- Email: [email protected]
soft surface interaction feedback in virtual environments
Mechatronics 24 (2014) 1092–1100
Contents lists available at ScienceDirect
Mechatronics journal homepage: www.elsevier.com/locate/mechatronics
Pseudo-haptics for rigid tool/soft surface interaction feedback in virtual environments Min Li a,⇑, Maisarah B. Ridzuan a, Sina Sareh a, Lakmal D. Seneviratne a,c, Prokar Dasgupta b, Kaspar Althoefer a a b c
Centre for Robotics Research (CoRe), Department of Informatics, Kings College London, London WC2R 2LS, UK Medical Research Council (MRC) Centre for Transplantation, King’s College London, King’s Health Partners, Guy’s Hospital, London SE1 9RT, UK College of Engineering, Khalifa University of Science, Technology and Research, Abu Dhabi, United Arab Emirates
a r t i c l e
i n f o
Article history: Received 7 January 2014 Accepted 21 July 2014 Available online 23 August 2014 Keywords: Haptic feedback Pseudo-haptic feedback Rigid tool/soft surface interaction Tumor identification
a b s t r a c t This paper proposes a novel pseudo-haptic soft surface stiffness simulation technique achieved by displaying the deformation of the soft surface and maneuvering an indenter avatar over a virtual soft surface by means of a touch-sensitive tablet. The visual feedback of the surface deformation and the alterations to the indenter avatar behavior produced by the proposed technique create the illusion of interaction with a hard inclusion embedded in the virtual soft surface. The proposed pseudo-haptics technique is validated with a series of experiments conducted by employing a tablet computer with an S-pen input and a tablet computer with a bare finger input. Tablet computers provide unique opportunities for presenting the pseudo-haptic (indenter avatar speed), haptic (contact reaction force from the device surface) and visual cues (surface information) at the same active point of interaction which facilitates information fusion. Hence, here, we evaluate the performance of tablet computers in identification of hard inclusions within virtual soft objects and compare it with the performance of a touchpad input device. A direct hand-soft surface interaction is used for benchmarking of this study. We found that compared with using a touchpad, both the sensitivity and the positive predictive value of the hard inclusion detection can be significantly improved by 33.3% and 13.9%, respectively, by employing tablet computers. Using tablet computers could produce results comparable to the direct hand-soft surface interaction in detecting hard inclusions in a soft object. The experimental results presented here confirm the potential of the proposed technique for conveying haptic information in rigid tool/soft surface interaction in virtual environments. Ó 2014 Elsevier Ltd. All rights reserved.
1. Introduction Pseudo-haptic methods create an illusion of haptic feedback via a visual display [1–4] without the need for the use of relatively complex, bulky and expensive haptic devices [5]. The modification of speed and size of the cursor for simulating bumps and holes using a computer mouse and a desktop computer was proposed by Lécuyer et al. [6]. By applying this method, image textures on the screen can be pseudo-haptically explored using a computer and a computer mouse. Shiota and Kashihara [7] proposed to generate pseudo-haptic feedback from drag operations on an iPad. The virtual object generally follows the finger movement. If the object ⇑ Corresponding author. Tel.: +44 (0)2078482902. E-mail addresses: [email protected], [email protected] (M. Li), maisarah_ [email protected] (M.B. Ridzuan), [email protected] (S. Sareh), lakmal. [email protected] (L.D. Seneviratne), [email protected] (P. Dasgupta), [email protected] (K. Althoefer). http://dx.doi.org/10.1016/j.mechatronics.2014.07.004 0957-4158/Ó 2014 Elsevier Ltd. All rights reserved.
moves slower in comparison with the finger movement, the user can feel the object is heavy. Thus, pseudo-haptic methods provide possible low-cost solutions for more realistic experiences in interactive virtual environments e.g. video games. In our previous research, this two-dimensional (2D) pseudo-haptic feedback was applied to soft tissue stiffness simulation [8]. The 2D hand motion was input by using a computer mouse while the visual information was displayed on a computer screen. By reducing the ratio between the indenter avatar (cursor) displacement and the input device (computer mouse) displacement, a resistance to motion when the indenter approaches an embedded hard nodule in soft tissue during sliding palpation was successfully simulated. Rigid tool/soft surface interactions contain both indenting and sliding behaviors. To simulate indenting behavior, the input of a third dimension of motion or force needs to be enabled. With the aid of pressure-sensitive technology, touchscreens can detect different force levels applied by the user to the surface. Sinclair et al. [9,10] developed a 3D touchscreen with force feedback and
1093
M. Li et al. / Mechatronics 24 (2014) 1092–1100
haptic texture. This device detects contact forces and provides force feedback by robotically moving the screen along the z-axis, providing opportunities for the simulation of the stiffness property of soft surfaces. Therefore, they proposed applying this technique to enable surgeons to ‘‘palpate’’ brain tissue when touching and looking at the MRI brain scan. Apart from the possible applications in hospitals, this device also has promising applications in museums for interactive experiences with untouchable, valuable art works and antiques. However, it is difficult to integrate this technique into compact, mobile, and wireless handheld devices that are becoming increasingly popular, e.g. tablet computers and smart phones. When interactions happen between a rigid tool and a soft surface, deformations occur at the soft surface side, where the change in the indentation depth and the deformation of the surface provide clues on the stiffness property of the surface [8]. To provide a more realistic feedback during rigid tool/soft surface interaction, visual feedback of tissue surface deformation is needed [11]. However, tissue surface deformation was not displayed in the aforementioned other feedback methods. The implementation of pseudo-haptic feedback and providing visual feedback of the soft surface deformation in portable devices such as smart phones and tablet computers is promising [2,12]. However, it is concerned by the poor pressure-sensitivity of some smart phones and tablet computers. Therefore, in some cases, separate pressure-sensitive input devices were introduced and the haptic interaction and the visual information were presented at different locations. For instance, Kimura and Nojima [2] has used a smart phone enhanced with two film-type force sensors to create an illusory sense of softness. This was realized by resizing of the rectangle displayed on the phone when the user squeezes the phone. Kokubun et al. [12] displayed the deformation of the black and white polygons-formed surface on a smart phone to generate pseudo-haptic sensations when pushing on a separate rear touch interface. Ridzuan et al. [13] proposed to convey stiffness information of soft surfaces by modifying the visual deformation depth of a virtual surface in accordance with the contact force and the stiffness property of the soft surfaces. This was implemented by using a tablet computer and strain gauges placed on both left and right sides of the support plate underneath. Compared to the other aforementioned pseudo-haptic methods, in Ridzuan et al. [13]’s system, the haptic and visual information was presented at the same active point of interaction when using a tablet computer. This made the users feel as though their finger or the stylus could penetrate the pressure-sensitive surface and be extended into the digital world to manipulate virtual tissues behind the screens directly which is called immersive illusion [13,14]. According to their experimental results, this method can produce a similar stiffness sensation as perceived when interacting with real soft surfaces. However, each virtual sample was assigned with one homogeneous stiffness property and only vertical interaction (indentation behavior) with the virtual surface was enabled during their experiments. In order to reduce the complexity of the system caused by a separate pressure-sensitive input device, tablet computers with higher quality of the pressure-sensitivity should be employed. In this paper, we propose to combine the sliding and indenting rigid tool/soft surface interactions to convey a non-homogeneously distributed stiffness property of soft surface. This is implemented by integrating the pseudo-haptic feedback described in our previous report [8] with the visualization of soft surface deformation and employing tablet computers. Soft surface stiffness data is acquired using a rolling/sliding indentation method [15–18] and then conveyed using a force-sensitive tablet computer. The major advantage of the proposed technique is that both the haptic and visual cues are presented at the same active point of interaction. Other advantages include compactness, portability, wireless
operation, ease-of-use and low cost. The effectiveness of the proposed technique is evaluated by the measure of its ability to identify hard inclusions within soft surfaces in a simulation. In order to demonstrate the beneficial effects of immersive illusion [13,14] on pseudo-haptic sensation, we compare the performance of tablet computers in identification of hard inclusions with a touchpad input device. Note that in the case of using a touchpad input device, the contact force is exerted from the touchpad while the visual information is displayed on a computer screen. Here, hand-soft surface interaction was used for experimental benchmarking. The potential applications of our approach range from interactive gaming to medical training. In Section 2.1, two types of reaction forces between the rigid tool tip and soft surfaces are simulated. Sections 2.2 and 2.3 explain the concept and algorithm of applying pseudo-haptics to the soft surface stiffness simulation and the hard inclusions identification process by using a force-sensitive tablet computer and a force-sensitive touchpad. Section 3 describes the process of soft surface stiffness data acquisition, the validation test protocol; Section 4 presents the results of the validation studies. Conclusions are described in Section 5. 2. Methodology 2.1. Rigid tool/soft surface interaction The soft surface stiffness 3D haptic data which is used for the purposes of this paper comes from the rolling/sliding indentation method introduced by [15–18] to detect tissue abnormalities. The rolling indentation method works as follows: During the superficial rolling/sliding, the indenter slides over a soft object surface. As the indenter approaches a hard inclusion embedded in the soft object an increase of lateral force fx is experienced (see Fig. 1). During indenting behavior, a tool is pressed on the surface of the soft object to explore its stiffness pattern. The reaction force fy increases as the indentation depth increases. When the areas with hard inclusions underneath are pressed, the reaction forces are greater than those of other areas which are free from inclusions. In our previous research [8], the lateral reaction force fx of the rolling behavior and the normal reaction force fy of the indenting behavior were simulated and visualized separately. In this study, we have implemented simultaneous simulation of the two forces fx and fy to provide a more intuitive visual feedback to the user. 2.2. Pseudo-haptic soft surface stiffness simulation using tablet computers When the user moves the input pen or his/her finger on the tablet screen towards a relatively stiffer area from Po to P, the indenter avatar moves slower following the input (see Fig. 2), and an illusion of resistance to motion will be experienced. In this way, the virtual
v f
Indenter
Soft Object
fx
fy Indentation Depth P
Fig. 1. Reaction forces in rigid tool/soft surface interaction.
1094
M. Li et al. / Mechatronics 24 (2014) 1092–1100
i-3x i-2x
i-x i-3
i-1
i
i-2
i+1
i+2
i+3
i+x
Fig. 2. Indenter avatar speed-changing strategy and visualization of soft surface deformation using a tablet computer: the input pen moves from Po to P; P0 has the same stiffness as Po and P has a higher stiffness than Po; the indenter avatar first follows the input pen from Po to P0 and then moves from P0 to Pm; VF is the virtual force generated by using pseudo-haptic feedback algorithm.
forces (VF) can be perceived visually along with the direction of the movement. The flow chart of the pseudo-haptic soft surface stiffness simulation using a force-sensitive tablet computer is shown in Fig. 3. The user moves the input pen or a fingertip tangent to the surface whilst applying a normal force on the force-sensitive tablet computer. The normal force exerted on the tablet surface, together with the 2-DOF movement kinematics of the pen tip over the surface of the tablet computer are the two inputs of the system. The outputs are the normal reaction force from the tablet surface, the virtual resistance along the movement direction generated by using pseudo-haptic feedback, and the soft surface deformation shown on the screen. It should be mentioned that, here, the deformation of the soft surface is calculated based on the indentation depth, the model of soft object surface deformation curvature, and the avatar movement update. The tissue curvature is calculated based on the indentation depth and the model of soft object surface deformation curvature [19]. The deformation of the soft surface during interaction is displayed in real time using a geometrical deformable soft surface model, which is established based on a predefined finite element model considering the indentation depth and the indenter diameter. The details of this model are presented in [19]. As a node of the mesh is pressed by the indenter, the normal vertex of this node is redefined according to the depth of the indenter (see Figs. 4 and 5). Note that, as a result of the indentation of a specific node, the normal vertices of nodes located nearby on the mesh are also affected. If the indentation depth increases, the number of affected
Pre-Measured Stiffness Distribution
Model of Deformation Indentation Depth Curvature of Soft Object Surface Stiffness Value
Force Value Mapping of Haptic Surface
Position
Force Value
Pseudo-haptic Algorithm
Indenter Avatar Position
Soft Object Surface Curvature Soft Object Deformation Display
2-DOF Pen Movement
Force Level
Tangent Hand Motion and Normal Force Reaction force Input Fig. 3. The flow chart of the pseudo-haptic palpation simulation using a forcesensitive tablet computer.
i+2x i+3x Fig. 4. The location of the indenter avatar is on node i; the normal vertex of this node is redefined according to the depth of the indenter; the normal vertices of nodes located nearby on the mesh are also affected; as the indentation depth increases, the number of affected nodes increases.
nodes will also increase. Thus, the pattern of the deformation of soft surface is displayed graphically. It should be mentioned that if the surface is displayed parallel to the tablet surface, the deformation cannot be visually captured. Here, we display the surface using a perspective view with an angle of 45° towards the user to effectively visualize the deformation. Two prominent types of commercially available tablet computers are used in this study – Samsung Galaxy Note 10.1 (using an Spen) and Motorola Xoom (using the user’s bare finger). Samsung Galaxy Note 10.1 has a dimension of 262 180 8.9 mm and a weight of 600 g. Motorola Xoom is 249 167.8 12.9 mm and 730 g. Samsung Note (using a Wacom S-pen) has 1024 levels of pressure sensitivity. Motorola Xoom tablet senses the area of touch: when the touched area is broader, it recognizes the applied force as larger. The return value from the tablet computers is the force level, which needs to be mapped with force magnitude. A Force/Torque sensor ATI Mini 40 (a 6-DOF sensor with a force sensing range of ±10 N in x and y direction, ±30 N in z direction and a torque sensing range of ±0.5 nm; normal force resolution is 0.01 N) was placed under the tablet computer to record the applied force and map the force levels defined by the tablet computer to actual force values. Since the dimension of the force sensor (40 mm in diameter) are smaller than the tablets, two plastic plates were fixed to the two sides of the force sensor to increase the contact area between the force sensor and the touchpad as well as the table surface underneath. During the experiment, a user held the pen or used an index finger to move tangentially over the tablet’s force-sensitive surface, hence, exerting a normal force on it. In order to avoid torques transmitted to the force sensor when pressing at the edges of the tablet, the pressing was done exactly above the sensor during the mapping process. The normal force values fz were recorded via a data acquisition card (NI PCI-6013) and a LABVIEW program and stored in a text file. Force levels were captured via an Android program and stored in another text file. The force applied on the tablet surface was approximately a sawtooth waveform with different amplitude each pressing time. The peaks of the force values acquired from the Mini 40 force sensor and the force levels read from the tablet computer were used as the synchronization markers. The force level-force value relationships are described in Eqs. (1) and (2) (see Figs. 6 and 7), according to which the force levels (fl) are converted to the normal force values (fn). Then the indentation depth dm (z) is calculated as a function of the pressure applied to the tablet (fn) employing a lookup table of force matrices linearly interpolated between stored values:
M. Li et al. / Mechatronics 24 (2014) 1092–1100
1095
Fig. 5. The indenter avatar (sphere) and the deformation of the soft object surface; as the indentation depth increases, the affected tissue surface area becomes larger.
where fl is the force level reading using the getPressure() method in Android SDK and fn is the corresponding normal force. The modification of the indenter movement is realized by adding a time delay to the indenter avatar’s rendering task when the indenter itself is approaching a stiffer area. If the indenter passes over the stiffer area, the indenter avatar continues to follow the contact point. The delay time is determined by Eq. (3):
4.5 4.0 3.5
y = 0.1008e4.2081x R² = 0.9243
Force (N)
3.0 2.5 2.0 1.5
td ¼ Df m;
ð3Þ
Df ¼ f f l ;
ð4Þ
1.0 0.5 0.0
0
0.2
0.4
0.6
0.8
1
Force level Fig. 6. Force level and force value mapping of the Samsung Note tablet (using an Spen).
14 12
y = 0.0772e3.0727x R² = 0.9423
Force (N)
10 8 6 4 2 0 0.0
0.5
1.0
1.5
2.0
Force level Fig. 7. Force level and force value mapping of the Motorola Xoom tablet (using a bare finger of the user).
f n ¼ 0:1008e4:2081f l ;
ð1Þ
f n ¼ 0:0772e3:0727f l ;
ð2Þ
where the value of the reaction force f is acquired from the resultant reaction force matrices formed during the rolling indentation; fl is the reaction force value at the last avatar position; Df is the reaction force difference; and m is a scalar value. The calculated time delay is then added to the program frame interval time. To ensure the user can notice the indenter avatar when it lags behind the contact point, the minimum delay time should be set higher than the time interval between frame updates, the frame interval for this program is 30 ms. Due to the phenomenon of persistence of vision of the eye, an afterimage is thought to persist for approximately one twentyfifth of a second on the retina, the delay time needs to be set to be higher than 40 ms. Here we set to be 50 ms. It is designed that the user should notice the minimum reaction force difference of 0.1 N, thus, m is 50/0.1 = 500. We tested the movement speed of the S-pen on the tablet surface. The movement speed was ranging from about 27 mm/s to 36.5 mm/s. Here, actively adding delay is a means to reduce the movement speed of the indenter avatar compared to the hand movement when the stiffness is increasing. It adds a small distance between the hand position and the indenter avatar position. Apart from the reaction force difference Df, the total time delay is also determined by the nodule size. For example, to move over a hard nodule with a diameter of 10 mm, 185 ms to 137 ms (depending on the movement speed) is needed to reach the middle of the hard nodule where the reaction force difference becomes negative. After this stage, the indenter avatar again follows the hand movement and the delay time is less than 185 ms. It should be mentioned that the delay time was chosen to be sufficiently small (just enough to be noticeable to the users and enable
1096
M. Li et al. / Mechatronics 24 (2014) 1092–1100
them to identify hard nodules) to minimize its negative influence on the user’s workload. Fig. 8 shows the user interfaces of the pseudo-haptic soft surface stiffness simulation using the two tablet computers. Human fingertips are larger than the S-pen tip. Touching the screen with a bare finger can sometimes obstruct the vision of the user. In order to prevent the finger of the user from obstructing the view, the indenter avatar is set to be apart from the interaction point at a 15 mm distance.
Input Pen Po
Indenter Avatar
S-pen
(a)
VF
Touchpad
Indenter Avatar
(b) Fig. 8. Pseudo-haptic soft surface stiffness simulation using tablet computers: (a) Samsung Galaxy Note 10.1 (using an S-pen) and (b) Motorola Xoom (using a bare finger of the user).
Pm’ P’
dm do
Graphical interface on a computer monitor Fig. 9. Pseudo-haptic feedback using a touchpad device: the avatar display distance dm; VF is the virtual force generated by using pseudo-haptic feedback algorithm.
5 4.5
y = 0.0827e0.0039x R² = 0.9703
4
Force (N)
3.5 3 2.5 2 1.5 1 0.5 0 0
200
400
600
800
1000
1200
Force level Fig. 10. Force level and force value mapping of the force-sensitive touchpad.
f n ¼ 0:0827e0:0039f l ;
ð5Þ
where normal reaction force values (fn) is acquired from the reaction force matrices in sliding indentation [15–18]; fl is the force level reading from the touchpad data package. The coordinates of the touchpad surface are linearly mapped to the soft object surface. The force levels ( fl) read from the touchpad data package are converted to the normal force values ( fn) according to the Eq. (5) first. The position of the interaction on the surface of the soft object is calculated based on the mapping relationship between the touchpad surface and the soft object surface. Then the indentation depth dm (z) is calculated as a function of the pressure applied to the touchpad ( fn) employing a look-up table of force matrices linearly interpolated between stored values. The resultant reaction force can be acquired via the lookup table of force matrices according to the indentation depth dm (z). Then, the difference of the reaction forces (Df) between the current avatar position and the last avatar position is calculated as:
Df ¼ f f l ; 15 mm distance
Po’
D
P
2.3. Pseudo-haptic soft surface stiffness simulation using a forcesensitive touchpad motion input device In order to prove the beneficial effects of immersive illusion [13,14] on pseudo-haptic sensation using tablet computers, a touchpad is used as a control condition. As shown in Fig. 9, using a touchpad as a vertical force and horizontal motion input device, visual and haptic information are presented at different point of interaction – the contact force is exerted from the touchpad via the special pen while the visual information displays on a computer monitor. When the user moves the input device towards a relatively stiffer area over a distance (D), the indenter avatar display ratio will be updated to be lower than the original default ratio (Rm < Ro, Rm is the modified ratio and Ro is the original default ratio). Thus, the modified avatar displacement dm will be lower than the original default indenter avatar displacement do (dm=Rm D, do = Ro D). An illusion of resistance to motion will be experienced. In this way, the virtual forces (VF) can be perceived visually along with the direction of the movement. A force-sensitive Wacom BAMBOO Pen & Touch touchpad (a 2D haptic input device, 248 176 8.5 mm in dimension, 125 85 mm for touch sensitivity, 360 g weight) is used here. The touchpad has 1024 force levels. Similar test as described in Section 2.2 was conducted to map the force levels read from the touchpad to force magnitudes. 57 sets of data points were obtained. The least square method was used to produce the regression Eq. described in Eq. (5) (R2 = 0.9703, see Fig. 10).
Indenter Avatar
dm = Rm ∙ D, do = Ro ∙ D
ð6Þ
where fl is the reaction force value at the current avatar position; fl is the reaction force value at the last avatar position. When the soft surface stiffness in current position is stiffer than the tissue in the previous position (Df > 0), the movement distance is reduced:
dm ðx; yÞ ¼ Rmt Dðx; yÞ;
ð7Þ
Rmt ¼ Ro =ðf þ 1Þ;
ð8Þ
where Ro is the original default ratio between the avatar displacement and the input device displacement (avatar display ratio) calculated based on the coordinates mapping relationship between
1097
M. Li et al. / Mechatronics 24 (2014) 1092–1100
the touchpad surface and the soft object surface; and Rmt is the modified avatar tangent display ratio. When the soft surface in current position has the same stiffness or is softer than in the previous position (Df 6 0), the position of the indenter avatar is calculated based on the coordinates mapping relationship between the touchpad surface and the soft object surface. Thus, there is a jump between the previous position and the current position. This jump is used as a helpful sign for the user to locate hard nodules. 3. Evaluation experiments 3.1. Measuring soft surface stiffness distribution The stiffness distribution matrices used in the pseudo-haptic soft surface stiffness simulation corresponded to a 120 120 25 mm3 silicone block with three spherical nodules embedded inside (see Fig. 11). The phantom was manufactured using RTV6166 (TECHSIL Limited, UK) (A: B = 3: 7). The nodules were made from STAEDTLER Mars plastic 526 50 (47–50 Shore A) rubber. The diameters of the spherical nodules used were 10 mm, 8 mm, and 6 mm, respectively. All of the nodules were buried at a depth of 6 mm; this distance was measured from the top of the each nodule to the silicone surface. To obtain a rolling/sliding stiffness map [15–17], 59 straight-line trajectories (121 mm long and parallel to the x-axis with an interval of 2 mm between trajectories along the y-axis) were defined. A robot arm (Mitsubishi RV-6SL 6-DOF robotic manipulator with a positioning accuracy of 0.01 mm) was programmed to move attached the sliding indentation probe along these 59 straight-line trajectories at a speed of 30 mm/s with a constant rolling indentation depth. The silicone phantom tissue was covered by a transparent plastic sheet and was lubricated during the sliding indentation process to reduce the friction. The normal and tangent components of reaction forces were recorded at a sampling rate of 100 Hz using a data acquisition card (NI PCI-6013) and a LABVIEW program. The data acquired here then was processed in MATLAB by using linear interpolation to make the number of data points on each line to be 159. This experiment was repeated for a range of rolling indentation depths between 2 mm and 7 mm with intervals of 1 mm. Six 159 59 normal and tangent force distribution matrices were created, which allowed us to obtain a stiffness distribution map for the whole silicone block surface. Here, force distribution matrices were recorded using a Nano17-ATI force sensor (resolution: 0.003 N). Subsequently, this information was used in the algorithm of pseudo-haptic feedback. 3.2. Test protocol In order to validate our pseudo-haptic soft surface stiffness simulation approach, a series of experiments were conducted
Sliding Indentation Probe
Transparent Plastic Sheet
y o
B A
C
x Fig. 11. The silicone block and the locations of the three embedded hard inclusions (A, B, C).
employing the aforementioned three types of 2-DOF force-sensitive haptic surfaces. Pseudo-haptic feedback relies on visual display, so people with uncorrectable visual impairment cannot participant in the test. Here, two groups of participants, those who have normal vision and those with corrected vision, participated in an empirical study. The first group (Group I) consisted of twenty participants of which nineteen right-handed, one left-handed, one with palpation experience, and all with an engineering background while the second group (Group II) consisted of twenty participants, all right-handed, with an engineering background but no palpation experience (see Table 1). Group I conducted the tests of manual hard inclusions detection using direct hand touch (Manual I) and the pseudo-haptic soft surface stiffness simulation using a force-sensitive touchpad (Touchpad). Group II participated in the tests of manual hard inclusions detection using direct hand touch (Manual II) and the pseudo-haptic soft surface stiffness simulation using a tablet with an S-pen input (S-pen) and a tablet with a bare finger input (Bare finger). Prior to the start of each test, participants were also asked to do a practice run with known tumor locations. Subsequently, they were requested to manipulate the input device, palpate the virtual soft surface and observe the change of the ratio between the indenter avatar displacement and the input device displacement. These participants were also asked to report the positions of hard inclusions, where they detected them. Throughout the whole program of tests, we recorded the correctly and wrongly identified hard inclusions. It is worth mentioning that during all the aforementioned tests, the same stiffness map applied (one silicone/nodule block was used), but the orientation of the stiffness map was different from test to test: so the participants would not learn the nodules’ locations from the earlier tests. In order to eliminate the influence of surface orientation on the nodule detection performance, the surface orientation was pseudo-randomly changed during the experiment. For each participant, the tests were conducted in a pseudo random order. During the manual hard inclusions detection test, the transparent silicone block surface was covered by a black plastic sheet to prevent the locations of the hard inclusions from being visible to the user. 4. Results and discussion 4.1. Results Sensitivity Se [20], which relates to the test’s ability to identify positive results of the presence of hard nodules, is defined as sum over all the n trials of the True Positives (TP) divided by the actual number of hard inclusions or the sum of TP and False Negatives (FN), namely: n X Se ¼ TP i i¼1
, n X ðTP i þ FNi Þ:
ð9Þ
i¼1
Se was calculated for each feedback method as the proportion of true positives over the total of all trials conducted by all participants. The overall Se of each test is shown in Fig. 12. Table 1 Overview of demographics and experience of the two groups. Item
Group I
Group II
Age range Gender Handedness Palpation experience Engineering background VR simulator
23–42 $: 6; #: 14 R: 19; L: 1 1 20 0
20–30 $: 7; #: 13 R: 20; L: 0 0 20 0
M. Li et al. / Mechatronics 24 (2014) 1092–1100
Wilson score intervals [21] can be used to test the difference of two proportions [22] and have good properties even for a small number of trials (less than 30) and/or an extreme probability [23,24]. In this study, Wilson score intervals were calculated for sensitivity at a 95% confidence level using the following formula:
rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi# ^ð1 p ^Þ z2 z2 p ^þ z þ 2 ; p 2n 4n n
ð10Þ
^ is the proportion of successes estimated from the statistical where p sample; z is the 1 a/2 percentile of a standard normal distribution; a is the error percentile and n is the sample size. Here, since the confidence level is 95%, the error a is 5%. The best Se was achieved with the tablet and S-pen (91.7%). The Se of each pairs of tests was compared. It was conducted by comparing the observed probabilities (p1 and p2) with a combined interval (CI), which was calculated by the following formula [25]:
CI ¼
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðP1 p1 Þ2 þ ðP2 p2 Þ2 ;
ð11Þ
where P1 is the upper bound of p1 and P2 is the lower bound of p2 if p1 < p2. If |p1–p2| > CI, there is a significant difference between the two tests. Table 2 shows the test results. In comparison with Group II, Group I had a higher Se (88.3% vs. 73.3%). However, the two methods had no significant difference in Se (CI = 0.148, Dp = 0.100). This proved that there was no significant difference in the touch perception abilities of these two groups. One can notice that both the Se of using tablet and S-pen, and tablet and bare finger were significantly higher than using a touchpad. Thus, direct touch and immersive illusion was proven to be superior to when visual and haptic information did not present at the same point of interaction. In comparison with using a tablet with a bare finger input, using a tablet with an S-pen input had higher Se (91.7% vs. 85%). However, the difference was not significant (CI = 0.157, Dp = 0.067). Fig. 13 compares the detection sensitivity of nodules A, B and C through the different pseudo-haptic feedback methods. Using the touchpad, the Se of nodule B was the highest (75%), followed by nodule A (65%); it was interesting to note that nodule B had a higher Se despite being smaller than nodule A. The smallest tumor C had a low Se of 15%. Using the tablet with an S-pen input, the Se of both nodule A and B were 100%. The smallest tumor, C, had a Se of 75%. Using the tablet with a bare finger input, both the Se of nodule A and B were 95%. The smallest tumor C had a Se of 65%.
100%
91.7%
Sensivity
80% 60%
85.0%
83.3% 73.3%
Item
Combined interval (CI)
Probability difference (Dp)
Significance?
Manual II & Manual I Manual I & Touchpad S-pen & Bare finger S-pen & Touchpad S-pen & Manual II Bare finger & Touchpad Bare finger & Manual II
0.148 0.167 0.157 0.156 0.136 0.165 0.146
0.100 0.316 0.067 0.400 0.184 0.333 0.117
No Yes No Yes Yes Yes No
120% 100% 80%
Touchpad
60%
S-pen
40%
Bare finger
20%
Manual I Manual II
0% A
B
C
Hard nodules Fig. 13. Nodule detection sensitivities of nodule A, B, and C with Wilson score intervals at a 95% confidence level.
Fig. 14 presents the Positive Predictive Values PPV [26], or precision rates, which are defined as sum over all the n trials of the TP divided by the test outcome positive or the sum of TP and False Positives (FP) – participants claim there is a hard nodule when there is no one, namely:
PPV ¼
n X
, TPi
i¼1
n X ðTP i þ FPi Þ:
ð12Þ
i¼1
PPV was calculated as the proportion of true positives over the sum of true positives and false negatives in all trails. The best PPV was achieved by using the tablet with an S-pen input (100% with 95% confidential interval: 93.5–100%) followed by using the tablet with a bare finger input (100%, 93.0–100%)). Using the touchpad had the lowest PPV (86.1%: 71.3–97.1%). The PPV using the touchpad (CI = 0.139, Dp = 0.167; CI = 0.142, Dp = 0.167; CI = 0.142, Dp = 0.167; CI = 0.145, Dp = 0.167) was significantly lower than the other four tests. This proves that the performance of hard nodule detection using tablet computers has no significant difference from the gold standard direct hand-soft object interaction (Manual I and Manual II). Also, the tablet computer enabled presenting visual cues and contact forces at the same interaction point and
51.7%
40% 20% 0%
Touchpad
S-pen
Bare finger
Manual I
Manual II
Feedback Method Fig. 12. Overall nodule detection sensitivities with Wilson score intervals at a 95% confidence level: touchpad represents the pseudo-haptic soft surface stiffness simulation using a touchpad; S-pen represents the experiment using a tablet with a motion and force input from an S-pen; bare finger represents the experiment using a tablet with a motion and force input from a bare finger of the user; Manual I represents hard inclusions detection using hand-soft surface interaction conducted by Group I; Manual II represents hard inclusions detection using hand-soft surface interaction conducted by Group II.
Posive predicve value
1 2 1 þ zn
"
Table 2 Comparison of sensitivities.
Sensivity
1098
110%
100.0%
100.0%
100.0%
100.0%
Bare finger
Manual I
Manual II
100% 90%
86.1%
80% 70% 60% 50% Touchpad
S-pen
Feedback method Fig. 14. Positive predictive values with Wilson score intervals at a 95% confidence level.
M. Li et al. / Mechatronics 24 (2014) 1092–1100
achieving significantly better performance than the case of exerting contact force from the touchpad while displaying visual information on a computer monitor. 4.2. Discussion Using a bare finger to provide the indentation and sliding input makes the contact more natural than using a pen. However, we noticed that the Se of using a tablet and a bare finger was lower than that of using a tablet and an S-pen. One reason for this is that the fingertips are larger than the S-pen tip. Touching the screen with a bare finger can sometimes obstruct the vision of the user during the experiments. Another reason can be the saturation problem of the pressure level measurement of this device, namely, using the contact surface area to estimate the contact force is not reliable when the contact area reaches its maximum while the contact force is still increasing. Hence, a better pressure-sensitive touch screen for bare finger interaction is required. Due to the fragility of the tablets, accurate robots were not used to press the tablet surfaces during the force/force level mapping experiments. Since humans might add jitters when pressing on the tablets, multiple tests were conducted with the same human hand to minimize the negative influence. Motorola Xoom tablet senses the area of touch: when the touched area is broader, it recognizes the applied force as larger. For Motorola Xoom tablet, ideally, calibration needs to be done for each user before using the system for greater accuracy. Here, to make our experiment more time-efficient for each user, the calibration was done only once prior to the start of the experiments. Although the size of contact area would vary among different users for the same amount of applied force, there would be a recognizable difference in size of the contact area between light and hard pressings. With this calibration result, the size of contact area can still be used to estimate the magnitude of normal force for the deformation of the virtual soft object. A perspective view of the surface was rendered using OpenGL. This surface and the indenter avatar were smaller at the rear end than in the front because of the increase of the distance to the observer. Since the smaller interaction area at the rear end compared with the front part, the perception may be affected. While the tissue surface was 120 120, the hard nodules were all located at the central area (60 60) of the virtual tissue. During the experiment, the participants were told to mainly explore the central area of the virtual tissue. Therefore, we assume the decrease of interaction accuracy for the area at the rear end did not have significant influence on our experimental results. 5. Conclusions The performance of the proposed pseudo-haptics rigid tool/soft surface interaction technique has been assessed using a force-sensitive touchpad with a pen input, a tablet computer with an S-pen input, and a tablet computer with a user’s bare finger input. Hard inclusions were detected more effectively on tablet computers. Sensitivity and positive predictive value were lower when the force-sensitive touchpad was used as visual and haptic information was presented at different points of interaction. Notably, the detection of hard inclusions done by applying direct touch or immersive illusion on tablet computers was comparable to detection performed with the use of hand/soft surface interaction. The proposed technique impresses with its performance in the detection of hard inclusions which rivals detection done via hand/ soft surface interaction. With the video gaming community always on the lookout for more realistic experiences, our proposed technique is sure to find many applications in gaming.
1099
Acknowledgements The work described in this paper was partially funded by the Vattikuti Foundation, the GSTT Charity, National Institute for Health Research (NIHR) Biomedical Research Centre based at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London, the European Commission’s Seventh Framework Programme under grant agreement 287728 in the framework of EU Project STIFFFLOP, and the China Scholarship Council. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.
References [1] Lécuyer A, Coquillart S, Kheddar A, Richard P, Coiffet P. Pseudo-haptic feedback: can isometric input devices simulate force feedback? In: Proceedings IEEE virtual reality. IEEE Comput. Soc; 2000. p. 83–90. [2] Kimura T, Nojima T. Pseudo-haptic feedback on softness induced by grasping motion. In: EuroHaptics; 2012. p. 202–5. [3] Dominjon L, Lecuyer A, Burkhardt J, Richard P, Richir S. Influence of control/ display ratio on the perception of mass of manipulated objects in virtual environments. In: Proceedings of the IEEE virtual reality; 2005. p. 19–25. [4] Argelaguet F, Jáuregui D. Elastic images: perceiving local elasticity of images through a novel pseudo-haptic deformation effect. ACM Trans Appl Percept 2013;10(3):17(1-14). [5] Khan S. Design and optimization of parallel haptic devices. 2012. p. 6. [6] Lécuyer A, Burkhardt JM, Tan CH. A study of the modification of the speed and size of the cursor for simulating pseudo-haptic bumps and holes. ACM Trans Appl Percept 2008;5(3):1–21. [7] Shiota G, Kashihara A. Cognitive effects of concept map generation with pseudo-haptic feedback. In: 2013 12th International conference on information technology based higher education and training (ITHET); 2013. p. 1–6. [8] Li M, Liu H, Seneviratne LD, Althoefer K. Tissue stiffness simulation and abnormality localization using pseudo-haptic feedback. In: IEEE international conference on robotics and automation. Saint Paul, Minnesota, USA; 2012. p. 5359–64. [9] Sinclair M, Pahud M, Benko H. 3D Touchscreen with Force Feedback and Haptic Texture. In: Proceedings of the 2013 ACM international conference on interactive tabletops and surfaces. St Andrews, United Kingdom; 2013. p. 287–96. [10] Sinclair M, Pahud M, Benko H. TouchMover: actuated 3D touchscreen with haptic feedback. In: Proceedings of the 2013 ACM international conference on interactive tabletops and surfaces; 2013. p. 287–96. [11] Li A, Sareh S, Ridzuan M, Seneviratne LD, Dasgupta P, Wurdemann HA, Althoefer K. Multi-fingered palpation using pseudo-haptic feedback. In: Hamyln symposium on medical robotics; 2014. Accepted. [12] Kokubun A, Ban Y, Narumi T, Tanikawa T, Hirose M. Visuo-haptic interaction with mobile rear touch interface. In: ACM SIGGRAPH 2013 Posters on – SIGGRAPH ’13. New York, NY, USA: ACM Press; 2013. p. 38. [13] Ridzuan M, Makino Y, Takemura K. Direct touch haptic display using immersive illusion with interactive virtual finger. In: EuroHaptics; 2012. p. 432–44. [14] Withana A, Makino Y, Kondo M, Sugimoto M, Kakehi G, Inami M. ImpAct: immersive haptic stylus to enable direct touch and manipulation for surface computing. Comput Entertain 2010;8(2):1. [15] Liu H, Noonan DP, Challacombe BJ, Dasgupta P, Seneviratne LD, Althoefer K. Rolling mechanical imaging for tissue abnormality localization during minimally invasive surgery. IEEE Trans Biomed Eng 2010;57(2):404–14. [16] Liu H, Li J, Poon Q, Seneviratne LD, Althoefer K. Miniaturized force-indentation depth sensor for tissue abnormality identification during laparoscopic surgery. In: International conference on robotics and automation (ICRA); 2010. p. 3654–9. [17] Noonan DP, Liu H, Zweiri YH, Althoefer KA, Seneviratne LD. A dual-function wheeled probe for tissue viscoelastic property identification during minimally invasive surgery. In: IEEE international conference on robotics and automation. Institute of Electrical and Electronics Engineers Inc.; 2007. p. 2629–34. [18] Liu H, Li J, Song X, Seneviratne LD, Althoefer K. Rolling indentation probe for tissue abnormality identification during minimally invasive surgery. IEEE Trans Robot 2011;27(3):450–60. [19] Li M, Seneviratne LD, Dasgupta P, Althoefer KA. Virtual palpation system. In: International conference on intelligent robots and systems workshop ‘‘Learning and Interaction in Haptic Robots’’; 2012. [20] Altman D, Bland J. Diagnostic test 1: sensitivity and specificity. BMJ 1994;308:1552. [21] Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc 1927;22:209–12. [22] Beauliu-Prevost D. Confidence Intervals: from tests of statistical significance to confidence intervals, range hypotheses and substantial effects. Tutorials Quantitative Methods Psychology 2006;2(1):11–9.
1100
M. Li et al. / Mechatronics 24 (2014) 1092–1100
[23] Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat Med 1998;17:857–72. [24] Newcombe RG, Altman DG. Proportions and their differences. In: Mltman DG, Machin D, Bryant TN, Gardner MJ, editors. Statistics with confidence: confidence intervals and statistical guidelines. London: BMJ Books; 2000. p. 46–8.
[25] Wallis S. Binomial confidence intervals and contingency tests: mathematical fundamentals and the evaluation of alternative methods. J Quant Linguist 2013;20(3):178–208. [26] Fawcett T. An introduction to ROC analysis. Pattern Recognit Lett 2006;27(8):861–74.
Contents lists available at ScienceDirect
Mechatronics journal homepage: www.elsevier.com/locate/mechatronics
Pseudo-haptics for rigid tool/soft surface interaction feedback in virtual environments Min Li a,⇑, Maisarah B. Ridzuan a, Sina Sareh a, Lakmal D. Seneviratne a,c, Prokar Dasgupta b, Kaspar Althoefer a a b c
Centre for Robotics Research (CoRe), Department of Informatics, Kings College London, London WC2R 2LS, UK Medical Research Council (MRC) Centre for Transplantation, King’s College London, King’s Health Partners, Guy’s Hospital, London SE1 9RT, UK College of Engineering, Khalifa University of Science, Technology and Research, Abu Dhabi, United Arab Emirates
a r t i c l e
i n f o
Article history: Received 7 January 2014 Accepted 21 July 2014 Available online 23 August 2014 Keywords: Haptic feedback Pseudo-haptic feedback Rigid tool/soft surface interaction Tumor identification
a b s t r a c t This paper proposes a novel pseudo-haptic soft surface stiffness simulation technique achieved by displaying the deformation of the soft surface and maneuvering an indenter avatar over a virtual soft surface by means of a touch-sensitive tablet. The visual feedback of the surface deformation and the alterations to the indenter avatar behavior produced by the proposed technique create the illusion of interaction with a hard inclusion embedded in the virtual soft surface. The proposed pseudo-haptics technique is validated with a series of experiments conducted by employing a tablet computer with an S-pen input and a tablet computer with a bare finger input. Tablet computers provide unique opportunities for presenting the pseudo-haptic (indenter avatar speed), haptic (contact reaction force from the device surface) and visual cues (surface information) at the same active point of interaction which facilitates information fusion. Hence, here, we evaluate the performance of tablet computers in identification of hard inclusions within virtual soft objects and compare it with the performance of a touchpad input device. A direct hand-soft surface interaction is used for benchmarking of this study. We found that compared with using a touchpad, both the sensitivity and the positive predictive value of the hard inclusion detection can be significantly improved by 33.3% and 13.9%, respectively, by employing tablet computers. Using tablet computers could produce results comparable to the direct hand-soft surface interaction in detecting hard inclusions in a soft object. The experimental results presented here confirm the potential of the proposed technique for conveying haptic information in rigid tool/soft surface interaction in virtual environments. Ó 2014 Elsevier Ltd. All rights reserved.
1. Introduction Pseudo-haptic methods create an illusion of haptic feedback via a visual display [1–4] without the need for the use of relatively complex, bulky and expensive haptic devices [5]. The modification of speed and size of the cursor for simulating bumps and holes using a computer mouse and a desktop computer was proposed by Lécuyer et al. [6]. By applying this method, image textures on the screen can be pseudo-haptically explored using a computer and a computer mouse. Shiota and Kashihara [7] proposed to generate pseudo-haptic feedback from drag operations on an iPad. The virtual object generally follows the finger movement. If the object ⇑ Corresponding author. Tel.: +44 (0)2078482902. E-mail addresses: [email protected], [email protected] (M. Li), maisarah_ [email protected] (M.B. Ridzuan), [email protected] (S. Sareh), lakmal. [email protected] (L.D. Seneviratne), [email protected] (P. Dasgupta), [email protected] (K. Althoefer). http://dx.doi.org/10.1016/j.mechatronics.2014.07.004 0957-4158/Ó 2014 Elsevier Ltd. All rights reserved.
moves slower in comparison with the finger movement, the user can feel the object is heavy. Thus, pseudo-haptic methods provide possible low-cost solutions for more realistic experiences in interactive virtual environments e.g. video games. In our previous research, this two-dimensional (2D) pseudo-haptic feedback was applied to soft tissue stiffness simulation [8]. The 2D hand motion was input by using a computer mouse while the visual information was displayed on a computer screen. By reducing the ratio between the indenter avatar (cursor) displacement and the input device (computer mouse) displacement, a resistance to motion when the indenter approaches an embedded hard nodule in soft tissue during sliding palpation was successfully simulated. Rigid tool/soft surface interactions contain both indenting and sliding behaviors. To simulate indenting behavior, the input of a third dimension of motion or force needs to be enabled. With the aid of pressure-sensitive technology, touchscreens can detect different force levels applied by the user to the surface. Sinclair et al. [9,10] developed a 3D touchscreen with force feedback and
1093
M. Li et al. / Mechatronics 24 (2014) 1092–1100
haptic texture. This device detects contact forces and provides force feedback by robotically moving the screen along the z-axis, providing opportunities for the simulation of the stiffness property of soft surfaces. Therefore, they proposed applying this technique to enable surgeons to ‘‘palpate’’ brain tissue when touching and looking at the MRI brain scan. Apart from the possible applications in hospitals, this device also has promising applications in museums for interactive experiences with untouchable, valuable art works and antiques. However, it is difficult to integrate this technique into compact, mobile, and wireless handheld devices that are becoming increasingly popular, e.g. tablet computers and smart phones. When interactions happen between a rigid tool and a soft surface, deformations occur at the soft surface side, where the change in the indentation depth and the deformation of the surface provide clues on the stiffness property of the surface [8]. To provide a more realistic feedback during rigid tool/soft surface interaction, visual feedback of tissue surface deformation is needed [11]. However, tissue surface deformation was not displayed in the aforementioned other feedback methods. The implementation of pseudo-haptic feedback and providing visual feedback of the soft surface deformation in portable devices such as smart phones and tablet computers is promising [2,12]. However, it is concerned by the poor pressure-sensitivity of some smart phones and tablet computers. Therefore, in some cases, separate pressure-sensitive input devices were introduced and the haptic interaction and the visual information were presented at different locations. For instance, Kimura and Nojima [2] has used a smart phone enhanced with two film-type force sensors to create an illusory sense of softness. This was realized by resizing of the rectangle displayed on the phone when the user squeezes the phone. Kokubun et al. [12] displayed the deformation of the black and white polygons-formed surface on a smart phone to generate pseudo-haptic sensations when pushing on a separate rear touch interface. Ridzuan et al. [13] proposed to convey stiffness information of soft surfaces by modifying the visual deformation depth of a virtual surface in accordance with the contact force and the stiffness property of the soft surfaces. This was implemented by using a tablet computer and strain gauges placed on both left and right sides of the support plate underneath. Compared to the other aforementioned pseudo-haptic methods, in Ridzuan et al. [13]’s system, the haptic and visual information was presented at the same active point of interaction when using a tablet computer. This made the users feel as though their finger or the stylus could penetrate the pressure-sensitive surface and be extended into the digital world to manipulate virtual tissues behind the screens directly which is called immersive illusion [13,14]. According to their experimental results, this method can produce a similar stiffness sensation as perceived when interacting with real soft surfaces. However, each virtual sample was assigned with one homogeneous stiffness property and only vertical interaction (indentation behavior) with the virtual surface was enabled during their experiments. In order to reduce the complexity of the system caused by a separate pressure-sensitive input device, tablet computers with higher quality of the pressure-sensitivity should be employed. In this paper, we propose to combine the sliding and indenting rigid tool/soft surface interactions to convey a non-homogeneously distributed stiffness property of soft surface. This is implemented by integrating the pseudo-haptic feedback described in our previous report [8] with the visualization of soft surface deformation and employing tablet computers. Soft surface stiffness data is acquired using a rolling/sliding indentation method [15–18] and then conveyed using a force-sensitive tablet computer. The major advantage of the proposed technique is that both the haptic and visual cues are presented at the same active point of interaction. Other advantages include compactness, portability, wireless
operation, ease-of-use and low cost. The effectiveness of the proposed technique is evaluated by the measure of its ability to identify hard inclusions within soft surfaces in a simulation. In order to demonstrate the beneficial effects of immersive illusion [13,14] on pseudo-haptic sensation, we compare the performance of tablet computers in identification of hard inclusions with a touchpad input device. Note that in the case of using a touchpad input device, the contact force is exerted from the touchpad while the visual information is displayed on a computer screen. Here, hand-soft surface interaction was used for experimental benchmarking. The potential applications of our approach range from interactive gaming to medical training. In Section 2.1, two types of reaction forces between the rigid tool tip and soft surfaces are simulated. Sections 2.2 and 2.3 explain the concept and algorithm of applying pseudo-haptics to the soft surface stiffness simulation and the hard inclusions identification process by using a force-sensitive tablet computer and a force-sensitive touchpad. Section 3 describes the process of soft surface stiffness data acquisition, the validation test protocol; Section 4 presents the results of the validation studies. Conclusions are described in Section 5. 2. Methodology 2.1. Rigid tool/soft surface interaction The soft surface stiffness 3D haptic data which is used for the purposes of this paper comes from the rolling/sliding indentation method introduced by [15–18] to detect tissue abnormalities. The rolling indentation method works as follows: During the superficial rolling/sliding, the indenter slides over a soft object surface. As the indenter approaches a hard inclusion embedded in the soft object an increase of lateral force fx is experienced (see Fig. 1). During indenting behavior, a tool is pressed on the surface of the soft object to explore its stiffness pattern. The reaction force fy increases as the indentation depth increases. When the areas with hard inclusions underneath are pressed, the reaction forces are greater than those of other areas which are free from inclusions. In our previous research [8], the lateral reaction force fx of the rolling behavior and the normal reaction force fy of the indenting behavior were simulated and visualized separately. In this study, we have implemented simultaneous simulation of the two forces fx and fy to provide a more intuitive visual feedback to the user. 2.2. Pseudo-haptic soft surface stiffness simulation using tablet computers When the user moves the input pen or his/her finger on the tablet screen towards a relatively stiffer area from Po to P, the indenter avatar moves slower following the input (see Fig. 2), and an illusion of resistance to motion will be experienced. In this way, the virtual
v f
Indenter
Soft Object
fx
fy Indentation Depth P
Fig. 1. Reaction forces in rigid tool/soft surface interaction.
1094
M. Li et al. / Mechatronics 24 (2014) 1092–1100
i-3x i-2x
i-x i-3
i-1
i
i-2
i+1
i+2
i+3
i+x
Fig. 2. Indenter avatar speed-changing strategy and visualization of soft surface deformation using a tablet computer: the input pen moves from Po to P; P0 has the same stiffness as Po and P has a higher stiffness than Po; the indenter avatar first follows the input pen from Po to P0 and then moves from P0 to Pm; VF is the virtual force generated by using pseudo-haptic feedback algorithm.
forces (VF) can be perceived visually along with the direction of the movement. The flow chart of the pseudo-haptic soft surface stiffness simulation using a force-sensitive tablet computer is shown in Fig. 3. The user moves the input pen or a fingertip tangent to the surface whilst applying a normal force on the force-sensitive tablet computer. The normal force exerted on the tablet surface, together with the 2-DOF movement kinematics of the pen tip over the surface of the tablet computer are the two inputs of the system. The outputs are the normal reaction force from the tablet surface, the virtual resistance along the movement direction generated by using pseudo-haptic feedback, and the soft surface deformation shown on the screen. It should be mentioned that, here, the deformation of the soft surface is calculated based on the indentation depth, the model of soft object surface deformation curvature, and the avatar movement update. The tissue curvature is calculated based on the indentation depth and the model of soft object surface deformation curvature [19]. The deformation of the soft surface during interaction is displayed in real time using a geometrical deformable soft surface model, which is established based on a predefined finite element model considering the indentation depth and the indenter diameter. The details of this model are presented in [19]. As a node of the mesh is pressed by the indenter, the normal vertex of this node is redefined according to the depth of the indenter (see Figs. 4 and 5). Note that, as a result of the indentation of a specific node, the normal vertices of nodes located nearby on the mesh are also affected. If the indentation depth increases, the number of affected
Pre-Measured Stiffness Distribution
Model of Deformation Indentation Depth Curvature of Soft Object Surface Stiffness Value
Force Value Mapping of Haptic Surface
Position
Force Value
Pseudo-haptic Algorithm
Indenter Avatar Position
Soft Object Surface Curvature Soft Object Deformation Display
2-DOF Pen Movement
Force Level
Tangent Hand Motion and Normal Force Reaction force Input Fig. 3. The flow chart of the pseudo-haptic palpation simulation using a forcesensitive tablet computer.
i+2x i+3x Fig. 4. The location of the indenter avatar is on node i; the normal vertex of this node is redefined according to the depth of the indenter; the normal vertices of nodes located nearby on the mesh are also affected; as the indentation depth increases, the number of affected nodes increases.
nodes will also increase. Thus, the pattern of the deformation of soft surface is displayed graphically. It should be mentioned that if the surface is displayed parallel to the tablet surface, the deformation cannot be visually captured. Here, we display the surface using a perspective view with an angle of 45° towards the user to effectively visualize the deformation. Two prominent types of commercially available tablet computers are used in this study – Samsung Galaxy Note 10.1 (using an Spen) and Motorola Xoom (using the user’s bare finger). Samsung Galaxy Note 10.1 has a dimension of 262 180 8.9 mm and a weight of 600 g. Motorola Xoom is 249 167.8 12.9 mm and 730 g. Samsung Note (using a Wacom S-pen) has 1024 levels of pressure sensitivity. Motorola Xoom tablet senses the area of touch: when the touched area is broader, it recognizes the applied force as larger. The return value from the tablet computers is the force level, which needs to be mapped with force magnitude. A Force/Torque sensor ATI Mini 40 (a 6-DOF sensor with a force sensing range of ±10 N in x and y direction, ±30 N in z direction and a torque sensing range of ±0.5 nm; normal force resolution is 0.01 N) was placed under the tablet computer to record the applied force and map the force levels defined by the tablet computer to actual force values. Since the dimension of the force sensor (40 mm in diameter) are smaller than the tablets, two plastic plates were fixed to the two sides of the force sensor to increase the contact area between the force sensor and the touchpad as well as the table surface underneath. During the experiment, a user held the pen or used an index finger to move tangentially over the tablet’s force-sensitive surface, hence, exerting a normal force on it. In order to avoid torques transmitted to the force sensor when pressing at the edges of the tablet, the pressing was done exactly above the sensor during the mapping process. The normal force values fz were recorded via a data acquisition card (NI PCI-6013) and a LABVIEW program and stored in a text file. Force levels were captured via an Android program and stored in another text file. The force applied on the tablet surface was approximately a sawtooth waveform with different amplitude each pressing time. The peaks of the force values acquired from the Mini 40 force sensor and the force levels read from the tablet computer were used as the synchronization markers. The force level-force value relationships are described in Eqs. (1) and (2) (see Figs. 6 and 7), according to which the force levels (fl) are converted to the normal force values (fn). Then the indentation depth dm (z) is calculated as a function of the pressure applied to the tablet (fn) employing a lookup table of force matrices linearly interpolated between stored values:
M. Li et al. / Mechatronics 24 (2014) 1092–1100
1095
Fig. 5. The indenter avatar (sphere) and the deformation of the soft object surface; as the indentation depth increases, the affected tissue surface area becomes larger.
where fl is the force level reading using the getPressure() method in Android SDK and fn is the corresponding normal force. The modification of the indenter movement is realized by adding a time delay to the indenter avatar’s rendering task when the indenter itself is approaching a stiffer area. If the indenter passes over the stiffer area, the indenter avatar continues to follow the contact point. The delay time is determined by Eq. (3):
4.5 4.0 3.5
y = 0.1008e4.2081x R² = 0.9243
Force (N)
3.0 2.5 2.0 1.5
td ¼ Df m;
ð3Þ
Df ¼ f f l ;
ð4Þ
1.0 0.5 0.0
0
0.2
0.4
0.6
0.8
1
Force level Fig. 6. Force level and force value mapping of the Samsung Note tablet (using an Spen).
14 12
y = 0.0772e3.0727x R² = 0.9423
Force (N)
10 8 6 4 2 0 0.0
0.5
1.0
1.5
2.0
Force level Fig. 7. Force level and force value mapping of the Motorola Xoom tablet (using a bare finger of the user).
f n ¼ 0:1008e4:2081f l ;
ð1Þ
f n ¼ 0:0772e3:0727f l ;
ð2Þ
where the value of the reaction force f is acquired from the resultant reaction force matrices formed during the rolling indentation; fl is the reaction force value at the last avatar position; Df is the reaction force difference; and m is a scalar value. The calculated time delay is then added to the program frame interval time. To ensure the user can notice the indenter avatar when it lags behind the contact point, the minimum delay time should be set higher than the time interval between frame updates, the frame interval for this program is 30 ms. Due to the phenomenon of persistence of vision of the eye, an afterimage is thought to persist for approximately one twentyfifth of a second on the retina, the delay time needs to be set to be higher than 40 ms. Here we set to be 50 ms. It is designed that the user should notice the minimum reaction force difference of 0.1 N, thus, m is 50/0.1 = 500. We tested the movement speed of the S-pen on the tablet surface. The movement speed was ranging from about 27 mm/s to 36.5 mm/s. Here, actively adding delay is a means to reduce the movement speed of the indenter avatar compared to the hand movement when the stiffness is increasing. It adds a small distance between the hand position and the indenter avatar position. Apart from the reaction force difference Df, the total time delay is also determined by the nodule size. For example, to move over a hard nodule with a diameter of 10 mm, 185 ms to 137 ms (depending on the movement speed) is needed to reach the middle of the hard nodule where the reaction force difference becomes negative. After this stage, the indenter avatar again follows the hand movement and the delay time is less than 185 ms. It should be mentioned that the delay time was chosen to be sufficiently small (just enough to be noticeable to the users and enable
1096
M. Li et al. / Mechatronics 24 (2014) 1092–1100
them to identify hard nodules) to minimize its negative influence on the user’s workload. Fig. 8 shows the user interfaces of the pseudo-haptic soft surface stiffness simulation using the two tablet computers. Human fingertips are larger than the S-pen tip. Touching the screen with a bare finger can sometimes obstruct the vision of the user. In order to prevent the finger of the user from obstructing the view, the indenter avatar is set to be apart from the interaction point at a 15 mm distance.
Input Pen Po
Indenter Avatar
S-pen
(a)
VF
Touchpad
Indenter Avatar
(b) Fig. 8. Pseudo-haptic soft surface stiffness simulation using tablet computers: (a) Samsung Galaxy Note 10.1 (using an S-pen) and (b) Motorola Xoom (using a bare finger of the user).
Pm’ P’
dm do
Graphical interface on a computer monitor Fig. 9. Pseudo-haptic feedback using a touchpad device: the avatar display distance dm; VF is the virtual force generated by using pseudo-haptic feedback algorithm.
5 4.5
y = 0.0827e0.0039x R² = 0.9703
4
Force (N)
3.5 3 2.5 2 1.5 1 0.5 0 0
200
400
600
800
1000
1200
Force level Fig. 10. Force level and force value mapping of the force-sensitive touchpad.
f n ¼ 0:0827e0:0039f l ;
ð5Þ
where normal reaction force values (fn) is acquired from the reaction force matrices in sliding indentation [15–18]; fl is the force level reading from the touchpad data package. The coordinates of the touchpad surface are linearly mapped to the soft object surface. The force levels ( fl) read from the touchpad data package are converted to the normal force values ( fn) according to the Eq. (5) first. The position of the interaction on the surface of the soft object is calculated based on the mapping relationship between the touchpad surface and the soft object surface. Then the indentation depth dm (z) is calculated as a function of the pressure applied to the touchpad ( fn) employing a look-up table of force matrices linearly interpolated between stored values. The resultant reaction force can be acquired via the lookup table of force matrices according to the indentation depth dm (z). Then, the difference of the reaction forces (Df) between the current avatar position and the last avatar position is calculated as:
Df ¼ f f l ; 15 mm distance
Po’
D
P
2.3. Pseudo-haptic soft surface stiffness simulation using a forcesensitive touchpad motion input device In order to prove the beneficial effects of immersive illusion [13,14] on pseudo-haptic sensation using tablet computers, a touchpad is used as a control condition. As shown in Fig. 9, using a touchpad as a vertical force and horizontal motion input device, visual and haptic information are presented at different point of interaction – the contact force is exerted from the touchpad via the special pen while the visual information displays on a computer monitor. When the user moves the input device towards a relatively stiffer area over a distance (D), the indenter avatar display ratio will be updated to be lower than the original default ratio (Rm < Ro, Rm is the modified ratio and Ro is the original default ratio). Thus, the modified avatar displacement dm will be lower than the original default indenter avatar displacement do (dm=Rm D, do = Ro D). An illusion of resistance to motion will be experienced. In this way, the virtual forces (VF) can be perceived visually along with the direction of the movement. A force-sensitive Wacom BAMBOO Pen & Touch touchpad (a 2D haptic input device, 248 176 8.5 mm in dimension, 125 85 mm for touch sensitivity, 360 g weight) is used here. The touchpad has 1024 force levels. Similar test as described in Section 2.2 was conducted to map the force levels read from the touchpad to force magnitudes. 57 sets of data points were obtained. The least square method was used to produce the regression Eq. described in Eq. (5) (R2 = 0.9703, see Fig. 10).
Indenter Avatar
dm = Rm ∙ D, do = Ro ∙ D
ð6Þ
where fl is the reaction force value at the current avatar position; fl is the reaction force value at the last avatar position. When the soft surface stiffness in current position is stiffer than the tissue in the previous position (Df > 0), the movement distance is reduced:
dm ðx; yÞ ¼ Rmt Dðx; yÞ;
ð7Þ
Rmt ¼ Ro =ðf þ 1Þ;
ð8Þ
where Ro is the original default ratio between the avatar displacement and the input device displacement (avatar display ratio) calculated based on the coordinates mapping relationship between
1097
M. Li et al. / Mechatronics 24 (2014) 1092–1100
the touchpad surface and the soft object surface; and Rmt is the modified avatar tangent display ratio. When the soft surface in current position has the same stiffness or is softer than in the previous position (Df 6 0), the position of the indenter avatar is calculated based on the coordinates mapping relationship between the touchpad surface and the soft object surface. Thus, there is a jump between the previous position and the current position. This jump is used as a helpful sign for the user to locate hard nodules. 3. Evaluation experiments 3.1. Measuring soft surface stiffness distribution The stiffness distribution matrices used in the pseudo-haptic soft surface stiffness simulation corresponded to a 120 120 25 mm3 silicone block with three spherical nodules embedded inside (see Fig. 11). The phantom was manufactured using RTV6166 (TECHSIL Limited, UK) (A: B = 3: 7). The nodules were made from STAEDTLER Mars plastic 526 50 (47–50 Shore A) rubber. The diameters of the spherical nodules used were 10 mm, 8 mm, and 6 mm, respectively. All of the nodules were buried at a depth of 6 mm; this distance was measured from the top of the each nodule to the silicone surface. To obtain a rolling/sliding stiffness map [15–17], 59 straight-line trajectories (121 mm long and parallel to the x-axis with an interval of 2 mm between trajectories along the y-axis) were defined. A robot arm (Mitsubishi RV-6SL 6-DOF robotic manipulator with a positioning accuracy of 0.01 mm) was programmed to move attached the sliding indentation probe along these 59 straight-line trajectories at a speed of 30 mm/s with a constant rolling indentation depth. The silicone phantom tissue was covered by a transparent plastic sheet and was lubricated during the sliding indentation process to reduce the friction. The normal and tangent components of reaction forces were recorded at a sampling rate of 100 Hz using a data acquisition card (NI PCI-6013) and a LABVIEW program. The data acquired here then was processed in MATLAB by using linear interpolation to make the number of data points on each line to be 159. This experiment was repeated for a range of rolling indentation depths between 2 mm and 7 mm with intervals of 1 mm. Six 159 59 normal and tangent force distribution matrices were created, which allowed us to obtain a stiffness distribution map for the whole silicone block surface. Here, force distribution matrices were recorded using a Nano17-ATI force sensor (resolution: 0.003 N). Subsequently, this information was used in the algorithm of pseudo-haptic feedback. 3.2. Test protocol In order to validate our pseudo-haptic soft surface stiffness simulation approach, a series of experiments were conducted
Sliding Indentation Probe
Transparent Plastic Sheet
y o
B A
C
x Fig. 11. The silicone block and the locations of the three embedded hard inclusions (A, B, C).
employing the aforementioned three types of 2-DOF force-sensitive haptic surfaces. Pseudo-haptic feedback relies on visual display, so people with uncorrectable visual impairment cannot participant in the test. Here, two groups of participants, those who have normal vision and those with corrected vision, participated in an empirical study. The first group (Group I) consisted of twenty participants of which nineteen right-handed, one left-handed, one with palpation experience, and all with an engineering background while the second group (Group II) consisted of twenty participants, all right-handed, with an engineering background but no palpation experience (see Table 1). Group I conducted the tests of manual hard inclusions detection using direct hand touch (Manual I) and the pseudo-haptic soft surface stiffness simulation using a force-sensitive touchpad (Touchpad). Group II participated in the tests of manual hard inclusions detection using direct hand touch (Manual II) and the pseudo-haptic soft surface stiffness simulation using a tablet with an S-pen input (S-pen) and a tablet with a bare finger input (Bare finger). Prior to the start of each test, participants were also asked to do a practice run with known tumor locations. Subsequently, they were requested to manipulate the input device, palpate the virtual soft surface and observe the change of the ratio between the indenter avatar displacement and the input device displacement. These participants were also asked to report the positions of hard inclusions, where they detected them. Throughout the whole program of tests, we recorded the correctly and wrongly identified hard inclusions. It is worth mentioning that during all the aforementioned tests, the same stiffness map applied (one silicone/nodule block was used), but the orientation of the stiffness map was different from test to test: so the participants would not learn the nodules’ locations from the earlier tests. In order to eliminate the influence of surface orientation on the nodule detection performance, the surface orientation was pseudo-randomly changed during the experiment. For each participant, the tests were conducted in a pseudo random order. During the manual hard inclusions detection test, the transparent silicone block surface was covered by a black plastic sheet to prevent the locations of the hard inclusions from being visible to the user. 4. Results and discussion 4.1. Results Sensitivity Se [20], which relates to the test’s ability to identify positive results of the presence of hard nodules, is defined as sum over all the n trials of the True Positives (TP) divided by the actual number of hard inclusions or the sum of TP and False Negatives (FN), namely: n X Se ¼ TP i i¼1
, n X ðTP i þ FNi Þ:
ð9Þ
i¼1
Se was calculated for each feedback method as the proportion of true positives over the total of all trials conducted by all participants. The overall Se of each test is shown in Fig. 12. Table 1 Overview of demographics and experience of the two groups. Item
Group I
Group II
Age range Gender Handedness Palpation experience Engineering background VR simulator
23–42 $: 6; #: 14 R: 19; L: 1 1 20 0
20–30 $: 7; #: 13 R: 20; L: 0 0 20 0
M. Li et al. / Mechatronics 24 (2014) 1092–1100
Wilson score intervals [21] can be used to test the difference of two proportions [22] and have good properties even for a small number of trials (less than 30) and/or an extreme probability [23,24]. In this study, Wilson score intervals were calculated for sensitivity at a 95% confidence level using the following formula:
rffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi# ^ð1 p ^Þ z2 z2 p ^þ z þ 2 ; p 2n 4n n
ð10Þ
^ is the proportion of successes estimated from the statistical where p sample; z is the 1 a/2 percentile of a standard normal distribution; a is the error percentile and n is the sample size. Here, since the confidence level is 95%, the error a is 5%. The best Se was achieved with the tablet and S-pen (91.7%). The Se of each pairs of tests was compared. It was conducted by comparing the observed probabilities (p1 and p2) with a combined interval (CI), which was calculated by the following formula [25]:
CI ¼
qffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi ðP1 p1 Þ2 þ ðP2 p2 Þ2 ;
ð11Þ
where P1 is the upper bound of p1 and P2 is the lower bound of p2 if p1 < p2. If |p1–p2| > CI, there is a significant difference between the two tests. Table 2 shows the test results. In comparison with Group II, Group I had a higher Se (88.3% vs. 73.3%). However, the two methods had no significant difference in Se (CI = 0.148, Dp = 0.100). This proved that there was no significant difference in the touch perception abilities of these two groups. One can notice that both the Se of using tablet and S-pen, and tablet and bare finger were significantly higher than using a touchpad. Thus, direct touch and immersive illusion was proven to be superior to when visual and haptic information did not present at the same point of interaction. In comparison with using a tablet with a bare finger input, using a tablet with an S-pen input had higher Se (91.7% vs. 85%). However, the difference was not significant (CI = 0.157, Dp = 0.067). Fig. 13 compares the detection sensitivity of nodules A, B and C through the different pseudo-haptic feedback methods. Using the touchpad, the Se of nodule B was the highest (75%), followed by nodule A (65%); it was interesting to note that nodule B had a higher Se despite being smaller than nodule A. The smallest tumor C had a low Se of 15%. Using the tablet with an S-pen input, the Se of both nodule A and B were 100%. The smallest tumor, C, had a Se of 75%. Using the tablet with a bare finger input, both the Se of nodule A and B were 95%. The smallest tumor C had a Se of 65%.
100%
91.7%
Sensivity
80% 60%
85.0%
83.3% 73.3%
Item
Combined interval (CI)
Probability difference (Dp)
Significance?
Manual II & Manual I Manual I & Touchpad S-pen & Bare finger S-pen & Touchpad S-pen & Manual II Bare finger & Touchpad Bare finger & Manual II
0.148 0.167 0.157 0.156 0.136 0.165 0.146
0.100 0.316 0.067 0.400 0.184 0.333 0.117
No Yes No Yes Yes Yes No
120% 100% 80%
Touchpad
60%
S-pen
40%
Bare finger
20%
Manual I Manual II
0% A
B
C
Hard nodules Fig. 13. Nodule detection sensitivities of nodule A, B, and C with Wilson score intervals at a 95% confidence level.
Fig. 14 presents the Positive Predictive Values PPV [26], or precision rates, which are defined as sum over all the n trials of the TP divided by the test outcome positive or the sum of TP and False Positives (FP) – participants claim there is a hard nodule when there is no one, namely:
PPV ¼
n X
, TPi
i¼1
n X ðTP i þ FPi Þ:
ð12Þ
i¼1
PPV was calculated as the proportion of true positives over the sum of true positives and false negatives in all trails. The best PPV was achieved by using the tablet with an S-pen input (100% with 95% confidential interval: 93.5–100%) followed by using the tablet with a bare finger input (100%, 93.0–100%)). Using the touchpad had the lowest PPV (86.1%: 71.3–97.1%). The PPV using the touchpad (CI = 0.139, Dp = 0.167; CI = 0.142, Dp = 0.167; CI = 0.142, Dp = 0.167; CI = 0.145, Dp = 0.167) was significantly lower than the other four tests. This proves that the performance of hard nodule detection using tablet computers has no significant difference from the gold standard direct hand-soft object interaction (Manual I and Manual II). Also, the tablet computer enabled presenting visual cues and contact forces at the same interaction point and
51.7%
40% 20% 0%
Touchpad
S-pen
Bare finger
Manual I
Manual II
Feedback Method Fig. 12. Overall nodule detection sensitivities with Wilson score intervals at a 95% confidence level: touchpad represents the pseudo-haptic soft surface stiffness simulation using a touchpad; S-pen represents the experiment using a tablet with a motion and force input from an S-pen; bare finger represents the experiment using a tablet with a motion and force input from a bare finger of the user; Manual I represents hard inclusions detection using hand-soft surface interaction conducted by Group I; Manual II represents hard inclusions detection using hand-soft surface interaction conducted by Group II.
Posive predicve value
1 2 1 þ zn
"
Table 2 Comparison of sensitivities.
Sensivity
1098
110%
100.0%
100.0%
100.0%
100.0%
Bare finger
Manual I
Manual II
100% 90%
86.1%
80% 70% 60% 50% Touchpad
S-pen
Feedback method Fig. 14. Positive predictive values with Wilson score intervals at a 95% confidence level.
M. Li et al. / Mechatronics 24 (2014) 1092–1100
achieving significantly better performance than the case of exerting contact force from the touchpad while displaying visual information on a computer monitor. 4.2. Discussion Using a bare finger to provide the indentation and sliding input makes the contact more natural than using a pen. However, we noticed that the Se of using a tablet and a bare finger was lower than that of using a tablet and an S-pen. One reason for this is that the fingertips are larger than the S-pen tip. Touching the screen with a bare finger can sometimes obstruct the vision of the user during the experiments. Another reason can be the saturation problem of the pressure level measurement of this device, namely, using the contact surface area to estimate the contact force is not reliable when the contact area reaches its maximum while the contact force is still increasing. Hence, a better pressure-sensitive touch screen for bare finger interaction is required. Due to the fragility of the tablets, accurate robots were not used to press the tablet surfaces during the force/force level mapping experiments. Since humans might add jitters when pressing on the tablets, multiple tests were conducted with the same human hand to minimize the negative influence. Motorola Xoom tablet senses the area of touch: when the touched area is broader, it recognizes the applied force as larger. For Motorola Xoom tablet, ideally, calibration needs to be done for each user before using the system for greater accuracy. Here, to make our experiment more time-efficient for each user, the calibration was done only once prior to the start of the experiments. Although the size of contact area would vary among different users for the same amount of applied force, there would be a recognizable difference in size of the contact area between light and hard pressings. With this calibration result, the size of contact area can still be used to estimate the magnitude of normal force for the deformation of the virtual soft object. A perspective view of the surface was rendered using OpenGL. This surface and the indenter avatar were smaller at the rear end than in the front because of the increase of the distance to the observer. Since the smaller interaction area at the rear end compared with the front part, the perception may be affected. While the tissue surface was 120 120, the hard nodules were all located at the central area (60 60) of the virtual tissue. During the experiment, the participants were told to mainly explore the central area of the virtual tissue. Therefore, we assume the decrease of interaction accuracy for the area at the rear end did not have significant influence on our experimental results. 5. Conclusions The performance of the proposed pseudo-haptics rigid tool/soft surface interaction technique has been assessed using a force-sensitive touchpad with a pen input, a tablet computer with an S-pen input, and a tablet computer with a user’s bare finger input. Hard inclusions were detected more effectively on tablet computers. Sensitivity and positive predictive value were lower when the force-sensitive touchpad was used as visual and haptic information was presented at different points of interaction. Notably, the detection of hard inclusions done by applying direct touch or immersive illusion on tablet computers was comparable to detection performed with the use of hand/soft surface interaction. The proposed technique impresses with its performance in the detection of hard inclusions which rivals detection done via hand/ soft surface interaction. With the video gaming community always on the lookout for more realistic experiences, our proposed technique is sure to find many applications in gaming.
1099
Acknowledgements The work described in this paper was partially funded by the Vattikuti Foundation, the GSTT Charity, National Institute for Health Research (NIHR) Biomedical Research Centre based at Guy’s and St Thomas’ NHS Foundation Trust and King’s College London, the European Commission’s Seventh Framework Programme under grant agreement 287728 in the framework of EU Project STIFFFLOP, and the China Scholarship Council. The views expressed are those of the authors and not necessarily those of the NHS, the NIHR or the Department of Health.
References [1] Lécuyer A, Coquillart S, Kheddar A, Richard P, Coiffet P. Pseudo-haptic feedback: can isometric input devices simulate force feedback? In: Proceedings IEEE virtual reality. IEEE Comput. Soc; 2000. p. 83–90. [2] Kimura T, Nojima T. Pseudo-haptic feedback on softness induced by grasping motion. In: EuroHaptics; 2012. p. 202–5. [3] Dominjon L, Lecuyer A, Burkhardt J, Richard P, Richir S. Influence of control/ display ratio on the perception of mass of manipulated objects in virtual environments. In: Proceedings of the IEEE virtual reality; 2005. p. 19–25. [4] Argelaguet F, Jáuregui D. Elastic images: perceiving local elasticity of images through a novel pseudo-haptic deformation effect. ACM Trans Appl Percept 2013;10(3):17(1-14). [5] Khan S. Design and optimization of parallel haptic devices. 2012. p. 6. [6] Lécuyer A, Burkhardt JM, Tan CH. A study of the modification of the speed and size of the cursor for simulating pseudo-haptic bumps and holes. ACM Trans Appl Percept 2008;5(3):1–21. [7] Shiota G, Kashihara A. Cognitive effects of concept map generation with pseudo-haptic feedback. In: 2013 12th International conference on information technology based higher education and training (ITHET); 2013. p. 1–6. [8] Li M, Liu H, Seneviratne LD, Althoefer K. Tissue stiffness simulation and abnormality localization using pseudo-haptic feedback. In: IEEE international conference on robotics and automation. Saint Paul, Minnesota, USA; 2012. p. 5359–64. [9] Sinclair M, Pahud M, Benko H. 3D Touchscreen with Force Feedback and Haptic Texture. In: Proceedings of the 2013 ACM international conference on interactive tabletops and surfaces. St Andrews, United Kingdom; 2013. p. 287–96. [10] Sinclair M, Pahud M, Benko H. TouchMover: actuated 3D touchscreen with haptic feedback. In: Proceedings of the 2013 ACM international conference on interactive tabletops and surfaces; 2013. p. 287–96. [11] Li A, Sareh S, Ridzuan M, Seneviratne LD, Dasgupta P, Wurdemann HA, Althoefer K. Multi-fingered palpation using pseudo-haptic feedback. In: Hamyln symposium on medical robotics; 2014. Accepted. [12] Kokubun A, Ban Y, Narumi T, Tanikawa T, Hirose M. Visuo-haptic interaction with mobile rear touch interface. In: ACM SIGGRAPH 2013 Posters on – SIGGRAPH ’13. New York, NY, USA: ACM Press; 2013. p. 38. [13] Ridzuan M, Makino Y, Takemura K. Direct touch haptic display using immersive illusion with interactive virtual finger. In: EuroHaptics; 2012. p. 432–44. [14] Withana A, Makino Y, Kondo M, Sugimoto M, Kakehi G, Inami M. ImpAct: immersive haptic stylus to enable direct touch and manipulation for surface computing. Comput Entertain 2010;8(2):1. [15] Liu H, Noonan DP, Challacombe BJ, Dasgupta P, Seneviratne LD, Althoefer K. Rolling mechanical imaging for tissue abnormality localization during minimally invasive surgery. IEEE Trans Biomed Eng 2010;57(2):404–14. [16] Liu H, Li J, Poon Q, Seneviratne LD, Althoefer K. Miniaturized force-indentation depth sensor for tissue abnormality identification during laparoscopic surgery. In: International conference on robotics and automation (ICRA); 2010. p. 3654–9. [17] Noonan DP, Liu H, Zweiri YH, Althoefer KA, Seneviratne LD. A dual-function wheeled probe for tissue viscoelastic property identification during minimally invasive surgery. In: IEEE international conference on robotics and automation. Institute of Electrical and Electronics Engineers Inc.; 2007. p. 2629–34. [18] Liu H, Li J, Song X, Seneviratne LD, Althoefer K. Rolling indentation probe for tissue abnormality identification during minimally invasive surgery. IEEE Trans Robot 2011;27(3):450–60. [19] Li M, Seneviratne LD, Dasgupta P, Althoefer KA. Virtual palpation system. In: International conference on intelligent robots and systems workshop ‘‘Learning and Interaction in Haptic Robots’’; 2012. [20] Altman D, Bland J. Diagnostic test 1: sensitivity and specificity. BMJ 1994;308:1552. [21] Wilson EB. Probable inference, the law of succession, and statistical inference. J Am Stat Assoc 1927;22:209–12. [22] Beauliu-Prevost D. Confidence Intervals: from tests of statistical significance to confidence intervals, range hypotheses and substantial effects. Tutorials Quantitative Methods Psychology 2006;2(1):11–9.
1100
M. Li et al. / Mechatronics 24 (2014) 1092–1100
[23] Newcombe RG. Two-sided confidence intervals for the single proportion: comparison of seven methods. Stat Med 1998;17:857–72. [24] Newcombe RG, Altman DG. Proportions and their differences. In: Mltman DG, Machin D, Bryant TN, Gardner MJ, editors. Statistics with confidence: confidence intervals and statistical guidelines. London: BMJ Books; 2000. p. 46–8.
[25] Wallis S. Binomial confidence intervals and contingency tests: mathematical fundamentals and the evaluation of alternative methods. J Quant Linguist 2013;20(3):178–208. [26] Fawcett T. An introduction to ROC analysis. Pattern Recognit Lett 2006;27(8):861–74.