Flexible tactile sensor array for distributed tactile sensing and slip detection in robotic hand grasping

Sensors and Actuators A 297 (2019) 111512

Contents lists available at ScienceDirect

Sensors and Actuators A: Physical journal homepage: www.elsevier.com/locate/sna

Flexible tactile sensor array for distributed tactile sensing and slip detection in robotic hand grasping Yancheng Wang a,b,∗ , Xin Wu b , Deqing Mei a,b , Lingfeng Zhu b , Jianing Chen b a

State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou, 310027, China Key Laboratory of Advanced Manufacturing Technology of Zhejiang Province, School of Mechanical Engineering, Zhejiang University, Hangzhou, 310027, China b

a r t i c l e

i n f o

Article history: Received 1 November 2018 Received in revised form 16 July 2019 Accepted 16 July 2019 Available online 17 August 2019 Keywords: Flexible tactile sensor Three-axis force Slippage detection Discrete wavelet transform Distributed force Grasping Robotic hand

a b s t r a c t Distributed tactile information sensing is crucial for the stable grasping and manipulation of intelligent robotics. This paper presents a flexible tactile sensor array with spatial resolution of 3.5 mm that can be easily worn on the robotic hand for distributed three-axis contact force sensing in grasping applications. The proposed tactile sensor array has 3 × 3 sensing units, each unit has a five-electrode pattern’s design and using conductive rubber as the sensing material. The fabricated prototype of the tactile sensor array has good flexibility, and its performance is characterized with high sensitivities: 0.471 V/N in x-axis and ˜ N measure0.466 V/N in y-axis. As for z-axis, the sensitivities are 0.201 V/N at 06˜ N and 0.067 V/N at 615 ment ranges. Then the tactile sensor array and its scanning circuit are integrated into the robotic hand for distributed three-axis contact force perception when grasping different objects. By using discrete wavelet transform analysis, the threshold values of wavelet coefficients for slip detection can be determined, and the slippage during robotic grasping of objects can be successfully detected. Therefore, the developed flexible tactile sensor array has the ability of detecting distributed contact forces and slippage simultaneously, and could be used for robotic dexterous grasping and manipulations. © 2019 Elsevier B.V. All rights reserved.

1. Introduction Autonomous manipulation, such as stable grasping of objects, is one of the key functions of industrial and social robotics [1]. For dexterous manipulation, robotics must be able to adapt to the environment, especially when interacting with unknown objects [2]. Slippage is a common phenomenon in object grasping, and usually occurs when the applied grasping force is insufficient. However, if the contact force between the object and robotic hand fingers is too large, may leads to the damage and/or deformation of soft and brittle objects [3]. Therefore, a tactile sensor array with the abilities of both high flexibility and sensitivity in contact force perception is generally required because it can obtain abundant tactile information for robotic grasping force control and slippage prevention [4,5]. Therefore, development of a high sensitive flexible tactile sensor array with the abilities of detecting the distributed contact forces and slippage simultaneously is urgent in intelligent robotics. Recently, several types of tactile sensors have been proposed using piezoelectric [6,7], piezoresistive [8,9], capacitive [10,11], and

∗ Corresponding author at: State Key Laboratory of Fluid Power and Mechatronic Systems, School of Mechanical Engineering, Zhejiang University, Hangzhou, 310027, China. E-mail address: [email protected] (Y. Wang). https://doi.org/10.1016/j.sna.2019.07.036 0924-4247/© 2019 Elsevier B.V. All rights reserved.

optical [12,13] sensing principles. The finger-shaped BioTac sensor array has an elastomeric skin inflated by a conductive liquid over a bone-like core, the contact force deforms the skin and underlying fluid, and resulting the changes of electrical impedance of an array of electrodes arranged on the bone-like core [14]. The optical-typed tactile sensors, such as Gelsight [12] and GelSlim [13], can convert signals of the contact deformation into high-resolution tactile images and thus can achieve high spatial resolution and high sensitivity to the contact forces. Piezoresistive tactile sensors, using conductive rubber or strain gauges as the sensing element, usually have a simple structural design and feature high sensitivity and stability. Kim et al. [15] proposed a flexible piezoresistive electronic skin by forming a pyramidal shape with organic material doped with carbon nanotubes, the sensor’s resistance will be changed when applying external pressures on it. The pyramid-shaped structure of the piezoresistive material can be easily deformed and leads to high sensitivity. Yeo et al. [16] fabricated a thin and flexible microfluidic piezoresistive tactile sensor by filling the conductive liquid into a uniquely designed elastic protrusion structure. This sensor exhibited high sensitivity of 0.06 kPa−1 and can be used to discriminate the surface amplitude changes below 0.5 mm. Charalambides et al. [17] developed a large-area, fully-flexible tactile sensor array, composed of 12 units that can cover the entire palm and used to detect robotic hand grasping forces to avoid slippage.

2

Y. Wang et al. / Sensors and Actuators A 297 (2019) 111512

However, this sensor array is characterized with relatively lower sensitivity and spatial resolution, while better stability for largearea of tactile force sensing [18]. For these proposed sensor array, the protruding bump structure can concentrate the external force on the sensing area to enhance the sensitivity. However, the excessively thick protrusions will limit the flexibility of the sensor array and interfere with robotic hand dexterous movement for different objects grasping. Therefore, the structural design of the tactile sensor array with high sensitivity of contact force sensing and the ability for the integration of robotic hand still needs to be investigated. Sensing of distributed contact forces and slippage detection are two general requirements for dexterous robotic hand manipulation. Distributed contact force sensing can be achieved by the structural design of tactile sensor array, while slippage detection remains a key challenge. Based on the measured contact forces, several methods have also been proposed for slip detection and can be classified into three types: 1) Detecting the variation in friction coefficients. By measuring both normal and tangential forces, and then calculating the ratio of tangential to normal forces as the friction coefficient or breakaway friction ratio [19]. This value is constant in stable grasping, and will be abruptly changed when slipping occurs [20]. 2) Detecting the oscillation and/or vibration at moment of slipping [21,22]. The different shape and surface roughness of the grasped objects will introduce an oscillation or small vibration when slipping happens. Goger et al. [23] used the short-time Fourier transform (STFT) with a window function to analyze the measured tactile signals, and transmitted the transformed signal to a PC terminal to classify the slippage occurrence. 3) Detection of high frequency components in the measured tactile forces by using wavelet transform analysis, because the slip signal usually contains high-frequency component characteristic at early stage of slipping. Yang et al. [24] used the discrete wavelet transform (DWT) to analyze the tactile sensor signals to separate the high-frequency components. By observing the variation trends, the differences can be distinguished between the stable grasping and slipping to identify the slippage occurrence. Deng et al. [25] employed a force-sensitive resistor (FSR) sensor with the empirical mode decomposition (EMD) method to detect slipping in robotic hand grasping control. Previously, we also used the DWT to analyze the measured contact forces and to set a proper threshold value for slip detection [26]. In this study, we followed this research to develop a novel flexible tactile sensor array to measure the contact forces and detect slippage in robotic hand grasping of different objects. In order to overcome the limitations caused by the empirical setting of the threshold value [26], a new method to determine the threshold values for the flexible tactile sensor array for slippage detection will be studied. In this work, we propose a flexible tactile sensor array with 3 × 3 sensing units for distributed contact force sensing and slip detection in robotic hand grasping. In Section 2, the structural design and working principle of this flexible tactile sensor array is presented. The fabrication process and procedure to make the prototype of the flexible tactile sensor array is described. In Section 3, the experiment setup and procedures for three-axis force-sensing performance characterization and robotic hand grasping force measurements are conducted. This is followed by results and discussion in Section 4.

2. Flexible tactile sensor array

structure: bottom flexible electrode fabricated on a PET substrate, middle conductive rubber as the sensing material, and top PDMS bump layer, as in Fig. 1(a). The PDMS bump layer has a 3 × 3 array of hemisphere-shaped bumps with radius of 3.5 mm, which can transmit external force to the conductive rubber and lead to the resistance change. The conductive rubber layer consists of nine round-shaped conductive rubber chips. Each rubber chip has a diameter of 3.0 mm and thickness of 0.5 mm. The roomtemperature vulcanized (RTV) adhesive provides a close connection between the rubber chips and the upper PDMS bump and lower electrode layers. The bottom copper electrodes fabricated on a soft PET substrate has fairly good flexibility and strength to prevent the sensor array from the damage of mechanical tearing during the applications. For the designed tactile sensor array, the thicknesses of the bottom electrode layer, conductive rubber chip, and top PDMS bump layer are about 0.1, 0.5, and 0.8 mm, respectively. The spatial resolution for distributed force sensing is defined as the distance between adjacent sensing units and equals 3.5 mm. Therefore, the overall dimensions of the proposed tactile sensor array are 12.0, 12.0, and 1.4 mm in length, width, and thickness, respectively. 2.2. Three-axis force sensing principle For the proposed tactile sensor array, the conductive rubber chips are the essential sensitive elements and their resistance changes when external force is applied. To enable the sensing unit to measure both normal and tangential forces, a patterned electrode layer with five electrodes beneath each sensing unit is used as shown in Fig. 1(c). The central electrode serves as a common electrode and four peripheral electrodes each work independently. These five electrodes beneath the conductive rubber chip can generate four resistors of R1 , R2 , R3 , and R4 [26]. A xyz coordinate is defined on the top surface of the conductive rubber chip with the zaxis perpendicular to the rubber chip, as in Fig. 2(a). When external force (F) is applied on the bump surface, this force can be decomposed into three components: Fx , Fy , and Fz , as shown in Fig. 2(b) and (c). The normal force (Fz ) compresses the PDMS bump, and induced a uniform pressure applied on the top surface of the conductive rubber chip. Thus the resistances of four resistors will be decreased and have almost the same resistance changes. For tangential force (Fx ), the equivalent distance of Fx in x-axis is marked as cx . The applied Fx will generate a torque in the x-z plane, which can be regarded as an approximately linear-distributed normal force applied on the top surface of the conductive rubber chip. As for the effect of the applied uniform normal force and linear-distributed normal force, a trapezoidal distributed normal force (Fnx ) in z-axis can be used for simplification, as shown in Fig. 2(b). Therefore, the resistance of R2 will becomes smaller and the resistance of R4 increases, while the resistances of R1 and R3 will have almost the same changes. As for the applied Fy , a trapezoidal distributed normal force (Fny ) can also be used to study the effects of Fy in y-axis, as shown in Fig. 2(c). Assuming that there is a linear relationship between the output voltages and applied force, Fx , Fy , and Fz can be calculated as Fy = k1 V1 − k3 V3

(1)

Fx = k2 V2 − k4 V4

(2)

Fz = ˛1 V1 + ˛2 V2 + ˛3 V3 + ˛4 V4

(3)

2.1. Structural design Fig. 1 shows the schematic view of the flexible tactile sensor array. It has 3 × 3 sensing units, and each unit has a three-layered

where V1 , V2 , V3 , and V4 are the measured output voltages of four resistors; and k1 , k2 , k3 , k4 , ˛1 , ˛2 , ˛3 , ˛4 are the calibration coefficients, which can be determined by experimental calibration.

Y. Wang et al. / Sensors and Actuators A 297 (2019) 111512

3

Fig. 1. (a) Schematic view of flexible tactile sensor array, (b) cross-section view of the sensing unit, (c) five electrodes form four resistors.

Fig. 2. Three-axis force sensing principle.

2.3. Fabrication procedure The procedure to fabricate the proposed flexible tactile sensor array is illustrated in Fig. 3, it mainly consists of five steps: Step 1: Bottom electrode layer fabrication. In Fig. 3(a), the patterned copper electrodes are first printed on a flexible printed circuit board (FPCB), and are directly fabricated on the top surface of a thin PET film. To connect the conductive rubber chips and patterned electrodes, a thin layer of conductive adhesive (Shanghai AiBOND Trading Co., Ltd, China) is uniformly placed on the top surface of the electrodes by screen printing through a stainless-steel mask (Fig. 3(b)). The thickness of the mask is 100 ␮m, so the thickness of the printed conductive adhesive can be precisely controlled as 100 ␮m. Step 2: Conductive rubber chip alignment. The conductive rubber is first cut into small round-shaped chips with diameter of 3.0 mm. These rubber chips are then aligned to the corresponding electrodes and placed on the top surface, as indicated in Fig. 3(c). A slight pressure is applied on the top to ensure the stable connection

between the rubber chips and copper electrodes. Then the device is heated at 70 ◦ C for 3 h to cure the conductive adhesive. Step 3: RTV adhesive encapsulation. After full solidification of the conductive adhesive, an insulating RTV adhesive (Shenzhen Ausbond Co. Ltd, China) is coated around the rubber chips. They can prevent the misalignment of the rubber chips and electrodes to avoid the contact with adjacent chips, as in Fig. 3(d). Step 4: PDMS bump fabrication. A plastic mold with hemispherical structure (Fig. 3(e)) is firstly fabricated by the FDM printing process. The PDMS prepolymer and its curing agent (Sylgard 184 A and B) are mixed at a weight ratio of 5:1, and then degassed in a vacuum chamber for 10 min. Then, the mixture is poured into the plastic mold and further degassed in the chamber for another 15 min. After that, the mixture is heated up to 70 ◦ C for 6 h for completely curing, the cured PDMS bump layer is peeled from the mold. Step 5: Assembly of the tactile sensor array. The PDMS bump and conductive rubber chips are aligned with each other and assembled as in Fig. 3(f). For full solidification, the tactile sensor is placed on a heating platform and heated up to 80 ◦ C for 6 h.

4

Y. Wang et al. / Sensors and Actuators A 297 (2019) 111512

Fig. 3. Fabrication procedure of flexible tactile sensor array.

Fig. 4. Photograph of the fabricated tactile sensor array.

The final fabricated tactile sensor array is shown in Fig. 4. To facilitate attachment to a robotic hand, a peripheral electrode is utilized, and the sensor array is connected with the FPC connector through gold fingers. To demonstrate the flexibility and ease of wear, the tactile sensor array and its peripheral electrodes are

easily worn on a human thumb finger (Fig. 4), showing that the developed tactile sensor array can integrate with a non-planar surface and would be suitable for humanoid robotic hand gripping applications.

Y. Wang et al. / Sensors and Actuators A 297 (2019) 111512

5

Fig. 5. Schematic diagram of the scanning circuit.

3. Experimental setup and procedure

3.2. Calibration of the sensor array

3.1. Scanning circuit

Fig. 6 shows the experimental setup for characterizing the sensing performance of the tactile sensor array. The fabricated sensor array is mounted on a commercialized 3-axis force-load cell (Interface 3A120) with resolution of 0.01 N and measurement range of 0–50 N. The load cell can measure the force applied on the sensing unit in x-, y-, and z-axes, respectively. A round-shaped aluminum bar with a diameter of 3.0 mm is attached to the linear motion stage to apply normal and shear forces on the sensing unit, as shown in Fig. 6. During the experiments, the induced voltages and applied forces are measured by the scanning circuit and load cell, respectively. The relationships between the measured voltage, resistance changes, and applied forces are analyzed and used to calibrate the performance of the tactile sensor array for three-axis force sensing.

For our tactile sensor array, the distributed three-axis force can be measured and recorded based on the designed scanning circuit, as shown in Fig. 5. Because the tactile sensor array has 3 × 3 sensing units, and each sensing unit has four resistors. So overall, 12 rows are connected to a 12 single-pass analog switches (ADG1206) through the FPC. These analog switches are controlled by a microcontroller unit (MCU, TMS320F2812). When a specific four-digit coding is output to the analog switch, the corresponding row of the sensor’s resistor will be selected and powered by a voltage of 5.0 V. The output voltage of each column is connected to a single operational amplifier with a reference resistor Rref with the aim of reducing the crosstalk between the sensing units. To record the output voltages, a three-channel data acquisition (DAQ) card is used to select the column and acquire the signals from each unit. For example, R2 of the middle sensing unit is selected and its voltage is acquired by the DAQ, as indicated by the read lines shown in Fig. 5. These data will be transmitted to the PC via an RS232 interface, and the voltage signals are then recorded and used for further data analysis.

3.3. Robotic hand grasping force and slipping measurements After calibration, the developed tactile sensor array is worn on the ReFlex three-finger robot hand (RightHand Robotics, Inc.), as shown in Fig. 7. This robotic hand has three under-actuated fingers and four dynamic servo motors, and is position controlled by the ROS control system. Due to the structural design of the flexible tactile sensor array, it can be tightly wrapped on the robotic

6

Y. Wang et al. / Sensors and Actuators A 297 (2019) 111512

Fig. 6. Experimental setup to calibrate the performance of the tactile sensor array.

sensor array and objects are plane, cylinder, and sphere surfaces, respectively. These three objects are weighted about 286.5, 263.2, and 203.1 g, respectively. Both robotic grasping and slipping experiments were carried out, and experimental procedures are described as: Firstly, the robotic hand was controlled to seize the object. After 5 s, the robotic hand lifted the grasped object for 8 s, as in Fig. 7. Secondly, a small interference by dropping a small copper billet to the object was performed to make the object slipped a little bit (for about 2 s), the robotic hand still grasped the object. Thirdly, the robotic hand was slowly opened and leave the object on the desktop to avoid abrupt slippage. The time between the slipping and hand opening was about 10 s. For each slipping, the same tests were repeated three times for each grasped object.

4. Results and discussion 4.1. Three-axis force sensing calibration

Fig. 7. Tactile sensor array worn on the robotic hand finger for grasping of a beaker.

thumb finger, as shown in Fig. 7. During grasping experiments, three objects are selected: a hexagonal-shaped glass, a cylindrical glass beaker, and a plastic ball. The contact surfaces between the

For the developed tactile sensor array, the force measurement ranges for the x-, y-, and z-axes are 0.6, 0.6, and 15 N, respectively. To characterize three-axis sensing performance, the calibration coefficients k1 , k2 , k3 , k4 , ˛1 , ˛2 , ˛3 , ˛4 need to be determined by experiments. For z-axis force sensing, we used the setup shown in Fig. 6 to apply normal force on the top surface of the bump. For xand y-axis force measurements, we first applied an almost constant normal force (about 3.7 N) on the top, and then applied tangential force by moving the linear stage in x- and y-axis directions, respectively.

Y. Wang et al. / Sensors and Actuators A 297 (2019) 111512

7

Fig. 8. The measured voltages of four resistors and calculated voltages in three axes of the sensing unit of U31 : (a) normal force in the z-axis, tangential force in (b) x-axis and (c) y-axis.

The original output voltages of the sensing unit’s four resistors versus applied force in three-axes are plotted in Fig. 8. We can see that the measured voltages of four resistors increased as the applied force in the z-axis was increased, as shown in Fig. 8(a). As tangential force was applied in the x- or y-axes, the voltages of two resistors in the same direction changed. Typically, the measured voltage of one resistor increased and the others decreased. For example, when a constant normal force of 3.7 N was applied (Vz = 1.1 V), as the tangential force in the y-axis increased (the direction is from R3 to R1 ), the measured voltage of V1 increased and the value of V3 decreased, as shown in Fig. 8(b). Using the regression fitted method to analyze the measured voltages, the calibration coefficients k1 , k2 , k3 , k4 , ˛1 , ˛2 , ˛3 , ˛4 can be calculated based on Eqs. (1)–(3). Here, k1 = k3 = 1.98 N/V and k2 = k4 = 1.73 N/V. For z-axis normal force sensing, ˛1 = 4.84 N/V,

˛2 = 5.28 N/V, ˛3 = 4.31 N/V, ˛4 = 5.21 N/V are calculated when Fz is lower than 6.0 N. When Fz > 6 N, the values of ˛1 , ˛2 , ˛3 , and ˛4 are calculated as 14.81, 12.79, 17.91, and 17.36 N/V, respectively. In general, the sensitivity of the sensor can be expressed by the voltage changes caused by the force per unit Newton. As for our developed tactile sensor, the applied normal force will cause four voltages change, and the tangential force will cause two voltage change in the corresponding directions. Here, we used Vz , Vx , Vy to represent the total voltage change caused by three-axis forces, and this relationship can be expressed as Vz = V1 + V2 + V3 + V4

(4)

Vy = V1 − V3

(5)

Vx = V2 − V4

(6)

8

Y. Wang et al. / Sensors and Actuators A 297 (2019) 111512

Fig. 9. Three-axis force measurement results of grasping the cylindrical glass beaker.

Thus, the calculated voltages in three-axes versus measured forces are shown in Fig. 8, linear fittings were performed for the converted voltages. The results show that the calculated voltage in three-axis has a relatively good linear response with the applied forces. The slope of the fitted linear line can be regarded as the sensitivity of this sensing unit. Typically, the sensitivities for xand y-axes force measurements are calculated as 0.471 V/N and 0.466 V/N, respectively. For z-axis force sensing, there are two sensitivities: 0.201 V/N for the range of 0–6.0 N and 0.067 V/N for the range of 6.0–15.0 N. This indicated that the developed flexible tactile sensor array can be used for small normal force sensing with high sensitivity and large force measurement during the applications. As for the cross-talk effects during x- and y-axes force measurements, they can be eliminated as shown in Fig. 8(b) and (c). Further, the fitting accuracies in z-, x-, and y-axes are evaluated as 3.93%, 10.52%, and 11.81%, respectively. 4.2. Distributed contact force sensing in robotic hand grasping As described above, the developed tactile sensor array has been worn on a robotic hand finger for grasping force measurement. Fig. 9 shows the measured results of contact forces when grasping a cylindrical glass beaker. Six distinct stages can be observed in the force curves. Stage I is robotic hand slowly touching the glass breaker and the contact force is increased to grasp it, the measured Fx , Fy , and Fz are increased from zero to an almost constant value, respectively. Stage II is object grasping and lifting. The robotic hand was lifted so that the glass breaker left the desktop. The measured shear force in y-axis increases, and a slight fluctuation in Fz can be observed. Stage III is the first stable grasping, where the glass breaker was stably grasped for 10 s. In Stage IV, an external interfering force was applied to the beaker to make it slip a short distance while the beaker and the hand remain contact. This stage takes about 2 s and an abrupt drop and rise in the measured forces can be observed. Stage V is the second stable grasp, in which the robotic hand stably grasped the glass breaker for about 8 s. In Stage VI, the robotic hand was controlled to slowly release the beaker

on the desktop. The measured forces were reduced gradually and decreased to zero. The measured forces in these six stages match well with the described experimental procedure and robotic hand motion, which was presented in the experimental section. Also, due to the contacts between the robotic hand finger and the cylindrical convex surfaced beaker, only two columns of the PDMS bumps were contacted with the glass beaker during the grasping experiments. Therefore, the measured forces of U11 , U21 , and U31 are zero, as shown in Fig. 9. Generally, the developed flexible tactile sensor array could be used for contact force-sensing in robotic hand grasping applications. The distribution of the contact force between the object and tactile sensor array is shown in Fig. 10 for grasping of different objects, such as the hexagonal-shaped glass, cylindrical glass beaker, and plastic ball. From top to bottom are the results of grasping a hexagonal-shaped glass, cylindrical glass beaker, and plastic ball, respectively. The left column shows the measured normal forces in Fig. 10 which occur at t = 15 s during the first stable grasping stage. The right column shows the contact normal forces at t = 21 s and occur after the slipping and second stable grasping stages. As in Fig. 10, we can see that the contact forces distributed at the locations of each sensing unit are generally not uniform. For grasping of the hexagonal-shaped glass, the contact surface between the sensing unit and glass surface is a plane. During stable grasping stages, all nine sensing units are in full contact with the object. Thus, the variations in the measured normal forces for each sensing unit at t = 15 s and t = 21 s are generally small; the largest differences between the measured forces is about 0.18 N, as shown in Fig. 10(a). For grasping of the cylindrical glass beaker, the contact between the object and the tactile sensor array is a cylindrical curved surface, so just two columns of the sensing units are in contact with the beaker. The differences in the measured contact forces of each sensing unit become greater than that of grasping the hexagonalshaped glass, as shown in Fig. 10(b). For grasping of the plastic ball, only four sensing units were in contact with the object. When the object slipped, the contact position changed from the upper four

Y. Wang et al. / Sensors and Actuators A 297 (2019) 111512

9

Fig. 10. The measured contact force distribution at t = 15 s and t = 21 s when grasping (a) a hexagonal-shaped glass, (b) a cylindrical glass beaker, and (c) a plastic ball.

units (U11 , U12 , U21 , and U22 ) to the lower four sensing units (U21 , U22 , U31 , and U32 ), as shown in Fig. 10(c). 4.3. Slippage detection in robotic hand application 4.3.1. Threshold value setting of the wavelet coefficients For robotic hand grasping, slippage detection is a critical challenge, because slippage may leads to unstable grasping or external disturbance on the object. Also, the slip signal usually contains a characteristic of high-frequency components which is hard to be detected directly. By using Fourier transformation, spectrum analysis, or wavelet transform analysis, the high-frequency components of the measured signals can be distinguished. Previous studies have shown that the wavelet coefficients of the contact force at slipping moment become greater, and can be used for slippage detection [23,27,28]. However, they all judge the slip only by setting the threshold based on the measured normal force of one single unit, the threshold value is also given by experience and there is no clear setting criterion. For the tactile sensor array with ability of threeaxis force sensing, this method would be rather simple and not sufficient for the accurate detection of slipping. Therefore, a more stringent threshold setting for the sensor array needs to be studied. The first step for slip detection is to determine the threshold value of the wavelet coefficient for each sensing unit. A series of robotic hand grasping experiments were performed to grasp three

different objects (hexagonal-shaped glass, cylindrical glass beaker, and plastic ball) and measure the induced contact forces. The procedures of grasping experiments were the same as described in Section 3.3. In addition to the loading, stable grasping, and unloading stages, we also added a lifting stage (acceleration of the object) as a control process for the slip detection. We still followed our previous study and used discrete wavelet transform (DWT) to analyze the measured contact three-axis forces. Here, we chose the sensing unit U21 as an example; Fig. 11(a) shows the measured Fz , Fx , and Fy for grasping of the hexagonal-shaped glass. In the DWT, a third-order wavelet decomposition using the Haar wavelet was performed to construct the approximations and details from the coefficients, as shown in Fig. 11(b)-(d). We can see that a significant fluctuation occurred in the detailed output signals of the z-axis, and this fluctuation is much greater than that in x- and y-axes. Thus, the threshold values of the wavelet coefficients in z-, x-, and y-axes can be determined separately. In this study, we calculated the average value (M) and the value of 3␴ in the wavelet coefficients from 19 s to 22 s during the slipmoment, as shown in Fig. 11(b). The ␴ can be calculated as ping

   n 2 (di − M) , where di is the wavelet coefficient, n is the  =  1n i=1

number of wavelet coefficients. For the z-, x-, and y-axes, the cal-

10

Y. Wang et al. / Sensors and Actuators A 297 (2019) 111512

Fig. 11. (a) Measured Fz , Fx , and Fy in the sensing unit of U21 ; DWT analysis results of (b) Fz , (c) Fx , and (d) Fy for slip detection. Table 1 Threshold value of the wavelet coefficients for slip detection. Grasped objects

Hexagonal shaped glass cup

Cylindrical shaped beaker

Spherical shaped plastic ball

Axes

z x y z x y z x y

Sensing units

Values

U11

U12

U13

U21

U22

U23

U31

U32

U33

Average

0.047 0.008 0.011 — — — 0.077 0.008 0.009

0.064 0.012 0.018 0.075 0.021 0.018 0.061 0.011 0.007

0.056 0.010 0.013 0.060 0.015 0.029 — — —

0.055 0.009 0.013 — — — 0.081 0.013 0.013

0.043 0.014 0.017 0.059 0.021 0.022 0.062 0.011 0.010

0.051 0.008 0.012 0.066 0.019 0.024 — — —

0.052 0.012 0.016 — — — 0.069 0.013 0.012

0.062 0.016 0.013 0.064 0.015 0.019 0.065 0.015 0.014

0.065 0.014 0.011 0.053 0.025 0.023 — — —

0.055 ± 0.008 0.011 ± 0.003 0.014 ± 0.003 0.063 ± 0.007 0.019 ± 0.004 0.023 ± 0.004 0.069 ± 0.008 0.012 ± 0.002 0.011 ± 0.003

culated M and 3 are 0.064 and 0.003, 0.015 and 0.002, 0.016 and 0.001, respectively. We can see that both the fluctuations at the loading and releasing stages are much lower than that of M-3␴ for the z-, x-, and y-axes, respectively. Thus, the value of M-3␴ can be used to set the threshold value of the wavelet coefficient for slip detection. If the fluctuation of the wavelet coefficient exceeds this value, then we can estimate that slipping occurs. Based on this method, we calculated the threshold values of the wavelet coefficient of each sensing unit for grasping of different objects. Table 1 summarizes the calculated threshold value of the wavelet coefficients for slip detection. Generally, the threshold values in the z-axis are greater than in the x- and y-axes, because the normal contact force is also greater than that of the tangential forces (Fx and Fy ). As for the grasping of different objects, the threshold values in the same axis are generally at the same level. Therefore, we can determine the threshold values of the wavelet coefficients as 0.06 in z-axis, 0.01 in x-axis, and 0.01 in y-axis for the developed tactile sensor array for slip detection in robotic hand grasping applications. To determine threshold values, several researchers have used the maximum value of the wavelet coefficients at the stable grasp-

ing stage as the basis for slip detection [27,28]. This value is usually too large, and it has the possibility of misjudging slippage. Setting of the threshold value of the wavelet coefficients in this work is still far from optimal. The methodologies to determine the optimal wavelet coefficient threshold value will be conducted in future work. In this study, we used the calculated M-3␴ values of 0.06, 0.01, and 0.01 in z-, x-, and y-axes as the threshold to distinguish the occurrence of slippage in robotic hand grasping. 4.3.2. Slippage detection in robotic hand grasping Using the threshold values set in the above section, the slip occurrence can be identified. Figs. 12 and 13 show the results of slip identification in robotic hand grasping of a cylindrical glass beaker and plastic ball, respectively. The blue, yellow, and purple planes of 0.06, 0.01, and 0.01 represent the threshold values in the z-, y-, and x-axis, respectively. For grasping the cylindrical glass beaker in Fig. 12, it is obvious that the wavelet coefficients of the z-axis are larger than the threshold values at about 21–23 s. Thus, we can conclude that slippage occurred at this moment. The same results can be observed in the x- and y-axes, as shown in

Y. Wang et al. / Sensors and Actuators A 297 (2019) 111512

11

Fig. 12. Slip detection based on the DWT analysis of the measured (a) Fz, (b) Fx, and (c) Fy in grasping the cylindrical glass beaker.

Fig. 13. Slip detection based on DWT analysis of the measured (a) Fz, (b) Fx, and (c) Fy in plastic ball grasping.

Fig. 12(b) and (c). During the loading and releasing stages, some obvious large wavelet coefficient values can be found, although most of them did not exceed the threshold values and cannot be viewed as an occurrence of slipping. For grasping the cylindrical glass beaker, the U13 , U23 , and U23 sensing units are not in contact with the beaker. Both the measured contact forces (Fig. 10) and cal-

culated wavelet coefficient values (Fig. 12) are generally small. In Fig. 12(b), in addition to the significantly larger wavelet coefficient values at loading, slipping, and releasing stages, there are relatively large wavelet coefficients when the beaker is lifted. This is because the Fy suddenly increases when the beaker is lifted. This increase is much more significant than that in the loading stage. When the

12

Y. Wang et al. / Sensors and Actuators A 297 (2019) 111512

beaker is lifted, the PDMS bump will have a distinct deformation, leading the conductive rubber to experience an abrupt change in force and resistance. On the other hand, the lifting of the beaker itself may cause a vibration and interference of the robotic hand movement, and so may possibly cause more slipping. So the calculated wavelet coefficient values may be greater than in the stable grasping stages, as shown in Fig. 12(b). Fig. 13 shows the slippage detection results for grasping the spherical plastic ball. Because there are only four sensing units (U11 , U12 , U21 , and U22 ) that are in contact with the ball, several sensing units have a wavelet coefficient of zero compared to grasping of the cylindrical beaker. For the z-axis, the overall trends of the wavelet coefficients are similar to those of cylindrical beaker grasping. Values of the wavelet coefficients exceed the threshold can be seen as slippage occurs. As the ball slips, it is worth noticing that U32 and U33 are suddenly compressed and deformed, and produce a negative maximum wavelet coefficient. Some researchers have used a negative wavelet coefficient value instead of a positive value to classify the slippage occurrence [24]. In this work, for grasping the spherical ball, changes in the contact positions at the sensing units can also be used to demonstrate the occurrence of slippage, and the calculated wavelet coefficient values in this moment are generally greater and exceed the pre-defined threshold values. As depicted in Figs. 10 and 13, when four sensing units (U11 , U12 , U21 , and U22 ) first contact the spherical ball, the maximum wavelet coefficient values of these four sensing units occur at about 19.97 s. When the ball slips, the contact position at the tactile sensor array changes, and the sensing units of U21 , U22 , U31 , and U32 are again in contact with the ball. The new calculated maximum wavelet coefficient values appeared at 20.71 s, which means that the duration of the ball slipping lasted for about 0.74 s. The changed contact position at the tactile sensor array can also be used to classify the direction of slipping. Based on the estimated slip duration time of 0.74 s and the spacing between the sensing units of 3.5 mm, the slipping speed can also be estimated as about 4.67 mm/s. Thus, slipping during robotic hand grasping can be successfully detected by using the developed flexible tactile sensor array. 5. Conclusions In this study, we developed a flexible tactile sensor array with 3 × 3 sensing units for distributed three-axis force sensing and slippage detection in robotic hand grasping applications. The developed tactile sensor array has a novel multilayered structural design with five electrodes in one sensing unit for three-axis force sensing and a small portion of hemisphere bump to increase the sensitivity. The fabricated tactile sensor array has high spatial resolution of 3.5 mm and high flexibility, which can be easily wrapped onto the robotic hand without interfering with the hand movement. Experimental characterization showed that the developed tactile sensor array has high sensing performance. The sensitivities for xand y- axis force sensing are 0.471 and 0.466 V/N. As for z- axis, the sensor array has two sensitivities of 0.201 V/N at 06˜ N and ˜ N measurement ranges. Robotic hand grasping 0.067 V/N at 615 experiments showed that the flexible tactile sensor array can measure the distributed contact forces when grasping different objects. Furthermore, we presented a methodology by using the DWT and Harr wavelet to analyze the measured forces and set the wavelet coefficient threshold values in three axes for slip detection. The results demonstrated that the slippage occurrence can be successfully detected using the developed tactile sensor array. The results obtained in this study open the opportunity by using the flexible tactile sensor array for real-time contact force sensing and slippage detection in robotic hand grasping manipulations. Because the shape and surface texture of the grasped objects and movement of the robotic hand all have effects on the slippage

generation and affect the threshold value setting of the wavelet coefficients. In future work, the methodology for determining the optimal wavelet coefficient threshold values of the tactile sensor array for slippage detection will be developed, the mechanism behind the slipping phenomenon will be investigated. Further, the method for robotic hand grasping feedback control based on the measurement of contact forces and detection of slippage occurrence will be conducted. Acknowledgments This work was supported by the National Natural Science Foundation of China (51575485 and 51821093), Zhejiang Provincial Funds for Distinguished Young Scientists of China (LR19E050001), and Zhejiang Province Key Research and Development Plan Projects (Grant No. 2018C01053). References [1] A. Bicchi, Hands for dexterous manipulation and robust grasping: a difficult road toward simplicity, IEEE Trans. Robot. Autom. 16 (6) (2000) 652–662. [2] Y. Chebotar, K. Hausman, Z. Su, G.S. Sukhatme, S. Schaal, Self-supervised regrasping using spatio-temporal tactile features and reinforcement learning, Proc. IEEE International Conference on Intelligent Robots and Systems (IROS) (2016) 1960–1966. [3] M. Stachowsky, T. Hummel, M. Moussa, H.A. Abdullah, A slip detection and correction strategy for precision robot grasping, IEEE/ASME Trans. Mechatron. 21 (5) (2016) 2214–2226. [4] S.J. Kim, S.G. Baek, H. Moon, H.R. Choi, J.C. Koo, Development of a capacitive slip sensor using internal air gap, Microsyst. Technol. 24 (186) (2018) 1–6. [5] H. Yousef, M. Boukallel, K. Althoefer, Tactile sensing for dexterous in-hand manipulation in robotics—a review, Sens. Actuators A Phys. 167 (2) (2011) 171–187. [6] W. Wu, L. Wang, Y. Li, F. Zhang, L. Lin, S. Niu, D. Chenet, X. Zhang, Y. Hao, T.F. Heinz, J. Hone, Z.L. Wang, Piezoelectricity of single-atomic-layer MoS2 for energy conversion and piezotronics, Nature 514 (7523) (2014) 470–474. [7] C.H. Chuang, M.S. Wang, Y.C. Yu, C.L. Mu, K.F. Lu, C.T. Lin, Flexible tactile sensor for the grasping control of robot fingers, Proc. International Conference on Advanced Robotics and Intelligent Systems (2013) 141–146. [8] H. Liu, M. Dong, W. Huang, J. Gao, K. Dai, J. Guo, G. Zheng, C. Liu, C. Shen, Z. Guo, Lightweight conductive graphene/thermoplastic polyurethane foams with ultrahigh compressibility for piezoresistive sensing, J. Mater. Chem. C 5 (1) (2016) 73–83. [9] Y.C. Wang, K.L. Xi, D.Q. Mei, G.H. Liang, Z.C. Chen, A flexible tactile sensor array based on pressure conducitive rubber for contact force measurement and slip detection, J. Robot. Mechatron. 28 (3) (2017) 378–385. [10] Y.C. Wang, G.H. Liang, D.Q. Mei, Z.C. Chen, Flexible tactile sensor array mounted on the curved surface: analytical modeling and experimental validation, IEEE J. Microelectromech. Syst. 26 (5) (2017) 1002–1011. [11] C.M. Oddo, M. Controzzi, L. Beccai, C. Cipriani, M.C. Carrozza, Roughness encoding for discrimination of surfaces in artificial active-touch, IEEE Trans. Robot. 27 (3) (2011) 522–533. [12] S.Y. Dong, W.Z. Yuan, E.H. Adelson, Improved GelSight tactile sensor for measuring geometry and slip, 2017 IEEE International Conference on Intelligent Robots And Systems (IROS) (2017) 137–144. [13] E. Donlon, S.Y. Dong, M. Liu, J.H. Li, E. Adelson, A. Rodriguez, GelSlim: a high-resolution, compact, robost, and calibrated tactile-sensing finger, 2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) (2018) 1927–1934. [14] J.A. Fishel, G.E. Loeb, Bayesian exploration for intelligent identification of textures, Front. Neurorobot. 6 (2012) 1–20. [15] Y. Kim, A. Chortos, W. Xu, Y. Liu, J.Y. Oh, D. Son, J. Kang, A.M. Foudeh, C. Zhu, Y. Lee, S. Niu, J. Liu, R. Pfattner, Z. Bao, T.W. Lee, A bioinspired flexible organic artificial afferent nerve, Science 360 (6392) (2018) 998–1003. [16] J.C. Yeo, Z.J. Liu, Z.Q. Zhang, P. Zhang, Z.P. Wang, C.T. Lim, Wearable mechanotransduced tactile sensor for haptic perception, Adv. Mater. Technol. 2 (6) (2017), 1700006. [17] A. Charalambides, S. Bergbreiter, Rapid manufacturing of mechanoreceptive skins for slip detection in robotic grasping, Adv. Mater. Technol. 2 (1) (2017), 1600188. [18] S. Stefano, C. Valentina, C. Giancarlo, P. Candido Fabrizio, Flexible tactile sensing based on piezoresistive composites: a review, Sensors 14 (3) (2014) 5296–5332. [19] A. Cavallo, G. De Maria, C. Natale, S. Pirozzi, Slipping detection and avoidance based on Kalman filter, Mechatronics 24 (5) (2014) 489–499. [20] X. Song, H. Liu, K. Althoefer, T. Nanayakkara, L.D. Seneviratne, Efficient break-away friction ratio and slip prediction based on haptic surface exploration, IEEE Trans. Robot. 30 (1) (2017) 203–219. [21] R.D. Howe, M.R. Cutkosky, Sensing skin acceleration for slip and texture perception, 1989 Proceedings of IEEE International Conference on Robotics and Automation (ICRA) (1989) 145–150.

Y. Wang et al. / Sensors and Actuators A 297 (2019) 111512 [22] Z. Kappassov, J.A. Corrales, V. Perdereau, Tactile sensing in dexterous robot hands – review, Rob. Auton. Syst. 74 (2015) 195–220. [23] D. Goger, N. Gorges, H. Worn, Tactile sensing for an anthropomorphic robotic hand: hardware and signal processing, Proc. IEEE International Conference on Robotics and Automation (ICRA) (2009) 2972–2978. [24] H. Yang, X. Hu, L. Cao, F. Sun, A new slip-detection method based on pairwise high frequency components of capacitive sensor signals, Proc. International Conference on Information Science and Technology (2015) 56–61. [25] H. Deng, G. Zhong, X. Li, W. Nie, Slippage and deformation preventive control of bionic prosthetic hands, IEEE/ASME Trans. Mechatron. 22 (2) (2016) 888–897. [26] Y. Wang, K. Xi, D. Mei, Slip detection in prosthetic hand grasping by using the discrete wavelet transform analysis, Proc. IEEE International Conference on Advanced Intelligent Mechatronics (AIM) (2016) 1485–1490. [27] S. Teshigawara, S. Shimizu, K. Tadakuma, A. Ming, M. Shimojo, M. Ishikawa, High sensitivity slip sensor using pressure conductive rubber, IEEE Sens. J. 143 (2) (2009) 988–991. [28] H. Deng, Y. Zhang, X.G. Duan, Wavelet transformation-based fuzzy reflex control for prosthetic hands to prevent slip, IEEE Trans. Ind. Electron. 64 (5) (2017) 3718–3726.

Biographies Yancheng Wang, Associate Professor, he received the Ph.D. degree from the College of Mechanical Engineering, Zhejiang University, China, in 2010. He is currently an

13

Associate Professor of Mechanical Engineering at Zhejiang University. His current research interests include biomedical design and manufacturing, and biomechatronics. Xin Wu received the B.S. degree in mechanical engineering from Zhejiang University, Hangzhou, China, in 2015, where he is currently pursuing for the Master degree with research on the design of flexible tactile sensor array. Deqing Mei, Professor, he received the Ph.D. degree from the College of Mechanical Engineering, Zhejiang University, Hangzhou, China, in 2000, where he is currently a Professor with the State Key Lab of Fluid Power & Mechatronic Systems, College of Mechanical Engineering. His research interests include micro-manufacturing, and microelectromechanical systems. Lingfeng Zhu received the B.S. degree in mechanical engineering from Zhejiang University, Hangzhou, China, in 2015, where he is currently pursuing for the PhD degree with research on the micro-manufacturing, and microelectromechanical systems. Jianing Chen received the B.S. degree in mechanical engineering from Zhejiang University, Hangzhou, China, in 2016, where he is currently pursuing for the Master degree with research on the design of flexible tactile sensor array.