- Email: [email protected]
Construction of force haptic reappearance system based on Geomagic Touch haptic device
Journal Pre-proof
Construction of Force Haptic Reappearance System Based on Geomagic Touch Haptic Device Yushan Tang , Shan Liu Member IEEE , Yaru Deng , Yuhui Zhang , Lirong Yin , Wenfeng Zheng Member IEEE PII: DOI: Reference:
S0169-2607(19)31362-8 https://doi.org/10.1016/j.cmpb.2020.105344 COMM 105344
To appear in:
Computer Methods and Programs in Biomedicine
Received date: Revised date: Accepted date:
14 August 2019 16 January 2020 16 January 2020
Please cite this article as: Yushan Tang , Shan Liu Member IEEE , Yaru Deng , Yuhui Zhang , Lirong Yin , Wenfeng Zheng Member IEEE , Construction of Force Haptic Reappearance System Based on Geomagic Touch Haptic Device, Computer Methods and Programs in Biomedicine (2020), doi: https://doi.org/10.1016/j.cmpb.2020.105344
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2020 Published by Elsevier B.V.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
1
Highlights In this paper, the force-displacement, stress relaxation and creep data of the soft tissue model were collected and compared with those of the real soft tissue.After comparison, it is found that the data of this model are basically within the range of actual measured values, and the model has certain accuracy and validity, which is superior to the empirical Particle-Spring model.The frame rate in this paper is much higher than the frame rate required for real-time performance, which verifies the real-time performance of the system.
Yushan Tang is with the School of Automation, University of Electronic Science and Technology of China, Chengdu 610054, Sichuan, P. R. China (e-mail: [email protected] https://orcid.org/0000-0002-8357-6595). Yaru Deng is with the School of Automation, University of Electronic Science and Technology of China,Chengdu 610054, Sichuan, P. R. China(e-mail:[email protected], https://orcid.org/0000-0003-4004-0448). Yuhui Zhang is with the School of Automation, University of Electronic Science and Technology of China, Chengdu 610054, Sichuan, P. R. China (e-mail: [email protected]) Shan Liu was with the School of Automation, University of Electronic Science and Technology of China, Chengdu 610054, Sichuan, P.R.China, now she is with the Department of Modelling, Simulation, and Visualization Engineering, Old Domain University, Norfolk, VA 23529, USA(e-mail: [email protected] ORCID: 0000−0002−8040−0367). Lirong Yin is with the Department of Geography and Anthropology, Louisiana State University, Baton Rouge, LA 70803, United States (e-mail: [email protected], [email protected], ORCID：0000-0002-5022-610X). Wenfeng Zheng* is with the School of Automation, University of Electronic Science and Technology of China, Chengdu 610054, Sichuan, P. R. China (e-mail: [email protected] ORCID：0000-0002-8486-1654).
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
2
Construction of Force Haptic Reappearance System Based on Geomagic Touch Haptic Device (August. 2019) Yushan Tang, Shan Liu, Member, IEEE. Yaru Deng, Yuhui Zhang, Lirong Yin, Wenfeng Zheng*, Member, IEEE, Corresponding author: Wenfeng Zheng, E-mail:[email protected]
Abstract—Force haptic reappearance technology is considered to be one of the top ten technologies that can change human life in the future. It has broad application prospects and market demand. Most of the existing medical robots, especially the remote diagnosis and treatment robots, lack haptic feedback, or the calculation of feedback force is insufficient. Haptic reappearance technology is an effective method to solve the problem of haptic presence and improve the practicability of medical robot. The ultimate goal of the force haptic reappearance system is to let the operator feel the haptic feedback when interacting with the soft tissue model in the virtual environment in real time. Haptic device is the necessary condition to realize force haptic reappearance, and it is an essential part of the system. Its important role is to introduce the external force imposed by the operator into the virtual environment, and let the operator feel the force in the virtual environment, which effectively guarantees the operator's sense of reality and immersion when interacting with the virtual environment. Therefore, starting with the key technology of force and haptic reappearance system, this paper studies the construction of force and haptic reappearance system. Soft tissue surface model is drawn by OpenGL, and hand model is drawn by 3Ds Max. The haptic reappearance and visual feedback of soft tissue model of hand palpation are realized. The quality of feedback is evaluated. The haptic reappearance is stable and realistic, and the visual feedback is smooth. This indicates that the system has a certain application value and is worth to promote to the public. Index Terms—3Ds Max, Force and haptic, Geomagic Touch, OpenGL, Soft tissue model
I. INTRODUCTION
H
APTIC devices are electromechanical devices with input and output functions, which allow them to move with several degrees of freedom. Operators can manipulate the haptic devices to feel the objects in the virtual environment and produce realistic haptic sensation. The realization of force and haptic reappearance system is to feed back the reaction force produced in the interaction process to the operator through the haptic device, so the haptic device is the necessary condition to realize force and haptic reappearance.
So far, great progress has been made in the research of haptic devices. Universities, institutes and companies around the world have developed a variety of devices, some of which have even been commercialized, such as Novin’s Falcon device [1], Force Dimension’s Omega device and Delta device[2], SensAble’s Phantom device[3] and so on. The French National Institute of Information and Automation (INRIA)[4] maintains the world’s leading edge in this field. Cotin uses Laparoscopic Impulse Engine () force feedback system to create a liver anatomy model based on finite element method and realize haptic feedback, allowing doctors to manipulate, deform and cut the liver model [5] . The open source SOFA simulation platform developed by their team is widely used worldwide[6].
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
3
Electronic Visualization Laboratory (EVL) of the University of Illinois at Chicago, USA, developed a personal augmented reality immersion system PARIS based on stereo projection using Phantom Desktop haptic deviceError! Reference source not found. .Subsequently, Luciano added the integrated haptic device Immersive Touch on the basis of PARIS system to form a haptic augmented reality systemError! Reference source not found.. Its hardware integrated 3D stereo visualization, force feedback, head and hand tracking and 3D audio. In 2015, in collaboration with the University of Chicago, the Medical College of the University of Chicago built an immersive aneurysm shearing system with two haptic devices, which realized the haptic simulation of cerebral aneurysm clipping operationError! Reference source not found.. These applications have inspired more scholars to analyze and discuss the technique of force and haptic reappearance. The Department of Computer Science at Stanford University has also developed an oral and maxillofacial surgery system with two-hand haptic feedback for surgeons to learn and useError! Reference source not found..Wang used two Phantom Omni haptic devices to train neurosurgery for the first timeError! Reference source not found.. Based on NVIDIA’s physical engine PhysX, Brazil’s Macel designed an interaction surgical simulation framework with haptic feedback using Phantom haptic deviceError! Reference source not found.. Suzuki of Japan put forward the model of filling ball. With Phantom Omni haptic device, the cutting simulation of soft tissue inside stomach is better realizedError! Reference source not found. . Tercero of Nagoya University proposed a force feedback method for vascular surgery simulation based on photo elastic effectError! Reference source not found..The precise vascular model was established by magnetic resonance and CT images, and the haptic reconstruction of remote catheter interventional surgery was also realized by Phantom Omni[15]. These experiments have led to further applications of the sense of touch in medicine. The core technology used in this paper is the Geomagic Touch haptic Device [16], which is a mid-grade professional impedance haptic force feedback device. It is also the most affordable haptic device at present. It is made of metal parts and injection plastics. Its safety has been certified by FCC, CE and RoHS. Its main working parameters are shown in Table Ⅰ. The characteristics are as follows: 1.Small size, compact structure, easy to carry, with comfortable molded rubber, textured paint and removable stylus, two integrated instantaneous stylus switches and multi-functional indicator lights. 2.Solid grip, sensitive response, 3-DOF force feedback and 6-DOF position sensor can provide greater feedback force and workspace. 3.The interface is the firewall interface of IEEE 1394. It can be directly connected to the computer’s USB port through RJ45 Ethernet cable and USB Ethernet adapter. 4.Support a variety of haptic applications, allowing users to customize. The ultimate goal of force and haptic reappearance system is to make the operator feel real-time haptic feedback when interacting with soft tissue model in virtual environment. In this paper, the most difficult and challenging task is to simulate the human hand palpation behavior using haptic devices. We need to model a soft tissue model that is very similar to the real soft tissue, and let participants interact with it, and evaluate the quality of the system according to the results of participants' feedback. The highlights of our work is that we successfully use Geomagic Touch haptic device and 3Ds Max to construct a relatively real and stable human-computer interactive palpation system.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
working space weight Scope of activities Friction Maximum Output Force Location resolution Rigidity
Force Feedback (6 Degrees of Freedom) inertia Interface Support platform
4
TABLE I Main Working Parameters of Geomagic Touch 160 mm x 120 mm x 70 mm 3pound Hand Motion with Wrist Joint as Axis <0.26N 3.3N > 450dpi/~0.055mm X axis > 1.26N/mm Y axis > 2.31N/mm Z axis > 1.02N/mm X, y and Z axes 45g IEEE 1394 Fireline Interface Personal Computer with Intel/AMD Chip
The main work of this paper is as follows: 1.Based on Geomagic Touch haptic device, the force and haptic reappearance system is constructed. 2.Based on the construction of force-haptic reappearance system, this paper use 3Ds Max[17] to simulate hand palpation of soft tissue model. 3.The operator manipulates the human hand in the virtual environment to interact with the soft tissue model in the virtual environment by manipulating the haptic device. 4.The effects of interaction are discussed and analyzed, and further research directions are proposed in this paper. II.
HARDWARE AND SOFTWARE CONDITIONS FOR BUILDING THE SYSTEM
A. Hardware Conditions of the System
The hardware of force and haptic reappearance system of Tele-diagnosis and treatment robot mainly includes computer and haptic equipment. The computer is responsible for the construction, control and coordination of the virtual environment. The computer configuration in this paper is Windows 10 environment, Intel (R) Core (TM) i7-4790 CPU @3.60 GHz, 8GB memory and Acer LCD display. The haptic device in this paper is Geomagic Touch haptic device of American 3D Systems Company. B. Software Conditions of the System
Microsoft Visual Studio 2013 is used as the programming environment for the force and haptic reappearance system of Tele-diagnosis and treatment robot. The programming language is C/C++.The overall implementation framework of the system is based on OpenHaptics. It is an upgraded version of the previous GHOST (General Haptics Open Software Toolkit) SDK. Although its name is Open, it is not open source. This article uses OpenHaptics version 3.4, which is the latest version. OpenHaptics simplifies programming by encapsulating basic steps common to haptic/graphical applications, and has good compatibility. It supports Win32 and X64 development platforms on 64-bit systems. OpenHaptics Touch Device Driver (GTDD), Haptic Device API (HDAPI), Haptic Library API (HLAPI) and Quick API are the main components of OpenHaptics Touch Device Driver (GTDD), as shown in Fig.1.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
5
Fig. 1 The Component of OpenHaptics
GTDD is the driver of Geomagic Touch haptic device, which guarantees the communication between computer and haptic device. HDAPI provides low-level access to haptic devices and control over the runtime of configuration drivers. HLAPI is a high-level API for haptic rendering after OpenGL graphics rendering. It allows the reuse of existing OpenGL code and can easily and quickly add touch to graphics applications, greatly simplifying the synchronization of touch and graphics threads. HLAPI is based on HDAPI[18]. HLAPI still needs to use HDAPI functions in the initialization, security processing, status and location query of haptic devices. HLAPI also provides haptic effects such as gravity and inertia. Quick API is a miniature API. It encapsulates many functions on the basis of HDAPI and HLAPI through STL template library of C++.Quick API provides two versions, one is Win32 and the other is GLUT version of OpenGL.Win32 version supports multi-window applications, while GLUT version does not support them. The GLUT version is independent of the operating system, while the Windows 32 version is only applicable to Microsoft’s Windows operating system. After the Open GL and HLAPI initializations, the HDAPI is called to initialize the force feedback device. HLAPI processes events at regular intervals to obtain Proxy values and position values of force feedback devices, which are then calculated by HDAPI to give the resultant force. The HDAPI for communicating with force feedback devices in program development consists of two parts: Device and Scheduler. The Device section manages the haptic devices, also known as PHANTOM Desktop, including state management, parameter setting, power transfer, and the ability to manage multiple haptic devices in tandem simultaneously. The state includes sensor position, button pressed or not, sensor speed, etc. The Scheduler is responsible for managing the high-frequency threads to transmit power and to accept or send out the device state. Since the Scheduler section is provided in the API, the consumer can leave the high frequencies required for haptic processing to the API, as long as the Scheduler of the HDAPI communicates with and executes the instructions. From Fig.1, it can also be known that the refresh rates of haptic feedback and visual feedback are different. The refresh rates of haptic device force are one order of magnitude higher than those required for image display [21]. This difference stems from the psychophysics of human perception. Usually, the refresh rate of visual feedback is not less than 30Hz, and the haptic feedback can become undistorted when the frequency is above 300Hz and can obtain satisfactory results when the frequency is above 1000Hz. Because of the above reasons, the frequency range of visual feedback is above 30Hz but the range of haptic feedback is above 1000Hz. Therefore, haptic rendering and
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
6
graphical visual feedback are usually executed simultaneously in different threads of the same process to meet their respective refresh rates. The callback mechanism of HLAPI implements that haptic rendering model is specified in the same thread at the same speed as graphics, instead of synchronizing the state of haptic and graphical threads in haptic threads. At the same time, events occurring during haptic rendering are transmitted to client programs through callback functions. It can also be seen from Fig.1 that HLAPI has two buffers, depth buffer and feedback buffer[19], for processing haptic rendering calculations based on geometric primitives, transformations and material attributes. When the feedback buffer stores points, line segments and polygons for haptic rendering, the depth buffer uses images read from the depth buffer for haptic rendering. For a large number of primitives, the deep buffer occupies less memory and is more efficient, while the feedback buffer is more efficient for a small number of primitives. Generally speaking, the depth buffer is not as accurate as the feedback buffer. Although the difference in accuracy is almost undetectable, the feedback buffer must be used to render the constrained points and lines, because the depth buffer cannot capture the points and lines. In specific applications, the type of buffer should be selected according to actual needs. In this paper, the feedback buffer is used. In order to better realize force and haptic reappearance and realistic visual feedback, and reuse the existing OpenGL code, this paper uses HLAPI framework to construct haptic system, and the development process as shown in Fig.2 is obtained. The haptic feedback model used in this paper is the proxy model proposed by Ruspini et al. The principle of the model generally uses a sphere without mass to represent the agent of the end of the haptic device in the virtual environment. When the operator moves the haptic device, the agent moves with it. In the proxy model, two location information of haptic device are saved: the device end location and the surface control point location (proxy). Among them, the surface control point is a point that closely follows the end position of the haptic device but is limited to the surface of all touchable objects. The operator manipulates the haptic device to interact with the soft tissue model in the virtual environment. When the end of the haptic device does not contact the soft tissue model, the position of the surface control point coincides with the position of the end of the device. When the end of the device actually penetrates the soft tissue surface, the position of the surface control point is the projection of the actual position of the end of the device on the soft tissue surface. At this time, a virtual spring damper is simulated between the surface control point and the actual position of the end of the device to calculate the force transmitted to the haptic device. The operator can feel the force from the haptic device, which is force feedback, which can simulate the haptic effect when contacting with the soft tissue model in the virtual environment. The feedback force can be calculated by the depth of "penetration", and the calculation method is shown in (1). •
𝐹 = 𝐾𝑥𝑑 + 𝐷𝑥𝑑
(1)
By setting the values of K and D, the haptic characteristics of the model surface can be controlled. Where K is the stiffness coefficient and can be set to any value between 0 and 1, 0 is the softest surface and 1 is the hardest surface. The haptic device has stable rendering capability, and setting K to an unreasonably high value will lead to instability or rigidity. D is the damping coefficient, which exists to reduce vibration, and the setting range is also between 0 and 1. Soft tissue is relatively soft. The parameter of force feedback in this paper is set to K=0.15, D=0.9. As shown in Fig.2, initialization is the first step after the program starts. Initialization includes the initialization of OpenGL, the initialization of haptic devices and the calibration of haptic devices [20]. Firstly, the haptic devices must be calibrated to obtain accurate three-dimensional position data. Secondly, all applications in a virtual environment must determine the appropriate mapping of the haptic workspace to the graphical scene so that all visible objects
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
7
can be touched. Haptic workspace is the reachable physical space of haptic devices. Basically, the whole working area is chosen, that is, the working area shown in Table Ⅰ. All haptic rendering in HLAPI is done in haptic frames, which begin with HLBeginframe () and end with HLEndframe (). Typically, each graphics rendering frame has a corresponding haptic rendering frame. HLAPI’s deformation rendering is used to render the established soft tissue model (proxy rendering). The content of deformation rendering is surrounded by HLBeginshape () and HLEndshape (). HLAPI captures the geometric objects specified by OpenGL and assigns unique identifiers, and performs haptic rendering on them. In order to facilitate the interaction with soft tissue models in virtual environment, visual haptic devices are needed, that is, display the location of haptic devices in space, and observe the changes when they interact with other objects in the scene. Three-dimensional cursor rendering is to achieve this function. The three-dimensional cursor defines the haptic interface points, and draws the location of the agent of the haptic device in the virtual scene (which can be obtained by the driver GTDD), as well as its size (which can be customized according to the display needs). Because there can be multiple haptic devices, there can be multiple three-dimensional cursors.
Fig. 2 HLAPI Development Process
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
8
The model drawing of force and haptic reappearance system of Tele-diagnosis and treatment robot mainly refers to the establishment of soft tissue model and palpation hand model. For force and haptic reappearance system, simple models can be directly drawn by OpenGL, while complex models are usually built by professional threedimensional modeling software such as AutoDesk 3Ds Max, SolidWorks, FreeForm, etc., and then imported into the development environment of Visual Studio through the program. III. METHOD A. Soft Organization Model Drawing Based on OpenGL ：
In this paper, a cross-platform graphics library OpenGL is used to build and display soft tissue surface model. OpenGL is known as "a software interface of graphics hardware". It is a procedural rather than descriptive graphics API. It has no window management and can be used to create graphics with excellent visual quality. OpenGL has the following characteristics: 1.OpenGL is an open source, cross-system, cross-language professional graphics program interface, which defines a special data type. 2.It has high portability and can provide powerful and convenient graphics library for the system. 3.It has a very fast speed, and its rendering speed is much faster than that of ray tracker or software rendering engine. 4.It is widely used, covering engineering, games and other modeling applications. Soft tissue modeling with OpenGL basically follows the workflow shown in Fig.3. Rasterization is the process of converting vertices into sheets. In fact, it fills the pixels between each coordinate point to form line segments. Grating can enhance the 3D sensation of the model, but the corresponding illumination processing and projection transformation must be done before rasterizing.
Fig. 3 The Workflow of OpenGL
In order to make the edge of the deformation region smooth during the deformation of soft tissue model, and more in line with the effect of the actual deformation of soft tissue, the mass spring soft tissue model is established based on the triangle topological structure. According to the topological structure of the selected mass-spring soft tissue model, the parameters of the spring itself are determined. The mass of a single mass is equal to 0.01. If the original length of the structural spring connecting the adjacent particles in the horizontal and vertical directions is set as 1, the original length of the shear spring in the diagonal direction is equal to 2, and the original length of the bending spring connecting the interval particles in the horizontal and vertical directions is equal to 2.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
9
According to the topological structure of the soft tissue model above, the soft tissue model established by OpenGL is shown in Fig.4.The soft tissue model shown in Fig.5 consists of 625 (25 x 25) particles and 3502 Voigt spring elements. The yellow points in the figure represent the particles, and the rose-red lines represent the spring elements connected between the particles. The soft tissue model that does not display the particle spring after rendering is shown in Fig.5.
Fig. 4 Soft Tissue Model
Fig. 5 Soft Tissue Model Rendered
B. Hand Model Rendering Based on 3Ds Max：
OpenGL is relatively difficult to use in complex three-dimensional modeling, so the human hand threedimensional model in force and haptic reappearance system is established and managed by Autodesk’s professional three-dimensional modeling software, 3Ds Max 2014.3Ds Max is a high-performance three-dimensional modeling and rendering software, which is widely used in design, video animation and games and other industries. It has low requirement for PC configuration, powerful modeling function, good expansibility, simple operation and easy to use. In this paper, the human hand model established by the classical polygon modeling method of 3D software in 3Ds Max 2014 is shown in Fig.7.The hand model shown in Fig.6 contains 7822 vertices and 15640 faces. Exporting the established hand model to obj format and importing it into Visual Studio 2013 development environment through program can greatly improve the efficiency of modeling.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
10
Fig. 6 Human Hand Model Established by 3Ds Max 2014
The human model in obj format is imported into Visual Studio 2013 development environment, using GLM header file and implementation file, which defines the corresponding vertex, color, material and structure of the model description of the obj file. Add the above file in the program, read the obj file with glmReadObj () function, and draw and display the model with glmDraw () function. The 3D model built by 3Ds Max does not necessarily match the size of the soft tissue model built by OpenGL in visual display, so the glmScale () function is also used to zoom the model in a certain proportion. The model built by 3Ds Max in this paper shows too small after importing in the original proportion, so the hand model is enlarged 15 times. The reason why we need to build a complex human hand model is that in the process of interaction, the touching and pressing activities of participants on soft tissues need to be simulated as realistically as possible through this human hand model. Therefore, the human hand model we built must be realistic, so as to make the experimental results accurate enough, and make the test response as sensitive as possible. According to the principle of proxy model, haptic devices are generally modeled as a sphere in virtual environment. To simulate hand palpation, it is necessary to model the proxy of haptic devices into an imported hand model in the three-dimensional cursor rendering stage. C.
Realization of Force and Haptic Reappearance System Based on Geomagic Touch：
Based on the above software and hardware conditions, soft tissue model and hand palpation model, a force and haptic reappearance system based on Geomagic Touch haptic device is constructed, as shown in Fig.7. The specific workflow of force and haptic reappearance system is as follows: 1.Initialization, calibration of haptic devices, setting force-haptic characteristics to create virtual environments, and determining error handling mechanisms such as motion overrun. 2.OpenGL initialization, the introduction of hand model, drawing soft tissue surface model. 3.Operators operate haptic devices to manipulate human hands in virtual environments and interact with soft tissue models in virtual environments. 4.Real-time acquisition of the position of the end of the haptic device, and collision detection based on the location information. If the collision occurs, the collision point is determined, and the external force is applied to the particle and the nearest circle of particles around it.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
11
5.According to the applied external force, the force of the soft tissue model is calculated, the particle velocity and displacement are calculated iteratively, the model is updated, and the deformation rendering is carried out. 6.Haptic rendering, calculating the feedback force, and feedback to the operator through the device, circular reciprocating, realizing the force and haptic reappearance and visual feedback in the process of human-computer interaction. 7.After the interaction, do some cleaning work to ensure the correct release of haptic device resources and memory resources.
Fig. 7 Force and Haptic Reappearance System
IV. RESULTS OF FINDING
A. Hand Palpation Experiment： Palpation is a process in which surgeons touch and press tissues or body organs with their fingers to determine if they have abnormalitiesError! Reference source not found..Run the program in Visual Studio 2013 development environment, hold the end of Geomagic Touch haptic device, control the hand movement in virtual environment through the end of mobile device, press or stretch the hand on the screen along the soft tissue surface, and simulate the touch operation. The shape change of the soft tissue model can be seen on the display screen, and the force and haptic information generated in the process of pressing and stretching the model is fed back to the operator through the haptic device. With the increase of press and stretch depth, the feedback force also changes. When the hand leaves the soft tissue model, the deformation of the model gradually decreases and finally restores to its original state. Soft tissue does not deform when the hand is not in contact with the soft tissue model, as shown in Fig.8.Soft tissue deforms and sinks downward when the surface of the soft tissue model is pressed by hand, as shown in Fig.9. The hand stretches the soft tissue model, and the soft tissue deforms and protrudes upward, as shown in Fig.10.When the hand leaves the soft tissue model, the deformation of the model slowly recovers, as shown in Fig.11.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
12
Fig. 8 Hand Contactless Model
Fig. 9 Hand Pressure Deformation
Fig. 10 Hand Tensile Deformation
OpenHaptics can also simulate gravity effects. Gravity effect is added to the soft tissue surface model. Fig. 12 gives two kinds of soft tissue models after applying gravity. The observation state of the soft tissue with added gravity from the perspective of top to bottom is expressed as (a) and the state of the soft tissue as seen from the bottom-up perspective is expressed as (b). In order to show the obvious topological structure of particle-spring. In previous studies, in order to stabilize the whole force and haptic reappearance system, the soft tissue model is constrained by deformation, and the particles around the model are fixed. When gravity is added, it will show the effect as shown in Fig.12. If the fixed particles around are cancelled, the model will gradually fall under the action of gravity and eventually disappear in the graphical display interface. If only deformation constraints are applied to a particle in one side direction, the soft tissue will be suspended in the graphical display interface like a curtain when gravity effect is added.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
13
Fig. 11 Deformation Recovery
(a)
(b) Fig. 12 Applies Gravity Effects. (a) From a Up-bottom Perspective; (b) From a Bottom-up Perspective
Compression and stretching were performed on the soft tissue model with added gravity effect, and the deformation effect was shown in Fig. 13 and Fig. 14. As it can be seen from the two figures, the compression and stretching effects of soft tissue under gravity still have a certain degree of fidelity. It is worth noting that the effect of force feedback of soft tissue model under gravity is different. If the applied gravity is opposite to the direction of operation, the operator will feel more reaction force. B. Quality Evaluation of Haptic Reappearance： At present, there are no quantitative indicators to evaluate the quality of haptic feedback. Since the main user of force and touch feedback system is human, the main criterion to evaluate the system performance should be human experience of using the system, so in this paper, subjective evaluation in the form of questionnaires is the main method. Referring to the artificial evaluation index used in haptic reappearance of surgical simulator in reference [22], this paper invites 10 participants to operate haptic devices and interact with soft tissue models in virtual environment to evaluate the quality of haptic reappearance of the system subjectively from three aspects: authenticity, stability and ease of use. In this experiment, there is no specific quantitative standard for the user's experience of the haptic representation system using the force, so in this paper, we take the user's sense of use as the evaluation index. When 8 out of 10 participants have positive comments on the system, it is deemed that the system could reproduce and feedback force and touch more realistically. Authenticity refers to whether the force in the virtual environment can be felt through the haptic device during the deformation process after the deformation of soft tissue. The greater the operation force, the greater the deformation. With the increase of the deformation depth, the greater the feedback force, and whether the deformation can be restored after the removal of the external force. Usability refers to the ease of interaction between haptic devices and soft tissue models. Stability refers to whether the feedback force is stable and there is no severe jitter during the whole palpation interaction process.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
14
10 participants in the test ranged in age from 23 to 27, with a male-to-female ratio of 1:1 and right-handed habits. Before all participants are tested, demonstration is done first, and then operation steps are explained. Attentions are given. Participants are asked to focus on the force feedback given by haptic devices when touching soft tissues. Finally, the authenticity, ease of use and stability of the force and haptic reappearance system are evaluated according to their subjective feelings.
Fig. 13 Tensile Deformation under Gravity
Fig. 14 Pressure Deformation under Gravity
As for the authenticity of the feedback, all participants felt haptic feedback in the process of manipulating the haptic device and interacting with the soft tissue model, and with the increase of deformation depth, the feedback force increased obviously. After the withdrawal of external force, the soft tissue model can indeed recover the deformation, and the haptic feedback has better authenticity. However, the haptic feedback of most participants in the stretching process is more obvious than that in the pressing process, presumably due to the need to overcome the inertia and gravity of the haptic device during the stretching process. In terms of usability, participants generally believed that it was easy to interact between haptic devices and soft tissue models. Individual participants felt that they were not used to it at first, so they would be better if they adapted to it. As far as the stability of the system is concerned, most participants do not feel obvious jitter during the whole interaction process. A few participants think that they will feel slight jitter when stretching to the deformation limit, and the system has better stability. C. Contrast Experiments on External Force Application：
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
15
Generally, the external force is applied only to the collision point. This paper optimizes the method by applying the external force to the collision point and a circle of particles around it. The effect of only applying external force to the impact point is shown in Fig.15, while the effect of optimized deformation is shown in Fig.16.In order to facilitate comparison, the particle and spring of the model are displayed, and the agent of haptic devices in virtual environment is modeled as a small sphere.
Fig. 15 Deformation Effect of External Force on Collision Point
Fig. 16 Optimized Deformation Effect
Compared with Fig.15 and Fig.16, it can be seen that if only external force is applied to the collision point, the deformation of the collision point will be too large during deformation rendering, while the deformation of other particles will be relatively small, showing a peak shape and not smooth. After the external force is applied to the collision point and a circle of particles around the collision point for optimization, the shape change of the interaction part is smooth and relatively more real. It should be noted that although Fig.15 and Fig.16 only compare the deformation rendering effect of soft tissue models with different external forces applied during stretching operation, the situation is similar when pressing, which will not be discussed here. D. Analysis of Real-time Influencing Factors： Real-time performance of force and haptic rendering system of Tele-diagnosis and treatment robot is generally measured by frame rateError! Reference source not found.. Research shows that as long as the soft tissue model is deformed at a rate greater than 25 fps, it can basically meet the requirements of visual refreshment. Previous studies give the
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
16
frame rate of soft tissue model in 625:80s, which can meet the real-time requirement. Now change the particle number of the soft tissue model and analyze the change of the frame rate. Change the particle number of the soft tissue model, perform the interaction operation, calculate and record the frame rate of the interaction, draw the frame rate curves of the model in the time of 400 and 900:80s respectively, and compare the frame rate of the model with that of 625:00, the results are shown in Fig.17. From Fig.17, we can see that the more particles in soft tissue model, the lower the frame rate, the fewer particles and the higher the frame rate.
Fig. 17 Frame Rate Curves with Different Particle Points
V. DISCUSSION
From the test of force and haptic reappearance system based on Geomagic Touch Haptic device, we have found some results in the last section. Considering the above experimental results, we can propose the following discussion in this section: 1)
2)
3)
In the hand palpation experiment and the quality evaluation of haptic reappearance, we can conclude that based on this system, operators can not only visually see the process of soft tissue deformation in virtual environment, but also feel the force and its change process through haptic devices. Visual feedback (deformation rendering) is smooth, haptic feedback is effective, and simulation results are realistic. It can better meet the comprehensive requirements of real-time, stability and authenticity in the interaction process of force-haptic reappearance. In the experiment of applying external force, if only external force is applied to the collision point, the deformation of the collision point will be too large, while the deformation of other particles is relatively small, and the deformation is peak-shaped and not smooth. When the external force is applied to the collision point and a circle of particles around it, the deformation of the interaction part becomes smoother and more real. From the analysis of real-time influencing factors, we can infer that the more particles in soft tissue model, the lower the frame rate, the fewer particles, the higher the frame rate. It can be inferred that, under the same hardware conditions, the complexity of the model has an important impact on the real-time performance of the system, and the more complex the model, the worse the real-time performance of the system. As far as the model is concerned, the number of particles determines the number of spring elements, and then affects
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
17
the efficiency of collision detection and deformation rendering, which naturally affects the real-time performance of the system. In this paper, some progress has been made in the construction of force and haptic reappearance system, but there are still some shortcomings, which can be further studied and launched: 1)
2)
3)
Establish more complex and realistic soft tissue models, and incorporate medical tools to further explore the force-haptic rendering and deformation rendering of soft tissue cutting, suture, injection, puncture and other complex operations. In addition to realizing force-haptic reappearance, we can also consider how to judge whether there are lesions in soft tissues by haptic feedback. At the same time, a more complete system is designed to integrate various soft tissue models, so that operators can choose different models to simulate force and haptic reappearance under different operations according to their needs. In the current experiment, the quality evaluation of the haptic feedback system is mainly based on the user's experience and other artificial evaluation, which cannot be quantified and will also produce errors according to the user. It is hoped that a set of quantitative evaluation criteria for haptic feedback system can be obtained in future experiments, so as to evaluate the authenticity, stability and ease of use of haptic feedback system more accurately. VI. CONCLUSION
In this paper, a force and haptic reappearance system is constructed based on Geomagic Touch haptic device. Firstly, through the overall design of the system structure, the hardware and software conditions of the system are described. Then, according to the topological structure of the soft tissue model, the soft tissue surface model is built with OpenGL, a cross-platform graphics library, and the touch hand model in obj format is built with the professional three-dimensional modeling software 3Ds Max 2014 to import into the development environment. Under the action of non-gravity and gravity, the pressing and stretching operation of human hand on soft tissue model is simulated, and the interaction effect is discussed and analyzed. Since the questionnaires of the 10 participants in the experiment showed that the haptic feedback system could quickly feedback soft tissue deformation, forces to their hands, and the feeling of it is realistic, and show a positive evaluation to this system, we believe that the haptic feedback is stable and real, and the visual feedback is smooth. Finally, the number of particles in the soft tissue model is changed to perform interactive operation, the frame rate during the interaction is calculated and recorded, and the results are compared and analyzed, which shows that the number of particles in the soft tissue model affects the real-time performance of the system.
Declaration of Competing Interest
We would like to submit our manuscript entitled “Construction of Force Haptic Reapperence System Based on Geomagic Touch Haptic Device” to your journal. We confirm that this work is original and has not been published elsewhere nor is it currently under review for publication elsewhere. We assure that: (i) no support, financial or otherwise, has been received from any organization that may have an interest in the submitted work ; and (ii) there are no other relationships or activities that could appear to have influenced the submitted work."Thank you for your consideration.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
18
ACKNOWLEGDEMENT This work was jointly supported by the Sichuan Science and Technology Program(2019YJ0189) and the Fundamental Research Funds for the Central Universities (ZYGX2019J059).
REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18]
[19] [20] [21] [22]
Le H H, Loomes M J, Loureiro R C V. Group interaction through a multi-modal haptic framework[C]//2016 12th International Conference on Intelligent Environments (IE). IEEE, 2016: 62-67 Arata J, Kondo H, Sakaguchi M, et al. Development of a haptic device “DELTA-4” using parallel link mechanism[C]//2009 IEEE International Conference on Robotics and Automation. IEEE, 2009: 294-300. Massie T H, Salisbury J K. The phantom haptic interface: A device for probing virtual objects[C]//Proceedings of the ASME winter annual meeting, symposium on haptic interfaces for virtual environment and teleoperator systems. 1994, 55(1): 295-300. Qiao B, Chen G, Ye X. The research of soft tissue deformation based on mass-spring model[C]//2009 International Conference on Mechatronics and Automation. IEEE, 2009: 4655-4660. S. Cotin, H. Delingette, N. Ayache. Real-time elastic deformations of soft tissues for surgery simulation[J]. IEEE transactions on Visualization and Computer Graphics, 1999, 5(1): 62-73 H. Talbot, N. Haouchine, I. Peterlik, et al. Surgery Training, Planning and Guidance Using the SOFA Framework[C]. Eurographics. 2015 A. Ohnson, D. Sandin, G. Dawe, et al. Developing the paris: Using the cave to prototype a new vr display[J]. Proceedings of IPT, 2000: 19-20 C. Luciano, P. Banerjee, L. Florea, et al. Design of the immersivetouch: a high-performance haptic augmented virtual reality system[C]. 11th International Conference on Human-Computer Interaction, Las Vegas, NV. 2005 A. Alaraj, C. J. Luciano, D. P. Bailey, et al. Virtual Reality Cerebral Aneurysm Clipping Simulation With Real-time Haptic Feedback[J]. Neurosurgery, 2015, 11 Suppl 2(1): 52 S. C. Schvartzman, R. Silva, K. Salisbury, et al. Computer-aided trauma simulation system with haptic feedback is easy and fast for oral-maxillofacial surgeons to learn and use[J]. Journal of Oral and Maxillofacial Surgery, 2014, 72(10): 1984-1993 P. Wang, A. A. Becker, I. A. Jones, et al. A virtual reality surgery simulation of cutting and retraction in neurosurgery with force-feedback[J]. Computer methods and programs in biomedicine, 2006, 84(1): 11-18 A. Maciel, T. Halic, Z. Lu, et al. Using the PhysX engine for Physics-based Virtual Surgery with Force Feedback[J]. International Journal of Medical Robotics & Computer Assisted Surgery, 2010, 5(3): 341-353 N. Suzuki, A. Hattori, S. Ieiri, et al. VR Training System for Endoscopic Surgery Robot[J]. Chinese Journal of Clinicians, 2011, 7264(17): 117-129 C. Tercero, Z. Najdovski, S. Ikeda, et al. Haptic feedback in endovascular tele-surgery simulation through vasculature phantom morphology changes[C]. World Haptics. 2013:359-364 Silva A J, Ramirez O A D, Vega V P, et al. Phantom omni haptic device: Kinematic and manipulability[C]//2009 Electronics, Robotics and Automotive Mechanics Conference (CERMA). IEEE, 2009: 193-198 Kuber R, Zhu S, Arber Y, et al. Augmenting the non-visual web browsing process using the geomagic touch haptic device[J]. ACM SIGACCESS Accessibility and Computing, 2014 (109): 4-10 Reinhart C, Breton P F. Experimental validation of Autodesk® 3ds Max® Design 2009 and DAYSIM 3.0[J]. Leukos, 2009, 6(1): 7-35. Itkowitz B, Handley J, Zhu W. The OpenHaptics/spl trade/toolkit: a library for adding 3D Touch/spl trade/navigation and haptics to graphics applications[C]//First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference. IEEE, 2005: 590-591. Roth D, Ismail F, Bedi S. Mechanistic modelling of the milling process using an adaptive depth buffer[J]. Computer-Aided Design, 2003, 35(14): 12871303. Kadlecek P. Overview of current developments in haptic APIs[J]. Proceedings of CESCG, 2011, 2011. Avila R S, Sobierajski L M. A haptic interaction method for volume visualization[C]//Proceedings of Seventh Annual IEEE Visualization'96. IEEE, 1996: 197-204. T. Y. Fang, P. C. Wang, C. H. Liu, et al. Evaluation of a haptics-based virtual reality temporal bone simulator for anatomy and surgery training[J]. Computer methods and programs in biomedicine, 2014, 113(2): 674-681
Yushan.Tang received the Bachelor’s degree in Automation Engineering from the University of Electronic Science and Technology of China in 2019. She is currently pursuing the master’s degree in control science and technology at University of Electronic Science and Technology of China.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
19
Shan.Liu received the Bachelor’s degree in Mechanical Automation from Hubei University of Technology in 2001.She received the Master’s degree in Systems Engineering from Huazhong University of Science and Technology in 2004. She received the Doctor’s degree in Artificial Intelligence from Huazhong University of Science and Technology in 2008. From 2006 to 2007, she is a visiting scholar in the Department of BioDesign from Arizona state University in USA. From 2013 to 2014, she is a visiting scholar in Louisiana State University and A & M College. She is now a Lecturer of the University of Electronic Science and Technology of China in the School of Automation Engineering since 2008. Dr.Liu is now a member of IEEE.
Yaru.Deng received the Bachelor’s degree in Automation Engineering from the University of Electronic Science and Technology of China in 2019. She is currently pursuing the master’s degree in control science and technology at University of Electronic Science and Technology of China.
Lirong.Yin received the Bachelor’s degree of GIS from University of Iowa in 2019. She is currently pursuing the Master’s degree in the Department of Geography and Anthropology at Louisiana state university.
Wenfeng.Zheng received the Doctor’s degree in Earth Exploration and Information Technology from Chengdu University of Technology in 2008. He is now working as a associate professor in the School of Automation Engineering at University of Electronic Science and Technology of China since 2008. he was a visiting scholar in the Department of Environmental Sciences at Louisiana state university from 2011 to 2013. Dr. Zheng is a member of Association for Computing Machinery since 2009, a member of IEEE since 2009. He is a member of America Association Geographer since 2016, and a member of American Geophysical Union since 2018. Dr.Zheng also owns a membership of China Association of Inventions since 2018 .
Construction of Force Haptic Reappearance System Based on Geomagic Touch Haptic Device Yushan Tang , Shan Liu Member IEEE , Yaru Deng , Yuhui Zhang , Lirong Yin , Wenfeng Zheng Member IEEE PII: DOI: Reference:
S0169-2607(19)31362-8 https://doi.org/10.1016/j.cmpb.2020.105344 COMM 105344
To appear in:
Computer Methods and Programs in Biomedicine
Received date: Revised date: Accepted date:
14 August 2019 16 January 2020 16 January 2020
Please cite this article as: Yushan Tang , Shan Liu Member IEEE , Yaru Deng , Yuhui Zhang , Lirong Yin , Wenfeng Zheng Member IEEE , Construction of Force Haptic Reappearance System Based on Geomagic Touch Haptic Device, Computer Methods and Programs in Biomedicine (2020), doi: https://doi.org/10.1016/j.cmpb.2020.105344
This is a PDF file of an article that has undergone enhancements after acceptance, such as the addition of a cover page and metadata, and formatting for readability, but it is not yet the definitive version of record. This version will undergo additional copyediting, typesetting and review before it is published in its final form, but we are providing this version to give early visibility of the article. Please note that, during the production process, errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain. © 2020 Published by Elsevier B.V.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
1
Highlights In this paper, the force-displacement, stress relaxation and creep data of the soft tissue model were collected and compared with those of the real soft tissue.After comparison, it is found that the data of this model are basically within the range of actual measured values, and the model has certain accuracy and validity, which is superior to the empirical Particle-Spring model.The frame rate in this paper is much higher than the frame rate required for real-time performance, which verifies the real-time performance of the system.
Yushan Tang is with the School of Automation, University of Electronic Science and Technology of China, Chengdu 610054, Sichuan, P. R. China (e-mail: [email protected] https://orcid.org/0000-0002-8357-6595). Yaru Deng is with the School of Automation, University of Electronic Science and Technology of China,Chengdu 610054, Sichuan, P. R. China(e-mail:[email protected], https://orcid.org/0000-0003-4004-0448). Yuhui Zhang is with the School of Automation, University of Electronic Science and Technology of China, Chengdu 610054, Sichuan, P. R. China (e-mail: [email protected]) Shan Liu was with the School of Automation, University of Electronic Science and Technology of China, Chengdu 610054, Sichuan, P.R.China, now she is with the Department of Modelling, Simulation, and Visualization Engineering, Old Domain University, Norfolk, VA 23529, USA(e-mail: [email protected] ORCID: 0000−0002−8040−0367). Lirong Yin is with the Department of Geography and Anthropology, Louisiana State University, Baton Rouge, LA 70803, United States (e-mail: [email protected], [email protected], ORCID：0000-0002-5022-610X). Wenfeng Zheng* is with the School of Automation, University of Electronic Science and Technology of China, Chengdu 610054, Sichuan, P. R. China (e-mail: [email protected] ORCID：0000-0002-8486-1654).
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
2
Construction of Force Haptic Reappearance System Based on Geomagic Touch Haptic Device (August. 2019) Yushan Tang, Shan Liu, Member, IEEE. Yaru Deng, Yuhui Zhang, Lirong Yin, Wenfeng Zheng*, Member, IEEE, Corresponding author: Wenfeng Zheng, E-mail:[email protected]
Abstract—Force haptic reappearance technology is considered to be one of the top ten technologies that can change human life in the future. It has broad application prospects and market demand. Most of the existing medical robots, especially the remote diagnosis and treatment robots, lack haptic feedback, or the calculation of feedback force is insufficient. Haptic reappearance technology is an effective method to solve the problem of haptic presence and improve the practicability of medical robot. The ultimate goal of the force haptic reappearance system is to let the operator feel the haptic feedback when interacting with the soft tissue model in the virtual environment in real time. Haptic device is the necessary condition to realize force haptic reappearance, and it is an essential part of the system. Its important role is to introduce the external force imposed by the operator into the virtual environment, and let the operator feel the force in the virtual environment, which effectively guarantees the operator's sense of reality and immersion when interacting with the virtual environment. Therefore, starting with the key technology of force and haptic reappearance system, this paper studies the construction of force and haptic reappearance system. Soft tissue surface model is drawn by OpenGL, and hand model is drawn by 3Ds Max. The haptic reappearance and visual feedback of soft tissue model of hand palpation are realized. The quality of feedback is evaluated. The haptic reappearance is stable and realistic, and the visual feedback is smooth. This indicates that the system has a certain application value and is worth to promote to the public. Index Terms—3Ds Max, Force and haptic, Geomagic Touch, OpenGL, Soft tissue model
I. INTRODUCTION
H
APTIC devices are electromechanical devices with input and output functions, which allow them to move with several degrees of freedom. Operators can manipulate the haptic devices to feel the objects in the virtual environment and produce realistic haptic sensation. The realization of force and haptic reappearance system is to feed back the reaction force produced in the interaction process to the operator through the haptic device, so the haptic device is the necessary condition to realize force and haptic reappearance.
So far, great progress has been made in the research of haptic devices. Universities, institutes and companies around the world have developed a variety of devices, some of which have even been commercialized, such as Novin’s Falcon device [1], Force Dimension’s Omega device and Delta device[2], SensAble’s Phantom device[3] and so on. The French National Institute of Information and Automation (INRIA)[4] maintains the world’s leading edge in this field. Cotin uses Laparoscopic Impulse Engine () force feedback system to create a liver anatomy model based on finite element method and realize haptic feedback, allowing doctors to manipulate, deform and cut the liver model [5] . The open source SOFA simulation platform developed by their team is widely used worldwide[6].
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
3
Electronic Visualization Laboratory (EVL) of the University of Illinois at Chicago, USA, developed a personal augmented reality immersion system PARIS based on stereo projection using Phantom Desktop haptic deviceError! Reference source not found. .Subsequently, Luciano added the integrated haptic device Immersive Touch on the basis of PARIS system to form a haptic augmented reality systemError! Reference source not found.. Its hardware integrated 3D stereo visualization, force feedback, head and hand tracking and 3D audio. In 2015, in collaboration with the University of Chicago, the Medical College of the University of Chicago built an immersive aneurysm shearing system with two haptic devices, which realized the haptic simulation of cerebral aneurysm clipping operationError! Reference source not found.. These applications have inspired more scholars to analyze and discuss the technique of force and haptic reappearance. The Department of Computer Science at Stanford University has also developed an oral and maxillofacial surgery system with two-hand haptic feedback for surgeons to learn and useError! Reference source not found..Wang used two Phantom Omni haptic devices to train neurosurgery for the first timeError! Reference source not found.. Based on NVIDIA’s physical engine PhysX, Brazil’s Macel designed an interaction surgical simulation framework with haptic feedback using Phantom haptic deviceError! Reference source not found.. Suzuki of Japan put forward the model of filling ball. With Phantom Omni haptic device, the cutting simulation of soft tissue inside stomach is better realizedError! Reference source not found. . Tercero of Nagoya University proposed a force feedback method for vascular surgery simulation based on photo elastic effectError! Reference source not found..The precise vascular model was established by magnetic resonance and CT images, and the haptic reconstruction of remote catheter interventional surgery was also realized by Phantom Omni[15]. These experiments have led to further applications of the sense of touch in medicine. The core technology used in this paper is the Geomagic Touch haptic Device [16], which is a mid-grade professional impedance haptic force feedback device. It is also the most affordable haptic device at present. It is made of metal parts and injection plastics. Its safety has been certified by FCC, CE and RoHS. Its main working parameters are shown in Table Ⅰ. The characteristics are as follows: 1.Small size, compact structure, easy to carry, with comfortable molded rubber, textured paint and removable stylus, two integrated instantaneous stylus switches and multi-functional indicator lights. 2.Solid grip, sensitive response, 3-DOF force feedback and 6-DOF position sensor can provide greater feedback force and workspace. 3.The interface is the firewall interface of IEEE 1394. It can be directly connected to the computer’s USB port through RJ45 Ethernet cable and USB Ethernet adapter. 4.Support a variety of haptic applications, allowing users to customize. The ultimate goal of force and haptic reappearance system is to make the operator feel real-time haptic feedback when interacting with soft tissue model in virtual environment. In this paper, the most difficult and challenging task is to simulate the human hand palpation behavior using haptic devices. We need to model a soft tissue model that is very similar to the real soft tissue, and let participants interact with it, and evaluate the quality of the system according to the results of participants' feedback. The highlights of our work is that we successfully use Geomagic Touch haptic device and 3Ds Max to construct a relatively real and stable human-computer interactive palpation system.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
working space weight Scope of activities Friction Maximum Output Force Location resolution Rigidity
Force Feedback (6 Degrees of Freedom) inertia Interface Support platform
4
TABLE I Main Working Parameters of Geomagic Touch 160 mm x 120 mm x 70 mm 3pound Hand Motion with Wrist Joint as Axis <0.26N 3.3N > 450dpi/~0.055mm X axis > 1.26N/mm Y axis > 2.31N/mm Z axis > 1.02N/mm X, y and Z axes 45g IEEE 1394 Fireline Interface Personal Computer with Intel/AMD Chip
The main work of this paper is as follows: 1.Based on Geomagic Touch haptic device, the force and haptic reappearance system is constructed. 2.Based on the construction of force-haptic reappearance system, this paper use 3Ds Max[17] to simulate hand palpation of soft tissue model. 3.The operator manipulates the human hand in the virtual environment to interact with the soft tissue model in the virtual environment by manipulating the haptic device. 4.The effects of interaction are discussed and analyzed, and further research directions are proposed in this paper. II.
HARDWARE AND SOFTWARE CONDITIONS FOR BUILDING THE SYSTEM
A. Hardware Conditions of the System
The hardware of force and haptic reappearance system of Tele-diagnosis and treatment robot mainly includes computer and haptic equipment. The computer is responsible for the construction, control and coordination of the virtual environment. The computer configuration in this paper is Windows 10 environment, Intel (R) Core (TM) i7-4790 CPU @3.60 GHz, 8GB memory and Acer LCD display. The haptic device in this paper is Geomagic Touch haptic device of American 3D Systems Company. B. Software Conditions of the System
Microsoft Visual Studio 2013 is used as the programming environment for the force and haptic reappearance system of Tele-diagnosis and treatment robot. The programming language is C/C++.The overall implementation framework of the system is based on OpenHaptics. It is an upgraded version of the previous GHOST (General Haptics Open Software Toolkit) SDK. Although its name is Open, it is not open source. This article uses OpenHaptics version 3.4, which is the latest version. OpenHaptics simplifies programming by encapsulating basic steps common to haptic/graphical applications, and has good compatibility. It supports Win32 and X64 development platforms on 64-bit systems. OpenHaptics Touch Device Driver (GTDD), Haptic Device API (HDAPI), Haptic Library API (HLAPI) and Quick API are the main components of OpenHaptics Touch Device Driver (GTDD), as shown in Fig.1.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
5
Fig. 1 The Component of OpenHaptics
GTDD is the driver of Geomagic Touch haptic device, which guarantees the communication between computer and haptic device. HDAPI provides low-level access to haptic devices and control over the runtime of configuration drivers. HLAPI is a high-level API for haptic rendering after OpenGL graphics rendering. It allows the reuse of existing OpenGL code and can easily and quickly add touch to graphics applications, greatly simplifying the synchronization of touch and graphics threads. HLAPI is based on HDAPI[18]. HLAPI still needs to use HDAPI functions in the initialization, security processing, status and location query of haptic devices. HLAPI also provides haptic effects such as gravity and inertia. Quick API is a miniature API. It encapsulates many functions on the basis of HDAPI and HLAPI through STL template library of C++.Quick API provides two versions, one is Win32 and the other is GLUT version of OpenGL.Win32 version supports multi-window applications, while GLUT version does not support them. The GLUT version is independent of the operating system, while the Windows 32 version is only applicable to Microsoft’s Windows operating system. After the Open GL and HLAPI initializations, the HDAPI is called to initialize the force feedback device. HLAPI processes events at regular intervals to obtain Proxy values and position values of force feedback devices, which are then calculated by HDAPI to give the resultant force. The HDAPI for communicating with force feedback devices in program development consists of two parts: Device and Scheduler. The Device section manages the haptic devices, also known as PHANTOM Desktop, including state management, parameter setting, power transfer, and the ability to manage multiple haptic devices in tandem simultaneously. The state includes sensor position, button pressed or not, sensor speed, etc. The Scheduler is responsible for managing the high-frequency threads to transmit power and to accept or send out the device state. Since the Scheduler section is provided in the API, the consumer can leave the high frequencies required for haptic processing to the API, as long as the Scheduler of the HDAPI communicates with and executes the instructions. From Fig.1, it can also be known that the refresh rates of haptic feedback and visual feedback are different. The refresh rates of haptic device force are one order of magnitude higher than those required for image display [21]. This difference stems from the psychophysics of human perception. Usually, the refresh rate of visual feedback is not less than 30Hz, and the haptic feedback can become undistorted when the frequency is above 300Hz and can obtain satisfactory results when the frequency is above 1000Hz. Because of the above reasons, the frequency range of visual feedback is above 30Hz but the range of haptic feedback is above 1000Hz. Therefore, haptic rendering and
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
6
graphical visual feedback are usually executed simultaneously in different threads of the same process to meet their respective refresh rates. The callback mechanism of HLAPI implements that haptic rendering model is specified in the same thread at the same speed as graphics, instead of synchronizing the state of haptic and graphical threads in haptic threads. At the same time, events occurring during haptic rendering are transmitted to client programs through callback functions. It can also be seen from Fig.1 that HLAPI has two buffers, depth buffer and feedback buffer[19], for processing haptic rendering calculations based on geometric primitives, transformations and material attributes. When the feedback buffer stores points, line segments and polygons for haptic rendering, the depth buffer uses images read from the depth buffer for haptic rendering. For a large number of primitives, the deep buffer occupies less memory and is more efficient, while the feedback buffer is more efficient for a small number of primitives. Generally speaking, the depth buffer is not as accurate as the feedback buffer. Although the difference in accuracy is almost undetectable, the feedback buffer must be used to render the constrained points and lines, because the depth buffer cannot capture the points and lines. In specific applications, the type of buffer should be selected according to actual needs. In this paper, the feedback buffer is used. In order to better realize force and haptic reappearance and realistic visual feedback, and reuse the existing OpenGL code, this paper uses HLAPI framework to construct haptic system, and the development process as shown in Fig.2 is obtained. The haptic feedback model used in this paper is the proxy model proposed by Ruspini et al. The principle of the model generally uses a sphere without mass to represent the agent of the end of the haptic device in the virtual environment. When the operator moves the haptic device, the agent moves with it. In the proxy model, two location information of haptic device are saved: the device end location and the surface control point location (proxy). Among them, the surface control point is a point that closely follows the end position of the haptic device but is limited to the surface of all touchable objects. The operator manipulates the haptic device to interact with the soft tissue model in the virtual environment. When the end of the haptic device does not contact the soft tissue model, the position of the surface control point coincides with the position of the end of the device. When the end of the device actually penetrates the soft tissue surface, the position of the surface control point is the projection of the actual position of the end of the device on the soft tissue surface. At this time, a virtual spring damper is simulated between the surface control point and the actual position of the end of the device to calculate the force transmitted to the haptic device. The operator can feel the force from the haptic device, which is force feedback, which can simulate the haptic effect when contacting with the soft tissue model in the virtual environment. The feedback force can be calculated by the depth of "penetration", and the calculation method is shown in (1). •
𝐹 = 𝐾𝑥𝑑 + 𝐷𝑥𝑑
(1)
By setting the values of K and D, the haptic characteristics of the model surface can be controlled. Where K is the stiffness coefficient and can be set to any value between 0 and 1, 0 is the softest surface and 1 is the hardest surface. The haptic device has stable rendering capability, and setting K to an unreasonably high value will lead to instability or rigidity. D is the damping coefficient, which exists to reduce vibration, and the setting range is also between 0 and 1. Soft tissue is relatively soft. The parameter of force feedback in this paper is set to K=0.15, D=0.9. As shown in Fig.2, initialization is the first step after the program starts. Initialization includes the initialization of OpenGL, the initialization of haptic devices and the calibration of haptic devices [20]. Firstly, the haptic devices must be calibrated to obtain accurate three-dimensional position data. Secondly, all applications in a virtual environment must determine the appropriate mapping of the haptic workspace to the graphical scene so that all visible objects
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
7
can be touched. Haptic workspace is the reachable physical space of haptic devices. Basically, the whole working area is chosen, that is, the working area shown in Table Ⅰ. All haptic rendering in HLAPI is done in haptic frames, which begin with HLBeginframe () and end with HLEndframe (). Typically, each graphics rendering frame has a corresponding haptic rendering frame. HLAPI’s deformation rendering is used to render the established soft tissue model (proxy rendering). The content of deformation rendering is surrounded by HLBeginshape () and HLEndshape (). HLAPI captures the geometric objects specified by OpenGL and assigns unique identifiers, and performs haptic rendering on them. In order to facilitate the interaction with soft tissue models in virtual environment, visual haptic devices are needed, that is, display the location of haptic devices in space, and observe the changes when they interact with other objects in the scene. Three-dimensional cursor rendering is to achieve this function. The three-dimensional cursor defines the haptic interface points, and draws the location of the agent of the haptic device in the virtual scene (which can be obtained by the driver GTDD), as well as its size (which can be customized according to the display needs). Because there can be multiple haptic devices, there can be multiple three-dimensional cursors.
Fig. 2 HLAPI Development Process
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
8
The model drawing of force and haptic reappearance system of Tele-diagnosis and treatment robot mainly refers to the establishment of soft tissue model and palpation hand model. For force and haptic reappearance system, simple models can be directly drawn by OpenGL, while complex models are usually built by professional threedimensional modeling software such as AutoDesk 3Ds Max, SolidWorks, FreeForm, etc., and then imported into the development environment of Visual Studio through the program. III. METHOD A. Soft Organization Model Drawing Based on OpenGL ：
In this paper, a cross-platform graphics library OpenGL is used to build and display soft tissue surface model. OpenGL is known as "a software interface of graphics hardware". It is a procedural rather than descriptive graphics API. It has no window management and can be used to create graphics with excellent visual quality. OpenGL has the following characteristics: 1.OpenGL is an open source, cross-system, cross-language professional graphics program interface, which defines a special data type. 2.It has high portability and can provide powerful and convenient graphics library for the system. 3.It has a very fast speed, and its rendering speed is much faster than that of ray tracker or software rendering engine. 4.It is widely used, covering engineering, games and other modeling applications. Soft tissue modeling with OpenGL basically follows the workflow shown in Fig.3. Rasterization is the process of converting vertices into sheets. In fact, it fills the pixels between each coordinate point to form line segments. Grating can enhance the 3D sensation of the model, but the corresponding illumination processing and projection transformation must be done before rasterizing.
Fig. 3 The Workflow of OpenGL
In order to make the edge of the deformation region smooth during the deformation of soft tissue model, and more in line with the effect of the actual deformation of soft tissue, the mass spring soft tissue model is established based on the triangle topological structure. According to the topological structure of the selected mass-spring soft tissue model, the parameters of the spring itself are determined. The mass of a single mass is equal to 0.01. If the original length of the structural spring connecting the adjacent particles in the horizontal and vertical directions is set as 1, the original length of the shear spring in the diagonal direction is equal to 2, and the original length of the bending spring connecting the interval particles in the horizontal and vertical directions is equal to 2.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
9
According to the topological structure of the soft tissue model above, the soft tissue model established by OpenGL is shown in Fig.4.The soft tissue model shown in Fig.5 consists of 625 (25 x 25) particles and 3502 Voigt spring elements. The yellow points in the figure represent the particles, and the rose-red lines represent the spring elements connected between the particles. The soft tissue model that does not display the particle spring after rendering is shown in Fig.5.
Fig. 4 Soft Tissue Model
Fig. 5 Soft Tissue Model Rendered
B. Hand Model Rendering Based on 3Ds Max：
OpenGL is relatively difficult to use in complex three-dimensional modeling, so the human hand threedimensional model in force and haptic reappearance system is established and managed by Autodesk’s professional three-dimensional modeling software, 3Ds Max 2014.3Ds Max is a high-performance three-dimensional modeling and rendering software, which is widely used in design, video animation and games and other industries. It has low requirement for PC configuration, powerful modeling function, good expansibility, simple operation and easy to use. In this paper, the human hand model established by the classical polygon modeling method of 3D software in 3Ds Max 2014 is shown in Fig.7.The hand model shown in Fig.6 contains 7822 vertices and 15640 faces. Exporting the established hand model to obj format and importing it into Visual Studio 2013 development environment through program can greatly improve the efficiency of modeling.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
10
Fig. 6 Human Hand Model Established by 3Ds Max 2014
The human model in obj format is imported into Visual Studio 2013 development environment, using GLM header file and implementation file, which defines the corresponding vertex, color, material and structure of the model description of the obj file. Add the above file in the program, read the obj file with glmReadObj () function, and draw and display the model with glmDraw () function. The 3D model built by 3Ds Max does not necessarily match the size of the soft tissue model built by OpenGL in visual display, so the glmScale () function is also used to zoom the model in a certain proportion. The model built by 3Ds Max in this paper shows too small after importing in the original proportion, so the hand model is enlarged 15 times. The reason why we need to build a complex human hand model is that in the process of interaction, the touching and pressing activities of participants on soft tissues need to be simulated as realistically as possible through this human hand model. Therefore, the human hand model we built must be realistic, so as to make the experimental results accurate enough, and make the test response as sensitive as possible. According to the principle of proxy model, haptic devices are generally modeled as a sphere in virtual environment. To simulate hand palpation, it is necessary to model the proxy of haptic devices into an imported hand model in the three-dimensional cursor rendering stage. C.
Realization of Force and Haptic Reappearance System Based on Geomagic Touch：
Based on the above software and hardware conditions, soft tissue model and hand palpation model, a force and haptic reappearance system based on Geomagic Touch haptic device is constructed, as shown in Fig.7. The specific workflow of force and haptic reappearance system is as follows: 1.Initialization, calibration of haptic devices, setting force-haptic characteristics to create virtual environments, and determining error handling mechanisms such as motion overrun. 2.OpenGL initialization, the introduction of hand model, drawing soft tissue surface model. 3.Operators operate haptic devices to manipulate human hands in virtual environments and interact with soft tissue models in virtual environments. 4.Real-time acquisition of the position of the end of the haptic device, and collision detection based on the location information. If the collision occurs, the collision point is determined, and the external force is applied to the particle and the nearest circle of particles around it.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
11
5.According to the applied external force, the force of the soft tissue model is calculated, the particle velocity and displacement are calculated iteratively, the model is updated, and the deformation rendering is carried out. 6.Haptic rendering, calculating the feedback force, and feedback to the operator through the device, circular reciprocating, realizing the force and haptic reappearance and visual feedback in the process of human-computer interaction. 7.After the interaction, do some cleaning work to ensure the correct release of haptic device resources and memory resources.
Fig. 7 Force and Haptic Reappearance System
IV. RESULTS OF FINDING
A. Hand Palpation Experiment： Palpation is a process in which surgeons touch and press tissues or body organs with their fingers to determine if they have abnormalitiesError! Reference source not found..Run the program in Visual Studio 2013 development environment, hold the end of Geomagic Touch haptic device, control the hand movement in virtual environment through the end of mobile device, press or stretch the hand on the screen along the soft tissue surface, and simulate the touch operation. The shape change of the soft tissue model can be seen on the display screen, and the force and haptic information generated in the process of pressing and stretching the model is fed back to the operator through the haptic device. With the increase of press and stretch depth, the feedback force also changes. When the hand leaves the soft tissue model, the deformation of the model gradually decreases and finally restores to its original state. Soft tissue does not deform when the hand is not in contact with the soft tissue model, as shown in Fig.8.Soft tissue deforms and sinks downward when the surface of the soft tissue model is pressed by hand, as shown in Fig.9. The hand stretches the soft tissue model, and the soft tissue deforms and protrudes upward, as shown in Fig.10.When the hand leaves the soft tissue model, the deformation of the model slowly recovers, as shown in Fig.11.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
12
Fig. 8 Hand Contactless Model
Fig. 9 Hand Pressure Deformation
Fig. 10 Hand Tensile Deformation
OpenHaptics can also simulate gravity effects. Gravity effect is added to the soft tissue surface model. Fig. 12 gives two kinds of soft tissue models after applying gravity. The observation state of the soft tissue with added gravity from the perspective of top to bottom is expressed as (a) and the state of the soft tissue as seen from the bottom-up perspective is expressed as (b). In order to show the obvious topological structure of particle-spring. In previous studies, in order to stabilize the whole force and haptic reappearance system, the soft tissue model is constrained by deformation, and the particles around the model are fixed. When gravity is added, it will show the effect as shown in Fig.12. If the fixed particles around are cancelled, the model will gradually fall under the action of gravity and eventually disappear in the graphical display interface. If only deformation constraints are applied to a particle in one side direction, the soft tissue will be suspended in the graphical display interface like a curtain when gravity effect is added.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
13
Fig. 11 Deformation Recovery
(a)
(b) Fig. 12 Applies Gravity Effects. (a) From a Up-bottom Perspective; (b) From a Bottom-up Perspective
Compression and stretching were performed on the soft tissue model with added gravity effect, and the deformation effect was shown in Fig. 13 and Fig. 14. As it can be seen from the two figures, the compression and stretching effects of soft tissue under gravity still have a certain degree of fidelity. It is worth noting that the effect of force feedback of soft tissue model under gravity is different. If the applied gravity is opposite to the direction of operation, the operator will feel more reaction force. B. Quality Evaluation of Haptic Reappearance： At present, there are no quantitative indicators to evaluate the quality of haptic feedback. Since the main user of force and touch feedback system is human, the main criterion to evaluate the system performance should be human experience of using the system, so in this paper, subjective evaluation in the form of questionnaires is the main method. Referring to the artificial evaluation index used in haptic reappearance of surgical simulator in reference [22], this paper invites 10 participants to operate haptic devices and interact with soft tissue models in virtual environment to evaluate the quality of haptic reappearance of the system subjectively from three aspects: authenticity, stability and ease of use. In this experiment, there is no specific quantitative standard for the user's experience of the haptic representation system using the force, so in this paper, we take the user's sense of use as the evaluation index. When 8 out of 10 participants have positive comments on the system, it is deemed that the system could reproduce and feedback force and touch more realistically. Authenticity refers to whether the force in the virtual environment can be felt through the haptic device during the deformation process after the deformation of soft tissue. The greater the operation force, the greater the deformation. With the increase of the deformation depth, the greater the feedback force, and whether the deformation can be restored after the removal of the external force. Usability refers to the ease of interaction between haptic devices and soft tissue models. Stability refers to whether the feedback force is stable and there is no severe jitter during the whole palpation interaction process.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
14
10 participants in the test ranged in age from 23 to 27, with a male-to-female ratio of 1:1 and right-handed habits. Before all participants are tested, demonstration is done first, and then operation steps are explained. Attentions are given. Participants are asked to focus on the force feedback given by haptic devices when touching soft tissues. Finally, the authenticity, ease of use and stability of the force and haptic reappearance system are evaluated according to their subjective feelings.
Fig. 13 Tensile Deformation under Gravity
Fig. 14 Pressure Deformation under Gravity
As for the authenticity of the feedback, all participants felt haptic feedback in the process of manipulating the haptic device and interacting with the soft tissue model, and with the increase of deformation depth, the feedback force increased obviously. After the withdrawal of external force, the soft tissue model can indeed recover the deformation, and the haptic feedback has better authenticity. However, the haptic feedback of most participants in the stretching process is more obvious than that in the pressing process, presumably due to the need to overcome the inertia and gravity of the haptic device during the stretching process. In terms of usability, participants generally believed that it was easy to interact between haptic devices and soft tissue models. Individual participants felt that they were not used to it at first, so they would be better if they adapted to it. As far as the stability of the system is concerned, most participants do not feel obvious jitter during the whole interaction process. A few participants think that they will feel slight jitter when stretching to the deformation limit, and the system has better stability. C. Contrast Experiments on External Force Application：
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
15
Generally, the external force is applied only to the collision point. This paper optimizes the method by applying the external force to the collision point and a circle of particles around it. The effect of only applying external force to the impact point is shown in Fig.15, while the effect of optimized deformation is shown in Fig.16.In order to facilitate comparison, the particle and spring of the model are displayed, and the agent of haptic devices in virtual environment is modeled as a small sphere.
Fig. 15 Deformation Effect of External Force on Collision Point
Fig. 16 Optimized Deformation Effect
Compared with Fig.15 and Fig.16, it can be seen that if only external force is applied to the collision point, the deformation of the collision point will be too large during deformation rendering, while the deformation of other particles will be relatively small, showing a peak shape and not smooth. After the external force is applied to the collision point and a circle of particles around the collision point for optimization, the shape change of the interaction part is smooth and relatively more real. It should be noted that although Fig.15 and Fig.16 only compare the deformation rendering effect of soft tissue models with different external forces applied during stretching operation, the situation is similar when pressing, which will not be discussed here. D. Analysis of Real-time Influencing Factors： Real-time performance of force and haptic rendering system of Tele-diagnosis and treatment robot is generally measured by frame rateError! Reference source not found.. Research shows that as long as the soft tissue model is deformed at a rate greater than 25 fps, it can basically meet the requirements of visual refreshment. Previous studies give the
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
16
frame rate of soft tissue model in 625:80s, which can meet the real-time requirement. Now change the particle number of the soft tissue model and analyze the change of the frame rate. Change the particle number of the soft tissue model, perform the interaction operation, calculate and record the frame rate of the interaction, draw the frame rate curves of the model in the time of 400 and 900:80s respectively, and compare the frame rate of the model with that of 625:00, the results are shown in Fig.17. From Fig.17, we can see that the more particles in soft tissue model, the lower the frame rate, the fewer particles and the higher the frame rate.
Fig. 17 Frame Rate Curves with Different Particle Points
V. DISCUSSION
From the test of force and haptic reappearance system based on Geomagic Touch Haptic device, we have found some results in the last section. Considering the above experimental results, we can propose the following discussion in this section: 1)
2)
3)
In the hand palpation experiment and the quality evaluation of haptic reappearance, we can conclude that based on this system, operators can not only visually see the process of soft tissue deformation in virtual environment, but also feel the force and its change process through haptic devices. Visual feedback (deformation rendering) is smooth, haptic feedback is effective, and simulation results are realistic. It can better meet the comprehensive requirements of real-time, stability and authenticity in the interaction process of force-haptic reappearance. In the experiment of applying external force, if only external force is applied to the collision point, the deformation of the collision point will be too large, while the deformation of other particles is relatively small, and the deformation is peak-shaped and not smooth. When the external force is applied to the collision point and a circle of particles around it, the deformation of the interaction part becomes smoother and more real. From the analysis of real-time influencing factors, we can infer that the more particles in soft tissue model, the lower the frame rate, the fewer particles, the higher the frame rate. It can be inferred that, under the same hardware conditions, the complexity of the model has an important impact on the real-time performance of the system, and the more complex the model, the worse the real-time performance of the system. As far as the model is concerned, the number of particles determines the number of spring elements, and then affects
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
17
the efficiency of collision detection and deformation rendering, which naturally affects the real-time performance of the system. In this paper, some progress has been made in the construction of force and haptic reappearance system, but there are still some shortcomings, which can be further studied and launched: 1)
2)
3)
Establish more complex and realistic soft tissue models, and incorporate medical tools to further explore the force-haptic rendering and deformation rendering of soft tissue cutting, suture, injection, puncture and other complex operations. In addition to realizing force-haptic reappearance, we can also consider how to judge whether there are lesions in soft tissues by haptic feedback. At the same time, a more complete system is designed to integrate various soft tissue models, so that operators can choose different models to simulate force and haptic reappearance under different operations according to their needs. In the current experiment, the quality evaluation of the haptic feedback system is mainly based on the user's experience and other artificial evaluation, which cannot be quantified and will also produce errors according to the user. It is hoped that a set of quantitative evaluation criteria for haptic feedback system can be obtained in future experiments, so as to evaluate the authenticity, stability and ease of use of haptic feedback system more accurately. VI. CONCLUSION
In this paper, a force and haptic reappearance system is constructed based on Geomagic Touch haptic device. Firstly, through the overall design of the system structure, the hardware and software conditions of the system are described. Then, according to the topological structure of the soft tissue model, the soft tissue surface model is built with OpenGL, a cross-platform graphics library, and the touch hand model in obj format is built with the professional three-dimensional modeling software 3Ds Max 2014 to import into the development environment. Under the action of non-gravity and gravity, the pressing and stretching operation of human hand on soft tissue model is simulated, and the interaction effect is discussed and analyzed. Since the questionnaires of the 10 participants in the experiment showed that the haptic feedback system could quickly feedback soft tissue deformation, forces to their hands, and the feeling of it is realistic, and show a positive evaluation to this system, we believe that the haptic feedback is stable and real, and the visual feedback is smooth. Finally, the number of particles in the soft tissue model is changed to perform interactive operation, the frame rate during the interaction is calculated and recorded, and the results are compared and analyzed, which shows that the number of particles in the soft tissue model affects the real-time performance of the system.
Declaration of Competing Interest
We would like to submit our manuscript entitled “Construction of Force Haptic Reapperence System Based on Geomagic Touch Haptic Device” to your journal. We confirm that this work is original and has not been published elsewhere nor is it currently under review for publication elsewhere. We assure that: (i) no support, financial or otherwise, has been received from any organization that may have an interest in the submitted work ; and (ii) there are no other relationships or activities that could appear to have influenced the submitted work."Thank you for your consideration.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
18
ACKNOWLEGDEMENT This work was jointly supported by the Sichuan Science and Technology Program(2019YJ0189) and the Fundamental Research Funds for the Central Universities (ZYGX2019J059).
REFERENCES [1] [2] [3] [4] [5] [6] [7] [8] [9] [10] [11] [12] [13] [14] [15] [16] [17] [18]
[19] [20] [21] [22]
Le H H, Loomes M J, Loureiro R C V. Group interaction through a multi-modal haptic framework[C]//2016 12th International Conference on Intelligent Environments (IE). IEEE, 2016: 62-67 Arata J, Kondo H, Sakaguchi M, et al. Development of a haptic device “DELTA-4” using parallel link mechanism[C]//2009 IEEE International Conference on Robotics and Automation. IEEE, 2009: 294-300. Massie T H, Salisbury J K. The phantom haptic interface: A device for probing virtual objects[C]//Proceedings of the ASME winter annual meeting, symposium on haptic interfaces for virtual environment and teleoperator systems. 1994, 55(1): 295-300. Qiao B, Chen G, Ye X. The research of soft tissue deformation based on mass-spring model[C]//2009 International Conference on Mechatronics and Automation. IEEE, 2009: 4655-4660. S. Cotin, H. Delingette, N. Ayache. Real-time elastic deformations of soft tissues for surgery simulation[J]. IEEE transactions on Visualization and Computer Graphics, 1999, 5(1): 62-73 H. Talbot, N. Haouchine, I. Peterlik, et al. Surgery Training, Planning and Guidance Using the SOFA Framework[C]. Eurographics. 2015 A. Ohnson, D. Sandin, G. Dawe, et al. Developing the paris: Using the cave to prototype a new vr display[J]. Proceedings of IPT, 2000: 19-20 C. Luciano, P. Banerjee, L. Florea, et al. Design of the immersivetouch: a high-performance haptic augmented virtual reality system[C]. 11th International Conference on Human-Computer Interaction, Las Vegas, NV. 2005 A. Alaraj, C. J. Luciano, D. P. Bailey, et al. Virtual Reality Cerebral Aneurysm Clipping Simulation With Real-time Haptic Feedback[J]. Neurosurgery, 2015, 11 Suppl 2(1): 52 S. C. Schvartzman, R. Silva, K. Salisbury, et al. Computer-aided trauma simulation system with haptic feedback is easy and fast for oral-maxillofacial surgeons to learn and use[J]. Journal of Oral and Maxillofacial Surgery, 2014, 72(10): 1984-1993 P. Wang, A. A. Becker, I. A. Jones, et al. A virtual reality surgery simulation of cutting and retraction in neurosurgery with force-feedback[J]. Computer methods and programs in biomedicine, 2006, 84(1): 11-18 A. Maciel, T. Halic, Z. Lu, et al. Using the PhysX engine for Physics-based Virtual Surgery with Force Feedback[J]. International Journal of Medical Robotics & Computer Assisted Surgery, 2010, 5(3): 341-353 N. Suzuki, A. Hattori, S. Ieiri, et al. VR Training System for Endoscopic Surgery Robot[J]. Chinese Journal of Clinicians, 2011, 7264(17): 117-129 C. Tercero, Z. Najdovski, S. Ikeda, et al. Haptic feedback in endovascular tele-surgery simulation through vasculature phantom morphology changes[C]. World Haptics. 2013:359-364 Silva A J, Ramirez O A D, Vega V P, et al. Phantom omni haptic device: Kinematic and manipulability[C]//2009 Electronics, Robotics and Automotive Mechanics Conference (CERMA). IEEE, 2009: 193-198 Kuber R, Zhu S, Arber Y, et al. Augmenting the non-visual web browsing process using the geomagic touch haptic device[J]. ACM SIGACCESS Accessibility and Computing, 2014 (109): 4-10 Reinhart C, Breton P F. Experimental validation of Autodesk® 3ds Max® Design 2009 and DAYSIM 3.0[J]. Leukos, 2009, 6(1): 7-35. Itkowitz B, Handley J, Zhu W. The OpenHaptics/spl trade/toolkit: a library for adding 3D Touch/spl trade/navigation and haptics to graphics applications[C]//First Joint Eurohaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems. World Haptics Conference. IEEE, 2005: 590-591. Roth D, Ismail F, Bedi S. Mechanistic modelling of the milling process using an adaptive depth buffer[J]. Computer-Aided Design, 2003, 35(14): 12871303. Kadlecek P. Overview of current developments in haptic APIs[J]. Proceedings of CESCG, 2011, 2011. Avila R S, Sobierajski L M. A haptic interaction method for volume visualization[C]//Proceedings of Seventh Annual IEEE Visualization'96. IEEE, 1996: 197-204. T. Y. Fang, P. C. Wang, C. H. Liu, et al. Evaluation of a haptics-based virtual reality temporal bone simulator for anatomy and surgery training[J]. Computer methods and programs in biomedicine, 2014, 113(2): 674-681
Yushan.Tang received the Bachelor’s degree in Automation Engineering from the University of Electronic Science and Technology of China in 2019. She is currently pursuing the master’s degree in control science and technology at University of Electronic Science and Technology of China.
> REPLACE THIS LINE WITH YOUR PAPER IDENTIFICATION NUMBER (DOUBLE-CLICK HERE TO EDIT) <
19
Shan.Liu received the Bachelor’s degree in Mechanical Automation from Hubei University of Technology in 2001.She received the Master’s degree in Systems Engineering from Huazhong University of Science and Technology in 2004. She received the Doctor’s degree in Artificial Intelligence from Huazhong University of Science and Technology in 2008. From 2006 to 2007, she is a visiting scholar in the Department of BioDesign from Arizona state University in USA. From 2013 to 2014, she is a visiting scholar in Louisiana State University and A & M College. She is now a Lecturer of the University of Electronic Science and Technology of China in the School of Automation Engineering since 2008. Dr.Liu is now a member of IEEE.
Yaru.Deng received the Bachelor’s degree in Automation Engineering from the University of Electronic Science and Technology of China in 2019. She is currently pursuing the master’s degree in control science and technology at University of Electronic Science and Technology of China.
Lirong.Yin received the Bachelor’s degree of GIS from University of Iowa in 2019. She is currently pursuing the Master’s degree in the Department of Geography and Anthropology at Louisiana state university.
Wenfeng.Zheng received the Doctor’s degree in Earth Exploration and Information Technology from Chengdu University of Technology in 2008. He is now working as a associate professor in the School of Automation Engineering at University of Electronic Science and Technology of China since 2008. he was a visiting scholar in the Department of Environmental Sciences at Louisiana state university from 2011 to 2013. Dr. Zheng is a member of Association for Computing Machinery since 2009, a member of IEEE since 2009. He is a member of America Association Geographer since 2016, and a member of American Geophysical Union since 2018. Dr.Zheng also owns a membership of China Association of Inventions since 2018 .