- Email: [email protected]
Perception-aware packet-loss resilient compression for networked haptic systems
Computer Communications 36 (2013) 1621–1628
Contents lists available at SciVerse ScienceDirect
Computer Communications journal homepage: www.elsevier.com/locate/comcom
Perception-aware packet-loss resilient compression for networked haptic systems Jalal Awed, Imad H. Elhajj ⇑, Ali Chehab, Ayman Kayssi American University of Beirut, Electrical and Computer Engineering, Beirut, Lebanon
a r t i c l e
i n f o
Article history: Available online 9 July 2013 Keywords: Haptic compression Haptic communication Perceptual coding Telepresence Packet-loss resilience
a b s t r a c t Haptic systems are increasingly being used in various applications such as virtual training, remote presence, and telesurgery. Human perception characteristics play a major role in the design of perceptual compression methods for haptic systems. However, the performance of these methods is jeopardized when packet loss occurs in the network. This study presents a packet loss resilient perceptual compression method, the modified prediction deadband method, as a novel enhancement on the linear prediction deadband method. The proposed method shows a significant improvement in user experience compared to other previously published methods, without a considerable loss in the compression ratio. Moreover, the method presents many advantages over other error-resilient compression methods, since it has a low footprint on the computational cost and on the memory consumption of the system. In addition, it does not require any previous knowledge or statistics regarding the state of the network. Ó 2013 Elsevier B.V. All rights reserved.
1. Introduction Haptic systems found their way to the Internet as more applications offer users online haptic feedback as part of the multimedia streams along with audio, video, and others. However, these systems generate a high packet rate as they require a high sampling frequency (1 kHz) in order to ensure stability and user immersion [1]. Moreover, like any other application using the Internet, haptic systems are faced with delay, jitter, and packet loss, which impair the performance of these systems, and alter the user experience. The implementation of efficient and robust compression methods in haptic systems can help reduce the effect of these network limitations. State of the art compression methods rely on the limitations of human perception of force variations. As such, the deadband approach introduced in [2,3] applies Weber’s law to achieve a reduction in packet rate by only sending force samples with a perceivable difference (found to be higher than 10%) from the last sample sent or predicted force. Different flavors of the deadband approach were also proposed, combining prediction techniques and stochastic methods, in order to increase the robustness of the compression method against packet loss in the network [4,5]. The objective of this work is to present a modified prediction deadband compression method to improve the robustness and immersion of the haptic system in case of packet loss. This method does not introduce significant computational or memory overhead ⇑ Corresponding author. Tel.: +961 70184884. E-mail addresses: [email protected] (J. Awed), [email protected] (I.H. Elhajj), [email protected] (A. Chehab), [email protected] (A. Kayssi). 0140-3664/$ - see front matter Ó 2013 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.comcom.2013.06.006
and does not require previous knowledge of the state of the network nor of the packet loss statistics. In order to assess the advantages and limitations of the proposed method, its performance was compared to that of different perceptual compression methods in a network prone to packet loss. The remainder of this paper is organized as follows: the next section discusses the previous work conducted in compression methods. Section 3 introduces the compression method proposed to overcome packet loss in the network. Section 4 describes the experimental setup used to test the performance of the proposed perception-based compression method, compares it to other methods discussed in the literature, and analyzes the obtained results. Finally, Section 5 presents the conclusion and the possible future work.
2. Previous work Finding an effective and efficient compression method became an important research subject since haptic applications found their way into industry [6]; in this regard, different approaches were proposed, some of them being based on statistical compression methods tailored to the requirements and specifications of haptic systems. Others explore the limitations of the haptic system’s components – perception limitations of humans and technical limitations of machines – to develop lossy compression methods. Prediction models were also devised in compression methods. And combinations of two or more of these approaches were also assessed in different studies.
1622
J. Awed et al. / Computer Communications 36 (2013) 1621–1628
Authors in [7–9] proposed the use of sampling and quantization techniques, and entropy encoders such as DPCM, Adaptive DPCM and Huffman encoding to compress haptic signals exchanged between the teleoperator and the human interface. These methods have been implemented in a general framework that is tolerant to network delays in haptic communication systems; a modular scheme has been proposed for this framework, including compression modules, prediction modules, and coding modules. However, these approaches do not address the high packet rates generated and exchanged in a haptic networked control system. Prediction models were also used to compress haptic data. In [10,11] a lossy method based on passive interpolative and extrapolative compression algorithms for haptic information is proposed. The extrapolative algorithm passively forecasts the original signal, therefore achieving a reduction of 89% in the network traffic. This method is also said to be perceptually transparent to users, according to the experimental results. As we mentioned in the introduction, haptic systems generate – when working over a network such as the Internet – a high packet rate as they require a high sampling frequency (1 kHz) in order to ensure stability and user immersion [1]. Hinterseer et al. were the first to target the issue of reducing the packet rate, as they introduced in [2] a deadband-based compression method that combines the deadband concept used in networked control systems [12] with Weber’s law mentioned earlier. The deadband approach is a technique used in control systems which exploits the technical limitations and system response (sensors and actuators precision, etc.) to reduce network traffic. In other words, any data that would not induce a difference in the system behavior is neither processed nor sent over the network. This method achieves a reduction in the packet rate by dropping any haptic signals unperceivable by humans and only transmitting data to the network when the sampled data value is above a certain threshold referred to as the Just Noticeable Difference (JND). This threshold is determined using psychophysical experiments [22,23]. Equation (1) illustrates the implementation of the deadband method, where force1 is the last value sent, force2 is the current force value, and DBF is the constant deadband factor. The value of force2 is sent only if it satisfies this inequality:
force2 P force1 DBF force1
ð1Þ
As for the reconstruction of the haptic signal on the receiver side, a modified version of the ‘‘hold last sample’’ approach is proposed in order to preserve the passivity of the system and hence its stability. The process of compressing and reconstructing haptic signals is illustrated in Fig. 1. Only samples with filled circles are sent,
Fig. 1. Principle of deadband data reduction.
and the signal is reconstructed from these samples on the receiver side. The perception thresholds (boundaries of gray zones) are a function of the haptic stimulus intensity. This compression technique has proven to be efficient in terms of bandwidth consumption and packet rate, without altering the user experience or introducing high computational cost. In [13,14] the aforementioned authors continued investigating the effect of using this compression method on stability. They studied the reconstruction method in case of a constant deadband, a relative deadband, and position drift to ensure system stability. The authors of [3] extend the concept of the deadband approach from one dimension (1 Degree of Freedom, DoF, teleoperator) to three dimensions (3 DoF teleoperator). They proposed a deadzone approach where forces are treated as vectors in space, and the deadzone is a spherical shape around the tip of the vector; any force changes falling within this deadzone will be neglected. This method has proven to be more efficient than separately applying the deadband approach on each of the force components. To determine the radius of this deadzone, the authors implemented an experiment where users are required to touch a virtual sphere with a virtual tip using the haptic device. During the movement, force magnitudes are changed and users had to report when they felt this change. To further increase the savings in the data transmitted, studies began to introduce prediction models to the control systems of the haptic elements. In [15], a linear prediction model for the force signal is implemented on both sides of the system. On the operator side a force predictor is used to estimate the current force value from previous values sent from the teleoperator side. On the teleoperator side, the same predictor is running in parallel and force values are sent over the network if their current actual value differs from the predicted value by a JND (Fig. 2). In Fig. 2, only samples illustrated in large filled circles are transmitted and used for updating the predictor [16]. Results obtained showed a significant decrease in the generated packet rate without deteriorating the user’s haptic experience. In [17], the combined prediction–compression approach was also used to compensate for the effects of the network delay between the operator and teleoperator. Three mechanisms were presented to achieve this purpose: a motion prediction to partially compensate for network delays on the teleoperator side, a force prediction module to reduce dependency on delayed force signals, and a haptic data compression to reduce data packets transmitted and therefore the required bandwidth. Double exponential smoothing was implemented in the motion module. As for the force prediction module, it is implemented as an online learning model where the operator receives a contact force computed from previous and delayed force and position measurements, whenever the real time force value is delayed. Finally, haptic data compression is achieved through a combined deadband and prediction approach. This approach was found to stabilize the control loops while improving the performance of the system as well as decreasing the packets sent over the network. In addition, no influence on the operator immersion was observed. Authors in [18] have also reported further reduction in packet rate when using fast Kalman filtering on the input signal before applying the deadband method along with the linear predictor. The performance of the perceptual compression methods over communication networks depends greatly on the successful reconstruction of the transmitted packets; therefore, the quality of the communication network plays an important role in the haptic systems using these compression methods. Packet loss is a primary factor in the deterioration of performance since in a compressed stream each packet contains a high density of information. Mitigating packet loss is complicated by the fact that these losses are bursty and hard to predict [19]. It is worth noting that packet loss is
1623
J. Awed et al. / Computer Communications 36 (2013) 1621–1628
Fig. 2. Illustration of combining deadband approach with linear prediction model.
meant to account for packets that are lost due to routing issues and packets that are delayed beyond real-time use. Brandi et al. [4] conducted a study to assess the effect of packet loss in the communication channel on the performance of the deadband approach with linear prediction, and they identified three main artifacts when packet loss occurs in the network: the Bouncing effect, the Inverted Force effect (glue effect) and the Roughness effect. Furthermore, the authors proposed an error resilient perceptual haptic data reduction scheme to mitigate the effects of packet loss. The scheme implements a complex probabilistic method combined with a binary tree to determine when to send additional samples. Several update trigger criteria were proposed such as the Expected Deviation and the Sum of Probabilities. This scheme induces a high computational cost; therefore, a variation was proposed in [5] to reduce the complexity of this method by adding the calculated gradient of the linear prediction and sending it along with the update packet. This addition would lead to a linear growth of the tree instead of the exponential growth as in the previous method. However, these methods still suffer from several limitations: All the update trigger criteria depend greatly on predicting the probability of packet loss, which is difficult to determine in a real network, where the packet losses are bursty and hard to predict [19]. In case of losing the retransmitted updates, the method will suffer from the same artifacts mentioned above. In addition, these methods require a high computational cost and considerable memory consumption due to the tree construction.
The difference between the current force magnitude and the recent update is larger than the deadband value (similar to the original deadband method). The update packet will include the current force value along with the gradient of the line used to predict the next samples. This gradient is calculated from averaging the slope between the current update sample and the last updated sample as stated in Eq. (2):
~ gj ¼
~ f j ~ f j1 tj tj1
ð2Þ
f j is the current update force, ~ f j1 is the last update sent, t j is where ~ f , is the time of the update ~ f . the time of the update ~ j
j1
At the receiver side the predictor will predict the next samples based on the gradient and the force sample received up to the point where the difference between the predicted force and the last received force sample is equal to the deadband value, then the receiver will hold the last force value predicted and wait for the next update sample, as described in Eq. (3), where tk is the time where f j k: k~ g j :ðti tj Þk reaches DBF k~
( ~ vi ¼
~ fj g j ðti tj Þ þ ~ ~ f g ðt t Þ þ ~ j
k
j
j
when ti 6 t k when ti > tk
Fig. 3 illustrates the reconstruction procedure on the receiver side. The filled dots on the left are the samples that are sent over the network. However, one of these samples is lost (the dotted line on the right side), and the reconstruction rule is applied.
In the next section we introduce an alternative perceptual based compression method to improve the robustness and immersion of the haptic system in case of packet loss and assess its performance along with the original deadband method, the method with linear prediction, and the low-complexity method [5] mentioned earlier in this section. 3. Proposed modified prediction deadband method The proposed method is a combination of the original deadband method and of the linear prediction approach. On the sender side, updates will be issued in any of these cases: The difference between the current force magnitude and the predicted force on the receiver is larger than the deadband value (similar to the deadband with linear prediction method).
ð3Þ
Fig. 3. Signal reconstruction on the receiver side in case of packet loss.
1624
J. Awed et al. / Computer Communications 36 (2013) 1621–1628
This method preserves the benefits of prediction techniques on the reconstruction of the haptic signal, while minimizing the errors produced by packet loss, showing a better resilience to this loss compared to the original deadband and linear prediction alone. In addition, this method provides a major advantage over other error resilient techniques for its low computational cost, low complexity and its low memory requirement. It is important to note that in case of the implementation of the original deadband method, packet loss would affect the stiffness of the virtual object. In a real environment this could lead to disastrous effects such as the penetration of the surface due to wrong perception of its stiffness, since an increase in the force applied by the user will not result in a corresponding increase in the force feedback or a resistance from the object due to the packet loss in the communication network. Fig. 4 illustrates one of the main advantages of the proposed method over the linear prediction approach. The large gray circles represent the lost samples, the black dots represent the sent and received samples while the white dots represent skipped or predicted samples. As clearly shown in part (c) of the figure, a packet loss is leading to a considerable drift of the reconstructed signal from the original signal, leading to deterioration in user perception. Part (d) of the figure shows how the proposed method limits this drift and bounds it to a value close to the deadband, therefore making the packet loss more transparent to the user. As shown in Fig. 4, the update trigger criteria in the proposed method help reduce the average error between the original signal and the rendered signal. In case of the original deadband method, f ~ f k and when E P DBF an upthe instantaneous error is E ¼ k~ i
j
f i ~ f j k P DBF j a new update is sent. Hence, the maximum erE0 ¼ k~ ror that can occur before triggering a new update is E < DBFp. The error occurring at trigger time is not necessarily the maximum error as in the previous cases. The error, E, occurring at trigger time satisfies the condition 0 6 E K DBF p depending on the criterion that triggered the update. Therefore, the error when triggering a new update is – on average – smaller than that of the previous methods. In this case, losing a packet has a smaller effect on the user experience as the density of information is reduced at the cost of minor reduction in compression.
3.1. Experimental setup The experimental setup consists of a three-dimensional virtual environment containing a static toroidal object provided by the CHAI3D library [20]. The participant can explore the remote environment via the SensAble PHANToM Omni haptic device with a 1 kHz sampling rate of the 3DoFs. In order to isolate the effect of packet loss on haptic systems, the network was emulated on the machine rendering the virtual environment, with one way controllable packet loss rates applied only to the force-feedback channel. The packet loss function was implemented using a Bernoulli distribution function, which allows the generation of random Booleans with a desired probability [19]. The experimental setup is illustrated in Fig. 5. Four force compression methods are assessed under different packet loss scenarios:
j
f j k; ~ fj f i is the original force and ~ date is sent, where DBF j ¼ DBF k~ is the current force. Hence, the maximum error between the original signal and the rendered signal, which occurs when triggering a new update, is E ffi DBF j . As for the case of the deadband method with linear prediction, the error is f ~ f k and when E P DBF p an update is sent, where DBF p ¼ E ¼ k~ i
1. Original deadband described in [3] with a 10% deadband factor. 2. Deadband compression method with the linear prediction as used in [5]. 3. Low-complexity error-resilient compression method using the ‘‘Expected Deviation’’ criterion described in [5] with the acknowledgment delayed by 30 m s. 4. Modified prediction deadband method introduced in this paper with a 10% deadband factor.
j
f p k, ~ f p is the current predicted force. Hence, the maximum DBF k~ error, which occurs when triggering a new update, is E ffi DBFp. However, in the case of using the modified prediction deadband f i ~ f p k and when E P DBF or method, the error is E ¼ k~
These methods are assessed under 10%, 20%, 40%, and 60% packet loss rates.
p
F
F
t
t
c. Reconstructed signal using linear prediction when a packet is lost
a. Original signal
t b. Reconstructed signal with no packet loss
X
X
t
d. Reconstructed signal using proposed modified linear prediction when a packet is lost
Fig. 4. Illustration of a signal reconstruction in case of packet loss using the linear prediction and the proposed modified prediction methods.
J. Awed et al. / Computer Communications 36 (2013) 1621–1628
Fig. 5. The experimental setup.
The experiment begins with a practice phase where each subject is asked to actively explore the virtual environment shown in Fig. 5. In the practice phase no compression and no packet loss is introduced on the force-feedback channel. In addition to freely exploring the environment, the subject is asked to specifically perform the following moves: Move the Phantom stylus (rendered as a small red sphere) on the surface of the torus. Tap on the torus. Statically press on the torus surface. Tap while moving on the surface. The remainder of the experiment is divided into four blocks; each block represents a packet loss rates and contains four 50-second sessions, each session using one of the four compression methods mentioned earlier applied to the signal. The blocks are rendered randomly to each participant and so is the order of the compression methods within each block.The participant is asked to perform the aforementioned moves in each session in order to evaluate the method used and to rate the session based on the rating scale shown in Table 1. The reference session used in the practice phase could be freely accessed at any time during the experiment, and is also rendered at the beginning of each block to help the participants recall the feeling and therefore assist them in rating the subsequent sessions. In addition to the subjective rating of the participant, the data percentage compression ratio (C.R.) is recorded for each method in each session during the experiment, according to:
C:R: ¼ 1
number of samples sent 100 number of samples generated
ð4Þ
4. Results and analysis Before showing the results for subjective real-time testing, we present results from an objective offline test. 4.1. Offline experiment An objective offline test was performed where the reconstructed signal is compared with the original signal. Therefore, an interval of force samples was captured and recorded when interacting with the virtual environment for around 30 s. The recorded
Table 1 Subjective rating scale. Description Excellent Good Fair Poor Very
Rating (%) no problems minor problems some problems several problems bad too many problems
100 75 50 25 0
1625
force samples varied around 1.35 N (as an average) with a standard deviation of 1.03 N. These samples were then fed into the compression algorithms, the communication channel module with different packet loss rates, and finally the reconstruction algorithms. The RMSE (Root Mean Square Error) between the original and the reconstructed signal were calculated. The calculated RMSEs are shown in Table 2. Results show that the proposed compression method has the smallest RMSE for all packet loss rates. For example, at 20% packet loss, the error obtained for the proposed method is approximately equal to the error obtained for the other methods when no packet loss was applied. These results are confirmed by the subjective rating of the participants for the proposed method, even with the presence of a 20% packet loss. The proposed method showed a lower percentage compression ratio 87% compared to the original deadband and the deadband with the linear prediction methods which have 89% and 94% compression, respectively. As for the low complexity error resilient method it showed 92% compression ratio at no packet loss, and decreased as the packet loss increased (91%, 90% and 89% compression ratios at 10%, 20% and 40% packet loss respectively). However, the lower compression ratio can be considered as an acceptable compromise to obtain a significantly better user experience and better RMSE. It is worth noting that since the compression ratio is influenced by the user’s movement and her/his reaction to changes in perception, the compression ratios obtained from the offline test only give an idea of the expected compression ratios when a user is actually controlling the haptic device. 4.2. Subjective experimental results Fifteen subjects participated in this experiment (Dominant Hand: 1 left, 14 right. Gender: 3 Females, 12 Males). The subjects were asked to rate the performance of the different compression methods as described earlier. Table 3 shows the average subjective ratings of the participants in percentage and their standard deviation (STDEV), along with the percentage compression ratio (C.R.) of the tested compression methods under different packet loss rates. Figs. 6 and 7 provide a graphical representation of the results shown in Table 3, where the proposed method (M4) clearly outperforms the other published methods in terms of human perception. In addition, Fig. 6 demonstrates the amount of variation in performance for each method with the increase of the packet loss. The proposed method shows a small drop in performance, while the linear prediction deadband showed a drastic drop in performance when packet loss increases. From Table 3 and Figs. 6 and 7 the following observations can be made: The original deadband method (M1) shows a fair performance (around 65% rating) with low packet loss rate. However, as the packet loss increases, the performance deteriorates slowly remaining in the fair range. The percentage compression ratio is stable around 90%. The deadband with linear prediction method (M2) shows the worst performance among the four compression methods. As packet loss increases, participants were experiencing unstable behavior and highly distorted telepresence. The low complexity error resilient method (M3), originally designed to overcome the effects of packet loss of method (M2), shows a noticeable amelioration in performance compared to M2. However, as packet loss increases, its performance decreases, but remained in the ‘‘fair’’ zone. On the other hand, the percentage compression ratio of this method decreases from approximately 90% to 80% as the packet loss increases. A significant drawback of this method is that it assumes a known packet loss rate, and implements the countermeasure based
1626
J. Awed et al. / Computer Communications 36 (2013) 1621–1628
Table 2 Root Mean Square Error and percentage compression ratio of the tested compression methods. Compression method
Packet loss rate 0%
M1. M2. M3. M4.
Original deadband (DB) DB with linear prediction Low complexity error resilient Proposed modified prediction DB
10%
20%
40%
RMSE
C.R. (%)
RMSE
C.R. (%)
RMSE
C.R. (%)
RMSE
C.R. (%)
0.089 0.086 0.085 0.063
89 94 92 87
0.138 0.28 0.281 0.072
89 94 91 87
0.165 0.396 0.344 0.094
89 94 90 87
0.25 0.749 0.318 0.154
89 94 89 87
Table 3 Subjective ratings %, their standard deviation and the percentage compression ratio (C.R.) of the compression methods under various packet loss rates. Compression methods
Packet loss rate 10%
20%
40%
60%
S. STDEV Compression S. STDEV Compression S. STDEV Compression S. STDEV Compression Rating (%) ratio (%) Rating (%) ratio (%) Rating (%) ratio (%) Rating (%) ratio (%) M1. Original deadband (DB) M2. DB with linear prediction M3. Low complexity error resilient M4. Proposed modified prediction DB
65.3
19.1
91.3
68
16
90.6
58
20.2
90.8
54.6
17.9
90.8
67.3
17.9
92.6
48.3
24.9
92.5
25.3
26.6
92.7
14
23.9
92.8
72.3
15.2
90.6
71
11.8
89.5
62.6
18.8
86.3
53.3
20.2
82.1
83.8
7.79
88.5
81.6
11.1
88.4
72
17.4
88.4
54.8
17.8
87.6
100 95 90 85 80 75 70
Subjective Rating %
65 60 55 50 45 40 35 30 25 20
1. Original Deadband (DB)
15
2. DB with Linear Predicon
10
3. Low Complexity Error Resillient 4. Proposed Modified predicon DB
5 0 10
20
40
60
Packet Loss Rate % Fig. 6. Subjective ratings of the compression methods under different packet loss rates.
on this ratio. Therefore, any inaccuracy in this ratio would lead to a further deterioration in the performance (inherited from method M2).
The modified prediction deadband method (M4) shows a clear amelioration of the user experience, even when the packet loss increases in the network. A decrease in the percentage
1627
J. Awed et al. / Computer Communications 36 (2013) 1621–1628 95 94 93
M2
92
M1
91
M3
Compression ratio %
90 89
M4
88 87 86 85 84 1. Original Deadband (DB)
83
2. DB with Linear Predicon 82 3. Low Complexity Error Resillient 81
4. Proposed Modified predicon DB
80 10
20
40
60
Packet Loss Rate % Fig. 7. The percentage compression ratios (C.R.) of the compression methods under various packet loss rates.
compression ratio only around 2.5% was observed in comparison to method (M1), but this decrease is highly compensated for by more than 10% increase in the user ratings compared to methods (M1), (M2) and (M3). In addition, Internet statistics [21] show that the global packet loss rate over a period of 30 days was around 20%. Based on this packet loss rate, the proposed modified prediction deadband method (M4) provides a significant improvement in perception (20%, 69% and 15% over the previously published compression methods M1, M2 and M3, respectively) at the cost of a slight decrease in percentage compression ratio (2%, 4% and 1% compared to methods M1, M2 and M3, respectively). Moreover, the subjective quality ratings of the proposed method remained above the level considered ‘‘good’’ by the participants (over 75%). It is important to note that the proposed method does not introduce additional processing overhead to the haptic system when compared to the other methods. In the proposed modified prediction deadband method (M4), on the sender side, the system first predicts the rendered value based on the gradient and the previous sent force (1 subtraction, 1 multiplication, and 1 addition). Then the method executes an ‘‘if condition’’ with 4 absolute value operations, 2 subtractions, and 2 multiplications. If the sample should be sent, the gradient is calculated using 2 subtractions and 1 division; else, no computations are carried out. On the receiver side, the predicted value is calculated using 1 subtraction, 1 multiplication, and 1 addition. Then the method executes an ‘‘if condition’’ with 2 absolute value operations, 1 subtraction, and 1 multiplication, and renders the appropriate force value. The low complexity error resilient method (M3), on the sender side, first calculates the predicted value (1 subtraction, 1 multiplication, and 1 addition). Then the method executes an ‘‘if condition’’ with 2 absolute value operations, 1 subtraction, and 1
multiplication. If the sample should be sent, the gradient is calculated using 2 subtractions and 1 division, and the tree is updated (memory access). Else when the sample is not sent (which is the case most of the time), the ‘‘update criterion’’ is calculated using a loop of (m) iterations, where m depends on the receipt of an acknowledgment or the maximum number before resetting the tree. On the receiver side, the predicted value is calculated using 1 subtraction, 1 multiplication, and 1 addition. The original deadband method (M1), on the sender side, consists of an ‘‘if condition’’ with 2 absolute value operations, 1 subtraction, and 1 multiplication. On the receiver side, the method renders the update once it is received with no computations and keeps rendering the same value until the next update. The deadband with linear prediction method (M2), on the sender side, calculates the predicted value (1 subtraction, 1 multiplication, and 1 addition). Then the method executes an ‘‘if condition’’ with 2 absolute value operations, 1 subtraction, and 1 multiplication. If the sample should be sent, the gradient is calculated using 2 subtractions and 1 division operation. On the receiver side, the predicted value is calculated using 1 subtraction, 1 multiplication, and 1 addition. Moreover, since there is little dependency on previous samples sent as in previous methods, no memory overhead is introduced. Finally, this method does not assume knowledge of packet loss rate, and therefore does not require any statistics or information regarding the state of the network, as it is the case in the published method (M3). 5. Conclusion and future work This work addressed transparent communication in haptic systems. The Internet is now an important component of these systems; it is crucial to apply an efficient compression method that respects the real-time constraints of the haptic system while being
1628
J. Awed et al. / Computer Communications 36 (2013) 1621–1628
resistant to packet loss. The modified prediction deadband method was proposed, that modifies the linear prediction deadband method, to mitigate the effects of packet loss in networked haptic systems. Experimental results showed a noticeable improvement in performance compared to other compression methods previously proposed. A major improvement of more than 10% was observed, while only around 2.5% decrease in percentage compression ratio was sacrificed compared to the deadband method. Finally, the proposed method has a negligible computational cost and requires no previous knowledge or assumptions about the status of the network or the packet loss rate. As future work, the implementation of the modified prediction deadband method could be extended to the forward channel on the operator side in addition to its implementation in the feedback channel on the teleperator side where the change in performance of the overall system could be assessed. Furthermore, future work would include the assessment of the performance of the proposed method in delayed networks. Acknowledgments Research funded by the American University of Beirut University Research Board and the Lebanese National Council of Scientific Research.
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
References [17] [1] E. Steinbach, S. Hirche, M.O. Ernst, F. Brandi, R. Chaudhari, J. Kammerl, I. Vittorias, Haptic communications, in: Proceedings of the IEEE, Frontiers of Audiovisual Communications, 2011. [2] P. Hinterseer, E. Steinbach, S. Hirche, M. Buss, A novel, psychophysically motivated transmission approach for haptic data streams in telepresence and teleaction systems, in: Proceedings of International Conference Acoustics, Speech, and Signal Processing, Philadelphia, PA, vol. 2, March 2005, pp. ii/ 1097–ii/1100. [3] P. Hinterseer, E. Steinbach, A psychophysically motivated compression approach for 3D haptic data, in: Proceedings of the IEEE Haptics Symposium, Alexandria, VA, USA, March 2006, pp. 35–41. [4] F. Brandi, J. Kammerl, E. Steinbach, Error-resilient perceptual coding for networked haptic interaction, in: Proceedings of the ACM Multimedia, Firenze, Italy, October 2010, pp. 351–360. [5] F. Brandi, E. Steinbach, Low-complexity error-resilient data reduction approach for networked haptic sessions, in: Proceedings of IEEE International Symposium on Haptic Audio-Visual Environments and Games (HAVE) Nanchang, China, October 2011. [6] V. Nitsch, B. Fäber, L. Geiger, P. Hinterseer, E. Steinbach, An experimental study of lossy compression in a real telepresence and teleaction system, in: Proceedings of IEEE International Workshop on Haptic Audio Visual
[18]
[19] [20]
[21] [22]
[23]
Environments and their Applications, Ottawa, Canada, October 2008, pp. 75– 80. C. Shahabi, M.R. Kolahdouzan, G. Barish, R. Zimmermann, D. Yao, K. Fu, L. Zhang, Alternative techniques for the efficient acquisition of haptic data, in: ACM SIGMETRICS/Performance 2001, June 2001. C. Shahabi, A. Ortega, M.R. Kolahdouzan A comparison of different haptic compression techniques, in: IEEE International Conference on Multimedia and Expo, 2002, pp. 657–660. K. Hikichi, H. Morino, I. Fukuda, S. Matsumoto, Y. Yasuda, I. Arimoto, M. Iijima, K. Sezaki, Architecture of haptics communication system for adaptation to network environments, in: Proceedings of IEEE International Conference Multimedia Expo, Tokyo, Japan, August 2001, pp. 744–747. M. Kuschel, P. Kremer, M. Buss, Passive haptic data-compression methods with perceptual coding for bilateral presence systems, IEEE Trans, Syst. Man Cybern. A Syst. Humans 39 (6) (2009) 1142–1151. P. Kremer, M. Kuschel, C. Preusche, M. Buss, G. Hirzinger, Passive event-based extrapolation for lossy haptic data compression in bilateral presence systems, in: Proceedings of IEEE International Conference on Robotics and Automation, Kobe, Japan, May 2009. J.R. Moyne, P.G. Otanez, D.M. Tilbury, Using deadbands to reduce communication in networked control systems, in: Proceedings of the American Control Conference, Anchorage, Alaska, May 2002. S. Hirche, P. Hinterseer, E. Steinbach, M. Buss, Network traffic reduction in haptic telepresence systems by deadband control, in: Proceedings of the 16th IFAC World Congress. Prague, Czech Republic, 2005. S. Hirche, P. Hinterseer, E. Steinbach, M. Buss, Towards deadband control in networked teleoperation systems, in: Proceedings of IFAC World Congress, International Federation of Autom, Control, Prague, Czech Republic, 2005. P. Hinterseer, E. Steinbach, S. Chaudhuri, Model-based data compression for 3D virtual haptic teleinteraction, in: IEEE International Conference on Consumer Electronics, ICCE 2006, January 2006. E. Steinbach, S. Hirche, J. Kammerl, I. Vittorias, R. Chaudhari, Haptic data compression and communication, IEEE Signal Process. Mag. 28 (1) (2011) 87– 96. S. Clarke, G. Schillhuber, M.F. Zaeh, H. Ulbrich, Telepresence across delayed networks: a combined prediction and compression approach, in: IEEE International Workshop on Haptic Audio Visual Environments and their Applications, November 2006, pp. 23–24. P. Hinterseer, E. Steinbach, S. Chaudhuri, Perception-based compression of haptic data streams using Kalman filters, in: Proceedings of IEEE International Conference Acoust, Speech, Signal Process, 2006. (accessed October25, 2012). F. Conti, F. Barbagli, D. Morris, C. Sewell, CHAI 3D: an open-source library for the rapid development of haptic scenes, in: Proceedings of IEEE World Haptics, Pisa, Italy, Mar 2005. The Internet Traffic Report
Contents lists available at SciVerse ScienceDirect
Computer Communications journal homepage: www.elsevier.com/locate/comcom
Perception-aware packet-loss resilient compression for networked haptic systems Jalal Awed, Imad H. Elhajj ⇑, Ali Chehab, Ayman Kayssi American University of Beirut, Electrical and Computer Engineering, Beirut, Lebanon
a r t i c l e
i n f o
Article history: Available online 9 July 2013 Keywords: Haptic compression Haptic communication Perceptual coding Telepresence Packet-loss resilience
a b s t r a c t Haptic systems are increasingly being used in various applications such as virtual training, remote presence, and telesurgery. Human perception characteristics play a major role in the design of perceptual compression methods for haptic systems. However, the performance of these methods is jeopardized when packet loss occurs in the network. This study presents a packet loss resilient perceptual compression method, the modified prediction deadband method, as a novel enhancement on the linear prediction deadband method. The proposed method shows a significant improvement in user experience compared to other previously published methods, without a considerable loss in the compression ratio. Moreover, the method presents many advantages over other error-resilient compression methods, since it has a low footprint on the computational cost and on the memory consumption of the system. In addition, it does not require any previous knowledge or statistics regarding the state of the network. Ó 2013 Elsevier B.V. All rights reserved.
1. Introduction Haptic systems found their way to the Internet as more applications offer users online haptic feedback as part of the multimedia streams along with audio, video, and others. However, these systems generate a high packet rate as they require a high sampling frequency (1 kHz) in order to ensure stability and user immersion [1]. Moreover, like any other application using the Internet, haptic systems are faced with delay, jitter, and packet loss, which impair the performance of these systems, and alter the user experience. The implementation of efficient and robust compression methods in haptic systems can help reduce the effect of these network limitations. State of the art compression methods rely on the limitations of human perception of force variations. As such, the deadband approach introduced in [2,3] applies Weber’s law to achieve a reduction in packet rate by only sending force samples with a perceivable difference (found to be higher than 10%) from the last sample sent or predicted force. Different flavors of the deadband approach were also proposed, combining prediction techniques and stochastic methods, in order to increase the robustness of the compression method against packet loss in the network [4,5]. The objective of this work is to present a modified prediction deadband compression method to improve the robustness and immersion of the haptic system in case of packet loss. This method does not introduce significant computational or memory overhead ⇑ Corresponding author. Tel.: +961 70184884. E-mail addresses: [email protected] (J. Awed), [email protected] (I.H. Elhajj), [email protected] (A. Chehab), [email protected] (A. Kayssi). 0140-3664/$ - see front matter Ó 2013 Elsevier B.V. All rights reserved. http://dx.doi.org/10.1016/j.comcom.2013.06.006
and does not require previous knowledge of the state of the network nor of the packet loss statistics. In order to assess the advantages and limitations of the proposed method, its performance was compared to that of different perceptual compression methods in a network prone to packet loss. The remainder of this paper is organized as follows: the next section discusses the previous work conducted in compression methods. Section 3 introduces the compression method proposed to overcome packet loss in the network. Section 4 describes the experimental setup used to test the performance of the proposed perception-based compression method, compares it to other methods discussed in the literature, and analyzes the obtained results. Finally, Section 5 presents the conclusion and the possible future work.
2. Previous work Finding an effective and efficient compression method became an important research subject since haptic applications found their way into industry [6]; in this regard, different approaches were proposed, some of them being based on statistical compression methods tailored to the requirements and specifications of haptic systems. Others explore the limitations of the haptic system’s components – perception limitations of humans and technical limitations of machines – to develop lossy compression methods. Prediction models were also devised in compression methods. And combinations of two or more of these approaches were also assessed in different studies.
1622
J. Awed et al. / Computer Communications 36 (2013) 1621–1628
Authors in [7–9] proposed the use of sampling and quantization techniques, and entropy encoders such as DPCM, Adaptive DPCM and Huffman encoding to compress haptic signals exchanged between the teleoperator and the human interface. These methods have been implemented in a general framework that is tolerant to network delays in haptic communication systems; a modular scheme has been proposed for this framework, including compression modules, prediction modules, and coding modules. However, these approaches do not address the high packet rates generated and exchanged in a haptic networked control system. Prediction models were also used to compress haptic data. In [10,11] a lossy method based on passive interpolative and extrapolative compression algorithms for haptic information is proposed. The extrapolative algorithm passively forecasts the original signal, therefore achieving a reduction of 89% in the network traffic. This method is also said to be perceptually transparent to users, according to the experimental results. As we mentioned in the introduction, haptic systems generate – when working over a network such as the Internet – a high packet rate as they require a high sampling frequency (1 kHz) in order to ensure stability and user immersion [1]. Hinterseer et al. were the first to target the issue of reducing the packet rate, as they introduced in [2] a deadband-based compression method that combines the deadband concept used in networked control systems [12] with Weber’s law mentioned earlier. The deadband approach is a technique used in control systems which exploits the technical limitations and system response (sensors and actuators precision, etc.) to reduce network traffic. In other words, any data that would not induce a difference in the system behavior is neither processed nor sent over the network. This method achieves a reduction in the packet rate by dropping any haptic signals unperceivable by humans and only transmitting data to the network when the sampled data value is above a certain threshold referred to as the Just Noticeable Difference (JND). This threshold is determined using psychophysical experiments [22,23]. Equation (1) illustrates the implementation of the deadband method, where force1 is the last value sent, force2 is the current force value, and DBF is the constant deadband factor. The value of force2 is sent only if it satisfies this inequality:
force2 P force1 DBF force1
ð1Þ
As for the reconstruction of the haptic signal on the receiver side, a modified version of the ‘‘hold last sample’’ approach is proposed in order to preserve the passivity of the system and hence its stability. The process of compressing and reconstructing haptic signals is illustrated in Fig. 1. Only samples with filled circles are sent,
Fig. 1. Principle of deadband data reduction.
and the signal is reconstructed from these samples on the receiver side. The perception thresholds (boundaries of gray zones) are a function of the haptic stimulus intensity. This compression technique has proven to be efficient in terms of bandwidth consumption and packet rate, without altering the user experience or introducing high computational cost. In [13,14] the aforementioned authors continued investigating the effect of using this compression method on stability. They studied the reconstruction method in case of a constant deadband, a relative deadband, and position drift to ensure system stability. The authors of [3] extend the concept of the deadband approach from one dimension (1 Degree of Freedom, DoF, teleoperator) to three dimensions (3 DoF teleoperator). They proposed a deadzone approach where forces are treated as vectors in space, and the deadzone is a spherical shape around the tip of the vector; any force changes falling within this deadzone will be neglected. This method has proven to be more efficient than separately applying the deadband approach on each of the force components. To determine the radius of this deadzone, the authors implemented an experiment where users are required to touch a virtual sphere with a virtual tip using the haptic device. During the movement, force magnitudes are changed and users had to report when they felt this change. To further increase the savings in the data transmitted, studies began to introduce prediction models to the control systems of the haptic elements. In [15], a linear prediction model for the force signal is implemented on both sides of the system. On the operator side a force predictor is used to estimate the current force value from previous values sent from the teleoperator side. On the teleoperator side, the same predictor is running in parallel and force values are sent over the network if their current actual value differs from the predicted value by a JND (Fig. 2). In Fig. 2, only samples illustrated in large filled circles are transmitted and used for updating the predictor [16]. Results obtained showed a significant decrease in the generated packet rate without deteriorating the user’s haptic experience. In [17], the combined prediction–compression approach was also used to compensate for the effects of the network delay between the operator and teleoperator. Three mechanisms were presented to achieve this purpose: a motion prediction to partially compensate for network delays on the teleoperator side, a force prediction module to reduce dependency on delayed force signals, and a haptic data compression to reduce data packets transmitted and therefore the required bandwidth. Double exponential smoothing was implemented in the motion module. As for the force prediction module, it is implemented as an online learning model where the operator receives a contact force computed from previous and delayed force and position measurements, whenever the real time force value is delayed. Finally, haptic data compression is achieved through a combined deadband and prediction approach. This approach was found to stabilize the control loops while improving the performance of the system as well as decreasing the packets sent over the network. In addition, no influence on the operator immersion was observed. Authors in [18] have also reported further reduction in packet rate when using fast Kalman filtering on the input signal before applying the deadband method along with the linear predictor. The performance of the perceptual compression methods over communication networks depends greatly on the successful reconstruction of the transmitted packets; therefore, the quality of the communication network plays an important role in the haptic systems using these compression methods. Packet loss is a primary factor in the deterioration of performance since in a compressed stream each packet contains a high density of information. Mitigating packet loss is complicated by the fact that these losses are bursty and hard to predict [19]. It is worth noting that packet loss is
1623
J. Awed et al. / Computer Communications 36 (2013) 1621–1628
Fig. 2. Illustration of combining deadband approach with linear prediction model.
meant to account for packets that are lost due to routing issues and packets that are delayed beyond real-time use. Brandi et al. [4] conducted a study to assess the effect of packet loss in the communication channel on the performance of the deadband approach with linear prediction, and they identified three main artifacts when packet loss occurs in the network: the Bouncing effect, the Inverted Force effect (glue effect) and the Roughness effect. Furthermore, the authors proposed an error resilient perceptual haptic data reduction scheme to mitigate the effects of packet loss. The scheme implements a complex probabilistic method combined with a binary tree to determine when to send additional samples. Several update trigger criteria were proposed such as the Expected Deviation and the Sum of Probabilities. This scheme induces a high computational cost; therefore, a variation was proposed in [5] to reduce the complexity of this method by adding the calculated gradient of the linear prediction and sending it along with the update packet. This addition would lead to a linear growth of the tree instead of the exponential growth as in the previous method. However, these methods still suffer from several limitations: All the update trigger criteria depend greatly on predicting the probability of packet loss, which is difficult to determine in a real network, where the packet losses are bursty and hard to predict [19]. In case of losing the retransmitted updates, the method will suffer from the same artifacts mentioned above. In addition, these methods require a high computational cost and considerable memory consumption due to the tree construction.
The difference between the current force magnitude and the recent update is larger than the deadband value (similar to the original deadband method). The update packet will include the current force value along with the gradient of the line used to predict the next samples. This gradient is calculated from averaging the slope between the current update sample and the last updated sample as stated in Eq. (2):
~ gj ¼
~ f j ~ f j1 tj tj1
ð2Þ
f j is the current update force, ~ f j1 is the last update sent, t j is where ~ f , is the time of the update ~ f . the time of the update ~ j
j1
At the receiver side the predictor will predict the next samples based on the gradient and the force sample received up to the point where the difference between the predicted force and the last received force sample is equal to the deadband value, then the receiver will hold the last force value predicted and wait for the next update sample, as described in Eq. (3), where tk is the time where f j k: k~ g j :ðti tj Þk reaches DBF k~
( ~ vi ¼
~ fj g j ðti tj Þ þ ~ ~ f g ðt t Þ þ ~ j
k
j
j
when ti 6 t k when ti > tk
Fig. 3 illustrates the reconstruction procedure on the receiver side. The filled dots on the left are the samples that are sent over the network. However, one of these samples is lost (the dotted line on the right side), and the reconstruction rule is applied.
In the next section we introduce an alternative perceptual based compression method to improve the robustness and immersion of the haptic system in case of packet loss and assess its performance along with the original deadband method, the method with linear prediction, and the low-complexity method [5] mentioned earlier in this section. 3. Proposed modified prediction deadband method The proposed method is a combination of the original deadband method and of the linear prediction approach. On the sender side, updates will be issued in any of these cases: The difference between the current force magnitude and the predicted force on the receiver is larger than the deadband value (similar to the deadband with linear prediction method).
ð3Þ
Fig. 3. Signal reconstruction on the receiver side in case of packet loss.
1624
J. Awed et al. / Computer Communications 36 (2013) 1621–1628
This method preserves the benefits of prediction techniques on the reconstruction of the haptic signal, while minimizing the errors produced by packet loss, showing a better resilience to this loss compared to the original deadband and linear prediction alone. In addition, this method provides a major advantage over other error resilient techniques for its low computational cost, low complexity and its low memory requirement. It is important to note that in case of the implementation of the original deadband method, packet loss would affect the stiffness of the virtual object. In a real environment this could lead to disastrous effects such as the penetration of the surface due to wrong perception of its stiffness, since an increase in the force applied by the user will not result in a corresponding increase in the force feedback or a resistance from the object due to the packet loss in the communication network. Fig. 4 illustrates one of the main advantages of the proposed method over the linear prediction approach. The large gray circles represent the lost samples, the black dots represent the sent and received samples while the white dots represent skipped or predicted samples. As clearly shown in part (c) of the figure, a packet loss is leading to a considerable drift of the reconstructed signal from the original signal, leading to deterioration in user perception. Part (d) of the figure shows how the proposed method limits this drift and bounds it to a value close to the deadband, therefore making the packet loss more transparent to the user. As shown in Fig. 4, the update trigger criteria in the proposed method help reduce the average error between the original signal and the rendered signal. In case of the original deadband method, f ~ f k and when E P DBF an upthe instantaneous error is E ¼ k~ i
j
f i ~ f j k P DBF j a new update is sent. Hence, the maximum erE0 ¼ k~ ror that can occur before triggering a new update is E < DBFp. The error occurring at trigger time is not necessarily the maximum error as in the previous cases. The error, E, occurring at trigger time satisfies the condition 0 6 E K DBF p depending on the criterion that triggered the update. Therefore, the error when triggering a new update is – on average – smaller than that of the previous methods. In this case, losing a packet has a smaller effect on the user experience as the density of information is reduced at the cost of minor reduction in compression.
3.1. Experimental setup The experimental setup consists of a three-dimensional virtual environment containing a static toroidal object provided by the CHAI3D library [20]. The participant can explore the remote environment via the SensAble PHANToM Omni haptic device with a 1 kHz sampling rate of the 3DoFs. In order to isolate the effect of packet loss on haptic systems, the network was emulated on the machine rendering the virtual environment, with one way controllable packet loss rates applied only to the force-feedback channel. The packet loss function was implemented using a Bernoulli distribution function, which allows the generation of random Booleans with a desired probability [19]. The experimental setup is illustrated in Fig. 5. Four force compression methods are assessed under different packet loss scenarios:
j
f j k; ~ fj f i is the original force and ~ date is sent, where DBF j ¼ DBF k~ is the current force. Hence, the maximum error between the original signal and the rendered signal, which occurs when triggering a new update, is E ffi DBF j . As for the case of the deadband method with linear prediction, the error is f ~ f k and when E P DBF p an update is sent, where DBF p ¼ E ¼ k~ i
1. Original deadband described in [3] with a 10% deadband factor. 2. Deadband compression method with the linear prediction as used in [5]. 3. Low-complexity error-resilient compression method using the ‘‘Expected Deviation’’ criterion described in [5] with the acknowledgment delayed by 30 m s. 4. Modified prediction deadband method introduced in this paper with a 10% deadband factor.
j
f p k, ~ f p is the current predicted force. Hence, the maximum DBF k~ error, which occurs when triggering a new update, is E ffi DBFp. However, in the case of using the modified prediction deadband f i ~ f p k and when E P DBF or method, the error is E ¼ k~
These methods are assessed under 10%, 20%, 40%, and 60% packet loss rates.
p
F
F
t
t
c. Reconstructed signal using linear prediction when a packet is lost
a. Original signal
t b. Reconstructed signal with no packet loss
X
X
t
d. Reconstructed signal using proposed modified linear prediction when a packet is lost
Fig. 4. Illustration of a signal reconstruction in case of packet loss using the linear prediction and the proposed modified prediction methods.
J. Awed et al. / Computer Communications 36 (2013) 1621–1628
Fig. 5. The experimental setup.
The experiment begins with a practice phase where each subject is asked to actively explore the virtual environment shown in Fig. 5. In the practice phase no compression and no packet loss is introduced on the force-feedback channel. In addition to freely exploring the environment, the subject is asked to specifically perform the following moves: Move the Phantom stylus (rendered as a small red sphere) on the surface of the torus. Tap on the torus. Statically press on the torus surface. Tap while moving on the surface. The remainder of the experiment is divided into four blocks; each block represents a packet loss rates and contains four 50-second sessions, each session using one of the four compression methods mentioned earlier applied to the signal. The blocks are rendered randomly to each participant and so is the order of the compression methods within each block.The participant is asked to perform the aforementioned moves in each session in order to evaluate the method used and to rate the session based on the rating scale shown in Table 1. The reference session used in the practice phase could be freely accessed at any time during the experiment, and is also rendered at the beginning of each block to help the participants recall the feeling and therefore assist them in rating the subsequent sessions. In addition to the subjective rating of the participant, the data percentage compression ratio (C.R.) is recorded for each method in each session during the experiment, according to:
C:R: ¼ 1
number of samples sent 100 number of samples generated
ð4Þ
4. Results and analysis Before showing the results for subjective real-time testing, we present results from an objective offline test. 4.1. Offline experiment An objective offline test was performed where the reconstructed signal is compared with the original signal. Therefore, an interval of force samples was captured and recorded when interacting with the virtual environment for around 30 s. The recorded
Table 1 Subjective rating scale. Description Excellent Good Fair Poor Very
Rating (%) no problems minor problems some problems several problems bad too many problems
100 75 50 25 0
1625
force samples varied around 1.35 N (as an average) with a standard deviation of 1.03 N. These samples were then fed into the compression algorithms, the communication channel module with different packet loss rates, and finally the reconstruction algorithms. The RMSE (Root Mean Square Error) between the original and the reconstructed signal were calculated. The calculated RMSEs are shown in Table 2. Results show that the proposed compression method has the smallest RMSE for all packet loss rates. For example, at 20% packet loss, the error obtained for the proposed method is approximately equal to the error obtained for the other methods when no packet loss was applied. These results are confirmed by the subjective rating of the participants for the proposed method, even with the presence of a 20% packet loss. The proposed method showed a lower percentage compression ratio 87% compared to the original deadband and the deadband with the linear prediction methods which have 89% and 94% compression, respectively. As for the low complexity error resilient method it showed 92% compression ratio at no packet loss, and decreased as the packet loss increased (91%, 90% and 89% compression ratios at 10%, 20% and 40% packet loss respectively). However, the lower compression ratio can be considered as an acceptable compromise to obtain a significantly better user experience and better RMSE. It is worth noting that since the compression ratio is influenced by the user’s movement and her/his reaction to changes in perception, the compression ratios obtained from the offline test only give an idea of the expected compression ratios when a user is actually controlling the haptic device. 4.2. Subjective experimental results Fifteen subjects participated in this experiment (Dominant Hand: 1 left, 14 right. Gender: 3 Females, 12 Males). The subjects were asked to rate the performance of the different compression methods as described earlier. Table 3 shows the average subjective ratings of the participants in percentage and their standard deviation (STDEV), along with the percentage compression ratio (C.R.) of the tested compression methods under different packet loss rates. Figs. 6 and 7 provide a graphical representation of the results shown in Table 3, where the proposed method (M4) clearly outperforms the other published methods in terms of human perception. In addition, Fig. 6 demonstrates the amount of variation in performance for each method with the increase of the packet loss. The proposed method shows a small drop in performance, while the linear prediction deadband showed a drastic drop in performance when packet loss increases. From Table 3 and Figs. 6 and 7 the following observations can be made: The original deadband method (M1) shows a fair performance (around 65% rating) with low packet loss rate. However, as the packet loss increases, the performance deteriorates slowly remaining in the fair range. The percentage compression ratio is stable around 90%. The deadband with linear prediction method (M2) shows the worst performance among the four compression methods. As packet loss increases, participants were experiencing unstable behavior and highly distorted telepresence. The low complexity error resilient method (M3), originally designed to overcome the effects of packet loss of method (M2), shows a noticeable amelioration in performance compared to M2. However, as packet loss increases, its performance decreases, but remained in the ‘‘fair’’ zone. On the other hand, the percentage compression ratio of this method decreases from approximately 90% to 80% as the packet loss increases. A significant drawback of this method is that it assumes a known packet loss rate, and implements the countermeasure based
1626
J. Awed et al. / Computer Communications 36 (2013) 1621–1628
Table 2 Root Mean Square Error and percentage compression ratio of the tested compression methods. Compression method
Packet loss rate 0%
M1. M2. M3. M4.
Original deadband (DB) DB with linear prediction Low complexity error resilient Proposed modified prediction DB
10%
20%
40%
RMSE
C.R. (%)
RMSE
C.R. (%)
RMSE
C.R. (%)
RMSE
C.R. (%)
0.089 0.086 0.085 0.063
89 94 92 87
0.138 0.28 0.281 0.072
89 94 91 87
0.165 0.396 0.344 0.094
89 94 90 87
0.25 0.749 0.318 0.154
89 94 89 87
Table 3 Subjective ratings %, their standard deviation and the percentage compression ratio (C.R.) of the compression methods under various packet loss rates. Compression methods
Packet loss rate 10%
20%
40%
60%
S. STDEV Compression S. STDEV Compression S. STDEV Compression S. STDEV Compression Rating (%) ratio (%) Rating (%) ratio (%) Rating (%) ratio (%) Rating (%) ratio (%) M1. Original deadband (DB) M2. DB with linear prediction M3. Low complexity error resilient M4. Proposed modified prediction DB
65.3
19.1
91.3
68
16
90.6
58
20.2
90.8
54.6
17.9
90.8
67.3
17.9
92.6
48.3
24.9
92.5
25.3
26.6
92.7
14
23.9
92.8
72.3
15.2
90.6
71
11.8
89.5
62.6
18.8
86.3
53.3
20.2
82.1
83.8
7.79
88.5
81.6
11.1
88.4
72
17.4
88.4
54.8
17.8
87.6
100 95 90 85 80 75 70
Subjective Rating %
65 60 55 50 45 40 35 30 25 20
1. Original Deadband (DB)
15
2. DB with Linear Predicon
10
3. Low Complexity Error Resillient 4. Proposed Modified predicon DB
5 0 10
20
40
60
Packet Loss Rate % Fig. 6. Subjective ratings of the compression methods under different packet loss rates.
on this ratio. Therefore, any inaccuracy in this ratio would lead to a further deterioration in the performance (inherited from method M2).
The modified prediction deadband method (M4) shows a clear amelioration of the user experience, even when the packet loss increases in the network. A decrease in the percentage
1627
J. Awed et al. / Computer Communications 36 (2013) 1621–1628 95 94 93
M2
92
M1
91
M3
Compression ratio %
90 89
M4
88 87 86 85 84 1. Original Deadband (DB)
83
2. DB with Linear Predicon 82 3. Low Complexity Error Resillient 81
4. Proposed Modified predicon DB
80 10
20
40
60
Packet Loss Rate % Fig. 7. The percentage compression ratios (C.R.) of the compression methods under various packet loss rates.
compression ratio only around 2.5% was observed in comparison to method (M1), but this decrease is highly compensated for by more than 10% increase in the user ratings compared to methods (M1), (M2) and (M3). In addition, Internet statistics [21] show that the global packet loss rate over a period of 30 days was around 20%. Based on this packet loss rate, the proposed modified prediction deadband method (M4) provides a significant improvement in perception (20%, 69% and 15% over the previously published compression methods M1, M2 and M3, respectively) at the cost of a slight decrease in percentage compression ratio (2%, 4% and 1% compared to methods M1, M2 and M3, respectively). Moreover, the subjective quality ratings of the proposed method remained above the level considered ‘‘good’’ by the participants (over 75%). It is important to note that the proposed method does not introduce additional processing overhead to the haptic system when compared to the other methods. In the proposed modified prediction deadband method (M4), on the sender side, the system first predicts the rendered value based on the gradient and the previous sent force (1 subtraction, 1 multiplication, and 1 addition). Then the method executes an ‘‘if condition’’ with 4 absolute value operations, 2 subtractions, and 2 multiplications. If the sample should be sent, the gradient is calculated using 2 subtractions and 1 division; else, no computations are carried out. On the receiver side, the predicted value is calculated using 1 subtraction, 1 multiplication, and 1 addition. Then the method executes an ‘‘if condition’’ with 2 absolute value operations, 1 subtraction, and 1 multiplication, and renders the appropriate force value. The low complexity error resilient method (M3), on the sender side, first calculates the predicted value (1 subtraction, 1 multiplication, and 1 addition). Then the method executes an ‘‘if condition’’ with 2 absolute value operations, 1 subtraction, and 1
multiplication. If the sample should be sent, the gradient is calculated using 2 subtractions and 1 division, and the tree is updated (memory access). Else when the sample is not sent (which is the case most of the time), the ‘‘update criterion’’ is calculated using a loop of (m) iterations, where m depends on the receipt of an acknowledgment or the maximum number before resetting the tree. On the receiver side, the predicted value is calculated using 1 subtraction, 1 multiplication, and 1 addition. The original deadband method (M1), on the sender side, consists of an ‘‘if condition’’ with 2 absolute value operations, 1 subtraction, and 1 multiplication. On the receiver side, the method renders the update once it is received with no computations and keeps rendering the same value until the next update. The deadband with linear prediction method (M2), on the sender side, calculates the predicted value (1 subtraction, 1 multiplication, and 1 addition). Then the method executes an ‘‘if condition’’ with 2 absolute value operations, 1 subtraction, and 1 multiplication. If the sample should be sent, the gradient is calculated using 2 subtractions and 1 division operation. On the receiver side, the predicted value is calculated using 1 subtraction, 1 multiplication, and 1 addition. Moreover, since there is little dependency on previous samples sent as in previous methods, no memory overhead is introduced. Finally, this method does not assume knowledge of packet loss rate, and therefore does not require any statistics or information regarding the state of the network, as it is the case in the published method (M3). 5. Conclusion and future work This work addressed transparent communication in haptic systems. The Internet is now an important component of these systems; it is crucial to apply an efficient compression method that respects the real-time constraints of the haptic system while being
1628
J. Awed et al. / Computer Communications 36 (2013) 1621–1628
resistant to packet loss. The modified prediction deadband method was proposed, that modifies the linear prediction deadband method, to mitigate the effects of packet loss in networked haptic systems. Experimental results showed a noticeable improvement in performance compared to other compression methods previously proposed. A major improvement of more than 10% was observed, while only around 2.5% decrease in percentage compression ratio was sacrificed compared to the deadband method. Finally, the proposed method has a negligible computational cost and requires no previous knowledge or assumptions about the status of the network or the packet loss rate. As future work, the implementation of the modified prediction deadband method could be extended to the forward channel on the operator side in addition to its implementation in the feedback channel on the teleperator side where the change in performance of the overall system could be assessed. Furthermore, future work would include the assessment of the performance of the proposed method in delayed networks. Acknowledgments Research funded by the American University of Beirut University Research Board and the Lebanese National Council of Scientific Research.
[7]
[8]
[9]
[10]
[11]
[12]
[13]
[14]
[15]
[16]
References [17] [1] E. Steinbach, S. Hirche, M.O. Ernst, F. Brandi, R. Chaudhari, J. Kammerl, I. Vittorias, Haptic communications, in: Proceedings of the IEEE, Frontiers of Audiovisual Communications, 2011. [2] P. Hinterseer, E. Steinbach, S. Hirche, M. Buss, A novel, psychophysically motivated transmission approach for haptic data streams in telepresence and teleaction systems, in: Proceedings of International Conference Acoustics, Speech, and Signal Processing, Philadelphia, PA, vol. 2, March 2005, pp. ii/ 1097–ii/1100. [3] P. Hinterseer, E. Steinbach, A psychophysically motivated compression approach for 3D haptic data, in: Proceedings of the IEEE Haptics Symposium, Alexandria, VA, USA, March 2006, pp. 35–41. [4] F. Brandi, J. Kammerl, E. Steinbach, Error-resilient perceptual coding for networked haptic interaction, in: Proceedings of the ACM Multimedia, Firenze, Italy, October 2010, pp. 351–360. [5] F. Brandi, E. Steinbach, Low-complexity error-resilient data reduction approach for networked haptic sessions, in: Proceedings of IEEE International Symposium on Haptic Audio-Visual Environments and Games (HAVE) Nanchang, China, October 2011. [6] V. Nitsch, B. Fäber, L. Geiger, P. Hinterseer, E. Steinbach, An experimental study of lossy compression in a real telepresence and teleaction system, in: Proceedings of IEEE International Workshop on Haptic Audio Visual
[18]
[19] [20]
[21] [22]
[23]
Environments and their Applications, Ottawa, Canada, October 2008, pp. 75– 80. C. Shahabi, M.R. Kolahdouzan, G. Barish, R. Zimmermann, D. Yao, K. Fu, L. Zhang, Alternative techniques for the efficient acquisition of haptic data, in: ACM SIGMETRICS/Performance 2001, June 2001. C. Shahabi, A. Ortega, M.R. Kolahdouzan A comparison of different haptic compression techniques, in: IEEE International Conference on Multimedia and Expo, 2002, pp. 657–660. K. Hikichi, H. Morino, I. Fukuda, S. Matsumoto, Y. Yasuda, I. Arimoto, M. Iijima, K. Sezaki, Architecture of haptics communication system for adaptation to network environments, in: Proceedings of IEEE International Conference Multimedia Expo, Tokyo, Japan, August 2001, pp. 744–747. M. Kuschel, P. Kremer, M. Buss, Passive haptic data-compression methods with perceptual coding for bilateral presence systems, IEEE Trans, Syst. Man Cybern. A Syst. Humans 39 (6) (2009) 1142–1151. P. Kremer, M. Kuschel, C. Preusche, M. Buss, G. Hirzinger, Passive event-based extrapolation for lossy haptic data compression in bilateral presence systems, in: Proceedings of IEEE International Conference on Robotics and Automation, Kobe, Japan, May 2009. J.R. Moyne, P.G. Otanez, D.M. Tilbury, Using deadbands to reduce communication in networked control systems, in: Proceedings of the American Control Conference, Anchorage, Alaska, May 2002. S. Hirche, P. Hinterseer, E. Steinbach, M. Buss, Network traffic reduction in haptic telepresence systems by deadband control, in: Proceedings of the 16th IFAC World Congress. Prague, Czech Republic, 2005. S. Hirche, P. Hinterseer, E. Steinbach, M. Buss, Towards deadband control in networked teleoperation systems, in: Proceedings of IFAC World Congress, International Federation of Autom, Control, Prague, Czech Republic, 2005. P. Hinterseer, E. Steinbach, S. Chaudhuri, Model-based data compression for 3D virtual haptic teleinteraction, in: IEEE International Conference on Consumer Electronics, ICCE 2006, January 2006. E. Steinbach, S. Hirche, J. Kammerl, I. Vittorias, R. Chaudhari, Haptic data compression and communication, IEEE Signal Process. Mag. 28 (1) (2011) 87– 96. S. Clarke, G. Schillhuber, M.F. Zaeh, H. Ulbrich, Telepresence across delayed networks: a combined prediction and compression approach, in: IEEE International Workshop on Haptic Audio Visual Environments and their Applications, November 2006, pp. 23–24. P. Hinterseer, E. Steinbach, S. Chaudhuri, Perception-based compression of haptic data streams using Kalman filters, in: Proceedings of IEEE International Conference Acoust, Speech, Signal Process, 2006.