Evaluation of edge-based interaction on a square smartwatch

Accepted Manuscript

Evaluation of Edge-based Interaction on a Square Smartwatch Sunggeun Ahn, Jaeyeon Lee, Keunwoo Park, Geehyuk Lee PII: DOI: Reference:

S1071-5819(17)30119-2 10.1016/j.ijhcs.2017.08.004 YIJHC 2145

To appear in:

International Journal of Human-Computer Studies

Received date: Revised date: Accepted date:

4 October 2016 14 August 2017 24 August 2017

Please cite this article as: Sunggeun Ahn, Jaeyeon Lee, Keunwoo Park, Geehyuk Lee, Evaluation of Edge-based Interaction on a Square Smartwatch, International Journal of Human-Computer Studies (2017), doi: 10.1016/j.ijhcs.2017.08.004

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

ACCEPTED MANUSCRIPT

Highlights • Exploration of edge-based interaction for a square smartwatch: single-edge, multiple-edge, and, edge x screen techniques. • Evaluation of 1D and 2D pointing using square smartwatch edges in comparison with touchscreen-based pointing.

CR IP T

• Qualitative and quantitative empirical data for the design of edge x screen interaction.

AC

CE

PT

ED

M

AN US

• Discussion of design options of edge x screen interaction based on the empirical results.

1

ACCEPTED MANUSCRIPT

Evaluation of Edge-based Interaction on a Square Smartwatch Sunggeun Ahn, Jaeyeon Lee, Keunwoo Park, Geehyuk Lee∗ HCI Lab, KAIST 291 Daehakro, Yuseong, Daejeon 34141, South Korea

CR IP T

Abstract

AN US

Researchers are proposing many approaches to overcome the usability problem of a smartwatch owing to its small touchscreen. One of the promising approaches is to use touch-sensing edges to expand the control space of a smartwatch. We considered possible interaction techniques using touch-sensing edges in combination with the smartwatch touchscreen: single-edge, multi-edge, and edge × screen (edge and touchscreen in combination). We call these techniques square watch interaction (SWI) techniques in this paper because they exploit the form factor of a square smartwatch. To explore the design space and evaluate the usability of the SWI techniques, we implemented a square smartwatch prototype with touch-sensitive edges, and conducted a series of user experiments. The experiment results showed that the SWI techniques enable precise 1D pointing and occlusion-free 2D pointing. The experiments also produced empirical data that reflect human manual skills for the edge × screen techniques. The produced empirical data will provide a practical guideline for the application of the edge × screen techniques. Keywords: smartwatch user interface, edge-based interaction, edge × screen interaction, square watch interaction 1. Introduction

AC

CE

PT

ED

M

The computing power of a smartwatch and its network environment are already good enough to make it a computing platform for diverse mobile applications. A more difficult challenge seems to be developing new user interfaces and new interaction techniques to overcome usability problems due to its small touchscreen. For example, the occlusion problem of a touchscreen interface is now more pronounced than before because its touchscreen size is comparable to the finger size. In addition, its small touchscreen may make useful multi-finger gestures, such as pinch-to-zoom, impractical. To overcome the small screen problem of a smartwatch, researchers have proposed many new user interface ideas. A common approach is to extend the control space beyond the touchscreen. The control space can be extended to the edges of a watch (Oakley and Lee, 2014), to the space above a watch (Harrison and Hudson, 2009), or even to the skin area around a watch (Laput et al., 2014). Among the various ways to extend the control space of a smartwatch, we are particularly interested in extending the control space to the edges of a watch, specifically the four side faces of a square watch, because of the following advantages: 1) Watch edges are physically contiguous with the touchscreen. Mapping from the edges to the GUI controls is almost direct. 2) Watch edges provide a physical ∗ Corresponding

author. Email addresses: [email protected] (Sunggeun Ahn), [email protected] (Jaeyeon Lee), [email protected] (Keunwoo Park), [email protected] (Geehyuk Lee)

guide for a stable finger movement. Moving a finger along an edge enables a precise operation. 3) The control space combining the touchscreen and the edges is small. Manipulating the touchscreen and the edges simultaneously with one hand is easy. The implication of the first two advantages seems clear: the edges will make precise 1D physical controls and do not occlude the screen. The third advantage is, in fact, more appealing to us and prompted us to imagine many possible operations with the simultaneous use of the touchscreen and the edges, which we call edge × screen techniques in the following sections. For instance, touching an icon with a thumb on an edge may invoke a secondary function for the icon. In addition, sliding on the edge with another finger on a volume icon may change the music volume. We conducted a brainstorming session to collect possible scenarios using a touchscreen and edges for a smartwatch with a square form factor. While compiling the scenarios, which we call square watch interaction (SWI), we thought of the following questions. Will the edges really make a precise and occlusion-free physical control? Will the two finger operations covering the edges and the screen be really convenient? Among the various combinations of the edges and the screen areas, which one will be more usable? The goal of the current study was to answer the above questions. Toward this goal, we constructed a smartwatch prototype with touch-sensing edges, and, using the prototype, we implemented five main SWI operations that we selected from the SWI scenarios. Finally, we designed and conducted a series of user experiments to answer the above questions. The results of the experiments support

Preprint submitted to International Journal of Human-Computer Studies

August 30, 2017

ACCEPTED MANUSCRIPT

the advantage of precise and occlusion-free edge-based operations in comparison with touchscreen-based operations. In addition, the results provide empirical data that can be used as a basis for the design of various SWI operations, which we believe is the main contribution of the current study.

using a grasping pattern around the smartwatch. A patent by Yoo and Kim (2015) describes a comprehensive set of interaction techniques using the edges of a square smartwatch. The patent includes interaction techniques using one edge or two edges in combination for 1D pointing, 2D pointing, and other screen manipulation operations. Kubo et al. (2016) utilized the combination of sequential inputs on the touchscreen and bezel of a rectangular smartwatch as input gestures. They showed that the four sides of the screen bezel could be used as distinct input spaces. We could gain from these previous works many valuable insights for utilizing the edges of a square watch. First, movements along the edge may be more precise and stable because of tactile guidance. This also means that we may utilize a touch on the edge for a more precise input. Second, the four edges of a rectangular smartwatch may be used as distinct input spaces. Third, the touchscreen and the edges are close, and, therefore, we may manipulate easily the touchscreen and the edges in combination using a single hand. These insights suggest the potential of the input space using the edges of a square watch. We noted, however, that this input space has not been fully explored, and, in particular, we needed empirical data for designing the edge-based input space. Therefore, our main focus in the current study is to produce empirical data to highlight the strong and weak points of edge-based interaction and provide a basis for the optimal design of various edge-based interaction techniques for a square smartwatch.

2. Related Work

M

AN US

CR IP T

In this section, we review the diverse approaches to overcome the limitations of the small touchscreen of a smartwatch. Researchers have proposed the use of space around a smartwatch, including the mid-air space and the skin surface, as an extended input channel for a smartwatch. Kim et al. (2007) proposed the Gesture Watch, a contact-less hand-gesture recognition system using infrared proximity sensors. Harrison and Hudson (2009) achieved a more precise gesture interaction by using magnetic sensors and a magnet attached to a fingertip. Laput et al. (2014) utilized the skin around a smartwatch as an input space by projecting small, touch-sensitive icons onto the skin. Sensing techniques using electric fields (Zhang et al., 2016c) and acoustic waves (Zhang et al., 2016a) were developed to capture touch inputs on the skin near a smartwatch. More recently, Sridhar et al. (2017) utilized a wearable depth sensor to capture multi-finger gestures on and above the skin on the back of a hand. Han et al. (2015) proposed an interaction strategy for a hover-tracking smartwatch that allows a user to continue a touch-screen operation into the mid-air space. Researchers also proposed the use of other parts of a smartwatch as an input space, including the bezel of the screen (Zhang et al., 2016b), the body of a smartwatch (Xiao et al., 2014; Ogata and Imai, 2015; Yeo et al., 2016; Seyed et al., 2016), and the band area (Lyons et al., 2012; Perrault et al., 2013; Funk et al., 2014; Ahn et al., 2015) of a smartwatch. Among the different parts of a smartwatch, the screen edge is the most popularly explored interaction area except for the touchscreen. Early researchers focused on the possibility that an edge provides tactile guidance for a precise and stable linear movement. Blasko and Feiner (2004) proposed the use of tactile landmarks to enhance and enrich stroke interactions on the edge of a smartwatch. Blask´ o and Feiner (2006) also showed that tactile landmarks are helpful in eyes-free stroke interaction with a cursorless numeric entry system. Ashbrook et al. (2008) investigated the effectiveness of round watch interaction techniques wherein the watch bezel is used as a movement guidance. Kerber et al. (2016) investigated the usability of the digital crown of Apple Watch and the rotatable bezel of Samsung Gear S2. More recently, researchers have explored possible interaction techniques utilizing the touch-sensitive side edge of a watch. Oakley and Lee (2014) explored interaction techniques using the side edge of a round smartwatch. They proposed occlusion-free target selection and gesture input

3. Square Watch Interaction

PT

ED

Below, we summarize SWI scenarios arranged in three categories: 1) using a single edge, 2) using multiple edges at the same time, and 3) using the touchscreen and an edge in combination. 3.1. Single Edge Interaction

AC

CE

An edge can be mapped to a 1D control such as a list control and a scroll bar. As shown in Figure 1a, list items on the smartwatch screen may be small and direct pointing may be difficult owing to occlusion by the finger. In this case, an edge that is parallel to the list control may be used to select an item. The resulting interaction is occlusionfree and nearly direct, similar to the old technique of using an offset cursor (Potter et al., 1988). In Figure 1b, the right edge, mapped to a vertical scroll bar, is used to scroll the page. Similarly, the bottom edge may be mapped to a horizontal scroll bar. The mapping between the edges and the scroll bars in these cases is intuitive and the scroll operations are occlusion free. 3.2. Multiple Edge Interaction Selecting a small target in the smartwatch screen is difficult owing to occlusion by a finger. Figure 1c shows a technique using a horizontal edge and a vertical edge in combination to define a point on the screen. Selecting a 3

ACCEPTED MANUSCRIPT

Figure 2: The SWI prototype with a linear multi-touch sensor around a watch: (a) top-down view and (b) side view showing the sensor.

AN US

CR IP T

3.3.1. An edge as a modifier Modifiers are used for expanding the input vocabulary. In touchscreen interfaces, a long touch or multiple touch points may be regarded as a modifier. However, using a long touch is often annoying and using multiple fingers on a small touchscreen may not be convenient. As an alternative, an edge may be used as a modifier for a touchscreen operation. Figure 1e shows an example. A user touches a name in a contact list to make a call. If a user touches a name while touching the bottom edge, however, the same touch may open a page showing the contact’s details. Figure 1f shows another example. The user slides a finger on the screen to scroll the contact list. If the user does it while touching the bottom edge, however, the list may scroll faster, e.g., jumping to the beginnings of the alphabetical sections of the list. From these examples, we expect that some screen areas may be difficult to reach while a certain edge is being touched owing to the ergonomic constraints of the hand.

ED

M

Figure 1: Single-edge (a and b), multi-edge (c and d), and edge × screen (e – h) interaction examples: (a) 1D pointing, (b) 1D scrolling, (c) 2D pointing, (d) zooming in and out, (e) alternative function, (f) alternative scrolling, (g) changing the music volume, and (h) changing a number.

AC

CE

PT

target in this way may be slower than touching a target on the touchscreen directly, but is occlusion free and may enable precise pointing. Two finger gestures for zooming or rotation on a small smartwatch screen may be difficult. An alternative technique using two edges in combination is shown in Figure 1d. Zooming can be done using the bottom edge and the right edge. Moving away from the bottom-right corner is reminiscent of an enlarging gesture, and moving the other way toward the same corner is reminiscent of a reducing gesture; these two operations can be mapped to zooming-in and zooming-out, respectively. Rotation may be done by using two opposite sides, such as the top edge and the bottom edge. Touching two opposite edges and sliding the fingers in opposite directions as if twisting the watch are reminiscent of a rotating gesture.

3.3.2. Selective Control Selecting an object on the smartwatch screen may be easy, but changing the properties of the selected object may not be easy owing to the limited screen space. Selective control is a possible solution for this situation. Figure 1g shows a selective control scenario. The user changes the music volume by sliding a finger on the right edge while touching a volume control on the touchscreen. Figure 1h shows another scenario, wherein the user changes the minute field of an alarm setting by sliding on the bottom edge while touching the minute field. From these examples, we may expect that there will be preferred combinations of a screen area and an edge owing to the ergonomic constraints of the hand. 4. SWI Prototype SWI requires a square smartwatch with touch-sensing edges. Each edge should be able to detect a finger touch as well as its position on the edge. Figure 2 shows the SWI prototype. We constructed a multi-touch sensor around a smartwatch (LG G Watch, 280 × 280 resolution, 29.6 mm × 29.6 mm screen). The multi-touch sensor is based on the TouchString concept introduced in Gu and Lee (2011). The sensor consists of a series of 32 infrared LEDs and 32 photodiodes (8 LEDs and 8 diodes on each side). The output of the sensor, a

3.3. Edge × Screen Interaction In this section, we present interaction scenarios wherein the touchscreen and the edges are used in combination. The edge has the advantage of precise and stable 1D input, whereas the touchscreen has the advantage of direct 2D input. Their complementary combination is expected to enable efficient novel input techniques. 4

ACCEPTED MANUSCRIPT

We then conducted a preliminary experiment and five main experiments to examine the unit SWI operations. The first goal of the experiments was to compare the SWI pointing operations with their touchscreen counterpart operations. The second but more important goal of the experiments was to produce empirical reference data for the design of the unit operations. For instance, there can be many possible edge and direction combinations in the case of Shift-Drag, and we need an empirical ground to determine the best combinations. In the first subsection, we describe the participants because some of the experiments were conducted sequentially with the same participants. In the subsequent subsections we describe the preliminary and the five main experiments.

CR IP T

Figure 3: Multi-touch sensing: (a) a proximity image from the sensor and (b) the corresponding touch situation.

32-pixel proximity image around the watch, is sent to a PC via an Arduino board for signal processing. The bar graphs in Figure 3a visualize the proximity image when two fingers are on the edges of the watch, as shown in Figure 3b. The position of the finger on the right edge, for instance, can be estimated by calculating the centroid of the finger image on the right edge. It is also required to know the touch state of a finger, i.e., whether a finger is on the sensor or over the sensor. The touch state can be determined from the amplitude of a finger image. The signal processing steps to determine the position of a finger are straightforward. However, some trial-and-error procedures were needed for the robust detection of touch events. A hysteresis mechanism was used to avoid chattering while a finger is detached from the edge. In addition, the derivative of the sensor output, as well as the output value itself, was used for more robust detection of touch events. The PC sends the signal processing results to the watch in the form of GUI events via a Bluetooth link. We share the implementation details of the SWI prototype on our web page1 for those who may be interested in reproducing our study.

5.1. Participants

5. Experiments

PT

ED

M

AN US

The preliminary experiment and Experiments 1 and 2 were conducted together in sequence with the same 12 participants recruited from our university (5 males and 7 females, mean age = 21.25 (SD = 3.38)). They were all right-handed and smartphone users, but none of them was a smartwatch user. Experiment 3 was conducted with 16 right-handed participants recruited from our university (13 males and 3 females, average age = 22.31 (SD = 2.91)). All of them were smartphone users, and four of them had an experience of using a smartwatch. None of them participated in the previous experiments. Experiments 4 and 5 were conducted together with 16 right-handed participants recruited from our university (13 males and 3 females, mean age = 21.06 (SD = 2.14)). All of them were smartphone users, and two of them had an experience of using a smartwatch. None of them participated in the previous experiments. The order of Experiment 4 and 5 was counterbalanced across the participants. In all experiments, all participants wore a smartwatch on their left arm while performing the experiment tasks.

To examine SWI empirically, we first chose five unit SWI operations that underlie the diverse SWI scenarios reviewed in Section 3.

CE

5.2. Preliminary Experiment This experiment was preliminary to Experiment 1; the goal of this experiment was to determine the edge to be used in Experiment 1. In a pilot study, we learned that the right edge and the bottom edge were clearly preferred to the left edge and the upper edge; however, we could not determine the preference between the right edge and the bottom edge. Therefore, the preliminary experiment was a within-subject experiment for the 1D Select task with two independent variables: Edge (Right and Bottom) and Size (5, 10, 20, 35, 70, and 140 pixels, where 1 pixel = 0.106 mm). The measured metrics were the mean completion time (Completion Time) and the error rate (Error Rate). The whole experiment took approximately 20 min for each participant. The main conclusion was that Edge had no significant main effect on Completion Time or on Error Rate. In addition, there was no significant Edge × Size interaction

AC

• 1D Select - to select a target in a 1D list. • 2D Select - to select a target in a 2D space. • Shift-Select - to select a screen area while touching an edge. • Shift-Drag - to drag in the screen while touching an edge. • Pivot-Slide - to slide along an edge while touching a screen area. 1 http://hcil.kaist.ac.kr/?page

id=2196

5

ACCEPTED MANUSCRIPT

Figure 4: 1D Select task: (a) selecting a target with an edge, (b) a target in a vertical bar, and (c) a target in a horizontal bar.

CR IP T

effect on Completion Time or on Error Rate. The trends in Completion Time and Error Rate as Size increased were similar to those of Experiment 1 (the Edge case). However, 10 out of the 12 participants answered that they preferred the right edge to the bottom edge. On the basis of these results, we decided to use the right edge in Experiment 1.

Figure 5: Trends of (a) Completion Time (CT) and (b) Error Rate (ER) for the 2(T ech) × 2(W idth) factorial cases as Size increased in Experiment 1. Error bars represent standard errors.

5.3. Experiment 1: 1D Select

Latin square. In each session, the participants performed ten 1D Select trials for each of the six sizes. The position of the target was random, and the order of Size was shuffled randomly. In total, each participant performed 720 (3 blocks × 4 sessions × 6 sizes × 10 trials) pointing trials. We measured the mean completion time (Completion Time) and the error rate (Error Rate) for each session. The participants were not allowed to retry a selection in a single trial, and, therefore, Error Rate was simply the rate of failed trials. This was also true in all experiments in the current study.

M

AN US

The goal of this experiment was to evaluate the performance of the edge-based 1D pointing technique in comparison with its touchscreen-based counterpart. In particular, we were interested in the benefit of the occlusion-free aspect of the edge-based technique. Our hypothesis was that the performance of the edge-based technique would only be affected by Size, whereas that of the touchscreen-based technique would be affected by both Size and W idth.

5.3.1. Task As shown in Figure 4a, a vertical or horizontal bar with a target appears on the screen. The user is required to select the target. When the user touches an edge along the bar, a thin red cursor appears and moves with the finger. When the user moves the cursor over the target, the target turns magenta. When the user releases the touch, the target is finally selected. The watch provides a feedback about the success of the operation: a brief vibration for success and a black screen for failure. There are two task parameters as explained in Figure 4b and c: the size (Size) and the width (W idth) of the target. Note that Size refers to the length of the target along the direction of the finger movement in both cases.

AC

CE

PT

ED

5.3.3. Results Figure 5 shows the trends of Completion Time and Error Rate in the 2(T ech) × 2(W idth) factorial cases as Size increased. In all cases, Completion Time tended to decrease and Error Rate tended to decrease as Size increased. Completion Time for the Screen case appears to be smaller than for the Edges case. Error Rate in the Screen and Narrow case appeared to be affected considerably by Size. 2(T ech) × 2(W idth) × 6(Size) RM-ANOVA (with a Greenhouse-Geisser adjustment) results on Completion Time and Error Rate are shown in Table 1. All independent variables had a significant main effect on Completion Time and Error Rate except that W idth did not have a significant main effect on Completion Time. Post hoc paired-sample t-tests with Bonferroni correction showed that the Edges case had a significantly lower Error Rate than the Screen case when Size was smaller than 20 pixels (2.12 mm) in the Narrow case (p < 0.001 for all cases). To examine the effect of W idth and Size on Error Rate in each of the two T ech cases, we conducted a 2(W idth)×6(Size) RM-ANOVA on Error Rate for each of the two T ech cases (Edges and Screen), and the results are shown in Table 2. As expected, Size only had a significant main effect on Error Rate in the Edges case, whereas both

5.3.2. Design and Procedure The experiment had a within-subject factorial design with three independent variables: T ech (Edges or Screen), W idth (Narrow (20 pixels) or Wide (80 pixels)), and Size (5, 10, 20, 35, 70, and 140 pixels where 1 pixel = 0.106 mm). The participants performed one practice block and three main blocks. Each block consisted of 2(T ech) × 2(W idth) factorial sessions in the W idth-major order. In other words, the participants repeated a session for each T ech for the first W idth and repeated a session for each T ech for the next W idth. The order of the factorial conditions was counterbalanced across participants by using a balanced 6

ACCEPTED MANUSCRIPT

Table 1: Results of the 2(T ech) × 2(W idth) × 6(Size) RM-ANOVA on Completion Time (CT) and Error Rate (ER) in Experiment 1. Shaded p-values represent significant effects. Effect

CT

ER

df

df2

F-value

p-value

T ech(T ) W idth(W ) Size(S) T ×W T ×S W ×S T ×W ×S

1.000 1.000 1.090 1.000 1.207 1.479 1.596

11.000 11.000 11.986 11.000 13.272 16.159 17.560

80.300 1.705 132.019 0.865 4.405 2.107 2.135

< .001 .218 < .001 .372 < .050 .162 .155

T ech W idth Size T ×W T ×S W ×S T ×W ×S

1.000 1.000 1.458 1.000 1.955 2.865 2.668

11.000 11.000 16.037 11.000 21.501 31.513 29.349

82.980 120.418 75.903 120.889 13.526 15.424 19.204

< < < < < < <

Figure 6: 2D Select task: (a) selecting a target with two edges, (b) selecting a target with the touchscreen, and (c) the size of a target.

.001 .001 .001 .001 .001 .001 .001

ror rate) than touchscreen-based pointing, in particular for small targets.

CR IP T

DV

Effect

df

df2

F-value

p-value

Edges

W idth(W ) Size(S) W ×S

1.000 2.005 2.239

11.000 22.057 24.634

0.496 20.898 0.475

.496 < .001 .793

Screen

W idth Size W ×S

1.000 1.422 2.742

11.000 15.638 30.164

168.691 55.817 30.305

< .001 < .001 < .001

M

Condition

AN US

Table 2: Results of the 2(W idth) × 6(Size) RM-ANOVA on Error Rate for the two T ech cases (Edges and Screen) in Experiment 1. Shaded p-values represent significant effects.

5.4.1. Task As shown in Figure 6a, a circular black target appears on the screen. The user is required to select the target. When the user touches the right and bottom edges with his/her fingers, a cross-hair (with a small blue circle at the center) appears on the screen and moves as the two fingers move. When the user moves the cross-hair over the target, the target turns cyan. When the user releases one of the fingers, the target is finally selected. The watch provides a feedback about the success of the operation in the same way as in the 1D Select case. Figure 6b shows the touchscreen counterpart; the target is selected when the finger is lifted from the screen. There is only one task parameter as shown in Figure 6c: the diameter (Size) of the target.

ED

W idth and Size had a significant main effect on Error Rate in the Screen case.

AC

CE

PT

5.3.4. Discussion In terms of Error Rate, edge-based pointing had an advantage over touchscreen-based pointing for narrow targets as we expected because it does not have an occlusion problem. In terms of Completion Time, however, touchscreen-based pointing was better than edge-based pointing regardless of the target width. We speculate that this difference was due to the indirect nature of edge-based pointing. In fact, three participants said that they felt ‘remoteness’ when using an edge owing to the separation between a touched position and a cursor position. The main conclusion of this experiment seems to be that edge-based pointing is more effective for accurate 1D pointing but is slower than touchscreen-based pointing. 5.4. Experiment 2: 2D Select

5.4.2. Design and Procedure The experiment had a within-subject factorial design with two independent variables: T ech (Edges or Screen) and Size (8, 16, 24, 32, 40, and 48 pixels where 1 pixel= 0.106 mm). The participants performed one practice block and five main blocks. Each block consisted of two (T ech) sessions. The order of T ech was counterbalanced across participants. In each session, the participants performed ten 2D Select trials for each of the six sizes. The position of the target was random, and the order of Size was shuffled randomly among the 60 trials (6 sizes × 10 trials). In total, each participant performed 600 (5 blocks × 2 sessions × 6 sizes × 10 trials) trials. We measured the mean completion time (Completion Time) of the task and the error rate (Error Rate) for each session. After the experiment, we collected subjective feedback about the two techniques from the participants. 5.4.3. Results Figure 7 shows the trends of Completion Time and Error Rate for the two T ech cases as Size increased. In both cases, both Completion Time and Error Rate tended to decrease as Size increased. Completion Time for the Screen case appeared to be shoter than that for the Edges case for all sizes. Error Rate in the Screen case appeared to be affected more by Size than that in the Edges case.

The goal of this experiment was to evaluate the performance of the edge-based 2D pointing technique in comparison with its touchscreen-based counterpart. Owing to the occlusion-free nature of edge-based pointing, we expected that it would be particularly advantageous for 2D pointing. Therefore, our hypothesis was that edge-based pointing would be more accurate (i.e., having a lower er7

ACCEPTED MANUSCRIPT

Figure 8: Shift-Select task: (a) an example, (b) the four edges to hold, and (c) the four quadrants to touch.

CR IP T

hand, all participants said that edge-based pointing was good for accurate pointing. There were higher individual variations in Error Rate in the case of edge-based pointing; 6 out of 12 participants maintained an Error Rate below 30% even for the smallest target (8 pixels; 0.848 mm), and one of them showed an 8% Error Rate for the smallest target. The main finding of this experiment seems to be that edge-based pointing is more effective for selecting a small target but is slower than touchscreen-based pointing.

Figure 7: Trends of (a) Completion Time (CT) and (b) Error Rate (ER) for the two T ech cases as Size increased in Experiment 2. Error bars represent standard errors.

Effect

df

df2

F-value

p-value

CT

T ech(T ) Size(S) T ×S

1.000 2.712 2.365

11.000 29.834 26.016

1207.939 176.920 122.816

< .001 < .001 < .001

ER

T ech Size T ×S

1.000 2.061 2.837

11.000 22.674 31.207

23.989 253.423 62.060

< .001 < .001 < .001

5.5. Experiment 3: Shift-Select

M

T ech

AN US

Table 3: Results of the 2(T ech) × 6(Size) RM-ANOVA on Completion Time (CT) and Error Rate (ER) in Experiment 2. Shaded p-values represent significant effects.

ED

The results of the 2(T ech) × 6(Size) RM-ANOVA (with a Greenhouse-Geisser adjustment) on Completion Time and Error Rate are shown in Table 3. All independent variables had a significant main effect on both of the dependent variables. To examine the effect of T ech on Error Rate in each of the six Size cases, we conducted six post hoc pairedsample t-tests on Error Rate with Bonferroni corrections. The results indicated that touchscreen pointing had a significantly lower Error Rate than edge-based pointing when the target size was 48 pixels (p < 0.050). On the other hand, edge pointing has a significantly lower Error Rate than touchscreen pointing when the target size is 8 or 16 pixels (p < 0.001 for both cases).

Shift-Select is a two-finger operation usually performed by using a thumb and an index finger. The usability of Shift-Select may depend on many design parameters such as the finger for holding an edge and the target position on the screen. The goal of this experiment was to identify good combinations of design parameters for Shift-Select. In other words, we did not have any hypothesis in advance but aimed to produce reference data showing the relative usability scores of different Shift-Select design possibilities.

AC

CE

PT

5.5.1. Task As shown in Figure 8a, a blue mark indicating a target edge appears, and a black square mark indicating a target screen quadrant appears. The user is required to touch the indicated quadrant while touching the indicated edge. The watch provides a brief haptic feedback in response to the touch and release actions. The watch provides visual feedback about the touched points on the edge during the practice blocks, but does not during the main blocks. After each trial, the screen stays blue for 500 ms, enforcing the user to wait. The user is instructed to aim at the marks on the screen and the edge, but does not need to be precise to complete the task. There are two task parameters, as shown in Figure 8b and c: the edge to hold (Edge) and the quadrant to touch (Quadrant).

5.4.4. Discussion Edge-based pointing took longer Completion Time than touchscreen-based pointing for all target sizes. On the other hand, edge-based pointing exhibited a reasonably lower Error Rate (mean = 19.8%, SD = 13.8) even when the target size was as small as 16 pixels (1.696 mm), whereas touchscreen-based pointing exhibited a considerably higher Error Rate (mean = 55.8%, SD = 16.8) when the target size was 16 pixels. Four participants felt discomfort while pointing using an edge, and they preferred using a touchscreen for selecting a large target. On the other

5.5.2. Design and Procedure We conducted two sub-experiments for the Thumb-Shift and Index-Shift cases. Thumb-Shift means that the thumb is used for shifting, i.e., one holds an edge with the thumb and selects a target with the index finger. Likewise, IndexShift means that the index finger is used for shifting, i.e., one holds an edge with the index finger and selects a target with the thumb. Except for the finger used for shifting, the 8

ACCEPTED MANUSCRIPT

M

AN US

CR IP T

two sub-experiments were identical. Each sub-experiment had a within-subject factorial design with two independent variables: Edge (Right, Top, Left, and Bottom) and Quadrant (target screen area: Q1, Q2, Q3, and Q4. See Figure 8c). We decided to conduct two sub-experiments for the Thumb-Shift and Index-Shift cases separately, instead of conducting a single combined experiment where the shifting finger is an independent variable, because a strong interaction between the shifting finger and the two independent variables, Edge and Quadrant, was obvious, which, in turn, would make the interpretation of the statistical analysis results unnecessarily involved. Participants performed one practice block, five main blocks, and one evaluation block. Each block consisted of four sessions (one for each of the Edge cases). The order of Edge was counterbalanced across participants using a balanced Latin square. In each session, the participants performed five Shift-Select trials for each of the four Quadrant cases. The order of Quadrant was shuffled randomly. In total, each participant performed 400 (5 blocks × 4 sessions × 4 quadrants × 5 trials) trials during the main sessions. In the evaluation block, the participants rated the comfort (Comfort) scores of the factorial conditions using a 7-point Likert scale (1 = very uncomfortable and 7 = very comfortable) while trying Shift-Select in the factorial conditions. Using the data from the five main blocks, we computed the mean completion time of the task (Completion Time) and the precision of the target selection (Precision) for each of the 4(Edge) × 4(Quadrant) factorial conditions. Precision was the standard deviation of the selection errors of the five samples in each session.

Figure 9: Results of Experiment 3: (a) Comfort, (b) Completion Time (CT), and (c) Precision (PR) data for the 4(Edge) × 4(Quadrant) factorial conditions from the two sub-experiments.

ED

on Comfort, we used the aligned rank transform (Wobbrock et al., 2011) before performing the RM-ANOVA. The results are shown in Table 4. Significant effects are indicated by shaded p-values. In all cases, Edge had a significant main effect on all of the three metrics.

PT

5.5.3. Results Figure 9 shows the Comfort, Completion Time, and Precision data for the 4(Edge) × 4(Quadrant) factorial conditions from the two sub-experiments. Each of the crossshaped tables in the figure summarizes 16 data values for the 16 factorial conditions. The four wings in the table correspond to the four Edge cases, and the four quadrants in each wing correspond to the four Quadrant cases. For example, the number 1.81 in the top wing of the first table in Figure 9a is the Comfort value for the Top and Q2 condition in the Thumb-Shift case. The trends of Comfort, Completion Time, and Precision seemed to be highly dependent on Edge, but did not seem to be dependent on Quadrant as much. Figure 10 shows the dependency of Comfort, Completion Time, and Precision on Quadrant and Edge. The curves for the Thumb-Shift case look inverted with respect to the curves for the Index-Shift cases, reflecting the opposing roles of the thumb and the index finger. We conducted 4(Edge) × 4(Quadrant) RM-ANOVA tests on Completion Time, Precision, and Comfort for each sub-experiment. We used a Greenhouse-Geisser adjustment whenever the sphericity was violated. In the tests

AC

CE

5.5.4. Discussion In the post-experiment interview, we could identify three types of ergonomic problems in the Shift-Select operation: 1) twisting of the wrist, 2) crossing between the thumb and the index finger, and 3) crowding of the thumb and the index finger. All participants mentioned the wrist twisting problem, 13 of them mentioned the finger crossing problem, and 7 of them mentioned the finger crowding problem. The Comfort data shown in Figure 9a reflect the wrist twisting problem clearly. When the top edge or the bottom edge was used for shifting, the severity of the twisting problem seemed to swing between two extremes depending on the finger used for shifting. When the left or the right edge was used for shifting, some twisting always occured regardless of the finger used for shifting, but the severity of the problem was mild. The finger crossing problem occured for some combinations of an edge and a quadrant, but they did not seem to 9

ACCEPTED MANUSCRIPT

Table 4: Results of the 4(Edge) × 4(Quadrant) RM-ANOVA on Completion Time (CT), Precision (PR), and Comfort (CF) for each subexperiment in Experiment 3. Shaded p-values represent significant effects. Effect

df

df2

CF F-value

p-value

df

df2

CT F-value

p-value

df

df2

PR F-value

p-value

3.000 1.986 9.000

45.000 29.788 135.000

68.533 9.881 4.344

< .001 < .001 < .001

1.448 1.829 2.810

12.219 27.428 42.143

21.718 3.567 3.097

< .001 < .050 < .050

3.000 2.016 3.221

5.291 42.143 48.314

45.000 0.693 4.072

< .005 .509 < .010

3.000 1.776 3.844

45.000 26.638 57.658

59.969 3.948 2.379

< .001 < .050 .065

1.346 1.733 1.945

20.185 25.988 29.176

8.180 1.743 1.201

< .010 .197 .314

3.000 3.000 4.557

45.000 45.000 68.353

3.027 1.015 1.308

< .050 .377 .273

Thumb-Shift Edge(E) Quadrant(Q) E×Q

AN US

Edge Quadrant E×Q

CR IP T

Index-Shift

Figure 11: Shift-Drag task: (a) an example, (b) the four edges to hold, and (c) the four drag directions.

M

Figure 10: Dependency of Comfort, Completion Time (CT) and Precision (PR) on Edge and Quadrant in Experiment 3.

PT

ED

be as problematic as the wrist twisting problem, possibly because the screen size of a smartwatch is small. For the same reason, however, the finger crowding problem may be more problematic, and it suggests not to place a ShiftSelect target near the edge used for shifting.

5.6. Experiment 4: Shift-Drag Shift-Drag is similar to Shift-Select except that it is used for dragging a target instead of selecting it. Therefore, we used the results from Experiment 3 to predetermine the finger for shifting: the index finger for the right and top edges and the thumb for the left and bottom edges. The main question in this experiment was which of the (EdgeandDirection) combinations is good for Shift-Drag. In this experiment again, we did not have any hypothesis in advance but aimed to produce reference data showing the relative usability scores of different Shift-Drag design possibilities. We describe Experiments 4 and 5 in separate sections, even though they were, in fact, conducted in sequence with the same participants. The order of the two experiments was counterbalanced across participants.

size) appears on the screen. The user is required to drag along the line from the head while maintaining a touch on the indicated edge. The watch provides brief haptic feedback in response to the touch and release actions. The watch provides visual feedback about the touched points on the edge and drag paths on the screen during the practice blocks, but does not during the main blocks. After each trial, the screen stays blue for 500 ms, enforcing the user to wait. There are two task parameters as shown in Figure 8b and c: the edge to hold (Edge) and the drag direction (Direction).

AC

CE

5.6.2. Design and Procedure The experiment had a within-subject factorial design with two independent variables: Edge (Right, Top, Left, and Bottom) and Direction (counter-clockwise (CCW), clockwise (CW), toward the edge (Near), and away from the edge (Far). See Figure 11c). The participants performed 1 practice block, 20 main blocks, and 1 evaluation block. In each block, the participants performed one Shift-Drag trial for each of the 16 (4(Edge) × 4(Direction)) combinations. The order of the 16 combinations was shuffled randomly. In total, each participant performed 320 trials (20 blocks × 16 combinations × 1 trial) during the main blocks. In the evaluation block, the participants rated the comfort (Comfort) of each of the 16 combinations using a 7-point Likert scale (1 = very uncomfortable and 7 = very comfortable). Using the data from the main blocks, we computed the mean completion time of the task (Completion Time).

5.6.1. Task As shown in Figure 11a, a blue mark indicating a target edge appears and a headed line (5/7 of the screen 10

ACCEPTED MANUSCRIPT

CR IP T

Figure 12: Results of Experiment 4: (a) Comfort (CF) and (b) Completion Time (CT) data for the 4(Edge) × 4(Direction) factorial conditions.

Table 5: Results of the 4(Edge) × 4(Direction) RM-ANOVA on Comfort (CF) and Completion Time (CT) in Experiment 4. Shaded p-values represent significant effects. Effect

df

df2

F-value

p-value

CF

Edge(E) Direction(D) E×D

3.000 3.000 9.000

45.000 45.000 135.000

44.002 6.515 3.392

< .050 < .001 < .001

CT

Edge Direction E×D

2.024 1.955 9.000

30.359 29.326 135.000

11.605 6.400 8.096

< .001 < .005 < .001

AN US

DV

5.6.4. Discussion Shift-Drag in the Near and Far directions is similar to the two-finger pinch gesture, and, therefore, the corresponding experiment results may be compared with those of Hoggan et al. (2013a), who analyzed the ergonomic problems of the pinch gesture. They reported that the pinch gesture takes more time when the two fingers move horizontally, which may correspond to the Left and Right edge cases in our case. The Completion Time of the LeftNear condition was relatively longer and seemed to support the correspondence, but the Completion Time of the Right-Near condition did not. Hoggan et al. (2013a) also reported that the pinch-in gesture is usually faster than the pinch-out gesture except for certain hand postures. This is in contrast to our results, wherein, in all edge cases, the Far direction exhibited a faster operation. These differences may be attributed to the constraint by the edge on the finger orientation in our case, as well as on its position. Shift-Drag in the CW and CCW directions is similar to the two-finger rotation gesture, and, therefore, the corresponding experiment results may be compared with those of Hoggan et al. (2013b), who analyzed the ergonomic problems of the rotation gesture. They reported that the rotation gesture in the CCW direction was faster than that in the CW direction when the two fingers started at 60◦ in the horizontal direction. However, when the two fingers started at 120◦ in the horizontal direction, the rotation gesture in the CCW direction was slower than that in the CW direction. In our case, an RM-ANOVA indicated that Direction had a significant main effect on Comfort and Completion Time, but the differences between the Direction cases seemed to be less outstanding than those found in Hoggan et al. (2013b). The differences may be due to the small rotation angle in our case compared with the usual two-finger rotation gestures. The effects of Edge on Comfort and Completion Time seemed to be clear in Figure 12 a and b. In particular, Left exhibited the worst scores among the four edges.

PT

ED

M

5.6.3. Results Figure 12 shows the Comfort and Completion Time data for the 4(Edge) × 4(Direction) factorial conditions. Each of the square tables in the figure summarizes the data values for the four edges for one of the four directions. For example, the number 4.81 in the first square in Figure 12a is the Comfort value for the Top-CW combination. The trends of Comfort and Completion Time seemded to be highly dependent on Edge, but did not seem to be dependent on Direction as much. The left edge appeared to give bad results for any directions. We conducted 4(Edge) × 4(Direction) RM-ANOVA tests on Completion Time with Greenhouse-Geisser adjustment whenever the sphericity was violated. In the tests on Comfort, we used the aligned rank transform before performing the RM-ANOVA. The results are shown in Table 5. Significant effects are indicated by shaded p-values. In all cases, both Edge and Direction had a significant main effect on all of the two metrics. We also conducted post hoc paired t-tests with Bonferroni corrections to compare Comfort and Completion Time between the Edge cases and between the Direction cases. Among the Edge cases, there was a significant difference between Left and each of the other cases (p < 0.010) in terms of Comfort and Completion Time. Among the Direction cases, there was a significant difference between the CCW-Near pair in terms of Comfort (p < 0.010), and between the CW-Far pair in terms of Completion Time (p < 0.010).

CE

5.7. Experiment 5: Pivot-Slide

AC

Pivot-Slide is similar to Shift-Drag except that the finger on the edge is the one moving instead of the finger on the screen. Therefore, we used the results from Experiment 3 to predetermine the finger for sliding on an edge; the index finger is for the right edge and the top edge, whereas the thumb is for the left edge and the bottom edge. The main question in this experiment was which of the (Edge, Quadrant, and Direction) combinations was good for Pivot-Slide. In this experiment again, we did not have any hypothesis in advance, but aimed to produce reference data showing the relative usability scores of different Pivot-Slide design possibilities. 5.7.1. Task As shown in Figure 13a, a gray bar with a red head appears on one of the four sides of the screen and a gray 11

ACCEPTED MANUSCRIPT

CR IP T

Figure 13: Pivot-Slide task: (a) an example, (b) the four quadrants and the four edges, and (c) the two directions on the four edges.

mark indicating a target quadrant appears. As the user slides his/her finger along the edge from the red head while maintaining a fingertip in the indicated quadrant, the color of the quadrant changes every 40 pixel (4.24 mm) movement. The color order is red, orange, yellow, green, blue, and purple. The goal of the task is to change the color of the quadrant to match the color of the other quadrants. The watch provides brief haptic feedback in response to the touch and release actions. After each trial, the screen stays blue for 500 ms, enforcing the user to wait. There are three task parameters, as shown in Figure 13b and c: the quadrant to hold (Quadrant), the edge to slide on (Edge), and the slide direction (Direction). The user is instructed to complete the task without clutching if possible.

AN US

Figure 14: Results of Experiment 5: (a) Comfort (CF) and (b) Completion Time (CT) data for the 4(Edge) × 4(Quadrant) × 2(Direction) factorial conditions.

Table 6: Results of the 4(Edge) × 4(Quadrant) × 2(Direction) RMANOVA on Comfort (CF) and Completion Time (CT) in Experiment 5. Shaded p-values represent significant effects. Effect

df

df2

F-value

CF

Edge(E) Quadrant(Q) Direction(D) E×Q E×D Q×D E×Q×D

3.000 2.101 1.000 4.123 3.000 1.821 3.710

45.000 31.509 15.000 61.848 45.000 27.315 55.654

31.896 3.580 8.157 10.250 15.811 2.440 1.783

< < < < <

CT

Edge Quadrant Direction E×Q E×D Q×D E×Q×D

2.113 3.000 1.000 3.588 1.864 2.335 3.752

31.689 45.000 15.000 53.827 27.956 35.019 56.279

20.151 0.026 9.340 5.127 8.973 1.348 1.855

< .001 .994 < .010 < .005 < .001 .274 .135

PT

ED

M

5.7.2. Design and Procedure The experiment had a within-subject factorial design with three independent variables: Edge (Right, Top, Left, or Bottom), Quadrant (Q1, Q2, Q3, or Q4), and Direction (counterclockwise (CCW) or clockwise (CW). See Figure 13c). The participants performed 1 practice block, 20 main blocks, and 1 evaluation block. In each block, the participants performed one Pivot-Slide trial for each of the 32 combinations (4(Edges) × 4(Quadrant) × 2(Direction)). The order of the combinations was shuffled randomly. In total, each participant performed 640 (20 blocks × 32 combinations × 1 trial) trials during the main blocks. In the evaluation block, the participants rated the comfort (Comfort) of each of the 32 combinations using a 7-point Likert scale (1 = very uncomfortable and 7 = very comfortable). Using the data from the main blocks, we computed the mean completion time of the task (Completion Time).

DV

p-value .001 .050 .050 .001 .001 .110 .150

AC

CE

We used a Greenhouse-Geisser adjustment whenever the sphericity was violated. In the case of Comfort, we used the aligned rank transform before performing the RMANOVA. The results are shown in Table 6. Significant effects are indicated by shaded p-values. In all cases, both Direction and Edge had a significant main effect on Comfort and Completion Time.

5.7.3. Results Figure 14 shows the Comfort and Completion Time data for the 4(Edge) × 4(Quadrant) × 2(Direction) factorial conditions. Each of the cross-shaped tables in the figure summarizes the data in the same way as in Figure 9, and the tables on the left and on the right are for the CCW and CW cases, respectively. The trends of Comfort and Completion Time seemed to be highly dependent on Edge, but did not seem to be dependent on Quadrant as much. We conducted 4(Edge) × 4(Quadrant) × 2(Direction) RM-ANOVA tests on Comfort and Completion Time.

5.7.4. Discussion We mentioned earlier the following three ergonomic problems: wrist twisting, finger crossing, and finger crowding. The results of Experiment 5 shown in Figure 14 also reflect these problems. An outstanding trend was that the Top and Bottom cases were better than the other Edge cases in terms of Comfort and Completion Time. This may be explained by the wrist twisting problem; the hand in the Top and Bottom cases suffered less from the wrist 12

ACCEPTED MANUSCRIPT

CR IP T

pants do not represent diverse user groups. We recruited right-handed participants only because our prototype was designed to be worn on the left wrist. We believe however that our results may be applicable to left-handed users if we invert our results laterally. In addition, our participants were all in their early twenties. The severity of the ergonomic problems that we identified in the current study may differ for different age groups. It may be reasonable to expect that the ergonomic problems may be more pronounced for older age groups. Second, we controlled the usage condition; the participants performed tasks while seated on a chair comfortably. Users elsewhere may use a smartwatch in various conditions and may perform worse in a dynamic condition than in a stable condition. However, the performance drop may not be as significant as one might expect because users may be helped by the guidance of the edges. In addition, users may perform more accurately when they have one finger anchored near the watch (Pizza et al., 2016). Therefore, edge × screen operations in particular may not be affected significantly because users in this case perform the operations with one finger on the watch. Third, we also controlled the usage posture; the participants performed tasks while maintaining the watch-worn arm at approximately 30” with respect to the coronal plane of the body. The angle of the watch-worn arm may have an effect on the wrist twisting problem, and, therefore, our results may be valid only for this specific posture. However, we believe that the actual usage posture of a smartwatch a real world setting may not be very different from the posture that we used in our study.

Figure 15: Marginal means of Comfort (CF) and Completion Time (CT) (a) for the Edge × Quadrant factorial cases and (b) for the Edge × Direction factorial cases in Experiment 5.

M

AN US

twisting problem. Figure 14 seems to also reflect the other two problems; however, to examine the effects of the other two problems better, we plotted the marginal means of Comfort and Completion Time for different quadrants for each of the four Edge cases, as shown in Figure 15a. Consider the Right case for example; Q1 was worst and Q3 was best in terms of Comfort and Completion Time. This may be explained by the finger crossing problem and the finger crowding problem; Q1 seemed to have both problems, whereas Q3 did not have either. The curves for the other Edge cases in the graphs may be explained similarly by the two problems. On the basis of the Comfort and Completion Time data shown in Figure 14, we may favor the four edges for Pivot-Slide in the following order: Top, Bottom, Right, and Left. Table 6 shows that Direction had a significant main effect on both Comfort and Completion Time. This was, in fact, contrary to our expectation: we thought that Comfort and Completion Time would not depend on Direction because hand and finger motions were symmetric in the two Direction cases. To examine the effect of Direction better, we plotted the marginal means of Comfort and Completion Time for different Direction cases for each of the four Edge cases, as shown in Figure 15b. The effect of Direction seemed to be different for different Edge cases. In the Right case, CW was better than CCW in terms of Comfort and Completion Time, but the trend was reversed in the Bottom case. A guideline that we may learn from these results is that the top edge should be favored for bidirectional Pivot-Slide operations because it seems to be affected least by Direction.

AC

CE

PT

ED

6.2. Square vs Round Interaction We assumed the use of a watch with a square form factor throughout the study. Some of our results may be highly dependent on the square form factor, whereas the others may not. The edge-based pointing techniques may be applicable only to the case of a square watch because they assume a rectangular coordinate system. However, our findings about their characteristics may be equally applicable to the case of a circular watch. For instance, we found out that they are not fast owing to their indirect nature. In addition, we found out that the edge-based pointing techniques are precise because they are free from an occlusion problem. The edge-based pointing techniques for a circular watch, as shown in Oakley and Lee (2014), may share the same characteristics of being slow but precise. The edge × screen techniques may be less dependent on the square form factor; the edges-as-modifier techniques, e.g., Shift-Select and Shift-Drag, may be equally applicable to a circular watch. The selective-control techniques, e.g., Pivot-Slide, may be more suitable for a circular watch because a slide operation along a circular edge may be less constrained than a slide operation along a linear, segmented edge. The segmented edge of a square watch may be an advantage, compared with the continuous edge of

6. General Discussions 6.1. Generalizability of the Results Care needs to be exercised when generalizing the experimental results of the current study. First, our partici13

ACCEPTED MANUSCRIPT

a circular watch, when distinct modifiers are needed, but may be a disadvantage when a longer slide operation along the edge is needed.

techniques. The produced empirical data are expected to provide a practical guideline for the application of the edge × screen SWI techniques. Immediate future work includes 1) the design of smartwatch GUI controls leveraging on the SWI techniques and 2) the evaluation of SWI-based smartwatch GUI designs in a real smartwatch environment.

6.3. Reflections on SWI after the studies In this paper, we investigated edge-based interaction techniques for a square smartwatch and could identify their complementary characteristics (slow but precise) and new possibilities when they were combined with touchscreen operations. The complementary characteristics of edge-based operations may be exploited in the GUI design of a smartwatch. In the case of browsing a long list, such as a contact list, touchscreen operations, which are fast but imprecise, may be used to select an item, whereas edge-based operations, which are slow but precise, may be used to scroll the list, possibly with an absolute mapping from the edge to the whole list. The new possibilities of the edge × screen operations may help overcome the limitation of the small watch screen by reducing the number of GUI controls. The empirical data from our study may provide a guideline for mapping functions to different operations. In the case of ShiftSelect, for example, a frequently used function, such as selecting an item, may be mapped to the bottom-edge combination because it is fast and comfortable, whereas an infrequently used or dangerous function, such as deleting an item, may be mapped to the left-edge combination because Shift-Select with the left edge is slow and uncomfortable, but is precise. We started our study on SWI to overcome the limitation of the small touchscreen of a smartwatch. As we proceeded with our study, our attention was drawn to the possibility that edge-based interaction may become eyes-free. Interaction on the touchscreen is rarely eyes-free because the touchscreen does not provide any clear tactile cues. In the case of edge-based interaction, the tactile cues by the edges and the kinesthetic sense of the hand may enable users to manipulate the smartwatch GUI without looking at it. This possibility clearly deserves an empirical study in the future.

References

CR IP T

References

AC

7. Conclusion

CE

PT

ED

M

AN US

Ahn, Y., Hwang, S., Yoon, H., Gim, J., Ryu, J.-h., 2015. BandSense: Pressure-sensitive Multi-touch Interaction on a Wristband. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. CHI EA ’15. ACM, New York, NY, USA, pp. 251–254. URL http://doi.acm.org/10.1145/2702613.2725441 Ashbrook, D., Lyons, K., Starner, T., 2008. An Investigation into Round Touchscreen Wristwatch Interaction. In: Proceedings of the 10th International Conference on Human Computer Interaction with Mobile Devices and Services. MobileHCI ’08. ACM, New York, NY, USA, pp. 311–314. URL http://doi.acm.org/10.1145/1409240.1409276 Blasko, G., Feiner, S., Oct 2004. An Interaction System for Watch Computers Using Tactile Guidance and Bidirectional Segmented Strokes. In: Wearable Computers, 2004. ISWC 2004. Eighth International Symposium on. Vol. 1. pp. 120–123. URL https://doi.org/10.1109/ISWC.2004.6 Blask´ o, G., Feiner, S., 2006. Evaluation of an eyes-free cursorless numeric entry system for wearable computers. In: Wearable Computers, 2006 10th IEEE International Symposium on. IEEE, pp. 21–28. URL https://doi.org/10.1109/ISWC.2006.286338 Funk, M., Sahami, A., Henze, N., Schmidt, A., 2014. Using a Touchsensitive Wristband for Text Entry on Smart Watches. In: CHI ’14 Extended Abstracts on Human Factors in Computing Systems. CHI EA ’14. ACM, New York, NY, USA, pp. 2305–2310. URL http://doi.acm.org/10.1145/2559206.2581143 Gu, J., Lee, G., 2011. TouchString: A Flexible Linear Multi-touch Sensor for Prototyping a Freeform Multi-touch Surface. In: Proceedings of the 24th Annual ACM Symposium Adjunct on User Interface Software and Technology. UIST ’11 Adjunct. ACM, New York, NY, USA, pp. 75–76. URL http://doi.acm.org/10.1145/2046396.2046430 Han, J., Ahn, S., Lee, G., 2015. Transture: Continuing a Touch Gesture on a Small Screen into the Air. In: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems. CHI EA ’15. ACM, New York, NY, USA, pp. 1295–1300. URL http://doi.acm.org/10.1145/2702613.2732849 Harrison, C., Hudson, S. E., 2009. Abracadabra: Wireless, Highprecision, and Unpowered Finger Input for Very Small Mobile Devices. In: Proceedings of the 22Nd Annual ACM Symposium on User Interface Software and Technology. UIST ’09. ACM, New York, NY, USA, pp. 121–124. URL http://doi.acm.org/10.1145/1622176.1622199 Hoggan, E., Nacenta, M., Kristensson, P. O., Williamson, J., Oulasvirta, A., Lehti¨ o, A., 2013a. Multi-touch pinch gestures: Performance and ergonomics. In: Proceedings of the 2013 ACM International Conference on Interactive Tabletops and Surfaces. ITS ’13. ACM, New York, NY, USA, pp. 219–222. URL http://doi.acm.org/10.1145/2512349.2512817 Hoggan, E., Williamson, J., Oulasvirta, A., Nacenta, M., Kristensson, P. O., Lehti¨ o, A., 2013b. Multi-touch rotation gestures: Performance and ergonomics. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’13. ACM, New York, NY, USA, pp. 3047–3050. URL http://doi.acm.org/10.1145/2470654.2481423

We explored possible interaction techniques utilizing the edges and screen of a square smartwatch. We presented the SWI scenario in three categories, and conducted a series of experiments to examine five SWI techniques underlying the SWI scenarios. The first two experiments identified the advantages and disadvantages of the SWI pointing techniques in comparison with their touchscreen counterparts. In particular, the experiments showed that SWI enables precise 1D pointing and occlusion-free 2D pointing compared with its touchscreen counterparts. The next three experiments produced empirical data that could help understand human manual skills for the edge × screen SWI 14

ACCEPTED MANUSCRIPT

Kerber, F., Kiefer, T., L¨ ochtefeld, M., 2016. Investigating interaction techniques for state-of-the-art smartwatches. In: Proceedings of the 2016 CHI Conference Extended Abstracts on Human Factors in Computing Systems. CHI EA ’16. ACM, New York, NY, USA, pp. 2540–2547. URL http://doi.acm.org/10.1145/2851581.2892302 Kim, J., He, J., Lyons, K., Starner, T., Oct 2007. The Gesture Watch: A Wireless Contact-free Gesture based Wrist Interface. In: Wearable Computers, 2007 11th IEEE International Symposium on. pp. 15–22. URL https://doi.org/10.1109/ISWC.2006.286338 Kubo, Y., Shizuki, B., Tanaka, J., 2016. B2b-swipe: Swipe gesture for rectangular smartwatches from a bezel to a bezel. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. CHI ’16. ACM, New York, NY, USA, pp. 3852–3856. URL http://doi.acm.org/10.1145/2858036.2858216 Laput, G., Xiao, R., Chen, X. A., Hudson, S. E., Harrison, C., 2014. Skin Buttons: Cheap, Small, Low-powered and Clickable Fixedicon Laser Projectors. In: Proceedings of the 27th Annual ACM Symposium on User Interface Software and Technology. UIST ’14. ACM, New York, NY, USA, pp. 389–394. URL http://doi.acm.org/10.1145/2642918.2647356 Lyons, K., Nguyen, D., Ashbrook, D., White, S., 2012. Facet: A Multi-segment Wrist Worn System. In: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology. UIST ’12. ACM, New York, NY, USA, pp. 123–130. URL http://doi.acm.org/10.1145/2380116.2380134 Oakley, I., Lee, D., 2014. Interaction on the Edge: Offset Sensing for Small Devices. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’14. ACM, New York, NY, USA, pp. 169–178. URL http://doi.acm.org/10.1145/2556288.2557138 Ogata, M., Imai, M., 2015. SkinWatch: Skin Gesture Interaction for Smart Watch. In: Proceedings of the 6th Augmented Human International Conference. AH ’15. ACM, New York, NY, USA, pp. 21–24. URL http://doi.acm.org/10.1145/2735711.2735830 Perrault, S. T., Lecolinet, E., Eagan, J., Guiard, Y., 2013. Watchit: Simple Gestures and Eyes-free Interaction for Wristwatches and Bracelets. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’13. ACM, New York, NY, USA, pp. 1451–1460. URL http://doi.acm.org/10.1145/2470654.2466192 Pizza, S., Brown, B., McMillan, D., Lampinen, A., 2016. Smartwatch in vivo. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. CHI ’16. ACM, New York, NY, USA, pp. 5456–5469. URL http://doi.acm.org/10.1145/2858036.2858522 Potter, R. L., Weldon, L. J., Shneiderman, B., 1988. Improving the accuracy of touch screens: An experimental evaluation of three strategies. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’88. ACM, New York, NY, USA, pp. 27–32. URL http://doi.acm.org/10.1145/57167.57171 Seyed, T., Yang, X.-D., Vogel, D., 2016. Doppio: A reconfigurable dual-face smartwatch for tangible interaction. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. CHI ’16. ACM, New York, NY, USA, pp. 4675–4686. URL http://doi.acm.org/10.1145/2858036.2858256 Sridhar, S., Markussen, A., Oulasvirta, A., Theobalt, C., Boring, S., 2017. Watchsense: On- and above-skin input sensing through a wearable depth sensor. In: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems. CHI ’17. ACM, New York, NY, USA, pp. 3891–3902. URL http://doi.acm.org/10.1145/3025453.3026005 Wobbrock, J. O., Findlater, L., Gergle, D., Higgins, J. J., 2011. The aligned rank transform for nonparametric factorial analyses using only anova procedures. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’11. ACM, New York, NY, USA, pp. 143–146. URL http://doi.acm.org/10.1145/1978942.1978963

AC

CE

PT

ED

M

AN US

CR IP T

Xiao, R., Laput, G., Harrison, C., 2014. Expanding the input expressivity of smartwatches with mechanical pan, twist, tilt and click. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. CHI ’14. ACM, New York, NY, USA, pp. 193–196. URL http://doi.acm.org/10.1145/2556288.2557017 Yeo, H.-S., Lee, J., Bianchi, A., Quigley, A., 2016. Sidetap & slingshot gestures on unmodified smartwatches. In: Proceedings of the 29th Annual Symposium on User Interface Software and Technology. UIST ’16 Adjunct. ACM, New York, NY, USA, pp. 189–190. URL http://doi.acm.org/10.1145/2984751.2984763 Yoo, H., Kim, M., 07 2015. Electronic device. Samsung Display Devices Co., Ltd, KOR Patent 10-2015-0076568. URL http://patent.ndsl.kr/patDetail.do?cn=KOR1020130164933 Zhang, C., Bedri, A., Reyes, G., Bercik, B., Inan, O. T., Starner, T. E., Abowd, G. D., 2016a. Tapskin: Recognizing on-skin input for smartwatches. In: Proceedings of the 2016 ACM on Interactive Surfaces and Spaces. ISS ’16. ACM, New York, NY, USA, pp. 13– 22. URL http://doi.acm.org/10.1145/2992154.2992187 Zhang, C., Yang, J., Southern, C., Starner, T. E., Abowd, G. D., 2016b. Watchout: Extending interactions on a smartwatch with inertial sensing. In: Proceedings of the 2016 ACM International Symposium on Wearable Computers. ISWC ’16. ACM, New York, NY, USA, pp. 136–143. URL http://doi.acm.org/10.1145/2971763.2971775 Zhang, Y., Zhou, J., Laput, G., Harrison, C., 2016c. Skintrack: Using the body as an electrical waveguide for continuous finger tracking on the skin. In: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems. CHI ’16. ACM, New York, NY, USA, pp. 1491–1503. URL http://doi.acm.org/10.1145/2858036.2858082

15