- Email: [email protected]
Impact of video-mediated communication on simulated service encounters
interacting
with Computers
~018 no 2 (1996)
193-206
Impact of video-mediated communication on simulated service encounters Anne H. Anderson, Alison Newlands, Jim Mullin, Anne Marie Fleming, Gwyneth Doherty-Sneddon and Jeroen Van der Velden*
The results are reported of three studies of collaborative problem-solving in a simulated travel agency where communication between travel agent and customers is supported by a videolink and shared multimedia tools. The video-mediated contexts (WCs) were compared with face-to-face and audio-only interactions in terms of the success of the task outcome, the process of communication and decision making and user satisfaction. WC did not deliver the same benefits as face-to-face interactions. The possible reasons for this are explored as well as the implications of the data for evaluation techniques. Keywords: cooperative
video-mediated work, dialogue
communication, analysis
computer-supported
Previous research on the impact of video-mediated communication (VMC) on collaboration has produced mixed findings. Whether or not the fact that one can see one’s partner has a beneficial effect seemsto be highly dependent on the type of collaborative task. Reviews of the literature report little benefit in seeing one’s partner when performing collaborative problem-solving tasks, while other studies suggest that the visual channel is of benefit for more ‘social’ tasks, such as negotiation or bargaining (Williams, 1977). Other reviews report equally mixed findings with respect to the impact of VMC on communicative processes (Sellen, in press).One of the problems with much of the research is that studies comparing video-mediated or face-to-face interactions with audio-only conditions have tended to focus on one or two measures in isolation - for example, task outcome (Chapanis; 1975; Short et al., 1976; Williams, 1977), or structural aspects of the communicative process such as turn-taking (Sellen, 1992; O’Connaill et aI., 1994), or user satisfaction (Tang and Isaacs, 1993). However, as writers such as Monk et al. (1995) argue, a multidimensional approach to the evaluation of VMC is likely to be more informative about the Human Communication G12 9YR, LJK *Work Psychology Unit, 0953-5438/96/$15.00 PII SO953-5438(96)0102%9
Research Technical
Centre,
Department
University
(0 1996 Elsevier
Science
of Delft,
of Psychology, 2628 BL Delft,
B.V. All rights
University
of Glasgow,
Glasgow
The Netherlands
reserved
193
relative benefits of this technology. Such an approach has also been adopted by researchers such as Olson et al. (1992) and Strauss and McGrath (1994). We need to understand the relationship between these variables in order to get a clearer picture of how technology mediates communication and collaboration. In this paper we describe a number of studies we have conducted on the impact of VMC on collaboration where we adopt such an approach; exploring the effect of such technologies on task outcome, the process of communication and user satisfaction. We conduct studies in the laboratory where we attempt to simulate the kind of application - the provision of travel agent services supported by video link which is being developed in several commercial domains.
General
approach
In our research on the impact of VMC, we explore the performance of users on two collaborative tasks, the Map Task (Brown et al., 1984), and the Travel Game (Anderson et al., 1994). In the Map Task participants collaborate to complete a map route as accurately as possible. This work is described in the paper by O’Malley et al. (1996). Here we focus on simulated service encounters, involving a task called the Travel Game where the participants are engaged in planning an itinerary around the USA. Their goal is to visit as many destinations as possible given the restrictions imposed by available connecting flights. Both tasks produce objective measures of task success against which the corresponding task dialogues can be compared. We use an experimental paradigm when conducting evaluation studies, i.e. we systematically compare the performance of users who attempt such tasks when the system incorporates video technology or provides only an audio link. In addition we compare the performance of subjects who tackle the tasks when communicating face-to-face or with only an audio link. As well as calculating task success we also perform detailed analyses of the structure and content of the task dialogues. This involves examining the lengths of the dialogues, in turns and words, and examining how the process of turn-taking is managed. For the Travel Game we examine the decision-making process, and how frequently in the dialogue clients change their plans and decisions. We also examine the lengths of the dialogues in the different conditions and the turntaking behaviour of speakers. In addition we conduct a detailed post-task questionnaire on aspects of user satisfaction with the task, communication and technology. We report on three studies of the impact of VMC, comparing face-to-face and audio-only interactions with collaborations supported by VMC. We assess the impact of VMC on Travel Game collaborations with different sizes of video display and explore the effect of VMC on remote collaborations across the European Internet.
Background:
the benefits
of face-to-face
interaction
In earlier research on collaborative problem-solving, we had found that in face-toface interaction, participants needed to say significantly less to achieve the same 194
Interacting
with Computers
vol 8 no 2 (1996)
level of performance than in audio-only conditions (Boyle et al., 1994). This study is unusual in showing subtle but significant benefits of the availability of visual signals for collaborative problem-solving. Most earlier studies, which focused only on task outcome, showed no advantage for face-to-face problem-solving (Davis, 1971; Chapanis et al., 1972; Williams, 1977). Only in tasks involving conflict or negotiation was there some evidence of a benefit for communication with visual contact (Morley and Stephenson, 1969; 1970; Short, 1974). The task we used in the Boyle et al. (1994) study and in subsequent explorations of VMC is a form of collaborative problem-solving known as the Map Task (Brown et al., 1984). Two participants each have a copy of a schematic map. One subject, the instruction-giver, has a route shown on his or her copy of the map and his or her task is to describe this route so that the instruction-follower can draw this on his or her copy of the map. Some landmarks differ between the maps providing problem points which the participants have to overcome. The task produces an objective measure of task success - how well the route has been completed, and provides clear and comparable communicative goals for participants. In the Boyle et al. study 32 pairs of undergraduate subjects tackled versions of the Map Task sitting at opposite sides of a table, either communicating face-to-face or with a screen between them. In face-to-face dialogues, speakers used 28% fewer turns and 20% fewer words than in the audio-only condition. Yet face-to-face participants achieved as good levels of task performance with this reduced verbal input. The interaction was also managed more smoothly in face-to-face collaboration, with 8.7% of turns containing interruptions compared to 12% of turns in the audio-only condition. These face-to-face advantages suggest that speakers can use visual signals to supplement the information presented verbally and to assistin managing the process of turn-taking. From detailed analysis of the task dialogues we have found that in face-to-face communication speakers and listeners can use visual signals to check that the communication is progressing smoothly so they need to elicit and provide less verbal feedback from their conversational partner, (see Doherty-Sneddon, et al., forthcoming for details).
Study
1: Impact
of VMC
on travel
collaborations
In the studies described here we use a different collaborative task. The Travel Game was designed to be a form of collaborative problem-solving which involved a more social dimension than the simple exchange of information involved in the Map Task. It is thought that visual signals are useful in establishing social presence and have a greater impact on tasks involving negotiation. We thus expect that the extent to which clients felt at ease with the communicative situation would depend in part upon seeing their conversational partner. As well as including questions about such aspects in our task questionnaire, we also attempted to identify features of the task dialogues which might reflect such a difference. In the Travel Game clients have to make decisions about their travel plans based on the information they have obtained from the travel agent. To optimize their itineraries they might have to request a considerable amount of information from Anderson
et al.
195
the travel agent who has to consult files to check on available flight connections and timings for each such request. We considered that the extent to which the client felt comfortable in involving the travel agent in additional information searches would depend in part on the degree of social presence in the communicative setting. In face-to-face interaction we predicted the client would feel more relaxed about making additional requests for input from the travel agent. Where no visual contact was available to provide emotional or expressive cues we predicted that such requests would be less common. Design and subjects Undergraduate subjects are assigned the role of client. They have to communicate with the travel agent to access flight information required to plan their itinerary across the US. Their goal is to visit as many cities and states in the USA on an Airline Travel pass which has restrictions on lengths of stay in a single state and financial penalties for changing airline or backtracking to an earlier destination. They have an audio link, two shared screen facilities, one a map of the US, one a whiteboard where the travel decisions are logged. In half the tasks they also have a videolink showing the face of their conversational partner in a 3.5 x 4.5 inch window. In the comparison conditions participants either tackle a paper-andpencil version of the task while communicating face-to-face, or via an audio link between adjacent rooms. In the comparison paper-and-pencil conditions, both participants have their own copy of a map of the US showing the major airports. The travel agent has the timetable of flight connections, and she fills in the shared travel log which is visible to both. For the VMC conditions the Travel Game was presented on two 20 in colour monitors on SunSPARC 10 workstations running NV 3.2 (Frederick, 1994), and operating over a dedicated local area network. The video image refreshed at 4-5 frames/s. There was an audio link through microphones and headphones, providing full duplex sound. Figure 1 illustrates the Travel Game with videowindows. The travel agent was a member of the research team who is familiar with the multimedia communication tools, she was instructed to work to a loose script in her interactions with clients to ensure that she was equally helpful to all clients. The experimental hypotheses were not known to her. Her task was to provide information from standard files about the availability of flight connections and timings and to log the client’s itinerary on the shared screen or notebook. Task performances were judged by the number of cities and states visited in the session. All performances are recorded for transcription and analysis. Twenty participants took part in a comparison of face-to-face and audio-only conditions, where the information was presented on paper. The subjects were undergraduates at the University of Glasgow who volunteered to participate in exchange for the chance to compete for a cash prize of E20 for the best itinerary recorded in the Travel Game. Ten subjects were randomly assigned to each condition. A further 20 volunteers participated in the second part of the study, again each was randomly assigned to one condition, attempting the task in either video-mediated or audio-only conditions as described above. Minor changes to the rules of the Travel Game to ensure that it could be completed within 196
interacting with
Computers
~018
no 2 (1996)
Figure 1. Illustration
of Travel Gamescreendisplay
15 minutes were introduced between the two parts of this study. This means the dialogues were shorter for transcription and analysis, and in subsequent studies participants could attempt more than one version of the Travel Game task in a single session. We standardise for length in the cross-condition comparisons described below.
Procedure The subjects were given the following instructions for the Travel game: “You have a map of the USA to plan your 14-day trip. The itinerary which covers the most cities and states will win the prize money. You have $600 but you have spent $499 on an Atlantic Air Passwhich enables you to travel on any Atlantic Air Path. The Travel Agent has these travel details and the flight timetable and connections. You must spend at least 24 hours in any one state but you can travel within the state in that time. The travel agent will note your itinerary as you plan. There’s a surcharge of $50 if you backtrack to an airport you have already visited. There’s a surcharge of $50 if you wish to travel to a non-Atlantic airport. You have 20 (or 15) minutes. Good luck.”
Results of study 1 Table 1 shows a summary of the results of Studies 1,2 and 3. When we compared the task performance data, no significant differences emerged. Participants managed to visit just as many destinations in face-to-face Anderson
et al.
197
Table
1. Overview
Study
Measures Task performance
Study 1 Travel Game (Local) Face-to-face versus audio-only VMC versus audio-only Study 2 Travel Game (Local) VMC Large image VMC Small image audio-only Study 3 Travel Game (Remote) VMC versus audio-only
of studies
Dialogue length
Number of interruptions
No difference
Face-to-face significantly
No difference
No difference
No difference
Face-to-face more optional changes of plans No difference
No difference
No difference
No difference
No difference
No difference
No difference
No difference
No difference
shorter
No difference
Dialogue structure
and audio-only conditions. In the computer-presented Travel Game the itineraries did not differ when participants did or did not have a video link to the travel agent. Subjects achieved good levels of task outcome in all conditions. As in previous studies in the literature, problem-solving performance was generally unaffected by communicative context. We next examined the lengths of the dialogues in the different conditions. We found that when the client and travel agent communicated face-to-face, the travel agent used 22% fewer words than when they were using an audio link between rooms. Due to the changes made in the overall time limit of the task between the two parts of Study 1, separate analyses were carried out on the data from the face-toface/audio and computer-mediated conditions. In the comparison of face-to-face and audio-only dialogues, the Analysis of Variance on the number of words spoken showed a significant effect of role (travel agent versus client, Fl, 18 = 346.8, ~7< 0.0001). Travel agents said more with the mean number of words spoken by the agent 1495 compared to 596 words from the client. The main effect of context (face-to-face versus audio-only interactions) did not reach significance (F1,18 = 1.69, p = 0.2), although the means showed that on average there seemed to be more words spoken in audio-only dialogues with the mean of 1103 compared to 988 in face-to-face. There was a significant interaction between context and role, (F1,18 = 11.51, p < 0.005). Within this interaction, simple effects tests showed that the travel agent said significantly more in the audio-only than in the face-to-face condition, 198
Interacting
with Computers
vol8
no 2 (1996)
(F1,18 = 8.01, p < 0.01, mean words spoken equalled 1635 versus 1356). The length advantage found in face-to-face interaction on the Map Task by Boyle et al. (1994) has been replicated in another collaborative task. When the dialogues from the computer-mediated conditions were analysed, the Analysis of Variance again showed the significant effect of role (F1,18 = 88.9, p < O.OOl), with agents again saying more than clients, with mean-words spoken equalling 753 and 379 respectively. The computer-mediated conditions also showed the significant interaction of role and context (VMC versus audio-only, F1,18 = 5.001, p < 0.05). Within this interaction there was no simple effects of context (Fs < l), and indeed the mean number of words spoken by the travel agent was slightly higher in the VMC condition. There was no significant main effect of context, (F < 1) and the means were almost identical at 568 words in VMC and 564 words in the audio-only condition. VMC failed to deliver the length advantage of face-to-face interaction. To explore the structure and the content of the task dialogues, two types of analyses were conducted. First, turn-taking was examined by calculating the number and rates of interruptions in each dialogue. An analysis of rates of interruptions across the four conditions of the study showed a significant effect of mediated-communication, (F1,38 = 6.46, p < 0.01). On average speakers interrupted one another less often in the second part of the study where the task and communication were computer-mediated. There was no significant effect of visual signals but across the four conditions the rate of interruptions rose from 10.9% of turns containing an interruption in the plus videowindow condition, 12% in minus videowindow, 13.8% in face-to-face and 14.5% in audio-only. Post hoc t-tests showed a significant difference (p < 0.05) between the ends of this distribution. The combination of access to visual signals and the greater formality of mediated communication seem to combine to reduce the incidence of overlapping speech in VMC. We next wanted to explore the content of the clients’ contributions in the Travel Game. Our dialogue analysis aimed at exploring the impact of visual signals on the decision-making process. We assumed that in face-to-face communication the greater sense of shared social presence would lead clients to be more willing to engage the travel agent in extra searches and thus there would be more optional changes of travel plans as the clients sought to optimize their itineraries. To test this hypothesis we looked at the decision-making process in each dialogue. We coded all the changes to the planned itineraries made by clients, and coded these as optional or forced changes. Forced changes were those which resulted from the information provided by the travel agent, such as there being no available flight connection between two cities. Optional changes were those where the client decided on a new destination from choice for example, by revising an earlier part of their itinerary, asking for alternative possible destinations, using the surcharge option in the rules of the game to backtrack to a previous destination. All such optional changes involve more work from the travel agent in terms of additional information searches or alterations to the logged itinerary. When the number of such forced and optional decision changes were calculated, we found that there were significantly more optional changes in face-to-face than in audio-only interactions. On average there were 6.6 optional Anderson
et al.
199
changes per dialogue in face-to-face interactions compared to 2.7 in the audioonly condition, (t 1,18 = 8.17, p < 0.0001). No such difference emerged when we compared dialogues from videowindow and audio-only conditions, where the number of such optional changes was low, with on average 0.4 such optional changes in the VMC condition compared to 1.3 in the comparable audio-only condition. VMC again failed to deliver the same benefits to users as face-to-face interaction. Was this because users were dissatisfied with the technology or felt uncomfortable with this form of task presentation or communication? We explored such issues in a detailed post-task questionnaire given to all participants. We asked participants how easy it was to communicate with the travel agent, asking for a choice from ‘very easy’ to ‘very difficult’. The questionnaire showed that in face-to-face interactions 90% of subjects felt that it was very easy to communicate with the travel agent compared with 50% of audio-only participants. In the videowindow condition again only 50% of participants felt it was very easy to communicate but surprisingly this rose to 80% in the corresponding audio-only condition. Chi squared tests conducted on the questionnaire responses showed that the distribution of answers concerning easeof communication differed significantly between face-to-face and audio-only conditions, (X (1) = 3.8, p = 0.05), but did not differ between the VMC and Audioonly conditions in the second part of the study. How easy was it to make changes to the planned itinerary? In face-to-face communication 70% felt it was very easy compared to 20% in the audio-only condition. In the videowindow condition 30% felt it was very easy to make changes compared to 20% in the corresponding audio-only condition. Chi squared tests again revealed significantly different responses between face-to-face and audio conditions, (X (1) = 4.24, p < 0.05), but no significant differences between the VMC and audio-only conditions. We asked participants to rate their satisfaction with the final outcome of the consultation with the travel agent. In face-to-face interactions 100% reported that they were satisfied or very satisfied with the outcome, as did 90% in the audio-only condition. In the mediated conditions, user ratings were again high with 80% in both conditions satisfied or very satisfied with the outcome. We also asked participants to gauge if they would have found the task easier in the other condition i.e. face-to-face z~ersusremote and vice-versa. Only 10% of the face-to-face participants felt the consultation would have been improved in the audio-only condition but 90% of the audio-only felt things would have been easier face-to-face. None of the videowindow users would have preferred to do without the visual link and 80% felt face-to-face would have been an improvement. Only 30% of participants with no videowindow felt this addition would have been an improvement, but 60% would have preferred a face-to-face interaction. The questionnaire data provides some useful confirmation that the subjective impressions of users is largely in accord with the objective measures of task performance and communication. Although VMC users were satisfied with their overall performance, and often commented on the appeal of the new technology, they felt slightly less at ease in the communication process than those engaged in face-to-face interaction. VMC did not seem to generate the same sense of social 200
Interacting
with Computers
ml 8 no 2 (1996)
presence as face-to-face communication same technology without the addition
Study
2: Impact
and indeed showed of video signals.
of size of videowindow
no benefits over the
on collaboration
In Study 2 we attempted to increase the sense of social presence Travel Game by increasing the size of the videowindow.
in VMC in the
Design and subjects We composed two versions of the Travel Game featuring the west or east of the USA and participants tackled the two versions of the task in two-out-of-three conditions. Thirty-six undergraduate subjects at the University of Glasgow took part in this study. Again the volunteers competed for a cash prize of E20 for the best itinerary. The conditions were Small videowindows, where the image as in the previous study measured 3.5 x 4.5 inches, Large videowindows where the image measured 6.5 x 8 inches, and No videowindow. Subjects were randomly assigned to one of three conditions tackling two versions of the Travel Game either with Large and Small videowindows, Small and No videowindows or Large and No videowindows. All orders of presentation were counterbalanced. The task instructions were the same as those used in Study 1. Did increasing the video image of the conversational partner increase the sense of social presence ? Did this form of VMC afford the advantages of faceto-face interaction in terms of dialogue length or decision-making processes? Were users more satisfied with the communicative context with larger video images?
Results of Study 2 The results of Study 2 were rather disappointing. Increasing the size of the video image had no effect on task outcome, dialogue length or decision-making. Indeed no significant differences on any of these measures were found across the conditions of the experiment. Analyses of Variance were conducted on number of destinations visited, number of words spoken by travel agent and customer, and the number of optional changes to itineraries made by customers. The communicative context (Large, Small and No videowindows) did not produce any significant main effects or interactions, (all Fs < 1). The number of cities visited on average across Large, Small and No videowindow Conditions were 10.8, 11.6 and 11.6 respectively. The total numbers of words spoken across the three conditions, Large, Small and No videowindows were similar with means of 1005, 946 and 973 respectively. The contributions from the travel agent also showed little difference with means of 680, 649 and 660 for the Large, Small and No videowindow conditions. The Analysis revealed only the expected main effect of task role (Fl, 69 = 502.1, P < 0.001) with travel agents again producing more words than customers means 663 and 311, respectively. The number of optional changes of decisions made by clients was small and again similar across all three conditions with means of 1.75, 1.58 and 1.42. Anderson et al.
201
So not only did we not observe increased social presence, we again failed to find any noticeable advantage in the objective data for the addition of video signals to a collaborative task. In the subjective impressions of users, as revealed in the questionnaire responses, there were some hints of possible benefits of VMC in general and of large images in particular. When using large videowindows, 75% of users felt that it was very easy to make changes to their itineraries compared to 54% for small windows and 25% for no videowindows. None of these differences however reached statistical significance when tested by Chi squared tests. In two studies of VMC we have failed to find significant objectively measurable benefits for VMC, (see also the data reported in O’Malley et al. for a similar picture from a different collaborative task). We have found a number of benefits for the presences of visual cues in face-to-face communication, but these have not been replicated in VMC. The subjective impressions of users give some suggestions that there may be a preference for systems which incorporate videolinks between users even if these do not seem to have been translated into differences in task performance or communicative style. In our most recent study we explored a context where the possibly subtle benefits of VMC may prove of value: distance collaboration. We investigated the impact of VMC on task outcome, communication and user satisfaction, where users are aware that their conversational partner is several hundreds of miles away, in a different country.
Study 3: Impact
of VMC on remote collaborations
This study was designed to assess the impact of VMC on long-distance collaboration over the European Internet. The quality of video and audio links provided by the Internet is rather variable. The UK-Netherlands link is generally one of the best connections. In pilot tests and in the main study described here we found that the video signal rarely suffered break up of the image and the audio signal was clearly audible on almost all occasions.
Design and subjects Twelve undergraduates at the University of Glasgow participated as tourists in this Travel Game study for a case prize of BE20as before. Each tackled two versions of the Travel Game, one with and one without a video link. The order of presentation was counterbalanced. The Travel Agent was at the Technical University of Delft in the Netherlands, but operated to the same script as had been used in earlier studies. The students were informed that this was a long distance link between Glasgow and Delft, and the travel agent although a fluent English-speaker spoke with a noticeable Dutch accent. In pretask chat whilst setting up the task, the conversation clearly indicated that this was a genuine long-distance experiment. The same task instructions were used as in Studies 1 and 2. The communication link over the Internet used public domain network video and audio tools (nv, nt, and vat). The same SunSPARC stations were used as in Studies 1 and 2. 202
Interacting
with Computers
~018 no 2 (1996)
Results of Study 3 The first finding was that the Internet connection between the UK and the Netherlands was able to support this kind of collaboration. Of 24 experimental sessions, only one had to be abandoned and rerun due to technical difficulties with the network link. Reasonably good audio and video reception was obtained in the remaining sessions, with clear sound reception and little noticeable lag or break-up of video images. The results of the Travel Game showed no difference in the task outcome for video- or audio-link conditions. The mean number of cities visited in the VMC condition was 9.5 compared to 9.66 in the audio-only condition. T-tests showed these conditions did not differ significantly. Although this again shows no advantage for VMC on outcome measures it is encouraging in other respects. The level of task success was similar to that obtained in earlier studies. The necessary differences imposed by using network communication tools, such as the need to click the pointer in a designated area of the screen before speaking, and the performance of the Internet compared to a dedicated local area network, did not impair performance on the task. When we assessed the communication process in Study 3, we found that there was no difference in the length of the dialogues or in the number of changes of plan between the VMC and audio-only conditions. An Analysis of Variance conducted on the number of words spoken, revealed the expected effect of task role (Fl,ll = 35.9, p < 0.001) with the travel agent saying more than the customers -with mean number of words 372 and 211, respectively. There was no significant main effect or interaction with communicative context (Fs < 1). The average length of the dialogue was 600 words in VMC compared to 567 in audio-only, with the Travel Agent saying almost exactly the same number of words on average in both conditions: VMC 374, audio-only 370 (F < 1). The number of optional changes which customers made to their itineraries was similarly low in both VMC and Audio-only interactions, with a mean of 1 and 0.58, respectively, and a T-test showed these data did not differ significantly. The questionnaire data confirmed some subjective benefits for VMC. Although 83% of users found it very easy to communicate in both conditions, when users were asked about social presence, i.e., how aware they were of the travel agent 92% of responses in VMC were ‘fairly’ or ‘very aware’ compared to only 50% in the audio-only condition. A Chi squared test showed that the distribution of the responses to this question was significantly different in the VMC and audio-only conditions, (X (1) = 6.32, p < 0.05). When asked how often they were worried that they had lost contact with the travel agent, 50% of VMC were ‘rarely’ or ‘not at all worried’ about this compared to only 25% in the audio-only condition, and again a Chi squared test showed the responses different significantly between the VMC and Audio-only conditions, (X (1) = 5.0, p < 0.05). These data suggest that particularly in long-distance collaboration there may be some benefits for VMC in establishing and maintaining social presence.
Discussion VMC Anderson
potentially et al.
offers
us the benefits
of face-to-face
interaction
while
we 203
collaborate with a distant partner. We have tried to illustrate those benefits that accrue in face-to-face interaction and which VMC systems might be designed to emulate. We found that the visual signals in face-to-face conversation seem to be used in several ways. In face-to-face communication speakers need to say less to complete the task successfully. From previous research we believe this is because visual cues can be used to judge that communication is proceeding smoothly and hence less verbal feedback is needed. Visual cues in face-to-face conversations may be used to establish a sense of social copresence which makes us feel at ease with our conversational partner, and so feel able to ask for additional information or help in a shared task. Video-mediated technology has been designed to facilitate collaborative communication at a distance. To simulate such conditions we compared task performance and dialogues between participants in different rooms. In our travel agency simulations, VMC did not result in shorter dialogues, nor in encounters where the clients felt relaxed enough to make frequent optional changes to their itineraries. This pattern of results, where VMC does not deliver the same benefits as face-to-face interaction is similar to our findings from a different collaborative task, reported in O’Malley et al. (1996). Our small study of long-distance collaboration suggests that VMC may offer some advantages in this area by making the conversational partners more aware of one another and reassuring them that they remain in contact even across a remote Internet connection. Further analyses of these remote dialogues continues, to explore whether there are any differences in dialogue structure which reflect these subjective impressions. It remains to be seen whether the effects of VMC which we observed are transient and would diminish with increased experience of VMC. This factor may be important, as reports in the literature which claim added-value for VMC (e.g., Tang and Isaacs, 1993; Olson et al., 1994; Isaacs and Tang, 1994) either use fairly lengthy exposure or repeated exposure over time. What we have shown has implications for the development and use of videomediating technologies. The visual channel is useful for task efficiency even for a task where social factors are not a central component (compared to, for example, negotiation and bargaining tasks). The impact of the visual channel depends upon the medium through which it is transmitted. VMC does not necessarily bring the same efficiency benefits to communication as face-to-face interaction. We have been concerned with the impact on collaboration of video-mediated images of the conversational partner. The conclusions which we might draw from these studies are that the benefits of face-to-face communication are rather subtle and the forms of VMC which we have investigated to date do not seem able to replicate these advantages. This is, of course, only one possible use of video data in collaborative tasks. The possibility of sharing other forms of video data would almost certainly have a large impact on task performance and communication. In the Travel Game task we provided shared accessto relevant video data in the form of the shared map of the US, and the whiteboard where travel decisions were recorded. If such information had not been accessible to both participants we assume the task would have been considerably more difficult and required longer dialogues between tourist and travel agent. In future studies we intend to go 204
Interacting
zuith Computers
vol 8 no 2 (I 996)
further in making systematic comparisons of the impact upon users of relevant video data compared to video images of the conversational partner. A final point we wish to make is methodological. Monk et aI. (1994) propose that a full evaluation of communication technology requires measures of both task outcome and communicative process. This is the approach which we have adopted. Only by looking at both of these aspects in conjunction can we adequately evaluate the impact of VMC. By looking at task performance alone (as in many earlier studies) we would have concluded that there is no difference between video-mediated, face-to-face and audio-only communication. By looking only at measures of process, such as the number of turns per dialogue, we may have concluded that VMC is either equivalent to, or worse than audio-only communication. By looking at process and outcome together we see that communicative structure, verbal and non-verbal, differs across the contexts in ways which offer advantages and disadvantages to the efficiency of the interactions. The visual channel provided by VMC is useful to speakers, but the remoteness of the interaction invokes a change in communication style which offsets the benefits. Our studies show that here is no simple answer to whether VMC ‘works’ or offers advantages. In general, however, the pattern is a complex one. Successful collaboration depends upon a number of complementary processes: how the process of communication is managed, the level of task success achieved and the users’ satisfaction with all the aspects of these processes. VMC has been shown to influence each of these aspects of collaboration within our laboratory studies. While designers will often be concerned with the impact of VMC on users in the workplace using systems for more extended time periods than those we employed, the data we have presented suggests that such studies must also deploy evaluation methods which consider the multidimensional aspects of collaboration. For some commercial applications, however, the initial impact on customers of a new form of service provision is also important. If users’ first experience is unsatisfactory they may well seek out alternative, traditional forms of service. Although VMC users performed well on our task and generally expressed their satisfaction with the technology and the outcome, their subjective reports were not always consistent. Despite these positive signs, they also reported that they would travel for up to an hour to engage in a face-to-face interaction but would accept VMC if more effort than this was required. In any of our studies we would obtain a different answer to the question of whether VMC ‘works’ if we considered task performance, communication process or any aspect of user satisfaction, in isolation. The same dangers apply to studies on the introduction or long-term use of multimedia services in the workplace.
Acknowledgements The research reported in this paper was supported by the Research Council grant to the Human Communication Glasgow University. Study three was facilitated by support action, Project 5: Practical Assessment of Multimedia Wide Anderson
et al.
Economic and Social Research Centre at from the EC COST 14 Area CSCW. 205
References Boyle, E.A., Anderson, A.H. and Newlands, A. (1994) ‘The effects of eye contact on dialogue and performance in a co-operative problem solving task’ Languageand Speech, 37,1,1-20 Brown, G., Anderson, A.H., Yule, G. and Shillcock, R. (1984) Teaching TaIk Cambridge University Press Chapanis, A. (1971) ‘Preclude to 2001: explorations in human communications’ American Psychologist 26, 949-961 Chapanis, A. (1975) ‘Interactive human communication’ Scientijic American 232, 3&42 Chapanis, A., Ochsman, R.B., Parrish, R.N. and Weeks, G.D. (1972) ‘Studies in interactive communication: the effects of four communication modes on the behaviour of teams during co-operative problem-solving’ Human Factors 14,487-509 Davies, M.F. (1971) ‘Co-operative problem-solving: a follow-up study’ Report No. E/71252/ DVS Post Office Long Range Intelligence Division, Cambridge UK Doherty-Sneddon, G., Anderson, A.H., O’Malley, C., Langton, S., Garrod, S. and Bruce, (forthcoming) ‘Face-to-face communication and VMC: a comparison of dialogue structure and task performance’ 1. Experimental Psychol. Applied (under review) Isaacs, E.A. and Tang, J.C. (1994) ‘What video can and cannot do for collaboration: a case study’ Multimedia Systems 2, 63-73 Monk, A., McCarthy, J., Watts, L. and Daly-Jones, 0. (1994) ‘Measures of process’ in MacLeod, M. and Murray, D. (Eds) Evaluationfor CSCW Springer Verlag Morley, I.E. and Stephenson, G.M. (1969) ‘Interpersonal and interparty exchange: a laboratory simulation of an industrial negotiation at the plant level’ Brit. J. Psychol. 60, 543-545 O’Connaill, B., Whittaker, S. and Wilbur, S. (1993) ‘Conversations over videoconferences: an evaluation of videomediated interaction’ Human-Computer Interaction 8, 389428 O’Malley, C., Langton, S., Anderson, A., Doherty-Sneddon, G. and Bruce, V. (1996) ‘Comparison of face-to-face and video-mediated interaction’ Interacting with Computers 8,2,177-192 Olson, J.S., Olson, G.M. and Meader, D.K. (1994) ‘What mix of video and audio is useful for remote real-time work’ Proc. Workshop on VMC, CSCW 1994 (Chapel Hill, NC, USA) ACM Press Olson, G.M., Olson, J.S., Carter, M. and Storrosten, M. (1992) ‘Small group design meetings: an analysis of collaboration‘ Human-Computer Interacfion 8,389428 Sellen, A.J. (1992) ‘Speech patterns in video-mediated conversations’ Proc. CHI 1992 ACM Press, 49-59 Sellen, A.J. (in press) ‘Remote conversations: the effects of mediating talk with technology Human-Computer Interaction Short, J. (1974) ‘Effects of medium of communication on experimental negotiation’ Human Relations 27, 225-243 Short, J., Williams, E. and Christie, B. (1976) The Social Psychology of Telecommunications Wiley Strauss, S. and McGrath, J. (1994) ‘Does the medium matter? The interaction of task and technology on group performance and members reactions’ J. Appl. Psychol. 79, 87-97 Tang, J.C. and Isaacs, E.A. (1993) ‘Why do users like video?’ Studies of Multimedia Collaboration Computer-supported Co-operative Work 1, 163196 Williams, E. (1977) ‘Experimental comparisons of face-to-face and video-mediated communication’ Psychol. Bull. 84, 96%976 206
Interucting with Computers vol8 no 2 (1996)
with Computers
~018 no 2 (1996)
193-206
Impact of video-mediated communication on simulated service encounters Anne H. Anderson, Alison Newlands, Jim Mullin, Anne Marie Fleming, Gwyneth Doherty-Sneddon and Jeroen Van der Velden*
The results are reported of three studies of collaborative problem-solving in a simulated travel agency where communication between travel agent and customers is supported by a videolink and shared multimedia tools. The video-mediated contexts (WCs) were compared with face-to-face and audio-only interactions in terms of the success of the task outcome, the process of communication and decision making and user satisfaction. WC did not deliver the same benefits as face-to-face interactions. The possible reasons for this are explored as well as the implications of the data for evaluation techniques. Keywords: cooperative
video-mediated work, dialogue
communication, analysis
computer-supported
Previous research on the impact of video-mediated communication (VMC) on collaboration has produced mixed findings. Whether or not the fact that one can see one’s partner has a beneficial effect seemsto be highly dependent on the type of collaborative task. Reviews of the literature report little benefit in seeing one’s partner when performing collaborative problem-solving tasks, while other studies suggest that the visual channel is of benefit for more ‘social’ tasks, such as negotiation or bargaining (Williams, 1977). Other reviews report equally mixed findings with respect to the impact of VMC on communicative processes (Sellen, in press).One of the problems with much of the research is that studies comparing video-mediated or face-to-face interactions with audio-only conditions have tended to focus on one or two measures in isolation - for example, task outcome (Chapanis; 1975; Short et al., 1976; Williams, 1977), or structural aspects of the communicative process such as turn-taking (Sellen, 1992; O’Connaill et aI., 1994), or user satisfaction (Tang and Isaacs, 1993). However, as writers such as Monk et al. (1995) argue, a multidimensional approach to the evaluation of VMC is likely to be more informative about the Human Communication G12 9YR, LJK *Work Psychology Unit, 0953-5438/96/$15.00 PII SO953-5438(96)0102%9
Research Technical
Centre,
Department
University
(0 1996 Elsevier
Science
of Delft,
of Psychology, 2628 BL Delft,
B.V. All rights
University
of Glasgow,
Glasgow
The Netherlands
reserved
193
relative benefits of this technology. Such an approach has also been adopted by researchers such as Olson et al. (1992) and Strauss and McGrath (1994). We need to understand the relationship between these variables in order to get a clearer picture of how technology mediates communication and collaboration. In this paper we describe a number of studies we have conducted on the impact of VMC on collaboration where we adopt such an approach; exploring the effect of such technologies on task outcome, the process of communication and user satisfaction. We conduct studies in the laboratory where we attempt to simulate the kind of application - the provision of travel agent services supported by video link which is being developed in several commercial domains.
General
approach
In our research on the impact of VMC, we explore the performance of users on two collaborative tasks, the Map Task (Brown et al., 1984), and the Travel Game (Anderson et al., 1994). In the Map Task participants collaborate to complete a map route as accurately as possible. This work is described in the paper by O’Malley et al. (1996). Here we focus on simulated service encounters, involving a task called the Travel Game where the participants are engaged in planning an itinerary around the USA. Their goal is to visit as many destinations as possible given the restrictions imposed by available connecting flights. Both tasks produce objective measures of task success against which the corresponding task dialogues can be compared. We use an experimental paradigm when conducting evaluation studies, i.e. we systematically compare the performance of users who attempt such tasks when the system incorporates video technology or provides only an audio link. In addition we compare the performance of subjects who tackle the tasks when communicating face-to-face or with only an audio link. As well as calculating task success we also perform detailed analyses of the structure and content of the task dialogues. This involves examining the lengths of the dialogues, in turns and words, and examining how the process of turn-taking is managed. For the Travel Game we examine the decision-making process, and how frequently in the dialogue clients change their plans and decisions. We also examine the lengths of the dialogues in the different conditions and the turntaking behaviour of speakers. In addition we conduct a detailed post-task questionnaire on aspects of user satisfaction with the task, communication and technology. We report on three studies of the impact of VMC, comparing face-to-face and audio-only interactions with collaborations supported by VMC. We assess the impact of VMC on Travel Game collaborations with different sizes of video display and explore the effect of VMC on remote collaborations across the European Internet.
Background:
the benefits
of face-to-face
interaction
In earlier research on collaborative problem-solving, we had found that in face-toface interaction, participants needed to say significantly less to achieve the same 194
Interacting
with Computers
vol 8 no 2 (1996)
level of performance than in audio-only conditions (Boyle et al., 1994). This study is unusual in showing subtle but significant benefits of the availability of visual signals for collaborative problem-solving. Most earlier studies, which focused only on task outcome, showed no advantage for face-to-face problem-solving (Davis, 1971; Chapanis et al., 1972; Williams, 1977). Only in tasks involving conflict or negotiation was there some evidence of a benefit for communication with visual contact (Morley and Stephenson, 1969; 1970; Short, 1974). The task we used in the Boyle et al. (1994) study and in subsequent explorations of VMC is a form of collaborative problem-solving known as the Map Task (Brown et al., 1984). Two participants each have a copy of a schematic map. One subject, the instruction-giver, has a route shown on his or her copy of the map and his or her task is to describe this route so that the instruction-follower can draw this on his or her copy of the map. Some landmarks differ between the maps providing problem points which the participants have to overcome. The task produces an objective measure of task success - how well the route has been completed, and provides clear and comparable communicative goals for participants. In the Boyle et al. study 32 pairs of undergraduate subjects tackled versions of the Map Task sitting at opposite sides of a table, either communicating face-to-face or with a screen between them. In face-to-face dialogues, speakers used 28% fewer turns and 20% fewer words than in the audio-only condition. Yet face-to-face participants achieved as good levels of task performance with this reduced verbal input. The interaction was also managed more smoothly in face-to-face collaboration, with 8.7% of turns containing interruptions compared to 12% of turns in the audio-only condition. These face-to-face advantages suggest that speakers can use visual signals to supplement the information presented verbally and to assistin managing the process of turn-taking. From detailed analysis of the task dialogues we have found that in face-to-face communication speakers and listeners can use visual signals to check that the communication is progressing smoothly so they need to elicit and provide less verbal feedback from their conversational partner, (see Doherty-Sneddon, et al., forthcoming for details).
Study
1: Impact
of VMC
on travel
collaborations
In the studies described here we use a different collaborative task. The Travel Game was designed to be a form of collaborative problem-solving which involved a more social dimension than the simple exchange of information involved in the Map Task. It is thought that visual signals are useful in establishing social presence and have a greater impact on tasks involving negotiation. We thus expect that the extent to which clients felt at ease with the communicative situation would depend in part upon seeing their conversational partner. As well as including questions about such aspects in our task questionnaire, we also attempted to identify features of the task dialogues which might reflect such a difference. In the Travel Game clients have to make decisions about their travel plans based on the information they have obtained from the travel agent. To optimize their itineraries they might have to request a considerable amount of information from Anderson
et al.
195
the travel agent who has to consult files to check on available flight connections and timings for each such request. We considered that the extent to which the client felt comfortable in involving the travel agent in additional information searches would depend in part on the degree of social presence in the communicative setting. In face-to-face interaction we predicted the client would feel more relaxed about making additional requests for input from the travel agent. Where no visual contact was available to provide emotional or expressive cues we predicted that such requests would be less common. Design and subjects Undergraduate subjects are assigned the role of client. They have to communicate with the travel agent to access flight information required to plan their itinerary across the US. Their goal is to visit as many cities and states in the USA on an Airline Travel pass which has restrictions on lengths of stay in a single state and financial penalties for changing airline or backtracking to an earlier destination. They have an audio link, two shared screen facilities, one a map of the US, one a whiteboard where the travel decisions are logged. In half the tasks they also have a videolink showing the face of their conversational partner in a 3.5 x 4.5 inch window. In the comparison conditions participants either tackle a paper-andpencil version of the task while communicating face-to-face, or via an audio link between adjacent rooms. In the comparison paper-and-pencil conditions, both participants have their own copy of a map of the US showing the major airports. The travel agent has the timetable of flight connections, and she fills in the shared travel log which is visible to both. For the VMC conditions the Travel Game was presented on two 20 in colour monitors on SunSPARC 10 workstations running NV 3.2 (Frederick, 1994), and operating over a dedicated local area network. The video image refreshed at 4-5 frames/s. There was an audio link through microphones and headphones, providing full duplex sound. Figure 1 illustrates the Travel Game with videowindows. The travel agent was a member of the research team who is familiar with the multimedia communication tools, she was instructed to work to a loose script in her interactions with clients to ensure that she was equally helpful to all clients. The experimental hypotheses were not known to her. Her task was to provide information from standard files about the availability of flight connections and timings and to log the client’s itinerary on the shared screen or notebook. Task performances were judged by the number of cities and states visited in the session. All performances are recorded for transcription and analysis. Twenty participants took part in a comparison of face-to-face and audio-only conditions, where the information was presented on paper. The subjects were undergraduates at the University of Glasgow who volunteered to participate in exchange for the chance to compete for a cash prize of E20 for the best itinerary recorded in the Travel Game. Ten subjects were randomly assigned to each condition. A further 20 volunteers participated in the second part of the study, again each was randomly assigned to one condition, attempting the task in either video-mediated or audio-only conditions as described above. Minor changes to the rules of the Travel Game to ensure that it could be completed within 196
interacting with
Computers
~018
no 2 (1996)
Figure 1. Illustration
of Travel Gamescreendisplay
15 minutes were introduced between the two parts of this study. This means the dialogues were shorter for transcription and analysis, and in subsequent studies participants could attempt more than one version of the Travel Game task in a single session. We standardise for length in the cross-condition comparisons described below.
Procedure The subjects were given the following instructions for the Travel game: “You have a map of the USA to plan your 14-day trip. The itinerary which covers the most cities and states will win the prize money. You have $600 but you have spent $499 on an Atlantic Air Passwhich enables you to travel on any Atlantic Air Path. The Travel Agent has these travel details and the flight timetable and connections. You must spend at least 24 hours in any one state but you can travel within the state in that time. The travel agent will note your itinerary as you plan. There’s a surcharge of $50 if you backtrack to an airport you have already visited. There’s a surcharge of $50 if you wish to travel to a non-Atlantic airport. You have 20 (or 15) minutes. Good luck.”
Results of study 1 Table 1 shows a summary of the results of Studies 1,2 and 3. When we compared the task performance data, no significant differences emerged. Participants managed to visit just as many destinations in face-to-face Anderson
et al.
197
Table
1. Overview
Study
Measures Task performance
Study 1 Travel Game (Local) Face-to-face versus audio-only VMC versus audio-only Study 2 Travel Game (Local) VMC Large image VMC Small image audio-only Study 3 Travel Game (Remote) VMC versus audio-only
of studies
Dialogue length
Number of interruptions
No difference
Face-to-face significantly
No difference
No difference
No difference
Face-to-face more optional changes of plans No difference
No difference
No difference
No difference
No difference
No difference
No difference
No difference
No difference
shorter
No difference
Dialogue structure
and audio-only conditions. In the computer-presented Travel Game the itineraries did not differ when participants did or did not have a video link to the travel agent. Subjects achieved good levels of task outcome in all conditions. As in previous studies in the literature, problem-solving performance was generally unaffected by communicative context. We next examined the lengths of the dialogues in the different conditions. We found that when the client and travel agent communicated face-to-face, the travel agent used 22% fewer words than when they were using an audio link between rooms. Due to the changes made in the overall time limit of the task between the two parts of Study 1, separate analyses were carried out on the data from the face-toface/audio and computer-mediated conditions. In the comparison of face-to-face and audio-only dialogues, the Analysis of Variance on the number of words spoken showed a significant effect of role (travel agent versus client, Fl, 18 = 346.8, ~7< 0.0001). Travel agents said more with the mean number of words spoken by the agent 1495 compared to 596 words from the client. The main effect of context (face-to-face versus audio-only interactions) did not reach significance (F1,18 = 1.69, p = 0.2), although the means showed that on average there seemed to be more words spoken in audio-only dialogues with the mean of 1103 compared to 988 in face-to-face. There was a significant interaction between context and role, (F1,18 = 11.51, p < 0.005). Within this interaction, simple effects tests showed that the travel agent said significantly more in the audio-only than in the face-to-face condition, 198
Interacting
with Computers
vol8
no 2 (1996)
(F1,18 = 8.01, p < 0.01, mean words spoken equalled 1635 versus 1356). The length advantage found in face-to-face interaction on the Map Task by Boyle et al. (1994) has been replicated in another collaborative task. When the dialogues from the computer-mediated conditions were analysed, the Analysis of Variance again showed the significant effect of role (F1,18 = 88.9, p < O.OOl), with agents again saying more than clients, with mean-words spoken equalling 753 and 379 respectively. The computer-mediated conditions also showed the significant interaction of role and context (VMC versus audio-only, F1,18 = 5.001, p < 0.05). Within this interaction there was no simple effects of context (Fs < l), and indeed the mean number of words spoken by the travel agent was slightly higher in the VMC condition. There was no significant main effect of context, (F < 1) and the means were almost identical at 568 words in VMC and 564 words in the audio-only condition. VMC failed to deliver the length advantage of face-to-face interaction. To explore the structure and the content of the task dialogues, two types of analyses were conducted. First, turn-taking was examined by calculating the number and rates of interruptions in each dialogue. An analysis of rates of interruptions across the four conditions of the study showed a significant effect of mediated-communication, (F1,38 = 6.46, p < 0.01). On average speakers interrupted one another less often in the second part of the study where the task and communication were computer-mediated. There was no significant effect of visual signals but across the four conditions the rate of interruptions rose from 10.9% of turns containing an interruption in the plus videowindow condition, 12% in minus videowindow, 13.8% in face-to-face and 14.5% in audio-only. Post hoc t-tests showed a significant difference (p < 0.05) between the ends of this distribution. The combination of access to visual signals and the greater formality of mediated communication seem to combine to reduce the incidence of overlapping speech in VMC. We next wanted to explore the content of the clients’ contributions in the Travel Game. Our dialogue analysis aimed at exploring the impact of visual signals on the decision-making process. We assumed that in face-to-face communication the greater sense of shared social presence would lead clients to be more willing to engage the travel agent in extra searches and thus there would be more optional changes of travel plans as the clients sought to optimize their itineraries. To test this hypothesis we looked at the decision-making process in each dialogue. We coded all the changes to the planned itineraries made by clients, and coded these as optional or forced changes. Forced changes were those which resulted from the information provided by the travel agent, such as there being no available flight connection between two cities. Optional changes were those where the client decided on a new destination from choice for example, by revising an earlier part of their itinerary, asking for alternative possible destinations, using the surcharge option in the rules of the game to backtrack to a previous destination. All such optional changes involve more work from the travel agent in terms of additional information searches or alterations to the logged itinerary. When the number of such forced and optional decision changes were calculated, we found that there were significantly more optional changes in face-to-face than in audio-only interactions. On average there were 6.6 optional Anderson
et al.
199
changes per dialogue in face-to-face interactions compared to 2.7 in the audioonly condition, (t 1,18 = 8.17, p < 0.0001). No such difference emerged when we compared dialogues from videowindow and audio-only conditions, where the number of such optional changes was low, with on average 0.4 such optional changes in the VMC condition compared to 1.3 in the comparable audio-only condition. VMC again failed to deliver the same benefits to users as face-to-face interaction. Was this because users were dissatisfied with the technology or felt uncomfortable with this form of task presentation or communication? We explored such issues in a detailed post-task questionnaire given to all participants. We asked participants how easy it was to communicate with the travel agent, asking for a choice from ‘very easy’ to ‘very difficult’. The questionnaire showed that in face-to-face interactions 90% of subjects felt that it was very easy to communicate with the travel agent compared with 50% of audio-only participants. In the videowindow condition again only 50% of participants felt it was very easy to communicate but surprisingly this rose to 80% in the corresponding audio-only condition. Chi squared tests conducted on the questionnaire responses showed that the distribution of answers concerning easeof communication differed significantly between face-to-face and audio-only conditions, (X (1) = 3.8, p = 0.05), but did not differ between the VMC and Audioonly conditions in the second part of the study. How easy was it to make changes to the planned itinerary? In face-to-face communication 70% felt it was very easy compared to 20% in the audio-only condition. In the videowindow condition 30% felt it was very easy to make changes compared to 20% in the corresponding audio-only condition. Chi squared tests again revealed significantly different responses between face-to-face and audio conditions, (X (1) = 4.24, p < 0.05), but no significant differences between the VMC and audio-only conditions. We asked participants to rate their satisfaction with the final outcome of the consultation with the travel agent. In face-to-face interactions 100% reported that they were satisfied or very satisfied with the outcome, as did 90% in the audio-only condition. In the mediated conditions, user ratings were again high with 80% in both conditions satisfied or very satisfied with the outcome. We also asked participants to gauge if they would have found the task easier in the other condition i.e. face-to-face z~ersusremote and vice-versa. Only 10% of the face-to-face participants felt the consultation would have been improved in the audio-only condition but 90% of the audio-only felt things would have been easier face-to-face. None of the videowindow users would have preferred to do without the visual link and 80% felt face-to-face would have been an improvement. Only 30% of participants with no videowindow felt this addition would have been an improvement, but 60% would have preferred a face-to-face interaction. The questionnaire data provides some useful confirmation that the subjective impressions of users is largely in accord with the objective measures of task performance and communication. Although VMC users were satisfied with their overall performance, and often commented on the appeal of the new technology, they felt slightly less at ease in the communication process than those engaged in face-to-face interaction. VMC did not seem to generate the same sense of social 200
Interacting
with Computers
ml 8 no 2 (1996)
presence as face-to-face communication same technology without the addition
Study
2: Impact
and indeed showed of video signals.
of size of videowindow
no benefits over the
on collaboration
In Study 2 we attempted to increase the sense of social presence Travel Game by increasing the size of the videowindow.
in VMC in the
Design and subjects We composed two versions of the Travel Game featuring the west or east of the USA and participants tackled the two versions of the task in two-out-of-three conditions. Thirty-six undergraduate subjects at the University of Glasgow took part in this study. Again the volunteers competed for a cash prize of E20 for the best itinerary. The conditions were Small videowindows, where the image as in the previous study measured 3.5 x 4.5 inches, Large videowindows where the image measured 6.5 x 8 inches, and No videowindow. Subjects were randomly assigned to one of three conditions tackling two versions of the Travel Game either with Large and Small videowindows, Small and No videowindows or Large and No videowindows. All orders of presentation were counterbalanced. The task instructions were the same as those used in Study 1. Did increasing the video image of the conversational partner increase the sense of social presence ? Did this form of VMC afford the advantages of faceto-face interaction in terms of dialogue length or decision-making processes? Were users more satisfied with the communicative context with larger video images?
Results of Study 2 The results of Study 2 were rather disappointing. Increasing the size of the video image had no effect on task outcome, dialogue length or decision-making. Indeed no significant differences on any of these measures were found across the conditions of the experiment. Analyses of Variance were conducted on number of destinations visited, number of words spoken by travel agent and customer, and the number of optional changes to itineraries made by customers. The communicative context (Large, Small and No videowindows) did not produce any significant main effects or interactions, (all Fs < 1). The number of cities visited on average across Large, Small and No videowindow Conditions were 10.8, 11.6 and 11.6 respectively. The total numbers of words spoken across the three conditions, Large, Small and No videowindows were similar with means of 1005, 946 and 973 respectively. The contributions from the travel agent also showed little difference with means of 680, 649 and 660 for the Large, Small and No videowindow conditions. The Analysis revealed only the expected main effect of task role (Fl, 69 = 502.1, P < 0.001) with travel agents again producing more words than customers means 663 and 311, respectively. The number of optional changes of decisions made by clients was small and again similar across all three conditions with means of 1.75, 1.58 and 1.42. Anderson et al.
201
So not only did we not observe increased social presence, we again failed to find any noticeable advantage in the objective data for the addition of video signals to a collaborative task. In the subjective impressions of users, as revealed in the questionnaire responses, there were some hints of possible benefits of VMC in general and of large images in particular. When using large videowindows, 75% of users felt that it was very easy to make changes to their itineraries compared to 54% for small windows and 25% for no videowindows. None of these differences however reached statistical significance when tested by Chi squared tests. In two studies of VMC we have failed to find significant objectively measurable benefits for VMC, (see also the data reported in O’Malley et al. for a similar picture from a different collaborative task). We have found a number of benefits for the presences of visual cues in face-to-face communication, but these have not been replicated in VMC. The subjective impressions of users give some suggestions that there may be a preference for systems which incorporate videolinks between users even if these do not seem to have been translated into differences in task performance or communicative style. In our most recent study we explored a context where the possibly subtle benefits of VMC may prove of value: distance collaboration. We investigated the impact of VMC on task outcome, communication and user satisfaction, where users are aware that their conversational partner is several hundreds of miles away, in a different country.
Study 3: Impact
of VMC on remote collaborations
This study was designed to assess the impact of VMC on long-distance collaboration over the European Internet. The quality of video and audio links provided by the Internet is rather variable. The UK-Netherlands link is generally one of the best connections. In pilot tests and in the main study described here we found that the video signal rarely suffered break up of the image and the audio signal was clearly audible on almost all occasions.
Design and subjects Twelve undergraduates at the University of Glasgow participated as tourists in this Travel Game study for a case prize of BE20as before. Each tackled two versions of the Travel Game, one with and one without a video link. The order of presentation was counterbalanced. The Travel Agent was at the Technical University of Delft in the Netherlands, but operated to the same script as had been used in earlier studies. The students were informed that this was a long distance link between Glasgow and Delft, and the travel agent although a fluent English-speaker spoke with a noticeable Dutch accent. In pretask chat whilst setting up the task, the conversation clearly indicated that this was a genuine long-distance experiment. The same task instructions were used as in Studies 1 and 2. The communication link over the Internet used public domain network video and audio tools (nv, nt, and vat). The same SunSPARC stations were used as in Studies 1 and 2. 202
Interacting
with Computers
~018 no 2 (1996)
Results of Study 3 The first finding was that the Internet connection between the UK and the Netherlands was able to support this kind of collaboration. Of 24 experimental sessions, only one had to be abandoned and rerun due to technical difficulties with the network link. Reasonably good audio and video reception was obtained in the remaining sessions, with clear sound reception and little noticeable lag or break-up of video images. The results of the Travel Game showed no difference in the task outcome for video- or audio-link conditions. The mean number of cities visited in the VMC condition was 9.5 compared to 9.66 in the audio-only condition. T-tests showed these conditions did not differ significantly. Although this again shows no advantage for VMC on outcome measures it is encouraging in other respects. The level of task success was similar to that obtained in earlier studies. The necessary differences imposed by using network communication tools, such as the need to click the pointer in a designated area of the screen before speaking, and the performance of the Internet compared to a dedicated local area network, did not impair performance on the task. When we assessed the communication process in Study 3, we found that there was no difference in the length of the dialogues or in the number of changes of plan between the VMC and audio-only conditions. An Analysis of Variance conducted on the number of words spoken, revealed the expected effect of task role (Fl,ll = 35.9, p < 0.001) with the travel agent saying more than the customers -with mean number of words 372 and 211, respectively. There was no significant main effect or interaction with communicative context (Fs < 1). The average length of the dialogue was 600 words in VMC compared to 567 in audio-only, with the Travel Agent saying almost exactly the same number of words on average in both conditions: VMC 374, audio-only 370 (F < 1). The number of optional changes which customers made to their itineraries was similarly low in both VMC and Audio-only interactions, with a mean of 1 and 0.58, respectively, and a T-test showed these data did not differ significantly. The questionnaire data confirmed some subjective benefits for VMC. Although 83% of users found it very easy to communicate in both conditions, when users were asked about social presence, i.e., how aware they were of the travel agent 92% of responses in VMC were ‘fairly’ or ‘very aware’ compared to only 50% in the audio-only condition. A Chi squared test showed that the distribution of the responses to this question was significantly different in the VMC and audio-only conditions, (X (1) = 6.32, p < 0.05). When asked how often they were worried that they had lost contact with the travel agent, 50% of VMC were ‘rarely’ or ‘not at all worried’ about this compared to only 25% in the audio-only condition, and again a Chi squared test showed the responses different significantly between the VMC and Audio-only conditions, (X (1) = 5.0, p < 0.05). These data suggest that particularly in long-distance collaboration there may be some benefits for VMC in establishing and maintaining social presence.
Discussion VMC Anderson
potentially et al.
offers
us the benefits
of face-to-face
interaction
while
we 203
collaborate with a distant partner. We have tried to illustrate those benefits that accrue in face-to-face interaction and which VMC systems might be designed to emulate. We found that the visual signals in face-to-face conversation seem to be used in several ways. In face-to-face communication speakers need to say less to complete the task successfully. From previous research we believe this is because visual cues can be used to judge that communication is proceeding smoothly and hence less verbal feedback is needed. Visual cues in face-to-face conversations may be used to establish a sense of social copresence which makes us feel at ease with our conversational partner, and so feel able to ask for additional information or help in a shared task. Video-mediated technology has been designed to facilitate collaborative communication at a distance. To simulate such conditions we compared task performance and dialogues between participants in different rooms. In our travel agency simulations, VMC did not result in shorter dialogues, nor in encounters where the clients felt relaxed enough to make frequent optional changes to their itineraries. This pattern of results, where VMC does not deliver the same benefits as face-to-face interaction is similar to our findings from a different collaborative task, reported in O’Malley et al. (1996). Our small study of long-distance collaboration suggests that VMC may offer some advantages in this area by making the conversational partners more aware of one another and reassuring them that they remain in contact even across a remote Internet connection. Further analyses of these remote dialogues continues, to explore whether there are any differences in dialogue structure which reflect these subjective impressions. It remains to be seen whether the effects of VMC which we observed are transient and would diminish with increased experience of VMC. This factor may be important, as reports in the literature which claim added-value for VMC (e.g., Tang and Isaacs, 1993; Olson et al., 1994; Isaacs and Tang, 1994) either use fairly lengthy exposure or repeated exposure over time. What we have shown has implications for the development and use of videomediating technologies. The visual channel is useful for task efficiency even for a task where social factors are not a central component (compared to, for example, negotiation and bargaining tasks). The impact of the visual channel depends upon the medium through which it is transmitted. VMC does not necessarily bring the same efficiency benefits to communication as face-to-face interaction. We have been concerned with the impact on collaboration of video-mediated images of the conversational partner. The conclusions which we might draw from these studies are that the benefits of face-to-face communication are rather subtle and the forms of VMC which we have investigated to date do not seem able to replicate these advantages. This is, of course, only one possible use of video data in collaborative tasks. The possibility of sharing other forms of video data would almost certainly have a large impact on task performance and communication. In the Travel Game task we provided shared accessto relevant video data in the form of the shared map of the US, and the whiteboard where travel decisions were recorded. If such information had not been accessible to both participants we assume the task would have been considerably more difficult and required longer dialogues between tourist and travel agent. In future studies we intend to go 204
Interacting
zuith Computers
vol 8 no 2 (I 996)
further in making systematic comparisons of the impact upon users of relevant video data compared to video images of the conversational partner. A final point we wish to make is methodological. Monk et aI. (1994) propose that a full evaluation of communication technology requires measures of both task outcome and communicative process. This is the approach which we have adopted. Only by looking at both of these aspects in conjunction can we adequately evaluate the impact of VMC. By looking at task performance alone (as in many earlier studies) we would have concluded that there is no difference between video-mediated, face-to-face and audio-only communication. By looking only at measures of process, such as the number of turns per dialogue, we may have concluded that VMC is either equivalent to, or worse than audio-only communication. By looking at process and outcome together we see that communicative structure, verbal and non-verbal, differs across the contexts in ways which offer advantages and disadvantages to the efficiency of the interactions. The visual channel provided by VMC is useful to speakers, but the remoteness of the interaction invokes a change in communication style which offsets the benefits. Our studies show that here is no simple answer to whether VMC ‘works’ or offers advantages. In general, however, the pattern is a complex one. Successful collaboration depends upon a number of complementary processes: how the process of communication is managed, the level of task success achieved and the users’ satisfaction with all the aspects of these processes. VMC has been shown to influence each of these aspects of collaboration within our laboratory studies. While designers will often be concerned with the impact of VMC on users in the workplace using systems for more extended time periods than those we employed, the data we have presented suggests that such studies must also deploy evaluation methods which consider the multidimensional aspects of collaboration. For some commercial applications, however, the initial impact on customers of a new form of service provision is also important. If users’ first experience is unsatisfactory they may well seek out alternative, traditional forms of service. Although VMC users performed well on our task and generally expressed their satisfaction with the technology and the outcome, their subjective reports were not always consistent. Despite these positive signs, they also reported that they would travel for up to an hour to engage in a face-to-face interaction but would accept VMC if more effort than this was required. In any of our studies we would obtain a different answer to the question of whether VMC ‘works’ if we considered task performance, communication process or any aspect of user satisfaction, in isolation. The same dangers apply to studies on the introduction or long-term use of multimedia services in the workplace.
Acknowledgements The research reported in this paper was supported by the Research Council grant to the Human Communication Glasgow University. Study three was facilitated by support action, Project 5: Practical Assessment of Multimedia Wide Anderson
et al.
Economic and Social Research Centre at from the EC COST 14 Area CSCW. 205
References Boyle, E.A., Anderson, A.H. and Newlands, A. (1994) ‘The effects of eye contact on dialogue and performance in a co-operative problem solving task’ Languageand Speech, 37,1,1-20 Brown, G., Anderson, A.H., Yule, G. and Shillcock, R. (1984) Teaching TaIk Cambridge University Press Chapanis, A. (1971) ‘Preclude to 2001: explorations in human communications’ American Psychologist 26, 949-961 Chapanis, A. (1975) ‘Interactive human communication’ Scientijic American 232, 3&42 Chapanis, A., Ochsman, R.B., Parrish, R.N. and Weeks, G.D. (1972) ‘Studies in interactive communication: the effects of four communication modes on the behaviour of teams during co-operative problem-solving’ Human Factors 14,487-509 Davies, M.F. (1971) ‘Co-operative problem-solving: a follow-up study’ Report No. E/71252/ DVS Post Office Long Range Intelligence Division, Cambridge UK Doherty-Sneddon, G., Anderson, A.H., O’Malley, C., Langton, S., Garrod, S. and Bruce, (forthcoming) ‘Face-to-face communication and VMC: a comparison of dialogue structure and task performance’ 1. Experimental Psychol. Applied (under review) Isaacs, E.A. and Tang, J.C. (1994) ‘What video can and cannot do for collaboration: a case study’ Multimedia Systems 2, 63-73 Monk, A., McCarthy, J., Watts, L. and Daly-Jones, 0. (1994) ‘Measures of process’ in MacLeod, M. and Murray, D. (Eds) Evaluationfor CSCW Springer Verlag Morley, I.E. and Stephenson, G.M. (1969) ‘Interpersonal and interparty exchange: a laboratory simulation of an industrial negotiation at the plant level’ Brit. J. Psychol. 60, 543-545 O’Connaill, B., Whittaker, S. and Wilbur, S. (1993) ‘Conversations over videoconferences: an evaluation of videomediated interaction’ Human-Computer Interaction 8, 389428 O’Malley, C., Langton, S., Anderson, A., Doherty-Sneddon, G. and Bruce, V. (1996) ‘Comparison of face-to-face and video-mediated interaction’ Interacting with Computers 8,2,177-192 Olson, J.S., Olson, G.M. and Meader, D.K. (1994) ‘What mix of video and audio is useful for remote real-time work’ Proc. Workshop on VMC, CSCW 1994 (Chapel Hill, NC, USA) ACM Press Olson, G.M., Olson, J.S., Carter, M. and Storrosten, M. (1992) ‘Small group design meetings: an analysis of collaboration‘ Human-Computer Interacfion 8,389428 Sellen, A.J. (1992) ‘Speech patterns in video-mediated conversations’ Proc. CHI 1992 ACM Press, 49-59 Sellen, A.J. (in press) ‘Remote conversations: the effects of mediating talk with technology Human-Computer Interaction Short, J. (1974) ‘Effects of medium of communication on experimental negotiation’ Human Relations 27, 225-243 Short, J., Williams, E. and Christie, B. (1976) The Social Psychology of Telecommunications Wiley Strauss, S. and McGrath, J. (1994) ‘Does the medium matter? The interaction of task and technology on group performance and members reactions’ J. Appl. Psychol. 79, 87-97 Tang, J.C. and Isaacs, E.A. (1993) ‘Why do users like video?’ Studies of Multimedia Collaboration Computer-supported Co-operative Work 1, 163196 Williams, E. (1977) ‘Experimental comparisons of face-to-face and video-mediated communication’ Psychol. Bull. 84, 96%976 206
Interucting with Computers vol8 no 2 (1996)