Experience at IBM with group support systems: A field study

183

Experiences at IBM with Group Support Systems : A Field Study Jay NUNAMAKER . Doug VOGEL, Alan HEMINGER, and Ben MARTZ

Introduction

Department of Management Information Systems, College of Business and Public Administration, Unioersity of Arizona, Tucson, AZ 85721, USA

Research literature attention to Group Support Systems is increasing . Framework papers [e.g . . Huber (1984), DeSanctis and Gallupe (1987)] have been followed by reports of experimental studies [e .g ., Lewis (1982), Gallupe (1985), Watson (1987), Beauclair (1987), Zigurs (1987), A . Easton (1988), G . Easton (1988)] . Unfortunately, only minor at-

Ron GROHOWSKI and Chris MCGGFF IBM Corporation, Bethesda, MD 20817, USA Although numerous laboratory studies have been conducted, virtually no attention has been given to how well an operational Group Support System functions in a real-world, organizational setting . This paper presents the results of a Group Support System field study conducted at an IBM site . Data collected included session pre- and post-session questionnaires and facilitator observations plus followup interviews with managers and participants . Process and outcome effectiveness, efficiency, and user satisfaction were consistently higher for Group Support Systems compared to no automated support. Further, those who had used the automated system before consistently had a higher mean score on questions of process effectiveness . A comparison of man-hours expended resulted in a reported 56% savings attributable to Group Support System use . The overwhelmingly positive results of this field study contradict some laboratory experiment findings and support others. Directions for future field and experimental research to resolve apparent differences and provide further clarification are identified . Keywords : Group Support Systems, GDSS, Productivity, Effectiveness, Efficiency, Field Study Jay F. Nonsmoker, Jr., is Head of the Department of Management Information Systems and is a Professor of Management Information Systems (MIS) and Computer Science at the University of Arizona . He received a PhD from Case Institute of Technology in systems engineering and operations research . He was an Associate Professor of Computer Science and Industrial Administration at Purdue University . Dr. Nunamaker joined the faculty at the University of Arizona in 1974 to develop the MIS program . He has authored numerous papers on group decision support systems, the automation of software construction, performance evaluation of computer systems, decision support systems for systems analysis and design, and has lectured throughout Europe, Russia, Asia, and South America . Dr. Nonsmoker is Chairman of the Association for Computing Machinery (ACM) Curriculum Committee on Information Systems . North-Holland Decision Support Systems 5(1989)183-196

Douglas R. Vogel is an Assistant Professor of MIS. He has been involved with computers and computer systems in various capacities for over 20 years. He received his M .S . in Computer Science from U .C .L.A . in 1972 and his Ph .D. in MIS from the University of Minnesota in 1986 where he was also research coordinator for the MIS Research Center . His current research interests bridge the business and academic communities in addressing questions of the impact of management information systems on aspects of interpersonal communication, group decision making . and organizational productivity . Alan Heminger is currently Assistant Professor at Indiana University School of Business in the Department of operation and Systems Management . He received an MS in Counseling from California State University, Hayward, School of Educational Psychology, in 1978 . In 1988 he completed his PhD in MIS at the University of Arizona . His current research interests are in the area of Group Decision Support Systems (GDSS) . with particular interest in the assessment of their implementation in operational environments, and the problem of hindsight bias as it affects group work . Heminger was selected to attend the 1987 ICIS Doctoral Consortium . William Benjamin Mark, Jr. graduated from the College of William and Mary in 1981 with a B .B .A . In 1985, he received his M .S. from the University of Arizona and entered the PhD . program seeking a concentration in Management Information Systems . His expected date of completion is Spring, 1989 . His experiences cover the design and coding of software for work groups, the facilitation of work groups at Arizona's facilities, and the coordination of the implementation of the work group software at a manufacturing site . His current interests include Group Decision Support Systems ; Electronic Business Planning Tools; Small Group Theory in an Electronic Environment .

0167-9236/89/$3 .50 © 1989, Elsevier Science Publishers B.V . (North-Holland)



184

!. Nanamaker et al . / Experiences at IBM with Group Support Systems

tendon has been given to field experiences with Group Support Systems (e .g., Applegate (1986), Martz (1989), Heminger (1989)] and the literature that does exist tends to contradict the results of many experimental studies . This is indeed a problem if results of academic research are to be applied effectively in business settings . The challenge is to reflect upon field observations and integrate aspects of field and experimental research to achieve a more comprehensive understanding of the implications of real world application of Group Support Systems . This paper presents the results of a Group Support System field study conducted at an IBM manufacturing site having 6,000 employees . Data collected included pre- and post-session questionnaires and facilitator observations plus followup interviews with managers and participants on issues of process and outcome effectiveness, efficiency, and user satisfaction . A comparison of the actual number of man-hours expended with the number that had been anticipated also was made . Results of the study are compared with those of experimental studies addressing similar issues . Limitations are discussed and directions for future field and experimental research to resolve difference are identified .

Ronald Grohowski is the Director of Management Information Systems and Telecommunications at the IBM Systems Integration Division in Bethesda, Maryland . Mr . Grohowski has a BS in Mathematics from Wilkes College and an MS in Operations Research from Union College in New York . Ronald Grohowski is responsible for tactical and strategic planning, business control guidance, DP equipment planning and acquisition and telecommunications/networking activities . He is also responsible for the direction of over 1,000 1/S professionals and over 1,000 MIPS for 16,000 employees . Christopher McCca is an IBM Program Manager at the IBM Systems Integration Division in Bethesda, Maryland . He received a BS degree in 1980 from the University of Scranton and has done postgraduate studies in Ergonomics/ Human Factors at Texas Tech University . As a program manager he is involved in decision center information systems strategy and computer supported cooperative work . The systems integration division supports a population of 19,300 professionals ith 10 3090s, 26 308xs, 19 4381s, that provide 14 MVS systems and 19 VM systems.

Research Model The research model used in conjunction with this research is illustrated in fig . 1 . The model was based on extensive experience coupled with a review of relevant literature . The variables in the model are representative of those variables studied most often in past Group Decision Support System and computer-mediated communication research . Readers are directed to Nanamaker, Applegate, and Konsynski (1988) and Dennis et al . (1988) for additional background on the research model . As shown in fig . 1, the model addresses issues related to individual, group, project, and organizational levels of analysis that we feel are particularly relevant to Group Support System design and implementation . At the center of the model, the characteristics of the group, task, context, and technology are represented as influencing process and, together with that process, establishing the group work environment . The environment, in turn, drives group outcomes that then provide feedback to the organization's decision-making environment . A group's characteristics include its size and the composite of experience, cohesiveness, motivation, and history that constitutes group member attitudes and involvement . Task characteristics include task type, complexity, degree of

Task Group Process Houtcomes~" Context

Work Gp, Technology Environment i

IOrg . D ecision-making Envir onment Fig . 1 . The Research Model .



J . hunanwker et at. / Experiences at IBM

The purpose of this section is to outline the chronology of events and experiences that took place in conjunction with implementation of University of Arizona Group Support System software at an IBM site . Facility use overviews are provided . Data collection procedures are reviewed .

White Board

Facilitator Console and Network File Server

P

Barco Projector

Workstations r-

f Li

Fig . 2. Corporate Site Facility .

Systems

185

Site : Phase 1 Following visits by IBM corporate personnel to University of Arizona facilities that included "hands-on" use of the work group support tools to address a major corporate planning problem, company representatives expressed satisfaction with the process and decided to install similar facilities within the corporation for operational use on a daily basis . The site selected for this installation is an IBM manufacturing plant with approximately 6,000 employees, located in a rural setting in upstate New York . A room to house the Group Support System was remodelled according to the design of an operational facility at the

Implementation Chronology

Wall Mounted Projection Screen

Group Support

Grant Support: Phase 0 The University of Arizona was one of 13 schools awarded $2 million grants through IBM's Management of Information Systems (MOTS) program . A portion of this grant and other grants were used to develop software and facilities extending prior University of Arizona Group Support System activities . Experience with the software and facilities established a foundation from which the University of Arizona proceeded in encouraging Group Support System installation and evaluation in a corporate setting .

rationality . and clarity . Context characteristics include aspects of the organizational environment such as incentives, reward system, and organizational culture . Technology characteristics include hardware, software, and setting configuration . Process includes aspects of the procedures, anonymity, level of participation, facilitation, and interaction of group members that influence group outcomes such as satisfaction, quality of outcomes, time required to reach resolution, consensus, and decision confidence . This model provided the framework for Group Support System implementation and evaluation at an IBM installation . The study addressed the research question of how well an operational Group Support System functions in a real-world, organizational setting?

White Board

with



I. Nunamaker et al. / Experiences at IBM with Group Support Systems

186

Table I Chronology of Phase 1 Implementation Activities . Date

Activity

4/87 5/87 5/87 9/87 9/87 12/87

Initial site visit Facility construction consultation Prototype installation in temporary facility Formal Group Support System installation Facilitation training Operational evaluation

University of Arizona . The corporate site facility is illustrated in fig . 2. In the room, a U-shaped table is equipped with 10 networked microcomputers. An additional microcomputer attached to a large screen projection system is also on the network to permit display of work done at individual workstations or of aggregated information from the total group. An adjacent small room is designated for "backroom" functions such as printing out session results . The Phase 1 software implementation process, initiated in the Spring of 1987 with a site visit, was concluded in December of 1987 with a corporate evaluative report . Data collection using Phase 1 software commenced in January 1988 . A Phase 2 system implementation process was initiated in September 1988 . Table 1 presents a summary of Phase 1 implementation activities. Close communication and cooperation between IBM and the University of Arizona ensured that the implementation proceeded smoothly and established a strong foundation for extended facility use and evaluative efforts . The transfer of technology from an academic research environment to a corporate mainstream application was a new experience for both the University of Arizona and IBM groups involved in this implementation . Prior University of Arizona experience had been based on six academic implementations . The cooperation led to success in implementing the software at the initial site that has subsequently resulted in Group Support System implementation at six other sites . Further site expansion is envisioned in the near future . Facility Use

Problem solving groups from throughout the site were encouraged to use the facility. Groups most often heard of the facility through word-ofmouth from other groups . Group participants ranged from the plant manager and high level executives to shop floor personnel . Representa-

rives of several management levels were often included in the same group . Some groups were on-going entities while others were newly established . Group size ranged from 4 to 10 members with an average size of 8 . Larger groups tended to be new ones having been assigned a particular task and representing a wide variety of functional areas . Group members typically had detailed knowledge of one aspect of the problem area and general knowledge of the problem domain . As such, the knowledge domains of group members were distinct yet overlapping relative to the question at hand . Tasks addressed by the groups were for the most part of a planning and problem solving nature, e.g ., strategic planning process problems, factors contributing to cost overruns, and functional area data processing needs . Some were cross organizational involving many functional areas ; others represented many management/ employee layers within a particular functional area . The groups participating in sessions brought with them myriad problem domains, e.g ., requirements analysis, strategic planning, and resource allocation . The majority of tasks were complex to the extent that they required creativity and had no known "right" answer, particularly for the larger groups of size 8 to 10. Groups also tended to address tasks that were oriented towards evaluating a set of issues. A pre-planning meeting was typically held before each group session to align the best use of the tools with the task to be undertaken . The technology provided consists of three linked software tools . An Electronic Brainstorming tool supports idea generation, allowing group members to simultaneously and anonymously share comments on a specific question . An Issue Analyzer tool helps group members identify and consolidate key focus items resulting from idea generation . Support is also provided for integrating external information to support identified focus items . A Voting tool provides a variety of prioritizing methods including Likert scales, rank ordering, and multiple choice . All group members cast private ballots . Accumulated results are displayed, at which time action items and an agenda for a future session, if appropriate, are identified . A scenario demonstrating use of the tools is as follows : A manager responsible for improving shop floor control was having difficulties identifying problem

J . Nunamaker e1 a7. / Experiences at IBM with Group Support Systems

areas that were hindering the process . Those persons knowledgeable about each of the subfunctions seemed unable to isolate primary causes and identify potential solutions that could result in improved productivity . In fact, a two hour meeting of half a dozen of the key participants had resulted in a number of arguments and no solutions . The manager had previously elicited IS support but, without defined requirements, no progress had been made . The manager elected to try the Group Support System in an attempt to resolve issues and develop a plan of action including information system requirements to improve the shop floor control process . The manager met with the facilitator to express the objective she sought and to understand how the Group Support System might be used . It was mutually decided to use the Electronic Brainstorming tool with the question "What are the key issues in improving shop floor control?" followed by Issue Analysis and Voting . A session agenda and time was established for 10 of the key participants including the manager and two junior analysts assigned to investigate possibilities of information systems support . The Brainstorming session lasted for 35 minutes during which time the participants generated 645 lines of comments . Comments included issues, ideas, and clarifications as group members shared information . At the end of the brainstorming session, the manager reflected that for the first time she was able to get meaningful answers to questions associated with shop floor control issues. The analysts noted that they were beginning to better understand the interrelated nature of the overall shop floor control process . A 30 minute period of focus item identification followed by 45 minutes of issue consolidation and face-to-face discussion resulted in a generalized list of requirements for effective shop floor control improvement . Each group member prioritized the list in terms of importance to improved shop floor control and cast a private ballot using the Voting tool . The accumulated results were displayed to the group . After 10 minutes of discussion, the session was concluded with comments from the manager thanking the participants and directing the analysts to proceed towards the development of information systems support for the shop floor control process . The manager was given hardcopy of all of the session input .

187

Data Collection

A particularly important aspect of the installation and use of the Group Support System at the IBM site was collection of evaluative data from a broad range of end users addressing a variety of tasks . Data collection was accomplished by combining information contained in system log files with that collected from on-line pre- and post-session questionnaires filled out by group members . facilitator observations, and followup interviews with managers and participants on aspects of process and three key objectives : outcome effectiveness, efficiency, and user satisfaction . Session log file, questionnaire, and interview details are provided in the following section . Additional evaluative measures included the time required for completing the project, number of meetings required, the length of individual meetings, and the number of people in the group meetings as well as cost measures that were separated as administrative and man-hour costs . Actual man-hour data were aligned with anticipated time estimates based on leader experience and historical precedents . Initial test data were collected in December 1987 . Data collection commenced in January 1988 in conjunction with the pilot testing of the data collection process, the focus of this paper, representing the beginning of a multi-year study . To facilitate cross-site comparison, data continue to be collected at the IBM sites that are now beginning to use Group Support System software . Each site has a major focus (e.g ., customer partnerships, product development cycle) to further investigate the domain of Group Support System applicability as well as provide opportunities for replication and independent confirmation of findings at the initial site . Each site has its own group of facilitators and support staff .

Study Results The major thrust of this research was to assess the use of an operational Group Support System in an organizational setting . An important facet of this study was to go beyond user acceptance to ascertain reaction to the Group Support System process and outcome in terms of effectiveness, efficiency, and user satisfaction . This section will report the summarized results from sessions using the Group Support System.



188

J. Nanamaker et af. / Experiences at IB .M with Group Support Systems

Effectiveness

For purposes of this study, Group Support System effectiveness was comprised of two components: quality of session process and quality of outcome . One measure of the quality of session process is the degree to which the participants took part in the process and contributed to its outcome . The quality of outcome is a measure of the degree to which the system provided the product that the session initiator desired . Further indication of outcome quality comes from followup on how the results from the system actually were used . The quality of session process was measured through the use of log files, through which it is possible to learn the evenness of participation among the participants, and of information gathered from the participant post-session questionnaires, and follow-up interview forms . Analysis of log file data revealed that, compared with traditional group dynamics, Group Support System use tends to equalize participation . Results were independent of participant typing skills and familiarity with micro-computer technology . As an additional indication of Group Support System process effectiveness, participants were asked to

`R, of responses (n =441) strongly disagree

"It was very effective . Some of the reasons for this assessment are the ability to get all participants together, to capture their thoughts anonymously, the ability to use data for action, and the ability to step strong managers through the process without personalities getting in the way ." "It was very effective for dynamic problem solving, for generating ideas and for working with group dynamics." "We don't have another vehicle which could do what we did for the way we used the system. The brainstorming capability and the anonymity feature were particularly important ."

Table 2 Levels of Agreement about System Effectiveness . Post-session questionnaire statement

express their agreement with a number of statements about the system, using a five-point Likert scale. Table 2 presents participant responses . These responses indicate a strong agreement among the participants that the system did provide process effectiveness . Further, those who had used the automated system before consistently had a higher mean score on questions of process effectiveness (p < 0 .0001) . These responses indicate a significant increase in belief in the effectiveness of the Group Support System process for those who had used the system before over those who had not . In addition, the question "How would you rate the effectiveness of the automated process?" was asked in the followup interviews with participants . Comments included :

strongly agree

1

2

3

4

5

Mean

The computer-aided process is better than the manual process.

5 .5

3 .9

11 .4

27 .4

51 .8

4 .162

The computer-aided process helps the group generate ideas .

5 .0

3 .6

6 .8

24 .8

59 .8

4 .307

Effectiveness of the output of the Group Support System was addressed in the followup interviews with managers who were also participants . Comments made in response to the question : "What has happened as a result of the automated session(s)? (In general terms, without discussing data.)" included :

The computer-aided process helps the group identify key ideas .

4 .8

5 .2

9 .8

31 .4

48 .9

4 .143

The computer-aided process helps the group achieve its goals .

4 .8

5.2

9 .8

31 .4

48 .9

4 .143

The group's problem-solving process was fair .

"Our department and two others put our ideas together and established a mission . We have since implemented that mission ."

5 .5

2 .5

15 .8

36 .3

40 .0

4.027

"It resulted in moving to the next phase of the project."

"All sessions have led to follow-up meetings . We definitely have used the data that was generated."

J . Nunamaker et d. / Experiences at IBM with Group Support Systems

"The session was input to many different plans . The plans were then developed ."

Table 3 Comparison of Projected Manual and Automation Supported Man-hour Requirements .

"We got enough information to make a decision . ,

People in session 7 10

Comments in response to the question : "What has been done with the data that were generated during the session(s)?" included

"The data were distributed to all members of the department . Open items were displayed at all hi-monthly meetings until closed ." Efficiency

tot . avg.

204 .50 47 .50 240 .00

Manhours actual

Percent man-hours saved 52 .32 56 .84

132 .00 94 .00 220 .00 400 .00 88 .00 88 .00 20 .00

40.00 36.00 46 .00 89 .50 9850 43 .00 43 .50 4 .50

83 .33 72 .73 51 .06 59 .32 15 .38 51 .14 50 .57 77 .50

42 .00 88 .00 30 .00 41 .00 264 .00 36 .00 60 .00 42 .00

29 .00 43 .00 17 .70 30 .00 116 .00 16 .00 31 .50 29 .00

30 .95 51 .14 41 .00 26 .83 56 .06 55 .56 47 .50 30 .95

10 10 5 9 7 10 9 8

72 .00 110 .00 38 .00 153 .00 296 .00 192 .00 72 .00 62 .00 136 .00 100 .00 41 .00 192 .00

40 .50 18 .00 24 .00 31 .00 128 .00 62 .00 40 .50 35 .00 39 .50 44 .50 22 .50 62 .00

43 .75 83 .64 36 .84 79 .74 56 .76 67 .71 43 .75 43 .55 70 .96 55 .50 45 .12 67 .71

247 8

3601 .00 120 .03

1378 .70 45 .96

1665 .19 55 .51

9 10 5 7 9 6 9 8 7 4 6 9

"The data were presented to management. The rank-order charts and histograms were used in later meetings . The raw data were given to management for their review ."

Manhours projected

97 .50 20.50

10 10 10 10 10 9 8 6

"It was fed back to the participants for further work in all cases . It was kept as a book and select people worked with it to provide feedback to the group ."

The efficiency of the system as used in this study is an indication of the relative costs and benefits to the organization compared with doing the same function manually . It was not possible to run parallel sessions with control groups to measure efficiency directly . However, prior to use of the facility and without knowledge of automated support capabilities, each group leader was required to recommend and document a feasible project schedule for the accomplishment of his or her group's objectives, based on previous experience with similar projects . These schedules defined historical parameters for the projects and a baseline for comparison with the efficiency of the automated support . The plan provided by the group leader was then translated into an outline for use of the automated support tools . After completion of the project, expectations before use of the tools was compared with what actually occurred. Further, the output from the sessions was independently evaluated by a knowledgeable third party to gain a measure of what it would take to arrive at the same level of accomplishment using traditional manual processes . Overall, there is reason to believe that the results

189

Number of sessions : 29

gave at least a reasonable approximation of the estimated parameters, based on the years of experience of the various group initiators . As illustrated in table 3, man-hours were saved in every case recorded, with an average per session saving of 55 .51% . Percentages have been used to compensate for varying project lengths . A matched-pairs r-test was significant at a level p = 0.0001 . The data align with that recorded in the pilot test that preceded formal data collection . In the pilot test of 11 groups, an average man-hour savings of 61% was reported in conjunction with a 92% average calendar reduction in time required to complete a project . Results from data subse-



190

J. Nunamaker et at. / Experiences at IBM with Group Support Systems

quently collected from nine groups at a second IBM corporate site in a metropolitan setting reflected average man-hour savings of 55.6%. These results strongly confirm the robust nature of man-hour savings because the second site is primarily administratively oriented, as opposed to the manufacturing orientation of the initial site . The time savings reported are independent of particular site characteristics on a group level of analysis . High levels of performance in terms of manhour savings were strongly correlated with the degree to which the group's task was stated clearly and concisely (p = 0 .004) . Larger groups (i .e., size 8 to 10) tended slightly to outperform smaller groups relative to expectations . Man-hour savings were independent of the individual knowledge completeness of group members, ongoing nature of the group, degree to which cooperation was required, and composition relative to number of management levels or different departments represented . However, more formal, more recently established, and less cohesive groups tended to achieve higher levels of man-hour savings relative to expectations from similar groups that met without benefit of automated support . These groups also tended to be larger . Participants were also asked, first in post-session questionnaires and later in follow-up interviews, to provide data on system efficiency . In the post-session questionnaire, one of the questions asked the participants to indicate their levels of agreement with a statement about system efficiency . Results are indicated in table 4 . Seventytwo percent of the participants regarded the process to be efficient . Followup interviews with managers who were also participants provided additional evidence of the efficiency of the Group Support System . In

Table 4 Levels of Agreement about System Efficiency. Post-session questionnaire statement

The groups problem solving process was efficient

% of responses (n = 387) strongly disagree

strongly agree

1

2

3

4

5

Mean

4 .8

5 .0

18.6

34 .2

37 .4

3 .946

response to the question : "How would you compare the automated process with similar manual processes?" comments from these managers included : "The manual process takes approximately 3 to 4 times longer ." "We could not have done the process we did manually within a similar time frame ." Additional responses to the question : "Looking back on the automated group work session(s), what stands out most in your mind?" included : "A lot of ideas were generated . A lot was accomplished in a short period of time . It provided a very organized way to do things ." "The speed of the process . The way that ideas got flowing and how those ideas triggered other ideas ." "The volume of information generated in the short amount of time ." User Satisfaction

User satisfaction was evaluated in three ways . First, utilization rates of the Group Support System were maintained as an indication of general user acceptance and satisfaction . . Second, the post-session questionnaire provided self reports of user satisfaction . Finally, interviews with 17 users of the automated system were conducted to obtain a broader range of personal impressions . Each of these measures will be reviewed in turn . One of the strongest measures of this criterion is the utilization rate of the system . Since it was opened in October 1987, the room has been fully utilized, according to the records of those responsible for facilitating sessions . Currently, there is a three week waiting period for use of the room . In fact, the inability to get into the room as soon as desired has been mentioned by some users as a problem with using the system . A further measure of the acceptance of the system by the organization has been the decision to install automated group rooms at additional company sites . Thus, user acceptance, which can be considered a measure of satisfaction, has been demonstrated . The post-session questionnaire provided self reports of user satisfaction . Two questions were



J. Nunamaker et al. ; Experiences at /BM with Group Support .Systemc

characteristics were maintained constant throughout the evaluation . These results contradict some laboratory experiment findings and support others, as we shall see in the following section.

Table 5 Post-session questionnaire statements

I am satisfied with the computeraided process . The groups problem solving process was satisfying.

191

T of responses (n = 387) strongly disagree

strongly agree --i

1

2

4

5

Mean

5 .7

3.6

31 .2

47 .4

4 .109

Comparison with Laboratory Experiments

5 .3

3 .7

12 .1

15 .8

34 .1 41 .2

4 .023

specifically directed at this area of interest . The questions and some response data follow . (see Table 5 .) Some of the followup interview comments to the question : "Looking back on the automated group work session(s), what stands out most in your mind?" provide insight into the reported levels of satisfaction . Comments included : "The synergism of the group . The ability to record data . And the preciseness of the process procedures ." "The way that the room knocked down barriers. It is a good vehicle for communicating and good for brainstorming . As a result, the decision room assisted in developing a rational outcome." "Its advantages are that it makes people freer to give ideas and discuss them openly. The participants were less apprehensive than in manual meetings ." "The best thing for me about the automated session was the way that it allowed people from many areas to mesh a plan ." "The openness of the process and its lack of intimidation . This was because of the anonymity of the process ." Referring again to the research model illustrated in fig . 1, it has been shown that we gathered data on Group Support System process and outcome effectiveness, efficiency, and user satisfaction under varying group, task, and context characteristics . The most striking conclusion is the overall positive nature of the results independent of these characteristics. Technological

Key points in this study are related to process and outcome effectiveness- efficiency, and user satisfaction. This section will compare field study results with results from laboratory experiments selected to have been conducted under conditions as similar as possible to the field conditions described. Although comparisons are adversely affected by wide variations in group, task, and technology characteristics as well as measures employed and degree of control, some issues that should influence future research seem to be emerging . Effectiveness

Experimental studies of Group Support System effectiveness have had mixed results . In terms of participation, Lewis (1982), Zigurs (1987), and G . Easton (1988) found that participation was positively influenced through use of a Group Support System . Beauclair (1987) and Gallupe (1985) found no effect. Ruble (1984), Beauclair (1987), and G . Easton (1988) found no difference in decision quality comparing automated support to control groups . Steeb and Johnston (1981) and Easton (1988), however, found that decision quality was improved through use of a Group Support System . Gallupe (1985) may have found the key to some of these differences, noting that quality was particularly improved for more complex problems . This suggests that sacrificing task complexity in order to speed task solution or enhance quantitative capability in experimental settings may destroy the distinctive competence of automated support, particularly in situations where the group size is small and participant knowledge domains are similar. Like some experimental studies, ours found participation to be much more evenly distributed than normally occurs in manual sessions . Unlike the experimental studies, the group tasks in our study were by nature complex and required the coordinated input of many people for solution .



192

J. Nunamaker et al. / Experiences at IBM with Group Support Systems

Time required to sufficiently address all issues averaged 3 .83 hours . System effectiveness was measured in terms of perceived quality of the process and output . Post-session questionnaire responses were consistently in strong agreement that the computer-aided process was better than manual processes, thereby confirming, in part, Gallupe's (1988) conclusion that, in particular, quality was improved for more complex problems . Comments in response to the question : "What do you consider to be advantages of using the automated system?" included : "I see four main advantages to the automated system . These are (1) the structuring that the system imposes on the process, (2) the anonymity which allows so much open participation, (3) the way that personalities are taken out of the process so that the process becomes more rational and (4) the amount of data which is automatically captured ." "The documentation of the session is very useful . This allows immediate access to the data . It is also an advantage to be able to bring data from outside the session . In one session, we brought in the data from two other meetings and this moved the meeting right along ." Efficiency

Experimental studies have consistently noted that use of automated support either makes no difference in terms of time [e .g., Gallupe (1985), Beauclair (1987)] or tends to take longer than manual processes [e .g . . Steeb and Johnston (1981), G. Easton (1988)] . Reasons for these results may have been the difficulty of comparing manual and automated processes as well as a focus on smaller group sizes . Automated support by nature is structured . Control groups may or may not follow structured techniques. Unstructured control groups have been reported to work faster than groups receiving manual support [e.g., Watson (1987), Easton (1988)] in small group settings (i .e., size 3 or 4), albeit often with poorer quality output. Easton, however, in a comparison of groups that were unstructured, groups that received manually structured support, and groups that received technologically structured support reported that

the groups that had automated structure completed the task more quickly than manually structured groups . In our field study, efficiency is documented through a combination of time savings based on experience, post-session questionnaire data, and comments from the followup interviews . In no case did an automated session take longer than a comparable manual session . An average man-hour savings of over 55 percent was reported, independent of site . Questionnaire data expressed strong agreement that the process was efficient . Followup interviews consistently commended the ability of the system to focus thoughts and save time . For example : "Time saving is absolutely the biggest advantage of the system . The confidence in the inputs is twice as high with the automated system. The comments increased in intensity as the situation went on . It was a good learning experience for the employees." Thus, there is partial agreement between experimental and field results when differences resulting from the introduction of structure are equalized . Group size, though, may be an overriding consideration in time savings . The efficiency of automated support becomes increasingly apparent as group size rises . Based on observation [e .g., Vogel et al. (1987)], groups of size 8 or more tend to benefit more than groups of size 3 or 4, consistent with our field study findings . User Satisfaction

Experimental studies have expressed mixed results regarding user satisfaction . Some studies have reported high levels of satisfaction [e.g ., Steeb and Johnston (1981), Applegate (1985), Faston (1988)] with Group Support Systems . Others have found no difference in satisfaction levels that were attributed to the presence or absence of automated support [e.g ., Beauclair (1987), G. Easton (1988)]. Yet others have reported dissatisfaction with automated support processes [e.g ., Gallupe (1985), Watson (1987)] . Several explanations are possible . First, it is extremely difficult to measure differences in satisfaction unless groups have experienced both manual and automated support for equivalent tasks . In many controlled studies, subjects experiencing the automated support have no

J . Nunamaker et al . / Experiences at IBM with Group Support Sustems

direct basis for comparison with an unsupported process . Second, dissatisfaction with the technology in general may be a contributing factor . Many Group Support Systems have been technologically unsophisticated relative to contemporary systems . Finally, experimental subjects rarely have a vested interest in the outcome of comparison studies and therefore are likely to be less enthusiastic than field study groups. In our study, user satisfaction with the system continues to be confirmed as ongoing use of the room is accompanied by positive user responses on both exit questionnaires and followup interviews . As noted earlier, since use of the system began in October 1987, The room has been fully utilized and there currently is a three-week waiting period for scheduling sessions . Additional sites are being implemented . User questionnaire data reflected strong satisfaction with the computer-aided process and the group problem solving process in general . These findings are especially relevant, given the maturity of the session participants and their familiarity with an equivalent manual process . Finally, followup interview comments on advantages of use of the automated system included : "1 am less apprehensive about commenting when using the decision room . It is an advantage to have all input captured . I like the quick response of the system ." "The system brings out more honest ideas without the emotional content . It removes the emotional side of the process ." "Also the anonymity of the input allows participants to change positions on an issue without embarrassment ." Overall, this field study addressed usefulness and acceptance of the University of Arizona Group Support System software . Interestingly, results were independent of typing and micro-computer skills as well as of the number of sessions (both manual and automated) that users had participated in . Thus the findings seem reasonably robust within the site studied . User acceptance of the system at an organizational level has also been demonstrated by the corporate decision to install this system at additional sites . Thus, at both site and organizational levels there has developed an acceptance of the automated system for group

193

collaborative work . No study or system . however, is without limitations and opportunities for improvement .

Limitations and Future Research The limitations pointed out and suggestions made in this section will be focused on opportunities for additional clarification of the impact of Group Support System technology based on the model illustrated in fig . 1 . An important aspect of the use of the tools across multiple sites will be evaluation of the impact and implications of extended Group Support System functionality across a wider variety of groups and tasks . Group Although several hundred groups have used University of Arizona and IBM facilities, we have only scratched the surface in terms of understanding the implications of varying group characteristics on the process and outcome effectiveness . efficiency, and satisfaction resulting from use of Group Support Systems . Pre-planning data relevant to evaluation of Group Support System impact includes project record keeping detail, e .g ., group name, leader, and number of participants . Background information includes prior use of the facilities and corresponding tasks as well as history of the group in terms of activities and meeting history . Data captured at the time of the session should include participant name, title, and other information necessary to generate a session participant roster and coordinate formation of a session database . Yet to be addressed are many broader questions of the effect of group and individual characteristics on Group Support System process and outcome . These can only be answered after longitudinal analysis of data . Task Many issues relating to the impact of the nature of the task on Group Support System process and outcome remain to be addressed . Task dimensions explored to date do not even approach the boundaries of the application domain -- indicating that the successful application of Group Support Systems seems limited only by the imagination of the users . Accumulation of pre-planning information assists in capturing task variety since task

J. Nunamaker et at. / Experiences at IBM with Group Support Systems

194

information includes the goal or objective of the project and type of task(s) envisioned as well as estimated number of sessions . Yet to be explored, however, are numerous corporate task environments extending vertically within a functional areas well as horizontally across functional areas that require particularly heavy integration of information between sessions and across groups . Furthermore, data to be stored in Issue Analysis "side files" and extrapolated on-line should be identified to facilitate participant access during the session . The overall purpose of this information is to help the facilitator develop an appropriate agenda of tool use for the group and to track system use at the project level and organizational levels, extending the record beyond a single session or small number of related sessions . Context

Use at multiple sites presents the opportunity to evaluate Group Support System impact under widely varying site characteristics that can be identified positively as influencing successful use of Group Support Systems . Such characteristics of the site as number of employees, location, overall mission) should be recorded . Site tasks, processes, and use of manual tools or techniques should be documented . Kinds of groups and related site experience that might influence Group Support System acceptance should be captured . Decision making processes as well as control functions should be noted. Aspects of decision room management (e .g., executive sponsor, operating sponsor, facilitator background) should be recorded . Failure to record site data prior to Group Support System installation has the potential to create data analysis confusion and confounding effects . Technology Limitations

Followup interviews with managers who were also participants included the question : "What do you consider to be disadvantages of using the automated system?" Comments included : "The lack of an agreement to follow up and put the system in place . The action plan needed other organizational involvement . That hasn't been pursued yet . Getting the right representatives into the room is crucial . More stations would have helped . We could have used 20 stations ."

In the follow-up interviews, each of the interviewed participants was also asked for recommendations for improving the Group Support System . Their responses to the question : "What changes would you like to see made in the automated system? Why?" included : "I would like to see Issue Analysis made easier to use . It is clumsy . Also, I didn't like that in EBS I sometimes did not see all the files that were in the system. It would be very helpful to be able to link comments to comments ." "I would like to see the system on line (in my office) as well as in the decision room environment ." "I would like to see the system handle more participants . Often, ten people are not enough to do what needs to be done . I would also like to have the underlying comments made available during the ranking process." IBM and the University of Arizona have taken actions to address these suggestions . Additional facilities are being developed to increase the number of workstations. System functionality has been extended to permit group members to work on group questions in their own offices with less need to meet face-to-face . This means that better use can be made of decision room time . Brainstorming comments have been numbered for reference purposes. A search feature has been added to the Issue Analyzer to assist in organizing information . Capability to review comments during the ranking process has been added. Additional tools support more in-depth exploration of issues and provide more support for multi-criteria decision making situations. Extended capacity for integrating information across sessions and between groups has been developed, including support for generation of action items to drive future session agendas . These extended capabilities provide opportunities to further our understanding of the implications of the impact of technology on various aspects of Group Support System process and outcome . Extended tool functionality will allow us automatically to record tool usage statistics as well as to examine the benefits of integrating information across sessions and between groups . Data provided through this capability complement those

J. Nunamaker et al . / Experiences or IBM with Group Support Systems

gathered from the on-line questionnaires currently filled out by participants at the conclusion of each session to capture individual reactions and those gleaned from facilitator and follow-up interviews with randomly selected managers and participants . Data collection is currently underway from similar groups established to perform equivalent tasks, with the use or non-use of the Group Support System the variable used as a means for comparison . The overall goal is to collect necessary and sufficient data while minimizing the impact on expenditure of corporate personnel time .

Process Numerous opportunities exist to capture and make use of session process information. Log-in information recorded by participants and the facilitator provides a link between the pre-planning session and future sessions . Additional information captured by the system during the session includes type and duration of tool use as well as summary participation statistics . Pop-up screens are available to the facilitator to record observations and key events during the group session that might provide Group Support System evaluation insights as well as action items and task assignments for future sessions . At the completion of the session, the facilitator is prompted to record his/her reactions and observations, including impressions of group dynamics (e .g ., how well the group worked together) and ease of facilitation . Additional questions address task complexity and accomplishment of objectives . Reflections on the suitability of the technology to the task and the group are also recorded . These are very rich sources of data to assist both in the evaluation process and in consideration of suggestions for system modification and/or procedural improvement to better meet end user and organizational needs . Additional requested data on session "key words" and "wrap-up" information are particularly relevant to integrating information across sessions and between groups involved in larger projects . Follow-up interviews are particularly important in evaluating the longer term impacts of the system and associated output . Feedback from managers and session participants can provide insight into salient determinants of success as well as the ultimate usefulness and disposition of session output.

195

Conclusion This field study has documented Group Support System process and outcome effectiveness, efficiency, and user satisfaction at an IBM site . The corporate decision to develop additional sites underscores user and corporate acceptance and satisfaction with the use of Group Support Systems . The overwhelmingly positive results of this field study contradict some laboratory experiment findings and support others . Additional research is warranted to expand field observations and integrate aspects of field and experimental research in order observations and integrate aspects of field and experimental research in order to achieve a more comprehensive understanding of the implications for organizations of the adoption of Group Support Systems . This paper is directed towards that end .

References Applegate, L_ "Idea Management in Organization Planning ." Unpublished Doctoral Dissertation, University of Arizona, 1986 . Beauclair, R ., "An Experimental Study of the Effects of Group Decision Support System Process Support Applications on Small Group Decision Making," Unpublished Doctoral Dissertation, Indiana University, 1987 . Dennis, A ., George, J., Jessup, L., Nunamaker, l ., and Vogel, D ., "Information Technology to Support Electronic Meetings," MIS Quarterly, December 1988. Dennis, A ., Heminger, A ., Nunamaker, l ., and vogei, D ., "Bringing GDSS to Corporate Planning: the Burr-Brown Experience;" University of Arizona Working Paper, 1988 . DeSanctis, G . and Gallupe, B ., "A Foundation for the Study of Group Decision Support Systems," Management Science, 33 (5), May, 1987, pp . 589-609 . Easton, A., "An Experimental Investigation of Automated versus Manual Support for Stakeholder Identification and Assumption Surfacing in Small Groups," Unpublished Doctoral Dissertation, University of Arizona, 1988 . Easton, G ., "Group Decision Support System versus Face-toFace Communication for Collaborative Group Work : An Experimental Investigation," Unpublished Doctoral Dissertation. University of Arizona, 1988 . Gallupe, B ., "The Impact of Task Difficulty on the Use of a Group Decision Support System", Unpublished Doctoral Dissertation, University of Minnesota, 1985 . Heminger, A ., "Assessment of a Group Decision Support System in a Field Setting," Unpublished Doctoral Dissertation, University of Arizona, 1989 . Huber, G ., "Issues in the Design of Group Decision Support Systems," MIS Quarterly, September, 1984 .



196

J. Nunamaker et al. / Experiences at IBM with Group Support Systems

Lewis, F., "Facilitator: A Microcomputer Decision Support Systems for Small Groups," Unpublished Doctoral Dissertation, University of Louisville, 1982 . Martz, B ., "Information Systems Infrastructure for Manufacturing Planning Systems," University of Arizona, Unpublished doctoral dissertation, 1989 . Nunamaker, J ., Applegate, L ., and Konsynski, B ., "ComputerAided Deliberation : Model Management and Group Decision Support," Journal of Operations Research, Nov : Dec., 1988 . Nunamaker, J ., Vogel, D., and Konsynski, B ., "Interaction of Task and Technology to Support Large Groups," Decision Support Systems, forthcoming . Steeb, R . and Johnston, SC., "A Computer-Based Interactive

System for Group Decision Making," IEEE Transactions on Systems, Man, and Cybernetics, August, 1981, pp . 544552 . Vogel, D. and Nunamaker , J., "Health Service Group Use of Automated Planning Support" Administrative Radiology, September. 1988 . Vogel, D ., Nunamaker, J ., Applegate, L. and Konsynski, B ., "Group Decision Support Systems : Determinants of Success," Proceedings of the 7th International Conference on Decision Support Systems, June 8-11, 1987 . Zigurs, I ., "The Effect of Computer Based Support on Influence Attempts and Patterns in Small Group DecisionMaking," Unpublished Doctoral Dissertation . University of Minnesota, 1987 .