- Email: [email protected]
Participatory design of a collaborative clinical trial protocol writing system
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
journal homepage: www.intl.elsevierhealth.com/journals/ijmi
Participatory design of a collaborative clinical trial protocol writing system Chunhua Weng a,∗ , David W. McDonald b , Dana Sparks c , Jason McCoy d , John H. Gennari e a
Center of Pathology and Oncology Informatics, University of Pittsburgh, PA, United States Information School, University of Washington, Seattle, United States c The Southwest Oncology Group, United States d Cancer Research and Biostatistics, Seattle, WA, United States e Department of Medical Education and Biomedical Informatics, University of Washington, Seattle, WA, United States b
a r t i c l e
i n f o
a b s t r a c t
Article history:
Objective: To explore concrete approaches to socio-technical design of collaborative health-
Received 5 March 2006
care information systems and to design a groupware technology for collaborative clinical
Accepted 11 May 2006
trial protocol writing. Method: We conducted “quick and dirty ethnography” through semi-structured interviews, observational studies, and work artifacts analysis to understand the group work for protocol
Keywords:
development. We used participatory design through evolutionary prototyping to explore the
Participatory design
feature space of a collaborative writing system. Our design strategies include role-based user
Quick and dirty ethnography
advocacy, formative evaluation, and change management.
Clinical trial protocol
Results: Quick and dirty ethnography helped us efficiently understand relevant work prac-
Socio-technical approach
tice, and participatory design helped us engage users into design and bring out their tacit work knowledge. Our approach that intertwined both techniques helped achieve a “workinformed and user-oriented” design. This research leads to a collaborative writing system that supports in situ communication, group awareness, and effective work progress tracking. The usability evaluation results have been satisfactory. The system design is being transferred to an organizational tool for daily use. © 2006 Elsevier Ireland Ltd. All rights reserved.
1.
Introduction
Many researchers have realized the importance of social factors and organizational issues for system design [1–3]. However, concrete methods that account for social factors in system design are largely unavailable or have insufficient guidelines. Participatory design (PD) is one approach to socio-technical designs [4]. It is a proactive design method that explicitly advocates active user participation throughout
∗
the design process. Unfortunately, PD has not been widely adopted in healthcare information system designs [5]. The major challenges involved in PD include long prototyping design cycles, difficulties to carry out representative participant selection, and changing or conflicting user inputs. PD without sufficient considerations of work practice may abuse user opinions and generate a perfect technical solution for a wrong workflow. Therefore, our research has two goals: one is to explore the design space of a collaborative clinical trial
Corresponding author. Tel.: +1 412 513 5340; fax: +1 412 647 5380. E-mail addresses: [email protected] (C. Weng), [email protected] (D.W. McDonald), [email protected] (D. Sparks), [email protected] (J. McCoy), [email protected] (J.H. Gennari). 1386-5056/$ – see front matter © 2006 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.ijmedinf.2006.05.035
S246
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
protocol writing system, and the other is to enrich the participatory design theory by trying to answer these four research questions: 1. How can we make ethnographic studies complement participatory design methods? 2. How can we best apply participatory design to the design of collaborative information systems in this healthcare setting? 3. Who should be chosen to best represent users in a participatory design of a groupware technology? 4. What are effective strategies or incentives for active user participation? Our research setting is The Southwest Oncology Group (SWOG), one of a number of cooperative programs that develop cancer clinical trial protocols under the direction of National Cancer Institute (NCI). SWOG members include approximately 4000 leading clinical experts, statisticians, protocol editors, and other clinical trial workers across the country. Each SWOG clinical trial protocol takes 4–9 months to author and the resulting protocol ranges from 60 to more than 100 pages. A combined use of Microsoft Word and email systems is inefficient and error-prone for the collaborative protocol writing processes. Prior to our efforts, SWOG had tried different systems to support their protocol authoring process. However, none of those systems were adopted. One possible reason is that those systems failed to address the group work challenges. Our research aims to fill this gap. This paper focuses on our participatory design experiences at SWOG. In the rest of this paper, we first elaborate on our design methodology, and then we present how it led to a usable system design. We conclude with a discussion about the effectiveness and generalizability of our research methodology.
2.
Design methodology
To design the right system in the right way in healthcare settings, it is important to understand both the work context and user preferences. In this research, to achieve a design grounded in work and users, we intertwined “quick and dirty ethnography” [6,7] and participatory design as Fig. 1 shows below. Based on this model, an iterative design process consists of three components: (1) quick and dirty ethnography assembles instances of working roles and their work activities, and sensitizes designers with selected focus of the work practices; (2) participatory design engages role-based user
advocates, who understand users and are able to articulate users needs to designers, in carrying out system design through evolutionary prototyping; (3) formative evaluations are carried out through evaluative ethnography to understand the interaction between user advocates and the prototypes, and also to provide feedback for ethnography and participatory design. Evaluation results drive further field ethnography or help adjust subsequent participatory designs, and then improve the prototypes. In the conventional social–technical design model, field studies and system designs are widely apart, and field studies are mostly design oriented. With this method, there has been a better synergy between field studies and participatory designs. As Fig. 1 shows, the prototype serves two roles: one as an evolving representation of the design ideas and the other as a probing tool to deepen the field knowledge. Change management has been carried out in PD sessions through discussions with participants about the design feasibility for improving work experience, the possible changes to the current work practice, and the tradeoff between benefits and costs of embracing new changes. The research has received a human subject approval from University of Washington at Seattle. The details for quick and dirty ethnography, participatory design, and evaluative ethnography in the above model are provided below.
2.1.
Quick and dirty ethnography
To understand the current work practice, we carried out ethnography using the practical “quick and dirty ethnography” strategy developed by Hughes et al. [6]. Quick and dirty ethnography is different from conventional ethnography in that it does not provide an exhaustive description of the whole work context; instead it strategically focuses on selected aspects of the work and provides relevant work details to designers in a short time period [7]. In this research, we are particularly interested in group communication, information sharing and integration, and the workflow of protocol writing. The following open-ended research questions are the foci of our ethnography: 1. What are the major steps in the protocol writing process? 2. Who is involved in each step and what are their job responsibilities? 3. How is group communication carried out? 4. What is the most challenging part of the writing? 5. How do protocol writers coordinate the group work? 6. What is the protocol review and revision process like? Our ethnographic methods include semi-structured interviews with SWOG protocol writers, observation of protocol review committee (PRC) meetings, and protocol development artifact collection and analysis. These approaches leveraged information gained from one other.
2.1.1.
Fig. 1 – An iterative grounded design.
Observational study
Over a period of about 6 months, we observed PRC meetings. At the time of our study, protocols were typically reviewed twice in PRC meetings. Each meeting reviews an average of two protocols, and meetings occurred about once a week. Meetings
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
were attended primarily by statisticians, although one protocol coordinator manager from San Antonio also attended via conference call. The first and the last author of this paper participated and collected notes from the meetings, as well as the official PRC notes that documented the suggested revisions and discussion points around a particular protocol. We observed where in protocols the major questions or discussion occur, how reviewers respond to other reviewers’ comments, how reviewers have common ideas or different opinions, and what review comments are typical. We collected notes from about 30 different protocols for later content analysis.
2.1.2.
iterate protocol document revisions based on suggestions in comments from experts. Therefore, effective support for group awareness [11], in situ communication, and comments incorporation is greatly needed by protocol writers. Interestingly, we found that the process diagrams created by different protocol writers were not exactly the same. Protocol writers have disparate views of the work process and tool support needs. This finding alludes to the importance of involving representative participants for each writing role in the design so that we could pattern their shared needs while coordinating their individual preference in a single system. These results effectively informed our participatory design as described below.
Artifact content analysis
The artifacts that we have collected and analyzed include (1) about 80 selective protocol development emails, which were forwarded to us by protocol writers; (2) organizational protocol development guidelines or standards; and (3) around 1140 protocol review comments for 32 protocols. From these comments, we used grounded theory [8] to derive comment categories, major communication patterns, and group work problems. The major categories of comments are ordered as “revision suggestion,” “question,” “inconsistency,” “answer,” “incompleteness,” and “incompliance with standards provided by NCI.” Therefore, most comments carry a substitution text for revision. This observation led to our design for supporting editors to access review annotations when they revise protocols [4].
2.1.3.
S247
Semi-structured interview
Initially, we identified three major roles in protocol development: protocol coordinator, principle investigator, and statistician. We recruited interviewees for each role using a snowball sampling method. There were seven protocol coordinators in total in SWOG at the time of study. We were able to interview three of them and their manager from San Antonio to understand the work of a protocol coordinator. We also interviewed four statisticians and three principle investigators who were recently involved in protocol development and were located in Seattle. During the second part of this research, based on new knowledge gained from our evaluative ethnography, we interviewed two more supportive roles for protocol development: data coordinator and PRC members, who perform protocol review. We interviewed one data coordinator and two PRC members. Each semi-structured interview lasted about an hour [9]; some interviews took 2 h because those interviewees were willing to talk and provide more details. During the interviews, we used a “process query” technique by asking each interviewee to draw a diagram of the writing process based on their perceptions. Moreover, we asked our interviewees to recall some extreme scenarios using the “Critical Incident Techniques” (CIT) [10]. Our users told us stories about a few extremely complex protocol development cases. CIT helped us get real personal experiences while minimizing interference from stereotypical reactions. This ethnographic study provided us with an in-depth understanding of design-sensitive work practice details, such as communication problems, group interactions, and workflow. We found that comments were essential to protocol writing processes because reviewers convey their feedback and carry out discussions or negotiations with other reviewers through comments. On the other hand, editors and authors
2.2.
Participatory design
Our participatory design process spanned almost 2 years; it followed our first part of ethnography study and happened in parallel to our evaluative ethnography and the second part of work process ethnography. Our major techniques were rolebased user advocacy, iterative prototyping, and change management.
2.2.1.
Role-based user advocacy
User-centered design has been proven to help ensure the usability of a system. However, an open research question is “Who could be the participants that represent users?” Muller defined six approaches to selecting representative users based on interpretations from the perspectives of statistics, politics, design practice, and grounded theory [12]. Among these methods, the grounded theory approach ensures diversity among users, and the design practice approach finds the politically representative users. Since our work involved multidisciplinary users working on different roles, we wanted to combine the features of both approaches. We selected politically representative users for each role in the protocol writing process identified by our ethnographic fieldwork and made sure that they were advocates of a new technology [13]. User advocates evaluated the system prototype in their real work practice and provides timely feedback to designers. Therefore, we called our approach “role-based user advocacy”. We selected participants across three roles to form a design team with us: one protocol editor, one study coordinator, and two statisticians. Although there has been some turnover in individuals, we have kept representatives from all three roles closely involved throughout our iterative prototyping design process. In addition, we also kept a couple of policy makers in the loop all the time. Periodically, we reported the status of the design and demonstrated the ongoing design to these policy makers. The policy makers understand the organization’s culture, provide leadership, and provide incentives to other participatory users to try out new technology. We found that timely communication with these leading roles is useful and important for a smooth system adoption later.
2.2.2.
Iterative prototyping
As Scacchi pointed out, the classic socio-technical design or participatory design approach does not provide critical insights, tools, or guidelines beyond “user participation” [2]. One open research question that we tried to address during this process is “What counts as an active participation and
S248
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
how to encourage it?” We utilized two effective props: system prototype mockup and evolutionary prototyping. Both ethnographic study results and input from participants inform the design. First, to assess the logical feasibility of our system design, we began with a paper-based mockup interface and scenarios of a new workflow. These mockups were developed using Microsoft Visio, a drawing package that helps create fake screenshots that look as if from real windows systems. Scenarios helped reconstruct work like contexts in which users are encouraged to formulate design ideas as domain experts. We performed cognitive walkthrough on the scenarios for our participants and elicited their feedback to refine the design [14]. This process was cost-effective and gave a vivid representation of the potential system outlook. The scenarios provided sufficient details for the logical flow of functionalities to participants. All participants considered the artifacts intuitive and stimulating. After we reached consent with participants on the interface and workflow design using mockups, we settled on the logical feasibility of our system design, and moved onto web-based prototypes to assess the technical feasibility of the potential design by considering issues such as data backend and concurrency control strategies. We went through two major rounds of system infrastructure changes. Our initial prototype used a file-based backend, which supported the old SWOG work practice of “only one editor is authorized to make changes on documents” perfectly. However, participants felt encouraged by the first prototype and were willing to try a more synchronous collaborative work mode by enabling all the authors to write multiple sections of a protocol concurrently. This new request was not supported by the file system; therefore, we extend the system to a database-driven web application. Participants provided very positive feedback for the second prototype, which remains as the current design. Web-based prototyping enabled concrete experience and modification by prospective users. Throughout our prototyping design, we held participatory design meetings with our participants as regular as once every 2 weeks. We consulted them about the usability of the interface design and system functionalities. Our participants were enthusiastic and felt encouraged because we incorporated many creative design ideas from them. We also felt that the iterative prototyping helped us build a mutual understanding and trust between system designers and user participants. We understood their work practice better, and they understood the technical feasibility better.
2.2.3.
Change management through negotiations
A big barrier in innovation deployment is resistance to changes from users [15,16]. Markus identified three categories of user resistance: user-centered, system-centered, and interactional [17]. Since participatory design emphasized user involvement, here we managed user-centered resistance: a resistance to innovation caused by a lack of knowledge or a reluctance to change on the user side. At SWOG we tried to minimize this resistance through frequent negotiation of changes with users. For example, we prepared “What are new changes” documents prior to each participatory evaluation session and gave participants a clear comparison of
their current work practice and the potential new work practice supported by our system. We elicited their feedback and discussed the tradeoffs with them. This approach served as educational sessions that helped participants understand design rationale, which turned out to be an important factor for them when they made system adoption decisions later.
2.3.
Evaluative ethnography
Formative evaluation is an integral part of our participatory design process. Since our design was exploratory, there was no prior peer system; a self-reflective evaluation is more appropriate than comparison-based evaluations. Formative evaluation throughout the design process helped us collect continuous feedback from participants as the design evolved. Our evaluation methods largely follow Kaplan’s suggestions [3] from the following two aspects: (1) set up evaluation objectives from multiple perspectives and from the beginning of system design; and (2) use multiple evaluation methods. As Kaplan pointed out, well recognized areas of system evaluation in the medical informatics literature include: (1) technical and system features that affect system use, (2) cost–benefit analysis, (3) user acceptance, and (4) patient outcomes. Here we are particularly interested in evaluation areas (1) and (3); (2) and (4) would be the long-term evaluation objectives that could be carried out later. Therefore, we created a set of usability evaluation questions for these two selected areas. Below are some example evaluation questions: 1. Can the new design fulfill the required functionalities that are supported by current work mode? 2. Can the new design address the range of problems that we observed via our ethnographic fieldwork? 3. Can the new design improve the group work experiences of these users? 4. Is the new system easy to use and are the system features appropriate and useful? Following Kaplan’s suggestion that evaluation should use multiple methods [3], our approach integrated interviews, focus-groups, and system use observations. We designed a staged evaluation plan, including the following steps: (1) system testing from a software engineering perspective; (2) taskoriented, role-centered, and scenario-based user testing; and (3) short-term field trials in a natural work setting. We kept participants closely involved in each stage of design and evaluation while incrementally increasing the degree of complexity of group interactions in different evaluation stages so that we could keep some control and better refine the design according to feedback from participants. So far, we have carried out two field trials for the system focused on interactions between protocol reviewers and protocol coordinators.
3.
Research results
Quick and dirty ethnography, participatory design, and parallel evaluative ethnography form a integrated design process in this research and effectively lead to satisfactory research
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
S249
Fig. 2 – With our PCAT infrastructure, users have a single web portal to the evolving document as well as comments and a library of standards. Communication is still open, but all users work with the same document.
results, including a dynamic comment model that facilitates collaborative writing at the data management level [18], a collaborative clinical trial writing system prototype [4], and a description of collaborative protocol writing process at SWOG with some important design implications [19,20]. Below we briefly described the major group work problems, features of our collaborative writing system, and formative evaluation results.
3.1.
Group work problems
At SWOG, the process of iteratively reviewing and revising protocol documents was error-prone and led to a number of work-coordination and communication problems. We identified problems such as ineffective version control of evolving documents, inefficient group communication via emails, divergent views of the process by different roles, and ineffective group coordination [20]. The major unmet needs of current protocol writers are: (1) awareness of the shared workspace and individual contributions to the evolving protocol, (2) a shared repository of protocol and comments with version control for both, and (3) an integrated reviewing and revising tool that has wide accessibility.
3.2.
Current system designs
Our designs so far consist of two parts. The first part is a generic comment model that facilitates in situ document reviewing and communications [18]. The model defines dynamic text anchor in changing documents, discussion context, version information for a comment, as well as activity transition status to support group awareness. Based on this model, we also arrive at a web-based protocol collaborative writing tool (PCAT) that strengthens iterative reviewing and revisions [4]. In our PCAT system, we support a new workflow model, whose comparison with the old workflow model in SWOG is illustrated in Fig. 2. Fig. 3 shows a screenshot of the PCAT interface for protocol reviewers. The interface is divided into three panels. The left panel displays threaded comments. The top right panel displays the text of a protocol document, where a reviewer can select arbitrary text and insert comments; the commented text is then highlighted. Each reviewer is represented with a different color. The bottom right panel displays the details of a comment, including incorporation status, the time when the comment is made, who makes the comment, who is supposed to receive the comment, and other such metadata that we describe in our comment model [18].
Fig. 3 – A screenshot of our PCAT system.
S250
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
A comment author can edit or delete his or her own comment, and any participant can attach new comments to any comment to facilitate online negotiations or discussions in the context of the shared document. Moreover, a floating window is displayed to show the recent activity of group members and to provide group awareness to collaborative protocol writers.
3.3.
System formative evaluation results
Our formative evaluations have been focused on eliciting feedback about the usability of our design of the new workflow, user interface, and other system features. Our usability evaluation studies showed that users liked our system design, and particularly the following four aspects of our collaborative writing system. First, users liked “comment incorporating” and “notification service” very much. They considered that these features could speed up the reviewing process and help group members stay abreast of the ongoing progress easily. Second, interoperability between the system and the old tools such as email and MS Word was highly preferred. Users needed smooth transition of work between two systems for the best flexibility. We also realize that this is important for any innovation diffusion. We should not force users to give up their familiar tools easily; it is better to give them more than one option and let themselves decide what to keep and what to use. Third, users preferred an “all-in-one” design. In another word, users liked integration. The system currently supports email, editing, commenting, progress report, document management and sharing, and many other features, which are far beyond our original intentions to strengthen the reviewing and revising activities. Such an integrated tool enables users to carry out all work in a single environment. Last, “less is more” is an important principle for work-oriented system design. Filters for information about group awareness or comments are very well received and heavily used. Most users commented that they liked to see information specific to their working document or their group, and they liked direct links to their interested information without navigating through multiple page. In addition, we found that “change management” remains a challenging task. Users tended to swing between the old practice and the new workflow model. Our users often fought with the question of “Is this change necessary and valuable?” We need further research to find out the answer; maybe a longterm field trial of the system can help convince users, where cost–benefit tradeoffs can be more explicitly measured. Moreover, we realize that change management requires efforts from at least two levels: individual level (micro level) and organizational level (macro level). Our “change explanation” documents helped individual participants understand the system design and facilitated change management at a micro level. However, a large organization like SWOG has hundreds of potential users of this system who are used to the old standard of e-mail and MS Word. Negotiating changes among the statisticians, protocol coordinators, and data coordinators is certainly a challenging task but getting the study coordinators, a constantly changing list of investigators who may or may not be computer savvy and are distributed widely as well as involved sparsely in this type of work, to use the system is a much more daunting task. It is an unknown quantity to manage changes at the organizational level.
3.4.
Design adoption status
Overall, we are encouraged by our evaluation study results. Our design participants so far have given enthusiastic support and positive feedback for the design. Many healthcare information system designs face adoption challenges. In this pilot research, we not only engaged users throughout the system design process, but also won the adoption of a pilot group of SWOG. Currently the statistical center’s leader is making efforts to transfer the prototype system into a fully-functional tool that will eventually be used daily in the statistical center of SWOG, which plays the major role in SWOG protocol development. The advantage of starting with a small branch of SWOG is that we can control the complicating factors from distributed clinical trial investigators for now. We also believe that this is a great opportunity to enable further collaborative research with other parts of SWOG or other collaborative clinical trials research organizations.
4.
Discussion
At the beginning of this paper, we raised four research questions about participatory design. At this point, we can answer these questions based on our experiences from this research. First, ethnography could be an integral part of a participatory design process and could be used to increase the sensibility of system design to the working context. Our quick and dirty ethnography proved to be practical and efficient in this work. We system designers did not strive for an exhaustive work process description; instead, we just focused on potential work problems that are relevant to the design. We think an important step is to define research questions and provide a focus for a “quick and dirty” ethnography. Interviews, observations, and working artifact analysis working together can sensitize designers to potential work problems in an efficient way. Second, collaborative healthcare information system should consider multiple roles and their needs together. Role-based user advocacy is an effective strategy toward collaborative system design. Third, to engage users actively in a participatory design, evolutionary system prototypes are a significant improvement from other artifacts because they are concrete artifacts that users can interact with and they are a source of further design ideas; otherwise, it is hard for users to articulate what they want, what might work for them, etc. Moreover, evaluative ethnography should be conducted along the way because it provides timely feedback to designers and helps designers adjust the focus of ethnography studies. Otherwise, some design mistakes might be too costly to fix later. Therefore, we conclude that our design model as illustrated in Fig. 1 is effective. We hope it could be useful for other healthcare information system design. One of our broad research interests is to improve the process of specifying software requirements. Currently many healthcare system designers still follow conventional requirement specification documents. The problem is that a large portion of system requirements is tacit and hard to articulate at the beginning of a system design process. As a result, the generated requirement specification documents often convey vague or incomplete system requirements, and are subject
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
to changes over time. Throughout our participatory design, we found that user requirements are emergent and change often; therefore, we did not use a static requirement specification document. Instead, we had many design proposals, versioned system descriptions, notes on user feedback, a list of design modification tasks and priority features, and notes from meetings with different stakeholders of the project. We view these artifacts as a “dynamic requirements document” that evolves over time as the project proceeds. Such dynamic requirements documents need version control and these versioned documents could provide design rationale to facilitate future system maintenance. It would be interesting to conduct further study to find out whether this kind of “dynamic requirement specification document” is generalizable to other software projects. Much of health care work is collaborative; thus, information systems design should consider group work requirements. So far few groupware systems that directly address multiple user roles have been studied in medical informatics. Thus, we found it important to consider computer supported cooperative work (CSCW) [21]. This field offers insights into the design of groupware systems. For example, we leveraged ideas about group awareness and work-coordination from CSCW. We also applied the idea of “role-based user advocacy” into our participatory design approach, where we included participants from all significant roles in the authoring process. This ensures that the system will include incentives for all users, rather than just for certain stakeholders. Our research extended our understanding of “change management” during system design in the healthcare setting. Healthcare setting is a loosely-coupled collaborative environment. Any system’s installation may affect a very big pool of users with diverse training backgrounds, different levels of expertise, and varied system requirements. How to manage changes at the macro level to address many users is a new research problem that we discovered from our research process. We think this problem is especially important for collaborative healthcare information system design. Further research needs address this problem. In summary, we believe that role-based user advocacy, iterative evolutionary prototypes, and ethnography are all integral parts to a work-oriented and user-centered design solution. These techniques together with CSCW concepts and technology helped us reduce user resistance and usability errors during early prototyping and arrive at a feasible and userpreferred system design. We have designed our system and applied our methodology specifically in the work setting of SWOG collaborative protocol authoring; however, we hope that our approach can be more broadly applied to healthcare groupware design in general.
Acknowledgements We thank the director of SWOG Statistical Center, Dr. John Crowley, for his constant support in this research and other SWOG participants for their active involvement, including
S251
Jacqueline Benedetti, Jim Moon, Diana Lowry, and others. We also thank all reviewers for the helpful comments.
references
[1] J. Ash, Organizational factors that influence information technology diffusion in academic health sciences centers, J. Am. Med. Inform. Assoc. 4 (2) (1997) 102–111. [2] W. Scacchi, Socio-technical design, in: W.S. Bainbridge (Ed.), The Encyclopedia of Human-Computer Interaction, Berkshire Publishing Group, Great Barrington (MA), 2004. [3] B. Kaplan, Addressing organizational issues into the evaluation of medical systems, J. Am. Med. Inform. Assoc. 4 (2) (1997) 94–101. [4] C. Weng, J. Gennari, D. McDonald, A collaborative clinical trial protocol writing system, Medinfo 11 (2004) 1481– 1486. ¨ [5] C. Sjoberg, T. Timpka, Participatory design of information systems in health care, J. Am. Med. Inform. Assoc. 5 (2) (1998) 177–183. [6] J. Hughes, et al., Moving out from the control room: ethnography in system design, in: Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, 1994, pp. 429–439. [7] A. Crabtree, Designing Collaborative Systems: A Practical Guide to Ethnography, Springer, 2003. [8] B.G. Glaser, A.L. Strauss, The Discovery of Grounded Theory, Aldine Publishing Company, New York, 1967. [9] L.E. Wood, Semi-structured interviewing for user-centered design, Interactions 4 (2) (1997) 48–61. [10] J. Flanagan, The critical incident technique, Psychol. Bull. 51 (4) (1954) 327–359. [11] P. Dourish, V. Bellotti, Awareness and coordination in shared workspaces, in: Proceedings of CSCW’92, ACM Press, New York, 1992, pp. 107–114. [12] M. Muller, D.R. Millen, What makes a representative user representative? A participatory poster, in: Proceedings of CHI’01, Seattle, 2001, pp. 101–102. [13] P. Mambrey, G. Mark, U. Pankoke-Babatz, User advocacy in participatory design: designer’s experiences with a new communication channel, Comput. Support. Coop. Work 7 (1998) 291–313. [14] D. Rowley, D. Rhoades, The cognitive jog-through: a fast-paced user interface evaluation procedure, in: Proceedings of CHI’92, 1992, pp. 389–395. [15] D.M. Berwick, Disseminating innovations in health care, JAMA 289 (2003) 1969–1975. [16] N.M. Lorenzi, R.T. Riley, Managing change: an overview, J. Am. Med. Inform. Assoc. 7 (2) (2000) 116–124. [17] M.L. Markus, Power, politics, and MIS implementation, Comm. ACM 26 (1983) 430–444. [18] C. Weng, J. Gennari, Asynchronous collaborative writing through annotations, in: Proceedings of CSCW’04, ACM Press, New York, 2004, pp. 578–581. [19] D.W. McDonald, C. Weng, J.H. Gennari, The multiple views of inter-organizational authoring, in: Proceedings of CSCW’04, ACM Press, New York, 2004, pp. 564–573. [20] J. Gennari, et al., An ethnographic study of collaborative clinical trial protocol writing, Medinfo 11 (2004) 1461–1465. [21] W. Pratt, et al., Incorporating ideas from computer-supported cooperative work, J. Biomed. Inform. 37 (2) (2004) 128–137.
journal homepage: www.intl.elsevierhealth.com/journals/ijmi
Participatory design of a collaborative clinical trial protocol writing system Chunhua Weng a,∗ , David W. McDonald b , Dana Sparks c , Jason McCoy d , John H. Gennari e a
Center of Pathology and Oncology Informatics, University of Pittsburgh, PA, United States Information School, University of Washington, Seattle, United States c The Southwest Oncology Group, United States d Cancer Research and Biostatistics, Seattle, WA, United States e Department of Medical Education and Biomedical Informatics, University of Washington, Seattle, WA, United States b
a r t i c l e
i n f o
a b s t r a c t
Article history:
Objective: To explore concrete approaches to socio-technical design of collaborative health-
Received 5 March 2006
care information systems and to design a groupware technology for collaborative clinical
Accepted 11 May 2006
trial protocol writing. Method: We conducted “quick and dirty ethnography” through semi-structured interviews, observational studies, and work artifacts analysis to understand the group work for protocol
Keywords:
development. We used participatory design through evolutionary prototyping to explore the
Participatory design
feature space of a collaborative writing system. Our design strategies include role-based user
Quick and dirty ethnography
advocacy, formative evaluation, and change management.
Clinical trial protocol
Results: Quick and dirty ethnography helped us efficiently understand relevant work prac-
Socio-technical approach
tice, and participatory design helped us engage users into design and bring out their tacit work knowledge. Our approach that intertwined both techniques helped achieve a “workinformed and user-oriented” design. This research leads to a collaborative writing system that supports in situ communication, group awareness, and effective work progress tracking. The usability evaluation results have been satisfactory. The system design is being transferred to an organizational tool for daily use. © 2006 Elsevier Ireland Ltd. All rights reserved.
1.
Introduction
Many researchers have realized the importance of social factors and organizational issues for system design [1–3]. However, concrete methods that account for social factors in system design are largely unavailable or have insufficient guidelines. Participatory design (PD) is one approach to socio-technical designs [4]. It is a proactive design method that explicitly advocates active user participation throughout
∗
the design process. Unfortunately, PD has not been widely adopted in healthcare information system designs [5]. The major challenges involved in PD include long prototyping design cycles, difficulties to carry out representative participant selection, and changing or conflicting user inputs. PD without sufficient considerations of work practice may abuse user opinions and generate a perfect technical solution for a wrong workflow. Therefore, our research has two goals: one is to explore the design space of a collaborative clinical trial
Corresponding author. Tel.: +1 412 513 5340; fax: +1 412 647 5380. E-mail addresses: [email protected] (C. Weng), [email protected] (D.W. McDonald), [email protected] (D. Sparks), [email protected] (J. McCoy), [email protected] (J.H. Gennari). 1386-5056/$ – see front matter © 2006 Elsevier Ireland Ltd. All rights reserved. doi:10.1016/j.ijmedinf.2006.05.035
S246
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
protocol writing system, and the other is to enrich the participatory design theory by trying to answer these four research questions: 1. How can we make ethnographic studies complement participatory design methods? 2. How can we best apply participatory design to the design of collaborative information systems in this healthcare setting? 3. Who should be chosen to best represent users in a participatory design of a groupware technology? 4. What are effective strategies or incentives for active user participation? Our research setting is The Southwest Oncology Group (SWOG), one of a number of cooperative programs that develop cancer clinical trial protocols under the direction of National Cancer Institute (NCI). SWOG members include approximately 4000 leading clinical experts, statisticians, protocol editors, and other clinical trial workers across the country. Each SWOG clinical trial protocol takes 4–9 months to author and the resulting protocol ranges from 60 to more than 100 pages. A combined use of Microsoft Word and email systems is inefficient and error-prone for the collaborative protocol writing processes. Prior to our efforts, SWOG had tried different systems to support their protocol authoring process. However, none of those systems were adopted. One possible reason is that those systems failed to address the group work challenges. Our research aims to fill this gap. This paper focuses on our participatory design experiences at SWOG. In the rest of this paper, we first elaborate on our design methodology, and then we present how it led to a usable system design. We conclude with a discussion about the effectiveness and generalizability of our research methodology.
2.
Design methodology
To design the right system in the right way in healthcare settings, it is important to understand both the work context and user preferences. In this research, to achieve a design grounded in work and users, we intertwined “quick and dirty ethnography” [6,7] and participatory design as Fig. 1 shows below. Based on this model, an iterative design process consists of three components: (1) quick and dirty ethnography assembles instances of working roles and their work activities, and sensitizes designers with selected focus of the work practices; (2) participatory design engages role-based user
advocates, who understand users and are able to articulate users needs to designers, in carrying out system design through evolutionary prototyping; (3) formative evaluations are carried out through evaluative ethnography to understand the interaction between user advocates and the prototypes, and also to provide feedback for ethnography and participatory design. Evaluation results drive further field ethnography or help adjust subsequent participatory designs, and then improve the prototypes. In the conventional social–technical design model, field studies and system designs are widely apart, and field studies are mostly design oriented. With this method, there has been a better synergy between field studies and participatory designs. As Fig. 1 shows, the prototype serves two roles: one as an evolving representation of the design ideas and the other as a probing tool to deepen the field knowledge. Change management has been carried out in PD sessions through discussions with participants about the design feasibility for improving work experience, the possible changes to the current work practice, and the tradeoff between benefits and costs of embracing new changes. The research has received a human subject approval from University of Washington at Seattle. The details for quick and dirty ethnography, participatory design, and evaluative ethnography in the above model are provided below.
2.1.
Quick and dirty ethnography
To understand the current work practice, we carried out ethnography using the practical “quick and dirty ethnography” strategy developed by Hughes et al. [6]. Quick and dirty ethnography is different from conventional ethnography in that it does not provide an exhaustive description of the whole work context; instead it strategically focuses on selected aspects of the work and provides relevant work details to designers in a short time period [7]. In this research, we are particularly interested in group communication, information sharing and integration, and the workflow of protocol writing. The following open-ended research questions are the foci of our ethnography: 1. What are the major steps in the protocol writing process? 2. Who is involved in each step and what are their job responsibilities? 3. How is group communication carried out? 4. What is the most challenging part of the writing? 5. How do protocol writers coordinate the group work? 6. What is the protocol review and revision process like? Our ethnographic methods include semi-structured interviews with SWOG protocol writers, observation of protocol review committee (PRC) meetings, and protocol development artifact collection and analysis. These approaches leveraged information gained from one other.
2.1.1.
Fig. 1 – An iterative grounded design.
Observational study
Over a period of about 6 months, we observed PRC meetings. At the time of our study, protocols were typically reviewed twice in PRC meetings. Each meeting reviews an average of two protocols, and meetings occurred about once a week. Meetings
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
were attended primarily by statisticians, although one protocol coordinator manager from San Antonio also attended via conference call. The first and the last author of this paper participated and collected notes from the meetings, as well as the official PRC notes that documented the suggested revisions and discussion points around a particular protocol. We observed where in protocols the major questions or discussion occur, how reviewers respond to other reviewers’ comments, how reviewers have common ideas or different opinions, and what review comments are typical. We collected notes from about 30 different protocols for later content analysis.
2.1.2.
iterate protocol document revisions based on suggestions in comments from experts. Therefore, effective support for group awareness [11], in situ communication, and comments incorporation is greatly needed by protocol writers. Interestingly, we found that the process diagrams created by different protocol writers were not exactly the same. Protocol writers have disparate views of the work process and tool support needs. This finding alludes to the importance of involving representative participants for each writing role in the design so that we could pattern their shared needs while coordinating their individual preference in a single system. These results effectively informed our participatory design as described below.
Artifact content analysis
The artifacts that we have collected and analyzed include (1) about 80 selective protocol development emails, which were forwarded to us by protocol writers; (2) organizational protocol development guidelines or standards; and (3) around 1140 protocol review comments for 32 protocols. From these comments, we used grounded theory [8] to derive comment categories, major communication patterns, and group work problems. The major categories of comments are ordered as “revision suggestion,” “question,” “inconsistency,” “answer,” “incompleteness,” and “incompliance with standards provided by NCI.” Therefore, most comments carry a substitution text for revision. This observation led to our design for supporting editors to access review annotations when they revise protocols [4].
2.1.3.
S247
Semi-structured interview
Initially, we identified three major roles in protocol development: protocol coordinator, principle investigator, and statistician. We recruited interviewees for each role using a snowball sampling method. There were seven protocol coordinators in total in SWOG at the time of study. We were able to interview three of them and their manager from San Antonio to understand the work of a protocol coordinator. We also interviewed four statisticians and three principle investigators who were recently involved in protocol development and were located in Seattle. During the second part of this research, based on new knowledge gained from our evaluative ethnography, we interviewed two more supportive roles for protocol development: data coordinator and PRC members, who perform protocol review. We interviewed one data coordinator and two PRC members. Each semi-structured interview lasted about an hour [9]; some interviews took 2 h because those interviewees were willing to talk and provide more details. During the interviews, we used a “process query” technique by asking each interviewee to draw a diagram of the writing process based on their perceptions. Moreover, we asked our interviewees to recall some extreme scenarios using the “Critical Incident Techniques” (CIT) [10]. Our users told us stories about a few extremely complex protocol development cases. CIT helped us get real personal experiences while minimizing interference from stereotypical reactions. This ethnographic study provided us with an in-depth understanding of design-sensitive work practice details, such as communication problems, group interactions, and workflow. We found that comments were essential to protocol writing processes because reviewers convey their feedback and carry out discussions or negotiations with other reviewers through comments. On the other hand, editors and authors
2.2.
Participatory design
Our participatory design process spanned almost 2 years; it followed our first part of ethnography study and happened in parallel to our evaluative ethnography and the second part of work process ethnography. Our major techniques were rolebased user advocacy, iterative prototyping, and change management.
2.2.1.
Role-based user advocacy
User-centered design has been proven to help ensure the usability of a system. However, an open research question is “Who could be the participants that represent users?” Muller defined six approaches to selecting representative users based on interpretations from the perspectives of statistics, politics, design practice, and grounded theory [12]. Among these methods, the grounded theory approach ensures diversity among users, and the design practice approach finds the politically representative users. Since our work involved multidisciplinary users working on different roles, we wanted to combine the features of both approaches. We selected politically representative users for each role in the protocol writing process identified by our ethnographic fieldwork and made sure that they were advocates of a new technology [13]. User advocates evaluated the system prototype in their real work practice and provides timely feedback to designers. Therefore, we called our approach “role-based user advocacy”. We selected participants across three roles to form a design team with us: one protocol editor, one study coordinator, and two statisticians. Although there has been some turnover in individuals, we have kept representatives from all three roles closely involved throughout our iterative prototyping design process. In addition, we also kept a couple of policy makers in the loop all the time. Periodically, we reported the status of the design and demonstrated the ongoing design to these policy makers. The policy makers understand the organization’s culture, provide leadership, and provide incentives to other participatory users to try out new technology. We found that timely communication with these leading roles is useful and important for a smooth system adoption later.
2.2.2.
Iterative prototyping
As Scacchi pointed out, the classic socio-technical design or participatory design approach does not provide critical insights, tools, or guidelines beyond “user participation” [2]. One open research question that we tried to address during this process is “What counts as an active participation and
S248
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
how to encourage it?” We utilized two effective props: system prototype mockup and evolutionary prototyping. Both ethnographic study results and input from participants inform the design. First, to assess the logical feasibility of our system design, we began with a paper-based mockup interface and scenarios of a new workflow. These mockups were developed using Microsoft Visio, a drawing package that helps create fake screenshots that look as if from real windows systems. Scenarios helped reconstruct work like contexts in which users are encouraged to formulate design ideas as domain experts. We performed cognitive walkthrough on the scenarios for our participants and elicited their feedback to refine the design [14]. This process was cost-effective and gave a vivid representation of the potential system outlook. The scenarios provided sufficient details for the logical flow of functionalities to participants. All participants considered the artifacts intuitive and stimulating. After we reached consent with participants on the interface and workflow design using mockups, we settled on the logical feasibility of our system design, and moved onto web-based prototypes to assess the technical feasibility of the potential design by considering issues such as data backend and concurrency control strategies. We went through two major rounds of system infrastructure changes. Our initial prototype used a file-based backend, which supported the old SWOG work practice of “only one editor is authorized to make changes on documents” perfectly. However, participants felt encouraged by the first prototype and were willing to try a more synchronous collaborative work mode by enabling all the authors to write multiple sections of a protocol concurrently. This new request was not supported by the file system; therefore, we extend the system to a database-driven web application. Participants provided very positive feedback for the second prototype, which remains as the current design. Web-based prototyping enabled concrete experience and modification by prospective users. Throughout our prototyping design, we held participatory design meetings with our participants as regular as once every 2 weeks. We consulted them about the usability of the interface design and system functionalities. Our participants were enthusiastic and felt encouraged because we incorporated many creative design ideas from them. We also felt that the iterative prototyping helped us build a mutual understanding and trust between system designers and user participants. We understood their work practice better, and they understood the technical feasibility better.
2.2.3.
Change management through negotiations
A big barrier in innovation deployment is resistance to changes from users [15,16]. Markus identified three categories of user resistance: user-centered, system-centered, and interactional [17]. Since participatory design emphasized user involvement, here we managed user-centered resistance: a resistance to innovation caused by a lack of knowledge or a reluctance to change on the user side. At SWOG we tried to minimize this resistance through frequent negotiation of changes with users. For example, we prepared “What are new changes” documents prior to each participatory evaluation session and gave participants a clear comparison of
their current work practice and the potential new work practice supported by our system. We elicited their feedback and discussed the tradeoffs with them. This approach served as educational sessions that helped participants understand design rationale, which turned out to be an important factor for them when they made system adoption decisions later.
2.3.
Evaluative ethnography
Formative evaluation is an integral part of our participatory design process. Since our design was exploratory, there was no prior peer system; a self-reflective evaluation is more appropriate than comparison-based evaluations. Formative evaluation throughout the design process helped us collect continuous feedback from participants as the design evolved. Our evaluation methods largely follow Kaplan’s suggestions [3] from the following two aspects: (1) set up evaluation objectives from multiple perspectives and from the beginning of system design; and (2) use multiple evaluation methods. As Kaplan pointed out, well recognized areas of system evaluation in the medical informatics literature include: (1) technical and system features that affect system use, (2) cost–benefit analysis, (3) user acceptance, and (4) patient outcomes. Here we are particularly interested in evaluation areas (1) and (3); (2) and (4) would be the long-term evaluation objectives that could be carried out later. Therefore, we created a set of usability evaluation questions for these two selected areas. Below are some example evaluation questions: 1. Can the new design fulfill the required functionalities that are supported by current work mode? 2. Can the new design address the range of problems that we observed via our ethnographic fieldwork? 3. Can the new design improve the group work experiences of these users? 4. Is the new system easy to use and are the system features appropriate and useful? Following Kaplan’s suggestion that evaluation should use multiple methods [3], our approach integrated interviews, focus-groups, and system use observations. We designed a staged evaluation plan, including the following steps: (1) system testing from a software engineering perspective; (2) taskoriented, role-centered, and scenario-based user testing; and (3) short-term field trials in a natural work setting. We kept participants closely involved in each stage of design and evaluation while incrementally increasing the degree of complexity of group interactions in different evaluation stages so that we could keep some control and better refine the design according to feedback from participants. So far, we have carried out two field trials for the system focused on interactions between protocol reviewers and protocol coordinators.
3.
Research results
Quick and dirty ethnography, participatory design, and parallel evaluative ethnography form a integrated design process in this research and effectively lead to satisfactory research
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
S249
Fig. 2 – With our PCAT infrastructure, users have a single web portal to the evolving document as well as comments and a library of standards. Communication is still open, but all users work with the same document.
results, including a dynamic comment model that facilitates collaborative writing at the data management level [18], a collaborative clinical trial writing system prototype [4], and a description of collaborative protocol writing process at SWOG with some important design implications [19,20]. Below we briefly described the major group work problems, features of our collaborative writing system, and formative evaluation results.
3.1.
Group work problems
At SWOG, the process of iteratively reviewing and revising protocol documents was error-prone and led to a number of work-coordination and communication problems. We identified problems such as ineffective version control of evolving documents, inefficient group communication via emails, divergent views of the process by different roles, and ineffective group coordination [20]. The major unmet needs of current protocol writers are: (1) awareness of the shared workspace and individual contributions to the evolving protocol, (2) a shared repository of protocol and comments with version control for both, and (3) an integrated reviewing and revising tool that has wide accessibility.
3.2.
Current system designs
Our designs so far consist of two parts. The first part is a generic comment model that facilitates in situ document reviewing and communications [18]. The model defines dynamic text anchor in changing documents, discussion context, version information for a comment, as well as activity transition status to support group awareness. Based on this model, we also arrive at a web-based protocol collaborative writing tool (PCAT) that strengthens iterative reviewing and revisions [4]. In our PCAT system, we support a new workflow model, whose comparison with the old workflow model in SWOG is illustrated in Fig. 2. Fig. 3 shows a screenshot of the PCAT interface for protocol reviewers. The interface is divided into three panels. The left panel displays threaded comments. The top right panel displays the text of a protocol document, where a reviewer can select arbitrary text and insert comments; the commented text is then highlighted. Each reviewer is represented with a different color. The bottom right panel displays the details of a comment, including incorporation status, the time when the comment is made, who makes the comment, who is supposed to receive the comment, and other such metadata that we describe in our comment model [18].
Fig. 3 – A screenshot of our PCAT system.
S250
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
A comment author can edit or delete his or her own comment, and any participant can attach new comments to any comment to facilitate online negotiations or discussions in the context of the shared document. Moreover, a floating window is displayed to show the recent activity of group members and to provide group awareness to collaborative protocol writers.
3.3.
System formative evaluation results
Our formative evaluations have been focused on eliciting feedback about the usability of our design of the new workflow, user interface, and other system features. Our usability evaluation studies showed that users liked our system design, and particularly the following four aspects of our collaborative writing system. First, users liked “comment incorporating” and “notification service” very much. They considered that these features could speed up the reviewing process and help group members stay abreast of the ongoing progress easily. Second, interoperability between the system and the old tools such as email and MS Word was highly preferred. Users needed smooth transition of work between two systems for the best flexibility. We also realize that this is important for any innovation diffusion. We should not force users to give up their familiar tools easily; it is better to give them more than one option and let themselves decide what to keep and what to use. Third, users preferred an “all-in-one” design. In another word, users liked integration. The system currently supports email, editing, commenting, progress report, document management and sharing, and many other features, which are far beyond our original intentions to strengthen the reviewing and revising activities. Such an integrated tool enables users to carry out all work in a single environment. Last, “less is more” is an important principle for work-oriented system design. Filters for information about group awareness or comments are very well received and heavily used. Most users commented that they liked to see information specific to their working document or their group, and they liked direct links to their interested information without navigating through multiple page. In addition, we found that “change management” remains a challenging task. Users tended to swing between the old practice and the new workflow model. Our users often fought with the question of “Is this change necessary and valuable?” We need further research to find out the answer; maybe a longterm field trial of the system can help convince users, where cost–benefit tradeoffs can be more explicitly measured. Moreover, we realize that change management requires efforts from at least two levels: individual level (micro level) and organizational level (macro level). Our “change explanation” documents helped individual participants understand the system design and facilitated change management at a micro level. However, a large organization like SWOG has hundreds of potential users of this system who are used to the old standard of e-mail and MS Word. Negotiating changes among the statisticians, protocol coordinators, and data coordinators is certainly a challenging task but getting the study coordinators, a constantly changing list of investigators who may or may not be computer savvy and are distributed widely as well as involved sparsely in this type of work, to use the system is a much more daunting task. It is an unknown quantity to manage changes at the organizational level.
3.4.
Design adoption status
Overall, we are encouraged by our evaluation study results. Our design participants so far have given enthusiastic support and positive feedback for the design. Many healthcare information system designs face adoption challenges. In this pilot research, we not only engaged users throughout the system design process, but also won the adoption of a pilot group of SWOG. Currently the statistical center’s leader is making efforts to transfer the prototype system into a fully-functional tool that will eventually be used daily in the statistical center of SWOG, which plays the major role in SWOG protocol development. The advantage of starting with a small branch of SWOG is that we can control the complicating factors from distributed clinical trial investigators for now. We also believe that this is a great opportunity to enable further collaborative research with other parts of SWOG or other collaborative clinical trials research organizations.
4.
Discussion
At the beginning of this paper, we raised four research questions about participatory design. At this point, we can answer these questions based on our experiences from this research. First, ethnography could be an integral part of a participatory design process and could be used to increase the sensibility of system design to the working context. Our quick and dirty ethnography proved to be practical and efficient in this work. We system designers did not strive for an exhaustive work process description; instead, we just focused on potential work problems that are relevant to the design. We think an important step is to define research questions and provide a focus for a “quick and dirty” ethnography. Interviews, observations, and working artifact analysis working together can sensitize designers to potential work problems in an efficient way. Second, collaborative healthcare information system should consider multiple roles and their needs together. Role-based user advocacy is an effective strategy toward collaborative system design. Third, to engage users actively in a participatory design, evolutionary system prototypes are a significant improvement from other artifacts because they are concrete artifacts that users can interact with and they are a source of further design ideas; otherwise, it is hard for users to articulate what they want, what might work for them, etc. Moreover, evaluative ethnography should be conducted along the way because it provides timely feedback to designers and helps designers adjust the focus of ethnography studies. Otherwise, some design mistakes might be too costly to fix later. Therefore, we conclude that our design model as illustrated in Fig. 1 is effective. We hope it could be useful for other healthcare information system design. One of our broad research interests is to improve the process of specifying software requirements. Currently many healthcare system designers still follow conventional requirement specification documents. The problem is that a large portion of system requirements is tacit and hard to articulate at the beginning of a system design process. As a result, the generated requirement specification documents often convey vague or incomplete system requirements, and are subject
i n t e r n a t i o n a l j o u r n a l o f m e d i c a l i n f o r m a t i c s 7 6 S ( 2 0 0 7 ) S245–S251
to changes over time. Throughout our participatory design, we found that user requirements are emergent and change often; therefore, we did not use a static requirement specification document. Instead, we had many design proposals, versioned system descriptions, notes on user feedback, a list of design modification tasks and priority features, and notes from meetings with different stakeholders of the project. We view these artifacts as a “dynamic requirements document” that evolves over time as the project proceeds. Such dynamic requirements documents need version control and these versioned documents could provide design rationale to facilitate future system maintenance. It would be interesting to conduct further study to find out whether this kind of “dynamic requirement specification document” is generalizable to other software projects. Much of health care work is collaborative; thus, information systems design should consider group work requirements. So far few groupware systems that directly address multiple user roles have been studied in medical informatics. Thus, we found it important to consider computer supported cooperative work (CSCW) [21]. This field offers insights into the design of groupware systems. For example, we leveraged ideas about group awareness and work-coordination from CSCW. We also applied the idea of “role-based user advocacy” into our participatory design approach, where we included participants from all significant roles in the authoring process. This ensures that the system will include incentives for all users, rather than just for certain stakeholders. Our research extended our understanding of “change management” during system design in the healthcare setting. Healthcare setting is a loosely-coupled collaborative environment. Any system’s installation may affect a very big pool of users with diverse training backgrounds, different levels of expertise, and varied system requirements. How to manage changes at the macro level to address many users is a new research problem that we discovered from our research process. We think this problem is especially important for collaborative healthcare information system design. Further research needs address this problem. In summary, we believe that role-based user advocacy, iterative evolutionary prototypes, and ethnography are all integral parts to a work-oriented and user-centered design solution. These techniques together with CSCW concepts and technology helped us reduce user resistance and usability errors during early prototyping and arrive at a feasible and userpreferred system design. We have designed our system and applied our methodology specifically in the work setting of SWOG collaborative protocol authoring; however, we hope that our approach can be more broadly applied to healthcare groupware design in general.
Acknowledgements We thank the director of SWOG Statistical Center, Dr. John Crowley, for his constant support in this research and other SWOG participants for their active involvement, including
S251
Jacqueline Benedetti, Jim Moon, Diana Lowry, and others. We also thank all reviewers for the helpful comments.
references
[1] J. Ash, Organizational factors that influence information technology diffusion in academic health sciences centers, J. Am. Med. Inform. Assoc. 4 (2) (1997) 102–111. [2] W. Scacchi, Socio-technical design, in: W.S. Bainbridge (Ed.), The Encyclopedia of Human-Computer Interaction, Berkshire Publishing Group, Great Barrington (MA), 2004. [3] B. Kaplan, Addressing organizational issues into the evaluation of medical systems, J. Am. Med. Inform. Assoc. 4 (2) (1997) 94–101. [4] C. Weng, J. Gennari, D. McDonald, A collaborative clinical trial protocol writing system, Medinfo 11 (2004) 1481– 1486. ¨ [5] C. Sjoberg, T. Timpka, Participatory design of information systems in health care, J. Am. Med. Inform. Assoc. 5 (2) (1998) 177–183. [6] J. Hughes, et al., Moving out from the control room: ethnography in system design, in: Proceedings of the 1994 ACM Conference on Computer Supported Cooperative Work, 1994, pp. 429–439. [7] A. Crabtree, Designing Collaborative Systems: A Practical Guide to Ethnography, Springer, 2003. [8] B.G. Glaser, A.L. Strauss, The Discovery of Grounded Theory, Aldine Publishing Company, New York, 1967. [9] L.E. Wood, Semi-structured interviewing for user-centered design, Interactions 4 (2) (1997) 48–61. [10] J. Flanagan, The critical incident technique, Psychol. Bull. 51 (4) (1954) 327–359. [11] P. Dourish, V. Bellotti, Awareness and coordination in shared workspaces, in: Proceedings of CSCW’92, ACM Press, New York, 1992, pp. 107–114. [12] M. Muller, D.R. Millen, What makes a representative user representative? A participatory poster, in: Proceedings of CHI’01, Seattle, 2001, pp. 101–102. [13] P. Mambrey, G. Mark, U. Pankoke-Babatz, User advocacy in participatory design: designer’s experiences with a new communication channel, Comput. Support. Coop. Work 7 (1998) 291–313. [14] D. Rowley, D. Rhoades, The cognitive jog-through: a fast-paced user interface evaluation procedure, in: Proceedings of CHI’92, 1992, pp. 389–395. [15] D.M. Berwick, Disseminating innovations in health care, JAMA 289 (2003) 1969–1975. [16] N.M. Lorenzi, R.T. Riley, Managing change: an overview, J. Am. Med. Inform. Assoc. 7 (2) (2000) 116–124. [17] M.L. Markus, Power, politics, and MIS implementation, Comm. ACM 26 (1983) 430–444. [18] C. Weng, J. Gennari, Asynchronous collaborative writing through annotations, in: Proceedings of CSCW’04, ACM Press, New York, 2004, pp. 578–581. [19] D.W. McDonald, C. Weng, J.H. Gennari, The multiple views of inter-organizational authoring, in: Proceedings of CSCW’04, ACM Press, New York, 2004, pp. 564–573. [20] J. Gennari, et al., An ethnographic study of collaborative clinical trial protocol writing, Medinfo 11 (2004) 1461–1465. [21] W. Pratt, et al., Incorporating ideas from computer-supported cooperative work, J. Biomed. Inform. 37 (2) (2004) 128–137.