Intelligent multimedia repositories (IMRs) for project estimation and management: an empirical study

Int. J. Human – Computer Studies (1996) 45 , 443 – 482

Intelligent multimedia repositories (IMRs) for project estimation and management: an empirical study BARRY G. SILVERMAN

AND

NABIL BEDEWI

Institute for Artificial Intelligence , George Washington Uniy ersity , Staughton Hall , Room 206 , Washington DC 20052 , USA. email:barryêseas.gwu.edu (Receiy ed 24 October 1995 and accepted in rey ised form 28 May 1996) This research explores whether the use of multimedia and intelligent agents foster the reuse of artifacts from a repository. That is, can a repository enhance reuse effectivity if it can (1) offer diverse media for conveying an artifact’s information, and (2) utilize agents that support human reuse processes? A study was conducted with 33 professional respondents in a software project estimation repository. Performance and reaction data were collected on the purpose, role, usefulness, impact, and importance of 14 media / artifact categories. Results show the repository improves performance significantly, and the multimedia and agents play an important role. Specific lessons learned are offered for the design of reuse repositories, the use of multimedia, and the role for intelligent agents. ÷ 1996 Academic Press Limited

1. Introduction Reusable artifact repositories are an important asset for improving team productivity and product line quality. From simple clip art and reusable script libraries of desktop applications, to the more complex domains of software parts reuse libraries managed by Computer Aided Software Environments (CASEs), this trend is proliferating. In this paper we report on whether there are effective ways to even further enhance this reuse-supported work. This research views reuse-supported work as spiraling through two basic steps: construction and refinement. Initially, to get past the ‘‘blank page’’ one must create and construct. A draft solution assembled from roughly appropriate reuse artifacts is acceptable. Once a draft is constructed, users then attempt a refinement step. Partial failures of reuse artifacts are noticed and refinements are attempted. This cycle may repeat until a quality, budgetary or time limit is reached. Variations of this two step process are not new and they appear in a number of problem solving methods (e.g. Bacon, 1620, Peirce, 1935; Polya, 1945; among others). An interesting question is whether repositories can (or should): (1) be better attuned to how humans use the two step reuse-supported work process, (2) be given media appropriate to the domain content and knowledge the repository artifacts need to convey, and (3) be given intelligent agent abilities to better support the process and domain media manipulation. As a precursor to designing such a system , this research explores the question: could reuse repositories with intelligent, multimedia artifacts improve user productivity and performance quality relative to repositories that lack intelligent multimedia designs? What intelligent, multimedia elements do users want and need? What elements will they use? The purpose is to 443 1071-5819 / 96 / 100443 1 40$18.00 / 0

÷ 1996 Academic Press Limited

444

B. G. SILVERMAN AND N. BEDEWI

examine whether artifacts themselves convey more information when represented in a different medium. Is there something about the design of artifact-media that adds true value to work productivity and quality. Similarly, if repositories are to be mainstreamed, how important are intelligent agents with deep knowledge of both the two process reuse step and the specific domain of the repository? This research does not intend to answer such questions in general, but seeks to answer them by conducting an empirical investigation about selected design features for a specific repository. Specifically, we conducted an experiment where 33 users attempted tasks both with and without a specific repository’s help. The user performance and reaction data shows that repositories are useful and important, but they rarely have the information users want. Also, participants reuse artifacts they are familiar with and / or can rapidly become familiar with (e.g. through a phone call to a trusted colleague). Any media or agent that reduces obstacles to reuse (by making artifacts readily understood) will have an important impact. Our results provide many insights and lessons learned on better designs for media and agents. The results also show that users often have clearer ideas than developers on what those better designs should be. Accommodating the full set of user-desired media and agent needs raises significant research challenges that are addressed later in this paper. The domain of the repository studied here is software component estimation (budget, manpower, project duration), specifically, software for sorting routines. There are tools for helping project managers make their forecasts such as COCOMO (Boehm, 1981), SME (Kistler & Valett, 1992), or various metrics based approaches (Conte, Dunsmore & Shen, 1986). Such tools are useful when there is a large sample of project data for the specific components of interest in the domain of the organization that adopts the tool. More often though, organizations do not adopt such tools, and even when they do, project leaders favor informal tools and general heuristical reasoning to make component estimates. Software project managers around the world make thousands of estimates a day using variations of the two-step heuristic process (construct, refine). This is analogical reasoning from completed, similar component estimates (e.g. Silverman, Moustakis, Liebowitz, Bokhari & Tsolakis, 1983; Silverman, 1985; Bedewi, 1995). That is most software project managers reuse prior estimates for analogical components that they are aware of. To support such efforts most project managers save past estimates for future reuse, or they obtain them from trusted colleagues who recently had to complete such an estimate. These saved estimates constitute informal repositories that exist in electronic spreadsheets, gantt charts, and documents on various managers’ personal computers. As organizations migrate to CASE tools and groupware (e.g. Lotus’ notes), these repository artifacts are increasingly available to a wider set of users. Further details on the software estimation problem, and on the specific groupware-based reuse repository adopted for this study exist in Section 2 of this paper. Section 3 explains the design of the research to study intelligent multimedia repositories (IMRs). Results and discussion of results are in Sections 4 and 5. Before proceeding to these sections, however, we first present a short discussion of multimedia and intelligent multimedia.

INTELLIGENT MULTIMEDIA REPOSITORIES

445

1.1. NEED FOR GUIDELINES ON MULTIMEDIA

One of the central issues of this research concerns the ecologically valid representations of knowledge for reusable artifacts (in the estimation domain). What media should artifact knowledge be represented in a maximize reuse? Given the recent ‘‘explosion’’ of media alternatives, and the fact that it means many things to many people, this section discusses two main topics. First, a ‘‘definition’’ of multimedia as used in this research is presented. Second, the lack of satisfactory previous research on when and where to include diverse media is reviewed. From the broadest perspective, multimedia is the result of combining the communication, entertainment, and computer fields. From a practical perspective, some of the common elements of multimedia are found in television, film, graphic art, books, magazines, radio, and animation (Holsinger, 1994). According to Holsinger, none of the items mentioned above are ‘‘true’’ multimedia since they lack one important aspect, namely, interactivity. For example, currently no interaction is possible with a TV program (except maybe turning the channel). However, with Interactive Television on the horizon, consumers will have far greater control over what they watch, such as ‘‘controlling’’ the camera angle to watch a particular sporting event. This research breaks down multimedia into three groups: classic, derivative, and combined. Classic multimedia is what most people are familiar with, i.e. sound, video, graphics, and text. They are certainly the most prevalent in our day-to-day life. Derivative multimedia are subsets of classic multimedia. For example, in project management, Gantt charts and spreadsheets could be used for capturing and conveying schedule information. Gantt charts are a type of graphics, whereas spreadsheets are tables whose elements are most often text. A fax is another example that could potentially be thought of as having two derivative states. When a fax is first received, it is an image, a subset of graphics. However, it could also be converted to text through the use of optical character recognition software. The advent of ‘‘container’’ type applications, such as Electronic Conferencing, and workgroup tools (e.g. Lotus notes) allows one to define the combined multimedia group. For example, project reviews could be performed through an electronic conferening tool. In turn, this tool would capture project status in text, schedule in Gantt chart, lessons learned in audio, etc. This research will also treat the following items as combined multimedia: Projects Repository, Attribute Importance, Estimation Agent, and Help. Turning to the literature, in the domain of software estimation there are no references to studies that look at how software estimators improve from classic, derivative, or combined multimedia to facilitate estimation. In searching more broadly for artifact design guidance, most of the literature is aimed at a level above this. While useful, it is aimed at the interface or at the general-purpose level, rather than at the artifact knowledge presentation issue. For example, since the early 1980s, research has shown the importance of familiar metaphors for ease of use (Shneiderman, 1982). Vincennes and Rasmussen (1992) further show the importance of using ecological interfaces to reduce cognitive burden. Ecological interfaces are ones that look beyond familiar metaphors, and try to represent task knowledge at

446

B. G. SILVERMAN AND N. BEDEWI

the level at which users need it during task performance. This reduces transformation, conversion, and other knowledge manipulation needs. Vincennes and Rasmussen (1992) indicate ecological interfaces may actually improve performance relative to familiar metaphors, yet research is needed to identify ‘‘ecological artifacts’’ for reuse repositories. At the repository-wide level, there a proliferation of guidelines on design of hypermedia, in general, and of homepages, in particular. These offer (often conflicting) suggestions for media quantity on each screen; placement of that information around the screen; depth and breadth of browsable linkages; and so on (Blattner & Dannenberg, 1992; Gray et al. , 1993; Maybury, 1993; Nabkel & Shafrir, 1995). By adopting a consistent set of such suggestions one can improve the coherence of and navigation through a repository. Equally useful are the interactive multimedia guideline studies in the general literature, most of which are at the same level as the other principles already enunciated in this section (e.g. Knight, 1992; Reeves, 1992; Park & Hannafin, 1993). For instance, ‘‘use familiar metaphors’’ is also a design principle recommended in the Park and Hannafin (1993) study. The media selection guidance that’s offered (e.g. use video for showing action scenes) may be thought of as ‘‘first level’’ guidelines. These guidelines are necessary, but insufficient, for the purposes of selecting appropriate mixes of classic, derivative, and combined media in the design of specific reuse artifacts for a given domain. In 1992, Silverman and Murray completed a study on factors affecting human – computer collaboration in a combat simulator repository. This study attempted to follow a number of the guidelines just mentioned. It compared users in traditional repository database browse and query mode to users who could depict the combat scenario they wanted to simulate by drawing it on a terrain map with the help of a map library, a set of draggable combatant icons, a smart scenario generation template, and a conversational intelligent agent (case based reasoner) that collects the reusable components from the repository and / or engages in dialogues to help the users find suitable substitutes (see Murray, Feggos, Weinstein & Silverman, 1987; Murray, 1988). This elevated the dialogue to a more ecologically valid level, and caused the machine to adapt to the humans’ domain. The results show significant improvements occur when repositories are domain-engineered well enough to: (1) replace standard menu and button interfaces with familiar metaphors; (2) replace unstructured artifact browsing and repository navigation with an ecological, whole-task-structuring interface (smart scenario template); and (3) substitute conversational, task-knowledgeable agents for direct search construction (Murray, 1988; Silverman, 1992b ). Elements of this kind of capability are increasingly called for by designers of CASE environments using terms such as ‘‘transformational programming’’ (Lowry, 1992; Winograd, 1995), ‘‘method abstraction and mapping’’ (Gennari, Tu et al. , 1994), and ‘‘reuse driven software process’’ (Hodges, 1995), among others. While the research just mentioned elevates the overall retrieval dialogue to a more ecologically valid level, it stops short of solving the individual artifact content problem. Once an artifact is retrieved, one is still left with the problem of how best to convey its knowledge content to the users. None of the authors cited thus far have yet provided specific guidelines as to when different media should be used to

INTELLIGENT MULTIMEDIA REPOSITORIES

447

represent individual artifact knowledge to respository users. That gap, in part, created the need for the current study. 1.2. AGENTS AND CONVERSATIONAL INTERFACES

An important trend in human – computer interaction is to enhance user support through the use of intelligent, conversational agents. That is, as environments become overly feature-rich, agents can be added that respond to users’ goals and intentions and that help users perform the construct and refine steps. Intelligent or conversational agents are computer programs that recognize organizational contexts, teamwork interaction needs, and humans’ information management activities. They autonomously anticipate and carry out low level tasks that improve productivity in these settings. To the extent possible, they hide the details of these tasks from the users, interacting with users at the overall goal level. These agents may, at times, need to ask the user for specific instructions, explain how they’re conducting a task, or ask for clarification. In general though, an intelligent agent should be able to determine intentions; manage unexpected changes in the conduct of its tasks; and adapt and learn over time (e.g. Seel, 1991; Sheth & Maes, 1993; Etzioni & Weld, 1994). Some examples of agents’ tasks are anticipating / carrying out simple information handling tasks and providing reminders, alerts, suggestions, criticisms, and adaptive learning in the team oriented setting. For instance, a group authoring environment such as Lotus notes can include agents that variously route work to proper recipients, remind team members when document versions are due, and alert members when changes are added by others. Still other agents help a user keep track of team members and how to reach them. Similarly, a CASE environment performs similar ‘‘conversational’’ services for the programming team. Over time as the group repository grows larger, such agents could form the basis of suggesting artifacts in the constructive step as well as adaptations during refinement (e.g. Silverman, Chang & Feggos, 1989; Silverman, 1994; Wooldridge & Jennings, 1995). One of the goals of this research is to explore what services repository users want of intelligent, conversational agents. In order to study this role, this effort examined the use of two types of agent approaches: critiquing (for refinement) and case based reasoning (for construction and for repository updates). The Critiquing Approach is a way of presenting a reasoned opinion about a product or action and in engaging in a dialogue about that opinion. It is analogous to human critiquing in real life. Critics are composed of rules or specialists that present ‘‘for / against’’ arguments and explanations for different issues on a given product or action the user has initiated. They are suited for ill-structured and complex problem domains since their purpose is not to present a primary source or complete solution but to interact as a set of cooperative specialists working with the user towards a solution. The estimation assistant presented below is this type of agent. A Case-Based Reasoning (CBR) is the second type of agent used here. It is ideally suited to constructive step support, e.g. Silverman et al. (1983), Kolodner (1993), Silverman, Chang and Feggos (1989). Basically, a CBR system examines the user’s description (however incomplete) of the target problem, compares it with a

448

B. G. SILVERMAN AND N. BEDEWI

repository of previous cases, and retrieves the most similar case(s). The similar cases are offered to support the user and as contrast to users’ unaided judgmental estimations. A CBR-based agent also could contain an adaptation facility (or critic) for adjusting between the previous case to the current problem, or it could be left to the user to perform the refinement step. There are two added strengths to a CBR-based repository agent. First, it only needs partial knowledge of the domain, which is very useful in ill-structured domains such as design or estimation. Second, its performance improves over time as new (good and bad) cases are added to the respository. Such an agent is included in this study as will be elaborated shortly.

2. Intelligent multimedia repository (IMR) for reuse of artifacts The previous sections describe multimedia and intelligent agents. This section puts these two topics together in the context of an intelligent multimedia repository (IMR) for reuse of project estimation artifacts. Figure 1 provides an overview of this synthesis. The IMR is managed by a piece of groupware on the left side of Figure 1 that less distributed user clients share a server’s reuse artifacts. This component supports client / server file sharing. On the far right of Figure 1 is the server. Through their client’s interface, the users may directly access the reuse artifacts and the various media these artifacts come in, or that help explain details about an artifact (derivation, assumptions, purpose, etc.). In the middle are two intelligent agents. One agent uses case-based reasoning to support the respository users in their reuse decisions, including making reuse suggestions. The second agent is an ‘‘estimation Artifacts/media Repository groupware (Lotus notes)

Problem description

Primary user interface

Help

Data collection

Conversational critic/agent (CBR esteem) D D E c o m m u n i c a t i o n

Previous cases

Attribute importance

Estimation assistance

Secondary user interface

Video L a u n c h e d e x e c u t a b l e

Audio

Text

Graphics

Electronic conference

Spreadsheet

Gantt chart

FIGURE 1. The IMR consists of a piece of groupware that connects users on client PCs to the server and its multimedia artifacts. A conversational CBR agent recommends reusable artifacts to users, while an estimation assistant supports / critiques user actions.

INTELLIGENT MULTIMEDIA REPOSITORIES

449

assistant’’ for estimation decision support and for critiquing / repairing inappropriate heuristics. Figure 1 also shows further detail about each of these three components. This detail is relevant to how the components supported users during the experiment conducted here. The repository is managed by Lotus notes, a workgroup tool that serves four main functions. First, it is the IMR’s primary client user interface, e.g. task descriptions (hypothetical projects) are presented to the respondents through this interface, along with the Q&A for the Target Attributes’ values. Respondents also provide their project estimates through this interface. Second, Help is provided through the use of PopUps (respondent clicks a field / label surrounded by a rectangle, and more information ‘‘pops up’’), DocLinks (similar to hypertext), and demos (screen captures with motion and audio demonstrations of different procedures). For the purpose of this research, Help is considered one of the experimental media. Third, protocols are collected and the final on-line questionnaire is also answered through Lotus notes. Fourth, it is the mechanism for sending commands to the case-base reasoning tool described below. The use of Lotus notes is necessary since the CBR tool used here had a simpler user interface that was not very user friendly. Esteem (÷ 1993, Esteem Software, Inc.) is an off-the-shelf case based reasoning tool that was embedded to test the intelligent agent features of interest in the IMR. It is used to hold the domain structure (attributes), portions of the experimental domain’s reusable cases, the functionality for changing the case’s attribute weights, a rule base for estimate adjustment, and the capability to ‘‘launch’’ other applications. Basically, Notes sends a Dynamic Data Exchange (DDE) message containing target case information to Esteem whenever a respondent presses a button for retrieving previous cases, weight adjustment, or estimation adjustment. In response, Esteem provides its own user interface for the appropriate function (button) selected. For the purpose of this research, the capabilities to ‘‘house’’ cases, change attribute information, and invoke the estimation assistant are considered three different media. Esteem’s strengths are two-fold. First, it numerically ranks the problem descriptions of previous cases for similarity to a new target problem based on attribute matching. The user can manipulate a number of attribute parameters that impact the search result, however, due to time constraints they were only exposed to changing the attributes’ importance (weight) in the trails described below. Second, is its capability to invoke a rule base for authoring added conversational agent support. For this research, the rule base was developed as the ‘‘estimation assistant’’ to adjust the cost and effort of the selected ‘‘best’’ matching previous case to the target case. For the purpose of this research it is a simple rule-base. It compares the six attribute values of the target problem to the selected previous case and recommends a ‘‘%’’ adjustment for the cost and effort. The general form of the rules is as follows. IF value of kattribute namel:Target Problem is Not Equal to value of kattribute namel:Previous Case THEN Recommend an ‘‘X%’’ adjustment

450

B. G. SILVERMAN AND N. BEDEWI

The specific attributes used in the rules will be described shortly. The adjustment factors ranged from 5 to 20%. These adjustment factors are not based on any empirical data. They are intended mainly to demonstrate relative importance and variety of adjustments. The respondents had the option of accepting the adjustments recommended or setting them to whatever value they liked. The third component of the IMR holds the reusable artifacts. The reuse artifacts are multi-media objects in native applications. These consist of the following Artifacts / Media / Application triplet combinations. Naturally, every time the respondent ‘‘launches’’ one of those artifacts’ tools, they have to use its respective interface.

Artifacts / Media / Applications $ Project Description / Text / Microsoft Write $ Algorithm Example / Video / Microsoft Video for Windows $ Lessons Learned / Audio / Microsoft Media Player $ Conference Meeting Information / Electronic Conferencing / Intel ProShare $ Algorithm Flowchart / Graphics / Clear Software Inc. allClear $ Initial Schedule / Spreadsheet / Microsoft Excel $ Initial Schedule / Gant Chart / Microsoft Project

Exactly what media to use with each specific artifact is the focus of this research. For starting values, this research used first level principles mentioned earlier (e.g. ‘‘familiar metaphors’’) and common sense. Thus for example, having a video clip of someone reading a script of lessons learned would be wasteful. This would be better served through the use of Audio alone. The actual domain of the IMR was restricted to one that would be widely known, so respondents could be readily found for the experiment. Specifically, the repository was set up for a domain of sorting routines. In total, six reusable past sorting cases were incorporated into the IMR. The six righthand columns of Table 1 list the six sorting routine cases that serve as the previous projects for this research. The rows of Table 1 show 22 attributes of each reusable case that are stored as the repository artifacts. The last six hold pointers to the supporting multimedia that was mentioned in the previous section. Six of the first 10 attributes served as the ‘‘target’’ problem definition: SortAlgorithmName, ItemsSorted, MaxRecordsToSort, ImplementationLanguage, ImplementationHardware, and ImplementationOpSystem. That is, the users are presented with a question (and choice of answers) for each one of these attributes. The case based reasoning agent uses those answers to ‘‘score’’ the previous cases relative to the target problem currently being solved. Other attributes in Table 1 also could be part of the matching process, and in a ‘‘real’’ domain there would be many more attributes. However, due to the time constraints of this experiment, and to allow the respondents time to explore the artifacts / media presented, only six attributes were selected as representative.

MFLgFilesRadix01 RadixSort Alphanumeric 10 1 100000 No Yes COBOL IBM 3070 MVS 20000 30000 2 3 PersonMonths No c: \ library \ sched1.mpp y : \ library \ radxsort.avi c: \ library \ lessons1.wav c: \ library \ project1.mg c: \ library \ radxsort.aci c: \ library \ project1.wri

Attributes

SortAlgorithName ItemsSorted NumberOfFieldsPerRecord NumberOfFieldsToSort MaxRecordsTcSort SortFieldsOrderImportant SortDirectionImportant ImplementationLanguage ImplementationHardware ImplementationOpSystem PlannedCost ActualCost PlannedEffort ActualEffort Effort Units WithinCost / AndSchedule InitialSchedule VideoOfAlgorithmExample AudioOfLessonLearned ConferenceMgInformation AlgorithmFlowchart ProjectDescription

RadixSort Alphanumeric 10 3 20000 Yes Yes FORTRAN 486 Windows 15000 25000 1.5 2.5 PersonMonths No c: \ library \ sched5.mpp y : \ library \ radxsort.avi c: \ library \ lessons5.wav c: \ library \ project5.mtg c: \ library \ radxsort.acl c: \ library \ project5.wri

MFPortRadix01 QuickSort Integers 10 3 5000 No Yes C 486 Windows 10000 10000 1 1 PersonMonths Yes c: \ library \ sched4.mpp y : \ library \ quiksort.avi c: \ library \ lessons4.wav c: \ library \ project4.mtg c: \ library \ quiksort.scl c: \ library \ project4.wri

MFPortQuick01 QuickSort Alphanumeric 5 2 1000 No Yes FORTRAN IBM 3070 MVS 20000 25000 2 2.5 PersonMonths No c: \ library \ sched2.mpp y : \ library \ quiksort.avi c: \ library \ lessons2.wav c: \ library \ project2.mtg c; \ library \ quiksort.acl c: \ library \ project2.wri

MFMdFilesQuick01

Case Name

TABLE 1 Cases in the repository

SelectionSort Letters 5 1 1000 No No C IBM 3070 MVS 10000 10000 1 1 PersonMonths Yes c: \ library \ sched3.mpp y: \ library \ selsort.avi c: \ library \ lessonsc3.wav c: \ library \ project3.mtg c: \ library \ selsort.acl c: \ library \ project3wri

MFSmFilesSelec01

SelectionSort Integers 5 3 2000 Yes Yes C Pentium OS2 10000 15000 1 1.5 PersonMonths No c: \ library / sched6.mpp y: \ library \ selsort.avi c: \ library \ lessons6.wav c: \ library \ project6.mtg c: \ library \ selsort.acl c: \ library \ project6.wri

MFPortSelect01

INTELLIGENT MULTIMEDIA REPOSITORIES

451

452

B. G. SILVERMAN AND N. BEDEWI

3. Design of the empirical study This research explores a range of research questions concerning what intelligent multimedia features users will and will not use as summarized in the following overview discussion. 3.1. RESEARCH QUESTIONS

The driving question behind this research is: do (or do not) the use of multimedia and intelligent agents foster the reuse of artifacts from a repository? This question is too broad if the answers are to guide the design of interfaces. Repository designers want to know which intelligent agent services and what multimedia artifacts work best and are accepted, adopted, and used. To get at these issues, questions must be asked about each component (media / artifact, agent service, etc.) of the IMR such as: $ PURPOSE: Is the reason for including that component apparent to the respondents? $ ROLE: Do the respondents access the component? find a role for it? $ USEFULNESS: Is the IMR and its components useful in respondents’ work (e.g. estimating) processes? $ IMPACT: What is the impact of the components on the respondents’ performance? $ IMPORTANCE: How important are the components to the organization’s work groups? These types of questions guided the design of specific questionnaires that are displayed with the results presentation in Section 4. 3.2. SAMPLE

A total of 33 project leaders / managers / sr. developers participated in this study, 11 from NASA, and 22 from a number of private organizations. These respondents averaged over 10 years experience, and 30 had at least 5 years experience as both software developers and project leads. The respondents consisted of 11 females and 22 males with over half holding MS or PhD degrees (all had college degrees). These respondents thus are a sample of expert practitioners. 3.3. TASKS

The respondents’ task was to provide cost and duration estimates for two hypothetical projects in the sorting domain. Project 1 Description . Assume that you are a manager in a company that specializes in adapting their library of Sorting tools to solve business problems across multiple platforms, operating systems and software languages. One of your clients has requested that you estimate how much it would cost and how long it would take to develop a sort algorithm that takes a list of integers (200 – 3000 elements), and sorts it in ascending order. This algorithm has to be implemented in C11, and will be running in an IBM PC compatible environment.

INTELLIGENT MULTIMEDIA REPOSITORIES

453

Project 2 Description . Again, assume that you are a manager in a company that specializes in adapting their library of sorting tools to solve business problems across multiple platforms, operating systems and software languages. You have been assigned another project (independent of Project no. 1), this time, another client would like you to develop a PC based application that would catalogue and track all the project documentation they generate. The maximum number of documents to track would not exceed 1500. The customer would like to be able to sort their catalogue by at least Author, Document Type, and Date Composed; descending or ascending order per field. Your task is to estimate the cost and duration for developing the sorting program portion of this application.

3.4. INTERVENTION ASSIGNMENTS

The whole experiment was conducted on-line, without experimenter intervention. All respondent dialogues, project taskings, questionnaires, etc. were administered through the Lotus notes interface (this looked and behaved like a home page), unless another application was temporarily launched for viewing a specific media / artifact. In the first project, respondents were only given links to access an online calculator and an empty spreadsheet (Excel). For the second project they had buttons and hotlinks to access the IMR’s agents and artifacts, including the case based reasoning and estimation agents and the full set of multimedia artifacts in the repository. This environment was setup with the assumption that the respondents would have NO prior knowledge with any of these tools, except the knowledge of using a mouse. For this study, the following Artifacts / Media pairs were used. Initial Project Schedule / Gantt Chart Initial Project Schedule / Spreadsheet lessons Learned / Audio Clips Project Description / Text Project Attributes / Text Algorithm Flowcharts / Graphics Project Initiation Meeting / Electronic Conference Sort Algorithm Example / Video Clips Analogous Projects / Projects Repository Project Attribute Weights / Atribute Importance Estimate Adjustments / Estimation Assistant Help—Instructions / PopUps Help—Sort Algorithm Descriptions / DocLinks Help—How To.. / Demos The choice of the artifacts / media pairs was intended to represent a sampling of the variety of information encountered in the software estimation and sorting domains. The level of detail in each artifact attempted, through hypermedia, to appeal to different respondent types, namely; managers, project leads, and senior developers (those wanting the most detail).

454

B. G. SILVERMAN AND N. BEDEWI

FIGURE 2. Screen illustrating a typical question about a repository attribute. This screen shows the question, menu of answers, and help in the form of popups (encased in the box) and doclinks (represented by the tiny page icons).

3.5. PROCEDURE

The following is the all-electronic procedure. Respondents are given a general description of this experiment. They are asked to read an on-line consent form, and press OK on the screen to begin the experiment. Next, they are presented with the description of Project no. 1 followed by a form asking them to provide the estimates. This serves as a baseline of work performance without either the respository or its intelligent multimedia interfaces, artifacts, and components. After completing Project no. 1, respondents are presented with the description for Project no. 2, followed by a set of six questions in sequence. These questions represent the six attributes that are used by the case based reasoning agent for analogical comparison. Figure 2 shows the screen structure for each of these questions. When answering each question, respondents are presented with a standard set of choice buttons at the base of the screen. Pushing these buttons from Figure 3 precipitates most of the interactions one can have with the repository. We describe these six buttons in the following six bullets. $ View Projects Repository. This triggers the CBR agent to retrieve a closest match, ranked list of analogous projects. Figure 4 offers an example of such a screen. The ranking is based on a weighted score of the answers to the six attribute questions of the Figure 3 type of screens. At this point respondents can select one of those previous projects to get more information, e.g. textual information such

INTELLIGENT MULTIMEDIA REPOSITORIES

455

FIGURE 3. Screen illustrating a typical set of choices a respondent sees after each question. This screen shows a choice question near the top of the screen, choice buttons at the bottom of the screen, and help in the middle of the screen in the form of PopUps (encased in boxes), a DocLink (represented by the tiny page icons), and Demos (represented by VideoCams).

as, actual vs. planned cost, video clip describing an example of the sort algorithm used, and so on. Figure 5 illustrates the screen that users encounter when they drill down to examine a given reusable past case. $ Change Attribute Importance. This allows the respondents to decide how important each attribute is for the CBR agent to determine closest match. Pushing this button results in a simple table where users can edit the weights on the six attribute values. $ Invoke Estimation Assistant. Based on the estimation agent’s rule-base, this

FIGURE 4. Screen illustrating the CBR agent’s recommended reuse cases sorted by relative score. Score is computed based on user’s answers to six attribute questions of earlier screens.

456

B. G. SILVERMAN AND N. BEDEWI

FIGURE 5. Screen showing partial detail of a given case. Only six case features are displayed at a time. Users may scroll down / up to view other features, or double click on media-artifact buttons (right side of screen) to view that item.

button suggests an estimated cost and duration by comparing Project no. 2 attribute values (i.e. answers to the six questions) to the previous (analogous) project selected. Figure 6 illustrates one of the sequence of screens involved in the dialog with the estimation assistant. At other times in the dialogues users encounter screens (1) asking if they want a recommendation, (2) asking for the rule base adjustment weights, and asking if they want an explanation of the rules leading to the estimate recommendation. $ Perform Manual Estimate. If they do not wish to have the estimation assistant automatically compute the estimate, this button allows the respondents to provide their own estimates at any time they feel ready. Figure 7 shows how this manual estimate is requested, as well as how support is offered and how the is rationale elicited. $ Project no. 2 Requirements. This button brings back a popup screen describing the task for the project. $ Undo Last Answer. This final button on Figure 3 (except for continue and exit) allows the respondents to change their answer to the LAST attribute question encountered.

FIGURE 6. One of the estimation agent’s screens. This screen is making a recommendation for altering a reused case’s cost estimate based on the agent’s internal rules about hardware differences.

INTELLIGENT MULTIMEDIA REPOSITORIES

457

FIGURE 7. Screen illustrating how the user may perform a manual estimate for project cost and duration. The screen offers support tools (spreadsheet, calculator), requests units for the estimates, and elicits the rationale for the estimates as well. At the base of the screen is a demo videoIcon.

This cycle of attribute questions-button choices-subscreens repeats for each of the six questions. This allows the respondents ample opportunity to select which artifacts / media pairs they want to explore and to drill down to whatever levels of detail they desire. If they have not already used it, after completing the answers to the six questions, the respondents are presented with a form (Figure 4) asking them to provide the estimates (similar to Project no. 1). Once an estimate is provided for Project no. 2, it signifies the end of the estimating tasks, and the respondents are asked to complete an on-line questionnaire. The questions in the questionnaire concern their opinions of each of the media / artifact pairs, and of the various screens of the repository and the intelligent agent. This approach does not compare the repository with and without intelligent multimedia. As mentioned at the outset, the field is still immature and we did not know enough to design a comparative study. Instead this design provides insight into the research questions about what elements or components respondents prefer in the repository interface. Given the descriptive nature of this study, and the variety of artifacts / media pairs studied, time was predicted to be the biggest constraint. It was planned that the experiment would last 1.5 h, 45 min for the task performance, and 45 min for the questionnaire.

458

B. G. SILVERMAN AND N. BEDEWI

3.6. MEASURES

Three types of information was collected from each respondent: demographic data, performance statistics, and reactions / protocols. Demographic data was captured by a brief questionnaire. For the performance statistics, the system captured keystroke data such as answers provided, media / artifacts encountered, etc. Often performance was computed by comparing task results on Projects 1 vs. 2. Protocol information collected by questionnaires after the two project task was intended to capture, among other things, what the respondents got from each of the media / artifacts, and their impressions of the IMR’s various services. In general, a number of questions required the respondent to select a value from a five point Likert scale (strongly agree, agree, undecided, disagree, strongly disagree), other questions were open ended. 3.7. STUDY OBJECTIVE

One may wonder why all the respondents were assigned two different project tasks—one with and one without the repository. That is, why we did not just divide our respondents in half and have half perform both tasks without the repository, and the other half perform both tasks with the repository. This would be a useful design if we were simply interested in benchmarking or baselining the performance differences. But baselining was not our goal. Recall this is an exploratory study, intended to elicit what all the participants thought of the respository and its media-artifacts. We wanted the largest possible set of media-artifact reactions. Still, to allow for some baselining, Task 1 is significantly simpler than Task 2. Thus any performance improvement of Task 2 (with repository) relative to Task 1 (without repository) could be potentially insightful, though not definitive. A follow up baseline study would be needed for definitive results.

4. Results The results in this section address the questions of purpose, usefulness, role, impact, and importance of the overall IMR as well as of the specific value of various media / artifacts and agents in the repository. In general, the results reflect both the respondents’ perceived answers from reaction questionnaires, interleaved with actual results from their task performance. The results throughout this paper are presented in tabular form. Each table of results is broken into part A and B. Part A shows raw scores on the Likert scale, where 5 is strongly agree, 4 is agree, 3 is undecided, 2 is disagree, 1 is strongly disagree, and 0 is did not encounter / no answer given. The cells of the table indicate the number of respondents who selected a particular score. Part B of each table includes three columns that summarize the results of each row. The column labelled ‘‘relative ranking’’ is computed as the sum of responses in the Strongly Agree and Agree columns in part A. In other words, relative ranking shows the raw count of all who are favorable to the item in the given row. The column labeled ‘‘normalized’’ is the ratio of relative ranking to total sample of respondants who encountered the item in a given row. This normalized score thus

459

INTELLIGENT MULTIMEDIA REPOSITORIES

reflects the fraction (percent) of the sample in favor of the item. The rows of each table are sorted according to the normalized score so the reader can get a quick sense of what the respondants favored most. Lastly, the column labeled ‘‘mean’’ is computed for each row as a ratio of the summation of the (Likert score 3 number of repondents) for each column of Part A divided by the number of respondants who encountered the item (i.e. 33 2 Number of 0 responses in Part A). The mean thus shows the relative likes and dislikes of all the respondents. As we discuss the tables, we will point out where the normalized rank differs from the ranking based on the mean. 4.1. QUESTIONS ABOUT IMR AND MEDIA / ARTIFACT ‘‘PURPOSE’’ The first items of interest were how many respondents understood the purpose of the repository in the context of the task. If they understood the repository’s purpose, the next questions concerned whether they encountered a given media (and its respective artifact) and whether they understood why it was in the repository. The tables discussed in this section summarize the findings. In Table 2(a), each cell holds a count of the total number of respondents that gave that answer, e.g. 10 respondents responded that they strongly agree with the statement that the reason for including the audio clips was apparent to them. Using the responses in Table 2(a), Table 2(b) (right three columns) was generated to provide a ranked list of whether the particular reason for including a medium was apparent or not. The numbers in the second column of Table 2(b) were generated by summing the respondents’ responses under ‘‘strongly agree’’ and ‘‘agree’’ from Table 2(a) for each medium. In general, these scores reflect an understanding of the reason that the repository itself and the diverse media / artifacts were included. Many of the lower score levels reflect the fact that, as expected, every medium was not encountered by each respondent (see last columns of Table 2(a)). This is due to a number of reasons. First, the respondents had about 1 h to perform both

TABLE 2 ‘‘Purpose ’’: respondents ’ ratings that ‘‘the reason for including a giy en media / artifact is apparent ’’

Media Gantt charts Help Projects repository Spreadsheets Textual information Estimation assistant Video Audio Attribute importance Electronic conferencing Graphics

Likert scale —————————— 5 4 3 2 1 10 15 22 1 13 14 8 10 10 3

9 17 9 12 17 15 13 11 13 9 8

1 1 3 1 2 3 4 2 1

1 1 1 1 1 1 1

1

1 2

0

Mean

Relative ranking

Normalized

14

4.53 4.42 4.55 3.86 4.30 4.25 4.17 4.20 4.14 3.75 3.36

19 32 31 13 30 29 21 21 23 12 8

1.00 0.97 0.94 0.93 0.91 0.91 0.88 0.84 0.82 0.75 0.73

19 1 9 8 5 17 22

460

B. G. SILVERMAN AND N. BEDEWI

estimation tasks. Second, the media / artifacts were developed to appeal to different levels of respondents (manager, leads, etc). Third, the respondents’ ‘‘exploration’’ path would have limited what they encountered. Fourth, it is also possible that some of the respondents listed under this column may have noticed the access path to a given artifact’s medium but chose not to launch it. For these reasons, the last column of Table 2(b) is computed. The numbers in that column are the same data as the second column of Table 2(b) except they have been divided by the number of respondents that encountered that medium. Thus they are normalized to discount this effect, as mentioned above. The results in Table 2(b) show three groupings of normalized media rankings: above 0.9, 0.8 to 0.9, and below 0.8. In the top category, one finds six media / artifacts, most of which were both widely encountered and exhibit high normalized scores. The exception is Gantt charts and spreadsheets which were scored highly when encountered, but appear to be encountered infrequently. The Spreadsheets and Gantt charts were included to convey schedule information. However, their presence was split between the six cases in the repository. If respondents did not explore all cases then the expectation was that about half of them would not encounter the spreadsheets and the other half would not encounter the Gantt charts. The fact that this occurred, may imply one of two things, (1) the schedules were not that important for the respondents to explore and (2) the respondents were under the influence of the ‘‘confirmation bias,’’ hence they were not exploring other cases once they ‘‘zeroed in’’ on a similar case [see Bedewi (1995) for a further discussion of judgment biases in respondent performance]. It is worth noting, that Graphics and Electronic Conferencing were both the least encountered and lowest normalized rank of any media. The graphic attribute was labeled in the project repository as ‘‘Algorithm Flowchart’’, and if a respondent did not feel that a flowchart of a sort algorithm was necessary for estimating, then they would have ignored it. The fact that half the respondents did not encounter Electronic Conferencing was interesting / puzzling. From pre-study interviews with 25 respondents drawn from the same population, it was concluded that the respondents rely on peer review / consensus as a method for estimating in real world projects. The electronic conference was intended to support this need. Again time could be a factor here, or perhaps respondents did not understand the purpose of the ‘‘Electronic Conferencing’’ button. For the most part use of the mean rather than the normalized score did not alter the three groupings of media (0.9, 0.8 to 0.9, and below 0.8) with one exception, spreadsheets. The spreadsheet drops out of Group 1 based on mean rather than normalized score. This may be because the mean is overly sensitive to whether respondents rated an item as strongly agree vs. agree. Thus the mean for spreadsheets would appear to contradict the normalized score. 4.2. QUESTIONS ABOUT IMR AND MEDIA / ARTIFACT ‘‘ROLE’’

4.2 .1 . Oy erall IMR role As with most decision making domains, if more information is available (up to a threshold), better decisions should be made. The next question asked the respondents

461

INTELLIGENT MULTIMEDIA REPOSITORIES

TABLE 3 ‘‘Role ’’: respondents reactions that they had sufficient information to perform the task (Project 1: without repository , Project 2: with repository ) (a)

(b) Likert scale —————————– 5 4 3 2 1

Project 1 Project 2

4 5

10 21

5 2

11 5

3 0

0

Mean

Apparent ranking

Normalized

0 0

3.03 3.79

14 26

0.42 0.79

if they had enough information to perform the task. As can be seen from Table 3, only 42% of the respondents ‘‘strongly agreed’’ or ‘‘agreed’’ that they had enough information for Project no. 1, compared to 79% for Project no. 2. Their feelings of insufficient information for Project no. 1 are interesting considering how simple that task was. Likewise their feelings are equally interesting that the IMR nearly doubles the information sufficiency for Project no. 2. Thus, the IMR fills a role, but not completely so. There is an information gap even with the IMR.

4.2 .2 . Specific media/ artifact roles The previous answers determined there is an overall role for the IMR of providing information. The next set of questions probed the precise informing role played by each of the individual artifacts in their respective media. Table 4(a) summarizes the respondents’ responses of how well the different artifacts / media provide information. Table 4(b), in turn, aggregates these into an apparent ranking and a normalized score of how the different artifacts / media support the IMR’s informational role. As before, the apparent rankings were derived by adding the respondents responses for strongly agree and agree, and the normalization was computed to account for media not encountered by the full sample of respondents. One of the interesting patterns in Table 4(b) is that the role and normalized rankings correlate fairly closely. Respondents found the artifacts about as informative as the frequency with which they visited these artifacts. The artifacts of the IMR that played the biggest role in helping them come up with their estimate were the top three artifacts: help, project descriptions and attributes, and analogous projects. Of somewhat lesser value, though still significant, were the next two artifacts, Gantt charts and video. Table 4(b) also reflects the fact that respondents found Gantt charts a more informative medium than Spreadsheets for conveying the schedule artifact. Use of the mean to interpret Table 4(b) does not change the findings except for two artifacts / media. Analogous projects / projects repository swaps places with

462

B. G. SILVERMAN AND N. BEDEWI

TABLE 4 Respondents ’ answers to : ‘‘does each media / artifact fulfill its information proy iding ‘‘role ’’?’’ (is the artifact / medium informatiy e ?) (a)

(b)

Artifacts / media PopUps, DocLinks, Demos / Help Proj. Descr. & Attributes / Textual Information Analogous Projects / Projects Repository Initial Project Schedules / Gantt Charts Sort Algorithm Example / Video Proj. Attribute Weights / Attribute Importance Proj. Initiation Meeting / Electronic Conferencing Lessons Learned / Audio Initial Project Schedules / Spreadsheets Project Estimates Adjustor / Estimation Assistant Algorithm Flowcharts / Graphics

Likert scale —————————– 5 4 3 2 1

0

Mean

Relative ranking

Normalized

16

15

2

4.52

31

1.00

16

15

1

1

4.47

31

0.97

18

12

2

1

4.50

30

0.94

8

8

3

14

4.26

16

0.84

8

11

2

3

9

4.00

19

0.79

7

14

5

2

5

3.93

21

0.75

5 10

6 9

1 5

2 2

18 7

3.80 4.04

11 19

0.73 0.73

1

9

2

2

19

3.64

10

0.71

6

14

9

2

2

3.77

20

0.65

1

4

2

4

21

3.00

5

0.42

1

1

textual Information but both remain in the top three artifacts / media. Lessons learned / audio moves up to the second grouping.

4.3. QUESTIONS ABOUT IMR AND MEDIA / ARTIFACTS ‘‘USEFULNESS’’

The next set of questions concerned the usefulness of the IMR and its media / artifacts in filling their role. For example, a reuse library should be useful for reducing overall effort and for reducing time to complete the task. Due to research constraints (i.e. funding of the authors, limited access to the respondents, and use of a hypothetical set of tasks) there was not direct way to observe and measure such reductions. Instead, respondent opinion polls were used to address this topic.

463

INTELLIGENT MULTIMEDIA REPOSITORIES

TABLE 5 Various questions concerning oy erall impressions (a)

(b)

Media NMR ‘‘Usefulness’’ for reducing organizational workload ‘‘Usefulness’ of IMR for reducing time to produce an estimate ‘‘Importance’’: the IMR would be accepted by subject’s organization If adopted, the IMR would be ‘‘important’’ to subject and organizational work If adopted, the IMR would ‘‘impact’’ organizational procedures

Likert scale —————————– 5 4 3 2 1

6

23

3

1

12

14

6

7

17

8

7

20

5

1

13

16

2

1

1 1

1

0

Mean

Apparent ranking

Normalized

4.00

29

0.88

4.12

26

0.79

3.88

24

0.73

4.00

27

0.82

4.18

29

0.88

4.3 .1 . Oy erall IMR estimation usefulness First, the respondents were asked if the IMR were adopted in their organization, would it reduce their workload. From Table 5 (row 1), 88% of the respondents ‘‘strongly agreed’’ or ‘‘agreed’’ with this statement. The following is a sampling of the respondents responses on the usefulness of the IMR to their workload. (1) Will be easier and faster to generate estimates for new projects, hence reduce the workload. (2) Fewer crises due to poor estimates. (3) Increase the workload in the beginning, but save time eventually. (4) It will make it easier for management to see why more resources are needed. (5) Will improve resource load balancing. (6) Reduced estimating time will provide more time for improving the estimates’ quality. (7) Workloads will increase if project budgets do not account for saving project information. Next, the respondents were asked if implementing the IMR in their organization would speed up the time required to produce an estimate. From Table 5 (row 2), 79% of the respondents ‘‘strongly agreed’’ or ‘‘agreed’’ with this statement. The following is a sampling of the respondents responses as to the role the IMR would play in speeding up their estimating: (1) Would not have to ‘‘run around’’ looking for previous cost data.

464 (2) (3) (4) (5) (6) (7)

B. G. SILVERMAN AND N. BEDEWI

Estimates will be faster once the initial setup and training is complete. If it is accessible on-line in a client / server environment. Less rework and information gathering. Less experienced people will be able to estimate. Consensus will be reached quicker and with more confidence. Will probably not help with new (research) type projects.

4.3 .2 . Specific media / artifact estimation ‘‘usefulness’’ Having established that the IMR is perceived as useful for reducing work / saving time, the next questions are to determine how the individual media / artifacts support these benefits. Tables 6(a) and 6(b) summarize the respondents’ responses. The most useful artifacts, as Table 6(b) shows, are pretty much the same as those that played the most informative role [earlier Table 4(b)]. An exception is that the IMR-generated project estimates were found to be useful, rising to the number

TABLE 6 ‘‘Usefulness ’’: which of the media / artifacts reduced workload and speed completion of the task ? (is the artifact / medium useful ?) (a)

(b)

Artifacts / media Analogous projects / projects repository Project description & attributes / textual information Project estimates adjustor / estimation assistant Lessons learned / audio Popups, doclinks, demos / help Project attribute weights / attribute importance Initial project schedules / Gantt charts Project initiation meeting / electronic conferencing Initial project schedules / spreadsheets Sort algorithm example / video Algorithm flowcharts / graphics

Likert scale —————————— 5 4 3 2 1 0 1

Mean

Relative ranking

Normalized

4.48

31

0.94

1

4.44

29

0.91

2 8

4.00 3.76

25 19

0.81 0.76

3.76

23

0.70

21

10

1

17

12

3

10 6

15 13

2 2

4 2

6

17

6

4

4

15

3

5

1

5

3.57

19

0.68

6

6

2

2

3

14

3.53

12

0.63

7

1

4

3

18

2.80

7

0.47

1

5

5

2

1

19

3.21

6

0.43

1

7

7

9

1

8

2.92

8

0.32

1

3

6

3

20

2.15

1

0.08

2

465

INTELLIGENT MULTIMEDIA REPOSITORIES

three spot. One suspects these estimates while not informative, were useful for other reasons we will discuss in the interpretation of results section. Another exception is that the first two most informative items from Table 4 have fallen to fifth and tenth place in the usefulness on the information they provide. Clearly not all information is equally useful. None of these findings change if one uses the mean rather than the normalized score. 4.4. QUESTIONS ABOUT IMR AND MEDIA / ARTIFACT ‘‘IMPACT’’ It is often interesting to assess if respondents’ perceptions about roles and usefulness of a technology match the way they actually use that technology. The opinions just presented indicated the primary use of the IMR is to save time / reduce workload. Data presented in this section show the IMR provides other benefits as well.

4.4 .1 . Oy erall IMR decision impact This is, of course, the primary area where task performance impacts, rather than strictly reaction assessments, could also be collected. In particular, the actual cost and duration estimates from the respondents are the estimation task outputs. These results show the extent of the repositories impact. As the first row of Table 7(a) shows, the mean cost estimate across the respondents rose from $6032 to $11 320 when the repository was introduced. Likewise the standard deviations dropped to less than half of the pre-repository amount [second row of Table 7(a)]. Similar mean and standard deviation shifts were observed for duration estimates. These results suggest the respondents are significantly shifted in both their anchoring (mean) and adjusting (deviations) processes by the repository. (To verify this finding, one would have to redo the results with 33 more respondents using a repository on Task 1 and no repository on Task 2—see ‘‘study objective’’ in Section 3). Moving back to reaction data, the respondents were next asked if they thought TABLE 7 ‘‘Impact ’’ of the oy erall repository (a) Respondents ’ project cost estimates for Project 1 & Project 2 Cost ($) Project no. 1

Project no. 2

6032.50 4597.78

11 320.00 2190.40

Mean S.D.

(b) Respondents ’ confidence that their estimates will be on target

Mean S.D.

Project no. 1

Project no. 2

Project no. 2 – Project no. 1

62.88 26.70

81.19 15.10

18.31 211.70

466

B. G. SILVERMAN AND N. BEDEWI

TABLE 8 ‘‘Impact ’’: how likely respondents thought their cost and duration estimates would be met (Project 1: without repository , Project 2: with repository ) (a)

(b) Likert scale —————————– 5 4 3 2 1

Project 1 Project 2

5 4

9 23

12 4

6 2

1 0

0

Mean

Apparent ranking

Normalized

0 0

3.33 3.88

14 27

0.42 0.82

their estimates for cost and duration Table 8, 42% of the respondents ‘‘strongly agreed’’ or ‘‘agreed’’ with that statement for Project no. 1, compared to 82% for Project no. 2. The respondents were also asked the following questions: ‘‘On a scale of 0 – 100, how confident are you that your estimates would be on target?’’. As can be seen from Table 7(b), the average improvement in confidence between Project no. 1 and Project no. 2 was 18.31%. These last two reaction questions demonstrate that the respondents’ confidence in their estimates improved with the IMR, and they felt that they could meet the cost & duration commitment. Specifically, 91% of the respondents indicated that the IMR improved their estimates, and only 9% were undecided. Thus a second role for the IMR, in addition to reducing effort / saving time, would seem to lie in improvement in the quality of work as well. 4.4 .2 . Specific media / artifact decision ‘‘impact’’ Having determined how the IMR improves estimate quality, the next question was to determine if the estimates would have been different without each of the various media. Earlier sections already probed the artifacts that are useful. The goal of these questions is to isolate what media do and do not help improve estimate quality. These answers begin to focus attention on the issues of what media seem fruitful. Table 9 summarizes the respondents responses about media impact. In Table 9(b), one can see that the respondents would like any repository, with the text based media having the best impact of the remaining items. Clearly these responses reflect the respondents’ strongly felt need (expressed in pre-survey discussions) to fill the gap and for organizations to start building repositories. These findings basically remained unchanged if one uses the mean rather than the normalized score. 4.5. QUESTIONS ABOUR IMR AND MEDIA / ARTIFACT ‘‘IMPORTANCE’’

4.5 .1 . Oy erall IMR importance We define importance as consisting of three factors: acceptance; inter-group effect; positive impact on procedures. Given the positive feedback from the respondents regarding the IMR, we were also interested in their projection on how important the IMR would be to their respective organizations. This was accomplished through a series of questions that are documented below. From Table 5 (row 3), 73% of the respondents ‘‘strongly agreed’’ or ‘‘agreed’’ with the statement that this tool would be accepted in their organization.

467

INTELLIGENT MULTIMEDIA REPOSITORIES

TABLE 9 ‘‘Impact ’’ of each medium for improy ing performance (a)

(b)

Media

Likert scale —————————– 5 4 3 2 1

Projects repository Textual information Estimation assistant Help Gantt charts Attribute importance Audio Video Spreadsheets Electronic conferencing Graphics

16 14 17 12 10 9 6 14 8 3 2 13 4 5 8 3 1 4 1

2 1 8 3 3 7 9 6 6 7 6

2 4 8 1 5 5 10 2 2 3

2 4 1 2 2 2 3 3

0

Mean

Relative ranking

Normalized

1 2 2

4.31 4.38 3.81 3.42 3.26 3.36 3.20 2.77 2.93 2.75 2.38

30 29 19 20 11 15 9 8 4 4 1

0.94 0.91 0.61 0.61 0.58 0.54 0.36 0.31 0.29 0.25 0.08

14 5 8 7 19 17 20

The following is a sampling of the reasons why the respondents though this tool would be accepted (and important). (1) (2) (3) (4) (5) (6) (7) (8)

Information would not be lost from one project to another. A standard estimating method could emerge out of this tool. Encourages organized approach to project planning. Access to ‘‘lessons learned’’ would be very valuable. Faster estimating would be possible. A comprehensive set of tools would be available to the manager. Estimates would be more accurate. Would improve communications across organizations.

Some respondents did ‘‘voice’’ some concern such as those following. (1) (2) (3) (4) (5) (6)

Garbage In Garbage Out still holds. Needs to be modified to suit specific organizational structure. It would not support detailed estimating. Populating the database has to be very easy and very quick. Needs to be expanded to cover more variables. Would have to prove its worth.

The respondents were also asked if the IMR would affect the way they worked with other groups in the organization. As shown in Table 5 (row 4), 82% of the respondents either ‘‘strongly agreed’’ or ‘‘agreed’’ with that it would affect them. The following is a sampling of the respondents responses as to how the IMR might affect (and be important to) the way they work with other groups. (1) (2) (3) (4)

Would eliminate some of the assumptions made during the estimation process. Some people might not want to use it since their mistakes will be seen. There would be more benefit from a group effort. Would identify weaknesses in the estimation process.

468

B. G. SILVERMAN AND N. BEDEWI

Information would be shared regardless of location. Easier to achieve consensus and management buy-in. Easier to justify estimates to other groups. Faster Estimating would be possible. Estimates would be more consistent and standardized. Could slow the groups down if more information tracking is needed. Could eliminate problems with documentation ‘‘hand-offs’’. Increased business with other organizations if there is confidence in the estimates. (13) Allows reuse without ‘‘bugging’’ original project team. (5) (6) (7) (8) (9) (10) (11) (12)

Next, the respondents were asked if implementing the IMR in their organization would impact existing procedures. From Table 5 (row 5), 88% of the respondents ‘‘strongly agreed’’ or ‘‘agreed’’ with this statement. The following is a sampling of the respondents responses as to how the IMR would impact existing organizational procedures. (1) (2) (3) (4) (5) (6)

More informed and formal estimation process. People would have to do detailed cost and time estimates. Historical data would have to be entered and maintained. People will be forced to think about project planning. More evaluation of projects after completion. People would be required to get corporate histories to justify their estimates.

Procedures will need to be implemented to insure that the tool is used properly. 4.5 .2 . Specific media/ artifact ‘‘importance’’ Having determined overall IMR importance, we next investigated how important the respondents thought each medium was. Table 10 summarizes the respondents responses to that question. The results reaffirm the previous finding on ‘‘impact’’ TABLE 10 ‘‘Importance ’’ of each medium in the IMR (a)

Media Projects repository Textual information Estimation assistant Help Gantt charts Audio Attribute importance Spreadsheets Electronic conferencing Video Graphics

(b) Likert scale —————————— 5 4 3 2 1 16 12 16 10 7 17 6 16 3 9 3 12 2 14 1 6 8 3 7 2

3 3 5 4 2 6 6 1 2 4 5

1 1 2 5 3 2 4 2 3 9 4

2 2 2 1 2 2 2 3

0

Mean

Relative ranking

Normalized

1 3 2

4.31 4.37 3.94 3.58 3.42 3.48 3.44 3.17 3.07 3.00 2.43

28 26 24 22 12 15 16 7 8 10 2

0.88 0.87 0.77 0.67 0.63 0.60 0.59 0.58 0.53 0.40 0.14

14 8 6 21 18 8 19

INTELLIGENT MULTIMEDIA REPOSITORIES

469

that organizations desperately need to build repositories. Once this is done, issues about choice of media will become more germaine. 4.6. QUESTIONS ABOUT IMR AND MEDIA / ARTIFACT ‘‘RECOMMENDATIONS’’

4.6 .1 . Collectiy e likes / dislikes So far we’ve presented the results for each artifact / media separately. The next set of results addresses the questions dealing with what the respondents liked / disliked about the media / artifact presented in this experiment, and what other artifacts they’d like to see presented in which media. Table A1 (a) and (b) in the Appendix plots the media / artifacts across the top, and a selection of ‘‘likes’’ and ‘‘dislikes’’ down the side. Individual respondents suggested these items on their own. The ‘‘totals’’ row of the ‘‘likes’’ [Table A1(a)] parallels most of the media / artifact ratings already discussed. The ‘‘totals’’ columns of the ‘‘likes’’ portion of the table reveals some new information, however. Specifically, they like items that are informative yet short / concise, and more than that they like to be able to compare across projects and view attribute importance weights. This latter item, is somewhat contradictory of earlier indications. Also, they want easy access, quick access, yet comprehensive coverage. In terms of the ‘‘dislikes’’ portion of Table A1, they wanted more detail in some cases, yet felt some of the information in the repository was variously too long, too detailed, too short, and poor in quality. Lastly, we asked what additional artifacts they would like to see in a given medium. Table A2, also in the Appendix, shows their suggestions down the side and the media across the top. Some of the more frequently asked for items include actual vs. planned schedules in Gantt chart form, lessons learned in audio / video, project team info in almost any medium, decision rationales in audio / video, estimated time and cost in spreadsheets (!), and issues / problems and resolutions in audio / video. Given the previously summarized results about the importance of a textual repository, it is surprising that the most popular item on this table is video (58 suggestions). Text is second with 40 suggestions, and audio is third with 31 suggestions. Apparently, respondents have a limit for the amount of textual artifacts they want. When debriefed about these items, they said they wanted to see the body language and hear the voice intonations of past participants sharing schedule difficulties, lessons learned, decision rationales, and problems / resolutions. They did not think text could do it justice.

5. Discussion of results There are two possible interpretations of the results just presented. The first interpretation is that a ‘‘a repository of any kind is needed, and the presence of intelligent multimedia is relatively unimportant.’’ A number of findings seemingly point to this interpretation. The repository itself was one of the artifacts tested in the experiment. Estimators’ accuracy, confidence, and speed all improved dramatically in the trial with the repository as compared to the trial without the repository. Also, the respondents fairly uniformly rated the repository as the highest artifact in terms of all metrics—workload reduction, time savings, organizational impact, importance, acceptance, and so on. Generally speaking, only the system help features and the

470

B. G. SILVERMAN AND N. BEDEWI

textual content of the repository were rated as highly. All other media-artifacts were rated less highly, and were encountered less often. The good news, if one accepts this first interpretation, is that organizations can get all the benefits of a software artifact repository without having to invest in any special media, intelligent agents, computational aids, or other artifact embellishments. Clearly this is true for small scale repositories with simplistic domains. Sorting algorithms are the type of program that most respondents have past experience with, and are relatively knowledgeable about. This experiment was constrained to that type of repository to cut down on the intrusion into the respondents’ work time (recall that all respondents are professional software leaders, managers, etc.). Given the simplicity of the domain, it is all the more impressive that the repository had the impact that it did on both task performance and user reaction. The fact that this impact occurred raises some important points about estimation task support needs and judgmental biases that we will return to in a moment. The problems with this first interpretation are severalfold. First, it ignores results presented above that indicate that video and audio are some of the most wanted items, and the specific audio / video clips we included were not of the artifacts the respondents wished to see in those media. Second, when one scales up to a full sized repository for a complex domain the repository-wide browsing problem grows, as does the difficulty of satisfying respondents’ demands for quick, rapid retrieval of relevant, informative items. Agents become more relevant. Third, respondents will be less familiar with the reusable artifacts they will encounter in a full-scale repository, and with the purpose, definition, contents, and derivation of those artifacts. There is overwhelming evidence that respondents reuse artifacts they are familiar with and / or can become rapidly familiar with such as through a phone call to the artifact’s author (e.g. Silverman et al. , 1983; Silverman, 1985, 1992a ; Silverman & Moustakis, 1987; Murray et al. , 1987; Murray, 1988; Silverman & Bedewi, 1993). Any media or agents that reduces obstacles to reuse for even a subset of the overall user group will have an important impact. In a classic article on the estimation task, Kahneman and Tversky (1972) refer to the ‘‘availability heuristic’’—the most ay ailable past estimates are reused, regardless of goodness of fit. Any errors this introduces are further compounded since the reused estimate serves as an ‘‘anchor’’ from which the new estimate is adjusted. Given the time pressures of the task and the frequency with which the task recurs, respondents fall back on heuristics such as ‘‘availability’’ and ‘‘anchoring and adjustment.’’ This is why their estimates were not as good when the repository was omitted from the experiment. In such a setting, ecological validity concerns must over-ride the familiar metaphor arguments if repositories are to achieve their full potential. Interpretations of respondents’ reactions as ‘‘any repository’’ is sufficient should not and cannot be trusted. A refinement to the first interpretation that accounts for ecological validity and task support needs would seem to be called for. A second interpretation that offers such a refinement is that ‘‘a textual repository is needed’’, and intelligent multimedia enhances the availability of, and adjustments possible to, artifacts for a significant number of users’’. This interpretation is not so difficult to accept, particularly since many of the media-artifacts were rated fairly highly by a still significant number of

INTELLIGENT MULTIMEDIA REPOSITORIES

471

respondents. The reasons behind the popularity of video and audio in the last set of results (Table A2) also bears this out. This interpretation means the organization must invest more in the design of the repository and its agents and in the media of the artifacts. It also means we need to look more carefully at what the results tell us about designs, media, and agents. The next few sections provide that closer look. 5.1. LESSONS LEARNED ABOUT REPOSITORY DESIGN

There are several lessons learned in these results about the overall repository design. First, the repository of the experiment was rich in media, following a ‘‘something for everyone’’ design philosophy. This profusion of media artifacts lead to a situation where most respondents did not encounter the full range of media of the repository. This was also the reason why the various repository help features were rated as important as the repository itself. Nevertheless, there were only a few mediaartifacts that seemed to be fairly uniformly disliked (e.g. schedules in spreadsheets). If anything, the results indicate the respondents like the richness and easily think up even more media artifacts they would find useful (see earlier Table A2). It seems there is a large appetite for media-artifacts, and repository designers won’t go wrong no matter how much they try to satiate it. The design of our study offers no insight, though, on the benefits of individual media, or on how much to expend in delivering such items. A second lesson learned concerns the organizing and sequencing of the mediaartifacts. When the users brought up a previous estimation case, the various media-artifacts could be scrolled past in sequential order. If a respondent did not scroll far enough, or scrolled too fast, various media-artifacts would not be encountered. In general, the non-textual media (video, audio and derivative) were at the end of the case, with audio, video, and teleconferencing links being the last things encountered. It would probably be worthwhile to invest in a single screen that overviewed the contents of each case, without the need for scrolling. One could even envision a GUI that would set aside reserved territory holding a preview of each media-artifact. This would make the artifacts more ‘‘available,’’ and reduce the ‘‘not encountered’’ syndrome. Another important set of lessons of the findings so far, concerns how to help the respondents with the ‘‘cultural’’ change that this environment would afford. Given that the IMR environment would be embedded as a shared, collaborative environment, it would support a ‘‘paradigm shift’’ in terms of how estimates are performed. Since different individuals / organizations (including the ‘‘customer’’ organization) repeatedly need various levels of estimates, this architecture must be embedded in their main work environment to encourage them to obtain / adapt reusable cases rather than constantly re-invent them. Likewise, the groupware feature allows users to retrieve other estimators’ reusable work without overly burdening that case author or development organization. To do so, respondents need help overcoming their unfamiliarity with some of the media-artifacts that support past authors explaining the rationale, derivation, and / or purpose of their past case. Most notably this includes the videoconferencing ability, but also how to access other types of media artifacts. Finally, this architecture also provides the dual benefit of immediate user support combined with long term organizational ‘‘corporate memory’’ evolution. An organization with this architecture should not have to embark on some

472

B. G. SILVERMAN AND N. BEDEWI

‘‘grand’’ metrics collection effort. Instead, users will build the repository incrementally for existing projects. Over time, enough reusable cases will be collected, so that most new situations can receive task support and metrics and parametric models could be developed. Our lessons learned to date show that all these potential benefits are foreign to many users. Adopters must allow for a training interval where organizational performance drops initially, but then quickly improves past the pre-repository levels. 5.2. LESSONS LEARNED ABOUT MULTIMEDIA

Discussion has already been offered in general terms about the use of the media-artifacts, the design of the media-artifacts, and the desires for additional media-artifacts. This section reviews each of these topics in more detail offering specific lessons learned. Particular attention is paid to the task relevancy of the media-artifacts. As mentioned earlier, having an electronic repository of analogous projects was the most important artifact from the respondents’ perspective. This was followed by text based information of project descriptions. Interestingly, respondents ranked their own experience third as an ‘‘artifact’’ used for estimating purposes. For initial estimating (i.e. anchoring on a previous case), respondents gave fourth place rank to project attributes (e.g. Cost, HW, SW) information in textual format. However, for final estimating (i.e. adjustment of a previous case), respondents ranked fourth the cost adjustments recommended by a rule-based estimation assistant (intelligent agent). The study also revealed that some multimedia tools such as electronic conferencing were greatly underutilized, yet from the pre-trial interviews it was apparent that such a tool was highly desired and needed. As already mentioned, this lack of usage could be attributed to the time pressure the respondents were under, and / or the to fact that they did not understand the purpose of the tool. ‘‘Classic’’ multimedia such as video and audio were found to be ‘‘nice’’, ‘‘exciting’’, etc., however, the respondents were unable to utilize these for estimate adjustment purposes. From an organizational perspective, the implications of this finding are significant. Given the budgetary constraints organizations are under, for estimating purposes, ‘‘classic’’ multimedia should either be constructed more carefully, or omitted. An organization embarking on the development of a project repository could bypass costly setups for such media as video and audio for the first phase of the effort. Before initiating such media, they need to resolve the challenge of how to structure and present the content of the classic media. This is a good point to examine lessons learned and guidelines for designing the artifacts. Generally, the respondents want the artifacts’ content to be comprehensive, structured, easily accessible, short and concise. They also want to view the same artifacts across multiple past cases simultaneously. Detailed information has to be hidden but quickly accessible. These ‘‘wants’’ add to the sophistication of the territory-reserving GUI described in the previous section. In an earlier section, this paper mentioned that ‘‘intrusiveness and emotional impact’’ might be a concern. The lessons learned here give insight about how information should be organized, and what should be allowed to intrude, when.

INTELLIGENT MULTIMEDIA REPOSITORIES

473

Respondents also indicated emotional impact is important and that the previous projects’ artifacts should be personalized as much as possible. This includes the following. (1) Capturing ‘‘body language’’ on video. (2) Capturing lessons learned throughout the project cycle with the original voice ‘‘tones’’. (3) Capturing project participants’ original comments, decision rationale, and issues throughout the project life cycle. This personalized information was often cited as the most vivid encountered by the respondents when accessing previous cases while performing the project estimate. Respondents complained that the quality of the video presented should be improved and that a couple of the audio clips were hard to hear. These types of comments will prove challenging for an organization attempting to capture personalized information mentioned above, to improve emotional impact, and to create comprehensive yet concise slips. Given that this information will be captured ‘‘dynamically’’ during the life of ongoing projects and by ‘‘amateurs’’, balancing between artifact content, quality, and production cost may be an obstacle not easily overcome. Further research would be useful here to develop templates of and procedures for the ongoing creation of such media-artifacts by project personnel. Results to date from the design rationale community may contribute to this effort (Bewtra, Bentz & Truszkowski, 1992; Truszkowski & Balin, 1992). Finally, we list some of the more popular artifacts / media that the respondents wanted captured in the repository and that were not part of the experimental repository. These include Demos / Video, Design / Graphics, Actual vs. Planned Schedules / Gantt Charts, Project Team Info / Misc., Lessons Learned / Audio & Video, Estimated Time & Cost / Spreadsheet. Again, one would assume these must be instantiated with the same quality, content, and production standards alluded to above. 5.3. LESSONS LEARNED ABOUT AGENTS

The results also include several interesting insights about intelligent agents. First, even though sorting algorithms are of the size (and familiarity) that most managers could intuitively estimate, only 42% of the respondent felt they had sufficient information for the task without the repository, as compared to 79% with the repository (Table 3). Similarly, respondents who thought their estimates would be met rose from 42% to 82% when the repository was added [Table 7(b)]. At the same time their mean group cost estimate rose by 88% yet the standard deviation in the group estimate dropped in half [Table 7(a)]. These numbers indicate the respondents are being given strong anchors, ones they trust, as well as estimation assistance they find acceptable (they do not adjust very far from the anchor). Indeed post-usage questionnaires and debriefing results reveal 91% of respondents feel the repository improved their estimates, saved them time, and supported their acquisition and processing efforts. These overall results are suggestive that the intelligent agents played a measurable ‘‘anchoring and adjustment’’ or decision support role. Indeed, out of dozens of media-artifacts they encountered, respondents consistently rated the acquisition (CBR) and estimate adjustment agents as important and

474

B. G. SILVERMAN AND N. BEDEWI

useful features. In terms of the CBR agent, they often used its highest recommended case, but rarely rated its attribute weights of past projects among the most important media / artifacts. Also, they wanted an interface that let them display and compare several past project estimates simultaneously. In terms of the estimation assistant, they felt it was useful at the start of the estimate adjusting process (to get them going), but then they ignored its final recommendations and relied on their own judgment in the end. The lessons learned here give insight about how information should be organized, agent vs. human control of tasks , and what should be allowed to intrude , when. In terms of parameters affecting agent intervention and control, when one has a significant descriptive or prescriptive model of the process / domain, and of where humans do and do not perform well, it is easier to see where agents should interrupt, take control, and provide support. Where this knowledge is absent, agents should generally be either passive (user started and controlled) or highly plastic and flexible in their interventions. For example, estimation agent is only activated by user button selection, and its suggestions / findings are easily overridden. The value of this design philosophy was realized in this repository. Specifically, the project estimators enjoyed the luxury of intervention by the estimation aid at the start of that task, but they took over control once the agent had got them going. Even though they felt the agent was naive, they liked its intervention to get them started. Our descriptive model of this domain never lead us to expect this usage of the estimation agent. Fortunately we had designed it so users could override its advice. In terms of parameters affecting agent acceptance, trust, and emotional impact, the most surprising results encountered in these studies were probably the extent of detail and media that users want from repositories. The estimation repository users want detail down to the body language and voice inflections of the previous artifact creators’ explaining design rationales. If repositoires of the future are to capture and retain such levels of detail, this poses high performance computing challenges for agents in every step of the reuse process—creation and refinement. As the telecommunications, video and computing fields continue to merge, the repository users may find the artifacts they want most are indeed in the repositories. The problem facing repository agent developers is how to use content-based search to retrieve only the most relevant clip segments, and to support the transformations and links to other media that are need for reuse supported problem solving.

6. Concluding comments As reusable artifact repositories are growing in breadth, media, and content, their popularity is rising. More and more users engage in reuse-supported work, a sort of ‘‘mainstreaming’’ of repositories is beginning. To foster this trend, this research explored whether repositories can (or should) (1) be better attuned to how humans use the two step reuse-supported work process (construct, refine), (2) be given media appropriate to the domain content and knowledge the repository artifacts need to convey, and (3) be given intelligent agent abilities to better support the process and domain media manipulation. The investigations were constrained to a repository domain of project estimation, and many media-artifact findings and lessons learned were presented for that domain.

INTELLIGENT MULTIMEDIA REPOSITORIES

475

Based on our earlier work in other reuse domains, we feel a few of our current study’s findings can be generalized beyond the sorting project estimation domain. The following are suggested as potential cross-domain lessons learned. $ Almost any form of reuse repository is better than none. $ The respondents were more aware than the repository engineers (the authors) of the various media each of the domain-specific artifacts should be encoded in. They are a resource that should not be ignored. $ Respondents have legitimate needs, and significant appetites, for all of the media. Items like digital video, desktop conferencing, audio clips, etc. can be ‘‘gold plating’’ if used indiscriminately, but they improve project performance when used appropriately. $ Human work performance (time, effort, quality) appears correlated to the usefulness of the repository. $ Accomodating the full set of multimedia needs in the repository, raises significant research challenges for agents that can do content-based searches and keep the retrieved artifacts informative yet concise and relevant, quick to access yet comprehensive in coverage, and good in quality yet captured by laypersons. $ There also are needs for reuse-oriented process agents in repositories that support users during (1) constructive activities such as attribute-based retrieval, a comparative display and analyse agent (similarities / dissimilarities), and good ‘‘anchor’’ suggesting / familiarizing agents and (2) refinement activities such as reusable artifact modification assistant agents, over- / under-adjustment critics, and other performance support steps. Building a repository that supports all these needs is a challenge for the future. The research described in this paper, hopefully, illuminates some of the pathway toward meeting that challenge. The NASA / GSFC is gratefully acknowledged for sponsoring this study. Also, we appreciate and thank the 33 respondents from NASA and industry who gave so generously of their time, particularly, when completing the ‘‘questionnaire from h—’’.

References BACON, F. (1620). Noy um Organum. Cambridge: Cambridge Universtity Press. BEDEWI, N. (1995). Judgmental forecasting in software projects: exploration of design guidelines for multimedia , reusable artifacts to support analogical reasoning heuristics , Ph.D. Thesis, George Washington University, Washington, DC. BEWTRA, M., BENTZ, R. & TRUSZKOWSKI, W. (1992). KAPTUR—a tool (system) to support software reuse. Strategies and Tools for Software Reuse Workshop Proceedings. Menlo Park, CA: AAAI / MIT Press. BLATTNER, M. & DANNENBERG, R., Eds. (1992). Multimedia. New York, NY: ACM Press. BLOOR, R. (1991). Repository technology. DBMS , 15 , 17 – 23. BOEHM, B. W. (1981). Software Engineering Economics. Englewood Cliffs, NJ: Prentice-Hall. CONTE, S. D., DUNSMORE, H. E. & SHEN, V. Y. (1986). Software Engineering Metrics and Models. New York, NY: Benjamin / Cummings Publishing. ETZIONI, O. & WELD, D. (1994). A softbot based interface to the internet. CACM , 37 , 72 – 75. GENNARI, J. H., TU, S. W. et al. (1994). Mapping domains to methods in support of reuse. International Journal of Human – Computer Studies , 41 , 399 – 424. GRAY, R. et al. , Eds. (1993). Intelligent User Interfaces. New York, NY: ACM Press. HODGES, B. (1995). Reuse driy en software process (RSP )—a product line approach to reuse.

476

B. G. SILVERMAN AND N. BEDEWI

The Navy / STARS Demonstration Experience Report, Crystal City: US Navy (STARS CDRL A017R). HOLSINGER, E. (1994). How Multimedia Works. Ziff-Davis Press. KAHNEMAN, D. & TVERSKY, A. (1973). On the psychology of prediction. Psychologic Rey iew , 80, 237 – 251. KISTLER, D., VALETT, J. (1992). Software management eny ironment (SME ) concepts and architecture. Technical Report NASA / GSFC SEL 89-103, Greenbelt, MD. KNIGHT, P. (1992). Factors to consider in evaluating multimedia platforms for widespread curricular adoption. Educational Technology , May, 25 – 27. LOWRY, M. R. (1992). Software engineering in the twenty first century. AI Magazine , 14 , 71 – 87. MAYBURY, M., Ed. (1993). Intelligent Multimedia Interfaces. Menlo Park, CA: AAAI / MIT Press. MURRAY, A. (1988). A user oriented approach for assessing multiple paradigm knowledge engineering practices. GWU / IAI Dissertation, available from University Microfilms. MURRAY, A., FEGGOS, K., WEINSTEIN, M. & SILVERMAN, B. G. (1987). A lisp machine-based scenario generator for the INNOVATOR expert-system. Military Applications of AI , pp. 90 – 110. Silver Spring Operations Research Society of America. NABKEL, J. & SHAFRIR, E. (1995). Blazing the trail: design considerations for interactive information pioneers. SIGCHI Bulletin , 27 , 45 – 54. PARK, I. & HANNAFIN, M. J. (1993). Empirically-based guidelines for the design of interactive multimedia. Educational Technology Research and Dey elopment , 41 , 63 – 85. PEIRCE, C. S. (1935). Collected Papers. Cambridge: Cambridge University Press. POLYA, G. (1945). How to Soly e It. Princeton, NJ: Princeton University Press. REEVES, T. C. (1992). Evaluating interactive multimedia. Educational Technology , May, 47 – 53. SEEL, N. R. (1991). Agents in the human – computer interface. IEEE Colloquium on Intelligent Agents , pp. 1 – 5. London: IEE. SHETH, B. & MAES, P. (1993). Evolving agents for personalized information filtering. Proceedings of Ninth Conference on Artificial Intelligence Applications , pp. 342 – 355. Los Alamitos, CA: IEEE Computer Society Press. SHNEIDERMAN, . (1982). Designing the User Interface. Reading, MA: Addison-Wesley. SILVERMAN, B. G. (1994). Intelligent electronic collaborators for mirror worlds. Distributed Collaboration Technologies for Software Engineering , ARPA Workshop Proceedings. Arlington: ARPA / SISTO. SILVERMAN, B. G. (1992a ). Critiquing Human Error: A Knowledge Based Human – Computer Collaboratiy e Approach. London: Academic Press. SILVERMAN, B. G. (1992b ). Human – computer collaboration. Human – Computer Interaction Journal , 7 , 165 – 196. SILVERMAN, B. G. (1985). Software cost and productivity improvement: an analogical view. Computer Magazine , 18 , 86 – 96. SILVERMAN, B. G. & BEDEWI, N. E. (1993). Toward the development of a rule base for critiquing project management errors during software development. Expert Critiquing Systems Research Workshop Proceedings , pp. 55 – 64. Menlo Park, CA: AAAI / MIT Press. SILVERMAN, B. G., CHANG, J. & FEGGOS, K. (1989). Blackboard system generator: an alternative distributed problem solving paradigm. IEEE Transactions on Systems , Man , and Cybernetics , 19 , 334 – 355. SILVERMAN, B. G. & MOUSTAKIS, V. (1987). INNOVATOR: representations and heuristics of inventor / engineers. Building Expert Systems for Business , pp. 402 – 439. Reading, MA: Addison-Wesley. SILVERMAN, B. G., MOUSTAKIS, V., LIEBOWITZ, J., BOKHARI, I. & TSOLAKIS, A. (1983). Resource planning by analogy: the SCAT support system. Microcomputers: Tools or Toys ? Gaithersburg: ACM / NBS. TRUSZKOWSKI, W. & BAILIN, S. (1992). KAPTUR: Knowledge Acquisition for Presery ation of Tradeoffs and Underlying Rationales. Greenbelt, NASA / GSFC / Code 520 pamphlet. VINCENNES, I. & RASMUSSEN, J. (1992). Designing ecological interfaces. IEEE Transactions on Systems , Man , & Cybernetics , 22, 1071 – 88.

477

INTELLIGENT MULTIMEDIA REPOSITORIES

WINOGRAD, T. (1995). From Programming environments to environments for designing. CACM , 38 , 65 – 74. WOOLRIDGE, M. & JENNINGS, N. R. (1995). Intelligent agents: theory and practice. Knowledge Engineering Rey iew , 10 , 115 – 152. Paper accepted for publication by Associate Editor, Dr. A. Monk.

Appendix

TABLE A1 Respondents ’ likes / dislikes with media / artifacts

1 2

4

3

2

1

4

2

2 1

5 1 1

2

21

1

1

Help / popups, doclinks, demons

1 3 2 4

Estimation assitant project estimates adjustor

1 3 3

Attribute importance / project attribute weights

1

1 4 5 2

1 1

1

Projects repository / analogous projects

1

Electronic conferencing / project initiation

1 5

Gantt charts / initial project schedules

Graphics / algorithm flowcharts

5 1

Spreadsheets / initial project schedules

Video / sort algorithm example

Short / concise 10 Informative 3 Well organized 4 Comprehensive 1 Quick access to information 3 Easy access to information 3 Ease of comparison with other projects 4 Review at own pace 1 Previous projects 1 nuances Easier than reading Personal Break from reading Why project was over budget Ease of recall of slipped project Better for retention Could visualize overall project schedule Combination of text & graphics Retention of information for historical reasons Similarity ranking Attribute weighting Good check against own judgment Did no clutter screen Totals 30

Audio / lessons learned

Text / project description & attributes

(a) Likes

1 1 1

2 8 1 4

21 28 16 16

1

6

16

5

19

8

44 9 4

12

2

1

8 1 2

1

2

2

1

1 2

2

2

2 1 2

1 2

4 1

5 16

9 21

4 1

19

T o t a l s

14

4

5

18

18

39

23

21

6 40

1 6

478

B. G. SILVERMAN AND N. BEDEWI

Slow if too much information Hard to locate important information, if too much information Did not tell problems encountered Too many attributes to compare manually Some terms were not clear Needed more details in some cases Too short / terse Sometimes fast Hard to hear Unorganized Information could not be quantified No lessons learned on successful projects Preferred it in video Too long Quality ‘‘poor’’ Tool detailed Too technical Could not visualize overall project schedule Prefer Gantt chart Too many colors (color blind) Not networked Too much scrolling Redundant information (but necessary for context purposes) No ‘‘hypertext’’ No time to under stand purpose

Help / popups, doclinks, demons

Estimation assitant project estimates adjustor

Attribute importance / project attribute weights

Projects repository / analogous projects

Electronic conferencing / project initiation

Gantt charts / initial project schedules

Spreadsheets / initial project schedules

Graphics / algorithm flowcharts

Video / sort algorithm example

Audio / lessons learned

Text / project description & attributes

(b) Dislikes

T o t a l s

1

1

1

1

1

1

1

1

2

2

1

5 2 3 1 1

1

1

6 1

1

1

2 1

1

1

3

14 3 3 1 1 1

1 5 8 2 1

4

2 5 11 3 1 4

2 1

2 1

3 1

1 1 2

1 1 2

1 1

1 1

2

2

479

INTELLIGENT MULTIMEDIA REPOSITORIES

‘‘Score’’ could mislead, and excourage no exploration Scrolling not standard Interface NOT intuitive Unable to view multiple projects at one time Could not filter out irrelevant factors / select other attributes Initially confusing ‘‘Back-out’’ capability Not clear how it impacts estimate No ‘‘future’’ look capability Having to ‘‘click & hold’’ left mouse button Totals

1

Help / popups, doclinks, demons

Estimation assitant project estimates adjustor

Attribute importance / project attribute weights

Projects repository / analogous projects

Electronic conferencing / project initiation

Gantt charts / initial project schedules

Spreadsheets / initial project schedules

Graphics / algorithm flowcharts

Video / sort algorithm example

Audio / lessons learned

Text / project description & attributes

(b) Dislikes —(Continued ).

1 2

1 2

1

1

3

1

2 5 1

1

5

3 1 1

5 7 2

1

1 1

7

16

17

4

8

6

5

T o t a l s

11

10

8

1

2 5

2

TABLE A2 Respondents ’ suggestions for additional artifacts they would like to see in a giy en media

T o t a l s User comments Developer comments

1 1

1 1

480

B. G. SILVERMAN AND N. BEDEWI

Estimation assitant project estimates adjustor

Attribute importance / project attribute weights

Projects repository / analogous projects

Electronic conferencing / project initiation

Gantt charts / initial project schedules

Spreadsheets / initial project schedules

Graphics / algorithm flowcharts

1

2 2

1

6 9

2

1

8 2

4 3

1

1

1

6 11 10

1

2

3

1

1

6

1

1

2 4

18

1

9

1

T o t a l s

6

1

4

Help / popups, doclinks, demons

Video / sort algorithm example

Discussion of complex subject Tutorial Operational comparisons (e.g. GUI, control center) Demos Design Actual vs. planned schedules Reason for project dependencies Scatter plots of project attributes Project team information, skill level, mix, etc. Miscellaneous decisions rationale Human factor, office politics, funding, etc. Estimated time & cost Baselines vs. new requirements Cost impact changed reqs Sub-system definitions Lessons learned (e.g. summary, by phase, all project members, etc.) File & data element types Popup help for each button Documentation

Audio / lessons learned

Text / project description & attributes

TABLE A2 (Continued.)

1 4

1

28

5

4

6

22

1

1

1

17

2

2

4

1

1

1

17 1

3

1

1

1

1

3

10

1 1 1

8

3

1

25 1

1

2 1

3 1

481

INTELLIGENT MULTIMEDIA REPOSITORIES

Help Business problem Proposed solution Project / task descriptions Issues / problems & resolution Audio clips transcriptions Reuse characteristics of modules Hypertexed docs. Detailed customer requirements Tools used Reasons for not meeting time & cost Future requirements Accuracy of estimates Estimator’s experience Project complexity No. of interfaces No. of users Requirements traceability Testing plan Recording of customer meetings Why other alternative(s) not selected Project manager explaining how estimate was performed Customer praise / complaints Assumptions made & how they changed over time

1 1 2

1

5

1

1

5

1 1 1

Help / popups, doclinks, demons

Estimation assitant project estimates adjustor

T o t a l s 1 3 3 7

5

4

15

1

1

1 1 2 1 1

Attribute importance / project attribute weights

Projects repository / analogous projects

Electronic conferencing / project initiation

Gantt charts / initial project schedules

Spreadsheets / initial project schedules

Graphics / algorithm flowcharts

Video / sort algorithm example

Audio / lessons learned

Text / project description & attributes

TABLE A2 (Continued.)

1 1 2

1

2 1

1

4 3

1 2

5

1

1

1

1

1 1 1 1

1 1 1 1

1 1

1 1 1

1

1

1

1 1

1

2

1 1

1

3

1

482

B. G. SILVERMAN AND N. BEDEWI

Miscellaneous presentations (project review, design review, etc.) Development team & working environment System functional flow Projected related flows, relations, illustrations GUI Formatted data & formulas Checklist / decision trees Miscellaneous meetings (scheduling, etc.) All aspects of project cycle LOC Unrelated projects Ability to sort attributes Keyword dictionary Pending project Development methodology Weights assigned by previous estimators Longevity, maintainability, etc. DB requirements Testing, installation Guidelines Total 40

10

1

Help / popups, doclinks, demons

Estimation assitant project estimates adjustor

Attribute importance / project attribute weights

Projects repository / analogous projects

Electronic conferencing / project initiation

Gantt charts / initial project schedules

Spreadsheets / initial project schedules

Graphics / algorithm flowcharts

Video / sort algorithm example

Audio / lessons learned

Text / project description & attributes

TABLE A2 (Continued.)

1

1

12 1

2

1

1

4 5

4 6

1 1

1

1

1

1

5

3 3 1

1

2

2

2 1

1

1

1

1 1 1

1 1 20

22

25

21

27

3 1

1

1

58

3

5 3 2 1

31

T o t a l s

13

13

1 7

1 1