Identifying the difficulties of object-oriented development

The Journal of Systems and Software 64 (2002) 23–36 www.elsevier.com/locate/jss

Identifying the difficulties of object-oriented development Steven D. Sheetz

*

Department of Accounting and Information Systems, Virginia Polytechnic Institute and State University, Pamplin College of Business, 3007 Pamplin Hall, Mailcode 0101, Blacksburg, VA 24061, USA Received 25 May 2001; received in revised form 30 June 2001; accepted 27 August 2001

Abstract Identifying the perceptions of developers that use object-oriented (OO) system development techniques is necessary to understand why they are described as difficult to learn and use. Professional developers with a broad range of experience were asked to share their perceptions of the issues that contribute to the difficulties of using OO techniques. The 67 developers primarily were from the tele-communications and systems consulting industries in a large metropolitan area. Seventeen small groups of developers with similar levels of experience completed a 2.5-h group cognitive mapping process using a group support system (GSS). Each GSS session consisted of activities to identify difficult issues, define categories that classify the issues by similarity, rate the importance of the categories for causing difficulties, and identify causal relationships among the categories to form a cognitive map that represents the group’s shared perceptions of the difficulties of using OO techniques. The 1279 issues identified were organized into 141 categories by the 17 groups. These 141 group-generated categories were merged to identify a set of 9 overall categories to allow comparisons of perceptions across groups and levels of experience. The results reveal a common ‘‘core’’ of difficult issues associated with using OO techniques that was shared by novices, intermediates, and experts. Overall category group cognitive maps reveal substantial differences in the causal relationships perceived by novices and experts. Ó 2002 Published by Elsevier Science Inc. Keywords: Cognitive mapping; Mental models; Group support systems; Analysis and design; Object-oriented; Novice/expert

1. Introduction Despite claims about the naturalness and usefulness of object-oriented (OO) techniques (Booch, 1991; Coad and Yourdon, 1991a,b; Rosson and Alpert, 1990; Rosson and Gold, 1989), the adoption of OO techniques has been slower than anticipated (Fichman and Kemerer, 1993; Rosson and Carroll, 1990). Difficulties in learning and using OO techniques have likely contributed to this slower adoption. Understanding the issues that contribute to difficulties of using OO techniques requires investigating what designers perceive as contributing to the complexity of OO systems. That is, to identify the difficulties of using OO techniques we must discover the designer’s mental model of using OO techniques. This mental model is based on experiences, and hence perceptions, of using OO techniques. Although the model may be incomplete, *

Tel.: +1-540-231-6096; fax: +1-540-231-2511. E-mail address: [email protected] (S.D. Sheetz).

0164-1212/02/$ - see front matter Ó 2002 Published by Elsevier Science Inc. PII: S 0 1 6 4 - 1 2 1 2 ( 0 2 ) 0 0 0 1 9 - 5

based on incorrect assumptions, or simply naive, the complexity of OO systems for designers is based on their mental model (Norman, 1988). Limitations or weaknesses in OO approaches also have been identified (Moynihan, 1996; Vessey and Conger, 1994). Understanding developers views also provides for revising OO techniques to make them more effective and efficient. Cognitive mapping is a research method that captures people’s perceptions of their environment (Axlerod, 1976; Huff, 1990). These perceptions represent a snapshot of their mental models. Applying cognitive mapping across levels of experience results in a series of snapshots that reflect difficulties encountered by OO developers as they gain expertise. This paper reports systems developers’ perceptions of the difficulties of developing OO systems. Similarities and differences in the mental models of novice, intermediate, and expert system developers are presented as evaluations of group cognitive maps of shared perceptions pertaining to the difficulty of using OO techniques.

24

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

The following section presents the research question and study methodology. The results are then reported followed by discussion of the implications of the results, and conclusions.

2. Research question Reconciling views of the benefits of OO techniques with reports of the difficulties of learning and using these techniques is the motivation of this research. Studying developers that use OO techniques is interesting due to the need to identify difficulties they associate with learning and using these approaches to system development (Booch, 1991; Coad and Yourdon, 1991a,b; Fichman and Kemerer, 1993; Kalakota et al., 1993; Manns and Nelson, 1993; Rosson and Carroll, 1990). Understanding developers’ perspectives is required for designing and refining effective processes of developing information systems (Rosson et al., 1988). An explicit focus on the most difficult to understand (complex) issues developers perceive provides a rationale for focusing resources to support their efforts. These resources may be applied to change developers by enhancing training or to change OO techniques by refining processes. Understanding developers’ perceptions requires a method for representing knowledge. Complexity theory represents knowledge as consisting of content, i.e., facts, ideas, or beliefs about a domain, interrelated by structure, i.e., causal relationships among elements of knowledge content (Streufert and Streufert, 1978; Streufert and Swezey, 1986). This representation provides a basis for investigating the content and structure of the knowledge people have about their environment. The content and structure of knowledge of OO systems development techniques at different levels of experience should be consistent with studies that show differences between novices and experts in other areas of computer system analysis, design, and programming. Such differences between novices and experts in various computer interaction situations have been documented (Adelson, 1984; Doane et al., 1990; Elsom-Cook, 1989; Guindon, 1990; McKeithen et al., 1981; Kieras and Polson, 1985; Pennington, 1987; Rist, 1991; Sutcliffe and Maiden, 1992). Thus, developers with differing levels of experience must be included in studies attempting to identify the range of issues perceived as difficult in OO development (Shanteau and Stewart, 1992). This expectation is due to the need for OO and other techniques to solve the same ill-structured problem of developing computer systems (Guindon, 1990; Kalakota et al., 1993). Differences in the content of knowledge for experts and novices including elements of a design schema (Adelson, 1984; Guindon, 1990; Jeffries et al., 1981; Manns and Nelson, 1993) and the ability to apply known

solutions to solve problems (Rist, 1991) suggest that the content of knowledge will be different for experts and novices. These content differences are associated with beliefs about what makes a good design or program and how to generate such a design or program. Experts have such beliefs; novices do not. Identifying knowledge content also should reveal overlaps in concepts and issues that are shared by novices and experts. That is, even though researchers often focus on differences, this study benefits from identifying similarities among perceptions across experience levels. Which in this case may represent difficulties that persist over time and reveal aspects of OO techniques that may benefit from revision. The structure of knowledge also has been shown to be different for novice and expert programmers. The structure of knowledge is different because the organization of knowledge is different, e.g., recalled programming language keywords were grouped using mnemonics (a syntactic view) by novices versus groupings by programming function (a semantic view) recalled by experts (McKeithen et al., 1981). One structure, novice, allows finding the definitions of keywords, i.e., all the things you can do, which then can be evaluated to determine if they meet the needs of a function being implemented. This results in the unnecessary retrieval and evaluation of keywords that do not apply to the current function. Experts on the other hand, have a knowledge structure that allows searching keywords by desired function. This is more efficient than novice structure because it eliminates both finding extraneous keywords and evaluating them, i.e., only the keywords that can implement the function are discovered, ideally the one that fits best the task. The goal of understanding the views of developers pertaining to the difficulty of using OO techniques and expectations of differences in knowledge content and structure of novices and experts lead to the following research question: What issues are perceived as complex, i.e., difficult to understand, about developing information systems using OO techniques, for developers at different levels of experience? The content of knowledge for three levels of experience were identified and compared using a content analysis approach (Kerlinger, 1973) applied to the responses of the developers. The structure of knowledge was compared using causal relationships the developers identified among their elements of knowledge content.

3. Methodology Seventeen groups of professional system developers were studied using a quasi-experimental design. The

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

25

The study environment was a major university near three technologically advanced metropolitan areas. The site was appropriate due to the availability of a GSS environment and the availability of knowledgeable participants. The GSS sessions were conducted in computer training classrooms using the VisionQuest software of Collaborative Technologies Corporation. The configuration allowed both verbal and electronic interaction and is consistent with the computer-based requirements that have been identified to support GSS and cognitive mapping (Eden, 1990; Dennis et al., 1988; Hickling, 1990; Johansen, 1988; Liou and Nunmaker, 1990; Weick and Meader, 1993).

Due to the relatively small percentage of developers that use OO techniques and the difficulty of convincing professionals to participate in research projects, random sampling techniques were not feasible. The largest number of participants (43%) came from software consulting organizations. Telecommunication companies provided the next largest percentage (38%) of participants. A smaller percentage (19%) of the participants came from other industries. Regardless of industry, most of the participants were developing application systems, rather than class libraries or utilities. Fifty-one percent of participants held a bachelors degree in information systems or computer science, an additional 40% held both undergraduate and graduate degrees. Participants averaged four traditional programming languages and two additional OO programming languages. Most had written many lines of code and participated in multiple development projects. Nearly all participants had used structured analysis and design methodologies, e.g., DeMarco (1978). Most participants at all levels of experience had evaluated at least one OO methodology, e.g., Booch (1991). Each of the 17 groups averaging 4 participants was assigned to a level of OO experience based on the average OO development experience of the members in the group. Six novices groups averaged 7–26 months of OO experience with a mean of 16 months. Five intermediate groups averaged 32–44 months of OO experience with a mean of 39 months. Six experts groups averaged 60–90 months of OO experience with a mean of 78 months. Participants at each level of OO experience were compared statistically using descriptive measures of experience using OO techniques (Glass and Hopkins, 1984). These analyses were conducted to determine if participants assigned to the levels of OO experience actually differed on measures of OO experience. As shown in Table 1, the OO experience data supports the assignment of groups to the expert, intermediate, and novice levels of OO experience.

3.2. Sample

3.3. Procedures

Participants were placed into groups based on their level of experience using OO techniques by their managers. That is, managers of the participants were asked to assign their employees to more or less experienced groups. Individuals from the same company and level of experience participated in the same GSS session. Often the participants in a group were working together on the same project. The participants were well qualified to comment on systems development as indicated by their extensive experience in the field. Participants were not paid to participate, they participated solely on the basis of gaining an understanding of the difficulties of using OO techniques through the results of the research.

Each participant completed one GSS session with participants at a similar level of experience. Group support systems research suggests that GSS can be applied to implement research methods (Anson et al., 1992; Davis et al., 1992; Dennis et al., 1990–1991; Hoffer et al., 1990; Gallupe et al., 1991). Sheetz et al. (1994) demonstrates the use of GSS to automate data collection for cognitive mapping. The steps completed by each group in a GSS are presented in Table 2. These steps are similar to procedures used in previous research (Bougon, 1983; Bougon, 1992; Bougon et al., 1990; Sheetz et al., 1994; Sheetz et al., 1997). The GSS software provides anonymity for participants during brainstorming of concepts, concept

seventeen groups consisted of 67 developers with varying levels of experience using OO techniques. A cognitive mapping approach was followed to allow the views of the participants to guide the investigation. Cognitive mapping is a research method that implements the constructs of complexity theory (Eden, 1988; Levi and Tetlock, 1980; Montazemi and Chan, 1990; Montazemi and Conrath, 1986; Weick and Bougon, 1986; Zhang, 1989; Zhang et al., 1992). Procedures are included to identify knowledge content as elements of meaning and knowledge structure as causal relationships among content elements. The procedures consisted of activities to elicit a broad range of ideas, facts, issues, or beliefs from the participants, then converge these responses into a set of categories derived by the participants, and lastly to define causal relationships among their categories. These procedures were automated using group support system (GSS) software. Each group participated in one GSS session. Multiple groups of participants for three levels of experience were evaluated to increase the likelihood that the major causes of difficulty were identified. 3.1. Environment

26

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

Table 1 Measures of descriptive information of the experience of participants Measure

Exp. level

Means

S.D.

Sig. F

Contrasts

t Value

t Sig. level

Months of OO experience Months of object-oriented programming experience

N

8.78

9.14

.000

Experts! ¼ Novices

5.78

.000***

I E

26.05 38.37

20.52 21.26

Intermediates! ¼ Novices Experts! ¼ Intermediates

3.47 2.27

.001*** .026**

N

7.41

9.27

Experts! ¼ Novices

8.71

.000***

I E

15.90 38.32

11.71 14.93

Intermediates! ¼ Novices Experts! ¼ Intermediates

2.47 5.97

.016** .000***

N

16.19

17.59

Experts! ¼ Novices

7.86

.000***

I E

41.95 76.68

27.50 32.60

Intermediates! ¼ Novices Experts! ¼ Intermediates

3.45 4.27

.001*** .000***

N

.56

2.32

Experts! ¼ Novices

3.28

.002***

I E

3.29 8.32

2.21 13.45

Intermediates! ¼ Novices Experts! ¼ Intermediates

1.19 2.01

.240 .049**

N

.23

.26

Experts! ¼ Novices

3.15

.002***

I E

.32 .47

.27 .23

Intermediates! ¼ Novices Experts! ¼ Intermediates

1.16 1.91

.251 .060*

N

2.07

1.24

Experts! ¼ Novices

4.61

.000***

I E

3.10 5.68

1.15 4.45

Intermediates! ¼ Novices Experts! ¼ Intermediates

1.34 3.13

.184 .003***

N

19K

36K

Experts! ¼ Novices

3.08

.003***

I E

18K 140K

22K 242K

Intermediates! ¼ Novices Experts! ¼ Intermediates

).03 2.95

.975 .004***

N

.70

2.54

Experts! ¼ Novices

2.26

.027**

I E

1.57 5.00

3.87 10.82

Intermediates! ¼ Novices Experts! ¼ Intermediates

.47 1.71

.640 .093*

N

1.15

.36

Experts! ¼ Novices

4.44

.000***

I E

1.19 2.05

.68 .97

Intermediates! ¼ Novices Experts! ¼ Intermediates

.21 4.00

.831 .000***

Months of object-oriented analysis experience

Months of object-oriented experience

Months of object-oriented management experience

Object-oriented experience/ total experience

Counts of OO activities Number of object-oriented development projects

Lines of object-oriented code written

Number of object-oriented employees managed

Number of object-oriented programming languages

.000

.000

.007

.010

.000

.005

.074

.000

*

p 6 :10. p 6 :05. *** p 6 :01. **

categorization, ratings of importance, and identification of relationships. During brainstorming participants can see the ideas of others, but do not know who entered the idea. For all other steps, individuals interact with the software without any indication of the responses of others. Anonymity provides participants the ability to respond freely (Connolly et al., 1990; Jessup et al., 1990; Gallupe et al., 1988). However, verbal processes for defining categories, discussing the results of categorizing the concepts, and discussing mean ratings provides group members some understanding of the perspectives of other group members. In the current sample of coworkers it is likely that the participants had an existing

understanding of the views of others in their experimental group. Throughout the session, the researcher is restricted to non-reactive procedural guidance only. It is likely that the group influenced the views of the individuals. This is true of most work situations. However, this group and individual process is necessary to uncover shared perceptions. Many systems developers have strong opinions about appropriate techniques and other development issues. Sheetz et al. (1994) suggests that individual views obtained before a group session are reiterated by the individual in the group setting, i.e., the group process did not appear to alter the opinions of the individuals.

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

27

Table 2 Steps in the GSS cognitive mapping approach Activity 1. Elicit concepts Introduction Concept identification 2. Identify categories Category identification

(Break) 3. Classify concepts Concept categorizations Discussion 4. Rank categories Category rating step 1 Discussion Category rating step 2 Discussion Category rating step 3 5. Define relationships Identify relationships

Description

Duration (min)

Visionquest tool

Describe system use; sign informed consent form; discuss framing statement and stall diagram Elicit characteristics, concepts, and/or issues that contribute to (increase or decrease) the difficulties of using OO techniques

15

Comment cards and brainwriting Brainwriting

Elicit categories to group concepts by similarity; agree on category definitions and names

20

40

Verbal for participants, facilitator uses compactor

10 Classify the concepts into (top) ten categories

20

Compactor

Discuss classification

10

Compactor

Rate each category on a 7-point scale, from important to extremely important Group means are calculated and discussed Same as category rating step 1 Group means are calculated and discussed Same as category rating step 1

5

Rating

10 5 10 5

Verbal discussion Rating Verbal discussion Rating

30

Scoring

Each individual is given a comparison matrix of the categories in a rating task. They rate each comparison on a scale of 3 to þ3, from strong negative influence to no influence to strong positive influence of one category on another category

The following framing statement was used to set the context of the study and focus the participants: We are interested in understanding the process of using Object-Oriented systems analysis, design, and programming techniques. It would be useful to know what you believe is difficult to understand about using these techniques? The statement was worded broadly so that the participants’ recall of difficult issues would drive the results. More detailed cues have the potential of introducing researcher bias. In Table 2, the majority of steps, i.e., steps 1–4, and GSS session time, i.e., on average 155 min out of 185, were devoted to the group working together to establish the shared meaning for the categories to use when defining their individual cognitive maps. Known as ‘‘congregating labels,’’ these shared meanings allow individually defined cognitive maps to be combined to form a group cognitive map. Without such a process idiosyncratic individual cognitive maps would be very difficult to integrate (Cossette and Audet, 1992; Langfield-Smith, 1992), especially on the scale necessary for this study. These procedures resulted in 17 sets of concepts, categories, categorizations, ratings of category importance, and cognitive maps. The concepts and category

definitions were grouped by experience level and analyzed using content analysis (Kerlinger, 1973) to identify similarities and differences in the perceptions of the novices, intermediates, and experts. Meaning units derived from this analysis provides a qualitative understanding of similarities and differences in the content of knowledge across experience levels. Cognitive maps representing the shared views of the novices, intermediates, and experts were compared in terms of the relationships and causal themes they contained. 3.4. Deriving overall category cognitive maps of shared perceptions Identifying and comparing group cognitive maps representing novices, intermediates and experts requires a common framework applicable across all levels of experience. The 17 category identification activities resulted in 141 non-unique categories. A set of ‘‘overall’’ categories was developed using a Q-sort process to identify the common framework capable of combining the maps of individuals to form a cognitive map for a level of experience. The Q-sort process consisted of three raters independently evaluating the 141 group category names and definitions generated by the 17 groups. The raters grouped categories based on similarity, i.e., they determined the meaning of a category then placed the category in a group with other categories that address

28

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

similar issues. After completing the grouping process each rater named his/her groups of categories resulting in three separate named groupings of the categories. The three sets of groupings were then combined by the researcher by matching the categories with the groupings. For example, a grouping by rater 1 was combined with a grouping by rater 2 because they both contained the same categories. The names of the sets of groupings defined by each rater and the final set of overall categories are included in Appendix A. Overall categories were required to combine responses within experience levels and to compare cognitive maps across experience levels. Overall category cognitive maps were derived for each level of OO experience using the individual participant cognitive maps and the assignments of group categories to overall categories. Software was developed that used the output of the VisionQuest GSS to complete following steps. First, an overall category cognitive map was derived for each participant (Eden et al., 1992). Since each group category was assigned to an overall category, deriving the overall category cognitive maps involved simply replacing the names of the group categories in the map with the names of the overall categories. When multiple group categories were assigned to the same overall category, that overall category would appear multiple times in the individual’s map. These multiple occurrences were then merged into one. To derive the overall category cognitive map that represented the shared perceptions of participants for a level of OO experience two matrices were created from the individual overall category cognitive maps of the participants at that level of OO experience. The first matrix contained the number of participants that identified each of the possible relationships between the overall categories. The second matrix contained the sum of strengths assigned by the participants for each relationship. An average strength of the relationships matrix was calculated by dividing each cell of the second matrix (total relationship strengths) by the corresponding cell of the first matrix (number of participants that identified the relationships). This calculation yields a matrix of average relationship strengths representing the shared perceptions of the participants at a level of OO experience. The number of participants and the average relationship strength matrices were used to identify overall category cognitive maps of varying levels of agreement for each level of experience. Level of agreement refers to the number of participants that identified a relationship between two categories (Weick and Bougon, 1986). For example, if all participants identified the relationship, level of agreement was 100%. If half of the participants identified a relationship then level of agreement was 50%. Identifying a level of agreement overall category map for one

level of experience consisted of the following steps: (1) selecting the desired level of agreement, e.g., 66%; (2) determining the number of participants at that level of experience that represent that level of agreement, e.g., 14 of 21 novices; (3) searching the number of participants matrix for cells that exceed that number of participants; and (4) using the corresponding cells in the average strength matrix to represent each relationship on a cognitive map. Cognitive maps representing 100% through 10% levels of agreement were identified for each level of experience. This amounts to looking first at the relationships that all participants at the level of experience identified, then progressively looking at relationships that fewer and fewer participants identified, until the map includes relationships identified by any participant. As fewer participants are required to agree, i.e., level of agreement is decreased, more relationships qualify to be represented. For example, 100% agreement maps have very few relationships, while 10% agreement maps may have all categories connected to all other categories. Observing the maps in this manner provides a feeling of the ‘‘emergence of complexity’’ for the researcher attempting to understand the responses of the participants. 3.5. Strengths and weaknesses of the methodology Strengths of the methodology include the substantial experience of the participants, in-depth understanding of the participants’ perceptions from the qualitative analyses, and the large sample size relative to other cognitive mapping or qualitative studies. Weaknesses of the methodology include the nonrandom sample and reliance on the participants to accurately recall the difficulties they experienced using OO techniques.

4. Results The content of knowledge was identified using a content analysis approach applied to the concepts and category definitions. The content analysis identified meaning units in the words and phrases used by the participants. This analysis was performed to identify the range of issues contained in the participants’ responses. Similarities and differences revealed by the overall category cognitive maps illustrate the participants’ structure of knowledge. That is, how difficulties related to one issue impact the difficulty of other issues. 4.1. Content analysis of concepts and categories Over the entire set of responses similarities and differences existed between the three levels of experience. One hundred and sixteen unique meaning units were

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

identified from the 1297 concepts and 141 category definitions provide by the 17 groups. Meaning units were identified to provide a broad perspective of the responses of the participants. Fig. 1 shows a graphical representation of the overlap of the meaning units. Circles in Fig. 1 were scaled to represent the number of meaning units identified by participants for that level of OO experience. The number in each area of the diagram is the number of meaning units found in the responses of the participants represented by the overlapping circles. For example the number ‘‘35’’ in the center of Fig. 1 indicates that the same 35 meaning units were found in the responses of experts, intermediates, and novices. As shown in Fig. 1, substantial overlap existed among the meaning units found in the concepts and category definitions. These results indicate that the content of knowledge for novices and experts has distinct similarities and differences. Only 10% of the meaning units found in the intermediate responses were not found in the responses of either novices or experts. Experts and novices identified substantially more unique meaning units with 28% and 33%, respectively. Thus, participants at each level of experience shared a core of beliefs pertaining to the difficulties of using OO techniques. Table 3 shows the meaning units in each overlapping area of Fig. 1. The table shows knowledge content of the perceived difficulties of using OO techniques identified by participants across levels of experience and unique to each level of experience. The meaning units and their organization reported in Table 3 provide a perspective of the perceptions of these developers. 4.2. Overall category cognitive maps The 17 category identification processes, one per group, resulted in an average of 8 categories that were similar for many groups. The overall category identifi-

Fig. 1. Overlap of the content of knowledge for levels of OO experience: Venn diagram of shared meaning units.

29

cation process resulted in the following set of overall categories: analysis/modeling (AM), design (DE), education (ED), implementation (IM), methodologies (ME), project management (PM), reuse (RE), testing (TE), tools (TO), and unknown (UN). Unknown was the overall category name used for the group categories that were placed in different Q-sort categories by all three raters. Of the 141 group categories identified, 133 (94%) were placed in the same overall category by at least two of the three raters; all three raters placed 73 (52%) of the group categories in the same overall category; and 8 (6%) were placed in different overall categories by all three raters. Kappa ¼ :55 (Siegel and Castellan, 1988, p. 284) indicated there was moderate agreement (Landis and Koch, 1977) among the raters on the assignment of group categories to the final set of overall categories. This result is significantly different from zero agreement with a calculated z ¼ 21:36; p < :001. The overall categories provide a knowledge structure derived from the knowledge content and structure of the participants. Understanding how the structure of knowledge is different for the levels of experience is related to how their responses are spread across the overall categories and how they perceive the overall categories to be interrelated. Novices, intermediates, and experts identified about the same number of concepts and categories. Additionally, the number of relationships between categories showed no differences across experience levels. Differences in knowledge structure are in the components of knowledge included or emphasized by the participants at the different levels of experience. These differences are shown in Table 4 by different numbers of categories and concepts for different overall categories and in the overall category cognitive maps by relationships among different overall categories. Table 4 shows how many of the group categories and concepts were placed in each overall category for each group. Cells in Table 4 contain two numbers. The first number is the number of categories the group identified that were placed in the overall category by the Q-sort raters. The second number is the number of concepts that were placed in those group categories by a majority of participants during their GSS session. The assignment of concepts and group categories to overall categories provides a comparison of the structure of knowledge across the levels of experience. Review of Table 4 shows that experts were more interested in Education, Project Management, and Tools issues, while novices were more focused on Analysis/Modeling, Implementation, and Testing. Overall category cognitive maps were derived for levels of agreement ranging from 100% to 10%. No relationships existed on the 100% or 90% level of agreement maps for any of the levels of experience. Selecting

30

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

Table 3 Panel A: Meaning units identified by participants at multiple levels of experience Meaning units identified by novices, intermediates, and experts Abstraction How system implemented Non-OO systems Assigning work Identifying objects Object interaction Benefits of adopting OO Inheritance Paradigm shift Choosing methodologies Interfacing w/Non-OO tools Placement Converting design to code Lack of standards Problem understanding Creating reusable objects Learning Project management techniques Developer communication Learning OO Relationships (is a, has a) Finding objects to reuse Managing iterations Reusing analysis & Design Flow of control Methodologies Reusing code

Scheduling issues Scope Training developers Training mangers Understanding methods Using OO languages/tools Using results of analysis What system must do

Meaning units identified by experts and novices Goal is working system Language Graphical user interfaces Managing change Iterative development Need mentors

OO lifecycle support Polymorphism Standards for OO

Meaning units identified by experts and intermediates Class libraries Implementation Costs of adopting OO Managing user expectations Estimating projects Maturity of methodology Hype of marketers Non-problem classes

OO Life cycle Performance Problem domain Production environment

Terminology Understanding OO concepts

Meaning units identified by intermediates and novices Analysis and design phases Development phases Correctness Documenting for reuse

Modeling Reuse

Upfront cost/long term benefits What OO can & Can’t do

Panel B: Unique meaning units identified by participants at each level of experience Meaning units identified only by experts Abstraction levels Drawing diagrams Make or buy class libraries Coding issues Dynamic communication Map OO principles to tools Data ownership Effects on organization Naive methodologists Defending against tools How reuse accomplished Number of operations Defining attributes Industry direction Programming languages Defining methods Iteration Real OO tools Meaning units identified only by intermediates Human–computer interface Maintenance Implementation objects Metrics

OO Philosophy Problem decomposition

Meaning units identified only by novices Communication Management buy-in Documentation Managing reuse across project Encapsulation Mapping prob. to implementation How variables used Mapping req. to model Integration testing OO infrastructures Large # available classes OO Model

Partitioning Partitioning business problem Persistence Programming standards Reusability Skill training

levels of agreements for presenting an accurate picture of the views of the participants for each level of experience was difficult. This is due to the lack of consensus on relationships identified. Thus, levels of agreements for presentation were selected to show a substantial number of relationships identified by a representative sample of the participants at a level of experience. This resulted in selecting 40% for novices and experts, but a higher level of agreement was necessary to limit the number of relationships appearing on the intermediate map and was deemed acceptable due to the goal of capturing a broad perspective of the difficulties faced by OO developers. An overall category cognitive map for the each level of experience is presented in Fig. 2. In the maps strong

Reusable dev. practices Tools Unlearning When structured techniques better Writing test code in objects

System testing Testing tools Understanding class hierarchy Understanding found objects Unit testing When to subclass

(þ3 or 3) relationships are indicated by thick lines, moderate (þ2 or 2) relationships by medium lines, and slight relationships (þ1 or 1) by thin lines. Inverse relationships, i.e., an increase (decrease) in difficulties due to issues in the origin category cause a decrease (increase) in difficulties due to the destination category are marked with a minus sign (). Relationships not marked indicate positive relationships, i.e., an increase (decrease) in the origin causes a increase (decrease), in the destination. Seventy percent, i.e., 18, of the 27 participants in the novice groups identified a strong positive relationship from implementation to analysis/modeling. Fifteen, i.e., 55%, of the novices viewed implementation as slightly impacting methodologies.

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

31

Table 4 Numbers of categories and concepts associated with each overall category Group # Novices 2 3 6 11 14 16 Total

ME

AM

1, 17 2, 14

1, 11

1, 3 4, 15 7, 49

Intermediates 5 10 12 2, 19 13 2, 9 15 1, 30 Total Experts 1 4 7 8 9 17 Total

5, 58

DE

IM

TE

RE

2, 20 1, 0

1, 10 1, 23

1, 5 1, 10

1, 18

21 3 16 15 11 13

1, 7 2, 12 1, 4

1, 8

1, 1, 3, 1, 1, 1,

1, 6 1, 2

1, 12 1, 2

1, 4

2, 12 1, 31

1, 19 1, 3

5, 39

3, 41

8, 69

6, 31

4, 29

5, 23

4, 56

3, 32

2, 10

2, 16 1, 25

13 3 15 7 11

1, 5

1, 8 1, 11

1, 1, 2, 2, 1,

1, 2, 2, 1, 1,

1, 5

2, 16

1, 11

7, 48

7, 62

1, 5

1, 7

1, 1, 1, 1, 1, 1,

1, 5, 3, 1, 1, 1,

2, 1, 2, 1, 1, 1,

1, 7 3, 16 1, 5

1, 2 2, 7

1, 1, 1, 1, 1,

7, 38

6, 50

5, 49

1, 10

PM

ED

3, 15

1, 13

1, 10

1, 4

1, 8

2, 5

1, 12 1, 18 1, 19

TO

13 18 20 8 4 11

3 4 32 6 17

3 46 13 6 3 8

Non-major concepts

# categories, # concepts

2 20 1 1 3 9

9, 96 6, 54 10, 67 6, 58 10, 73 8, 90

36

49, 438

12 7 2 3 4

9, 73 6, 51 9, 104 10, 49 8, 90

1, 2

28

42, 367

1, 6 1, 0

15 30 9 10 19 7

9, 76 9, 120 7, 64 7, 75 9, 79 10, 78 50, 492

2, 6 1, 12 1, 5 4, 23

1, 2

2, 18 1, 3

1, 4 1, 13 1, 4

2, 16

1, 12

1, 2 3, 15

3, 43

5, 27

4, 39

5, 26

2, 16

3, 17

5, 66

12, 74

8, 79

3, 15

90

17, 104

13, 40

18, 144

9, 52

9, 62

11, 100

23, 178

18, 173

8, 40

154

All groups Total 15, 150

1, 5

19 14 4 14 4 11

6 10 18 6 8

UN

1, 9

141, 1297

Column headings are category codes. AM––analysis/modeling, DE––design, ED––education, IM––implementation, ME––methodologies, PM–– project management, RE––reuse, TE––testing, TO––tools, and UN––unknown. Each cell contains the number of categories placed in the overall category and the number of concepts that were in those categories.

The overall cognitive map where 11, i.e., about 40%, of the 27 novices agreed that each relationship existed is presented in Fig. 2a. In this map with 17 relationships, implementation was part of 13 relationships, while the next most cognitively central overall categories, analysis/ modeling and testing each was involved with only 5 relationships. Thus, it is clear that implementation was much more central to the views of these novices than were the issues in the other overall categories. Eighty-five percent, i.e., 18, of the 21 participants in the intermediate groups identified positive relationships from implementation to education and from project management to education. Seventy percent, i.e., 15, of the intermediates identified positive relationships from implementation to analysis/modeling and from analysis/ modeling to education. Education was the most cognitively central of the overall categories. The project management category was also perceived as an important source of difficulties influencing the analysis/modeling, implementation, and education categories for 60% of intermediates. A small majority, i.e., 11% or 52%, of the participants in the intermediate groups identified the 17 relationships

presented in Fig. 2b. Education, participating in 8 relationships, is the most cognitively central category followed by project management, implementation, and methodologies each with 6 relationships. Generally, the cognitive centrality of the categories did range as widely for the intermediates as it did for the novices. That is, no one category was perceived as dramatically more important than the others. Intermediates saw the Reuse, Testing, and Tools categories as not participating in any relationship. Eighty percent, i.e., 15 of the 19 participants in the expert groups identified a relationship from education to project management. Thirteen, i.e., about 68%, of the participants identified a strong relationship from project management to education and a relationship from education to tools. Twelve, i.e., 63%, of the participants identified a relationship from the project management category to the tools category. Eight, i.e., 42%, of the 19 experts identified the 18 relationships presented in the overall category cognitive map shown in Fig. 2c. Education was the most cognitively central category participating in 9 relationships. Project management, implementation, and tools each

32

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

mately influencing the first category in the series. For example in Fig. 2a, a loop of strong relationships exists beginning in the analysis/modeling category, moving next to the implementation category, then on to the design category, and finally back to the analysis/modeling category. Loops occur between two or more than two categories. The loops uncovered in the overall category maps of the novices, intermediates, and experts from Fig. 2 are presented in Table 5. Comparison of these loops shows how different the perceptions of the novices and experts were in terms of causal themes among categories. It seems that these novices and experts share few perceptions from this perspective. Specifically, the novices and experts have in common only one two-category loop, i.e., between implementation and reuse (IM ! RE). This is out of 15 loops for the experts and nine loops for the novices. The loops also show that intermediates are more like experts than they are like novices, with a focus on the impact of education and project management that is absent in the novices’ perceptions.

5. Discussion

Fig. 2. Cognitive maps representing the shared views of novices, intermediates, and experts.

participated in six relationships. Implementation was seen as a source of difficult issues. Controlling the difficulties of project management was a goal for these experts. Education, tools and reuse were seen as ways to achieve this goal. Novices were more focused on implementation than experts, and experts were more focused on education and project management than novices. Novices were more focused on analysis/modeling, experts were more focused on methodologies. The categories with the lowest values of cognitive centrality across all three levels of experience were design, reuse, and testing. Novices were more focused on design and testing than were experts, and experts were more focused on reuse than novices. 4.3. Loop analysis A loop of causation is formed when a series of categories are viewed as influencing each other and ulti-

The structure and content of knowledge of the novices, intermediates, and experts exhibited similarities and differences. These results are discussed in terms of themes of difficult issues apparent at each experience level. Together the themes show how perceptions of difficulty evolve with experience using OO techniques. Implementation issues overwhelmed novices. The meaning units they identified, e.g., encapsulation and testing, and those they shared with experts, e.g., flow of control and creating reusable objects, were dominated by issues related to modeling, design, coding and testing (see Table 3). This Implementation Theme also dominated the novice overall category cognitive map. In Fig. 2a, the implementation category participated in 11 of the 17 relationships and all loops except one. Other categories with multiple relationships also were implementation oriented. For example, the analysis/modeling, testing, reuse, and tools categories all address aspects of implementing OO systems. Fig. 3a shows the relationships that novices and experts identified. Like the shared meaning units, these relationships also are focused around implementation. The perceptions of intermediate groups also exhibited the Implementation Theme. Table 3a shows that the meaning units shared by novices and intermediates are associated with the implementation at higher levels of abstraction such as development phases versus the more code focused meaning units. Of the meaning units only identified by intermediates, half were oriented toward implementation, but toward issues at higher level than the novice level focus on code, e.g., human–computer

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

33

Table 5 Loops of causal flow through overall category cognitive maps for novices, intermediates and experts Two-category loops

Three-category loops

Four or more category loops

Novices

IM ! AM IM ! RE IM ! TE IM ! ME AM ! TO

IM ! DE ! AM IM ! TO ! AM IM ! TE ! RE IM ! TE ! ED

Intermediates

ED ! ME ED ! IM ED ! PM IM ! DE AM ! ME ME ! PM

ED ! IM ! DE ED ! IM ! AM ED ! ME ! AM ED ! ME ! PM ED ! PM ! IM ME ! PM ! AM

ED ! ME ! PM ! AM ED ! PM ! IM ! DE ED ! PM ! IM ! AM ED ! IM ! AM ! ME IM ! AM ! ME ! PM ED ! ME ! PM ! IM ! AM ED ! ME ! PM ! IM ! DE

Experts

ED ! IM ED ! PM ED ! ME ED ! TO PM ! TO IM ! RE

ED ! ME ! PM ED ! PM ! TO ED ! RE ! IM ED ! RE ! PM

ED ! ME ! PM ! TO ED ! IM ! RE ! PM ED ! RE ! PM ! TO ED ! IM ! RE ! PM ! TO

Cell contents are category codes. AM––analysis/modeling, DE––design, ED––education, IM––implementation, ME––methodologies, PM––project management, RE––reuse, TE––testing, TO––tools. Each cell contains the number of categories placed in the overall category and the number of concepts that were in those categories.

interface and problem decomposition. In Fig. 2b, the analysis/modeling and implementation categories participate in many relationships supporting the existence of the Implementation Theme in the cognitive maps of the intermediates. The implementation category also participates in 11 of the 19 loops identified by the intermediates in Table 5. The most prevalent theme in the meaning units identified by both intermediates and experts and both intermediates and novices (Table 3a) are associated with issues such as OO life cycle, maturity of methodology, managing user expectations, estimating projects, development phases, and documenting for reuse. These issues represent a focus on methodology and project management issues that did not exist in the responses of the novices. This Methodology and Project Management Theme represents a step in thinking about OO development beyond specifics of how to accomplish individual activities required to implement systems to the processes required to integrate activities and deliver systems. The intermediate cognitive map presented in Fig. 2b also exhibits this theme. Strong relationships among the project management and methodology categories and the large number of relationships associated with these categories reveal this is a focus of the intermediate groups. In addition, loops that include strong relationships with the Analysis/Modeling category and relationships with implementation show how this theme is integrated with the Implementation Theme. The intermediates identified 8 relationships not identified by either novices or experts in their overall category cognitive maps (see Fig. 2). These relationships were

primarily associated with integration of the Implementation and Methodology and Project Management themes. Finally, the intermediates’ overall category cognitive map reveals a focus on education as the issue most highly related to other categories, participating in 8 of 17 relationships. Table 5 shows that Education is also the most dominant category in the loops identified by the intermediates, influencing 14 of the 19 loops of causal reasoning. This reveals a view that education can have substantive influence on the difficulties of perceived for each of the other issues, this represents an Education Theme in the views of the intermediates. Experts’ perceptions included the many issues related to the implementation, reuse, and tools aspects of the Implementation Theme. This is consistent with the views of novices and intermediates, but highly focused on how tools and reuse can influence implementation. This is a focus on the efficiency of implementation aspects of this theme and substantively different from the perspectives of the novices and intermediates which included more analysis/modeling and testing aspects, i.e., effectiveness aspects. It seems that novices and intermediates are having difficulties ‘‘getting it right’’, while experts are able to assume correctness, they know that part, and now focus on ‘‘getting it fast’’ through reuse and tools. The Education Theme found in the intermediates’ map is dominant in the experts’ overall category cognitive map (Fig. 2c). The education category participates in 9 of the 18 relationships in the map and overwhelmingly influences causation through 12 of the 14 loops identified (Table 5).

34

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

The importance of the Methodology and Project Management Theme is present although less important for experts than for intermediates. This was indicated by the lower number relationships and strengths of relationships identified by experts, combined with reduced integration of this theme with the implementation categories. Comparing the maps in Fig. 3 clearly reveals the differences in the perceptions of the experts and novices. For example, comparing the education category in the experts’ map, i.e., Fig. 3b, to the education category in the novices’ map, i.e., Fig. 3c, reveals dramatic differences in the perceptions of the influence of education. The views of the experts and the novices also differed in their perceived influences of the implementation category. Substantively, the second most cognitively central category in each map, i.e., project management for experts (Fig. 3b) and analysis/modeling for novices (Fig.

3c), was not part of the map of the other level of experience. This shows that the perceptions of the experts and novices were substantially different in terms of how difficulties of OO interact to impact system development. 5.1. Implications Resources devoted to easing the adoption of or transition to OO techniques should specifically consider developer experience levels and provide training oriented toward reducing the difficulties of OO development for developers at that level. The core of meaning units shared by participants at all levels of experience in Table 3 and the relationships common to both novices and experts in Fig. 3a provide the potential for the greatest return on education resources. It seems that a better understanding of the interaction of analysis/ modeling, implementation, and reuse could reduce perceived difficulties for multiple levels of experience. Perhaps changes to OO methodologies are appropriate to improve the integration of these issues. Novices could benefit from development life cycle training with examples of the relationships among analyses/models, designs, and implementations, including testing and tool aspects of OO techniques. In other words, a focus on addressing their biggest concern, i.e., the Implementation Theme. More experienced developers should benefit from understanding how tools, methodologies, and reuse influence project management, i.e., addressing issues of their Methodology and Project Management Theme. Research should focus on determining the statistical significance of the themes in the perspectives of OO developers. Understanding difficulties encountered in the use of OO techniques benefits from illustration of the range of issues relevant using OO techniques by OO developers found in the meaning units. Understanding the perceptions of industry professionals is ideal for identifying topics for applied research and suggests revisions in OO approaches.

6. Conclusions

Fig. 3. Shared and unique relationships of novice and expert overall category cognitive maps.

The content of knowledge for the experts and novices included substantial differences and similarities. Intermediates shared knowledge content with both novices and experts. The structure of knowledge for the novices and experts differed significantly from many perspectives. Experts were more focused than novices on project management and education issues. Novices were more focused than experts on implementation, analysis/modeling, and testing. These findings were consistent for meaning units derived from concepts and the overall category cognitive maps of the shared perceptions of the participants.

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

This study confirms that organizations adopting OO techniques must provide education programs across a range of topics. Additionally, they may wish to enhance/ customize the OO approaches they adopt for their situation. The range of issues identified can be addressed by organizations through training to make the adoption and use of OO techniques more efficient and effective. Specifically, it seems that novices will benefit from education focused on how to do the activities of OO development. Experts should benefit from education focused on how to manage OO projects and from applying their experience to enhancing this aspect of OO. Appendix A Overall categories were required to compare responses across groups and levels of experience. These overall categories were defined using a Q-sort process completed by three raters. The following table includes the names the raters assigned to their Q-sort categories and the names selected for the overall categories.

Rater 1, Q-sort categories

Rater 2, Q-sort categories

Methodologies Methods, process Modeling & Analysis/ OO concepts modeling, cognitive stuff Integration Design ImplementaHigh-level tion/prog. Imp., lowlevel imp. Testing/main- General tenance stuff youhave to do Reuse Reuse Tools, other Tools/ GUI Education/ Educatraining tion, stuff Management/ people issues, marketplace

Project management Issues

Rater 3, Q-sort categories

Overall Q-sort categories

Methodologies/market Analysis

Methodologies Analysis/ modeling

Design Implementation

Design Implementation

Reuse Tools

Testing/ maintenance Reuse Tools

Education, culture/ mindsets Project management

Education Project management Unknown

References Adelson, B., 1984. When novices surpass experts: the difficulty of a task may increase with expertise. J. Exp. Psychol. 10, 483–495.

35

Anson, R.G., Fellers, J.W., Bostrom, R.P., Chidambaram, L., 1992. Using group support systems to facilitate the research process. In: Nunamaker Jr., J.F., Sprague, R. (Eds.), Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences. IEEE Computer Society Press, Los Alamitos, CA, pp. 70–79. Axlerod, R., 1976. Structure of Decision––The Cognitive Maps of Political Elites. Princeton University Press, Princeton, NJ. Booch, G., 1991. Object-Oriented Design with Applications. Benjamin/Cummings, Redwood City, CA. Bougon, M.G., 1983. Uncovering cognitive maps––the self Q technique. In: Morgan, G. (Ed.), Beyond Method: Strategies for Social Research. Sage, Beverly Hills, CA, pp. 173–188. Bougon, M.G., 1992. Congregate cognitive maps: a unified dynamic theory of organization and strategy. J. Manage. Stud. 29 (3), 369– 389. Bougon, M.G., Baird, N., Komocar, J.M., Ross, W., 1990. Identifying strategic loops: the self Q interviews. In: Huff, A.S. (Ed.), Mapping Strategic Thought. Wiley, Chichester, England, pp. 327–354. Coad, P., Yourdon, E., 1991a. Object-Oriented Analysis, second ed. Yourdon Press, Englewood Cliffs, NJ. Coad, P., Yourdon, E., 1991b. Object-Oriented Design. Yourdon Press, Englewood Cliffs, NJ. Connolly, T., Jessup, L.M., Valacich, J.S., 1990. Effects of anonymity and evaluative tone on idea generation in computer-mediated groups. Manage. Sci. 36, 689–703. Cossette, P., Audet, M., 1992. Mapping of idiosyncratic thought. J. Manage. Stud. 29, 325–347. Davis, G.B., Nunamaker, J.F., Watson, H.J., Wynne, B.E., 1992. The use of a collaborative work system for the study of the key issues facing information systems managers: a comparison of issues and data collection methods from previous studies. In: Nunamaker Jr., J.F., Sprague, R. (Eds.), Proceedings of the Twenty-Fifth Hawaii International Conference on System Sciences. IEEE Computer Society Press, Los Alamitos, pp. 46–59. DeMarco, T., 1978. Structured Analysis and System Specification. Prentice-Hall, Englewood Cliffs, NJ. Dennis, A.R., George, J.F., Jessup, L.M., Nunamaker Jr., J.F., Vogel, D., 1988. Information technology to support meetings. MIS Quart. 12, 591–624. Dennis, A.R., Nunamaker Jr., J.F., Vogel, D., 1990–1991. A comparison of laboratory and field research in the study of electronic meeting systems. J. Manage. Inf. Syst. 7, 107–135. Doane, S.M., Pellegrino, J.W., Klatsky, R.L., 1990. Expertise in a computer operating system: conceptualization and performance. Human–Comput. Interact. 5, 267–304. Eden, C., 1988. Cognitive mapping: a review. Eur. J. Oper. Res. 36, 1–13. Eden, C., 1990. Managing the environment as a means to managing complexity. In: Eden, C., Radford, J. (Eds.), Tackling Strategic Problems: The Role of Group Decision Support. Sage, London, pp. 154–161. Eden, C., Ackermann, F., Cropper, S., 1992. The analysis of cause maps. J. Manage. Stud. 29, 309–324. Elsom-Cook, M., 1989. Acquisition of computing skills. In: Calley, A.M., Beech, J.R. (Eds.), Acquisition and Performance of Cognitive Skills. Wiley, New York, pp. 137–165. Fichman, R.G., Kemerer, C.F., 1993. Adoption of software engineering process innovations: the case of object orientation. Sloan Manage. Rev., 7–22. Gallupe, R.B., Bastianutti, L.M., Cooper, W.H., 1991. Unlocking brainstorms. J. Appl. Psychol. 76, 137–142. Gallupe, R.B., DeSanctis, G., Dickson, G.W., 1988. Computer-based support for group problem finding: an experimental investigation. MIS Quart. 12, 277–296. Glass, G.V., Hopkins, K.D., 1984. Statistical Methods in Education and Psychology. Prentice-Hall, Englewood Cliffs, NJ. Guindon, R., 1990. Designing the design process: exploiting opportunistic thoughts. Human–Comput. Interact. 5, 305–344.

36

S.D. Sheetz / The Journal of Systems and Software 64 (2002) 23–36

Hickling, A., 1990. ‘Decision Spaces’: a scenario about designing appropriate rooms for group decision management. In: Eden, C., Radford, J. (Eds.), Tackling Strategic Problems: The Role of Group Decision Support. Sage, London, pp. 154–161. Hoffer, J.A., Anson, R., Bostrom, R.P., Michaele, S.J., 1990. Identifying the root causes of data systems planning problems: an application of the PLEXSYS electronic meeting support system. In: Nunamaker Jr., J.F. (Ed.), Proceedings of the Twenty-Third Hawaii International Conference on System Sciences. IEEE Computer Society Press, Los Alamitos, CA, pp. 30–39. Huff, A.S., 1990. Mapping Strategic Thought. Wiley, Chichester, England. Jeffries, R., Turner, A.A., Polson, P.G., Atwood, M.E., 1981. The processes involved in designing software. In: Anderson, J.R. (Ed.), Cognitive Skills and Their Acquisition. Lawrence Erlbaum, Hillsdale, NJ, pp. 255–283. Jessup, L.M., Connolly, T., Gallegher, J., 1990. The effects of anonymity on group process in an idea-generating task. MIS Quart. 14, 313–321. Johansen, R., 1988. Groupware: Computer Support for Business Teams. Free Press, New York. Kalakota, R., Rathnam, S., Whinston, A.B., 1993. The role of complexity in object-oriented systems development. In: Nunamaker Jr., J.F., Sprague, R. (Eds.), Proceedings of the TwentySixth Hawaii International Conference on System Sciences. IEEE Computer Society Press, Los Alamitos, CA, pp. 70–79. Kerlinger, F.N., 1973. Foundations of Behavioral Research, second ed. Holt, Rinehart & Winston, New York. Kieras, D.E., Polson, P.G., 1985. An approach to the formal analysis of user complexity. Int. J. Man–Machine Stud. 22, 365–394. Landis, J.R., Koch, G.G., 1977. The measurement of observer agreement for categorial data. Biometrics 33, 159–174. Langfield-Smith, K., 1992. Exploring the need for a shared cognitive map. J. Manage. Stud. 29, 349–368. Levi, A., Tetlock, P.E., 1980. A cognitive analysis of Japan’s 1941 decision for war. J. Conflict Resol. 24, 195–211. Liou, Y.I., Nunamaker Jr., J.F., 1990. Ambiguity in human communication and the design of computer-mediated-communication systems. In: Nunamaker Jr., J.F. (Ed.), Proceedings of the TwentyThird Hawaii International Conference on System Sciences. IEEE Computer Society Press, Los Alamitos, CA, pp. 40–49. Manns, M.L., Nelson, H.J., 1993. An exploration of schema development in procedure-oriented programmers learning object-oriented technology. In: Proceedings of the Fourteenth International Conference on Information Systems, Orlando, FL. McKeithen, K.B., Reitman, J.S., Rueter, H.H., Hirtle, S.C., 1981. Knowledge organization and skill differences in computer programmers. Cognitive Psychol. 13, 307–325. Montazemi, A.R., Conrath, D.W., 1986. The use of cognitive mapping for information requirements analysis. MIS Quart. 10, 44–55. Montazemi, A.R., Chan, L., 1990. An analysis of the structure of expert knowledge. Eur. J. Oper. Res. 45, 275–292. Moynihan, T., 1996. Experimental comparison of object-orientation and functional-decomposition as paradigms for communicating system functionality to users. J. Syst. Software 33 (2), 163–169. Norman, D., 1988. The Psychology of Everyday Things. Basic Books, New York. Pennington, N., 1987. Stimulus structures and mental representations in expert comprehension of computer programs. Cognitive Psychol. 19, 295–341.

Rist, R.S., 1991. Knowledge creation and retrieval in program design: a comparison of novice and intermediate student programmers. Human–Comput. Interact. 6, 1–46. Rosson, M.B., Maass, S., Kellog, W.A., 1988. The designer as user: building requirements for design tools from design practice. Commun. ACM 31, 1288–1298. Rosson, M.B., Gold, E., 1989. Problem-solution mapping in objectoriented design. In: Meyrowitz, N. (Ed.), OOPSLA-89 Conference Proceedings. SIGPLAN Notices 24 (10), 7–10. Rosson, M.B., Alpert, S.R., 1990. The cognitive consequences of object-oriented design. Human–Comput. Interact. 5, 345–379. Rosson, M.B., Carroll, J.M., 1990. Climbing the Smalltalk Mountain. SIGCHI Bull. 21 (3), 76–79. Shanteau, J., Stewart, T.R., 1992. Why study expert decision making. Some historical perspectives and comments. Organ. Behav. Human Dec. Process. 53, 95–106. Sheetz, S.D., Tegarden, D.P., Kozar, K.A., Zigurs, I., 1994. A group support systems approach to cognitive mapping. J. Manage. Inf. Syst. 11 (1), 31–57. Sheetz, S.D., Irwin, G., Tegarden, D.P., Nelson, J., Monarchi, D.E., 1997. Individual and group views of the difficulties of using objectoriented techniques. J. Manage. Inf. Syst. 14 (2), 101–129. Siegel, S., Castellan Jr., N.J., 1988. Nonparametric Statistics for the Behavioral Sciences. McGraw-Hill, New York. Streufert, S., Streufert, S.C., 1978. Behavior in the Complex Environment. Winston & Sons, New York. Streufert, S., Swezey, R.W., 1986. Complexity, Managers, and Organizations. Academic Press, New York. Sutcliffe, A.G., Maiden, N.A., 1992. Analyzing the novice analyst: cognitive models in software engineering. Int. J. Man–Machine Stud. 36, 719–740. Vessey, I., Conger, S., 1994. Requirement specification: learning object, process, and data methodologies. Commun. ACM 37 (5), 102–113. Weick, K.E., Bougon, M.G., 1986. Organizations as cognitive maps: charting ways to success and failure. In: Sims, H., Gioia, D. (Eds.), The Thinking Organization: Dynamics of Organizational Cognition. Jossey-Bass, San Francisco. Weick, K.E., Meader, D.K., 1993. Sensemaking and group support systems. In: Jessup, M., Valacich, J.S. (Eds.), Group Support Systems: New Perspectives. Macmillan, New York, pp. 230–251. Zhang, W., 1989. POOL2––a generic system for cognitive map development and decision analysis. IEEE Trans. Systems Man Cybernet 19, 31–39. Zhang, W., Chen, S., Wang, W., King, R.S., 1992. A cognitive-mapbased approach to the coordination of distributed cooperative agents. IEEE Trans. Systems Man Cybernet 22, 103–114. Steven D. Sheetz is an Assistant Professor of Accounting and Information Systems at the Pamplin College of Business at Virginia Tech. He received his Ph.D. in Information Systems from the University of Colorado. His research interests include the cognitive complexity of developing information systems, learning and use of Object-Oriented development techniques, participatory design, and the application of Group Support Systems technology. He has published articles in Decision Support Systems, International Journal of Human–Computer Studies, Journal of Management Information Systems, Journal of Systems and Software, and Object-Oriented Systems. He also holds a MBA from the University of Northern Colorado and a B.S. in Computer Science from Texas Tech University. He has substantial industry experience in database design and systems development.