The impacts of quality and productivity perceptions on the use of software process improvement innovations

Information and Software Technology 47 (2005) 543–553 www.elsevier.com/locate/infsof

The impacts of quality and productivity perceptions on the use of software process improvement innovations Gina C. Greena,*, Alan R. Hevnerb,1, Rosann Webb Collinsb,2 b

a Information Systems Department, Hankamer School of Business, P.O. Box 98005, Baylor University, Waco, TX 76798, USA Information Systems and Decision Sciences Department, College of Business Administration, University of South Florida, Tampa, FL 33620, USA

Received 13 June 2004; revised 14 October 2004; accepted 21 October 2004 Available online 25 December 2004

Abstract Numerous software process improvement (SPI) innovations have been proposed to improve software development productivity and system quality; however, their diffusion in practice has been disappointing. This research investigates the adoption of the Personal Software Process on industrial software projects. Quantitative and qualitative analyses reveal that perceived increases in software quality and development productivity, project management benefits, and innovation fit to development tasks, enhance the usefulness of the innovation to developers. Results underscore the need to enrich current technology acceptance models with these constructs, and serve to encourage project managers to adopt formal SPI methods if developers perceive the methods will have positive impacts on their productivity and system quality. q 2004 Elsevier B.V. All rights reserved. Keywords: Software process improvement (SPI) Innovations; Personal software process; Software quality; Development productivity

1. Introduction The search for new ideas and innovations to improve software development productivity and system quality continues to be a key focus of industrial and academic research. Fichman and Kemerer [16,17] define software process innovations simply as any new way of constructing application software products: technologies that change an IT group’s process for developing software applications. This change is motivated by a need for improvement of software development: these methods “offer something in exchange for a change in work habits, and sometimes in exchange for a ‘culture’ change. reduce cost or effort, duration or time to market, or defects, and most offer to reduce all of these” [44, p. 112]. Much attention has been given to process-based approaches based on the premise that system development * Corresponding author. Tel.: C1 254 710 6210; fax: C1 254 710 1091. E-mail addresses: [email protected] (G.C. Green), ahevner@ coba.usf.edu (A.R. Hevner), [email protected] (R. Webb Collins). 1 Tel.: C1 813 974 6753; fax: 1 813 974 6749. 2 Tel.: C1 813 974 6754; fax: C1 813 974 6749. 0950-5849/$ - see front matter q 2004 Elsevier B.V. All rights reserved. doi:10.1016/j.infsof.2004.10.004

outcomes are determined by the capability of the systems development process. “Process improvement has emerged as an important paradigm for managing software development” [43, p. 271]. Recent attempts to improve software development practices have focused on defining, monitoring, and measuring development activities in an effort to identify and subsequently verify areas of improvement. Work in this area has resulted in a number of software process improvement (SPI) innovations such as the Capability Maturity Model Integration (CMMI) [6], the Personal Software Process (PSP) [25], the Team Software Process (TSP) [26], Cleanroom development methods [38], and others. Proponents of SPIs have asserted that well-defined and measured processes eventually lead to gains in software quality and developer productivity, yet there is evidence that use of SPI techniques has not successfully been diffused in the software engineering community [4,13,19,34,52]. Recent research has attempted to understand why this is so [9,39,44]. One approach has been to examine human factors in SPI adoption, in order to identify key factors that motivate software developers to use SPIs. Baddoo and Hall [5] find in their field research that ‘visible success’ is the key

544

G.C. Green et al. / Information and Software Technology 47 (2005) 543–553

motivating factor for developers and managers to sustain use of SPIs. Baddoo and Hall describe visible success of a SPI as evidence that a particular SPI can be successful, both in general and within specific development environments. A SPI’s success can be defined in many ways. Dyba [12], in a study of the organizational factors that influence the success of SPIs, operationalizes SPI success as perceptions of competence, perceptions of performance, perceptions of improved productivity, and perceptions of improved customer satisfaction. Our study defines a SPI’s ‘visible’ success in terms of (1) perceptions of how easy it is to use, (2) perceptions of improved productivity, (3) perceptions of improved quality, and (4) perceptions of the usefulness of the SPI. Perceptions of ease of use and usefulness are chosen because of the vast amount of research suggesting these factors are key to understanding technology use. Quality and productivity benefits are chosen because they are the most frequently claimed benefits of SPIs that make them useful to organizations. Visible success is a concept that is related to the cognitive instrumental process in the Revised Technology Acceptance Model, in which the instrumental factors of output quality, result demonstrability, and output quality of an information system, combined with job relevance, drive usefulness, which in turn impacts intentions to use a system [50]. In a study of the acceptance of two software development methods, Khalifa and Verner [29] find a significant, positive relationship between developers’ perceptions of quality resulting from using the development methods and actual use of those methods. Taken together, these research results suggest that developers, like information system users, respond to the actual instrumental value of a software process improvement innovation, and that a positive response, termed usefulness, is associated with greater use or intentions to use the SPI. What has not been tested in the specific SPI context is which factors are central to determining the usefulness of the SPI. In addition, research has not tested the comparative impact of usefulness and the traditional adoption factor of ease of use in motivating either intent to use or actual use of a SPI. This research extends the work of Baddoo and Hall [5] and Khalifa and Verner [29] by identifying elements key to visible success of a SPI. We use a combination of quantitative and qualitative methods to test whether these elements are crucial for determining usefulness of SPIs. The quantitative part of this study examines whether perceived gains in developer productivity and software quality are significant factors in determining the instrumental value of a SPI to software developers. In addition, we compare the relative impact on use of both perceptions of the usefulness of a SPI and its ease of use. The qualitative study is a content analysis of software developers’ comments about SPI use in order to extract additional, important factors. The overall goal of the research is to improve the likelihood of success in the implementation of formal SPI methods in

development organizations by conceptual development of what drives usefulness of a SPI, and by testing the relative importance of the factors motivating SPI use.

2. Research background 2.1. Usefulness of SPIs In software engineering literature, two frequently cited objectives of SPI innovations are to reduce development costs through improved developer productivity and to improve end user satisfaction with the resulting software by reducing software defects [33]. Thus, for managers and developers to perceive a SPI as useful, they would likely expect to see gains in quality and/or productivity as a result of using the SPI. Riemenschneider et al. [41], using Davis’ Technology Acceptance Model (TAM) [11], study factors that determine software developers’ intent to use development methodologies. They find that the usefulness of the methodology, compatibility of the methodology with work practices, and the opinions of developers’ peers and managers influence their intention to sustain use of a methodology. They further note that to be useful to developers, the methodology must enable developers to be more productive and achieve higher levels of performance in their jobs. Some case studies of individual organizations provide encouraging data on quality and productivity gains from SPI use [13,14,33]. McGibbon [33], reporting on business case justifications for SPIs, summarizes literature that reports gains in productivity from 9 to 100% and reductions in postrelease defects from 60 to 100% for organizations that have implemented CMM concepts. Glass [19] reviews research on seven categories of software technology innovations, noting that early excitement over the benefits of many technology improvements has likely been misplaced. However, his summary of research finds a fairly consistent pattern of gains in programmer productivity and software quality in those organizations that report using process improvement innovations such as software inspections and CMM. Ferguson et al. [14] review impacts of Personal Software Process (PSP) use in three organizations. As a result of PSP use, one organization observed one postrelease defect; another observed no post-release defects. A third organization reported 78% improvement in acceptance test software quality and 7.4% improvement in developer productivity. These studies and others suggest organizations that implement SPI techniques can expect gains in both quality and productivity. However, other studies suggest limited impacts from SPI use. In their survey of 67 developers in New England-based development organizations, Kuilboer and Ashrafi [30] examine developers’ perceptions of impacts of SPI use. They find that two aspects of software quality considered most important to software developers are correctness

G.C. Green et al. / Information and Software Technology 47 (2005) 543–553

and reliability. Over 48% of these developers noted improvements in software correctness as a result of using SPI techniques and nearly 57% reported improvements in software reliability. However, when questioned about impacts of SPI use on project schedule and cost (factors that ultimately impact developer productivity), nearly 70% of those who were able to draw a conclusion reported that there was either no impact in these areas, or that schedules and costs actually increased. Similar results have been reported in studies of software reuse [18]. These differing results suggest that individual software developers who have experience with a software process improvement innovation will vary in their perception of the extent to which the SPI improves software quality and productivity. Both software quality and developer productivity are components of workplace performance. In research on technology use, the construct perceived usefulness is defined as an assessment by the user of the degree to which the technology will improve workplace performance [11]. Consistent with the approach of Kuilboer and Ashrafi [30], we examine how productivity and quality from use of a SPI are perceived by the developers themselves, and investigate how these two factors are combined by individual developers to assess the usefulness of the SPI for their work. Thus, we hypothesize: H1: Higher perceptions of software quality improvements from using a SPI will be associated with higher perceptions of the usefulness of the SPI. H2: Higher perceptions of developer productivity gains from using a SPI will be associated with higher perceptions of the usefulness of the SPI. In the qualitative part of this study, we investigate whether there are additional factors that influence developers’ perceptions of the usefulness of the SPI. The results of the qualitative analysis will provide a further test of the importance of quality and productivity, as well as identify other factors that may impact perceptions of value. 2.2. Usefulness and ease of use of SPIs In the research literature, it has been noted that successful IT innovations are innovations that are actually used by the intended users, since the “true business value from any information technology would derive only through appropriate use by its target user group” [2, p. 85]. Many studies have identified users’ perceptions of an innovation’s characteristics as important to ensuring use of the innovation. In early diffusion of innovations work, Rogers [45] identifies five perceptions about an innovation that affects the rate of diffusion of the innovation: relative advantage, compatibility, complexity, observability, and trialability. Davis [10] studied the impact of two of these characteristics on innovation use: perceived ease of use (which was equated to Rogers’ complexity) and perceived

545

usefulness (which was equated to Rogers’ relative advantage). Commenting on perceived usefulness, Davis asserts that “people tend to use.an application to the extent they believe it will help them perform their job better” [10, p. 320]. Studies have shown a strong, positive relationship between perceived usefulness of a system and intentions to use a system [49]. Such relationships are also found in the context of system development approaches. For example, a recent field study of software developers finds that perceived usefulness of a software development methodology is the strongest determinant of their intention to use the methodology [22]. Conradi and Dyba [8] study perceptions of how SPIs support the expression and dissemination of knowledge. They note that adherence to SPIs cannot be expected unless, among other things, the processes are demonstrated to be useful to developers. Through subsequent interviews, Conradi and Dyba find support for this assertion, noting general agreement among developers and managers that “there is no point in having [SPIs] that are not considered useful” [8, p. 271]. However, Davis also states that an implied prerequisite of usefulness is that the application is perceived as relatively easy to use; that is, although an application may be considered useful, this potential benefit could be “outoutweighed by the effort of using the application” [10, p. 320]. Subsequent IS studies examining these two IT characteristics under the guidance of TAM also find that perceived ease of use is necessary for a technology to be considered useful (e.g. [22,48]). This is particularly relevant to the SPI context, in which Kuilboer and Ashrafi [30] find that some developers state that development costs and schedules increase when using a new SPI. Thus, it would appear that the degree to which the use of a SPI is perceived to be free of effort (perceived ease of use) impacts developers’ perceptions of the usefulness of a SPI, and the degree to which the SPI is perceived to improve developers’ work (perceived usefulness) impacts successful SPI use. In particular, we expect: H3: Higher perceptions of the ease of use of the SPI will be associated with higher perceptions of SPI usefulness. H4: Higher perceptions of the usefulness of the SPI will be associated with higher levels of SPI use. A more direct relationship between perceptions of ease of use and intentions to use/actual use of a technology has also been included in research models of antecedents of technology use. In the original TAM model, perceived ease of use is posited as impacting both perceived usefulness and use [10]. In the revised TAM model, the relationships between perceived ease of use and both perceived usefulness and use are retained [50]. Venkatesh et al. [51] test a unified model of information technology acceptance that explains greater variation in use than previous models. In this new model perceived ease of use (as part of effort expectancy) does not impact usefulness (termed performance

546

G.C. Green et al. / Information and Software Technology 47 (2005) 543–553

Fig. 1. Research model: factors impacting SPI use.

expectancy), but does impact use through behavioral intention to use. Therefore, we also hypothesize: H5: Higher perceptions of the ease of use of the SPI will be associated with higher levels of SPI use. In summary, Fig. 1 shows the research model to be examined in this study.

3. Research methodology 3.1. Research context Software process improvement (SPI) innovations continue to emerge as the field attempts to solve problems of cost overruns, schedule overruns, and customer dissatisfaction with poor quality software products While tools (e.g. CASE), techniques (e.g. object orientation) and development languages have all been studied as examples of SPI innovations [3,16,17,46], Ravichandran and Rai [42] argue that isolated efforts are unlikely to be successful, and that software process management should be viewed as part of an effort to create an organizational system for quality improvement. Their theory, based on a knowledge management perspective, is that higher level normative models of process improvement such as CMMI, BOOTSTRAP and Trillium impact software process capability through both systematic knowledge creation and embedding of that knowledge in the software development process [43]. The SPI innovation studied in this research is the Personal Software Process (PSP), which is a normative model that embodies both knowledge creation and knowledge embedding. PSP supports individual developers to become more productive and disciplined and is both a process framework and set of software development methods. PSP requires that developers continuously improve their skills, so that productivity and product quality improve over time. Similar to other quality assessments, PSP establishes four levels of maturity in development.

At the initial level, PSP asks developers to focus on knowledge creation; documentation of their current practices and recording of productivity and quality metrics. Later levels require developers to use processes that embody best-practice knowledge (e.g. effective specification methods, defect prevention) as well as to adapt processes based on their own knowledge and environment. Training on PSP helps developers progress through these levels with procedures and methods for documentation and measurement, project planning and estimation, identification and elimination of process problems, design and code reviews, and adaptation of methods to specific working environments [25]. While some results of PSP application have been positive in terms of productivity, quality, and important long-term benefits to individual developers (e.g. greater skills and discipline), other reports also reveal difficulties in the transition of the PSP processes and methods into practice [14,28,34]. PSP is a particularly appropriate research context for the study of the implementation of a SPI innovation because of the role of the Software Engineering Institute (SEI), which defines PSP’s advantages and benefits, as well as the organizational changes needed for its implementation. SEI also provides training on PSP, so the costs and knowledge barriers to PSP implementation are significantly reduced [15,47]. In addition, PSP is a software development innovation that is implemented at an individual level, which supports research at the individual level of analysis. 3.2. Data collection In order to examine and test the research hypotheses, we conducted a field survey of software developers who had adopted the PSP within their projects. The content validity of items on the survey was assessed via (1) a pretest with professors, graduate students, and software development professionals, and (2) a pilot study involving software development professionals. The pre-test of the instrument resulted in a few modifications to item wording to increase

G.C. Green et al. / Information and Software Technology 47 (2005) 543–553 Table 1 Survey items and sources of items (items in bold are used in hypotheses testing) Variable

Item #

Item wording

Source

Quality

QUAL1

Use of PSP has enhanced the functionality of applications that I build Use of PSP has decreased the number of errors in software products I build Software developed with PSP requires less maintenance Use of PSP has significantly improved my documentation of software products Use of PSP has improved the quality of software products I build Use of PSP has made me more conscious of software quality

[27]

Use of PSP has greatly speeded up my development of new applications Use of PSP has definitely made me more productive Use of PSP has significantly reduced the time I spend in software development

[27]

I found it easy to use PSP The tools and techniques of PSP were clear and understandable I found PSP to be flexible to implement It was easy for me to become skillful at using PSP

[10]

Using PSP improved my job performance I feel that use of PSP was successful in transforming me into a more disciplined software engineer Using PSP made it easier to do my job I found PSP useful in my job

[10]

The proportion of projects I use PSP with is:. How often do you use PSP?

[27]

QUAL2

QUAL4 QUAL5

QUAL3

QUAL6

Productivity

PROD1

PROD2 PROD3

Ease of use

EOU1 EOU2

EOU3 EOU4

Usefulness

USEFUL1 USEFUL2

USEFUL3 USEFUL4 Use

USE1 USE2

Researcher

547

With the assistance of the Software Engineering Institute, software developers who had been trained in the PSP and were using PSP on software development projects were contacted via email and invited to participate in the study. At the time of the survey, the SEI had trained over 2000 individuals in PSP. They estimated that approximately 300 software developers were using PSP on real projects [24]. Our target sampling frame represented the developers who accepted the invitation to participate in the study. Onehundred fifty-four developers agreed to participate in the study and were sent surveys via postal mail or email according to their indicated preference. If surveys were not received within four weeks, participants were contacted and reminded to complete and return the survey as quickly as possible. Seventy-one completed surveys were returned, resulting in a 46% response rate. Of these, eight were unusable, resulting in a sample size of 63 for this study. The 63 respondents represent 24 different organizations. Approximately half of the study sample classified themselves as programmers. Approximately 80% of the study respondents were male. The average age of respondents was approximately 33 years and over 40% had masters degrees. 3.3. Instrument validation

[23]

clarity and the reformatting of rating scales to ensure more consistency in scoring of variables. The pilot test of the instrument resulted in the elimination of two items that significantly reduced scale reliabilities. Table 1 lists the questions contained on the final survey as well as the source of the questions. Respondents answered most items using a 7-point Likert scale. Variable scores were determined by taking the average of the item scores for each variable. The exception to this is the USE measure: because one item used a 5-point scale and the other used a 7-point scale, these items were summed to produce the score for this variable.

To determine the reliability of scales, Cronbach’s alpha was computed for each scale. Initial alpha values were improved by the removal of three items in the Quality scale and one item in the Ease of Use scale. Results of the final reliability assessment are shown in Table 2. All scales exhibit reliability levels greater than .70, the recommended minimum level of acceptable reliability [36]. Construct validity was assessed in two ways. First, a principal axis factor analysis was conducted. Because of the relatively small sample size, two factor analyses were completed: one with the Quality, Productivity, and Ease of Use items, and the other with the Usefulness items only. Tables 3a and 3b show the results of these analyses. Items measuring the ease of use, usefulness, and productivity constructs load on their respective factors as expected. However, one item measuring the perceived quality construct (QUAL4) cross-loads weakly on both the Quality and Productivity factors. Because maintainability is a wellknown measure of software quality [37], we chose to keep the QUAL4 item with the Quality scale. The second assessment of construct validity was to examine the correlations between constructs to see if any Table 2 Reliability coefficients Scale

Cronbach’s alpha

Ease of use Usefulness Quality Productivity

.72 .90 .78 .83

548

G.C. Green et al. / Information and Software Technology 47 (2005) 543–553

Table 3a Factor analysis of quality, productivity, and ease of use items

Table 4 Correlations (square root of average variance is extracted on the diagonals)

Variable

Item

Factor 1

Factor 2

Factor 3

Quality

QUAL2 QUAL3 QUAL4 PROD1 PROD2 PROD3 EOU1 EOU2 EOU4

.869 .978 .320 .072 .326 .175 .276 .008 K.008

.258 .198 .329 .741 .731 .741 K.020 .162 .069

.117 .059 .072 .081 .106 .087 .848 .574 .654

Productivity

Ease of use

Table 3b Factor analysis of usefulness items Variable

Item

Factor 1

Usefulness

USEFUL1 USEFUL2 USEFUL3 USEFUL4

.824 .731 .860 .907

correlations exceeded the shared variance across construct items. As shown in Table 4, the shared variances across items exceed the correlations across constructs, further supporting construct validity.

Quality Productivity Ease of use Usefulness Use

3

PLS-Graph Version 3 was used to conduct the partial least squares analysis.

.853 .439** .225 .789** .353**

.867 .269* .524** .199

Ease of use

.799 .212 .118

Usefulness

Use

.876 .557**

1.000

PSP use is also associated with increased perception of the usefulness of PSP to developers (H2); and subsequently, increased perceptions of the usefulness of PSP is associated with increased levels of PSP Use (H4). Surprisingly, however, there is no support for the hypothesized relationships involving the ease of use construct (H3 and H5). More positive perceptions of Table 5 Quantitative analysis descriptive statistics Variable

Item

Quality PSP use enhances functionality of applications PSP use decreases errors in software PSP use improves quality of software Software requires less maintenance PSP use improves software documentation PSP use improves software quality consciousness Productivity PSP use speeds up development of new applications PSP use made me more productive PSP use reduced the time spent in development

4.1. Quantitative analysis of the research model Table 5 shows descriptive data on model variables. In general, developers give PSP good marks on most independent variables. Developers rate perceived quality impacts from PSP use and perceived usefulness of PSP fairly high (meansZ4.93 and 5.01, respectively). On average, assessments of perceived productivity benefits from using PSP and perceptions of how easy PSP is to use are close to neutral (meansZ4.06 and 4.36, respectively). Fig. 2 summarizes the results of the PLS analysis. An examination of the path coefficients reveals support for three of the five hypotheses. Increased perception of quality improvements from PSP use is associated with an increased perception of the usefulness of PSP to the developer (H1); increased perception of productivity improvements from

Productivity

Note: *significant at .05; **significant at .01.

4. Results Hypotheses were tested using both quantitative and qualitative analyses. Partial least squares (PLS) analysis3 was used to quantitatively test our research hypotheses. Content analysis was then used to analyze written comments provided by survey respondents.

Quality

Ease of use Easy to use PSP The tools and techniques of PSP are clear and understandable PSP flexible to implement Easy to become skillful at using PSP Usefulness PSP improved job performance PSP successful in trans.me into a more disciplined software eng PSP made it easier to do job PSP useful in job Usea The proportion of projects that use PSP How often do you use PSP? a

Mean

SD

4.93 3.80

0.94 1.59

5.33

1.23

5.40

1.25

4.70

1.55

4.46

1.58

5.88

1.30

4.06 3.93

1.24 1.51

4.39

1.38

3.90

1.34

4.36 4.01 4.58

1.03 1.37 1.57

4.61 4.23

1.40 1.32

5.01 4.92 5.31

1.31 1.50 1.42

4.63 5.17 7.65 3.43

1.61 1.45 3.27 1.43

4.71

1.69

Note: USE items used different measurement scales. The first item used a 5-point scale; the second item used a 7-point scale.

G.C. Green et al. / Information and Software Technology 47 (2005) 543–553

549

Fig. 2. Results of hypothesis testing.

the ease of PSP use are not found to impact subsequent PSP use, either directly (H5) or indirectly (H3). The lack of support for these two hypotheses would, on the surface, appear to contradict existing studies which find that perceived ease of use impacts perceived usefulness [1,10,48]. However, some studies find that the impact of ease of use varies with age and gender, and is greatest during early stages of use, ‘when process issues represent hurdles to be overcome, and later become overshadowed by instrumentality concerns’ [51, p. 450] In the most recent, unified model of technology acceptance, the relationship between perceived ease of use and usefulness (in this study, H3) has been eliminated [51]. Because the subjects in this study were trained on PSP and had used PSP, on average, on two projects, they may have moved beyond the early concern about the effort required for use. In order to further investigate the relationships between ease of use, usefulness, and use, we performed a qualitative analysis of the written comments provided by survey respondents in search of other relationships not examined by the research model. 4.2. Qualitative analysis of the research model 4.2.1. Procedure A section of our survey was provided for developers to supply written comments concerning their use of PSP Using a content analysis approach similar to that used by Rainer and Hall [40], we analyzed the comment data supplied by survey respondents. Of the 63 completed surveys used in the quantitative analysis, 44 of them (70%) contained writein comments. In addition, nine out of the 10 pilot study participants also supplied write-in comments. Therefore, there were 53 sets of written comments available for analysis. Each set of comments was analyzed and then categorized according to the PSP use-related issue(s) it addressed. Some comment sets addressed multiple use-related issues.

Ten surveys contained comments that simply clarified the job that the respondent performed or commented on the survey itself; these comment sets were removed from further analysis leaving 43 comment sets to be categorized. 4.2.2. Results Nine issues faced by developers emerged from the content analysis of comments. A description of each of these broad issues, as well as the number of comment sets addressing each issue, is documented in Table 6. Of the four independent variables in the research model, two (quality and productivity impacts) were clearly of importance to developers in their use of PSP, as evidenced by the fact that nearly one-third (NZ14) of the comment sets addressed these issues. In general, more comments reported improvements in software quality (NZ6) than did those reporting gains in productivity (NZ1). In fact, most comments addressing the issue of productivity noted that there had not yet been any visible impact (NZ2), or that productivity had actually decreased (NZ10). These results are consistent with the descriptive data reported in this study (see Table 5) as well as existing research on quality and productivity impacts of SPIs [30]. Yet several respondents indicated that they continue to use the PSP even if there has been little evidence of productivity or quality improvements to date. This observation is consistent with the quantitative PLS analysis which suggests that quality and productivity improvements do not impact SPI use directly, but do have a direct bearing on the degree to which the SPI is considered useful to the developer. Ease of use did not appear explicitly as an issue in comment data; however, some of the ‘;Documentation and Packaging’ concerns could be related to ease of use. With this assumption, the low number of comment sets addressing the Documentation and Packaging issue (NZ3 or 7%) is consistent with the quantitative analysis of the research model which indicates that ease of use may not be a

550

G.C. Green et al. / Information and Software Technology 47 (2005) 543–553

Table 6 PSP usage issues discovered during qualitative analysis (NZ43 comments analyzed) Issue

Issue description

Number of comments

Task-technology fit

Challenges and importance of targeting the PSP to specific environments, activities, and/ or projects Described ways developers used PSP for project management-related tasks (both work and personal projects) and personal benefits gained from using PSP for these activities Comments clarifying the type of software quality and developer productivity impacts that resulted from use of the PSP Need for management commitment to the PSP initiative in terms of supportiveness and, in some cases, providing standards/controls Concerns related to having too much process documentation Need for automated tools to support the PSP Need for entire teams to use the PSP, as opposed to few individuals Issues related to the importance, lack, or adequacy of training Identifying individual characteristics of developers that may impact which developer chooses to use a SPI, such as ‘methodical’, ‘smart’, ‘self-motivated’

18

Project management benefits Quality and productivity impacts Management support and control Documentation and packaging Tool support Team support Training Individual characteristics of developers

significant factor in explaining the usefulness of a SPI to a developer.

5. Discussion As indicated by the results of the quantitative analysis, perceived productivity and quality benefits explain 66% of the variance in the perceived usefulness of the SPI; however, perceived usefulness explains only 32% of the variance in SPI use. While this percentage is significant and generally consistent with other studies of IT use (e.g. in [10] perceived usefulness accounts for 36% of variance in use), it suggests that there may be other important factors that explain sustained SPI use. The additional issues uncovered during our qualitative analysis of comment data gives us insight into what some of these factors may be. Comments concerning quality and productivity impacts, and documentation and packaging (ease of use surrogate) support variables already contained on the research model. Descriptive statistics shown in Table 5 give some insight into specific ways PSP has impacted developer productivity and software quality. Examination of comment data gives further insight into this issue: † “[PSP] makes me more aware of typical syntax and data type errors I commonly make [based on data from] the past.PSP has also made me more confident in estimations of time to complete an assignment.” † “[PSP] gives me the structure I need to come back to programming tasks and pick them up and start up again with hope that I can maintain a high level of quality throughout.” † “.[PSP] helps me keep excellent track of defects at all levels.” † “What I use from PSP [most frequently] is time estimation., informal design review., [and] code walkthroughs.”

15 14 7 3 3 3 2 2

These comments point to time recording/estimation, defect recording, design and code inspections, and task planning as key features of PSP that have helped to improve quality and productivity. The remaining seven issues uncovered during the qualitative analysis are not included in the research model. The importance of management and automated tool support [21], adequate training [7], individual characteristics [50,51], and team support of SPI initiatives [21] have been well-studied and validated in the research literature and will not be discussed further in this paper. Less studied in the literature is the importance of matching the SPI innovation to the tasks performed by individual developers (task-technology fit) and the potential of the SPI to provide project management benefits. These two issues received the most attention in the comments data, with twothirds of the analyzed comment sets (NZ29) addressing one or both issues. These comments suggest that there are additional benefits and considerations that may motivate software developers to implement a SPI. 5.1. Task-technology fit The issue receiving the most attention in the comments data (NZ18) dealt with the importance of matching the SPI to the developer’s task or task environment. Comments suggest that when developers are responsible for tasks such as maintenance and debugging, work on exploratory projects, or programming in 4GL or Unix environments, they find the PSP to be more difficult to adapt, constraining their use of PSP in these environments. For example: † “The difficulty I have had with applying PSP has been trying to keep pace with changing platforms and languages and trouble adapting it to a maintenance environment.” † “More study needs to be done on how to use PSP with new tools/environments.”

G.C. Green et al. / Information and Software Technology 47 (2005) 543–553

† “.challenges we have.involve use of PSP during Maintenance phases of a project, and use of PSP with 4GLs.” † “I.work on small projects lasting [up to] 30 hours.I can usually use my intuition to accurately estimate such small projects.” † “.colleagues active in other activities like maintenance and support.leads to a weak usage of PSP.” † “What I use from PSP is creation of techniques for tasks I do frequently.” † “It is my experience that the techniques are useful when you are working on projects where a large percentage of what needs to be done is known and understood. In developing for new platforms or unfamiliar products, I found that PSP estimates were off by as much as a factor of 4.In our environment, we are constantly doing new things, making PSP.usefulness difficult [in this situation].” The above comments and others like them suggest that development managers should ensure that use of SPIs is clearly targeted for or tailorable to specific development tasks and environments. This observation has support in research examining task-technology fit. Goodhue [20] suggests that a lack of fit between an IT and the user’s task can inhibit use of the IT. Indeed, Goodhue [20] and others [31,32] suggest that the fit of the technology to the task of the user impacts the usefulness of the technology to the user. Broad implementations of SPIs will meet with less success than those implementations where SPI use is tailored to the specific needs of software developers.

551

† “We have a better knowledge of where we are on projects and what resources we require.” Principles gained from training and use of SPIs in software development may spill over to other work and non-work activities. The types of benefits described in these comments are more difficult to measure quantitatively; however, to some extent, they involve aspects of organizational learning. Current research leaves no doubt that organizational learning (which begins with individual learning) is essential to the viability and competitiveness of organizations [35]. Hardgrave et al. [22] note that ‘skill acquisition benefits’ is an important aspect of usefulness omitted by the technology acceptance model. Increased skills that developers gain by use of a particular development methodology can enhance their value as a software development professional. Hardgrave et al. call for future research to encompass this ‘perceived long-term consequences’ concept. The present research findings address this call and provide preliminary support for perceived long-term consequences increasing the usefulness of a SPI innovation to software developers. Since it is clear from this and other studies that productivity gains in particular cannot always be assured when implementing SPIs, we recommend that management use the personal and professional benefits that can be gained from SPI usage as an additional motivator for developers to adopt SPI techniques. Management can also use this benefit when promoting SPI efforts to upper management based on its value to organizational learning and competitiveness.

5.2. Project management and personal benefits 6. Conclusions Although quality and productivity gains are the most often-touted benefits of process innovations, the secondmost cited issue in the comments data relate to improvements in managing and understanding projects, as well as managing personal tasks (NZ15). † “.I have spent.very little time performing the design, code, test portions of the [development] cycle.HowHowever, I do continue to use PSP in my own personal projects and have found it beneficial there.” † “.on one small project, [PSP] did not improve my productivity or reduce my error rate.however, I understand the flow of my work better and know where to look for trouble. I can head off trouble before it starts.” † “.I have not noticed a significant amount of time saved. What I gained most from PSP was more effective techniques of reviewing my work.realizing the importance of following a checklist of review items.[coming] up with an organized method of reviewing my work.” † “I used PSP principles when giving presentations, keeping minutes, etc.”

This study has examined factors that motivate developers to adopt and sustain use of SPIs Previous research has noted the importance of SPIs having ‘visible success’ in order to motivate developers to use them. This research has been an attempt to uncover what constitutes visible success of a SPI to developers. Through a combination of quantitative and qualitative analyses, we have verified critical success factors for the success of a SPI to software developers. The SPI must demonstrate (or show the promise of demonstrating) positive impacts on quality and/or productivity and the SPI must be perceived as useful to the particular developer. From developer comments, we also find other factors that may enhance the usefulness of a SPI to a developer, including perceived personal and project management benefits, as well as how well the SPI fits the task environment of the developer. Interestingly, ease of use, when evaluated against potential SPI benefits such as gains in quality and/or productivity, is not of similar importance to software developers when deciding

552

G.C. Green et al. / Information and Software Technology 47 (2005) 543–553

to use process innovations, at least when they have moved beyond initial SPI use. No amount of ease of use can compensate for lack of visible benefits to the individual developer. There are limitations with this study to be noted. Because the innovation chosen at the time of the survey was fairly recent, and the study was not longitudinal, a few respondents noted that it was too early to assess productivity gains. Therefore, some respondents did not respond to productivity-related items (in which case the responses were not included in data analysis). Future research should repeat our survey of PSP use to determine if the perceived productivity impacts reported in this study vary significantly from actual productivity impacts. Another potential limitation of this study is that only one SPI (PSP) was evaluated, such that generalizability across SPIs is limited. While PSP is an example of a higher-level normative model for the entire development process, it is unique in that the focus is on use by individual developers. This is in contrast to other SPIs that must be adopted by an entire team (TSP) or an organizational unit (CMMI). However, our research model is based on established theory which has been applied to several IT innovations. Thus, we feel that future studies examining the diffusion of other SPIs will find results consistent with those reported in this study. Finally, consistent with most non-experimental research designs, there is a lack of formal control over study variables. While the relationship directions that we have proposed in our model are consistent with previous experimental and non-experimental diffusion research, a further test of our model in an experimental setting would further validate our findings. In addition to maintaining the traditional software engineering focus of improving software quality and developer productivity, this research should inspire development managers to emphasize more qualitative benefits of SPIs to software developers, as these benefits appear to be of great importance to an individual’s sustained use of SPIs. Further, managers are encouraged to have a clear understanding of the types of projects and development activities best suited to the SPIs they consider for implementation. SPIs that offer promising quantitative and qualitative benefits but do not readily fit the organization’s development environment should be tailored to fit the organization’s environment or perhaps avoided altogether.

References [1] D. Adams, R. Nelson, P. Todd, Perceived usefulness, ease of use, and usage of information technology: a replication, MIS Quarterly 16 (2) (1992) 227–248. [2] R. Agarwal, Individual acceptance of information technologies in: R.W. Zmud (Ed.), Framing the Domain of IT Management, Pinnaflex Educational Resources, Cincinnati, OH, 2000, pp. 85–104.

[3] R. Agrawal, J. Prasad, A field study of the adoption of software process innovations by information systems professionals, IEEE Transactions on Engineering Management 47 (3) (2000) 295–308. [4] P. Armour, The laws of software process, Communications of the ACM 44 (1) (2001) 15–17. [5] N. Badoo, T. Hall, Motivators of software process improvement: an analysis of practitioners’ views, The Journal of Systems and Software 62 (2002) 85–96. [6] M. Chrissis, M. Konrad, S. Shrum, CMMI: Guidelines for Process Integration and Product Improvement, Addison-Wesley, Reading , MA, 2003. [7] D.R. Compeau, L. Olfman, M. Sein, J. Webster, End user training and learning, Communications of the ACM 39 (7) (1995) 24–26. [8] R. Conradi, T. Dyba, An empirical study on the utility of formal routines to transfer knowledge and experience in: V. Gruhn (Ed.), Proceedings of the European Software Engineering Conference 2001 (ESEC ‘2001), Vienna, 10–14 Sept., ACM/IEEE CS Press, New York, 2001, pp. 268–276. [9] R. Conradi, A. Fuggetta, Improving software process improvement, IEEE Software July/August (2002) 92–99. [10] F. Davis, Perceived usefulness, perceived ease of use, and user acceptance of information technology, MIS Quarterly 13 (3) (1989) 319–339. [11] F. Davis, R. Bagozzi, P. Warshaw, User acceptance of computer technology: a comparison of two theoretical models, Management Science 35 (8) (1989) 982–1002. [12] T. Dyba, Enabling Software Process Improvement: an Investigation on the Importance of Organizational Issues, Unpublished thesis, Norwegian University of Science and Technology, Trondheim, Norway, 2001. [13] C. Ebert, Technical controlling and software process improvement, The Journal of Systems and Software 46 (1999) 25–39. [14] P. Ferguson, W. Humphrey, S. Khajenoori, S. Macke, A. Matvya, Results of applying the personal software process, IEEE Computer 30 (1997) 24–31. [15] R.G. Fichman, The diffusion and assimilation of information technology innovations in: R.W. Zmud (Ed.), Framing the Domains of IT Management, Pinnaflex Press, Cinncinati, OH, 2000, pp. 105–127. [16] R.G. Fichman, C.F. Kemerer, Adoption of software engineering process innovations: the case of object orientation, Sloan Management Review 34 (2) (1993) 7–22. [17] R.G. Fichman, C.F. Kemerer, The illusory diffusion of innovation: an examination of assimilation gaps, Information Systems Research 10 (3) (1999) 255–275. [18] W. Frakes, G. Succi, An industrial study of reuse, quality, and productivity, The Journal of Systems and Software 57 (2001) 99–106. [19] R. Glass, The Realities of software technology payoffs, Communications of the ACM 42 (2) (1999) 74–79. [20] D. Goodhue, Understanding user evaluations of information systems, Management Science 41 (12) (1995) 1827–1844. [21] G. Green, A. Hevner, Perceived Control of Software Developers and Its Impact on the Successful Diffusion of Information Technology, Carnegie Mellon University, Software Engineering Institute, CMU/SEI-98-SR-013, 1999. [22] B. Hardgrave, F. Davis, C. Riemenschneider, Investigating determinants of software developers’ intentions to follow methodologies, Journal of Management Information Systems 20 (1) (2003) 123–151. [23] J. Hartwick, H. Barki, Explaining the role of user participation in information system use, Management Science 40 (4) (1994) 440–465. [24] W. Hayes, J. Over, The personal software process (PSP): an empirical study of the impact of PSP on individual engineers, Technical Report SEI-97-TR-001, Software Engineering Institute, Carnegie Mellon University, Pittsburgh, 1997. [25] W. Humphrey, Introduction to the Personal Software Process, Addison-Wesley, Reading, MA, 1997.

G.C. Green et al. / Information and Software Technology 47 (2005) 543–553 [26] W. Humphrey, Introduction to the Team Software Process, AddisonWesley, Reading, MA, 2000. [27] J. Iivari, Why are CASE tools not used?, Communications of the ACM 39 (10) (1996) 94–103. [28] J. Kamatar, W. Hayes, An experience report on the personal software process, IEEE Software 17 (6) (2000) 85–89. [29] M. Khalifa, J. Verner, Drivers for software development method usage, IEEE Transactions on Engineering Management 47 (3) (2000) 360–369. [30] J. Kuilboer, N. Ashrafi, Software process and product improvement: an empirical assessment, Information and Software Technology 42 (2000) 27–34. [31] K. Lim, I. Benbasat, The effect of multimedia on perceived equivocality and perceived usefulness of information systems, MIS Quarterly 24 (3) (2000) 449–479. [32] K. Mathieson, E. Peacock, W. Chin, Extending the technology acceptance model: the influence of perceived user resources, The DATABASE for Advances in Information Systems 32 (3) (2001) 86–112. [33] T. McGibbon, A business case for software process improvement revised, DACS State-of-the-Art Report, Rome Laboratory, September 1999, http://www.dacs.dtic.mil/techs/roispi2. [34] M. Morisio, Measurement processes are software, too, The Journal of Systems and Software 49 (1999) 17–31. [35] S. Nambisan, R. Agarwal, M. Tanniru, Organizational mechanisms for enhancing user innovation in information technology, MIS Quarterly 23 (3) (1999) 365–395. [36] J. Nunnally, Psychometric Theory, McGraw-Hill, New York, 1978. [37] R. Pressman, Software Engineering: a Practitioner’s Approach, McGraw-Hill, New York, 1997. [38] S. Prowell, C. Trammell, R. Linger, J. Poore, Cleanroom Software Engineering: Technology and Practice, Addison Wesley, Reading, MA, 1999. [39] A. Rainer, T. Hall, Key success factors for implementing software process improvement: a maturity-based analysis, The Journal of Systems and Software 62 (2002) 71–84.

553

[40] A. Rainer, T. Hall, A quantitative and qualitative analysis of factors affecting software process, The Journal of Systems and Software 66 (2003) 7–21. [41] C. Riemenschneider, B. Hardgrave, F. Davis, Explaining software developer acceptance of methodologies: a comparison of five theoretical models, IEEE Transactions on Software Engineering 28 (12) (2002) 1135–1145. [42] T. Ravichandran, A. Rai, Quality management in systems development: an organizational system perspective, MIS Quarterly 24 (3) (2000) 381–415. [43] T. Ravichandran, A. Rai, Structural analysis of the impact of knowledge creation and knowledge embedding on software process capability, IEEE Transactions on Engineering Management 50 (3) (2003) 270–284. [44] S. Rifkin, Why software process innovations are not adopted, IEEE Software 18 (4) (2001) 112C. [45] E. Rogers, Diffusion of Innovations, third ed., Free Press, New York, 1983. [46] S. Sharma, A. Rai, CASE deployment in IS organizations, Communications of the ACM 43 (1) (2000) 80–88. [47] E. Swanson, N. Ramiller, The organizing vision in information systems innovation, Organization Science 8 (5) (1997) 458–474. [48] B. Szajna, Empirical evaluation of the revised technology acceptance model, Management Science 42 (1) (1996) 85–92. [49] V. Venkatesh, Creating favorable user perceptions: exploring the role of intrinsic motivation, MIS Quarterly 23 (2) (1999) 239–260. [50] V. Venkatesh, M. Morris, Why don’t men ever stop to ask for directions? Gender, social influence, and their role in technology acceptance and usage behavior, MIS Quarterly 23 (1) (2000) 115– 139. [51] V. Venkatesh, M. Morris, G. Davis, F. Davis, User acceptance of information technology: toward a unified view, MIS Quarterly 27 (3) (2003) 425–450. [52] D. Wilson, T. Hall, N. Baddoo, A framework for evaluation and prediction of software process improvement success, The Journal of Systems and Software 59 (2001) 135–142.