- Email: [email protected]
Social networks and coordination performance of distributed software development teams
Journal of High Technology Management Research 20 (2009) 52–61
Contents lists available at ScienceDirect
Journal of High Technology Management Research
Social networks and coordination performance of distributed software development teams Liaquat Hossain a,⁎, David Zhu b a b
Project Management Graduate Programme, The University of Sydney, NSW 2006, Australia Accenture, Sydney, Australia
a r t i c l e
i n f o
Available online 16 April 2009 Keywords: Coordination Social networks Performance Software development Defect management OSS (Open Source Software)
a b s t r a c t In this study, we explore the coordination performance of the geographically distributed software development teams by exploring OSS (Open Source Software) development dataset available through SourceForge.com. OSS team structures have traditionally been geographically dispersed and therefore, the coordination of post release activities such as testing efforts have been carried out by means of communication via electronic forms, such as email or message boards and forums. In our current communication-enriched environment, best practices for coordination are adopted by all software projects yet some still fail to achieve their target performance. Does team structure have any bearing on the performance outcome of the project? How does the communication between teams and their external parties affect ultimate success or failure of projects? We seek to answer above questions by applying existing theories and analytical methods from social networks for exploring the coordination performance of defect management activities found in OSS projects. We propose social networks based theoretical model for exploring distributed coordination structure and apply that for the case of OSS defect management process for exploring the structural properties, which induce the greatest coordination performance. The outcome of our suggest that there is correlation between certain network measures such as density, centrality and betweenness and coordination performance measures of defect management systems such as quality and timeliness. © 2009 Elsevier Inc. All rights reserved.
1. Introduction The Open Source approach to software development involves a group of loosely knitted volunteers to collaborate over a public medium of communication, most popularly the Internet, to create software (de Souza, Froehlich, & Dourish, 2005). The source code is open to public access and it is this readily available, which result in faster and more responsive development cycles, thus producing more robust and secure software. These inherent characteristic has some interesting connotations to OSS project teams and how the software is tested. What is phenomenal about an OSS projects, is that its participants tend to form a community that is “bounded by their shared interest in using/developing the system” (Ye & Kishida, 2003). In this study, the particular coordination activities we are interested are related to bug reporting, fixing and knowledge sharing post release date of the OSS. The general focus of existing studies have been coordination between developers during pre-release phase of the project with minimal attempts to include OSS project community's involvement. We suggest that the evolution of any OSS project originates from the input or identification of defects from the OSS project's community. We explore questions such as: Does the degree of centrality, betweenness and density of the network have any bearing on the number of defect fixed per software promotion? Does the degree of centrality, betweenness and density of the network have any bearing on the number of defects reported at different severity levels? Does the degree of centrality, betweenness and density of the network have any bearing on
⁎ Tel.: +61 2 9306 9110; fax: +61 2 9351 8642. E-mail address: [email protected] (L. Hossain). 1047-8310/$ – see front matter © 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.hitech.2009.02.007
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
53
the average number of days for a defect to be fixed for each project team? Does the degree of centrality, betweenness and density of the network have any bearing on the average number of days for responses by developers on defects? 2. Towards a social network based coordination approach The conceptual origin of social network analysis can be traced as far back as the 1930's when leading ‘Gestalt’ theorists Jacob Moreno devised the ‘sociogram’ to represent the formal properties of social configuration (Scott, 2000). Moreno's (1930) sociogram (as cited in Scott, 2000) depicted individuals using points and their social relationship with others by lines, he argued using this type of representation researchers could identify leaders(stars) and isolated individuals(isolates), to uncover asymmetry and reciprocity and to map chains of connection. In modern studies, researchers has taken used Moreno's (1930) suggestions and analysed networks based on either team structure. (Pearce & David, 1983) concluded there to be a definite relationship between ‘organisation design variables and group performance’ from their study of two types of organizational design models each with their own setup of stars, isolates and liaisons and levels of reciprocity. The framework of social network analysis was finally crystallized in the 1950's through the works of fieldworkers associated with the Department of Social Anthropology at Manchester University. Clyde Michell's can be attributed to turning the existing studies of graph theory used for sociometric analysis and formulating it to fit a specific sociological framework (Scott, 2000). John Barnes conducted intensive studies on the part kinship, friendship and neighbouring relationships played in the formation of communities. Barnes claimed that ‘the whole of social life’ could be set as ‘a set of points some which are joined by lines’ to form a ‘total network of relations’. The informal sphere of interpersonal relations can be seen as a part of a ‘partial network’, of this total network (Barnes, 1954). The final breakthrough that solidified the global properties of SNA (social Networks Analysis) in all fields of social life was made at Harvard by Harrison White and his associates. The first part was the development of algebraic models of groups using set theory to model relationships and the second was the development of multidimensional scaling, which was a technique for translating relationships into social distances and for mapping social space, which was the key idea behind Lewins work (Scott, 2000). The earliest example of combining theories of networks with coordination can be found in research by Alex Bavelas in the early 1950's. The experiment was devised so that individuals had to find other person who possesses the card with the same symbol as they did. The way of communication was through written messages and the group was setup in different formations of networks. The result of their research was that groups in the star or Y networks solved the problem the quickest, with fewest errors and least mistakes and those at the center of the star or Y network recorded the highest level of satisfaction (Kosfeld, 2003). de Souza et al. (2005) conducted on social network based investigation into distributed software development with its focus using artifacts and activities to determine the social structure of the software project (de Souza et al., 2005). With regards to examining distributed coordination, Sandusky and Gasser (2005) observed the impact of ‘negotiation’ as a mechanism of coordination in distributed software problem management systems (SWPM). The research was centered on publicly accessible bug repositories for OSS and negotiation in the SWPM process (Sandusky & Gasser, 2005). It is important to note that from the research it was found that OSS community bug repositories represent the major coordination mechanism in OSS projects. Finally, Gutwin, Penner et al. (2004) studied the coordination of distributed software development focusing on group awareness with the aim of unearthing how such geographically dispersed group stay aware of each other work and coordinates the development of large complex software projects. The result of the study outlined the importance of organizational culture and text based communication tools to maintain awareness (Gutwin, Penner et al., 2004). From the literature reviewed, it is evident that focus of the work has been conducted in the field of distributed software development of OSS has primarily been on prerelease coordination activities. For example, the focus of bug repositories study by Sandusky and Gasser (2005) was negotiation and not coordination measures. This leaves scope for investigating social network properties and their implications on coordination measures of OSS teams. 2.1. Structural variables of social networks for understanding distributed coordination of OSS There are certain variables of social networks that will be the focus of our analysis for the coordination in OSS projects. There are some general concepts and definitions that will be extensively used in the analysis sections that need to be firstly introduced. These definitions will help form the foundation of our social network based analysis of the OSS projects and their wider community. Social networks analysis is deeply rooted in graph theory, however it is important to distinguish that a graph in this context is ‘simply a set of lines connection points’ and graph theory consists of a ‘body of mathematical axioms and formulae that describe the properties of the patterns formed by the lines’ (Scott, 2000). • • • •
Two points are said to be adjacent if a line connects them Those points which are adjacent to a particular point is said to be the neighbourhood The total number of points within a point's neighbourhood is said to be its degree of connection Points can be connected directly by a single line or indirectly connected by a sequence of lines. The sequences of lines is said to be a walk • A walk in which each line and point is distinct is called a path • The length of a path is said to be the number of lines that make up the path • The distance between two points is the length of the shortest path between them.
54
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
In Fig. 1 we take A as our focus of analysis. A has the neighbourhood of B, C, D, E, F, G, H and the degree of connection of the entire neighbourhood for A equal 7. The length of the path of A to F can be simply 1 if we directly connect to it or it can be via B, C, D, and E which will give its length to be 5. However if we're calculating distance from A to F it will be 1 since it is the shortest length. 2.2. Density of the network In graph theory, density describes the general level of linkages among the points on a graph. When this is applied to the social network context each point of the graph represent individuals of the network and each line is a representation of interaction or communication between the two individuals. As describe in (Scott, 2000), density depends on two parameters; inclusiveness of the graph and the sum of the degrees of its points. Inclusiveness refers the total number of points in a graph minus the number of isolated points. An Isolated point is one which has no lines connected to it and contributes nothing to the density of the graph. The general formula for density is: l nðn − 1Þ = 2 where l is the number of lines present and n is number of points present. This measure uses the number of lines present in a graph compared to the number of lines theoretically possible. The number of points n representing actors involved in the network is the key factor here which has implications to the total number of lines possible. The formula for density can be applied in one of two ways, the ‘ego-centric’ or ‘socio-centric’ approach. The ego-centric approach concentrates on the density of the links surround a particular agent while the ‘socio-centric’ approach focuses on the pattern of connections in the network as a whole. For the purpose of this paper the ego-centric approach will be used because of the nature of OSS test is incredibly distributed in its participants. Studies by (Ye & Kishida, 2003) have shown that for OSS communities most participants are regarded as passive users and as for the Apache which was the focus of their study, the figure was up around the 99% proportion were either readers or passive users. 2.3. Centrality of the network The topic of centrality has been at center of many academic papers and empirical investigations and its origin is derived from the sociometric term of the ‘star’. Bavelas can be accredited as being the pioneer of the establishing the formal properties of centrality. In the late 1940's and early 1950's, Bavelas and Dermont (1951) and his colleges at Massachusetts Institute of Technology directed one of the first research studies into the applications of centrality. The research concluded that ‘centrality is related to group efficiency in problem-solving, perception of leadership and the personal satisfaction of participants’ (Freeman, 1979). The proliferation of the study of centrality on the context social network analysis has been proven to be far reaching. There are numerous measures that have been put forth, some of which include degree centrality, closeness, betweenness, and the rush index just to name a few. Borgatti argues that these measures often make certain assumptions on characteristics of the network. If these characteristics were not applied then we would lose the ability to fully
Fig. 1. Social network.
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
55
interpret these measures (Borgatti, 2005). Therefore in the following section the network characteristics that will be assumed for our analysis will be identified. 2.4. Local and global centrality A point is considered locally central if it has a large number of connections with other points within its neighbourhood. For this study we will be using a degree-based measure of local centrality which is simply computed by the number of points to which a point is adjacent. We can also extend this measure to include indirect connections and set a cut-off distant to which we measure its local centrality. For example if we set the cut-off distant at 2 that means a networks measure of local centrality will include the direct connection between two points A and H plus the connection between H and B (see Fig. 2 — Local and Global centrality (Scott, 2000)). A point is consider globally central when it has ‘a position of strategic significance in the overall structure of the network’ (Scott, 2000). Freeman proposed another measure based on what he called ‘closeness’ which identify the globally central points of the entire network. A point is said to be the globally central if it lies at shortest distance from many other points, therefore global centrality is calculated by the sum of the geodesic distances from one point to all other points within the network (Scott, 2000). From the definition we see that in Fig. 2 — Local and Global centrality (Scott, 2000) the global central of the network is point B because it has the lowest sum of geodesic distances from all other points in the network. Freeman (1977) adds further to the concept of centrality with what he terms as betweenness, which measures the extent to which a particular point lies between the various other points in the graph. Freeman explains that in terms of centrality it is not only the person with the most connections with the entire network that should be of interest, but also the person “in a group that is strategically located on the shortest communications path connecting pairs of others” (Freeman, 1977) shall also be referred to as a central actor. This person assumes responsibility of passing on information between the two pairs he is connected with, therefore the performance of a system depends on the coordination of activities from this central person. Freeman's approach centers about the idea of local dependency where a point is dependent on another if the paths which connect it to the other points pass through this point (Scott, 2000). What is of interest is that the role points like H and L play in the performance of OSS teams, whether their presence as intermediaries aids or hinders the average time bugs gets fixed. The model proposed here forms the basis of analysis conducted on the coordination of distributed open source software teams from a social network perspective (see Fig. 3 for Social Network based model for coordination). The model is based on some of the key structural variables as supported by the investigation into existing literature of social networks and coordination. From the measures of density, centrality and betweenness we can analysis the properties of the network for an OSS team for a particular
Fig. 2. Local and global centrality (Scott, 2000).
56
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
Fig. 3. Social network based model for coordination.
project. Then through examining the coordination measures of quality and timeliness, we can rank the project team in terms of their coordination performance. From these scores, we can then compare different OSS teams and form conclusions on whether each of the structural variables has influences on the coordination of distributed teams. However, there are mitigating factors which affect the results of the analysis. Factors such as defect severity and development team size needs to be kept in mind so that a basis for comparison can be formed to ensure the integrity of the results. 2.5. Density as a network variable affecting quality and timeliness Density as described earlier is a measure of the number of connections formed within a network as a proportion of total possible connections between all nodes of a network. Studies into density have been conducted such as the one by (Crowston & Howison, 2006), where they have found a negative correlation between the project size and density of the network. From their empirical work it would seem that the connections a for certain key participants remain the same while the project grows in size thus making the network less dense(Crowston & Howison, 2006). From a coordination perspective, research has been done by Dinh-Trong and Bieman (2004) into comparing the defect density and general performance measures between different projects and seeing if indeed OSS produces better quality software (Dinh-Trong & Bieman, 2004). Therefore it would be of interest to find the correlation, if any, between the network variable density and key performance indicators of OSS project. 2.6. Centrality as a network variable affecting quality and timeliness For the purpose of this study, the ego-centric approach to measuring centrality is used to investigate the correlation between levels of centrality within a network and the quality and timeliness of OSS project. Taking the definition that a highly centralized network is one that core developers are involved in high number of defect fixing, (Crowston & Howison, 2006) governed an empirical study into centralization in OSS team communications and found there to be larger projects with higher participants are lowly centralized. Other works contributes to the belief that OSS projects the core team consist of 10–15% of the total number of people make the majority contribution to the project (Ari, 2001). This again has implications on the centrality of the network and more to our interest the degree of centrality of key individuals and their affect on our defined coordination measure of quality and timeliness. 2.7. Betweenness as a network variable affecting quality and timeliness Betweenness, as described by (Scott, 2000) is the measure of the extent of which and actor can play the part of a ‘broker’ or ‘gatekeeper’ with a potential for control over others. This definition raises an interesting question, how influential is the role of a
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
57
broker or ‘liaisons’(Pearce & David, 1983) in coordinating bug fixing activities? Or does existence of too many liaisons within a network hinder the quality and timeliness of the OSS project? Studies based on betweenness by Newman (2005) have taken measures based on random information flows throughout a network and identifying first the most traveled paths and then those actors that are located along these paths (Newman, 2005). These actors signify high level of betweenness and centrality. Obviously this has connotations upon the coordination measures within our context of OSS bug fixing where the actions of those actors with high betweenness measure can potentially make or break the success of the project because they are those that control the follow of information between groups to which they are connected. Hypothesis. Quality of defect-management activities is influenced by social network measures. H1. Quality of code measure is influenced by degree centrality, node betweenness and density of the OSS team. H2(a). Defect density measure is influenced by degree centrality, node betweenness and density of the OSS team (high-severity). H2(b). Defect density measure is influenced by degree centrality, node betweenness and density of the OSS team (medium severity). H2(c). Defect density measure is influenced by degree centrality, node betweenness and density of the OSS team (low severity). Timeliness of defect-management activities is influenced by social network measure H3. Time-of-fix measure is influence by degree centrality, node betweenness and density of the OSS team. H4. Time-of-response measure is influence by degree centrality, node betweenness and density of the OSS teams. 3. Defect management process as a coordinated system Distributed Software Test teams are a recent trend that has emerged due to quality and ease of communication across vast physical locations and the boundless efforts of all corporations to reduce cost associated to software development. As a result entire projects are now divided so that teams in charge of different phases of the project can be located oceans apart. The relevance of coordination theory comes to the forefront when we analysis Distributed Software testing because apart from the multiple levels of communication from the tester to steering committee that must be coordinated, the actors must also contend with the physical distance between each team. By the using dependencies of coordination theory we can deduce theoretical models that would help achieve the goal of software testing more effectively. In order to provide some context on the coordinated system we are applying measures of coordination, it is only appropriate to outline the defect management process that distributed OSS teams must go through to ensure only legitimate and valid defects are accepted to be fixed. The Defect Management process is considered a coordinated system because it involves actors (developers and authors of the defect), performing interdependent tasks (raising and fixing defects) to achieve a common goal (advancing the OSS). How do we know what is the most effective application of each coordination model? To answer this question we need apply our understanding of coordination components universal to all Coordinated Systems and whether the different methods of applying these components affect result of existing metrics used to measure the effectiveness of software integration testing. Before we elaborate the measures of OSS testing and how these relate to coordination, we need to step back and justify the validity of such metrics in regards to measuring coordination. We have previously established that OSS testing can be categories as a Coordinated System. This system comprises of actors that perform interdependent tasks to achieve goals, the ultimate goal being to produce software to the community on a timely fashion, error free and satisfies the requirements specified. In order to achieve these goals, the actors working together and liaising with community members must manage the components of coordination such as resources allocation, producer/consumer relationship, simultaneity constraints and task dependencies. The measures of effectiveness include (i) quality and (ii) timeliness all of which are aligned with the ultimate goal of testing OSS projects. Quality measures have a direct correlation with the Producer/Consumer relationship component of Coordination theory. The core development team (producer) releases the software to the OSS community (consumer) then active or passive user logs defect reports when they arrive at an incorrect or unexpected behavior from the system. The cause of the defect invariably relates to the Producer/Consumer relationship, particularly the prerequisite constraints, transfer and usability aspects. To ensure that the software produced has a low quality of code metrics value and defect density metric value we must coordinate activities so that Prerequisite Constraints such as environment connectivity are correct. We must also ensure vital knowledge of the system is transferred from for example the development team to the test teams. The last element of Usability will naturally fall into place once the previous elements are managed accurately. Quality of Code metric captures the relation between the number of weighted defects and the size of the product release. WF NCSPT where WF is number of weighted defect found post release software release. The weight of each defect is dependent on the severity. NCSPT is the number of commit statements per time period. In this metrics the lower the number is, indicates the fewer defects or less serious defects found thus the higher is the quality of the code. Traditionally quality of code is measured using per
58
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
thousand lines of code, however this method of measure created efficiencies such as “bloaty” (Mockus, Fielding et al., 2002). With proprietary software the common practice in minimizes the effects of “bloaty” code, was to only counting every thousand lines of new code. Also the incremental delivery model of software development means that with each release only sections of new code is built on existing baseline of code from the previous release. By only taking the new lines of code into consideration a more accurate depiction of the quality of the code. In the spirit of avoiding “bloaty” code to affect the accuracy of the measure of quality this paper used NCSPT because with regards to OSS projects they are tracked according to commit statements which are logged and track on OSS project website. Each commit statement indicates patches and fixes to the software thus the proportionality between number of total defects raised and number of commit statement indicators if the quality of the code is high or low. Naturally high a number indicates that many defects have arisen for each commit statement which is a clear sign of poor quality of code. Defect Density metrics shows the relation between numbers of weighted defects per severity level and the total number of defects detected post release. WPSL ⁎100 WF where WPSL is number of weighted defects per severity level found during the post release period of the OSS and WF represents the total number of weighted defects found. Using this metrics we find that, higher the number shows higher ratio of defects were detected before release and higher effectiveness of the test. The motivating factor behind this measure is that we can arrive at a comparison based on number of defects of a certain severity level as a proportion of overall defect numbers. Naturally projects with high quality will have fewer defects within the high range of defect severity. The successful result of the following Timeliness Metrics can be associated the management of Simultaneity constraints and task dependencies. Imagine a typical software testing phase, with absolute certainty there dependencies on which activities can be performed first, in parallel and last. It is up those in the position to coordinate the activities to ensure that those tasks categorized as high priority because many other tasks depend on its success are run first. Those that can be run in parallel are organized in such a way and those many dependencies on other activities are run last. Although coordination theory is a very powerful method of analyzing areas of improvement for OSS testing, there still exist limitation and questions that must be answered. For example what type of affects does personal characteristics or properties of actors have on the outcome of the testing? Would group structure affect which activities that can be performed? In a distributed network of groups that must perform tasks that or interdependent, what type of links induce the best overall outcome for the software integration testing phases? A Social Network based approach will help complete this picture and offer a more round model when combined with our coordination model for improving the outcomes of software integration testing. Density as described earlier is a measure of the number of connections formed within a network as a proportion of total possible connections between all nodes of a network. Studies into density have been conducted such as the one by (Crowston and Howison (2006), where they have found a negative correlation between the project size and density of the network. From their empirical work it would seem that the connections a for certain key participants remain the same while the project grows in size thus making the network less dense(Crowston & Howison, 2006). From a coordination perspective, research has been done by Dinh-Trong and Bieman (2004) into comparing the defect density and general performance measures between different projects and seeing if indeed OSS produces better quality software (Dinh-Trong & Bieman, 2004). Therefore, it would be of interest to find the correlation, if any, between the network variable density and key performance indicators of OSS project. Therefore, in this paper, we only test the following two assumptions: H1. Quality of code measure is influenced by degree centrality, node betweenness and density of the OSS team. H2. Defect density measure is influenced by degree centrality, node betweenness and density of the OSS team (high-medium-low severity). 4. Data source Data was collected via the monthly data-dump SF.net provided to the Department of Computer Science and Engineering, University of Notre Dame for sole purpose of supporting academic and scholarly research on the Free/Open Source Software phenomenon. We were granted access to the university's specialist wiki-supported website dedicated to the study of Free/Libre/ Open Source Software projects. This site provided the SF.net ER-diagrams, database schema definition and SQL query functionalities require to extract the necessary data specified in previous section. The SQL query tool was a web-based program that allowed for simple SQL queries and timed out when more complex queries were required to be executed. Therefore, the list of projects used for analysis was chosen from the 100 most actively participated projects according to monthly download rate, forum activity and pages view. From this list of 100 only the projects with more than 7 developers and 200 bug reports were included. We felt that 7 developers provided a robust enough network for our studies and 200 bug reports represented enough instances of group interaction between the actors of the network. According to these criteria of selection a group of 45 projects were identified to be appropriate for use in our analysis into social network of OSS team. After the short list of projects that is to be included in the analysis was established we went about extracting the necessary data from appropriate tables and fields according the ER-Diagram and schema definitions provided by Notre Dame University. SQL
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
59
queries were executed to extract the necessary data and the results of each query were saved as a text file with ‘:’ (colons) used as separators for each field. Each text file contains all closed status bug reports for one of the 45 projects which fit the selection criteria. The type of refinement needed for coordination measures of timeliness included conversion of the time-stamped fields such as artifact.open_date, artifact.close_date and artifact_message.adddate, because SF.net uses UNIX formatted time-stamps for date fields. Unix time describes time as the number of seconds elapsed since midnight UTC of 1st January, 1970, therefore the following formula was applied in Excel to convert the time-stamp into a more meaningful manner. UNIXtime + ð365⁎70 + 19Þ⁎86400 − 0:41667 86400 Once the time format has been changed we used existing Excel functions to calculate the duration for each closed was fixed according the artifact.open_date and artifact.close_date then summing the total of the durations and dividing the by the total number of bugs we have a coordination measure of Time-to-fix-Bugs metrics for each project. A similar approach was used to calculate the Time-to-Respond metrics but instead the artifact_message.adddate was subtracted from the artifact.open_date with the average time similarly calculated by totaling the duration of each response and dividing by the total number of artifact_message. For the measures of quality, defect density required aggregation of the different categories of the aftifact.priority field. Currently SF.net's tracker system uses a 9-level priority system to categorize the severity of each bug, with 9 being the most severe and 1 being the least. This system presents ambiguity when used in our analysis of the defect density when defining the difference between a priority 5 bug and priority 4 or 6 bug. Thus to clarify the severity of the bugs raised for each project we have concluded that due to the number of bugs found in each of the 9 priority levels, 1–3 priority bugs are deemed “low-severity”, 4–6 priority bugs referred to as “moderate severity” and finally 7–9 priority bugs classified as “high-priority”. For the final measure of quality of code, total number of bugs raised and commit statement executed for each project were extracted from the Project summary page. 5. Results and discussions Using the multiple regression model of analysis we arrived at a set of results which we can use to interpret the relationship between dependent variables of coordination and independent variables of social network characteristics. This section reports on the findings from this analysis conducted. We analyse the regression output of the social network independent variables (centrality, betweenness and density) regressed on both measures of quality of coordination respectively. From our analysis of the relationship between the quality of code measure and degree centrality, betweenness and density Excel processed the following output, and regression model (Table 1): Y ¼ 2:593 þ 11:361⁎Centrality þ 9:136⁎Betweenness−3:862⁎Density We can only accept the hypothesis if for F-test we obtain an F-statistic greater than 2.61 (critical value for F at 3.41 degrees of freedom), additional to this the significance-F value that is less than 0.05 which is the predetermined level of significance. In term of the significances of the independent variables, each t-statistic needs be greater than 1.6829 or less than − 1.6829 (critical value for a two-tailed t-test at 41 degrees of freedom) and p-values less than 0.05. If these conditions are met then we can conclude that there is sufficient evidence of a relationship between quality of code and centrality, betweenness and density. Interpreting the output we accepted hypothesis because the F value of 16.428 which is greater than the critical value of 2.61 for a 5% level of significance. Furthermore, we concluded that each independent variable has strong significance to the model with either respective p-values all
Table 1 Quality of code output. Regression statistics Multiple R R2 Adjusted R2 Standard error Observation
Regression Residual Total
Centrality Betweennes Density
0.788236 0.62131599 0.59909888 13.5134966 45 df
SS
MS
F
Significance F
3 41 44
8999.8732 7487.19761 7735.61699
2999.958 182.6146
16.42781
0.0062626
Coefficients
Standard Error
t statistics
p-value
11.3614902 15.1359258 0.86176736
6.25387107 6.39409508 0.35632672
1.816713 2.367172 2.41848
9.36E− 16 2.323E− 12 0.000032
60
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
being greater than 1.6829 which is the critical values for t-distribution test at 5% level of significance. Therefore each of the βi's contribute to the model. The moderate Adjusted R2 suggest that the model as a whole has average explanatory power and around 60% of the variations in observed values for quality of code can be explained by the variations of values of centrality, betweenness and density. From the strong p-values of the independent variables we can conclude the following: (i) for each percentage increase of centrality, on average it will increase the quality of code measure by 11.361 units, holding all other variables constant; (ii) for each percentage increase of betweenness, on average it will increase the quality of code measure by 15.136 units, holding all other variables constant; and, (iii) for each percentage increase of density, on average it will decrease the quality of code measure by 3.862 units, holding all other variables constant. What these conclusions illustrate is that while centrality and betweenness of the network have a positive correlation with quality performance of OSS teams, interestingly density has a negative correlation. This result suggests that increasing the level of centrality and betweenness of an OSS team will increase the number of bugs fixed per commit statement. However, increasing the communication links between actors will actually decrease this measure of quality in the coordination of defect management activities. For the measure of defect density three separate regression analysis was conducted according three levels of defect that was classified for each OSS project. This is to take into the account of the mitigating factor of defect severity levels. As also previously discussed when a defect is recognized as valid it is assigned in severity or priority level which is an indication of the affect this defect has on the project. Naturally higher severity defect demand more attention because it affects the behavior of the software to a greater extent. For this reason we decided to conduct separate regression analysis on the three levels of defect severity to see what relationship exists between that and centrality, betweenness and density. The regression analysis produced the following output and model: Y ¼ 2:593 þ 11:361⁎Centrality þ 9:136⁎Betweenness−3:862⁎Density Here, we can only accept the hypothesis if for an F-test we obtain an F-static greater than 2.61 (critical value for F at 3, 41 degrees of freedom), additional to this the significance-F value that is less than 0.05 which is the predetermined level of significance. In term of the significances of the independent variables, each t-stat needs be greater than 1.6829 or less than −1.6829 (critical value for a two-tailed t-test at 41 degrees of freedom) and p-values less than 0.05. If these conditions are met then we can conclude that there is sufficient evidence of a relationship between quality of code and centrality, betweenness and density. Interpreting the output we can only tentatively accept the H2(a) because whilst the F value of 4.310 is greater than the critical value of 2.61 for a 5% level of significance. However, each independent variable has shows poor insignificance to the model with their respective p-values all being greater than − 1.6829 and less than 1.6829 which is the critical values for t-distribution test at 5% level of significance. Therefore, each of the βi's does not contribute to the model. Furthermore the low Adjusted R2 suggest that the model as a wholes has poor explanatory power and only around 30% of the variations in observed values for defect density for high severity defects can be explained by the variations of values of centrality, betweenness and density. The above results suggested that we do further analysis into the model and the decision was taken to included an interaction term into the model. The adjusted model included the introduction of centrality ⁎ density and betweenness ⁎ density. These two interaction terms were chosen because networks with high degrees of density tend to have lower degrees of centrality and betweenness, this is due to the increase connections between actors thus reducing the centralization measure of individual actors. Intuitively this assumption can be made because as the ties amongst actors increases there is less need for actors to communicate through other actors instead they will more likely to contact the source of the knowledge consequently reducing the need to central or between actors (Table 2). Table 2 High severity defect output. Regression statistics Multiple R R2 Adjusted R2 Standard error Observation
Regression Residual Total
Centrality Betweennes Density
0.4828337 0.2331284 0.1998723 0.0008997 45 df
SS
MS
F
Significance F
3 41 44
0.22459875 0.80244947 1.02704823
0.07486625 0.01957193
3.82518
0.01665228
Coefficients
Standard error
t statistics
p-value
− 0.077590007 − 0.096314342 − 0.008513927
0.08544898 0.08690062 0.0758123
− 0.9080273 − 1.1083269 − 0.1123030
0.176248 0.322982 0.017061
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
61
6. Conclusions This study suggests that there is a correlation between social network characteristics and strong and poor performing projects in an OSS environment. The projects which were analysed using our theoretical model all fitted the criteria of more than 7 developers and more 200 defect reports. The conditions we have set on the data help validate the strength of the results because such criterions provide a robust network of interaction between its actors also it facilitates varying sized networks. Analysis of the data displayed a normal distribution which fitted our proposed parameterized, regression analysis. We believe that we have made a significant contribution into the study of distributed work groups, social network analysis and coordination. Studies by Madey, Freech et al. (2002) have investigated the social network phenomenon, power-law relationship(Madey, Freeh et al., 2002), in OSS development teams and suggested to look into degrees of separation of the connection which we have analysed through density and centrality. References Ari, I. (2001). Quantitative Analysis of Open Source Software Projects. [Retrieved on 11/7/07]. Barnes, J. A. (1954). "Human Relations." Class and Committee in a Norwegian Island Parish 7. [Retrieved on 12/7/07]. Bavelas, A. (1950). Communication Patterns in Task-Oriented Groups. Journal of the Acoustical Society of America, 22(6), 725−730. Bavelas, A., & Dermont, B. (1951). An Experimental Approach to Organizational Communication. Personnel Psychology, 27. Borgatti, S. (2005). Centrality and network flows. Social Networks, 27, 55−71 [Retrieved on 9/7/07]. Crowston, K., & Howison, J. (2006). Hierarchy and centralization in free and open source software team communication. Knowledge, Techology and Policy, 18(4), 65−85 [Retrieved on 9/7/07]. Dinh-Trong, T., & Bieman, J. M. (2004). Open source software development: a case study of FreeBSD. 10th International Symposium on software metrics (pp. 96−105). [Retrieved on 12/7/07]. Freeman, L. C. (1977). A set of measures of centrality based upon betweeness. Sociometry, 40, 35−41 [Retrieved on 9/7/07]. Freeman, L. C. (1979). Centrality in social networks: conceptual clarification. Social Networks, 1, 215−239 [Retrieved on 1/8/07]. Gutwin, C., Penner, R., & Schneider, K. (2004). Group awareness in distributed software development. Computer supported cooperative work, 72−81. Kosfeld, M. (2003). Network experiments. Zurich IEER Working Papers (pp. 152). [Retrieved on 13/7/07]. Madey, G., Freeh, V., Tynan, R. (2002). The open source software development phenomenon: An analysis based on social network theory. Eighth Americas Conference on Information Systems (pp. 1806–1813). Mockus, A., Fielding, R. T., & Herbsleb, J. D. (2002). Two case studies of open source software development: Apache and Mozilla. ACM Transactions on Software Engineering and Methodology, 11(3), 309−346. Newman, M. E. J. (2005). A measure of betweenness centrality based on random walks. Social Networks, 27(1), 39−54 [Retrieved on 14/7/07]. Pearce, J. A., & David, F. R. (1983). A social network approach to organisational design-performance. The Academy of Management Review, 8(3), 436−444 [Retrieved on 7/7/07]. Sandusky, R. J., & Gasser, L. (2005). Negotiation and the coordination of information and activity in distributed software problem management. Conference on Supporting Group Work (pp. 187−196). [Retrieved on 17/7/07]. Scott, J. (2000). Social Network Analysis: A handbook. [Retrieved on 9/7/07]. de Souza, C.R.B., Froehlich, J., & Dourish, P. (2005) Seeking the Source: Software Source Code as a Social and Technical Artifact. ACM International Conference on Supporting Group Work (GROUP 2005) (pp: 197–206). Ye, Y., & Kishida, K. (2003). Towards an understanding of the motivation open source software developers. International Conference on Software Engineering (pp. 419−429). [Retrieved on 8/7/07].
Contents lists available at ScienceDirect
Journal of High Technology Management Research
Social networks and coordination performance of distributed software development teams Liaquat Hossain a,⁎, David Zhu b a b
Project Management Graduate Programme, The University of Sydney, NSW 2006, Australia Accenture, Sydney, Australia
a r t i c l e
i n f o
Available online 16 April 2009 Keywords: Coordination Social networks Performance Software development Defect management OSS (Open Source Software)
a b s t r a c t In this study, we explore the coordination performance of the geographically distributed software development teams by exploring OSS (Open Source Software) development dataset available through SourceForge.com. OSS team structures have traditionally been geographically dispersed and therefore, the coordination of post release activities such as testing efforts have been carried out by means of communication via electronic forms, such as email or message boards and forums. In our current communication-enriched environment, best practices for coordination are adopted by all software projects yet some still fail to achieve their target performance. Does team structure have any bearing on the performance outcome of the project? How does the communication between teams and their external parties affect ultimate success or failure of projects? We seek to answer above questions by applying existing theories and analytical methods from social networks for exploring the coordination performance of defect management activities found in OSS projects. We propose social networks based theoretical model for exploring distributed coordination structure and apply that for the case of OSS defect management process for exploring the structural properties, which induce the greatest coordination performance. The outcome of our suggest that there is correlation between certain network measures such as density, centrality and betweenness and coordination performance measures of defect management systems such as quality and timeliness. © 2009 Elsevier Inc. All rights reserved.
1. Introduction The Open Source approach to software development involves a group of loosely knitted volunteers to collaborate over a public medium of communication, most popularly the Internet, to create software (de Souza, Froehlich, & Dourish, 2005). The source code is open to public access and it is this readily available, which result in faster and more responsive development cycles, thus producing more robust and secure software. These inherent characteristic has some interesting connotations to OSS project teams and how the software is tested. What is phenomenal about an OSS projects, is that its participants tend to form a community that is “bounded by their shared interest in using/developing the system” (Ye & Kishida, 2003). In this study, the particular coordination activities we are interested are related to bug reporting, fixing and knowledge sharing post release date of the OSS. The general focus of existing studies have been coordination between developers during pre-release phase of the project with minimal attempts to include OSS project community's involvement. We suggest that the evolution of any OSS project originates from the input or identification of defects from the OSS project's community. We explore questions such as: Does the degree of centrality, betweenness and density of the network have any bearing on the number of defect fixed per software promotion? Does the degree of centrality, betweenness and density of the network have any bearing on the number of defects reported at different severity levels? Does the degree of centrality, betweenness and density of the network have any bearing on
⁎ Tel.: +61 2 9306 9110; fax: +61 2 9351 8642. E-mail address: [email protected] (L. Hossain). 1047-8310/$ – see front matter © 2009 Elsevier Inc. All rights reserved. doi:10.1016/j.hitech.2009.02.007
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
53
the average number of days for a defect to be fixed for each project team? Does the degree of centrality, betweenness and density of the network have any bearing on the average number of days for responses by developers on defects? 2. Towards a social network based coordination approach The conceptual origin of social network analysis can be traced as far back as the 1930's when leading ‘Gestalt’ theorists Jacob Moreno devised the ‘sociogram’ to represent the formal properties of social configuration (Scott, 2000). Moreno's (1930) sociogram (as cited in Scott, 2000) depicted individuals using points and their social relationship with others by lines, he argued using this type of representation researchers could identify leaders(stars) and isolated individuals(isolates), to uncover asymmetry and reciprocity and to map chains of connection. In modern studies, researchers has taken used Moreno's (1930) suggestions and analysed networks based on either team structure. (Pearce & David, 1983) concluded there to be a definite relationship between ‘organisation design variables and group performance’ from their study of two types of organizational design models each with their own setup of stars, isolates and liaisons and levels of reciprocity. The framework of social network analysis was finally crystallized in the 1950's through the works of fieldworkers associated with the Department of Social Anthropology at Manchester University. Clyde Michell's can be attributed to turning the existing studies of graph theory used for sociometric analysis and formulating it to fit a specific sociological framework (Scott, 2000). John Barnes conducted intensive studies on the part kinship, friendship and neighbouring relationships played in the formation of communities. Barnes claimed that ‘the whole of social life’ could be set as ‘a set of points some which are joined by lines’ to form a ‘total network of relations’. The informal sphere of interpersonal relations can be seen as a part of a ‘partial network’, of this total network (Barnes, 1954). The final breakthrough that solidified the global properties of SNA (social Networks Analysis) in all fields of social life was made at Harvard by Harrison White and his associates. The first part was the development of algebraic models of groups using set theory to model relationships and the second was the development of multidimensional scaling, which was a technique for translating relationships into social distances and for mapping social space, which was the key idea behind Lewins work (Scott, 2000). The earliest example of combining theories of networks with coordination can be found in research by Alex Bavelas in the early 1950's. The experiment was devised so that individuals had to find other person who possesses the card with the same symbol as they did. The way of communication was through written messages and the group was setup in different formations of networks. The result of their research was that groups in the star or Y networks solved the problem the quickest, with fewest errors and least mistakes and those at the center of the star or Y network recorded the highest level of satisfaction (Kosfeld, 2003). de Souza et al. (2005) conducted on social network based investigation into distributed software development with its focus using artifacts and activities to determine the social structure of the software project (de Souza et al., 2005). With regards to examining distributed coordination, Sandusky and Gasser (2005) observed the impact of ‘negotiation’ as a mechanism of coordination in distributed software problem management systems (SWPM). The research was centered on publicly accessible bug repositories for OSS and negotiation in the SWPM process (Sandusky & Gasser, 2005). It is important to note that from the research it was found that OSS community bug repositories represent the major coordination mechanism in OSS projects. Finally, Gutwin, Penner et al. (2004) studied the coordination of distributed software development focusing on group awareness with the aim of unearthing how such geographically dispersed group stay aware of each other work and coordinates the development of large complex software projects. The result of the study outlined the importance of organizational culture and text based communication tools to maintain awareness (Gutwin, Penner et al., 2004). From the literature reviewed, it is evident that focus of the work has been conducted in the field of distributed software development of OSS has primarily been on prerelease coordination activities. For example, the focus of bug repositories study by Sandusky and Gasser (2005) was negotiation and not coordination measures. This leaves scope for investigating social network properties and their implications on coordination measures of OSS teams. 2.1. Structural variables of social networks for understanding distributed coordination of OSS There are certain variables of social networks that will be the focus of our analysis for the coordination in OSS projects. There are some general concepts and definitions that will be extensively used in the analysis sections that need to be firstly introduced. These definitions will help form the foundation of our social network based analysis of the OSS projects and their wider community. Social networks analysis is deeply rooted in graph theory, however it is important to distinguish that a graph in this context is ‘simply a set of lines connection points’ and graph theory consists of a ‘body of mathematical axioms and formulae that describe the properties of the patterns formed by the lines’ (Scott, 2000). • • • •
Two points are said to be adjacent if a line connects them Those points which are adjacent to a particular point is said to be the neighbourhood The total number of points within a point's neighbourhood is said to be its degree of connection Points can be connected directly by a single line or indirectly connected by a sequence of lines. The sequences of lines is said to be a walk • A walk in which each line and point is distinct is called a path • The length of a path is said to be the number of lines that make up the path • The distance between two points is the length of the shortest path between them.
54
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
In Fig. 1 we take A as our focus of analysis. A has the neighbourhood of B, C, D, E, F, G, H and the degree of connection of the entire neighbourhood for A equal 7. The length of the path of A to F can be simply 1 if we directly connect to it or it can be via B, C, D, and E which will give its length to be 5. However if we're calculating distance from A to F it will be 1 since it is the shortest length. 2.2. Density of the network In graph theory, density describes the general level of linkages among the points on a graph. When this is applied to the social network context each point of the graph represent individuals of the network and each line is a representation of interaction or communication between the two individuals. As describe in (Scott, 2000), density depends on two parameters; inclusiveness of the graph and the sum of the degrees of its points. Inclusiveness refers the total number of points in a graph minus the number of isolated points. An Isolated point is one which has no lines connected to it and contributes nothing to the density of the graph. The general formula for density is: l nðn − 1Þ = 2 where l is the number of lines present and n is number of points present. This measure uses the number of lines present in a graph compared to the number of lines theoretically possible. The number of points n representing actors involved in the network is the key factor here which has implications to the total number of lines possible. The formula for density can be applied in one of two ways, the ‘ego-centric’ or ‘socio-centric’ approach. The ego-centric approach concentrates on the density of the links surround a particular agent while the ‘socio-centric’ approach focuses on the pattern of connections in the network as a whole. For the purpose of this paper the ego-centric approach will be used because of the nature of OSS test is incredibly distributed in its participants. Studies by (Ye & Kishida, 2003) have shown that for OSS communities most participants are regarded as passive users and as for the Apache which was the focus of their study, the figure was up around the 99% proportion were either readers or passive users. 2.3. Centrality of the network The topic of centrality has been at center of many academic papers and empirical investigations and its origin is derived from the sociometric term of the ‘star’. Bavelas can be accredited as being the pioneer of the establishing the formal properties of centrality. In the late 1940's and early 1950's, Bavelas and Dermont (1951) and his colleges at Massachusetts Institute of Technology directed one of the first research studies into the applications of centrality. The research concluded that ‘centrality is related to group efficiency in problem-solving, perception of leadership and the personal satisfaction of participants’ (Freeman, 1979). The proliferation of the study of centrality on the context social network analysis has been proven to be far reaching. There are numerous measures that have been put forth, some of which include degree centrality, closeness, betweenness, and the rush index just to name a few. Borgatti argues that these measures often make certain assumptions on characteristics of the network. If these characteristics were not applied then we would lose the ability to fully
Fig. 1. Social network.
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
55
interpret these measures (Borgatti, 2005). Therefore in the following section the network characteristics that will be assumed for our analysis will be identified. 2.4. Local and global centrality A point is considered locally central if it has a large number of connections with other points within its neighbourhood. For this study we will be using a degree-based measure of local centrality which is simply computed by the number of points to which a point is adjacent. We can also extend this measure to include indirect connections and set a cut-off distant to which we measure its local centrality. For example if we set the cut-off distant at 2 that means a networks measure of local centrality will include the direct connection between two points A and H plus the connection between H and B (see Fig. 2 — Local and Global centrality (Scott, 2000)). A point is consider globally central when it has ‘a position of strategic significance in the overall structure of the network’ (Scott, 2000). Freeman proposed another measure based on what he called ‘closeness’ which identify the globally central points of the entire network. A point is said to be the globally central if it lies at shortest distance from many other points, therefore global centrality is calculated by the sum of the geodesic distances from one point to all other points within the network (Scott, 2000). From the definition we see that in Fig. 2 — Local and Global centrality (Scott, 2000) the global central of the network is point B because it has the lowest sum of geodesic distances from all other points in the network. Freeman (1977) adds further to the concept of centrality with what he terms as betweenness, which measures the extent to which a particular point lies between the various other points in the graph. Freeman explains that in terms of centrality it is not only the person with the most connections with the entire network that should be of interest, but also the person “in a group that is strategically located on the shortest communications path connecting pairs of others” (Freeman, 1977) shall also be referred to as a central actor. This person assumes responsibility of passing on information between the two pairs he is connected with, therefore the performance of a system depends on the coordination of activities from this central person. Freeman's approach centers about the idea of local dependency where a point is dependent on another if the paths which connect it to the other points pass through this point (Scott, 2000). What is of interest is that the role points like H and L play in the performance of OSS teams, whether their presence as intermediaries aids or hinders the average time bugs gets fixed. The model proposed here forms the basis of analysis conducted on the coordination of distributed open source software teams from a social network perspective (see Fig. 3 for Social Network based model for coordination). The model is based on some of the key structural variables as supported by the investigation into existing literature of social networks and coordination. From the measures of density, centrality and betweenness we can analysis the properties of the network for an OSS team for a particular
Fig. 2. Local and global centrality (Scott, 2000).
56
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
Fig. 3. Social network based model for coordination.
project. Then through examining the coordination measures of quality and timeliness, we can rank the project team in terms of their coordination performance. From these scores, we can then compare different OSS teams and form conclusions on whether each of the structural variables has influences on the coordination of distributed teams. However, there are mitigating factors which affect the results of the analysis. Factors such as defect severity and development team size needs to be kept in mind so that a basis for comparison can be formed to ensure the integrity of the results. 2.5. Density as a network variable affecting quality and timeliness Density as described earlier is a measure of the number of connections formed within a network as a proportion of total possible connections between all nodes of a network. Studies into density have been conducted such as the one by (Crowston & Howison, 2006), where they have found a negative correlation between the project size and density of the network. From their empirical work it would seem that the connections a for certain key participants remain the same while the project grows in size thus making the network less dense(Crowston & Howison, 2006). From a coordination perspective, research has been done by Dinh-Trong and Bieman (2004) into comparing the defect density and general performance measures between different projects and seeing if indeed OSS produces better quality software (Dinh-Trong & Bieman, 2004). Therefore it would be of interest to find the correlation, if any, between the network variable density and key performance indicators of OSS project. 2.6. Centrality as a network variable affecting quality and timeliness For the purpose of this study, the ego-centric approach to measuring centrality is used to investigate the correlation between levels of centrality within a network and the quality and timeliness of OSS project. Taking the definition that a highly centralized network is one that core developers are involved in high number of defect fixing, (Crowston & Howison, 2006) governed an empirical study into centralization in OSS team communications and found there to be larger projects with higher participants are lowly centralized. Other works contributes to the belief that OSS projects the core team consist of 10–15% of the total number of people make the majority contribution to the project (Ari, 2001). This again has implications on the centrality of the network and more to our interest the degree of centrality of key individuals and their affect on our defined coordination measure of quality and timeliness. 2.7. Betweenness as a network variable affecting quality and timeliness Betweenness, as described by (Scott, 2000) is the measure of the extent of which and actor can play the part of a ‘broker’ or ‘gatekeeper’ with a potential for control over others. This definition raises an interesting question, how influential is the role of a
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
57
broker or ‘liaisons’(Pearce & David, 1983) in coordinating bug fixing activities? Or does existence of too many liaisons within a network hinder the quality and timeliness of the OSS project? Studies based on betweenness by Newman (2005) have taken measures based on random information flows throughout a network and identifying first the most traveled paths and then those actors that are located along these paths (Newman, 2005). These actors signify high level of betweenness and centrality. Obviously this has connotations upon the coordination measures within our context of OSS bug fixing where the actions of those actors with high betweenness measure can potentially make or break the success of the project because they are those that control the follow of information between groups to which they are connected. Hypothesis. Quality of defect-management activities is influenced by social network measures. H1. Quality of code measure is influenced by degree centrality, node betweenness and density of the OSS team. H2(a). Defect density measure is influenced by degree centrality, node betweenness and density of the OSS team (high-severity). H2(b). Defect density measure is influenced by degree centrality, node betweenness and density of the OSS team (medium severity). H2(c). Defect density measure is influenced by degree centrality, node betweenness and density of the OSS team (low severity). Timeliness of defect-management activities is influenced by social network measure H3. Time-of-fix measure is influence by degree centrality, node betweenness and density of the OSS team. H4. Time-of-response measure is influence by degree centrality, node betweenness and density of the OSS teams. 3. Defect management process as a coordinated system Distributed Software Test teams are a recent trend that has emerged due to quality and ease of communication across vast physical locations and the boundless efforts of all corporations to reduce cost associated to software development. As a result entire projects are now divided so that teams in charge of different phases of the project can be located oceans apart. The relevance of coordination theory comes to the forefront when we analysis Distributed Software testing because apart from the multiple levels of communication from the tester to steering committee that must be coordinated, the actors must also contend with the physical distance between each team. By the using dependencies of coordination theory we can deduce theoretical models that would help achieve the goal of software testing more effectively. In order to provide some context on the coordinated system we are applying measures of coordination, it is only appropriate to outline the defect management process that distributed OSS teams must go through to ensure only legitimate and valid defects are accepted to be fixed. The Defect Management process is considered a coordinated system because it involves actors (developers and authors of the defect), performing interdependent tasks (raising and fixing defects) to achieve a common goal (advancing the OSS). How do we know what is the most effective application of each coordination model? To answer this question we need apply our understanding of coordination components universal to all Coordinated Systems and whether the different methods of applying these components affect result of existing metrics used to measure the effectiveness of software integration testing. Before we elaborate the measures of OSS testing and how these relate to coordination, we need to step back and justify the validity of such metrics in regards to measuring coordination. We have previously established that OSS testing can be categories as a Coordinated System. This system comprises of actors that perform interdependent tasks to achieve goals, the ultimate goal being to produce software to the community on a timely fashion, error free and satisfies the requirements specified. In order to achieve these goals, the actors working together and liaising with community members must manage the components of coordination such as resources allocation, producer/consumer relationship, simultaneity constraints and task dependencies. The measures of effectiveness include (i) quality and (ii) timeliness all of which are aligned with the ultimate goal of testing OSS projects. Quality measures have a direct correlation with the Producer/Consumer relationship component of Coordination theory. The core development team (producer) releases the software to the OSS community (consumer) then active or passive user logs defect reports when they arrive at an incorrect or unexpected behavior from the system. The cause of the defect invariably relates to the Producer/Consumer relationship, particularly the prerequisite constraints, transfer and usability aspects. To ensure that the software produced has a low quality of code metrics value and defect density metric value we must coordinate activities so that Prerequisite Constraints such as environment connectivity are correct. We must also ensure vital knowledge of the system is transferred from for example the development team to the test teams. The last element of Usability will naturally fall into place once the previous elements are managed accurately. Quality of Code metric captures the relation between the number of weighted defects and the size of the product release. WF NCSPT where WF is number of weighted defect found post release software release. The weight of each defect is dependent on the severity. NCSPT is the number of commit statements per time period. In this metrics the lower the number is, indicates the fewer defects or less serious defects found thus the higher is the quality of the code. Traditionally quality of code is measured using per
58
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
thousand lines of code, however this method of measure created efficiencies such as “bloaty” (Mockus, Fielding et al., 2002). With proprietary software the common practice in minimizes the effects of “bloaty” code, was to only counting every thousand lines of new code. Also the incremental delivery model of software development means that with each release only sections of new code is built on existing baseline of code from the previous release. By only taking the new lines of code into consideration a more accurate depiction of the quality of the code. In the spirit of avoiding “bloaty” code to affect the accuracy of the measure of quality this paper used NCSPT because with regards to OSS projects they are tracked according to commit statements which are logged and track on OSS project website. Each commit statement indicates patches and fixes to the software thus the proportionality between number of total defects raised and number of commit statement indicators if the quality of the code is high or low. Naturally high a number indicates that many defects have arisen for each commit statement which is a clear sign of poor quality of code. Defect Density metrics shows the relation between numbers of weighted defects per severity level and the total number of defects detected post release. WPSL ⁎100 WF where WPSL is number of weighted defects per severity level found during the post release period of the OSS and WF represents the total number of weighted defects found. Using this metrics we find that, higher the number shows higher ratio of defects were detected before release and higher effectiveness of the test. The motivating factor behind this measure is that we can arrive at a comparison based on number of defects of a certain severity level as a proportion of overall defect numbers. Naturally projects with high quality will have fewer defects within the high range of defect severity. The successful result of the following Timeliness Metrics can be associated the management of Simultaneity constraints and task dependencies. Imagine a typical software testing phase, with absolute certainty there dependencies on which activities can be performed first, in parallel and last. It is up those in the position to coordinate the activities to ensure that those tasks categorized as high priority because many other tasks depend on its success are run first. Those that can be run in parallel are organized in such a way and those many dependencies on other activities are run last. Although coordination theory is a very powerful method of analyzing areas of improvement for OSS testing, there still exist limitation and questions that must be answered. For example what type of affects does personal characteristics or properties of actors have on the outcome of the testing? Would group structure affect which activities that can be performed? In a distributed network of groups that must perform tasks that or interdependent, what type of links induce the best overall outcome for the software integration testing phases? A Social Network based approach will help complete this picture and offer a more round model when combined with our coordination model for improving the outcomes of software integration testing. Density as described earlier is a measure of the number of connections formed within a network as a proportion of total possible connections between all nodes of a network. Studies into density have been conducted such as the one by (Crowston and Howison (2006), where they have found a negative correlation between the project size and density of the network. From their empirical work it would seem that the connections a for certain key participants remain the same while the project grows in size thus making the network less dense(Crowston & Howison, 2006). From a coordination perspective, research has been done by Dinh-Trong and Bieman (2004) into comparing the defect density and general performance measures between different projects and seeing if indeed OSS produces better quality software (Dinh-Trong & Bieman, 2004). Therefore, it would be of interest to find the correlation, if any, between the network variable density and key performance indicators of OSS project. Therefore, in this paper, we only test the following two assumptions: H1. Quality of code measure is influenced by degree centrality, node betweenness and density of the OSS team. H2. Defect density measure is influenced by degree centrality, node betweenness and density of the OSS team (high-medium-low severity). 4. Data source Data was collected via the monthly data-dump SF.net provided to the Department of Computer Science and Engineering, University of Notre Dame for sole purpose of supporting academic and scholarly research on the Free/Open Source Software phenomenon. We were granted access to the university's specialist wiki-supported website dedicated to the study of Free/Libre/ Open Source Software projects. This site provided the SF.net ER-diagrams, database schema definition and SQL query functionalities require to extract the necessary data specified in previous section. The SQL query tool was a web-based program that allowed for simple SQL queries and timed out when more complex queries were required to be executed. Therefore, the list of projects used for analysis was chosen from the 100 most actively participated projects according to monthly download rate, forum activity and pages view. From this list of 100 only the projects with more than 7 developers and 200 bug reports were included. We felt that 7 developers provided a robust enough network for our studies and 200 bug reports represented enough instances of group interaction between the actors of the network. According to these criteria of selection a group of 45 projects were identified to be appropriate for use in our analysis into social network of OSS team. After the short list of projects that is to be included in the analysis was established we went about extracting the necessary data from appropriate tables and fields according the ER-Diagram and schema definitions provided by Notre Dame University. SQL
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
59
queries were executed to extract the necessary data and the results of each query were saved as a text file with ‘:’ (colons) used as separators for each field. Each text file contains all closed status bug reports for one of the 45 projects which fit the selection criteria. The type of refinement needed for coordination measures of timeliness included conversion of the time-stamped fields such as artifact.open_date, artifact.close_date and artifact_message.adddate, because SF.net uses UNIX formatted time-stamps for date fields. Unix time describes time as the number of seconds elapsed since midnight UTC of 1st January, 1970, therefore the following formula was applied in Excel to convert the time-stamp into a more meaningful manner. UNIXtime + ð365⁎70 + 19Þ⁎86400 − 0:41667 86400 Once the time format has been changed we used existing Excel functions to calculate the duration for each closed was fixed according the artifact.open_date and artifact.close_date then summing the total of the durations and dividing the by the total number of bugs we have a coordination measure of Time-to-fix-Bugs metrics for each project. A similar approach was used to calculate the Time-to-Respond metrics but instead the artifact_message.adddate was subtracted from the artifact.open_date with the average time similarly calculated by totaling the duration of each response and dividing by the total number of artifact_message. For the measures of quality, defect density required aggregation of the different categories of the aftifact.priority field. Currently SF.net's tracker system uses a 9-level priority system to categorize the severity of each bug, with 9 being the most severe and 1 being the least. This system presents ambiguity when used in our analysis of the defect density when defining the difference between a priority 5 bug and priority 4 or 6 bug. Thus to clarify the severity of the bugs raised for each project we have concluded that due to the number of bugs found in each of the 9 priority levels, 1–3 priority bugs are deemed “low-severity”, 4–6 priority bugs referred to as “moderate severity” and finally 7–9 priority bugs classified as “high-priority”. For the final measure of quality of code, total number of bugs raised and commit statement executed for each project were extracted from the Project summary page. 5. Results and discussions Using the multiple regression model of analysis we arrived at a set of results which we can use to interpret the relationship between dependent variables of coordination and independent variables of social network characteristics. This section reports on the findings from this analysis conducted. We analyse the regression output of the social network independent variables (centrality, betweenness and density) regressed on both measures of quality of coordination respectively. From our analysis of the relationship between the quality of code measure and degree centrality, betweenness and density Excel processed the following output, and regression model (Table 1): Y ¼ 2:593 þ 11:361⁎Centrality þ 9:136⁎Betweenness−3:862⁎Density We can only accept the hypothesis if for F-test we obtain an F-statistic greater than 2.61 (critical value for F at 3.41 degrees of freedom), additional to this the significance-F value that is less than 0.05 which is the predetermined level of significance. In term of the significances of the independent variables, each t-statistic needs be greater than 1.6829 or less than − 1.6829 (critical value for a two-tailed t-test at 41 degrees of freedom) and p-values less than 0.05. If these conditions are met then we can conclude that there is sufficient evidence of a relationship between quality of code and centrality, betweenness and density. Interpreting the output we accepted hypothesis because the F value of 16.428 which is greater than the critical value of 2.61 for a 5% level of significance. Furthermore, we concluded that each independent variable has strong significance to the model with either respective p-values all
Table 1 Quality of code output. Regression statistics Multiple R R2 Adjusted R2 Standard error Observation
Regression Residual Total
Centrality Betweennes Density
0.788236 0.62131599 0.59909888 13.5134966 45 df
SS
MS
F
Significance F
3 41 44
8999.8732 7487.19761 7735.61699
2999.958 182.6146
16.42781
0.0062626
Coefficients
Standard Error
t statistics
p-value
11.3614902 15.1359258 0.86176736
6.25387107 6.39409508 0.35632672
1.816713 2.367172 2.41848
9.36E− 16 2.323E− 12 0.000032
60
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
being greater than 1.6829 which is the critical values for t-distribution test at 5% level of significance. Therefore each of the βi's contribute to the model. The moderate Adjusted R2 suggest that the model as a whole has average explanatory power and around 60% of the variations in observed values for quality of code can be explained by the variations of values of centrality, betweenness and density. From the strong p-values of the independent variables we can conclude the following: (i) for each percentage increase of centrality, on average it will increase the quality of code measure by 11.361 units, holding all other variables constant; (ii) for each percentage increase of betweenness, on average it will increase the quality of code measure by 15.136 units, holding all other variables constant; and, (iii) for each percentage increase of density, on average it will decrease the quality of code measure by 3.862 units, holding all other variables constant. What these conclusions illustrate is that while centrality and betweenness of the network have a positive correlation with quality performance of OSS teams, interestingly density has a negative correlation. This result suggests that increasing the level of centrality and betweenness of an OSS team will increase the number of bugs fixed per commit statement. However, increasing the communication links between actors will actually decrease this measure of quality in the coordination of defect management activities. For the measure of defect density three separate regression analysis was conducted according three levels of defect that was classified for each OSS project. This is to take into the account of the mitigating factor of defect severity levels. As also previously discussed when a defect is recognized as valid it is assigned in severity or priority level which is an indication of the affect this defect has on the project. Naturally higher severity defect demand more attention because it affects the behavior of the software to a greater extent. For this reason we decided to conduct separate regression analysis on the three levels of defect severity to see what relationship exists between that and centrality, betweenness and density. The regression analysis produced the following output and model: Y ¼ 2:593 þ 11:361⁎Centrality þ 9:136⁎Betweenness−3:862⁎Density Here, we can only accept the hypothesis if for an F-test we obtain an F-static greater than 2.61 (critical value for F at 3, 41 degrees of freedom), additional to this the significance-F value that is less than 0.05 which is the predetermined level of significance. In term of the significances of the independent variables, each t-stat needs be greater than 1.6829 or less than −1.6829 (critical value for a two-tailed t-test at 41 degrees of freedom) and p-values less than 0.05. If these conditions are met then we can conclude that there is sufficient evidence of a relationship between quality of code and centrality, betweenness and density. Interpreting the output we can only tentatively accept the H2(a) because whilst the F value of 4.310 is greater than the critical value of 2.61 for a 5% level of significance. However, each independent variable has shows poor insignificance to the model with their respective p-values all being greater than − 1.6829 and less than 1.6829 which is the critical values for t-distribution test at 5% level of significance. Therefore, each of the βi's does not contribute to the model. Furthermore the low Adjusted R2 suggest that the model as a wholes has poor explanatory power and only around 30% of the variations in observed values for defect density for high severity defects can be explained by the variations of values of centrality, betweenness and density. The above results suggested that we do further analysis into the model and the decision was taken to included an interaction term into the model. The adjusted model included the introduction of centrality ⁎ density and betweenness ⁎ density. These two interaction terms were chosen because networks with high degrees of density tend to have lower degrees of centrality and betweenness, this is due to the increase connections between actors thus reducing the centralization measure of individual actors. Intuitively this assumption can be made because as the ties amongst actors increases there is less need for actors to communicate through other actors instead they will more likely to contact the source of the knowledge consequently reducing the need to central or between actors (Table 2). Table 2 High severity defect output. Regression statistics Multiple R R2 Adjusted R2 Standard error Observation
Regression Residual Total
Centrality Betweennes Density
0.4828337 0.2331284 0.1998723 0.0008997 45 df
SS
MS
F
Significance F
3 41 44
0.22459875 0.80244947 1.02704823
0.07486625 0.01957193
3.82518
0.01665228
Coefficients
Standard error
t statistics
p-value
− 0.077590007 − 0.096314342 − 0.008513927
0.08544898 0.08690062 0.0758123
− 0.9080273 − 1.1083269 − 0.1123030
0.176248 0.322982 0.017061
L. Hossain, D. Zhu / Journal of High Technology Management Research 20 (2009) 52–61
61
6. Conclusions This study suggests that there is a correlation between social network characteristics and strong and poor performing projects in an OSS environment. The projects which were analysed using our theoretical model all fitted the criteria of more than 7 developers and more 200 defect reports. The conditions we have set on the data help validate the strength of the results because such criterions provide a robust network of interaction between its actors also it facilitates varying sized networks. Analysis of the data displayed a normal distribution which fitted our proposed parameterized, regression analysis. We believe that we have made a significant contribution into the study of distributed work groups, social network analysis and coordination. Studies by Madey, Freech et al. (2002) have investigated the social network phenomenon, power-law relationship(Madey, Freeh et al., 2002), in OSS development teams and suggested to look into degrees of separation of the connection which we have analysed through density and centrality. References Ari, I. (2001). Quantitative Analysis of Open Source Software Projects. [Retrieved on 11/7/07]. Barnes, J. A. (1954). "Human Relations." Class and Committee in a Norwegian Island Parish 7. [Retrieved on 12/7/07]. Bavelas, A. (1950). Communication Patterns in Task-Oriented Groups. Journal of the Acoustical Society of America, 22(6), 725−730. Bavelas, A., & Dermont, B. (1951). An Experimental Approach to Organizational Communication. Personnel Psychology, 27. Borgatti, S. (2005). Centrality and network flows. Social Networks, 27, 55−71 [Retrieved on 9/7/07]. Crowston, K., & Howison, J. (2006). Hierarchy and centralization in free and open source software team communication. Knowledge, Techology and Policy, 18(4), 65−85 [Retrieved on 9/7/07]. Dinh-Trong, T., & Bieman, J. M. (2004). Open source software development: a case study of FreeBSD. 10th International Symposium on software metrics (pp. 96−105). [Retrieved on 12/7/07]. Freeman, L. C. (1977). A set of measures of centrality based upon betweeness. Sociometry, 40, 35−41 [Retrieved on 9/7/07]. Freeman, L. C. (1979). Centrality in social networks: conceptual clarification. Social Networks, 1, 215−239 [Retrieved on 1/8/07]. Gutwin, C., Penner, R., & Schneider, K. (2004). Group awareness in distributed software development. Computer supported cooperative work, 72−81. Kosfeld, M. (2003). Network experiments. Zurich IEER Working Papers (pp. 152). [Retrieved on 13/7/07]. Madey, G., Freeh, V., Tynan, R. (2002). The open source software development phenomenon: An analysis based on social network theory. Eighth Americas Conference on Information Systems (pp. 1806–1813). Mockus, A., Fielding, R. T., & Herbsleb, J. D. (2002). Two case studies of open source software development: Apache and Mozilla. ACM Transactions on Software Engineering and Methodology, 11(3), 309−346. Newman, M. E. J. (2005). A measure of betweenness centrality based on random walks. Social Networks, 27(1), 39−54 [Retrieved on 14/7/07]. Pearce, J. A., & David, F. R. (1983). A social network approach to organisational design-performance. The Academy of Management Review, 8(3), 436−444 [Retrieved on 7/7/07]. Sandusky, R. J., & Gasser, L. (2005). Negotiation and the coordination of information and activity in distributed software problem management. Conference on Supporting Group Work (pp. 187−196). [Retrieved on 17/7/07]. Scott, J. (2000). Social Network Analysis: A handbook. [Retrieved on 9/7/07]. de Souza, C.R.B., Froehlich, J., & Dourish, P. (2005) Seeking the Source: Software Source Code as a Social and Technical Artifact. ACM International Conference on Supporting Group Work (GROUP 2005) (pp: 197–206). Ye, Y., & Kishida, K. (2003). Towards an understanding of the motivation open source software developers. International Conference on Software Engineering (pp. 419−429). [Retrieved on 8/7/07].