- Email: [email protected]
Measuring social networks when forming information system project teams
Accepted Manuscript
Measuring social networks when forming information system project teams Roberto Latorre, Javier Suarez ´ PII: DOI: Reference:
S0164-1212(17)30208-X 10.1016/j.jss.2017.09.019 JSS 10041
To appear in:
The Journal of Systems & Software
Received date: Revised date: Accepted date:
7 July 2016 4 July 2017 21 September 2017
Please cite this article as: Roberto Latorre, Javier Suarez, Measuring social networks when ´ forming information system project teams, The Journal of Systems & Software (2017), doi: 10.1016/j.jss.2017.09.019
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Highlights • Socio-technical framework for team assignments based on social-network
CR IP T
analysis • Social networks estimating compatibility according to previous collaborations
• Social networks estimating compatibility according to social skills
• Industrial insights into the applicability and usability in real-life case-
AC
CE
PT
ED
M
AN US
studies
1
ACCEPTED MANUSCRIPT
Measuring social networks when forming information system project teams
a Escuela
CR IP T
Roberto Latorrea,∗, Javier Su´ arezb Polit´ ecnica Superior, Universidad Aut´ onoma de Madrid, 28049, Madrid, Spain b Kymatic Soluciones Inform´ aticas, Madrid, Spain
Abstract
AN US
Despite the advances in the project management field, little is known about how creating performing software teams in a systematical and repeatable way.
The technical dimension is not enough to achieve this. Software development is a complex collaborative process where people and interpersonal relationships – i.e., how people interact, behave and organize significantly – influence the project success. We define a framework to assign people to projects from this
M
socio-technical perspective. Social networks characterizing the interplay between teammates are built and analyzed to predict productive collaborations
ED
and identify adequate team-members depending on the needs of the organization and the kind of project. A noteworthy novelty of these social networks is that they estimate compatibility between coworkers according to previous
PT
collaborations, but also to individuals’ social skills. This allows analyzing the compatibility between people who have not worked together before. We present results of using the proposed framework in a multinational corporation during
CE
a more-than-two-year period. Our in-company experiments emphasize that we can significantly improve the expected outcomes characterizing and measuring
AC
the social interaction between coworkers. Social aspects discussed here may be highly relevant in the context of distributed software engineering, since it implies new challenges in the interplay among coworkers.. ∗ Corresponding
author Email address: [email protected] (Roberto Latorre)
Preprint submitted to Journal of Sytems and Software
September 22, 2017
ACCEPTED MANUSCRIPT
Keywords: Team member allocation, Team formation problem, Social skills, Social network analysis, Interpersonal relationships, Collaboration network
CR IP T
1. Introduction
Information System (IS) projects in the industrial environment are commonly considered to be successfully completed when they are delivered on time,
on budget and with the appropriate level of quality. With this definition in 5
mind, multiple factors can influence the temporal, financial and quality goals
AN US
of industrial IS projects during their lifetime (Charette, 2005). For instance,
underestimated projects commonly fail: the ones delivered with the appropriate level of quality, are often completed late and/or with over cost; while the released to meet the budget usually do not meet system requirements and are 10
unreliable systems. Another fundamental factor to complete the project successfully is an appropriate team composition (Fenton and Neil, 1999; Linberg,
M
1999; El Emam and Koru, 2008). The decision of who participates in the project has therefore a critical impact on success. Nevertheless, experience shows that, to improve the expected outcomes, the project team usually takes a backseat at the planning stage and we often focus the attention on other factors, such
ED
15
as determining the project tasks and workload, and optimizing the time-line
PT
scheduling for developing the system. Assuming the existence of such mechanisms to control these other factors, this paper addresses the team formation problem in the context of real-life industrial IS projects. Forming effective and efficient working teams is a complicated endeavor be-
CE
20
cause multiple factors influence the effective performance and productivity of a
AC
group of people working together. Traditional methods for selecting people to perform a given task are based on the candidates’ estimated performance and/or the best fitting among the individuals’ abilities and the set of abilities required
25
for the task to perform (Lievens et al., 2002). In the context of software engineering, the need of highly technical specialized skills for developing software has been subject of discussion for a long time. Thus, human resources (HR) allo-
3
ACCEPTED MANUSCRIPT
cation in software industry is mainly driven by the candidates’ know-what and know-how, i.e., technical knowledge or expertise acquired through experience 30
or formal training. For instance, these are the basis of traditional role-based
CR IP T
staffing strategies (Bandinelli et al., 1996; Zhu et al., 2006) that individually assign people to development roles with the goal of identifying the candidates with the necessary technical skills to complete each development activity with
the minimum training time and/or cost (Tsai et al., 2003; Barreto et al., 2008; 35
Otero et al., 2009; Ngo-The and Ruhe, 2009; Chen and Zhang, 2013). However, a team is more than the sum of the individual team members’ attributes. Different
AN US
individuals have different working styles and preferences, and simply bringing
together the best candidates as a function of their estimated performance or technical abilities does not ensure creating a high-performing team (Fitzpatrick 40
and Askin, 2005; Acu˜ na et al., 2008; Adams and Anantatmula, 2010; Feldt et al., 2010). Software development is an intense collaborative process (Sawyer, 2004; Luna-Reyes et al., 2005) where cooperation and social interactions among team-
M
mates (Lakhanpal, 1993; Sawyer, 2001; Gorla and Lam, 2004), or even among people belonging to different working teams (Oh et al., 2006), can significantly affect to project success. From this socio-technical perspective, an important
ED
45
step to enhance productivity and quality outcomes of IS project teams is to be able to identify compatible coworkers to improve social interactions within the
PT
team and avoid, or at least minimize, conflicts. A commonly accepted view in psychology is that the personality of an individual can be described by sets of personality traits, i.e., fixed sets of patterns
CE
50
defining the way people tend to think, feel and behave in different situations. This concept is extensively used in the HR management domain (Robertson
AC
and Smith, 2001) trying to estimate the level of development of an individual in different generic (skills or attributes of the personal behavior that can be
55
considered as a behavioral feature) and technical competencies (particular skills specifically related to a specific job or task, e.g., business analysis or project management). Software engineering research efforts in this line are directed toward identifying the required features for each development task or role (Acu˜ na 4
ACCEPTED MANUSCRIPT
and Juristo, 2004; Balthazard et al., 2004; Acu˜ na et al., 2006; Licorish and 60
MacDonell, 2015) and finding the best matching among people and these features (Drigas et al., 2004; Otero et al., 2009; Otero and Otero, 2012). Related to
CR IP T
our work, some authors claim that some competencies can explain and predict how a person interact and collaborate with other people in such a way that
variations of these behavioral and social traits can significantly influence the 65
dynamics and effective productivity of a development team (Blaskovich, 2008;
Adams and Anantatmula, 2010; Feldt et al., 2010). Under this view, it has been proposed that creating productive software development teams requires an ade-
traits (Turley and Bieman, 1995).
AN US
quate balance across team members’ technical skills, experience and personality
In spite of the increasing interest on non-technical and social aspects for the
70
research community, in practice, HR allocation techniques in software industry usually only respond to who does what. Assignments are typically made to tasks or roles individually and issues related to how individuals are assembled to form
75
M
the team are often disregarded. We propose here a general socio-technical framework to form teams globally. The motivation behind this framework is finding
ED
an optimal trade-off between technical and social aspects in order to have a successful project. The goal is to form a team of experts with the required technical knowledge, but also collaborating as a team in an effective manner. Our
80
PT
proposal benefits of some of the socio-technical aspects previously discussed in the research literature. During the evaluation of the technical dimension, we
CE
look for an optimal fitting among the technical needs and the candidates’ skills. During the evaluation of the social dimension, the aim is to build and analyze networks of software professionals characterizing and measuring their interper-
AC
sonal relationships. These social networks allow us to study the compatibility
85
among people who are involved in the project and identify those individuals that, working together, may potentially have an optimal joint-performance to complete the project. A relevant novelty of the proposed framework is that it does not only measure networks connecting software professionals, but also connecting the social skills they use to communicate, persuade or interact with other 5
ACCEPTED MANUSCRIPT
90
people (e.g., abilities such as verbal and non-verbal communication, empathy or leadership). Results obtained in multiple in-company experiments suggest that characterizing and measuring the interplay between coworkers when forming
CR IP T
software project teams can significantly improve the expected outcomes. We would like to emphasize that the proposed framework is general in the 95
sense that it can be apply to different kind of companies. The framework defines rules to evaluate and align relevant technical and social aspects affecting perfor-
mance of a software development team, but the information required to build
and measure the proposed social networks (e.g., project performance estimators
100
AN US
or level of knowledge/expertise in a given skill) can be obtained from different sources and/or be estimated using different metrics. This allows adapting the
framework to companies with different organizational structures and processes: from small companies to multinational corporations.
M
2. Related work
2.1. Socio-technical approaches to software development Interpersonal relationships are associations among two or more people formed
ED
105
in the context of a social interaction, e.g., friendship, affinity or communications. In this work, we are interested in collaborations between software professionals
PT
within a development team. Multiple types of personal and professional interpersonal relationships arise between coworkers in an IS project. Over the last 110
years, there is an increasing interest of these relationships on development tasks
CE
and literature on software engineering recognizes the importance of social aspects in the software development process (e.g., see Tamburri et al. (2013)). A
AC
non-trivial IS project can be considered a socio-technical endeavor where both technical and social factors influence the development process (Sawyer, 2004;
115
Luna-Reyes et al., 2005; Herbsleb, 2007). The technical component consists of the technical properties of the software to be developed, the development techniques or the technology employed. The social component includes the organization and the individuals involved in the development process, their attitudes
6
ACCEPTED MANUSCRIPT
and behaviors. Some authors claim that socio-technical congruence is a good in120
dicator of organizations performance in carrying out IS projects (Cataldo et al., 2006, 2008). Other investigations suggest that interpersonal relationships be-
CR IP T
tween coworkers determine the likelihood of development teams to be cohesive and productive (Yang and Tang, 2004; Adams and Anantatmula, 2010). 2.2. Social network analysis and the staffing problem
Socio-technical approaches to software development demand to assess the
125
relationships among coworkers in order to find compatible teammates during
AN US
the personnel assignment process. However, when putting together a team, it is difficult to determine who can efficiently work with whom. Compatibility can be given by technical (e.g., compatible programmers use a similar programming 130
style and naming strategy) and/or social aspects (e.g., fluidly communications and knowledge sharing between teammates).
Interpersonal relationships can be represented and analyzed with a social
M
network (Moreno, 1934), i.e., a graph where each node corresponds to an individual and links represent relationships between them. Social Network Analysis (SNA) has been applied in multiple scenarios from different scientific areas (Al-
ED
135
bert and Barab´ asi, 2002). In the context of software engineering, it emerges as a tool for capturing the underlying interaction processes within develop-
PT
ment teams and studying different aspects of the collaboration and organization among software professionals. Thus, SNA has been used, for instance, to iden140
tify top developers and their attitudes and knowledge sharing behaviors (Sowe
CE
et al., 2008; Licorish and MacDonell, 2014, 2015) or to predict failures and vulnerabilities in software products (Meneely et al., 2008; Wolf et al., 2009; Shin
AC
et al., 2011). SNA is also extensively used to analyze the Open Source Software development process (Crowston and Scozzi, 2002; Madey et al., 2002; Duche-
145
neaut, 2005; Bagozzi and Dholakia, 2006). More related to our work is the investigation by Lappas et al. Lappas et al.
(2009), who apply SNA to the problem of assigning people to a working task. Given a pool of candidates with different skills, a social network that captures 7
ACCEPTED MANUSCRIPT
the compatibility among these individuals and a particular task that requires 150
a set of skills, Lappas et al. (2009) addresses the problem of identifying the individuals best suited to perform the given task. Extensions of this algorithm
CR IP T
have been proposed later, for example, to be applied when a skill requires more than one expert (Li and Shan, 2010).
All these works provide a useful theoretical framework successfully tested 155
with scientific-collaboration graphs (Newman, 2001). Nevertheless, when you try to apply this framework to real IS projects in industry, you find some lim-
itations. The main issue is that Lappas et al. look for groups of individuals
AN US
that “together” have the required skills regardless of who displays each skill.
This is not a real scenario in industry, where projects demand individuals with 160
a specific set of skills. Furthermore, Lappas et al. focus on the network algorithms and do not address the construction of the social network capturing the compatibility among coworkers. In their tests, they build scientific-collaboration networks from the DBLP bibliography database where two scientists have a pre-
165
M
vious successful collaboration when they appear in the database as coauthors of at least one manuscript. In the following section, we review different common
ED
approaches to build social networks of software professionals. 2.3. Strategies to build social networks in software engineering
PT
Literature describes three main strategies to build networks representing social relationships among software professionals. The most common strategy is 170
based on the correlation observed between work performance and effective com-
CE
munications between coworkers (e.g., see Yang and Tang (2004)). Coworkers in an IS project can interact verbally and/or textually through different chan-
AC
nels. Investigations that use communications to identify relationships among coworkers found several problems in tracing and characterizing verbal commu-
175
nications, even when they were computer-mediated. Then, most of these works build collaboration networks using different heuristics to mine knowledge from mailing lists (Ducheneaut, 2005; Bird et al., 2006; Abreu and Premraj, 2009), bug tracking tools (Bhattacharya and Neamtiu, 2010; Bhattacharya et al., 2012) 8
ACCEPTED MANUSCRIPT
or version control systems (Bird et al., 2006; Schuler and Zimmermann, 2008; 180
Meneely and Williams, 2011; Bettenburg and Hassan, 2012), between others. Thus, they establish a connection between individuals that interchange a given
CR IP T
number of emails, commit the same file in a software repository or comment the same bug in a bug tracking system. Some studies show that relevant links
can be missing from the data in the social networks constructed with these 185
strategies (Bachmann et al., 2010).
A second strategy consists of using the hierarchical structure of the company
to identify the edges of the social network. This strategy has been used to
AN US
study how organizational structures interfere with task assignments and their influence on communications (Damian et al., 2013) and the resulting software 190
quality (Nagappan et al., 2008).
A last strategy consists of building social networks connecting participants in the same project (Madey et al., 2002; Xu et al., 2006). Note that these networks are equivalent to the scientific-collaboration networks mentioned above.
195
M
An interesting work where this strategy is used to find compatible programmers is Surian et al. (2011). In this work, they build collaboration networks con-
ED
taining three types of nodes – developers, projects and project properties – and define a metric to quantify the compatibility among developers as a function of the projects they have worked together before and their properties. The idea
200
PT
inspiring this metric is that two teammates working in the same project have common interests and skills. Surian et al. report “reasonable” results based on
CE
an input developer and snapshots from Sourceforge.Net. However, they do not address the construction of teams composed of people with different skills and
AC
roles.
205
These three strategies have been successfully applied to different relevant
problems in software engineering. Nevertheless, as we mention above, from our perspective, they have some limitations to be applied to the team allocation problem in industry. In addition to these limitations, we consider crucial to establish how positive or negative the interactions among coworkers are, and none of them does it. For instance, non-productive teammates can interchange 9
ACCEPTED MANUSCRIPT
210
a great amount of emails or systematically participate in failing projects. Social networks built in the context of the team formation problem must capture this information to avoid including individuals with negative relationships in future
CR IP T
working teams. To solve the drawbacks raised, we use a different approach. As Surian et al. (2011), we consider that the most reliable source for informa215
tion about compatibility between coworkers is their participation in previous projects, but we introduce a joint-performance estimator weighting each con-
nection in the social network (see Section 3.2 for details). Barreto et al. (2008) defines a similar “productivity modifier” characterizing the productivity of an
220
AN US
individual. The difference between this productivity score and ours is that Bar-
reto et al. assign the scores to individuals – nodes in terms of the social network – and we assign them to interpersonal relationships – i.e., connections. Moreover, we introduce an additional social score to analyze the collective compatibility of a group of software professionals, not only the compatibility between two
225
M
particular individuals.
3. Framework definition
ED
Our work departs from the hypothesis that the expected outcomes of new IS projects can be significantly improved if the project team is not only selected as a
PT
function of technical capabilities, but also considering interpersonal relationships within the team. Under this view, the goal of the proposed socio-technical 230
framework is maximizing the success probability of new teams finding an optimal
CE
trade-off between technical and social aspects. To achieve this goal, we build different compatibility social networks for employees and new candidates. The proposed framework is generic in the sense that it can be adapted to the needs
AC
of different type of companies. In particular, different strategies and metrics
235
can be used to construct and measure the different social networks proposed. Let assume a forming-storming-norming-performing team life cycle (Tuck-
man and Jensen, 1977), where a new team advances from the forming to the performing stage to become an effective team. The simple and intuitive idea behind
10
ED
M
AN US
CR IP T
ACCEPTED MANUSCRIPT
PT
Figure 1: Schematic representation of the proposed socio-technical framework.
our proposal is putting together compatible teammates with the required technical skill in order to facilitate this transition and minimize the storming stage
CE
240
impact. By compatible, here we mean people that working together reach a high joint-performance. To predict the most adequate project team composition, the
AC
framework analyzes multiple variables. Depending on the company, this includes different project metrics, the employees’ technical and non-technical capabili-
245
ties, and/or the social interactions within the team, between others. To briefly provide an intuitive view of this analysis, Fig. 1 illustrates schematically the proposed framework:
11
ACCEPTED MANUSCRIPT
1. First, we identify the candidates best suited to participate in the project (Section 3.1.1). When possible, these will be the members of the different teams to evaluate and the nodes of our social networks. Suitability is
250
CR IP T
determined through the matching between the attitudes required for each task or role (Pi , identified with a different color in Fig. 1) and the available human resources’ attitudes.
2. Connections in the social networks are established as a function of previous collaborations, i.e., projects where the candidates worked together before.
255
Then, we extract the information of similar projects from a database of
AN US
completed projects (Section 3.1.2).
3. Once potential candidates and projects to rate their previous collaborations have been found, we build two kind of compatibility networks. On one hand, social networks characterizing the interplay between team-
260
mates in terms of previous productive and non-productive relationships (Section 3.2.1). On the other hand, social networks connecting candidates
M
with compatible and incompatible social skills (Section 3.2.2). Individuals in the social-skill networks can be connected even when they have not worked together before.
ED
265
4. Finally, the potential candidates are combined using different strategies (Section 3.3), and the resulting teams are analyzed with SNA to estimate
PT
the most effective assignments for new projects or to include new members in existing teams (Section 3.4).
As we mention above, this general schema can be adapted to the specific
CE
270
context of the company where the framework is applied. In the following sections, we describe possible strategies to construct and measure our teammate
AC
and social-skill networks. 3.1. Identification of nodes and links
275
3.1.1. Find potential candidates Nodes of our social networks are the potential candidates to participate in the project. Like in any other strategy for assigning people to a working task, 12
ACCEPTED MANUSCRIPT
potential candidates are identified as a function of the fitting among the candidate’s abilities and the set of required abilities. To group these abilities, we use 280
the concept of profile – defined as the set of specific attitudes for each task or
CR IP T
role in the project. A profile can include different type of attributes: technical and non-technical skills, personality traits, limiting individual factors (e.g., trav-
eling availability) and personal preferences. Note that this requires identifying
and grading every candidate’s technical and non-technical skills and personality 285
traits (see threats to validity in Section 6). While some existing decision support
systems for assigning people to IS project teams already incorporate the candi-
AN US
dates’ technical and non-technical skills (Andr´e et al., 2011; e Silva and Costa,
2013), considering limiting individual factors and personal preferences is a novelty of our proposal. The need of taking into account limiting individual fac290
tors is obvious for real-life industrial projects. Regarding personal preferences, experience indicates that when an employee is continuously assigned to tasks he/she does not like to perform and is qualified to perform a higher-level job,
M
at the end, a decreasing performance likely takes place. Thus, a possible sample of web developer’s profile is Pdeveloper = {java, TDD, german competencies, stress tolerance, not traveling availability, distributed project}.
ED
295
In the simpler case, profiles only include technical skills, since this is the minimal information to identify those candidates with the required technical
PT
knowledge. The inclusion of other elements is not mandatory and depends on the company and the information it manages. To match the required profiles and the available human resources’ profiles,
CE
300
these are considered fuzzy sets in order to manage uncompleted associations between the required and the actual level of knowledge/expertise in a given skill
AC
or capability (Drigas et al., 2004). When there is not a perfect match between profiles and candidates, we use the best-fitted resource (BFR) methodology
305
proposed in Otero et al. (2009). BFR analysis quantifies how prior knowledge in other skills contributes to the learning of a given skill. This analysis allows us to build effectively teams even when the optimal profiles are not completely available. 13
ACCEPTED MANUSCRIPT
3.1.2. Find similar projects We measure networks built considering the projects where individuals have
310
worked before. Multiple variables make an IS project to be unique. Therefore,
CR IP T
the generation of our social networks is driven only by data of similar projects to the one for which the new team is being formed. Similarity between projects is
estimated using a clustering algorithm. These algorithms allow us to compute 315
and compare similarity vectors between projects. In our implementations we perform a k-means clustering analysis (MacQueen, 1967; Xu and Wunsch, 2005). The identification of similar projects extensively depends on the data stored
AN US
in the corresponding database of projects (see threats to validity in Section 6).
Results reported in this paper consider vectors with the following features to 320
determine similarity among projects: type of project, area of knowledge, working methodology, estimated workload, complexity, risk level, team size, budget and geographical distribution. We would like to highlight the latter, since geographical distribution usually is disregarded during the project team compo-
involves teams split over locations and countries.
ED
325
M
sition (beyond language competencies). However, current software production
3.2. Building compatibility networks 3.2.1. Teammate networks
PT
A teammate network – e.g., see Fig. 2 – is social network that consists of nodes representing individuals (IT M = {I1 , ..., In }) and links connecting them and quantifying their expected compatibility with a compatibility score (RT M =
CE
330
{CT MIi Ij }). The more compatible a teammate-teammate relationship in the network, the higher the estimated joint-performance of the corresponding indi-
AC
viduals. Links between teammates are bidirectional and CT MIi Ij = CT MIj Ii . If two individuals are not connected in the network, we have no evidence re-
335
garding their compatibility. This is different from being incompatible. Although the previous definition is conceptually simple, it is not easy to
determine a priori the compatibility among coworkers. Similar approaches to ours described in the related work assume that two individuals are compatible 14
AN US
CR IP T
ACCEPTED MANUSCRIPT
Figure 2: Example of teammate network. Scores in the links indicate how compatible the corresponding interpersonal relationship is. For simplicity in the graphical representation,
M
positive relationships are plotted in green color; and negative in red.
and have a successful collaboration simply if they previously worked in the same project. From our perspective, this and other similar heuristics do not capture
ED
340
the compatibility among coworkers in a real-life scenario, because they do not measure whether interpersonal relationship were actually positive. To solve
PT
these drawbacks, we rate connections in our teammate networks using two types of metrics: subjective metrics based on the teammates’ personal opinion about their interactions, and objective metrics based on the collective performance
CE
345
of previous teams. Thus, subjective teammate networks are constructed as a function of a subjective metric; and objective teammate networks as a function
AC
of an objective metric.
350
We would like to emphasize again that the proposed framework is defined
to be applicable in different contexts. In this regard, the compatibility score of a teammate-teammate relationship can be given by different subjective or objective metrics depending on the particular company. It can be expressed
15
ACCEPTED MANUSCRIPT
qualitatively or quantitatively, and obviously, the possible values and the interpretation of these values vary depending on the strategy and metrics used 355
to build the corresponding teammate network. The only relevant point is the
CR IP T
order relation between the possible values. In the case of quantitative data, we assume that the higher value corresponds to the more compatible relationships.
If this is not the case, it can be easily changed. To simplify the analysis, qualitative data are converted into numerical values maintaining the order relation. 360
For example, in a “positive-negative-neutral” Likert scale, we can assign 1 to “negative”, 2 to “neutral” and 3 to “positive”.
AN US
Subjective teammate networks are based on the idea that only teammates
know how they interact during daily work. Then, connections in these networks are rated according to the assessment the teammates themselves supply. 365
Depending on the context of the company, different strategies can be used to achieve this. The HR management domain provides us with different tools to evaluate interactions among coworkers. In particular, we highlight here the so-
M
called 360-degree feedback (London and Beatty, 1993), since this is one of the mechanisms most frequently used for development and assessment purposes in multiple organizations. Briefly, 360-degree surveys gather feedback on an indi-
ED
370
vidual from different sources, typically including self-evaluation, subordinates, peers and supervisors. Subjects have to answer questions rating a partner in
PT
different topics related to team climate and team dynamics (e.g., leadership, teamwork, decision-making, job knowledge or productivity, between others). Then, statistical analysis can be used to obtain comprehensive performance and
CE
375
behavioral evaluations of each individual. When some of these appraisal systems is implemented in the organization, they usually provide us with information
AC
we can use for building subjective teammate networks. Another mechanism to collect the information required to build subjective teammate networks is using
380
one of the questionnaires described in Appendix B. This a simple manner to obtain these data when they are not available in the company, as well as when the existing assessment process requires too much effort from the employees (see threats to validity). 16
M
AN US
CR IP T
ACCEPTED MANUSCRIPT
Figure 3: Simple illustrative example of teammate network (left panel) and social-skill network (right panel) built based on the two projects on top. In this example, the networks are gener-
ED
ated considering the normalized effort deviation as team performance estimator. As in Fig. 1, green and red connections correspond to positive and negative relationships respectively.
385
PT
Unlike subjective networks, objective ones are not built on the basis of the employees’ opinion, but on an objective evaluation of their previous collaborations. Thus, objective metrics are joint-performance estimators that assume
CE
that members of performing teams are compatible and have positive relationships among them; while failing projects have non-compatible teammates with
AC
negative relationships. This requires a global project performance/success es-
390
timator, being the score of a particular link the sum of this estimator in all the projects where the corresponding teammates worked together (Fig. 3, left panel). The advantage of this strategy is that this kind of metrics is quite common in IS companies. Multiple metrics have been discussed in software engineering literature for project success, e.g., on-time/on-budget completion, 17
ACCEPTED MANUSCRIPT
395
system quality or effort/budget deviation (DeLone and McLean, 2003; Espinosa et al., 2006). The proposed framework adapts to the information managed by the company, and teammate networks can be based on any of these metrics (see
CR IP T
threats to validity in Section 6). Finally, we also test teammate networks that combine subjective and ob400
jective metrics. The motivation behind these networks is to identify negative
personal relationships among teammates even when they reach a high level of accuracy when working together and vice versa. To calculate the score of the
we use the following equation:
AN US
connection between two individuals (Ii and Ij ) in these collaboration networks,
CT MIi Ij = sig0.25 (CT MIsi Ij ) · sig3.0 (CT MIoi Ij ) · CT MIoi Ij 405
(1)
where sigβ (x) = 1/(1 + eβx ) is the sigmoid function, and being CT MIsi Ij and CT MIoi Ij the normalized subjective and objective score of the corresponding
M
link, respectively.
In the results section, we compare outcomes obtained with teammate net-
410
ED
works built with different strategies and metrics. 3.2.2. Social-skill networks
The motivation behind the social-skill network is the existence of particular
PT
combinations of social skills – i.e., personality traits and abilities a person uses to interact with other people – that make two people “connect” during team-
CE
work and have a productive professional relationship (Blaskovich, 2008; Adams 415
and Anantatmula, 2010; Feldt et al., 2010). In this situation, the assessment of relationships among coworkers as a function of their social skills may help to
AC
determine the optimal project team composition, in particular when they have not previously worked together (e.g., consider companies with a high turnover rate). Note that in this situation a teammate networks do not provide informa-
420
tion regarding compatibility. Different works in literature investigate the personality traits present in exceptional software engineers (Turley and Bieman, 1995), in the individuals that 18
ACCEPTED MANUSCRIPT
dominate team communication (Licorish and MacDonell, 2015) or the ones required to perform a specific development activity (Acu˜ na and Juristo, 2004; 425
Acu˜ na et al., 2006). However, we do not found definitive results pointing out
CR IP T
which are the relevant skills to be included in our social-skill network. Therefore, along with a group of occupational psychologists belonging to the HR
department of one of our industrial partners, we defined a list of 11 social skills
to be measured in the social profile of an individual: written communication, 430
verbal and non-verbal communication, active listening, teamwork, negotiation,
sociability, empathy, agreeableness, leadership. Appendix A contains a brief
AN US
description of what we mean by each of these social skills. The intention of
this 11-skill social profile is to determine and predict each candidate’s ability to communicate and interact with coworkers avoiding or minimizing conflicts 435
within the working team (see threats to validity).
The question now is how to determine an individual’s social profile. Again, the HR management domain provides us with tools to determine the level of
M
development of an individual in different social skills, including the ones of our social profile. For example, the Myers-Briggs Test Indicator (MBTI), the Big Five (BF) and the Sixteen Personality Factor Questionnaire (16PF) are com-
ED
440
monly used in software engineering literature (e.g., see Hardiman (1997); Acu˜ na and Juristo (2004); Feldt et al. (2010); Andr´e et al. (2011); Licorish and Mac-
PT
Donell (2015)). These are simple personality tests nowadays readily available online which we use in different in-company experiments. In addition to these tests, multiple organizations implement their own competencies assessment sys-
CE
445
tem to identify their employee’s personality traits. Depending on the company,
AC
the framework can receive the social profiles from these or other sources. A social-skill network is equivalent to a teammate network, but reflect-
ing the correlation in terms of productivity between different skills (Fig. 2,
450
right panel). Nodes in these networks represent skill-level of development pairs (SSK = {S1 , ..., Sn }). We consider as possible levels of development for the different social skills, as possible values in the corresponding metric used to assess candidates in this skill. Once identified the nodes in the network, to 19
ACCEPTED MANUSCRIPT
establish connections between them, we depart from an objective teammate 455
network. Two nodes (Si and Sj ) are connected when this network contains a link between individuals with the corresponding skill-level of development pairs.
CR IP T
The compatibility score of the link (CSKSi Sj ) is then calculated as the sum of the scores of all the connections between individuals with these skills in the teammate network. Social-skill networks do not directly measure compatibil460
ity among candidates. They serve to compute an objective metric (C(I1 , I2 )),
which can be used to build a new teammate network (IT M 0 = {I1 , ..., In }; RT M 0 = {CT MI0i Ij = C(I1 , I2 )}) reflecting the compatibility among poten-
AN US
tial candidates from the perspective of their social profiles. Teammate networks built in this way can be measured as any other objective or subjective team465
mate network (Section 3.4). Formally, given the social profiles of two individuals (SPI1 = {s1I1 , ..., snI1 } and SPI2 = {s1I2 , ..., snI2 }) and a social-skill network (SSK = {S1 , ..., Sn }; RSK = {CSKSi Sj }), C(I1 , I2 ) is estimated as follows: XX
M
C(I1 , I2 ) =
i
j
CSKsi
sj I1 I2
(2)
As an example, let consider the following social profiles {negotiation: high; 470
ED
leadership: high; listening: medium} and {negotiation: low; leadership: low; listening: low}. Given the social-skill network depicted in the right panel of
0.2.
PT
Fig. 3, the compatibility between two individuals with these social profiles is
CE
3.3. Strategies for defining potential teams for the project In our socio-technical approach, the technical dimension is evaluated in a
475
first stage to identify potential teams for the project. Note that we use the
AC
term technical in a broad sense, since our profiles can include technical and non-technical skills, personality traits, limiting individual factors and personal preferences (Section 3.1.1). To do this initial identification, we use a BFR methodology to match the required profiles and the available resources’ profiles
480
(see Section 3.1.1). Multiple combinations of candidates can satisfy the project technical requirements producing a great amount of potential teams, especially 20
ACCEPTED MANUSCRIPT
when the number of available resources is high. Then, SNA can be applied to every potential team to assess the social dimension (Section 3.4). However, we can use algorithms based on our social networks to limit the number of assessment and drive the direction of our predictions. 3.3.1. Individuals compatibility strategy
CR IP T
485
The individuals compatibility strategy departs from the premise that people
who have successfully worked together before will work successfully again. Note
that this does not imply maintaining permanent teams composed by the same people. In the context of the proposed socio-technical framework, working suc-
AN US
490
cessfully means having a high compatible score in the teammate network. Then, we only generate teams composed of individuals included in the project’s teammate network. If a compatible teammate is not found for any of the required profiles, additional team members are assigned considering the project’s social495
skill network (see Section 3.3.2). These second assignments are made taking
included in the same team.
M
into account that incompatible teammates in the teammate network are never
ED
Our results indicate that the individuals compatibility strategy is the best strategy to substitute a key member (e.g., the team leader) in already existing 500
teams once a project has begun.
PT
3.3.2. Social-skills compatibility strategy The social-skills compatibility strategy consists of creating the team finding
CE
compatible individuals as a function of their social skills, i.e., considering the information provided by the project’s social-skill network. In production, this
505
strategy is mainly used to create teams including people who have never worked
AC
together, e.g., employees recently hired. 3.3.3. Star-based strategy Some works in literature suggest that people with highly developed social
skills can facilitate project success (Licorish and MacDonell, 2014, 2015). The 510
proposed teammate networks allow building teams based on this idea. If we 21
ACCEPTED MANUSCRIPT
compute the Katz centrality (Katz, 1953; Grindrod et al., 2011; Sun and Tang, 2011) of every node in the teammate network of a project, we can identify individuals that normally interplay effectively with their coworkers. We call
515
CR IP T
these individuals social-stars. The social-star-based strategy consists of building the team around some of these individuals. Once the social-stars are selected, the team is completed using the social-skill network.
Similarly to the social-star-based strategy, the team can be formed around
individuals highly skilled to perform a given role or task. In this case, the technical-stars are selected as a function of the project technical needs and the team is also completed using the social-skill network.
AN US
520
3.3.4. Split strategy
The split strategy consists of splitting a performing team in two new teams. New teams are completed with individuals with the same or similar social skills to the ones observed in the members of the original team. This strategy is mainly used with groups of programmers or testers with the idea of increasing
M
525
the pool of effective relationships.
ED
3.4. Analyzing compatibility networks To mine knowledge from the proposed social networks, we compute two measures for each potential team (T ). On one hand, the number of positive links within the team (P RT ). Depending on the strategy and the metric used to con-
PT
530
struct the network, positive has a different interpretation. For example, in the
CE
case of a subjective teammate network based on 5-point qualitative metric that rates relationships between teammates as “not effective”, “somewhat effective”, “effective”, “very effective” and “extremely effective”, we will consider positive those links labeled as “effective”, “very effective” or “extremely effective”. On
AC
535
the other hand, we compute the team social score (T SST ), defined as the cu-
mulative compatibility score of the relationships between every team member within the team. Formally, if T is composed by the teammates {tm1 , ..., tmm }: XX T SST = CT Mtmi tmj (3) i
j
22
ACCEPTED MANUSCRIPT
As the compatibility scores computed for links belonging to two distinct 540
teammate networks, team scores derived from different networks are not comparable. In this sense, it allows comparing teams in the context of a given social
CR IP T
network, i.e., teams assessed under the same conditions. The greater the social score of a team, the higher its estimated success probability.
Once computed these two values, the goal is to find an optimal balance be545
tween maximizing T SS and P RT . Let consider a simple illustrative example
departing from the teammate network depicted in Fig. 2. Our goal is to create a new 4-member team with a unique common required profile, which, for simplic-
AN US
ity in the example, all subjects in the network have. SNA allows us to compute easily the T SS value for all the possible teams: 26 out of 70 possible 4-member 550
teams do not have negative relationships, being T1 = {I4 , I6 , I7 , I8 } the team with the greatest T SS (T SST1 = 17). Then, only considering T SS, the network indicates that T1 is the best team to be assigned to the project. Nevertheless, P RT1 = 2, i.e., we only have evidence of 2 out 6 positive relationships among
555
M
the members of T1 . If we now consider T2 = {I1 , I3 , I4 , I6 }, T SST2 = 16 and P RT2 = 5. This is also a good recommendation from the team social score
ED
viewpoint (cf. T SST1 and T SST2 ), but the teammate network provides additional evidence of a good joint-performance between all members of T2 except I3 and I6 . Furthermore, assuming transitive relationships between software
560
PT
professionals, i.e., relationships between collaborators of collaborators (Surian et al., 2011), we could predict that I3 and I6 will be compatible too, because
CE
both are compatible with I1 and I4 . Transitive relationships of a given individual are computed by the inspection with depth 2 of the nodes accessible from the corresponding node in the network. Then, considering the information pro-
AC
vided by the network of Fig. 2, T2 is the 4-member team with a greatest success
565
probability.
23
ACCEPTED MANUSCRIPT
4. Proposal validation To validate our proposal, we carried out different kind of experiments in our industrial partners. In these experiments, we addressed two main goals. First,
570
CR IP T
to verify our proposal applicability and the suggested teams coherence. Second,
to validate whether the proposed teams were actually effective and productive. For statistical analysis, we group our industrial partners into four categories
depending on the number of employees. According to the definitions currently in
used in the European Union1 , companies with less than 50 and 250 employees are considered small- and medium-sized companies, respectively; while large-sized
enterprises have more than 250 employees. Although some of the medium- and
AN US
575
large-sized companies have employees located in offices from different countries, we consider an additional category (multinational organization) for more-than1000-employee enterprises with teams split over multiple countries.
M
4.1. Simulation of team assignments
To look into the applicability and coherence of our proposal in a given com-
580
pany, we first performed simulations of team assignments. These are proof-of-
ED
concepts in which we define a series of required profiles for fake projects and use the framework to form the corresponding team with the company’s current employees. Depending on the company, required profiles are obtained from the planning of previously completed projects and/or they are randomly chosen from
PT
585
a pool of predefined profiles. Then, a group of project managers, team leaders
CE
and/or HR management professionals designated by the company validates the assignments. When they reject a team, they must indicate the rejection causes. Before running the simulations of team assignments, the framework must
be adapted to the company. The aims of this initial phase are to identify the
AC
590
information already available in the company that can be used to build the social 1 EUR-Lex:
96/280/EC: Commission Recommendation of 3 April 1996 concerning the
definition of small and medium-sized enterprises, Official Journal L 107, 30/04/1996 P. 0004 - 0009
24
ACCEPTED MANUSCRIPT
networks, and define the protocols to capture required data. The company’s HR department usually participated in the framework adaptation. In many cases, they proposed elements to incorporate into the social networks – e.g., additional competencies and/or social skills they include in the employee’s profile or specific
CR IP T
595
metrics provided by the tools they use to assess team climate. To quantify the
adaptability and applicability of the framework, we calculate the percentage of companies where simulations could be run after the initial adaptation phase.
Once the framework is adapted to the company, simulations of team assign600
ments allow us to compare coherence of the teams proposed by social networks
AN US
generated with different metrics and strategies. Depending on the data available in the company, we can construct, measure and compare:
• Subjective teammate networks generated as a function of the employees’ responses to questionnaires Q1 and Q2 of Appendix B. 605
• Subjective teammate networks built with the information provided by any
M
existing tool to evaluate coworkers’ collaborations.
• Objective teammate networks generated using different project metrics.
ED
For statistical analysis, we distinguish between four success/performance indicators. Effort and budget deviation – i.e., the difference among es610
timated and actual effort/budget – since these are the most commonly
PT
available indicators in our industrial partners. Another project metrics, such as customer satisfaction or different quality measures, are only avail-
CE
able in a few companies. We group all these metrics into a unique category in order to get enough statistical evidence. Finally, a last simple
AC
615
metric, which can be even used when the company does not keep track of any project metric, consists of rating the projects as “successful” or “unsuccessful” depending on whether they are completed on schedule and meeting budget and requirements.
• Teammate networks combining different objective and subjective metrics 620
(Eq. 1). 25
ACCEPTED MANUSCRIPT
• Social-skill networks built with the subjects’ social profiles. Coherence of the teams obtained from networks constructed with different strategies is evaluated in terms of the percentage of accepted teams. These
625
CR IP T
values are compared with results obtained in control experiments where team
members are selected randomly using two possible strategies. With the first strategy, all the team members are selected randomly from the pool of available
resources independently of their skills. With the second one, each member is selected randomly too, but now from the set of candidates best suited for each
630
AN US
task or role. 4.2. Pilot experiments
Simulations of team assignments allow us to verify the applicability of our proposal and the coherence of the suggested teams, but they do not validate whether these teams are actually effective. To do this, we carried out pilot
635
M
experiments where the proposed framework served as a tool to help decisionmakers allocate the team of real IS projects. In the lead-up to these experiments, we always performed simulations of team assignments to adapt the framework
scenario.
ED
to the company and identify the social networks providing better results in each
In the pilot experiments, the final decision about the team composition is on the managers’ hands. Then, as in the simulations, we study the percentage
PT
640
of teams suggested by our social networks and assigned to real projects. We
CE
use this value to quantify the managers’ confidence in the framework. Note the difference between accepting a team in a simulation and assigning this team to a production project in the pilot experiments. In addition to this implicit measure of the managers’ perception, we also analyze what they think the framework
AC
645
brings to the team allocation process. For that, they have to fill in with answers a questionnaire with the following questions: • QPE1. How helpful do you think the tool for selecting the project team is? (1-5) 26
ACCEPTED MANUSCRIPT
• QPE2. I identified suited people for the projects who I had not taken into
650
consideration (1-5)
CR IP T
• QPE3. The tool allows forming more efficient project teams (1-5) • QPE4. The tool could be used to form project teams automatically (1-5) The questions are expressed on a Likert scale, where 1-5 mean “Not at 655
all helpful”, “Slightly helpful”, “Moderately helpful”, “Very helpful” and “Ex-
tremely helpful” for QPE1; and “Strongly disagree”, “Disagree”, “Neither agree nor disagree”, “Agree” and “Strongly agree” for QPE2-4.
AN US
Once a team is assigned to a production project, we verify whether this assignment was effective by evaluating four common project productivity metrics: 660
a binary success indicator, effort deviation, budget deviation and customer satisfaction. In a study like ours, it is nearly impossible to quantify to what extent the project benefited of a specific team composition as compared to other as-
in the same real project.
4.3. Large-scale production studies
ED
665
M
signments, since we cannot compare results obtained by different teams working
After the pilot experiments, some companies adopted the proposed framework as a supporting tool helping decision-makers during the creation of teams
PT
for IS projects. This demonstrates their confidence in our socio-technical approach, but moreover the large-scale production studies allow us to verify improvements in the team allocation process (see threats to validity).
CE
670
Although as in the pilot experiments, we cannot compare results of different
teams undertaken the same project, in the large-scale studies we do have enough
AC
statistical sample to compare performance of teams formed using the proposed framework and teams formed using other strategies (the specific strategy de-
675
pends on the company and the project). Comparisons are made in terms of different measures, mainly productivity and quality metrics (e.g., effort/budget deviation or customer satisfaction), depending on the company and the expected benefits. To minimize possible bias, results of the teams suggested by 27
ACCEPTED MANUSCRIPT
Personality traits (Yes-No)
Collaborations (Yes-No)
Applicability
Small-sized
9-6
2-13
0-15
100%
Medium-sized
16-3
10-9
8-11
Large-sized
5-0
4-1
2-3
Multinational
2-0
2-0
2-0
TOTAL
41
41
41
CR IP T
Project success (Yes-No)
Type of company
89.5% 100%
100%
95.1%
Table 1: Summary of the companies where the framework was tested. Middle columns indicate
AN US
the number of companies with the corresponding measures already available before performing
the simulations of assignments. The last column shows the percentage of companies where the framework was successfully applied.
the framework are compared with baselines established averaging the results of 680
projects completed by teams formed with the allocation process that we want to
M
improve. An improvement in the metrics analyzed in the large-scale production studies does not guarantee the assignment of the most effective team, but it gave us enough statistical evidence to verify that the framework has a positive
5. Results
PT
685
ED
impact on the company.
5.1. Applicability of the proposed framework
CE
To date, we have simulated team assignments in 41 organizations of different size and with different internal processes and hierarchical structure (Table 1). This has allowed us to validate the proposed framework adaptability and applicability to multiple scenarios. The framework was successfully tested in 39
AC
690
out 41 (95%) of our industrial partners. In these 39 companies, we were able to capture the required data and/or adapt the framework to the data previously managed by the company and run the simulations. The two companies where the simulations failed had quite similar proper-
695
ties: around 100 software professionals involved in the project, the only informa28
ACCEPTED MANUSCRIPT
tion available of these subjects was their curricular data, and historical data of projects were not available. In both companies, we found two problems during the framework adaptation. First, it was nearly impossible to build a reliable
700
CR IP T
base of knowledge of successful and unsuccessful previous teams. Second, the data acquisition process required some effort on the company: employees must
perform different tests to identify their personality, behavioral and social traits;
and they must complete the questionnaires of Appendix B to assess their interpersonal relationships with coworkers. Due to these issues, the project was
canceled before running the simulations. In this regard, it must be said that data capture was a bottleneck in some medium- and large-sized companies be-
AN US
705
cause of the number of questionnaires/evaluations each employee had to perform before running the simulations (see threats to validity). In spite of this, simulations were successfully run in six medium and one large enterprise with similar characteristics to the companies where they were canceled (i.e., without pre710
vious data regarding personality traits and collaborations). Four of these six
M
medium-sized companies are currently using the framework as a supporting tool
ED
to form the production teams of their IS projects. 5.2. Teams coherence
The simulations of team assignments indicate that the proposed social networks in general suggest coherent project teams (Fig. 4). No matter the type
PT
715
of company or the strategy used to construct the network, the experts designated by the company consider around 90% of the suggested teams coherent
CE
and valid for the project (cf. control experiments). The greater acceptance percentage occurs for small-sized companies due to the small number of potential candidates. For instance, acceptance is 100% in the three companies with less
AC
720
than 12 employees. However, this is not a significant result. Projects in these companies often required very homogeneous development profiles (e.g., android developer with knowledge in JSON, XML, Java and SQL), and most of the employees had these skills. This makes nearly all the possible candidates to be
725
considered adequate for the project. This observation is corroborated by the 29
CR IP T
ACCEPTED MANUSCRIPT
Figure 4: Acceptance percentage of the teams suggested by our teammates and social-skill networks in the simulations of team assignments. Results of the control experiments are
grouped into teams randomly selected from the whole set of available resources (“Random”)
AN US
and the ones selected from the set of candidates best suited for each task or role (“Random best”). Objective and subjective teammate networks are grouped into different categories depending on the metric used to build the network. “HR tool” corresponds to networks based on data extracted from an existing tool (note that none of the small-sized companies has this kind of tool). “Success indicator” corresponds to a binary successful/unsuccessful metric. “Other” includes project metrics but effort and budget deviation. See Section 4.1 for details.
M
control experiments, where teams selected randomly in small companies have a high acceptance percentage (88% or 96% depending on the strategy used to
ED
select the team members). Therefore, we cannot draw any conclusion about coherence of the teams suggested by the framework in small-sized companies. 730
We discuss below the helpfulness in these organizations.
PT
Comparing the different strategies to generate teammate networks, teams proposed by objective networks have in general a better acceptance than the
CE
proposed by subjective networks. This occurs regardless of the objective team performance estimator or the mechanisms used to assess subjectively interper-
735
sonal relationships (Fig. 4). Although it seems obvious that only the team
AC
members themselves have a detailed knowledge of the social interactions in the project, the simulations suggest that objective joint-performance estimators have a greater accuracy to capture efficiency of these interactions, and, therefore, objective networks contain more relevant information for us. A more
740
detailed analysis corroborates this observation. For example, teams formed with
30
AN US
CR IP T
ACCEPTED MANUSCRIPT
Figure 5: Objective and a subjective teammate network generated for a medium-sized company with projects completed within the last five years. For simplicity in the visualization, we only show connection for one employee (represented in blue in both networks). The jointperformance indicator used to build the objective network is the binary success indicator, i.e.,
M
successful projects compute +1 and failing projects −1. The subjective network is based on the last-five-year data extracted from an assessment tool already existing in the company. It scores the relationships between employees with a value between 0 and 100. The greater
ED
the score, the more positive the interpersonal relationship (ranges 0-39, 40-60 and 61-100 correspond to negative, neutral and positive relationships, respectively). Colors in both cases are used to identify positive, negative and neutral relationships according to the information
PT
captured in the corresponding network.
the social-stars strategy have a greater acceptance when social-stars are iden-
CE
tified in objective teammate networks. These networks also allow us to detect positive relationships between individuals with limited social skills that do not appear in networks generated with subjective metrics. For example, the blue individual of Fig. 5 represents a senior software developer. In the last five years,
AC
745
he participated in 11 project, six completed successfully and five unsuccessfully. His social profile indicates that he has very limited social abilities: excellent and acceptable competence in written communication and leadership, respectively; and a poor or very poor level of development in the rest of social skills. This
750
translates into a difficulty interacting with teammates, which is reflected in the 31
ACCEPTED MANUSCRIPT
subjective teammate network (Fig. 5, right). Taking into account this network, this employee was never proposed in the simulations as member of a new team. However, while the network built with subjective metrics does not show any
755
CR IP T
positive connection, six productive interactions can be observed in the equivalent objective network (Fig. 5, left). In view of this network, for example, a
team containing the blue individual, I3, I6 and I7 will be likely productive (cf. connections in the right panel). If the large-scale production experiments con-
firm that teams established in this way are actually productive, this is a highly relevant property for the team assignment problem in the context of a complex
collaborative process like software development. The objective teammate net-
AN US
760
works would allow identifying teammates able to adapt their working styles to special circumstances as the difficulty in social interactions.
A possible explanation for the problems found in the assignments derived from subjective evaluation of interpersonal relationships is that these metrics 765
are biased by other positive or negative relationships, e.g., affinity or friendship,
M
which do not always translate into productive or non-productive professional relationships. It is important to highlight that this bias appears in the simple
ED
questionnaires proposed in Appendix B, but also in a well-accepted HR assessment of team climate like 360-degree feedback. In fact, our results indicate that 770
the teams suggested by networks generated using questionnaires Q1 and Q2, and
PT
the teams suggested by a HR-tool-based network have nearly the same components and a very similar acceptability (Fig. 4). This result points out that, for
CE
our needs, the simple questionnaires of Appendix B collect equivalent information to the feedback gathered by more complex assessment processes. In the
775
context of a social-technical approach like ours, this is a highly relevant result
AC
for companies without corporate mechanisms to assess collaborations and team dynamics. Note that none small companies, and only 8 out of 19 medium-sized implement this kind of tool.
780
Concerning results of the different objective teammate networks, one could
expect that the higher the accuracy of the success/performance estimator, the more reliable the corresponding network. Nevertheless, the stakeholders’ general 32
ACCEPTED MANUSCRIPT
perception is that the optimal results are obtained when the networks are built as a function of the effort deviation. Note that this metric largely depend on the estimation quality – e.g., if a project fails due to over-optimistic scheduling or under-staffing, the corresponding team is considered non-productive.
CR IP T
785
Despite the possible bias discussed above in the subjective teammate net-
works, our analysis points out that subjective evaluation of collaborations can add some value when it is incorporated as correction factor to objective networks. Networks built in this way slightly improve the percentage of acceptance 790
of objective teammates networks (Fig. 4). Studying the teams proposed by these
AN US
networks, we observe that the information provided by teammates about their
interpersonal relationships allows measuring personal preferences together with the productive dimension of professional collaborations. Given how the compatibility score is computed in these networks (Eq. 1), productive relationships 795
where both teammates think that the collaboration is positive (although biased by friendship) are considered “more productive”. In the same way, subjective
M
evaluation makes “more negative” those non-productive relationships between teammates with a non-good personal relationship.
800
ED
Finally, results of the social-skill networks are nearly the same to the obtained with the teammate networks (in all cases they have a greater than 90% acceptance). It is important to remind that the teams proposed by these net-
PT
works are not based on previous collaborations, but on the estimated compatibility between their members’ social skills. The larger the company size, the
CE
more coherent the teams proposed. This result corroborates our departing hy805
pothesis that putting together coworkers as a function of their social skills is a good strategy to build IS project teams (at least from the coherence point of
AC
view).
5.3. Managers’ perception
810
We performed pilot experiments in 28 industrial IS enterprises. Table 2
summarizes the properties and measures used to build the proposed social networks in each company. It is important to remind that these are fixed after a 33
ACCEPTED MANUSCRIPT
simulation phase (see Section 4.2).
2
small (19)
3*
small (44)
4
small (24)
5*
small (41)
6*
small (45)
7
small (25)
8
small (30)
9
small (32)
10*
medium (87)
11*
medium (213)
12*
medium (232)
14*
ED
15
medium (105)
16*
medium (91)
17*
medium (203)
18
medium (64)
19
medium (133)
20*
medium (215)
21*
medium (152)
medium (123) medium (99)
AC
CE
PT
13*
effort deviation effort deviation defect density success indicator success indicator success indicator success indicator effort deviation defect density effort deviation defect density effort deviation defect density success indicator success indicator budget deviation budget deviation success indicator effort deviation effort deviation effort deviation
Q2 Q2
Personality traits
34
#teams
CR IP T
small (48)
Subjective measure
existing HR tool
#accepted
27
26
MBTI-BF-16PF existing HR tool
10
10
21
20
MBTI-BF-16PF
17
15
MBTI-BF-16PF
36
32
—
MBTI-BF-16PF
34
32
Q1
MBTI-BF-16PF
14
11
Q1
MBTI-BF-16PF
12
11
Q1
MBTI-BF-16PF
6
6
—
MBTI-BF-16PF
40
35
—
MBTI-BF-16PF existing HR tool
52
40
82
74
MBTI-BF-16PF existing HR tool existing HR tool
25
22
35
26
35
28
MBTI-BF-16PF existing HR tool
41
29
67
58
MBTI-BF-16PF existing HR tool existing HR tool existing HR tool
28
27
24
21
79
71
56
47
Q2 Q2 Q1
AN US
1*
Objective measure
M
Company
Type of company (participants)
Q1 Q2 existing HR tool Q2 existing HR tool existing HR tool Q2 — existing HR tool existing HR tool
ACCEPTED MANUSCRIPT
medium (111)
23
medium (55)
24*
large (289)
25
large (325)
26*
large (312)
27
large (450)
28*
multinational (1819)
TOTAL
effort deviation effort deviation effort deviation effort deviation success indicator budget deviation effort deviation
Subjective measure
Personality traits
#teams
#accepted
—
MBTI-BF-16PF
21
17
CR IP T
22*
Objective measure
Q1 existing HR tool Q2 Q2
Q1 existing HR tool
MBTI-BF-16PF existing HR tool existing HR tool
MBTI-BF-16PF existing HR tool existing HR tool
AN US
Company
Type of company (participants)
6
6
88
71
56
43
109
94
71
63
380
340
1472
1275
Table 2: Features of the pilot experiments. Note that, in line with the results discussed in Section 5.2, we always measured team-
M
mates networks built based on objective metrics, or on the combination of objective and subjective metrics. Companies identified
ED
with (*) incorporated the proposed framework to their staffing pro-
PT
tocol and/or are currently performing large-scale studies.
We first analyze the project managers’ opinion about the framework. Table 3 shows the distribution of their answers to the questionnaire regarding the helpfulness of the framework during team assignments (Section 4.2). In general,
CE 815
they think that the framework helps decision-making, allowing the identifica-
AC
tion of employees suited for the project that they did not know or, at least, they did not initially consider for the project. In spite of this, they believe that the team allocation must be supervised (cf. response to QPE4). Beyond the
820
general result, there are some significant differences in the managers’ perception of the framework depending on the company size. A significant number of managers belonging to small companies do not consider the framework useful
35
2
3
4
Small-sized
40.1%
23.2%
2.8%
20.3%
Medium-sized
1.4%
5.7%
9.5%
60.4%
Large-sized
2.2%
11.1%
6.5%
40.4%
Multinational
0.8%
1.6%
8.4%
38.1%
TOTAL
6.1%
8.0%
7.7%
45.5%
Small-sized
51.4%
10.2%
2.3%
27.1%
Medium-sized
1.2%
3.9%
1.9%
46.7%
46.3%
Large-sized
0.3%
4.6%
7.1%
42.9%
45.1%
Multinational
1.3%
2.4%
3.2%
52.4%
40.7%
TOTAL
7.1%
4.4%
3.4%
45.0%
40.1%
Small-sized
50.3%
7.3%
24.9%
16.9%
0.6%
Medium-sized
1.9%
2.9%
10.8%
69.7%
14.7%
Large-sized
6.8%
1.2%
9.9%
34.0%
48.1%
Multinational
0.0%
1.3%
9.5%
39.5%
49.7%
TOTAL
8.3%
2.6%
12.0%
47.7%
29.4%
Small-sized
68.9%
24.3%
6.8%
0.0%
0.0%
PT
5
Medium-sized
53.1%
34.0%
8.5%
4.4%
0.0%
Large-sized
92.3%
7.7%
0.0%
0.0%
0.0%
Multinational
97.9%
0.5%
1.6%
0.0%
0.0%
TOTAL
75.2%
18.4%
4.6%
1.8%
0.0%
CR IP T
1
CE
QPE4
13.6%
23.0%
39.8%
51.1%
32.8% 9.0%
AN US
M
ED
QPE3
QPE2
QPE1
ACCEPTED MANUSCRIPT
AC
Table 3: Distribution of the managers’ responses to the questions about the framework helpfulness. They are expressed on a Likert-type scale as follows. QPE1: 1 = Not at all helpful; 2 = Slightly helpful; 3 = Moderately helpful; 4 = Very helpful; and 5 = Extremely helpful. QPE2-4: 1 = Strongly disagree; 2 = Disagree; 3 = Neither agree nor disagree; 4 = Agree; and 5 = Strongly agree. See Section 4.2 for details.
36
ACCEPTED MANUSCRIPT
(40.1% not at all helpful; and 23.2% slightly helpful). They believe that they can make team assignments as a function of their experience and knowledge 825
about the potential team members, and the framework does not provide them
CR IP T
with any additional information. This is in line with the results obtained in the simulations of team assignments, and can be something expected in companies where everyone knows each other (cf. response to QPE2). Interestingly, all the managers of small companies who consider the framework very helpful or 830
extremely helpful belong to more-than-40-employee companies. As the number of potential candidates grows, this perception disappears and managers tend to
AN US
think that the proposed social networks help them understand the contribution of the people they manage.
In addition to the managers’ opinion, we also analyze their confidence in the 835
framework by means of the percentage of teams accepted and assigned to real projects. Figure 6 shows that this value is high (around 85%, see also column “#accepted” in Table 3), but always lower than the percentage of accepted
M
teams in the simulations. The analysis of the rejection causes indicates that this is because, in production, managers tend to reject the teams composed of people that do not have worked together yet. Although in the simulations they
ED
840
consider these teams coherent and valid for the project, in production they prefer not to do these assignments when a team composed of previous teammates can
PT
be formed. As some of the suggested teams successfully advances, confidence in the framework assignments grows, and a greater number of teams based on the social-skill networks is accepted and assigned to new projects. Note that,
CE
845
in spite of the initial doubts, the mean acceptance is around 87% (86% if we do
AC
not take into account the small-sized companies). 5.4. Team effectiveness validation
850
Figure 7 shows the outcomes of the teams assigned to real IS projects from
different perspectives. We first consider that a project is successfully completed when the system is delivered on-time, within-budget and meets the required features and functions (“success indicator”). Likely, this is the indicator that best 37
CR IP T
ACCEPTED MANUSCRIPT
Figure 6: Percentage of acceptance of the teams suggested by our teammates and social-skill
CE
PT
ED
M
AN US
networks in the pilot experiments.
AC
Figure 7: Outcomes of the project teams suggested by the socio-technical framework.
summarizes the team success. From this perspective, 87.9% of the teams can
be considered successful teams. Fails from this perspective are mainly a conse-
855
quence of scheduling deviations (∼ 83% of the failing projects). Interestingly, only 6% of the failing teams do it because of the system quality. Results from the effort, budget and customer satisfaction viewpoint are nearly equivalent, al-
38
ACCEPTED MANUSCRIPT
though we would like to highlight results from the effort deviation perspective. In some projects, the team needs more time than the estimated to complete 860
some tasks. This does not always translate into a budget deviation or a quality
the effort deviation are the more reliable.
CR IP T
decrease, but it can explain why the teammate networks built as a function of
In view of the results shown in Fig. 7, we can conclude that the teams suggested by the framework are in general effective and efficient. 865
5.5. An empirical study
AN US
To date, 17 out of 39 companies (Table 2) where the framework was successfully applied are currently running large-scale production studies, i.e., 44% of the companies (41% considering all the companies where we tried to apply the
framework; 54% if we do not include companies with less than 20 employees; and 870
61% if we only consider those companies that performed pilot experiments). The most relevant empirical study is ongoing at a multinational corporation based
M
in Europe, Asia and South America (company 28 of Table 2). After performing different simulations and pilot experiments with employees and projects belong-
875
ED
ing to different countries, the company decided to address the improvement of the people assignment process in nine software factories that showed a general deviation in scheduling and budget. They were located in Spain (2), France (3),
PT
Germany (1), Mexico (1) and Argentina (2). In this section, we focus on the results of measuring the proposed social networks when assigning people to the projects faced by these nine software factories during more than three years. Appendix C describes the steps leading up to the framework application.
CE 880
AC
5.5.1. Effectiveness and efficiency improvements The expected benefits of applying the framework in the nine software fac-
tories were mainly related to effectiveness and efficiency enhancements. Then, we compare the productivity and quality results of teams formed before and
885
after the framework start-up. In particular, the analyses are made in terms of effort deviation (productivity) and defects per KLOC (quality). Baselines for
39
Individuals
Social-skills
Social-stars
Tech-stars
Split
Effort deviation
0.84 beall
0.81 beall
0.85 beall
0.85 beall
0.84 beall
0.83 beall
Defect density
0.77 bdall
0.75 bdall
0.79 bdall
0.79 bdall
0.76 bdall
0.77 bdall
Effort deviation
0.85 bees
0.82 bees
0.85 bees
0.85 bees
0.87 bees
0.86 bees
Defect density
0.79 bdes
0.77 bdes
0.83 bdes
0.81 bdes
0.78 bdes
0.79 bdes
Effort deviation
0.81 bef r
0.77 bef r
0.83 bef r
0.85 bef r
0.80 bef r
0.81 bef r
Defect density
0.76 bdf r
0.73 bdf r
0.79 bdf r
0.76 bdf r
0.78 bdf r
0.75 bdf r
Effort deviation
0.89 bede
0.90 bede
0.89 bede
0.87 bede
0.91 bede
0.88 bede
Defect density
0.79 bdde
0.75 bdde
0.81 bdde
0.84 bdde
0.74 bdde
0.79 bdde
Effort deviation
0.82 bemx
Defect density
0.73 bdmx
Effort deviation
0.81 bear
Defect density
0.79 bdar
CR IP T
Overall
AN US
ar
mx
de
fr
es
all
ACCEPTED MANUSCRIPT
0.77 bemx
0.84 bemx
0.86 bemx
0.83 bemx
0.81 bemx
0.71 bdmx
0.70 bdmx
0.71 bdmx
0.77 bdmx
0.75 bdmx
0.78 bear
0.84 bear
0.83 bear
0.80 bear
0.82 bear
0.79 bdar
0.84 bdar
0.84 bdar
0.72 bdar
0.77 bdar
Table 4: Productivity and quality measures of the on-site projects undertaken by teams
M
formed with the support of the proposed framework. Table 5 displays equivalent measures for distributed projects. Results are shown as a function of the three-year baselines for effort deviation (bex ) and defect density (bdx ); globally (“all”) and grouped by country (“es”,
ED
“fr”, “de”, “mx” and “ar” correspond to Spain, France, Germany, Mexico and Argentine, respectively). Columns display results grouped by the strategy followed to form the team.
PT
comparison are computed with the projects completed within the three years prior to start applying the framework (see Appendix C). Table 4 shows the
CE
outcomes obtained in the on-site projects – i.e., projects where all the team 890
members belong to the same software factory – completed by teams formed with the framework support during a more-than-three-year period. Analyzing
AC
the overall results, we observe a general enhancement, both in productivity and quality. The mean effort deviation is 16% fewer; while the mean defect density has decreased 23%. Improvements by country are coherent with these values.
895
Teams composed of teammates who have successfully worked together before
generally are more effective (cf. the individuals compatibility strategy and the rest of strategies in Table 4). Most members of these teams know each other and 40
ACCEPTED MANUSCRIPT
have close professional relationships. The existence of prior positive relationships within the team favors collaboration and coordination and facilitates be900
coming a performing team. Conversely, the teams formed using other strategies
CR IP T
need time to grow and consolidate the teammate-teammate relationships. This implies that these teams follow a more pronounced forming-storming-norming-
performing pattern (cf. red and blue traces in Fig. 8). However, in spite of the
more pronounced forming, storming and norming phases, they usually become 905
performing teams after a relative short formative period. Note that the mean
effort deviation in Table 4 is similar independently of the strategy followed to
AN US
select the project team. This result is observed even in teams created using
the social-skills compatibility strategy, which does not take into account previous collaborations to evaluate relationships among coworkers. This observation 910
served to decide using this strategy to assign people to small non-critical projects in order to create new teammate-teammate relationships. These new relationships usually become effective during the project, and then the new compatible
M
teammates were assigned to projects that were more critical. In this regard, we would like to emphasize the high turnover rate: 26%, 24.5%, 19.5%, 28.5% and 17% in the last year in the Spanish, French, German, Mexican and Argentinean
ED
915
software factories. In this situation, it seems critical the formation of effective relationships among the employees recently hired.
PT
Although the statistical sample is enough to minimize bias, the improvements could be consequence of other factors, e.g., a more skilled employee recruiting. Furthermore, effort deviation and defect density are simple and in some
CE
920
sense unreliable measures of productivity and quality. Therefore, we would like to emphasize the stakeholders’ general perception. They believe that the use
AC
of the proposed social networks led to a noteworthy improvement in the new teams’ efficiency, which minimized the scheduling and budget problems in the
925
nine software factories. In fact, they have started three new large-scale production studies: one involving three software factories in India; another with five software factories in Mexico and five in Argentina; and the last one with 15 software factories located in Europe (3 in Spain, 8 in France and 4 in Germany). 41
CR IP T
ACCEPTED MANUSCRIPT
AN US
Figure 8: Evolution of the mean normalized performance of 20 different on-site French teams assigned to a more-than-one-year project. Red trace corresponds to 20 teams created using the individuals compatibility strategy. Blue trace corresponds to teams created using the social-skills compatibility strategy (7), the social-stars strategy (7) and the split strategy (6). The analysis covers the 12 months after the team allocation and its assignment to a new project. Dashed line denotes the French three-year performance baseline.
M
5.5.2. Emphasizing the relevance of social interactions
Beyond the general results shown in Table 4, additional observations after
930
ED
measuring the proposed social networks during more than 36 months can be summarized in the following points: • Results of teams created with the social-stars and the technical-stars strat935
PT
egy are nearly the same. This emphasizes the relevance of interpersonal relationships for project success and suggests that social skills can be as
CE
important as technical skills. In some cases, the social-stars strategy can produce teams with a better mean performance (see Spanish and Ger-
AC
man teams in Table 4). In step with these results, multiple projects were
940
successfully completed by development teams not having all the required technical skills, but having a high social score and a large amount of positive interpersonal relationships between their members. A detailed analysis indicates that, although from a technical viewpoint these were not good team assignments, the projects met their budget and quality ob-
42
ACCEPTED MANUSCRIPT
jectives due to an effective knowledge sharing. As expected, there are fluid 945
collaborations within the working team, but also with employees outside the team. The proposed social score is not defined to predict this kind of
CR IP T
collaborative behavior. However, our analysis points out that the greater the number of social-stars in a project team, the higher the knowledge sharing outside the team. 950
• Taking into account the results of Licorish and MacDonell (2014, 2015), the social-starts strategy was initially proposed thinking in people in man-
agement, architectural and development roles. Nevertheless, depending
AN US
on the country, around 27-39% of the individuals identified as social-stars perform an ancillary role. These results point out again the relevance 955
of the social aspects. Despite that, we would like to highlight the IS project leaders’ profile. It is well-known that project leaders play a crucial role and can be a major source of errors that lead to failure due to
M
bad decision-making (Charette, 2005). However, we have found that relationships between project leaders and the rest of team members are also 960
critical for project success. Multiple non-effective teams become effective
ED
under the command of project leaders with social skills compatible with the ones of the team members. Note that we do not only refer to specific
PT
project leaders, but to different leaders with specific social-skills. • Teammates with limited social skills (e.g., people showing a low team965
work capacity) can also constitute high-performing teams. These effective
CE
relationships usually appear between highly skilled technicians that, in general, only have productive relationships with similar peers. This kind
AC
of teams successfully performs small projects or specific – and usually
970
complex – tasks. If this is not the case, the team needs complementary members with additional social skills. • When a performing team has a high maturity level, replace some of its members does not significantly affect the team’s effective productivity. In this regard, we would like to stress results obtained with the split strategy 43
ACCEPTED MANUSCRIPT
since, departing from a performing team; it permits obtaining two nearly 975
equivalent teams to the original one. This strategy increases the number of productive teammate-teammate relationships.
CR IP T
• Finally, teammates and social-skill networks have helped to create project
teams distributed across different software factories. This was not a common practice in the company. Then, initially, distributed teams were 980
created using the social-skills compatibility strategy. A great percentage of these initial teams failed, even when teammates belonged to the same
country. This suggests the need of different social skills for interacting
AN US
with distributed coworkers. Once the number of productive distributed teammate-teammate relationships grew, performing teams were formed 985
using the individuals compatibility strategy. Afterwards, the increased number of productive relationships allows successfully using other assignment strategies. For instance, high-performing teams have been created
M
around distributed social-stars identified in the teammate networks. Currently, distributed teams’ mean performance and quality are nearly the 990
same than the on-site teams (cf. Tables 4 and 5). Although it is difficult
ED
to draw general conclusions due to the individual-dependent nature of social interaction, distributed teammate networks allow us to analyze how
PT
compatible social profiles change depending on the teammate’s location. For example, around 87% of people that successfully interact with cowork-
995
ers of other countries show a high empathy level. Similarly, more than
CE
75% of these people also have a medium-high adaptability level. In other cases, a dependence on the language used to communicate is observed.
AC
For example, around 72% of the effective teammates who communicate in
1000
English show a high active listening level. There even exist skills needed to interact successfully with coworkers of specific countries.
44
ACCEPTED MANUSCRIPT
Spanish
English
Effort deviation
0.84 beall
0.83 besp
0.87 been
Defect density
0.84 bdall
0.85 bdsp
0.84 bden
CR IP T
Overall
Table 5: Outcomes of the distributed teams composed of people belonging to different software
factories. Columns labeled as “Spanish” and “English” indicate the language used to communicate. Communication is in Spanish language if the project involves people from Spain,
Mexico and/or Argentina; and it is in English when teammates are from Spain, France and/or Germany.
AN US
6. Threats to Validity
To limit the scope of our claims as well as to explain their utility, the first important fact to take into account is that this is an empirical study. Like most empirical studies, the validity of our results is subject to several threats (Basili 1005
et al., 1999).
M
Concerning the generalization of the findings, we obtained successful outcomes in companies of different size, with different organizational structures and different internal processes. Nevertheless, results presented in this paper
1010
ED
should not be taken as valid in the general sense. Although a Hawthorne effect is not possible, the environment in real industrial projects cannot be controlled to the same extent as in isolated research experiments. In this regard, it is impor-
PT
tant to note that each project is different and that team dynamics are complex. These factors make nearly impossible to control all the variables affecting suc-
CE
cess or failure of an IS project. Thus, there exist several threats concerning
1015
external factors that may affect our results. In particular, we do not exactly know whether the performance and quality improvements observed in the pi-
AC
lot experiments and the large-scale production studies can be attributed to the selection of more adequate coworkers or to many other possible factors. Furthermore, obtaining optimal results in the metrics analyzed in the pilot experiments
1020
and the large-scale production studies does not guarantee the assignment of the most effective team in a general sense. In the first case, our results must be in-
45
ACCEPTED MANUSCRIPT
terpreted as the formation of effective teams from parameters established by the company in each particular case. In the second case, they must be interpreted as an improvement in team assignment process of the company. Regarding the statistical conclusion validity, we analyze results derived from
CR IP T
1025
tests and pilot experiments performed in 39 different companies. This includes
thousands of simulations and real project assignments. Furthermore, we report detailed results of using the proposed framework during more than three years
in nine software factories belonging to a multinational corporation. This gives 1030
us enough statistics to consider that our results are statistically valid.
AN US
We assume that the most reliable source for information about compati-
bility between coworkers is their common participation in previous projects. In particular, we define compatibility as a function of objective global metrics related to success/performance of completed projects. This definition is also 1035
subject to some threats to validity. With this approach, we cannot guarantee that some relevant nodes and/or links from the network can be missing from the
M
source data, as it happens with other strategies for constructing compatibility networks for software professionals (Bachmann et al., 2010). If the database
1040
ED
of previous projects is not complete, multiple nodes or links can be missing. Similarly, predictions of the social networks also depend on the reliability of the metrics managed by the company. Depending on the particular data managed
PT
by the company, in our implementations we mainly used the following variables to measure project success/performance: on-time completion, within-budget
CE
completion, budget/effort deviation, system quality and customer satisfaction. 1045
These are the variables identified in literature as the main metrics for global IS projects success (DeLone and McLean, 2003; Espinosa et al., 2006). However,
AC
we also obtained successful outcomes with other metrics such lines of code per unit of effort or time taken to complete a task. Finally, with our definition, we only analyze intra-team relationships, when relationships among team members
1050
and the rest of the organization can also be relevant. In spite of these threats, our empirical results demonstrate that the abstraction that we propose is a good approach for the team allocation problem. 46
ACCEPTED MANUSCRIPT
Regarding the adaptability to different organizations, the proposed social networks can be built using data and metrics commonly managed by indus1055
trial companies. The employees’ technical and non-technical capabilities can be
CR IP T
captured from curricular data, which are commonly available in industrial companies. Personality traits and interpersonal relationships can be assessed with
different metrics. These metrics include estimators from the HR management domain that can be obtained from tools commonly available in medium/large1060
sized enterprises – e.g., 360-degree feedback surveys (London and Beatty, 1993) or competencies assessment (Robertson and Smith, 2001). If the company does
AN US
not have any of these tools, MBTI, BF and 16PF are simple personality tests
nowadays readily available online. These three tests are commonly used for personnel assessment in software engineering literature (e.g., see Hardiman (1997); 1065
Acu˜ na and Juristo (2004); Feldt et al. (2010); Andr´e et al. (2011); Licorish and MacDonell (2015)) and we successfully used them in some of our experiments. In this regard, we strongly recommend dealing with the help of professionals of
M
the organization’s HR department, since they know the mechanisms existing in the company to assess employees and their collaborations. This aims to adapt the framework to the specific characteristic of the company. For instance, in
ED
1070
some of our in-company experiments, the HR department proposed additional competencies measured by their own personality tests to be included in the
PT
social networks.
Concerning the applicability of the proposed framework, one of the main drawbacks is related to the data required to build and measure the social net-
CE
1075
works. It needs specific data related to projects (e.g., features, participants and success/performance metrics) and employees (e.g., detailed profiles or estima-
AC
tors of their social abilities). This need can make the framework not useful in some organizations. On one hand, although as we indicate above it can be
1080
easily adapted to the data managed by each company, not all the companies manage and keep track of the information required. In these cases, not only the team selection process must be implemented, but also additional processes to capture and keep historical data of projects and employees. On the other 47
ACCEPTED MANUSCRIPT
hand, the larger the company, the greater the volume of data to be captured 1085
and analyzed. In this regard, it is important to note that, although subjective data about working interpersonal relationships among coworkers can contribute
CR IP T
to improve teamwork, this information is not mandatory for the framework. In fact, objective teammate networks generated with a, in some sense, unreliable productivity measure as the effort deviation, provide more relevant information 1090
than those built with data subjective data.
7. Discussion
AN US
Many researchers and practitioners in software engineering hold to a socio-
technical view of IS projects (Sawyer, 2004; Luna-Reyes et al., 2005; Herbsleb, 2007; Tamburri et al., 2013). From this view, both the technical and social 1095
aspects of a project are crucial to performance and, therefore, should be taken into account to succeed. While multiple mechanisms to evaluate and improve
M
the technical dimension of a project are extensively applied in software industry, the mechanisms to control and manage the social dimension are still unclear. The framework presented in this paper takes advantage of some previously discussed elements and incorporates novel principles for globally assigning people
ED
1100
to IS projects from a socio-technical perspective. We propose the assessment of
PT
the candidates’ technical skills, experience and personality traits, but also their interpersonal relationships. For that, the framework leans on the information managed by the company to create personal profiles of each employee and to construct and measure different social networks which objectively and/or sub-
CE
1105
jectively characterize the collaborations among coworkers. This characterization can be made according to previous (successful or unsuccessful) collaborations
AC
as well as to a measure of the compatibility between teammates estimated as a function of their social skills. For companies where the required information
1110
is not available, we define possible mechanisms to collect it in a simple manner. We provide industrial insights into the framework applicability in different contexts, obtaining successful outcomes according to the parameters established
48
ACCEPTED MANUSCRIPT
by the corresponding company and with social networks constructed and measured with distinct strategies and metrics, which points out the generality of 1115
our proposal.
CR IP T
How to build efficient and effective teams is one of the most relevant questions in software engineering management. In practice, the decision of who participates in a project normally is on the managers’ hands. Although we also
found companies where the assignments were made by the HR department as 1120
a function of curricular data. Managers usually make decisions based on their
experience and subjective knowledge about the potential team members. In
AN US
our in-company experiments, we observed that in this scenario decision makers
tended to count on the same people and many new projects directly inherited teams of previously completed projects. This practice, that might be evident in 1125
small companies with a limited staff, also happened in medium- and large-sized companies and even in cases of recurrent failing teams. In many occasions, this was due, on one hand, to the lack of knowledge sharing among managers and, on
M
the other hand, to the lack of proper methods to evaluate personnel capabilities. Experience showed that maintaining teams composed by the same people was not always an effective practice, as successful teams for a given project might
ED
1130
not be appropriate for another one. Moreover, in the organizations with more complex structures, managers could be part of many different teams, which
PT
could be distributed across different locations or even countries. In these cases, they did not know every coworker daily work and did not always fully understand the contribution of a particular member to the team and his/her impact
CE
1135
on team dynamics. In these cases, the same as when they were made by the HR department, assignments were often made as a function of the candidates’ cur-
AC
ricular data. Therefore, we found a strong argument for a tool like the proposed framework, defining mechanisms to obtain wide-ranging information to form an
1140
accurate picture of coworkers’ performance and their interactions and, leaning on this information, infer adequate configurations for a specific project. In this regard, experiments in real-life industrial projects pointed out that managers, in general, appreciated the existence of such mechanisms, but they preferred to re49
ACCEPTED MANUSCRIPT
ceive the information and define their owns teams according to their experience 1145
(Table 3). The teammate networks provided managers with information that allowed
CR IP T
them to identify potential effective relationships among employees they did not know. They were also suitable to point to groups of teammates that usually work together in failing projects. In these networks, the team social score max1150
imizes the former relationships and minimizes the latter within new working teams. The pilot experiments and the long-term production studies showed
that forming teams based on this value is an effective strategy to create per-
AN US
forming teams. Since social-skill networks are not based on previous collaborations, the compatibility information they provided for decision makers was 1155
more difficult to understand than the one provided by the teammate networks. At a first glance, the managers were reluctant to accept these relationships and form teams based on them. However, the experiments also demonstrated the general good performance of these teams (cf. results of the individuals and the
1160
M
social-skills compatibility strategy in Table 4). From a more theoretical perspective, results obtained from the social-skill networks support the thesis that
ED
specific combinations of social skills allow two teammates to collaborate more effectively in an IS project. In this regard, we would like to highlight the particular case of some employees with limited social capabilities having difficulty
1165
PT
interacting with others (e.g., see Fig. 5). The analysis of our social networks permitted to identify coworkers with compatible social skills and the ability to
CE
adapt their working styles to these individuals. Beyond the benefits of forming more performing teams, these team assignments also increased the general personnel satisfaction. In addition to the direct benefits of creating more effective
AC
working teams, some of the staffing strategies implemented using the proposed
1170
social networks, e.g., the split strategy, had relevant positive medium/long-term benefits in exchange for a little short-term performance penalty (Fig. 8). This strategy is conceived to generate two performing teams from one existing performing team. In our experiments this was usually achieved (Table 4). Then, new productive teammate-teammate relationships provided an increased capac50
ACCEPTED MANUSCRIPT
1175
ity to face successfully new projects for the company. This is a highly valuable result for organizations with a high turnover rate. On the down side, the information required to build the proposed social
CR IP T
networks. To implement the framework in an industrial enterprise, we need: • A detailed knowledge of each employee’s technical skills and capabilities is mandatory to find potential candidates to participate in the project.
1180
• Access to historical productivity/quality metrics of each project is mandatory to build objective teammate networks.
AN US
• To take advantage of the social-skill networks, it is needed to know each employee’s social profile. This is likely the information more difficult to collect.
1185
Beyond the use of the proposed framework, our results emphasize the importance of how participants in an IS project collaborate, communicate and, in
M
general, work together as a team. In line with previous findings (Fitzpatrick and Askin, 2005; Acu˜ na et al., 2008; Adams and Anantatmula, 2010; Feldt et al., 2010), this points out that the social aspects can be as relevant as the technical
ED
1190
aspects for successfully completing a project (cf. results of the social-start and the technical-start strategy in Table 4), and supports the hypothesis that we can
PT
significantly improve our outcomes paying attention to interpersonal relationships when putting together a team. Social factors may enhance (or diminish) 1195
the sum of individual team members’ attitudes. In this sense, for instance,
CE
we found individuals with a low productivity that significantly increased their performance when moved to a team with different teammates; or projects suc-
AC
cessfully completed by teams not covering all the required technical skills but composed of people with compatible social skills. Obviously, people are not the
1200
only factor to be considered, and we cannot disregard other critical aspects for the project success such as the planning, project tasks, workload or scheduling, between others. The benefits of considering software development in industry as a socio51
ACCEPTED MANUSCRIPT
technical process were observed independently of the company size. Never1205
theless, they were less apparent in small-sized enterprises where the reduced number of employees, and usually also of projects, limited the number of possi-
CR IP T
ble assignments. In this regard, it is important to highlight that implementing a framework like ours to find an optimal trade-off between technical and so-
cial aspects does not have sense in this kind of companies. In addition to the 1210
limitation in the number of possible teams, managers in these companies often
know the rest of employees during daily work (Table 3). Although we have not enough statistics to raise a general conclusion, throughout our experiments we
AN US
established a threshold of 20 software professional to consider the framework helpful.
We have developed social networks characterizing interpersonal relationships
1215
within the team. These relationships are strongly related to group factors such as cohesion, conflicts or team climate, between others. However, social factors in an IS project are not limited to intra-team aspects. Of course, interpersonal re-
1220
M
lationships outside the working team can significantly impact on performance. Our teammate networks could incorporate additional elements characterizing
ED
interpersonal relationships outside project teams. For instance, Oh et al. (2006) found a correlation between team effectiveness and what they called the group social capital – the set of resources available to the team through its members
1225
PT
interpersonal relationships within the team and with other members of the organization. In companies with the ability to compute the group social capital
CE
proposed by Oh et al., it should be possible to develop a strategy for team assignments trying to balance this value and our team social score. Given the current trend towards globalization in the software development
AC
industry, we encourage software organizations to systematically incorporate re-
1230
lationships between coworkers to their team staffing processes. In this scenario, software production involves teams split over locations, countries, and even companies. Despite the increasing attention to distributed software projects, there is a limited understanding of team dynamics that makes these projects succeed or fail. In a distributed development team, there are many additional factors 52
ACCEPTED MANUSCRIPT
1235
influencing social interactions (Holmstrom et al., 2006; Espinosa et al., 2006). Between others, these include temporal distance (i.e., people working at different time zones which implies few overlapping hours), socio-cultural differences
CR IP T
(e.g., native language and/or culture differences) and geographic distance. Due to these communication issues, in a distributed project there exist less commu1240
nications among coworkers: each individual communicate with fewer coworkers
and interactions are less frequent (Olson and Olson, 2000). Even some au-
thors claim that communications are less effective (Herbsleb and Mockus, 2003). Therefore, distributed software engineering poses a challenge to the development
1245
AN US
process, demanding additional behavioral traits and new social abilities in the
interaction among software professionals. Results reported in this paper in the context of distributed projects are a good departing point to incorporate SNA to the team formation problem in software industry.
file
M
Appendix A. Proposal of competencies to include in the social pro-
This Appendix briefly describes the competencies and social skills measured
1250
ED
in our social-skill networks. As we indicate in the main text, these data can be obtained from different tools depending on the company. This implies that in different social-skills network the same value has different meanings and inter-
1255
PT
pretations.
• Written communication: Ability to write clearly and effectively in order
CE
to express information via a written channel
AC
• Verbal and non-verbal communication: Ability to give information and
1260
actively manage the communication process through oral channels.
• Active listening: Ability to accurately receive and interpret messages in the verbal communication process. • Teamwork: Ability to contribute to teams and improve their effectiveness through personal commitment. 53
ACCEPTED MANUSCRIPT
• Negotiation: Ability to talk or consult together with someone for the purpose of mutual arrangement using a win-win philosophy. • Sociability: Ability to get on well with a wide range of people and build
1265
CR IP T
long term trusting relationships.
• Empathy: Ability to see the world as another person, to share and understand another person’s feelings, needs, concerns and/or emotional state.
• Agreeableness: Ability to be kind, sympathetic, cooperative, warm and
AN US
considerate with other.
1270
• Leadership: Ability to guide and inspire others towards achieving goals. • Motivation: Ability to support and encourage individuals, so that they give their best.
• Adaptability: Ability to respond and adapt to changing circumstances and to manage, solve problems and provide solutions.
M
1275
ED
Appendix B. Questionnaires to evaluate teammates collaborations Information required to built subjective teammate networks can be extracted from an appraisal system previously implemented in the company. However,
1280
PT
these tools require of consistent organizational processes and they are not available in many companies. Furthermore, they usually require too many effort
CE
from subjects (e.g., in some of the appraisals already existing in our industrial partners each individual has to complete questionnaires with more than 200 questions). Then, in order to simplify the process, a group of occupational psy-
AC
chologists of one of our partners helped us define two possible questionnaites to
1285
collect subjective information regarding the teammates’ collaborations in a simple manner (Table B.6). One of the goals of the simulations of team assignments (Section 4.1) is comparing the results obtained with social networks generated on the basis of information provided by some existing tool in the company, and the ones obtained with questionnaires Q1 and Q2. 54
ACCEPTED MANUSCRIPT
ID
Question
Q1
What is the impact of Ana on your work? (1-5)
Questionnaire Q2
CR IP T
Questionnaire Q1
Question
Q1
How productive is your work with Ana? (0-10)
Q2
How effective is your work with Ana? (0-10)
Q3
How well does Ana communicate with you? (0-10)
Q4
How well does Ana collaborate with you? (0-10)
Q5
How well does Ana help you with your work? (0-10)
Q6
How respectfully does Ana treat you? (0-10)
Q7
How quickly does Ana respond your requests? (0-10)
Q8
How well does Ana handle criticism? (0-10)
Q9
How professionally does Ana behave? (0-10)
M
AN US
ID
ED
Table B.6: Questionnaires to collect subjective evaluations of the teammates’ collaborations.
Questionnaire Q1 was defined to capture with a single question whether a
1290
PT
subject considers positive, negative or neutral his/her professional relationship with a particular teammate (Ana in the example). Using a common strategy in this kind of evaluations, the only question of Q1 is expressed on a five-point
CE
Likert-type scale (1 = Extremely negative; 3 = Neither positive nor negative;
1295
and 5 = Extremely positive). In questionnaire Q2, subjects have to rate nine
AC
questions with a numerical value between 0 and 10. These questions refer to well-known key aspects in social interactions within a work team – e.g., communication or knowledge sharing (Lievens et al., 2002; Chen et al., 2004).
1300
When the network is built leaning on one of these questionnaires, connections
among potential candidates are scored with the mean value of the responds provided by the two candidates about their collaboration.
55
ACCEPTED MANUSCRIPT
Appendix C. Background of the empirical study Appendix C.1. Framework adaptation
1305
CR IP T
We made the framework adaptation along with professionals of the Spanish HR department. The company kept track, between others, of the following rel-
evant data for all the projects undertaken by the software factories included in
the study: type of project, area of knowledge, working team, working methodology, estimated workload, complexity, risk level, budget, effort/budget deviation, customer satisfaction, defect density and software complexity. We used some of
these data (type of project, area of knowledge, working methodology, estimated
AN US
1310
workload, complexity, risk level, team size, budget and geographical distribution) to generate the similarity vectors employed to characterize and compare projects. While effort/budget deviation, customer satisfaction, defect density and software complexity were used to generate and test objective teammate net1315
works based on different metrics. For the simulations and the pilot experiments,
M
we generated a base of knowledge with all the projects completed within the last three years in the nine software factories. Once the large-scale production
ED
study started up, this was weekly updated. Baselines for comparison in these experiments were also computed with the initial last-three-year data. These 1320
baselines did not have changed along the study.
PT
The company also implemented at organizational level two tools whose data were suited to construct our social networks: a 360-degree feedback appraisal and a competencies assessment system. Every year, each employee has to pro-
CE
vide self-evaluation and assess a group of coworkers including his/her supervisor,
1325
subordinates and some colleagues. Feedback collected of an individual in the
AC
360-degree assessment is weighted to calculate overall individual and collective metrics measuring different aspect related to team work. During the framework adaptation phase, we decided to test the use of two of these metrics. On one hand, a 0-to-10 quantitative value rating teammate-teammate relationships: 0
1330
corresponds to relationships with negative impact on team dynamics and 10 to relationships between coworkers with an extremely positive interplay, respec-
56
ACCEPTED MANUSCRIPT
tively. This metric was used to test 360-feedback-based teammate networks. On the other hand, a qualitative score that ranges from 0 (not developed at all) and 10 (highly developed) the level of development of an individual in different competencies which could be linked with the skills of our social profile (Appendix
CR IP T
1335
A). Competencies assessment, for its part, supports new employees recruitment. Between others, it assesses with a 0-to-10 quantitative value 11 competencies
that also could be used to characterize an employee’s social profile. When this information was not available for an employee included in the study, he/she had to complete the competencies assessment process.
AN US
1340
Appendix C.2. Results of the simulations and the pilot experiments During the simulations of team assignments leading up to the framework application in the nine software factories selected by the company, we tested the following social networks:
• Subjective teammate networks based on Q1, Q2 and 360-feedback.
M
1345
• Objective teammate networks based on effort/budget deviation, customer
ED
satisfaction, defect density and software complexity. • Teammate networks based on all the possible combinations of subjective
PT
and objectives metrics.
• Social-skill networks based on the information extracted from the existing
1350
360-feedback and competencies assessment tools, both individually and
CE
averaging the value obtained from each tool. Based on the HR department professionals’ experience, in the social profiles, and therefore in the nodes
AC
of the social-skill networks, we converted the 0-to-10 values obtained from
1355
these tools into a five-point qualitative scale: “very poor” ([0−2)), “poor” ([2 − 4)), “acceptable”, ([4 − 6)),“good” ([6 − 8)) and “excellent” ([8 − 10]). The simulations corroborated that the objective teammate networks based
on the effort deviation were the most reliable (Fig. 4). The subjective teammate networks based on Q1, Q2 and 360-feedback had poorer and equivalent results. 57
ACCEPTED MANUSCRIPT
1360
And when we combined effort deviation and a subjective metric, no matter which of them, results were very similar to the obtained with the effort deviation (94% vs. 93% of acceptance). Then, we decided to test these latter teammate
CR IP T
network in different pilot experiments. In particular, given the institutional mechanisms to keep updated the 360-feedback assessment, we used these data 1365
instead of questionnaires Q1 and Q2. References
Abreu, R., Premraj, R., 2009. How developer communication frequency relates
AN US
to bug introducing changes. Proceedings of the joint international and annual ERCIM workshops on Principles of software evolution (IWPSE) and software evolution (Evol) workshops - IWPSE-Evol 09 .
1370
Acu˜ na, S., Juristo, N., Moreno, A., 2006. Emphasizing human capabilities in software development. IEEE Software 23, 94–101.
M
Acu˜ na, S.T., G´ omez, M., Juristo, N., 2008. Towards understanding the relationship between team climate and software quality – a quasi-experimental study. Empir Software Eng 13, 401–434.
ED
1375
Acu˜ na, S.T., Juristo, N., 2004. Assigning people to roles in software projects.
PT
Softw: Pract. Exper. 34, 675–696. Adams, S.L., Anantatmula, V., 2010. Social and behavioral influences on team
CE
process. Proj Mgmt Jrnl 41, 89–98. 1380
Albert, R., Barab´ asi, A.L., 2002. Statistical mechanics of complex networks.
AC
Rev. Mod. Phys. 74, 47–97.
Andr´e, M., Baldoqu´ın, M.G., Acu˜ na, S.T., 2011. Formal model for assigning
1385
human resources to teams in software projects. Information and Software Technology 53, 259–275.
Bachmann, A., Bird, C., Rahman, F., Devanbu, P., Bernstein, A., 2010. The missing links: Bugs and bug-fix commits, in: Proceedings of the Eighteenth 58
ACCEPTED MANUSCRIPT
ACM SIGSOFT International Symposium on Foundations of Software Engineering, ACM, New York, NY, USA. pp. 97–106. Bagozzi, R.P., Dholakia, U.M., 2006. Open source software user communities: A
CR IP T
study of participation in linux user groups. Management Science 52, 10991115.
1390
Balthazard, P., Potter, R.E., Warren, J., 2004. Expertise, extraversion and group interaction styles as performance indicators in virtual teams. ACM SIGMIS Database 35, 41.
Bandinelli, S., Di Nitto, E., Fuggetta, A., 1996. Supporting cooperation in the spade-1 environment. IEEE Trans. Software Eng. 22, 841–865.
AN US
1395
Barreto, A., Barros, M.d.O., Werner, C.M., 2008. Staffing a software project: A constraint satisfaction and optimization-based approach. Computers & Operations Research 35, 3073–3089.
Basili, V., Shull, F., Lanubile, F., 1999. Building knowledge through families of
M
experiments. IEEE Trans. Software Eng. 25, 456473.
1400
ED
Bettenburg, N., Hassan, A.E., 2012. Studying the impact of social interactions on software quality. Empir Software Eng 18, 375–431. Bhattacharya, P., Iliofotou, M., Neamtiu, I., Faloutsos, M., 2012. Graph-based
PT
analysis and prediction for software evolution. 2012 34th International Conference on Software Engineering (ICSE) .
1405
CE
Bhattacharya, P., Neamtiu, I., 2010. Fine-grained incremental learning and multi-feature tossing graphs to improve bug triaging. 2010 IEEE International
AC
Conference on Software Maintenance .
Bird, C., Gourley, A., Devanbu, P., Gertz, M., Swaminathan, A., 2006. Mining
1410
email social networks. Proceedings of the 2006 international workshop on Mining software repositories - MSR 06 .
59
ACCEPTED MANUSCRIPT
Blaskovich, J.L., 2008. Exploring the effect of distance: An experimental investigation of virtual collaboration, social loafing, and group decisions. Journal of Information Systems 22, 27–46. Cataldo, M., Herbsleb, J.D., Carley, K.M., 2008. Socio-technical congruence.
CR IP T
1415
Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement - ESEM 08 .
Cataldo, M., Wagstrom, P.A., Herbsleb, J.D., Carley, K.M., 2006. Identification of coordination requirements. Proceedings of the 2006 20th anniversary
AN US
conference on Computer supported cooperative work - CSCW 06 .
1420
Charette, R.N., 2005. Why software fails [software failure]. IEEE Spectrum 42, 42–49.
Chen, G., Donahue, L.M., Klimoski, R.J., 2004. Training undergraduates to
tion 3, 27–40.
1425
M
work in organizational teams. Academy of Management Learning & Educa-
Chen, W.N., Zhang, J., 2013. Ant Colony Optimization for Software Project
ED
Scheduling and Staffing with an Event-Based Scheduler. IEEE Transactions on Software Engineering 39, 1–17.
PT
Crowston, K., Scozzi, B., 2002. Open source software projects as virtual organisations: competency rallying for software development. IEE Proc., Softw.
1430
CE
149, 3.
Damian, D., Helms, R., Kwan, I., Marczak, S., Koelewijn, B., 2013. The role
AC
of domain knowledge and cross-functional communication in socio-technical
1435
coordination. 2013 35th International Conference on Software Engineering (ICSE) .
DeLone, W.H., McLean, E.R., 2003. The delone and mclean model of information systems success: A ten-year update. Journal of Management Information Systems 19, 9–30.
60
ACCEPTED MANUSCRIPT
Drigas, A., Kouremenos, S., Vrettos, S., Vrettaros, J., Kouremenos, D., 2004. An expert system for job matching of the unemployed. Expert Systems with
1440
Applications 26, 217–224.
CR IP T
Ducheneaut, N., 2005. Socialization in an open source software community: A socio-technical analysis. Computer Supported Cooperative Work (CSCW) 14, 323368. 1445
El Emam, K., Koru, A.G., 2008. A replicated survey of it software project failures. IEEE Software 25, 84–90.
AN US
Espinosa, J.A., DeLone, W., Lee, G., 2006. Global boundaries, task processes
and is project success: a field study. Information Technology & People 19, 345–370. 1450
Feldt, R., Angelis, L., Torkar, R., Samuelsson, M., 2010. Links between the personalities, views and attitudes of software engineers. Information and
M
Software Technology 52, 611–624.
Fenton, N.E., Neil, M., 1999. A critique of software defect prediction models.
1455
ED
IEEE Transactions on Software Engineering 25, 675–689. Fitzpatrick, E.L., Askin, R.G., 2005. Forming effective worker teams with multi-
PT
functional skill requirements. Computers & Industrial Engineering 48, 593– 608.
CE
Gorla, N., Lam, Y.W., 2004. Who should work with whom? Communications of the ACM 47, 79–82.
Grindrod, P., Parsons, M.C., Higham, D.J., Estrada, E., 2011. Communicability
AC
1460
across evolving networks. Phys. Rev. E 83.
Hardiman, L., 1997. Personality types and software engineers. Computer 30, 10–10.
61
ACCEPTED MANUSCRIPT
Herbsleb, J., Mockus, A., 2003. An empirical study of speed and communication in globally distributed software development. IEEE Trans. Software Eng. 29,
1465
481494.
CR IP T
Herbsleb, J.D., 2007. Global software engineering: The future of socio-technical coordination. Future of Software Engineering (FOSE 07) .
Holmstrom, H., Conchuir, E., Agerfalk, P., Fitzgerald, B., 2006. Global software
development challenges: A case study on temporal, geographical and socio-
1470
cultural distance. 2006 IEEE International Conference on Global Software
AN US
Engineering (ICGSE06) .
Katz, L., 1953. A new status index derived from sociometric analysis. Psychometrika 18, 39–43. 1475
Lakhanpal, B., 1993. Understanding the factors influencing the performance of software development groups: An exploratory group-level analysis. Informa-
M
tion and Software Technology 35, 468–473.
Lappas, T., Liu, K., Terzi, E., 2009. Finding a team of experts in social net-
ED
works. Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD 09 .
1480
PT
Li, C.T., Shan, M.K., 2010. Team formation for generalized tasks in expertise social networks. 2010 IEEE Second International Conference on Social
CE
Computing .
Licorish, S.A., MacDonell, S.G., 2014. Understanding the attitudes, knowledge sharing behaviors and task performance of core developers: A longitudinal
AC
1485
study. Information and Software Technology 56, 1578–1596.
Licorish, S.A., MacDonell, S.G., 2015. Communication and personality profiles of global software developers. Information and Software Technology 64, 113– 131.
62
ACCEPTED MANUSCRIPT
1490
Lievens, F., van Dam, K., Anderson, N., 2002. Recent trends and challenges in personnel selection. Personnel Review 31, 580–601. Linberg, K.R., 1999. Software developer perceptions about software project
CR IP T
failure: a case study. Journal of Systems and Software 49, 177–192.
London, M., Beatty, R.W., 1993. 360-degree feedback as a competitive advantage. Hum. Resour. Manage. 32, 353–372.
1495
Luna-Reyes, L.F., Zhang, J., Ramn Gil-Garca, J., Cresswell, A.M., 2005. Information systems development as emergent socio-technical change: a practice
AN US
approach. Eur J Inf Syst 14, 93105.
MacQueen, J.B., 1967. Some methods for classification and analysis of multivariate observations, in: Proc. of the fifth Berkeley Symposium on Mathematical
1500
Statistics and Probability, University of California Press. pp. 281–297. Madey, G., Freeh, V., Tynan, R., 2002. The open source software develop-
1505
ED
Proceedings , 1806–1813.
M
ment phenomenon: An analysis based on social network theory. AMCIS 2002
Meneely, A., Williams, L., 2011. Socio-technical developer networks. Proceeding of the 33rd international conference on Software engineering - ICSE 11 .
PT
Meneely, A., Williams, L., Snipes, W., Osborne, J., 2008. Predicting failures with developer networks and social network analysis. Proceedings of the
CE
16th ACM SIGSOFT International Symposium on Foundations of software engineering - SIGSOFT 08/FSE-16 .
1510
AC
Moreno, J.L., 1934. Who shall survive?: A new approach to the problem of human interrelations. .
Nagappan, N., Murphy, B., Basili, V., 2008. The influence of organizational
1515
structure on software quality. Proceedings of the 13th international conference on Software engineering - ICSE 08 .
63
ACCEPTED MANUSCRIPT
Newman, M.E.J., 2001. Scientific collaboration networks. ii. shortest paths, weighted networks, and centrality. Phys. Rev. E 64. Ngo-The, A., Ruhe, G., 2009. Optimized resource allocation for software release
1520
CR IP T
planning. IEEE Transactions on Software Engineering 35, 109–123.
Oh, H., Labianca, G., Chung, M.H., 2006. A multilevel model of group social capital. Academy of Management Review 31, 569–582.
Olson, G., Olson, J., 2000. Distance matters. Human-Computer Interaction 15,
AN US
139–178.
Otero, L.D., Centeno, G., Ruiz-Torres, A.J., Otero, C.E., 2009. A systematic approach for resource allocation in software projects. Computers & Industrial
1525
Engineering 56, 1333–1339.
Otero, L.D., Otero, C.E., 2012. A fuzzy expert system architecture for capability assessments in skill-based environments. Expert Systems with Applications
Robertson, I.T., Smith, M., 2001. Personnel selection. Journal of Occupational
ED
1530
M
39, 654–662.
and Organizational Psychology 74, 441–472. Sawyer, S., 2001. Effects of intra-group conflict on packaged software develop-
PT
ment team performance. Inform Syst J 11, 155–178.
CE
Sawyer, S., 2004. Software development teams. Commun. ACM 47, 9599. 1535
Schuler, D., Zimmermann, T., 2008.
Mining usage expertise from version
AC
archives. Proceedings of the 2008 international workshop on Mining software repositories - MSR 08 .
Shin, Y., Meneely, A., Williams, L., Osborne, J.A., 2011. Evaluating com-
1540
plexity, code churn, and developer activity metrics as indicators of software vulnerabilities. IEEE Transactions on Software Engineering 37, 772–787.
64
ACCEPTED MANUSCRIPT
e Silva, L.C., Costa, A.P.C.S., 2013. Decision model for allocating human resources in information system projects. International Journal of Project Management 31, 100–108.
CR IP T
Sowe, S.K., Stamelos, I., Angelis, L., 2008. Understanding knowledge sharing activities in free/open source software projects: An empirical study. Journal
1545
of Systems and Software 81, 431–446.
Sun, J., Tang, J., 2011. A survey of models and algorithms for social influence analysis. Social Network Data Analytics , 177214.
AN US
Surian, D., Liu, N., Lo, D., Tong, H., Lim, E.P., Faloutsos, C., 2011. Rec-
ommending people in developers collaboration network. 2011 18th Working
1550
Conference on Reverse Engineering .
Tamburri, D.A., Lago, P., Vliet, H.v., 2013. Organizational social structures for software engineering. ACM Computing Surveys 46, 135.
M
Tsai, H.T., Moskowitz, H., Lee, L.H., 2003. Human resource selection for software development projects using taguchis parameter design. European Jour-
1555
ED
nal of Operational Research 151, 167–180. Tuckman, B.W., Jensen, M.A.C., 1977. Stages of small-group development
PT
revisited. Group & Organization Management 2, 419–427. Turley, R.T., Bieman, J.M., 1995. Competencies of exceptional and nonexceptional software engineers. Journal of Systems and Software 28, 19–38.
CE
1560
Wolf, T., Schroter, A., Damian, D., Nguyen, T., 2009. Predicting build failures
AC
using social network analysis on developer communication. 2009 IEEE 31st International Conference on Software Engineering .
Xu, J., Christley, S., Madey, G., 2006. Application of social network analysis to
1565
the study of open source software. The Economics of Open Source Software Development , 247–269.
65
ACCEPTED MANUSCRIPT
Xu, R., Wunsch, D., I., 2005. Survey of clustering algorithms. Neural Networks, IEEE Transactions on 16, 645–678. Yang, H.L., Tang, J.H., 2004. Team structure and team performance in is
CR IP T
development: a social network perspective. Information & Management 41,
1570
335–349.
Zhu, H., Zhou, M., Seguin, P., 2006. Supporting software development with
roles. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Sys-
AC
CE
PT
ED
M
AN US
tems and Humans 36, 1110–1123.
66
CR IP T
ACCEPTED MANUSCRIPT
AN US
1575
Roberto Latorre received the B.S. degree in computer engineering and the Ph.D. in computer science and telecommunications from Universidad Aut´ onoma de Madrid, Madrid, Spain, in 2000 and 2008, respectively. He has an extensive experience in software industry, where he worked with multiple companies as IT 1580
architect and software developer. Since 2002 he has been a professor at Escuela
M
Politcnica Superior at Universidad Autnoma de Madrid, where he currently is
AC
CE
PT
ED
Profesor Contratado Doctor.
1585
Javier Su´ a]rez is a software engineer and a researcher at Kymatic Soluciones Informticas. He received the I B.S. degree in computer engineering from Universidad Politcnica de Madrid. He is an expert in Speech technologies and software development.
67
Measuring social networks when forming information system project teams Roberto Latorre, Javier Suarez ´ PII: DOI: Reference:
S0164-1212(17)30208-X 10.1016/j.jss.2017.09.019 JSS 10041
To appear in:
The Journal of Systems & Software
Received date: Revised date: Accepted date:
7 July 2016 4 July 2017 21 September 2017
Please cite this article as: Roberto Latorre, Javier Suarez, Measuring social networks when ´ forming information system project teams, The Journal of Systems & Software (2017), doi: 10.1016/j.jss.2017.09.019
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
ACCEPTED MANUSCRIPT
Highlights • Socio-technical framework for team assignments based on social-network
CR IP T
analysis • Social networks estimating compatibility according to previous collaborations
• Social networks estimating compatibility according to social skills
• Industrial insights into the applicability and usability in real-life case-
AC
CE
PT
ED
M
AN US
studies
1
ACCEPTED MANUSCRIPT
Measuring social networks when forming information system project teams
a Escuela
CR IP T
Roberto Latorrea,∗, Javier Su´ arezb Polit´ ecnica Superior, Universidad Aut´ onoma de Madrid, 28049, Madrid, Spain b Kymatic Soluciones Inform´ aticas, Madrid, Spain
Abstract
AN US
Despite the advances in the project management field, little is known about how creating performing software teams in a systematical and repeatable way.
The technical dimension is not enough to achieve this. Software development is a complex collaborative process where people and interpersonal relationships – i.e., how people interact, behave and organize significantly – influence the project success. We define a framework to assign people to projects from this
M
socio-technical perspective. Social networks characterizing the interplay between teammates are built and analyzed to predict productive collaborations
ED
and identify adequate team-members depending on the needs of the organization and the kind of project. A noteworthy novelty of these social networks is that they estimate compatibility between coworkers according to previous
PT
collaborations, but also to individuals’ social skills. This allows analyzing the compatibility between people who have not worked together before. We present results of using the proposed framework in a multinational corporation during
CE
a more-than-two-year period. Our in-company experiments emphasize that we can significantly improve the expected outcomes characterizing and measuring
AC
the social interaction between coworkers. Social aspects discussed here may be highly relevant in the context of distributed software engineering, since it implies new challenges in the interplay among coworkers.. ∗ Corresponding
author Email address: [email protected] (Roberto Latorre)
Preprint submitted to Journal of Sytems and Software
September 22, 2017
ACCEPTED MANUSCRIPT
Keywords: Team member allocation, Team formation problem, Social skills, Social network analysis, Interpersonal relationships, Collaboration network
CR IP T
1. Introduction
Information System (IS) projects in the industrial environment are commonly considered to be successfully completed when they are delivered on time,
on budget and with the appropriate level of quality. With this definition in 5
mind, multiple factors can influence the temporal, financial and quality goals
AN US
of industrial IS projects during their lifetime (Charette, 2005). For instance,
underestimated projects commonly fail: the ones delivered with the appropriate level of quality, are often completed late and/or with over cost; while the released to meet the budget usually do not meet system requirements and are 10
unreliable systems. Another fundamental factor to complete the project successfully is an appropriate team composition (Fenton and Neil, 1999; Linberg,
M
1999; El Emam and Koru, 2008). The decision of who participates in the project has therefore a critical impact on success. Nevertheless, experience shows that, to improve the expected outcomes, the project team usually takes a backseat at the planning stage and we often focus the attention on other factors, such
ED
15
as determining the project tasks and workload, and optimizing the time-line
PT
scheduling for developing the system. Assuming the existence of such mechanisms to control these other factors, this paper addresses the team formation problem in the context of real-life industrial IS projects. Forming effective and efficient working teams is a complicated endeavor be-
CE
20
cause multiple factors influence the effective performance and productivity of a
AC
group of people working together. Traditional methods for selecting people to perform a given task are based on the candidates’ estimated performance and/or the best fitting among the individuals’ abilities and the set of abilities required
25
for the task to perform (Lievens et al., 2002). In the context of software engineering, the need of highly technical specialized skills for developing software has been subject of discussion for a long time. Thus, human resources (HR) allo-
3
ACCEPTED MANUSCRIPT
cation in software industry is mainly driven by the candidates’ know-what and know-how, i.e., technical knowledge or expertise acquired through experience 30
or formal training. For instance, these are the basis of traditional role-based
CR IP T
staffing strategies (Bandinelli et al., 1996; Zhu et al., 2006) that individually assign people to development roles with the goal of identifying the candidates with the necessary technical skills to complete each development activity with
the minimum training time and/or cost (Tsai et al., 2003; Barreto et al., 2008; 35
Otero et al., 2009; Ngo-The and Ruhe, 2009; Chen and Zhang, 2013). However, a team is more than the sum of the individual team members’ attributes. Different
AN US
individuals have different working styles and preferences, and simply bringing
together the best candidates as a function of their estimated performance or technical abilities does not ensure creating a high-performing team (Fitzpatrick 40
and Askin, 2005; Acu˜ na et al., 2008; Adams and Anantatmula, 2010; Feldt et al., 2010). Software development is an intense collaborative process (Sawyer, 2004; Luna-Reyes et al., 2005) where cooperation and social interactions among team-
M
mates (Lakhanpal, 1993; Sawyer, 2001; Gorla and Lam, 2004), or even among people belonging to different working teams (Oh et al., 2006), can significantly affect to project success. From this socio-technical perspective, an important
ED
45
step to enhance productivity and quality outcomes of IS project teams is to be able to identify compatible coworkers to improve social interactions within the
PT
team and avoid, or at least minimize, conflicts. A commonly accepted view in psychology is that the personality of an individual can be described by sets of personality traits, i.e., fixed sets of patterns
CE
50
defining the way people tend to think, feel and behave in different situations. This concept is extensively used in the HR management domain (Robertson
AC
and Smith, 2001) trying to estimate the level of development of an individual in different generic (skills or attributes of the personal behavior that can be
55
considered as a behavioral feature) and technical competencies (particular skills specifically related to a specific job or task, e.g., business analysis or project management). Software engineering research efforts in this line are directed toward identifying the required features for each development task or role (Acu˜ na 4
ACCEPTED MANUSCRIPT
and Juristo, 2004; Balthazard et al., 2004; Acu˜ na et al., 2006; Licorish and 60
MacDonell, 2015) and finding the best matching among people and these features (Drigas et al., 2004; Otero et al., 2009; Otero and Otero, 2012). Related to
CR IP T
our work, some authors claim that some competencies can explain and predict how a person interact and collaborate with other people in such a way that
variations of these behavioral and social traits can significantly influence the 65
dynamics and effective productivity of a development team (Blaskovich, 2008;
Adams and Anantatmula, 2010; Feldt et al., 2010). Under this view, it has been proposed that creating productive software development teams requires an ade-
traits (Turley and Bieman, 1995).
AN US
quate balance across team members’ technical skills, experience and personality
In spite of the increasing interest on non-technical and social aspects for the
70
research community, in practice, HR allocation techniques in software industry usually only respond to who does what. Assignments are typically made to tasks or roles individually and issues related to how individuals are assembled to form
75
M
the team are often disregarded. We propose here a general socio-technical framework to form teams globally. The motivation behind this framework is finding
ED
an optimal trade-off between technical and social aspects in order to have a successful project. The goal is to form a team of experts with the required technical knowledge, but also collaborating as a team in an effective manner. Our
80
PT
proposal benefits of some of the socio-technical aspects previously discussed in the research literature. During the evaluation of the technical dimension, we
CE
look for an optimal fitting among the technical needs and the candidates’ skills. During the evaluation of the social dimension, the aim is to build and analyze networks of software professionals characterizing and measuring their interper-
AC
sonal relationships. These social networks allow us to study the compatibility
85
among people who are involved in the project and identify those individuals that, working together, may potentially have an optimal joint-performance to complete the project. A relevant novelty of the proposed framework is that it does not only measure networks connecting software professionals, but also connecting the social skills they use to communicate, persuade or interact with other 5
ACCEPTED MANUSCRIPT
90
people (e.g., abilities such as verbal and non-verbal communication, empathy or leadership). Results obtained in multiple in-company experiments suggest that characterizing and measuring the interplay between coworkers when forming
CR IP T
software project teams can significantly improve the expected outcomes. We would like to emphasize that the proposed framework is general in the 95
sense that it can be apply to different kind of companies. The framework defines rules to evaluate and align relevant technical and social aspects affecting perfor-
mance of a software development team, but the information required to build
and measure the proposed social networks (e.g., project performance estimators
100
AN US
or level of knowledge/expertise in a given skill) can be obtained from different sources and/or be estimated using different metrics. This allows adapting the
framework to companies with different organizational structures and processes: from small companies to multinational corporations.
M
2. Related work
2.1. Socio-technical approaches to software development Interpersonal relationships are associations among two or more people formed
ED
105
in the context of a social interaction, e.g., friendship, affinity or communications. In this work, we are interested in collaborations between software professionals
PT
within a development team. Multiple types of personal and professional interpersonal relationships arise between coworkers in an IS project. Over the last 110
years, there is an increasing interest of these relationships on development tasks
CE
and literature on software engineering recognizes the importance of social aspects in the software development process (e.g., see Tamburri et al. (2013)). A
AC
non-trivial IS project can be considered a socio-technical endeavor where both technical and social factors influence the development process (Sawyer, 2004;
115
Luna-Reyes et al., 2005; Herbsleb, 2007). The technical component consists of the technical properties of the software to be developed, the development techniques or the technology employed. The social component includes the organization and the individuals involved in the development process, their attitudes
6
ACCEPTED MANUSCRIPT
and behaviors. Some authors claim that socio-technical congruence is a good in120
dicator of organizations performance in carrying out IS projects (Cataldo et al., 2006, 2008). Other investigations suggest that interpersonal relationships be-
CR IP T
tween coworkers determine the likelihood of development teams to be cohesive and productive (Yang and Tang, 2004; Adams and Anantatmula, 2010). 2.2. Social network analysis and the staffing problem
Socio-technical approaches to software development demand to assess the
125
relationships among coworkers in order to find compatible teammates during
AN US
the personnel assignment process. However, when putting together a team, it is difficult to determine who can efficiently work with whom. Compatibility can be given by technical (e.g., compatible programmers use a similar programming 130
style and naming strategy) and/or social aspects (e.g., fluidly communications and knowledge sharing between teammates).
Interpersonal relationships can be represented and analyzed with a social
M
network (Moreno, 1934), i.e., a graph where each node corresponds to an individual and links represent relationships between them. Social Network Analysis (SNA) has been applied in multiple scenarios from different scientific areas (Al-
ED
135
bert and Barab´ asi, 2002). In the context of software engineering, it emerges as a tool for capturing the underlying interaction processes within develop-
PT
ment teams and studying different aspects of the collaboration and organization among software professionals. Thus, SNA has been used, for instance, to iden140
tify top developers and their attitudes and knowledge sharing behaviors (Sowe
CE
et al., 2008; Licorish and MacDonell, 2014, 2015) or to predict failures and vulnerabilities in software products (Meneely et al., 2008; Wolf et al., 2009; Shin
AC
et al., 2011). SNA is also extensively used to analyze the Open Source Software development process (Crowston and Scozzi, 2002; Madey et al., 2002; Duche-
145
neaut, 2005; Bagozzi and Dholakia, 2006). More related to our work is the investigation by Lappas et al. Lappas et al.
(2009), who apply SNA to the problem of assigning people to a working task. Given a pool of candidates with different skills, a social network that captures 7
ACCEPTED MANUSCRIPT
the compatibility among these individuals and a particular task that requires 150
a set of skills, Lappas et al. (2009) addresses the problem of identifying the individuals best suited to perform the given task. Extensions of this algorithm
CR IP T
have been proposed later, for example, to be applied when a skill requires more than one expert (Li and Shan, 2010).
All these works provide a useful theoretical framework successfully tested 155
with scientific-collaboration graphs (Newman, 2001). Nevertheless, when you try to apply this framework to real IS projects in industry, you find some lim-
itations. The main issue is that Lappas et al. look for groups of individuals
AN US
that “together” have the required skills regardless of who displays each skill.
This is not a real scenario in industry, where projects demand individuals with 160
a specific set of skills. Furthermore, Lappas et al. focus on the network algorithms and do not address the construction of the social network capturing the compatibility among coworkers. In their tests, they build scientific-collaboration networks from the DBLP bibliography database where two scientists have a pre-
165
M
vious successful collaboration when they appear in the database as coauthors of at least one manuscript. In the following section, we review different common
ED
approaches to build social networks of software professionals. 2.3. Strategies to build social networks in software engineering
PT
Literature describes three main strategies to build networks representing social relationships among software professionals. The most common strategy is 170
based on the correlation observed between work performance and effective com-
CE
munications between coworkers (e.g., see Yang and Tang (2004)). Coworkers in an IS project can interact verbally and/or textually through different chan-
AC
nels. Investigations that use communications to identify relationships among coworkers found several problems in tracing and characterizing verbal commu-
175
nications, even when they were computer-mediated. Then, most of these works build collaboration networks using different heuristics to mine knowledge from mailing lists (Ducheneaut, 2005; Bird et al., 2006; Abreu and Premraj, 2009), bug tracking tools (Bhattacharya and Neamtiu, 2010; Bhattacharya et al., 2012) 8
ACCEPTED MANUSCRIPT
or version control systems (Bird et al., 2006; Schuler and Zimmermann, 2008; 180
Meneely and Williams, 2011; Bettenburg and Hassan, 2012), between others. Thus, they establish a connection between individuals that interchange a given
CR IP T
number of emails, commit the same file in a software repository or comment the same bug in a bug tracking system. Some studies show that relevant links
can be missing from the data in the social networks constructed with these 185
strategies (Bachmann et al., 2010).
A second strategy consists of using the hierarchical structure of the company
to identify the edges of the social network. This strategy has been used to
AN US
study how organizational structures interfere with task assignments and their influence on communications (Damian et al., 2013) and the resulting software 190
quality (Nagappan et al., 2008).
A last strategy consists of building social networks connecting participants in the same project (Madey et al., 2002; Xu et al., 2006). Note that these networks are equivalent to the scientific-collaboration networks mentioned above.
195
M
An interesting work where this strategy is used to find compatible programmers is Surian et al. (2011). In this work, they build collaboration networks con-
ED
taining three types of nodes – developers, projects and project properties – and define a metric to quantify the compatibility among developers as a function of the projects they have worked together before and their properties. The idea
200
PT
inspiring this metric is that two teammates working in the same project have common interests and skills. Surian et al. report “reasonable” results based on
CE
an input developer and snapshots from Sourceforge.Net. However, they do not address the construction of teams composed of people with different skills and
AC
roles.
205
These three strategies have been successfully applied to different relevant
problems in software engineering. Nevertheless, as we mention above, from our perspective, they have some limitations to be applied to the team allocation problem in industry. In addition to these limitations, we consider crucial to establish how positive or negative the interactions among coworkers are, and none of them does it. For instance, non-productive teammates can interchange 9
ACCEPTED MANUSCRIPT
210
a great amount of emails or systematically participate in failing projects. Social networks built in the context of the team formation problem must capture this information to avoid including individuals with negative relationships in future
CR IP T
working teams. To solve the drawbacks raised, we use a different approach. As Surian et al. (2011), we consider that the most reliable source for informa215
tion about compatibility between coworkers is their participation in previous projects, but we introduce a joint-performance estimator weighting each con-
nection in the social network (see Section 3.2 for details). Barreto et al. (2008) defines a similar “productivity modifier” characterizing the productivity of an
220
AN US
individual. The difference between this productivity score and ours is that Bar-
reto et al. assign the scores to individuals – nodes in terms of the social network – and we assign them to interpersonal relationships – i.e., connections. Moreover, we introduce an additional social score to analyze the collective compatibility of a group of software professionals, not only the compatibility between two
225
M
particular individuals.
3. Framework definition
ED
Our work departs from the hypothesis that the expected outcomes of new IS projects can be significantly improved if the project team is not only selected as a
PT
function of technical capabilities, but also considering interpersonal relationships within the team. Under this view, the goal of the proposed socio-technical 230
framework is maximizing the success probability of new teams finding an optimal
CE
trade-off between technical and social aspects. To achieve this goal, we build different compatibility social networks for employees and new candidates. The proposed framework is generic in the sense that it can be adapted to the needs
AC
of different type of companies. In particular, different strategies and metrics
235
can be used to construct and measure the different social networks proposed. Let assume a forming-storming-norming-performing team life cycle (Tuck-
man and Jensen, 1977), where a new team advances from the forming to the performing stage to become an effective team. The simple and intuitive idea behind
10
ED
M
AN US
CR IP T
ACCEPTED MANUSCRIPT
PT
Figure 1: Schematic representation of the proposed socio-technical framework.
our proposal is putting together compatible teammates with the required technical skill in order to facilitate this transition and minimize the storming stage
CE
240
impact. By compatible, here we mean people that working together reach a high joint-performance. To predict the most adequate project team composition, the
AC
framework analyzes multiple variables. Depending on the company, this includes different project metrics, the employees’ technical and non-technical capabili-
245
ties, and/or the social interactions within the team, between others. To briefly provide an intuitive view of this analysis, Fig. 1 illustrates schematically the proposed framework:
11
ACCEPTED MANUSCRIPT
1. First, we identify the candidates best suited to participate in the project (Section 3.1.1). When possible, these will be the members of the different teams to evaluate and the nodes of our social networks. Suitability is
250
CR IP T
determined through the matching between the attitudes required for each task or role (Pi , identified with a different color in Fig. 1) and the available human resources’ attitudes.
2. Connections in the social networks are established as a function of previous collaborations, i.e., projects where the candidates worked together before.
255
Then, we extract the information of similar projects from a database of
AN US
completed projects (Section 3.1.2).
3. Once potential candidates and projects to rate their previous collaborations have been found, we build two kind of compatibility networks. On one hand, social networks characterizing the interplay between team-
260
mates in terms of previous productive and non-productive relationships (Section 3.2.1). On the other hand, social networks connecting candidates
M
with compatible and incompatible social skills (Section 3.2.2). Individuals in the social-skill networks can be connected even when they have not worked together before.
ED
265
4. Finally, the potential candidates are combined using different strategies (Section 3.3), and the resulting teams are analyzed with SNA to estimate
PT
the most effective assignments for new projects or to include new members in existing teams (Section 3.4).
As we mention above, this general schema can be adapted to the specific
CE
270
context of the company where the framework is applied. In the following sections, we describe possible strategies to construct and measure our teammate
AC
and social-skill networks. 3.1. Identification of nodes and links
275
3.1.1. Find potential candidates Nodes of our social networks are the potential candidates to participate in the project. Like in any other strategy for assigning people to a working task, 12
ACCEPTED MANUSCRIPT
potential candidates are identified as a function of the fitting among the candidate’s abilities and the set of required abilities. To group these abilities, we use 280
the concept of profile – defined as the set of specific attitudes for each task or
CR IP T
role in the project. A profile can include different type of attributes: technical and non-technical skills, personality traits, limiting individual factors (e.g., trav-
eling availability) and personal preferences. Note that this requires identifying
and grading every candidate’s technical and non-technical skills and personality 285
traits (see threats to validity in Section 6). While some existing decision support
systems for assigning people to IS project teams already incorporate the candi-
AN US
dates’ technical and non-technical skills (Andr´e et al., 2011; e Silva and Costa,
2013), considering limiting individual factors and personal preferences is a novelty of our proposal. The need of taking into account limiting individual fac290
tors is obvious for real-life industrial projects. Regarding personal preferences, experience indicates that when an employee is continuously assigned to tasks he/she does not like to perform and is qualified to perform a higher-level job,
M
at the end, a decreasing performance likely takes place. Thus, a possible sample of web developer’s profile is Pdeveloper = {java, TDD, german competencies, stress tolerance, not traveling availability, distributed project}.
ED
295
In the simpler case, profiles only include technical skills, since this is the minimal information to identify those candidates with the required technical
PT
knowledge. The inclusion of other elements is not mandatory and depends on the company and the information it manages. To match the required profiles and the available human resources’ profiles,
CE
300
these are considered fuzzy sets in order to manage uncompleted associations between the required and the actual level of knowledge/expertise in a given skill
AC
or capability (Drigas et al., 2004). When there is not a perfect match between profiles and candidates, we use the best-fitted resource (BFR) methodology
305
proposed in Otero et al. (2009). BFR analysis quantifies how prior knowledge in other skills contributes to the learning of a given skill. This analysis allows us to build effectively teams even when the optimal profiles are not completely available. 13
ACCEPTED MANUSCRIPT
3.1.2. Find similar projects We measure networks built considering the projects where individuals have
310
worked before. Multiple variables make an IS project to be unique. Therefore,
CR IP T
the generation of our social networks is driven only by data of similar projects to the one for which the new team is being formed. Similarity between projects is
estimated using a clustering algorithm. These algorithms allow us to compute 315
and compare similarity vectors between projects. In our implementations we perform a k-means clustering analysis (MacQueen, 1967; Xu and Wunsch, 2005). The identification of similar projects extensively depends on the data stored
AN US
in the corresponding database of projects (see threats to validity in Section 6).
Results reported in this paper consider vectors with the following features to 320
determine similarity among projects: type of project, area of knowledge, working methodology, estimated workload, complexity, risk level, team size, budget and geographical distribution. We would like to highlight the latter, since geographical distribution usually is disregarded during the project team compo-
involves teams split over locations and countries.
ED
325
M
sition (beyond language competencies). However, current software production
3.2. Building compatibility networks 3.2.1. Teammate networks
PT
A teammate network – e.g., see Fig. 2 – is social network that consists of nodes representing individuals (IT M = {I1 , ..., In }) and links connecting them and quantifying their expected compatibility with a compatibility score (RT M =
CE
330
{CT MIi Ij }). The more compatible a teammate-teammate relationship in the network, the higher the estimated joint-performance of the corresponding indi-
AC
viduals. Links between teammates are bidirectional and CT MIi Ij = CT MIj Ii . If two individuals are not connected in the network, we have no evidence re-
335
garding their compatibility. This is different from being incompatible. Although the previous definition is conceptually simple, it is not easy to
determine a priori the compatibility among coworkers. Similar approaches to ours described in the related work assume that two individuals are compatible 14
AN US
CR IP T
ACCEPTED MANUSCRIPT
Figure 2: Example of teammate network. Scores in the links indicate how compatible the corresponding interpersonal relationship is. For simplicity in the graphical representation,
M
positive relationships are plotted in green color; and negative in red.
and have a successful collaboration simply if they previously worked in the same project. From our perspective, this and other similar heuristics do not capture
ED
340
the compatibility among coworkers in a real-life scenario, because they do not measure whether interpersonal relationship were actually positive. To solve
PT
these drawbacks, we rate connections in our teammate networks using two types of metrics: subjective metrics based on the teammates’ personal opinion about their interactions, and objective metrics based on the collective performance
CE
345
of previous teams. Thus, subjective teammate networks are constructed as a function of a subjective metric; and objective teammate networks as a function
AC
of an objective metric.
350
We would like to emphasize again that the proposed framework is defined
to be applicable in different contexts. In this regard, the compatibility score of a teammate-teammate relationship can be given by different subjective or objective metrics depending on the particular company. It can be expressed
15
ACCEPTED MANUSCRIPT
qualitatively or quantitatively, and obviously, the possible values and the interpretation of these values vary depending on the strategy and metrics used 355
to build the corresponding teammate network. The only relevant point is the
CR IP T
order relation between the possible values. In the case of quantitative data, we assume that the higher value corresponds to the more compatible relationships.
If this is not the case, it can be easily changed. To simplify the analysis, qualitative data are converted into numerical values maintaining the order relation. 360
For example, in a “positive-negative-neutral” Likert scale, we can assign 1 to “negative”, 2 to “neutral” and 3 to “positive”.
AN US
Subjective teammate networks are based on the idea that only teammates
know how they interact during daily work. Then, connections in these networks are rated according to the assessment the teammates themselves supply. 365
Depending on the context of the company, different strategies can be used to achieve this. The HR management domain provides us with different tools to evaluate interactions among coworkers. In particular, we highlight here the so-
M
called 360-degree feedback (London and Beatty, 1993), since this is one of the mechanisms most frequently used for development and assessment purposes in multiple organizations. Briefly, 360-degree surveys gather feedback on an indi-
ED
370
vidual from different sources, typically including self-evaluation, subordinates, peers and supervisors. Subjects have to answer questions rating a partner in
PT
different topics related to team climate and team dynamics (e.g., leadership, teamwork, decision-making, job knowledge or productivity, between others). Then, statistical analysis can be used to obtain comprehensive performance and
CE
375
behavioral evaluations of each individual. When some of these appraisal systems is implemented in the organization, they usually provide us with information
AC
we can use for building subjective teammate networks. Another mechanism to collect the information required to build subjective teammate networks is using
380
one of the questionnaires described in Appendix B. This a simple manner to obtain these data when they are not available in the company, as well as when the existing assessment process requires too much effort from the employees (see threats to validity). 16
M
AN US
CR IP T
ACCEPTED MANUSCRIPT
Figure 3: Simple illustrative example of teammate network (left panel) and social-skill network (right panel) built based on the two projects on top. In this example, the networks are gener-
ED
ated considering the normalized effort deviation as team performance estimator. As in Fig. 1, green and red connections correspond to positive and negative relationships respectively.
385
PT
Unlike subjective networks, objective ones are not built on the basis of the employees’ opinion, but on an objective evaluation of their previous collaborations. Thus, objective metrics are joint-performance estimators that assume
CE
that members of performing teams are compatible and have positive relationships among them; while failing projects have non-compatible teammates with
AC
negative relationships. This requires a global project performance/success es-
390
timator, being the score of a particular link the sum of this estimator in all the projects where the corresponding teammates worked together (Fig. 3, left panel). The advantage of this strategy is that this kind of metrics is quite common in IS companies. Multiple metrics have been discussed in software engineering literature for project success, e.g., on-time/on-budget completion, 17
ACCEPTED MANUSCRIPT
395
system quality or effort/budget deviation (DeLone and McLean, 2003; Espinosa et al., 2006). The proposed framework adapts to the information managed by the company, and teammate networks can be based on any of these metrics (see
CR IP T
threats to validity in Section 6). Finally, we also test teammate networks that combine subjective and ob400
jective metrics. The motivation behind these networks is to identify negative
personal relationships among teammates even when they reach a high level of accuracy when working together and vice versa. To calculate the score of the
we use the following equation:
AN US
connection between two individuals (Ii and Ij ) in these collaboration networks,
CT MIi Ij = sig0.25 (CT MIsi Ij ) · sig3.0 (CT MIoi Ij ) · CT MIoi Ij 405
(1)
where sigβ (x) = 1/(1 + eβx ) is the sigmoid function, and being CT MIsi Ij and CT MIoi Ij the normalized subjective and objective score of the corresponding
M
link, respectively.
In the results section, we compare outcomes obtained with teammate net-
410
ED
works built with different strategies and metrics. 3.2.2. Social-skill networks
The motivation behind the social-skill network is the existence of particular
PT
combinations of social skills – i.e., personality traits and abilities a person uses to interact with other people – that make two people “connect” during team-
CE
work and have a productive professional relationship (Blaskovich, 2008; Adams 415
and Anantatmula, 2010; Feldt et al., 2010). In this situation, the assessment of relationships among coworkers as a function of their social skills may help to
AC
determine the optimal project team composition, in particular when they have not previously worked together (e.g., consider companies with a high turnover rate). Note that in this situation a teammate networks do not provide informa-
420
tion regarding compatibility. Different works in literature investigate the personality traits present in exceptional software engineers (Turley and Bieman, 1995), in the individuals that 18
ACCEPTED MANUSCRIPT
dominate team communication (Licorish and MacDonell, 2015) or the ones required to perform a specific development activity (Acu˜ na and Juristo, 2004; 425
Acu˜ na et al., 2006). However, we do not found definitive results pointing out
CR IP T
which are the relevant skills to be included in our social-skill network. Therefore, along with a group of occupational psychologists belonging to the HR
department of one of our industrial partners, we defined a list of 11 social skills
to be measured in the social profile of an individual: written communication, 430
verbal and non-verbal communication, active listening, teamwork, negotiation,
sociability, empathy, agreeableness, leadership. Appendix A contains a brief
AN US
description of what we mean by each of these social skills. The intention of
this 11-skill social profile is to determine and predict each candidate’s ability to communicate and interact with coworkers avoiding or minimizing conflicts 435
within the working team (see threats to validity).
The question now is how to determine an individual’s social profile. Again, the HR management domain provides us with tools to determine the level of
M
development of an individual in different social skills, including the ones of our social profile. For example, the Myers-Briggs Test Indicator (MBTI), the Big Five (BF) and the Sixteen Personality Factor Questionnaire (16PF) are com-
ED
440
monly used in software engineering literature (e.g., see Hardiman (1997); Acu˜ na and Juristo (2004); Feldt et al. (2010); Andr´e et al. (2011); Licorish and Mac-
PT
Donell (2015)). These are simple personality tests nowadays readily available online which we use in different in-company experiments. In addition to these tests, multiple organizations implement their own competencies assessment sys-
CE
445
tem to identify their employee’s personality traits. Depending on the company,
AC
the framework can receive the social profiles from these or other sources. A social-skill network is equivalent to a teammate network, but reflect-
ing the correlation in terms of productivity between different skills (Fig. 2,
450
right panel). Nodes in these networks represent skill-level of development pairs (SSK = {S1 , ..., Sn }). We consider as possible levels of development for the different social skills, as possible values in the corresponding metric used to assess candidates in this skill. Once identified the nodes in the network, to 19
ACCEPTED MANUSCRIPT
establish connections between them, we depart from an objective teammate 455
network. Two nodes (Si and Sj ) are connected when this network contains a link between individuals with the corresponding skill-level of development pairs.
CR IP T
The compatibility score of the link (CSKSi Sj ) is then calculated as the sum of the scores of all the connections between individuals with these skills in the teammate network. Social-skill networks do not directly measure compatibil460
ity among candidates. They serve to compute an objective metric (C(I1 , I2 )),
which can be used to build a new teammate network (IT M 0 = {I1 , ..., In }; RT M 0 = {CT MI0i Ij = C(I1 , I2 )}) reflecting the compatibility among poten-
AN US
tial candidates from the perspective of their social profiles. Teammate networks built in this way can be measured as any other objective or subjective team465
mate network (Section 3.4). Formally, given the social profiles of two individuals (SPI1 = {s1I1 , ..., snI1 } and SPI2 = {s1I2 , ..., snI2 }) and a social-skill network (SSK = {S1 , ..., Sn }; RSK = {CSKSi Sj }), C(I1 , I2 ) is estimated as follows: XX
M
C(I1 , I2 ) =
i
j
CSKsi
sj I1 I2
(2)
As an example, let consider the following social profiles {negotiation: high; 470
ED
leadership: high; listening: medium} and {negotiation: low; leadership: low; listening: low}. Given the social-skill network depicted in the right panel of
0.2.
PT
Fig. 3, the compatibility between two individuals with these social profiles is
CE
3.3. Strategies for defining potential teams for the project In our socio-technical approach, the technical dimension is evaluated in a
475
first stage to identify potential teams for the project. Note that we use the
AC
term technical in a broad sense, since our profiles can include technical and non-technical skills, personality traits, limiting individual factors and personal preferences (Section 3.1.1). To do this initial identification, we use a BFR methodology to match the required profiles and the available resources’ profiles
480
(see Section 3.1.1). Multiple combinations of candidates can satisfy the project technical requirements producing a great amount of potential teams, especially 20
ACCEPTED MANUSCRIPT
when the number of available resources is high. Then, SNA can be applied to every potential team to assess the social dimension (Section 3.4). However, we can use algorithms based on our social networks to limit the number of assessment and drive the direction of our predictions. 3.3.1. Individuals compatibility strategy
CR IP T
485
The individuals compatibility strategy departs from the premise that people
who have successfully worked together before will work successfully again. Note
that this does not imply maintaining permanent teams composed by the same people. In the context of the proposed socio-technical framework, working suc-
AN US
490
cessfully means having a high compatible score in the teammate network. Then, we only generate teams composed of individuals included in the project’s teammate network. If a compatible teammate is not found for any of the required profiles, additional team members are assigned considering the project’s social495
skill network (see Section 3.3.2). These second assignments are made taking
included in the same team.
M
into account that incompatible teammates in the teammate network are never
ED
Our results indicate that the individuals compatibility strategy is the best strategy to substitute a key member (e.g., the team leader) in already existing 500
teams once a project has begun.
PT
3.3.2. Social-skills compatibility strategy The social-skills compatibility strategy consists of creating the team finding
CE
compatible individuals as a function of their social skills, i.e., considering the information provided by the project’s social-skill network. In production, this
505
strategy is mainly used to create teams including people who have never worked
AC
together, e.g., employees recently hired. 3.3.3. Star-based strategy Some works in literature suggest that people with highly developed social
skills can facilitate project success (Licorish and MacDonell, 2014, 2015). The 510
proposed teammate networks allow building teams based on this idea. If we 21
ACCEPTED MANUSCRIPT
compute the Katz centrality (Katz, 1953; Grindrod et al., 2011; Sun and Tang, 2011) of every node in the teammate network of a project, we can identify individuals that normally interplay effectively with their coworkers. We call
515
CR IP T
these individuals social-stars. The social-star-based strategy consists of building the team around some of these individuals. Once the social-stars are selected, the team is completed using the social-skill network.
Similarly to the social-star-based strategy, the team can be formed around
individuals highly skilled to perform a given role or task. In this case, the technical-stars are selected as a function of the project technical needs and the team is also completed using the social-skill network.
AN US
520
3.3.4. Split strategy
The split strategy consists of splitting a performing team in two new teams. New teams are completed with individuals with the same or similar social skills to the ones observed in the members of the original team. This strategy is mainly used with groups of programmers or testers with the idea of increasing
M
525
the pool of effective relationships.
ED
3.4. Analyzing compatibility networks To mine knowledge from the proposed social networks, we compute two measures for each potential team (T ). On one hand, the number of positive links within the team (P RT ). Depending on the strategy and the metric used to con-
PT
530
struct the network, positive has a different interpretation. For example, in the
CE
case of a subjective teammate network based on 5-point qualitative metric that rates relationships between teammates as “not effective”, “somewhat effective”, “effective”, “very effective” and “extremely effective”, we will consider positive those links labeled as “effective”, “very effective” or “extremely effective”. On
AC
535
the other hand, we compute the team social score (T SST ), defined as the cu-
mulative compatibility score of the relationships between every team member within the team. Formally, if T is composed by the teammates {tm1 , ..., tmm }: XX T SST = CT Mtmi tmj (3) i
j
22
ACCEPTED MANUSCRIPT
As the compatibility scores computed for links belonging to two distinct 540
teammate networks, team scores derived from different networks are not comparable. In this sense, it allows comparing teams in the context of a given social
CR IP T
network, i.e., teams assessed under the same conditions. The greater the social score of a team, the higher its estimated success probability.
Once computed these two values, the goal is to find an optimal balance be545
tween maximizing T SS and P RT . Let consider a simple illustrative example
departing from the teammate network depicted in Fig. 2. Our goal is to create a new 4-member team with a unique common required profile, which, for simplic-
AN US
ity in the example, all subjects in the network have. SNA allows us to compute easily the T SS value for all the possible teams: 26 out of 70 possible 4-member 550
teams do not have negative relationships, being T1 = {I4 , I6 , I7 , I8 } the team with the greatest T SS (T SST1 = 17). Then, only considering T SS, the network indicates that T1 is the best team to be assigned to the project. Nevertheless, P RT1 = 2, i.e., we only have evidence of 2 out 6 positive relationships among
555
M
the members of T1 . If we now consider T2 = {I1 , I3 , I4 , I6 }, T SST2 = 16 and P RT2 = 5. This is also a good recommendation from the team social score
ED
viewpoint (cf. T SST1 and T SST2 ), but the teammate network provides additional evidence of a good joint-performance between all members of T2 except I3 and I6 . Furthermore, assuming transitive relationships between software
560
PT
professionals, i.e., relationships between collaborators of collaborators (Surian et al., 2011), we could predict that I3 and I6 will be compatible too, because
CE
both are compatible with I1 and I4 . Transitive relationships of a given individual are computed by the inspection with depth 2 of the nodes accessible from the corresponding node in the network. Then, considering the information pro-
AC
vided by the network of Fig. 2, T2 is the 4-member team with a greatest success
565
probability.
23
ACCEPTED MANUSCRIPT
4. Proposal validation To validate our proposal, we carried out different kind of experiments in our industrial partners. In these experiments, we addressed two main goals. First,
570
CR IP T
to verify our proposal applicability and the suggested teams coherence. Second,
to validate whether the proposed teams were actually effective and productive. For statistical analysis, we group our industrial partners into four categories
depending on the number of employees. According to the definitions currently in
used in the European Union1 , companies with less than 50 and 250 employees are considered small- and medium-sized companies, respectively; while large-sized
enterprises have more than 250 employees. Although some of the medium- and
AN US
575
large-sized companies have employees located in offices from different countries, we consider an additional category (multinational organization) for more-than1000-employee enterprises with teams split over multiple countries.
M
4.1. Simulation of team assignments
To look into the applicability and coherence of our proposal in a given com-
580
pany, we first performed simulations of team assignments. These are proof-of-
ED
concepts in which we define a series of required profiles for fake projects and use the framework to form the corresponding team with the company’s current employees. Depending on the company, required profiles are obtained from the planning of previously completed projects and/or they are randomly chosen from
PT
585
a pool of predefined profiles. Then, a group of project managers, team leaders
CE
and/or HR management professionals designated by the company validates the assignments. When they reject a team, they must indicate the rejection causes. Before running the simulations of team assignments, the framework must
be adapted to the company. The aims of this initial phase are to identify the
AC
590
information already available in the company that can be used to build the social 1 EUR-Lex:
96/280/EC: Commission Recommendation of 3 April 1996 concerning the
definition of small and medium-sized enterprises, Official Journal L 107, 30/04/1996 P. 0004 - 0009
24
ACCEPTED MANUSCRIPT
networks, and define the protocols to capture required data. The company’s HR department usually participated in the framework adaptation. In many cases, they proposed elements to incorporate into the social networks – e.g., additional competencies and/or social skills they include in the employee’s profile or specific
CR IP T
595
metrics provided by the tools they use to assess team climate. To quantify the
adaptability and applicability of the framework, we calculate the percentage of companies where simulations could be run after the initial adaptation phase.
Once the framework is adapted to the company, simulations of team assign600
ments allow us to compare coherence of the teams proposed by social networks
AN US
generated with different metrics and strategies. Depending on the data available in the company, we can construct, measure and compare:
• Subjective teammate networks generated as a function of the employees’ responses to questionnaires Q1 and Q2 of Appendix B. 605
• Subjective teammate networks built with the information provided by any
M
existing tool to evaluate coworkers’ collaborations.
• Objective teammate networks generated using different project metrics.
ED
For statistical analysis, we distinguish between four success/performance indicators. Effort and budget deviation – i.e., the difference among es610
timated and actual effort/budget – since these are the most commonly
PT
available indicators in our industrial partners. Another project metrics, such as customer satisfaction or different quality measures, are only avail-
CE
able in a few companies. We group all these metrics into a unique category in order to get enough statistical evidence. Finally, a last simple
AC
615
metric, which can be even used when the company does not keep track of any project metric, consists of rating the projects as “successful” or “unsuccessful” depending on whether they are completed on schedule and meeting budget and requirements.
• Teammate networks combining different objective and subjective metrics 620
(Eq. 1). 25
ACCEPTED MANUSCRIPT
• Social-skill networks built with the subjects’ social profiles. Coherence of the teams obtained from networks constructed with different strategies is evaluated in terms of the percentage of accepted teams. These
625
CR IP T
values are compared with results obtained in control experiments where team
members are selected randomly using two possible strategies. With the first strategy, all the team members are selected randomly from the pool of available
resources independently of their skills. With the second one, each member is selected randomly too, but now from the set of candidates best suited for each
630
AN US
task or role. 4.2. Pilot experiments
Simulations of team assignments allow us to verify the applicability of our proposal and the coherence of the suggested teams, but they do not validate whether these teams are actually effective. To do this, we carried out pilot
635
M
experiments where the proposed framework served as a tool to help decisionmakers allocate the team of real IS projects. In the lead-up to these experiments, we always performed simulations of team assignments to adapt the framework
scenario.
ED
to the company and identify the social networks providing better results in each
In the pilot experiments, the final decision about the team composition is on the managers’ hands. Then, as in the simulations, we study the percentage
PT
640
of teams suggested by our social networks and assigned to real projects. We
CE
use this value to quantify the managers’ confidence in the framework. Note the difference between accepting a team in a simulation and assigning this team to a production project in the pilot experiments. In addition to this implicit measure of the managers’ perception, we also analyze what they think the framework
AC
645
brings to the team allocation process. For that, they have to fill in with answers a questionnaire with the following questions: • QPE1. How helpful do you think the tool for selecting the project team is? (1-5) 26
ACCEPTED MANUSCRIPT
• QPE2. I identified suited people for the projects who I had not taken into
650
consideration (1-5)
CR IP T
• QPE3. The tool allows forming more efficient project teams (1-5) • QPE4. The tool could be used to form project teams automatically (1-5) The questions are expressed on a Likert scale, where 1-5 mean “Not at 655
all helpful”, “Slightly helpful”, “Moderately helpful”, “Very helpful” and “Ex-
tremely helpful” for QPE1; and “Strongly disagree”, “Disagree”, “Neither agree nor disagree”, “Agree” and “Strongly agree” for QPE2-4.
AN US
Once a team is assigned to a production project, we verify whether this assignment was effective by evaluating four common project productivity metrics: 660
a binary success indicator, effort deviation, budget deviation and customer satisfaction. In a study like ours, it is nearly impossible to quantify to what extent the project benefited of a specific team composition as compared to other as-
in the same real project.
4.3. Large-scale production studies
ED
665
M
signments, since we cannot compare results obtained by different teams working
After the pilot experiments, some companies adopted the proposed framework as a supporting tool helping decision-makers during the creation of teams
PT
for IS projects. This demonstrates their confidence in our socio-technical approach, but moreover the large-scale production studies allow us to verify improvements in the team allocation process (see threats to validity).
CE
670
Although as in the pilot experiments, we cannot compare results of different
teams undertaken the same project, in the large-scale studies we do have enough
AC
statistical sample to compare performance of teams formed using the proposed framework and teams formed using other strategies (the specific strategy de-
675
pends on the company and the project). Comparisons are made in terms of different measures, mainly productivity and quality metrics (e.g., effort/budget deviation or customer satisfaction), depending on the company and the expected benefits. To minimize possible bias, results of the teams suggested by 27
ACCEPTED MANUSCRIPT
Personality traits (Yes-No)
Collaborations (Yes-No)
Applicability
Small-sized
9-6
2-13
0-15
100%
Medium-sized
16-3
10-9
8-11
Large-sized
5-0
4-1
2-3
Multinational
2-0
2-0
2-0
TOTAL
41
41
41
CR IP T
Project success (Yes-No)
Type of company
89.5% 100%
100%
95.1%
Table 1: Summary of the companies where the framework was tested. Middle columns indicate
AN US
the number of companies with the corresponding measures already available before performing
the simulations of assignments. The last column shows the percentage of companies where the framework was successfully applied.
the framework are compared with baselines established averaging the results of 680
projects completed by teams formed with the allocation process that we want to
M
improve. An improvement in the metrics analyzed in the large-scale production studies does not guarantee the assignment of the most effective team, but it gave us enough statistical evidence to verify that the framework has a positive
5. Results
PT
685
ED
impact on the company.
5.1. Applicability of the proposed framework
CE
To date, we have simulated team assignments in 41 organizations of different size and with different internal processes and hierarchical structure (Table 1). This has allowed us to validate the proposed framework adaptability and applicability to multiple scenarios. The framework was successfully tested in 39
AC
690
out 41 (95%) of our industrial partners. In these 39 companies, we were able to capture the required data and/or adapt the framework to the data previously managed by the company and run the simulations. The two companies where the simulations failed had quite similar proper-
695
ties: around 100 software professionals involved in the project, the only informa28
ACCEPTED MANUSCRIPT
tion available of these subjects was their curricular data, and historical data of projects were not available. In both companies, we found two problems during the framework adaptation. First, it was nearly impossible to build a reliable
700
CR IP T
base of knowledge of successful and unsuccessful previous teams. Second, the data acquisition process required some effort on the company: employees must
perform different tests to identify their personality, behavioral and social traits;
and they must complete the questionnaires of Appendix B to assess their interpersonal relationships with coworkers. Due to these issues, the project was
canceled before running the simulations. In this regard, it must be said that data capture was a bottleneck in some medium- and large-sized companies be-
AN US
705
cause of the number of questionnaires/evaluations each employee had to perform before running the simulations (see threats to validity). In spite of this, simulations were successfully run in six medium and one large enterprise with similar characteristics to the companies where they were canceled (i.e., without pre710
vious data regarding personality traits and collaborations). Four of these six
M
medium-sized companies are currently using the framework as a supporting tool
ED
to form the production teams of their IS projects. 5.2. Teams coherence
The simulations of team assignments indicate that the proposed social networks in general suggest coherent project teams (Fig. 4). No matter the type
PT
715
of company or the strategy used to construct the network, the experts designated by the company consider around 90% of the suggested teams coherent
CE
and valid for the project (cf. control experiments). The greater acceptance percentage occurs for small-sized companies due to the small number of potential candidates. For instance, acceptance is 100% in the three companies with less
AC
720
than 12 employees. However, this is not a significant result. Projects in these companies often required very homogeneous development profiles (e.g., android developer with knowledge in JSON, XML, Java and SQL), and most of the employees had these skills. This makes nearly all the possible candidates to be
725
considered adequate for the project. This observation is corroborated by the 29
CR IP T
ACCEPTED MANUSCRIPT
Figure 4: Acceptance percentage of the teams suggested by our teammates and social-skill networks in the simulations of team assignments. Results of the control experiments are
grouped into teams randomly selected from the whole set of available resources (“Random”)
AN US
and the ones selected from the set of candidates best suited for each task or role (“Random best”). Objective and subjective teammate networks are grouped into different categories depending on the metric used to build the network. “HR tool” corresponds to networks based on data extracted from an existing tool (note that none of the small-sized companies has this kind of tool). “Success indicator” corresponds to a binary successful/unsuccessful metric. “Other” includes project metrics but effort and budget deviation. See Section 4.1 for details.
M
control experiments, where teams selected randomly in small companies have a high acceptance percentage (88% or 96% depending on the strategy used to
ED
select the team members). Therefore, we cannot draw any conclusion about coherence of the teams suggested by the framework in small-sized companies. 730
We discuss below the helpfulness in these organizations.
PT
Comparing the different strategies to generate teammate networks, teams proposed by objective networks have in general a better acceptance than the
CE
proposed by subjective networks. This occurs regardless of the objective team performance estimator or the mechanisms used to assess subjectively interper-
735
sonal relationships (Fig. 4). Although it seems obvious that only the team
AC
members themselves have a detailed knowledge of the social interactions in the project, the simulations suggest that objective joint-performance estimators have a greater accuracy to capture efficiency of these interactions, and, therefore, objective networks contain more relevant information for us. A more
740
detailed analysis corroborates this observation. For example, teams formed with
30
AN US
CR IP T
ACCEPTED MANUSCRIPT
Figure 5: Objective and a subjective teammate network generated for a medium-sized company with projects completed within the last five years. For simplicity in the visualization, we only show connection for one employee (represented in blue in both networks). The jointperformance indicator used to build the objective network is the binary success indicator, i.e.,
M
successful projects compute +1 and failing projects −1. The subjective network is based on the last-five-year data extracted from an assessment tool already existing in the company. It scores the relationships between employees with a value between 0 and 100. The greater
ED
the score, the more positive the interpersonal relationship (ranges 0-39, 40-60 and 61-100 correspond to negative, neutral and positive relationships, respectively). Colors in both cases are used to identify positive, negative and neutral relationships according to the information
PT
captured in the corresponding network.
the social-stars strategy have a greater acceptance when social-stars are iden-
CE
tified in objective teammate networks. These networks also allow us to detect positive relationships between individuals with limited social skills that do not appear in networks generated with subjective metrics. For example, the blue individual of Fig. 5 represents a senior software developer. In the last five years,
AC
745
he participated in 11 project, six completed successfully and five unsuccessfully. His social profile indicates that he has very limited social abilities: excellent and acceptable competence in written communication and leadership, respectively; and a poor or very poor level of development in the rest of social skills. This
750
translates into a difficulty interacting with teammates, which is reflected in the 31
ACCEPTED MANUSCRIPT
subjective teammate network (Fig. 5, right). Taking into account this network, this employee was never proposed in the simulations as member of a new team. However, while the network built with subjective metrics does not show any
755
CR IP T
positive connection, six productive interactions can be observed in the equivalent objective network (Fig. 5, left). In view of this network, for example, a
team containing the blue individual, I3, I6 and I7 will be likely productive (cf. connections in the right panel). If the large-scale production experiments con-
firm that teams established in this way are actually productive, this is a highly relevant property for the team assignment problem in the context of a complex
collaborative process like software development. The objective teammate net-
AN US
760
works would allow identifying teammates able to adapt their working styles to special circumstances as the difficulty in social interactions.
A possible explanation for the problems found in the assignments derived from subjective evaluation of interpersonal relationships is that these metrics 765
are biased by other positive or negative relationships, e.g., affinity or friendship,
M
which do not always translate into productive or non-productive professional relationships. It is important to highlight that this bias appears in the simple
ED
questionnaires proposed in Appendix B, but also in a well-accepted HR assessment of team climate like 360-degree feedback. In fact, our results indicate that 770
the teams suggested by networks generated using questionnaires Q1 and Q2, and
PT
the teams suggested by a HR-tool-based network have nearly the same components and a very similar acceptability (Fig. 4). This result points out that, for
CE
our needs, the simple questionnaires of Appendix B collect equivalent information to the feedback gathered by more complex assessment processes. In the
775
context of a social-technical approach like ours, this is a highly relevant result
AC
for companies without corporate mechanisms to assess collaborations and team dynamics. Note that none small companies, and only 8 out of 19 medium-sized implement this kind of tool.
780
Concerning results of the different objective teammate networks, one could
expect that the higher the accuracy of the success/performance estimator, the more reliable the corresponding network. Nevertheless, the stakeholders’ general 32
ACCEPTED MANUSCRIPT
perception is that the optimal results are obtained when the networks are built as a function of the effort deviation. Note that this metric largely depend on the estimation quality – e.g., if a project fails due to over-optimistic scheduling or under-staffing, the corresponding team is considered non-productive.
CR IP T
785
Despite the possible bias discussed above in the subjective teammate net-
works, our analysis points out that subjective evaluation of collaborations can add some value when it is incorporated as correction factor to objective networks. Networks built in this way slightly improve the percentage of acceptance 790
of objective teammates networks (Fig. 4). Studying the teams proposed by these
AN US
networks, we observe that the information provided by teammates about their
interpersonal relationships allows measuring personal preferences together with the productive dimension of professional collaborations. Given how the compatibility score is computed in these networks (Eq. 1), productive relationships 795
where both teammates think that the collaboration is positive (although biased by friendship) are considered “more productive”. In the same way, subjective
M
evaluation makes “more negative” those non-productive relationships between teammates with a non-good personal relationship.
800
ED
Finally, results of the social-skill networks are nearly the same to the obtained with the teammate networks (in all cases they have a greater than 90% acceptance). It is important to remind that the teams proposed by these net-
PT
works are not based on previous collaborations, but on the estimated compatibility between their members’ social skills. The larger the company size, the
CE
more coherent the teams proposed. This result corroborates our departing hy805
pothesis that putting together coworkers as a function of their social skills is a good strategy to build IS project teams (at least from the coherence point of
AC
view).
5.3. Managers’ perception
810
We performed pilot experiments in 28 industrial IS enterprises. Table 2
summarizes the properties and measures used to build the proposed social networks in each company. It is important to remind that these are fixed after a 33
ACCEPTED MANUSCRIPT
simulation phase (see Section 4.2).
2
small (19)
3*
small (44)
4
small (24)
5*
small (41)
6*
small (45)
7
small (25)
8
small (30)
9
small (32)
10*
medium (87)
11*
medium (213)
12*
medium (232)
14*
ED
15
medium (105)
16*
medium (91)
17*
medium (203)
18
medium (64)
19
medium (133)
20*
medium (215)
21*
medium (152)
medium (123) medium (99)
AC
CE
PT
13*
effort deviation effort deviation defect density success indicator success indicator success indicator success indicator effort deviation defect density effort deviation defect density effort deviation defect density success indicator success indicator budget deviation budget deviation success indicator effort deviation effort deviation effort deviation
Q2 Q2
Personality traits
34
#teams
CR IP T
small (48)
Subjective measure
existing HR tool
#accepted
27
26
MBTI-BF-16PF existing HR tool
10
10
21
20
MBTI-BF-16PF
17
15
MBTI-BF-16PF
36
32
—
MBTI-BF-16PF
34
32
Q1
MBTI-BF-16PF
14
11
Q1
MBTI-BF-16PF
12
11
Q1
MBTI-BF-16PF
6
6
—
MBTI-BF-16PF
40
35
—
MBTI-BF-16PF existing HR tool
52
40
82
74
MBTI-BF-16PF existing HR tool existing HR tool
25
22
35
26
35
28
MBTI-BF-16PF existing HR tool
41
29
67
58
MBTI-BF-16PF existing HR tool existing HR tool existing HR tool
28
27
24
21
79
71
56
47
Q2 Q2 Q1
AN US
1*
Objective measure
M
Company
Type of company (participants)
Q1 Q2 existing HR tool Q2 existing HR tool existing HR tool Q2 — existing HR tool existing HR tool
ACCEPTED MANUSCRIPT
medium (111)
23
medium (55)
24*
large (289)
25
large (325)
26*
large (312)
27
large (450)
28*
multinational (1819)
TOTAL
effort deviation effort deviation effort deviation effort deviation success indicator budget deviation effort deviation
Subjective measure
Personality traits
#teams
#accepted
—
MBTI-BF-16PF
21
17
CR IP T
22*
Objective measure
Q1 existing HR tool Q2 Q2
Q1 existing HR tool
MBTI-BF-16PF existing HR tool existing HR tool
MBTI-BF-16PF existing HR tool existing HR tool
AN US
Company
Type of company (participants)
6
6
88
71
56
43
109
94
71
63
380
340
1472
1275
Table 2: Features of the pilot experiments. Note that, in line with the results discussed in Section 5.2, we always measured team-
M
mates networks built based on objective metrics, or on the combination of objective and subjective metrics. Companies identified
ED
with (*) incorporated the proposed framework to their staffing pro-
PT
tocol and/or are currently performing large-scale studies.
We first analyze the project managers’ opinion about the framework. Table 3 shows the distribution of their answers to the questionnaire regarding the helpfulness of the framework during team assignments (Section 4.2). In general,
CE 815
they think that the framework helps decision-making, allowing the identifica-
AC
tion of employees suited for the project that they did not know or, at least, they did not initially consider for the project. In spite of this, they believe that the team allocation must be supervised (cf. response to QPE4). Beyond the
820
general result, there are some significant differences in the managers’ perception of the framework depending on the company size. A significant number of managers belonging to small companies do not consider the framework useful
35
2
3
4
Small-sized
40.1%
23.2%
2.8%
20.3%
Medium-sized
1.4%
5.7%
9.5%
60.4%
Large-sized
2.2%
11.1%
6.5%
40.4%
Multinational
0.8%
1.6%
8.4%
38.1%
TOTAL
6.1%
8.0%
7.7%
45.5%
Small-sized
51.4%
10.2%
2.3%
27.1%
Medium-sized
1.2%
3.9%
1.9%
46.7%
46.3%
Large-sized
0.3%
4.6%
7.1%
42.9%
45.1%
Multinational
1.3%
2.4%
3.2%
52.4%
40.7%
TOTAL
7.1%
4.4%
3.4%
45.0%
40.1%
Small-sized
50.3%
7.3%
24.9%
16.9%
0.6%
Medium-sized
1.9%
2.9%
10.8%
69.7%
14.7%
Large-sized
6.8%
1.2%
9.9%
34.0%
48.1%
Multinational
0.0%
1.3%
9.5%
39.5%
49.7%
TOTAL
8.3%
2.6%
12.0%
47.7%
29.4%
Small-sized
68.9%
24.3%
6.8%
0.0%
0.0%
PT
5
Medium-sized
53.1%
34.0%
8.5%
4.4%
0.0%
Large-sized
92.3%
7.7%
0.0%
0.0%
0.0%
Multinational
97.9%
0.5%
1.6%
0.0%
0.0%
TOTAL
75.2%
18.4%
4.6%
1.8%
0.0%
CR IP T
1
CE
QPE4
13.6%
23.0%
39.8%
51.1%
32.8% 9.0%
AN US
M
ED
QPE3
QPE2
QPE1
ACCEPTED MANUSCRIPT
AC
Table 3: Distribution of the managers’ responses to the questions about the framework helpfulness. They are expressed on a Likert-type scale as follows. QPE1: 1 = Not at all helpful; 2 = Slightly helpful; 3 = Moderately helpful; 4 = Very helpful; and 5 = Extremely helpful. QPE2-4: 1 = Strongly disagree; 2 = Disagree; 3 = Neither agree nor disagree; 4 = Agree; and 5 = Strongly agree. See Section 4.2 for details.
36
ACCEPTED MANUSCRIPT
(40.1% not at all helpful; and 23.2% slightly helpful). They believe that they can make team assignments as a function of their experience and knowledge 825
about the potential team members, and the framework does not provide them
CR IP T
with any additional information. This is in line with the results obtained in the simulations of team assignments, and can be something expected in companies where everyone knows each other (cf. response to QPE2). Interestingly, all the managers of small companies who consider the framework very helpful or 830
extremely helpful belong to more-than-40-employee companies. As the number of potential candidates grows, this perception disappears and managers tend to
AN US
think that the proposed social networks help them understand the contribution of the people they manage.
In addition to the managers’ opinion, we also analyze their confidence in the 835
framework by means of the percentage of teams accepted and assigned to real projects. Figure 6 shows that this value is high (around 85%, see also column “#accepted” in Table 3), but always lower than the percentage of accepted
M
teams in the simulations. The analysis of the rejection causes indicates that this is because, in production, managers tend to reject the teams composed of people that do not have worked together yet. Although in the simulations they
ED
840
consider these teams coherent and valid for the project, in production they prefer not to do these assignments when a team composed of previous teammates can
PT
be formed. As some of the suggested teams successfully advances, confidence in the framework assignments grows, and a greater number of teams based on the social-skill networks is accepted and assigned to new projects. Note that,
CE
845
in spite of the initial doubts, the mean acceptance is around 87% (86% if we do
AC
not take into account the small-sized companies). 5.4. Team effectiveness validation
850
Figure 7 shows the outcomes of the teams assigned to real IS projects from
different perspectives. We first consider that a project is successfully completed when the system is delivered on-time, within-budget and meets the required features and functions (“success indicator”). Likely, this is the indicator that best 37
CR IP T
ACCEPTED MANUSCRIPT
Figure 6: Percentage of acceptance of the teams suggested by our teammates and social-skill
CE
PT
ED
M
AN US
networks in the pilot experiments.
AC
Figure 7: Outcomes of the project teams suggested by the socio-technical framework.
summarizes the team success. From this perspective, 87.9% of the teams can
be considered successful teams. Fails from this perspective are mainly a conse-
855
quence of scheduling deviations (∼ 83% of the failing projects). Interestingly, only 6% of the failing teams do it because of the system quality. Results from the effort, budget and customer satisfaction viewpoint are nearly equivalent, al-
38
ACCEPTED MANUSCRIPT
though we would like to highlight results from the effort deviation perspective. In some projects, the team needs more time than the estimated to complete 860
some tasks. This does not always translate into a budget deviation or a quality
the effort deviation are the more reliable.
CR IP T
decrease, but it can explain why the teammate networks built as a function of
In view of the results shown in Fig. 7, we can conclude that the teams suggested by the framework are in general effective and efficient. 865
5.5. An empirical study
AN US
To date, 17 out of 39 companies (Table 2) where the framework was successfully applied are currently running large-scale production studies, i.e., 44% of the companies (41% considering all the companies where we tried to apply the
framework; 54% if we do not include companies with less than 20 employees; and 870
61% if we only consider those companies that performed pilot experiments). The most relevant empirical study is ongoing at a multinational corporation based
M
in Europe, Asia and South America (company 28 of Table 2). After performing different simulations and pilot experiments with employees and projects belong-
875
ED
ing to different countries, the company decided to address the improvement of the people assignment process in nine software factories that showed a general deviation in scheduling and budget. They were located in Spain (2), France (3),
PT
Germany (1), Mexico (1) and Argentina (2). In this section, we focus on the results of measuring the proposed social networks when assigning people to the projects faced by these nine software factories during more than three years. Appendix C describes the steps leading up to the framework application.
CE 880
AC
5.5.1. Effectiveness and efficiency improvements The expected benefits of applying the framework in the nine software fac-
tories were mainly related to effectiveness and efficiency enhancements. Then, we compare the productivity and quality results of teams formed before and
885
after the framework start-up. In particular, the analyses are made in terms of effort deviation (productivity) and defects per KLOC (quality). Baselines for
39
Individuals
Social-skills
Social-stars
Tech-stars
Split
Effort deviation
0.84 beall
0.81 beall
0.85 beall
0.85 beall
0.84 beall
0.83 beall
Defect density
0.77 bdall
0.75 bdall
0.79 bdall
0.79 bdall
0.76 bdall
0.77 bdall
Effort deviation
0.85 bees
0.82 bees
0.85 bees
0.85 bees
0.87 bees
0.86 bees
Defect density
0.79 bdes
0.77 bdes
0.83 bdes
0.81 bdes
0.78 bdes
0.79 bdes
Effort deviation
0.81 bef r
0.77 bef r
0.83 bef r
0.85 bef r
0.80 bef r
0.81 bef r
Defect density
0.76 bdf r
0.73 bdf r
0.79 bdf r
0.76 bdf r
0.78 bdf r
0.75 bdf r
Effort deviation
0.89 bede
0.90 bede
0.89 bede
0.87 bede
0.91 bede
0.88 bede
Defect density
0.79 bdde
0.75 bdde
0.81 bdde
0.84 bdde
0.74 bdde
0.79 bdde
Effort deviation
0.82 bemx
Defect density
0.73 bdmx
Effort deviation
0.81 bear
Defect density
0.79 bdar
CR IP T
Overall
AN US
ar
mx
de
fr
es
all
ACCEPTED MANUSCRIPT
0.77 bemx
0.84 bemx
0.86 bemx
0.83 bemx
0.81 bemx
0.71 bdmx
0.70 bdmx
0.71 bdmx
0.77 bdmx
0.75 bdmx
0.78 bear
0.84 bear
0.83 bear
0.80 bear
0.82 bear
0.79 bdar
0.84 bdar
0.84 bdar
0.72 bdar
0.77 bdar
Table 4: Productivity and quality measures of the on-site projects undertaken by teams
M
formed with the support of the proposed framework. Table 5 displays equivalent measures for distributed projects. Results are shown as a function of the three-year baselines for effort deviation (bex ) and defect density (bdx ); globally (“all”) and grouped by country (“es”,
ED
“fr”, “de”, “mx” and “ar” correspond to Spain, France, Germany, Mexico and Argentine, respectively). Columns display results grouped by the strategy followed to form the team.
PT
comparison are computed with the projects completed within the three years prior to start applying the framework (see Appendix C). Table 4 shows the
CE
outcomes obtained in the on-site projects – i.e., projects where all the team 890
members belong to the same software factory – completed by teams formed with the framework support during a more-than-three-year period. Analyzing
AC
the overall results, we observe a general enhancement, both in productivity and quality. The mean effort deviation is 16% fewer; while the mean defect density has decreased 23%. Improvements by country are coherent with these values.
895
Teams composed of teammates who have successfully worked together before
generally are more effective (cf. the individuals compatibility strategy and the rest of strategies in Table 4). Most members of these teams know each other and 40
ACCEPTED MANUSCRIPT
have close professional relationships. The existence of prior positive relationships within the team favors collaboration and coordination and facilitates be900
coming a performing team. Conversely, the teams formed using other strategies
CR IP T
need time to grow and consolidate the teammate-teammate relationships. This implies that these teams follow a more pronounced forming-storming-norming-
performing pattern (cf. red and blue traces in Fig. 8). However, in spite of the
more pronounced forming, storming and norming phases, they usually become 905
performing teams after a relative short formative period. Note that the mean
effort deviation in Table 4 is similar independently of the strategy followed to
AN US
select the project team. This result is observed even in teams created using
the social-skills compatibility strategy, which does not take into account previous collaborations to evaluate relationships among coworkers. This observation 910
served to decide using this strategy to assign people to small non-critical projects in order to create new teammate-teammate relationships. These new relationships usually become effective during the project, and then the new compatible
M
teammates were assigned to projects that were more critical. In this regard, we would like to emphasize the high turnover rate: 26%, 24.5%, 19.5%, 28.5% and 17% in the last year in the Spanish, French, German, Mexican and Argentinean
ED
915
software factories. In this situation, it seems critical the formation of effective relationships among the employees recently hired.
PT
Although the statistical sample is enough to minimize bias, the improvements could be consequence of other factors, e.g., a more skilled employee recruiting. Furthermore, effort deviation and defect density are simple and in some
CE
920
sense unreliable measures of productivity and quality. Therefore, we would like to emphasize the stakeholders’ general perception. They believe that the use
AC
of the proposed social networks led to a noteworthy improvement in the new teams’ efficiency, which minimized the scheduling and budget problems in the
925
nine software factories. In fact, they have started three new large-scale production studies: one involving three software factories in India; another with five software factories in Mexico and five in Argentina; and the last one with 15 software factories located in Europe (3 in Spain, 8 in France and 4 in Germany). 41
CR IP T
ACCEPTED MANUSCRIPT
AN US
Figure 8: Evolution of the mean normalized performance of 20 different on-site French teams assigned to a more-than-one-year project. Red trace corresponds to 20 teams created using the individuals compatibility strategy. Blue trace corresponds to teams created using the social-skills compatibility strategy (7), the social-stars strategy (7) and the split strategy (6). The analysis covers the 12 months after the team allocation and its assignment to a new project. Dashed line denotes the French three-year performance baseline.
M
5.5.2. Emphasizing the relevance of social interactions
Beyond the general results shown in Table 4, additional observations after
930
ED
measuring the proposed social networks during more than 36 months can be summarized in the following points: • Results of teams created with the social-stars and the technical-stars strat935
PT
egy are nearly the same. This emphasizes the relevance of interpersonal relationships for project success and suggests that social skills can be as
CE
important as technical skills. In some cases, the social-stars strategy can produce teams with a better mean performance (see Spanish and Ger-
AC
man teams in Table 4). In step with these results, multiple projects were
940
successfully completed by development teams not having all the required technical skills, but having a high social score and a large amount of positive interpersonal relationships between their members. A detailed analysis indicates that, although from a technical viewpoint these were not good team assignments, the projects met their budget and quality ob-
42
ACCEPTED MANUSCRIPT
jectives due to an effective knowledge sharing. As expected, there are fluid 945
collaborations within the working team, but also with employees outside the team. The proposed social score is not defined to predict this kind of
CR IP T
collaborative behavior. However, our analysis points out that the greater the number of social-stars in a project team, the higher the knowledge sharing outside the team. 950
• Taking into account the results of Licorish and MacDonell (2014, 2015), the social-starts strategy was initially proposed thinking in people in man-
agement, architectural and development roles. Nevertheless, depending
AN US
on the country, around 27-39% of the individuals identified as social-stars perform an ancillary role. These results point out again the relevance 955
of the social aspects. Despite that, we would like to highlight the IS project leaders’ profile. It is well-known that project leaders play a crucial role and can be a major source of errors that lead to failure due to
M
bad decision-making (Charette, 2005). However, we have found that relationships between project leaders and the rest of team members are also 960
critical for project success. Multiple non-effective teams become effective
ED
under the command of project leaders with social skills compatible with the ones of the team members. Note that we do not only refer to specific
PT
project leaders, but to different leaders with specific social-skills. • Teammates with limited social skills (e.g., people showing a low team965
work capacity) can also constitute high-performing teams. These effective
CE
relationships usually appear between highly skilled technicians that, in general, only have productive relationships with similar peers. This kind
AC
of teams successfully performs small projects or specific – and usually
970
complex – tasks. If this is not the case, the team needs complementary members with additional social skills. • When a performing team has a high maturity level, replace some of its members does not significantly affect the team’s effective productivity. In this regard, we would like to stress results obtained with the split strategy 43
ACCEPTED MANUSCRIPT
since, departing from a performing team; it permits obtaining two nearly 975
equivalent teams to the original one. This strategy increases the number of productive teammate-teammate relationships.
CR IP T
• Finally, teammates and social-skill networks have helped to create project
teams distributed across different software factories. This was not a common practice in the company. Then, initially, distributed teams were 980
created using the social-skills compatibility strategy. A great percentage of these initial teams failed, even when teammates belonged to the same
country. This suggests the need of different social skills for interacting
AN US
with distributed coworkers. Once the number of productive distributed teammate-teammate relationships grew, performing teams were formed 985
using the individuals compatibility strategy. Afterwards, the increased number of productive relationships allows successfully using other assignment strategies. For instance, high-performing teams have been created
M
around distributed social-stars identified in the teammate networks. Currently, distributed teams’ mean performance and quality are nearly the 990
same than the on-site teams (cf. Tables 4 and 5). Although it is difficult
ED
to draw general conclusions due to the individual-dependent nature of social interaction, distributed teammate networks allow us to analyze how
PT
compatible social profiles change depending on the teammate’s location. For example, around 87% of people that successfully interact with cowork-
995
ers of other countries show a high empathy level. Similarly, more than
CE
75% of these people also have a medium-high adaptability level. In other cases, a dependence on the language used to communicate is observed.
AC
For example, around 72% of the effective teammates who communicate in
1000
English show a high active listening level. There even exist skills needed to interact successfully with coworkers of specific countries.
44
ACCEPTED MANUSCRIPT
Spanish
English
Effort deviation
0.84 beall
0.83 besp
0.87 been
Defect density
0.84 bdall
0.85 bdsp
0.84 bden
CR IP T
Overall
Table 5: Outcomes of the distributed teams composed of people belonging to different software
factories. Columns labeled as “Spanish” and “English” indicate the language used to communicate. Communication is in Spanish language if the project involves people from Spain,
Mexico and/or Argentina; and it is in English when teammates are from Spain, France and/or Germany.
AN US
6. Threats to Validity
To limit the scope of our claims as well as to explain their utility, the first important fact to take into account is that this is an empirical study. Like most empirical studies, the validity of our results is subject to several threats (Basili 1005
et al., 1999).
M
Concerning the generalization of the findings, we obtained successful outcomes in companies of different size, with different organizational structures and different internal processes. Nevertheless, results presented in this paper
1010
ED
should not be taken as valid in the general sense. Although a Hawthorne effect is not possible, the environment in real industrial projects cannot be controlled to the same extent as in isolated research experiments. In this regard, it is impor-
PT
tant to note that each project is different and that team dynamics are complex. These factors make nearly impossible to control all the variables affecting suc-
CE
cess or failure of an IS project. Thus, there exist several threats concerning
1015
external factors that may affect our results. In particular, we do not exactly know whether the performance and quality improvements observed in the pi-
AC
lot experiments and the large-scale production studies can be attributed to the selection of more adequate coworkers or to many other possible factors. Furthermore, obtaining optimal results in the metrics analyzed in the pilot experiments
1020
and the large-scale production studies does not guarantee the assignment of the most effective team in a general sense. In the first case, our results must be in-
45
ACCEPTED MANUSCRIPT
terpreted as the formation of effective teams from parameters established by the company in each particular case. In the second case, they must be interpreted as an improvement in team assignment process of the company. Regarding the statistical conclusion validity, we analyze results derived from
CR IP T
1025
tests and pilot experiments performed in 39 different companies. This includes
thousands of simulations and real project assignments. Furthermore, we report detailed results of using the proposed framework during more than three years
in nine software factories belonging to a multinational corporation. This gives 1030
us enough statistics to consider that our results are statistically valid.
AN US
We assume that the most reliable source for information about compati-
bility between coworkers is their common participation in previous projects. In particular, we define compatibility as a function of objective global metrics related to success/performance of completed projects. This definition is also 1035
subject to some threats to validity. With this approach, we cannot guarantee that some relevant nodes and/or links from the network can be missing from the
M
source data, as it happens with other strategies for constructing compatibility networks for software professionals (Bachmann et al., 2010). If the database
1040
ED
of previous projects is not complete, multiple nodes or links can be missing. Similarly, predictions of the social networks also depend on the reliability of the metrics managed by the company. Depending on the particular data managed
PT
by the company, in our implementations we mainly used the following variables to measure project success/performance: on-time completion, within-budget
CE
completion, budget/effort deviation, system quality and customer satisfaction. 1045
These are the variables identified in literature as the main metrics for global IS projects success (DeLone and McLean, 2003; Espinosa et al., 2006). However,
AC
we also obtained successful outcomes with other metrics such lines of code per unit of effort or time taken to complete a task. Finally, with our definition, we only analyze intra-team relationships, when relationships among team members
1050
and the rest of the organization can also be relevant. In spite of these threats, our empirical results demonstrate that the abstraction that we propose is a good approach for the team allocation problem. 46
ACCEPTED MANUSCRIPT
Regarding the adaptability to different organizations, the proposed social networks can be built using data and metrics commonly managed by indus1055
trial companies. The employees’ technical and non-technical capabilities can be
CR IP T
captured from curricular data, which are commonly available in industrial companies. Personality traits and interpersonal relationships can be assessed with
different metrics. These metrics include estimators from the HR management domain that can be obtained from tools commonly available in medium/large1060
sized enterprises – e.g., 360-degree feedback surveys (London and Beatty, 1993) or competencies assessment (Robertson and Smith, 2001). If the company does
AN US
not have any of these tools, MBTI, BF and 16PF are simple personality tests
nowadays readily available online. These three tests are commonly used for personnel assessment in software engineering literature (e.g., see Hardiman (1997); 1065
Acu˜ na and Juristo (2004); Feldt et al. (2010); Andr´e et al. (2011); Licorish and MacDonell (2015)) and we successfully used them in some of our experiments. In this regard, we strongly recommend dealing with the help of professionals of
M
the organization’s HR department, since they know the mechanisms existing in the company to assess employees and their collaborations. This aims to adapt the framework to the specific characteristic of the company. For instance, in
ED
1070
some of our in-company experiments, the HR department proposed additional competencies measured by their own personality tests to be included in the
PT
social networks.
Concerning the applicability of the proposed framework, one of the main drawbacks is related to the data required to build and measure the social net-
CE
1075
works. It needs specific data related to projects (e.g., features, participants and success/performance metrics) and employees (e.g., detailed profiles or estima-
AC
tors of their social abilities). This need can make the framework not useful in some organizations. On one hand, although as we indicate above it can be
1080
easily adapted to the data managed by each company, not all the companies manage and keep track of the information required. In these cases, not only the team selection process must be implemented, but also additional processes to capture and keep historical data of projects and employees. On the other 47
ACCEPTED MANUSCRIPT
hand, the larger the company, the greater the volume of data to be captured 1085
and analyzed. In this regard, it is important to note that, although subjective data about working interpersonal relationships among coworkers can contribute
CR IP T
to improve teamwork, this information is not mandatory for the framework. In fact, objective teammate networks generated with a, in some sense, unreliable productivity measure as the effort deviation, provide more relevant information 1090
than those built with data subjective data.
7. Discussion
AN US
Many researchers and practitioners in software engineering hold to a socio-
technical view of IS projects (Sawyer, 2004; Luna-Reyes et al., 2005; Herbsleb, 2007; Tamburri et al., 2013). From this view, both the technical and social 1095
aspects of a project are crucial to performance and, therefore, should be taken into account to succeed. While multiple mechanisms to evaluate and improve
M
the technical dimension of a project are extensively applied in software industry, the mechanisms to control and manage the social dimension are still unclear. The framework presented in this paper takes advantage of some previously discussed elements and incorporates novel principles for globally assigning people
ED
1100
to IS projects from a socio-technical perspective. We propose the assessment of
PT
the candidates’ technical skills, experience and personality traits, but also their interpersonal relationships. For that, the framework leans on the information managed by the company to create personal profiles of each employee and to construct and measure different social networks which objectively and/or sub-
CE
1105
jectively characterize the collaborations among coworkers. This characterization can be made according to previous (successful or unsuccessful) collaborations
AC
as well as to a measure of the compatibility between teammates estimated as a function of their social skills. For companies where the required information
1110
is not available, we define possible mechanisms to collect it in a simple manner. We provide industrial insights into the framework applicability in different contexts, obtaining successful outcomes according to the parameters established
48
ACCEPTED MANUSCRIPT
by the corresponding company and with social networks constructed and measured with distinct strategies and metrics, which points out the generality of 1115
our proposal.
CR IP T
How to build efficient and effective teams is one of the most relevant questions in software engineering management. In practice, the decision of who participates in a project normally is on the managers’ hands. Although we also
found companies where the assignments were made by the HR department as 1120
a function of curricular data. Managers usually make decisions based on their
experience and subjective knowledge about the potential team members. In
AN US
our in-company experiments, we observed that in this scenario decision makers
tended to count on the same people and many new projects directly inherited teams of previously completed projects. This practice, that might be evident in 1125
small companies with a limited staff, also happened in medium- and large-sized companies and even in cases of recurrent failing teams. In many occasions, this was due, on one hand, to the lack of knowledge sharing among managers and, on
M
the other hand, to the lack of proper methods to evaluate personnel capabilities. Experience showed that maintaining teams composed by the same people was not always an effective practice, as successful teams for a given project might
ED
1130
not be appropriate for another one. Moreover, in the organizations with more complex structures, managers could be part of many different teams, which
PT
could be distributed across different locations or even countries. In these cases, they did not know every coworker daily work and did not always fully understand the contribution of a particular member to the team and his/her impact
CE
1135
on team dynamics. In these cases, the same as when they were made by the HR department, assignments were often made as a function of the candidates’ cur-
AC
ricular data. Therefore, we found a strong argument for a tool like the proposed framework, defining mechanisms to obtain wide-ranging information to form an
1140
accurate picture of coworkers’ performance and their interactions and, leaning on this information, infer adequate configurations for a specific project. In this regard, experiments in real-life industrial projects pointed out that managers, in general, appreciated the existence of such mechanisms, but they preferred to re49
ACCEPTED MANUSCRIPT
ceive the information and define their owns teams according to their experience 1145
(Table 3). The teammate networks provided managers with information that allowed
CR IP T
them to identify potential effective relationships among employees they did not know. They were also suitable to point to groups of teammates that usually work together in failing projects. In these networks, the team social score max1150
imizes the former relationships and minimizes the latter within new working teams. The pilot experiments and the long-term production studies showed
that forming teams based on this value is an effective strategy to create per-
AN US
forming teams. Since social-skill networks are not based on previous collaborations, the compatibility information they provided for decision makers was 1155
more difficult to understand than the one provided by the teammate networks. At a first glance, the managers were reluctant to accept these relationships and form teams based on them. However, the experiments also demonstrated the general good performance of these teams (cf. results of the individuals and the
1160
M
social-skills compatibility strategy in Table 4). From a more theoretical perspective, results obtained from the social-skill networks support the thesis that
ED
specific combinations of social skills allow two teammates to collaborate more effectively in an IS project. In this regard, we would like to highlight the particular case of some employees with limited social capabilities having difficulty
1165
PT
interacting with others (e.g., see Fig. 5). The analysis of our social networks permitted to identify coworkers with compatible social skills and the ability to
CE
adapt their working styles to these individuals. Beyond the benefits of forming more performing teams, these team assignments also increased the general personnel satisfaction. In addition to the direct benefits of creating more effective
AC
working teams, some of the staffing strategies implemented using the proposed
1170
social networks, e.g., the split strategy, had relevant positive medium/long-term benefits in exchange for a little short-term performance penalty (Fig. 8). This strategy is conceived to generate two performing teams from one existing performing team. In our experiments this was usually achieved (Table 4). Then, new productive teammate-teammate relationships provided an increased capac50
ACCEPTED MANUSCRIPT
1175
ity to face successfully new projects for the company. This is a highly valuable result for organizations with a high turnover rate. On the down side, the information required to build the proposed social
CR IP T
networks. To implement the framework in an industrial enterprise, we need: • A detailed knowledge of each employee’s technical skills and capabilities is mandatory to find potential candidates to participate in the project.
1180
• Access to historical productivity/quality metrics of each project is mandatory to build objective teammate networks.
AN US
• To take advantage of the social-skill networks, it is needed to know each employee’s social profile. This is likely the information more difficult to collect.
1185
Beyond the use of the proposed framework, our results emphasize the importance of how participants in an IS project collaborate, communicate and, in
M
general, work together as a team. In line with previous findings (Fitzpatrick and Askin, 2005; Acu˜ na et al., 2008; Adams and Anantatmula, 2010; Feldt et al., 2010), this points out that the social aspects can be as relevant as the technical
ED
1190
aspects for successfully completing a project (cf. results of the social-start and the technical-start strategy in Table 4), and supports the hypothesis that we can
PT
significantly improve our outcomes paying attention to interpersonal relationships when putting together a team. Social factors may enhance (or diminish) 1195
the sum of individual team members’ attitudes. In this sense, for instance,
CE
we found individuals with a low productivity that significantly increased their performance when moved to a team with different teammates; or projects suc-
AC
cessfully completed by teams not covering all the required technical skills but composed of people with compatible social skills. Obviously, people are not the
1200
only factor to be considered, and we cannot disregard other critical aspects for the project success such as the planning, project tasks, workload or scheduling, between others. The benefits of considering software development in industry as a socio51
ACCEPTED MANUSCRIPT
technical process were observed independently of the company size. Never1205
theless, they were less apparent in small-sized enterprises where the reduced number of employees, and usually also of projects, limited the number of possi-
CR IP T
ble assignments. In this regard, it is important to highlight that implementing a framework like ours to find an optimal trade-off between technical and so-
cial aspects does not have sense in this kind of companies. In addition to the 1210
limitation in the number of possible teams, managers in these companies often
know the rest of employees during daily work (Table 3). Although we have not enough statistics to raise a general conclusion, throughout our experiments we
AN US
established a threshold of 20 software professional to consider the framework helpful.
We have developed social networks characterizing interpersonal relationships
1215
within the team. These relationships are strongly related to group factors such as cohesion, conflicts or team climate, between others. However, social factors in an IS project are not limited to intra-team aspects. Of course, interpersonal re-
1220
M
lationships outside the working team can significantly impact on performance. Our teammate networks could incorporate additional elements characterizing
ED
interpersonal relationships outside project teams. For instance, Oh et al. (2006) found a correlation between team effectiveness and what they called the group social capital – the set of resources available to the team through its members
1225
PT
interpersonal relationships within the team and with other members of the organization. In companies with the ability to compute the group social capital
CE
proposed by Oh et al., it should be possible to develop a strategy for team assignments trying to balance this value and our team social score. Given the current trend towards globalization in the software development
AC
industry, we encourage software organizations to systematically incorporate re-
1230
lationships between coworkers to their team staffing processes. In this scenario, software production involves teams split over locations, countries, and even companies. Despite the increasing attention to distributed software projects, there is a limited understanding of team dynamics that makes these projects succeed or fail. In a distributed development team, there are many additional factors 52
ACCEPTED MANUSCRIPT
1235
influencing social interactions (Holmstrom et al., 2006; Espinosa et al., 2006). Between others, these include temporal distance (i.e., people working at different time zones which implies few overlapping hours), socio-cultural differences
CR IP T
(e.g., native language and/or culture differences) and geographic distance. Due to these communication issues, in a distributed project there exist less commu1240
nications among coworkers: each individual communicate with fewer coworkers
and interactions are less frequent (Olson and Olson, 2000). Even some au-
thors claim that communications are less effective (Herbsleb and Mockus, 2003). Therefore, distributed software engineering poses a challenge to the development
1245
AN US
process, demanding additional behavioral traits and new social abilities in the
interaction among software professionals. Results reported in this paper in the context of distributed projects are a good departing point to incorporate SNA to the team formation problem in software industry.
file
M
Appendix A. Proposal of competencies to include in the social pro-
This Appendix briefly describes the competencies and social skills measured
1250
ED
in our social-skill networks. As we indicate in the main text, these data can be obtained from different tools depending on the company. This implies that in different social-skills network the same value has different meanings and inter-
1255
PT
pretations.
• Written communication: Ability to write clearly and effectively in order
CE
to express information via a written channel
AC
• Verbal and non-verbal communication: Ability to give information and
1260
actively manage the communication process through oral channels.
• Active listening: Ability to accurately receive and interpret messages in the verbal communication process. • Teamwork: Ability to contribute to teams and improve their effectiveness through personal commitment. 53
ACCEPTED MANUSCRIPT
• Negotiation: Ability to talk or consult together with someone for the purpose of mutual arrangement using a win-win philosophy. • Sociability: Ability to get on well with a wide range of people and build
1265
CR IP T
long term trusting relationships.
• Empathy: Ability to see the world as another person, to share and understand another person’s feelings, needs, concerns and/or emotional state.
• Agreeableness: Ability to be kind, sympathetic, cooperative, warm and
AN US
considerate with other.
1270
• Leadership: Ability to guide and inspire others towards achieving goals. • Motivation: Ability to support and encourage individuals, so that they give their best.
• Adaptability: Ability to respond and adapt to changing circumstances and to manage, solve problems and provide solutions.
M
1275
ED
Appendix B. Questionnaires to evaluate teammates collaborations Information required to built subjective teammate networks can be extracted from an appraisal system previously implemented in the company. However,
1280
PT
these tools require of consistent organizational processes and they are not available in many companies. Furthermore, they usually require too many effort
CE
from subjects (e.g., in some of the appraisals already existing in our industrial partners each individual has to complete questionnaires with more than 200 questions). Then, in order to simplify the process, a group of occupational psy-
AC
chologists of one of our partners helped us define two possible questionnaites to
1285
collect subjective information regarding the teammates’ collaborations in a simple manner (Table B.6). One of the goals of the simulations of team assignments (Section 4.1) is comparing the results obtained with social networks generated on the basis of information provided by some existing tool in the company, and the ones obtained with questionnaires Q1 and Q2. 54
ACCEPTED MANUSCRIPT
ID
Question
Q1
What is the impact of Ana on your work? (1-5)
Questionnaire Q2
CR IP T
Questionnaire Q1
Question
Q1
How productive is your work with Ana? (0-10)
Q2
How effective is your work with Ana? (0-10)
Q3
How well does Ana communicate with you? (0-10)
Q4
How well does Ana collaborate with you? (0-10)
Q5
How well does Ana help you with your work? (0-10)
Q6
How respectfully does Ana treat you? (0-10)
Q7
How quickly does Ana respond your requests? (0-10)
Q8
How well does Ana handle criticism? (0-10)
Q9
How professionally does Ana behave? (0-10)
M
AN US
ID
ED
Table B.6: Questionnaires to collect subjective evaluations of the teammates’ collaborations.
Questionnaire Q1 was defined to capture with a single question whether a
1290
PT
subject considers positive, negative or neutral his/her professional relationship with a particular teammate (Ana in the example). Using a common strategy in this kind of evaluations, the only question of Q1 is expressed on a five-point
CE
Likert-type scale (1 = Extremely negative; 3 = Neither positive nor negative;
1295
and 5 = Extremely positive). In questionnaire Q2, subjects have to rate nine
AC
questions with a numerical value between 0 and 10. These questions refer to well-known key aspects in social interactions within a work team – e.g., communication or knowledge sharing (Lievens et al., 2002; Chen et al., 2004).
1300
When the network is built leaning on one of these questionnaires, connections
among potential candidates are scored with the mean value of the responds provided by the two candidates about their collaboration.
55
ACCEPTED MANUSCRIPT
Appendix C. Background of the empirical study Appendix C.1. Framework adaptation
1305
CR IP T
We made the framework adaptation along with professionals of the Spanish HR department. The company kept track, between others, of the following rel-
evant data for all the projects undertaken by the software factories included in
the study: type of project, area of knowledge, working team, working methodology, estimated workload, complexity, risk level, budget, effort/budget deviation, customer satisfaction, defect density and software complexity. We used some of
these data (type of project, area of knowledge, working methodology, estimated
AN US
1310
workload, complexity, risk level, team size, budget and geographical distribution) to generate the similarity vectors employed to characterize and compare projects. While effort/budget deviation, customer satisfaction, defect density and software complexity were used to generate and test objective teammate net1315
works based on different metrics. For the simulations and the pilot experiments,
M
we generated a base of knowledge with all the projects completed within the last three years in the nine software factories. Once the large-scale production
ED
study started up, this was weekly updated. Baselines for comparison in these experiments were also computed with the initial last-three-year data. These 1320
baselines did not have changed along the study.
PT
The company also implemented at organizational level two tools whose data were suited to construct our social networks: a 360-degree feedback appraisal and a competencies assessment system. Every year, each employee has to pro-
CE
vide self-evaluation and assess a group of coworkers including his/her supervisor,
1325
subordinates and some colleagues. Feedback collected of an individual in the
AC
360-degree assessment is weighted to calculate overall individual and collective metrics measuring different aspect related to team work. During the framework adaptation phase, we decided to test the use of two of these metrics. On one hand, a 0-to-10 quantitative value rating teammate-teammate relationships: 0
1330
corresponds to relationships with negative impact on team dynamics and 10 to relationships between coworkers with an extremely positive interplay, respec-
56
ACCEPTED MANUSCRIPT
tively. This metric was used to test 360-feedback-based teammate networks. On the other hand, a qualitative score that ranges from 0 (not developed at all) and 10 (highly developed) the level of development of an individual in different competencies which could be linked with the skills of our social profile (Appendix
CR IP T
1335
A). Competencies assessment, for its part, supports new employees recruitment. Between others, it assesses with a 0-to-10 quantitative value 11 competencies
that also could be used to characterize an employee’s social profile. When this information was not available for an employee included in the study, he/she had to complete the competencies assessment process.
AN US
1340
Appendix C.2. Results of the simulations and the pilot experiments During the simulations of team assignments leading up to the framework application in the nine software factories selected by the company, we tested the following social networks:
• Subjective teammate networks based on Q1, Q2 and 360-feedback.
M
1345
• Objective teammate networks based on effort/budget deviation, customer
ED
satisfaction, defect density and software complexity. • Teammate networks based on all the possible combinations of subjective
PT
and objectives metrics.
• Social-skill networks based on the information extracted from the existing
1350
360-feedback and competencies assessment tools, both individually and
CE
averaging the value obtained from each tool. Based on the HR department professionals’ experience, in the social profiles, and therefore in the nodes
AC
of the social-skill networks, we converted the 0-to-10 values obtained from
1355
these tools into a five-point qualitative scale: “very poor” ([0−2)), “poor” ([2 − 4)), “acceptable”, ([4 − 6)),“good” ([6 − 8)) and “excellent” ([8 − 10]). The simulations corroborated that the objective teammate networks based
on the effort deviation were the most reliable (Fig. 4). The subjective teammate networks based on Q1, Q2 and 360-feedback had poorer and equivalent results. 57
ACCEPTED MANUSCRIPT
1360
And when we combined effort deviation and a subjective metric, no matter which of them, results were very similar to the obtained with the effort deviation (94% vs. 93% of acceptance). Then, we decided to test these latter teammate
CR IP T
network in different pilot experiments. In particular, given the institutional mechanisms to keep updated the 360-feedback assessment, we used these data 1365
instead of questionnaires Q1 and Q2. References
Abreu, R., Premraj, R., 2009. How developer communication frequency relates
AN US
to bug introducing changes. Proceedings of the joint international and annual ERCIM workshops on Principles of software evolution (IWPSE) and software evolution (Evol) workshops - IWPSE-Evol 09 .
1370
Acu˜ na, S., Juristo, N., Moreno, A., 2006. Emphasizing human capabilities in software development. IEEE Software 23, 94–101.
M
Acu˜ na, S.T., G´ omez, M., Juristo, N., 2008. Towards understanding the relationship between team climate and software quality – a quasi-experimental study. Empir Software Eng 13, 401–434.
ED
1375
Acu˜ na, S.T., Juristo, N., 2004. Assigning people to roles in software projects.
PT
Softw: Pract. Exper. 34, 675–696. Adams, S.L., Anantatmula, V., 2010. Social and behavioral influences on team
CE
process. Proj Mgmt Jrnl 41, 89–98. 1380
Albert, R., Barab´ asi, A.L., 2002. Statistical mechanics of complex networks.
AC
Rev. Mod. Phys. 74, 47–97.
Andr´e, M., Baldoqu´ın, M.G., Acu˜ na, S.T., 2011. Formal model for assigning
1385
human resources to teams in software projects. Information and Software Technology 53, 259–275.
Bachmann, A., Bird, C., Rahman, F., Devanbu, P., Bernstein, A., 2010. The missing links: Bugs and bug-fix commits, in: Proceedings of the Eighteenth 58
ACCEPTED MANUSCRIPT
ACM SIGSOFT International Symposium on Foundations of Software Engineering, ACM, New York, NY, USA. pp. 97–106. Bagozzi, R.P., Dholakia, U.M., 2006. Open source software user communities: A
CR IP T
study of participation in linux user groups. Management Science 52, 10991115.
1390
Balthazard, P., Potter, R.E., Warren, J., 2004. Expertise, extraversion and group interaction styles as performance indicators in virtual teams. ACM SIGMIS Database 35, 41.
Bandinelli, S., Di Nitto, E., Fuggetta, A., 1996. Supporting cooperation in the spade-1 environment. IEEE Trans. Software Eng. 22, 841–865.
AN US
1395
Barreto, A., Barros, M.d.O., Werner, C.M., 2008. Staffing a software project: A constraint satisfaction and optimization-based approach. Computers & Operations Research 35, 3073–3089.
Basili, V., Shull, F., Lanubile, F., 1999. Building knowledge through families of
M
experiments. IEEE Trans. Software Eng. 25, 456473.
1400
ED
Bettenburg, N., Hassan, A.E., 2012. Studying the impact of social interactions on software quality. Empir Software Eng 18, 375–431. Bhattacharya, P., Iliofotou, M., Neamtiu, I., Faloutsos, M., 2012. Graph-based
PT
analysis and prediction for software evolution. 2012 34th International Conference on Software Engineering (ICSE) .
1405
CE
Bhattacharya, P., Neamtiu, I., 2010. Fine-grained incremental learning and multi-feature tossing graphs to improve bug triaging. 2010 IEEE International
AC
Conference on Software Maintenance .
Bird, C., Gourley, A., Devanbu, P., Gertz, M., Swaminathan, A., 2006. Mining
1410
email social networks. Proceedings of the 2006 international workshop on Mining software repositories - MSR 06 .
59
ACCEPTED MANUSCRIPT
Blaskovich, J.L., 2008. Exploring the effect of distance: An experimental investigation of virtual collaboration, social loafing, and group decisions. Journal of Information Systems 22, 27–46. Cataldo, M., Herbsleb, J.D., Carley, K.M., 2008. Socio-technical congruence.
CR IP T
1415
Proceedings of the Second ACM-IEEE international symposium on Empirical software engineering and measurement - ESEM 08 .
Cataldo, M., Wagstrom, P.A., Herbsleb, J.D., Carley, K.M., 2006. Identification of coordination requirements. Proceedings of the 2006 20th anniversary
AN US
conference on Computer supported cooperative work - CSCW 06 .
1420
Charette, R.N., 2005. Why software fails [software failure]. IEEE Spectrum 42, 42–49.
Chen, G., Donahue, L.M., Klimoski, R.J., 2004. Training undergraduates to
tion 3, 27–40.
1425
M
work in organizational teams. Academy of Management Learning & Educa-
Chen, W.N., Zhang, J., 2013. Ant Colony Optimization for Software Project
ED
Scheduling and Staffing with an Event-Based Scheduler. IEEE Transactions on Software Engineering 39, 1–17.
PT
Crowston, K., Scozzi, B., 2002. Open source software projects as virtual organisations: competency rallying for software development. IEE Proc., Softw.
1430
CE
149, 3.
Damian, D., Helms, R., Kwan, I., Marczak, S., Koelewijn, B., 2013. The role
AC
of domain knowledge and cross-functional communication in socio-technical
1435
coordination. 2013 35th International Conference on Software Engineering (ICSE) .
DeLone, W.H., McLean, E.R., 2003. The delone and mclean model of information systems success: A ten-year update. Journal of Management Information Systems 19, 9–30.
60
ACCEPTED MANUSCRIPT
Drigas, A., Kouremenos, S., Vrettos, S., Vrettaros, J., Kouremenos, D., 2004. An expert system for job matching of the unemployed. Expert Systems with
1440
Applications 26, 217–224.
CR IP T
Ducheneaut, N., 2005. Socialization in an open source software community: A socio-technical analysis. Computer Supported Cooperative Work (CSCW) 14, 323368. 1445
El Emam, K., Koru, A.G., 2008. A replicated survey of it software project failures. IEEE Software 25, 84–90.
AN US
Espinosa, J.A., DeLone, W., Lee, G., 2006. Global boundaries, task processes
and is project success: a field study. Information Technology & People 19, 345–370. 1450
Feldt, R., Angelis, L., Torkar, R., Samuelsson, M., 2010. Links between the personalities, views and attitudes of software engineers. Information and
M
Software Technology 52, 611–624.
Fenton, N.E., Neil, M., 1999. A critique of software defect prediction models.
1455
ED
IEEE Transactions on Software Engineering 25, 675–689. Fitzpatrick, E.L., Askin, R.G., 2005. Forming effective worker teams with multi-
PT
functional skill requirements. Computers & Industrial Engineering 48, 593– 608.
CE
Gorla, N., Lam, Y.W., 2004. Who should work with whom? Communications of the ACM 47, 79–82.
Grindrod, P., Parsons, M.C., Higham, D.J., Estrada, E., 2011. Communicability
AC
1460
across evolving networks. Phys. Rev. E 83.
Hardiman, L., 1997. Personality types and software engineers. Computer 30, 10–10.
61
ACCEPTED MANUSCRIPT
Herbsleb, J., Mockus, A., 2003. An empirical study of speed and communication in globally distributed software development. IEEE Trans. Software Eng. 29,
1465
481494.
CR IP T
Herbsleb, J.D., 2007. Global software engineering: The future of socio-technical coordination. Future of Software Engineering (FOSE 07) .
Holmstrom, H., Conchuir, E., Agerfalk, P., Fitzgerald, B., 2006. Global software
development challenges: A case study on temporal, geographical and socio-
1470
cultural distance. 2006 IEEE International Conference on Global Software
AN US
Engineering (ICGSE06) .
Katz, L., 1953. A new status index derived from sociometric analysis. Psychometrika 18, 39–43. 1475
Lakhanpal, B., 1993. Understanding the factors influencing the performance of software development groups: An exploratory group-level analysis. Informa-
M
tion and Software Technology 35, 468–473.
Lappas, T., Liu, K., Terzi, E., 2009. Finding a team of experts in social net-
ED
works. Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining - KDD 09 .
1480
PT
Li, C.T., Shan, M.K., 2010. Team formation for generalized tasks in expertise social networks. 2010 IEEE Second International Conference on Social
CE
Computing .
Licorish, S.A., MacDonell, S.G., 2014. Understanding the attitudes, knowledge sharing behaviors and task performance of core developers: A longitudinal
AC
1485
study. Information and Software Technology 56, 1578–1596.
Licorish, S.A., MacDonell, S.G., 2015. Communication and personality profiles of global software developers. Information and Software Technology 64, 113– 131.
62
ACCEPTED MANUSCRIPT
1490
Lievens, F., van Dam, K., Anderson, N., 2002. Recent trends and challenges in personnel selection. Personnel Review 31, 580–601. Linberg, K.R., 1999. Software developer perceptions about software project
CR IP T
failure: a case study. Journal of Systems and Software 49, 177–192.
London, M., Beatty, R.W., 1993. 360-degree feedback as a competitive advantage. Hum. Resour. Manage. 32, 353–372.
1495
Luna-Reyes, L.F., Zhang, J., Ramn Gil-Garca, J., Cresswell, A.M., 2005. Information systems development as emergent socio-technical change: a practice
AN US
approach. Eur J Inf Syst 14, 93105.
MacQueen, J.B., 1967. Some methods for classification and analysis of multivariate observations, in: Proc. of the fifth Berkeley Symposium on Mathematical
1500
Statistics and Probability, University of California Press. pp. 281–297. Madey, G., Freeh, V., Tynan, R., 2002. The open source software develop-
1505
ED
Proceedings , 1806–1813.
M
ment phenomenon: An analysis based on social network theory. AMCIS 2002
Meneely, A., Williams, L., 2011. Socio-technical developer networks. Proceeding of the 33rd international conference on Software engineering - ICSE 11 .
PT
Meneely, A., Williams, L., Snipes, W., Osborne, J., 2008. Predicting failures with developer networks and social network analysis. Proceedings of the
CE
16th ACM SIGSOFT International Symposium on Foundations of software engineering - SIGSOFT 08/FSE-16 .
1510
AC
Moreno, J.L., 1934. Who shall survive?: A new approach to the problem of human interrelations. .
Nagappan, N., Murphy, B., Basili, V., 2008. The influence of organizational
1515
structure on software quality. Proceedings of the 13th international conference on Software engineering - ICSE 08 .
63
ACCEPTED MANUSCRIPT
Newman, M.E.J., 2001. Scientific collaboration networks. ii. shortest paths, weighted networks, and centrality. Phys. Rev. E 64. Ngo-The, A., Ruhe, G., 2009. Optimized resource allocation for software release
1520
CR IP T
planning. IEEE Transactions on Software Engineering 35, 109–123.
Oh, H., Labianca, G., Chung, M.H., 2006. A multilevel model of group social capital. Academy of Management Review 31, 569–582.
Olson, G., Olson, J., 2000. Distance matters. Human-Computer Interaction 15,
AN US
139–178.
Otero, L.D., Centeno, G., Ruiz-Torres, A.J., Otero, C.E., 2009. A systematic approach for resource allocation in software projects. Computers & Industrial
1525
Engineering 56, 1333–1339.
Otero, L.D., Otero, C.E., 2012. A fuzzy expert system architecture for capability assessments in skill-based environments. Expert Systems with Applications
Robertson, I.T., Smith, M., 2001. Personnel selection. Journal of Occupational
ED
1530
M
39, 654–662.
and Organizational Psychology 74, 441–472. Sawyer, S., 2001. Effects of intra-group conflict on packaged software develop-
PT
ment team performance. Inform Syst J 11, 155–178.
CE
Sawyer, S., 2004. Software development teams. Commun. ACM 47, 9599. 1535
Schuler, D., Zimmermann, T., 2008.
Mining usage expertise from version
AC
archives. Proceedings of the 2008 international workshop on Mining software repositories - MSR 08 .
Shin, Y., Meneely, A., Williams, L., Osborne, J.A., 2011. Evaluating com-
1540
plexity, code churn, and developer activity metrics as indicators of software vulnerabilities. IEEE Transactions on Software Engineering 37, 772–787.
64
ACCEPTED MANUSCRIPT
e Silva, L.C., Costa, A.P.C.S., 2013. Decision model for allocating human resources in information system projects. International Journal of Project Management 31, 100–108.
CR IP T
Sowe, S.K., Stamelos, I., Angelis, L., 2008. Understanding knowledge sharing activities in free/open source software projects: An empirical study. Journal
1545
of Systems and Software 81, 431–446.
Sun, J., Tang, J., 2011. A survey of models and algorithms for social influence analysis. Social Network Data Analytics , 177214.
AN US
Surian, D., Liu, N., Lo, D., Tong, H., Lim, E.P., Faloutsos, C., 2011. Rec-
ommending people in developers collaboration network. 2011 18th Working
1550
Conference on Reverse Engineering .
Tamburri, D.A., Lago, P., Vliet, H.v., 2013. Organizational social structures for software engineering. ACM Computing Surveys 46, 135.
M
Tsai, H.T., Moskowitz, H., Lee, L.H., 2003. Human resource selection for software development projects using taguchis parameter design. European Jour-
1555
ED
nal of Operational Research 151, 167–180. Tuckman, B.W., Jensen, M.A.C., 1977. Stages of small-group development
PT
revisited. Group & Organization Management 2, 419–427. Turley, R.T., Bieman, J.M., 1995. Competencies of exceptional and nonexceptional software engineers. Journal of Systems and Software 28, 19–38.
CE
1560
Wolf, T., Schroter, A., Damian, D., Nguyen, T., 2009. Predicting build failures
AC
using social network analysis on developer communication. 2009 IEEE 31st International Conference on Software Engineering .
Xu, J., Christley, S., Madey, G., 2006. Application of social network analysis to
1565
the study of open source software. The Economics of Open Source Software Development , 247–269.
65
ACCEPTED MANUSCRIPT
Xu, R., Wunsch, D., I., 2005. Survey of clustering algorithms. Neural Networks, IEEE Transactions on 16, 645–678. Yang, H.L., Tang, J.H., 2004. Team structure and team performance in is
CR IP T
development: a social network perspective. Information & Management 41,
1570
335–349.
Zhu, H., Zhou, M., Seguin, P., 2006. Supporting software development with
roles. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Sys-
AC
CE
PT
ED
M
AN US
tems and Humans 36, 1110–1123.
66
CR IP T
ACCEPTED MANUSCRIPT
AN US
1575
Roberto Latorre received the B.S. degree in computer engineering and the Ph.D. in computer science and telecommunications from Universidad Aut´ onoma de Madrid, Madrid, Spain, in 2000 and 2008, respectively. He has an extensive experience in software industry, where he worked with multiple companies as IT 1580
architect and software developer. Since 2002 he has been a professor at Escuela
M
Politcnica Superior at Universidad Autnoma de Madrid, where he currently is
AC
CE
PT
ED
Profesor Contratado Doctor.
1585
Javier Su´ a]rez is a software engineer and a researcher at Kymatic Soluciones Informticas. He received the I B.S. degree in computer engineering from Universidad Politcnica de Madrid. He is an expert in Speech technologies and software development.
67