Learning and best practices for learning in open-source software communities

Computers & Education 63 (2013) 98–108

Contents lists available at SciVerse ScienceDirect

Computers & Education journal homepage: www.elsevier.com/locate/compedu

Learning and best practices for learning in open-source software communities Vandana Singh a, *, Lila Holt b,1 a b

School of Information Sciences, University of Tennessee, Knoxville, 449 Communications Building, 1345 Circle Park Drive, Knoxville, TN 37996-0341, USA Instructional Technology, 415 Claxton Complex, 1122 Volunteer Boulevard, Knoxville, TN 37996-3400, USA

a r t i c l e i n f o

a b s t r a c t

Article history: Received 14 June 2012 Received in revised form 30 November 2012 Accepted 1 December 2012

This research is about participants who use open-source software (OSS) discussion forums for learning. Learning in online communities of education as well as non-education-related online communities has been studied under the lens of social learning theory and situated learning for a long time. In this research, we draw parallels among these two types of communities and explore what can be learned from open-source software communities about online learning. Thematic network analysis was used to code the qualitative data from the open-ended questions in the survey and the interviews. The results indicate that learning in online open-source software communities encompasses much more than just learning about the software being discussed. 283 Open-source forum participants were surveyed, and 21 were interviewed to develop an understanding of the challenges to learning in these communities as well as to identify the practices that promote learning. Identifying these practices helps to understand online learning and enables the integration of best practices into online education. Ó 2012 Elsevier Ltd. All rights reserved.

Keywords: Computer-mediated communication Cooperative/collaborative learning Interactive learning environments Learning communities Teaching/learning strategies

1. Introduction Open-source software (OSS) communities have been a source of learning for participants since their creation. Online education is increasingly becoming a dominant part of our education system. The Sloan Consortium found that 66% of postsecondary institutions reported an increased demand for new distance education course offerings as well as an increased demand for their existing distance education coursework (Allen & Seaman, 2010). According to Allen and Seaman (2010), “The 2010 Sloan Survey of Online Learning reveals that enrollment rose by almost one million students from a year earlier. The survey of more than 2500 colleges and universities nationwide finds approximately 5.6 million students were enrolled in at least one online course in fall 2009, the most recent term for which figures are available.” Additionally, the research firm Ambient (Ambient Research Firm, 2012) predicts that 3.55 million students will be taking classes online by 2014, while the number of students taking all of their courses in on-campus classrooms will drop to 5.14 million. In an effort to adapt to these demands, scholars are striving to develop tools and pedagogies to facilitate learning in online environments. While different avenues exist for this exploration, one way to identify new ways for facilitation is by examining current online ecosystems that incorporate learning. Various online communities exist that may provide valuable insights into online learning. In particular, a positive influence of interacting in open-source software communities has been the opportunity to learn while participating (Koh, Kim, Butler, & Bock, 2007). In an effort to further the knowledge base of pedagogies, this paper contributes insights on how the OSS community uses the forums for learning and solving problems. It explores the motivations for joining these communities, the learning that occurs in the online communities, and the challenges to learning. Not all the activities in an online forum lead to learning, so this paper examines the specific practices identified by participants and moderators that aid in learning in OSS forums. Through the guidance of multidimensional best practices inquiry (Petr & Walter, 2005), the practices of the participants of open-source software communities are evaluated to create a list of best practices that support learning. Understanding these practices that promote learning in OSS communities can guide us to develop similar learning

* Corresponding author. Tel.: þ1 865 974 2785. E-mail addresses: [email protected] (V. Singh), [email protected] (L. Holt). 1 Tel.: þ1 865 805 0538. 0360-1315/$ – see front matter Ó 2012 Elsevier Ltd. All rights reserved. http://dx.doi.org/10.1016/j.compedu.2012.12.002

V. Singh, L. Holt / Computers & Education 63 (2013) 98–108

99

communities in online education. The fact that these communities, seemingly autocratic and without the presence of a designated instructor/teacher, play an active role in learning for the participants is significant and deserves an in-depth inquiry. In conclusion, this article presents ways in which the best practices identified from an OSS community can be integrated into online education. 2. Literature review This section presents research to understand the attitudes and motivations for joining OSS communities and to understand the practices and activities in online communities, including open-source communities and online education communities. Research related to forumbased communities is presented, and the role of users in these communities is described. The key objective of this research is to understand what promotes learning in these online communities; hence, the theories and research on social learning that describe the process of learning in a social environment, online as well as in physical communities, are presented. 2.1. Attitudes and motivations in open-source software communities Researchers interested in open-source software communities have long been asking questions such as, “What are the factors driving participants to contribute to OSS communities?” and “How do OSS project groups keep these participants engaged in the communities?” There are many studies that provide insights into the motivations of open-source software participants. Table 1 presents the past research on the motivations for individual participation in open-source software communities. This table adds to the literature review on individual motivations listed in the open-source research review done by Crowston et al. (2012). A variety of lenses exist for the examination of attitudes and motivations in online communities. For example, Xu and Jones (2010) discuss the impact of identification and obligation on an OSS developer’s voluntary efforts in software creation. They also discuss the types of trust, such as affective and cognitive trust, and emotional attachments that must develop between the members of a community in order to hold that community together. Hew and Hara (2007) found that the most common motivators for users contributing knowledge to online environments were feelings of collectivism and reciprocity, while Burnett, Besant, and Chatman (2001) address how normative behaviors manifest within online communities and how such behaviors might manifest differently than they do in face-to-face situations. Furthermore, in terms of the learning that occurs in online contexts, various researchers have studied the impact not only in forums but also in wikis and Community Question Answering (CQA) sites. Bruns and Humphreys (2007) found that within the wiki-based learning environment they analyzed, it must be creative, collaborative, and communicative in order for the users to learn and contribute to the community. 2.2. Practices and activities in online communities Within the online forum communities, various practices and tools have supported both user and moderator activities. The following section describes some of those practices and how they are used within the forums. 2.3. Forum member practices and tools Musicant, Ren, Johnson, and Riedl (2011) looked at the mentoring practices within Wikipedia. They found that most contributors to Wikipedia are more focused on “establishing their legitimacy rather than acting proactively to guide, protect, and support the long-term grown of adoptees” (p. 173). In the paper, they suggest strengthening communication between mentors and protégés in the “Adopt-aUser” program, including threaded discussions, simplicity in cross-page communication, and more reliable notifications of message receipts. They also encouraged protégés and mentors to pursue social connections outside of Wikipedia to strengthen such a relationship. Searching forums has been found to be critical for learning. Both the structure of the information that creates the documentation and the tools used to find information impact participant learning. For example, research of online tools includes the use of search tools within Table 1 Literature review on individual participation in online learning communities. Authors

Individual motivations and categories identified

Hars and Ou (2002)

Internal factors (e.g., intrinsic motivation, altruism) and external rewards (e.g., expected future returns, personal needs; compensation) Identification as a Linux user; personal improvement; supporting independent software and networking within the Linux community, pure enjoyment of programming Reputation Reward motives such as career development; personal improvement

Hertel, Niedner, and Herrmann (2003) Hann, Roberts, and Slaughter (2004) Hann, Roberts, Slaughter, and Fielding (2002) Hars and Ou (2002) Orman (2008) Ghosh (1998) Shah (2006) Ye and Kishida (2003) Lakhani and von Hippel (2003) Lerner and Tirole (2002) Fang and Neufeld (2009) Shah (2006) Wu, Gerlach, and Young (2007) Bagozzi and Dholakia (2006) Bergquist and Ljungberg (2001)

Enjoyment based motivations such as fun Sharing and learning opportunities User needs; reciprocity Individuals’ motives are not static, but evolve over time may join and leave once their needs are met or choose to stay and participation becomes a hobby Social identity; contribution to group action Giving to the community and learning by peer review process

100

V. Singh, L. Holt / Computers & Education 63 (2013) 98–108

a Linux community using Google’s search engine (Filho, Olson, & Geus, 2010). The researchers found that well-structured documentation is more effective for novice users as opposed to finding information via keyword search. Additionally, they found that such information allows users to browse as opposed to knowing exactly what they want to find. Cao, Cong, Cui, Jensen, and Zhang (2009) looked at search tools from CQA sites such as Yahoo! Answers. They write that such communities contain much important information that should be available via question search. They propose a new method of exploiting the classifications of questions to improve the information found via question search. The practice of rating users for information value has been researched in forums. Motoyama, McCoy, Levchenko, Savage, and Voelker (2011) looked at how users are rated in underground forums, or forums “where participants exchange information on abusive tactics and engage in the sale of illegal goods and services” (p. 71). They found that the rating system offered more credibility to users’ threads. Savolainen (2011) examined the criteria users applied in the judgment of information quality in online discussion forums. He found that the only types of judgment exhibited in such forums were negative in nature, whereas positive judgments took on a more silent role in the forums. Kim and Han (2009) examined users’ trust and reputation in CQA sites such as Yahoo! Answers. They recommend that the providers of the site regularly monitor the quality of information and the amount of time in which it is delivered. These practices may increase the trust in overall information quality. Tools have been made available in online environments to support this ranking of users. Bouguessa, Dumoulin, and Wang (2008) found that within CQA sites such as Yahoo! Answers it is important to develop tools that identify the most authoritative users. Having a ranking that identifies the users that contribute high-quality content can contribute to more learning in the community. Harper, Moy, and Konstan (2009) examined types of posts in CQA sites such as Yahoo! Answers. They defined the difference between conversational questions and informational questions and the potential archival value of each. Conversational questions by nature had lower archival value because they did not contribute much knowledge to the overall community and were more social in nature. They suggested using tools to classify questions as either conversational or informational in order to contribute to more learning from such sites. Hansen (2009) found similar results within a technical support community, implying that “the same characteristics that make the conversation successful (personal, immediate, and socially engaging nature) make reuse of the conversation problematic” (p. 155). Hansen discusses developing search tools that help support the most effective conversation reuse and contribute to learning from the site. 2.4. Moderator practices Along with analyzing the general users, best practices for moderators have been examined in research. Yates, Wagner, and Majchrzak (2010) define moderators in wiki communities as “knowledge shapers” who modify posts. Rather than contributing new posts, knowledge shapers are users who “move the community forward by looking for differences and similarities in perspectives, integrating where they can, and highlighting the differences when they cannot” (p. 553). Panzarasa, Opsahl, and Carley (2009) also describe the role of the moderator as being responsible for “improving the security of communication and enhancing the effectiveness of information diffusion” (p. 930). With regard to keeping a library of information in order to contribute knowledge, Sureka, Goyal, and Rastogi (2011) analyzed the bug report data of Mozilla Firefox and found that archives and databases help provide “hidden and interesting patterns useful to project managers and developers” (p. 203). For example, Barzilay, Hazzan, and Yehudai (2011) looked at how developers provided examples in LinkedIn group discussions. They found a diverse range of approaches in the examples provided. While some provided many detailed examples, other developers opposed using examples altogether. Still another role of the moderator is to help integrate newcomers. Nov, Naaman, and Ye (2010) examined what could potentially motivate newcomers to participate in an online community and found that more experienced users may be motivated by self-development purposes, while this has the opposite effect on newcomers. In terms of helping newcomers, Thom-Santelli, Cosley, and Gay (2009) found tensions between the experts and novices of a mobile social tagging system for a museum collection. They found that experts exhibited territoriality toward newcomers’ contributions to the system and museum in general. Experts were also more likely to negatively criticize contributions made by others. The researchers found that such behaviors may intimidate and discourage novices from contributing to the site. Jones, Poulsen, Maiden, and Zachos (2011) suggest encouraging all users, including lurkers, to contribute to the site by “asking participants to take turns in facilitating, and introduction of a visible reward system for different kinds of contribution” (p. 350). They suggest that encouraging all types of users to participate leads to more effective collaborations. Ankolekar, Sycara, Herbsleb, Kraut, and Welty (2006) developed a prototype Semantic Web system called “Dhruv” which they suggested would aid novice developers to better understand error reports. This understanding would then help novices to participate in error resolution more effectively. They cite a “lack of documentation in OSS communities” which may turn away novices, so a system such a Dhruv can help compensate for such a lack of documentation (p. 583). 2.5. Social learning in online communities Social learning theory (Bandura, 1977) suggests that people learn by observation in social situations and that they will begin to act like people they observe even without external incentives. Social learning theories posit that learning is fundamentally social in nature and that it involves interactions of people at some level, whether direct or mediated. Swan and Shea (2005) argue that social learning theories must be addressed in any discussion of online learning. In the situated learning view of learning communities, such as the one presented by Bransford, Brown, and Cocking (1999), knowledge is inseparable from practice, and practice is inseparable from the communities in which it occurs. In other words, the practices of a community are an integral part of the learning that occurs in that community. This is also reflected in the concept of communities of practice as defined by Wenger (1997); the communities studied clearly demonstrate that communities of practice are characterized by mutual engagement, joint enterprise, shared repertoire, and negotiated meaning. In his view, the authentic communities of learning inherently have these characteristics, and these are also the characteristics that can be developed to foster communities of practice that promote knowledge creation and learning. An important part of Wenger’s notion of communities of practice is the idea that all learning is situated in practice and that all practice is essentially social in nature. It is these practices in open-source software

V. Singh, L. Holt / Computers & Education 63 (2013) 98–108

101

communities that most interests the author of this article. This research asks which practices promote learning for the members of these communities. Lave and Wenger (1991) characterized learning practices as processes of participation in which beginners are initially relatively peripheral to the activities of a community, but as they learn the practices their participation becomes more central. For an environment of apprenticeship to be a productive environment of learning, there need to be opportunities for learners to observe and then practice activities which move them into more “legitimate” participation in the community. The literature review shows that the reasons for joining online communities vary from getting problems solved to learning to reciprocity. The members’ experiences within these communities are shaped by how the community members interact with them and each other. The role of the existing community members is very important in integrating new members into the community. New members learn by interacting with the community and being exposed to the community’s knowledge. In this research, the participants of the forums were asked what practices from other forum members promote learning for them. The main objective is to understand what the community members can do to promote learning for newcomers and existing members alike. In summary, research has examined online communities and found that attitudes, motivations, tools, and practices in various online ecosystems do affect the interactions and the success of the communities. Practices and behaviors of community members have a very strong influence on learning and knowledge creation. In the next few sections, this paper presents results from a study that asked participants of open-source software communities why they join forums and which practices of forum members promote learning in these communities. The best practices identified by the community members are then grouped into best practices by different user groups and are used as guidelines or recommendations for online education environments that rely on discussion-based learning. 3. Research questions 1. What are the motivations for joining open-source discussion forums? 2. What are the forum participants learning in open-source discussion forums? 3. What are the practices that promote learning for forum participants?

4. Methods This study instituted a qualitative design consisting of observations of tools used in the forums and interactions in the forums, an openended questionnaire, and in-depth interviews of OSS forum participants, thus creating triangulation (Creswell & Miller, 2000). Initial tools were obtained by observation of the forums and a review of the literature. Following the development of the list of tools, an open-ended questionnaire was created. The questionnaire encompassed four goals. While some quantitative data was gathered, the use of the data was not for proving statistical significance but rather to create a clearer lens through which to examine the results. The first goal of the questionnaire was to identify characteristics of the respondents. Second, by using a questionnaire a greater number of responses could be obtained than solely through interviews. Third, the results from the questionnaire were used to help focus the interview questions. Finally, the questionnaire helped to identify participants who would be willing to be interviewed. Requests for completion of the questionnaire were sent to OSS forum owners as well as users of OSS email lists. Approximately 60 requests to forums/lists were solicited. The lists solicited were from open-source users and/or developers from at least 58 different groups. Examples of the lists include Firefox, Linux, Debian, Koha, Thunderbird, Ubuntu, Edubuntu, Seamonkey, Open-source CAD, Java, Fedora, OpenOffice, etc. from both user and developer sides. As an incentive for filling out the questionnaire, the participants were entered into a drawing for a gift certificate equal in value to a tablet computer. From the respondents to the questionnaire, 21 individuals were interviewed. These personal interviews allowed for the collection of rich data from which to examine the tools. The interviews were performed over a month with time in between the interviews for analysis and readjusting of the interview questions (Corbin & Strauss, 1990). During the course of the interviews, an emerging theme of best practices for users and moderators was identified. Subsequently, an extra question was added asking the participants to describe best practices in both the use and moderation of the forums. Upon completion of the interviews, codes and themes were developed. These codes and themes were then subsequently coded by a second coder to ascertain the consistency in coding data. The inter-rater reliability was .95. Research has argued that for qualitative studies where the questions change and evolve, such as in grounded theory, trying to maintain inter-rater reliability may result in a more superficial data analysis (Morse, 1997). Thus, as the in-depth interviews and evaluation continued, more detailed coding and analysis were conducted, and no further inter-rater reliability was assessed. Within the practices and activities, some have been identified as “best practices” to aid in use of the forums. Resnick and Vaughan (2006) define best practices as “certain design ideas [that] have shown a modicum of superiority to other ideas and have been adopted by respected practitioners in at least some circumstances” (p. 781). Identifying best practices from the responses follows the steps for a multidimensional best practices inquiry (Petr & Walter, 2005). Through the qualitative process of coding, question formation and desired outcomes were identified. Subsequently, the results of the literature and the findings of the questionnaires and interviews created multidimensional sources of knowledge from which to base the results. Through the lens of the literature and the respondents’ input, thoughts were coded and themed, creating a basis for summary as well as comparison and contrast. The quality of the sources, more than quantity, was examined, and value criteria identified through the study were applied as will be noted in the results. The results of this process are presented in Fig. 1 in the thematic network analysis diagram. The thematic network analysis diagram was created using the structure of the thematic network as defined by Attride-Sterling (2001). The lens through which this study’s results are evaluated is a combination of pragmatism and constructivism, as pragmatism allows for the view that one epistemology does not apply in all cases (Creswell, 2007), and constructivism allows for participants to interact in the coconstruction of knowledge (Hatch, 2002).

102

V. Singh, L. Holt / Computers & Education 63 (2013) 98–108

Fig. 1. Thematic network analysis for best practices in open-source online communities.

5. Results and discussion Results were gathered from participants from a variety of OSS forums; the total number of respondents to the questionnaire was 223. Of these respondents, 94.2% were male, with 5.8% being female. The age range was from 18 to 79, with the average age being 41. Overwhelmingly, the ethnicity was Caucasian (76%). Additionally, 68.6% of the respondents were college graduates. Thus, the average respondent was an educated, white male in his mid-40s. 5.1. Motivations to join forums As a response to the first research question, the forum participants from this study echoed the responses from the literature review. The OSS forums benefited a participant by allowing him or her to join a community that shares an interest in using and promoting OSS. Respondents to the survey characterized the forums and their motivation to participate as “outstanding,” “revolutionizing,” “cutting edge,” “love to learn,” and “giving back.” Hence, the motivations identified parallel research findings of reciprocity (Hew & Hara, 2007) and creativity (Bruns & Humphreys, 2007). Additionally, forums extended and supplemented documentation or manuals for open-source software, thus allowing participants to look for information beyond formal written documentation. Research results found that participants joined the OSS forums for several reasons. Problem solving was indicated as the main reason participants joined the forums. As noted by a respondent, “one must use forums to improve and find quickly an answer to a problem.” Below is a representative quote from one of the interviews: “I think I’d just conclude that forum learning is, IMHO (in my humble opinion), the hands-down best way to learn open-source software available, since it changes too fast for printed publications to keep up, and because it’ll always be able to help with the issue you’re having, regardless of how obscure or hard-to-describe (as in an index or table of contents).” Table 2 lists the motivations shared by forum participants in the survey. Looking for answers for problems, understanding issues related to software, helping others, and learning were the most frequently cited motivations for participation among the survey respondents. 5.2. Learning in forums The results for the second research question highlighted the different things that the forum participants had to learn when they join these forums. The “learning” that is mentioned in the previous section mostly refers to learning the content of the forums, but that is not the only type of learning that is needed in order to integrate into the discussion forums. The participants of the survey and interviews mentioned the following types of learning for integrating into the communities: 1) 2) 3) 4)

Learning Learning Learning Learning

how to get help in the forums the forum subject forum expectations about current developments in the product

V. Singh, L. Holt / Computers & Education 63 (2013) 98–108

103

Table 2 Motivations shared by surveyed forum participants.

5) 6) 7) 8)

Learning Learning Learning Learning

to to to to

Motivation for joining forums

# of Survey respondents

Looking for answers to problems Understanding and discussing software issues Sense of community – everyone using same resources Help others Learning Ask/answer Continued learning/SDL Saves time Personal reason/work Flame Need hand holding Teaching

70 68 67 61 60 10 4 2 2 1 1 1

communicate in the forum make good posts search for specific things help others

For identifying the best practices that promote learning in these online communities, the results from the motivations question were used to create criteria for what makes a best practice. A practice was considered a best practice if it 1) allows for problem solving, 2) increases the knowledge base for all, 3) allows for learning and improvement, and/or 4) allows a user to quickly achieve his or her personal goal for participation. The practices as reported by the forum participants were then evaluated in light of these goals. A list of best practices from different user groups in the forums is presented below. 5.3. Best practices thematic network The results of this analysis are based on open-ended questions from the survey and open-ended interview questions. Therefore, instead of a quantitative analysis, the researcher created a thematic network analysis as defined by Attride-Sterling (2001). In this case, the global theme is best practices and the organizing themes are newcomer best practices, user best practices, moderator best practices, and administration best practices. The basic themes are the specific best practices presented in the diagram and the write-up following the diagram. Fig. 1 shows the thematic network analysis for best practices in open-source online communities. Some best practices for users and newcomers are common and hence they are connecting to both the organizing themes in the diagram. They are the same best practices but they occurred in both contexts. 5.4. Best practices for forum administration Best practices begin with the creation of the forums themselves and the focal point for which they exist. A comment from a respondent noted that a forum should be “very simple to not only get information but to ask for information. And that’s key, it’s gotta be easy to get onto, it’s gotta provide relevant information, and it’s gotta make the user comfortable to not only ask the questions but comfortable in answering the questions as well.” This indicates that an initial best practice is that a forum should be easy to access. This best practice is further substantiated by another respondent: “some of the forum owners or forum moderators simply require too much information to gain access to the forum.” Four interview responders out of 21 insisted that the ease of access is very important to enhance participation in forums. While the literature review did not directly address forum access, ease of access relates directly to time and also possibly to the sense of trust necessary to participate, as seen in Xu and Jones (2010). From these responses, the subsequent best practice was defined: Best practice 1: A forum should be easy to access. Unless a forum is easy to access, no learning will occur due to a lack of participation. Thus, this best practice meets the goal of allowing learning. The ease of entry supports the importance of time as a factor for participation. 5.5. Best practices for forum newcomers As a best practice for new users, they are encouraged to learn how the forums work before posting. The “newbies” as they are referred to in forums are expected to read the rules for a particular forum before participating. A forum user indicated “usually, when one joins a forum., they get a link to the written rules... I wish everyone would actually read those and keep them in mind.” This practice was shared by one moderator out of the five moderators interviewed. A further best practice identified and repeated by five interview respondents was to find a mentor to aid in integration into the forum. A respondent indicated this finding: “Also, find a mentor, someone who will work with you ... teach you the ropes.” These findings confirm the literature of the positive mentoring practices of Kim and Han (2009) when incorporating newcomers into the use of wikis. In summary, the best practices for newcomers are as follows: Best practice 2: Newcomers should read the rules before participating. Best practice 3: Newcomers should find a mentor. These practices are a safeguard that may help reduce clutter from inappropriate posts in the forum. Thus, they achieve the goal of helping all participants to find information more quickly.

104

V. Singh, L. Holt / Computers & Education 63 (2013) 98–108

5.6. Best practices for forum users In continuing best practices, forum participants should always check Frequently Asked Questions (FAQs) and search the forum for answers before posting a question. As noted by a respondent, “On OSS forums, people are generally expected to search for their own answers BEFORE turning to others.” Furthermore, another respondent indicated, “the other thing is I would like to let people – reading, reading some technical you know manuals or actual – before they ask the question because it helps to ask a better question maybe not to find the right answer but to it’ll definitely to help them ask the right question.” The result of this best practice is to both ask better questions and reduce duplication of posts in the forum, thus reducing the number of posts and threads to search through. Overall, then, self-directed initiatives to learn from documented sources are expected by forum members. Not only does this reduce the duplicate documentation load, but forum participants also learn to ask the right questions. A best practice for posting to the forum includes making a “good” post. Characteristics of a good post as identified by interview and survey respondents are that it is friendly, detailed, contributes to forum knowledge, and captures forum members’ attention. Among the interviewees, 14 out of 21 said that the posts should be friendly, 17 said that the posts should be detailed, six said it should contribute to forum knowledge, and four said that the post should capture attention. While one participant noted that he enjoyed a good “flaming” (i.e., being derogatory toward a poster), in general flaming and rude posts were identified as undesired in forums, and comments stressed the importance of friendliness. More than half of interviewees (13 out of 21) said that flaming should not happen and that it increases the time to read the messages to get to relevant posts. These results parallel the literature, where it was found that archiving conversational posts was more difficult and created additional posts to search through that lacked additional knowledge information (Hansen, 2009; Harper et al., 2009). The amount of detail required in posting problems on a forum was also noted by the responders to aid the poster’s learning while not wasting the “experts’” time: “And that’s, that’s part of the idea, every teacher knows that a student takes a little long to formulate the question. He has a better chance of stumbling across the answer and also there’s a chance of figuring it out. In case in any of his neurons are actually working. So, so it’s partly a teaching exercise but it’s also about self-defense for somebody like me cause I don’t want just waste my time in a guessing game with somebody.” Taking the time to research and post details afterward was identified as critical to learning and forum function. Again, these details leave little room for off-topic social conversation. Detailed posts included being as specific as possible when posting either a problem or a solution. Problem posts should include specific environments, error messages, sequences of events, etc. After posting a problem, the poster should check back to learn how to solve it. A forum participant noted, “I always encourage people to report back when they present a problem. In most cases they do and are elated that all is now functioning the way it should. That’s a learning process.” One moderator out of the five interviewed insisted that solution posts should require the poster to test the solution before posting as well as provide a detailed solution and summary. Fourteen out of 21 interviewees reiterated that solution posts may include hyperlinks to documentation or previous solutions. Citing where solutions may be found was also a best practice to help participants solve problems and learn, as indicated by this respondent: “you know just to say ‘hey this is where I found this information and it took me’ however long it took me to find it.” It was also noted that if the solution was rather short, it was better to simply post solution steps on the forum as opposed to having the other participants jump to other links. As forum members indicated, one reason they chose to participate in the forums is to keep up with new advances in the specific software. Thus, posts should contribute to forum knowledge. A forum member indicated, “If a question not only helps you but others as well, it would make it a good question as we share our thoughts and experience while replying to it.” In addition, creating posts that capture the attention of the forum participants aids in that endeavor. Another best practice that three out of five moderators mentioned is to make sure to identify where in the forum to post a question. In other words, is the question in the right place? Respondents also disclosed that a best practice is to keep personal copies of information and locations for future reference. One respondent imparted, “If I find something that’s relevant to me that’ll help me out in the future, I’ll cut that out and go post it on a blog page.” Still another respondent replied, “If I find something that took me a long time to discover, that I’ll likely need again, I’ll copy/paste it into a .txt file on my backed-up drive, so that I can use it later.” This reduces the amount of time in future searches of the forums. In summary, the best practices and goals met are as follows: Best practice 4: Always search for an answer before posting a question. This helps by reducing the size of search material on the forum, aiding in better question formation, and saving people time in answering. Additionally, it helps the participants learn by possibly finding solutions themselves before posting. Best practice 5: All participants should post questions that capture forum members’ attention and are detailed, friendly, and contribute to forum knowledge. Posts that add to forum knowledge are important to the participants. Best practice 6: Details in posts should include the environment, a complete description of the procedure taken, any error messages, etc. as applicable to the question. A post with more details leads to better question formation. This increases the available knowledge base as well as helps the participants to solve the problem. Best practice 7: Questions should be posted in the appropriate place in the forum. Organization is important for better access to the overall information and for reducing clutter. Best practice 8: Participants who post questions should check back for solutions and provide feedback on those solutions. Feedback on correct or incorrect solutions creates a complete artifact for future participant reference and increases the knowledge base. Best practice 9: Participants who provide solutions to problems should test their solutions before posting. This reduces the clutter of incorrect information and creates a more accurate knowledge base. Best practice 10: Solutions should include hyperlinks and citations where further information or solutions can be found. This practice provides for a more complete learning opportunity while allowing quicker access to the information. Best practice 11: Keep information that might be needed again in a private library for easy access. Experienced users noted this was an efficient procedure and thus reduced future search time.

V. Singh, L. Holt / Computers & Education 63 (2013) 98–108

105

5.7. Time-saving practices Time (defined as the speed of finding information in the forums) was a theme interwoven throughout the tools and behaviors. Being able to find information quickly and thus reducing the time spent to learn was a primary focus for the participants. Tools were identified that saved time. Among them were assigning tags, subscribing to threads (either via RSS or email), using avatars to quickly decide which answers might be more relevant or correct, using screenshots, monitoring hot topics, etc. These tools aided in both saving time and identifying accurate information that helped the participant learn. The following example demonstrated both the time-saving aspects of tools: “An uploaded avatar is important because a lot of us associate an avatar with a particular user and it helps us quickly realize if this is a ‘good poster’ with valuable insights or some dumb spammer or wannabe who hasn’t got a clue. Makes reading the forums faster.” As with the tools previously identified, certain behaviors were identified as best practices for saving time specifically. Respondents found that having to read posts that “flamed” (i.e., rude/ugly responses to a previous poster or an online argument) were a waste of time, and 13 out of 21 interviewees said that users should not flame. Since the respondents did not have enough time to read for pleasure anyway, having to traverse the flaming posts was irritating. Also, another behavior that would save time is making detailed posts (both when asking questions and providing answers) so the reader does not have to hunt for further information. As noted by a respondent, “Ask intelligent questions... The quality of the reply is often directly related to the quality of the question. A lot of us ‘experts’ are old-timers and we expect a certain amount of detail. I don’t have time, generally, to research someone’s problem too deeply, so the more detail and useful information/ description they can provide, the better.” The time it takes to post a detailed question has another advantage for learning. Detailed answers and questions are especially helpful for the visually impaired. That is, having all information in the post without having to link to other sites was much easier for the visually impaired screen readers to navigate. Still other behavior that would save time is reading the first and last post of a thread. The first post would provide an overview of the problem, while the last post would most likely indicate whether the problem had been solved. Thus, one would not have to waste time reading other posts if no relevant information in the thread was found. A summary of practices that focus on saving time is as follows: Best Practice 12: Use tools as appropriate. Best practice 13: Don’t flame – this only creates more junk to read through. Best practice 14: Provide details for most items without links. This is especially time-saving for visually impaired users. Best practice 15: Read the first and last responses in a post to see if the question/answer is appropriate for your situation. 5.8. Moderator best practices Best practices for moderators are minimal but were identified as essential by forum participants. A best practice for forum moderators is to keep moderation to a minimum as much as possible; six non-moderators out of 16 emphasized this point. For example, a respondent related, “I’m a big fan of very limited moderation... moderation should be kept to A: moving posts to the correct subforum, B: keeping the discussion civil and on track and C: handling ‘problem users.’” Although the desire was for minimal moderation, moderators are charged with keeping order and harmony in the forum, as the following response notes: “For instance, for some reason people ask the stupid questions and people just keep posting on the poor guy telling ‘hey you stupid guy what are you asking? You’re an idiot’ and other things. All the things are unnecessary and uncalled for. So the moderator should be active, and should prevent these situations.” Nine non-moderators said that moderators should help and encourage the newbies; four even said that if needed, moderators should do “hand holding” for newcomers. Respondents also indicated that the moderator is the one to set precedence of being tactful and not on a vendetta against forum participants. Additionally, it is important for moderators to keep unanswered posts from becoming buried before they are addressed. Threads that become too long should be examined by a moderator and reorganized using special tools for ease of following and learning. Moderators should also aid in keeping information on the forums accurate, as the following respondent indicates: “Sometimes you get an ‘expert’ who is really out to lunch and can harm others if they follow bad advice. Moderators and others have to tactfully deal with them – fast.” Again, respondents think tact and minimal interference are key for moderators. Lastly, the role of the moderator is one of being helpful and respectful to forum members. Moderators should not be hesitant to take on the role of mentor for new members. When talking about helping the newcomers, one moderator indicated that “I have a library of 200 screenshots that I use to choose to illustrate points I’m trying to make.” Ten interviewees claimed that they copy content offline to help others on the forum, and another ten said that they use sticky notes to refer to previous threads. In summary, the moderator best practices are as follows: Best practice 16: Keep moderation to a minimum. Do only what is required to keep harmony on the forum. This helps reduce “flame” and excessive off-topic posts while allowing for learning. Best practice 17: Help newcomers integrate into the forum. This increases participation and allows for an expanded knowledge base. Best practice 18: Keep a private library of helpful materials for newcomers and others that can easily be copied and pasted in response to forum members. This reduces response time in helping with repeatedly asked questions and keeps the excessive material off the forum for all to traverse. Best practice 19: Keep posts aligned and unanswered questions from being buried – help participants find information easier. Best practice 20: Set the tone for the forum as one of tact and friendliness. This encourages participation, increasing the posts and the knowledge base while reducing the posts of extra-curricular socialization. Best practice 21: Make sure responses are accurate by monitoring people who continually post misinformation. This helps save participants’ time in following false leads and misinformation. When examining moderator best practices, the concept of filing notes and examples in a personal library for retrieval to help newcomers and others with frequent needs confirms the studies of Sureka et al. (2011) and Barzilay et al. (2011). In both cases, the archives, libraries, and examples retained by the moderators were used in interacting with the social communities. This study confirms that as a best practice for

106

V. Singh, L. Holt / Computers & Education 63 (2013) 98–108

moderators. Additionally, best practices for moderators were identified based on keeping information organized for ease of retrieval by forum members and not allowing the forum to be cluttered with flaming or inaccurate information. 5.9. Implications of best practices for distance education/online education Based on the above best practices that promote learning in open-source software communities and the theories of social learning mentioned in the literature review section, this section presents some recommendations for promoting learning in discussion forum-based online education. Participation in forums is a pre-requisite for any community to exist and any learning to happen. During the interview, one of the participants mentioned that it is difficult to get participation in distance education settings; therefore, the type of community that is built in open-source forums is not seen very often in a classroom situation. He said, “In 4 years, the only time I ever got students to actually participate in forums (I taught Linux courses at the Community College level to degree and transfer students) was to grade them on it.” This quote from the participant was quite significant for the author who also teaches online courses in a graduate program. The fact that the students only participate when they are forced to through incentives has been a point of discussion in the literature for many years (Burkett, Leard, & Spector, 2004; Klisc, McGill, & Hobbs, 2009; Laurillard, 2002; Leh, 2002; Seo, 2007). Social learning theory tells us that interaction among members of a community is essential for learning, and communities of practice are identified by the presence of mutual engagement, joint enterprise, shared repertoire, and negotiated meaning. In classroom settings, these characteristics have to be created to ensure that a community of practice is formed. The fact that these are students from a shared class should be able to create these characteristics of the community. This is not always the case in classroom settings, but we see that the online communities of open-source software exhibit the characteristics of a community of practice and promote learning. What is so special about these online communities of open-source software that is missing from the students in an online education course? To answer this question, one must look back at the best practices that promote learning. By focusing on the best practices, one can see that one of the underlying understandings of all these practices is that the participants are in the forum to solve a problem. The participants join the forum to solve a problem, they help others to receive solutions (reciprocity), and they collect and create information to solve a problem. They contribute to the forum to create a knowledge base that will be useful to them as well as others. They devise practices to ensure that knowledge is being created and efficient solutions are being provided to the forum users. It is not a structured lecture with a topic like a classroom setting; rather, the entire forum exists to solve a problem that a participant is encountering. The forum serves a purpose of solving problems, and therefore the incentive is in the form of solutions for the participants. How can this be achieved in a classroom setting? In an online classroom, discussions are usually about the topics in class, which do engage students with the content but do not necessarily develop a community. Students participate to receive grades/points, not because they want to help themselves or one another. However, if class discussion forums are created around solving a problem, will that create a community of mutual engagement, joint enterprise, shared repertoire, and negotiated meaning? To test this hypothesis, the author created two sets of discussion forums in a graduate course and told the students that the first set of forums will be graded for its content – the students had to post one message every week to demonstrate that they have understood the topic of the week. The second forum was created as a place to ask questions for their final project. The students were told that their participation on the second forum would not be graded. The final project involved learning about an open-source content management system and creating an informational website on it. This was a new and reasonably complex topic for them, so there was scope for a lot of questions. The second, un-graded forum was completely devoted to solving their problems and helping others. Surprisingly, the forum that was not graded had higher participation than the graded forum, and the content of the non-graded forum included students asking questions, providing resources to other students, helping other students, and developing a community where they were comfortable sharing their frustrations and triumphs. Another key difference between the two environments (open-source online communities vs. online classes) is the fact that the participation in open-source communities is voluntary and participation in online classes is a requirement. That is the reason why in the anecdote mentioned above, the problem-solving discussion forum was not graded, so that students do not feel obligated to participate but instead do so voluntarily. This voluntary participation is thus comparable to participation in an open-source community. On the other hand, a student might be in a class that is required but not want to be there, making his or her presence in class not voluntary. This test was done to get an idea of whether there is promise in the claim that to replicate the community strengths of open-source software communities, classroom discussions should be focused around solving a problem and not just discussing topics for grades/points. This anecdotal evidence is being shared to show the initial strength of the idea. A study that is more in-depth, structured, and at a larger scale is essential to explore this concept further. 6. Conclusion This research is based on open-source software discussion forums and learning. An initial survey of 283 forum participants was conducted; out of those, 21 survey respondents self-identified for participation in follow-up interviews. Data collected from the survey, especially the open-ended questions in the survey and data collected from interviews, informed this research to understand participant motivations and forum best practices for different roles (newcome, user, moderator, and administrator). Best practices that promote learning open-source software communities were developed and presented using thematic network analysis. Based on these best practices, recommendations to integrate them in online education are presented with ideas for future research. Acknowledgment This material is based in part upon work supported by the National Science Foundation under Grant Numbers 0935156. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation.

V. Singh, L. Holt / Computers & Education 63 (2013) 98–108

107

References Allen, E. I., & Seaman, J. (2010). Class differences: online education in the United States, 2010. The Sloan Consortium report. Retrieved from http://sloanconsortium.org/ publications/survey/class_differences. Ambient Research Firm. (2012). The worldwide market for self-paced elearning products and services: 2011–2016 forecast and analysis. http://www.ambientinsight.com/ Reports/eLearning.aspx. Ankolekar, A., Sycara, K., Herbsleb, J., Kraut, R., & Welty, C. (2006). Supporting online problem-solving communities with the semantic web. Paper presented at the WWW ’06: the 15th international conference on world wide web. Attride-Sterling, J. (2001). Thematic networks: an analysis tool for qualitative research. Qualitative Research, 1(3), 385–405. Bagozzi, R. P., & Dholakia, U. M. (2006). Open source software user communities: a study of participation in Linux user groups. Management Science, 52(7), 1099–1115. Bandura, A. (1977). Social learning theory. New York, NY: General Learning Press. Barzilay, O., Hazzan, O., & Yehudai, A. (2011). Using social media to study the diversity of example usage among professional developers. Paper presented at the ESEC/FSE ’11: the 19th ACM SIGSOFT symposium and the 13th European conference on Foundations of Software Engineering. Bergquist, M., & Ljungberg, J. (2001). The power of gifts: organizing social relationships in open source communities. Information Systems Journal, 11(4), 305–320. Bouguessa, M., Dumoulin, B., & Wang, S. (2008). Identifying authoritative actors in question-answering forums: The case of Yahoo! Answers. Paper presented at the KDD ’08: the 14th ACM SIGKDD international conference on knowledge discovery and data mining. Bransford, J., Brown, A. L., & Cocking, R. R. (Eds.), (1999). How people learn: Brain, mind, experience, and school. Washington, D.C.: National Academy Press. Bruns, A., & Humphreys, S. (2007). Building collaborative capacities in learners: The M/cyclopedia project revisited. Paper presented at the WikiSym ’07: The 2007 international symposium on wikis. Burkett, R., Leard, C., & Spector, B. (2004). Using an electronic bulletin board in science teacher education: issues and trade-offs. The Journal of Interactive Online Learning, 3(1), 1–9, Retrieved from http://www.ncolr.org/jiol/issues/pdf/3.1.1.pdf. Burnett, G., Besant, M., & Chatman, E. A. (2001). Small worlds: normative behavior in virtual communities and feminist bookselling. Journal of the American Society for Information Science and Technology, 52(7), 536–547. Cao, X., Cong, G., Cui, B., Jensen, C. S., & Zhang, C. (2009). The use of categorization information in language models for question retrieval. In Proceedings of the 18th ACM conference on information and knowledge management (CIKM) (pp. 265–274). New York, NY: ACM Press. Corbin, J. M., & Strauss, A. (1990). Grounded theory research: procedures, canons, and evaluative criteria. Qualitative Sociology, 13(1), 3–21. Creswell, J. W. (2007). Qualitative inquiry & research design: Choosing among five approaches (2nd ed.). Thousand Oaks, CA: Sage Publications. Creswell, J. W., & Miller, D. L. (2000). Determining validity in qualitative inquiry. Theory into Practice, 39(3), 124–130. Crowston, K., Wei, K., Howison, J., & Wiggins, A. (February 2012). Free/Libre open-source software development: what we know and what we do not know. ACM Computing Surveys, 44(2), 35, Article 7. Fang, Y., & Neufeld, D. (2009). Understanding sustained participation in open source software projects. Journal of Management Information Systems, 25(4), 9–50. Filho, F. F., Olson, G. M., & Geus, P. (2010). Kolline: A task-oriented system for collaborative information seeking. Paper presented at the SIGDOC ’10: the 28th ACM international conference on design of communication. Ghosh, R. A. (1998). FM interview with Linus Torvalds: what motivates free software developers? First Monday, 3(3). Hann, I. H., Roberts, J., & Slaughter, S. A. (2004). Why developers participate in open source software projects: an empirical investigation. In Proceedings of the 25th international conference on information systems. Hann, I. H., Roberts, J., Slaughter, S. A., & Fielding, R. (2002). Economic incentives for participating in open source software projects. In Proceedings of the 23rd international conference on information systems (pp. 365–372). Hansen, D. L. (2009). Overhearing the crowd: an empirical examination of conversation reuse in a technical support community. In Proceedings of the fourth international conference on communities and technologies (C&T ’09) (pp. 155–164). New York, NY, USA: ACM. http://doi.acm.org/10.1145/1556460.1556484. Harper, F. M., Moy, D., & Konstan, J. A. (2009). Facts or friends? Distinguishing informational and conversational questions in social Q&A sites. In Proceedings of the 27th international conference on human factors in computing systems (CHI ’09) (pp. 759–768). New York, NY: ACM Press. http://dx.doi.org/10.1145/1518701.1518819. Hars, A., & Ou, S. S. (2002). Working for free? Motivations for participating in open-source projects. International Journal of Electronic Commerce, 6(3), 25–39. Hatch, A. J. (2002). Doing qualitative research in educational settings. Albany, NY: State University of New York Press. Hertel, G., Niedner, S., & Herrmann, S. (2003). Motivation of software developers in open source projects: an internet-based survey of contributors to the Linux kernel. Research Policy, 32(7), 1159–1177. Hew, K. F., & Hara, N. (2007). Knowledge sharing in online environments: a qualitative case study. Journal of the American Society for Information Science and Technology, 58(14), 2310–2324. http://dx.doi.org/10.1002/asi.20698. Jones, S., Poulsen, A., Maiden, N., & Zachos, K. (2011). User roles in asynchronous distributed collaborative idea generation. Paper presented at the C&C ’11: the 8th annual conference on creativity and cognition. Kim, B., & Han, I. (2009). The role of trust belief and its antecedents in a community-driven knowledge environment. Journal of the American Society for Information Science and Technology, 60(5), 1012–1026. Klisc, C., McGill, T., & Hobbs, V. (2009). The effect of assessment on the outcomes of asynchronous online discussion as perceived by instructors. Australasian Journal of Educational Technology, 25(5), 666–682, Retrieved from http://www.ascilite.org.au/ajet/ajet25/klisc.html. Koh, J., Kim, Y. G., Butler, B., & Bock, G. W. (2007). Encouraging participation in virtual communities. Communications of the ACM, 50(2), 68–73. Lakhani, K. R., & von Hippel, E. (2003). How open source software works: “free” user-to-user assistance. Research Policy, 32(6), 923–943. http://dx.doi.org/10.1016/S00487333(02)00095-1. Laurillard, D. (2002). Rethinking university teaching: A framework for the effective use of learning technologies (2nd ed.). New York, NY: Routledge. Lave, J., & Wenger, E. (1991). Situated learning: Legitimate peripheral participation. Cambridge, UK: Cambridge University Press. Leh, A. (2002). Action research on hybrid courses and their online communities. Educational Media International, 39(1), 31–38, Retrieved from http://dx.doi.org/10.1080/ 09523980210131204. Lerner, J., & Tirole, J. (2002). Some simple economics of open source. Journal of Industrial Economic, 50(2), 197–234. Morse, J. M. (1997). “Perfectly healthy, but dead”: the myth of inter-rater reliability. Qualitative Health Research, 7(4), 445–447. Motoyama, M., McCoy, D., Levchenko, K., Savage, S., & Voelker, G. M. (2011). An analysis of underground forums. Paper presented at the ACM SIGCOMM conference on internet measurement. Musicant, D. R., Ren, Y., Johnson, J. A., & Riedl, J. (2011). Mentoring in Wikipedia: A clash of cultures. Paper presented at the WikiSym 11: the 7th international symposium on wikis and open collaboration. Nov, O., Naaman, M., & Ye, C. (2010). Analysis of participation in an online photo-sharing community: a multidimensional perspective. Journal of the American Society for Information Science and Technology, 61(3), 555–566. Orman, W. H. (2008). Giving it away for free? The nature of job-market signaling by open-source software developers. B.E. Journal of Economics, Analysis and Policy, 8(1), Article 12. Panzarasa, P., Opsahl, T., & Carley, K. M. (2009). Patterns and dynamics of users’ behavior and interaction: network analysis of an online community. Journal of the American Society for Information Science and Technology, 60(5), 911–932. Petr, C. G., & Walter, U. M. (2005). Best practices inquiry: a multidimensional, value-critical framework. Journal of Social Work Education, 41(2), 251–267. Resnick, M. L., & Vaughan, M. W. (2006). Best practices and future visions for search user interfaces. Journal of the American Society for Information Science and Technology, 57(6), 781–787. Savolainen, R. (2011). Judging the quality and credibility of information in internet discussion forums. Journal of the American Society for Information Science and Technology, 62(7), 1243–1256. http://dx.doi.org/10.1002/asi.21546. Seo, K. (2007). Utilizing peer moderating in online discussions: addressing the controversy between teacher moderation and nonmoderation. The American Journal of Distance Education, 21(1), 21–36, Retrieved from http://dx.doi.org/10.1080/08923640701298688. Shah, S. K. (2006). Motivation, governance, and the viability of hybrid forms in open source software development. Management Science, 52(7), 1000–1014. Sureka, A., Goyal, A., & Rastogi, A. (2011). Using social network analysis for mining collaboration data in a defect tracking system for risk and vulnerability analysis. In Proceedings of the 4th India software engineering conference (ISEC ’11) (pp. 195–204). New York, NY: ACM. http://dx.doi.org/10.1145/1953355.1953381. Swan, K., & Shea, P. (2005). The development of virtual learning communities. In S. R. Hiltz, & R. Goldman (Eds.), Asynchronous learning networks: The research frontier (pp. 239–260). New York, NY: Hampton Press.

108

V. Singh, L. Holt / Computers & Education 63 (2013) 98–108

Thom-Santelli, J., Cosley, D. R., & Gay, G. (2009). What’s mine is mine: territoriality in collaborative authoring. In Proceedings of the 27th international conference on human factors in computing systems (CHI ’09) (pp. 1481–1484). New York, NY: ACM. http://dx.doi.org/10.1145/1518701.1518925. Wenger, E. (1997). Communities of practice: Learning, meaning, and identity. New York: Cambridge University Press. Wu, C. G., Gerlach, J. H., & Young, C. E. (2007). An empirical analysis of open source software developers’ motivations and continuance intentions. Information Management, 44(3), 253–262. Xu, B., & Jones, D. R. (2010). Volunteers’ participation in open source software development: a study from the social-relational perspective. The Database for Advances in Information Systems, 41(3), 69–84. Yates, D., Wagner, C., & Majchrzak, A. (2010). Factors affecting shapers of organizational wikis. Journal of the American Society for Information Science and Technology, 61(3), 543–554. Ye, Y., & Kishida, K. (2003). Toward an understanding of the motivation of open source software developers. In Proceedings of international conference on software engineering (ICSE’03).