How to find people who can help to answer a question? – Analyses of metrics and machine learning in online communities

Computers in Human Behavior xxx (2014) xxx–xxx

Contents lists available at ScienceDirect

Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh

How to find people who can help to answer a question? – Analyses of metrics and machine learning in online communities Thiago Baesso Procaci, Sean Wolfgand Matsui Siqueira ⇑, Maria Helena Lima Baptista Braz, Leila Cristina Vasconcelos de Andrade Department of Applied Informatics (DIA), Federal University of the State of Rio de Janeiro (UNIRIO), Av. Pasteur, 458, Urca, 22290-240 Rio de Janeiro, RJ, Brazil

a r t i c l e

i n f o

Article history: Available online xxxx Keywords: Collaborative learning Online communities Experts Reputation Bow Tie structure Machine learning

a b s t r a c t Online communities have become important places for users to exchange information and build knowledge. In these communities, people ask and answer questions, learn with each other, but some problems may occur such as not getting an answer or getting contradictory ones. In order to increase the responsiveness of the communities, it would be important to identify people who are willing to help and who provide good answers in such communities, whom we call reliable users. We investigated various components of online communities and users’ attributes looking for a correlation between these characteristics and the users’ reputation in these communities. After that, we proposed the usage of two machine learning techniques, artificial neural network and clustering algorithm, with the users’ attributes for finding reliable sources. The results show that the usage of an artificial neural network is a good approach as around 90% of the users were correctly identified while the clustering algorithm makes to find groups of reliable users more easily. Ó 2014 Elsevier Ltd. All rights reserved.

1. Introduction Due to an increased demand for knowledge and a restricted availability of internal resources and competences for fulfilling such demands, people look for specialized knowledge in external sources (Constant, Sproull, & Kiesler, 1996; Fritzen, Siqueira, & Andrade, 2013; Prates, Fritzen, Siqueira, Braz, & Andrade, 2013; West, Salter, Vanhaverbeke, & Chesbrough, 2014) that usually come in the form of web sites or online communities. Online communities (or online social networks), which are intended to promote knowledge sharing, are effective places for finding help because, in general, they are formed by individuals that share common interests and voluntarily work together to expand their comprehension of a knowledge domain (Alan, Jian, Abrahams, Fan, & Zhang, 2013; Benson, Morgan, & Tennakoon, 2012; Marques, Krejci, Siqueira, Pimentel, & Braz, 2013). Usually, the members of these communities do not know each other, may be identified by pseudonyms and are willing to help each other for a number of reasons, for example altruism, reputation, expected reciprocity and the learning benefits (Kollock, 1999; Lakhani & Von Hippel, 2000; Lin & Huang, 2013).

⇑ Corresponding author. Tel.: +55 21 25308051; fax: +55 21 38736400. E-mail address: [email protected] (S.W.M. Siqueira).

We investigate how to find the most reliable members, i.e. people who are willing to help other people and who can provide good answers in online communities, and present the results in this article. The process of posting questions in online communities and waiting for answers is called social query (Banerjee & Basu, 2008; Morris, Teevan, & Panovich, 2010; Souza, Magalhães, Costa, & Fechine, 2013) and it could be considered a more qualitative alternative to search engines. According to Horowitz and Kamvar (2010), some problems such as more contextualized questions, recommendation’s requests, advices and opinions are better solved by people. However, the use of social query also has some limitations. When a question is posted in a community, unexpected results can be found such as: getting wrong or contradictory answers; continued responses even after the problem was solved; and never getting an answer, since some communities tend to prioritize the visualization of the most recent posts (Morris et al., 2010; Paul, Hong, & Chi, 2011). Considering this scenario of informal learning in online communities and looking for ways of minimizing some of the limitations of social query (more specifically, wrong answers and no answers at all), we performed a previous study in an online community that has no reputation mechanism (Procaci, Siqueira, & Andrade, 2014), but as the validation of the approach was limited to the evaluation of only one expert, in this work we consider five different online

http://dx.doi.org/10.1016/j.chb.2014.12.026 0747-5632/Ó 2014 Elsevier Ltd. All rights reserved.

Please cite this article in press as: Procaci, T. B., et al. How to find people who can help to answer a question? – Analyses of metrics and machine learning in online communities. Computers in Human Behavior (2014), http://dx.doi.org/10.1016/j.chb.2014.12.026

2

T.B. Procaci et al. / Computers in Human Behavior xxx (2014) xxx–xxx

communities with reputation mechanism. We studied the users’ attributes that could allow for inferring a high reputation (i.e., the reliable users) in each community and if data from one community could be used in other communities in order to find the most reliable users. In addition, considering the users’ attributes, we then proposed a novel approach, based on machine learning, using an artificial neural network and a clustering algorithm, to identify the reliable users. The remaining of this article is organized as follows: Section 2 presents related work. In Section 3 we describe the empirical study of five different online communities, aiming to characterize and explore the online networks. In addition, the results of the analyses of possible indicators that allow finding the reliable users, the artificial neural network and the clustering algorithm are also presented. Finally, in Section 4 we draw some conclusions and present future work. 2. Related work Studies for finding the reliable users (experts) in a community have already been explored in other works in the scientific community. Some of them focus on information retrieval techniques with natural language processing, also known as document-based approach, to identify user’s competences (Ackerman & McDonald, 1996; Krulwich & Burkey, 1996; Streeter & Lochbaum, 1998). In this approach, usually, the texts that are produced in a virtual environment are represented as a term vector (words or tokens) with their respective frequency. Then, it is possible to infer which kind of competence a user has, based on his discourse. However, the information retrieval approach makes it difficult to capture the level of competence of each user, since it is difficult to judge if a user provides a good answer by only parsing and processing his texts posted in a community (Zhang, Ackerman, & Adamic, 2007), thus, according to Littlepage and Mueller (1997), it is a limited approach. Balog, Azzopardi, and Rijke (2009) proposed a way for identifying the reliable users based on queries executed on an environment and a collection of texts associated to the candidates for reliable users. Their work based on information retrieval techniques and probabilistic methods aimed at setting the relevance between a query and the candidates for reliable users. Another similar work was proposed in (Liu, Wang, Johri, Zhou, & Fan 2012), in which a framework automatically generated the specialized user profiles of a community. These profiles had information about the user competences and were built based on the association between the community’s topics with the common user profile. Fernandes and Siqueira (2013) also presented an approach for automatically building users’ profiles in online social networks based on the content of their messages. Another approach is the use of ranking algorithms in graph for finding reliable users in a network. Algorithms were applied to a community (represented by a graph) and a number was assigned for each user, representing his competence degree on a subject. Campbell, Maglio, Cozzi, and Dom (2003) and Dom, Eiron, Cozzi, and Zhang (2003) used the HITS ranking algorithm on graphs for finding the reliable users that composed an email list. The results of these studies were rousing, since the approach based on graphs was effective. However, these studies have a weakness: the online network size was relatively small, meaning that the results might not reflect the reality. Zhang et al. (2007) proposed an algorithm based on graph for the same goal, but applied to a traditional discussion forum. Although their approach was interesting, the authors concluded, through the simulations, that the communities with different characteristics should be analyzed separately because of the particular characteristics that can influence the results, and then adaptations could be necessary in the measures

or techniques they applied. Alan et al. (2013) proposed a new approach for identifying reliable users when they built a hybrid model based on the information retrieval approach with the ranking algorithms on the graph approach. Banerjee and Basu (2008) presented a probabilistic algorithm that enabled directing the questions to the users who were more apt to answer them. This algorithm worked based on repeated past actions in the online social network. Davitz, Yu, Basu, Gutelius, and Harris (2007) developed a similar work, in which there was a global entity (agent) in the system that monitored the online social network and decided which users would receive (visualize) a specific posted question through probabilistic analysis. However, this solution based on agents was tested only on a small community. Souza et al. (2013) proposed an algorithm for finding reliable users based on a list of the user’s followers on Twitter. Their idea was to find the follower with the profile that was more adequate to answer a question in Twitter. Their results were interesting because the proposed algorithm was effective to find reliable users on Twitter. Some systems for finding expert consider document-based expertise indicators and social relationships (Kautz, Selman, & Shah, 1997; Mattox, Maybury, & Morey, 1999). Serdyukov, Rode, and Hiemstra (2007) proposed a relevance propagation algorithm that considered not only the documents directly related to an expert but also those indirectly related through document–author relationships. They used additional information such as hyperlinks in documents, user feedback and the organizational structure. The evaluation showed that their approach outperformed the methods that considered only documents directly related to experts. Another interesting approach is to use knowledge management to find experts or reliable users in a community. Awad and Ghaziri (2004) mentioned that knowledge management derives knowledge from several sources, including personnel practices knowledge, such as the available expertise, the quality service they provide and how to find experts. Ismail and Ahmad (2011) state that knowledge management enables to identify people who have high expertise in some area and connect them to people who need help. Ismail and Ahmad (2012) presented an analytical framework for an effective online personal knowledge management of knowledge worker. This framework enables to deeper understand people skills (expertise) and needs. After that, it makes possible to connect people who can help each other. This framework was quantitatively evaluated (questionnaire survey) by 118 workers from eight organizations from Malaysia. Li, Liao, and Lai (2012) proposed a recommender system, which generates expert recommendations based on semantic similarity, profession and reliability, social intimacy and popularity, and social network-based Markov Chain models for knowledge sharing in online forum communities. The results were interesting, however, according to the authors, the recommendations evaluation needs improving (for instance, inviting the users from the communities to participate in the evaluation). In our work, we revisited the approaches based on graph with ranking algorithms combined with the information retrieval approach for finding reliable users in online communities. Our approach is based on graphs because the communities are represented through this form, while based on information retrieval because we extracted metadata (information about the users) from the communities for making the analyses. However, we propose a different way of finding reliable users. We made analyses on parts of the communities and then we used one of them for training our artificial neural network. The purpose of dividing the communities is to find a good set of training data for the artificial neural network. Once trained, we used the artificial neural network to identify the reliable users on the communities. Besides, we made analyses using a clustering algorithm in order to create groups of

Please cite this article in press as: Procaci, T. B., et al. How to find people who can help to answer a question? – Analyses of metrics and machine learning in online communities. Computers in Human Behavior (2014), http://dx.doi.org/10.1016/j.chb.2014.12.026

T.B. Procaci et al. / Computers in Human Behavior xxx (2014) xxx–xxx

similar users. Through these groups, we expected to find the ones where the probability of finding a reliable user is higher. 3. Empirical study We conducted empirical studies on five communities in order to identify with more confidence the reliable users. We analyzed some users’ attributes aiming at reaching a common strategy that could be applied to different online social networks. In this work, we assume the user is directly reliable according to his reputation. 3.1. Dataset and general characteristics of the communities For analyzing the measures and testing the strategies for identifying the reliable users, it was necessary to extract a set of data from real online communities. We chose five distinct communities of questions and answers: Biology Q&A1 (BQA), a community for learning biology; English Language and Usage2 (ELU), a community focused on the learning of the English language; Physics Q&A3 (PQA), a community focused on the learning of physics; Mathematics Q&A4 (MQA), a community focused on the learning of math; and Travel Answers5 (TA), a community intended for elucidating doubts on travels. These five communities have similar characteristics, although they attract different participants. Usually, in this kind of online communities, the discussions are structured in threads, i.e. a user posts a question or topic and soon after that, other users post answers or comments related to the question. In addition, each thread belongs to at least one category of the community (for instance: electromagnetism, genetics, calculus etc.) and each user is evaluated by other users based on their posted questions or answers. This evaluation scheme allows the users to build their reputation in the network. However, in some communities there may be no reputation mechanism. In addition, usually the reputation depends on other users’ evaluation. In our work, we aim at minimize such dependency and provide an automatic mechanism for finding the reliable users. In order to extract data from these communities we developed a crawler that consumed data from each community. It was developed using Python programming language. The crawler sent several HTTP requests to each community and then, saved all returned data in several files. We read all the files and discarded the data not needed in our work. We were only interested in data related to the users’ participation. Then, we transformed these data into two models: class model and graph. The former allowed us to extract simple users’ attributes (such as number of answers) as well as to calculate metrics which do not depend on any graph based algorithm (such as entropy and z-score). The latter (graph model) allowed us to extract measures based on graphs, such as indegree and the results provided by the Page Rank algorithm. Afterward, we evaluated the metrics using statistical correlation. The measures, attributes and the evaluation are described in the next sections. Table 1 shows general characteristics of extracted data from the network, such as: number of users, number of messages, number of answers, number of comments and number of threads (which is the number of main posts or topics). Through Table 1, it is possible to notice that the Mathematics community is the biggest one and the Biology community is the smallest one considering the number of messages, answers, comments, threads and users. The average 1 2 3 4 5

http://biology.stackexchange.com/. http://english.stackexchange.com/. http://physics.stackexchange.com/. http://math.stackexchange.com/. http://travel.stackexchange.com/.

3

amounts of characters of the posts are similar in the five communities. However, the average size of a thread is bigger in the English Language and Usage community. This may indicate that this community has bigger (or deeper) discussions when compared to the others. 3.2. Representing the communities as graphs In order to make the necessary analyses, we represented the communities as directed graphs, following the proposal presented in (Zhang et al., 2007). In this scheme, the graph nodes represent the users while the edges characterize their interactions. Then, if a user A posts a question and the user B answers it, the graph has two nodes (A and B) representing the users (respectively A and B). In addition, this graph has an edge that departs from node A in direction to B, symbolizing that B answered A. We extended this model in order to better represent the interactions among the users of the communities. As in the analyzed communities it is also possible to comment a question or an answer, if user X comments a question of user Y, then an edge departing from the user (node) Y and arriving at the user (node) X is also represented. Similarly, if the user Z comments an answer of the user K, then an edge departing from the user (node) K and arriving at the user (node) Z is also represented. The five communities that were analyzed in this work were represented through graphs and their characteristics are presented in Table 2. 3.3. The Bow Tie structure After representing the network as a graph, it was possible to use the Bow Tie structure (Broder et al., 2000), to analyze the communities according to six distinct components: Core, IN, OUT, Tendrils, Tubes and Disconnected. The Core component has the users who frequently answer each other. Considering the graph representation, the Core components are the strongly connected nodes of a graph. The IN component has the users who only make questions and get answers from some Core member (i.e., a node that has indegree equals to zero, outdegree bigger than zero and an edge that departs from it and arrives at some Core member). Then, the nodes that constitute the IN are those that can reach some member of Core, but can’t be reached by any member of Core. The OUT component contains the users who only answer questions posted by some member of the Core component (an OUT member is a node whose outdegree is equal to zero, the indegree is bigger than zero and there is an edge that departs from a Core member and reaches the OUT member). Then, the OUT members can be reached by Core members, but can’t reach the Core. The Tendrils and Tubes components are connected to the IN or OUT components, but are not connected to the Core. The Tendrils are those who can be reached by some member from IN and do not reach the Core or those who can reach some OUT member and can’t be reached by someone from Core. On the other hand, the Tubes are those who can be reached by some member of IN and do not reach the Core and those who can reach some member of OUT and can’t reach someone from Core. The Disconnected are those who do not fit any of the previous components. Table 3 shows the structures of the five analyzed communities, which are places where people are willing to help and be helped. The Core component varies from 41.3% to 50.5%, which means almost half of the users are willing to collaborate with each other. The IN component of the communities BQA (32.6%) and MQA (33.9%) are fairly bigger than the IN from ELU (25.9%), PQA (29.8%) and TA (28.9%). It means that in these two communities (BQA and MQA) there are more people interested in only asking questions and not answering (or helping) anyone when compared

Please cite this article in press as: Procaci, T. B., et al. How to find people who can help to answer a question? – Analyses of metrics and machine learning in online communities. Computers in Human Behavior (2014), http://dx.doi.org/10.1016/j.chb.2014.12.026

4

T.B. Procaci et al. / Computers in Human Behavior xxx (2014) xxx–xxx

Table 1 General characteristics of the online social networks. Community

Number of messages

Number of threads

Number of answers

Number of comments

Average size of a thread

Average amount of characters / posts

Number of users

BQA ELU PQA MQA TA

25,828 326,915 250,337 1,035,275 42,322

4,549 30,044 31,678 149,948 5,529

5,734 79,978 51,838 215,346 10,526

15,545 216,893 166,821 669,981 26,267

3 6 4 4 4

314 236 328 222 275

2,317 20,408 15,753 51,245 3,579

Table 2 Data from the communities graphs. Community

Number of nodes

Number of edges

BQA ELU PQA MQA TA

2,317 20,408 15,753 51,245 3,579

10,316 149,993 96,938 418,995 16,792

to the others (ELU, PQA and TA). The OUT component presents bigger numbers in TA (22.6%) and ELU (18.9%) compared to the others (PQA, BQA and MQA), which means there are more people willing to answer, but do not make questions. The Tubes, Tendrils and Disconnected have smaller numbers. The Bow Tie structure is important in the context of this work because its components are used to investigate the strategies of discovering the dataset on the training process of the artificial neural network used to identify the reliable users. Therefore, in this work we analyze each component of the Bow Tie structure separately, aiming at seeking evidences of reliable users. 3.4. Users’ attributes We analyzed some users’ attributes in order to seek for evidences that a user is reliable (has a high reputation). These attributes were analyzed for the whole network as well as for the specific components of the Bow Tie structure. The attributes we chose for the analyses were: (1) number of answers and number of comments: represent the users’ answers and comments within the network and may indicate the reputation of a user in a network; (2) z-score: combines the number of questions with the number of answers of a user. Answering to several questions may indicate the user is reliable, but asking many questions may indicate he is not. This metric provides a balance between the number of questions and answers of a user; (3) indegree: represents the number of people the user has answered. We believe the user reputation may depend on the amount of people he answers; (4) user entropy: a measure for analyzing the focus of a user in specific subjects in the community, to ascertain the relationship between the user focus in specific subjects and his reputation; and (5) Page Rank (Page, Brin, Motwani, & Winograd, 1998): several works use ranking algorithms for finding the reliable users, but not considering specific parts of the network. The idea was to analyze if it works in such strategies. Then, all these

measures were extracted from the network and analyzed separately for each component of the Bow Tie structure. Table 4 shows the correlations of the user attributes of each community with the reputation from the network considering each component of the Bow Tie structure and considering the whole network (General). In this table, the label ‘‘Answer#’’ stands for the number of answers, while ‘‘Com#’’ is the number of comments, and ‘‘A + C’’ the sum of the number of answers with the number of comments. In addition to the number of answers and comments, we considered the z-score, which is a measure proposed by Zhang et al. (2007) and can indicate the user reputation or expertise in the network. That work showed how the z-score calculation was created, reaching the formula:

Z-score ¼ ðQ  AÞ=ðQ þ AÞ1=2

ð3Þ

Then, the z-score of each user was calculated considering the number of questions (variable ‘‘Q’’) and the number of answers (variable ‘‘A’’) that he posted. In the context of this work, the variable ‘‘A’’ was the sum of the number of answers with the number of comments of a user, since a comment can be considered as an answer to a question or a compliment to another answer. The zscore provided a good measure for indicating user reputation, although it was not the best one. In order to capture how answers and comments of a user are focused on specific categories (or topics) of the communities, we considered the user entropy. The more grouped are the answers or comments of a user in a specific category, the lower is the entropy and higher is the focus. A person who has high entropy usually answers or comments topics in several categories, i.e. this person has a lower focus in specific subjects. The user entropy can be defined by the formula (Adamic, Zhang, & Ackerman, 2008):

entropy ¼ 

X

P i i

 log2ðP i Þ

ð1Þ

The ‘‘P’’ in the formula is used to determine the ability of a user to transmit information. In this work, we call this ability as user participation. In order to explain this formula in the context of this work, imagine a user who posted ten answers in a community of questions and answers, in which three of the answers were related to the calculus category, three were related to the algebra category and four were related to probability. Then, we calculate the ‘‘P’’ of each category, an indicator of participation of a user in a specific category. For instance, the ‘‘P’’ for this user in the calculus category

Table 3 Comparing the components of the Bow Tie structure.

BQA ELU PQA MQA TA

Core (%)

IN (%)

OUT (%)

Tendrils (%)

Tubes (%)

Disconnected (%)

47.5 48.0 49.1 50.5 41.3

32.6 25.9 29.8 33.9 28.9

11.5 18.9 13.0 8.5 22.6

2.7 3.1 3.1 1.7 3.2

0.00 0.06 0.01 0.01 0.03

5.70 4.04 4.99 5.39 3.97

Please cite this article in press as: Procaci, T. B., et al. How to find people who can help to answer a question? – Analyses of metrics and machine learning in online communities. Computers in Human Behavior (2014), http://dx.doi.org/10.1016/j.chb.2014.12.026

5

T.B. Procaci et al. / Computers in Human Behavior xxx (2014) xxx–xxx Table 4 Pearson correlation coefficient – attributes.

General BQA Core BQA IN BQA OUT BQA Tendrils BQA General ELU Core ELU IN ELU OUT ELU Tendrils ELU General PQA Core PQA IN PQA OUT PQA Tendrils PQA General MQA Core MQA IN MQA OUT MQA Tendrils MQA General TA Core TA IN TA OUT TA Tendrils TA

Answer#

Com#

A+C

z-score

Indegree

Entropy

Page rank

0.92 0.93 0.05 0.14 0.03 0.92 0.93 0.07 0.06 0.10 0.91 0.90 0.07 0.29 0.20 0.89 0.87 0.07 0.02 0.02 0.94 0.97 0.21 0.36 0.14

0.84 0.83 0.11 0.43 0.47 0.76 0.76 0.16 0.39 0.28 0.73 0.74 0.10 0.38 0.26 0.88 0.85 0.10 0.38 0.18 0.83 0.85 0.28 0.18 0.02

0.89 0.90 0.12 0.34 0.40 0.82 0.83 0.17 0.35 0.19 0.81 0.80 0.11 0.28 0.17 0.90 0.87 0.11 0.37 0.17 0.91 0.89 0.31 0.38 0.16

0.79 0.83 0.008 0.36 0.26 0.81 0.81 0.05 0.40 0.14 0.70 0.69 0.02 0.36 0.15 0.81 0.78 0.02 0.41 0.13 0.76 0.81 0.12 0.41 0.09

0.91 0.92 – 0.32 0.15 0.88 0.86 – 0.36 0.20 0.82 0.81 – 0.25 0.16 0.90 0.86 – 0.34 0.16 0.93 0.92 – 0.39 0.09

0.59 0.58 0.11 0.30 0.25 0.36 0.34 0.12 0.40 0.20 0.33 0.32 0.09 0.37 0.22 0.36 0.33 0.11 0.41 0.33 0.44 0.45 0.25 0.43 0.16

0.90 0.91 – 0.29 0.14 0.86 0.84 – 0.32 0.21 0.81 0.80 – 0.31 0.13 0.89 0.87 – 0.32 0.12 0.91 0.90 – 0.28 0.05

is 0.3 since 3 of the 10 answers were posted in the calculus category (i.e., the division 3/10). For calculating this user’s entropy, it is necessary to execute the following calculation:

comments and the z-score are moderate indicators of the reputation.

ððPcalc  log2ðPcalcÞÞ þ ðPalg  log2ðPalgÞÞ þ ðPprob  log2ðPprobÞÞÞ

3.5. Other users’ attributes

ðð0:3  log2ð0:3ÞÞ þ ð0:3  log2ð0:3ÞÞ þ ð0:4  log2ð0:4ÞÞÞ ¼ 1:57 ð2Þ Then, the users’ entropies in the communities were calculated in order to analyze if a user that is focused in a specific category has higher reputation. However, the results show the entropy (less focus in the subjects) is a weak indicator of reputation. Considering only the parts of the communities (Bow Tie components), the best correlations were provided by the Core components, since the most participative (active) users (those who ask and answer) are found in it. The correlation between the user attributes and reputation in the IN components were weak most of the times. The reason may be the fact the members of the IN component only make questions and therefore they do not build a good reputation in the network. On the other hand, the correlations provided by the OUT components, composed of users who only answer questions, sometimes were moderate and sometimes weak. It may indicate that only answering questions is not enough to build high reputation. It seems that to build high reputation needs reciprocity during the interactions. The Tendrils components provided not so good correlations, maybe because they are much smaller. The correlations for the Tubes components of each community were not calculated because they are too small, so no conclusion drawn from them would be relevant. Comparing the correlations provided by the Core components and by the whole network (General), it is possible to notice that they are very similar. It means that the Core components can represent well the whole community when trying to find the reliable users. In summary, after the analysis of the network as a whole or its parts in the five communities, we can conclude that the number of answers (as well as the number of answers and comments), the indegree and page rank provide the best correlations with the reputation. In addition, the analysis of the whole network and the Core component are the most representative. The number of

When studying online communities and representing it as graphs, it is common to see others classical graph metrics being analyzed such as betweenness, closeness and eigenvector. However, some algorithms used to compute these metrics require high CPU utilization (time complexity) and it may take a long time to bring the results. Due to this fact, the usage of these metrics can be a bottleneck during analyses involving huge graphs, despite the results it can bring. Betweenness is a measure of a node within a graph and quantifies the number of times a node acts as a bridge along the shortest path between two other nodes. Although there is also the edge betweenness, it is not discussed in this work. In connected graphs there is a distance metric between all pairs of nodes, defined by the length of their shortest paths. The farness of a node is defined as the sum of its distances to all other nodes, and its closeness is defined as the inverse of the farness. Thus, the more central a node is, then the lower is its total distance to all other nodes. Closeness can be regarded as a measure of how long it will take to spread information from a node to all other nodes. The complexity time to calculate the betweenness and the closeness for all nodes of a graph is O(n3) (Zhuge & Zhang, 2010; Bader & Madduri, 2006). There are strategies to improve the complexity time for these metrics like presented in (Brandes, 2001), but it is out of the scope of this work. Eigenvector is a measure of the influence of a node in a network. It assigns relative scores to all nodes in the network based on the concept that connections to high-scoring nodes contribute more to the score of the node in question than equal connections to low-scoring nodes. The Page Rank algorithm is a variant of the eigenvector measure. For this reason, we do not calculate this measure for all nodes of our graphs. We extracted a subgraph from each community graph as presented in Section 3.2 to make it easier to calculate the

Please cite this article in press as: Procaci, T. B., et al. How to find people who can help to answer a question? – Analyses of metrics and machine learning in online communities. Computers in Human Behavior (2014), http://dx.doi.org/10.1016/j.chb.2014.12.026

6

T.B. Procaci et al. / Computers in Human Behavior xxx (2014) xxx–xxx

Table 5 Data from the communities subgraphs.

Table 7 User classification.

Community

Number of nodes

Number of edges

Condition

Classification

BQA ELU PQA MQA TA

1100 3001 3001 3001 1479

7711 31,010 29,811 13,652 12,302

UR 6 M1 M1 < UR 6 M2 M2 < UR 6 M3 M3 < UR 6 M4 M4 < UR 6 M5 M5 < UR 6 MAX

Not reliable Little reliable Fairly reliable Reliable Very reliable Extremely reliable

Table 6 Pearson correlation coefficient – subgraphs. Subgraph

Betweenness

Closeness

Eigenvector

BQA ELU PQA MQA TA

0.92 0.94 0.84 0.87 0.94

0.45 0.19 0.17 0.18 0.39

0.88 0.80 0.67 0.80 0.78

betweenness, closeness and eigenvector. Each extracted subgraph contains only nodes from the Core component of each community. As already said, the Core component seems to represent well the whole community and each node that was selected to be in the subgraph was randomly selected from the Core. The Table 5 shows the subgraphs’ data. After that, we calculated the correlations between the users’ reputation and the three measures described in this section (betweenness, closeness and eigenvector). In Table 6 one can see that the correlations involving the betweenness measure obtained the best results. A node with high betweenness value means that it can connect different parts of a network, avoiding creating a disconnected graph. The eigenvector also obtained good correlations, similar to Page Rank presented in the previous section. The closeness measure obtained worst correlations. It means that a measure of spreading information is not so useful when trying to find the reliable users from a community. Summarizing the results in the context of this work, the higher the values of the betweenness and eigenvector measures, the greater are the chances of a user being reliable. However, considering the results and processing time and the results for the users’ attributes and the graph measures, the previous approach is more interesting than considering the graph metrics. 3.6. Using artificial neural networks Given the set of attributes of a user, we used an artificial neural network called Multilayer Perceptron to classify a user into reliable or not. We decided to use the Multilayer Perceptron because this method is able to classify data that are not linearly separable. The goal was to classify the users into reliable or not, using the user’s attributes (which are not linearly separable). First, it was necessary to train the neural network and we used the users’ attributes (described in Section 3.4) as the inputs. We did not use the attributes described in Section 3.5 as the inputs because we did not calculate it for all users of the community due to its

computational cost. For the outputs (desired results) of the neural network we embraced a classification provided by an arithmetic means scheme. The scheme is based on the reputation provided by the communities, which is built based on evaluations made by their own users. In other words, a user has his answers, comments and questions evaluated by others users. Successive arithmetic means of the reputation allow us to identify who are the reliable users of the community. First, we had to find the highest and the lowest reputation in the community. The Fig. 1 shows an example of this scheme considering the highest reputation equals to 100 (MAX) and the lowest equals to 0 (MIN). Once known the highest and the lowest reputation scores, we calculated the arithmetic mean between them. We called M1 the result of this operation. Then, we calculated the arithmetic mean between M1 and MAX and called the result M3. Afterward, we calculated the arithmetic mean between M1 and M3 and we called the result M2. After that, we calculated the arithmetic mean between M3 and MAX and we called the result M4. At last, we calculated the arithmetic mean between M4 and MAX and we called the result M5. It is important to mention that we decided to use the arithmetic mean because we wanted the extreme values (MAX and MIN) influencing the means. In other words, the classification of a user depends on the most and the least reliable user in the communities. After these calculations, we considered the reliability of a user according to Table 7. The label UR from Table 7 stands for the user reputation given by the communities. After the definitions of the inputs (users’ attributes) and the outputs (desired results provided by reputation arithmetic means) for our neural network, we started the training process with the users’ attributes only from the Core component of each community as the training dataset, due to the good correlations (as described in the Section 3.4). After training, we compared the classification provided by the neural network using all attributes of all users and the classification provided by the reputation and the arithmetic means scheme. The Table 8 shows the results obtained after running the trained neural network of each scenario (in the columns). The Table 8 also shows that the neural network classified correctly most of the users. However, just verifying the percent of correctly classified users is not enough to make any conclusion since this metric does not take into account the classification occurring by chance. Given this problem, we used Cohen’s kappa coefficient in order to check if running our classifier (neural network) several times can bring the similar classifications. In other words, we want to know the agreement of each classification

Fig. 1. Reputation arithmetic means.

Please cite this article in press as: Procaci, T. B., et al. How to find people who can help to answer a question? – Analyses of metrics and machine learning in online communities. Computers in Human Behavior (2014), http://dx.doi.org/10.1016/j.chb.2014.12.026

7

T.B. Procaci et al. / Computers in Human Behavior xxx (2014) xxx–xxx Table 8 Comparison: neural network x reputation.

Correctly classified users (%) Incorrectly classified users (%) Kappa statistic Training dataset

BQA

ELU

PQA

MQA

TA

BQA

ELU

PQA

MQA

TA

89.2 10.8

91.0 9.0

90.5 9.6

92.1 7.9

91.5 8.5

88.7 11.3

91.5 8.5

90.0 10.0

92.2 7.8

90.2 9.8

0.476 Core BQA

0.551 Core ELU

0.4707 Core PQA

0.4491 Core MQA

0.5401 Core TA

0.4718 20% Core BQA

0.5294 20% Core ELU

0.5099 20% Core PQA

0.4339 20% Core MQA

0.582 20% Core TA

Table 9 Cross-validation.

Correctly classified users (%) Incorrectly classified users (%) Kappa statistic

BQA

ELU

PQA

MQA

TA

88.6 11.4 0.4506

91.6 8.4 0.5099

90.1 9.9 0.481

92.1 7.9 0.467

91.6 8.4 0.5475

(if the classification of a user persists after running our neural network many times). Cohen’s kappa measures the agreement among classifiers and each item into mutually exclusive categories. Landis and Koch (1977) characterized Cohen’s kappa values like: less than 0 as indicating no agreement among classifiers; 0 to 0.20 as slight; between 0.20 and 0.40 as fair; between 0.40 and 0.60 as moderate; between 0.60 and 0.80 as substantial; and above 0.80 as almost perfect agreement. Fleiss (1981) characterized Cohen’s kappa values similarly: over 0.75 as excellent, 0.40 to 0.75 as fair to good, and below 0.40 as poor. Notice that all values are between 0.41 and 0.60, which according to Landis and Koch (1977) is a moderate indicator and according to Fleiss (1981) is fair to good. Through these tests, we can conclude that our approach seems to be a promising method to identify the reliable users from the communities. Another interesting fact is that using only 20% of the Core as the training dataset can bring similar results when compared to the whole Core. We tested others training datasets, for instance using random users instead of the Core. However sometimes it fails due to the random choice of the training dataset. Therefore, it is safer to use the Core component. Given that online communities are dynamic and the number of users can increase or decrease any time, it is necessary to check if the proposed neural network can bring similar results when the data involved changes. In order to verify if the neural network

results when new dataset appears, we performed the k-fold cross-validation model. The k-fold is used to estimate how well a predictive model will perform in practice given that new data may appears frequently (Kohavi, 1995). In k-fold cross-validation, the original dataset is divided into k equal size subsamples. Of the k subsamples, one subsample is retained as the validation data for testing the model, and the remaining k - 1 subsamples are used as training dataset. The cross-validation process is repeated several times, with each of the k subsamples used exactly once as the validation data. The Table 9 shows the cross-validation results performed on our online communities, using 10 subsamples. It is possible to see that the cross-validation results are similar to the results presented in the Table 8. Through the cross-validation test, we can conclude that our neural network will perform well considering dataset changes. 3.7. Using the Core from another community as training dataset In order to analyze the possibility of using a previous trained neural network in other communities, we performed some tests, training the neural network using the attributes of the Core users from one community and then, trying to classify the users from another community. The Tables 10 and 11 show the results of the classification using the Core component from other communities. In the Tables 10 and 11, we can see that neural networks can correctly classify most of the users. Most of the scenarios presented in Tables 10 and 11 show that more 80% of users were correctly classified, considering the entire Core as the training dataset or 20% of it. In addition, most of the values of Cohen’s kappa are above 0.40, which is a moderate (Landis & Koch, 1977) or fair to good (Fleiss, 1981) indicator. All cases in which Cohen’s kappa measures were less than 0.40 was involved the BQA community (sometimes as the community where the neural network was tested and

Table 10 Comparison: neural network x reputation (using the Core component from different communities). Community

Correctly classified users (%)

Incorrectly classified users (%)

Kappa statistic

Training dataset

BQA BQA BQA BQA ELU ELU ELU ELU PQA PQA PQA PQA MQA MQA MQA MQA TA TA TA TA

86.49 87.69 78.67 87.87 90.53 91.39 91.26 90.90 89.53 89.83 90.20 90.00 90.34 90.50 91.24 90.65 90.66 90.94 90.51 87.84

13.51 12.31 21.33 12.13 9.47 8.61 8.74 9.10 10.47 10.17 9.80 10.00 9.66 9.50 8.76 9.35 9.34 9.06 9.49 12.16

0.3795 0.3974 0.3225 0.4091 0.4518 0.4837 0.4627 0.5103 0.427 0.4974 0.4555 0.476 0.3999 0.4751 0.4304 0.443 0.4687 0.5424 0.4461 0.4248

Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core

ELU PQA MQA TA BQA PQA MQA TA BQA ELU MQA TA BQA ELU PQA TA BQA ELU PQA MQA

Please cite this article in press as: Procaci, T. B., et al. How to find people who can help to answer a question? – Analyses of metrics and machine learning in online communities. Computers in Human Behavior (2014), http://dx.doi.org/10.1016/j.chb.2014.12.026

8

T.B. Procaci et al. / Computers in Human Behavior xxx (2014) xxx–xxx

Table 11 Comparison: neural network x reputation (using 20% of the Core component from different communities). Community

Correctly classified users (%)

Incorrectly classified users (%)

Kappa statistic

Training dataset

BQA BQA BQA BQA ELU ELU ELU ELU PQA PQA PQA PQA MQA MQA MQA MQA TA TA TA TA

87.57 87.61 88.34 87.69 89.58 91.25 91.23 91.35 89.32 90.16 89.89 90.19 89.85 91.18 90.93 91.02 89.91 91.01 91.22 90.66

12.43 12.39 11.66 12.31 10.42 8.75 8.77 8.65 10.68 9.84 10.11 9.81 10.15 8.82 9.07 8.98 10.09 8.99 8.78 9.34

0.3763 0.401 0.434 0.3761 0.4338 0.5258 0.4448 0.5122 0.4369 0.4523 0.4047 0.4652 0.3888 0.4372 0.4559 0.4401 0.4528 0.4837 0.5289 0.44

20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20% 20%

sometimes as the training dataset). The BQA community is the smallest community among those analyzed, and perhaps for this reason, the results provided by the Cohen’s kappa were not so good. Thus, when using this approach to make any user classification, a special attention must be given to the size of the community. However, in some cases, even using smaller communities such as BQA and TA, the Cohen’s kappa obtained good results. Apparently, choosing smaller communities in this approach is not a safe option. The most consistent values of Cohen’s kappa occurred in larger communities with similar sizes, such as PQA and ELU. Thus, according to the scope of this research, the usage of a previous trained neural network seems to be a good approach when using data from larger communities (size similar to PQA at least) and, after that, apply it to a community with similar size.

Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core Core

ELU PQA MQA TA BQA PQA MQA TA BQA ELU MQA TA BQA ELU PQA TA BQA ELU PQA MQA

3.8. Using clustering algorithm In order to check if another approach can bring better results than the usage of the artificial neural network, we tested a clustering algorithm called K-means (Macqueen, 1967). The idea of this kind of algorithm is to make groups of similar objects and it is one of the most famous clustering algorithms. The K-means algorithm tries to make group of objects just using its input data. In other words, unlike the artificial neural networks, it is not necessary to inform the desired outputs to clustering algorithms, but just to provide the input data. Usually, in this kind of algorithm, an object is assigned to a group according to similarity. If an object is similar to another one, they will be assigned to the same group. In the context of this research, the objects we want to put in different groups are the users.

Table 12 Groups provided by K-means. Group

Community

Number of users

Average number of answers

Number of not reliable

Number of little reliable

Number of fairly reliable

Number of reliable

Number of very reliable

Number of extremely reliable

Probability of being reliable or above

Probability of being fairly reliable or below

0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4 0 1 2 3 4

BQA BQA BQA BQA BQA PQA PQA PQA PQA PQA ELU ELU ELU ELU ELU MQA MQA MQA MQA MQA TA TA TA TA TA

20 2 000 7 55 235 175 27 7 14,967 577 11 18,898 51 205 1 243 47,080 18 3 223 767 157 2 950 31 81 514 3

62.45 0.4455 172.1429 23.45455 4.438298 77.44 288 1202 0.702546 17.75043 1 007.273 0.716372 378.9216 92.53659 11.66613 0.424533 1 806.389 10.1542 85.11864 395.7134 0.445763 125.4516 25.44444 3.402724 473

0 1 904 0 0 86 0 0 1 13,752 40 0 17,922 1 1 224 45,083 0 1 167 8 2 2 891 0 1 274 0

0 81 0 3 73 2 0 0 915 88 0 841 0 5 433 1 599 0 1 120 53 0 56 0 11 175 0

0 14 0 13 63 14 1 0 296 284 0 135 0 33 500 381 0 751 303 2 3 0 34 63 0

7 1 1 29 13 120 1 0 4 159 0 0 6 140 86 17 0 180 348 63 0 16 35 2 0

8 0 3 10 0 37 12 0 0 6 2 0 29 24 0 0 3 5 51 60 0 11 0 0 1

5 0 3 0 0 2 13 6 0 0 9 0 15 2 0 0 15 0 4 30 0 4 0 0 2

100% 0.05% 100% 70.90909% 5.531915% 90.85714% 96.2963% 85.71429% 0.026725% 28.59619% 100% 0% 98.03922% 80.97561% 6.918745% 0.036109% 100% 5.739994% 52.54237% 97.45223% 0% 100% 43.20988% 0.389105% 100%

0% 99.95% 0% 29.09091% 94.46809% 9.142857% 3.703704% 14.28571% 99.97327% 71.40381% 0% 100% 1.960784% 19.02439% 93.08126% 99.96389% 0% 94.26001% 47.45763% 2.547771% 100% 0% 56.79012% 99.61089% 0%

Please cite this article in press as: Procaci, T. B., et al. How to find people who can help to answer a question? – Analyses of metrics and machine learning in online communities. Computers in Human Behavior (2014), http://dx.doi.org/10.1016/j.chb.2014.12.026

T.B. Procaci et al. / Computers in Human Behavior xxx (2014) xxx–xxx

Fig. 2. Probability of being reliable or above.

Fig. 3. Probability of being fairly reliable or below.

When running the K-means algorithm, it is necessary to choose the number N of groups to be created. Then, the K-means algorithm randomly selects N objects (in our case, users) to be the centroid of each group and the other objects are associated to the nearest centroid. After that, the centroids of each group are recalculated and usually the new centroid is the most central object of a group. This procedure is repeated until a satisfactory level of convergence is found. We used the K-means to find the reliable users of the communities entering the users’ attributes as the inputs data, and we configured it to create five groups of users. The idea was to make groups of users with similar attributes and we expected to be able to find the groups in which the reliable users were. The Table 12 shows the groups formed by the K-means algorithm, where each group has the number of users that belongs to it, the average number of answers provided by the members of the groups and the number of users according to the classification provided by Table 7. We can see that, in general, the higher is the average number of answers of a group, the better is the probability of finding a reliable user in it. This fact can be better noticed at Figs. 2 and 3, where the blue (solid) line presented the data in Table 12 (average number of answers, probability of being reliable or above and probability of being fairly reliable or below) and the red (dotted) line shows logarithmic trend line of the blue line data. These figures show that the higher is the average number of answers of a group, the lower is the number of users that belongs to this group. 4. Conclusion In this article we presented the characteristics of five communities of questions and answers. First, these communities were characterized according to the Bow Tie and, according to the size of the Core component, the majority of the members of the communities of study seemed to be willing to help each other. We also analyzed

9

the correlation between the users’ entropy (focus on specific subjects) and their reputation in the network and we concluded that the entropy is moderately correlated with the user reputation when we consider the whole network or only the components Core and OUT. It means a user that does not focus his participation in the network in specific categories (i.e. has a high entropy), probably has a high reputation. However, it is noteworthy that the categories of the communities are from the same main theme and the results could be different if different themes were considered. We also analyzed and correlated several users’ attributes with the users’ respective reputation. Good correlations were obtained with the number of messages, the indegree and the page rank, in the whole network as well as the Core component, which showed to be a good approach for finding reliable users. We also used some classical graphs metrics such as betweenness, closeness and eigenvector. We saw that betweenness and eigenvector could represent the reputation of a user due to the good correlations. However, the betweenness calculation demands high computational processing and therefore may not be applicable in some cases such as in huge communities. We used an artificial neural network in order to identify the reliable users in a community, using the Core component on the training phase. The results show that the usage of artificial neural network can be a good method for finding the reliable users in a community. Although achieving good results, this approach has one limitation as it still depends on the training of the neural network. Then, we also tested the correlation considering only 20% of the Core members for the training phase, which still proved to provide good results. Notice, that the network may provide the information about some of the reliable users or it could be obtained from a group of professionals, for example. In addition, we analyzed the possibility of using a previous trained neural network in some community and apply it to another one. We made tests using the Core (and 20% of it) as the training dataset in several scenarios and found good outcomes when using larger communities with similar sizes. Finally, we performed tests using the K-means clustering algorithm in order to make groups of similar users. After some tests, we discovered that the higher is the average number of answers of a group, the greater is the probability of finding a reliable user in it. We expect that the results obtained in this work can support students in an online community, facilitating their contact with more experienced people as it may indicate the discovery of possible tutors or instructors in an informal learning environment. As future works, we intend to identify the most adequate person to answer a specific question. For instance, an expert in software engineering may not be the most adequate to answer a question on compilers. Then, a possible future work is to build a model that allows finding the most adequate people to answer a specific question (willing to help and knowledgeable). This model can be built using Semantic Web technologies since it has been widely used to organize and manage knowledge from several different information domains (Labra Gayo, Ordóñez De Pablos, & Cueva Lovelle, 2010; Lytras & Ordóñez de Pablos, 2011; Lytras, Sakkopoulos, & Ordóñez de Pablos, 2009). Considering this approach, one possibility is to make analysis (like the ones presented on this work) in order to extract knowledge from a community and then, organize it in a Semantic Web model such as ontologies. After building this semantic model, we expect to easily find the reliable users using inference rules. Acknowledgments This work was partially supported by FAPERJ (through Grants E-26/170028/2008 INC&T Program – Project: Brazilian Institute

Please cite this article in press as: Procaci, T. B., et al. How to find people who can help to answer a question? – Analyses of metrics and machine learning in online communities. Computers in Human Behavior (2014), http://dx.doi.org/10.1016/j.chb.2014.12.026

10

T.B. Procaci et al. / Computers in Human Behavior xxx (2014) xxx–xxx

of Research on Web Science, and E-26-102.256/2013 – BBP/Bursary Associa: Exploring a Semantic and Social Learning-Teaching Environment), and CNPQ (Project: 557.128/2009-9, INCT on Web Science). References Ackerman, M. S., & McDonald, D. W. (1996). Answer Garden 2: Merging organizational memory with collaborative help. In Proceedings of CSCW ‘96 (pp. 97–105). Boston, MA: ACM Press. Adamic, L., Zhang J., Bakshy E., & Ackerman, M. S. (2008). Knowledge sharing and yahoo answers: Everyone knows something. In Proceedings of the 17th international conference on World Wide Web, April 21–25, Beijing, China. Alan, W. G., Jian, J., Abrahams, A. S., Fan, W., & Zhang, Z. (2013). ExpertRank: A topicaware expert finding algorithm for online knowledge communities. Decision Support Systems, 0167-9236, 54(3), 1442–1451. http://dx.doi.org/10.1016/ j.dss.2012.12.020. Awad, E. M., & Ghaziri, H. M. (2004). Knowledge management. New Jersey: Pearson Education Ltd.. Bader, D., & Madduri, K. (2006). Parallel algorithms for evaluating centrality indices in real-world networks. Proceedings of 35th int’l conference on parallel processing (ICPP). Columbus, OH: IEEE Computer Society. Balog, K., Azzopardi, L., & Rijke, M. D. (2009). A language modeling framework for expert finding. Information Processing & Management, 45(1), 1–19. Banerjee, A., & Basu, S. (2008). A social query model for decentralized search. Proceedings of 2nd workshop on social network mining and analysis. ACM Press. Benson, V., Morgan, S., & Tennakoon, H. (2012). A framework for knowledge management in higher education using social networking. International Journal of Knowledge Society Research (IJKSR), 3(2), 44–54. http://dx.doi.org/10.4018/ jksr.2012040104. Brandes, U. (2001). A faster algorithm for betweenness centrality. Journal of Mathematical Sociology, 25(2), 163–177. Broder, A., Kumar, R., Maghoul, F., Raghavan, P., Rajagopalan, S., Stata, R., et al. (2000). Graph structure in the Web. Computer Networks, 33(1–6), 309–320. Campbell, C. S., Maglio, P. P., Cozzi, A., & Dom, B. (2003). Expertise identification using email communications. In The twelfth international conference on information and knowledge management, New Orleans, LA (pp. 528–231). Constant, D., Sproull, L., & Kiesler, S. (1996). The kindness of strangers: The usefulness of electronic weak ties for technical advice. Organization Science, 7(2), 119. Davitz, J., Yu, J., Basu, S., Gutelius, D., & Harris, A. (2007). iLink: Search and routing in social networks. In Proceedings of the 13th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 931–940). ACM Press. Dom, B., Eiron, I., Cozzi, A., & Zhang, Y. (2003). Graph-based ranking algorithms for e-mail expertise analysis. In DMKD (pp. 42–48). New York, NY: ACM Press. Fernandes, G. T., & Siqueira, S. W. M. (2013). Building users profiles dynamically through analysis of messages in collaborative systems. In Proceedings international conference WWW/Internet, 2013 (Vol. 12, pp. 1–8). Forth Worth. Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York: John Wiley. ISBN 0-471-26370-2. Fritzen, E., Siqueira, S. W. M., & Andrade, L. C. V. (2013). Contextualized search enriched with open data for supporting collaborative learning in social networks (Busca Contextualizada Enriquecida com Dados Abertos para Apoiar a Aprendizagem Colaborativa em Redes Sociais). Brazilian Journal on Computers and Education (Revista Brasileira de Informática na Educação), 21(3), 25–39. http://dx.doi.org/10.5753/RBIE.2013.21.03.25 [in Portuguese]. Horowitz, D., & Kamvar, S. (2010). The anatomy of a large-scale social search engine. In Proceedings of the 19th international conference on World Wide Web (WWW) (pp. 431–440). ACM Press. Ismail S. & Ahmad M. S. (2011). Emergence of social intelligence in social network: A quantitative analysis for agent-mediated PKM processes. In Proc. of the ICIMu 2011 conference, Malaysia. Ismail, S. & Ahmad, M. S. (2012). Effective personal knowledge management: A proposed online framework. In Proceedings of international conference on information and knowledge society. Kautz, H., Selman, B., & Shah, M. (1997). Referral web: Combining social networks and collaborative filtering. Communications of the ACM, 40(3), 63–65. Kohavi, Ron (1995). A study of cross-validation and bootstrap for accuracy estimation and model selection. In Proceedings of the fourteenth international joint conference on artificial intelligence (Vol. 2, pp. 1137–1143). San Mateo, CA: Morgan Kaufmann (12) CiteSeerX: 10.1.1.48.529. Kollock, P. (1999). The economies of online cooperation: Gifts and public goods in cyberspace. In M. A. Smith & P. Kollock (Eds.), Communities in cyberspace. London: Routledge.

Krulwich, B. & Burkey, C. (1996). ContactFinder agent: Answering bulletin board questions with referrals. In the 13th national conference on artificial intelligence, Portland, OR, pp. 10–15. Labra Gayo, J. E., Ordóñez De Pablos, P., & Cueva Lovelle, J. M. (2010). WESONET: Applying semantic web technologies and collaborative tagging to multimedia web information systems. Computers in Human Behaviour, 26(2), 205–209. Lakhani, K., & Von Hippel, E. (2000). How open source software works: ‘‘free’’ userto-user assistance. Research Policy, 32(6), 923–943. Landis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for categorical data. Biometrics, 33(1), 159–174. http://dx.doi.org/10.2307/ 2529310. JSTOR 2529310. PMID 843571. Li, Yung-Ming, Liao, Tzu-Fong, & Lai, Cheng-Yang (2012). A social recommender mechanism for improving knowledge sharing in online forums. Information Processing & Management, 48(5), 978–994. Lin, F., & Huang, H. (2013). Why people share knowledge in virtual communities?: The use of Yahoo! Kimo Knowledge+ as an example. Internet Research, 23(2), 133–159. http://dx.doi.org/10.1108/10662241311313295. Littlepage, G. E., & Mueller, A. L. (1997). Recognition and utilization of expertise in problem-solving groups: Expert characteristics and behavior. Group Dynamics: Theory, Research, and Practice, 1, 324–328. Liu, X., Wang, G. A., Johri, A., Zhou, M., & Fan, W. (2012). Harnessing global expertise: A comparative study of expertise profiling methods for online communities. Information Systems Frontiers, 1–13. Lytras, M. D., & Ordóñez de Pablos, P. (2011). Software technologies in knowledge society. Journal of Universal Computer Science, 17(9), 1219–1221. Lytras, M. D., Sakkopoulos, E., & Ordóñez de Pablos, P. (2009). Semantic Web and Knowledge Management for the health domain: state of the art and challenges for the Seventh Framework Programme (FP7) of the European Union (2007– 2013). International Journal of Technology Management, 47(1/2/3). 239–249.7. Macqueen, J. B. (1967). Some methods for classification and analysis of multivariate observations. In Proceedings of 5th Berkeley symposium on mathematical statistics and probability (pp. 281–297). University of California Press. MR 0214227. Zbl 0214.46201. Retrieved 2009–04-07. Marques, A. M., Krejci, R., Siqueira, S. W. M., Pimentel, M., & Braz, M. H. L. B. (2013). Structuring the discourse on social networks for learning: Case studies on blogs and microblogs. Computers in Human Behavior, 29(2), 395–400. http:// dx.doi.org/10.1016/j.chb.2012.03.001. Mattox, D., Maybury, M., & Morey, D. (1999). Enterprise expert and knowledge discovery. In Proceedings of the 8th international conference on human-computer interaction (pp. 23–27). Germany: Munich. Morris, M., Teevan, J., & Panovich, K. (2010). What do people ask their social networks, and why?: A survey study of status message Q&A behavior. In Proceedings of the 28th international conference on human factors in computing systems (CHI) (pp. 1739–1748). ACM Press. Page, L., Brin, S., Motwani, R., & Winograd, T. (1998). The pagerank citation ranking: Bringing order to the web. In Stanford digital library technologies project. Paul, S., Hong, L., & Chi, E. (2011). Is twitter a good place for asking questions? a characterization study. In The eighth international conference on internet and web applications and services conference on weblogs and social media (ICWSM), pp. 578–581. Prates, J. C., Fritzen, E., Siqueira, S. W. M., Braz, M. H. L. B., & Andrade, L. C. V. (2013). Contextual web searches in Facebook using learning materials and discussion messages. Computers in Human Behavior, 29(2), 386–394. http://dx.doi.org/ 10.1016/j.chb.2012.06.002. Procaci, T. B., Siqueira, S. W. M., & Andrade, L. C. V. (2014). Finding experts on Facebook communities: Who knows more? International Journal of Knowledge Society Research. Serdyukov, P., Rode, H., & Hiemstra, D. (2007). University of Twente at the TREC 2007 Enterprise Track: modeling relevance propagation for the expert search task. In The 16th text retrieval conference (TREC 2007), Enterprise Track, Gaithersburg, MD. Souza, C. C., Magalhães, J. J., Costa, E. B., & Fechine, J. M. (2013). Social query: A query routing system for Twitter. In: The eighth international conference on Internet and Web Applications and Services (ICIW). Streeter, L. & Lochbaum, K. (1998). Who knows: A system based on automatic representation of semantic structure. In Proceedings of RIAO (pp. 380–388). West, J., Salter, A., Vanhaverbeke, W., & Chesbrough, H. (2014). Open innovation: The next decade. Research Policy, 43(5), 805–811. Zhang, J., Ackerman, M. S., & Adamic, L. (2007). Expertise networks in online communities: structure and algorithms. In Proceedings of the 16th international conference on World Wide Web, May 08–12. Alberta, Canada: Banff. Zhuge, H., & Zhang, J. (2010). Topological centrality and its e-science applications. Journal of American Society for Information Science and Technology, 61(9), 1824–1841.

Please cite this article in press as: Procaci, T. B., et al. How to find people who can help to answer a question? – Analyses of metrics and machine learning in online communities. Computers in Human Behavior (2014), http://dx.doi.org/10.1016/j.chb.2014.12.026