Understanding the determinants of online review helpfulness: A meta-analytic investigation

Accepted Manuscript Understanding the determinants of online review helpfulness: A meta-analytic investigation

Hong Hong, Di Xu, G. Alan Wang, Weiguo Fan PII: DOI: Reference:

S0167-9236(17)30119-7 doi: 10.1016/j.dss.2017.06.007 DECSUP 12858

To appear in:

Decision Support Systems

Received date: Revised date: Accepted date:

7 December 2016 7 May 2017 27 June 2017

Please cite this article as: Hong Hong, Di Xu, G. Alan Wang, Weiguo Fan , Understanding the determinants of online review helpfulness: A meta-analytic investigation, Decision Support Systems (2017), doi: 10.1016/j.dss.2017.06.007

This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

ACCEPTED MANUSCRIPT Understanding the Determinants of Online Review Helpfulness: A Meta-Analytic Investigation Hong Hong1, Di Xu1, G. Alan Wang2*, Weiguo Fan3 : Department of Management Science, Xiamen University, Fujian, China

2

: Department of Business Information Technology, Virginia Tech, Blacksburg, VA

3

: Department of Accounting and Information Systems, Virginia Tech, Blacksburg, VA

RI

PT

1

SC

Contact information: [email protected]

NU

Abstract: Online consumer reviews can help customers reduce uncertainty and risks faced in online

MA

shopping. However, the studies examining the determinants of perceived review helpfulness produce mixed findings. We review extant research about the determinant factors of perceived online review

D

helpfulness. All review related determinants (i.e., review depth, review readability, linear review rating,

PT E

quadratic review rating, review age) and two reviewer related determinants (i.e., reviewer information disclosure and reviewer expertise) are found to have inconsistent conclusions on how they affect

CE

perceived review helpfulness. We conduct a meta-analysis to examine those determinant factors in

AC

order to reconcile the contradictory findings about their influence on perceived review helpfulness. The meta-analysis results affirm that review depth, review age, reviewer information disclosure, and reviewer expertise have positive influences on review helpfulness. Review readability and review rating are found to have no significant influence on review helpfulness. Moreover, we find that helpfulness measurement, online review platform, and product type are the three factors that cause mixed findings in extant research. Keywords: Online customer reviews; Review helpfulness; Meta-analysis; Review

1

ACCEPTED MANUSCRIPT

1. Introduction With the fast development of electronic commerce, online shopping plays an increasingly important role in our daily lives because of its low cost and convenience. Compared to shopping at brick-and-mortar stores, online shopping is unique in its temporal and spatial separation of buyers and

PT

sellers (Luo et al. 2012). It is difficult for consumers to experience products or services before buying

RI

them. Hence, consumers have to face a higher degree of uncertainty and risk in online shopping than

SC

that in in-store shopping.

Recent advances in information technologies have enabled the use of user-generated content (UGC)

NU

such as online customer reviews, which help consumers decrease the uncertainty and risks associated

MA

with online purchasing decisions. Online customer reviews are defined as peer-generated evaluations about products or services posted on retailer or third-party websites (Mudambi et al. 2010). Typically,

D

an online review includes a star rating (usually ranges from 1 to 5 stars) and written comments about the

PT E

experience of using a product or service and critique about product or service features (Mudambi et al. 2010). Online reviews, often viewed as online WOM, can impact product sales (Dellarocas et al. 2007;

CE

Duan et al. 2008; Liu 2006) through increasing consumers’ awareness about products (Chen et al. 2004;

AC

Godes et al. 2004; Liu 2006) or impacting consumers’ attitudes toward products (Wang et al. 2015b). Therefore, many businesses utilize online WOM as a new marketing strategy (Chen et al. 2008). In order to better utilize user-generated content, Wang et al. (2013) develop an effective knowledge management system for facilitating information seeking and sharing in online communities. However, the problem of information overload and conflicting comments in online reviews can get consumers confused. For example, comment spamming in online consumer reviews may decrease the efficiency of consumers’ decision-making (Chen et al. 2011). Therefore, it is important for researchers and 2

ACCEPTED MANUSCRIPT practitioners to understand how online consumers perceive the helpfulness of online reviews (Hao et al. 2010). Both e-commerce practitioners and researchers have examined ways of obtaining helpful reviews. Many e-commerce websites, such as Amazon and Yahoo! Movie, provide a helpfulness feedback

PT

mechanism for online consumer reviews. The mechanism has been found effective in promoting sales.

RI

According to Spool (2009), a simple review feedback question asked by Amazon, “do you think this

SC

review is helpful”, had brought Amazon $2.7 billion extra revenue ①. However, user participation is critical to the success of user-specified helpfulness feedback. Lu et al. (2010) find that a large

NU

proportion of reviews have few or no helpfulness feedbacks, particularly the more recent ones. Newly

MA

written reviews do not have enough time to accumulate helpful votes (Kim et al. 2006), therefore, user-specified helpfulness feedback are too sparse for online users to assess the helpfulness of reviews

D

(Tang et al. 2013). On the other hand, researchers have examined review characteristics, such as review

PT E

rating and content, that can be used as indicators of perceived review helpfulness. However, they have reached mixed findings (Yin et al. 2016; Yin 2012). For example, Pan et al. (2011) find that reviews

CE

with high star ratings are more likely to be perceived as helpful reviews. In another study, Mudambi et

AC

al. (2010) argue that star ratings and online review helpfulness have an inverted "U" shaped relationship for experience products. Moreover, Chua et al. (2015) imply a negative curvilinear relationship between review ratings and perceived review helpfulness. The mixed findings on the determinants of perceived review helpfulness create confusions to both academic researchers and e-commerce practitioners. It is common to have mixed research findings in social and behavioral sciences (Hunter et al. 1982). A single research study is often constrained by its research context and may not sufficiently consider the

Spool J M. “The Magic behind Amazon’s 2.7 Billion Dollar Question”, http://www.uie.com/articles/magicbehindamazon/. 3

①

ACCEPTED MANUSCRIPT complexity of the constructs being investigated. Meta-analysis is used to find a common truth behind all conceptually similar studies by statistically combining their results (King et al. 2005). Meta-analysis has been applied to study the word-of-mouth (WOM) effect. For example, Floyd et al. (2014) conduct a meta-analysis on 26 empirical studies to examine how review valence and volume influence the

PT

elasticity of retailer sales. You et al. (2015) conduct a meta-analysis on the direct effect of online WOM

RI

on sales and the moderating effect of product characteristics, industry characteristics, and platform

SC

characteristics. The meta-analysis conducted by Babić Rosario et al. (2016) aims to contribute to the understanding of the influence of platform characteristics, product characteristics, and online WOM

NU

metrics on the relationship between online WOM and sales. Although WOM and its impact on product

MA

sales are important research topics, those meta-analysis studies fail to explain how online customers perceive the helpfulness of online consumer reviews. Lee et al. (2016) find that perceived online review

D

helpfulness mediates the relationship between online review characteristics and product sales. It is

PT E

important to understand how online customers perceive the helpfulness of online customer reviews in addition to its ultimate impact on product sales. The findings can potentially help consumers identify

CE

helpful reviews and help e-commerce practitioners build effective online review platforms.

AC

In this study we aim to conduct a meta-analysis to reconcile and understand the mixed findings related to the determinants of online review helpfulness instead of product sales. The perception of review helpfulness is an important step in the decision-making process of online shopping. Based on marketing literature, Mudambi et al. (2010) argue that information search and evaluation of alternatives are important steps before customers make purchase decisions. The perception of review helpfulness can significantly change the outcome of information search and alternative evaluations. Existing research that studies the impact of customer reviews on products sale uses review helpfulness ratings or 4

ACCEPTED MANUSCRIPT helpfulness votes without explaining what makes an online review helpful to customers. Purnawirawan et al. (2015) has attempted a meta-analysis study in order to understand the impact of online review valence on perceived review helpfulness. However, their study does not consider other important determinants of online review helpfulness. We therefore attempt to fill this research gap by conducting a

PT

comprehensive meta-analysis to examine all the major determinants of perceived online review

RI

helpfulness found in existing studies, and investigate the impact of various moderators on the direct

SC

effects.

The rest of this paper is structured as follows. We first review existing studies related to the

NU

determinants of perceived review helpfulness and put forward our research hypotheses in Section 2. In

MA

Section 3 we present our research methodology and data collection process. In Section 4 we report the meta-analysis results and discuss the reasons why mixed findings exist on the relationships between

D

review helpfulness and its determinants. In the final section, we conclude our paper by discussing the

PT E

contributions, limitations, and future directions of our study. 2. Literature Review and Hypotheses Development

CE

In order to accurately understand online review helpfulness, many researchers have proposed the

AC

determinant factors that may influence perceived review helpfulness. The objective is to understand what kind of online consumer reviews is helpful for fellow consumers. In addition, practitioners can better understand the characteristics of online reviews that may influence consumers’ perceived helpfulness in their processes of making purchasing decisions (Ghose et al. 2011). Review helpfulness studies commonly use two types of data sources: (1) First-hand data collected using surveys or questionnaires (Cheung et al. 2012; Connors et al. 2011; Schlosser 2011); (2) Second-hand data scraped from online review systems provided by e-commerce practitioners. Collecting first-hand data is often 5

ACCEPTED MANUSCRIPT time consuming and subject to common method bias (Podsakoff et al. 2003). Second-hand data have the advantage of quickly collecting a large number of reviews and have been commonly used in online customer review studies. Our meta-analysis will focus on those studies using second-hand data collected from platforms hosted by either retailers themselves or third party providers.

PT

2.1. Major determinants of review helpfulness

RI

Based on an extensive literature search, we categorize the determinants of review helpfulness into two

SC

categories: (1) Review related factors that are derived from review ratings and contents, including review depth, review readability, linear review rating, quadratic review rating, and review age; (2)

NU

Reviewer related factors that are derived from reviewers’ background and self-description, including

MA

reviewer information disclosure, reviewer expertise, reviewer expert label, reviewer friend number, and reviewer fan number. Table 1 provides a summary of those determinants, including their definitions and

D

operationalization used in different studies. As Table 1 shows, 7 out of 10 identified review helpfulness

PT E

determinants show inconsistent findings in their relationships with perceived review helpfulness. Only 3 out of 5 reviewer related factors, namely reviewer expert label, reviewer friend number, and reviewer

CE

fan number, have consistent findings over their positive influence on perceived review helpfulness. All

AC

review related determinants and 2 out of 5 reviewer related factors (i.e., reviewer information disclosure and reviewer expertise) are found to have inconsistent influence on review helpfulness. Therefore, our literature review suggests that a meta-analysis is necessary to understand and reconcile the contradictory findings on those review helpfulness determinants. Following the meta-analysis study conducted by Purnawirawan et al. (2015), we formulate expected relationships between online review helpfulness and its determinants based on the major tendency found in existing literature (Table 1). For conflicting relationship found between a determinant and review helpfulness, we adopt a majority 6

ACCEPTED MANUSCRIPT principle. For a determinant with equal study number of positive and negative relationship with review helpfulness (e.g., review readability), we expect the impact of this determinant on review helpfulness is unclear. And the expected relationships would be tested by meta-analysis results.

Expected relationship Positive

Unclear

AC

CE

PT E

D

MA

NU

SC

RI

PT

Table 1. Major determinants of perceived review helpfulness in existing studies Definition or Prior Category Determinant Representative study measure finding Review Review Total number of Positive Siering et al. (2013), Wu (2013), Kang et related depth words of a al. (2016), Salehan et al. (2016), Ullah et factors review; review al. (2015), Yin et al. (2016), Kuan et al. word number; (2015), Chua et al. (2015), Baek et al. word count; (2012), Quaschning et al. (2015), Einar et review length; al. (2015), Yin (2012), Mudambi et al. review (2010), Willemsen et al. (2011), Bao et al. elaborateness. (2016), Zhang et al. (2010), Ghose et al. (2007), Yin et al. (2012), Yan et al. (2012), Yan et al. (2013), Pan et al. (2011), Ahmad et al. (2015), Lee et al. (2016), Yin et al. (2014a), Yin et al. (2014b), Cheng et al. (2015), Hlee et al. (2016), Zhou et al. (2015), Zhu et al. (2014), Park et al. (2015), Liu et al. (2015), Fang et al. (2016), Guo et al. (2016), Li et al. (2016) Negative Racherla et al. (2012) Review Ease of Positive Kang et al. (2016), Korfiatis et al. (2012), readability understanding of Ghose et al. (2011), Park et al. (2015), reviews, Liu et al. (2015), Hlee et al. (2016) measured by Negative Korfiatis et al. (2012), Yin et al. (2014b), Gunning’s fog Zhu et al. (2014), Yin et al. (2014a), Fang index, Automated et al. (2016), Zhou et al. (2015) readability index and the Coleman-Liau index. The lower the measure, the more readable the text is. Linear Review rating Positive Wu (2013), Wu et al. (2011), Huang et al. review usually ranges (2015), Ullah et al. (2015), Quaschning et rating from one star to al. (2015), Einar et al. (2015), Korfiatis et five stars. al. (2012), Willemsen et al. (2011), Yin et al. (2012), Yan et al. (2012), Pan et al. (2011), Ahmad et al. (2015), Liu et al. (2015)

7

Positive

ACCEPTED MANUSCRIPT

Positive

Review age

Review elapsed days; days elapsed after the review being posted.

Positive

Reviewer information disclosure

Disclosure of personal information, e.g., real name, self-photo, location, reviewer identity. Total number of reviews on the platform published by the reviewer. Dummy variable of whether the reviewer has expert/elite badge, rank 10,000 label; credibility. Reviewers’ follow number; reviewer out-degree centrality. Reviewers’ fans; reviewer in-degree centrality.

Reviewer expert label

Positive

Positive

Yin (2012), Cheng et al. (2015), Zhu et al. (2014), Park et al. (2015)

Positive

MA

Negative

Reviewer fan number

Positive

NU

Negative Positive

SC

RI

Racherla et al. (2012), Park et al. (2015), Liu et al. (2015), Ghose et al. (2007), Pan et al. (2011), Yin et al. (2014b), Zhu et al. (2014), Yin et al. (2014a), Kwok et al. (2016), Zhou et al. (2015) Yin et al. (2012) Einar et al. (2015), Willemsen et al. (2011), Forman et al. (2008), Ghose et al. (2011), Zhou et al. (2015), Park et al. (2015), Liu et al. (2015), Guo et al. (2016) Ghose et al. (2011)

Racherla et al. (2012)

Positive

Kuan et al. (2015), Baek et al. (2012), Zhou et al. (2015), Yin et al. (2014a), Guo et al. (2016)

Positive

Positive

Huang et al. (2015), Yin (2012), Racherla et al. (2012), Zhou et al. (2015), Cheng et al. (2015), Zhu et al. (2014), Liu et al. (2015), Yin et al. (2014a), Guo et al. (2016) Li et al. (2015), Yin (2012), Yin et al. (2012)

Positive

Negative

AC

Reviewer friend number

Positive

D

Reviewer expertise

Negative

PT E

Reviewer related factors

Review rating*Review rating; Quadratic term of review rating.

CE

Quadratic review rating

Wang et al. (2015a), Chua et al. (2015), Zhang et al. (2010), Liao et al. (2013), Yin et al. (2014b), Racherla et al. (2012), Chen et al. (2014), Zhou et al. (2015), Zhu et al. (2014), Park et al. (2015), Kwok et al. (2016), Guo et al. (2016) Baek et al. (2012), Mudambi et al. (2010), Racherla et al. (2012), Park et al. (2015), Liu et al. (2015) Chua et al. (2015), Wu et al. (2011), Yin et al. (2012)

PT

Negative

Positive

Positive

2.2. Moderating effects Examining the influence of potential moderators in a meta-analysis can help shed light on the 8

ACCEPTED MANUSCRIPT inconsistent findings obtained from previous studies that were conducted in different research contexts (Hong et al. 2013). Similar to other meta-analysis studies on online WOM, such as Purnawirawan et al. (2015) and de Matos et al. (2008), the identification of potential moderators is restricted to the variables that exist in the studies included in the meta-analysis and that may influence the direct relationship

PT

between online review characteristics and its consequences. As a result, we have identified three

RI

potential moderators, namely the measure of review helpfulness, review platform, and product type.

SC

The measures being used may impact the studied relationships, because different item characteristics may lead different measures to bias construct validity (Podsakoff et al. 2003). Hong et al. (2013)

NU

investigate the moderating role of measures of service climate and rating sources as moderators

MA

between service climate and outcomes. As for our research question, measures for review helpfulness are not unified in existing studies (i.e., some studies use helpful vote ratio as the measurement, while

PT E

inconsistent findings.

D

others use helpful vote number to measure), hence, measures of review may be responsible for

Floyd et al. (2014) use review platform as a theoretical variable to conduct meta-analysis and find

CE

that whether the reviews are obtained from a third-party platform has influence on sales elasticities. Gu

AC

et al. (2012) find that external reviews from third-party platforms have more influence on consumers’ decision making than internal reviews from retailer-hosted platforms. Consumers may have different mechanisms to evaluate review helpfulness from different platforms. So we use review platform as a potential reason for extant mixed findings. Product type is often used as a moderator in studies on online review helpfulness, e.g., Baek et al. (2012) and Mudambi et al. (2010). Purnawirawan et al. (2015) use product type as a moderator to conduct the meta-analysis on the influence of review valence on review helpfulness, and purchasing 9

ACCEPTED MANUSCRIPT intention etc. We also use product type as a moderator in our meta-analysis. Table 2 summarizes the definition and operationalization of the three moderators. We will explain the hypothesized moderating effects of the three factors in the rest of this section. 2.2.1. Operationalization of online review helpfulness

PT

The operationalization of the review helpfulness measure is a methodological moderator that

RI

may potentially explain the inconsistent relationships between determinants and online review

SC

helpfulness in prior studies (Schmidt et al. 2014). In their seminal paper, Mudambi et al. (2010) define online review helpfulness as consumers’ perceived value of online reviews while shopping online. They

NU

propose to use the ratio of helpful votes to total votes received as the measure of perceived review

MA

helpfulness. However, several subsequent studies choose to use a variant measure, namely the absolute total number of helpful votes received, as the perceived helpfulness measure (Park et al. 2015; Racherla

D

et al. 2012; Yin et al. 2014a; Yin et al. 2014b; Zhu et al. 2014). The change in measurement is mainly

PT E

caused by the restriction set forth by their data sets collected from Yelp reviews, which only displays the total number of helpful votes. Amazon reviews, which is another commonly used dataset in online

CE

review studies, allow the calculation of the helpful vote ratio because both the number of helpful votes

AC

and the total number of votes are available. Ahmad et al. (2015) find that the helpful vote ratio is a better measurement after comparing both measures in their study. We will review papers using both helpful vote ratio and helpful vote number and summarize determinant factors of review helpfulness in extant research. We consider both operationalization methods in our meta-analysis because both measures will co-exist in future online review studies as long as data restrictions remain. Studies using different review helpfulness measures, namely the helpful vote ratio and the number of helpful vote, generally have conflicting findings on the same determinant factor of review helpfulness, except on the factors of 10

ACCEPTED MANUSCRIPT reviewer expert label and reviewer friend number. Therefore, the discrepancy in the review helpfulness measure can be an important cause for the mix findings. Thus, we put forward the following hypothesis: H1: The operationalization of the online review helpfulness measure can moderate the relationships between determinants on online review helpfulness.

PT

2.2.2. Context-specific factors

RI

We consider two moderating factors, namely, review platform and product type, that are

SC

related to the research context in which online review helpfulness studies take place. Online reviews can be divided into internal online reviews (e.g., reviews from Amazon) and external

NU

online reviews (e.g., reviews from TripAdvisor) according to the platforms from which they are

MA

obtained. Online review data collected in different studies vary from the platforms hosted by retailers themselves to the platforms hosted by third-party providers. Different data sources may impact the

D

findings over the determinants of perceived review helpfulness. Consumers can obtain product

PT E

information from a variety of sources. User-generated contents such as online WOM are more trustworthy than seller-generated contents (Bickart et al. 2001). Xiang et al. (2017) examine

CE

information quality of online reviews on TripAdvisor, Expedia, and Yelp, and find that there are huge

AC

discrepancies in the representation of the hotel industry on these platforms. Park et al. (2012) find that about half of consumers would scan online reviews in multiple platforms while shopping online, hence, both types of online reviews play a role in consumers’ decision-making process. After estimating the relative impact of different online review sources on product sales, Gu et al. (2012) find that consumers are less influenced by internal reviews. External review platforms usually do not sell products, therefore, those platform providers are less likely to manipulate online reviews than retailers. That is primarily the reason why consumers consider external reviews from third-party sources as being more trustworthy 11

ACCEPTED MANUSCRIPT (Floyd et al. 2014). Trustworthiness has been found to positively influence consumers’ attitudes towards the product, Floyd et al. (2014) find external reviews have significantly higher sales elasticities than internal reviews. The influence of determinants on helpfulness perceived by consumers may different for reviews from different platforms. Hence, we pose the following hypothesis:

PT

H2: Online review platform can moderate the relationships between determinants and online review

RI

helpfulness.

SC

Existing online review helpfulness studies are usually focused on a particular type of products. The difference in product type may also change how review helpfulness determinants influence

NU

perceived review helpfulness. Consumers may read online reviews from different perspectives

MA

depending on their purpose of reading online reviews during information search stage. The purpose of reading online reviews can be different depending on what products consumers intend to buy (Baek et al.

D

2012). Product type is usually regarded as a moderator between determinants and online review

PT E

helpfulness (Baek et al. 2012; Lee et al. 2016; Mudambi et al. 2010; Purnawirawan et al. 2015; Siering et al. 2013). Nelson (1970) categorize products into search products and experience products. Search

CE

products, such as electronics, are those products whose quality evaluation can be obtained via

AC

information searches before a purchase is made. Experience products, such as restaurants or music, are those products whose quality can only be evaluated with certainty after purchase and actual usage. The major difference between search products and experience products lies in different degree of uncertainty of product quality before purchasing (Luo et al. 2012). Quality evaluation of search products is objective while that of experience products is subjective and dependent on usage experience (Sen et al. 2007). Girard et al. (2010) confirm that consumers have to face greater pre-purchase uncertainties and risks for experience products than search products, because consumers lack product 12

ACCEPTED MANUSCRIPT experience before purchase of experience products (Park et al. 2013; Zhu et al. 2010). A search product can be evaluated based on the sellers’ information, but the sellers’ information is not enough to make a purchase decision for an experience product (Baek et al. 2012). Therefore, they might seek additional information as an effort to reduce perceived risk and uncertainty (Park et al. 2009). Therefore, we

PT

propose the following hypothesis:

RI

H3: The product type of online reviews can moderate the relationships between determinants and

SC

online review helpfulness. 2.2.3. Summary

NU

In order to understand how the review helpfulness measure, review platform, and product type may

MA

affect the findings related to perceived review helpfulness, we will conduct three subgroup analyses to compare the findings between subgroups. For the moderator analysis, we first partitioned the data into

D

individual subgroups based on the categories presented in Table 2. We can categorize existing studies,

PT E

which is summarized in Table 1, into two subgroups based on the helpfulness measure used (i.e., 34 studies using helpful vote ratio to measure review helpfulness and 15 studies using helpful vote number

CE

to measure review helpfulness) ② , two subgroups based on the review platform (i.e., 28 studies

AC

investigate reviews from internal review platform and 20 studies investigate reviews from external review platform), and two subgroups based on the product type (i.e., 37 research studies experience product reviews and 17 research studies search product reviews) ③.

Moderator Helpfulness measure

② ③

Table 2. Moderator definitions Definition Operationalization The measurement of online review helpfulness.

Helpful vote ratio: The ratio of the number of helpful votes to the total number of votes received. Helpful vote number: Total number of helpful votes.

Ahmad and Laroche (2015) use both helpful vote ratio and helpful vote number to measure review helpfulness. 9 studies investigate reviews of both experience and search product, the product type cannot be figured out for 3 studies.

13

Study number 34 15

ACCEPTED MANUSCRIPT Review platform

Product type

The review platform where online reviews are collected. The product type in the research context.

Internal review platform: Online reviews are obtained from retailer-hosted platforms. External review platform: Online reviews are collected from third party-hosted platforms. Experience product: The products whose quality can only be evaluated with certainty after purchase and usage. Search product: The products whose quality evaluation can be obtained via searches before a purchase is made.

28 20 37 17

PT

3. Research Methodology

RI

Originated from Fisher’s “combining P values” method, meta-analysis was developed into “combining

SC

statistics” (Glass 1976). It is a popular statistical method to systematically synthesize and analyze the quantitative results from empirical studies that address similar research questions (Lipsey et al. 2001;

NU

Schmidt et al. 2014). It allows a mathematical combination of correlations between two or more

MA

variables to reconcile inconsistent findings due to measurement errors, low statistical power, and difference in the research context (Ortiz de Guinea et al. 2012). Therefore, meta-analysis results can

D

reconcile conflicting research findings and offer directions and insight for future studies (Schmidt et al.

PT E

2014). Meta-analysis was first used in the medical and psychological fields. Now it is a popular method in both natural science and social science areas (Hjørland 2001). It has been applied to information

CE

systems research since 2000 (Hwang 1996; King et al. 2005). King et al. (2005) discuss the application

AC

of meta-analysis in the field of information systems and consider meta-analysis as a formal and systematic literature review method. Meta-analysis has been applied to study online word-of-mouth effect, such as Floyd et al. (2014), You et al. (2015), and Babić Rosario et al. (2016). In this study, we used meta-analysis to mathematically synthesize the results of previous studies on the determinants of online review helpfulness. There are two major steps in the classic meta-analysis method. The first step is to extract statistical test data and effect sizes from multiple studies. The second step is to standardize the effect sizes and subject them to a null hypothesis in order to test the statistical relationships (Glass 14

ACCEPTED MANUSCRIPT 1976). We will describe how we conduct our meta-analysis in detail in Section 3.3. 3.1. The Selection of Relevant Studies In order to avoid publication bias, this study used a multi-channel literature search. For English studies, we searched literature from commonly used digital library databases such as ScienceDirect, EBSCO,

PT

SAGE, and Taylor & Francis. In addition, we manually searched related papers from four prestigious

RI

information systems (IS) journals where research related to perceived review helpfulness are mostly

SC

likely to be published, including MIS Quarterly, Information Systems Research, Journal of Management Information Systems, Journal of the Association for Information Systems, and Decision

NU

Support Systems. Online consumer reviews have also been extensively studied in the field of marketing

MA

because of its impact on product sales and firm performance. Therefore, we also searched papers from three prestigious marketing journals, namely Journal of Consumer Research, Journal of Marketing, and

D

Journal of Marketing Research. In addition to published articles, we also downloaded working papers

PT E

from the Social Science Research Network (SSRN) database. Keywords used for literature search are online reviews, web reviews, user-generated content, usefulness, and perceived helpfulness.

CE

Based on our research goal, we made the following selection criteria when including related studies:

AC

(1) it is a published or working academic paper; (2) its research topic is related to online consumer reviews, the word-of-mouth effect, or user-generated content; (3) it examines the influence of online review on perceived review helpfulness or product sales; (4) it is an empirical study on perceived review helpfulness. After downloading relevant papers from the databases mentioned above, we applied the following exclusion rules to the relevant papers: (1) Studies that do not report correlation coefficient values (r) or the sample size, both of which are necessary to study the effect size in meta-analysis (Hameed et al. 2012); (2) Studies that use first-hand data such as survey or experiment 15

ACCEPTED MANUSCRIPT data. For each included study, we identified the main determinants of perceived review helpfulness that we summarized in Table 1. Kirca et al. (2005) argue that meta-analysis could be conducted with at least three studies, hence we did not do meta-analysis on those determinants with less than three studies. We also removed two studies that reported correlation coefficients larger than the critical value of 1, as

PT

Comprehensive Meta-Analysis (CMA) 2.0 cannot process correlation coefficients with absolute value

RI

larger than 1. As a result, our final dataset for meta-analysis included forty-two studies. Field (2001)

SC

suggests that a meta-analysis should have at least fifteen studies. Our meta-analysis certainly satisfies

3.2. The Coding Procedure for Moderators

NU

that criterion.

MA

We extracted descriptive information such as author information, published year, paper title, paper source, review platform, studied product, and sample size from selected studies. The effect size metric

D

selected for meta-analysis is the correlation coefficient. For the moderating factors that we consider in

PT E

our meta-analysis, we manually coded each included study based on its review helpfulness measure, review platform, and product type. For the review helpfulness measure, we labelled each study as either

CE

“helpful vote number” if online review helpfulness is measured with the total number of helpful votes

AC

or “helpful vote ratio” if it was measured with the ratio of the number of helpful votes to the total number of votes. For online review platform, studies were categorized into two categories: “internal platform” and “external platform”. A study was coded as “internal platform” if reviews are obtained from retailer-hosted platforms (e.g., Amazon). Conversely, if online reviews are collected from third-party infomediaries (e.g., Yelp and TripAdvisor). For the product type, we coded each study as “experience product” or “search product” based on the specific product type in its research context. 3.3. Statistical Analysis 16

ACCEPTED MANUSCRIPT To conduct the meta-analysis, we first extracted effect sizes from the included studies. The effect size is defined as “the degree to which the phenomenon is present in the population or the degree to which the null hypothesis is false” (Cohen 1988). The larger value of the effect size is, the greater degree to which the subject phenomenon is manifested. In this paper we adopted the correlation coefficient r as the

PT

effect and obtained the reported coefficients in each included study. The meta-analysis consists of three

RI

steps:

be calculated using Equation 1 (Lipsey et al. 2001).

(1)

NU

Fisher ' s Z i =0.5*log((1+ r i )/(1- r i ))

SC

Step 1: Calculate the Fisher’s Z and combine effect sizes (i.e., correlation coefficients). Fisher’s Z can

MA

where r i is the correlation coefficient extracted from study i.

The weighted-average Fisher’s Z was calculated using Equation 2. n

(2)

D

Fisher ' sZ   wi  Fisher ' s Z i

PT E

i 1

where wi is the weight of study i, which equals to the ratio of sample size of study i to the overall

CE

sample size of all the studies considered in the meta-analysis. The weighted-average Fisher’s Z was converted to a combined effect size r using Equation 3. (3)

AC

r  (e2 Fisher ' sZ i  1)(e2 Fisher ' sZ i  1)

Step 2: Test the significance of the combined effect size. The p-value associated with the combined effect size indicates its statistical significance. Step 3: Conduct subgroup analyses. The Q-statistic measured by the weighted variance of the effect size statistics (Martin 2008) is used to test homogeneity between the studies that belong to different subgroups. It describes the percentage of the variability in the effect estimate that is due to

17

ACCEPTED MANUSCRIPT heterogeneity rather than sampling error or chance.

4. Meta-Analysis Results 4.1. Calculation of Effect Sizes

PT

We used the Comprehensive Meta-Analysis (CMA) 2.0 software to conduct our statistics analysis. CMA 2.0 generates either a fixed-effect model or a random-effect model. Based on the result of

RI

Q-statistics which rejects the homogeneity assumption across studies (Martin 2008), we adopted the

NU

SC

random-effect model for our meta-analysis. The meta-analysis results are summarized in Table 3.

AC

CE

PT E

D

MA

Table 3. Meta-analytic effect sizes of online review factors on helpfulness Factor Factor Significant study Sample Combined P-value Q-value category number size effect size (Homogeneity test) Review Review depth 29 588031 0.114 0.000*** 14135.975*** related Review 15 238169 0.014 0.298 516.646*** factors readability Linear review 18 445657 0.054 0.279 17600.581*** rating Quadratic 6 40322 0.055 0.409 935.672*** review rating Review age 7 186345 0.094 0.001*** 728.753*** ** Reviewer Reviewer 9 304044 0.250 0.032 24175.430*** related information factors disclosure Reviewer expert 6 270649 0.201 0.001*** 4523.685*** label Notes: Kirca et al. (2005) argue that meta-analysis could be conducted with at least three studies, hence we did not do meta-analysis on factors with significant study number less than three. ** : p<0.05, ***: p<0.01

The meta-analysis results show that two review related determinants, namely, review depth measured by the number of words in a review and review age measured by the number of days that a review exists, and two reviewer related factors, namely, reviewer information disclosure and reviewer expert label, have significantly positive influence on perceived review helpfulness. Our results also show that there is no significant statistical relationship between the other two review related determinants, namely 18

ACCEPTED MANUSCRIPT review readability and review rating (both linear review rating and quadratic review rating), and review helpfulness. According to the results of the homogeneity test, the Q-values are significant for all variables, rejecting the homogeneity assumption across studies. This indicates that the variability in effect sizes

PT

exceeds what would be expected from sampling error (Purnawirawan et al. 2015). In order to further

RI

explain the variability in our findings, we conducted subgroup analyses using moderator variables (i.e.,

SC

review helpfulness measurement, review platform, and product type) and report our findings in the following section.

NU

4.2. Subgroup Analysis

MA

As we discussed earlier, discrepancy in the review helpfulness measure, review platform, and product type may cause heterogeneity in the findings on the determinants of perceived review helpfulness.

D

Table 4 summarizes the findings of our subgroup analyses that reveal the moderating effect of the three

PT E

factors between review helpfulness determinants and perceived review helpfulness.

AC

CE

Table 4. The moderating effects of helpfulness measurement, online review platform, and product type Moderator Factor Significant Sample Combined P-value Q-value of two groups study number size effect size (Homogeneity test) Review helpfulness measure Helpful vote Review 18 398058 0.113 0.000*** 210.437*** ratio depth Helpful vote 11 189973 0.119 0.000*** number Helpful vote Review 7 119537 -0.001 0.829 9.275*** ratio readability Helpful vote 8 118632 0.026 0.308 number Helpful vote Linear 9 222334 0.081 0.197 1538.241*** ratio review Helpful vote rating 9 223323 0.027 0.190 number Helpful vote Quadratic 3 27417 0.017 0.893 11.984*** ratio review Helpful vote rating 3 12905 0.093 0.014** number 19

ACCEPTED MANUSCRIPT

CE

PT E

D

MA

NU

SC

RI

PT

Helpful vote Review age 4 113783 0.111 0.021** 8.360*** ratio Helpful vote 3 72562 0.071 0.058* number Helpful vote Reviewer 5 155421 0.084 0.350 14104.75*** ratio information Helpful vote disclosure 4 148623 0.456 0.007*** number Helpful vote Reviewer 3 115681 0.258 0.070** 389.057* ratio expert label Helpful vote 3 154968 0.143 0.000*** number Review platform Internal Review 18 395644 0.100 0.001*** 340.214*** * depth External 11 175566 0.137 0.078 Internal Linear 11 240524 0.129 0.082* 5219.393*** ** Review External 7 205133 -0.063 0.033 rating Internal Quadratic 3 27417 0.017 0.893 11.984*** ** review External 3 12905 0.093 0.014 rating Internal Review age 3 112097 0.110 0.051* 5.892** ** External 4 74248 0.081 0.013 Product type Search Review 8 51513 0.068 0.074* 267.279*** *** depth Experience 18 359304 0.137 0.003 Search Review 3 8332 0.026 0.308 2.71 readability Experience 12 229837 0.016 0.268 Search Linear 4 27666 0.161 0.453 12234.233*** *** review Experience 12 341528 0.162 0.000 rating Notes: Kirca et al. (2005) argue that meta-analysis could be conducted with at least three studies, hence we did not do meta-analysis on factors with significant study number less than three. * : p<0.1, **: p<0.05, ***: p<0.01

AC

For the review helpfulness measure, the Q-statistic values calculated for all review helpfulness determinants are statistically significant. The findings, which are in support of our hypothesis H1, show that the discrepancy in the review helpfulness measure used in the existing studies moderates the effects of review helpfulness determinants on perceived review helpfulness. It is one of the reasons why inconsistent findings were found in existing studies. The effects of review depth, review age, and reviewer expert label on review helpfulness are significant regardless of which review helpfulness measurement is used. They are more reliable determinants than others. However, review age and 20

ACCEPTED MANUSCRIPT reviewer expert label have stronger effects on perceived review helpfulness when the helpfulness vote ratio is used as the review helpfulness measure. The finding is consistent with that of Ahmad et al. (2015). Review depth, on the other hand, has a stronger effect on perceived review helpfulness when the total number of helpful votes is used as the review helpfulness measure. Review readability and linear

PT

review rating remain to be insignificant determinants for review helpfulness regardless of which review

RI

helpfulness measure is used. Quadratic review rating and reviewer information disclosure only have a

SC

significant influence on perceived review helpfulness when the number of helpful votes is used as the review helpfulness measure. After comparing the different effects of predictor variables on the helpful

NU

vote number and the ratio of helpful vote number to total vote number, Ahmad et al. (2015) argue that

MA

the ratio of helpful vote number to total vote number is a better measurement for review helpfulness because of the noise in the data with helpful vote number as the dependent variable. Using helpful vote

D

number as review helpfulness lacks an important information of total vote number, and helpfulness of

PT E

reviews with the same helpful number but different total vote number is different (Ahmad et al. 2015). For the online review platform, we did not conduct subgroup analysis on review readability, reviewer

CE

information disclosure, and reviewer expert label due to insufficient number of studies. The

AC

homogeneity test result shows that online review platform significantly moderates the relationships between review depth, linear and quadratic review rating, and review age on review helpfulness, thus supporting H2. When online reviews are obtained from an internal review platform, review age has a stronger effect on perceived review helpfulness. When reviews are obtained externally, review depth has a stronger effect on perceived review helpfulness while quadratic review rating becomes a significant determinant of perceived review helpfulness. Moreover, linear review rating has a positive influence on review helpfulness for internally managed reviews while the influence becomes negative 21

ACCEPTED MANUSCRIPT for externally managed reviews. For external reviews, review rating has a “U” shape relationship with review helpfulness, indicating that online consumers perceive extreme reviews (positive or negative) as more helpful than moderate reviews. It in line with Racherla et al. (2012) and Park et al. (2015), who obtain review data from internal managed platform Yelp. For internal reviews, review rating has

PT

positive influence on review helpfulness, implying that consumers are more likely to find positively

RI

rated reviews helpful for internally managed reviews. In order to reduce risk of loss more than

SC

enhancing the gain (Kahneman et al. 1979), rational consumers will take negative reviews more seriously and discounting the positive reviews Hu et al. (2007). Hence, external reviews from

NU

third-party platforms have more influence on consumers’ decision making than internal reviews from

MA

retailer-hosted platforms (Gu et al. 2012).

For the moderating effect of product type, we could not conduct subgroup analyses for review age,

D

linear review rating, reviewer information disclosure, and reviewer expert label due to insufficient data.

PT E

Our Q-statistic results show that product type significantly moderates the effects of review depth and linear review rating (with the exception of review readability) on perceived review helpfulness,

CE

partially supporting H3. More specially, review depth and linear review rating exert relatively larger

AC

positive influence on review helpfulness for experience products than for search products. Since review rating can be regarded as a proxy variable for review valence (Dellarocas et al. 2004), our results about linear review rating is consistent with the meta-analysis results of Purnawirawan et al. (2015), which show that online reviews valence exerts more influence on review helpfulness of experience products. Consumers have to face higher level of uncertainty while shopping experience products online than search products, because it is impossible to experience an experience product before buying it (Park et al. 2013; Zhu et al. 2010). In order to reduce uncertainty and risks of buying experience products online, 22

ACCEPTED MANUSCRIPT consumers have stronger motivation to seek not only firm-generated content but also online reviews (Park et al. 2009). While buying search product online, firm-generated content is enough to evaluate product quality (Baek et al. 2012), hence, consumers pay relatively less attention to online reviews. As a consequence, the influence of determinants (e.g., review depth and linear review rating) on review

PT

helpfulness are stronger for experience products than search products.

RI

Based on our subgroup analysis results, we can conclude that review helpfulness measurement

SC

significantly moderates the effects of review depth, review readability, review rating, review age, reviewer information disclosure, and reviewer expert label on review helpfulness; online review

NU

platform moderates the effects of review depth, review rating, and review age on review helpfulness;

MA

product type moderates the effects of review depth and linear review rating on review helpfulness. Our findings can help review helpfulness researchers identify appropriate determinants for review

D

helpfulness in each specific research context. Furthermore, our findings raise additional research

PT E

questions on why review helpfulness measure, review platform, and product type moderate the determinants of review helpfulness.

CE

4.3. Result Summary

AC

We summarize the results of meta-analysis and subgroup analysis in Table 5. In order to test our expectations and hypotheses, we also include them in Table 5.

Factor Review depth

Review readability

Expected relationship Positive

Unclear

Table 5. Results summary Meta-analysis Subgroup analysis result Moderator Subgroup *** 0.114 Helpfulness measure Helpful vote ratio (Q-value: 210.437***) Helpful vote number Review platform Internal review *** (Q-value: 340.214 ) External review Product type Search product *** (Q-value: 267.279 ) Experience product 0.014 Helpfulness measure Helpful vote ratio (Q-value: 9.275*** ) Helpful vote number 23

Result 0.113*** 0.119*** 0.100*** 0.137* 0.001*** 0.078* -0.001 0.026

ACCEPTED MANUSCRIPT

Positive

0.055

Review age

Positive

0.094***

Reviewer information disclosure Reviewer expert label

Positive

0.250**

0.201***

0.456***

Helpfulness measure (Q-value: 389.057*)

Helpful vote ratio Helpful vote number

0.258** 0.143***

MA

Positive

Helpful vote number

PT

Quadratic review rating

0.026 0.016 0.081 0.027 0.129** -0.063** 0.161 0.162*** 0.017 0.093** 0.017 0.093** 0.111** 0.071* 0.110* 0.081** 0.084

RI

0.054

Search product Experience product Helpful vote ratio Helpful vote number Internal review External review Search product Experience product Helpful vote ratio Helpful vote number Internal review External review Helpful vote ratio Helpful vote number Internal review External review Helpful vote ratio

SC

Positive

NU

Linear review rating

Product type (Q-value: 2.71) Helpfulness measure (Q-value: 1538.241***) Review platform (Q-value: 5219.393***) Product type (Q-value: 12234.233***) Helpfulness measure (Q-value: 11.984***) Review platform (Q-value: 11.984***) Helpfulness measure (Q-value: 8.360***) Review platform (Q-value: 5.892**) Helpfulness measure (Q-value: 14104.75***)

As we can see from Table 5, the expected positive influence of review depth, review age, reviewer

D

information disclosure, and reviewer expert label is supported by meta-analysis results. There is no

PT E

significant statistical relationship between the other two review related determinants, namely review readability and review rating (both linear review rating and quadratic review rating), and review

CE

helpfulness.

According to the Q-value of heterogeneity test between subgroups, all the hypotheses on moderators

AC

are supported with the exception of the moderating effect of product type between review readability and review helpfulness. The positive influence of review depth, review age, and reviewer expert label on review helpfulness is robust with no direction change for all moderator subgroups. Review readability has no significant impact on review helpfulness (measured by helpful vote number or helpful vote ratio) of both experience and search product. The influence of reviewer information disclosure on review helpfulness is positive when review helpfulness is measured by

24

ACCEPTED MANUSCRIPT helpful vote number, which is consistent with our expectation. In contrast, reviewer information disclosure has no significant influence on review helpfulness measured by helpful vote ratio. Review rating has no significant influence on review helpfulness measured by helpful vote ratio, but has positive curve relationship with review helpfulness measured by helpful vote number. Review rating

PT

has a “U” shape relationship with review helpfulness for external reviews and has positive influence on

RI

internal review helpfulness. Review rating has no significant influence on search product review

SC

helpfulness but has positive influence on experience product review helpfulness. Our meta-analysis results reconcile the conflicting results of existing studies and confirm the

NU

moderating effect of review helpfulness measure, product type, and review platform.

MA

5. Discussions and Future Work 5.1. Conclusions

D

We reviewed extant research about the determinant factors of perceived online review helpfulness. Two

PT E

types of factors were found to have influence on perceived helpfulness, namely review related and reviewer related factors. All review related factors (i.e., review depth, review readability, linear review

CE

rating, quadratic review rating, review age) and two out of five reviewer related factors (i.e., reviewer

helpfulness.

AC

information disclosure and reviewer expertise) are found to have inconsistent influences on review

A meta-analysis was conducted on review helpfulness determinants to reconcile the contradictory findings on the influence of online review related factors on perceived review helpfulness. The results of meta-analysis show that review depth, review age, and reviewer related factors measured by whether a reviewer disclose his/her own information or has an expert label have significantly positive influence on review helpfulness. Review readability and review rating do not show significant influence on 25

ACCEPTED MANUSCRIPT review helpfulness. We also performed subgroup analysis to compare the differences between groups. The results of subgroup analysis show that the impacts of determinant factors on the online review helpfulness are moderated by helpfulness measurement, online review platform, and product type. Helpfulness measurement, online review platform, and product type are found to be the three factors

PT

responsible for most of the mixed findings. More specifically, helpfulness measurement significantly

RI

moderates the effects of review depth, review readability, review rating, review age, reviewer

SC

information disclosure, and reviewer expert label on review helpfulness; online review platform moderates the effects of review depth, review rating, and review age on review helpfulness; product

NU

type of online reviews moderates the effects of review depth and review rating on review helpfulness.

MA

5.2. Research Implications

Our study has both theoretical and practical implications. From a theoretical perspective, our study

D

enriches the literature on online review helpfulness. A great deal of research has been carried out to

PT E

understand the helpfulness of online reviews. However, there are mixed findings with regards to the determinants of perceived review helpfulness and how they influence helpfulness. Our study integrates

CE

existing research, reconciles their findings, and clarifies the reasons behind the inconsistencies in

AC

existing studies. Our study summarizes what has been done about online review helpfulness determinants and provide directions for future studies. From a practical perspective, our findings help both sellers and consumers to better identify helpful online reviews among an enormous amount of reviews and improve the efficiency of their decision making. Long reviews with high rating are more likely perceived helpful for experience products. Consumers are more likely to find positively rated reviews helpful for internally managed reviews and perceive extreme reviews (positive or negative) as more helpful for external reviews. Our findings will 26

ACCEPTED MANUSCRIPT also benefit platform managers to design better review systems. The review helpful vote number and review helpful ratio two measurements can be used to measure review helpfulness, platform managers should choose appropriate review helpfulness assessment system based on both product and platform characteristics.

PT

5.3. Limitations and Future Work

RI

This study has several limitations. First, our meta-analysis study only focuses on review meta-data,

SC

reviewer-related features, and review readability, without considering more advanced features that are made possible by the development of text mining techniques. We also surveyed elite conferences in CS

NU

domain such as EMNLP, WWW, ICDM, SIGCHI, RANLP, RecSys, AAAI, and CIKM to obtain studies

MA

related to review helpfulness. Table 6 summarizes the main determinants of review helpfulness investigated in CS literature. In addition to review meta-data, reviewer-related characteristics, and

D

review readability, linguistic review features such as syntactic, semantic feature, and lexical features are

PT E

also important features in review helpfulness prediction or classification tasks. Linguistic features are seldom studied in empirical studies related to review helpfulness in the information systems or

CE

marketing domain. When there are enough empirical studies that use linguistic features as the

AC

determinants of review helpfulness, we can update our meta-analysis to include those features. Table 6. Summary of the determinants of review helpfulness investigated in CS literature Determinant Definition Related work Review meta-data Capture information independent Cao et al. (2011), Krishnamoorthy (2015), Lee et al. of the review text (Kim et al. (2014), Lu et al. (2010), Liu et al. (2008), Otterbacher 2006), e.g., review rating, review (2009), Kim et al. (2006), Weimer et al. (2007) extremity, review age. Reviewer-related Features related to reviewers, Tang et al. (2013), Ngo-Ye et al. (2014), O'Mahony et characteristics e.g., reviewer’s reputation, social al. (2009), Lu et al. (2010), Liu et al. (2008), Lee et al. network. (2014), Otterbacher (2009) Review The easy-to-read degree of a Martin et al. (2014), Liu et al. (2007), Otterbacher readability review. (2009), Krishnamoorthy (2015), Liu et al. (2008) Syntactic feature Features that take into account Kim et al. (2006), Weimer et al. (2007), Martin et al. the Part-Of-Speech (POS) (2014), Lu et al. (2010) 27

ACCEPTED MANUSCRIPT

Semantic feature

Lexical feature

tagging of the words in the text. Related to the substance of the review, i.e., the meaning of the words in the review. Capture the words observed in the reviews, e.g., unigram, bigram, spelling error frequency.

Kim et al. (2006), Cao et al. (2011), Zhang et al. (2006), O'Mahony et al. (2009) Kim et al. (2006), Weimer et al. (2007), Cao et al. (2011), Krishnamoorthy (2015), Lee et al. (2014), Zhang et al. (2006), O'Mahony et al. (2009), Lu et al. (2010), Liu et al. (2007), Otterbacher (2009)

PT

Second, we reviewed forty-seven published articles related to online review helpfulness, among

RI

which forty-two were included in the meta-analysis. Although we have tried our best to find all relevant

SC

articles in top information systems journals, it is likely that we might have missed important studies that should have been included in our meta-analysis. Second, we mainly examine the direct effects of

NU

determinant factors on perceived helpfulness and the moderating effects of helpfulness measurement,

MA

product type, and online review platform. There could be other variables, such as data source (first/second hand data) and regression method, that the studies included in our meta-analysis did not

D

consider. Thus, they were not included in our meta-analysis. Third, the weakness of meta-analysis

PT E

method of losing contextual information cannot be ignored. The meta-analysis result cannot reveal all the differences in the research contexts of the studies considered. Therefore, more detailed analysis is

AC

CE

needed to explain mixed findings in extant research.

Acknowledgement

The authors would like to thank the editors and reviewers for their helpful and constructive suggestions. This research was supported by the Natural Science Foundation of China (Grant# 71572122, Grant# 71671153, Grant# 71671154 and Grant# 71531013) and China Scholarship Council (Grant# 201506310121).

28

ACCEPTED MANUSCRIPT References：

AC

CE

PT E

D

MA

NU

SC

RI

PT

Ahmad, S. N., and Laroche, M. 2015. "How do expressed emotions affect the helpfulness of a product review? Evidence from reviews using latent semantic analysis," International Journal of Electronic Commerce (20:1), pp. 76-111. Babić Rosario, A., Sotgiu, F., De Valck, K., and Bijmolt, T. H. A. 2016. "The effect of electronic word of mouth on sales: A meta-analytic review of platform, product, and metric factors," Journal of Marketing Research (53:3), pp. 297-318. Baek, H., Ahn, J., and Choi, Y. 2012. "Helpfulness of online consumer reviews: Readers' objectives and review cues," International Journal of Electronic Commerce (17:2), pp. 99-126. Bao, Z., and Chau, M. 2016. "The effect of collective rating on the perception of online reviews," in Proceedings of the 20th Pacific Asia Conference on Information Systems: Chiayi. Bickart, B., and Schindler, R. M. 2001. "Internet forums as influential sources of consumer information," Journal of Interactive Marketing (15:3), pp. 31-40. Cao, Q., Duan, W., and Gan, Q. 2011. "Exploring determinants of voting for the “helpfulness” of online user reviews: A text mining approach," Decision Support Systems (50:2), pp. 511-521. Chen, C. C., and Tseng, Y.-D. 2011. "Quality evaluation of product reviews using an information quality framework," Decision Support Systems (50:4), pp. 755-768. Chen, P.-Y., Wu, S.-Y., and Yoon, J. 2004. "The impact of online recommendations and consumer feedback on sales," in Proceedings of the 25th International Conference on Information Systems: Washington, DC. Chen, Y., and Xie, J. 2008. "Online consumer review: Word-of-mouth as a new element of marketing communication mix," Management Science (54:3), pp. 477-491. Chen, Z., and Xu, F. 2014. "Study on impact factors of voting for helpfulness of online reviews in E-commerce," Journal of Modern Information (34:1), pp. 18-22. Cheng, Y.-H., and Ho, H.-Y. 2015. "Social influence's impact on reader perceptions of online reviews," Journal of Business Research (68:4), pp. 883-887. Cheung, C. M.-Y., Sia, C.-L., and Kuan, K. K. Y. 2012. "Is this review believable? A study of factors affecting the credibility of online consumer reviews from an ELM perspective," Journal of the Association for Information Systems (13:8), pp. 618-635. Chua, A. Y. K., and Banerjee, S. 2015. "Understanding review helpfulness as a function of reviewer reputation, review rating, and review depth," Journal of the Association for Information Science and Technology (66:2), pp. 354-362. Cohen, J. 1988. Statistical power analysis for the behavioral sciences (second edition), New Jercy: Published Hillsdale. Connors, L., Mudambi, S. M., and Schuff, D. Year. "Is it the review or the reviewer? A multi-method approach to determine the antecedents of online review helpfulness," Proceedings of the 44th Hawaii International Conference on System Sciences, Hawaii, 2011. de Matos, C. A., and Rossi, C. A. V. 2008. "Word-of-mouth communications in marketing: a meta-analytic review of the antecedents and moderators," Journal of the Academy of Marketing Science (36:4), pp. 578-596. Dellarocas, C., Awad, N. F., and Zhang, X. 2004. "Exploring the value of online reviews to organizations: Implications for revenue forecasting and planning," in Proceedings of the 25th International Conference on Information Systems: Washington, DC. 29

ACCEPTED MANUSCRIPT

AC

CE

PT E

D

MA

NU

SC

RI

PT

Dellarocas, C., Zhang, X., and Awad, N. F. 2007. "Exploring the value of online product reviews in forecasting sales: The case of motion pictures," Journal of Interactive Marketing (21:4), pp. 23-45. Duan, W., Gu, B., and Whinston, A. B. 2008. "Do online reviews matter? — An empirical investigation of panel data," Decision Support Systems (45:4), pp. 1007-1016. Einar, B., Havro, L. J., and Moen, O. 2015. "An empirical investigation of self-selection bias and factors influencing review helpfulness," International Journal of Business and Management (10:7), pp. 16-30. Fang, B., Ye, Q., Kucukusta, D., and Law, R. 2016. "Analysis of the perceived value of online tourism reviews: Influence of readability and reviewer characteristics," Tourism Management (52), pp. 498-506. Field, A. P. 2001. "Meta-analysis of correlation coefficients: A Monte Carlo comparison of fixed- and random-effects methods," Psychological Methods (6:2), pp. 161-180. Floyd, K., Freling, R., Alhoqail, S., Cho, H. Y., and Freling, T. 2014. "How online product reviews affect retail sales: A meta-analysis," Journal of Retailing (90:2), pp. 217-232. Forman, C., Ghose, A., and Wiesenfeld, B. 2008. "Examining the relationship between reviews and sales: The role of reviewer identity disclosure in electronic markets," Information Systems Research (19:3), pp. 291-313. Ghose, A., and Ipeirotis, P. G. Year. "Designing novel review ranking systems: Predicting usefulness and impact of reviews," Proceedings of the 9th International Conference on Electronic Commerce, Minnesota, 2007. Ghose, A., and Ipeirotis, P. G. 2011. "Estimating the helpfulness and economic impact of product reviews: Mining text and reviewer characteristics," IEEE Transactions on Knowledge and Data Engineering (23:10), pp. 1498-1512. Girard, T., and Dion, P. 2010. "Validating the search, experience, and credence product classification framework," Journal of Business Research (63:9-10), pp. 1079-1087. Glass, G. V. 1976. "Primary, secondary, and meta-analysis of research," Educational Researcher (5:10), pp. 3-8. Godes, D., and Mayzlin, D. 2004. "Using online conversations to study word-of-mouth communication," Marketing Science (23:4), pp. 545-560. Gu, B., Park, J., and Konana, P. 2012. "Research note—The impact of external word-of-mouth sources on retailer sales of high-involvement products," Information Systems Research (23:1), pp. 182-196. Guo, B., and Zhou, S. 2016. "What makes population perception of review helpfulness: An information processing perspective," Electronic Commerce Research. Hameed, M. A., Counsell, S., and Swift, S. 2012. "A meta-analysis of relationships between organizational characteristics and IT innovation adoption in organizations," Information & Management (49:5), pp. 218-232. Hao, Y., Ye, Q., and Li, Y. 2010. "Research on online impact factors of customer reviews usefulness based on movie reviews data," Journal of Management Sciences in China (13:8), pp. 78-88,96. Hjørland, B. 2001. "Why is meta analysis neglected by information scientists?," Journal of the American Society for Information Science and Technology (52:13), pp. 1193-1194. Hlee, S., Lee, J., Yang, S.-B., and Koo, C. 2016. "An empirical examination of online restaurant reviews (Yelp.com): Moderating roles of restaurant type and self-image disclosure," in Information and 30

ACCEPTED MANUSCRIPT

AC

CE

PT E

D

MA

NU

SC

RI

PT

Communication Technologies in Tourism, A. Inversini and R. Schegg (eds.), Springer International Publishing, pp. 339-353. Hong, Y., Liao, H., Hu, J., and Jiang, K. 2013. "Missing link in the service profit chain: A meta-analytic review of the antecedents, consequences, and moderators of service climate," Journal of Applied Psychology (98:2), pp. 237-267. Hu, N., Pavlou, P. A., and Zhang, J. 2007. "Why do online product reviews have a j-shaped distribution? Overcoming biases in online word-of-mouth communication," unpublised paper. Huang, A. H., Chen, K., Yen, D. C., and Tran, T. P. 2015. "A study of factors that contribute to online review helpfulness," Computers in Human Behavior (48), pp. 17-27. Hunter, J. E., Schmidt, F. L., and Jackson, G. B. 1982. Meta-analysis: Cumulating research findings across studies, Sage Publications. Hwang, M. I. 1996. "The use of meta-analysis in MIS research: Promises and problems," Data Base for Advances in Information Systems (27:3), pp. 35-48. Kahneman, D., and Tversky, A. 1979. "Prospect theory: An analysis of decision under risk," Econometrica (47:2), pp. 263-292. Kang, Y., and Zhou, L. 2016. "Longer is better? A case study of product review helpfulness prediction," in Proceedings of the 22nd Americas Conference on Information Systems: San Diego. Kim, S.-M., Pantel, P., Chklovski, T., and Pennacchiotti, M. 2006. "Automatically assessing review helpfulness," in Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing, Association for Computational Linguistics: Sydney, pp. 423-430. King, W. R., and He, J. 2005. "Understanding the role and methods of meta-analysis in IS research," Communications of the Association for Information Systems (16:1), pp. 665-686. Kirca, A. H., Jayachandran, S., and Bearden, W. O. 2005. "Marketing orientation: A meta-analytic review and assessment of its antecedents and impact on performance," Journal of Marketing (69), pp. 24-41. Korfiatis, N., García-Bariocanal, E., and Sánchez-Alonso, S. 2012. "Evaluating content quality and helpfulness of online product reviews: The interplay of review helpfulness vs. review content," Electronic Commerce Research and Applications (11:3), pp. 205-217. Krishnamoorthy, S. 2015. "Linguistic features for review helpfulness prediction," Expert Systems with Applications (42:7), pp. 3751-3759. Kuan, K. K. Y., Hui, K.-L., Prasarnphanich, P., and Lai, H.-Y. 2015. "What makes a review voted? An empirical investigation of review voting in online review systems," Journal of the Association for Information Systems (16:1), pp. 48-71. Kwok, L., and Xie, K. L. 2016. "Factors contributing to the helpfulness of online hotel reviews," International Journal of Contemporary Hospitality Management (28:10), pp. 2156-2177. Lee, S., and Choeh, J. Y. 2014. "Predicting the helpfulness of online reviews using multilayer perceptron neural networks," Expert Systems with Applications (41:6), pp. 3041-3046. Lee, S., and Choeh, J. Y. 2016. "The determinants of helpfulness of online reviews," Behaviour & Information Technology (35:10), pp. 853-863. Li, H., Zhang, Z., Janakiraman, R., and Meng, F. 2016. "How review sentiment and readability affect online peer evaluation votes? –An examination combining reviewer’s social identity and social network," in Proceedings of the 2016 TTRA Annual International Conference: Colorado. Li, Q., Cui, J., and Gao, Y. 2015. "The influence of social capital in an online community on online review quality in China," in 48th Hawaii International Conference on System Sciences, pp. 31

ACCEPTED MANUSCRIPT

AC

CE

PT E

D

MA

NU

SC

RI

PT

562-570. Liao, C., Cai, C., and Li, Y. 2013. "An empirical study on influence factors of online reviews' helpfulness in E-commerce," Soft Science (27:5), pp. 46-50. Lipsey, M. W., and Wilson, D. B. 2001. Practical meta-analysis, California: PublishedSage Publication. Liu, J., Cao, Y., Lin, C.-Y., Huang, Y., and Zhou, M. 2007. "Low-quality product review detection in opinion summarization," in Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning, Association for Computational Linguistics: Prague, pp. 334-342. Liu, Y. 2006. "Word of mouth for movies: Its dynamics and impact on box office revenue," Journal of Marketing (70:3), pp. 74-89. Liu, Y., Huang, X., An, A., and Yu, X. 2008. "Modeling and predicting the helpfulness of online reviews," in Eighth IEEE International Conference on Data Mining, IEEE: Pisa, pp. 443-452. Liu, Z., and Park, S. 2015. "What makes a useful online review? Implication for travel product websites," Tourism Management (47), pp. 140-151. Lu, Y., Tsaparas, P., Ntoulas, A., and Polanyi, L. 2010. "Exploiting social context for review quality prediction," in Proceedings of the International World Wide Web Conference, ACM: Raleigh, pp. 691-700. Luo, J., Ba, S., and Zhang, H. 2012. "The effectiveness of online shopping characteristics and well-designed websites on satisfaction," Mis Quarterly (36:4), pp. 1131-1144. Martin, C. M. 2008. "A meta-analytic investigation of the relationship between emotional intelligence and leadership effectiveness," in Department of Educational Leadership, East Carolina University, pp. Martin, L., and Pu, P. 2014. "Prediction of helpful reviews using emotions extraction," in Proceedings of the Twenty-Eighth AAAI Conference on Artificial Intelligence, Association for the Advancement of Artificial Intelligence: Québec City, pp. 1551-1557. Mudambi, S. M., and Schuff, D. 2010. "What makes a helpful online review? A study of customer reviews on Amazon.com," Mis Quarterly (34:1), pp. 185-200. Nelson, P. 1970. "Information and consumer behavior," Journal of Political Economy (78:2), pp. 311-329. Ngo-Ye, T. L., and Sinha, A. P. 2014. "The influence of reviewer engagement characteristics on online review helpfulness: A text regression model," Decision Support Systems (61), pp. 47-58. O'Mahony, M. P., and Smyth, B. 2009. "Learning to recommend helpful hotel reviews," in Proceedings of the Third ACM Conference on Recommender Systems, ACM: New York, pp. 305-308. Ortiz de Guinea, A., Webster, J., and Staples, D. S. 2012. "A meta-analysis of the consequences of virtualness on team functioning," Information & Management (49:6), pp. 301-308. Otterbacher, J. 2009. ""Helpfulness" in online communities: A measure of message quality," in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, ACM: Boston, pp. 955-964. Pan, Y., and Zhang, J. Q. 2011. "Born unequal: A study of the helpfulness of user-generated product reviews," Journal of Retailing (87:4), pp. 598-612. Park, C., and Lee, T. M. 2009. "Information direction, website reputation and eWOM effect: A moderating role of product type," Journal of Business Research (62:1), pp. 61-67. Park, J., Gu, B., and Lee, H. 2012. "The relationship between retailer-hosted and third-party hosted 32

ACCEPTED MANUSCRIPT

AC

CE

PT E

D

MA

NU

SC

RI

PT

WOM sources and their influence on retailer sales," Electronic Commerce Research and Applications (11:3), pp. 253-261. Park, S.-B., and Park, D.-H. 2013. "The effect of low- versus high-variance in product reviews on product evaluation," Psychology & Marketing (30:7), pp. 543-554. Park, S., and Nicolau, J. L. 2015. "Asymmetric effects of online consumer reviews," Annals of Tourism Research (50), pp. 67-83. Podsakoff, P. M., MacKenzie, S. B., Lee, J. Y., and Podsakoff, N. P. 2003. "Common method biases in behavioral research: A critical review of the literature and recommended remedies," Journal of Applied Psychology (88:5), pp. 879-903. Purnawirawan, N., Eisend, M., De Pelsmacker, P., and Dens, N. 2015. "A meta-analytic investigation of the role of valence in online reviews," Journal of Interactive Marketing (31), pp. 17-27. Quaschning, S., Pandelaere, M., and Vermeir, I. 2015. "When consistency matters: The effect of valence consistency on review helpfulness," Journal of Computer-Mediated Communication (20:2), pp. 136-152. Racherla, P., and Friske, W. 2012. "Perceived ‘usefulness’ of online consumer reviews: An exploratory investigation across three services categories," Electronic Commerce Research and Applications (11:6), pp. 548-559. Salehan, M., and Kim, D. J. 2016. "Predicting the performance of online consumer reviews: A sentiment mining approach to big data analytics," Decision Support Systems (81), pp. 30-40. Schlosser, A. E. 2011. "Can including pros and cons increase the helpfulness and persuasiveness of online reviews? The interactive effects of ratings and arguments," Journal of Consumer Psychology (21:3), pp. 226-239. Schmidt, F. L., and Hunter, J. E. 2014. Methods of meta-analysis: Correcting error and bias in research findings (third edition), California: PublishedSage publications. Sen, S., and Lerman, D. 2007. "Why are you telling me this? An examination into negative consumer reviews on the Web," Journal of Interactive Marketing (21:4), pp. 76-94. Siering, M., and Muntermann, J. Year. "What drives the helpfulness of online product reviews? From stars to facts and emotions," 11th International Conference on Wirtschaftsinformatik, Leipzig, 2013, pp. 103-118. Tang, J., Gao, H., Hu, X., and Liu, H. 2013. "Context-aware review helpfulness rating prediction," in Proceedings of the 7th ACM Conference on Recommender Systems, ACM: Hong Kong, pp. 1-8. Ullah, R., Zeb, A., and Kim, W. 2015. "The impact of emotions on the helpfulness of movie reviews," Journal of Applied Research and Technology (13:3), pp. 359-363. Wang, C.-C., Li, M.-Z., and Yang, Y. Y. H. 2015a. "Perceived usefulness of word-of-mouth: An analysis of sentimentality in product reviews," in International Conference on Multidisciplinary Social Networks Research, Springer Berlin Heidelberg: Matsuyama, pp. 448-459. Wang, G. A., Jiao, J., Abrahams, A. S., Fan, W., and Zhang, Z. 2013. "ExpertRank: A topic-aware expert finding algorithm for online knowledge communities," Decision Support Systems (54:3), pp. 1442-1451. Wang, S., Cunningham, N. R., and Eastin, M. S. 2015b. "The impact of eWOM message characteristics on the perceived effectiveness of online consumer reviews," Journal of Interactive Advertising (15:2), pp. 151-159. Weimer, M., and Gurevych, I. 2007. "Predicting the perceived quality of web forum posts," in Proceedings of the Conference on Recent Advances in Natural Language Processing: Borovets, 33

ACCEPTED MANUSCRIPT

AC

CE

PT E

D

MA

NU

SC

RI

PT

pp. 643-648. Willemsen, L. M., Neijens, P. C., Bronner, F., and De Ridder, J. A. 2011. "“Highly recommended!” The content characteristics and perceived usefulness of online consumer reviews," Journal of Computer-Mediated Communication (17:1), pp. 19-38. Wu, P. F. 2013. "In search of negativity bias: An empirical study of perceived helpfulness of online reviews," Psychology & Marketing (30:11), pp. 971-984. Wu, P. F., Heijden, H. V. D., and Korfiatis, N. T. Year. "The influences of negativity and review quality on the helpfulness of online reviews," Proceedings of the 32nd International Conference on Information Sysyems, Shanghai, 2011. Xiang, Z., Du, Q., Ma, Y., and Fan, W. 2017. "A comparative analysis of major online review platforms: Implications for social media analytics in hospitality and tourism," Tourism Management (58), pp. 51-65. Yan, J., Zhang, L., and Zhang, L. 2012. "An empirical study of the impact of review content on online eviews helpfulness in E-commerce," Information Science (30:5), pp. 713-716,719. Yan, Q., and Meng, Y. 2013. "Factors affecting the perceived usefulness of online reviews: An empirical study based on online film reviews," Chinese Journal of Management Science (21), pp. 126-131. Yin, D., Mitra, S., and Zhang, H. 2016. "Research note—When do consumers value positive vs. negative reviews? An empirical investigation of confirmation bias in online word of mouth," Information Systems Research (27:1), pp. 131-144. Yin, G. 2012. "Which kind of online reviews do consumers consider more useful: Based on social factor perspective " Management World:(12), pp. 115-124. Yin, G., Liu, W., and Zhu, S. 2012. "What makes a helpful online review? The perspective of information adoption and social network," Library and Information Service (56:16), pp. 140-147. Yin, G., Wei, L., Xu, W., and Chen, M. Year. "Exploring heuristic cues for consumer perceptions of online reviews helpfulness: The case of Yelp.com," Proceedings of the 18th Pacific Asia Conference on Information Systems, Chengdu, 2014a, pp. 52-63. Yin, G., Zhang, Q., and Li, Y. Year. "Effects of emotional valence and arousal on consumer perceptions of online review helpfulness," Proceedings of the 20th Americas Conference on Information Systems, Savannah, 2014b. You, Y., Vadakkepatt, G. G., and Joshi, A. M. 2015. "A meta-analysis of electronic word-of-mouth elasticity," Journal of Marketing (79:2), pp. 19-39. Zhang, J. Q., Craciun, G., and Shin, D. 2010. "When does electronic word-of-mouth matter? A study of consumer product reviews," Journal of Business Research (63:12), pp. 1336-1341. Zhang, Z., and Varadarajan, B. 2006. "Utility scoring of product reviews," in Proceedings of the 15th ACM International Conference on Information and Knowledge Management, ACM: Arlington, pp. 51-57. Zhou, S., and Guo, B. 2015. "The interactive effect of review rating and text sentiment on review helpfulness," E-Commerce and Web Technologies, pp. 100-111. Zhu, F., and Zhang, X. 2010. "Impact of online consumer reviews on sales:Themoderating role of product and consumer characteristics " Journal of Marketing (74:2), pp. 133-148. Zhu, L., Yin, G., and He, W. 2014. "Is this opinion leader’s review useful? Peripheral cues for online review helpfulness," Journal of Electronic Commerce Research (15:4), pp. 267-280. 34

ACCEPTED MANUSCRIPT

AC

CE

PT E

D

MA

NU

SC

RI

PT

Biographical notes: Hong Hong is a doctoral student of School of Management at Xiamen University. She is currently also a visiting student in the Pamplin College of Business at Virginia Tech funded by China Scholarship Council. Her research interests are social media, information technology and electronic commerce. She has published papers in conferences such as Journal of Electronic Commerce Research, Americas Conference on Information Systems (AMCIS) and Hawaii International Conference on System Sciences (HICSS). Di Xu is a Full Professor of School of Management at Xiamen University. His research interests are technology and innovation management, information system and e-commerce. He has published papers in journals such as International Journal of Production Economics, Information and Management, Expert Systems with Applications and Discrete Dynamics in Nature and Society. G. Alan Wang* is an Associate Professor in the Department of Business Information Technology at Virginia Tech. He received the Ph.D. in Management Information Systems from the University of Arizona, the M.S. in Industrial Engineering from Louisiana State University, and the B.E. in Industrial Management & Engineering from Tianjin University. His research interests text mining, data mining, knowledge discovery, web and social media analytics, entity resolution and security informatics, service computing, and quality engineering. He has published more than 20 refereed articles in various prestigious journals including Productions and Operations Management, Decision Support Systems, Journal of Business Ethics, Journal of the AIS, Communications of the ACM, IEEE Transactions of Systems, Man and Cybernetics (Part A), IEEE Computer, Group Decision and Negotiation, Journal of the American Society for Information Science and Technology, and Expert Systems with Applications. Weiguo Fan is a R. B. Pamplin Professor of Accounting and Information Systems and Full Professor of Computer Science (courtesy) at the Virginia Polytechnic Institute and State University (Virginia Tech). He received his Ph.D. in Business Administration from the Ross School of Business, University of Michigan, Ann Arbor, in 2002, a M. Sce in Computer Science from the National University of Singapore in 1997, and a B. E. in Information and Control Engineering from the Xi’an Jiaotong University, P.R. China, in 1995. His research interests focus on the design and development of novel information technologies—information retrieval, data mining, text analytics, social media analytics, business intelligence techniques—to support better business information management and decision making. He has published more than 150 refereed journal and conference papers. His research has appeared in many premier IT/IS/OM journals such as Information Systems Research, Journal of Management Information Systems, Productions and Operations Management, IEEE Transactions on Knowledge and Data Engineering, Information Systems, Communications of the ACM, Information and Management, Journal of the American Society on Information Science and Technology, Information Processing and Management, Decision Support Systems, ACM Transactions on Internet Technology, ACM Transactions on Management Information Systems, Pattern Recognition, IEEE Intelligent Systems, Journal of Informetrics, Information Systems Frontiers, Journal of Computer Information Systems, Pattern Recognition Letters, International Journal of e-Collaboration, and International Journal of Electronic Business.

35

ACCEPTED MANUSCRIPT Highlights We reviewed extant research about the determinant factors of perceived online review helpfulness. We conduct a meta-analysis to reconcile the contradictory findings on the influence of determinant factors on perceived review helpfulness.

PT

Helpfulness measurement, online review source, and product type are found to be responsible for most of the mixed findings.

AC

CE

PT E

D

MA

NU

SC

RI

Our findings help both sellers and consumers to better identify helpful online reviews and improve the efficiency of their decision making.

36