Intrusion of software robots into journalism: The public's and journalists' perceptions of news written by algorithms and human journalists

Computers in Human Behavior 71 (2017) 291e298

Contents lists available at ScienceDirect

Computers in Human Behavior journal homepage: www.elsevier.com/locate/comphumbeh

Full length article

Intrusion of software robots into journalism: The public's and journalists' perceptions of news written by algorithms and human journalists Jaemin Jung a, Haeyeop Song a, *, Youngju Kim b, Hyunsuk Im a, Sewook Oh b a b

School of Management Engineering, KAIST, Seoul, South Korea Korea Press Foundation, Press Center Bldg., Taepyeongno 1-ga, Jung-gu, Seoul, 100-750, South Korea

a r t i c l e i n f o

a b s t r a c t

Article history: Received 11 July 2016 Received in revised form 6 February 2017 Accepted 6 February 2017 Available online 7 February 2017

This study adopted a two (author: algorithm or journalist) by two (notification of author: real or inverse) between subject design to investigate how the public and journalists perceive the quality of algorithmswritten articles compared with human journalist's work. Findings showed that both the public and journalists' evaluations were varied by the manipulation of author notification. That is, the public gave higher scores to the algorithm's work when it was notified as the real author, but they gave lower scores to the algorithm's work when the author was notified as a journalist. It confirmed the public's negative attitude toward journalists' credibility and craving for new information and communication technology (ICT) products/services in Korea. Based on journalists' resistance to change and innovation and the theory of prejudice, it was expected that journalists would be favorable to another journalist's work and unfavorable to an algorithm's work. However, contrary to the hypothetical expectation, journalists also gave higher scores to an algorithm's work and lower scores to a journalist's work. Implications relating to the intrusion of algorithm-written articles into journalism were discussed. © 2017 Elsevier Ltd. All rights reserved.

Keywords: Robot journalism Journalist Algorithms Technology adoption Ingroup-outgroup distinction

1. Introduction The rise of the Internet has changed not only how news is distributed and consumed but also how it is produced. The Internet has drawn more people and organizations into news production. Beyond journalists in traditional news companies, there are also professional bloggers and online news startups. Even the general public acts as news providers. A recent development in the realm of journalism is software-generated content. Journalism entered a new phase with the rise of computer-written or automatically produced news articles. This advanced technological development has led to a new type of journalismdrobot journalism. News companies have started to work with algorithms to operate and publish software-generated news articles. Templates are produced in journalistic but iterative processes that require painstaking manual work. Although the technology is still in an

* Corresponding author. E-mail addresses: [email protected] (J. Jung), [email protected] (H. Song), [email protected] (Y. Kim), [email protected] (H. Im), [email protected] (S. Oh). http://dx.doi.org/10.1016/j.chb.2017.02.022 0747-5632/© 2017 Elsevier Ltd. All rights reserved.

early market phase, automated journalism has arrived in newsrooms. For example, Forbes uses an artificial-intelligence platform provided by the technology company Narrative Science to generate automated news on corporate earnings and stock prices from live datasets and content harvested from previous articles (Gani & Haddou, 2014). AP partnered with Automated Insights to begin automating quarterly earnings reports and now publishes 3000 such financial stories every quarter (Miller, 2015). After an earthquake hit Los Angeles one morning, it took only 3 min for the LA Times to write and publish an article about it online. It was written by an algorithm (Neal, 2014). Beyond the U.S. news media, companies in other regions introduced algorithm-written articles. Chinese social and gaming giant Tencent published its first business report written by an algorithm in September 2015. It was written in Chinese and completed in just 1 min by Dreamwriter, a Tencent-designed algorithm (He, 2015). The Financial News in Korea ran an article reporting stock market news written by IamFNBOT, an algorithm writer (Son, 2016). Algorithms are used not only in news writing but also in news selection and editing. In the Guardian from the U.K., the algorithmic newspaper software selects the most popular Guardian articles and

292

J. Jung et al. / Computers in Human Behavior 71 (2017) 291e298

assembles them into a weekly printed newspaper (Ellis, 2013). It is well known that search engines like Google also use algorithms to display news on its site. As the technologies are intruding into the creation of news, scholars have started to examine the changing nature of journalism amid data abundance, computational exploration, and algorithmic emphasis with growing significance for the media industry and for journalism as practice and profession (Lewis, 2015; also see the special issue: “Journalism in an era of big data” of Digital Journalism, 2015). Among the data-oriented practices emerging in journalism, Carlson (2015) noted that none appear to be as potentially disruptive as automated journalism, insofar as it calls up concerns about the future of journalistic labor, news compositional forms, and the very foundation of journalistic authority. Meanwhile, automated content with no human intervention beyond the initial programming can make journalists free from handling basic works and afford time to focus on more investigative reporting. It could be an opportunity for media companies to reinvent news production system by generating news faster, at a larger scale, and with fewer errors. It also could be beneficial for audience pursuing more news and unbiased reporting. In fact, journalists are not among the first to feel both the pressure and opportunity of automation. There have been periodic warnings in the last two centuries that automation and new technology were going to wipe our large number of middle class jobs (Autor, 2015). Such concerns have recently regained prominence. In their widely discussed book The Second Machine Age, Brynjolfsson and McAfee (2014) offered an unsettling picture of the likely effects of automation on employment. They asserted that there's never been a better time to be a worker with special skills or the right education, because these people can use technology to create and capture value. However, there's never been a worse time to be a worker with only ‘ordinary’ skills and abilities to offer, because computers, robots, and other digital technologies are acquiring these skills and abilities at an extraordinary rate (Brynjolfsson & McAfee, 2014, p. 11). In this new era of automated journalism, to activate discussion on automated news creation and its implications for journalists, journalism, news industry, and audience, it is necessary to explore the public's and journalists' perceptions and evaluation of news written by algorithms. How does the public perceive and evaluate the quality of computer-written news articles? Can they tell the differences between articles written by a human journalist and algorithms? Journalists perceive themselves as a professional group differentiated with the public, because they have expertise and duty. Then, how do journalists' perception and evaluations of algorithm-written articles compare with those from the public? This study compares the perceived quality of algorithm-written articles with the work by human journalists. To answer the questions, we sampled both general readers and journalists and conducted a series of experiments with human- and algorithm-written articles. 2. Literature review The discourse around the use of computers and software to gather, distribute, and publish content has different kinds of labels (Clerwall, 2014). One term embracing it broadly is “computational journalism,” which is described as “the combination of algorithms, data, and knowledge from the social sciences to supplement the accountability function of journalism” (Hamilton & Turner, 2009, p. 2). Other terms are “robot journalism” (Clerwall, 2014; Dawson, 2010; Van Dalen, 2012) and “automated content” and “algorithmic news” (Anderson, 2013; Bunz, 2010; Levy, 2012). Computational and algorithmic journalism might reshape the

cultural practice of news creation due to the hybrid nature of newsroom sense-making technologies. Anderson (2013) argues that the human becomes partially obdurate and the material partly intentional. For this reason, there have been increasing attention to the impact of algorithm and data in journalism. Some of recent studies focused on the computational journalism using qualitative interviews and textual analysis. Young and Hermida (2015) found that computational thinking and techniques emerged in a (dis) continuous evolution of organizational norms, practices, content, identities, and technologies that interdependently led to new product. Carlson (2015) contended with the emergent practice of automated news content creation both in how it alters the working practices of journalists and how it affects larger understandings of what journalism is, and how it ought to operate. Other studies examined the relationship between journalism and data. Anderson (2015) investigated historicized relationship between journalism and big data, and then asserted we need to consider the material objects (whether interviews, documents, human observations, or other objects) that underlie journalistic processes. Parasie (2015) examined the question about to what extent would the processing of huge datasets allow journalists to produce new types of revelation based on the epistemological approach. Diakopoulos (2015) identified the algorithmic power as something worthy of scrutiny by computational journalists interested in accountability reporting. He tried to show how transparency might be used to effectively adhere to journalistic norms in the use of newsroom algorithms. Using algorithm as part of the journalistic process is not, by any means, a new phenomenon. However, the use of software to actually write a news story is a new advancement (Clerwall, 2014). Being rather new, less attention has been given to automatically generated articles and their implications for journalistic practice and audience response, especially in empirical studies. With the advent of algorithms-written articles, scholars attempted to test the quality of the automatically produced article. The first study was conducted by Christer Clerwall (2014) in Sweden. He investigated how readers perceive software-generated content in relation to similar content written by a journalist. Although there were no statistically significant differences due to the limited sample sized46 respondentsdthe experiment revealed interesting and mixed results. While the softwaregenerated content was perceived as more descriptive and boring, it was also considered to be more informative, objective, accurate, and trustworthy. In this study, the respondents were also asked to assess whether the text had been written by a journalist or by a computer. Of the 27 respondents who read the software-generated text, 10 thought a journalist wrote it and 17 thought it was software-generated. Out of 18 journalists, 8 thought a journalist wrote it, but 10 thought software wrote it. In other words, respondents failed to assess the author of the article correctlydwhether it was written by a journalist or a computer. Overall, journalistic content produced by an algorithm was not or was barely discernible from content written by a journalist in this experiment. Another study explored the perceived credibility of algorithmwritten news articles, searching specifically for differences and similarities between journalists and news consumers in the Netherlands (Van der Kaa & Krahmer, 2014). In this study, 168 native Dutch speakers and 64 Dutch journalists were asked to evaluate the perceived levels of the expertise and trustworthiness of news articles (sports and finance news) written by algorithms. Similar to the previous study (Clerwall, 2014), news consumers perceived the levels of trustworthiness and expertise of the algorithm and journalist equally.

J. Jung et al. / Computers in Human Behavior 71 (2017) 291e298

The lack of differences between algorithm-generated content and articles written by human journalists (Clerwall, 2014; Van der Kaa & Krahmer, 2014) may be seen as an indicator that the software is doing a good job, or it may indicate that the journalist is doing a poor jobdor perhaps both are doing a good (or poor) job. The articles tested in the previous study (Van der Kaa & Krahmer, 2014), which investigate the quality of softwaregenerated news, were written by algorithms, and only the author (computer or journalist) was manipulated. The manipulations, “this article is written by a computer” or “this article is written by a journalist,” were shown on every page. Findings showed that journalists perceived the trustworthiness of a journalist to be much higher than that of an algorithm. In other words, journalists gave a higher score to the article noted as “a work by journalist,” although it was the same content. On the other hand, consumers valued the perceived trustworthiness of a computer writer, although they were slightly negative about the expertise of a journalist. Again, they evaluated the same content and were only manipulated by the notification of the author (computer or journalist). Previous attempts that explored the perceived quality of algorithm-written news articles yielded similar results. Both the public and journalists failed to recognize the differences between news written by an algorithm and a journalist. The quality evaluations of the articles were varied by manipulation of the author. However, they did not figure out why the respondents answered in a different way. This study assumes that the general public's and journalists' stereotypes and different attitudes toward algorithms and journalists might yield different perceptions of news written by algorithms and human journalists. 2.1. The public's attitude toward journalists and technology The image of journalists in popular culture is quite masculine. They are aggressive, self-reliant, tough, ambitious, cynical, cocky, and unsympathetic. Female journalists who are more compassionate, caring, maternal, and sympathetic have fought to overcome this contradiction throughout the 20th century and are still fighting the battle today (Saltzman, 2003). Most portrayals of working journalists highlighted many negative characteristics that branded the profession for decades (Evensen, 2000; Welford, 1998). Bridger (1997) analyzed 59 films containing portrayals of photojournalists and found the stereotypes framed in negative images. A number of movies such as “Network” and “Broadcast News” cynically depicted journalists as greedy, aggressive, and cold-hearted people who are manipulating the news-gathering process to achieve their professional success. Regardless of the real lives of journalists, they are stereotyped negatively. Even in the context of South Korea, the place in which the current study is conducted, journalists are blamed as unreliable and unethical because of a recent event. There was a Sewol ferry tragedy on April 16, 2014. The South Korean ferry capsized while carrying 476 people, who were mostly high school students. In all, more than 300 passengers and crewmembers died in the disaster. The country's mainstream media critically failed in their jobs right in the midst of the disaster. Public criticism has been mounting against the Korean media because they dismissed the truth, published provocative stories, and parroted the government's announcements without checking the facts and questioning them. A growing number of people have called reporters “giregi”da mix of the Korean words meaning reporter and trashdin the wake of the Sewol tragedy. Although journalists regret their wrongdoing and apologized to the public, they cannot yet seem to earn back the people's trust (Cho, 2014). Other evidence shows the public's low credibility to journalists in Korea. The Korea Press Foundation reports an audience's survey

293

with more than 5000 nationals every other year (Korea Press Foundation, 2014). From 2006 to 2014, although the public's evaluation for social influence of journalists was above average, journalists' credibility decreased from 3.00 to 2.68 out of 5.00 points. The score for ethical transparency of journalists also decreased from 2.95 to 2.77. According to these results, journalists in Korea fall short of the required ethical standards that are set in the audience's minds. Historically, trust in journalists and news organizations has not been high in Korea. In particular, misinformation about the Sewol ferry sinking in 2014 and about the outbreak of Middle East Respiratory Syndrome in 2015 affected trust in the news media. A survey on the news consumption conducted by Reuters Institute for the Study of Journalism in 2016 revealed that Koreans' trust for journalists and news organizations was only 17%, which was ranked the 25th out of 26 developed countries participated in the survey (Reuters Institute, 2016). It also might be necessary to look at the Korean consumers' attitudes toward new technology to predict the public's perception of algorithm-written articles. Koreans crave innovative technology adoption and novelty seeking (Choi, Jung, & Lee, 2013; Jung, ChanOlmsted, Park, & Kim, 2012). In February 2015, Korea attained the world's highest smartphone-penetration rate, owing to the country's well-built wireless network system, abundant handset choices, and a cultural propensity toward information and communication technology (ICT) products (Ministry of Science, ICT, & Future Planning, 2015). Large-scale media content and service supply and consumption are available through diverse digital devices such as smartphones and tablet computers. South Korea is a noteworthy market given its highly developed broadband infrastructure and robust consumer demand for ICT and new media content/platform/devices. As one of the world trendsetters in consumer adoption of an emerging content and platform, the findings from the Korean experience might be extrapolated to better understand the nature of consumer evaluation on algorithmwritten articles in other country settings. In line with negative stereotypes and low credibility of journalists and the public's craving for new technology adoption in Korea, it might be expected that the general public is less favorable to journalists' work and more positive toward algorithms' work. Thus, we hypothesize that when the author of the article is noted as an algorithm, the evaluation will be higher, but it will be lower when the author of the article is manipulated as a journalist and vice versa. H1. Regardless of the real author, the public is more likely to be favorable to an algorithm's work and less favorable to a human journalist's article. That is, the public's evaluation will be different by manipulation of author notification. 2.2. Journalists' responses to change If the difference between the algorithm's work and journalist's article is not evident, the first concern from journalists will be job security. In fact, algorithm-written news is not the first threat to journalists. In response to new developments, journalists have often proved to be a conservative community that protects its boundaries (e.g., Lahav & Reich, 2011; Van Dalen, 2012). By stressing the boundaries between the outsider and the journalistic profession, hegemony may be sustained. Previous studies have examined the manner by which professionalization works in the field of journalism. Since Thomas Gieryn (1983) suggested the concept of “boundary work”, many researchers have devoted to the demarcation problem and how does it work within a large occupational, political, and economic social structure. For example, Zelizer (1992) identifies journalists as an “interpretive community”

294

J. Jung et al. / Computers in Human Behavior 71 (2017) 291e298

whose authority stems from discursive source operating both inside and outside the professional sphere. However, one obvious thing is professional claims serve to draw boundary line between those on the “inside” and “outside of the profession. Historical cases reveal how the journalists have been drawn the boundaries between the outsider and the journalistic profession to sustain their hegemony. In the mid-1970s, Computerized Typesetting System (CTS) and Computer Assisted Reporting (CAR) as new technology have deeply affected journalism as labor. CTS have substituted the metal type in newspaper printing and CAR have substituted handwriting in reporting. The computer reduced the labor intensity of journalists in editing and reporting (Howard, 2014). These technologies also reduced the number of employees in the newsroom. For example, the ‘operator’ as technician of the metal type has been disappeared by computer. But, disappeared occupational group in newsroom such as operator did not belong to journalistic profession. These groups were assistant of journalists. Of course, journalists had been compelled to adopt computer as a new reporting and editing device. However, journalists thought of computer as assistant of them. Because they thought of themselves as professionals that are not substituted essentially at that time (Deuze, 2005). The rise of Internet and other forms of participatory media can serve as an example of how journalists respond to new technological developments (e.g., Lowrey & Mackay, 2008; Singer, 2003, 2005; Steensen, 2011). In the mid-1990s, journalists defended their hegemony over journalistic work by stressing the reliability of the information they provide, owing to their professional skills. Thereby they intended to ‘‘strengthen journalists’ claims over the tasks of gathering news and ensuring accuracy’’ (Lowrey, 2006, p. € 494). Ornebring (2013) interviewed with professional journalists and showed that the journalists turn to traditional claims based on expertise and duty when professional journalists want to demarcate the boundary between them and citizen journalism. Journalists face another challenge from machine-written newsdthe robot journalist. Again they are forced to position themselves in relation to an external threat, which potentially challenges their jurisdiction over their main tasks. Van Dalen (2012) studied the response of the journalistic community to automatic content creation by analyzing 68 newspaper articles and blog posts covering Statsheet, an automated sports news creation site, during 2010. Contrary to previous studies that described the journalistic community as conservative and defensive, journalists who wrote about robot journalists did not reject the new development. Van Dalen (2012) interpreted this finding for three different reasons. First, the journalists who wrote about robot journalists can be seen as early adopters who are more open to change than the journalistic community at large. Second, because automated content creation is mostly applied to sports journalism rather than domains like politics or international affairs, journalists might be more generous. Finally, the journalists who wrote about automated-content analysis are observant of a trend that still remains largely abstract and does not influence their own work directly. Cognitive psychologists have found that people are significantly affected by instability in their environments, tend to fear the unknown, and strive to the status quo (Brockner, Grover, Reed, Dewitt, & O'Malley, 1987; Wanberg & Banas, 2000). Introduction of a new system in media organizations has commonly faced resistance from journalists and yielded negative results. Giles (1995) argued in his book on newsroom management that resistance is a rational response to change because change invariably involves losses in routines, relationships, and traditions. Although there were positive or neutral responses on the newsroom convergence experiment (Dupagne & Garrison, 2006; Erdal, 2009), a number of innovation projects, including newsroom restructuring, adoption of

computer systems in newsrooms, experimenting with multimedia and social networking, and partnering with other media, have been fleeting and peripheral to the core operation (Liu, 2006; Lowrey, 2005, 2009, 2011; Aviles, Leon, Sanders, & Harrison, 2004; Daniels & Hollifield, 2002; Domingo, 2008; Gade, 2004; Garcia, 2008; Russial, 2009; Sheffer & Schultz, 2009; Sterling, 2008). With journalists' negative reactions to innovation, it is necessary to pay attention to journalists' occupational homogeneity as an ingroup. Journalists might draw a boundary between the outsider and the journalistic profession (Van Dalen, 2012). Contemporary research on intergroup relations, prejudice, and discrimination appears to accept, at least implicitly, the idea that ingroup favoritism and outgroup negativity are reciprocally related. The idea can be derived from a number of theoretical assumptions. In an environment of scarcity, individuals needed to band together in groups to compete with other groups for survival (Sumner, 1906). Another approach to ingroup-outgroup distinction can be explained by the zero-sum perspective in which attachment and positive affect toward an ingroup is achieved through distance and negative affect toward corresponding outgroups. In fact, it is a sense universally true that “we” are more peaceful, trustworthy, friendly, and honest than “they.” This is reinforced by a general preference for the familiar over the unfamiliar (Brewer, 1999). Following zero-sum perspective, journalists would perceive algorithms replace their job. Otherwise, journalist would perceive the algorithms' works as complementary. Thus, we might expect that journalists have more positive attitudes toward ingroup members' (human journalists') work and negative attitudes toward outgroup members' (algorithms') work. Following the same logic, we can expect that journalists will evaluate a journalist's work lower when the author is noted as an algorithm. H2. Regardless of the real author, journalists are more likely to be favorable to a journalist's article and less favorable to an algorithm's work. That is, journalists' evaluations will be different by manipulation of author notification. 3. Methods 3.1. Selection of stimulus This study selected baseball game articles generated by software that developed by a university research team in Korea. Automated journalism requires high-quality data in structured and machinereadable formats. For this reason, automation works particularly well in domains such as finance, sports, or weather, where data providers make sure that the underlying data are accurate and reliable (Graefe, 2016). Other topics in different sections such as politics, economics, or culture might have more relevance with societal level. However, this study has no choice but to use baseball game articles due to the limited technology in Korea at this moment. The articles used in the experiments should be representative for both algorithm-written content and human journalists' work. We found that algorithm-generated articles are very similar in format and content. We compared the similarity of 10 articles by asking 20 people to indicate the quality (well written, clear, professional, and credible) with a 5-point Likert scale. There were no statistical differences among the articles, and we selected 1 of the 10 articles randomly. In comparison to algorithm-written work, content written by human journalists varies in quality and quantity. Thus, more attention should be paid in selecting representative human journalists' articles. First, we selected 10 articles reporting the same-

J. Jung et al. / Computers in Human Behavior 71 (2017) 291e298

day ball game result. These articles were written by human journalists working for different news companies and were shortened in order to match the length of the one written by the algorithm. Then, we asked 20 respondents to select one article that seemed to be most similar with the algorithm's work already chosen. Additionally, these 20 respondents evaluated the quality of the 10 articles based on the four components (well written, clear, professional, and credible). The results of both tests for selecting a human article yielded the same results. Eight respondents out of 20 selected one article as the most similar to the example written by an algorithm, and the mean score of that article (M ¼ 3.23) was almost equal to the mean of the selected algorithm article (M ¼ 3.22). 3.2. Pretest: comparing an algorithm's and journalist's work without author notification We conducted a pretest on 201 people (103 male and 98 female) that were recruited from a national online panel to validate the stimuli used for the quality evaluation (Mage ¼ 39.6, SD ¼ 11.24). A professional survey company administered the survey. Respondents were given a small reward for completing the questionnaire. The research firm, Hankuk Research in Seoul Korea, emailed a link containing a questionnaire to respondents listed on the panel of the research firm. Participants read one of two articles (written by an algorithm or journalist) without any notification about the author. After reading the article, participants indicated the extent to which they agree on a 5-point scale (1 ¼ not at all to 5 ¼ extremely) regarding the four components of article quality (well written, clear, professional, and credible). The four items were grouped as one factor, and the Cronbach alpha for internal reliability was 0.81. As we expected, there was no statistical difference between the two groups (algorithm: M ¼ 3.21; journalist: M ¼ 3.31; F ¼ 1.269, p ¼ 0.261). Results of the pretest confirmed that the general public failed to discriminate the quality of the algorithm's and human journalist's articles when the author of the articles was not disclosed. Thus, it validated the appropriateness of the stimuli. 4. Study 1: evaluation by the general public Study 1 was designed to test hypothesis 1: the general public's preference for an algorithm-written article over a human journalist's work. When the author is noted correctly, the public would give higher scores on the algorithm's work. However, the evaluation of the algorithm's work would be lower when the author was noted as the journalist. Participants were presented with an article written by an algorithm or human journalist and asked to evaluate the quality of the article. 4.1. Participants and procedure This study consisted of 400 individuals (200 male and 200 female) recruited from a national online panel who were asked to read an article and then answer a series of questions (Mage ¼ 39.3, SD ¼ 11.60). Subjects recruited from a professional research company, Hankuk Research in Seoul Korea. This study adopted a two (author of the article: algorithm or journalist) by two (notification of author: real or inverse) between subject design. The survey was distributed to 6259 people randomly. Among them, 1934 people opened the link and 1335 tried to begin the survey. The survey process was continued until a total of 400 completed questionnaires had been gathered. Note that though the remaining respondents also completed the questionnaires, they were not included in the final analysis because of late responses and budget limitations. The 400 participants were

295



  

:ULWWHQ E\ $OJRULWKP :ULWWHQ E\ +XPDQ

5HDO %\OLQH

,QYHUVH %\OLQH

Fig. 1. The public's evaluation on the articles written by an algorithm and a human journalist.

randomly assigned to one of four conditions until each quota is over. The manipulations, “this article is written by an algorithm” or “this article is written by a journalist,” were shown on the instructions. One group evaluated an article written by an algorithm and was instructed as such. The second group evaluated an article written by an algorithm but instructed as reverse. That is, although the article was written by an algorithm, they were told it was a human journalist's work. The third group evaluated an article written by a human journalist and was instructed as such. The participants of the fourth group were assigned to the article written by a human journalist but were instructed as reverse. After reading the assigned article, each respondent answered the quality of the article (well written, clear, professional, and credible) with a 5-point Likert scale (1 ¼ not at all to 5 ¼ extremely). 4.2. Results We tested our prediction using an analysis of variance (ANOVA) with the author of the article and notification of the author as the factors. As shown in Fig. 1, the author of the article by notification of the author interaction was significant (F ¼ 6.301, p < 0.05). The public gave a higher score (M ¼ 3.39) to the algorithm's work when it was notified as the real author, but they gave a lower score (M ¼ 3.21) to the algorithm's work when the author was noted as a journalist. Although the article was written by an algorithm, evaluation was lower when the instructions stated it was a journalist's work. Meanwhile, the public gave a lower score (M ¼ 3.31) to the journalist's article when the author was noted as a journalist, but they gave a higher score (M ¼ 3.41) to the journalist's work when the author was noted as an algorithm. Thus, regardless of the real author, the public is likely to be more favorable to an algorithm's work and less favorable to a human journalist's article. H1 was supported. Based on the first study, we figured out that the public gave higher evaluation to an algorithm's work when it was noted as the real author, but they gave lower scores to an algorithm's work when the author was notified as the reverse. On the other hand, the public gave lower evaluation to the human journalist's article when the author was noted as a journalist, but they gave higher scores to the journalist's work when the author was noted as the reverse. Thus, whether quality evaluation by the public on an algorithm's work is higher than a journalist's work depends on the manipulation of author notification. 5. Study 2: evaluation of journalists Study 2 was designed to test hypothesis 2: journalists' preferences for a human journalist's article over an algorithm-written

296

J. Jung et al. / Computers in Human Behavior 71 (2017) 291e298

article. When the author is disclosed truthfully, journalists gave higher scores to a journalist's work. However, the evaluation would be lower when the author was noted as an algorithm. Participants were presented with an article written by an algorithm or human journalist and asked to evaluate the quality of the article. 5.1. Participants and procedure There were 164 journalists (111 male and 53 female) recruited via snowballing who were given a link to an article at random order and were asked to read it and then answer a series of questions evaluating the quality of the article (Mage ¼ 38.5, SD ¼ 7.12). Snowball sampling is a nonprobability approach to sampling design and inference in hard-to-reach, or equivalently, hidden population. Standard statistical sampling method requires a list of population members from which the sample can be drawn. Although snowball sampling for journalist hinders the generalization of the results, this study adopted the snowball sampling method because it is difficult to gather journalists' samples systematically. This study selected the first participants among the list of journalists, which participated the previous research in Korea Press Foundation. Study 2 also adopted a two (author of the article: algorithm or journalist) by two (notification of author: real or inverse) between subject design. Participants were assigned to one of four conditions identical to Study 1. Each cell had about 40 participants. The manipulations, “this article is written by an algorithm” or “this article is written by a journalist,” were shown on the instructions. Two groups were aware of the correct author of each article, and the other two groups read the article with wrong information about the author of the article. After reading the assigned article, each respondent ranked the quality of the article (well written, clear, professional, and credible) on a 5-point Likert scale (1 ¼ not at all to 5 ¼ extremely). 5.2. Results To test H2, journalists' preferences for a human journalist's work over an algorithm's work, ANOVA was conducted. As shown in Fig. 2, interaction between the author of the article and the notification of author was significant (F ¼ 8.984, p < 0.01). When the author of the article was noted as the real author, journalists reading an algorithm's article rated higher (M ¼ 3.23), but they gave a lower score (M ¼ 2.85) to an algorithm's work when the author was noted as a journalist. Journalists gave a lower score (M ¼ 3.13) to a journalist's work when the author was noted as a journalist, but they gave a higher score (M ¼ 3.33) to a journalist's work when the author was notified as an algorithm.

   :ULWWHQ E\ 

$OJRULWKP :ULWWHQ E\ +XPDQ

5HDO %\OLQH

,QYHUVH %\OLQH

Fig. 2. The journalists' evaluation on the article written by an algorithm and a human journalist.

In Study 2, we figured out that journalists gave a higher evaluation to an algorithm's work when it was noted as the real author, but they gave lower scores to an algorithm's work when the author was noted as a journalist. On the other hand, journalists gave a lower evaluation on a human journalist's article when the author was noted as a journalist, but they gave higher scores to a journalist's work when the author was noted as an algorithm. Thus, whether quality evaluation by journalists on an algorithm's work is higher than on a journalist's work depends on the manipulation of author notification. It should be noted, however, that the direction of interaction was against our hypothetical expectation. That is, regardless of the real author, journalists are more likely to be favorable to an algorithm's work and less favorable to a human journalist's article. Thus, H2 was not supported. Detailed explanation is provided in the next section. 6. Conclusion and discussion The current study compared the public's and journalists' perceptions of algorithm-written article with that of a human journalist's article and investigated the effect of manipulation by author notification with a series of experiments. Consistent to previous studies testing the quality of algorithmwritten articles in different country settings, this study confirmed that the public failed to discern an algorithm-written article from a human journalist's article and gave undifferentiated evaluation on quality without author notification context. It proved that although algorithm-written articles are in their initial stages in Korea, the quality of algorithm-written articles is quite comparable to that of human journalists. Regarding the public's evaluation of article quality, we found interaction effect of author by true/reverse notification of author. That is, the public gave higher scores on the algorithm-written article when the author was noted as an algorithm. However, they gave lower scores on the algorithm's article when the author was noted as a human journalist by manipulation. On the other hand, the public gave lower scores on a journalist's article when the author was noted truthfully. They gave higher scores on a journalist's article when the author was noted as an algorithm. It was the expected results based on the public's negative attitude toward journalists' credibility and craving for new ICT products/services in Korea. The results of the journalists' evaluations were against expectations. According to the previous theory of prejudice, journalists would give higher scores to ingroup members (i.e., a journalist's work) and lower scores to outgroup members (i.e., an article written by an algorithm). However, journalists gave scores in the opposite way in this study. That is, journalists also gave higher scores to the algorithm-written article when the author was noted as an algorithm but gave lower scores to the algorithm's work when the author was noted as a human journalist. As the public did, journalists also gave lower scores to the journalist's article when the author was noted truthfully. However, they gave higher scores to the journalist's article when the author was noted as an algorithm. Presumably, it might be interpreted as the following reasons. Allport (1954) recognized that attachment to one's ingroup does not necessarily require hostility toward the outgroup. Brewer (1999) empirically confirmed Allport's contention that ingroup favoritism and outgroup prejudice are separable phenomena and that the origin of identification and attachment to ingroups is independent of intergroup conflict. Following Allport’s (1954) and Brewer’s (1999) alternative views, it is assumed that although human journalists in this experiment perceive a journalist who wrote the tested article as a member of the ingroup, the journalists' ingroup identification did not necessarily lead to negative evaluation of the algorithm.

J. Jung et al. / Computers in Human Behavior 71 (2017) 291e298

Moreover, in the absence of realistic conflict between human journalists and algorithm writers, neither strong ingroup attachment nor outgroup hostility was effective. In fact, since this study only tested a sports article, journalists in general did not perceive strong ingroup attachment to the human article. Since algorithmwritten articles are published in sports and stock market reporting areas in a limited way in Korea, journalists did not seem to show any hostility to algorithm-written news yet. Different from the zero-sum perspective between ingroup attachment and outgroup distance, journalists do not see the rise of the algorithm writer as a replacement that leads to a zero-sum game. Rather, journalists seem to expect algorithms would work as complementary in journalism by helping human journalists to have more time to focus on more investigative reporting. According to previous literatures, journalists framed an algorithm as a tool to facilitate systematic coverage and freeing journalists to add more depth, context, and the human touch, as well as possibly decrease costs (Young & Hermida, 2015). Van Dalen (2012) reached a similar conclusion about the impact of algorithmic journalism. Professional claims obviously serve to draw boundary lines between those on the inside and outside of the profession. Similar to CTS case, journalists seem to consider an algorithm as assistant and believe professionals are not substituted essentially. Another interpretation is possible with the notion of concentric loyalties. When comparing to a higher loyalty in more inclusive collectives such as nations and ethnicity, profession is the lessinclusive subgroup (Allport, 1954; Brewer, 1999). In other words, since concentric loyalty among journalists is weaker, it did not spawn outgroup hostility. It is necessary to discuss why both the public and journalists gave higher evaluation to algorithms' work. South Korea has had a massive investment in robotics and automation during the last two decades and a positive public spin which might affect the way people perceive machines and feel trust towards automated systems. Automation bias is a well-researched cognitive tendency that leads to systematic deviations from a standard of rationality and good judgement. There is a risk that people depend excessively on automated systems and, this could be a probable explanation behind the results. The results can be also influenced by a more general, positive first impression of algorithm-written article. The public and journalists both thought, “this is actually not bad for an algorithm.” According to the expectation formation-disconfirmation theory (Bhattacherjee, 2001; Oliver, 1977; Olson & Dover, 1979), consumers' responses after initial trial of any product and service might be “it was poorer than I expected” or “it was better than I expected.” Thus, we might infer that higher evaluation of an algorithm's work was caused from the confirmation. In other words, expectation for an algorithm's work was presumably low, but after reading the article, people thought it was better than expected. An article written by an algorithm was a positive surprise to people. After reading the journalist's article, people were disappointed because it could not meet the quality expectations. Since journalists are more strict and professional than the public on the evaluation of article quality, journalists seemed to be more disappointed and thus gave lower scores to the article written by a human journalist. Considering an algorithm-written article is perceived as “more than expected,” more attention should be paid to the impacts of robot journalism in the near future. Scholars and some journalists have an optimistic view. With algorithms handling basic routine work tasks, journalists would be freed up to focus on more qualified assignments such as long-form or in-depth investigative reporting, which cannot be expected from an algorithm (Flew, Spurgeon, Daniel, & Swift, 2012). Automated content creation can also help to cover stories for small audiences, which now go uncovered. Van

297

Dalen (2012) adds that journalists see the advantages of an algorithm as an unbiased reporter. Tow Center for Digital Journalism suggests other potentials for algorithms in journalism (Graefe, 2016). Algorithms are able to generate news faster, at a larger scale, and potentially with fewer errors than human journalists. Algorithms can use the same data to tell stories in multiple languages and from different angles, thus personalizing them to an individual reader's preferences. Algorithms also have the potential to generate news on demand by creating stories in response to users' questions about the data. There are also concerns about this new phenomenon. Algorithms rely on data and assumptions, both of which are subject to biases and errors. As a result, algorithms could produce outcomes that were unexpected, unintended, and contained errors (Graefe, 2016). It might lead to the ethical quandary. If factual errors are discovered, does the software take responsibility? There are other ethical questions such as transparency and copyright. Although most journalists expect the robots will work as complementary to rather than replacing their jobs, automated content creation can be serious competition and a threat to the job security of journalists performing basic routine tasks (Van Dalen, 2012). In fact, changes in news organizations have been due to coercion from above and to weak-tie connectivity with the organization's market and readers (Lowrey, 2011). To accommodate for an algorithm-written article and develop high-quality journalism with the help of algorithms, more communication and accurate division of work roles within media organizations and understanding readers' needs are necessary. 7. Limitations and suggestions Despite its contributions to the literature on the perceptions of algorithm-written articles, this study has several limitations. First, the algorithm was developed to create sports articles due to the limited technology at this moment, so this study only tested the perceived quality of sports articles. Other news in different sections such as politics, economics, or culture might yield different results. Future research should take into account other content created by algorithms. Second, this study sample consisted only of South Koreans. South Korea has highly developed broadband and mobile infrastructures and features a congested media-consumption environment. Nevertheless, perception and adoption of algorithmwritten articles might differ across countries, and attitudes toward journalists and new ICT services could vary. Moreover, South Korea is a country with a collectivist cultural orientation. It would be valuable to investigate possible differences with other countries that are more individualistically inclined. Thus, the generalizability of our results is limited and requires caution. Third, snowball sampling for journalists hinders the generalization of the results. Although acknowledging the difficulty in gathering journalists' samples systematically, the diverse background and personal work roles might have influence on the perception of algorithm-written articles. Future research should pay more attention in sampling journalists. Lastly, the articles used may not be very representative for either algorithm-written articles or content written by humans. To validate the representativeness of the tested articles, we followed a thorough sample selection process and performed a pretest with 201 samples. Nonetheless, we indicate that the experiment comprised one article from each category, making the risk of generalization. References Allport, G. W. (1954). The nature of prejudice. Cambridge, MA: Addison-Wesley.

298

J. Jung et al. / Computers in Human Behavior 71 (2017) 291e298

Anderson, C. W. (2013). Towards a sociology of computational and algorithmic journalism. New Media & Society, 15(7), 1005e1021. http://dx.doi.org/10.1177/ 1461444812465137. Anderson, C. W. (2015). Between the unique and the pattern: Historical tensions in our understanding of quantitative journalism. Digital Journalism, 3(3), 349e363. http://dx.doi.org/10.1080/21670811.2014.976407. Autor, D. H. (2015). Why are there still so many jobs? The history and future of workplace automation. Journal of Economic Perspectives, 29(3), 3e30. Aviles, J. A. G., Leon, B., Sanders, K., & Harrison, J. (2004). Journalists at digital television newsrooms in Britain and Spain: Workflow and multiskilling in a competitive environment. Journalism Studies, 5(1), 87e100. Bhattacherjee, A. (2001). Understanding information systems continuance: An expectation-confirmation model. MIS Quarterly, 25(3), 351e370. Brewer, M. B. (1999). The psychology of prejudice: Ingroup love or outgroup hate? Journal of Social Issues, 55(3), 429e444. Bridger, E. (1997). From the ridiculous to the sublime: Stereotypes of photojournalists in the movies. Visual Communication Quarterly, 4(1), 4e11. http:// dx.doi.org/10.1080/15551399709363342. Brockner, J., Grover, S., Reed, T., DeWitt, R., & O'Malley, M. (1987). Survivors' reaction to layoffs: We get by with a little help from our friends. Administrative Sciences Quarterly, 32(4), 526e541. Brynjolfsson, E., & McAfee, A. (2014). The second machine age: Work, progress, and prosperity in a time of brilliant technologies. New York and London: W.W. Norton & Company. Bunz, M. (2010). In the US, algorithms are already reporting the news. Guardian. Retrived from http://www.guardian.co.uk/media/pda/2010/mar/30/digitalmedia-algorithms-reporting-journalism. Carlson, M. (2015). The robotic reporter. Digital Journalism, 3(3), 416e431. Cho, C. (2014). Sewol disaster reveals failure of mass media as watchdog. Korea Herald. Retrieved from http://www.koreaherald.com/view.php?ud¼20140516001217. Choi, J., Jung, J., & Lee, S. (2013). What causes users to switch from a local to a global social network site? The cultural, social, economic, and motivational factors of Facebook's globalization. Computers in Human Behavior, 29(6), 2665e2673. Clerwall, C. (2014). Enter the robot journalist. Journalism Practice, 8(5), 519e531. http://dx.doi.org/10.1080/17512786.2014.883116. Daniels, G., & Hollifield, A. (2002). Time of turmoil: Short and long-term effects of organizational change on newsroom employees. Journalism & Mass Communication Quarterly, 79(3), 661e680. Dawson, R. (2010). The rise of robot journalists. In Trends in the living networks. Retrieved from http://rossdawsonblog.com/weblog/archives/2010/04/the_rise_ of_rob.html. Deuze, M. (2005). What is journalism? Professional identity and ideology of journalists reconsidered. Journalism, 6(4), 442e464. Diakopoulos, N. (2015). Algorithmic accountability. Digital Journalism, 3(3), 398e415. Domingo, D. (2008). Interactivity in the daily routines of online newsrooms: Dealing with an uncomfortable myth. Journal of Computer-Mediated Communication, 13(3), 680e704. Dupagne, M., & Garrison, B. (2006). The meaning and influence of convergence. Journalism Studies, 7(2), 237e255. http://dx.doi.org/10.1080/14616700500533569. Ellis, J. (2013). The Guardian experiments with a robot-generated newspaper with the long good read. Nieman Lab. Retrieved from http://www.niemanlab.org/2013/12/theguardian-experiments-with-a-robot-generated-newspaper-with-the-long-goodread/. Erdal, I. J. (2009). Cross-media (Re)production cultures. Convergence, 15(2), 215e231. http://dx.doi.org/10.1177/1354856508105231. Evensen, B. (2000). The drunken journalist: The biography of a film stereotype. Lanham, Md.: Scarecrow Press. Flew, T., Spurgeon, C., Daniel, A., & Swift, A. (2012). The promise of computational journalism. Journalism Practice, 6(2), 157e171. http://dx.doi.org/10.1080/ 17512786.2011.616655. Gade, P. (2004). Newspapers and organizational development: Management and journalist perceptions of newsroom cultural change. Journalism and Communication Monographs, 6(1), 5e55. Gani, A., & Haddou, L. (2014). Could robots be the journalists of the future? Guardian. Retrieved from http://www.theguardian.com/media/shortcuts/2014/mar/16/ could-robots-be-journalist-of-future. Garcia, E. P. (2008). Print and online newsrooms in Argentinian media: Autonomy and professional identity. In C. Paterson, & D. Domingo (Eds.), Making online news: The ethnography of new media production (pp. 61e76). New York: Peter Lang. Gieryn, T. F. (1983). Boundary-work and the demarcation of science from nonscience: Strains and interests in professional ideologies of scientists. American Sociological Review, 48(6), 781e795. Giles, R. H. (1995). Newsroom management: A guide to theory and practice. MI: Media Management Books. Graefe, A. (2016). Guide to automated journalism. Tow Center for Digital Journalism. Retrieved from http://towcenter.org/research/guide-to-automated-journalism/. Hamilton, J. T., & Turner, F. (2009). Accountability through algorithm: Developing the field of computational journalism. In Center for advanced study in the behavioral sciences summer workshop, Stanford, CA, july 27e31. Retrieved from http://dewitt.sanford.duke.edu/wpcontent/uploads/2011/12/About-3Research-B-cj-1-finalreport.pdf. He, H. (2015). End of the road for journalists? Tencent's Robot reporter 'Dreamwriter' churns out perfect 1,000-word news story in 60 seconds. In South China

morning post. Retrieved from http://www.scmp.com/tech/china-tech/article/ 1857196/end-road-journalists-robot-reporter-dreamwriter-chinas-tencent. Howard, A. B. (2014). The art and science of data-driven journalism. Towcenter. Retrieved from http://towcenter.org/the-art-and-science-of-data-driven-journalism. Jung, J., Chan-Olmsted, S., Park, B., & Kim, Y. (2012). Factors affecting e-book reader awareness, interest, and intention to use. New Media & Society, 14(2), 204e224. Korea Press Foundation. (2014). National audience survey 2014. Seoul: Korea Press Foundation. Lahav, H., & Reich, A. (2011). Authors and poets write the news. Journalism Studies, 12(5), 624e641. Levy, S. (2012). Can machines write better news stories than humans?. In Wired. Retrieved from http://www.wired.com/gadgetlab/2012/04/can-an-algorithmwrite-a-better-news-story-than-ahuman-reporter/. Lewis, S. C. (2015). Journalism in an era of big data. Digital Journalism, 3(3), 321e330. http://dx.doi.org/10.1080/21670811.2014.976399. Liu, C.-D. (2006). De-skilling effects on journalists: ICTs and the labour process of Taiwanese newspaper reporters. Canadian Journal of Communication, 31(3), 695e714. Lowrey, W. (2005). Commitment to newspaper TV partnering: A test of the impact of institutional isomorphism. Journalism & Mass Communication Quarterly, 82(3), 495e514. Lowrey, W. (2006). Mapping the journalism-blogging relationship. Journalism, 7(4), 477e500. Lowrey, W. (2009). Institutional roadblocks: Assessing journalism's response to changing audiences. In Z. Papacharissi (Ed.), Journalism and citizenship: New agendas (pp. 44e68). London: Routledge. Lowrey, W. (2011). Institutionalism, news organizations and innovation. Journalism Studies, 12(1), 64e79. http://dx.doi.org/10.1080/1461670X.2010.511954. Lowrey, W., & Mackay, J. B. (2008). Journalism and blogging: A test of a model of occupational competition. Journalism Practice, 2(1), 64e81. Miller, R. (2015). AP's 'robot journalists' are writing their own stories now. The Verge. Retrieved from http://www.theverge.com/2015/1/29/7939067/ap-journalismautomation-robots-financial-reporting. Ministry of Science, ICT, & Future Planning. (2015). Number of smartphone subscribers. Available at: http://www.msip.go.kr/web/msipContents/contents.do? mId¼MTQ2. Neal, R. W. (2014). Robo-journalism: LA times bot writes and publishes earthquake article in 3 minutes. Retrieved from. International Business Times http://www. ibtimes.com/robo-journalism-la-times-bot-writes-publishes-earthquakearticle-3-minutes-1562397. Oliver, R. L. (1977). Effect of expectation and disconfirmation on post exposure product evaluations: An alternative interpretation. Journal of Applied Psychology, 62(4), 480e486. Olson, J. C., & Dover, P. A. (1979). Disconfirmation of consumer expectations through product trial. Journal of Applied Psychology, 64(2), 179e189. € Ornebring, H. (2013). Anything you can do, I can do better? Professional journalists on citizen journalism in six European countries. International Communication Gazette, 75(1), 35e53. Parasie, S. (2015). Data-Driven revelation? Digital Journalism, 3(3), 364e380. Reuters Institute. (2016). Digital news repot 2016. Retrieved from http:// reutersinstitute.politics.ox.ac.uk/sites/default/files/Digital-News-Report-2016. pdf?utm_source¼digitalnewsreport.org&utm_medium¼referral. Russial, J. (2009). Growth of multimedia not extensive at newspapers. Newspaper Research Journal, 30(3), 58e74. Saltzman, J. (2003). Sob sisters: The image of the female journalist in popular culture. University of Southern California. Sheffer, M. L., & Schultz, B. (2009). Blogging from the management perspective: A follow-up study. International Journal on Media Management, 11(1), 9e17. Singer, J. B. (2003). Who are these guys? The online challenge to the notion of journalistic professionalism. Journalism, 4(2), 139e163. Singer, J. B. (2005). The political J-blogger; ‘normalizing’ a new media form to fit old norms and practices. Journalism, 6(2), 173e198. Son, J. (2016). Will robot reporters replace humans? The Korean Observer. Retrieved from http://www.koreaobserver.com/will-robot-reporters-replace-humans-63057/. Steensen, S. (2011). Online journalism and the promises of new technology: A critical review and look ahead. Journalism Studies, 12(3), 311e327. Sterling, J. (2008). A plan for a U.S. newspaper industry counterattack against disruptive innovators. Strategy and Leadership, 36(1), 20e26. Sumner, W. G. (1906). Folkways. New York: Ginn. Van Dalen, A. (2012). The algorithms behind the headlines. Journalism Practice, 6(5e6), 648e658. http://dx.doi.org/10.1080/17512786.2012.667268. Van der Kaa, H. A. J., & Krahmer, E. J. (2014). Journalist versus news consumer: The perceived credibility of machine written news. In Proceedings of the computationþJournalism conference. NY. Retrived from http://compute-cuj.org/cj-2014/ cj2014_session4_paper2.pdf. Wanberg, C. R., & Banas, J. T. (2000). Predictors and outcomes of openness to changes in a reorganizing workplace. Journal of Applied Psychology, 85(1), 132e142. http://dx.doi.org/10.1037/0021-9010.85.1.132. Welford, H. (1998). Please, don't stereotype journalists (or midwives). AIMS Journal, 10(1), 15e16. Young, M. L., & Hermida, A. (2015). From Mr. and Mrs. outlier to central tendencies. Digital Journalism, 3(3), 381e397. Zelizer, B. (1992). Covering the body: The Kennedy assassination, the media, and the shaping of collective memory. University of Chicago Press.