Methods of data collection in English empirical linguistics research: Results of a recent survey

Language Sciences 78 (2020) 101263

Contents lists available at ScienceDirect

Language Sciences journal homepage: www.elsevier.com/locate/langsci

Methods of data collection in English empirical linguistics research: Results of a recent surveyq,qq Ignacio M. Palacios Martínez Department of English and German, University of Santiago de Compostela, Spain

a r t i c l e i n f o

a b s t r a c t

Article history: Received 5 April 2019 Received in revised form 10 November 2019 Accepted 2 December 2019 Available online xxx

Most handbooks on research in English Linguistics state that in conducting any kind of research-based study, the method of data collection should be in agreement with the purpose of the research itself. However, the reality is that many studies into language pay very little attention to this central element of research. In this paper, which can be regarded as a partial replication of Krug and Schlüter (2013) study, I will reflect on this issue, basing my observations primarily on a survey carried out to identify the most common methods of data collection currently used in English empirical linguistics. For this purpose, the abstracts of 1,143 papers published in 2017 from 32 international and high impact journals were consulted, together with a sample of over 200 papers which were examined in their entirety. The journals were grouped according to seven subdisciplines, to see the extent to which there was a correlation between the method of data collection used and the nature of the subdiscipline in question. General findings were broadly as expected. Experimental studies were the most prevalent, followed by corpora, interviews, questionnaires, case studies, ethnographic projects, computer-mediated communication, observation and grammaticality judgement tests. However, in a fifth of all the studies considered it was not in fact clear what method of data collection had been used. The data also show that there was a correspondence between the subdiscipline, and the particular method of data collection used. Contrary to what could be expected, the combination of several research methods was infrequent. Ó 2019 Elsevier Ltd. All rights reserved.

1. Introduction Few would question the claim that carrying out research in English Linguistics today, as in any other scientific discipline, requires a specific kind of technical training. As Wei and Moyer (2008: 18) have noted, “research is an activity that requires analytical and critical thinking at all stages”. One can also add that it is systematic and principled (Brown, 2014: 3). This may

q The term “English Linguistics” as used in the title and throughout the paper should not be taken to be representative of “Linguistics” more broadly. Irrespective of the impact that English Linguistics may have on the disciplinary nature of Linguistics as a whole, the two terms are distinguished for the purposes of this paper. qq For generous financial support, I am grateful to the Spanish Ministry of Economy and Competitiveness (grant PGC2018-093622-B-I00), the European Regional Development Fund and the Galician Ministry of Innovation and Industry (grants ED431D 2017/09 and ED431B 2018/05). I am also indebted to the journal editors and reviewers for their comments and suggestions on this study. E-mail address: [email protected]. URL: http://www.spertus.es/ignacio.html https://doi.org/10.1016/j.langsci.2019.101263 0388-0001/Ó 2019 Elsevier Ltd. All rights reserved.

2

I.M. Palacios Martínez / Language Sciences 78 (2020) 101263

explain why some university undergraduate degree programmes, and most postgraduate ones in English Linguistics now include a module dedicated to the nature of linguistic research and its methodology, encompassing questions related to the different stages of research, such as the selection of a topic, the general structure and organisation of the project, reviewing the literature, the formulation of hypotheses and the statement of aims, the selection and application of the different data collection methods, the design of research instruments and tools, the analysis and interpretation of findings, basic statistical treatment of data, the framing of conclusions, the use and citing of bibliographical references, and the rules of academic language and stylistic conventions. This, together with the natural development of the discipline, may also account for the recent appearance of a number of handbooks on language research which deal with all these issues in response to the need to train newcomers in the field, or to clarify important issues for current researchers. Some of these manuals are general in nature, with a clearly pedagogical purpose (Wray et al., 1998; Wray and Bloomer, 2006; Bowern, 2008; Johnson, 2008; Sealey, 2010; Litosseliti, 2010), while others focus more specifically on language variation and change (Vaux and Cooper, 1999; Tagliamonte, 2006, 2012; Krug and Schlüter, 2013), linguistic fieldwork (Newman and Ratliff, 2001; Bowern, 2008), qualitative research (Mason, 1996; Rasinger, 2008; Silverman, 2015), quantitative studies (Tĕsitelová; 1992; McEnery and Wilson, 2001) or on applied linguistics, second language acquisition and language learning (Brown, 1988, 2014; Brown and Rodgers, 2002; Hatch and Lazaraton, 1991; Nunan, 1992; Tarone et al., 1994; Dörnyei, 2003, 2007; Doughty and Long, 2003; Wei and Moyer, 2008; Brown and Coombe, 2015; Loewen and Plonsky, 2015; Mackey and Gass, 2015; McKinley and Rose, 2017; Phakiti et al., 2018). Independently of the approach taken, what they all share is the broad principle that in conducting any kind of research-based study, the choice of the appropriate method of data collection is crucial, and that this should be directly connected with the purpose of the research. “Research methods are inextricably linked with the research questions being asked”, as Litosseliti (2010: 2) notes. Indeed, this choice is perhaps the most important one for any project, in that it conditions the collection of data, the findings, and the subsequent analysis and interpretation of these. Findings will only be valid and reliable if the method used for data collection has been appropriately selected, suits the intended purpose and responds to plausible, practical and well-articulated working hypotheses; the findings themselves might not in themselves confirm a new theory (Mayo, 2018; Stanford, 2016), be very innovative or lead to a ground-breaking contribution in the field, yet at least the study will be based on sound data collection methods and a robust treatment of those data. Tagliamonte (2006: 17) has argued that “the most fundamental challenge for sociolinguistic research is how to obtain appropriate linguistic data to analyse”, something reiterated by Krug and Schlüter (2013: 5): “Concerning the types of data used in empirical linguistics in particular, researchers have a wide spectrum of possibilities at their disposal, the choice of which depends on the specific purpose of the investigation” (emphasis added). The selection of a research method is of course not the only factor that should be borne in mind. The theoretical framework adopted is also likely to play an important role, method and framework often being closely connected. By theoretical framework I mean the general approach taken, be it cognitive, functionalist, systemic or constructivist. “A theoretical framework consists of concepts and, together with their definitions and reference to relevant scholarly literature, existing theory that is used for a particular study” (Abend, 2008: 173). A distinction should also be made between the notions of research methodology and data collection; the former is wider in scope, and generally includes not only the source or instrument(s) used for the retrieval of data but also information on the theoretical underpinnings of a study, the type of analysis to be used, potential problems in this process, and the decisions made at each stage. In some cases, particularly when experiments are conducted, the description of participants, the context of the investigation, and the procedures may also be a necessary part of the general research methodology. In spite of the direct connection between type of method collection and purpose, it is relatively frequent to find empirical studies in which the relationship between research methods and objectives is not clearly drawn, and this not only impacts on the success of the study itself, but also on the process of submitting and reviewing proposals for conference and journal papers. Nunan (1992: 3) has noted that “research is a systematic process of inquiry consisting of three elements or components: (1) a question, problem, or hypothesis, (2) data, (3) analysis and interpretation of data”. If these three elements are all relevant, the second, the data, is undoubtedly crucial. In light of the above, it is my aim here to reflect on central issues in empirical language research by addressing a number of questions: the topic areas most widely studied, methods of data-collection used, connections between research methods and linguistic disciplines, evolution, if any, in language research over recent decades, etc. I believe it is important that linguists take some time to reflect on the nature of language research as interesting conclusions can be gathered from this reflection, which will definitely have an impact on their own practice as linguists. For this purpose, a survey of a large sample of papers published in high-impact linguistics journals will be conducted. Findings from this analysis, encompassing both abstracts and full papers, will provide a broad and representative picture of the state of language research according to the sample of papers analysed, and will also make it possible to identify fruitful areas for further research. This paper can then be regarded as a partial replication of Krug and Schlüter’s study, although their analysis was simpler and had a limited focus in terms of sample size and the linguistic subfields considered. They surveyed only a total of 361 abstracts extracted from journals published between 2009 and 2010, and their analysis was restricted only to find out

I.M. Palacios Martínez / Language Sciences 78 (2020) 101263

3

whether any affinities could be found between data collection methods and language subdisciplines. Other issues included in the current study were out of their scope.

2. Research questions In the tradition of linguistic studies, several dichotomies have been established in terms of different types of research: primary versus secondary, quantitative versus qualitative, case study versus statistical analysis, theoretical versus empirical, statistical versus interpretive, analytical versus nomological, fieldwork versus no fieldwork, hypothetic-deductive versus inductive, etc (Nunan, 1992: chapter 1; Tĕsitelová; 1992; Mason, 1996; Brown, 1988, 2014; Newman and Ratliff, 2001; Rasinger, 2008; Wei and Moyer, 2008; Brown and Coombe, 2015; Silverman, 2015). For the purpose of this paper, I will differentiate broadly between introspective and empirical approaches to the study of language. Introspective studies are based mainly on the researcher’s own analysis and intuition. The data presented, where these exist, are provided by the researchers themselves, who formulate their hypotheses and seek to confirm or disconfirm (Stanford, 2016) them through discussing examples of their own, or those provided in previous studies, without resorting to any new empirical evidence derived from field-work or from the analysis of information collected by means of questionnaires, interviews, observation, ethnographic studies, texts, experiments or corpora. They normally deal with issues that are mainly theoretical. In contrast, linguists who adopt an empirical perspective use one or more methods of data collection to make firmer their hypotheses and to draw some conclusions, which thus are based on attested facts. In this paper I will concentrate on the latter approach. The following research questions will then be considered: 1. What were the most popular topics and issues in 2017 that were being investigated in the linguistics journals under review? How did the topics vary according to the different linguistic subfields in question? An understanding of those linguistic areas recently being researched will allow us to reflect on the possible reasons for the selection of such issues; whether it is because they have important implications for the understanding of language, because they are not restricted to one individual language but comparisons can be drawn across different languages, or because they have received very little attention thus far; moreover we can also identify those areas in which work is not currently being done, and in this way decide on what sort of contribution we ourselves can make to the field. “The function of research is to add to our knowledge of the world and to demonstrate the ‘truth’ of the commonsense notions we have about the world”, as Nunan (1991: 10) argues. Information on the topic areas most commonly addressed in research is of potential interest to junior researchers who have yet to decide on a research topic, as well as to experienced scholars who may be looking for new lines of work, or indeed who are seeking new perspectives on a familiar field. It may also be thought-provoking from a meta-scientific point of view so as to appreciate the current state of research in the field, and to possibly provide reference values for studies on methodological change over time. It will also be of interest to consider why there are some topic areas which are really popular whereas others are almost ignored. Are there specific issues which can be considered more fashionable than others according to the period of time in question? 2. What were the most common methods of data collection used in 2017 in the linguistics journals under review? A broad knowledge of what sort of evidence modern linguists use in their work, and how this is obtained, will help us to understand new developments in the field and will also inform own research. It will be interesting to see whether language researchers develop new methods of data collection according to ongoing needs. 3. To what extent were these methods related to the linguistic subdiscipline in question? One might expect, for example, that the use of corpora would be the favoured option for historical linguistics, whereas interviews and ethnographic studies would be preferred in sociolinguistics, experimental studies in psycholinguistics and cognitive linguistics. This in turn would lead us to believe that the method used is conditioned to a large extent by the nature of the subdiscipline. In this respect, Krug and Schlüter (2013: 2) refer to the existence of ‘pet methods’ of certain subdisciplines of linguistic research. Our findings, however, will need to provide evidence to confirm this. 4. How common was it to find studies in which several methods of data collection were combined in the linguistic journals under review? There seems to be a growing tendency for combining quantitative and qualitative approaches in the analysis of data (Angouri, 2010), since “for deeper understanding of an object or phenomenon and thus also language it is necessary to know not only its qualitative but also its quantitative side (by counting and measuring)” (Tĕsitelová; 1992: 11). This is particularly so when dealing with certain specific issues or language areas. A similar view is maintained by Wei and Moyer (2008: 28), who claim that “in the field of bilingualism a combination of methods may be most appropriate because a bilingual system is not so neatly defined as the language system of monolingual speakers”; however, this is not always reflected in the different research methods used. This said, it should be borne in mind that the use of different sets of data and collection instruments does not necessarily justify their need nor guarantee the high quality of the results. As Sealey (2010: 32) claims, “a rag-bag of data of many different kinds is unlikely to help you answer your research questions!”. 5. Did particular language journals promote or favour a specific methodology regarding data collection? In terms of guidelines for contributors, or in the general statement of their aims and purposes, some journals clearly state that they will only

4

I.M. Palacios Martínez / Language Sciences 78 (2020) 101263

accept papers in which a corpus-based methodology has been adopted, that they are more oriented towards theoretical than to empirical studies, or that they welcome experimental studies over any other method of data collection, etc. Once again, data from this survey will serve to bear out or call into question such an assumption and discuss the possible consequences of this. 6. What role did new technologies and the internet play in the data collection methods of the selected journals? Technology has introduced changes in the way we see the world as well as in the use of the language, to the extent that new concepts have emerged such as netspeak or internet or electronic language, “a type of language displaying features that are unique to the Internet. arising out of its character as a medium which is electronic, global, and interactive” (Crystal, 2001: 18). This has given rise to the emergence of a large number of digital or computer-mediated communication genres, such as (video)blogs, forums, Twitter, electronic mails, etc. (Herring, 2001; Androutosopoulos and Beisswenger, 2008; Thurlow and Mroczek, 2011; Squires, 2016), which deserve close study in their own right and which can also offer new data for the study of language in these new formats. Thus, it would be of interest to see the degree to which all such recent phenomena are reflected in current data collection methods and how digital genres have been incorporated into language research. Since this study will be cross-sectional rather than longitudinal, the reference point will be the results of previous surveys similar in nature to the one presented here, particularly Krug and Schlüter’s study (2013).

3. Method In order to address the questions above, a sample of 1,143 abstracts from papers published during 2017 in a total of 32 high-impact international journals were consulted. These were considered to be representative of their fields due to their high impact factor, their citation counts, their comprehensive nature, covering different linguistic subfields, the balance of quantitative and qualitative studies which they represent in terms of the material selected, and the fact that they are highly respected within the linguistics community. Whereas it is true that some of these publications might have been replaced by others on the same lines, the guiding principle has been to evolve a sample which can be regarded as minimally representative for each area of linguistic research. Abstracts were considered as the basis for the analysis since a well-conceived abstract should provide information on the purpose and main components of the study. The length of abstracts varied from around 200 to 400 words, and they all followed a similar structure in terms of their content: brief introduction, statement of purpose or working hypotheses, research methodology used, and main findings. Some of the journals in question provided guidelines for abstracts in their instructions for contributors: “A concise and factual abstract of 150 words or less is required. The abstract should state briefly the purpose of the research, the principal results and major conclusions. An abstract is often presented separately from the article, so it must be able to stand alone. For this reason, References should be avoided, but if essential, then cite the author(s) and year(s). Also, non-standard or uncommon abbreviations should be avoided, but if essential they must be defined at their first mention in the abstract itself”. (Brain and Language website, instructions for contributors); , last access, July 27, 2019). “At the beginning of the paper: title, author’s name, author’s affiliation followed by a short abstract (100–150 words, summarising the contents of the paper, e. g. hypothesis/ main course of the argument, data, methodology, main results)”. (English Worldwide website, guidelines for contributors; , last access, July 27, 2019). However, in most cases this information was less detailed: “Your article should include an abstract of approx. 200 words and 3–5 relevant keywords”. (International Journal of the Sociology of Language website, guidelines for contributors; , last access, July 27, 2019). “An abstract of no more than 150 words is also required, together with at least 3 keywords”. (Language in Society website, guidelines for contributors; last access, July 27, 2019). The journals selected were classified into seven categories according to their field of study: (i) Historical Linguistics, (ii) Dialect and Varieties, (iii) Language Contact, Typology and Anthropological Linguistics, (iv) Sociolinguistics, (v) Psycholinguistics, (vi) First and Second Language Acquisition and Teaching, and (vii) Theoretical Linguistics, including here grammar, phonetics/phonology, discourse and pragmatics. The initial intention had been to use just three or four broad categories, but once a preliminary analysis was made, it became clear that an expansion in the number of linguistic disciplines was required to achieve a more fine-grained analysis. Thus, it was that Dialect and Varieties, which might plausibly have been taken as a part of Sociolinguistics, was finally considered as an independent category, given the high volume of papers published here. Furthermore, as mentioned above, the aim was to replicate as far as possible the study conducted by Krug and Schlüter (2013),

I.M. Palacios Martínez / Language Sciences 78 (2020) 101263

5

which covered the period 2009 to 2010. In the current survey, an additional category for what has been referred as “Theoretical Linguistics” has been added here including grammar, phonetics/phonology, discourse and pragmatics; the category “Theoretical Linguistics” did not figure specifically in Krug and Schlüter’s study, yet it seems to be of great significance for the present survey. As before, I am aware that further subdivisions could have been made, but the categories as they stand were felt to offer the greatest scope for insightful analysis. The abstracts were closely examined, then coded and classified according to their journal and date of publication, topic, linguistic subfield, and the method of data collection used. According to the results of a first analysis, ten main methods of data collection were identified as the most common: corpora/databases, ethnography, interviews, case studies, questionnaires, experiments, observation, data extracted from computer-mediated communication (digital genres), grammar and acceptability judgment tests, and a category of other, reserved for those cases where it was not possible to classify the method identified under any of the previous categories. The grouping of the different research methods was not without difficulties, since in some cases the abstract did not provide specific information on the data collection method used, or gave insufficient details here, and in other papers authors provided their own data or discussed evidence drawn from previous studies; in the latter two cases, these abstracts were discarded. Review or state-of-the-art papers were also excluded from the count, and the same applied to those which did not include a clear empirical component. This was occasionally the case with so-called Special Issues, in that these often focus on a particular topic; take, for example, Issue 3 of Volume 36 of World Englishes, which essentially compiles commentaries on Post-Brexit English, with the majority of contributors discussing and reflecting on questions related to this general theme, without providing any empirical data. The following table lists all the journals considered for the analysis together with the corresponding volumes and issues. (see Table 1). As already noted, these journals were all published in 2017 and can be regarded as representative of their respective subdisciplines according to the criteria mentioned above. Some additional criteria were established for selection, an important one being that the journals be listed in the main bibliographical sources and databases, such as Scopus, MLA Bibliography, European Reference Index for the Humanities and Social Sciences (ERIH PLUS), Communication & Mass Media Index, ERA: Educational Research Abstracts Online (T&F), IBR & IBZ: International Bibliographies of Periodical Literature (KG Saur), Journal Citation Reports/Social Science Edition (Clarivate Analytics) and Linguistics Collection (ProQuest). In order to confirm the findings from the initial analysis of the abstracts, the findings were cross-checked. To this end, 228 papers, corresponding to a proportionate stratified random sample of 20% of the total number of papers, were also examined

Table 1 Journals and volumes/issues included in the analysis. Subdiscipline

Journal

Volumes & issues analysed

Historical linguistics

Diachronica Folia Linguistica Historica American Speech English World-Wide Dialectologia et Geolinguistica World Englishes Journal of Language Contact Journal of Pidgin and Creole Languages Languages in Contrast Linguistic Typology Gender and Language International Journal of the Sociology of Language Journal of Sociolinguistics Language in Society Applied Psycholinguistics Brain and Language Cognitive Linguistics Journal of Neurolinguistics Language and Cognition Language, Cognition and Neuroscience ELT Journal International Journal of Bilingualism Journal of Child Language Language Learning Modern Language Journal Research in the Teaching of English Studies in Second Language Acquisition Journal of English Linguistics English Language and Linguistics Journal of Pragmatics Lingua Language

34, 1,2,3,4 51-1,2,3 92, 1,2,3,4 36, 1,2,3 25,1 (only 1 issue) 36, 1,2,3,4 10, 1,2,3 32, 1-2 17, 1-2 21, 1,2,3 11, 1,2,3,4 244, 245, 247, 248 21, 1,2,3,4,5 46, 1,2,3,4,5 38, 1,2,3,4,5,6 164–175 28, 1,2,3,4 41, 42, 43A, 43B, 44 9, 1,2,3,4 1,2,4,5,6, 9,10 71, 1,2,3,4 21, 1,2,3,4,5,6 44, 1,2,3,4,5,6 67, 1,2,3,4 & S1 101, 1,2,3,4 51, 1,2,3,4 39, 1,2,3,4 45, 1,2,3,4 21, 1,2,3 117–122 185–200 93, 1,2,3,4

Dialects and varieties

Language contact, typology & anthropological linguistics

Sociolinguistics

Cognitive linguistics, psycho & neurolinguistics

First and second language acquisition and teaching

Theoretical linguistics (grammar, phonetics/phonology, discourse and pragmatics)

6

I.M. Palacios Martínez / Language Sciences 78 (2020) 101263

in their entirety; for this, a random sample of the papers from the different subdisciplines was selected, taking into account the total number of articles per category, and with each paper having an equal probability of being included in the final sample, that is, using non-biased selection criteria (Tĕsitelová; 1992: 33; Johnson, 2008: 35). These 228 papers were then coded and classified according to subdiscipline and the criteria described above. The aim here was to increase the validity and reliability of the study since, as mentioned above, in some cases authors did not make explicit mention of the method collection used in their abstracts and, as a result, it was difficult to come to firm conclusions on the evidence of abstracts alone. However, going through the whole paper allowed me to gather more detailed information about the research method used, the criteria adopted for the selection of that method and the connections between the research method and the aims of the paper. Thus, the analysis of these papers in their entirety provided additional information that was not evident or indeed available from their abstracts alone. 4. Results In this section 1 will try to answer the research questions established above, using tables and figures to enhance the presentation of findings. Research Question 1: What were the most popular topics and issues in 2017 that were being investigated in the linguistics journals under review? How did the topics vary according to the different linguistic subfields in question? Table 2, below, sets out the overall results here. Broad areas of work have been established within each subdiscipline for the purposes of classification and as an attempt to organize the material coherently. Thus, within Historical Linguistics I have distinguished the subcategories of morphosyntax, including here topic areas, such as conditionals, passives and antipassives, number suppletion, double negation, and phonetics and phonology (mainly sound changes), and the same applies to Dialects and Varieties, in which an independent group has been reserved for varieties of English, due to the high number of papers dealing with new Englishes and outer circle dialects. Within the papers dealing with Language Contact, Typology and Anthropological Linguistics, a special entry has been created for contact languages, e.g. Reo Rapa, Black Cape Dutch, Engst, since a notable proportion of articles focused on these areas. The category Sociolinguistics includes four main areas, with language and gender seeing the highest number of papers together with those dealing with language and identity constructions, and language planning and diversity. The Grammar, Phonetics/Phonology, Discourse and Pragmatics group shows internal balance, with studies belonging to each of these areas together with some on phonetics and phonology; some of the areas studied were genitive alternation, progressive aspect, motion verbs together with metaphor, irony, reformulation markers, and the vowels of contemporary RP. It is in the field of Psycholinguistics where I found the widest range of topics, including those concerned with processing and acquisition, problems in communication, reading and listening, language learning, and brain, language and cognition. Finally, within the final group, First and Second Language Acquisition and Language Teaching, the division was relatively simple, since most of the papers belong to either second language acquisition (heritage language, effects of age of acquisition, vocabulary development, individual differences in L2 morphology learning, etc.) or language teaching and assessment (vocabulary teaching, reading and spelling difficulties, creative writing, teacher talk, etc.). Although Table 2 provides an overview of the main topic areas in linguistic research in 2017 according to the sample of journals analysed, this should not be regarded as exhaustive, since a number of other topics do not feature in the table, either because they were rather marginal in terms of subject matter, or there were a small number of them. Tables 3–6 together with Figs. 1 and 2 show the remaining findings, and will allow us to address the remaining research questions formulated above. Research Question 2: What were the most common methods of data collection used in 2017 in the linguistics journals under review? Looking at the raw figures, experimental studies were the most prevalent, amounting to a third (33%) of the data collection methods used, followed by corpus analysis (18%), interviews (4%), case studies (3%), tests (3%), questionnaires (3%), ethnographic studies (2%), computer-mediated-communication (blogs, Twitter, forums, Facebook, etc.) (2%) and observation (1%). Thus, experimental studies together with corpus analysis represented half of the data collection methods used. This finding cannot be regarded as unexpected since corpus linguistics is gradually growing in importance, boosted by the recent compilation of corpora of many different types; also, experiments here were often closely linked to linguistic disciplines such as Psycholinguistics, language processing, and second language acquisition, all of which have attracted considerable attention recently due to their applied and practical nature, their specific attention to the solving of problems identified in individuals and in our society in broad terms, and their connections to other fields that extend beyond linguistics, that is, their multidisciplinary character. Although the number of abstracts selected for each of the subfields differed considerably (for example, a total of 52, 56 and 67 for Historical Linguistics, Language Contact and Dialect and Varieties respectively, but some 312, 274, 257 and 125 for Psycholinguistics, Theoretical Linguistics, FLA and SLA, and Sociolinguistics), the findings here clearly confirmed the prevalence of experimental studies and corpus analysis as the main data collection methods. These general results are in agreement with those of Krug and Schlüter (2013: 8), although, as mentioned above, their analysis was simpler and had a limited focus. Experimental studies can include a wide variety of tests which in our sample consisted of eye-tracking experiments, subjects’ response to different types of data input, or the completion of a wide range of tasks (written, oral, reading, listening, sequence-recall, picture description, auditory phoneme counting, sentence comprehension, computer-assisted, metalinguistic awareness, self-paced reading, sentence completion, segmentation, sentence repetition, picture naming, memory,

I.M. Palacios Martínez / Language Sciences 78 (2020) 101263

Table 2 Most common topics in research according to different linguistic subfields. Subdiscipline

Topics

Historical Linguistics

Morphosyntax: conditionals, double negation, argument marking & constructions, intransitive motion constructions, passives & antipassives, proper names, possessive inflection, noun class prefixation, number suppletion in adjectives. Phonetics & Phonology: Sound changes. Varieties of English: African American English, Smith Island English, South Asian Englishes, Irish English, Euro Englishes, Tswana English. Morphosyntax: get-passives, ergativity, noun phrase complexity, negative concord. Phonetics and Phonology: rhoticity, central vowels in Andalusian Spanish. Other: language change, language ideologies. Contact languages: Reo Rapa, the Paluai, Black Cape Dutch, Engst. Morphosyntax: complementisers, clausal subordination, diminutive suffixes, compounding, discourse markers, question tags, word-formation, coordination, standard negation in sign language, agreement mismatch in possessive constructions, person as an inflectional category, causation. Other: language mixing and diachronic change, speech technologies, learning English through new technologies, heritage learning, improving motivation to learn English. Language and gender: gender ideologies and sexism, empowerment in women's blogs, gender stereotypes, emerging genders, gender representations in the press, gender and sarcasm, gender and masculinity, discourse, gender and sexuality. Language and identity constructions: ethnolinguistic identity, social meaning, stance and class identities, stance and negotiation. Language planning, diversity and endangerment: language and globalization in South and central Asia, mixed languages. Other: vocatives, incorporating translation in sociolinguistic research, place reference in story beginnings, th-stopping among urban British youth, narratives of migration in Facebook. Morphosyntax: genitive alternation, adjective intensification, general extenders, verbs in the progressive, inversion in clause initial adverbs, first person null subjects, ditransitives, address terms in family life, object agreement in American English, NP recursion over time, motion verbs, evidentiality, the subjunctive, the grammar of English pronouns. Phonetics and Phonology: constraints in T/D deletion in English, social stratification of nonprevocalic /r/ in Edinburgh speech, the vowels of contemporary RP, tone and intonation in discourse management. Pragmatics and Discourse: adaptability in new media, metaphor, irony and sarcasm in political discourse, the pragmatics of borrowing, the microanalysis of online data, metadiscourse markers across different registers, reformulation markers and their functions, the pragmatics of manipulation, multimodal artifacts and the texture of viewpoint, the linguistics of humour, presuppositions. Processing and acquisition: sentence comprehension, contextual effects in second language processing, comprehension of scientific metaphors, speech perception, conceptual representations and figurative language, processing of word-level stress, acquisition of tense morphology, language exposure, the priming of word order, individual differences in first language fluency. Problems in communication: semantic interferences, semantic ambiguity, communicative failures, aphasia, communicative-pragmatic disorders. Reading and Listening: reading network in dyslexia, visual salience, reading development, vocabulary size and auditory word recognition, listening comprehension of academic and everyday language in FLA and SLA. Language learning: sleep and language learning, literacy, second language practice distribution, corrective feed-back. Brain, language and cognition: working memory and language, neural correlates, genes, brain and language, metalinguistic awareness, levels of metaphor, metaphors of time, cognitive grammar and English nominalizations, implicature understanding, the grammar of temporal motion, eye-tracking the effect of word-order. Acquisition, learning and development: heritage language, acquisition of differential object marking, acquisition of noun phrases, third language acquisition, effects of age of acquisition on verbal memory, acquisition of metonymy, word learning, vocabulary development, acquisition of mood, acquisition of relative clauses, development of gesture at speech, developmental phonological disorders, individual differences in L2 morphology learning, task complexity on L2 writing, corpus-based approaches to language learning, effects of mode and task complexity on second language production, early foreign language learning in school, age effects in first language attrition, fluency development before, during and after residence abroad, motivation, learner beliefs, recasts, self-directed language development, translanguaging, formulaic sequences, processing of input, measures of implicit knowledge. Teaching and evaluation: formative assessment, evaluation of classroom peer teaching, teacher talk, collocations and their place in language teaching, reading and spelling difficulties, curriculum development and change, vocabulary teaching, resources for teaching and learning tense and aspect, teachers' mediation strategies, creative writing.

Dialects & varieties

Language contact, typology & Anthropological linguistics

Sociolinguistics

Theoretical Linguistics (grammar, phonetics/ phonology, discourse & pragmatics)

Psycholinguistics

First and second language acquisition and language teaching

7

8

I.M. Palacios Martínez / Language Sciences 78 (2020) 101263

Table 3 Methods of data collection according to linguistic subfield: raw figures and general percentages (abstracts). Subdiscipl.

Historical Linguistics Dialects & varieties Lg. contact, typology, anthrop. Linguistics Socioling. Theoretical Linguistics (grammar, phonetics, Discourse, Pragmatics) Psycholing. FLA & SLA Total

Corpus/Datab. Ethn.

Int.

Case study Quest. Exp.

R

%

R

%

R

%

%

R

%

21 24 16

40% 36% 29%

0 1 0

0% 1% 0%

0 5 5

0% 4 7% 3 9% 3

8% 4% 5%

0 6 1

0 2 9% 4 2% 0

30 67

24% 24%

22 18% 9 0 0% 3

7% 6 1% 6

5% 2%

6 1

5% 2 0% 31

2% 3 11% 1

21 22 201

7% 9% 18%

0 0% 3 1% 26 2%

2% 4% 3%

R

1 0% 5 18 7% 10 41 4% 37

R

Obs.

CMC

Grammar/Accept. Other/uncl./no indic. T judgem. tests

%

R

%

R

R

4% 6% 0%

0 1 0

0% 0 1% 1 0% 1

%

%

R

%

0% 0% 0%

25 22 30

48% 33% 54%

52 67 56

2% 3 2% 1 0% 21 8% 10

1% 4%

43 134

34% 49%

125 274

1 0% 227 73% 1 0% 0 0% 11 15 6% 115 45% 6 2% 0 0% 10 30 3% 381 33% 12 1% 26 2% 32

4% 4% 3%

45 58 357

14% 23% 31%

312 257 1143

0% 0 1% 0 2% 0

Ethn.: Ethnographic; Int.: Interviews; Quest.: Questionnaires; Exp.: Experiments; Obs.: Observation; CMC: Computer-mediated communication.

Table 4 Methods of data collection according to linguistic subfield: raw figures (20% sample of the total number of papers, examined in their entirety). Subdiscipline

Corpus/Datab.

Ethn.

Int. Typological Analysis

Quest.

Exp. Obs. Radio & Classroom TV Data data

Gramm./Accept. Other/uncl./no judgem. tests indic.

Total

Historical Linguistics Dialects & varieties Lg. Contact, typology, anthrop. Linguistics Socioling. Theoretical Linguistics (grammar, phonetics, pragmatics, discourse) Psycholing. FLA & SLA Total

8 5 7

0 2 0

0 0 1

0 0 2

0 1 0

0 2 0

0 1 0

0 0 0

0 0 0

0 0 0

3 2 2

11 13 12

5 24

1 0

5 2

0 3

3 2

1 6

0 0

4 4

0 0

0 5

6 9

25 55

4 4 57

0 1 4

0 3 11

0 0 5

1 3 10

43 28 80

0 4 5

0 0 8

0 3 3

0 0 5

12 6 40

60 52 228

Ethn.: Ethnographic; Int.: Interviews; Quest.: Questionnaires; Exp.: Experiments; Obs.: Observation.

Table 5 Methods of data collection according to linguistic subfield: general percentages (20% sample of the total number of papers, examined in their entirety). Subdiscipline

Corpus/Database

Ethn.

Int.

Typological analysis

Quest.

Exp.

Obs.

Radio & TV data

Classroom data

Gramm./Accept. judgem. tests

Other/uncl./no indic.

Historical Linguistics Dialects & varieties Lg. contact, typology, anthrop. Linguistics Socioling. Theoretical Linguistics (grammar, phonetics, pragmatics, discourse) Psycholing. FLA & SLA Total

73% 37% 58%

0 15% 0

0 0 8%

0 0 17%

0 8% 0

0 15% 0

0 8% 0

0 0

0 0

0 0 0

27% 15% 17%

20% 44%

4% 0

20% 4%

0 5%

12% 4%

4% 11%

0 0

16% 7%

0

0 9%

24% 16%

7% 8% 25%

0 2% 2%

0 6% 5%

0 0 2%

2% 6% 4%

72% 54% 35%

0 8% 2%

0 0 4%

0 6% 1%

0 0 2%

20% 12% 18%

Ethn.: Ethnographic; Int.: Interviews; Quest.: Questionnaires; Exp.: Experiments; Obs.: Observation.

I.M. Palacios Martínez / Language Sciences 78 (2020) 101263

9

Table 6 Methods of data collection in Sociolinguistics.

Gender & Language International Journal of the Sociology of Language Journal of Sociolinguistics Language in Society Total a

Corpus/ Ethn. Database

Interv. Case study Question. Diaries Experiments CMC

Observation Tests

Unclear

More Total than 1 method

11 3

0 13

0 3

1 3

2 3

0 1

0 0

0 0

1 0

0 0

8 22

1 3

23 48

9

5

2

2

0

0

2

0

0

0

8

1

28

7 30/24%

4 4 0 22/17.6% 9/7.2% 6/4.8%

0 5/4%

0 0 1/0.8% 2/1.6%

3 2 3/2.4% 3/2.4%

Note that this value is not recorded when summing the total number of abstracts considered.

Fig. 1. General methods of data collection (percentages).

Fig. 2. Data collection methods in Sociolinguistics (percentages).

1 5 3 1/0.8% 43/34.4% 8a

26 125

10

I.M. Palacios Martínez / Language Sciences 78 (2020) 101263

narrative retelling, etc.) under different conditions. Comparisons were sometimes drawn between experimental and control groups. In the case of corpus-based studies, the authors included in our survey tended to use either historical corpora (Korean Historical Database, Penn Corpora, York-Toronto-Helsinki Parsed Corpus of Old English Prose, ARCHER, Diachronic Corpus of Present-Day Spoken English) or more recent ones (Toronto English Archive, International Corpus of English, International Corpus of Learner English, TIME Magazine Corpus, Bergen Corpus of London Teenage Language, London English Corpus, Corpus of Contemporary American English), as well as dictionaries such as the OED and databases, although authors sometimes designed and compiled their own corpora where this was necessary. The remaining collection methods were clearly far less prevalent, with observation occupying the final position. In addition to this, it should be noted that in almost a third of the abstracts considered (31%), linguists made use of other data collection methods which could not be classified under any of the previous categories and also very frequently it was not clear exactly what method of data collection had been used, or at least the abstract provided no specific indication of this, given the case that for some journals the length of the abstract required was very short indeed. The strict limits on the length of abstracts that journals imposed might have something to do with this. However, such a finding is still certainly quite surprising, given that in most journal guidelines for authors it is clearly stated that the methodology used should be included. Schreier (2013: 17-18) has previously drawn attention to this issue: “The most important sections of articles published in peer-reviewed journals or handbooks are dedicated to the presentation and evaluation of findings, and readers are typically left with sparse information as to how the scrutinized data were actually collected.” It might be supposed from this that researchers in general were more concerned with the presentation of the results of their studies than with how they arrived at them. One might also think that these authors may simply not have followed the journals’ guidelines to the full extent, or may have believed that the inclusion in the abstract of the research method used was a secondary matter, despite the fact that abstracts are typically the first thing that people read about one’s article, and indeed are frequently included on their own in databases and in indexing and abstracting journals. Finally, the results from the sample of papers that were considered in their entirety confirmed the previous findings, with no major differences found (see Tables 4 and 5). Once again, experimental studies and those based on corpus analysis were most common, representing 35% and 25% of the total, respectively, followed by interviews at a rate of just 5%. As expected, the unspecific group ‘other’ now decreased to 18%, in that a full reading of papers allowed me to classify these more accurately. In this respect, the survey of the sample of full papers added valuable extra information. Research Question 3: To what extent were these methods related to the linguistic subdiscipline in question? As expected, the data in Table 3 show a direct correspondence between the linguistic subdiscipline and the particular method of data collection used. Corpus analysis was the most common method in studies on Historical Linguistics (40%), Dialects/Varieties (36%), Language Contact (29%), Sociolinguistics (24%) and Theoretical Linguistics (24%). However, experimental studies were far more frequent in Psycholinguistics (73%), and First and Second Language Acquisition (45%). Finally, grammaticality and acceptance judgment tests were more widely used in Grammar studies, as well as in First and Second Language Acquisition. It is also interesting to observe how the use of corpora was becoming more and more popular in those subdisciplines where linguists worked with large samples of data, drew comparisons across different varieties at all linguistic levels, and were interested in statistical proofs relating to hypotheses. One could gather from this that quantitative studies tended to prevail over qualitative ones although this was not always the case, since statistical analysis were on many occasions accompanied by qualitative studies. The opposite tended to apply in Neurolinguistics and Psycholinguistics, where bare numbers were often not as important as the trends that emerged from experiments. As before, results for the abstracts were borne out by the analysis of the sample of full papers, in that no relevant differences were found here (See Tables 4 and 5). Moreover, these figures showed higher numbers than before for corpus analysis in historical linguistic research, at a rate of 73%, Dialects and varieties (37%), Language contact (58%) and Sociolinguistics (20%). Meanwhile, the category of Theoretical Linguistics (grammar, phonetics and discourse) now amounted to 44%. This new evidence also showed higher proportions than before for experimental studies in the disciplines of Psycholinguistics, and First and Second Language Acquisition, at rates of 72% and 54%, respectively. As Table 6 and Fig. 2 show, the case of Sociolinguistics deserves special attention, in that a broader spectrum of methods was found, with corpus analysis and ethnographic studies ranked first and second, with percentages of 24% and 17.6%, respectively, followed by sociolinguistic interviews (7.2%), case studies (4.8%) and questionnaires (4%); digital genres (forums, blogs) (2.4%), with observation and experimental studies occupying the final two positions (around 2% each). However, Sociolinguistics was clearly the subfield in which most variation was found, with diaries, focus group discussions, and (critical) discourse/conversation analysis also used. On this occasion, the analysis of the full papers showed slight differences compared to that of abstracts, with corpus analysis and interviews ranking first, at a percentage of 20%, followed by studies based on data extracted from radio and TV programmes (16%) and questionnaires (12%) (See Tables 4 and 5 above). A connection might be perceived here between the high number of topics studied from a sociolinguistic perspective and the wide variety of methods of data collection identified. Furthermore, in terms of what happens with each of the journals in this group, the data showed that all of them, except for International Journal of the Sociology of Language (IJSL), followed a similar trend, with the use of corpora as a research method

I.M. Palacios Martínez / Language Sciences 78 (2020) 101263

11

clearly prevailing. In the case of IJSL, ethnographic studies represented 25% of studies, versus 6% for those using corpora. This difference may have to do with a change in the perspective or approach taken, in that contributions in IJSL may have placed more emphasis on the role played by language in society, that is, authors may have taken a more sociological perspective, whereas in the other journals considered authors may have been more concerned with a particular language phenomenon in its own right and its impact on society. Research question 4: How common was it to find studies in which several methods of data collection were combined in the linguistics journals under review? The combination of several research methods was infrequent (around 5%), something that was also recorded by Krug and Schlüter (2013: 9) in their study: “What is most striking in this survey is the fact that the articles whose abstracts were surveyed hardly ever indicate than more than one methodology is applied to the phenomenon under investigation”. This included those cases where, due to the complex and multidisciplinary nature of the study, it would have been advisable to do so. The methods most frequently combined were interviews and questionnaires, interviews and observation, corpus analysis and data extracted from digital genres, observation and focus groups, corpus analysis and interviews, different types of tasks and judgment tests, etc. This combination of research methods was found to be more common in the subfields of Sociolinguistics, Psycholinguistics and First and Second Language Acquisition, and may be as the result of the applied linguistic nature of these areas, in which a problem-solving and interdisciplinary approach to the study of linguistic issues is often adopted (McCarthy, 2001: 1; Cook, 2003: 10-11). This finding could also explain why in these fields I also found the highest number of papers presented jointly by several researchers, rather than by a single author. Each member of the group or team contributed to the whole with their own expertise, with the purpose of making the whole paper more solid and valuable. The use of only one source of data for analysis could be regarded as contrary to the general notion that a mixture of different research methods should be favoured, as many scholars advise: “The results from whatever method applied will be strengthened by results obtained from another method” (Rosenbach, 2013: 278). As Angouri (2010: 31) has also argued: “I discuss the notions of ‘integrating’ and ‘mixing’ both at the level of overarching paradigms (namely mixed methods, qualitative and quantitative) but also at the level of specific methodologies associated with fields of study.” This combination of data-gathering methods is often necessary when carrying out corpus analysis, since corpora, no matter how large they are, often present an insufficient number of tokens or other items under investigation for solid conclusions to be reached, and thus researchers are compelled to look elsewhere for additional data, be it through interviews, questionnaires or digital genres. The same may apply to questionnaire-based studies from which it is not always possible to attain definitive results, since the trends identified are not always well-defined, requiring interviews or the use of observation to compensate for such deficiencies, and in this way to obtain more valid and reliable data. Moreover, the combination of different research instruments may allow a wider number of research questions to be addressed, and is particularly common in multidisciplinary research where scholars from different subdisciplines introduce the methods that are associated with each of these. However, this does not necessarily mean, as mentioned above, that such a mixing of research methods necessarily leads to higher quality research, as this will depend on a rigorous and thorough analysis of the data. Research question 5: Did particular journals promote or favour a specific method regarding data collection? In some journals, such as Applied Psycholinguistics, Brain & Language, English Language and Linguistics, Diachronica, English World-Wide, a particular method of data collection seemed to prevail over all others. The former two of these belonged to a subdiscipline that has already been identified as leaning towards experimental evidence, while the latter three belonged to subdisciplines that tend to use corpora (see Table 3). However, it was not clear whether this was due to the specific policies of the journals or to the nature of the studies published therein. To answer this, I would need to explore in depth the views of the relevant editors here. Sometimes, but not always, clues could be gleaned from the general description of the journal in question. Thus, in Applied Psycholinguistics (AP) Diachronica and Language in Society, the following accounts were found in their introductory descriptions: “AP publishes original research papers on the psychological processes involved in language. It examines language development, language use and language disorders in adults and children with a particular emphasis on crosslanguage studies. The journal gathers together the best work from a variety of disciplines including linguistics, psychology, reading, education, language learning, speech and hearing, and neurology” (AP website; , last access, April 4, 2019) “Diachronica provides a forum for the presentation and discussion of information concerning all aspects of language change in any and all languages of the globe. Contributions which combine theoretical interest and philological acumen are especially welcome”. (Diachronica website; , last access, April 4, 2019) “Language in Society is an international journal of sociolinguistics concerned with language and discourse as aspects of social life. The journal publishes empirical studies of general theoretical, comparative or methodological interest to students and scholars in sociolinguistics, linguistic anthropology and related fields”. (Language in Society website; , last access, April 4, 2019)

12

I.M. Palacios Martínez / Language Sciences 78 (2020) 101263

Note how in Applied Psycholinguistics very little was said about the approach that was particularly favoured, with the emphasis on cross-language studies and muldisciplinarity. However, in Diachronica and Language and Society, explicit mention was made of work of theoretical interest and of an empirical nature, respectively. Research Question 6: What role did new technologies and the Internet play in the data collection methods of the selected journals? To answer this question more accurately, I would have to compare samples of abstracts over time and conduct a longitudinal study. However, with the evidence from the current survey and taking Krug and Schlüter (2013) also as reference, I can confirm, as expected, that English language research in 2017 broadly followed general research traditions, although new methods and sources of data collection were also being introduced, such as materials and samples extracted from computermediated communication genres and the use of more sophisticated tools and experiments reflecting advances in new and developing technologies (Hundt et al., 2007; Androutosopoulos and Beisswenger, 2008; Herring, 2010; Thurlow and Mroczek, 2011). Corpora that consisted of web-based texts (e.g. GloWbE) and even CMC corpora (Beisswenger and Storrer, 2008) were also becoming more and more popular. Computer-mediated communication, that is, the so-called digital genres, which included the selection and analysis of samples extracted from the general Web and from Twitter, blogs, forums, emails, instant messaging, chats, Facebook, Flickr, YouTube, Skype, MySpace, Instagram, etc., were used mainly as sources of data collection in discourse/pragmatic analysis and in sociolinguistic studies (Crystal, 2001; Androutosopoulos and Beisswenger, 2008; Herring, 2010; Squires, 2016). These were also found widely in Media and Communication Studies, although this area and representative journals of this such as Journal of Computer-Mediated Communication, Language@Internet, Communication and Media Studies, Media, Culture and Society, to mention just a few, were not included in our analysis. It might also be assumed that the impact of new genres and technologies may have had an effect on the analysis of data, which was also gradually becoming more sophisticated and technical, with the inclusion of complex statistical analyses and more intricate calculations.

5. Final words and suggestions for further research I hope the findings and discussion here will serve as food for thought regarding our role as language researchers, and that this study has shed some light on the decisions we make when selecting a particular method of data collection. These reflections have been contrasted with data that illustrate real practice by linguists, and that also show how each of the linguistic subdisciplines considered conditioned the type of research method selected, and vice versa. In this respect, it is important to note that overall experimental studies together with corpus analysis represented half of the data collection methods. When considering the different subdisciplines independently, Sociolinguistics stood out as the subfield where most variation of data collection methods was found. I have also presented a broad picture of those topics most widely investigated. This has allowed me to identify language issues which, despite their long traditions in Linguistics, continue to attract attention, possibly because of their complexity or universality, or because it is still possible to contribute something new to the field or to deal with them from a different perspective or approach. The results obtained also showed that some language journals promoted a specific data collection method although in most cases there was not clear indication of this. Interestingly, the analysis confirmed the emergence of new trends in language research, especially with the advent of digital genres and the significant impact of technology, both in society and in our mode of expression, as well as noting the most popular data collection methods within the various linguistic subfields. I also found that the combination of different research methods was not as common as might have been expected given the fact that on many occasions a single method was used. These findings also revealed that future meta-methodological investigations should be conducted with specific fields of study (e.g. historical linguistics, World Englishes, first and second language teaching and acquisition). Homogeneity in terms of the phenomena or contexts studied will lead to more informed discussions of the relative merits of different methods. This paper, however, also has some limitations in terms of the size of the sample of journals studied and the period of publication of these (restricted to 2017), and in the fact that the first part of the analysis extrapolates findings from abstracts to entire articles; to address this latter issue, a sample of full papers (20% of the total) was also considered in the analysis which has contributed to confirm and modulate some of the first findings obtained. It would, then, be useful to expand the period and enlarge the sample of journals for each of the linguistic subfields considered. Questionnaires aimed at the authors and editors of the journals, as a means of further exploring some of these issues, would also enhance our understanding. It might also be worth investigating whether some methods of data collection are regarded by the linguistics community as more scientific than others, in a similar vein to what Schreier (2013: 17) has described with reference to the perception that fieldwork is undervalued: “It seems uncontroversial to say that fieldwork competence has not been highly valued or has been underestimated by the research community, which again may explain why it has not received a prominent place in the literature until very recently”. In this paper I have reiterated the need for a close connection between the method of data collection and the purpose of the study, but I have also noted the importance of looking at the type of analysis conducted and how it is related to the research method. Finally, we might explore the extent to which the research being conducted in English empirical linguistics

I.M. Palacios Martínez / Language Sciences 78 (2020) 101263

13

follows the same patterns as in general linguistics. It seems to me that there is a clear need for more studies here, in which linguists reflect on their research practices and which provide hard data on issues central to their practice. References Abend, Gabriel, 2008. The meaning of theory. Sociol. Theory 26, 173–199. Androutosopoulos, J., Beisswenger, M., 2008. Potentials and limitations of discourse-centred online ethnography. Language@Internet 7, 1–21. Angouri, Jo, 2010. Quantitative, qualitative or both? Combining methods in linguistic research. In: Litosseliti, Lia (Ed.), Research Methods in Linguistics. Continuum, London, pp. 29–49. Beisswenger, Michael, Storrer, Angelika, 2008. Corpora of computer-mediated communication. In: Lüdeling, Anke, Kytö, Merja (Eds.), Corpus Linguistics. An International Handbook, vol. 1, pp. 292–308. Berlin. New York (Series: Handbücher zur Sprach und Kommunikationswissenschaft /Handbooks of Linguistics and Communication Science 29.1). Bowern, Claire, 2008. Linguistic Fieldwork: A Practical Guide. Palgrave Macmillan, Houndmills. Brown, James Dean, 1988. Understanding Research in Second Language Learning: A Teacher’s Guide to Statistics and Research Design. Cambridge University Press, New York. Brown, James Dean, 2014. Mixed Methods Research for TESOL. Edinburgh University Press, Edinburgh. Brown, James Dean, Coombe, Christine (Eds.), 2015. The Cambridge Guide to Research in Language Teaching and Learning. Cambridge University Press, Cambridge. Brown, James Dean, Rodgers, Theodore S., 2002. Doing Second Language Research. Oxford University Press, Oxford. Cook, Guy, 2003. Applied Linguistics. Oxford University Press, Oxford. Crystal, David, 2001. Language and the Internet. Cambridge University Press, Cambridge. Dörnyei, Zoltan, 2003. Questionnaires in Second Language Research: Construction, Administration, and Processing. Lawrence Erlbaum Associates, Mahwah, N. J. Dörnyei, Zoltan, 2007. Research Methods in Applied Linguistics. Oxford University Press, Oxford. Doughty, Catherine, Long, Michael H., 2003. The Handbook of Second Language Acquisition. Blackwell, Oxford. Hatch, Evelyn, Lazaraton, Anne, 1991. Design and Statistics for Applied Linguistics. Heinle and Heinle Pub, Boston. Herring, S., 2010. Computer-mediated conversation: introduction and overview. Language@Internet 7, 1–21. Hundt, Marianne, Nesselhauf, Nadja, Biewer, Carolin (Eds.), 2007. Corpus Linguistics and the Web. Rodopi, Amsterdam. Johnson, Keith, 2008. Quantitative Methods in Linguistics. Blackwell, Malden, Oxford, Carlton. Krug, Manfred, Schlüter, Julia (Eds.), 2013. Research Methods in Language Variation and Change. Cambridge University Press, Cambridge. Litosseliti, Lia (Ed.), 2010. Research Methods in Linguistics. Continuum, London and New York. Loewen, Shawn, Plonsky, Luke, 2015. An A-Z of Applied Linguistics Research Methods. Palgrave, New York, NY. Mackey, Alison, Gass, Susan, 2015. Second Language Research: Methodology and Design. Routledge, New York, NY. Mason, Jennifer, 1996. Qualitative Researching. Sage Publications, London. Mayo, Deborah G., 2018. Statistical Inference as Severe Testing: How to Get beyond the Statistics Wars. Cambridge University Press, Cambridge. McCarthy, Michael, 2001. Issues in Applied Linguistics. Cambridge University Press, Cambridge. McEnery, T., Wilson, A., 2001. An Introduction. In: Corpus Linguistics, second ed. Edinburgh University Press, Edinburgh, pp. 1–24. McKinley, Jim, Rose, Heath, 2017. Doing Research in Applied Linguistics. Routledge, New York, NY. Newman, Paul, Ratliff, Martha, 2001. Linguistic Fieldwork. Cambridge University Press, Cambridge. Nunan, David, 1992. Research Methods in Language Learning. Cambridge University Press, Cambridge. Phakiti, Aek, De Costa, Peter I., Plonsky, Luke, Starfield, Sue (Eds.), 2018. The Palgrave Handbook of Applied Linguistic Research. Palgrave Macmillan, London. Rasinger, Sebastian M., 2008. Quantitative Research in Linguistics. An Introduction. Continuum, London. Rosenbach, Anette, 2013. Combining elicitation data with corpus data. In: Krug, Manfred, Schlüter, Julia (Eds.), Research Methods in Language Variation and Change. Cambridge University Press, Cambridge, pp. 278–294. Schreier, Daniel, 2013. Collecting ethnographic and sociolinguistic data. In: Krug, Manfred, Schlüter, Julia (Eds.), Research Methods in Language Variation and Change. Cambridge University Press, Cambridge, pp. 17–39. Sealey, Alison, 2010. Researching English Language: A Resource Book for Students. Routledge, London. Silverman, David, 2015. Interpreting Qualitative Data, fifth ed. Sage, London. Squires, Lauren, 2016. English in Computer-Mediated Communication: Variation, Representation, and Change. De Guyter, Berlin, Boston. Stanford, Kyle, 2016. Underdetermination of scientific theory. In: Zalta, Edward N. (Ed.), Stanford Encyclopedia of Philosophy Archive. Spring 2016 Edition. https://plato.stanford.edu/archives/spr2016/entries/sicientific-underdetermination/. (Accessed 28 July 2019). Tagliamonte, Sali, 2006. Analysing Sociolinguistic Variation. Cambridge University Press, Cambridge. Tagliamonte, Sali, 2012. Variationist Sociolinguistics: Change, Observation, Interpretation. John Wiley & Sons, London. Tarone, Elaine, Gass, Susan M., Cohen, Andrew, 1994. Research Methodology in Second Language Acquisition. Lawrence Erlbaum, Hillsdale, NJ. Tĕsitelová, Marie, 1992. Quantitative Linguistics. Linguistic and Literary Studies in Eastern Europe 37. John Benjamins, Amsterdam. Thurlow, Crispin, Mroczek, Kristine (Eds.), 2011. Digital Discourse: Language in the New Media (Oxford Studies in Sociolinguistics). Oxford University Press, Oxford. Vaux, B., Cooper, J., 1999. Introduction to Linguistic Field Methods. Münchem: Lincom Europa. Wei, Li, Moyer, Melissa G. (Eds.), 2008. The Blackwell Guide to Research Methods in Bilingualism and Multilingualism. Blackwell, Oxford. Wray, Alison, Bloomer, Aileen, 2006. Projects in Linguistics. A Practical Guide to Researching Language. Arnold, New York. Wray, Alison, Trott, Kate, Bloomer, Aileen, 1998. Projects in Linguistics. A Practical Guide to Researching Language. Arnold, London.