A systematic review of derived verbal behavior research

Journal of Contextual Behavioral Science 12 (2019) 128–148

Contents lists available at ScienceDirect

Journal of Contextual Behavioral Science journal homepage: www.elsevier.com/locate/jcbs

A systematic review of derived verbal behavior research a,⁎

a

b

a

Conny Raaymakers , Yors Garcia , Krystal Cunningham , Lisa Krank , Lauren Nemer-Kaiser a b

b

T

The Chicago School of Professional Psychology, 350 N Orleans, Suite 1050, Chicago, IL 60654-1822, USA Developmental Enhancement Behavioral Health, 854 Washington Ave., Suite 600, Holland, MI 49423, USA

ARTICLE INFO

ABSTRACT

Keywords: Verbal behavior Applied behavior analysis Relational frame theory Autism Derived responding

Practitioners in the field of autism treatment have historically been challenged by training complex language repertoires in individuals with an autism spectrum disorder, specifically when traditional training techniques have demonstrated a failure to promote a generalized repertoire. The inclusion of research from the field of Relational Frame Theory (RFT) may assist the practitioner in developing treatment targets that demonstrate measurable progress in language and cognitive skills. Since the call for a potential synthesis between Skinner's Verbal Behavior and RFT, there has been an increasing trend in publications on derived verbal behavior, with the majority of research articles focused on the intraverbal, often utilizing conditional discrimination training, and a near even split between using participants with a diagnosis of autism and those without any diagnoses. A summary of the research within the field of derived relational responding on each of Skinner's verbal operants is provided, followed by recommendations for practitioners.

1. Introduction Since the publication of Lovaas (1987) seminal research on the lifealtering effects of behavior intervention for individuals with autism, the demand for competent individuals trained to carry out behavioral interventions with individuals with autism has grown substantially (Smith & Iadarola, 2015). According to recent assessments, the demand for Board Certified Behavior Analysts has increased 116% from 2012 to 2014, with over 50% of jobs requiring experience working with autism (Burning Glass Technologies, 2015). A review of autism treatment practices has found communication to be a main objective of behavioral intervention, along with academic readiness and higher cognitive functions (Grow & Kodak, 2010; Howard, Ladew, & Pollack, 2009). Given the emphasis on language in autism treatment, practitioners carrying out behavioral interventions should be fluent in programming to promote the acquisition of advanced language and cognitive skills in an effective and efficient manner (see Dixon, Belisle, Rehfeldt, & Root, 2018; Ming, Moran, & Stewart, 2014). Various conceptual and treatment models have been utilized within the autism literature, including the Lovaas model (Carr & Firth, 2005; Smith & Iadarola, 2015), and models based on Skinner's conceptual analysis of language, coined Verbal Behavior (VB) or Applied Verbal Behavior (AVB; for further review see Kates-McElrath & Axelrod, 2006). Generally speaking, a verbal behavior curriculum would focus on the functional independence of verbal operants, rather than receptive

⁎

versus expressive language categories found in the Lovaas approach (Gamba, Goyos, & Petursdottir, 2015). In addition, there would be a heavy focus on a functional analysis of language, and identification of the stimulus conditions under which a response should be trained (e.g., motivating operations and mand training; Bondy, Esch, Esch, & Sundberg, 2010). Given the practical value of Skinner (1957) conceptualization of language, and its utility to teach complex language as the main objective of behavioral intervention programming (Grow & Kodak, 2010; Howard et al., 2009), it is surprising that few studies have examined more advanced skills, such as tact extensions, private verbal behavior, disguised mands, and autoclitics; in fact, few have reported outcomes on reductions in social communication deficits in general (Dymond & Alonso-Alvarez, 2010; Najdowski, Bergstrom, Tarbox, & Clair, 2017; Presti & Moderato, 2016; Sautter & LeBlanc, 2006). As noted by Dixon, Belisle, McKeel, et al. (2017), by focusing solely on Skinner's elementary verbal operants, and utilizing assessments that have yet to be empirically validated, practitioners are failing to take into consideration the vast research that has been published on derived relational responding (DRR). The impact of Skinner's Verbal Behavior (1957) continues to be debated within the field of behavior analysis, with research noting the majority of articles citing Skinner's Verbal Behavior (80%) is non-empirical in nature (i.e., conceptual articles), with a focus on the early verbal operants (e.g., tacts, mands), and overwhelmingly utilize individuals with language differences as participants (Dymond & Alonso-

Correspondence to: Applied Behavior Analysis Online Program. The Chicago School of Professional Psychology, 350 New Orleans, Suite 1050, Chicago, USA. E-mail address: [email protected] (C. Raaymakers).

https://doi.org/10.1016/j.jcbs.2019.02.006 Received 17 May 2018; Received in revised form 31 January 2019; Accepted 20 February 2019 2212-1447/ © 2019 Association for Contextual Behavioral Science. Published by Elsevier Inc. All rights reserved.

Journal of Contextual Behavioral Science 12 (2019) 128–148

C. Raaymakers, et al.

Alvarez, 2010; McKeel, Rowsey, Belisle, Dixon, & Szekely, 2015; Presti & Moderato, 2016; Sautter & LeBlanc, 2006), with limited outcomes demonstrating a generative language repertoire (McLay, Sutherland, Church, & Tyler-Merrick, 2013; Ming et al., 2014). In contrast, 55% of research on derived relational responding (DRR) is empirical in nature (O’Connor, Farrell, Munnelly, & McHugh, 2017) with a focus on complex skills such as grammar, derived intraverbals, derived tacts, metonymical tacts, and deictic frames (Cullinan & Vitale, 2008; Daar, Negrelli, & Dixon, 2015; Dixon, Belisle, & Stanley, 2017; McHugh & Reed, 2008; Murphy & Barnes-Holmes, 2010a; Ruiz-Sánchez & Montoya-Rodríguez, 2014), and predominately utilize adults or children with typically developing language as participants. The most complex verbal operant studied in the literature on verbal behavior, the intraverbal, has seen an increase in empirical research in recent years; however, the focus continues to be on basic intraverbal questions (e.g., simple fill-in statements, categorization; Stauch, LaLonde, Plavnick, Bak, & Gatewood, 2017) via direct training of the operant (e.g., stimulus control transfer techniques; Aguirre, Valentino, & LeBlanc, 2016). The DRR literature offers practical applicability and efficiency in teaching by demonstrating the acquisition of verbal operants in the absence of direct training (May, Hawkins, & Dymond, 2013); whereas, stimulus control transfer techniques, while successful in establishing new repertoires, typically require several trials to reach criterion (Feng, Chou, & Lee, 2015; Ingvarsson & Hollobaugh, 2011; Ingvarsson & Le, 2011). In addition to offering an efficient teaching method, DRR challenges the traditional view of verbal behavior focused exclusively on the speaker, as well as verbal behavior's lack of research on symbolic behavior (Barnes-Holmes, Barnes-Holmes, & Cullinan, 2000). Three major theoretical accounts have explained the formation of complex language and cognition: naming (Horne & Lowe, 1996), stimulus equivalence (Sidman & Tailby, 1982), and relational frame theory (RFT; Hayes, Barnes-Holmes, & Roche, 2001). While there are conceptual differences among the three approaches (Stewart, McElwee, & Ming, 2013), they remain the three predominant viewpoints in explaining DRR. There has been substantial practical research behind each approach, for example, naming research has been widely conducted on the acquisition of categorization in children (Miguel, Petursdottir, Carr, & Michael, 2008), whereas, the emergence of equivalence across a variety of skill sets has been studied, including the acquisition of reading, mathematics, geography, and money skills in both typically developing individuals, as well as those with autism (McLay et al., 2013). Further, stimulus equivalence procedures have been used to teach complex verbal behavior such as derived textual responses, derived categorical responses, derived intraverbals, and qualifying autoclitics (Dixon, Belisle, & Stanley, 2017; May et al., 2013; Sprinkle & Miguel, 2013). Given the vast application of stimulus equivalence technology, it is surprising that only recently has it been applied to teach derived verbal operants (Belloso-Díaz & Pérez-González, 2015a, 2015b; Zaring-Hinkle, Carp, & Lepper, 2016). Similarly, RFT has had an increase in empirical research over the last thirty years (Dixon, Belisle, McKeel, et al., 2017; O’Connor et al., 2017), with this type of relational training demonstrating an increase in IQ scores and standardized language measures in children with autism (Dixon, Belisle, Stanley, & Rowsey, 2018), as well as neurotypical children and adolescents (Cassidy, Roche, Colbert, Stewart, & Grey, 2016; Colbert, Tyndall, Roche, & Cassidy, 2018). Several years ago, Barnes-Holmes et al. (2000) suggested a possible synthesis between Skinner's theory of verbal behavior (VB) and RFT, as a means to encourage research on the study of complex language and cognition, especially in those with language deficits as seen in individuals with developmental disabilities. Since the publication of the Barnes-Holmes et al. (2000) article, there have been several articles published on derived verbal behavior; however, an up to date quantitative analysis has not yet been conducted. A quantitative analysis of the research is necessary in order to identify gaps in research, common findings, as well as inform practitioners on evidenced based practices. It

is important to highlight that this systematic review takes a wider definition of DRR, that is, we take into consideration all the research that has been conducted in stimulus equivalence and RFT in teaching derived verbal behavior. Although Barnes-Holmes et al. (2000) proposed exclusively integrating RFT and verbal behavior; we consider that the field of autism treatment will benefit from reviewing both accounts. 2. Method 2.1. Search procedure The search criteria focused on research within the field of RFT, and stimulus equivalence pertaining to the derived verbal operants (i.e., mands, tacts, intraverbal, echoic, textual, dictation, and autoclitics). Studies were initially identified through three databases: PsycINFO, Web of Science, and Educational Resources in Education Clearinghouse (ERIC). The following keywords, Boolean operators and truncation were utilized as part of the initial identification: [stimulus equivalence, OR relational frame theory] AND [mand, OR tact, OR intraverbal, OR echoic, OR, textual, OR dictation, OR autoclitic] AND [untaught, OR derived, OR emergent] These key words were selected in order to focus on language that was not the result of direct training, using the language typically found in derived relational responding research (e.g., stimulus equivalence, RFT). However, in order to ensure studies that used alternative descriptors for verbal operants (e.g., requesting, labeling) were not overlooked an additional search using the following keywords, Boolean operators and truncation were used: [stimulus equivalence, OR relational frame theory] AND [demand, OR request, OR command, OR inquire, OR, label, OR expressive, OR vocal imitation, OR mimic, OR reading, OR receptive, OR dictation] AND [untaught, OR derived, OR emergent]. The search was limited to English speaking and peer-reviewed journals. Upon completion of the database searches, an additional step was completed in which the first two authors looked individually in four major behavior analytic journals, The Analysis of Verbal Behavior, The Journal of Contextual Behavioral Science, The Journal of Applied Behavior Analysis, and The Psychological Record, for additional papers that may have been missed through the database searches, using the same keywords from the database search. The Journal of Applied Behavior Analysis as well as The Analysis of Verbal Behavior were selected due to recent citation analyses indicating 57% of articles on the verbal operants were published in these journals (De Souza, Akers, & Fisher, 2017), while The Journal of Contextual Behavioral Science and The Psychological Record were selected for their frequency in publishing RFT literature. 2.2. Inclusion and exclusion criteria In order to be included in the review, the study had to be empirical in nature (i.e., no conceptual articles or review articles), and published between January 1st, 2000 and December 31st, 2017. In addition, dissertations, and studies that utilized common verbal behavior approaches (e.g., transfer-of-stimulus-control), were excluded. Although emergence and untaught relations are mentioned in studies that use transfer of stimulus control procedures in the form of prompt and prompt fading techniques (Bloh, 2008), the processes they refer to did not meet our criterion of derived relational responding, as the process involves a history of direct training and reinforcement. Last, participant demographics (e.g., diagnosis, age, gender) and experimental design were variables identified for coding purposes only. After further analysis, articles that alluded to naming theory were also excluded from the analysis, as these articles typically focused on the direct training of listener and tact repertoires, and tested for generalization of listener and tact repertoires to novel stimuli. The type of responding observed in these studies did not meet our criteria of derived relational responding. 129

Journal of Contextual Behavioral Science 12 (2019) 128–148

C. Raaymakers, et al.

2.3. Data extraction

appeared was calculated (Fig. 2). Due to some articles utilizing both individuals with various diagnoses, as well as individuals without diagnoses, the total number of articles in each category did not add up to the total number of articles review (52). Of the 52 articles, 54% (28 articles) utilized individuals with a diagnosis of ASD or pervasive developmental disorder not otherwise specified (PDD-NOS), while 37% (19 articles) included individuals without any known diagnosis. The remaining articles utilized individuals with varying forms of intellectual or cognitive delays (12 articles), language or speech impairment (7 articles), cerebral palsy, seizure disorder, blindness, or other health impairment (6 articles), anxiety disorders and mood disorders (2 articles), psychotic disorders (2 articles), impulse control disorders (1 article) or obsessive compulsive disorder (1 article). Of the 52 articles, only one study used a between-subjects design (Zaine, Domeniconi, & de Rose, 2014), while the remaining utilized a within-subject treatment design, and approximately half of the studies used two to three participants, with only four studies using 10 or more subjects (Belloso-Díaz & Pérez-González, 2015a, 2015b; PérezGonzález, Belloso-Díaz, Caramés-Méndez, & Alonso-Álvarez, 2014; Zaine et al., 2014). In addition, 68% of experiments within the articles utilized a multiple baseline design, or a derivation of this design (e.g., multiple probe design, non-concurrent multiple baseline), 16% used a pre-test/post-test design, and the remaining 16% used a variety of designs (e.g., alternating treatments, parallel treatments, reversal, withdrawal, or between-groups). In terms of language assessments used, a large number of articles (n = 23) did not include a formal language assessment. Of those that did include language assessments, the VBMAPP (6 articles), Wechsler Intelligence Scale (WAIS, WIPPSI, WISC; 6 articles) and Peabody Picture Vocabulary Test (PPVT; 4 articles) were most commonly used. Finally, inter-observer agreement was reported in all articles, while procedural integrity data was only reported in 19 of the experiments, and one article reported on social validity (Allan, Vladescu, Kisamore, Reeve, & Sidener, 2015).

The selected papers were categorized based on participant demographics (e.g., number of participants, age of participants, and diagnoses) to identify any trends in terms of diagnoses or age range of participants within the DRR literature. In addition, to identify trends on the focus and outcomes of DRR research, data were collected on which operant was studied, whether it was directly trained or derived (e.g., if the authors trained a mand, and then tested for a derived intraverbal), the experimental design utilized, and reliability or procedural integrity measures, social validity measures, probes for generalization, and main findings. Finally, to identify information relative to necessary skillsets of participants to derive language, prerequisite skills listed for participants (i.e., inclusionary criterion), language assessments utilized, the identified independent variable(s) or training procedures used (e.g., match to sample), and the criterion for mastery of the training procedures was also coded. 2.4. Reliability and inter-coder agreement Each of the three research assistants reviewed the articles assigned to them, and were responsible for extrapolating relevant information from the select articles. Training followed a behavior skills training format, in which a model of the coding was conducted, followed by an opportunity for the assistant to complete coding on a sample article with the researcher providing feedback. When inter-coding agreement on two sample articles was 100%, the coders completed their randomly assigned set of articles. Thirty percent of each set of articles were randomly selected for inter-coder agreement. Inter-coder agreement was calculated by adding the number of cells per article in which there was an agreement on the data for each cell, and dividing by agreements plus disagreements. Inter-coder agreement for participant demographics (number of participants, age range, and diagnoses) and main findings was 100% across all three assistants; mean inter-coder agreement for experimental design (operant trained, operant derived, experimental design used, and training methods employed) was 92%, with a range of 85–100%. Inter-coder agreement for additional measures (language assessments, IOA/PI, generalization probes, and social validity measures) was 95%, with a range of 90–100%. For those areas in which there was a disagreement, the principal investigator met with the coder to discuss until inter-coder agreement reached 100%.

2.5.2. Operants studied and main findings Each article was coded based on the operants directly trained, the criterion for acquisition of each target, and the operant that was derived. A summary of findings based on the operant trained follows. 2.5.3. Mand Eleven articles focused on the mand, either in terms of direct training of an operant (e.g., tact) and testing for a derived mand (Nuzzolo-Gomez & Greer, 2004; Rosales & Rehfeldt, 2007; Still, May, Rehfeldt, Whelan, & Dymond, 2015), or directly training a listener response and testing for derived manding (Ribeiro, Elias, Goyos, & Miguel, 2010), or mand training and testing for derived mands (Halvey & Rehfeldt, 2005; Murphy, Barnes-Holmes, & Barnes-Holmes, 2005; Murphy & Barnes-Holmes, 2009a, 2009b; Murphy & Barnes-Holmes, 2010a, 2010b; Rehfeldt & Root, 2005). Overall, eight of the 11 articles focused solely on derived mands and no other derived verbal operant. In terms of specific prerequisite skills required for entry in to the experiment, only two articles included this information (Ribeiro et al., 2010; Rosales & Rehfeldt, 2007). Most articles (82%) did not include prerequisite skills, however, did include general information regarding participant repertoires. Of those articles that included participant repertoires, most describe an established echoic, tact, mand, and intraverbal repertoires, with few articles indicating the skillset measured through formal tests, such as the Wechsler Adult Intelligence Scale (WAIS; Halvey & Rehfeldt, 2005; Rosales & Rehfeldt, 2007; Still et al., 2015). In terms of training procedures, nine of the 11 articles (82%) utilized conditional discrimination training, specifically match to sample training, to establish relationships between stimuli. For example, Murphy and Barnes‐Holmes (2010b) taught three adolescent boys to mand for an item using arbitrary stimuli (i.e., A1), they were then taught to match the arbitrary stimulus A to a second stimulus B, and upon mastery were taught to match B to C. Participants were then

2.5. Results A total of 110 papers were identified through initial database searching (see Fig. 1). After removing duplicates, a total of 69 papers were screened for eligibility. A review of those 69 papers resulted in 44 articles further excluded due to being non-behavior analytic in nature, referenced naming hypothesis, or were conceptual in nature. A search in four additional journals: The Analysis of Verbal Behavior, The Journal of Contextual Behavioral Science, The Journal of Applied Behavior Analysis, and The Psychological Record resulted in an additional set of 27 papers. A total of 52 articles were reviewed and coded. From this set of 52 articles, six were further excluded for not focusing on derived responding, or alluding to naming hypothesis. From there, a search for alternative descriptors for verbal operants was conducted. After removing duplicates, conceptual articles, and articles that did not have a direct focus on derived verbal behavior, six additional articles were discovered. The following is a summary of findings for the 52 articles reviewed (see Fig. 1). 2.5.1. Demographics The articles were reviewed for participant diagnoses, age range, number of participants, and experimental designs. If an article contained more than one experiment, each experiment was coded separately. In terms of diagnosis, the frequency upon which each diagnosis 130

Journal of Contextual Behavioral Science 12 (2019) 128–148

C. Raaymakers, et al.

Fig. 1. Example of the steps taken to identify records in the current review.

Fig. 2. Diagnoses of participants across experimental studies.

tested for their ability to mand for an item using the arbitrary stimulus C. The remaining two articles involved the direct training of an operant (listener responding and tacting) and testing for derived manding (Nuzzolo-Gomez & Greer, 2004; Ribeiro et al., 2010). Mastery criterion for training conditions was typically between 89%

and 100% correct, with only one article requiring less than 89% accuracy in responding (Still et al., 2015). However, trial blocks varied greatly from article to article, with some studies requiring as little as 8 out of 9 responses correct across three trial blocks, to as much as 22 out of 25 correct responses across two trial blocks. In most articles (91%), 131

Journal of Contextual Behavioral Science 12 (2019) 128–148

C. Raaymakers, et al.

all participants derived mands, though in four experiments involving mand training and testing for derived mands (Murphy & BarnesHolmes, 2009b; Murphy & Barnes-Holmes, 2010a, 2010b; NuzzoloGomez & Greer, 2004), multiple exemplar instruction was necessary for derived mands to occur. Finally, only one article cited any probes for generalization (Halvey & Rehfeldt, 2005), with two of the three participants demonstrating generalization of mands across settings.

Rosales et al., 2011; Ziomek & Rehfeldt, 2008) at least one participant in the study did not derive the tact to mastery criterion. The remaining four articles in which all participants demonstrated derived responding to mastery criterion, two of the four (Nuzzolo-Gomez & Greer, 2004; Speckman et al., 2012) utilized multiple exemplar instruction. Finally, only three articles cited any probes for generalization (Halvey & Rehfeldt, 2005; Rosales et al., 2011; Ziomek & Rehfeldt, 2008), with most participants demonstrating generalization of tacts across settings, stimuli, or communicative partners.

2.5.4. Tact Twenty-six articles focused on the tact, however, unlike the research on mand training, the research on tact training predominately (81%) focused on tact training and testing for a derived operants other than the tact (e.g., testing for derived intraverbals made up 67% of the studies). Only 14 of the 26 articles focused on derived tacting in some capacity (Byrne, Rehfeldt, & Aguirre, 2014; Elias & Goyos, 2013; Elias, Goyos, Saunders, & Saunders, 2008; Groskreutz, Karsina, Miguel, & Groskreutz, 2010; Halvey & Rehfeldt, 2005; Keintz, Miguel, Kao, & Finn, 2011; Kodak & Paden, 2015; May, Downs, Marchant, & Dymond, 2016; Nuzzolo-Gomez & Greer, 2004; Petursdottir & Haflidadottir, 2009; Ribeiro et al., 2010; Rosales, Rehfeldt, & Lovett, 2011; Speckman, Greer, & Rivera-Valdes, 2012; Ziomek & Rehfeldt, 2008), which will be the focus of this section. More than half of the articles (9) did not include prerequisite skills; however, the majority (13) did include general information regarding participant repertoires. Of those articles that included participant repertoires, six articles described repertoires through language assessments, such as the VB-MAPP or Inventory for Client and Agency Planning (ICAP; Byrne et al., 2014; Groskreutz et al., 2010; Halvey & Rehfeldt, 2005; May et al., 2016; Ribeiro et al., 2010; Ziomek & Rehfeldt, 2008), while the remainder provided general summaries of repertoires. For those articles that included pre-requisite skills, all were specific to the requirements of the experimental study (e.g., generalized identify matching, or chance level of responding to textual stimuli). In terms of training, the majority of articles that focused on derived tacts utilized conditional discrimination training (match to sample; 21%), or listener training procedures (36%). Only one article utilized a non-traditional listener training procedure (i.e., stimulus pairing observation procedure; Byrne et al., 2014), one article trained mands using the Picture Exchange Communication System (PECS) and tested for derived tacts (Ziomek & Rehfeldt, 2008), and three articles utilized multiple exemplar instruction (Nuzzolo-Gomez & Greer, 2004; Rosales et al., 2011; Speckman et al., 2012). Similar to the research on the mand, the mastery criterion for training varied from article to article, with articles requiring 8 out of 9 responses correct across either a single trial block (e.g., Byrne et al., 2014), two trial blocks (Groskreutz et al., 2010; Keintz et al., 2011), or up to three trial blocks (Halvey & Rehfeldt, 2005). Two articles required 7 out of 8 responses across one trial block or up to two trial blocks (May et al., 2016; Rosales et al., 2011), and only one article required 10 out of 12 correct responses across two sessions (Petursdottir & Haflidadottir, 2009). One article required a pre-determined number of consecutive correct responses (e.g., 10 consecutive correct responses; Speckman et al., 2012). The remaining articles utilized percentages as mastery criterion, either in isolation, or in combination with pre-determined number of trial blocks (e.g., Speckman et al., 2012), and ranged from either 90% across two sessions or 100% within one session (Elias & Goyos, 2013; Elias et al., 2008; Speckman et al., 2012) or 100% accuracy across a trial block (Nuzzolo-Gomez & Greer, 2004; Ribeiro et al., 2010; Ziomek & Rehfeldt, 2008). Latency in responding varied from 5 s up to 10 s, before a response was marked as incorrect due to lack of responding. Of the fourteen articles focused on derived tacts, derivation was varied. In 71% of the articles (Byrne et al., 2014; Elias & Goyos, 2013; Elias et al., 2008; Halvey & Rehfeldt, 2005; Keintz et al., 2011; Kodak & Paden, 2015; May et al., 2016; Petursdottir & Haflidadottir, 2009;

2.5.5. Intraverbal The intraverbal was the most widely studied operant, with 30 of the 52 articles (58%) focused on the intraverbal. Training was either the direct training of an intraverbal and testing for an alternative derived intraverbal (e.g., testing for a reverse intraverbal; 30%), eight (27%) focused on directly training a listener response or a separate operant (e.g., tact or mand) and testing for derived intraverbals, ten articles used packaged training of listener response training and operant training (e.g., mand, tact, intraverbal) and tested for derived intraverbals (33%), and three articles used match to sample training and tested for derived intraverbals. Most articles (70%) did not include prerequisite skills, or inclusionary criterion, however, the majority (21) did include general information regarding participant repertoires. For those articles that included prerequisite skills, only one provided specific repertoires that were not specific to the experimental question (Allan et al., 2015). Of those articles that described participant repertoires, 16 articles described repertoires through standardized language assessments, such as Vineland, or Inventory for Client and Agency Planning (ICAP), or through performance based assessments like the VB-MAPP (Allan et al., 2015; Daar et al., 2015; Devine, Carp, Hiett, & Petursdottir, 2016; Dixon, Belisle, Stanley et al.,2017; Grannan & Rehfeldt, 2012; Greer & Yuan, 2008; Kodak & Paden, 2015; May et al., 2013, 2016; Noro, 2005; Pérez‐González, García‐Asenjo, Williams, & Carnerero, 2007; Shillingsburg, Frampton, Cleveland, & Cariveau, 2017; Smith et al., 2016; Vallinger-Brown & Rosales, 2014; Ziomek & Rehfeldt, 2008), while the remainder provided general summaries of repertoires. A large number of the studies (18; 60%) participant repertoires were described as no known delays in language, or advanced communication (e.g., spoke in complete sentences, more than 500 mands, tacts, and 50 or more intraverbals; Allan et al., 2015; Belloso-Díaz & Pérez-González, 2015a; Belloso-Díaz & Pérez-González, 2015b; Belloso-Díaz & PérezGonzález, 2016; Carp & Petursdottir, 2012; Daar et al., 2015; Dickes & Kodak, 2015; Dounavi, 2014; Greer, Yaun, & Gautreaux, 2005; Keintz et al., 2011; Kodak & Paden, 2015; May et al., 2016; Mellor, Barnes, & Rehfeldt, 2015; Miguel, Petursdottir, & Carr, 2005; Pérez-González, Herszlikowicz, & Williams, 2008; Pérez-González et al., 2014; PérezGonzález & García-Asenjo, 2016; Petursdottir & Haflidadottir, 2009). The remainder of the articles described repertoires of moderate communication (e.g., spoke in two to four word sentences, less than 500 mands, tacts, and 25–50 known intraverbals; Allan et al., 2015; Greer & Yuan, 2008; Noro, 2005; Pérez‐González et al., 2007; Smith et al., 2016; Vallinger-Brown & Rosales, 2014), and only one article described participant repertoires on the low end (e.g., tact scores as low as a 6 on the VBMAPP, and an IV score as low as 0 on the VB-MAPP; Shillingsburg et al., 2017). Those articles that focused on directly training a listener response or a separate operant (e.g., mand training), most utilized some form of tact training procedure (4; Devine et al., 2016; Grannan & Rehfeldt, 2012; May et al., 2013; Mellor et al., 2015). For example, Devine et al. (2016) taught seven neurotypical children to label stick figures using nonsense syllables when asked “who is this?” using vocal prompts and differential reinforcement (in the form of token delivery) for independent responding. Once participants met mastery criterion, defined as 100%

132

Journal of Contextual Behavioral Science 12 (2019) 128–148

C. Raaymakers, et al.

correct responding across three trial blocks, intraverbal probes were conducted. Similarly, an additional seven articles used a combined package containing tact training and other training procedures (e.g., intraverbal, or listener responding; Belloso-Díaz & Pérez-González, 2015a; Belloso-Díaz & Pérez-González, 2016; Dounavi, 2014; Miguel et al., 2005; Pérez-González & García-Asenjo, 2016; Petursdottir & Haflidadottir, 2009; Shillingsburg et al., 2017). For example, Miguel et al. (2005) taught six neurotypical children to label pictures of stimuli using adjectives (e.g., empty, full, lit, unlit) to the question “how is it?”. Upon completion of tact training, participants were then probed on their ability to select the stimuli when presented with the antecedent stimulus “point to __”. Following listener response training, the participants were taught to respond to the contextual cues of “same” and “opposite” via match to sample training. Upon completion of tact training, listener response training, and match to sample training, intraverbal probes (e.g., “name the opposite of lit”) were conducted. Findings suggest that intraverbals emerged as a result of the auditoryvisual conditional discrimination training (e.g. point to the same, point to the opposite) versus the listener response, or tact training only conditions. Findings on derived intraverbals using tact training procedures were mixed: of those articles that used tact training procedures (Devine et al., 2016; Grannan & Rehfeldt, 2012; May et al., 2013; Mellor et al., 2015) or used tact training in combination with intraverbal training or listener response training (Belloso-Díaz & Pérez-González, 2015a, 2016; Dounavi, 2014; Miguel et al., 2005; Pérez-González & García-Asenjo, 2016; Petursdottir & Haflidadottir, 2009; Shillingsburg et al., 2017) the occurrence of derived responding in 100% of participants was found in four of the 11 articles (Belloso-Díaz & Pérez-González, 2015a; Dounavi, 2014; Grannan & Rehfeldt, 2012; Shillingsburg et al., 2017). Rather, most articles had at least one participant that did not derive intraverbals to criterion. Of the remaining experiments focused on derived intraverbals, three utilized listener training (Dixon, Belisle, Stanley, Daar, & Williams, 2016; Smith et al., 2016; Vallinger-Brown & Rosales, 2014), and two used a combination of listener and intraverbal training (Kodak & Paden, 2015; May et al., 2016). Across the five experiments using listener response training, alone or as part of a package, in only one study did the participants derived 100% of the intraverbal relations (Dixon et al., 2016). For example, in Smith et al. (2016) one participant required additional tact training in order to demonstrate emergence of intraverbals, whereas in Vallinger-Brown and Rosales (2014) transfer of stimulus control procedure was required for intraverbals to emerge in three of the four participants. In addition to listener training, one article focused on a combination of dictation taking and intraverbal training (Greer et al., 2005), one trained mands (Ziomek & Rehfeldt, 2008), and three focused on match to sample training on derived intraverbals (Daar et al., 2015; Keintz et al., 2011; Noro, 2005). The remaining nine experiments focused on directly training an intraverbal, and testing for derived intraverbals. The findings of these series of papers were similar to those of tact training: at least one participant failed to demonstrate derived intraverbals to mastery criterion. For example, Allan et al. (2015) demonstrated reverse intraverbals in three of the four participants, Pérez‐González et al. (2007) found that participants required the direct training of both intraverbals and reverse intraverbals before the novel intraverbals emerged, however, Dickes and Kodak (2015) demonstrated that training of the original intraverbal was sufficient to produce reverse intraverbals.

autoclitic frames (Luke, Greer, Singer-Dudek, & Keohane, 2011; Speckman et al., 2012) one focused on dictation taking (Greer et al., 2005), and nine focused on textual responses (De Souza & Rehfeldt, 2013; Elias & Goyos, 2013; Elias et al., 2008; Groskreutz et al., 2010; Keintz et al., 2011; Miguel, Yang, Finn, & Ahearn, 2009; Sprinkle & Miguel, 2012; Still et al., 2015; Zaine et al., 2014). The two articles focused on derived autoclitic frames (Luke et al., 2011; Speckman et al., 2012), both utilized multiple exemplar training as the independent variable, and tested for the use of the autoclitic frames with novel mands and tacts. Both articles demonstrated derived responding to novel stimuli using trained autoclitic frames. One article focused on derived dictation taking (Greer et al., 2005), however, two articles utilized dictation taking to test for other derived verbal operants (e.g., mands; De Souza & Rehfeldt, 2013; Still et al., 2015). Similar to the research on autoclitics, multiple exemplar instruction was used for training of dictation taking. For example, Greer et al. (2005) found that all participants spelled untaught responses either vocally (e.g., intraverbal) or written (e.g., dictation taking) with 80–100% accuracy. Finally, nine articles focused on derived textual responding (De Souza & Rehfeldt, 2013; Elias & Goyos, 2013; Elias et al., 2008; Groskreutz et al., 2010; Keintz et al., 2011; Miguel et al., 2009; Sprinkle & Miguel, 2012; Still et al., 2015; Zaine et al., 2014). Three of the nine articles had 100% of their participants demonstrate derived textual responses (De Souza & Rehfeldt, 2013; Groskreutz et al., 2010; Miguel et al., 2009), three of the remaining six experiments demonstrated that the conditional discrimination training procedure produced greater derivation in the other verbal operants tested (e.g., tact) than dictation taking (Elias & Goyos, 2013; Keintz et al., 2011; Zaine et al., 2014). 3. Discussion A review of research articles published between 2000 and 2017 found 52 articles on derived verbal operants (sans naming hypothesis) with most found in the area of intraverbals; the total number of studies focused on derived intraverbals was nearly equal to studies focused on all other operants combined (Fig. 3). Additionally, it appears that there has been an overall increasing trend in publication of research on derived verbal operants over the last seventeen years (Fig. 4). The increasing trend in DRR literature is in line with recent citation analyses on RFT (O’Connor et al., 2017); however, while O’Connor et al. (2017) found a greater number of empirical publications on RFT (n = 160), a large portion of the studies focused on establishing frames of coordination, opposition, or comparison. Taken together, these findings suggest that while there has been an increasing focus within the RFT literature on the verbal operants, the growth pales in comparison to the research focused on other areas (e.g., comparative frames). In the current review, one article utilized advanced statistical analyses (Devine et al., 2016). Fisher and Lerman (2014) argued for an increase in use of statistical analyses in applied behavior analysis research in an attempt to conduct meta-analyses to measure the efficacy of a particular intervention (for detailed information on the benefits of statistical analyses in behavior analytic research readers should reference Fisher & Lerman, 2014). Future research in the area of DRR may want to include statistical analyses such as effect sizes, error analysis, or multilevel analysis, in order to assist practitioners in identifying best practice methods for deriving a specific verbal operant in individuals with autism (see Manolov & Moeyaert, 2017). For example, Young (2018) recommends behavior analysts move away from measuring behavior as responses per unit time, and utilize measures such as interresponse time, and in doing so can use additional statistical analysis such as logistic regression for repeated measures. As noted, statistical analyses aid in providing a more accurate summary of findings across participants, similarly, social validity data aid in determining whether the findings are of social importance to the

2.5.6. Textual, dictation, and autoclitic Twelve articles involved derived verbal operants other than the mand, tact, and intraverbal. Of those articles, match to sample was the most common procedure used to train and test for derived verbal operants. Of those twelve articles, two focused on testing for derived

133

Journal of Contextual Behavioral Science 12 (2019) 128–148

C. Raaymakers, et al.

Fig. 3. Derived verbal operant studied across review articles.

community (Wolf, 1978), through the use of questionnaires, comparative assessments (e.g., evaluating the participants’ outcomes with that of a peer group), or through assessing the generalization and/or maintenance of the outcomes (Carr, Austin, Britton, Kellum, & Bailey, 1999; Snodgrass, Chung, Meadan, & Halle, 2018). While the importance of social validity data is without question, analyses within the field have reported weak social validation practices (see Carr et al., 1999; Snodgrass et al., 2018 for proposed explanations); similarly, in the current review social validity data was present in only one article (Allan et al., 2015). Finally, in terms of additional indicators of social validity (i.e., generalization and maintenance), only eight studies in the current review specifically note probes for generalization or maintenance. Future research in derived relational responding should focus on noting probes for generalization or maintenance of the operant, in order to compare these results to those observed in verbal behavior research. With regards to the current review, the majority of the articles focused on derived intraverbals, utilizing tact training as part of the

teaching procedure. This is in direct contrast to what is observed in the verbal behavior literature, where the most commonly studied operants are the mand and tact (see De Souza et al., 2017; Petursdottir & Devine, 2017). While derived relational responding did not emerge consistently across all participants, the most effective training procedures for derived intraverbal responding appeared to be either tact training, or intraverbal training (e.g., training an A-B and B-C intraverbal, and testing for A-C or C-A intraverbal). A recent review of the multiple control of the intraverbal by Stauch et al. (2017) found only five studies in the verbal behavior literature that brought behavior under divergent stimulus control; one of those five articles was also included in the current review on derived relational responding (Grannan & Rehfeldt, 2012). Given that Stauch et al. (2017) argue that little is known regarding divergent control and the intraverbal, future research may want to review the extent to which training methods in DRR research support multiple control of the intraverbal (see Lee, Chou, & Feng, 2017, for an example). The mand and tact were the next most studied operant (11 and 14

Fig. 4. Frequency of derived verbal operant studies published between 2001 and 2017.

134

Journal of Contextual Behavioral Science 12 (2019) 128–148

C. Raaymakers, et al.

However, Fuller and Fienup (2018) did not investigate the number of observations (i.e., sessions) therefore, further studies should investigate the effects of increased observation on retention (e.g., mastery criterion across two versus three observations). Of note, for those studies that used a rolling block, versus those that utilized block mastery, recent studies have found that each mastery type produced similar results on derived responding, therefore, the format does not appear to affect whether derived relations emerge in participants (Fienup & Brodsky, 2017). There are several limitations that should be noted in this review. Excluding articles that referenced naming hypothesis in this review is a major limitation; future research may want to include this area in reviews of derived relational responding to identify any potential differences in terms of operant studied (e.g., echoics), training methods employed, as well as findings. In addition, it is possible that limiting the search to three databases (i.e., PsycINFO, Web of Science, and ERIC), as well as limiting our search to four major behavior analytic journals, resulted in missed articles. Future research may want to expand to include other journals that study verbal operants, including Behavioral Interventions, The Journal of Speech Language Pathology-Applied Behavior Analysis, Research in Developmental Disabilities, or the Journal of Autism and Developmental Disorders. Along those lines, another limitation within this review is the lack of information regarding the complexity of the verbal operant tested. This is relevant information, given the heavy focus in the DRR literature on complex language and cognition. Future research may want to focus on coding articles not only in terms of the type of operant derived, but also in terms of its complexity. The results of this review suggest possible areas of further exploration for practitioners. Given that the research on derived mands and tacts both utilized conditional discrimination training procedures (e.g., matching to sample), it suggests this skillset may be a necessary prerequisite skill for more complex language, and thus a necessary focus of behavioral programming. From there, practitioners may want to focus on establishing listener responding skills, as this has been found to aid in the acquisition of derived tacts and intraverbals. In addition, it is likely that practitioners would want to move away from percentage as mastery criterion, and move towards trial block or rolling block mastery; this is especially important when criterions are set at 80% (Iversen, 2016). Future research may want to conduct comparative research between traditional mand training techniques and those described in the derived relational responding literature, regarding the efficiency of each method in establishing a generalized mand repertoire (e.g., trials to criterion). In addition, while there has been research comparing stimulus control transfer techniques on intraverbal acquisition (Ingvarsson & Hollobaugh, 2011; Ingvarsson & Le, 2011), future research may want to compare the efficacy of category tact training, or intraverbal training, versus stimulus control transfer techniques (e.g., prompt and prompt fading) on the acquisition of derived intraverbals. Finally, with the publication of recent assessments and curriculum based on relational frame theory research (e.g., PEAK Relational Training System), practitioners may want to assess language gains made through these curricula versus other verbal behavior curricula via standardized measures, such as the Peabody Picture Vocabulary Test (PPVT), Stanford Binet, or Wechsler Intelligence Scale for Children (WISC). Until then, combining training techniques outlined in the studies included in this review may aid practitioners in establishing a repertoire in which derived responding occurs, thus providing a solid foundation for more complex language.

articles, respectively), with conditional discrimination training procedures most widely used to test for derived manding, and conditional discrimination training procedures or listener response training used to test for derived tacting. The most effective training procedure for derived mands appeared to be conditional discrimination training procedures, whereas multiple exemplar instruction appeared to be the most effect method for derived tacts. This information is relevant to practitioners in the field, specifically, utilizing conditional discrimination training procedures (e.g., match to sample) during mand training in order to promote a repertoire in which mands appear without a history of direct reinforcement. In addition, only one study employed the use of stimulus pairing observation procedure (SPOP; Byrne et al., 2014) with promising results, practitioners may want to continue to research this method of training conditional discriminations and its effects on derived tacts. There was a significant lack of research noted in the area of echoics. Given the echoic is viewed as an important variable in the establishment of emergent listener and speaker repertoires, it is possible the exclusion of articles referencing naming hypothesis resulted in skewed data, eliminating any current research on echoics. However, it is likely this was not the case given recent studies have shown generally limited research on the echoic as the dependent variable (De Souza et al., 2017). Rather, the echoic is typically utilized as part of the training for other verbal operants. This was demonstrated in the current review, where the echoic was referenced as part of the independent variable, for example pairing echoic with listener responding, when the LR alone did not result in the emergence of intraverbals (Smith et al., 2016). Future research may want to analyze the extent to which echoics are a necessary prerequisite in the establishment of derived mands, tacts or intraverbals, or as part of a treatment package including listener response training. Unfortunately, not all articles contained participant prerequisite skills to further analyze what basic repertoire may be needed in order for derived relational responding to occur. Formal standardized language assessments were described in approximately half of the studies (n = 24), while an additional five studies used non-standardized measures (e.g., VBMAPP, ABLLS-R). This is in line with a recent review on the intraverbal in the verbal behavior literature, where the amount of detail regarding participant repertoires varied greatly across articles (Stauch et al., 2017). While focused solely on equivalence relations, O'Donnell and Saunders (2003) described the importance of documenting subjects’ language abilities in order to compare results across studies. As noted here, it is difficult to compare the results of each study when language skills were not described using formal assessments, and participation selection criterion (prerequisite skills) were listed in only 31% of articles (n = 16). Further, practitioners are unable to determine which training procedure would be beneficial to their clients, if variables such as prerequisite skills, or existing verbal behavior repertoires, are excluded from research. Finally, mastery criterion varied from experiment to experiment, with some utilizing a percentage correct, and others utilizing block mastery, with few utilizing rolling block mastery criteria. This phenomenon has been studied recently; Fuller and Fienup (2018) randomly assigned a 50%, 80%, or 90% mastery criterion to five target behaviors and did a parametric analysis on maintenance probes for retention. For all participants, the 90% mastery criterion provided the highest retention on maintenance checks with less variability in performance, whereas, for two participants there was no difference in performance on maintenance checks for skills set at 50% and 80% mastery, but greatest variability in performance observed at the 50% mastery criterion.

135

136

Prerequisites and participant repertoires

Experiment I and II: 6 Spanish speaking children, four males and two females. No language assessments used

Three 7 year olds with a diagnosis of autism and severe language delays. VB-MAPP, Battelle Developmental Inventory, Preschool Language Scale 4th Edition (PLS)

Belloso-Díaz et al. (2016)

Byrne et al. (2014)

Concurrent multiple probe design across participants

Non-concurrent multiplebaseline across-participants used in both experiments

Ten Spanish speaking children Multiple (6–7 years). No language assess- Baseline ment used Design

PretestTen Spanish speaking children (5–6 years). No language assess- posttest ment used

Provided target response within 5 s, required 8/9 correct responses on 9 trial block of LRtact

Provided target response within 5 s, 12 correct consecutive responses

None listed

No prerequisite skills listed for inclusion; repertoires described as mand, tact, and LR repertoires consistent with level I of VBMAPP

Provided target response within 5 s, mastery criterion was 12 correct consecutive responses

Provide target response within 5 s, three consecutive correct responses in some phases; 12 correct consecutive responses required in others

Provided target response within 5 s, with 100% correct responding across 2 sessions on IV probes before reverse IVs were tested

Criterion

None listed except no known learning disability

None listed

Multiple probe Participant repertoires: design across Attending, eye contact, generalparticipants ized vocal imitation, 50–100 tacts, and LR 50–100 in, no problem bx. Two participants described as level III on VBMAPP for mands, tacts, listener, and IV, one described as level II for mands, and level III for remaining operants, and one described as level I and II for operants

Experimental design

Belloso-Díaz et al. (2015b)

Belloso-Díaz et al. (2015a)

Four males with autism (9–18 years).

Allan et al. (2015).

Expressive Vocabulary Test 2nd Ed. (EVT-2) and VB-MAPP

Participants and assessment measures

Author/s

Reliability, procedural integrity, social validity

Operant tested: tact, listener response

Direct training: stimulus pairing observation procedure (SPOP) combined with multiple exemplar instruction (MEI)

Operant tested: symmetrical intraverbals

Operant tested: remaining ABC intraverbals Direct training: Tact training in experiment I, intraverbal training in experiment II

Direct training: randomly assigned to either the train categories, exemplars, and AB/BC intraverbals or the exemplars and AB/BC intraverbals

Operant tested: reverse intraverbal Direct training: Experiment I utilized tact training procedures only, Experiment II combined tact training and intraverbal training Operant tested: intraverbals

Generalization: none

Generalization: none IOA ranged from 98.5% to 100% across participants; PI ranged from 99.5%− 99.9% across participants. No social validity measures

No social validity measures

IOA: 99.8%. PI:100%

No social validity measures Generalization: none

Generalization: none IOA: 99.8%. PI:100%

No social validity measures

IOA= 99.8%. PI= 100%

Nine items derived from Treatment Acceptability Rating Form (TARF) were scored using a 5 pt Likert scale. Social validity averaged between 4.0 and 4.5 Generalization: none

Direct training: IOA= 100%, PI= 100% Intraverbal training consisting of multiple-exemplar training, bidirectional stimulus-response teaching formats, reinforcement, and single-word antecedent verbal stimuli

Operant Trained and Derived

SPOP in conjunction with MEI was shown to be effective in establishing some of the tact and listener relations for the three participants but accuracy on tact was always lower.

Exp I: 5/6 participants demonstrated emergence of probed intraverbals; Exp II: 5/ 6 participants demonstrated emergence of the probed intraverbals.

Teaching the categories together with the exemplars and the AB and BC intraverbals, had a strong influence in the emergence of the probed ABC intraverbals

In experiment I, three of three children demonstrated the emergence of an intraverbal. In experiment II, five of seven demonstrate the emergence of an intraverbal.

Three (of four) participants demonstrated the emergence of reverse intraverbals.

Main findings

C. Raaymakers, et al.

Journal of Contextual Behavioral Science 12 (2019) 128–148

137

Three children with ASD, between 4 and 6 years. Language assessed through the ABLLS

Two makes with ASD between 13 and 15 years. Language assessed through WISC-IV

Dickes et al. (2015)

Dixon et al. (2016)

Three men, 18–20 years, diagnosed with intellectual disabilities. No language assessments listed

De Souza et al. (2013)

Seven typically developing children, 3y10 months-5 y1 month. Language assessed through the Peabody Picture Vocabulary Test-4th Ed.

Thee participants, between 10 and 11 years; two diagnosed with ASD, one diagnosed with ASD and OCD. Skills measured through ABLLS-R

Daar et al. (2015)

Devine et al. (2016)

Nine typically developing 6–7 year olds. No language assessments used

Carp et al. (2012)

No prerequisites listed. Participant repertories: LR repertoire, able to carry conversations (IV repertoire), and could ID and tact letters

No prerequisites listed. Participant prerequisites: difficulty answering wh-questions. Repertories include spoken 4–5 word sentences, independent mands, simple IV fill-ins

None listed

Concurrent multiple probe across participants

Multiple baseline across categories

Participant prerequisites: all participants demonstrated reflexive, symmetric, and some transitive and equivalence relations per a pre-test; repertoires not described

No prerequisites for inclusion listed; participant repertoires: follow multiple-step instructions, generalized echoic and motor imitation repertoires, had at least 300 mands and tacts and 50 or more IVs

Post-test, No prerequisites for inclusion within-subjects listed; participant repertoires design described as PPVT scores from 95 to 134 for four participants, no scores for 3 participants

Multiple baseline

Concurrent multiple baseline across participants and behaviors

Non-concurrent multipleprobe design across participants

Direct training: match-tosample tasks, intraverbaltransfer trials followed each phase.

Operant tested: Intraverbal

Provided target response within 20 s, 100% correct responding for a minimum of five sessions

Operant tested: intraverbal and listener

Operant tested: Intraverbal/reverse intraverbal Direct training: conditional discrimination training

Provided target response within Direct training: 5 s, training-used progressive Intraverbal time delay with mastery criterion for intraverbal training as two consecutive sessions at or above 90%.

Provided target response within 5 s; mastery for tact training was 3 consecutive trial blocks at 100%, passing for IV tests was 87.5% correct

Operant tested: Intraverbal Direct training: Tact

Operant tested: Textual training with compound stimuli

Exp I: dictation taking alone produced high accuracy in vocal spelling for all three participants. Exp II: All participants met criterion for derived stimulus relations after conditional discrimination instruction

Functional relationship demonstrated between training and emergence of IV to novel wh-questions for two of three participants,

Emergence of all novel intraverbals was observed for 4 of the 6 participants, but only after completing both A-B and B-C training.

Generalization: none

IOA data 100% for both participants, no PI data reported, no social validity data.

Generalization: novel stimulus sets participants did not show mastery of relations at baseline, after training A-B relation to five sessions at 100%, all participants demonstrated derived B-A, and A-C relations

3 (of 7) participants demonstrated high accuracy for IVs that had multiple control, 4 with intermediate accuracy, 5 with high accuracy when intraverbal responding required control by only one stimulus element, all participants demonstrated emergence of reverse intraverbal IOA ranged from 99.2% to 100% teaching original intraverbals PI ranged from 99.2% to 100%. was sufficient to produce No social validity data emergence of reverse intraverbals.

IOA ranged from

Generalization: none 99% to 100% for all other participants. PI on intraverbal and tact instruction ranged from 97.2% to 100%; no social validity data Generalization: none

IOA ranged from 97% to 100%; PI ranged from 86% to 96%; no social validity

Generalization: none

No procedural integrity data or social validity data Generalization: none IOA:100%; no procedural integrity, no social validity measures

Direct training: A-B intra- IOA ranged from 94.5% to 100% verbal training, and B-C in- on training, and 99.2–100% on traverbal training to critest trials. terion

Operant tested: Intraverbal Provided target response within Direct training: visual-vi5 s, criterion for mastery was 23 sual conditional discrimiof 25 words (93%) or higher for nation instruction proce3 consecutive sessions dure

Provided target response within 3 s, 90% or above on both RIT and ITT, across two consecutive sessions.

Provided target response within 5 s on training trials; criterion for moving between phases was 3, 6 and 12; test criterion was 12

C. Raaymakers, et al.

Journal of Contextual Behavioral Science 12 (2019) 128–148

138

Eight 5–6 year olds, four in each Multiple baseexperiment. In exp. I, two diag- line probe nosed w/ speech impairment/LI, one w/ autism/MR, one autism/ speech impairment; exp. II all with ASD. Language assessed through Stanford Binet Verbal IQ

Concurrent multiple probe design across participants

Grannan et al. (2012) Two five-year-old males, diagnosed with autism. Language assessed through the PLS-4 and VB-MAPP

Greer et al. (2005)

Multiple baseline design across stimuli

Five children age 5 years, two participants were deaf, no diagnoses for the remaining participants; no language assessments listed

Elias et al. (2013)

Multiple baseline across stimuli

Elias et al. (2008)

Seven participants, between the ages of 21–61 years. Diagnosed with mild to severe MR, epilepsy, CP. Language assessed through the WAIS Test of Nonverbal Intelligence Form A, Hiskey Nebraska Test of Learning Aptitudes, French Pictorial Test of Intelligence

Pretestposttest design replication across stimulus sets and participants

Three children, one with intelMultiple probe lectual disability and speech and across particilanguage impairment, one with pants cognitive delay, one with ASD. Language assessed through the PEAK-DT

Dounavi et al. (2014) Two native Spanish speakers, between the ages of 29–37 years. No diagnoses; language assessed through online test for English language proficiency

Dixon, Belisle, Stanley, et al. (2017)

No prerequisites for inclusion listed; participant repertoires described as beginning writing and reading repertoires; none could spell words in the experiment either written or spoken; no joint stimulus control of spelling written/spoken words

No prerequisites listed; participant repertoires: age equivalent of 3y 6 m to 3y 7 m on PLS-4

Participant prerequisites: chance level of responding for picture to printed word, and printed word–picture relations; selected for being either the oldest two in classroom, or only two deaf participants in class. Participant repertoires not described

Participant prerequisites: none listed; participant repertoires: all participants had a history of sign language from beginner to proficient

Participant prerequisites: high to low level of English proficiency. Participant repertoires not described.

Generalization: none

Operant tested: Intraverbal Direct training: foreign language tact training, native-to-foreign or foreignto-native intraverbal training

Provided target response within 3 s; met a criterion of 90% or better for two successive sessions

Provided target response within 5 s; mastery criterion was 100% across 2 sessions for tact, 89% across two trial blocks for tact category, and 94% across 2 trial blocks for MTS

Operant tested: Intraverbal, transcription (of untaught sets)

Operant tested: Intraverbal Direct training: Intraverbal (vocally spell) or transcription (write the words) in an individually determined set of words

Operant tested: B-A (tact), C-A (textual), BC and CB observing responses Direct training: tacts, followed by category tact instruction and match to sample instruction

Five of seven participants demonstrated emergence, with higher accuracy on the B-D relation for the first group of stimuli trained; lower accuracy demonstrated on subsequently trained stimuli

Tact training and native-toforeign intraverbal (N-F) training resulted in the emergence of IV responses over F-N IV training

Relational training was effective at promoting the emergence of categorical responding, and two of the three participants demonstrated the emergence of additional intraverbal responding without training.

Generalization: none

IOA: 100%. No procedural integrity or social validity data listed.

Generalization: none

IOA:99% for both participants, PI= ranged from 99–100%; no social validity data

Generalization: none

After MEI, all students spelled untaught responses with 80–100% accuracy. The second experiment replicated the first experiment's results.

Both participants emitted multiple correct intraverbal responses during posttest probes

IOA:100%, PI not calculated due All participants showed the to being presented by computer emergence of most signs when program; no social validity data prompted by the presence of the pictures and printed words. The total BA tact responses were higher than correct CA textual responses.

Generalization: none

Reliability data 100%; no social validity data listed

Generalization: none

IOA ranged from 99.7% to 99.8%. No PI data listed, no social validity data listed

IOA:97%; No PI data listed, no social validity data

Direct training: Intraverbal categorical relations

Operant tested: Intraverbal 100% correct responding within Direct training: condi1 session, or 90% across two tional discrimination sessions for A-B and C-B trained training relations; did not hit 100% for tested A-C relation, then retrained A-B and C-B; tested for BD and C-D once at 100% for A-C relation for a signed response to be marked correct had to occur within 10 s from stimulus presentation; Operant tested: Listener response, tact, tact textual Provided target response within Direct training: A-B 5 s, mastery criterion was 100% training (listener response) within 1 session, or 90% across 2 and A-C (LR-text) teaching sessions; required 100% on all relations before testing for derived

Provided target response within 3 s, mastery criterion during training was 30 of 30 correct responses for two consecutive sessions

Participant prerequisites: poor Mastery criterion was a PEAK academic performance, inability score greater than 90 across to correctly ID stimuli based on three consecutive trial blocks category; participant repertoires not described other than PEAKDT scores listed between 42 and 108.

C. Raaymakers, et al.

Journal of Contextual Behavioral Science 12 (2019) 128–148

139

Keintz et al. (2011)

Halvey et al. (2005)

Groskreutz et al. (2010)

Greer et al. (2008) Time lagged multiple probe across participants

Direct training: Prerequisites: did not emit past trained to a minimum of 90% accuracy across 2 sessions for 20 Intraverbal (SEI and MEI) tense for regular or irregular learn units verbs; participant repertoires: participants had established listener, speaker, beginning reader and writer skills, participants varied in social repertoires from participating in back and forth conversation to engaging in parallel and cooperative play;

Two participants age 6, all with a diagnosis of autism; no language assessments used

Pretestposttest

No prerequisites for inclusion listed; participant repertoires: speaking in multiple word sentences or speaking in prompted 2–4 word sentences

Provided target response within 5 s, mastery criterion was two consecutive blocks of 8/9 correct unprompted responses for training

Generalization: none IOA: 100%. No procedural integrity data; no social validity data.

Generalization: none

Operant tested: Tact, textual, IV

IOA: 100%. No procedural integrity data; no social validity data.

Generalization: none

IOA: 98%. Procedural Integrity: 100%; no social validity data listed

IOA: Experiment I: 100% for probe and instructional sessions, Experiment II: 97% for probe sessions and 100% of instructional sessions; PI data: 100% for both. Generalization: tested intervention with preschool aged children in experiment II

Operant tested: Mand, tact Direct training: MTS training of auditory name to coin, coin to printed price, auditory price to printed price.

Operant tested: Intraverbal (across novel sets) Six participants between 8 and PretestPrerequisites for inclusion: gen- Provided target response within Direct training: MTS 14 years, with a diagnosis of posttest eralized identity-matching, his- 5 s, participant responded cortraining of auditory (picautism. Language skills assessed tory of conditional relations with rectly and independently on 8 of ture names), visual stimuli through Peabody Picture visual and auditory sample; par- 9 trials across two consecutive (pictures), and written Vocabulary Test (PPVT) ticipant repertoires: spoke in sessions and two different words. complete sentences or commutherapists. nicated through PECS with 100–200 icons. LR repertoire tested through PPVT with scores between 4y 7 mo-1y 9 mo. Operant tested: Tact, textual Provided target response within Direct training: mand for Three participants, 26–57y, one Multiple probe No prerequisites for inclusion preferred items, 1 item 10 s, request training required across partici- listed; participant repertoires: with severe MR, CP, impulse 8/9 correct for 3 consecutive (9 from each category using ICAP scores between age control disorder, one with severe pants equivalent of 1 y 8 m- 3 y. 11 m; block) trials, conditional discri- the category name (i.e. MR and psychotic d/o NOS, and "fruit") using DTT. Category mination training required crione severe MR, epilepsy, CP. language use described as 1–2 match to sample training word utterances, with or without terion of 16/18 correct reLanguage assessed through (i.e. pear to apple). sponses for at least three manual sign Stanford-Binet Verbal IQ, and consecutive blocks Social & Communicative subsection of ICAP

Experiment I: four between 6 and 7 year olds, Experiment II: three 4 year olds, Diagnosed with language delay, health impairments, developmental delays, autism. Language assessed through Stanford-Binet, VABS Expressive One-Word picture vocabulary test; social repertoire assessed through CABAS Curriculum and Inventory of Repertoires for Children from Pre-School through Kindergarten and First Grade, NY State Standards: K-2.

After training coin and name (AB), coin and price (BC), and price and coin name (AC) 1 participant showed 7 untrained relations; 1 participant showed 4 untrained relations; with the BE (tact) observed in both, and textual in one; the IV (DE, AF) was not derived to criterion in either

All participants demonstrated untrained mands, 2 participants demonstrated the emergence of untrained tacts. 1 participant showed emergent generalization of mands across settings, 1 showed emergent generalization across settings for mands, tacts, and stimulus relations.

For all 6 participants, training resulted in derived tact and textual relations.

Experiment I: all participants required two sets of MEI for mastery criterion to be met for novel regular and irregular verb sets; Experiment II: 2 of the 3 participants required 1 set of MEI, the third required two sets

C. Raaymakers, et al.

Journal of Contextual Behavioral Science 12 (2019) 128–148

140

Three participants age 11–16y, all with a diagnosis of autism. Language assessed through the British Picture Vocabulary ScaleSecond Edition

Three participants age 2–5y, no diagnosis. Language assessed through The British Picture Vocabulary Scale; Expressive One-word Picture Vocabulary Test

May et al. (2013)

May et al. (2016)

Mellor et al. (2015)

Eight participants; 4 in each exp. Exp. I: 5–7y w/ ASD, Exp. II ages 3–5y, typically developing. Language assessed through NY State Curricular Standards, Preschool Inventory of Repertoires for K, and Verbal Capabilities Checklist

Luke et al. (2011)

Provided target response within 5 s, mastery criterion was 2 consecutive sessions at or above 90%; test trials required 2 sessions at or above 80%

No prerequisites for inclusion listed; participant repertoires described as scoring within normal range on BPVS and Expressive one-word picture vocab test

No prerequisites for inclusion listed; participant repertoires described as age equivalent scores on BPVS of 5y 5 m to 9y 7m

On pre-training, required 100% accuracy, on training trials (LR and IV) required 87.5% correct during one session;

On pre-training required 100% accuracy, 10 s response latency; minimum of 87.5% on training trials (10 s latency)

No social validity data Generalization: application to typically developing children, and across frames IOA ranged from 98% to 100%; no procedural integrity data, no social validity data

Direct training: Four training phases: tacting character; tacting what the character eats; mixed tacting name and food; mixed training with reduced feedback. Operant tested: Generalization: none Intraverbal Direct training: LR English IOA ranged from 97% to 99.6%; word; IV, mixed LR and IV no procedural integrity data; no social validity data

Operant tested: Autoclitic

Experiment 1: IOA: 99%; Procedural Integrity: 97%; Experiment 2: IOA: 95%, Procedural Integrity: 99%

Generalization: none

Direct training: LRFFC fill- IOA: 100%. No procedural inin tasks and intraverbal fill- tegrity data; no social validity in tasks data.

Operant tested: Intraverbal, tact, LRFFC Participants were selected due to Provided target response within Direct training: low accuracy in responding on 5 s, mastery criterion for train Experiment 1: MEI across 4 pre-test probes of autoclitic trials was 90% correct responses autoclitic frames; frames with known items; reacross 2 sessions Experiment 2: 2 participertoires were not described pants trained using speaker training only and 2 participants using listener and speaker training

No prerequisites for inclusion listed; participant repertoires: 1500–2000 tacts, and emergent IVs (50) follow multiple step instructions; PPVT scores ranged from 2y 6 m - 2y 8 m

Operant tested: Intraverbal

Generalization: none

Operant tested: Tact items Generalization: none in Welsh, LR when given name in Welsh Four participants between 6 and Multiple probe Prerequisites for inclusion: based Provided target response within Direct training: tact audi- IOA: IV responses: 98–99%; Lip movement: 85–100%; Latency: 7 y, no diagnosis. No language on teacher recommendation; 5–7 s, mastery for ATI was 100% tory stimuli, tact sound 82–99.6%; Instruction sessions: assessments listed participant repertoires not listed across 3 sessions, mastery for 99.5%. Procedural integrity: auditory imaging was 90% 99.7–99.9%; no social validity across three sessions data

Multiple baseline design across stimuli

Non-concurrent multiple baseline across participants

Experiment 1: pre- and posttest Experiment 2: single case, time lagged, multiple probe across participants

Two participants aged 3–4y, all Adapted alterdiagnosed with autism; language nating treatassessed through the PPVT ments design within a concurrent MBD across participants

Kodak et al. (2015)

Intraverbals increased for three of four participants following both tact phases. Auditory imagining instruction was sufficient for 2 participants, and 2 required some direct instruction.

For 2 participants, the emergence of derived relations occurred and maintained across maintenance post-tests; for one participant, one set of words showed derived relations, the other did not

Participants responded at 100% accuracy in the emergent relations test probes except for one student who had 1 error during a post-test probe.

Exp. I: All 4 participants showed criterion-level responding after intervention on both the acquisition probe for the autoclitic frame with known stimuli and on tests for abstraction to mand & tacts. Exp.II: All participants achieved criterion-level responding on the acquisition of autoclitic frames with known stimuli; 3 of 4 achieved criteria on test for abstraction

Untrained repertoire emerged in one participant with IV training. For the second participant, both LRFFC and IV training produced untrained responses, tacting during LRFFC trials increased.

C. Raaymakers, et al.

Journal of Contextual Behavioral Science 12 (2019) 128–148

Three participants aged 4–10y, two typically developing, one diagnosed with severe speech delay. No language assessment listed.

Seven participants between 5 and 9 y, all with a diagnosis of autism. No language assessment listed

Murphy et al. (2005)

Reversal

Withdrawal

Exp. I: three participants 7–11y, Reversal with a diagnosis of autism, Exp. II: 3 participants (1 completed both) with autism between 7 and 11y. Language assessed through the Preschool Inventory of Repertoires for Kindergarten

Murphy et al. (2009a)

Murphy et al. (2009b)

Two males age 6, both with a Concurrent diagnosis of autism. No language multiple baseassessments listed. line design

Provided target response within 10 s, tact, LR, and IV training phases required mastery criterion of 100% across three consecutive trial blocks

Direct training: tact training of exemplars and category, LR, and IV training (providing members of a category)

141

Operant tested: Mands (C1/C2)

Operant tested: Intraverbal Prerequisites for inclusion: based Provided target response within Direct training: Match to on prior exposure to visual 5 s, mastery criterion for each sample (match spoken acschedules; participant reperrelation and mixed training was tivity names to pictures of toires: speak in 3–5 word sentwo consecutive nine-trial blocks the activity, and match tences, tact a variety of stimuli, of 8/9 correct unprompted respoken activity names to limited sight words, able to sit sponses written activity names.) for five minutes Operant tested: Textual No prerequisites for inclusion Required 18 out of 20 correct on Direct training: Mand listed; participant repertoires: pre-experimental procedure be- training of more or less several hundred tacts, mands, fore moving on to experiment; using arbitrary stimuli. and intraverbals all training trials required a Match to sample training of minimum of 18 of 20 correct A1-B1, A2-B2, B1-C1, B2responses across two successive C2. After tested, reversed trial blocks. conditional discrimination training of B1-C2, B2-C1. Operant tested: Mand Direct training: Mand Mand training required 22/25 Prerequisites for inclusion was ability to add and subtract single correct responses for two conse- training of A stimuli, conditional discrimination cutive blocks, conditional disdigit numbers; participant retraining of A to B and B to crimination training required pertoires not included C. Experiment II replicated 22/25 correct responses across experiment I except no two consecutive blocks “smilies” were added or removed Operant tested: Mand No prerequisites for inclusion Mastery for mand training was 1 Direct training: Mand listed; participant repertoires: an block of six correct mand trials training using arbitrary stimuli A1&A2. MTS training established echoic, mand, tact, (18 consecutive mands), for A to B and B to C. Reverse autoclitic and IV conditional discrimination training mastery criterion was training B1-C2 & C2-B1. set at 10/10 correct responses across 1 trial block

Six participants between 3 and 5 Multiple base- No prerequisites for inclusion y, no diagnosis. No language line across sti- listed; participant repertoires not assessments listed muli (word ca- listed tegories)

Miguel et al. (2009)

Miguel et al. (2005)

Generalization: none

Generalization: none IOA: 100%; no procedural integrity data; no social validity data

Generalization: none IOA: 100%, no procedural integrity data; no social validity data

Generalization: none IOA: 100%; No procedural integrity data; no social validity data

IOA: 92–100%. No procedural integrity data; no social validity data

Generalization: none

IOA: 98–99%; Procedural Integrity: 98–100%; no social validity data

When detailed instructions were provided, 6 participants demonstrated derived mands, when replicated with minimal instructions, 3 non-naive and one naive participant demonstrated derived mands. During the reversal, 1 participant demonstrated derived transfer of mand functions across 2 reversals; one participant demonstrated derived transfer across one reversal.

The participants showed 5 derived mands and went on to show 5 reorganized derived mands, but some participants required additional training during tests 2 and 3.

Across the two experiments, participants demonstrated derived transfer of more/less mand functions, derived transfer of reversed more/less mands, and a return to baseline derived more/less mands.

After conditional discrimination training, both participants were able to complete activity schedules based on written words only.

In all cases, direct training produced the most intraverbal responses. Both MTT and RDT resulted in fewer intraverbal responses per probe, regardless of the order in which the conditions were presented.

C. Raaymakers, et al.

Journal of Contextual Behavioral Science 12 (2019) 128–148

Three participants, 14y, all with Posttest a diagnosis of autism. No language assessments listed.

One participant age 5 y 9 mo, diagnosed with autism and MMR, language assessed through WPPSI

Murphy et al. (2010b)

Noro (2005)

142 Provided target response within No prerequisites for inclusion 5 s, requirements was 12 conselisted; participant repertoires: cutive correct responses generalized echoic, 300 tacts, mands, social IV's and had some autoclitic frames, could read, write; scores on PPVT ranged from 2.92 to 6y.

Two children age 6–8y, diagnosed with Pervasive Developmental Disorder, language assessed through the PPVT

Eight children, 5–6y of age; no diagnoses. No language assessments listed.

Pérez-González et al. (2008)

Baseline instruction criterion was 100% correct for one 15learn unit; MEI criterion was 100% correct responding for each operant for one session (15 trials)

Pérez-González et al. (2007)

Generalization: none IOA: 100%; No procedural integrity data; no social validity data

Operant tested: Mands Direct training: Mand training with arbitrary stimuli (A1-A5)

Operant tested: Mands, tacts Direct training: Intraverbal and reverse intraverbal

Operant tested: Intraverbal (oral naming of emotion when given scenario) Direct training: MEI across mand and tact conditions for a subset of adjectiveobject pairs.

Operant tested: Intraverbal and reverse intraverbal Multiple probe No prerequisites for inclusion Exp. I: provided target response Direct training: across subjects listed; participant repertoires not within 2 s; 3–6 consecutive cor- Intraverbal training in described. rect responses. Final phase reExperiment I; IV category quired 12 consecutive correct training in Experiment II; responses; Exp. II and III similar Experiment III: replication in set up of two Operant tested: Intraverbal

Multiple baseline across participants

IOA: 95%, no procedural integrity data; no social validity data

Direct training: Mand training with arbitrary stimuli, then trained on A-E and reorganized A-E relations.

Generalization: replication within subjects

IOA: 99.32%; no procedural integrity data; no social validity data.

Generalization: none

IOA: 99.8%; no procedural integrity data; no social validity data

Generalization: none

IOA: range from 98%− 100%; no procedural integrity data; no social validity data

Generalization: none

Match to sample training (A Generalization: none to B, then B to C). Operant tested: Mands Delivered reinforcer on a FR3 Direct training: MTS IOA: 98%; No procedural integschedule, error correction intraining to teach A-B rela- rity data; no social validity data volved verbal feedback; criterion tion and C-A relation was 80% correct across 2 consecutive trial blocks

Mastery for mand training was 22/25 correct responses for 2 consecutive blocks, same criterion for conditional discrimination training

Minimum of 26/30 trials correct, across 2 consecutive trial blocks, with no errors on last 15 trials

No prerequisites for inclusion listed; participant repertoires: manding with frame "I want" and tact with frame "It's a"

No prerequisites for inclusion listed; participant repertoire: verbal IQ 57, Performance IQ 94, full scale IQ 69, could read Japanese hiragana characters, follow 1 step instructions, and communicate in 2–3 word sentences

No prerequisites for inclusion listed; participant repertoires: several hundred mands, tacts, and IV's, extensive MTS experience

No prerequisites for inclusion listed; participant repertoires: several hundred mands, tacts, and IVs, no fluency in basic math

Nuzzolo-Gomez et al. Four participants ages 6–9y, two Multiple probe (2004) with a diagnosis of autism, one across participants with speech impairment, one with MR. No language assessments listed.

Pre-test, Posttest

Seven participants between the Reversal ages of 5–14y, three participants without any diagnoses, four participants diagnosed with autism. No language assessments listed.

Murphy et al. (2010a)

8 children who participated in Exp. I and II showed emergence of novel intraverbals. Three children demonstrated novel intraverbals with additional stimuli sets.

Novel intraverbals did not emerge in any participant until intraverbal and reverse intraverbal were directly trained

After MEI, the untaught mand or tact verbal operants for adjective-object pairs emerged.

After training A-B and C-A relations, B-C relations emerged, but not on the first test. Tests in which the participant was asked to respond intraverbally when presented with scenarios, participant scored above 80% across all three probes.

Each participant demonstrated derived mands by using the C stimuli to mand, but one participant required ME training

Testing showed derived transformation of functions for 7 participants; 2 required exemplar training.

C. Raaymakers, et al.

Journal of Contextual Behavioral Science 12 (2019) 128–148

Five Spanish speaking children 3y, no diagnoses. No language assessments listed.

Two children, 5y, no diagnoses. No language assessments listed.

Pérez-González et al. (2016)

Petursdottir et al. (2009)

Multiple baseline design across participants, and adapted alternating treatments design

PretestPosttest design, with multiple probes

Multiple baseline across participants

143

Ribeiro et al. (2010)

Two participants 16–20 y, diagnosed with intellectual disabilities; language assessed through the WAIS and Columbia Mental Maturity Scale

Multiple probe Prerequisites for inclusion: vocal across stimuli repertoires that were unintelligible; participant repertoires: no prior experience with Brazilian sign language, had fine motor abilities; prior experience with MTS tasks; IQ from 43 to 52; language ranged from no functional language to two word utterances

Operant tested: Mands, tacts

Operant tested: Mands Mastery criterion established for Direct training: Listener each stimulus set was 100% of responding correct selections in a block during LR training

Operant tested: tact and intraverbal First 3 phases of PECS training to Direct training: Mands, listed criterion; conditional dis- conditional discrimination crimination training: criterion training was 8 out of 9 trials correct

Provided target response within 5 s, 3 or 12 consecutive responses based on phase, exemplar training: ranged from 4 or 12 consecutive correct responses based on phase

Direct training: Exp. I: A-B and B-C Intraverbals Exp. II: A-B and B-C intraverbals and categories. Exp. III: teaching exemplars prior to A-B and B-C intraverbals Operant tested: Intraverbals No prerequisites for inclusion Tact training criterion: 3–12 Direct training: Tact listed, participant repertoires not consecutive correct trials; audio- training; Auditory-Visual described. visual CD training: 3–16 conse- conditional discrimination cutive correct trials. Auditorytraining, Auditory-Auditory Auditory training 3–6 consecuconditional discrimination tive correct responses, or 14 or training; MEI more correct responses. Auditory-Auditory CD with tact training: 30/32 trials correct Operant tested: Intraverbals Perquisites for inclusion: each All training continued until Direct training: Listener participant could tact each stithere were 10 of 12 correct training, tact training, and mulus in their native language responses on each set in two native foreign intraverbal and vocally imitate its name in consecutive sessions training the target foreign language. Participant repertoires: no known language or developmental delay, spoke Icelandic

Prerequisites for inclusion: no more than one question answered correctly on pretest probes; participant repertoires not described

Rehfeldt et al. (2005) Three adults, 20–34y, diagnosed Multiple probe No prerequisites for inclusion with MR; no language assessacross partici- listed; participant repertoires: ments listed pants limited to no functional language; participants were on medication

Eleven Spanish speaking adults; age not listed; no diagnoses. No language assessments listed.

Pérez-González et al. (2014)

Generalization: none

Generalization: none IOA: 100%; no procedural integrity data; no social validity data

IOA: 98%; no procedural integrity data, no social validity data

Generalization: none

IOA: 100%; no procedural integrity data taken; no social validity data

Generalization: none

IOA: 99.21%; no procedural data; no social validity data

Generalization: none

IOA: 99.32%; no procedural integrity data; no social validity data

For both participants, training relations between signs presented through video and pictures produced emergent signed tacts and mands.

Emergence of derived requesting skills in individuals with severe developmental disabilities was observed.

Listener training resulted in increases in correct tact and intraverbal responding for both participants but not to posttest passing criterion; training of a single relation did not result in the emergence of all possible untrained relations to mastery criterion

Four children demonstrated the emergence of all the intraverbals, the fifth child demonstrated the emergence of three of the four intraverbals

Ten out of the eleven participants showed the emergence of the probed ABC intraverbals.

C. Raaymakers, et al.

Journal of Contextual Behavioral Science 12 (2019) 128–148

Five participants, age 6–15y, all diagnosed with autism. Language assessed through the VABS

Smith et al. (2016)

144

Sprinkle et al. (2012) Four participants between 5 y, 11 mo.- 7 y 9 mo. with a diagnosis of autism. No language assessments listed

Exp. I: three children, 4–5y, two with a language delay, one typically developing. Exp. II: three children, 3–4y, one with a language delay, two typically developing. Language assessed through the REEL, Rosetti Infant Toddler Language Scale, or CELF- Preschool

Six participants, age 4–8y, five with a diagnosis of autism, one with a diagnosis of PDD-NOS. Language assessed through the VB-MAPP

Shillingsburg et al. (2017)

Speckman et al. (2012)

Four Spanish speaking children age 3y, no diagnoses. No language assessments listed.

Rosales et al. (2011)

Alternating treatments design

Exp. I: delayed multiple-probe design; Exp. II: non-concurrent multiple probe design

Non-concurrent multiple baseline across participants

Multiple probe across behavior

Multiple probe across stimulus sets

Two adults, 34–58y; one with Multiple probe severe MR, one with anxiety across particidisorder and psychotic disorder pants NOS. Language assessed through the social and communicative subsection of the ICAP

Rosales et al. (2007)

No prerequisites for inclusion listed; participant repertoires: echoic and match to sample repertoires, and some tacting skills. Two participants had a beginning textual repertoire

No prerequisites for inclusion listed; participant repertoires not described other than having a language delay

Operant tested: tact, textual

Operant tested: Tact with autoclitic frame Utilized a progressive prompt Direct training: listener delay, once at 2 consecutive training of A-B and A-C sessions with a minimum of 8/9 relation, tact training, texcorrect, prompt was reduced tual

Exp. I: SEI training; 2 s to respond; Twenty-trial sessions with mastery criterion of 90% for two consecutive sessions or 100% for one session. Exp. II: SEI and MEI, Criterion for mastery for each set was 90% for one session or 10 consecutive correct responses

Operant tested: Intraverbal Direct training: Exp. I: tact training with single exemplar and ME training; Exp. II: replication of experiment I with change of abstraction of autoclitic.

Mastery criterion for mand Direct training: Mands, training: 8 out 9 consecutive conditional discrimination correct responses per trial block; conditional discrimination training had a mastery criterion of 89% correct Operant tested: Mands No prerequisites for inclusion LR training mastery criterion: 7/ Direct training: ME lislisted; no description of partici- 8 (88%) correct responses over 2 tener response training and pant repertoires listed other than consecutive trial blocks; ME tact training all Spanish speaking training: 3 consecutive correct responses. Operant tested: Tacts No prerequisites for inclusion Training trials required 3 s reDirect training: LRFFC listed; participant repertoires: sponding to instruction; test training, tact function VBMAPP scores between 50.5 probes were 5 s; training trials training, IV training if and 102.5; IV scores between 0 required 100% correct reneeded and 7, and tact scores between 6 sponding across 2 sessions. though 13.5, LRFFC scores between 0.5 and 6.5 Operant tested: Intraverbals No prerequisites for inclusion LR training required correct re- Direct training: Listener listed; participant repertoires: sponse within 3 s; 90% correct response training; LR and 20–75 + mands, 50–250 tacts, responding in one 10 trial block echoic if needed, or LR and and 50–75 + IVs; scores on the tact training VABS between 59 and 89

Prerequisites for inclusion: lack of functional communication; participant repertoires: unknown IQ to IQ 36; ICAP equivalent scores of 1y 8 m, to 3y 11 m.

Generalization: none

IOA averaged 98%, range of 97–99% across participants; procedural integrity data collected averaged 96.7%, with a range of 95–98% across participants; no social validity data

Generalization: none

IOA Exp. I and II: 100%; Procedural integrity: Exp. I and II: 100%; no social validity data

Generalization: none

IOA: 98.9%; no procedural integrity data; no social validity data

Generalization: none

Generalization: novel stimuli IOA: 94.5%− 98.9%; no procedural integrity data; no social validity data

Generalization: none IOA: 96.89%− 99.86%; Procedural Integrity data: 99.82%− 99.84%; no social validity data

IOA: 97–100%; no procedural integrity data; no social validity data

Listener training demonstrated derived speaker responding in 3 of the 4 participants, and emergence of picture-word matching for 3 of 4 participants; speaker training resulted in the emergence of listener skills and picture-word matching in all participants

Untaught tact responses containing the autoclitic frame emerged following MEI; this was not apparent after SEI; when tested on a novel set, all participants showed 90% or greater accuracy in responding

Four (out of five) participants demonstrated the emergence of intraverbal responses following listener training. One participant required additional tact training

All participants demonstrated emergence of untrained intraverbal relations. Four (of six) demonstrated some degree of emergence of untrained intraverbals in sets that hadn’t received any training.

All participants demonstrated emergence of derived tact relations following MET; 2 of the 4 met all derived relations to mastery criterion.

Conditional discrimination training can lead to the emergence of derived manding

C. Raaymakers, et al.

Journal of Contextual Behavioral Science 12 (2019) 128–148

Ziomek et al. (2008)

Zaine et al. (2014)

Prerequisites for participation: limited history with systematic instruction; participant repertoires: IQ of 24, Stanford Binet mental age of 24 m- 2y 8 m, ICAP adaptive behavior scores between 2y 5 m and 3y 11 m, ICAP communication of 1y 8 m1y 10 m

Prerequisites for participation: history of difficulty reading. Participant repertoires: not listed

Fourteen participants between 9 Between subjects design and 15 years, diagnosed with mild to moderate intellectual disabilities. Language assessed through the WISC-III

Three participants, 42–52y, one Alternating diagnosed with mild MR, mood treatments deswings, depression, intermittent sign explosive disorder, and blindness. One diagnosed with profound MR, intermittent explosive disorder, seizure disorder, and obesity, and one diagnosed with mild MR, mood swings, and depression. Language assessed via the ICAP

No prerequisites for participation listed; participant repertoires: level 2 of VBMAPP for IVs, and mand and tact repertoires in Level 2 and 3

No prerequisites for inclusion listed; participant repertoires: BPVS scores on receptive and expressive language abilities lower than chronological age

Non-concurrent multiple probe across participants and adapted alternating treatments design

Exp. II: Multiple probe, multiple baseline design

Exp. I: 8 children, 3–12y, Exp. II: Exp. I: pretestthree children, 4–12y; all diagposttest nosed with autism. Language assessed through the BPVS-III and Expressive Vocabulary Test2nd Ed.

Vallinger-Brown et al. Three participants, 4–7y, all di(2014) agnosed with ASD. Language assessed through the VB-MAPP

Still et al. (2015)

Operant tested: Mands Direct training: Listener response training, stimulus pairing

145

Operant tested: Tacts, intraverbals

Operant tested: textual Trained first 3 phases of PECS to Direct training: PECS training, manual sign mastery criterion outlined in training manual; mastery for manual mands (sign) was 100% accuracy per trial block, for transitive EO mand training, mastery criterion was 100% accuracy for two consecutive trial blocks.

Generalization: novel settings and people

Generalization: none IOA: 100%; no procedural integrity data; no social validity data

IOA data collected on intraverbal responses only, IOA 100% for all participants; no procedural integrity taken as presentation of stimuli was computerized; no social validity.

Generalization: none

Generalization: none IOA: 96.5–98.8%; procedural integrity: 99.6%; no social validity data

Experiment II: 100%; no procedural integrity data; no social validity data

Direct training: Exp. I: IOA Experiment I: 97–100% Mand training and conditional discrimination training Exp. II: conditional discrimination training

Operant tested: Intraverbal Direct training: simple Participants were required to respond with 100% accuracy on and conditional discrimination training six A-B trials (match name to picture) before training would begin. Criterion to move across phases was no more than 1 incorrect response within each session, across 2 consecutive sessions

Criterion was up to 10 s to respond; tact training of stimuli: 100% for one complete 12 trial block; listener training: 11/12 correct

Criterion was up to 10 s to respond; mand training: 75% independent manding in one block; conditional discrimination training 75–100% correct responding across 2 blocks of 16 trials

Mands established using PECS generalized across settings and communicative partners. Untrained tacts and intraverbals using PECS were shown to emerge for some of the participants following PECS training.

Both groups had high accuracy with tacting, matching picture to picture, and listener responding tests at baseline, but scored low on LR dictated word (textual). At post-test the percent correct did not change for the control group, but increased for the experimental group. Experimental group scored higher on C-B, and B-C tests and all participants in control correctly read (C-D relation) at least one word on post-test, with 50% correctly reading at least half of the words

Both procedures were effective at producing some untaught intraverbal responses, but a transfer of stimulus control procedure was required for two of the three participants

Across both experiments, all but one participant demonstrated derived requesting.

C. Raaymakers, et al.

Journal of Contextual Behavioral Science 12 (2019) 128–148

Journal of Contextual Behavioral Science 12 (2019) 128–148

C. Raaymakers, et al.

Conflicts of interest

equivalence relations of geometry skills in Students with autism: An application of the PEAK-E curriculum. The Analysis of Verbal Behavior, 32(1), 38–45. https://doi. org/10.1007/s40616-016-0051-9. Dixon, M. R., Belisle, J., Stanley, C. R., Speelman, R. C., Rowsey, K. E., Kime, D., & Daar, J. H. (2017). Establishing derived categorical responding in children with disabilities using the PEAK-E curriculum. Journal of Applied Behavior Analysis, 50(1), 134–145. https://doi.org/10.1002/jaba.355. Dixon, M. R., Belisle, J., McKeel, A., Whiting, S., Speelman, R., Daar, J. H., & Rowsey, K. (2017). An internal and critical review of the PEAK relational training system for children with autism and related intellectual disabilities: 2014–2017. The Behavior Analyst, 40(2), 493–521. https://doi.org/10.1007/s40614-017-0119-4. Dixon, M. R., Belisle, J., Rehfeldt, R. A., & Root, W. B. (2018). Why we are still not acting to save the world: The upward challenge of a post-Skinnerian behavior science. Perspectives on Behavior Science, 41, 241–267. https://doi.org/10.1007/s40614-0180162-9. Dixon, M. R., Belisle, J., Stanley, C. R., & Rowsey, K. (2018). Student outcomes after 1 year of front line staff implementation of the PEAK curriculum. Behavioral Interventions, 33(2), 185–195. https://doi.org/10.1002/bin.1516. Dymond, S., & Alonso-Alvarez, B. (2010). The selective impact of Skinner's verbal behavior on empirical research: A reply to Schlinger (2008). Psychological Record, 60, 355–360. https://doi.org/10.1007/BF03395712. Dounavi, K. (2014). Tact training versus bidirectional intraverbal training in teaching a foreign language. Journal of Applied Behavior Analysis, 47(1), 165–170. https://doi. org/10.1002/jaba.86. Elias, N. C., Goyos, C., Saunders, M., & Saunders, R. (2008). Teaching manual signs to adults with mental retardation using matching-to-sample procedures and stimulus equivalence. Analysis of Verbal Behavior, 24, 1–13. https://doi.org/10.1007/ bf03393053. Elias, N. C., & Goyos, C. (2013). Mimetic relation as matching-to-sample observing response and the emergence of speaker relations in children with and without hearing impairments. The Psychological Record, 63(1), 131–140. https://doi.org/10.11133/j. tpr.2013.63.1.010. Feng, H., Chou, W. C., & Lee, G. T. (2015). Effects of tact prompts on acquisition and maintenance of divergent intraverbal responses by a child with autism. Focus on Autism and Other Developmental Disabilities, 32, 133–141. https://doi.org/10.1177/ 1088357615610540. Fienup, D. M., & Brodsky, J. (2017). Effects of mastery criterion on the emergence of derived equivalence relations. Journal of Applied Behavior Analysis, 50(4), 843–848. https://doi.org/10.1002/jaba.416. Fisher, W. W., & Lerman, D. C. (2014). It has been said that, “There are three degrees of falsehoods: Lies, damn lies, and statistics”. Journal of School Psychology, 52(2), 243–248. https://doi.org/10.1016/j.jsp.2014.01.001. Fuller, J. L., & Fienup, D. M. (2018). A preliminary analysis of mastery criterion level: Effects on response maintenance. Behavior Analysis in Practice, 11, 1–8. https://doi. org/10.1007/s40617-017-0201-0. Gamba, J., Goyos, C., & Petursdottir, A. I. (2015). The functional independence of mands and tacts: Has it been demonstrated empirically? The Analysis of Verbal Behavior, 31, 10–38. https://doi.org/10.1007/s40616-014-0026-7. Grannan, L., & Rehfeldt, R. A. (2012). Emergent intraverbal responses VIA tact and match-to-sample instruction. Journal of Applied Behavior Analysis, 45(3), 601–605. https://doi.org/10.1901/jaba.2012.45-601. Greer, R. D., Yaun, L., & Gautreaux, G. (2005). Novel dictation and intraverbal responses as a function of a multiple exemplar instructional history. The Analysis of Verbal Behavior, 21(1), 99–116. https://doi.org/10.1007/BF03393012. Greer, R. D., & Yuan, L. (2008). How kids learn to say the darnedest things: The effect of multiple exemplar instruction on the emergence of novel verb usage. The Analysis of Verbal Behavior, 24(1), 103–121. https://doi.org/10.1007/bf03393060. Groskreutz, N. C., Karsina, A., Miguel, C. F., & Groskreutz, M. P. (2010). Using complex auditory—visual samples to produce emergent relations in children with autism. Journal of Applied Behavior Analysis, 43(1), 131–136. https://doi.org/10.1901/jaba. 2010.43-131. Grow, L. L., & Kodak, T. (2010). Recent research on emergent verbal behavior: Clinical applications and future directions. Journal of Applied Behavior Analysis, 43, 775–778. https://doi.org/10.1901/jaba.2010.43-775. Halvey, C., & Rehfeldt, R. A. (2005). Expanding vocal requesting repertoires via relational responding in adults with severe developmental disabilities. The Analysis of Verbal Behavior, 21(1), 13–25. https://doi.org/10.1007/BF03393007. Hayes, S. C., Barnes-Holmes, D., & Roche, B. (2001). Relational frame theory: A postSkinnerian account of human language and cognition. New York: Kluwer Academic. Horne, P. J., & Lowe, C. F. (1996). On the origins of naming and other symbolic behavior. Journal of the Experimental Analysis of Behavior, 65, 185–241. https://doi.org/10. 1901/jeab.1996.65-185. Howard, H. A., Ladew, P., & Pollack, E. G. (2009). The National Autism Center's National Standards Project “Findings and Conclusions”. Randolph, MA: National Autism Center. Ingvarsson, E. T., & Hollobaugh, T. (2011). A comparison of prompting tactics to establish intraverbals in children with autism. Journal of Applied Behavior Analysis, 44, 659–664. https://doi.org/10.1901/jaba.2011.44-659. Ingvarsson, E. T., & Le, D. D. (2011). Further evaluation of prompting tactics for establishing intraverbal responding in children with autism. The Analysis of Verbal Behavior, 27, 75–93. https://doi.org/10.1007/BF03393093. Iversen, I. H. (2016). Problems with “percent correct” in conditional discrimination tasks. European Journal of Behavior Analysis. https://doi.org/10.1080/15021149.2016. 1139368. Keintz, K. S., Miguel, C. F., Kao, B., & Finn, H. E. (2011). Using conditional discrimination training to produce emergent relations between coins and their values in children with autism. Journal of Applied Behavior Analysis, 44(4), 909–913. https://doi.org/10.

None. Funding This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. References Aguirre, A. A., Valentino, A. L., & LeBlanc, L. A. (2016). Empirical investigations of the intraverbal: 2005–2015. The Analysis of Verbal Behavior, 32, 139–153. https://doi. org/10.1007/s40616-016-0064-4. Allan, A. C., Vladescu, J. C., Kisamore, A. N., Reeve, S. A., & Sidener, T. M. (2015). Evaluating the emergence of reverse intraverbals in children with autism. The Analysis of Verbal Behavior, 31(1), 59–75. https://doi.org/10.1007/s40616-0140025-8. Barnes-Holmes, D., Barnes-Holmes, Y., & Cullinan, V. (2000). Relational frame theory and Skinner's Verbal Behavior: A possible synthesis. The Behavior Analyst, 23, 69–84. https://doi.org/10.1007/BF03392000. Belloso-Díaz, C., & Pérez-González, L. A. (2015aa). Effect of learning tacts or tacts and intraverbals on the emergence of intraverbals about verbal categorization. The Psychological Record, 65(4), 749–760. https://doi.org/10.1007/s40732-015-0145-0. Belloso-Díaz, C., & Pérez-González, L. A. (2015bb). Exemplars and categories necessary for the emergence of intraverbals about transitive reasoning in typically developing children. The Psychological Record, 65(3), 541–556. https://doi.org/10.1007/s40732015-0131-6. Belloso-Díaz, C., & Pérez-González, L. A. (2016). Emergence of symmetrical intraverbals facilitated by learning skills with the intraverbal responses. The Psychological Record, 66(2), 269–281. https://doi.org/10.1007/s40732-016-0169-0. Bloh, C. (2008). Assessing transfer of stimulus control procedures across learners with autism. The Analysis of Verbal Behavior, 24(1), 87–101. https://doi.org/10.1007/ BF03393059. Bondy, A., Esch, B. E., Esch, J. W., & Sundberg, M. (2010). Questions on verbal behavior and its application to individuals with autism: An interview with the experts. The Behavior Analyst Today, 11(3), 186–205. https://doi.org/10.1037/h0100700. Burning Glass Technologies (2015). US Behavior Analyst Workforce: Understanding the National Demand for Behavior Analysts. Retrieved from 〈http://bacb.com/wpcontent/uploads/2015/10/151009-burning-glass-report.pdf〉. Byrne, B. L., Rehfeldt, R. A., & Aguirre, A. A. (2014). Evaluating the effectiveness of the stimulus pairing observation procedure and multiple exemplar instruction on tact and listener responses in children with autism. The Analysis of Verbal Behavior, 30(2), 160–169. https://doi.org/10.1007/s40616-014-0020-0. Carp, C. L., & Petursdottir, A. I. (2012). Effects of two training conditions on the emergence of novel intraverbals: An extension of Pérez-González et al. (2008). The Psychological Record, 62(2), 187. https://doi.org/10.1007/BF03395797. Carr, J. E., Austin, J. L., Britton, L. N., Kellum, K. K., & Bailey, J. S. (1999). An assessment of social validity trends in applied behavior analysis. Behavioral Interventions, 14(4), 223–231. https://doi.org/10.1002/(sici)1099-078x(199910/12)14:4<223::aidbin37>3.0.co;2-y. Cassidy, S., Roche, B., Colbert, D., Stewart, I., & Grey, I. M. (2016). A relational frame skills training intervention to increase general intelligence and scholastic aptitude. Learning and Individual Differences, 47, 222–235. https://doi.org/10.1016/j.lindif. 2016.03.001. Carr, J. E., & Firth, A. M. (2005). The verbal behavior approach to early and intensive behavioral intervention for autism: A call for additional empirical support. Journal of Early and Intensive Behavior Intervention, 2(1), 18–27. https://doi.org/10.1037/ h0100297. Colbert, D., Tyndall, I., Roche, B., & Cassidy, S. (2018). Can SMART training really increase intelligence? A replication study. Journal of Behavioral Education, 1–23. Cullinan, V., & Vitale, A. (2008). The contribution of Relational Frame Theory to the development of interventions for impairments of language and cognition. The Journal of Speech and Language Pathology-Applied Behavior Analysis, 3, 122–135. https://doi. org/10.1037/h0100254. Daar, J. H., Negrelli, S., & Dixon, M. R. (2015). Derived emergence of WH question–answers in children with autism. Research in Autism Spectrum Disorders, 19, 59–71. https://doi.org/10.1016/j.rasd.2015.06.004. De Souza, A. A., Akers, J. S., & Fisher, W. W. (2017). Empirical application of Skinner's verbal behavior to interventions for children with autism: A Review. The Analysis of Verbal Behavior, 33(2), 229–259. https://doi.org/10.1007/s40616-017-0093-7. De Souza, A. A., & Rehfeldt, R. A. (2013). Effects of dictation‐taking and match‐to‐sample training on listing and spelling responses in adults with intellectual disabilities. Journal of Applied Behavior Analysis, 46(4), 792–804. https://doi.org/10.1002/ jaba.75. Devine, B., Carp, C. L., Hiett, K. A., & Petursdottir, A. I. (2016). Emergence of intraverbal responding following tact instruction with compound stimuli. Analysis of Verbal Behavior, 32(2), 154–170. https://doi.org/10.1007/s40616-016-0062-6. Dickes, N. R., & Kodak, T. (2015). Evaluating the emergence of reverse intraverbals following intraverbal training in young children with autism spectrum disorder. Behavioral Interventions, 30, 169–190. https://doi.org/10.1002/bin.1412. Dixon, M. R., Belisle, J., Stanley, C. R., Daar, J. H., & Williams, L. A. (2016). Derived

146

Journal of Contextual Behavioral Science 12 (2019) 128–148

C. Raaymakers, et al. 1901/jaba.2011.44-909. Kodak, T., & Paden, A. R. (2015). A comparison of intraverbal and listener training for children with autism spectrum disorder. The Analysis of Verbal Behavior, 31(1), 137–144. https://doi.org/10.1007/s40616-015-0033-3. Lee, G. T., Chou, W. C., & Feng, H. (2017). Using intraverbal prompts to increase divergent intraverbal responses by a child with autism. Behavioral Interventions, 32(4), 434–444. https://doi.org/10.1002/bin.1496. Lovaas, O. I. (1987). Behavioral treatment and normal educational and intellectual functioning in young autistic children. Journal of Consulting and Clinical Psychology, 55, 3–9. https://doi.org/10.1037/0022-006X.55.1.3. Luke, N., Greer, R. D., Singer-Dudek, J., & Keohane, D. D. (2011). The emergence of autoclitic frames in atypically and typically developing children as a function of multiple exemplar instruction. The Analysis of Verbal Behavior, 27(1), 141–156. https://doi.org/10.1007/BF03393098. Kates-McElrath, K., & Axelrod, S. (2006). Behavioral intervention for autism: A distinction between two behavior analytic approaches. The Behavior Analyst Today, 7, 242–252. https://doi.org/10.1037/h0100085. Manolov, R., & Moeyaert, M. (2017). Recommendations for choosing single-case data analytical techniques. Behavior Therapy, 48(1), 97–114. https://doi.org/10.1016/j. beth.2016.04.008. May, R. J., Hawkins, E., & Dymond, S. (2013). Brief report: Effects of tact training on emergent intraverbal vocal responses in adolescents with autism. Journal of Autism and Developmental Disorders, 43, 996–1004. https://doi.org/10.1007/s10803-0121632-7. May, R. J., Downs, R., Marchant, A., & Dymond, S. (2016). Emergent verbal behavior in preschool children learning a second language. Journal of Applied Behavior Analysis, 49(3), 711–716. https://doi.org/10.1002/jaba.301. McKeel, A. N., Rowsey, K. E., Belisle, J., Dixon, M. R., & Szekely, S. (2015). Teaching complex verbal operants with the PEAK relational training system. Behavior Analysis in Practice, 8(2), 241–244. https://doi.org/10.1007/s40617-015-0067-y. McLay, L. K., Sutherland, D., Church, J., & Tyler-Merrick, G. (2013). The formation of equivalence classes in individuals with autism spectrum disorder: A review of the literature. Research in Autism Spectrum Disorders, 7, 418–431. https://doi.org/10. 1016/j.rasd.2012.11.002. McHugh, L., & Reed, P. (2008). Using relational frame theory to build grammar in children with autistic spectrum conditions. The Journal of Speech-Language Pathology and Applied Behavior Analysis, 3, 60–77. https://doi.org/10.1037/h0100233. Mellor, J. R., Barnes, C. S., & Rehfeldt, R. A. (2015). The effects of auditory tact and auditory imagining instructions on the emergence of novel intraverbals. The Analysis of Verbal Behavior, 31(2), 236–254. https://doi.org/10.1007/s40616-015-0036-0. Miguel, C. F., Petursdottir, A. I., & Carr, J. E. (2005). The effects of multiple-tact and receptive-discrimination training on the acquisition of intraverbal behavior. The Analysis of Verbal Behavior, 21(1), 27–41. https://doi.org/10.1007/BF03393008. Miguel, C. F., Petursdottir, A. I., Carr, J. E., & Michael, J. (2008). The role of naming in stimulus categorization by preschool children. Journal of the Experimental Analysis of Behavior, 89(3), 383–405. Miguel, C. F., Yang, H. G., Finn, H. E., & Ahearn, W. H. (2009). Establishing derived textual control in activity schedules with children with autism. Journal of Applied Behavior Analysis, 42(3), 703–709. https://doi.org/10.1901/jaba.2009.42-703. Murphy, C., Barnes-Holmes, D., & Barnes-Holmes, Y. (2005). Derived mand training in children with autism: Synthesizing Skinner's verbal behavior with relational frame theory. Journal of Applied Behavior Analysis, 38(4), 445–462. https://doi.org/10. 1901/jaba.2005.97-04. Murphy, C., & Barnes-Holmes, D. (2009aa). Derived more/less relational mands with children diagnosed with autism. Journal of Applied Behavior Analysis, 42, 252–268. https://doi.org/10.1901/jaba.2009.42-253. Murphy, C., & Barnes-Holmes, D. (2009bb). Establishing derived manding for specific amounts with three children: An attempt at synthesizing Skinner's Verbal Behavior with relational frame theory. The Psychological Record, 59(1), 75–91. https://doi.org/ 10.1007/BF03395650. Murphy, C., & Barnes-Holmes, D. (2010a). Establishing complex derived manding with children with and without a diagnosis of autism. The Psychological Record, 60, 489–503. https://doi.org/10.1007/BF03395650. Murphy, C., & Barnes‐Holmes, D. (2010b). Establishing five derived mands in three adolescent boys with autism. Journal of Applied Behavior Analysis, 43(3), 537–541. https://doi.org/10.1901/jaba.2010.43-537. Ming, S., Moran, L., & Stewart, I. (2014). Derived relational responding and generative language: Applications and future directions for teaching individuals with autism spectrum disorders. European Journal of Behavior Analysis, 15, 199–224. https://doi. org/10.1080/15021149.2014.11434722. Najdowski, A. C., Bergstrom, R., Tarbox, J., & Clair, M. S. (2017). Teaching children with autism to respond to disguised mands. Journal of Applied Behavior Analysis, 50, 733–743. https://doi.org/10.1002/jaba.413. Noro, F. (2005). Using stimulus equivalence procedures to teach receptive emotional labeling to a child with autistic disorder. Japanese Journal of Special Education, 42(6), 483–496. https://doi.org/10.6033/tokkyou.42.483. Nuzzolo-Gomez, R., & Greer, R. D. (2004). Emergence of untaught mands or tacts of novel adjective-object pairs as a function of instructional history. The Analysis of Verbal Behavior, 20(1), 63–76. https://doi.org/10.1007/BF03392995. O’Connor, M., Farrell, L., Munnelly, A., & McHugh, L. (2017). Citation analysis of relational frame theory: 2009–2016. Journal of Contextual Behavioral Science, 6, 152–158. https://doi.org/10.1016/j.jcbs.2017.04.009. O'Donnell, J., & Saunders, K. J. (2003). Equivalence relations in individuals with language limitations and mental retardation. Journal of the Experimental Analysis of Behavior, 80(1), 131–157. https://doi.org/10.1901/jeab.2003.80-131. Pérez‐González, L. A., García‐Asenjo, L., Williams, G., & Carnerero, J. J. (2007).

Emergence of intraverbal antonyms in children with pervasive developmental disorder. Journal of Applied Behavior Analysis, 40(4), 697–701. https://doi.org/10.1901/ jaba.2007.697-701. Pérez-González, L. A., Herszlikowicz, K., & Williams, G. (2008). Stimulus relations analysis and the emergence of novel intraverbals. The Psychological Record, 58(1), 95–129. https://doi.org/10.1007/BF03395605. Pérez-González, L. A., Belloso-Díaz, C., Caramés-Méndez, M., & Alonso-Álvarez, B. (2014). Emergence of complex intraverbals determined by simpler intraverbals. The Psychological Record, 64(3), 509–526. https://doi.org/10.1007/s40732-014-0047-6. Pérez-González, L. A., & García-Asenjo, L. (2016). Emergence of intraverbals with antonyms derived from relations with verbal and nonverbal stimuli. The Psychological Record, 66(3), 351–368. https://doi.org/10.1007/s40732-016-0177-0. Petursdottir, A. I., & Haflidadottir, R. D. (2009). A comparison of four strategies for teaching a small foreign language vocabulary. Journal of Applied Behavior Analysis, 42, 497–745. https://doi.org/10.1901/jaba.2009.42-685. Petursdottir, A. I., & Devine, B. (2017). The impact of verbal behavior on the scholarly literature from 2005 to 2016. The Analysis of Verbal Behavior, 33(2), 212–228. https://doi.org/10.1007/s40616-017-0089-3. Presti, G., & Moderato, P. (2016). Verbal behavior: What is really researched? An analysis of the papers published in TAVB over 30 years. European Journal of Behavior Analysis, 17, 166–181. https://doi.org/10.1080/15021149.2016.1249259. Rehfeldt, R. A., & Root, S. L. (2005). Establishing derived requesting skills in adults with severe developmental disabilities. Journal of Applied Behavior Analysis, 38(1), 101–105. https://doi.org/10.1901/jaba.2005.106-03. Ribeiro, D. M., Elias, N. C., Goyos, C., & Miguel, C. F. (2010). The effects of listener training on the emergence of tact and mand signs by individuals with intellectual disabilities. The Analysis of Verbal Behavior, 26(1), 65–72. https://doi.org/10.1007/ BF03393084. Rosales, R., & Rehfeldt, R. A. (2007). Contriving transitive conditioned establishing operations to establish derived manding skills in adults with severe developmental disabilities. Journal of Applied Behavior Analysis, 40(1), 105–121. https://doi.org/10. 1901/jaba.2007.117-05. Rosales, R., Rehfeldt, R. A., & Lovett, S. (2011). Effects of multiple exemplar training on the emergence of derived relations in preschool children learning a second language. Analysis of Verbal Behavior, 27, 61–74. https://doi.org/10.1007/BF03393092. Ruiz-Sánchez, L. J., & Montoya-Rodríguez, M. M. (2014). A review of empirical relational frame theory articles in children and adolescents with developmental disabilities. European Journal of Child Development, Education and Psychopathology, 2, 17–26. Sautter, R. A., & LeBlanc, L. A. (2006). Empirical applications of Skinner's analysis of verbal behavior with humans. The Analysis of Verbal Behavior, 22, 35–48. https://doi. org/10.1007/BF03393025. Shillingsburg, M. A., Frampton, S. E., Cleveland, S. A., & Cariveau, T. (2017). A clinical application of procedures to promote the emergence of untrained intraverbal relations with children with autism. Learning and Motivation Advance Online Publication. https://doi.org/10.1016/j.lmot.2017.02.003. Sidman, M., & Tailby, W. (1982). Conditional discrimination vs. matching to sample: An expansion of the testing paradigm. Journal of the Experimental Analysis of Behavior, 37(1), 5–22. https://doi.org/10.1901/jeab.1982.37-5. Skinner, B. F. (1957). Verbal behavior. Acton, MA: Copley Publishing Group. Smith, D. P., Eikeseth, S., Fletcher, S. E., Montebelli, L., Smith, H. R., & Taylor, J. C. (2016). Emergent intraverbal forms may occur as a result of listener training for children with autism. The Analysis of Verbal Behavior, 32(1), 27–37. https://doi.org/ 10.1007/s40616-016-0057-3. Smith, T., & Iadarola, S. (2015). Evidence base update for autism spectrum disorder. Journal of Clinical Child & Adolescent Psychology, 44, 897–922. https://doi.org/10. 1080/15374416.2015.1077448. Snodgrass, M. R., Chung, M. Y., Meadan, H., & Halle, J. W. (2018). Social validity in single-case research: A systematic literature review of prevalence and application. Research in Developmental Disabilities, 74, 160–173. https://doi.org/10.1016/j.ridd. 2018.01.007. Speckman, J., Greer, R. D., & Rivera-Valdes, C. (2012). Multiple exemplar instruction and the emergence of generative production of suffixes as autoclitic frames. The Analysis of Verbal Behavior, 28(1), 83–99. https://doi.org/10.1007/BF03393109. Sprinkle, E. C., & Miguel, C. F. (2012). The effects of listener and speaker training on emergent relations in children with autism. The Analysis of Verbal Behavior, 28(1), 111–117. https://doi.org/10.1007/bf03393111. Sprinkle, E. C., & Miguel, C. F. (2013). Establishing derived textual activity schedules in children with autism. Behavioral Interventions, 28(3), 185–202. https://doi.org/10. 1002/bin.1365. Stauch, T., LaLonde, K., Plavnick, J. B., Bak, M. S., & Gatewood, K. (2017). Intraverbal training for individuals with autism: The current status of multiple control. The Analysis of Verbal Behavior, 33, 98–116. https://doi.org/10.1007/s40616-0170079-5. Stewart, I., McElwee, J., & Ming, S. (2013). Language generativity, response generalization, and derived relational responding. The Analysis of Verbal Behavior, 29(1), 137–155. https://doi.org/10.1007/BF03393131. Still, K., May, R. J., Rehfeldt, R. A., Whelan, R., & Dymond, S. (2015). Facilitating derived requesting skills with a touchscreen tablet computer for children with autism spectrum disorder. Research in Autism Spectrum Disorders, 19, 44–58. https://doi.org/10. 1016/j.rasd.2015.04.006. Vallinger-Brown, M., & Rosales, R. (2014). An investigation of stimulus pairing and listener training to establish emergent intraverbals in children with autism. The Analysis of Verbal Behavior, 30(2), 148–159. https://doi.org/10.1007/s40616-014-0014-y. Wolf, M. M. (1978). Social validity: The case for subjective measurement or how applied behavior analysis is finding its heart. Journal of Applied Behavior Analysis, 11(2), 203–214. https://doi.org/10.1901/jaba.1978.11-203.

147

Journal of Contextual Behavioral Science 12 (2019) 128–148

C. Raaymakers, et al. Young, M. E. (2018). A place for statistics in behavior analysis. Behavior Analysis: Research and Practice, 18(2), 193–202. https://doi.org/10.1037/bar0000099. Zaine, I., Domeniconi, C., & de Rose, J. C. (2014). Simple and conditional discrimination and specific reinforcement in teaching reading: An intervention package. The Analysis of Verbal Behavior, 30(2), 193–204. https://doi.org/10.1007/s40616. Zaring-Hinkle, B., Carp, C. L., & Lepper, T. L. (2016). An evaluation of two stimulus

equivalence training sequences on the emergence of novel intraverbals. The Analysis of Verbal Behavior, 32(2), 171–193. https://doi.org/10.1007/s40616-016-0072-4. Ziomek, M. M., & Rehfeldt, R. A. (2008). Investigating the acquisition, generalization, and emergence of untrained verbal operants for mands acquired using the Picture Exchange Communication System in adults with severe developmental disabilities. The Analysis of Verbal Behavior, 24(1), 15–30. https://doi.org/10.1007/BF03393054.

148