Who made the Aurignacian and other early Upper Paleolithic industries?

Journal of Human Evolution 57 (2009) 11–26

Contents lists available at ScienceDirect

Journal of Human Evolution journal homepage: www.elsevier.com/locate/jhevol

Who made the Aurignacian and other early Upper Paleolithic industries? Shara E. Bailey a, c, *, Timothy D. Weaver b, c, Jean-Jacques Hublin c a

Department of Anthropology, New York University, 25 Waverly Place, New York, NY 10003, USA Department of Anthropology, University of California, Davis, One Shields Avenue, Davis, CA 95616, USA c Department of Human Evolution, Max Planck Institute for Evolutionary Anthropology, Deutscher Platz 6, D-04103 Leipzig, Germany b

a r t i c l e i n f o

a b s t r a c t

Article history: Received 16 March 2008 Accepted 19 February 2009

The Aurignacian is typically taken as a marker of the spread of anatomically modern humans into Europe. However, human remains associated with this industry are frustratingly sparse and often limited to teeth. Some have suggested that Neandertals may, in fact, be responsible for the Aurignacian and the earliest Upper Paleolithic industries. Although dental remains are frequently considered to be taxonomically undiagnostic in this context, recent research shows that Neandertals possess a distinct dental pattern relative to anatomically modern humans. Even so, it is rare to find mandibles or maxillae that preserve all or most of their teeth; and, the probability of correctly identifying individuals represented by only a few teeth or a single tooth is unknown. We present a Bayesian statistical approach to classifying individuals represented exclusively by teeth into two possible groups. The classification is based on dental trait frequencies and sample sizes for ‘known’ samples of 95 Neandertals and 63 Upper Paleolithic modern humans. In a cross validation test of the known samples, 89% of the Neandertals and 89% of the Upper Paleolithic modern humans were classified correctly. We then classified an ‘unknown’ sample of 52 individuals: 34 associated with Aurignacian or other (non-Chaˆtelperronian) early Upper Paleolithic industries, 15 associated with the Chaˆtelperronian, and three unassociated. Of the 34 early Upper Paleolithic-associated individuals, 29 were assigned to modern humans, which is well within the range expected (95% of the time 26–33) with an 11% misclassification rate for an entirely modern human sample. These results provide some of the strongest evidence that anatomically modern humans made the Aurignacian and other (non-Chaˆtelperronian) early Upper Paleolithic industries. Ó 2009 Elsevier Ltd. All rights reserved.

Keywords: Neandertal Modern humans Homo sapiens Non-metric traits Dental morphology Chaˆtelperronian Bayesian statistics Classification

Introduction The paucity of well-preserved human remains associated with the earliest archaeological assemblages in Europe has led to continuous debates about who were the makers of these assemblages. Around the time at which Neandertals were replaced by anatomically modern humans in Europe (40,000 to 35,000 BP), a variety of lithic assemblages have been assigned to 1) the late Middle Paleolithic (e.g., Mousterian of Acheulean Tradition B or MATB, Denticulate Mousterian, Eastern Micoquian, etc.); 2) ‘‘transitional industries’’ with mixed technological and typological characteristics (e.g., Chaˆtelperronian, Szeletian, Uluzzian, etc.); and 3) genuine early Upper Paleolithic assemblages, mostly represented by the early Aurignacian, which is preceded and/or partly contemporary with a so-called ‘‘proto-Aurignacian.’’ The origin of the Aurignacian/proto-Aurignacian remains an open question, but it could be preceded in Eastern and Central Europe by various ‘‘Initial Upper Paleolithic’’ industries.

* Corresponding author. E-mail address: [email protected] (S.E. Bailey) 0047-2484/$ – see front matter Ó 2009 Elsevier Ltd. All rights reserved. doi:10.1016/j.jhevol.2009.02.003.

To date, all the diagnostic human remains found in a late Middle Paleolithic context are Neandertal. Of the ‘‘transitional’’ assemblages only one, the Chaˆtelperronian, has yielded significant human remains in two sites; these are also clearly Neandertal (Le´veˆque et al., 1993; Hublin et al., 1996; Bailey and Hublin, 2006). Because the Chaˆtelperronian shows affinities withdand has the same geographical distribution asdthe local MATB, some have interpreted it as the result of acculturation of the local Neandertals by incoming anatomically modern humans bearing ‘genuine’ Upper Paleolithic industries (e.g., Demars and Hublin, 1989; Harrold, 1989; Hublin et al., 1996). It has been argued that the development of other ‘transitional industries’ in Europe may result from a similar process. The occurrence of ‘genuine’ Upper Paleolithic industries is typically viewed as marking the spread of anatomically modern humans into Europe (Klein, 1999). Although no clear counter-examples to this association exist, there are relatively few human remains associated with the earliest Upper Paleolithic industries, particularly those older than 30,000 radiocarbon years (Gambier, 1989; Churchill and Smith, 2000a; Mellars, 2004; Trinkaus, 2005). At 31,000 BP, the Aurignacianassociated Mladecˇ fossils are the exception (Wild et al., 2005; Frayer et al., 2006; Wolpoff et al., 2006). The remaining human fossils securely associated with the Aurignacian (and other early

12

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26

UP Modern Neandertal 100 90 80 70 60 50 40 30 20 10 0 UI2 Shovel

UP3 MxPAR

LP4 TC

LM1 MTC

Figure 1. Frequency distribution of selected non-metric dental traits in Upper Paleolithic modern humans and Neandertals. UI2 Shovel: shoveling of the upper lateral incisor; UP3 MxPAR: Maxillary premolar accessory ridges on the upper P3; LP4 TC: Transverse crest (crest connecting protoconid and metaconid) of the lower P4; LM1 MTC: Mid-trigonid crest (connecting protoconid and metaconid) of the lower M1.

Upper Paleolithic industries) are said to be, for the most part, undiagnostic or poorly dated (see Churchill and Smith, 2000a for a review). The re-dating of fossils like Vogelherd (Conard et al., 2004) and Velika Pec´ina (Smith et al., 1999), from the early Upper Paleolithic to the Holocene has prompted some researchers to claim that it may be hasty to assume that anatomically modern humans were responsible for the Aurignacian (e.g., Conard et al., 2004), at least its earliest phases. Although dental remains are often found in association with early Upper Paleolithic industries, they have typically received little weight in diagnosing taxa. This is primarily because traditional taxonomic assessments were based on metrical aspects (which overlap greatly in late Pleistocene hominins) or a few morphological traits limited to a few teeth (e.g., taurodont molars, shovel shaped incisors). Recently, on assessing human remains associated with an early Aurignacian assemblage from Brassempouy, HenryGambier et al. (2004) concluded that the dental remains were undiagnostic, agreeing with Conard et al. (2004) that the makers of the earliest Aurignacian is open to question. This raises two questions that will be addressed in this paper: (1) is there truly no basis for assuming that anatomically modern humans were responsible for the early genuine Upper Paleolithic?

and, (2) are dental remains, particularly isolated teeth, taxonomically undiagnostic and therefore uninformative in this context? With regard to the first question, although certain specimens have been re-dated to the Holocene (see above), the dating of other specimens has confirmed that at least some anatomically modern humans are associated with the earliest Aurignacian. These include the aforementioned specimens from Mladecˇ (Wild et al., 2005), as well as some specimens from Les Rois (Dujardin and Tymula, 2004) and some from Isturitz (Turq et al., 1999). Unfortunately, the earliest anatomically modern human fossils from Europe (e.g., Pes¸tera cu Oase) lack archaeological association (Trinkaus et al., 2003b). There is no question that the Mladecˇ specimens are anatomically modern, although some researchers claim they preserve various archaic, perhaps Neandertal, characters (Frayer, 1992, 1997; Wolpoff, 2001). Likewise, some argue that the specimens from Les Rois are mixed in morphology (Trinkaus, 2007), while others have proposed that some of these specimens may represent Neandertals (Ramirez-Rozzi, pers. comm.). With regard to the claim that isolated teeth are taxonomically uninformative in the late Pleistocene (Henry-Gambier et al., 2004), this assertion is based on the fact that there is overlap in the ranges of variation for basic metric traits (i.e., length, breadth, and indices derived from these), and that all dental traits found in Neandertals can also be found in anatomically modern humans. Figure 1 illustrates the latter point. Frequency data for four traits are presented in Upper Paleolithic modern humans and Neandertals. It is clear that the differences between these two groups are in trait frequencies rather than trait presence or absence. In some cases the discrepancy is more exaggerated than in others (e.g., lower M1 mid-trigonid crest vs. upper I2 shoveling). Studies of morphometric variables show a similar pattern: Neandertals are on the extremes of shape variation but there is overlap between groups (Bailey, 2004; Bailey and Lynch, 2005; Martino´n-Torres et al., 2006; Go´mez-Robles et al., 2007; Souday, 2008). The question is, if there is overlap between the two groups for each feature considered on its own, does that mean teeth are taxonomically undiagnostic? Studies have shown that specific trait combinations in certain teeth clearly distinguish Neandertals from modern humans (Bailey, 2002a, 2007). These teeth, as well as those that show the most marked trait frequency differences between the two groups, are particularly useful in distinguishing Neandertals from anatomically modern humans. As such, they can be considered highly diagnostic. They include the upper incisors, the lower fourth premolar, the permanent first molars, and the deciduous second molars (Table 1, Fig. 2). Ultimately, we aim to determine whether or not dental nonmetric data alone can distinguish between Neandertals and Upper

Table 1 Diagnostic teeth for distinguishing between Neandertals and Upper Paleolithic modern humans (see text, Fig. 2). Tooth

Trait

Neandertals

Upper Paleolithic modern humans

Upper I1 and I2

Shoveling, labial curvature, tuberculum dentale Occlusal polygon area Crown shape

High frequency and expression; frequently co-occur Small (<30% crown area) Skewed

Low to moderate frequency and expression; rarely co-occur Large (>30% crown area) Square

Asymmetry, transverse crest, multiple lingual cusps Mid-trigonid crest Crown outline Crown outline

High frequency; frequently co-occur High frequency Rounded Rounded

Low to moderate frequency; rarely co-occur Rare to absent Angular Angular

M1 Lower P4 M1–3 M1a dm2a

Ref.b 1–4 1–3 5 6–8 9–10 11 7 7

Note: Superscript indicates upper dentition, subscript indicates lower dentition; I refers to incisors, C refers to canines, P refers to premolars, M refers to molars, lower case m refers to deciduous molar; number indicates tooth position. a Morphometric assessment of crown outline of M1 and dm2 were not used in this study and are listed here only for completeness. b 1-Bailey (2006); 2-Crummett (1995); 3-Mizoguchi (1985); 4-Martı´non-Torres et al., 2007; 5-Bailey (2004); 6-Go´mez-Robles et al., (2007); 7-Souday (2008); 8-Bailey (2004); 9-Bailey and Lynch (2005); 10-Martino´n-Torres et al., (2006); 11-Bailey (2002a).

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26

13

Figure 2. Diagnostic teeth and traits in Neandertals and Upper Paleolithic modern humans. 1: Incisors comparing lingual tubercles, shoveling, and lingual curvature in a Neandertal stonice) (1b); 2: Upper first molar comparing the small occlusal polygon area (OPA) and (Krapina) (1a) to the lack of these traits in an Upper Paleolithic modern human (Dolnı´ Ve skewed shape in a Neandertal (Krapina) (2a) to the larger OPA and squarer shape in the Upper Paleolithic modern human (La Madeleine) (2b); 3: Lower fourth premolar comparing multiple lingual cusps, transverse crest and lingual crown asymmetry, in a Neandertal (Krapina) (3a) to the lack of these traits in an Upper Paleolithic modern human (Forneau du Diable); 4: Lower first molar comparing the mid-trigonid crest and rounded shape* in a Neandertal (Le Moustier) (4a) to the lack of a mid-trigonid crest and more rectangular shape* in an Upper Paleolithic modern human (Grotte des Abeilles); 5: lower second deciduous molar comparing the mid-trigonid crest and rounded shape* in Neandertals (Arcysur-Cure) (5a) to the lack of a mid-trigonid crest and more rectangular shape in an Upper Paleolithic modern human (Isturitz). (Teeth are not to scale). * See Souday (2008) for a morphometric analysis of crown shape in Neandertals and anatomically modern humans.

Paleolithic modern humans, and, if successful, to provide a method that would help diagnose unidentified fossil teeth, especially those associated with the earliest Upper Paleolithic industries. Because there are many more dental remains than cranial or postcranial remains associated with early Upper Paleolithic industries, a reliable classification of the dental remains has the potential to resolve the issue of who made the Aurignacian and other early Upper Paleolithic industries. With this in mind, the goals of this study are to: 1. develop a reliable method to estimate taxonomic affinity (visa`-vis Neandertals and Upper Paleolithic modern humans) of indeterminate or contentious specimens based solely on dental non-metric variables; and 2. if successful, to apply the method to a sample of human teeth associated with early Upper Paleolithic industries, many of which are considered to be controversial, undiagnostic, or to show a ‘mosaic’ of Neandertal and modern human traits. For these specimens our goal is to determine the posterior probabilities of membership to Neandertal vs. Upper Paleolithic modern human groups. We chose a phenetic rather than a cladistic approach to our problem. Patterns of variation between groups in most dental traits

are typically thought to be neutral (Scott and Turner, 1997), resulting from mutation, genetic drift, and gene flow rather than from natural selection. If this assumption is correct, then one should obtain similar results using either a phenetic (based on all observable similarities or differences) or a cladistic (based only on derived similarities or differences) approach. When divergence is neutral, the overall amount of morphological differencedwhich is the quantity measured by phenetic approachesdwill be proportional to the time since divergence (Lynch, 1989; Weaver et al., 2008). The traits that contribute to morphological difference will, by definition, be derived; so, the derived features counted by cladistic approaches will also accumulate with time (Pagel, 2002; Weaver et al., 2008). Because both the overall amount of morphological difference and the number of derived features are essentially proxies for time since divergence, both phenetic and cladistic approaches should give similar results if divergence is neutral. Additionally, since Neandertal and Upper Paleolithic modern groups vary in trait frequency rather than in trait presence or absence, a cladistic approach would require assigning character states to trait frequencies. When this method (Thiele, 1993) was applied to an analysis of middle-late Pleistocene hominins and contemporary humans, the results from the cladistic analysis largely mirrored those of the phenetic analysis (Bailey, 2002b). Similar agreement between phenetic and cladistic analyses were

14

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26

found by Martı´non-Torres et al. (2007) on a large sample of African and Eurasian hominins. We believe the agreement between phenetic and cladistic analyses based on dental non-metric traits supports the assumption that, while there may be certain exceptions, dental morphological divergence is primarily neutral. Therefore, we felt that using a cladistic approach was neither necessary nor would be more informative than a phenetic approach. Materials Since our objective is to address the question of whether or not human remains from the European late Pleistocene belong to Neandertals vs. modern humans, we limit our ‘known’ samples to these two groups. The affinities of these known specimens are either generally uncontested and/or determined from non-dental data. We include 95 Neandertals and 63 Upper Paleolithic-associated modern humans (Table 2, and Appendix A). The Neandertal sample consists of European specimens spanning ca. 130,000 years ago (e.g., Krapina) to ca. 30,000 years ago (e.g., Vindija). Previous studies have shown that there are no significant differences in dental morphology between later and earlier Neandertals, suggesting that, at least dentally, they comprise a homogeneous group (Bailey, 2002b, 2007; Martino´n-Torres et al., 2006; Go´mez-Robles et al., 2007). The modern human sample includes fossils associated with Upper Paleolithic industries (Aurignacian, Gravettian, Solutrean, and Magdalenian), dating to between about 30,000 and 11,000 years ago. We exclude contemporary modern humans and the earliest (non-European) anatomically modern humans from the modern human sample. We do this primarily because our questions pertain specifically to distinguishing between fossil rather than contemporary human groups. Moreover, contemporary humans are heterogeneous across the globe, and the dental morphology of contemporary Europeans differs significantly from that of Upper Paleolithic modern humans (Bailey, 2007, 2008). We exclude the earliest modern humans from Africa and Asia for similar reasons. First, they represent a different geographic region from those of interest. Second, although they are nearly as divergent from Neandertals as are Upper Paleolithic modern humans (Bailey, 2006: Table 15), they diverge in a different way, such that there are significant dental differences between the earliest modern humans outside of Europe and those associated with the European Upper Paleolithic (Bailey, 2007: Table 3). Therefore, unlike ‘Neandertals,’ ‘early modern humans’ do not comprise a dentally homogenous group. Our ‘unknown’ sample includes fossils associated with the earliest Upper Paleolithic industries, including those considered ‘transitional’ industries (Table 3). Our primary unknown sample consists of 34 individuals associated with Aurignacian or other early Upper Paleolithic (EUP) industries (excluding the Chaˆtelperronian) and three individuals that are unassociated or for which the provenience is unknown. We place Chaˆtelperronianassociated teeth into a second unknown sample of 15 individuals because there is substantial prior evidence that the Chaˆtelperronian was made by Neandertals rather than by modern humans (Le´veˆque and Vandermeersch, 1980; Hublin et al., 1996). All these fossils date to between w34,000 14C BP and w29,000 14C BP. Although there is little dispute regarding the Neandertal affinity of Saint-Ce´saire (associated with the Chaˆtelperronian) or the modern affinity of the Mladecˇ skulls (associated with the early Aurignacian), many of the other fossils in the unknown sample are somewhat controversial, being thought by some to be ‘undiagnostic’ or to be mosaic (i.e., Neandertal plus modern) in character (Gambier, 1989; Churchill and Smith, 2000b; Henry-Gambier et al., 2004).

Table 2 Specimens comprising the ‘known sample’ and number of traits used. Group

Specimen

No. Features Used

Neandertal

Krapina DP#1 Krapina DP#2 Krapina DP#3 Krapina DP#4 Krapina DP#5 Krapina DP#6 Krapina DP#8 Krapina DP#10 Krapina DP#11 Krapina DP#12 Krapina DP#13 Krapina DP#17 Krapina DP#18 Krapina DP#19 Krapina DP#20 Krapina DP#21 Krapina DP#22 Krapina DP#23 Krapina DP#24 Krapina DP#25 Krapina DP#27 Krapina DP#28 Krapina DP#29 Krapina DP#30 Krapina DP#31 Krapina DP#32 Krapina DP#33 Krapina DP#34 Krapina DP#35 Krapina 40 Krapina Maxilla B Krapina Maxilla C and Mandible C Krapina Composite 1a Krapina Composite 2a Krapina Composite 3a Krapina Mandible F Malarnaud Monsempron 1953-1 Monsempron misc Regourdou Arcy-sur-Cure 40 (Renne: Level XIV) Arcy-sur-Cure 39 (Renne: Level XII) Arcy-sur-Cure #41 (Renne: Level XIV) Arcy-sur-Cure #43 (Renne: Level XIV) Arcy-sur-Cure 45 (Galerie Schoepflin: Level IV5) Arcy-sur-Cure mandible (Hye`ne: Level IVb6) Arcy-sur-Cure 9 (Hye`ne: IVb6) Gibraltar: Devil’s Tower Ochoz Ku˚lna Petit-Puymoyen 1 Petit-Puymoyen 2 Petit-Puymoyen 3 Petit-Puymoyen 4A Petit-Puymoyen 4B Petit-Puymoyen 211 Petit-Puymoyen 1975-30-5 Hortus II Hortus III Hortus IV Hortus V Hortus VI Hortus VII Taubach La Fate II

19 22 20 58 34 37 6 20 11 18 16 11 34 27 17 6 8 47 8 4 26 7 7 12 13 9 9 10 7 4 3 24 47 46 11 6 6 32 6 23 5 4 6 5 4 7 6 12 9 18 18 11 9 7 7 14 26 16 15 16 23 11 7 5 7

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26 Table 2 Appendix A. (continued) Group

Upper Paleolithic Modern

15

Table 2 Appendix A. (continued)

Specimen

No. Features Used

La Fate VI La Fate XII La Fate XIII Roc du Marsal Ciota Ciara 2 (Monte Fenera) Ciota Ciara 3 (Monte Fenera) Grotte Taddeo Rep H #1 Grotte Taddeo Rep H #2 Grotte Taddeo Rep L Guattari III Saccopastore 1 Saccopastore 2 Vindija Vi149 Vindija Vi146 Vindija Vi259 Vindija Vi148 Spy 1 Spy 2 Le Moustier La Quina 5 La Quina 9 La Quina 18 Montgaudier 5 Combe Grenal Chateauneuf 2 Marillac La Ferrassie 8 Obi Rakhmat Subalyuk 1 Subalyuk 2

5 4 2 2 6 5 2 3 8 14 4 12 2 9 4 16 9 5 60 15 16 9 7 13 8 7 7 24 18 3

Les Vachons Roc de Combe 4 Lagar Velho Dolnı´ Veˇstonice 13 Dolnı´ Veˇstonice 14 Dolnı´ Veˇstonice 15 Dolnı´ Veˇstonice 16 Dolnı´ Veˇstonice 31 Dolnı´ Veˇstonice 37 Dolnı´ Veˇstonice 36 Vindija 289 level Fd Pavlov 1 Pavlov 3 Pavlov 2 Abri Pataud 1 Abri Pataud 2 Abri Blanchard 1956–46 Abri Labatut 1956–47 Mieslingtal Grotte des Abeilles a 1,2 Grotte des Abeilles 3 Lespugue La Gravette Balla Barlang #68.145.1 Bervavolgy #68.142.1 Gruta do Caldeira˜o 1 Cisterna 1 La Madeleine Peche de la Boissiere B1 Peche de la Boissiere B2 Farincourt Farincourt Laugerie Basse St. Germaine-la-Rivie`re B4 St. Germaine-la-Rivie`re B3 St. Germaine-la-Rivie`re B5 St. Germaine-la-Rivie`re B6, B7a St. Germaine-la-Rivie`re 1a, 2 St. Germaine-la-Rivie`re 3 St. Germaine-la-Rivie`re 6 St. Germaine-la-Rivie`re 9 St. Germaine-la-Rivie`re 10 St. Germaine-la-Rivie`rea 11, 21, 18, 7 St. Germaine-la-Rivie`re 12

12 3 46 36 37 49 4 6 4 9 2 1 7 8 45 3 7 9 6 11 4 15 3 7 12 9 18 11 7 4 5 27 46 14 7 7 4 5 7 2 1 5 13 3

Group

Specimen St. Germaine-la-Rivie`re 14 St. Germaine-la-Rivie`re 15 St. Germaine-la-Rivie`re 16 St. Germaine-la-Rivie`rea 19,20 St. Germaine-la-Rivie`re 7unnumbered Gough’s Cave a 22,87 & 49 Gough’s Cave 253/263 Gough’s Cave 139 Oberkassel D999 Oberkassel unnumbered Isturitz 1950-6 Isturitz 1950-10-3 Isturitz 1950-9 (IV-105) Isturitz 1950-10-2 Isturitz IV 1942/1950 Isturitz mandible series 7B 1950-4-1 La Chaud 4 La Chaud 5 La Chaud 3 La Chaud 83

No. Features Used 6 4 3 9 3 38 11 15 35 7 6 7 6 2 6 5 17 26 19 6

Note: For more detailed information on which teeth are preserved for individual specimens, please contact the lead author. a Indicates a ‘composite’ individual (see text).

Because a good deal of our sample is comprised of loose teeth, ‘individuals’ could be represented by one to many teeth. Where it made sense to do so, we group isolated teeth into composite individuals because it increases the number of traits that can be evaluated, which provides greater power for group assignment. We take a conservative approach: we group teeth only if they were associated in the same level or square, were not duplicates, and showed similar wear and/or development. In some cases the individual groupings had been previously assigned. For example, most of the Krapina individuals consist of the Krapina Dental People assembled by Radovcˇic´ et al. (1988). In other cases it was evident that teeth belonged to the same individual based on morphology (i.e., they were antimeres), wear, and/or matching interproximal facets. In the end, composite individuals (assigned by SEB) make up only a small portion of our entire sample (see Table 2). Combining teeth into composite individuals in the ‘known’ sample may influence the cross validation assessment of accuracy because it results in a greater number of traits per individual (i.e., more information). However, composite individuals in the known sample have no effect on the group assignment of the unknown individuals. This is because our method assigns group membership based on overall trait frequencies in the two groups and not the combination of traits observed in particular individuals in each of the groups. Methods Dental traits We use different trait lists in two different analyses (Tables 4 and 7). Seventy-one traits are used in the first analysis, but the maximum number of traits that could be scored on any one individual was 66. Twenty-seven traits are used in the second analysis, and the maximum number that could be used for any one individual was 26. However, in both analyses most individuals preserve fewer than half these traits. Dental traits are scored either following the Arizona State University Dental Anthropology System (or ASUDAS; Turner et al., 1991; Burnett, 1998), or for those traits not currently part of the

16

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26

Table 3 Specimens comprising the ‘unknown’ fossil sample. Group Early Upper Paleolithic

Chaˆtelperronian

No association

a

Specimen Mladecˇ (9a, 9b, 10)a Mladecˇ 2 Mladecˇ 1 Vindija 289 Fontechevade 1 Fontechevade 2 Font de Gaume 2 La Ferrasie 7 La Ferrassie 10 La Ferrasie 11 Derava´ Skala } Ista´llo´sko Vindijaa G1 Vi287 & Vi290 Bacho Kiro 599 Brassempouya 1046, 3040, 542 Brassempouya 262, 441 Brassempouy 884 Brassempouy 16 Brassempouy 2206 Grotte des Roisa mandible A, plus 6 Grotte des Roisa R50 mandible B, plus R51, 22, R51, 23 Grotte des Roisa 3, R50 Grotte des Roisa 32, 18, unnumbered tooth Grotte des Roisa 45, 15, 54 Grotte des Roisa 19, 24, 40, 6 Grotte des Rois 5 Grotte des Rois 29 Grotte des Rois R8 Grotte des Rois R51-30 Grotte des Rois R16 Grotte des Rois R51-14, R31 Grotte des Rois 55 Grotte des Rois unnumbered Grotte des Rois unnumbered St Ce´saire 1 St Ce´saire 2 Arcy-sur-Cure 4 Arcy-sur-Cure 5 Arcy-sur-Cure 6 Arcy-sur-Cure 7 Arcy-sur-Cure 13 Arcy-sur-Cure 16 Arcy-sur-Cure 17 Arcy-sur-Curea 19, 20 Arcy-sur-Cure 21 Arcy-sur-Curea 23, 24 Arcy-sur-Cure 30 Arcy-sur-Cure 32 Arcy-sur-Cure 35 Oase 1 Oase 2 Vindija Vi 76 (229)

Table 4 Set of dental traits used in the ‘complete’ trait approach. No. Features Used 11 9 6 1 4 7 7 4 7 4 7 7 8 6 9 3 6 4 2 22

Tooth/presence Maxilla ASU ASU ASU ASU

2–4 2–7 1–6 2–6

Labial convexity Shoveling Double Shoveling Tuberculum dentale

ASU 2–4 ASU 1–6 ASU 2–6

Shoveling Double Shoveling Tuberculum dentale

ASU ASU ASU ASU ASU

Shoveling Double Shoveling Tuberculum dentale Bushman canine Distal accessory ridge

I2

C

/C 2–4 1–6 2–6 1–3 2–5

P3 SEBþ Burnettþ Burnett 2–4

13 14 12 16 21 4 4 7 7 7 9 7 4 4 51 3 6 7 1 1 5 1 2 8 7 8 7 1 7 12 5 1

Indicates a ‘composite’ individual (see text).

ASUDAS, following the standards outlined by Bailey (2002b). Trait expressions for the ASUDAS traits are scored using a combination of written descriptions and reference plaques. The additional traits from Bailey (2002b) do not have associated reference plaques; however, most are scored as present or absent (e.g., bifurcated median ridge of P3). For traits scored on a continuum of expression, the expression is dichotomized into presence/absence based on standard breakpoints used in other dental morphological analyses (Irish, 1998; Bailey, 2002b). One metric trait, the reduced occlusal polygon area (OPA), is included because of its ability to discriminate Neandertals and non-Neandertals (Bailey, 2004). In order to translate this metric trait (a measure of relative area) into a discrete trait (present/absent), ‘‘presence’’ is recorded if the OPA is <30% of the total crown area.

Tooth/presence Mandible

I1

ASUþ SEBþ

P4 SEBþ Burnettþ Burnett 2–4 ASUþ SEBþ

Medial ridge Triangular ridge bifurcation Maxillary premolar accessory ridges (MxPAR) Accessory cusps Transverse crest

Medial ridge Triangular ridge bifurcation Maxillary premolar accessory ridges (MxPAR) Accessory cusps Transverse crest

M1 ASU 1–5 ASU 3–7 SEB <30% SEBþ

Cusp 5 Carabelli’s cusp Occusal polygon area Mesial accessory cusps

M2 ASU 0–2 ASU 1–5 ASU 3–7 SEBþ

Hypocone reduction Cusp 5 Carabelli’s cusp Mesial accessory cusps

M3 ASU 0–3 ASU 0–2 ASU 1–5 ASU 3–7 SEBþ ASU P or R

Metacone reduction Hypocone reduction Cusp 5 Carabelli’s cusp Mesial accessory cusp Reduced/peg

ASU 2–5

Distal accessory ridge

P3 ASU 2–9 SEB 1–2

Lingual cusp number Transverse crest

SEB 2–3

Distal accessory ridge

SEB SEB SEB SEB

Mesial accessory ridge Mesial lingual groove Distal lingual groove Asymmetry

2–3 þ þ 1–2

P4 ASU 2–9 SEB 1 SEB 1–2

Lingual cusp number Mesial metaconid placement Transverse crest

SEB 2–3 SEB 2-3 SEB 1–2

Distal accessory ridge Mesial accessory ridge Asymmetry

M1 ASU Y ASU 4 ASU 2–3 ASUþ SEB 1–3 ASU 1–6 ASU 2–4

Y-pattern Cusp number Deflecting wrinkle Distal trigonid crest Mid-trigonid crest Cusp 6 Cusp 7

M2 ASU Y ASU 4 ASU 2–3 ASUþ SEB 1–3 ASU 1–6 ASU 2–4

Y-pattern Four cusped Deflecting wrinkle Distal trigonid crest Mid-trigonid crest Cusp 6 Cusp 7

M3 ASU Y ASU 4 ASU 2–3 ASUþ SEB 1–3 ASU 1–6 ASU 2–4

Y-pattern Four cusped Deflecting wrinkle Distal trigonid crest Mid-trigonid crest Cusp 6 Cusp 7

Although both left and right sides of the dentition are scored, we rely on the ‘individual count method’ to calculate trait frequencies. In accordance with this method, when a tooth is present on both sides, only the side with the highest expression is used in the analysis (Turner and Scott, 1977; Scott and Turner, 1997). We use two different methods to analyze our data, each of which is based on different assumptions. The first methoddthe ‘complete trait’ analysisdutilizes all dental trait information available for a particular individual. Because this analysis involves

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26

17

using multiple teeth in a tooth field (e.g., M1, M2, and M3), it assumes either that 1) all traits are completely independent within the fields; or 2) the pattern of dependencies differs between groups (e.g., Neandertals and Upper Paleolithic modern humans) in such a way that if one did not know the group, knowing the state of a trait for one tooth in the class would not allow the state to be predicted accurately for other teeth in the class (i.e., correlation patterns differ between groups). In contemporary humans, we know that traits are not completely independent within a tooth field. Scott and Turner (1997:111) report that most interclass correlations fall between 0.25 and 0.60. Unfortunately, we could not calculate interclass correlations for our fossil samples because only a small percentage of individuals possessed multiple teeth in a given tooth field. For example, only 6% of the known individuals possessed all three upper molars on which Carabelli’s trait could be scored, and only about 8% possessed both upper I1 and I2 on which shoveling could be scored. Although it was not possible to calculate correlations in our fossil groups, we have reason to believe that they would be different from those observed in contemporary humans. For example, while there is 100% concordance between the presence of shoveling for the upper I1 and I2 in Neandertals (n ¼ 8), in a geographically mixed contemporary human sample (Bailey, unpublished data) the interclass correlation (tetrachoric correlation) for this trait is a moderate 0.55 (n ¼ 55). Given the above, a method that relied on estimates of correlations from contemporary humans would likely produce inaccurate results. In addition, it is possible the patterns of trait dependency are different between Neandertals and Upper Paleolithic modern humans. Typically, statistical analyses of dental morphology (e.g., biodistance studies) utilize only one tooth from a tooth field to avoid issues of trait correlation. Often the ‘key’ tooth is the one that is considered most stable according to the morphogenetic field concept (Dahlberg, 1945; Scott and Turner, 1997) and thought to most accurately reflect the underlying genotype for a particular trait. The ‘key’ tooth may also be the tooth that represents the center of a morphogenetic field. For example, for tuberculum dentale (lingual tubercles) the center of the morphogenetic field is the upper I2, while for shoveling the center of the morphogenetic field is the upper I1. These ‘key’ teeth have been previously worked out for most traits of the ASUDAS (Turner et al., 1991). Unfortunately, an analysis based exclusively on ‘key’ teeth would have excluded about 9% of our known individuals and 17% of our unknown individuals for which ‘key’ teeth were not preserved. Therefore, we devised the second methoddthe modified ‘key tooth’ analysisdto deal with this problem. This method uses only the ‘key’ tooth in a given field unless that tooth is absent, in which case another tooth in the tooth field is substituted. Because this method substitutes, say, an M2 for an M1, it assumes that both teeth provided exactly the same information. In other words, the modified ‘key’ tooth analysis assumes complete dependence of dental traits on teeth in the same tooth field. Data from contemporary humans suggest that tooth traits within a tooth field are neither completely independent nor completely dependent. Instead, reality lies somewhere in between. In lieu of using a method that takes into account interclass correlations that we could not estimate in fossils, we felt our two alternative methods, representing the two extremes of complete dependence and complete independence, would sufficiently address issues related to interclass trait correlation.

Paleolithic modern human, P(M), and assign it to the group with the higher posterior probability. The posterior probabilities are the probabilities that the unknown individual comes from each of the two groups, given that it possesses or lacks certain dental traits and assuming it certainly derives from either Neandertal or Upper Paleolithic modern groups, (P(N) þ P(M) ¼ 1). The latter assumption seems reasonable given that we have constrained the time and geographic locations, but it may not be valid in other situations. Starting with the case of a single trait that is present on the unknown, using Bayes’ Rule (Berry, 1996) we can calculate the posterior probability of being a Neandertal from the prior probabilities (indicated here by ‘‘R’’ for the two occurrences of this letter in the word ‘‘prior’’ to distinguish from the ‘‘P’’ for posterior probabilities) of the unknown being a Neandertal, R(N), or a modern human, R(M), and the likelihoods, which are the probabilities of the unknown having the trait present if it comes from a Neandertal, LðtjNÞ, or a modern human, LðtjMÞ, as

Classification

interval [0, 1] into equal portions of length 1k. If we assume equal prior probabilities for each fi and prior probabilities of zero for any other value in the interval [0, 1] then it follows from Bayes’ Rule and Eq. (2) that

To classify an unknown tooth or partial dentition we calculate its posterior probability of being a Neandertal, P(N), or an Upper

PðNÞ ¼

LðtjNÞRðNÞ : LðtjNÞRðNÞ þ LðtjMÞRðMÞ

(1)

An equation analogous to Eq. (1) gives the posterior probability of being a modern human. We can extend this approach to multiple traits if we assume the different traits are independent within groups (conditionally independent given group membership). We consider whether the assumption of conditional independence is reasonable in the previous section. By assuming conditional independence, we can calculate the posterior probabilities for a set of traits by sequentially applying Bayes’ Rule, so the posterior probabilities based on the 1st trait become the prior probabilities for the 2nd trait, the posterior probabilities based on the 2nd trait become the prior probabilities for the 3rd trait, and so on. For the 1st trait we assume equal (uniform) priors, R(N) ¼ R(M) ¼ 0.5. This seems reasonable given that our study is motivated by the claim that there is no reason to assume in advance that, for example, an unknown from the Aurignacian has a higher probability of being a Neandertal or a modern human. The final step is to calculate the likelihoods for Neandertals and modern humans for each trait. Taking Neandertals as an example, LðtjNÞ is the probability a particular dental trait will be present on a Neandertal tooth and 1  LðtjNÞ is the probability the trait will be absent. LðtjMÞ is calculated with comparable steps. Suppose in the Neandertal known sample there are r individuals out of n total individuals with the trait present, giving a fraction p ¼ nr of the Neandertal sample with the trait present. If the fraction of the Neandertal population with the trait present is f, then LðtjNÞ ¼ f. However, we only know p rather than f, and LðtjNÞsp unless p ¼ f. The probability mass function for the binomial distribution (Casella and Berger, 2002) gives the probability of obtaining the observed r in a sample of n known Neandertal individuals for a given fi :

Pðrjfi ; nÞ ¼

Let fi ¼

n! fr ð1  fi Þnr : r!ðn  rÞ! i n

0; 1; .; k k k k

o

(2)

where k is a constant used to divide the

18

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26

n! fr f nr r!ðnrÞ! i ð1  i Þ n! fr f nr r!ðnrÞ! i ð1  i Þ

fr ð1  fi Þnr : fri ð1  fi Þnr

¼ Pi

Pðfi jr; nÞ ¼ P

(3)

It follows from the Law of Total Probability (Berry, 1996) and Eq. (3) that

LðtjNÞ ¼

X

P

frþ1 fr ð1  fi Þnr ð1  fi Þnr i fi P i r : nr ¼ P r fi ð1  fi Þ fi ð1  fi Þnr

(4)

Eq. (4) will provide a good approximation for LðtjNÞ if k is large, but we can find an exact solution by converting to the continuous case, which gives

R rþ1 f ð1  fÞnr df LðtjNÞ ¼ R r : f ð1  fÞnr df

(5)

Obtaining analytical solutions to the integrals in Eq. (5) results in

LðtjNÞ ¼

ðr þ 1Þ!ðn þ 1Þ! rþ1 ¼ ; r!ðn þ 2Þ! nþ2

(6)

so with no known Neandertals LðtjNÞ ¼ 0:5, because n ¼ r ¼ 0. Eq. (6) further shows that LðtjNÞ depends on n as well as p. Factoring p from the numerator of Eq. (6) and taking the limit as n/N gives lim

n/N

  p n þ 1p nþ2

¼ p;

(7)

which demonstrates that as the known Neandertal sample size increases, LðtjNÞ approaches p. The importance of considering sample size (n) and not just trait frequencies (p  100) can be illustrated with the following example. Suppose we have an unknown tooth that has a particular dental trait present. First, without considering the known sample at all, if we assume the tooth must either come from a Neandertal or a modern human (P(N) þ P(M) ¼ 1) and there is no prior reason to choose one group over the other (R(N) ¼ R(M) ¼ 0.5), then the probability that it comes from a Neandertal is 0.5. Now imagine that the known sample consists of one Neandertal and one modern human, and the trait is present for the Neandertal but absent for the modern human. In this case, the known sample frequencies would be 100% for Neandertals and 0% for modern humans. Just considering the frequencies alone, it would appear that the probability of the unknown tooth being a Neandertal is now 1, but, in fact, this would only be the case if we knew all Neandertals had the trait and all modern humans lacked the trait. The actual probability is given by Eqs. (1) and (6) as 0.67, which is between the prior probability (0.5) and the fraction of known Neandertals with the trait present (1). Because the known sample is small it is closer to 0.5. As shown by Eq. (7), if the known sample size increases and all Neandertals continue to have the trait and all modern humans continue to lack the trait, then the probability the tooth comes from a Neandertal will become closer and closer to 1 (Fig. 3). In essence, the utility of a trait for classification is not only a function of how different the frequencies are for Neandertals and modern humans, but also a function of how many individuals those frequencies are based on. Cross validation We use cross validation (Krzanowski, 2000) to test the accuracy of our method in classifying unknown individuals. Misclassification probabilities from cross validation are virtually unbiased estimates

Figure 3. Graph illustrating the relationship between the sample size and posterior probabilities (see text).

of the actual misclassification probabilities for unknown individuals. Additionally, on some level, it does not matter what theoretical assumptions the method is based ondi.e., trait independence or dependencedas long as we can show empirically that it works. The cross-validation steps are: 1. Select one individual from the known sample. 2. Classify this individual based on all the other individuals in the known sample. This is important because the selected individual is not included in the sample used to calculate the classification, so it mimics an unknown individual. 3. Repeat steps 1 and 2 for all the individuals in the known sample. 4. Calculate the number of individuals who were correctly classified. Results The cross validation test using the complete set of traits showed that 89% of the known sample was correctly assigned to their respective groups, which is a misclassification rate of 11% (Table 5). Table 6 provides the details of the misclassified individuals. Ten of the misclassified individuals (four Neandertals and six Upper Paleolithic modern humans) had relatively low (<65%)1 posterior probabilities for group membership. The remaining misclassified individuals had posterior probabilities ranging from 66% to 65%. The overall classification results in our modified key tooth analysis were very similar to those of the complete trait analysis (Table 8). The total correct classification increased slightly from 89% (complete trait analysis) to 91% (modified key tooth analysis). The classification of known Neandertals increased from 89% to 90%, while the classification for Upper Paleolithic modern humans increased from 89% to 92%. Table 9 provides the details of the misclassified individuals. Nine of the misclassified individuals (five Neandertal and four Upper Paleolithic modern) had relatively low (<65%) posterior probabilities for group membership. The remaining misclassified individuals had posterior probabilities ranging from 66%–75%. Two Neandertals (Krapina DP 8 and 25) and two

1

see Discussion for why posterior probabilities below 65% are considered ‘low’.

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26 Table 5 Cross validation test classifying known Neandertals and Upper Paleolithic modern humans using the ‘‘complete trait’’ approach (see text). Correct

Incorrect

Neandertal Upper Paleolithic modern

89% (n ¼ 85) 89% (n ¼ 56)

11% (n ¼ 10) 11% (n ¼ 7)

Total

89% (n ¼ 141)

11% (n ¼ 17)

Upper Paleolithic modern humans (one from St. Germain-la-Rivie`re and one from Isturitz) that were misclassified in the complete analysis were correctly classified in the modified key tooth analysis. In sum, both complete trait and modified key tooth analyses produced highly accurate results. The 89–92% accuracy achieved is better than is typical for sex determination based on cranial morphology (Duric et al., 2004; Williams and Rogers, 2006). Because the cross validation results were nearly the same for either method, one could argue that no preference should be given to one approach over the other. However, three individuals that were classified in the first analysis (two of which classified correctly) were unclassified in the second analysis because both the ‘key’ tooth and the ‘back up’ tooth were absent. Thus, while either method produces highly accurate results, we preferred to apply the complete trait analysis to our unknown sample. While we are not advocating that the complete trait approach will always be the best approach, in a general sense, we chose to use it here because it produced comparable results to the ‘‘modified key tooth’’ approach and allowed us to classify more unknown individuals.

Unknown specimens Tables 10 and 11 present the classification results for the unknown sample. We found that 85% (29/34) of the specimens associated with Aurignacian/early Upper Paleolithic industries were assigned to Upper Paleolithic modern humans, which is well within the range expected (95% of the time 26–33), with an 11% misclassification rate for an entirely modern human sample. We found the lower and upper ends of the expected range as the 2.5% and 97.5% quantiles, respectively, of a binomial probability distribution with 34 samples and an 89% chance of correct classification. In contrast to the Aurignaican/EUP sample, 93% (14/15) of the specimens associated with the Chaˆtelperronian classified as Neandertals. We do note that for one of the Chaˆtelperronian Table 6 Cross validation: Misclassified individuals using the complete trait approach. Specimen

Posterior Probability

Neandertals classified as Upper Paleolithic modern Krapina 40 90% 55% KDP 25a 74% KDP 32a 63% KDP 8a Arcy-sur-Cure 9 72% Grotte Taddeo Rep H 65% Ochoz 55% Vindija Vi259 88% Petit Puymoyen 3 56% Le Fate XIII 74% Upper Paleolithic moderns classified as Neandertals stonice 16 Dolnı´ Ve 60% Pavlov 1 61% La Gravette 54% St. Germain-la-Rivie`re 19 & 20 60% St. Germain-la-Rivie`re (unnumbered) 62% Isturitz 1950-10-2 53% Isturitz IV 1942/1950 66% a

KDP ¼ Krapina Dental Person.

No. traits 4 4 9 6 6 3 9 4 9 2 4 1 3 9 3 2 6

19

Table 7 Set of dental traits for modified ‘key’ tooth approach.a Tooth/presence

Maxilla

I1 ASU 2–4 ASU 2–7 ASU 1–6

Labial convexity Shoveling Double shoveling

ASU 2–6

Tuberculum dentale

Tooth/presence

Mandible

I1

None

I2

None

/C

None

P3 SEB 2–3 SEBþ

Distal accessory ridge Mesial lingual groove

P4 ASU 2–9

Lingual cusp number

I2 C/ ASU 1–3 ASU 2–5

Bushman canine Distal accessory ridge

P3

none

P4 Burnettþ Burnettþ ASUþ

Triangular ridge bifurcation Maxillary premolar accessory ridges Accessory cusps

M1 ASU 1-5 ASU 3–7 SEB <30%

Cusp 5 Carabelli’s cusp Occlusal polygon area

M2 ASU 0–2

Hypocone reduction

M3 ASU P or R

Reduced/peg

SEB 1 SEB 1–2 SEB 1–2

Mesial metaconid placement Transverse crest Asymmetry

M1 ASU 2–3 ASUþ SEB 1–3 ASU 1–6 ASU 2–4

Deflecting wrinkle Distal trigonid crest Mid-trigonid crest Cusp 6 Cusp 7

M2 ASU Y ASU 4

Y-pattern Four cusped

M3

None

a I2 was substituted when I1 was absent and vice versa. P3 was substituted when P4 was absent. M2 (or M3 if necessary) was substituted when M1 was absent. M3 was substituted when M2 was absent.

specimens assigned to the Neandertal group (Arcy-sur-Cure #16) and one of the Aurignacian-associated specimens assigned to the Upper Paleolithic modern group (Mladecˇ 2) the posterior probabilities were low (59% and 54%, respectively). Of the unassociated individuals, two (Oase 2 and Vindija 76) classified as Neandertal and one (Oase 1) classified as an Upper Paleolithic modern human. The details of the classifications are presented in the discussion below. Discussion Many of the fragmentary remains associated with the earliest Upper Paleolithic industries are dental. Rather than assume that isolated teeth are uninformative, we aimed to determine whether or not these fragmentary remains could be attributed to a taxonomic group. Our first step was to test whether or not dental nonmetric traits alone were capable of distinguishing between known Neandertals and Upper Paleolithic modern human groups. The second step was to determine the taxonomic affinity of dental remains associated with early Upper Paleolithic industries (w30,000 years ago). Table 8 Cross validation test classifying known Neandertals and Upper Paleolithic modern humans using the modified ‘key’ tooth approach (see text). Correct

Incorrect

Neandertal Upper Paleolithic modern

90% (n ¼ 85) 92% (n ¼ 56)

10% (n ¼ 9) 8% (n ¼ 5)

Total

91% (n ¼ 141)

9% (n ¼ 14)

20

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26

Table 9 Cross validation: Misclassified individuals using the modified ‘key’ trait approach. Individual

Posterior Probability

Specimen

Neandertals classified as Upper Paleolithic modern Krapina 40 70% 73% KDPa 32 Arcy-sur-Cure 9 51% Ochoz 56% Vindija Vi259 75% Petit Puymoyen 3 59% Grotte Taddeo Rep H 64% Spy 2 52% Le Fate XIII 74% Upper Paleolithic moderns classified as Neandertals Dolnı´ Vestonice 16 53% Pavlov 1 61% La Gravette 51% St. Germain-la-Rivie`re (unnumbered) 62% Isturitz IV 1942/1950 66% a

No. traits

Table 11 Details of Table 10 showing number of traits used and posterior probability of classification.

2 4 4 8 3 7 2 4 2 3 1 1 3 6

KDP ¼ Krapina Dental Person.

Step 1: cross validation Based on a cross validation test, our classification approaches turned out to be quite accurate, distinguishing correctly between known Neandertals and Upper Paleolithic modern humans w90% of the time. Because the complete trait classification method allowed a greater number of individuals to be classified, we preferred it to the modified ‘key’ tooth approach. The remaining discussion will refer only to the complete trait method. An examination of the misclassified individuals (Table 6) showed that many retained only a few observable traits, suggesting a relationship between the number of traits and the probability of misclassification. Individuals in the misclassified sample had a maximum of nine traits, but it is important to point out that nine should not be taken as a cutoff above which classification is assumed to be 100% accurate. This low maximum may be due, in part, to small sample size. There are only 16 misclassified individuals, so just by chance they may happen to be those with fewer traits, especially given that the median number of traits in the known sample is only eight. Maxima are notoriously dependent on sample size. However, if we take random samples of 16 from the individuals in the entire known sample, on average the maximum number of traits in these samples is 46 (based on 10,000 random samples). This value is lower than 66, the maximum number of traits in the whole sample, but still much higher than nine, the observed maximum for the misclassified sample. These results suggest that, although it is not possible to establish a secure cutoff for the number of traits above which classification errors decrease substantially, in general, individuals with fewer traits are more likely to be misclassified. On the other hand, many individuals with nine traits or fewer were classified correctly. Examination of these specimens showed that they preserved at least one of the diagnostic teeth outlined earlier (upper incisors, permanent M1, or lower P4). In sum, accurate classification can depend both on how many teeth/traits are preserved and which teeth are preserved. In addition to the relationship between the number of traits and correct classification, there is a relationship between the posterior

No. traits Posterior Probability

Aurignacian/Early Upper Paleolithic classified as Neandertal Mladecˇ composite (9a, 9b, 10) 11 La Ferrassie 7 4 Grotte des Rois R50, Mand B 13 3 Vindija 289 Level Fd Chaˆtelperronian classified as Upper Paleolithic Modern Arcy-sur-Cure 17 2

80% 62% 54% 89% 54%

probability and frequency of correctly classified individuals (Fig. 4). This is expected but encouraging nonetheless. None of the known individuals with posterior probabilities of 90% or more were misclassified. However, there are reasons why posterior probabilities of more than 90% should not be taken as 100% certainty in the classification. First, although cross validation gives a virtually unbiased estimate of the misclassification probabilities, with any estimate there is error around it. Second, the misclassification probabilities assume the unknown individuals come from one of two groups. If the unknown is from a third group, it could still classify with a high posterior probability in one of the two groups. So, we suggest that posterior probabilities of 90% should be taken to indicate very high, but not necessarily 100%, confidence of correct classification. With lower posterior probabilities the frequency of misclassifications increased. At posterior probabilities of 65% or higher, 5% or fewer individuals were misclassified. Therefore, we believe that posterior probabilities above 65% or higher provide relatively secure group assignments. Below 65%, however, group assignment should be interpreted cautiously. Before moving to our discussion of the unknown classification results, one individual in our known sample warrants additional comment because of the role it has played in the debate surrounding interbreeding between Neandertals and Upper Paleolithic modern humans. Lagar Velho. The interpretation of the Lagar Velho child has been the topic of debate for some time because of the claim that it represents the result of long-term Neandertal-modern human

Table 10 Classification results of the ‘unknown’ sample using ‘complete’ trait analysis.

Aurignacian/Early Upper Paleolithic Chaˆtelperronian Unassociated (Oase 1&2, Vindija Vi76)

Neandertal

Upper Paleolithic Modern

15% (n ¼ 5) 93% (n ¼ 14) 67% (n ¼ 2)

85% (n ¼ 29) 7% (n ¼ 1) 33% (n ¼ 1)

Figure 4. Graph illustrating the relationship between posterior probabilities and the percent of correctly classified individuals.

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26

21

hybridization (contrast Duarte et al., 1999 and Zilha˜o and Trinkaus, 2002 with Tattersall and Schwartz, 1999). While clearly a modern human, the primary basis of the claim for hybridization is tibial to femoral length proportions and diaphyseal robusticity, but other aspects of the skeleton (e.g., juxtamastoid projection, semispinalis capitis fossae) are said to indicate a mosaic pattern as well (Zilha˜o and Trinkaus, 2002). The dentition of Lagar Velho is well preserved and it was possible to record variation in 54 of the 71 dental traits. Importantly, the dentition preserves the highly diagnostic upper incisors, upper first molar, lower fourth premolar, and lower first molar. The posterior probability of this specimen belonging to the Upper Paleolithic modern human sample was indistinguishable from 100%. While the specimen does exhibit mild shoveling and moderate lingual tubercle development on the upper I1, it lacks the mesio-distal convexity that is so typical of Neandertal incisors (Crummett, 1994; Martı´non-Torres et al., 2007). Moreover, the upper I1s exhibit mild double shovelingda trait that is found in only 1 of 23 (4.3%) Neandertals. As suggested by the high posterior probability of belonging to the Upper Paleolithic modern group, the remaining teeth are all highly ‘modern’ in their morphology. Step 2: classification of unknown individuals The majority of the Aurignacian/EUP-associated specimens were classified as Upper Paleolithic modern humans (29/34), and most (n ¼ 26) had relatively high (>75%) posterior probabilities. All but one of the Chaˆtelperronian-associated individuals were classified as Neandertals (14/15), and most (n ¼ 9) had relatively high (>75%) posterior probabilities. Three individuals were included in the analysis even though they had no association (Oase 1 and 2) or no known provenience (Vindija Vi 76). Two of these (Oase 2 and Vindija 76) were assigned to the Neandertal group and one (Oase 1) was assigned to the Upper Paleolithic modern group. Below we discuss individual sites from the unknown sample in greater details with reference to their age and archaeological association. Mladecˇ. The Mladecˇ sample included partial maxillary dentitions of two crania (Mladecˇ 1 and 2), and a composite individual (made up of Mladecˇ 9, an upper right canine and P3, and Mladecˇ 10, an upper right M3). All individuals are associated with an Aurignacian industry (Wolpoff et al., 2006), and we predicted that they would be assigned to the Upper Paleolithic modern group. As expected, Mladecˇ 1 and 2 were assigned to this group. Mladecˇ 1 classified with a posterior probability of 99% based on six traits; however, the classification of Mladecˇ 2 was less secure with a posterior probability of only 53% based on nine traits. It is worth noting that only one of the nine traits was scorable on the upper M1 (Carabelli’s cusp), the most diagnostic of the molars. Unexpectedly, the composite individual was assigned to the Neandertal group with a posterior probability of 80% based on 11 traits. The disparate classifications are surprising considering that teeth from all three individuals have been dated directly using 14C to w31,000 BP (Wild et al., 2005). With a posterior probability of 80%, the classification of the composite individual is relatively secure. Indeed, the canine presents traits that are seen in high frequencies in Neandertals (e.g., a well-developed lingual tubercle and shoveling; Fig. 5). It should be pointed out, however, that there is uncertainty surrounding how the Mladecˇ remains were accumulated in the cave. In a review of the context of the Main Cave, from which all three individuals derive (Frayer et al., 2006), Svoboda (2000) suggests that it was neither a living site nor a site that was frequently visited. Rather, he maintains that the human remains likely accumulated by falling through chimneys or fissures in the cave. Therefore, while these results are certainly intriguing, we caution against reading too

Figure 5. The upper canine from the composite Mladecˇ individual showing marked shoveling and lingual tubercle (centimeter scale).

much into the association of a few Neandertal-looking teeth of questionable provenience with the Aurignacian material at Mladecˇ. Grotte de Fonte´chavade. In our analysis, the two teeth from Fonte´chevade (F1B and F2) were assigned to the Upper Paleolithic modern human group with relatively high posterior probabilities (84% and 76%, respectively). The classification for F1B (an upper first molar) was based on four traits, while the classification of F2 (a lower first molar) was based on seven traits. This classification is not unexpected given both teeth derive from level B, which contains Aurignacian artifacts and dates to 31,710  45 BP (Dujardin, 2002). Our classification agrees with the assessment of Garralda (2006: 333) who suggested that, while quite large, the teeth were likely anatomically modern. Font de Gaume. The lower first molar from Font de Gaume (FG2) was assigned with high probability (96%) to the Upper Paleolithic modern human group based on seven traits. Indeed, it has been attributed to an Aurignacian level (de Sonneville-Bordes and Prat, 1969). A previous assessment concluded that it was impossible to determine whether or not FG2 belonged to an Upper Paleolithic modern human or a Neandertal (Gambier et al., 1990). However, we note that the tooth lacks a mid-trigonid crest and it possesses a weak anterior fovea. The absence of these two traits occurs in high frequencies in Upper Paleolithic modern humans. As opposed to being unclassifiable, our analysis strongly suggests the Font de Gaume tooth belonged to an Upper Paleolithic modern human. La Ferrassie. We analyzed three isolated teeth from La Ferrassie: a lower M2 (LF #11), a lower M3 (LF #10), and an upper I1 (LF #7). The lower M2 and M3dtreated as a single individualdwere assigned to the Upper Paleolithic modern human group with a 99% posterior probability based on nine traits.

22

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26

The incisor, however, was assigned to the Neandertal group with a 63% posterior probability. This was not predicted given that it derives from level E’, which is associated with artifacts that are considered ‘ancient’ Aurignacian (de Sonneville-Bordes, 1960) and is thought to date to just earlier than 34,00032,000 BP (Churchill and Smith, 2000a). However, the relatively low posterior probability associated with its classification suggests that caution is warranted in interpreting this tooth. Previously, Gambier et al. (1990) concluded that the incisor is taxonomically unclassifiable, although they noted that the size fits better with fossil modern humans. We also note that the notably short root places the tooth among Upper Paleolithic modern humans (Bailey, 2005). In sum, the tooth possesses a mosaic of features including some that are more closely aligned with Neandertals (e.g., shoveling) and others that are more closely aligned with Upper Paleolithic modern humans (e.g., straight labial contour). This mosaic is reflected in the relatively low posterior probability associated with its classification. Dzerava´ Ska´la (Pa´lffy). The molar from Dzerava´ Ska´la is most likely the germ of a lower left M2 (although without roots, an M1 cannot be excluded). We ran our analysis both ways (as an M1 and as an M2) and the results were the same. Based on seven traits, the tooth was assigned to the Upper Paleolithic group with a high posterior probability (indistinguishable from 100%). The association of this tooth with an archaeological industry is uncertain given that it was recovered from a ‘cytoturbated sediment,’ which contained ‘both Aurignacian and Szeletian type artifacts’ (Churchill and Smith, 2000a:78). Although the original excavator likened this tooth to Neandertals because of its wide anterior fovea (Hillebrand, 1914), the tooth is quite ‘modern’ in its morphology, having only four cusps and lacking a mid-trigonid crest. } . The lower M1 from Ista´llo }sko Cave was assigned to Ista´llosko the Upper Paleolithic modern human group with a high posterior probability (99%) based on seven traits. This molar is said to be associated with an early Upper Paleolithic industry referred to as early Aurignacian or Bachokirian (Mala´n, 1954; Hahn, 1993). Although it is the oldest tooth in our ‘unknown’ sample, dating to 39,700 BP (Allsworth-Jones, 1990), its morphology is quite ‘modern.’ Particularly important to its classification were the absence of a hypoconulid and the absence of a mid-trigonid crest. The absence of these two traits is observed in a much higher frequency in Upper Paleolithic modern humans than in Neandertals. Bacho Kiro. The Bacho Kiro site has yielded several specimens. The specimen #599 comprises a partial mandible with a deciduous m2 and permanent M1. Based on six traits on the M1 (X fissure pattern, 5 cusps, and lack of: deflecting wrinkle, distal trigonid crest, cusp 6, and cusp 7), it was assigned with a moderate posterior probability (66%) to the Upper Paleolithic modern human group. Bacho Kiro #599 comes from layer 7 of the site. This layer yielded a limited series of artifacts variably assigned to the Bachokirian or to the Aurignacian. However, the AMS date on charcoal at 32,200  780 BP (OxA 3182; Hedges et al., 1994) obtained in this layer supports the latter. Glen and Kaczanowski (1982) found the tooth dimensions of the permanent M1 to be indistinguishable from early modern and Neandertal samples. Churchill and Smith (2000a) concluded generally that the specimens from Bacho Kiro are ambiguous but closer to modern humans. Although the classification is not unexpected given its context, our confidence in it is not particularly high. This is partially because the posterior probability is in the ‘caution’ range (less than 65%), and partially because we were only able to examine a cast of the original. Normally, this would not be an issue except in this case the cast does not permit us to determine whether the fissure dividing the mesial cusps is natural (sagittal sulcus) or a crack. Examination

of a photograph of the original from Glen and Kaczanowski (1982) did not resolve this matter. Therefore, the mid-trigonid crest was recorded as ‘missing data.’ Since the mid-trigonid crest is a highly diagnostic trait, its presence would push the classification to the Neandertal group, while its absence would make the posterior probability of belonging to the Upper Paleolithic modern human group much higher. Ultimately, we feel the classification of this tooth should wait until examination of the original is possible. Brassempouy. Although we were not allowed to examine the Brassempouy dental remains directly, we were able to record expression for 19 traits based on the description and photographs published by Henry-Gambier et al. (2004). The eight teeth were combined into five individuals (see Table 2) based on wear, development, and comments in the original publication. The Brassempouy dental remains are associated with an early Aurignacian industry and have been dated to 30,000–34,000 BP (HenryGambier et al., 2004). As expected, all five ‘individuals’ were assigned to the Upper Paleolithic modern human group. Four of these were assigned with high posterior probabilities (w90%), while the fifth had a posterior probability of 63% Henry-Gambier et al. (2004) have claimed that it is not possible to assign the teeth to Neandertals or modern humans because most of the traits they possess can be found in both groups. Our analysis assigned these teeth to the Upper Paleolithic modern human group because they possess combinations of traits that are typical of modern humans, which include an upper M2 with a reduced hypocone; a lower P4 that lacks asymmetry, possesses a single lingual cusp, and lacks a transverse crest; and a lower M2 that lacks the mid-trigonid crest. Thus, our analysis supports earlier conclusions about the Brassempouy teeth based on visual inspection only (Bailey and Hublin, 2005) and provides quantification for this earlier assessment in the form of posterior probabilities. Pes¸tera cu Oase. Two individuals from the cave of Pes¸tera cu Oase possess partial dentitions. Our results show that, based on 12 traits, Oase 1 has a high posterior probability (99%) of belonging to the Upper Paleolithic modern human group. Oase 2, on the other hand, has a high posterior probability (97%) of belonging to the Neandertal group based on six traits (cusp 5, Carabelli’s cusp on upper M1 and M2, unreduced hypocone on upper M2, and unreduced upper M3). Neither specimen is associated with an archaeological industry. However, their age is similar to, and possibly older than, the Upper Paleolithic-associated specimens in our unknown sample (Trinkaus et al., 2003a,b; Rougier et al., 2007). Given that the cranium of Oase 2 is clearly not that of a Neandertal (Rougier et al., 2007), the assignment of this individual to the Neandertal group was unexpected. Trinkaus (2007) has suggested that, while essentially ‘modern,’ both Oase 1 and 2 exhibit a mosaic of cranio-dental features, some of which are archaic (e.g., dental proportions, long and flat frontal bone), and others apparently derived towards anatomically modern humans (parietal curvature, absence of supraorbital torus) or towards Neandertals (unilateral lingual bridging of the mandibular canal). It is important to note that the dental traits aligning Oase 2 with Neandertals are archaic in nature, as they are observed in other fossil hominins as well (Bailey, 2002b, 2006; Martı´non-Torres et al., 2007). It is unfortunate that incisor morphology could not be assessed (teeth are missing), and that the upper M1s are too worn to ascertain occlusal polygon shape and occlusal polygon area, since these are features that are likely derived for the Neandertals/ Neandertal lineage (Bailey, 2004; Go´mez-Robles et al., 2007). Considering that some of the most diagnostic features of the upper dentition could not be assessed and that our approach is not 100% accurate, we caution against over-interpreting the classification of Oase 2.

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26

Grotte des Rois. The analysis of the entire Les Rois sample (n ¼ 15) showed that they have an overwhelmingly ‘modern’ signal. Fourteen of the fifteen individuals had high posterior probabilities of belonging to the Upper Paleolithic modern human group (>80%). This is not unexpected considering the teeth are associated with an Aurignacian industry (Dujardin, 2000) and most come from unit B, which has been dated to 28,715  145 BP using AMS C14. One individual (Mandible B), however, was classified as Neandertal with a low posterior probability (54%) based on 13 traits. Ramirez Rozzi (pers. comm.) has recently suggested that Mandible B represents a Neandertal, based largely on the asymmetrical P4 together with some aspects of the corpus. Trinkaus (2007) has also argued that the specimens from Les Rois are mixed in morphology. While the P4 is asymmetrical, a large sagittal crack in the crown exaggerates this feature and the remaining aspects of the tooth are distinctively not Neandertal-like (it lacks a transverse crest and multiple lingual cusps: Fig. 6). In the end, we do not consider the posterior probability of 54% to be compelling enough to conclude, based on dental traits, that there were Neandertals present at Les Rois. Vindija. All three ‘unknown’ individuals from the Vindija sample were assigned to the Neandertal group. For one of these individuals (Vi 229, an upper M) the provenience is unknown. Its classification was based on a single trait (lack of M2 hypocone reduction) and it was assigned with a low posterior probability (60%). Two teeth recovered from level G1 were treated as a single individual on the basis of morphology and wear (Vi 287, an upper C, and Vi 290, an upper I1). Their assignment to the Neandertal group, with a posterior probability of 95%, was based on eight traits. This is interesting, given that the G1 layer has been interpreted as a ‘‘transitional’’ industry (Karavanic´, 1995; Ahern et al., 2004). It has yielded a scarce industry composed of Mousterian-like artifacts with some Aurignacian-looking bone artifacts, among them only one diagnostic artifact: a split-base bone point. However, some have questioned the integrity of the assemblage based on clear signs

23

of cryoturbation and a recorded high level of carnivore activity (Zilha¨o and d’Errico, 1999a,b). Therefore, whether the G1 layer of Vindjia truly represents a genuine early Upper Paleolithic layer remains an open question. The final individual, recovered from the Aurignacian level Fd (Vi 289, an I2), was also assigned to the Neandertal group with a high posterior probability (88%). The two traits used to assign this individualdmarked shoveling and lingual tubercle developmentdare, indeed, very Neandertal-like. Arcy-sur-Cure. All but one ‘individual’ from the Arcy-sur-Cure sample (n ¼ 13) were assigned to the Neandertal group with posterior probabilities ranging from 59% to 99.9%. Most teeth were classified with relatively high posterior probabilities (>75%); those that were not (n ¼ 4) were classified on the basis of single traits. A single partial tooth crown was assigned to the Upper Paleolithic modern human group based on two traits (lower M1 lacking C7 and having more than four cusps). It had a low posterior probability (54%) of belonging to this group. For this reason, we do not place much confidence in its assignment. The overwhelming Neandertal signal of this sample confirms conclusions of an earlier study, based on visual inspection (Bailey and Hublin, 2006). This is not surprising given the teeth were recovered from deposits associated with a Chaˆtelperronian industry, likely dating ca. w32,000 BP (see Bailey and Hublin, 2006 for review) and that a temporal bone from an infant has also been determined to belong to a Neandertal (Hublin et al., 1996). St. Ce´saire. Both specimens from St. Ce´saire were assigned to the Neandertal group. The more complete individual (St. Ce´saire 1), represented primarily by the right half of both upper and lower dentitions, preserved 51 traits and was classified with a posterior probability of 98%. St. Ce´saire 2 (an upper canine and upper M2) was classified with a lower probability (66%), presumably because it preserved only three traits, for which the frequency differences between Neandertals and Upper Paleolithic moderns is minimal (lack of Cusp 5 and Carabelli’s cusp on M2 and canine shoveling). Thus, our dental morphological assessment agrees with previous assessments of the St. Ce´saire individuals as Neandertal based on cranial morphology (Le´veˆque and Vandermeersch, 1980; Stringer et al., 1984; Trinkaus et al., 1998). In several cases our discussion refers to diagnostic characters, for which frequencies in Upper Paleolithic moderns and Neandertals are very different (Bailey, 2002a, 2004, 2006; Bailey and Lynch, 2005). In some cases the presence or absence of one or more of these traits plays a significant role in classification. One could argue that individuals who preserve highly diagnostic teeth could be classified without going through the above statistical exercise. This may be so. However, we feel that our approach is a better way to proceed because it allows us to quantify the probability that an individual belongs to a particular group. Our method also does not require that individuals possess the most diagnostic teeth (e.g., upper incisors, lower P4, upper and lower M1), as in some cases correct classifications were made without them. Finally, we feel our analysis of the effect of sample size, number of traits, and posterior probabilities on group assignment provides the researcher with important information for interpreting the classification of individuals. Conclusions

Figure 6. The lower right P4 from the mandible from Les Rois showing damage (a wide sagittal crack) to the crown.

We have shown that dental features, non-metric traits in particular, provide important information for distinguishing between Neandertals and modern humans during the late Pleistocene. Our approach, which emphasizes trait frequencies as well as trait combinations, shows very clearly that dental non-metric traits alone can provide a wealth of information for taxonomic

24

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26

identification in this context. While it may be ideal to have multiple teeth represented in a given specimen, it is also possible to correctly classify isolated diagnostic teeth with high probability (e.g., isolated M1s from Krapina and the Mousterian levels of Arcy-sur-Cure were assigned correctly to Neandertals with 81% and 82% probability, respectively). In total, our taxonomic classifications, which relied primarily on incomplete dentitions and often single teeth, are 89% accurate; thus they are more reliable than sex determinations from the skull (Duric et al., 2004; Williams and Rogers, 2006). Therefore, we believe that we have developed a reliable method for distinguishing between Neandertals and modern humans for individuals represented solely by teeth. It was certainly possible to distinguish Neandertal from modern human teeth before we undertook this study. What we have contributed is a way to quantify the probability of these taxonomic assignments. This quantification is important, especially when dealing with two human groups that differ in trait frequency, rather than presence or absence of certain traits. In addition, we were able to apply our approach to an unknown fossil sample to determine that 85% of the specimens/individuals associated with the Aurignacian/EUP can be attributed to modern humans. In contrast, 93% of the specimens/individuals associated with the Chaˆtelperronian can be attributed to Neandertals. The percentages of specimens classified as Neandertals in the first group and as Upper Paleolithic modern humans in the second are comparable to the percentages of misclassified specimens observed in our Neandertal and Upper Paleolithic modern human reference samples. There is, therefore, no reason to assume that these distributions primarily result from anything else but the limitations of the method and the expected variation within homogeneous populations of early Upper Paleolithic modern humans and late Neandertals. Several teeth associated with the Aurignacian but that were previously considered ‘undiagnostic’ (Gambier et al., 1990; Churchill and Smith, 2000a; Garralda, 2006) have been assigned to the Upper Paleolithic modern human group (e.g., Font de Gaume 2, Dzerava´ Ska´la). In other cases the results of our study confirm earlier conclusions based on non-statistical methods: the Chaˆtelperronianassociated Arcy-sur-Cure teeth belong to Neandertals (Bailey and Hublin, 2006) and the Aurignacian-associated Brassempouy teeth belong to modern humans (Bailey and Hublin, 2005). Our study focused exclusively on the affinities of late Pleistocene fossils in order to address specific questions relevant to this time period, but the potential is there to extend our approach to other time periods as well. In order to do so, however, it would be necessary to have secure associations between teeth and crania (and enough of them) to be able to establish a known sample against which unknown isolated teeth or partial dentitions could be compared. Acknowledgements We thank S. Anto´n and the three anonymous reviewers for their helpful suggestions for improving this manuscript. We also thank F. Bailey, C. Bailey, and C. Weaver for their proof reading assistance, C. Verna for background research on several specimens and Claire Letourneux for help with sorting out the Isturitz sample. In addition, SEB is grateful to all the people who provided access to, and assistance with, the fossils used in this study: E. Trinkaus, at Washington University, St. Louis; C. Stringer and R. Kruszynski at the Natural History Museum, London; Y. Rak of the Sackler School of Medicine, Tel Aviv; J. Zilha˜o at Bristol University; G. Manzi at the University of Rome; G. Giacobini of the University of Turin; F. Mallegni of the University of Pisa; G. Koufos of the Aristotle University of Thessaloniki; P. Mennecier of the Museum of Mann, Paris; H. de Lumley at the Institute of Human Paleontology, Paris;

J. Leopold of the Museum of National Antiquities, St. Germain-enLaye; M. Tavaso and F. Marchal at the Laboratory of Historical Geology, Marseille; J.-J. Cleyet-Merle and A. Morala at the National Museum of Prehistory, Les Eyzies; V. Merlin-Anglade and G. Marchesseau at the Museum of Perigord; E. Ladier at the Museum of Natural History, Montauban; S. Dusek at the Museum of Prehistory, Weimar; H.-E. Joachim at the State Museum of the Rhine, Bonn; W. Menghin at the Museum of Prehistory, Berlin; the late N. Farsan of the University of Heidelberg; M. Teschler-Nicola at the Natural History Museum, Vienna; R. Orban and P. Semal of the Royal Institute of Natural Sciences of Belgium, Brussells; J. Radovcˇic´ at the Croatian Natural History Museum, Zagreb; M. Paunovicˇ of the Institute for Quaternary Geology and Paleontology, Zagreb; J. Svoboda of the Institute of Archaeology, Paleolithic and Paleoethnology stonice; I. Pap of the Hungarian National Research Center, Dolnı´ Ve History Museum, Budapest and M. Dockalova at the Moravian Museum, Brno. Data collection and research were supported by funding from the Max Planck Society, as well as grants from the National Science Foundation (BCS-0002481) and LSB Leakey Foundation. Appendix A: Samples used in this study. All samples are original fossils unless otherwise noted. Sample

Institution

Krapina

Croatian Natural History Museum, Zagreb Museum of Man, Paris Institute for Human Paleontology, Paris Museum of Perigord, Pe´rigueux Museum of Man, Paris Natural History Museum, London Moravian Museum, Brno Moravian Museum, Brno Institute for Human Paleontology, Paris Faculty of Medicine, Marseilles Institute for Prehistory, Jena University of Turin National Museum of Prehistory, Les Eyzies University of Turin University of Turin University of Pisa University of Pisa Italian Institute of Human Paleontology, Rome University of Rome Royal Institute of Natural Sciences of Belgium, Brussels Museum of Prehistory and Early History, Berlin Museum of National Antiquities, St. Germain-en-Laye; Museum of Man, Paris Washington University, courtesy of E. Trinkaus Washington University, courtesy of E. Trinkaus Washington University, courtesy of E. Trinkaus Washington University, courtesy of E. Trinkaus National Museum of Prehistory, Les Eyzies courtesy of Michelle Glantz, University of Colorado Hungarian National History Museum, Budpest National Museum of Prehistory, Les Eyzies National Museum of Prehistory, Les Eyzies

Malarnaud Monsempron Regourdou Arcy-sur-Cure Gibraltar: Devil’s Tower Ochoz Ku˚lna Petit-Puymoyen Hortus (casts) Taubach La Fate Roc du Marsal Ciota Ciara (Monte Fenera) Ciutarun 1 (Monte Fenera) Grotte Taddeo Grotte Poggio Guattari Saccopastore Spy Le Moustier La Quina

Montgaudier (casts) Combe Grenal (casts) Chateauneuf (casts) Marillac (casts) La Ferrassie Obi Rakhmat (casts) Subalyuk Les Vachons Roc de Combe Capelle

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26 Appendix A. (continued) Sample

Institution

Lagar Velho

Portuguese Institute of Archaeology, Lisbon Institute of Archaeology, Paleolithic and Paleoethnology Research Center, Dolnı´ Veˇstonice Institute of Archaeology, Paleolithic and Paleoethnology Research Center, Dolnı´ Veˇstonice Museum of Man, Paris Institute for Human Paleontology, Paris Institute for Human Paleontology, Paris Natural History Museum, Vienna National Museum of Prehistory, Les Eyzies Institute for Human Paleontology, Paris National Museum of Prehistory, Les Eyzies Hungarian National History Museum, Budapest Hungarian National History Museum, Budapest Portuguese Institute of Archaeology, Lisbon Portuguese Institute of Archaeology, Lisbon National Museum of Prehistory, Les Eyzies National Museum of Prehistory, Les Eyzies Institute for Human Paleontology, Paris Museum of Man, Paris National Museum of Prehistory, Les Eyzies; Institute for Human Paleontology, Paris Natural History Museum, London State Museum of the Rhine, Bonn Institute for Human Paleontology, Paris Institute for Human Paleontology, Paris Natural History Museum, Vienna Institute for Human Paleontology, Paris National Museum of Prehistory, Les Eyzies Hungarian National History Museum, Budapest Hungarian National History Museum, Budapest Institute for Quaternary Geology and Paleontology, Zagreb Natural History Museum, London Henry-Gambier et al. (2004)

Dolnı´ Veˇstonice

Pavlov

Abri Pataud Abri Blanchard Abri Labatut Mieslingtal Grotte des Abeilles Lespugue La Gravette Balla Barlang Bervavolgy Gruta do Caldeira˜o Cisterna La Madeleine Peche de la Boissiere Farincourt Laugerie Basse St. Germaine-la-Rivie`re

Gough’s Cave Oberkassel Isturitz La Chaud Mladecˇ Fontechevade Font de Gaume Derava´ Skala } Ista´llo´sko Vindija Bacho Kiro (cast) Brassempouy (photographs/ descriptions) Grotte des Rois St. Ce´saire Oase

Institute for Human Paleontology, Paris University of Bordeaux Speleological Institute, Bucharest

References Ahern, J.C., Karavanic´, I., Paunovic´, M., Jankovic´, I., Smith, F.H., 2004. New discoveries and interpretations of hominid fossils and artifacts from Vindija Cave, Croatia. J. Hum. Evol. 46, 27–67. Allsworth-Jones, P., 1990. The Szeletian and the stratigraphic succession in central Europe and adjacent areas: main trends, recent results, and problems for resolution. In: Mellars, P. (Ed.), The Emergence of Modern Humans: an archaeological perspective. Cornell University Press, New York, pp. 160–242. Bailey, S.E., 2002a. A closer look at Neanderthal postcanine dental morphology. I. The mandibular dentition. Anat. Rec. 269, 148–156. Bailey, S.E., 2002b. Neandertal Dental Morphology: implications for modern human origins. Ph.D. Dissertation, Arizona State University. Bailey, S.E., 2004. A morphometric analysis of maxillary molar crowns of MiddleLate Pleistocene hominins. J. Hum. Evol. 47, 183–198. Bailey, S.E., 2005. Diagnostic dental differences between Neandertals and Upper Paleolithic modern humans: getting to the root of the matter. In: Zadzinska, E. (Ed.), Current Trends in Dental Morphology Research. University of qo´dz´ Press (Poland), qo´dz´, pp. 201–210. Bailey, S.E., 2006. Beyond shovel shaped incisors: Neandertal dental morphology in a comparative context. Period. Biol. 108, 253–267.

25

Bailey, S.E., 2007. The evolution of non-metric dental variation in Europe. Mitt. Ges. Urgesch. 15, 9–30. Bailey, S.E., 2008. Inter and intraspecific variation in Pan tooth crown morphology: implications for Neandertal taxonomy. In: Irish, J.D., Nelson, G.C. (Eds.), Technique and Application in Dental Anthropology. Cambridge University Press, Cambridge, pp. 293–317. Bailey, S.E., Hublin, J.-J., 2005. Who made the Early Aurignacian? A reconsideration of the Brassempouy dental remains. Bull. Mem. Soc. Anthropol. Paris 17, 115–121. Bailey, S.E., Hublin, J.-J., 2006. Dental remains from Grotte du Renne at Arcy-surCure (Yonne). J. Hum. Evol. 50, 485–508. Bailey, S.E., Lynch, J.M., 2005. Diagnostic differences in mandibular P4 shape between Neandertals and anatomically modern humans. Am. J. Phys. Anthropol. 126, 268–277. Berry, D.A., 1996. Statistics: a Bayesian perspective. Wadsworth, Belmont. Burnett, S.E., 1998. Maxillary Premolar Accessory Ridges (MxPAR): Worldwide Occurrence and Utility in Population Differentiation. Masters Thesis, Arizona State University. Casella, G., Berger, R.L., 2002. Statistical Inference. Duxbury, Pacific Grove. Churchill, S., Smith, F., 2000a. Makers of the early Aurignacian of Europe. Yrbk. Phys. Anthropol. 43, 61–115. Churchill, S., Smith, F., 2000b. A modern human humerus from the early Aurignacian of Vogelherdhohle (Stetten, Germany). Am. J. Phys. Anthropol. 112, 251–273. Conard, N.J., Grootes, P.M., Smith, F.H., 2004. Unexpectedly recent dates for human remains from Vogelherd. Nature 430, 198–201. Crummett, T., 1994. The Evolution of Shovel Shaping: Regional and Temporal Variation in Human Incisor Morphology. Ph.D. Dissertation, University of Michigan. Crummett, T., 1995. The three-dimensions of shovel-shaping. In: Moggi-Cecchi, J (Ed.) Aspects of Dental Bioology, Paleontology, Anthropology, and Evolution. International Institute for the Study of Man, Florence. pp. 305-313 Dahlberg, A., 1945. The changing dentition of man. J. Am. Dent. Assoc. 32, 676–680. Demars, P.Y., Hublin, J.-J., 1989. La transition ne´andertaliens/hommes de type moderne en Europe occidentale: aspects paleontologiques et culturels. In: Otte, M., Laville, H. (Eds.), L’homme Ne´andertal 7: l’extinction. ERAUL, Lie`ge, pp. 29–42. Duarte, C., Maurı´cio, J., Pettitt, P., Souto, P., Trinkaus, E., van der Plicht, H., Zilha˜o, J., 1999. The early Upper Paleolithic human skeleton from the Abrigo do Lagar Velho (Portugal) and modern human emergence in Iberia. Proc. Natl. Acad. Sci. U.S.A. 96, 7604–7609. Dujardin, V., 2000. Une excursion en Charente pre´historique en 1935. D’apre`s les cahiers de Jean Morel. Bull. Soc. Archaeol. Hist. Charente 1, 35–48. Dujardin, V., 2002. Datations au radiocarbone. http://www.vdujardin.com/14C.html (accessed 10.02.09). Dujardin, V., Tymula, S., 2004. Relecture chronologique de sites anciennement fouille´s: datation des sites pale´olithiques et e´pipale´olithiques de Poitou-Charentes. Bull. Me´m. Soc. Pre´hist. France 102, 771–788. Duric, M., Rakocevic, Z., Donic, D., 2004. The reliability of sex determination of skeletons from forensic context in the Balkans. Forensic Sci. Int. 147, 159–164. Frayer, D., 1992. The persistence of Neanderthal features in post-Neanderthal Europeans. In: Bra¨uer, G., Smith, F. (Eds.), Continuity or Replacement: Controversies in Homo sapiens Evolution. AA Balkema, Rotterdam. Frayer, D., 1997. Perspectives on Neanderthals as ancestors. In: Clark, G., Willermet, C. (Eds.), Conceptual Issues in Modern Human Origins Research. Aldine de Gruyter, New York, pp. 107–116. Frayer, D., Jelı´nek, J., Oliva, M., Wolpoff, M., 2006. Aurignacian male crania, jaws and teeth from the Mladecˇ Caves, Moravia, Czech Republic. In: Teschler-Nicola, M. (Ed.), Early Modern Humans at the Moravian Gate: the Mladecˇ Caves and Their Remains. Springer, New York, pp. 185–272. Gambier, D., 1989. Fossil hominids from the early Upper Palaeolithic (Aurignacian) of France. In: Mellars, P., Stringer, C. (Eds.), The Human Revolution: Behavioral and Biological Perspectives on the Origins of Modern Humans. Princeton University Press, Princeton, pp. 194–211. Gambier, D., Houet, F., Tillier, A.M., 1990. Dents de Font de Gaume Chatelperronien et Aurignacien, et de La Ferrassie (Aurignacien ancien) en Dordogne. Pale´o 2,143–152. Garralda, M.D., 2006. Las gentes del Paleolitico Superior antiguo de Europa Occidental. In: Maillo, J.M., Baquedano, E. (Eds.), Zona Arqueolo´gica 7. Miscela´nea en homenaja a Victoria Cabrera. Museo Arqueolo´gico Regional, Madrid, pp. 320–335. Glen, E., Kaczanowski, K., 1982. Human remains. In: Kozkowski, J.K. (Ed.), Excavation in the Bacho Kiro Cave (Bulgaria) Final Report. PWN, Warszawa, pp. 75–79. Go´mez-Robles, A., Martino´n-Torres, M., Bermu´dez De Castro, J.M., Margvelashvili, A., Bastir, M., Arsuaga, J.L., Pe´rez-Pe´rez, A., Estebaranz, F., Martı´nez, L.M., 2007. A geometric morphometric analysis of hominin upper first molar shape. J. Hum. Evol. 55, 627–638. Hahn, J., 1993. L’origine du Pale´olithique supe´rieur en Europe centrale: les datations C14. In: Valde´s, V. (Ed.), El Orı´gen del hombre moderno en el suroeste de Europe. Universidad Nacional de Educatio´n a Distancia, Madrid, pp. 61–80. Harrold, F.B., 1989. Mousterian, Chaˆtelperronian and early Aurignacian in Western Europe: continuity or discontinuity? In: Mellars, P., Stringer, C.B. (Eds.), The Human Revolution. Princeton University Press, Princeton, pp. 677–713. Hedges, R.E.M., Housley, R.A., Bronk Ramsey, C., van Klinken, G.J., 1994. Radiocarbon dates from the Oxford AMS system: Archaeometry datelist 18. Archaeometry 36, 337–374. Henry-Gambier, D., Maureille, B., White, R., 2004. Vestiges humains des niveaux de l’Aurignacien ancien du site de Brassempouy (Landes). Bull. Mem. Soc. Anthropol. Paris 16, 49–87.

26

S.E. Bailey et al. / Journal of Human Evolution 57 (2009) 11–26

Hillebrand, J., 1914. Az 1913 e´vi barlangkutatasaim ereme´nyei. Barlangkutates 2, 115–124. Hublin, J.-J., Spoor, F., Braun, M., Zonneveld, F.W., Condemi, S., 1996. A late Neanderthal associated with Upper Palaeolithic artefacts. Nature 381, 224–226. Irish, J., 1998. Ancestral dental traits in recent sub-Saharan Africans and the origins of modern humans. J. Hum. Evol. 34, 81–98. Karavanic´, I., 1995. Upper Paleolithic occupation levels and late-occurring Neandertal at Vindija Cave (Croatia) in the context of Central Europe and the Balkans. J. Anthropol. Res. 51, 10–35. Klein, R., 1999. The Human Career. University of Chicago Press, Chicago. Krzanowski, W., 2000. Principles of Multivariate Analysis: a User’s Perspective, revised ed. Oxford University Press, Oxford. Le´veˆque, F., Backer, A.M., Guilbaud, M. (Eds.), 1993. Context of a Late Neandertal. Prehistory Press, Madison, WI. Le´veˆque, F., Vandermeersch, B., 1980. De´couverte de restes humains dans un niveau castelperronien a` Saint-Ce´saire (Charente-Maritime). C.R. Acad. Sci. Paris 291, 187–189. Lynch, M., 1989. Phylogenetic hypotheses under the assumption of neutral quantitative-genetic variation. Evolution 43, 1–17. Mala´n, M., 1954. Zahnkeim aus der zweiten Aurignacien Schicht der Ho¨le von } sko. Acta Archaeol. Hung. 5, 145–148. Ista´llo Martı´non-Torres, M., Bermudez De Castro, J.M., Go´mez-Robles, A., Arsuaga, J.L., Carbonell, E., Lordkipanidze, D., Manzi, G., Margvelashvili, A., 2007. Dental evidence on hominin dispersals during the Middle Pleistocene. Proc. Natl. Acad. Sci. U.S.A. 104, 13279–13282. Martino´n-Torres, M., Mastir, M., Bermu´dez De Castro, J.M., Go´mez, A., Sarmiento, S., Muela, A., Arsuaga, J.L., 2006. Hominin lower second premolar morphology: evolutionary inferences through geometric mrophometric analysis. J. Hum. Evol. 50, 523–533. Mellars, P., 2004. Neanderthals and the modern human colonization of Europe. Nature 432, 461–465. Mizoguchi, Y., 1985. Shovelling: A Statistical Analysis of its Morphology. University of Tokyo Press. Pagel, M., 2002. Phylogenetic inference: methods. In: Pagel, M. (Ed.), Encyclopedia of Evolution. Oxford University Press, Oxford, pp. 895–904. Radovcˇic´, J., Smith, F.H., Trinkaus, E., Wolpoff, M.H., 1988. The Krapina Hominids: an Illustrated Catalog of Skeletal Collection. Mladost Publishing, Croatian Natural History Museum, Zagreb. Rougier, H., Milota, S., Rodrigo, R., Gherase, M., Sarcina, L., Moldovan, O., Zilha˜o, J., Constantin, S., Franciscus, R., Zollikofer, C., Ponce De Leo´n, M.S., Trinkaus, E., 2007. Pes¸tera cu Oase 2 and the cranial morphology of early modern Europeans. Proc. Natl. Acad. Sci. U.S.A. 104, 1165–1170. Scott, G.R., Turner II, C.G., 1997. The Anthropology of Modern Human Teeth. Dental Morphology and its Variation in Recent Human Populations. Cambridge University Press, Cambridge. Smith, F., Trinkaus, E., Pettitt, P., Karavanicˇ, I., Paunovicˇ, M., 1999. Direct radiocarbon dates for Vindija G1 and Velika Pec´ina Late Pleistocene hominid remains. Proc. Natl. Acad. Sci. U.S.A. 96, 12281–12286. de Sonneville-Bordes, D., 1960. Le Pale´olithique Supe´rieur en Pe´rigord. Delmas, Bordeaux. de Sonneville-Bordes, D., Prat, F., 1969. De´couvertes re´centes de Pale´olithique supe´rieur a` la grotte de Font de Gaume (Dordogne). Quaternaria 11, 115–132. Souday, C., 2008. Analyse morphome´trique des molaires de´ciduales et de´finitives dans le genre Homo: prespectives phyloge´ne´tiques et bioge´ographiques. Docteur, Muse´um National d’Histoire Naturelle.

Stringer, C., Hublin, J., Vandermeersch, B., 1984. The origin of anatomically modern humans in Western Europe. In: Smith, F., Spencer, F. (Eds.), The Origins of Modern Humans: a World Survey of the Fossil Evidence. Alan, R. Liss, New York, pp. 51–135. Svoboda, J., 2000. The depositional context of the Early Upper Paleolithic human prusy (Zlaty  ku˚n ˇ ) and Mladecˇ Caves, Czech Republic. J. fossils from the Kone Hum. Evol. 38, 523–536. Tattersall, I., Schwartz, J.H., 1999. Hominids and hybrids: the place of Neanderthals in human evolution. Proc. Natl. Acad. Sci. U.S.A. 96, 7117–7119. Thiele, K., 1993. The Holy Grail of the perfect character: the cladistic treatment of morphometric data. Cladistics 9, 275–304. Trinkaus, E., 2005. Early modern humans. Annu. Rev. Anthropol. 34, 207–230. Trinkaus, E., 2007. European early modern humans and the fate of the Neandertals. Proc. Natl. Acad. Sci. U.S.A. 104, 7367–7372. Trinkaus, E., Milota, S., Rodrigo, R., Mircea, G., Moldovan, O., 2003a. Early modern human cranial remains from the Pes¸tera cu Oase, Romania. J. Hum. Evol. 45, 245–253. Trinkaus, E., Moldovan, O., Milota, S., Bilgar, A., Sarcina, L., Athreya, S., Bailey, S.E., Rodrigo, R., Mircea, G., Higham, T., Ramsey, C., van der Plicht, J., 2003b. An early modern human from the Pes¸tera cu Oase, Romania. Proc. Natl. Acad. Sci. U.S.A. 100, 11231–11236. Trinkaus, E., Ruff, C.B., Churchill, S.E., Vandermeersch, B., 1998. Locomotion and body proportions of the Saint-Cesaire 1 Chaˆtelperronian Neandertal. Proc. Natl. Acad. Sci. U.S.A. 95, 5836–5840. Turner II, C.G., Nichol, C.R., Scott, G.R., 1991. Scoring procedures for key morphological traits of the permanent dentition: The Arizona State University Dental Anthropology System. In: Kelley, M., Larsen, C. (Eds.), Advances in Dental Anthropology. Wiley Liss, New York, pp. 13–31. Turner II, C.G., Scott, G.R., 1977. Dentition of Easter Islanders. In: Dahlberg, A., Graber, T. (Eds.), Orofacial Growth and Development. Mouton Publishers, The Hague, pp. 229–249. Turq, A., Normand, C., Valladas, H., 1999. Saint-Martin-d’Arberoue: Grotte d’Isturitz: Bilan Scientifique 1998. Direction Re´gionale des Affaires Culturelles, Service Re´gional de l’Archae´ologie, Bordeaux. Weaver, T.D., Roseman, C.C., Stringer, C.B., 2008. Close correspondence between quantitative- and molecular-genetic divergence times for Neandertals and modern humans. Proc. Natl. Acad. Sci. U.S.A. 105, 4645–4649. Wild, E.M., Teschler-Nicola, M., Kutschera, W., Steier, P., Trinkaus, E., Wanek, W., 2005. Direct dating of Early Upper Paleolithic human remains from Mladecˇ. Nature 435, 322–335. Williams, B.A., Rogers, T.L., 2006. Evaluating the accuracy and precision of cranial morphological traits for sex determination. J. Forensic Sci. 51, 729–735. Wolpoff, M., 2001. Modern human ancestry at the peripheries: A test of the replacement theory. Science 291, 293–297. Wolpoff, M., Frayer, D., Jelı´nek, J., 2006. Aurignacian female crania and teeth from the Mladecˇ Caves, Moravia, Czech Republic. In: Teschler-Nicola, M. (Ed.), Early Modern Humans at the Moravian Gate. The Mladecˇ Caves and Their Remains. SpringerWienNewYork, New York, pp. 273–340. Zilha¨o, J., d’Errico, F., 1999a. The chronology and taphonomy of the earliest Aurignacian and its implications for the understanding of Neanderthal extinction. J. World Prehist. 13, 1–68. Zilha˜o, J., d’Errico, F., 1999b. Reply to Straus, Mellars and Otte. Curr. Anthropol. 40, 355–364. Zilha˜o, J., Trinkaus, E. (Eds.), 2002. Portrait of the Artist as a Child. The Gravettian Human Skeleton from the Abrigo do Lagar Velho and its Archaeological Context. Instituto Portugues de Arqueologia, Lisbon.