- Email: [email protected]
Cue words and the topic structure of spoken discourse: The case of Swedish men ‘but’
ELSEVIER
Journal of Pragmatics 33 (2001) 1061-1081 www.elsevier.nl/locate/pragma
Cue words and the topic structure of spoken discourse" The case of Swedish m e n 'but Merle Home,* Petra Hansson, G6sta Bruce, Johan Frid, Marcus Filipsson Department of Linguistics and Phonetics, Lund University, Helgonabacken 12, S-223 62 Lund, Sweden Received 24 March 1999; revised version 6 June 2000
Abstract The Swedish cue word men 'but' can mark the boundary between both different topic units as well as topic-internal units in spontaneous speech. The goal of this study is to see if these two functions of men can be distinguished on the basis of their local prosodic correlates and co-occurring lexical items. Men-tokens in spontaneous narrations were labelled as to their function, first using text-only data. The 'strong' tokens (categorized identically by all labellers) were subsequently seen to be clearly differentiated into two classes on the basis of related prosodic parameters and co-occurring lexical items. This distinction was, however, not found for the corresponding 'weak' tokens which were subsequently relabelled using both text and speech nor for the data-base as a whole. A test using a neural network trained using strong tokens was seen to be able to correctly categorize 90% of the strong men-tokens as to their associated boundary-type (topic-shift vs. topic-internal). The results show that cue words along with their prosodic correlates and co-occurring lexical items constitute a constellation of important information for understanding how segmentation of spoken discourse is produced and understood. © 2001 Elsevier Science B.V. All rights reserved. Keywords: Cue word; Discourse marker; Prosody; Spoken discourse; Topic structure; Speech recognition; Swedish
~ This research has been supported by a grant from the Swedish HSFR/NUTEK Language Technology Programme and by Telia Research. We would like to thank Mechtild Tronnier for assistance with statistical analyses and St6phane Di Cesar6 for help with programming. We are also very grateful to two anonymous referees whose comments have led to considerable improvements in the paper. * Corresponding author. E-mail: [email protected] 0378-2166/01/$ - see front matter © 2001 Elsevier Science B.V. All rights reserved. PII: S0378-2166(00)00044-8
1062
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
I. Introduction Within the area of speech recognition and understanding, one area of current research is centered around the issue of segmentation of spontaneous speech into topic units and topic-internal units. (This information can be useful e.g. in referentresolution algorithms and in algorithms for recognizing and synthesizing topic boundaries.) As is known, prosodic cues constitute an important source of information on discourse segmentation (Grosz and Hirschberg, 1992; Ostendorf et al., 1993). In previous studies, we have analysed the role played in particular by 'rightedge' prosodic cues such as phrase accents and final lengthening (Home et al., 1995; Bruce et al., 1993) in signalling boundaries, particularly in read speech (see also de Pijper and Sanderman, 1994). One problem encountered in investigations on discourse segmentation is that the same prosodic parameters (e.g. F0 (fundamental frequency)-reset, final lengthening, pause duration) that are used to mark the boundary between topic-internal units (often correlated with a level of prosodic structure termed the Prosodic Phrase (Home and Filipsson, 1995) or Intonational Phrase (Nespor and Vogel, 1986)) are also used to mark the boundary between higher-level topic units (corresponding to a prosodic constituent which is termed a Prosodic Utterance (Nespor and Vogel, 1986; Home and Filipsson, 1995)). What is most often involved is a relative difference in the degree of phonetic realization of the different parameters marking the different boundaries. Thus, it can be difficult to know where one should set the limit for e.g. F0-reset in order to be able to distinguish between a Prosodic Phrase boundary and a Prosodic Utterance boundary. Furthermore, it is not at all clear if one always can (or should) try to draw a strict boundary between these two, since it is not clear that speakers themselves do. Perhaps it is more appropriate to regard segmentation as a gradient parameter where speakers, due to variation in the extent of speech planning, also use varying degrees of clarity in boundary signalling which in turn, is interpreted as having varying degrees of meaningfulness by the listener (see Swerts, 1997). 1.1. Cue words
One additional type of information that can be used to facilitate the segmentation of speech into topic units that we have not earlier investigated is cue words that together with prosodic cues as well as other lexical/syntactic cues, mark the beginning of different units of discourse structure. (We use the term 'cue word' following Hirschberg and Litman, 1993. Other terms used to refer to the same concept are 'discourse markers' (Schiffrin, 1987), 'pragmatic markers' (Fraser, 1990) and 'discourse operators' (Redeker, 1991)). Cue words constitute local lexical information that can be used, together with other types of information, to determine if the upcoming unit is associated with a Prosodic Phrase boundary/topic-internal boundary or a higher Prosodic Utterance/topic-shift boundary (see Nakajima and Allen, 1993). The prosodic characteristics of the cue word itself are important since one and the same cue word can occur at the beginning of both new topic units as well as topic-internal units.
M. Home et al. / Journal of Pragmatics 33 (2001) 1061-1081
1063
1.2. The Swedish cue w o r d men 'but'
This study reports on the Swedish cue word m e n which corresponds to English 'but' in spontaneous speech. M e n is classified lexically as a conjunction, a classification which reflects its function within sentence grammar to link together two or more clauses. In this function, m e n expresses a local (topic-internal) contradiction. Following are some examples from our data which consists of Swedish narrations of a fragment of a silent film:
(1) Topic-internal
men: a. m a n kunde ldsa ndt litet av det diir brevet m e n det var viildigt otydligt 'you could read a little bit of the letter but it was very unclear' b. fi~rst tror m a n att han dr d6d m e n det var han inte 'first you think he's dead but he wasn't'
In addition to this topic-internal function, m e n has another function in spoken language, i.e. to introduce a new topic unit or to return to a previous topic: (2) Topic-shift men: a. sd f d r han hjiilp upp dd och dd f 6 r s v i n n e r dom hiir ... hotelldirekt6ren m e d k o m p a n i ut ... och han sldpar sig bort till skdpet diir kappan hdnger ... m e n dd k o m m e r det in en liten rant ... s o m ... nja hon dr vdl ndn sdn hiir h u s m o r p d hotellet
'so he gets help up and then they go out ... the hotel manager and company ... and he drags himself over to the wardrobe where the coat is hanging ... but then a little lady comes in ... who ... yea she's one of those matrons at the hotel ... ' b . . . . d eh den hiir g a m l e m a n n e n ... j a g vet inte o m han bor ihop m e d sin dotter eller sin f r u m e n i alla f a l l f r u n hon b a k a r ndn f o r m av kaka eller tdrta ...
'... and uh this old man ... I don't know if he lives together with his daughter or his wife but anyway his wife is baking some kind of cake or tart ...' If it were possible to distinguish between the two different kinds of m e n illustrated in (1) and (2), one could use this information in speech processing algorithms (along with other information sources) to facilitate segmentation into topic-shift and topicinternal boundaries in spontaneous speech.
2. Previous studies In recent years, a number of studies have been reported on the function of cue words in discourse (Schiffrin, 1987; Fraser, 1990; Mosegaard Hansen, 1997; Byron and Heeman, 1997). Few studies, however, have concentrated on the prosodic correlates of these words (see, however, Hirschberg and Litman, 1993; Fretheim, 1989).
1064
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
Hirschberg and Litman (1993) examined both textual and prosodic features of n o w and attempted to find a set of features that could be used to distinguish between n o w ' s S(entential) function (adverbial) and its D(iscourse) function. In its D(iscourse) function, n o w , like Swedish m e n , marks the beginning of a new topic or
a return to a previous one. Hirschberg and Litman arrived at the following results after examining 100 cases of n o w : i.
Discourse n o w constituted most often a (prosodic) phrase on its own (41.3% of the cases). Sentence n o w hardly ever constituted a phrase by itself. ii. Discourse n o w appeared most often at the beginning of a (prosodic) phrase (98.4%). Sentential n o w appeared most often in non-initial position (86.5%). iii. Discourse n o w was more often deaccentuated than sentential n o w . iv. Discourse n o w co-occurred with other cue-words, e.g. w e l l n o w ... Since Swedish m e n is lexically a conjunction, and thus always occurs at the beginning of a clause, linear position (syntactic clause-initial/non-initial) cannot be used as a parameter in distinguishing between m e n ' s functions in discourse as in the case of English n o w (see (ii) above). However, one can expect that additional kinds of information such as prosodic correlates and co-occurring lexical cues could be associated with the two different functions of m e n . Following Schiffrin (1987), we are thus assuming that the meaning component [CONTRAST] associated with m e n (like English but) can have different scopes in discourse (e.g. 'referential' contrast at a relatively local, topic-internal level) and 'action contrast' at a higher topic-shift level. Fig. 1 illustrates how S - m e n a n d D - m e n are associated with these different levels of discourse structure. It also illustrates the levels of prosodic structure assumed to correlate with the discourse structure.
3. Current study In our investigation of Swedish m e n , we decided to conduct the study somewhat differently from Hirschberg and Litman (1993). Four of the authors (MH, GB, PH, MF) first examined the data using only the written transcriptions in order to see how many cases of m e n could be clearly associated with the beginning of either a new topic unit or a topic-internal unit. In doing this, we used the same labels as Hirschberg and Litman (1993), i.e. D for D(iscourse) or topic-shift m e n and S for (Sentential) or topic-internal m e n . Subsequently, auditory and visual acoustic information were used to see if prosody would aid listeners in classifying the tokens of m e n that were not classified in the same way by all four authors. 3.1. D a t a , s p e a k e r s a n d l a b e l l i n g g u i d e l i n e s
As data, we used 21 spontaneous narrations of a fragment of a silent film ( T h e l a s t l a u g h ) . All speakers were from southern Sweden and spoke a variety of southern Swedish. 15 of the speakers were female and 6 were male.
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
1065
PU=
/ ~
\
\
D - M ~ ~ ' ~ _ ~ , _
[FO-reset]
PPh
" F0-Reset. ' 1 Partial
)<
S-MEN
PW
PW
pph=ToPIC-INTERNALUNIT
i
PW
/
i
~
PW
L~
-'~z~
Fig. 1. Prosodic structure showing how (D)-men and (S)-men are associated with the beginning of a PU (Prosodic Utterance) and a PPh (Prosodic Phrase), respectively. PW stands for Prosodic Word. The prosodic structure is further shown to be correlated with two levels of discourse structure, i.e. PU boundaries align with discourse topic unit boundaries, while PPh boundaries align with topic-internal unit boundaries. The figure also shows (within square brackets) a number of prosodic correlates known to be associated with the edges of PU's and PPh's: F0-reset, H% and L% boundary tones, Final lengthening and Pause (= Silent interval).
All together the narrations included 157 tokens of men which were classified as either S, D or S/D (in cases where it was impossible to decide on either S or D). In order to facilitate the labeling task, a n u m b e r of guidelines and examples (like those above in (1) and (2) were used when categorizing men-tokens. Following is a translation of these guidelines:
- S-men expresses some kind of local (topic-internal) contradiction. Often it is a referent under discussion that is contrasted with another referent. It can even be two verbs or two phrases, e.g. two predicates with the same subject that are contrasted. This local contrast should remain if one replaces men with dock ' h o w e v e r ' or fast ' t h o u g h ' (see examples in (1)). D-men does not express any local contrast but rather introduces an utterance which begins a new topic or takes up a previous topic. In this function, men can often be left out or replaced with och ' a n d ' without changing the meaning. Men often co-occurs with i alia fall ' a n y w a y ' in this function (see examples in (2)). - If both a S(entential) (topic-internal) and a D(iscourse) (new topic) interpretation seem possible, label men as S/D. -
1066
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
4. Analysis of data labelled from text alone In about 50% of the cases, the author-labellers were in complete agreement as to the classification of men in the initial classification using text only. That is to say, on the basis of textual information alone, we could classify almost half of the cases (81). Of these 81 cases, 41 were classified as D-men, 39 as S-men and 1 as S/D. Following Swerts (1997: 515), these cases can be said to involve 'strong' boundary marking where "boundary strength is computed as the proportion of subjects agreeing on a given break". 4.1. Prosodic analysis and lexical co-occurrence patterns
Subsequent to the textual analysis of men, we made a prosodic analysis of the cases where the labellers were in agreement in order to determine which, if any, of the parameters chosen for study constituted reliable distinguishing characteristics of the two types of boundaries associated with men. The prosodic analysis was conducted using the ESPS/waves environment. In addition to the acoustic phonetic analysis, we also examined the data for patterns of co-occurring lexical items. In the prosodic analysis, the following local parameters were examined: (a) Preceding pause: one would expect a relatively longer pause before a D-men than a S-men since a D-men marks the beginning of a new topic unit and these units (often corresponding to written paragraphs) are most often preceded by longer pauses than the beginning of a new topic-internal unit (often corresponding to a clause or sentence) (Fant and Kruckenberg, 1989; Strangert, 1993; Home et al., 1995). (In the present study, pauses are defined as periods of silence (excluding occlusion phases for stop consonants) identified in the acoustic wave form.) (b) F0-reset on men: one would expect a relatively larger F0-reset on D-men than on S-men for the same reason as in (a) (see e.g. Brown et al., 1980; Grosz and Hirschberg, 1992; Sluijter and Terken, 1993; Swerts and Geluykens, 1993). (In this study, we measure 'reset' (the rise in F0 (fundamental frequency)) from the lowest F0 value in the syllable rhyme of the final syllable of the word preceding men to the highest F0 value in the vowel of men.) (c) Duration of men. One would expect that D-men would have a greater duration than S-men since the degree of coherence between D-men and what follows (i.e. new topic or return to a previous topic) is less than that between topic-internal S-men and that which follows. S-men are often perceived as short and reduced, while D-men are perceived as prominent (drawn out) reflecting the speaker's plarming of a new discourse topic. Note that this is just the opposite characterization of n o w ' s discourse form in comparison with its sentential form (Hirschberg and Litman, 1993). This is obviously due to the lexical category of the words. Since now is an adverb, it is often prominent in its 'Sentential' function when it constitutes new information, whereas men is a conjunction and by default reduced in its 'Sentential' connective function. Consequently, in their
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
1067
'Discourse' functions, one would then expect n o w and m en to exhibit a level of prominence which is clearly different from that they have in their 'Sentential' functions. (d) Prosodic Phrasing. As in the case of now, one would expect that m en in its discourse function of signalling a topic shift would constitute a Prosodic Phrase on its own, i.e. both preceded and followed by pauses reflecting the extra planning time involved when making a topic shift. In the lexical co-occurrence analysis, we looked at the word class of items following men. As Hirschberg and Litman showed in the case of now, in its 'Discourse' function, this marker often co-occurs with other cue words/phrases e.g. well now. Swedish men is no exception. In its more S(entential), topic-internal discourse function, on the other hand, m e n is often followed by a pronoun referring back to a previously introduced discourse referent. 4.2. Results
Results from the prosodic and lexical analysis are summarized in Table 1. Readers should be aware of the fact that the use of t-tests to test significance levels is questionable since we factor out the possible influence of subjects. In order to eliminate the possible influence of subjects, an ANOVA is called for. This kind of test was difficult to run on our limited database, however, due to the large amount of data that would have to be discarded. We have decided therefore to present the t-test results here, since the potential application of our findings lies in automatic speech recognition, and since speech recognizers must be able to handle all speakers' speech, we feel that the results of the t-tests give at the very least a number of interesting clues as to which prosodic correlates are more relevant than others to use in the development of automatic classifiers. Furthermore, we describe below how a neural network can use the results of the multiple speaker data to correctly classify men-tokens. 4.2.1. Prosodic correlates 4.2.1.1. Duration. Measurements of the absolute duration of m en revealed that D(iscourse)-men was on the average 310 ms (= milliseconds) long while S(entential)-men was on the average 170 ms, i.e. 55% the duration of D(iscourse) men. This
difference was statistically significant (p = 0.002). 4.2.1.2. FO-reset. Measurements of F0 (fundamental frequency) were made in the rhyme of the word preceding m e n (lowest F0 value) and in the vowel of m en (highest F0 value). Of the 41 tokens of D - m e n , 32 (78%) were characterized by a positive F0-reset (i.e where the F0 level rose). Since the phrase-final F0 level preceding m en was associated with glottalization ('creaky voice') in a number of cases, and since glottalization has the effect of lowering F0 (Dilley et al., 1996), we factored out the cases of reset associated with phrase-final glottalization into a separate group. (Glottalization was identified on the basis of irregularities in the acoustic wave form.) It is seen in Table 1 that the D - m e n tokens without reset are characterized by a mean
1068
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
reset of 5.7 ST (= Semitones) which is in line with the size of the reset one would expect at a topic-shift boundary. The mean reset associated with glottalization is, on the other hand, 13.8 ST. As for the S-men tokens, the tokens not associated with glottalization showed a mean reset of 2.2 ST (normal at a topic-internal Prosodic Phrase boundary), whereas those occurring after phrase-final glottalization exhibited a correspondingly larger reset of 7.6 ST. Disregarding potential speaker variation, these differences are significant (p = 0.0238 for glottalized reset and 0.0448 for nonglottalized reset). 4.2.1.3. Preceding pause. Measurements of pause duration revealed that 66% of D(iscourse) men (N = 27) were associated with a preceding pause that was on the average 750 ms long. Of the S(entential) men tokens, 38% (N = 15) were preceded by a pause which was on the average 460 ms long. (It is to be noted that in the measurements of duration, no consideration was taken to rate of speech.) Although the mean difference is quite large, it is not significant (p = 0.1274). 4.2.1.4. Prosodic phrasing. As in the case of English now, Swedish D(iscourse) men constituted a separate Prosodic Phrase in 34% of the D-tokens, i.e. was both preceded and followed by a pause. None of the S-men, on the other hand were characterized in this way. Fig. 2a shows an example of a D-men characterized by a relatively long duration, a preceding pause and a large F0-reset). Fig. 2b presents an example of a S-men, characterized by a relatively short duration, absence of F0-reset and no preceding pause.
4.2.2. Co-occurring lexical items In its D(iscourse) function, it was seen, as expected, that men co-occurs with other similar words (conjunctions, pause fillers, particles). Unlike now, however, the discourse markers co-occurring with men come in a position following men instead of preceding it. In 26 of the 41 cases of men labelled as D(iscourse) (63%), one of the five following labels was associated with another discourse-related cue-word or pause marker: alltsd, sedan (sd), i alia fall, i varje fall, men, (eh)(just) dd(sd), eh, ja, sd, dd, ngir. Only two of the tokens of men labelled as S were so characterized and both of them involved a filled pause segment (eh). On the other hand, 24 of the 39 tokens of men (62%) labelled as S were followed immediately by a pronoun, while only five (12%) of the D-men were so characterized. Of the five, one was the nonreferential pronoun det 'it' = Eng. 'there' (det var ndgra sddana fattigbarn 'there were some poor children'. The other four were personal pronouns followed in turn by discourse markers, e.g. han v?intade i alla fall pd hotellet 'h._eewaited anyway at the hotel'. Thus, one can hypothesize that the lexical co-occurrence cue for identifying D-men (i.e. a following discourse-related cue word marker) is quite strong. 4.3. Summary of findings for data labelled from text-only Results of the acoustic analysis of prosodic parameters associated with the topicshift (D-men) and topic-internal (S-men) show considerable mean differences in F0reset and absolute word duration. A clear difference in prosodic phrasing wherein
M, Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
1069
i-
•2 ~
!:i
--
li I.
-r.
!._
•
I -
u.-
!
L
-~t~
t~
,
I
~=~o.~
1070
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
i~
e~
=.
e.
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
1071
Table 1 Results from the labelling from text only study* Prosodic correlates s.d. rain Duration (ms) D (41) 310 240 40 S (39) 170 130 10 FO-reset (glottalized) (ST) D (17/41) 13.8 6.0 0.7 S (9/39) 7.6 6.7 1.8 FO-reset (non-glottalized) (ST) D (15/41) 5.7 5.6 0.1 S (17/39) 2.2 3.6 0.01 Preceding pause (ms) D (27/41) 750 670 30 S (15/39) 460 350 40 Men as separate Prosodic Phrase (Pauses before and after) D (14/41) (34%) S (0/39) (0%)
max 1170 600
t = 3.2 (78 dr) p = 0.0020
25.6 19.1
t = 2.41 (24 dr) p = 0.0238
16.4 12.9
t = 2.093 (30 dr) p = 0.0448
2320 1080
t = 1.557 (40 df) p = 0.1274
Co-occurring lexical cues Following cue word (one of the following 5 labels) D 26/41 (63%) S 2/39 (5%)
Following pronoun D 5/41 (12%) S 24/39 (62%)
* Prosodic correlates (Duration, F0-reset, Preceding Pause duration, and status as a separate Prosodic Phrase) as well as co-occurring lexical items (following Cue word, following Pronoun) associated with men-tokens occurring at the beginning of a new topical unit (D-men) and at the beginning of topic-internal units (S-men). Means (X), standard deviations (s.d.), minimum (min) and maximum (max) values for the continuous prosodic features Duration, F0-reset and Preceding pause are presented. (df) stands for 'degrees of freedom'. For the non-continuous features (Men as a separate Prosodic Phrase and Co-occurring lexical items), results are presented as fractions of the total number of S- and D-men tokens that are characterized by the feature. T-test results are to be interpreted with caution since they factor out potential speaker differences.
D - m e n , but not S - m e n constitutes an in d e p e nd en t Prosodic Phrase was also observed. T h e d i f f e r e n c e in p r e c e d i n g pause duration for all speakers p o o l e d together, h o w e v e r , was not significant, a result in line with that reported in Swerts and G e l u y k e n s (1993) for a single speaker. T h e tw o categories o f m e n were further strongly associated with different c o - o c c u r r i n g lexical categories, i.e. D - m e n were in o v e r 6 0 % o f the cases f o l l o w e d by another cue word, w h i l e S - m e n w e r e in o v e r 6 0 % o f the cases f o l l o w e d by a pronoun. In s u m m a r y , although the labelling o f the data in this study was done on text alone, it is seen that the ' s t r o n g ' boundaries are associated with a w h o l e constellation o f both lexical and prosodic cues that serve to mark their function in discourse.
1072
M. Home et al. /Journal of Pragmatics 33 (2001) 1061-1081
5. Analysis of data labelled from text-with-speech On the basis of the results obtained from the acoustic analysis of the tokens of S-men and D - m e n labelled from text alone, we initially expected that by listening to the data and observing the acoustic signal associated with the tokens of men that
were not categorized in the same way by all four labellers, prosodic information might provide some clues that would aid in coming to some agreement as to the categorization of men. Three of the authors (MF was not available at this time) thus re-examined all the cases of men (n = 75) which had not received a unique label during the first part of the study. Six of these were discarded since they constituted parts of speech disfluencies. The remaining 69 were examined and each case was individually discussed until labellers agreed on a unique categorization for each token. 30 tokens were classified as topic-internal (S), 29 as topic-shift (D) and 10 as ambiguous (S/D). During the process, the general impression the labellerauthors had was that, although access to the speech signal DID help in some cases, there was nevertheless a rather high degree of uncertainty as to the classification of men. This is partly reflected in the fact that 10 tokens were still assigned a S/D label, i.e. it was unclear as to how they should be classified even after listening to the speech. 5.1. Results 5.1.1. Prosodic correlates
This general uncertainty as to the classification of the cases of men that were unclear from their text-only labelling is reflected in the prosodic analysis of these tokens after reexamination and retagging using speech as well. Results are presented in Table 2. As can be seen, none of the prosodic parameters examined exhibited significant differences between S and D-men tokens. 5.1.1.1. Duration. As can be seen from Table 2, there was hardly any difference between S-men and D-men as regards their absolute duration in the labelled-fromspeech data (210 vs. 220 ms, respectively). 5.1.1.2. FO-reset. Differences in positive F0-reset between S- and D-men in the data labelled from listening were not at all significant. After factoring out glottalization, there is almost no difference in the values for F0-reset. 5.1.1.3. Preceding pause. Differences in preceding pause length for the tokens of men tagged by listening are quite similar to those tagged from text. 17 of the 29 cases labelled as D (59%) were preceded by a pause having a mean duration of 740 ms whereas only 6 of the tokens labelled as S (20%) were associated with a preceding pause which had a mean duration of 540 ms. Thus the pause duration preceding D-men labelled from speech is almost the same as that associated with the D-men labelled from text whereas the mean duration of S-men is 90 ms greater here than for the tokens labelled from text. Thus the overall difference is not as large. 5.1.1.4. Prosodic phrasing. As in the data labelled from text-alone, a clear cue to the classification of men is its status as an independent Prosodic Phrase (surrounded
M. Horne et al. /Journal of Pragmatics 33 (2001) 1061-1081
1073
by pauses). In the data labelled from text and speech, three of the D-men constituted independent Prosodic Phrases while none of the S-men were so characterized. 5.1.2. Co-occurring lexical items Even the local lexical information associated with the tokens of men labelled from speech was not significantly different in the two cases. Whereas nine cases (31%) of D-men were followed by another cue word, six tokens of S-men (17%) were also followed by a cue word. Even following pronouns did not help to distinguish the two cases of men. Although 60% of the S-men were followed by a pronoun, 41% of the D-men were also followed by a pronoun. Table 2 Results from the labelling from text-with-speech study* Prosodic correlates s.d.
min
max
210 222
160 200
20 40
690 920
t = 0.204 (57 df) p = 0.8391
10.7 10.4
6.6 4.1
0.4 2.6
22.9 14.0
t = 0.119 (16 df) p = 0.9064
3.1 3.2
1.5 4.6
0.8 0.1
6.8 16.2
t = 0.587 (23 df) p = 0.9536
D (17/29) 740 530 70 S (6/30) 540 420 60 Men as separate Prosodic Phrase (Pauses before and after) D (3/29) (10%) S (0/30) (0%)
1930 1080
t = 0.812 (21 df) p = 0.4261
Duration (ms) D (29) S (30)
FO-reset (glottalized) (ST) D (11/29) S (7/30)
FO-reset (non-glottalized) (ST) D (13/29) S (12/30)
Preceding pause (ms)
Co-occurring lexical cues
Following cue word
Following pronoun
(one of the following 5 labels) D 9/29 (31%) S 5/30 (17%)
D 12/29 (41%) S 14/30 (47%)
* Prosodic correlates (Duration, F0-reset, Preceding Pause duration, and status as a separate Prosodic Phrase) as well as co-occurring lexical items (following Cue word, following Pronoun) associated with men-tokens occurring at the begilming of a new topic unit (D-men) and at the beginning of topic-internal units (S-men). Means (X), standard deviations (s.d.), minimum (rain) and maximum (max) values for the continuous prosodic features Duration, F0-reset and Preceding pause are presented. (df) stands for 'degrees of freedom'. For the non-continuous features (Men as a separate Prosodic Phrase and Co-occurring lexical items), results are presented as fractions of the total number of S- and D-men tokens that are characterized by the feature. T-test results are to be interpreted with caution since they factor out potential speaker differences.
1074
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
5.2. Summary o f findings for data labelled using text-with-speech
No clear differences in the values for the prosodic parameters nor in the patterning of co-occurring lexical items were seen in this subset of data labelled using textwith-speech. This indicates that these 'weak' boundaries are weak even with regard to prosodic parameters and further corroborates the idea developed in Swerts (1997) that, with respect to the hearer/reader, not all boundaries are equally meaningful and this is reflected in the degree of 'strength' with which they are linguistically realized.
6. Analysis of complete database A final analysis of the data was made in order to see if any of the prosodic parameters and co-occurring lexical items could be used to distinguish between S- and D-men when pooling both sets of data (labelled from text-alone and labelled from text-with-speech). Results are presented in Table 3. As can be seen, the only parameter which alone shows a significant difference between S- and D-men is men's absolute duration (p = 0.0203). Another clear indication of men's D-status is its occurrence as a separate prosodic phrase. Whereas 24% (N = 17) of the D-men constitute a separate Prosodic Phrase, none of the S-men are characterized in this way. As regards the patteming of co-occurring lexical items, one can see that half of the D-men are followed by another cue word and 55% of the S-men are followed by a pronoun. Only 12% of the S-men were followed by another cue word. 25% of the Dmen, however, were also followed by a pronoun.
7. Implications for speech recognition/understanding From a recognition point of view, one could imagine, all other things being equal, that it is necessary to develop some way of clearly classifying all instances of men as either D (topic-shift) or S (topic-internal). However, it is not clear that it is important or necessary to be able to classify all tokens of men. From a speech understanding point of view, it stands to reason that it is the clearly marked cases (correlated with 'strong' boundaries (Swerts, 1997, 1998)) that should be important for discourse processing since it is these that the speaker probably intends the listener to pay particular attention to. In our study, these are the cases in the labelling from text-alone part where all labellers agreed as to the categorization of men. The 'strong' D-boundaries are the ones clearly associated with a new topic boundary, while the 'strong' S-boundaries are those that are clearly topic-internal. The unclear cases (related to 'weak' boundaries), on the other hand, which are not distinctly marked in any way, do not have as clear a function in discourse either and can for the most part probably be disregarded in speech processing since they are not interpretable/meaningful for the listener. Following this line of reasoning, and assuming that it is possible to distinguish between 'strong' and 'weak' cases of the cue word, we thought it would be interesting
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
1075
Table 3 Results from all data (labelled from text and labelled from text and speech) pooled together* Prosodic correlates s.d.
min
max
Duration (ms) D (70)
271
217
20
1170
t -- 2.347 (137
S (69)
194
169
10
920
12.6 8.8
6.3 5.8
0.4 0.2
25.6 19.1
t = 1.974 (42 df) p = 0.0549
4.5 2.5
4.4 3.9
0.1 0.01
16.4 16.2
t = 1.796 (56 df) p = 0.0778
2320 1080
t = 1.803 (63 df) p = 0.0761
dO p = 0.0203
FO-reset (glottalized) (ST) D (28/70) S (16/69)
FO-reset (non-glottalized) (ST) D (28/70) S (29/69)
Preceding pause (ms) D (44/70) 740 610 30 S (21/69) 480 360 40 Men as separate Prosodic Phrase (Pauses before and after) D (17/70) (24%) S (0/69) (0%) Co-occurring lexical cues
Following cue word
Following pronoun
(one of the following 5 labels) D 35/70 (50%) D 17/70 (25%) S 8/69 (12%) S 38/69 (55%) * Prosodic correlates (Duration, F0-reset, Preceding Pause duration, and status as a separate Prosodic Phrase) as well as co-occurring lexical items (following Cue word, following Pronoun) associated with men-tokens at the beginning of a new topical unit (D-men) and at the beginning of topic-internal units (S-men). Means (~), standard deviations (s.d.), minimum (min) and maximum (max) values for the continuous prosodic features Duration, F0-reset and Preceding pause are presented. (df) stands for 'degrees of freedom'. For the non-continuous features (Men as a separate Prosodic Phrase and Co-occurring lexical items), results are presented as fractions of the total number of S- and D-men tokens that are characterized by the feature. T-test results are to be interpreted with caution since they factor out potential
to see to what extent an automatic classifier could be developed to distinguish between S- and D-men within the 'strong' group. We will first describe an attempt to use a 'linear classifier' and then an experiment using a non-linear neural network.
8. A u t o m a t i c classification o f 'strong' men-tokens
8.1. Linear classifiers By using linear classifiers, each relevant parameter can be used independently to classify the data set. The method can then determine the threshold that splits the data into S-men and D-men subsets in such a way that the maximum number of correct
1076
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
classifications is obtained. The advantage of this method is that the resulting threshold value is easy to interpret. We did this for the three acoustic parameters 'preceding pause duration' (ppd), 'word duration' (dur) and 'F0-reset' (fzr). These thresholds, and the correct classification rates are presented in Table 4. Table 4 Threshold values for the three prosodic parameters 'preceding pause duration' (ppd) (in seconds (s)), 'word duration' (dur) (in seconds (s)) and 'F0-reset' (fzr) (in Semitones (ST)) used in a linear classifier for distinguishing between the 'strong' S-men and D-men tokens
Threshold Correct (%)
ppd (s)
dur (s)
fzr (ST)
0.232 58.5
0.358 65.0
8.265 70.0
If one combines these three linear classifiers, and uses a decision function where two parameters are enough to classify a men-token as S or D the rate of correct classifications rises to 71.5 %. The weakness of this method, however, is that it does not take into account the relative strength that each parameter can have. For example, if the parameters ppd and d u r both yield the category S-men, the value of fzr is not taken into consideration, even though a very high value (i.e. > 10 ST) of this parameter would be a much stronger indication of a D-men. In order to avoid this weakness a non-linear method, namely, a neural network, was subsequently tested. 8.2. Non-linear classifiers: Neural networks
Neural networks are computer models based on the operation of components of the human brain. The particular strength of neural networks lies in their power to generalise, classify and find patterns in multi-dimensional data. The disadvantage with this method, however, is that the threshold values for each parameter are obscured. However, in the attempt to reach the best possible classification, the neural network method has proven to be superior. When a neural network is supplied with some measured parameters the task of the network is to map these input features onto a classification state, that is, given the acoustic features mentioned above as input, the neural network can be trained to decide which type of class category (D-men or S-men) they match most closely. Neural networks have been applied successfully to many aspects of speech processing, see e.g. Kohonen (1988) for an approach to speech recognition, Sejnowski and Rosenberg (1986) for speech synthesis, and Johansson (1995) for language acquisition. (See Beale and Jackson (1990) for a more general introduction to neural networks.) It may be objected that a neural network is too powerful a method for the present task, since there are relatively few parameters and cases. However, we feel that due to the nature of the data (multispeaker) as well as the goal (to recognize boundary
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
1077
types realized by several interacting parameters), a non-linear method is the only method that is feasible and that can handle these interacting cues and give some reliable indication as to the relevance of the parameters that are used in the categorization (see also Ostendorf (2000) on this issue).
8.2.1. Data In order to build a classifier with a neural network, the network needs (minimally) two data sets: a training set and a validation set. In the data described above, there were 80 cases of 'strong' boundaries labelled from text only. Three of these were discarded since the F0 reset parameter could not be measured due to glottalisation. The remaining 77 cases were divided into two groups, one consisting of 33 cases and the other of 44 cases. These groups were to be used as training and validation data. Care was taken so as to get an even distribution of speakers in both groups, to include all speakers in both groups, and to have an approximately even distribution of S-men and D-men cases in each group (55% and 48% D-men, respectively). This means that the baseline performance of the network (choosing the most likely category) is around 50%. 8.2.2. Network details Two neural networks were designed and are schematically represented in Fig. 3. One used three input nodes for the parameters 'preceding pause duration' (ppd), 'word duration' (dur) and 'F0 reset' (fzr). This net thus only used prosodic correlates. The other net had two additional input nodes for the lexical information 'Following cue word' (c-w) and 'Following pronoun' (pro), thus having a total of five input nodes. Both nets had a hidden layer with two nodes, and an output layer with one node for the classification as D-men or S-men. All neural network processing and simulation was performed with the SNNS package from IPVR in Stuttgart (University of Stuttgart, 1996). The training was performed using the Resilient backpropagation (Rprop) scheme. The input was raw data, normalised around the mean of each parameter, so that input values were in the range of -1 to 1. Since the number of cases is quite small, there is a risk that the actual distribution of data in the two sets affects the outcome of the network's performance. Therefore the network was trained in two different sessions. In the first session, the smaller data set was used as training data and in the second the larger data set was used as training data. Note, however, that the two training sessions were run independently of each other. In neither case was the validation data included in the training set. The results presented constitute an average of the two different sessions. 8.2.3. Results When using the three acoustic parameters (ppd, dur, fzr), the rate of correct classifications is 90%. When the lexical features (c-w, pro) are included, the rate of correct classifications decreases to 82%. This might seem counterintuitive: adding more information should increase the network's performance. The explanation for this result, however, lies no doubt in the fact that the two lexical features are good indicators of D-men or S-men only when they OCCUR. In the cases when
1078
M. Home et al. / Journal of Pragmatics 33 (2001) 1061-1081
Output(SorD) ~ 0
0 ppd dur
A
Hiddenl a y e r ~ 0
fzr
0 0 0 0 0 ppd dur fzr c-w pro
Fig. 3. Schematic representation of the neural networks experimented with in this study. The one on the left uses three input nodes for the prosodic parameters 'preceding pause duration' (ppd), 'word duration' (dur) and 'F0 reset' (fzr). The net on the right uses two additional input nodes for the information on co-occurring lexical items: 'Following cue word' (e-w) and 'Following pronoun' (pro). Both nets have a hidden layer with two nodes, and an output layer with one node for the classification as D-men or S-men.
they do not occur, the distribution between D-men and S-men is almost 40-60, which introduces uncertainty in the network. Since there are more cases where they do not occur than when they do occur, the number of correctly classified cases decreases. (The same result would have applied to the parameter 'Separate prosodic phrase' if it had been used.) The recall and precision rates are 84,2 % and 97,0% for S-men, and 97,4% and 86,4% for D-men. The overall error rate is 18,4% for S and 17,9% for D. These rather high rates are due to the small number of cases in the data. 8.2.4. Discussion The combination of the prosodic parameters preceding pause duration, word duration, and F0-reset can predict the status of the cue word men as D-men or S-men correctly in 90% of the 'strong' cases. The tendencies observed in the data can thus be utilized to produce a rather accurate classifier. It also indicates that listeners use an aggregate, rather than one particular feature, when they distinguish between the two categories. It should be pointed out, however, that in the network described here, we have only used 'strong' cases to train the network since we were interested to see if the prosodic and lexical information from several speakers was potentially useful in categorizing the two different categories of men, S- and D-. Thus, if one were to attempt to implement such a classification network in a general system for recognizing strong S- and D- men-tokens, the system would have to be able to distinguish the strong from the weak tokens. It is not clear if this could be done in an optimal way with the present network. Therefore, if one's goal were to actually build a network for recognizing topic shift boundaries (in other words, for recognizing strong
M. Home et al. / Journal of Pragmatics 33 (2001) 1061-1081
1079
D - m e n ) , it would probably be more advisable to use three categories to train the system from the very beginning, i.e. S, D and U (for 'Unclear'). Moreover, a more implementation-oriented network could be trained with other (or more refined) prosodic and lexical parameters than those that we have considered here. For example, the case where m e n is both preceded and followed by a pause is a very strong (if not decisive) cue to its classification as D. Yet, we have not built this information into the network discussed here. As we have further noticed, the absence vs. the presence of another cue word following m e n is another very strong indicator of a Dm e n . However, it is possibly the case that the size of the window we have used in this study is too large. In order to exclude potential cue words that are in fact not cue words, the size of the window should be reduced from five words following the cue word to something like three words.
9. Conclusion From this investigation of the cue-word m e n 'but' in Swedish, it has become evident that it, along with its associated prosodic correlates and co-occurring lexical categories, constitutes an important source of information for recognizing both topicshift and topic-internal boundaries in spontaneous speech. Results from an exploratory study using a neural network show that it is possible to attain a high degree of recognition of 'strong' boundaries by using the cue word and the associated parameters chosen for this study. The results are also interesting for speech synthesis since in order to generate coherent discourse, it is important to be able to model the different kinds of prosodic constituent boundaries that occur in natural speech. The present study shows that a whole constellation of prosodic and lexical cues need to be taken into consideration in order to understand how speakers perceive and produce topic-related boundaries in spontaneous speech.
References Beale, Russel and Tom Jackson, 1990. Neural computing: an introduction. Bristol: IOP Publishing. Brown, Gillian, Karen Currie and Joanne Kenworthy, 1980. Questions of intonation. London: Croom Helm. Bruce, G6sta, Bjtiru GranstRim, Kjell Gustafson and David House, 1993. Interaction of F0 and duration in the perception of prosodic phrasing in Swedish. In: B. Granstr6m and L. Nord, eds., Nordic prosody VI, 7-22. Stockholm: Almqvist & Wiksell. Byron, Donna K. and Peter A. Heeman, 1997. Discourse marker use in task-oriented spoken dialog. Proceedings of Eurospeech 97 (Rhodes, Greece), 2223-2226. Dilley, Laura, Stefanie Shattuck-Hufnagel and Marl Ostendorf, 1996. Glottalization of word-initial vowels as a function of prosodic structure. Journal of Phonetics 24: 423-444. Fant, Gunnar and Anita Kruckenberg, 1989. Preliminaries to the study of Swedish prose reading and reading style. Speech Transmission Laboratory, Quarterly Progress and Status Report (Royal Institute of Technology, Stockholm) 2. Fraser, Bruce, 1990. An approach to discourse markers. Journal of Pragmatics 14: 383-395. Fretheim, Thorstein, 1989. The two faces of the Norwegian inference particle da. In: Harald Weydt, ed., Sprechen mit Partikeln, 403-415. Berlin: de Gruyter.
1080
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
Grosz, Barbara and Julia Hirschberg, 1992. Some intonational characteristics of discourse structure. Proceedings of the 2nd International Conference on Spoken Language Processing (Banff, Canada), 429-432. Hirschberg, Julia and Diane Litman, 1993. Empirical studies on disambiguation of cue phrases. Computational Linguistics 19: 501-530. Home, Merle and Marcus Filipsson, 1995. Computational modelling and generation of prosodic structure in Swedish. Proceedings of the International Congress of Phonetic Sciences (Stockholm) 4: 364-367. Home, Merle, Eva Strangert and Mattias Heldner, 1995. Prosodic boundary strength in Swedish: Final lengthening and silent interval duration. Proceedings of the International Congress of Phonetic Sciences (Stockholm) 1: 170-173. Johansson, Christer, 1995. The development of verb forms in connectionist nets. Licentiate thesis, Department of Linguistics and Phonetics, Lund University. Kohonen, Teuvo, 1988. The 'neural' phonetic typewriter. Computer 21(3): 11-22. Mosegaard Hansen, Maj-Britt, 1997. Alors and done in spoken French: A reanalysis. Journal of Pragmatics 28: 153-187. Nakajima, Shin'ya and James F. Allen, 1993. A study on prosody and discourse stmcture in cooperative dialogues. Phonetica 50: 197-210. Nespor, Marina and Irene Vogel, 1986. Prosodic phonology. Dordrecht: Foris. Ostendorf, Mari, 2000. Prosodic boundary detection. In: Merle Home, ed., Prosody: Theory and experiment. Studies presented to GOsta Bmce, 263-279. Dordrecht: Kluwer. Ostendorf, Mari, Colin W. Wightman and N.M. Veilleux, 1993. Parse scoring with prosodic information: An analysis/synthesis approach. Computer Speech and Language 7: 193-210. Pijper, Jan R. de and Angelien Sanderman, 1994. On the perceptual strength of prosodic boundaries and its relation to suprasegmental cues. Journal of the Acoustical Society of America 96: 2037-2047. Redeker, Gisela. 1991. Linguistic markers of discourse structure. Linguistics 29:1139-1172. Schiffrin, Deborah, 1987. Discourse markers. Cambridge: Cambridge University Press. Sejnowski, Terrence J. and Charles R. Rosenberg, 1986. NETtalk: A parallel network that learns to read aloud. Cognitive Science 14:179-211. Sluijter, Agaath M.C. and Jacques M.B. Terken, 1993. Beyond sentence prosody: Paragraph intonation in Dutch. Phonetica 50: 180-188. Strangert, Eva, 1993. Speaking style and pausing. Phonum 2: 121-137. Swerts, Marc, 1997. Prosodic features at discourse boundaries of different strength. Journal of the Acoustical Society of America 101: 514-521. Swerts, Marc, 1998. Filled pauses as markers of discourse stmcture. Journal of Pragmatics 30: 485-496. Swerts, Marc and Ronald Geluykens, 1993. The prosody of information units in spontaneous monologues. Phonetica 50: 189-196. University of Stuttgart, 1996. Stuttgart Neural Network Simulator Manual. Institute of Parallel and Distributed High-Performance Systems, University of Stuttgart.
Merle Horne Ph.D. (www.ling.lu.se/persons/Merle) (born 1950), Senior lecturer in general linguistics at Lund University, Sweden. Involved in interdisciplinary research on prosody and its relation to information structure in spontaneous discourse. Currently leading a project on Swedish dialogue prosody. Petra Hansson (www.ling.lu.se/persons/Petra) (born 1974) is a doctoral student in phonetics at Lund University. Thesis research centered on dialogue prosody, in particular prosodic factors related to prominence and phrasing. G6sta Bruce Ph.D. (www.ling.lu.se/persons/Gosta) (born 1947), Professor of Phonetics at Lund University, Sweden. Is internationally well-known for his work on modelling Swedish prosody. Currently leading a project on the phonetics and phonology of Swedish dialects.
M. Home et al. /Journal of Pragmatics 33 (2001) 1061-1081
1081
Johau Frid (www.ling.lu.se/persons/JohanF) (born 1970) is a doctoral student in phonetics at Lund University. Research centered on computational aspects of prosody related to speech synthesis and recognition.
Marcus Filipsson (born 1967) was a research engineer at the Dept. of Linguistics during the time the study reported on here was being conducted. He is currently employed by Telelogic, Malm6, Sweden.
Journal of Pragmatics 33 (2001) 1061-1081 www.elsevier.nl/locate/pragma
Cue words and the topic structure of spoken discourse" The case of Swedish m e n 'but Merle Home,* Petra Hansson, G6sta Bruce, Johan Frid, Marcus Filipsson Department of Linguistics and Phonetics, Lund University, Helgonabacken 12, S-223 62 Lund, Sweden Received 24 March 1999; revised version 6 June 2000
Abstract The Swedish cue word men 'but' can mark the boundary between both different topic units as well as topic-internal units in spontaneous speech. The goal of this study is to see if these two functions of men can be distinguished on the basis of their local prosodic correlates and co-occurring lexical items. Men-tokens in spontaneous narrations were labelled as to their function, first using text-only data. The 'strong' tokens (categorized identically by all labellers) were subsequently seen to be clearly differentiated into two classes on the basis of related prosodic parameters and co-occurring lexical items. This distinction was, however, not found for the corresponding 'weak' tokens which were subsequently relabelled using both text and speech nor for the data-base as a whole. A test using a neural network trained using strong tokens was seen to be able to correctly categorize 90% of the strong men-tokens as to their associated boundary-type (topic-shift vs. topic-internal). The results show that cue words along with their prosodic correlates and co-occurring lexical items constitute a constellation of important information for understanding how segmentation of spoken discourse is produced and understood. © 2001 Elsevier Science B.V. All rights reserved. Keywords: Cue word; Discourse marker; Prosody; Spoken discourse; Topic structure; Speech recognition; Swedish
~ This research has been supported by a grant from the Swedish HSFR/NUTEK Language Technology Programme and by Telia Research. We would like to thank Mechtild Tronnier for assistance with statistical analyses and St6phane Di Cesar6 for help with programming. We are also very grateful to two anonymous referees whose comments have led to considerable improvements in the paper. * Corresponding author. E-mail: [email protected] 0378-2166/01/$ - see front matter © 2001 Elsevier Science B.V. All rights reserved. PII: S0378-2166(00)00044-8
1062
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
I. Introduction Within the area of speech recognition and understanding, one area of current research is centered around the issue of segmentation of spontaneous speech into topic units and topic-internal units. (This information can be useful e.g. in referentresolution algorithms and in algorithms for recognizing and synthesizing topic boundaries.) As is known, prosodic cues constitute an important source of information on discourse segmentation (Grosz and Hirschberg, 1992; Ostendorf et al., 1993). In previous studies, we have analysed the role played in particular by 'rightedge' prosodic cues such as phrase accents and final lengthening (Home et al., 1995; Bruce et al., 1993) in signalling boundaries, particularly in read speech (see also de Pijper and Sanderman, 1994). One problem encountered in investigations on discourse segmentation is that the same prosodic parameters (e.g. F0 (fundamental frequency)-reset, final lengthening, pause duration) that are used to mark the boundary between topic-internal units (often correlated with a level of prosodic structure termed the Prosodic Phrase (Home and Filipsson, 1995) or Intonational Phrase (Nespor and Vogel, 1986)) are also used to mark the boundary between higher-level topic units (corresponding to a prosodic constituent which is termed a Prosodic Utterance (Nespor and Vogel, 1986; Home and Filipsson, 1995)). What is most often involved is a relative difference in the degree of phonetic realization of the different parameters marking the different boundaries. Thus, it can be difficult to know where one should set the limit for e.g. F0-reset in order to be able to distinguish between a Prosodic Phrase boundary and a Prosodic Utterance boundary. Furthermore, it is not at all clear if one always can (or should) try to draw a strict boundary between these two, since it is not clear that speakers themselves do. Perhaps it is more appropriate to regard segmentation as a gradient parameter where speakers, due to variation in the extent of speech planning, also use varying degrees of clarity in boundary signalling which in turn, is interpreted as having varying degrees of meaningfulness by the listener (see Swerts, 1997). 1.1. Cue words
One additional type of information that can be used to facilitate the segmentation of speech into topic units that we have not earlier investigated is cue words that together with prosodic cues as well as other lexical/syntactic cues, mark the beginning of different units of discourse structure. (We use the term 'cue word' following Hirschberg and Litman, 1993. Other terms used to refer to the same concept are 'discourse markers' (Schiffrin, 1987), 'pragmatic markers' (Fraser, 1990) and 'discourse operators' (Redeker, 1991)). Cue words constitute local lexical information that can be used, together with other types of information, to determine if the upcoming unit is associated with a Prosodic Phrase boundary/topic-internal boundary or a higher Prosodic Utterance/topic-shift boundary (see Nakajima and Allen, 1993). The prosodic characteristics of the cue word itself are important since one and the same cue word can occur at the beginning of both new topic units as well as topic-internal units.
M. Home et al. / Journal of Pragmatics 33 (2001) 1061-1081
1063
1.2. The Swedish cue w o r d men 'but'
This study reports on the Swedish cue word m e n which corresponds to English 'but' in spontaneous speech. M e n is classified lexically as a conjunction, a classification which reflects its function within sentence grammar to link together two or more clauses. In this function, m e n expresses a local (topic-internal) contradiction. Following are some examples from our data which consists of Swedish narrations of a fragment of a silent film:
(1) Topic-internal
men: a. m a n kunde ldsa ndt litet av det diir brevet m e n det var viildigt otydligt 'you could read a little bit of the letter but it was very unclear' b. fi~rst tror m a n att han dr d6d m e n det var han inte 'first you think he's dead but he wasn't'
In addition to this topic-internal function, m e n has another function in spoken language, i.e. to introduce a new topic unit or to return to a previous topic: (2) Topic-shift men: a. sd f d r han hjiilp upp dd och dd f 6 r s v i n n e r dom hiir ... hotelldirekt6ren m e d k o m p a n i ut ... och han sldpar sig bort till skdpet diir kappan hdnger ... m e n dd k o m m e r det in en liten rant ... s o m ... nja hon dr vdl ndn sdn hiir h u s m o r p d hotellet
'so he gets help up and then they go out ... the hotel manager and company ... and he drags himself over to the wardrobe where the coat is hanging ... but then a little lady comes in ... who ... yea she's one of those matrons at the hotel ... ' b . . . . d eh den hiir g a m l e m a n n e n ... j a g vet inte o m han bor ihop m e d sin dotter eller sin f r u m e n i alla f a l l f r u n hon b a k a r ndn f o r m av kaka eller tdrta ...
'... and uh this old man ... I don't know if he lives together with his daughter or his wife but anyway his wife is baking some kind of cake or tart ...' If it were possible to distinguish between the two different kinds of m e n illustrated in (1) and (2), one could use this information in speech processing algorithms (along with other information sources) to facilitate segmentation into topic-shift and topicinternal boundaries in spontaneous speech.
2. Previous studies In recent years, a number of studies have been reported on the function of cue words in discourse (Schiffrin, 1987; Fraser, 1990; Mosegaard Hansen, 1997; Byron and Heeman, 1997). Few studies, however, have concentrated on the prosodic correlates of these words (see, however, Hirschberg and Litman, 1993; Fretheim, 1989).
1064
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
Hirschberg and Litman (1993) examined both textual and prosodic features of n o w and attempted to find a set of features that could be used to distinguish between n o w ' s S(entential) function (adverbial) and its D(iscourse) function. In its D(iscourse) function, n o w , like Swedish m e n , marks the beginning of a new topic or
a return to a previous one. Hirschberg and Litman arrived at the following results after examining 100 cases of n o w : i.
Discourse n o w constituted most often a (prosodic) phrase on its own (41.3% of the cases). Sentence n o w hardly ever constituted a phrase by itself. ii. Discourse n o w appeared most often at the beginning of a (prosodic) phrase (98.4%). Sentential n o w appeared most often in non-initial position (86.5%). iii. Discourse n o w was more often deaccentuated than sentential n o w . iv. Discourse n o w co-occurred with other cue-words, e.g. w e l l n o w ... Since Swedish m e n is lexically a conjunction, and thus always occurs at the beginning of a clause, linear position (syntactic clause-initial/non-initial) cannot be used as a parameter in distinguishing between m e n ' s functions in discourse as in the case of English n o w (see (ii) above). However, one can expect that additional kinds of information such as prosodic correlates and co-occurring lexical cues could be associated with the two different functions of m e n . Following Schiffrin (1987), we are thus assuming that the meaning component [CONTRAST] associated with m e n (like English but) can have different scopes in discourse (e.g. 'referential' contrast at a relatively local, topic-internal level) and 'action contrast' at a higher topic-shift level. Fig. 1 illustrates how S - m e n a n d D - m e n are associated with these different levels of discourse structure. It also illustrates the levels of prosodic structure assumed to correlate with the discourse structure.
3. Current study In our investigation of Swedish m e n , we decided to conduct the study somewhat differently from Hirschberg and Litman (1993). Four of the authors (MH, GB, PH, MF) first examined the data using only the written transcriptions in order to see how many cases of m e n could be clearly associated with the beginning of either a new topic unit or a topic-internal unit. In doing this, we used the same labels as Hirschberg and Litman (1993), i.e. D for D(iscourse) or topic-shift m e n and S for (Sentential) or topic-internal m e n . Subsequently, auditory and visual acoustic information were used to see if prosody would aid listeners in classifying the tokens of m e n that were not classified in the same way by all four authors. 3.1. D a t a , s p e a k e r s a n d l a b e l l i n g g u i d e l i n e s
As data, we used 21 spontaneous narrations of a fragment of a silent film ( T h e l a s t l a u g h ) . All speakers were from southern Sweden and spoke a variety of southern Swedish. 15 of the speakers were female and 6 were male.
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
1065
PU=
/ ~
\
\
D - M ~ ~ ' ~ _ ~ , _
[FO-reset]
PPh
" F0-Reset. ' 1 Partial
)<
S-MEN
PW
PW
pph=ToPIC-INTERNALUNIT
i
PW
/
i
~
PW
L~
-'~z~
Fig. 1. Prosodic structure showing how (D)-men and (S)-men are associated with the beginning of a PU (Prosodic Utterance) and a PPh (Prosodic Phrase), respectively. PW stands for Prosodic Word. The prosodic structure is further shown to be correlated with two levels of discourse structure, i.e. PU boundaries align with discourse topic unit boundaries, while PPh boundaries align with topic-internal unit boundaries. The figure also shows (within square brackets) a number of prosodic correlates known to be associated with the edges of PU's and PPh's: F0-reset, H% and L% boundary tones, Final lengthening and Pause (= Silent interval).
All together the narrations included 157 tokens of men which were classified as either S, D or S/D (in cases where it was impossible to decide on either S or D). In order to facilitate the labeling task, a n u m b e r of guidelines and examples (like those above in (1) and (2) were used when categorizing men-tokens. Following is a translation of these guidelines:
- S-men expresses some kind of local (topic-internal) contradiction. Often it is a referent under discussion that is contrasted with another referent. It can even be two verbs or two phrases, e.g. two predicates with the same subject that are contrasted. This local contrast should remain if one replaces men with dock ' h o w e v e r ' or fast ' t h o u g h ' (see examples in (1)). D-men does not express any local contrast but rather introduces an utterance which begins a new topic or takes up a previous topic. In this function, men can often be left out or replaced with och ' a n d ' without changing the meaning. Men often co-occurs with i alia fall ' a n y w a y ' in this function (see examples in (2)). - If both a S(entential) (topic-internal) and a D(iscourse) (new topic) interpretation seem possible, label men as S/D. -
1066
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
4. Analysis of data labelled from text alone In about 50% of the cases, the author-labellers were in complete agreement as to the classification of men in the initial classification using text only. That is to say, on the basis of textual information alone, we could classify almost half of the cases (81). Of these 81 cases, 41 were classified as D-men, 39 as S-men and 1 as S/D. Following Swerts (1997: 515), these cases can be said to involve 'strong' boundary marking where "boundary strength is computed as the proportion of subjects agreeing on a given break". 4.1. Prosodic analysis and lexical co-occurrence patterns
Subsequent to the textual analysis of men, we made a prosodic analysis of the cases where the labellers were in agreement in order to determine which, if any, of the parameters chosen for study constituted reliable distinguishing characteristics of the two types of boundaries associated with men. The prosodic analysis was conducted using the ESPS/waves environment. In addition to the acoustic phonetic analysis, we also examined the data for patterns of co-occurring lexical items. In the prosodic analysis, the following local parameters were examined: (a) Preceding pause: one would expect a relatively longer pause before a D-men than a S-men since a D-men marks the beginning of a new topic unit and these units (often corresponding to written paragraphs) are most often preceded by longer pauses than the beginning of a new topic-internal unit (often corresponding to a clause or sentence) (Fant and Kruckenberg, 1989; Strangert, 1993; Home et al., 1995). (In the present study, pauses are defined as periods of silence (excluding occlusion phases for stop consonants) identified in the acoustic wave form.) (b) F0-reset on men: one would expect a relatively larger F0-reset on D-men than on S-men for the same reason as in (a) (see e.g. Brown et al., 1980; Grosz and Hirschberg, 1992; Sluijter and Terken, 1993; Swerts and Geluykens, 1993). (In this study, we measure 'reset' (the rise in F0 (fundamental frequency)) from the lowest F0 value in the syllable rhyme of the final syllable of the word preceding men to the highest F0 value in the vowel of men.) (c) Duration of men. One would expect that D-men would have a greater duration than S-men since the degree of coherence between D-men and what follows (i.e. new topic or return to a previous topic) is less than that between topic-internal S-men and that which follows. S-men are often perceived as short and reduced, while D-men are perceived as prominent (drawn out) reflecting the speaker's plarming of a new discourse topic. Note that this is just the opposite characterization of n o w ' s discourse form in comparison with its sentential form (Hirschberg and Litman, 1993). This is obviously due to the lexical category of the words. Since now is an adverb, it is often prominent in its 'Sentential' function when it constitutes new information, whereas men is a conjunction and by default reduced in its 'Sentential' connective function. Consequently, in their
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
1067
'Discourse' functions, one would then expect n o w and m en to exhibit a level of prominence which is clearly different from that they have in their 'Sentential' functions. (d) Prosodic Phrasing. As in the case of now, one would expect that m en in its discourse function of signalling a topic shift would constitute a Prosodic Phrase on its own, i.e. both preceded and followed by pauses reflecting the extra planning time involved when making a topic shift. In the lexical co-occurrence analysis, we looked at the word class of items following men. As Hirschberg and Litman showed in the case of now, in its 'Discourse' function, this marker often co-occurs with other cue words/phrases e.g. well now. Swedish men is no exception. In its more S(entential), topic-internal discourse function, on the other hand, m e n is often followed by a pronoun referring back to a previously introduced discourse referent. 4.2. Results
Results from the prosodic and lexical analysis are summarized in Table 1. Readers should be aware of the fact that the use of t-tests to test significance levels is questionable since we factor out the possible influence of subjects. In order to eliminate the possible influence of subjects, an ANOVA is called for. This kind of test was difficult to run on our limited database, however, due to the large amount of data that would have to be discarded. We have decided therefore to present the t-test results here, since the potential application of our findings lies in automatic speech recognition, and since speech recognizers must be able to handle all speakers' speech, we feel that the results of the t-tests give at the very least a number of interesting clues as to which prosodic correlates are more relevant than others to use in the development of automatic classifiers. Furthermore, we describe below how a neural network can use the results of the multiple speaker data to correctly classify men-tokens. 4.2.1. Prosodic correlates 4.2.1.1. Duration. Measurements of the absolute duration of m en revealed that D(iscourse)-men was on the average 310 ms (= milliseconds) long while S(entential)-men was on the average 170 ms, i.e. 55% the duration of D(iscourse) men. This
difference was statistically significant (p = 0.002). 4.2.1.2. FO-reset. Measurements of F0 (fundamental frequency) were made in the rhyme of the word preceding m e n (lowest F0 value) and in the vowel of m en (highest F0 value). Of the 41 tokens of D - m e n , 32 (78%) were characterized by a positive F0-reset (i.e where the F0 level rose). Since the phrase-final F0 level preceding m en was associated with glottalization ('creaky voice') in a number of cases, and since glottalization has the effect of lowering F0 (Dilley et al., 1996), we factored out the cases of reset associated with phrase-final glottalization into a separate group. (Glottalization was identified on the basis of irregularities in the acoustic wave form.) It is seen in Table 1 that the D - m e n tokens without reset are characterized by a mean
1068
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
reset of 5.7 ST (= Semitones) which is in line with the size of the reset one would expect at a topic-shift boundary. The mean reset associated with glottalization is, on the other hand, 13.8 ST. As for the S-men tokens, the tokens not associated with glottalization showed a mean reset of 2.2 ST (normal at a topic-internal Prosodic Phrase boundary), whereas those occurring after phrase-final glottalization exhibited a correspondingly larger reset of 7.6 ST. Disregarding potential speaker variation, these differences are significant (p = 0.0238 for glottalized reset and 0.0448 for nonglottalized reset). 4.2.1.3. Preceding pause. Measurements of pause duration revealed that 66% of D(iscourse) men (N = 27) were associated with a preceding pause that was on the average 750 ms long. Of the S(entential) men tokens, 38% (N = 15) were preceded by a pause which was on the average 460 ms long. (It is to be noted that in the measurements of duration, no consideration was taken to rate of speech.) Although the mean difference is quite large, it is not significant (p = 0.1274). 4.2.1.4. Prosodic phrasing. As in the case of English now, Swedish D(iscourse) men constituted a separate Prosodic Phrase in 34% of the D-tokens, i.e. was both preceded and followed by a pause. None of the S-men, on the other hand were characterized in this way. Fig. 2a shows an example of a D-men characterized by a relatively long duration, a preceding pause and a large F0-reset). Fig. 2b presents an example of a S-men, characterized by a relatively short duration, absence of F0-reset and no preceding pause.
4.2.2. Co-occurring lexical items In its D(iscourse) function, it was seen, as expected, that men co-occurs with other similar words (conjunctions, pause fillers, particles). Unlike now, however, the discourse markers co-occurring with men come in a position following men instead of preceding it. In 26 of the 41 cases of men labelled as D(iscourse) (63%), one of the five following labels was associated with another discourse-related cue-word or pause marker: alltsd, sedan (sd), i alia fall, i varje fall, men, (eh)(just) dd(sd), eh, ja, sd, dd, ngir. Only two of the tokens of men labelled as S were so characterized and both of them involved a filled pause segment (eh). On the other hand, 24 of the 39 tokens of men (62%) labelled as S were followed immediately by a pronoun, while only five (12%) of the D-men were so characterized. Of the five, one was the nonreferential pronoun det 'it' = Eng. 'there' (det var ndgra sddana fattigbarn 'there were some poor children'. The other four were personal pronouns followed in turn by discourse markers, e.g. han v?intade i alla fall pd hotellet 'h._eewaited anyway at the hotel'. Thus, one can hypothesize that the lexical co-occurrence cue for identifying D-men (i.e. a following discourse-related cue word marker) is quite strong. 4.3. Summary of findings for data labelled from text-only Results of the acoustic analysis of prosodic parameters associated with the topicshift (D-men) and topic-internal (S-men) show considerable mean differences in F0reset and absolute word duration. A clear difference in prosodic phrasing wherein
M, Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
1069
i-
•2 ~
!:i
--
li I.
-r.
!._
•
I -
u.-
!
L
-~t~
t~
,
I
~=~o.~
1070
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
i~
e~
=.
e.
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
1071
Table 1 Results from the labelling from text only study* Prosodic correlates s.d. rain Duration (ms) D (41) 310 240 40 S (39) 170 130 10 FO-reset (glottalized) (ST) D (17/41) 13.8 6.0 0.7 S (9/39) 7.6 6.7 1.8 FO-reset (non-glottalized) (ST) D (15/41) 5.7 5.6 0.1 S (17/39) 2.2 3.6 0.01 Preceding pause (ms) D (27/41) 750 670 30 S (15/39) 460 350 40 Men as separate Prosodic Phrase (Pauses before and after) D (14/41) (34%) S (0/39) (0%)
max 1170 600
t = 3.2 (78 dr) p = 0.0020
25.6 19.1
t = 2.41 (24 dr) p = 0.0238
16.4 12.9
t = 2.093 (30 dr) p = 0.0448
2320 1080
t = 1.557 (40 df) p = 0.1274
Co-occurring lexical cues Following cue word (one of the following 5 labels) D 26/41 (63%) S 2/39 (5%)
Following pronoun D 5/41 (12%) S 24/39 (62%)
* Prosodic correlates (Duration, F0-reset, Preceding Pause duration, and status as a separate Prosodic Phrase) as well as co-occurring lexical items (following Cue word, following Pronoun) associated with men-tokens occurring at the beginning of a new topical unit (D-men) and at the beginning of topic-internal units (S-men). Means (X), standard deviations (s.d.), minimum (min) and maximum (max) values for the continuous prosodic features Duration, F0-reset and Preceding pause are presented. (df) stands for 'degrees of freedom'. For the non-continuous features (Men as a separate Prosodic Phrase and Co-occurring lexical items), results are presented as fractions of the total number of S- and D-men tokens that are characterized by the feature. T-test results are to be interpreted with caution since they factor out potential speaker differences.
D - m e n , but not S - m e n constitutes an in d e p e nd en t Prosodic Phrase was also observed. T h e d i f f e r e n c e in p r e c e d i n g pause duration for all speakers p o o l e d together, h o w e v e r , was not significant, a result in line with that reported in Swerts and G e l u y k e n s (1993) for a single speaker. T h e tw o categories o f m e n were further strongly associated with different c o - o c c u r r i n g lexical categories, i.e. D - m e n were in o v e r 6 0 % o f the cases f o l l o w e d by another cue word, w h i l e S - m e n w e r e in o v e r 6 0 % o f the cases f o l l o w e d by a pronoun. In s u m m a r y , although the labelling o f the data in this study was done on text alone, it is seen that the ' s t r o n g ' boundaries are associated with a w h o l e constellation o f both lexical and prosodic cues that serve to mark their function in discourse.
1072
M. Home et al. /Journal of Pragmatics 33 (2001) 1061-1081
5. Analysis of data labelled from text-with-speech On the basis of the results obtained from the acoustic analysis of the tokens of S-men and D - m e n labelled from text alone, we initially expected that by listening to the data and observing the acoustic signal associated with the tokens of men that
were not categorized in the same way by all four labellers, prosodic information might provide some clues that would aid in coming to some agreement as to the categorization of men. Three of the authors (MF was not available at this time) thus re-examined all the cases of men (n = 75) which had not received a unique label during the first part of the study. Six of these were discarded since they constituted parts of speech disfluencies. The remaining 69 were examined and each case was individually discussed until labellers agreed on a unique categorization for each token. 30 tokens were classified as topic-internal (S), 29 as topic-shift (D) and 10 as ambiguous (S/D). During the process, the general impression the labellerauthors had was that, although access to the speech signal DID help in some cases, there was nevertheless a rather high degree of uncertainty as to the classification of men. This is partly reflected in the fact that 10 tokens were still assigned a S/D label, i.e. it was unclear as to how they should be classified even after listening to the speech. 5.1. Results 5.1.1. Prosodic correlates
This general uncertainty as to the classification of the cases of men that were unclear from their text-only labelling is reflected in the prosodic analysis of these tokens after reexamination and retagging using speech as well. Results are presented in Table 2. As can be seen, none of the prosodic parameters examined exhibited significant differences between S and D-men tokens. 5.1.1.1. Duration. As can be seen from Table 2, there was hardly any difference between S-men and D-men as regards their absolute duration in the labelled-fromspeech data (210 vs. 220 ms, respectively). 5.1.1.2. FO-reset. Differences in positive F0-reset between S- and D-men in the data labelled from listening were not at all significant. After factoring out glottalization, there is almost no difference in the values for F0-reset. 5.1.1.3. Preceding pause. Differences in preceding pause length for the tokens of men tagged by listening are quite similar to those tagged from text. 17 of the 29 cases labelled as D (59%) were preceded by a pause having a mean duration of 740 ms whereas only 6 of the tokens labelled as S (20%) were associated with a preceding pause which had a mean duration of 540 ms. Thus the pause duration preceding D-men labelled from speech is almost the same as that associated with the D-men labelled from text whereas the mean duration of S-men is 90 ms greater here than for the tokens labelled from text. Thus the overall difference is not as large. 5.1.1.4. Prosodic phrasing. As in the data labelled from text-alone, a clear cue to the classification of men is its status as an independent Prosodic Phrase (surrounded
M. Horne et al. /Journal of Pragmatics 33 (2001) 1061-1081
1073
by pauses). In the data labelled from text and speech, three of the D-men constituted independent Prosodic Phrases while none of the S-men were so characterized. 5.1.2. Co-occurring lexical items Even the local lexical information associated with the tokens of men labelled from speech was not significantly different in the two cases. Whereas nine cases (31%) of D-men were followed by another cue word, six tokens of S-men (17%) were also followed by a cue word. Even following pronouns did not help to distinguish the two cases of men. Although 60% of the S-men were followed by a pronoun, 41% of the D-men were also followed by a pronoun. Table 2 Results from the labelling from text-with-speech study* Prosodic correlates s.d.
min
max
210 222
160 200
20 40
690 920
t = 0.204 (57 df) p = 0.8391
10.7 10.4
6.6 4.1
0.4 2.6
22.9 14.0
t = 0.119 (16 df) p = 0.9064
3.1 3.2
1.5 4.6
0.8 0.1
6.8 16.2
t = 0.587 (23 df) p = 0.9536
D (17/29) 740 530 70 S (6/30) 540 420 60 Men as separate Prosodic Phrase (Pauses before and after) D (3/29) (10%) S (0/30) (0%)
1930 1080
t = 0.812 (21 df) p = 0.4261
Duration (ms) D (29) S (30)
FO-reset (glottalized) (ST) D (11/29) S (7/30)
FO-reset (non-glottalized) (ST) D (13/29) S (12/30)
Preceding pause (ms)
Co-occurring lexical cues
Following cue word
Following pronoun
(one of the following 5 labels) D 9/29 (31%) S 5/30 (17%)
D 12/29 (41%) S 14/30 (47%)
* Prosodic correlates (Duration, F0-reset, Preceding Pause duration, and status as a separate Prosodic Phrase) as well as co-occurring lexical items (following Cue word, following Pronoun) associated with men-tokens occurring at the begilming of a new topic unit (D-men) and at the beginning of topic-internal units (S-men). Means (X), standard deviations (s.d.), minimum (rain) and maximum (max) values for the continuous prosodic features Duration, F0-reset and Preceding pause are presented. (df) stands for 'degrees of freedom'. For the non-continuous features (Men as a separate Prosodic Phrase and Co-occurring lexical items), results are presented as fractions of the total number of S- and D-men tokens that are characterized by the feature. T-test results are to be interpreted with caution since they factor out potential speaker differences.
1074
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
5.2. Summary o f findings for data labelled using text-with-speech
No clear differences in the values for the prosodic parameters nor in the patterning of co-occurring lexical items were seen in this subset of data labelled using textwith-speech. This indicates that these 'weak' boundaries are weak even with regard to prosodic parameters and further corroborates the idea developed in Swerts (1997) that, with respect to the hearer/reader, not all boundaries are equally meaningful and this is reflected in the degree of 'strength' with which they are linguistically realized.
6. Analysis of complete database A final analysis of the data was made in order to see if any of the prosodic parameters and co-occurring lexical items could be used to distinguish between S- and D-men when pooling both sets of data (labelled from text-alone and labelled from text-with-speech). Results are presented in Table 3. As can be seen, the only parameter which alone shows a significant difference between S- and D-men is men's absolute duration (p = 0.0203). Another clear indication of men's D-status is its occurrence as a separate prosodic phrase. Whereas 24% (N = 17) of the D-men constitute a separate Prosodic Phrase, none of the S-men are characterized in this way. As regards the patteming of co-occurring lexical items, one can see that half of the D-men are followed by another cue word and 55% of the S-men are followed by a pronoun. Only 12% of the S-men were followed by another cue word. 25% of the Dmen, however, were also followed by a pronoun.
7. Implications for speech recognition/understanding From a recognition point of view, one could imagine, all other things being equal, that it is necessary to develop some way of clearly classifying all instances of men as either D (topic-shift) or S (topic-internal). However, it is not clear that it is important or necessary to be able to classify all tokens of men. From a speech understanding point of view, it stands to reason that it is the clearly marked cases (correlated with 'strong' boundaries (Swerts, 1997, 1998)) that should be important for discourse processing since it is these that the speaker probably intends the listener to pay particular attention to. In our study, these are the cases in the labelling from text-alone part where all labellers agreed as to the categorization of men. The 'strong' D-boundaries are the ones clearly associated with a new topic boundary, while the 'strong' S-boundaries are those that are clearly topic-internal. The unclear cases (related to 'weak' boundaries), on the other hand, which are not distinctly marked in any way, do not have as clear a function in discourse either and can for the most part probably be disregarded in speech processing since they are not interpretable/meaningful for the listener. Following this line of reasoning, and assuming that it is possible to distinguish between 'strong' and 'weak' cases of the cue word, we thought it would be interesting
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
1075
Table 3 Results from all data (labelled from text and labelled from text and speech) pooled together* Prosodic correlates s.d.
min
max
Duration (ms) D (70)
271
217
20
1170
t -- 2.347 (137
S (69)
194
169
10
920
12.6 8.8
6.3 5.8
0.4 0.2
25.6 19.1
t = 1.974 (42 df) p = 0.0549
4.5 2.5
4.4 3.9
0.1 0.01
16.4 16.2
t = 1.796 (56 df) p = 0.0778
2320 1080
t = 1.803 (63 df) p = 0.0761
dO p = 0.0203
FO-reset (glottalized) (ST) D (28/70) S (16/69)
FO-reset (non-glottalized) (ST) D (28/70) S (29/69)
Preceding pause (ms) D (44/70) 740 610 30 S (21/69) 480 360 40 Men as separate Prosodic Phrase (Pauses before and after) D (17/70) (24%) S (0/69) (0%) Co-occurring lexical cues
Following cue word
Following pronoun
(one of the following 5 labels) D 35/70 (50%) D 17/70 (25%) S 8/69 (12%) S 38/69 (55%) * Prosodic correlates (Duration, F0-reset, Preceding Pause duration, and status as a separate Prosodic Phrase) as well as co-occurring lexical items (following Cue word, following Pronoun) associated with men-tokens at the beginning of a new topical unit (D-men) and at the beginning of topic-internal units (S-men). Means (~), standard deviations (s.d.), minimum (min) and maximum (max) values for the continuous prosodic features Duration, F0-reset and Preceding pause are presented. (df) stands for 'degrees of freedom'. For the non-continuous features (Men as a separate Prosodic Phrase and Co-occurring lexical items), results are presented as fractions of the total number of S- and D-men tokens that are characterized by the feature. T-test results are to be interpreted with caution since they factor out potential
to see to what extent an automatic classifier could be developed to distinguish between S- and D-men within the 'strong' group. We will first describe an attempt to use a 'linear classifier' and then an experiment using a non-linear neural network.
8. A u t o m a t i c classification o f 'strong' men-tokens
8.1. Linear classifiers By using linear classifiers, each relevant parameter can be used independently to classify the data set. The method can then determine the threshold that splits the data into S-men and D-men subsets in such a way that the maximum number of correct
1076
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
classifications is obtained. The advantage of this method is that the resulting threshold value is easy to interpret. We did this for the three acoustic parameters 'preceding pause duration' (ppd), 'word duration' (dur) and 'F0-reset' (fzr). These thresholds, and the correct classification rates are presented in Table 4. Table 4 Threshold values for the three prosodic parameters 'preceding pause duration' (ppd) (in seconds (s)), 'word duration' (dur) (in seconds (s)) and 'F0-reset' (fzr) (in Semitones (ST)) used in a linear classifier for distinguishing between the 'strong' S-men and D-men tokens
Threshold Correct (%)
ppd (s)
dur (s)
fzr (ST)
0.232 58.5
0.358 65.0
8.265 70.0
If one combines these three linear classifiers, and uses a decision function where two parameters are enough to classify a men-token as S or D the rate of correct classifications rises to 71.5 %. The weakness of this method, however, is that it does not take into account the relative strength that each parameter can have. For example, if the parameters ppd and d u r both yield the category S-men, the value of fzr is not taken into consideration, even though a very high value (i.e. > 10 ST) of this parameter would be a much stronger indication of a D-men. In order to avoid this weakness a non-linear method, namely, a neural network, was subsequently tested. 8.2. Non-linear classifiers: Neural networks
Neural networks are computer models based on the operation of components of the human brain. The particular strength of neural networks lies in their power to generalise, classify and find patterns in multi-dimensional data. The disadvantage with this method, however, is that the threshold values for each parameter are obscured. However, in the attempt to reach the best possible classification, the neural network method has proven to be superior. When a neural network is supplied with some measured parameters the task of the network is to map these input features onto a classification state, that is, given the acoustic features mentioned above as input, the neural network can be trained to decide which type of class category (D-men or S-men) they match most closely. Neural networks have been applied successfully to many aspects of speech processing, see e.g. Kohonen (1988) for an approach to speech recognition, Sejnowski and Rosenberg (1986) for speech synthesis, and Johansson (1995) for language acquisition. (See Beale and Jackson (1990) for a more general introduction to neural networks.) It may be objected that a neural network is too powerful a method for the present task, since there are relatively few parameters and cases. However, we feel that due to the nature of the data (multispeaker) as well as the goal (to recognize boundary
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
1077
types realized by several interacting parameters), a non-linear method is the only method that is feasible and that can handle these interacting cues and give some reliable indication as to the relevance of the parameters that are used in the categorization (see also Ostendorf (2000) on this issue).
8.2.1. Data In order to build a classifier with a neural network, the network needs (minimally) two data sets: a training set and a validation set. In the data described above, there were 80 cases of 'strong' boundaries labelled from text only. Three of these were discarded since the F0 reset parameter could not be measured due to glottalisation. The remaining 77 cases were divided into two groups, one consisting of 33 cases and the other of 44 cases. These groups were to be used as training and validation data. Care was taken so as to get an even distribution of speakers in both groups, to include all speakers in both groups, and to have an approximately even distribution of S-men and D-men cases in each group (55% and 48% D-men, respectively). This means that the baseline performance of the network (choosing the most likely category) is around 50%. 8.2.2. Network details Two neural networks were designed and are schematically represented in Fig. 3. One used three input nodes for the parameters 'preceding pause duration' (ppd), 'word duration' (dur) and 'F0 reset' (fzr). This net thus only used prosodic correlates. The other net had two additional input nodes for the lexical information 'Following cue word' (c-w) and 'Following pronoun' (pro), thus having a total of five input nodes. Both nets had a hidden layer with two nodes, and an output layer with one node for the classification as D-men or S-men. All neural network processing and simulation was performed with the SNNS package from IPVR in Stuttgart (University of Stuttgart, 1996). The training was performed using the Resilient backpropagation (Rprop) scheme. The input was raw data, normalised around the mean of each parameter, so that input values were in the range of -1 to 1. Since the number of cases is quite small, there is a risk that the actual distribution of data in the two sets affects the outcome of the network's performance. Therefore the network was trained in two different sessions. In the first session, the smaller data set was used as training data and in the second the larger data set was used as training data. Note, however, that the two training sessions were run independently of each other. In neither case was the validation data included in the training set. The results presented constitute an average of the two different sessions. 8.2.3. Results When using the three acoustic parameters (ppd, dur, fzr), the rate of correct classifications is 90%. When the lexical features (c-w, pro) are included, the rate of correct classifications decreases to 82%. This might seem counterintuitive: adding more information should increase the network's performance. The explanation for this result, however, lies no doubt in the fact that the two lexical features are good indicators of D-men or S-men only when they OCCUR. In the cases when
1078
M. Home et al. / Journal of Pragmatics 33 (2001) 1061-1081
Output(SorD) ~ 0
0 ppd dur
A
Hiddenl a y e r ~ 0
fzr
0 0 0 0 0 ppd dur fzr c-w pro
Fig. 3. Schematic representation of the neural networks experimented with in this study. The one on the left uses three input nodes for the prosodic parameters 'preceding pause duration' (ppd), 'word duration' (dur) and 'F0 reset' (fzr). The net on the right uses two additional input nodes for the information on co-occurring lexical items: 'Following cue word' (e-w) and 'Following pronoun' (pro). Both nets have a hidden layer with two nodes, and an output layer with one node for the classification as D-men or S-men.
they do not occur, the distribution between D-men and S-men is almost 40-60, which introduces uncertainty in the network. Since there are more cases where they do not occur than when they do occur, the number of correctly classified cases decreases. (The same result would have applied to the parameter 'Separate prosodic phrase' if it had been used.) The recall and precision rates are 84,2 % and 97,0% for S-men, and 97,4% and 86,4% for D-men. The overall error rate is 18,4% for S and 17,9% for D. These rather high rates are due to the small number of cases in the data. 8.2.4. Discussion The combination of the prosodic parameters preceding pause duration, word duration, and F0-reset can predict the status of the cue word men as D-men or S-men correctly in 90% of the 'strong' cases. The tendencies observed in the data can thus be utilized to produce a rather accurate classifier. It also indicates that listeners use an aggregate, rather than one particular feature, when they distinguish between the two categories. It should be pointed out, however, that in the network described here, we have only used 'strong' cases to train the network since we were interested to see if the prosodic and lexical information from several speakers was potentially useful in categorizing the two different categories of men, S- and D-. Thus, if one were to attempt to implement such a classification network in a general system for recognizing strong S- and D- men-tokens, the system would have to be able to distinguish the strong from the weak tokens. It is not clear if this could be done in an optimal way with the present network. Therefore, if one's goal were to actually build a network for recognizing topic shift boundaries (in other words, for recognizing strong
M. Home et al. / Journal of Pragmatics 33 (2001) 1061-1081
1079
D - m e n ) , it would probably be more advisable to use three categories to train the system from the very beginning, i.e. S, D and U (for 'Unclear'). Moreover, a more implementation-oriented network could be trained with other (or more refined) prosodic and lexical parameters than those that we have considered here. For example, the case where m e n is both preceded and followed by a pause is a very strong (if not decisive) cue to its classification as D. Yet, we have not built this information into the network discussed here. As we have further noticed, the absence vs. the presence of another cue word following m e n is another very strong indicator of a Dm e n . However, it is possibly the case that the size of the window we have used in this study is too large. In order to exclude potential cue words that are in fact not cue words, the size of the window should be reduced from five words following the cue word to something like three words.
9. Conclusion From this investigation of the cue-word m e n 'but' in Swedish, it has become evident that it, along with its associated prosodic correlates and co-occurring lexical categories, constitutes an important source of information for recognizing both topicshift and topic-internal boundaries in spontaneous speech. Results from an exploratory study using a neural network show that it is possible to attain a high degree of recognition of 'strong' boundaries by using the cue word and the associated parameters chosen for this study. The results are also interesting for speech synthesis since in order to generate coherent discourse, it is important to be able to model the different kinds of prosodic constituent boundaries that occur in natural speech. The present study shows that a whole constellation of prosodic and lexical cues need to be taken into consideration in order to understand how speakers perceive and produce topic-related boundaries in spontaneous speech.
References Beale, Russel and Tom Jackson, 1990. Neural computing: an introduction. Bristol: IOP Publishing. Brown, Gillian, Karen Currie and Joanne Kenworthy, 1980. Questions of intonation. London: Croom Helm. Bruce, G6sta, Bjtiru GranstRim, Kjell Gustafson and David House, 1993. Interaction of F0 and duration in the perception of prosodic phrasing in Swedish. In: B. Granstr6m and L. Nord, eds., Nordic prosody VI, 7-22. Stockholm: Almqvist & Wiksell. Byron, Donna K. and Peter A. Heeman, 1997. Discourse marker use in task-oriented spoken dialog. Proceedings of Eurospeech 97 (Rhodes, Greece), 2223-2226. Dilley, Laura, Stefanie Shattuck-Hufnagel and Marl Ostendorf, 1996. Glottalization of word-initial vowels as a function of prosodic structure. Journal of Phonetics 24: 423-444. Fant, Gunnar and Anita Kruckenberg, 1989. Preliminaries to the study of Swedish prose reading and reading style. Speech Transmission Laboratory, Quarterly Progress and Status Report (Royal Institute of Technology, Stockholm) 2. Fraser, Bruce, 1990. An approach to discourse markers. Journal of Pragmatics 14: 383-395. Fretheim, Thorstein, 1989. The two faces of the Norwegian inference particle da. In: Harald Weydt, ed., Sprechen mit Partikeln, 403-415. Berlin: de Gruyter.
1080
M. Horne et al. / Journal of Pragmatics 33 (2001) 1061-1081
Grosz, Barbara and Julia Hirschberg, 1992. Some intonational characteristics of discourse structure. Proceedings of the 2nd International Conference on Spoken Language Processing (Banff, Canada), 429-432. Hirschberg, Julia and Diane Litman, 1993. Empirical studies on disambiguation of cue phrases. Computational Linguistics 19: 501-530. Home, Merle and Marcus Filipsson, 1995. Computational modelling and generation of prosodic structure in Swedish. Proceedings of the International Congress of Phonetic Sciences (Stockholm) 4: 364-367. Home, Merle, Eva Strangert and Mattias Heldner, 1995. Prosodic boundary strength in Swedish: Final lengthening and silent interval duration. Proceedings of the International Congress of Phonetic Sciences (Stockholm) 1: 170-173. Johansson, Christer, 1995. The development of verb forms in connectionist nets. Licentiate thesis, Department of Linguistics and Phonetics, Lund University. Kohonen, Teuvo, 1988. The 'neural' phonetic typewriter. Computer 21(3): 11-22. Mosegaard Hansen, Maj-Britt, 1997. Alors and done in spoken French: A reanalysis. Journal of Pragmatics 28: 153-187. Nakajima, Shin'ya and James F. Allen, 1993. A study on prosody and discourse stmcture in cooperative dialogues. Phonetica 50: 197-210. Nespor, Marina and Irene Vogel, 1986. Prosodic phonology. Dordrecht: Foris. Ostendorf, Mari, 2000. Prosodic boundary detection. In: Merle Home, ed., Prosody: Theory and experiment. Studies presented to GOsta Bmce, 263-279. Dordrecht: Kluwer. Ostendorf, Mari, Colin W. Wightman and N.M. Veilleux, 1993. Parse scoring with prosodic information: An analysis/synthesis approach. Computer Speech and Language 7: 193-210. Pijper, Jan R. de and Angelien Sanderman, 1994. On the perceptual strength of prosodic boundaries and its relation to suprasegmental cues. Journal of the Acoustical Society of America 96: 2037-2047. Redeker, Gisela. 1991. Linguistic markers of discourse structure. Linguistics 29:1139-1172. Schiffrin, Deborah, 1987. Discourse markers. Cambridge: Cambridge University Press. Sejnowski, Terrence J. and Charles R. Rosenberg, 1986. NETtalk: A parallel network that learns to read aloud. Cognitive Science 14:179-211. Sluijter, Agaath M.C. and Jacques M.B. Terken, 1993. Beyond sentence prosody: Paragraph intonation in Dutch. Phonetica 50: 180-188. Strangert, Eva, 1993. Speaking style and pausing. Phonum 2: 121-137. Swerts, Marc, 1997. Prosodic features at discourse boundaries of different strength. Journal of the Acoustical Society of America 101: 514-521. Swerts, Marc, 1998. Filled pauses as markers of discourse stmcture. Journal of Pragmatics 30: 485-496. Swerts, Marc and Ronald Geluykens, 1993. The prosody of information units in spontaneous monologues. Phonetica 50: 189-196. University of Stuttgart, 1996. Stuttgart Neural Network Simulator Manual. Institute of Parallel and Distributed High-Performance Systems, University of Stuttgart.
Merle Horne Ph.D. (www.ling.lu.se/persons/Merle) (born 1950), Senior lecturer in general linguistics at Lund University, Sweden. Involved in interdisciplinary research on prosody and its relation to information structure in spontaneous discourse. Currently leading a project on Swedish dialogue prosody. Petra Hansson (www.ling.lu.se/persons/Petra) (born 1974) is a doctoral student in phonetics at Lund University. Thesis research centered on dialogue prosody, in particular prosodic factors related to prominence and phrasing. G6sta Bruce Ph.D. (www.ling.lu.se/persons/Gosta) (born 1947), Professor of Phonetics at Lund University, Sweden. Is internationally well-known for his work on modelling Swedish prosody. Currently leading a project on the phonetics and phonology of Swedish dialects.
M. Home et al. /Journal of Pragmatics 33 (2001) 1061-1081
1081
Johau Frid (www.ling.lu.se/persons/JohanF) (born 1970) is a doctoral student in phonetics at Lund University. Research centered on computational aspects of prosody related to speech synthesis and recognition.
Marcus Filipsson (born 1967) was a research engineer at the Dept. of Linguistics during the time the study reported on here was being conducted. He is currently employed by Telelogic, Malm6, Sweden.