- Email: [email protected]
Development of a method for automatic basso continuo playing
Information Processing and Management 47 (2011) 440–451
Contents lists available at ScienceDirect
Information Processing and Management journal homepage: www.elsevier.com/locate/infoproman
Development of a method for automatic basso continuo playing Masahiro Niitsuma a,⇑, Masaki Matsubara b, Masaki Oono c, Hiroaki Saito b a
School of Music and Sonic Arts, Queen’s University Belfast, BT7 1NN, United Kingdom Graduate School of Science and Technology, Keio University Hiyoshi, 3-14-1 Kouhoku-ku Yokohama-shi, Kanagawa-ken, Japan Department of Communication Engineering & Informatics, Faculty of Informatics and Engineering, The University of Electro-Communications, 1-5-1 Chofugaoka, Chofu-shi, Tokyo, Japan
b c
a r t i c l e
i n f o
Article history: Received 4 November 2008 Received in revised form 13 October 2010 Accepted 4 November 2010 Available online 26 November 2010 Keywords: Basso continuo Beam search Dynamic programming Harmonisation Baroque music Music information retrieval Computational musicology
a b s t r a c t The purpose of this study is to develop an automatic basso continuo playing system. In order to find a musically appropriate sequence of chords, we propose the principle of ‘‘harmony cost’’, which is defined as the sum of two different costs: one is the ‘‘local cost’’ which indicates the unlikelihood of a certain chord allocation and the other is the ‘‘transition cost’’, which indicates the unlikelihood of a certain connection between two chords. Automatic basso continuo playing is realised by searching for the optimal chord sequence, which minimises the accumulated harmony cost. The proposed method is evaluated by three experiments. Each experiment verifies the effectiveness of the proposed method. Ó 2010 Elsevier Ltd. All rights reserved.
1. Introduction In baroque music composers indicated the accompaniment of their works in the notes of the bass part with numerical figures, and performers improvised the accompaniment of them on the basis of this information. This figured bass is called basso continuo. Fig. 1a shows the accompaniment part of the Brandenburg concerto No. 5 by J.S. Bach. In Fig. 1a, only the first chord for the right hand is written. Instead of writing the other chords in the same manner, a small Arabic numeral is written on the part of the left hand, which is the clue to the proper realisation of the chords that musically accompany the other parts. Fig. 1b is one example of the realisation of Fig. 1a. Basso continuo playing requires knowledge and experience in various aspects such as theory of harmony, historical background, character of instruments, and the art of accompaniment, thus preventing the proper realisation especially for the nonspecialist of baroque music. Therefore in this age a considerable number of people have to practise playing instruments, for example recorders, without basso continuo. These circumstances have brought about a great demand for a system that plays basso continuo automatically. Many projects that demonstrate theoretical expertise are primarily concerned with computing stylistically correct compositions. The automatic counterpoint program by Schottstaedt (1989), the Experiments in Musical Intelligence (EMI) system by Cope (2004), and the CHORAL expert system by Ebcioglu (1988) are examples of this. This paper presents an automatic basso continuo playing method based on ‘‘harmony cost’’, which indicates the unlikelihood of chords. In order to find a musically appropriate sequence of chords, harmony cost is defined as the sum of two different costs; the local cost which indicates the unlikelihood of a certain chord location and the transition cost, which
⇑ Corresponding author. E-mail address: [email protected] (M. Niitsuma). 0306-4573/$ - see front matter Ó 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.ipm.2010.11.001
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
441
Fig. 1. An example of original score (a) and realised score (b).
indicates the unlikelihood of a certain connection between two chords. Automatic basso continuo playing is realised by searching for the optimal chord sequence, which minimises accumulated harmony cost. Additionally, the use of the beam search algorithm significantly reduces the amount of memory used, thereby enabling the basso continuo realisation regardless of its length. This paper is structured as follows: Section 2 discusses an overview of basso continuo playing and the proposed methodology; Section 3 describes a simulation in order to clarify how the proposed method operates; Section 4 illustrates three experiments to evaluate the proposed method; finally Section 5 offers concluding remarks and a discussion of future work. 2. A method for automatic basso continuo playing 2.1. Overview of basso continuo playing This section shows an overview of basso continuo realisation. Firstly, figures are interpreted. The principle of figuring is to notate only intervals1 over the bass note. Both numerical figures and a unique key are enough for complete identification of an interval. For example, Fig. 2 indicates the actual interval indicated by each numerical figure; bar one and two are the case of C major and bar three and four are that of C minor. If more than two figures are played at the same time, chords are constructed, and how different figures should be played at the same time can be decided on the basis of a bass note and numerical figures. This decision is the first task of basso continuo realisation. Secondly, the spacing2 of chords is decided. Even though three figures are given, this leaves room for considerable variety in the construction of a chord. Two general types are open position and close position. Close position occurs when the three upper voices are as close together as possible. In open position, the upper voices are separated so that a chord tone could be inserted between either alto and soprano, tenor and alto, or both pairs of voices (Aldwell & Schachter, 2003). These chords are usually constructed by playing all the notes indicated by the figures; these are called complete chords. Alternatively, they can be played leaving some notes out; these are called incomplete chords. Fig. 3 indicates how the same figured bass can be played differently in four-part texture, which consists of four different voices: soprano, alto, tenor, and bass. A significant number of difficulties are involved in these decisions; e.g., careless spacing of chords leads to a large movement of each voice, or an awkward motion in the long musical context; in the more elaborated realisation, chords have to be allocated so that they generate motivic development or counterpoint. These difficulties can only be resolved by players’ experience and sense of beauty. As understood from the existence of a considerable number of textbooks stating the rules of continuo playing, a rule-based system can imitate such a player’s skills to some extent. However, a purely rule-based system can not imitate human flexibility to violate such rules to prioritise other aspects such as their own musical tastes, the specific style of the piece, or the limitation of the performer’s skills.
1 The distance between two pitches. Intervals are traditionally labelled according to the number of steps they embrace in a diatonic scale, counted inclusively: thus from C up to D or down to B is a 2nd, another step up to E or down to A makes a 3rd, etc. (Mark Lindley et al., 2010). 2 The arrangement of the notes of a chord, according to the requirements of the individual voices. If the upper voices are close together, the spacing is described as close position, or close harmony; if not, the arrangement is called open position, or open harmony (Rushton, 2010).
442
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8
Fig. 2. An example of figures and their implications: bar one and two indicate that of C major and bar three and four indicate that of C minor.
Fig. 3. An example of chord spacing: (a) and (b) are close position chords; (c) and (d) are open position chords; (a) and (c) are complete chords; and (b) and (d) are incomplete chords.
The main objective of this paper is to mimic these human skills of basso continuo playing by integrating rule-based methods with beam search dynamic programming. Possible chords are generated by a certain number of rules from a given figured bass, which is followed by beam search dynamic programming to search for the optimal chord sequence. Dynamic programming enables basso continuo realisation without stating all the detailed rules concerning chord allocation and voice leading, and the use of the beam search makes it possible to consider long musical contexts with a small amount of memory. Moreover, many different results can be generated thorough this process, thereby realising the most important element of basso continuo playing, which is that it varies from performer to performer. 2.2. Overview of the proposed method To simplify the process, the following basic assumptions are made in the proposed method: (1) Bass parts are figured completely and the key is clear. Generally, bass parts tend to be figured incompletely from a theoretical point of view. In this paper, it is assumed that figures are complete enough to specify the basic types of chords, to focus on how to construct chords from complete figures and how to allocate them. Automatic figure analysis from incompletely figured bass is reported in Knopke and Wead (2007); however, there remain unsolved problems with this type of analysis and so it can not be widely used yet. (2) Basso continuo is realised in the four-part vocal writing style. Generally, basso continuo tends to be instrumental with a flexible number of voices. However, four-part vocal writing tends to be the first material for learners of basso continuo playing. The main reason is that it has applicability to music of greater complexity even though it is simple. Much complex instrumental music is based on a framework of four-part vocal writing. Therefore, the four-part style is assumed in the proposed method. (3) Only harmonic notes are produced unless the figure indicates non-harmonic notes.3 The final goal of basso continuo playing is an art of accompaniment. However, only when the player has mastered all the elementary technical and practical aspects of basso continuo playing, can purely musical aspects such as rhythm, articulation, and supporting
3 Non-harmonic notes are in a sense melodic ornaments, and many of the names used to describe them have been borrowed from the terminology for ornamentation (e.g. appoggiatura, broderie, vorschlag) (Drabkin, 2010).
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
443
the soloist be considered (Christensen, 2002). Therefore, such musical embellishments are excluded. Moreover, the solo part is not considered. In fact, a significant number of historical sources stress that the position of chords should be treated not in relation to the solo part but to the bass. The most important thing is to find the best voice leading and the best connection of the harmonies (Christensen, 2002, p. 40). The proposed method consists of three modules: the figure interpretation module, the chord allocation module, and the module which finds the most appropriate chord sequence. In the first module, the input figures are interpreted on the basis of forty-six rules which have been formulated according to modern textbooks. In the second module, the figures augmented by the figure interpretation module are received, and the possible chords in which all voices are arranged in the proper range are stored. In the third module, the optimal sequence of chords is sought by the dynamic programming method. Finally, the complete accompaniment is produced in MIDI format (Huber, 2007) from which a musical score is generated (output). The detail of each module is demonstrated in the following sections. 2.3. Figure interpretation Three figures are necessary to realise basso continuo into a four-part chord. Even though figures are complete and the basic type of chords is specified, there still exist different ways to play them in four-part chords. There is a certain number of rules in this interpretation method, which are extracted into forty-six rules (Table 1) according to modern textbooks. In the figured bass notation used for the theory of harmony, the interpretation of one specific figure is unique (Aldwell & Schachter, 2003); however, in the Baroque era, there were a lot of interpretations, and performers had to judge which was the most appropriate. This proper judgement can be realised by using figure interpretation cost which is addressed in Section 2.5. 2.4. Chord allocation In this module, the figures interpreted by the figure interpretation module are received, and the possible chords in which all voices are set in proper range are stored. For example, if bass note ‘‘D’’ and three figures ‘‘3 5 8’’ are input, three pitch spellings ‘‘F A D’’ are generated. Next, each pitch spelling is assigned to each part. Finally, the MIDI number of each part is obtained by deciding the octave. Chords satisfying Eq. (1) are stored as possible chords.
bass 6 ten 6 alt 6 sop 6 MaxSop
ð1Þ
where sop, alt and ten denote the MIDI number of each part and MaxSop denotes the highest MIDI number sop can reach. This procedure makes it possible to discard any chords which include the crossover between any parts. Fig. 4 shows all the chord allocations of ‘‘3 5 8’’ above ‘‘D’’ note in the case of D major piece, generated by Eq. (1), with MaxSop = 81. How to choose an appropriate chord sequence is described in the next section.
Table 1 Figure interpretation rules (excerpts). Figure
Interpretation
None 6 6 6 6 7 24 25
3 3 3 3 3 3 2 2
5 6 3 6 4 5 4 5
8 6 6 8 6 7 6 5
Fig. 4. Possible chord allocations of ‘‘3 5 8’’ above ‘‘D’’; generated by Eq. (1) with MaxSop = 81.
444
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
Fig. 5. Two different allocations of the same chord: (a) realisation in close position; and (b) same chord in open position. There is higher likelihood of close position chord being simulated by lowering the chord location cost of (a) than that of (b).
Table 2 An example of factors for chord location cost. N: Negative rule, P: Positive rule. Factor
N/P
Each voice is not set in a proper range Leading note is doubled Close position Not prepared seventh Not prepared ninth
N N P N N
2.5. Finding the most appropriate chord sequence The optimal sequence of chords can be found by Dynamic Programming (DP) (Huang, Acero, & Hon, 2001). DP searches for the route with minimum cost, which is defined as two kinds of costs; the local cost and the transition cost. Let Xt be a set of all possible chords which can be located at time t, and let xt 2 Xt be a certain chord at time t. Ideally speaking, the optimal chord sequence should be sought for each internal cadence by using a sliding window with beam search. However, automatic cadence analysis is not practical at this moment. Therefore, optimal chord sequence of entire pieces is sought. 2.5.1. Local cost Local cost L(xt) indicates the unlikelihood of a certain chord xt 2 Xt at time t. L(xt) consists of two different costs: figure interpretation cost L1(xt) and chord location cost L2(xt). Let k denote an adjustment factor, then L(xt) is defined by Eq. (2).
Lðxt Þ ¼ L1 ðxt Þ þ kL2 ðxt Þ
ð2Þ
Figure interpretation cost L1(xt) represents the unlikelihood of a certain interpretation of figure. The general factor for the figure interpretation cost is the scale degree of a bass note. Some figures can be interpreted differently depending on composers and regions. For instance, the figure ‘‘6’’ is usually interpreted as the first inversion chord, and is interpreted as ‘‘3 3 6’’, ‘‘3 6 6’’ or ‘‘3 6 8’’. However, some theorists and composers used figure ‘‘6’’ as ‘‘3 4 6’’. This preference can be reflected by lowering the cost of the interpretation of ‘‘6’’ to ‘‘3 4 6’’ to make it less than the other interpretation. This results in the generation of different accompaniment with the harsh sound of ‘‘4’’. Chord location cost L2(xt) indicates the unlikelihood of a certain chord xt at time t. For instance, the cost of a close chord (Fig. 5a) is smaller than that of an open chord (Fig. 5b), since a close chord is more common than an open chord unless there is a specific reason for it. Table 2 lists some of the factors for chord location cost such as leading note4 doubling and unprepared5 seventh and ninth. The larger k becomes, the more complete chords become and the more usual spacing tends to be used with the general figure interpretation. On the contrary, the smaller k becomes, the more special figure interpretation is used, even though it causes poor chord spacing.
4 The seventh degree of the major, harmonic minor, or ascending melodic minor scale, so called because it lies a semitone below the tonic and therefore has a strong tendency to lead up to it (Kennedy & Bourne, 2010). 5 In part-writing, the softening of the dissonant effect of an accented non-harmonic note (i.e. an appoggiatura) by presenting it as a consonant note in the previous chord; the result is called a prepared appoggiatura. The dissonance is softened further if the preparation note is tied to the appoggiatura, creating a suspension (Kennedy & Bourne, 2010).
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
445
Fig. 6. Two different chord transition above the same figured bass: (a) realisation without successive motion; (b) realisation with successive octave between Soprano and Tenor. There is low likelihood of successive motion being simulated by lowering the transition cost of (a) than that of (b).
Table 3 An example of factors for transition cost. Factor
N/P
Hidden octave Hidden unison Successive fifth Successive octave Successive unison Contrary motion with other voices Motion in the same direction as the previous direction of the soprano part Leap larger than octave Seventh interval motion Augmented interval motion
N N N N N P P N N N
Table 4 A guideline for weighting of the factors of each cost; the importance of each factor is shown on a scale of one to five. Factor
Importance (1–5)
Melodic motion of soprano part Smooth connection of each part Successive octave Successive unison Hidden octave Hidden fifth Leading unresolved Seventh unresolved Missing third Augmented skip
5 4 5 5 3 3 2 2 1 1
2.5.2. Transition cost Transition cost T(xt, xt+1) indicates the unlikelihood of the connection between two chords when there is a certain chord xt at time t and chord xt+1 at time t + 1. For instance, Fig. 6b has a successive6 octave between soprano and tenor, therefore the transition cost of Fig. 6b is larger than that of Fig. 6a. Table 3 lists some of the factors for transition cost. 2.5.3. Determining cost It would be ideal to decide the cost automatically by machine learning methods. However, it is difficult at this point to obtain a large amount of training data of realised basso continuo. Moreover, because basso continuo realisation varies from one performer to another, it would not be appropriate to calculate the general weight. Therefore, the cost was decided by humans by trial and error. A major guideline for this weighting is shown in Table 4. This weighting varies from performer to performer, and these guidelines are not the only solution. 2.5.4. Searching by DP In standard DP, accumulated cost is memorised at each state and the route is traced back to obtain the best state sequence. If this method is used in basso continuo realisation, the amount of memory needed becomes O(en) as the length of music n increases. Let us denote by jXtj the number of chords that can be located at time t, and by a the upper bound of capacity required to memorise one chord, then the amount of memory needed M(n) can be expressed as Eq. (3). 6 In part-writing, the simultaneous duplication of the melodic line of one part by another at the interval of a perfect 5th or an octave, or any equivalent compound interval (Drabkin, 2010).
446
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
MðnÞ ¼ Pnt¼1 a jX t j
ð3Þ n
Let us put b = max{jX1j, jX2j, . . . , jXnj}, then the amount of memory needed will be denoted as Eq. (4), which exceeds O(e ).
MðnÞ P ðabÞn
ð4Þ
This is a crucial problem because the length of music is variable. As demonstrated in Tillmann and Ney (2003), the amount of memory used can be reduced by using the beam search algorithm, which only keeps states with small cost only. However, the naive beam search might neglect some interpretation of figures without considering continuing chords, which may lead to inappropriate chord connections. This problem can be handled by searching backwards, which enables the figure interpretation in the light of the following chords. Moreover, the amount of branching can be decreased significantly by searching backwards since tonal music tends to converge to cadence. Let l denote an adjustmental factor, then ‘‘harmony cost’’ P(xt, xt+1) can be defined by Eq. (5).
Pðxt ; xtþ1 Þ :¼
Lðxt Þ þ l Tðxt ; xtþ1 Þ
if 1 6 t 6 n 2
Lðxt Þ þ Lðxtþ1 Þ þ Tðxt ; xtþ1 Þ if t ¼ n 1
ð5Þ
The larger l becomes the more the chords tend to be incomplete and the more unusual the spacing becomes. On the contrary, the smaller l becomes, the more the chords tend to become complete with usual spacing. Automatic basso continuo playing is realised by searching for the optimal chord sequence, which minimises the accumulated harmony cost calculated on the basis of Eq. (5). Let the input figured bass notes be b1, b2, . . ., bt, . . ., bn, all the possible chords above bt be chord(bt), an array for chords be V, the number of elements of V be jVj, an array for chord sequences be W, and all the possible chord sequences obtained by connecting each of all the elements of W and V be VW, then the searching algorithm is explained as shown in Algorithm 1. Algorithm 1. Search for the optimal chord sequence. Input: A set of figured bass notes b1, b2, . . ., bt, . . ., bn Output: An optimal chord sequence 1: initialise W and V 2: for t = n to 1 do 3: V chord(bt) 4: if t = n then 5: wj Vj 6: W V 7: else 8: W VW 9: end if 10: keep the best w elements of W on the basis of the accumulated harmony cost calculated by Eq. (5), and remove the other elements 11: end for 12: return the best chord sequence from W
3. Simulation This section looks at a simulation of the proposed method. The format of the input data is shown in Fig. 7a. In the header, key, meter and whether there is an anacrusis7 or not are read. In this example, the system reads C major, 4/4 and no anacrusis. In the body part of the input data, pitch, duration, and figures of the bass notes are read line-by-line, and stored in chronological order. In this example, E, a quarter-note and ‘‘6’’ are stored on the first line. On the second line, D, a quarter-note and ‘‘6’’ are stored. On the third line, C, a quarter-note and no figures are stored. On the fourth line, B (‘‘,’’ indicates that this note is an octave lower than the default height), a quarter-note and ‘‘6’’ are stored. On the fifth line, C, a half-note and no figures are stored. Each figure is interpreted as three figures. In this example, no figures are interpreted as ‘‘3 5 8’’, and ‘‘6’’ is interpreted as ‘‘3 3 6’’, ‘‘3 6 6’’, ‘‘3 6 8’’, and ‘‘3 4 6’’. These figures are processed by the chord allocation module to generate all the possible chords in which all voices are set in proper range. On the first line, it receives the bass note E and figures ‘‘3 3 6’’ (one of four interpretations). G is obtained as the third above the bass note, C is obtained as the sixth above the bass note, and so forth. This procedure decides each pitch spelling which composes one chord, followed by the calculation of possible allocation. From the calculated chord candidates, the beam search seeks the optimal chord sequence, which minimises the accumulated harmony cost. In this example, the beam width is equal to the number of possible chords (39 in number) above the last
7
Anacrusis refers to an anticipatory note occurring before the first bar-line of a piece.
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
(a)
447
Header Key : C Meter : 4/4 Anacrusis or not : normal Body No.
Note
Duration
1
Figures
E
4
6
2
D
4
6
3
C
4
4
B,
4
6
5 . . .
C . . .
2 . . .
. . .
(b)
Fig. 7. An example of the input file and input score.
Fig. 8. Hypothesised score by the proposed method from the input file shown in Fig. 7.
bass note. Finally the output for the best chord sequence is in Lilypond format,8 thus creating a MIDI file and a score file. An example of the output file is shown in Fig. 8, where the figure ‘‘6’’ is interpreted differently by its location; ‘‘6’’ above the first bass note is interpreted as ‘‘3 6 8’’; on the other hand ‘‘6’’ above the second bass note is interpreted as ‘‘3 4 6’’. 4. Experiment This section addresses three experiments to evaluate the effectiveness of the proposed method. In the first experiment, the amount of memory used was verified. In the second experiment, four soloists played the melody parts of four different pieces with three different accompaniments, including the output of the proposed system. In the third experiment, a professional continuo player evaluated the scores of the accompaniments hypothesised by the proposed method. All the experiments were carried out on Intel Core Duo 2.16 GHz CPU with 2GB memory.
8 Lilypond is an automated engraving system. It formats music beautifully and automatically, and has a friendly syntax for its input files. It is free software and available from http://lilypond.org/.
448
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
4.1. Experiment 1 In the first experiment, the correlation between the amount of memory used and the number of input bass notes was investigated. The input file was excerpted from the D. Purcell Sonata in F major. In Fig. 9, the X-axis indicates the number of input bass notes, and the Y-axis indicates the amount of memory used (Fig. 9a) and the execution time (Fig. 9b). In both Figs. 9a and b, the effect of the beam search algorithm appears clearly. For example, the amount of memory used without the beam search, when n = 4, is 4.117 megabytes; on the other hand, that with the beam search is 0.0011 megabytes. 4.2. Experiment 2 In the second experiment, four alto recorder soloists played four different pieces with three different accompaniments. They have been familiar with music from infancy, and have a good command of alto recorders. Table 5 shows the pieces used in the experiment, which are different from the pieces by which the costs were determined, and incomplete figures and key change information were supplemented in advance. The soloists were asked to evaluate each accompaniment on a scale of one to five. Three accompaniments of each piece were prepared as MIDI files, where all files have the same volume and the same tone colour.
(a)
Used memory (Megabytes)
Bass-Part only (BPO) Four-Part by a professional musician (FPPM) Four-Part by our system (FPOS)
withBeam withoutBeam
10 1 0.1 0.001 0.0001
1
2
3
4
5
Number of inputs
(b) Execution time (msec)
10 10 10 10 10 10
5
withBeam withoutBeam 4
3
2
1
0
1
2
3
4
5
Number of inputs Fig. 9. A comparison of the amount of memory used and execution time.
Table 5 Details of the pieces used for experiment. Title
Key
The number of bars
D. Purcell Sonata Telemann Sonata Marcello Sonata Barsanti Sonata
F Dur d moll F Dur g moll
52 17 38 38
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451 5
449
BPO FPPM
4.5 FPOS
Score
4 3.5 3 2.5 2 1.5 1 Purcell Telemann Marcello Barsanti
Piece Fig. 10. Evaluated value for each piece by the soloists.
Fig. 11. Two different realisations of the same bass excerpted from Purcell’s sonata (a) are realised by the proposed method and (b) by a professional. The arpeggiation employed in (b) was evaluated positively by some subjects.
Table 6 Negative factors pointed out by professionals. No.
Factor
(1) (2)
Non-harmonic tone is harmonised Soprano part is so high that it obscures melody part
Fig. 10 illustrates that the evaluation of four-part by our system (FPOS) is higher than only basso (BPO), and almost the same as that by the professional musician (FPPM). With regard to the pieces of Purcell, because the harmonic changes are few, chords were played with arpeggiation in FPPM shown in Fig. 11b. Some subjects expressed their opinions regarding the FPPM of Purcell that chords played with arpeggiation sounded harmonious. On the other hand, chords were merely played with the same rhythm as that of bass notes in FPOS as shown in Fig. 11a, and there was an opinion about the FPOS of Purcell that the chords were felt to be harsh. This indicates that sustaining one chord with the same harmony without arpeggiation can become harsh and tedious. 4.3. Experiment 3 In the third experiment, a professional of basso continuo evaluated realised scores to find out how to improve the proposed method. No. (1) shown in Table 6 is caused by harmonising non-harmonic tones. Rapid bass movement could sound noisy when all the notes are harmonised as shown in Fig. 12. In order to solve this problem, it is essential that non-harmonic tone be properly distinguished from harmonic tone. No. (2) implies that the soprano part could become so high that it obscures the melody part. Both of these problems stemmed from the assumptions shown in Section 2.2, thereby implying that realisation without each assumption should be investigated.
450
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
Fig. 12. Two different realisations of the same bass excerpted from Marcello’s sonata (a) are realised by the proposed method and (b) by a professional. The harmonisation of all the 16th notes in (a) was pointed out as a negative factor (1) shown in Table 6.
5. Conclusion and future work In this paper, a method for automatic basso continuo playing was proposed. In order to find a musically appropriate sequence of chords, ‘‘harmony cost’’ was defined as the sum of two different costs, and the beam search algorithm was adopted to reduce the amount of memory used. Experimental results showed the effectiveness of our approach and its similarity to human realisation. In order to achieve the complete development of an automatic basso continuo playing method, time and effort will be devoted to further research. To decide the cost automatically by machine learning, a database of realised basso continuo has to be developed. The implementation of a more historical manner of basso continuo playing is an interesting subject, which may require cooperation with musicologists. As mentioned in Section 2.2, this paper deals with the foundation of basso continuo playing, and basso continuo playing without each and every assumption made in this paper requires further research: firstly, basso continuo playing in a more instrumental style with a variable number of voices should be investigated; secondly, figure analysis from unfigured or incompletely figured bass seems to be a worthwhile subject to investigate; finally, a more elaborate realisation with embellishments or interaction with the solo part will involve many problems, thereby stimulating further research. We intend to work on these tasks on the basis of the method proposed in this paper.
Acknowledgement This research was conducted under the auspices of the Exploratory IT Human Resources Project 2006 of the InformationTechnology Promotion Agency, Japan, and the financial support by the Nakajima Foundation. References Aldwell, E., & Schachter, C. (2003). Harmony & voice leading. New York: Schirmer Books. Christensen, J. B. (2002). 18th century continuo playing: A historical guide to the basics. Kassel: Barenreiter. Cope, D. (2004). Computer models of musical creativity. Cambridge: MIT Press. Drabkin, W. (2010). Consecutive fifths, consecutive octaves. The new grove dictionary of music and musicians, 2nd ed. Retrieved 21.07.10. Drabkin, W. (2010). Non-harmonic note. The new grove dictionary of music and musicians, 2nd ed. Retrieved 21.07.10. Ebcioglu, K. (1988). An expert system for harmonizing four-part chorales. Computer Music Journal, 12(3), 43–51. Huang, X., Acero, A., & Hon, H.-W. (2001). Spoken language processing: A guide to theory, algorithm, and system development. London: Prentice Hall. Huber, D. M. (2007). The MIDI manual, a practical guide to MIDI in the project studio (3rd ed.). London: Focal Press. Kennedy, M. & Bourne, J. (Eds.), (2010). Preparation. The new grove dictionary of music and musicians, 2nd ed. Retrieved 21.07.10. Kennedy, M. & Bourne, J. (Eds.), (2010). Leading note. The new grove dictionary of music and musicians, 2nd ed. Retrieved 21.07.10.
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
451
Knopke, I. & Wead, A. (2007). Computer-based implementation of basso continuo rules for figured bass realizations. In Proceedings of international computer music conference (Vol. 2, pp. 188–191). Mark Lindley, M. C. & Greated, C. (2010). Interval. The new grove dictionary of music and musicians, 2nd ed. Retrieved 21.07.10. Rushton, J. (2010). Spacing. The new grove dictionary of music and musicians, 2nd ed. Retrieved 21.07.10. Schottstaedt, W. (1989). Automatic species counterpoint. In M. Mathews & J. Pierce (Eds.), Current directions in computer music research (pp. 225–262). Tillmann, C., & Ney, H. (2003). Word reordering and a dynamic programming beam search algorithm for statistical machine translation. Computational Linguistics, 29(1), 97–133.
Contents lists available at ScienceDirect
Information Processing and Management journal homepage: www.elsevier.com/locate/infoproman
Development of a method for automatic basso continuo playing Masahiro Niitsuma a,⇑, Masaki Matsubara b, Masaki Oono c, Hiroaki Saito b a
School of Music and Sonic Arts, Queen’s University Belfast, BT7 1NN, United Kingdom Graduate School of Science and Technology, Keio University Hiyoshi, 3-14-1 Kouhoku-ku Yokohama-shi, Kanagawa-ken, Japan Department of Communication Engineering & Informatics, Faculty of Informatics and Engineering, The University of Electro-Communications, 1-5-1 Chofugaoka, Chofu-shi, Tokyo, Japan
b c
a r t i c l e
i n f o
Article history: Received 4 November 2008 Received in revised form 13 October 2010 Accepted 4 November 2010 Available online 26 November 2010 Keywords: Basso continuo Beam search Dynamic programming Harmonisation Baroque music Music information retrieval Computational musicology
a b s t r a c t The purpose of this study is to develop an automatic basso continuo playing system. In order to find a musically appropriate sequence of chords, we propose the principle of ‘‘harmony cost’’, which is defined as the sum of two different costs: one is the ‘‘local cost’’ which indicates the unlikelihood of a certain chord allocation and the other is the ‘‘transition cost’’, which indicates the unlikelihood of a certain connection between two chords. Automatic basso continuo playing is realised by searching for the optimal chord sequence, which minimises the accumulated harmony cost. The proposed method is evaluated by three experiments. Each experiment verifies the effectiveness of the proposed method. Ó 2010 Elsevier Ltd. All rights reserved.
1. Introduction In baroque music composers indicated the accompaniment of their works in the notes of the bass part with numerical figures, and performers improvised the accompaniment of them on the basis of this information. This figured bass is called basso continuo. Fig. 1a shows the accompaniment part of the Brandenburg concerto No. 5 by J.S. Bach. In Fig. 1a, only the first chord for the right hand is written. Instead of writing the other chords in the same manner, a small Arabic numeral is written on the part of the left hand, which is the clue to the proper realisation of the chords that musically accompany the other parts. Fig. 1b is one example of the realisation of Fig. 1a. Basso continuo playing requires knowledge and experience in various aspects such as theory of harmony, historical background, character of instruments, and the art of accompaniment, thus preventing the proper realisation especially for the nonspecialist of baroque music. Therefore in this age a considerable number of people have to practise playing instruments, for example recorders, without basso continuo. These circumstances have brought about a great demand for a system that plays basso continuo automatically. Many projects that demonstrate theoretical expertise are primarily concerned with computing stylistically correct compositions. The automatic counterpoint program by Schottstaedt (1989), the Experiments in Musical Intelligence (EMI) system by Cope (2004), and the CHORAL expert system by Ebcioglu (1988) are examples of this. This paper presents an automatic basso continuo playing method based on ‘‘harmony cost’’, which indicates the unlikelihood of chords. In order to find a musically appropriate sequence of chords, harmony cost is defined as the sum of two different costs; the local cost which indicates the unlikelihood of a certain chord location and the transition cost, which
⇑ Corresponding author. E-mail address: [email protected] (M. Niitsuma). 0306-4573/$ - see front matter Ó 2010 Elsevier Ltd. All rights reserved. doi:10.1016/j.ipm.2010.11.001
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
441
Fig. 1. An example of original score (a) and realised score (b).
indicates the unlikelihood of a certain connection between two chords. Automatic basso continuo playing is realised by searching for the optimal chord sequence, which minimises accumulated harmony cost. Additionally, the use of the beam search algorithm significantly reduces the amount of memory used, thereby enabling the basso continuo realisation regardless of its length. This paper is structured as follows: Section 2 discusses an overview of basso continuo playing and the proposed methodology; Section 3 describes a simulation in order to clarify how the proposed method operates; Section 4 illustrates three experiments to evaluate the proposed method; finally Section 5 offers concluding remarks and a discussion of future work. 2. A method for automatic basso continuo playing 2.1. Overview of basso continuo playing This section shows an overview of basso continuo realisation. Firstly, figures are interpreted. The principle of figuring is to notate only intervals1 over the bass note. Both numerical figures and a unique key are enough for complete identification of an interval. For example, Fig. 2 indicates the actual interval indicated by each numerical figure; bar one and two are the case of C major and bar three and four are that of C minor. If more than two figures are played at the same time, chords are constructed, and how different figures should be played at the same time can be decided on the basis of a bass note and numerical figures. This decision is the first task of basso continuo realisation. Secondly, the spacing2 of chords is decided. Even though three figures are given, this leaves room for considerable variety in the construction of a chord. Two general types are open position and close position. Close position occurs when the three upper voices are as close together as possible. In open position, the upper voices are separated so that a chord tone could be inserted between either alto and soprano, tenor and alto, or both pairs of voices (Aldwell & Schachter, 2003). These chords are usually constructed by playing all the notes indicated by the figures; these are called complete chords. Alternatively, they can be played leaving some notes out; these are called incomplete chords. Fig. 3 indicates how the same figured bass can be played differently in four-part texture, which consists of four different voices: soprano, alto, tenor, and bass. A significant number of difficulties are involved in these decisions; e.g., careless spacing of chords leads to a large movement of each voice, or an awkward motion in the long musical context; in the more elaborated realisation, chords have to be allocated so that they generate motivic development or counterpoint. These difficulties can only be resolved by players’ experience and sense of beauty. As understood from the existence of a considerable number of textbooks stating the rules of continuo playing, a rule-based system can imitate such a player’s skills to some extent. However, a purely rule-based system can not imitate human flexibility to violate such rules to prioritise other aspects such as their own musical tastes, the specific style of the piece, or the limitation of the performer’s skills.
1 The distance between two pitches. Intervals are traditionally labelled according to the number of steps they embrace in a diatonic scale, counted inclusively: thus from C up to D or down to B is a 2nd, another step up to E or down to A makes a 3rd, etc. (Mark Lindley et al., 2010). 2 The arrangement of the notes of a chord, according to the requirements of the individual voices. If the upper voices are close together, the spacing is described as close position, or close harmony; if not, the arrangement is called open position, or open harmony (Rushton, 2010).
442
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8
Fig. 2. An example of figures and their implications: bar one and two indicate that of C major and bar three and four indicate that of C minor.
Fig. 3. An example of chord spacing: (a) and (b) are close position chords; (c) and (d) are open position chords; (a) and (c) are complete chords; and (b) and (d) are incomplete chords.
The main objective of this paper is to mimic these human skills of basso continuo playing by integrating rule-based methods with beam search dynamic programming. Possible chords are generated by a certain number of rules from a given figured bass, which is followed by beam search dynamic programming to search for the optimal chord sequence. Dynamic programming enables basso continuo realisation without stating all the detailed rules concerning chord allocation and voice leading, and the use of the beam search makes it possible to consider long musical contexts with a small amount of memory. Moreover, many different results can be generated thorough this process, thereby realising the most important element of basso continuo playing, which is that it varies from performer to performer. 2.2. Overview of the proposed method To simplify the process, the following basic assumptions are made in the proposed method: (1) Bass parts are figured completely and the key is clear. Generally, bass parts tend to be figured incompletely from a theoretical point of view. In this paper, it is assumed that figures are complete enough to specify the basic types of chords, to focus on how to construct chords from complete figures and how to allocate them. Automatic figure analysis from incompletely figured bass is reported in Knopke and Wead (2007); however, there remain unsolved problems with this type of analysis and so it can not be widely used yet. (2) Basso continuo is realised in the four-part vocal writing style. Generally, basso continuo tends to be instrumental with a flexible number of voices. However, four-part vocal writing tends to be the first material for learners of basso continuo playing. The main reason is that it has applicability to music of greater complexity even though it is simple. Much complex instrumental music is based on a framework of four-part vocal writing. Therefore, the four-part style is assumed in the proposed method. (3) Only harmonic notes are produced unless the figure indicates non-harmonic notes.3 The final goal of basso continuo playing is an art of accompaniment. However, only when the player has mastered all the elementary technical and practical aspects of basso continuo playing, can purely musical aspects such as rhythm, articulation, and supporting
3 Non-harmonic notes are in a sense melodic ornaments, and many of the names used to describe them have been borrowed from the terminology for ornamentation (e.g. appoggiatura, broderie, vorschlag) (Drabkin, 2010).
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
443
the soloist be considered (Christensen, 2002). Therefore, such musical embellishments are excluded. Moreover, the solo part is not considered. In fact, a significant number of historical sources stress that the position of chords should be treated not in relation to the solo part but to the bass. The most important thing is to find the best voice leading and the best connection of the harmonies (Christensen, 2002, p. 40). The proposed method consists of three modules: the figure interpretation module, the chord allocation module, and the module which finds the most appropriate chord sequence. In the first module, the input figures are interpreted on the basis of forty-six rules which have been formulated according to modern textbooks. In the second module, the figures augmented by the figure interpretation module are received, and the possible chords in which all voices are arranged in the proper range are stored. In the third module, the optimal sequence of chords is sought by the dynamic programming method. Finally, the complete accompaniment is produced in MIDI format (Huber, 2007) from which a musical score is generated (output). The detail of each module is demonstrated in the following sections. 2.3. Figure interpretation Three figures are necessary to realise basso continuo into a four-part chord. Even though figures are complete and the basic type of chords is specified, there still exist different ways to play them in four-part chords. There is a certain number of rules in this interpretation method, which are extracted into forty-six rules (Table 1) according to modern textbooks. In the figured bass notation used for the theory of harmony, the interpretation of one specific figure is unique (Aldwell & Schachter, 2003); however, in the Baroque era, there were a lot of interpretations, and performers had to judge which was the most appropriate. This proper judgement can be realised by using figure interpretation cost which is addressed in Section 2.5. 2.4. Chord allocation In this module, the figures interpreted by the figure interpretation module are received, and the possible chords in which all voices are set in proper range are stored. For example, if bass note ‘‘D’’ and three figures ‘‘3 5 8’’ are input, three pitch spellings ‘‘F A D’’ are generated. Next, each pitch spelling is assigned to each part. Finally, the MIDI number of each part is obtained by deciding the octave. Chords satisfying Eq. (1) are stored as possible chords.
bass 6 ten 6 alt 6 sop 6 MaxSop
ð1Þ
where sop, alt and ten denote the MIDI number of each part and MaxSop denotes the highest MIDI number sop can reach. This procedure makes it possible to discard any chords which include the crossover between any parts. Fig. 4 shows all the chord allocations of ‘‘3 5 8’’ above ‘‘D’’ note in the case of D major piece, generated by Eq. (1), with MaxSop = 81. How to choose an appropriate chord sequence is described in the next section.
Table 1 Figure interpretation rules (excerpts). Figure
Interpretation
None 6 6 6 6 7 24 25
3 3 3 3 3 3 2 2
5 6 3 6 4 5 4 5
8 6 6 8 6 7 6 5
Fig. 4. Possible chord allocations of ‘‘3 5 8’’ above ‘‘D’’; generated by Eq. (1) with MaxSop = 81.
444
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
Fig. 5. Two different allocations of the same chord: (a) realisation in close position; and (b) same chord in open position. There is higher likelihood of close position chord being simulated by lowering the chord location cost of (a) than that of (b).
Table 2 An example of factors for chord location cost. N: Negative rule, P: Positive rule. Factor
N/P
Each voice is not set in a proper range Leading note is doubled Close position Not prepared seventh Not prepared ninth
N N P N N
2.5. Finding the most appropriate chord sequence The optimal sequence of chords can be found by Dynamic Programming (DP) (Huang, Acero, & Hon, 2001). DP searches for the route with minimum cost, which is defined as two kinds of costs; the local cost and the transition cost. Let Xt be a set of all possible chords which can be located at time t, and let xt 2 Xt be a certain chord at time t. Ideally speaking, the optimal chord sequence should be sought for each internal cadence by using a sliding window with beam search. However, automatic cadence analysis is not practical at this moment. Therefore, optimal chord sequence of entire pieces is sought. 2.5.1. Local cost Local cost L(xt) indicates the unlikelihood of a certain chord xt 2 Xt at time t. L(xt) consists of two different costs: figure interpretation cost L1(xt) and chord location cost L2(xt). Let k denote an adjustment factor, then L(xt) is defined by Eq. (2).
Lðxt Þ ¼ L1 ðxt Þ þ kL2 ðxt Þ
ð2Þ
Figure interpretation cost L1(xt) represents the unlikelihood of a certain interpretation of figure. The general factor for the figure interpretation cost is the scale degree of a bass note. Some figures can be interpreted differently depending on composers and regions. For instance, the figure ‘‘6’’ is usually interpreted as the first inversion chord, and is interpreted as ‘‘3 3 6’’, ‘‘3 6 6’’ or ‘‘3 6 8’’. However, some theorists and composers used figure ‘‘6’’ as ‘‘3 4 6’’. This preference can be reflected by lowering the cost of the interpretation of ‘‘6’’ to ‘‘3 4 6’’ to make it less than the other interpretation. This results in the generation of different accompaniment with the harsh sound of ‘‘4’’. Chord location cost L2(xt) indicates the unlikelihood of a certain chord xt at time t. For instance, the cost of a close chord (Fig. 5a) is smaller than that of an open chord (Fig. 5b), since a close chord is more common than an open chord unless there is a specific reason for it. Table 2 lists some of the factors for chord location cost such as leading note4 doubling and unprepared5 seventh and ninth. The larger k becomes, the more complete chords become and the more usual spacing tends to be used with the general figure interpretation. On the contrary, the smaller k becomes, the more special figure interpretation is used, even though it causes poor chord spacing.
4 The seventh degree of the major, harmonic minor, or ascending melodic minor scale, so called because it lies a semitone below the tonic and therefore has a strong tendency to lead up to it (Kennedy & Bourne, 2010). 5 In part-writing, the softening of the dissonant effect of an accented non-harmonic note (i.e. an appoggiatura) by presenting it as a consonant note in the previous chord; the result is called a prepared appoggiatura. The dissonance is softened further if the preparation note is tied to the appoggiatura, creating a suspension (Kennedy & Bourne, 2010).
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
445
Fig. 6. Two different chord transition above the same figured bass: (a) realisation without successive motion; (b) realisation with successive octave between Soprano and Tenor. There is low likelihood of successive motion being simulated by lowering the transition cost of (a) than that of (b).
Table 3 An example of factors for transition cost. Factor
N/P
Hidden octave Hidden unison Successive fifth Successive octave Successive unison Contrary motion with other voices Motion in the same direction as the previous direction of the soprano part Leap larger than octave Seventh interval motion Augmented interval motion
N N N N N P P N N N
Table 4 A guideline for weighting of the factors of each cost; the importance of each factor is shown on a scale of one to five. Factor
Importance (1–5)
Melodic motion of soprano part Smooth connection of each part Successive octave Successive unison Hidden octave Hidden fifth Leading unresolved Seventh unresolved Missing third Augmented skip
5 4 5 5 3 3 2 2 1 1
2.5.2. Transition cost Transition cost T(xt, xt+1) indicates the unlikelihood of the connection between two chords when there is a certain chord xt at time t and chord xt+1 at time t + 1. For instance, Fig. 6b has a successive6 octave between soprano and tenor, therefore the transition cost of Fig. 6b is larger than that of Fig. 6a. Table 3 lists some of the factors for transition cost. 2.5.3. Determining cost It would be ideal to decide the cost automatically by machine learning methods. However, it is difficult at this point to obtain a large amount of training data of realised basso continuo. Moreover, because basso continuo realisation varies from one performer to another, it would not be appropriate to calculate the general weight. Therefore, the cost was decided by humans by trial and error. A major guideline for this weighting is shown in Table 4. This weighting varies from performer to performer, and these guidelines are not the only solution. 2.5.4. Searching by DP In standard DP, accumulated cost is memorised at each state and the route is traced back to obtain the best state sequence. If this method is used in basso continuo realisation, the amount of memory needed becomes O(en) as the length of music n increases. Let us denote by jXtj the number of chords that can be located at time t, and by a the upper bound of capacity required to memorise one chord, then the amount of memory needed M(n) can be expressed as Eq. (3). 6 In part-writing, the simultaneous duplication of the melodic line of one part by another at the interval of a perfect 5th or an octave, or any equivalent compound interval (Drabkin, 2010).
446
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
MðnÞ ¼ Pnt¼1 a jX t j
ð3Þ n
Let us put b = max{jX1j, jX2j, . . . , jXnj}, then the amount of memory needed will be denoted as Eq. (4), which exceeds O(e ).
MðnÞ P ðabÞn
ð4Þ
This is a crucial problem because the length of music is variable. As demonstrated in Tillmann and Ney (2003), the amount of memory used can be reduced by using the beam search algorithm, which only keeps states with small cost only. However, the naive beam search might neglect some interpretation of figures without considering continuing chords, which may lead to inappropriate chord connections. This problem can be handled by searching backwards, which enables the figure interpretation in the light of the following chords. Moreover, the amount of branching can be decreased significantly by searching backwards since tonal music tends to converge to cadence. Let l denote an adjustmental factor, then ‘‘harmony cost’’ P(xt, xt+1) can be defined by Eq. (5).
Pðxt ; xtþ1 Þ :¼
Lðxt Þ þ l Tðxt ; xtþ1 Þ
if 1 6 t 6 n 2
Lðxt Þ þ Lðxtþ1 Þ þ Tðxt ; xtþ1 Þ if t ¼ n 1
ð5Þ
The larger l becomes the more the chords tend to be incomplete and the more unusual the spacing becomes. On the contrary, the smaller l becomes, the more the chords tend to become complete with usual spacing. Automatic basso continuo playing is realised by searching for the optimal chord sequence, which minimises the accumulated harmony cost calculated on the basis of Eq. (5). Let the input figured bass notes be b1, b2, . . ., bt, . . ., bn, all the possible chords above bt be chord(bt), an array for chords be V, the number of elements of V be jVj, an array for chord sequences be W, and all the possible chord sequences obtained by connecting each of all the elements of W and V be VW, then the searching algorithm is explained as shown in Algorithm 1. Algorithm 1. Search for the optimal chord sequence. Input: A set of figured bass notes b1, b2, . . ., bt, . . ., bn Output: An optimal chord sequence 1: initialise W and V 2: for t = n to 1 do 3: V chord(bt) 4: if t = n then 5: wj Vj 6: W V 7: else 8: W VW 9: end if 10: keep the best w elements of W on the basis of the accumulated harmony cost calculated by Eq. (5), and remove the other elements 11: end for 12: return the best chord sequence from W
3. Simulation This section looks at a simulation of the proposed method. The format of the input data is shown in Fig. 7a. In the header, key, meter and whether there is an anacrusis7 or not are read. In this example, the system reads C major, 4/4 and no anacrusis. In the body part of the input data, pitch, duration, and figures of the bass notes are read line-by-line, and stored in chronological order. In this example, E, a quarter-note and ‘‘6’’ are stored on the first line. On the second line, D, a quarter-note and ‘‘6’’ are stored. On the third line, C, a quarter-note and no figures are stored. On the fourth line, B (‘‘,’’ indicates that this note is an octave lower than the default height), a quarter-note and ‘‘6’’ are stored. On the fifth line, C, a half-note and no figures are stored. Each figure is interpreted as three figures. In this example, no figures are interpreted as ‘‘3 5 8’’, and ‘‘6’’ is interpreted as ‘‘3 3 6’’, ‘‘3 6 6’’, ‘‘3 6 8’’, and ‘‘3 4 6’’. These figures are processed by the chord allocation module to generate all the possible chords in which all voices are set in proper range. On the first line, it receives the bass note E and figures ‘‘3 3 6’’ (one of four interpretations). G is obtained as the third above the bass note, C is obtained as the sixth above the bass note, and so forth. This procedure decides each pitch spelling which composes one chord, followed by the calculation of possible allocation. From the calculated chord candidates, the beam search seeks the optimal chord sequence, which minimises the accumulated harmony cost. In this example, the beam width is equal to the number of possible chords (39 in number) above the last
7
Anacrusis refers to an anticipatory note occurring before the first bar-line of a piece.
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
(a)
447
Header Key : C Meter : 4/4 Anacrusis or not : normal Body No.
Note
Duration
1
Figures
E
4
6
2
D
4
6
3
C
4
4
B,
4
6
5 . . .
C . . .
2 . . .
. . .
(b)
Fig. 7. An example of the input file and input score.
Fig. 8. Hypothesised score by the proposed method from the input file shown in Fig. 7.
bass note. Finally the output for the best chord sequence is in Lilypond format,8 thus creating a MIDI file and a score file. An example of the output file is shown in Fig. 8, where the figure ‘‘6’’ is interpreted differently by its location; ‘‘6’’ above the first bass note is interpreted as ‘‘3 6 8’’; on the other hand ‘‘6’’ above the second bass note is interpreted as ‘‘3 4 6’’. 4. Experiment This section addresses three experiments to evaluate the effectiveness of the proposed method. In the first experiment, the amount of memory used was verified. In the second experiment, four soloists played the melody parts of four different pieces with three different accompaniments, including the output of the proposed system. In the third experiment, a professional continuo player evaluated the scores of the accompaniments hypothesised by the proposed method. All the experiments were carried out on Intel Core Duo 2.16 GHz CPU with 2GB memory.
8 Lilypond is an automated engraving system. It formats music beautifully and automatically, and has a friendly syntax for its input files. It is free software and available from http://lilypond.org/.
448
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
4.1. Experiment 1 In the first experiment, the correlation between the amount of memory used and the number of input bass notes was investigated. The input file was excerpted from the D. Purcell Sonata in F major. In Fig. 9, the X-axis indicates the number of input bass notes, and the Y-axis indicates the amount of memory used (Fig. 9a) and the execution time (Fig. 9b). In both Figs. 9a and b, the effect of the beam search algorithm appears clearly. For example, the amount of memory used without the beam search, when n = 4, is 4.117 megabytes; on the other hand, that with the beam search is 0.0011 megabytes. 4.2. Experiment 2 In the second experiment, four alto recorder soloists played four different pieces with three different accompaniments. They have been familiar with music from infancy, and have a good command of alto recorders. Table 5 shows the pieces used in the experiment, which are different from the pieces by which the costs were determined, and incomplete figures and key change information were supplemented in advance. The soloists were asked to evaluate each accompaniment on a scale of one to five. Three accompaniments of each piece were prepared as MIDI files, where all files have the same volume and the same tone colour.
(a)
Used memory (Megabytes)
Bass-Part only (BPO) Four-Part by a professional musician (FPPM) Four-Part by our system (FPOS)
withBeam withoutBeam
10 1 0.1 0.001 0.0001
1
2
3
4
5
Number of inputs
(b) Execution time (msec)
10 10 10 10 10 10
5
withBeam withoutBeam 4
3
2
1
0
1
2
3
4
5
Number of inputs Fig. 9. A comparison of the amount of memory used and execution time.
Table 5 Details of the pieces used for experiment. Title
Key
The number of bars
D. Purcell Sonata Telemann Sonata Marcello Sonata Barsanti Sonata
F Dur d moll F Dur g moll
52 17 38 38
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451 5
449
BPO FPPM
4.5 FPOS
Score
4 3.5 3 2.5 2 1.5 1 Purcell Telemann Marcello Barsanti
Piece Fig. 10. Evaluated value for each piece by the soloists.
Fig. 11. Two different realisations of the same bass excerpted from Purcell’s sonata (a) are realised by the proposed method and (b) by a professional. The arpeggiation employed in (b) was evaluated positively by some subjects.
Table 6 Negative factors pointed out by professionals. No.
Factor
(1) (2)
Non-harmonic tone is harmonised Soprano part is so high that it obscures melody part
Fig. 10 illustrates that the evaluation of four-part by our system (FPOS) is higher than only basso (BPO), and almost the same as that by the professional musician (FPPM). With regard to the pieces of Purcell, because the harmonic changes are few, chords were played with arpeggiation in FPPM shown in Fig. 11b. Some subjects expressed their opinions regarding the FPPM of Purcell that chords played with arpeggiation sounded harmonious. On the other hand, chords were merely played with the same rhythm as that of bass notes in FPOS as shown in Fig. 11a, and there was an opinion about the FPOS of Purcell that the chords were felt to be harsh. This indicates that sustaining one chord with the same harmony without arpeggiation can become harsh and tedious. 4.3. Experiment 3 In the third experiment, a professional of basso continuo evaluated realised scores to find out how to improve the proposed method. No. (1) shown in Table 6 is caused by harmonising non-harmonic tones. Rapid bass movement could sound noisy when all the notes are harmonised as shown in Fig. 12. In order to solve this problem, it is essential that non-harmonic tone be properly distinguished from harmonic tone. No. (2) implies that the soprano part could become so high that it obscures the melody part. Both of these problems stemmed from the assumptions shown in Section 2.2, thereby implying that realisation without each assumption should be investigated.
450
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
Fig. 12. Two different realisations of the same bass excerpted from Marcello’s sonata (a) are realised by the proposed method and (b) by a professional. The harmonisation of all the 16th notes in (a) was pointed out as a negative factor (1) shown in Table 6.
5. Conclusion and future work In this paper, a method for automatic basso continuo playing was proposed. In order to find a musically appropriate sequence of chords, ‘‘harmony cost’’ was defined as the sum of two different costs, and the beam search algorithm was adopted to reduce the amount of memory used. Experimental results showed the effectiveness of our approach and its similarity to human realisation. In order to achieve the complete development of an automatic basso continuo playing method, time and effort will be devoted to further research. To decide the cost automatically by machine learning, a database of realised basso continuo has to be developed. The implementation of a more historical manner of basso continuo playing is an interesting subject, which may require cooperation with musicologists. As mentioned in Section 2.2, this paper deals with the foundation of basso continuo playing, and basso continuo playing without each and every assumption made in this paper requires further research: firstly, basso continuo playing in a more instrumental style with a variable number of voices should be investigated; secondly, figure analysis from unfigured or incompletely figured bass seems to be a worthwhile subject to investigate; finally, a more elaborate realisation with embellishments or interaction with the solo part will involve many problems, thereby stimulating further research. We intend to work on these tasks on the basis of the method proposed in this paper.
Acknowledgement This research was conducted under the auspices of the Exploratory IT Human Resources Project 2006 of the InformationTechnology Promotion Agency, Japan, and the financial support by the Nakajima Foundation. References Aldwell, E., & Schachter, C. (2003). Harmony & voice leading. New York: Schirmer Books. Christensen, J. B. (2002). 18th century continuo playing: A historical guide to the basics. Kassel: Barenreiter. Cope, D. (2004). Computer models of musical creativity. Cambridge: MIT Press. Drabkin, W. (2010). Consecutive fifths, consecutive octaves. The new grove dictionary of music and musicians, 2nd ed.
M. Niitsuma et al. / Information Processing and Management 47 (2011) 440–451
451
Knopke, I. & Wead, A. (2007). Computer-based implementation of basso continuo rules for figured bass realizations. In Proceedings of international computer music conference (Vol. 2, pp. 188–191). Mark Lindley, M. C. & Greated, C. (2010). Interval. The new grove dictionary of music and musicians, 2nd ed.