- Email: [email protected]
Man-machine interaction in creative applications
Int. J. Man-Machine Studies (1971) 3, 1-11
Man-Machine Interaction in Creative Applications J. K.
PULFER
National Research Council of Canada, Ottawa, Canada (Received 27 April 1970) Some aspects of creative activity are reviewed, and means by which the processes involved may be augmented with the help of a digital computer are discussed. A computer aided facility for writing, arranging and producing simple musical compositions is described to illustrate these concepts.
Introduction One possible definition of creative activity is "The process of translating a concept or idea (perhaps vague or poorly formed) into some form of physical embodiment which can be appreciated and shared by others." The physical embodiment might vary all the way from a written or spoken description, to something as concrete as a piece of sculpture or the performance of a new musical composition. Sometimes the creative process is thought of as a flash of inspiration in which the artist or composer produces his work in final unchangeable form. In many cases, however, the process involves much trial and error, with the creator modifying his mental image of his creation as it takes shape. He interacts with the creative medium (and, in fact, with his total environment including his past experiences) in a conversational way, learning the "language" in which he can express himself as he goes' along. This description applies to creativity in the pure and applied sciences just as well as it does to painting, writing, sculpture and the composition and performance of musical works. In this paper the advantages and problems inherent in the use of a digital computer as an aid in creative activity are discussed, and a computer facility for the composition, arrangement and live production of music is taken as an illustration.
1
2
J. K. PULFER
Creation with a Computer In many cases, the production of a work of art may be considered in two distinct stages: the first stage is the translation of the idea into a description or set of instructions; the second stage is the translation of these instructions into the final embodiment. For example, an architect translates his ideas for a building into blueprints, and then the builder translates these into steel, glass and concrete. A composer of music translates his basic idea into a composition plan and writes out the final form as sheet music, and then the musicians translate these into sounds during a performance. When a computer is used to aid in creative activity its functions are often those just described. Let us take music as the example. The computer provides the equivalent of (1) pen, (2) paper, (3) the musician and his instruments. A simple diagram of the process is shown in Fig. 1. The dashed line
Data base or Logical entity or
or
Information Embodiment structure or of idea
FIG. 1. Creative activity is often a conversational process between the artist and the creative medium. In this illustration, the creative medium (below the dividing line) is a digital computer.
MAN-MACHINE INTERACTION IN CREATIVE APPLICATIONS
3
separates man (above) from computer (below). The composer, by means of the input interface (pen), constructs and modifies an embodiment of his idea (the composition) in the computer memory (the paper). The computer then translates this entity just produced into a form (sound) that the user can appreciate and criticize by means of the output interface (the musician and his instrument). After hearing his creation, the composer may then modify it and listen to it again, the process continuing around and around the loop until he is satisfied or gives up in disgust. If we change the appropriate key words, the preceding paragraph is an equally good description of architectural or engineering design, the formulation of new mathematical models describing the physical world, or writing and producing a new play. As with any new medium, the computer also offers the artist a stimulus to create in the form of the unexpected. If the response he receives to a given input is new and different, it may trigger a whole new approach. If a computer is indeed to be able to assist in these activities and to augment creative output of individuals, and if it is to do this in a way which is economically practical, then the design of the three rectangular blocks in figure I is vital. The input interface must provide the user with a means for constructing and manipulating his creation by means of a set of commands which are easy to learn because they are familiar, that is, if he is constructing a drawing, he should have items at his disposal which behave like pencil, paper, brushes, paint, erasers, etc. The output interface should show him the item while he is drawing it, so that he can have immediate feedback to "guide his hand" and allow correction of errors during the drawing. The information structure or logical entity being produced should have a flexible format that allows for storing, recalling, combining, and copying, parts which go together to make up the final whole. Constraints Up to this point, it has been assumed that the best possible way to design the computer would be to make it transparent. That is to make it look to the user as though it were not even present, so that whatever idea occurred to him, it could be rapidly formed into a final creation. This is not necessarily true. An architect may like to have the constraints of a Tee-square and set-square to help him draw straight lines and square corners. A musician may want to stay within the constraints of a chromatic scale, standard pitch, and a
4
J. K. PULFER
restricted set of time durations and speeds so that melody, harmony and counterpoint can be combined in a meaningful way. Figure 2 illustrates the creative process with constraints applied during both the constructive and interpretive phases.
Information structure
FIG. 2. For continuity, it is often desirable that the medium apply some constraints during creative activity.
As with the other aspects of the computer system, the constraints must be very carefully thought out. While they can relieve the user of a large amount of unnecessary and boring detail, they can also restrict his freedom to the point where the resulting productions are dull and mechanical. The dotted line in Fig. 2 is intended to indicate another basic functional boundary in creative activity. The boxes above the line indicate hardware and programming elements which are in a sense independent of the type of structure being built up. These items do not need to be changed when a new type of activity is undertaken. On the other hand, the constructing, manipulating and interpreting packages as well as the constraints are specific to a particular field; let us say, music. Most computer languages now available
MAN-MACIDNE INTERACTION IN CREATIVE APPLICATIONS
5
for constructing these blocks are unsatisfactory either because they are mathematically oriented, or because they result in cumbersome and slow programs. As a result we are usually left with the situation where an artistprogrammer team is formed, the artist uses the system without having intimate control over the functions of the blocks he uses, and the programmer builds blocks without fully appreciating the needs of the artists. At this point, it is probably worth while to describe a computer music facility. A discussion of its good and bad points may then serve to clarify some of the concepts described above. The Music Package
Figure 3 is a block diagram illustrating the music facility. The input interface consists of two parts. One is a keyboard and positioning wheel which allows a melody to be entered in the form of notes, rests and other musical
Melody
FIG. 3. A facility for composing and arranging experimental musical inventions is one
example of computer participation in creative work.
6
J. K. PULFER
symbols. While this is being done, one of the output interpreters presents to the user a picture on a screen which looks much like a sheet of music on which he can see his music being written on a staff (Fig. 4). By means of the key-
•
11. FIG.
4. The music is displayed on a screen as symbols on a staff in a format closely resembling sheet music.
board the user can also enter control information into the melody. Commands which define attack and decay, key and logical boundaries of portions of the melody are available. Loudness and speed are also parameters which can be defined, and individual notes can be marked for amplitude stress. The constructing package also includes editing facilities which allow easy modification of melodies. Any number of notes or control signs may be inserted, deleted or over-written as the composer desires. The second inputoutput interface pair allows him to sketch on the screen an amplitude vs. time curve which will define the waveform and thus the timbre of the sounds to be created. ' A third interpreter takes the melody and waveforms just created and "plays" them through a loudspeaker in the form of live music. Constraints are active during both the constructive and interpretive phases. The computer only allows notes with pitches on the chromatic scale to be entered, and produces only the corresponding pitch sounds. Time durations of notes are limited to multiples of the duration of a "12 note" for a given choice of "tempo". An arrangement facility is also available which allows the composer to state (in effect) "play the first part twice with a slow decay and a mellow timbre, then play the second portion with a deep reed-like tone, followed by the first part again one octave higher and with a very sharp decay." The user may at any time name and store away melodies, timbres, and even arrangements that he likes, and recall and combine these later in other contexts. Figure 5 illustrates some of the functions associated with the package.
MAN-MACHINE INTERACTION IN CREATIVE APPLICATIONS
7
Auxiliary routines are available which allow rapid checking of the duration of individual parts of a melody, transposition and inversion-all done interactively at the display. A more detailed technical description of the hardware and software is given in the Appendix.
Input
Add delete select generate sublists
Scan display execute
Menus ar lists
FIG. 5. Many of the housekeeping aspects of creative activity-such as putting away, recalling, combining, and subdividing component parts-can also be conveniently handled by the computer.
At no time is it necessary for the user to learn how to program the computer, or in fact even to know how to operate it other than through making some choices from names presented to him on the screen. After being introduced to the music package and being shown its basic functions, he can proceed to learn the "language" by trial and error.
8
J. K. PULFER
The question which naturally occurs to the mUSICian is, "How can a facility like this be made available to users and how much is it going to cost?" The facility described above costs in excess of $150,000, althoughit was designed for man-computer communications studies and is probably more costly than a specific user's facility would be. Let us say that reductions brought about by decreasing costs of some part of the facility hardware and some simplification of the facility for a specific use would result in a cost of roughly $100,000 excluding housing and maintenance. This is still probably too expensive for a single individual. On the other hand, even the most creative user would not be able to use a facility like this full time. Sharing between five to ten artistic users and selling the remaining time for more routine applications would surely make the hardware costs tolerable. Ways of solving the organizational problems involved in sharing would be difficult, but the potential increase in the productivity of each individual is a strong motivation.
Discussion We are currently doing a study in which composers from various disciplines and with differing backgrounds are using the computer as a medium for composing and arranging music. The intent is not to develop a good music facility, but rather to find out more about the interaction problems in a creative environment. In the past year a considerable number of composers and musicians have experimented with the facility and of these, 12 made serious use of it. The latter users can be divided into three categories: (1) Commercially oriented users who are interested in producing music stored on analogue magnetic tape which can then be supplied to a prospective buyer in finished form. These people use the system as a musical instrument which can provide results which are in some way different from those produced by standard instruments. (2) Non-commercially oriented users who are interested in composing or producing a type of music which is too complex for them to visualize without external aids and is difficult or impossible for performers to generate using standard instruments. Examples of this type are people experimenting with music composed by computer. (3) Users whose primary goal is education. These people want to learn about the type of music a computer can produce and to build up
MAN-MACHINE INTERACTION IN CREATIVE APPLICATIONS
9
their knowledge of the interaction of rhythmic, dynamic and pitch patterns. Users of all three types have been encouraged because each has something to contribute to a better understanding of the use of computers in a creative environment. Some of the contributions which the users have made are related to the following questions: (1) What stumbling blocks (in their opinions) were present when they tried to communicate in musical terms with the computer? (2) What types of input and output media are most meaningful and useful to people who think in musical terms? (3) At what level of complexity did the composer find it useful to have external aids to help him create? (4) What aspects of instant playback of the music he wrote were useful and stimulating in the creative process? (5) To what extent was it valuable and to what extent was it a hindrance to retain the constraints of the classical musical language in communicating with a computer to compose music? The answers to these questions together with similar answers from users in other areas, such as art and film animation, have helped to formulate basic design philosophies for hardware and software. From my point of view, faults and weaknesses in the music package can be divided into two classes. There are outright limitations in the type of music and type of sounds which can be produced which are imposed by the speed of the computer and the approach taken to synthesizing the waveforms within the machine. As an example, at present the output is monophonic, and harmony cannot be produced "live" without sacrificing some other parameters such as tone quality. Such limitations although perhaps serious to some composers are of secondary interest here. They can presumably be reduced to acceptable levels by buying a more powerful computer, and higher quality hardware. On the other hand the composer may point out that he finds it intuitively difficult to think of timbre or tone quality as being specified by a waveform on a cathode ray tube screen, and that he would much rather communicate in a different sort of language. Such a fault would be a real hindrance to creative activity. One measure of the success of a facility for aiding creative activity, is that if it is used by a number of different people, the results should reflect the character and weaknesses of the individual users rather than the
10
J. K. PULFER
faults of the machine. As an example a common criticism of music that we have produced on computers in the past is that the timing is too regular-"it sounds too mechanical". In this case, the timing is completely under the control of the composer and it is his responsibility to inform the machine how he would like to position the notes in time. Surelyit would be wrong to insert a constraint in the program which says in effect-try to interpret the music as a sequence of phrases and adjust the timing to suit-this is just the sort of judgment that the artist would like to keep for himself so that he could put his feeling into the music. On the whole, user reaction to date has been enthusiastic and we hope that some of the ideas generated may be useful to other users of computers in creative fields. References NOLL, A. M. (1967). The digital computer as a creative medium. IEEE Spectrum 4 (10), 89-95.
NOLL, A. M. (1967). Computers and the visual arts. Design and Planning, No.2. New Yark, Hastings House Publications Inc. MATHEWS, M. V. (1967). Coupling the computer to arts and humanities. Proc. 23rd Ann. Meet. Am. Con! Academic Deans, Los Angeles, Calif. January. I would like to acknowledge the large amount of constructive criticism offered by my engineering colleagues as well as by artists and musicians who have seen the facility. The ideas contributed have been very useful in helping formulate the thoughts presented in this paper. APPENDIX Technical Description of the Music System For the benefit of those interested in more of the technical details of the music system, the following is a summary of some of the hardware and software aspects. The system uses an SEL 840A digital computer. Word size is 24 bits, and cycle time is 1·75 Jlsec. The computer has a core memory of 16 K words (48 K bytes) and a disc memory of approximately 1·2 million words. Other peripherals include a digital magnetic tape unit, high speed paper tape, typewriter terminals, a digital display with its own core memory and display processor, and an analogue interface. A small organ keyboard is also interfaced as a digital input device. The computer has a multilevel hardware priority interrupt system which is connected to, among other things, two hardware real time clocks, and a multi-track analogue tape recorder.
MAN-MACHINE INTERACTION IN CREATIVE APPLICATIONS
11
The display hardware includes a number of cathode ray tube displays, a light pen, a mouse and a pair of 10 bit shaft position encoders which can be adjusted by the operator. The software can be divided into two parts. One part is the system programs which are core resident, taking up about 4K of core memory. The system programs include the normal executive and input-output programs, but they also include programs which are useful in interactive creative work. As an example, a subroutine which displays on the cathode ray tube alist of choices, and returns to the calling program after the choice is made is extremely important in interactive control of a music system. A second system program provides easy access to the disc library in an interactive manner to users with no knowledge of computer functions. From the user point of view, the music library directory is a set of menus which appear in the cathode ray tube and which form a hierarchical tree structure with complete freedom of cross linking. The names on the menus might represent melodies, tables defining timbre, tables defining envelopes, or even programs for sound generation. (Of course they might equally well refer to pictures if the system is being used for art or film animation.) The remaining 12K of core is available to the music system programs. It is occupied by the various sound generating and data manipulating routines which are useful during composition. Each "note" of a melody is stored as one 24 bit computer word. The word is subdivided into fields which contain numbers defining pitch, duration, and control information such as loudness, tempo, decay rate, etc. Each sound generator simultaneously makes use of three independent time rates. Pitch is determined by counting the crystal clock controlling the computer cycle time. Tempo or the duration of a note is determined by counting an external crystal controlled real time clock which interrupts the pitch generating program. Phase and frequency of the clock are controlled by the computer. The attack and decay and any other dynamic functions are controlled by a second external clock which is manually adjustable. Sound generator programs are available for a number of different techniques of waveform generation with monophonic output, or for limited waveform selection and a polyphonic output. One of the sound generators can operate under the control of the organ keyboard. Melodies can be played and heard by a musician, and then the resultant "sheet music" can be inspected. Finally, a routine is available which prints out a composed melody in musical symbols on a graphical plotter providing a permanent paper record of the music.
Man-Machine Interaction in Creative Applications J. K.
PULFER
National Research Council of Canada, Ottawa, Canada (Received 27 April 1970) Some aspects of creative activity are reviewed, and means by which the processes involved may be augmented with the help of a digital computer are discussed. A computer aided facility for writing, arranging and producing simple musical compositions is described to illustrate these concepts.
Introduction One possible definition of creative activity is "The process of translating a concept or idea (perhaps vague or poorly formed) into some form of physical embodiment which can be appreciated and shared by others." The physical embodiment might vary all the way from a written or spoken description, to something as concrete as a piece of sculpture or the performance of a new musical composition. Sometimes the creative process is thought of as a flash of inspiration in which the artist or composer produces his work in final unchangeable form. In many cases, however, the process involves much trial and error, with the creator modifying his mental image of his creation as it takes shape. He interacts with the creative medium (and, in fact, with his total environment including his past experiences) in a conversational way, learning the "language" in which he can express himself as he goes' along. This description applies to creativity in the pure and applied sciences just as well as it does to painting, writing, sculpture and the composition and performance of musical works. In this paper the advantages and problems inherent in the use of a digital computer as an aid in creative activity are discussed, and a computer facility for the composition, arrangement and live production of music is taken as an illustration.
1
2
J. K. PULFER
Creation with a Computer In many cases, the production of a work of art may be considered in two distinct stages: the first stage is the translation of the idea into a description or set of instructions; the second stage is the translation of these instructions into the final embodiment. For example, an architect translates his ideas for a building into blueprints, and then the builder translates these into steel, glass and concrete. A composer of music translates his basic idea into a composition plan and writes out the final form as sheet music, and then the musicians translate these into sounds during a performance. When a computer is used to aid in creative activity its functions are often those just described. Let us take music as the example. The computer provides the equivalent of (1) pen, (2) paper, (3) the musician and his instruments. A simple diagram of the process is shown in Fig. 1. The dashed line
Data base or Logical entity or
or
Information Embodiment structure or of idea
FIG. 1. Creative activity is often a conversational process between the artist and the creative medium. In this illustration, the creative medium (below the dividing line) is a digital computer.
MAN-MACHINE INTERACTION IN CREATIVE APPLICATIONS
3
separates man (above) from computer (below). The composer, by means of the input interface (pen), constructs and modifies an embodiment of his idea (the composition) in the computer memory (the paper). The computer then translates this entity just produced into a form (sound) that the user can appreciate and criticize by means of the output interface (the musician and his instrument). After hearing his creation, the composer may then modify it and listen to it again, the process continuing around and around the loop until he is satisfied or gives up in disgust. If we change the appropriate key words, the preceding paragraph is an equally good description of architectural or engineering design, the formulation of new mathematical models describing the physical world, or writing and producing a new play. As with any new medium, the computer also offers the artist a stimulus to create in the form of the unexpected. If the response he receives to a given input is new and different, it may trigger a whole new approach. If a computer is indeed to be able to assist in these activities and to augment creative output of individuals, and if it is to do this in a way which is economically practical, then the design of the three rectangular blocks in figure I is vital. The input interface must provide the user with a means for constructing and manipulating his creation by means of a set of commands which are easy to learn because they are familiar, that is, if he is constructing a drawing, he should have items at his disposal which behave like pencil, paper, brushes, paint, erasers, etc. The output interface should show him the item while he is drawing it, so that he can have immediate feedback to "guide his hand" and allow correction of errors during the drawing. The information structure or logical entity being produced should have a flexible format that allows for storing, recalling, combining, and copying, parts which go together to make up the final whole. Constraints Up to this point, it has been assumed that the best possible way to design the computer would be to make it transparent. That is to make it look to the user as though it were not even present, so that whatever idea occurred to him, it could be rapidly formed into a final creation. This is not necessarily true. An architect may like to have the constraints of a Tee-square and set-square to help him draw straight lines and square corners. A musician may want to stay within the constraints of a chromatic scale, standard pitch, and a
4
J. K. PULFER
restricted set of time durations and speeds so that melody, harmony and counterpoint can be combined in a meaningful way. Figure 2 illustrates the creative process with constraints applied during both the constructive and interpretive phases.
Information structure
FIG. 2. For continuity, it is often desirable that the medium apply some constraints during creative activity.
As with the other aspects of the computer system, the constraints must be very carefully thought out. While they can relieve the user of a large amount of unnecessary and boring detail, they can also restrict his freedom to the point where the resulting productions are dull and mechanical. The dotted line in Fig. 2 is intended to indicate another basic functional boundary in creative activity. The boxes above the line indicate hardware and programming elements which are in a sense independent of the type of structure being built up. These items do not need to be changed when a new type of activity is undertaken. On the other hand, the constructing, manipulating and interpreting packages as well as the constraints are specific to a particular field; let us say, music. Most computer languages now available
MAN-MACIDNE INTERACTION IN CREATIVE APPLICATIONS
5
for constructing these blocks are unsatisfactory either because they are mathematically oriented, or because they result in cumbersome and slow programs. As a result we are usually left with the situation where an artistprogrammer team is formed, the artist uses the system without having intimate control over the functions of the blocks he uses, and the programmer builds blocks without fully appreciating the needs of the artists. At this point, it is probably worth while to describe a computer music facility. A discussion of its good and bad points may then serve to clarify some of the concepts described above. The Music Package
Figure 3 is a block diagram illustrating the music facility. The input interface consists of two parts. One is a keyboard and positioning wheel which allows a melody to be entered in the form of notes, rests and other musical
Melody
FIG. 3. A facility for composing and arranging experimental musical inventions is one
example of computer participation in creative work.
6
J. K. PULFER
symbols. While this is being done, one of the output interpreters presents to the user a picture on a screen which looks much like a sheet of music on which he can see his music being written on a staff (Fig. 4). By means of the key-
•
11. FIG.
4. The music is displayed on a screen as symbols on a staff in a format closely resembling sheet music.
board the user can also enter control information into the melody. Commands which define attack and decay, key and logical boundaries of portions of the melody are available. Loudness and speed are also parameters which can be defined, and individual notes can be marked for amplitude stress. The constructing package also includes editing facilities which allow easy modification of melodies. Any number of notes or control signs may be inserted, deleted or over-written as the composer desires. The second inputoutput interface pair allows him to sketch on the screen an amplitude vs. time curve which will define the waveform and thus the timbre of the sounds to be created. ' A third interpreter takes the melody and waveforms just created and "plays" them through a loudspeaker in the form of live music. Constraints are active during both the constructive and interpretive phases. The computer only allows notes with pitches on the chromatic scale to be entered, and produces only the corresponding pitch sounds. Time durations of notes are limited to multiples of the duration of a "12 note" for a given choice of "tempo". An arrangement facility is also available which allows the composer to state (in effect) "play the first part twice with a slow decay and a mellow timbre, then play the second portion with a deep reed-like tone, followed by the first part again one octave higher and with a very sharp decay." The user may at any time name and store away melodies, timbres, and even arrangements that he likes, and recall and combine these later in other contexts. Figure 5 illustrates some of the functions associated with the package.
MAN-MACHINE INTERACTION IN CREATIVE APPLICATIONS
7
Auxiliary routines are available which allow rapid checking of the duration of individual parts of a melody, transposition and inversion-all done interactively at the display. A more detailed technical description of the hardware and software is given in the Appendix.
Input
Add delete select generate sublists
Scan display execute
Menus ar lists
FIG. 5. Many of the housekeeping aspects of creative activity-such as putting away, recalling, combining, and subdividing component parts-can also be conveniently handled by the computer.
At no time is it necessary for the user to learn how to program the computer, or in fact even to know how to operate it other than through making some choices from names presented to him on the screen. After being introduced to the music package and being shown its basic functions, he can proceed to learn the "language" by trial and error.
8
J. K. PULFER
The question which naturally occurs to the mUSICian is, "How can a facility like this be made available to users and how much is it going to cost?" The facility described above costs in excess of $150,000, althoughit was designed for man-computer communications studies and is probably more costly than a specific user's facility would be. Let us say that reductions brought about by decreasing costs of some part of the facility hardware and some simplification of the facility for a specific use would result in a cost of roughly $100,000 excluding housing and maintenance. This is still probably too expensive for a single individual. On the other hand, even the most creative user would not be able to use a facility like this full time. Sharing between five to ten artistic users and selling the remaining time for more routine applications would surely make the hardware costs tolerable. Ways of solving the organizational problems involved in sharing would be difficult, but the potential increase in the productivity of each individual is a strong motivation.
Discussion We are currently doing a study in which composers from various disciplines and with differing backgrounds are using the computer as a medium for composing and arranging music. The intent is not to develop a good music facility, but rather to find out more about the interaction problems in a creative environment. In the past year a considerable number of composers and musicians have experimented with the facility and of these, 12 made serious use of it. The latter users can be divided into three categories: (1) Commercially oriented users who are interested in producing music stored on analogue magnetic tape which can then be supplied to a prospective buyer in finished form. These people use the system as a musical instrument which can provide results which are in some way different from those produced by standard instruments. (2) Non-commercially oriented users who are interested in composing or producing a type of music which is too complex for them to visualize without external aids and is difficult or impossible for performers to generate using standard instruments. Examples of this type are people experimenting with music composed by computer. (3) Users whose primary goal is education. These people want to learn about the type of music a computer can produce and to build up
MAN-MACHINE INTERACTION IN CREATIVE APPLICATIONS
9
their knowledge of the interaction of rhythmic, dynamic and pitch patterns. Users of all three types have been encouraged because each has something to contribute to a better understanding of the use of computers in a creative environment. Some of the contributions which the users have made are related to the following questions: (1) What stumbling blocks (in their opinions) were present when they tried to communicate in musical terms with the computer? (2) What types of input and output media are most meaningful and useful to people who think in musical terms? (3) At what level of complexity did the composer find it useful to have external aids to help him create? (4) What aspects of instant playback of the music he wrote were useful and stimulating in the creative process? (5) To what extent was it valuable and to what extent was it a hindrance to retain the constraints of the classical musical language in communicating with a computer to compose music? The answers to these questions together with similar answers from users in other areas, such as art and film animation, have helped to formulate basic design philosophies for hardware and software. From my point of view, faults and weaknesses in the music package can be divided into two classes. There are outright limitations in the type of music and type of sounds which can be produced which are imposed by the speed of the computer and the approach taken to synthesizing the waveforms within the machine. As an example, at present the output is monophonic, and harmony cannot be produced "live" without sacrificing some other parameters such as tone quality. Such limitations although perhaps serious to some composers are of secondary interest here. They can presumably be reduced to acceptable levels by buying a more powerful computer, and higher quality hardware. On the other hand the composer may point out that he finds it intuitively difficult to think of timbre or tone quality as being specified by a waveform on a cathode ray tube screen, and that he would much rather communicate in a different sort of language. Such a fault would be a real hindrance to creative activity. One measure of the success of a facility for aiding creative activity, is that if it is used by a number of different people, the results should reflect the character and weaknesses of the individual users rather than the
10
J. K. PULFER
faults of the machine. As an example a common criticism of music that we have produced on computers in the past is that the timing is too regular-"it sounds too mechanical". In this case, the timing is completely under the control of the composer and it is his responsibility to inform the machine how he would like to position the notes in time. Surelyit would be wrong to insert a constraint in the program which says in effect-try to interpret the music as a sequence of phrases and adjust the timing to suit-this is just the sort of judgment that the artist would like to keep for himself so that he could put his feeling into the music. On the whole, user reaction to date has been enthusiastic and we hope that some of the ideas generated may be useful to other users of computers in creative fields. References NOLL, A. M. (1967). The digital computer as a creative medium. IEEE Spectrum 4 (10), 89-95.
NOLL, A. M. (1967). Computers and the visual arts. Design and Planning, No.2. New Yark, Hastings House Publications Inc. MATHEWS, M. V. (1967). Coupling the computer to arts and humanities. Proc. 23rd Ann. Meet. Am. Con! Academic Deans, Los Angeles, Calif. January. I would like to acknowledge the large amount of constructive criticism offered by my engineering colleagues as well as by artists and musicians who have seen the facility. The ideas contributed have been very useful in helping formulate the thoughts presented in this paper. APPENDIX Technical Description of the Music System For the benefit of those interested in more of the technical details of the music system, the following is a summary of some of the hardware and software aspects. The system uses an SEL 840A digital computer. Word size is 24 bits, and cycle time is 1·75 Jlsec. The computer has a core memory of 16 K words (48 K bytes) and a disc memory of approximately 1·2 million words. Other peripherals include a digital magnetic tape unit, high speed paper tape, typewriter terminals, a digital display with its own core memory and display processor, and an analogue interface. A small organ keyboard is also interfaced as a digital input device. The computer has a multilevel hardware priority interrupt system which is connected to, among other things, two hardware real time clocks, and a multi-track analogue tape recorder.
MAN-MACHINE INTERACTION IN CREATIVE APPLICATIONS
11
The display hardware includes a number of cathode ray tube displays, a light pen, a mouse and a pair of 10 bit shaft position encoders which can be adjusted by the operator. The software can be divided into two parts. One part is the system programs which are core resident, taking up about 4K of core memory. The system programs include the normal executive and input-output programs, but they also include programs which are useful in interactive creative work. As an example, a subroutine which displays on the cathode ray tube alist of choices, and returns to the calling program after the choice is made is extremely important in interactive control of a music system. A second system program provides easy access to the disc library in an interactive manner to users with no knowledge of computer functions. From the user point of view, the music library directory is a set of menus which appear in the cathode ray tube and which form a hierarchical tree structure with complete freedom of cross linking. The names on the menus might represent melodies, tables defining timbre, tables defining envelopes, or even programs for sound generation. (Of course they might equally well refer to pictures if the system is being used for art or film animation.) The remaining 12K of core is available to the music system programs. It is occupied by the various sound generating and data manipulating routines which are useful during composition. Each "note" of a melody is stored as one 24 bit computer word. The word is subdivided into fields which contain numbers defining pitch, duration, and control information such as loudness, tempo, decay rate, etc. Each sound generator simultaneously makes use of three independent time rates. Pitch is determined by counting the crystal clock controlling the computer cycle time. Tempo or the duration of a note is determined by counting an external crystal controlled real time clock which interrupts the pitch generating program. Phase and frequency of the clock are controlled by the computer. The attack and decay and any other dynamic functions are controlled by a second external clock which is manually adjustable. Sound generator programs are available for a number of different techniques of waveform generation with monophonic output, or for limited waveform selection and a polyphonic output. One of the sound generators can operate under the control of the organ keyboard. Melodies can be played and heard by a musician, and then the resultant "sheet music" can be inspected. Finally, a routine is available which prints out a composed melody in musical symbols on a graphical plotter providing a permanent paper record of the music.