Learning with Harmony Space : an overview …

                        Learning with Harmony Space : an overview
                                                Simon Holland
                                          Department of Computing,
                          Faculty of Mathematics, Open University, Milton Keynes, UK
                              Tel: UK-908-653148 email: s.holland@open.ac.uk


                                                  Abstract
         Recent developments are presented in the evolution of Harmony Space , an interface that
         exploits cognitive theories of harmony. Harmony Space is an interface that exploits both
         Artificial Intelligence and Human Computer Interaction methodologies to help beginners
         learn about and make use of aspects of tonal harmony rapidly. Use is made of Longuet-
         Higgins' and Balzano's cognitive theories of tonal harmony. The interface allows entities of
         interest (notes, chords, chord progressions, key areas, modulations) to be manipulated via
         direct manipulation techniques using a single principled spatial metaphor to make a wide
         range of musical tasks simple for novices to perform. The interface can also be used by
         experienced musicians to make a range of expert tasks more tractable than by using
         conventional tools and notations. The interface is highly interactive and multi-modal, using
         two pointing devices and spatial, aural and kinaesthetic cues that all map uniformly into the
         underlying theory. Some recent implementations of Harmony Space are discussed, together
         with some of the musical tasks which they make tractable for beginners and experienced
         musicians. Aspects of the simple, consistent, principled framework behind the interface are
         outlined.

Introduction
     Harmony Space is a highly interactive interface for exploring harmony and learning to compose music.
The interface is based on two cognitive theories of harmony. Harmony Space exploits artificial intelligence
and human computer interaction methodologies to help beginners learn about and make use of tonal
harmony.The interface was originally inspired by Longuet-Higgins (1962) theory of the perception of
harmony, and some current versions of the interface still reflect this influence. However, for many purposes,
it has turned out to be more appropriate to use Balzano's 1982 group theoretic characterisation of the
perception of harmony as a grounding for the interface. For a detailed discussion of the reasons behind this
decision, see Holland (1989)

Balzano's Theory
     To help understand the interface, it is useful to outline informally the key result of Balzano's theory. For
accounts of the derivation, see Balzano (1980) and Holland (1989). The basic idea behind Balzano's theory can
be summarised as follows.Balzano's theory identifies the 12-fold western pitch set (more or less irrespective of
tuning) as being isomorphic with C12 , the cyclic group of order 12. This identification is possible because
the notes of the scale have a natural ordering (their pitch) which is circular (under octave equivalence). The
subgroups and group theoretic properties of C12 turn out to characterise many of the harmonic properties of
the pitch-set very parsimoniously. Loosely speaking, the key result is that most of the central relationships
of tonal harmony can be expressed economically as simple, spatially expressible relationships in an array in
which notes of the chromatic scale are arranged as follows (Figure 1)
         · 4 semitones (major thirds) on one axis,
         · 3 semitones (minor thirds) on the other axis.
     There is some evidence that this representation (the "thirds space" representation) is intimately connected
with the way in which people perceive and process tonal harmony. Of course, there are other competing
psychological accounts; for example, see Watkins and Dyson (1985) and many others.

Basics of the interface
     We will now examine some of the basic ideas behind the interface. Let us start with one simple version
of the display from the current Macintosh version (Figure 1) The notes are numbered from 0-11 in ascending
pitch order. Each circle represents a note. Clicking on the circle with the mouse pointer causes that circle to
darken and the corresponding note to sound via an external synthesiser.
     The horizontal and vertical axes of the grid correspond to movements in semitone step sizes 3 and 4, for
the theoretical reasons noted above. The diagonal axes also turn out to have interesting properties. The NW
axis yields a semitone ordering. The NE axis corresponds to the circle of fifths. From an educational point of
view, these are two important dimensions that western musicians constantly make use of (and novices need to
learn about). They have simple physical, spatial interpretations in the interface. The NW axis is important
since, for example, motion along it is much used in melodies, and relates to the notion of melodic 'closeness'.
The NE axis (i.e. the circle of fifths, based on regular semitone steps of size seven) is also very important in
tonal music, for example being fundamental to the movement of bass lines, chord sequences, and key
progressions. Alphabetical, and various other forms of labelling are available(Figure 2). (Due to limitations
of screen space in the version shown, the note F#/Gb is labelled F#, but this has no theoretical significance.)
     By default, all of the notes on the screen, when played as roots, normally sound in the same octave. This
means that there are only really 12 distinguishable notes on the screen. The pattern of 12 notes has been
propagated, like a wallpaper pattern, to allow certain other patterns to strike the eye more easily. To play any
note up one or more octaves, the centre or right mouse button is pressed (on the SparcStation version), (or
the mouse button is pressed after shifting octave using the alphanumeric keyboard using the Macintosh
version). To make octave height visible as well as audible, the note circles are illuminated in different shades
of gray depending on the octave displacement. A data glove or high precision multiple-touch touchscreen
might make a better controller than a mouse, but these are not currently provided.

The diatonic scale and other scales
     By accessing the scale menu, it is possible to cause diagonal slices of any length from 1 to 12 to be
shaded in. This corresponds to picking out certain notes from the full set of 12 notes for special attention, i.e.
forming scales. While diagonal slices on the NE axis are by no means the only way to form scales, they are
of particular importance from the point of view of the group theoretic account of harmonic perception
(Balzano, 1980). Indeed, such scales have particular musical significance. For example, a diagonal section of
length 5 taken from the circle of fifths corresponds to the pentatonic scale, and length 6 corresponds to the
medieval scale, Guido's hexachord. By far the most important scale is the slice of length 7 (figure 2 ), the
familiar diatonic scale. With a scale of size 7, the diagonal slice is visually folded alongside itself (Figure 2)
and appears like a repeating box shape with a chunk cut out. To make it easier for the eye to distinguish
positions in the repeating pattern, we can separate the boxes from each other, purely as a graphical device as
in figure 2. Notes in the white area are now the notes of the diatonic scale, and notes in the black area are
chromatic notes outside the diatonic scale We will refer to the pattern of white notes (the visible shape of the
diatonic scale in the interface) as the 'key window'.

Key areas and modulation
     Using the arrow keys on the alphanumeric keypad, we can move the key window (i.e., the white diatonic
enclosure) around over the fixed grid of notes, which corresponds to modulation. For example, if we slide the
window up the diagonal cycle of fifths axis starting from the key of C, this corresponds to modulating to the
dominant. The key immediately below on the diagonal axis is the subdominant. There are 12 distinguishable
positions for the key window (i.e. 12 major keys are distinguished). In general, keys that are musically
closely related are nearby, and more remote keys are further away. A second mouse or a mole (a mouse for the
foot) might be a preferable controller to arrow keys, but this is not currently provided. We can use the
interface as it stands to play simple melodies (though the current version is not especially well suited to this)
changing octave using mouse buttons or control keys as required. Note that if we run the mouse down the SE
axis with the mouse button depressed, we we will hear the chromatic scale. If we run down the same axis but
play only the notes in the key window, (which is easily arranged by setting all 'black' notes to respond with
silence), we will hear the diatonic scale.

Chord construction
    So far we have discussed only the playing of single notes. The interface allows the user to change the
number of notes that will normally be played in each chord (i.e. playing in triads, seventh, ninths, etc) at any
time by using a menu or control key. (Better input devices might make this smoother). The interface can help
beginners study the way that scale tone diads, triads, sevenths, ninths etc build differently up on different
degrees of the scale, independently of key transpositions. The rule for building chords on any degree of the
scale is easy to appreciate visually. The rule is simply that the chord must expand along and up, and must fit
into the key window. To quote one child's description, "The notes bounce upwards". The visual behaviour
gives the beginner a simple rationale for understanding why different degrees of the scale are associated with
different heard chord qualities (namely: they conform to fit the shape of the key window: that is, the diatonic
scale).

Inversions, voicing and altered chords
     In the simplest case, all of the notes in a natural chord are sounded in root inversion in close position. In
the current Sparc version, by using the three mouse buttons, root, first and second inversions can be played
(in the Mac version, control keys may be used to play any inversion). Visually, in accordance with the
metaphor for octave height already noted above, notes played in different octaves from their root are shaded
accordingly to show their displacement in the z-plane. In addition to this manual method of control, a chord
editor can be used to voice chords in any way desired. This allows appropriately voiced chords of whatever size
(triad, seventh, etc) to be associated systematically with given degrees of the scale, or played on demand
individually when required.So, for example, one can edit in a favourite Ravel or Debussy voicing scheme. The
same chord editing mechanism that can be used to set up appropriate voicings can also be used to set up entire
new chord scales, for example, scales that involve altered chords, or minor or modal variants.

Triads and tonal centres
      In Harmony Space , there is a one-to-one mapping between chord quality and visual chord shape,
independently of key. Triads can be identified as the maximally compact three element shapes that will fit into
the diatonic scale. There can be seen to be exactly three chords of the first such shape (the major triads), and
exactly three chords of the second such shape (the minor chords). There is only one chord of the third shape,
(the diminished chord).Both major and minor chords are layed out in groups of three with a clearly visible
central major chord and a clearly visible minor chord. The existence and location of the major and minor tonal
centres are thus visually highlighted in a pedagogically useful way. This spatial metaphor for the two tonal
centres reflects a hypothesis by Wilding-White (1961), as follows. Balzano (1980) notes that before three note
harmonic textures became the norm, six of the seven possible notes in the diatonic scale were extensively
used as central notes. At about the time when triadic textures became the norm, this modal system of
harmony (with no especially privileged 'centres') collapsed to be replaced by a system in which just two
centres (major and minor) came to predominate as stable starting and finishing points. Balzano speculates that
the emergence of two structural tonal centres and the adoption of triadic textures were mutually dependent.
Balzano (1980) quotes Wilding-White (1961) to the effect that the central notion behind a tonic is not a
central note, but a central triad. Harmony Space can be used to illustrate this hypothesis in a way which is
easily demonstrated to beginners.

Accompaniment and analysis
     Harmony Space can be used to analyse and play chord progressions. (Indeed complete beginners can start
making music straight away ignoring the theoretical discussions we have just entertained). Complete
beginners can be taught very rapidly to play accompaniments to pieces in the major or minor mode in any
key using typical two, three and four chord patterns, with a clear idea of where to find tonal centres. More
extended 'classical harmonic movements' involving establishing a tonal centre, jumping to some non-tonic
triad and then moving back to the tonic in a straight line through all intermediate triads along the cycle of
fifths axis are equally easy to play. Such sequences can be played with appropriate chord qualities, as diads,
triads, sevenths or ninths, by making single vertical straight line gestures. Some harmonically rich forms of
tonal music typically use complex progressions that make use of moving targets (i.e modulations) in
systematic ways. Harmony Space offers the opportunity of understanding such harmonic patterns
strategically, in a series of trajectories, rather than just at chord by chord level. For example, the chord
sequence for John Coltrane's Giant Steps, which modulates every two chords or so, is typically considered
hard to play and to memorize. The Harmony Space trace reveals its to be a very ingenious but essentially
simple sequence of V I and II V I chords (i.e straight lines) modulating 'westwards' down in major thirds
every two or three chords. The sequence makes use of the "pun" that four such steps bring you back to the
original key. Informal trials have shown that at least some children with minimal keyboard skills can be
taught to play the chord sequences for Giants Steps in Harmony Space in a minute or two, and once learned,
it is very easily remembered. Interestingly, once the trick has been grasped, Harmony Space allows the player
to explore what simple (or radical ) variants on the basic idea might also work musically. For example, what
happens if the V I's modulate consitently up a major third, or up a minor third, or down a tone? Harmony
Space makes it very easy to explore. Each idea presents problems to the ear, which then have to be solved,
producing new potential pieces. Contrasting uses of Harmony Space include ingenious harmonisation games
devised by Pat Howard of the Open University, playable by beginners to teach them to harmonise material
such as Bach chorales. Note that as currently designed, novices require guidance in the use of Harmony Space
- there is no claim that simple exposure to Harmony Space without guidance would by itself teach novices.

Some educational uses of Harmony Space
     Harmony Space has at least six general uses; musical instrument, particularly for sketching chord
sequences; tool for musical analysis; tool for learning basic harmonisation; learning tool for exploring the
theory of tonal music; discovery learning tool for composing and modifying chord sequences; and notation for
aspects of chord sequences not obvious in conventional notations. Discussions of examples of each of these
activities, as carried out by novices, can be found in in Holland (1989).

Implementations of Harmony Space, and some variants
     A family of variants of the interaface have been implemented in prototype on the SparcStation specialised
for various musical tasks. The basic implemented variants include: a microtonal version; a just intonation
version; a version switchable to either Longuet-Higgins or Balzano configuration; a version for graphically
performing harmonic analysis by graphical best fit; a version able to drive melodic and rhythmic figural
patterns; a version as a recursive programming language able to drive the interface; a version designed to
integrate with an intelligent tutor for music composition. Many other variants have been designed,
prototyped or otherwise realised, including a human-powered performance-event version staged as a co-
operative game at the Utrecht School of Art with Peter Desain and Henkjan Honing.

Formative evaluations
      An informal qualitative investigation (Holland, 1989) using an early prototype version demonstrated that
the interface made it possible for previously musically untrained novices to learn in the space of between ten
minutes and two and a half hours to perform a range of analytical, theoretical and compositional musical
tasks that would normally be well beyond the scope of beginners without extensive training. It was
demonstrated that non-musically trained users (including a child) could use and make sense of the interface
given initial guidance.Full details can be found in (Holland, 1989).The interface has been evaluated more
recently with two classes of 12 years olds as part of regular music lessons a part of a pilot evaluation -
details will be published when the resulting data has been analysed.

Related work
     In many respects, Harmony Space is unique. Its relationship to various independently developed interfaces
including Music Mouse, Harmony Grid, and Longuet-Higgins light organ is described in Holland (1989). A
complete declarative model of the interface has been implemented as a logic program, which allows Harmony
Space to be linked with a constraint-based musical planner to form the core of an intelligent tutor for music
composition.

Limitations
     There are some aspects of harmony that Harmony Space does not represent well, for example, voice-
leading and the visualisation and control of harmony in a metrical and rhythmic context. Harmony Space
emphasises vertical aspects (in the musical traditional sense) at the expense of linear aspects of harmony,
although the Sparc version of Harmony Space can demonstrate several aspects of voice leading reasonably
well using techniques sketched in (Holland, 1986). The current implementations of Harmony Space would
benefit from various simple refinements including the following. On the current Macintosh version (less so
the Sparc version) melodic and rhythmic context is not provided by any built-in features; rhythmic playing is
hampered somewhat by response speed; only three or four levels of z-displacement are currently visible; the
constituent notes of chords are not independently manipulable while sounding; mode changing and key
changing, and label changing cause slight delays; inversions, voicings and non-standard chord scales should be
more easily changeable. Other facilities are required including: a simple visual record playback facility, a
MIDI-IN facility, more flexible ways of using the notes of a chord (via alberti patterns, appeggiation patterns
and finer manual control) and a keyboard display (on the Macintosh version).

Further work
     Around sixty further work projects related to Harmony Space are itemised in Holland (1989). We will
note two or three outstanding ones. Harmony Space is based on theories that reconceptualise the domain in
question. The reconceptualisations call for and give an opportunity to make radical changes to the curriculum
and teaching approaches in this area. The developing of new curricula and teaching methods to take advantage
of Harmony Space's theoretical basis has begun, but needs much more work. Various technical improvements
are required. Various major variants of the interface await resources to implement, such as the full graphical
analysis tool, and full associated discovery based composition tutors, currently only partly implemented.

Conclusions
     AI and Education researchers have long been interested in taking AI theories (of particular domains, ways
of learning, and styles of teaching) and of finding ways to exploit them for educational purposes. One less
advertised way of doing this is to dispense with a teaching component, an expert component, or a student
model, and to use something like the following strategy.

· Investigate as deeply as possible any cognitive theories or AI theories of the domain in question and gather
     computational models.
· Taking into account the best current HCI practice, try to devise an interface that represents abstract domain
     concepts in perceptually concrete, interactive, manipulable ways.
· Try to make concepts and relationships perceptually salient which should be conceptually important.
· Reduce cognitive load by representing explicitly and uniformly relationships that would otherwise have to
     be learned or calculated.

This approach is not at all new (Holland, 1989) , but it is far from having yet been fully exploited. We have
    presented just one example of this approach to using AI in Education.
    The best ideas in AI and Education and HCI acting in concert are probably yet to be devised.




                                                 Figure 1
                                               Figure 2

References
Balzano, G. J. (1980). The Group-theoretic Description of 12-fold and Microtonal Pitch systems. Computer
    Music Journal 4:4. Winter 1980.
Longuet-Higgins, H.C., (1962a); Letter to a Musical Friend. Music Review, August 1962 pp. 244-248
Longuet-Higgins, H.C., (1962b) Second letter to a Musical Friend. Music Review, Nov. 1962.
Holland, S. (1992) "Interface Design for Empowerment: A case study from music" in Holland, S., and
    Edwards, A. (1992) (Eds) "Multimedia Interface Design in Education". Springer Verlag, Hiedelberg
Holland, S. and Elsom-Cook, M. (1990) "Architecture of a knowledge-based music tutor". In Guided
    Discovery Tutoring, Ed. Elsom-Cook, M. Paul Chapman Publishing Ltd, London.
Holland, S. (1989) "Artificial Intelligence, Education and Music". PhD thesis published internally as OU IET
    CITE Report No. 88, July 1989.
Holland, S. (1987) "Direct Manipulation tools for novices based on new cognitive theories of harmony". pp
    182 -189 Proceedings of 1987 International Computer Music Conference.
Holland, S. (1986). Design considerations for a human-computer interface using 12-tone three dimensional
    Harmony Space to aid novices to learn aspects of harmony and composition. CITE Report No. 7, Open
    University, Milton Keynes.
Watkins, J. and Dyson, Mary C. (1985) On the perception of tone sequences and melodies. In Howell, Peter,
    Cross, Ian, and West, Robert (Eds.) Musical Structure and Cognition. Academic Press, London.
Wilding-White, R. (1961) "Tonality and Scale theory". Journal of Music Theory 5:275-286