Information about http://mcs.open.ac.uk/sh2/hspNATO.pdf

Interface design for empowerment: a case study from …

Tags: aberdeen scotland, artificial intelligence and education, balzano, case study, chord sequences, cognitive theories, computing science, education techniques, formal musical education, harmonic relationships, modified version, multimedia interface design, music composition, principled, science university, simon holland, springer verlag, tonal harmony, university of aberdeen, vocabulary,
Pages: 19
Language: english
Created: Tue Dec 12 19:22:28 2000

Display cached document

Page 1

Page 2

Page 3

Page 4

Page 5

Page 6

Page 7

Page 8

Page 9

Page 10

Page 11

Page 12

Page 13

Page 14

Page 15

Page 16

Page 17

Page 18

Page 19

Interface design for empowerment: a case study from
music

Simon Holland
Department of Computing Science,
University of Aberdeen,
Aberdeen, Scotland, AB9 2UB

Abstract

The work reported here is part of a wider project [5,6,7,8] to find ways of using
artificial intelligence and education techniques to encourage and facilitate music
composition by novices. The project is aimed at novices with little or no formal
musical education, especially those outside a formal educational setting. For this
reason, we have used illustrations and vocabulary from popular music and jazz,
although the work applies equally to tonal harmony in general (see the appendix at the
end of this paper for the conventions used for notating chord sequences). The research
exploits two recent cognitive theories of harmony due to Longuet-Higgins [11,12] and
Balzano [1] which give rise to principled and elegant representations for harmonic
relationships. In this discussion we will concentrate on the use of a modified version
of Longuet-Higgins' theory, although we have obtained very closely related results [6]
using a version of Balzano's theory.

This paper appeared as
Holland, S. (1992) Interface Design for Empowerment: A case study from music in
Holland, S., and Edwards, A. (1992) (Eds) Multimedia Interface Design in Education.
Springer Verlag, Hiedelberg

1
Interface design for empowerment: a case study from
music

Simon Holland
Department of Computing Science,
University of Aberdeen,
Aberdeen, Scotland, AB9 2UB

1. Introduction

2. Longuet-Higgins' theory of harmony

Longuet-Higgins' theory of harmony [11,12] investigates the
properties of an array of notes arranged in ascending perfect
fifths on one axis and major thirds on the other axis (figure 1)1.
Longuet-Higgins' representation turns out to be a good framework
for theories explaining how people perceive and process tonal
harmony [15]. Longuet-Higgins' [11,12] theory asserts that the set
of intervals that occur in Western tonal music are those between
notes whose frequencies are in ratios expressible as the product of
the three prime factors 2, 3, and 5 and no others. Given this
premise, it follows from the fundamental theorem of arithmetic

1 In Longuet-Higgins' presentations of the theory, and in all discussions of it in the
psychological literature, the convention is that ascending perfect fifths appear on
the x-axis and the ascending major thirds on the y-axis. We reverse this for
educational purposes on two grounds. Firstly, it allows students to switch more
easily between the Balzano representation and the 12-note version of the
Longuet-Higgins representation. (The x-axes become coincident and the y-axes
are seen to be related by a 'shear' operation.) Secondly, the V-I movements that
dominate Western tonal harmony at so many different levels become aligned
with physical gravity in a metaphor useful to novices.

2
that the set of three intervals consisting of the octave, the perfect
fifth and the major third is the only co-ordinate system that can
allow all intervals in musical use (and only those intervals) to be
given unique co-ordinates. We can represent this graphically by
laying out notes in a three dimensional grid with notes ascending
in octaves, perfect thirds and major fifths along the three axes.
The octave dimension is discarded in most discussions on grounds
of octave equivalence and of practical convenience for focussing
on the other two dimensions (figure 1).

Db F A C# E# Gx

Gb Bb D F# A# Cx

Cb Eb G B D# Fx

Fb Ab C E G# B#
key
Perfect Bbb Db F A C# E# window
fifths
for key of
Ebb Gb Bb D F# A# C Major
Abb Cb Eb G B D#

Major thirds
Figure 1. Longuet-Higgins' note array
Diagram adapted from Longuet-Higgins [11]
The theory has been of great interest in the cognitive psychology
of music [9] as a framework for explaining how people perceive
and process tonal harmony. Our chosen focus here is on applying
the theory to develop new educational tools.

2.1 Keys and modulation
We begin by looking at how various 'static' relationships in
harmony appear in this representation. In diagrams such as figure
1, all of the notes of the diatonic scale are 'clumped' into a
compact region. For example, all of the notes of C major, and no
other notes, are contained in the box or window in figure 1. If we
imagine the box or window as being free to slide around over the
fixed grid of notes, we will see that moving the window vertically
upwards or downwards, for example, corresponds to modulation
to the dominant and subdominant keys respectively. Other keys

3
can be found by sliding the window in other directions. Despite
the repetition of note names, it is important to realise that notes
with the same name in different positions are not the same note,
but notes with the same name in different key relationships.
However, for the purposes of educating novices in the elementary
facts of tonal harmony it turns out to be convenient to map
Longuet-Higgins' space onto the twelve note vocabulary of a
fixed-tuning instrument, resulting in a 12-note, two-dimensional
version of Longuet-Higgins' space. The collapse to the 12-fold
space makes it apparently impossible to make distinctions about
note spelling that could be made in the original space. However,
we can console ourselves with the thought than in this respect it is
no more misleading than a piano keyboard. (And we will see
shortly that it makes many harmonic relationships far clearer
than a piano keyboard does.)
As a result of our decision, the double sharps and double flats of
figure 1 are lost, and the space now repeats exactly in all
directions (figure 2). Notes with the same name really are the
same note in this space. In fact a little thought will show that the
space is in fact a torus, which we have unfolded and repeated like
a wallpaper pattern. Instead of a single key window there are
replicated copies of the same key window.

4
D F B D F B D F

G B E G B E G B

C E A C E A C E

F A Db F A Db F A

B D F B D F B D
key
E G B E G B E G windows
for
A C E A C A C
perfect E C major
fifths Db F A Db F A Db F
using
12-note F B D F B D F B
pitch set B E G G B E
B E

E A C E A C E A

A Db F A Db F A Db

major thirds using
12-note pitch set
Figure 2. 12-note version of Longuet-Higgins' note array
Note that we have used arbitrary spellings in these diagrams (e.g..
F# instead of Gb etc.), but we could equally easily use neutral
semitone numbers or some other preferred convention.

2.2 Chords and tonal centres
Let us now turn to look at the representation of triads and tonal
centres. In 12-note versions of Longuet-Higgins' space, major
triads correspond to L-shapes (figure 3). A triad consists of three
maximally close distinct notes in a configuration that can fit a key
window. The dominant and subdominant triads are maximally
close to the tonic triad. We can see from the diagram that the
three primary triads contain all the notes in the diatonic scale.
Notice also that we have a clear spatial metaphor for the
centrality of the tonic - the tonic triad is literally the central one
of the three major triads of any major key. We can make similar
observations for the minor triads. Minor triads correspond to
rotated L-shapes. Like major triads, they are maximally compact
three-element objects that can fit a key window. The three

5
secondary triads generate the natural minor (and major) scale
(We can deal with harmonic and melodic minor scales by
introducing variant key window shapes, but we will not pursue
this here). Also, the space gives a clear visual metaphor for the
centrality of the relative minor triad among the secondary triads2.
Completing the full set of scale tone triads for the major scale, the
diminished triad is a sloping straight line. Seventh or ninth chords
similarly have memorable and consistent shapes in the 12-note
space. See figure 4 for the representation of scale tone sevenths.

D D F D

G B G B G B D

C E C E C E G

F A F A F AC
E
D D D
B D B D F A
G B B G
* B BD
G

C E C E C E
F A diminished
A
F A F
chord approached

Major triads Minor triads Diminished
in C major in C Major chord
I, IV and V II, III and VI VII
Figure 3 Triads in the 12-fold space

3 A computer-based interface
We will now present the essential points of the design of an
interface, Harmony Space, based on the Longuet-Higgins'
representation. Several versions of the interface have been
implemented that exhibit in various form all of the key design
decisions described below. There is a grid of notes displayed on a
computer screen, each circle representing a note. Two pointing
devices, such as mice, are provided. One mouse controls the
location of a cursor that highlights and sounds any note-circle it

2 The 'centrality of the tonic' argument as applied to the tonal centre of the minor mode
is borrowed from Balzano [1]. It is not valid in the full non-repeating Longuet-
Higgins space but works in the 12-note Longuet-Higgins version.

6
passes over, provided the mouse button is down at the time.
More generally, the mouse can control the location of the root of a
diad, triad, seventh or ninth chord. (We will refer to the number
of notes in the chord as its 'chord-size'.) The chord-size can be
varied by the user using a second pointing device or control keys.
As the root is moved around, the quality of the chord
automatically changes appropriately for the position of the root in
the scale. So for example, unless overridden with the other mouse
by the user, the chord on the tonic will be a major triad (or major
seventh if we are using sevenths) and the chord on the supertonic
will be a minor triad (or minor seventh if we are using sevenths).
We refer to these chord qualities as the default chord qualities
for the chord-size and degree of the scale. Of course, default chord
qualities will sometimes need to be overridden. As we have
already mentioned in passing, this is controlled using a second
pointing device (control keys are used in all versions implemented
to date).
Although the qualities of chords are assigned automatically by
default as the root is moved by the user, there is a clear visual
metaphor for the basis of the automatic choice, because the shape
of the chord appears to change to fit the physical constraint of the
key window. The second pointing device is also used to move the
key window. Moving this pointing device corresponds to changing
key. If, for example we modulate by moving the window while
sounding the same chord root, the chord quality may change. Once
again there is a clear visual metaphor for what is happening since
the shape of the chord will appear to be "squeezed" to fit the new
position of the key window. Note circles can be displayed with
alphabetical pitch names (e.g. C, G, Eb, etc.) or functional names
(e.g. I, V, IIIb, etc.). The alphabetical pitch name associated with a
given note circle remains fixed irrespective of the position of the
key window, whereas the functional name associated with a note
circle varies as the key window is moved, in accordance with the
meaning of functional names.
The interface is linked to a synthesizer so that everything we
have described can be heard at the same time. The resulting
interface of which we have given an outline description is called
"Harmony Space".

7
D D F D D

G G B G B
B G B

C E C E C E C E

F F A F A A
A F
D B D
D D D
B D B D

G B G B G B G B

C E C E C E
C E

F A F A F A F A
B
Scale tone Scale tone Dominant Diminished
Major sevenths Minor sevenths seventh chord
in C Major in C Major chord V VII
I and IV II, III and VI

Figure 4. Scale tone seventh chords in 12-fold space

3.1 Representing harmonic progression
So far we have looked at the representation of key areas and
chords in the 12-fold space. Let us now move on to look at
harmonic progression and succession. It turns out that many of
the fundamental harmonic progressions of Western tonal music
correspond to very simple paths in Harmony Space. These
patterns do not appear to have been noted explicitly in previous
discussions of Longuet-Higgins' or Balzano's theories, perhaps
because Longuet-Higgins' theory is usually considered in the non-
repeating form where these patterns do not appear, and Balzano's
theory is usually applied to quite different purposes.
Firstly, the I V I progression which is so commonplace in tonal
music can be seen as one that begins on the central major triad of
the key, and then moves to a maximally close neighbour before
returning home (figure 3). Similarly, progressions involving I, IV
and V can be seen as oscillating either side of the tonal centre by
the smallest possible step and then returning home.
Moving onto wider chord vocabularies, we notice that
fundamental progressions like II V I, VI II V I , III VI II V I
etc. (see the appendix for the chord notation convention)
correspond to straight lines vertically downward in the 12-fold
space with a tonal centre as their target (figure 5). We refer to
straight line motions in 12-fold Harmony Space to tonal goals as
harmonic trajectories.

8
III VII
III
VI VI
VI
II II II II
a V V V V
I I I I

b
c
Figure 5. Circles of fifths in 12-fold space
The circle of fifths is of fundamental importance in western tonal
music. We can distinguish between two classes of circle of fifths.
In the first case, 'real' circle of fifths, the root moves in straight
lines down the perfect fifth axis, sounding all roots on its path,
irrespective of whether they are inside or outside the key window
- e.g. figure 5c (including half-shaded points). In the second case
(tonal circle of fifths) only notes in the key window are sounded.
In Harmony Space this corresponds to straight lines that "jump"
where necessary to avoid notes outside the key window - e.g.
figure 5c excluding half-shaded points. For example, if we are in
the key of C, the root is forced to make an irregular jump of a
diminished fifth from F to B in order to stay in key. Note that the
"jumps" can be drawn equivalently as "bends" (e.g. figure 5b) due
to the fact that different occurrences of notes with the same name
in the space are equivalent notes.
Using the interface, we can audibly and visibly play tonal circles
of fifths simply by making a vertical straight line gesture with the
mouse. The chord quality can be seen and heard flexing to fit
within the key window (figures 3 and 4). This works even if
there are modulations (movements of the key window) mid-chord
sequence. To play a real circle of fifths, we simply switch off the

9
option that prevents us sounding roots in the chromatic area
outside the key window. (Note that for chords with roots outside
the key window, there are no "obvious" chord qualities. Such
chords may be assigned some arbitrary quality in advance or may
be given an appropriate quality by hand with the second pointing
device as they occur.)

3.2 Manipulating and representing arbitrary harmonic sequences
In general, it turns out that straight line gestures on various axes
in Harmony Space, particularly gestures ending on tonal centres,
are of particular importance or interest in Western tonal harmony.
The circle of fifths progressions already seen are in many ways
the most important. Some related progressions are discussed
below.
If we reverse the direction of movement on the circle of fifths axis
and consider chord sequences moving vertically upwards, we
have what might be called, following Steedman [18], extended
plagal sequences and cadences. Short plagal cadences are very
common but extended chord sequences of this sort (i.e. chord
sequences like I V II VI etc.) are rare, probably for reasons
explored by Steedman [18]. Extended plagal chord sequences are
occasionally used as the basis for short pieces, for example "Hey
Joe" (popular arr. Jimi Hendrix).

III
IV

Figure 6. Scalic progressions
Turning to other axes of Harmony Space, scalic sequences (i.e. we
use this term to mean movement up and down the diatonic scale)
can be represented as diagonal trajectories constrained to remain
within the key windows (figure 6 ). So for example, the chord
sequences I II III II I, IV III II I etc. can be represented as

10
diagonal trajectories or diagonal oscillations. Scalic root movement
is frequent in tonal music in short sequences and is often used as
the basis of harmonic sequences in modal music, for example,
commonly by Michael Jackson and Phil Collins.
If the constraint is removed that the root must stay within the
key window, scalic sequences become chromatic sequences (figure
7). Chromatic chord succession is widely used in some jazz
dialects. Such chord progressions coincide on every other chord
with a circle of fifths progression, and are viewed in some
circumstances by jazz musicians as substitutes for circles of fifths
(in a jazz practice known as "tritone substitution").
Extended straight line harmonic trajectories in 'real' (chromatic)
major or minor thirds are not harmonically very useful because
they touch few roots in any given key. Diatonic progressions in
thirds,where the roots move through alternating intervals of
major and minor thirds, are sometimes used as a basis for pieces
and can be played easily in Harmony Space with a zig-zagging
gesture.
In summary, simple extended physical gestures such as straight
lines along appropriate axes towards tonal centres correspond to
important harmonic progressions in tonal harmony. Other basic
progressions correspond to further simple patterns (not explored
here). In general, we can play any desired chord sequences in
Harmony Space by making gestures in the appropriate directions.
It is important to note that a visual formalism is not being
proposed as a substitute for listening. However, Harmony Space
can allow novices without instrumental skills and without
knowledge of standard theory or terminology to gain experience
of controlling and analysing such sequences. Harmony Space is
also a good place to learn music theory if a novice so desires.

11
I
bIIx
II
III bIIIo

IVo

bV�

Figure 7 Chromatic progressions

4 Informal qualitative investigation
An informal qualitative investigation was carried out with a small
number subjects to discover whether Harmony Space is usable by
novices and to find out whether it can enable them to perform
musical tasks that would normally be difficult for novices to carry
out by other means. We summarise briefly the results of this
investigation. Full details can be found in Holland [6]. In brief, it
was demonstrated that musical novices with no previous musical
training can be taught, using Harmony Space, in the space of
between 10 minutes and two and a half hours to carry out tasks
including:
� Perform harmonic analyses of the chord functions and
modulations (ignoring inversions) of such pieces as Mozart's "Ave
Verum Corpus". The harmonic analysis was performed on a
version played in triads, in close position, in root inversion. This
task was carried out by one subject after only 10 minutes
training.
� Accompany sung performances of songs, playing the correct
chords, on the basis of simple verbal instructions or
demonstrations. Songs were selected with a range of contrasting
harmonic constructions.
� Learn and perform simple strategies for composing chord
sequences using 'musical plans' such as 'return home', 'cautious
exploration', 'moving goal-post' and 'modal harmonic ostinato'.
More details of 'musical plans' can be found in Holland [5,6].
� Modify existing chord sequences in musically 'sensible' ways, for
example perform what jazz musicians refer to as 'tritone
substitutions' on simple jazz chord sequences.
� Play and recognise various classes of abstractly described,

12
musically useful chord sequences in various keys both
diatonically and chromatically.
� Carry out various musical tasks, such as to recognise and
distinguish chord qualities; to use the rule for scale tone chord
construction; to locate major and minor tonic degrees in any key;
and to make use of the rationale for the centrality of the major
and minor tonics.
� Locate, recognise and distinguish examples of important
harmonic entities and phenomena. For example: identify examples
of the major and minor tonics, modulations, and major and minor
triads in various keys.
We will summarise the most important limitations of the
investigation. It was a qualitative evaluation. The sample was
small (five subjects), though of varied age, nationality and social
background. Only single sessions were used. The harmonic
analyses were performed on reduced harmonic versions played in
triads, in close position, in root inversion. Given these limitations,
the investigation indicated that beginners with no previous
musical training and a wide range of ages can be taught very
quickly to use the interface. It was demonstrated that musical
novices can be taught in a matter of minutes using the prototype
to carry out a range of musical tasks that it would typically take
weeks or months for beginners to learn by conventional methods.

5 Related work
To the best of our knowledge, the work described in this chapter
was the first application of Longuet-Higgins' theory for
educational purposes. Holland [8] appears to be the first discussion
of use of Longuet-Higgins' representation for controlling a musical
instrument, and the implemented prototype appears to be the
first such instrument constructed. A number of musical
interfaces, each related in some way to Harmony Space are
described below. Each was developed independently of the others.
The first device is Longuet-Higgins' light organLonguet-Higgins
(reported in Steedman [17] page 127) connected each key of an
electronic organ to an square array of light bulbs illuminating
note names. This device was the first instrument to make explicit
use of Longuet-Higgins' theory. It allowed music played on the
organ to be displayed in Longuet-Higgins' non-repeating space.
However, the question did not arise of working in the opposite
direction to allow the grid representation to control the organ.
The use of the non-repeating space means that the paths we have
discussed do not emerge as straight lines. The key window does
not appear to have been represented on the display.
The first computer-controlled device using a generalised two-

13
dimensional note-array (the meaning of this should become clear
in a moment) with pointing device to control a musical
instrument seems to be Levitt's program Harmony Grid (figure 8)
[10]. Harmony Grid runs on an Apple Macintosh. It displays a two
dimensional grid of notes where the interval between adjacent
notes may be adjusted independently for the x and y axes to any
arbitrary number of semitone steps. The grid display can control
or display the output of any musical instrument with a MIDI
interface (Musical Instrument Digital Interface - an industry
interconnection standard). Harmony Grid can be configured as a
special case to the grid layouts of Balzano's space or Longuet-
Higgins 12-fold space. However, the question of key windows, or
their analogue, is not considered in Harmony Grid. This means that
Harmony Grid does not make explicit the bulk of the relationships
and structures described in this chapter. The mouse can control
chords, but their quality must be adjusted manually - there is no
notion of inheriting or constraining chord quality from a key
window. Hence modulation can be carried out only by knowing
and manually selecting the appropriate chord quality as each
chord is played. Levitt's pioneering program is a superb
educational tool, robustly implemented with a good real time
response and many practical features. It was the first
implemented program of its kind. The many differences and
similarities between Harmony Space and Harmony Grid raise very
interesting musical, educational and interface design issues, but
they are beyond the scope of this chapter.

Figure 8 Harmony Grid
Balzano has worked on the design of computer-based educational

14
tools for learning about music. At a conference, Balzano [2]
referred to an educational tool based on his group-theoretic
approach to harmony, but this does not appear to have been
discussed in the literature yet.

6 Recent developments and limitations
The original implementations of Harmony Space (1986) were
experimental prototypes designed to show the coherence and
practicality of the design. The prototypes served their intended
purpose, but were too slow and basic to make them easy to use.
More recently, much more sophisticated and general versions of
Harmony Space have been implemented on SPARC workstations
working with James Bisset and Colin Watson. These versions are
very accurate and fast, and the latest version is switchable
between many configurations including the Balzano and Longuet-
Higgins spaces, as well as several theoretically interesting
microtonal spaces. It can be linked to a variety of rhythmical
filters, and is a "two and a half dimensional" interface, making
use of shading to show information about octaves and inversion.
This version has facilities for recording performances, and being
driven by a guided discovery intelligent tutoring system for music
composition [6]. A performance event using a specially
constructed human-powered version of Harmony Space was
performed at the Utrecht Art School's Centre for Knowledge
Technology at the invitation of Peter Design and Henkjan Honing
in 1990. Part of the aim was to allow people to experience and
control harmony and melody with the movement of their whole
bodies. In a series of games, participants moved around in a large
Harmony Space grid marked out on the floor. Their movements
'controlled' a specially trained group of musicians whose playing
was partly determined by the Harmony Space configuration. A
large, specially constructed wooden key window was shifted
around under the players' feet to control modulations. Games
included 'exploratory walks', polyphonic games, improvisatory
games and discovery learning games.
A straightforward extension of the present research would be a
systematic educational evaluation of Harmony Space. One simple
empirical investigation planned is to take a small group of
students and find out the extent to which composition, analysis,
accompaniment and music theory skills learned using harmony
space can be transferred to keyboards.

There are some aspects of harmony that Harmony Space does not represent well, for
example, voice-leading, the control and representation of voicing and inversion, and the
visualisation and control of harmony in a metrical context. Harmony Space emphasises
vertical aspects (in the traditional musical sense) at the expense of linear aspects of
harmony. To a large extent this is an inherent limitation of Harmony Space, although

15
Harmony Space can demonstrate some special cases of voice leading rather well [8].
Some other limitations of the research are as follows: Harmony Space deals only with
tonal harmony; Harmony Space is not very well suited to dealing with melody and
rhythm; inverted harmonic functions have been ignored, some musical terms and
notations have been used in unusual ways. Details of possible ways of addressing
some of these problems can be found in Holland [6].

7 Harmony Space and interface design
Harmony Space exemplifies and combines a high density of
generally applicable interface techniques for making abstract or
inaccessible entities and relationships accessible. Theories of
interface design are scarce, but it is suggested that Harmony Space
may provide a useful precedent for future research on interfaces
designed to promote accessibility. Four techniques simultaneously
exemplified by Harmony Space can be identified as follows;
� Make visible a representation underlying a theory of the
domain. In the case of Harmony Space, Longuet-Higgins' 12-fold
space (and the structures implicit in it) are made concretely
visible and directly manipulable. In TPM (the Transparent Prolog
Machine, Eisenstadt and Brayshaw [4], the proof trees underlying
Prolog execution are made dynamically visible (but not directly
manipulable).
� Map a task analogically from a sensory modality in which a task
is difficult or impossible for some class of user into another
modality where it is easy or at least possible to perform. Apart
from Harmony Space, other interfaces that use this technique
include Edward's Soundtrack [3], Harmony Grid and Eisenstadt and
Brayshaw's TPM [4]. Soundtrack maps a WIMP graphic interface
into the auditory modality so that blind and partially sighted
users can use it.
� Use a single, uniform, principled metaphor to render abstract,
theoretical relationships into a form that can be concretely
experienced and experimented with. In the case of Harmony
Space, the abstractions are those of music theory. A striking
recent example of another interface using this technique is ARK
(the Alternative Reality Kit), due to Smith [16]. In ARK, all objects
represented in the interface have position, velocity and mass and
can be manipulated and 'thrown' with a mouse-driven 'hand'. The
laws of motion and gravity are enforced but may be modified,
allowing students to perform alternative universe experiments.
Much of the experiences it deals with cannot normally otherwise
be experienced in the original modality and must normally be
approached using abstract formulae. ARK uses the abstractions of
physics to create a world in which such local barriers can be
surmounted, allowing students access to normally inaccessible
experiences.

16
� Search for and exploit a single metaphor that allows principles
of consistency, simplicity, reduction of short term memory load
and exploitation of existing knowledge to be used. The aim is to
make a task normally difficult for novices easy to to perform.
Harmony Space does this (this is analysed in detail in Holland [6]),
but perhaps the best known example of such an interface is the
Xerox Star (and latterly Macintosh) interfaces, which exploit a
shared office metaphor and uniform commands consistently
across a range of contexts.
None of these techniques is new by itself. However, just as the
notion of 'direct manipulation' (Shneiderman, [14]) characterised
existing practices in a useful way, it may be that new
combinations of practices are starting to emerge whose
identification and analysis would be of general value in designing
interfaces to promote accessibility. An interesting research
program would be to analyse these and other highly empowering
interfaces, with the aim of explicitly characterising and
generalising the techniques they use, and establishing the extent
to which they can be applied to wider domains.

8 Conclusions
We have presented a theoretically motivated design for a
computer-based interface for exploring tonal harmony. The
interface exploits cognitive theories to give principled, uniform,
metaphors in two sensory modalities (visual and kinaesthetic) for
harmonic relationships and processes occurring in a third
modality (the auditory).
Various versions of the interface have been constructed. A
qualitative investigation has demonstrated that the interface can
enable musical novices to learn very quickly to perform a range of
tasks that would normally be very difficult for them to do.
It has been shown that many of the fundamental harmonic
progressions of Western tonal music correspond to very simple
paths in Harmony Space. These patterns do not appear to have
been noted previously in discussions of Longuet-Higgins' or
Balzano's theories, perhaps because Longuet-Higgins' theory is
usually considered in the non-repeating form where these
patterns do not appear, and Balzano's theory is usually applied to
quite different purposes.
To the best of our knowledge, the work described in this chapter
was the first application of Longuet-Higgins' theory for
educational purposes, and the first use of Longuet-Higgins'
representation for controlling a musical instrument.
We have identified a number of generally applicable interface
design approaches in common to the following: Harmony Space, an
interface for Prolog programming, an interface for Newtonian

17
physics, and an auditory interfaces for blind users. It is suggested
that analysis of such design approaches may contribute to general
theoretical frameworks for the design of highly empowering
interfaces.

Acknowledgements
Thanks to Mark Elsom-Cook. This paper would not have existed
without Mark Steedman's suggestion that Longuet-Higgins' theory
was a good area to explore for educational applications. Thanks to
Tim O'Shea for much help. Thanks to Mark Steedman, John
Sloboda, Trevor Bray, Richard Middleton, Alistair Edwards and
Mike Baker for comments on earlier drafts. Thanks to Christopher
Longuet-Higgins, Ed Lisle and Mike Baker for valuable
discussions. Thanks to James Bisset and Colin Watson for their
enthusiastic work on new implementations. The support by the
ESRC of this work under doctoral research studentship
C00428525025 is gratefully acknowledged.
This paper incorporates edited, supplemented, and revised
versions of some material that appeared as part of an article in
the proceedings of the 1987 International Computer Music
Conference as Holland [7].

Appendix: Chord symbol conventions
Chord symbol conventions are based on Mehegan [13]. Roman
numerals representing scale tone triads or sevenths are written
in capitals, irrespective of major or minor quality (e.g. I II II IV
V etc.). Roman numerals represent triads of the quality normally
associated with the degree of the tonality (or modality)
prevailing. We call this quality the "default" quality. In the jazz
example, Roman numerals indicate scale-tone sevenths rather
than triads. The following post-fix symbols are used to annotate
Roman chord symbols to override the chord quality as follows ; x
- dominant, o - diminished, � - half diminished, m - minor, M -
major. The following post-fix convention is used to alter indicated
degrees of the scale; "#3" means default chord quality but with
sharpened 3rd, "#7" means default chord quality but with
sharpened 7th etc. The following post-fix convention is used to
add notes to chords e.g. "+6" means default chord quality with
added scale-tone sixth 6th. The prefixes # and b move all notes of
the otherwise indicated chord a semitone up or down.

References
1. Balzano, G. J.: The Group-theoretic Description of 12-fold and Microtonal
Pitch systems. Computer Music Journal 4:4. Winter 1980

2. Balzano, G.: Restructuring the Curriculum for design: Music, Mathematics

18
and Psychology. Machine Mediated Learning, 2: 1&2. (1987)

3. Edwards, A. D. N.: Integrating synthetic speech with other auditory cues
in graphical computer programs for blind users, Proceedings of the IEE
International Conference on Speech Input and Output, London. (1986)

4. Eisenstadt, M. and Brayshaw, M. The Transparent Prolog Machine
(TPM): an execution model and graphical debugger for logic programming.
Journal of Logic Programming. (1988)

5. Holland, S. and Elsom-Cook, M.: "Architecture of a knowledge-based
music tutor". In Guided Discovery Tutoring, M. Elsom-Cook (ed.). M.
Paul Chapman Publishing Ltd, London. (1990)

6. Holland, S. Artificial Intelligence, Education and Music. PhD thesis,
published as CITE report 88. Open University, Milton Keynes, England.
(1989)

7. Holland, S. "Direct Manipulation tools for novices based on new cognitive
theories of harmony". pp 182 -189 Proceedings of 1987 International
Computer Music Conference. (1987)

8. Holland, S.: Design considerations for a human-computer interface using
12-tone three dimensional Harmony Space to aid novices to learn aspects of
harmony and composition. CITE Report No. 7, Open University, Milton
Keynes. (1986)

9. Howell, P., Cross, I. and West, R. (Eds.): Musical Structure and
Cognition. Academic Press, London. (1985)

10. Levitt, D. H. and Joffe, E.: Harmony grid .Computer program, MIT.
(1986)

11. Longuet-Higgins, H.C.: Letter to a Musical Friend. Music Review,
August 1962 pp. 244-248 (1962a)

12. Longuet-Higgins, H.C.: Second letter to a Musical Friend. Music Review,
Nov. 1962. (1962b)

13. Mehegan, J.: Jazz Improvisation I: Tonal and Rhythmic Principles.
Watson-Guptil Publications, New York. (1959)

14. Shneiderman, B.: The future of interactive systems and the emergence of
direct manipulation. Behaviour and Information Technology 1: 237-256.
(1982)

15. Sloboda, J.: The Musical Mind: The Cognitive Psychology of Music.
Clarendon Press, Oxford. (1985)

16. Smith, R.B.: "Experiences with the Alternate Reality Kit: An example of
the tension between literalism and Magic". Proceedings of the Conference
on Human Factors in Computer Systems, Toronto, April, 1987.

17. Steedman, M. : The formal description of musical perception. Unpublished
Phd, University of Edinburgh. (1972)

18. Steedman, M. : A generative grammar for jazz chord sequences. Music
Perception 2:1. (1983).