Chapter 8

Interactive Musical Instruments

Topics: Computer musical instruments, graphical user interfaces, graphics objects and widgets, event-driven programming, callback functions, Play class, continuous pitch, audio samples, MIDI sequences, paper prototyping, iterative refinement, keyboard events, mouse events, virtual piano, parallel lists, scheduling future events.

8.1 Overview

This chapter explores graphical user interfaces and the development of interactive musical instruments. While it takes years to master playing a guitar or violin, it is much easier for beginners to play a computer-based musical instrument, especially if they already have experience with computer games and other graphical applications. The goal here is not to replace traditional instruments or to undermine the years of experience required to achieve expertise in performing with them, but to introduce and engage more people in musical performance and composition. Also, interactive computer-based musical instruments offer considerable versatility. They can be used by a single performer or by multiple performers in ensembles, like Laptop Orchestras or iPad Ensembles. It is also possible to have an ensemble that includes both traditional instruments and computer-based instruments. Computer-based musical instruments can be customized to particular compositions or musical styles. They may be used to perform music at impromptu musical happenings (such as at a party or a coffee shop) or be part of formally composed, avant garde orchestral pieces.

8.2 Building Musical Instruments

To assist live interaction with our programs, we explore graphical user interfaces (GUIs) in this chapter. Through the various GUI primitives available, we can design unique interactive musical instruments for live performance. While such instruments use a GUI for interaction, under the hood they utilize the various Python building blocks we have seen so far. Additionally, developing interactive musical instruments through Python (i.e., using the GUI and music libraries) facilitates music making by other people, including your friends, who do not know how to program in Python. You can design a GUI for each specific piece or music-making activity. Then, through this GUI, the instrument players (i.e., the “end-users”) can make music without having to know how to program in Python.

You (the programmer and GUI designer) become a “digital luthier” of sorts—a digital instrument maker, who can enable and shape the musical experience of others. Your ability to program musical interactions and design unique, innovative GUIs gives you immense power, because your design choices will affect the type of music your end-users can make and, ultimately, their creative expression.

The ability to shape (to both enhance and constrain) other people’s expression through your programs results in software similar to that of any ready-made product, including ready-made music production software (such as GarageBand, Audacity, and Ableton Live). Such software tools impose conceptual and compositional constraints, which were put in place (either by design or inadvertently) by their developers. These constraints focus the types of musical expression possible through them, as discussed by Magnusson (2010). This is why it is so important to know how to develop your own tools—so you can explore new ways to channel your creative expression. This argument is made very clearly and forcefully by Douglas Rushkoff in Program or Be Programmed (Rushkoff 2010).

Of course, building effective, usable instruments (digital or physical) is hard work. In this chapter, we will lay the foundation by learning some basic human–computer interaction (HCI) ideas, including usability, design analysis concepts, and paper prototyping. We will also learn the basics of GUI development, using the provided GUI library (a simplified, yet powerful library for developing graphical user interfaces). Finally, we will further explore functions and event-driven programming.

8.3 Graphical User Interfaces

In this chapter we develop music applications that have a graphical user interface (GUI). For this we use the GUI library provided with the book. As with the music library, the GUI library follows Alan Kay’s maxim that “Simple things should be simple, and complex things should be possible” (Davidson 1993). Appendix C contains the complete list of graphical objects and GUI widgets available in this library.

To access the GUI library, you need to use the following statement:

from gui import *

From that point on, your program can access all the available GUI objects and functionality.

8.3.1 Creating Displays

A program’s GUI exists inside a display (window). Displays contain other GUI components (graphics objects and widgets). A program may have several displays open. Displays may contain any number of GUI components, but they cannot contain another display.

The Display class is the most important component in the GUI library. Display objects have the following attributes:

• Title—a string, which appears at the top display border
• Width—a positive integer, which defines the number of horizontal pixels inside the display (x axis)
• Height—a positive integer, which defines the number of vertical pixels inside the display (y axis)

For example, the following:

d = Display("First Display", 400, 100)

creates a display with the given title, width, and height (as shown in Figure 8.1 and Figure 8.2).

Notice how the GUI library uses the style of the operating system at hand. (This is because it is built on top of the Java Swing library, which has this desirable characteristic.)*

Once a display has been created, you can add GUI widgets and other graphical objects, using the following function:

d.add(object, x, y)

where object is a GUI widget or graphical object (presented below). The coordinates x, y specify where in the display to place the object.

Fact: The origin of a display (0, 0) is at the top-left corner.

8.3.2 Graphics Objects

Once you have created a display, you can add a wide variety of graphical objects. Adding a graphics object to a display actually draws the corresponding shape onto the display.

The available shapes are Line, Circle, Point, Oval, Rectangle, and Polygon. Each of these shapes can have a specific color and thickness (measured in pixels). Additionally, 2D shapes (i.e., circle, oval, rectangle, and polygon) can be drawn filled (i.e., solid) or not (i.e., outline).

These graphics primitives allow you to create any graphics composition imaginable (clearly, some easier to create than others).

For example, the following code

d = Display("First Display", 400, 100)

c = Circle(200, 50, 10)	# x, y, and radius

d.add(c)

r = Rectangle(180, 30, 220, 70) # left-top and right-bottom corners

d.add(r)

l = Line(160, 50, 240, 50) # x, y of two points

d.add(l)

l1 = Line(200, 10, 200, 90)

d.add(l1)

creates the compound graphics shape in Figure 8.3.

8.3.2.1 Exercise

Create a function containing the code used to create the graphics shape in Figure 8.3. This function should have three arguments, namely, display, x, and y. The first argument is the display to draw the shape. The other arguments are the x and y coordinates of the shape’s center on the display. Hint: Modify the above code to use relative coordinates from x and y. For example, the circle should be drawn at precisely x and y, whereas the rectangle’s top-left corner should be at x-20 and y-20, etc.

Converting from absolute coordinates to relative coordinates (from a given location, e.g., center point) allow us to generalize and reuse code for drawing complex graphics shapes.

8.3.3 Showing Display Coordinates

For convenience, Display objects provide the function:

d.showMouseCoordinates()

which shows the coordinates of the cursor, using a tool tip.† This allows you to discover the coordinates of where to place various GUI components and thus simplifies GUI development.

To turn off this functionality, use the function:

d.hideMouseCoordinates()

For an overview of available Display (and other graphics objects) functions, see Appendix C.

8.4 Case Study: Random Circles

This case study demonstrates how to combine some of the programming building blocks we have learned so far (namely, randomness, loops, and GUI functions) to draw random circles on a display.

# randomCircles.py

#

# Demonstrates how to draw random circles on a GUI display.

#

from gui import *

from random import *

numberOfCircles = 1000	# how many circles to draw

# create display

d = Display("Random Circles", 600, 400)

# draw various filled circles with random position, radius, color for i in range(numberOfCircles):

	# create a random circle, and place it on the display

	# get random position and radius

	x = randint(0, d.getWidth()-1)	# x may be anywhere on display

	y = randint(0, d.getHeight()-1)	# y may be anywhere on display

	radius = randint(1, 40)	# random radius (1-40 pixels)

	# get random color (RGB)

	red = randint(0, 255)	# random R (0-255)

	green = randint(0, 255)	# random G (0-255)

	blue = randint(0, 255)	# random B (0-255)

	color = Color(red, green, blue)	# build color from random RGB

	# create a filled circle from random values

	c = Circle(x, y, radius, color, True)

	# finally, add circle to the displayd.add(c)

# now, all circles have been added

Every time you run this program, it generates 1000 random circles and places them on the created display (see Figure 8.4). The display has a default width of 600 pixels and height of 400 pixels.

The center of each circle is placed at x, y coordinates which are selected randomly from 0 to d.getWidth() and 0 to d.getHeight(), respectively. As their names suggest, these display functions provide the actual dimensions of the display d.

Notice how the color of each circle is created using Color(red, green, blue), where red, green, and blue are the color’s RGB color. Each is assigned a random value between 0 and 255 (which is the range of RGB values).

8.4.1 Exercises

1. Write a program that draws a 1000 random shapes (include rectangles, lines, ovals, and polygons). For this exercise, utilize the compact way of drawing shapes on a display, that is, with the display functions drawCircle(), drawLine(), and so on (see Appendix C).
2. Write a program that draws a 1000 compound shapes similar to the one in Figure 8.3. Hint: Create a function that draws one such shape using relative coordinates (see earlier exercise). Then call this function 1000 times (in a loop), providing random x and y coordinates, ranging between 0 and the corresponding display border (width or height).

8.5 GUI Widgets

In addition to graphics objects, the GUI library supports a wide variety of widgets. Widgets are graphical components used for input and output. They receive user input and/or display information. Widgets are the interactive components out of which GUIs are built.

The available widgets include:

• Label—objects that present textual information.
• Button—objects that can be pressed by the user to perform an action.
• Checkbox—objects that can be selected (or deselected) by the user.
• Slider—objects that can be adjusted by the user to input a value.
• DropDownList—objects that contain items which can be selected by the user.
• TextBox—objects that allow the user to enter a single line of text.
• TextArea—objects that allow the user to enter multiple lines of text.
• Icon—objects that allow displaying of external images (.jpg or .png).
• Menu—objects that contain items which can be selected by the user. Menu objects are fixed at the menu bar (top), whereas DropDownList objects are placed anywhere on a display.

Below we will see various examples of how to use these widgets to build interactive musical instruments.

8.5.1 Event-Driven Programming

Interaction and building GUIs require a new way of thinking about programming. Up until now, our code was executed in the order specified by an algorithm (combining sequence, selection, and iteration). Every now and then, we introduced user interaction, through the input() function. This had the potential to change the order of execution, through if statements examining the value entered by the user (e.g., if the value is negative do this, otherwise do that). This interaction made our programs more dynamic and customizable, based on user input.

GUI widgets provide a new and more powerful way to get input from the user. Instead of the user typing text to a program prompt (via the input() function), GUI widgets allow the design of intricate user interfaces, which support diverse input scenarios and user interactions.

However, this requires a new way of thinking.

8.5.2 Callback Functions

With GUI widgets, we have no way of knowing when a user will interact with a GUI widget. Therefore, GUI widgets allow you to specify a function to be called when they are activated (i.e., the user interacts with them). In other words, the function gets called as a result of the user interaction. If the user does not interact with the widget, the function never gets called.

Definition: A callback function is a function associated with a GUI widget that gets called only when (and if) the user interacts with that widget.‡

Callback functions are user-defined functions, which allow us to associate arbitrary functionality to a widget. It is through a combination of widgets and their callback functions that we implement the functionality of a GUI.

Definition: Event-driven programming refers to the style of programming introduced by GUIs, where code is not executed in the order specified by a fixed algorithm but, instead, code is executed in reaction to user-initiated events (if and when these may occur) from the user interface.

Event-driven programming and callback functions are not confined to GUIs. Other mechanisms for program interaction, for example, MIDI input/output and Open Sound Control (OSC), employ the same idea (as we will see in Chapter 9). In other words, you use components that receive user input (from wherever this input comes). Then you assign callback functions to be executed when (and if) user input arrives. These functions process the user input and make it available to other parts of the program (if needed). The collective functionality of an event-driven program is put together through individual callback functions associated with interactive program elements (e.g., widgets).

8.6 Case Study: A Simple Musical Instrument

The following program demonstrates event-driven programming. It creates the GUI shown in Figure 8.5. This GUI control surface allows the starting and the stopping of a single note. Soon we will see how to generalize this idea to create more versatile interactive musical instruments.

The GUI consists of two Button widgets. The first starts a note. The second stops the note. Under the hood, each button utilizes its own callback function, which performs the desired functionality when (and if) the button is pressed.

Fact: Callback functions have a fixed number of arguments, specified by the widget that calls them.

For example, callback functions for Button widgets must accept zero parameters.§ If we use a function with the wrong number of arguments, this will cause a runtime error (i.e., the error will be reported when the button is pressed at runtime). So it is very important to follow documentation and test GUI code carefully. If you do not test a particular widget, you will not see an error (even if one is waiting to happen) until a user comes across it.

The following program makes use of several Play class functions we have not seen before. These functions, namely, Play.noteOn() and Play.noteOff(), facilitate interactive performance by allowing the duration of a note to be determined in real time by the end user. This is in contrast to Play.midi(), which we have used extensively and is geared toward the playback of compositional, noninteractive, algorithmic music.

Play.noteOn() and Play.noteOff() are explained in more detail in the next section. For now, simply accept that they exist and that they do (pretty much) what their names suggest.

# simpleButtonInstrument.py

#

# Demonstrates how to create a instrument consisting of two buttons,

# one to start a note, and another to end it.

#

from gui import *

from music import *

# create display

d = Display("Simple Button Instrument", 270, 130)

pitch = A4	# pitch of note to be played

# define callback functions

def startNote():	# function to start the note

 global pitch	# we use this global variable

 Play.noteOn(pitch)	# start the note

def stopNote():	# function to stop the note

 global pitch	# we use this global variable

 Play.noteOff(pitch)	# stop the note

# next, create the button widgets and assign their callback functions

b1 = Button("On", startNote)

b2 = Button("Off", stopNote)

# finally, add buttons to the display

d.add(b1, 90, 30)

d.add(b2, 90, 65)

Notice the two callback functions, startNote() and stopNote() that trigger the sending of noteOn and noteOff messages.¶ The only thing that makes them callback functions is our intention to use them as such. They are regular user-defined functions, which happen to accept zero parameters. What makes them “callback functions” is that we assign them as callback functions to the two Button objects. Since Button objects require callback functions with zero arguments, we made sure that these functions do indeed require zero arguments.** Also, in order to assign them as callback functions, Python requires that they are defined before the Button objects are created.

Finally, notice the statement (used in both of the functions):

	global pitch # we use this global variable

This states that functions startNote() and stopNote() use a variable defined outside them. Normally, such a variable would be passed through the parameter list. However, this is not the case here, as explained below.

8.6.1 Python Global Statement

As we mentioned above, callback functions have a fixed number of arguments, specified by the widget that calls them. If these functions require access to other variables (e.g., a label to update or a display to draw on), these variables need to be specified somehow. Normally, such variables are passed through the argument list. However, since Button objects (and other GUI widgets) control the parameter list of callback functions, we have to use the Python global statement.

For example, in the above functions, the statement

global pitch

informs the interpreter that the variable pitch, which is defined outside the function, is needed (i.e., will be used by) the function. This is equivalent to having this variable in the function argument list, except we do not have to physically put it there.††

Good Style: The global statement should be used minimally—mainly in callback functions. For normal functions, such variables should be passed in as arguments.

According to the Python manual, “variables that are only referenced inside a function are implicitly global. If a variable is assigned a new value anywhere within the function’s body, it’s assumed to be a local. If a variable is ever assigned a new value inside the function, the variable is implicitly local, and you need to explicitly declare it as global.”

8.6.2 Exercise

Expand the above program to create a second set of buttons for a different instrument (your choice) playing another note (also your choice).

8.7 Play Class

The Play class, as its name suggests, contains functions related to playing music in real time.

In previous chapters we used the function Play.midi(). As you recall, this function allows the playing of any musical material (Note, Phrase, Part, or Score). Actually, you can have several calls to Play.midi() active at the same time—each will play its own musical material in parallel with the others.

Fact: Play.midi() makes extensive use of computational resources, so the number of parallel calls is limited by the computational load of your system.

As stated earlier, Play.midi() is geared toward compositional music making.

Additionally, the Play class provides several functions geared toward interactive (e.g., GUI) music making. These functions are intended for building interactive musical instruments:

Function	Description
Play.noteOn(pitch, velocity, channel)	Starts pitch sounding. Specifically, it sends a NOTE_ON message with pitch (0–127), at given velocity (0–127—default is 100), to be played on channel (0–15—default is 0) through the Java synthesizer.
Play.noteOff(pitch, channel)	Stops pitch from sounding. Specifically, it sends a NOTE_OFF message with pitch (0–127), on given channel (0–15—default is 0) through the Java synthesizer. If the pitch is not sounding on this channel, this has no effect.
Play.allNotesOff()	Stops all notes from sounding on all channels.
Play.setInstrument(instrument, channel)	Sets a MIDI instrument (0–127—default is 0) for the given channel (0–15, default is 0). Any notes played through channel will sound using instrument.*
Play.getInstrument(channel)	Returns the MIDI instrument (0–127) assigned to channel (0–15, default is 0).
Play.setVolume(volume, channel)	Sets the global (main) volume (0–127) for this channel (0–15). This is different from the velocity level of individual notes—see Play.noteOn().
Play.getVolume(channel)	Returns the global (main) volume (0–127) for this channel (0–15).
Play.setPanning(position, channel)	Sets the global (main) panning position (0–127) for this channel (0–15). The default position is in the middle (64).
Play.getPanning(channel)	Returns the global (main) position (0–127) for this channel (0–15).
Play.setPitchBend(bend, channel)	Sets the pitch bend for this channel (0–15—default is 0) to the Java synthesizer object. Pitch bend ranges from −8192 (max downward bend) to 8191 (max upward bend). No pitch bend is 0 (which is the default). If you exceed these values the outcome is undefined (it may wrap around or it may cap, depending on the system.)
	Continued
Play.getPitchBend(channel)	Returns the current pitch bend for this channel (0–15—default is 0).
Play.frequencyOn(frequency, volume, channel)	Starts a note sounding at the given frequency and volume (0–127—default is 100), on channel (0–15—default is 0). Warning: You should play only one frequency per channel. (Since this uses pitch bend indirectly, it will affect the pitch of all other notes sounding on this channel.)
Play.frequencyOff(frequency, channel)	Stops a note sounding at the given frequency on channel (0–15—default is 0). Warning: You should play only one frequency per channel. (Since the frequency gets translated to a pitch and a pitch bend, this will also affect notes with nearby frequencies on this channel.)
Play.allFrequenciesOff()	Same as Play.allNotesOff(). Stops all notes from sounding on all channels.
* You should avoid using both Play.setInstrument() and Play.midi(), since the latter also makes instrument assignments indirectly (as specified by its musical material, e.g., a Part).

Finally, the Play class provides an advanced function for scheduling notes to be played at a later time:

8.8 Case Study: An Audio Instrument for Continuous Pitch control

In this section, we explore how to incorporate audio samples (as opposed to MIDI) into our music-making activities. Being able to play arbitrary audio files through our programs opens new doors for composition, performance, and creative expression.

This case study demonstrates how to use GUI functions to create a simple instrument for changing the volume and frequency of an audio loop in real time. The following program creates the GUI control surface shown in Figure 8.6.

This GUI surface consists of two sliders to control frequency and volume, respectively. It also uses two labels to provide feedback to the user about what the current sliders’ values are (as they change, in real time).

# continuousPitchInstrumentAudio.py

#

# Demonstrates how to use sliders and labels to create an instrument

# for changing volume and frequency of an audio loop in real time.

#

from gui import *

from music import *

# load audio sample

a = AudioSample("moondog.Bird_sLament.wav")

# create display

d = Display("Continuous Pitch Instrument", 270, 200)

# set slider ranges (must be integers)

minFreq = 440	# frequency slider range 

maxFreq = 880	# (440 Hz is A4, 880 Hz is A5)

minVol = 0	# volume slider range

maxVol = 127

# create labels

label1 = Label("Freq: " + str(minFreq) + " Hz") # set initial text

label2 = Label("Vol: " + str(maxVol))

# define callback functions (called every time the slider changes)

def setFrequency(freq):	# function to change frequency

 global label1, a	# label to update, and audio to adjust

 a.setFrequency(freq)

 label1.setText("Freq: " + str(freq) + " Hz")	# update label

def setVolume(volume):	# function to change volume

 global label2, a	# label to update, and audio to adjust

 a.setVolume(volume)

 label2.setText("Vol: " + str(volume))	# update label

# next, create two slider widgets and assign their callback functions

#Slider(orientation, lower, upper, start, eventHandler)

slider1 = Slider(HORIZONTAL, minFreq, maxFreq, minFreq, setFrequency)

slider2 = Slider(HORIZONTAL, minVol, maxVol, maxVol, setVolume)

# add labels and sliders to display

d.add(label1, 40, 30)

d.add(slider1, 40, 60)

d.add(label2, 40, 120)

d.add(slider2, 40, 150)

# start the sound

a.loop()

First, notice the class AudioSample is used to load an audio file into the program. AudioSample is very useful for building interactive musical instruments because it opens the door to recorded sound (as opposed to MIDI). This important class is presented in detail below. For now, observe how easy it is to load an audio file and play it in a loop. This is done with the following statements:

a = AudioSample("moondog.Bird_sLament.wav")

a.loop()

where “moondog.Bird_sLament.wav” is an audio sample (WAV audio file) containing the opening phrase of Moondog’s piece “Bird’s Lament” (1969). Available for download at http://jythonMusic.org this audio file should be stored in the same folder as the program. The second statement, a.loop(), starts the audio and repeats it indefinitely (see the next section for a complete list of AudioSample functions).

Notice the two functions, setFrequency() and setVolume(). Since they are used as slider callback functions, they are required to have only one argument, namely, the value of the slider. These functions are called every time the corresponding slider has been adjusted.

These functions, in addition to adjusting the sound, update the corresponding labels. Therefore, these variables, namely, label1 and label2, respectively, are identified via global statements (as discussed earlier).

Finally, notice the statement:

la bel1.setText("Freq: " + str(freq) + " Hz") # update label

This updates the text being displayed by label1. The label text is a string, so we use a combination of strings and the “+” (string concatenation) operator. Since variable freq is an integer (slider values are always integers), we use the str() conversion function to convert the value of freq to a string (so that it can be concatenated with the other strings). The result becomes the single string displayed in label1.

Good Style: Always provide feedback to the user via the GUI about the result of his/her actions. This makes users feel in control, which makes your software more usable.

8.9 AudioSample Class

The AudioSample class, as its name suggests, contains functions related to playing audio samples in real time. An audio sample is a sound object created from an external audio file, which can be played, looped, paused, resumed, and stopped.

8.9.1 Creating Audio Samples

The AudioSample class has the following attributes:

• filename – a string, the name of the audio file to be loaded (.wav or .aif)
• pitch – a MIDI pitch (0–127) which will be associated with the audio sample (default is A4)‡‡
• volume – a positive integer (0–127), which sets the initial volume of playback (default is 127)

The following function creates a new AudioSample, so you need to save it in a variable (in order to use it later).

Function	Description
AudioSample(filename, pitch, volume)	Creates an audio sample from the audio file specified in filename (supported formats are WAV and AIF— 16, 24, and 32 bit PCM, and 32-bit float). Parameter pitch (optional) specifies a MIDI note number to be used for playback (default is A4). Parameter volume (optional) specifies a MIDI note velocity to be used for playback (default is 127).

For example, an audio sample may be created as follows:

a = AudioSample("rainstorm.wav", C4, 100)

creates an audio sample, a, with pitch C4 and volume 100.

An audio sample can be played once, looped, stopped, paused, and resumed. Also, we can change its pitch or frequency in real time (through pitch shifting) by using setPitch() or setFrequency().

Once an audio sample, a, has been created, the following functions are available:

Function	Description
a.play() a.play(start, size)	Play the sample once. If start and size are provided, the sample is played from milliseconds start until milliseconds start+size (default is 0 and −1, respectively, meaning from beginning to end).
a.loop() a.loop(times, start, size)	Repeat the sample indefinitely. Optional parameters times specifies the number of times to repeat (default is −1, indefinitely). If start and size are provided, looping occurs between milliseconds start and milliseconds start+size (default is 0 and −1, respectively, meaning from beginning to end).
a.stop()	Stops sample playback immediately.
a.pause()	Pauses sample playback (remembers current position for resume).
a.resume()	Resumes sample playback (from the paused position).
a.isPlaying()	Returns True if the sample is still playing, False otherwise.

The following functions control audio sample playback parameters (pitch, frequency, and volume).

Function	Description
a.setPitch(pitch)	Sets the sample pitch (0–127) through pitch shifting from sample’s base pitch.
a.getPitch()	Returns the sample’s current pitch (it may be different from the default pitch).
a.setFrequency(freq)	Sets the sample pitch frequency (in Hz). This is equivalent to setPitch(), except it provides more granularity (accuracy). For instance, pitch A4 is the same as frequency 440 Hz.
a.getFrequency()	Returns the current playback frequency.
a.setVolume(volume)	Sets the volume (amplitude) of the sample (volume ranges from 0 to 127).
a.getVolume()	Returns the current volume (amplitude) of the sample (volume ranges from 0 to 127).
a.getFrameRate()	Returns the sample’s recording rate (e.g., 44100.0 Hz).
a.setPanning(panning)	Sets the panning of the sample (panning ranges from 0–127).
a.getPanning()	Returns the current panning of the sample (panning ranges from 0–127).

8.9.2 Exercise

Using buttons, sliders, and other GUI elements, create a Music Production Controller (MPC).

• MPCs are electronic musical instruments (originally produced by Akai) that feature a grid of buttons that allow a user to play back (trigger) various samples.
• Use an external audio editor (such as Audacity) to capture and manipulate arbitrary audio files and live inputs.
• Then plan a music performance using the produced audio samples together with your custom MPC.

8.10 MidiSequence Class

The MidiSequence class provides playback functionality for MIDI sequences that is similar to the functionality described above for audio samples. This parallelism in the functionality of the two classes allows us to easily mix sounds from audio and MIDI material (files) in interactive musical instruments.

8.10.1 Creating MIDI Sequences

MidiSequence objects can be created from an external MIDI file (as well as Note, Phrase, Part, and Score objects).

The MidiSequence class has the following attributes:

• Material—a string, the name of a MIDI file to be loaded (.mid) or a Note, Phrase, Part, and Score object
• Pitch—a MIDI pitch (0−127) which will be associated with the material (default is A4)§§
• Volume—a positive integer (0−127), which sets the initial volume of playback (default is 127)

The following function creates a new MidiSequence, so you need to save it in a variable (in order to use it later).

Function	Description
MidiSequence(material, pitch, volume)	Creates a MIDI sequence from the MIDI material specified in material (this may be a filename of an external MIDI file or a Note, Phrase, Part, and Score object. Parameter pitch (optional) specifies a MIDI note number to be used for playback (default is A4). Parameter volume (optional) specifies a MIDI note velocity to be used for playback (default is 127).

For example, a MIDI sequence may be created as follows:

m = MidiSequence("beat1.mid", C4, 100)

creates a MIDI sequence, m, with pitch C4 and volume 100.

A MIDI sequence can be played once, looped, stopped, paused, and resumed. Also, we may change its pitch, tempo, and volume. These changes happen immediately.

Once a MIDI sequence, m, has been created, the following functions are available:

Function	Description
m.play()	Play the MIDI sequence once.
m.loop()	Repeat the MIDI sequence indefinitely.
m.stop()	Stops MIDI sequence playback immediately.
Continued
m.pause()	Pauses MIDI sequence playback (remembers current position for resume).
m.resume()	Resumes MIDI sequence playback (from the paused position).
m.isPlaying()	Returns True if the MIDI sequence is still playing, False otherwise.

The following functions control MIDI sequence playback parameters (pitch, frequency, and volume).

Function	Description
m.setPitch(pitch)	Set the MIDI sequence’s playback pitch (0–127) by transposing the MIDI material.
m.getPitch()	Returns the MIDI sequence’s playback pitch (0–127).
m.setTempo(tempo)	Set MIDI sequence’s playback tempo in beats per minute (e.g., 60).
m.getTempo()	Return MIDI sequence’s playback tempo (in beats per minute).
m.getDefaultTempo()	Return MIDI sequence’s default tempo (in beats per minute).
m.setVolume(volume)	Returns the volume of the MIDI sequence (0–127).
m.getVolume()	Returns the current volume of the MIDI sequence (0–127).

8.10.2 Exercises

1. Extend the Music Production Controller (MPC) of the previous exercise to include MIDI sequences. Using algorithmic techniques explored in earlier chapters, develop a few interesting MIDI sequences. Plan a music performance using the combined audio samples and MIDI sequences through your custom MPC.
2. Given everything you have seen so far (in this chapter, and before), you can now create a wide variety of musical laptop instruments. Your imagination and musical creativity is the limit. Design an innovative interactive musical instrument. Interview a few musicians. Start with a paper prototype. Construct a functional prototype and refine it using input from your target audience. Plan a performance using this new instrument, possibly together with other traditional and software instruments.

8.11 Paper Prototyping

Paper prototyping is a technique from the field of human–computer interaction. It involves constructing a mock-up of the GUI on paper (usually via paper and pencil) (Moggridge 2007, Stone et al. 2005, Greenberg et al. 2011).

Definition: Paper prototypes are throwaway models of the GUI of a program, involving rough (usually hand-sketched) drawings.

Since drawing on paper is much faster than programming, paper prototyping allows you to explore and refine the GUI design with minimal effort. Once the design is finalized on paper, we can code it. Again, consider the programmer maxim “2 hours of design can save you 20 hours of coding.” It also most definitely applies to GUI programming.

Fact: Although paper prototyping may appear simplistic, it is used widely in industry for GUI development.

Paper prototyping provides valuable user feedback early enough in the development process. This helps shape the GUI design with minimal effort. The earlier it is done, the better. The more often it is done, the better. It simply saves programming effort.

Consider, on the other hand, if you wait to seek user input on your design until after you have programmed your GUI. If you make the changes suggested by users, you throw away part of your program and start again. If you resist making those changes, the users will be unhappy with (and may avoid using) your program. Either way you end up wasting valuable effort.

Good Style: Use paper prototypes (even if rough sketches) whenever possible. They save considerable development time. Also they allow you to explore aesthetic choices before you implement a particular idea.

For example, Figure 8.7 shows the paper prototype for the GUI shown in Figure 8.6.

8.12 A Simple Methodology for Developing GUIs

This section presents a simple methodology for developing GUIs through Python and the GUI library. It involves iterative refinement, through these three steps:

1. Develop a paper prototype. Get up to five representative users.¶¶ Show them the paper prototype. Ask them to imagine using it to perform the intended task (e.g., make music). Ask them for their thoughts and preferences. For example, do users like a horizontal slider for some functionality, or would they prefer something else? Where should this slider be placed? What should the labels contain? What should the colors be? And so on. When the users talk, listen. Do not explain (or even worse, defend) your initial design choices. Your goal is to learn from the users what they like (hence, the cheap implementation on paper—it’s very easy to change). Iterate, as necessary, until the paper prototype captures the design that most users prefer. Once the paper prototype has evolved to capture all possible user ideas and preferences, you may move to the next step.
2. Create a blank Display object with the dimensions specified in the paper prototype. Use the showMouseCoordinates() function to identify the approximate position of the various GUI widgets and graphics objects. Notate those coordinates on the paper prototype. If necessary, make numeric adjustments on the paper prototype (see Figure 8.7).
3. Once you have the approximate coordinates for the various GUI components, write code that creates them and places them on the display. Running your program interactively, through the Python interpreter, use the display function move() to fine-tune the position of the various GUI components relative to each other. If necessary, go back and adjust the size of the display. Notate those changes on the paper prototype. Make final adjustments in your code, and present to users. Be ready to make additional adjustments. If necessary, go back to step (1), especially if users suggest a radical change.

These steps may be repeated as more functionality and GUI components are added. By developing a system through this iterative methodology, you can focus on the most important design aspects first. Then you can add more details. By involving users early on in the design process, you are more likely to develop a system that is useable and appreciated by the intended user group. Your development effort will be well spent.

8.12.1 Listen, Listen, Listen

Listening is the most important skill of a GUI developer. Yet it is the hardest thing to learn to do.

When paper prototyping, beginner designers feel the need to explain why they designed something a certain way. At the moment you start explaining (or telling the user how to do something) during a paper prototyping session, you have biased the user toward your design and lost the opportunity to get new, unbiased ideas.

Expert GUI designers remain mostly silent during such sessions, and only ask questions to help probe the user’s mind as to what the user expected to see. It is helpful to realize that, once you have collected all user preferences and desires, you can still decide not to use them. The choice is yours. The goal of paper prototyping is to collect these user preferences and desires, early on in the development effort, so that you can build the best, most usable system possible.

8.13 Event Handling

The GUI library provides a wide variety of functions to handle keyboard and mouse events. These functions are available for every GUI object (i.e., displays, graphics objects, and widgets). If desired, you can customize every component on your GUI to respond differently to user keyboard or mouse actions. This provides immense power for building various interactive behaviors and functionalities.

Every GUI object listens for user actions (e.g., mouse click, mouse drag, typing a key, etc.) and allows you to specify a callback function to be executed if and when a user action occurs. You decide which events to handle and on which objects.

8.13.1 Keyboard Events

Every widget in the GUI library supports the following functions: onKeyType(), onKeyDown(), onKeyUp(). Each of these functions accepts one argument, namely, the callback function you have created (programmed) to handle this particular event.

For example, in the next case study, the function clearOnSpacebar() is given as an argument to the display’s function onKeyDown(). In other words, clearOnSpacebar() will be called every time the user presses a key while the display is in focus.

Callback functions for keyboard events are passed the key that was typed, pressed, or released, accordingly. The callback function decides what to do with this information. As an example of this, see the function clearOnSpacebar() below.

8.13.2 Mouse Events

Every GUI object provides the following functions: onMouseClick(), onMouseDown(), onMouseUp(), onMouseMove(), onMouseDrag(), onMouseEnter(), onMouseExit(). Each of these functions accepts one argument, namely, the callback function you have created (programmed) to handle this particular event.

Callback functions for mouse events are given the x and y coordinates of the mouse. These are the coordinates of where the corresponding mouse event happened on the GUI. The callback function decides what to do with these coordinates. As an example of this, see the function beginCircle() below.

8.13.2.1 Example

To explore mouse events, let’s start the Python interpreter and try the following:

>>> from gui import *

>>> d1 = Display("Playing with Events", 200, 355)

>>> d1.getWidth()

200

>>> d1.getHeight()

355

This creates a display with the given title, and with width 200 and height 355 (as demonstrated by the calls to display’s getWidth() and getHeight() functions). Nothing new here.

Now let’s draw a circle at position (50, 100) with radius 81 pixels, using the color orange. Leave the circle unfilled and make the outline 4 pixels thick:

>>> c1 = d1.drawCircle(50, 100, 81, Color.ORANGE, False, 4)

Let’s move the circle to coordinates (80, 120):

>>> d1.move(c1, 80, 120)

Now let’s create a function that moves the circle to those coordinates when called. Below, we define the function interactively. If we wish this function to be permanent, we need to define it inside a program. (Notice the ellipses, “...”. These are automatically generated by the interpreter to indicate that it is expecting more input (i.e., the function body is not complete). To finalize the function body, we hit the <Enter> key twice.)

>>> def moveCircle():

... global c1, d1

... d1.move(c1, 80, 220)

Now let’s create a button that calls this function when pressed. Also, let’s add it to the display:

>>> b1 = Button("Move Circle", moveCircle)

>>> d1.add(b1, 25, 307)

Now let’s press the button. This should move the circle to coordinates (80, 200). Then let’s manually move the circle to coordinates (20, 20), through the code below.

>>> d1.move(c1, 20, 20)

Alternate back and forth between pressing the button and manually moving the circle.

Next, let’s create another function. This function will be associated with a mouse event (below), so let’s define it to accept two parameters, that is, the x and y coordinates of the mouse position (when the mouse event occurs).

>>> def moveCircle1(x, y):

... global d1, c1

... d1.move(c1, x, y)

Let’s call this function manually a few times.

>>> moveCircle1(0, 0)

>>> moveCircle1(10, 30)

>>> moveCircle1(0, 0)

Finally, let’s associate it with the display’s mouse-click event:

>>> d1.onMouseClick(moveCircle1)

Now, clicking the mouse anywhere on the display moves the circle to that position. Pressing the button moves it back to coordinates (80, 220). Play with this interactive functionality a little to fully understand it.

8.13.2.2 Exercises

1. Make the circle follow the mouse movement (Hint: See display’s onMouseMove() function.)
2. How could this be used to create a 2D slider? Hint: The only thing you need to do is to add a few more statements inside the moveCircle1() function, to adjust the parameters of, say, an AudioSample (e.g., connect x value (mouse coordinate) with frequency and y value with volume). There are numerous possibilities here to build innovative interactive instrument displays.
3. What other interactive possibilities can you think of for controlling a circle using GUI elements? Try some more things out. How about adding more graphical objects (with different colors, etc.)? What about using images (GUI icons) instead of graphical objects?

8.13.3 Case Study: Drawing Musical Circles

This program demonstrates how to use event handling to build a simple interactive musical instrument. The user plays notes by drawing circles (see Figure 8.8).

The diameter of the circle determines the pitch of the note (within the major scale). To draw a circle, the user presses the mouse on the display and starts moving the mouse. The point where the mouse is released determines the size of the circle (also the pitch of the note).

The mathematics used to map the diameter of the circle is relatively simple, but generates a musical effect that is interesting. Although the user might expect that doubling of the diameter will double the pitch of the note, that’s not the case. It takes a little while to figure out how mouse distance corresponds to pitch, and it is precisely this challenge that makes this “game” interesting—not unlike regular video games.

# simpleCircleInstrument.py

#

# Demonstrates how to use mouse and keyboard events to build a simple

# drawing musical instrument.

#

from gui import *

from music import *

from math import sqrt

### initialize variables ######################

minPitch = C1 # instrument pitch range

maxPitch = C8

# create display

d = Display("Circle Instrument") # default dimensions (600 x 400)

d.setColor(Color(51, 204, 255)) # set background to turquoise

beginX = 0 # holds starting x coordinate for next circle

beginY = 0 # olds starting y coordinate

# maximum circle diameter - same as diagonal of display

maxDiameter = sqrt(d.getWidth()**2 + d.getHeight()**2) # calculate it

### define callback functions ######################

def beginCircle(x, y): # for when mouse is pressed

	global beginX, beginY

	beginX = x # remember new circle's coordinates

	beginY = y

def endCircleAndPlayNote(endX, endY): # for when mouse is released

global beginX, beginY, d, maxDiameter, minPitch, maxPitch

	# calculate circle parameters

	# first, calculate distance between begin and end points

	diameter = sqrt((beginX-endX)**2 + (beginY-endY)**2)

	diameter = int(diameter)	# in pixels - make it an integer

	radius = diameter/2	# get radius

	centerX = (beginX + endX)/2	# circle center is halfway between...

	centerY = (beginY + endY)/2	# ...begin and end points

	# draw circle with yellow color, unfilled, 3 pixels thick

	d.drawCircle(centerX, centerY, radius, Color.YELLOW, False, 3)

	# create note

	pitch = mapScale(diameter, 0, maxDiameter, minPitch, maxPitch,

		MAJOR_SCALE)

	# invert pitch (larger diameter, lower pitch)

	pitch = maxPitch - pitch

	# and play note

	Play.note(pitch, 0, 5000)	# start immediately, hold for 5 secs

def clearOnSpacebar(key):	# for when a key is pressed

	global d

	# if they pressed space, clear display and stop the music

	if key == VK_SPACE:

	d.removeAll()	# remove all shapes

	Play.allNotesOff()	# stop all notes

### assign callback functions to display event handlers ######################

d.onMouseDown(beginCircle)

d.onMouseUp(endCircleAndPlayNote)

d.onKeyDown(clearOnSpacebar)

First, notice the use of the math library.

from math import *

This library contains various mathematical constants (e.g., pi) and functions. Here we are using it for the square root function, that is, sqrt().***

Notice the call to display’s setColor() function:

d.setColor(Color(51, 204, 255)) # set background to turquoise

This sets the display background to the specified color using RGB values. However, if you call setColor() without parameters, it brings up a color chooser from which you can pick the desired color. This color picker also displays the RGB values, which allows you to hardcode them in your program to recreate a given color. This is a very useful feature.

8.13.3.1 Defining Callback Functions

As seen earlier, the callback functions are regular user-defined functions with specific argument lists. Functions intended to be mouse event-handlers need to accept two arguments, namely, the x and y coordinates of the mouse, when the event happens (e.g., mouse click). To be keyboard event-handlers, functions need to accept one argument, namely, the key pressed/released/typed (for more information see Appendix C).

In order for callback functions to do anything interesting, they usually require more information. But, since their argument lists are fixed and cannot be extended, this additional information is made available to them through global variables.

Good Style: Global variables should be avoided.†††

Fact: In the case of callback functions (since their argument lists are specified by the event handler they are connected to), global variables are a necessity.

The above program’s callback functions use a few global variables to provide information to each other and the main program (see the global statements in these functions). Of these, the variables beginX and beginY are of particular importance. They are set by the callback function beginCircle(). This function is associated with the display’s onMouseDown event, for example,

d.onMouseDown(beginCircle)

This indicates that the function beginCircle() will be called when the user presses down the (left) mouse button inside the display. Its arguments, x and y, will be automatically set to the mouse x and y coordinates at that moment. The function saves these x and y coordinates to the global variables beginX and beginY for further use by the program.

In particular, the callback function endCircleAndPlayNote() uses those variables to determine the diameter of the circle to draw, and from that to determine the pitch of the note to play. This function is associated with the display’s onMouseUp event, as in

d.onMouseUp(endCircleAndPlayNote)

This means that function endCircleAndPlayNote() will be called when the user releases the mouse button. Its arguments, x and y, will be automatically set to the mouse x and y coordinates at that moment. The function uses these x and y coordinates to finalize and draw the circle, and to construct and play the corresponding note.

Finally, the callback function clearOnSpacebar() clears the display and stops all notes from sounding. This function is associated with the display’s keyDown event, for example,

d.onKeyDown(clearOnSpacebar)

This means that the function clearOnSpacebar() will be called when the user presses any key on the keyboard. Its argument, key, will be automatically set to the virtual code of the key pressed. If the key is the spacebar (i.e., virtual key VK_SPACE), the function clears the display and stops all notes.

In summary, by associating callback functions with GUI events, we synthesize the functionality of the GUI control surface.

Figure 8.8 shows the image generated from one musical performance with this instrument. Part of the performance is lost, namely, the pitch and timing of the notes. What remains is still artistically interesting in its own right. This also demonstrates the creative possibilities of combining the GUI and music libraries to build interactive musical instruments.

8.13.3.2 Exercises

1. Explore different mappings between circle radius and note pitch. One possibility is to go up one octave when the radius is doubled. (Hint: For this, you might find function Note.freqToMidiPitch(freq) useful—see Appendix B.) Explore other possibilities. Finding a balance between challenge and predictability might make the instrument interesting in various ways.
2. Explore different ways to map interactions with sound for this instrument. Possibilities include connecting x position of circle with pitch, connecting y position of circle with volume, connecting radius of circle with volume, and/or connecting pitch with color of circle. Can you think of other possibilities?
3. What other musical instruments can you design, combining GUI controls, mouse/keyboard events, and music-making functions? There are many intriguing possibilities. Can you think of a few? Can you think of instruments that could be complementary to each other (for example, instruments you can use to create a laptop band)? What about instruments that involve game play as part of the interaction?

8.14 Case Study: A Virtual Piano

This case study demonstrates how to create an interactive musical instrument that incorporates images. The following program combines GUI elements to create a realistic piano which can be played through the computer keyboard.

It uses an image of a complete piano octave, namely, iPianoOctave.png, to display a piano keyboard with 12 keys unpressed. Then, to generate the illusion of piano keys being pressed, it selectively adds the following images to the display (see Figure 8.9).

• iPianoWhiteLeftDown.png (used for “pressing” keys C and F),
• iPianoBlackDown.png (used for “pressing” any black key),
• iPianoWhiteCenterDown.png (used for “pressing” keys D, G, and A), and
• iPianoWhiteRightDown.png (used for “pressing” keys E and B).

# iPianoSimple.py

#

# Demonstrates how to build a simple piano instrument playable

# through the computer keyboard.

#

from music import *

from gui import *

Play.setInstrument(PIANO)	# set desired MIDI instrument (0-127)

# load piano image and create display with appropriate size

pianoIcon = Icon("iPianoOctave.png")	# image for complete piano

display = Display("iPiano", pianoIcon.getWidth(),

	pianoIcon.getHeight())

display.add(pianoIcon)	# place image at top-left corner

# load icons for pressed piano keys

cDownIcon	= Icon("iPianoWhiteLeftDown.png")	# C

cSharpDownIcon	= Icon("iPianoBlackDown.png")	# C sharp

dDownIcon	= Icon("iPianoWhiteCenterDown.png")	# D

# ...continue loading icons for additional piano keys

# remember which keys are currently pressed

keysPressed = []

#####################################################################

# define callback functions

def beginNote(key):

	"""This function will be called when a computer key is pressed.

	It starts the corresponding note, if the key is pressed for

	 the first time (i.e., counteracts the key-repeat function of computer keyboards).

	"""

	global display	# display surface to add icons

	global keysPressed	# list to remember which keys are pressed

	print "Key pressed is " + str(key)  # show which key was pressed

	if key == VK_Z and key not in keysPressed:

	display.add(cDownIcon, 0, 1)	# "press" this piano key

	Play.noteOn(C4)	# play corresponding note

	keysPressed.append(VK_Z)	 # avoid key-repeat

	elif key == VK_S and key not in keysPressed:

	display.add(cSharpDownIcon, 45, 1)	# "press" this piano key

	Play.noteOn(CS4)	# play corresponding note

	keysPressed.append(VK_S)	# avoid key-repeat

	elif key == VK_X and key not in keysPressed:

	display.add(dDownIcon, 76, 1)	# "press" this piano key

	Play.noteOn(D4)	# play corresponding note

	keysPressed.append(VK_X)	# avoid key-repeat

	#...continue adding elif's for additional piano keys

def endNote(key):

	"""This function will be called when a computer key is released.

	It stops the corresponding note.

	"""

	global display	# display surface to add icons

	global keysPressed	# list to remember which keys are pressed

	if key = = VK_Z:

	display.remove(cDownIcon)	# "release" this piano key

	Play.noteOff(C4)	# stop corresponding note

	keysPressed.remove(VK_Z)	# and forget key

	elif key = = VK_S:

	display.remove(cSharpDownIcon)	# "release" this piano key

	Play.noteOff(CS4)	# stop corresponding note

	keysPressed.remove(VK_S)	# and forget key

	elif key = = VK_X:

	display.remove(dDownIcon)	# "release" this piano key

	Play.noteOff(D4)	# stop corresponding note

	keysPressed.remove(VK_X)	# and forget key

	#...continue adding elif's for additional piano keys

#####################################################################

# associate callback functions with GUI events

display.onKeyDown(beginNote)

display.onKeyUp(endNote)

Let’s explore this code. First, the following statement

Play.setInstrument(PIANO) # set desired MIDI instrument (0-127)

sets the timbre generated by this instrument.

Then the following statements

# load piano image and create display with appropriate size

pi anoIcon = Icon("iPianoOctave.png") # image for complete piano

di splay = Display("iPiano", pianoIcon.getWidth(),

	pianoIcon.getHeight())

load the full piano image (actually one octave), create a display with the exact size (given the image dimensions), and place the image on it.‡‡‡

Now that we have the full (one-octave) keyboard shown the display, we can create the illusion of different keys getting pressed by superimposing (i.e., adding) various images of pressed keys onto the display. So, next we load those images, one image for each key we want pressed. The following code demonstrates how to load such images. It is left as an exercise for you to complete it.

# load icons for pressed piano keys

cDownIcon = Icon("iPianoWhiteLeftDown.png")	# C

cS harpDownIcon = Icon("iPianoBlackDown.png")	# C sharp

dDownIcon = Icon("iPianoWhiteCenterDown.png")	# D

#...continue loading icons for additional piano keys

Next we create a list to remember which keys are pressed. This is needed because when we keep a key pressed, most computer keyboards initiate the key-repeat function, i.e., they type multiple instances of the same character. (While this is convenient when we type, say, an email, it is quite disturbing when trying to play a piano; we do not want to get several note repeats from a single key press. For this reason, when a key is pressed, we add it to the keysPressed list. We remove it when the key is released. This way we can differentiate between a new key press and one that was generated by the computer keyboard’s key-repeat functionality.

# remember which keys are currently pressed

keysPressed = []

Next we define the two callback functions to be used when a key is pressed and when a key is released, respectively.

The first callback function, beginNote(), is assigned to the display’s onKeyDown event handler. As mentioned above, since computer keyboards usually have a key-repeat action (i.e., if you keep pressing a key, it repeats), this function will be called repeatedly when a key is held pressed. This would have the effect of restarting the corresponding note, which is undesirable. For this reason, we use the keysPressed list to remember which keys are currently being pressed, so that when the key-repeat action causes repeated calls of beginNote() for the same key press, we can ignore them.

This function associates a given computer key with a piano key, using the following code:

if key == VK_Z and key not in keysPressed:

	display.add(cDownIcon, 0, 1)	# "press" this piano key

	Play.noteOn(C4)	 # play corresponding note

	keysPressed.append(VK_Z)	 # avoid key-repeat

In the above case, if the key pressed is “Z” (i.e., virtual key VK_Z)§§§:

• We superimpose the proper key-pressed image (i.e., image cDownIcon) at the proper place on the display (i.e., x, y coordinates 0, 1).
• We start playing the proper note (i.e., C4).
• Finally, we add this virtual key to the keysPressed list. This way, when the key-repeat action starts and function beginNote() is called again with the same virtual key, the if condition (i.e., key not in keysPressed) will prevent the code from executing again.

Similarly, when the key is released, the endNote() callback function is called. This function associates a given computer key with a piano key, using the following code:

if key == VK_Z:

	display.remove(cDownIcon)	# "release" this piano key

	Play.noteOff(C4)	# stop corresponding note

	keysPressed.remove(VK_Z)	# and forget key

In the above case, if the key pressed is “Z” (i.e., virtual key VK_Z):

• We remove the pressed-down image (so that it creates the illusion that the piano key has been released).
• We stop the corresponding note from sounding.
• Finally, we remove the released key from the keysPressed list, so that if it is pressed again, we can act on it.

The last part of the code associates the two callback functions with the corresponding keyboard events on the display:

# associate callback functions with GUI events

display.onKeyDown(beginNote)

display.onKeyUp(endNote)

8.14.1 Exercise

Complete the functionality of the above program. In other words, make the remaining piano keys (F sharp, G, G sharp, A, A sharp, and B) functional. Keep in mind that, if we wish to use the same key-pressed image for several keys, we need to load it several times (i.e., once for each key we want to use it with). This allows us to have several keys appear to be pressed simultaneously.

Note: To improve typing accuracy, some computer keyboards do not allow certain keys to be typed together, such as Z, S, and X. Therefore, you may notice that your virtual piano has the same limitation.

8.14.2 A Variation, Using Parallel Lists

As you work through the above exercise, you will soon realize the repetitive nature of the task. For each of the computer keys you wish to associate with a virtual piano key, you have to add a special elif case in each of the two callback functions (in addition to loading the corresponding key-pressed icon). This can take some time to implement.

The following variation takes advantage of the repetitive nature of this task. It introduces parallel lists to hold related information:

• downKeyIcons holds the icons corresponding to “pressed” piano keys,
• virtualKeys holds the virtual keys (e.g., VK_Z, etc.) corresponding to the above piano keys (pressing these keys on the computer keyboard “presses” the corresponding piano keys),
• pitches holds the note pitches corresponding to the above (piano) keys, and
• iconWidths holds the X coordinate of each “pressed” piano key icon, so it perfectly aligns with the underlying “unpressed” piano octave icon.

These lists can be extended to support more keys (currently, only 6 keys are functional).

# iPianoParallel.py

#

# Demonstrates how to build a simple piano instrument playable

# through the computer keyboard.

#

from music import *

from gui import *

Play.setInstrument(PIANO)	# set desired MIDI instrument (0–127)

# load piano image and create display with appropriate size

pianoIcon = Icon("iPianoOctave.png")	# image for complete piano

d = Display("iPiano", pianoIcon.getWidth(), pianoIcon.getHeight())

d.add(pianoIcon)	# place image at top-left corner

# NOTE: The following loads a partial list of icons for pressed piano

#	keys, and associates them (via parallel lists) with the 

# virtual keys corresponding to those piano keys and the corresponding

# pitches. These lists should be expanded to cover the whole octave

# (or more).

# load icons for pressed piano keys 

# (continue loading icons for additional piano keys)

downKeyIcons = [] # holds all down piano-key icons

downKeyIcons.append(Icon("iPianoWhiteLeftDown.png"))	# C 

downKeyIcons.append(Icon("iPianoBlackDown.png"))	# C sharp

downKeyIcons.append(Icon("iPianoWhiteCenterDown.png"))	# D

downKeyIcons.append(Icon("iPianoBlackDown.png"))	# D sharp

downKeyIcons.append(Icon("iPianoWhiteRightDown.png"))	# E

downKeyIcons.append(Icon("iPianoWhiteLeftDown.png"))	# F

# lists of virtual keys and pitches corresponding to above piano keys

virtualKeys	= [VK_Z, VK_S, VK_X, VK_D, VK_C, VK_V]

pitches	= [C4, CS4, D4, DS4, E4, F4]

# create list of display positions for downKey icons

#

# NOTE: This as hardcoded – they depend on the used images!

#

iconLeftXCoordinates = [0, 45, 76, 138, 150, 223]

keysPressed = [] # holds which keys are currently pressed

#####################################################################

# define callback functions

def beginNote(key):

	"""Called when a computer key is pressed. Implements the 

	corresponding piano key press (i.e., adds key-down icon on 

	display, and starts note). Also, counteracts the key-repeat 

	function of computer keyboards.

	"""

	# loop through all known virtual keys

	for i in range(len(virtualKeys)):

	# if this is a known key (and NOT already pressed)

	if key == virtualKeys[i] and key not in keysPressed: 

	# "press" this piano key (by adding pressed key icon)

	d.add(downKeyIcons[i], iconLeftXCoordinates[i], 0)

	Play.noteOn(pitches[i])	# play corresponding note

	keysPressed.append(key)	# avoid key-repeat

def endNote(key):

	"""Called when a computer key is released. Implements the 

	corresponding piano key release (i.e., removes key-down icon, 

	and stops note).

	"""

	# loop through known virtual keys

	for i in range(len(virtualKeys)):

	# if this is a known key (we can assume it is already pressed)

	if key == virtualKeys[i]:

	# "release" this piano key (by removing pressed key icon)

	d.remove(downKeyIcons[i])

	Play.noteOff(pitches[i])	# stop corresponding note

	keysPressed.remove(key)	# and forget key

#####################################################################

# associate callback functions with GUI events

d.onKeyDown(beginNote)

d.onKeyUp(endNote)

This revision demonstrates that for most programming tasks there may be different algorithms that accomplish the same task. In this case, the first variation is simpler to understand, but more lengthy, repetitious, and thus error-prone. The second variation is more elegant and easier to extend—to add more functioning keys to the virtual piano you simply add more items to the parallel lists at the top of the program.

Notice the use of for loops in the two callback functions to iterate through the known virtual keys (virtualKeys list) to see if we need to act on the key generating the keyboard event. For example,

# loop through all known virtual keys

for i in range(len(virtualKeys)):

	# if this is a known key (and NOT already pressed)

	if key == virtualKeys[i] and key not in keysPressed:

	# "press" this piano key (by adding pressed key icon)

	d.add(downKeyIcons[i], iconLeftXCoordinates[i], 0)

	Play.noteOn(pitches[i])	# play corresponding note

	keysPressed.append(key)	# avoid key-repeat

Notice how we use the index of the virtual key, i, to access the corresponding information in the parallel lists. This replaces the repetitive sequence of if/elif statements for each of the known virtual keys seen in the original program.

8.14.2.1 Exercises

1. Complete the functionality of the above program. In other words, make the remaining piano keys (F sharp, G, G sharp, A, A sharp, and B) functional. Keep in mind that, if we wish to use the same key-pressed image for several keys, we need to load it several times (i.e., once for each key we want to use it with). This allows us to have several keys appearing to be pressed simultaneously.
2. Extend the above program to display a two octave piano. (Hint: Load the “iPianoOctave.png” image twice and place the second instance at the end of the first instance. Use the icon getWidth() function to make this more general.)
3. Create a second display with an additional piano. Figure out which computer keyboard keys to assign to which piano display and notes. Assign a harpsichord timbre and pick registers accordingly to create a four octave, “two-decker” harpsichord instrument.
4. Create your own GUI version of audio mixer slider (e.g., mixer master volume). It should consist of two images, that is, the base of the slider and the knob. The knob should be moved by dragging the mouse. (Hint: Use the icon’s onMouseDrag() function.) Constrain the knob’s movement to be only vertical (i.e., ignore the mouse x coordinate); also constrain the vertical movement to begin and end using the base image’s position and height.
5. Once your GUI version of the audio mixer slider is ready (see above), use it to control the volume of an AudioSample. (Hint: Similarly to the example in the Mouse Events section above, include all necessary code inside the callback function associated with slider’s onMouseDrag event.)
6. Design a collection of GUI control knobs, ribbons, and related components to be used in building more advanced surfaces for interactive musical instruments. This collection can be easily implemented using Python classes (presented in the next section).

8.15 Scheduling Future Events

So far we have seen how to capture user input through graphical user interfaces to drive our program’s behavior, that is, event-driven programming.

Sometimes it is also useful to schedule computer-initiated (as opposed to user-initiated) events to happen sometime in the future. This allows us, among other things, to build buttons that temporarily change color (or shape) when pressed, etc. To do so, we simply change the color (or shape) of a graphical object when it is clicked on and then schedule another change in color (or shape) to occur a fraction of a second later.

This is accomplished using Timer objects. Timer objects are given a function to call, and a delay specifying after how much time to call it.¶¶¶ The next case study demonstrates how to do this.

8.15.1 Case Study: Random Circles with Timer

This case study presents a generative music application consisting of circles drawn randomly onto a display.

• Every circle is connected to a note— as the circle is drawn, a pitch is sounded.
• The color of the circle determines the pitch of the note—the redder the color, the lower the note.
• The circle’s size (radius) determines the volume of the note—the bigger the radius, the louder the note.

To make the music aesthetically pleasing, pitches are selected from the major scale. This concept is inspired by Brian Eno’s “Bloom” musical app for smartphones.

Circles are initially drawn once per 0.5 seconds (500 milliseconds), or at a rate of 2 circles per second. This rate works quite well with adding circles to a display, possibly reminding the viewer of drops of rain. Additionally, the program presents a secondary display (see Figure 8.10), which allows the user to control the delay between successive drops (i.e., circle-notes).

Here is the code:

# randomCirclesTimed.py

#

# Demonstrates how to generate a musical animation by drawing random

# circles on a GUI display using a timer. Each circle generates

# a note - the redder the color, the lower the pitch; also,

# the larger the radius, the louder the note. Note pitches come

# from the major scale.

#

from gui import *

from random import *

from music import *

delay = 500 # initial delay between successive circle/notes

##### create display on which to draw circles #####

d = Display("Random Timed Circles with Sound")

# define callback function for timer

def drawCircle():

	"""Draws one random circle and plays the corresponding note."""

	global d	# we will access the display

	x = randint(0, d.getWidth())	# x may be anywhere on display

	y = randint(0, d.getHeight())	# y may be anywhere on display

	radius = randint(5, 40)	# random radius (5-40 pixels)

	# create a red-to-brown-to-blue gradient (RGB)

	red = randint(100, 255)	# random R component (100-255)

	blue = randint(0, 100)	# random B component (0-100)

	color = Color(red, 0, blue)	# create color (green is 0)

	c = Circle(x, y, radius, color, True)	# create filled circle

	d.add(c)	# add it to the display

	# now, let's create note based on this circle

	# the redder the color, the lower the pitch (using major scale)

	pitch = mapScale(255-red+blue, 0, 255, C4, C6, MAJOR_SCALE)

	# the larger the circle, the louder the note

	dynamic = mapValue(radius, 5, 40, 20, 127)

	# and play note (start immediately, hold for 5 secs)

	Play.note(pitch, 0, 5000, dynamic)

# create timer for animation

t = Timer(delay, drawCircle)	# one circle per 'delay' milliseconds

##### create display with slider for user input #####

title = "Delay"

xPosition = d.getWidth()/3 # set initial position of display

yPosition = d.getHeight() + 45

d1 = Display(title, 250, 50, xPosition, yPosition)

# define callback function for slider

def timerSet(value):

	global t, d1, title  # we will access these variables

	t.setDelay(value)

	d1.setTitle(title + "(" + str(value) + "msec)")

# create slider

s1 = Slider(HORIZONTAL, 10, delay*2, delay, timerSet)

d1.add(s1, 25, 10)

# everything is ready, so start animation (i.e., start timer)

t.start()

First, notice the delay variable used to initialize the initial delay between successive circles.

delay = 500 # initial delay between successive circle/notes

The next section of the code creates the main display, d, and a callback function, called drawCircle(). This function will be called by the Timer object repeatedly, every delay (e.g., 500) milliseconds, to draw a new circle and play a note.

The Timer object is created as follows:

# create timer for animation

t = Timer(delay, drawCircle) # one circle per 'delay' milliseconds

As seen in the next section, the Timer function expects a delay and a callback function. It creates a Timer object which automatically calls this function every delay milliseconds.

A program may have several timers going on at once. Timers may be started (so that they do their intended function) or stopped. Also, we may change their delay time. Actually, the rest of the program does precisely that, that is, it creates a secondary display consisting of a slider, which allows the end-user to change the timer’s delay.

##### create control surface #####

title = "Delay"

xP osition = d.getWidth()/3 # set initial position of display

yPosition = d.getHeight() + 45

d1 = Display(title, 250, 50, xPosition, yPosition)

Notice the positioning of this secondary display. It is placed below the main display, relative to the main display’s the width (i.e., xPosition = d.getWidth()/3) and height (i.e., yPosition = d.getHeight() + 45).

The slider is created with a callback function, timerSet(), which expects the value of the slider and uses this value to set the Timer object’s delay, as well as the title of the secondary display.

# define callback function for slider

def timerSet(value):

global t, d1, title

t.setDelay(value)

d1.setTitle(title + "(" + str(value) + " msec)")

# create slider

s1 = Slider(HORIZONTAL, 10, delay*2, delay, timerSet)

d1.add(s1, 25, 10)

Notice how the slider is created to range from 10 milliseconds to twice the initial delay that may be set at the beginning of the program, and with an initial value equal to delay. Remember that, as mentioned in the previous chapter, every time the user moves the slider button the system calls the provided (callback) function, timerSet(), passing to it the slider’s new value.

Finally, in order for the application to come alive, we need to start the timer:

# everything is ready, so start animation (i.e., start timer)

t.start()

The following section provides more information on creating and managing Timer objects.

8.15.2 The Timer Class

The GUI library supports scheduling future tasks through the Timer class. Timer objects are used to schedule when and how often to perform a certain task (i.e., to call a given function). These tasks are not confined to graphical events; as demonstrated in the previous case study, they can be any computer-generated task, including musical tasks.

8.15.2.1 Creating Timers

Timer objects are used to schedule functions to be executed after a given time interval, repeatedly or once. The following function creates a new Timer, so you need to save it in a variable (so you can use it later).

Function	Description
Timer(delay, function, parameters, repeat)	Creates a new Timer to execute function after delay time interval (in milliseconds). The optional parameter parameters is a list of parameters to pass to the function (when called). The optional parameter repeat (boolean—default is True) determines if the timer will go on indefinitely.

For example, the following:

t = Timer(500, Play.noteOn, [A4], True)

creates a Timer t, which will call function Play.noteOn(A4) repeatedly every 500 milliseconds (i.e., half second). In order for a timer to operate, it needs to get started:

t.start()

Once a Timer t has been created, the following functions are available:

Function	Description
t.start()	Starts timer t.
t.stop()	Stops timer t.
t.getDelay()	Returns the delay time interval of timer t (in milliseconds).
t.setDelay(delay)	Sets a new delay time interval for timer t (in milliseconds). This allows us to change the speed of the animation, after some event occurs.
t.isRunning(delay)	Returns True if timer t is running (has been started), False otherwise.
t.setFunction(function, parameters)	Sets the function to execute. The optional parameter parameters is a list of parameters to pass to the function (when called).
t.getRepeat()	Returns True if timer t is set to repeat, False otherwise.
t.setRepeat(flag)	If flag is True, timer t is set to repeat (this also starts the timer, if stopped). Otherwise, if flag is False, timer t is set to not repeat (this stops the timer, if running).

8.16 Summary

This chapter focused on the design and development of graphical user interfaces (GUIs) and interactive software instruments. We discussed a number of new real-time musical functions, including Play.noteOn() and Play.noteOff(), as well as how to load in audio samples and MIDI sequences and make them part of our interactive software. We also saw the GUI library which opens new interaction possibilities. We talked about paper prototypes and how industry experts use them extensively (along with similar GUI development methodologies) while developing interactive software. Although starting with code might feel like the right thing to do, in the case of GUIs, a little paper prototyping goes a long way. We also saw how the various GUI widgets make the instrument interfaces come alive with visual animations synchronized to user input and sounds. Finally, we presented the Timer class, which enables the scheduling of functions to be executed in the future. This sets the stage for some power possibilities that will be explored further in the remaining chapters.

---