Chapter 7

Sonification and Big Data

Topics: Data sonification, mapValue() and mapScale(), Kepler, Python strings, music from text, Guido d’Arezzo, nested loops, file input/output, Python while loop, big data, biosignal sonification, defining functions, image sonification, Python images, visual soundscapes.

7.1 Overview

According to Scaletti (1993), “the idea of representing data in sound is an ancient one. For the ancient Greeks music was not an art-for-art’s sake, practiced in a vacuum, but a manifestation of the same ratios and relationships as those found in geometry or in the positions and behaviors of the planets.” Pythagoras, Plato, and Aristotle worked on quantitative expressions of proportion and beauty, such as the golden ratio. Pythagoreans, for instance, quantified harmonious musical intervals in terms of proportions (ratios) of the numbers 1, 2, 3, 4, and 5 (as we discussed earlier in Chapter 1). This became the basis for the scales, modes, and tuning systems used in Western music.

As Johannes Kepler (among others) observed, the universe is flowing and fluctuating at different time scales and size scales. We only hear a small range of these fluctuations as sound, namely, when they vibrate at frequencies between 20 and 20,000 times per second (Hertz or Hz). Similarly, we only see (a different) small range of these vibrations as visible light (red through purple). The limits of our senses make it difficult for us to fully appreciate how our cosmos is put together in harmonically (hence musically) meaningful ways. It is this harmony in the proportions and organizations of the parts that make our universe stay together, allowing us to be in it and experience it. It is this harmony that Kepler studied and tried to formalize and communicate through his three laws of planetary motion. Kepler also tried to map the harmony he observed in the astronomical data available at his time as sounds, that is, to sonify them (Kepler 1619).

Sonification allows us to capture and better experience phenomena that are outside our sensory range by mapping values into sound structures that we can perceive by listening to them. Data for sonification may come from any measurable vibration or fluctuation, such as planetary orbits, magnitudes of earthquakes, positions of branches on a tree, lengths of words in this chapter, and so on.

In addition to facilitating insight and analysis, sonification can also be used to make interesting music and sound, just like visualization of data can make interesting images. If music can involve the imitation of nature (mimesis), then sonification is a valuable tool for modern music making.

7.2 Data Sonification

When dealing with big data (i.e., large amounts of data), sometimes it is easier to hear patterns by turning the data into sound, as opposed to looking at, say, thousands or millions of numbers.

Definition: Data sonification involves mapping numbers to musical parameters, such as pitch, duration, volume, pan position, and timbre, in order to better perceive patterns in the data and appreciate their characteristics.

A clear and simple example of sonification is the Geiger counter, a device which measures radiation and turns this measurement to audible clicks. The number (frequency) of clicks is directly related to the radiation level in the vicinity of the device.

Another example of sonification is the biosignal monitor that we see at hospitals. These are the devices that monitor signals measured from the human body, such as heart rate. These monitors (similarly to the Geiger counter) provide constant sonic biofeedback, allowing the human operator to focus on other tasks.

7.2.1 The mapValue() Function

Mapping a value from one range to another is very common in sonification. For instance, you may want to map a list of numbers (say, from 20.0 to 110.0) to pitch values (say, from 30 to 90). This becomes more realistic if, say, you are interested in global warming and want to explore how temperatures change over time. By converting temperatures to pitch values, you can actually hear these changes.

The music library provides the mapValue() function precisely for this task. This function accepts the following arguments:

• value—the number to be mapped
• minValue—the lowest possible number to be mapped
• maxValue—the highest possible number to be mapped
• minResult—the lowest value of the destination range
• maxResult—the highest value of the destination range

For example,

>>> from music import * # mapValue() is defined in this library

>>> mapValue(24, 0, 100, 32, 212) # celsius to fahrenheit

75

This converts 24°C to 75°F. To verify that this conversion really works, let us try converting the extremes:

>>> mapValue(0, 0, 100, 32, 212) # lowest temperature

32

So, when we give the lowest temperature in the temperature range (i.e., 0), mapValue() returns the lowest pitch in the destination range (i.e., 32).

>>> mapValue(100, 0, 100, 32, 212) # highest temperature

212

Also, when we give the highest temperature in the temperature range (i.e., 100), mapValue() returns the highest pitch in the destination range (i.e., 212).

>>> mapValue(56.7, 0.0, 100.0, 32.0, 212.0) # arbitrary temperature

134.06

Using float numbers returns float numbers.

Having tested this, we can assume that the rest of the values are mapped linearly (as if we took two number lines, one from 0 to 100 and another from 32 to 212, and superimposed them).

A useful feature is that mapValue() will return a value using the data type of the destination range. So, for example, providing float (real) values in the destination range generates a float result:

>>> mapValue(56.7, 0.0, 100.0, 32.0, 212.0)

134.06

7.2.2 The mapScale() Function

The mapScale() function is similar to mapValue(), expect that it has an additional 6th argument:

• scale—(optional) the musical scale to be used in the destination range*

This function quantizes (sieves) the numeric results to fit pitches within the provided scale. The key of the scale is determined by minResult. For example, any C pitch (e.g., C4 (60), C5 (72), etc.) corresponds to the key of C any D pitch corresponds to the key of D and so on.

Accordingly, mapScale() returns integer values, since they are meant to be used as pitches.

The difference between mapValue() and mapScale() is demonstrated as follows. Note that the first examples use duration constants, demonstrating another useful musical application for mapValue().

>>> from music import *

>>> from random import *

>>> mapValue(0.6, 0, 1, TN, WN)

2.4499999999999997

>>> mapValue(0.999, 0.0, 1.0, TN, WN)

3.996125

>>> mapScale(0.5, 0, 1, 0, 127, MAJOR_SCALE)

64

>>> mapScale(0.66, 0, 1, 40, 80, MINOR_SCALE)

66

Whereas mapValue() returns a float value with lots of accuracy, mapScale() returns an integer.

Here are some examples using a provided scale:

>>> for i in range(0, 13):

print i, "—>", mapScale(i, 0, 12, 0, 12, MAJOR_SCALE)

0 —> 0

1 —> 0

2 —> 2

3 —> 2

4 —> 4

5 —> 4

6 —> 5

7 —> 7

8 —> 7

9 —> 9

10 —> 9

11 —> 11

12 —> 12

Here we can clearly see how the result is adjusted to fit the pitches in the C major scale. We know the scale is in C from the minResult (4th argument), which is 0. Pitch 0 is a C pitch, namely, C_1.†

Constraining pitches to a scale allows us to generate more aesthetically pleasing sonifications.

Finally, since scales are simply lists of pitch classes (between 0 and 11),‡ for example,

>>> MAJOR_SCALE

[0, 2, 4, 5, 7, 9, 11]

>>> MINOR_SCALE

[0, 2, 3, 5, 7, 8, 10]

>>>

it is possible to use your own list, such as [0, 5, 8]. The only constraint is that the numbers range from 0 to 11. So, for example,

>>> mapScale(97.5, 20, 110, 30, 90, [0, 5, 8])

80

Here the scale has pitches C (0), F (5), and G sharp (8). If we had used mapValue(), the result would have been 81.7, but rounding to the nearest C, F, or G sharp gives us GS5, which is 80.

To understand better how mapScale() works with scales, try different scales with the following loop:

for i in range(22):

	print mapScale(i, 0, 22, C3, C6, scale)

where scale is a particular scale or pitch class list.

7.3 Case Study: Kepler—“Harmonies of the World” (1619)

In 1619 Johannes Kepler wrote his “Harmonices Mundi (Harmonies of the World)” book (Kepler 1619). While Pythagoreans only talked about the “music of the spheres,” Kepler discovered physical harmonies in planetary motion. As a result, he became a key figure in the development of astronomy and modern physics.

Following Kepler’s studies, in the late 1700s Johann Daniel Titius and Johann Elert Bode independently contributed to a model of the symmetries and proportions of our solar system. Their formula, known as the Titius–Bode law (or simply Bode’s law), predicts the positions of the planets in our solar system. Actually, it also predicted the asteroid belt between Mars and Jupiter (long before it was discovered), but fail to account for the irregularly moving Neptune and the (now demoted non-planet) Pluto.

The following program, harmonicesMundi.py, sonifies one aspect of the celestial organization of planets. In particular, it converts the orbital velocities of the planets to musical notes.

The planets’ mean orbital velocities (in kilometers per second) are as follows:

We will map this range of velocities to a range of MIDI pitches, say, C1 to C6. To do this mapping, we can use the mapScale() function.

# harmonicesMundi.py

#

# Sonify mean planetary velocities in the solar system.

#

from music import *

# create a list of planet mean orbital velocities

# Mercury, Venus, Earth, Mars, Ceres, Jupiter, 

# Saturn, Uranus, Neptune, Pluto

planetVelocities = [47.89, 35.03, 29.79, 24.13, 17.882, 13.06, 

	9.64, 6.81, 5.43, 4.74]

# get minimum and maximum velocities:

minVelocity = min(planetVelocities)

maxVelocity = max(planetVelocities)

# calculate pitches

planetPitches = []	# holds list of sonified velocities

planetDurations = []	# holds list of durations

for velocity in planetVelocities:

	# map a velocity to pitch and save it

	pitch = mapScale(velocity, minVelocity, maxVelocity, C1, C6, CHROMATIC_SCALE)

	planetPitches.append(pitch)

	planetDurations.append(EN)	# for now, keep duration fixed

# create the planet melodies

melody1 = Phrase(0.0)	# starts at beginning

melody2 = Phrase(10.0)	# starts 10 beats into the piece

melody3 = Phrase(20.0)	# starts 20 beats into the piece

# create melody 1 (theme)

melody1.addNoteList(planetPitches, planetDurations)

# melody 2 starts 10 beats into the piece and

# is elongated by a factor of 2

melody2 = melody1.copy()

melody2.setStartTime(10.0)

Mod.elongate(melody2, 2.0)

# melody 3 starts 20 beats into the piece and

# is elongated by a factor of 4

melody3 = melody1.copy()

melody3.setStartTime(20.0)

Mod.elongate(melody3, 4.0)

# rep eat melodies appropriate times, so they will end together

Mod.repeat(melody1, 8)

Mod.repeat(melody2, 3)

# create parts with different instruments and add melodies

part1 = Part("Eighth Notes", PIANO, 0)

part2 = Part("Quarter Notes", FLUTE, 1)

part3 = Part("Half Notes", TRUMPET, 3)

part1.addPhrase(melody1)

part2.addPhrase(melody2)

part3.addPhrase(melody3)

# finally, create, view, and write the score

score = Score("Celestial Canon")

score.addPart(part1)

score.addPart(part2)

score.addPart(part3)

View.sketch(score)

Play.midi(score)

Write.midi(score, "harmonicesMundi.mid")

Notice how we first create the list of planetary velocities. For this case study, we also include Ceres (i.e., the asteroid belt) and (the recently demoted non-planet) Pluto.

planetVelocities = [47.89, 35.03, 29.79, 24.13, 17.882, 13.06, 

	9.64, 6.81, 5.43, 4.74]

Notice how we use the list functions, min() and max(), to find the minimum and maximum velocity values.

minVelocity = min(planetVelocities)

maxVelocity = max(planetVelocities)

We could have done this by hand by scanning the list ourselves and hard-coded the min and max numbers in the program; however, this use of “magic numbers” (as they are called by seasoned software developers) can get us into trouble.

Good Style: Avoid using magic numbers in your programs.

Definition: A magic number is a number in our programs that seems to come out of nowhere, that is, it is not derived automatically (calculated) from the program’s data.

Even if we document such magic numbers profusely (a minimum requirement if we have to use them), they become a place of future error, as the program evolves (and most programs do evolve). So as to avoid using magic numbers in the preceding code, we added code to calculate min and max from the list of velocities. Thus, if the list ever changes (or it is read from a data file, as we will see soon), min() and max() will continue to do the right thing. These functions, together with mapValue(), are very useful in sonification tasks.

Notice how mapScale() is utilized to sonify the orbital velocities. For example, assume that variable velocity contains the orbital velocity of a planet, and variables minVelocity and maxVelocity contain the smallest and largest of all planet velocities, respectively. Then this statement

pitch = mapScale(velocity, minVelocity, maxVelocity, C1, C6, CHROMATIC_SCALE)

generates a pitch value positioned in the range C1 to C6 similarly to how velocity is positioned in the range minVelocity to maxVelocity.

Next we use a for loop to map each velocity in planetVelocities to the corresponding pitch (in the range 24 to 84). Notice how we create an empty list ahead of time (outside of the loop) and use the list operation append() to add each new pitch value, as it is created, to the end of the pitches list.

# calculate pitches

planetPitches = []	# holds list of sonified velocities

for velocity in planetVelocities:

 # map a velocity to pitch and save it

 pitch = mapValue(velocity, minVelocity, maxVelocity, C1, C6)

 planetPitches.append(pitch)

Now the list planetPitches is created and contains one pitch per planet. Notice how it is parallel to the list planetVelocities—for every planet velocity there is now a pitch, in the corresponding list positions.

Finally, we are ready to sonify. What follows is similar to the types of things we have seen in earlier chapters. We have many options for how we can sonify these data. One possibility is to just play the melody generated by the pitches. Another possibility would be to create a musical piece using the above melody as a component. For instance, we might use the original melody as a bass line, add chords, drums, and additional musical material.

Here, in the spirit of J.S. Bach and Arvo Pärt, we build a canon from the sonified orbital velocities. To do this, we treat the melody as the theme and use canonic devices (seen in Chapter 4) to create a celestial canon. We choose to play the melody concurrently, against itself, using different durations (see Figure 7.1). This is similar to Arvo Pärt’s musical structure for Cantus in Memoriam (see case study in Chapter 4).

The preceding code implements this canon structure. Notice how we create three different melodies, melody1, melody2, and melody3, to contain the same pitch list (the one sonified earlier) but with different durations, adding up to 5 beats, 10 beats, and 20 beats, respectively. Then we adjust the start time of each melody (i.e., start at 0.0, 10.0, and 20.0), so that they are introduced incrementally, but end together.

7.3.1 Exercise

Modify the preceding sonification to use a different musical structure; there are many possibilities. Aim for something aesthetically pleasing. Consider using different scales with function mapScale(). Experiment with different scales, including your own pitch sets.

Search the Internet for interesting astronomical data, download them, and sonify them using a similar approach to the one explored here. For musical inspiration, listen to Olivier Messiaen’s “Mode de valeurs et d’intensités” for piano (1949), in which he creates musical material in a systematic way focusing on durations and intensities.

7.4 Python Strings

In Chapter 2, we discussed how Python uses strings to represent text data. String is another Python data type, such as int, float, list, and boolean. A string contains a sequence of characters. Strings are enclosed in quotes.

Definition: A Python string consists of zero or more characters, enclosed in quotes.

There are three types of quotes that may be used, single quotes (‘ ’), double quotes (“”), or three double quotes (“““”””). For example,

'This is a string - 1, 2, 3.'

Here is the same string using double quotes:

"This is a string - 1, 2, 3."

And again using three double quotes:

"""This is a string - 1, 2, 3."""

You can use any one of these three representations. Most programmers prefer the first two, since clearly they are most economical. The only reason to use the three double quotes is that such strings may contain newlines, that is, they can span several lines. This is not the case with the other quotes, for example,

"""This is a

string - 1,

2, 3."""

Strings are similar to lists in that they are sequences. As a result, everything we learned about lists, in terms of indexing, applies to strings as well. For example, normal indexing and slice operations apply:

>>> s = "This a string - 1, 2, 3."

>>> s[0]

'T'

>>> s[1]

'h'

>>> s[1:4]

'his'

>>> s[1:4] == s[10:13]

False

>>> s[2] == s[10]

True

>>> s[2]

'i'

Also, we can iterate through strings like we did with lists:

>>> s1 = "Hello!"

>>> for character in s1:

...  print character

...

H

e

l

l

o

!

In the last example, what would happen if we put a comma at the end of the print statement? Why? §

String characters are represented internally using numbers, according to the ASCII standard.¶ The complete listing of the ASCII characters (and the numbers used to represent them) is easily available on the Internet. Most of the time we can ignore this. However, sometimes knowing the internal representation can be useful.

In Python, we can use function ord() which, given a string with a single character as argument, returns the corresponding ASCII number. For example,

>>> ord('A')

65

>>> ord('a')

97

>>> ord(' ')

32

>>> ord('!')

33

>>> ord('0')

48

One thing to remember is that not all ASCII characters are printable, that is, some of them are used for formatting (e.g., newline character, tab, backspace, etc.). The printable ASCII characters range from space (ASCII value 32) to the tilde character, “~” (ASCII 126).

Having access to the ASCII numbers makes it straightforward to generate music from text. It is not that different from generating music from planetary orbital velocities, as we see in the following text.

7.4.1 Case Study: Music from Text

A simple way to algorithmically dictate musical structures is to follow some existing data patterns. One source of data patterns is astronomical data, as we saw earlier. Another source of patterns is natural language (e.g., English). Since languages have inherent structure—as described by Noam Chomsky (1957) and George K. Zipf (1949)—it is reasonable to expect that music based on text might maintain some of the expressiveness inherent in this structure. The sonification of text can, similar to all sonifications, use simple or complex mappings between the text and sound.

The following program, textMusic.py, demonstrates how to generate music from text. Using the ord() Python built-in function, this program converts the values of ASCII characters to MIDI pitches. For variety, note durations are randomized; other note properties (volume, etc.) are the same for all notes.

# textMusic.py

#

# It demonstrates how to generate music from text.

# It converts the values of ASCII characters to MIDI pitch.

# Note duration is picked randomly from a weighted-probability list.

# All other music parameters (volume, panoramic, instrument, etc.)

# are kept constant.

#

from music import *

from random import *

# Define text to sonify.

# Excerpt from Herman Melville's "Moby-Dick", Epilogue (1851)

text = """The drama's done. Why then here does any one step forth? - Because one did survive the wreck. """

##### define the data structure

textMusicScore = Score("Moby-Dick melody", 130)

textMusicPart  = Part("Moby-Dick melody", GLOCK, 0)

textMusicPhrase = Phrase()

# create durations list (factors correspond to probability)

durations = [HN] + [QN]*4 + [EN]*4 + [SN]*2

##### create musical data

for character in text: # loop enough times

	value = ord(character) # convert character to ASCII number

	# map printable ASCII values to a pitch value

	pitch = mapScale(value, 32, 126, C3, C6, PENTATONIC_SCALE, C2)

	# map printable ASCII values to a duration value

	index = mapValue(value, 32, 126, 0, len(durations)-1)

	duration = durations[index]

	print "value", value, "becomes pitch", pitch, 

	print "and duration", duration 

	dynamic = randint(60, 120)	# get a random dynamic

	note = Note(pitch, duration, dynamic)  # create note

	textMusicPhrase.addNote(note) # and add it to phrase

# now, all characters have been converted to notes

# add ending note (same as last one - only longer)

note = Note(pitch, WN)

textMusicPhrase.addNote(note) 

##### combine musical material

textMusicPart.addPhrase(textMusicPhrase)

textMusicScore.addPart(textMusicPart)

##### view score and write it to a MIDI file

View.show(textMusicScore)

Play.midi(textMusicScore)

Write.midi(textMusicScore, "textMusic.mid")

Note that the string to sonify is at the top of the code. If you change this string, you will get different (yet similar music). Why? The music generated depends on the relative probabilities of characters in the English language (and not on the actual words, or, even further, the meaning of those words). It would be interesting to explore how to somehow map the meaning of words (or actual words) to note pitch. This would involve more work—beyond the scope of this chapter, for sure—but definitely something that could be explored using Python.

Also, notice how we use a for loop to iterate through every character in the string (stored in variable text). Actually, for loops work not only with lists, but also with any sequence, such as a string. (Remember that a string is a sequence of characters.) In the body of this loop, we get the ASCII number corresponding to each character and map it to a pitch.

ASCII characters are represented by integers from 0 to 127. However, in a regular text file, we could have mapped them directly to MIDI pitches (also from 0 to 127), that is,

pitch = ord(character)

But instead we opted for a more musical mapping utilizing the PENTATONIC_SCALE, that is,

pi tch = mapScale(value, 0, 127, C2, C6, PENTATONIC_SCALE)

Notice the print statement in the loop, which is commented out. It can be used to output the original value and the mapped pitch and duration values.

Good Style: Use strategically placed print statements (as above) to figure out what the code is doing (especially if things do not work as expected). Delete them after the code is ready.

Print statements are a programmer’s best friend. Use them extensively while you are developing your code. Then you can remove them (or comment them out, as above—they may come in handy again, after you have made a change in your code).

7.4.1.1 Exercise

Consider ways to make the earlier sonification more interesting.

• Perhaps by adding variety to parameters such as note lengths, dynamics, and panning.
• Consider how this variety might come from the text itself instead of simply harnessing randomness?
• What are some other text characteristics that we can use? (Hint: Some more ideas are introduced in the next case study.)

7.4.2 String Library Functions

Python provides a string library with useful string functions. Table 7.1 presents the most common ones (assume s is a string).

Here are some examples:

>>> from string import *

>>> s = 'hello world!'

>>> capitalize(s)

'Hello world!'

>>> upper(s)

'HELLO WORLD!'

>>> s

'hello world!'

Note that string functions do not modify the string provided as an argument; they just return a new string. For example, after the above, s still has its original value.

To modify s, we need to do this:

>>> s = capitalize(s)

>>> s

'Hello world!'

Here is another example,

>>> s = 'hello world!'

>>> find(s, 'e')

1

The result, 1, indicates that the first occurrence of substring “e” appears in position 1 (where “h” is at position 0).**

Finally, the following demonstrates functions split() and join():

>>> s = 'hello world!'

>>> s1 = split(s, ' ')

>>> s1

['hello', 'world!']

>>> s2 = join(sl, ' ')

>>> s2

'hello world!'

These are a few of the string functions available in the Python string library. They should be sufficient for most tasks related to music making. Feel free to explore more online.††

7.4.3 Case Study: Guido d’Arezzo—“Word Music” (ca. 1000)

One of the oldest known algorithmic music processes is a rule-based algorithm that selects each note based on the letters in a text, credited to Guido d’Arezzo (991–1033). Originally, the intention was that the melody was a sung phrase and the text was the lyric to be sung. Each vowel in the text is associated with a pitch. The duration of notes comes from the word length.

Although d’Arezzo’s original intention was simply to provide an approximate composition guide, here we formalize and automate these rules. This is an approximation to d’Arezzo’s algorithm, adapted to text written in ASCII and to modern musical sensibilities.

# guidoWordMusic.py

#

# Creates a melody from text using the following rules:

#

# 1) Vowels specify pentatonic pitch, 'a' is C4, 'e' is D4,

#  'i' is E4, 'o' is G4, and 'u' is A4.

#

# 2) Consonants extend the duration of the previous note (if any).

#

from music import *

from string import *

# this is the text to be sonified

text = """One of the oldest known algorithmic music processes is a rule-based algorithm that selects each note based on the letters in a text, credited to Guido d'Arezzo."""

text = lower(text)  # convert string to lowercase

# define vowels and corresponding pitches (parallel sequences),

# i.e., first vowel goes with first pitch, and so on.

vowels	= "aeiou"

vowelPitches	= [C4, D4, E4, G4, A4]

# define consonants

consonants = "bcdfghjklmnpqrstvwxyz"

# define parallel lists to hold pitches and durations

pitches  = []

durations = []

# factor used to scale durations

durationFactor = 0.1  # higher for longer durations

# separate text into words (using space as delimiter)

words = split(text)

# iterate through every word in the text

for word in words:

	# iterate through every character in this word

	for character in word:

	# is this character a vowel?

	if character in vowels:

	# yes, so find its position in the vowel list

	index = find(vowels, character)

	#print character, index

	# and use position to find the corresponding pitch

	pitch = vowelPitches[index] 

	# finally, remember this pitch

	pitches.append(pitch)

	# create duration from the word length

	duration = len(word) * durationFactor

	# and remember it

	durations.append(duration)

# now, pitches and durations have been created

# so, add them to a phrase

melody = Phrase()

melody.addNoteList(pitches, durations)

# view and play melody

View.notation(melody)

Play.midi(melody)

In this program, we import the string library to use functions lower(), split(), and find().

We use function lower() to convert to lowercase the characters in variable text. We do this, because we have defined rules in terms of lowercase vowels. If we did not convert text to lowercase we would need to duplicate rules for uppercase letters.

Notice how we define two parallel sequences (a string and a list). The first one, vowels, holds the vowels we wish to associate with notes. The second one, vowelPitches, holds the pitches associated with those vowels. Since both are indexed starting at 0, we can use them to associate vowels with pitches.‡‡ For example,

>>> vowels = "aeiou"

>>> vowelPitches = [C4, D4, E4, G4, A4]

>>> vowels[0]

'a'

>>> vowelPitches[0]

60		# same as C4

Also, notice how we define two parallel lists, pitches and durations, to store musical data for the notes.

7.4.4 Python Nested Loops

Loops may be nested. This kind of hierarchical processing (loops within loops) is quite common as a programming pattern, and we will see another example of it later in this chapter. It is common in programming because hierarchical structures are common in the world. As a geographic example of a hierarchy, consider how people live in suburbs, suburbs are in regions, regions are in states, states are in a country, and so on. This structure could be used as the basis for an algorithm to process details about all people in a particular geographic area.

Definition: Nested loops use two or more loops, one inside the other, to process data.

In the earlier program, the outer loop iterates as usual. For each iteration of the outer loop, the inner loop does a complete run (i.e., it does all its iterations).§§

The outer loop repeats for the number of words in the text. For each word selected, the inner loop repeats for the number of characters in the word. The Python interpreter goes back and forth between the outer and inner loops, until all the text has been processed.

Let us now take a look at the nested loop in our recent musical case study in more detail:

# separate text into words (using space as delimiter)

words = split(text)

# iterate through every word in the text

for word in words:

	# iterate through every character in this word

	for character in word:

	# is this character a vowel?

	if character in vowels:

	# yes, so find its position in the vowel list

	index = find(vowels, character)

	#print character, index

	 # a nd use position to find the corresponding pitch

	pitch = vowelPitches[index]

	# finally, remember this pitch

	pitches.append(pitch)

	# create duration from the word length

	duration = len(word) * durationFactor

	# and remember it

	durations.append(duration)

First, the split() function divides the text string using spaces as a delimiter (separator). This does mean that punctuation is considered part of “words”, but we will live with that for now.

The variable word in the outer loop holds each word in the text in turn. The inner loop act on each word by iterating through each character looking for a vowel. When it finds a vowel, it creates a note by mapping the vowel to its pitch, and the current word length to duration. In particular,

if character in vowels:

if the current character is a vowel, we find its index in the list of vowels,

index = find(vowels, character)

Since vowels is parallel with vowelPitches, we use this index to find the corresponding pitch:

pitch = vowelPitches[index]

The note duration is calculated from the current word length. Because word lengths may typically range from 1 to 7 (or so), we multiply by a duration factor (0.1). This creates more appropriate note durations.

# create duration from the word length

duration = len(word) * durationFactor

# and remember it

durations.append(duration)

Once we have the pitch and duration, we add this to the pitches and durations lists:

pitches.append(pitch)

durations.append(duration)

The rest of the program is straightforward. After the loop, we use the two lists, pitches and durations, to construct a phrase. Finally, we view that phrase and play it.

7.4.5 Exercise

Explore adding musical rules for numbers, punctuation, and whitespace.

• Decide how such characters may affect the panning and/or dynamic values of, say, subsequent notes.
• Consider how to handle punctuation such as commas and full stops. For example, they may add rests. (Hint: The string library defines list punctuation to hold all punctuation characters. Notice how this is similar to the list vowels, in the above program.)

7.5 File Input and Output

Python makes it very easy to input and output data through files. Files are an easy way to store and transfer information. This section explains how to have our programs open files in order to read in or write out information.

7.5.1 Reading Files

Opening a file is easy in Python—it is done with the open() function. For example, the following

data = open("someFile.txt", "r")

opens the file “someFile.txt” and stores in it the variable data. This file is expected to be in the same folder as your program.

Function open() expects two arguments, both strings. The first argument is the name of the file to be opened. The second argument determines if the file is opened for reading (i.e., “r”), for writing (i.e., “w”), or for appending (i.e., “a”).

Once a file has been opened for reading, the following functions can be used to read information from it:

• read()—reads the complete file in as a single (long!) string
• readline()—reads the next line as a string
• readlines()—reads all lines together as a list of strings

For example, the following code:

book = open("book.txt", "r")

text = book.read()

book.close()

opens the file “book.txt,” then reads the complete file as a string into the variable text. After reading the data, it is very important to close all files that have been opened. This is done via the close() function. Otherwise, open files are occupying memory for no reason.

Once you have read data from a file, you may use functions from the string library to extract whatever information you need. You must know the format an input file has if you are to successfully read it. Some possible input file formats include one word per line, two words separated by a semicolon, or, say, a sequence of numbers all on one line separated by whitespace. There are infinite possibilities; but as long as your program can anticipate (or better still knows) how data is structured inside a file (i.e., the file format), it can access what it needs through string functions.

The following is a more efficient way to read every line from a file. Actually, a file in Python is a sequence of lines. And since for loops operate on sequences (such as lists, strings, etc.), the following does not need to use readline() explicitly. In this example, we simply read every line and output it (via a print statement):

book = open("book.txt", "r")

for line in book:

	print line

book.close()

In your code, you should replace the print statement with whatever code is needed to process the input data (e.g., append it to a list, add it to a variable, etc.).

7.5.2 Writing Files

If a program needs to store data for later processing, this can be done by writing a file. Again, you open a file, but for writing this time. This is done using the open() function with the “w” argument (or “a” for appending). Then, we use the write() function to write information into the file.

• write(s)—writes string s to the file

Notice how this function requires a string. In other words, if we need to output other data (such as numbers), we need to first convert it to a string. This is done with the str() Python function.¶¶

For example, the following

data = open("numbers.txt", "w")

for i in range(1000):

	data.write(str(i))

	data.write('
')

data.close()

opens file "numbers.txt" for writing and stores a sequence of numbers into it. Notice how write() does not automatically output newlines. To get a newline, you need to explicitly output a newline character, " ".

Opening a file for writing overwrites (clears out) any earlier contents. To add data use “a” instead of “w,” which appends new material to an existing file.

Notice how we close the file via the close() function, once done with writing data into it. Otherwise some data may be lost.

7.5.3 Exercises

1. Write a program that creates a file called “randomNumbers.txt,” which contains a sequence of 20 random integers between 0 and 100 (inclusive). (Hint: See function randint() in Chapter 6.)
2. Go to Project Gutenberg on the internet and download Edgar Allan Poe’s “The Gold-Bug.” Modify the “Word Music” case study above to use words from a file. Decide what part of this book you would like to sonify and create an input text file accordingly from Poe’s e-book.

7.6 Python while Loop

In addition to the for loop, Python also provides the while loop. While loops have the following format:

while <condition>:

	<body>

where condition is a boolean expression (built with variables, relational and logical operators), similar to the ones we have used in if statements.

While the condition evaluates to True, the loop continues to iterate (hence its name—the while loop). As soon as the condition evaluates to False, the loop terminates.***

For example, the following while loop iterates 100 times.

i = 0

while i < 100:

	print i

	i = i + 1

This is equivalent to this for loop:

for i in range(100):

	print i

Actually, for loops are useful for iterating over a sequence (i.e., a fixed number of times). On the other hand, while loops are useful when we do not know in advance how many times we need to iterate. For example, consider the following:

# get input from user (perform error checking)

pi tch = input("Enter a MIDI pitch (0-127): ") # get first value

wh ile pitch < 0 or pitch > 127: # if value is wrong, try again

	print "The value you entered,", pitch,

	pr int "is outside the specified range. Please try again."

	pitch = input("Enter a MIDI pitch (0-127): ")

# now, we have a proper pitch value

Notice how the number of times this will loop is unknown—it depends on the user’s input. It may loop 0 times (if the user enters a valid input on the first attempt) or, say, 12 times, if the user makes 12 errors (highly unlikely, but possible).†††

Incidentally, this example performs error checking for the last case study in Chapter 2. Modify that program and run it again. While loops are great for implementing error checking and providing the user with opportunities to correct input errors.

7.6.1 Exercise

Write a program that reads in a file called “randomNumbers.txt,” which contains a sequence of random integers between 0 and 10 (inclusive).

• Have it generate notes using these input values as pitches. Make it terminate when the input value is 0. (Hint: This program should use the same while loop pattern as earlier, that is, read the first value outside the loop and then start looping. Inside the loop the program should create a note from the last integer (pitch) read in and then get the next integer from the input file. The loop should terminate when the input value is 0.)
• Verify that your program terminates when the first value in the file is 0. In this case, no notes should be generated.

7.7 Big Data

As a result of the proliferation of computers and technologies for data collection and storage over the last 60 years, society is encountering a deluge of data. This phenomenon of our times is called Big Data.

Definition: Big Data refers to modern data sets that have become too voluminous or complex to analyze and comprehend with traditional computing techniques.

Sources for such data sets include meteorology, astronomy and other physical systems, genomics, Internet social networks, and finance/business statistics.

Computers are perfect for doing repetitive tasks without getting tired. We can use them to create sonifications and/or visualizations of massive data sets, such as biosignals, weather data, etc. This can help to observe, analyze, and comprehend patterns in such data sets.

This section shows how to do this using Python and the music library. It also introduces the image library, which allows us to manipulate data in images.

7.7.1 Case Study: Biosignal Sonification

In this case study, we explore the processing and sonification of data from biological processes. Figure 7.2 displays heart data, captured by measuring blood pressure over time.

Additionally, Figure 7.3 displays skin conductance, captured by measuring electrical conductivity between two fingers over time (the more sweaty the fingers get, the higher the skin conductance).

These images demonstrate the concept of visualization. Visualization is one way to experience big data.

Definition: Visualization involves mapping data to the visual characteristics of an image, such as lines, colors, size, shapes, etc., in order to perceive patterns and better appreciate their characteristics.‡‡‡

Visualization is complementary to sonification, in that the human visual and auditory processing systems have evolved to perceive different types of patterns. Since the focus of this book is on music and programming, here we explore sonifying biological signals.§§§

The data presented in Figure 7.2 and Figure 7.3 are actually stored in a data file. In order to sonify these data, we first need to understand their format (i.e., how they are stored in our data file).

In the following is a sample of the actual raw data displayed in Figure 7.2 and Figure 7.3:

20:39:51.560	1.84	1.880

20:39:51.593	3.13	1.953

20:39:51.627	3.14	1.970

20:39:51.660	3.13	1.975

20:39:51.693	3.13	1.969

20:39:51.727	3.14	1.978

20:39:51.760	3.13	2.027

20:39:51.793	3.13	2.315

20:39:51.827	3.14	2.489

20:39:51.860	3.14	2.466

The preceding data was captured at a rate of approximately 30 measurements per second. The complete data file is available online at http://jythonMusic.org.

The data format consists of three columns (fields). These are the time of measurement (e.g., 20:39:51.560), the skin conductance at that time (e.g., 1.84), and the particular blood pressure at the time (e.g., 1.880).

7.7.1.1 Sonification Design

To analyze data through sonification, we need to find a way to map these data to sound. We pose the following questions: How can we map characteristics of these data to musical parameters? Are there some characteristics that are more important than others? Are there certain musical parameters better suited to sonify these data characteristics? For a given data set, there may be many ways to answer these questions.

In the following are some possibilities for the preceding data set.

First of all, there is no correct way to map data to sound. Again, the trick is to decide what aspects of the data you would like to make easily perceivable by mapping them to sound parameters. Moreover, in the context of music making, you might also consider what aspects of the data might contribute to more interesting music.

Here are some possibilities:

• Map skin data to pitch (remember to scale to a preferred integer range, e.g., C3–C6).
• Map heart data also to pitch (e.g., add some variety to pitch).
• Map heart data to dynamic (remember to scale to 0–127).

The following program, sonifyBiosignals.py, demonstrates these rules:

# sonifyBiosignals.py

#

# Sonify skin conductance and heart data to pitch and dynamic.

#

# Sonification design:

#

# * Skin conductance is mapped to pitch (C3 - C6).

# * Heart value is mapped to a pitch variation (0 to 24).

# * Heart value is mapped to dynamic (0 - 127).

#

# NOTE: We quantize pitches to the C Major scale.

#

from music import *

from string import *

# first let's read in the data

data = open("biosignals.txt", "r")

# read and process every line

skinData = []	# holds skin data

heartData = []	# holds heart data

for line in data:

	time, skin, heart = split(line)	# extract the three values

	skin = float(skin)	# convert from string to float

	heart = float(heart)	# convert from string to float

	skinData.append(skin)	# keep the skin data

	heartData.append(heart)	# keep the heart data

# now, heartData contains all the heart values

data.close() # done, so let's close the file

##### define the data structure

biomusicScore = Score("Biosignal sonification", 150)

biomusicPart  = Part(PIANO, 0)

biomusicPhrase = Phrase()

# let's find the range extremes

heartMinValue	= min(heartData)

heartMaxValue	= max(heartData)

skinMinValue	= min(skinData)

skinMaxValue	= max(skinData)

# let's sonify the data

i = 0;  # point to first value in data

while i < len(heartData):  # while there are more values, loop

	# map skin-conductance to pitch

	pitch = mapScale(skinData[i], skinMinValue, skinMaxValue, C3, C6, MAJOR_SCALE, C4)

	# map heart data to a variation of pitch

	pitchVariation = mapScale(heartData[i], heartMinValue,) heartMaxValue, 0, 24, MAJOR_SCALE, C4)

	# also map heart data to dynamic

	dynamic = mapValue(heartData[i], heartMinValue, heartMaxValue, 0, 127)

	 # finally, combine pitch, pitch variation, and dynamic into note note = Note(pitch + pitchVariation, TN, dynamic)

	# add it to the melody so far

	biomusicPhrase.addNote(note)

	# point to next value in heart and skin data

	i= i + 1

# now, biomusicPhrase contains all the sonified values

##### combine musical material

biomusicPart.addPhrase(biomusicPhrase)

biomusicScore.addPart(biomusicPart)

##### view score and write it to a MIDI file

View.sketch(biomusicScore)

Write.midi(biomusicScore, "sonifyBiosignals.mid")

Play.midi(biomusicScore)

First we import the necessary libraries and open the data file. Then we read every line from the file, using a loop, and split the data into its constituent parts. It so happens that the raw heart data is the third number on every line.¶¶¶

# read and process every line

skinData = []	# holds skin data

heartData = []	# holds heart data

for line in data:

time, skin, heart = split(line)	# extract the three values

skin = float(skin)			# convert from string to float

heart = float(heart)			 # convert from string to float

skinData.append(skin)			 # keep the skin data

heartData.append(heart)			 # keep the heart data

# now, heartData contains all the heart values

data.close()	# done, so let's close the file

Notice how we create an empty list, heartData, to hold the heart rate data from the data file. Inside the loop, we get a line from the data file at a time, extract the value of interest (i.e., raw heart data), and append it to this list. This is a very common pattern in data manipulation, called the accumulator pattern.

Definition: The accumulator pattern uses a loop to accumulate data, one item per iteration, in some container. For example, appending to a list, concatenating to a string, or adding to an integer variable.

Notice how we use the string function, split(line), to separate the single line (a string) to be split into a list of three substrings. The statement

time, skin, heart = split(line)

automatically “unpacks” the list of three substrings returned by split(line) to three strings and assigns each to the corresponding variable.

One thing about big data is that the data files may need some “massaging” (preprocessing) to end up in a format we want. In the above example, we assume that the data file has gone through this step, and thus no error checking is required in the code.

7.7.1.2 Python Parallel Assignment

Python provides a shorthand operation to assign several variables, in parallel, to corresponding values.

For example,

>>> o, p, q = [4, 5, 6]

>>> o

4

>>> p

5

>>> q

6

Notice how the three variables, o, p, and q, automatically get assigned to each item in the list.

>>> a, b, c = "123"

>>> a

'1'

>>> b

'2'

>>> c

'3'

This also works with strings (since both strings and lists are sequences). Finally, parallel assignment allows us to easily swap values between two variables:

>>> x = 2

>>> y = 3

>>> x

2

>>> y

3

>>> x, y = y, x

>>> x

3

>>> y

2

Notice how x and y are originally 2 and 3, respectively. After the parallel assignment (i.e., x, y = y, x), their values have been reversed. Parallel assignment is a very convenient operation.

Let’s get back to the program. The line

heart = float(heart)

converts the string representation of a number to a float number. This is necessary since data from a file is read in as a string.

Upon completion of the loop, all skin and heart data values are now accumulated in the skinData and heartData lists.

The rest of the program performs the sonification of these values. First we create the musical data structure,

##### define the data structure

biomusicScore	= Score("Biosignal sonification", 60)

biomusicPart	= Part(PIANO, 0)

biomusicPhrase	= Phrase()

Then we find the minimum and maximum data values (needed by mapValue() below).

# let's find the range extremes

minValue = min(heartData)

maxValue = max(heartData)

skinMinValue = min(skinData)

skinMaxValue = max(skinData)

We loop through the list of float numbers and sonify them. We have a few sonification rules:

• Each skin conductance value is mapped to a pitch in the range C3 to C6 (using the major scale).
• Each heart value is mapped to a pitch variation in the range 0 to 24. The idea is to combine both data values (skin conductance and heart) in a single melody. This makes sense since skin conductance tends to be very monotonous (without much fluctuation—see Figure 7.3). The heart value is producing interesting heart-like fluctuations (see Figure 7.2). Alternatively, we could use two melodies (and observe the harmonies they generate).
• Since a note’s dynamic level is an important musical parameter, we also map the heart values to it, in order to generate interesting dynamic variation.
• For simplicity, we leave the note duration constant (TN).****

This is just one of many possible mappings of the data. The following code implements this sonification design:

# let's sonify the data

i = 0;  # point to first value in data

while i < len(heartData):  # while there are more values, loop

	# map skin-conductance to pitch

	pitch = mapScale(skinData[i], skinMinValue, skinMaxValue, C3, C6, MAJOR_SCALE, C4)

	# map heart data to a variation of pitch

	pitchVariation = mapScale(heartData[i], heartMinValue, heartMaxValue, 0, 24, MAJOR_SCALE, C4)

	# also map heart data to dynamic

	dynamic = mapValue(heartData[i], heartMinValue, heartMaxValue, 0, 127)

	# finally, combine pitch, pitch variation, and dynamic into note

	note = Note(pitch + pitchVariation, TN, dynamic)

	# add it to the melody so far

	biomusicPhrase.addNote(note)

	# point to next value in heart and skin data

	i = i + 1

Notice the use of a while loop (instead of a for loop). This is equivalent to

for i in range(len(heartData)):

Notice how, with the while loop, we need to manually handle the initialization of i to 0 (right before the loop) and the increment of i (i.e., i = i + 1, at the very end of the loop).

Upon completion of the loop, biomusicPhrase contains all the notes resulting from sonifying the heart data. The rest of the program is straightforward. Notice how the View.sketch() function creates a visualization (of the sonification) of the data. Actually, we could focus more on this visual representation, as opposed to the sonification itself. In other words, we could use the musical notes as an intermediate form to create visualizations of data. In the next section, we will see another way to possibly create visualizations, by manipulating image files. Still, the primary focus of this book remains making music with Python.

7.7.2 Exercises

1. Skin-conductance level could be sonified as a separate melody. Then both “melodies” could be played in parallel. Also, we could use different MIDI instruments for the two melodies by putting them in different parts on different MIDI channels. What other combinations are possible? See the Kepler case study, presented earlier, for inspiration.
2. What other ways of sonifying the heart data can you think of? For example, you could capture the difference between two values and use it as note duration. (Hint: Remember to take the absolute value of the difference, i.e., use function abs(), since the difference may be negative.) In such a sonification, the duration of the note signifies the change in heart contraction between two consecutive measurements. Since these measurements are equally spaced in time, the longer the note, the more work was exerted by the heart (during those two measurements). This is a useful pattern, which is currently lost in a sea of data. What other useful patterns can you think of?
3. The US National Climatic Data Center (http://www.ncdc.noaa.gov) has a vast archive of historical weather (and other) data. For example, use the provided average monthly temperatures for California from 1895 to 2004. Find ways to make apparent the trends of global warming through sonification (and visualization) of data.
4. Find other sources of data (e.g., stock market, population sizes, astronomical data) and try to answer important questions or present interesting aspects of these data through sonification.

7.8 Python Functions

So far, we have seen various useful Python functions, such as abs(), round(), len(), mapValue(), and upper(). These functions are already built into Python (or one of its libraries), so that you can easily perform a needed task. For example,

>>> x = -3

>>> x = abs(x)

>>> x

3

>>> y = 3.6

>>> y = round(y)

>>> y

4.0

>>> z = ["this", "is", "fun"]

>>> len(z)

3

A nice feature of Python is that it allows you to create your own functions.

This is useful when you have code that performs a common task, which you end up using many times in your program. Defining a function allows you to “package” that code, so that you can reuse it easily, as we saw earlier.

Definition: A Python function is a piece of code that performs a specific task. Once defined, it may be used many times, without having to write it again and again.

7.8.1 Defining Functions

Functions are very easy to define. Simply take the code you want to package as a function, indent it (by, say, 3 spaces), and add a header line:

def functionName():

  <your indented code goes here>

For example, let us define a function that plays a violin note:

# define function to play a C4 quarter note using violin sound

def playViolinNote():

	note = Note(C4, QN)

	phrase = Phrase()

	phrase.addNote(note)

	phrase.setInstrument(VIOLIN)

	Play.midi(phrase)

The first line, in bold, is called the function header. It specifies the name to use when calling this function. Any arguments the function may need to get its job done are listed between the parentheses. In the earlier example, function playViolingNote() requires no argument.

Now, we can call function playViolinNote() as follows††††:

playViolinNote()

Every time you call this function, it plays a C4 quarter note using a violin timbre.

Functions are more useful (or general) when you can pass information to them, to customize what they do. For example, the preceding function definition can become more versatile by allowing us to specify the note’s pitch and duration:

# define function to play a specified note using violin sound

def playViolinNote(pitch, duration):

	note = Note(pitch, duration)

	phrase = Phrase()

	phrase.addNote(note)

	phrase.setInstrument(VIOLIN)

	Play.midi(phrase)

Now, we can call this function as follows:

>>> playViolinNote(C4, WN)

>>> playViolinNote(E4, HN)

>>> playViolinNote(G4, QN)

to play different notes. Notice how much more versatile this function has become.

In summary, functions hide unnecessary details from the main code. Also, they allow us to reuse code efficiently, because we define a function once and use it many times. Without functions, our programs would be much larger and harder to understand and modify.

7.8.2 Exercise

Define a function called playNote() which accepts a pitch, duration, and instrument. (You may assume the music library has already been imported.) Try it out with different notes and instruments (see Appendix A).

7.8.3 Returning Values

Python functions may return a value. This can be very useful when you are “packaging” code that produces a value (e.g., a number, a musical phrase, etc.) to be used by the caller.

For example, here is our own implementation of Python’s abs() function. This function returns the absolute value of a number.

# returns the absolute value of a number

def absolute(number):

	if number < 0:  # is the argument negative?

	number = -1 * number  # yes, so make it positive

	# now, the number is positive (either way)

	return number

Notice the return statement at the bottom of the function body. When evaluated, this statement passes back the specified value to the caller of the function. For example,

>>> x = absolute(-2.3)

>>> x

2.3

In other words, when we call absolute(−2.3), the argument, −2.3, is associated with the parameter number (see function definition) above. Then the body of the function executes with variable number having the value −2.3. The if statement’s condition evaluates to True, and so the sign of number changes (multiplying a number by −1 changes its sign). Finally, the return statement takes the value of variable number and returns it to the caller (as seen in the preceding interpreter example).

A return statement is similar to a print statement. Both take a value (or expression) and send it somewhere else. The print statement sends it to the computer screen. The return statement sends the value back to the statement that called the function. The difference is that, unlike print, the return statement also stops the function from executing, so that a value may be returned to the caller, and so the caller can continue its work.

A function may have several return statements. As soon as one is encountered, the function stops executing, and control returns to the caller. However, it is considered bad style to have more than one return statement.

Good Style: A function should have at most one return statement.‡‡‡‡

If a function does not have a return statement, it stops executing after the last statement in its body. Then control returns to the caller.

Let us see another example.

Python provides various functions for lists, such as len(), sum(), min(), and max(). However, Python does not provide a function for calculating the average of a list. Here we define such a function:

# returns the average of a numeric list

def avg(someList):

	total = sum(someList) # get the total of all items

	length = len(someList) # get the number of items

	# next, convert to float (for accuracy) and calculate average

	result = float(total) / float(length)

	# finally, return result to caller

	return result

7.8.4 Exercises

1. Division by zero is always a problem. Update the preceding function to check if length is 0. If so, it should output an error message and return the special value None. Or you can use the statement:

	raise ZeroDivisionError(string)

where string contains the error message. This will result in a typical Python error message similar to the ones we first saw in Chapter 2.

1. Write a function called createMelody(), which returns a phrase of notes with pitches and durations randomly selected from a scale and a list, respectively. The function should accept three parameters, namely, numNotes, scale, and durations (a list of durations) to choose from. For example, the following call

	 phrase = createMelody(10, PENTATONIC_SCALE, [QN, SN, EN])

should generate a phrase of 10 notes from the pentatonic scale with durations randomly selected from the provided list. (Hint: See the “Pentatonic Melody Generator” case study in Chapter 6.)

1. Write a function named createCantusVoice(), which returns a phrase containing one of the voices in Arvo Part’s “Cantus in Memoriam” seen in Chapter 4. The function should accept three parameters, namely, pitches, durations, and an elongationAmount (a float). For example, the following

	pitches = [A5, G5, F5, E5, D5, C5, B4, A4]

	durations = [HN, QN, HN, QN, HN, QN, HN, QN]

	voice = createCantusVoice(pitches, durations, 1.0)

should generate a phrase with 8 notes starting with an A5 half note (HN), followed by a G5 quarter note (QN),... and ending with an A4 quarter note. Whereas the following

	 voice = createCantusVoice(pitches, durations, 2.0)

should generate a phrase with 8 notes starting with an A5 whole note (WN), followed by a G5 half note (HN),... and ending with an A4 half note. (Hint: You may use Mod.elongate() in the body of your function.)

1. Add two parameters to the createCantusVoice() function defined earlier, namely, transposition and repetitions. The first specifies the amount by which to transpose the pitches in the result, and the second the number of times to repeat the result. (Hint: You may use Mod.transpose() and Mod.repeat() in the function body.)
2. Using the createCantusVoice() function defined earlier, write a program that generates variations on Arvo Pärt’s compositional idea, using different scales (e.g., C major scale) and different pitch movement (e.g., upward motion, or downward and upward motion in one scale, and so on). For example, the program may use the defined function as follows:

	pitches = [A5, G5, F5, E5, D5, C5, B4, A4]

	durations = [HN, QN, HN, QN, HN, QN, HN, QN]

	voice1 = createCantusVoice(pitches, durations, 0, 1.0)

	voice2 = createCantusVoice(pitches, durations, -12, 2.0)

	voice3 = createCantusVoice(pitches, durations, -24, 4.0)

	voice4 = createCantusVoice(pitches, durations, -36, 8.0)

to generate the different voices in the piece before adding them to a part.

7.8.5 Scope of Variables

In Python, variables that are declared outside a function are considered global; they have global scope. Global variables are visible anywhere within the program and can be accessed even from within functions. Variables that are defined within a function are only visible (accessible) within that function. We say that the scope of a variable is limited to the function.

Definition: The scope of a variable is the space within a program where that variable is visible (i.e., it can be accessed and possibly modified).

Ideally, a function should avoid using global variables.

Good Style: If a function needs any data, then that data should be passed to it through its parameter list.

Good Style: If a function changes any data outside its scope, it should return them via the return statement.

7.9 Image Sonification

One very interesting application of sonification is to make music from aesthetically pleasing or otherwise interesting images. Again, the idea is to find ways to map interesting patterns in one medium (in this case, images) to interesting patterns in another medium (in this case, sound). Image sonification is accomplished by reading in an image, which is a two-dimensional list of pixels (or pictures elements), and exploring ways to map these pixels to notes. Each pixel contains information about color in one small area of a picture.

7.9.1 Python Images

Your Python programs can read, manipulate, and write images through the image library. To access the image library, import it.

from image import *

An image is represented by Python as a two-dimensional table (i.e., matrix) of pixels.

Definition: An image is a table of x (width) * y (height) pixels.

For example, the following code creates a blank (black) image of 100 by 100 pixels.

img = Image (100, 100)

So an image has width * height number of pixels. The origin (0, 0) is at top left.

Definition: A pixel is a picture element or a point in an image.

Each pixel contains a list of three values; the red, green, and blue (or RGB) components of that point in the image. Each RGB value ranges from 0 to 255. For example,

• [255, 255, 255] is white
• [0, 0, 0] is black
• [255, 0, 0] is bright red
• [150, 0, 0] is a darker red
• [0, 255, 0] is bright green
• [0, 0, 255] is bright blue
• [255, 255, 0] is yellow
• [150, 150, 150] is a tone of gray

7.9.2 Image Library Functions

The Python image library provides various useful functions.§§§§ The most common ones are shown in Table 7.2.

The following case study applies some of these functions to generate music from images.

7.9.3 Case Study: Visual Soundscape

A soundscape refers to a musical composition that incorporates sounds recorded from, and/or music that depicts the characteristics of, an environment (e.g., a city soundscape or a forest soundscape).

In this case study, we explore how, through sonification of image data using image library functions, we create interesting musical artifacts by mapping visual aspects of an image into corresponding musical aspects. Also, similarly to sonification of other media (e.g., biosignals, weather data, financial markets, etc.), we may capture and explore structural characteristics of images that may not be as evident visually.

As we mentioned earlier, images consist of pixels (or picture elements). A pixel is the elemental data that digitally represents a single point in the original scene (as captured by a camera). The number of pixels in an image depends on the quality (or resolution) of the digital camera. For example, Figure 7.4 (depicting a sunset at the Loutraki seaside resort in Greece) consists of 320 × 213 pixels.

Similarly to other sonification activities, the ways we can map pixels to sound are not prescribed. A rule of thumb is to find what inspires you about a particular image and explore how you might convert that to sound. So image sonification involves imagination and artistic exploration.

The image in Figure 7.4 has a very nice gradient that gets brighter from left to right. The sun is not shown but can be imagined. There is a clear horizontal division between the sea and sky. The mountains, on the left, provide a contrast to the color of the sea and sky. Finally, the image gradient is interrupted by the (somewhat noisy) visual layers and the sea at the bottom half of the image. Clearly, there is enough structural variety in the visual domain to provide interesting analogies in the musical domain. All this can be exploited by selecting certain rows (or columns) of pixels, scanning the image left-to-right (or up-and-down), and converting individual pixels or areas of pixels to musical notes or passages.

7.9.3.1 Sonification Design

In this case study, we use the following sonification rules:

• Left-to-right pixel (column) position is mapped to time (actually, note start time);
• Brightness (or luminosity) of a pixel (i.e., average RGB value) is mapped to pitch (the brighter the pixel, the higher the pitch);
• Redness of a pixel (R value) is mapped to duration (the redder the pixel, the longer the note); and
• Blueness of a pixel (B value) is mapped to dynamic (the bluer the pixel, the louder the note).

These mappings were selected as a result of experimentation, simply because they sounded interesting, given this image. A little experimentation will go a long way.

Using the same sonification scheme with other images will likely generate interesting results. For instance, you could select a new image with this scheme in mind. Or you could create/modify an image (e.g., via Photoshop) with the particular sonification scheme in mind.

However, the most appropriate way is to pick an image and then design a set of sonification rules to use that matches its features. The image choice and sonification rules are intimately connected.

The following program implements the preceding rules to sonify the image in Figure 7.4.

# sonifyImage.py

#

# Demonstrates how to create a soundscape from an image. 

# It also demonstrates how to use functions.

# It loads a jpg image and scans it from left to right. 

# Pixels are mapped to notes using these rules:

# 

# + left-to-right column position is mapped to time,

# + luminosity (pixel brightness) is mapped to pitch within a scale, 

# + redness (pixel R value) is mapped to duration, and 

# + blueness (pixel B value) is mapped to volume.

#

from music import *

from image import *

from random import *

##### define data structure

soundscapeScore = Score("Loutraki Soundscape", 60)

soundscapePart = Part(PIANO, 0) 

##### define musical parameters

scale = MIXOLYDIAN_SCALE

minPitch = 0	# MIDI pitch (0-127)

maxPitch = 127

minDuration = 0.8	# duration (1.0 is QN)

maxDuration = 6.0

minVolume = 0	# MIDI velocity (0-127)

maxVolume = 127

# start time is randomly displaced by one of these 

# durations (for variety)

timeDisplacement = [DEN, EN, SN, TN]

##### read in image (origin (0, 0) is at top left)

image = Image("soundscapeLoutrakiSunset.jpg")

# specify image pixel rows to sonify - this depends on the image!

pixelRows = [0, 53, 106, 159, 212]

width = image.getWidth()	# get number of columns in image

height = image.getHeight()	# get number of rows in image

##### define function to sonify one pixel

# Returns a note wrapped in a phrase created from sonifying 

# the RGB values of 'pixel' found on given column.

def sonifyPixel(pixel, col):

	red, green, blue = pixel # get pixel RGB value

	luminosity = (red + green + blue) / 3	# calculate brightness

	# map luminosity to pitch (the brighter the pixel, the higher

	# the pitch) using specified scale

	pitch = mapScale(luminosity, 0, 255, minPitch, maxPitch, scale)

	# map red value to duration (the redder the pixel, the longer

	# the note)

	duration = mapValue(red, 0, 255, minDuration, maxDuration)

	# map blue value to dynamic (the bluer the pixel, the louder

	# the note)

	dynamic = mapValue(blue, 0, 255, minVolume, maxVolume)

	# create note and return it to caller

	note = Note(pitch, duration, dynamic)  

	# use column value as note start time (e.g., 0.0, 1.0, and so on)

	startTime = float(col)	# time is a float

	# add some random displacement for variety

	startTime = startTime + choice(timeDisplacement)

	# wrap note in a phrase to give it a start time

	# (Phrases have start time, Notes do not)

	phrase = Phrase(startTime)	# create phrase with given start time

	phrase.addNote(note)	# and put note in it 

	# done sonifying this pixel, so return result

	return phrase

##### create musical data

# sonify image pixels

for row in pixelRows:	# iterate through selected rows

	for col in range(width):	# iterate through all pixels on this row

	# get pixel at current coordinates (col and row)

	pixel = image.getPixel(col, row)

	# sonify this pixel (we get a note wrapped in a phrase)

	phrase = sonifyPixel(pixel, col)

	# put result in part

	soundscapePart.addPhrase(phrase)

	# now, all pixels on this row have been sonified

# now, all pixelRows have been sonified, and soundscapePart 

# contains all notes

##### combine musical material

soundscapeScore.addPart(soundscapePart)

##### view score and write it to an audio and MIDI files

View.sketch(soundscapeScore)

Write.midi(soundscapeScore, "soundscapeLoutrakiSunset.mid")

First, notice how the top-level comments describe what the program does and, in particular, the sonification rules employed by it. Keeping such comments at the beginning allows you to understand what a program does several months/years afterwards, without having to read the code again line for line. It takes great discipline to write quality comments. However, it saves time later on.¶¶¶¶ Actually, experienced programmers first write comments and then code. A common programming aphorism goes, “if you cannot comment it, you cannot code it.” This is especially true for writing elegant code.

Notice how we have grouped the definitions of musical parameters at the top. These values are set once and are used throughout the program.

Next, we load the image and define a list of pixel rows to sonify. The program generates one melodic line from each pixel row. Figure 7.5 shows these pixel rows, where row 0 is at the top, row 53 is the second from the top, and so on.

pixelRows = [0, 53, 106, 159, 212]

Usually, it is a bad idea to hard-code such numbers in a program. For instance, what would happen if we changed the image we read in (e.g., made it smaller or larger)? We make an exception here and hard code the numbers. Our rationale is that the choice of rows to sonify depends on the selected image. Hence, we can think of these as constants related to the selected image.

7.9.3.2 Defining a Function

Next comes the function definition. This function expects a pixel and its column coordinate. The pixel provides the RGB values for pitch, duration, and dynamic. The column is used to specify note start time (the piece evolves from left to right). By encapsulating the sonification scheme in a function, we help modularize our code. This makes it easier to change our code in the future, for example, to create different sonification schemes (and encapsulate them into different functions).

##### define function to sonify one pixel

# Returns a note wrapped in a phrase created from sonifying 

# the RGB values of 'pixel' found on given column.

def sonifyPixel(pixel, col):

	red, green, blue = pixel # get pixel RGB value

	luminosity = (red + green + blue) / 3  # calculate brightness

	# map luminosity to pitch (the brighter the pixel, the higher

	# the pitch) using specified scale

	pitch = mapScale(luminosity, 0, 255, minPitch, maxPitch, scale)

	# map red value to duration (the redder the pixel, the longer 

	# the note)

	duration = mapValue(red, 0, 255, minDuration, maxDuration)

	# map blue value to dynamic (the bluer the pixel, the louder 

	# the note)

	dynamic = mapValue(blue, 0, 255, minVolume, maxVolume)

	# create note and return it to caller

	note = Note(pitch, duration, dynamic)

	# use column value as note start time (e.g., 0.0, 1.0, and so on)

	startTime = float(col)	# time is a float

	# add some random displacement for variety

	startTime = startTime + choice(timeDisplacement)

	# wrap note in a phrase to give it a start time

	# (Phrases have start time, Notes do not)

	phrase = Phrase(startTime)	# create phrase with given start time

	phrase.addNote(note)	# and put note in it 

	# done sonifying this pixel, so return result

	return phrase

Some general things to know about functions:

• Functions need to be defined before they are called.
• As with variable names, try to come up with meaningful function names (e.g., sonifyPixel). If the name of the function communicates what it does, this makes our code easier to read.
• Notice the two parameters, pixel and column. These variables will store the values provided when the function is called. We use these variables in the function body to make use of these values.
• The body of the function is indented. It contains all the code that implements what the function does. This is similar to the bodies of loops and if statements.

Now let us explore what this function does. The first statement,

red, green, blue = pixel

is parallel assignment. Variable pixel contains a list of three RGB values, for example, [243, 124, 0]. The parallel assignment provides a convenient way to “unpack” the three RGB values from pixel.

The rest of the function body uses red, green, and blue to calculate the note pitch, duration, and dynamic, according to the sonification scheme described earlier.

The following statements

startTime = float(col) # time is a float

# add some random displacement for variety

startTime = startTime + choice(timeDisplacement)

use the column coordinate of the pixel to specify start time for the note created for this pixel. Since notes do not have a start time, we wrap the note into a phrase and set the phrase’s start time accordingly. Finally, since the columns are pretty rigid, that is, they take integral values (i.e., 0, 1, 2, 3, and so on), we add a random displacement (selected from the list timeDisplacement, i.e., [DEN, EN, SN, TN]). This makes the music less “square.”

The last statement in the function

return phrase

returns the created phrase to the function caller.

7.9.4 Python Nested Loops (again)

Next we use two loops, one inside the other. Again, this is called a nested loop.

The outer loop iterates as usual. For each iteration of the outer loop, the inner loop does a complete run (i.e., it does all its iterations).***** A helpful trick is to think of the inner loop as a simple statement. As all statements in the body of the outer loop, it needs to be executed completely.

##### create musical data

# sonify image pixels

for row in pixelRows:	# iterate through selected rows

	for col in range(width):	# iterate through all pixels on this row

	# get pixel at current coordinates (col and row)

	pixel = image.getPixel(col, row)

	# sonify this pixel (we get a note wrapped in a phrase)

	phrase = sonifyPixel(pixel, col)

	# put result in part

	soundscapePart.addPhrase(phrase)

	# now, all pixels on this row have been sonified

# now, all pixelRows have been sonified, and soundscapePart 

# contains all notes

As the earlier comments indicate, every time the inner loop finishes, one more row of pixels has been sonified. The inner loop gets every pixel on this row, sonifies it (using the sonifyPixel() function), and adds the sonified result (a phrase) to soundscapePart.

The outer loop repeats this process for every row in the pixelRows list. Changing pixelRows (at the top of the program) will automatically adjust which areas of the image are sonified.

We end the program as usual, that is, by putting the part in a score, visualizing the score, and saving the score in a MIDI file.

7.9.5 Exercises

The earlier program is only the beginning. Many avenues for experimentation exist.

1. Try different musical scales, such as MAJOR_SCALE and PENTATONIC_SCALE.
2. Modify the above program to put each row into a different part (i.e., define a total of 5 parts). Experiment with different instruments per part. Also, experiment with transposing each part into a different register (octave).
3. Select a different image (e.g., try a symmetrical or fractal image). What sonification ideas come to mind? How can you map the symmetry to musical form most effectively?
4. Try other possibilities such as include scanning the image top-to-bottom (as opposed to left-to-right), averaging areas of the image to generate musical events, changing how we sonify each pixel, mapping pixels to chords (as opposed to single notes), opening several images at once, and so on.

7.10 Summary

This chapter has explored some ideas and programming techniques for sonifying and visualizing data. Importantly, they may be used to perceive hidden, important patterns in big data. Once you understand the principles of sonification, the sky is the limit! Topics covered include data scaling using the mapValue() and mapScale() functions. A number of Python language features have been introduced. These include strings, reading and writing files, the while loop, and how to define your own functions. These have been applied to sonification case studies that demonstrate how these can be used for both artistic and scientific objectives. Explore and enjoy.

---