Chapter 7
Topics: Data sonification, mapValue() and mapScale(), Kepler, Python strings, music from text, Guido d’Arezzo, nested loops, file input/output, Python while loop, big data, biosignal sonification, defining functions, image sonification, Python images, visual soundscapes.
According to Scaletti (1993), “the idea of representing data in sound is an ancient one. For the ancient Greeks music was not an art-for-art’s sake, practiced in a vacuum, but a manifestation of the same ratios and relationships as those found in geometry or in the positions and behaviors of the planets.” Pythagoras, Plato, and Aristotle worked on quantitative expressions of proportion and beauty, such as the golden ratio. Pythagoreans, for instance, quantified harmonious musical intervals in terms of proportions (ratios) of the numbers 1, 2, 3, 4, and 5 (as we discussed earlier in Chapter 1). This became the basis for the scales, modes, and tuning systems used in Western music.
As Johannes Kepler (among others) observed, the universe is flowing and fluctuating at different time scales and size scales. We only hear a small range of these fluctuations as sound, namely, when they vibrate at frequencies between 20 and 20,000 times per second (Hertz or Hz). Similarly, we only see (a different) small range of these vibrations as visible light (red through purple). The limits of our senses make it difficult for us to fully appreciate how our cosmos is put together in harmonically (hence musically) meaningful ways. It is this harmony in the proportions and organizations of the parts that make our universe stay together, allowing us to be in it and experience it. It is this harmony that Kepler studied and tried to formalize and communicate through his three laws of planetary motion. Kepler also tried to map the harmony he observed in the astronomical data available at his time as sounds, that is, to sonify them (Kepler 1619).
Sonification allows us to capture and better experience phenomena that are outside our sensory range by mapping values into sound structures that we can perceive by listening to them. Data for sonification may come from any measurable vibration or fluctuation, such as planetary orbits, magnitudes of earthquakes, positions of branches on a tree, lengths of words in this chapter, and so on.
In addition to facilitating insight and analysis, sonification can also be used to make interesting music and sound, just like visualization of data can make interesting images. If music can involve the imitation of nature (mimesis), then sonification is a valuable tool for modern music making.
When dealing with big data (i.e., large amounts of data), sometimes it is easier to hear patterns by turning the data into sound, as opposed to looking at, say, thousands or millions of numbers.
Definition: Data sonification involves mapping numbers to musical parameters, such as pitch, duration, volume, pan position, and timbre, in order to better perceive patterns in the data and appreciate their characteristics.
A clear and simple example of sonification is the Geiger counter, a device which measures radiation and turns this measurement to audible clicks. The number (frequency) of clicks is directly related to the radiation level in the vicinity of the device.
Another example of sonification is the biosignal monitor that we see at hospitals. These are the devices that monitor signals measured from the human body, such as heart rate. These monitors (similarly to the Geiger counter) provide constant sonic biofeedback, allowing the human operator to focus on other tasks.
Mapping a value from one range to another is very common in sonification. For instance, you may want to map a list of numbers (say, from 20.0 to 110.0) to pitch values (say, from 30 to 90). This becomes more realistic if, say, you are interested in global warming and want to explore how temperatures change over time. By converting temperatures to pitch values, you can actually hear these changes.
The music library provides the mapValue() function precisely for this task. This function accepts the following arguments:
For example,
>>> from music import * # mapValue() is defined in this library
>>> mapValue(24, 0, 100, 32, 212) # celsius to fahrenheit
75
This converts 24°C to 75°F. To verify that this conversion really works, let us try converting the extremes:
>>> mapValue(0, 0, 100, 32, 212) # lowest temperature
32
So, when we give the lowest temperature in the temperature range (i.e., 0), mapValue() returns the lowest pitch in the destination range (i.e., 32).
>>> mapValue(100, 0, 100, 32, 212) # highest temperature
212
Also, when we give the highest temperature in the temperature range (i.e., 100), mapValue() returns the highest pitch in the destination range (i.e., 212).
>>> mapValue(56.7, 0.0, 100.0, 32.0, 212.0) # arbitrary temperature
134.06
Using float numbers returns float numbers.
Having tested this, we can assume that the rest of the values are mapped linearly (as if we took two number lines, one from 0 to 100 and another from 32 to 212, and superimposed them).
A useful feature is that mapValue() will return a value using the data type of the destination range. So, for example, providing float (real) values in the destination range generates a float result:
>>> mapValue(56.7, 0.0, 100.0, 32.0, 212.0)
134.06
The mapScale() function is similar to mapValue(), expect that it has an additional 6th argument:
This function quantizes (sieves) the numeric results to fit pitches within the provided scale. The key of the scale is determined by minResult. For example, any C pitch (e.g., C4 (60), C5 (72), etc.) corresponds to the key of C any D pitch corresponds to the key of D and so on.
Accordingly, mapScale() returns integer values, since they are meant to be used as pitches.
The difference between mapValue() and mapScale() is demonstrated as follows. Note that the first examples use duration constants, demonstrating another useful musical application for mapValue().
>>> from music import *
>>> from random import *
>>> mapValue(0.6, 0, 1, TN, WN)
2.4499999999999997
>>> mapValue(0.999, 0.0, 1.0, TN, WN)
3.996125
>>> mapScale(0.5, 0, 1, 0, 127, MAJOR_SCALE)
64
>>> mapScale(0.66, 0, 1, 40, 80, MINOR_SCALE)
66
Whereas mapValue() returns a float value with lots of accuracy, mapScale() returns an integer.
Here are some examples using a provided scale:
>>> for i in range(0, 13):
print i, "—>", mapScale(i, 0, 12, 0, 12, MAJOR_SCALE)
0 —> 0
1 —> 0
2 —> 2
3 —> 2
4 —> 4
5 —> 4
6 —> 5
7 —> 7
8 —> 7
9 —> 9
10 —> 9
11 —> 11
12 —> 12
Here we can clearly see how the result is adjusted to fit the pitches in the C major scale. We know the scale is in C from the minResult (4th argument), which is 0. Pitch 0 is a C pitch, namely, C_1.†
Constraining pitches to a scale allows us to generate more aesthetically pleasing sonifications.
Finally, since scales are simply lists of pitch classes (between 0 and 11),‡ for example,
>>> MAJOR_SCALE
[0, 2, 4, 5, 7, 9, 11]
>>> MINOR_SCALE
[0, 2, 3, 5, 7, 8, 10]
>>>
it is possible to use your own list, such as [0, 5, 8]. The only constraint is that the numbers range from 0 to 11. So, for example,
>>> mapScale(97.5, 20, 110, 30, 90, [0, 5, 8])
80
Here the scale has pitches C (0), F (5), and G sharp (8). If we had used mapValue(), the result would have been 81.7, but rounding to the nearest C, F, or G sharp gives us GS5, which is 80.
To understand better how mapScale() works with scales, try different scales with the following loop:
for i in range(22):
print mapScale(i, 0, 22, C3, C6, scale)
where scale is a particular scale or pitch class list.
In 1619 Johannes Kepler wrote his “Harmonices Mundi (Harmonies of the World)” book (Kepler 1619). While Pythagoreans only talked about the “music of the spheres,” Kepler discovered physical harmonies in planetary motion. As a result, he became a key figure in the development of astronomy and modern physics.
Following Kepler’s studies, in the late 1700s Johann Daniel Titius and Johann Elert Bode independently contributed to a model of the symmetries and proportions of our solar system. Their formula, known as the Titius–Bode law (or simply Bode’s law), predicts the positions of the planets in our solar system. Actually, it also predicted the asteroid belt between Mars and Jupiter (long before it was discovered), but fail to account for the irregularly moving Neptune and the (now demoted non-planet) Pluto.
The following program, harmonicesMundi.py, sonifies one aspect of the celestial organization of planets. In particular, it converts the orbital velocities of the planets to musical notes.
The planets’ mean orbital velocities (in kilometers per second) are as follows:
We will map this range of velocities to a range of MIDI pitches, say, C1 to C6. To do this mapping, we can use the mapScale() function.
# harmonicesMundi.py
#
# Sonify mean planetary velocities in the solar system.
#
from music import *
# create a list of planet mean orbital velocities
# Mercury, Venus, Earth, Mars, Ceres, Jupiter,
# Saturn, Uranus, Neptune, Pluto
planetVelocities = [47.89, 35.03, 29.79, 24.13, 17.882, 13.06,
9.64, 6.81, 5.43, 4.74]
# get minimum and maximum velocities:
minVelocity = min(planetVelocities)
maxVelocity = max(planetVelocities)
# calculate pitches
planetPitches = [] # holds list of sonified velocities
planetDurations = [] # holds list of durations
for velocity in planetVelocities:
# map a velocity to pitch and save it
pitch = mapScale(velocity, minVelocity, maxVelocity, C1, C6, CHROMATIC_SCALE)
planetPitches.append(pitch)
planetDurations.append(EN) # for now, keep duration fixed
# create the planet melodies
melody1 = Phrase(0.0) # starts at beginning
melody2 = Phrase(10.0) # starts 10 beats into the piece
melody3 = Phrase(20.0) # starts 20 beats into the piece
# create melody 1 (theme)
melody1.addNoteList(planetPitches, planetDurations)
# melody 2 starts 10 beats into the piece and
# is elongated by a factor of 2
melody2 = melody1.copy()
melody2.setStartTime(10.0)
Mod.elongate(melody2, 2.0)
# melody 3 starts 20 beats into the piece and
# is elongated by a factor of 4
melody3 = melody1.copy()
melody3.setStartTime(20.0)
Mod.elongate(melody3, 4.0)
# rep eat melodies appropriate times, so they will end together
Mod.repeat(melody1, 8)
Mod.repeat(melody2, 3)
# create parts with different instruments and add melodies
part1 = Part("Eighth Notes", PIANO, 0)
part2 = Part("Quarter Notes", FLUTE, 1)
part3 = Part("Half Notes", TRUMPET, 3)
part1.addPhrase(melody1)
part2.addPhrase(melody2)
part3.addPhrase(melody3)
# finally, create, view, and write the score
score = Score("Celestial Canon")
score.addPart(part1)
score.addPart(part2)
score.addPart(part3)
View.sketch(score)
Play.midi(score)
Write.midi(score, "harmonicesMundi.mid")
Notice how we first create the list of planetary velocities. For this case study, we also include Ceres (i.e., the asteroid belt) and (the recently demoted non-planet) Pluto.
planetVelocities = [47.89, 35.03, 29.79, 24.13, 17.882, 13.06,
9.64, 6.81, 5.43, 4.74]
Notice how we use the list functions, min() and max(), to find the minimum and maximum velocity values.
minVelocity = min(planetVelocities)
maxVelocity = max(planetVelocities)
We could have done this by hand by scanning the list ourselves and hard-coded the min and max numbers in the program; however, this use of “magic numbers” (as they are called by seasoned software developers) can get us into trouble.
Good Style: Avoid using magic numbers in your programs.
Definition: A magic number is a number in our programs that seems to come out of nowhere, that is, it is not derived automatically (calculated) from the program’s data.
Even if we document such magic numbers profusely (a minimum requirement if we have to use them), they become a place of future error, as the program evolves (and most programs do evolve). So as to avoid using magic numbers in the preceding code, we added code to calculate min and max from the list of velocities. Thus, if the list ever changes (or it is read from a data file, as we will see soon), min() and max() will continue to do the right thing. These functions, together with mapValue(), are very useful in sonification tasks.
Notice how mapScale() is utilized to sonify the orbital velocities. For example, assume that variable velocity contains the orbital velocity of a planet, and variables minVelocity and maxVelocity contain the smallest and largest of all planet velocities, respectively. Then this statement
pitch = mapScale(velocity, minVelocity, maxVelocity, C1, C6, CHROMATIC_SCALE)
generates a pitch value positioned in the range C1 to C6 similarly to how velocity is positioned in the range minVelocity to maxVelocity.
Next we use a for loop to map each velocity in planetVelocities to the corresponding pitch (in the range 24 to 84). Notice how we create an empty list ahead of time (outside of the loop) and use the list operation append() to add each new pitch value, as it is created, to the end of the pitches list.
# calculate pitches
planetPitches = [] # holds list of sonified velocities
for velocity in planetVelocities:
# map a velocity to pitch and save it
pitch = mapValue(velocity, minVelocity, maxVelocity, C1, C6)
planetPitches.append(pitch)
Now the list planetPitches is created and contains one pitch per planet. Notice how it is parallel to the list planetVelocities—for every planet velocity there is now a pitch, in the corresponding list positions.
Finally, we are ready to sonify. What follows is similar to the types of things we have seen in earlier chapters. We have many options for how we can sonify these data. One possibility is to just play the melody generated by the pitches. Another possibility would be to create a musical piece using the above melody as a component. For instance, we might use the original melody as a bass line, add chords, drums, and additional musical material.
Here, in the spirit of J.S. Bach and Arvo Pärt, we build a canon from the sonified orbital velocities. To do this, we treat the melody as the theme and use canonic devices (seen in Chapter 4) to create a celestial canon. We choose to play the melody concurrently, against itself, using different durations (see Figure 7.1). This is similar to Arvo Pärt’s musical structure for Cantus in Memoriam (see case study in Chapter 4).
The preceding code implements this canon structure. Notice how we create three different melodies, melody1, melody2, and melody3, to contain the same pitch list (the one sonified earlier) but with different durations, adding up to 5 beats, 10 beats, and 20 beats, respectively. Then we adjust the start time of each melody (i.e., start at 0.0, 10.0, and 20.0), so that they are introduced incrementally, but end together.
Modify the preceding sonification to use a different musical structure; there are many possibilities. Aim for something aesthetically pleasing. Consider using different scales with function mapScale(). Experiment with different scales, including your own pitch sets.
Search the Internet for interesting astronomical data, download them, and sonify them using a similar approach to the one explored here. For musical inspiration, listen to Olivier Messiaen’s “Mode de valeurs et d’intensités” for piano (1949), in which he creates musical material in a systematic way focusing on durations and intensities.
In Chapter 2, we discussed how Python uses strings to represent text data. String is another Python data type, such as int, float, list, and boolean. A string contains a sequence of characters. Strings are enclosed in quotes.
Definition: A Python string consists of zero or more characters, enclosed in quotes.
There are three types of quotes that may be used, single quotes (‘ ’), double quotes (“”), or three double quotes (“““”””). For example,
'This is a string - 1, 2, 3.'
Here is the same string using double quotes:
"This is a string - 1, 2, 3."
And again using three double quotes:
"""This is a string - 1, 2, 3."""
You can use any one of these three representations. Most programmers prefer the first two, since clearly they are most economical. The only reason to use the three double quotes is that such strings may contain newlines, that is, they can span several lines. This is not the case with the other quotes, for example,
"""This is a
string - 1,
2, 3."""
Strings are similar to lists in that they are sequences. As a result, everything we learned about lists, in terms of indexing, applies to strings as well. For example, normal indexing and slice operations apply:
>>> s = "This a string - 1, 2, 3."
>>> s[0]
'T'
>>> s[1]
'h'
>>> s[1:4]
'his'
>>> s[1:4] == s[10:13]
False
>>> s[2] == s[10]
True
>>> s[2]
'i'
Also, we can iterate through strings like we did with lists:
>>> s1 = "Hello!"
>>> for character in s1:
... print character
...
H
e
l
l
o
!
In the last example, what would happen if we put a comma at the end of the print statement? Why? §
String characters are represented internally using numbers, according to the ASCII standard.¶ The complete listing of the ASCII characters (and the numbers used to represent them) is easily available on the Internet. Most of the time we can ignore this. However, sometimes knowing the internal representation can be useful.
In Python, we can use function ord() which, given a string with a single character as argument, returns the corresponding ASCII number. For example,
>>> ord('A')
65
>>> ord('a')
97
>>> ord(' ')
32
>>> ord('!')
33
>>> ord('0')
48
One thing to remember is that not all ASCII characters are printable, that is, some of them are used for formatting (e.g., newline character, tab, backspace, etc.). The printable ASCII characters range from space (ASCII value 32) to the tilde character, “~” (ASCII 126).
Having access to the ASCII numbers makes it straightforward to generate music from text. It is not that different from generating music from planetary orbital velocities, as we see in the following text.
A simple way to algorithmically dictate musical structures is to follow some existing data patterns. One source of data patterns is astronomical data, as we saw earlier. Another source of patterns is natural language (e.g., English). Since languages have inherent structure—as described by Noam Chomsky (1957) and George K. Zipf (1949)—it is reasonable to expect that music based on text might maintain some of the expressiveness inherent in this structure. The sonification of text can, similar to all sonifications, use simple or complex mappings between the text and sound.
The following program, textMusic.py, demonstrates how to generate music from text. Using the ord() Python built-in function, this program converts the values of ASCII characters to MIDI pitches. For variety, note durations are randomized; other note properties (volume, etc.) are the same for all notes.
# textMusic.py
#
# It demonstrates how to generate music from text.
# It converts the values of ASCII characters to MIDI pitch.
# Note duration is picked randomly from a weighted-probability list.
# All other music parameters (volume, panoramic, instrument, etc.)
# are kept constant.
#
from music import *
from random import *
# Define text to sonify.
# Excerpt from Herman Melville's "Moby-Dick", Epilogue (1851)
text = """The drama's done. Why then here does any one step forth? - Because one did survive the wreck. """
##### define the data structure
textMusicScore = Score("Moby-Dick melody", 130)
textMusicPart = Part("Moby-Dick melody", GLOCK, 0)
textMusicPhrase = Phrase()
# create durations list (factors correspond to probability)
durations = [HN] + [QN]*4 + [EN]*4 + [SN]*2
##### create musical data
for character in text: # loop enough times
value = ord(character) # convert character to ASCII number
# map printable ASCII values to a pitch value
pitch = mapScale(value, 32, 126, C3, C6, PENTATONIC_SCALE, C2)
# map printable ASCII values to a duration value
index = mapValue(value, 32, 126, 0, len(durations)-1)
duration = durations[index]
print "value", value, "becomes pitch", pitch,
print "and duration", duration
dynamic = randint(60, 120) # get a random dynamic
note = Note(pitch, duration, dynamic) # create note
textMusicPhrase.addNote(note) # and add it to phrase
# now, all characters have been converted to notes
# add ending note (same as last one - only longer)
note = Note(pitch, WN)
textMusicPhrase.addNote(note)
##### combine musical material
textMusicPart.addPhrase(textMusicPhrase)
textMusicScore.addPart(textMusicPart)
##### view score and write it to a MIDI file
View.show(textMusicScore)
Play.midi(textMusicScore)
Write.midi(textMusicScore, "textMusic.mid")
Note that the string to sonify is at the top of the code. If you change this string, you will get different (yet similar music). Why? The music generated depends on the relative probabilities of characters in the English language (and not on the actual words, or, even further, the meaning of those words). It would be interesting to explore how to somehow map the meaning of words (or actual words) to note pitch. This would involve more work—beyond the scope of this chapter, for sure—but definitely something that could be explored using Python.
Also, notice how we use a for loop to iterate through every character in the string (stored in variable text). Actually, for loops work not only with lists, but also with any sequence, such as a string. (Remember that a string is a sequence of characters.) In the body of this loop, we get the ASCII number corresponding to each character and map it to a pitch.
ASCII characters are represented by integers from 0 to 127. However, in a regular text file, we could have mapped them directly to MIDI pitches (also from 0 to 127), that is,
pitch = ord(character)
But instead we opted for a more musical mapping utilizing the PENTATONIC_SCALE, that is,
pi tch = mapScale(value, 0, 127, C2, C6, PENTATONIC_SCALE)
Notice the print statement in the loop, which is commented out. It can be used to output the original value and the mapped pitch and duration values.
Good Style: Use strategically placed print statements (as above) to figure out what the code is doing (especially if things do not work as expected). Delete them after the code is ready.
Print statements are a programmer’s best friend. Use them extensively while you are developing your code. Then you can remove them (or comment them out, as above—they may come in handy again, after you have made a change in your code).
Consider ways to make the earlier sonification more interesting.
Python provides a string library with useful string functions. Table 7.1 presents the most common ones (assume s is a string).
Common string library functions (assume s is a string)
Function |
Description |
capitalize(s) |
Turn first letter of s into upper case. |
upper(s) |
Turn every letter of s into upper case. |
lower(s) |
Turn every letter of s into lower case. |
count(s, substring) |
Count occurrences of substring in s. |
find(s, substring) |
Find leftmost occurrence of substring in s (returns the position in s, starting at 0, or −1 if not found). |
rfind(s, substring) |
Find rightmost occurrence of substring in s (returns the position in s, starting at 0 , or −1 if not found). |
replace(s, old, new) |
Replace first occurrence of old with new in string s . |
strip(s) |
Remove whitespace from both ends of string s . |
lstrip(s) |
Remove whitespace from left side of string s . |
rstrip(s) |
Remove whitespace from right side of string s . |
split(s, char) |
Split string s into a list of substrings, using char as the delimiter (char is removed from the result). If char is omitted, it defaults to whitespace (space, tab, newline). |
join(listOfStrings, char) |
Join a list of strings into a single string, with char being the spacer placed in between the strings joined together (opposite of split). |
char in string |
Checks if char is in string. |
Here are some examples:
>>> from string import *
>>> s = 'hello world!'
>>> capitalize(s)
'Hello world!'
>>> upper(s)
'HELLO WORLD!'
>>> s
'hello world!'
Note that string functions do not modify the string provided as an argument; they just return a new string. For example, after the above, s still has its original value.
To modify s, we need to do this:
>>> s = capitalize(s)
>>> s
'Hello world!'
Here is another example,
>>> s = 'hello world!'
>>> find(s, 'e')
1
The result, 1, indicates that the first occurrence of substring “e” appears in position 1 (where “h” is at position 0).**
Finally, the following demonstrates functions split() and join():
>>> s = 'hello world!'
>>> s1 = split(s, ' ')
>>> s1
['hello', 'world!']
>>> s2 = join(sl, ' ')
>>> s2
'hello world!'
These are a few of the string functions available in the Python string library. They should be sufficient for most tasks related to music making. Feel free to explore more online.††
One of the oldest known algorithmic music processes is a rule-based algorithm that selects each note based on the letters in a text, credited to Guido d’Arezzo (991–1033). Originally, the intention was that the melody was a sung phrase and the text was the lyric to be sung. Each vowel in the text is associated with a pitch. The duration of notes comes from the word length.
Although d’Arezzo’s original intention was simply to provide an approximate composition guide, here we formalize and automate these rules. This is an approximation to d’Arezzo’s algorithm, adapted to text written in ASCII and to modern musical sensibilities.
# guidoWordMusic.py
#
# Creates a melody from text using the following rules:
#
# 1) Vowels specify pentatonic pitch, 'a' is C4, 'e' is D4,
# 'i' is E4, 'o' is G4, and 'u' is A4.
#
# 2) Consonants extend the duration of the previous note (if any).
#
from music import *
from string import *
# this is the text to be sonified
text = """One of the oldest known algorithmic music processes is a rule-based algorithm that selects each note based on the letters in a text, credited to Guido d'Arezzo."""
text = lower(text) # convert string to lowercase
# define vowels and corresponding pitches (parallel sequences),
# i.e., first vowel goes with first pitch, and so on.
vowels = "aeiou"
vowelPitches = [C4, D4, E4, G4, A4]
# define consonants
consonants = "bcdfghjklmnpqrstvwxyz"
# define parallel lists to hold pitches and durations
pitches = []
durations = []
# factor used to scale durations
durationFactor = 0.1 # higher for longer durations
# separate text into words (using space as delimiter)
words = split(text)
# iterate through every word in the text
for word in words:
# iterate through every character in this word
for character in word:
# is this character a vowel?
if character in vowels:
# yes, so find its position in the vowel list
index = find(vowels, character)
#print character, index
# and use position to find the corresponding pitch
pitch = vowelPitches[index]
# finally, remember this pitch
pitches.append(pitch)
# create duration from the word length
duration = len(word) * durationFactor
# and remember it
durations.append(duration)
# now, pitches and durations have been created
# so, add them to a phrase
melody = Phrase()
melody.addNoteList(pitches, durations)
# view and play melody
View.notation(melody)
Play.midi(melody)
In this program, we import the string library to use functions lower(), split(), and find().
We use function lower() to convert to lowercase the characters in variable text. We do this, because we have defined rules in terms of lowercase vowels. If we did not convert text to lowercase we would need to duplicate rules for uppercase letters.
Notice how we define two parallel sequences (a string and a list). The first one, vowels, holds the vowels we wish to associate with notes. The second one, vowelPitches, holds the pitches associated with those vowels. Since both are indexed starting at 0, we can use them to associate vowels with pitches.‡‡ For example,
>>> vowels = "aeiou"
>>> vowelPitches = [C4, D4, E4, G4, A4]
>>> vowels[0]
'a'
>>> vowelPitches[0]
60 # same as C4
Also, notice how we define two parallel lists, pitches and durations, to store musical data for the notes.
Loops may be nested. This kind of hierarchical processing (loops within loops) is quite common as a programming pattern, and we will see another example of it later in this chapter. It is common in programming because hierarchical structures are common in the world. As a geographic example of a hierarchy, consider how people live in suburbs, suburbs are in regions, regions are in states, states are in a country, and so on. This structure could be used as the basis for an algorithm to process details about all people in a particular geographic area.
Definition: Nested loops use two or more loops, one inside the other, to process data.
In the earlier program, the outer loop iterates as usual. For each iteration of the outer loop, the inner loop does a complete run (i.e., it does all its iterations).§§
The outer loop repeats for the number of words in the text. For each word selected, the inner loop repeats for the number of characters in the word. The Python interpreter goes back and forth between the outer and inner loops, until all the text has been processed.
Let us now take a look at the nested loop in our recent musical case study in more detail:
# separate text into words (using space as delimiter)
words = split(text)
# iterate through every word in the text
for word in words:
# iterate through every character in this word
for character in word:
# is this character a vowel?
if character in vowels:
# yes, so find its position in the vowel list
index = find(vowels, character)
#print character, index
# a nd use position to find the corresponding pitch
pitch = vowelPitches[index]
# finally, remember this pitch
pitches.append(pitch)
# create duration from the word length
duration = len(word) * durationFactor
# and remember it
durations.append(duration)
First, the split() function divides the text string using spaces as a delimiter (separator). This does mean that punctuation is considered part of “words”, but we will live with that for now.
The variable word in the outer loop holds each word in the text in turn. The inner loop act on each word by iterating through each character looking for a vowel. When it finds a vowel, it creates a note by mapping the vowel to its pitch, and the current word length to duration. In particular,
if character in vowels:
if the current character is a vowel, we find its index in the list of vowels,
index = find(vowels, character)
Since vowels is parallel with vowelPitches, we use this index to find the corresponding pitch:
pitch = vowelPitches[index]
The note duration is calculated from the current word length. Because word lengths may typically range from 1 to 7 (or so), we multiply by a duration factor (0.1). This creates more appropriate note durations.
# create duration from the word length
duration = len(word) * durationFactor
# and remember it
durations.append(duration)
Once we have the pitch and duration, we add this to the pitches and durations lists:
pitches.append(pitch)
durations.append(duration)
The rest of the program is straightforward. After the loop, we use the two lists, pitches and durations, to construct a phrase. Finally, we view that phrase and play it.
Explore adding musical rules for numbers, punctuation, and whitespace.
Python makes it very easy to input and output data through files. Files are an easy way to store and transfer information. This section explains how to have our programs open files in order to read in or write out information.
Opening a file is easy in Python—it is done with the open() function. For example, the following
data = open("someFile.txt", "r")
opens the file “someFile.txt” and stores in it the variable data. This file is expected to be in the same folder as your program.
Function open() expects two arguments, both strings. The first argument is the name of the file to be opened. The second argument determines if the file is opened for reading (i.e., “r”), for writing (i.e., “w”), or for appending (i.e., “a”).
Once a file has been opened for reading, the following functions can be used to read information from it:
For example, the following code:
book = open("book.txt", "r")
text = book.read()
book.close()
opens the file “book.txt,” then reads the complete file as a string into the variable text. After reading the data, it is very important to close all files that have been opened. This is done via the close() function. Otherwise, open files are occupying memory for no reason.
Once you have read data from a file, you may use functions from the string library to extract whatever information you need. You must know the format an input file has if you are to successfully read it. Some possible input file formats include one word per line, two words separated by a semicolon, or, say, a sequence of numbers all on one line separated by whitespace. There are infinite possibilities; but as long as your program can anticipate (or better still knows) how data is structured inside a file (i.e., the file format), it can access what it needs through string functions.
The following is a more efficient way to read every line from a file. Actually, a file in Python is a sequence of lines. And since for loops operate on sequences (such as lists, strings, etc.), the following does not need to use readline() explicitly. In this example, we simply read every line and output it (via a print statement):
book = open("book.txt", "r")
for line in book:
print line
book.close()
In your code, you should replace the print statement with whatever code is needed to process the input data (e.g., append it to a list, add it to a variable, etc.).
If a program needs to store data for later processing, this can be done by writing a file. Again, you open a file, but for writing this time. This is done using the open() function with the “w” argument (or “a” for appending). Then, we use the write() function to write information into the file.
Notice how this function requires a string. In other words, if we need to output other data (such as numbers), we need to first convert it to a string. This is done with the str() Python function.¶¶
For example, the following
data = open("numbers.txt", "w")
for i in range(1000):
data.write(str(i))
data.write(' ')
data.close()
opens file "numbers.txt" for writing and stores a sequence of numbers into it. Notice how write() does not automatically output newlines. To get a newline, you need to explicitly output a newline character, " ".
Opening a file for writing overwrites (clears out) any earlier contents. To add data use “a” instead of “w,” which appends new material to an existing file.
Notice how we close the file via the close() function, once done with writing data into it. Otherwise some data may be lost.
In addition to the for loop, Python also provides the while loop. While loops have the following format:
while <condition>:
<body>
where condition is a boolean expression (built with variables, relational and logical operators), similar to the ones we have used in if statements.
While the condition evaluates to True, the loop continues to iterate (hence its name—the while loop). As soon as the condition evaluates to False, the loop terminates.***
For example, the following while loop iterates 100 times.
i = 0
while i < 100:
print i
i = i + 1
This is equivalent to this for loop:
for i in range(100):
print i
Actually, for loops are useful for iterating over a sequence (i.e., a fixed number of times). On the other hand, while loops are useful when we do not know in advance how many times we need to iterate. For example, consider the following:
# get input from user (perform error checking)
pi tch = input("Enter a MIDI pitch (0-127): ") # get first value
wh ile pitch < 0 or pitch > 127: # if value is wrong, try again
print "The value you entered,", pitch,
pr int "is outside the specified range. Please try again."
pitch = input("Enter a MIDI pitch (0-127): ")
# now, we have a proper pitch value
Notice how the number of times this will loop is unknown—it depends on the user’s input. It may loop 0 times (if the user enters a valid input on the first attempt) or, say, 12 times, if the user makes 12 errors (highly unlikely, but possible).†††
Incidentally, this example performs error checking for the last case study in Chapter 2. Modify that program and run it again. While loops are great for implementing error checking and providing the user with opportunities to correct input errors.
Write a program that reads in a file called “randomNumbers.txt,” which contains a sequence of random integers between 0 and 10 (inclusive).
As a result of the proliferation of computers and technologies for data collection and storage over the last 60 years, society is encountering a deluge of data. This phenomenon of our times is called Big Data.
Definition: Big Data refers to modern data sets that have become too voluminous or complex to analyze and comprehend with traditional computing techniques.
Sources for such data sets include meteorology, astronomy and other physical systems, genomics, Internet social networks, and finance/business statistics.
Computers are perfect for doing repetitive tasks without getting tired. We can use them to create sonifications and/or visualizations of massive data sets, such as biosignals, weather data, etc. This can help to observe, analyze, and comprehend patterns in such data sets.
This section shows how to do this using Python and the music library. It also introduces the image library, which allows us to manipulate data in images.
In this case study, we explore the processing and sonification of data from biological processes. Figure 7.2 displays heart data, captured by measuring blood pressure over time.
Additionally, Figure 7.3 displays skin conductance, captured by measuring electrical conductivity between two fingers over time (the more sweaty the fingers get, the higher the skin conductance).
These images demonstrate the concept of visualization. Visualization is one way to experience big data.
Definition: Visualization involves mapping data to the visual characteristics of an image, such as lines, colors, size, shapes, etc., in order to perceive patterns and better appreciate their characteristics.‡‡‡
Visualization is complementary to sonification, in that the human visual and auditory processing systems have evolved to perceive different types of patterns. Since the focus of this book is on music and programming, here we explore sonifying biological signals.§§§
The data presented in Figure 7.2 and Figure 7.3 are actually stored in a data file. In order to sonify these data, we first need to understand their format (i.e., how they are stored in our data file).
In the following is a sample of the actual raw data displayed in Figure 7.2 and Figure 7.3:
20:39:51.560 1.84 1.880
20:39:51.593 3.13 1.953
20:39:51.627 3.14 1.970
20:39:51.660 3.13 1.975
20:39:51.693 3.13 1.969
20:39:51.727 3.14 1.978
20:39:51.760 3.13 2.027
20:39:51.793 3.13 2.315
20:39:51.827 3.14 2.489
20:39:51.860 3.14 2.466
The preceding data was captured at a rate of approximately 30 measurements per second. The complete data file is available online at http://jythonMusic.org.
The data format consists of three columns (fields). These are the time of measurement (e.g., 20:39:51.560), the skin conductance at that time (e.g., 1.84), and the particular blood pressure at the time (e.g., 1.880).
To analyze data through sonification, we need to find a way to map these data to sound. We pose the following questions: How can we map characteristics of these data to musical parameters? Are there some characteristics that are more important than others? Are there certain musical parameters better suited to sonify these data characteristics? For a given data set, there may be many ways to answer these questions.
In the following are some possibilities for the preceding data set.
First of all, there is no correct way to map data to sound. Again, the trick is to decide what aspects of the data you would like to make easily perceivable by mapping them to sound parameters. Moreover, in the context of music making, you might also consider what aspects of the data might contribute to more interesting music.
Here are some possibilities:
The following program, sonifyBiosignals.py, demonstrates these rules:
# sonifyBiosignals.py
#
# Sonify skin conductance and heart data to pitch and dynamic.
#
# Sonification design:
#
# * Skin conductance is mapped to pitch (C3 - C6).
# * Heart value is mapped to a pitch variation (0 to 24).
# * Heart value is mapped to dynamic (0 - 127).
#
# NOTE: We quantize pitches to the C Major scale.
#
from music import *
from string import *
# first let's read in the data
data = open("biosignals.txt", "r")
# read and process every line
skinData = [] # holds skin data
heartData = [] # holds heart data
for line in data:
time, skin, heart = split(line) # extract the three values
skin = float(skin) # convert from string to float
heart = float(heart) # convert from string to float
skinData.append(skin) # keep the skin data
heartData.append(heart) # keep the heart data
# now, heartData contains all the heart values
data.close() # done, so let's close the file
##### define the data structure
biomusicScore = Score("Biosignal sonification", 150)
biomusicPart = Part(PIANO, 0)
biomusicPhrase = Phrase()
# let's find the range extremes
heartMinValue = min(heartData)
heartMaxValue = max(heartData)
skinMinValue = min(skinData)
skinMaxValue = max(skinData)
# let's sonify the data
i = 0; # point to first value in data
while i < len(heartData): # while there are more values, loop
# map skin-conductance to pitch
pitch = mapScale(skinData[i], skinMinValue, skinMaxValue, C3, C6, MAJOR_SCALE, C4)
# map heart data to a variation of pitch
pitchVariation = mapScale(heartData[i], heartMinValue,) heartMaxValue, 0, 24, MAJOR_SCALE, C4)
# also map heart data to dynamic
dynamic = mapValue(heartData[i], heartMinValue, heartMaxValue, 0, 127)
# finally, combine pitch, pitch variation, and dynamic into note note = Note(pitch + pitchVariation, TN, dynamic)
# add it to the melody so far
biomusicPhrase.addNote(note)
# point to next value in heart and skin data
i= i + 1
# now, biomusicPhrase contains all the sonified values
##### combine musical material
biomusicPart.addPhrase(biomusicPhrase)
biomusicScore.addPart(biomusicPart)
##### view score and write it to a MIDI file
View.sketch(biomusicScore)
Write.midi(biomusicScore, "sonifyBiosignals.mid")
Play.midi(biomusicScore)
First we import the necessary libraries and open the data file. Then we read every line from the file, using a loop, and split the data into its constituent parts. It so happens that the raw heart data is the third number on every line.¶¶¶
# read and process every line
skinData = [] # holds skin data
heartData = [] # holds heart data
for line in data:
time, skin, heart = split(line) # extract the three values
skin = float(skin) # convert from string to float
heart = float(heart) # convert from string to float
skinData.append(skin) # keep the skin data
heartData.append(heart) # keep the heart data
# now, heartData contains all the heart values
data.close() # done, so let's close the file
Notice how we create an empty list, heartData, to hold the heart rate data from the data file. Inside the loop, we get a line from the data file at a time, extract the value of interest (i.e., raw heart data), and append it to this list. This is a very common pattern in data manipulation, called the accumulator pattern.
Definition: The accumulator pattern uses a loop to accumulate data, one item per iteration, in some container. For example, appending to a list, concatenating to a string, or adding to an integer variable.
Notice how we use the string function, split(line), to separate the single line (a string) to be split into a list of three substrings. The statement
time, skin, heart = split(line)
automatically “unpacks” the list of three substrings returned by split(line) to three strings and assigns each to the corresponding variable.
One thing about big data is that the data files may need some “massaging” (preprocessing) to end up in a format we want. In the above example, we assume that the data file has gone through this step, and thus no error checking is required in the code.
Python provides a shorthand operation to assign several variables, in parallel, to corresponding values.
For example,
>>> o, p, q = [4, 5, 6]
>>> o
4
>>> p
5
>>> q
6
Notice how the three variables, o, p, and q, automatically get assigned to each item in the list.
>>> a, b, c = "123"
>>> a
'1'
>>> b
'2'
>>> c
'3'
This also works with strings (since both strings and lists are sequences). Finally, parallel assignment allows us to easily swap values between two variables:
>>> x = 2
>>> y = 3
>>> x
2
>>> y
3
>>> x, y = y, x
>>> x
3
>>> y
2
Notice how x and y are originally 2 and 3, respectively. After the parallel assignment (i.e., x, y = y, x), their values have been reversed. Parallel assignment is a very convenient operation.
Let’s get back to the program. The line
heart = float(heart)
converts the string representation of a number to a float number. This is necessary since data from a file is read in as a string.
Upon completion of the loop, all skin and heart data values are now accumulated in the skinData and heartData lists.
The rest of the program performs the sonification of these values. First we create the musical data structure,
##### define the data structure
biomusicScore = Score("Biosignal sonification", 60)
biomusicPart = Part(PIANO, 0)
biomusicPhrase = Phrase()
Then we find the minimum and maximum data values (needed by mapValue() below).
# let's find the range extremes
minValue = min(heartData)
maxValue = max(heartData)
skinMinValue = min(skinData)
skinMaxValue = max(skinData)
We loop through the list of float numbers and sonify them. We have a few sonification rules:
This is just one of many possible mappings of the data. The following code implements this sonification design:
# let's sonify the data
i = 0; # point to first value in data
while i < len(heartData): # while there are more values, loop
# map skin-conductance to pitch
pitch = mapScale(skinData[i], skinMinValue, skinMaxValue, C3, C6, MAJOR_SCALE, C4)
# map heart data to a variation of pitch
pitchVariation = mapScale(heartData[i], heartMinValue, heartMaxValue, 0, 24, MAJOR_SCALE, C4)
# also map heart data to dynamic
dynamic = mapValue(heartData[i], heartMinValue, heartMaxValue, 0, 127)
# finally, combine pitch, pitch variation, and dynamic into note
note = Note(pitch + pitchVariation, TN, dynamic)
# add it to the melody so far
biomusicPhrase.addNote(note)
# point to next value in heart and skin data
i = i + 1
Notice the use of a while loop (instead of a for loop). This is equivalent to
for i in range(len(heartData)):
Notice how, with the while loop, we need to manually handle the initialization of i to 0 (right before the loop) and the increment of i (i.e., i = i + 1, at the very end of the loop).
Upon completion of the loop, biomusicPhrase contains all the notes resulting from sonifying the heart data. The rest of the program is straightforward. Notice how the View.sketch() function creates a visualization (of the sonification) of the data. Actually, we could focus more on this visual representation, as opposed to the sonification itself. In other words, we could use the musical notes as an intermediate form to create visualizations of data. In the next section, we will see another way to possibly create visualizations, by manipulating image files. Still, the primary focus of this book remains making music with Python.
So far, we have seen various useful Python functions, such as abs(), round(), len(), mapValue(), and upper(). These functions are already built into Python (or one of its libraries), so that you can easily perform a needed task. For example,
>>> x = -3
>>> x = abs(x)
>>> x
3
>>> y = 3.6
>>> y = round(y)
>>> y
4.0
>>> z = ["this", "is", "fun"]
>>> len(z)
3
A nice feature of Python is that it allows you to create your own functions.
This is useful when you have code that performs a common task, which you end up using many times in your program. Defining a function allows you to “package” that code, so that you can reuse it easily, as we saw earlier.
Definition: A Python function is a piece of code that performs a specific task. Once defined, it may be used many times, without having to write it again and again.
Functions are very easy to define. Simply take the code you want to package as a function, indent it (by, say, 3 spaces), and add a header line:
def functionName():
<your indented code goes here>
For example, let us define a function that plays a violin note:
# define function to play a C4 quarter note using violin sound
def playViolinNote():
note = Note(C4, QN)
phrase = Phrase()
phrase.addNote(note)
phrase.setInstrument(VIOLIN)
Play.midi(phrase)
The first line, in bold, is called the function header. It specifies the name to use when calling this function. Any arguments the function may need to get its job done are listed between the parentheses. In the earlier example, function playViolingNote() requires no argument.
Now, we can call function playViolinNote() as follows††††:
playViolinNote()
Every time you call this function, it plays a C4 quarter note using a violin timbre.
Functions are more useful (or general) when you can pass information to them, to customize what they do. For example, the preceding function definition can become more versatile by allowing us to specify the note’s pitch and duration:
# define function to play a specified note using violin sound
def playViolinNote(pitch, duration):
note = Note(pitch, duration)
phrase = Phrase()
phrase.addNote(note)
phrase.setInstrument(VIOLIN)
Play.midi(phrase)
Now, we can call this function as follows:
>>> playViolinNote(C4, WN)
>>> playViolinNote(E4, HN)
>>> playViolinNote(G4, QN)
to play different notes. Notice how much more versatile this function has become.
In summary, functions hide unnecessary details from the main code. Also, they allow us to reuse code efficiently, because we define a function once and use it many times. Without functions, our programs would be much larger and harder to understand and modify.
Define a function called playNote() which accepts a pitch, duration, and instrument. (You may assume the music library has already been imported.) Try it out with different notes and instruments (see Appendix A).
Python functions may return a value. This can be very useful when you are “packaging” code that produces a value (e.g., a number, a musical phrase, etc.) to be used by the caller.
For example, here is our own implementation of Python’s abs() function. This function returns the absolute value of a number.
# returns the absolute value of a number
def absolute(number):
if number < 0: # is the argument negative?
number = -1 * number # yes, so make it positive
# now, the number is positive (either way)
return number
Notice the return statement at the bottom of the function body. When evaluated, this statement passes back the specified value to the caller of the function. For example,
>>> x = absolute(-2.3)
>>> x
2.3
In other words, when we call absolute(−2.3), the argument, −2.3, is associated with the parameter number (see function definition) above. Then the body of the function executes with variable number having the value −2.3. The if statement’s condition evaluates to True, and so the sign of number changes (multiplying a number by −1 changes its sign). Finally, the return statement takes the value of variable number and returns it to the caller (as seen in the preceding interpreter example).
A return statement is similar to a print statement. Both take a value (or expression) and send it somewhere else. The print statement sends it to the computer screen. The return statement sends the value back to the statement that called the function. The difference is that, unlike print, the return statement also stops the function from executing, so that a value may be returned to the caller, and so the caller can continue its work.
A function may have several return statements. As soon as one is encountered, the function stops executing, and control returns to the caller. However, it is considered bad style to have more than one return statement.
Good Style: A function should have at most one return statement.‡‡‡‡
If a function does not have a return statement, it stops executing after the last statement in its body. Then control returns to the caller.
Let us see another example.
Python provides various functions for lists, such as len(), sum(), min(), and max(). However, Python does not provide a function for calculating the average of a list. Here we define such a function:
# returns the average of a numeric list
def avg(someList):
total = sum(someList) # get the total of all items
length = len(someList) # get the number of items
# next, convert to float (for accuracy) and calculate average
result = float(total) / float(length)
# finally, return result to caller
return result
raise ZeroDivisionError(string)
where string contains the error message. This will result in a typical Python error message similar to the ones we first saw in Chapter 2.
phrase = createMelody(10, PENTATONIC_SCALE, [QN, SN, EN])
should generate a phrase of 10 notes from the pentatonic scale with durations randomly selected from the provided list. (Hint: See the “Pentatonic Melody Generator” case study in Chapter 6.)
pitches = [A5, G5, F5, E5, D5, C5, B4, A4]
durations = [HN, QN, HN, QN, HN, QN, HN, QN]
voice = createCantusVoice(pitches, durations, 1.0)
should generate a phrase with 8 notes starting with an A5 half note (HN), followed by a G5 quarter note (QN),... and ending with an A4 quarter note. Whereas the following
voice = createCantusVoice(pitches, durations, 2.0)
should generate a phrase with 8 notes starting with an A5 whole note (WN), followed by a G5 half note (HN),... and ending with an A4 half note. (Hint: You may use Mod.elongate() in the body of your function.)
pitches = [A5, G5, F5, E5, D5, C5, B4, A4]
durations = [HN, QN, HN, QN, HN, QN, HN, QN]
voice1 = createCantusVoice(pitches, durations, 0, 1.0)
voice2 = createCantusVoice(pitches, durations, -12, 2.0)
voice3 = createCantusVoice(pitches, durations, -24, 4.0)
voice4 = createCantusVoice(pitches, durations, -36, 8.0)
to generate the different voices in the piece before adding them to a part.
In Python, variables that are declared outside a function are considered global; they have global scope. Global variables are visible anywhere within the program and can be accessed even from within functions. Variables that are defined within a function are only visible (accessible) within that function. We say that the scope of a variable is limited to the function.
Definition: The scope of a variable is the space within a program where that variable is visible (i.e., it can be accessed and possibly modified).
Ideally, a function should avoid using global variables.
Good Style: If a function needs any data, then that data should be passed to it through its parameter list.
Good Style: If a function changes any data outside its scope, it should return them via the return statement.
One very interesting application of sonification is to make music from aesthetically pleasing or otherwise interesting images. Again, the idea is to find ways to map interesting patterns in one medium (in this case, images) to interesting patterns in another medium (in this case, sound). Image sonification is accomplished by reading in an image, which is a two-dimensional list of pixels (or pictures elements), and exploring ways to map these pixels to notes. Each pixel contains information about color in one small area of a picture.
Your Python programs can read, manipulate, and write images through the image library. To access the image library, import it.
from image import *
An image is represented by Python as a two-dimensional table (i.e., matrix) of pixels.
Definition: An image is a table of x (width) * y (height) pixels.
For example, the following code creates a blank (black) image of 100 by 100 pixels.
img = Image (100, 100)
So an image has width * height number of pixels. The origin (0, 0) is at top left.
Definition: A pixel is a picture element or a point in an image.
Each pixel contains a list of three values; the red, green, and blue (or RGB) components of that point in the image. Each RGB value ranges from 0 to 255. For example,
The Python image library provides various useful functions.§§§§ The most common ones are shown in Table 7.2.
Image library functions (assume img is an image)
Function |
Description |
Image(filename) |
Reads in a jpg or png file called filename (a string) and shows an image. It returns the image, so it should be stored in a variable, for example, img = Image("sunset.jpg") |
Image(width, height) |
Returns an empty (blank) image with provided width and height . It returns the image, so it should be stored in a variable, for example, img = Image(200, 300) |
img.getWidth() |
Returns the width of image img . |
img.getHeight() |
Returns the height of image img . |
img.getPixel(col, row) |
Returns this pixel’s RGB values (a list, e.g., [255, 0, 0]), where col is the image column, and row is the image row. The image origin (0, 0) is at top left. |
img.setPixel(col, row, RGBlist) |
Sets this pixel’s RGB values, for example [255, 0, 0], where col is the image column, and row is the image row. The image origin (0, 0) is at top left. |
img.show() |
Displays the image img in a window. |
img.hide() |
Hides the image window (if any). |
img.write(filename) |
Writes image img to the jpg or png filename . |
The following case study applies some of these functions to generate music from images.
A soundscape refers to a musical composition that incorporates sounds recorded from, and/or music that depicts the characteristics of, an environment (e.g., a city soundscape or a forest soundscape).
In this case study, we explore how, through sonification of image data using image library functions, we create interesting musical artifacts by mapping visual aspects of an image into corresponding musical aspects. Also, similarly to sonification of other media (e.g., biosignals, weather data, financial markets, etc.), we may capture and explore structural characteristics of images that may not be as evident visually.
As we mentioned earlier, images consist of pixels (or picture elements). A pixel is the elemental data that digitally represents a single point in the original scene (as captured by a camera). The number of pixels in an image depends on the quality (or resolution) of the digital camera. For example, Figure 7.4 (depicting a sunset at the Loutraki seaside resort in Greece) consists of 320 × 213 pixels.
Similarly to other sonification activities, the ways we can map pixels to sound are not prescribed. A rule of thumb is to find what inspires you about a particular image and explore how you might convert that to sound. So image sonification involves imagination and artistic exploration.
The image in Figure 7.4 has a very nice gradient that gets brighter from left to right. The sun is not shown but can be imagined. There is a clear horizontal division between the sea and sky. The mountains, on the left, provide a contrast to the color of the sea and sky. Finally, the image gradient is interrupted by the (somewhat noisy) visual layers and the sea at the bottom half of the image. Clearly, there is enough structural variety in the visual domain to provide interesting analogies in the musical domain. All this can be exploited by selecting certain rows (or columns) of pixels, scanning the image left-to-right (or up-and-down), and converting individual pixels or areas of pixels to musical notes or passages.
In this case study, we use the following sonification rules:
These mappings were selected as a result of experimentation, simply because they sounded interesting, given this image. A little experimentation will go a long way.
Using the same sonification scheme with other images will likely generate interesting results. For instance, you could select a new image with this scheme in mind. Or you could create/modify an image (e.g., via Photoshop) with the particular sonification scheme in mind.
However, the most appropriate way is to pick an image and then design a set of sonification rules to use that matches its features. The image choice and sonification rules are intimately connected.
The following program implements the preceding rules to sonify the image in Figure 7.4.
# sonifyImage.py
#
# Demonstrates how to create a soundscape from an image.
# It also demonstrates how to use functions.
# It loads a jpg image and scans it from left to right.
# Pixels are mapped to notes using these rules:
#
# + left-to-right column position is mapped to time,
# + luminosity (pixel brightness) is mapped to pitch within a scale,
# + redness (pixel R value) is mapped to duration, and
# + blueness (pixel B value) is mapped to volume.
#
from music import *
from image import *
from random import *
##### define data structure
soundscapeScore = Score("Loutraki Soundscape", 60)
soundscapePart = Part(PIANO, 0)
##### define musical parameters
scale = MIXOLYDIAN_SCALE
minPitch = 0 # MIDI pitch (0-127)
maxPitch = 127
minDuration = 0.8 # duration (1.0 is QN)
maxDuration = 6.0
minVolume = 0 # MIDI velocity (0-127)
maxVolume = 127
# start time is randomly displaced by one of these
# durations (for variety)
timeDisplacement = [DEN, EN, SN, TN]
##### read in image (origin (0, 0) is at top left)
image = Image("soundscapeLoutrakiSunset.jpg")
# specify image pixel rows to sonify - this depends on the image!
pixelRows = [0, 53, 106, 159, 212]
width = image.getWidth() # get number of columns in image
height = image.getHeight() # get number of rows in image
##### define function to sonify one pixel
# Returns a note wrapped in a phrase created from sonifying
# the RGB values of 'pixel' found on given column.
def sonifyPixel(pixel, col):
red, green, blue = pixel # get pixel RGB value
luminosity = (red + green + blue) / 3 # calculate brightness
# map luminosity to pitch (the brighter the pixel, the higher
# the pitch) using specified scale
pitch = mapScale(luminosity, 0, 255, minPitch, maxPitch, scale)
# map red value to duration (the redder the pixel, the longer
# the note)
duration = mapValue(red, 0, 255, minDuration, maxDuration)
# map blue value to dynamic (the bluer the pixel, the louder
# the note)
dynamic = mapValue(blue, 0, 255, minVolume, maxVolume)
# create note and return it to caller
note = Note(pitch, duration, dynamic)
# use column value as note start time (e.g., 0.0, 1.0, and so on)
startTime = float(col) # time is a float
# add some random displacement for variety
startTime = startTime + choice(timeDisplacement)
# wrap note in a phrase to give it a start time
# (Phrases have start time, Notes do not)
phrase = Phrase(startTime) # create phrase with given start time
phrase.addNote(note) # and put note in it
# done sonifying this pixel, so return result
return phrase
##### create musical data
# sonify image pixels
for row in pixelRows: # iterate through selected rows
for col in range(width): # iterate through all pixels on this row
# get pixel at current coordinates (col and row)
pixel = image.getPixel(col, row)
# sonify this pixel (we get a note wrapped in a phrase)
phrase = sonifyPixel(pixel, col)
# put result in part
soundscapePart.addPhrase(phrase)
# now, all pixels on this row have been sonified
# now, all pixelRows have been sonified, and soundscapePart
# contains all notes
##### combine musical material
soundscapeScore.addPart(soundscapePart)
##### view score and write it to an audio and MIDI files
View.sketch(soundscapeScore)
Write.midi(soundscapeScore, "soundscapeLoutrakiSunset.mid")
First, notice how the top-level comments describe what the program does and, in particular, the sonification rules employed by it. Keeping such comments at the beginning allows you to understand what a program does several months/years afterwards, without having to read the code again line for line. It takes great discipline to write quality comments. However, it saves time later on.¶¶¶¶ Actually, experienced programmers first write comments and then code. A common programming aphorism goes, “if you cannot comment it, you cannot code it.” This is especially true for writing elegant code.
Notice how we have grouped the definitions of musical parameters at the top. These values are set once and are used throughout the program.
Next, we load the image and define a list of pixel rows to sonify. The program generates one melodic line from each pixel row. Figure 7.5 shows these pixel rows, where row 0 is at the top, row 53 is the second from the top, and so on.
pixelRows = [0, 53, 106, 159, 212]
Usually, it is a bad idea to hard-code such numbers in a program. For instance, what would happen if we changed the image we read in (e.g., made it smaller or larger)? We make an exception here and hard code the numbers. Our rationale is that the choice of rows to sonify depends on the selected image. Hence, we can think of these as constants related to the selected image.
Next comes the function definition. This function expects a pixel and its column coordinate. The pixel provides the RGB values for pitch, duration, and dynamic. The column is used to specify note start time (the piece evolves from left to right). By encapsulating the sonification scheme in a function, we help modularize our code. This makes it easier to change our code in the future, for example, to create different sonification schemes (and encapsulate them into different functions).
##### define function to sonify one pixel
# Returns a note wrapped in a phrase created from sonifying
# the RGB values of 'pixel' found on given column.
def sonifyPixel(pixel, col):
red, green, blue = pixel # get pixel RGB value
luminosity = (red + green + blue) / 3 # calculate brightness
# map luminosity to pitch (the brighter the pixel, the higher
# the pitch) using specified scale
pitch = mapScale(luminosity, 0, 255, minPitch, maxPitch, scale)
# map red value to duration (the redder the pixel, the longer
# the note)
duration = mapValue(red, 0, 255, minDuration, maxDuration)
# map blue value to dynamic (the bluer the pixel, the louder
# the note)
dynamic = mapValue(blue, 0, 255, minVolume, maxVolume)
# create note and return it to caller
note = Note(pitch, duration, dynamic)
# use column value as note start time (e.g., 0.0, 1.0, and so on)
startTime = float(col) # time is a float
# add some random displacement for variety
startTime = startTime + choice(timeDisplacement)
# wrap note in a phrase to give it a start time
# (Phrases have start time, Notes do not)
phrase = Phrase(startTime) # create phrase with given start time
phrase.addNote(note) # and put note in it
# done sonifying this pixel, so return result
return phrase
Some general things to know about functions:
Now let us explore what this function does. The first statement,
red, green, blue = pixel
is parallel assignment. Variable pixel contains a list of three RGB values, for example, [243, 124, 0]. The parallel assignment provides a convenient way to “unpack” the three RGB values from pixel.
The rest of the function body uses red, green, and blue to calculate the note pitch, duration, and dynamic, according to the sonification scheme described earlier.
The following statements
startTime = float(col) # time is a float
# add some random displacement for variety
startTime = startTime + choice(timeDisplacement)
use the column coordinate of the pixel to specify start time for the note created for this pixel. Since notes do not have a start time, we wrap the note into a phrase and set the phrase’s start time accordingly. Finally, since the columns are pretty rigid, that is, they take integral values (i.e., 0, 1, 2, 3, and so on), we add a random displacement (selected from the list timeDisplacement, i.e., [DEN, EN, SN, TN]). This makes the music less “square.”
The last statement in the function
return phrase
returns the created phrase to the function caller.
Next we use two loops, one inside the other. Again, this is called a nested loop.
The outer loop iterates as usual. For each iteration of the outer loop, the inner loop does a complete run (i.e., it does all its iterations).***** A helpful trick is to think of the inner loop as a simple statement. As all statements in the body of the outer loop, it needs to be executed completely.
##### create musical data
# sonify image pixels
for row in pixelRows: # iterate through selected rows
for col in range(width): # iterate through all pixels on this row
# get pixel at current coordinates (col and row)
pixel = image.getPixel(col, row)
# sonify this pixel (we get a note wrapped in a phrase)
phrase = sonifyPixel(pixel, col)
# put result in part
soundscapePart.addPhrase(phrase)
# now, all pixels on this row have been sonified
# now, all pixelRows have been sonified, and soundscapePart
# contains all notes
As the earlier comments indicate, every time the inner loop finishes, one more row of pixels has been sonified. The inner loop gets every pixel on this row, sonifies it (using the sonifyPixel() function), and adds the sonified result (a phrase) to soundscapePart.
The outer loop repeats this process for every row in the pixelRows list. Changing pixelRows (at the top of the program) will automatically adjust which areas of the image are sonified.
We end the program as usual, that is, by putting the part in a score, visualizing the score, and saving the score in a MIDI file.
The earlier program is only the beginning. Many avenues for experimentation exist.
This chapter has explored some ideas and programming techniques for sonifying and visualizing data. Importantly, they may be used to perceive hidden, important patterns in big data. Once you understand the principles of sonification, the sky is the limit! Topics covered include data scaling using the mapValue() and mapScale() functions. A number of Python language features have been introduced. These include strings, reading and writing files, the while loop, and how to define your own functions. These have been applied to sonification case studies that demonstrate how these can be used for both artistic and scientific objectives. Explore and enjoy.
* The default is CHROMATIC_SCALE.
† Actually, mapScale() has an extra, optional parameter, key, which could specify a key different from the one implied by minResult.
‡ In music theory, such a list is called a pitch class set. Pitch class sets have been used extensively by modern composers.
§ The print statement was covered in Chapter 2.
¶ ASCII stands for American Standard for Information Interchange.
** Python sequences (such as strings and lists) are always indexed starting at 0, that is, the first item is at position 0.
†† Chances are you will find the functions you need in the string library. If not, you should be able to easily construct what you need building upon the tools in the library.
‡‡ We have seen this idea before. For instance, in Chapter 3, we used parallel lists of pitches and durations to create phrases of notes.
§§ For example, if the outer loop iterates 5 times and the inner loop iterates 10 times, then for each iteration of the outer loop, the inner loop will iterate 10 times, for a total of 50 times.
¶¶ Python has similar functions to convert data to other Python data types as needed, such as int(), float(), bool(), etc.
*** Clearly, in order for a while loop to terminate, the variable (or variables) used in the loop condition have to be altered inside the loop body. Why?
††† Actually, an even better test would include checking the data type of the input, pitch, to ensure that the user entered an integer. This would be accomplished using the function type(pitch) seen in Chapter 2.
‡‡‡ Compare with the definition of data sonification at the beginning of the chapter.
§§§ Nevertheless, the concepts presented here (e.g., data mapping) also apply to visualization.
¶¶¶ In most sonification (or other data processing) tasks, you may need to clean up the data file (e.g., remove a header line or some copyright text) before loading it into your program. Doing this manually (i.e., with a text editor) saves time and simplifies your code.
**** Of course, note duration is another valuable musical parameter which could be used effectively to communicate patterns in data.
†††† For simplicity, let us assume the music library has already been imported.
‡‡‡‡ This is a special case of the software engineering guideline “one-way in, one-way out” that suggests there should be only one way to get into a piece of code and only one way to get out of it. By following this guideline, your code becomes easier to understand and modify. In the case of multiple return statements, this guideline is violated. Most modern programming languages provide structures that enforce the one-way in, one-way out style of programming.
§§§§ The image library is not standard. Similarly to the music library, it is provided with this textbook.
¶¶¶¶ As mentioned earlier, most programs are written once, but modified over and over.
***** For example, if the outer loop iterates 5 times and the inner loop iterates 10 times, then for each iteration of the outer loop, the inner loop will iterate 10 times, for a total of 50 times.