7.1.1 Inertial Measurement Unit

Our sensing means will be the LPMS-B2 IMU that has Bluetooth. The range is sufficient to work in a ballet studio.

The IMU has many other outputs that we will not use. A close-up of the IMU is shown in Figure 7.2. Table 7.1 gives the parameters for the IMU.

Table 7.1

LPMS-B2: 9-Axis Inertial Measurement Unit (IMU).

Parameter	Description
Bluetooth	2.1 + EDR / Low Energy (LE) 4.1
Communication distance	<  20 m
Orientation range	Roll: ± 180∘; Pitch: ± 90∘; Yaw: ± 180∘
Resolution	< 0.01∘
Accuracy	< 0.5∘(static), < 2∘ RMS (dynamic)
Accelerometer	3 axes, ± 2∕ ± 4∕ ± 8∕ ± 16 g, 16 bits
Gyroscope 3 axes	± 125∕ ± 245∕ ± 500∕ ± 1000∕ ± 2000∘/s, 16 bits
Data output format	2 raw data / Euler angle / quaternion
Data transmission rate up to	2400Hz

We will first work out the details of the data acquisition. We will then build a deep learning algorithm to train the system and later take data and classify the pirouette as being a pirouette done by a particular dancer. We’ll build up the data acquisition by first writing the MATLAB code to acquire the data. We will then create functions to display the data. We will then integrate it all into a GUI. Finally, we will create a deep learning classification system.

7.1.2 Physics

A pirouette is a complex multiflexible body problem. The pirouette is initiated by the dancer doing a plié and then using their muscles to generate a torque about the spin axis and forces to get onto their pointe shoe and over their center of mass. Their muscles quickly stop the translational motion so that they can focus on balancing as they are turning. We will assume that the dancer has a rigid body. Once a dancer is in the pirouette position, this is true. A dancer spots their head, that is, holds it focused on a target until the body has turned too far and then rapidly rotates it back to the target. However, we will ignore this for simplicity. The equation of rotational motion is known as Euler’s equation and is

(7.1)

where ω is the angular rate and I is the inertia. T is our external torque. The external torque is due to a push off the floor and gravity. This is a vector equation. The vectors are T and ω. I is a 3×3 matrix.

(7.2)

(7.3)

Each component is a value about a particular axis. For example, T_x is the torque about the x-axis attached to the dancer. Figure 7.3 shows the system. We will only be concerned with rotation assuming all translational motion is damped. If the dancer’s center of mass is not above the box of their pointe shoe, they will experience an overturning torque.

The dynamical model is three first-order coupled differential equations. Angular rate ω is the state, that is, the quantity being differentiated. The equation says that the external torque (due to pushing off the floor or due to pointe shoe drag) is equal to the angular acceleration plus the Euler coupling term. This equation assumes that the body is rigid. For a dancer, it means they are rotating and no part is moving with respect to any other part. Now in a proper pirouette, this is never true if you are spotting. But let’s suppose you are one of those dancers who don’t spot. Let’s forget about the angular rate coupling term, which only matters if the angular rate is large. Let’s just look at the first two terms which is . Expanded

(7.4)

Let’s look at the equation for T_z. We just multiply the first row of the inertia matrix times the angular rate vector:

(7.5)

This means a torque around the z-axis influences the angular rates about all three axes.

We can write out the required torque:

(7.6)

This is the perfect pirouette push-off because it only creates rotation about the vertical axis, which is what we want in a pirouette. To this, you need to add the forces needed to get on pointe with your center of mass over your pointe shoe tip.

While turning, the only significant external torque is due to friction between the pointe shoe tip and the floor. Friction resists both turning motion and translational motion. You don’t want a slight side force, perhaps due to a less than a great partner, to cause you to slide.

Our IMU measures angular rates and linear accelerations. Angular rates are the quantities in the Euler equations. However, since the IMU is not at the dancer’s center of mass, it will also measure angular accelerations along with the acceleration of the center of mass. We locate it at the dancer’s waist so it is not too far from the spin axis, but it still sees a component:

(7.7)

where r_IMU is the vector from the dancer’s center of mass to the IMU.

For a dancer doing a pirouette, Euler’s equation is not sufficient. A dancer can transfer momentum internally to stop a pirouette and needs a little jump to get on demi-pointe or pointe. To model this, we add additional terms:

(7.8)

(7.9)

(7.10)

(7.11)

where m is the mass, F is the vertical force, and z is the vertical direction. The vertical force is the difference between the dancer’s leg force upward and gravity downward. I_i is the internal inertia for control, and I_s is the head inertia. I includes both of these already. That is, I is the total body inertia that includes the internal “wheel,” body and head. T_i is the internal torque. T_h is the head torque (for spotting). The internal torques, T_i and T_h, are between the body and the internal “wheel” or head. For example, T_h causes the head to move one way and the body the other. If you are standing, the torque you produce from your feet against the floor prevents your body from rotating. T is the external torque due to friction and the initial push-off by the feet. The unit vector is

(7.12)

There are six equations in total. The first is a vector equation with three components; the second two are scalar equations. The vector equation is three equations, and each scalar equation is just one equation. We can use these to create a simulation of a dancer. The second component models all z-axis internal rotation, including spotting.

7.2.1 Problem

We want to get data from the Bluetooth IMU.

7.2.2 Solution

We will use the MATLAB bluetooth function. We’ll create a function to read data from the IMU.

7.2.3 How It Works

We will write an interface for the Bluetooth device. First, make sure the IMU is charged. Connect it to your computer as shown in Figure 7.4. Push the button on the back. This turns it on and off. The status is indicated by the LED. The IMU comes with support software from the vendor, but you will not need any of their software as MATLAB does all the hard work for you.

Let’s try commanding the IMU. Type btInfo = instrhwinfo(’Bluetooth’) and you should get the following:

This shows that your IMU is discoverable. There is no support package available from MathWorks. Now type b = Bluetooth(btInfo.RemoteIDs1,1) (this can be slow). The number is the channel. The Bluetooth function requires the Instrument Control Toolbox for MATLAB.

Note that the Communication State Status shows closed. The device is opened by typing fopen(b). If you don’t have this device, just type

This says it cannot recognize remote names or IDs. You may need a support package for your device in this case.

We read the binary and put it into a data structure using DataFromIMU. typecast converts from bytes to float.

We’ve wrapped all of this into the script BluetoothTest.m. We print out a few samples of the data to make sure our bytes are aligned correctly.

The first number in each row vector is the sample, the next three are the angular rates from the gyro, the next three are the accelerations, and the last four are the quaternion. The acceleration is mostly in the -z direction which means that +z is in the button direction. Bluetooth, like all wireless connections, can be problematic. If you get this error

turn the IMU on and off. You might also have to restart MATLAB at times. This is because RemoteNames is empty, and this test is assuming it will not be. MATLAB then gets confused.

7.3.1 Problem

We want to use quaternions to represent the orientation of our dancers in our deep learning system.

7.3.2 Solution

Implement basic quaternion operations. We need quaternion operations to process the quaternions from the IMU.

7.3.3 How It Works

Quaternions are the preferred mathematical representation of orientation. Propagating a quaternion requires fewer operations than propagating a transformation matrix and avoids singularities that occur with Euler angles. A quaternion has four elements, which corresponds to a unit vector a and angle of rotation ϕ about that vector. The first element is termed the “scalar component” s, and the next three elements are the “vector” components v. This notation is shown as follows [29]:

(7.13)

The “unit” quaternion which represents zero rotation from the initial coordinate frame has a unit scalar component and zero vector components. This is the same convention used on the Space Shuttle, although other conventions are possible.

(7.14)

In order to transform a vector from one coordinate frame a to another b using a quaternion q_ab, the operation is

(7.15)

using quaternion multiplication with the vectors defined as quaternions with a scalar part equal to zero, or

(7.16)

For example, the quaternion

(7.17)

represents a pure rotation about the x-axis. The first element is 0.7071 and equals the . We cannot tell the direction of rotation from the first element. The second element is the first component of the unit vector, which in this case is

(7.18)

times the argument . Since the sign is positive, the rotation must be a positive 90^∘ rotation.

We only need one routine that converts the quaternion, which comes from the IMU, into a transformation matrix for visualization. We do this because multiplying a 3× n array of vectors for the vertices of our 3D model by a matrix is much faster than transforming each vector with a quaternion.

Note that the diagonal terms have the same form. The off-diagonal terms also all have the same form.

7.4.1 Problem

We want to simulate a dancer for readers who don’t have access to the hardware.

7.4.2 Solution

We will write a right-hand side for the dancer based on the preceding equations and write a simulation with a control system.

7.4.3 How It Works

The right-hand side implements the dancer model. It includes an internal control “wheel” and a degree of freedom for the head movement. The default data structure is returned if you call it without arguments.

The remainder mechanizes the equations given earlier. We add an equation for the integral of the z-axis rate. This makes the control system easier to write. We also include the gravitational acceleration in the force equation.

The simulation setup gets default parameters from RHSDancer.

It then sets up the control system. We use a proportional derivative controller for the z position and a rate damper to stop the pirouette. The position control is done by the foot muscles. The rate damping is our internal damper wheel.

The simulation loop calls the right-hand side and the control system. We call RHS Dancer.m to get the linear acceleration.

The control system includes a torque and force pulse to get the pirouette going.

The remainder of the script plots the results and outputs the data, which would have come from the IMU, into a file.

Simulation results for a double pirouette are shown in Figure 7.5. We stop the turn at 6.5 seconds, hence the pulse.

You can create different dancers by varying the mass properties and the control parameters.

We didn’t implement spotting (looking at the audience as much as possible during the turn) control. It would rotate the head so that it faces forward whenever the head was within 90 degrees or so of the front. We’d need to add a head angle for that purpose to the right-hand side, much like we added the z-axis angle.

7.5.1 Problem

We want to display data from the IMU in real time. This will allow us to monitor the pirouettes.

7.5.2 Solution

Use plot with drawnow to implement multiple figures of plots.

7.5.3 How It Works

The main function is a switch statement with two cases. The function also has a built-in demo. The first case, initialize, initializes the plot figures. It stores everything in a data structure that is returned on each function call. This is one way for a function to have a memory. We return the data structure from each subfunction.

The first case, initialize, initializes the figure window.

The second case, update, updates the data displayed in the plot. It leaves the existing figures, subplots, and labels in place and just updates the plots of the line segments with new data. It can change the size of the axes as needed. The function adds a line segment for each new data point. This way, no storage is needed external to the plot. It reads xdata and ydata and appends the new data to those arrays.

The built-in demo plots six numbers. It updates the axes in time once. It sets up a figure window with six plots. You need to create the figure and save the figure handle before calling GUIPlots.

The pause in the demo just slows down plotting so that you can see the updates. The height (the last number in g.pos) is the height of each plot. If you happen to set the locations of the plots out of the figure window, you will get a MATLAB error. g.tLim gives the initial time limits in seconds. The upper limit will expand as data is entered.

Figure 7.6 shows the real-time plots at the end of the demo.

7.6.1 Problem

We want to display the dancer’s orientation in real time.

7.6.2 Solution

Use patch to draw an OBJ model in a three-dimensional plot. The figure is easier to understand than the four quaternion elements. Our solution can handle three-axis rotation although typically we will only see single-axis rotation.

7.6.3 How It Works

We start with our Ballerina.obj file. It only has vertices and faces. A 3D drawing consists of a set of vertices. Each vertex is a point in space. The vertices are organized into faces. Each face is a triangle. Triangles are used for 3D drawings because they always form a plane. 3D processing hardware is designed to work with triangles, so this also gives the fastest results. The obj files for our software can only contain triangles. Each face can have only three vertices. Generally, obj files can have any size polynomials, that is, faces with more than three points. Most sources of obj files can provide tessellation services to convert polygons with more than three vertices into triangles. LoadOBJ.m will not draw models with anything other than triangular faces.

The main part of the function uses a case statement to handle the three actions. The first action just returns the defaults, which is the name of the default obj file. The second reads in the file and initializes the patches. The third updates the patches. patch is the MATLAB name for a set of triangles. The function can be passed through a figure handle. A figure handle tells it into which figure the 3D model should be drawn. This allows it to be used as part of a GUI, as well shown in the next section.

Initialize loads the obj file. It creates a figure and saves the object data structure. It sets shading to interpolated and lighting to Gouraud. Gouraud is a type of lighting model named after its inventor. It then creates the patches and sets up the axis system. We save handles to all the patches for updating later. We also place a light near the object to illuminate the patches.

In Update, we convert the quaternion to a matrix because it is faster to matrix multiply all the vertices with one matrix multiplication. The vertices are n by 3 so we transpose before the matrix multiplication. We use the patch handles to update the vertices. The two options at the end are to create movie frames or just update the drawing.

This is the built-in demo. We vary the 1 and 4 elements of the quaternion to get rotation about the z-axis.

Figure 7.7 shows two orientations of the dancer during the demo. The demo produces an animation of the dancer rotating about the z-axis.

The rotation is slow because of the number of vertices. The figure is not articulated so the entire figure is rotated as a rigid body. We don’t employ texture maps. In any case, the purpose of this function is just to show orientation so it doesn’t matter.

7.7.1 Problem

We need to attach the IMU to our dancer.

7.7.2 Solution

We use the arm strap that is available from the manufacturer. We buy an elastic belt and make one that fits around the dancer’s waist.

7.7.3 How It Works

Remove the holder from the LPMS-B2 Holder. Cut the belt at the buckle and slide the holder onto the belt. Sew the belt at the buckle.

The sensor on a dancer is shown in Figure 7.9. We had the dancer stand near the laptop during startup. We didn’t have any range problems during the experiments. We didn’t try it with across-the-floor movement as one would have during grande allegro.

7.8.1 Problem

We want to test the data acquisition system. This will find any problems with the data acquisition process.

7.8.2 Solution

Have a dancer do changements, which are small jumps changing the foot position on landing.

7.8.3 How It Works

The dancer puts on the sensor belt, we push the calibrate button, then they do a series of changements. The dancer stands about 2 m from your computer to make acquisition easier. The dancer will do small jumps, known as changements. A changement is a small jump where the feet change positions starting from the fifth position. If the right foot is in the fifth position front at the start, it is in the back at the finish. Photos are shown in Figure 7.10.

The time scale is a bit long. You can see that the calibration does not lead us to a natural orientation in the axis system in the GUI. It doesn’t matter from a data collection point of view but is an improvement we should make in the future. The changement is shown in Figure 7.11. The dancer is still at the beginning and end.

The interface to the Bluetooth device doesn’t do any checking or stream control. Some Bluetooth data collection errors occur from time to time. Typically, they happen after 40 seconds of data collection.

If this happens, turn the device on and off. Restart MATLAB if that doesn’t work.

Another bluetooth error is

This is a MATLAB error and requires restarting MATLAB. It doesn’t happen very often. We ran the entire data collection with four dancers doing ten pirouettes each without ever experiencing the problem.

7.9.1 Problem

We want to classify the pirouettes of our four dancers.

7.9.2 Solution

Create an LSTM that classifies pirouettes according to the dancer. The four labels are the dancers’ names.

7.9.3 How It Works

The script takes one file and displays it. Figure 7.12 shows a double pirouette. It lasts only a few seconds.

We load in the data and limit the range to six seconds. Sometimes, the IMU would run longer due to human error (the data collection did not stop after the pirouette ended). We also remove bad sets.

Four Ryoko sets were lost due to errors in data collection, as shown in the following output:

We set up and then train the neural network. We use a bidirectional LSTM to classify the sequences. There are ten features: four quaternion measurements, three rate gyro measurements, and three accelerometer measurements. The four quaternion numbers are coupled through the relationship

(7.19)

However, this should not impact the learning accuracy aside from slowing down the learning.

The bidirectional LSTM is a good choice because we have access to the full sequence. For a classifier using bilstmLayer, we must set the ’outputMode’ to ’last’. This is followed by a fully connected layer, a softmax for producing normalized maximums, and finally the classification layer.

The layers as displayed are shown as follows:

The training GUI is shown in Figure 7.13. It converges fairly well.

We test the neural network against the unused data.

The accuracy achieved is printed as follows:

This result, >  80%, is pretty good considering the limited amount of data. Interestingly, the deep learning network could distinguish the dancers’ pirouettes. The data itself did not show any differences that were easy to identify to the human eye. Calibration could have been done better to make the data more consistent between dancers. It would have been interesting to collect data on multiple days. Other experiments would be to classify pirouettes done in pointe shoes and without. We might also have had the dancers do different types of turns to see if the network could still identify the dancer.

7.10.1 Problem

Build a data acquisition GUI to display the real-time data and output it into training sets.

7.10.2 Solution

Integrate all the preceding recipes into a GUI.

7.10.3 How It Works

We aren’t going to use MATLAB’s GUIDE or app tools to build our GUI. We will hand-code it, which will give you a better idea of how a GUI works.

We will use nested functions for the GUI. The inner functions have access to all variables in the outer functions. This also makes using callbacks easy as shown in the following code snippet:

A callback is a function called by a uicontrol when the user interacts with the control. When you first open the GUI, it will look for the Bluetooth device. This can take a while.

Everything in DrawGUI has access to variables in DancerGUI. The GUI is shown in Figure 7.14. The 3D orientation display is in the upper-left corner. Real-time plots are on the right. Buttons are on the lower left, and the movie window is on the right.

The remaining three lines display the time, the angular rate vector, and the acceleration vector as numbers. This is the same data that is plotted.

The first part creates the figure and draws the GUI. It initializes all the fields for GUIPlots. It reads in a default picture for the movie window as a placeholder.

The notation

is latex format. This will generate ω_x in the plot labels.

The next part tries to find Bluetooth. It first sees if Bluetooth is available at all. It then enumerates all Bluetooth devices. It looks through the list to find our IMU.

The following is the run loop. If no IMU is present, it synthesizes data. If the IMU is found, the GUI reads data from the IMU in 91-byte chunks. The uiwait is to wait until the user hits the start button. When used for testing, the IMU should be on the dancer. The dancer should remain still when the start button is pushed. It will then calibrate the IMU. Calibration fixes the quaternion reference and removes the gravitational acceleration. You can also hit the calibration button at any time.

The following closes the GUI. It uses a variable set in one of the callbacks.

This code displays the IMU data.

The drawing code uses uicontrol to create all the buttons. GUIPlots and Quaterni onVisualization are also initialized. The uicontrol that require an action have callbacks.

uicontrol takes parameter pairs, except for the first argument that can be a figure handle. There are a lot of parameter pairs. The easiest way to explore them is to type

All types of uicontrol that handle user interaction have “callbacks” that are functions that do something when the button is pushed or a menu item is selected. We have five uicontrol with callbacks. The start/stop button callback uses uiwait and uiresume to start and stop data collection.

The Quit button uses questdlg to ask if you want to save the data that has been stored in the GUI. This produces the modal dialog shown in Figure 7.15.

The Clear callback function clears the data storage arrays. It resets the quaternion to a unit quaternion.

The calibrate callback runs the calibration procedure.

The last, SaveFile, saves the recorded data into a mat-file for use by the deep learning algorithm.

We make it easier for the user to save files by reading the directory and adding a number to the end of the dancer filename that is one greater than the last filename number.