Resources

Here are some resources:

• FluidSynth home page ( http://sourceforge.net/apps/trac/fluidsynth/ )

• FluidSynth download page ( http://sourceforge.net/projects/fluidsynth/ )

• FluidSynth 1.1 developer documentation ( http://fluidsynth.sourceforge.net/api/ )

• SourceArchive’s fluidsynth documentation ( http://fluidsynth.sourcearchive.com/documentation/1.1.5-1/main.html )

Players

fluidsynth is a command-line MIDI player. It runs under ALSA with the following command line:

fluidsynth -a alsa -l <soundfont> <files...>

Play MIDI Files

The FluidSynth API consists of the following:

• A sequencer created using new_fluid_player

• A synthesizer created using new_fluid_synth

• An audio player created using new_fluid_audio_driver that runs in a separate thread

• A “settings” object that can be used to control many features of the other components, created with new_fluid_settings and modified with calls such as fluid_settings_setstr

A typical program to play a sequence of MIDI files using ALSA follows. It creates the various objects, sets the audio player to use ALSA, and then adds each sound font and MIDI file to the player. The call to fluid_player_play then plays each MIDI file in turn. This program is just a repeat of the program shown in Chapter 20.

#include <fluidsynth.h>
#include <fluid_midi.h>

int main(int argc, char** argv)
{
    int i;
    fluid_settings_t* settings;
    fluid_synth_t* synth;
    fluid_player_t* player;
    fluid_audio_driver_t* adriver;

    settings = new_fluid_settings();
    fluid_settings_setstr(settings, "audio.driver", "alsa");
    synth = new_fluid_synth(settings);
    player = new_fluid_player(synth);

    adriver = new_fluid_audio_driver(settings, synth);
    /* process command line arguments */
    for (i = 1; i < argc; i++) {
        if (fluid_is_soundfont(argv[i])) {
            fluid_synth_sfload(synth, argv[1], 1);
        } else {
            fluid_player_add(player, argv[i]);
        }
    }
    /* play the midi files, if any */
    fluid_player_play(player);
    /* wait for playback termination */
    fluid_player_join(player);
    /* cleanup */
    delete_fluid_audio_driver(adriver);
    delete_fluid_player(player);
    delete_fluid_synth(synth);
    delete_fluid_settings(settings);
    return 0;
}

Extending FluidSynth with Callbacks

Callbacks are functions registered with an application that are called when certain events occur. To build a karaoke player, you need to know the following:

• When a file is loaded so that you can extract all the lyrics from it for display at the right times

• When each meta lyric or text event occurs as output from a sequencer so that you can see what lyric is about to be sung

The first of these is fairly straightforward: FluidSynth has a function fluid_player_load that will load a file. You can change the code to add a suitable callback into that function that will give you access to the loaded MIDI file.

Getting lyric or text events out of a sequencer is not so easy, since they are never meant to appear! The MIDI specification allows these event types within a MIDI file, but they are not wire types so should never be sent from a sequencer to a synthesizer. The Java MIDI API makes them available by an out-of-band call to a meta event handler. FluidSynth just throws them away.

On the other hand, FluidSynth already has a callback to handle MIDI events sent from the sequencer to the synthesizer. It is the function fluid_synth_handle_midi_event and is set with the call fluid_player_set_playback_callback. What you need to do is to first alter the existing FluidSynth code so that lyric and text events are passed through. Then insert a new playback callback that will intercept those events and do something with them while passing on all other events to the default handler. The default handler will ignore any such events anyway, so it does not need to be changed.

I have added one new function to FluidSynth, fluid_player_set_onload_callback, and added appropriate code to pass on some meta events. Then it is a matter of writing an onload callback to walk through the MIDI data from the parsed input file and writing a suitable MIDI event callback to handle the intercepted meta events while passing the rest through to the default handler.

These changes have been made to give a new source package fluidsynth-1.1.6-karaoke.tar.bz2. If you just want to work from a patch file, that is fluid.patch. The patch has been submitted to the FluidSynth maintainers.

To build from this package, do the same as you normally would.

tar jxf fluidsynth-1.1.6-karaoke.tar.bz2
cd fluidsynth-1.1.6
./configure
make clean
make

To get ALSA support, you will need to have installed the libasound2-dev package, like for Jack and other packages. You probably won’t have many of them installed, so don’t run make install or you will overwrite the normal fluidsynth package, which will probably have more features.

The previous program modified to just print out the lyric lines and the lyric events as they occur is karaoke_player.c, shown here:

#include <fluidsynth.h>
#include <fluid_midi.h>

/**
 * This MIDI event callback filters out the TEXT and LYRIC events
 * and passes the rest to the default event handler.
 * Here we just print the text of the event, more
 * complex handling can be done
 */
int event_callback(void *data, fluid_midi_event_t *event) {
    fluid_synth_t* synth = (fluid_synth_t*) data;
    int type = fluid_midi_event_get_type(event);
    int chan = fluid_midi_event_get_channel(event);
    if (synth == NULL) printf("Synth is null
");
    switch(type) {
    case MIDI_TEXT:
        printf("Callback: Playing text event %s (length %d)
",
               (char *) event->paramptr, event->param1);
        return  FLUID_OK;

    case MIDI_LYRIC:
        printf("Callback: Playing lyric event %d %s
",
               event->param1, (char *) event->paramptr);
        return  FLUID_OK;
    }
    return fluid_synth_handle_midi_event( data, event);
}

/**
 * This is called whenever new data is loaded, such as a new file.
 * Here we extract the TEXT and LYRIC events and just print them
 * to stdout. They could e.g. be saved and displayed in a GUI
 * as the events are received by the event callback.
 */
int onload_callback(void *data, fluid_player_t *player) {
    printf("Load callback, tracks %d 
", player->ntracks);
    int n;
    for (n = 0; n < player->ntracks; n++) {
        fluid_track_t *track = player->track[n];
        printf("Track %d
", n);
        fluid_midi_event_t *event = fluid_track_first_event(track);
        while (event != NULL) {
            switch (event->type) {
            case MIDI_TEXT:
            case MIDI_LYRIC:
                printf("Loaded event %s
", (char *) event->paramptr);
            }
            event = fluid_track_next_event(track);
        }
    }
    return FLUID_OK;
}

int main(int argc, char** argv)
{
    int i;
    fluid_settings_t* settings;
    fluid_synth_t* synth;
    fluid_player_t* player;
    fluid_audio_driver_t* adriver;
    settings = new_fluid_settings();
    fluid_settings_setstr(settings, "audio.driver", "alsa");
    fluid_settings_setint(settings, "synth.polyphony", 64);
    synth = new_fluid_synth(settings);
    player = new_fluid_player(synth);

    /* Set the MIDI event callback to our own functions rather than the system default */
    fluid_player_set_playback_callback(player, event_callback, synth);

    /* Add an onload callback so we can get information from new data before it plays */
    fluid_player_set_onload_callback(player, onload_callback, NULL);

    adriver = new_fluid_audio_driver(settings, synth);
    /* process command line arguments */
    for (i = 1; i < argc; i++) {
        if (fluid_is_soundfont(argv[i])) {
            fluid_synth_sfload(synth, argv[1], 1);
        } else {
            fluid_player_add(player, argv[i]);
        }
    }
    /* play the midi files, if any */
    fluid_player_play(player);
    /* wait for playback termination */
    fluid_player_join(player);
    /* cleanup */
    delete_fluid_audio_driver(adriver);
    delete_fluid_player(player);
    delete_fluid_synth(synth);
    delete_fluid_settings(settings);
    return 0;
}

Assuming the new fluidsynth package is in an immediate subdirectory, to compile the program, you will need to pick up the local includes and libraries.

gcc -g -I fluidsynth-1.1.6/include/ -I fluidsynth-1.1.6/src/midi/ -I fluidsynth-1.1.6/src/utils/ -c -o karaoke_player.o karaoke_player.c

gcc karaoke_player.o -Lfluidsynth-1.1.6/src/.libs -l fluidsynth -o karaoke_player

To run the program, you will also need to pick up the local library and the sound font file.

export LD_LIBRARY_PATH=./fluidsynth-1.1.6/src/.libs/
./karaoke_player /usr/share/soundfonts/FluidR3_GM.sf2 54154.mid

The output for a typical KAR file is as follows:

Load callback, tracks 1
Track 0
Loaded event #
Loaded event 0
Loaded event 0
Loaded event 0
Loaded event 1
Loaded event

...

Callback: Playing lyric event 2 #
Callback: Playing lyric event 2 0
Callback: Playing lyric event 2 0
Callback: Playing lyric event 2 0
Callback: Playing lyric event 2 1
Callback: Playing lyric event 3

Displaying and Coloring Text with Gtk

While there are many ways in which karaoke text can be displayed, a common pattern is to display two lines of text: the currently playing line and the next one. The current line is progressively highlighted and on completion is replaced by the next line.

In Chapter 25 you did that. But the Java libraries have not been polished and are distinctly slow and heavyweight. They also seem to be low priority on Oracle’s development schedule for Java. So, here you will look at an alternative GUI and make use of the FluidSynth library. I chose the Gtk library for the reasons outlined in Chapter 15.

The first task is to build up an array of lyric lines as the file is loaded. You are asssuming KAR format files with up-front information as to the title, and so on, prefixed with @ and with newlines prefixed with .

struct _lyric_t {
    gchar *lyric;
    long tick;
};
typedef struct _lyric_t lyric_t;

struct _lyric_lines_t {
    char *language;
    char *title;
    char *performer;
    GArray *lines; // array of GString *
};
typedef struct _lyric_lines_t lyric_lines_t;

GArray *lyrics;
lyric_lines_t lyric_lines;

void build_lyric_lines() {
    int n;
    lyric_t *plyric;
    GString *line = g_string_new("");
    GArray *lines =  g_array_sized_new(FALSE, FALSE, sizeof(GString *), 64);

    lyric_lines.title = NULL;

    for (n = 0; n < lyrics->len; n++) {
        plyric = g_array_index(lyrics, lyric_t *, n);
        gchar *lyric = plyric->lyric;
        int tick = plyric->tick;

        if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'L')) {
            lyric_lines.language =  lyric + 2;
            continue;
        }

        if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'T')) {
            if (lyric_lines.title == NULL) {
                lyric_lines.title = lyric + 2;
            } else {
                lyric_lines.performer = lyric + 2;
            }
            continue;
        }

        if (lyric[0] == '@') {
            // some other stuff like @KMIDI KARAOKE FILE
            continue;
        }

        if ((lyric[0] == '/') || (lyric[0] == '')) {
            // start of a new line
            // add to lines
            g_array_append_val(lines, line);
            line = g_string_new(lyric + 1);
        }  else {
            line = g_string_append(line, lyric);
        }
    }
    lyric_lines.lines = lines;

    printf("Title is %s, performer is %s, language is %s
",
           lyric_lines.title, lyric_lines.performer, lyric_lines.language);
    for (n = 0; n < lines->len; n++) {
        printf("Line is %s
", g_array_index(lines, GString *, n)->str);
    }
}

This is called from the onload callback.

int onload_callback(void *data, fluid_player_t *player) {
    long ticks = 0L;
    lyric_t *plyric;

    printf("Load callback, tracks %d 
", player->ntracks);
    int n;
    for (n = 0; n < player->ntracks; n++) {
        fluid_track_t *track = player->track[n];
        printf("Track %d
", n);
        fluid_midi_event_t *event = fluid_track_first_event(track);
        while (event != NULL) {
            switch (fluid_midi_event_get_type (event)) {
            case MIDI_TEXT:
            case MIDI_LYRIC:
                /* there's no fluid_midi_event_get_sysex()
                   or fluid_midi_event_get_time() so we
                   have to look inside the opaque struct
                */
                ticks += event->dtime;
                printf("Loaded event %s for time %d
",
                       event->paramptr,
                       ticks);
                plyric = g_new(lyric_t, 1);
                plyric->lyric = g_strdup(event->paramptr);
                plyric->tick = ticks;
                g_array_append_val(lyrics, plyric);
            }
            event = fluid_track_next_event(track);
        }
    }

    printf("Saved %d lyric events
", lyrics->len);
    for (n = 0; n < lyrics->len; n++) {
        plyric = g_array_index(lyrics, lyric_t *, n);
        printf("Saved lyric %s at %d
", plyric->lyric, plyric->tick);
    }

    build_lyric_lines();
}

The standard GUI part is to build an interface consisting of two labels, one above the other to hold lines of lyrics. This is just ordinary Gtk.

The final part is to handle lyric or text events from the sequencer. If the event is a , then the current text in a label must be replaced with new text, after a small pause. Otherwise, the text in the label has to be progressively colored to indicate what is next to be played.

In Chapter 15, I discussed using Cairo to draw in pixbufs and using Pango to structure the text. The Gtk label understands Pango directly, so you just use Pango to format the text and display it in the label. This involves constructing an HTML string with the first part colored red and the rest in black. This can be set in the label, and there is no need to use Cairo.

The program is gtkkaraoke_player.c.

#include <fluidsynth.h>
#include <fluid_midi.h>
#include <string.h>

#include <gtk/gtk.h>

/* GString stuff from https://developer.gnome.org/glib/2.31/glib-Strings.html
   Memory alloc from https://developer.gnome.org/glib/2.30/glib-Memory-Allocation.html
   Packing demo from https://developer.gnome.org/gtk-tutorial/2.90/x386.html
   Thread stuff from https://developer.gnome.org/gtk-faq/stable/x481.html
   GArrays from http://www.gtk.org/api/2.6/glib/glib-Arrays.html
   Pango attributes from http://www.ibm.com/developerworks/library/l-u-pango2/
   Timeouts at http://www.gtk.org/tutorial1.2/gtk_tut-17.html
 */

struct _lyric_t {
    gchar *lyric;
    long tick;

};
typedef struct _lyric_t lyric_t;

struct _lyric_lines_t {
    char *language;
    char *title;
    char *performer;
    GArray *lines; // array of GString *
};
typedef struct _lyric_lines_t lyric_lines_t;

GArray *lyrics;

lyric_lines_t lyric_lines;

fluid_synth_t* synth;

GtkWidget *lyric_labels[2];

fluid_player_t* player;

int current_panel = -1;  // panel showing current lyric line
int current_line = 0;  // which line is the current lyric
gchar *current_lyric;   // currently playing lyric line
GString *front_of_lyric;  // part of lyric to be coloured red
GString *end_of_lyric;    // part of lyric to not be coloured

gchar *markup[] = {"<span foreground="red">",
                   "</span><span foreground="black">",
                   "</span>"};
gchar *markup_newline[] = {"<span foreground="black">",
                   "</span>"};
GString *marked_up_label;

struct _reset_label_data {
    GtkLabel *label;
    gchar *text;
    PangoAttrList *attrs;
};

typedef struct _reset_label_data reset_label_data;

/**
 * redraw a label some time later
 */
gint reset_label_cb(gpointer data) {
    reset_label_data *rdata = ( reset_label_data *) data;

    if (rdata->label == NULL) {
        printf("Label is null, cant set its text 
");
        return FALSE;
    }

    printf("Resetting label callback to "%s"
", rdata->text);

    gdk_threads_enter();

    gchar *str;
    str = g_strconcat(markup_newline[0], rdata->text, markup_newline[1], NULL);

    PangoAttrList *attrs;
    gchar *text;
    pango_parse_markup (str, -1,0, &attrs, &text, NULL, NULL);

    gtk_label_set_text(rdata->label, text);
    gtk_label_set_attributes(rdata->label, attrs);

    gdk_threads_leave();

    GtkAllocation* alloc = g_new(GtkAllocation, 1);
    gtk_widget_get_allocation((GtkWidget *) (rdata->label), alloc);
    printf("Set label text to "%s"
", gtk_label_get_text(rdata->label));
    printf("Label has height %d width %d
", alloc->height, alloc->width);
    printf("Set other label text to "%s"
",
           gtk_label_get_text(rdata->label == lyric_labels[0] ?
                              lyric_labels[1] : lyric_labels[0]));
    gtk_widget_get_allocation((GtkWidget *) (rdata->label  == lyric_labels[0] ?
                              lyric_labels[1] : lyric_labels[0]), alloc);
    printf("Label has height %d width %d
", alloc->height, alloc->width);

    return FALSE;
}

/**
 * This MIDI event callback filters out the TEXT and LYRIC events
 * and passes the rest to the default event handler.
 * Here we colour the text in a Gtk label
 */
int event_callback(void *data, fluid_midi_event_t *event) {
    fluid_synth_t* synth = (fluid_synth_t*) data;
    int type = fluid_midi_event_get_type(event);
    int chan = fluid_midi_event_get_channel(event);
    if (synth == NULL) printf("Synth is null
");
    switch(type) {
    case MIDI_TEXT:
        printf("Callback: Playing text event %s (length %d)
",
               (char *) event->paramptr, event->param1);

        if (((char *) event->paramptr)[0] == '') {
            // we've got a new line, change the label text on the NEXT panel
            int next_panel = current_panel; // really (current_panel+2)%2
            int next_line = current_line + 2;
            gchar *next_lyric;

            if (current_line + 2 >= lyric_lines.lines->len) {
                return FLUID_OK;
            }
            current_line += 1;
            current_panel = (current_panel + 1) % 2;

            // set up new line as current line
            char *lyric =  event->paramptr;

            // find the next line from lyric_lines array
            current_lyric = g_array_index(lyric_lines.lines, GString *, current_line)->str;

            // lyric is in 2 parts: front coloured, end uncoloured
            front_of_lyric = g_string_new(lyric+1); // lose 
            end_of_lyric = g_string_new(current_lyric);
            printf("New line. Setting front to %s end to "%s"
", lyric+1, current_lyric);

            // update label for next line after this one
            char *str = g_array_index(lyric_lines.lines, GString *, next_line)->str;
            printf("Setting text in label %d to "%s"
", next_panel, str);

            next_lyric = g_array_index(lyric_lines.lines, GString *, next_line)->str;

            gdk_threads_enter();

            // change the label after one second to avoid visual "jar"
            reset_label_data *label_data;
            label_data = g_new(reset_label_data, 1);
            label_data->label = (GtkLabel *) lyric_labels[next_panel];
            label_data->text = next_lyric;
            g_timeout_add(1000, reset_label_cb, label_data);

            // Dies if you try to flush at this point!
            // gdk_flush();

            gdk_threads_leave();
        } else {
            // change text colour as chars are played, using Pango attributes
            char *lyric =  event->paramptr;
            if ((front_of_lyric != NULL) && (lyric != NULL)) {
                // add the new lyric to the front of the existing coloured
                g_string_append(front_of_lyric, lyric);
                char *s = front_of_lyric->str;
                printf("Displaying "%s"
", current_lyric);
                printf("  Colouring "%s"
", s);
                printf("  Not colouring "%s"
", current_lyric + strlen(s));

                // todo: avoid memory leak
                marked_up_label = g_string_new(markup[0]);
                g_string_append(marked_up_label, s);
                g_string_append(marked_up_label, markup[1]);
                g_string_append(marked_up_label, current_lyric + strlen(s));
                g_string_append(marked_up_label, markup[2]);
                printf("Marked up label "%s"
", marked_up_label->str);

                /* Example from http://www.ibm.com/developerworks/library/l-u-pango2/
                 */
                PangoAttrList *attrs;
                gchar *text;
                gdk_threads_enter();
                pango_parse_markup (marked_up_label->str, -1,0, &attrs, &text, NULL, NULL);
                printf("Marked up label parsed ok
");
                gtk_label_set_text((GtkLabel *) lyric_labels[current_panel],
                                   text);
                gtk_label_set_attributes(GTK_LABEL(lyric_labels[current_panel]), attrs);
                // Dies if you try to flush at this point!
                //gdk_flush();

                gdk_threads_leave();
            }
        }
        return  FLUID_OK;

    case MIDI_LYRIC:
        printf("Callback: Playing lyric event %d %s
",
               event->param1, (char *) event->paramptr);
        return  FLUID_OK;

    case MIDI_EOT:
        printf("End of track
");
        exit(0);
    }
    // default handler for all other events
    return fluid_synth_handle_midi_event( data, event);
}

/*
 * Build array of lyric lines from the MIDI file data
 */
void build_lyric_lines() {
    int n;
    lyric_t *plyric;
    GString *line = g_string_new("");
    GArray *lines =  g_array_sized_new(FALSE, FALSE, sizeof(GString *), 64);

    lyric_lines.title = NULL;

    for (n = 0; n < lyrics->len; n++) {
        plyric = g_array_index(lyrics, lyric_t *, n);
        gchar *lyric = plyric->lyric;
        int tick = plyric->tick;

        if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'L')) {
            lyric_lines.language =  lyric + 2;
            continue;
        }

        if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'T')) {
            if (lyric_lines.title == NULL) {
                lyric_lines.title = lyric + 2;
            } else {
                lyric_lines.performer = lyric + 2;
            }
            continue;
        }

        if (lyric[0] == '@') {
            // some other stuff like @KMIDI KARAOKE FILE
            continue;
        }

        if ((lyric[0] == '/') || (lyric[0] == '')) {
            // start of a new line
            // add to lines
            g_array_append_val(lines, line);
            line = g_string_new(lyric + 1);
        }  else {
            line = g_string_append(line, lyric);
        }
    }
    lyric_lines.lines = lines;

    printf("Title is %s, performer is %s, language is %s
",
           lyric_lines.title, lyric_lines.performer, lyric_lines.language);
    for (n = 0; n < lines->len; n++) {
        printf("Line is %s
", g_array_index(lines, GString *, n)->str);
    }

}

/**
 * This is called whenever new data is loaded, such as a new file.
 * Here we extract the TEXT and LYRIC events and save them
 * into an array
 */
int onload_callback(void *data, fluid_player_t *player) {
    long ticks = 0L;
    lyric_t *plyric;

    printf("Load callback, tracks %d 
", player->ntracks);
    int n;
    for (n = 0; n < player->ntracks; n++) {
        fluid_track_t *track = player->track[n];
        printf("Track %d
", n);
        fluid_midi_event_t *event = fluid_track_first_event(track);
        while (event != NULL) {
            switch (fluid_midi_event_get_type (event)) {
            case MIDI_TEXT:
            case MIDI_LYRIC:
                /* there's no fluid_midi_event_get_sysex()
                   or fluid_midi_event_get_time() so we
                   have to look inside the opaque struct
                */
                ticks += event->dtime;
                printf("Loaded event %s for time %ld
",
                       (char *) event->paramptr,
                       ticks);
                plyric = g_new(lyric_t, 1);
                plyric->lyric = g_strdup(event->paramptr);
                plyric->tick = ticks;
                g_array_append_val(lyrics, plyric);
            }
            event = fluid_track_next_event(track);
        }
    }

    printf("Saved %d lyric events
", lyrics->len);
    for (n = 0; n < lyrics->len; n++) {
        plyric = g_array_index(lyrics, lyric_t *, n);
        printf("Saved lyric %s at %ld
", plyric->lyric, plyric->tick);
    }

    build_lyric_lines();

    // stick the first two lines into the labels so we can see
    // what is coming
    gdk_threads_enter();
    char *str = g_array_index(lyric_lines.lines, GString *, 1)->str;
    gtk_label_set_text((GtkLabel *) lyric_labels[0], str);
    str = g_array_index(lyric_lines.lines, GString *, 2)->str;
    gtk_label_set_text((GtkLabel *) lyric_labels[1], str);
    // gdk_flush ();

    /* release GTK thread lock */
    gdk_threads_leave();

    return FLUID_OK;
}

/* Called when the windows are realized
 */
static void realize_cb (GtkWidget *widget, gpointer data) {
    /* now we can play the midi files, if any */
    fluid_player_play(player);
}

static gboolean delete_event( GtkWidget *widget,
                              GdkEvent  *event,
                              gpointer   data )
{
    /* If you return FALSE in the "delete-event" signal handler,
     * GTK will emit the "destroy" signal. Returning TRUE means
     * you don't want the window to be destroyed.
     * This is useful for popping up 'are you sure you want to quit?'
     * type dialogs. */

    g_print ("delete event occurred
");

    /* Change TRUE to FALSE and the main window will be destroyed with
     * a "delete-event". */

    return TRUE;
}

/* Another callback */
static void destroy( GtkWidget *widget,
                     gpointer   data )
{
    gtk_main_quit ();
}

int main(int argc, char** argv)
{

    /* set up the fluidsynth stuff */
    int i;
    fluid_settings_t* settings;

    fluid_audio_driver_t* adriver;
    settings = new_fluid_settings();
    fluid_settings_setstr(settings, "audio.driver", "alsa");
    fluid_settings_setint(settings, "synth.polyphony", 64);
    fluid_settings_setint(settings, "synth.reverb.active", FALSE);
    fluid_settings_setint(settings, "synth.sample-rate", 22050);
    synth = new_fluid_synth(settings);
    player = new_fluid_player(synth);

    lyrics = g_array_sized_new(FALSE, FALSE, sizeof(lyric_t *), 1024);

    /* Set the MIDI event callback to our own functions rather than the system default */
    fluid_player_set_playback_callback(player, event_callback, synth);

    /* Add an onload callback so we can get information from new data before it plays */
    fluid_player_set_onload_callback(player, onload_callback, NULL);

    adriver = new_fluid_audio_driver(settings, synth);
    /* process command line arguments */
    for (i = 1; i < argc; i++) {
        if (fluid_is_soundfont(argv[i])) {
            fluid_synth_sfload(synth, argv[1], 1);
        } else {
            fluid_player_add(player, argv[i]);
        }
    }

    // Gtk stuff now

   /* GtkWidget is the storage type for widgets */
    GtkWidget *window;
    GtkWidget *button;
    GtkWidget *lyrics_box;

    /* This is called in all GTK applications. Arguments are parsed
     * from the command line and are returned to the application. */
    gtk_init (&argc, &argv);

    /* create a new window */
    window = gtk_window_new (GTK_WINDOW_TOPLEVEL);

    /* When the window is given the "delete-event" signal (this is given
     * by the window manager, usually by the "close" option, or on the
     * titlebar), we ask it to call the delete_event () function
     * as defined above. The data passed to the callback
     * function is NULL and is ignored in the callback function. */
    g_signal_connect (window, "delete-event",
                      G_CALLBACK (delete_event), NULL);

    /* Here we connect the "destroy" event to a signal handler.
     * This event occurs when we call gtk_widget_destroy() on the window,
     * or if we return FALSE in the "delete-event" callback. */
    g_signal_connect (window, "destroy",
                      G_CALLBACK (destroy), NULL);

    g_signal_connect (window, "realize", G_CALLBACK (realize_cb), NULL);

    /* Sets the border width of the window. */
    gtk_container_set_border_width (GTK_CONTAINER (window), 10);

    // Gtk 3.0 deprecates gtk_vbox_new in favour of gtk_grid
    // but that isn't in Gtk 2.0, so we ignore warnings for now
    lyrics_box = gtk_vbox_new(TRUE, 1);
    gtk_widget_show(lyrics_box);

    char *str = "  ";
    lyric_labels[0] = gtk_label_new(str);
    lyric_labels[1] = gtk_label_new(str);

    gtk_widget_show (lyric_labels[0]);
    gtk_widget_show (lyric_labels[1]);

    gtk_box_pack_start (GTK_BOX (lyrics_box), lyric_labels[0], TRUE, TRUE, 0);
    gtk_box_pack_start (GTK_BOX (lyrics_box), lyric_labels[1], TRUE, TRUE, 0);

    /* This packs the button into the window (a gtk container). */
    gtk_container_add (GTK_CONTAINER (window), lyrics_box);

    /* and the window */
    gtk_widget_show (window);

    /* All GTK applications must have a gtk_main(). Control ends here
     * and waits for an event to occur (like a key press or
     * mouse event). */
    gtk_main ();

    /* wait for playback termination */
    fluid_player_join(player);
    /* cleanup */
    delete_fluid_audio_driver(adriver);
    delete_fluid_player(player);
    delete_fluid_synth(synth);
    delete_fluid_settings(settings);
    return 0;
}

When run, it looks like Figure 27-1.

Figure 27-1. Caption

Playing a Background Video with Gtk

Chapter 15 showed how to play a background video with images (using pixbufs), text (using Cairo), and colored text (using Pango). You can extend that by adding in the dynamic text display for playing karaoke.

You can capture each lyric line in a structure, which keeps the whole line, the part that has been sung already, the Pango markup for the line, and the Pango attributes.

typedef struct _coloured_line_t {
    gchar *line;
    gchar *front_of_line;
    gchar *marked_up_line;
    PangoAttrList *attrs;
} coloured_line_t;

This is updated each time a MIDI lyric event occurs, in a thread listening to the FluidSynth sequencer.

A separate thread plays the video and on each frame overlays the frame image with the current and next lyric. This is set into a GdkImage for display by Gtk.

The program is gtkkaraoke_player_video_pango.c.

#include <fluidsynth.h>
#include <fluid_midi.h>
#include <string.h>

#include <gtk/gtk.h>

#include <libavcodec/avcodec.h>
#include <libavformat/avformat.h>
#include <libswscale/swscale.h>

// saving as pixbufs leaks memory
//#define USE_PIXBUF

/* run by
   gtkkaraoke_player_video_pango /usr/share/sounds/sf2/FluidR3_GM.sf2 /home/newmarch/Music/karaoke/sonken/songs/54154.kar
*/

/*
 * APIs:
 * GString: https://developer.gnome.org/glib/2.28/glib-Strings.html
 * Pango text attributes: https://developer.gnome.org/pango/stable/pango-Text-Attributes.html#pango-parse-markup
 * Pango layout: http://www.gtk.org/api/2.6/pango/pango-Layout-Objects.html
 * Cairo rendering: https://developer.gnome.org/pango/stable/pango-Cairo-Rendering.html#pango-cairo-create-layout
 * Cairo surface_t: http://cairographics.org/manual/cairo-cairo-surface-t.html
 * GTK+ 3 Reference Manual: https://developer.gnome.org/gtk3/3.0/
 * Gdk Pixbufs: https://developer.gnome.org/gdk/stable/gdk-Pixbufs.html
 */

struct _lyric_t {
    gchar *lyric;
    long tick;

};
typedef struct _lyric_t lyric_t;

struct _lyric_lines_t {
    char *language;
    char *title;
    char *performer;
    GArray *lines; // array of GString *
};
typedef struct _lyric_lines_t lyric_lines_t;

GArray *lyrics;

lyric_lines_t lyric_lines;

typedef struct _coloured_line_t {
    gchar *line;
    gchar *front_of_line;
    gchar *marked_up_line;
    PangoAttrList *attrs;
#ifdef USE_PIXBUF
    GdkPixbuf *pixbuf;
#endif
} coloured_line_t;

coloured_line_t coloured_lines[2];

fluid_synth_t* synth;

GtkWidget *image;
#if GTK_MAJOR_VERSION == 2
GdkPixmap *dbuf_pixmap;
#endif

int height_lyric_pixbufs[] = {300, 400}; // vertical offset of lyric in video

fluid_player_t* player;

int current_panel = 1;  // panel showing current lyric line
int current_line = 0;  // which line is the current lyric
gchar *current_lyric;   // currently playing lyric line
GString *front_of_lyric;  // part of lyric to be coloured red
//GString *end_of_lyric;    // part of lyrci to not be coloured

// Colours seem to get mixed up when putting a pixbuf onto a pixbuf
#ifdef USE_PIXBUF
#define RED blue
#else
#define RED red
#endif

gchar *markup[] = {"<span font="28" foreground="RED">",
                   "</span><span font="28" foreground="white">",
                   "</span>"};
gchar *markup_newline[] = {"<span foreground="black">",
                           "</span>"};
GString *marked_up_label;

/* FFMpeg vbls */
AVFormatContext *pFormatCtx = NULL;
AVCodecContext *pCodecCtx = NULL;
int videoStream;
struct SwsContext *sws_ctx = NULL;
AVCodec *pCodec = NULL;

void markup_line(coloured_line_t *line) {
    GString *str =  g_string_new(markup[0]);
    g_string_append(str, line->front_of_line);
    g_string_append(str, markup[1]);
    g_string_append(str, line->line + strlen(line->front_of_line));
    g_string_append(str, markup[2]);
    printf("Marked up label "%s"
", str->str);

    line->marked_up_line = str->str;
    // we have to free line->marked_up_line

    pango_parse_markup(str->str, -1,0, &(line->attrs), NULL, NULL, NULL);
    g_string_free(str, FALSE);
}

#ifdef USE_PIXBUF
void update_line_pixbuf(coloured_line_t *line) {
    //return;
    cairo_surface_t *surface;
    cairo_t *cr;

    int lyric_width = 480;
    int lyric_height = 60;
    surface = cairo_image_surface_create (CAIRO_FORMAT_ARGB32,
                                          lyric_width, lyric_height);
    cr = cairo_create (surface);

    PangoLayout *layout;
    PangoFontDescription *desc;

    // draw the attributed text
    layout = pango_cairo_create_layout (cr);
    pango_layout_set_text (layout, line->line, -1);
    pango_layout_set_attributes(layout, line->attrs);

    // centre the image in the surface
    int width, height;
    pango_layout_get_pixel_size(layout,
                                &width,
                                &height);
    cairo_move_to(cr, (lyric_width-width)/2, 0);

    pango_cairo_update_layout (cr, layout);
    pango_cairo_show_layout (cr, layout);

    // pull the pixbuf out of the surface
    unsigned char *data = cairo_image_surface_get_data(surface);
    width = cairo_image_surface_get_width(surface);
    height = cairo_image_surface_get_height(surface);
    int stride = cairo_image_surface_get_stride(surface);
    printf("Text surface width %d height %d stride %d
", width, height, stride);

    GdkPixbuf *old_pixbuf = line->pixbuf;
    line->pixbuf = gdk_pixbuf_new_from_data(data, GDK_COLORSPACE_RGB, 1, 8, width, height, stride, NULL, NULL);
    cairo_surface_destroy(surface);
    g_object_unref(old_pixbuf);
}
#endif

/**
 * This MIDI event callback filters out the TEXT and LYRIC events
 * and passes the rest to the default event handler.
  */
int event_callback(void *data, fluid_midi_event_t *event) {
    fluid_synth_t* synth = (fluid_synth_t*) data;
    int type = fluid_midi_event_get_type(event);
    int chan = fluid_midi_event_get_channel(event);
    if (synth == NULL) printf("Synth is null
");

    //return 0;

    switch(type) {
    case MIDI_TEXT:
        printf("Callback: Playing text event %s (length %d)
",
               (char *) event->paramptr, (int) event->param1);

        if (((char *) event->paramptr)[0] == '') {
            int next_panel = current_panel; // really (current_panel+2)%2
            int next_line = current_line + 2;
            gchar *next_lyric;

            if (current_line + 2 >= lyric_lines.lines->len) {
                return FLUID_OK;
            }
            current_line += 1;
            current_panel = (current_panel + 1) % 2;

            // set up new line as current line
            char *lyric =  event->paramptr;
            current_lyric = g_array_index(lyric_lines.lines, GString *, current_line)->str;
            front_of_lyric = g_string_new(lyric+1); // lose 
            printf("New line. Setting front to %s end to "%s"
", lyric+1, current_lyric);

            coloured_lines[current_panel].line = current_lyric;
            coloured_lines[current_panel].front_of_line = lyric+1;
            markup_line(coloured_lines+current_panel);
#ifdef USE_PIXBUF
            update_line_pixbuf(coloured_lines+current_panel);
#endif
            // update label for next line after this one
            next_lyric = g_array_index(lyric_lines.lines, GString *, next_line)->str;

            marked_up_label = g_string_new(markup_newline[0]);

            g_string_append(marked_up_label, next_lyric);
            g_string_append(marked_up_label, markup_newline[1]);
            PangoAttrList *attrs;
            gchar *text;
            pango_parse_markup (marked_up_label->str, -1,0, &attrs, &text, NULL, NULL);

            coloured_lines[next_panel].line = next_lyric;
            coloured_lines[next_panel].front_of_line = "";
            markup_line(coloured_lines+next_panel);
#ifdef USE_PIXBUF
            update_line_pixbuf(coloured_lines+next_panel);
#endif
        } else {
            // change text colour as chars are played
            char *lyric =  event->paramptr;
            if ((front_of_lyric != NULL) && (lyric != NULL)) {
                g_string_append(front_of_lyric, lyric);
                char *s = front_of_lyric->str;
                coloured_lines[current_panel].front_of_line = s;
                markup_line(coloured_lines+current_panel);
#ifdef USE_PIXBUF
                update_line_pixbuf(coloured_lines+current_panel);
#endif
            }
        }
        return  FLUID_OK;

    case MIDI_LYRIC:
        printf("Callback: Playing lyric event %d %s
", (int) event->param1, (char *) event->paramptr);
        return  FLUID_OK;

    case MIDI_EOT:
        printf("End of track
");
        exit(0);
    }
    return fluid_synth_handle_midi_event( data, event);
}

void build_lyric_lines() {
    int n;
    lyric_t *plyric;
    GString *line = g_string_new("");
    GArray *lines =  g_array_sized_new(FALSE, FALSE, sizeof(GString *), 64);

    lyric_lines.title = NULL;

    for (n = 0; n < lyrics->len; n++) {
        plyric = g_array_index(lyrics, lyric_t *, n);
        gchar *lyric = plyric->lyric;
        int tick = plyric->tick;

        if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'L')) {
            lyric_lines.language =  lyric + 2;
            continue;
        }

        if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'T')) {
            if (lyric_lines.title == NULL) {
                lyric_lines.title = lyric + 2;
            } else {
                lyric_lines.performer = lyric + 2;
            }
            continue;
        }

        if (lyric[0] == '@') {
            // some other stuff like @KMIDI KARAOKE FILE
            continue;
        }

        if ((lyric[0] == '/') || (lyric[0] == '')) {
            // start of a new line
            // add to lines
            g_array_append_val(lines, line);
            line = g_string_new(lyric + 1);
        }  else {
            line = g_string_append(line, lyric);
        }
    }
    lyric_lines.lines = lines;

    printf("Title is %s, performer is %s, language is %s
",
           lyric_lines.title, lyric_lines.performer, lyric_lines.language);
    for (n = 0; n < lines->len; n++) {
        printf("Line is %s
", g_array_index(lines, GString *, n)->str);
    }

}

/**
 * This is called whenever new data is loaded, such as a new file.
 * Here we extract the TEXT and LYRIC events and just print them
 * to stdout. They could e.g. be saved and displayed in a GUI
 * as the events are received by the event callback.
 */
int onload_callback(void *data, fluid_player_t *player) {
    long ticks = 0L;
    lyric_t *plyric;

    printf("Load callback, tracks %d 
", player->ntracks);
    int n;
    for (n = 0; n < player->ntracks; n++) {
        fluid_track_t *track = player->track[n];
        printf("Track %d
", n);
        fluid_midi_event_t *event = fluid_track_first_event(track);
        while (event != NULL) {
            switch (fluid_midi_event_get_type (event)) {
            case MIDI_TEXT:
            case MIDI_LYRIC:
                /* there's no fluid_midi_event_get_sysex()
                   or fluid_midi_event_get_time() so we
                   have to look inside the opaque struct
                */
                ticks += event->dtime;
                printf("Loaded event %s for time %ld
",
                       (char *) event->paramptr,
                       ticks);
                plyric = g_new(lyric_t, 1);
                plyric->lyric = g_strdup(event->paramptr);
                plyric->tick = ticks;
                g_array_append_val(lyrics, plyric);
            }
            event = fluid_track_next_event(track);
        }
    }

    printf("Saved %d lyric events
", lyrics->len);
    for (n = 0; n < lyrics->len; n++) {
        plyric = g_array_index(lyrics, lyric_t *, n);
        printf("Saved lyric %s at %ld
", plyric->lyric, plyric->tick);
    }

    build_lyric_lines();

    return FLUID_OK;
}

static void overlay_lyric(cairo_t *cr,
                          coloured_line_t *line,
                          int ht) {
    PangoLayout *layout;
    int height, width;

    if (line->line == NULL) {
        return;
    }

    layout = pango_cairo_create_layout (cr);
    pango_layout_set_text (layout, line->line, -1);
    pango_layout_set_attributes(layout, line->attrs);
    pango_layout_get_pixel_size(layout,
                                &width,
                                &height);
    cairo_move_to(cr, (720-width)/2, ht);

    pango_cairo_update_layout (cr, layout);
    pango_cairo_show_layout (cr, layout);

    g_object_unref(layout);
}

static void pixmap_destroy_notify(guchar *pixels,
                                  gpointer data) {
    printf("Ddestroy pixmap
");
}

static void *play_background(void *args) {
    /* based on code from
       http://www.cs.dartmouth.edu/∼xy/cs23/gtk.html
       http://cdry.wordpress.com/2009/09/09/using-custom-io-callbacks-with-ffmpeg/
    */

    int i;
    AVPacket packet;
    int frameFinished;
    AVFrame *pFrame = NULL;

    int oldSize;
    char *oldData;
    int bytesDecoded;
    GdkPixbuf *pixbuf;
    AVFrame *picture_RGB;
    char *buffer;

#if GTK_MAJOR_VERSION == 2
    GdkPixmap *pixmap;
    GdkBitmap *mask;
#endif

    pFrame=avcodec_alloc_frame();

    i=0;
    picture_RGB = avcodec_alloc_frame();
    buffer = malloc (avpicture_get_size(PIX_FMT_RGB24, 720, 576));
    avpicture_fill((AVPicture *)picture_RGB, buffer, PIX_FMT_RGB24, 720, 576);

    while(av_read_frame(pFormatCtx, &packet)>=0) {
        if(packet.stream_index==videoStream) {
            //printf("Frame %d
", i++);
            usleep(33670);  // 29.7 frames per second
            // Decode video frame
            avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished,
                                  &packet);
            int width = pCodecCtx->width;
            int height = pCodecCtx->height;

            sws_ctx = sws_getContext(pCodecCtx->width, pCodecCtx->height, pCodecCtx->pix_fmt, pCodecCtx->width, pCodecCtx->height, PIX_FMT_RGB24, SWS_BICUBIC, NULL, NULL, NULL);

            if (frameFinished) {
                printf("Frame %d
", i++);

                sws_scale(sws_ctx,  (uint8_t const * const *) pFrame->data, pFrame->linesize, 0, pCodecCtx->height, picture_RGB->data, picture_RGB->linesize);

                pixbuf = gdk_pixbuf_new_from_data(picture_RGB->data[0], GDK_COLORSPACE_RGB, 0, 8, 720, 480, picture_RGB->linesize[0], pixmap_destroy_notify, NULL);

                /* start GTK thread lock for drawing */
                gdk_threads_enter();

#define SHOW_LYRIC
#ifdef SHOW_LYRIC
                // Create the destination surface
                cairo_surface_t *surface = cairo_image_surface_create (CAIRO_FORMAT_ARGB32,
                                                                       width, height);
                cairo_t *cr = cairo_create(surface);

                // draw the background image
                gdk_cairo_set_source_pixbuf(cr, pixbuf, 0, 0);
                cairo_paint (cr);

#ifdef USE_PIXBUF
                // draw the lyric
                GdkPixbuf *lyric_pixbuf = coloured_lines[current_panel].pixbuf;
                if (lyric_pixbuf != NULL) {
                    int width = gdk_pixbuf_get_width(lyric_pixbuf);
                    gdk_cairo_set_source_pixbuf(cr,
                                                lyric_pixbuf,
                                                (720-width)/2,
                                                 height_lyric_pixbufs[current_panel]);
                    cairo_paint (cr);
                }

                int next_panel = (current_panel+1) % 2;
                lyric_pixbuf = coloured_lines[next_panel].pixbuf;
                if (lyric_pixbuf != NULL) {
                    int width = gdk_pixbuf_get_width(lyric_pixbuf);
                    gdk_cairo_set_source_pixbuf(cr,
                                                lyric_pixbuf,
                                                (720-width)/2,
                                                 height_lyric_pixbufs[next_panel]);
                    cairo_paint (cr);
                }
#else

                overlay_lyric(cr,
                              coloured_lines+current_panel,
                              height_lyric_pixbufs[current_panel]);

                int next_panel = (current_panel+1) % 2;
                overlay_lyric(cr,
                              coloured_lines+next_panel,
                              height_lyric_pixbufs[next_panel]);
#endif
                pixbuf = gdk_pixbuf_get_from_surface(surface,
                                                     0,
                                                     0,
                                                     width,
                                                     height);

                gtk_image_set_from_pixbuf((GtkImage*) image, pixbuf);

                g_object_unref(pixbuf);         /* reclaim memory */
                //g_object_unref(layout);
                cairo_surface_destroy(surface);
                cairo_destroy(cr);
#else
        gtk_image_set_from_pixbuf((GtkImage*) image, pixbuf);
#endif /* SHOW_LYRIC */

                /* release GTK thread lock */
                gdk_threads_leave();
            }
        }
        av_free_packet(&packet);
    }
    sws_freeContext(sws_ctx);

    printf("Video over!
");
    exit(0);
}

static void *play_midi(void *args) {
    fluid_player_play(player);

    printf("Audio finished
");
    //exit(0);
}

/* Called when the windows are realized
 */
static void realize_cb (GtkWidget *widget, gpointer data) {
    /* start the video playing in its own thread */
    pthread_t tid;
    pthread_create(&tid, NULL, play_background, NULL);

    /* start the MIDI file playing in its own thread */
    pthread_t tid_midi;
    pthread_create(&tid_midi, NULL, play_midi, NULL);
}

static gboolean delete_event( GtkWidget *widget,
                              GdkEvent  *event,
                              gpointer   data )
{
    /* If you return FALSE in the "delete-event" signal handler,
     * GTK will emit the "destroy" signal. Returning TRUE means
     * you don't want the window to be destroyed.
     * This is useful for popping up 'are you sure you want to quit?'
     * type dialogs. */

    g_print ("delete event occurred
");

    /* Change TRUE to FALSE and the main window will be destroyed with
     * a "delete-event". */

    return TRUE;
}

/* Another callback */
static void destroy( GtkWidget *widget,
                     gpointer   data )
{
    gtk_main_quit ();
}

int main(int argc, char** argv)
{
    XInitThreads();

    int i;

    fluid_settings_t* settings;

    fluid_audio_driver_t* adriver;
    settings = new_fluid_settings();
    fluid_settings_setstr(settings, "audio.driver", "alsa");
    //fluid_settings_setint(settings, "lash.enable", 0);
    fluid_settings_setint(settings, "synth.polyphony", 64);
    fluid_settings_setint(settings, "synth.reverb.active", FALSE);
    fluid_settings_setint(settings, "synth.sample-rate", 22050);
    synth = new_fluid_synth(settings);
    player = new_fluid_player(synth);

    lyrics = g_array_sized_new(FALSE, FALSE, sizeof(lyric_t *), 1024);

    /* Set the MIDI event callback to our own functions rather than the system default */
    fluid_player_set_playback_callback(player, event_callback, synth);

    /* Add an onload callback so we can get information from new data before it plays */
    fluid_player_set_onload_callback(player, onload_callback, NULL);

    adriver = new_fluid_audio_driver(settings, synth);

    /* process command line arguments */
    for (i = 1; i < argc; i++) {
        if (fluid_is_soundfont(argv[i])) {
            fluid_synth_sfload(synth, argv[1], 1);
        } else {
            fluid_player_add(player, argv[i]);
        }
    }

    /* FFMpeg stuff */

    AVFrame *pFrame = NULL;
    AVPacket packet;

    AVDictionary *optionsDict = NULL;

    av_register_all();

    if(avformat_open_input(&pFormatCtx, "short.mpg", NULL, NULL)!=0) {
        printf("Couldn't open video file
");
        return -1; // Couldn't open file
    }

    // Retrieve stream information
    if(avformat_find_stream_info(pFormatCtx, NULL)<0) {
        printf("Couldn't find stream information
");
        return -1; // Couldn't find stream information
    }

    // Dump information about file onto standard error
    av_dump_format(pFormatCtx, 0, argv[1], 0);

    // Find the first video stream
    videoStream=-1;
    for(i=0; i<pFormatCtx->nb_streams; i++)
        if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_VIDEO) {
            videoStream=i;
            break;
        }
    if(videoStream==-1)
        return -1; // Didn't find a video stream

    for(i=0; i<pFormatCtx->nb_streams; i++)
        if(pFormatCtx->streams[i]->codec->codec_type==AVMEDIA_TYPE_AUDIO) {
            printf("Found an audio stream too
");
            break;
        }

    // Get a pointer to the codec context for the video stream
    pCodecCtx=pFormatCtx->streams[videoStream]->codec;

    // Find the decoder for the video stream
    pCodec=avcodec_find_decoder(pCodecCtx->codec_id);
    if(pCodec==NULL) {
        fprintf(stderr, "Unsupported codec!
");
        return -1; // Codec not found
    }

    // Open codec
    if(avcodec_open2(pCodecCtx, pCodec, &optionsDict)<0) {
        printf("Could not open codec
");
        return -1; // Could not open codec
    }

    sws_ctx =
        sws_getContext
        (
         pCodecCtx->width,
         pCodecCtx->height,
         pCodecCtx->pix_fmt,
         pCodecCtx->width,
         pCodecCtx->height,
         PIX_FMT_YUV420P,
         SWS_BILINEAR,
         NULL,
         NULL,
         NULL
         );

    /* GTK stuff now */

    /* GtkWidget is the storage type for widgets */
    GtkWidget *window;
    GtkWidget *button;
    GtkWidget *lyrics_box;

    /* This is called in all GTK applications. Arguments are parsed
     * from the command line and are returned to the application. */
    gtk_init (&argc, &argv);

    /* create a new window */
    window = gtk_window_new (GTK_WINDOW_TOPLEVEL);

    /* When the window is given the "delete-event" signal (this is given
     * by the window manager, usually by the "close" option, or on the
     * titlebar), we ask it to call the delete_event () function
     * as defined above. The data passed to the callback
     * function is NULL and is ignored in the callback function. */
    g_signal_connect (window, "delete-event",
                      G_CALLBACK (delete_event), NULL);

    /* Here we connect the "destroy" event to a signal handler.
     * This event occurs when we call gtk_widget_destroy() on the window,
     * or if we return FALSE in the "delete-event" callback. */
    g_signal_connect (window, "destroy",
                      G_CALLBACK (destroy), NULL);

    g_signal_connect (window, "realize", G_CALLBACK (realize_cb), NULL);

    /* Sets the border width of the window. */
    gtk_container_set_border_width (GTK_CONTAINER (window), 10);

    lyrics_box = gtk_vbox_new(TRUE, 1);
    gtk_widget_show(lyrics_box);

    /*
    char *str = "     ";
    lyric_labels[0] = gtk_label_new(str);
    str =  "World";
    lyric_labels[1] = gtk_label_new(str);
    */

    image = gtk_image_new();

    //image_drawable = gtk_drawing_area_new();
    //gtk_widget_set_size_request (canvas, 720, 480);
    //gtk_drawing_area_size((GtkDrawingArea *) image_drawable, 720, 480);

    //gtk_widget_show (lyric_labels[0]);
    //gtk_widget_show (lyric_labels[1]);

    gtk_widget_show (image);

    //gtk_box_pack_start (GTK_BOX (lyrics_box), lyric_labels[0], TRUE, TRUE, 0);
    //gtk_box_pack_start (GTK_BOX (lyrics_box), lyric_labels[1], TRUE, TRUE, 0);
    gtk_box_pack_start (GTK_BOX (lyrics_box), image, TRUE, TRUE, 0);
    //gtk_box_pack_start (GTK_BOX (lyrics_box), canvas, TRUE, TRUE, 0);
    //gtk_box_pack_start (GTK_BOX (lyrics_box), image_drawable, TRUE, TRUE, 0);

    /* This packs the button into the window (a gtk container). */
    gtk_container_add (GTK_CONTAINER (window), lyrics_box);

    /* and the window */
    gtk_widget_show (window);

    /* All GTK applications must have a gtk_main(). Control ends here
     * and waits for an event to occur (like a key press or
     * mouse event). */
    gtk_main ();

    return 0;

    /* wait for playback termination */
    fluid_player_join(player);
    /* cleanup */
    delete_fluid_audio_driver(adriver);
    delete_fluid_player(player);
    delete_fluid_synth(synth);
    delete_fluid_settings(settings);

    return 0;
}

The application looks like Figure 27-2.

Figure 27-2. Caption

Conclusion

By extending FluidSynth, it can be made into a karaoke player in various ways. It is quite heavy in CPU usage, though. On my laptop, the final version runs at about 100 percent CPU.