TiMidity and Jack

In Chapter 17, I discussed playing MIDI files using Jack. Jack is designed to link audio sources and sinks in arbitrary configurations. By running qjackctl, you can link, for example, microphone outputs to speaker inputs. This is done by dragging capture_1 to playback_1 , and so on, and it looks like Figure 28-2.

Figure 28-2. qjackctl showing microphone to speakers

If TiMidity is then run with Jack output, you get instant karaoke. You can also see the lyrics played in real time using the --trace option.

timidity -Oj --trace 54154.mid

The connections are shown in qjackctl in Figure 28-3.

Figure 28-3. qjackctl showing TiMidity

The lyric display is klunky and will be improved later.

TiMidity Interface

You will need to have the TiMidity source downloaded from SourceForge TiMidity++ ( http://sourceforge.net/projects/timidity/?source=dlp ).

In Chapter 21, I discussed two alternative ways of building applications using TiMidity.

• You can build a front end with TiMidity as a library back end.

• You can use standard TiMidity with a custom-built interface as the back end to TiMidity.

Both options are possible here, with one wrinkle: if you want to capture MIDI events, then you have to do so as a back end to TiMidity, which requires that you build a TiMidity interface.

To recap, the different interface files for TiMidity are stored in the directory interface and include files such as dumb_c.c for the dumb interface. They all revolve around a data structure ControlMode defined in timidity/controls.h.

typedef struct {
  char *id_name, id_character;
  char *id_short_name;
  int verbosity, trace_playing, opened;

  int32 flags;

  int  (*open)(int using_stdin, int using_stdout);
  void (*close)(void);
  int (*pass_playing_list)(int number_of_files, char *list_of_files[]);
  int  (*read)(int32 *valp);
  int  (*write)(char *buf, int32 size);
  int  (*cmsg)(int type, int verbosity_level, char *fmt, ...);
  void (*event)(CtlEvent *ev);  /* Control events */
} ControlMode;

For the simplest values of the functions in this structure, see the code for the dumb interface in interface/dumb_c.c.

For dealing with lyrics, the main field to set is the function event(). This will be passed a pointer to a CtlEvent, which is defined in timidity/controls.h.

typedef struct _CtlEvent {
    int type;           /* See above */
    ptr_size_t v1, v2, v3, v4;/* Event value */
} CtlEvent;

The type field distinguishes a large number of event types such as CTLE_NOW_LOADING and CTLE_PITCH_BEND. The type of interest to you is CTLE_LYRIC.

Typical code to handle this is in interface/dumb_c.c, which prints event information to output.

static void ctl_event(CtlEvent *e)
{
    switch(e->type) {
      case CTLE_LYRIC:
        ctl_lyric((int)e->v1);
        break;
   }
}

static void ctl_lyric(int lyricid)
{
    char *lyric;

    lyric = event2string(lyricid);
    if(lyric != NULL)
    {
        if(lyric[0] == ME_KARAOKE_LYRIC)
        {
            if(lyric[1] == '/' || lyric[1] == '')
            {
                fprintf(outfp, "
%s", lyric + 2);
                fflush(outfp);
            }
            else if(lyric[1] == '@')
            {
                if(lyric[2] == 'L')
                    fprintf(outfp, "
Language: %s
", lyric + 3);
                else if(lyric[2] == 'T')
                   fprintf(outfp, "Title: %s
", lyric + 3);
                else
                    fprintf(outfp, "%s
", lyric + 1);
            }
            else
            {
                fputs(lyric + 1, outfp);
                fflush(outfp);
            }
        }
        else
        {
            if(lyric[0] == ME_CHORUS_TEXT || lyric[0] == ME_INSERT_TEXT)
                fprintf(outfp, "
");
            fputs(lyric + 1, outfp);
            fflush(outfp);
        }
    }
}

Getting the List of Lyrics

The failing of the current interfaces in TiMidity with regard to karaoke is that while they can show the lyrics as they are played, they don’t show the lyric lines and progressively highlight them as they are played. For that, you need the set of lyrics.

TiMidity in fact builds a list of lyrics and makes them accessible. It has a function event2string() that takes an integer parameter from 1 upward. For each value, it returns the string of a lyric or text event, finally returning NULL on the end of the list. The first character returned is a type parameter; the rest is the string. Using GLib functions, you can build up an array of lines for a KAR file with the following:

struct _lyric_t {
    gchar *lyric;
    long tick; // not used here
};
typedef struct _lyric_t lyric_t;

struct _lyric_lines_t {
    char *language;
    char *title;
    char *performer;
    GArray *lines; // array of GString *
};
typedef struct _lyric_lines_t lyric_lines_t;

GArray *lyrics;
lyric_lines_t lyric_lines;

static void build_lyric_lines() {
    int n;
    lyric_t *plyric;
    GString *line = g_string_new("");
    GArray *lines =  g_array_sized_new(FALSE, FALSE, sizeof(GString *), 64);

    lyric_lines.title = NULL;

    n = 1;
    char *evt_str;
    while ((evt_str = event2string(n++)) != NULL) {
        gchar *lyric = evt_str+1;

        if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'L')) {
            lyric_lines.language =  lyric + 2;
            continue;
        }

        if ((strlen(lyric) >= 2) && (lyric[0] == '@') && (lyric[1] == 'T')) {
            if (lyric_lines.title == NULL) {
                lyric_lines.title = lyric + 2;
            } else {
                lyric_lines.performer = lyric + 2;
            }
            continue;
        }

        if (lyric[0] == '@') {
            // some other stuff like @KMIDI KARAOKE FILE
            continue;
        }

        if ((lyric[0] == '/') || (lyric[0] == '')) {
            // start of a new line
            // add to lines
            g_array_append_val(lines, line);
            line = g_string_new(lyric + 1);
        }  else {
            line = g_string_append(line, lyric);
        }
    }
    lyric_lines.lines = lines;

    printf("Title is %s, performer is %s, language is %s
",
           lyric_lines.title, lyric_lines.performer, lyric_lines.language);
    for (n = 0; n < lines->len; n++) {
        printf("Line is %s
", g_array_index(lines, GString *, n)->str);
    }
}

The function build_lyric_lines() should be called from the CTLE_LOADING_DONE branch of ctl_event().

TiMidity Options

If you choose to use TiMidity as the front end, then you need to run it with suitable options. These include turning tracing on and also dynamically loading your new interface. This can be done, for example, with the following for a “v” interface in the current directory:

timidity -d. -iv --trace  --trace-text-meta ...
.

The alternative is building a main program that calls TiMidity as a library. The command-line parameters to TiMidity then have to be included as hard-coded parameters in the application. One is easy: the CtlMode has a field trace_playing and setting it to 1 turns tracing on. Including text events as lyric events requires digging a bit deeper into TiMidity but just requires (shortly after initializing the library) the following:

extern int opt_trace_text_meta_event;
opt_trace_text_meta_event = 1;

Playing Lyrics Using Pango + Cairo + Xlib

I want to be able to play my karaoke files on the Raspberry Pi and similar systems on a chip (SOCs). Unfortunately, the Raspberry Pi has a grossly underpowered CPU, so I have ended up using a CubieBoard 2.

Anything involving heavy graphics is not possible on this CPU. All of the MIDI players hit close to (or over) 100 percent CPU usage just by themselves. So, the system discussed in the next section, showing background video, isn’t feasible on the Raspberry Pi without the use of the GPU, which is discussed in my book Raspberry Pi GPU Audio Video Programming. The programs discussed in the sequel play fine on any current laptops and desktops.

In this section, you use TiMidity as the MIDI player with a minimal back end to display the lyrics as they are played. The lowest level of GUI support is used, namely, Xlib. This can be used to draw text using low-level Xlib calls such as XDrawImageString. This works fine with ASCII languages and, with appropriate font choices, with other languages in the ISO-8859 family.

Asian languages are harder to deal with in standard C. They involve 1- or 2-byte characters when using an encoding such as UTF-8. To manage these, it is easiest to switch to a library designed to handle them such, such as Cairo.

Cairo is good for drawing simple text. For example, for Chinese characters you have to find a font that will allow you to draw them. Alternatively, you can jump up one further level to Pango. Pango looks after all the font issues and produces glyphs that are sent to the X server.

That approach is adopted in the following interface, x_code.c.

The essential difference between the previous naive interface section and the Xlib interface of this section lies, of course, in the drawing. The function build_lyric_lines gives you access to the set of lines to render. Additional data types are required for Pango and Cairo as follows:

GArray *lyrics;
GString *lyrics_array[NUM_LINES];

lyric_lines_t lyric_lines;

typedef struct _coloured_line_t {
    gchar *line;
    gchar *front_of_line;
    gchar *marked_up_line;
    PangoAttrList *attrs;
} coloured_line_t;

int height_lyric_pixbufs[] = {100, 200, 300, 400}; // vertical offset of lyric in video
int coloured_text_offset;

// int current_panel = 1;  // panel showing current lyric line
int current_line = 0;  // which line is the current lyric
gchar *current_lyric;   // currently playing lyric line
GString *front_of_lyric;  // part of lyric to be coloured red
//GString *end_of_lyric;    // part of lyrci to not be coloured

gchar *markup[] = {"<span font="28" foreground="RED">",
                   "</span><span font="28" foreground="white">",
                   "</span>"};
gchar *markup_newline[] = {"<span foreground="black">",
                           "</span>"};
GString *marked_up_label;

PangoFontDescription *font_description;

cairo_surface_t *surface;
cairo_t *cr;

The markup string will draw played text in red and unplayed text in white, while markup_newline will clear the previous line. The principal drawing functions are as follows:

static void paint_background() {
    cr = cairo_create(surface);
    cairo_set_source_rgb(cr, 0.0, 0.8, 0.0);
    cairo_paint(cr);
    cairo_destroy(cr);
}

static void set_font() {
    font_description = pango_font_description_new ();
    pango_font_description_set_family (font_description, "serif");
    pango_font_description_set_weight (font_description, PANGO_WEIGHT_BOLD);
    pango_font_description_set_absolute_size (font_description, 32 * PANGO_SCALE);
}

static int draw_text(char *text, float red, float green, float blue, int height, int offset) {
  // See http://cairographics.org/FAQ/
  PangoLayout *layout;
  int width, ht;
  cairo_text_extents_t extents;

  layout = pango_cairo_create_layout (cr);
  pango_layout_set_font_description (layout, font_description);
  pango_layout_set_text (layout, text, -1);

  if (offset == 0) {
      pango_layout_get_size(layout, &width, &ht);
      offset = (WIDTH - (width/PANGO_SCALE)) / 2;
  }

  cairo_set_source_rgb (cr, red, green, blue);
  cairo_move_to (cr, offset, height);
  pango_cairo_show_layout (cr, layout);

  g_object_unref (layout);
  return offset;
}

The function to initialize X and Cairo is as follows:

static void init_X() {
    int screenNumber;
    unsigned long foreground, background;
    int screen_width, screen_height;
    Screen *screen;
    XSizeHints hints;
    char **argv = NULL;
    XGCValues gcValues;
    Colormap colormap;
    XColor rgb_color, hw_color;
    Font font;
    //char *FNAME = "hanzigb24st";
    char *FNAME = "-misc-fixed-medium-r-normal--0-0-100-100-c-0-iso10646-1";

    display = XOpenDisplay(NULL);
    if (display == NULL) {
        fprintf(stderr, "Can't open dsplay
");
        exit(1);
    }
    screenNumber = DefaultScreen(display);
    foreground = BlackPixel(display, screenNumber);
    background = WhitePixel(display, screenNumber);

    screen = DefaultScreenOfDisplay(display);
    screen_width = WidthOfScreen(screen);
    screen_height = HeightOfScreen(screen);

    hints.x = (screen_width - WIDTH) / 2;
    hints.y = (screen_height - HEIGHT) / 2;
    hints.width = WIDTH;
    hints.height = HEIGHT;
    hints.flags = PPosition | PSize;

    window = XCreateSimpleWindow(display,
                                 DefaultRootWindow(display),
                                 hints.x, hints.y, WIDTH, HEIGHT, 10,
                                 foreground, background);

    XSetStandardProperties(display, window,
                           "TiMidity", "TiMidity",
                           None,
                           argv, 0,
                           &hints);

    XMapWindow(display, window);

    set_font();
    surface = cairo_xlib_surface_create(display, window,
                                        DefaultVisual(display, 0), WIDTH, HEIGHT);
    cairo_xlib_surface_set_size(surface, WIDTH, HEIGHT);

    paint_background();

    /*
    cr = cairo_create(surface);
    draw_text(g_array_index(lyric_lines.lines, GString *, 0)->str,
              0.0, 0.0, 1.0, height_lyric_pixbufs[0]);
    draw_text(g_array_index(lyric_lines.lines, GString*, 1)->str,
              0.0, 0.0, 1.0, height_lyric_pixbufs[0]);
    cairo_destroy(cr);
    */
    XFlush(display);
}

The key function is ctl_lyric, which is responsible for handling lyrics as they are played. If the lyric signals and end of line with or /, then it has to update the current_line. The next lines redraw the text of each line and then progressively step through the current line, coloring the first part red and the rest in white.

static void ctl_lyric(int lyricid)
{
    char *lyric;

    current_file_info = get_midi_file_info(current_file, 1);

    lyric = event2string(lyricid);
    if(lyric != NULL)
        lyric++;
    printf("Got a lyric %s
", lyric);

    if ((*lyric == '') || (*lyric == '/')) {

        int next_line = current_line + NUM_LINES;
        gchar *next_lyric;

        if (current_line + NUM_LINES < lyric_lines.lines->len) {
            current_line += 1;

            // update label for next line after this one
            next_lyric = g_array_index(lyric_lines.lines, GString *, next_line)->str;

        } else {
            current_line += 1;
            lyrics_array[(next_line-1) % NUM_LINES] = NULL;
            next_lyric = "";
        }

        // set up new line as current line
        if (current_line < lyric_lines.lines->len) {
            GString *gstr = g_array_index(lyric_lines.lines, GString *, current_line);
            current_lyric = gstr->str;
            front_of_lyric = g_string_new(lyric+1); // lose     slosh
        }
        printf("New line. Setting front to %s end to "%s"
", lyric+1, current_lyric);

        // Now draw stuff
        paint_background();

        cr = cairo_create(surface);

        int n;
        for (n = 0; n < NUM_LINES; n++) {

            if (lyrics_array[n] != NULL) {
                draw_text(lyrics_array[n]->str,
                          0.0, 0.0, 0.5, height_lyric_pixbufs[n], 0);
            }
        }
        // redraw current and next lines
        if (current_line < lyric_lines.lines->len) {
            if (current_line >= 2) {
                // redraw last line still in red
                GString *gstr = lyrics_array[(current_line-2) % NUM_LINES];
                if (gstr != NULL) {
                    draw_text(gstr->str,
                              1.0, 0.0, 00,
                              height_lyric_pixbufs[(current_line-2) % NUM_LINES],
                              0);
                }
            }
            // draw next line in brighter blue
            coloured_text_offset = draw_text(lyrics_array[(current_line-1) % NUM_LINES]->str,
                      0.0, 0.0, 1.0, height_lyric_pixbufs[(current_line-1) % NUM_LINES], 0);
            printf("coloured text offset %d
", coloured_text_offset);
        }

        if (next_line < lyric_lines.lines->len) {
            lyrics_array[(next_line-1) % NUM_LINES] =
                g_array_index(lyric_lines.lines, GString *, next_line);
        }

        cairo_destroy(cr);
        XFlush(display);

    } else {
        // change text colour as chars are played
        if ((front_of_lyric != NULL) && (lyric != NULL)) {
            g_string_append(front_of_lyric, lyric);
            char *s = front_of_lyric->str;
            //coloured_lines[current_panel].front_of_line = s;

            cairo_t *cr = cairo_create(surface);

            // See http://cairographics.org/FAQ/
            draw_text(s, 1.0, 0.0, 0.0,
                      height_lyric_pixbufs[(current_line-1) % NUM_LINES],
                      coloured_text_offset);

            cairo_destroy(cr);
            XFlush(display);

        }
    }
}

The file x_code.c is compiled with the following:

CFLAGS =   $(shell pkg-config --cflags gtk+-$(V).0 libavformat libavcodec libswscale libavutil )  -ITiMidity++-2.14.0/timidity/ -ITiMidity++-2.14.0/utils

LIBS =  -lasound -l glib-2.0 $(shell pkg-config --libs gtk+-$(V).0  libavformat libavcodec libavutil libswscale) -lpthread -lX11

gcc  -fPIC $(CFLAGS) -c -o x_code.o x_code.c $(LIBS)
 gcc -shared -o if_x.so x_code.o $(LIBS)

Again, this uses a locally compiled and built version of TiMidity because the Ubuntu version crashes. It is run with the following:

TiMidity++-2.14.0/timidity/timidity -d. -ix --trace --trace-text-meta <KAR file>

Playing a Background Video with Gtk

In Chapter 27, I discussed a program to show lyrics overlaid onto a movie. Apart from the previous considerations, the rest of the application follows similarly to the FluidSynth case: build a set of lyric lines, display them using Pango over Gtk pixbufs, and when a new lyric event occurs, update the corresponding colors in the lyric line.

All of the dynamic action needs to occur out of the back end of TiMidity, particularly in the function ctl_event. Other parts such as initializing FFmpeg and Gtk must also occur in the back end when using standard TiMidity. If TiMidity is used as a library, this initialization could occur in the front or the back. For simplicity, you just place it all in the back in the file video_code.c.

As in the previous section, you have some initial data structures and values and will have an array of two lines of coloured_line_t.

struct _lyric_t {
    gchar *lyric;
    long tick;

};
typedef struct _lyric_t lyric_t;

struct _lyric_lines_t {
    char *language;
    char *title;
    char *performer;
    GArray *lines; // array of GString *
};
typedef struct _lyric_lines_t lyric_lines_t;

GArray *lyrics;

lyric_lines_t lyric_lines;

typedef struct _coloured_line_t {
    gchar *line;
    gchar *front_of_line;
    gchar *marked_up_line;
    PangoAttrList *attrs;
#ifdef USE_PIXBUF
    GdkPixbuf *pixbuf;
#endif
} coloured_line_t;

coloured_line_t coloured_lines[2];

GtkWidget *image;

int height_lyric_pixbufs[] = {300, 400}; // vertical offset of lyric in video

int current_panel = 1;  // panel showing current lyric line
int current_line = 0;  // which line is the current lyric
gchar *current_lyric;   // currently playing lyric line
GString *front_of_lyric;  // part of lyric to be coloured red
//GString *end_of_lyric;    // part of lyrci to not be coloured

// Colours seem to get mixed up when putting a pixbuf onto a pixbuf
#ifdef USE_PIXBUF
#define RED blue
#else
#define RED red
#endif

gchar *markup[] = {"<span font="28" foreground="RED">",
                   "</span><span font="28" foreground="white">",
b                   "</span>"};
gchar *markup_newline[] = {"<span foreground="black">",
                           "</span>"};
GString *marked_up_label;

There are now essentially two blocks of code: one to keep the array of colored lines up-to-date as each new lyric is played and one to play the video with the colored lines on top. The first block has three functions: markup_line to prepare a string with the HTML markup, update_line_pixbuf to create a new pixbuf by applying the Pango attributes to the marked-up line, and ctl_lyric, which is triggered on each new lyric event.

The first two functions are as follows:

void markup_line(coloured_line_t *line) {
    GString *str =  g_string_new(markup[0]);
    g_string_append(str, line->front_of_line);
    g_string_append(str, markup[1]);
    g_string_append(str, line->line + strlen(line->front_of_line));
    g_string_append(str, markup[2]);
    printf("Marked up label "%s"
", str->str);

    line->marked_up_line = str->str;
    // we have to free line->marked_up_line

    pango_parse_markup(str->str, -1,0, &(line->attrs), NULL, NULL, NULL);
    g_string_free(str, FALSE);
}

void update_line_pixbuf(coloured_line_t *line) {
    //return;
    cairo_surface_t *surface;
    cairo_t *cr;

    int lyric_width = 480;
    int lyric_height = 60;
    surface = cairo_image_surface_create (CAIRO_FORMAT_ARGB32,
                                          lyric_width, lyric_height);
    cr = cairo_create (surface);

    PangoLayout *layout;
    PangoFontDescription *desc;

    // draw the attributed text
    layout = pango_cairo_create_layout (cr);
    pango_layout_set_text (layout, line->line, -1);
    pango_layout_set_attributes(layout, line->attrs);

    // centre the image in the surface
    int width, height;
    pango_layout_get_pixel_size(layout,
                                &width,
                                &height);
    cairo_move_to(cr, (lyric_width-width)/2, 0);

    pango_cairo_update_layout (cr, layout);
    pango_cairo_show_layout (cr, layout);

    // pull the pixbuf out of the surface
    unsigned char *data = cairo_image_surface_get_data(surface);
    width = cairo_image_surface_get_width(surface);
    height = cairo_image_surface_get_height(surface);
    int stride = cairo_image_surface_get_stride(surface);
    printf("Text surface width %d height %d stride %d
", width, height, stride);

    GdkPixbuf *old_pixbuf = line->pixbuf;
    line->pixbuf = gdk_pixbuf_new_from_data(data, GDK_COLORSPACE_RGB, 1, 8, width, height, stride, NULL, NULL);
    cairo_surface_destroy(surface);
    g_object_unref(old_pixbuf);
}

The function to handle each new lyric event needs to work out if a newline event has occurred, which is when the lyric is the single character. Then it needs to update the current_line index and also to replace the previous line by a new one. Once that is done, for all events the current line is marked and its pixmap generated for drawing. The ctl_lyric function is as follows:

static void ctl_lyric(int lyricid)
{
    char *lyric;

    current_file_info = get_midi_file_info(current_file, 1);

    lyric = event2string(lyricid);
    if(lyric != NULL)
        lyric++;
    printf("Got a lyric %s
", lyric);
    if (*lyric == '') {
        int next_panel = current_panel; // really (current_panel+2)%2
        int next_line = current_line + 2;
        gchar *next_lyric;

        if (current_line + 2 >= lyric_lines.lines->len) {
            return;
        }
        current_line += 1;
        current_panel = (current_panel + 1) % 2;

        // set up new line as current line
        current_lyric = g_array_index(lyric_lines.lines, GString *, current_line)->str;
        front_of_lyric = g_string_new(lyric+1); // lose 
        printf("New line. Setting front to %s end to "%s"
", lyric+1, current_lyric);

        coloured_lines[current_panel].line = current_lyric;
        coloured_lines[current_panel].front_of_line = lyric+1;
        markup_line(coloured_lines+current_panel);
#ifdef USE_PIXBUF
        update_line_pixbuf(coloured_lines+current_panel);
#endif
        // update label for next line after this one
        next_lyric = g_array_index(lyric_lines.lines, GString *, next_line)->str;

        marked_up_label = g_string_new(markup_newline[0]);

        g_string_append(marked_up_label, next_lyric);
        g_string_append(marked_up_label, markup_newline[1]);
        PangoAttrList *attrs;
        gchar *text;
        pango_parse_markup (marked_up_label->str, -1,0, &attrs, &text, NULL, NULL);

        coloured_lines[next_panel].line = next_lyric;
        coloured_lines[next_panel].front_of_line = "";
        markup_line(coloured_lines+next_panel);
        update_line_pixbuf(coloured_lines+next_panel);
    } else {
        // change text colour as chars are played
        if ((front_of_lyric != NULL) && (lyric != NULL)) {
            g_string_append(front_of_lyric, lyric);
            char *s = front_of_lyric->str;
            coloured_lines[current_panel].front_of_line = s;
            markup_line(coloured_lines+current_panel);
            update_line_pixbuf(coloured_lines+current_panel);
        }
    }
}

static gboolean draw_image(gpointer user_data) {
    GdkPixbuf *pixbuf = (GdkPixbuf *) user_data;

    gtk_image_set_from_pixbuf((GtkImage *) image, pixbuf);
    gtk_widget_queue_draw(image);
    g_object_unref(pixbuf);

    return G_SOURCE_REMOVE;
}

The function to play the video and overlay the colored lines has nothing essentially new. It reads a frame from the video and puts it into a pixbuf. Then for each of the lyric panels, it draws the colored line into the pixbuf. Finally, it calls gdk_threads_add_idle so that Gtk can draw the pixbuf in its main thread. The function play_background is as follows:

static void *play_background(void *args) {

    int i;
    AVPacket packet;
    int frameFinished;
    AVFrame *pFrame = NULL;

    int oldSize;
    char *oldData;
    int bytesDecoded;
    GdkPixbuf *pixbuf;
    AVFrame *picture_RGB;
    char *buffer;

    pFrame=av_frame_alloc();

    i=0;
    picture_RGB = avcodec_frame_alloc();
    buffer = malloc (avpicture_get_size(PIX_FMT_RGB24, 720, 576));
    avpicture_fill((AVPicture *)picture_RGB, buffer, PIX_FMT_RGB24, 720, 576);

    int width = pCodecCtx->width;
    int height = pCodecCtx->height;

    sws_ctx = sws_getContext(pCodecCtx->width, pCodecCtx->height, pCodecCtx->pix_fmt,
                                               pCodecCtx->width, pCodecCtx->height, PIX_FMT_RGB24,
                                               SWS_BICUBIC, NULL, NULL, NULL);

    while(av_read_frame(pFormatCtx, &packet)>=0) {
        if(packet.stream_index==videoStream) {
            //printf("Frame %d
", i++);
            usleep(33670);  // 29.7 frames per second
            // Decode video frame
            avcodec_decode_video2(pCodecCtx, pFrame, &frameFinished,
                                  &packet);

            if (frameFinished) {
                //printf("Frame %d
", i++);

                sws_scale(sws_ctx,  (uint8_t const * const *) pFrame->data, pFrame->linesize, 0,
                                                  pCodecCtx->height, picture_RGB->data, picture_RGB->linesize);

                pixbuf = gdk_pixbuf_new_from_data(picture_RGB->data[0], GDK_COLORSPACE_RGB, 0, 8,
                                                  width, height, picture_RGB->linesize[0],
                                                  pixmap_destroy_notify, NULL);

                // Create the destination surface
                cairo_surface_t *surface = cairo_image_surface_create (CAIRO_FORMAT_ARGB32,
                                                                       width, height);
                cairo_t *cr = cairo_create(surface);

                // draw the background image
                gdk_cairo_set_source_pixbuf(cr, pixbuf, 0, 0);
                cairo_paint (cr);

                // draw the lyric
                GdkPixbuf *lyric_pixbuf = coloured_lines[current_panel].pixbuf;
                if (lyric_pixbuf != NULL) {
                    int width = gdk_pixbuf_get_width(lyric_pixbuf);
                    gdk_cairo_set_source_pixbuf(cr,
                                                lyric_pixbuf,
                                                (720-width)/2,
                                                height_lyric_pixbufs[current_panel]);
                    cairo_paint (cr);
                }

                int next_panel = (current_panel+1) % 2;
                lyric_pixbuf = coloured_lines[next_panel].pixbuf;
                if (lyric_pixbuf != NULL) {
                    int width = gdk_pixbuf_get_width(lyric_pixbuf);
                    gdk_cairo_set_source_pixbuf(cr,
                                                lyric_pixbuf,
                                                (720-width)/2,
                                                height_lyric_pixbufs[next_panel]);
                    cairo_paint (cr);
                }

                pixbuf = gdk_pixbuf_get_from_surface(surface,
                                                     0,
                                                     0,
                                                     width,
                                                     height);
                gdk_threads_add_idle(draw_image, pixbuf);

         /* reclaim memory */
                sws_freeContext(sws_ctx);
                g_object_unref(layout);
                cairo_surface_destroy(surface);
                cairo_destroy(cr);

            }
        }
        av_free_packet(&packet);
    }
    sws_freeContext(sws_ctx);

    printf("Video over!
");
    exit(0);
}

It is run with the following:

TiMidity++-2.14.0/timidity/timidity -d. -iv --trace --trace-text-meta <KAR file>

In appearance, it looks like Figure 28-4.

Figure 28-4. Caption

Background Video with TiMidity as Library

The code for this follows the same structure as the code in Chapter 21. It is in the file gtkkaraoke_player_video_pango.c.

#include <stdio.h>
#include <stdlib.h>

#include "sysdep.h"
#include "controls.h"

extern ControlMode  *video_ctl;
extern ControlMode  *ctl;

static void init_timidity() {
    int err;

    timidity_start_initialize();

    if ((err = timidity_pre_load_configuration()) != 0) {
        printf("couldn't pre-load configuration file
");
        exit(1);
    }

    err += timidity_post_load_configuration();

    if (err) {
        printf("couldn't post-load configuration file
");
        exit(1);
    }

    timidity_init_player();

    extern int opt_trace_text_meta_event;
    opt_trace_text_meta_event = 1;

    ctl = &video_ctl;
    //ctl->trace_playing = 1;
    //opt_trace_text_meta_event = 1;

}

#define MIDI_FILE "54154.kar"

static void *play_midi(void *args) {
    char *argv[1];
    argv[0] = MIDI_FILE;
    int argc = 1;

    timidity_play_main(argc, argv);

    printf("Audio finished
");
    exit(0);
}

int main(int argc, char** argv)
{

    int i;

    /* TiMidity */
    init_timidity();
    play_midi(NULL);

    return 0;
}

Background Video with TiMidity as Front End

The interface needs to be built as a shared library with the following:

if_video.so: video_code.c
        gcc  -fPIC $(CFLAGS) -c -o video_code.o video_code.c $(LIBS)
        gcc -shared -o if_video.so video_code.o $(LIBS)

TiMidity is then run with options.

timidity -d. -iv --trace  --trace-text-meta

As before, it crashes TiMidity from the Ubuntu distro but works fine with TiMidity built from source in the current Linux environment.

Adding Microphone Input

At this stage you have a single application that can play a MIDI file, play a background movie, and display highlighted lyrics on top of the video. There is no microphone input to sing along.

Singing along can be handled either within this application or by an external process. If you want to include it in the current application, then you will have to build a mixer for two audio streams. Java does this in the Java Sound package, but in C you would need to do that yourself. It can be done in ALSA but would involve complex ALSA mixer code.

Jack makes it easy to mix audio, from different processes. The earlier section showed how to do that.

A long-term goal is to include scoring, and so on. However, that takes you into the realm of deep signal processing (identifying notes sung using algorithms such as YIN, for example), which is beyond the scope of this book.

Conclusion

This chapter showed you how to use TiMidity as a MIDI player for a karaoke system. On my laptop it uses about 35 percent of the CPU with Gtk 3.0.