©  Jan Newmarch 2017

Jan Newmarch, Linux Sound Programming, 10.1007/978-1-4842-2496-0_10

10. GStreamer

Jan Newmarch1 

(1)Oakleigh, Victoria, Australia

GStreamer is a library of components that can be hooked together in complex pipelines. It can be used for filtering, converting formats, and mixing. It can handle both audio and video formats, but this chapter covers only audio. It looks at the user-level mechanisms for using GStreamer and also the programming model for linking GStreamer components . A reference is given for writing new components.

Resources

Here are some resources:

Overview

GStreamer uses a pipeline model to connect elements, which are sources, filters, and sinks. Figure 10-1 shows the model.

A435426_1_En_10_Fig1_HTML.gif
Figure 10-1. GStreamer pipeline model

Each element has zero or more pads, which can be source pads producing data or sink pads consuming data, as shown in Figure 10-2.

A435426_1_En_10_Fig2_HTML.gif
Figure 10-2. GStreamer source and sink pads

Pads can be static or may be dynamically created or destroyed in response to events. For example, to process a container file such as MP4, the element has to read enough of the file before it can work out the format of the contained object, such as an H.264 video. Once that is done, it can create a source pad for the next stage to consume the data.

GStreamer is not restricted to linear pipelines like command languages such as bash. A demuxer, for example, may need to separate audio from video and process each separately, as in Figure 10-3.

A435426_1_En_10_Fig3_HTML.gif
Figure 10-3. Complex GStreamer pipeline

Elements follow a state model, like the following:

  • GST_STATE_NULL

  • GST_STATE_READY

  • GST_STATE_PAUSED

  • GST_STATE_PLAYING

Generally elements will be created and moved from NULL to PLAYING. Finer control is available with the other states.

Elements can also generate events that contain information on the state of the data stream. Events are generally handled internally but may be watched, such as events signaling the end of a data stream or the format of a data stream.

A plug-in is a loadable block of code. Generally a plug-in contains the implementation of a single element, but it may contain more.

Each pad has an associated list of capabilities. Each capability is a statement about what the pad can handle. This includes information about data types (for example, audio/raw), format (S32LE, U32LE, S16LE, U16LE, and so on), data rate (for example, 1-2147483647 bits per second), and so on. When a source pad is linked to a sink pad, these capabilities are used to determine how the elements will communicate.

Command-Line Processing

There are three levels of dealing with GStreamer: by using the command line, by writing C programs (or Python, Perl, C++, and so on) to link elements, or by writing new elements. This section covers command-line tools.

gst-inspect

The command gst-inspect(on my Ubuntu system, gst-inspect-1.0) with no arguments shows the list of plug-ins, their elements, and a brief description. A brief extract is as follows:

...
audiomixer:  liveadder: AudioMixer
audioparsers:  aacparse: AAC audio stream parser
audioparsers:  ac3parse: AC3 audio stream parser
audioparsers:  amrparse: AMR audio stream parser
audioparsers:  dcaparse: DTS Coherent Acoustics audio stream parser
audioparsers:  flacparse: FLAC audio parser
audioparsers:  mpegaudioparse: MPEG1 Audio Parser
audioparsers:  sbcparse: SBC audio parser
audioparsers:  wavpackparse: Wavpack audio stream parser
audiorate:  audiorate: Audio rate adjuster
...

This shows that the plug-in audioparsers contains a number of elements such as aacparse, which is an “AAC audio stream parser.”

When run with a plug-in as an argument, gst-inspect shows a little more detail about the plug-in.

$gst-inspect-1.0 audioparsers
Plugin Details:
  Name                     audioparsers
  Description              Parsers for various audio formats
  Filename                 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstaudioparsers.so
  Version                  1.8.1
  License                  LGPL
  Source module            gst-plugins-good
  Source release date      2016-04-20
  Binary package           GStreamer Good Plugins (Ubuntu)
  Origin URL               https://launchpad.net/distros/ubuntu/+source/gst-plugins-good1.0

  aacparse: AAC audio stream parser
  amrparse: AMR audio stream parser
  ac3parse: AC3 audio stream parser
  dcaparse: DTS Coherent Acoustics audio stream parser
  flacparse: FLAC audio parser
  mpegaudioparse: MPEG1 Audio Parser
  sbcparse: SBC audio parser
  wavpackparse: Wavpack audio stream parser

  8 features:
  +-- 8 elements

In particular, note that it is from the module gst-plugins-good. Plug-ins are categorized as to stability, licenses, and so on.

When run with the element as an argument, gst-inspectshows a lot of information about the element.

$gst-inspect-1.0 aacparse
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                AAC audio stream parser
  Klass                    Codec/Parser/Audio
  Description              Advanced Audio Coding parser
  Author                   Stefan Kost <[email protected]>

Plugin Details:
  Name                     audioparsers
  Description              Parsers for various audio formats
  Filename                 /usr/lib/x86_64-linux-gnu/gstreamer-1.0/libgstaudioparsers.so
  Version                  1.8.1
  License                  LGPL
  Source module            gst-plugins-good
  Source release date      2016-04-20
  Binary package           GStreamer Good Plugins (Ubuntu)
  Origin URL               https://launchpad.net/distros/ubuntu/+source/gst-plugins-good1.0

GObject
 +----GInitiallyUnowned
       +----GstObject
             +----GstElement
                   +----GstBaseParse
                         +----GstAacParse

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      audio/mpeg
            mpegversion: { 2, 4 }

  SRC template: 'src'
    Availability: Always
    Capabilities:
      audio/mpeg
                 framed: true
            mpegversion: { 2, 4 }
          stream-format: { raw, adts, adif, loas }

Element Flags:
  no flags set

Element Implementation:
  Has change_state() function: gst_base_parse_change_state

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "aacparse0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  disable-passthrough : Force processing (disables passthrough)
                        flags: readable, writable
                        Boolean. Default: false

This shows that it can take audio/mpeg version 2 or 4 and convert the data into audio/mpeg version 2 or 4 in a variety of formats.

gst-discoverer

The command gst-discoverer(on my system gst-discoverer-1.0) can be used to give information about resources such as files or URIs. On an audio file called audio_01.ogg, it gives the following information:

$gst-discoverer-1.0 enigma/audio_01.ogg
Analyzing file:enigma/audio_01.ogg
Done discovering file:enigma/audio_01.ogg

Topology:
  container: Ogg
    audio: Vorbis

Properties:
  Duration: 0:02:03.586666666
  Seekable: yes
  Tags:
      encoder: Xiph.Org libVorbis I 20020717
      encoder version: 0
      audio codec: Vorbis
      nominal bitrate: 112001
      bitrate: 112001
      container format: Ogg

gst-device-monitor

This command can give a lot of information about the devices on your system:

$gst-device-monitor-1.0
Probing devices...

Device found:

        name  : Monitor of Built-in Audio Digital Stereo (HDMI)
        class : Audio/Source
        caps  : audio/x-raw, format=(string){ S16LE, S16BE, F32LE, F32BE, S32LE, S32BE, S24LE, S24BE, S24_32LE, S24_32BE, U8 }, layout=(string)interleaved, rate=(int)[ 1, 2147483647 ], channels=(int)[ 1, 32 ];
                audio/x-alaw, rate=(int)[ 1, 2147483647 ], channels=(int)[ 1, 32 ];
                audio/x-mulaw, rate=(int)[ 1, 2147483647 ], channels=(int)[ 1, 32 ];
        properties:
                device.description = "Monitor of Built-in Audio Digital Stereo (HDMI)"
                device.class = monitor
                alsa.card = 0
                alsa.card_name = "HDA Intel HDMI"
                alsa.long_card_name = "HDA Intel HDMI at 0xf7214000 irq 52"
                alsa.driver_name = snd_hda_intel
                device.bus_path = pci-0000:00:03.0
                sysfs.path = /devices/pci0000:00/0000:00:03.0/sound/card0
                device.bus = pci
                device.vendor.id = 8086
                device.vendor.name = "Intel Corporation"
                device.product.id = 160c
                device.product.name = "Broadwell-U Audio Controller"
                device.form_factor = internal
                device.string = 0
                module-udev-detect.discovered = 1
                device.icon_name = audio-card-pci
...

That is plenty of information about the audio capabilities of my HDMI monitor, and it is followed by other information about the audio and video capabilities of my other devices.

gst-play

This program is a one-stop shop to play all sorts of media files and URIs, as follows:

      $gst-play-1.0 enigma/audio_01.ogg

gst-launch

The gst-launch program allows you to build a pipeline of commands to process media data. The format is as follows:

      gst-launch <elmt> [<args>] ! <elmt> [<args>] ! ...

For example, to play a WAV file through ALSA, use the following:

      $gst-launch-1.0 filesrc location=enigma/audio_01.wav ! wavparse ! alsasink

The hardest part of using GStreamer pipelines appears to be in choosing the appropriate plug-ins. This looks like a bit of a fine art; see the GStreamer cheat sheet at http://wiki.oz9aec.net/index.php/Gstreamer_cheat_sheet for help.

For example, Ogg files are a container format, usually containing Vorbis audio streams and Theora video streams (although they can contain other data formats). They play either the audio or the video or both, and the streams have to be extracted from the container using a demultiplexer, decoded, and then played. There are a variety of ways of just playing the audio, including these three:

    $gst-launch-1.0 filesrc location=enigma/audio_01.ogg ! oggdemux ! vorbisdec ! audioconvert ! alsasink

    $gst-launch-1.0 filesrc location=enigma/audio_01.ogg ! oggdemux ! vorbisdec ! autoaudiosink

    $gst-launch-1.0 uridecodebin uri=file:enigma/audio_01.ogg ! audioconvert ! autoaudiosink

The syntax of GStreamer pipelines allows a pipeline to be split into multiple pipes, for example to manage both the audio and video streams. This is covered in the online documentation of GStreamer.

C Programming

The same pipeline principles hold as for gst-launch, but of course at the C programming level there is far more plumbing to look after. The following program from the GStreamer SDK Basic tutorials at http://docs.gstreamer.com/display/GstSDK/Basic+tutorials does the same as the last of the gst-launch examples ($gst-launch-1.0 uridecodebin uri=... ! audioconvert ! autoaudiosink).

The GStreamer elements are created by calls such as this:

data.source = gst_element_factory_make ("uridecodebin", "source");

The pipeline is built with this:

data.pipeline = gst_pipeline_new ("test-pipeline")
gst_bin_add_many (GST_BIN (data.pipeline), data.source, data.convert , data.sink, NULL);

Eventually all the elements have to be linked. Right now, convert and sink can be linked with the following:

gst_element_link (data.convert, data.sink)

The URI to play is set with the following:

g_object_set (data.source, "uri", "http://docs.gstreamer.com/media/sintel_trailer-480p.webm", NULL);

The data source is a container; in my previous example it was an Ogg container, and here it’s a web media URL. This will not create a source pad on the data source element until enough of the data has been read to determine the data format and parameters. Consequently, the C program has to add an event handler for pad-added, which it does with this:

g_signal_connect (data.source, "pad-added", G_CALLBACK (pad_added_handler), &data);

When a pad is added to the source, the pad_added_handler will be called. This does much type checking and getting the new pad but finally does the key step of linking the source and convert elements.

gst_pad_link (new_pad, sink_pad)

Then the application starts playing by changing the state to PLAYING and waits for normal termination (GST_MESSAGE_EOS) or other messages.

gst_element_set_state (data.pipeline, GST_STATE_PLAYING);
bus = gst_element_get_bus (data.pipeline);
msg = gst_bus_timed_pop_filtered (bus, GST_CLOCK_TIME_NONE,
        GST_MESSAGE_STATE_CHANGED | GST_MESSAGE_ERROR | GST_MESSAGE_EOS);

The last section of code does cleanup. The complete program is as follows:

#include <gst/gst.h>

/* Structure to contain all our information, so we can pass it to callbacks */
typedef struct _CustomData {
  GstElement *pipeline;
  GstElement *source;
  GstElement *convert;
  GstElement *sink;
} CustomData;

/* Handler for the pad-added signal */
static void pad_added_handler (GstElement *src, GstPad *pad, CustomData *data);

int main(int argc, char *argv[]) {
  CustomData data;
  GstBus *bus;
  GstMessage *msg;
  GstStateChangeReturn ret;
  gboolean terminate = FALSE;

  /* Initialize GStreamer */
  gst_init (&argc, &argv);

  /* Create the elements */
  data.source = gst_element_factory_make ("uridecodebin", "source");
  data.convert = gst_element_factory_make ("audioconvert", "convert");
  data.sink = gst_element_factory_make ("autoaudiosink", "sink");

  /* Create the empty pipeline */
  data.pipeline = gst_pipeline_new ("test-pipeline");

  if (!data.pipeline || !data.source || !data.convert || !data.sink) {
    g_printerr ("Not all elements could be created.
");
    return -1;
  }

  /* Build the pipeline. Note that we are NOT linking the source at this
   * point. We will do it later. */
  gst_bin_add_many (GST_BIN (data.pipeline), data.source, data.convert , data.sink, NULL);
  if (!gst_element_link (data.convert, data.sink)) {
    g_printerr ("Elements could not be linked.
");
    gst_object_unref (data.pipeline);
    return -1;
  }

  /* Set the URI to play */
  g_object_set (data.source, "uri", "http://docs.gstreamer.com/media/sintel_trailer-480p.webm", NULL);

  /* Connect to the pad-added signal */
  g_signal_connect (data.source, "pad-added", G_CALLBACK (pad_added_handler), &data);

  /* Start playing */
  ret = gst_element_set_state (data.pipeline, GST_STATE_PLAYING);
  if (ret == GST_STATE_CHANGE_FAILURE) {
    g_printerr ("Unable to set the pipeline to the playing state.
");
    gst_object_unref (data.pipeline);
    return -1;
  }

  /* Listen to the bus */
  bus = gst_element_get_bus (data.pipeline);
  do {
    msg = gst_bus_timed_pop_filtered (bus, GST_CLOCK_TIME_NONE,
        GST_MESSAGE_STATE_CHANGED | GST_MESSAGE_ERROR | GST_MESSAGE_EOS);

    /* Parse message */
    if (msg != NULL) {
      GError *err;
      gchar *debug_info;

      switch (GST_MESSAGE_TYPE (msg)) {
        case GST_MESSAGE_ERROR:
          gst_message_parse_error (msg, &err, &debug_info);
          g_printerr ("Error received from element %s: %s
", GST_OBJECT_NAME (msg->src), err->message);
          g_printerr ("Debugging information: %s
", debug_info ? debug_info : "none");
          g_clear_error (&err);
          g_free (debug_info);
          terminate = TRUE;
          break;
        case GST_MESSAGE_EOS:
          g_print ("End-Of-Stream reached.
");
          terminate = TRUE;
          break;
        case GST_MESSAGE_STATE_CHANGED:
          /* We are only interested in state-changed messages from the pipeline */
          if (GST_MESSAGE_SRC (msg) == GST_OBJECT (data.pipeline)) {
            GstState old_state, new_state, pending_state;
            gst_message_parse_state_changed (msg, &old_state, &new_state, &pending_state);
            g_print ("Pipeline state changed from %s to %s:
",
                gst_element_state_get_name (old_state), gst_element_state_get_name (new_state));
          }
          break;
        default:
          /* We should not reach here */
          g_printerr ("Unexpected message received.
");
          break;
      }
      gst_message_unref (msg);
    }
  } while (!terminate);

  /* Free resources */
  gst_object_unref (bus);
  gst_element_set_state (data.pipeline, GST_STATE_NULL);
  gst_object_unref (data.pipeline);
  return 0;
}

/* This function will be called by the pad-added signal */
static void pad_added_handler (GstElement *src, GstPad *new_pad, CustomData *data) {
  GstPad *sink_pad = gst_element_get_static_pad (data->convert, "sink");
  GstPadLinkReturn ret;
  GstCaps *new_pad_caps = NULL;
  GstStructure *new_pad_struct = NULL;
  const gchar *new_pad_type = NULL;

  g_print ("Received new pad '%s' from '%s':
", GST_PAD_NAME (new_pad), GST_ELEMENT_NAME (src));

  /* If our converter is already linked, we have nothing to do here */
  if (gst_pad_is_linked (sink_pad)) {
    g_print ("  We are already linked. Ignoring.
");
    goto exit;
  }

  /* Check the new pad's type */
  new_pad_caps = gst_pad_get_caps (new_pad);
  new_pad_struct = gst_caps_get_structure (new_pad_caps, 0);
  new_pad_type = gst_structure_get_name (new_pad_struct);
  if (!g_str_has_prefix (new_pad_type, "audio/x-raw")) {
    g_print ("  It has type '%s' which is not raw audio. Ignoring.
", new_pad_type);
    goto exit;
  }

  /* Attempt the link */
  ret = gst_pad_link (new_pad, sink_pad);
  if (GST_PAD_LINK_FAILED (ret)) {
    g_print ("  Type is '%s' but link failed.
", new_pad_type);
  } else {
    g_print ("  Link succeeded (type '%s').
", new_pad_type);
  }

exit:
  /* Unreference the new pad's caps, if we got them */
  if (new_pad_caps != NULL)
    gst_caps_unref (new_pad_caps);

  /* Unreference the sink pad */
  gst_object_unref (sink_pad);
}

Writing Plug-ins

Writing new GStreamer plug-ins is a nontrivial task. The document “GStreamer Writer’s Guide” at https://gstreamer.freedesktop.org/data/doc/gstreamer/head/pwg/html/index.html gives extensive advice on this.

Conclusion

This chapter looked at using GStreamer both from the command line and from an example C program. There is a huge list of available plug-ins that will meet the many needs of audio/visual developers. I have only touched the surface of GStreamer, and it has many other features, including integration with the GTK toolkit.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset