Programming with the OpenMAX AL multimedia library in Android NDK

OpenMAX AL is an application-level multimedia library in C. Android NDK multimedia APIs are based on the OpenMAX AL 1.0.1 standard with Android-specific extensions. The API is available for Android 4.0 or higher. We should note that the API is evolving and the Android NDK team mentioned that the future version of OpenMAX AL API may require developers to change their code.

Getting ready...

Before we start coding with the OpenMAX AL library, it is important to understand some basics about the library. We will briefly describe the library in the following text.

OpenMAX AL refers to the Application Layer interface of the Open Media Acceleration (OpenMAX) library. It is a royalty-free, cross-platform, C-language application level API for developers to create multimedia applications. Its main features include media recording, media playback, media controls (for example, brightness control), and effects. Compared to OpenSL ES library, OpenMAX AL provides features for both video and audio, but it lacks certain audio features like 3D audio and audio effects which OpenSL ES can provide. Some applications may need to use both libraries.

OpenMAX AL defines two profiles, namely media playback and media player/recorder. Android does not implement all features required by either profile, therefore the OpenMAX AL library in Android does not conform either profile. In addition, Android implements some features specific to Android.

The main features provided by Android OpenMAX AL implementation is the ability to process the MPEG-2 transport stream. We can demultiplex the stream, decode the video and audio, and render them as audio output or to the phone screen. This library allows us to have complete control over the media data before it is passed for presentation. For example, we can call OpenGL ES functions to apply graphics effect on video data before rendering it.

For a detailed description of what is supported on Android, we can refer to the OpenMAX AL for Android documentation available with the Android NDK under the docs/openmaxal/ folder.

The design of OpenMAX AL library is similar to OpenSL ES library. They both adopt an object-oriented approach and the fundamental concepts including objects and interfaces are the same. Readers should refer to the previous recipe for a detailed explanation on these concepts.

How to do it...

The following steps describe how to create a simple Android video playback application using the OpenMAX AL functions:

  1. Create an Android application named OpenMAXSLDemo. Set the package name as cookbook.chapter7.openmaxsldemo. Refer to the Loading native libraries and registering native methods recipe of Chapter 2, Java Native Interface for more detailed instructions.
  2. Right-click on the project OpenMAXSLDemo, select Android Tools | Add Native Support.
  3. Add a Java file named MainActivity.java in the package cookbook.chapter7.openmaxsldemo. This Java file loads the native library OpenMAXSLDemo, sets the view, and calls the native methods to play the video.
  4. Add the mylog.h and OpenMAXSLDemo.c files in the jni folder. A part of the code in OpenMAXSLDemo.c is showed in the following code snippet.

    naCreateEngine creates and realizes the engine object and the output mix object.

    void naCreateEngine(JNIEnv* env, jclass clazz) {
   XAresult res;
   res = xaCreateEngine(&engineObject, 0, NULL, 0, NULL, NULL);
   res = (*engineObject)->Realize(engineObject, XA_BOOLEAN_FALSE);
   res = (*engineObject)->GetInterface(engineObject, XA_IID_ENGINE, &engineEngine);
   res = (*engineEngine)->CreateOutputMix(engineEngine, &outputMixObject, 0, NULL, NULL);
   res = (*outputMixObject)->Realize(outputMixObject, XA_BOOLEAN_FALSE);
}

    naCreateStreamingMediaPlayer creates and realizes a media player object with the data source and data sink. It obtains the buffer queue interface and registers the AndroidBufferQueueCallback function as the callback function. The callback function will be invoked after a buffer is processed:

    jboolean naCreateStreamingMediaPlayer(JNIEnv* env, jclass clazz, jstring filename) {
   XAresult res;
   const char *utf8FileName = (*env)->GetStringUTFChars(env, filename, NULL);
   file = fopen(utf8FileName, "rb");
   XADataLocator_AndroidBufferQueue loc_abq = { XA_DATALOCATOR_ANDROIDBUFFERQUEUE, NB_BUFFERS };
   XADataFormat_MIME format_mime = {XA_DATAFORMAT_MIME, XA_ANDROID_MIME_MP2TS, XA_CONTAINERTYPE_MPEG_TS };
   XADataSource dataSrc = {&loc_abq, &format_mime};
   XADataLocator_OutputMix loc_outmix = { XA_DATALOCATOR_OUTPUTMIX, outputMixObject };
   XADataSink audioSnk = { &loc_outmix, NULL };
   XADataLocator_NativeDisplay loc_nd = {XA_DATALOCATOR_NATIVEDISPLAY,       
           (void*)theNativeWindow, NULL};
   XADataSink imageVideoSink = {&loc_nd, NULL};
   XAboolean required[NB_MAXAL_INTERFACES] = {XA_BOOLEAN_TRUE, XA_BOOLEAN_TRUE};
   XAInterfaceID iidArray[NB_MAXAL_INTERFACES] = {XA_IID_PLAY, XA_IID_ANDROIDBUFFERQUEUESOURCE};
   res = (*engineEngine)->CreateMediaPlayer(engineEngine, &playerObj, &dataSrc, NULL,   &audioSnk, &imageVideoSink, NULL, NULL, NB_MAXAL_INTERFACES, iidArray, required );
   (*env)->ReleaseStringUTFChars(env, filename, utf8FileName);
   res = (*playerObj)->Realize(playerObj, XA_BOOLEAN_FALSE);
   res = (*playerObj)->GetInterface(playerObj, XA_IID_PLAY, &playerPlayItf);
   res = (*playerObj)->GetInterface(playerObj, XA_IID_ANDROIDBUFFERQUEUESOURCE, &playerBQItf);
   res = (*playerBQItf)->SetCallbackEventsMask(playerBQItf, XA_ANDROIDBUFFERQUEUEEVENT_PROCESSED);
   res = (*playerBQItf)->RegisterCallback(playerBQItf, AndroidBufferQueueCallback, NULL);
   if (!enqueueInitialBuffers(JNI_FALSE)) {
       return JNI_FALSE;
   }
   res = (*playerPlayItf)->SetPlayState(playerPlayItf, XA_PLAYSTATE_PAUSED);
   res = (*playerPlayItf)->SetPlayState(playerPlayItf, XA_PLAYSTATE_PLAYING);
   return JNI_TRUE;
}

    AndroidBufferQueueCallback is the callback function registered to refill the buffer with media data or handle commands:

    XAresult AndroidBufferQueueCallback(XAAndroidBufferQueueItf caller, void *pCallbackContext, void *pBufferContext,  void *pBufferData, XAuint32 dataSize,  XAuint32 dataUsed, const XAAndroidBufferItem *pItems, XAuint32 itemsLength) {
   XAresult res;
   int ok;
   ok = pthread_mutex_lock(&mutex);
   if (discontinuity) {
       if (!reachedEof) {
           res = (*playerBQItf)->Clear(playerBQItf);
           rewind(file);
            (void) enqueueInitialBuffers(JNI_TRUE);
       }
       discontinuity = JNI_FALSE;
       ok = pthread_cond_signal(&cond);
       goto exit;
   }
   if ((pBufferData == NULL) && (pBufferContext != NULL)) {
       const int processedCommand = *(int *)pBufferContext;
       if (kEosBufferCntxt == processedCommand) {
           goto exit;
       }
   }
   if (reachedEof) {
       goto exit;
   }
   size_t nbRead;
   size_t bytesRead;
   bytesRead = fread(pBufferData, 1, BUFFER_SIZE, file);
   if (bytesRead > 0) {
       if ((bytesRead % MPEG2_TS_PACKET_SIZE) != 0) {
           LOGI(2, "Dropping last packet because it is not whole");
       }
       size_t packetsRead = bytesRead / MPEG2_TS_PACKET_SIZE;
       size_t bufferSize = packetsRead * MPEG2_TS_PACKET_SIZE;
       res = (*caller)->Enqueue(caller, NULL, pBufferData, bufferSize, NULL, 0);
   } else {
       XAAndroidBufferItem msgEos[1];
       msgEos[0].itemKey = XA_ANDROID_ITEMKEY_EOS;
       msgEos[0].itemSize = 0;
       res = (*caller)->Enqueue(caller, (void *)&kEosBufferCntxt, NULL, 0, msgEos, sizeof(XAuint32)*2);
       reachedEof = JNI_TRUE;
   }
exit:
   ok = pthread_mutex_unlock(&mutex);
   return XA_RESULT_SUCCESS;
}
  5. Add an Android.mk file in the jni folder with the following content:
    LOCAL_PATH := $(call my-dir)
include $(CLEAR_VARS)
LOCAL_MODULE    := OpenMAXSLDemo
LOCAL_SRC_FILES := OpenMAXSLDemo.c
LOCAL_LDLIBS := -llog
LOCAL_LDLIBS    += -landroid
LOCAL_LDLIBS    += -lOpenMAXAL
include $(BUILD_SHARED_LIBRARY)
  6. We can use the NativeMedia.ts video file available in the samples/native-media/ directory for testing. The following command can be used to put the video file into the /sdcard/ directory of the testing Android device:
    $ adb push NativeMedia.ts /sdcard/
  7. Build and start the Android application. We can see the GUI as shown in the following screenshot:
    How to do it...

    We can press Play to start playing the video.

How it works...

In this recipe, we used the OpenMAX AL library to implement a simple video player.

Use and build with the OpenMAX AL multimedia library:

In order to call the API functions, we must add the following line to our code:

#include <OMXAL/OpenMAXAL.h>

If we are also using Android-specific features, we should include another header:

#include <OMXAL/OpenMAXAL_Android.h>

In the Android.mk file, we must add the following line to link to the OpenMAX AL multimedia library:

LOCAL_LDLIBS += libOpenMAXAL

OpenMAX AL video playback

Our sample project that is a simplified version of the native media project comes with the Android NDK. The following diagram illustrates how the application works:

OpenMAX AL video playback

In our code, we created and realized the engine and output mix objects at naCreateEngine function. At the naCreateStreamingMediaPlayerfunction function, we created and realized the media player object with the audio data sink set as output mix, video data sink set as native display, and data source set as the Android buffer queue.

When a buffer is consumed, the callback function AndroidBufferQueueCallback is invoked, where we refill the buffer with data from the NativeMedia.ts file and enqueue it to the buffer queue.

There's more....

OpenMAX AL is a complex library. The specification is a good reference when developing applications with OpenMAX AL and it is available with the Android NDK. The Android NDK also comes with a native-media example, which is a good example of how to use the API.

..................Content has been hidden....................

You can't read the all page of ebook, please click here login for view all page.
Reset