This chapter looks at video processing using the Broadcom GPU on the Raspberry Pi.
Building Programs
You can build the programs in this chapter using the following Makefile:
DMX_INC = -I/opt/vc/include/ -I /opt/vc/include/interface/vmcs_host/ -I/opt/vc/include/interface/vcos/pthreads -I/opt/vc/include/interface/vmcs_host/linuxEGL_INC =OMX_INC = -I /opt/vc/include/ILOMX_ILCLIENT_INC = -I/opt/vc/src/hello_pi/libs/ilclientINCLUDES = $(DMX_INC) $(EGL_INC) $(OMX_INC) $(OMX_ILCLIENT_INC)CFLAGS=-g -DRASPBERRY_PI -DOMX_SKIP64BIT $(INCLUDES)CPPFLAGS =DMX_LIBS = -L/opt/vc/lib/ -lbcm_host -lvcos -lvchiq_arm -lpthreadEGL_LIBS = -L/opt/vc/lib/ -lEGL -lGLESv2OMX_LIBS = -lopenmaxilOMX_ILCLIENT_LIBS = -L/opt/vc/src/hello_pi/libs/ilclient -lilclientLDLIBS = $(DMX_LIBS) $(EGL_LIBS) $(OMX_LIBS) $(OMX_ILCLIENT_LIBS)all: il_decode_video il_render_video
Video Components
The Raspberry Pi has a number of OpenMAX components specifically for video processing, as shown here:
camera
egl_render
video_decode
video_encode
video_render
video_scheduler
video_splitter
Video Formats
OpenMAX has a number of data structures used to get and set information about components. You have seen some of these before.
OMX_PORT_PARAM_TYPE: This is used with the index parameter OMX_IndexParamVideoInit in a call to OMX_GetParameter. This gives the number of video ports and the port number of the first port.
OMX_PARAM_PORTDEFINITIONTYPE: This is used with the index parameter OMX_PARAM_PORTDEFINITIONTYPE to give, for each port, the number of buffers, the size of each buffer, and the direction (input or output) of the port.
OMX_VIDEO_PORTDEFINITIONTYPE: This is a field of the OMX_PARAM_PORTDEFINITIONTYPE for videos.
OMX_VIDEO_PARAM_PORTFORMATTYPE: This is used to get information about the different formats supported by each port.
Some of these were discussed in Chapter 8.
You haven’t looked at the field OMX_VIDEO_PORTDEFINITIONTYPE, which is part of the port definition information. It contains the following relevant fields:
typedef struct OMX_VIDEO_PORTDEFINITIONTYPE {
OMX_STRING cMIMEType;
OMX_NATIVE_DEVICETYPE pNativeRender;
OMX_U32 nFrameWidth;
OMX_U32 nFrameHeight;
OMX_S32 nStride;
OMX_U32 nSliceHeight;
OMX_U32 nBitrate;
OMX_U32 xFramerate;
OMX_BOOL bFlagErrorConcealment;
OMX_VIDEO_CODINGTYPE eCompressionFormat;
OMX_COLOR_FORMATTYPE eColorFormat;
OMX_NATIVE_WINDOWTYPE pNativeWindow;
} OMX_VIDEO_PORTDEFINITIONTYPE;The last two but one fields are the current values set for the port. The possible values are obtained from the next structure OMX_VIDEO_PARAM_PORTFORMATTYPE, so I will discuss it in the next paragraph. The major fields you get here, though, are parameters about the video size: nFrameWidth, nFrameHeight, nStride, and nSliceHeight. These are parameters you often need if you are, say, saving a decoded video in a different format such as MPEG4.
The OMX_VIDEO_PARAM_PORTFORMATTYPE is defined (in section 4.3.5 in the 1.1.2 specification) as follows:
typedef struct OMX_VIDEO_PARAM_PORTFORMATTYPE {OMX_U32 nSize;OMX_VERSIONTYPE nVersion;OMX_U32 nPortIndex;OMX_U32 nIndex;OMX_VIDEO_CODINGTYPE eCompressionFormat;OMX_COLOR_FORMATTYPE eColorFormat;OMX_U32 xFramerate;} OMX_VIDEO_PARAM_PORTFORMATTYPE;
The first two fields are common to all OpenMAX structures. The nPortIndex field is the port you are looking at. The nIndex field is to distinguish between all the different format types supported by this port. The eCompressionFormat and eColorFormat fields give information about the format, while the last field gives the frame rate of the video in frames per second.
The values for OMX_VIDEO_CODINGTYPE are given in Table 4-66 of the 1.1.2 specification and on the RPi are given in the file /opt/vc/include/IL/OMX_Video.h, as shown here:
typedef enum OMX_VIDEO_CODINGTYPE {OMX_VIDEO_CodingUnused, /** Value when coding is N/A */OMX_VIDEO_CodingAutoDetect, /** Autodetection of coding type */OMX_VIDEO_CodingMPEG2, /** AKA: H.262 */OMX_VIDEO_CodingH263, /** H.263 */OMX_VIDEO_CodingMPEG4, /** MPEG-4 */OMX_VIDEO_CodingWMV, /** all versions of Windows Media Video */OMX_VIDEO_CodingRV, /** all versions of Real Video */OMX_VIDEO_CodingAVC, /** H.264/AVC */OMX_VIDEO_CodingMJPEG, /** Motion JPEG */OMX_VIDEO_CodingKhronosExtensions = 0x6F000000, /** Reserved region for introducing Khronos Standard Extensions */OMX_VIDEO_CodingVendorStartUnused = 0x7F000000, /** Reserved region for introducing Vendor Extensions */OMX_VIDEO_CodingVP6, /** On2 VP6 */OMX_VIDEO_CodingVP7, /** On2 VP7 */OMX_VIDEO_CodingVP8, /** On2 VP8 */OMX_VIDEO_CodingYUV, /* raw YUV video */OMX_VIDEO_CodingSorenson, /** Sorenson */OMX_VIDEO_CodingTheora, /** Theora */OMX_VIDEO_CodingMVC, /** H.264/MVC */OMX_VIDEO_CodingMax = 0x7FFFFFFF} OMX_VIDEO_CODINGTYPE;
The type OMX_COLOR_FORMATTYPE is more informative. It is defined in the file /opt/vc/include/IL/OMX_IVCommon.h, as follows:
typedef enum OMX_COLOR_FORMATTYPE {OMX_COLOR_FormatUnused,OMX_COLOR_FormatMonochrome,OMX_COLOR_Format8bitRGB332,OMX_COLOR_Format12bitRGB444,OMX_COLOR_Format16bitARGB4444,OMX_COLOR_Format16bitARGB1555,OMX_COLOR_Format16bitRGB565,OMX_COLOR_Format16bitBGR565,OMX_COLOR_Format18bitRGB666,OMX_COLOR_Format18bitARGB1665,OMX_COLOR_Format19bitARGB1666,OMX_COLOR_Format24bitRGB888,OMX_COLOR_Format24bitBGR888,OMX_COLOR_Format24bitARGB1887,OMX_COLOR_Format25bitARGB1888,OMX_COLOR_Format32bitBGRA8888,OMX_COLOR_Format32bitARGB8888,OMX_COLOR_FormatYUV411Planar,OMX_COLOR_FormatYUV411PackedPlanar,OMX_COLOR_FormatYUV420Planar,OMX_COLOR_FormatYUV420PackedPlanar,OMX_COLOR_FormatYUV420SemiPlanar,OMX_COLOR_FormatYUV422Planar,OMX_COLOR_FormatYUV422PackedPlanar,OMX_COLOR_FormatYUV422SemiPlanar,OMX_COLOR_FormatYCbYCr,OMX_COLOR_FormatYCrYCb,OMX_COLOR_FormatCbYCrY,OMX_COLOR_FormatCrYCbY,OMX_COLOR_FormatYUV444Interleaved,OMX_COLOR_FormatRawBayer8bit,OMX_COLOR_FormatRawBayer10bit,OMX_COLOR_FormatRawBayer8bitcompressed,OMX_COLOR_FormatL2,OMX_COLOR_FormatL4,OMX_COLOR_FormatL8,OMX_COLOR_FormatL16,OMX_COLOR_FormatL24,OMX_COLOR_FormatL32,OMX_COLOR_FormatYUV420PackedSemiPlanar,OMX_COLOR_FormatYUV422PackedSemiPlanar ,OMX_COLOR_Format18BitBGR666,OMX_COLOR_Format24BitARGB6666,OMX_COLOR_Format24BitABGR6666,OMX_COLOR_FormatKhronosExtensions = 0x6F000000, /** Reserved region for introducing Khronos Standard Extensions */OMX_COLOR_FormatVendorStartUnused = 0x7F000000, /** Reserved region for introducing Vendor Extensions */OMX_COLOR_Format32bitABGR8888,OMX_COLOR_Format8bitPalette,OMX_COLOR_FormatYUVUV128,OMX_COLOR_FormatRawBayer12bit,OMX_COLOR_FormatBRCMEGL,OMX_COLOR_FormatBRCMOpaque,OMX_COLOR_FormatYVU420PackedPlanar,OMX_COLOR_FormatYVU420PackedSemiPlanar,OMX_COLOR_FormatMax = 0x7FFFFFFF} OMX_COLOR_FORMATTYPE;
That’s a rather large number of formats!
Running the program info from Chapter 8 shows the following for the video_decode component:
Video ports:Ports start on 130There are 2 open portsPort 130 has 20 buffers (minimum 1) of size 81920Direction is inputSupported video formats are:Video format encoding 0x14Video compression format 0x4Video format encoding 0x14Video compression format 0x7Video format encoding 0x14Video compression format 0x7F000007Video format encoding 0x14Video compression format 0x8Video format encoding 0x14Video compression format 0x5Video format encoding 0x14Video compression format 0x3Video format encoding 0x14Video compression format 0x2Video format encoding 0x14Video compression format 0x7F000001Video format encoding 0x14Video compression format 0x7F000002Video format encoding 0x14Video compression format 0x7F000003Video format encoding 0x14Video compression format 0x6Video format encoding 0x14Video compression format 0x7F000004Video format encoding 0x14Video compression format 0x7F000005Video format encoding 0x14Video compression format 0x7F000006Video format encoding 0x14Video compression format 0x0No more formats supportedPort 131 has 1 buffers (minimum 1) of size 115200Direction is outputSupported video formats are:Video format encoding 0x14Video compression format 0x0No more formats supported
The compression formats are as follows:
0x2: OMX_VIDEO_CodingMPEG2
0x3: OMX_VIDEO_CodingH263
0x4: OMX_VIDEO_CodingMPEG4
0x5: OMX_VIDEO_CodingWMV
0x6: OMX_VIDEO_CodingRV
0x7: OMX_VIDEO_CodingAVC
0x8: OMX_VIDEO_CodingMJPEG
0x7F000001: OMX_VIDEO_CodingVP6
0x7F000002: OMX_VIDEO_CodingVP7
0x7F000003: OMX_VIDEO_CodingVP8
0x7F000004: OMX_VIDEO_CodingYUV
0x7F000005: OMX_VIDEO_CodingSorenson
0x7F000006: OMX_VIDEO_CodingTheora
Currently I have not confirmed any of these except for H.264 (CodingAVC) .
Decoding an H.264 File
The program to decode a video into an uncompressed format is essentially the same as the one for decoding an image. There is an essential difference: reading one block is not enough for the component to get information about the video file format, and multiple block reads generally have to occur.
In this example, you just slap a loop around reading the initial blocks until a PortSettingsChanged event occurs. The call to ilclient_wait_for_event will time out for the first set of calls, so you drop the timeout period from 10 seconds down to 2 seconds. This isn’t any good for interactive viewing, of course, but here you are just decoding.
The significant change from the image-decoding program is this initial loop:
int port_settings_changed = 0;while (!port_settings_changed) {buff_header =ilclient_get_input_buffer(component,130,1 /* block */);if (buff_header != NULL) {read_into_buffer_and_empty(fp,component,buff_header,&toread);// If all the file has been read in, then// we have to re-read this first block.// Broadcom bug?if (toread <= 0) {printf("Rewinding ");// wind back to start and repeatfp = freopen(VIDEO, "r", fp);toread = get_file_size(VIDEO);}}// try if this block sets params for output porterr = ilclient_wait_for_event(component,OMX_EventPortSettingsChanged,131, 0, 0, 1,ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED,2000) ;if (err < 0) {printf("Wait for port settings changed timed out ");} else {port_settings_changed = 1;}}
The full program is il_decode_video.c.
#include <stdio.h>#include <stdlib.h>#include <sys/stat.h>#include <OMX_Core.h>#include <OMX_Component.h>#include <bcm_host.h>#include <ilclient.h>#define video "/opt/vc/src/hello_pi/hello_video/test.h264"void printState(OMX_HANDLETYPE handle) {// elided}char *err2str(int err) {return "elided";}void eos_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {fprintf(stderr, "Got eos event ");}void error_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {fprintf(stderr, "OMX error %s ", err2str(data));}int get_file_size(char *fname) {struct stat st;if (stat(fname, &st) == -1) {perror("Stat'ing video file");return -1;}return(st.st_size);}static void set_video_decoder_input_format(COMPONENT_T *component) {// set input video formatprintf("Setting video decoder format ");OMX_VIDEO_PARAM_PORTFORMATTYPE videoPortFormat;//setHeader(&videoPortFormat, sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));memset(&videoPortFormat, 0, sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));videoPortFormat.nSize = sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE);videoPortFormat.nVersion.nVersion = OMX_VERSION;videoPortFormat.nPortIndex = 130;videoPortFormat.eCompressionFormat = OMX_VIDEO_CodingAVC;OMX_SetParameter(ilclient_get_handle(component),OMX_IndexParamVideoPortFormat, &videoPortFormat);}OMX_ERRORTYPE read_into_buffer_and_empty(FILE *fp,COMPONENT_T *component,OMX_BUFFERHEADERTYPE *buff_header,int *toread) {OMX_ERRORTYPE r;int buff_size = buff_header->nAllocLen;int nread = fread(buff_header->pBuffer, 1, buff_size, fp);buff_header->nFilledLen = nread;*toread -= nread;printf("Read %d, %d still left ", nread, *toread);if (*toread <= 0) {printf("Setting EOS on input ");buff_header->nFlags |= OMX_BUFFERFLAG_EOS;}r = OMX_EmptyThisBuffer(ilclient_get_handle(component),buff_header);if (r != OMX_ErrorNone) {fprintf(stderr, "Empty buffer error %s ",err2str(r));}return r;}OMX_ERRORTYPE save_info_from_filled_buffer(COMPONENT_T *component,OMX_BUFFERHEADERTYPE * buff_header) {OMX_ERRORTYPE r ;printf("Got a filled buffer with %d, allocated %d ",buff_header->nFilledLen,buff_header->nAllocLen);if (buff_header->nFlags & OMX_BUFFERFLAG_EOS) {printf("Got EOS on output ");exit(0);}// and then refill itr = OMX_FillThisBuffer(ilclient_get_handle(component),buff_header);if (r != OMX_ErrorNone) {fprintf(stderr, "Fill buffer error %s ",err2str(r));}return r;}void get_output_port_settings(COMPONENT_T *component) {OMX_PARAM_PORTDEFINITIONTYPE portdef;printf("Port settings changed ");// need to setup the input for the resizer with the output of the// decoderportdef.nSize = sizeof(OMX_PARAM_PORTDEFINITIONTYPE);portdef.nVersion.nVersion = OMX_VERSION;portdef.nPortIndex = 131;OMX_GetParameter(ilclient_get_handle(component),OMX_IndexParamPortDefinition, &portdef);unsigned int uWidth =(unsigned int) portdef.format.video.nFrameWidth;unsigned int uHeight =(unsigned int) portdef.format.video.nFrameHeight;unsigned int uStride =(unsigned int) portdef.format.video.nStride;unsigned int uSliceHeight =(unsigned int) portdef.format.video.nSliceHeight;printf("Frame width %d, frame height %d, stride %d, slice height %d ",uWidth,uHeight,uStride,uSliceHeight);printf("Getting format Compression 0x%x Color Format: 0x%x ",(unsigned int) portdef.format.video.eCompressionFormat,(unsigned int) portdef.format.video.eColorFormat);}int main(int argc, char** argv) {int i;char *componentName;int err;ILCLIENT_T *handle;COMPONENT_T *component;FILE *fp = fopen(video, "r");int toread = get_file_size(video);OMX_BUFFERHEADERTYPE *buff_header;componentName = "video_decode";bcm_host_init();handle = ilclient_init() ;if (handle == NULL) {fprintf(stderr, "IL client init failed ");exit(1);}if (OMX_Init() != OMX_ErrorNone) {ilclient_destroy(handle);fprintf(stderr, "OMX init failed ");exit(1);}ilclient_set_error_callback(handle,error_callback,NULL);ilclient_set_eos_callback(handle,eos_callback,NULL);err = ilclient_create_component(handle,&component,componentName,ILCLIENT_DISABLE_ALL_PORTS|ILCLIENT_ENABLE_INPUT_BUFFERS|ILCLIENT_ENABLE_OUTPUT_BUFFERS);if (err == -1) {fprintf(stderr, "Component create failed ");exit(1);}printState(ilclient_get_handle(component));err = ilclient_change_component_state(component,OMX_StateIdle);if (err < 0) {fprintf(stderr, "Couldn't change state to Idle ");exit(1);}printState(ilclient_get_handle(component));// must be before we enable buffersset_video_decoder_input_format(component);// input portilclient_enable_port_buffers(component, 130,NULL, NULL, NULL);ilclient_enable_port(component, 130);err = ilclient_change_component_state(component,OMX_StateExecuting);if (err < 0) {fprintf(stderr, "Couldn't change state to Executing ");exit(1);}printState(ilclient_get_handle(component));// Read the first block so that the component can get// the dimensions of the video and call port settings// changed on the output port to configure itint port_settings_changed = 0;while (!port_settings_changed) {buff_header =ilclient_get_input_buffer(component,130,1 /* block */);if (buff_header != NULL) {read_into_buffer_and_empty(fp,component,buff_header,&toread);// If all the file has been read in, then// we have to re-read this first block.// Broadcom bug?if (toread <= 0) {printf("Rewinding ");// wind back to start and repeatfp = freopen(video, "r", fp);toread = get_file_size(video);}}// wait for first input block to set params for output porterr = ilclient_wait_for_event(component,OMX_EventPortSettingsChanged,131, 0, 0, 1,ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED,2000);if (err < 0) {printf("Wait for port settings changed timed out ");} else {port_settings_changed = 1;}err = ilclient_remove_event(component,OMX_EventPortSettingsChanged,131, 0, 0, 1);if (err < 0) {printf("Wait for remove port settings changed timed out ");} else {port_settings_changed = 1;}}get_output_port_settings(component);// now enable output port since port params have been setilclient_enable_port_buffers(component, 131,NULL, NULL, NULL);ilclient_enable_port(component, 131);// now work through the filewhile (toread > 0) {OMX_ERRORTYPE r;// do we have an input buffer we can fill and empty?buff_header =ilclient_get_input_buffer(component,130,1 /* block */);if (buff_header != NULL) {read_into_buffer_and_empty(fp,component,buff_header,&toread);}// do we have an output buffer that has been filled?buff_header =ilclient_get_output_buffer(component,131,0 /* no block */);if (buff_header != NULL) {save_info_from_filled_buffer(component,buff_header);} else {printf("No filled buffer ");}}while (1) {printf("Getting last output buffers ");buff_header =ilclient_get_output_buffer(component,131,1 /* block */);if (buff_header != NULL) {save_info_from_filled_buffer(component,buff_header);}}exit(0);}
The program runs with no command-line parameters . The Videocore library comes with a standard H.264 example clip, /opt/vc/src/hello_pi/hello_video/test.h264, from Big Buck Bunny. It reads multiple frames before a PortSettngsChanged event occurs at which time the video format can be determined.
...Read 81920, 30353369 still leftEmpty buffer doneWait for port settings changed timed outWait for remove port settings changed timed outRead 81920, 30271449 still leftEmpty buffer doneWait for remove port settings changed timed outPort settings changedFrame width 1920, frame height 1080, stride 1920, slice height 1088Getting format Compression 0x0 Color Format: 0x14...
Rendering an H.264 Video
In a similar manner to decoding an image, rendering a video closely follows the code for rendering an image. However, there is now a major difference in how you wait for the initial PortSettingsChanged event. You have seen these two possibilities so far:
Read the first part of the data and then block, waiting for the PortSettingsChanged event that you know will come.
Keep reading data, each time block waiting for the PortSettingsChanged event and timing out if it doesn’t arrive.
Neither of these two is acceptable for the immediate rendering of videos. It may take more than one read, and waiting on timeouts is too slow. Instead, you need to adopt an alternative approach based on the ilclient_remove_event function, which returns true if it can remove an event from the list of those events already seen.
The revised wait for the PortSettingsChanged event is as follows:
while there is more dataread a new bufferempty the bufferif a PortSettingsChanged event has occurredbreakif there is no more datablock waiting for a PortSettingsChanged eventif failed on timeouterrorelsebreak
(This algorithm should be generally applicable. But it breaks on the tunneled image rendering for unknown reasons.)
The video-rendering program using this is il_render_video.c.
#include <stdio.h>#include <stdlib.h>#include <sys/stat.h>#include <OMX_Core.h>#include <OMX_Component.h>#include <bcm_host.h>#include <ilclient.h>#define VIDEO "/opt/vc/src/hello_pi/hello_video/test.h264"char *video_file = VIDEO;void printState(OMX_HANDLETYPE handle) {// elided}char *err2str(int err) {return "elided";}void eos_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {fprintf(stderr, "Got eos event ");}void error_callback(void *userdata, COMPONENT_T *comp, OMX_U32 data) {fprintf(stderr, "OMX error %s ", err2str(data));}int get_file_size(char *fname) {struct stat st;if (stat(fname, &st) == -1) {perror("Stat'ing video file");return -1;}return(st.st_size);}unsigned int uWidth;unsigned int uHeight;OMX_ERRORTYPE read_into_buffer_and_empty(FILE *fp,COMPONENT_T *component,OMX_BUFFERHEADERTYPE *buff_header,int *toread) {OMX_ERRORTYPE r;int buff_size = buff_header->nAllocLen;int nread = fread(buff_header->pBuffer, 1, buff_size, fp);buff_header->nFilledLen = nread;*toread -= nread;printf("Read %d, %d still left ", nread, *toread);if (*toread <= 0) {printf("Setting EOS on input ");buff_header->nFlags |= OMX_BUFFERFLAG_EOS;}r = OMX_EmptyThisBuffer(ilclient_get_handle(component),buff_header);if (r != OMX_ErrorNone) {fprintf(stderr, "Empty buffer error %s ",err2str(r));}return r;}static void set_video_decoder_input_format(COMPONENT_T *component) {int err;// set input video formatprintf("Setting video decoder format ");OMX_VIDEO_PARAM_PORTFORMATTYPE videoPortFormat;//setHeader(&videoPortFormat, sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));memset(&videoPortFormat, 0, sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE));videoPortFormat.nSize = sizeof(OMX_VIDEO_PARAM_PORTFORMATTYPE);videoPortFormat.nVersion.nVersion = OMX_VERSION;videoPortFormat.nPortIndex = 130;videoPortFormat.eCompressionFormat = OMX_VIDEO_CodingAVC;err = OMX_SetParameter(ilclient_get_handle(component),OMX_IndexParamVideoPortFormat, &videoPortFormat);if (err != OMX_ErrorNone) {fprintf(stderr, "Error setting video decoder format %s ", err2str(err));exit(1);} else {printf("Video decoder format set up ok ");}}void setup_decodeComponent(ILCLIENT_T *handle,char *decodeComponentName,COMPONENT_T **decodeComponent) {int err;err = ilclient_create_component(handle,decodeComponent,decodeComponentName,ILCLIENT_DISABLE_ALL_PORTS|ILCLIENT_ENABLE_INPUT_BUFFERS|ILCLIENT_ENABLE_OUTPUT_BUFFERS);if (err == -1) {fprintf(stderr, "DecodeComponent create failed ");exit(1);}printState(ilclient_get_handle(*decodeComponent));err = ilclient_change_component_state(*decodeComponent,OMX_StateIdle);if (err < 0) {fprintf(stderr, "Couldn't change state to Idle ");exit(1);}printState(ilclient_get_handle(*decodeComponent));// must be before we enable buffersset_video_decoder_input_format(*decodeComponent);}void setup_renderComponent(ILCLIENT_T *handle,char *renderComponentName,COMPONENT_T **renderComponent) {int err;err = ilclient_create_component(handle,renderComponent,renderComponentName,ILCLIENT_DISABLE_ALL_PORTS|ILCLIENT_ENABLE_INPUT_BUFFERS);if (err == -1) {fprintf(stderr, "RenderComponent create failed ");exit(1);}printState(ilclient_get_handle(*renderComponent));err = ilclient_change_component_state(*renderComponent,OMX_StateIdle);if (err < 0) {fprintf(stderr, "Couldn't change state to Idle ");exit(1);}printState(ilclient_get_handle(*renderComponent));}int main(int argc, char** argv) {int i;char *decodeComponentName;char *renderComponentName;int err;ILCLIENT_T *handle;COMPONENT_T *decodeComponent;COMPONENT_T *renderComponent;FILE *fp;int toread;OMX_BUFFERHEADERTYPE *buff_header;if (argc == 2) {video_file = argv[1];}if ((fp = fopen(video_file, "r")) == NULL) {fprintf(stderr, "Can't open: %s ", video_file);exit(2);}if ((toread = get_file_size(video_file)) == -1) {fprintf(stderr, "Can't stat: %s ", video_file);exit(2);}decodeComponentName = "video_decode";renderComponentName = "video_render";bcm_host_init();handle = ilclient_init();if (handle == NULL) {fprintf(stderr, "IL client init failed ");exit(1);}if (OMX_Init() != OMX_ErrorNone) {ilclient_destroy(handle);fprintf(stderr, "OMX init failed ");exit(1);}ilclient_set_error_callback(handle,error_callback,NULL);ilclient_set_eos_callback(handle,eos_callback,NULL);setup_decodeComponent(handle, decodeComponentName, &decodeComponent);setup_renderComponent(handle, renderComponentName, &renderComponent);// both components now in Idle state, no buffers, ports disabled// input portilclient_enable_port_buffers(decodeComponent, 130,NULL, NULL, NULL);ilclient_enable_port(decodeComponent, 130);err = ilclient_change_component_state(decodeComponent,OMX_StateExecuting);if (err < 0) {fprintf(stderr, "Couldn't change state to Executing ");exit(1);}printState(ilclient_get_handle(decodeComponent));// Read blocks and break out when we see a// PortSettingsChangedwhile (toread > 0) {buff_header =ilclient_get_input_buffer(decodeComponent,130,1 /* block */);if (buff_header != NULL) {read_into_buffer_and_empty(fp,decodeComponent,buff_header,&toread);// If all the file has been read in, then// we have to re-read this first block.// Broadcom bug?if (toread <= 0) {printf("Rewinding ");// wind back to start and repeatfp = freopen(video_file, "r", fp);toread = get_file_size(video_file);}}if (toread > 0 && ilclient_remove_event(decodeComponent,OMX_EventPortSettingsChanged,131, 0, 0, 1) == 0) {printf("Removed port settings event ");break;} else {printf("No portr settting seen yet ");}// wait for first input block to set params for output portif (toread == 0) {// wait for first input block to set params for output porterr = ilclient_wait_for_event(decodeComponent,OMX_EventPortSettingsChanged,131, 0, 0, 1,ILCLIENT_EVENT_ERROR | ILCLIENT_PARAMETER_CHANGED,2000);if (err < 0) {fprintf(stderr, "No port settings change ");//exit(1);} else {printf("Port settings changed ");break;}}}// set the decode component to idle and disable its portserr = ilclient_change_component_state(decodeComponent,OMX_StateIdle);if (err < 0) {fprintf(stderr, "Couldn't change state to Idle ");exit(1);}ilclient_disable_port(decodeComponent, 131);ilclient_disable_port_buffers(decodeComponent, 131,NULL, NULL, NULL);// set up the tunnel between decode and render portserr = OMX_SetupTunnel(ilclient_get_handle(decodeComponent),131,ilclient_get_handle(renderComponent),90);if (err != OMX_ErrorNone) {fprintf(stderr, "Error setting up tunnel %X ", err);exit(1);} else {printf("Tunnel set up ok ");}// Okay to go back to processing data// enable the decode output portsOMX_SendCommand(ilclient_get_handle(decodeComponent),OMX_CommandPortEnable, 131, NULL);ilclient_enable_port(decodeComponent, 131);// enable the render output portsOMX_SendCommand(ilclient_get_handle(renderComponent),OMX_CommandPortEnable, 90, NULL);ilclient_enable_port(renderComponent, 90);// set both components to executing stateerr = ilclient_change_component_state(decodeComponent,OMX_StateExecuting);if (err < 0) {fprintf(stderr, "Couldn't change state to Idle ");exit(1);}err = ilclient_change_component_state(renderComponent,OMX_StateExecuting);if (err < 0) {fprintf(stderr, "Couldn't change state to Idle ");exit(1);}// now work through the filewhile (toread > 0) {OMX_ERRORTYPE r;// do we have a decode input buffer we can fill and empty?buff_header =ilclient_get_input_buffer(decodeComponent,130,1 /* block */);if (buff_header != NULL) {read_into_buffer_and_empty(fp,decodeComponent,buff_header,&toread);}}ilclient_wait_for_event(renderComponent,OMX_EventBufferFlag,90, 0, OMX_BUFFERFLAG_EOS, 0,ILCLIENT_BUFFER_FLAG_EOS, 10000);printf("EOS on render ");sleep(10);exit(0);}
The program takes the default H.264 but can be overridden by a command-line option. It plays the short segment and then terminates.
Going Full-Screen
The Broadcom version of OpenMAX has a nonstandard configuration mechanism that can be applied to video-rendering components. This can be done as follows:
OMX_CONFIG_DISPLAYREGIONTYPE display_region;display_region.nSize = sizeof(OMX_CONFIG_DISPLAYREGIONTYPE);display_region.nVersion.nVersion = OMX_VERSION;display_region.nPortIndex = 90;display_region.set = OMX_DISPLAY_SET_FULLSCREEN | OMX_DISPLAY_SET_NOASPECT;display_region.fullscreen = OMX_TRUE;display_region.noaspect = OMX_TRUE;err = OMX_SetConfig(ilclient_get_handle(*renderComponent),OMX_IndexConfigDisplayRegion,&display_region);if(err != OMX_ErrorNone) {fprintf(stderr, "Failed to set Full screen %x %s ",err, err2str(err));exit(1);}
Conclusion
This chapter has looked at decoding and rendering H.264 video streams. Playing MP4 files is covered in Chapter 16.
Resources
OpenMAX IL: The Standard for Media Library Portability: https://www.khronos.org/openmax/il/
OpenMAX IL 1.1.2 Specification: https://www.khronos.org/registry/omxil/specs/OpenMAX_IL_1_1_2_Specification.pdf
Raspberry Pi OpenMAX forum: www.raspberrypi.org/forums/viewforum.php?f=70
VMCS-X OpenMAX IL Components: www.jvcref.com/files/PI/documentation/ilcomponents/
Source code for ARM side libraries for interfacing to Raspberry Pi GPU: https://github.com/raspberrypi/userland