The key component of OpenMAX is the component. OpenMAX couples the modules in the meida flow process by abstracting them into components. Under the OpenMAX standard, the data flow is transmitted, processed and displayed through components. In this article, you do not need to understand the detailed internal implementation mechanism of the component, nor do you need to know the code implementation forms of various methods (if you encounter a small amount of code form analysis instructions, you can skip temporarily, just need to know what the work of this code is to complete). What you need to know is what the component is by reading this article? What's its use? What is the main internal composition structure? More detailed introduction will be introduced in the following articles.
- OpenMAX IL spec Manual Download: https://www.khronos.org/files/openmax_il_spec_1_0.pdf
- OpenMAX IL sample download: https://www.khronos.org/files/openmax/sample_implementation/OMX_CONF_MyComponent_Alt.c
- OpenMAX IL header file download: https://www.khronos.org/files/openmax/headers/omx_il_v1/omx_il_v1.zip
What is a component
Component is the abstraction of modules in meida video stream by OpenMAX, such as video input module, video coding module and video decoding module, which can be understood as XXX component. OpenMAX provides a complete component-based programming solution, including data stream exchange and synchronization. In the OpenMAX IL layer, the component represents an independent functional module. The component can be source (video input), destination (video output), codec, filter, separator (audio-video separation), mixer (audio-video mixing) or any other data processing module. A component can represent a hardware device, a software codec, a processor, or a combination thereof, depending on how the component is implemented.
A typical component structure is shown in the figure below:
Among them, video coding and video decoding can exist as a component.
Internal structure of components
The following figure is the internal structure diagram of an abstract component given in the spec Manual of OpenMAX IL:
As can be seen from the above figure, there are probably several components inside the component:
- Component handle (component descriptor, similar to file descriptor).
- Configure related structure methods (set/get parameter/configuration).
- Command queue.
- port.
- buffer management of ports.
- Component event handle (used to send events to IL Client).
- buffer sending module.
The operation process of IL Client for components is as follows:
Component handle
The component handle is the keepsake of the IL Client operation component, which can be regarded as the same thing as the file descriptor in Linux or Windows file programming. In file programming, all operations of the file are completed according to the file descriptor, and all operations of the component are also completed by the component handle. You can treat a component handle as an abstract instance of the entire component.
Configuration related structure method
The main methods are set/get parameter and set/get configuration. Through these two methods, IL Client can set and obtain the properties of components. For OpenMAX, the functions of the two methods are different. Note: these callback methods need to implement their own specific action code.
-
Parameter
-
Set Parameter
The relevant component code of OpenMAX is OMX_SetParameter macro (the detailed implementation of this macro will be introduced in the following articles). Using this macro, a parameter structure will be sent from the IL Client to the component, which contains various parameter information to be set to the component. When the macro is used, the relevant callback function inside the component will be called, and then the relevant component settings will be made according to the passed structure parameters. For example, you can set the port number index of the buffer provider of the component. -
Get Parameter
There is no need to explain this in detail. Naturally, it is to obtain the parameters of the component. The specific component code of OpenMAX is OMX_GetParameter this macro.
-
-
Configuration
-
Set Configuration
The specific code related to OpenMAX is OMX_SetConfig macro (the detailed implementation of this macro will be introduced in the later article). Using this macro will set the values of relevant configurations of components. The macro can be called at any time after the component is initialized and the state changes to Loaded. To call this, you need to provide the memory address of the configuration structure and the relevant internal members of the initialized structure. After the macro is used, the IL Client can discard the relevant configuration structure (a copy of the relevant configuration will be copied inside the component). For example, you can set the time stamp of the component, time jump unit and time scaling factor. -
Get Configuration
The reason is the same as above, but this method is used to obtain the component configuration. Specifically, OMX is the relevant code of OpenMAX_ Getconfig macro.
-
-
Macro definition code implementation (don't worry too much about its further implementation details, which will be explained later)
/* get/set parameter Implementation of macro definition */
#define OMX_GetParameter( \
hComponent, \
nParamIndex, \
pComponentParameterStructure) \
((OMX_COMPONENTTYPE*)hComponent)->GetParameter( \
hComponent, \
nParamIndex, \
pComponentParameterStructure) /* Macro End */
#define OMX_SetParameter( \
hComponent, \
nParamIndex, \
pComponentParameterStructure) \
((OMX_COMPONENTTYPE*)hComponent)->SetParameter( \
hComponent, \
nParamIndex, \
pComponentParameterStructure) /* Macro End */
/* get/set configuration Implementation of macro definition */
#define OMX_GetConfig( \
hComponent, \
nConfigIndex, \
pComponentConfigStructure) \
((OMX_COMPONENTTYPE*)hComponent)->GetConfig( \
hComponent, \
nConfigIndex, \
pComponentConfigStructure) /* Macro End */
#define OMX_SetConfig( \
hComponent, \
nConfigIndex, \
pComponentConfigStructure) \
((OMX_COMPONENTTYPE*)hComponent)->SetConfig( \
hComponent, \
nConfigIndex, \
pComponentConfigStructure) /* Macro End */
Command queue
The command queue is used for various controls of IL Client and components, such as the following commands (the action code corresponding to the control command needs to be implemented by itself):
| command | function |
|---|---|
| OMX_CommandStateSet | Set the status of the component |
| OMX_CommandFlush | Flush the buffer of relevant port |
| OMX_CommandPortDisable | Stops the specified port |
| OMX_CommandPortEnable | Enable the specified port |
| OMX_CommandMarkBuffer | Mark a buffer to specify which component will respond to this event |
In the sample of OpenMAX, the command queue is implemented in the form of pipe, which is divided into:
- Create a new pipe (using the pipe function) in the initialization code of the component. At this time, you can get a pipe instance similar to the file handle.
- Wait for the pipe to be readable (a command arrives) in the internal thread of the component, and use the select function to wait. The pipe instance generated in the previous step can be used as a file descriptor (file handle).
- In the relevant callback function (for command sending, the macro is defined as OMX_SendCommand) write command data into the pipe. Using the write function, the parameters are the pipe instance and command structure obtained in the first step.
- If the select of the internal thread of the component detects that the pipe is readable, it immediately reads out a command for execution. After execution, go to step 2 again to cycle.
For the time being, you don't need to know the code implementation details of the component command queue or the generation process of the command. You just need to know the general function of the command
In addition to the above command queue management in the form of pipe combined with read and write, you can also build a producer consumer data queue model to manage the command queue. In fact, the above method can be regarded as a producer consumer model, and its workflow is shown in the figure below:
The internal thread of the component in the figure above can be regarded as a consumer, while the IL Client can be regarded as a producer. The producer constantly writes commands (production of commands) to the command pipe database, and the consumer constantly reads commands (consumption commands) from the command pipe. The two are carried out in different steps without interference, and the commands will not be lost due to the busy system.
There is also a self implemented producer consumer model, which is realized through linked list. Its basic process is shown in the figure below:
The component maintains two two two-way linked lists through list_head structure. One linked list stores the empty command structure, and the other one stores the newly generated command structure. IL Client continuously takes out a command structure from the empty command structure list, and then fills in the commands to be executed (if the linked list is empty, you need to re apply for a new command structure), Finally, it is put into the filled command structure chain list, and the internal thread of the component continuously takes a command from the prepared command structure chain list, and then distributes and executes it. Finally, the executed command structure is sent back to the empty command structure chain list for the next filling.
port
-
What is PORT used for?
PORT acts as an agent for data exchange between components (embassies abroad). Through the PORT port, two components can transfer data to each other, and can also carry out other controls. -
How to exchange data through PORT?
Components can be bound. After the component binding is completed, the instantiated component handles and binding PORT information of the components of both sides are stored in the PORT ports of the bound components, and then the buffer related callback functions of the bound components are called through PORT, In this way, components can call each other's internal methods to realize data exchange (including data sending and data recycling). -
How to define a PORT?
The definition of a PORT requires three structures. Under the OpenMAX standard, these three structures are OMX_PARAM_PORTDEFINITIONTYPE，OMX_VIDEO_PARAM_PORTFORMATTYPE，OMX_ PARAM_ Buffersupporttype. The second structure is related to the PORT type of the component, including video, image, audio and other types. Here is the video type. These three structure descriptions include PORT type, PORT number, COMP handle and other information. -
How to establish a connection through PORT?
Call the internal method of the component to negotiate connection (negotiate to establish diplomacy). Specifically, OMX is used in OpenMAX_ The macro definition of setuptunnel needs to call the macro definition of both components when binding components. Finally, it can call back to a ComponentTunnelRequest method inside both components to complete the binding of components (the specific code content of this method needs to be implemented by itself).
Note: OMX in cedarc_ Neither setuptunnel nor ComponentTunnelRequest has been implemented. The implementation method in cedarc will be analyzed in detail later.
buffer management of ports
The buffer management of the port is similar to the command queue. In the sample of OpenMAX, the buffer management is also similar to the command queue management. The data generation and reading process is as follows:
The management of data in the internal thread of the component in the sample of OpenMAX is completed through two linked lists (not necessarily two in actual use, but the maintenance mode of the linked list can be customized). One is InputList and the other is OutputList, which store the input and output buffer data respectively.
Note: the buffer mentioned here does not refer to the actual video data or image and audio data, but a buffer header description structure, which contains the starting address, size, buffer type, format and other information of the actual buffer data. The management of the buffer inside the component refers to the management of the buffer description structure. Just imagine that if it is video data, a frame has a size of several meters. If you copy these meters of data every time, the CPU resources will be exhausted. Therefore, buffer management is carried out in the form of passing only the information description of buffer, Only when you really need to process the buffer can you read the buffer data from the actual buffer storage address according to the buffer information description.
Event handle for component
The component will register an OMX with the IL Client during initialization_ Callback function set of callbackktype type. Its structure prototype is as follows:
typedef struct OMX_CALLBACKTYPE
{
/** The EventHandler method is used to notify the application when an
event of interest occurs. Events are defined in the OMX_EVENTTYPE
enumeration. Please see that enumeration for details of what will
be returned for each type of event. Callbacks should not return
an error to the component, so if an error occurs, the application
shall handle it internally. This is a blocking call.
The application should return from this call within 5 msec to avoid
blocking the component for an excessively long period of time.
@param hComponent
handle of the component to access. This is the component
handle returned by the call to the GetHandle function.
@param pAppData
pointer to an application defined value that was provided in the
pAppData parameter to the OMX_GetHandle method for the component.
This application defined value is provided so that the application
can have a component specific context when receiving the callback.
@param eEvent
Event that the component wants to notify the application about.
@param nData1
nData will be the OMX_ERRORTYPE for an error event and will be
an OMX_COMMANDTYPE for a command complete event and OMX_INDEXTYPE for a
OMX_PortSettingsChanged event.
@param nData2
nData2 will hold further information related to the event. Can be OMX_STATETYPE for
a OMX_CommandStateSet command or port index for a OMX_PortSettingsChanged event.
Default value is 0 if not used. )
@param pEventData
Pointer to additional event-specific data (see spec for meaning).
*/
OMX_ERRORTYPE (*EventHandler)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_PTR pAppData,
OMX_IN OMX_EVENTTYPE eEvent,
OMX_IN OMX_U32 nData1,
OMX_IN OMX_U32 nData2,
OMX_IN OMX_PTR pEventData);
/** The EmptyBufferDone method is used to return emptied buffers from an
input port back to the application for reuse. This is a blocking call
so the application should not attempt to refill the buffers during this
call, but should queue them and refill them in another thread. There
is no error return, so the application shall handle any errors generated
internally.
The application should return from this call within 5 msec.
@param hComponent
handle of the component to access. This is the component
handle returned by the call to the GetHandle function.
@param pAppData
pointer to an application defined value that was provided in the
pAppData parameter to the OMX_GetHandle method for the component.
This application defined value is provided so that the application
can have a component specific context when receiving the callback.
@param pBuffer
pointer to an OMX_BUFFERHEADERTYPE structure allocated with UseBuffer
or AllocateBuffer indicating the buffer that was emptied.
@ingroup buf
*/
OMX_ERRORTYPE (*EmptyBufferDone)(
OMX_IN OMX_HANDLETYPE hComponent,
OMX_IN OMX_PTR pAppData,
OMX_IN OMX_BUFFERHEADERTYPE* pBuffer);
/** The FillBufferDone method is used to return filled buffers from an
output port back to the application for emptying and then reuse.
This is a blocking call so the application should not attempt to
empty the buffers during this call, but should queue the buffers
and empty them in another thread. There is no error return, so
the application shall handle any errors generated internally. The
application shall also update the buffer header to indicate the
number of bytes placed into the buffer.
The application should return from this call within 5 msec.
@param hComponent
handle of the component to access. This is the component
handle returned by the call to the GetHandle function.
@param pAppData
pointer to an application defined value that was provided in the
pAppData parameter to the OMX_GetHandle method for the component.
This application defined value is provided so that the application
can have a component specific context when receiving the callback.
@param pBuffer
pointer to an OMX_BUFFERHEADERTYPE structure allocated with UseBuffer
or AllocateBuffer indicating the buffer that was filled.
@ingroup buf
*/
OMX_ERRORTYPE (*FillBufferDone)(
OMX_OUT OMX_HANDLETYPE hComponent,
OMX_OUT OMX_PTR pAppData,
OMX_OUT OMX_BUFFERHEADERTYPE* pBuffer);
}
The three callback members are implemented by the IL Client. The callback of these three callback members can be triggered by some event inside the component. For example, the EventHandler callback member will be called when the component state transition is completed, and the other two will be called when the EmptyBuffer and FillThisBuffer operations are completed. Use these callback functions, The IL Client can receive several message events from within the component, so as to achieve some synchronization and receive feedback from the component.
buffer sending module
tunnel mode (binding mode)
The implementation of this module is an important part inside the component. It is usually located after the command distribution of the thread inside the component (tunnel mode - i.e. binding mode). You can refer to the thread implementation part of OpenMAX sample. The actual buffer sending is completed through the PORT port, which calls the EmptyThisBuffer method of the bound component to complete the data transmission. Let's look at a picture:
As can be seen from the figure, if component A is the data provider, the complete data transmission of one frame is:
- Component A calls OMX of component B (implemented through PORT port)_ Emptythisbuffer (B, Pbuffer) macro to transfer data from A to B.
- Component B calls OMX of component A (implemented through PORT port)_ Fillthisbuffer (A, Pbuffer) macro to complete the return process of data from B to A.
If component B is the provider of data:
- Component B calls OMX of component A (implemented through PORT port)_ Fillthisbuffer (A, Pbuffer) macro is used to complete the transfer process of buffer data from B to A.
- The component A fills the buffer data structure received, and then calls the component B (through the PORT port) OMX_. Emptythisbuffer (B, Pbuffer) macro to realize the return process of data from a to B.
The above process is completed in the internal threads of both components. The specific position within the thread depends on the function of the component and its specific implementation method. The routine is not fixed, and you can make some changes according to your own needs.
Note: openmax in cedarc only implements the above component A as the data provider.
Non tunnel mode (unbound mode)
In this mode, the two sides of data transmission become IL Client and component, and the whole process becomes:
- IL Client via OMX_ The FillThisBuffer (phandle, 1, pbufferout) callback method provides an empty buffer structure to the Output port of component A to be filled. This macro finally completes the call to the FillThisBuffer callback method of component a through the IL Core.
- Then, the IL Client transfers a buffer data to component A. at this time, component a can process the buffer, and then fill the new buffer data generated after processing into the empty buffer structure received in the previous step.
- After component A fills the buffer in the Output, component A will call OMX_FillBufferDone(pBufferOut) method to notify the IL Client to receive the processed data.
- After IL Client data processing, OMX will be called again_ Fill this buffer (phandle, 1, pbufferout) to return the buffer to the Output list and wait for it to be filled again.
- After component a processes the received buffer data, it will call OMX_EmptyBufferDone(pBufferIn) method to notify IL Client that component A has completed the data processing of receiving buffer.
The above process description may seem a little confused. Take an example to illustrate. For example, if component A is a decoding component, the whole process becomes:
- IL Client provides an empty buffer structure to the Output buffer list of component A (decoding component) to store the decoded data. Note that at this time, only the buffer header description is passed to A (see the above introduction to buffer header description - buffer management of port), and the real buffer data storage address is in the IL Client.
- IL Client sends a frame of buffer data to be decoded to the Input port of component A.
- After component A decodes, the decoded data is copied to the buffer queue of the Output port (according to the buffer description, the decoded data is directly copied to the actual buffer storage address of the IL Client). At the same time, the IL Client is notified through callback that the buffer is available (the decoded data is available).
- IL Client processes the decoded data, such as saving it as a file, and then returns the buffer to component A to be filled with the decoded data of the next frame.
- Component A notifies IL Client through callback that the frame data has been decoded. You can transfer the data of the next frame.
State transition of components
- Why state transitions?
Through the state transition of components, the tentative, start and recovery of data transmission between components can be controlled, which can be used for data synchronization and data exchange switch between components. - How to control the status of components?
It is controlled through the callback function provided inside the component, mainly sending OMX through the SendCommand callback_ Commandstateset command. - What are the States?
There are 6 states in total. See the following figure for the types, state characteristics and the conversion relationship between them:
As you can see, the state of the component is initialized to OMX as soon as it is initialized_ Stateloaded, and then wait until the resource is ready, the IL Client will set the state of the component to OMX_StateIdle. When you need to transfer data, you need to set the state to OMX_StateExecuting: the process from component stop to destruction is just the opposite of the above.
- When the component is initialized, the final state changes to Executing, and the data flows normally.
- The data transmission between components stops and the status changes to Idle.
- The status of the component changes to Loaded, and the latter component needs to call the FillThisBuffer callback method of the previous component to return the data.
- Components are officially destroyed and all resources are released.
This article roughly outlines the overall framework of a component, and then introduces the relevant structures inside the component and callback function members.
Article reference:
List of C language_ Head bidirectional linked list
Initial knowledge of OpenMAX programming