[Gstreamer-openmax] Discussion on the hardware accelerator solution in GstOpenMAX project.

[Felipe Contreras]

Hi,

On Wed, 2008-07-30 at 15:00 +0800, ext Ling Shi wrote:
> Hi, all
> I'm in a research project to port gstreamer into embedded system. Now,
> we encounter the issue on how to integrate hardware accelerators
> (DSP/GPU) into gst. After evaluating different solution, we think
> GstOpenMAX project may be the best one for us, because
> 
>   1) OpenMAX is an industry standard
>   2) more and more DSP/GPU vendor support OpenMAX
> 
> But, I still have several questions on this project.
> 1. Does Nokia N8xx serials use GstOpenMAX?
> I know Nokia engineers lead this project. I also know N8xx serials use
> gstreamer as default playback engine, and it uses TI OMAP 2420, which
> has DSP. Can anyone tell me if N8xx use GstOpenMAX? If no, does N8xx
> plan to use it in the future?

The current Maemo Products don't use OpenMAX IL, they use TI's DSP
directly through the open source version of the DSP bridge (DSP
gateway).

The plan is to use OpenMAX IL so we can choose between different
implementations without much effort.

TI has started to provide their OpenMAX IL source code:
http://omapzoom.org/gf/project/openmax/wiki/

> 2. What's GstOpenMAX plan to support DSP/GPU in the future?
> I review several plugins in GstOpenMAX and find current design can
> only support none-tunnel communication.  It's not the best solution in
> hardware, because of bad performance. So, we plan to improve it by
> adding tunneled or proprietary communication. Do you have such plan?
> If yes, can we involve in design?

Indeed, as Bruno mentions, NXP has contributed code that adds support
for tunneled communication. It is maintained in a separate branch and
will soon be merged to the master branch.

> In addition, most accelerators work as a decoder and a render. It
> means, the encoded data sent to it will directly be decoded and
> rendered, and will not be retrieved back again. How the gst or omx
> organize its pipeline in this situation? We are evaluating two
> solutions.
> 
> ===Solution 1===
> We can design a super omx sink component to cover decoder and render.
> This is the solution is used by N8xx.
> src ! demux ! sink
>                          |
>                        super omx sink
>                          |
>                    +--------------------------------------------+
>                     |    hardware accelerator                 |
>                    +--------------------------------------------+

The disadvantage of this solution is that it requires the creation of
many elements to cover all the possible combination of elements. This
becomes specially a problem when you add for example some filtering,
like a volume control, etc.

> ===Solution 2===
> We can separate omx decoder, omx post processer, and omx sink
> elements. We enhance decoder, post processor, and sink plugin in
> GstOpenMAX. If GstOpenMAX plugin found its neighborhood are GstOpenMAX
> plugin, it will try to establish tunneled communication or proprietary
> communication firstly. It means, although we have 3 OMX plugin in gst
> pipeline, there is no data in gst pad and omx port. The last two
> gst/omx plugin only provide control function, but not support process
> data flow. Of cause, If the connection is failed, it will use
> none-tunnel communication.
> 
> src ! demux ! decoder ! post processor ! sink
>                             |                     |                  |
>                      omx dec          omx pp       omx sink
>                             |                     |                  |
>                    +--------------------------------------------+
>                     |    hardware accelerator                 |
>                    +--------------------------------------------+
> 
> It seems solution 2 is more flexible. How about your suggestion on the
> 2 solutions? Which one is feasible? Do you have other solutions?

This is exactly how NXP implemented it and seems to be working fine.

The problem I see with both solutions is that there will be A/V sync
issues when using OMX sinks in tunneling mode. The idea is to solve
these issues by mapping the OMX clock to the GST clock. However, this
hasn't been implemented yet.

Best regards.

-- 
Felipe Contreras