[Gstreamer-openmax] Discussion on the hardware accelerator solution in GstOpenMAX project.

[Victor Manuel Jáquez Leal]

Even though I haven't work in these things for a while, I used to and
have thoughts about them.

I haven't read the NXP tunneling implementation in gstopenmax but once
we implemented something related: when the omx gst element is linked
with another omx gst element a tunnel is setup, but as no buffers
traverse the pipeline, because the buffer communication is done
beneath the omx layer, we had to push ghost buffer (empty buffers with
calculated metadata), and those ghost buffers simulated the gst a/v
sync, nevertheless the real a/v sync was done by omx.

That solution is not sound: might work in some cases but we couldn't
assure it for every case.

We dismissed the super sinks since the beginning of the development,
because, as you mentioned, is not a flexible solution.

But we have a trade of: the first solution is not concordant with the
gstreamer philosophy, and the second is not concordant with the omx
philosophy, because a semantic overlapping, as in the buffer
communication assumptions, the state management among the components,
etc.

Nowadays I'm more convinced that a supersink could be the best
solution to integrate gst and omx in the A/V playback use case:

1. it's easy to build and setup the omx pipelines given the caps
2. it's easy to control the sync
3. it's easy add gst interfaces as volume, contrast, et all
4. it's easy to manage the state among the omx components
5. no dirty hacks as ghostbuffers
6. afaik the supersink elements can be autoplugged by playbin2

I admit the supersinks break the flexibility offered by gst, but as
far as i can foresee, is the straight strategy to obtain the
performance promised by omx in his interop profile.

Maybe one day the interop profile won't be necessary when the pBuffer
in the bufferheader leave its readonly property...

Víctor Manuel Jáquez Leal

On Wed, Jul 30, 2008 at 9:00 AM, Ling Shi <shil66 at gmail.com> wrote:
> Hi, all
> I'm in a research project to port gstreamer into embedded system. Now, we
> encounter the issue on how to integrate hardware accelerators (DSP/GPU) into
> gst. After evaluating different solution, we think GstOpenMAX project may be
> the best one for us, because
>
>   1) OpenMAX is an industry standard
>   2) more and more DSP/GPU vendor support OpenMAX
>
> But, I still have several questions on this project.
> 1. Does Nokia N8xx serials use GstOpenMAX?
> I know Nokia engineers lead this project. I also know N8xx serials use
> gstreamer as default playback engine, and it uses TI OMAP 2420, which has
> DSP. Can anyone tell me if N8xx use GstOpenMAX? If no, does N8xx plan to use
> it in the future?
>
> 2. What's GstOpenMAX plan to support DSP/GPU in the future?
> I review several plugins in GstOpenMAX and find current design can only
> support none-tunnel communication.  It's not the best solution in hardware,
> because of bad performance. So, we plan to improve it by adding tunneled or
> proprietary communication. Do you have such plan? If yes, can we involve in
> design?
>
> In addition, most accelerators work as a decoder and a render. It means, the
> encoded data sent to it will directly be decoded and rendered, and will not
> be retrieved back again. How the gst or omx organize its pipeline in this
> situation? We are evaluating two solutions.
>
> ===Solution 1===
> We can design a super omx sink component to cover decoder and render. This
> is the solution is used by N8xx.
> src ! demux ! sink
>                          |
>                        super omx sink
>                          |
>                    +--------------------------------------------+
>                     |    hardware accelerator                 |
>                    +--------------------------------------------+
>
> ===Solution 2===
> We can separate omx decoder, omx post processer, and omx sink elements. We
> enhance decoder, post processor, and sink plugin in GstOpenMAX. If
> GstOpenMAX plugin found its neighborhood are GstOpenMAX plugin, it will try
> to establish tunneled communication or proprietary communication firstly. It
> means, although we have 3 OMX plugin in gst pipeline, there is no data in
> gst pad and omx port. The last two gst/omx plugin only provide control
> function, but not support process data flow. Of cause, If the connection is
> failed, it will use none-tunnel communication.
>
> src ! demux ! decoder ! post processor ! sink
>                             |                     |                  |
>                      omx dec          omx pp       omx sink
>                             |                     |                  |
>                    +--------------------------------------------+
>                     |    hardware accelerator                 |
>                    +--------------------------------------------+
>
> It seems solution 2 is more flexible. How about your suggestion on the 2
> solutions? Which one is feasible? Do you have other solutions?
>
> Thank you very much.
> -------------------------------------------------------------------------
> This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
> Build the coolest Linux based applications with Moblin SDK & win great
> prizes
> Grand prize is a trip for two to an Open Source event anywhere in the world
> http://moblin-contest.org/redirect.php?banner_id=100&url=/
> _______________________________________________
> Gstreamer-openmax mailing list
> Gstreamer-openmax at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-openmax
>
>