[Gstreamer-openmax] Discussion on the hardware accelerator solution in GstOpenMAX project.

[Victor Manuel Jáquez Leal]

Hi all,

>> I haven't read the NXP tunneling implementation in gstopenmax but once
>> we implemented something related: when the omx gst element is linked
>> with another omx gst element a tunnel is setup, but as no buffers
>> traverse the pipeline, because the buffer communication is done
>> beneath the omx layer, we had to push ghost buffer (empty buffers with
>> calculated metadata), and those ghost buffers simulated the gst a/v
>> sync, nevertheless the real a/v sync was done by omx.
>
> If the A/V sync was done in the omx layer then why are the ghost buffers needed?

Last Friday I glanced the tunneling-v3  branch in github and I could
not grasp some details which I'm worry about:

1) AFAIK: in order that a pipeline could change from prerrolling to
playing, the sink must receive at least one buffer. If you have an OMX
sink in tunnel with a previous element the pipeline will never leave
the prerolling state.

2) When the first gst buffer traverse its part of the pipeline the
stream negotiation is done among the linked elements. So, if no
buffers traverse some portion of the pipeline because they are
tunneled, those elements in the tunnel will never inform about their
real configured caps to the GStreamer client application.

> The gst base sink requires buffers in order to do A/V sync. If the
> sink doesn't receive the buffers, then it doesn't do the sync, but it
> still works.

But, as I said, the pipeline won't leave the prerolling state in the
case of a tunneled omx sink.

>> 1. it's easy to build and setup the omx pipelines given the caps
>
> What if you want a post-processing element in the middle? Or you want
> an encoder+decoder (transcoder)?

Yes, the supersink solutions are not flexible, but they might provide
an effective solution to a common use case.

> I don't think all the omx pipelines can be built based on the caps.

Not at all, but if you have a fixed hardware (as happens in the
embedded world) you'll only need the input stream caps to build up the
OMX pipeline to render the stream.

Anyway, this is not a real argument, it's just workaround a problem in
the supersink concept.

>> 4. it's easy to manage the state among the omx components
>
> I'm not so sure about that. If a gst element is mapped to a single omx
> component it's easier to see what's happening.

That's another issue which I couldn't figure out in the tunneling-v3 branch.

According to the spec (Page 122, Figure 3-10. State Transition to Idle
in the Case of Tunneled Components) if you change to idle state to a
component which is not a "buffer supplier" the CommandStateSet
callback won't be triggered until the other component in the tunnel
have changed to idle also.

AFAIK, each change state in gstopenmax is done sequentially:
change_state, wait_for_state.  And it could cause problems when the
component which is the buffer supplier in the tunnel is the posterior
in the tunneled: you'll get a dead-lock waiting for the first
component to change its state. And that could be the case, in example,
of tunneled videosink.

In the case of a supersink that situation is easily to overcome.

> Actually I think that was the difficult part in the tunneling branch:
> align gst and omx components.

Yes, it is.

>> Maybe one day the interop profile won't be necessary when the pBuffer
>> in the bufferheader leave its readonly property...
>
> Even in that case tunneling might help; there would be less memory
> allocations for gst buffers.

Yes that might be true. And also could be the situation where the
tunneled components never use general purpose memory ;)

> I think you are basing your ideas on the fact that in order to support
> A/V sync properly gst should do it by receiving real buffers on the
> element. Even if you have a video decoder sink that receives real gst
> buffers in order to do A/V sync, the sync will be done _before_ the
> decoding, so by the time the buffers reach the renderer some time
> would have been spent, and the sync would be lost.
>
> In discussions with different parties I believe the consensus has been
> that mapping the omx clock to a gst clock is the right way to go. In
> this way you have all the flexibility of gst pipelines, omx
> efficiency, and you have proper A/V sync.

You're right. When I tried to find a way to map those clocks I found
some creepy problems in the OMX clock implementation so I dropped it,
but yes I remember it was possibly... in theory at least.

Nevertheless, in the supersink solution, the mapping won't be
necessary, because the supersink will only attend the (not-exposed)
OMX clock :)

> I'm a pragmaticist, so I'm not saying that approach would work, but I
> don't see any way it shouldn't, so I think we should better try it
> first.

I'm agree.

Cheers!

vmjl