architectural question (Android): gstreamer / openmax / libfslvpuwrap / imxvpuapi

[Carlos Rafael Giani]

The suggestion with disabling frame reordering makes sense. As the VPU 
documentation states, many h.264 videos don't actually use B-frames, yet 
enable frame reordering. With frame reordering, there is zero 
possibility for low-latency decoding.

imxvpuapi should in theory work on Android. I never tested it there 
though. But, you could also try gstreamer-imx. With these plugins, you 
can decode with the VPU without OpenMAX. An additional option is to 
restrict h.264 video streams to the baseline profile, which does not 
contain video streams (this is the profile that is often used for 
real-time video feed encoding).

On 2014-10-06 21:10, Stephen Webb wrote:
> Sebastien,
>
> Thank you very much for the reply.  It was helpful, but of course it 
> raises more questions. :)
>
> Maybe it would be more clear if I explain what I'm trying to do and 
> someone could point out problems / make suggestions.
>
> 1.  I'm using Android on a Compulab Utilite iMX6 platform.
> 2.  I'm currently using Gstreamer as a general way to decode video 
> files -  I want to retain this.
> 3.  I am adding support for hardware decode of H264 stream - *low 
> latency is important.*
> 4.  When I use the iMX6 hardware decode path in Gstreamer I get 8+ 
> frames of latency:
>      decoder  = gst_element_factory_make 
> ("amcviddec-omxfreescalestdvideodecoderavcv3hwbased",  "decoder");
> 5.  I googled for other people with similar issues and came across 
> suggestions like this: (effectively "set nReorderEnable=false if your 
> stream doesn't use B frames"):
> https://community.freescale.com/thread/304322
>
> So I went down the path of changing vpudec.c to change 
> nReorderEnable=false etc...*I think this may be where my mistake was 
> as I'm realizing that this may not work on Android*.  Similarly I 
> found the imxvpuapi library as a way to achieve the same thing, but 
> again I'm now thinking this isn't for Android.
>
> Basically I want to retain gstreamer for my general video file 
> playback needs and I also want to come up with a way to decode a live 
> H264 stream with low latency.  It is OK for the low latency decode to 
> be "tied" to the iMX6 platform. *I now think that using OpenMax IL 
> directly (with the caveat you mentioned) is probably my best option*.
>
> Does that seem right?
>
> Steve
>
> On Mon, Oct 6, 2014 at 10:55 AM, Sebastian Dröge 
> <sebastian at centricular.com <mailto:sebastian at centricular.com>> wrote:
>
>     On Mo, 2014-10-06 at 10:47 -0600, Stephen Webb wrote:
>     > I'm trying to understand how these things fit together / relate
>     to each
>     > other.  I'm afraid that I may not know enough to even ask
>     intelligent
>     > questions at this point, so if someone can point me to
>     documentation or
>     > other resources tat would help, that would be great.
>     >
>     > My current questions:
>     >
>     > 1.  How do gstreamer and openmax relate to each other?  Is it
>     best to think
>     > of gstreamer as a replacement/alternative to Openmax?  A
>     replacement for
>     > the OpenMax AL?  They both seem to do a lot of the same things.
>
>     GStreamer is a bit like OpenMAX AL and OpenMAX IL together, but has an
>     even broader scope than those too.
>
>     GStreamer can use OpenMAX IL via gst-omx (so you can use e.g. hardware
>     codecs that have an OpenMAX IL API in GStreamer pipelines). In theory
>     you could also implement the OpenMAX IL and AL API on top of
>     GStreamer.
>
>     > 2.  I was under the impression that OpenMax was the standard way
>     to get
>     > multimedia support on Android - as in it is required in some way
>     when doing
>     > multimedia work on Android (specifically when any low level
>     hardware access
>     > comes into play).  Having dug in and looked around a fair
>     amount, I don't
>     > see much mention of OpenMax - it doesn't seem so central at all.
>
>     OpenMAX IL is not public API on Android but many devices use it
>     (but you
>     shouldn't use it directly unless you target exactly a specific
>     device),
>     OpenMAX AL is public API on Android but last time I looked only
>     supported MPEG TS.
>
>     The public API for accessing codecs on Android is the Java
>     android.media.MediaCodec API, which is supported by the GStreamer
>     androidmedia plugin.
>
>     > 2.  I found the imxvpuapi project.  If this works on Android
>     this seems
>     > like exactly what I need.
>     > a.  Does it work on Android?
>     > b.  With the only dependency being libfslvpuwrap / vpu_wrapper.h
>     this
>     > appears to interface directly with the freescale vpu driver (no
>     OMX, no
>     > gstreamer).  Is this right?  Does this work on Android?  If so, am I
>     > confused about how Android locks things down and doesn't allow
>     applications
>     > to get low level hardware access?
>
>     I don't know any answers for this one, but you're constraining
>     yourself
>     to iMX based devices then in any case :)
>
>     --
>     Sebastian Dröge, Centricular Ltd - http://www.centricular.com
>     Expertise, Straight from the Source
>
>     _______________________________________________
>     gstreamer-devel mailing list
>     gstreamer-devel at lists.freedesktop.org
>     <mailto:gstreamer-devel at lists.freedesktop.org>
>     http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
>
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20141008/ca66c1ef/attachment.html>