BGRA for omxh264enc and omxh264dec

Mon Mar 14 04:39:25 UTC 2016

Hi Sebastian.

Looking into the to the problem of converting and/or resize I420 or NV12
output from the omxh264dec (and others), there seem to be ways to both
convert to other raw formats as well as perhaps resize the output all using
the GPU.

The OpenMAX Il-component *video_splitter *(input port 250, output ports
251-254 - only one is needed) can take input as I420 and output as
OMX_COLOR_Format32bitARGB8888 or OMX_COLOR_Format32bitBGRA8888 (one of
which I believe is called a byte stream of R, G, B and A bytes (RGBA)) and
possibly OMX_COLOR_Format24bitRGB888 or OMX_COLOR_Format24bitBGR888 (one of
which is believed to be RGB). I'm not sure it can do both RGB(A) and BGR(A).

Anyway, it should be possible to connect the encoder's port 131 to the
video_splitter's port 250, set the splitter port 250 format to I420,
perhaps provide buffers for this, set the splitter port 251 to RGBA or RGB
and feed it data buffers from downstream (perhaps shm) for zero-copying. An
of course the video_splitter component has to be started and prepared for
state chage etc.

For video resizing (and conversion) the OpenMAX image component *resize*
can take I420 input on port 60 AND scale AND convert to image formats such
as RGBA and RGB (I am 95% sure) and output these to output port 61 on data
buffers from downstreams (perhaps shm).

All this of course applies to the other omxXXXdec modules.

The same should be possible for the input for omxh264enc and the other
encoders in a way so the encoders can take I420/RGB/RGBA at any size and
the decoders can output I420/RGB/RGBA at any size.

The "any size" may be to take it to far. There are some restrictions for
either the encoder and the decoder. For one of them, the width and height
must both be even numbers and for the other, both must be a multipla of 16.
I just don't know if you can link OpenMAX video modules to OpenMAX image
modules. At least some timing may be set and controlled out-of-band when
going over an image module .... perhaps.

I tried looking to the GStreamer code, but I can't really see where to add
the functionality and I am still struggling with the very poorly documented
OpenMAX examples and docs.

I hope this is worth looking at. Especailly converting from I420 to RGBA
for is nearly impossible on the Pi-platform for 720p and quite impossible
for 1080p with the current gstreamer version, conversion alone take a lot
of CPU not really present.

Does it help to open a ticket for an improvement?

Best regards
Peter Maersk-Moller

On Tue, Feb 23, 2016 at 3:18 PM, Sebastian Dröge <sebastian at centricular.com>
wrote:

> On Di, 2016-02-23 at 13:43 +0100, Peter Maersk-Moller wrote:
> >
> > > I420 is already supported, no?
> > My mistake. So far it seems I420 is the only one that works.
> > Testing the following pipeline:
> >
> > gst-launch-1.0 -v videotestsrc is-live=1 ! 'video/x-
> > raw,format='$format ! omxh264enc ! fakesink
> >
> > with these formats :I420 YV12 YUY2 UYVY AYUV RGBx BGRx xRGB xBGR RGBA
> > BGRA ARGB ABGR RGB BGR Y41B Y42B YVYU Y444 v210 v216 NV12 NV21 NV16
> > NV61 NV24 GRAY8 GRAY16_BE GRAY16_LE v308 RGB16 BGR16 RGB15 BGR15 UYVP
> > A420 RGB8P YUV9 YVU9 IYU1 ARGB64 AYUV64 r210 I420_10LE I420_10BE
> > I422_10LE I422_10BE Y444_10LE Y444_10BE GBR GBR_10LE GBR_10BE
> > NV12_64Z32 A420_10LE A420_10BE A422_10LE A422_10BE A444_10LE
> > A444_10BE
> >
> > Then only format that works is I420.
> >
> > Shouldn't 'gst-inspect-1.0 omxh264enc' report that the only format it
> > takes is I420?
>
> As said in the previous mail, it support I420 *and* NV12 currently.
> It's just that on the RPi the encoder only supports I420 and nothing
> else.
>
> So what it should do is to report I420 and NV12 in general, and in the
> RPi specific gstomx.conf you would override the pad template to only
> list I420.
>
> > > For the other formats, adding support for them would definitely be
> > > a
> > > good idea. Can you file a bug for this, ideally with a patch?
> > I can. Is the above description enough for the bug report?
> >
> > For the patch, can you point to the code where you detect hardware
> > capabilities before reporting it for the sink pad of omxh264enc ?
> > Is the code in omxh264enc prepared for reporting multiple sink
> > capabilities, or just one. Not sure I know how to extend the code to
> > report multiple capabilities. When done, I can try to look into how
> > to make the hardware encoder accept other formats than I420 as it
> > should be able to. Maybe it needs some of these *broadcom tunnels*
> > for conversion that some of the encoding examples for BCM2835/2836
> > shows.
>
> There is no hardware capability probing API for this (or it's not
> implemented anywhere in a meaningful way). The code I showed you is
> where the format specific handling is, the other changes would be for
> the pad templates which would just be hardcoded... in code with a
> default and in the gstomx.conf for the RPi.
>
> --
> Sebastian Dröge, Centricular Ltd · http://www.centricular.com
>
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20160314/d4723ffc/attachment.html>