omxh264enc: incorrectly advertised sink capabilities?

[Milian Wolff]

Hey there,

I'm trying to optimize a gstreamer pipeline running on a Raspberry Pi 3b+. 
According to gst-shark, a lot of proctime is spend in a videoconvert I added 
to get the pipeline working at all.

The simiplified version looks like this:

```
gst-launch-1.0 gltestsrc ! glcolorconvert ! "video/x-
raw(memory:GLMemory),format=RGB16" ! queue ! gldownload ! omxh264enc control-
rate=1 target-bitrate=1145000 ! video/x-h264,profile=baseline ! h264parse ! 
avimux ! filesink location=test.avi
```

I.e. I have some part of the pipeline that lives in OpenGL land and thus 
operates on RGBA frames. Then I wanted to convert that via glcolorconvert to a 
format that can be understood by omxh264enc before downloading, but this does 
not work without a videoconvert before the `omxh264enc`, despite the fact that 
omxh264enc advertises the following:

```
Factory Details:
  Rank                     primary + 1 (257)
  Long-name                OpenMAX H.264 Video Encoder
  Klass                    Codec/Encoder/Video/Hardware
  Description              Encode H.264 video streams
  Author                   Sebastian Dröge <sebastian.droege at collabora.co.uk>

Plugin Details:
  Name                     omx
  Description              GStreamer OpenMAX Plug-ins
  Filename                 /usr/lib/gstreamer-1.0/libgstomx.so
  Version                  1.16.1
  License                  LGPL
  Source module            gst-omx
  Source release date      2019-09-23
  Binary package           GStreamer OpenMAX Plug-ins source release
  Origin URL               Unknown package origin
...

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      video/x-raw
                 format: { (string)NV16_10LE32, (string)NV12_10LE32, 
(string)NV16, (string)YUY2, (string)YVYU, (string)UYVY, (string)N
V12, (string)I420, (string)RGB16, (string)BGR16, (string)ABGR, (string)ARGB, 
(string)BGR, (string)GRAY8 }
                  width: [ 1, 2147483647 ]
                 height: [ 1, 2147483647 ]
              framerate: [ 0/1, 2147483647/1 ]
```

The error I get, for all RGB-like formats (BGR16, ABGR, ABGR, ARGB, BGR) is:

```
0:00:00.216186510  1356  0x1454e60 WARN                 basesrc gstbasesrc.c:
3072:gst_base_src_loop:<gltestsrc0> error: Internal data stream error.
0:00:00.216280364  1356  0x1454e60 WARN                 basesrc gstbasesrc.c:
3072:gst_base_src_loop:<gltestsrc0> error: streaming stopped, reason not-
negotiated (-4)
ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal 
data stream error.
```

Looking at the source for gst-omx I wonder why these formats are included in 
GST_OMX_VIDEO_SUPPORTED_FORMATS? They do not seem to be supported internally 
at all by omxh264enc, if I'm not mistaken? Are those maybe formats supported 
by omxh264dec, i.e. the decoder, but not by the encoder I'm trying to use?

Generally, does someone know if the hardware encoder would in theory support 
being fed with such frames? Is this complicated to implement for someone with 
zero experience in writing gst plugins and OMX? or would it be somewhat 
straight forward? Could someone maybe give me some hints as to what one should 
do roughly - then I would try to implement this and upstream that code.

My hunch is that this would greatly improve the performance of my specific 
pipeline, as I could get rid of the on-CPU videoconvert and leverage the 
glcolorconvert instead.

Cheers
-- 
Milian Wolff | milian.wolff at kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3826 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20200406/a43eedfb/attachment-0001.bin>