omxh264enc: incorrectly advertised sink capabilities?
Milian Wolff
milian.wolff at kdab.com
Mon Apr 6 13:32:54 UTC 2020
On Montag, 6. April 2020 11:44:17 CEST Milian Wolff wrote:
> Hey there,
>
> I'm trying to optimize a gstreamer pipeline running on a Raspberry Pi 3b+.
> According to gst-shark, a lot of proctime is spend in a videoconvert I added
> to get the pipeline working at all.
>
> The simiplified version looks like this:
>
> ```
> gst-launch-1.0 gltestsrc ! glcolorconvert ! "video/x-
> raw(memory:GLMemory),format=RGB16" ! queue ! gldownload ! omxh264enc
> control- rate=1 target-bitrate=1145000 ! video/x-h264,profile=baseline !
> h264parse ! avimux ! filesink location=test.avi
> ```
>
> I.e. I have some part of the pipeline that lives in OpenGL land and thus
> operates on RGBA frames. Then I wanted to convert that via glcolorconvert to
> a format that can be understood by omxh264enc before downloading, but this
> does not work without a videoconvert before the `omxh264enc`, despite the
> fact that omxh264enc advertises the following:
>
> ```
> Factory Details:
> Rank primary + 1 (257)
> Long-name OpenMAX H.264 Video Encoder
> Klass Codec/Encoder/Video/Hardware
> Description Encode H.264 video streams
> Author Sebastian Dröge
> <sebastian.droege at collabora.co.uk>
>
> Plugin Details:
> Name omx
> Description GStreamer OpenMAX Plug-ins
> Filename /usr/lib/gstreamer-1.0/libgstomx.so
> Version 1.16.1
> License LGPL
> Source module gst-omx
> Source release date 2019-09-23
> Binary package GStreamer OpenMAX Plug-ins source release
> Origin URL Unknown package origin
> ...
>
> Pad Templates:
> SINK template: 'sink'
> Availability: Always
> Capabilities:
> video/x-raw
> format: { (string)NV16_10LE32, (string)NV12_10LE32,
> (string)NV16, (string)YUY2, (string)YVYU, (string)UYVY, (string)N
> V12, (string)I420, (string)RGB16, (string)BGR16, (string)ABGR, (string)ARGB,
> (string)BGR, (string)GRAY8 }
> width: [ 1, 2147483647 ]
> height: [ 1, 2147483647 ]
> framerate: [ 0/1, 2147483647/1 ]
> ```
>
> The error I get, for all RGB-like formats (BGR16, ABGR, ABGR, ARGB, BGR) is:
>
> ```
> 0:00:00.216186510 1356 0x1454e60 WARN basesrc
> gstbasesrc.c: 3072:gst_base_src_loop:<gltestsrc0> error: Internal data
> stream error. 0:00:00.216280364 1356 0x1454e60 WARN
> basesrc gstbasesrc.c: 3072:gst_base_src_loop:<gltestsrc0> error: streaming
> stopped, reason not- negotiated (-4)
> ERROR: from element /GstPipeline:pipeline0/GstGLTestSrc:gltestsrc0: Internal
> data stream error.
> ```
>
> Looking at the source for gst-omx I wonder why these formats are included in
> GST_OMX_VIDEO_SUPPORTED_FORMATS? They do not seem to be supported
> internally at all by omxh264enc, if I'm not mistaken? Are those maybe
> formats supported by omxh264dec, i.e. the decoder, but not by the encoder
> I'm trying to use?
>
> Generally, does someone know if the hardware encoder would in theory support
> being fed with such frames? Is this complicated to implement for someone
> with zero experience in writing gst plugins and OMX? or would it be
> somewhat straight forward? Could someone maybe give me some hints as to
> what one should do roughly - then I would try to implement this and
> upstream that code.
>
> My hunch is that this would greatly improve the performance of my specific
> pipeline, as I could get rid of the on-CPU videoconvert and leverage the
> glcolorconvert instead.
My hunch seems to be correct - the following patch applied to `gst-omx` in tag
1.16.1 improves the performance significantly, but the video's colors are all
wrong. I.e. red turns orange, green turns violet and blue turns yellow... So
clearly there's more to be done but I have no clue what would be needed to get
there... Does anyone have a suggestion what to try?
```
diff --git a/omx/gstomxvideoenc.c b/omx/gstomxvideoenc.c
index dfc03c5..c1a5b71 100644
--- a/omx/gstomxvideoenc.c
+++ b/omx/gstomxvideoenc.c
@@ -1881,6 +1881,11 @@ gst_omx_video_enc_configure_input_buffer
(GstOMXVideoEnc * self,
((port_def.format.video.nFrameHeight + 1) / 2));
break;
+ case OMX_COLOR_Format32bitBGRA8888:
+ case OMX_COLOR_Format32bitARGB8888:
+ port_def.nBufferSize = 4 * (port_def.format.video.nStride *
port_def.format.video.nFrameHeight);
+ break;
+
default:
GST_ERROR_OBJECT (self, "Unsupported port format %x",
port_def.format.video.eColorFormat);
@@ -3211,8 +3216,8 @@ filter_supported_formats (GList * negotiation_map)
case GST_VIDEO_FORMAT_NV12_10LE32:
case GST_VIDEO_FORMAT_NV16:
case GST_VIDEO_FORMAT_NV16_10LE32:
- //case GST_VIDEO_FORMAT_ABGR:
- //case GST_VIDEO_FORMAT_ARGB:
+ case GST_VIDEO_FORMAT_ABGR:
+ case GST_VIDEO_FORMAT_ARGB:
cur = g_list_next (cur);
continue;
default:
```
Additionally, I would really appreciate if someone could explain the following
to me: When I try to feed `omxh264enc` after applying the above patch with
`ARGB` input I get:
```
0:00:02.013120260 2588 0x1fc3e60 DEBUG GST_CAPS gstutils.c:
3111:gst_pad_peer_query_caps:<omxh264enc-omxh264enc0:src> peer query returned
video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ],
framerate=(fraction)[ 0/1, 2147483647/1 ], profile=(string)baseline, stream-
format=(string)byte-stream, alignment=(string)au; video/x-h264, width=(int)[
16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1
], profile=(string)baseline
0:00:02.013263854 2588 0x1fc3e60 LOG default gstvideoutilsprivate.c:
128:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> template caps
video/x-raw, format=(string)I420; video/x-raw, format=(string)NV12; video/x-
raw, format=(string)ABGR
0:00:02.013389583 2588 0x1fc3e60 LOG default gstvideoutilsprivate.c:
129:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> allowed caps
video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ],
framerate=(fraction)[ 0/1, 2147483647/1 ], profile=(string)baseline, stream-
format=(string)byte-stream, alignment=(string)au; video/x-h264, width=(int)[
16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1
], profile=(string)baseline
0:00:02.013797604 2588 0x1fc3e60 LOG default gstvideoutilsprivate.c:
138:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> intersecting
with video/x-raw(memory:GLMemory), format=(string)ARGB, width=(int)[ 1,
2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1,
2147483647/1 ], texture-target=(string){ 2D, rectangle, external-oes }; video/
x-raw(memory:DMABuf), format=(string)ARGB, width=(int)[ 1, 2147483647 ],
height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ];
video/x-raw, format=(string)ARGB, width=(int)[ 1, 2147483647 ], height=(int)[
1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:02.013934218 2588 0x1fc3e60 LOG default gstvideoutilsprivate.c:
147:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> proxy caps
EMPTY
0:00:02.014005156 2588 0x1fc3e60 LOG omxvideoenc
gstomxvideoenc.c:3258:gst_omx_video_enc_getcaps:<omxh264enc-omxh264enc0>
Supported caps EMPTY
```
But for `ABGR` it works like the following - why? Shouldn't the code handle
both now?
```
0:00:02.026753801 2733 0xf85c60 DEBUG GST_CAPS gstutils.c:
3111:gst_pad_peer_query_caps:<omxh264enc-omxh264enc0:src> peer query returned
video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ],
framerate=(fraction)[ 0/1, 2147483647/1 ], profile=(string)baseline, stream-
format=(string)byte-stream, alignment=(string)au; video/x-h264, width=(int)[
16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1
], profile=(string)baseline
0:00:02.026899114 2733 0xf85c60 LOG default gstvideoutilsprivate.c:
128:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> template caps
video/x-raw, format=(string)I420; video/x-raw, format=(string)NV12; video/x-
raw, format=(string)ABGR
0:00:02.027026353 2733 0xf85c60 LOG default gstvideoutilsprivate.c:
129:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> allowed caps
video/x-h264, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ],
framerate=(fraction)[ 0/1, 2147483647/1 ], profile=(string)baseline, stream-
format=(string)byte-stream, alignment=(string)au; video/x-h264, width=(int)[
16, 4096 ], height=(int)[ 16, 4096 ], framerate=(fraction)[ 0/1, 2147483647/1
], profile=(string)baseline
0:00:02.027436614 2733 0xf85c60 LOG default gstvideoutilsprivate.c:
138:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> intersecting
with video/x-raw(memory:GLMemory), format=(string)ABGR, width=(int)[ 1,
2147483647 ], height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1,
2147483647/1 ], texture-target=(string){ 2D, rectangle, external-oes }; video/
x-raw(memory:DMABuf), format=(string)ABGR, width=(int)[ 1, 2147483647 ],
height=(int)[ 1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ];
video/x-raw, format=(string)ABGR, width=(int)[ 1, 2147483647 ], height=(int)[
1, 2147483647 ], framerate=(fraction)[ 0/1, 2147483647/1 ]
0:00:02.027611770 2733 0xf85c60 LOG default gstvideoutilsprivate.c:
147:__gst_video_element_proxy_getcaps:<omxh264enc-omxh264enc0> proxy caps
video/x-raw, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ],
framerate=(fraction)[ 0/1, 2147483647/1 ], format=(string)ABGR
0:00:02.027717499 2733 0xf85c60 LOG omxvideoenc
gstomxvideoenc.c:3258:gst_omx_video_enc_getcaps:<omxh264enc-omxh264enc0>
Supported caps video/x-raw, width=(int)[ 16, 4096 ], height=(int)[ 16, 4096 ],
framerate=(fraction)[ 0/1, 2147483647/1 ], format=(string)ABGR
```
--
Milian Wolff | milian.wolff at kdab.com | Senior Software Engineer
KDAB (Deutschland) GmbH, a KDAB Group company
Tel: +49-30-521325470
KDAB - The Qt, C++ and OpenGL Experts
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3826 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20200406/fecfa5ef/attachment.bin>
More information about the gstreamer-devel
mailing list