Issue with decoding H264 frames using appsrc in GStreamer
Michael Gruner
michael.gruner at ridgerun.com
Mon Jul 10 20:37:53 UTC 2023
You should be able to use the decoder, yes. Try placing a h264parse right after the appsrc. That will read the H264 bitstream, parse it, accumulate it to form full NAL units and perform stream format conversions if they are needed by the decoder.
-Michael
> On 10 Jul 2023, at 09:04, Israel Díaz <isra601988 at gmail.com> wrote:
>
> Ohh. Thank you but it seems I don't have access to gst_buffer_new_memdup since I am using 1.18
>
> The other option to not copy I guess is not suitable for me because encoded_frame will be freed by another component (the encoded data is part of a bigger object that is free when the object call its destructor)
>
> So finally I opted for this
>
> buffer = gst_buffer_new_allocate (NULL, frame_size, NULL);
> gst_buffer_map (buffer, &map, GST_MAP_WRITE);
> memcpy((guchar *)map.data, encoded_frame, gst_buffer_get_size( buffer ) );
> gst_buffer_unmap(buffer, &map);
>
>
> Now I don't see those broken/invalid nal Type: 0 messages but currently I still don't see that the raw file is receiving bytes.
> Do you think it is possible to use the decoder with this configuration? Maybe I need to follow the approach of need-data to feed the decoder or How Can I check if the decoder is actually processing everything?
>
> On Fri, Jul 7, 2023 at 4:50 PM Michael Gruner <michael.gruner at ridgerun.com <mailto:michael.gruner at ridgerun.com>> wrote:
>> Hi Israel
>>
>> This assignment is invalid:
>>
>> gst_buffer_map(buffer, &map, GST_MAP_WRITE);
>> map.data = encoded_frame;
>> gst_buffer_unmap(buffer, &map);
>>
>> You need to copy the data using a plain memcpy or
>> by creating the new buffer directly as a copy using gst_buffer_new_memdup.
>>
>> If you would like to avoid the copy, then you can use gst_buffer_new_wrapped or gst_buffer_new_wrapped_full but you need to be careful that: 1. The pointer remains valid the while the buffer is alive and 2. The pointer is freed when the buffer is unreffed.
>>
>> Michael
>>
>>> On 7 Jul 2023, at 04:02, Israel Díaz via gstreamer-devel <gstreamer-devel at lists.freedesktop.org <mailto:gstreamer-devel at lists.freedesktop.org>> wrote:
>>>
>>>
>>> Dear GStreamer community,
>>>
>>> I am currently facing an issue while attempting to decode H264 frames using the appsrc element in GStreamer. I have constructed the following pipeline within my application:
>>>
>>> appsrc name=mysource ! video/x-h264, alignment=au, stream-format=byte-stream ! h264parse ! video/x-h264, alignment=au, stream-format=byte-stream ! omxh264dec ! video/x-raw, format=NV16_10LE32, width=1920, height=1080, framerate=30/1 ! filesink location=ball.raw
>>>
>>> To feed the appsrc with frames, I am reading an H264 file frame by frame since I store each frame in a structure along with its size. Then, I push the frames into the appsrc using the following function:
>>>
>>> void VDRE2GStreamerDecodingDisplaySinkTask::PushBufferToGStreamer(uint8_t* encoded_frame, uint32_t frame_size)
>>> {
>>> GstFlowReturn ret;
>>> GstMapInfo map;
>>> GstBuffer *buffer = gst_buffer_new_and_alloc(frame_size);
>>> gst_buffer_map(buffer, &map, GST_MAP_WRITE);
>>> map.data = encoded_frame;
>>> gst_buffer_unmap(buffer, &map);
>>>
>>> /* Push the buffer into the appsrc */
>>> g_signal_emit_by_name(m_appsrc, "push-buffer", buffer, &ret);
>>>
>>> /* Free the buffer now that we are done with it */
>>> gst_buffer_unref(buffer);
>>>
>>> if (ret != GST_FLOW_OK) {
>>> g_print("PushBufferToGStreamer: Error!\n");
>>> }
>>> }
>>>
>>> However, when I execute the pipeline and set GST_DEBUG=4, I encounter the following error messages:
>>>
>>> Decode Thread: Frame to Process : 1 :s 246
>>> Decode Thread: Frame to Process : 2 :s 232
>>> Decode Thread: Frame to Process : 3 :s 854
>>> Decode Thread: Frame to Process : 4 :s 952
>>> Decode Thread: Frame to Process : 5 :s 350
>>> Decode Thread: Frame to Process : 6 :s 1038
>>> Decode Thread: Frame to Process : 7 :s 1447
>>> Decode Thread: Frame to Process : 8 :s 1316
>>> Decode Thread: Frame to Process : 9 :s 1297
>>> Decode Thread: Frame to Process : 10 :s 1205
>>> 0:00:02.370465675 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 29 will be dropped
>>> 0:00:02.370549896 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 1140 will be dropped
>>> Decode Thread: Frame to Process : 11 :s 1222
>>> Decode Thread: Frame to Process : 12 :s 1189
>>> Decode Thread: Frame to Process : 13 :s 1094
>>> Decode Thread: Frame to Process : 14 :s 930
>>> Decode Thread: Frame to Process : 15 :s 1000
>>> Decode Thread: Frame to Process : 16 :s 888
>>> Decode Thread: Frame to Process : 17 :s 901
>>> Decode Thread: Frame to Process : 18 :s 788
>>> Decode Thread: Frame to Process : 19 :s 833
>>> Decode Thread: Frame to Process : 20 :s 764
>>> Decode Thread: Frame to Process : 21 :s 727
>>> Decode Thread: Frame to Process : 22 :s 616
>>> Decode Thread: Frame to Process : 23 :s 728
>>> Decode Thread: Frame to Process : 24 :s 622
>>> Decode Thread: Frame to Process : 25 :s 609
>>> Decode Thread: Frame to Process : 26 :s 657
>>> Decode Thread: Frame to Process : 27 :s 651
>>> Decode Thread: Frame to Process : 28 :s 621
>>> Decode Thread: Frame to Process : 29 :s 534
>>> Decode Thread: Frame to Process : 30 :s 2989
>>> 0:00:03.033182378 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 29 will be dropped
>>> 0:00:03.033258709 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 117 will be dropped
>>> 0:00:03.033309439 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 29 will be dropped
>>> 0:00:03.033369800 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 1468 will be dropped
>>> Decode Thread: Frame to Process : 31 :s 411
>>> Decode Thread: Frame to Process : 32 :s 465
>>> Decode Thread: Frame to Process : 33 :s 514
>>> Decode Thread: Frame to Process : 34 :s 520
>>> Decode Thread: Frame to Process : 35 :s 519
>>> Decode Thread: Frame to Process : 36 :s 476
>>> Decode Thread: Frame to Process : 37 :s 448
>>> Decode Thread: Frame to Process : 38 :s 544
>>> Decode Thread: Frame to Process : 39 :s 523
>>> Decode Thread: Frame to Process : 40 :s 454
>>> Decode Thread: Frame to Process : 41 :s 561
>>> Decode Thread: Frame to Process : 42 :s 409
>>> Decode Thread: Frame to Process : 43 :s 506
>>> Decode Thread: Frame to Process : 44 :s 330
>>> Decode Thread: Frame to Process : 45 :s 2180
>>> 0:00:03.530095934 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 29 will be dropped
>>> 0:00:03.530192075 6548 0xffff8c01ff60 WARN h264parse gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid nal Type: 0 Unknown, Size: 117 will be dropped
>>> 0:00:03.530251666 654
>>>
>>>
>>> So then I analyzed the encoded_data to check if there are valid NAL units:
>>>
>>> have analyzed the encoded_frame data to determine if it contains valid NAL units. Here's a summary of your findings for the first five data sets:
>>>
>>> Encoded data values in hexadecimal: 0, size: 3241
>>> 00 00 00 01 27 7A 00 28 B6 D0 0C E8 07 80 22 7E 27 01 6A 02 04 04 80 00 00 03 00 80 00 00 1E 19 A8 00 3D 09 00 01 6E 37 FF FE 05 00 00 00 01 28 EE 3C
>>>
>>> Encoded data values in hexadecimal: 1, size: 246
>>> 00 00 00 01 21 E2 01 08 42 DF 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 09 36 B5 84 05 B0 C9 8C DA C9 6F 9D F0 00
>>>
>>> Encoded data values in hexadecimal: 2, size: 232
>>> 00 00 00 01 21 E4 02 08 42 DF 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 16 D0 00 00 01 21 00 1F E3 90 08 21
>>>
>>> Encoded data values in hexadecimal: 3, size: 854
>>> 00 00 00 01 21 E6 03 08 8B 7F 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 06 2C 00 00 01 21 00 1F E3 98 0C 22
>>>
>>> Encoded data values in hexadecimal: 4, size: 952
>>> 00 00 00 01 21 E8 04 0E DF 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03 00 00 97 80 00 00 01 21 00 1F E3 A0 10 3B
>>>
>>> It seems that the analyzed data contains valid NAL units, as expected. I have attempted a similar test using a file source and launching the pipeline with gst-launch as follows:
>>>
>>> gst-launch-1.0 -e filesrc location=ball2.h264 ! video/x-h264, alignment=au, width=1920, height=1080, framerate=30/1 ! h264parse ! omxh264dec ! video/x-raw, format=NV16_10LE32, width=1920, height=1080, framerate=30/1 ! filesink location=ball_gst.raw
>>>
>>> During this test, I did not encounter any issues or error messages with GST_DEBUG=4. Additionally, the raw file was populated with bytes, unlike the appsrc version where the content always appears empty (presumably because it is not being decoded at any point).
>>>
>>> I even tried with a identity dump=true element between filesrc and h264parse and I also have the same bytes I am injecting to appsrc So I don't know what can be the difference
>>>
>>> Thank you in advance.
>>> Best regards.
>>> Israel.
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20230710/a686fd9b/attachment-0001.htm>
More information about the gstreamer-devel
mailing list