Issue with decoding H264 frames using appsrc in GStreamer
Israel Díaz
isra601988 at gmail.com
Tue Jul 11 12:33:20 UTC 2023
That was the thing!! removing the caps specification!!
If anyone is wondering how to do this pipeline. This is the final code to
create it
(Finally I add a fakesink to avoid the generation of big raw files)
m_pipeline = gst_pipeline_new("mypipeline");
m_appsrc = gst_element_factory_make("appsrc", "mysource");
GstElement *h264_parser = gst_element_factory_make("h264parse",
"myparser");
GstElement *decoder = gst_element_factory_make("omxh264dec", "mydec");
GstElement *fakesink = gst_element_factory_make("fakesink", "mysink");
/* g_object_set(G_OBJECT (filesink), "location", "ball_inside.raw",
NULL); */
// Check if all elements were created
if (!m_pipeline && m_appsrc && h264_parser && decoder && fakesink)
{
g_print("Could not gst_element_factory_make, terminating\n");
return;
}
// Create gstreamer loop
loop = g_main_loop_new(NULL, FALSE);
// add a message handler
bus = gst_pipeline_get_bus(GST_PIPELINE(m_pipeline));
bus_watch_id = gst_bus_add_watch(bus, bus_call, loop);
gst_object_unref(bus);
// add all elements into pipeline
gst_bin_add_many(GST_BIN(m_pipeline), m_appsrc, h264_parser, decoder,
fakesink, NULL);
// link elements into pipe: appsrc -> jpegenc -> appsink
gst_element_link(m_appsrc, h264_parser);
gst_element_link(h264_parser, decoder);
gst_element_link(decoder, fakesink);
GstPad *encoder_pad;
encoder_pad = gst_element_get_static_pad(decoder, "src");
gst_pad_add_probe(encoder_pad, GST_PAD_PROBE_TYPE_BUFFER,
(GstPadProbeCallback)probe_source_callback, NULL, NULL);
encoder_pad = gst_element_get_static_pad(decoder, "sink");
gst_pad_add_probe(encoder_pad, GST_PAD_PROBE_TYPE_BUFFER,
(GstPadProbeCallback)probe_sink_callback, NULL, NULL);
gst_object_unref (encoder_pad);
g_print("Setting g_main_loop_run to GST_STATE_PLAYING\n");
// Start pipeline so it could process incoming data
gst_element_set_state(m_pipeline, GST_STATE_PLAYING);
I also add those Pads to check that I have buffers on both sides of the
decoder.
Anyone know why that was the reason to fix this? (Removing the caps)
Thank you
On Mon, Jul 10, 2023 at 10:41 PM Michael Gruner <michael.gruner at ridgerun.com>
wrote:
> Nevermind, I just re-read that you had a parser already. Try removing the
> caps specification and letting the parser come with its own set of caps.
>
> On 10 Jul 2023, at 14:37, Michael Gruner <michael.gruner at ridgerun.com>
> wrote:
>
> You should be able to use the decoder, yes. Try placing a h264parse right
> after the appsrc. That will read the H264 bitstream, parse it, accumulate
> it to form full NAL units and perform stream format conversions if they are
> needed by the decoder.
>
> -Michael
>
> On 10 Jul 2023, at 09:04, Israel Díaz <isra601988 at gmail.com> wrote:
>
> Ohh. Thank you but it seems I don't have access to gst_buffer_new_memdup
> since I am using 1.18
>
> The other option to not copy I guess is not suitable for me because
> encoded_frame will be freed by another component (the encoded data is part
> of a bigger object that is free when the object call its destructor)
>
> So finally I opted for this
>
> buffer = gst_buffer_new_allocate (NULL, frame_size, NULL);
> gst_buffer_map (buffer, &map, GST_MAP_WRITE);
> memcpy((guchar *)map.data, encoded_frame, gst_buffer_get_size(
> buffer ) );
> gst_buffer_unmap(buffer, &map);
>
>
> Now I don't see those *broken/invalid nal Type: 0* messages but
> currently I still don't see that the raw file is receiving bytes.
> Do you think it is possible to use the decoder with this configuration?
> Maybe I need to follow the approach of need-data to feed the decoder or How
> Can I check if the decoder is actually processing everything?
>
> On Fri, Jul 7, 2023 at 4:50 PM Michael Gruner <michael.gruner at ridgerun.com>
> wrote:
>
>> Hi Israel
>>
>> This assignment is invalid:
>>
>> gst_buffer_map(buffer, &map, GST_MAP_WRITE);
>> * map.data = encoded_frame;*
>> gst_buffer_unmap(buffer, &map);
>>
>> You need to copy the data using a plain memcpy or
>> by creating the new buffer directly as a copy using gst_buffer_new_memdup.
>>
>> If you would like to avoid the copy, then you can use
>> gst_buffer_new_wrapped or gst_buffer_new_wrapped_full but you need to be
>> careful that: 1. The pointer remains valid the while the buffer is alive
>> and 2. The pointer is freed when the buffer is unreffed.
>>
>> Michael
>>
>> On 7 Jul 2023, at 04:02, Israel Díaz via gstreamer-devel <
>> gstreamer-devel at lists.freedesktop.org> wrote:
>>
>>
>> Dear GStreamer community,
>>
>> I am currently facing an issue while attempting to decode H264 frames
>> using the appsrc element in GStreamer. I have constructed the following
>> pipeline within my application:
>>
>> appsrc name=mysource ! video/x-h264, alignment=au,
>> stream-format=byte-stream ! h264parse ! video/x-h264, alignment=au,
>> stream-format=byte-stream ! omxh264dec ! video/x-raw, format=NV16_10LE32,
>> width=1920, height=1080, framerate=30/1 ! filesink location=ball.raw
>>
>> To feed the appsrc with frames, I am reading an H264 file frame by frame
>> since I store each frame in a structure along with its size. Then, I push
>> the frames into the appsrc using the following function:
>>
>> void
>> VDRE2GStreamerDecodingDisplaySinkTask::PushBufferToGStreamer(uint8_t*
>> encoded_frame, uint32_t frame_size)
>> {
>> GstFlowReturn ret;
>> GstMapInfo map;
>> GstBuffer *buffer = gst_buffer_new_and_alloc(frame_size);
>> gst_buffer_map(buffer, &map, GST_MAP_WRITE);
>> map.data = encoded_frame;
>> gst_buffer_unmap(buffer, &map);
>>
>> /* Push the buffer into the appsrc */
>> g_signal_emit_by_name(m_appsrc, "push-buffer", buffer, &ret);
>>
>> /* Free the buffer now that we are done with it */
>> gst_buffer_unref(buffer);
>>
>> if (ret != GST_FLOW_OK) {
>> g_print("PushBufferToGStreamer: Error!\n");
>> }
>> }
>>
>> However, when I execute the pipeline and set GST_DEBUG=4, I encounter the
>> following error messages:
>>
>> Decode Thread: Frame to Process : 1 :s 246
>> Decode Thread: Frame to Process : 2 :s 232
>> Decode Thread: Frame to Process : 3 :s 854
>> Decode Thread: Frame to Process : 4 :s 952
>> Decode Thread: Frame to Process : 5 :s 350
>> Decode Thread: Frame to Process : 6 :s 1038
>> Decode Thread: Frame to Process : 7 :s 1447
>> Decode Thread: Frame to Process : 8 :s 1316
>> Decode Thread: Frame to Process : 9 :s 1297
>> Decode Thread: Frame to Process : 10 :s 1205
>> 0:00:02.370465675 6548 0xffff8c01ff60 WARN h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 29 will be dropped
>> 0:00:02.370549896 6548 0xffff8c01ff60 WARN h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 1140 will be dropped
>> Decode Thread: Frame to Process : 11 :s 1222
>> Decode Thread: Frame to Process : 12 :s 1189
>> Decode Thread: Frame to Process : 13 :s 1094
>> Decode Thread: Frame to Process : 14 :s 930
>> Decode Thread: Frame to Process : 15 :s 1000
>> Decode Thread: Frame to Process : 16 :s 888
>> Decode Thread: Frame to Process : 17 :s 901
>> Decode Thread: Frame to Process : 18 :s 788
>> Decode Thread: Frame to Process : 19 :s 833
>> Decode Thread: Frame to Process : 20 :s 764
>> Decode Thread: Frame to Process : 21 :s 727
>> Decode Thread: Frame to Process : 22 :s 616
>> Decode Thread: Frame to Process : 23 :s 728
>> Decode Thread: Frame to Process : 24 :s 622
>> Decode Thread: Frame to Process : 25 :s 609
>> Decode Thread: Frame to Process : 26 :s 657
>> Decode Thread: Frame to Process : 27 :s 651
>> Decode Thread: Frame to Process : 28 :s 621
>> Decode Thread: Frame to Process : 29 :s 534
>> Decode Thread: Frame to Process : 30 :s 2989
>> 0:00:03.033182378 6548 0xffff8c01ff60 WARN h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 29 will be dropped
>> 0:00:03.033258709 6548 0xffff8c01ff60 WARN h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 117 will be dropped
>> 0:00:03.033309439 6548 0xffff8c01ff60 WARN h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 29 will be dropped
>> 0:00:03.033369800 6548 0xffff8c01ff60 WARN h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 1468 will be dropped
>> Decode Thread: Frame to Process : 31 :s 411
>> Decode Thread: Frame to Process : 32 :s 465
>> Decode Thread: Frame to Process : 33 :s 514
>> Decode Thread: Frame to Process : 34 :s 520
>> Decode Thread: Frame to Process : 35 :s 519
>> Decode Thread: Frame to Process : 36 :s 476
>> Decode Thread: Frame to Process : 37 :s 448
>> Decode Thread: Frame to Process : 38 :s 544
>> Decode Thread: Frame to Process : 39 :s 523
>> Decode Thread: Frame to Process : 40 :s 454
>> Decode Thread: Frame to Process : 41 :s 561
>> Decode Thread: Frame to Process : 42 :s 409
>> Decode Thread: Frame to Process : 43 :s 506
>> Decode Thread: Frame to Process : 44 :s 330
>> Decode Thread: Frame to Process : 45 :s 2180
>> 0:00:03.530095934 6548 0xffff8c01ff60 WARN h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 29 will be dropped
>> 0:00:03.530192075 6548 0xffff8c01ff60 WARN h264parse
>> gsth264parse.c:1562:gst_h264_parse_handle_frame:<myparser> broken/invalid
>> nal Type: 0 Unknown, Size: 117 will be dropped
>> 0:00:03.530251666 654
>>
>>
>> So then I analyzed the encoded_data to check if there are valid NAL units:
>>
>> have analyzed the encoded_frame data to determine if it contains valid
>> NAL units. Here's a summary of your findings for the first five data sets:
>>
>> Encoded data values in hexadecimal: 0, size: 3241
>> 00 00 00 01 27 7A 00 28 B6 D0 0C E8 07 80 22 7E 27 01 6A 02 04 04 80 00
>> 00 03 00 80 00 00 1E 19 A8 00 3D 09 00 01 6E 37 FF FE 05 00 00 00 01 28 EE
>> 3C
>>
>> Encoded data values in hexadecimal: 1, size: 246
>> 00 00 00 01 21 E2 01 08 42 DF 00 00 03 00 00 03 00 00 03 00 00 03 00 00
>> 03 00 00 03 00 00 03 00 00 09 36 B5 84 05 B0 C9 8C DA C9 6F 9D F0 00
>>
>> Encoded data values in hexadecimal: 2, size: 232
>> 00 00 00 01 21 E4 02 08 42 DF 00 00 03 00 00 03 00 00 03 00 00 03 00 00
>> 03 00 00 03 00 00 03 00 00 03 00 16 D0 00 00 01 21 00 1F E3 90 08 21
>>
>> Encoded data values in hexadecimal: 3, size: 854
>> 00 00 00 01 21 E6 03 08 8B 7F 00 00 03 00 00 03 00 00 03 00 00 03 00 00
>> 03 00 00 03 00 00 03 00 00 03 00 06 2C 00 00 01 21 00 1F E3 98 0C 22
>>
>> Encoded data values in hexadecimal: 4, size: 952
>> 00 00 00 01 21 E8 04 0E DF 00 00 03 00 00 03 00 00 03 00 00 03 00 00 03
>> 00 00 03 00 00 03 00 00 03 00 00 97 80 00 00 01 21 00 1F E3 A0 10 3B
>>
>> It seems that the analyzed data contains valid NAL units, as expected. I
>> have attempted a similar test using a file source and launching the
>> pipeline with gst-launch as follows:
>>
>> gst-launch-1.0 -e filesrc location=ball2.h264 ! video/x-h264,
>> alignment=au, width=1920, height=1080, framerate=30/1 ! h264parse !
>> omxh264dec ! video/x-raw, format=NV16_10LE32, width=1920, height=1080,
>> framerate=30/1 ! filesink location=ball_gst.raw
>>
>> During this test, I did not encounter any issues or error messages with
>> GST_DEBUG=4. Additionally, the raw file was populated with bytes, unlike
>> the appsrc version where the content always appears empty (presumably
>> because it is not being decoded at any point).
>>
>> I even tried with a identity dump=true element between filesrc and
>> h264parse and I also have the same bytes I am injecting to appsrc So I
>> don't know what can be the difference
>>
>> Thank you in advance.
>> Best regards.
>> Israel.
>>
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20230711/832e85f6/attachment-0001.htm>
More information about the gstreamer-devel
mailing list