Adding MetaData to GstBuffer
Roshan Chaudhari
rgc183 at gmail.com
Tue Sep 28 07:30:47 UTC 2021
I am using 1.19.0.1.
I will try with raw text data or using multiple pipelines in the same
process instead using udp.
Thanks,
Roshan
On Tue, Sep 28, 2021 at 12:04 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
wrote:
> That makes sense, Your metadata is present on the sink pad of rtpstpay but
> not present afterwards because now that audio frame has been converted into
> RTP packet(s).
>
> It's really weird that at the sender you attached rtpgstdepay and was able
> to obtain the metadata back but not at the receiver side.
>
> Btw, I am using Gstreamer 1.16.2.
>
> Best Regards,
> Eslam Ahmed
>
>
> On Tue, Sep 28, 2021 at 7:02 AM Roshan Chaudhari <rgc183 at gmail.com> wrote:
>
>> Quick update, from sender, I probed it on every element before rtpgstpay,
>> meta data is present. On sinkpad of rtpgstpay, it is present but not on
>> srcpad. If I add rtpgstdepay after rtpgstpay, data is present on srcpad of
>> rtpgstdepay, so looks like sender is fine and audioconvert is not messing
>> with the data.
>>
>> However, on the receiver, if I query after rtpgstdepay, data is not
>> present.
>>
>> On Mon, Sep 27, 2021 at 10:26 PM Roshan Chaudhari <rgc183 at gmail.com>
>> wrote:
>>
>>> I see some weird behaviour:
>>>
>>> I add metadata on first element of sender:
>>>
>>> ADD_PAD_PROBE(source, "src", Callback1,
>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>
>>> and then:
>>>
>>> if(!gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>>> rtpgstpay, rtpsink, NULL)){
>>> g_error("Failed to link ");
>>> }
>>>
>>> ADD_PAD_PROBE(rtpsink, "sink", Callback,
>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>
>>> with this, on probing on last element, I dont get metadata.. however,
>>> removing rtpgstpay above, I see metadata on rtpsink..
>>> Also if I replace sink -> src in ADD_PAD_PROBE, I dont get metadata with
>>> or without rtpgstpay.
>>>
>>>
>>>
>>> On Mon, Sep 27, 2021 at 9:16 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>> wrote:
>>>
>>>> If I am gonna take a quick guess, I think if you probe GstMeta at the
>>>> sender after `audioconvert` you will not find it!
>>>>
>>>> This is basically addressed by metadata transformation which basically
>>>> relates to how a particular element will handle (either by
>>>> passing/dropping/transforming) the GstMeta attached with its incoming
>>>> buffer.
>>>> To break it down, `audioconvert` inherits from `GstBaseTransform`
>>>> which has a default of implementation of the transform_meta() method. See
>>>> https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c#GstBaseTransformClass::transform_meta
>>>>
>>>> Now if `audioconvert` doesn't override GstBaseTransform's default
>>>> implementation (which is true by my guess), then `audioconvert` will only copy
>>>> all the metadata (GstMeta) without tags (default implementation).
>>>> These tags are the ones you pass to gst_meta_api_type_register() and
>>>> if you have tagged your GstMetaAPI (i.e. tags are not null) then
>>>> `audioconvert` will not copy (drop) the metadata into the output buffer.
>>>>
>>>> Best Regards,
>>>> Eslam Ahmed
>>>>
>>>>
>>>> On Mon, Sep 27, 2021 at 5:05 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>>> wrote:
>>>>
>>>>> Your code should work.
>>>>>
>>>>> However, I stumbled upon something like that but in a video branch.
>>>>> For me it was an element in the pipeline that was dropping the GstMeta
>>>>> before it gets payloaded. I think your case might be similar.
>>>>>
>>>>> Try probing the GstBuffer and check the existing metadata after every
>>>>> element after you have injected it and make sure that it enters the
>>>>> payloader as is. Please confirm that first! If so then we can dig into the
>>>>> solution.
>>>>>
>>>>> Best Regards,
>>>>> Eslam Ahmed
>>>>>
>>>>>
>>>>> On Mon, Sep 27, 2021 at 1:00 PM Roshan Chaudhari <rgc183 at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> So I have created sample application with sender adding metadata and
>>>>>> receiver reading it. I am using templates for gst_Add_meta from here:
>>>>>> http://lifestyletransfer.com/how-to-add-metadata-to-gstreamer-buffer-in-python/
>>>>>>
>>>>>>
>>>>>> Relevant code from my sender and receiver as follows:
>>>>>> However, if I try to read meta data below in the same pipeline, it
>>>>>> works.
>>>>>>
>>>>>> rtpsink is udpsink and rtpsrc is udpsrc.
>>>>>>
>>>>>> Sender:
>>>>>>
>>>>>> #define AUDIO_CAPS "audio/x-raw,format=S16LE,channels=1"
>>>>>>
>>>>>> gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>>>>>> rtpgstpay, rtpsink, NULL)
>>>>>>
>>>>>> ADD_PAD_PROBE(source, "src", Callback,
>>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>>
>>>>>> GstPadProbeReturn Callback(GstPad *pad,
>>>>>>
>>>>>> GstPadProbeInfo *info,
>>>>>> gpointer
>>>>>> user_data) {
>>>>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>>>>
>>>>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>>>>
>>>>>> g_print("\n data sending from source ");
>>>>>> GstMapInfo map = GST_MAP_INFO_INIT;
>>>>>>
>>>>>> GstMetaMarking* meta = GST_META_MARKING_ADD(buffer);
>>>>>>
>>>>>> meta->in_timestamp = 99000;
>>>>>>
>>>>>> // Below works
>>>>>> /*GstMetaMarking* meta1 = GST_META_MARKING_GET(buffer);
>>>>>> if(meta1)
>>>>>> g_print("\n data is %lu ", meta1->in_timestamp);
>>>>>> */
>>>>>> return GST_PAD_PROBE_OK;
>>>>>>
>>>>>> }
>>>>>>
>>>>>>
>>>>>> Receiver:
>>>>>>
>>>>>> // capsfilterAFX are same as sender AUDIO_CAPS
>>>>>>
>>>>>> gboolean res=gst_element_link_many(rtpsrc, rtpgstdepay,
>>>>>> capsfilterAFX, audioconver,audioresample, fakesink,NULL);
>>>>>> g_assert(res==TRUE);
>>>>>>
>>>>>> ADD_PAD_PROBE(capsfilterAFX, "src", Callback,
>>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>>
>>>>>> GstPadProbeReturn Callback(GstPad *pad,
>>>>>> GstPadProbeInfo *info,
>>>>>> gpointer user_data) {
>>>>>>
>>>>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>>>> //MaxineDSPipeline *pipeline = (MaxineDSPipeline *)user_data;
>>>>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>>>>
>>>>>> GstMetaMarking* meta = GST_META_MARKING_GET(buffer);
>>>>>> if(!meta)
>>>>>> g_print("\n it is null ");
>>>>>> else
>>>>>> g_print("\n data received %lu", meta->in_timestamp);
>>>>>>
>>>>>> //g_print("\n data received");
>>>>>> return GST_PAD_PROBE_OK;
>>>>>> }
>>>>>>
>>>>>>
>>>>>>
>>>>>> However, my receiver, always get data null.
>>>>>>
>>>>>>
>>>>>>
>>>>>> -
>>>>>> Thanks,
>>>>>> Roshan
>>>>>>
>>>>>> On Mon, Sep 27, 2021 at 12:50 PM Eslam Ahmed <
>>>>>> eslam.ahmed at avidbeam.com> wrote:
>>>>>>
>>>>>>> You are welcome!
>>>>>>>
>>>>>>> With gstrtppay/gstrtpdepay you should be able to
>>>>>>> serialize/deserialize the GstBuffer over the network no matter the type of
>>>>>>> data it contains. GstBuffer at this point could contain text,
>>>>>>> encoded/raw frame or an audio frame.
>>>>>>>
>>>>>>> Best Regards,
>>>>>>> Eslam Ahmed
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Sep 27, 2021 at 9:09 AM Roshan Chaudhari <rgc183 at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Thank you Michael and Eslam!
>>>>>>>>
>>>>>>>> @Eslam, Seems like all the above options you mentioned are specific
>>>>>>>> to MPEG, or RTP?
>>>>>>>>
>>>>>>>> I am looking for a solution which could be applied to any data such
>>>>>>>> as text, audio, video data.
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Thanks,
>>>>>>>> Roshan
>>>>>>>>
>>>>>>>> On Sun, Sep 26, 2021 at 2:21 PM Eslam Ahmed <
>>>>>>>> eslam.ahmed at avidbeam.com> wrote:
>>>>>>>>
>>>>>>>>> Roshan,
>>>>>>>>>
>>>>>>>>> Regarding Transport Stream, you can use the MPEG-TS container to
>>>>>>>>> contain your encoded stream as well as your metadata in a klv-format. Check
>>>>>>>>> this out:
>>>>>>>>> https://gstreamer-devel.narkive.com/GlIqaK1k/example-code-for-muxing-klv-meta-x-klv-with-mpegtsmux-plugins-bad-and-gstreamer-1-8-3
>>>>>>>>>
>>>>>>>>> Just make sure to push klv metadata every video frame otherwise
>>>>>>>>> the pipeline would stall. One useful thing to note is that your
>>>>>>>>> metadata need not be in a klv format, you can just push arbitrary bytes as
>>>>>>>>> long as you know how to interpret them after the demuxer at the
>>>>>>>>> receiver end.
>>>>>>>>>
>>>>>>>>> Another approach is to use payloaders (e.g. rtph264pay) and encode
>>>>>>>>> your metadata as RTP extension headers. see
>>>>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_add_extension_onebyte_header.
>>>>>>>>> If your metadata is per-frame basis, then most probably and based on the
>>>>>>>>> encoded frame size, one frame might end up having multiple RTP packets so
>>>>>>>>> you will need to handle where to insert your metadata. You can use the
>>>>>>>>> marker bit for that. see
>>>>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_get_marker
>>>>>>>>>
>>>>>>>>> Final approach which is much more complicated which will require
>>>>>>>>> you to create a new custom gstreamer media type (e.g. video/x-mytype) and
>>>>>>>>> insert your custom metadata via gst_buffer_add_meta() in the
>>>>>>>>> GstBuffer and use rtpgstpay/rtpgstdepay which guarantee to fully
>>>>>>>>> serialize/deserialize the GstBuffer over the network. You would also need
>>>>>>>>> to develop 2 gstreamer elements, one to prepend/append your metadata to
>>>>>>>>> your au-aligned H264 and the other to extract the metadata from
>>>>>>>>> video/x-mytype and restore the original stream.
>>>>>>>>>
>>>>>>>>> Hope that helps!
>>>>>>>>>
>>>>>>>>> Best Regards,
>>>>>>>>> Eslam Ahmed
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Sat, Sep 25, 2021 at 10:00 PM Michael Gruner via
>>>>>>>>> gstreamer-devel <gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>>>>
>>>>>>>>>> Hello Roshan
>>>>>>>>>>
>>>>>>>>>> The GstMeta is not transmitted through network. If your want a
>>>>>>>>>> standard way to transmit metadata over the network along with audio and
>>>>>>>>>> video you may look into Transport Stream. This will allow you to insert
>>>>>>>>>> your custom metadata without breaking existing decoders. GStreamer already
>>>>>>>>>> has support for this.
>>>>>>>>>>
>>>>>>>>>> Another option may be to use a codec that allows you to insert
>>>>>>>>>> custom data within the compressed bitstream (like SEI in H264 or H265).
>>>>>>>>>> That, however is a bit more tricky to get right.
>>>>>>>>>>
>>>>>>>>>> Michael
>>>>>>>>>> www.ridgerun.con
>>>>>>>>>>
>>>>>>>>>> On 25 Sep 2021, at 08:24, Roshan Chaudhari via gstreamer-devel <
>>>>>>>>>> gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> I am trying to append some metadata to the data transferred over
>>>>>>>>>> udp using gstreamer.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> 1. I would like to know whether metadata in GstBuffer is
>>>>>>>>>> transferred/retained when transferred over the network or it is only
>>>>>>>>>> retained in that pipeline? I have tried using my custom metadata with
>>>>>>>>>> gst_buffer_add_meta() at the udpsink, however, when I try to query it on
>>>>>>>>>> udpsrc on other machine using gst_buffer_get_meta(), metadata is not
>>>>>>>>>> present in GstBuffer.
>>>>>>>>>> 2.
>>>>>>>>>> 3. If this is not carried over the network, what would be the
>>>>>>>>>> other way to add metadata? I could write custom plugin and append to actual
>>>>>>>>>> data, so my custom encoder and decoder knows how to extract real data and
>>>>>>>>>> pass it to next stage in the pipeline. But this way, it puts restriction on
>>>>>>>>>> the receiver side to have decoder if my data contains metadata.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> -
>>>>>>>>>> Thanks,
>>>>>>>>>> Roshan
>>>>>>>>>>
>>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210928/1e584e9a/attachment-0001.htm>
More information about the gstreamer-devel
mailing list