Adding MetaData to GstBuffer
Eslam Ahmed
eslam.ahmed at avidbeam.com
Tue Sep 28 06:34:03 UTC 2021
That makes sense, Your metadata is present on the sink pad of rtpstpay but
not present afterwards because now that audio frame has been converted into
RTP packet(s).
It's really weird that at the sender you attached rtpgstdepay and was able
to obtain the metadata back but not at the receiver side.
Btw, I am using Gstreamer 1.16.2.
Best Regards,
Eslam Ahmed
On Tue, Sep 28, 2021 at 7:02 AM Roshan Chaudhari <rgc183 at gmail.com> wrote:
> Quick update, from sender, I probed it on every element before rtpgstpay,
> meta data is present. On sinkpad of rtpgstpay, it is present but not on
> srcpad. If I add rtpgstdepay after rtpgstpay, data is present on srcpad of
> rtpgstdepay, so looks like sender is fine and audioconvert is not messing
> with the data.
>
> However, on the receiver, if I query after rtpgstdepay, data is not
> present.
>
> On Mon, Sep 27, 2021 at 10:26 PM Roshan Chaudhari <rgc183 at gmail.com>
> wrote:
>
>> I see some weird behaviour:
>>
>> I add metadata on first element of sender:
>>
>> ADD_PAD_PROBE(source, "src", Callback1,
>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>
>> and then:
>>
>> if(!gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>> rtpgstpay, rtpsink, NULL)){
>> g_error("Failed to link ");
>> }
>>
>> ADD_PAD_PROBE(rtpsink, "sink", Callback,
>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>
>> with this, on probing on last element, I dont get metadata.. however,
>> removing rtpgstpay above, I see metadata on rtpsink..
>> Also if I replace sink -> src in ADD_PAD_PROBE, I dont get metadata with
>> or without rtpgstpay.
>>
>>
>>
>> On Mon, Sep 27, 2021 at 9:16 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>> wrote:
>>
>>> If I am gonna take a quick guess, I think if you probe GstMeta at the
>>> sender after `audioconvert` you will not find it!
>>>
>>> This is basically addressed by metadata transformation which basically
>>> relates to how a particular element will handle (either by
>>> passing/dropping/transforming) the GstMeta attached with its incoming
>>> buffer.
>>> To break it down, `audioconvert` inherits from `GstBaseTransform` which
>>> has a default of implementation of the transform_meta() method. See
>>> https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c#GstBaseTransformClass::transform_meta
>>>
>>> Now if `audioconvert` doesn't override GstBaseTransform's default
>>> implementation (which is true by my guess), then `audioconvert` will only copy
>>> all the metadata (GstMeta) without tags (default implementation).
>>> These tags are the ones you pass to gst_meta_api_type_register() and if
>>> you have tagged your GstMetaAPI (i.e. tags are not null) then
>>> `audioconvert` will not copy (drop) the metadata into the output buffer.
>>>
>>> Best Regards,
>>> Eslam Ahmed
>>>
>>>
>>> On Mon, Sep 27, 2021 at 5:05 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>> wrote:
>>>
>>>> Your code should work.
>>>>
>>>> However, I stumbled upon something like that but in a video branch. For
>>>> me it was an element in the pipeline that was dropping the GstMeta before
>>>> it gets payloaded. I think your case might be similar.
>>>>
>>>> Try probing the GstBuffer and check the existing metadata after every
>>>> element after you have injected it and make sure that it enters the
>>>> payloader as is. Please confirm that first! If so then we can dig into the
>>>> solution.
>>>>
>>>> Best Regards,
>>>> Eslam Ahmed
>>>>
>>>>
>>>> On Mon, Sep 27, 2021 at 1:00 PM Roshan Chaudhari <rgc183 at gmail.com>
>>>> wrote:
>>>>
>>>>> So I have created sample application with sender adding metadata and
>>>>> receiver reading it. I am using templates for gst_Add_meta from here:
>>>>> http://lifestyletransfer.com/how-to-add-metadata-to-gstreamer-buffer-in-python/
>>>>>
>>>>>
>>>>> Relevant code from my sender and receiver as follows:
>>>>> However, if I try to read meta data below in the same pipeline, it
>>>>> works.
>>>>>
>>>>> rtpsink is udpsink and rtpsrc is udpsrc.
>>>>>
>>>>> Sender:
>>>>>
>>>>> #define AUDIO_CAPS "audio/x-raw,format=S16LE,channels=1"
>>>>>
>>>>> gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>>>>> rtpgstpay, rtpsink, NULL)
>>>>>
>>>>> ADD_PAD_PROBE(source, "src", Callback,
>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>
>>>>> GstPadProbeReturn Callback(GstPad *pad,
>>>>>
>>>>> GstPadProbeInfo *info,
>>>>> gpointer
>>>>> user_data) {
>>>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>>>
>>>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>>>
>>>>> g_print("\n data sending from source ");
>>>>> GstMapInfo map = GST_MAP_INFO_INIT;
>>>>>
>>>>> GstMetaMarking* meta = GST_META_MARKING_ADD(buffer);
>>>>>
>>>>> meta->in_timestamp = 99000;
>>>>>
>>>>> // Below works
>>>>> /*GstMetaMarking* meta1 = GST_META_MARKING_GET(buffer);
>>>>> if(meta1)
>>>>> g_print("\n data is %lu ", meta1->in_timestamp);
>>>>> */
>>>>> return GST_PAD_PROBE_OK;
>>>>>
>>>>> }
>>>>>
>>>>>
>>>>> Receiver:
>>>>>
>>>>> // capsfilterAFX are same as sender AUDIO_CAPS
>>>>>
>>>>> gboolean res=gst_element_link_many(rtpsrc, rtpgstdepay, capsfilterAFX,
>>>>> audioconver,audioresample, fakesink,NULL);
>>>>> g_assert(res==TRUE);
>>>>>
>>>>> ADD_PAD_PROBE(capsfilterAFX, "src", Callback,
>>>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>>>
>>>>> GstPadProbeReturn Callback(GstPad *pad,
>>>>> GstPadProbeInfo *info,
>>>>> gpointer user_data) {
>>>>>
>>>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>>> //MaxineDSPipeline *pipeline = (MaxineDSPipeline *)user_data;
>>>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>>>
>>>>> GstMetaMarking* meta = GST_META_MARKING_GET(buffer);
>>>>> if(!meta)
>>>>> g_print("\n it is null ");
>>>>> else
>>>>> g_print("\n data received %lu", meta->in_timestamp);
>>>>>
>>>>> //g_print("\n data received");
>>>>> return GST_PAD_PROBE_OK;
>>>>> }
>>>>>
>>>>>
>>>>>
>>>>> However, my receiver, always get data null.
>>>>>
>>>>>
>>>>>
>>>>> -
>>>>> Thanks,
>>>>> Roshan
>>>>>
>>>>> On Mon, Sep 27, 2021 at 12:50 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>>>> wrote:
>>>>>
>>>>>> You are welcome!
>>>>>>
>>>>>> With gstrtppay/gstrtpdepay you should be able to
>>>>>> serialize/deserialize the GstBuffer over the network no matter the type of
>>>>>> data it contains. GstBuffer at this point could contain text,
>>>>>> encoded/raw frame or an audio frame.
>>>>>>
>>>>>> Best Regards,
>>>>>> Eslam Ahmed
>>>>>>
>>>>>>
>>>>>> On Mon, Sep 27, 2021 at 9:09 AM Roshan Chaudhari <rgc183 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Thank you Michael and Eslam!
>>>>>>>
>>>>>>> @Eslam, Seems like all the above options you mentioned are specific
>>>>>>> to MPEG, or RTP?
>>>>>>>
>>>>>>> I am looking for a solution which could be applied to any data such
>>>>>>> as text, audio, video data.
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Thanks,
>>>>>>> Roshan
>>>>>>>
>>>>>>> On Sun, Sep 26, 2021 at 2:21 PM Eslam Ahmed <
>>>>>>> eslam.ahmed at avidbeam.com> wrote:
>>>>>>>
>>>>>>>> Roshan,
>>>>>>>>
>>>>>>>> Regarding Transport Stream, you can use the MPEG-TS container to
>>>>>>>> contain your encoded stream as well as your metadata in a klv-format. Check
>>>>>>>> this out:
>>>>>>>> https://gstreamer-devel.narkive.com/GlIqaK1k/example-code-for-muxing-klv-meta-x-klv-with-mpegtsmux-plugins-bad-and-gstreamer-1-8-3
>>>>>>>>
>>>>>>>> Just make sure to push klv metadata every video frame otherwise the
>>>>>>>> pipeline would stall. One useful thing to note is that your metadata need
>>>>>>>> not be in a klv format, you can just push arbitrary bytes as long as you
>>>>>>>> know how to interpret them after the demuxer at the receiver end.
>>>>>>>>
>>>>>>>> Another approach is to use payloaders (e.g. rtph264pay) and encode
>>>>>>>> your metadata as RTP extension headers. see
>>>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_add_extension_onebyte_header.
>>>>>>>> If your metadata is per-frame basis, then most probably and based on the
>>>>>>>> encoded frame size, one frame might end up having multiple RTP packets so
>>>>>>>> you will need to handle where to insert your metadata. You can use the
>>>>>>>> marker bit for that. see
>>>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_get_marker
>>>>>>>>
>>>>>>>> Final approach which is much more complicated which will require
>>>>>>>> you to create a new custom gstreamer media type (e.g. video/x-mytype) and
>>>>>>>> insert your custom metadata via gst_buffer_add_meta() in the
>>>>>>>> GstBuffer and use rtpgstpay/rtpgstdepay which guarantee to fully
>>>>>>>> serialize/deserialize the GstBuffer over the network. You would also need
>>>>>>>> to develop 2 gstreamer elements, one to prepend/append your metadata to
>>>>>>>> your au-aligned H264 and the other to extract the metadata from
>>>>>>>> video/x-mytype and restore the original stream.
>>>>>>>>
>>>>>>>> Hope that helps!
>>>>>>>>
>>>>>>>> Best Regards,
>>>>>>>> Eslam Ahmed
>>>>>>>>
>>>>>>>>
>>>>>>>> On Sat, Sep 25, 2021 at 10:00 PM Michael Gruner via gstreamer-devel
>>>>>>>> <gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>>>
>>>>>>>>> Hello Roshan
>>>>>>>>>
>>>>>>>>> The GstMeta is not transmitted through network. If your want a
>>>>>>>>> standard way to transmit metadata over the network along with audio and
>>>>>>>>> video you may look into Transport Stream. This will allow you to insert
>>>>>>>>> your custom metadata without breaking existing decoders. GStreamer already
>>>>>>>>> has support for this.
>>>>>>>>>
>>>>>>>>> Another option may be to use a codec that allows you to insert
>>>>>>>>> custom data within the compressed bitstream (like SEI in H264 or H265).
>>>>>>>>> That, however is a bit more tricky to get right.
>>>>>>>>>
>>>>>>>>> Michael
>>>>>>>>> www.ridgerun.con
>>>>>>>>>
>>>>>>>>> On 25 Sep 2021, at 08:24, Roshan Chaudhari via gstreamer-devel <
>>>>>>>>> gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hello,
>>>>>>>>>
>>>>>>>>> I am trying to append some metadata to the data transferred over
>>>>>>>>> udp using gstreamer.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> 1. I would like to know whether metadata in GstBuffer is
>>>>>>>>> transferred/retained when transferred over the network or it is only
>>>>>>>>> retained in that pipeline? I have tried using my custom metadata with
>>>>>>>>> gst_buffer_add_meta() at the udpsink, however, when I try to query it on
>>>>>>>>> udpsrc on other machine using gst_buffer_get_meta(), metadata is not
>>>>>>>>> present in GstBuffer.
>>>>>>>>> 2.
>>>>>>>>> 3. If this is not carried over the network, what would be the
>>>>>>>>> other way to add metadata? I could write custom plugin and append to actual
>>>>>>>>> data, so my custom encoder and decoder knows how to extract real data and
>>>>>>>>> pass it to next stage in the pipeline. But this way, it puts restriction on
>>>>>>>>> the receiver side to have decoder if my data contains metadata.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> -
>>>>>>>>> Thanks,
>>>>>>>>> Roshan
>>>>>>>>>
>>>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210928/c4b7b9f8/attachment-0001.htm>
More information about the gstreamer-devel
mailing list