Adding MetaData to GstBuffer
Roshan Chaudhari
rgc183 at gmail.com
Mon Sep 27 16:56:08 UTC 2021
I see some weird behaviour:
I add metadata on first element of sender:
ADD_PAD_PROBE(source, "src", Callback1,
GST_PAD_PROBE_TYPE_BUFFER, nullptr);
and then:
if(!gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
rtpgstpay, rtpsink, NULL)){
g_error("Failed to link ");
}
ADD_PAD_PROBE(rtpsink, "sink", Callback,
GST_PAD_PROBE_TYPE_BUFFER, nullptr);
with this, on probing on last element, I dont get metadata.. however,
removing rtpgstpay above, I see metadata on rtpsink..
Also if I replace sink -> src in ADD_PAD_PROBE, I dont get metadata with or
without rtpgstpay.
On Mon, Sep 27, 2021 at 9:16 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
wrote:
> If I am gonna take a quick guess, I think if you probe GstMeta at the
> sender after `audioconvert` you will not find it!
>
> This is basically addressed by metadata transformation which basically
> relates to how a particular element will handle (either by
> passing/dropping/transforming) the GstMeta attached with its incoming
> buffer.
> To break it down, `audioconvert` inherits from `GstBaseTransform` which
> has a default of implementation of the transform_meta() method. See
> https://gstreamer.freedesktop.org/documentation/base/gstbasetransform.html?gi-language=c#GstBaseTransformClass::transform_meta
>
> Now if `audioconvert` doesn't override GstBaseTransform's default
> implementation (which is true by my guess), then `audioconvert` will only copy
> all the metadata (GstMeta) without tags (default implementation).
> These tags are the ones you pass to gst_meta_api_type_register() and if
> you have tagged your GstMetaAPI (i.e. tags are not null) then
> `audioconvert` will not copy (drop) the metadata into the output buffer.
>
> Best Regards,
> Eslam Ahmed
>
>
> On Mon, Sep 27, 2021 at 5:05 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
> wrote:
>
>> Your code should work.
>>
>> However, I stumbled upon something like that but in a video branch. For
>> me it was an element in the pipeline that was dropping the GstMeta before
>> it gets payloaded. I think your case might be similar.
>>
>> Try probing the GstBuffer and check the existing metadata after every
>> element after you have injected it and make sure that it enters the
>> payloader as is. Please confirm that first! If so then we can dig into the
>> solution.
>>
>> Best Regards,
>> Eslam Ahmed
>>
>>
>> On Mon, Sep 27, 2021 at 1:00 PM Roshan Chaudhari <rgc183 at gmail.com>
>> wrote:
>>
>>> So I have created sample application with sender adding metadata and
>>> receiver reading it. I am using templates for gst_Add_meta from here:
>>> http://lifestyletransfer.com/how-to-add-metadata-to-gstreamer-buffer-in-python/
>>>
>>>
>>> Relevant code from my sender and receiver as follows:
>>> However, if I try to read meta data below in the same pipeline, it works.
>>>
>>> rtpsink is udpsink and rtpsrc is udpsrc.
>>>
>>> Sender:
>>>
>>> #define AUDIO_CAPS "audio/x-raw,format=S16LE,channels=1"
>>>
>>> gst_element_link_many(source, wavparse, audioconv,capsfilterAFX,
>>> rtpgstpay, rtpsink, NULL)
>>>
>>> ADD_PAD_PROBE(source, "src", Callback,
>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>
>>> GstPadProbeReturn Callback(GstPad *pad,
>>> GstPadProbeInfo
>>> *info,
>>> gpointer
>>> user_data) {
>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>>
>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>
>>> g_print("\n data sending from source ");
>>> GstMapInfo map = GST_MAP_INFO_INIT;
>>>
>>> GstMetaMarking* meta = GST_META_MARKING_ADD(buffer);
>>>
>>> meta->in_timestamp = 99000;
>>>
>>> // Below works
>>> /*GstMetaMarking* meta1 = GST_META_MARKING_GET(buffer);
>>> if(meta1)
>>> g_print("\n data is %lu ", meta1->in_timestamp);
>>> */
>>> return GST_PAD_PROBE_OK;
>>>
>>> }
>>>
>>>
>>> Receiver:
>>>
>>> // capsfilterAFX are same as sender AUDIO_CAPS
>>>
>>> gboolean res=gst_element_link_many(rtpsrc, rtpgstdepay, capsfilterAFX,
>>> audioconver,audioresample, fakesink,NULL);
>>> g_assert(res==TRUE);
>>>
>>> ADD_PAD_PROBE(capsfilterAFX, "src", Callback,
>>> GST_PAD_PROBE_TYPE_BUFFER, nullptr);
>>>
>>> GstPadProbeReturn Callback(GstPad *pad,
>>> GstPadProbeInfo *info,
>>> gpointer user_data) {
>>>
>>> GstBuffer *buffer = (GstBuffer *)info->data;
>>> //MaxineDSPipeline *pipeline = (MaxineDSPipeline *)user_data;
>>> GstMapInfo inmap = GST_MAP_INFO_INIT;
>>>
>>> GstMetaMarking* meta = GST_META_MARKING_GET(buffer);
>>> if(!meta)
>>> g_print("\n it is null ");
>>> else
>>> g_print("\n data received %lu", meta->in_timestamp);
>>>
>>> //g_print("\n data received");
>>> return GST_PAD_PROBE_OK;
>>> }
>>>
>>>
>>>
>>> However, my receiver, always get data null.
>>>
>>>
>>>
>>> -
>>> Thanks,
>>> Roshan
>>>
>>> On Mon, Sep 27, 2021 at 12:50 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>> wrote:
>>>
>>>> You are welcome!
>>>>
>>>> With gstrtppay/gstrtpdepay you should be able to serialize/deserialize
>>>> the GstBuffer over the network no matter the type of data it contains.
>>>> GstBuffer at this point could contain text, encoded/raw frame or an audio
>>>> frame.
>>>>
>>>> Best Regards,
>>>> Eslam Ahmed
>>>>
>>>>
>>>> On Mon, Sep 27, 2021 at 9:09 AM Roshan Chaudhari <rgc183 at gmail.com>
>>>> wrote:
>>>>
>>>>> Thank you Michael and Eslam!
>>>>>
>>>>> @Eslam, Seems like all the above options you mentioned are specific to
>>>>> MPEG, or RTP?
>>>>>
>>>>> I am looking for a solution which could be applied to any data such as
>>>>> text, audio, video data.
>>>>>
>>>>>
>>>>> --
>>>>> Thanks,
>>>>> Roshan
>>>>>
>>>>> On Sun, Sep 26, 2021 at 2:21 PM Eslam Ahmed <eslam.ahmed at avidbeam.com>
>>>>> wrote:
>>>>>
>>>>>> Roshan,
>>>>>>
>>>>>> Regarding Transport Stream, you can use the MPEG-TS container to
>>>>>> contain your encoded stream as well as your metadata in a klv-format. Check
>>>>>> this out:
>>>>>> https://gstreamer-devel.narkive.com/GlIqaK1k/example-code-for-muxing-klv-meta-x-klv-with-mpegtsmux-plugins-bad-and-gstreamer-1-8-3
>>>>>>
>>>>>> Just make sure to push klv metadata every video frame otherwise the
>>>>>> pipeline would stall. One useful thing to note is that your metadata need
>>>>>> not be in a klv format, you can just push arbitrary bytes as long as you
>>>>>> know how to interpret them after the demuxer at the receiver end.
>>>>>>
>>>>>> Another approach is to use payloaders (e.g. rtph264pay) and encode
>>>>>> your metadata as RTP extension headers. see
>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_add_extension_onebyte_header.
>>>>>> If your metadata is per-frame basis, then most probably and based on the
>>>>>> encoded frame size, one frame might end up having multiple RTP packets so
>>>>>> you will need to handle where to insert your metadata. You can use the
>>>>>> marker bit for that. see
>>>>>> https://gstreamer.freedesktop.org/documentation/rtplib/gstrtpbuffer.html?gi-language=c#gst_rtp_buffer_get_marker
>>>>>>
>>>>>> Final approach which is much more complicated which will require you
>>>>>> to create a new custom gstreamer media type (e.g. video/x-mytype) and
>>>>>> insert your custom metadata via gst_buffer_add_meta() in the
>>>>>> GstBuffer and use rtpgstpay/rtpgstdepay which guarantee to fully
>>>>>> serialize/deserialize the GstBuffer over the network. You would also need
>>>>>> to develop 2 gstreamer elements, one to prepend/append your metadata to
>>>>>> your au-aligned H264 and the other to extract the metadata from
>>>>>> video/x-mytype and restore the original stream.
>>>>>>
>>>>>> Hope that helps!
>>>>>>
>>>>>> Best Regards,
>>>>>> Eslam Ahmed
>>>>>>
>>>>>>
>>>>>> On Sat, Sep 25, 2021 at 10:00 PM Michael Gruner via gstreamer-devel <
>>>>>> gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>
>>>>>>> Hello Roshan
>>>>>>>
>>>>>>> The GstMeta is not transmitted through network. If your want a
>>>>>>> standard way to transmit metadata over the network along with audio and
>>>>>>> video you may look into Transport Stream. This will allow you to insert
>>>>>>> your custom metadata without breaking existing decoders. GStreamer already
>>>>>>> has support for this.
>>>>>>>
>>>>>>> Another option may be to use a codec that allows you to insert
>>>>>>> custom data within the compressed bitstream (like SEI in H264 or H265).
>>>>>>> That, however is a bit more tricky to get right.
>>>>>>>
>>>>>>> Michael
>>>>>>> www.ridgerun.con
>>>>>>>
>>>>>>> On 25 Sep 2021, at 08:24, Roshan Chaudhari via gstreamer-devel <
>>>>>>> gstreamer-devel at lists.freedesktop.org> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hello,
>>>>>>>
>>>>>>> I am trying to append some metadata to the data transferred over udp
>>>>>>> using gstreamer.
>>>>>>>
>>>>>>>
>>>>>>> 1. I would like to know whether metadata in GstBuffer is
>>>>>>> transferred/retained when transferred over the network or it is only
>>>>>>> retained in that pipeline? I have tried using my custom metadata with
>>>>>>> gst_buffer_add_meta() at the udpsink, however, when I try to query it on
>>>>>>> udpsrc on other machine using gst_buffer_get_meta(), metadata is not
>>>>>>> present in GstBuffer.
>>>>>>> 2.
>>>>>>> 3. If this is not carried over the network, what would be the
>>>>>>> other way to add metadata? I could write custom plugin and append to actual
>>>>>>> data, so my custom encoder and decoder knows how to extract real data and
>>>>>>> pass it to next stage in the pipeline. But this way, it puts restriction on
>>>>>>> the receiver side to have decoder if my data contains metadata.
>>>>>>>
>>>>>>>
>>>>>>> -
>>>>>>> Thanks,
>>>>>>> Roshan
>>>>>>>
>>>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/gstreamer-devel/attachments/20210927/fafedd7d/attachment-0001.htm>
More information about the gstreamer-devel
mailing list