[gst-devel] sending extra data along with buffers

Tue Jan 22 08:12:17 CET 2008

Hi,

Quoting Jan Schmidt <thaytan at noraisin.net>:

> Stefan Kost wrote:
>> hi,
>>
>> I've put a summary on the wiki:
>> http://gstreamer.freedesktop.org/wiki/BufferMetadata
>>
>> please add your use-cases and ideas.
>>
>
> I don't quite know how to have a dialogue on a wiki page. Seems like a
> good way to document a design, but a poor way to generate it.
>
Yes having the dialog here is good. Still it would be good to update  
the wiki page at the same time to have the running draft there.
>
> Regarding what's there in the wiki at the moment:
>
> I particularly dislike the idea of using the element name to identify
> data. It's pretty much exactly what I said I *don't* think the design
> should be - a dumping ground in which to place arbitrary data.
>
> I think any data that requires intimate knowledge of the actual element
> generating it belongs in a private element event or message.

In the past days I wrote a test programm that uses tag events and it  
shows that they are indeed in sync with the buffer flow. Means one  
element sends both buffers and tag-events out on its src pad. Then the  
connected peer element will receive them in the same order. (tried  
with two basetransform elements).
I'll make that a bit clearer in the docs after the freeze.

This definitely makes events the better mechanism for realy private stuff.
>
> On the other hand, all the examples in the wiki look to me like things
> that can be considered tied to the caps - that is, they are extra
> descriptions of the content of the buffer, and knowledge of how to
> interpret them can be derived from the caps, or some extra property in
> the caps.

What about caps like
video/whatever, width=320, height=240,...,extradata=(fields,subregion);

that would mean that the extradata has two structures named 'fields'  
and 'subregion'. Regarding the name 'extradata', it is still caps  
data. Only not essential for all plugins.

>
> I also think that we should adopt a policy that any such 'extradata' (I
> want a better name, people!) is free to be dropped by any element that
> doesn't explicitly support it. I want whitelisting, rather than in any
> way expecting elements to pass through the data unless they know better.
>
An element that e.g. just changes image contrast would/should not need  
to drop the interlaced field info. I means that we need to define  
those subsets, their name and purpose.

Stefan

>
> J.
>
>> Stefan
>>
>> Quoting Stefan Kost <ensonic at hora-obscura.de>:
>>
>>> Hi,
>>>
>>> Jan Schmidt schrieb:
>>>
>>>> <quote who="Stefan Kost">
>>>>
>>>>
>>>>> hi,
>>>>>
>>>>> I am looking for a way to send some private data along with a buffer.
>>>>> During data capture I get some metadata about capture settings   
>>>>> and I would
>>>>> like to store/analyze them later along with the data stream. I   
>>>>> tried to add
>>>>> a payload item to caps and put a GstBuffer there. But this seems to be
>>>>> painful and I am afraid that it will cause renegotiation as the content
>>>>> changes for each buffer. Has anyone been doing something similar?
>>>>>
>>>>> alternative (better approach) seems to be using a buffer subclass, right?
>>>>>
>>>>> Attached is my current test. Its a bit hackish and fails. One thing I was
>>>>> wondering is that fakesrc send buffer with caps=NULL. A bug?
>>>>>
>>>>>        From time to time, I've thought about adding something   
>>>>> like this directly
>>>>>
>>>> to GstBuffer, for carrying information whose presence been negotiated in
>>>> the caps, but whose content should be allowed to change without
>>>> renegotiating.
>>>>
>>>> Examples:
>>>>  * Pan and scan (video subregion) position information
>>>>  * interlacing markers (top field, bottom field)
>>>>  * theora granulepos, instead of slapping it in the 'offset' field
>>>>  * MPEG Decoding timestamps, which we currently never send along.
>>>>
>>>> My thought was to add a GstStructure to GstBuffer which is allowed to be
>>>> NULL.
>>>>
>>>> Wim has expressed concern that it would cause performance issues, and it
>>>> would certainly require extending various bits of GStreamer to support
>>>> properly (buffer serialisation/deserialisation in GDP, for example). Also,
>>>> the semantics of how to handle the 'extradata' when transforming a buffer
>>>> need to be clearly defined, ie the extradata has to be clearly specified
>>>> by the caps attached to the buffer, and there needs to be a way for it to
>>>> be handled by elements that don't understand the extradata they've been
>>>> given. It can't just be a dumping ground in which to place arbitrary data,
>>>> or I don't think it can be made to work.
>>>>
>>>> J.
>>>>
>>>>
>>> Maybe some GstTaglist features can help. I am thinking of the     
>>> merge-modes here.
>>> Atleast this is needed when mixing data.
>>>
>>> No idea how to express constraints that would render the data useless. E.g.
>>> muxing two streams removes the 'frame' context from video frames.   
>>>   Thus the muxer
>>> would not copy/merge the extra-data to outgoing buffers.
>>>
>>> I am not so much concerned with performance. In most cases it   
>>> would   not be used.
>>> Freeing buffers would need to release the taglst if its not NULL. Copying
>>> buffers need to copy it.
>>>
>>> As we have hierarchical structures now, we could probably require that the
>>> producer uses its element name as a root tag. This way the taglist  
>>>    could carry
>>> multiple data-sets identified in a unique way. Elements that can   
>>> use the data
>>> need to know the name of the element that created it.
>>>
>>> I'll think more about it - may be we can use our shiny new wiki to  
>>>    draft it :)
>>>
>>> Stefan
>>>
>>>
>>> -------------------------------------------------------------------------
>>> This SF.net email is sponsored by: Microsoft
>>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>>> _______________________________________________
>>> gstreamer-devel mailing list
>>> gstreamer-devel at lists.sourceforge.net
>>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>>
>>>
>>
>>
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Microsoft
>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>
>>