[gst-devel] sending extra data along with buffers

[Jan Schmidt]

Stefan Kost wrote:
> hi,
>
> I've put a summary on the wiki:
> http://gstreamer.freedesktop.org/wiki/BufferMetadata
>
> please add your use-cases and ideas.
>   

I don't quite know how to have a dialogue on a wiki page. Seems like a 
good way to document a design, but a poor way to generate it.

Regarding what's there in the wiki at the moment:

I particularly dislike the idea of using the element name to identify 
data. It's pretty much exactly what I said I *don't* think the design 
should be - a dumping ground in which to place arbitrary data.

I think any data that requires intimate knowledge of the actual element 
generating it belongs in a private element event or message.

On the other hand, all the examples in the wiki look to me like things 
that can be considered tied to the caps - that is, they are extra 
descriptions of the content of the buffer, and knowledge of how to 
interpret them can be derived from the caps, or some extra property in 
the caps.

I also think that we should adopt a policy that any such 'extradata' (I 
want a better name, people!) is free to be dropped by any element that 
doesn't explicitly support it. I want whitelisting, rather than in any 
way expecting elements to pass through the data unless they know better.

J.

> Stefan
>
> Quoting Stefan Kost <ensonic at hora-obscura.de>:
>   
>> Hi,
>>
>> Jan Schmidt schrieb:
>>     
>>> <quote who="Stefan Kost">
>>>
>>>       
>>>> hi,
>>>>
>>>> I am looking for a way to send some private data along with a buffer.
>>>> During data capture I get some metadata about capture settings and I would
>>>> like to store/analyze them later along with the data stream. I tried to add
>>>> a payload item to caps and put a GstBuffer there. But this seems to be
>>>> painful and I am afraid that it will cause renegotiation as the content
>>>> changes for each buffer. Has anyone been doing something similar?
>>>>
>>>> alternative (better approach) seems to be using a buffer subclass, right?
>>>>
>>>> Attached is my current test. Its a bit hackish and fails. One thing I was
>>>> wondering is that fakesrc send buffer with caps=NULL. A bug?
>>>>
>>>>         
>>>> From time to time, I've thought about adding something like this directly
>>>>         
>>> to GstBuffer, for carrying information whose presence been negotiated in
>>> the caps, but whose content should be allowed to change without
>>> renegotiating.
>>>
>>> Examples:
>>>   * Pan and scan (video subregion) position information
>>>   * interlacing markers (top field, bottom field)
>>>   * theora granulepos, instead of slapping it in the 'offset' field
>>>   * MPEG Decoding timestamps, which we currently never send along.
>>>
>>> My thought was to add a GstStructure to GstBuffer which is allowed to be
>>> NULL.
>>>
>>> Wim has expressed concern that it would cause performance issues, and it
>>> would certainly require extending various bits of GStreamer to support
>>> properly (buffer serialisation/deserialisation in GDP, for example). Also,
>>> the semantics of how to handle the 'extradata' when transforming a buffer
>>> need to be clearly defined, ie the extradata has to be clearly specified
>>> by the caps attached to the buffer, and there needs to be a way for it to
>>> be handled by elements that don't understand the extradata they've been
>>> given. It can't just be a dumping ground in which to place arbitrary data,
>>> or I don't think it can be made to work.
>>>
>>> J.
>>>
>>>       
>> Maybe some GstTaglist features can help. I am thinking of the   
>> merge-modes here.
>> Atleast this is needed when mixing data.
>>
>> No idea how to express constraints that would render the data useless. E.g.
>> muxing two streams removes the 'frame' context from video frames.   
>> Thus the muxer
>> would not copy/merge the extra-data to outgoing buffers.
>>
>> I am not so much concerned with performance. In most cases it would   
>> not be used.
>> Freeing buffers would need to release the taglst if its not NULL. Copying
>> buffers need to copy it.
>>
>> As we have hierarchical structures now, we could probably require that the
>> producer uses its element name as a root tag. This way the taglist   
>> could carry
>> multiple data-sets identified in a unique way. Elements that can use the data
>> need to know the name of the element that created it.
>>
>> I'll think more about it - may be we can use our shiny new wiki to   
>> draft it :)
>>
>> Stefan
>>
>>
>> -------------------------------------------------------------------------
>> This SF.net email is sponsored by: Microsoft
>> Defy all challenges. Microsoft(R) Visual Studio 2008.
>> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.sourceforge.net
>> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>>
>>     
>
>
>
> -------------------------------------------------------------------------
> This SF.net email is sponsored by: Microsoft
> Defy all challenges. Microsoft(R) Visual Studio 2008.
> http://clk.atdmt.com/MRT/go/vse0120000070mrt/direct/01/
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.sourceforge.net
> https://lists.sourceforge.net/lists/listinfo/gstreamer-devel
>
>