How to track buffer caps in gstreamer 1.x ?

Sat Jan 31 06:18:57 PST 2015

Thanks, Thiago!

It's great to have some guidance at this stage. Very much appreciated.

Ben

On Thu, Jan 29, 2015 at 10:06 PM, Thiago Santos <thiagoss at osg.samsung.com>
wrote:

>  On 01/29/2015 02:35 PM, Ben Bridgwater wrote:
>
>  Thanks Thiago - I appreciate your responsiveness!
>
>  I'm at the planning stage of a port-to-gstreamer project, so I'm trying
> to understand it better before I make any final design decisions.
>
>  My application is an interactive spectrogram with a Qt GUI that let's
> the user modify FFT (spectrogram) parameters (which will mean gstreamer
> caps changes) on-the-fly as audio data is streaming. I support both
> real-time streaming (e.g. from a microphone source) as well as manual audio
> file "browsing" where the user can use a scroll bar to seek through the
> audio file with the visible portion being pushed thru the processing
> pipeline in response to scroll movement. My out-of-order buffer processing
> requirement comes from the Qt GUI which maintains a display buffer of the
> stream chunks (buffers) it has received from the pipeline, and may need to
> render them (needing data format/caps) in any order per the Qt paintEvents
> it receives.
>
> You can store the buffers with caps in GstSamples, you don't need to copy
> the caps or the buffer, just keep a ref to them in the GstSample and the
> overhead will be on the allocation of a struct, which shouldn't be that
> much.
>
> If you don't need to pass this buffer around you can also have some other
> custom way of identifying the format internally in your element if you
> believe performance would be improved.
>
>
>  The application currently uses my own pipeline/element design, but I
> want to port to gstreamer in order to support a wider variety of audio
> sources as well as spectrogram-synced audio playback capability.
>
>  Right now the GUI is the final sink of my pipeline (audio-source ->
> audio-resampling -> audio-preemphasis -> FFT -> Qt-signal-adapter -> Qt
> GUI), but in the future I will be feeding the output into a machine
> learning program, and so would like to make my FFT/analysis component a
> greamer element rather than appsink in order to retain flexibility. I may
> also split this "FFT" component into multiple elements as it also does some
> compute-heavy FFT post-processing that could benefit from being
> multithreaded via gstreamer queue elements.
>
> Good idea, it will make it more reusable.
>
>
>  In the port to gstreamer, I'm planning to implement the GUI-driven FFT
> parameter changes via a caps filter inserted after the FFT element, via
> which I can set the desired caps and force the FTT to adapt. The downstream
> Qt GUI will therefore receive a stream of caps change events and buffers
> and per the out-of-order processing requirement I need an efficient way to
> map from buffers to corresponding caps/data format. I'm thinking that if I
> could add a "caps ID" tag to the GstBuffers (metadata or reuse some
> existing struct member) then I could maintain a corresponding "caps ID" to
> caps map in the GUI to make the association.
>
> You don't necessarily need the capsfilter, your element can change the
> requirements on its pads dynamically and push a 'reconfigure' event
> upstream to notify other elements that they should try to renegotiate to
> another format. This is essentially what the capsfilter would be doing when
> you set a new caps to it.
>
>
>  Hopefully this explains what I'm trying to do. Any suggestions or advice
> are is very welcome, especially regarding the out-of-order buffer-to-caps
> mapping.
>
> Hope it helps, success on your porting!
>
>
>  Ben
>
>
> On Thu, Jan 29, 2015 at 11:46 AM, Thiago Santos <thiagoss at osg.samsung.com>
> wrote:
>
>>  On 01/29/2015 12:52 PM, Ben Bridgwater wrote:
>>
>>  Thanks, Thiago.
>>
>>  Yes, in my application (because of the out-of-order buffer processing)
>> I do need to explicitly associate buffers with caps rather than just rely
>> on the caps event and buffer ordering.
>>
>>  What is your application exactly? Are you using appsink, fakesink or a
>> custom sink? Or is this another type of element?
>>
>>
>>  I assume gstreamer pipeline elements can't actually pass GstSamples
>> rather than GstBuffers - this is just a convenience class for a sink to
>> associate buffers/caps/segments rather than a type that would be passed
>> from element to element?
>>
>>  Yes, only buffers are passed through pads.
>>
>>
>>  It seems another option for me might be to use GstBuffer metadata to
>> add the tagging/caps info I need. Would I be right in thinking that an
>> element could receive a GstBuffer, add some metadata, then pass the
>> GstBuffer with added metadata down the pipleline? Would an existing
>> element/plugin care if it received a buffer with added metadata that it
>> wasn't expecting? Presumably adding metadata involves an additional malloc?
>>
>>  Is there any way to add any metadata (maybe just an integer tag) to a
>> GstBuffer without doing an additional alloc? Any GstBuffer members than it
>> would be harmless to repurpose as a tag, perhaps?
>>
>>  There is GstMeta that is a flexible metadata system that can be
>> associated with buffers. You need to make sure that other elements in the
>> pipeline preserve the meta when passing buffers around (specially the ones
>> that create new buffers for outputing, like decoders). There are also some
>> flags in GstBuffer itself that might be useful for you.
>>
>> In any case, associating a Caps to a Buffer is a bad idea as it is
>> already implicitly associated by the previous caps event.
>>
>> Perhaps you could provide more details on your use case so we can provide
>> better guidance.
>>
>>
>>  Thanks,
>> Ben
>>
>> On Thu, Jan 29, 2015 at 10:05 AM, Thiago Santos <thiagoss at osg.samsung.com
>> > wrote:
>>
>>>   On 01/29/2015 11:53 AM, Ben Bridgwater wrote:
>>>
>>>  Hi,
>>> I'm new to gstreamer development - converting an existing application
>>> (interactive spectrogram) from using my own pipeline to use gstreamer
>>> instead.
>>>
>>>  I gather that in gstreamer 0.1 the caps of a pad sourcing a buffer
>>> were also recorded in the buffer itself, but since 1.0 that is no longer
>>> the case, and it's up to elements to tracks caps via caps events instead...
>>>
>>>  My question is what are the options and best practices with gstreamer
>>> 1.x for tracking buffer caps in the case where caps are changing rapidly
>>> and buffers may be processed out of order? In my case this requirement
>>> comes from a downstream display buffer where paint events may request
>>> arbitrary parts of the display to be updated - corresponding to arbitrary
>>> sequences of buffers needing to be processed, and the need to know what the
>>> caps (data format) are for each buffer.
>>>
>>>
>>>  The GstCaps are not stored on buffers anymore, they are only present
>>> on GstPads. Instead of calling gst_pad_set_caps or pushing a buffer with a
>>> caps set, now elements should push a caps event before the buffers and all
>>> buffers following a caps event should have the caps of that event. You can
>>> use GstPad's API to get the current negotiated caps on a pad and all
>>> buffers flowing on that pad should have that caps until a new event
>>> arrives. Check
>>> http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstPad.html#gst-pad-get-current-caps
>>>
>>> So, to keep track of buffers and caps, you need to look for the caps
>>> event. For example, suppose you receive the following events and buffers:
>>>
>>> CAPS_1 BUF_A BUF_B BUF_C CAPS_2 BUF_D BUF_E
>>>
>>> Buffers A B and C will be on format defined by CAPS_1, while buffers D
>>> and E will be on format CAPS_2.
>>>
>>> I hope it makes it clear. If for some reason you need to store buffer +
>>> caps together you can use the GstSample :
>>> https://developer.gnome.org/gstreamer/stable/gstreamer-GstSample.html
>>>
>>>
>>>  Thanks for any pointers and advice!
>>>
>>>  Ben
>>>
>>>
>>>
>>>  _______________________________________________
>>> gstreamer-devel mailing listgstreamer-devel at lists.freedesktop.orghttp://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>>
>>>
>>>
>>> --
>>> Thiago Sousa Santos
>>> Senior Multimedia Engineer, Open Source Group
>>> Samsung Research America - Silicon Valley
>>>
>>>
>>> _______________________________________________
>>> gstreamer-devel mailing list
>>> gstreamer-devel at lists.freedesktop.org
>>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>>
>>>
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing listgstreamer-devel at lists.freedesktop.orghttp://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>>
>>
>> --
>> Thiago Sousa Santos
>> Senior Multimedia Engineer, Open Source Group
>> Samsung Research America - Silicon Valley
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>>
>
>
> _______________________________________________
> gstreamer-devel mailing listgstreamer-devel at lists.freedesktop.orghttp://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
>
> --
> Thiago Sousa Santos
> Senior Multimedia Engineer, Open Source Group
> Samsung Research America - Silicon Valley
>
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20150131/fcba898b/attachment-0001.html>