How to track buffer caps in gstreamer 1.x ?
Thiago Santos
thiagoss at osg.samsung.com
Thu Jan 29 19:06:40 PST 2015
On 01/29/2015 02:35 PM, Ben Bridgwater wrote:
> Thanks Thiago - I appreciate your responsiveness!
>
> I'm at the planning stage of a port-to-gstreamer project, so I'm
> trying to understand it better before I make any final design decisions.
>
> My application is an interactive spectrogram with a Qt GUI that let's
> the user modify FFT (spectrogram) parameters (which will mean
> gstreamer caps changes) on-the-fly as audio data is streaming. I
> support both real-time streaming (e.g. from a microphone source) as
> well as manual audio file "browsing" where the user can use a scroll
> bar to seek through the audio file with the visible portion being
> pushed thru the processing pipeline in response to scroll movement. My
> out-of-order buffer processing requirement comes from the Qt GUI
> which maintains a display buffer of the stream chunks (buffers) it has
> received from the pipeline, and may need to render them (needing data
> format/caps) in any order per the Qt paintEvents it receives.
You can store the buffers with caps in GstSamples, you don't need to
copy the caps or the buffer, just keep a ref to them in the GstSample
and the overhead will be on the allocation of a struct, which shouldn't
be that much.
If you don't need to pass this buffer around you can also have some
other custom way of identifying the format internally in your element if
you believe performance would be improved.
>
> The application currently uses my own pipeline/element design, but I
> want to port to gstreamer in order to support a wider variety of audio
> sources as well as spectrogram-synced audio playback capability.
>
> Right now the GUI is the final sink of my pipeline (audio-source ->
> audio-resampling -> audio-preemphasis -> FFT -> Qt-signal-adapter ->
> Qt GUI), but in the future I will be feeding the output into a machine
> learning program, and so would like to make my FFT/analysis component
> a greamer element rather than appsink in order to retain flexibility.
> I may also split this "FFT" component into multiple elements as it
> also does some compute-heavy FFT post-processing that could benefit
> from being multithreaded via gstreamer queue elements.
Good idea, it will make it more reusable.
>
> In the port to gstreamer, I'm planning to implement the GUI-driven FFT
> parameter changes via a caps filter inserted after the FFT element,
> via which I can set the desired caps and force the FTT to adapt. The
> downstream Qt GUI will therefore receive a stream of caps change
> events and buffers and per the out-of-order processing requirement I
> need an efficient way to map from buffers to corresponding caps/data
> format. I'm thinking that if I could add a "caps ID" tag to the
> GstBuffers (metadata or reuse some existing struct member) then I
> could maintain a corresponding "caps ID" to caps map in the GUI to
> make the association.
You don't necessarily need the capsfilter, your element can change the
requirements on its pads dynamically and push a 'reconfigure' event
upstream to notify other elements that they should try to renegotiate to
another format. This is essentially what the capsfilter would be doing
when you set a new caps to it.
>
> Hopefully this explains what I'm trying to do. Any suggestions or
> advice are is very welcome, especially regarding the out-of-order
> buffer-to-caps mapping.
Hope it helps, success on your porting!
>
> Ben
>
>
> On Thu, Jan 29, 2015 at 11:46 AM, Thiago Santos
> <thiagoss at osg.samsung.com <mailto:thiagoss at osg.samsung.com>> wrote:
>
> On 01/29/2015 12:52 PM, Ben Bridgwater wrote:
>> Thanks, Thiago.
>>
>> Yes, in my application (because of the out-of-order buffer
>> processing) I do need to explicitly associate buffers with caps
>> rather than just rely on the caps event and buffer ordering.
> What is your application exactly? Are you using appsink, fakesink
> or a custom sink? Or is this another type of element?
>>
>> I assume gstreamer pipeline elements can't actually pass
>> GstSamples rather than GstBuffers - this is just a convenience
>> class for a sink to associate buffers/caps/segments rather than a
>> type that would be passed from element to element?
> Yes, only buffers are passed through pads.
>>
>> It seems another option for me might be to use GstBuffer metadata
>> to add the tagging/caps info I need. Would I be right in thinking
>> that an element could receive a GstBuffer, add some metadata,
>> then pass the GstBuffer with added metadata down the pipleline?
>> Would an existing element/plugin care if it received a buffer
>> with added metadata that it wasn't expecting? Presumably adding
>> metadata involves an additional malloc?
>>
>> Is there any way to add any metadata (maybe just an integer tag)
>> to a GstBuffer without doing an additional alloc? Any GstBuffer
>> members than it would be harmless to repurpose as a tag, perhaps?
> There is GstMeta that is a flexible metadata system that can be
> associated with buffers. You need to make sure that other elements
> in the pipeline preserve the meta when passing buffers around
> (specially the ones that create new buffers for outputing, like
> decoders). There are also some flags in GstBuffer itself that
> might be useful for you.
>
> In any case, associating a Caps to a Buffer is a bad idea as it is
> already implicitly associated by the previous caps event.
>
> Perhaps you could provide more details on your use case so we can
> provide better guidance.
>
>>
>> Thanks,
>> Ben
>>
>> On Thu, Jan 29, 2015 at 10:05 AM, Thiago Santos
>> <thiagoss at osg.samsung.com <mailto:thiagoss at osg.samsung.com>> wrote:
>>
>> On 01/29/2015 11:53 AM, Ben Bridgwater wrote:
>>> Hi,
>>> I'm new to gstreamer development - converting an existing
>>> application (interactive spectrogram) from using my own
>>> pipeline to use gstreamer instead.
>>>
>>> I gather that in gstreamer 0.1 the caps of a pad sourcing a
>>> buffer were also recorded in the buffer itself, but since
>>> 1.0 that is no longer the case, and it's up to elements to
>>> tracks caps via caps events instead...
>>>
>>> My question is what are the options and best practices with
>>> gstreamer 1.x for tracking buffer caps in the case where
>>> caps are changing rapidly and buffers may be processed out
>>> of order? In my case this requirement comes from a
>>> downstream display buffer where paint events may request
>>> arbitrary parts of the display to be updated - corresponding
>>> to arbitrary sequences of buffers needing to be processed,
>>> and the need to know what the caps (data format) are for
>>> each buffer.
>>
>> The GstCaps are not stored on buffers anymore, they are only
>> present on GstPads. Instead of calling gst_pad_set_caps or
>> pushing a buffer with a caps set, now elements should push a
>> caps event before the buffers and all buffers following a
>> caps event should have the caps of that event. You can use
>> GstPad's API to get the current negotiated caps on a pad and
>> all buffers flowing on that pad should have that caps until a
>> new event arrives. Check
>> http://gstreamer.freedesktop.org/data/doc/gstreamer/head/gstreamer/html/GstPad.html#gst-pad-get-current-caps
>>
>> So, to keep track of buffers and caps, you need to look for
>> the caps event. For example, suppose you receive the
>> following events and buffers:
>>
>> CAPS_1 BUF_A BUF_B BUF_C CAPS_2 BUF_D BUF_E
>>
>> Buffers A B and C will be on format defined by CAPS_1, while
>> buffers D and E will be on format CAPS_2.
>>
>> I hope it makes it clear. If for some reason you need to
>> store buffer + caps together you can use the GstSample :
>> https://developer.gnome.org/gstreamer/stable/gstreamer-GstSample.html
>>
>>>
>>> Thanks for any pointers and advice!
>>>
>>> Ben
>>>
>>>
>>>
>>> _______________________________________________
>>> gstreamer-devel mailing list
>>> gstreamer-devel at lists.freedesktop.org <mailto:gstreamer-devel at lists.freedesktop.org>
>>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>>
>> --
>> Thiago Sousa Santos
>> Senior Multimedia Engineer, Open Source Group
>> Samsung Research America - Silicon Valley
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org
>> <mailto:gstreamer-devel at lists.freedesktop.org>
>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>>
>>
>>
>>
>> _______________________________________________
>> gstreamer-devel mailing list
>> gstreamer-devel at lists.freedesktop.org <mailto:gstreamer-devel at lists.freedesktop.org>
>> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
> --
> Thiago Sousa Santos
> Senior Multimedia Engineer, Open Source Group
> Samsung Research America - Silicon Valley
>
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> <mailto:gstreamer-devel at lists.freedesktop.org>
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
>
>
>
>
> _______________________________________________
> gstreamer-devel mailing list
> gstreamer-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/gstreamer-devel
--
Thiago Sousa Santos
Senior Multimedia Engineer, Open Source Group
Samsung Research America - Silicon Valley
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20150130/9fa3fbdb/attachment-0001.html>
More information about the gstreamer-devel
mailing list