[gst-devel] Audio visualizations

[Matteo Landi]

On Fri, Aug 6, 2010 at 9:01 PM, Stefan Kost <ensonic at hora-obscura.de> wrote:
> Am 06.08.2010 14:48, schrieb Matteo Landi:
>> On Thu, Aug 5, 2010 at 11:29 PM, Stefan Kost <ensonic at hora-obscura.de> wrote:
>>> Am 27.07.2010 18:53, schrieb Matteo Landi:
>>>> Hi guys,
>>>> these days I was wondering how to use gstreamer in order to create
>>>> some audio visualizations like an oscilloscope or a spectrum analyzer.
>>>>
>>>> The application consist of a window with a drawing area used to
>>>> display the visualizations.
>>>> Regarding the pipeline, I thought about something like
>>>>
>>>> filesrc !
>>>> decodebin !
>>>> audioconvert !
>>>> audio/x-raw-int, channels=1, rate=44100, width=16, depth=16, endianness=1234 !
>>>> tee name=t !
>>>>   queue !
>>>>     fakesink name=fakesink signal-handoffs=true t. !
>>>>   queue !
>>>>     autoaudiosink
>>>>
>>>> with the handoff callback linked to a method which spawn a new thread,
>>>> process the signal, and queue a redraw of the window.
>>>
>>> Have a look at gst-plugins-good/test/examples/spectrum/
>>
>> Thank you, at first I haven't noticed the demo used to display bars
>> values read from the spectrum built in plugin. I was trying to
>> re-implement my own just for the curiosity-sake, nothing else :P
>>
>>>
>>> Anyway, don't draw to gtk-windows from a custom thread. Rather do processing in
>>> a gst element and post the data via GstBus to the main thread and draw there.
>>
>> Indeed, I suppose I was not been too clear while explaining my approach:
>>
>> 1 the fakesink queue a callback linked to the event (this is equal to
>> listening for messages on the bus): the only difference is that I need
>> more time for audio processing (fft) inside that callback
>> 2 invoke a redraw method using gobject.idle_add with the fft data
>> processed at the previous step
>> 3 draw the content on a backup cairo surface and invalidate the window
>> 4 automatically redraw the window (paint the surface) using the
>> generated expose event.
>>
>> The main difference I see with the code you linked [1], is that I
>> don't care at all about all those clocks operations (are they needed?)
>
> The clock stuff just sync the drawing with the playback. Just imagine lots of
> slow elements after the spectrum element.

I imagined it. I thought the sync=true property was enough.

>>
>> I talked about threads, because I was thinking about spawning a new
>> thread for the audio process:
>> 1 fakesink callback which spawns a new thread
>> 2 threaded audio processing
>> 3 gobject.idleadd ..
>> 4 ...
>>
>
> So you want to do the fft yourself, instead of using gstreamers? Have a look at
> http://versia.com/category/spek/ until 0.5 it used gstreamer like I described above.

Thanks, I'll give it a look asap.

>
>> There last one obscure thing: why the demo example does not initialize
>> threads for gobject? I thought it was necessary while working
>> simultaneously with gtk and gst..
>>
> It should, but gst_init does that also.

Reading the docs of gst_init ([1]), it comes out that g_threads_init()
should be invoked explicitly.. that is in contrast with the example
inside the documentation or not?


Regards,


[1] http://www.gstreamer.net/data/doc/gstreamer/head/gstreamer/html/gstreamer-Gst.html#gst-init

>
> Stefan
>
>>>
>>> Stefan
>>>
>>>>
>>>> I got few problems with the implementation:
>>>> 1 - I often get core dumped, and I cannot understand why: all the
>>>> drawing actions are done inside the main thread, so it should be ok
>>>> ...
>>>> 2 - Synchronization between audio and video: the two sources are not
>>>> synchronized.. maybe is it something related to the queues elements?
>>>>
>>>> Have you got any advices regarding the chosen implementation, and how
>>>> to solve the listed problems?
>>>>
>>>> Thanks in advance for the attention.
>>>>
>>>>
>>>
>>>
>>
>> Regards,
>>
>>
>> [1] http://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/tests/examples/spectrum/demo-audiotest.c
>>
>>
>
>


-- 
Matteo Landi
http://www.matteolandi.net/