[gst-devel] Audio visualizations

[Stefan Kost]

Am 07.08.2010 13:47, schrieb Matteo Landi:
> On Fri, Aug 6, 2010 at 9:01 PM, Stefan Kost <ensonic at hora-obscura.de> wrote:
>> Am 06.08.2010 14:48, schrieb Matteo Landi:
>>> On Thu, Aug 5, 2010 at 11:29 PM, Stefan Kost <ensonic at hora-obscura.de> wrote:
>>>> Am 27.07.2010 18:53, schrieb Matteo Landi:
>>>>> Hi guys,
>>>>> these days I was wondering how to use gstreamer in order to create
>>>>> some audio visualizations like an oscilloscope or a spectrum analyzer.
>>>>>
>>>>> The application consist of a window with a drawing area used to
>>>>> display the visualizations.
>>>>> Regarding the pipeline, I thought about something like
>>>>>
>>>>> filesrc !
>>>>> decodebin !
>>>>> audioconvert !
>>>>> audio/x-raw-int, channels=1, rate=44100, width=16, depth=16, endianness=1234 !
>>>>> tee name=t !
>>>>>   queue !
>>>>>     fakesink name=fakesink signal-handoffs=true t. !
>>>>>   queue !
>>>>>     autoaudiosink
>>>>>
>>>>> with the handoff callback linked to a method which spawn a new thread,
>>>>> process the signal, and queue a redraw of the window.
>>>>
>>>> Have a look at gst-plugins-good/test/examples/spectrum/
>>>
>>> Thank you, at first I haven't noticed the demo used to display bars
>>> values read from the spectrum built in plugin. I was trying to
>>> re-implement my own just for the curiosity-sake, nothing else :P
>>>
>>>>
>>>> Anyway, don't draw to gtk-windows from a custom thread. Rather do processing in
>>>> a gst element and post the data via GstBus to the main thread and draw there.
>>>
>>> Indeed, I suppose I was not been too clear while explaining my approach:
>>>
>>> 1 the fakesink queue a callback linked to the event (this is equal to
>>> listening for messages on the bus): the only difference is that I need
>>> more time for audio processing (fft) inside that callback
>>> 2 invoke a redraw method using gobject.idle_add with the fft data
>>> processed at the previous step
>>> 3 draw the content on a backup cairo surface and invalidate the window
>>> 4 automatically redraw the window (paint the surface) using the
>>> generated expose event.
>>>
>>> The main difference I see with the code you linked [1], is that I
>>> don't care at all about all those clocks operations (are they needed?)
>>
>> The clock stuff just sync the drawing with the playback. Just imagine lots of
>> slow elements after the spectrum element.
> 
> I imagined it. I thought the sync=true property was enough.

Setting sync=true on the sink is making the playback synced. In the example we
sync drawing to our custom graphs the screen to the playback time.

> 
>>>
>>> I talked about threads, because I was thinking about spawning a new
>>> thread for the audio process:
>>> 1 fakesink callback which spawns a new thread
>>> 2 threaded audio processing
>>> 3 gobject.idleadd ..
>>> 4 ...
>>>
>>
>> So you want to do the fft yourself, instead of using gstreamers? Have a look at
>> http://versia.com/category/spek/ until 0.5 it used gstreamer like I described above.
> 
> Thanks, I'll give it a look asap.
> 
>>
>>> There last one obscure thing: why the demo example does not initialize
>>> threads for gobject? I thought it was necessary while working
>>> simultaneously with gtk and gst..
>>>
>> It should, but gst_init does that also.
> 
> Reading the docs of gst_init ([1]), it comes out that g_threads_init()
> should be invoked explicitly.. that is in contrast with the example
> inside the documentation or not?

As I said, the examples should do that. I will improve the spectrum example soon
(already did that in buzztard).

Stefan

> 
> 
> Regards,
> 
> 
> [1] http://www.gstreamer.net/data/doc/gstreamer/head/gstreamer/html/gstreamer-Gst.html#gst-init
> 
>>
>> Stefan
>>
>>>>
>>>> Stefan
>>>>
>>>>>
>>>>> I got few problems with the implementation:
>>>>> 1 - I often get core dumped, and I cannot understand why: all the
>>>>> drawing actions are done inside the main thread, so it should be ok
>>>>> ...
>>>>> 2 - Synchronization between audio and video: the two sources are not
>>>>> synchronized.. maybe is it something related to the queues elements?
>>>>>
>>>>> Have you got any advices regarding the chosen implementation, and how
>>>>> to solve the listed problems?
>>>>>
>>>>> Thanks in advance for the attention.
>>>>>
>>>>>
>>>>
>>>>
>>>
>>> Regards,
>>>
>>>
>>> [1] http://cgit.freedesktop.org/gstreamer/gst-plugins-good/tree/tests/examples/spectrum/demo-audiotest.c
>>>
>>>
>>
>>
> 
>