[gst-devel] GSmartMix, a smart audio mixer :)

[Marc-André Lureau]

hi,

I am pleased to introduce GSmartMix. GSmartMix has recently been accepted to
Google SoC. It aims is to bring a new sound experience in Gnome. Thus, I
guess this effort should be discussed here.  Ensonic, Stefan Kost, is the
mentor of this project. Raphel Slinckx is the co-mentor.

If you are not already familiar with this project, you could have a
look at http://live.gnome.org/GSmartMix
before reading this.  I will also do my best to change the GNOME sound
support, but *it's not* the main goal. For the moment, GSmartMix is a
project that I want to design to be run aside and optionnaly, without any
API change.

The following are some technical suggestions on GSmartMix, I would like to
discuss with you.

Please note:
GNOME API and esd discussions will go in gnome.multimedia mailing list.
I don't make any difference between rule or class for the moment: it's a
description of the sound: "music", "voice", "event"...

=== UI ===

The ui provide a per class sound level sliders, and some rule parameters:
- foreground sound level
- background sound level
- fading speed
- and list of class that we put in foreground when this class has to play
(queue policy: first in stay in fg (fifo), last in go in fg (lifo), none
(all fg))

I would like to add a new tab in the GNOME "Volume Control" application with
this content: http://live.gnome.org/GSmartMix/TechnicalChoices?action=AttachFile


Regarding the Topaz mockup of volume controler and the per-app volume
control ( http://static.flickr.com/20/70003494_668cfdc0dd.jpg): GSmartMix
benefits will partially fit this. I don't know how we could deal with
"multiple instance of an application" problem... This is why I was briefly
thinking of a Metacity integration, in order to differenciate each sound
source easily. But I am wondering if one "foreground app" per class for
common sounds (voice, video, music..) will help to clean up this list. This
way, only "foreground app" will be shown in the list. Only some class, like
"events" won't need this "front" scheme. We don't care of a per app "event"
sound level probably.Is it really what we want?

=== Sinks: a new element or a gstbaseaudiosink integration ===

I have tried to depict my initial overview of GSmartMix here:
http://live.gnome.org/GSmartMix/TechnicalChoices?action=AttachFile&do=view&target=overview.png

Different design solutions can be thought up. What will be the best or the
consensus?

==== "Base patch" solution ====

The idea is to handle dbus communication at the baseaudiosink class level.
ensonic, how would be handled the volume control/transformation at this
level?
This way, every audiosink will be seen over dbus.

Drawback:
- all compiled audio sinks will have a libdbus dependency (if we compile
gstlib with this support)
- intrusive patch in gst
- every app will have to suffer/benefit the smart level control

==== "New element" solution ====

Formerly, I was thinking of a kind of GStreamer "identity" bin elements,
inserted before the real audio sink. They will catch some gstramer events
(status/streaminfo..) and send them over dbus. After that they will obey
GSmartMix server orders and update a volume level. The "volume" element
manipulated would be found by looking up the parent bin. The
"volume_element" property could be overriden. If none is found, a new one
will be added in GSmartSink bin in order to control volume output. This way,
I hope to be able to provide GSmartMix without having to change anything in
the application side and in GStreamer.

Look at this sketch:
http://photos1.blogger.com/blogger/2015/2884/1600/sec.2.png


Drawback:
- another plugin => a different pipeline
- all applications/usage will not benefit from GSmartMix (those who use
directly alsasink for instance)
- more complex solution?

==== Polypaudio ====

jeff pointed me out Polypaudio. I was not thinking hard of using it, because
it sounded like an odd but good esd replacement. Which is not the aim of my
work. But, I must admit I am quite impressed by its state. It features some
of the required interfaces for my project. Definetly, my will is to fill the
gab between gnome, gstreamer and polypaudio.

Recent discussion with lennart on #polypaudio about GSmartMix:
http://pastebin.com/747392

Drawback (if the effort is to patch polypaudio):
- the volume level adjusted at the server side will not be updated in the
client side (for example, the volume slider of your favourite music player):
I believe I can fix this or is there already a solution in polypaudio? The
polypaudio author, Lennart, will post on this subject, onthe  GStreamer
GstMixer flaw.
- we still depends officialy on a sound server (and not on ALSA/dmix), which
could be badly interpreted

==== Remarks ====

- I believe sinks must initialise and "block" sound output until the dbus
server did not reply to adjust their level.

- of course, I would also like to bring new things to the client side, like
the ability to describe sound, like the ability to give sound "class" and
"priority". Could it be done with standard GStramer API, with a bus message
?

=== GSmartMix, a dbus server ===

I like the simplicity of devil's pie. I wonder if I could improve s-exp
interpreter to support GSmartMix needs (fifo on class, loops on other sinks
?).  So that the server could be written with light C code. If more freedom
is required, Python is probably the language of choice.
But, it could be written differently.

I also wonder how the dbus server could monitor its slaves: the
communication need to be in a "connected" mode with the peer (I think of an
application crash, an abnormal behaviour...).

-- 
Marc-André Lureau, GSmartMix
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.freedesktop.org/archives/gstreamer-devel/attachments/20060530/84df5640/attachment.htm>