Multimedia support (Re: X Developers' Summit)

[Helge Bahmann]

Am Montag, 16. Juli 2007 13:20 schrieb Colin Guthrie:
> This sounds interesting. Have you heard of/do you use internally
> PulseAudio? http://pulseaudio.org/ It has good support for network
> transparent sound and fairly reasonable support with end user
> applications (including alsa plugin and esd compatible daemon
> support).... It would be good to utilise a compatible interface if at
> all possible....

Sounds interesting, have not seen it before (I know arts, jack, esd and mas 
reasonably well), but I will have a look into it. But what I am trying to do 
is fundamentally different in at least two ways:

- audio data is routed through the X protocol; no separate audio connection, 
no separate audio daemon (*); I *think* that any system using a separate 
audio daemon will have a very hard time providing synchronisation with video 
(unless you move *all* multimedia to the daemon, bypassing X entirely; 
however if you do this, you end up duplicating a lot of X functionality for 
images...)
- imperative audio processing primitives instead of a flow-graph based 
approaches most other audio systems use (from a quick browsing through the 
documentation I gather that PulseAudio is flow-graph based)

To illustrate the last point a bit better... I am introducing "samplebuffers" 
(akin to pixmaps/pictures) as a new server-side resource to, well, store 
audio samples; primitive operations allow to composite and transform data 
between different samplebuffers (akin to drawing operations); two other 
resource classes "playbackcontext" and "recordingcontext" operate on the data 
contained in samplebuffers to drive DAC/ADC (vaguely akin to visuals).

Audio processing is performed by the client actively issuing audio compositing 
commands instead of building up a server-side transformation flow graph. Yes, 
this means it is the clients responsibility to ensure that commands are 
executed in time. The approach is thus *considerably* more low-level than a 
flow-graph based system, and IMHO an imperative programming model better 
mirrors the existing X graphics architecture.

I am sorry that I am just takling about it without you being able to look at 
the code, but I promise it will be cleaned up and released in at most 2-3 
weeks' time :)

(*) Yes my PoC implementation currently has real-time issues with current 
(single-threaded) xorg server. None of the problems are unfixable, and I 
think it is worthwhile to do it -- which is the main reason I would like to 
get in touch with you X developers to see if you agree (and of course 
convince you otherwise if you don't :)

Best regards
Helge Bahmann
-- 
Mathematicians stand on each other's shoulders while computer scientists stand 
on each other's toes.
-- Richard Hamming