[gst-devel] Game-style audio

[Steve Fink]

On 6/1/07, Michal Benes <michal.benes at itonis.tv> wrote:
> Hi,
>
>         I, personally, would use SDL for a game-style audio. From what
> you are describing, you will run into some more trickier parts of
> GStreamer like time synchronization among different samples and dynamic
> pipe rebuilding. But it is difficult to tell, if we do not know what
> exactly should your application do.

I am currently using SDL (or more specifically, SDL_mixer). It's what
I'm trying to get away from. :-) SDL_mixer gets me 80% of the way
there very easily and with a straightforward API. But in exchange,
that last 20% is nearly impossible and would require abusing its
effect functionality. And I've stumbled across at least two bugs so
far (one in another dependent library, one in SDL_mixer). Also,
although I didn't mention it in that last message (it's in an upcoming
one, once I get the basics down a little better), I need video
playback with and without A/V sync, and gstreamer seems very well
suited for that part. So I'd like to use it for everything.

I'm not sure what else would be useful to describe about my
application. You can look at <http://www.reactrix.com/index.php>, but
it's hard to figure out what exactly we're doing from that.

I'm working on an application that takes camera input and produces...
um... "interactive experiences". Anything from minigames to the effect
of rippling water or bouncing balls. As an unkind comparison, I
sometimes refer to it as "a full-body interactive screensaver". The
important thing is that users feel like they're directly influencing
the system and getting immediate feedback. The set of things I want to
be able to do with audio range from:

 - basic fire-and-forget masses of short game-type sound clips
 - pitch-bending to add variety and realism to physics-based collisions
 - pitch-bending to use fewer samples (eg a piano keyboard)
 - fairly long clips with visual effects synchronized to event points
within the sound stream
 - cross-fading different music tracks
 - playing back video with synced audio

Well, I'll keep trying to plow ahead for a while, and see how far I
get. If I come up with any useful plugins for managing this type of
audio, I'll post them here. (The LGPL is very nice for forcing
management to allow me to do open source work during my day job.)