[gst-devel] totem and osssink?

[Ronald S. Bultje]

Hi Benjamin & all,

osssink has some issues. I recently tested totem (assuming it worked,
since everyone was very enthusiastic about it)...

Some issues that I currently see, in no particular order:
* osssink would start playing noise as soon as an underrun occurred.
This has been this way for several weeks (given the code), but I only
was notified of this several days ago, and fixed it yesterday. Casting
issue from signed (negative) to unsigned.
* osssink doesn't survive seeking. Someone removed the discont event
handler and - how surprisingly - didn't put anything back in return.
Just added that back.
* A/V sync issues. This one is harder to solve, although the problem can
be explained in a few sentences. We currently use sound as a master
clock and video as slave. Sound will play and will try to keep to the
incoming buffer timestamps if possible. Video follows by waiting on the
sound clock. Issue one is if video lags. There is now way for audio to
know this, since video exposes no clock. It only exposes element timing,
but that's not available to other elements. The new clocking system was
supposed to solve it, but doesn't. By that time, the queue element had a
limit of 1s in buffers, which means that video could lag at worst 1s x
max(num_queues_in_audio_part, num_queues_in_video_part), assuming we're
using one source (demuxer). Reason is fairly simple. Right now, someone
(?) changed the queue to a max. of no time and 256 buffers (which can
get up to several gig of video, btw), but which is also a lot of audio
(mp3, for example, is approximately 0.03s per buffer, so this is roughly
10 seconds) which causes video to allow to lag up to ten times as much.
Indeed, if I heat up my CPU to 100% use, the video lags far more than
1-2 seconds (15? 20?) before sound starts skipping. This change to queue
needs to be reverted. If there was any specific reason, that user should
use properties to increase queue's bufsize.
* as soon as one buffer (audio) is dropped at full queues and CPU of
100%, it becomes unfixable. Reason is probably my lack of understanding
(see next point). osssink will believe that we are lagging buffers, and
will thus drop a buffer ('throw frame' part in chain()). We can, after
this, either update the audio clock or not. If we don't, then we will
after a while get timestamps that fit our clock's old time. then we
update the clock to the current time (gst_audio_clock_set_time()) and
we're back to the previous situation. What's worse is that - since we
don't update our clock - video might be waiting for our clock, which is
a deadlock. It will wait 'videotime - audiotime' until displaying the
next frame (clocking magic - just assume this), which gets one fps unit
worse per displayed frame and soon leads to a stall in video display.
Since queues are filled, it doesn't solve itself. Since audio lags, too,
the audio clock never gets updated, so that doesn't solve it either. In
the case where we do update the clock, we lose sound for the rest of the
movie because we will never actually *decrease* our time difference
between audio buffers and audio clock. The deeper issue is that I don't
know how to synchronize or change the time of the audio clock here after
a discont or framedrop. Someone please help me here.
* Probably the main issue: I don't know how clocking works. We have
several time units in osssink, one being the element clock, one being
the audio clock, one being the oss clock, one being the buffer
timestamps, one being the elementtime and one being the buffertime (see
chain()). I have no f***ing clue how those relate to each other or what
each of those represents, and I cannot fix osssink if nobody tells me. I
need documentation. Benjamin, please. Explain what you did here, what's
what. Especially audio clock vs. oss clock vs. element clock and which
does what. And how - according to you - A/V sync and timing should be
done, for the element itself, other elements and applications. The code
does *not* speak for itself.

I can hopefully get totem going (sort-of) before 0.8.0. It'd be nice if
people would either make the dependencies of totem some less modern (I
don't want to run CVS gtk/glib/gnome just for one app) or if people
would be a bit more verbose on bugs. Developers included. :).

Ronald

Btw, a gst-launch pipeline playing a movie (divx5,mp3,full-PAL) takes
+/- 75% CPU. The same pipeline in player takes >100%. Where does this
overhead come from?

-- 
Ronald Bultje <rbultje at ronald.bitfreak.net>
Linux Video/Multimedia developer