[gst-devel] why MPEG fails

[Ronald Bultje]

Hi,

Johan asked me to explain why some MPEG files don't play well in Totem, so
here goes.

MPEG files are a system stream, which contains 1-N elementary streams
(e.g. video, audio). Those streams are cut randomly, so unframed, and
prepended by a small header which contains a timestamp. The timestamp is
used in GStreamer in the GstBuffer, which is sent to the appropriate
decoders (e.g. mpeg2dec or mad).

Totem creates a pipeline from libgstplay which basically is a spider with
video/audiosink attached. All this is a single thread, since spider can
not insert queues. ASCII art:

                                   .src%d ! osssink
filesrc location=file.mpg ! spider
                                   .src%d ! ximagesink

Spider will insert mpegdemux, mpeg2dec and mad, so we get, virtually:

                                       .video_00 ! mpeg2dec ! ximagesink
filesrc location=file.mpeg ! mpegdemux
                                       .audio_00 ! mad ! osssink

There's also conversion elements but those are omitted for clarity. Now,
mpegdemux will spit out a buffers. However, we can never be sure that the
two streams inside the system, streams are actually aligned synchronized
w.r.t. each other. So inside the MPEG stream, this is a valid bytestream
(buffer-stream ;) ):

audio[0.0s] : video[1.0s] : audio[0.1s] : video[1.1s]

Those who know scheduling knows what comes next: mpegdemux will spit out
an audio buffer, mad decodes, sends to osssink, it plays. osssink provides
the master clock, so the clock is updated from 0.0s to 0.1s. Then, it
spits out the video buffer, ximagesink does a gst_element_wait() for 1s,
and displays. During this 1s, nothing happens, because we're
single-threaded! We go back to audio, video (again 1s wait, because the
clock-diff is still 1s between video and audio). The result is a 1fps
movie perception for the user.

The MPEG specifications suggest to queue data to prevent this. One way of
doing that is by using the queue element. This plays fine:

                                      .video_00 ! { queue ! ... }
filesrc location=file.mpg ! mpegdemux
                                      .audio_00 ! { queue ! ... }

Another is to use bufpens between elements, which is what the basic
scheduler does. Therefore, totem --gst-scheduler=basicgthread will plays
those MPEG movies fine, too.

I'm not sure which is better. I just know that we don't want to ship 0.8.0
doing this. :). I suggest switching to basicgthread. It has less bugs,
performs as well and works better in practice. I'll leave that open for
others to comment on, though.

Ronald