[Bug 736655] basesink: preroll issue for some clips which audio is shorter than video

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=736655

--- Comment #158 from GstBlub <gstblub at gmail.com> ---
Created attachment 352589
  --> https://bugzilla.gnome.org/attachment.cgi?id=352589&action=edit
queue2: Add buffer-full-time, buffer-full-buffers, buffer-full-size properties

I spent a good amount of time investigating this issue.  I have come to
conclusion that buffering cannot be implemented as documented, without
potentially running into this lockup issue on the state change to PAUSED.  It
does not appear to be a bug, more a deficiency in gstreamer's architectural
design, or assumptions made that shouldn't be made.  I'll explain why, later. 
Here's what an application (in this case an audio player) will have to do to
work around this issue reliably:

1. Create playbin as usual, but do NOT use GST_PLAY_FLAG_BUFFERING.  Note
however, that playbin2 may still emit buffering messages (bug in
gstplaysink.c).
2. Create a bin that contains a queue2 element (let's call it "bufferqueue"). 
This element then is followed by whatever you need, in my case an appsink.  If
playing back (e.g. using some audio sink) you will most likely want the "sync"
property of your sink is set to TRUE.  Be sure to set the "async" property to
FALSE, we do NOT want the state change to PAUSED to be asynchronous (I'll
explain later, why)!  Assign this bin to the "audio-sink" property of the
playbin element.
3. The application should maintain a "target state" that indicates whether the
user wants PLAYING or PAUSED.  This is important because the pipeline state may
be at PAUSED due to buffering, but the buffering logic will need to bring it
back to PLAYING once buffered up, but leave it at PAUSED if the user paused
playback during buffering.  Be sure to handle the case where buffering was
disabled (See #6 and #7.b)
4. Create a "begin_buffering()" function that will set your queue2
("bufferqueue") properties to the following: use-buffering=TRUE,
max-size-buffers=0, max-size-bytes=0, max-size-time=0, buffer-full-time=<some
non-zero value indicating how much decoded audio you want to buffer>.  This
effectively changes the "bufferqueue" to an unbounded queue that will accept as
much data as it is handed, but will report a 100% fill level once the decoded
audio reaches what you set for the buffer-full-time property (attached patch
required).  The queue2 element should be in this state ONLY when the pipeline
is at PAUSED state *AND* it is at PAUSED state due to buffering.
5. Create a "done_buffering()" function that will set your queue2
("bufferqueue") properties to the following: use-buffering=TRUE,
max-size-buffers=0, max-size-bytes=0, max-size-time=<non-zero value indicating
how much decoded audio this queue should, max>, buffer-full-time=0.  This
effectively changes the "bufferqueue" to a bounded queue that will not hold on
to more decoded audio data than max-size-time indicates.  The queue2 element
should be in this state whenever the pipeline is at any state other than
PAUSED, or when it is at PAUSED due to the user requested pausing playback
("target state" indicating PAUSED) and if applicable, buffering completed.
6. Create a "disable_buffering()" function that will set your queue2
("bufferqueue") properties to the following: user-buffering=FALSE,
max-size-buffers=0, max-size-bytes=0, max-size-time=<non-zero value indicating
how much decoded audio this queue should, max>, buffer-full-time=0.  This
effectively changes the "bufferqueue" to a bounded queue that will not hold on
to more decoded audio data than max-size-time indicates, and it will no longer
emit buffering messages.
7. When handling buffering messages, the application needs to do the following:
7.a Check if GST_MESSAGE_SRC() of the buffering message is equal to the queue2
element you created ("bufferqueue"), ignore the message if it's coming from any
other element
7.b You might want to check whether this is a live pipeline (buffer message
indicates GST_BUFFERING_LIVE or the last pipeline state change to PAUSED
returned GST_STATE_CHANGE_NO_PREROLL).  You might want to disable the buffering
logic in this case and call your "disable_buffering()" function.  Be sure to
record that fact so that your application will proceed to PLAYING (See #3) once
at PAUSED, because you're never going to get any more buffering messages.
7.b If your application determines that buffering should begin (e.g. when 0% is
reported and 100% has never been reached), call your "begin_buffering()"
function and take the pipeline to PAUSED.
7.c If your application determines that buffering completed (e.g. when 100% is
reported), call your "done_buffering()" function, and if the user's target
state indicates PLAYING, then take the pipeline back to PLAYING.
8. When initially starting playback, call your "begin_buffering()" function,
and then take the pipeline to PAUSED (so that you can check whether it's a live
pipeline by checking for GST_STATE_CHANGE_NO_PREROLL). Do *NOT* take it
directly to PLAYING.  Once at paused, if the user's "target state" indicates
PLAYING and buffering has not started or been disabled, proceed to PLAYING.

Essentially, what you need is a queue2 element that is unbounded during
buffering and bounded otherwise.  Also, you want this queue2 at the decoding
phase.  Currently, playbin2, uridecodebin, etc have queues that emit buffering
messages, but these messages don't really make sense.  E.g. uridecodebin has a
queue2 that reports buffering of the original source (e.g. souphttpsrc), in my
case that means this queue constantly jumps from 0% to 100% (depending on
source) because it may read a large chunk of data (in one buffer) that
instantly fills the queue2 and then once it pushed that buffer out it is
instantly drained, reporting 0%.  The other problem with playbin2's buffering
is that there is a queue in the playsink that reports buffering based on the
decoded data.  And then there's also a multiqueue that may emit buffering
messages.  And the application gets all of those, but none of these really make
sense (at least not to me).

And here's where the design flaw comes into play.  Let's say you get a
buffering message of 0%.  In my case it was because souphttpsrc is just pushing
a e.g. 1 MB sized buffer of undecoded audio and the 500kb buffer it pushed
before had just drained the queue2 emitting that message).  When the
application takes the pipeline to PAUSED, any push to the sink element will
block due to prerolling.  However, this 1 MB buffer that souphttpsrc is pushing
is now producing a whole bunch of audio data, so it flows into the decoder and
essentially fills up all the queues maintained by playbin2/playsink.  Now the
first buffers are finally making it out of playbin2 and they hit the sink
element, but because it's in PAUSED this push is now blocked.  So it's sitting
there, waiting to goto PLAYING, but because the entire chain (up to and
including the source) is blocked (because all the queues are filled), no more
buffering messages get emitted.  The problem really is that all theses queues
are bounded in size, so they're set to not exceed a certain amount of data,
which is not really a problem unless you're going to PAUSED and hit the
prerolling wait.  Then there's that chance that the entire chain all the way to
the source element is blocked and it can't get out of this state.  The only
solution to solve this situation reliably is a dedicated queue2 for buffering
purposes, and it should probably be operating on the decoded data, which needs
to be unbounded during buffering phases and bounded otherwise.  This allows
everything prior to that queue to at least complete the push during buffering,
regardless of how much data this raw source ballooned into.  Because a
unbounded queue2 could never emit buffering messages with values that make
sense, I had to enhance the logic to allow defining a fill level that ought to
be considered 100%, despite the queue being able to accept more data.  This
unbounded queue2 now may receive a large amount of buffer, but it allows it to
emit a 100% e.g. in the middle of pushing that large 1 MB buffer downstream,
which then kicks everything back to PLAYING, unblocking the preroll wait in the
sink element.

Also, if "async" on the sink element is not set to FALSE then the state changed
from PLAYING back to PAUSED (e.g. due to buffering) is not going to complete
because it expects to be taken to PLAYING again.  This probably could be
handled by the application, but it probably gets complicated/weird.

I hope this all makes sense.  I believe the documentation of how buffering
should be implemented is completely insufficient and will not implement a
buffering strategy that will work reliably.  Depending on source, timing, etc.
it's almost guaranteed you'll run into this "deadlock" unless you implement
buffering as outlined above.

Also, for me, I can make sense of this design problem easiest by thinking about
a source element pushing a large amount of encoded raw data that will get
decoded into a large amount of buffers and/or data (in time), enough to fill up
the entire chain, blocking the push all the way up to the source while in the
prerolling wait on the sink element that is in PAUSED.

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.