[Bug 733959] hlsdemux: download bitrate alogoritmus don't reflect real download rate

[GStreamer (GNOME Bugzilla)]

https://bugzilla.gnome.org/show_bug.cgi?id=733959

--- Comment #20 from Duncan Palmer <dpalmer at digisoft.tv> ---

I've prototyped something along the following lines which is (I think) a lot
simpler than the queueing ideas we've discussed. It implements the following;

    - The kernel and possibly libsoup have buffers which fill continually when
we're retrieving a HLS segment. The combined size of these buffers is
upstream_bufsize.
    - hlsdemux has a local buffer of size hlsdemux_bufsize, where
hlsdemux_bufsize > upstream_bufsize.
    - We can measure bitrate by measuring how long it takes to fill the portion
of the hlsdemux buffer between offsets upstream_bufsize and hlsdemux_bufsize.
We're not attempting to measure how long it takes to download a segment, but
simply measure bitrate.

The prototype uses a GstBuffer which is filled with data arriving in
_src_chain() until it becomes full. The contents of the buffer are then pushed
downstream.

    - The download_start_time variable is set when the buffer level increases
above upstream_bufsize.
    - The download_total_bytes variable is incremented by hlsdemux_bufsize -
upstream_bufsize (so we don't take account of all data received, only the
portion which we should have had to wait to download).
    - In the EOS handler for the src, we push out anything remaining in the
GstBuffer, but do not update download_total_bytes or download_total_time.

This prototype implementation breaks when we use HLS content with small
segments, as the GstBuffer will never fill before EOS. This can be solved in a
final implementation by measuring how long it takes to download the entire
segment for segments which are < hlsdemux_bufsize. This works because we never
get blocked pushing downstream with these small segments, as there'll be one
push only, on src handler EOS.

There is no way to determine what upstream_bufsize is at runtime. It is the
size of the socket receive buffer (which I believe encompasses the TCP receive
window) plus whatever buffering libsoup may introduce. Without hacking on
libsoup, we can't find out this information. So, I think the easiest option is
to introduce a property to set it. I've experimented with values of 128k and
256k (the prototype assumes hlsdemux_bufsize == upstream_bufsize * 2) on my
STB. Both work well, I do see some evidence of the original problem when using
128k, however this is small enough that I could live with it. The correct value
will also vary depending how much memory the machine has, as I believe this
effects the actual socker readbuffer sizes, unless they are explicitly set.

Testing (quick and dirty so far)

    - Throttle the connection to 5120 kbit/s using dripls.
    - Play back. The initial variant chosen is the first (and highest in the
case of my test content), requiring about 10000 kbit/s.
    - Using the original bitrate calculation algorithm, the bitrate is
calculated at about 10000 kbit/s after downloading this fragment. About 3
seconds is spent pushing data downstream, and 2 seconds waiting for data to
arrive from souphttpsrc.
    - Using the prototype, the bitrate is calculated at about 4800 kbit/s.
    - Playback for 90 seconds using the prototype, and observe that the bitrate
is always about right.

An interesting observation from the testing is that the time taken to push
buffers downstream were consistently smaller with the prototype. I presume this
is because we were dealing in < 4K buffers previously, whereas we're now
dealing with 256K buffers, and so are not pushing the first multiqueue into
overrun so quickly. The logs show evidence to support this, but I can't say for
sure.

I'd appreciate comments on this approach. I'll upload a patch later tonite, or
tomorrow. Currently, what I have doesn't apply cleanly to 1.4, or at all to
master (I'm working on a 1.4 based branch).

-- 
You are receiving this mail because:
You are the QA Contact for the bug.
You are the assignee for the bug.