[Spice-devel] Status of spice-html5 and video streaming

[Jeremy White]

Hey all,

I wanted to provide an update on this thread, as I've again spent
considerable time with the issue.

The overall status is slightly improved; you can now play videos for a
while, although you generally lose audio after a period of time.

I have learned several things.  First, a large browser window (e.g.
1500x900) sends draw operations that will take the current spice-html5
client substantial time to draw; on the order of 75 or more ms.  That
long delay then plays havoc with the Media Source Extensions that we are
using.

That is, those mechanisms really do not seem to handle discontinuous
feeds very gracefully.  (I also have a mystery problem with my current
XSpice test setup, where every 30 seconds, I get a 60 ms delay in
transmitting audio samples.  Coincidentally, that shows the problem
nicely as well).

Second, the new gstreamer vp8 codec is not a magic bullet.  Video
streaming with that codec 'feels' a little bit better than regular
mjpeg, but that may just be subjective wishful thinking on my part.  It
largely mitigates the slow draw issue described above, but all it takes
is one slow draw to trigger the audio glitch.  And once the audio is
glitched, it doesn't come back.

My next steps are roughly as follows:
  - Mitigate the long draw operations
    * This could be as simple as studying the EyeOS client to leverage
it's LZ code speed up (and then forcing spice into using LZ)
    * Implementing LZ4 support could be a nice win
  - Determine why the discontinuous feed causes so much heartburn, and
work to mitigate it.  This is worst, and reproducible with audio, so I
plan to focus there and then bring any lessons learned back into the
video handling later.

Cheers,

Jeremy

On 06/05/2015 10:05 AM, Jeremy White wrote:
> Hey all,
> 
> I've spent a lot of time on spice-html5 and video streaming in the past
> week.  I wanted to share the status while it is fresh in my mind, as I'm
> headed off on vacation for a week or so.
> 
> I guess I also have the hope that someone else, expert in Media Source
> Extensions and WebSockets, may have some insight as well.
> 
> Right now, with the very latest commit, Xspice video streaming, with
> sound, 'works', for generous definitions of work.  Smaller videos
> generally work fairly well.  But by the time you get up to a large
> browser window (e.g. > 1024x768), your video will show, with stuttering,
> but you'll lose sound.
> 
> I think I have one more patch that may improve Chrome's audio behavior
> to push it into the generally tolerable range, even with large videos
> (but that is essentially a hack, not a full solution).  Haven't found an
> equivalent for Firefox yet.
> 
> The fundamental issue, as far as I can figure, is that the inbound
> WebSocket traffic flat out overwhelms the browser.  Further, it appears
> as though the inbound request processing takes precedence over other
> browser functions.  It seems that receiving a great deal of inbound
> traffic essentially starves the browser rendering engine.  So getting a
> lot of inbound video frames prevents the frames from rendering, causing
> us to fall further behind - you get the picture.  The just pushed stream
> report does a great deal to mitigate this, for video playback
> 
> For example, even just sending audio (e.g. no video, just playing a
> song), Chrome does not appear to play as fast as it should.  That is, I
> get a 'canplaythrough' signal after 0.2 seconds of data, but
> audio.currentTime does not move off 0 until 0.46 seconds; I think the
> very act of receiving and buffering those additional data packets keeps
> the audio player from playing.  I may toy with trying to send larger
> clumps of audio data to see if that improves that situation.
> 
> I haven't had a chance to work this in with the vp8 code, which I had
> hoped would provide a dramatic improvement.  Unfortunately, the sized
> stream message really creates havoc with vp8 streams, and you really
> can't play a video without the sized stream messages.  See:
>   http://lists.freedesktop.org/archives/spice-devel/2015-May/019948.html
> 
> My thinking is to try to build some metrics around the websocket / media
> source extensions and see if I can prove out some of the apparently
> uncivilized behaviors.   I've also (briefly) experimented with trying to
> use timers to defer work (e.g. trying to get a poor mans yield()), but
> haven't gotten far enough to decide if it's viable or not.  If anyone
> else had insight on the websockets / media source extensions, I'd love
> to hear it.
> 
> Cheers,
> 
> Jeremy
> _______________________________________________
> Spice-devel mailing list
> Spice-devel at lists.freedesktop.org
> http://lists.freedesktop.org/mailman/listinfo/spice-devel
>