[Spice-devel] Status of spice-html5 and video streaming

[Jeremy White]

Hey all,

I've spent a lot of time on spice-html5 and video streaming in the past
week.  I wanted to share the status while it is fresh in my mind, as I'm
headed off on vacation for a week or so.

I guess I also have the hope that someone else, expert in Media Source
Extensions and WebSockets, may have some insight as well.

Right now, with the very latest commit, Xspice video streaming, with
sound, 'works', for generous definitions of work.  Smaller videos
generally work fairly well.  But by the time you get up to a large
browser window (e.g. > 1024x768), your video will show, with stuttering,
but you'll lose sound.

I think I have one more patch that may improve Chrome's audio behavior
to push it into the generally tolerable range, even with large videos
(but that is essentially a hack, not a full solution).  Haven't found an
equivalent for Firefox yet.

The fundamental issue, as far as I can figure, is that the inbound
WebSocket traffic flat out overwhelms the browser.  Further, it appears
as though the inbound request processing takes precedence over other
browser functions.  It seems that receiving a great deal of inbound
traffic essentially starves the browser rendering engine.  So getting a
lot of inbound video frames prevents the frames from rendering, causing
us to fall further behind - you get the picture.  The just pushed stream
report does a great deal to mitigate this, for video playback

For example, even just sending audio (e.g. no video, just playing a
song), Chrome does not appear to play as fast as it should.  That is, I
get a 'canplaythrough' signal after 0.2 seconds of data, but
audio.currentTime does not move off 0 until 0.46 seconds; I think the
very act of receiving and buffering those additional data packets keeps
the audio player from playing.  I may toy with trying to send larger
clumps of audio data to see if that improves that situation.

I haven't had a chance to work this in with the vp8 code, which I had
hoped would provide a dramatic improvement.  Unfortunately, the sized
stream message really creates havoc with vp8 streams, and you really
can't play a video without the sized stream messages.  See:
  http://lists.freedesktop.org/archives/spice-devel/2015-May/019948.html

My thinking is to try to build some metrics around the websocket / media
source extensions and see if I can prove out some of the apparently
uncivilized behaviors.   I've also (briefly) experimented with trying to
use timers to defer work (e.g. trying to get a poor mans yield()), but
haven't gotten far enough to decide if it's viable or not.  If anyone
else had insight on the websockets / media source extensions, I'd love
to hear it.

Cheers,

Jeremy