[Spice-devel] [spice-server v2 8/9] reds: add support to ranks for video codecs

[Francois Gouget]

On Tue, 13 Dec 2016, Victor Toso wrote:

> Hi,
> 
> Thanks again for your feedback.
> 
> On Tue, Dec 13, 2016 at 08:06:50PM +0100, Francois Gouget wrote:
> > So your complexity objection is about the format in which to send the
> > client's codec preferences to the server: you want to send it as an
> > array rather than as a single string.
> 
> No. "gstreamer" is not a video-codec, it is an *encoder*.

Of course. Let me rephrase.

At this point I don't care whether you send the client's preferences as 
{"vp8", "h264", "mjpeg"} or as {{"vp8", 1}, {"h264", 1}, {"mjpeg", 2}} 
 or as "vp8;h264;mjpeg". That's an implementation detail.

Fortunately I think you let slip a valid reason for your proposal:

[...]
> Client says that prefers vp8, maybe due hw decoding capability.


So let's build a proper case for your proposal:

There are two cases where the codec used to encode the video really 
matters to the client:

1) Mobile devices such as smartphones and tablets

   These usually have processors that would struggle to keep up with 
   software decoding. Furthermore software decoding is more power 
   inefficient and would quickly run down their battery. For these two 
   reasons it is important that they be able to get video in a format 
   compatible with their hardware decoding capabilities whenever 
   possible.

   I would also note that there is no client for these platforms (that I 
   know of), but the Spice protocol should not be an obstacle to 
   implementing a good client for these platforms.

2) Laptops operating on battery power

   Laptops usually have processors that are powerful enough to perform 
   software decoding without trouble. That may change as new codecs are 
   added that require more processing power for decoding although if 
   past history is any indicator processors quickly catch up (but 
   it may be unwise to bet against a plateauing of CPU speeds).

   Another factor is that for laptops too hardware decoding is more 
   power efficient. So even if they could do software decoding, ensuring 
   they get video in a format compatible with their hardware 
   capabilities would help increase their battery life.

So how can the client get a say in the selection of the video codec 
while at the same time letting the server administrator be in charge 
too?

There are two requirements:

1) The client must have a way of expressing preferences.
   The current capability based system does not allow that.

2) Since the server preferences override those of the client, they 
   must be able to give the same priority to multiple codecs so the 
   client preferences can be used to break these ties.


And now, with the rationale firmly established, we can get into the 
implementation details.


Server side
-----------

On the server side the simplest solution is to extend the current 
preferences system by adding a rank, as you proposed, so we have a 
semi-colon separated list of encoder:codec:rank triplets. The higher the 
rank the lower the preference for the codec.

There are a few details to decide on however:

1) What should the highest rank be: zero or one? 

   Under the current system any encoder:codec that is not in the 
   preferences list will not be used and for backward compatibility this 
   should be preserved.

   In your proposal an encoder:codec with a rank of 0 is to be disabled 
   which is the same as not putting it in the list in the first place. 
   This seems redundant and I usually consider redundancy to be bad as 
   it typically means unnecessary complexity.

   Do we have a scenario where we would need to be able to put an 
   encoder:codec in the preferences string but still want it to not be 
   used?

   If not then should the highest rank be 0 rather than 1?

2) What should the lowest rank be?

   In you proposal you argue for the rank to be a single digit. This 
   saves us from having to deal with crazy inputs such as 
   "12345678901234567890". However a single digit seems like a pretty 
   small space and it's not really much easier to parse than the 
   multiple digits case.

   Should we just say it's any integer we can parse? Or a two digit 
   number?

3) If one sets the preferences as "gstreamer:h264:2;spice:mjpeg", what 
   should the rank of "spice:mjpeg" be?

   I argue that it should be the lowest rank (i.e. something like 9 or 
   99 or whichever value we pick in point 2).

4) Should wa call this a rank or a priority?

   The problem with calling it a rank is that the entry with the highest 
   numerical value is the one least likely to be used. This can be 
   confusing.

   As proof that it is confusing, I initially used the "highest rank" 
   term for both point 1 (which is about 0 vs. 1) and point 2 (which is 
   about the other end: 9, 99 or more).

   Would calling it a priority be better? That would imply flipping 
   things over so 99 actually has a higher priority than 2. Then 
   "spice:mjpeg" could get priority 0, i.e. the lowest priority.


Client side
-----------

On the client side what we need is:

1) A new Spice message to send the client preferences since the 
   capabilities system has no way of ordering capabilities.

2) A format to express the client preferences.

The format could be a simple semi-colon separated list of codecs since 
there is no need for ranks on the client side. Or it could be an array 
of strings to save on parsing if that easy to handle within the Spice 
protocol.

It could also be some format that includes ranks if that helps us reuse 
code. However this would mean exposing implementation details on the 
client side which we could get stuck with. So it may be best to avoid.

Can the client send a message expresing its codec preferences to a 
server that does not support said message? Does this require adding a 
server capability?

Then what should we do with the current video capabilities?

It seems like we have to keep them for backward compatibility but 
presumably we will not add new capabilities for VP9, h265, etc?

-- 
Francois Gouget <fgouget at codeweavers.com>