[Spice-devel] [spice-server v2 8/9] reds: add support to ranks for video codecs

Wed Dec 14 07:53:49 UTC 2016

Hi,

You rock. I'll use this as reference for future proposals, many thanks!

On Wed, Dec 14, 2016 at 04:10:55AM +0100, Francois Gouget wrote:
> > Client says that prefers vp8, maybe due hw decoding capability.
>
> So let's build a proper case for your proposal:
>
> There are two cases where the codec used to encode the video really
> matters to the client:
>
> 1) Mobile devices such as smartphones and tablets
>
>    These usually have processors that would struggle to keep up with
>    software decoding. Furthermore software decoding is more power
>    inefficient and would quickly run down their battery. For these two
>    reasons it is important that they be able to get video in a format
>    compatible with their hardware decoding capabilities whenever
>    possible.
>
>    I would also note that there is no client for these platforms (that
>    I know of), but the Spice protocol should not be an obstacle to
>    implementing a good client for these platforms.
>
> 2) Laptops operating on battery power
>
>    Laptops usually have processors that are powerful enough to perform
>    software decoding without trouble. That may change as new codecs
>    are added that require more processing power for decoding although
>    if past history is any indicator processors quickly catch up (but
>    it may be unwise to bet against a plateauing of CPU speeds).
>
>    Another factor is that for laptops too hardware decoding is more
>    power efficient. So even if they could do software decoding,
>    ensuring they get video in a format compatible with their hardware
>    capabilities would help increase their battery life.

Power consumption and decoding efficiency, valid reasons.

Each codec might have different overall quality and bandwidth usage and
client might prefer one to another because of that too (which might or
not be related to decoding efficiency).

> So how can the client get a say in the selection of the video codec
> while at the same time letting the server administrator be in charge
> too?
>
> There are two requirements:
>
> 1) The client must have a way of expressing preferences.
>    The current capability based system does not allow that.
>
> 2) Since the server preferences override those of the client, they
>    must be able to give the same priority to multiple codecs so the
>    client preferences can be used to break these ties.
>
>
> And now, with the rationale firmly established, we can get into the
> implementation details.
>
>
> Server side
> -----------
>
> On the server side the simplest solution is to extend the current
> preferences system by adding a rank, as you proposed, so we have a
> semi-colon separated list of encoder:codec:rank triplets. The higher the
> rank the lower the preference for the codec.
>
> There are a few details to decide on however:
>
> 1) What should the highest rank be: zero or one?
>
>    Under the current system any encoder:codec that is not in the
>    preferences list will not be used and for backward compatibility this
>    should be preserved.
>
>    In your proposal an encoder:codec with a rank of 0 is to be disabled
>    which is the same as not putting it in the list in the first place.
>    This seems redundant and I usually consider redundancy to be bad as
>    it typically means unnecessary complexity.

The spice-server API is spice_server_set_video_codecs() and so we
consider that any encoder:codec pair that are not present should be
disabled.

I would prefer that encoder:codec pair that are not present would
receive the default rank/priority value which would mean *not* disabled,
but less likely to be used; To disable a encoder:codec, it should be
explicit required to be said so.

The reason for that is that application implementing spice-server will
need to be changed every time we include a new video-codec capability if
the application wants to support it otherwise it will be disabled as
its no present in the input string of spice_server_set_video_codecs()

>    Do we have a scenario where we would need to be able to put an
>    encoder:codec in the preferences string but still want it to not be
>    used?

Internally we use a GArray of encoder/video-codecs and as a detail of
implementation, I think it is much better to have a GArray with all
video-codecs with their ranks set accordingly.

>
>    If not then should the highest rank be 0 rather than 1?

I got a bit confused with "What should the highest rank be: zero or one"
I first read I thought you mean s/highest/lowest a 0 < 9 :) - But I'll
keep commenting following the nomination you gave.

I think this redundancy is not a bad one so I would go for 0 to
disabled, 1 being the default/highest rank.

> 2) What should the lowest rank be?
>
>    In you proposal you argue for the rank to be a single digit. This
>    saves us from having to deal with crazy inputs such as
>    "12345678901234567890". However a single digit seems like a pretty
>    small space and it's not really much easier to parse than the
>    multiple digits case.
>
>    Should we just say it's any integer we can parse? Or a two digit
>    number?

One digit means 10 values. User can use 9 values to set their preference
in the host... I would say it is more then enough and very easy to
extend if necessary.

>
> 3) If one sets the preferences as "gstreamer:h264:2;spice:mjpeg", what
>    should the rank of "spice:mjpeg" be?
>
>    I argue that it should be the lowest rank (i.e. something like 9 or
>    99 or whichever value we pick in point 2).

9 would be higher priority then 2. So, the default should be 1.

> 4) Should wa call this a rank or a priority?

Either is fine but one important point to make is that the value
attached to them and how they matter, my proposal is:

0: disabled
1: default - (lowest priority possible)
2: higher priority then 1
...
9: highest priority.

>    The problem with calling it a rank is that the entry with the highest
>    numerical value is the one least likely to be used. This can be
>    confusing.

Yeah, I think we both are confused at this point :)

I will use the priority term as it seems easier to understand:
- 9 has higher priority then 1
- 9 has higher rank then 1

>    As proof that it is confusing, I initially used the "highest rank"
>    term for both point 1 (which is about 0 vs. 1) and point 2 (which is
>    about the other end: 9, 99 or more).
>
>    Would calling it a priority be better? That would imply flipping
>    things over so 99 actually has a higher priority than 2. Then
>    "spice:mjpeg" could get priority 0, i.e. the lowest priority.
>
>
> Client side
> -----------
>
> On the client side what we need is:
>
> 1) A new Spice message to send the client preferences since the
>    capabilities system has no way of ordering capabilities.
>
> 2) A format to express the client preferences.
>
> The format could be a simple semi-colon separated list of codecs since
> there is no need for ranks on the client side. Or it could be an array
> of strings to save on parsing if that easy to handle within the Spice
> protocol.

As we already have SpiceVideoCodecType [0] and we only want to state the
order of preference, I would go with an array as I did with these enum
as value.

[0] https://cgit.freedesktop.org/spice/spice-protocol/tree/spice/enums.h#n147

In the server we will simply sort the existing GArray in a new array. It
is also easier to validate too; Using string does not reduce the need of
parsing IMHO.

> It could also be some format that includes ranks if that helps us reuse
> code. However this would mean exposing implementation details on the
> client side which we could get stuck with. So it may be best to avoid.

Yes :) - My first approach was using ranks in the client side but it was
bad.

> Can the client send a message expresing its codec preferences to a
> server that does not support said message? Does this require adding a
> server capability?

It requires adding capability

> Then what should we do with the current video capabilities?
>
> It seems like we have to keep them for backward compatibility but
> presumably we will not add new capabilities for VP9, h265, etc?

The video-codec preference message and capability does not exclude the
need of per video-codec capability. We will keep including video-codec
capability in server and client.

I think we will be able to keep backward compatibility for everything
related to the video-codec preference :)

Cheers,
  toso
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/spice-devel/attachments/20161214/ba28475b/attachment.sig>