[Spice-devel] On channel priority

[Frediano Ziglio]

> 
> Hi,
> 
> I started looking at "webdav: copying big files makes the session
> unusable" bug https://gitlab.freedesktop.org/spice/spice-gtk/issues/7,
> and had a few ideas worth sharing. Of course, one of the problem with
> this is how to measure the benefit when doing any additional tweaking.
> 
> The actual design and implementation of Spice (client & server) is
> that each channel has its own socket and IO is done asynchronously.
> Some channels have additional socket tweaking. But I don't think we
> have a general reference or guideline on how prioritization is being
> done. Afaik, there is no packet priority ordering in client & server
> code, however there is some flow control in various channels (ack
> count, agent tokens, char-device tokens, display/cursor max-pipe-size)
> which may not be enough to throttle the channels and give enough
> bandwidth to others channels, but may influence the fairness &
> interactivity experience.
> 
> So lower than Spice itself, I think the following settings are
> influencing channel prioritization:
> - glib source priority (a context iteration dispatch highest priority
> sources)

Not much to solve what a proper QoS would do. The reason is that
these priorities affects mainly CPU bound work but the QoS on network
means that we are network bound instead. It is true that events with
more priority would be run before other events but as the sockets
are asynchronous and we are network bound simply the high priorities
sockets will be fill their buffer first, events will be turned off
on high priority events and on next loop iteration sockets with lower
priority will queue data too. As we are network bound this will affect
probably network queueing less than expected.

> - socketopt TCP_CORK (manual control of buffering)
> - socketopt TCP_NODELAY (no buffering)

These are designed and used more for packet fragmentation reducing,
not QoS.

> - socketopt SO_PRIORITY (linux socket queuing priority)

Would be good to check what this setting is achieving, I think
default Linux queuing respect the socket priority.

> - socketopt IP_TOS (the Type-Of-Service, lowdelay etc)

We use for sound which is expected to be low bandwidth but
we want it to have less latency. Note that setting a socket
with the wrong TOS (for instance setting low delay on display
which is surely high bandwidth) cause the connection to get
quite bad limiting the usage of this flag.

> - thread priority (not really, since we use single-thread design)

display/cursor are run on different thread, but similar argument
could be raised for glib priorities, these settings affects more
CPU bound usage.

> 
> For various reasons, glib source priority is not being used at the moment.
> But socketopt are being used:
> - TCP_CORK: used by server on display channel for manual buffering
> - TCP_NODELAY: used on all channels in spice-gtk. server code is more
> tricky to follow, but reds_init_client_connection() sets it to TRUE by
> default. There is some tweaking in sound channel based on bandwidth,
> which I am not convinced really is worth.

Not for QoS.

> - SO_PRIORITY: used by spice server on sound channel with maximum
> non-admin value (6).

Surely worth testing this.

> - IPTOS_LOWDELAY: used by spice server on sound channel

Didn't to measurement but I think is worth.

> 
> So, in general, there isn't much socket/io prioritization, except on
> sound channels in the server side. Because of the variety of channels,
> devices, and use cases, it is hard to generalize the priorities. But I
> think we could try better. I'll try to review each channel and make
> some educated guess:
> 
> MainChannel
> - control channel & data transfer (file & c&p)
> - bandwidth mixed, high-latency okay
> - migration: should be handled first
> - suggest: lowering priority, queuing prioritisation within channel
> could be worthwhile for data
> 
> DisplayChannel
> - mostly server driven
> - high bandwidth, low-latency preferable, except for streams?
> - migration: high priority
> - suggest: normal priority, queuing prioritisation within channel
> could be worthwhile for streams. client prioritisation could be higher
> for stream QoS reports
> 
> InputsChannel
> - mostly client driven
> - low bandwidth, low-latency
> - migration: high priority (higher than display?)
> - suggest: highering priority in client side
> 
> CursorChannel
> - server driven
> - low bandwidth, low-latency
> - migration: high priority
> - suggest: highering priority
> 
> PlaybackChannel
> - server driven
> - medium bandwidth, low-latency
> - migration: high priority (higher than display?)
> - suggest: current server settings? + glib source priority
> 
> RecordChannel
> - server control, client data
> - medium bandwidth, low-latency
> - migration: high priority (higher than display?)
> - suggest: current server settings? + glib source priority
> 
> TunnelChannel
> - deprecated
> 
> SmartcardChannel
> - client driven
> - low bandwidth, high latency ok
> - migration: low priority
> - suggest: lowering priority
> 
> UsbredirChannel:
> - data channel, but depends on the device being redirected
> - suggest by default: highering priority for interrupt devices
> (input), lowering priority for bulk devices (hdd). Normal priority for
> isochronous devices (webcam & audio I think). Eventually, add a way to
> specify channel priority per device being redirected.
> 
> PortChannel:
> - marginal/experimental: it depends on the use case
> 
> WebDAVChannel:
> - data channel
> - high bandwidth, high latency ok
> - migration: low priority
> - suggest: lowering priority
> 

About migration mainly you marked all high priority beside smartcard and
webdav which I don't think they have much data to move on migration.

I would keep low delay for sound channels, human ears are able to
distinguish delays in the order of ms (1/2) on sounds (I know that
we use queueing and buffers but reducing the latency won't hurt).

Agreed to try to lower webdav/smartcard priority. Not sure what's
the SO_PRIORITY settings (not saying is the only way to do QoS).

Not sure about main channel, too many stuff there (from settings,
to clipboard/files to other management stuff), I would need a full
list.

> If we allow tweaking priority on the client side, it would make sense
> to communicate it to the server, so that both side use similar
> settings (for example, for USB devices). For now, I think we can try
> to improve the situation with reasonable defaults.
> 
> I have started modifying spice-gtk to do some experimentations, but I
> thought you may have other ideas or point of view. I'd also like to
> gather ideas for how to make measurements.
> 

I remember Victor did some work on QoS too and have some patches too.

> Thanks
> 
> 
> --
> Marc-André Lureau

Frediano