High delay of video-streams

[Michael Scherle]

Hello spice developers,

we are trying to develop an Open Source virtual desktop infrastructure 
to be deployed at multiple German universities as described, by my 
colleagues, in the paper which I have put in the attachment. The 
solution based on openstack, qemu, spice... Our plan is also to have VM 
instances with virtual GPUs (SR-IOV). Due to the resulting requirements, 
it is necessary to transmit the image data as a video stream.
We have seen Vivek Kasireddy recent work on spice which solves exactly 
this problem. However, when we tested it, we noticed a very high input 
to display delay (400 ms+ but only if the image data is transferred as 
video-stream). However, the problem seems to be a more general spice 
problem or is there something wrong with our setup or are there special 
parameters that we are missing?

Our setup:

QEMU: https://gitlab.freedesktop.org/Vivek/qemu/-/commits/spice_gl_on_v2
Spice: https://gitlab.freedesktop.org/Vivek/spice/-/commits/encode_dmabuf_v6
virt-viewer
Intel HW decoder/encoder (but same with sw)

I have looked into what is causing the delay and have noticed that 
encoding only takes about 3-4ms. In general, the image seems to reach 
the client in less than 15ms.
The main problem seems to be that gstreamer gets a very high 
margin(https://gitlab.freedesktop.org/spice/spice-gtk/-/blob/master/src/channel-display.c?ref_type=heads#L1773) 
and therefore waits a long time before starting decoding. And the reason 
for the high margin seems to be the bad mm_time_offset 
https://gitlab.freedesktop.org/spice/spice-gtk/-/blob/master/src/spice-session.c?ref_type=heads#L2418 
which is used to offset the server time to the client time (with some 
margin). And this variable is set by the spice server to initially 400 
ms 
https://gitlab.freedesktop.org/spice/spice/-/blob/master/server/reds.cpp?ref_type=heads#L3062 
and gets updated with the latency 
https://gitlab.freedesktop.org/spice/spice/-/blob/master/server/reds.cpp?ref_type=heads#L2614 
(but only increased). I still need to see how this latency is calculated.

Am I missing something or is this design not intended for transmitting 
interactive content via video stream?
Temporarily overwriting the margin and tweaking parameter settings on 
the msdkh264dec brought the delay to about 80-100ms, which is not yet 
optimal but usable. To see what is technical possible on my setup, I 
made a comparison using moonlight/sunshine which resulted in an delay of 
20-40ms.

Our goal is to achieve some round trip time similar to the 
moonlight/sunshine scenario to achieve a properly usable desktop experience.

Greetings
Michael
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Cloud__OpenStack_VDI-1.pdf
Type: application/pdf
Size: 676164 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/spice-devel/attachments/20240315/f69e1201/attachment-0001.pdf>