[Spice-devel] Trying to turn on gstreamer vaapi {en,de}coding

Uri Lublin ulublin at redhat.com
Mon Nov 28 13:04:10 UTC 2022 

Hi,

On Sun, Nov 27, 2022 at 4:04 PM Victor Toso <victortoso at redhat.com> wrote:

> Hi,
> On Fri, Nov 25, 2022 at 12:32:57PM +0000, Lévai, Dániel wrote:
> > Hi all!
> >
> > First time here, and was a bit hesitant to post tbh, as this is
> > not a "-devel" sort of thing, but reading through the posts
> > here it seamed there have been some technical questions
> > already, so I'm trying my luck. The post is a bit longwinded,
> > it has a bunch of debug outputs and parameters in-line, please
> > let me know if you prefer sharing these in another way.
>
> It is fine to post it here.
>
> > I'm using a couple of VMs with libvirt/qemu hosted on an Ubuntu
> > 22.04.1 LTS, accessing them via Spice from an Arch system. The
> > connection between Hypevisor and client are via Internet, so
> > despite 'localhost' being the host in the tests/examples below,
> > they don't actually run on the same system, it's just a tunnel
> > via `stunnel`.
>
> > What I wanted to do is speed up the display (drawing) so it
> > won't lag that much when browsing through spreadsheets and
> > using console (with small fonts, admittedly). The resolution
> > also plays a role in this at 2560x1440 (only one spice
> > display).
>
> Note that host video encoding will lag a bit. This is what you
> are trying. It should be better with spice-streaming-agent by
> setting up a video card to the guest, an example:
>
>
> https://www.spice-space.org/demos.html#accelerated-spice-streaming-with-intel


>
> > What I've tried already is changing the spice section in the domain
> configuration in libvirt:
> >     <graphics type='spice' port='5912' autoport='no' listen='127.0.0.1'>
> >       <listen type='address' address='127.0.0.1'/>
> >       <image compression='auto_glz'/>
> >       <jpeg compression='never'/>
> >       <zlib compression='never'/>
> >       <streaming mode='all'/>
> >       <filetransfer enable='yes'/>
> >       <gl enable='no'/>
> >     </graphics>
>
> > Which translates into this -spice qemu parameter:
> > -spice
> port=5912,addr=127.0.0.1,disable-ticketing=on,image-compression=auto_glz,jpeg-wan-compression=never,zlib-glz-wan-compression=never,streaming-video=all,seamless-migration=on
>
> > ... with the complete command as follows:
> > /usr/bin/qemu-system-x86_64 -name guest=arch,debug-threads=on -S -object
> {"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-22-arch/master-key.aes"}
> -machine
> pc-q35-4.2,usb=off,vmport=off,dump-guest-core=off,memory-backend=pc.ram
> -accel kvm -cpu
> Cooperlake,ss=on,vmx=on,pdcm=on,hypervisor=on,tsc-adjust=on,sha-ni=on,umip=on,waitpkg=on,gfni=on,vaes=on,vpclmulqdq=on,rdpid=on,movdiri=on,movdir64b=on,fsrm=on,md-clear=on,avx-vnni=on,xsaves=on,ibpb=on,ibrs=on,amd-stibp=on,amd-ssbd=on,hle=off,rtm=off,avx512f=off,avx512dq=off,avx512cd=off,avx512bw=off,avx512vl=off,avx512vnni=off,avx512-bf16=off,taa-no=off
> -m 16384 -object
> {"qom-type":"memory-backend-ram","id":"pc.ram","size":17179869184}
> -overcommit mem-lock=off -smp 4,sockets=4,cores=1,threads=1 -no-user-config
> -nodefaults -chardev socket,id=charmonitor,fd=38,server=on,wait=off -mon
> chardev=charmonitor,id=monitor,mode=control -rtc base=utc,driftfix=slew
> -global kvm-pit.lost_tick_policy=delay -no-hpet -no-shutdown -global
> ICH9-LPC.disable_s3=1 -global ICH9-LPC.disable_s4=1 -boot strict=on -device
> pcie-root-port,port=16,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x2
> -device pcie-root-port,port=17,chassis=2,id=pci.2,bus=pcie.0,addr=0x2.0x1
> -device pcie-root-port,port=18,chassis=3,id=pci.3,bus=pcie.0,addr=0x2.0x2
> -device pcie-root-port,port=19,chassis=4,id=pci.4,bus=pcie.0,addr=0x2.0x3
> -device pcie-root-port,port=20,chassis=5,id=pci.5,bus=pcie.0,addr=0x2.0x4
> -device pcie-root-port,port=21,chassis=6,id=pci.6,bus=pcie.0,addr=0x2.0x5
> -device pcie-root-port,port=22,chassis=7,id=pci.7,bus=pcie.0,addr=0x2.0x6
> -device pcie-root-port,port=23,chassis=8,id=pci.8,bus=pcie.0,addr=0x2.0x7
> -device qemu-xhci,p2=15,p3=15,id=usb,bus=pci.2,addr=0x0 -device
> virtio-serial-pci,id=virtio-serial0,bus=pci.3,addr=0x0 -blockdev
> {"driver":"file","filename":"/var/lib/libvirt/images/archlinux.qcow2","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}
> -blockdev
> {"node-name":"libvirt-2-format","read-only":false,"driver":"qcow2","file":"libvirt-2-storage","backing":null}
> -device
> virtio-blk-pci,bus=pci.4,addr=0x0,drive=libvirt-2-format,id=virtio-disk0,bootindex=1
> -device ide-cd,bus=ide.0,id=sata0-0-0 -netdev
> tap,fd=39,id=hostnet0,vhost=on,vhostfd=41 -device
> virtio-net-pci,netdev=hostnet0,id=net0,bus=pci.1,addr=0x0 -chardev
> pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0
> -chardev socket,id=charchannel0,fd=35,server=on,wait=off -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0
> -chardev spicevmc,id=charchannel1,name=vdagent -device
> virtserialport,bus=virtio-serial0.0,nr=2,chardev=charchannel1,id=channel1,name=com.redhat.spice.0
> -device usb-tablet,id=input0,bus=usb.0,port=1 -device
> virtio-keyboard-pci,id=input2,bus=pci.7,addr=0x0 -device
> usb-mouse,id=input3,bus=usb.0,port=4 -audiodev
> {"id":"audio1","driver":"spice"} -spice
> port=5912,addr=127.0.0.1,disable-ticketing=on,image-compression=auto_glz,jpeg-wan-compression=never,zlib-glz-wan-compression=never,streaming-video=all,seamless-migration=on
> -device virtio-vga,id=video0,max_outputs=1,bus=pcie.0,addr=0x1 -device
> ich9-intel-hda,id=sound0,bus=pcie.0,addr=0x1b -device
> hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0,audiodev=audio1 -chardev
> spicevmc,id=charredir0,name=usbredir -device
> usb-redir,chardev=charredir0,id=redir0,bus=usb.0,port=2 -chardev
> spicevmc,id=charredir1,name=usbredir -device
> usb-redir,chardev=charredir1,id=redir1,bus=usb.0,port=3 -device
> virtio-balloon-pci,id=balloon0,bus=pci.5,addr=0x0 -object
> {"qom-type":"rng-random","id":"objrng0","filename":"/dev/urandom"} -device
> virtio-rng-pci,rng=objrng0,id=rng0,bus=pci.6,addr=0x0 -sandbox
> on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny
> -msg timestamp=on
>
> > Basically, I thought turning 'streaming mode'/'streaming-video'
> > to 'all' would make spice think it needs to stream the whole
> > screen content regardless of its heuristics detecting a video
> > stream.
>
> I'm replying by memory here, so I might be a little off. But just
> to add a bit more info. The streaming mode set to 'all' means
> that all regions being updated too often in the Display are
> included in one video stream that encapsulate those regions. The
> 'filter' mode, iirc, would split the different regions in
> different video streams.
>
> The whole screen streaming only happens if needed, even with
> mode set to 'all'.
>

Only areas on the screen that are recognized as video are sent as
video-stream, not
the whole screen.
IIRC, "all" is for all areas that are recognized, while "filter" does not
stream small areas.
Every area/region has its own video-stream.

What you (Daniel) want is for qemu (+spice-server) to, at intervals, render
the
whole screen, hardware-encode it and send it as a video stream to the
client.
Currently, this feature is not implemented.
I think some years ago Frediano wrote some code that does that as an
experiment. IIRC, there was an issue with the gstreamer ORC library
running in the environment libvirt creates for the qemu-kvm process.


Regards,
    Uri


> > I thought if I could enable HW encoding/decoding with
> > these options, I'd get better performance out of it.
>
> Yeah, that's a fair assumption. The main issue in general is with
> latency that video encoding adds, be it software or hardware.
>
> <snipped, package information>

>
> > As part of troubleshooting here, I checked if I have the
> > necessary codecs/encoders/decoders with gstreamer on the
> > different systems:
> >
> > On the hypervisor host (with Intel Alderlake (12th gen) graphics):
> >
> > As my unrpivileged user:
> > $ gst-inspect-1.0 |grep -F -i vaapi
> > vaapi:  vaapiav1dec: VA-API AV1 decoder
> > vaapi:  vaapidecodebin: VA-API Decode Bin
> > vaapi:  vaapih264dec: VA-API H264 decoder
> > vaapi:  vaapih264enc: VA-API H264 encoder
> > vaapi:  vaapih265dec: VA-API H265 decoder
> > vaapi:  vaapih265enc: VA-API H265 encoder
> > vaapi:  vaapijpegdec: VA-API JPEG decoder
> > vaapi:  vaapijpegenc: VA-API JPEG encoder
> > vaapi:  vaapimpeg2dec: VA-API MPEG2 decoder
> > vaapi:  vaapimpeg2enc: VA-API MPEG-2 encoder
> > vaapi:  vaapioverlay: VA-API overlay
> > vaapi:  vaapipostproc: VA-API video postprocessing
> > vaapi:  vaapisink: VA-API sink
> > vaapi:  vaapivc1dec: VA-API VC1 decoder
> > vaapi:  vaapivp9dec: VA-API VP9 decoder
> > vaapi:  vaapivp9enc: VA-API VP9 encoder
> >
> > As the libvirt user that runs qemu:
> > # sudo -u libvirt-qemu gst-inspect-1.0 |grep -F -i vaapi
> > vaapi:  vaapiav1dec: VA-API AV1 decoder
> > vaapi:  vaapidecodebin: VA-API Decode Bin
> > vaapi:  vaapih264dec: VA-API H264 decoder
> > vaapi:  vaapih264enc: VA-API H264 encoder
> > vaapi:  vaapih265dec: VA-API H265 decoder
> > vaapi:  vaapih265enc: VA-API H265 encoder
> > vaapi:  vaapijpegdec: VA-API JPEG decoder
> > vaapi:  vaapijpegenc: VA-API JPEG encoder
> > vaapi:  vaapimpeg2dec: VA-API MPEG2 decoder
> > vaapi:  vaapimpeg2enc: VA-API MPEG-2 encoder
> > vaapi:  vaapioverlay: VA-API overlay
> > vaapi:  vaapipostproc: VA-API video postprocessing
> > vaapi:  vaapisink: VA-API sink
> > vaapi:  vaapivc1dec: VA-API VC1 decoder
> > vaapi:  vaapivp9dec: VA-API VP9 decoder
> > vaapi:  vaapivp9enc: VA-API VP9 encoder
> >
> > On the client (with Intel CometLake (10th gen) graphics):
> > As my user running the client (spicy/remote-viewer):
> > $ gst-inspect-1.0 |grep -F -i vaapi
> > vaapi:  vaapidecodebin: VA-API Decode Bin
> > vaapi:  vaapih264dec: VA-API H264 decoder
> > vaapi:  vaapih264enc: VA-API H264 encoder
> > vaapi:  vaapih265dec: VA-API H265 decoder
> > vaapi:  vaapih265enc: VA-API H265 encoder
> > vaapi:  vaapijpegdec: VA-API JPEG decoder
> > vaapi:  vaapijpegenc: VA-API JPEG encoder
> > vaapi:  vaapimpeg2dec: VA-API MPEG2 decoder
> > vaapi:  vaapimpeg2enc: VA-API MPEG-2 encoder
> > vaapi:  vaapioverlay: VA-API overlay
> > vaapi:  vaapipostproc: VA-API video postprocessing
> > vaapi:  vaapisink: VA-API sink
> > vaapi:  vaapivc1dec: VA-API VC1 decoder
> > vaapi:  vaapivp8dec: VA-API VP8 decoder
> > vaapi:  vaapivp8enc: VA-API VP8 encoder
> > vaapi:  vaapivp9dec: VA-API VP9 decoder
> >
> > (a good common ground could be VP9 as the host can encode and
> > the client can decode, or h264/h265 as both can encode/decode)
>
> There are a lot of settings that you'll likely need to tweak in
> spice server (host) for good optimization and even hardware
> encoding. You can see at [0] that the plugin for video encoding
> is hardcoded with x264enc. I can't see if you have it or not in
> your host because of the grep, but that could be another reason
> for not being able to encode in h264 (edit: seems to be present
> in the logs)
>
> [0]
> https://gitlab.freedesktop.org/spice/spice/-/blob/master/server/gstreamer-encoder.c#L887
>
> The spice-gtk (client) uses GStreamer's most generic elements to
> give it the possibility to setup a video decoding in the best way
> possible, preferably with hardware encoded although that was not
> guaranteed when I implemented that logic.
>
> > As an example of using vaapi, tested with mpv (--vo=gpu):
>

<snipped, it works>


> > This works fine, while also can check the GPU utilization with
> > intel_gpu_top - the /Video/ metric keeps changing during
> > playback.
> >
> > So I'm testing with `spicy` first and foremost because it
> > actually has a menu option to switch between video encodings,
> > which I don't know how to do with e.g. remote-viewer or
> > virt-viewer.
> >
> > $ GST_DEBUG=4 spicy --spice-debug -p 5912 -h localhost
> >
> > Long story short, while the client output shows that gstreamer
> > picks up its vaapi plugin, and actually loads the intel iHD DRI
> > driver, it still doesn't use the GPU to decode.
>
> > I'm afraid something lies underneath the configuration and the
> > protocol that I don't understand. Maybe the host doesn't even
> > encode its stream into e.g. h264 even though the client
> > (=spicy) requests so, therefore it can't very well decode it
> > with the GPU.
>
> > I imagined something along the lines of Spice encoding the
> > stream on the host with the help of the GPU (gstreamer + vaapi)
> > and then the client could decode it on the other end with its
> > GPU as well - as far as they agree on a codec that both support
> > (more specifically the host supports encoding to it and the
> > client suppports decoding from it).
> >
> > I was afraid of posting the full output with --spice-debug,
> > becuase it had a ton of "decode-glz.c:335 decode_header:" and
> > "scroll event" lines, so I snipped these out from it.
>
>
<snipped, log lines>

> If you have any suggestions as to what I am doing wrong or what
> > I'm missing, I'd be greatful if you'd let me know.
>
> From what I could see, you have set the h264 video-codec
> preference at 13:06:46 and terminated spicy around 13:06:49.
>
> So, perhaps the first issue is the assumption that as soon as you
> set the video-codec preference, that the stream will start. It'll
> not. The streaming will happen based on update frequency on the
> display. For example, a clock updating the seconds, once every
> second, will not create a video streaming. You should try running
> an App that has high update frequency or just a video.
>
> GStreamer needs to be enabled in spice-server at build time too.
> You can check with ldd in libspice-server.so...
>
> So, the logs from spice-server would be useful to know what issue
> might have happened at encoding if it doesn't work.
>
> Just trying to give you some ideas and directions to go.
>
> Cheers,
> Victor
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/spice-devel/attachments/20221128/be26e295/attachment-0001.htm>

More information about the Spice-devel mailing list