[Spice-devel] [Qemu-devel] Vioserial of Windows guest OS on Qemu 0.15

Vadim Rozenfeld vrozenfe at redhat.com
Mon Mar 19 05:02:55 PDT 2012


On Monday, March 19, 2012 01:16:56 PM Charles.Tsai-蔡清海-研究發展部 wrote:
> Vadim,
> 
> What do you mean "qemu configuration string"? Is it the build configuration
> or command line option? Please let me know it. Thanks.
> 
> 
Sorry, that was ambiguous.
I need the command line in order to reproduce the problem.

Cheers,
Vadim.

> 
> 
> -----Original Message-----
> From: Vadim Rozenfeld [mailto:vrozenfe at redhat.com]
> Sent: Monday, March 19, 2012 5:17 PM
> To: Charles.Tsai-蔡清海-研究發展部
> Cc: Michael Roth; Stefan Hajnoczi; spice-devel at lists.freedesktop.org; Alex
> Huang-黃必賢-研究發展部; Alon Levy; qemu-devel; Paul Lu-盧偉智-研究發展部 Subject: Re:
> [Qemu-devel] [Spice-devel] Vioserial of Windows guest OS on Qemu 0.15
> 
> On Monday, March 19, 2012 08:29:41 AM Charles.Tsai-蔡清海-研究發展部 wrote:
> > Hi Vadim,
> > 
> > 	We recently encountered with many issues with new Windows Virio
> > 
> > driver running on Qemu 1.0 and Qemu 1.0.50. My initial investigation
> > was found that the new Viro driver could not be installed  before Qemu
> > 1.0.50. But my colleague right now can install the new Virio driver on
> > Qemu 1.0. Such a finding is quite a bit surprise to me.
> 
> Does Windows refuse to install a new driver or just stuck?
> 
> > 	When we tested the new Virio driver on both Qemu 1.0 and 1.0.50, we
> > 
> > found the Virio driver failed to work after disabling and enabling the
> > driver a few times. However, the failure case largely depends on the
> > type of machine. On my machine, I did not see such a problem.
> 
> Are you running SMP or UP guest? What about others?
> 
> > 	Here are a few of questions that I want to clarify for us.
> > 	
> > 	1. Can the new VirIO driver be installed prior Qemu 1.0.50(the
> > 
> > current head branch)?
> 
> Yes, as far as I know.
> 
> > 2. Does your QA team verify the new VirtIo driver prior Qemu 1.0.50?
> > Do you recommend us to do such a test or debugging?
> 
> Our QA test our drivers on top of RHEL only. I don't know exactly what is
> the difference between our latest qemu-kvm and upstream
> 
> > 	3. When we disable and enable the new VirtIO driver several times on
> > 
> > either Qemu 1.0 or Qemu 1.0.50, VirtIo driver failed to work after
> > enabling the new VirtIO driver. What we found in this particular case
> > is that the VirtIO driver was blocked inside the DPC routine where the
> > driver tried to ACK the interrupt. Based on my colleague's 		  
> > observation,
> 
> the
> 
> > I/O read from DPC was not trapped by the front-end driver(a Linux
> > driver implemented for VirtIO)? Do you have any idea why this could
> > happen? Also, did you 	   	   find similar issues by you or your QA?
> 
> I know for sure that we have at least one race condition described here:
> https://bugzilla.redhat.com/show_bug.cgi?id=750773
> Hence, my question is whether your guest is SMP or UP?
> 
> > 	Please let us know these as soon as possible since we have a big
> > 
> > trouble to make the Virio driver(new and old driver) work stably in
> > any Qemu official release including Qemu 1.0.50.
> 
> I'm going to push some changes, mostly related to the above bug.
> The DPC problem is completely new for me. Please post your qemu
> configuration string and I'll try to reproduce it and investigate this
> problem.
> 
> Best regards,
> Vadim.
> 
> > -----Original Message-----
> > From: Vadim Rozenfeld [mailto:vrozenfe at redhat.com]
> > Sent: Wednesday, February 08, 2012 3:47 PM
> > To: Charles.Tsai-蔡清海-研究發展部
> > Cc: Michael Roth; Stefan Hajnoczi; spice-devel at lists.freedesktop.org;
> > Alex
> > Huang-黃必賢-研究發展部; Alon Levy; qemu-devel; Paul Lu-盧偉智-研究發展部 Subject: Re:
> > [Qemu-devel] [Spice-devel] Vioserial of Windows guest OS on Qemu 0.15
> > 
> > I'm afraid to give you any recommendations regarding QEMU stability.
> > I usually work with qemu-kvm repository. But we are always doing WHQL
> > process on RHEL platform only.
> > 
> > Maybe someone from the SPICE team can comment on your question?
> > 
> > Best regards,
> > Vadim.
> > 
> > ----- Original Message -----
> > From: "Charles.Tsai-蔡清海-研究發展部" <charles.tsai at cloudena.com>
> > To: "Vadim Rozenfeld" <vrozenfe at redhat.com>
> > Cc: "Michael Roth" <mdroth at linux.vnet.ibm.com>, "Stefan Hajnoczi"
> > <stefanha at gmail.com>, spice-devel at lists.freedesktop.org, "Alex
> > Huang-黃必賢-研究發展部" <alex.huang at cloudena.com>, "Alon Levy"
> > <alevy at redhat.com>, "qemu-devel" <qemu-devel at nongnu.org>, "Paul
> > Lu-盧偉智-研究發展部" <paul.lu at cloudena.com> Sent: Wednesday, February 8, 2012
> > 4:39:22 AM
> > Subject: RE: [Qemu-devel] [Spice-devel] Vioserial of Windows guest OS
> > on Qemu 0.15
> > 
> > Vadim,
> > 
> > Thanks. I tested Windows guest OS driver the over qemu-kvm-devel
> > 1.0.50 from git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git and it
> > worked. The latest Windows guest driver from
> > (https://github.com/YanVugenfirer/) seems to have to work with Qemu
> > 1.0.50. My previous test ran over Qemu 1.0 and that was why it failed to
> > work.
> > 
> > As you mentioned, the latest driver from
> > (https://github.com/YanVugenfirer/) is much more stable than the old one.
> > But those drivers must work with Qemu 1.0.50 which was not verified to
> > be stable. Do you think Qemu 1.0.50 is stable to be used? Please give
> > us a recommendation.
> > 
> > 
> > -----Original Message-----
> > From: Vadim Rozenfeld [mailto:vrozenfe at redhat.com]
> > Sent: Wednesday, February 08, 2012 3:06 AM
> > To: Charles.Tsai-蔡清海-研究發展部
> > Cc: Michael Roth; Stefan Hajnoczi; spice-devel at lists.freedesktop.org;
> > Alex
> > Huang-黃必賢-研究發展部; Alon Levy; qemu-devel; Paul Lu-盧偉智-研究發展部 Subject: Re:
> > [Qemu-devel] [Spice-devel] Vioserial of Windows guest OS on Qemu 0.15
> > 
> > I have tried to reproduce the problem several times without success.
> > 
> > QEMU:
> > QEMU 1.0.50 from git://git.qemu.org/qemu.git ,and qemu-kvm-devel
> > 1.0.50 from git://git.kernel.org/pub/scm/virt/kvm/qemu-kvm.git
> > 
> > My guest info:
> > OS Name:                   Microsoft Windows 7 Enterprise
> > OS Version:                6.1.7600 N/A Build 7600
> > 
> > QEMU command line:
> > sudo
> > /home/vrozenfe/work/upstream/qemu/x86_64-softmmu/qemu-system-x86_64 -m
> > 1024 -smp 1,sockets=1,cores=1,threads=1 -name SERIAL -uuid
> > d9388815-ddd3-c38e-33c2-a9d5fcc7a775 -monitor stdio -rtc
> > base=localtime -device
> > virtio-serial-pci,id=virtio-serial0,bus=pci.0,multifunction=on,addr=0x
> > 5.0x
> > 0 -drive
> > file=/home/vrozenfe/work/images/win7.qcow2,if=none,id=drive-ide0-0-0,f
> > orma
> > t=qcow2 -device
> > ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=
> > 1
> > -netdev tap,id=hostnet0 -device
> > e1000,netdev=hostnet0,id=net0,mac=22:3A:40:3F:2F:12,bus=pci.0,multifun
> > ctio
> > n=on,addr=0x3.0x0 -chardev pty,id=charserial0 -device
> > isa-serial,chardev=charserial0,id=serial0 -chardev
> > spicevmc,id=charchannel0,name=vdagent -device
> > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=chann
> > el0,
> > name=com.redhat.spice.0 -usb -device usb-tablet,id=input0 -device
> > virtio-balloon-pci,id=balloon0,bus=pci.0,multifunction=on,addr=0x4.0x0
> > -cdrom /home/vrozenfe/work/virtio-win-1.4.0.iso -snapshot -enable-kvm
> > 
> > Drivers:
> > Both, MS signed from RHEL6.2 virtio-win RPM, as well as compiled by
> > myself with the sources from our internal repository (which contains
> > absolutely the same code as in Yan's repository).
> > 
> > 
> > 
> > ----- Original Message -----
> > From: "Charles.Tsai-蔡清海-研究發展部" <charles.tsai at cloudena.com>
> > To: "Vadim Rozenfeld" <vrozenfe at redhat.com>
> > Cc: "Michael Roth" <mdroth at linux.vnet.ibm.com>, "Stefan Hajnoczi"
> > <stefanha at gmail.com>, spice-devel at lists.freedesktop.org, "Alex
> > Huang-黃必賢-研究發展部" <alex.huang at cloudena.com>, "Alon Levy"
> > <alevy at redhat.com>, "qemu-devel" <qemu-devel at nongnu.org>, "Paul
> > Lu-盧偉智-研究發展部" <paul.lu at cloudena.com> Sent: Monday, February 6, 2012
> > 3:08:55 PM
> > Subject: RE: [Qemu-devel] [Spice-devel] Vioserial of Windows guest OS
> > on Qemu 0.15
> > 
> > Vadim,
> > 
> > I did the same test again today and no luck to made it work.
> > Let us what you find since we might have missed some points when we
> > did the test.
> > 
> > 
> > -----Original Message-----
> > From: Vadim Rozenfeld [mailto:vrozenfe at redhat.com]
> > Sent: Sunday, February 05, 2012 2:14 AM
> > To: Charles.Tsai-蔡清海-研究發展部
> > Cc: Michael Roth; Stefan Hajnoczi; spice-devel at lists.freedesktop.org;
> > Alex
> > Huang-黃必賢-研究發展部; Alon Levy; qemu-devel; Paul Lu-盧偉智-研究發展部 Subject: Re:
> > [Qemu-devel] [Spice-devel] Vioserial of Windows guest OS on Qemu 0.15
> > 
> > Thanks,
> > I'll check this problem.
> > Vadim.
> > 
> > ----- Original Message -----
> > From: "Charles.Tsai-蔡清海-研究發展部" <charles.tsai at cloudena.com>
> > To: "Charles.Tsai-蔡清海-研究發展部" <charles.tsai at cloudena.com>, "Vadim
> > Rozenfeld" <vrozenfe at redhat.com> Cc: "Michael Roth"
> > <mdroth at linux.vnet.ibm.com>, "Stefan Hajnoczi" <stefanha at gmail.com>,
> > spice-devel at lists.freedesktop.org,
> > "Alex Huang-黃必賢-研究發展部" <alex.huang at cloudena.com>, "Alon Levy"
> > <alevy at redhat.com>, "qemu-devel" <qemu-devel at nongnu.org>, "Paul
> > Lu-盧偉智-研究發展部" <paul.lu at cloudena.com> Sent: Saturday, February 4, 2012
> > 5:39:26 AM
> > Subject: RE: [Qemu-devel] [Spice-devel] Vioserial of Windows guest OS
> > on Qemu 0.15
> > 
> > Vadim,
> > 
> > We tested the free build driver on 32-bit Windows 7 and the symptom
> > was the same. In other words, the latest driver source from
> > https://github.com/YanVugenfirer/ is buggy and it was not workable at
> > all.
> > 
> > 
> > 
> > -----Original Message-----
> > From: Charles.Tsai-蔡清海-研究發展部
> > Sent: Friday, February 03, 2012 12:01 PM
> > To: 'Vadim Rozenfeld'
> > Cc: Michael Roth; Stefan Hajnoczi; spice-devel at lists.freedesktop.org;
> > Alex
> > Huang-黃必賢-研究發展部; Alon Levy; qemu-devel; Paul Lu-盧偉智-研究發展部 Subject: RE:
> > [Qemu-devel] [Spice-devel] Vioserial of Windows guest OS on Qemu 0.15
> > 
> > Vadim,
> > 
> > I downloaded the latest Windows guest driver code from
> > (https://github.com/YanVugenfirer/) and built the test binary driver
> > code. When I tested the release build Vioserial driver on 64-bit Windows
> > 7, the driver seemed to wait for something to happen and the update
> > driver Windows dialog simply just stayed in busy state. Finally, the
> > Windows system entered a busy loop. However, if I tested the debugging
> > build driver, the driver could be installed. But it took for a long time
> > to complete.
> > 
> > Our installed Qemu is 1.0. Is there Qemu build requirement so that we
> > can verify the vioserial driver? From my test, the latest code from
> > (https://github.com/YanVugenfirer/) seemed to be worst than the one we
> > tested before. Let me know if we did anything wrong.
> > 
> > 
> > -----Original Message-----
> > From: Vadim Rozenfeld [mailto:vrozenfe at redhat.com]
> > Sent: Friday, January 20, 2012 8:48 PM
> > To: Charles.Tsai-蔡清海-研究發展部
> > Cc: Michael Roth; Stefan Hajnoczi; spice-devel at lists.freedesktop.org;
> > Alex
> > Huang-黃必賢-研究發展部; Alon Levy; qemu-devel Subject: Re: [Qemu-devel]
> > [Spice-devel] Vioserial of Windows guest OS on Qemu 0.15
> > 
> > This code is slightly buggy.
> > Please try Yan's repository
> > at github (https://github.com/YanVugenfirer/).
> > I believe that the most critical changes have been merged already by
> > Yan into this public repository. I will ask to update binaries and
> > sources at fedoraproject site as well.
> > 
> > Best regards,
> > Vadim.
> > 
> > 
> > ----- Original Message -----
> > From: "Charles.Tsai-蔡清海-研究發展部" <charles.tsai at cloudena.com>
> > To: "Vadim Rozenfeld" <vrozenfe at redhat.com>
> > Cc: "Michael Roth" <mdroth at linux.vnet.ibm.com>, "Stefan Hajnoczi"
> > <stefanha at gmail.com>, spice-devel at lists.freedesktop.org, "Alex
> > Huang-黃必賢-研究發展部" <alex.huang at cloudena.com>, "Alon Levy"
> > <alevy at redhat.com>, "qemu-devel" <qemu-devel at nongnu.org> Sent: Friday,
> > January 20, 2012 3:25:51 AM
> > Subject: RE: [Qemu-devel] [Spice-devel] Vioserial of Windows guest OS
> > on Qemu 0.15
> > 
> > Vadim,
> > 
> > We downloaded the driver source from the following website.
> > ===========================================================
> > wget --no-check-certificate
> > https://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/src/vir
> > tio-
> > win-prewhql-0.1-15-sources.zip
> > 
> > -----Original Message-----
> > From: Vadim Rozenfeld [mailto:vrozenfe at redhat.com]
> > Sent: Thursday, January 19, 2012 8:25 PM
> > To: Charles.Tsai-蔡清海-研究發展部
> > Cc: Michael Roth; Stefan Hajnoczi; spice-devel at lists.freedesktop.org;
> > Alex
> > Huang-黃必賢-研究發展部; Alon Levy; qemu-devel Subject: Re: [Qemu-devel]
> > [Spice-devel] Vioserial of Windows guest OS on Qemu 0.15
> > 
> > Just to be sure that we are on the same page:
> > could you tell me about the origin of the source?
> > Is it the latest from the Yan's repository at github.com?
> > 
> > ----- Original Message -----
> > From: "Charles.Tsai-蔡清海-研究發展部" <charles.tsai at cloudena.com>
> > To: "Vadim Rozenfeld" <vrozenfe at redhat.com>
> > Cc: "Michael Roth" <mdroth at linux.vnet.ibm.com>, "Stefan Hajnoczi"
> > <stefanha at gmail.com>, spice-devel at lists.freedesktop.org, "Alex
> > Huang-黃必賢-研究發展部" <alex.huang at cloudena.com>, "Alon Levy"
> > <alevy at redhat.com>, "qemu-devel" <qemu-devel at nongnu.org> Sent:
> > Thursday, January 19, 2012 12:06:16 PM
> > Subject: RE: [Qemu-devel] [Spice-devel] Vioserial of Windows guest OS
> > on Qemu 0.15
> > 
> > Vadim,
> > 
> > We built it from the driver source. Up to this moment, we always use
> > the same binary to test Qemu.
> > 
> > 
> > -----Original Message-----
> > From: Vadim Rozenfeld [mailto:vrozenfe at redhat.com]
> > Sent: Thursday, January 19, 2012 5:39 PM
> > To: Charles.Tsai-蔡清海-研究發展部
> > Cc: Michael Roth; Stefan Hajnoczi; spice-devel at lists.freedesktop.org;
> > Alex
> > Huang-黃必賢-研究發展部; Alon Levy; qemu-devel Subject: RE: [Qemu-devel]
> > [Spice-devel] Vioserial of Windows guest OS on Qemu 0.15
> > 
> > On Thu, 2012-01-19 at 16:33 +0800, Charles.Tsai-蔡清海-研究發展部 wrote:
> > > Vadim,
> > > 
> > > It is SMP system.
> > 
> > What about vioserial driver itself?
> > did you build it from sources or is
> > it one, available through RHEL channels?
> > 
> > > -----Original Message-----
> > > From: Vadim Rozenfeld [mailto:vrozenfe at redhat.com]
> > > Sent: Thursday, January 19, 2012 3:58 PM
> > > To: Charles.Tsai-蔡清海-研究發展部
> > > Cc: Michael Roth; Stefan Hajnoczi;
> > > spice-devel at lists.freedesktop.org;
> > > Alex Huang-黃必賢-研究發展部; Alon Levy; qemu-devel
> > > Subject: RE: [Qemu-devel] [Spice-devel] Vioserial of Windows guest
> > > OS on Qemu 0.15
> > > 
> > > On Thu, 2012-01-19 at 09:41 +0800, Charles.Tsai-蔡清海-研究發展部 wrote:
> > > > Vadim,
> > > > 
> > > > I tested on Qemu 1.0.50. and found the VioSerial driver had
> > > > problem to install on 64-bit Win7 guest. During the driver
> > > > installation, the system hung after the driver being installed.
> > > > After I rebooted the guest OS, the Vioserial driver work. The hang
> > > > system seemed to be found only during the driver installation.
> > > 
> > > On UP or SMP system?
> > > 
> > > > -----Original Message-----
> > > > From: Vadim Rozenfeld [mailto:vrozenfe at redhat.com]
> > > > Sent: Wednesday, January 18, 2012 4:57 AM
> > > > To: Michael Roth
> > > > Cc: Charles.Tsai-蔡清海-研究發展部; Stefan Hajnoczi;
> > > > spice-devel at lists.freedesktop.org; Alex Huang-黃必賢-研究發展部; Alon
> > > > Levy; qemu-devel
> > > > Subject: Re: [Qemu-devel] [Spice-devel] Vioserial of Windows guest
> > > > OS on Qemu 0.15
> > > > 
> > > > On Mon, 2012-01-16 at 19:50 -0600, Michael Roth wrote:
> > > > > On 01/15/2012 08:02 PM, Charles.Tsai-蔡清海-研究發展部 wrote:
> > > > > > Vadim,
> > > > > > 
> > > > > > Thank you for your prompt reply. Here are the information for
> > > > > > our test case.
> > > > > > 
> > > > > > 
> > > > > > 1) we use the following command line to launch the guest OS
> > > > > > 
> > > > > > 
> > > > > > /usr/bin/kvm -S -M pc-0.14 -enable-kvm -m 1024 -smp
> > > > > > 1,sockets=1,cores=1,threads=1 -name win_xp -uuid
> > > > > > d9388815-ddd3-c38e-33c2-a9d5fcc7a775 -nodefconfig -nodefaults
> > > > > > -chardev
> > > > > > socket,id=charmonitor,path=/var/lib/libvirt/qemu/win_xp.monito
> > > > > > r,
> > > > > > se rv er,nowait -mon
> > > > > > chardev=charmonitor,id=monitor,mode=readline
> > > > > > -rtc base=localtime
> > > > > > -device
> > > > > > virtio-serial-pci,id=virtio-serial0,bus=pci.0,multifunction=on
> > > > > > ,a
> > > > > > dd
> > > > > > r=
> > > > > > 0x5.0x0 -drive
> > > > > > file=/media/Images/Windows-XP.img,if=none,id=drive-ide0-0-0,fo
> > > > > > rm
> > > > > > at
> > > > > > =r
> > > > > > aw -device
> > > > > > ide-drive,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bo
> > > > > > ot
> > > > > > in
> > > > > > de
> > > > > > x=1
> > > > > > -netdev tap,fd=17,id=hostnet0
> > > > > > -device
> > > > > > rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:e8:dc:b1,bus=pci.
> > > > > > 0,
> > > > > > mu
> > > > > > lt
> > > > > > ifunction=on,addr=0x3.0x0
> > > > > > -chardev pty,id=charserial0
> > > > > > -device isa-serial,chardev=charserial0,id=serial0
> > > > > > -chardev spicevmc,id=charchannel0,name=vdagent
> > > > > > -device
> > > > > > virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,
> > > > > > id
> > > > > > =c
> > > > > > ha
> > > > > > nnel0,name=com.redhat.spice.0
> > > > > > -usb -device usb-tablet,id=input0 -spice
> > > > > > port=5900,addr=0.0.0.0,disable-ticketing
> > > > > > -vga qxl -global qxl-vga.vram_size=67108864 -device
> > > > > > virtio-balloon-pci,id=balloon0,bus=pci.0,multifunction=on,addr
> > > > > > =0
> > > > > > x4
> > > > > > .0
> > > > > > x0
> > > > > > 
> > > > > > 
> > > > > > 
> > > > > > 2). In Guest Windows XP OS
> > > > > > 
> > > > > > 
> > > > > > When the following callback function of the vioserial device
> > > > > > is called in guest OS. The allocated resources is empty.
> > > > > > 
> > > > > > 
> > > > > > VIOSerialEvtDevicePrepareHardware() ---This function is to get
> > > > > > the I/O address of the vioserial device and map the physical
> > > > > > address to the logical address space.
> > > > > > 
> > > > > > I added the following trace and the value of nListSize is ZERO.
> > > > > > TraceEvents(TRACE_LEVEL_INFORMATION, DBG_PNP, "%s
> > > > > > (nListSize=%d)\n", __FUNCTION__,nListSize);
> > > > > > 
> > > > > > 
> > > > > > So far, we have tested Qemu 0.14 without any problem but Qemu
> > > > > > 0.15 seemed to be broken in vioserial device. Let me know if
> > > > > > you need further information. Thanks.
> > > > > 
> > > > > Hi Charles,
> > > > > 
> > > > > What versions of the virtio-win drivers are you using?
> > > > > 
> > > > > I've been testing virtio-serial on windows using the latest
> > > > > qemu.git (1.0). Linux guests work fine, but I've been having
> > > > > various issues with Windows 7, XP SP3, and Server 2008 R1. XP
> > > > > SP3
> > > > > 
> > > > > works intermittently for me using RHEL6.0 virtio-win, as well as
> > > > > the
> 
> drivers at:
> > > > I have seen some virtio serial port initialization problems on
> > > > 1.0.50. Will try to look into this problem in the following week(s).
> > > > 
> > > > > http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bi
> > > > > n/
> > > > > 
> > > > > But I've been getting a mix of issues such as guest hangs,
> > > > > vioser-test failing to enumerate any virtio-serial devices, or
> > > > > various non-critical error messages from qemu that seem to
> > > > > coincide with the channel being open/closed (occasionally
> > > > > resulting in the channel becoming unresponsive).
> > > > > 
> > > > > Do any of these seem similar to the behaviour you're seeing? If
> > > > > so I'll see if the issues go away on 0.14.0 and follow-up with a
> > > > > git bisect.


More information about the Spice-devel mailing list