答复: KVM Intel graphic passthrough cause qemu pause
Zhenyu Wang
zhenyuw at linux.intel.com
Mon Dec 3 04:06:59 UTC 2018
Hi,
Our mail list has limit of attachment for 200KB so I've got two
warnings on big log attachment for this that's blocked now.
Pls follow https://github.com/intel/gvt-linux/wiki to open bug
on bugzilla, or I think new github issue would also be ok.
thanks
On 2018.11.30 00:53:13 +0000, Jianghuaping wrote:
> Hi all
>
> The attached xml is Guest’s Qemu parameters.
>
> Thanks
>
> Jiang huaping
>
> 发件人: jianghuaping (Cloud)
> 发送时间: 2018年11月29日 18:12
> 收件人: bailin (Cloud); 'Zhang, Xiong Y'; Yuan, Hang;
> 'intel-gvt-dev at lists.freedesktop.org'
> 抄送: Peng, Chao P; wangxuan (Cloud); Zeng, Harris; Wang, Hongbo; wangtao
> (Cloud); daishijun (Cloud); yandehan (CTS)
> 主题: 答复: KVM Intel graphic passthrough cause qemu pause
>
>
>
> Hi Chao and Henry
>
> Sub error 1 re-appeared after adding Qemu parameter “x-vga=0”, the attached
> is dmesg, syslog and qemu log.
>
> Thanks
>
>
>
> Jiang
>
>
>
> 发件人: bailin (Cloud)
> 发送时间: 2018年11月29日 10:10
> 收件人: jianghuaping (Cloud); 'Zhang, Xiong Y'; Yuan, Hang;
> 'intel-gvt-dev at lists.freedesktop.org'
> 抄送: Peng, Chao P; wangxuan (Cloud); Zeng, Harris; Wang, Hongbo; wangtao
> (Cloud); daishijun (Cloud); yandehan (CTS)
> 主题: 答复: KVM Intel graphic passthrough cause qemu pause
>
>
>
> Guest os is 32bit windows 10, version 1703
>
>
>
>
>
>
>
> 发件人: jianghuaping (Cloud)
> 发送时间: 2018年11月29日 9:40
> 收件人: 'Zhang, Xiong Y' <xiong.y.zhang at intel.com>; Yuan, Hang <
> hang.yuan at intel.com>; 'intel-gvt-dev at lists.freedesktop.org' <
> intel-gvt-dev at lists.freedesktop.org>
> 抄送: Peng, Chao P <chao.p.peng at intel.com>; wangxuan (Cloud) <wang.xuan at h3c.com
> >; bailin (Cloud) <berlin at h3c.com>; Zeng, Harris <harris.zeng at intel.com>; Wang,
> Hongbo <hongbo.wang at intel.com>; wangtao (Cloud) <wang.taoD at h3c.com>; daishijun
> (Cloud) <daishijun at h3c.com>; yandehan (CTS) <ydhan at h3c.com>
> 主题: 答复: KVM Intel graphic passthrough cause qemu pause
>
>
>
> The guest is windows 10.
>
> There is no this issue on windows 7 guest.
>
>
>
> 发件人: Zhang, Xiong Y [mailto:xiong.y.zhang at intel.com]
> 发送时间: 2018年11月29日 9:38
> 收件人: jianghuaping (Cloud); Yuan, Hang; 'intel-gvt-dev at lists.freedesktop.org'
> 抄送: Peng, Chao P; wangxuan (Cloud); bailin (Cloud); Zeng, Harris; Wang,
> Hongbo; wangtao (Cloud); daishijun (Cloud); yandehan (CTS); Zhang, Xiong Y
> 主题: RE: KVM Intel graphic passthrough cause qemu pause
>
>
>
> When this issue happens, it seems the guest Win7 OS doesn’t start.
>
> If this issue happens during guest OS boot up, I suggest to try win8 or win10,
> I never try win7.
>
>
>
> thanks
>
> From: intel-gvt-dev [mailto:intel-gvt-dev-bounces at lists.freedesktop.org] On
> Behalf Of Jianghuaping
> Sent: Wednesday, November 28, 2018 7:18 PM
> To: Yuan, Hang <hang.yuan at intel.com>; 'intel-gvt-dev at lists.freedesktop.org' <
> intel-gvt-dev at lists.freedesktop.org>
> Cc: Peng, Chao P <chao.p.peng at intel.com>; Wangxuan <wang.xuan at h3c.com>; Bailin
> <berlin at h3c.com>; Zeng, Harris <harris.zeng at intel.com>; Wang, Hongbo <
> hongbo.wang at intel.com>; Wangtao <wang.taoD at h3c.com>; Daishijun <
> daishijun at h3c.com>; Yandehan <ydhan at h3c.com>
> Subject: 答复: KVM Intel graphic passthrough cause qemu pause
>
>
>
> Hi Henry
>
> Double check,
>
> What you mean is the syslog under /var/log/syslog? We can get it when this
> issue re-appeared.
>
>
>
> Thanks
>
> Jiang
>
>
>
> 发件人: Yuan, Hang [mailto:hang.yuan at intel.com]
> 发送时间: 2018年11月28日 18:55
> 收件人: jianghuaping (Cloud); 'intel-gvt-dev at lists.freedesktop.org'
> 抄送: Zeng, Harris; Peng, Chao P; daishijun (Cloud); Wang, Hongbo; bailin
> (Cloud); yandehan (CTS); wangxuan (Cloud); wangtao (Cloud)
> 主题: RE: KVM Intel graphic passthrough cause qemu pause
>
>
>
> Do you have host kernel log? So we can help to take a look from GVT
> perspective.
>
>
>
> Regards,
>
> Henry
>
>
>
> From: Jianghuaping [mailto:jiang.huaping at h3c.com]
> Sent: Wednesday, November 28, 2018 5:56 PM
> To: Yuan, Hang <hang.yuan at intel.com>; 'intel-gvt-dev at lists.freedesktop.org' <
> intel-gvt-dev at lists.freedesktop.org>
> Cc: Zeng, Harris <harris.zeng at intel.com>; Peng, Chao P <chao.p.peng at intel.com>;
> Daishijun <daishijun at h3c.com>; Wang, Hongbo <hongbo.wang at intel.com>; Bailin <
> berlin at h3c.com>; Yandehan <ydhan at h3c.com>; Wangxuan <wang.xuan at h3c.com>;
> Wangtao <wang.taoD at h3c.com>
> Subject: 答复: KVM Intel graphic passthrough cause qemu pause
>
>
>
> Hi Hang
>
> The attached is two logs.
>
> v288C1A78-C34C-11E8-94EF-CEFAE916D900.log : suberror 1 qemu log
>
> v8FE70BB0-BC03-11E8-874B-B4FF2E460600.log : suberror 3 qemu log
>
>
>
> old kvm parameter is :
>
> /usr/bin/kvm -name guest=v03000200-0400-0500-0006-000700080009,debug-threads=on
> -S -object secret,id=masterKey0,format=raw,file=/var/run/lib/libvirt/qemu/
> domain-1-v03000200-0400-0500-/master-key.aes -machine pc-i440fx-2.12,accel=
> kvm,usb=off,dump-guest-core=off -cpu host,hv_time,hv_relaxed,hv_spinlocks=
> 0x2000,host-cache-info=on,l3-cache=off -m 3380 -realtime mlock=off -smp
> 2,sockets=1,cores=2,threads=1 -uuid 078fe81f-f60a-45ab-9aaa-b6b97028dc35
> -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/
> var/run/lib/libvirt/qemu/domain-1-v03000200-0400-0500-/
> monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control
> -chardev socket,id=charmonitor_cas,path=/var/run/lib/libvirt/qemu/
> domain-1-v03000200-0400-0500-/monitor.sock.cas,server,nowait -mon chardev=
> charmonitor_cas,id=monitor_cas,mode=control -rtc base=localtime,clock=
> vm,driftfix=slew -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
> PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=
> usb,pci_hotpluggable=off,bus=pci.0,addr=0x1.0x2 -device usb-ehci,id=
> usb1,pci_hotpluggable=off,bus=pci.0,addr=0x4 -device nec-usb-xhci,id=
> usb2,pci_hotpluggable=off,bus=pci.0,addr=0x5 -device virtio-scsi-pci,id=
> scsi1,pci_hotpluggable=off,bus=pci.0,addr=0x6 -device virtio-serial-pci,id=
> virtio-serial0,pci_hotpluggable=off,bus=pci.0,addr=0x7 -device usb-hub,id=
> hub0,bus=usb.0,port=1 -drive file=/vms/idv/data/win7,format=qcow2,if=none,id=
> drive-virtio-disk0,cache=directsync,aio=native -device virtio-blk-pci,scsi=
> off,bus=pci.0,addr=0x8,pci_hotpluggable=off,drive=drive-virtio-disk0,id=
> virtio-disk0,bootindex=1 -drive file=/vms/isos/virtio-win7.vfd,format=raw,if=
> none,id=drive-fdc0-0-0,readonly=on,cache=directsync,aio=native -global
> isa-fdc.driveA=drive-fdc0-0-0 -global isa-fdc.bootindexA=4 -drive if=none,id=
> drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=
> drive-ide0-0-0,id=ide0-0-0,bootindex=2 -netdev tap,fd=51,id=hostnet0,vhost=
> on,vhostfd=52 -device virtio-net-pci,pci_hotpluggable=off,netdev=hostnet0,id=
> net0,mac=9c:06:1b:6f:93:5f,bus=pci.0,addr=0x3,bootindex=3 -chardev pty,id=
> charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev
> socket,id=charchannel0,path=/var/lib/libvirt/qemu/
> v03000200-0400-0500-0006-000700080009.agent,server,nowait -device
> virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=
> org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=2 -device
> vfio-pci,pci_hotpluggable=off,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2
> -device vfio-pci,pci_hotpluggable=off,host=00:1f.3,id=hostdev1,bus=pci.0,addr=
> 0x9 -device usb-host,hostbus=1,hostaddr=3,id=hostdev2,bus=usb.0,port=1.1
> -device usb-host,hostbus=1,hostaddr=2,id=hostdev3,bus=usb.0,port=1.2 -set
> device.hostdev0.x-igd-opregion=on -set device.hostdev0.x-igd-gms=1 -msg
> timestamp=on
>
>
>
>
>
> Thanks
>
> Jiang
>
>
>
> 发件人: Yuan, Hang [mailto:hang.yuan at intel.com]
> 发送时间: 2018年11月28日 15:52
> 收件人: jianghuaping (Cloud); 'intel-gvt-dev at lists.freedesktop.org'
> 抄送: Zeng, Harris; Peng, Chao P; daishijun (Cloud); Wang, Hongbo; bailin
> (Cloud)
> 主题: RE: KVM Intel graphic passthrough cause qemu pause
>
>
>
> Hi Huaping,
>
>
>
> What’s your Qemu parameters to create the VM? Do you have host kernel log to
> share?
>
>
>
> Regards,
>
> Henry
>
>
>
> From: intel-gvt-dev [mailto:intel-gvt-dev-bounces at lists.freedesktop.org] On
> Behalf Of Jianghuaping
> Sent: Wednesday, November 28, 2018 10:34 AM
> To: 'intel-gvt-dev at lists.freedesktop.org' <intel-gvt-dev at lists.freedesktop.org>
> Cc: Zeng, Harris <harris.zeng at intel.com>; Peng, Chao P <chao.p.peng at intel.com>;
> Daishijun <daishijun at h3c.com>; Wang, Hongbo <hongbo.wang at intel.com>; Bailin <
> berlin at h3c.com>
> Subject: KVM Intel graphic passthrough cause qemu pause
>
>
>
> Hello Intel GVT experts.
>
> we are using Intel skylake I3 processor to run KVM virtualization(1 Centos
> Hypervisor +1 Windows guest). Intel graphic in I3 processor will be passed
> through Qemu guest(this is a Windows 10 1703 guest). We found kvm will appear
> “suberror 3” or “suberror 1”, and Qemu will pause, when win 10 guest
> reboot or resume from sleep. Looks like this issue related to EPT miconfig,
> could you please help us on this issue? The following are detail information.
>
> In these days, Intel kvm expert:Peng chao is also helping analyzing this issue.
>
> ----------------------------------------------------------------------
>
> Linux Kernel &Kvm version:
>
> [root at cvknode31 ~]# virsh version
>
> Compiled against library: libvirt 4.0.0
>
> Using library: libvirt 4.0.0
>
> Using API: QEMU 4.0.0
>
> Running hypervisor: QEMU 2.12.1
>
> [root at cvknode31 ~]# uname -a
>
> Linux cvknode31 4.14.0-generic #862.el7 SMP Wed May 23 19:40:09 CST 2018 x86_64
> x86_64 x86_64 GNU/Linux
>
> [root at cvknode31 ~]# cat /etc/redhat-release
>
> CentOS Linux release 7.5.1804 (Core)
>
>
>
>
>
> -----------------------------------------------------------------------------
>
> Reboot caused suberror 3 qemu log
>
> KVM internal error. Suberror: 3
>
> extra data[0]: 80000b0e
>
> extra data[1]: 31
>
> extra data[2]: 683
>
> extra data[3]: 88c70
>
> EAX=00000000 EBX=87b84120 ECX=87b862c0 EDX=80843120
>
> ESI=87b2afac EDI=80843120 EBP=87a3aa44 ESP=87a3aa40
>
> EIP=873d5cfa EFL=00210202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
>
> ES =0023 00000000 ffffffff 00c0f300 DPL=3 DS [-WA]
>
> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>
> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>
> DS =0023 00000000 ffffffff 00c0f300 DPL=3 DS [-WA]
>
> FS =0030 87b84000 00004a20 00409300 DPL=0 DS [-WA]
>
> GS =0000 00000000 ffffffff 00c00000
>
> LDT=0000 00000000 ffffffff 00c00000
>
> TR =0028 87b88a40 000020ab 00008b00 DPL=0 TSS32-busy
>
> GDT= 87b8e5c0 000003ff
>
> IDT= 87b8e9c0 000007ff
>
> CR0=80010033 CR2=87b2afb0 CR3=001a8000 CR4=001406e8
>
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=
> 0000000000000000
>
> DR6=00000000ffff0ff0 DR7=0000000000000400
>
> EFER=0000000000000800
>
>
>
> ---------------------------------------------------------------------------------------
>
> Sleep caused suberror 1 qemu log
>
> KVM internal error. Suberror: 1
>
> emulation failure
>
> EAX=00010008 EBX=00000000 ECX=00024000 EDX=00000000
>
> ESI=00000000 EDI=00000000 EBP=b4237a1c ESP=b42379e8
>
> EIP=ffd03000 EFL=00010246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
>
> ES =0023 00000000 ffffffff 00c0f300 DPL=3 DS [-WA]
>
> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
>
> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
>
> DS =0023 00000000 ffffffff 00c0f300 DPL=3 DS [-WA]
>
> FS =0030 81158000 00004a60 00409300 DPL=0 DS [-WA]
>
> GS =0000 00000000 ffffffff 00c00000
>
> LDT=0000 00000000 ffffffff 00c00000
>
> TR =0028 8113f000 000020ab 00008b00 DPL=0 TSS32-busy
>
> GDT= 81151000 000003ff
>
> IDT= 81151400 000007ff
>
> CR0=80010033 CR2=8136b000 CR3=9ffd2320 CR4=001406e9
>
> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=
> 0000000000000000
>
> DR6=00000000ffff0ff0 DR7=0000000000000400
>
> EFER=0000000000000800
>
>
>
> Thanks
>
> Jiang huaping
>
> 发件人: bailin (Cloud)
> 发送时间: 2018年11月28日 8:08
> 收件人: Peng, Chao P; Zeng, Harris
> 抄送: jianghuaping (Cloud); changlimin (Cloud)
> 主题: 答复: 答复: pause问题
>
>
>
> 用户态的堆栈如下,感觉这种操作普遍并且正常.
>
>
>
> 一个cpu在pci的空间访问,导致修改memslot
>
> #0 0x00007fee3ede95d7 in ioctl () from /lib64/libc.so.6
>
> #1 0x00005624101a28d7 in kvm_vm_ioctl (s=0x8ec8, s at entry=0x5624132e8e20, type=
> 881622144, type at entry=1075883590)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/accel/kvm/kvm-all.c:2075
>
> #2 0x00005624101a3207 in kvm_set_user_memory_region (slot=slot at entry=
> 0x5624132ea110, kml=0x5624132e9ec0)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/accel/kvm/kvm-all.c:277
>
> #3 0x00005624101a3640 in kvm_set_phys_mem (kml=0x5624132e9ec0, section=
> <optimized out>, add=true)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/accel/kvm/kvm-all.c:774
>
> #4 0x00005624101929e1 in address_space_update_topology_pass (as=as at entry=
> 0x562410e3b5e0 <address_space_memory>, adding=adding at entry=true,
>
> new_view=0x7fee106a03d0, new_view=0x7fee106a03d0, old_view=<optimized out>,
> old_view=<optimized out>)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/memory.c:933
>
> #5 0x0000562410192d94 in address_space_set_flatview (as=as at entry=
> 0x562410e3b5e0 <address_space_memory>)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/memory.c:1008
>
> #6 0x00005624101959c0 in memory_region_transaction_commit ()
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/memory.c:1060
>
> #7 0x0000562410391ab8 in pci_update_vga (pci_dev=0x562414c97bc0)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/hw/pci/pci.c:1167
>
> #8 0x0000562410392913 in pci_update_vga (pci_dev=0x562414c97bc0)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/hw/pci/pci.c:1161
>
> #9 pci_update_mappings (d=d at entry=0x562414c97bc0)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/hw/pci/pci.c:1333
>
> #10 0x0000562410392f59 in pci_default_write_config (d=d at entry=0x562414c97bc0,
> addr=addr at entry=4, val_in=val_in at entry=1024, l=2)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/hw/pci/pci.c:1380
>
> #11 0x00005624101dd49c in vfio_pci_write_config (pdev=0x562414c97bc0, addr=4,
> val=1024, len=<optimized out>)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/hw/vfio/pci.c:1222
>
> #12 0x00005624103998da in pci_host_config_write_common (pci_dev=0x562414c97bc0,
> addr=4, limit=256, val=1024, len=2)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/hw/pci/pci_host.c:66
>
> #13 0x000056241019341b in memory_region_write_accessor (mr=0x5624136715b0, addr
> =0, value=<optimized out>, size=2, shift=<optimized out>,
>
> mask=<optimized out>, attrs=...) at /home/uis-enterprise/f-idv/daemon/qemu/
> qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:530
>
> #14 0x0000562410191029 in access_with_adjusted_size (addr=addr at entry=0, value=
> value at entry=0x7fee25dbb4a8, size=size at entry=2,
>
> access_size_min=<optimized out>, access_size_max=<optimized out>,
>
> access_fn=access_fn at entry=0x5624101933a0 <memory_region_write_accessor>, mr
> =mr at entry=0x5624136715b0, attrs=attrs at entry=...)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/memory.c:597
>
> #15 0x0000562410195f75 in memory_region_dispatch_write (mr=mr at entry=
> 0x5624136715b0, addr=addr at entry=0, data=1024, size=size at entry=2,
>
> attrs=attrs at entry=...) at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/
> rpmbuild/BUILD/qemu-2.12/memory.c:1474
>
> #16 0x0000562410131342 in flatview_write_continue (mr=0x5624136715b0, l=2,
> addr1=0, len=2, buf=0x7fee459ed000 "", attrs=..., addr=3324,
>
> fv=0x7fee1c594e40) at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/
> rpmbuild/BUILD/qemu-2.12/exec.c:3166
>
> #17 flatview_write (fv=0x7fee1c594e40, addr=<optimized out>, attrs=..., buf=
> <optimized out>, len=<optimized out>)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/exec.c:3216
>
> #18 0x000056241013507f in address_space_write (as=<optimized out>, addr=
> <optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/exec.c:3332
>
> #19 0x0000562410135125 in address_space_rw (as=<optimized out>, addr=addr at entry
> =3324, attrs=..., attrs at entry=..., buf=<optimized out>,
>
> len=len at entry=2, is_write=is_write at entry=true) at /home/uis-enterprise/
> f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/exec.c:3343
>
> #20 0x00005624101a4ff6 in kvm_handle_io (count=1, size=2, direction=<optimized
> out>, data=<optimized out>, attrs=..., port=3324)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/accel/kvm/kvm-all.c:1730
>
> #21 kvm_cpu_exec (cpu=cpu at entry=0x56241344c3e0)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/accel/kvm/kvm-all.c:1970
>
> #22 0x000056241017f9e6 in qemu_kvm_cpu_thread_fn (arg=0x56241344c3e0)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/cpus.c:1229
>
> #23 0x00007fee3f0c8e25 in start_thread () from /lib64/libpthread.so.0
>
> #24 0x00007fee3edf2bad in clone () from /lib64/libc.so.6
>
>
>
>
>
> 异常的cpu的qemu堆栈
>
> #0 0x00007fee3ede95d7 in ioctl () from /lib64/libc.so.6
>
> #1 0x00005624101a4d02 in kvm_vcpu_ioctl (cpu=0x0, cpu at entry=0x5624133fe150,
> type=136, type at entry=44672)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/accel/kvm/kvm-all.c:2093
>
> #2 0x00005624101a4e5f in kvm_cpu_exec (cpu=cpu at entry=0x5624133fe150)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/accel/kvm/kvm-all.c:1930
>
> #3 0x000056241017f9e6 in qemu_kvm_cpu_thread_fn (arg=0x5624133fe150)
>
> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/
> qemu-2.12/cpus.c:1229
>
> #4 0x00007fee3f0c8e25 in start_thread () from /lib64/libpthread.so.0
>
> #5 0x00007fee3edf2bad in clone () from /lib64/libc.so.6
>
>
>
>
>
>
>
> 发件人: Peng, Chao P [mailto:chao.p.peng at intel.com]
> 发送时间: 2018年11月26日 16:40
> 收件人: Zeng, Harris <harris.zeng at intel.com>; bailin (Cloud) <berlin at h3c.com>
> 抄送: jianghuaping (Cloud) <jiang.huaping at h3c.com>; changlimin (Cloud) <
> changlimin at h3c.com>
> 主题: RE: 答复: pause问题
>
>
>
> 我看了代码，目前的现象和RCU的本来设计也是吻合的。所以即使memslot 为空，还不好说
> 这是问题。
>
>
>
> 一个思路是看看新添加memslot的操作是什么时机触发，是否与guest的特定行为有关？
>
>
>
> From: Zeng, Harris
> Sent: Saturday, November 24, 2018 10:13 PM
> To: Bailin <berlin at h3c.com>; Peng, Chao P <chao.p.peng at intel.com>
> Cc: Jianghuaping <jiang.huaping at h3c.com>; Changlimin <changlimin at h3c.com>
> Subject: Re: 答复: pause问题
>
>
>
> Hi Chao,
>
>
>
> Can you help to advise?
>
>
>
> Thanks,
>
> Harris
>
> Sent from my mobile phone.
>
>
> 在 2018年11月22日，上午10:47，Bailin <berlin at h3c.com> 写道：
>
> 不知道你们那边有没有具体线索,目前我们这边分析是内核函数
> kvm_vcpu_gfn_to_hva_prot可能有缺陷
>
> 目前我们这边没有更好的修改思路,也无法确认为什么这里返回错误,会最终导致ept
> misconfig.
>
>
>
>
>
> 出现问题时,memslot为空，但是在coredump中看到的不为空的准确原因明确了
>
>
>
> 出问题的cpu的堆栈
>
> #0 [ffffc900009d37a8] machine_kexec at ffffffff8105da32
>
> #1 [ffffc900009d3800] __crash_kexec at ffffffff8111a9ad
>
> #2 [ffffc900009d38c8] panic at ffffffff81085e2c
>
> #3 [ffffc900009d3948] paging64_walk_addr_generic at ffffffffc05cf4b3 [kvm]
>
> #4 [ffffc900009d3a30] paging64_gva_to_gpa at ffffffffc05cf6df [kvm]
>
> #5 [ffffc900009d3b48] emulator_read_write_onepage at ffffffffc05b6f5b [kvm]
>
> #6 [ffffc900009d3ba8] emulator_read_write at ffffffffc05b72d2 [kvm]
>
> #7 [ffffc900009d3bf8] segmented_read at ffffffffc05da519 [kvm]
>
> #8 [ffffc900009d3c38] x86_emulate_insn at ffffffffc05de3f0 [kvm]
>
> #9 [ffffc900009d3c88] x86_emulate_instruction at ffffffffc05c0149 [kvm]
>
> #10 [ffffc900009d3cf8] kvm_mmu_page_fault_ept_violation at ffffffffc05cd9ea
> [kvm]
>
> #11 [ffffc900009d3d30] kvm_arch_vcpu_ioctl_run at ffffffffc05c4510 [kvm]
>
> #12 [ffffc900009d3df0] kvm_vcpu_ioctl at ffffffffc05aa197 [kvm]
>
> #13 [ffffc900009d3e80] do_vfs_ioctl at ffffffff8126bbc9
>
> #14 [ffffc900009d3f00] sys_ioctl at ffffffff8126c1b4
>
> #15 [ffffc900009d3f38] do_syscall_64 at ffffffff810036fe
>
> #16 [ffffc900009d3f50] entry_SYSCALL_64_after_hwframe at ffffffff81a00081
>
>
>
>
>
> 发生异常的cpu调用这个函数，获取memslots，是没有问题的，这个函数除了索引数组
> ，
>
> 主要是在内核的lock debug打开时检查kvm->srcu，kvm->slots_lock的状态，而一般
> 情况下lock debug是不打开的
>
> static inline struct kvm_memslots *__kvm_memslots(struct kvm *kvm, int
> as_id)
>
> {
>
> return srcu_dereference_check(kvm->memslots[as_id], &kvm->srcu,
>
> lockdep_is_held(&kvm->slots_lock) ||
>
> !refcount_read(&kvm->users_count));
>
> }
>
> 实际上是memslots取错了，导致根据gfn查找的memslot错了，修改了调试信息
>
> unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn,
> bool *writable)
>
> {
>
> //struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu,
> gfn);
>
> struct kvm_memslots *slots = kvm_vcpu_memslots(vcpu);
>
> struct kvm_memory_slot *slot = __gfn_to_memslot(slots, gfn);
>
>
>
> if(NULL == slot){
>
> printk("%s,%d,slots=%p,vpuid=%d,gfn=%llx\
> n",__FUNCTION__,__LINE__,slots,vcpu->vcpu_id,gfn);
>
> }
>
>
>
> return gfn_to_hva_memslot_prot(slot, gfn, writable);
>
> }
>
>
>
> 出问题时的，memslot为空，但是memslots不为空，但是和当前内存中的不一致，
>
> [ 5621.133030] kvm_vcpu_gfn_to_hva_prot,1325,slots=ffff88011b6d0000,vpuid=
> 0,gfn=84
>
>
>
> 而coredump中读出的memslots
>
> crash> struct -x kvm.memslots 0xffff880118380000
>
> memslots = {0xffff88011ee40000, 0xffff88011ee10000}
>
>
>
> 0xffff88011ee40000这个memslots里面是有对应gfn的memslot，0x ffff88011b6d0000
> 是没有的
>
>
>
> 原因是coredump中看到的是另外一个cpu在install_new_memslots做了更新的memslots
> ，
>
> 而发生异常时的cpu读取的是更新前的memslots。
>
> 另外一个cpu的堆栈
>
> #0 [ffffc900009ffb08] __schedule at ffffffff818f8c6e
>
> #1 [ffffc900009ffb98] preempt_schedule_common at ffffffff818f94ad
>
> #2 [ffffc900009ffba8] _cond_resched at ffffffff818f94d8
>
> #3 [ffffc900009ffbb0] wait_for_completion at ffffffff818fa69c
>
> #4 [ffffc900009ffc08] __synchronize_srcu at ffffffff810f1447
>
> #5 [ffffc900009ffc70] install_new_memslots at ffffffffc05a5d89 [kvm]
>
> #6 [ffffc900009ffc90] __kvm_set_memory_region at ffffffffc05a7061 [kvm]
>
> #7 [ffffc900009ffda0] kvm_set_memory_region at ffffffffc05a72d6 [kvm]
>
> #8 [ffffc900009ffdc0] kvm_vm_ioctl at ffffffffc05a994b [kvm]
>
> #9 [ffffc900009ffe80] do_vfs_ioctl at ffffffff8126bbc9
>
> #10 [ffffc900009fff00] sys_ioctl at ffffffff8126c1b4
>
> #11 [ffffc900009fff38] do_syscall_64 at ffffffff810036fe
>
> #12 [ffffc900009fff50] entry_SYSCALL_64_after_hwframe at ffffffff81a00081
>
>
>
>
>
> static struct kvm_memslots *install_new_memslots(struct kvm *kvm,
>
> int as_id, struct kvm_memslots *slots)
>
> {
>
> struct kvm_memslots *old_memslots = __kvm_memslots(kvm, as_id);
>
>
>
> /*
>
> * Set the low bit in the generation, which disables SPTE caching
>
> * until the end of synchronize_srcu_expedited.
>
> */
>
> WARN_ON(old_memslots->generation & 1);
>
> slots->generation = old_memslots->generation + 1;
>
>
>
> rcu_assign_pointer(kvm->memslots[as_id], slots);
>
> synchronize_srcu_expedited(&kvm->srcu);
>
>
>
>
>
> 实际中另外一个cpu在更新memslots之前是加了slots的锁，lock debug没有打开时其
> 实是检测不到错误的。
>
> 而更新过程中，数据可能是不准确的。
>
> int kvm_set_memory_region(struct kvm *kvm,
>
> const struct kvm_userspace_memory_region
> *mem)
>
> {
>
> int r;
>
>
>
> mutex_lock(&kvm->slots_lock);
>
> r = __kvm_set_memory_region(kvm, mem);
>
> mutex_unlock(&kvm->slots_lock);
>
> return r;
>
> }
>
>
>
>
>
>
>
>
>
> -------------------------------------------------------------------------------------------------------------------------------------
> 本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出
> 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、
> 复制、
> 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并
> 删除本
> 邮件！
> This e-mail and its attachments contain confidential information from New
> H3C, which is
> intended only for the person or entity whose address is listed above. Any
> use of the
> information contained herein in any way (including, but not limited to,
> total or partial
> disclosure, reproduction, or dissemination) by persons other than the
> intended
> recipient(s) is prohibited. If you receive this e-mail in error, please
> notify the sender
> by phone or email immediately and delete it!
>
> _______________________________________________
> intel-gvt-dev mailing list
> intel-gvt-dev at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/intel-gvt-dev
--
Open Source Technology Center, Intel ltd.
$gpg --keyserver wwwkeys.pgp.net --recv-keys 4D781827
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 195 bytes
Desc: not available
URL: <https://lists.freedesktop.org/archives/intel-gvt-dev/attachments/20181203/574494d5/attachment-0001.sig>
More information about the intel-gvt-dev
mailing list