答复: KVM Intel graphic passthrough cause qemu pause
Bailin
berlin at h3c.com
Thu Nov 29 02:09:33 UTC 2018
Guest os is 32bit windows 10, version 1703
发件人: jianghuaping (Cloud)
发送时间: 2018年11月29日 9:40
收件人: 'Zhang, Xiong Y' <xiong.y.zhang at intel.com>; Yuan, Hang <hang.yuan at intel.com>; 'intel-gvt-dev at lists.freedesktop.org' <intel-gvt-dev at lists.freedesktop.org>
抄送: Peng, Chao P <chao.p.peng at intel.com>; wangxuan (Cloud) <wang.xuan at h3c.com>; bailin (Cloud) <berlin at h3c.com>; Zeng, Harris <harris.zeng at intel.com>; Wang, Hongbo <hongbo.wang at intel.com>; wangtao (Cloud) <wang.taoD at h3c.com>; daishijun (Cloud) <daishijun at h3c.com>; yandehan (CTS) <ydhan at h3c.com>
主题: 答复: KVM Intel graphic passthrough cause qemu pause
The guest is windows 10.
There is no this issue on windows 7 guest.
发件人: Zhang, Xiong Y [mailto:xiong.y.zhang at intel.com]
发送时间: 2018年11月29日 9:38
收件人: jianghuaping (Cloud); Yuan, Hang; 'intel-gvt-dev at lists.freedesktop.org'
抄送: Peng, Chao P; wangxuan (Cloud); bailin (Cloud); Zeng, Harris; Wang, Hongbo; wangtao (Cloud); daishijun (Cloud); yandehan (CTS); Zhang, Xiong Y
主题: RE: KVM Intel graphic passthrough cause qemu pause
When this issue happens, it seems the guest Win7 OS doesn’t start.
If this issue happens during guest OS boot up, I suggest to try win8 or win10, I never try win7.
thanks
From: intel-gvt-dev [mailto:intel-gvt-dev-bounces at lists.freedesktop.org] On Behalf Of Jianghuaping
Sent: Wednesday, November 28, 2018 7:18 PM
To: Yuan, Hang <hang.yuan at intel.com<mailto:hang.yuan at intel.com>>; 'intel-gvt-dev at lists.freedesktop.org' <intel-gvt-dev at lists.freedesktop.org<mailto:intel-gvt-dev at lists.freedesktop.org>>
Cc: Peng, Chao P <chao.p.peng at intel.com<mailto:chao.p.peng at intel.com>>; Wangxuan <wang.xuan at h3c.com<mailto:wang.xuan at h3c.com>>; Bailin <berlin at h3c.com<mailto:berlin at h3c.com>>; Zeng, Harris <harris.zeng at intel.com<mailto:harris.zeng at intel.com>>; Wang, Hongbo <hongbo.wang at intel.com<mailto:hongbo.wang at intel.com>>; Wangtao <wang.taoD at h3c.com<mailto:wang.taoD at h3c.com>>; Daishijun <daishijun at h3c.com<mailto:daishijun at h3c.com>>; Yandehan <ydhan at h3c.com<mailto:ydhan at h3c.com>>
Subject: 答复: KVM Intel graphic passthrough cause qemu pause
Hi Henry
Double check,
What you mean is the syslog under /var/log/syslog? We can get it when this issue re-appeared.
Thanks
Jiang
发件人: Yuan, Hang [mailto:hang.yuan at intel.com]
发送时间: 2018年11月28日 18:55
收件人: jianghuaping (Cloud); 'intel-gvt-dev at lists.freedesktop.org'
抄送: Zeng, Harris; Peng, Chao P; daishijun (Cloud); Wang, Hongbo; bailin (Cloud); yandehan (CTS); wangxuan (Cloud); wangtao (Cloud)
主题: RE: KVM Intel graphic passthrough cause qemu pause
Do you have host kernel log? So we can help to take a look from GVT perspective.
Regards,
Henry
From: Jianghuaping [mailto:jiang.huaping at h3c.com]
Sent: Wednesday, November 28, 2018 5:56 PM
To: Yuan, Hang <hang.yuan at intel.com<mailto:hang.yuan at intel.com>>; 'intel-gvt-dev at lists.freedesktop.org' <intel-gvt-dev at lists.freedesktop.org<mailto:intel-gvt-dev at lists.freedesktop.org>>
Cc: Zeng, Harris <harris.zeng at intel.com<mailto:harris.zeng at intel.com>>; Peng, Chao P <chao.p.peng at intel.com<mailto:chao.p.peng at intel.com>>; Daishijun <daishijun at h3c.com<mailto:daishijun at h3c.com>>; Wang, Hongbo <hongbo.wang at intel.com<mailto:hongbo.wang at intel.com>>; Bailin <berlin at h3c.com<mailto:berlin at h3c.com>>; Yandehan <ydhan at h3c.com<mailto:ydhan at h3c.com>>; Wangxuan <wang.xuan at h3c.com<mailto:wang.xuan at h3c.com>>; Wangtao <wang.taoD at h3c.com<mailto:wang.taoD at h3c.com>>
Subject: 答复: KVM Intel graphic passthrough cause qemu pause
Hi Hang
The attached is two logs.
v288C1A78-C34C-11E8-94EF-CEFAE916D900.log : suberror 1 qemu log
v8FE70BB0-BC03-11E8-874B-B4FF2E460600.log : suberror 3 qemu log
old kvm parameter is :
/usr/bin/kvm -name guest=v03000200-0400-0500-0006-000700080009,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/run/lib/libvirt/qemu/domain-1-v03000200-0400-0500-/master-key.aes -machine pc-i440fx-2.12,accel=kvm,usb=off,dump-guest-core=off -cpu host,hv_time,hv_relaxed,hv_spinlocks=0x2000,host-cache-info=on,l3-cache=off -m 3380 -realtime mlock=off -smp 2,sockets=1,cores=2,threads=1 -uuid 078fe81f-f60a-45ab-9aaa-b6b97028dc35 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/run/lib/libvirt/qemu/domain-1-v03000200-0400-0500-/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -chardev socket,id=charmonitor_cas,path=/var/run/lib/libvirt/qemu/domain-1-v03000200-0400-0500-/monitor.sock.cas,server,nowait -mon chardev=charmonitor_cas,id=monitor_cas,mode=control -rtc base=localtime,clock=vm,driftfix=slew -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,pci_hotpluggable=off,bus=pci.0,addr=0x1.0x2 -device usb-ehci,id=usb1,pci_hotpluggable=off,bus=pci.0,addr=0x4 -device nec-usb-xhci,id=usb2,pci_hotpluggable=off,bus=pci.0,addr=0x5 -device virtio-scsi-pci,id=scsi1,pci_hotpluggable=off,bus=pci.0,addr=0x6 -device virtio-serial-pci,id=virtio-serial0,pci_hotpluggable=off,bus=pci.0,addr=0x7 -device usb-hub,id=hub0,bus=usb.0,port=1 -drive file=/vms/idv/data/win7,format=qcow2,if=none,id=drive-virtio-disk0,cache=directsync,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,pci_hotpluggable=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/vms/isos/virtio-win7.vfd,format=raw,if=none,id=drive-fdc0-0-0,readonly=on,cache=directsync,aio=native -global isa-fdc.driveA=drive-fdc0-0-0 -global isa-fdc.bootindexA=4 -drive if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -netdev tap,fd=51,id=hostnet0,vhost=on,vhostfd=52 -device virtio-net-pci,pci_hotpluggable=off,netdev=hostnet0,id=net0,mac=9c:06:1b:6f:93:5f,bus=pci.0,addr=0x3,bootindex=3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/v03000200-0400-0500-0006-000700080009.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=2 -device vfio-pci,pci_hotpluggable=off,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2 -device vfio-pci,pci_hotpluggable=off,host=00:1f.3,id=hostdev1,bus=pci.0,addr=0x9 -device usb-host,hostbus=1,hostaddr=3,id=hostdev2,bus=usb.0,port=1.1 -device usb-host,hostbus=1,hostaddr=2,id=hostdev3,bus=usb.0,port=1.2 -set device.hostdev0.x-igd-opregion=on -set device.hostdev0.x-igd-gms=1 -msg timestamp=on
Thanks
Jiang
发件人: Yuan, Hang [mailto:hang.yuan at intel.com]
发送时间: 2018年11月28日 15:52
收件人: jianghuaping (Cloud); 'intel-gvt-dev at lists.freedesktop.org'
抄送: Zeng, Harris; Peng, Chao P; daishijun (Cloud); Wang, Hongbo; bailin (Cloud)
主题: RE: KVM Intel graphic passthrough cause qemu pause
Hi Huaping,
What’s your Qemu parameters to create the VM? Do you have host kernel log to share?
Regards,
Henry
From: intel-gvt-dev [mailto:intel-gvt-dev-bounces at lists.freedesktop.org] On Behalf Of Jianghuaping
Sent: Wednesday, November 28, 2018 10:34 AM
To: 'intel-gvt-dev at lists.freedesktop.org' <intel-gvt-dev at lists.freedesktop.org<mailto:intel-gvt-dev at lists.freedesktop.org>>
Cc: Zeng, Harris <harris.zeng at intel.com<mailto:harris.zeng at intel.com>>; Peng, Chao P <chao.p.peng at intel.com<mailto:chao.p.peng at intel.com>>; Daishijun <daishijun at h3c.com<mailto:daishijun at h3c.com>>; Wang, Hongbo <hongbo.wang at intel.com<mailto:hongbo.wang at intel.com>>; Bailin <berlin at h3c.com<mailto:berlin at h3c.com>>
Subject: KVM Intel graphic passthrough cause qemu pause
Hello Intel GVT experts.
we are using Intel skylake I3 processor to run KVM virtualization(1 Centos Hypervisor +1 Windows guest). Intel graphic in I3 processor will be passed through Qemu guest(this is a Windows 10 1703 guest). We found kvm will appear “suberror 3” or “suberror 1”, and Qemu will pause, when win 10 guest reboot or resume from sleep. Looks like this issue related to EPT miconfig, could you please help us on this issue? The following are detail information.
In these days, Intel kvm expert:Peng chao is also helping analyzing this issue.
----------------------------------------------------------------------
Linux Kernel &Kvm version:
[root at cvknode31 ~]# virsh version
Compiled against library: libvirt 4.0.0
Using library: libvirt 4.0.0
Using API: QEMU 4.0.0
Running hypervisor: QEMU 2.12.1
[root at cvknode31 ~]# uname -a
Linux cvknode31 4.14.0-generic #862.el7 SMP Wed May 23 19:40:09 CST 2018 x86_64 x86_64 x86_64 GNU/Linux
[root at cvknode31 ~]# cat /etc/redhat-release
CentOS Linux release 7.5.1804 (Core)
-----------------------------------------------------------------------------
Reboot caused suberror 3 qemu log
KVM internal error. Suberror: 3
extra data[0]: 80000b0e
extra data[1]: 31
extra data[2]: 683
extra data[3]: 88c70
EAX=00000000 EBX=87b84120 ECX=87b862c0 EDX=80843120
ESI=87b2afac EDI=80843120 EBP=87a3aa44 ESP=87a3aa40
EIP=873d5cfa EFL=00210202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0023 00000000 ffffffff 00c0f300 DPL=3 DS [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
DS =0023 00000000 ffffffff 00c0f300 DPL=3 DS [-WA]
FS =0030 87b84000 00004a20 00409300 DPL=0 DS [-WA]
GS =0000 00000000 ffffffff 00c00000
LDT=0000 00000000 ffffffff 00c00000
TR =0028 87b88a40 000020ab 00008b00 DPL=0 TSS32-busy
GDT= 87b8e5c0 000003ff
IDT= 87b8e9c0 000007ff
CR0=80010033 CR2=87b2afb0 CR3=001a8000 CR4=001406e8
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000800
---------------------------------------------------------------------------------------
Sleep caused suberror 1 qemu log
KVM internal error. Suberror: 1
emulation failure
EAX=00010008 EBX=00000000 ECX=00024000 EDX=00000000
ESI=00000000 EDI=00000000 EBP=b4237a1c ESP=b42379e8
EIP=ffd03000 EFL=00010246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0
ES =0023 00000000 ffffffff 00c0f300 DPL=3 DS [-WA]
CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]
SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]
DS =0023 00000000 ffffffff 00c0f300 DPL=3 DS [-WA]
FS =0030 81158000 00004a60 00409300 DPL=0 DS [-WA]
GS =0000 00000000 ffffffff 00c00000
LDT=0000 00000000 ffffffff 00c00000
TR =0028 8113f000 000020ab 00008b00 DPL=0 TSS32-busy
GDT= 81151000 000003ff
IDT= 81151400 000007ff
CR0=80010033 CR2=8136b000 CR3=9ffd2320 CR4=001406e9
DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000
DR6=00000000ffff0ff0 DR7=0000000000000400
EFER=0000000000000800
Thanks
Jiang huaping
发件人: bailin (Cloud)
发送时间: 2018年11月28日 8:08
收件人: Peng, Chao P; Zeng, Harris
抄送: jianghuaping (Cloud); changlimin (Cloud)
主题: 答复: 答复: pause问题
用户态的堆栈如下,感觉这种操作普遍并且正常.
一个cpu在pci的空间访问,导致修改memslot
#0 0x00007fee3ede95d7 in ioctl () from /lib64/libc.so.6
#1 0x00005624101a28d7 in kvm_vm_ioctl (s=0x8ec8, s at entry=0x5624132e8e20, type=881622144, type at entry=1075883590)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:2075
#2 0x00005624101a3207 in kvm_set_user_memory_region (slot=slot at entry=0x5624132ea110, kml=0x5624132e9ec0)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:277
#3 0x00005624101a3640 in kvm_set_phys_mem (kml=0x5624132e9ec0, section=<optimized out>, add=true)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:774
#4 0x00005624101929e1 in address_space_update_topology_pass (as=as at entry=0x562410e3b5e0 <address_space_memory>, adding=adding at entry=true,
new_view=0x7fee106a03d0, new_view=0x7fee106a03d0, old_view=<optimized out>, old_view=<optimized out>)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:933
#5 0x0000562410192d94 in address_space_set_flatview (as=as at entry=0x562410e3b5e0 <address_space_memory>)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:1008
#6 0x00005624101959c0 in memory_region_transaction_commit ()
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:1060
#7 0x0000562410391ab8 in pci_update_vga (pci_dev=0x562414c97bc0)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/hw/pci/pci.c:1167
#8 0x0000562410392913 in pci_update_vga (pci_dev=0x562414c97bc0)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/hw/pci/pci.c:1161
#9 pci_update_mappings (d=d at entry=0x562414c97bc0)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/hw/pci/pci.c:1333
#10 0x0000562410392f59 in pci_default_write_config (d=d at entry=0x562414c97bc0, addr=addr at entry=4, val_in=val_in at entry=1024, l=2)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/hw/pci/pci.c:1380
#11 0x00005624101dd49c in vfio_pci_write_config (pdev=0x562414c97bc0, addr=4, val=1024, len=<optimized out>)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/hw/vfio/pci.c:1222
#12 0x00005624103998da in pci_host_config_write_common (pci_dev=0x562414c97bc0, addr=4, limit=256, val=1024, len=2)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/hw/pci/pci_host.c:66
#13 0x000056241019341b in memory_region_write_accessor (mr=0x5624136715b0, addr=0, value=<optimized out>, size=2, shift=<optimized out>,
mask=<optimized out>, attrs=...) at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:530
#14 0x0000562410191029 in access_with_adjusted_size (addr=addr at entry=0, value=value at entry=0x7fee25dbb4a8, size=size at entry=2,
access_size_min=<optimized out>, access_size_max=<optimized out>,
access_fn=access_fn at entry=0x5624101933a0 <memory_region_write_accessor>, mr=mr at entry=0x5624136715b0, attrs=attrs at entry=...)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:597
#15 0x0000562410195f75 in memory_region_dispatch_write (mr=mr at entry=0x5624136715b0, addr=addr at entry=0, data=1024, size=size at entry=2,
attrs=attrs at entry=...) at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:1474
#16 0x0000562410131342 in flatview_write_continue (mr=0x5624136715b0, l=2, addr1=0, len=2, buf=0x7fee459ed000 "", attrs=..., addr=3324,
fv=0x7fee1c594e40) at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/exec.c:3166
#17 flatview_write (fv=0x7fee1c594e40, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/exec.c:3216
#18 0x000056241013507f in address_space_write (as=<optimized out>, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/exec.c:3332
#19 0x0000562410135125 in address_space_rw (as=<optimized out>, addr=addr at entry=3324, attrs=..., attrs at entry=..., buf=<optimized out>,
len=len at entry=2, is_write=is_write at entry=true) at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/exec.c:3343
#20 0x00005624101a4ff6 in kvm_handle_io (count=1, size=2, direction=<optimized out>, data=<optimized out>, attrs=..., port=3324)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:1730
#21 kvm_cpu_exec (cpu=cpu at entry=0x56241344c3e0)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:1970
#22 0x000056241017f9e6 in qemu_kvm_cpu_thread_fn (arg=0x56241344c3e0)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/cpus.c:1229
#23 0x00007fee3f0c8e25 in start_thread () from /lib64/libpthread.so.0
#24 0x00007fee3edf2bad in clone () from /lib64/libc.so.6
异常的cpu的qemu堆栈
#0 0x00007fee3ede95d7 in ioctl () from /lib64/libc.so.6
#1 0x00005624101a4d02 in kvm_vcpu_ioctl (cpu=0x0, cpu at entry=0x5624133fe150, type=136, type at entry=44672)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:2093
#2 0x00005624101a4e5f in kvm_cpu_exec (cpu=cpu at entry=0x5624133fe150)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:1930
#3 0x000056241017f9e6 in qemu_kvm_cpu_thread_fn (arg=0x5624133fe150)
at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/cpus.c:1229
#4 0x00007fee3f0c8e25 in start_thread () from /lib64/libpthread.so.0
#5 0x00007fee3edf2bad in clone () from /lib64/libc.so.6
发件人: Peng, Chao P [mailto:chao.p.peng at intel.com]
发送时间: 2018年11月26日 16:40
收件人: Zeng, Harris <harris.zeng at intel.com<mailto:harris.zeng at intel.com>>; bailin (Cloud) <berlin at h3c.com<mailto:berlin at h3c.com>>
抄送: jianghuaping (Cloud) <jiang.huaping at h3c.com<mailto:jiang.huaping at h3c.com>>; changlimin (Cloud) <changlimin at h3c.com<mailto:changlimin at h3c.com>>
主题: RE: 答复: pause问题
我看了代码，目前的现象和RCU的本来设计也是吻合的。所以即使memslot 为空， 还不好说这是问题。
一个思路是看看新添加memslot的操作是什么时机触发，是否与guest的特定行为有关？
From: Zeng, Harris
Sent: Saturday, November 24, 2018 10:13 PM
To: Bailin <berlin at h3c.com<mailto:berlin at h3c.com>>; Peng, Chao P <chao.p.peng at intel.com<mailto:chao.p.peng at intel.com>>
Cc: Jianghuaping <jiang.huaping at h3c.com<mailto:jiang.huaping at h3c.com>>; Changlimin <changlimin at h3c.com<mailto:changlimin at h3c.com>>
Subject: Re: 答复: pause问题
Hi Chao,
Can you help to advise?
Thanks,
Harris
Sent from my mobile phone.
在 2018年11月22日，上午10:47，Bailin <berlin at h3c.com<mailto:berlin at h3c.com>> 写道：
不知道你们那边有没有具体线索,目前我们这边分析是内核函数kvm_vcpu_gfn_to_hva_prot可能有缺陷
目前我们这边没有更好的修改思路,也无法确认为什么这里返回错误,会最终导致ept misconfig.
出现问题时,memslot为空，但是在coredump中看到的不为空的准确原因明确了
出问题的cpu的堆栈
#0 [ffffc900009d37a8] machine_kexec at ffffffff8105da32
#1 [ffffc900009d3800] __crash_kexec at ffffffff8111a9ad
#2 [ffffc900009d38c8] panic at ffffffff81085e2c
#3 [ffffc900009d3948] paging64_walk_addr_generic at ffffffffc05cf4b3 [kvm]
#4 [ffffc900009d3a30] paging64_gva_to_gpa at ffffffffc05cf6df [kvm]
#5 [ffffc900009d3b48] emulator_read_write_onepage at ffffffffc05b6f5b [kvm]
#6 [ffffc900009d3ba8] emulator_read_write at ffffffffc05b72d2 [kvm]
#7 [ffffc900009d3bf8] segmented_read at ffffffffc05da519 [kvm]
#8 [ffffc900009d3c38] x86_emulate_insn at ffffffffc05de3f0 [kvm]
#9 [ffffc900009d3c88] x86_emulate_instruction at ffffffffc05c0149 [kvm]
#10 [ffffc900009d3cf8] kvm_mmu_page_fault_ept_violation at ffffffffc05cd9ea [kvm]
#11 [ffffc900009d3d30] kvm_arch_vcpu_ioctl_run at ffffffffc05c4510 [kvm]
#12 [ffffc900009d3df0] kvm_vcpu_ioctl at ffffffffc05aa197 [kvm]
#13 [ffffc900009d3e80] do_vfs_ioctl at ffffffff8126bbc9
#14 [ffffc900009d3f00] sys_ioctl at ffffffff8126c1b4
#15 [ffffc900009d3f38] do_syscall_64 at ffffffff810036fe
#16 [ffffc900009d3f50] entry_SYSCALL_64_after_hwframe at ffffffff81a00081
发生异常的cpu调用 这个函数，获取memslots，是没有问题的，这个函数除了索引数组，
主要是在内核的lock debug打开时检查kvm->srcu，kvm->slots_lock的状态，而一般情况下lock debug是不打开的
static inline struct kvm_memslots *__kvm_memslots(struct kvm *kvm, int as_id)
{
return srcu_dereference_check(kvm->memslots[as_id], &kvm->srcu,
lockdep_is_held(&kvm->slots_lock) ||
!refcount_read(&kvm->users_count));
}
实际上是memslots取错了，导致根据gfn查找的memslot错了，修改了调试信息
unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, bool *writable)
{
//struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);
struct kvm_memslots *slots = kvm_vcpu_memslots(vcpu);
struct kvm_memory_slot *slot = __gfn_to_memslot(slots, gfn);
if(NULL == slot){
printk("%s,%d,slots=%p,vpuid=%d,gfn=%llx\n",__FUNCTION__,__LINE__,slots,vcpu->vcpu_id,gfn);
}
return gfn_to_hva_memslot_prot(slot, gfn, writable);
}
出问题时的，memslot为空，但是memslots不为空，但是和当前内存中的不一致，
[ 5621.133030] kvm_vcpu_gfn_to_hva_prot,1325,slots=ffff88011b6d0000,vpuid=0,gfn=84
而coredump中读出的memslots
crash> struct -x kvm.memslots 0xffff880118380000
memslots = {0xffff88011ee40000, 0xffff88011ee10000}
0xffff88011ee40000这个memslots里面是有对应gfn的memslot，0x ffff88011b6d0000是没有的
原因是coredump中看到的是另外一个cpu在install_new_memslots做了更新的memslots，
而发生异常时的cpu读取的是更新前的memslots。
另外一个cpu的堆栈
#0 [ffffc900009ffb08] __schedule at ffffffff818f8c6e
#1 [ffffc900009ffb98] preempt_schedule_common at ffffffff818f94ad
#2 [ffffc900009ffba8] _cond_resched at ffffffff818f94d8
#3 [ffffc900009ffbb0] wait_for_completion at ffffffff818fa69c
#4 [ffffc900009ffc08] __synchronize_srcu at ffffffff810f1447
#5 [ffffc900009ffc70] install_new_memslots at ffffffffc05a5d89 [kvm]
#6 [ffffc900009ffc90] __kvm_set_memory_region at ffffffffc05a7061 [kvm]
#7 [ffffc900009ffda0] kvm_set_memory_region at ffffffffc05a72d6 [kvm]
#8 [ffffc900009ffdc0] kvm_vm_ioctl at ffffffffc05a994b [kvm]
#9 [ffffc900009ffe80] do_vfs_ioctl at ffffffff8126bbc9
#10 [ffffc900009fff00] sys_ioctl at ffffffff8126c1b4
#11 [ffffc900009fff38] do_syscall_64 at ffffffff810036fe
#12 [ffffc900009fff50] entry_SYSCALL_64_after_hwframe at ffffffff81a00081
static struct kvm_memslots *install_new_memslots(struct kvm *kvm,
int as_id, struct kvm_memslots *slots)
{
struct kvm_memslots *old_memslots = __kvm_memslots(kvm, as_id);
/*
* Set the low bit in the generation, which disables SPTE caching
* until the end of synchronize_srcu_expedited.
*/
WARN_ON(old_memslots->generation & 1);
slots->generation = old_memslots->generation + 1;
rcu_assign_pointer(kvm->memslots[as_id], slots);
synchronize_srcu_expedited(&kvm->srcu);
实际中另外一个cpu在更新memslots之前是加了slots的锁，lock debug没有打开时其实是检测不到错误的。
而更新过程中，数据可能是不准确的。
int kvm_set_memory_region(struct kvm *kvm,
const struct kvm_userspace_memory_region *mem)
{
int r;
mutex_lock(&kvm->slots_lock);
r = __kvm_set_memory_region(kvm, mem);
mutex_unlock(&kvm->slots_lock);
return r;
}
-------------------------------------------------------------------------------------------------------------------------------------
本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出
的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、
或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本
邮件！
This e-mail and its attachments contain confidential information from New H3C, which is
intended only for the person or entity whose address is listed above. Any use of the
information contained herein in any way (including, but not limited to, total or partial
disclosure, reproduction, or dissemination) by persons other than the intended
recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender
by phone or email immediately and delete it!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.freedesktop.org/archives/intel-gvt-dev/attachments/20181129/c04f5ed3/attachment-0001.html>
More information about the intel-gvt-dev
mailing list