<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"> <head> <meta http-equiv="Content-Type" content="text/html; charset=gb2312"> <meta name="Generator" content="Microsoft Word 14 (filtered medium)"> <style></style> </head> <body lang="ZH-CN" link="#0563C1" vlink="#954F72"> <div class="WordSection1"> The guest is windows 10.<o:p></o:p> There is no this issue on windows 7 guest.<o:p></o:p> <o:p> </o:p> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> 发件人: Zhang, Xiong Y [mailto:xiong.y.zhang@intel.com] 发送时间: 2018年11月29日 9:38 收件人: jianghuaping (Cloud); Yuan, Hang; 'intel-gvt-dev@lists.freedesktop.org' 抄送: Peng, Chao P; wangxuan (Cloud); bailin (Cloud); Zeng, Harris; Wang, Hongbo; wangtao (Cloud); daishijun (Cloud); yandehan (CTS); Zhang, Xiong Y 主题: RE: KVM Intel graphic passthrough cause qemu pause<o:p></o:p> </div> </div> <o:p> </o:p> When this issue happens, it seems the guest Win7 OS doesn’t start.<o:p></o:p> If this issue happens during guest OS boot up, I suggest to try win8 or win10, I never try win7.<o:p></o:p> <a name="_MailEndCompose"></a><o:p> </o:p> thanks<o:p></o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> <a name="_____replyseparator"></a>From: intel-gvt-dev [<a href="mailto:intel-gvt-dev-bounces@lists.freedesktop.org">mailto:intel-gvt-dev-bounces@lists.freedesktop.org</a>] On Behalf Of Jianghuaping Sent: Wednesday, November 28, 2018 7:18 PM To: Yuan, Hang <<a href="mailto:hang.yuan@intel.com">hang.yuan@intel.com</a>>; 'intel-gvt-dev@lists.freedesktop.org' <<a href="mailto:intel-gvt-dev@lists.freedesktop.org">intel-gvt-dev@lists.freedesktop.org</a>> Cc: Peng, Chao P <<a href="mailto:chao.p.peng@intel.com">chao.p.peng@intel.com</a>>; Wangxuan <<a href="mailto:wang.xuan@h3c.com">wang.xuan@h3c.com</a>>; Bailin <<a href="mailto:berlin@h3c.com">berlin@h3c.com</a>>; Zeng, Harris <<a href="mailto:harris.zeng@intel.com">harris.zeng@intel.com</a>>; Wang, Hongbo <<a href="mailto:hongbo.wang@intel.com">hongbo.wang@intel.com</a>>; Wangtao <<a href="mailto:wang.taoD@h3c.com">wang.taoD@h3c.com</a>>; Daishijun <<a href="mailto:daishijun@h3c.com">daishijun@h3c.com</a>>; Yandehan <<a href="mailto:ydhan@h3c.com">ydhan@h3c.com</a>> Subject: 答复: KVM Intel graphic passthrough cause qemu pause<o:p></o:p> </div> </div> <o:p> </o:p> Hi Henry<o:p></o:p> Double check,<o:p></o:p> What you mean is the syslog under /var/log/syslog? We can get it when this issue re-appeared.<o:p></o:p> <o:p> </o:p> Thanks<o:p></o:p> Jiang <o:p></o:p> <o:p> </o:p> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> 发件人: Yuan, Hang [<a href="mailto:hang.yuan@intel.com">mailto:hang.yuan@intel.com</a>] 发送时间: 2018年11月28日 18:55 收件人: jianghuaping (Cloud); 'intel-gvt-dev@lists.freedesktop.org' 抄送: Zeng, Harris; Peng, Chao P; daishijun (Cloud); Wang, Hongbo; bailin (Cloud); yandehan (CTS); wangxuan (Cloud); wangtao (Cloud) 主题: RE: KVM Intel graphic passthrough cause qemu pause<o:p></o:p> </div> </div> <o:p> </o:p> Do you have host kernel log? So we can help to take a look from GVT perspective.<o:p></o:p> <o:p> </o:p> Regards,<o:p></o:p> Henry<o:p></o:p> <o:p> </o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Jianghuaping [<a href="mailto:jiang.huaping@h3c.com">mailto:jiang.huaping@h3c.com</a>] Sent: Wednesday, November 28, 2018 5:56 PM To: Yuan, Hang <<a href="mailto:hang.yuan@intel.com">hang.yuan@intel.com</a>>; 'intel-gvt-dev@lists.freedesktop.org' <<a href="mailto:intel-gvt-dev@lists.freedesktop.org">intel-gvt-dev@lists.freedesktop.org</a>> Cc: Zeng, Harris <<a href="mailto:harris.zeng@intel.com">harris.zeng@intel.com</a>>; Peng, Chao P <<a href="mailto:chao.p.peng@intel.com">chao.p.peng@intel.com</a>>; Daishijun <<a href="mailto:daishijun@h3c.com">daishijun@h3c.com</a>>; Wang, Hongbo <<a href="mailto:hongbo.wang@intel.com">hongbo.wang@intel.com</a>>; Bailin <<a href="mailto:berlin@h3c.com">berlin@h3c.com</a>>; Yandehan <<a href="mailto:ydhan@h3c.com">ydhan@h3c.com</a>>; Wangxuan <<a href="mailto:wang.xuan@h3c.com">wang.xuan@h3c.com</a>>; Wangtao <<a href="mailto:wang.taoD@h3c.com">wang.taoD@h3c.com</a>> Subject: 答复: KVM Intel graphic passthrough cause qemu pause<o:p></o:p> </div> </div> <o:p> </o:p> Hi Hang<o:p></o:p> The attached is two logs.<o:p></o:p> v288C1A78-C34C-11E8-94EF-CEFAE916D900.log : suberror 1 qemu log<o:p></o:p> v8FE70BB0-BC03-11E8-874B-B4FF2E460600.log : suberror 3 qemu log<o:p></o:p> <o:p> </o:p> old kvm parameter is :<o:p></o:p> /usr/bin/kvm -name guest=v03000200-0400-0500-0006-000700080009,debug-threads=on -S -object secret,id=masterKey0,format=raw,file=/var/run/lib/libvirt/qemu/domain-1-v03000200-0400-0500-/master-key.aes -machine pc-i440fx-2.12,accel=kvm,usb=off,dump-guest-core=off -cpu host,hv_time,hv_relaxed,hv_spinlocks=0x2000,host-cache-info=on,l3-cache=off -m 3380 -realtime mlock=off -smp 2,sockets=1,cores=2,threads=1 -uuid 078fe81f-f60a-45ab-9aaa-b6b97028dc35 -display none -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/run/lib/libvirt/qemu/domain-1-v03000200-0400-0500-/monitor.sock,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -chardev socket,id=charmonitor_cas,path=/var/run/lib/libvirt/qemu/domain-1-v03000200-0400-0500-/monitor.sock.cas,server,nowait -mon chardev=charmonitor_cas,id=monitor_cas,mode=control -rtc base=localtime,clock=vm,driftfix=slew -no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global PIIX4_PM.disable_s4=1 -boot strict=on -device piix3-usb-uhci,id=usb,pci_hotpluggable=off,bus=pci.0,addr=0x1.0x2 -device usb-ehci,id=usb1,pci_hotpluggable=off,bus=pci.0,addr=0x4 -device nec-usb-xhci,id=usb2,pci_hotpluggable=off,bus=pci.0,addr=0x5 -device virtio-scsi-pci,id=scsi1,pci_hotpluggable=off,bus=pci.0,addr=0x6 -device virtio-serial-pci,id=virtio-serial0,pci_hotpluggable=off,bus=pci.0,addr=0x7 -device usb-hub,id=hub0,bus=usb.0,port=1 -drive file=/vms/idv/data/win7,format=qcow2,if=none,id=drive-virtio-disk0,cache=directsync,aio=native -device virtio-blk-pci,scsi=off,bus=pci.0,addr=0x8,pci_hotpluggable=off,drive=drive-virtio-disk0,id=virtio-disk0,bootindex=1 -drive file=/vms/isos/virtio-win7.vfd,format=raw,if=none,id=drive-fdc0-0-0,readonly=on,cache=directsync,aio=native -global isa-fdc.driveA=drive-fdc0-0-0 -global isa-fdc.bootindexA=4 -drive if=none,id=drive-ide0-0-0,readonly=on -device ide-cd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=2 -netdev tap,fd=51,id=hostnet0,vhost=on,vhostfd=52 -device virtio-net-pci,pci_hotpluggable=off,netdev=hostnet0,id=net0,mac=9c:06:1b:6f:93:5f,bus=pci.0,addr=0x3,bootindex=3 -chardev pty,id=charserial0 -device isa-serial,chardev=charserial0,id=serial0 -chardev socket,id=charchannel0,path=/var/lib/libvirt/qemu/v03000200-0400-0500-0006-000700080009.agent,server,nowait -device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 -device usb-tablet,id=input0,bus=usb.0,port=2 -device vfio-pci,pci_hotpluggable=off,host=00:02.0,id=hostdev0,bus=pci.0,addr=0x2 -device vfio-pci,pci_hotpluggable=off,host=00:1f.3,id=hostdev1,bus=pci.0,addr=0x9 -device usb-host,hostbus=1,hostaddr=3,id=hostdev2,bus=usb.0,port=1.1 -device usb-host,hostbus=1,hostaddr=2,id=hostdev3,bus=usb.0,port=1.2 -set device.hostdev0.x-igd-opregion=on -set device.hostdev0.x-igd-gms=1 -msg timestamp=on<o:p></o:p> <o:p> </o:p> <o:p> </o:p> Thanks<o:p></o:p> Jiang<o:p></o:p> <o:p> </o:p> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> 发件人: Yuan, Hang [<a href="mailto:hang.yuan@intel.com">mailto:hang.yuan@intel.com</a>] 发送时间: 2018年11月28日 15:52 收件人: jianghuaping (Cloud); 'intel-gvt-dev@lists.freedesktop.org' 抄送: Zeng, Harris; Peng, Chao P; daishijun (Cloud); Wang, Hongbo; bailin (Cloud) 主题: RE: KVM Intel graphic passthrough cause qemu pause<o:p></o:p> </div> </div> <o:p> </o:p> Hi Huaping,<o:p></o:p> <o:p> </o:p> What’s your Qemu parameters to create the VM? Do you have host kernel log to share?<o:p></o:p> <o:p> </o:p> Regards,<o:p></o:p> Henry<o:p></o:p> <o:p> </o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: intel-gvt-dev [<a href="mailto:intel-gvt-dev-bounces@lists.freedesktop.org">mailto:intel-gvt-dev-bounces@lists.freedesktop.org</a>] On Behalf Of Jianghuaping Sent: Wednesday, November 28, 2018 10:34 AM To: 'intel-gvt-dev@lists.freedesktop.org' <<a href="mailto:intel-gvt-dev@lists.freedesktop.org">intel-gvt-dev@lists.freedesktop.org</a>> Cc: Zeng, Harris <<a href="mailto:harris.zeng@intel.com">harris.zeng@intel.com</a>>; Peng, Chao P <<a href="mailto:chao.p.peng@intel.com">chao.p.peng@intel.com</a>>; Daishijun <<a href="mailto:daishijun@h3c.com">daishijun@h3c.com</a>>; Wang, Hongbo <<a href="mailto:hongbo.wang@intel.com">hongbo.wang@intel.com</a>>; Bailin <<a href="mailto:berlin@h3c.com">berlin@h3c.com</a>> Subject: KVM Intel graphic passthrough cause qemu pause<o:p></o:p> </div> </div> <o:p> </o:p> Hello Intel GVT experts.<o:p></o:p> we are using Intel skylake I3 processor to run KVM virtualization(1 Centos Hypervisor +1 Windows guest). Intel graphic in I3 processor will be passed through Qemu guest(this is a Windows 10 1703 guest). We found kvm will appear “suberror 3” or “suberror 1”, and Qemu will pause, when win 10 guest reboot or resume from sleep. Looks like this issue related to EPT miconfig, could you please help us on this issue? The following are detail information.<o:p></o:p> In these days, Intel kvm expert:Peng chao is also helping analyzing this issue.<o:p></o:p> ----------------------------------------------------------------------<o:p></o:p> Linux Kernel &Kvm version:<o:p></o:p> [root@cvknode31 ~]# virsh version<o:p></o:p> Compiled against library: libvirt 4.0.0<o:p></o:p> Using library: libvirt 4.0.0<o:p></o:p> Using API: QEMU 4.0.0<o:p></o:p> Running hypervisor: QEMU 2.12.1<o:p></o:p> [root@cvknode31 ~]# uname -a<o:p></o:p> Linux cvknode31 4.14.0-generic #862.el7 SMP Wed May 23 19:40:09 CST 2018 x86_64 x86_64 x86_64 GNU/Linux<o:p></o:p> [root@cvknode31 ~]# cat /etc/redhat-release <o:p></o:p> CentOS Linux release 7.5.1804 (Core)<o:p></o:p> <o:p> </o:p> <o:p> </o:p> -----------------------------------------------------------------------------<o:p></o:p> Reboot caused suberror 3 qemu log<o:p></o:p> KVM internal error. Suberror: 3<o:p></o:p> extra data[0]: 80000b0e<o:p></o:p> extra data[1]: 31<o:p></o:p> extra data[2]: 683<o:p></o:p> extra data[3]: 88c70<o:p></o:p> EAX=00000000 EBX=87b84120 ECX=87b862c0 EDX=80843120<o:p></o:p> ESI=87b2afac EDI=80843120 EBP=87a3aa44 ESP=87a3aa40<o:p></o:p> EIP=873d5cfa EFL=00210202 [-------] CPL=0 II=0 A20=1 SMM=0 HLT=0<o:p></o:p> ES =0023 00000000 ffffffff 00c0f300 DPL=3 DS [-WA]<o:p></o:p> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]<o:p></o:p> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]<o:p></o:p> DS =0023 00000000 ffffffff 00c0f300 DPL=3 DS [-WA]<o:p></o:p> FS =0030 87b84000 00004a20 00409300 DPL=0 DS [-WA]<o:p></o:p> GS =0000 00000000 ffffffff 00c00000<o:p></o:p> LDT=0000 00000000 ffffffff 00c00000<o:p></o:p> TR =0028 87b88a40 000020ab 00008b00 DPL=0 TSS32-busy<o:p></o:p> GDT= 87b8e5c0 000003ff<o:p></o:p> IDT= 87b8e9c0 000007ff<o:p></o:p> CR0=80010033 CR2=87b2afb0 CR3=001a8000 CR4=001406e8<o:p></o:p> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 <o:p></o:p> DR6=00000000ffff0ff0 DR7=0000000000000400<o:p></o:p> EFER=0000000000000800<o:p></o:p> <o:p> </o:p> ---------------------------------------------------------------------------------------<o:p></o:p> Sleep caused suberror 1 qemu log<o:p></o:p> KVM internal error. Suberror: 1<o:p></o:p> emulation failure<o:p></o:p> EAX=00010008 EBX=00000000 ECX=00024000 EDX=00000000<o:p></o:p> ESI=00000000 EDI=00000000 EBP=b4237a1c ESP=b42379e8<o:p></o:p> EIP=ffd03000 EFL=00010246 [---Z-P-] CPL=0 II=0 A20=1 SMM=0 HLT=0<o:p></o:p> ES =0023 00000000 ffffffff 00c0f300 DPL=3 DS [-WA]<o:p></o:p> CS =0008 00000000 ffffffff 00c09b00 DPL=0 CS32 [-RA]<o:p></o:p> SS =0010 00000000 ffffffff 00c09300 DPL=0 DS [-WA]<o:p></o:p> DS =0023 00000000 ffffffff 00c0f300 DPL=3 DS [-WA]<o:p></o:p> FS =0030 81158000 00004a60 00409300 DPL=0 DS [-WA]<o:p></o:p> GS =0000 00000000 ffffffff 00c00000<o:p></o:p> LDT=0000 00000000 ffffffff 00c00000<o:p></o:p> TR =0028 8113f000 000020ab 00008b00 DPL=0 TSS32-busy<o:p></o:p> GDT= 81151000 000003ff<o:p></o:p> IDT= 81151400 000007ff<o:p></o:p> CR0=80010033 CR2=8136b000 CR3=9ffd2320 CR4=001406e9<o:p></o:p> DR0=0000000000000000 DR1=0000000000000000 DR2=0000000000000000 DR3=0000000000000000 <o:p></o:p> DR6=00000000ffff0ff0 DR7=0000000000000400<o:p></o:p> EFER=0000000000000800<o:p></o:p> <o:p> </o:p> Thanks <o:p></o:p> Jiang huaping<o:p></o:p> <div> <div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0cm 0cm 0cm"> 发件人: bailin (Cloud) 发送时间: 2018年11月28日 8:08 收件人: Peng, Chao P; Zeng, Harris 抄送: jianghuaping (Cloud); changlimin (Cloud) 主题: 答复: 答复: pause问题<o:p></o:p> </div> </div> <o:p> </o:p> 用户态的堆栈如下,感觉这种操作普遍并且正常.<o:p></o:p> <o:p> </o:p> 一个cpu在pci的空间访问,导致修改memslot<o:p></o:p> #0 0x00007fee3ede95d7 in ioctl () from /lib64/libc.so.6<o:p></o:p> #1 0x00005624101a28d7 in kvm_vm_ioctl (s=0x8ec8, s@entry=0x5624132e8e20, type=881622144, type@entry=1075883590)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:2075<o:p></o:p> #2 0x00005624101a3207 in kvm_set_user_memory_region (slot=slot@entry=0x5624132ea110, kml=0x5624132e9ec0)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:277<o:p></o:p> #3 0x00005624101a3640 in kvm_set_phys_mem (kml=0x5624132e9ec0, section=<optimized out>, add=true)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:774<o:p></o:p> #4 0x00005624101929e1 in address_space_update_topology_pass (as=as@entry=0x562410e3b5e0 <address_space_memory>, adding=adding@entry=true, <o:p></o:p> new_view=0x7fee106a03d0, new_view=0x7fee106a03d0, old_view=<optimized out>, old_view=<optimized out>)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:933<o:p></o:p> #5 0x0000562410192d94 in address_space_set_flatview (as=as@entry=0x562410e3b5e0 <address_space_memory>)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:1008<o:p></o:p> #6 0x00005624101959c0 in memory_region_transaction_commit ()<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:1060<o:p></o:p> #7 0x0000562410391ab8 in pci_update_vga (pci_dev=0x562414c97bc0)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/hw/pci/pci.c:1167<o:p></o:p> #8 0x0000562410392913 in pci_update_vga (pci_dev=0x562414c97bc0)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/hw/pci/pci.c:1161<o:p></o:p> #9 pci_update_mappings (d=d@entry=0x562414c97bc0)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/hw/pci/pci.c:1333<o:p></o:p> #10 0x0000562410392f59 in pci_default_write_config (d=d@entry=0x562414c97bc0, addr=addr@entry=4, val_in=val_in@entry=1024, l=2)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/hw/pci/pci.c:1380<o:p></o:p> #11 0x00005624101dd49c in vfio_pci_write_config (pdev=0x562414c97bc0, addr=4, val=1024, len=<optimized out>)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/hw/vfio/pci.c:1222<o:p></o:p> #12 0x00005624103998da in pci_host_config_write_common (pci_dev=0x562414c97bc0, addr=4, limit=256, val=1024, len=2)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/hw/pci/pci_host.c:66<o:p></o:p> #13 0x000056241019341b in memory_region_write_accessor (mr=0x5624136715b0, addr=0, value=<optimized out>, size=2, shift=<optimized out>, <o:p></o:p> mask=<optimized out>, attrs=...) at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:530<o:p></o:p> #14 0x0000562410191029 in access_with_adjusted_size (addr=addr@entry=0, value=value@entry=0x7fee25dbb4a8, size=size@entry=2, <o:p></o:p> access_size_min=<optimized out>, access_size_max=<optimized out>, <o:p></o:p> access_fn=access_fn@entry=0x5624101933a0 <memory_region_write_accessor>, mr=mr@entry=0x5624136715b0, attrs=attrs@entry=...)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:597<o:p></o:p> #15 0x0000562410195f75 in memory_region_dispatch_write (mr=mr@entry=0x5624136715b0, addr=addr@entry=0, data=1024, size=size@entry=2, <o:p></o:p> attrs=attrs@entry=...) at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/memory.c:1474<o:p></o:p> #16 0x0000562410131342 in flatview_write_continue (mr=0x5624136715b0, l=2, addr1=0, len=2, buf=0x7fee459ed000 "", attrs=..., addr=3324, <o:p></o:p> fv=0x7fee1c594e40) at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/exec.c:3166<o:p></o:p> #17 flatview_write (fv=0x7fee1c594e40, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/exec.c:3216<o:p></o:p> #18 0x000056241013507f in address_space_write (as=<optimized out>, addr=<optimized out>, attrs=..., buf=<optimized out>, len=<optimized out>)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/exec.c:3332<o:p></o:p> #19 0x0000562410135125 in address_space_rw (as=<optimized out>, addr=addr@entry=3324, attrs=..., attrs@entry=..., buf=<optimized out>, <o:p></o:p> len=len@entry=2, is_write=is_write@entry=true) at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/exec.c:3343<o:p></o:p> #20 0x00005624101a4ff6 in kvm_handle_io (count=1, size=2, direction=<optimized out>, data=<optimized out>, attrs=..., port=3324)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:1730<o:p></o:p> #21 kvm_cpu_exec (cpu=cpu@entry=0x56241344c3e0)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:1970<o:p></o:p> #22 0x000056241017f9e6 in qemu_kvm_cpu_thread_fn (arg=0x56241344c3e0)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/cpus.c:1229<o:p></o:p> #23 0x00007fee3f0c8e25 in start_thread () from /lib64/libpthread.so.0<o:p></o:p> #24 0x00007fee3edf2bad in clone () from /lib64/libc.so.6<o:p></o:p> <o:p> </o:p> <o:p> </o:p> 异常的cpu的qemu堆栈<o:p></o:p> #0 0x00007fee3ede95d7 in ioctl () from /lib64/libc.so.6<o:p></o:p> #1 0x00005624101a4d02 in kvm_vcpu_ioctl (cpu=0x0, cpu@entry=0x5624133fe150, type=136, type@entry=44672)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:2093<o:p></o:p> #2 0x00005624101a4e5f in kvm_cpu_exec (cpu=cpu@entry=0x5624133fe150)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/accel/kvm/kvm-all.c:1930<o:p></o:p> #3 0x000056241017f9e6 in qemu_kvm_cpu_thread_fn (arg=0x5624133fe150)<o:p></o:p> at /home/uis-enterprise/f-idv/daemon/qemu/qemu-2.12/rpmbuild/BUILD/qemu-2.12/cpus.c:1229<o:p></o:p> #4 0x00007fee3f0c8e25 in start_thread () from /lib64/libpthread.so.0<o:p></o:p> #5 0x00007fee3edf2bad in clone () from /lib64/libc.so.6<o:p></o:p> <o:p> </o:p> <o:p> </o:p> <o:p> </o:p> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> 发件人: Peng, Chao P [<a href="mailto:chao.p.peng@intel.com">mailto:chao.p.peng@intel.com</a>] 发送时间: 2018年11月26日 16:40 收件人: Zeng, Harris <<a href="mailto:harris.zeng@intel.com">harris.zeng@intel.com</a>>; bailin (Cloud) <<a href="mailto:berlin@h3c.com">berlin@h3c.com</a>> 抄送: jianghuaping (Cloud) <<a href="mailto:jiang.huaping@h3c.com">jiang.huaping@h3c.com</a>>; changlimin (Cloud) <<a href="mailto:changlimin@h3c.com">changlimin@h3c.com</a>> 主题: RE: 答复: pause问题<o:p></o:p> </div> </div> <o:p> </o:p> 我看了代码，目前的现象和RCU的本来设计也是吻合的。所以即使memslot 为空， 还不好说这是问题。<o:p></o:p> <o:p> </o:p> 一个思路是看看新添加memslot的操作是什么时机触发，是否与guest的特定行为有关？<o:p></o:p> <o:p> </o:p> <div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt"> <div> <div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm"> From: Zeng, Harris Sent: Saturday, November 24, 2018 10:13 PM To: Bailin <<a href="mailto:berlin@h3c.com">berlin@h3c.com</a>>; Peng, Chao P <<a href="mailto:chao.p.peng@intel.com">chao.p.peng@intel.com</a>> Cc: Jianghuaping <<a href="mailto:jiang.huaping@h3c.com">jiang.huaping@h3c.com</a>>; Changlimin <<a href="mailto:changlimin@h3c.com">changlimin@h3c.com</a>> Subject: Re: 答复: pause问题<o:p></o:p> </div> </div> <o:p> </o:p> Hi Chao,<o:p></o:p> <div> <o:p> </o:p> </div> <div> Can you help to advise?<o:p></o:p> </div> <div> <o:p> </o:p> </div> <div> Thanks,<o:p></o:p> </div> <div> Harris<o:p></o:p> <div id="AppleMailSignature"> Sent from my mobile phone.<o:p></o:p> </div> <div> 在 2018年11月22日，上午10:47，Bailin <<a href="mailto:berlin@h3c.com">berlin@h3c.com</a>> 写道：<o:p></o:p> </div> <blockquote style="margin-top:5.0pt;margin-bottom:5.0pt"> <div> 不知道你们那边有没有具体线索,目前我们这边分析是内核函数kvm_vcpu_gfn_to_hva_prot可能有缺陷<o:p></o:p> 目前我们这边没有更好的修改思路,也无法确认为什么这里返回错误,会最终导致ept misconfig.<o:p></o:p> <o:p></o:p> <o:p></o:p> 出现问题时,memslot为空，但是在coredump中看到的不为空的准确原因明确了<o:p></o:p> <o:p></o:p> 出问题的cpu的堆栈<o:p></o:p> #0 [ffffc900009d37a8] machine_kexec at ffffffff8105da32<o:p></o:p> #1 [ffffc900009d3800] __crash_kexec at ffffffff8111a9ad<o:p></o:p> #2 [ffffc900009d38c8] panic at ffffffff81085e2c<o:p></o:p> #3 [ffffc900009d3948] paging64_walk_addr_generic at ffffffffc05cf4b3 [kvm]<o:p></o:p> #4 [ffffc900009d3a30] paging64_gva_to_gpa at ffffffffc05cf6df [kvm]<o:p></o:p> #5 [ffffc900009d3b48] emulator_read_write_onepage at ffffffffc05b6f5b [kvm]<o:p></o:p> #6 [ffffc900009d3ba8] emulator_read_write at ffffffffc05b72d2 [kvm]<o:p></o:p> #7 [ffffc900009d3bf8] segmented_read at ffffffffc05da519 [kvm]<o:p></o:p> #8 [ffffc900009d3c38] x86_emulate_insn at ffffffffc05de3f0 [kvm]<o:p></o:p> #9 [ffffc900009d3c88] x86_emulate_instruction at ffffffffc05c0149 [kvm]<o:p></o:p> #10 [ffffc900009d3cf8] kvm_mmu_page_fault_ept_violation at ffffffffc05cd9ea [kvm]<o:p></o:p> #11 [ffffc900009d3d30] kvm_arch_vcpu_ioctl_run at ffffffffc05c4510 [kvm]<o:p></o:p> #12 [ffffc900009d3df0] kvm_vcpu_ioctl at ffffffffc05aa197 [kvm]<o:p></o:p> #13 [ffffc900009d3e80] do_vfs_ioctl at ffffffff8126bbc9<o:p></o:p> #14 [ffffc900009d3f00] sys_ioctl at ffffffff8126c1b4<o:p></o:p> #15 [ffffc900009d3f38] do_syscall_64 at ffffffff810036fe<o:p></o:p> #16 [ffffc900009d3f50] entry_SYSCALL_64_after_hwframe at ffffffff81a00081<o:p></o:p> <o:p></o:p> <o:p></o:p> 发生异常的cpu调用 这个函数，获取memslots，是没有问题的，这个函数除了索引数组，<o:p></o:p> 主要是在内核的lock debug打开时检查kvm->srcu，kvm->slots_lock的状态，而一般情况下lock debug是不打开的<o:p></o:p> static inline struct kvm_memslots *__kvm_memslots(struct kvm *kvm, int as_id)<o:p></o:p> {<o:p></o:p> return srcu_dereference_check(kvm->memslots[as_id], &kvm->srcu,<o:p></o:p> lockdep_is_held(&kvm->slots_lock) ||<o:p></o:p> !refcount_read(&kvm->users_count));<o:p></o:p> }<o:p></o:p> 实际上是memslots取错了，导致根据gfn查找的memslot错了，修改了调试信息<o:p></o:p> unsigned long kvm_vcpu_gfn_to_hva_prot(struct kvm_vcpu *vcpu, gfn_t gfn, bool *writable)<o:p></o:p> {<o:p></o:p> //struct kvm_memory_slot *slot = kvm_vcpu_gfn_to_memslot(vcpu, gfn);<o:p></o:p> struct kvm_memslots *slots = kvm_vcpu_memslots(vcpu);<o:p></o:p> struct kvm_memory_slot *slot = __gfn_to_memslot(slots, gfn);<o:p></o:p> <o:p></o:p> if(NULL == slot){<o:p></o:p> printk("%s,%d,slots=%p,vpuid=%d,gfn=%llx\n",__FUNCTION__,__LINE__,slots,vcpu->vcpu_id,gfn);<o:p></o:p> }<o:p></o:p> <o:p></o:p> return gfn_to_hva_memslot_prot(slot, gfn, writable);<o:p></o:p> }<o:p></o:p> <o:p></o:p> 出问题时的，memslot为空，但是memslots不为空，但是和当前内存中的不一致，<o:p></o:p> [ 5621.133030] kvm_vcpu_gfn_to_hva_prot,1325,slots=ffff88011b6d0000,vpuid=0,gfn=84<o:p></o:p> <o:p></o:p> 而coredump中读出的memslots<o:p></o:p> crash> struct -x kvm.memslots 0xffff880118380000<o:p></o:p> memslots = {0xffff88011ee40000, 0xffff88011ee10000}<o:p></o:p> <o:p></o:p> 0xffff88011ee40000这个memslots里面是有对应gfn的memslot，0x ffff88011b6d0000是没有的<o:p></o:p> <o:p></o:p> 原因是coredump中看到的是另外一个cpu在install_new_memslots做了更新的memslots，<o:p></o:p> 而发生异常时的cpu读取的是更新前的memslots。<o:p></o:p> 另外一个cpu的堆栈<o:p></o:p> #0 [ffffc900009ffb08] __schedule at ffffffff818f8c6e<o:p></o:p> #1 [ffffc900009ffb98] preempt_schedule_common at ffffffff818f94ad<o:p></o:p> #2 [ffffc900009ffba8] _cond_resched at ffffffff818f94d8<o:p></o:p> #3 [ffffc900009ffbb0] wait_for_completion at ffffffff818fa69c<o:p></o:p> #4 [ffffc900009ffc08] __synchronize_srcu at ffffffff810f1447<o:p></o:p> #5 [ffffc900009ffc70] install_new_memslots at ffffffffc05a5d89 [kvm]<o:p></o:p> #6 [ffffc900009ffc90] __kvm_set_memory_region at ffffffffc05a7061 [kvm]<o:p></o:p> #7 [ffffc900009ffda0] kvm_set_memory_region at ffffffffc05a72d6 [kvm]<o:p></o:p> #8 [ffffc900009ffdc0] kvm_vm_ioctl at ffffffffc05a994b [kvm]<o:p></o:p> #9 [ffffc900009ffe80] do_vfs_ioctl at ffffffff8126bbc9<o:p></o:p> #10 [ffffc900009fff00] sys_ioctl at ffffffff8126c1b4<o:p></o:p> #11 [ffffc900009fff38] do_syscall_64 at ffffffff810036fe<o:p></o:p> #12 [ffffc900009fff50] entry_SYSCALL_64_after_hwframe at ffffffff81a00081<o:p></o:p> <o:p></o:p> <o:p></o:p> static struct kvm_memslots *install_new_memslots(struct kvm *kvm,<o:p></o:p> int as_id, struct kvm_memslots *slots)<o:p></o:p> {<o:p></o:p> struct kvm_memslots *old_memslots = __kvm_memslots(kvm, as_id);<o:p></o:p> <o:p></o:p> /*<o:p></o:p> * Set the low bit in the generation, which disables SPTE caching<o:p></o:p> * until the end of synchronize_srcu_expedited.<o:p></o:p> */<o:p></o:p> WARN_ON(old_memslots->generation & 1);<o:p></o:p> slots->generation = old_memslots->generation + 1;<o:p></o:p> <o:p></o:p> rcu_assign_pointer(kvm->memslots[as_id], slots);<o:p></o:p> synchronize_srcu_expedited(&kvm->srcu);<o:p></o:p> <o:p></o:p> <o:p></o:p> 实际中另外一个cpu在更新memslots之前是加了slots的锁，lock debug没有打开时其实是检测不到错误的。<o:p></o:p> 而更新过程中，数据可能是不准确的。<o:p></o:p> int kvm_set_memory_region(struct kvm *kvm,<o:p></o:p> const struct kvm_userspace_memory_region *mem)<o:p></o:p> {<o:p></o:p> int r;<o:p></o:p> <o:p></o:p> mutex_lock(&kvm->slots_lock);<o:p></o:p> r = __kvm_set_memory_region(kvm, mem);<o:p></o:p> mutex_unlock(&kvm->slots_lock);<o:p></o:p> return r;<o:p></o:p> }<o:p></o:p> <o:p></o:p> <o:p></o:p> <o:p></o:p> <o:p></o:p> ------------------------------------------------------------------------------------------------------------------------------------- 本邮件及其附件含有新华三集团的保密信息，仅限于发送给上面地址中列出 的个人或群组。禁止任何其他人以任何形式使用（包括但不限于全部或部分地泄露、复制、 或散发）本邮件中的信息。如果您错收了本邮件，请您立即电话或邮件通知发件人并删除本 邮件！ This e-mail and its attachments contain confidential information from New H3C, which is intended only for the person or entity whose address is listed above. Any use of the information contained herein in any way (including, but not limited to, total or partial disclosure, reproduction, or dissemination) by persons other than the intended recipient(s) is prohibited. If you receive this e-mail in error, please notify the sender by phone or email immediately and delete it! <o:p></o:p> </div> </blockquote> </div> </div> </div> </div> </div> </div> </body> </html>