[Spice-devel] Postcopy+spice crash

[Uri Lublin]

On 12/05/2016 11:46 AM, Dr. David Alan Gilbert wrote:
> * Gerd Hoffmann (kraxel at redhat.com) wrote:
>> On Fr, 2016-12-02 at 17:44 +0000, Dr. David Alan Gilbert wrote:
>>> Hi Gerd,
>>>   I've got a moderately repeatable crash with spice playing
>>> a video + postcopy.  Some of the time I just get a warning
>>> (that I also get in precopy) but sometimes it turns into
>>> a backtrace;
>>>
>>> This is:
>>>   f24 guest playing youtube fullscreen.
>>>   migration between 2.7.0<->current head (had crash both ways)
>>>
>>> The warning I get with precopy most of the time is:
>>>   ./x86_64-softmmu/qemu-system-x86_64:26921): Spice-Warning **: red_memslots.c:94:validate_virt: virtual address out of range
>>
>> That is in spice-server.  Which version do you run?
>
> From the bottom of the post; spice-server-devel-0.12.4-19.el7.x86_64 (rhel 7)
>
>> Adding spice-devel to Cc:
>>
>>>     virt=0x7f5397ed002a+0x2925ff31 slot_id=1 group_id=1
>>>     slot=0x7f5397c00000-0x7f539bbfe000 delta=0x7f5397c00000
>>
>> Base address looks sane.
>> Size (0x2925ff31) is bogous.
>>
>> On a quick glance I'd blame the guest for sending corrupted commands.
>> Strange though that it happens on migration only, so there could be
>> a host issue too.  Or a timing issue triggered by migration.
>>
>> Which migration phase?
>
> This is the point at which it switches over in postcopy.

It looks like it's the vmstate (post) load phase of the qxl device on
destination host.
Maybe if you trace qxl device save/load related functions
on both src and dst hosts you'll see a difference.

>
>> Do you have seamless spice migration enabled?
>> If so: Does it still reproduce with seamless migration turned off?
>
> No I don't think so; I think the command line I was running was:
> ./x86_64-softmmu/qemu-system-x86_64 -vnc :0 -M pc-i440fx-2.7,accel=kvm -monitor stdio -netdev user,id=unet,hostfwd=tcp::2022-:22,hostfwd=tcp::2023-:2022 -device virtio-net-pci,netdev=unet -drive file=/home/vms/f24.qcow2,cache=none,id=disk,if=none  -device virtio-blk-pci,drive=disk -device virtio-balloon -vga qxl -device ich9-usb-ehci1 -device usb-tablet,id=in0 -device virtio-rng-pci -device AC97 -m 8192 -smp 4 -drive file=/home/vms/Fedora-Server-netinst-x86_64-23.iso,cache=none,id=cd,if=scsi -incoming tcp::4444

Note that VNC is used.

Uri.

>
>>> The crash I've had with postcopy is:
>>> red_dispatcher_loadvm_commands:
>>> id 0, group 0, virt start 0, virt end ffffffffffffffff, generation 0, delta 0
>>> id 1, group 1, virt start 7fbe83c00000, virt end 7fbe87bfe000, generation 0, delta 7fbe83c00000
>>> id 2, group 1, virt start 7fbe7fa00000, virt end 7fbe83a00000, generation 0, delta 7fbe7fa00000
>>> (./x86_64-softmmu/qemu-system-x86_64:22376): Spice-CRITICAL **: red_memslots.c:123:get_virt: slot_id 128 too big, addr=8000000000000000
>>>
>>> #0  0x00007fc0aa42f49d in read () from /lib64/libpthread.so.0
>>> #1  0x00007fc0a8c36c01 in spice_backtrace_gstack () from /lib64/libspice-server.so.1
>>> #2  0x00007fc0a8c3e4f7 in spice_logv () from /lib64/libspice-server.so.1
>>> #3  0x00007fc0a8c3e655 in spice_log () from /lib64/libspice-server.so.1
>>> #4  0x00007fc0a8bfc6de in get_virt () from /lib64/libspice-server.so.1
>>> #5  0x00007fc0a8bfcb73 in red_get_data_chunks_ptr () from /lib64/libspice-server.so.1
>>> #6  0x00007fc0a8bff3fa in red_get_cursor_cmd () from /lib64/libspice-server.so.1
>>> #7  0x00007fc0a8c0fd79 in handle_dev_loadvm_commands () from /lib64/libspice-server.so.1
>>> #8  0x00007fc0a8bf9523 in dispatcher_handle_recv_read () from /lib64/libspice-server.so.1
>>> #9  0x00007fc0a8c1d5a5 in red_worker_main () from /lib64/libspice-server.so.1
>>> #10 0x00007fc0aa428dc5 in start_thread () from /lib64/libpthread.so.0
>>> #11 0x00007fc0a61786ed in clone () from /lib64/libc.so.6
>>
>> Spice worker thread ...
>>
>>> red_dispatcher_loadvm_commands:
>>> id 0, group 0, virt start 0, virt end ffffffffffffffff, generation 0, delta 0
>>> id 1, group 1, virt start 7f3b93800000, virt end 7f3b977fe000, generation 0, delta 7f3b93800000
>>> id 2, group 1, virt start 7f3b8f400000, virt end 7f3b93400000, generation 0, delta 7f3b8f400000
>>> (/opt/qemu/v2.7.0/bin/qemu-system-x86_64:41053): Spice-CRITICAL **: red_memslots.c:123:get_virt: slot_id 80 too big, addr=5000000000000000
>>
>>
>> ... trying to decode a invalid qxl address.
>
> Yes one observation is that I think a few (all?) of the bad
> addresses I've seen there have been a single nybble followed by
> a lot of 0's.
>
>>> I'm using:
>>> spice-server-devel-0.12.4-19.el7.x86_64
>>
>> Ah, RHEL-7.3 host.
>>
>> cheers,
>>   Gerd
>>
>
> Dave
>
> --
> Dr. David Alan Gilbert / dgilbert at redhat.com / Manchester, UK
> _______________________________________________
> Spice-devel mailing list
> Spice-devel at lists.freedesktop.org
> https://lists.freedesktop.org/mailman/listinfo/spice-devel
>