[Spice-devel] changing the timing of spice client linking in migration (RHBZ #725009)

[Yonit Halperin]

On 08/25/2011 01:24 PM, Yonit Halperin wrote:
> Hi all,
> here is the original email I sent for #725009.
> ===> For this plan no change is required in libvirt. <===
>
> This plan does not bring back seamless migration to live. But it does
> make the current host switching less noticeable by the user.
> The difference from seamless migration is that in-progress operations
> fail (equivalent to disconnect and reconnect): audio, smartcard, usb (?
> not sure for which rhel version it is designated), vdagent (copy-paste).
>
> After some discussions with Alon, we came to the conclusion that in
> order to revive seamless migration we need to solve the following race
> conditions :
>
> race 1: target starts before the client managed to connect to it.
> possible solutions:
> 1) qxl savevm will hold (with timeout) migration till client is
> connected to target. Will it be acceptable by qemu? (Gerd?)
> However, qxl is not necessarily installed...
> 2) spice will try to connect the target immediately when it receives
> client_migrate_info from libvirt, and will wait till it succeeds (and by
> this will hold migration. (libvirt - can we count on executing
> client_migrate_info just before migration starts and not a long time
> before?)
>
> race 2: target starts before it manages to restore the source
> pre-migration state (which was sent to it from the src server, via the
> client).
> solution:
> the target won't attach any spice device interface till it restores its
> state (with timeout). i.e., it will queue any request from the quest,
> and will operate on them only after it has been restored.
>
 From the vm_start code (vl.c), it looks like the start notifier is 
called before the guest is actually started, so we can hold the target 
from starting till we recover it. Gerd, is this correct?

> other changes in spice:
> due to the broken seamless migration, spice channels were not maintained
> to keep supporting it (i.e., sending the minimal amount of data that is
> required for the target to restore the channel's state), especially the
> new channels (smartcard, usb(?), agent transmissions, etc.). So,
> implementation in this level is required as well.
>
Arnon, Alon, Hans, can you let me know how much work it would be to 
support seemless migration for agent, smartcard, and usb (is it relevant 
for rhel 6.3?)? Supporting seemless migration mainly requires (1) 
sending SPICE_MSG_MIGRATE_DATA with data that will allow the target side 
to be restored, so that the connection with the same client will stay 
valid. (2) handle SPICE_MSGC_MIGRATE_DATA in the target
See RedChannel::handle_migrate in the client for more details.
Is there any other channel I didn't think of that needs work for 
seemless migration (except for the display channel and surfaces issues)?

> Cheers,
> Yonit.
>
>
> On 08/16/2011 12:53 PM, Yonit Halperin wrote:
>> Hi,
>> Today, we handle migration in the client side by disconnecting from the
>> source and connecting to the target. The disconnection
>> and reconnection occur in response to
>> SPICE_MSG_MAIN_MIGRATE_SWITCH_HOST, which is sent from the source when
>> migration ends.
>> In order to solve RHBZ #725009, i.e., to avoid expiration of tickets at
>> migration target before spice client has attempted to connect, it has
>> been proposed that spice client will connect to the target when
>> migration starts, but will complete the connection only after
>> migration ends.
>> This pathway is more similar to how spice worked when we could support
>> seamless migration. But, in contrast to seamless migration,
>> we still need to initialize data on the client side upon switching to
>> the target (this cleanup was achieved by the explicit disconnection and
>> reconnection).
>>
>> Here are the details of the implementation I suggest:
>>
>> qemu
>> =====
>> ui/spice-core::migration_state_notifier should handle MIG_STATE_ACTIVE
>> (for migration start) and MIG_STATE_ERROR/CANCELLED by calling
>> spice_server_migrate_start, and spice_server_migrate_end, respectively.
>> These callbacks are currently declared in spice-experimental.h.
>>
>> spice-server
>> =============
>> (A) Migration source side
>>
>> * reds::spice_server_migrate_start:
>> send SPICE_MSG_MAIN_MIGRATE_SWITCH_HOST.
>> We can't use SPICE_MSG_MAIN_MIGRATE_BEGIN since it doesn't
>> include the certificate information we need. But we can change it
>> to be identical to SPICE_MSG_MAIN_MIGRATE_SWITCH_HOST.
>>
>> * reds::spice_server_migrate_end(completed)
>> - if (completed) => send SPICE_MSG_MIGRATE (flags=0) to all
>> connected channels (via Channel->migrate).
>> - if (!completed) => send SPICE_MSG_MAIN_MIGRATE_CANCEL
>>
>> (B) Migration target side
>>
>> reds identifies it is a migration target when the client connects with a
>> connection id != 0.
>> When linking to a migrated channels, a special treatment is required
>> (and not the support that is currently coded, since it is for seamless
>> migration).
>> For example:
>> - For the main channel, (1) network test is not required (2) no need for
>> SPICE_MSG_MAIN_INIT, but rather SPICE_MSG_MAIN_MULTI_MEDIA_TIME and
>> SPICE_MSG_MAIN_MOUSE_MODE. This way we will also save all the agent work
>> we preform when initializing the main channel in the client.
>> - For the display channel, we mustn't call display_channel_wait_for_init
>> immediately upon link, but we should expect it to arrive later (for
>> setting cache and dictionary sizes).
>> - For playback channel: we still need to send the current playback
>> status, as opposed to seamless migration.
>>
>> Spice client
>> ============
>> (A) SPICE_MSG_MAIN_MIGRATE_SWITCH_HOST
>> client connects to the target, but still stays connected to the
>> source host. It doesn't listen to the target sockets.
>> The link message to the target contains the connection_id of the
>> connection to the source (this allows the target server to identify
>> itself as a migration target).
>> For this part we can use most of the code in the class Migrate in
>> red_client.cpp
>> (B) SPICE_MSG_MIGRATE
>> We can use the code in red_channel::handle_migrate to switch the
>> channels and start listening to the target.
>> The difference is that we should implement differently the virtual
>> method RedChannel::on_migrate.
>> (1) Each channel must reset all the dynamic data that depends on
>> the server. For example: the display channel
>> needs to destroy all the surfaces and reset the caches and
>> dictionary; The playback and record channel need to stop
>> the current session, if there is an active one, etc.
>> (2) Each channel should send to the server the initalization
>> information it normally sends in RedChannel::on_connect.
>>
>> (C) SPICE_MSG_MAIN_MIGRATE_CANCEL
>> disconnects all the new channels. This code is already implemented
>> in spice-client.
>>
>> spice-protocol(?)/Backward compatibility
>> =========================================
>> should we bounce spice protocol version, or use capabilities? (if we
>> change SPICE_MSG_MAIN_MIGRATE_BEGIN structue, there is no question).
>>
>> New Spice-Server with old client will send only
>> SPICE_MSG_MAIN_MIGRATE_SWITCH_HOST, and only when migration completes
>> (same as today).
>> New client with old Spice-server will disconnect the source and will
>> connect the target upon receiving SPICE_MSG_MAIN_MIGRATE_SWITCH_HOST
>> (same as today).
>>
>> Any comments? If there are any tricky parts I missed, let me know.
>>
>> Cheers,
>> Yonit.
>