[Spice-devel] rfc seamless migration

[Yonit Halperin]

Hi,
On 06/10/2012 05:32 PM, Hans de Goede wrote:
> Hi,
>
> On 06/10/2012 11:05 AM, Yonit Halperin wrote:
>> Hi,
>>
>> As the qemu team rejected integrating spice connection migration in
>> qemu migration process, we remain with a solution that will involve
>> libvirt, and passing data from the src to the target via the client.
>> Before I continue with the implementation I'd like to hear your
>> comments on the details:
>>
>> Here is a reminder about the problems we face:
>> (1) Loss of data: we would like the client to continue the connection
>> from the same point the vm was stopped. For example, we want any
>> usb/smartcard devices to stay attached, and we don't want to lose any
>> data that was sent from the client to the vm, or partial data that was
>> read from a device, but hasn't reached its destination before migration.
>>
>> (2) The qemu process in the src side can be closed by libvirt as soon
>> as the migration state changes to "completed". Thus, we can't reliably
>> pass any data between the src server and the client after migration
>> has completed.
>>
>> These problems can be addressed by the following:
>> Add a qmp event for spice migration completion. libvirt will need to
>> wait not only for qemu migration completion, but also for this qmp
>> event, before it closes the src qemu.
>> Spice is required to know whether libvirt supports this, or not, in
>> order to decide which migration approach to take (semi or seamless).
>> For this aim, we will add a new parameter to the spice configuration
>> in the qemu command line (e.g., seamless-migration=on), and if it is
>> set by libvirt we can assume libvirt will wait for spice migration.
>> After qemu migration is completed, the src server will pass migration
>> data to the target via the client/s. When the clients disconnect from
>> the src and switch completely to the target, we send the new qmp event.
>>
>>
>> migration data transfer
>> =======================
>> Our historical MSG_MIGRATE pathway, provides support for sending all
>> pending outgoing data from the client to the server, and vice-versa,
>> before we fill the migration data.
>> Each channel defines its own migration data.
>> (1) MSG_MIGRATE is the last message that is sent from the src server
>> channel to the client, before MIGRATE_DATA.
>> (2) If the messages flags have MIGRATE_NEED_FLUSH, the client write
>> all its outgoing data, and then sends FLUSH to the server. (3) Then
>> the client
>> channel waits for MIGRATE_DATA message, and does nothing besides that.
>> (4) When it receives the message, it switches to the target completely
>> and passes it the migration data.
>>
>> (1) server channel--->MSG_MIGRATE...in-flight messages--->client
>> (2) client channel-->MSGC_FLUSH_MARK...in-flight messages-->server
>> (3) server channel-->MSG_MIGRATE_DATA-->client
>> (4) client channel-->MSGC_MIGRATE_DATA-->target server
>>
>> Obligatory migration data:
>> -------------------------
>> (1) agent/spicevmc/smartcard write buffer. i.e., data that reached the
>> server after savevm, and thus was not written to the device.
>> Currently, spicevmc and smartcard do not have write buffer, but since
>> buffers can reach the server after savevm, they should have one. I'm
>> not sure if even today they should attempt to write to the guest if it
>> is stopped. The agent code also can write to the guest even if it is
>> stopped; I think it is a bug.
>> (2) agent/smartcard partial data that had been read from the device
>> and wasn't sent to the client since its reading hasn't completed.
>> Currently we don't have such data for spicevmc, because we push to the
>> client any amount of data we read. In the future we might want to
>> control the rate and the size of data we send/receive, and then we
>> will have outgoing buffer.
>
> I'm still not a big fan of the concept of server data going through the
> client, this means the server
> will need to seriously sanity check what it receives to avoid
> potentially new attacks on it.
>
> I'm wondering why not do the following:
>
> 1) spicevmc device gets a savevm call, tell spice-server to send a
> message to the client telling it
> to stop sending more data to *this* server.
> 2) client sends an ack in response to the stop sending data server
> 3) server waits for ack.
> 4) savevm continues only after ack, which means all data which was in
> flight has been received.
>
> No more reason for obligatory data 1.
> And as you already point out 2, is not an issue atm.
>
> So no more obligatory reason to have server *state* pass through the
> client, which I still believe
> is just asking for security vulnerabilities.
>
>
Actually, even if qemu team hasn't rejected this solution, there was 
still a server state to transfer: the writes to the agent can fail, and 
the server holds a write queue for handling such writes. In addition, 
both the agent and the smartcard have a buffer for partial reads. I also 
think that though spicevmc doesn't have a write buffer, if in the future 
spicevmc will be used with other devices besides usb, it may require a 
write buffer.
>> Optional migration data:
>> --------------
>> - primary surface lossy region(*), or its extents
>> If we don't send it to the client, and jpeg is enabled, we will need
>> to resend the primary surface after migration, or set the lossy region
>> to the whole surface, and then each non opaque rendering operation
>> that involves the surface, will require resending parts of it losslessly.
>
> So this needs to be send to client, but not back to the server?
>
I meant that sending the lossy region or its extents (i.e., just a Rect) 
from the src to the target, can prevent sending the primary surface, or 
parts of it, to the client after it switches to the target server
>> - list of off-screen surfaces ids that have been sent to the client,
>> and their lossy region.
>> By keeping this data we will avoid on-demand resending surfaces that
>> already exist on the client side.
>
> The client already knows which off-screen surfaces ids it has been
> received, so it can just
> send these to the new server without having to receive them from the old
> one first.
>
The client does not track the lossy regions. Even though it can track 
them, I don't think the client should be aware of the internal state of 
the server. In addition, I think that in order to keep the migration 
logic as simple as possible, and less prone to bugs, we should aim to 
isolate the migration data synchronization logic as much as possible to 
one place, to the server.
>> - bitmaps cache - list of bitmaps ids + some internal cache
>> information for each bitmap.
>
> idem.
>
>> - active video streams: ids, destination box, etc.
>
> idem.
>
>> - session bandwidth (low/high): we don't want to perform the main
>> channel net test after the migration is completed, because it can take
>> time (we can't do it during the migration because the main loop is not
>> available). So we assume the bandwidth classification will stay the
>> same. When we will have a dynamic monitoring of bandwidth, we can drop
>> this.
>
> This I can live with being send through the client, but then not as
> opaque data, but have
> a special command for it. This could be useful in non migration cases
> too. If the client
> somehow already knows the channel characteristics.
>
>
>>
>> Though the above data is optional, part of it is important for
>> avoiding a slow start of the connection to target (e.g., sending the
>> primary lossy region, in order to avoid resending parts of it).
>>
>> In addition, if we wish to keep the client channels state the same,
>> and not require them (1) to send initialization data to the server,
>> and (2) to reset part of their state, we should also migrate other
>> server state details, like:
>> - the serial of the last message sent from the display channel
>> - main channel agent data tokens state
>> - size of the images cache (this is usually set by the client upon new
>> connection).
>> Including such information in the migration data will allow us to keep
>> the migration logic in the server. The alternative will be that the
>> client will reset part of its state after migration, either by self
>> initiative, or by specific messages sent from the server (it may
>> require new set of messages).
>>
>> (*) lossy-region=the region on the surface that contains bitmaps that
>> were compressed using jpeg
>>
>> Transparency of migration data:
>> ------------------------------
>> I think that the migration data shouldn't be part of spice protocol,
>> and that it should be opaque to the client, for the following reasons:
>
> As said before, I think that migration data should not be send through
> the spice protocol *at all* !
>
>> (a) The client is only a mediator, and it has nothing to do with the
>> data content.
>> (b) If the migration data of each channel is part of spice protocol,
>> every minor change to the migration data of one channel, will require
>> a new message and capability, and will make the support in migration
>> backward compatibility more cumbersome, as it will involve the client
>> as well. Moreover, If the client supports only migration data of ver
>> x, and the src and target both support ver x+1, we will suffer from
>> data loss.
>> (c) As for security issues, I don't think that it should raise a
>> problem since the client is trusted by both the src and the target.
>
> The client is trusted to access the *vm*, not the *host*, and this
> allows attacks on spice-server,
> which is running on the *host*.
>
The client is trusted to access the spice-server.

Best,
Yonit.
>
> Regards,
>
> Hans