When migrated using libvirt with "--copy-storage-all", at the end of
migration there is race between NBD mirroring task trying to do flush
and migration completion, both end up invalidating cache. Since qcow2
driver does not handle this situation very well, random crashes happen.
This disables the BDRV_O_INCOMING flag for the block device being migrated
once the cache has been invalidated.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
--
fixed parens by hand
Signed-off-by: Juan Quintela <quintela@redhat.com>
if a saved vm has unknown flags in the memory data qemu
currently simply ignores this flag and continues which
yields in an unpredictable result.
This patch catches all unknown flags and aborts the
loading of the vm. Additionally error reports are thrown
if the migration aborts abnormally.
Signed-off-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Juan Quintela <quintela@redhat.com>
The existing timeout is 30ms which on 100MB/s (1Gbit) gives us
3MB/s rate maximum. If we put some load on the guest, it is easy to
get page dirtying rate too big so live migration will never complete.
In the case of libvirt that means that the guest will be stopped
anyway after a timeout specified in the "virsh migrate" command and
this normally generates even bigger delay.
This changes max_downtime to 300ms which seems to be more
reasonable value.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Signed-off-by: Juan Quintela <quintela@redhat.com>
currently the value of the throughput field contains whatever
was the last calculated throughput shortly before the migration
finished.
This patch updates the post migration contents of the field to
the average throughput.
Signed-off-by: Peter Lieven <pl@kamp.de>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
expose the count that logs the times of updating the dirty bitmap to
end user.
Signed-off-by: ChenLiang <chenliang88@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
QEMU will assert if you attempt to start an outgoing migration on
a QEMU that's sitting waiting for an incoming migration (started
with -incoming), so disallow it with a proper error.
(This is a fix for https://bugzilla.redhat.com/show_bug.cgi?id=1086987 )
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Andreas Färber <afaerber@suse.de>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
This replaces DPRINTF macro with tracepoints.
This moves some messages from migration.c to savevm.c.
This adds tracepoint to signal about fileds failed to migrate.
Signed-off-by: Alexey Kardashevskiy <aik@ozlabs.ru>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Amit Shah <amit.shah@redhat.com>
If it returns an error, the migrated VM will not be started, but qemu
exits with an error message.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Benoit Canet <benoit@irqsave.net>
If enabled, set the thread name at creation (on GNU systems with
pthread_set_np)
Fix up all the callers with a thread name
Signed-off-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
Reviewed-by: Laszlo Ersek <lersek@redhat.com>
As far as we can tell, all known bugs have been fixed:
1. Parallel migrations are working
2. IPv6 migration is working
3. virt-test is working
I'm not comfortable sending the revised libvirt patch
until this is accepted or review suggestions are addressed,
(including pin-all support. It does not make sense to
remove experimental for one thing and not the other. That's
too many trips through the libvirt community).
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
When qemu do live migration with xbzrle, qemu malloc decoded_buf
at destination end but free it at source end. It will crash qemu
by double free error in some scenarios. Splitting the XBZRLE structure
for clear logic distinguishing src/dst side.
Signed-off-by: ChenLiang <chenliang88@huawei.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: GongLei <arei.gonglei@huawei.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
The migration thread appears to want to allow writeout to occur at full
speed rather than being rate limited during completion of state saving,
but sets the limit to INT_MAX when xfer_limit is INT64_MAX. This causes
problems if there's more than 2GB of state left to save at this point. It
probably ought to just be INT64_MAX instead.
Signed-off-by: Matthew Garrett <matthew.garrett@nebula.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Introduce MIG_STATE_CANCELLING state to avoid starting a new migration task while the previous one still exist.
Signed-off-by: Zeng Junliang <zengjunliang@huawei.com>
Signed-off-by: Zhang Haoyu <haoyu.zhang@huawei.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Avoid a bogus COMPLETED->CANCELLED transition.
There is a period of time from the timing of setting COMPLETED state to that of migration thread exits, so during which it's problematic in COMPLETED->CANCELLED transition.
Signed-off-by: Zeng Junliang <zengjunliang@huawei.com>
Signed-off-by: Zhang Haoyu <haoyu.zhang@huawei.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
If a user or QMP client enter a bad syntax for the migrate
command in QMP/HMP, then the migrate command will never succeed
from that point on.
For example, if you enter:
(qemu) migrate tcp;0:4444
migrate: Parameter 'uri' expects a valid migration protocol
Then the migrate command will always fail from now on:
(qemu) migrate tcp:0:4444
migrate: There's a migration process in progress
The problem is that qmp_migrate() sets the migration status to
MIG_STATE_SETUP and doesn't reset it on syntax error. This bug
was introduced by commit 29ae8a4133.
Reviewed-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
The printf args are uint64_t and with -Werr QEMU doesn't compile with
migration debugging turned on unless this is fixed. Fix it.
Signed-off-by: Christoffer Dall <christoffer.dall@linaro.org>
Signed-off-by: Juan Quintela <quintela@redhat.com>
This is an autogenerated patch using scripts/switch-timer-api.
Switch the entire code base to using the new timer API.
Note this patch may introduce some line length issues.
Signed-off-by: Alex Bligh <alex@alex.org.uk>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
include/qemu/timer.h has no need to include main-loop.h and
doing so causes an issue for the next patch. Unfortunately
various files assume including timers.h will pull in main-loop.h.
Untangle this mess.
Signed-off-by: Alex Bligh <alex@alex.org.uk>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
The qmp_migrate method uses the 'blk' and 'inc' parameter without
checking if they're valid or not (they may be uninitialized if
command is received via QMP)
Signed-off-by: Pawit Pornkitprasan <p.pawit@gmail.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
The "completed" stage sets total_time but not has_total_time and
thus it is not sent via QMP reply (but sent via HMP nevertheless)
Signed-off-by: Pawit Pornkitprasan <p.pawit@gmail.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
Commit 29ae8a4133 ("rdma: introduce
MIG_STATE_NONE and change MIG_STATE_SETUP state transition") changed the
state transitions during migration setup.
Spice used to be notified with MIG_STATE_ACTIVE and it detected this
using migration_is_active(). Spice is now notified with
MIG_STATE_SETUP and migration_is_active() no longer works.
Replace migration_is_active() with migration_in_setup() to fix spice
migration.
Cc: Michael R. Hines <mrhines@us.ibm.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
The migration thread runs outside the QEMU global mutex when possible.
Therefore we must notify migration state change *before* starting the
migration thread.
This allows registered listeners to act before live migration iterations
begin. Therefore they can get into a state that allows for live
migration. When the migration thread starts everything will be ready.
Without this patch there is a race condition during migration setup,
depending on whether the migration thread has already transitioned from
SETUP to ACTIVE state.
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Using the previous patches, we're now able to timestamp the SETUP
state. Once we have this time, let the user know about it in the
schema.
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
As described in the previous patch, until now, the MIG_STATE_SETUP
state was not really a 'formal' state. It has been used as a 'zero' state
(what we're calling 'NONE' here) and QEMU has been unconditionally transitioning
into this state when the QMP migration command was called. Instead we want to
introduce MIG_STATE_NONE, which is our starting state in the state machine, and
then immediately transition into the MIG_STATE_SETUP state when the QMP migrate
command is issued.
In order to do this, we must delay the transition into MIG_STATE_ACTIVE until
later in the migration_thread(). This is done to be able to timestamp the amount of
time spent in the SETUP state for proper accounting to the user during
an RDMA migration.
Furthermore, the management software, until now, has never been aware of the
existence of the SETUP state whatsoever. This must change, because, timing of this
state implies that the state actually exists.
These two patches cannot be separated because the 'query_migrate' QMP
switch statement needs to know how to handle this new state transition.
Reviewed-by: Juan Quintela <quintela@redhat.com>
Tested-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
This patch is in preparation for the next ones: Until now the MIG_STATE_SETUP
state was not really a 'formal' state. It has been used as a 'zero' state
and QEMU has been unconditionally transitioning into this state when
the QMP migrate command was called. In preparation for timing this state,
we have to make this state a a 'real' state which actually gets transitioned
from later in the migration_thread() from SETUP => ACTIVE, rather than just
automatically dropping into this state at the beginninig of the migration.
This means that the state transition function (migration_finish_set_state())
needs to be capable of transitioning from valid states _other_ than just
MIG_STATE_ACTIVE.
The function is in fact already capable of doing that, but was not allowing the
old state to be a parameter specified as an input.
This patch fixes that and only makes the transition if the current state
matches the old state that the caller intended to transition from.
Reviewed-by: Juan Quintela <quintela@redhat.com>
Tested-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Code that does need to be visible is kept
well contained inside this file and this is the only
new additional file to the entire patch.
This file includes the entire protocol and interfaces
required to perform RDMA migration.
Also, the configure and Makefile modifications to link
this file are included.
Full documentation is in docs/rdma.txt
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Chegu Vinod <chegu_vinod@hp.com>
Tested-by: Chegu Vinod <chegu_vinod@hp.com>
Tested-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
this patch adds a efficient encoding for zero blocks by
adding a new flag indicating a block is completely zero.
additionally bdrv_write_zeros() is used at the destination
to efficiently write these zeroes. depending on the implementation
this avoids that the destination target gets fully provisioned.
Signed-off-by: Peter Lieven <pl@kamp.de>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
# By Chegu Vinod
# Via Juan Quintela
* quintela/migration.next:
Force auto-convegence of live migration
Add 'auto-converge' migration capability
Introduce async_run_on_cpu()
Message-id: 1373664508-5404-1-git-send-email-quintela@redhat.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
If bdrv_flush_all() returns an error, there is an inconsistency in the
view of an image file between the source and the destination host.
Completing the migration would lead to corruption. Better abort
migration in this case.
To reproduce this case, try the following (ensures that there is
something to flush, and then fails that flush):
$ qemu-img create -f qcow2 test.qcow2 1G
$ cat blkdebug.cfg
[inject-error]
event = "flush_to_os"
errno = "5"
$ qemu-system-x86_64 -hda blkdebug:blkdebug.cfg:test.qcow2 -monitor stdio
(qemu) qemu-io ide0-hd0 "write 0 4k"
(qemu) migrate ...
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
The auto-converge migration capability allows the user to specify if they
choose live migration seqeunce to automatically detect and force convergence.
Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
We're already using them in several places, but __sync builtins are just
too ugly to type, and do not provide seqcst load/store operations.
Reviewed-by: Richard Henderson <rth@twiddle.net>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This capability allows you to disable dynamic chunk registration
for better throughput on high-performance links.
For example, using an 8GB RAM virtual machine with all 8GB of memory in
active use and the VM itself is completely idle using a 40 gbps infiniband link:
1. x-rdma-pin-all disabled total time: approximately 7.5 seconds @ 9.5 Gbps
2. x-rdma-pin-all enabled total time: approximately 4 seconds @ 26 Gbps
These numbers would of course scale up to whatever size virtual machine
you have to migrate using RDMA.
Enabling this feature does *not* have any measurable affect on
migration *downtime*. This is because, without this feature, all of the
memory will have already been registered already in advance during
the bulk round and does not need to be re-registered during the successive
iteration rounds.
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Chegu Vinod <chegu_vinod@hp.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Tested-by: Chegu Vinod <chegu_vinod@hp.com>
Tested-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
This exposes throughput (in megabits/sec) through QMP.
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Chegu Vinod <chegu_vinod@hp.com>
Tested-by: Chegu Vinod <chegu_vinod@hp.com>
Tested-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Michael R. Hines <mrhines@us.ibm.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
bandwidth_limit is double set in migrate_init(), remove one.
Signed-off-by: Lei Li <lilei@linux.vnet.ibm.com>
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
This reverts commit 7161082c8d.
Reverting this patch fixes a divide-by-zero error in qemu that can be
fairly reliably triggered by doing block migration. In this case, the
configuration/error was:
source: temp/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -L temp-bios
-M pc-i440fx-1.4 -m 512M -kernel boot/vmlinuz-x86_64 -initrd
boot/test-initramfs-x86_64.img.gz -vga std -append seed=1234 -drive
file=disk1.img,if=virtio -drive file=disk2.img,if=virtio -device
virtio-net-pci,netdev=net0 -netdev user,id=net0 -monitor
unix:/tmp/vm-hmp.sock,server,nowait -qmp
unix:/tmp/vm-qmp.sock,server,nowait -vnc :100
16837 Floating point exception(core dumped)
target: temp/x86_64-softmmu/qemu-system-x86_64 -enable-kvm -L temp-bios
-M pc-i440fx-1.4 -m 512M -kernel boot/vmlinuz-x86_64 -initrd
boot/test-initramfs-x86_64.img.gz -vga std -append seed=1234 -drive
file=target_disk1.img,if=virtio -drive file=target_disk2.img,if=virtio
-device virtio-net-pci,netdev=net0 -netdev user,id=net0 -incoming
unix:/tmp/migrate.sock -monitor
unix:/tmp/vm-hmp-incoming.sock,server,nowait -qmp
unix:/tmp/vm-qmp-incoming.sock,server,nowait -vnc :101
Receiving block device images
20 %
21 %
load of migration failed
This revert potentially re-introduces a bug that was present in 1.4,
but fixes a prevalent issue with block migration so we should revert
it for now and take an updated patch later.
Conflicts:
migration.c
* fixed up to remove logic introduced in 7161082c while leaving
changes in HEAD intact
Signed-off-by: Michael Roth <mdroth@linux.vnet.ibm.com>
Message-id: 1368739544-31021-1-git-send-email-mdroth@linux.vnet.ibm.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
Management apps like libvirt don't know to pay attention to
stderr unless there is a non-zero exit status.
* migration.c (process_incoming_migration_co): Exit with non-zero
status on failure.
Signed-off-by: Eric Blake <eblake@redhat.com>
Message-id: 1366149041-626-1-git-send-email-eblake@redhat.com
Signed-off-by: Anthony Liguori <aliguori@us.ibm.com>
The fcntl(fd, F_SETFL, O_NONBLOCK) flag is not specific to sockets.
Rename to qemu_set_nonblock() just like qemu_set_cloexec().
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Luiz Capitulino <lcapitulino@redhat.com>
during bulk stage of ram migration if a page is a
zero page do not send it at all.
the memory at the destination reads as zero anyway.
even if there is an madvise with QEMU_MADV_DONTNEED
at the target upon receipt of a zero page I have observed
that the target starts swapping if the memory is overcommitted.
it seems that the pages are dropped asynchronously.
this patch also updates QMP to return the number of
skipped pages in MigrationStats.
Signed-off-by: Peter Lieven <pl@kamp.de>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
The indirection is useless now. Backends can open s->file directly.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
With this patch, the migration_file is not needed anymore.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Rate limiting is now simply a byte counter; client call
qemu_file_rate_limit() manually to determine if they have to exit.
So it is possible and simple to move the functionality to QEMUFile.
This makes the remaining functionality of s->file redundant;
in the next patch we can remove it and write directly to s->migration_file.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
This patch extracts a few small changes from the next patch, which
are unrelated to adding generic rate-limiting functionality to
QEMUFile. Make migration_set_rate_limit a simple accessor, and
use qemu_file_set_rate_limit consistently. Also fix a typo where
INT_MAX should have been SIZE_MAX.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Prepare for when s->bytes_xfer will be removed.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Second, drop the file descriptor indirection, and write directly to the
QEMUFile.
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: Juan Quintela <quintela@redhat.com>
Alexey Kardashevskiy
Peter Lieven
ChenLiang
Dr. David Alan Gilbert
Kevin Wolf
Michael R. Hines
Orit Wasserman
Gonglei (Arei)
Edgar E. Iglesias
Matthew Garrett
Zhanghaoyu (A)
Luiz Capitulino
Michael Tokarev
Christoffer Dall
Alex Bligh
Pawit Pornkitprasan
Stefan Hajnoczi
Anthony Liguori
Chegu Vinod
Paolo Bonzini
Lei Li
Michael Roth
Eric Blake