docs/migration: Organize "Postcopy" page
Reorganize the page, moving things around, and add a few
headlines ("Postcopy internals", "Postcopy features") to cover sub-areas.
Reviewed-by: Cédric Le Goater <clg@redhat.com>
Link: https://lore.kernel.org/r/20240109064628.595453-9-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
This commit is contained in:
Peter Xu
parent
4c6f8a79ae
commit
21b17cd011
@ -1,6 +1,9 @@
|
|||||||
|
========
|
||||||
Postcopy
|
Postcopy
|
||||||
========
|
========
|
||||||
|
|
||||||
|
.. contents::
|
||||||
|
|
||||||
'Postcopy' migration is a way to deal with migrations that refuse to converge
|
'Postcopy' migration is a way to deal with migrations that refuse to converge
|
||||||
(or take too long to converge) its plus side is that there is an upper bound on
|
(or take too long to converge) its plus side is that there is an upper bound on
|
||||||
the amount of migration traffic and time it takes, the down side is that during
|
the amount of migration traffic and time it takes, the down side is that during
|
||||||
@ -14,7 +17,7 @@ Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
|
|||||||
doesn't finish in a given time the switch is made to postcopy.
|
doesn't finish in a given time the switch is made to postcopy.
|
||||||
|
|
||||||
Enabling postcopy
|
Enabling postcopy
|
||||||
-----------------
|
=================
|
||||||
|
|
||||||
To enable postcopy, issue this command on the monitor (both source and
|
To enable postcopy, issue this command on the monitor (both source and
|
||||||
destination) prior to the start of migration:
|
destination) prior to the start of migration:
|
||||||
@ -49,8 +52,71 @@ time per vCPU.
|
|||||||
``migrate_set_parameter`` is ignored (to avoid delaying requested pages that
|
``migrate_set_parameter`` is ignored (to avoid delaying requested pages that
|
||||||
the destination is waiting for).
|
the destination is waiting for).
|
||||||
|
|
||||||
Postcopy device transfer
|
Postcopy internals
|
||||||
------------------------
|
==================
|
||||||
|
|
||||||
|
State machine
|
||||||
|
-------------
|
||||||
|
|
||||||
|
Postcopy moves through a series of states (see postcopy_state) from
|
||||||
|
ADVISE->DISCARD->LISTEN->RUNNING->END
|
||||||
|
|
||||||
|
- Advise
|
||||||
|
|
||||||
|
Set at the start of migration if postcopy is enabled, even
|
||||||
|
if it hasn't had the start command; here the destination
|
||||||
|
checks that its OS has the support needed for postcopy, and performs
|
||||||
|
setup to ensure the RAM mappings are suitable for later postcopy.
|
||||||
|
The destination will fail early in migration at this point if the
|
||||||
|
required OS support is not present.
|
||||||
|
(Triggered by reception of POSTCOPY_ADVISE command)
|
||||||
|
|
||||||
|
- Discard
|
||||||
|
|
||||||
|
Entered on receipt of the first 'discard' command; prior to
|
||||||
|
the first Discard being performed, hugepages are switched off
|
||||||
|
(using madvise) to ensure that no new huge pages are created
|
||||||
|
during the postcopy phase, and to cause any huge pages that
|
||||||
|
have discards on them to be broken.
|
||||||
|
|
||||||
|
- Listen
|
||||||
|
|
||||||
|
The first command in the package, POSTCOPY_LISTEN, switches
|
||||||
|
the destination state to Listen, and starts a new thread
|
||||||
|
(the 'listen thread') which takes over the job of receiving
|
||||||
|
pages off the migration stream, while the main thread carries
|
||||||
|
on processing the blob. With this thread able to process page
|
||||||
|
reception, the destination now 'sensitises' the RAM to detect
|
||||||
|
any access to missing pages (on Linux using the 'userfault'
|
||||||
|
system).
|
||||||
|
|
||||||
|
- Running
|
||||||
|
|
||||||
|
POSTCOPY_RUN causes the destination to synchronise all
|
||||||
|
state and start the CPUs and IO devices running. The main
|
||||||
|
thread now finishes processing the migration package and
|
||||||
|
now carries on as it would for normal precopy migration
|
||||||
|
(although it can't do the cleanup it would do as it
|
||||||
|
finishes a normal migration).
|
||||||
|
|
||||||
|
- Paused
|
||||||
|
|
||||||
|
Postcopy can run into a paused state (normally on both sides when
|
||||||
|
happens), where all threads will be temporarily halted mostly due to
|
||||||
|
network errors. When reaching paused state, migration will make sure
|
||||||
|
the qemu binary on both sides maintain the data without corrupting
|
||||||
|
the VM. To continue the migration, the admin needs to fix the
|
||||||
|
migration channel using the QMP command 'migrate-recover' on the
|
||||||
|
destination node, then resume the migration using QMP command 'migrate'
|
||||||
|
again on source node, with resume=true flag set.
|
||||||
|
|
||||||
|
- End
|
||||||
|
|
||||||
|
The listen thread can now quit, and perform the cleanup of migration
|
||||||
|
state, the migration is now complete.
|
||||||
|
|
||||||
|
Device transfer
|
||||||
|
---------------
|
||||||
|
|
||||||
Loading of device data may cause the device emulation to access guest RAM
|
Loading of device data may cause the device emulation to access guest RAM
|
||||||
that may trigger faults that have to be resolved by the source, as such
|
that may trigger faults that have to be resolved by the source, as such
|
||||||
@ -130,7 +196,20 @@ processing.
|
|||||||
is no longer used by migration, while the listen thread carries on servicing
|
is no longer used by migration, while the listen thread carries on servicing
|
||||||
page data until the end of migration.
|
page data until the end of migration.
|
||||||
|
|
||||||
Postcopy Recovery
|
Source side page bitmap
|
||||||
|
-----------------------
|
||||||
|
|
||||||
|
The 'migration bitmap' in postcopy is basically the same as in the precopy,
|
||||||
|
where each of the bit to indicate that page is 'dirty' - i.e. needs
|
||||||
|
sending. During the precopy phase this is updated as the CPU dirties
|
||||||
|
pages, however during postcopy the CPUs are stopped and nothing should
|
||||||
|
dirty anything any more. Instead, dirty bits are cleared when the relevant
|
||||||
|
pages are sent during postcopy.
|
||||||
|
|
||||||
|
Postcopy features
|
||||||
|
=================
|
||||||
|
|
||||||
|
Postcopy recovery
|
||||||
-----------------
|
-----------------
|
||||||
|
|
||||||
Comparing to precopy, postcopy is special on error handlings. When any
|
Comparing to precopy, postcopy is special on error handlings. When any
|
||||||
@ -166,76 +245,6 @@ configurations of the guest. For example, when with async page fault
|
|||||||
enabled, logically the guest can proactively schedule out the threads
|
enabled, logically the guest can proactively schedule out the threads
|
||||||
accessing missing pages.
|
accessing missing pages.
|
||||||
|
|
||||||
Postcopy states
|
|
||||||
---------------
|
|
||||||
|
|
||||||
Postcopy moves through a series of states (see postcopy_state) from
|
|
||||||
ADVISE->DISCARD->LISTEN->RUNNING->END
|
|
||||||
|
|
||||||
- Advise
|
|
||||||
|
|
||||||
Set at the start of migration if postcopy is enabled, even
|
|
||||||
if it hasn't had the start command; here the destination
|
|
||||||
checks that its OS has the support needed for postcopy, and performs
|
|
||||||
setup to ensure the RAM mappings are suitable for later postcopy.
|
|
||||||
The destination will fail early in migration at this point if the
|
|
||||||
required OS support is not present.
|
|
||||||
(Triggered by reception of POSTCOPY_ADVISE command)
|
|
||||||
|
|
||||||
- Discard
|
|
||||||
|
|
||||||
Entered on receipt of the first 'discard' command; prior to
|
|
||||||
the first Discard being performed, hugepages are switched off
|
|
||||||
(using madvise) to ensure that no new huge pages are created
|
|
||||||
during the postcopy phase, and to cause any huge pages that
|
|
||||||
have discards on them to be broken.
|
|
||||||
|
|
||||||
- Listen
|
|
||||||
|
|
||||||
The first command in the package, POSTCOPY_LISTEN, switches
|
|
||||||
the destination state to Listen, and starts a new thread
|
|
||||||
(the 'listen thread') which takes over the job of receiving
|
|
||||||
pages off the migration stream, while the main thread carries
|
|
||||||
on processing the blob. With this thread able to process page
|
|
||||||
reception, the destination now 'sensitises' the RAM to detect
|
|
||||||
any access to missing pages (on Linux using the 'userfault'
|
|
||||||
system).
|
|
||||||
|
|
||||||
- Running
|
|
||||||
|
|
||||||
POSTCOPY_RUN causes the destination to synchronise all
|
|
||||||
state and start the CPUs and IO devices running. The main
|
|
||||||
thread now finishes processing the migration package and
|
|
||||||
now carries on as it would for normal precopy migration
|
|
||||||
(although it can't do the cleanup it would do as it
|
|
||||||
finishes a normal migration).
|
|
||||||
|
|
||||||
- Paused
|
|
||||||
|
|
||||||
Postcopy can run into a paused state (normally on both sides when
|
|
||||||
happens), where all threads will be temporarily halted mostly due to
|
|
||||||
network errors. When reaching paused state, migration will make sure
|
|
||||||
the qemu binary on both sides maintain the data without corrupting
|
|
||||||
the VM. To continue the migration, the admin needs to fix the
|
|
||||||
migration channel using the QMP command 'migrate-recover' on the
|
|
||||||
destination node, then resume the migration using QMP command 'migrate'
|
|
||||||
again on source node, with resume=true flag set.
|
|
||||||
|
|
||||||
- End
|
|
||||||
|
|
||||||
The listen thread can now quit, and perform the cleanup of migration
|
|
||||||
state, the migration is now complete.
|
|
||||||
|
|
||||||
Source side page map
|
|
||||||
--------------------
|
|
||||||
|
|
||||||
The 'migration bitmap' in postcopy is basically the same as in the precopy,
|
|
||||||
where each of the bit to indicate that page is 'dirty' - i.e. needs
|
|
||||||
sending. During the precopy phase this is updated as the CPU dirties
|
|
||||||
pages, however during postcopy the CPUs are stopped and nothing should
|
|
||||||
dirty anything any more. Instead, dirty bits are cleared when the relevant
|
|
||||||
pages are sent during postcopy.
|
|
||||||
|
|
||||||
Postcopy with hugepages
|
Postcopy with hugepages
|
||||||
-----------------------
|
-----------------------
|
||||||
|
|
||||||
@ -293,7 +302,7 @@ Retro-fitting postcopy to existing clients is possible:
|
|||||||
guest memory access is made while holding a lock then all other
|
guest memory access is made while holding a lock then all other
|
||||||
threads waiting for that lock will also be blocked.
|
threads waiting for that lock will also be blocked.
|
||||||
|
|
||||||
Postcopy Preemption Mode
|
Postcopy preemption mode
|
||||||
------------------------
|
------------------------
|
||||||
|
|
||||||
Postcopy preempt is a new capability introduced in 8.0 QEMU release, it
|
Postcopy preempt is a new capability introduced in 8.0 QEMU release, it
|
||||||
|
|||||||
Loading…
Reference in New Issue
Block a user