745c42912a
Loading of a VFIO device's data can take a substantial amount of time as the device may need to allocate resources, prepare internal data structures, etc. This can increase migration downtime, especially for VFIO devices with a lot of resources. To solve this, VFIO migration uAPI defines "initial bytes" as part of its precopy data stream. Initial bytes can be used in various ways to improve VFIO migration performance. For example, it can be used to transfer device metadata to pre-allocate resources in the destination. However, for this to work we need to make sure that all initial bytes are sent and loaded in the destination before the source VM is stopped. Use migration switchover ack capability to make sure a VFIO device's initial bytes are sent and loaded in the destination before the source stops the VM and attempts to complete the migration. This can significantly reduce migration downtime for some devices. Signed-off-by: Avihai Horon <avihaih@nvidia.com> Reviewed-by: Cédric Le Goater <clg@redhat.com> Tested-by: YangHang Liu <yanghliu@redhat.com> Acked-by: Alex Williamson <alex.williamson@redhat.com> Signed-off-by: Cédric Le Goater <clg@redhat.com>
188 lines
8.5 KiB
ReStructuredText
188 lines
8.5 KiB
ReStructuredText
=====================
|
|
VFIO device Migration
|
|
=====================
|
|
|
|
Migration of virtual machine involves saving the state for each device that
|
|
the guest is running on source host and restoring this saved state on the
|
|
destination host. This document details how saving and restoring of VFIO
|
|
devices is done in QEMU.
|
|
|
|
Migration of VFIO devices consists of two phases: the optional pre-copy phase,
|
|
and the stop-and-copy phase. The pre-copy phase is iterative and allows to
|
|
accommodate VFIO devices that have a large amount of data that needs to be
|
|
transferred. The iterative pre-copy phase of migration allows for the guest to
|
|
continue whilst the VFIO device state is transferred to the destination, this
|
|
helps to reduce the total downtime of the VM. VFIO devices opt-in to pre-copy
|
|
support by reporting the VFIO_MIGRATION_PRE_COPY flag in the
|
|
VFIO_DEVICE_FEATURE_MIGRATION ioctl.
|
|
|
|
When pre-copy is supported, it's possible to further reduce downtime by
|
|
enabling "switchover-ack" migration capability.
|
|
VFIO migration uAPI defines "initial bytes" as part of its pre-copy data stream
|
|
and recommends that the initial bytes are sent and loaded in the destination
|
|
before stopping the source VM. Enabling this migration capability will
|
|
guarantee that and thus, can potentially reduce downtime even further.
|
|
|
|
Note that currently VFIO migration is supported only for a single device. This
|
|
is due to VFIO migration's lack of P2P support. However, P2P support is planned
|
|
to be added later on.
|
|
|
|
A detailed description of the UAPI for VFIO device migration can be found in
|
|
the comment for the ``vfio_device_mig_state`` structure in the header file
|
|
linux-headers/linux/vfio.h.
|
|
|
|
VFIO implements the device hooks for the iterative approach as follows:
|
|
|
|
* A ``save_setup`` function that sets up migration on the source.
|
|
|
|
* A ``load_setup`` function that sets the VFIO device on the destination in
|
|
_RESUMING state.
|
|
|
|
* A ``state_pending_estimate`` function that reports an estimate of the
|
|
remaining pre-copy data that the vendor driver has yet to save for the VFIO
|
|
device.
|
|
|
|
* A ``state_pending_exact`` function that reads pending_bytes from the vendor
|
|
driver, which indicates the amount of data that the vendor driver has yet to
|
|
save for the VFIO device.
|
|
|
|
* An ``is_active_iterate`` function that indicates ``save_live_iterate`` is
|
|
active only when the VFIO device is in pre-copy states.
|
|
|
|
* A ``save_live_iterate`` function that reads the VFIO device's data from the
|
|
vendor driver during iterative pre-copy phase.
|
|
|
|
* A ``switchover_ack_needed`` function that checks if the VFIO device uses
|
|
"switchover-ack" migration capability when this capability is enabled.
|
|
|
|
* A ``save_state`` function to save the device config space if it is present.
|
|
|
|
* A ``save_live_complete_precopy`` function that sets the VFIO device in
|
|
_STOP_COPY state and iteratively copies the data for the VFIO device until
|
|
the vendor driver indicates that no data remains.
|
|
|
|
* A ``load_state`` function that loads the config section and the data
|
|
sections that are generated by the save functions above.
|
|
|
|
* ``cleanup`` functions for both save and load that perform any migration
|
|
related cleanup.
|
|
|
|
|
|
The VFIO migration code uses a VM state change handler to change the VFIO
|
|
device state when the VM state changes from running to not-running, and
|
|
vice versa.
|
|
|
|
Similarly, a migration state change handler is used to trigger a transition of
|
|
the VFIO device state when certain changes of the migration state occur. For
|
|
example, the VFIO device state is transitioned back to _RUNNING in case a
|
|
migration failed or was canceled.
|
|
|
|
System memory dirty pages tracking
|
|
----------------------------------
|
|
|
|
A ``log_global_start`` and ``log_global_stop`` memory listener callback informs
|
|
the VFIO dirty tracking module to start and stop dirty page tracking. A
|
|
``log_sync`` memory listener callback queries the dirty page bitmap from the
|
|
dirty tracking module and marks system memory pages which were DMA-ed by the
|
|
VFIO device as dirty. The dirty page bitmap is queried per container.
|
|
|
|
Currently there are two ways dirty page tracking can be done:
|
|
(1) Device dirty tracking:
|
|
In this method the device is responsible to log and report its DMAs. This
|
|
method can be used only if the device is capable of tracking its DMAs.
|
|
Discovering device capability, starting and stopping dirty tracking, and
|
|
syncing the dirty bitmaps from the device are done using the DMA logging uAPI.
|
|
More info about the uAPI can be found in the comments of the
|
|
``vfio_device_feature_dma_logging_control`` and
|
|
``vfio_device_feature_dma_logging_report`` structures in the header file
|
|
linux-headers/linux/vfio.h.
|
|
|
|
(2) VFIO IOMMU module:
|
|
In this method dirty tracking is done by IOMMU. However, there is currently no
|
|
IOMMU support for dirty page tracking. For this reason, all pages are
|
|
perpetually marked dirty, unless the device driver pins pages through external
|
|
APIs in which case only those pinned pages are perpetually marked dirty.
|
|
|
|
If the above two methods are not supported, all pages are perpetually marked
|
|
dirty by QEMU.
|
|
|
|
By default, dirty pages are tracked during pre-copy as well as stop-and-copy
|
|
phase. So, a page marked as dirty will be copied to the destination in both
|
|
phases. Copying dirty pages in pre-copy phase helps QEMU to predict if it can
|
|
achieve its downtime tolerances. If QEMU during pre-copy phase keeps finding
|
|
dirty pages continuously, then it understands that even in stop-and-copy phase,
|
|
it is likely to find dirty pages and can predict the downtime accordingly.
|
|
|
|
QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking``
|
|
which disables querying the dirty bitmap during pre-copy phase. If it is set to
|
|
off, all dirty pages will be copied to the destination in stop-and-copy phase
|
|
only.
|
|
|
|
System memory dirty pages tracking when vIOMMU is enabled
|
|
---------------------------------------------------------
|
|
|
|
With vIOMMU, an IO virtual address range can get unmapped while in pre-copy
|
|
phase of migration. In that case, the unmap ioctl returns any dirty pages in
|
|
that range and QEMU reports corresponding guest physical pages dirty. During
|
|
stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped
|
|
pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those
|
|
mapped ranges. If device dirty tracking is enabled with vIOMMU, live migration
|
|
will be blocked.
|
|
|
|
Flow of state changes during Live migration
|
|
===========================================
|
|
|
|
Below is the flow of state change during live migration.
|
|
The values in the parentheses represent the VM state, the migration state, and
|
|
the VFIO device state, respectively.
|
|
The text in the square brackets represents the flow if the VFIO device supports
|
|
pre-copy.
|
|
|
|
Live migration save path
|
|
------------------------
|
|
|
|
::
|
|
|
|
QEMU normal running state
|
|
(RUNNING, _NONE, _RUNNING)
|
|
|
|
|
migrate_init spawns migration_thread
|
|
Migration thread then calls each device's .save_setup()
|
|
(RUNNING, _SETUP, _RUNNING [_PRE_COPY])
|
|
|
|
|
(RUNNING, _ACTIVE, _RUNNING [_PRE_COPY])
|
|
If device is active, get pending_bytes by .state_pending_{estimate,exact}()
|
|
If total pending_bytes >= threshold_size, call .save_live_iterate()
|
|
[Data of VFIO device for pre-copy phase is copied]
|
|
Iterate till total pending bytes converge and are less than threshold
|
|
|
|
|
On migration completion, vCPU stops and calls .save_live_complete_precopy for
|
|
each active device. The VFIO device is then transitioned into _STOP_COPY state
|
|
(FINISH_MIGRATE, _DEVICE, _STOP_COPY)
|
|
|
|
|
For the VFIO device, iterate in .save_live_complete_precopy until
|
|
pending data is 0
|
|
(FINISH_MIGRATE, _DEVICE, _STOP)
|
|
|
|
|
(FINISH_MIGRATE, _COMPLETED, _STOP)
|
|
Migraton thread schedules cleanup bottom half and exits
|
|
|
|
Live migration resume path
|
|
--------------------------
|
|
|
|
::
|
|
|
|
Incoming migration calls .load_setup for each device
|
|
(RESTORE_VM, _ACTIVE, _STOP)
|
|
|
|
|
For each device, .load_state is called for that device section data
|
|
(RESTORE_VM, _ACTIVE, _RESUMING)
|
|
|
|
|
At the end, .load_cleanup is called for each device and vCPUs are started
|
|
(RUNNING, _NONE, _RUNNING)
|
|
|
|
Postcopy
|
|
========
|
|
|
|
Postcopy migration is currently not supported for VFIO devices.
|