0b2675c473
The term "QEMU global mutex" is identical to the more widely used Big
QEMU Lock ("BQL"). Update the code comments and documentation to use
"BQL" instead of "QEMU global mutex".
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Paul Durrant <paul@xen.org>
Reviewed-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Reviewed-by: Cédric Le Goater <clg@kaod.org>
Reviewed-by: Harsh Prateek Bora <harshpb@linux.ibm.com>
Message-id: 20240102153529.486531-6-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
131 lines
5.9 KiB
Plaintext
131 lines
5.9 KiB
Plaintext
Copyright (c) 2014-2017 Red Hat Inc.
|
|
|
|
This work is licensed under the terms of the GNU GPL, version 2 or later. See
|
|
the COPYING file in the top-level directory.
|
|
|
|
|
|
This document explains the IOThread feature and how to write code that runs
|
|
outside the BQL.
|
|
|
|
The main loop and IOThreads
|
|
---------------------------
|
|
QEMU is an event-driven program that can do several things at once using an
|
|
event loop. The VNC server and the QMP monitor are both processed from the
|
|
same event loop, which monitors their file descriptors until they become
|
|
readable and then invokes a callback.
|
|
|
|
The default event loop is called the main loop (see main-loop.c). It is
|
|
possible to create additional event loop threads using -object
|
|
iothread,id=my-iothread.
|
|
|
|
Side note: The main loop and IOThread are both event loops but their code is
|
|
not shared completely. Sometimes it is useful to remember that although they
|
|
are conceptually similar they are currently not interchangeable.
|
|
|
|
Why IOThreads are useful
|
|
------------------------
|
|
IOThreads allow the user to control the placement of work. The main loop is a
|
|
scalability bottleneck on hosts with many CPUs. Work can be spread across
|
|
several IOThreads instead of just one main loop. When set up correctly this
|
|
can improve I/O latency and reduce jitter seen by the guest.
|
|
|
|
The main loop is also deeply associated with the BQL, which is a
|
|
scalability bottleneck in itself. vCPU threads and the main loop use the BQL
|
|
to serialize execution of QEMU code. This mutex is necessary because a lot of
|
|
QEMU's code historically was not thread-safe.
|
|
|
|
The fact that all I/O processing is done in a single main loop and that the
|
|
BQL is contended by all vCPU threads and the main loop explain
|
|
why it is desirable to place work into IOThreads.
|
|
|
|
The experimental virtio-blk data-plane implementation has been benchmarked and
|
|
shows these effects:
|
|
ftp://public.dhe.ibm.com/linux/pdfs/KVM_Virtualized_IO_Performance_Paper.pdf
|
|
|
|
How to program for IOThreads
|
|
----------------------------
|
|
The main difference between legacy code and new code that can run in an
|
|
IOThread is dealing explicitly with the event loop object, AioContext
|
|
(see include/block/aio.h). Code that only works in the main loop
|
|
implicitly uses the main loop's AioContext. Code that supports running
|
|
in IOThreads must be aware of its AioContext.
|
|
|
|
AioContext supports the following services:
|
|
* File descriptor monitoring (read/write/error on POSIX hosts)
|
|
* Event notifiers (inter-thread signalling)
|
|
* Timers
|
|
* Bottom Halves (BH) deferred callbacks
|
|
|
|
There are several old APIs that use the main loop AioContext:
|
|
* LEGACY qemu_aio_set_fd_handler() - monitor a file descriptor
|
|
* LEGACY qemu_aio_set_event_notifier() - monitor an event notifier
|
|
* LEGACY timer_new_ms() - create a timer
|
|
* LEGACY qemu_bh_new() - create a BH
|
|
* LEGACY qemu_bh_new_guarded() - create a BH with a device re-entrancy guard
|
|
* LEGACY qemu_aio_wait() - run an event loop iteration
|
|
|
|
Since they implicitly work on the main loop they cannot be used in code that
|
|
runs in an IOThread. They might cause a crash or deadlock if called from an
|
|
IOThread since the BQL is not held.
|
|
|
|
Instead, use the AioContext functions directly (see include/block/aio.h):
|
|
* aio_set_fd_handler() - monitor a file descriptor
|
|
* aio_set_event_notifier() - monitor an event notifier
|
|
* aio_timer_new() - create a timer
|
|
* aio_bh_new() - create a BH
|
|
* aio_bh_new_guarded() - create a BH with a device re-entrancy guard
|
|
* aio_poll() - run an event loop iteration
|
|
|
|
The qemu_bh_new_guarded/aio_bh_new_guarded APIs accept a "MemReentrancyGuard"
|
|
argument, which is used to check for and prevent re-entrancy problems. For
|
|
BHs associated with devices, the reentrancy-guard is contained in the
|
|
corresponding DeviceState and named "mem_reentrancy_guard".
|
|
|
|
The AioContext can be obtained from the IOThread using
|
|
iothread_get_aio_context() or for the main loop using qemu_get_aio_context().
|
|
Code that takes an AioContext argument works both in IOThreads or the main
|
|
loop, depending on which AioContext instance the caller passes in.
|
|
|
|
How to synchronize with an IOThread
|
|
-----------------------------------
|
|
Variables that can be accessed by multiple threads require some form of
|
|
synchronization such as qemu_mutex_lock(), rcu_read_lock(), etc.
|
|
|
|
AioContext functions like aio_set_fd_handler(), aio_set_event_notifier(),
|
|
aio_bh_new(), and aio_timer_new() are thread-safe. They can be used to trigger
|
|
activity in an IOThread.
|
|
|
|
Side note: the best way to schedule a function call across threads is to call
|
|
aio_bh_schedule_oneshot().
|
|
|
|
The main loop thread can wait synchronously for a condition using
|
|
AIO_WAIT_WHILE().
|
|
|
|
AioContext and the block layer
|
|
------------------------------
|
|
The AioContext originates from the QEMU block layer, even though nowadays
|
|
AioContext is a generic event loop that can be used by any QEMU subsystem.
|
|
|
|
The block layer has support for AioContext integrated. Each BlockDriverState
|
|
is associated with an AioContext using bdrv_try_change_aio_context() and
|
|
bdrv_get_aio_context(). This allows block layer code to process I/O inside the
|
|
right AioContext. Other subsystems may wish to follow a similar approach.
|
|
|
|
Block layer code must therefore expect to run in an IOThread and avoid using
|
|
old APIs that implicitly use the main loop. See the "How to program for
|
|
IOThreads" above for information on how to do that.
|
|
|
|
Code running in the monitor typically needs to ensure that past
|
|
requests from the guest are completed. When a block device is running
|
|
in an IOThread, the IOThread can also process requests from the guest
|
|
(via ioeventfd). To achieve both objects, wrap the code between
|
|
bdrv_drained_begin() and bdrv_drained_end(), thus creating a "drained
|
|
section".
|
|
|
|
Long-running jobs (usually in the form of coroutines) are often scheduled in
|
|
the BlockDriverState's AioContext. The functions
|
|
bdrv_add/remove_aio_context_notifier, or alternatively
|
|
blk_add/remove_aio_context_notifier if you use BlockBackends, can be used to
|
|
get a notification whenever bdrv_try_change_aio_context() moves a
|
|
BlockDriverState to a different AioContext.
|