Commit Graph

111315 Commits

Author SHA1 Message Date
Fabiano Rosas
06833d83f8 migration/multifd: Allow multifd without packets
For the upcoming support to the new 'mapped-ram' migration stream
format, we cannot use multifd packets because each write into the
ramblock section in the migration file is expected to contain only the
guest pages. They are written at their respective offsets relative to
the ramblock section header.

There is no space for the packet information and the expected gains
from the new approach come partly from being able to write the pages
sequentially without extraneous data in between.

The new format also simply doesn't need the packets and all necessary
information can be taken from the standard migration headers with some
(future) changes to multifd code.

Use the presence of the mapped-ram capability to decide whether to
send packets.

This only moves code under multifd_use_packets(), it has no effect for
now as mapped-ram cannot yet be enabled with multifd.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-15-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Fabiano Rosas
9db1912513 migration/multifd: Decouple recv method from pages
Next patches will abstract the type of data being received by the
channels, so do some cleanup now to remove references to pages and
dependency on 'normal_num'.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-14-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Fabiano Rosas
402dd7ac1c migration/multifd: Rename MultiFDSend|RecvParams::data to compress_data
Use a more specific name for the compression data so we can use the
generic for the multifd core code.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-13-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Fabiano Rosas
c7076ec350 tests/qtest/migration: Add tests for mapped-ram file-based migration
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-12-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Fabiano Rosas
2f6b8826a5 migration/ram: Add incoming 'mapped-ram' migration
Add the necessary code to parse the format changes for the
'mapped-ram' capability.

One of the more notable changes in behavior is that in the
'mapped-ram' case ram pages are restored in one go rather than
constantly looping through the migration stream.

Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-11-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Fabiano Rosas
c2d5c4a7cb migration/ram: Add outgoing 'mapped-ram' migration
Implement the outgoing migration side for the 'mapped-ram' capability.

A bitmap is introduced to track which pages have been written in the
migration file. Pages are written at a fixed location for every
ramblock. Zero pages are ignored as they'd be zero in the destination
migration as well.

The migration stream is altered to put the dirty pages for a ramblock
after its header instead of having a sequential stream of pages that
follow the ramblock headers.

Without mapped-ram (current):        With mapped-ram (new):

 ---------------------               --------------------------------
 | ramblock 1 header |               | ramblock 1 header            |
 ---------------------               --------------------------------
 | ramblock 2 header |               | ramblock 1 mapped-ram header |
 ---------------------               --------------------------------
 | ...               |               | padding to next 1MB boundary |
 ---------------------               | ...                          |
 | ramblock n header |               --------------------------------
 ---------------------               | ramblock 1 pages             |
 | RAM_SAVE_FLAG_EOS |               | ...                          |
 ---------------------               --------------------------------
 | stream of pages   |               | ramblock 2 header            |
 | (iter 1)          |               --------------------------------
 | ...               |               | ramblock 2 mapped-ram header |
 ---------------------               --------------------------------
 | RAM_SAVE_FLAG_EOS |               | padding to next 1MB boundary |
 ---------------------               | ...                          |
 | stream of pages   |               --------------------------------
 | (iter 2)          |               | ramblock 2 pages             |
 | ...               |               | ...                          |
 ---------------------               --------------------------------
 | ...               |               | ...                          |
 ---------------------               --------------------------------
                                     | RAM_SAVE_FLAG_EOS            |
                                     --------------------------------
                                     | ...                          |
                                     --------------------------------

where:
 - ramblock header: the generic information for a ramblock, such as
   idstr, used_len, etc.

 - ramblock mapped-ram header: the new information added by this
   feature: bitmap of pages written, bitmap size and offset of pages
   in the migration file.

Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-10-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Fabiano Rosas
8d9e0d4100 migration: Add mapped-ram URI compatibility check
The mapped-ram migration format needs a channel that supports seeking
to be able to write each page to an arbitrary offset in the migration
stream.

Reviewed-by: "Daniel P. Berrangé" <berrange@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-9-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Fabiano Rosas
4ed49feb44 migration/ram: Introduce 'mapped-ram' migration capability
Add a new migration capability 'mapped-ram'.

The core of the feature is to ensure that RAM pages are mapped
directly to offsets in the resulting migration file instead of being
streamed at arbitrary points.

The reasons why we'd want such behavior are:

 - The resulting file will have a bounded size, since pages which are
   dirtied multiple times will always go to a fixed location in the
   file, rather than constantly being added to a sequential
   stream. This eliminates cases where a VM with, say, 1G of RAM can
   result in a migration file that's 10s of GBs, provided that the
   workload constantly redirties memory.

 - It paves the way to implement O_DIRECT-enabled save/restore of the
   migration stream as the pages are ensured to be written at aligned
   offsets.

 - It allows the usage of multifd so we can write RAM pages to the
   migration file in parallel.

For now, enabling the capability has no effect. The next couple of
patches implement the core functionality.

Acked-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-8-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Fabiano Rosas
7f5b50a401 migration/qemu-file: add utility methods for working with seekable channels
Add utility methods that will be needed when implementing 'mapped-ram'
migration capability.

Signed-off-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: "Daniel P. Berrangé" <berrange@redhat.com>
Link: https://lore.kernel.org/r/20240229153017.2221-7-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Fabiano Rosas
c05dfcb7f2 io: fsync before closing a file channel
Make sure the data is flushed to disk before closing file
channels. This is to ensure data is on disk and not lost in the event
of a host crash.

This is currently being implemented to affect the migration code when
migrating to a file, but all QIOChannelFile users should benefit from
the change.

Reviewed-by: "Daniel P. Berrangé" <berrange@redhat.com>
Acked-by: "Daniel P. Berrangé" <berrange@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-6-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Nikolay Borisov
0478b030fa io: implement io_pwritev/preadv for QIOChannelFile
The upcoming 'mapped-ram' feature will require qemu to write data to
(and restore from) specific offsets of the migration file.

Add a minimal implementation of pwritev/preadv and expose them via the
io_pwritev and io_preadv interfaces.

Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: "Daniel P. Berrangé" <berrange@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-5-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Nikolay Borisov
f1cfe39418 io: Add generic pwritev/preadv interface
Introduce basic pwritev/preadv support in the generic channel layer.
Specific implementation will follow for the file channel as this is
required in order to support migration streams with fixed location of
each ram page.

Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: "Daniel P. Berrangé" <berrange@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-4-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Nikolay Borisov
401e311ff7 io: add and implement QIO_CHANNEL_FEATURE_SEEKABLE for channel file
Add a generic QIOChannel feature SEEKABLE which would be used by the
qemu_file* apis. For the time being this will be only implemented for
file channels.

Signed-off-by: Nikolay Borisov <nborisov@suse.com>
Reviewed-by: "Daniel P. Berrangé" <berrange@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-3-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Fabiano Rosas
4aac6b1e9b migration/multifd: Cleanup multifd_recv_sync_main
Some minor cleanups and documentation for multifd_recv_sync_main.

Use thread_count as done in other parts of the code. Remove p->id from
the multifd_recv_state sync, since that is global and not tied to a
channel. Add documentation for the sync steps.

Reviewed-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240229153017.2221-2-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 15:42:04 +08:00
Bryan Zhang
2b57143231 tests/migration: Set compression level in migration tests
Adds calls to set compression level for `zstd` and `zlib` migration
tests, just to make sure that the calls work.

Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Link: https://lore.kernel.org/r/20240301035901.4006936-3-bryan.zhang@bytedance.com
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 14:14:55 +08:00
Bryan Zhang
b4014a2bf5 migration: Properly apply migration compression level parameters
Some glue code was missing, so that using `qmp_migrate_set_parameters`
to set `multifd-zstd-level` or `multifd-zlib-level` did not work. This
commit adds the glue code to fix that.

Signed-off-by: Bryan Zhang <bryan.zhang@bytedance.com>
Link: https://lore.kernel.org/r/20240301035901.4006936-2-bryan.zhang@bytedance.com
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 14:14:55 +08:00
Steve Sistare
87a2848715 migration: massage cpr-reboot documentation
Re-wrap the cpr-reboot documentation to 70 columns, use '@' for
cpr-reboot references, capitalize COLO and VFIO, and tweak the
wording.

Suggested-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Link: https://lore.kernel.org/r/1709218462-3640-1-git-send-email-steven.sistare@oracle.com
[peterx: s/qemu/QEMU per Markus's suggestion]
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-03-01 14:14:14 +08:00
Peter Maydell
c0c6a0e352 Migration pull request
- Fabiano's fixed-ram patches (1-5 only)
 - Peter's cleanups on multifd tls IOC referencing
 - Steve's cpr patches for vfio (migration patches only)
 - Fabiano's fix on mbps stats racing with COMPLETE state
 - Fabiano's fix on return path thread hang
 -----BEGIN PGP SIGNATURE-----
 
 iIcEABYKADAWIQS5GE3CDMRX2s990ak7X8zN86vXBgUCZd7AbhIccGV0ZXJ4QHJl
 ZGhhdC5jb20ACgkQO1/MzfOr1wbg0gDyA3Vg3pIqCJ+u+hLZ+QKxY/pnu8Y5kF+E
 HK2IdslQUQD+OX4ATUnl+CGMiVX9fjs1fKx0Z0Qetq8gC1YJF13yuA0=
 =P2QF
 -----END PGP SIGNATURE-----

Merge tag 'migration-next-pull-request' of https://gitlab.com/peterx/qemu into staging

Migration pull request

- Fabiano's fixed-ram patches (1-5 only)
- Peter's cleanups on multifd tls IOC referencing
- Steve's cpr patches for vfio (migration patches only)
- Fabiano's fix on mbps stats racing with COMPLETE state
- Fabiano's fix on return path thread hang

# -----BEGIN PGP SIGNATURE-----
#
# iIcEABYKADAWIQS5GE3CDMRX2s990ak7X8zN86vXBgUCZd7AbhIccGV0ZXJ4QHJl
# ZGhhdC5jb20ACgkQO1/MzfOr1wbg0gDyA3Vg3pIqCJ+u+hLZ+QKxY/pnu8Y5kF+E
# HK2IdslQUQD+OX4ATUnl+CGMiVX9fjs1fKx0Z0Qetq8gC1YJF13yuA0=
# =P2QF
# -----END PGP SIGNATURE-----
# gpg: Signature made Wed 28 Feb 2024 05:11:10 GMT
# gpg:                using EDDSA key B9184DC20CC457DACF7DD1A93B5FCCCDF3ABD706
# gpg:                issuer "peterx@redhat.com"
# gpg: Good signature from "Peter Xu <xzpeter@gmail.com>" [marginal]
# gpg:                 aka "Peter Xu <peterx@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg:          It is not certain that the signature belongs to the owner.
# Primary key fingerprint: B918 4DC2 0CC4 57DA CF7D  D1A9 3B5F CCCD F3AB D706

* tag 'migration-next-pull-request' of https://gitlab.com/peterx/qemu: (25 commits)
  migration: Use migrate_has_error() in close_return_path_on_source()
  migration: Join the return path thread before releasing to_dst_file
  migration: Fix qmp_query_migrate mbps value
  migration: options incompatible with cpr
  migration: update cpr-reboot description
  migration: stop vm for cpr
  migration: notifier error checking
  migration: refactor migrate_fd_connect failures
  migration: per-mode notifiers
  migration: MigrationNotifyFunc
  migration: remove postcopy_after_devices
  migration: MigrationEvent for notifiers
  migration: convert to NotifierWithReturn
  migration: remove error from notifier data
  notify: pass error to notifier with return
  migration/multifd: Drop unnecessary helper to destroy IOC
  migration/multifd: Cleanup outgoing_args in state destroy
  migration/multifd: Make multifd_channel_connect() return void
  migration/multifd: Drop registered_yank
  migration/multifd: Cleanup TLS iochannel referencing
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2024-02-28 17:27:10 +00:00
Peter Maydell
bfe8020c81 * target/i386: Fix physical address truncation on 32-bit PAE
* Remove globals for options -no-fd-bootchk and -win2k-hack
 -----BEGIN PGP SIGNATURE-----
 
 iQFIBAABCAAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmXebwQUHHBib256aW5p
 QHJlZGhhdC5jb20ACgkQv/vSX3jHroPozAf/Vgc9u6C+8PcPDrol6qxjI+EOHLNy
 7M3/OFpUkwLXuOSawb6syYxHpLS38fKRcsb2ninngUmbRWA6p+KNUizlAFMj7op5
 wJmtdamCwCwXXaw20SfWxx2Ih0JS7FQsRsU94HTOdaDB17C9+hBcYwcggsOAXCmq
 gyVenEF1mov2A4jLMhdVIRX784AAoEP+QAuhBKQBrQwRLCTTyNdHl7jXdB9w+2sh
 KafokoFLcozJHz/tN3AhRKy6zjPugJyQmJwBRuj9tstCILtXpvf/ZE/3pUq5l3ZY
 A6dCI0zWAlGNTkpKRXsMFozNIVP2htnyidy29XHptlY5acfjtQ++rMu3BQ==
 =WY4H
 -----END PGP SIGNATURE-----

Merge tag 'for-upstream' of https://gitlab.com/bonzini/qemu into staging

* target/i386: Fix physical address truncation on 32-bit PAE
* Remove globals for options -no-fd-bootchk and -win2k-hack

# -----BEGIN PGP SIGNATURE-----
#
# iQFIBAABCAAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmXebwQUHHBib256aW5p
# QHJlZGhhdC5jb20ACgkQv/vSX3jHroPozAf/Vgc9u6C+8PcPDrol6qxjI+EOHLNy
# 7M3/OFpUkwLXuOSawb6syYxHpLS38fKRcsb2ninngUmbRWA6p+KNUizlAFMj7op5
# wJmtdamCwCwXXaw20SfWxx2Ih0JS7FQsRsU94HTOdaDB17C9+hBcYwcggsOAXCmq
# gyVenEF1mov2A4jLMhdVIRX784AAoEP+QAuhBKQBrQwRLCTTyNdHl7jXdB9w+2sh
# KafokoFLcozJHz/tN3AhRKy6zjPugJyQmJwBRuj9tstCILtXpvf/ZE/3pUq5l3ZY
# A6dCI0zWAlGNTkpKRXsMFozNIVP2htnyidy29XHptlY5acfjtQ++rMu3BQ==
# =WY4H
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue 27 Feb 2024 23:23:48 GMT
# gpg:                using RSA key F13338574B662389866C7682BFFBD25F78C7AE83
# gpg:                issuer "pbonzini@redhat.com"
# gpg: Good signature from "Paolo Bonzini <bonzini@gnu.org>" [full]
# gpg:                 aka "Paolo Bonzini <pbonzini@redhat.com>" [full]
# Primary key fingerprint: 46F5 9FBD 57D6 12E7 BFD4  E2F7 7E15 100C CD36 69B1
#      Subkey fingerprint: F133 3857 4B66 2389 866C  7682 BFFB D25F 78C7 AE83

* tag 'for-upstream' of https://gitlab.com/bonzini/qemu:
  ide, vl: turn -win2k-hack into a property on IDE devices
  ide: collapse parameters to ide_init_drive
  target/i386: leave the A20 bit set in the final NPT walk
  target/i386: remove unnecessary/wrong application of the A20 mask
  target/i386: Fix physical address truncation
  target/i386: use separate MMU indexes for 32-bit accesses
  target/i386: introduce function to query MMU indices
  target/i386: check validity of VMCB addresses
  target/i386: mask high bits of CR3 in 32-bit mode
  vl, pc: turn -no-fd-bootchk into a machine property

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2024-02-28 14:23:21 +00:00
Peter Maydell
d316f1b146 Testing, gdbstub and plugin updates:
- fix some test/tcg license headers to GPLv2+
   - bump up check-tcg timeout to 120s
   - avoid re-building VM images too often
   - update OpenBSD to 7.4
   - use GDBFeature to build gdbstub XML
   - unify plugin vcpu count under qemu_plugin_num_vcpus
   - avoid spurious idle/resume callbacks on new vCPUs
   - ensure nios2-linux-user processes async work
   - call vcpu_init plugin callback through async work
   - define plugin helpers when registers being read
   - add plugin API for reading register values
   - add support for register tracking to execlog
   - update plugin docs with assumptions
   - mention plugins can trigger tb_flush in mttcg design doc
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmXfAv0ACgkQ+9DbCVqe
 KkQyogf/X6T5lWsdZGb22FOYzaTLf5gfCPXArIVN+GsjSae3dU6qy/qVM1VRJQPw
 mH8kvMY7QO5V9M2tL33WtZZg6hqWypXYU+Hit6sMmveKYMKS9ESEX28x3yybgt8Y
 fyDywNODX7bs8Wb6NQjVkZvTmM2llrHEtQXPffaXaPyxOAzlGTV9Mf3Sop9rk4nG
 8IchzLmOOQ7XnVst/KRyq+29oOYsbyUtj13tNeWBZ5iXFDT6Q/nGwPQ12U2Ztn9N
 FZvyzGG707dFaEDxIr4pl7n+lHJto29LMlSXlocANwG6wFNP3nfkSw/dXw3nkZZK
 pOfrQKvnnunJKBd7495LYZxTDe505Q==
 =/k97
 -----END PGP SIGNATURE-----

Merge tag 'pull-maintainer-updates-280224-1' of https://gitlab.com/stsquad/qemu into staging

Testing, gdbstub and plugin updates:

  - fix some test/tcg license headers to GPLv2+
  - bump up check-tcg timeout to 120s
  - avoid re-building VM images too often
  - update OpenBSD to 7.4
  - use GDBFeature to build gdbstub XML
  - unify plugin vcpu count under qemu_plugin_num_vcpus
  - avoid spurious idle/resume callbacks on new vCPUs
  - ensure nios2-linux-user processes async work
  - call vcpu_init plugin callback through async work
  - define plugin helpers when registers being read
  - add plugin API for reading register values
  - add support for register tracking to execlog
  - update plugin docs with assumptions
  - mention plugins can trigger tb_flush in mttcg design doc

# -----BEGIN PGP SIGNATURE-----
#
# iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmXfAv0ACgkQ+9DbCVqe
# KkQyogf/X6T5lWsdZGb22FOYzaTLf5gfCPXArIVN+GsjSae3dU6qy/qVM1VRJQPw
# mH8kvMY7QO5V9M2tL33WtZZg6hqWypXYU+Hit6sMmveKYMKS9ESEX28x3yybgt8Y
# fyDywNODX7bs8Wb6NQjVkZvTmM2llrHEtQXPffaXaPyxOAzlGTV9Mf3Sop9rk4nG
# 8IchzLmOOQ7XnVst/KRyq+29oOYsbyUtj13tNeWBZ5iXFDT6Q/nGwPQ12U2Ztn9N
# FZvyzGG707dFaEDxIr4pl7n+lHJto29LMlSXlocANwG6wFNP3nfkSw/dXw3nkZZK
# pOfrQKvnnunJKBd7495LYZxTDe505Q==
# =/k97
# -----END PGP SIGNATURE-----
# gpg: Signature made Wed 28 Feb 2024 09:55:09 GMT
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* tag 'pull-maintainer-updates-280224-1' of https://gitlab.com/stsquad/qemu: (29 commits)
  docs/devel: plugins can trigger a tb flush
  docs/devel: document some plugin assumptions
  docs/devel: lift example and plugin API sections up
  contrib/plugins: extend execlog to track register changes
  contrib/plugins: fix imatch
  tests/tcg: expand insn test case to exercise register API
  plugins: add an API to read registers
  plugins: create CPUPluginState and migrate plugin_mask
  gdbstub: expose api to find registers
  plugins: Use different helpers when reading registers
  cpu: call plugin init hook asynchronously
  linux-user: ensure nios2 processes queued work
  plugins: fix order of init/idle/resume callback
  plugins: add qemu_plugin_num_vcpus function
  plugins: remove previous n_vcpus functions from API
  gdbstub: Add members to identify registers to GDBFeature
  hw/core/cpu: Remove gdb_get_dynamic_xml member
  gdbstub: Infer number of core registers from XML
  gdbstub: Simplify XML lookup
  gdbstub: Change gdb_get_reg_cb and gdb_set_reg_cb
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2024-02-28 14:23:07 +00:00
Pierrick Bouvier
02ca5ec150 docs/devel: plugins can trigger a tb flush
When scoreboards need to be reallocated.

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20240213094009.150349-8-pierrick.bouvier@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-30-alex.bennee@linaro.org>
2024-02-28 09:11:42 +00:00
Alex Bennée
f87b220f2a docs/devel: document some plugin assumptions
While we attempt to hide implementation details from the plugin we
shouldn't be totally obtuse. Let the user know what they can and can't
expect with the various instrumentation options.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-29-alex.bennee@linaro.org>
2024-02-28 09:11:42 +00:00
Alex Bennée
b0b3c0f562 docs/devel: lift example and plugin API sections up
This makes them a bit more visible in the TCG emulation menu rather
than hiding them away bellow the ToC limit.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-28-alex.bennee@linaro.org>
2024-02-28 09:11:42 +00:00
Alex Bennée
af6e4e0a22 contrib/plugins: extend execlog to track register changes
With the new plugin register API we can now track changes to register
values. Currently the implementation is fairly dumb which will slow
down if a large number of register values are being tracked. This
could be improved by only instrumenting instructions which mention
registers we are interested in tracking.

Example usage:

  ./qemu-aarch64 -D plugin.log -d plugin \
     -cpu max,sve256=on \
     -plugin contrib/plugins/libexeclog.so,reg=sp,reg=z\* \
     ./tests/tcg/aarch64-linux-user/sha512-sve

will display in the execlog any changes to the stack pointer (sp) and
the SVE Z registers.

As testing registers every instruction will be quite a heavy operation
there is an additional flag which attempts to optimise the register
tracking by only instrumenting instructions which are likely to change
its value. This relies on the QEMU disassembler showing up the register
names in disassembly so is an explicit opt-in.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Cc: Akihiko Odaki <akihiko.odaki@daynix.com>
Based-On:  <20231025093128.33116-19-akihiko.odaki@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-27-alex.bennee@linaro.org>
2024-02-28 09:11:42 +00:00
Alex Bennée
9e096a76c7 contrib/plugins: fix imatch
We can't directly save the ephemeral imatch from argv as that memory
will get recycled.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-26-alex.bennee@linaro.org>
2024-02-28 09:11:42 +00:00
Alex Bennée
6036b9cfde tests/tcg: expand insn test case to exercise register API
This ensure we at least read every register the plugin API reports at
least once during the check-tcg checks.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-25-alex.bennee@linaro.org>
2024-02-28 09:11:42 +00:00
Alex Bennée
8df5e27cf7 plugins: add an API to read registers
We can only request a list of registers once the vCPU has been
initialised so the user needs to use either call the get function on
vCPU initialisation or during the translation phase.

We don't expose the reg number to the plugin instead hiding it behind
an opaque handle. For now this is just the gdb_regnum encapsulated in
an anonymous GPOINTER but in future as we add more state for plugins
to track we can expand it.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1706
Based-on:  <20231025093128.33116-18-akihiko.odaki@daynix.com>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Reviewed-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-24-alex.bennee@linaro.org>
2024-02-28 09:11:42 +00:00
Alex Bennée
c006147122 plugins: create CPUPluginState and migrate plugin_mask
As we expand the per-vCPU data for plugins we don't want to pollute
CPUState. For now this just moves the plugin_mask (renamed to
event_mask) as the memory callbacks are accessed directly by TCG
generated code.

Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-23-alex.bennee@linaro.org>
2024-02-28 09:11:42 +00:00
Alex Bennée
c3d0b46645 gdbstub: expose api to find registers
Expose an internal API to QEMU to return all the registers for a vCPU.
The list containing the details required to called gdb_read_register().

Based-on:  <20231025093128.33116-15-akihiko.odaki@daynix.com>
Cc: Akihiko Odaki <akihiko.odaki@daynix.com>
Reviewed-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-22-alex.bennee@linaro.org>
2024-02-28 09:11:42 +00:00
Akihiko Odaki
33a277fec0 plugins: Use different helpers when reading registers
This avoids optimizations incompatible when reading registers.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Message-Id: <20231213-gdb-v17-12-777047380591@daynix.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-21-alex.bennee@linaro.org>
2024-02-28 09:11:42 +00:00
Pierrick Bouvier
81cf548f53 cpu: call plugin init hook asynchronously
This ensures we run during a cpu_exec, which allows to call start/end
exclusive from this init hook (needed for new scoreboard API introduced
later).

async work is run before any tb is translated/executed, so we can
guarantee plugin init will be called before any other hook.

The previous change made sure that any idle/resume cb call will not be
done before initializing plugin for a given vcpu.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Message-Id: <20240213094009.150349-5-pierrick.bouvier@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-20-alex.bennee@linaro.org>
2024-02-28 09:11:40 +00:00
Alex Bennée
f2e8d2d899 linux-user: ensure nios2 processes queued work
While async processes are rare for linux-user we do use them from time
to time. The most obvious one is tb_flush when we run out of
translation space. We will also need this when we move plugin
vcpu_init to an async task.

Fix nios2 to follow its older, wiser and more stable siblings.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-19-alex.bennee@linaro.org>
2024-02-28 09:11:40 +00:00
Pierrick Bouvier
c490e681ec plugins: fix order of init/idle/resume callback
We found that vcpu_init_hook was called *after* idle callback.
vcpu_init is called from cpu_realize_fn, while idle/resume cb are called
from qemu_wait_io_event (in vcpu thread).

This change ensures we only call idle and resume cb only once a plugin
was init for a given vcpu.

Next change in the series will run vcpu_init asynchronously, which will
make it run *after* resume callback as well. So we fix this now.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Message-Id: <20240213094009.150349-4-pierrick.bouvier@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-18-alex.bennee@linaro.org>
2024-02-28 09:11:38 +00:00
Pierrick Bouvier
4a448b148c plugins: add qemu_plugin_num_vcpus function
We now keep track of how many vcpus were started. This way, a plugin can
easily query number of any vcpus at any point of execution, which
unifies user and system mode workflows.

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20240213094009.150349-3-pierrick.bouvier@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-17-alex.bennee@linaro.org>
2024-02-28 09:11:36 +00:00
Pierrick Bouvier
926e146eff plugins: remove previous n_vcpus functions from API
This information is already accessible using qemu_info_t during plugin
install.

We will introduce another function (qemu_plugin_num_vcpus) which
represent how many cpus were enabled, by tracking new cpu indexes.

It's a breaking change, so we bump API version.

Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20240213094009.150349-2-pierrick.bouvier@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-16-alex.bennee@linaro.org>
2024-02-28 09:11:34 +00:00
Akihiko Odaki
eb37086fb0 gdbstub: Add members to identify registers to GDBFeature
These members will be used to help plugins to identify registers.
The added members in instances of GDBFeature dynamically generated by
CPUs will be filled in later changes.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Message-Id: <20231213-gdb-v17-10-777047380591@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-15-alex.bennee@linaro.org>
2024-02-28 09:10:11 +00:00
Akihiko Odaki
f1a5287fc3 hw/core/cpu: Remove gdb_get_dynamic_xml member
This function is no longer used.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231213-gdb-v17-9-777047380591@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-14-alex.bennee@linaro.org>
2024-02-28 09:10:06 +00:00
Akihiko Odaki
ecd6f6a882 gdbstub: Infer number of core registers from XML
GDBFeature has the num_regs member so use it where applicable to
remove magic numbers.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Message-Id: <20231213-gdb-v17-8-777047380591@daynix.com>
[AJB: remove core reg check from microblaze read reg]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-13-alex.bennee@linaro.org>
2024-02-28 09:09:58 +00:00
Akihiko Odaki
ee59fa1dd5 gdbstub: Simplify XML lookup
Now we know all instances of GDBFeature that is used in CPU so we can
traverse them to find XML. This removes the need for a CPU-specific
lookup function for dynamic XMLs.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231213-gdb-v17-7-777047380591@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-12-alex.bennee@linaro.org>
2024-02-28 09:09:55 +00:00
Akihiko Odaki
66260159a7 gdbstub: Change gdb_get_reg_cb and gdb_set_reg_cb
Align the parameters of gdb_get_reg_cb and gdb_set_reg_cb with the
gdb_read_register and gdb_write_register members of CPUClass to allow
to unify the logic to access registers of the core and coprocessors
in the future.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231213-gdb-v17-6-777047380591@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-11-alex.bennee@linaro.org>
2024-02-28 09:09:49 +00:00
Akihiko Odaki
c494f8f529 gdbstub: Use GDBFeature for GDBRegisterState
Simplify GDBRegisterState by replacing num_regs and xml members with
one member that points to GDBFeature.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20231213-gdb-v17-5-777047380591@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-10-alex.bennee@linaro.org>
2024-02-28 09:09:43 +00:00
Akihiko Odaki
ac1e867100 gdbstub: Use GDBFeature for gdb_register_coprocessor
This is a tree-wide change to introduce GDBFeature parameter to
gdb_register_coprocessor(). The new parameter just replaces num_regs
and xml parameters for now. GDBFeature will be utilized to simplify XML
lookup in a following change.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231213-gdb-v17-4-777047380591@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-9-alex.bennee@linaro.org>
2024-02-28 09:09:34 +00:00
Akihiko Odaki
33a24910ae target/riscv: Use GDBFeature for dynamic XML
In preparation for a change to use GDBFeature as a parameter of
gdb_register_coprocessor(), convert the internal representation of
dynamic feature from plain XML to GDBFeature.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Message-Id: <20231213-gdb-v17-3-777047380591@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-8-alex.bennee@linaro.org>
2024-02-28 09:09:29 +00:00
Akihiko Odaki
1b53948ff8 target/ppc: Use GDBFeature for dynamic XML
In preparation for a change to use GDBFeature as a parameter of
gdb_register_coprocessor(), convert the internal representation of
dynamic feature from plain XML to GDBFeature.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20231213-gdb-v17-2-777047380591@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-7-alex.bennee@linaro.org>
2024-02-28 09:09:25 +00:00
Akihiko Odaki
690bd97b5b target/arm: Use GDBFeature for dynamic XML
In preparation for a change to use GDBFeature as a parameter of
gdb_register_coprocessor(), convert the internal representation of
dynamic feature from plain XML to GDBFeature.

Signed-off-by: Akihiko Odaki <akihiko.odaki@daynix.com>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20231213-gdb-v17-1-777047380591@daynix.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-6-alex.bennee@linaro.org>
2024-02-28 09:09:16 +00:00
Alex Bennée
8467ac75b3 tests/vm: update openbsd image to 7.4
The old links are dead so even if we have the ISO cached we can't
finish the install. Update to the current stable and tweak the install
strings.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2192
Tested-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-5-alex.bennee@linaro.org>
2024-02-28 09:09:16 +00:00
Alex Bennée
151b7dba39 tests/vm: avoid re-building the VM images all the time
The main problem is that "check-venv" is a .PHONY target will always
evaluate and trigger a full re-build of the VM images. While its
tempting to drop it from the dependencies that does introduce a
breakage on freshly configured builds.

Fortunately we do have the otherwise redundant --force flag for the
script which up until now was always on. If we make the usage of
--force conditional on dependencies other than check-venv triggering
the update we can avoid the costly rebuild and still run cleanly on a
fresh checkout.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2118
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-4-alex.bennee@linaro.org>
2024-02-28 09:09:16 +00:00
Alex Bennée
cdb5bfc9f3 tests/tcg: bump TCG test timeout to 120s
This is less than ideal but easier than making sure we get all the
iterations of the memory test. Update the comment accordingly.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-3-alex.bennee@linaro.org>
2024-02-28 09:09:11 +00:00
Alex Bennée
542b10bd14 tests/tcg: update licenses to GPLv2 as intended
My default header template is GPLv3 but for QEMU code we really should
stick to GPLv2-or-later (allowing others to up-license it if they
wish). While this is test code we should still be consistent on the
source distribution.

I wrote all of this code so its not a problem. However there remains
one GPLv3 file left which is the crt0-tc2x.S for TriCore.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240227144335.1196131-2-alex.bennee@linaro.org>
2024-02-28 09:09:04 +00:00
Cédric Le Goater
9425ef3f99 migration: Use migrate_has_error() in close_return_path_on_source()
close_return_path_on_source() retrieves the migration error from the
the QEMUFile '->to_dst_file' to know if a shutdown is required. This
shutdown is required to exit the return-path thread.

Avoid relying on '->to_dst_file' and use migrate_has_error() instead.

(using to_dst_file is a heuristic to infer whether
rp_state.from_dst_file might be stuck on a recvmsg(). Using a generic
method for detecting errors is more reliable. We also want to reduce
dependency on QEMUFile::last_error)

Suggested-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Cédric Le Goater <clg@redhat.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
[added some words about the motivation for this patch]
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20240226203122.22894-3-farosas@suse.de
Signed-off-by: Peter Xu <peterx@redhat.com>
2024-02-28 11:31:28 +08:00