Python 3 removed `dict.iteritems()` in favor of `dict.items()`. This
means the script currently doesn't work on Python 3.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Message-id: 20230926103436.25700-15-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
By moving the dynamic argument construction to keyword-arguments,
we can remove all of the specialized handling, and streamline it.
If a tracing method wants to access these, they can define the
kwargs, or ignore it be placing `**kwargs` at the end of the
function's arguments list.
Added deprecation warning to Analyzer class to make users aware
of the Analyzer2 class. No removal date is planned.
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Message-id: 20230926103436.25700-13-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Moved event processing to the Analyzer class to separate specific analyzer
logic (like caching and function signatures) from the _process function.
This allows for new types of Analyzer-based subclasses without changing
the core code.
Note, that the fn_cache is important for performance in cases where the
analyzer is branching away from the catch-all a lot. The cache has no
measurable performance penalty.
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Message-id: 20230926103436.25700-12-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
To avoid duplicate code depending on input types and to better handle
open/close of log with a context-manager, we move the logic of process into
_process.
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Message-id: 20230926103436.25700-11-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Moved event_mapping and event_id_to_name down one level in the function
call-stack to keep variable instantiation and usage closer (`process`
and `run` has no use of the variables; `read_trace_records` does).
Instead of passing event_mapping and event_id_to_name to the bottom of
the call-stack, we move their use to `read_trace_records`. This
separates responsibility and ownership of the information.
`read_record` now just reads the arguments from the file-object by
knowning the total number of bytes. Parsing it to specific arguments is
moved up to `read_trace_records`.
Special handling of dropped events removed, as they can be handled
by the general code.
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Message-id: 20230926103436.25700-10-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Instead of explicitly calling `begin` and `end`, we can change the class
to use the context-manager paradigm. This is mostly a styling choice,
used in modern Python code. But it also allows for more advanced analyzers
to handle exceptions gracefully in the `__exit__` method (not
demonstrated here).
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Message-id: 20230926103436.25700-9-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Define `SimpleException` to differentiate our exceptions from generic
exceptions (IOError, etc.). Adapted simpletrace to support this and
output to stderr.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Message-id: 20230926103436.25700-8-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
A failed call to `read_header` wouldn't be handled the same for the two
different code paths (one path would try to use `None` as a list).
Changed to raise exception to be handled centrally. This also allows for
easier unpacking, as errors has been filtered out.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Message-id: 20230926103436.25700-7-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
The call to `getargspec` was deprecated and in Python 3.11 it has been
removed in favor of `getfullargspec`. `getfullargspec` is compatible
with QEMU's requirement of at least Python version 3.6.
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Message-id: 20230926103436.25700-6-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Readability is subjective, but I've expanded the naming of the variables
and arguments, to help with understanding for new eyes on the code.
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: 20230926103436.25700-5-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
The arguments extracted from `sys.argv` named and unpacked to make it
clear what the arguments are and what they're used for.
The two input files were opened, but never explicitly closed. File usage
changed to use `with` statement to take care of this. At the same time,
ownership of the file-object is moved up to `run` function. Added option
to process to support file-like objects.
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Message-id: 20230926103436.25700-4-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
It wasn't clear where the constants and structs came from, so I added
comments to help.
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Message-id: 20230926103436.25700-3-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
It was unclear what was the supported public interface. I.e. when
refactoring the code, what functions/classes are important to retain.
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Mads Ynddal <m.ynddal@samsung.com>
Message-id: 20230926103436.25700-2-mads@ynddal.dk
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
* Simplify the NIC init code of the jazz machine a little bit
* Minor qtest and avocado fixes
-----BEGIN PGP SIGNATURE-----
iQJFBAABCAAvFiEEJ7iIR+7gJQEY8+q5LtnXdP5wLbUFAmURS8gRHHRodXRoQHJl
ZGhhdC5jb20ACgkQLtnXdP5wLbVn4A/+NQKFZcN7gVn5JXkK7kf6i01LNmAoqjj9
QeQL+WCoNC68OApw7DxIEnpBYT0G42NTHHx4SYeOvzJUzCpeWcxYzQUz58ObZML7
+OKsiOsaHu3/qOuihBCn43et6moLdDCWbee5Zr6JQv/Fjn3q3nEQZnJDWdw8vm1v
csYQJZOD6HelLVMmbLfl1szzrykDTT53NhPncH/SjPz6we17sKqHqmT6LBUIsXcV
u2LaowppKmT7Ooexu6SmsCagLhtWuYo1iGGcRqoojtRWo7eZtWLrAy2DJpyFkPBW
AIYBfntRISZv4eBGCxcVfvODD/Q4OXHuYTfGzD3m+ELJ6hUk/+d4/aHJ2hm+KEm+
AD0IpDtimaEmyQTPlaWHhhEur/82JZ+zYlxUMPf3+hglB/rbr6fhA0SMAV6nwR0r
N8jnB8UCml9oDxJVvDZyrcPMGFs1xlr5FVSHHEoL338SvSfjG3NOEtcNao9n6A8d
rO2CfPzI7peQhKWAzJL+qpnmenyIniH23tFnf2mpOZ0g45ZWtJeT0CXL3aQO3XAZ
m56pkM0d/etAHHRoLQ5D/iKZpwiTRLjdzsJ0gMAQsIuRlG/j5h+zou0vUMgm6F8F
igRHLxytlywZBTCABm2XIlKmaJp8hQlVQMpKsv/BwzTvzzk0GGS5d1qzzFt5WWR7
4rSalTn5Xuw=
=FioB
-----END PGP SIGNATURE-----
Merge tag 'pull-request-2023-09-25' of https://gitlab.com/thuth/qemu into staging
* Make keyutils independent from keyring in meson.build
* Simplify the NIC init code of the jazz machine a little bit
* Minor qtest and avocado fixes
# -----BEGIN PGP SIGNATURE-----
#
# iQJFBAABCAAvFiEEJ7iIR+7gJQEY8+q5LtnXdP5wLbUFAmURS8gRHHRodXRoQHJl
# ZGhhdC5jb20ACgkQLtnXdP5wLbVn4A/+NQKFZcN7gVn5JXkK7kf6i01LNmAoqjj9
# QeQL+WCoNC68OApw7DxIEnpBYT0G42NTHHx4SYeOvzJUzCpeWcxYzQUz58ObZML7
# +OKsiOsaHu3/qOuihBCn43et6moLdDCWbee5Zr6JQv/Fjn3q3nEQZnJDWdw8vm1v
# csYQJZOD6HelLVMmbLfl1szzrykDTT53NhPncH/SjPz6we17sKqHqmT6LBUIsXcV
# u2LaowppKmT7Ooexu6SmsCagLhtWuYo1iGGcRqoojtRWo7eZtWLrAy2DJpyFkPBW
# AIYBfntRISZv4eBGCxcVfvODD/Q4OXHuYTfGzD3m+ELJ6hUk/+d4/aHJ2hm+KEm+
# AD0IpDtimaEmyQTPlaWHhhEur/82JZ+zYlxUMPf3+hglB/rbr6fhA0SMAV6nwR0r
# N8jnB8UCml9oDxJVvDZyrcPMGFs1xlr5FVSHHEoL338SvSfjG3NOEtcNao9n6A8d
# rO2CfPzI7peQhKWAzJL+qpnmenyIniH23tFnf2mpOZ0g45ZWtJeT0CXL3aQO3XAZ
# m56pkM0d/etAHHRoLQ5D/iKZpwiTRLjdzsJ0gMAQsIuRlG/j5h+zou0vUMgm6F8F
# igRHLxytlywZBTCABm2XIlKmaJp8hQlVQMpKsv/BwzTvzzk0GGS5d1qzzFt5WWR7
# 4rSalTn5Xuw=
# =FioB
# -----END PGP SIGNATURE-----
# gpg: Signature made Mon 25 Sep 2023 04:58:48 EDT
# gpg: using RSA key 27B88847EEE0250118F3EAB92ED9D774FE702DB5
# gpg: issuer "thuth@redhat.com"
# gpg: Good signature from "Thomas Huth <th.huth@gmx.de>" [full]
# gpg: aka "Thomas Huth <thuth@redhat.com>" [full]
# gpg: aka "Thomas Huth <huth@tuxfamily.org>" [full]
# gpg: aka "Thomas Huth <th.huth@posteo.de>" [unknown]
# Primary key fingerprint: 27B8 8847 EEE0 2501 18F3 EAB9 2ED9 D774 FE70 2DB5
* tag 'pull-request-2023-09-25' of https://gitlab.com/thuth/qemu:
tests/avocado: fix waiting for vm shutdown in replay_linux
hw/mips/jazz: Simplify the NIC setup code
hw/mips/jazz: Move the NIC init code into a separate function
tests/qtest/netdev-socket: Do not test multicast on Darwin
tests/qtest/m48t59-test: Silence compiler warning with -Wshadow
tests/qtest/netdev-socket: Raise connection timeout to 120 seconds
meson.build: Make keyutils independent from keyring
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Commit 0db0fbb5cf ("Add conditional dependency for libkeyutils")
tried to provide a possibility for the user to disable keyutils
if not required by makeing it depend on the keyring feature. This
looked reasonable at a first glance (the unit test in tests/unit/
needs both), but the condition in meson.build fails if the feature
is meant to be detected automatically, and there is also another
spot in backends/meson.build where keyutils is used independently
from keyring. So let's remove the dependency on keyring again and
introduce a proper meson build option instead.
Cc: qemu-stable@nongnu.org
Fixes: 0db0fbb5cf ("Add conditional dependency for libkeyutils")
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1842
Message-ID: <20230824094208.255279-1-thuth@redhat.com>
Reviewed-by: "Daniel P. Berrangé" <berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
On parts that enumerate IA32_VMX_BASIC MSR bit as 1, any exception vector
can be delivered with or without an error code if the other consistency
checks are satisfied.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Don't assume specific parameter names like 'bs' or 'blk' in the
generated code, but use the actual name.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20230911094620.45040-8-kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Add a new wrapper type for GRAPH_WRLOCK functions that should be called
from coroutine context.
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
Reviewed-by: Emanuele Giuseppe Esposito <eesposit@redhat.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-ID: <20230911094620.45040-7-kwolf@redhat.com>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
AF_XDP is a network socket family that allows communication directly
with the network device driver in the kernel, bypassing most or all
of the kernel networking stack. In the essence, the technology is
pretty similar to netmap. But, unlike netmap, AF_XDP is Linux-native
and works with any network interfaces without driver modifications.
Unlike vhost-based backends (kernel, user, vdpa), AF_XDP doesn't
require access to character devices or unix sockets. Only access to
the network interface itself is necessary.
This patch implements a network backend that communicates with the
kernel by creating an AF_XDP socket. A chunk of userspace memory
is shared between QEMU and the host kernel. 4 ring buffers (Tx, Rx,
Fill and Completion) are placed in that memory along with a pool of
memory buffers for the packet data. Data transmission is done by
allocating one of the buffers, copying packet data into it and
placing the pointer into Tx ring. After transmission, device will
return the buffer via Completion ring. On Rx, device will take
a buffer form a pre-populated Fill ring, write the packet data into
it and place the buffer into Rx ring.
AF_XDP network backend takes on the communication with the host
kernel and the network interface and forwards packets to/from the
peer device in QEMU.
Usage example:
-device virtio-net-pci,netdev=guest1,mac=00:16:35:AF:AA:5C
-netdev af-xdp,ifname=ens6f1np1,id=guest1,mode=native,queues=1
XDP program bridges the socket with a network interface. It can be
attached to the interface in 2 different modes:
1. skb - this mode should work for any interface and doesn't require
driver support. With a caveat of lower performance.
2. native - this does require support from the driver and allows to
bypass skb allocation in the kernel and potentially use
zero-copy while getting packets in/out userspace.
By default, QEMU will try to use native mode and fall back to skb.
Mode can be forced via 'mode' option. To force 'copy' even in native
mode, use 'force-copy=on' option. This might be useful if there is
some issue with the driver.
Option 'queues=N' allows to specify how many device queues should
be open. Note that all the queues that are not open are still
functional and can receive traffic, but it will not be delivered to
QEMU. So, the number of device queues should generally match the
QEMU configuration, unless the device is shared with something
else and the traffic re-direction to appropriate queues is correctly
configured on a device level (e.g. with ethtool -N).
'start-queue=M' option can be used to specify from which queue id
QEMU should start configuring 'N' queues. It might also be necessary
to use this option with certain NICs, e.g. MLX5 NICs. See the docs
for examples.
In a general case QEMU will need CAP_NET_ADMIN and CAP_SYS_ADMIN
or CAP_BPF capabilities in order to load default XSK/XDP programs to
the network interface and configure BPF maps. It is possible, however,
to run with no capabilities. For that to work, an external process
with enough capabilities will need to pre-load default XSK program,
create AF_XDP sockets and pass their file descriptors to QEMU process
on startup via 'sock-fds' option. Network backend will need to be
configured with 'inhibit=on' to avoid loading of the program.
QEMU will need 32 MB of locked memory (RLIMIT_MEMLOCK) per queue
or CAP_IPC_LOCK.
There are few performance challenges with the current network backends.
First is that they do not support IO threads. This means that data
path is handled by the main thread in QEMU and may slow down other
work or may be slowed down by some other work. This also means that
taking advantage of multi-queue is generally not possible today.
Another thing is that data path is going through the device emulation
code, which is not really optimized for performance. The fastest
"frontend" device is virtio-net. But it's not optimized for heavy
traffic either, because it expects such use-cases to be handled via
some implementation of vhost (user, kernel, vdpa). In practice, we
have virtio notifications and rcu lock/unlock on a per-packet basis
and not very efficient accesses to the guest memory. Communication
channels between backend and frontend devices do not allow passing
more than one packet at a time as well.
Some of these challenges can be avoided in the future by adding better
batching into device emulation or by implementing vhost-af-xdp variant.
There are also a few kernel limitations. AF_XDP sockets do not
support any kinds of checksum or segmentation offloading. Buffers
are limited to a page size (4K), i.e. MTU is limited. Multi-buffer
support implementation for AF_XDP is in progress, but not ready yet.
Also, transmission in all non-zero-copy modes is synchronous, i.e.
done in a syscall. That doesn't allow high packet rates on virtual
interfaces.
However, keeping in mind all of these challenges, current implementation
of the AF_XDP backend shows a decent performance while running on top
of a physical NIC with zero-copy support.
Test setup:
2 VMs running on 2 physical hosts connected via ConnectX6-Dx card.
Network backend is configured to open the NIC directly in native mode.
The driver supports zero-copy. NIC is configured to use 1 queue.
Inside a VM - iperf3 for basic TCP performance testing and dpdk-testpmd
for PPS testing.
iperf3 result:
TCP stream : 19.1 Gbps
dpdk-testpmd (single queue, single CPU core, 64 B packets) results:
Tx only : 3.4 Mpps
Rx only : 2.0 Mpps
L2 FWD Loopback : 1.5 Mpps
In skb mode the same setup shows much lower performance, similar to
the setup where pair of physical NICs is replaced with veth pair:
iperf3 result:
TCP stream : 9 Gbps
dpdk-testpmd (single queue, single CPU core, 64 B packets) results:
Tx only : 1.2 Mpps
Rx only : 1.0 Mpps
L2 FWD Loopback : 0.7 Mpps
Results in skb mode or over the veth are close to results of a tap
backend with vhost=on and disabled segmentation offloading bridged
with a NIC.
Signed-off-by: Ilya Maximets <i.maximets@ovn.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com> (docker/lcitool)
Signed-off-by: Jason Wang <jasowang@redhat.com>
* target/i386: fix BQL handling of the legacy FERR interrupts
* target/i386: fix memory operand size for CVTPS2PD
* target/i386: Add support for AMX-COMPLEX in CPUID enumeration
* compile plugins on Darwin
* configure and meson cleanups
* drop mkvenv support for Python 3.7 and Debian10
* add wrap file for libblkio
* tweak KVM stubs
-----BEGIN PGP SIGNATURE-----
iQFIBAABCAAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmT5t6UUHHBib256aW5p
QHJlZGhhdC5jb20ACgkQv/vSX3jHroMmjwf+MpvVuq+nn+3PqGUXgnzJx5ccA5ne
O9Xy8+1GdlQPzBw/tPovxXDSKn3HQtBfxObn2CCE1tu/4uHWpBA1Vksn++NHdUf2
P0yoHxGskJu5iYYTtIcNw5cH2i+AizdiXuEjhfNjqD5Y234cFoHnUApt9e3zBvVO
cwGD7WpPuSb4g38hHkV6nKcx72o7b4ejDToqUVZJ2N+RkddSqB03fSdrOru0hR7x
V+lay0DYdFszNDFm05LJzfDbcrHuSryGA91wtty7Fzj6QhR/HBHQCUZJxMB5PI7F
Zy4Zdpu60zxtSxUqeKgIi7UhNFgMcax2Hf9QEqdc/B4ARoBbboh4q4u8kQ==
=dH7/
-----END PGP SIGNATURE-----
Merge tag 'for-upstream' of https://gitlab.com/bonzini/qemu into staging
* only build util/async-teardown.c when system build is requested
* target/i386: fix BQL handling of the legacy FERR interrupts
* target/i386: fix memory operand size for CVTPS2PD
* target/i386: Add support for AMX-COMPLEX in CPUID enumeration
* compile plugins on Darwin
* configure and meson cleanups
* drop mkvenv support for Python 3.7 and Debian10
* add wrap file for libblkio
* tweak KVM stubs
# -----BEGIN PGP SIGNATURE-----
#
# iQFIBAABCAAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmT5t6UUHHBib256aW5p
# QHJlZGhhdC5jb20ACgkQv/vSX3jHroMmjwf+MpvVuq+nn+3PqGUXgnzJx5ccA5ne
# O9Xy8+1GdlQPzBw/tPovxXDSKn3HQtBfxObn2CCE1tu/4uHWpBA1Vksn++NHdUf2
# P0yoHxGskJu5iYYTtIcNw5cH2i+AizdiXuEjhfNjqD5Y234cFoHnUApt9e3zBvVO
# cwGD7WpPuSb4g38hHkV6nKcx72o7b4ejDToqUVZJ2N+RkddSqB03fSdrOru0hR7x
# V+lay0DYdFszNDFm05LJzfDbcrHuSryGA91wtty7Fzj6QhR/HBHQCUZJxMB5PI7F
# Zy4Zdpu60zxtSxUqeKgIi7UhNFgMcax2Hf9QEqdc/B4ARoBbboh4q4u8kQ==
# =dH7/
# -----END PGP SIGNATURE-----
# gpg: Signature made Thu 07 Sep 2023 07:44:37 EDT
# gpg: using RSA key F13338574B662389866C7682BFFBD25F78C7AE83
# gpg: issuer "pbonzini@redhat.com"
# gpg: Good signature from "Paolo Bonzini <bonzini@gnu.org>" [full]
# gpg: aka "Paolo Bonzini <pbonzini@redhat.com>" [full]
# Primary key fingerprint: 46F5 9FBD 57D6 12E7 BFD4 E2F7 7E15 100C CD36 69B1
# Subkey fingerprint: F133 3857 4B66 2389 866C 7682 BFFB D25F 78C7 AE83
* tag 'for-upstream' of https://gitlab.com/bonzini/qemu: (51 commits)
docs/system/replay: do not show removed command line option
subprojects: add wrap file for libblkio
sysemu/kvm: Restrict kvm_pc_setup_irq_routing() to x86 targets
sysemu/kvm: Restrict kvm_has_pit_state2() to x86 targets
sysemu/kvm: Restrict kvm_get_apic_state() to x86 targets
sysemu/kvm: Restrict kvm_arch_get_supported_cpuid/msr() to x86 targets
target/i386: Restrict declarations specific to CONFIG_KVM
target/i386: Allow elision of kvm_hv_vpindex_settable()
target/i386: Allow elision of kvm_enable_x2apic()
target/i386: Remove unused KVM stubs
target/i386/cpu-sysemu: Inline kvm_apic_in_kernel()
target/i386/helper: Restrict KVM declarations to system emulation
hw/i386/fw_cfg: Include missing 'cpu.h' header
hw/i386/pc: Include missing 'cpu.h' header
hw/i386/pc: Include missing 'sysemu/tcg.h' header
Revert "mkvenv: work around broken pip installations on Debian 10"
mkvenv: assume presence of importlib.metadata
Python: Drop support for Python 3.7
configure: remove dead code
meson: list leftover CONFIG_* symbols
...
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Debian 10 is not anymore a supported distro, since Debian 12 was
released on June 10, 2023. Our supported build platforms as of today
all support at least 3.8 (and all of them except for Ubuntu 20.04
support 3.9):
openSUSE Leap 15.5: 3.6.15 (3.11.2)
CentOS Stream 8: 3.6.8 (3.8.13, 3.9.16, 3.11.4)
CentOS Stream 9: 3.9.17 (3.11.4)
Fedora 37: 3.11.4
Fedora 38: 3.11.4
Debian 11: 3.9.2
Debian 12: 3.11.2
Alpine 3.14, 3.15: 3.9.16
Alpine 3.16, 3.17: 3.10.10
Ubuntu 20.04 LTS: 3.8.10
Ubuntu 22.04 LTS: 3.10.12
NetBSD 9.3: 3.9.13*
FreeBSD 12.4: 3.9.16
FreeBSD 13.1: 3.9.18
OpenBSD 7.2: 3.9.17
Note: NetBSD does not appear to have a default meta-package, but offers
several options, the lowest of which is 3.7.15. However, "python39"
appears to be a pre-requisite to one of the other packages we request
in tests/vm/netbsd.
Since it is safe under our supported platform policy, bump our
minimum supported version of Python to 3.8. The two most interesting
features to have by default include:
- the importlib.metadata module, whose lack is responsible for over 100
lines of code in mkvenv.py
- improvements to asyncio, for example asyncio.CancelledError
inherits from BaseException rather than Exception
In addition, code can now use the assignment operator ':='
Because mypy now learns about importlib.metadata, a small change to
mkvenv.py is needed to pass type checking.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
While the option still needs to be parsed in the configure script
(it's needed by tests/tcg, and also to decide about recursing
into contrib/plugins), passing it to Meson can be done with -D
instead of using config-host.mak.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
HAX is deprecated since commits 73741fda6c ("MAINTAINERS: Abort
HAXM maintenance") and 90c167a1da ("docs/about/deprecated: Mark
HAXM in QEMU as deprecated"), released in v8.0.0.
Per the latest HAXM release (v7.8 [*]), the latest QEMU supported
is v7.2:
Note: Up to this release, HAXM supports QEMU from 2.9.0 to 7.2.0.
The next commit (https://github.com/intel/haxm/commit/da1b8ec072)
added:
HAXM v7.8.0 is our last release and we will not accept
pull requests or respond to issues after this.
It became very hard to build and test HAXM. Its previous
maintainers made it clear they won't help. It doesn't seem to be
a very good use of QEMU maintainers to spend their time in a dead
project. Save our time by removing this orphan zombie code.
[*] https://github.com/intel/haxm/releases/tag/v7.8.0
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230831082016.60885-1-philmd@linaro.org>
This reverts commit e8e4298fea.
ensuregroup allows to specify both the acceptable versions of avocado,
and a locked version to be used when avocado is not installed as a system
pacakge. This lets us install avocado in pyvenv/ using "mkvenv.py" and
reuse the distro package on Fedora and CentOS Stream (the only distros
where it's available).
ensuregroup's usage of "(>=..., <=...)" constraints when evaluating
the distro package, and "==" constraints when installing it from PyPI,
makes it possible to avoid conflicts between the known-good version and
a package plugins included in the distro.
This is because package plugins have "==" constraints on the version
that is included in the distro, and, using "pip install avocado==88.1"
on a venv that includes system packages will result in an error:
avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
But at the same time, if the venv does not include a system distribution
of avocado then we can install a known-good version and stick to LTS
releases.
Resolves: https://gitlab.com/qemu-project/qemu/-/issues/1663
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
The "expected failure" tests for decodetree result in the
error messages from decodetree ending up in logs and in
V=1 output:
>>> MALLOC_PERTURB_=226 /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86/pyvenv/bin/python3 /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/scripts/decodetree.py --output-null --test-for-error /mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86/../../tests/decode/err_argset1.decode
――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――― ✀ ――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
/mnt/nvmedisk/linaro/qemu-from-laptop/qemu/build/x86/../../tests/decode/err_argset1.decode:5: error: duplicate argument "a"
―――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――――
1/44 qemu:decodetree / err_argset1 OK 0.05s
This then produces false positives when scanning the
logfiles for strings like "error: ".
For the expected-failure tests, make decodetree print
"detected:" instead of "error:".
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-id: 20230720131521.1325905-1-peter.maydell@linaro.org
The POSIX definition of the 'read' utility requires that you
specify the variable name to set; omitting the name and
having it default to 'REPLY' is a bashism. If your system
sh is dash, then it will print an error message during build:
qemu/pc-bios/s390-ccw/../../scripts/git-submodule.sh: 106: read: arg count
Specify the variable name explicitly.
Fixes: fdb8fd8cb9 ("git-submodule: allow partial update of .git-submodule-status")
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-id: 20230720153038.1587196-1-peter.maydell@linaro.org
We are not mixing C++ with C code anymore, the only remaining
C++ code in qga/vss-win32/ is used for a plain C++ executable.
Thus we can remove the hacks for linking C code with the C++ linker
now to simplify meson.build a little bit, and also to avoid that
some C++ code sneaks in by accident again.
Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20230706064736.178962-1-thuth@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
When updating to the latest fedora the santizer found more leaks
inside xkbmap:
FAILED: pc-bios/keymaps/ar
/builds/stsquad/qemu/build-oss-fuzz/qemu-keymap -f pc-bios/keymaps/ar -l ara
=================================================================
==3604==ERROR: LeakSanitizer: detected memory leaks
Direct leak of 1424 byte(s) in 1 object(s) allocated from:
#0 0x56316418ebec in __interceptor_calloc (/builds/stsquad/qemu/build-oss-fuzz/qemu-keymap+0x127bec) (BuildId: a2ad9da3190962acaa010fa8f44a9269f9081e1c)
#1 0x7f60d4dc067e (/lib64/libxkbcommon.so.0+0x1c67e) (BuildId: b243a34e4e58e6a30b93771c256268b114d34b80)
#2 0x7f60d4dc2137 in xkb_keymap_new_from_names (/lib64/libxkbcommon.so.0+0x1e137) (BuildId: b243a34e4e58e6a30b93771c256268b114d34b80)
#3 0x5631641ca50f in main /builds/stsquad/qemu/build-oss-fuzz/../qemu-keymap.c:215:11
and many more. As we can't do anything about the library add a
suppression to keep the CI going with what its meant to be doing.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20230630180423.558337-8-alex.bennee@linaro.org>
gdbus-codegen doesn't support conditions or pre-processing.
Rather than duplicating D-Bus interfaces for win32 adaptation, let's
have a preprocess step, so we can have platform-specific interfaces.
The python script is based on
https://github.com/peitaosu/XML-Preprocessor, with bug fixes, some
testing and replacing lxml dependency with the built-in xml module.
This preprocessing syntax style is not very common, but is similar to
the one provided by WiX (https://wixtoolset.org/docs/v3/overview/preprocessor/)
or wixl, that we adopted in QEMU for packaging the guest agent.
Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20230606115658.677673-5-marcandre.lureau@redhat.com>
TBStats will be introduced to replace CONFIG_PROFILER totally, here
remove all CONFIG_PROFILER related stuffs first.
Signed-off-by: Vanderson M. do Rosario <vandersonmr2@gmail.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Fei Wu <fei2.wu@intel.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20230607122411.3394702-2-fei2.wu@intel.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
The call to git-submodule.sh done in configure may happen without a
previous checkout of the roms/SLOF submodule, or even without a
previous run of the script.
So, handle creating a .git-submodule-status file even in validate
mode. If git is absent, ensure that all passed directories exists
(because you should be in a fresh untar and will not have stale
arguments to git-submodule.sh) but do no other checks. If git
is present, ensure that .git-submodule-status contains an entry
for all submodules passed on the command line.
With this change, "ignore" mode is not needed anymore.
Reported-by: Nina Schoetterl-Glausch <nsg@linux.ibm.com>
Fixes: b11f9bd96f ("configure: move SLOF submodule handling to pc-bios/s390-ccw", 2023-06-06)
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
In restructuring to allow for internal emulation of Xen functionality,
I broke compatibility for Xen 4.6 and earlier. Fix this by explicitly
removing support for anything older than 4.7.1, which is also ancient
but it does still build, and the compatibility support for it is fairly
unintrusive.
Fixes: 15e283c5b6 ("hw/xen: Add foreignmem operations to allow redirection to internal emulation")
Signed-off-by: David Woodhouse <dwmw@amazon.co.uk>
Reviewed-by: Paul Durrant <paul@xen.org>
Message-Id: <20230412185102.441523-4-dwmw2@infradead.org>
Signed-off-by: Anthony PERARD <anthony.perard@citrix.com>
* meson.build tweaks
* revert avocado update
* always upgrade/downgrade locally installed Python packages
* switch from submodules to subprojects
* remove --with-git= option
* rename --enable-pypi to --enable-download, control submodules and subprojects too
-----BEGIN PGP SIGNATURE-----
iQFIBAABCAAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmR/Qu8UHHBib256aW5p
QHJlZGhhdC5jb20ACgkQv/vSX3jHroMmSwgAj5SHD8R+5D1UmptzBvI/72CfgqVv
MJa4O1LvHwUkuSmxX1MFFhRa0mo0bu6j+bPpvJ29zKS61ybVwJl87gnsRcDAMXe7
08YbcG35Chox6aZxbidUQtXm18JZ3F2aMtmxUuP0PR7LDjVXLV5FsjrHTIt8KuEZ
vUqq3IsVbc4FxCCC0ke2DzrtgpRCxYSdfPrj/t5WzAztAXId9r1zvUlCLN+FUpri
E3KIZYpkXZyOnJQ9W30KnsZo5QtDACwlIMBK6whSdoCjyNN7TwDdhNW8QkOueNO6
q3tLfwf5+u6uyEoaQTW+teE2oMXT8N4IJllRJj2RyQ1BFD49XhUUJmc33Q==
=b9QD
-----END PGP SIGNATURE-----
Merge tag 'for-upstream' of https://gitlab.com/bonzini/qemu into staging
* finish atomics revamp
* meson.build tweaks
* revert avocado update
* always upgrade/downgrade locally installed Python packages
* switch from submodules to subprojects
* remove --with-git= option
* rename --enable-pypi to --enable-download, control submodules and subprojects too
# -----BEGIN PGP SIGNATURE-----
#
# iQFIBAABCAAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmR/Qu8UHHBib256aW5p
# QHJlZGhhdC5jb20ACgkQv/vSX3jHroMmSwgAj5SHD8R+5D1UmptzBvI/72CfgqVv
# MJa4O1LvHwUkuSmxX1MFFhRa0mo0bu6j+bPpvJ29zKS61ybVwJl87gnsRcDAMXe7
# 08YbcG35Chox6aZxbidUQtXm18JZ3F2aMtmxUuP0PR7LDjVXLV5FsjrHTIt8KuEZ
# vUqq3IsVbc4FxCCC0ke2DzrtgpRCxYSdfPrj/t5WzAztAXId9r1zvUlCLN+FUpri
# E3KIZYpkXZyOnJQ9W30KnsZo5QtDACwlIMBK6whSdoCjyNN7TwDdhNW8QkOueNO6
# q3tLfwf5+u6uyEoaQTW+teE2oMXT8N4IJllRJj2RyQ1BFD49XhUUJmc33Q==
# =b9QD
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue 06 Jun 2023 07:30:07 AM PDT
# gpg: using RSA key F13338574B662389866C7682BFFBD25F78C7AE83
# gpg: issuer "pbonzini@redhat.com"
# gpg: Good signature from "Paolo Bonzini <bonzini@gnu.org>" [unknown]
# gpg: aka "Paolo Bonzini <pbonzini@redhat.com>" [unknown]
# gpg: WARNING: This key is not certified with a trusted signature!
# gpg: There is no indication that the signature belongs to the owner.
# Primary key fingerprint: 46F5 9FBD 57D6 12E7 BFD4 E2F7 7E15 100C CD36 69B1
# Subkey fingerprint: F133 3857 4B66 2389 866C 7682 BFFB D25F 78C7 AE83
* tag 'for-upstream' of https://gitlab.com/bonzini/qemu: (21 commits)
configure: remove --with-git-submodules=
build: remove git submodule handling from main makefile
meson: subprojects: replace berkeley-{soft,test}float-3 with wraps
pc-bios/s390-ccw: always build network bootloader
configure: move SLOF submodule handling to pc-bios/s390-ccw
meson: subprojects: replace submodules with wrap files
build: log submodule update from git-submodule.sh
git-submodule: allow partial update of .git-submodule-status
configure: rename --enable-pypi to --enable-download, control subprojects too
configure: remove --with-git= option
mkvenv: always pass locally-installed packages to pip
tests: Use separate virtual environment for avocado
Revert "tests/requirements.txt: bump up avocado-framework version to 101.0"
scsi/qemu-pr-helper: Drop support for 'old' libmultipath API
meson.build: Use -Wno-undef only for SDL2 versions that need it
meson.build: Group the audio backend entries in a separate summary section
meson.build: Group the network backend entries in a separate summary section
meson.build: Group the UI entries in a separate summary section
scripts: remove dead file
atomics: eliminate mb_read/mb_set
...
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reuse --enable/--disable-download to control git submodules as well.
Adjust the error messages of git-submodule.sh to refer to the new
option.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Unlike other subprojects, these require an overlay directory to include
meson rules to build the libraries. The rules are basically lifted
from tests/fp/meson.build, with a few changes to create platform.h
and publish a dependency.
The build defines are passed through a subproject option, and posted
back to users of the library via the dependency's compile_args.
The only remaining user of GIT_SUBMODULES and GIT_SUBMODULES_ACTION
is roms/SLOF, which is used to build pc-bios/s390-ccw. All other
roms submodules are only present to satisfy the license on pre-built
firmware blobs.
Best reviewed with --color-moved.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Compared to submodules, .wrap files have several advantages:
* option parsing and downloading is delegated to meson
* the commit is stored in a text file instead of a magic entry in the
git tree object
* we could stop shipping external dependencies that are only used as a
fallback, but not break compilation on platforms that lack them.
For example it may make sense to download dtc at build time, controlled
by --enable-download, even when building from a tarball. Right now,
this patch does the opposite: make-release treats dtc like libvfio-user
(which is not stable API and therefore hasn't found its way into any
distros) and keycodemap (which is a copylib, for better or worse).
dependency() can fall back to a wrap automatically. However, this
is only possible for libraries that come with a .pc file, and this
is not very common for libfdt even though the upstream project in
principle provides it; it also removes the control that we provide with
--enable-fdt={system,internal}. Therefore, the logic to pick system
vs. internal libfdt is left untouched.
--enable-fdt=git is removed; it was already a synonym for
--enable-fdt=internal.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Print exactly which submodules have been updated, by reusing the logic of
"git-submodule.sh validate" after executing "git submodule update --init'.
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
Allow a specific subdirectory to run git-submodule.sh with only a
subset of submodules, without removing the others from the
.git-submodule-status file.
This also allows scripts/git-submodule.sh to be more lenient:
validating an empty set of submodules is not a mistake.
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
The scenario for which --with-git= was introduced was to use a SOCKS proxy
such as tsocks. However, this was back in 2017 when QEMU's submodules
used the git:// protocol, and it is not as important when using the
"smart HTTP" backend; for example, neither "meson subprojects download"
nor scripts/checkpatch.pl obey the GIT environment variable.
So remove the knob, but test for the presence of git in the configure and
git-submodule.sh scripts, and suggest using --with-git-submodules=validate
+ a manual invocation of git-submodule.sh when git does not work. Hopefully
in the future the GIT environment variable will be supported by Meson.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This reverts commits eea2d14117 ("Makefile: remove $(TESTS_PYTHON)",
2023-05-26) and 9c6692db55 ("tests: Use configure-provided pyvenv for
tests", 2023-05-18).
Right now, there is a conflict between wanting a ">=" constraint when
using a distro-provided package and wanting a "==" constraint when
installing Avocado from PyPI; this would provide the best of both worlds
in terms of resiliency for both distros that have required packages and
distros that don't.
The conflict is visible also for meson, where we would like to install
the latest 0.63.x version but also accept a distro 1.1.x version.
But it is worse for avocado, for two reasons:
1) we cannot use an "==" constraint to install avocado if the venv
includes a system avocado. The distro will package plugins that have
"==" constraints on the version that is included in the distro, and, using
"pip install avocado==88.1" on a venv that includes system packages will
result in this error:
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
avocado-framework-plugin-varianter-yaml-to-mux 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
avocado-framework-plugin-result-html 98.0 requires avocado-framework==98.0, but you have avocado-framework 88.1 which is incompatible.
make[1]: Leaving directory '/home/berrange/src/virt/qemu/build'
2) we cannot use ">=" either if the venv does _not_ include a system
avocado, because that would result in the installation of v101.0 which
is the one we've just reverted.
So the idea is to encode the dependencies as an (acceptable, locked)
tuple, like this hypothetical TOML that would be committed inside
python/ and used by mkvenv.py:
[meson]
meson = { minimum = "0.63.0", install = "0.63.3", canary = "meson" }
[docs]
# 6.0 drops support for Python 3.7
sphinx = { minimum = "1.6", install = "<6.0", canary = "sphinx-build" }
sphinx_rtd_theme = { minimum = "0.5" }
[avocado]
avocado-framework = { minimum = "88.1", install = "88.1", canary = "avocado" }
Once this is implemented, it would also be possible to install avocado in
pyvenv/ using "mkvenv.py ensure", thus using the distro package on Fedora
and CentOS Stream (the only distros where it's available). But until
this is implemented, keep avocado in a separate venv. There is still the
benefit of using a single python for meson custom_targets and for sphinx.
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
scripts/test-driver.py was used when "make check" was already using meson
introspection data, but it did not execute "meson test". It is dead since
commit 3d2f73ef75 ("build: use "meson test" as the test harness", 2021-12-23).
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
If CONFIG_USER_ONLY is ok generically, so is CONFIG_SOFTMMU,
because they are exactly opposite.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Add a script to generate Coccinelle semantic patch
removing all pointless QOM cast macro uses.
Suggested-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230601093452.38972-2-philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Now we no longer have vcpu controlled trace events we can excise the
code that allows us to query its status.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 20230526165401.574474-9-alex.bennee@linaro.org
Message-Id: <20230524133952.3971948-8-alex.bennee@linaro.org>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
This makes it a little easier for developers to find where things
where being generated.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-id: 20230526165401.574474-5-alex.bennee@linaro.org
Message-Id: <20230524133952.3971948-5-alex.bennee@linaro.org>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
