Commit Graph

210 Commits

Author SHA1 Message Date
Jagannathan Raman
55116968de vfio-user: build library
add the libvfio-user library as a submodule. build it as a meson
subproject.

libvfio-user is distributed with BSD 3-Clause license and
json-c with MIT (Expat) license

Signed-off-by: Elena Ufimtseva <elena.ufimtseva@oracle.com>
Signed-off-by: John G Johnson <john.g.johnson@oracle.com>
Signed-off-by: Jagannathan Raman <jag.raman@oracle.com>
Reviewed-by: Stefan Hajnoczi <stefanha@redhat.com>
Message-id: c2adec87958b081d1dc8775d4aa05c897912f025.1655151679.git.jag.raman@oracle.com

[Changed submodule URL to QEMU's libvfio-user mirror on GitLab. The QEMU
project mirrors its dependencies so that it can provide full source code
even in the event that its dependencies become unavailable. Note that
the mirror repo is manually updated, so please contact me to make newer
libvfio-user commits available. If I become a bottleneck we can set up a
cronjob.

Updated scripts/meson-buildoptions.sh to match the meson_options.txt
change. Failure to do so can result in scripts/meson-buildoptions.sh
being modified by the build system later on and you end up with a dirty
working tree.
--Stefan]

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2022-06-15 16:42:33 +01:00
Alex Bennée
b56d1ee951 .gitlab: use less aggressive nproc on our aarch64/32 runners
Running on all 80 cores of our aarch64 runner does occasionally
trigger a race condition which fails the build. However the CI system
is not the time and place to play with much heisenbugs so turn down
the nproc to "only" use 40 cores in the build.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20220613171258.1905715-8-alex.bennee@linaro.org>
2022-06-14 00:15:06 +01:00
Daniel P. Berrangé
34776d80f3 gitlab: compare CIRRUS_nn vars against 'null' not ""
The GitLab variable comparisons don't have shell like semantics where
an unset variable compares equal to empty string. We need to explicitly
test against 'null' to detect an unset variable.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220608160651.248781-1-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220613171258.1905715-7-alex.bennee@linaro.org>
2022-06-14 00:15:06 +01:00
Thomas Huth
72ec89bfc5 gitlab-ci: Fix the build-cfi-aarch64 and build-cfi-ppc64-s390x jobs
The job definitions recently got a second "variables:" section by
accident and thus are failing now if one tries to run them. Merge
the two sections into one again to fix the issue.

And while we're at it, bump the timeout here (70 minutes are currently
not enough for the aarch64 job). The jobs are marked as manual anyway,
so if the user starts them, they want to see their result for sure and
then it's annoying if the job timeouts too early.

Fixes: e312d1fdbb ("gitlab: convert build/container jobs to .base_job_template")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Acked-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220603124809.70794-1-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220613171258.1905715-4-alex.bennee@linaro.org>
2022-06-14 00:15:06 +01:00
John Snow
1d8cf47e5b tests: run 'device-crash-test' from tests/venv
Remove the sys.path hacking from device-crash-test, and add in a little
user-friendly message for anyone who was used to running this script
directly from the source tree.

Modify the GitLab job recipes to create the tests/venv first, then run
device-crash-test from that venv.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20220526000921.1581503-10-jsnow@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-06-06 09:26:54 +02:00
Daniel P. Berrangé
28357dc525 gitlab: don't run CI jobs in forks by default
To preserve CI shared runner credits we don't want to run
pipelines on every push.

This sets up the config so that pipelines are never created
for contributors by default. To override this the QEMU_CI
variable can be set to a non-zero value. If set to 1, the
pipeline will be created but all jobs will remain manually
started. The contributor can selectively run jobs that they
care about. If set to 2, the pipeline will be created and
all jobs will immediately start.

This behavior can be controlled using push variables

  git push -o ci.variable=QEMU_CI=1

To make this more convenient define an alias

   git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
   git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"

Which lets you run

  git push-ci

to create the pipeline, or

  git push-ci-now

to create and run the pipeline

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-6-berrange@redhat.com>
[AJB: fix typo, replicate alias tips in ci.rst]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-33-alex.bennee@linaro.org>
2022-06-01 18:54:59 +01:00
Daniel P. Berrangé
e312d1fdbb gitlab: convert build/container jobs to .base_job_template
This converts the main build and container jobs to use the
base job rules, defining the following new variables

 - QEMU_JOB_SKIPPED - jobs that are known to be currently
   broken and should not be run. Can still be manually
   launched if desired.

 - QEMU_JOB_AVOCADO - jobs that run the Avocado integration
   test harness.

 - QEMU_JOB_PUBLISH - jobs that publish content after the
   branch is merged upstream

As build-tools-and-docs runs on master we declare the requirement of
building amd64-debian-container optional as it should already exits
once we merge.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-5-berrange@redhat.com>
[AJB: fix upstream typo, mention optional container req]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-32-alex.bennee@linaro.org>
2022-06-01 15:47:43 +01:00
Daniel P. Berrangé
16fee101d9 gitlab: convert static checks to .base_job_template
This folds the static checks into using the base job
template rules, introducing one new variable

 - QEMU_JOB_ONLY_FORKS - a job that should never run
   on an upstream pipeline. The information it reports
   is only applicable to contributors in a pre-submission
   scenario, not time of merge.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-4-berrange@redhat.com>
[AJB: fix typo]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-31-alex.bennee@linaro.org>
2022-06-01 15:47:43 +01:00
Daniel P. Berrangé
00125414ba gitlab: convert Cirrus jobs to .base_job_template
This folds the Cirrus job rules into the base job
template, introducing two new variables

  - QEMU_JOB_CIRRUS - identifies the job as making
    use of Cirrus CI via cirrus-run

  - QEMU_JOB_OPTIONAL - identifies the job as one
    that is not run by default, primarily due to
    resource constraints. It can be manually invoked
    by users if they wish to validate that scenario.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-3-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-30-alex.bennee@linaro.org>
2022-06-01 15:47:43 +01:00
Daniel P. Berrangé
6a0e7ea7b8 gitlab: introduce a common base job template
Currently job rules are spread across the various templates
and jobs, making it hard to understand exactly what runs in
what scenario. This leads to inconsistency in the rules and
increased maint burden.

The intent is that we introduce a common '.base_job_template'
which will have a general purpose 'rules:' block. No other
template or job should define 'rules:', but instead they must
rely on the inherited rules. To allow behaviour to be tweaked,
rules will be influenced by a number of variables with the
naming scheme 'QEMU_JOB_nnnn'.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-2-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-29-alex.bennee@linaro.org>
2022-06-01 15:47:43 +01:00
Alex Bennée
9e19fd7d4a tests/docker: update debian-amd64 with lcitool
The one minor wrinkle we need to account for is the netmap support
still requires building from source. We also include cscope and GNU
global as they are used in one of the builds.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: Luigi Rizzo <rizzo@iet.unipi.it>
Cc: Giuseppe Lettieri <g.lettieri@iet.unipi.it>
Cc: Vincenzo Maffione <v.maffione@gmail.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-12-alex.bennee@linaro.org>
2022-06-01 15:47:23 +01:00
Alex Bennée
4ec740e12d tests/docker: update debian-ppc64el-cross with lcitool
Use lcitool to update debian-ppc64el-cross to a Debian 11 based system.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-11-alex.bennee@linaro.org>
2022-06-01 13:08:09 +01:00
Alex Bennée
1e834d1714 tests/docker: update debian-mips64el-cross with lcitool
Use lcitool to update debian-mips64el-cross to a Debian 11 based system.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-10-alex.bennee@linaro.org>
2022-06-01 13:08:09 +01:00
Alex Bennée
432ae739c6 tests/docker: update debian-mipsel-cross with lcitool
Use lcitool to update debian-mipsel-cross to a Debian 11 based system.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-9-alex.bennee@linaro.org>
2022-06-01 13:08:09 +01:00
Alex Bennée
9480624187 tests/docker: update debian-armel-cross with lcitool
Use lcitool to update debian-armel-cross to a Debian 11 based system.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-8-alex.bennee@linaro.org>
2022-06-01 13:08:09 +01:00
Alex Bennée
18ad049d90 tests/docker: update debian-armhf-cross with lcitool
Use lcitool to update debian-armhf-cross to a Debian 11 based system.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220527153603.887929-7-alex.bennee@linaro.org>
2022-06-01 13:08:09 +01:00
Marc-André Lureau
861dca3707 gitlab-ci: add meson JUnit test result into report
This allows the gitlab UI to show the test results in different ways,
see doc:

  https://docs.gitlab.com/ee/ci/unit_test_reports.html#how-it-works

Previous we only reports avocado test results (.avocado_test_job_template),
with this change, the qemu/meson tests are also covered.

Signed-off-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20220525173411.612224-1-marcandre.lureau@redhat.com>
[AJB: expand the commit description]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-4-alex.bennee@linaro.org>
2022-06-01 13:06:32 +01:00
Thomas Huth
6fd4e75610 .gitlab-ci.d/crossbuilds: Fix the dependency of the cross-i386-tci job
The cross-i386-tci job uses the fedora-i386-cross image, so we should make sure
that the corresponding job that builds it (the i386-fedora-cross-container job)
has finished before we start the TCI job.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220524092600.89997-1-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220527153603.887929-3-alex.bennee@linaro.org>
2022-06-01 10:21:49 +01:00
Thomas Huth
658f2da1f3 .gitlab-ci.d/container-cross: Fix RISC-V container dependencies / stages
The "riscv64-debian-cross-container" job does not depend on any other
container job from the first stage, so we can move it to the first
stage, too.

The "riscv64-debian-test-cross-container" job needs the debian11
container, so we should add a proper "needs:" statement here.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220524093141.91012-1-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Message-Id: <20220527153603.887929-2-alex.bennee@linaro.org>
2022-06-01 10:21:49 +01:00
Thomas Huth
83602083b4 capstone: Remove the capstone submodule
Now that we allow compiling with Capstone v3.0.5 again, all our supported
build hosts should provide at least this version of the disassembler
library, so we do not need to ship this as a submodule anymore.

Message-Id: <20220516145823.148450-4-thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2022-05-18 08:54:22 +02:00
Thomas Huth
43ecd16678 capstone: Allow version 3.0.5 again
According to

 https://lore.kernel.org/qemu-devel/20200921174118.39352-1-richard.henderson@linaro.org/

there was an issue with Capstone 3.0.4 from Ubuntu 18, which was the reason
for bumping our minimum Capstone requirement to version 4.0. And indeed,
compiling with that version 3.0.4 from Ubuntu 18.04 still fails (after
allowing it with a hack in meson.build). But now that we've dropped support
for Ubuntu 18.04, that issue is not relevant anymore. Compiling with Capstone
version 3.0.5 (e.g. used in Ubuntu 20.04) seems to work fine, so let's allow
that version again.

Message-Id: <20220516145823.148450-3-thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2022-05-18 08:54:22 +02:00
Thomas Huth
f3f0cbaf97 Remove Ubuntu 18.04 container support from the repository
According to our "Supported build platforms" policy, we now do not support
Ubuntu 18.04 anymore. Remove the related container files and entries from
our CI.

Message-Id: <20220516115912.120951-1-thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2022-05-18 08:54:22 +02:00
Thomas Huth
e4e68c31db gitlab-ci: Switch the container of the 'check-patch' & 'check-dco' jobs
The 'check-patch' and 'check-dco' jobs only need Python and git for
checking the patches, so it's not really necessary to use a container
here that has all the other build dependencies installed. By using a
lightweight Alpine container, we can improve the runtime here quite a
bit, cutting it down from ca. 1:30 minutes to ca. 45 seconds.

Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220516082310.33876-1-thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2022-05-18 08:54:16 +02:00
Daniel P. Berrangé
29a493765e .gitlab-ci.d: export meson testlog.txt as an artifact
When running 'make check' we only get a summary of progress on the
console. Fortunately meson/ninja have saved the raw test output to a
logfile. Exposing this log will make it easier to debug failures that
happen in CI.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220509124134.867431-3-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Kevin Wolf <kwolf@redhat.com>
2022-05-12 12:27:32 +02:00
Yonggang Luo
5c570ef2f1 gitlab-ci: Upgrade mingw base package.
Signed-off-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Message-Id: <20220503200524.1868-3-luoyonggang@gmail.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2022-05-09 08:21:14 +02:00
Thomas Huth
b0dd0a3d74 tests: Drop perl-Test-Harness from the CI containers / VMs
The perl test harness is not necessary anymore since commit 3d2f73ef75
("build: use "meson test" as the test harness"). Thus remove it from
tests/lcitool/projects/qemu.yml, run "make lcitool-refresh" and manually
clean the remaining docker / vm files that are not managed by lcitool yet.

Message-Id: <20220329102808.423681-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2022-04-20 08:54:16 +02:00
Alex Bennée
efd080de76 gitlab: include new aarch32 job in custom-runners
Without linking it in it won't be presented on the UI. Also while
doing that fix the misnamed job from 20.40 to 20.04.

Fixes: cc44a16002 ("gitlab: add a new aarch32 custom runner definition")
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20220315121954.2283887-1-alex.bennee@linaro.org>
2022-03-23 10:38:09 +00:00
Alex Bennée
309df6acb2 gitlab: disable accelerated zlib for s390x
There appears to be a bug in the s390 hardware-accelerated version of
zlib distributed with Ubuntu 20.04, which makes our test
/i386/migration/multifd/tcp/zlib hit an assertion perhaps one time in
10. Fortunately zlib provides an escape hatch where we can disable the
hardware-acceleration entirely by setting the environment variable
DFLTCC to 0. Do this on all our CI which runs on s390 hosts, both our
custom gitlab runner and also the Travis hosts.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Cornelia Huck <cohuck@redhat.com>
Message-id: 20220321161151.3654386-1-alex.bennee@linaro.org
Cc: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2022-03-22 17:07:30 +00:00
Peter Maydell
dee3a86d54 * whpx fixes in preparation for GDB support (Ivan)
* VSS header fixes (Marc-André)
 * 5-level EPT support (Vitaly)
 * AMX support (Jing Liu & Yang Zhong)
 * Bundle changes to MSI routes (Longpeng)
 * More precise emulation of #SS (Gareth)
 * Disable ASAN testing
 -----BEGIN PGP SIGNATURE-----
 
 iQFIBAABCAAyFiEE8TM4V0tmI4mGbHaCv/vSX3jHroMFAmIwb5QUHHBib256aW5p
 QHJlZGhhdC5jb20ACgkQv/vSX3jHroOOUQf8DiNcq8XVVMdX946Qwa4pSxc4ZJtF
 X+RkNsscluuLJ2vGEFKwPVps6c6UPqAhXUruZOQmcLmma511MsyJrxyfd4iRgPD2
 tL1+n4RpfsbnTEGT8c6TFWWMEIOjLTbKmR/SIxuxpeVG3xlk6tlCevykrIdc90gP
 vQIByTGFx3GwiPyDo0j92mA/CsWLnfq6zQ2Tox1xCyt8R+QDimqG0KGLc5RAyiyC
 ZmilN2yaqizDfkIzinwHG6gP1NGwVUsrUNl4X9C4mwEMFnsXiyKP5n/BlDZ7w4Wb
 QXalFpPg1hJxRGGvyta6OF9VmCfmK9Q0FNVWm1lPE5adn3ECHFo6FJKvfg==
 =LVgf
 -----END PGP SIGNATURE-----

Merge tag 'for-upstream' of https://gitlab.com/bonzini/qemu into staging

* whpx fixes in preparation for GDB support (Ivan)
* VSS header fixes (Marc-André)
* 5-level EPT support (Vitaly)
* AMX support (Jing Liu & Yang Zhong)
* Bundle changes to MSI routes (Longpeng)
* More precise emulation of #SS (Gareth)
* Disable ASAN testing

# gpg: Signature made Tue 15 Mar 2022 10:51:00 GMT
# gpg:                using RSA key F13338574B662389866C7682BFFBD25F78C7AE83
# gpg:                issuer "pbonzini@redhat.com"
# gpg: Good signature from "Paolo Bonzini <bonzini@gnu.org>" [full]
# gpg:                 aka "Paolo Bonzini <pbonzini@redhat.com>" [full]
# Primary key fingerprint: 46F5 9FBD 57D6 12E7 BFD4  E2F7 7E15 100C CD36 69B1
#      Subkey fingerprint: F133 3857 4B66 2389 866C  7682 BFFB D25F 78C7 AE83

* tag 'for-upstream' of https://gitlab.com/bonzini/qemu: (22 commits)
  gitlab-ci: do not run tests with address sanitizer
  KVM: SVM: always set MSR_AMD64_TSC_RATIO to default value
  i386: Add Icelake-Server-v6 CPU model with 5-level EPT support
  x86: Support XFD and AMX xsave data migration
  x86: add support for KVM_CAP_XSAVE2 and AMX state migration
  x86: Add AMX CPUIDs enumeration
  x86: Add XFD faulting bit for state components
  x86: Grant AMX permission for guest
  x86: Add AMX XTILECFG and XTILEDATA components
  x86: Fix the 64-byte boundary enumeration for extended state
  linux-headers: include missing changes from 5.17
  target/i386: Throw a #SS when loading a non-canonical IST
  target/i386: only include bits in pg_mode if they are not ignored
  kvm/msi: do explicit commit when adding msi routes
  kvm-irqchip: introduce new API to support route change
  update meson-buildoptions.sh
  qga/vss: update informative message about MinGW
  qga/vss-win32: check old VSS SDK headers
  meson: fix generic location of vss headers
  vmxcap: Add 5-level EPT bit
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2022-03-15 14:41:16 +00:00
Paolo Bonzini
8a5606f644 gitlab-ci: do not run tests with address sanitizer
This is flaky and sometimes fails or hangs unexplicably.

Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2022-03-15 11:50:50 +01:00
Gerd Hoffmann
a46eff0a35 edk2/docker: use ubuntu 18.04
Upstream CI uses ubuntu 18.04 too, so pick
that version (instead of something newer).

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2022-03-15 09:32:22 +01:00
Gerd Hoffmann
392c8ac19b edk2/docker: install python3
python2 is not supported any more,
so go install python3 instead.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
2022-03-15 09:32:22 +01:00
Alex Bennée
7dc17ff794 gitlab: upgrade the job definition for s390x to 20.04
The new s390x machine has more of everything including the OS. As
18.04 will soon be going we might as well get onto something moderately
modern.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Christian Borntraeger <borntraeger@de.ibm.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Acked-by: Cornelia Huck <cohuck@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Cc: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220225172021.3493923-16-alex.bennee@linaro.org>
2022-02-28 16:42:35 +00:00
Alex Bennée
cc44a16002 gitlab: add a new aarch32 custom runner definition
Although running on aarch64 hardware we can still target 32bit builds
with a cross compiler and run the resulting binaries.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220225172021.3493923-10-alex.bennee@linaro.org>
2022-02-28 16:42:12 +00:00
Alex Bennée
7d5817a426 tests/docker: introduce debian-riscv64-test-cross
Cross building QEMU for riscv64 still involves messing about with sid
and ports. However for building tests we can have a slimmer compiler
only container which should be more stable.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220225172021.3493923-7-alex.bennee@linaro.org>
2022-02-28 16:42:02 +00:00
Alex Bennée
89767579ca tests/docker: update debian-s390x-cross with lcitool
A later compiler is needed for some upcomming tests so we might as
well migrate to an lcitool generated docker file.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Cc: David Hildenbrand <david@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220225172021.3493923-6-alex.bennee@linaro.org>
2022-02-28 16:42:02 +00:00
Alex Bennée
e3000245a0 tests/docker: update debian-arm64-cross with lcitool
Using lcitool update debian-arm64-cross to a Debian 11 based system.
As a result we can drop debian-arm64-test-cross just for building
tests.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20220225172021.3493923-5-alex.bennee@linaro.org>
2022-02-28 16:42:02 +00:00
Thomas Huth
74154d7e4a linux-user: Remove the deprecated ppc64abi32 target
It's likely broken, and nobody cared for picking it up again
during the deprecation phase, so let's remove this now.

Since this is the last entry in deprecated_targets_list, remove
the related code in the configure script, too.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Cédric Le Goater <clg@kaod.org>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20211215084958.185214-1-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220112112722.3641051-32-alex.bennee@linaro.org>
2022-02-09 13:29:38 +00:00
Stefan Hajnoczi
6340af7a94 gitlab: fall back to commit hash in qemu-setup filename
Personal repos may not have release tags (v6.0.0, v6.1.0, etc) and this
causes cross_system_build_job to fail when pretty-printing a unique
qemu-setup-*.exe name:

  version="$(git describe --match v[0-9]*)";
             ^^^^^^^^^^ fails ^^^^^^^^^^^
  mv -v qemu-setup*.exe qemu-setup-${version}.exe;

Fall back to the short commit hash if necessary. This fixes CI failures
that Greg Kurz and I experienced in our personal repos.

Cc: Greg Kurz <groug@kaod.org>
Cc: Gerd Hoffmann <kraxel@redhat.com>
Cc: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Greg Kurz <groug@kaod.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20220125173454.10381-1-stefanha@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220204204335.1689602-14-alex.bennee@linaro.org>
2022-02-09 12:08:42 +00:00
Philippe Mathieu-Daudé
d2b7bb099f tests: Manually remove libxml2 on MSYS2 runners
lcitool doesn't support MSYS2 targets, so manually remove
this now unnecessary library.

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220121154134.315047-7-f4bug@amsat.org>
Message-Id: <20220204204335.1689602-11-alex.bennee@linaro.org>
2022-02-09 12:08:42 +00:00
Philippe Mathieu-Daudé
dabee8381a tests/lcitool: Refresh submodule and remove libxml2
The previous commit removed all uses of libxml2.

Refresh lcitool submodule, update qemu.yml and refresh the generated
files by running:

  $ make lcitool-refresh

Note: This refreshment also removes libudev dependency on Fedora
and CentOS due to libvirt-ci commit 18bfaee ("mappings: Improve
mapping for libudev"), since "The udev project has been absorbed
by the systemd project", and lttng-ust on FreeBSD runners due to
libvirt-ci commit 6dd9b6f ("guests: drop lttng-ust from FreeBSD
platform").

Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220121154134.315047-6-f4bug@amsat.org>
Message-Id: <20220204204335.1689602-10-alex.bennee@linaro.org>
2022-02-09 12:08:42 +00:00
Anup Patel
4211fc5532 roms/opensbi: Remove ELF images
Now that all RISC-V machines can use OpenSBI BIN images, we remove
OpenSBI ELF images and also exclude these images from BIOS build.

Signed-off-by: Anup Patel <apatel@ventanamicro.com>
Reviewed-by: Alistair Francis <alistair.francis@wdc.com>
Reviewed-by: Bin Meng <bmeng.cn@gmail.com>
Signed-off-by: Alistair Francis <alistair.francis@wdc.com>
2022-01-21 15:52:57 +10:00
Brad Smith
e850897e67 FreeBSD: Upgrade to 12.3 release
Note, since libtasn1 was fixed in 12.3 [*], this commit re-enables GnuTLS.

[*] https://gitlab.com/gnutls/libtasn1/-/merge_requests/71

Signed-off-by: Brad Smith <brad@comstyle.com>
Tested-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Warner Losh <imp@bsdimp.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <YdUCQLVe5JSWZByQ@humpty.home.comstyle.com>
Message-Id: <20220105135009.1584676-31-alex.bennee@linaro.org>
2022-01-18 16:44:16 +00:00
Daniel P. Berrangé
c45a540f4b .gitlab-ci.d/cirrus: auto-generate variables with lcitool
The current Cirrus CI variables files were previously generated by using
lcitool. This change wires them up to the refresh script to make that
link explicit.

This changes the package list because libvirt-ci now knows about the
mapping for dtc on FreeBSD and macOS platforms.

The variables are also now emit in sorted order for stability across
runs.

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20211215141949.3512719-15-berrange@redhat.com>
Message-Id: <20220105135009.1584676-15-alex.bennee@linaro.org>
2022-01-18 16:42:42 +00:00
Daniel P. Berrangé
6815e63f31 tests/docker: remove ubuntu.docker container
This duplicates the ubuntu2004 container but with an inconsistent set of
packages.

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20211215141949.3512719-14-berrange@redhat.com>
Message-Id: <20220105135009.1584676-14-alex.bennee@linaro.org>
2022-01-18 16:42:42 +00:00
Daniel P. Berrangé
f178c62afc ci: explicitly skip I/O tests on alpine
The block I/O tests don't work on Alpine because their alternative libc
impl emits different strings for errnos, which breaks the expected
output matching. e.g.

=== IO: pattern 102
 wrote 512/512 bytes at offset 512
 512 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-qemu-img: Error while reading offset 0 of blkdebug:TEST_DIR/blkdebug.conf:TEST_DIR/t.IMGFMT: Input/output error
+qemu-img: Error while reading offset 0 of blkdebug:TEST_DIR/blkdebug.conf:TEST_DIR/t.IMGFMT: I/O error
 4
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1073741824
 Formatting 'TEST_DIR/t.IMGFMT.2', fmt=IMGFMT size=0

Currently the I/O tests are skipped as a side effect of the Alpine image
containing a minimal busybox 'sed' binary, rather than GNU sed. This is
a fragile assumption that will be invalidated when the dockerfile is
changed to be autogenerated from a standardized package list that
includes GNU sed.

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20211215141949.3512719-6-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220105135009.1584676-6-alex.bennee@linaro.org>
2022-01-18 16:42:41 +00:00
Thomas Huth
4cc75ce697 gitlab-ci: Enable docs in the centos job
We just ran into a problem that the docs don't build on RHEL8 / CentOS 8
anymore. Seems like these distros are using one of the oldest Sphinx
versions that we still have to support. Thus enable the docs build in
the CI on CentOS so that such bugs don't slip in so easily again.

Message-Id: <20220104091240.160867-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Marc-André Lureau <marcandre.lureau@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2022-01-05 11:10:13 +01:00
Richard Henderson
b5a3d8bc91 Fix some meson conversion breakage
Disable check-python-tox
 Fix emulation of hppa STBY insn
 -----BEGIN PGP SIGNATURE-----
 
 iQFRBAABCgA7FiEEekgeeIaLTbaoWgXAZN846K9+IV8FAmHTMwQdHHJpY2hhcmQu
 aGVuZGVyc29uQGxpbmFyby5vcmcACgkQZN846K9+IV9Rhgf/fFVZTVKscFhrnjzH
 T7PjP3c6TeVpFU8uIb6h1NC0Ugyo4rBds2Gg/qIE+/X8jvDIHa6/aW1HKacWq6Uq
 3Qea+FU56Oc8BpH1CQZEMR/U2ChDofvtcXf/PGHklnpgYCqZctkpNAFPdWlpT9Fx
 jy2Y29rGijV1Q2GEXqD8H6Ij6SX+QZBO5DYaiqGsLSWMS8W2a58rfahBBgslvL82
 UkpJ7TMzvERN8XPBlFOzvqzgSUX4CObAxn2/EiFT4rXukRbVuf5PbDtl2nNRtcS4
 Rfu59GsrHz0tgl3JTRZ/5ab7Na/w8ewknCUPtNygf0k8kKZn8IhDDMhT0ekepoZ9
 dsynQg==
 =WrNf
 -----END PGP SIGNATURE-----

Merge tag 'pull-misc-20220103' of https://gitlab.com/rth7680/qemu into staging

Fix some meson conversion breakage
Disable check-python-tox
Fix emulation of hppa STBY insn

# gpg: Signature made Mon 03 Jan 2022 09:31:48 AM PST
# gpg:                using RSA key 7A481E78868B4DB6A85A05C064DF38E8AF7E215F
# gpg:                issuer "richard.henderson@linaro.org"
# gpg: Good signature from "Richard Henderson <richard.henderson@linaro.org>" [ultimate]

* tag 'pull-misc-20220103' of https://gitlab.com/rth7680/qemu:
  gitlab: Disable check-python-tox
  target/hppa: Fix atomic_store_3 for STBY
  tests/tcg: Unconditionally use 90 second timeout
  tests/tcg: Use $cpu in configure.sh
  meson: Unify mips and mips64 in host_arch

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2022-01-03 09:34:41 -08:00
Richard Henderson
5c23f0c319 gitlab: Disable check-python-tox
Set this test to be manually run, until failures can be fixed.

Suggested-by: John Snow <jsnow@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2022-01-03 08:55:55 -08:00
Philippe Mathieu-Daudé
90f285fd83 MAINTAINERS: Change philmd's email address
The philmd@redhat.com email address will stop working on
2022-01-01, change it to my personal email address.

Update .mailmap in case anyone wants to send me an email
because of some past commit I authored.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20211231000759.707519-1-philmd@redhat.com>
2021-12-31 13:42:54 +01:00
Thomas Huth
9f8e6cad65 gitlab-ci: Speed up the msys2-64bit job by using --without-default-devices
The new msys2-64bit job is often running for more than 50 minutes - and
if the CI is currently loaded, it times out after 60 minutes. The job
has been declared with a bigger timeout, but seems like this is getting
ignored on the shared Gitlab-CI Windows runners, so we're currently
seeing a lot of failures with this job. Thus we have to reduce the time
it takes to finish this job. Since we want to test compiling the WHPX
and HAX accelerator code with this job, switching to another target CPU
is not really a good option, so let's reduce the amount of code that we
have to compile with the --without-default-devices switch instead.

Message-Id: <20211216082253.43899-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-12-17 09:12:37 +01:00
Thomas Huth
7876cba8fc gitlab-ci: Test compilation on Windows with MSYS2
Gitlab also provides runners with Windows, we can use them to
test compilation with MSYS2, in both, 64-bit and 32-bit.

However, it takes quite a long time to set up the VM, so to stay
in a reasonable time frame, we can only compile and check one
target here.

Message-Id: <20211115140623.104116-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-12-15 08:08:59 +01:00
Thomas Huth
f11b0a4892 gitlab-ci: Add cirrus-ci based tests for NetBSD and OpenBSD
Cirrus-CI provides KVM in their Linux containers, so we can also run
our VM-based NetBSD and OpenBSD build jobs there.
Since the VM installation might take a while, we only run the "help"
target on the first invocation to avoid timeouts, and then only check
the build during the next run, once the base image has been cached.
For the the build tests, we also only use very a limited set of target
CPUs since compiling in these VMs is not very fast (especially the
build on OpenBSD seems to be incredibly slow).

The jobs are marked as "manual" only, since this double-indirect setup
(with the cirrus-run script and VMs in the Cirrus-CI containers) might
fail more often than the other jobs, and since we can trigger a limited
amount of Cirrus-CI jobs at a time anyway (due to the restrictions in
the free tier of Cirrus). Thus these jobs are rather added as convenience
for contributors who would like to run the NetBSD/OpenBSD tests without
the need of downloading and installing the corresponding VM images on
their local machines.

Message-Id: <20211209103124.121942-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-12-15 08:08:59 +01:00
Thomas Huth
f462be4c06 gitlab-ci.d/buildtest: Add jobs that run the device-crash-test
The device-crash-test script has been quite neglected in the past,
so that it bit-rot quite often. Let's add CI jobs that run this
script for at least some targets, so that this script does not
regress that easily anymore.

Message-Id: <20211126162724.1162049-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-12-15 08:08:59 +01:00
Thomas Huth
d06f3bf922 gitlab-ci/cirrus: Increase timeout to 80 minutes
The jobs on Cirrus-CI sometimes get delayed quite a bit, waiting to
be scheduled, so while the build test itself finishes within 60 minutes,
the total run time of the jobs can be longer due to this waiting time.
Thus let's increase the timeout on the gitlab side a little bit, so
that these jobs are not marked as failing just because of the delay.

Message-Id: <20211116163309.246602-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-11-17 10:20:38 +01:00
Daniel P. Berrangé
9968de0a4a gitlab: skip cirrus jobs on master and stable branches
On the primary QEMU repository we want the CI jobs to run on the staging
branch as a gating CI test.

Cirrus CI has very limited job concurrency, so if there are too many
jobs triggered they'll queue up and hit the GitLab CI job timeout before
they complete on Cirrus.

If we let Cirrus jobs run again on the master branch immediately after
merging from staging, that just increases the chances jobs will get
queued and subsequently timeout.

The same applies for merges to the stable branches.

User forks meanwhile should be allowed to run Cirrus CI jobs freely.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20211116112757.1909176-1-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
2021-11-16 16:19:53 +00:00
Philippe Mathieu-Daudé
60bec83e8a gitlab-ci: Split custom-runners.yml in one file per runner
To ease maintenance, add the custom-runners/ directory and
split custom-runners.yml in 3 files, all included by the
current custom-runners.yml:
 - ubuntu-18.04-s390x.yml
 - ubuntu-20.04-aarch64.yml
 - centos-stream-8-x86_64.yml

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20211115095608.2436223-1-philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20211115142915.3797652-7-alex.bennee@linaro.org>
2021-11-16 16:19:53 +00:00
Cleber Rosa
d7c2e2b3f4 Jobs based on custom runners: add CentOS Stream 8
This introduces three different parts of a job designed to run
on a custom runner managed by Red Hat.  The goals include:

  a) propose a model for other organizations that want to onboard
     their own runners, with their specific platforms, build
     configuration and tests.

  b) bring awareness to the differences between upstream QEMU and the
     version available under CentOS Stream, which is "A preview of
     upcoming Red Hat Enterprise Linux minor and major releases".

  c) because of b), it should be easier to identify and reduce the gap
     between Red Hat's downstream and upstream QEMU.

The components of this custom job are:

  I) OS build environment setup code:

     - additions to the existing "build-environment.yml" playbook
       that can be used to set up CentOS/EL 8 systems.

     - a CentOS Stream 8 specific "build-environment.yml" playbook
       that adds to the generic one.

 II) QEMU build configuration: a script that will produce binaries with
     features as similar as possible to the ones built and packaged on
     CentOS stream 8.

III) Scripts that define the minimum amount of testing that the
     binaries built with the given configuration (point II) under the
     given OS build environment (point I) should be subjected to.

 IV) Job definition: GitLab CI jobs that will dispatch the build/test
     jobs (see points #II and #III) to the machine specifically
     configured according to #I.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Tested-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20211111160501.862396-2-crosa@redhat.com>
Message-Id: <20211115142915.3797652-6-alex.bennee@linaro.org>
2021-11-16 16:19:53 +00:00
Willian Rampazzo
bbbd9b6ec6 tests/acceptance: rename tests acceptance to tests avocado
In the discussion about renaming the `tests/acceptance` [1], the
conclusion was that the folders inside `tests` are related to the
framework running the tests and not directly related to the type of
the tests.

This changes the folder to `tests/avocado` and adjusts the MAKEFILE, the
CI related files and the documentation.

[1] https://lists.gnu.org/archive/html/qemu-devel/2021-05/msg06553.html

Reviewed-by: Niek Linnenbank <nieklinnenbank@gmail.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20211105155354.154864-3-willianr@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
2021-11-08 17:00:22 +01:00
Gerd Hoffmann
f5918a9928 microvm: add device tree support.
Allows edk2 detect virtio-mmio devices and pcie ecam.
See comment in hw/i386/microvm-dt.c for more details.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Sergio Lopez <slp@redhat.com>
Message-Id: <20211014193617.2475578-1-kraxel@redhat.com>
2021-11-02 17:24:17 +01:00
Philippe Mathieu-Daudé
bd0da3a3d4 roms/edk2: Only initialize required submodules
The EDK2 firmware images built to test QEMU do not require
the following submodules:

 - MdeModulePkg/Universal/RegularExpressionDxe/oniguruma
 - UnitTestFrameworkPkg/Library/CmockaLib/cmocka

The only submodules required are:

 - ArmPkg/Library/ArmSoftFloatLib/berkeley-softfloat-3
 - BaseTools/Source/C/BrotliCompress/brotli
 - CryptoPkg/Library/OpensslLib/openssl
 - MdeModulePkg/Library/BrotliCustomDecompressLib/brotli

Adapt the buildsys machinery to only initialize the required
submodules.

Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Gerd Hoffmann <kraxel@redhat.com>
Message-Id: <20211018105816.2663195-3-philmd@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2021-10-20 16:26:19 -07:00
Daniel P. Berrangé
f13abca0a3 gitlab: fix passing of TEST_TARGETS env to cirrus
A typo meant the substitution would not work, and the placeholder in the
target file didn't even exist.

The result was that tests were never run on the FreeBSD and macOS jobs,
only a basic build.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20210915125452.1704899-3-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210917162332.3511179-10-alex.bennee@linaro.org>
2021-10-12 08:37:05 +01:00
Daniel P. Berrangé
dcbad7a6ed gitlab: skip the check-patch job on the upstream repo
The check-patch job is intended to be used by contributors or
subsystem maintainers to see if there are style mistakes. The
false positive rate is too high to be used in a gating scenario
so should not run it on the upstream repo ever.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20210915125452.1704899-2-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210917162332.3511179-9-alex.bennee@linaro.org>
2021-10-12 08:37:05 +01:00
Richard Henderson
9f62025141 gitlab: Add cross-riscv64-system, cross-riscv64-user
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20210914185830.1378771-3-richard.henderson@linaro.org>
[AJB: add allow_failure]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210917162332.3511179-6-alex.bennee@linaro.org>
2021-10-12 08:37:05 +01:00
Alex Bennée
5c24acf320 tests/docker: promote debian-riscv64-cross to a full image
To be able to cross build QEMU itself we need to include a few more
libraries. These are only available in Debian's unstable ports repo
for now so we need to base the riscv64 image on sid with the the
minimal libs needed to build QEMU (glib/pixman).

The result works but is not as clean as using build-dep to bring in
more dependencies. However sid is by definition a shifting pile of
sand and by keeping the list of libs minimal we reduce the chance of
having an image we can't build. It's good enough for a basic cross
build testing of TCG.

Cc: "Daniel P. Berrangé" <berrange@redhat.com>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Message-Id: <20210914185830.1378771-2-richard.henderson@linaro.org>
[AJB: tweak allow_failure]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210917162332.3511179-5-alex.bennee@linaro.org>
2021-10-12 08:37:05 +01:00
Peter Maydell
7b7ab2d6c9 gitlab-ci: Mark manual-only jobs as allow_failure
If a gitlab CI job is marked as manual-only but is not marked
as allow_failure, then gitlab considers that the pipeline is
"blocked" until the job has been manually triggered. We need
to mark these manual-only jobs as also allow_failure: true
so that gitlab doesn't insist that they have run before it
will consider the pipeline to be complete.

Fixes: 4c9af1ea14
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20210915123412.8232-1-peter.maydell@linaro.org
Acked-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
2021-09-15 16:43:16 +01:00
Peter Maydell
4c9af1ea14 gitlab-ci: Make more custom runner jobs manual, and don't allow failure
Currently we define a lot of jobs for our custom runners:
for both aarch64 and s390x we have
 - all-linux-static
 - all
 - alldbg
 - clang (manual)
 - tci
 - notcg (manual)

This is overkill.  The main reason to run on these hosts is to get
coverage for the host architecture; we can leave the handling of
differences like debug vs non-debug to the x86 CI jobs.

The jobs are also generally running OK; they occasionally fail due to
timeouts, which is likely because we're overloading the machine by
asking it to run 4 CI jobs at once plus the ad-hoc CI.

Remove the 'allow_failure' tag from all these jobs, and switch the
s390x-alldbg, aarch64-all, s390x-tci and aarch64-tci jobs to manual.
(We keep -all on s390x and -alldbg on aarch64 just for diversity
of coverage.)

This will let us make the switch for s390x and aarch64 hosts from
the ad-hoc CI to gitlab.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Thomas Huth <thuth@redhat.com>
Message-id: 20210913101948.12600-1-peter.maydell@linaro.org
2021-09-14 17:03:03 +01:00
Thomas Huth
2ffd4d815e gitlab-ci: Don't try to use the system libfdt in the debian job
libfdt in Debian is too old to be usable for QEMU. So far we were
silently falling back to the internal dtc submodule, but since
this is wrong, let's remove the --enable-fdt=system switch here now.

Message-Id: <20210827151718.178988-1-thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Acked-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-09-06 09:07:00 +02:00
Peter Maydell
9c03aa87e5 Testing and plugin updates:
- fix typo in execlog plugin
   - clean-up and document gitlab FOO_RUNNER_AVAILABLE vars
   - fix plugin build issue on OSX and modules
   - add multi-core support to cache modelling plugin
   - clean-ups for plugin arg=FOO handling
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmEwqF4ACgkQ+9DbCVqe
 KkTCcAf/fCHK1hhXJIaxpFyMGvCkhYzUgKlpL4b05ofXFyQJ4JWp+0OadQzBcVIN
 PwsVVsXik/5ibKvQiud8wt3/kbr9gj5RZdwVITZMEpod6hy9Nt5oaUhxEZpiVs3f
 XZG17zaK8huwTwU6EtHWOnvIWMCuvBXpw8cM/6jF6pEiq162VJbiRo9L/aQ+n0Io
 TZTy9YqEbrZUTj+XrvgPV0TewjM6T/zTujXFJtyCzYJ3P3kx9z5a3/KzfyJI9qkk
 TYw0MX9MY8J5/sXcP6OjOqXqppsRe+G5Uaz0BoW9SvOmR/0nNGy/N8mVCZTD6mqs
 zMsWw5RyIZsZCyWEYj4TYWEORv5Kiw==
 =CBs1
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/stsquad/tags/pull-for-6.2-020921-1' into staging

Testing and plugin updates:

  - fix typo in execlog plugin
  - clean-up and document gitlab FOO_RUNNER_AVAILABLE vars
  - fix plugin build issue on OSX and modules
  - add multi-core support to cache modelling plugin
  - clean-ups for plugin arg=FOO handling

# gpg: Signature made Thu 02 Sep 2021 11:33:02 BST
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-for-6.2-020921-1: (22 commits)
  docs/devel: be consistent about example plugin names
  docs/deprecated: deprecate passing plugin args through `arg=`
  tests/plugins/syscalls: adhere to new arg-passing scheme
  tests/plugins/mem: introduce "track" arg and make args not positional
  tests/plugins/insn: made arg inline not positional and parse it as bool
  tests/plugins/bb: adapt to the new arg passing scheme
  docs/tcg-plugins: new passing parameters scheme for cache docs
  plugins/howvec: adapting to the new argument passing scheme
  plugins/hwprofile: adapt to the new plugin arguments scheme
  plugins/lockstep: make socket path not positional & parse bool arg
  plugins/hotblocks: Added correct boolean argument parsing
  plugins/hotpages: introduce sortby arg and parsed bool args correctly
  plugins/api: added a boolean parsing plugin api
  plugins: allow plugin arguments to be passed directly
  docs/devel/tcg-plugins: added cores arg to cache plugin
  plugins: sort exported symbol list
  plugins/cache: supported multicore cache modelling
  plugins: do not limit exported symbols if modules are active
  gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them
  gitlab-ci: Remove superfluous "dnf install" statement
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-09-03 14:23:36 +01:00
Thomas Huth
40e916b4bc gitlab-ci: Fix ..._RUNNER_AVAILABLE variables and document them
The patch that recently introduced the S390X_RUNNER_AVAILABLE variable
in custom-runners.yml missed that the bottom half of the file is rather
about aarch64 than s390x. Thus rename the S390X_RUNNER_AVAILABLE to
AARCH64_RUNNER_AVAILABLE in those jobs.

Finally mention both variables in our CI documentation, too.

Fixes: c5dd0f0342 ("Improve rules for the staging branch")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210730143809.717079-4-thuth@redhat.com>
[AJB: moved due to docu changes]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210806141015.2487502-5-alex.bennee@linaro.org>
2021-09-02 11:29:34 +01:00
Thomas Huth
d3ac658fbe gitlab-ci: Remove superfluous "dnf install" statement
The container already features meson and ninja, so there is no need
to try to install it with dnf again.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210730143809.717079-3-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210806141015.2487502-4-alex.bennee@linaro.org>
2021-09-02 11:29:34 +01:00
Thomas Huth
8d316275ad gitlab-ci: Merge "build-disabled" with "build-without-default-features"
Both jobs are testing more or less the same thing (building QEMU with
features disabled), so we are wasting precious CI cycles here by doing
this twice. Merge the jobs by using --without-default-features by default
and just adding some additional --disable-... switches which are not
covered by the generic switch (yet). And while we're at it, also test
compilation with "--disable-fdt" (which forces us to change the list
of targets in this job, though, since some targets do not work without
fdt).

Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210730143809.717079-2-thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210806141015.2487502-3-alex.bennee@linaro.org>
2021-09-02 11:29:34 +01:00
Vladimir Sementsov-Ogievskiy
c88f078948 iotests: move 222 to tests/image-fleecing
Give a good name to test file.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210824083856.17408-29-vsementsov@virtuozzo.com>
[hreitz: Adjust .gitlab-ci.d/buildtest.yml]
Signed-off-by: Hanna Reitz <hreitz@redhat.com>
2021-09-01 14:37:14 +02:00
Daniel P. Berrangé
a1f0f36838 gitlab: skip many more targets in windows cross builds
The windows cross builds still take way too long in gitlab CI, so need
more targets to be skipped. We don't want to hurt coverage of other
cross builds more though, so we let jobs fine tune with a new env
variale $CROSS_SKIP_TARGETS.

We take the set of targets that are considered relatively niche or
very old architectures, and skip approx half of them in win32 builds
and the other half of them in win64.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210810140653.3969823-3-berrange@redhat.com>
[thuth: Swapped the "CROSS_SKIP_TARGETS:" lines as suggested by philmd]
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-08-11 08:52:31 +02:00
Daniel P. Berrangé
a62354915b gitlab: exclude sparc-softmmu and riscv32-softmmu from cross builds
We need to cut down compile time by excluding more targets. Both these
targets still have their 64-bit variant enabled, so the loss of coverage
is mitigated to some degree.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210810140653.3969823-2-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-08-11 07:52:43 +02:00
Thomas Huth
c5dd0f0342 gitlab-ci.d/custom-runners: Improve rules for the staging branch
If maintainers are currently pushing to a branch called "staging"
in their repository, they are ending up with some stuck jobs - unless
they have a s390x CI runner machine available. That's ugly, we should
make sure that the related jobs are really only started if such a
runner is available. So let's only run these jobs if it's the
"staging" branch of the main repository of the QEMU project (where
we can be sure that the s390x runner is available), or if the user
explicitly set a S390X_RUNNER_AVAILABLE variable in their CI configs
to declare that they have such a runner available, too.

Fixes: 4799c21023 ("Jobs based on custom runners: add job definitions ...")
Message-Id: <20210728173857.497523-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-29 08:04:25 +02:00
Thomas Huth
e37264ebe3 gitlab-ci.d/buildtest: Mark the aarch64 and ppc64-s390x CFI jobs as manual
These two jobs are currently failing very often - the linker seems to
get killed due to out-of-memory problems. Since apparently nobody has
currently an idea how to fix that nicely, let's mark the jobs as manual
for the time being until someone comes up with a proper fix.

Message-Id: <20210728075141.400816-1-thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-29 08:04:25 +02:00
Thomas Huth
db1e119238 gitlab-ci.d/buildtest: Disable iotests 197 and 215
The iotests 197 and 215 are occasionally failing in the gitlab-CI now.
According to the log, the failure is "./common.rc: Killed" which might
be an indication that the process has been killed due to out-of-memory
reasons. Both tests are doing a big read with 2G that likely causes
this issue. It used to work fine in the gitlab-CI in the past, but
either the program is now requiring more free memory, or the the CI
containers have changed, so that the OOM condition now sometimes occurs.

Anyway, these two tests are not really suitable for CI containers if
they are doing things like huge reads (which is likely also the reason
why they haven't been added to the "auto" group in the past), so let's
simply disable them in the gitlab-CI now, too.

Message-Id: <20210727162542.318882-1-thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-29 08:04:25 +02:00
Philippe Mathieu-Daudé
c217fd8e36 gitlab-ci: Fix 'when:' condition in OpenSBI jobs
Jobs depending on another should not use the 'when: always'
condition, because if a dependency failed we should not keep
running jobs depending on it. The correct condition is
'when: on_success'.

Fixes: c6fc0fc1a7 ("gitlab-ci.yml: Add jobs to build OpenSBI firmware binaries")
Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210727142431.1672530-5-philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-29 07:56:01 +02:00
Philippe Mathieu-Daudé
59e8b62b22 gitlab-ci: Fix 'when:' condition in EDK2 jobs
Jobs depending on another should not use the 'when: always'
condition, because if a dependency failed we should not keep
running jobs depending on it. The correct condition is
'when: on_success'.

Fixes: 71920809ce ("gitlab-ci.yml: Add jobs to build EDK2 firmware binaries")
Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20210727142431.1672530-4-philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-29 07:56:01 +02:00
Philippe Mathieu-Daudé
d3a4e41da2 gitlab-ci: Fix 'when:' condition in acceptance_test_job_template
Jobs depending on another should not use the 'when: always'
condition, because if a dependency failed we should not keep
running jobs depending on it. The correct condition is
'when: on_success'.

Fixes: f56bf4caf7 ("gitlab: Run Avocado tests manually (except mainstream CI)")
Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20210727142431.1672530-3-philmd@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-29 07:56:01 +02:00
Peter Maydell
34fd92ab41 Doc, metadata, plugin and testing updates for 6.1-rc1:
- git ignore some file editor detritus
   - add overview on device emulation terminology
   - remove needless if leg in configure custom devices logic
   - numerous gitdm/mailmap updates
   - fix plugin_exit race for linux-user
   - fix a few bugs in cache modelling plugin
   - fix plugin calculation of physical address
   - handle pure assembler/linker tcg tests outside of docker
   - add tricore build to gitlab
   - remove superfluous MacOSX task
   - generalise the OpenBSI gitlab rules
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmD67ioACgkQ+9DbCVqe
 KkTKcwf/UlEl2wlYrvymoJy7caasD/UrlNwpaK90zRTxuofJjG2bXX/58EGBlgxe
 cjbX3841702HXAFDxzswmWX3yeXQVgNiXrEy5FapB7L3h/qC/BDVS4pz987TITNW
 b4lw/d05bQVGavcdJXPnX0CdV8IK4hpieDkdh/LGj5+CSAC39jvyVGTYBpfLKl8D
 cps5qcJcJuQjz4hatB1haZOV1WHLyw8KI0srfcEqkh3IImZFZ09frs3qqAxrYRqE
 Y62d7SIN5TX1XgWuTaf1mmaMVmbNi9DTdCxWpGYr4ZBLhhzKeljQvKbF45ZGQ0z9
 eBVBdxGJTX/WAKrimcbHxLo43pCo2Q==
 =7vzj
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/stsquad/tags/pull-for-6.1-rc1-230721-1' into staging

Doc, metadata, plugin and testing updates for 6.1-rc1:

  - git ignore some file editor detritus
  - add overview on device emulation terminology
  - remove needless if leg in configure custom devices logic
  - numerous gitdm/mailmap updates
  - fix plugin_exit race for linux-user
  - fix a few bugs in cache modelling plugin
  - fix plugin calculation of physical address
  - handle pure assembler/linker tcg tests outside of docker
  - add tricore build to gitlab
  - remove superfluous MacOSX task
  - generalise the OpenBSI gitlab rules

# gpg: Signature made Fri 23 Jul 2021 17:28:26 BST
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-for-6.1-rc1-230721-1: (28 commits)
  gitlab-ci: Extract OpenSBI job rules to reusable section
  gitlab-ci: Remove the second superfluous macos task
  gitlab: enable a very minimal build with the tricore container
  tests/tcg/configure.sh: add handling for assembler only builds
  plugins: Fix physical address calculation for IO regions
  plugins/cache: Fixed "function decl. is not a prototype" warnings
  plugins/cache: limited the scope of a mutex lock
  plugins/cache: Fixed a bug with destroying FIFO metadata
  tcg/plugins: implement a qemu_plugin_user_exit helper
  contrib/gitdm: add more individual contributor entries.
  contrib/gitdm: add a new interns group-map for GSoC/Outreachy work
  contrib/gitdm: add an explicit academic entry for BU
  contrib/gitdm: add group-map for Netflix
  contrib/gitdm: add domain-map for NVIDIA
  contrib/gitdm: add domain-map for Crudebyte
  contrib/gitdm: un-ironically add a mapping for LWN
  contrib/gitdm: add domain-map/group-map for Wind River
  contrib/gitdm: add domain-map for Eldorado
  contrib/gitdm: add domain-map/group-map mappings for Samsung
  gitdm.config: sort the corporate GroupMap entries
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-07-26 11:00:15 +01:00
Philippe Mathieu-Daudé
0a9487d80a gitlab-ci: Extract OpenSBI job rules to reusable section
All jobs depending on 'docker-opensbi' job must use at most all
the rules that triggers it. The simplest way to ensure that
is to always use the same rules. Extract all the rules to a
reusable section, and include this section (with the 'extends'
keyword) in both 'docker-opensbi' and 'build-opensbi' jobs.

The problem was introduced in commit c6fc0fc1a7 ("gitlab-ci.yml:
Add jobs to build OpenSBI firmware binaries"), but was revealed in
commit 91e9c47e50 ("docker: OpenSBI build job depends on OpenSBI
container").

This fix is similar to the one used with the EDK2 firmware job in
commit ac0595cf6b ("gitlab-ci: Extract EDK2 job rules to reusable
section").

Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210720164829.3949558-1-philmd@redhat.com>
Message-Id: <20210720232703.10650-30-alex.bennee@linaro.org>
2021-07-23 17:22:16 +01:00
Thomas Huth
e90c3c3c87 gitlab-ci: Remove the second superfluous macos task
While there might have been bigger differnces between the -base and
the -xcode images in the beginning, they almost vanished in the
current builds, e.g. when comparing the output of the "configure"
step after cleaning up the differences due to temporary path names,
I only get:

  $ diff -u /tmp/base.txt /tmp/xcode.txt
  --- /tmp/base.txt	2021-07-16 09:16:24.211427940 +0200
  +++ /tmp/xcode.txt	2021-07-16 09:16:43.029684274 +0200
  @@ -19,14 +19,14 @@
   Build type: native build
   Project name: qemu
   Project version: 6.0.50
  -C compiler for the host machine: cc (clang 12.0.0 "Apple clang version 12.0.0 (clang-1200.0.32.29)")
  +C compiler for the host machine: cc (clang 12.0.0 "Apple clang version 12.0.0 (clang-1200.0.32.28)")
   C linker for the host machine: cc ld64 609.8
   Host machine cpu family: x86_64
   Host machine cpu: x86_64
   Program sh found: YES (/bin/sh)
   Program python3 found: YES (/usr/local/opt/python@3.9/bin/python3.9)
   Program bzip2 found: YES (/usr/bin/bzip2)
  -C++ compiler for the host machine: c++ (clang 12.0.0 "Apple clang version 12.0.0 (clang-1200.0.32.29)")
  +C++ compiler for the host machine: c++ (clang 12.0.0 "Apple clang version 12.0.0 (clang-1200.0.32.28)")
   C++ linker for the host machine: c++ ld64 609.8
   Objective-C compiler for the host machine: clang (clang 12.0.0)
   Objective-C linker for the host machine: clang ld64 609.8

Since we're not using Xcode itself at all, it seems like it does not
make much sense anymore to waste compute cycles with two images here.
Thus let's delete the -xcode job now.

[AJB: fix up commit formatting which trips up b4]

Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210719073051.1559348-1-thuth@redhat.com>
Message-Id: <20210720232703.10650-29-alex.bennee@linaro.org>
2021-07-23 17:22:16 +01:00
Alex Bennée
39ce923732 gitlab: enable a very minimal build with the tricore container
Rather than base of the shared Debian 10 container which would require
us to bring in even more dependencies just bring in what is needed for
building tricore-softmmu in GitLab. We don't even remove the container
from the DOCKER_PARTIAL_IMAGES lest we cause more confusion.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210720232703.10650-28-alex.bennee@linaro.org>
2021-07-23 17:22:16 +01:00
Daniel P. Berrangé
eafadbbbac gitlab: only let pages be published from default branch
GitLab will happily publish pages generated by the latest CI pipeline
from any branch:

https://docs.gitlab.com/ee/user/project/pages/introduction.html

  "Remember that GitLab Pages are by default branch/tag agnostic
   and their deployment relies solely on what you specify in
   .gitlab-ci.yml. You can limit the pages job with the only
   parameter, whenever a new commit is pushed to a branch used
   specifically for your pages."

The current "pages" job is not limited, so it is happily publishing
docs content from any branch/tag in qemu.git that gets pushed to.
This means we're potentially publishing from the "staging" branch
or worse from outdated "stable-NNN" branches

This change restricts it to only publish from the default branch
in the main repository. For contributor forks, however, we allow
it to publish from any branch, since users will have arbitrarily
named topic branches in flight at any time.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210723113051.2792799-1-berrange@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-07-23 16:02:56 +02:00
Gerd Hoffmann
8619b5ddb5 ci: build & store windows installer
Build windows installer for qemu in gitlab CI,
store the result as artifact.

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20210623091137.1156959-2-kraxel@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-07-19 09:33:39 +02:00
Peter Maydell
438951e883 Testing and plugin updates:
- custom runner playbooks for configuring GitLab runners
   - integrate Cirrus jobs into GitLab via cirrus-run
   - clean-up docker package lists
   - bump NetBSD to 9.2
   - bump OpenBSD to 6.9
   - make test-mmap more hexagon friendly
   - fixup handling of hostaddr for plugins
   - disallow some incompatible plugin configurations
   - fix handling of -ldl for BSDs
   - remove some old unused symbols from the plugin symbol map
   - enable plugins by default for most TCG builds
   - honour main build -Wall settings for plugins
   - new execlog plugin
   - new cache modelling plugin
   - fix io_uring build regression
   - disable modular TCG on Darwin
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCgAdFiEEZoWumedRZ7yvyN81+9DbCVqeKkQFAmDu+xsACgkQ+9DbCVqe
 KkQ2XAf+MRBeSMX9J091iloiqSjpuobAcVuoIFciG7gEqWd2iTQVPDqJNgBJUANQ
 vTA7f89ljxj/FZjSZwLgR6cqY6X0ufXI1BLEIt7s78LJtVp14sHVo89GwNBzmRwo
 615T49KG8b5EBBU5YlVcAW/m8DlfgI4b1ufS/qHldOukKegu+haoCDjGG6RNpYNx
 mmXgLOBJiB/p2u2S73KTIa+3AEIIlsTJZpdLPus8hby+/Q4qB9t8YbdHaweyM9qs
 NjiojczvlbHLFd/IQSl6fqBS9QI+KWf4+oFd6ZB4THhr9/GUYTiMeLjRKucBgDQG
 TwGtd1RN3gAPI5aEJ0xlgCL/crz1qg==
 =dQl6
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/stsquad/tags/pull-testing-and-plugins-140721-5' into staging

Testing and plugin updates:

  - custom runner playbooks for configuring GitLab runners
  - integrate Cirrus jobs into GitLab via cirrus-run
  - clean-up docker package lists
  - bump NetBSD to 9.2
  - bump OpenBSD to 6.9
  - make test-mmap more hexagon friendly
  - fixup handling of hostaddr for plugins
  - disallow some incompatible plugin configurations
  - fix handling of -ldl for BSDs
  - remove some old unused symbols from the plugin symbol map
  - enable plugins by default for most TCG builds
  - honour main build -Wall settings for plugins
  - new execlog plugin
  - new cache modelling plugin
  - fix io_uring build regression
  - disable modular TCG on Darwin

# gpg: Signature made Wed 14 Jul 2021 15:56:27 BST
# gpg:                using RSA key 6685AE99E75167BCAFC8DF35FBD0DB095A9E2A44
# gpg: Good signature from "Alex Bennée (Master Work Key) <alex.bennee@linaro.org>" [full]
# Primary key fingerprint: 6685 AE99 E751 67BC AFC8  DF35 FBD0 DB09 5A9E 2A44

* remotes/stsquad/tags/pull-testing-and-plugins-140721-5: (44 commits)
  MAINTAINERS: Added myself as a reviewer for TCG Plugins
  docs/devel: Added cache plugin to the plugins docs
  plugins/cache: Added FIFO and LRU eviction policies
  plugins/cache: Enable cache parameterization
  plugins: Added a new cache modelling plugin
  docs/devel: tcg-plugins: add execlog plugin description
  contrib/plugins: add execlog to log instruction execution and memory access
  contrib/plugins: enable -Wall for building plugins
  tcg/plugins: enable by default for most TCG builds
  configure: stop user enabling plugins on Windows for now
  configure: add an explicit static and plugins check
  configure: don't allow plugins to be enabled for a non-TCG build
  tcg/plugins: remove some stale entries from the symbol list
  meson.build: relax the libdl test to one for the function dlopen
  meson.build: move TCG plugin summary output
  plugins: fix-up handling of internal hostaddr for 32 bit
  tests/tcg: make test-mmap a little less aggressive
  tests/vm: update openbsd to release 6.9
  tests/vm: update NetBSD to 9.2
  tests/docker: expand opensuse-leap package list
  ...

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-07-15 16:06:12 +01:00
Philippe Mathieu-Daudé
ac0595cf6b gitlab-ci: Extract EDK2 job rules to reusable section
All jobs depending on 'docker-edk2' job must use at most all
the rules that triggers it. The simplest way to ensure that
is to always use the same rules. Extract all the rules to a
reusable section, and include this section (with the 'extends'
keyword) in both 'docker-edk2' and 'build-edk2' jobs.

The problem was introduced in commit 71920809ce ("gitlab-ci.yml:
Add jobs to build EDK2 firmware binaries"), but was revealed in
commit 1925468ddb ("docker: EDK2 build job depends on EDK2
container") and eventually failed on CI:
https://gitlab.com/qemu-project/qemu/-/pipelines/335995843

Reported-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210714101003.3113726-1-philmd@redhat.com>
2021-07-14 22:28:17 +02:00
Alex Bennée
ba4dd2aabc tcg/plugins: enable by default for most TCG builds
Aside from a minor bloat to file size the ability to have TCG plugins
has no real impact on performance unless a plugin is actively loaded.
Even then the libempty.so plugin shows only a minor degradation in
performance caused by the extra book keeping the TCG has to do to keep
track of instructions. As it's a useful feature lets just enable it by
default and reduce our testing matrix a little.

We need to move our linker testing earlier so we can be sure we can
enable the loader module required. As we have ruled out static &
plugins in an earlier patch we can also reduce the indent a little.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Cc: Paolo Bonzini <pbonzini@redhat.com>
Message-Id: <20210709143005.1554-33-alex.bennee@linaro.org>
2021-07-14 15:54:13 +01:00
Daniel P. Berrangé
0e103a65ba gitlab: support for FreeBSD 12, 13 and macOS 11 via cirrus-run
This adds support for running 4 jobs via Cirrus CI runners:

 * FreeBSD 12
 * FreeBSD 13
 * macOS 11 with default XCode
 * macOS 11 with latest XCode

The gitlab job uses a container published by the libvirt-ci
project (https://gitlab.com/libvirt/libvirt-ci) that contains
the 'cirrus-run' command. This accepts a short yaml file that
describes a single Cirrus CI job, runs it using the Cirrus CI
REST API, and reports any output to the console.

In this way Cirrus CI is effectively working as an indirect
custom runner for GitLab CI pipelines. The key benefit is that
Cirrus CI job results affect the GitLab CI pipeline result and
so the user only has look at one CI dashboard.

[AJB: remove $TEMPORARILY_DISABLED condition, s/py37/py38/]

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20210625172211.451010-3-berrange@redhat.com>
Message-Id: <20210709143005.1554-8-alex.bennee@linaro.org>
2021-07-14 14:33:36 +01:00
Cleber Rosa
4799c21023 Jobs based on custom runners: add job definitions for QEMU's machines
The QEMU project has two machines (aarch64 and s390x) that can be used
for jobs that do build and run tests.  This introduces those jobs,
which are a mapping of custom scripts used for the same purpose.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210630012619.115262-5-crosa@redhat.com>
Message-Id: <20210709143005.1554-5-alex.bennee@linaro.org>
2021-07-14 14:31:48 +01:00
Cleber Rosa
b38a04f71f Jobs based on custom runners: documentation and configuration placeholder
As described in the included documentation, the "custom runner" jobs
extend the GitLab CI jobs already in place.  One of their primary
goals of catching and preventing regressions on a wider number of host
systems than the ones provided by GitLab's shared runners.

This sets the stage in which other community members can add their own
machine configuration documentation/scripts, and accompanying job
definitions.  As a general rule, those newly added contributed jobs
should run as "non-gating", until their reliability is verified (AKA
"allow_failure: true").

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210630012619.115262-2-crosa@redhat.com>
Message-Id: <20210709143005.1554-2-alex.bennee@linaro.org>
2021-07-14 14:31:48 +01:00
Wainer dos Santos Moschetta
6f651a6d84 python: Configure tox to skip missing interpreters
Currently tox tests against the installed interpreters, however if any
supported interpreter is absent then it will return fail. It seems not
reasonable to expect developers to have all supported interpreters
installed on their systems. Luckily tox can be configured to skip
missing interpreters.

This changed the tox setup so that missing interpreters are skipped by
default. On the CI, however, we still want to enforce it tests
against all supported. This way on CI the
--skip-missing-interpreters=false option is passed to tox.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210630184546.456582-1-wainersm@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Cleber Rosa <crosa@redhat.com>
2021-07-13 15:44:16 -04:00
John Snow
6f84d726f3 python: rename 'venv-check' target to 'check-pipenv'
Well, Cleber was right, this is a better name.

In preparation for adding a different kind of virtual environment check
(One that simply uses whichever version of Python you happen to have),
rename this test 'check-pipenv' so that it matches the CI job
'check-python-pipenv'.

Remove the "If you don't know which test to run" hint, because it's not
actually likely you have Python 3.6 installed to be able to run the
test. It's still the test I'd most prefer you to run, but it's not the
test you are most likely to be able to run.

Rename the 'venv' target to 'pipenv' as well, and move the more
pertinent help text under the 'check-pipenv' target.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-id: 20210629214323.1329806-8-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
2021-06-30 21:54:04 -04:00
Alex Bennée
66cf70149a scripts/checkpatch: roll diff tweaking into checkpatch itself
Rather than relying on external tweaks lets just do it inside
checkpatch's direct commitish handling which is QEMU specific code
anyway.

Suggested-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210623102749.25686-4-alex.bennee@linaro.org>
2021-06-25 10:08:33 +01:00
Philippe Mathieu-Daudé
e2d3017809 gitlab-ci: Split gprof-gcov job
This job is hitting the 70min limit, so split it in 2 tasks.

Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Message-Id: <20210525082556.4011380-7-f4bug@amsat.org>
2021-06-07 14:42:47 +01:00
Alex Bennée
b114a0b943 gitlab: work harder to avoid false positives in checkpatch
This copies the behaviour of patchew's configuration to make the diff
algorithm generate a minimal diff.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210602153247.27651-1-alex.bennee@linaro.org>
2021-06-07 14:42:47 +01:00
Peter Maydell
8c345b3e6a * Update the references to some doc files (use *.rst instead of *.txt)
* Bump minimum versions of some requirements after removing CentOS 7 support
 -----BEGIN PGP SIGNATURE-----
 
 iQJFBAABCAAvFiEEJ7iIR+7gJQEY8+q5LtnXdP5wLbUFAmC3L1IRHHRodXRoQHJl
 ZGhhdC5jb20ACgkQLtnXdP5wLbU8wA//aRAIn0Qa3xoZdsT434P99GHauCZ5ePq3
 meY4co69c+TfkQZ/b0xlyvT3+7bd9Ni92CQL7n/LtX5bs3pIhRjHNIsm50e4x6Da
 0jb422cKWfRIltXlCfdL/dnFRtJluH83M0seRGMvmhveuWPZ19oSIasEiQjeg//e
 KMNK5z4HEnJ1czb3Bf8p38bmmY/O/QEAA5wAqk7iJkJHz6T/GlqImLYYwpFPlHj6
 JQttm0aWsHrsqEXxnuV0/DT1yHyXDB6S4iuAvABZWhv/M/nCaXo0ib0gW5NPtRPo
 Yf3HO163F9/fewJCc1AUsBe1C/2UwmSWRhEtxpr9uuW2Mv9qEl3hkJwd4k6sEOvh
 U4i+GONC4eElPcmECKUfHA9EP+7faDs6xnM6Ev/PIEp+cPJ2QRfklZv4qpMUWYtb
 3KkADchOyVZAsdB8cGmnznDEVmno1Dt0adVUq8CF6uW6MwD3pb4838arrfwIfwOp
 g4yTI1AQQykkxxOaR7GSNoxWRti7TH4fzLhx/xXDd9TKIOWuOepiyuhB7+Q48WAJ
 6EW/JUIzOe7k3GsI0iBsk+y67ED2tpATiRKWWw4QS1BwIhNEMUqgZkucFfKE6Ze1
 B1Xw+Di+1CzrLHV6lCjtdLVmqWqcGcO43HqhSK4sodE87BDGGwy9UxE6oBVt2C1x
 HPQ6/omDlFk=
 =RAkI
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/thuth-gitlab/tags/pull-request-2021-06-02' into staging

* Update the references to some doc files (use *.rst instead of *.txt)
* Bump minimum versions of some requirements after removing CentOS 7 support

# gpg: Signature made Wed 02 Jun 2021 08:12:18 BST
# gpg:                using RSA key 27B88847EEE0250118F3EAB92ED9D774FE702DB5
# gpg:                issuer "thuth@redhat.com"
# gpg: Good signature from "Thomas Huth <th.huth@gmx.de>" [full]
# gpg:                 aka "Thomas Huth <thuth@redhat.com>" [full]
# gpg:                 aka "Thomas Huth <huth@tuxfamily.org>" [full]
# gpg:                 aka "Thomas Huth <th.huth@posteo.de>" [unknown]
# Primary key fingerprint: 27B8 8847 EEE0 2501 18F3  EAB9 2ED9 D774 FE70 2DB5

* remotes/thuth-gitlab/tags/pull-request-2021-06-02:
  configure: bump min required CLang to 6.0 / XCode 10.0
  configure: bump min required GCC to 7.5.0
  configure: bump min required glib version to 2.56
  tests/docker: drop CentOS 7 container
  tests/vm: convert centos VM recipe to CentOS 8
  crypto: drop used conditional check
  crypto: bump min gnutls to 3.5.18, dropping RHEL-7 support
  crypto: bump min gcrypt to 1.8.0, dropping RHEL-7 support
  crypto: drop back compatibility typedefs for nettle
  crypto: bump min nettle to 3.4, dropping RHEL-7 support
  patchew: move quick build job from CentOS 7 to CentOS 8 container
  block/ssh: Bump minimum libssh version to 0.8.7
  docs: fix references to docs/devel/s390-dasd-ipl.rst
  docs: fix references to docs/specs/tpm.rst
  docs: fix references to docs/devel/build-system.rst
  docs: fix references to docs/devel/atomics.rst
  docs: fix references to docs/devel/tracing.rst

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-06-02 17:08:11 +01:00
Daniel P. Berrangé
6f0748a048 tests/docker: drop CentOS 7 container
It has been over two years since RHEL-8 was released, and thus per the
platform build policy, we no longer need to support RHEL-7 as a build
target.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Acked-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20210514120415.1368922-10-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-06-02 09:11:32 +02:00