2f7350cd43
The `ccache` tool can be very effective at reducing compilation times when re-running pipelines with only minor changes each time. For example a fresh 'build-system-fedora' job will typically take 20 minutes on the gitlab.com shared runners. With ccache this is reduced to as little as 6 minutes. Normally meson would auto-detect existance of ccache in $PATH and use it automatically, but the way we wrap meson from configure breaks this, as we're passing in an config file with explicitly set compiler paths. Thus we need to add $CCACHE_WRAPPERSPATH to the front of $PATH. For unknown reasons if doing this in msys though, gcc becomes unable to invoke 'cc1' when run from meson. For msys we thus set CC='ccache gcc' before invoking 'configure' instead. A second problem with msys is that cache misses are incredibly expensive, so enabling ccache massively slows down the build when the cache isn't well populated. This is suspected to be a result of the cost of spawning processes under the msys architecture. To deal with this we set CCACHE_DEPEND=1 which enables ccache's 'depend_only' strategy. This avoids extra spawning of the pre-processor during cache misses, with the downside that is it less likely ccache will find a cache hit after semantically benign compiler flag changes. This is the lesser of two evils, as otherwise we can't use ccache at all under msys and remain inside the job time limit. If people are finding ccache to hurt their pipelines, it can be disabled by setting the 'CCACHE_DISABLE=1' env variable against their gitlab fork CI settings. Signed-off-by: Daniel P. Berrangé <berrange@redhat.com> Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-Id: <20230804111054.281802-2-berrange@redhat.com> Signed-off-by: Alex Bennée <alex.bennee@linaro.org> Message-Id: <20230829161528.2707696-2-alex.bennee@linaro.org>
198 lines
6.1 KiB
HTML
198 lines
6.1 KiB
HTML
.. _ci_var:
|
|
|
|
Custom CI/CD variables
|
|
======================
|
|
|
|
QEMU CI pipelines can be tuned by setting some CI environment variables.
|
|
|
|
Set variable globally in the user's CI namespace
|
|
------------------------------------------------
|
|
|
|
Variables can be set globally in the user's CI namespace setting.
|
|
|
|
For further information about how to set these variables, please refer to::
|
|
|
|
https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
|
|
|
|
Set variable manually when pushing a branch or tag to the user's repository
|
|
---------------------------------------------------------------------------
|
|
|
|
Variables can be set manually when pushing a branch or tag, using
|
|
git-push command line arguments.
|
|
|
|
Example setting the QEMU_CI_EXAMPLE_VAR variable:
|
|
|
|
.. code::
|
|
|
|
git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch
|
|
|
|
For further information about how to set these variables, please refer to::
|
|
|
|
https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
|
|
|
|
Setting aliases in your git config
|
|
----------------------------------
|
|
|
|
You can use aliases to make it easier to push branches with different
|
|
CI configurations. For example define an alias for triggering CI:
|
|
|
|
.. code::
|
|
|
|
git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
|
|
git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"
|
|
|
|
Which lets you run:
|
|
|
|
.. code::
|
|
|
|
git push-ci
|
|
|
|
to create the pipeline, or:
|
|
|
|
.. code::
|
|
|
|
git push-ci-now
|
|
|
|
to create and run the pipeline
|
|
|
|
|
|
Variable naming and grouping
|
|
----------------------------
|
|
|
|
The variables used by QEMU's CI configuration are grouped together
|
|
in a handful of namespaces
|
|
|
|
* QEMU_JOB_nnnn - variables to be defined in individual jobs
|
|
or templates, to influence the shared rules defined in the
|
|
.base_job_template.
|
|
|
|
* QEMU_CI_nnn - variables to be set by contributors in their
|
|
repository CI settings, or as git push variables, to influence
|
|
which jobs get run in a pipeline
|
|
|
|
* QEMU_CI_CONTAINER_TAG - the tag used to publish containers
|
|
in stage 1, for use by build jobs in stage 2. Defaults to
|
|
'latest', but if running pipelines for different branches
|
|
concurrently, it should be overridden per pipeline.
|
|
|
|
* QEMU_CI_UPSTREAM - gitlab namespace that is considered to be
|
|
the 'upstream'. This defaults to 'qemu-project'. Contributors
|
|
may choose to override this if they are modifying rules in
|
|
base.yml and need to validate how they will operate when in
|
|
an upstream context, as opposed to their fork context.
|
|
|
|
* nnn - other misc variables not falling into the above
|
|
categories, or using different names for historical reasons
|
|
and not yet converted.
|
|
|
|
Maintainer controlled job variables
|
|
-----------------------------------
|
|
|
|
The following variables may be set when defining a job in the
|
|
CI configuration file.
|
|
|
|
QEMU_JOB_CIRRUS
|
|
~~~~~~~~~~~~~~~
|
|
|
|
The job makes use of Cirrus CI infrastructure, requiring the
|
|
configuration setup for cirrus-run to be present in the repository
|
|
|
|
QEMU_JOB_OPTIONAL
|
|
~~~~~~~~~~~~~~~~~
|
|
|
|
The job is expected to be successful in general, but is not run
|
|
by default due to need to conserve limited CI resources. It is
|
|
available to be started manually by the contributor in the CI
|
|
pipelines UI.
|
|
|
|
QEMU_JOB_ONLY_FORKS
|
|
~~~~~~~~~~~~~~~~~~~
|
|
|
|
The job results are only of interest to contributors prior to
|
|
submitting code. They are not required as part of the gating
|
|
CI pipeline.
|
|
|
|
QEMU_JOB_SKIPPED
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
The job is not reliably successsful in general, so is not
|
|
currently suitable to be run by default. Ideally this should
|
|
be a temporary marker until the problems can be addressed, or
|
|
the job permanently removed.
|
|
|
|
QEMU_JOB_PUBLISH
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
The job is for publishing content after a branch has been
|
|
merged into the upstream default branch.
|
|
|
|
QEMU_JOB_AVOCADO
|
|
~~~~~~~~~~~~~~~~
|
|
|
|
The job runs the Avocado integration test suite
|
|
|
|
Contributor controlled runtime variables
|
|
----------------------------------------
|
|
|
|
The following variables may be set by contributors to control
|
|
job execution
|
|
|
|
QEMU_CI
|
|
~~~~~~~
|
|
|
|
By default, no pipelines will be created on contributor forks
|
|
in order to preserve CI credits
|
|
|
|
Set this variable to 1 to create the pipelines, but leave all
|
|
the jobs to be manually started from the UI
|
|
|
|
Set this variable to 2 to create the pipelines and run all
|
|
the jobs immediately, as was historicaly behaviour
|
|
|
|
QEMU_CI_AVOCADO_TESTING
|
|
~~~~~~~~~~~~~~~~~~~~~~~
|
|
By default, tests using the Avocado framework are not run automatically in
|
|
the pipelines (because multiple artifacts have to be downloaded, and if
|
|
these artifacts are not already cached, downloading them make the jobs
|
|
reach the timeout limit). Set this variable to have the tests using the
|
|
Avocado framework run automatically.
|
|
|
|
Other misc variables
|
|
--------------------
|
|
|
|
These variables are primarily to control execution of jobs on
|
|
private runners
|
|
|
|
AARCH64_RUNNER_AVAILABLE
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
If you've got access to an aarch64 host that can be used as a gitlab-CI
|
|
runner, you can set this variable to enable the tests that require this
|
|
kind of host. The runner should be tagged with "aarch64".
|
|
|
|
AARCH32_RUNNER_AVAILABLE
|
|
~~~~~~~~~~~~~~~~~~~~~~~~
|
|
If you've got access to an armhf host or an arch64 host that can run
|
|
aarch32 EL0 code to be used as a gitlab-CI runner, you can set this
|
|
variable to enable the tests that require this kind of host. The
|
|
runner should be tagged with "aarch32".
|
|
|
|
S390X_RUNNER_AVAILABLE
|
|
~~~~~~~~~~~~~~~~~~~~~~
|
|
If you've got access to an IBM Z host that can be used as a gitlab-CI
|
|
runner, you can set this variable to enable the tests that require this
|
|
kind of host. The runner should be tagged with "s390x".
|
|
|
|
CENTOS_STREAM_8_x86_64_RUNNER_AVAILABLE
|
|
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
|
If you've got access to a CentOS Stream 8 x86_64 host that can be
|
|
used as a gitlab-CI runner, you can set this variable to enable the
|
|
tests that require this kind of host. The runner should be tagged with
|
|
both "centos_stream_8" and "x86_64".
|
|
|
|
CCACHE_DISABLE
|
|
~~~~~~~~~~~~~~
|
|
The jobs are configured to use "ccache" by default since this typically
|
|
reduces compilation time, at the cost of increased storage. If the
|
|
use of "ccache" is suspected to be hurting the overall job execution
|
|
time, setting the "CCACHE_DISABLE=1" env variable to disable it.
|