In forks QEMU_CI=1 can be used to create a pipeline but not auto-run any
jobs. In upstream jobs always auto-run, which is equiv of QEMU_CI=2.
This supports setting QEMU_CI=1 in upstream, to disable job auto-run.
This can be used to preserve CI minutes if repushing a branch to staging
with a specific fix that only needs testing in limited scenarios.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20230608164018.2520330-6-berrange@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
In upstream context we only run pipelines on staging branches, and
limited publishing jobs on the default branch.
We don't want to run pipelines on stable branches, or tags, because
the content will have already been tested on a staging branch before
getting pushed.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20230608164018.2520330-5-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
If the stable staging branches publish containers under the 'latest' tag
they will clash with containers published on the primary staging branch,
as well as with each other. This introduces logic that overrides the
container tag when jobs run against the stable staging branches.
The CI_COMMIT_REF_SLUG variable we use expands to the git branch name,
but with most special characters removed, such that it is valid as a
docker tag name. eg 'staging-8.0' will get a slug of 'staging-8-0'
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20230608164018.2520330-4-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The CI rules have special logic for what happens in upstream. To enable
contributors who modify CI rules to test this logic, however, they need
to be able to override which repo is considered upstream. This
introduces the 'QEMU_CI_UPSTREAM' variable
git push gitlab <branch> -o ci.variable=QEMU_CI_UPSTREAM=berrange
to make it look as if my namespace is the actual upstream. Namespace in
this context refers to the path fragment in gitlab URLs that is above
the repository. Typically this will be the contributor's gitlab login
name.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20230608164018.2520330-3-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
We use a fixed container tag of 'latest' so that contributors' forks
don't end up with an ever growing number of containers as they work
on throwaway feature branches.
This fixed tag causes problems running CI upstream in stable staging
branches, however, because the stable staging branch will publish old
container content that clashes with that needed by primary staging
branch. This makes it impossible to reliably run CI pipelines in
parallel in upstream for different staging branches.
This introduces $QEMU_CI_CONTAINER_TAG global variable as a way to
change which tag container publishing uses. Initially it can be set
by contributors as a git push option if they want to override the
default use of 'latest' eg
git push gitlab <branch> -o ci.variable=QEMU_CONTAINER_TAG=fish
this is useful if contributors need to run pipelines for different
branches concurrently in their forks.
Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20230608164018.2520330-2-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-Id: <20230403134920.2132362-11-alex.bennee@linaro.org>
When handling pull requests in the staging branch, it often happens
that one of the job fails due to a problem, so that the pull request
can't be merged. Peter/Richard/Stefan then informs the sender of the
pull request and continues by pushing the next pending pull request
from another subsystem maintainer. Now the problem is that there might
still be lots of other running jobs in the pipeline of the first pull
request, eating up precious CI minutes though the pipeline is not
needed anymore. We can avoid this by marking the jobs as "interruptible".
With this setting, the jobs from previous pipelines are automatically
terminated when pushing a new one. If someone does not like this auto-
matic termination, it can still be disabled in the settings of the
repository. See this URL for details:
https://docs.gitlab.com/ee/ci/yaml/index.html#interruptible
Message-Id: <20230223191343.1064274-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Suggested-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Signed-off-by: Mark Cave-Ayland <mark.cave-ayland@ilande.co.uk>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20230124180127.1881110-3-alex.bennee@linaro.org>
The GitLab variable comparisons don't have shell like semantics where
an unset variable compares equal to empty string. We need to explicitly
test against 'null' to detect an unset variable.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220608160651.248781-1-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20220613171258.1905715-7-alex.bennee@linaro.org>
To preserve CI shared runner credits we don't want to run
pipelines on every push.
This sets up the config so that pipelines are never created
for contributors by default. To override this the QEMU_CI
variable can be set to a non-zero value. If set to 1, the
pipeline will be created but all jobs will remain manually
started. The contributor can selectively run jobs that they
care about. If set to 2, the pipeline will be created and
all jobs will immediately start.
This behavior can be controlled using push variables
git push -o ci.variable=QEMU_CI=1
To make this more convenient define an alias
git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"
Which lets you run
git push-ci
to create the pipeline, or
git push-ci-now
to create and run the pipeline
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-6-berrange@redhat.com>
[AJB: fix typo, replicate alias tips in ci.rst]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-33-alex.bennee@linaro.org>
This converts the main build and container jobs to use the
base job rules, defining the following new variables
- QEMU_JOB_SKIPPED - jobs that are known to be currently
broken and should not be run. Can still be manually
launched if desired.
- QEMU_JOB_AVOCADO - jobs that run the Avocado integration
test harness.
- QEMU_JOB_PUBLISH - jobs that publish content after the
branch is merged upstream
As build-tools-and-docs runs on master we declare the requirement of
building amd64-debian-container optional as it should already exits
once we merge.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-5-berrange@redhat.com>
[AJB: fix upstream typo, mention optional container req]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-32-alex.bennee@linaro.org>
This folds the static checks into using the base job
template rules, introducing one new variable
- QEMU_JOB_ONLY_FORKS - a job that should never run
on an upstream pipeline. The information it reports
is only applicable to contributors in a pre-submission
scenario, not time of merge.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-4-berrange@redhat.com>
[AJB: fix typo]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-31-alex.bennee@linaro.org>
This folds the Cirrus job rules into the base job
template, introducing two new variables
- QEMU_JOB_CIRRUS - identifies the job as making
use of Cirrus CI via cirrus-run
- QEMU_JOB_OPTIONAL - identifies the job as one
that is not run by default, primarily due to
resource constraints. It can be manually invoked
by users if they wish to validate that scenario.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-3-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-30-alex.bennee@linaro.org>
Currently job rules are spread across the various templates
and jobs, making it hard to understand exactly what runs in
what scenario. This leads to inconsistency in the rules and
increased maint burden.
The intent is that we introduce a common '.base_job_template'
which will have a general purpose 'rules:' block. No other
template or job should define 'rules:', but instead they must
rely on the inherited rules. To allow behaviour to be tweaked,
rules will be influenced by a number of variables with the
naming scheme 'QEMU_JOB_nnnn'.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20220526110705.59952-2-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20220527153603.887929-29-alex.bennee@linaro.org>
Daniel P. Berrangé
Alex Bennée
Thomas Huth
Mark Cave-Ayland