OpenE2K
/
qemu-e2k
13
4
Fork 0
Code Issues 4 Pull Requests Projects 2 Releases Activity

docs/devel: Explain how acceptance tests can be skipped 

Documented under the "Acceptance tests using the Avocado Framework"
section in testing.rst how environment variables are used to skip tests.

Signed-off-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Message-Id: <20210115210022.417996-1-wainersm@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
This commit is contained in:
Wainer dos Santos Moschetta 2021-01-15 18:00:22 -03:00 committed by Thomas Huth
parent 12a917a14f
commit cf5891ec53
1 changed files with 62 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

62
docs/devel/testing.rst
View File

@ -871,6 +871,68 @@ qemu_bin
The exact QEMU binary to be used on QEMUMachine.
Skipping tests
--------------
The Avocado framework provides Python decorators which allow for easily skip
tests running under certain conditions. For example, on the lack of a binary
on the test system or when the running environment is a CI system. For further
information about those decorators, please refer to::
https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#skipping-tests
While the conditions for skipping tests are often specifics of each one, there
are recurring scenarios identified by the QEMU developers and the use of
environment variables became a kind of standard way to enable/disable tests.
Here is a list of the most used variables:
AVOCADO_ALLOW_LARGE_STORAGE
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tests which are going to fetch or produce assets considered *large* are not
going to run unless that `AVOCADO_ALLOW_LARGE_STORAGE=1` is exported on
the environment.
The definition of *large* is a bit arbitrary here, but it usually means an
asset which occupies at least 1GB of size on disk when uncompressed.
AVOCADO_ALLOW_UNTRUSTED_CODE
~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There are tests which will boot a kernel image or firmware that can be
considered not safe to run on the developer's workstation, thus they are
skipped by default. The definition of *not safe* is also arbitrary but
usually it means a blob which either its source or build process aren't
public available.
You should export `AVOCADO_ALLOW_UNTRUSTED_CODE=1` on the environment in
order to allow tests which make use of those kind of assets.
AVOCADO_TIMEOUT_EXPECTED
~~~~~~~~~~~~~~~~~~~~~~~~
The Avocado framework has a timeout mechanism which interrupts tests to avoid the
test suite of getting stuck. The timeout value can be set via test parameter or
property defined in the test class, for further details::
https://avocado-framework.readthedocs.io/en/latest/guides/writer/chapters/writing.html#setting-a-test-timeout
Even though the timeout can be set by the test developer, there are some tests
that may not have a well-defined limit of time to finish under certain
conditions. For example, tests that take longer to execute when QEMU is
compiled with debug flags. Therefore, the `AVOCADO_TIMEOUT_EXPECTED` variable
has been used to determine whether those tests should run or not.
GITLAB_CI
~~~~~~~~~
A number of tests are flagged to not run on the GitLab CI. Usually because
they proved to the flaky or there are constraints on the CI environment which
would make them fail. If you encounter a similar situation then use that
variable as shown on the code snippet below to skip the test:
.. code::
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
def test(self):
do_something()
Uninstalling Avocado
--------------------