This patch, an updated version of
<https://sourceware.org/ml/libc-alpha/2014-01/msg00195.html>, makes it
possible for .test-result files for individual tests to contain XPASS
and XFAIL rather than PASS and FAIL in cases where failure is
expected. This replaces the marking of two individual tests with "-"
to cause them to be expected at makefile level to fail;
evaluate-test.sh will ensure it exits with status 0 for an expected
failure.
Tested x86_64.
* scripts/evaluate-test.sh: Take new argument indicating whether
failure is expected.
* Makeconfig (evaluate-test): Pass argument to evaluate-test.sh
indicating whether failure is expected.
* conform/Makefile (test-xfail-run-conformtest): New variable.
($(objpfx)run-conformtest.out): Don't expect to fail at makefile
level.
* posix/Makefile (test-xfail-annexc): New variable.
($(objpfx)annexc.out): Don't expect to fail at makefile level.
This patch, an updated version of
<https://sourceware.org/ml/libc-alpha/2014-01/msg00193.html>, starts
the process of generating explicit PASS or FAIL status for individual
glibc tests. It's based on Tomas Dohnalek's patch
<https://sourceware.org/ml/libc-alpha/2012-10/msg00278.html>, but is
deliberately more minimal: it doesn't try to cover any tests outside
of $(tests) / $(xtests) (that's for a later patch), nor does it put
the result together in an overall summary file (again, a later patch):
it just generates the .test-result files.
Thus, this patch keeps the overall logic for when a testsuite run
finishes completely unchanged: a test failing will terminate the run.
I think we *should* move to a more conventional approach where plain
"make check" does not terminate for an individual test failure, unless
e.g. you say "make stop-on-test-failure=y check", but that sort of
policy change is best done as a separate patch once the infrastructure
is in place to generate summary files for completed test runs (which
will entirely consist of PASS and XFAIL lines if the testsuite run
reaches the point of generating them, until such a policy change is
made).
Tested x86_64.
2014-02-14 Tomas Dohnalek <tdohnale@redhat.com>
Joseph Myers <joseph@codesourcery.com>
* Makeconfig (test-name): New variable.
(evaluate-test): Likewise.
* Makerules (do-test-clean): Remove .test-result files.
(common-mostlyclean): Likewise.
* Rules ($(objpfx)%.out): Use $(evaluate-test) in both rules.
* scripts/evaluate-test.sh: New file.
Joseph Myers
Tomas Dohnalek