This patch adds an xfail manifest for trunk for x86_64 builds. I find
this useful to determine whether my patch has introduced new failures.
The failures in these manifest are always present in trunk and
deciding what to ignore is not very straightforward.
I will keep maintaining this manifest out of clean builds. They are
not hard to maintain. Manifest files can be generated by going to the
top of the build directory and typing:
$ cd <top-of-bld-dir>
$ <path-to-src>/contrib/testsuite-management --produce_manifest
This will generate a .xfail file with the triple name of the target
you just built. Once this file exist you can run the validator again
on the build directory with no arguments. It should produce the
output:
$ cd <top-of-bld-dir>
$ <path-to-src>/contrib/testsuite-management/validate_failures.py
Source directory: <path-to-src>
Build target: x86_64-unknown-linux-gnu
Manifest: <path-to-src>/contrib/testsuite-management/x86_64-unknown-linux-gnu.xfail
Getting actual results from build
./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum
./x86_64-unknown-linux-gnu/libffi/testsuite/libffi.sum
./x86_64-unknown-linux-gnu/libgomp/testsuite/libgomp.sum
./x86_64-unknown-linux-gnu/libgo/libgo.sum
./x86_64-unknown-linux-gnu/boehm-gc/testsuite/boehm-gc.sum
./x86_64-unknown-linux-gnu/libatomic/testsuite/libatomic.sum
./x86_64-unknown-linux-gnu/libmudflap/testsuite/libmudflap.sum
./x86_64-unknown-linux-gnu/libitm/testsuite/libitm.sum
./x86_64-unknown-linux-gnu/libjava/testsuite/libjava.sum
./gcc/testsuite/g++/g++.sum
./gcc/testsuite/gnat/gnat.sum
./gcc/testsuite/ada/acats/acats.sum
./gcc/testsuite/gcc/gcc.sum
./gcc/testsuite/gfortran/gfortran.sum
./gcc/testsuite/obj-c++/obj-c++.sum
./gcc/testsuite/go/go.sum
./gcc/testsuite/objc/objc.sum
SUCCESS: No unexpected failures.
If the output shows new failures, you investigate them. If they are
not yours, you can add them to the xfail manifest (after reporting
them) and then commit the modified .xfail file.
Long term, I would like to have this script pull manifest files from
postings made to gcc-testresults. This way, we won't have to maintain
these .xfail files manually. In branches this is not a big problem,
but in trunk it may be a tad annoying.
From-SVN: r190404
I noticed recently that while the validator was accepting the
'expire=YYYYMMDD' attribute, it was not actually doing anything with
it.
This patch fixes the oversight.
2012-08-13 Diego Novillo <dnovillo@google.com>
* testsuite-management/validate_failures.py: Import datetime.
(TestResult.ExpirationDate): New.
(TestResult.HasExpired): New.
(ParseSummary): Call it. If it returns True, warn that the
expected failure has expired and do not add it to the set of
expected results.
(GetResults): Clarify documentation.
From-SVN: r190351
Some of the hosts were we run this script are still using Python 2.4.
This patch replaces the use of 'with ... as ...' to avoid syntax errors.
2012-07-26 Diego Novillo <dnovillo@google.com>
* testsuite-management/validate_failures.py: Do not use
'with ... as ...' constructs.
From-SVN: r189893
When I added the functionality to use other summary files for
reporting, I broke the generation of manifests. When sum files are
passed on, we need to use the list. Otherwise, we need to find summary
files in the build directory.
This patch factors out that logic into a new function and calls it
from both the reporting and generation routines.
Committed to trunk.
* testsuite-management/validate_failures.py (CollectSumFiles):
Rename from GetSumFiles.
(GetSumFiles): Factor out of CheckExpectedResults.
(CheckExpectedResults): Call it.
(ProduceManifest): Call it.
From-SVN: r189662
This patch adds three new arguments to validate_failures.py so
it can be used outside the build directory:
--ignore_missing_failures
When a failure is expected in the manifest but it is not found in
the actual results, the script produces a note alerting to this
fact. This means that the expected failure has been fixed, or it
did not run, or it may simply be flaky.
With this option, one can ask the script not to show the missing
failures.
--manifest
Name of the manifest file to use. By default, the script will
look for the manifest file in the source directory associated with
this build. With this option, one can point to any arbitrary
manifest file. I renamed the old --manifest flag to
--produce_manifest.
--results
Space-separated list of .sum files with the testing results to
check. The only content needed from these files are the lines
starting with FAIL, XPASS or UNRESOLVED.
From-SVN: r188217
This fixes an edge case in parsing summary lines. Some times, the
description field is missing (e.g., 'FAIL: libstdc++/abi_check'), so
the space that the pattern was looking for does not exist.
I've changed it to match any whitespace, which includes '\n'. I also
made it print the line that it fails to parse, in case there are other
problems like this in the future.
2012-03-02 Diego Novillo <dnovillo@google.com>
* testsuite-management/validate_failures.py (class TestResult): Fix
match pattern for the summary line. If there is a parsing failure,
show the line we failed to parse.
From-SVN: r184822
Diego Novillo
Diego Novillo
Rainer Orth
Doug Kwan