Commit Graph

28 Commits

Author SHA1 Message Date
Diego Novillo f15b972787 Fix stale use of old 'options' argument.
An earlier patch had made the command line options a global variable
_OPTIONS, but it had not renamed all the uses of the old options argument.

	* testsuite-management/validate_failures.py: Fix stale
	use of 'options'.

From-SVN: r194092
2012-12-03 09:53:37 -05:00
Doug Evans d5651dcfe8 validate_failures.py: Add support for @include, @remove directives in manifest files.
* testsuite-management/validate_failures.py: Add support for @include,
	@remove directives in manifest files.

From-SVN: r194008
2012-11-30 21:53:34 +00:00
Doug Evans f6fce9517e validate_failures.py: Add function GetManifestPath.
* testsuite-management/validate_failures.py: Add function
	GetManifestPath.  New global _MANIFEST_SUBDIR.

From-SVN: r194007
2012-11-30 20:06:27 +00:00
Doug Evans 74df1ad0fe validate_failures.py: Remove pass/fail indicator from result of GetBuildData.
* testsuite-management/validate_failures.py: Remove pass/fail
	indicator from result of GetBuildData.

From-SVN: r193968
2012-11-29 21:40:16 +00:00
Doug Evans bc5e01b107 validate_failures.py: Store options in global variable _OPTIONS.
* testsuite-management/validate_failures.py: Store options in global
	variable _OPTIONS.

From-SVN: r193967
2012-11-29 21:35:07 +00:00
Doug Evans cd1d95bd62 validate_failures.py: Rename variable manifest_name to manifest_path everywhere.
* testsuite-management/validate_failures.py: Rename variable
	manifest_name to manifest_path everywhere.

From-SVN: r193965
2012-11-29 21:23:22 +00:00
Doug Evans b4b5e1f715 validate_failures.py (CompareBuilds): Pass options.results to GetSumFiles for clean build.
* testsuite-management/validate_failures.py (CompareBuilds): Pass
	options.results to GetSumFiles for clean build.

From-SVN: r193952
2012-11-29 19:50:29 +00:00
Doug Evans 1099bb0a71 validate_failures.py: Use <target_alias> instead of <target>.
* testsuite-management/validate_failures.py: Use <target_alias> instead
	of <target>.  Minor whitespace changes.

From-SVN: r193904
2012-11-28 19:25:24 +00:00
Doug Evans b3891de433 validate_failures.py: Record ordinal with TestResult.
* testsuite-management/validate_failures.py: Record ordinal with
	TestResult.

From-SVN: r193903
2012-11-28 19:23:31 +00:00
Diego Novillo b436bf3805 Add a new option --clean_build to validate_failures.py
This is useful when you have two builds of the same compiler.  One with
your changes.  The other one, a clean build at the same revision.
Instead of using a manifest file, --clean_build will compare the
results it gather from the patched build against those it gathers from
the clean build.

Usage

$ cd /top/of/patched/gcc/bld
$ validate_failures.py --clean_build=clean/bld-gcc
Source directory: /usr/local/google/home/dnovillo/gcc/trunk
Build target:     x86_64-unknown-linux-gnu
Getting actual results from build directory .
        ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum
        ./x86_64-unknown-linux-gnu/libffi/testsuite/libffi.sum
        ./x86_64-unknown-linux-gnu/libgomp/testsuite/libgomp.sum
        ./x86_64-unknown-linux-gnu/libgo/libgo.sum
        ./x86_64-unknown-linux-gnu/boehm-gc/testsuite/boehm-gc.sum
        ./x86_64-unknown-linux-gnu/libatomic/testsuite/libatomic.sum
        ./x86_64-unknown-linux-gnu/libmudflap/testsuite/libmudflap.sum
        ./x86_64-unknown-linux-gnu/libitm/testsuite/libitm.sum
        ./x86_64-unknown-linux-gnu/libjava/testsuite/libjava.sum
        ./gcc/testsuite/g++/g++.sum
        ./gcc/testsuite/gnat/gnat.sum
        ./gcc/testsuite/ada/acats/acats.sum
        ./gcc/testsuite/gcc/gcc.sum
        ./gcc/testsuite/gfortran/gfortran.sum
        ./gcc/testsuite/obj-c++/obj-c++.sum
        ./gcc/testsuite/go/go.sum
        ./gcc/testsuite/objc/objc.sum
Getting actual results from build directory clean/bld-gcc
        clean/bld-gcc/x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum
        clean/bld-gcc/x86_64-unknown-linux-gnu/libffi/testsuite/libffi.sum
        clean/bld-gcc/x86_64-unknown-linux-gnu/libgomp/testsuite/libgomp.sum
        clean/bld-gcc/x86_64-unknown-linux-gnu/libgo/libgo.sum
        clean/bld-gcc/x86_64-unknown-linux-gnu/boehm-gc/testsuite/boehm-gc.sum
        clean/bld-gcc/x86_64-unknown-linux-gnu/libatomic/testsuite/libatomic.sum
        clean/bld-gcc/x86_64-unknown-linux-gnu/libmudflap/testsuite/libmudflap.sum
        clean/bld-gcc/x86_64-unknown-linux-gnu/libitm/testsuite/libitm.sum
        clean/bld-gcc/x86_64-unknown-linux-gnu/libjava/testsuite/libjava.sum
        clean/bld-gcc/gcc/testsuite/g++/g++.sum
        clean/bld-gcc/gcc/testsuite/gnat/gnat.sum
        clean/bld-gcc/gcc/testsuite/ada/acats/acats.sum
        clean/bld-gcc/gcc/testsuite/gcc/gcc.sum
        clean/bld-gcc/gcc/testsuite/gfortran/gfortran.sum
        clean/bld-gcc/gcc/testsuite/obj-c++/obj-c++.sum
        clean/bld-gcc/gcc/testsuite/go/go.sum
        clean/bld-gcc/gcc/testsuite/objc/objc.sum

SUCCESS: No unexpected failures.

2012-11-02  Diego Novillo  <dnovillo@google.com>

	* testsuite-management/validate_failures.py: Add option
	--clean_build to compare test results against another
	build.

From-SVN: r193105
2012-11-02 15:43:25 -04:00
Lawrence Crowl d4ac4ce2d3 This patch renames sbitmap iterators to unify them with the bitmap iterators.
Remove the unused EXECUTE_IF_SET_IN_SBITMAP_REV, which has an unconventional
interface.

Rename the sbitmap_iter_* functions to match bitmap's bmp_iter_* functions.
Add an additional parameter to the initialization and next functions to
match the interface in bmp_iter_*.  This extra parameter is mostly hidden
by the use of the EXECUTE_IF macros.

Rename the EXECUTE_IF_SET_IN_SBITMAP macro to EXECUTE_IF_SET_IN_BITMAP.  Its
implementation is now identical to that in bitmap.h.  To prevent redefinition
errors, both definitions are now guarded by #ifndef.  An alternate strategy
is to simply include bitmap.h from sbitmap.h.  As this would increase build
time, I have elected to use the #ifndef version.  I do not have a strong
preference here.

The sbitmap_iterator type is still distinctly named because it is often
declared in contexts where the bitmap type is not obvious.  There are less
than 40 uses of this type, so the burden to modify it when changing bitmap
types is not large.

Tested on x86-64, config-list.mk testing.


Index: gcc/ChangeLog

2012-10-31  Lawrence Crowl  <crowl@google.com>

	* sbitmap.h (sbitmap_iter_init): Rename bmp_iter_set_init and add
	unused parameter to match bitmap iterator.  Update callers.
	(sbitmap_iter_cond): Rename bmp_iter_set.  Update callers.
	(sbitmap_iter_next): Rename bmp_iter_next and add unused parameter to
	match bitmap iterator.  Update callers.
	(EXECUTE_IF_SET_IN_SBITMAP_REV): Remove unused.
	(EXECUTE_IF_SET_IN_SBITMAP): Rename EXECUTE_IF_SET_IN_BITMAP and
	adjust to be identical to the definition in bitmap.h.  Conditionalize
	the definition based on not having been defined.  Update callers.
	* bitmap.h (EXECUTE_IF_SET_IN_BITMAP): Conditionalize the definition
	based on not having been defined.  (To match the above.)

From-SVN: r193069
2012-11-01 21:02:15 +00:00
Diego Novillo 3b1de8eba7 validate_failures.py: Fix parsing of summary lines.
* testsuite-management/validate_failures.py: Fix parsing
	of summary lines.

From-SVN: r193039
2012-10-31 12:37:06 -04:00
Diego Novillo 6119d95c67 * testsuite-management/x86_64-unknown-linux-gnu.xfail: Update.
From-SVN: r192960
2012-10-29 15:35:35 -04:00
Diego Novillo 1996c0a6e0 x86_64-unknown-linux-gnu.xfail: Update.
2012-10-06  Diego Novillo  <dnovillo@google.com>

	* testsuite-management/x86_64-unknown-linux-gnu.xfail: Update.

From-SVN: r192168
2012-10-06 13:44:39 -04:00
Diego Novillo 5ad7a43ec6 * testsuite-management/x86_64-unknown-linux-gnu.xfail: Update.
From-SVN: r191166
2012-09-10 20:04:25 -04:00
Diego Novillo e8f6d0461b * testsuite-management/x86_64-unknown-linux-gnu.xfail: Update.
From-SVN: r190929
2012-09-04 09:23:10 -04:00
Diego Novillo 18da4303c1 Add an xfail manifest for x86_64-unknown-linux-gnu to trunk.
This patch adds an xfail manifest for trunk for x86_64 builds. I find
this useful to determine whether my patch has introduced new failures.
The failures in these manifest are always present in trunk and
deciding what to ignore is not very straightforward.

I will keep maintaining this manifest out of clean builds. They are
not hard to maintain. Manifest files can be generated by going to the
top of the build directory and typing:

$ cd <top-of-bld-dir>
$ <path-to-src>/contrib/testsuite-management --produce_manifest

This will generate a .xfail file with the triple name of the target
you just built.  Once this file exist you can run the validator again
on the build directory with no arguments.  It should produce the
output:

$ cd <top-of-bld-dir>
$ <path-to-src>/contrib/testsuite-management/validate_failures.py
Source directory: <path-to-src>
Build target:     x86_64-unknown-linux-gnu
Manifest:         <path-to-src>/contrib/testsuite-management/x86_64-unknown-linux-gnu.xfail
Getting actual results from build
        ./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum
        ./x86_64-unknown-linux-gnu/libffi/testsuite/libffi.sum
        ./x86_64-unknown-linux-gnu/libgomp/testsuite/libgomp.sum
        ./x86_64-unknown-linux-gnu/libgo/libgo.sum
        ./x86_64-unknown-linux-gnu/boehm-gc/testsuite/boehm-gc.sum
        ./x86_64-unknown-linux-gnu/libatomic/testsuite/libatomic.sum
        ./x86_64-unknown-linux-gnu/libmudflap/testsuite/libmudflap.sum
        ./x86_64-unknown-linux-gnu/libitm/testsuite/libitm.sum
        ./x86_64-unknown-linux-gnu/libjava/testsuite/libjava.sum
        ./gcc/testsuite/g++/g++.sum
        ./gcc/testsuite/gnat/gnat.sum
        ./gcc/testsuite/ada/acats/acats.sum
        ./gcc/testsuite/gcc/gcc.sum
        ./gcc/testsuite/gfortran/gfortran.sum
        ./gcc/testsuite/obj-c++/obj-c++.sum
        ./gcc/testsuite/go/go.sum
        ./gcc/testsuite/objc/objc.sum


SUCCESS: No unexpected failures.


If the output shows new failures, you investigate them. If they are
not yours, you can add them to the xfail manifest (after reporting
them) and then commit the modified .xfail file.

Long term, I would like to have this script pull manifest files from
postings made to gcc-testresults. This way, we won't have to maintain
these .xfail files manually. In branches this is not a big problem,
but in trunk it may be a tad annoying.

From-SVN: r190404
2012-08-14 22:25:19 -04:00
Diego Novillo a327112f68 validate_failures.py (ExpirationDate): Tidy comment.
* testsuite-management/validate_failures.py (ExpirationDate): Tidy
	comment.

From-SVN: r190354
2012-08-13 15:36:03 -04:00
Diego Novillo 6baa72251b validate_failures.py (ParseSummary): Fix comment.
* testsuite-management/validate_failures.py (ParseSummary): Fix
	comment.

From-SVN: r190352
2012-08-13 14:18:52 -04:00
Diego Novillo c577382e7b Implement support for expiring expected results in validate_failures.py.
I noticed recently that while the validator was accepting the
'expire=YYYYMMDD' attribute, it was not actually doing anything with
it.

This patch fixes the oversight.

2012-08-13  Diego Novillo  <dnovillo@google.com>

	* testsuite-management/validate_failures.py: Import datetime.
	(TestResult.ExpirationDate): New.
	(TestResult.HasExpired): New.
	(ParseSummary): Call it.  If it returns True, warn that the
	expected failure has expired and do not add it to the set of
	expected results.
	(GetResults): Clarify documentation.

From-SVN: r190351
2012-08-13 14:00:55 -04:00
Diego Novillo 55b073bae6 Do not use 'with ... as ...' in validate_failures.py
Some of the hosts were we run this script are still using Python 2.4.
This patch replaces the use of 'with ... as ...' to avoid syntax errors.

2012-07-26   Diego Novillo  <dnovillo@google.com>

	* testsuite-management/validate_failures.py: Do not use
	'with ... as ...' constructs.

From-SVN: r189893
2012-07-26 11:31:00 -04:00
Diego Novillo 29476fe162 Fix --produce_manifest flag in validate_failures.py.
When I added the functionality to use other summary files for
reporting, I broke the generation of manifests. When sum files are
passed on, we need to use the list. Otherwise, we need to find summary
files in the build directory.

This patch factors out that logic into a new function and calls it
from both the reporting and generation routines.

Committed to trunk.

	* testsuite-management/validate_failures.py (CollectSumFiles):
	Rename from GetSumFiles.
	(GetSumFiles): Factor out of CheckExpectedResults.
	(CheckExpectedResults): Call it.
	(ProduceManifest): Call it.

From-SVN: r189662
2012-07-19 10:35:18 -04:00
Diego Novillo 89d223b3ac validate_failures.py: Fix /usr/bin/python invocation.
* testsuite-management/validate_failures.py: Fix
	/usr/bin/python invocation.

From-SVN: r189631
2012-07-18 15:55:46 -04:00
Diego Novillo 831315d08c Extend validate_failures.py to run outside the build directory.
This patch adds three new arguments to validate_failures.py so
it can be used outside the build directory:

--ignore_missing_failures
    When a failure is expected in the manifest but it is not found in
    the actual results, the script produces a note alerting to this
    fact. This means that the expected failure has been fixed, or it
    did not run, or it may simply be flaky.

    With this option, one can ask the script not to show the missing
    failures.

--manifest
    Name of the manifest file to use.  By default, the script will
    look for the manifest file in the source directory associated with
    this build.  With this option, one can point to any arbitrary
    manifest file.  I renamed the old --manifest flag to
    --produce_manifest.

--results
    Space-separated list of .sum files with the testing results to
    check. The only content needed from these files are the lines
    starting with FAIL, XPASS or UNRESOLVED.

From-SVN: r188217
2012-06-04 20:28:56 -04:00
Rainer Orth 034a3fcd14 Use target_alias in validate_failures.py
* testsuite-management/validate_failures.py (GetBuildData): Use
	target_alias.

From-SVN: r186497
2012-04-16 14:49:00 +00:00
Diego Novillo 9c23e8b86d This fixes an edge case in parsing summary lines.
This fixes an edge case in parsing summary lines.  Some times, the
description field is missing (e.g., 'FAIL: libstdc++/abi_check'), so
the space that the pattern was looking for does not exist.

I've changed it to match any whitespace, which includes '\n'.  I also
made it print the line that it fails to parse, in case there are other
problems like this in the future.

2012-03-02   Diego Novillo  <dnovillo@google.com>

	* testsuite-management/validate_failures.py (class TestResult): Fix
	match pattern for the summary line.  If there is a parsing failure,
	show the line we failed to parse.

From-SVN: r184822
2012-03-02 15:43:29 -05:00
Doug Kwan 4ad29d0de3 2012-02-17 Doug Kwan <dougkwan@google.com>
* contrib/testsuite-management/validate_failures.py
	(GetMakefileValue): Check for cross compilers.

From-SVN: r184357
2012-02-17 23:55:00 +00:00
Diego Novillo a5baf3b8ce testsuite-management: New.
* testsuite-management: New.
	* testsuite-management/validate_failures.py: New.

From-SVN: r178833
2011-09-13 16:24:47 -04:00