* validate_failures.py: Loosen check for build directory.
State what failed if we couldn't find the source tree or
the target triplet.
From-SVN: r198064
When using validate_failures.py with --manifest and --results, we
don't need a GCC build directory. This is useful when using the
validator outside of the build tree. We were insisting on finding
a valid build tree regardless of those options.
Tested on x86_64. Committed to trunk.
* testsuite-management/validate_failures.py: Update
Copyright years.
Request contributions not to use Python features newer
than 2.4.
(GetBuildData): If this is not a build directory,
emit an error only if --results or --manifest are missing.
From-SVN: r195817
2012-12-01 Bernhard Reutner-Fischer <aldot@gcc.gnu.org>
* testsuite-management/validate_failures.py
(IsInterestingResult): Only strip line a second time if we did split.
Rephrase return statement while at it.
(CollectSumFiles): Also ignore .git directory.
From-SVN: r194182
An earlier patch had made the command line options a global variable
_OPTIONS, but it had not renamed all the uses of the old options argument.
* testsuite-management/validate_failures.py: Fix stale
use of 'options'.
From-SVN: r194092
compiler runs.
Usage documentation is in the script.
The script produces output of the form:
$ compare_two_ftime_report_sets "Log0/*perf" "Log3/*perf"
Arithmetic sample for timevar log files
"Log0/*perf"
and selecting lines containing "TOTAL" with desired confidence 95 is
trial count is 4, mean is 443.022 (95% confidence in 440.234 to 445.811),
std.deviation is 1.75264, std.error is 0.876322
Arithmetic sample for timevar log files
"Log3/*perf"
and selecting lines containing "TOTAL" with desired confidence 95 is
trial count is 4, mean is 441.302 (95% confidence in 436.671 to 445.934),
std.deviation is 2.91098, std.error is 1.45549
The first sample appears to be 0.39% larger,
with 60% confidence of being larger.
To reach 95% confidence, you need roughly 14 trials,
assuming the standard deviation is stable, which is iffy.
Tested on x86_64 builds.
Index: contrib/ChangeLog
2012-11-05 Lawrence Crowl <crowl@google.com>
* compare_two_ftime_report_sets: New.
From-SVN: r193277
This is useful when you have two builds of the same compiler. One with
your changes. The other one, a clean build at the same revision.
Instead of using a manifest file, --clean_build will compare the
results it gather from the patched build against those it gathers from
the clean build.
Usage
$ cd /top/of/patched/gcc/bld
$ validate_failures.py --clean_build=clean/bld-gcc
Source directory: /usr/local/google/home/dnovillo/gcc/trunk
Build target: x86_64-unknown-linux-gnu
Getting actual results from build directory .
./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum
./x86_64-unknown-linux-gnu/libffi/testsuite/libffi.sum
./x86_64-unknown-linux-gnu/libgomp/testsuite/libgomp.sum
./x86_64-unknown-linux-gnu/libgo/libgo.sum
./x86_64-unknown-linux-gnu/boehm-gc/testsuite/boehm-gc.sum
./x86_64-unknown-linux-gnu/libatomic/testsuite/libatomic.sum
./x86_64-unknown-linux-gnu/libmudflap/testsuite/libmudflap.sum
./x86_64-unknown-linux-gnu/libitm/testsuite/libitm.sum
./x86_64-unknown-linux-gnu/libjava/testsuite/libjava.sum
./gcc/testsuite/g++/g++.sum
./gcc/testsuite/gnat/gnat.sum
./gcc/testsuite/ada/acats/acats.sum
./gcc/testsuite/gcc/gcc.sum
./gcc/testsuite/gfortran/gfortran.sum
./gcc/testsuite/obj-c++/obj-c++.sum
./gcc/testsuite/go/go.sum
./gcc/testsuite/objc/objc.sum
Getting actual results from build directory clean/bld-gcc
clean/bld-gcc/x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum
clean/bld-gcc/x86_64-unknown-linux-gnu/libffi/testsuite/libffi.sum
clean/bld-gcc/x86_64-unknown-linux-gnu/libgomp/testsuite/libgomp.sum
clean/bld-gcc/x86_64-unknown-linux-gnu/libgo/libgo.sum
clean/bld-gcc/x86_64-unknown-linux-gnu/boehm-gc/testsuite/boehm-gc.sum
clean/bld-gcc/x86_64-unknown-linux-gnu/libatomic/testsuite/libatomic.sum
clean/bld-gcc/x86_64-unknown-linux-gnu/libmudflap/testsuite/libmudflap.sum
clean/bld-gcc/x86_64-unknown-linux-gnu/libitm/testsuite/libitm.sum
clean/bld-gcc/x86_64-unknown-linux-gnu/libjava/testsuite/libjava.sum
clean/bld-gcc/gcc/testsuite/g++/g++.sum
clean/bld-gcc/gcc/testsuite/gnat/gnat.sum
clean/bld-gcc/gcc/testsuite/ada/acats/acats.sum
clean/bld-gcc/gcc/testsuite/gcc/gcc.sum
clean/bld-gcc/gcc/testsuite/gfortran/gfortran.sum
clean/bld-gcc/gcc/testsuite/obj-c++/obj-c++.sum
clean/bld-gcc/gcc/testsuite/go/go.sum
clean/bld-gcc/gcc/testsuite/objc/objc.sum
SUCCESS: No unexpected failures.
2012-11-02 Diego Novillo <dnovillo@google.com>
* testsuite-management/validate_failures.py: Add option
--clean_build to compare test results against another
build.
From-SVN: r193105
This patch adds an xfail manifest for trunk for x86_64 builds. I find
this useful to determine whether my patch has introduced new failures.
The failures in these manifest are always present in trunk and
deciding what to ignore is not very straightforward.
I will keep maintaining this manifest out of clean builds. They are
not hard to maintain. Manifest files can be generated by going to the
top of the build directory and typing:
$ cd <top-of-bld-dir>
$ <path-to-src>/contrib/testsuite-management --produce_manifest
This will generate a .xfail file with the triple name of the target
you just built. Once this file exist you can run the validator again
on the build directory with no arguments. It should produce the
output:
$ cd <top-of-bld-dir>
$ <path-to-src>/contrib/testsuite-management/validate_failures.py
Source directory: <path-to-src>
Build target: x86_64-unknown-linux-gnu
Manifest: <path-to-src>/contrib/testsuite-management/x86_64-unknown-linux-gnu.xfail
Getting actual results from build
./x86_64-unknown-linux-gnu/libstdc++-v3/testsuite/libstdc++.sum
./x86_64-unknown-linux-gnu/libffi/testsuite/libffi.sum
./x86_64-unknown-linux-gnu/libgomp/testsuite/libgomp.sum
./x86_64-unknown-linux-gnu/libgo/libgo.sum
./x86_64-unknown-linux-gnu/boehm-gc/testsuite/boehm-gc.sum
./x86_64-unknown-linux-gnu/libatomic/testsuite/libatomic.sum
./x86_64-unknown-linux-gnu/libmudflap/testsuite/libmudflap.sum
./x86_64-unknown-linux-gnu/libitm/testsuite/libitm.sum
./x86_64-unknown-linux-gnu/libjava/testsuite/libjava.sum
./gcc/testsuite/g++/g++.sum
./gcc/testsuite/gnat/gnat.sum
./gcc/testsuite/ada/acats/acats.sum
./gcc/testsuite/gcc/gcc.sum
./gcc/testsuite/gfortran/gfortran.sum
./gcc/testsuite/obj-c++/obj-c++.sum
./gcc/testsuite/go/go.sum
./gcc/testsuite/objc/objc.sum
SUCCESS: No unexpected failures.
If the output shows new failures, you investigate them. If they are
not yours, you can add them to the xfail manifest (after reporting
them) and then commit the modified .xfail file.
Long term, I would like to have this script pull manifest files from
postings made to gcc-testresults. This way, we won't have to maintain
these .xfail files manually. In branches this is not a big problem,
but in trunk it may be a tad annoying.
From-SVN: r190404
I noticed recently that while the validator was accepting the
'expire=YYYYMMDD' attribute, it was not actually doing anything with
it.
This patch fixes the oversight.
2012-08-13 Diego Novillo <dnovillo@google.com>
* testsuite-management/validate_failures.py: Import datetime.
(TestResult.ExpirationDate): New.
(TestResult.HasExpired): New.
(ParseSummary): Call it. If it returns True, warn that the
expected failure has expired and do not add it to the set of
expected results.
(GetResults): Clarify documentation.
From-SVN: r190351
Some of the hosts were we run this script are still using Python 2.4.
This patch replaces the use of 'with ... as ...' to avoid syntax errors.
2012-07-26 Diego Novillo <dnovillo@google.com>
* testsuite-management/validate_failures.py: Do not use
'with ... as ...' constructs.
From-SVN: r189893
When I added the functionality to use other summary files for
reporting, I broke the generation of manifests. When sum files are
passed on, we need to use the list. Otherwise, we need to find summary
files in the build directory.
This patch factors out that logic into a new function and calls it
from both the reporting and generation routines.
Committed to trunk.
* testsuite-management/validate_failures.py (CollectSumFiles):
Rename from GetSumFiles.
(GetSumFiles): Factor out of CheckExpectedResults.
(CheckExpectedResults): Call it.
(ProduceManifest): Call it.
From-SVN: r189662
The new mklog script prints three spaces after the date part, which
is wrong. Thus fixed by adjusting the split pattern. Tested manually.
2012-05-31 Marek Polacek <polacek@redhat.com>
* mklog: Prevent printing three spaces after the date.
From-SVN: r188265
