glibc/math/README.libm-test

104 lines
4.3 KiB
Plaintext
Raw Normal View History

README for libm-test math test suite
====================================
The libm-test math test suite tests a number of function points of
math functions in the GNU C library. The following sections contain a
brief overview. Please note that the test drivers and the Perl script
"gen-libm-test.pl" have some options. A full list of options is
available with --help (for the test drivers) and -h for
"gen-libm-test.pl".
What is tested?
===============
The tests just evaluate the functions at specified points and compare
the results with precomputed values and the requirements of the ISO
C99 standard.
Besides testing the special values mandated by IEEE 754 (infinities,
NaNs and minus zero), some more or less random values are tested.
Files that are part of libm-test
================================
The main file is "libm-test.inc". It is platform and floating point
format independent. The file must be preprocessed by the Perl script
"gen-libm-test.pl". The results are "libm-test.c" and a file
"libm-test-ulps.h" with platform specific deltas.
The test drivers test-double.c, test-float.c, test-ldouble.c test the
normal double, float and long double implementation of libm. The test
drivers with an i in it (test-idouble.c, test-ifloat.c,
test-ildoubl.c) test the corresponding inline functions (where
available - otherwise they also test the real functions in libm).
"gen-libm-test.pl" needs a platform specific files with ULPs (Units of
Last Precision). The file is called "libm-test-ulps" and lives in
platform specific sysdep directory.
How can I generate "libm-test-ulps"?
====================================
New Makefile target `regen-ulps'. The wiki "Regeneration" page has this to say about update ULPs. "The libm-test-ulps files are semiautomatically updated. To update an ulps baseline, run each of the failing tests (test-float, test-double, etc.) with -u; this will generate a file called ULPs; concatenate each of those files with the existing libm-test-ulps file, after removing any entries for particularly huge numbers of ulps that you do not want to mark as expected. Then run gen-libm-test.pl -n -u FILE where FILE is the concatenated file produced in the previous step. This generates a file called NewUlps which is the new sorted version of libm-test-ulps." The same information is listed in math/README.libm-test, and is a lot of manual work that you often want to run over-and-over again while working on a particular test. The `regen-ulps' convenience target does this automatically for developers. We strictly assume the source tree is readonly and add a new --output-dir option to libm-test.inc to allow for writing out ULPs to $(objpfx). When run the new target does the following: * Starts with the baseline ULPs file. * Runs each of the libm math tests with -u. * Adds new changes seen with -u to the baseline. * Sorts and prepares the test output with gen-libm-test.pl. * Leaves math/NewUlps in your build tree to copy to your source tree, cleanup, and checkin. The math test documentation in math/README.libm-test is updated document the new Makefile target. --- 2013-04-06 Carlos O'Donell <carlos@redhat.com> * Makefile.in (regen-ulps): New target. * math/Makefile [ifneq (no,$(PERL)]: Declare regen-ulps with .PHONY. [ifneq (no,$(PERL)] (run-regen-ulps): New variable. [ifneq (no,$(PERL)] (regen-ulps): New target. [ifeq (no,$(PERL)] (regen-ulps): New target. * math/libm-test.inc (ulps_file_name): Define. (output_dir): New variable. (options): Add "output-dir" option. (parse_opt): Handle 'o' case. (main): If output_dir is non-NULL use it as a prefix otherwise use "". * math/README.libm-test: Update `How can I generate "libm-test-ulps"?'
2013-04-06 22:22:47 +02:00
To automatically generate a new "libm-test-ulps" run "make regen-ulps".
This generates the file "math/NewUlps" in the build directory. The file
contains the sorted results of all the tests. You can use the "NewUlps"
file as the machine's updated "libm-test-ulps" file. Copy "NewUlps" to
"libm-test-ulps" in the appropriate machine sysdep directory. Verify
the changes, post your patch, and check it in after review.
To manually generate a new "libm-test-ulps" file, first remove "ULPs"
file in the current directory, then you can execute for example:
./testrun.sh math/test-double -u --ignore-max-ulp=yes
This generates a file "ULPs" with all double ULPs in it, ignoring any
New Makefile target `regen-ulps'. The wiki "Regeneration" page has this to say about update ULPs. "The libm-test-ulps files are semiautomatically updated. To update an ulps baseline, run each of the failing tests (test-float, test-double, etc.) with -u; this will generate a file called ULPs; concatenate each of those files with the existing libm-test-ulps file, after removing any entries for particularly huge numbers of ulps that you do not want to mark as expected. Then run gen-libm-test.pl -n -u FILE where FILE is the concatenated file produced in the previous step. This generates a file called NewUlps which is the new sorted version of libm-test-ulps." The same information is listed in math/README.libm-test, and is a lot of manual work that you often want to run over-and-over again while working on a particular test. The `regen-ulps' convenience target does this automatically for developers. We strictly assume the source tree is readonly and add a new --output-dir option to libm-test.inc to allow for writing out ULPs to $(objpfx). When run the new target does the following: * Starts with the baseline ULPs file. * Runs each of the libm math tests with -u. * Adds new changes seen with -u to the baseline. * Sorts and prepares the test output with gen-libm-test.pl. * Leaves math/NewUlps in your build tree to copy to your source tree, cleanup, and checkin. The math test documentation in math/README.libm-test is updated document the new Makefile target. --- 2013-04-06 Carlos O'Donell <carlos@redhat.com> * Makefile.in (regen-ulps): New target. * math/Makefile [ifneq (no,$(PERL)]: Declare regen-ulps with .PHONY. [ifneq (no,$(PERL)] (run-regen-ulps): New variable. [ifneq (no,$(PERL)] (regen-ulps): New target. [ifeq (no,$(PERL)] (regen-ulps): New target. * math/libm-test.inc (ulps_file_name): Define. (output_dir): New variable. (options): Add "output-dir" option. (parse_opt): Handle 'o' case. (main): If output_dir is non-NULL use it as a prefix otherwise use "". * math/README.libm-test: Update `How can I generate "libm-test-ulps"?'
2013-04-06 22:22:47 +02:00
previously calculated ULPs, and running with the newly built dynamic
loader and math library (assumes you didn't install your build). Now
generate the ULPs for all other formats, the tests will be appending the
data to the "ULPs" file. As final step run "gen-libm-test.pl" with the
file as input and ask to generate a pretty printed output in the file
"NewUlps":
gen-libm-test.pl -u ULPs -n
New Makefile target `regen-ulps'. The wiki "Regeneration" page has this to say about update ULPs. "The libm-test-ulps files are semiautomatically updated. To update an ulps baseline, run each of the failing tests (test-float, test-double, etc.) with -u; this will generate a file called ULPs; concatenate each of those files with the existing libm-test-ulps file, after removing any entries for particularly huge numbers of ulps that you do not want to mark as expected. Then run gen-libm-test.pl -n -u FILE where FILE is the concatenated file produced in the previous step. This generates a file called NewUlps which is the new sorted version of libm-test-ulps." The same information is listed in math/README.libm-test, and is a lot of manual work that you often want to run over-and-over again while working on a particular test. The `regen-ulps' convenience target does this automatically for developers. We strictly assume the source tree is readonly and add a new --output-dir option to libm-test.inc to allow for writing out ULPs to $(objpfx). When run the new target does the following: * Starts with the baseline ULPs file. * Runs each of the libm math tests with -u. * Adds new changes seen with -u to the baseline. * Sorts and prepares the test output with gen-libm-test.pl. * Leaves math/NewUlps in your build tree to copy to your source tree, cleanup, and checkin. The math test documentation in math/README.libm-test is updated document the new Makefile target. --- 2013-04-06 Carlos O'Donell <carlos@redhat.com> * Makefile.in (regen-ulps): New target. * math/Makefile [ifneq (no,$(PERL)]: Declare regen-ulps with .PHONY. [ifneq (no,$(PERL)] (run-regen-ulps): New variable. [ifneq (no,$(PERL)] (regen-ulps): New target. [ifeq (no,$(PERL)] (regen-ulps): New target. * math/libm-test.inc (ulps_file_name): Define. (output_dir): New variable. (options): Add "output-dir" option. (parse_opt): Handle 'o' case. (main): If output_dir is non-NULL use it as a prefix otherwise use "". * math/README.libm-test: Update `How can I generate "libm-test-ulps"?'
2013-04-06 22:22:47 +02:00
Copy "NewUlps" to "libm-test-ulps" in the appropriate machine sysdep
directory.
Note that the test drivers have an option "-u" to output an unsorted
list of all epsilons that the functions have. The output can be read
in directly but it's better to pretty print it first.
"gen-libm-test.pl" has an option to generate a pretty-printed and
sorted new ULPs file from the output of the test drivers.
Contents of libm-test-ulps
==========================
Don't include individual test ulps in libm-test-ulps. As recently discussed <https://sourceware.org/ml/libc-alpha/2014-02/msg00670.html>, it doesn't seem particularly useful for libm-test-ulps files to contain huge amounts of data on ulps for individual tests; just the global maximum observed ulps for each function, together with the verification of exceptions, errno and special results such as infinities and NaNs for each test, suffices to verify that a function's behavior on the given test inputs is within the expected accuracy. Removing this data reduces source tree churn caused by updates to these files when libm tests are added, and reduces the frequency with which testsuite additions actually need libm-test-ulps changes at all. Accordingly, this patch removes that data, so that individual tests get checked against the global bounds for the given function and only generate an error if those are exceeded. Tested x86_64 (including verifying that if an ulps value is artificially reduced, the tests do indeed fail as they should and "make regen-ulps" generates the expected changes). * math/libm-test.inc (struct ulp_data): Don't refer to ulps for individual tests in comment. (libm-test-ulps.h): Don't refer to test_ulps in #include comment. (prev_max_error): New variable. (prev_real_max_error): Likewise. (prev_imag_max_error): Likewise. (compare_ulp_data): Don't refer to test names in comment. (find_test_ulps): Remove function. (find_function_ulps): Likewise. (find_complex_function_ulps): Likewise. (init_max_error): Take function name as argument. Look up ulps for that function. (print_ulps): Remove function. (print_max_error): Use prev_max_error instead of calling find_function_ulps. (print_complex_max_error): Use prev_real_max_error and prev_imag_max_error instead of calling find_complex_function_ulps. (check_float_internal): Take max_ulp parameter instead of calling find_test_ulps. Don't call print_ulps. (check_float): Update call to check_float_internal. (check_complex): Update calls to check_float_internal. (START): Pass argument to init_max_error. * math/gen-libm-test.pl (%results): Don't include "kind" information. (parse_ulps): Don't handle ulps of individual tests. (print_ulps_file): Likewise. (output_ulps): Likewise. * math/README.libm-test: Update. * manual/libm-err-tab.pl (parse_ulps): Don't handle ulps of individual tests. * sysdeps/aarch64/libm-test-ulps: Remove individual test ulps. * sysdeps/alpha/fpu/libm-test-ulps: Likewise. * sysdeps/arm/libm-test-ulps: Likewise. * sysdeps/i386/fpu/libm-test-ulps: Likewise. * sysdeps/ia64/fpu/libm-test-ulps: Likewise. * sysdeps/m68k/coldfire/fpu/libm-test-ulps: Likewise. * sysdeps/m68k/m680x0/fpu/libm-test-ulps: Likewise. * sysdeps/microblaze/libm-test-ulps: Likewise. * sysdeps/mips/mips32/libm-test-ulps: Likewise. * sysdeps/mips/mips64/libm-test-ulps: Likewise. * sysdeps/powerpc/fpu/libm-test-ulps: Likewise. * sysdeps/powerpc/nofpu/libm-test-ulps: Likewise. * sysdeps/s390/fpu/libm-test-ulps: Likewise. * sysdeps/sh/libm-test-ulps: Likewise. * sysdeps/sparc/fpu/libm-test-ulps: Likewise. * sysdeps/tile/libm-test-ulps: Likewise. * sysdeps/x86_64/fpu/libm-test-ulps: Likewise. * sysdeps/hppa/fpu/libm-test-ulps: Remove individual test ulps.
2014-03-05 16:02:38 +01:00
Since libm-test-ulps can be generated automatically, just a few notes.
The file contains lines for maximal errors of single functions, like:
Function "yn":
Don't include individual test ulps in libm-test-ulps. As recently discussed <https://sourceware.org/ml/libc-alpha/2014-02/msg00670.html>, it doesn't seem particularly useful for libm-test-ulps files to contain huge amounts of data on ulps for individual tests; just the global maximum observed ulps for each function, together with the verification of exceptions, errno and special results such as infinities and NaNs for each test, suffices to verify that a function's behavior on the given test inputs is within the expected accuracy. Removing this data reduces source tree churn caused by updates to these files when libm tests are added, and reduces the frequency with which testsuite additions actually need libm-test-ulps changes at all. Accordingly, this patch removes that data, so that individual tests get checked against the global bounds for the given function and only generate an error if those are exceeded. Tested x86_64 (including verifying that if an ulps value is artificially reduced, the tests do indeed fail as they should and "make regen-ulps" generates the expected changes). * math/libm-test.inc (struct ulp_data): Don't refer to ulps for individual tests in comment. (libm-test-ulps.h): Don't refer to test_ulps in #include comment. (prev_max_error): New variable. (prev_real_max_error): Likewise. (prev_imag_max_error): Likewise. (compare_ulp_data): Don't refer to test names in comment. (find_test_ulps): Remove function. (find_function_ulps): Likewise. (find_complex_function_ulps): Likewise. (init_max_error): Take function name as argument. Look up ulps for that function. (print_ulps): Remove function. (print_max_error): Use prev_max_error instead of calling find_function_ulps. (print_complex_max_error): Use prev_real_max_error and prev_imag_max_error instead of calling find_complex_function_ulps. (check_float_internal): Take max_ulp parameter instead of calling find_test_ulps. Don't call print_ulps. (check_float): Update call to check_float_internal. (check_complex): Update calls to check_float_internal. (START): Pass argument to init_max_error. * math/gen-libm-test.pl (%results): Don't include "kind" information. (parse_ulps): Don't handle ulps of individual tests. (print_ulps_file): Likewise. (output_ulps): Likewise. * math/README.libm-test: Update. * manual/libm-err-tab.pl (parse_ulps): Don't handle ulps of individual tests. * sysdeps/aarch64/libm-test-ulps: Remove individual test ulps. * sysdeps/alpha/fpu/libm-test-ulps: Likewise. * sysdeps/arm/libm-test-ulps: Likewise. * sysdeps/i386/fpu/libm-test-ulps: Likewise. * sysdeps/ia64/fpu/libm-test-ulps: Likewise. * sysdeps/m68k/coldfire/fpu/libm-test-ulps: Likewise. * sysdeps/m68k/m680x0/fpu/libm-test-ulps: Likewise. * sysdeps/microblaze/libm-test-ulps: Likewise. * sysdeps/mips/mips32/libm-test-ulps: Likewise. * sysdeps/mips/mips64/libm-test-ulps: Likewise. * sysdeps/powerpc/fpu/libm-test-ulps: Likewise. * sysdeps/powerpc/nofpu/libm-test-ulps: Likewise. * sysdeps/s390/fpu/libm-test-ulps: Likewise. * sysdeps/sh/libm-test-ulps: Likewise. * sysdeps/sparc/fpu/libm-test-ulps: Likewise. * sysdeps/tile/libm-test-ulps: Likewise. * sysdeps/x86_64/fpu/libm-test-ulps: Likewise. * sysdeps/hppa/fpu/libm-test-ulps: Remove individual test ulps.
2014-03-05 16:02:38 +01:00
idouble: 6
The keywords are float, ifloat, double, idouble, ldouble and ildouble
(the prefix i stands for inline).
Adding tests to libm-test.inc
=============================
The tests are evaluated by a set of special test macros. The macros
start with "TEST_" followed by a specification the input values, an
underscore and a specification of the output values. As an example,
the test macro for a function with input of type FLOAT (FLOAT is
either float, double, long double) and output of type FLOAT is
"TEST_f_f". The macro's parameter are the name of the function, the
input parameter, output parameter and optionally one exception
parameter.
The accepted parameter types are:
- "f" for FLOAT
- "b" for boolean - just tests if the output parameter evaluates to 0
or 1 (only for output).
- "c" for complex. This parameter needs two values, first the real,
then the imaginary part.
- "i" for int.
- "l" for long int.
- "L" for long long int.
- "F" for the address of a FLOAT (only as input parameter)
- "I" for the address of an int (only as input parameter)