Speedup tanf range reduction by using the new sincosf range
reduction algorithm. Overall code quality is improved due to
inlining, so there is a speedup even if no range reduction is
required.
tanf throughput gains on Cortex-A72:
* |x| < M_PI_4 : 1.1x
* |x| < M_PI_2 : 1.2x
* |x| < 2 * M_PI: 1.5x
* |x| < 120.0 : 1.6x
* |x| < Inf : 12.1x
* sysdeps/ieee754/flt-32/s_tanf.c (__tanf): Use fast range reduction.
The internal functions __kernel_sinf and __kernel_cosf are used only by
lgammaf_r. Removing the internal functions and using the generic sinf
and cosf is better overall. Benchmarking on Cortex-A72 shows the generic
sinf and cosf are 1.4x and 2.3x faster in the range |x| < PI/4, and 0.66x
and 1.1x for |x| < PI/2, so it should make lgammaf_r faster on average.
GLIBC regression tests pass on AArch64.
* sysdeps/ieee754/flt-32/e_lgammaf_r.c (sin_pif): Use __sinf/__cosf.
* sysdeps/ieee754/flt-32/k_cosf.c (__kernel_cosf): Remove all code.
* sysdeps/ieee754/flt-32/k_sinf.c (__kernel_sinf): Likewise.
The second patch improves performance of sinf and cosf using the same
algorithms and polynomials. The returned values are identical to sincosf
for the same input. ULP definitions for AArch64 and x64 are updated.
sinf/cosf througput gains on Cortex-A72:
* |x| < 0x1p-12 : 1.2x
* |x| < M_PI_4 : 1.8x
* |x| < 2 * M_PI: 1.7x
* |x| < 120.0 : 2.3x
* |x| < Inf : 3.0x
* NEWS: Mention sinf, cosf, sincosf.
* sysdeps/aarch64/libm-test-ulps: Update ULP for sinf, cosf, sincosf.
* sysdeps/x86_64/fpu/libm-test-ulps: Update ULP for sinf and cosf.
* sysdeps/x86_64/fpu/multiarch/s_sincosf-fma.c: Add definitions of
constants rather than including generic sincosf.h.
* sysdeps/x86_64/fpu/s_sincosf_data.c: Remove.
* sysdeps/ieee754/flt-32/s_cosf.c (cosf): Rewrite.
* sysdeps/ieee754/flt-32/s_sincosf.h (reduced_sin): Remove.
(reduced_cos): Remove.
(sinf_poly): New function.
* sysdeps/ieee754/flt-32/s_sinf.c (sinf): Rewrite.
This patch is a complete rewrite of sincosf. The new version is
significantly faster, as well as simple and accurate.
The worst-case ULP is 0.5607, maximum relative error is 0.5303 * 2^-23 over
all 4 billion inputs. In non-nearest rounding modes the error is 1ULP.
The algorithm uses 3 main cases: small inputs which don't need argument
reduction, small inputs which need a simple range reduction and large inputs
requiring complex range reduction. The code uses approximate integer
comparisons to quickly decide between these cases.
The small range reducer uses a single reduction step to handle values up to
120.0. It is fastest on targets which support inlined round instructions.
The large range reducer uses integer arithmetic for simplicity. It does a
32x96 bit multiply to compute a 64-bit modulo result. This is more than
accurate enough to handle the worst-case cancellation for values close to
an integer multiple of PI/4. It could be further optimized, however it is
already much faster than necessary.
sincosf throughput gains on Cortex-A72:
* |x| < 0x1p-12 : 1.6x
* |x| < M_PI_4 : 1.7x
* |x| < 2 * M_PI: 1.5x
* |x| < 120.0 : 1.8x
* |x| < Inf : 2.3x
* math/Makefile: Add s_sincosf_data.c.
* sysdeps/ia64/fpu/s_sincosf_data.c: New file.
* sysdeps/ieee754/flt-32/s_sincosf.h (abstop12): Add new function.
(sincosf_poly): Likewise.
(reduce_small): Likewise.
(reduce_large): Likewise.
* sysdeps/ieee754/flt-32/s_sincosf.c (sincosf): Rewrite.
* sysdeps/ieee754/flt-32/s_sincosf_data.c: New file with sincosf data.
* sysdeps/m68k/m680x0/fpu/s_sincosf_data.c: New file.
* sysdeps/x86_64/fpu/s_sincosf_data.c: New file.
This patch currently only affects aarch64.
The roundtoint and converttoint internal functions are only called with small
values, so 32 bit result is enough for converttoint and it is a signed int
conversion so the return type is changed to int32_t.
The original idea was to help the compiler keeping the result in uint64_t,
then it's clear that no sign extension is needed and there is no accidental
undefined or implementation defined signed int arithmetics.
But it turns out gcc does a good job with inlining so changing the type has
no overhead and the semantics of the conversion is less surprising this way.
Since we want to allow the asuint64 (x + 0x1.8p52) style conversion, the top
bits were never usable and the existing code ensures that only the bottom
32 bits of the conversion result are used.
On aarch64 the neon intrinsics (which round ties to even) are changed to
round and lround (which round ties away from zero) this does not affect the
results in a significant way, but more portable (relies on round and lround
being inlined which works with -fno-math-errno).
The TOINT_SHIFT and TOINT_RINT macros were removed, only keep separate code
paths for TOINT_INTRINSICS and !TOINT_INTRINSICS.
* sysdeps/aarch64/fpu/math_private.h (roundtoint): Use round.
(converttoint): Use lround.
* sysdeps/ieee754/flt-32/math_config.h (roundtoint): Declare and
document the semantics when TOINT_INTRINSICS is set.
(converttoint): Likewise.
(TOINT_RINT): Remove.
(TOINT_SHIFT): Remove.
* sysdeps/ieee754/flt-32/e_expf.c (__expf): Remove the TOINT_RINT code
path.
Since the addition of the _Float128 API, strfromf128 and printf_size use
__printf_fp to print _Float128 values. This is achieved by setting the
'is_binary128' member of the 'printf_info' structure to one. Now that
the format of long double on powerpc64le is getting a third option, this
mechanism is reused for long double values that have binary128 format
(i.e.: when -mabi=ieeelongdouble).
This patch adds __printf_sizeieee128 as an exported symbol, but doesn't
provide redirections from printf_size, yet. All redirections will be
installed in a future commit, once all other functions that print or
read long double values with binary128 format are ready. In
__printf_fp, when 'is_binary128' is one, the floating-point argument is
treated as if it was of _Float128 type, regardless of the value of
'is_long_double', thus __printf_sizeieee128 sets 'is_binary128' to the
same value of 'is_long_double'. Otherwise, double values would not be
printed correctly.
Tested for powerpc64le.
Ideally sign should be bool, but sometimes (e.g. in powf) it's more
efficient to pass a non-zero value than 1 to indicate that the sign
should be set. The fixed size int is less ambigous than unsigned
long.
* sysdeps/ieee754/flt-32/e_powf.c (__powf): Use uint32_t.
(exp2f_inline): Likewise.
* sysdeps/ieee754/flt-32/math_config.h (__math_oflowf): Likewise.
(__math_uflowf): Likewise.
(__math_may_uflowf): Likewise.
(__math_divzerof): Likewise.
(__math_invalidf): Likewise.
* sysdeps/ieee754/flt-32/math_errf.c (xflowf): Likewise.
(__math_oflowf): Likewise.
(__math_uflowf): Likewise.
(__math_may_uflowf): Likewise.
(__math_divzerof): Likewise.
(__math_invalidf): Likewise.
This patch adds __*ieee128 symbols for strfrom, strtold, strtold_l, wcstold
and wcstold_l functions. Redirection from *l to *ieee128 will be handled
in separate patch once we start building these new files.
2018-06-28 Rajalakshmi Srinivasaraghavan <raji@linux.vnet.ibm.com>
* sysdeps/ieee754/ldbl-128ibm-compat/Versions: Add __strfromieee128,
__strtoieee128, __strtoieee128_l,__wcstoieee128 and __wcstoieee128_l.
* sysdeps/ieee754/ldbl-128ibm-compat/strfromf128.c: New file.
* sysdeps/ieee754/ldbl-128ibm-compat/strtof128.c: New file.
* sysdeps/ieee754/ldbl-128ibm-compat/strtof128_l.c: New file.
* sysdeps/ieee754/ldbl-128ibm-compat/wcstof128.c: New file.
* sysdeps/ieee754/ldbl-128ibm-compat/wcstof128_l.c: New file.
Add a new libm-alias-float128.h in order to provide the __*ieee128
aliases for the existing *f128 that do not have a globally exported
symbol.
* sysdeps/ieee754/ldbl-128ibm-compat/Versions: New file.
* sysdeps/ieee754/ldbl-128ibm-compat/libm-alias-float128.h: New file.
As reported in bug 23272, the ldbl-96 implementation of fma (fma for
double, in terms of ldbl-96 as the internal arithmetic type, as used
on 32-bit x86) is missing some of the special-case handling for
non-finite arguments, resulting in incorrect NaN results when the
first two arguments are infinities, the third is finite and so the
infinities go through the logic for finite arguments. This patch
fixes it by handling all cases of non-finite arguments up front, with
additional fma tests for the problem cases being added to the
testsuite.
Tested for x86_64 and x86.
[BZ #23272]
* sysdeps/ieee754/ldbl-96/s_fma.c (__fma): Start by handling all
cases of non-finite arguments.
* math/libm-test-fma.inc (fma_test_data): Add more tests.
Building with recent GCC mainline for i686-linux-gnu is failing with:
../sysdeps/ieee754/flt-32/k_rem_pio2f.c: In function '__kernel_rem_pio2f':
../sysdeps/ieee754/flt-32/k_rem_pio2f.c:186:28: error: 'fq[0]' may be used uninitialized in this function [-Werror=maybe-uninitialized]
fv = math_narrow_eval (fq[0]-fv);
^
and
../sysdeps/ieee754/dbl-64/k_rem_pio2.c: In function '__kernel_rem_pio2':
../sysdeps/ieee754/dbl-64/k_rem_pio2.c:333:32: error: 'fq[0]' may be used uninitialized in this function [-Werror=maybe-uninitialized]
fv = math_narrow_eval (fq[0] - fv);
^
These are similar to -Warray-bounds cases for which the DIAG_* macros
are already used in those files: the array element is in fact always
initialized, but the reasoning that it is depends on another array not
having been all zero at an earlier point, which depends on the
functions not being called with zero arguments. Thus, this patch uses
DIAG_* to disable -Wmaybe-uninitialized for this code.
(The warning may be i686-specific because of math_narrow_eval somehow
perturbing what the compiler does with this code enough to cause the
warning. I don't know why it doesn't appear for i686-gnu.)
Tested with build-many-glibcs.py that this fixes the i686 build in
this configuration.
* sysdeps/ieee754/dbl-64/k_rem_pio2.c (__kernel_rem_pio2): Ignore
-Wmaybe-uninitialized around access to fq[0].
* sysdeps/ieee754/flt-32/k_rem_pio2f.c (__kernel_rem_pio2f):
Likewise.
This patch continues the math_private.h cleanup by stopping
math_private.h from including math-barriers.h and making the users of
the barrier macros include the latter header directly. No attempt is
made to remove any math_private.h includes that are now unused, except
in strtod_l.c where that is done to avoid line number changes in
assertions, so that installed stripped shared libraries can be
compared before and after the patch. (I think the floating-point
environment support in math_private.h should also move out - some
architectures already have fenv_private.h as an architecture-internal
header included from their math_private.h - and after moving that out
might be a better time to identify unused math_private.h includes.)
Tested for x86_64 and x86, and tested with build-many-glibcs.py that
installed stripped shared libraries are unchanged by the patch.
* sysdeps/generic/math_private.h: Do not include
<math-barriers.h>.
* stdlib/strtod_l.c: Include <math-barriers.h> instead of
<math_private.h>.
* math/fromfp.h: Include <math-barriers.h>.
* math/math-narrow.h: Likewise.
* math/s_nextafter.c: Likewise.
* math/s_nexttowardf.c: Likewise.
* sysdeps/aarch64/fpu/s_llrint.c: Likewise.
* sysdeps/aarch64/fpu/s_llrintf.c: Likewise.
* sysdeps/aarch64/fpu/s_lrint.c: Likewise.
* sysdeps/aarch64/fpu/s_lrintf.c: Likewise.
* sysdeps/i386/fpu/s_nextafterl.c: Likewise.
* sysdeps/i386/fpu/s_nexttoward.c: Likewise.
* sysdeps/i386/fpu/s_nexttowardf.c: Likewise.
* sysdeps/ieee754/dbl-64/e_atan2.c: Likewise.
* sysdeps/ieee754/dbl-64/e_atanh.c: Likewise.
* sysdeps/ieee754/dbl-64/e_exp.c: Likewise.
* sysdeps/ieee754/dbl-64/e_exp2.c: Likewise.
* sysdeps/ieee754/dbl-64/e_j0.c: Likewise.
* sysdeps/ieee754/dbl-64/e_sqrt.c: Likewise.
* sysdeps/ieee754/dbl-64/s_expm1.c: Likewise.
* sysdeps/ieee754/dbl-64/s_fma.c: Likewise.
* sysdeps/ieee754/dbl-64/s_fmaf.c: Likewise.
* sysdeps/ieee754/dbl-64/s_log1p.c: Likewise.
* sysdeps/ieee754/dbl-64/s_nearbyint.c: Likewise.
* sysdeps/ieee754/dbl-64/wordsize-64/s_nearbyint.c: Likewise.
* sysdeps/ieee754/flt-32/e_atanhf.c: Likewise.
* sysdeps/ieee754/flt-32/e_j0f.c: Likewise.
* sysdeps/ieee754/flt-32/s_expm1f.c: Likewise.
* sysdeps/ieee754/flt-32/s_log1pf.c: Likewise.
* sysdeps/ieee754/flt-32/s_nearbyintf.c: Likewise.
* sysdeps/ieee754/flt-32/s_nextafterf.c: Likewise.
* sysdeps/ieee754/k_standardl.c: Likewise.
* sysdeps/ieee754/ldbl-128/e_asinl.c: Likewise.
* sysdeps/ieee754/ldbl-128/e_expl.c: Likewise.
* sysdeps/ieee754/ldbl-128/e_powl.c: Likewise.
* sysdeps/ieee754/ldbl-128/s_fmal.c: Likewise.
* sysdeps/ieee754/ldbl-128/s_nearbyintl.c: Likewise.
* sysdeps/ieee754/ldbl-128/s_nextafterl.c: Likewise.
* sysdeps/ieee754/ldbl-128/s_nexttoward.c: Likewise.
* sysdeps/ieee754/ldbl-128/s_nexttowardf.c: Likewise.
* sysdeps/ieee754/ldbl-128ibm/e_asinl.c: Likewise.
* sysdeps/ieee754/ldbl-128ibm/s_fmal.c: Likewise.
* sysdeps/ieee754/ldbl-128ibm/s_nextafterl.c: Likewise.
* sysdeps/ieee754/ldbl-128ibm/s_nexttoward.c: Likewise.
* sysdeps/ieee754/ldbl-128ibm/s_nexttowardf.c: Likewise.
* sysdeps/ieee754/ldbl-128ibm/s_rintl.c: Likewise.
* sysdeps/ieee754/ldbl-96/e_atanhl.c: Likewise.
* sysdeps/ieee754/ldbl-96/e_j0l.c: Likewise.
* sysdeps/ieee754/ldbl-96/s_fma.c: Likewise.
* sysdeps/ieee754/ldbl-96/s_fmal.c: Likewise.
* sysdeps/ieee754/ldbl-96/s_nexttoward.c: Likewise.
* sysdeps/ieee754/ldbl-96/s_nexttowardf.c: Likewise.
* sysdeps/ieee754/ldbl-opt/s_nexttowardfd.c: Likewise.
* sysdeps/m68k/m680x0/fpu/s_nextafterl.c: Likewise.
This patch continues cleaning up the math_private.h header, which
contains lots of different definitions many of which are only needed
by a limited subset of files using that header (and some of which are
overridden by architectures that only want to override selected parts
of the header), by moving the math_narrow_eval macro out to a separate
math-narrow-eval.h header, only included by those files that need it.
That header is placed in include/ (since it's used in stdlib/, not
just files built in math/, but no sysdeps variants are needed at
present).
Tested for x86_64, and with build-many-glibcs.py. (Installed stripped
shared libraries change because of line numbers in assertions in
strtod_l.c.)
* include/math-narrow-eval.h: New file. Contents moved from ....
* sysdeps/generic/math_private.h: ... here.
(math_narrow_eval): Remove macro. Moved to math-narrow-eval.h.
[FLT_EVAL_METHOD != 0] (excess_precision): Likewise.
* math/s_fdim_template.c: Include <math-narrow-eval.h>.
* stdlib/strtod_l.c: Likewise.
* sysdeps/i386/fpu/s_f32xaddf64.c: Likewise.
* sysdeps/i386/fpu/s_f32xsubf64.c: Likewise.
* sysdeps/i386/fpu/s_fdim.c: Likewise.
* sysdeps/ieee754/dbl-64/e_cosh.c: Likewise.
* sysdeps/ieee754/dbl-64/e_gamma_r.c: Likewise.
* sysdeps/ieee754/dbl-64/e_j1.c: Likewise.
* sysdeps/ieee754/dbl-64/e_jn.c: Likewise.
* sysdeps/ieee754/dbl-64/e_lgamma_r.c: Likewise.
* sysdeps/ieee754/dbl-64/e_sinh.c: Likewise.
* sysdeps/ieee754/dbl-64/gamma_productf.c: Likewise.
* sysdeps/ieee754/dbl-64/k_rem_pio2.c: Likewise.
* sysdeps/ieee754/dbl-64/lgamma_neg.c: Likewise.
* sysdeps/ieee754/dbl-64/s_erf.c: Likewise.
* sysdeps/ieee754/dbl-64/s_llrint.c: Likewise.
* sysdeps/ieee754/dbl-64/s_lrint.c: Likewise.
* sysdeps/ieee754/flt-32/e_coshf.c: Likewise.
* sysdeps/ieee754/flt-32/e_exp2f.c: Likewise.
* sysdeps/ieee754/flt-32/e_expf.c: Likewise.
* sysdeps/ieee754/flt-32/e_gammaf_r.c: Likewise.
* sysdeps/ieee754/flt-32/e_j1f.c: Likewise.
* sysdeps/ieee754/flt-32/e_jnf.c: Likewise.
* sysdeps/ieee754/flt-32/e_lgammaf_r.c: Likewise.
* sysdeps/ieee754/flt-32/e_sinhf.c: Likewise.
* sysdeps/ieee754/flt-32/k_rem_pio2f.c: Likewise.
* sysdeps/ieee754/flt-32/lgamma_negf.c: Likewise.
* sysdeps/ieee754/flt-32/s_erff.c: Likewise.
* sysdeps/ieee754/flt-32/s_llrintf.c: Likewise.
* sysdeps/ieee754/flt-32/s_lrintf.c: Likewise.
* sysdeps/ieee754/ldbl-96/gamma_product.c: Likewise.
Adds a fast path to e_exp.c when |x| < 1.03972053527832.
When values are tested in isolation, reduction in execution
time is: aarch 30%, sparc 18%, x86 37%.
When comparing benchtests/bench.out which includes values
outside that range, the gains are:
aarch 8%, sparc 5%, x86 9%.
make check is clean (no increase in ulp for any math test).
Testing 20M values for each rounding mode in that range shows
approximately one in 200 values is off by 1 ulp. No value tested
for exp(x) changed by 2 or more ulp.
No observed change in performance or accuracy for x outside
fast path range.
These changes will be active for all platforms that don't provide
their own exp() routines. They will also be active for ieee754
versions of ccos, ccosh, cosh, csin, csinh, sinh, exp10, gamma, and
erf.
Refactor the sincos implementation - rather than rely on odd partial inlining
of preprocessed portions from sin and cos, explicitly write out the cases.
This makes sincos much easier to maintain and provides an additional 16-20%
speedup between 0 and 2^27. The overall speedup of sincos is 48% over this range.
Between 0 and PI it is 66% faster.
* sysdeps/ieee754/dbl-64/s_sin.c (__sin): Cleanup ifdefs.
(__cos): Likewise.
* sysdeps/ieee754/dbl-64/s_sin.c (__sincos): Refactor using the same
logic as sin and cos.
Refactor duplicated code into do_sin. Since all calls to do_sin use copysign to
set the sign of the result, move it inside do_sin. Small inputs use a separate
polynomial, so move this into do_sin as well (the check is based on the more
conservative case when doing large range reduction, but could be relaxed).
* sysdeps/ieee754/dbl-64/s_sin.c (do_sin): Use TAYLOR_SIN for small
inputs. Return correct sign.
(do_sincos): Remove small input check before do_sin, let do_sin set
the sign.
(__sin): Likewise.
(__cos): Likewise.
For huge inputs use the improved do_sincos function as well. Now no cases use
the correction factor returned by do_sin, do_cos and TAYLOR_SIN, so remove it.
* sysdeps/ieee754/dbl-64/s_sin.c (TAYLOR_SIN): Remove cor parameter.
(do_cos): Remove corp parameter and calculations.
(do_sin): Likewise.
(do_sincos): Remove cor variable.
(__sin): Use do_sincos for huge inputs.
(__cos): Likewise.
* sysdeps/ieee754/dbl-64/s_sincos.c (__sincos): Likewise.
(reduce_and_compute_sincos): Remove unused function.
This patch improves the accuracy of the range reduction. When the input is
large (2^27) and very close to a multiple of PI/2, using 110 bits of PI is not
enough. Improve range reduction accuracy to 136 bits. As a result the special
checks for results close to zero can be removed. The ULP of the polynomials is
at worst 0.55ULP, so there is no reason for the slow functions, and they can be
removed.
* sysdeps/ieee754/dbl-64/s_sin.c (reduce_sincos_1): Rename to
reduce_sincos, improve accuracy to 136 bits.
(do_sincos_1): Rename to do_sincos, remove fallbacks to slow functions.
(__sin): Use improved reduction and simplified do_sincos calculation.
(__cos): Likewise.
* sysdeps/ieee754/dbl-64/s_sincos.c (__sincos): Likewise.
This patch removes the large range reduction code and defers to the huge range
reduction code. The first level range reducer supports inputs up to 2^27,
which is way too large given that inputs for sin/cos are typically small
(< 10), and optimizing for a smaller range would give a significant speedup.
Input values above 2^27 are practically never used, so there is no reason for
supporting range reduction between 2^27 and 2^48. Removing it significantly
simplifies code and enables further speedups. There is about a 2.3x slowdown
in this range due to __branred being extremely slow (a better algorithm could
easily more than double performance).
* sysdeps/ieee754/dbl-64/s_sin.c (reduce_sincos_2): Remove function.
(do_sincos_2): Likewise.
(__sin): Remove middle range reduction case.
(__cos): Likewise.
* sysdeps/ieee754/dbl-64/s_sincos.c (__sincos): Remove middle range
reduction case.
This series of patches removes the slow patchs from sin, cos and sincos.
Besides greatly simplifying the implementation, the new version is also much
faster for inputs up to PI (41% faster) and for large inputs needing range
reduction (27% faster).
ULP is ~0.55 with no errors found after testing 1.6 billion inputs across most
of the range with mpsin and mpcos. The number of incorrectly rounded results
(ie. ULP >0.5) is at most ~2750 per million inputs between 0.125 and 0.5,
the average is ~850 per million between 0 and PI.
Tested on AArch64 and x86_64 with no regressions.
The first patch removes the slow paths for the cases where the input is small
and doesn't require range reduction. Update ULP tables for sin, cos and sincos
on AArch64 and x86_64.
* sysdeps/aarch64/libm-test-ulps: Update ULP for sin, cos, sincos.
* sysdeps/ieee754/dbl-64/s_sin.c (__sin): Remove slow paths for small
inputs.
(__cos): Likewise.
* sysdeps/x86_64/fpu/libm-test-ulps: Update ULP for sin, cos, sincos.
Jeff Law noticed that native PowerPC builds were broken by my having
made math_ldbl_opt.h not include math.h. nldbl-compat.c formerly got
math.h via libioP.h and math_ldbl_opt.h, *without* __NO_LONG_DOUBLE_MATH;
after my change it got it via nldbl-compat.h *with* __NO_LONG_DOUBLE_MATH,
but __NO_LONG_DOUBLE_MATH mode is forbidden on hosts that define
__HAVE_DISTINCT_FLOAT128, so the build breaks. This is the quick fix.
* sysdeps/ieee754/ldbl-opt/nldbl-compat.c: Include math.h
before nldbl-compat.h.
The sysdeps/ieee754/ldbl-opt version of math_ldbl_opt.h includes
math.h and math_private.h, despite not having any need for those
headers itself; the sysdeps/generic version doesn't. About 20 files
are relying on math_ldbl_opt.h to include math.h and/or math_private.h
for them, even though none of them necessarily used on a platform that
needs ldbl-opt support.
* sysdeps/ieee754/ldbl-opt/math_ldbl_opt.h: Don't include
math.h or math_private.h.
* sysdeps/alpha/fpu/s_isnan.c
* sysdeps/ieee754/ldbl-128ibm/s_ceill.c
* sysdeps/ieee754/ldbl-128ibm/s_floorl.c
* sysdeps/ieee754/ldbl-128ibm/s_llrintl.c
* sysdeps/ieee754/ldbl-128ibm/s_llroundl.c
* sysdeps/ieee754/ldbl-128ibm/s_lrintl.c
* sysdeps/ieee754/ldbl-128ibm/s_lroundl.c
* sysdeps/ieee754/ldbl-128ibm/s_rintl.c
* sysdeps/ieee754/ldbl-128ibm/s_roundl.c
* sysdeps/ieee754/ldbl-128ibm/s_truncl.c
* sysdeps/powerpc/powerpc32/power4/fpu/multiarch/e_hypot.c
* sysdeps/powerpc/powerpc32/power4/fpu/multiarch/e_hypotf.c:
* sysdeps/powerpc/powerpc64/fpu/multiarch/e_expf.c
* sysdeps/powerpc/powerpc64/fpu/multiarch/e_hypot.c
* sysdeps/powerpc/powerpc64/fpu/multiarch/e_hypotf.c:
Include math_private.h.
* sysdeps/ieee754/ldbl-64-128/s_finitel.c
* sysdeps/ieee754/ldbl-64-128/s_fpclassifyl.c
* sysdeps/ieee754/ldbl-64-128/s_isinfl.c
* sysdeps/ieee754/ldbl-64-128/s_isnanl.c
* sysdeps/ieee754/ldbl-64-128/s_signbitl.c
* sysdeps/powerpc/power7/fpu/s_logb.c:
Include math.h and math_private.h.
This patch mechanically removes all remaining uses, and the
definitions, of the following libio name aliases:
name replaced with
---- -------------
_IO_FILE FILE
_IO_fpos_t __fpos_t
_IO_fpos64_t __fpos64_t
_IO_size_t size_t
_IO_ssize_t ssize_t or __ssize_t
_IO_off_t off_t
_IO_off64_t off64_t
_IO_pid_t pid_t
_IO_uid_t uid_t
_IO_wint_t wint_t
_IO_va_list va_list or __gnuc_va_list
_IO_BUFSIZ BUFSIZ
_IO_cookie_io_functions_t cookie_io_functions_t
__io_read_fn cookie_read_function_t
__io_write_fn cookie_write_function_t
__io_seek_fn cookie_seek_function_t
__io_close_fn cookie_close_function_t
I used __fpos_t and __fpos64_t instead of fpos_t and fpos64_t because
the definitions of fpos_t and fpos64_t depend on the largefile mode.
I used __ssize_t and __gnuc_va_list in a handful of headers where
namespace cleanliness might be relevant even though they're
internal-use-only. In all other cases, I used the public-namespace
name.
There are a tiny handful of places where I left a use of 'struct _IO_FILE'
alone, because it was being used together with 'struct _IO_FILE_plus'
or 'struct _IO_FILE_complete' in the same arithmetic expression.
Because this patch was almost entirely done with search and replace, I
may have introduced indentation botches. I did proofread the diff,
but I may have missed something.
The ChangeLog below calls out all of the places where this was not a
pure search-and-replace change.
Installed stripped libraries and executables are unchanged by this patch,
except that some assertions in vfscanf.c change line numbers.
* libio/libio.h (_IO_FILE): Delete; all uses changed to FILE.
(_IO_fpos_t): Delete; all uses changed to __fpos_t.
(_IO_fpos64_t): Delete; all uses changed to __fpos64_t.
(_IO_size_t): Delete; all uses changed to size_t.
(_IO_ssize_t): Delete; all uses changed to ssize_t or __ssize_t.
(_IO_off_t): Delete; all uses changed to off_t.
(_IO_off64_t): Delete; all uses changed to off64_t.
(_IO_pid_t): Delete; all uses changed to pid_t.
(_IO_uid_t): Delete; all uses changed to uid_t.
(_IO_wint_t): Delete; all uses changed to wint_t.
(_IO_va_list): Delete; all uses changed to va_list or __gnuc_va_list.
(_IO_BUFSIZ): Delete; all uses changed to BUFSIZ.
(_IO_cookie_io_functions_t): Delete; all uses changed to
cookie_io_functions_t.
(__io_read_fn): Delete; all uses changed to cookie_read_function_t.
(__io_write_fn): Delete; all uses changed to cookie_write_function_t.
(__io_seek_fn): Delete; all uses changed to cookie_seek_function_t.
(__io_close_fn): Delete: all uses changed to cookie_close_function_t.
* libio/iofopncook.c: Remove unnecessary forward declarations.
* libio/iolibio.h: Correct outdated commentary.
* malloc/malloc.c (__malloc_stats): Remove unnecessary casts.
* stdio-common/fxprintf.c (__fxprintf_nocancel):
Remove unnecessary casts.
* stdio-common/getline.c: Use _IO_getdelim directly.
Don't redefine ssize_t.
* stdio-common/printf_fp.c, stdio_common/printf_fphex.c
* stdio-common/printf_size.c: Don't redefine size_t or FILE.
Remove outdated comments.
* stdio-common/vfscanf.c: Don't redefine va_list.
Remove the slow paths from pow. Like several other double precision math
functions, pow is exactly rounded. This is not required from math functions
and causes major overheads as it requires multiple fallbacks using higher
precision arithmetic if a result is close to 0.5ULP. Ridiculous slowdowns
of up to 100000x have been reported when the highest precision path triggers.
All GLIBC math tests pass on AArch64 and x64 (with ULP of pow set to 1).
The worst case error is ~0.506ULP. A simple test over a few hundred million
values shows pow is 10% faster on average. This fixes BZ #13932.
[BZ #13932]
* sysdeps/ieee754/dbl-64/uexp.h (err_1): Remove.
* benchtests/pow-inputs: Update comment for slow path cases.
* manual/probes.texi (slowpow_p10): Delete removed probe.
(slowpow_p10): Likewise.
* math/Makefile: Remove halfulp.c and slowpow.c.
* sysdeps/aarch64/libm-test-ulps: Set ULP of pow to 1.
* sysdeps/generic/math_private.h (__exp1): Remove error argument.
(__halfulp): Remove.
(__slowpow): Remove.
* sysdeps/i386/fpu/halfulp.c: Delete file.
* sysdeps/i386/fpu/slowpow.c: Likewise.
* sysdeps/ia64/fpu/halfulp.c: Likewise.
* sysdeps/ia64/fpu/slowpow.c: Likewise.
* sysdeps/ieee754/dbl-64/e_exp.c (__exp1): Remove error argument,
improve comments and add error analysis.
* sysdeps/ieee754/dbl-64/e_pow.c (__ieee754_pow): Add error analysis.
(power1): Remove function:
(log1): Remove error argument, add error analysis.
(my_log2): Remove function.
* sysdeps/ieee754/dbl-64/halfulp.c: Delete file.
* sysdeps/ieee754/dbl-64/slowpow.c: Likewise.
* sysdeps/m68k/m680x0/fpu/halfulp.c: Likewise.
* sysdeps/m68k/m680x0/fpu/slowpow.c: Likewise.
* sysdeps/powerpc/power4/fpu/Makefile: Remove CPPFLAGS-slowpow.c.
* sysdeps/x86_64/fpu/libm-test-ulps: Set ULP of pow to 1.
* sysdeps/x86_64/fpu/multiarch/Makefile: Remove slowpow-fma.c,
slowpow-fma4.c, halfulp-fma.c, halfulp-fma4.c.
* sysdeps/x86_64/fpu/multiarch/e_pow-fma.c (__slowpow): Remove define.
* sysdeps/x86_64/fpu/multiarch/e_pow-fma4.c (__slowpow): Likewise.
* sysdeps/x86_64/fpu/multiarch/halfulp-fma.c: Delete file.
* sysdeps/x86_64/fpu/multiarch/halfulp-fma4.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/slowpow-fma.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/slowpow-fma4.c: Likewise.
This patch adds the narrowing add functions from TS 18661-1 to glibc's
libm: fadd, faddl, daddl, f32addf64, f32addf32x, f32xaddf64 for all
configurations; f32addf64x, f32addf128, f64addf64x, f64addf128,
f32xaddf64x, f32xaddf128, f64xaddf128 for configurations with
_Float64x and _Float128; __nldbl_daddl for ldbl-opt. As discussed for
the build infrastructure patch, tgmath.h support is deliberately
deferred, and FP_FAST_* macros are not applicable without optimized
function implementations.
Function implementations are added for all relevant pairs of formats
(including certain cases of a format and itself where more than one
type has that format). The main implementations use round-to-odd, or
a trivial computation in the case where both formats are the same or
where the wider format is IBM long double (in which case we don't
attempt to be correctly rounding). The sysdeps/ieee754/soft-fp
implementations use soft-fp, and are used automatically for
configurations without exceptions and rounding modes by virtue of
existing Implies files. As previously discussed, optimized versions
for particular architectures are possible, but not included.
i386 gets a special version of f32xaddf64 to avoid problems with
double rounding (similar to the existing fdim version), since this
function must round just once without an intermediate rounding to long
double. (No such special version is needed for any other function,
because the nontrivial functions use round-to-odd, which does the
intermediate computation with the rounding mode set to round-to-zero,
and double rounding is OK except in round-to-nearest mode, so is OK
for that intermediate round-to-zero computation.) mul and div will
need slightly different special versions for i386 (using round-to-odd
on long double instead of precision control) because of the
possibility of inexact intermediate results in the subnormal range for
double.
To reduce duplication among the different function implementations,
math-narrow.h gets macros CHECK_NARROW_ADD, NARROW_ADD_ROUND_TO_ODD
and NARROW_ADD_TRIVIAL.
In the trivial cases and for any architecture-specific optimized
implementations, the overhead of the errno setting might be
significant, but I think that's best handled through compiler built-in
functions rather than providing separate no-errno versions in glibc
(and likewise there are no __*_finite entry points for these function
provided, __*_finite effectively being no-errno versions at present in
most cases).
Tested for x86_64 and x86, with both GCC 6 and GCC 7. Tested for
mips64 (all three ABIs, both hard and soft float) and powerpc with GCC
7. Tested with build-many-glibcs.py with both GCC 6 and GCC 7.
* math/Makefile (libm-narrow-fns): Add add.
(libm-test-funcs-narrow): Likewise.
* math/Versions (GLIBC_2.28): Add narrowing add functions.
* math/bits/mathcalls-narrow.h (add): Use __MATHCALL_NARROW .
* math/gen-auto-libm-tests.c (test_functions): Add add.
* math/math-narrow.h (CHECK_NARROW_ADD): New macro.
(NARROW_ADD_ROUND_TO_ODD): Likewise.
(NARROW_ADD_TRIVIAL): Likewise.
* sysdeps/ieee754/float128/float128_private.h (__faddl): New
macro.
(__daddl): Likewise.
* sysdeps/ieee754/ldbl-opt/Makefile (libnldbl-calls): Add fadd and
dadd.
(CFLAGS-nldbl-dadd.c): New variable.
(CFLAGS-nldbl-fadd.c): Likewise.
* sysdeps/ieee754/ldbl-opt/Versions (GLIBC_2.28): Add
__nldbl_daddl.
* sysdeps/ieee754/ldbl-opt/nldbl-compat.h (__nldbl_daddl): New
prototype.
* manual/arith.texi (Misc FP Arithmetic): Document fadd, faddl,
daddl, fMaddfN, fMaddfNx, fMxaddfN and fMxaddfNx.
* math/auto-libm-test-in: Add tests of add.
* math/auto-libm-test-out-narrow-add: New generated file.
* math/libm-test-narrow-add.inc: New file.
* sysdeps/i386/fpu/s_f32xaddf64.c: Likewise.
* sysdeps/ieee754/dbl-64/s_f32xaddf64.c: Likewise.
* sysdeps/ieee754/dbl-64/s_fadd.c: Likewise.
* sysdeps/ieee754/float128/s_f32addf128.c: Likewise.
* sysdeps/ieee754/float128/s_f64addf128.c: Likewise.
* sysdeps/ieee754/float128/s_f64xaddf128.c: Likewise.
* sysdeps/ieee754/ldbl-128/s_daddl.c: Likewise.
* sysdeps/ieee754/ldbl-128/s_f64xaddf128.c: Likewise.
* sysdeps/ieee754/ldbl-128/s_faddl.c: Likewise.
* sysdeps/ieee754/ldbl-128ibm/s_daddl.c: Likewise.
* sysdeps/ieee754/ldbl-128ibm/s_faddl.c: Likewise.
* sysdeps/ieee754/ldbl-96/s_daddl.c: Likewise.
* sysdeps/ieee754/ldbl-96/s_faddl.c: Likewise.
* sysdeps/ieee754/ldbl-opt/nldbl-dadd.c: Likewise.
* sysdeps/ieee754/ldbl-opt/nldbl-fadd.c: Likewise.
* sysdeps/ieee754/soft-fp/s_daddl.c: Likewise.
* sysdeps/ieee754/soft-fp/s_fadd.c: Likewise.
* sysdeps/ieee754/soft-fp/s_faddl.c: Likewise.
* sysdeps/powerpc/fpu/libm-test-ulps: Update.
* sysdeps/mach/hurd/i386/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/aarch64/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/alpha/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/arm/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/hppa/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/i386/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/ia64/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/m68k/coldfire/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/m68k/m680x0/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/microblaze/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/mips/mips32/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/mips/mips64/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/nios2/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/fpu/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc32/nofpu/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/libm-le.abilist: Likewise.
* sysdeps/unix/sysv/linux/powerpc/powerpc64/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/riscv/rv64/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-32/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/s390/s390-64/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/sh/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc32/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/sparc/sparc64/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/tile/tilegx32/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/tile/tilegx64/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/x86_64/64/libm.abilist: Likewise.
* sysdeps/unix/sysv/linux/x86_64/x32/libm.abilist: Likewise.
This patch continues preparations for adding TS 18661-1 narrowing libm
functions by adding the required testsuite infrastructure to test such
functions through the libm-test infrastructure.
That infrastructure is based around testing for a single type, FLOAT.
For the narrowing functions, FLOAT, the "main" type for testing, is
the function return type; the argument type is ARG_FLOAT. This is
consistent with how the code built once for each type,
libm-test-support.c, depends on FLOAT for such things as calculating
ulps errors in results but can already handle different argument types
(pointers, integers, long double for nexttoward).
Makefile machinery is added to handle building tests for all pairs of
types for which there are narrowing functions (as with non-narrowing
functions, aliases are tested just the same as the functions they
alias). gen-auto-libm-tests gains a --narrow option for building
outputs for narrowing functions (so narrowing sqrt and fma will share
the same inputs as non-narrowing, but gen-auto-libm-tests will be run
with and without that option to generate different output files). In
the narrowing case, the auto-libm-test-out-narrow-* files include
annotations for each test about what properties ARG_FLOAT must have to
be able to represent all the inputs for that test; those annotations
result in calls to the TEST_COND_arg_fmt macro.
gen-libm-test.pl has some minor updates to handle narrowing tests (for
example, arguments in such tests must be surrounded by ARG_LIT calls
instead of LIT calls). Various new macros are added to the C test
support code (for example, sNaN initializers need to be properly
typed, so arg_snan_value is added; other such arg_* macros are added
as it seems cleanest to do so, though some are not strictly required).
Special-casing of the ibm128 format to allow for its limitations is
adjusted to handle it as the argument format as well as as the result
format; thus, the tests of the new functions allow nonzero ulps only
in the case where ibm128 is the argument format, as otherwise the
functions correspond to fully-defined IEEE operations. The ulps in
question appear as e.g. 'Function: "add_ldouble"' in libm-test-ulps
(with 1ulp errors then listed for double and float for that function
in powerpc); no support is added to generate corresponding faddl /
daddl ulps listings in the ulps table in the manual.
For the previous patch, I noted the need to avoid spurious macro
expansions of identifiers such as "add". A test test-narrow-macros.c
is added to verify such macro expansions are successfully avoided, and
there is also a -mlong-double-64 version of that test for ldbl-opt.
This test is set up to cover the full set of relevant identifiers from
the start rather than adding functions one at a time as each function
group is added.
Tested for x86_64 (this patch in isolation, as well as testing for
various configurations in conjunction with the actual addition of
"add" functions).
* math/Makefile (test-type-pairs): New variable.
(test-type-pairs-f64xf128-yes): Likewise.
(tests): Add test-narrow-macros.
(libm-test-funcs-narrow): New variable.
(libm-test-c-narrow): Likewise.
(generated): Add $(libm-test-c-narrow).
(libm-tests-base-narrow): New variable.
(libm-tests-narrow): Likewise.
(libm-tests): Add $(libm-tests-narrow).
(libm-tests-for-type): Handle $(libm-tests-narrow).
(libm-test-c-narrow-obj): New variable.
($(libm-test-c-narrow-obj)): New rule.
($(foreach t,$(libm-tests-narrow),$(objpfx)$(t).c)): Likewise.
($(foreach f,$(libm-test-funcs-narrow),$(objpfx)$(o)-$(f).o)): Use
$(o-iterator) to set dependencies and CFLAGS.
* math/gen-auto-libm-tests.c: Document use for narrowing
functions.
(output_for_one_input_case): Take argument NARROW.
(generate_output): Likewise. Update call to
output_for_one_input_case.
(main): Take --narrow option. Update call to generate_output.
* math/gen-libm-test.pl (_apply_lit): Take macro name as argument.
(apply_lit): Update call to _apply_lit.
(apply_arglit): New function.
(parse_args): Handle "a" arguments.
(parse_auto_input): Handle format names using ":".
* math/README.libm-test: Document "a" parameter type.
* math/libm-test-support.h (ARG_TYPE_MIN): New macro.
(ARG_TYPE_TRUE_MIN): Likewise.
(ARG_TYPE_MAX): Likwise.
(ARG_MIN_EXP): Likewise.
(ARG_MAX_EXP): Likewise.
(ARG_MANT_DIG): Likewise.
(TEST_COND_arg_ibm128): Likewise.
(TEST_COND_ibm128_libgcc): Define conditional on [ARG_FLOAT].
(TEST_COND_arg_fmt): New macro.
(init_max_error): Update prototype.
* math/libm-test-support.c (test_ibm128): New variable.
(init_max_error): Take argument testing_ibm128 and set test_ibm128
instead of using [TEST_COND_ibm128] conditional.
(test_exceptions): Use test_ibm128 instead of TEST_COND_ibm128.
* math/libm-test-driver.c (STR_ARG_FLOAT): New macro.
[TEST_NARROW] (TEST_MSG): New definition.
(arg_plus_zero): New macro.
(arg_minus_zero): Likewise.
(arg_plus_infty): Likewise.
(arg_minus_infty): Likewise.
(arg_qnan_value_pl): Likewise.
(arg_qnan_value): Likewise.
(arg_snan_value_pl): Likewise.
(arg_snan_value): Likewise.
(arg_max_value): Likewise.
(arg_min_value): Likewise.
(arg_min_subnorm_value): Likewise.
[ARG_FLOAT] (struct test_aa_f_data): New struct type.
(RUN_TEST_LOOP_aa_f): New macro.
(TEST_SUFF): New macro.
(TEST_SUFF_STR): Likewise.
[!TEST_MATHVEC] (VEC_SUFF): Don't define.
(TEST_COND_any_ibm128): New macro.
(START): Use TEST_SUFF and TEST_SUFF_STR in initializer for
this_func. Update call to init_max_error.
* math/test-double.h (FUNC_NARROW_PREFIX): New macro.
* math/test-float.h (FUNC_NARROW_PREFIX): Likewise.
* math/test-float128.h (FUNC_NARROW_PREFIX): Likewise.
* math/test-float32.h (FUNC_NARROW_PREFIX): Likewise.
* math/test-float32x.h (FUNC_NARROW_PREFIX): Likewise.
* math/test-float64.h (FUNC_NARROW_PREFIX): Likewise.
* math/test-float64x.h (FUNC_NARROW_PREFIX): Likewise.
* math/test-math-scalar.h (TEST_NARROW): Likewise.
* math/test-math-vector.h (TEST_NARROW): Likewise.
* math/test-arg-double.h: New file.
* math/test-arg-float128.h: Likewise.
* math/test-arg-float32x.h: Likewise.
* math/test-arg-float64.h: Likewise.
* math/test-arg-float64x.h: Likewise.
* math/test-arg-ldouble.h: Likewise.
* math/test-math-narrow.h: Likewise.
* math/test-narrow-macros.c: Likewise.
* sysdeps/ieee754/ldbl-opt/test-narrow-macros-ldbl-64.c: Likewise.
* sysdeps/ieee754/ldbl-opt/Makefile (tests): Add
test-narrow-macros-ldbl-64.
(CFLAGS-test-narrow-macros-ldbl-64.c): New variable.
TS 18661-1 defines libm functions that carry out an operation (+ - * /
sqrt fma) on their arguments and return a result rounded to a
(usually) narrower type, as if the original result were computed to
infinite precision and then rounded directly to the result type
without any intermediate rounding to the argument type. For example,
fadd, faddl and daddl for addition. These are the last remaining TS
18661-1 functions left to be added to glibc. TS 18661-3 extends this
to corresponding functions for _FloatN and _FloatNx types.
As functions parametrized by two rather than one varying
floating-point types, these functions require infrastructure in glibc
that was not required for previous libm functions. This patch
provides such infrastructure - excluding test support, and actual
function implementations, which will be in subsequent patches.
Declaring the functions uses a header bits/mathcalls-narrow.h, which
is included many times, for each relevant pair of types. This will
end up containing macro calls of the form
__MATHCALL_NARROW (__MATHCALL_NAME (add), __MATHCALL_REDIR_NAME (add), 2);
for each family of narrowing functions. (The structure of this macro
call, with the calls to __MATHCALL_NAME and __MATHCALL_REDIR_NAME
there rather than in the definition of __MATHCALL_NARROW, arises from
the names such as "add" *not* themselves being reserved identifiers -
meaning it's necessary to avoid any indirection that would result in a
user-defined "add" macro being expanded.) Whereas for existing
functions declaring long double functions is disabled if _LIBC in the
case where they alias double functions, to facilitate defining the
long double functions as aliases of the double ones, there is no such
logic for the narrowing functions in this patch. Rather, the files
defining such functions are expected to use #define to hide the
original declarations of the alias names, to avoid errors about
defining aliases with incompatible types.
math/Makefile support is added for building the functions (listed in
libm-narrow-fns, currently empty) for all relevant pairs of types. An
internal header math-narrow.h is added for macros shared between
multiple function implementations - currently a ROUND_TO_ODD macro to
facilitate writing functions using the round-to-odd implementation
approach, and alias macros to create all the required function
aliases. libc_feholdexcept_setroundf128 and libc_feupdateenv_testf128
are added for use when required (only for x86_64). float128_private.h
support is added for ldbl-128 narrowing functions to be used for
_Float128.
Certain things are specifically omitted from this patch and the
immediate followups. tgmath.h support is deferred; there remain
unresolved questions about how the type-generic macros for these
functions are supposed to work, especially in the case of arguments of
integer type. The math.h / bits/mathcalls-narrow.h logic, and the
logic for determining what functions / aliases to define, will need
some adjustments to support the sqrt and fma functions, where
e.g. f32xsqrtf64 can just be an alias for sqrt rather than a separate
function. TS 18661-1 defines FP_FAST_* macros but no support is
included for defining them (they won't in general be true without
architecture-specific optimized function versions).
For each of the function groups (add sub mul div sqrt fma) there are
always six functions present (e.g. fadd, faddl, daddl, f32addf64,
f32addf32x, f32xaddf64). When _Float64x and _Float128 are supported,
there are seven more (e.g. f32addf64x, f32addf128, f64addf64x,
f64addf128, f32xaddf64x, f32xaddf128, f64xaddf128). In addition, in
the ldbl-opt case there are function names such as __nldbl_daddl (an
alias for f32xaddf64, which is not a reserved name in TS 18661-1, only
in TS 18661-3), for calls to daddl to be mapped to in the
-mlong-double-64 case. (Calls to faddl just get mapped to fadd, and
for sqrt and fma there won't be __nldbl_* functions because dsqrtl and
dfmal can just be mapped to sqrt and fma with -mlong-double-64.)
While there are six or thirteen functions present in each group (plus
__nldbl_* names only as an ABI, not an API), not all are distinct;
they fall in various groups of aliases. There are two distinct
versions built if long double has the same format as double; four if
they have distinct formats but there is no _Float64x or _Float128
support; five if long double has binary128 format; seven when
_Float128 is distinct from long double.
Architecture-specific optimized versions are possible, but not
included in my patches. For example, IA64 generally supports
narrowing the result of most floating-point instructions; Power ISA
2.07 (POWER8) supports double values as arguments to float
instructions, with the results narrowed as expected; Power ISA 3
(POWER9) supports round-to-odd for float128 instructions, so meaning
that approach can be used without needing to set and restore the
rounding mode and test "inexact". I intend to leave any such
optimized versions to the architecture maintainers. Generally in such
cases it would also make sense for calls to these functions to be
expanded inline (given -fno-math-errno); I put a suggestion for TS
18661-1 built-in functions at <https://gcc.gnu.org/wiki/SummerOfCode>.
Tested for x86_64 (this patch in isolation, as well as testing for
various configurations in conjunction with further patches).
* math/bits/mathcalls-narrow.h: New file.
* include/bits/mathcalls-narrow.h: Likewise.
* math/math-narrow.h: Likewise.
* math/math.h (__MATHCALL_NARROW_ARGS_1): New macro.
(__MATHCALL_NARROW_ARGS_2): Likewise.
(__MATHCALL_NARROW_ARGS_3): Likewise.
(__MATHCALL_NARROW_NORMAL): Likewise.
(__MATHCALL_NARROW_REDIR): Likewise.
(__MATHCALL_NARROW): Likewise.
[__GLIBC_USE (IEC_60559_BFP_EXT)]: Repeatedly include
<bits/mathcalls-narrow.h> with _Mret_, _Marg_ and __MATHCALL_NAME
defined.
[__GLIBC_USE (IEC_60559_TYPES_EXT)]: Likewise.
* math/Makefile (headers): Add bits/mathcalls-narrow.h.
(libm-narrow-fns): New variable.
(libm-narrow-types-basic): Likewise.
(libm-narrow-types-ldouble-yes): Likewise.
(libm-narrow-types-float128-yes): Likewise.
(libm-narrow-types-float128-alias-yes): Likewise.
(libm-narrow-types): Likewise.
(libm-routines): Add narrowing functions.
* sysdeps/i386/fpu/fenv_private.h [__x86_64__]
(libc_feholdexcept_setroundf128): New macro.
[__x86_64__] (libc_feupdateenv_testf128): Likewise.
* sysdeps/ieee754/float128/float128_private.h: Include
<math/math-narrow.h>.
[libc_feholdexcept_setroundf128] (libc_feholdexcept_setroundl):
Undefine and redefine.
[libc_feupdateenv_testf128] (libc_feupdateenv_testl): Likewise.
(libm_alias_float_ldouble): Undefine and redefine.
(libm_alias_double_ldouble): Likewise.
Nearly everything in _G_config.h is either junk or more appropriately
defined elsewhere:
* _G_fpos_t, _G_fpos64_t, and _G_BUFSIZ are already completely unused.
* All remaining uses of _G_va_list have been changed to __gnuc_va_list.
* The definition of _G_HAVE_ST_BLKSIZE/_IO_HAVE_ST_BLKSIZE has
been inlined into its sole use.
* The complete definition of _G_iconv_t has been moved to libio.h and
renamed _IO_iconv_t (all actual users used that name).
* _G_IO_IO_FILE_VERSION is vestigial; some code cares whether
_IO_stdin_used exists, but nothing looks at its value. I've
preserved the value as a hardwired constant in csu/init.c.
This means csu/init.c no longer needs to include anything.
* Many of the headers included by _G_config.h were already being
included directly by either either libio.h or stdio.h; the
remaining ones were moved to libio.h.
* _G_HAVE_MREMAP is still relevant, because mremap genuinely is a
Linux extension; it's not in POSIX and as far as I can tell it's
not available on the Hurd either. I also preserved _G_HAVE_MMAP,
since it's conceivable someone would want to port glibc to a
MMU-less, mmap-less environment in the future. Both are now always
defined to 1/0 as is the current convention, instead of the older
1/undef convention. These are the only symbols still defined in
_G_config.h.
* The actual inclusion of _G_config.h moves from libio.h to libioP.h,
as this is where a potential override of _G_HAVE_MMAP happens.
* The #ifdef logic in libioP.h controlling _IO_JUMPS_OFFSET has been
simplified.
After this patch, the only surviving _G_ symbols are the struct tag
names _G_fpos_t and _G_fpos64_t, which are preserved for the sake of
C++ mangled names in applications, and _G_HAVE_MMAP and _G_HAVE_MREMAP,
which do not seem worth renaming.
Installed stripped libraries are unchanged by this patch.
* bits/_G_config.h: Move back to sysdeps/generic/_G_config.h.
Delete all contents except for definitions of _G_HAVE_MMAP and
_G_HAVE_MREMAP. Add commentary explaining those two symbols.
* sysdeps/unix/sysv/linux/bits/_G_config.h: Move back to
sysdeps/unix/sysv/linux/_G_config.h. Make same content
change as above.
* libio/libio.h: Don't include bits/_G_config.h here.
Include stddef.h with __need_wchar_t defined. Include
bits/types/__mbstate_t.h, bits/types/wint_t.h, and gconv.h.
Define _IO_iconv_t here, directly.
Don't define _IO_HAVE_ST_BLKSIZE.
* libio/libioP.h: Include _G_config.h here. Move include of
shlib-compat.h up with rest of includes. Simplify conditionals
controlling definition of _IO_JUMPS_OFFSET.
* csu/init.c: Remove always-true #if around entire file.
Don't include stdio.h. Set _IO_stdin_used to hardwired
constant 0x20001, and update commentary.
* include/stdio.h, sysdeps/ieee754/ldbl-opt/nldbl-compat.h:
Replace all uses of _G_va_list with __gnuc_va_list.
* libio/filedoalloc.c: Use #if defined _STATBUF_ST_BLKSIZE
instead of #if _IO_HAVE_ST_BLKSIZE.
* libio/fileops.c: Test _G_HAVE_MREMAP with #if, not #ifdef.
* libio/iofdopen.c, libio/iofopen.c: Test _G_HAVE_MMAP with #if,
not #ifdef.
Remove the slow paths from log. Like several other double precision math
functions, log is exactly rounded. This is not required from math functions
and causes major overheads as it requires multiple fallbacks using higher
precision arithmetic if a result is close to 0.5ULP. Ridiculous slowdowns
of up to 100000x have been reported when the highest precision path triggers.
Interestingly removing the slow paths makes hardly any difference in practice:
the worst case error is still ~0.502ULP, and exp(log(x)) shows identical results
before/after on many millions of random cases. All GLIBC math tests pass on
AArch64 and x64 with no change in ULP error. A simple test over a few hundred
million values shows log is now 18% faster on average.
* manual/probes.texi (slowlog): Delete documentation of removed probe.
(slowlog_inexact): Likewise
* sysdeps/ieee754/dbl-64/e_log.c (__ieee754_log): Remove slow paths.
* sysdeps/ieee754/dbl-64/ulog.h: Remove unused declarations.
The general rule in glibc is that it's better for a macro to be always
defined, and tested with #if, than for it to be tested with #ifdef,
because the latter is prone to typos in the macro name as well as to
the header with the macro accidentally not being included in a file
testing it. (Testing with an "if" statement is even better, in those
cases where it's possible to do things that way, as it then means both
cases in the code get checked for syntax in glibc builds with either
value of the condition.)
math_private.h has several different groups of macros, meaning that
architectures wanting to override some of them need to define those
then include the generic version, which then defines macros if not
already defined. It's hard to avoid that arrangement completely, but
various cases can be improved by splitting out macros or groups of
macros into separate files.
This patch splits out the LDBL_CLASSIFY_COMPAT macro into a separate
ldbl-classify-compat.h header. This macro is tested with #ifdef; this
patch changes it to testing with #if, with a default definition to 0
in the generic header and then architecture-specific headers defining
it to 1.
Tested with build-many-glibcs.py that installed stripped shared
libraries are unchanged by the patch.
* sysdeps/generic/ldbl-classify-compat.h: New file.
* sysdeps/arm/ldbl-classify-compat.h: Likewise.
* sysdeps/m68k/coldfire/ldbl-classify-compat.h: Likewise.
* sysdeps/microblaze/ldbl-classify-compat.h: Likewise.
* sysdeps/mips/ldbl-classify-compat.h: Likewise.
* sysdeps/nios2/ldbl-classify-compat.h: Likewise.
* sysdeps/sh/ldbl-classify-compat.h: Likewise.
* sysdeps/ieee754/dbl-64/s_finite.c: Include
<ldbl-classify-compat.h>.
[LDBL_CLASSIFY_COMPAT]: Test value, not whether defined.
* sysdeps/ieee754/dbl-64/s_isinf.c: Include
<ldbl-classify-compat.h>.
[LDBL_CLASSIFY_COMPAT]: Test value, not whether defined.
* sysdeps/ieee754/dbl-64/s_isnan.c: Include
<ldbl-classify-compat.h>.
[LDBL_CLASSIFY_COMPAT]: Test value, not whether defined.
* sysdeps/ieee754/dbl-64/wordsize-64/s_finite.c: Include
<ldbl-classify-compat.h>.
[LDBL_CLASSIFY_COMPAT]: Test value, not whether defined.
* sysdeps/ieee754/dbl-64/wordsize-64/s_isinf.c: Include
<ldbl-classify-compat.h>.
[LDBL_CLASSIFY_COMPAT]: Test value, not whether defined.
* sysdeps/ieee754/dbl-64/wordsize-64/s_isnan.c: Include
<ldbl-classify-compat.h>.
[LDBL_CLASSIFY_COMPAT]: Test value, not whether defined.
* sysdeps/arm/math_private.h (LDBL_CLASSIFY_COMPAT): Remove macro.
* sysdeps/mips/math_private.h (LDBL_CLASSIFY_COMPAT): Likewise.
* sysdeps/m68k/coldfire/math_private.h: Remove file.
* sysdeps/microblaze/math_private.h: Likewise.
* sysdeps/nios2/math_private.h: Likewise.
* sysdeps/sh/math_private.h: Likewise.
As reported in bug 21314, building log1p and log1pf fails with -Os
because of a spurious -Wmaybe-uninitialized warning (reported there
for GCC 5 for MIPS, I see it also with GCC 7 for x86_64). This patch,
based on the patches in the bug, fixes this using the DIAG_* macros.
Tested for x86_64 with -Os that this eliminates those warnings and so
allows the build to progress further.
2018-02-01 Carlos O'Donell <carlos@redhat.com>
Ramin Seyed-Moussavi <lordrasmus@gmail.com>
Joseph Myers <joseph@codesourcery.com>
[BZ #21314]
* sysdeps/ieee754/dbl-64/s_log1p.c: Include <libc-diag.h>.
(__log1p): Disable -Wmaybe-uninitialized for -Os around
computation using c.
* sysdeps/ieee754/flt-32/s_log1pf.c: Include <libc-diag.h>.
(__log1pf): Disable -Wmaybe-uninitialized for -Os around
computation using c.
The ldbl-128ibm implementation of log1pl does ordered comparisons on a
negative qNaN argument, so resulting in spurious "invalid" exceptions
(for soft-float powerpc; hard-float only avoids this because of GCC
bug 58684 meaning ordered comparison instructions never get
generated). This patch fixes this by arranging for the test for NaN
or infinity arguments to handle negative arguments as well.
Tested for powerpc (soft float).
[BZ #22693]
* sysdeps/ieee754/ldbl-128ibm/s_log1pl.c (__log1pl): Handle
negative arguments in test for NaN or infinity argument.
The ldbl-128ibm implementations of lrintl and lroundl are missing
"invalid" exceptions for certain overflow cases when compiled with GCC
8. The cause of this is after-the-fact integer overflow checks that
fail when the compiler optimizes on the basis of integer overflow
being undefined; GCC 8 must be able to detect new cases of
undefinedness here.
Failure: lrint (-0x80000001p0): Exception "Invalid operation" not set
Failure: lrint_downward (-0x80000001p0): Exception "Invalid operation" not set
Failure: lrint_towardzero (-0x80000001p0): Exception "Invalid operation" not set
Failure: lrint_upward (-0x80000001p0): Exception "Invalid operation" not set
Failure: lround (-0x80000001p0): Exception "Invalid operation" not set
Failure: lround_downward (-0x80000001p0): Exception "Invalid operation" not set
Failure: lround_towardzero (-0x80000001p0): Exception "Invalid operation" not set
Failure: lround_upward (-0x80000001p0): Exception "Invalid operation" not set
(Tested that these failures occur before the patch for powerpc
soft-float, but the issue applies in principle for hard-float as well,
whether or not the particular optimizations in fact occur there at
present.)
This patch fixes the bug by ensuring the additions / subtractions in
question cast arguments to unsigned long int, or use 1UL as a constant
argument, so that the arithmetic occurs in an unsigned type with the
result then converted back to a signed type.
Tested for powerpc (soft-float).
[BZ #22690]
* sysdeps/ieee754/ldbl-128ibm/s_lrintl.c (__lrintl): Use unsigned
long int for arguments of possibly overflowing addition or
subtraction.
* sysdeps/ieee754/ldbl-128ibm/s_lroundl.c (__lroundl): Likewise.
As noted in bug 21309, dbl-64/e_pow.c contains signed int shifts that,
although the shift count is in the range [0, 31], shift bits into and
beyond the sign bit and so are undefined in ISO C. Although this is
defined in GNU C, this patch from the bug cleans up the code to avoid
those shifts.
Tested for x86_64.
[BZ #21309]
* sysdeps/ieee754/dbl-64/e_pow.c (checkint): Make m and n
unsigned.
These changes will be active for all platforms that don't provide
their own exp() routines. They will also be active for ieee754
versions of ccos, ccosh, cosh, csin, csinh, sinh, exp10, gamma, and
erf.
Typical performance gains is typically around 5x when measured on
Sparc s7 for common values between exp(1) and exp(40).
Using the glibc perf tests on sparc,
sparc (nsec) x86 (nsec)
old new old new
max 17629 395 5173 144
min 399 54 15 13
mean 5317 200 1349 23
The extreme max times for the old (ieee754) exp are due to the
multiprecision computation in the old algorithm when the true value is
very near 0.5 ulp away from an value representable in double
precision. The new algorithm does not take special measures for those
cases. The current glibc exp perf tests overrepresent those values.
Informal testing suggests approximately one in 200 cases might
invoke the high cost computation. The performance advantage of the new
algorithm for other values is still large but not as large as indicated
by the chart above.
Glibc correctness tests for exp() and expf() were run. Within the
test suite 3 input values were found to cause 1 bit differences (ulp)
when "FE_TONEAREST" rounding mode is set. No differences in exp() were
seen for the tested values for the other rounding modes.
Typical example:
exp(-0x1.760cd2p+0) (-1.46113312244415283203125)
new code: 2.31973271630014299393707e-01 0x1.db14cd799387ap-3
old code: 2.31973271630014271638132e-01 0x1.db14cd7993879p-3
exp = 2.31973271630014285508337 (high precision)
Old delta: off by 0.49 ulp
New delta: off by 0.51 ulp
In addition, because ieee754_exp() is used by other routines, cexp()
showed test results with very small imaginary input values where the
imaginary portion of the result was off by 3 ulp when in upward
rounding mode, but not in the other rounding modes. For x86, tgamma
showed a few values where the ulp increased to 6 (max ulp for tgamma
is 5). Sparc tgamma did not show these failures. I presume the tgamma
differences are due to compiler optimization differences within the
gamma function.The gamma function is known to be difficult to compute
accurately.
* sysdeps/ieee754/dbl-64/e_exp.c: Include <math-svid-compat.h> and
<errno.h>. Include "eexp.tbl".
(half): New constant.
(one): Likewise.
(__ieee754_exp): Rewrite.
(__slowexp): Remove prototype.
* sysdeps/ieee754/dbl-64/eexp.tbl: New file.
* sysdeps/ieee754/dbl-64/slowexp.c: Remove file.
* sysdeps/i386/fpu/slowexp.c: Likewise.
* sysdeps/ia64/fpu/slowexp.c: Likewise.
* sysdeps/m68k/m680x0/fpu/slowexp.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/slowexp-avx.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/slowexp-fma.c: Likewise.
* sysdeps/x86_64/fpu/multiarch/slowexp-fma4.c: Likewise.
* sysdeps/generic/math_private.h (__slowexp): Remove prototype.
* sysdeps/ieee754/dbl-64/e_pow.c: Remove mention of slowexp.c in
comment.
* sysdeps/powerpc/power4/fpu/Makefile [$(subdir) = math]
(CPPFLAGS-slowexp.c): Remove variable.
* sysdeps/x86_64/fpu/multiarch/Makefile (libm-sysdep_routines):
Remove slowexp-fma, slowexp-fma4 and slowexp-avx.
(CFLAGS-slowexp-fma.c): Remove variable.
(CFLAGS-slowexp-fma4.c): Likewise.
(CFLAGS-slowexp-avx.c): Likewise.
* sysdeps/x86_64/fpu/multiarch/e_exp-avx.c (__slowexp): Do not
define as macro.
* sysdeps/x86_64/fpu/multiarch/e_exp-fma.c (__slowexp): Likewise.
* sysdeps/x86_64/fpu/multiarch/e_exp-fma4.c (__slowexp): Likewise.
* math/Makefile (type-double-routines): Remove slowexp.
* manual/probes.texi (slowexp_p6): Remove.
(slowexp_p32): Likewise.
The default sysdeps/ieee754 fma implementations rely on exceptions and
rounding modes to achieve correct results through internal use of
round-to-odd. Thus, glibc configurations without support for
exceptions and rounding modes instead need to use implementations of
fma based on soft-fp.
At present, this is achieved via having implementation files in
soft-fp/ that are #included by sysdeps files for each glibc
configuration that needs them. In general this means such a
configuration has its own s_fma.c and s_fmaf.c.
TS 18661-1 adds functions that do an operation (+ - * / sqrt fma) on
arguments wider than the return type, with a single rounding of the
infinite-precision result to that return type. These are also
naturally implemented using round-to-odd on platforms with hardware
support for rounding modes and exceptions but lacking hardware support
for these narrowing operations themselves. (Platforms that have
direct hardware support for such narrowing operations include at least
ia64, and Power ISA 2.07 or later, which I think means POWER8 or
later.)
So adding the remaining TS 18661-1 functions would mean at least six
narrowing function implementations (fadd fsub fmul fdiv ffma fsqrt),
with aliases for other types and further implementations in some
configurations, that need to be overridden for configurations lacking
hardware exceptions and rounding modes. Requiring all such
configurations (currently seven of them) to have their own source
files for all those functions seems undesirable.
Thus, this patch adds a directory sysdeps/ieee754/soft-fp to contain
libm function implementations based on soft-fp. This directory is
then used via Implies from all the configurations that need it, so no
more files need adding to every such configuration when adding more
functions with soft-fp implementations. A configuration can still
selectively #include a particular file from this directory if desired;
thus, the MIPS #include of the fmal implementation is retained, since
that's appropriate even for hard float (because long double is always
implementated in software for MIPS64, so the soft-fp implementation of
fmal is better than the ldbl-128 one).
This also provides additional motivation for my recent patch removing
--with-fp / --without-fp: previously there was no need for correct use
of --without-fp for no-FPU ARM or SH3, and now we have autodetection
nofpu/ sysdeps directories can be used by this patch for those
configurations without imposing any new requirements on how glibc is
configured.
(The mips64/*/fpu/s_fma.c files added by this patch are needed to keep
the dbl-64 version of fma for double, rather than the ldbl-128 one,
used in that case.)
Tested with build-many-glibcs.py that installed stripped shared
libraries are unchanged by this patch.
* soft-fp/fmadf4.c: Move to ....
* sysdeps/ieee754/soft-fp/s_fma.c: ... here.
* soft-fp/fmasf4.c: Move to ....
* sysdeps/ieee754/soft-fp/s_fmaf.c: ... here.
* soft-fp/fmatf4.c: Move to ....
* sysdeps/ieee754/soft-fp/s_fmal.c: ... here.
* sysdeps/ieee754/soft-fp/Makefile: New file.
* sysdeps/arm/preconfigure.ac: Define with_fp_cond.
* sysdeps/arm/preconfigure: Regenerated.
* sysdeps/arm/nofpu/Implies: New file.
* sysdeps/arm/s_fma.c: Remove file.
* sysdeps/arm/s_fmaf.c: Likewise.
* sysdeps/m68k/coldfire/nofpu/Implies: New file.
* sysdeps/m68k/coldfire/nofpu/s_fma.c: Remove file.
* sysdeps/m68k/coldfire/nofpu/s_fmaf.c: Likewise.
* sysdeps/microblaze/Implies: Add ieee754/soft-fp.
* sysdeps/microblaze/s_fma.c: Remove file.
* sysdeps/microblaze/s_fmaf.c: Likewise.
* sysdeps/mips/mips32/nofpu/Implies: New file.
* sysdeps/mips/mips64/n32/fpu/s_fma.c: Likewise.
* sysdeps/mips/mips64/n32/nofpu/Implies: Likewise.
* sysdeps/mips/mips64/n64/fpu/s_fma.c: Likewise.
* sysdeps/mips/mips64/n64/nofpu/Implies: Likewise.
* sysdeps/mips/ieee754/s_fma.c: Remove file.
* sysdeps/mips/ieee754/s_fmaf.c: Likewise.
* sysdeps/mips/ieee754/s_fmal.c: Update include for move of fmal
implementation.
* sysdeps/nios2/Implies: Add ieee754/soft-fp.
* sysdeps/nios2/s_fma.c: Remove file.
* sysdeps/nios2/s_fmaf.c: Likewise.
* sysdeps/sh/nofpu/Implies: New file.
* sysdeps/sh/s_fma.c: Remove file.
* sysdeps/sh/s_fmaf.c: Likewise.
* sysdeps/tile/Implies: Add ieee754/soft-fp.
* sysdeps/tile/s_fma.c: Remove file.
* sysdeps/tile/s_fmaf.c: Likewise.
Wilco Dijkstra
Szabolcs Nagy
Gabriel F. T. Gomes
Rajalakshmi Srinivasaraghavan
Tulio Magno Quites Machado Filho
Joseph Myers
Florian Weimer
Patrick McGehearty
Zack Weinberg
Carlos O'Donell
Bernd Edlinger