As the discussion in the PR pointed out the RTL we have for the REDUC_PLUS
patterns are wrong. The UNSPECs are modelled as returning a vector and then
in an expand pattern we emit a vec_select of the 0th element to get the scalar.
This is incorrect as the instruction itself already only returns a single scalar
and by declaring it returns a vector it allows combine to push in a subreg into
the pattern, which causes reload to make duplicate moves.
This patch corrects this by removing the weird indirection and making the RTL
pattern model the correct semantics of the instruction immediately.
gcc/ChangeLog:
PR target/104049
* config/aarch64/aarch64-simd.md
(aarch64_reduc_plus_internal<mode>): Fix RTL and rename to...
(reduc_plus_scal_<mode>): ... This.
(reduc_plus_scal_v4sf): Moved.
(aarch64_reduc_plus_internalv2si): Fix RTL and rename to...
(reduc_plus_scal_v2si): ... This.
gcc/testsuite/ChangeLog:
PR target/104049
* gcc.target/aarch64/vadd_reduc-1.c: New test.
* gcc.target/aarch64/vadd_reduc-2.c: New test.
This patch extends the previous support for 16-byte vec_concat
so that it supports pairs of 4-byte elements. This too isn't
strictly a regression fix, since the 8-byte forms weren't affected
by the same problems as the 16-byte forms, but it leaves things in
a more consistent state.
gcc/
* config/aarch64/iterators.md (VDCSIF): New mode iterator.
(VDBL): Handle SF.
(single_wx, single_type, single_dtype, dblq): New mode attributes.
* config/aarch64/aarch64-simd.md (load_pair_lanes<mode>): Extend
from VDC to VDCSIF.
(store_pair_lanes<mode>): Likewise.
(*aarch64_combine_internal<mode>): Likewise.
(*aarch64_combine_internal_be<mode>): Likewise.
(*aarch64_combinez<mode>): Likewise.
(*aarch64_combinez_be<mode>): Likewise.
* config/aarch64/aarch64.cc (aarch64_classify_address): Handle
8-byte modes for ADDR_QUERY_LDP_STP_N.
(aarch64_print_operand): Likewise for %y.
gcc/testsuite/
* gcc.target/aarch64/vec-init-13.c: New test.
* gcc.target/aarch64/vec-init-14.c: Likewise.
* gcc.target/aarch64/vec-init-15.c: Likewise.
* gcc.target/aarch64/vec-init-16.c: Likewise.
* gcc.target/aarch64/vec-init-17.c: Likewise.
This patch is the second of two to remove the old
move_lo/hi_quad expanders and move_hi_quad insns.
gcc/
* config/aarch64/aarch64-simd.md (@aarch64_split_simd_mov<mode>):
Use aarch64_combine instead of move_lo/hi_quad. Tabify.
(move_lo_quad_<mode>, aarch64_simd_move_hi_quad_<mode>): Delete.
(aarch64_simd_move_hi_quad_be_<mode>, move_hi_quad_<mode>): Delete.
(vec_pack_trunc_<mode>): Take general_operand elements and use
aarch64_combine rather than move_lo/hi_quad to combine them.
(vec_pack_trunc_df): Likewise.
After previous patches, we have a (mostly new) group of vec_concat
patterns as well as vestiges of the old move_lo/hi_quad patterns.
(A previous patch removed the move_lo_quad insns, but we still
have the move_hi_quad insns and both sets of expanders.)
This patch is the first of two to remove the old move_lo/hi_quad
stuff. It isn't technically a regression fix, but it seemed
better to make the changes now rather than leave things in
a half-finished and inconsistent state.
This patch defines an aarch64_vec_concat expander that coerces the
element operands into a valid form, including the ones added by the
previous patch. This in turn lets us get rid of one move_lo/hi_quad
pair.
As a side-effect, it also means that vcombines of 2 vectors make
better use of the available forms, like vec_inits of 2 scalars
already do.
gcc/
* config/aarch64/aarch64-protos.h (aarch64_split_simd_combine):
Delete.
* config/aarch64/aarch64-simd.md (@aarch64_combinez<mode>): Rename
to...
(*aarch64_combinez<mode>): ...this.
(@aarch64_combinez_be<mode>): Rename to...
(*aarch64_combinez_be<mode>): ...this.
(@aarch64_vec_concat<mode>): New expander.
(aarch64_combine<mode>): Use it.
(@aarch64_simd_combine<mode>): Delete.
* config/aarch64/aarch64.cc (aarch64_split_simd_combine): Delete.
(aarch64_expand_vector_init): Use aarch64_vec_concat.
gcc/testsuite/
* gcc.target/aarch64/vec-init-12.c: New test.
vec_combine is really one instruction on aarch64, provided that
the lowpart element is in the same register as the destination
vector. This patch adds patterns for that.
The patch fixes a regression from GCC 8. Before the patch:
int64x2_t s64q_1(int64_t a0, int64_t a1) {
if (__BYTE_ORDER__ == __ORDER_BIG_ENDIAN__)
return (int64x2_t) { a1, a0 };
else
return (int64x2_t) { a0, a1 };
}
generated:
fmov d0, x0
ins v0.d[1], x1
ins v0.d[1], x1
ret
whereas GCC 8 generated the more respectable:
dup v0.2d, x0
ins v0.d[1], x1
ret
gcc/
* config/aarch64/predicates.md (aarch64_reg_or_mem_pair_operand):
New predicate.
* config/aarch64/aarch64-simd.md (*aarch64_combine_internal<mode>)
(*aarch64_combine_internal_be<mode>): New patterns.
gcc/testsuite/
* gcc.target/aarch64/vec-init-9.c: New test.
* gcc.target/aarch64/vec-init-10.c: Likewise.
* gcc.target/aarch64/vec-init-11.c: Likewise.
move_lo_quad_internal_<mode> and move_lo_quad_internal_be_<mode>
partially duplicate the later aarch64_combinez{,_be}<mode> patterns.
The duplication itself is a regression.
The only substantive differences between the two are:
* combinez uses vector MOV (ORR) instead of element MOV (DUP).
The former seems more likely to be handled via renaming.
* combinez disparages the GPR->FPR alternative whereas move_lo_quad
gave it equal cost. The new test gives a token example of when
the combinez behaviour helps.
gcc/
* config/aarch64/aarch64-simd.md (move_lo_quad_internal_<mode>)
(move_lo_quad_internal_be_<mode>): Delete.
(move_lo_quad_<mode>): Use aarch64_combine<Vhalf> instead of the above.
gcc/testsuite/
* gcc.target/aarch64/vec-init-8.c: New test.
This patch generalises the load_pair_lanes<mode> guard so that
it uses aarch64_check_consecutive_mems to check for consecutive
mems. It also allows the pattern to be used for STRICT_ALIGNMENT
targets if the alignment is high enough.
The main aim is to avoid an inline test, for the sake of a later patch
that needs to repeat it. Reusing aarch64_check_consecutive_mems seemed
simpler than writing an entirely new function.
gcc/
* config/aarch64/aarch64-protos.h (aarch64_mergeable_load_pair_p):
Declare.
* config/aarch64/aarch64-simd.md (load_pair_lanes<mode>): Use
aarch64_mergeable_load_pair_p instead of inline check.
* config/aarch64/aarch64.cc (aarch64_expand_vector_init): Likewise.
(aarch64_check_consecutive_mems): Allow the reversed parameter
to be null.
(aarch64_mergeable_load_pair_p): New function.
The aarch64_simd_vec_set<mode> define_insn takes memory operands,
so this patch makes the vec_set<mode> optab expander do the same.
gcc/
* config/aarch64/aarch64-simd.md (vec_set<mode>): Allow the
element to be an aarch64_simd_nonimmediate_operand.
This patch fixes some case in which *general_operand was used over
*nonimmediate_operand by patterns that don't accept immediates.
This avoids some complication with later patches.
gcc/
* config/aarch64/aarch64-simd.md (aarch64_simd_vec_set<mode>): Use
aarch64_simd_nonimmediate_operand instead of
aarch64_simd_general_operand.
(@aarch64_combinez<mode>): Use nonimmediate_operand instead of
general_operand.
(@aarch64_combinez_be<mode>): Likewise.
The Advanced SIMD movmisalign patterns didn't handle 16-bit
FP modes, which meant that the vector loop for:
void
test (_Float16 *data)
{
_Pragma ("omp simd")
for (int i = 0; i < 8; ++i)
data[i] = 1.0;
}
would be versioned for alignment.
This was causing some new failures in aarch64/sve/single_5.c:
FAIL: gcc.target/aarch64/sve/single_5.c scan-assembler-not \\tb
FAIL: gcc.target/aarch64/sve/single_5.c scan-assembler-not \\tcmp
FAIL: gcc.target/aarch64/sve/single_5.c scan-assembler-times \\tstr\\tq[0-9]+, 10
but I didn't look into what changed from earlier releases.
Adding the missing modes removes some existing xfails.
gcc/
* config/aarch64/aarch64-simd.md (movmisalign<mode>): Extend from
VALL to VALL_F16.
gcc/testsuite/
* gcc.target/aarch64/sve/single_5.c: Remove some XFAILs.
The VALL_F16MOV iterator now has the same modes as VALL_F16,
in the same order. This patch removes the former in favour
of the latter.
This doesn't fix a bug as such, but it's ultra-safe (no change in
object code) and it saves a follow-up patch from having to make
a false choice between the iterators.
gcc/
* config/aarch64/iterators.md (VALL_F16MOV): Delete.
* config/aarch64/aarch64-simd.md (mov<mode>): Use VALL_F16 instead
of VALL_F16MOV.
After the first patch in the series this updates the optabs to expect the
canonical sequence.
gcc/ChangeLog:
PR tree-optimization/102819
PR tree-optimization/103169
* config/aarch64/aarch64-simd.md (cml<fcmac1><conj_op><mode>4): Use
canonical order.
* config/aarch64/aarch64-sve.md (cml<fcmac1><conj_op><mode>4): Likewise.
This patch is sorting issue with LS64 intrinsics tests failing with
AArch64_be targets.
gcc/ChangeLog:
PR target/103729
* config/aarch64/aarch64-simd.md (aarch64_movv8di): Allow big endian
targets to move V8DI.
This patch is adding support for LS64 (Armv8.7-A Load/Store 64 Byte extension)
which is part of Armv8.7-A architecture. Changes include missing plumbing for
TARGET_LS64, LS64 data structure and intrinsics defined in ACLE. Machine
description of intrinsics is using new V8DI mode added in a separate patch.
__ARM_FEATURE_LS64 is defined if the Armv8.7-A LS64 instructions for atomic
64-byte access to device memory are supported.
New compiler internal type is added wrapping ACLE struct data512_t:
typedef struct {
uint64_t val[8];
} __arm_data512_t;
gcc/ChangeLog:
* config/aarch64/aarch64-builtins.c (enum aarch64_builtins):
Define AARCH64_LS64_BUILTIN_LD64B, AARCH64_LS64_BUILTIN_ST64B,
AARCH64_LS64_BUILTIN_ST64BV, AARCH64_LS64_BUILTIN_ST64BV0.
(aarch64_init_ls64_builtin_decl): Helper function.
(aarch64_init_ls64_builtins): Helper function.
(aarch64_init_ls64_builtins_types): Helper function.
(aarch64_general_init_builtins): Init LS64 intrisics for
TARGET_LS64.
(aarch64_expand_builtin_ls64): LS64 intrinsics expander.
(aarch64_general_expand_builtin): Handle aarch64_expand_builtin_ls64.
(ls64_builtins_data): New helper struct.
(v8di_UP): New define.
* config/aarch64/aarch64-c.c (aarch64_update_cpp_builtins): Define
__ARM_FEATURE_LS64.
* config/aarch64/aarch64.c (aarch64_classify_address): Enforce the
V8DI range (7-bit signed scaled) for both ends of the range.
* config/aarch64/aarch64-simd.md (movv8di): New pattern.
(aarch64_movv8di): New pattern.
* config/aarch64/aarch64.h (AARCH64_ISA_LS64): New define.
(TARGET_LS64): New define.
* config/aarch64/aarch64.md: Add UNSPEC_LD64B, UNSPEC_ST64B,
UNSPEC_ST64BV and UNSPEC_ST64BV0.
(ld64b): New define_insn.
(st64b): New define_insn.
(st64bv): New define_insn.
(st64bv0): New define_insn.
* config/aarch64/arm_acle.h (data512_t): New type derived from
__arm_data512_t.
(__arm_data512_t): New internal type.
(__arm_ld64b): New intrinsic.
(__arm_st64b): New intrinsic.
(__arm_st64bv): New intrinsic.
(__arm_st64bv0): New intrinsic.
* config/arm/types.md: Add new type ls64.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/acle/ls64_asm.c: New test.
* gcc.target/aarch64/acle/ls64_ld64b.c: New test.
* gcc.target/aarch64/acle/ls64_ld64b-2.c: New test.
* gcc.target/aarch64/acle/ls64_ld64b-3.c: New test.
* gcc.target/aarch64/acle/ls64_st64b.c: New test.
* gcc.target/aarch64/acle/ls64_ld_st_o0.c: New test.
* gcc.target/aarch64/acle/ls64_st64b-2.c: New test.
* gcc.target/aarch64/acle/ls64_st64bv.c: New test.
* gcc.target/aarch64/acle/ls64_st64bv-2.c: New test.
* gcc.target/aarch64/acle/ls64_st64bv-3.c: New test.
* gcc.target/aarch64/acle/ls64_st64bv0.c: New test.
* gcc.target/aarch64/acle/ls64_st64bv0-2.c: New test.
* gcc.target/aarch64/acle/ls64_st64bv0-3.c: New test.
* gcc.target/aarch64/pragma_cpp_predefs_2.c: Add checks
for __ARM_FEATURE_LS64.
This optimizes right shift rounding narrow instructions to
rounding add narrow high where one vector is 0 when the shift amount is half
that of the original input type.
i.e.
uint32x4_t foo (uint64x2_t a, uint64x2_t b)
{
return vrshrn_high_n_u64 (vrshrn_n_u64 (a, 32), b, 32);
}
now generates:
foo:
movi v3.4s, 0
raddhn v0.2s, v2.2d, v3.2d
raddhn2 v0.4s, v2.2d, v3.2d
instead of:
foo:
rshrn v0.2s, v0.2d, 32
rshrn2 v0.4s, v1.2d, 32
ret
On Arm cores this is an improvement in both latency and throughput.
Because a vector zero is needed I created a new method
aarch64_gen_shareable_zero that creates zeros using V4SI and then takes a subreg
of the zero to the desired type. This allows CSE to share all the zero
constants.
gcc/ChangeLog:
* config/aarch64/aarch64-protos.h (aarch64_gen_shareable_zero): New.
* config/aarch64/aarch64-simd.md (aarch64_rshrn<mode>,
aarch64_rshrn2<mode>): Generate rounding half-ing add when appropriate.
* config/aarch64/aarch64.c (aarch64_gen_shareable_zero): New.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/advsimd-intrinsics/shrn-1.c: New test.
* gcc.target/aarch64/advsimd-intrinsics/shrn-2.c: New test.
* gcc.target/aarch64/advsimd-intrinsics/shrn-3.c: New test.
* gcc.target/aarch64/advsimd-intrinsics/shrn-4.c: New test.
The problem here is aarch64_simd_dup<mode> use
the vw iterator rather than vwcore iterator. This causes
problems for the V4SF and V2DF modes. I changed both of
aarch64_simd_dup<mode> patterns to be consistent.
Committed as obvious after a bootstrap/test on aarch64-linux-gnu.
PR target/103170
gcc/ChangeLog:
* config/aarch64/aarch64-simd.md (aarch64_simd_dup<mode>):
Use vwcore iterator for the r constraint output string.
gcc/testsuite/ChangeLog:
* gcc.c-torture/compile/vector-dup-1.c: New test.
This removed the patterns to optimize the rounding shift and narrow.
The optimization is valid only for the truncating rounding shift and narrow,
for the rounding shift and narrow we need a different pattern that I will submit
separately.
This wasn't noticed before as the benchmarks did not run conformance as part of
the run, which we now do and this now passes again.
gcc/ChangeLog:
* config/aarch64/aarch64-simd.md (*aarch64_topbits_shuffle<mode>_le
,*aarch64_topbits_shuffle<mode>_be): Remove.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/shrn-combine-8.c: Update.
* gcc.target/aarch64/shrn-combine-9.c: Update.
There was some duplication between the maxmin_uns (uns for unspec
rather than unsigned) int attribute and the optab int attribute.
The difficulty for FMAXNM and FMINNM is that the instructions
really correspond to two things: the smax/smin optabs for floats
(used only for fast-math-like flags) and the fmax/fmin optabs
(used for built-in functions). The optab attribute was
consistently for the former but maxmin_uns had a mixture of both.
This patch renames maxmin_uns to fmaxmin and only uses it
for the fmax and fmin optabs. The reductions that previously
used the maxmin_uns attribute now use the optab attribute instead.
FMAX and FMIN are awkward in that they don't correspond to any
optab. It's nevertheless useful to define them alongside the
“real” optabs. Previously they were known as “smax_nan” and
“smin_nan”, but the problem with those names it that smax and
smin are only used for floats if NaNs don't matter. This patch
therefore uses fmax_nan and fmin_nan instead.
There is still some inconsistency, in that the optab attribute
handles UNSPEC_COND_FMAX but the fmaxmin attribute handles
UNSPEC_FMAX. This is because the SVE FP instructions, being
predicated, have to use unspecs in cases where the Advanced
SIMD ones could use rtl codes.
At least there are no duplicate entries though, so this seemed
like the best compromise for now.
gcc/
* config/aarch64/iterators.md (optab): Use fmax_nan instead of
smax_nan and fmin_nan instead of smin_nan.
(maxmin_uns): Rename to...
(fmaxmin): ...this and make the same changes. Remove entries
unrelated to fmax* and fmin*.
* config/aarch64/aarch64.md (<maxmin_uns><mode>3): Rename to...
(<fmaxmin><mode>3): ...this.
* config/aarch64/aarch64-simd.md (aarch64_<maxmin_uns>p<mode>):
Rename to...
(aarch64_<optab>p<mode>): ...this.
(<maxmin_uns><mode>3): Rename to...
(<fmaxmin><mode>3): ...this.
(reduc_<maxmin_uns>_scal_<mode>): Rename to...
(reduc_<optab>_scal_<mode>): ...this and update gen* call.
(aarch64_reduc_<maxmin_uns>_internal<mode>): Rename to...
(aarch64_reduc_<optab>_internal<mode>): ...this.
(aarch64_reduc_<maxmin_uns>_internalv2si): Rename to...
(aarch64_reduc_<optab>_internalv2si): ...this.
* config/aarch64/aarch64-sve.md (<maxmin_uns><mode>3): Rename to...
(<fmaxmin><mode>3): ...this.
* config/aarch64/aarch64-simd-builtins.def (smax_nan, smin_nan)
Rename to...
(fmax_nan, fmin_nan): ...this.
* config/aarch64/arm_neon.h (vmax_f32, vmax_f64, vmaxq_f32, vmaxq_f64)
(vmin_f32, vmin_f64, vminq_f32, vminq_f64, vmax_f16, vmaxq_f16)
(vmin_f16, vminq_f16): Update accordingly.
This patch adds extended costing to cost the creation of constants and the
manipulation of constants. The default values provided are based on
architectural expectations and each cost models can be individually tweaked as
needed.
The changes in this patch covers:
* Construction of PARALLEL or CONST_VECTOR:
Adds better costing for vector of constants which is based on the constant
being created and the instruction that can be used to create it. i.e. a movi
is cheaper than a literal load etc.
* Construction of a vector through a vec_dup.
gcc/ChangeLog:
* config/arm/aarch-common-protos.h (struct vector_cost_table): Add
movi, dup and extract costing fields.
* config/aarch64/aarch64-cost-tables.h (qdf24xx_extra_costs,
thunderx_extra_costs, thunderx2t99_extra_costs,
thunderx3t110_extra_costs, tsv110_extra_costs, a64fx_extra_costs): Use
them.
* config/arm/aarch-cost-tables.h (generic_extra_costs,
cortexa53_extra_costs, cortexa57_extra_costs, cortexa76_extra_costs,
exynosm1_extra_costs, xgene1_extra_costs): Likewise
* config/aarch64/aarch64-simd.md (aarch64_simd_dup<mode>): Add r->w dup.
* config/aarch64/aarch64.c (aarch64_rtx_costs): Add extra costs.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/vect-cse-codegen.c: New test.
This turns a bitwise inverse of an equality comparison with 0 into a compare of
bitwise nonzero (cmtst).
We already have one pattern for cmsts, this adds an additional one which does
not require an additional bitwise and.
i.e.
#include <arm_neon.h>
uint8x8_t bar(int16x8_t abs_row0, int16x8_t row0) {
uint16x8_t row0_diff =
vreinterpretq_u16_s16(veorq_s16(abs_row0, vshrq_n_s16(row0, 15)));
uint8x8_t abs_row0_gt0 =
vmovn_u16(vcgtq_u16(vreinterpretq_u16_s16(abs_row0), vdupq_n_u16(0)));
return abs_row0_gt0;
}
now generates:
bar:
cmtst v0.8h, v0.8h, v0.8h
xtn v0.8b, v0.8h
ret
instead of:
bar:
cmeq v0.8h, v0.8h, #0
not v0.16b, v0.16b
xtn v0.8b, v0.8h
ret
gcc/ChangeLog:
* config/aarch64/aarch64-simd.md (*aarch64_cmtst_same_<mode>): New.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/mvn-cmeq0-1.c: New test.
This turns truncate operations with a hi/lo pair into a single permute of half
the bit size of the input and just ignoring the top bits (which are truncated
out).
i.e.
void d2 (short * restrict a, int *b, int n)
{
for (int i = 0; i < n; i++)
a[i] = b[i];
}
now generates:
.L4:
ldp q0, q1, [x3]
add x3, x3, 32
uzp1 v0.8h, v0.8h, v1.8h
str q0, [x5], 16
cmp x4, x3
bne .L4
instead of
.L4:
ldp q0, q1, [x3]
add x3, x3, 32
xtn v0.4h, v0.4s
xtn2 v0.8h, v1.4s
str q0, [x5], 16
cmp x4, x3
bne .L4
gcc/ChangeLog:
* config/aarch64/aarch64-simd.md (*aarch64_narrow_trunc<mode>): New.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/narrow_high_combine.c: Update case.
* gcc.target/aarch64/xtn-combine-1.c: New test.
* gcc.target/aarch64/xtn-combine-2.c: New test.
* gcc.target/aarch64/xtn-combine-3.c: New test.
* gcc.target/aarch64/xtn-combine-4.c: New test.
* gcc.target/aarch64/xtn-combine-5.c: New test.
* gcc.target/aarch64/xtn-combine-6.c: New test.
This optimizes signed right shift by BITSIZE-1 into a cmlt operation which is
more optimal because generally compares have a higher throughput than shifts.
On AArch64 the result of the shift would have been either -1 or 0 which is the
results of the compare.
i.e.
void e (int * restrict a, int *b, int n)
{
for (int i = 0; i < n; i++)
b[i] = a[i] >> 31;
}
now generates:
.L4:
ldr q0, [x0, x3]
cmlt v0.4s, v0.4s, #0
str q0, [x1, x3]
add x3, x3, 16
cmp x4, x3
bne .L4
instead of:
.L4:
ldr q0, [x0, x3]
sshr v0.4s, v0.4s, 31
str q0, [x1, x3]
add x3, x3, 16
cmp x4, x3
bne .L4
Thanks,
Tamar
gcc/ChangeLog:
* config/aarch64/aarch64-simd.md (aarch64_simd_ashr<mode>): Add case cmp
case.
* config/aarch64/constraints.md (D1): New.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/shl-combine-2.c: New test.
* gcc.target/aarch64/shl-combine-3.c: New test.
* gcc.target/aarch64/shl-combine-4.c: New test.
* gcc.target/aarch64/shl-combine-5.c: New test.
When doing a (narrowing) right shift by half the width of the original type then
we are essentially shuffling the top bits from the first number down.
If we have a hi/lo pair we can just use a single shuffle instead of needing two
shifts.
i.e.
typedef short int16_t;
typedef unsigned short uint16_t;
void foo (uint16_t * restrict a, int16_t * restrict d, int n)
{
for( int i = 0; i < n; i++ )
d[i] = (a[i] * a[i]) >> 16;
}
now generates:
.L4:
ldr q0, [x0, x3]
umull v1.4s, v0.4h, v0.4h
umull2 v0.4s, v0.8h, v0.8h
uzp2 v0.8h, v1.8h, v0.8h
str q0, [x1, x3]
add x3, x3, 16
cmp x4, x3
bne .L4
instead of
.L4:
ldr q0, [x0, x3]
umull v1.4s, v0.4h, v0.4h
umull2 v0.4s, v0.8h, v0.8h
sshr v1.4s, v1.4s, 16
sshr v0.4s, v0.4s, 16
xtn v1.4h, v1.4s
xtn2 v1.8h, v0.4s
str q1, [x1, x3]
add x3, x3, 16
cmp x4, x3
bne .L4
Thanks,
Tamar
gcc/ChangeLog:
* config/aarch64/aarch64-simd.md
(*aarch64_<srn_op>topbits_shuffle<mode>_le): New.
(*aarch64_topbits_shuffle<mode>_le): New.
(*aarch64_<srn_op>topbits_shuffle<mode>_be): New.
(*aarch64_topbits_shuffle<mode>_be): New.
* config/aarch64/predicates.md
(aarch64_simd_shift_imm_vec_exact_top): New.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/shrn-combine-10.c: New test.
* gcc.target/aarch64/shrn-combine-5.c: New test.
* gcc.target/aarch64/shrn-combine-6.c: New test.
* gcc.target/aarch64/shrn-combine-7.c: New test.
* gcc.target/aarch64/shrn-combine-8.c: New test.
* gcc.target/aarch64/shrn-combine-9.c: New test.
Loops containing long long shifts fail to vectorize due to the vectorizer
not being able to recognize long long right shifts. This is due to a bug
in the iterator used for the vashr and vlshr patterns in aarch64-simd.md.
2021-08-09 Tejas Belagod <tejas.belagod@arm.com>
gcc/ChangeLog
PR target/101609
* config/aarch64/aarch64-simd.md (vlshr<mode>3, vashr<mode>3): Use
the right iterator.
gcc/testsuite/ChangeLog
* gcc.target/aarch64/vect-shr-reg.c: New testcase.
* gcc.target/aarch64/vect-shr-reg-run.c: Likewise.
As a general principle, vec_duplicate should be as close to the root
of an expression as possible. Where unary operations have
vec_duplicate as an argument, these operations should be pushed
inside the vec_duplicate.
This patch modifies unary operation simplification to push
sign/zero-extension of a scalar inside vec_duplicate.
This patch also updates all RTL patterns in aarch64-simd.md to use
the new canonical form.
gcc/ChangeLog:
2021-07-19 Jonathan Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd.md: Push sign/zero-extension
inside vec_duplicate for all patterns.
* simplify-rtx.c (simplify_context::simplify_unary_operation_1):
Push sign/zero-extension inside vec_duplicate.
The previous fix for this problem was wrong due to a subtle difference between
where NEON expects the RMW values and where intrinsics expects them.
The insn pattern is modeled after the intrinsics and so needs an expand for
the vectorizer optab to switch the RTL.
However operand[3] is not expected to be written to so the current pattern is
bogus.
Instead I rewrite the RTL to be in canonical ordering and merge them.
gcc/ChangeLog:
* config/aarch64/aarch64-simd-builtins.def (sdot, udot): Rename to..
(sdot_prod, udot_prod): ... This.
* config/aarch64/aarch64-simd.md (aarch64_<sur>dot<vsi2qi>): Merged
into...
(<sur>dot_prod<vsi2qi>): ... this.
(aarch64_<sur>dot_lane<vsi2qi>, aarch64_<sur>dot_laneq<vsi2qi>):
Change operands order.
(<sur>sadv16qi): Use new operands order.
* config/aarch64/arm_neon.h (vdot_u32, vdotq_u32, vdot_s32,
vdotq_s32): Use new RTL ordering.
There's a slight mismatch between the vectorizer optabs and the intrinsics
patterns for NEON. The vectorizer expects operands[3] and operands[0] to be
the same but the aarch64 intrinsics expanders expect operands[0] and
operands[1] to be the same.
This means we need different patterns here. This adds a separate usdot
vectorizer pattern which just shuffles around the RTL params.
There's also an inconsistency between the usdot and (u|s)dot intrinsics RTL
patterns which is not corrected here.
gcc/ChangeLog:
* config/aarch64/aarch64-builtins.c (TYPES_TERNOP_SUSS,
aarch64_types_ternop_suss_qualifiers): New.
* config/aarch64/aarch64-simd-builtins.def (usdot_prod): Use it.
* config/aarch64/aarch64-simd.md (usdot_prod<vsi2qi>): Re-organize RTL.
* config/aarch64/arm_neon.h (vusdot_s32, vusdotq_s32): Use it.
The current RTL for the vectorizer patterns for dot-product are incorrect.
Operand3 isn't an output parameter so we can't write to it.
This fixes this issue and reduces the number of RTL.
gcc/ChangeLog:
* config/aarch64/aarch64-simd-builtins.def (udot, sdot): Rename to...
(sdot_prod, udot_prod): ...These.
* config/aarch64/aarch64-simd.md (<sur>dot_prod<vsi2qi>): Remove.
(aarch64_<sur>dot<vsi2qi>): Rename to...
(<sur>dot_prod<vsi2qi>): ...This.
* config/aarch64/arm_neon.h (vdot_u32, vdotq_u32, vdot_s32, vdotq_s32):
Update builtins.
Model the zero-high-half semantics of the narrowing arithmetic Neon
instructions in the aarch64_<sur><addsub>hn<mode> RTL pattern.
Modeling these semantics allows for better RTL combinations while
also removing some register allocation issues as the compiler now
knows that the operation is totally destructive.
Add new tests to narrow_zero_high_half.c to verify the benefit of
this change.
gcc/ChangeLog:
2021-06-14 Jonathan Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd.md (aarch64_<sur><addsub>hn<mode>):
Change to an expander that emits the correct instruction
depending on endianness.
(aarch64_<sur><addsub>hn<mode>_insn_le): Define.
(aarch64_<sur><addsub>hn<mode>_insn_be): Define.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/narrow_zero_high_half.c: Add new tests.
Split the aarch64_<su>qmovn<mode> pattern into separate scalar and
vector variants. Further split the vector RTL pattern into big/
little endian variants that model the zero-high-half semantics of the
underlying instruction. Modeling these semantics allows for better
RTL combinations while also removing some register allocation issues
as the compiler now knows that the operation is totally destructive.
Add new tests to narrow_zero_high_half.c to verify the benefit of
this change.
gcc/ChangeLog:
2021-06-14 Jonathan Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd-builtins.def: Split generator
for aarch64_<su>qmovn builtins into scalar and vector
variants.
* config/aarch64/aarch64-simd.md (aarch64_<su>qmovn<mode>_insn_le):
Define.
(aarch64_<su>qmovn<mode>_insn_be): Define.
(aarch64_<su>qmovn<mode>): Split into scalar and vector
variants. Change vector variant to an expander that emits the
correct instruction depending on endianness.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/narrow_zero_high_half.c: Add new tests.
Split the aarch64_sqmovun<mode> pattern into separate scalar and
vector variants. Further split the vector pattern into big/little
endian variants that model the zero-high-half semantics of the
underlying instruction. Modeling these semantics allows for better
RTL combinations while also removing some register allocation issues
as the compiler now knows that the operation is totally destructive.
Add new tests to narrow_zero_high_half.c to verify the benefit of
this change.
gcc/ChangeLog:
2021-06-14 Jonathan Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd-builtins.def: Split generator
for aarch64_sqmovun builtins into scalar and vector variants.
* config/aarch64/aarch64-simd.md (aarch64_sqmovun<mode>):
Split into scalar and vector variants. Change vector variant
to an expander that emits the correct instruction depending
on endianness.
(aarch64_sqmovun<mode>_insn_le): Define.
(aarch64_sqmovun<mode>_insn_be): Define.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/narrow_zero_high_half.c: Add new tests.
Modeling the zero-high-half semantics of the XTN narrowing
instruction in RTL indicates to the compiler that this is a totally
destructive operation. This enables more RTL simplifications and also
prevents some register allocation issues.
Add new tests to narrow_zero_high_half.c to verify the benefit of
this change.
gcc/ChangeLog:
2021-06-11 Jonathan Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd.md (aarch64_xtn<mode>_insn_le):
Define - modeling zero-high-half semantics.
(aarch64_xtn<mode>): Change to an expander that emits the
appropriate instruction depending on endianness.
(aarch64_xtn<mode>_insn_be): Define - modeling zero-high-half
semantics.
(aarch64_xtn2<mode>_le): Rename to...
(aarch64_xtn2<mode>_insn_le): This.
(aarch64_xtn2<mode>_be): Rename to...
(aarch64_xtn2<mode>_insn_be): This.
(vec_pack_trunc_<mode>): Emit truncation instruction instead
of aarch64_xtn.
* config/aarch64/iterators.md (Vnarrowd): Add Vnarrowd mode
attribute iterator.
gcc/testsuite/ChangeLog:
* gcc.target/aarch64/narrow_zero_high_half.c: Add new tests.
Use the correct "neon_move_narrow_q" type attribute in RTL patterns
that generate XTN/XTN2 instructions.
This makes a material difference because these instructions can be
executed on both SIMD pipes in the Cortex-A57 core model, whereas the
"neon_shift_imm_narrow_q" attribute (in use until now) would suggest
to the scheduler that they could only execute on one of the two
pipes.
gcc/ChangeLog:
2021-05-18 Jonathan Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd.md: Use "neon_move_narrow_q"
type attribute in patterns generating XTN(2).
The existing vec_pack_trunc RTL pattern emits an opaque two-
instruction assembly code sequence that prevents proper instruction
scheduling. This commit changes the pattern to an expander that emits
individual xtn and xtn2 instructions.
This commit also consolidates the duplicate truncation patterns.
gcc/ChangeLog:
2021-05-17 Jonathan Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd.md (aarch64_simd_vec_pack_trunc_<mode>):
Remove as duplicate of...
(aarch64_xtn<mode>): This.
(aarch64_xtn2<mode>_le): Move position in file.
(aarch64_xtn2<mode>_be): Move position in file.
(aarch64_xtn2<mode>): Move position in file.
(vec_pack_trunc_<mode>): Define as an expander.
Split the aarch64_<sur>q<r>shr<u>n_n<mode> pattern into separate
scalar and vector variants. Further split the vector pattern into
big/little endian variants that model the zero-high-half semantics
of the underlying instruction - allowing for more combinations with
the write-to-high-half variant (aarch64_<sur>q<r>shr<u>n2_n<mode>.)
gcc/ChangeLog:
2021-05-14 Jonathan Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd-builtins.def: Split builtin
generation for aarch64_<sur>q<r>shr<u>n_n<mode> pattern into
separate scalar and vector generators.
* config/aarch64/aarch64-simd.md
(aarch64_<sur>q<r>shr<u>n_n<mode>): Define as an expander and
split into...
(aarch64_<sur>q<r>shr<u>n_n<mode>_insn_le): This and...
(aarch64_<sur>q<r>shr<u>n_n<mode>_insn_be): This.
* config/aarch64/iterators.md: Define SD_HSDI iterator.
Use UNSPEC_SQXTUN instead of UNSPEC_SQXTUN2 in aarch64_sqxtun2<mode>
patterns. This allows for more more aggressive combinations and
ultimately better code generation. The now redundant UNSPEC_SQXTUN2
is removed.
gcc/ChangeLog:
2021-05-14 Jonathn Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd.md: Use UNSPEC_SQXTUN instead
of UNSPEC_SQXTUN2.
* config/aarch64/iterators.md: Remove UNSPEC_SQXTUN2.
Implement saturating right-shift and narrow high Neon intrinsic RTL
patterns using a vec_concat of a register_operand and a VQSHRN_N
unspec - instead of just a VQSHRN_N unspec. This more relaxed pattern
allows for more aggressive combinations and ultimately better code
generation.
gcc/ChangeLog:
2021-03-04 Jonathan Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd.md (aarch64_<sur>q<r>shr<u>n2_n<mode>):
Implement as an expand emitting a big/little endian
instruction pattern.
(aarch64_<sur>q<r>shr<u>n2_n<mode>_insn_le): Define.
(aarch64_<sur>q<r>shr<u>n2_n<mode>_insn_be): Define.
Implement v[r]addhn2 and v[r]subhn2 Neon intrinsic RTL patterns using
a vec_concat of a register_operand and an ADDSUBHN unspec - instead
of just an ADDSUBHN2 unspec. This more relaxed pattern allows for
more aggressive combinations and ultimately better code generation.
This patch also removes the now redundant [R]ADDHN2 and [R]SUBHN2
unspecs and their iterator.
gcc/ChangeLog:
2021-03-03 Jonathan Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd.md (aarch64_<sur><addsub>hn2<mode>):
Implement as an expand emitting a big/little endian
instruction pattern.
(aarch64_<sur><addsub>hn2<mode>_insn_le): Define.
(aarch64_<sur><addsub>hn2<mode>_insn_be): Define.
* config/aarch64/iterators.md: Remove UNSPEC_[R]ADDHN2 and
UNSPEC_[R]SUBHN2 unspecs and ADDSUBHN2 iterator.
The sqdmlal2 patterns are hidden beneath the SBINQOPS iterator and unfortunately they don't match
canonical RTL because the simple accumulate operand comes in the first arm of the SS_PLUS.
This patch splits the SS_PLUS and SS_MINUS forms with the SS_PLUS operands set up to match
the canonical form, where the complex operand comes first.
gcc/ChangeLog:
* config/aarch64/aarch64-simd.md
(aarch64_sqdml<SBINQOPS:as>l2_lane<mode>_internal): Split into...
(aarch64_sqdmlsl2_lane<mode>_internal): ... This...
(aarch64_sqdmlal2_lane<mode>_internal): ... And this.
(aarch64_sqdml<SBINQOPS:as>l2_laneq<mode>_internal): Split into ...
(aarch64_sqdmlsl2_laneq<mode>_internal): ... This...
(aarch64_sqdmlal2_laneq<mode>_internal): ... And this.
(aarch64_sqdml<SBINQOPS:as>l2_n<mode>_internal): Split into...
(aarch64_sqdmlsl2_n<mode>_internal): ... This...
(aarch64_sqdmlal2_n<mode>_internal): ... And this.
The various sqdmlal2 and sqdmlsl2 expanders perform almost identical functions and can be
merged using code iterators and attributes to reduce the code in the MD file.
No behavioural change is expected.
gcc/ChangeLog:
* config/aarch64/aarch64-simd.md (aarch64_sqdmlal2<mode>): Merge into...
(aarch64_sqdml<SBINQOPS:as>l2<mode>): ... This.
(aarch64_sqdmlsl2<mode>): Delete.
(aarch64_sqdmlal2_lane<mode>): Merge this...
(aarch64_sqdmlsl2_lane<mode>): ... And this...
(aarch64_sqdml<SBINQOPS:as>l2_lane<mode>): ... Into this.
(aarch64_sqdmlal2_laneq<mode>): Merge this...
(aarch64_sqdmlsl2_laneq<mode>): ... And this...
(aarch64_sqdml<SBINQOPS:as>l2_laneq<mode>): ... Into this.
(aarch64_sqdmlal2_n<mode>): Merge this...
(aarch64_sqdmlsl2_n<mode>): ... And this...
(aarch64_sqdml<SBINQOPS:as>l2_n<mode>): ... Into this.
This patch removes the duplication between the mul_laneq<mode>3
and the older mul-lane patterns. The older patterns were previously
divided into two based on whether the indexed operand had the same mode
as the other operands or whether it had the opposite length from the
other operands (64-bit vs. 128-bit). However, it seemed easier to
divide them instead based on whether the indexed operand was 64-bit or
128-bit, since that maps directly to the arm_neon.h “q” conventions.
Also, it looks like the older patterns were missing cases for
V8HF<->V4HF combinations, which meant that vmul_laneq_f16 and
vmulq_lane_f16 didn't produce single instructions.
There was a typo in the V2SF entry for VCONQ, but in practice
no patterns were using that entry until now.
The test passes for both endiannesses, but endianness does change
the mapping between regexps and functions.
gcc/
* config/aarch64/iterators.md (VMUL_CHANGE_NLANES): Delete.
(VMULD): New iterator.
(VCOND): Handle V4HF and V8HF.
(VCONQ): Fix entry for V2SF.
* config/aarch64/aarch64-simd.md (mul_lane<mode>3): Use VMULD
instead of VMUL. Use a 64-bit vector mode for the indexed operand.
(*aarch64_mul3_elt_<vswap_width_name><mode>): Merge with...
(mul_laneq<mode>3): ...this define_insn. Use VMUL instead of VDQSF.
Use a 128-bit vector mode for the indexed operand. Use stype for
the scheduling type.
gcc/testsuite/
* gcc.target/aarch64/fmul_lane_1.c: New test.
Rewrite floating-point vml[as][q]_laneq Neon intrinsics to use RTL
builtins rather than relying on the GCC vector extensions. Using RTL
builtins allows control over the emission of fmla/fmls instructions
(which we don't want here.)
With this commit, the code generated by these intrinsics changes from
a fused multiply-add/subtract instruction to an fmul followed by an
fadd/fsub instruction. If the programmer really wants fmla/fmls
instructions, they can use the vfm[as] intrinsics.
gcc/ChangeLog:
2021-02-17 Jonathan Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd-builtins.def: Add
float_ml[as][q]_laneq builtin generator macros.
* config/aarch64/aarch64-simd.md (mul_laneq<mode>3): Define.
(aarch64_float_mla_laneq<mode>): Define.
(aarch64_float_mls_laneq<mode>): Define.
* config/aarch64/arm_neon.h (vmla_laneq_f32): Use RTL builtin
instead of GCC vector extensions.
(vmlaq_laneq_f32): Likewise.
(vmls_laneq_f32): Likewise.
(vmlsq_laneq_f32): Likewise.
Rewrite floating-point vml[as][q]_lane Neon intrinsics to use RTL
builtins rather than relying on the GCC vector extensions. Using RTL
builtins allows control over the emission of fmla/fmls instructions
(which we don't want here.)
With this commit, the code generated by these intrinsics changes from
a fused multiply-add/subtract instruction to an fmul followed by an
fadd/fsub instruction. If the programmer really wants fmla/fmls
instructions, they can use the vfm[as] intrinsics.
gcc/ChangeLog:
2021-02-16 Jonathan Wright <jonathan.wright@arm.com>
* config/aarch64/aarch64-simd-builtins.def: Add
float_ml[as]_lane builtin generator macros.
* config/aarch64/aarch64-simd.md (*aarch64_mul3_elt<mode>):
Rename to...
(mul_lane<mode>3): This, and re-order arguments.
(aarch64_float_mla_lane<mode>): Define.
(aarch64_float_mls_lane<mode>): Define.
* config/aarch64/arm_neon.h (vmla_lane_f32): Use RTL builtin
instead of GCC vector extensions.
(vmlaq_lane_f32): Likewise.
(vmls_lane_f32): Likewise.
(vmlsq_lane_f32): Likewise.
Tamar Christina
Richard Sandiford
Jakub Jelinek
Przemyslaw Wirkus
Andrew Pinski
Jonathan Wright
Tejas Belagod
Kyrylo Tkachov