linux/arch/arm64/kernel
Will Deacon 8e86f0b409 arm64: atomics: fix use of acquire + release for full barrier semantics
Linux requires a number of atomic operations to provide full barrier
semantics, that is no memory accesses after the operation can be
observed before any accesses up to and including the operation in
program order.

On arm64, these operations have been incorrectly implemented as follows:

	// A, B, C are independent memory locations

	<Access [A]>

	// atomic_op (B)
1:	ldaxr	x0, [B]		// Exclusive load with acquire
	<op(B)>
	stlxr	w1, x0, [B]	// Exclusive store with release
	cbnz	w1, 1b

	<Access [C]>

The assumption here being that two half barriers are equivalent to a
full barrier, so the only permitted ordering would be A -> B -> C
(where B is the atomic operation involving both a load and a store).

Unfortunately, this is not the case by the letter of the architecture
and, in fact, the accesses to A and C are permitted to pass their
nearest half barrier resulting in orderings such as Bl -> A -> C -> Bs
or Bl -> C -> A -> Bs (where Bl is the load-acquire on B and Bs is the
store-release on B). This is a clear violation of the full barrier
requirement.

The simple way to fix this is to implement the same algorithm as ARMv7
using explicit barriers:

	<Access [A]>

	// atomic_op (B)
	dmb	ish		// Full barrier
1:	ldxr	x0, [B]		// Exclusive load
	<op(B)>
	stxr	w1, x0, [B]	// Exclusive store
	cbnz	w1, 1b
	dmb	ish		// Full barrier

	<Access [C]>

but this has the undesirable effect of introducing *two* full barrier
instructions. A better approach is actually the following, non-intuitive
sequence:

	<Access [A]>

	// atomic_op (B)
1:	ldxr	x0, [B]		// Exclusive load
	<op(B)>
	stlxr	w1, x0, [B]	// Exclusive store with release
	cbnz	w1, 1b
	dmb	ish		// Full barrier

	<Access [C]>

The simple observations here are:

  - The dmb ensures that no subsequent accesses (e.g. the access to C)
    can enter or pass the atomic sequence.

  - The dmb also ensures that no prior accesses (e.g. the access to A)
    can pass the atomic sequence.

  - Therefore, no prior access can pass a subsequent access, or
    vice-versa (i.e. A is strictly ordered before C).

  - The stlxr ensures that no prior access can pass the store component
    of the atomic operation.

The only tricky part remaining is the ordering between the ldxr and the
access to A, since the absence of the first dmb means that we're now
permitting re-ordering between the ldxr and any prior accesses.

From an (arbitrary) observer's point of view, there are two scenarios:

  1. We have observed the ldxr. This means that if we perform a store to
     [B], the ldxr will still return older data. If we can observe the
     ldxr, then we can potentially observe the permitted re-ordering
     with the access to A, which is clearly an issue when compared to
     the dmb variant of the code. Thankfully, the exclusive monitor will
     save us here since it will be cleared as a result of the store and
     the ldxr will retry. Notice that any use of a later memory
     observation to imply observation of the ldxr will also imply
     observation of the access to A, since the stlxr/dmb ensure strict
     ordering.

  2. We have not observed the ldxr. This means we can perform a store
     and influence the later ldxr. However, that doesn't actually tell
     us anything about the access to [A], so we've not lost anything
     here either when compared to the dmb variant.

This patch implements this solution for our barriered atomic operations,
ensuring that we satisfy the full barrier requirements where they are
needed.

Cc: <stable@vger.kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Signed-off-by: Will Deacon <will.deacon@arm.com>
Signed-off-by: Catalin Marinas <catalin.marinas@arm.com>
2014-02-07 16:45:43 +00:00
..
vdso arm64: vdso: fix coarse clock handling 2014-02-05 11:55:30 +00:00
.gitignore arm64: Build infrastructure 2012-09-17 13:42:21 +01:00
Makefile arm64, jump label: optimize jump label implementation 2014-01-08 15:23:53 +00:00
arm64ksyms.c arm64: use generic strnlen_user and strncpy_from_user functions 2013-12-19 17:43:06 +00:00
asm-offsets.c arm64: kernel: cpu_{suspend/resume} implementation 2013-12-16 17:17:31 +00:00
cpu_ops.c arm64: Slightly improve the warning on CPU0 enable-method 2013-10-31 16:37:26 +00:00
cputable.c arm64: add CPU_HOTPLUG infrastructure 2013-10-25 11:33:21 +01:00
debug-monitors.c arm64: support single-step and breakpoint handler hooks 2013-12-19 17:43:11 +00:00
early_printk.c arm64: Fix duplicate definition of early_console 2013-05-13 11:44:53 +01:00
entry-fpsimd.S arm64: move FP-SIMD save/restore code to a macro 2012-12-05 11:26:50 +00:00
entry.S arm64: fix typo in entry.S 2014-01-13 13:55:13 +00:00
fpsimd.c arm64: kernel: implement fpsimd CPU PM notifier 2013-12-16 17:17:32 +00:00
head.S arm64: Remove unused __data_loc variable 2013-12-20 12:04:48 +00:00
hw_breakpoint.c arm64: kernel: restore HW breakpoint registers in cpu_suspend 2014-01-10 17:51:35 +00:00
hyp-stub.S arm64: add hypervisor stub 2012-12-05 11:26:49 +00:00
insn.c arm64: introduce aarch64_insn_gen_{nop|branch_imm}() helper functions 2014-01-08 15:21:29 +00:00
io.c arm64: Device specific operations 2012-09-17 13:42:04 +01:00
irq.c arm64: add CPU_HOTPLUG infrastructure 2013-10-25 11:33:21 +01:00
jump_label.c arm64, jump label: optimize jump label implementation 2014-01-08 15:23:53 +00:00
kuser32.S arm64: atomics: fix use of acquire + release for full barrier semantics 2014-02-07 16:45:43 +00:00
module.c arm64: move encode_insn_immediate() from module.c to insn.c 2014-01-08 15:21:29 +00:00
perf_event.c arm64: perf: add support for percpu pmu interrupt 2013-12-19 17:43:05 +00:00
process.c arm64: FIQs are unused 2014-01-30 13:51:43 +00:00
psci.c arm64: add PSCI CPU_OFF-based hotplug support 2013-10-25 11:33:21 +01:00
ptrace.c arm64: ptrace: avoid using HW_BREAKPOINT_EMPTY for disabled events 2013-12-19 17:41:25 +00:00
setup.c Merge tag 'arm64-suspend' of git://linux-arm.org/linux-2.6-lp into upstream 2013-12-19 17:57:51 +00:00
signal.c arm64: switch to generic sigaltstack 2013-02-14 09:17:29 -05:00
signal32.c Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs 2013-11-13 15:34:18 +09:00
sleep.S arm64: kernel: cpu_{suspend/resume} implementation 2013-12-16 17:17:31 +00:00
smp.c arm64: FIQs are unused 2014-01-30 13:51:43 +00:00
smp_spin_table.c arm64: big-endian: write CPU holding pen address as LE 2013-10-25 15:59:42 +01:00
stacktrace.c ARM64: fix framepointer check in unwind_frame 2013-12-19 17:43:10 +00:00
suspend.c arm64: kernel: fix per-cpu offset restore on resume 2014-01-24 14:27:40 +00:00
sys.c arm64: switch to generic sigaltstack 2013-02-14 09:17:29 -05:00
sys32.S arm64: compat: correct register concatenation for syscall wrappers 2013-10-25 15:59:36 +01:00
sys_compat.c compat: generic compat_sys_sched_rr_get_interval() implementation 2012-12-17 17:15:18 -08:00
time.c arch_timer: Move to generic sched_clock framework 2013-10-09 16:54:10 -07:00
traps.c arm64: debug: consolidate software breakpoint handlers 2013-06-12 11:23:02 +01:00
vdso.c arm64: vdso: update wtm fields for CLOCK_MONOTONIC_COARSE 2014-02-05 11:55:49 +00:00
vmlinux.lds.S arm64: Remove unused __data_loc variable 2013-12-20 12:04:48 +00:00