linux/kernel/rcu
Neeraj Upadhyay ef0dcab6d2 rcu: Allow only one expedited GP to run concurrently with wakeups
commit 4bc6b745e5 upstream.

The current expedited RCU grace-period code expects that a task
requesting an expedited grace period cannot awaken until that grace
period has reached the wakeup phase.  However, it is possible for a long
preemption to result in the waiting task never sleeping.  For example,
consider the following sequence of events:

1.	Task A starts an expedited grace period by invoking
	synchronize_rcu_expedited().  It proceeds normally up to the
	wait_event() near the end of that function, and is then preempted
	(or interrupted or whatever).

2.	The expedited grace period completes, and a kworker task starts
	the awaken phase, having incremented the counter and acquired
	the rcu_state structure's .exp_wake_mutex.  This kworker task
	is then preempted or interrupted or whatever.

3.	Task A resumes and enters wait_event(), which notes that the
	expedited grace period has completed, and thus doesn't sleep.

4.	Task B starts an expedited grace period exactly as did Task A,
	complete with the preemption (or whatever delay) just before
	the call to wait_event().

5.	The expedited grace period completes, and another kworker
	task starts the awaken phase, having incremented the counter.
	However, it blocks when attempting to acquire the rcu_state
	structure's .exp_wake_mutex because step 2's kworker task has
	not yet released it.

6.	Steps 4 and 5 repeat, resulting in overflow of the rcu_node
	structure's ->exp_wq[] array.

In theory, this is harmless.  Tasks waiting on the various ->exp_wq[]
array will just be spuriously awakened, but they will just sleep again
on noting that the rcu_state structure's ->expedited_sequence value has
not advanced far enough.

In practice, this wastes CPU time and is an accident waiting to happen.
This commit therefore moves the rcu_exp_gp_seq_end() call that officially
ends the expedited grace period (along with associate tracing) until
after the ->exp_wake_mutex has been acquired.  This prevents Task A from
awakening prematurely, thus preventing more than one expedited grace
period from being in flight during a previous expedited grace period's
wakeup phase.

Fixes: 3b5f668e71 ("rcu: Overlap wakeups with next expedited grace period")
Signed-off-by: Neeraj Upadhyay <neeraju@codeaurora.org>
[ paulmck: Added updated comment. ]
Signed-off-by: Paul E. McKenney <paulmck@kernel.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2020-03-05 16:43:50 +01:00
..
Kconfig rcu: Use CONFIG_PREEMPTION 2019-07-31 19:03:35 +02:00
Kconfig.debug rcu: Add support for consolidated-RCU reader checking 2019-08-09 11:00:35 -07:00
Makefile License cleanup: add SPDX GPL-2.0 license identifier to files with no license 2017-11-02 11:10:55 +01:00
rcu.h rcu: Add kernel parameter to dump trace after RCU CPU stall warning 2019-08-01 14:05:51 -07:00
rcu_segcblist.c rcu/nocb: Add bypass callback queueing 2019-08-13 14:37:32 -07:00
rcu_segcblist.h rcu/nocb: Add bypass callback queueing 2019-08-13 14:37:32 -07:00
rcuperf.c rcuperf: Make rcuperf kernel test more robust for !expedited mode 2019-08-01 14:30:22 -07:00
rcutorture.c rcu/nocb: Print no-CBs diagnostics when rcutorture writer unduly delayed 2019-08-13 14:38:24 -07:00
srcutiny.c srcu: Remove cleanup_srcu_struct_quiesced() 2019-03-26 14:39:24 -07:00
srcutree.c srcu: Apply *_ONCE() to ->srcu_last_gp_end 2020-02-11 04:35:08 -08:00
sync.c rcu/sync: Simplify the state machine 2019-05-28 09:05:23 -07:00
tiny.c rcu: rcu_qs -- Use raise_softirq_irqoff to not save irqs twice 2019-03-26 14:37:49 -07:00
tree.c rcu: Fix data-race due to atomic_t copy-by-value 2020-02-24 08:36:23 +01:00
tree.h rcu/nocb: Print no-CBs diagnostics when rcutorture writer unduly delayed 2019-08-13 14:38:24 -07:00
tree_exp.h rcu: Allow only one expedited GP to run concurrently with wakeups 2020-03-05 16:43:50 +01:00
tree_plugin.h rcu/nocb: Fix dump_tree hierarchy print always active 2020-02-24 08:36:22 +01:00
tree_stall.h Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2019-09-16 17:25:49 -07:00
update.c Merge branches 'consolidate.2019.08.01b', 'fixes.2019.08.12a', 'lists.2019.08.13a' and 'torture.2019.08.01b' into HEAD 2019-08-13 14:30:30 -07:00