2005-04-17 00:20:36 +02:00
|
|
|
/*
|
|
|
|
* Read-Copy Update mechanism for mutual exclusion
|
|
|
|
*
|
|
|
|
* This program is free software; you can redistribute it and/or modify
|
|
|
|
* it under the terms of the GNU General Public License as published by
|
|
|
|
* the Free Software Foundation; either version 2 of the License, or
|
|
|
|
* (at your option) any later version.
|
|
|
|
*
|
|
|
|
* This program is distributed in the hope that it will be useful,
|
|
|
|
* but WITHOUT ANY WARRANTY; without even the implied warranty of
|
|
|
|
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
|
|
|
|
* GNU General Public License for more details.
|
|
|
|
*
|
|
|
|
* You should have received a copy of the GNU General Public License
|
|
|
|
* along with this program; if not, write to the Free Software
|
|
|
|
* Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.
|
|
|
|
*
|
2008-01-25 21:08:24 +01:00
|
|
|
* Copyright IBM Corporation, 2001
|
2005-04-17 00:20:36 +02:00
|
|
|
*
|
|
|
|
* Authors: Dipankar Sarma <dipankar@in.ibm.com>
|
|
|
|
* Manfred Spraul <manfred@colorfullife.com>
|
|
|
|
*
|
|
|
|
* Based on the original work by Paul McKenney <paulmck@us.ibm.com>
|
|
|
|
* and inputs from Rusty Russell, Andrea Arcangeli and Andi Kleen.
|
|
|
|
* Papers:
|
|
|
|
* http://www.rdrop.com/users/paulmck/paper/rclockpdcsproof.pdf
|
|
|
|
* http://lse.sourceforge.net/locking/rclock_OLS.2001.05.01c.sc.pdf (OLS2001)
|
|
|
|
*
|
|
|
|
* For detailed explanation of Read-Copy Update mechanism see -
|
|
|
|
* http://lse.sourceforge.net/locking/rcupdate.html
|
|
|
|
*
|
|
|
|
*/
|
|
|
|
#include <linux/types.h>
|
|
|
|
#include <linux/kernel.h>
|
|
|
|
#include <linux/init.h>
|
|
|
|
#include <linux/spinlock.h>
|
|
|
|
#include <linux/smp.h>
|
|
|
|
#include <linux/interrupt.h>
|
|
|
|
#include <linux/sched.h>
|
|
|
|
#include <asm/atomic.h>
|
|
|
|
#include <linux/bitops.h>
|
|
|
|
#include <linux/percpu.h>
|
|
|
|
#include <linux/notifier.h>
|
|
|
|
#include <linux/cpu.h>
|
2006-03-23 12:00:19 +01:00
|
|
|
#include <linux/mutex.h>
|
2008-01-25 21:08:24 +01:00
|
|
|
#include <linux/module.h>
|
rcu: Teach RCU that idle task is not quiscent state at boot
This patch fixes a bug located by Vegard Nossum with the aid of
kmemcheck, updated based on review comments from Nick Piggin,
Ingo Molnar, and Andrew Morton. And cleans up the variable-name
and function-name language. ;-)
The boot CPU runs in the context of its idle thread during boot-up.
During this time, idle_cpu(0) will always return nonzero, which will
fool Classic and Hierarchical RCU into deciding that a large chunk of
the boot-up sequence is a big long quiescent state. This in turn causes
RCU to prematurely end grace periods during this time.
This patch changes the rcutree.c and rcuclassic.c rcu_check_callbacks()
function to ignore the idle task as a quiescent state until the
system has started up the scheduler in rest_init(), introducing a
new non-API function rcu_idle_now_means_idle() to inform RCU of this
transition. RCU maintains an internal rcu_idle_cpu_truthful variable
to track this state, which is then used by rcu_check_callback() to
determine if it should believe idle_cpu().
Because this patch has the effect of disallowing RCU grace periods
during long stretches of the boot-up sequence, this patch also introduces
Josh Triplett's UP-only optimization that makes synchronize_rcu() be a
no-op if num_online_cpus() returns 1. This allows boot-time code that
calls synchronize_rcu() to proceed normally. Note, however, that RCU
callbacks registered by call_rcu() will likely queue up until later in
the boot sequence. Although rcuclassic and rcutree can also use this
same optimization after boot completes, rcupreempt must restrict its
use of this optimization to the portion of the boot sequence before the
scheduler starts up, given that an rcupreempt RCU read-side critical
section may be preeempted.
In addition, this patch takes Nick Piggin's suggestion to make the
system_state global variable be __read_mostly.
Changes since v4:
o Changes the name of the introduced function and variable to
be less emotional. ;-)
Changes since v3:
o WARN_ON(nr_context_switches() > 0) to verify that RCU
switches out of boot-time mode before the first context
switch, as suggested by Nick Piggin.
Changes since v2:
o Created rcu_blocking_is_gp() internal-to-RCU API that
determines whether a call to synchronize_rcu() is itself
a grace period.
o The definition of rcu_blocking_is_gp() for rcuclassic and
rcutree checks to see if but a single CPU is online.
o The definition of rcu_blocking_is_gp() for rcupreempt
checks to see both if but a single CPU is online and if
the system is still in early boot.
This allows rcupreempt to again work correctly if running
on a single CPU after booting is complete.
o Added check to rcupreempt's synchronize_sched() for there
being but one online CPU.
Tested all three variants both SMP and !SMP, booted fine, passed a short
rcutorture test on both x86 and Power.
Located-by: Vegard Nossum <vegard.nossum@gmail.com>
Tested-by: Vegard Nossum <vegard.nossum@gmail.com>
Tested-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-02-26 03:03:42 +01:00
|
|
|
#include <linux/kernel_stat.h>
|
2005-04-17 00:20:36 +02:00
|
|
|
|
2008-05-12 21:21:05 +02:00
|
|
|
enum rcu_barrier {
|
|
|
|
RCU_BARRIER_STD,
|
|
|
|
RCU_BARRIER_BH,
|
|
|
|
RCU_BARRIER_SCHED,
|
|
|
|
};
|
|
|
|
|
2008-01-25 21:08:24 +01:00
|
|
|
static DEFINE_PER_CPU(struct rcu_head, rcu_barrier_head) = {NULL};
|
2006-03-08 06:55:33 +01:00
|
|
|
static atomic_t rcu_barrier_cpu_count;
|
2006-03-23 12:00:19 +01:00
|
|
|
static DEFINE_MUTEX(rcu_barrier_mutex);
|
2006-03-08 06:55:33 +01:00
|
|
|
static struct completion rcu_barrier_completion;
|
rcu: Teach RCU that idle task is not quiscent state at boot
This patch fixes a bug located by Vegard Nossum with the aid of
kmemcheck, updated based on review comments from Nick Piggin,
Ingo Molnar, and Andrew Morton. And cleans up the variable-name
and function-name language. ;-)
The boot CPU runs in the context of its idle thread during boot-up.
During this time, idle_cpu(0) will always return nonzero, which will
fool Classic and Hierarchical RCU into deciding that a large chunk of
the boot-up sequence is a big long quiescent state. This in turn causes
RCU to prematurely end grace periods during this time.
This patch changes the rcutree.c and rcuclassic.c rcu_check_callbacks()
function to ignore the idle task as a quiescent state until the
system has started up the scheduler in rest_init(), introducing a
new non-API function rcu_idle_now_means_idle() to inform RCU of this
transition. RCU maintains an internal rcu_idle_cpu_truthful variable
to track this state, which is then used by rcu_check_callback() to
determine if it should believe idle_cpu().
Because this patch has the effect of disallowing RCU grace periods
during long stretches of the boot-up sequence, this patch also introduces
Josh Triplett's UP-only optimization that makes synchronize_rcu() be a
no-op if num_online_cpus() returns 1. This allows boot-time code that
calls synchronize_rcu() to proceed normally. Note, however, that RCU
callbacks registered by call_rcu() will likely queue up until later in
the boot sequence. Although rcuclassic and rcutree can also use this
same optimization after boot completes, rcupreempt must restrict its
use of this optimization to the portion of the boot sequence before the
scheduler starts up, given that an rcupreempt RCU read-side critical
section may be preeempted.
In addition, this patch takes Nick Piggin's suggestion to make the
system_state global variable be __read_mostly.
Changes since v4:
o Changes the name of the introduced function and variable to
be less emotional. ;-)
Changes since v3:
o WARN_ON(nr_context_switches() > 0) to verify that RCU
switches out of boot-time mode before the first context
switch, as suggested by Nick Piggin.
Changes since v2:
o Created rcu_blocking_is_gp() internal-to-RCU API that
determines whether a call to synchronize_rcu() is itself
a grace period.
o The definition of rcu_blocking_is_gp() for rcuclassic and
rcutree checks to see if but a single CPU is online.
o The definition of rcu_blocking_is_gp() for rcupreempt
checks to see both if but a single CPU is online and if
the system is still in early boot.
This allows rcupreempt to again work correctly if running
on a single CPU after booting is complete.
o Added check to rcupreempt's synchronize_sched() for there
being but one online CPU.
Tested all three variants both SMP and !SMP, booted fine, passed a short
rcutorture test on both x86 and Power.
Located-by: Vegard Nossum <vegard.nossum@gmail.com>
Tested-by: Vegard Nossum <vegard.nossum@gmail.com>
Tested-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-02-26 03:03:42 +01:00
|
|
|
int rcu_scheduler_active __read_mostly;
|
2006-03-08 06:55:33 +01:00
|
|
|
|
2009-04-15 20:35:01 +02:00
|
|
|
static atomic_t rcu_migrate_type_count = ATOMIC_INIT(0);
|
|
|
|
static struct rcu_head rcu_migrate_head[3];
|
|
|
|
static DECLARE_WAIT_QUEUE_HEAD(rcu_migrate_wq);
|
|
|
|
|
2008-02-14 00:03:15 +01:00
|
|
|
/*
|
|
|
|
* Awaken the corresponding synchronize_rcu() instance now that a
|
|
|
|
* grace period has elapsed.
|
|
|
|
*/
|
rcu: add call_rcu_sched()
Fourth cut of patch to provide the call_rcu_sched(). This is again to
synchronize_sched() as call_rcu() is to synchronize_rcu().
Should be fine for experimental and -rt use, but not ready for inclusion.
With some luck, I will be able to tell Andrew to come out of hiding on
the next round.
Passes multi-day rcutorture sessions with concurrent CPU hotplugging.
Fixes since the first version include a bug that could result in
indefinite blocking (spotted by Gautham Shenoy), better resiliency
against CPU-hotplug operations, and other minor fixes.
Fixes since the second version include reworking grace-period detection
to avoid deadlocks that could happen when running concurrently with
CPU hotplug, adding Mathieu's fix to avoid the softlockup messages,
as well as Mathieu's fix to allow use earlier in boot.
Fixes since the third version include a wrong-CPU bug spotted by
Andrew, getting rid of the obsolete synchronize_kernel API that somehow
snuck back in, merging spin_unlock() and local_irq_restore() in a
few places, commenting the code that checks for quiescent states based
on interrupting from user-mode execution or the idle loop, removing
some inline attributes, and some code-style changes.
Known/suspected shortcomings:
o I still do not entirely trust the sleep/wakeup logic. Next step
will be to use a private snapshot of the CPU online mask in
rcu_sched_grace_period() -- if the CPU wasn't there at the start
of the grace period, we don't need to hear from it. And the
bit about accounting for changes in online CPUs inside of
rcu_sched_grace_period() is ugly anyway.
o It might be good for rcu_sched_grace_period() to invoke
resched_cpu() when a given CPU wasn't responding quickly,
but resched_cpu() is declared static...
This patch also fixes a long-standing bug in the earlier preemptable-RCU
implementation of synchronize_rcu() that could result in loss of
concurrent external changes to a task's CPU affinity mask. I still cannot
remember who reported this...
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Mathieu Desnoyers <mathieu.desnoyers@polymtl.ca>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
2008-05-12 21:21:05 +02:00
|
|
|
void wakeme_after_rcu(struct rcu_head *head)
|
2006-03-08 06:55:33 +01:00
|
|
|
{
|
2008-01-25 21:08:24 +01:00
|
|
|
struct rcu_synchronize *rcu;
|
|
|
|
|
|
|
|
rcu = container_of(head, struct rcu_synchronize, head);
|
|
|
|
complete(&rcu->completion);
|
2006-03-08 06:55:33 +01:00
|
|
|
}
|
2005-04-17 00:20:36 +02:00
|
|
|
|
|
|
|
/**
|
2008-01-25 21:08:24 +01:00
|
|
|
* synchronize_rcu - wait until a grace period has elapsed.
|
2005-04-17 00:20:36 +02:00
|
|
|
*
|
2008-01-25 21:08:24 +01:00
|
|
|
* Control will return to the caller some time after a full grace
|
|
|
|
* period has elapsed, in other words after all currently executing RCU
|
2005-04-17 00:20:36 +02:00
|
|
|
* read-side critical sections have completed. RCU read-side critical
|
|
|
|
* sections are delimited by rcu_read_lock() and rcu_read_unlock(),
|
|
|
|
* and may be nested.
|
|
|
|
*/
|
2009-01-04 22:03:02 +01:00
|
|
|
void synchronize_rcu(void)
|
|
|
|
{
|
|
|
|
struct rcu_synchronize rcu;
|
rcu: Teach RCU that idle task is not quiscent state at boot
This patch fixes a bug located by Vegard Nossum with the aid of
kmemcheck, updated based on review comments from Nick Piggin,
Ingo Molnar, and Andrew Morton. And cleans up the variable-name
and function-name language. ;-)
The boot CPU runs in the context of its idle thread during boot-up.
During this time, idle_cpu(0) will always return nonzero, which will
fool Classic and Hierarchical RCU into deciding that a large chunk of
the boot-up sequence is a big long quiescent state. This in turn causes
RCU to prematurely end grace periods during this time.
This patch changes the rcutree.c and rcuclassic.c rcu_check_callbacks()
function to ignore the idle task as a quiescent state until the
system has started up the scheduler in rest_init(), introducing a
new non-API function rcu_idle_now_means_idle() to inform RCU of this
transition. RCU maintains an internal rcu_idle_cpu_truthful variable
to track this state, which is then used by rcu_check_callback() to
determine if it should believe idle_cpu().
Because this patch has the effect of disallowing RCU grace periods
during long stretches of the boot-up sequence, this patch also introduces
Josh Triplett's UP-only optimization that makes synchronize_rcu() be a
no-op if num_online_cpus() returns 1. This allows boot-time code that
calls synchronize_rcu() to proceed normally. Note, however, that RCU
callbacks registered by call_rcu() will likely queue up until later in
the boot sequence. Although rcuclassic and rcutree can also use this
same optimization after boot completes, rcupreempt must restrict its
use of this optimization to the portion of the boot sequence before the
scheduler starts up, given that an rcupreempt RCU read-side critical
section may be preeempted.
In addition, this patch takes Nick Piggin's suggestion to make the
system_state global variable be __read_mostly.
Changes since v4:
o Changes the name of the introduced function and variable to
be less emotional. ;-)
Changes since v3:
o WARN_ON(nr_context_switches() > 0) to verify that RCU
switches out of boot-time mode before the first context
switch, as suggested by Nick Piggin.
Changes since v2:
o Created rcu_blocking_is_gp() internal-to-RCU API that
determines whether a call to synchronize_rcu() is itself
a grace period.
o The definition of rcu_blocking_is_gp() for rcuclassic and
rcutree checks to see if but a single CPU is online.
o The definition of rcu_blocking_is_gp() for rcupreempt
checks to see both if but a single CPU is online and if
the system is still in early boot.
This allows rcupreempt to again work correctly if running
on a single CPU after booting is complete.
o Added check to rcupreempt's synchronize_sched() for there
being but one online CPU.
Tested all three variants both SMP and !SMP, booted fine, passed a short
rcutorture test on both x86 and Power.
Located-by: Vegard Nossum <vegard.nossum@gmail.com>
Tested-by: Vegard Nossum <vegard.nossum@gmail.com>
Tested-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-02-26 03:03:42 +01:00
|
|
|
|
|
|
|
if (rcu_blocking_is_gp())
|
|
|
|
return;
|
|
|
|
|
2009-01-04 22:03:02 +01:00
|
|
|
init_completion(&rcu.completion);
|
|
|
|
/* Will wake me after RCU finished. */
|
|
|
|
call_rcu(&rcu.head, wakeme_after_rcu);
|
|
|
|
/* Wait for it. */
|
|
|
|
wait_for_completion(&rcu.completion);
|
|
|
|
}
|
2008-01-25 21:08:24 +01:00
|
|
|
EXPORT_SYMBOL_GPL(synchronize_rcu);
|
2006-06-27 11:54:04 +02:00
|
|
|
|
2009-06-25 18:08:16 +02:00
|
|
|
/**
|
|
|
|
* synchronize_rcu_bh - wait until an rcu_bh grace period has elapsed.
|
|
|
|
*
|
|
|
|
* Control will return to the caller some time after a full rcu_bh grace
|
|
|
|
* period has elapsed, in other words after all currently executing rcu_bh
|
|
|
|
* read-side critical sections have completed. RCU read-side critical
|
|
|
|
* sections are delimited by rcu_read_lock_bh() and rcu_read_unlock_bh(),
|
|
|
|
* and may be nested.
|
|
|
|
*/
|
|
|
|
void synchronize_rcu_bh(void)
|
|
|
|
{
|
|
|
|
struct rcu_synchronize rcu;
|
|
|
|
|
|
|
|
if (rcu_blocking_is_gp())
|
|
|
|
return;
|
|
|
|
|
|
|
|
init_completion(&rcu.completion);
|
|
|
|
/* Will wake me after RCU finished. */
|
|
|
|
call_rcu_bh(&rcu.head, wakeme_after_rcu);
|
|
|
|
/* Wait for it. */
|
|
|
|
wait_for_completion(&rcu.completion);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(synchronize_rcu_bh);
|
|
|
|
|
2005-12-12 09:37:05 +01:00
|
|
|
static void rcu_barrier_callback(struct rcu_head *notused)
|
|
|
|
{
|
|
|
|
if (atomic_dec_and_test(&rcu_barrier_cpu_count))
|
|
|
|
complete(&rcu_barrier_completion);
|
|
|
|
}
|
|
|
|
|
|
|
|
/*
|
|
|
|
* Called with preemption disabled, and from cross-cpu IRQ context.
|
|
|
|
*/
|
2008-05-12 21:21:05 +02:00
|
|
|
static void rcu_barrier_func(void *type)
|
2005-12-12 09:37:05 +01:00
|
|
|
{
|
|
|
|
int cpu = smp_processor_id();
|
2008-01-25 21:08:24 +01:00
|
|
|
struct rcu_head *head = &per_cpu(rcu_barrier_head, cpu);
|
2005-12-12 09:37:05 +01:00
|
|
|
|
|
|
|
atomic_inc(&rcu_barrier_cpu_count);
|
2008-05-12 21:21:05 +02:00
|
|
|
switch ((enum rcu_barrier)type) {
|
|
|
|
case RCU_BARRIER_STD:
|
|
|
|
call_rcu(head, rcu_barrier_callback);
|
|
|
|
break;
|
|
|
|
case RCU_BARRIER_BH:
|
|
|
|
call_rcu_bh(head, rcu_barrier_callback);
|
|
|
|
break;
|
|
|
|
case RCU_BARRIER_SCHED:
|
|
|
|
call_rcu_sched(head, rcu_barrier_callback);
|
|
|
|
break;
|
|
|
|
}
|
2005-12-12 09:37:05 +01:00
|
|
|
}
|
|
|
|
|
2009-04-15 20:35:01 +02:00
|
|
|
static inline void wait_migrated_callbacks(void)
|
|
|
|
{
|
|
|
|
wait_event(rcu_migrate_wq, !atomic_read(&rcu_migrate_type_count));
|
2009-06-25 18:08:16 +02:00
|
|
|
smp_mb(); /* In case we didn't sleep. */
|
2009-04-15 20:35:01 +02:00
|
|
|
}
|
rcu: rcu_barrier VS cpu_hotplug: Ensure callbacks in dead cpu are migrated to online cpu
cpu hotplug may happen asynchronously, some rcu callbacks are maybe
still on dead cpu, rcu_barrier() also needs to wait for these rcu
callbacks to complete, so we must ensure callbacks in dead cpu are
migrated to online cpu.
Paul E. McKenney's review:
Good stuff, Lai!!! Simpler than any of the approaches that I was
considering, and, better yet, independent of the underlying RCU
implementation!!!
I was initially worried that wake_up() might wake only one of two
possible wait_event()s, namely rcu_barrier() and the CPU_POST_DEAD code,
but the fact that wait_event() clears WQ_FLAG_EXCLUSIVE avoids that issue.
I was also worried about the fact that different RCU implementations have
different mappings of call_rcu(), call_rcu_bh(), and call_rcu_sched(), but
this is OK as well because we just get an extra (harmless) callback in the
case that they map together (for example, Classic RCU has call_rcu_sched()
mapping to call_rcu()).
Overlap of CPU-hotplug operations is prevented by cpu_add_remove_lock,
and any stray callbacks that arrive (for example, from irq handlers
running on the dying CPU) either are ahead of the CPU_DYING callbacks on
the one hand (and thus accounted for), or happened after the rcu_barrier()
started on the other (and thus don't need to be accounted for).
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
LKML-Reference: <49C36476.1010400@cn.fujitsu.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-03-20 10:40:06 +01:00
|
|
|
|
2008-05-12 21:21:05 +02:00
|
|
|
/*
|
|
|
|
* Orchestrate the specified type of RCU barrier, waiting for all
|
|
|
|
* RCU callbacks of the specified type to complete.
|
2005-12-12 09:37:05 +01:00
|
|
|
*/
|
2008-05-12 21:21:05 +02:00
|
|
|
static void _rcu_barrier(enum rcu_barrier type)
|
2005-12-12 09:37:05 +01:00
|
|
|
{
|
|
|
|
BUG_ON(in_interrupt());
|
2006-03-23 12:00:19 +01:00
|
|
|
/* Take cpucontrol mutex to protect against CPU hotplug */
|
|
|
|
mutex_lock(&rcu_barrier_mutex);
|
2005-12-12 09:37:05 +01:00
|
|
|
init_completion(&rcu_barrier_completion);
|
2008-01-25 21:08:24 +01:00
|
|
|
/*
|
rcupdate: fix bug of rcu_barrier*()
current rcu_barrier_bh() is like this:
void rcu_barrier_bh(void)
{
BUG_ON(in_interrupt());
/* Take cpucontrol mutex to protect against CPU hotplug */
mutex_lock(&rcu_barrier_mutex);
init_completion(&rcu_barrier_completion);
atomic_set(&rcu_barrier_cpu_count, 0);
/*
* The queueing of callbacks in all CPUs must be atomic with
* respect to RCU, otherwise one CPU may queue a callback,
* wait for a grace period, decrement barrier count and call
* complete(), while other CPUs have not yet queued anything.
* So, we need to make sure that grace periods cannot complete
* until all the callbacks are queued.
*/
rcu_read_lock();
on_each_cpu(rcu_barrier_func, (void *)RCU_BARRIER_BH, 1);
rcu_read_unlock();
wait_for_completion(&rcu_barrier_completion);
mutex_unlock(&rcu_barrier_mutex);
}
The inconsistency of the code and the comments show a bug here.
rcu_read_lock() cannot make sure that "grace periods for RCU_BH
cannot complete until all the callbacks are queued".
it only make sure that race periods for RCU cannot complete
until all the callbacks are queued.
so we must use rcu_read_lock_bh() for rcu_barrier_bh().
like this:
void rcu_barrier_bh(void)
{
......
rcu_read_lock_bh();
on_each_cpu(rcu_barrier_func, (void *)RCU_BARRIER_BH, 1);
rcu_read_unlock_bh();
......
}
and also rcu_barrier() rcu_barrier_sched() are implemented like this.
it will bring a lot of duplicate code. My patch uses another way to
fix this bug, please see the comment of my patch.
Thank Paul E. McKenney for he rewrote the comment.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-10-17 08:40:30 +02:00
|
|
|
* Initialize rcu_barrier_cpu_count to 1, then invoke
|
|
|
|
* rcu_barrier_func() on each CPU, so that each CPU also has
|
|
|
|
* incremented rcu_barrier_cpu_count. Only then is it safe to
|
|
|
|
* decrement rcu_barrier_cpu_count -- otherwise the first CPU
|
|
|
|
* might complete its grace period before all of the other CPUs
|
|
|
|
* did their increment, causing this function to return too
|
|
|
|
* early.
|
2008-01-25 21:08:24 +01:00
|
|
|
*/
|
rcupdate: fix bug of rcu_barrier*()
current rcu_barrier_bh() is like this:
void rcu_barrier_bh(void)
{
BUG_ON(in_interrupt());
/* Take cpucontrol mutex to protect against CPU hotplug */
mutex_lock(&rcu_barrier_mutex);
init_completion(&rcu_barrier_completion);
atomic_set(&rcu_barrier_cpu_count, 0);
/*
* The queueing of callbacks in all CPUs must be atomic with
* respect to RCU, otherwise one CPU may queue a callback,
* wait for a grace period, decrement barrier count and call
* complete(), while other CPUs have not yet queued anything.
* So, we need to make sure that grace periods cannot complete
* until all the callbacks are queued.
*/
rcu_read_lock();
on_each_cpu(rcu_barrier_func, (void *)RCU_BARRIER_BH, 1);
rcu_read_unlock();
wait_for_completion(&rcu_barrier_completion);
mutex_unlock(&rcu_barrier_mutex);
}
The inconsistency of the code and the comments show a bug here.
rcu_read_lock() cannot make sure that "grace periods for RCU_BH
cannot complete until all the callbacks are queued".
it only make sure that race periods for RCU cannot complete
until all the callbacks are queued.
so we must use rcu_read_lock_bh() for rcu_barrier_bh().
like this:
void rcu_barrier_bh(void)
{
......
rcu_read_lock_bh();
on_each_cpu(rcu_barrier_func, (void *)RCU_BARRIER_BH, 1);
rcu_read_unlock_bh();
......
}
and also rcu_barrier() rcu_barrier_sched() are implemented like this.
it will bring a lot of duplicate code. My patch uses another way to
fix this bug, please see the comment of my patch.
Thank Paul E. McKenney for he rewrote the comment.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-10-17 08:40:30 +02:00
|
|
|
atomic_set(&rcu_barrier_cpu_count, 1);
|
2008-07-15 23:02:33 +02:00
|
|
|
on_each_cpu(rcu_barrier_func, (void *)type, 1);
|
rcupdate: fix bug of rcu_barrier*()
current rcu_barrier_bh() is like this:
void rcu_barrier_bh(void)
{
BUG_ON(in_interrupt());
/* Take cpucontrol mutex to protect against CPU hotplug */
mutex_lock(&rcu_barrier_mutex);
init_completion(&rcu_barrier_completion);
atomic_set(&rcu_barrier_cpu_count, 0);
/*
* The queueing of callbacks in all CPUs must be atomic with
* respect to RCU, otherwise one CPU may queue a callback,
* wait for a grace period, decrement barrier count and call
* complete(), while other CPUs have not yet queued anything.
* So, we need to make sure that grace periods cannot complete
* until all the callbacks are queued.
*/
rcu_read_lock();
on_each_cpu(rcu_barrier_func, (void *)RCU_BARRIER_BH, 1);
rcu_read_unlock();
wait_for_completion(&rcu_barrier_completion);
mutex_unlock(&rcu_barrier_mutex);
}
The inconsistency of the code and the comments show a bug here.
rcu_read_lock() cannot make sure that "grace periods for RCU_BH
cannot complete until all the callbacks are queued".
it only make sure that race periods for RCU cannot complete
until all the callbacks are queued.
so we must use rcu_read_lock_bh() for rcu_barrier_bh().
like this:
void rcu_barrier_bh(void)
{
......
rcu_read_lock_bh();
on_each_cpu(rcu_barrier_func, (void *)RCU_BARRIER_BH, 1);
rcu_read_unlock_bh();
......
}
and also rcu_barrier() rcu_barrier_sched() are implemented like this.
it will bring a lot of duplicate code. My patch uses another way to
fix this bug, please see the comment of my patch.
Thank Paul E. McKenney for he rewrote the comment.
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2008-10-17 08:40:30 +02:00
|
|
|
if (atomic_dec_and_test(&rcu_barrier_cpu_count))
|
|
|
|
complete(&rcu_barrier_completion);
|
2005-12-12 09:37:05 +01:00
|
|
|
wait_for_completion(&rcu_barrier_completion);
|
2006-03-23 12:00:19 +01:00
|
|
|
mutex_unlock(&rcu_barrier_mutex);
|
rcu: rcu_barrier VS cpu_hotplug: Ensure callbacks in dead cpu are migrated to online cpu
cpu hotplug may happen asynchronously, some rcu callbacks are maybe
still on dead cpu, rcu_barrier() also needs to wait for these rcu
callbacks to complete, so we must ensure callbacks in dead cpu are
migrated to online cpu.
Paul E. McKenney's review:
Good stuff, Lai!!! Simpler than any of the approaches that I was
considering, and, better yet, independent of the underlying RCU
implementation!!!
I was initially worried that wake_up() might wake only one of two
possible wait_event()s, namely rcu_barrier() and the CPU_POST_DEAD code,
but the fact that wait_event() clears WQ_FLAG_EXCLUSIVE avoids that issue.
I was also worried about the fact that different RCU implementations have
different mappings of call_rcu(), call_rcu_bh(), and call_rcu_sched(), but
this is OK as well because we just get an extra (harmless) callback in the
case that they map together (for example, Classic RCU has call_rcu_sched()
mapping to call_rcu()).
Overlap of CPU-hotplug operations is prevented by cpu_add_remove_lock,
and any stray callbacks that arrive (for example, from irq handlers
running on the dying CPU) either are ahead of the CPU_DYING callbacks on
the one hand (and thus accounted for), or happened after the rcu_barrier()
started on the other (and thus don't need to be accounted for).
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
LKML-Reference: <49C36476.1010400@cn.fujitsu.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-03-20 10:40:06 +01:00
|
|
|
wait_migrated_callbacks();
|
2005-12-12 09:37:05 +01:00
|
|
|
}
|
2008-05-12 21:21:05 +02:00
|
|
|
|
|
|
|
/**
|
|
|
|
* rcu_barrier - Wait until all in-flight call_rcu() callbacks complete.
|
|
|
|
*/
|
|
|
|
void rcu_barrier(void)
|
|
|
|
{
|
|
|
|
_rcu_barrier(RCU_BARRIER_STD);
|
|
|
|
}
|
2005-12-12 09:37:05 +01:00
|
|
|
EXPORT_SYMBOL_GPL(rcu_barrier);
|
|
|
|
|
2008-05-12 21:21:05 +02:00
|
|
|
/**
|
|
|
|
* rcu_barrier_bh - Wait until all in-flight call_rcu_bh() callbacks complete.
|
|
|
|
*/
|
|
|
|
void rcu_barrier_bh(void)
|
|
|
|
{
|
|
|
|
_rcu_barrier(RCU_BARRIER_BH);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(rcu_barrier_bh);
|
|
|
|
|
|
|
|
/**
|
|
|
|
* rcu_barrier_sched - Wait for in-flight call_rcu_sched() callbacks.
|
|
|
|
*/
|
|
|
|
void rcu_barrier_sched(void)
|
|
|
|
{
|
|
|
|
_rcu_barrier(RCU_BARRIER_SCHED);
|
|
|
|
}
|
|
|
|
EXPORT_SYMBOL_GPL(rcu_barrier_sched);
|
|
|
|
|
rcu: rcu_barrier VS cpu_hotplug: Ensure callbacks in dead cpu are migrated to online cpu
cpu hotplug may happen asynchronously, some rcu callbacks are maybe
still on dead cpu, rcu_barrier() also needs to wait for these rcu
callbacks to complete, so we must ensure callbacks in dead cpu are
migrated to online cpu.
Paul E. McKenney's review:
Good stuff, Lai!!! Simpler than any of the approaches that I was
considering, and, better yet, independent of the underlying RCU
implementation!!!
I was initially worried that wake_up() might wake only one of two
possible wait_event()s, namely rcu_barrier() and the CPU_POST_DEAD code,
but the fact that wait_event() clears WQ_FLAG_EXCLUSIVE avoids that issue.
I was also worried about the fact that different RCU implementations have
different mappings of call_rcu(), call_rcu_bh(), and call_rcu_sched(), but
this is OK as well because we just get an extra (harmless) callback in the
case that they map together (for example, Classic RCU has call_rcu_sched()
mapping to call_rcu()).
Overlap of CPU-hotplug operations is prevented by cpu_add_remove_lock,
and any stray callbacks that arrive (for example, from irq handlers
running on the dying CPU) either are ahead of the CPU_DYING callbacks on
the one hand (and thus accounted for), or happened after the rcu_barrier()
started on the other (and thus don't need to be accounted for).
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
LKML-Reference: <49C36476.1010400@cn.fujitsu.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-03-20 10:40:06 +01:00
|
|
|
static void rcu_migrate_callback(struct rcu_head *notused)
|
|
|
|
{
|
|
|
|
if (atomic_dec_and_test(&rcu_migrate_type_count))
|
|
|
|
wake_up(&rcu_migrate_wq);
|
|
|
|
}
|
|
|
|
|
2009-08-15 18:53:48 +02:00
|
|
|
extern int rcu_cpu_notify(struct notifier_block *self,
|
|
|
|
unsigned long action, void *hcpu);
|
|
|
|
|
rcu: rcu_barrier VS cpu_hotplug: Ensure callbacks in dead cpu are migrated to online cpu
cpu hotplug may happen asynchronously, some rcu callbacks are maybe
still on dead cpu, rcu_barrier() also needs to wait for these rcu
callbacks to complete, so we must ensure callbacks in dead cpu are
migrated to online cpu.
Paul E. McKenney's review:
Good stuff, Lai!!! Simpler than any of the approaches that I was
considering, and, better yet, independent of the underlying RCU
implementation!!!
I was initially worried that wake_up() might wake only one of two
possible wait_event()s, namely rcu_barrier() and the CPU_POST_DEAD code,
but the fact that wait_event() clears WQ_FLAG_EXCLUSIVE avoids that issue.
I was also worried about the fact that different RCU implementations have
different mappings of call_rcu(), call_rcu_bh(), and call_rcu_sched(), but
this is OK as well because we just get an extra (harmless) callback in the
case that they map together (for example, Classic RCU has call_rcu_sched()
mapping to call_rcu()).
Overlap of CPU-hotplug operations is prevented by cpu_add_remove_lock,
and any stray callbacks that arrive (for example, from irq handlers
running on the dying CPU) either are ahead of the CPU_DYING callbacks on
the one hand (and thus accounted for), or happened after the rcu_barrier()
started on the other (and thus don't need to be accounted for).
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
LKML-Reference: <49C36476.1010400@cn.fujitsu.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-03-20 10:40:06 +01:00
|
|
|
static int __cpuinit rcu_barrier_cpu_hotplug(struct notifier_block *self,
|
|
|
|
unsigned long action, void *hcpu)
|
|
|
|
{
|
2009-08-15 18:53:48 +02:00
|
|
|
rcu_cpu_notify(self, action, hcpu);
|
rcu: rcu_barrier VS cpu_hotplug: Ensure callbacks in dead cpu are migrated to online cpu
cpu hotplug may happen asynchronously, some rcu callbacks are maybe
still on dead cpu, rcu_barrier() also needs to wait for these rcu
callbacks to complete, so we must ensure callbacks in dead cpu are
migrated to online cpu.
Paul E. McKenney's review:
Good stuff, Lai!!! Simpler than any of the approaches that I was
considering, and, better yet, independent of the underlying RCU
implementation!!!
I was initially worried that wake_up() might wake only one of two
possible wait_event()s, namely rcu_barrier() and the CPU_POST_DEAD code,
but the fact that wait_event() clears WQ_FLAG_EXCLUSIVE avoids that issue.
I was also worried about the fact that different RCU implementations have
different mappings of call_rcu(), call_rcu_bh(), and call_rcu_sched(), but
this is OK as well because we just get an extra (harmless) callback in the
case that they map together (for example, Classic RCU has call_rcu_sched()
mapping to call_rcu()).
Overlap of CPU-hotplug operations is prevented by cpu_add_remove_lock,
and any stray callbacks that arrive (for example, from irq handlers
running on the dying CPU) either are ahead of the CPU_DYING callbacks on
the one hand (and thus accounted for), or happened after the rcu_barrier()
started on the other (and thus don't need to be accounted for).
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
LKML-Reference: <49C36476.1010400@cn.fujitsu.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-03-20 10:40:06 +01:00
|
|
|
if (action == CPU_DYING) {
|
|
|
|
/*
|
|
|
|
* preempt_disable() in on_each_cpu() prevents stop_machine(),
|
|
|
|
* so when "on_each_cpu(rcu_barrier_func, (void *)type, 1);"
|
|
|
|
* returns, all online cpus have queued rcu_barrier_func(),
|
|
|
|
* and the dead cpu(if it exist) queues rcu_migrate_callback()s.
|
|
|
|
*
|
|
|
|
* These callbacks ensure _rcu_barrier() waits for all
|
|
|
|
* RCU callbacks of the specified type to complete.
|
|
|
|
*/
|
|
|
|
atomic_set(&rcu_migrate_type_count, 3);
|
|
|
|
call_rcu_bh(rcu_migrate_head, rcu_migrate_callback);
|
|
|
|
call_rcu_sched(rcu_migrate_head + 1, rcu_migrate_callback);
|
|
|
|
call_rcu(rcu_migrate_head + 2, rcu_migrate_callback);
|
2009-08-19 08:06:14 +02:00
|
|
|
} else if (action == CPU_DOWN_PREPARE) {
|
|
|
|
/* Don't need to wait until next removal operation. */
|
rcu: rcu_barrier VS cpu_hotplug: Ensure callbacks in dead cpu are migrated to online cpu
cpu hotplug may happen asynchronously, some rcu callbacks are maybe
still on dead cpu, rcu_barrier() also needs to wait for these rcu
callbacks to complete, so we must ensure callbacks in dead cpu are
migrated to online cpu.
Paul E. McKenney's review:
Good stuff, Lai!!! Simpler than any of the approaches that I was
considering, and, better yet, independent of the underlying RCU
implementation!!!
I was initially worried that wake_up() might wake only one of two
possible wait_event()s, namely rcu_barrier() and the CPU_POST_DEAD code,
but the fact that wait_event() clears WQ_FLAG_EXCLUSIVE avoids that issue.
I was also worried about the fact that different RCU implementations have
different mappings of call_rcu(), call_rcu_bh(), and call_rcu_sched(), but
this is OK as well because we just get an extra (harmless) callback in the
case that they map together (for example, Classic RCU has call_rcu_sched()
mapping to call_rcu()).
Overlap of CPU-hotplug operations is prevented by cpu_add_remove_lock,
and any stray callbacks that arrive (for example, from irq handlers
running on the dying CPU) either are ahead of the CPU_DYING callbacks on
the one hand (and thus accounted for), or happened after the rcu_barrier()
started on the other (and thus don't need to be accounted for).
Signed-off-by: Lai Jiangshan <laijs@cn.fujitsu.com>
Reviewed-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
LKML-Reference: <49C36476.1010400@cn.fujitsu.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-03-20 10:40:06 +01:00
|
|
|
/* rcu_migrate_head is protected by cpu_add_remove_lock */
|
|
|
|
wait_migrated_callbacks();
|
|
|
|
}
|
|
|
|
|
|
|
|
return NOTIFY_OK;
|
|
|
|
}
|
|
|
|
|
2005-04-17 00:20:36 +02:00
|
|
|
void __init rcu_init(void)
|
|
|
|
{
|
2009-08-15 18:53:48 +02:00
|
|
|
int i;
|
|
|
|
|
2008-01-25 21:08:24 +01:00
|
|
|
__rcu_init();
|
2009-08-15 18:53:48 +02:00
|
|
|
cpu_notifier(rcu_barrier_cpu_hotplug, 0);
|
|
|
|
|
|
|
|
/*
|
|
|
|
* We don't need protection against CPU-hotplug here because
|
|
|
|
* this is called early in boot, before either interrupts
|
|
|
|
* or the scheduler are operational.
|
|
|
|
*/
|
|
|
|
for_each_online_cpu(i)
|
|
|
|
rcu_barrier_cpu_hotplug(NULL, CPU_UP_PREPARE, (void *)(long)i);
|
2005-04-17 00:20:36 +02:00
|
|
|
}
|
|
|
|
|
rcu: Teach RCU that idle task is not quiscent state at boot
This patch fixes a bug located by Vegard Nossum with the aid of
kmemcheck, updated based on review comments from Nick Piggin,
Ingo Molnar, and Andrew Morton. And cleans up the variable-name
and function-name language. ;-)
The boot CPU runs in the context of its idle thread during boot-up.
During this time, idle_cpu(0) will always return nonzero, which will
fool Classic and Hierarchical RCU into deciding that a large chunk of
the boot-up sequence is a big long quiescent state. This in turn causes
RCU to prematurely end grace periods during this time.
This patch changes the rcutree.c and rcuclassic.c rcu_check_callbacks()
function to ignore the idle task as a quiescent state until the
system has started up the scheduler in rest_init(), introducing a
new non-API function rcu_idle_now_means_idle() to inform RCU of this
transition. RCU maintains an internal rcu_idle_cpu_truthful variable
to track this state, which is then used by rcu_check_callback() to
determine if it should believe idle_cpu().
Because this patch has the effect of disallowing RCU grace periods
during long stretches of the boot-up sequence, this patch also introduces
Josh Triplett's UP-only optimization that makes synchronize_rcu() be a
no-op if num_online_cpus() returns 1. This allows boot-time code that
calls synchronize_rcu() to proceed normally. Note, however, that RCU
callbacks registered by call_rcu() will likely queue up until later in
the boot sequence. Although rcuclassic and rcutree can also use this
same optimization after boot completes, rcupreempt must restrict its
use of this optimization to the portion of the boot sequence before the
scheduler starts up, given that an rcupreempt RCU read-side critical
section may be preeempted.
In addition, this patch takes Nick Piggin's suggestion to make the
system_state global variable be __read_mostly.
Changes since v4:
o Changes the name of the introduced function and variable to
be less emotional. ;-)
Changes since v3:
o WARN_ON(nr_context_switches() > 0) to verify that RCU
switches out of boot-time mode before the first context
switch, as suggested by Nick Piggin.
Changes since v2:
o Created rcu_blocking_is_gp() internal-to-RCU API that
determines whether a call to synchronize_rcu() is itself
a grace period.
o The definition of rcu_blocking_is_gp() for rcuclassic and
rcutree checks to see if but a single CPU is online.
o The definition of rcu_blocking_is_gp() for rcupreempt
checks to see both if but a single CPU is online and if
the system is still in early boot.
This allows rcupreempt to again work correctly if running
on a single CPU after booting is complete.
o Added check to rcupreempt's synchronize_sched() for there
being but one online CPU.
Tested all three variants both SMP and !SMP, booted fine, passed a short
rcutorture test on both x86 and Power.
Located-by: Vegard Nossum <vegard.nossum@gmail.com>
Tested-by: Vegard Nossum <vegard.nossum@gmail.com>
Tested-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Signed-off-by: Ingo Molnar <mingo@elte.hu>
2009-02-26 03:03:42 +01:00
|
|
|
void rcu_scheduler_starting(void)
|
|
|
|
{
|
|
|
|
WARN_ON(num_online_cpus() != 1);
|
|
|
|
WARN_ON(nr_context_switches() > 0);
|
|
|
|
rcu_scheduler_active = 1;
|
|
|
|
}
|