linux/init
Nicholas Piggin 6290602709 mm: add PageWaiters indicating tasks are waiting for a page bit
Add a new page flag, PageWaiters, to indicate the page waitqueue has
tasks waiting. This can be tested rather than testing waitqueue_active
which requires another cacheline load.

This bit is always set when the page has tasks on page_waitqueue(page),
and is set and cleared under the waitqueue lock. It may be set when
there are no tasks on the waitqueue, which will cause a harmless extra
wakeup check that will clears the bit.

The generic bit-waitqueue infrastructure is no longer used for pages.
Instead, waitqueues are used directly with a custom key type. The
generic code was not flexible enough to have PageWaiters manipulation
under the waitqueue lock (which simplifies concurrency).

This improves the performance of page lock intensive microbenchmarks by
2-3%.

Putting two bits in the same word opens the opportunity to remove the
memory barrier between clearing the lock bit and testing the waiters
bit, after some work on the arch primitives (e.g., ensuring memory
operand widths match and cover both bits).

Signed-off-by: Nicholas Piggin <npiggin@gmail.com>
Cc: Dave Hansen <dave.hansen@linux.intel.com>
Cc: Bob Peterson <rpeterso@redhat.com>
Cc: Steven Whitehouse <swhiteho@redhat.com>
Cc: Andrew Lutomirski <luto@kernel.org>
Cc: Andreas Gruenbacher <agruenba@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Mel Gorman <mgorman@techsingularity.net>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2016-12-25 11:54:48 -08:00
..
Kconfig Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2016-12-17 20:17:04 -08:00
Makefile kbuild: allow archs to select link dead code/data elimination 2016-09-09 10:47:00 +02:00
calibrate.c kernel: add calibration_delay_done() 2014-06-16 12:47:39 -06:00
do_mounts.c init: reduce rootwait polling interval time to 5ms 2016-12-12 18:55:10 -08:00
do_mounts.h init/do_mounts: initrd_load() can be boolean 2016-01-20 17:09:18 -08:00
do_mounts_initrd.c init/do_mounts: initrd_load() can be boolean 2016-01-20 17:09:18 -08:00
do_mounts_md.c init: disable sparse checking of the mount.o source files 2012-05-31 17:49:27 -07:00
do_mounts_rd.c init: use pr_cont() when displaying rotator during ramdisk loading. 2016-11-24 09:32:20 -08:00
init_task.c Replace <asm/uaccess.h> with <linux/uaccess.h> globally 2016-12-24 11:46:01 -08:00
initramfs.c kexec: split kexec_load syscall from kexec core code 2015-09-10 13:29:01 -07:00
main.c mm: add PageWaiters indicating tasks are waiting for a page bit 2016-12-25 11:54:48 -08:00
noinitramfs.c init: mark __user address space on string literals 2010-10-26 16:52:15 -07:00
version.c copy address of proc_ns_ops into ns_common 2014-12-04 14:34:47 -05:00