OpenE2K/linux
6
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
9607a85b67
linux/lib
History
Davidlohr Bueso 9607a85b67 rwsem: check counter to avoid cmpxchg calls 
This patch tries to reduce the amount of cmpxchg calls in the writer
failed path by checking the counter value first before issuing the
instruction.  If ->count is not set to RWSEM_WAITING_BIAS then there is
no point wasting a cmpxchg call.

Furthermore, Michel states "I suppose it helps due to the case where
someone else steals the lock while we're trying to acquire
sem->wait_lock."

Two very different workloads and machines were used to see how this
patch improves throughput: pgbench on a quad-core laptop and aim7 on a
large 8 socket box with 80 cores.

Some results comparing Michel's fast-path write lock stealing
(tps-rwsem) on a quad-core laptop running pgbench:

  | db_size | clients  |  tps-rwsem     |   tps-patch  |
  +---------+----------+----------------+--------------+
  | 160 MB   |       1 |           6906 |         9153 | + 32.5
  | 160 MB   |       2 |          15931 |        22487 | + 41.1%
  | 160 MB   |       4 |          33021 |        32503 |
  | 160 MB   |       8 |          34626 |        34695 |
  | 160 MB   |      16 |          33098 |        34003 |
  | 160 MB   |      20 |          31343 |        31440 |
  | 160 MB   |      30 |          28961 |        28987 |
  | 160 MB   |      40 |          26902 |        26970 |
  | 160 MB   |      50 |          25760 |        25810 |
  ------------------------------------------------------
  | 1.6 GB   |       1 |           7729 |         7537 |
  | 1.6 GB   |       2 |          19009 |        23508 | + 23.7%
  | 1.6 GB   |       4 |          33185 |        32666 |
  | 1.6 GB   |       8 |          34550 |        34318 |
  | 1.6 GB   |      16 |          33079 |        32689 |
  | 1.6 GB   |      20 |          31494 |        31702 |
  | 1.6 GB   |      30 |          28535 |        28755 |
  | 1.6 GB   |      40 |          27054 |        27017 |
  | 1.6 GB   |      50 |          25591 |        25560 |
  ------------------------------------------------------
  | 7.6 GB   |       1 |           6224 |         7469 | + 20.0%
  | 7.6 GB   |       2 |          13611 |        12778 |
  | 7.6 GB   |       4 |          33108 |        32927 |
  | 7.6 GB   |       8 |          34712 |        34878 |
  | 7.6 GB   |      16 |          32895 |        33003 |
  | 7.6 GB   |      20 |          31689 |        31974 |
  | 7.6 GB   |      30 |          29003 |        28806 |
  | 7.6 GB   |      40 |          26683 |        26976 |
  | 7.6 GB   |      50 |          25925 |        25652 |
  ------------------------------------------------------

For the aim7 worloads, they overall improved on top of Michel's
patchset.  For full graphs on how the rwsem series plus this patch
behaves on a large 8 socket machine against a vanilla kernel:

  http://stgolabs.net/rwsem-aim7-results.tar.gz

Signed-off-by: Davidlohr Bueso <davidlohr.bueso@hp.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2013-05-07 16:11:51 -07:00
..
lzo lib/lzo: Update LZO compression to current upstream version 2013-02-20 19:36:01 +01:00
mpi mpilib: use DIV_ROUND_UP and remove unused macros 2013-02-01 16:28:32 +11:00
raid6 lib/raid6: build proper files on corresponding arch 2012-12-13 19:51:04 +11:00
reed_solomon lib: Remove unnecessary inclusions of asm/semaphore.h 2008-04-18 22:17:17 -04:00
xz decompressors: fix typo "POWERPC" 2013-03-13 15:21:48 -07:00
zlib_deflate zlib: slim down zlib_deflate() workspace when possible 2011-03-22 17:44:17 -07:00
zlib_inflate inflate_fast: sout is already a short so ptr arith was off by one. 2010-03-12 15:52:44 -08:00
.gitignore X.509: Implement simple static OID registry 2012-10-08 13:50:18 +10:30
argv_split.c argv_split(): teach it to handle mutable strings 2013-04-29 18:28:19 -07:00
asn1_decoder.c Nothing all that exciting; a new module-from-fd syscall for those who want 2012-12-19 07:55:08 -08:00
atomic64_test.c atomic64_test: simplify the #ifdef for atomic64_dec_if_positive() test 2012-07-30 17:25:16 -07:00
atomic64.c lib: atomic64: Initialize locks statically to fix early users 2012-12-20 13:50:16 -08:00
audit.c audit: support the "standard" <asm-generic/unistd.h> 2011-05-04 14:41:28 -04:00
average.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
bcd.c usb/core: use bin2bcd() for bcdDevice in RH 2012-09-10 11:13:16 -07:00
bch.c lib: add shared BCH ECC library 2011-03-11 14:25:50 +00:00
bitmap.c propagate name change to comments in kernel source 2012-12-06 10:39:54 +01:00
bitrev.c lib: export bitrev16 2008-06-06 11:29:10 -07:00
bsearch.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
btree.c btree: catch NULL value before it does harm 2012-06-07 14:43:55 -07:00
bug.c taint: add explicit flag to show whether lock dep is still OK. 2013-01-21 17:17:57 +10:30
build_OID_registry X.509: Implement simple static OID registry 2012-10-08 13:50:18 +10:30
bust_spinlocks.c printk: Provide a wake_up_klogd() off-case 2013-03-22 16:41:20 -07:00
check_signature.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
checksum.c asm-generic headers: Allow yet more arch overrides in checksum.h 2013-02-11 20:00:33 +05:30
clz_tab.c lib: Fix multiple definitions of clz_tab 2012-02-02 10:34:23 +11:00
cmdline.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
cordic.c Docs: wording: functions -> algorithm 2011-10-29 21:20:22 +02:00
cpu_rmap.c lib: cpu_rmap: avoid flushing all workqueues 2013-01-11 14:54:54 -08:00
cpu-notifier-error-inject.c cpu: rewrite cpu-notifier-error-inject module 2012-07-30 17:25:22 -07:00
cpumask.c bootmem: fix wrong call parameter for free_bootmem() 2012-12-11 17:22:28 -08:00
crc7.c CRC7 support 2007-07-17 10:23:04 -07:00
crc8.c lib: crc8: add new library module providing crc8 algorithm 2011-06-03 15:01:06 -04:00
crc16.c
crc32.c sections: fix const sections for crc32 table 2012-10-06 03:04:46 +09:00
crc32defs.h crc32: select an algorithm via Kconfig 2012-03-23 16:58:38 -07:00
crc-ccitt.c
crc-itu-t.c
crc-t10dif.c [SCSI] lib: Add support for the T10 (SCSI) Data Integrity Field CRC 2008-07-12 08:22:32 -05:00
ctype.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
debug_locks.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
debugobjects.c hlist: drop the node parameter from iterators 2013-02-27 19:10:24 -08:00
dec_and_lock.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
decompress_bunzip2.c decompress_bunzip2: remove invalid vi modeline 2011-12-06 10:00:05 +01:00
decompress_inflate.c decompressors: check input size in decompress_inflate.c 2011-01-13 08:03:25 -08:00
decompress_unlzma.c treewide: Fix comment and string typo 'bufer' 2011-12-06 09:53:40 +01:00
decompress_unlzo.c lib/lzo: Rename lzo1x_decompress.c to lzo1x_decompress_safe.c 2013-02-20 19:36:00 +01:00
decompress_unxz.c Fix common misspellings 2011-03-31 11:26:23 -03:00
decompress.c lib/decompress.c: fix initconst 2013-04-30 17:04:09 -07:00
devres.c lib/devres.c: fix misplaced #endif 2013-02-27 19:10:09 -08:00
digsig.c Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security 2013-02-21 08:18:12 -08:00
div64.c Revert "math64: New div64_u64_rem helper" 2013-04-30 19:13:05 +02:00
dma-debug.c dma-debug: update DMA debug API to better handle multiple mappings of a buffer 2013-03-22 16:41:20 -07:00
dump_stack.c dump_stack: consolidate dump_stack() implementations and unify their behaviors 2013-04-30 17:04:02 -07:00
dynamic_debug.c dynamic_debug: reuse generic string_unescape function 2013-04-30 17:04:03 -07:00
dynamic_queue_limits.c bql: Avoid possible inconsistent calculation. 2012-05-31 18:18:17 -04:00
earlycpio.c lib: Add early cpio decoder 2012-09-30 18:02:20 -07:00
extable.c module: trim exception table on init free. 2009-06-12 21:47:04 +09:30
fault-inject.c lib/: rename random32() to prandom_u32() 2013-04-29 18:28:42 -07:00
fdt_ro.c of/lib: Allow scripts/dtc/libfdt to be used from kernel code 2012-07-23 13:54:52 +01:00
fdt_rw.c of/lib: Allow scripts/dtc/libfdt to be used from kernel code 2012-07-23 13:54:52 +01:00
fdt_strerror.c of/lib: Allow scripts/dtc/libfdt to be used from kernel code 2012-07-23 13:54:52 +01:00
fdt_sw.c of/lib: Allow scripts/dtc/libfdt to be used from kernel code 2012-07-23 13:54:52 +01:00
fdt_wip.c of/lib: Allow scripts/dtc/libfdt to be used from kernel code 2012-07-23 13:54:52 +01:00
fdt.c of/lib: Allow scripts/dtc/libfdt to be used from kernel code 2012-07-23 13:54:52 +01:00
find_last_bit.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
find_next_bit.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
flex_array.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
flex_proportions.c lib/flex_proportions.c: fix corruption of denominator in flexible proportions 2012-09-25 08:59:21 -07:00
gcd.c lib/gcd.c: prevent possible div by 0 2012-10-06 03:04:57 +09:00
gen_crc32table.c sections: fix const sections for crc32 table 2012-10-06 03:04:46 +09:00
genalloc.c genalloc: add devres support, allow to find a managed pool by device 2013-04-29 18:28:13 -07:00
halfmd4.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
hexdump.c dynamic_debug: dynamic hex dump 2013-01-17 12:19:09 -08:00
hweight.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
idr.c idr: introduce idr_alloc_cyclic() 2013-04-29 18:28:41 -07:00
inflate.c MN10300: Don't try and #include <linux/slab.h> in lib/inflate.c from bootloader 2010-08-12 09:51:35 -07:00
int_sqrt.c lib/int_sqrt.c: optimize square root algorithm 2013-04-29 18:28:19 -07:00
interval_tree_test_main.c random32: rename random32 to prandom 2012-12-17 17:15:26 -08:00
interval_tree.c mm: interval tree updates 2012-10-09 16:22:40 +09:00
iomap_copy.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
iomap.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
iommu-helper.c The following text was taken from the original review request: 2012-03-24 10:24:31 -07:00
ioremap.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
irq_regs.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
is_single_threaded.c kernel: is_current_single_threaded: don't use ->mmap_sem 2009-07-17 09:11:31 +10:00
jedec_ddr_data.c ddr: add LPDDR2 data from JESD209-2 2012-05-02 00:04:06 -07:00
kasprintf.c lib/kasprintf.c: use kmalloc_track_caller() to get accurate traces for kvasprintf 2012-10-11 08:50:15 +09:00
Kconfig Move utf16 functions to kernel core and rename 2013-04-15 21:23:03 +01:00
Kconfig.debug notifier-error-inject: fix module names in Kconfig 2013-04-30 17:04:09 -07:00
Kconfig.kgdb KGDB/KDB fixes and cleanups 2013-03-02 08:31:39 -08:00
Kconfig.kmemcheck kmemcheck: depend on HAVE_ARCH_KMEMCHECK 2009-07-01 22:28:44 +02:00
kfifo.c kfifo: fix kfifo_alloc() and kfifo_init() 2013-02-27 19:10:23 -08:00
klist.c Revert "driver core: check start node in klist_iter_init_node" 2012-04-19 19:17:30 -07:00
kobject_uevent.c netlink: hide struct module parameter in netlink_kernel_create 2012-09-08 18:46:30 -04:00
kobject.c kref: minor cleanup 2013-05-07 16:09:00 -07:00
kstrtox.c kstrto*: add documentation 2012-12-17 17:15:22 -08:00
kstrtox.h lib/kstrtox: common code between kstrto*() and simple_strto*() functions 2011-10-31 17:30:56 -07:00
lcm.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
libcrc32c.c libcrc32c: Fix "crc32c undefined" compilation error 2008-12-25 11:01:42 +11:00
list_debug.c rcu: Fix broken strings in RCU's source code. 2012-07-06 06:01:49 -07:00
list_sort.c lib/: rename random32() to prandom_u32() 2013-04-29 18:28:42 -07:00
llist.c Disintegrate and delete asm/system.h 2012-03-28 15:58:21 -07:00
locking-selftest-hardirq.h
locking-selftest-mutex.h
locking-selftest-rlock-hardirq.h
locking-selftest-rlock-softirq.h
locking-selftest-rlock.h
locking-selftest-rsem.h
locking-selftest-softirq.h
locking-selftest-spin-hardirq.h
locking-selftest-spin-softirq.h
locking-selftest-spin.h
locking-selftest-wlock-hardirq.h
locking-selftest-wlock-softirq.h
locking-selftest-wlock.h
locking-selftest-wsem.h
locking-selftest.c lockdep: Selftest: convert spinlock to raw spinlock 2013-02-19 08:43:35 +01:00
lru_cache.c hlist: drop the node parameter from iterators 2013-02-27 19:10:24 -08:00
Makefile Kconfig: consolidate CONFIG_DEBUG_STRICT_USER_COPY_CHECKS 2013-04-30 17:04:09 -07:00
md5.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
memory-notifier-error-inject.c memory: memory notifier error injection module 2012-07-30 17:25:22 -07:00
memweight.c string: introduce memweight() 2012-07-30 17:25:16 -07:00
nlattr.c netlink: add minlen validation for the new signed types 2012-08-30 13:11:46 -04:00
notifier-error-inject.c mode_t, whack-a-mole at 11... 2013-04-09 14:13:05 -04:00
notifier-error-inject.h fault-injection: notifier error injection 2012-07-30 17:25:22 -07:00
of-reconfig-notifier-error-inject.c powerpc+of: Rename and fix OF reconfig notifier error inject module 2012-12-14 10:32:52 +11:00
oid_registry.c Give the OID registry file module info to avoid kernel tainting 2013-05-05 14:38:00 -07:00
parser.c lib/parser.c: fix up comments for valid return values from match_number 2013-02-21 17:22:25 -08:00
pci_iomap.c lib: add NO_GENERIC_PCI_IOPORT_MAP 2012-01-31 23:19:47 +02:00
percpu_counter.c switch the protection of percpu_counter list to spinlock 2012-07-31 09:28:31 +04:00
percpu-rwsem.c percpu_rw_semaphore: add lockdep annotations 2012-12-17 17:15:18 -08:00
plist.c lib/plist.c: make plist test announcements KERN_DEBUG 2012-10-06 03:04:58 +09:00
pm-notifier-error-inject.c PM: PM notifier error injection module 2012-07-30 17:25:22 -07:00
prio_heap.c lib: fix sparse shadowed variable warning 2009-01-06 15:59:11 -08:00
proportions.c locking, lib/proportions: Annotate prop_local_percpu::lock as raw 2011-09-13 11:11:50 +02:00
radix-tree.c radix-tree: fix contiguous iterator 2012-06-05 10:46:40 -07:00
random32.c prandom: introduce prandom_bytes() and prandom_bytes_state() 2012-12-17 17:15:26 -08:00
ratelimit.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
rational.c lib: Change mail address of Oskar Schirmer 2012-05-17 15:18:37 +02:00
rbtree_test.c rbtree_test: add __init/__exit annotations 2013-04-30 17:04:07 -07:00
rbtree.c lib/rbtree.c: avoid the use of non-static __always_inline 2013-01-11 14:54:56 -08:00
reciprocal_div.c sch_red: Adaptative RED AQM 2011-12-08 19:52:43 -05:00
rwsem-spinlock.c rwsem: simplify __rwsem_do_wake 2013-05-07 07:20:16 -07:00
rwsem.c rwsem: check counter to avoid cmpxchg calls 2013-05-07 16:11:51 -07:00
scatterlist.c lib/scatterlist: sg_page_iter: support sg lists w/o backing pages 2013-03-27 17:13:44 +01:00
sha1.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
show_mem.c mm, show_mem: suppress page counts in non-blockable contexts 2013-04-29 15:54:28 -07:00
smp_processor_id.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
sort.c generic swap(): lib/sort.c: rename swap to swap_func 2009-01-08 08:31:14 -08:00
spinlock_debug.c lib/spinlock_debug: avoid livelock in do_raw_spin_lock() 2012-10-06 03:04:57 +09:00
stmp_device.c lib: add support for stmp-style devices 2012-04-20 23:27:08 +02:00
string_helpers.c lib/string_helpers: introduce generic string_unescape 2013-04-30 17:04:03 -07:00
string.c The following text was taken from the original review request: 2012-03-24 10:24:31 -07:00
strncpy_from_user.c word-at-a-time: make the interfaces truly generic 2012-05-26 11:33:40 -07:00
strnlen_user.c lib: Fix generic strnlen_user for 32-bit big-endian machines 2012-05-27 20:59:46 -07:00
swiotlb.c x86, kdump: Set crashkernel_low automatically 2013-04-17 12:35:32 -07:00
syscall.c lib: reduce the use of module.h wherever possible 2012-03-07 15:04:04 -05:00
test-kstrtox.c lib/test-kstrtox.c: mark const init data with __initconst instead of __initdata 2012-05-29 16:22:32 -07:00
test-string_helpers.c lib/string_helpers: introduce generic string_unescape 2013-04-30 17:04:03 -07:00
textsearch.c textsearch: doc - fix spelling in lib/textsearch.c. 2011-01-24 23:33:30 -08:00
timerqueue.c The following text was taken from the original review request: 2012-03-24 10:24:31 -07:00
ts_bm.c textsearch: ts_bm: support case insensitive searching in Boyer-Moore algorithm 2008-07-08 02:37:54 -07:00
ts_fsm.c textsearch: ts_fsm: return error on request for case insensitive search 2008-07-08 02:38:27 -07:00
ts_kmp.c textsearch: ts_kmp: support case insensitive searching in Knuth-Morris-Pratt algorithm 2008-07-08 02:38:09 -07:00
ucs2_string.c Move utf16 functions to kernel core and rename 2013-04-15 21:23:03 +01:00
usercopy.c Kconfig: consolidate CONFIG_DEBUG_STRICT_USER_COPY_CHECKS 2013-04-30 17:04:09 -07:00
uuid.c uuid: use prandom_bytes() 2013-04-29 18:28:42 -07:00
vsprintf.c vsprintf: Add extension %pSR - print_symbol replacement 2013-04-30 22:31:16 +02:00