linux/net
Johannes Weiner 3e32cb2e0a mm: memcontrol: lockless page counters
Memory is internally accounted in bytes, using spinlock-protected 64-bit
counters, even though the smallest accounting delta is a page.  The
counter interface is also convoluted and does too many things.

Introduce a new lockless word-sized page counter API, then change all
memory accounting over to it.  The translation from and to bytes then only
happens when interfacing with userspace.

The removed locking overhead is noticable when scaling beyond the per-cpu
charge caches - on a 4-socket machine with 144-threads, the following test
shows the performance differences of 288 memcgs concurrently running a
page fault benchmark:

vanilla:

   18631648.500498      task-clock (msec)         #  140.643 CPUs utilized            ( +-  0.33% )
         1,380,638      context-switches          #    0.074 K/sec                    ( +-  0.75% )
            24,390      cpu-migrations            #    0.001 K/sec                    ( +-  8.44% )
     1,843,305,768      page-faults               #    0.099 M/sec                    ( +-  0.00% )
50,134,994,088,218      cycles                    #    2.691 GHz                      ( +-  0.33% )
   <not supported>      stalled-cycles-frontend
   <not supported>      stalled-cycles-backend
 8,049,712,224,651      instructions              #    0.16  insns per cycle          ( +-  0.04% )
 1,586,970,584,979      branches                  #   85.176 M/sec                    ( +-  0.05% )
     1,724,989,949      branch-misses             #    0.11% of all branches          ( +-  0.48% )

     132.474343877 seconds time elapsed                                          ( +-  0.21% )

lockless:

   12195979.037525      task-clock (msec)         #  133.480 CPUs utilized            ( +-  0.18% )
           832,850      context-switches          #    0.068 K/sec                    ( +-  0.54% )
            15,624      cpu-migrations            #    0.001 K/sec                    ( +- 10.17% )
     1,843,304,774      page-faults               #    0.151 M/sec                    ( +-  0.00% )
32,811,216,801,141      cycles                    #    2.690 GHz                      ( +-  0.18% )
   <not supported>      stalled-cycles-frontend
   <not supported>      stalled-cycles-backend
 9,999,265,091,727      instructions              #    0.30  insns per cycle          ( +-  0.10% )
 2,076,759,325,203      branches                  #  170.282 M/sec                    ( +-  0.12% )
     1,656,917,214      branch-misses             #    0.08% of all branches          ( +-  0.55% )

      91.369330729 seconds time elapsed                                          ( +-  0.45% )

On top of improved scalability, this also gets rid of the icky long long
types in the very heart of memcg, which is great for 32 bit and also makes
the code a lot more readable.

Notable differences between the old and new API:

- res_counter_charge() and res_counter_charge_nofail() become
  page_counter_try_charge() and page_counter_charge() resp. to match
  the more common kernel naming scheme of try_do()/do()

- res_counter_uncharge_until() is only ever used to cancel a local
  counter and never to uncharge bigger segments of a hierarchy, so
  it's replaced by the simpler page_counter_cancel()

- res_counter_set_limit() is replaced by page_counter_limit(), which
  expects its callers to serialize against themselves

- res_counter_memparse_write_strategy() is replaced by
  page_counter_limit(), which rounds down to the nearest page size -
  rather than up.  This is more reasonable for explicitely requested
  hard upper limits.

- to keep charging light-weight, page_counter_try_charge() charges
  speculatively, only to roll back if the result exceeds the limit.
  Because of this, a failing bigger charge can temporarily lock out
  smaller charges that would otherwise succeed.  The error is bounded
  to the difference between the smallest and the biggest possible
  charge size, so for memcg, this means that a failing THP charge can
  send base page charges into reclaim upto 2MB (4MB) before the limit
  would have been reached.  This should be acceptable.

[akpm@linux-foundation.org: add includes for WARN_ON_ONCE and memparse]
[akpm@linux-foundation.org: add includes for WARN_ON_ONCE, memparse, strncmp, and PAGE_SIZE]
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Acked-by: Michal Hocko <mhocko@suse.cz>
Acked-by: Vladimir Davydov <vdavydov@parallels.com>
Cc: Tejun Heo <tj@kernel.org>
Cc: David Rientjes <rientjes@google.com>
Cc: Stephen Rothwell <sfr@canb.auug.org.au>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
2014-12-10 17:41:04 -08:00
..
6lowpan 6lowpan: Allow 6LoWPAN to be modular 2014-08-07 11:44:18 -07:00
9p 9p/trans_virtio: enable VQs early 2014-10-15 10:25:04 +10:30
802 net: set name_assign_type in alloc_netdev() 2014-07-15 16:12:48 -07:00
8021q net: better IFF_XMIT_DST_RELEASE support 2014-10-07 13:22:11 -04:00
appletalk Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2014-07-16 14:09:34 -07:00
atm net: better IFF_XMIT_DST_RELEASE support 2014-10-07 13:22:11 -04:00
ax25 net: Fix use after free by removing length arg from sk_data_ready callbacks. 2014-04-11 16:15:36 -04:00
batman-adv batman-adv: replace strnicmp with strncasecmp 2014-10-14 02:18:24 +02:00
bluetooth rfcomm, sched/wait: Fix broken wait construct 2014-11-04 07:17:47 +01:00
bridge bridge: Add missing policy entry for IFLA_BRPORT_FAST_LEAVE 2014-11-26 15:29:01 -05:00
caif caif_usb: use target structure member in memset 2014-10-14 16:05:45 -04:00
can can: add hash based access to single EFF frame filters 2014-05-19 09:38:24 +02:00
ceph libceph: change from BUG to WARN for __remove_osd() asserts 2014-11-13 22:26:34 +03:00
core Merge branch 'sched-core-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip 2014-12-09 21:21:34 -08:00
dcb dcbnl : Disable software interrupts before taking dcb_lock 2014-11-16 14:50:52 -05:00
dccp Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2014-10-18 09:31:37 -07:00
decnet af_decnet: Use time_after_eq 2014-08-22 12:23:11 -07:00
dns_resolver Merge commit 'v3.16' into next 2014-10-01 00:44:04 +10:00
dsa net: dsa: slave: Fix autoneg for phys on switch MDIO bus 2014-11-06 15:06:28 -05:00
ethernet net: Add function for parsing the header length out of linear ethernet frames 2014-09-05 17:47:02 -07:00
hsr net/hsr: Remove left-over never-true conditional code. 2014-07-11 15:04:40 -07:00
ieee802154 Merge tag 'master-2014-10-02' of git://git.kernel.org/pub/scm/linux/kernel/git/linville/wireless-next 2014-10-05 21:34:39 -04:00
ipv4 mm: memcontrol: lockless page counters 2014-12-10 17:41:04 -08:00
ipv6 tcp: fix possible NULL dereference in tcp_vX_send_reset() 2014-11-25 14:29:18 -05:00
ipx ipx: fix locking regression in ipx_sendmsg and ipx_recvmsg 2014-11-20 22:57:03 -05:00
irda irda: stop calling sk_prot->disconnect() on connection failure 2014-11-02 10:20:26 -08:00
iucv iucv: Convert pr_warning to pr_warn 2014-09-10 12:40:10 -07:00
key af_key: remove unnecessary break after return 2014-07-15 16:27:00 -07:00
l2tp l2tp: Refactor l2tp core driver to make use of the common UDP tunnel functions 2014-09-19 15:57:15 -04:00
lapb
llc net_dma: simple removal 2014-09-28 07:05:16 -07:00
mac80211 mac80211: minstrel_ht: fix a crash in rate sorting 2014-11-18 22:39:16 +01:00
mac802154 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next 2014-10-08 21:40:54 -04:00
mpls mpls: Allow mpls_gso to be built as module 2014-10-31 15:47:21 -04:00
netfilter Revert "netfilter: conntrack: fix race in __nf_conntrack_confirm against get_next_corpse" 2014-11-25 14:14:51 -05:00
netlabel netlabel: kernel-doc warning fix 2014-10-09 01:40:05 -04:00
netlink netlink: Properly unbind in error conditions. 2014-11-12 15:12:06 -05:00
netrom netrom: use linux/uaccess.h 2014-10-17 23:52:54 -04:00
nfc NFC: nci: Add support for proprietary RF Protocols 2014-09-24 02:02:24 +02:00
openvswitch openvswitch: Don't validate IPv6 label masks. 2014-11-20 22:56:13 -05:00
packet af_packet: fix sparse warning 2014-11-24 16:15:36 -05:00
phonet net: fix rcu access on phonet_routes 2014-10-06 18:16:30 -04:00
rds Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net 2014-10-18 09:31:37 -07:00
rfkill net: rfkill: kernel-doc warning fixes 2014-10-09 11:16:15 +02:00
rose rose: use %*ph specifier 2014-09-07 16:07:25 -07:00
rxrpc Merge branch 'next' of git://git.kernel.org/pub/scm/linux/kernel/git/jmorris/linux-security 2014-10-12 10:13:55 -04:00
sched sch_pie: schedule the timer after all init succeed 2014-10-29 14:28:01 -04:00
sctp net: sctp: fix memory leak in auth key management 2014-11-11 15:19:11 -05:00
sunrpc SUNRPC: Fix locking around callback channel reply receive 2014-11-19 12:03:20 -05:00
tipc tipc: fix lockdep warning when intra-node messages are delivered 2014-10-21 15:28:15 -04:00
unix af_unix: remove 0 assignment on static 2014-10-07 17:03:14 -04:00
vmw_vsock vsock: Make transport the proto owner 2014-05-05 13:13:50 -04:00
wimax wimax: convert printk to pr_foo() 2014-10-07 20:28:44 -04:00
wireless Here are a few fixes for the wireless stack: one fixes the 2014-10-27 13:38:15 -04:00
x25 net: Fix use after free by removing length arg from sk_data_ready callbacks. 2014-04-11 16:15:36 -04:00
xfrm net: skb_fclone_busy() needs to detect orphaned skb 2014-10-30 19:58:30 -04:00
Kconfig bpf: split eBPF out of NET 2014-10-27 19:09:59 -04:00
Makefile 6lowpan: introduce new net/6lowpan directory 2014-07-12 01:53:30 +02:00
compat.c net: sendmsg: fix NULL pointer dereference 2014-07-29 12:20:22 -07:00
nonet.c
socket.c File locking related changes for v3.18 (pile #1) 2014-10-11 13:21:34 -04:00
sysctl_net.c