OpenE2K
/
linux
6
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity
linux/block
History
Tejun Heo 36c38fb714 blkcg: use trylock on blkcg_pol_mutex in blkcg_reset_stats() 
During the recent conversion of cgroup to kernfs, cgroup_tree_mutex
which nests above both the kernfs s_active protection and cgroup_mutex
is added to synchronize cgroup file type operations as cgroup_mutex
needed to be grabbed from some file operations and thus can't be put
above s_active protection.

While this arrangement mostly worked for cgroup, this triggered the
following lockdep warning.

  ======================================================
  [ INFO: possible circular locking dependency detected ]
  3.15.0-rc3-next-20140430-sasha-00016-g4e281fa-dirty #429 Tainted: G        W
  -------------------------------------------------------
  trinity-c173/9024 is trying to acquire lock:
  (blkcg_pol_mutex){+.+.+.}, at: blkcg_reset_stats (include/linux/spinlock.h:328 block/blk-cgroup.c:455)

  but task is already holding lock:
  (s_active#89){++++.+}, at: kernfs_fop_write (fs/kernfs/file.c:283)

  which lock already depends on the new lock.

  the existing dependency chain (in reverse order) is:

  -> #2 (s_active#89){++++.+}:
  lock_acquire (arch/x86/include/asm/current.h:14 kernel/locking/lockdep.c:3602)
  __kernfs_remove (arch/x86/include/asm/atomic.h:27 fs/kernfs/dir.c:352 fs/kernfs/dir.c:1024)
  kernfs_remove_by_name_ns (fs/kernfs/dir.c:1219)
  cgroup_addrm_files (include/linux/kernfs.h:427 kernel/cgroup.c:1074 kernel/cgroup.c:2899)
  cgroup_clear_dir (kernel/cgroup.c:1092 (discriminator 2))
  rebind_subsystems (kernel/cgroup.c:1144)
  cgroup_setup_root (kernel/cgroup.c:1568)
  cgroup_mount (kernel/cgroup.c:1716)
  mount_fs (fs/super.c:1094)
  vfs_kern_mount (fs/namespace.c:899)
  do_mount (fs/namespace.c:2238 fs/namespace.c:2561)
  SyS_mount (fs/namespace.c:2758 fs/namespace.c:2729)
  tracesys (arch/x86/kernel/entry_64.S:746)

  -> #1 (cgroup_tree_mutex){+.+.+.}:
  lock_acquire (arch/x86/include/asm/current.h:14 kernel/locking/lockdep.c:3602)
  mutex_lock_nested (kernel/locking/mutex.c:486 kernel/locking/mutex.c:587)
  cgroup_add_cftypes (include/linux/list.h:76 kernel/cgroup.c:3040)
  blkcg_policy_register (block/blk-cgroup.c:1106)
  throtl_init (block/blk-throttle.c:1694)
  do_one_initcall (init/main.c:789)
  kernel_init_freeable (init/main.c:854 init/main.c:863 init/main.c:882 init/main.c:1003)
  kernel_init (init/main.c:935)
  ret_from_fork (arch/x86/kernel/entry_64.S:552)

  -> #0 (blkcg_pol_mutex){+.+.+.}:
  __lock_acquire (kernel/locking/lockdep.c:1840 kernel/locking/lockdep.c:1945 kernel/locking/lockdep.c:2131 kernel/locking/lockdep.c:3182)
  lock_acquire (arch/x86/include/asm/current.h:14 kernel/locking/lockdep.c:3602)
  mutex_lock_nested (kernel/locking/mutex.c:486 kernel/locking/mutex.c:587)
  blkcg_reset_stats (include/linux/spinlock.h:328 block/blk-cgroup.c:455)
  cgroup_file_write (kernel/cgroup.c:2714)
  kernfs_fop_write (fs/kernfs/file.c:295)
  vfs_write (fs/read_write.c:532)
  SyS_write (fs/read_write.c:584 fs/read_write.c:576)
  tracesys (arch/x86/kernel/entry_64.S:746)

  other info that might help us debug this:

  Chain exists of:
  blkcg_pol_mutex --> cgroup_tree_mutex --> s_active#89

   Possible unsafe locking scenario:

	 CPU0                    CPU1
	 ----                    ----
    lock(s_active#89);
				 lock(cgroup_tree_mutex);
				 lock(s_active#89);
    lock(blkcg_pol_mutex);

   *** DEADLOCK ***

  4 locks held by trinity-c173/9024:
  #0: (&f->f_pos_lock){+.+.+.}, at: __fdget_pos (fs/file.c:714)
  #1: (sb_writers#18){.+.+.+}, at: vfs_write (include/linux/fs.h:2255 fs/read_write.c:530)
  #2: (&of->mutex){+.+.+.}, at: kernfs_fop_write (fs/kernfs/file.c:283)
  #3: (s_active#89){++++.+}, at: kernfs_fop_write (fs/kernfs/file.c:283)

  stack backtrace:
  CPU: 3 PID: 9024 Comm: trinity-c173 Tainted: G        W     3.15.0-rc3-next-20140430-sasha-00016-g4e281fa-dirty #429
   ffffffff919687b0 ffff8805f6373bb8 ffffffff8e52cdbb 0000000000000002
   ffffffff919d8400 ffff8805f6373c08 ffffffff8e51fb88 0000000000000004
   ffff8805f6373c98 ffff8805f6373c08 ffff88061be70d98 ffff88061be70dd0
  Call Trace:
  dump_stack (lib/dump_stack.c:52)
  print_circular_bug (kernel/locking/lockdep.c:1216)
  __lock_acquire (kernel/locking/lockdep.c:1840 kernel/locking/lockdep.c:1945 kernel/locking/lockdep.c:2131 kernel/locking/lockdep.c:3182)
  lock_acquire (arch/x86/include/asm/current.h:14 kernel/locking/lockdep.c:3602)
  mutex_lock_nested (kernel/locking/mutex.c:486 kernel/locking/mutex.c:587)
  blkcg_reset_stats (include/linux/spinlock.h:328 block/blk-cgroup.c:455)
  cgroup_file_write (kernel/cgroup.c:2714)
  kernfs_fop_write (fs/kernfs/file.c:295)
  vfs_write (fs/read_write.c:532)
  SyS_write (fs/read_write.c:584 fs/read_write.c:576)

This is a highly unlikely but valid circular dependency between "echo
1 > blkcg.reset_stats" and cfq module [un]loading.  cgroup is going
through further locking update which will remove this complication but
for now let's use trylock on blkcg_pol_mutex and retry the file
operation if the trylock fails.

Signed-off-by: Tejun Heo <tj@kernel.org>
Reported-by: Sasha Levin <sasha.levin@oracle.com>
References: http://lkml.kernel.org/g/5363C04B.4010400@oracle.com
 		2014-05-05 13:48:18 -04:00
..
partitions block: Use macros from compiler.h instead of __attribute__((...)) 2014-02-18 12:20:01 -08:00
Kconfig block: change config option name for cmdline partition parsing 2013-09-30 14:31:02 -07:00
Kconfig.iosched
…
Makefile blk-mq: new multi-queue block IO queueing mechanism 2013-10-25 11:56:00 +01:00
blk-cgroup.c blkcg: use trylock on blkcg_pol_mutex in blkcg_reset_stats() 2014-05-05 13:48:18 -04:00
blk-cgroup.h Merge branch 'for-3.15' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup 2014-04-03 13:05:42 -07:00
blk-core.c block: fix regression with block enabled tagging 2014-04-09 21:54:06 -06:00
blk-exec.c blk-mq: merge blk_mq_insert_request and blk_mq_run_request 2014-03-21 08:57:37 -06:00
blk-flush.c blk-mq: merge blk_mq_insert_request and blk_mq_run_request 2014-03-21 08:57:37 -06:00
blk-integrity.c bio-integrity: Convert to bvec_iter 2013-11-23 22:33:50 -08:00
blk-ioc.c block: Substitute rcu_access_pointer() for rcu_dereference_raw() 2014-02-18 12:21:26 -08:00
blk-iopoll.c block: remove old blk_iopoll_enabled variable 2014-03-13 09:38:42 -06:00
blk-lib.c block: add cond_resched() to potentially long running ioctl discard loop 2014-02-12 09:36:37 -07:00
blk-map.c Merge branch 'for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/viro/vfs 2014-04-12 14:49:50 -07:00
blk-merge.c block: Explicitly handle discard/write same segments 2014-02-07 13:54:08 -07:00
blk-mq-cpu.c rt,blk,mq: Make blk_mq_cpu_notify_lock a raw spinlock 2014-03-21 08:57:56 -06:00
blk-mq-cpumap.c blk-mq: don't dump CPU -> hw queue map on driver load 2014-03-20 13:31:44 -06:00
blk-mq-sysfs.c blk-mq: don't dump CPU -> hw queue map on driver load 2014-03-20 13:31:44 -06:00
blk-mq-tag.c Merge branch 'for-linus' of git://git.kernel.dk/linux-block 2014-02-14 10:45:18 -08:00
blk-mq-tag.h blk-mq: new multi-queue block IO queueing mechanism 2013-10-25 11:56:00 +01:00
blk-mq.c blk-mq: fix potential stall during CPU unplug with IO pending 2014-04-07 08:17:18 -06:00
blk-mq.h blk-mq: merge blk_mq_insert_request and blk_mq_run_request 2014-03-21 08:57:37 -06:00
blk-settings.c bcache/md: Use raid stripe size 2014-01-08 13:05:09 -08:00
blk-softirq.c block: fix regression with block enabled tagging 2014-04-09 21:54:06 -06:00
blk-sysfs.c blk-mq: rework flush sequencing logic 2014-02-10 09:29:00 -07:00
blk-tag.c block: Reserve only one queue tag for sync IO if only 3 tags are available 2013-06-28 21:32:27 +02:00
blk-throttle.c cgroup: drop const from @buffer of cftype->write_string() 2014-03-19 10:23:54 -04:00
blk-timeout.c blk-mq: rework I/O completions 2014-02-10 09:27:31 -07:00
blk.h block: fix regression with block enabled tagging 2014-04-09 21:54:06 -06:00
bsg-lib.c
…
bsg.c hlist: drop the node parameter from iterators 2013-02-27 19:10:24 -08:00
cfq-iosched.c Merge branch 'for-3.15' of git://git.kernel.org/pub/scm/linux/kernel/git/tj/cgroup 2014-04-03 13:05:42 -07:00
cmdline-parser.c block: remove unrelated header files and export symbol 2014-01-21 20:18:26 -08:00
compat_ioctl.c kernel-wide: fix missing validations on __get/__put/__copy_to/__copy_from_user() 2013-09-11 15:58:18 -07:00
deadline-iosched.c block: Stop abusing csd.list for fifo_time 2014-02-24 14:46:32 -08:00
elevator.c block: fix regression with block enabled tagging 2014-04-09 21:54:06 -06:00
genhd.c block: Convert kmalloc_node(...GFP_ZERO...) to kzalloc_node(...) 2013-09-11 13:22:03 -06:00
ioctl.c block: replace IS_ERR and PTR_ERR with PTR_ERR_OR_ZERO 2013-11-08 09:05:31 -07:00
noop-iosched.c elevator: Fix a race in elevator switching 2013-07-03 13:25:24 +02:00
partition-generic.c Revert "loop: cleanup partitions when detaching loop device" 2013-04-08 10:12:11 +02:00
scsi_ioctl.c block: Fix memory leak in rw_copy_check_uvector() handling 2014-01-21 20:36:17 -08:00