Regression in 2.6.28-rc1: When I added the new state_mutex which
prevents corruption of raw1394's internal state when accessed by
multithreaded client applications, the following possible though
highly unlikely deadlock slipped in:
Thread A: Thread B:
- acquire mmap_sem - raw1394_write() or raw1394_ioctl()
- raw1394_mmap() - acquire state_mutex
- acquire state_mutex - copy_to/from_user(), possible page fault:
acquire mmap_sem
The simplest fix is to use mutex_trylock() instead of mutex_lock() in
raw1394_mmap(). This changes the behavior under contention in a way
which is visible to userspace clients. However, since multithreaded
access was entirely buggy before state_mutex was added and libraw1394's
documentation advised application programmers to use a handle only in a
single thread, this change in behaviour should not be an issue in
practice at all.
Since we have to use mutex_trylock() in raw1394_mmap() regardless
whether /dev/raw1394 was opened with O_NONBLOCK or not, we now use
mutex_trylock() unconditionally everywhere for state_mutex, just to have
consistent behavior.
Reported-by: Johannes Weiner <hannes@saeurebad.de>
Signed-off-by: Stefan Richter <stefanr@s5r6.in-berlin.de>
638570b543