binutils-gdb/gdb/gdbserver
Pedro Alves 2db9a4275c GNU/Linux: Stop using libthread_db/td_ta_thr_iter
TL;DR - GDB can hang if something refreshes the thread list out of the
target while the target is running.  GDB hangs inside td_ta_thr_iter.
The fix is to not use that libthread_db function anymore.

Long version:

Running the testsuite against my all-stop-on-top-of-non-stop series is
still exposing latent non-stop bugs.

I was originally seeing this with the multi-create.exp test, back when
we were still using libthread_db thread event breakpoints.  The
all-stop-on-top-of-non-stop series forces a thread list refresh each
time GDB needs to start stepping over a breakpoint (to pause all
threads).  That test hits the thread event breakpoint often, resulting
in a bunch of step-over operations, thus a bunch of thread list
refreshes while some threads in the target are running.

The commit adds a real non-stop mode test that triggers the issue,
based on multi-create.exp, that does an explicit "info threads" when a
breakpoint is hit.  IOW, it does the same things the as-ns series was
doing when testing multi-create.exp.

The bug is a race, so it unfortunately takes several runs for the test
to trigger it.  In fact, even when setting the test running in a loop,
it sometimes takes several minutes for it to trigger for me.

The race is related to libthread_db's td_ta_thr_iter.  This is
libthread_db's entry point for walking the thread list of the
inferior.

Sometimes, when GDB refreshes the thread list from the target,
libthread_db's td_ta_thr_iter can somehow see glibc's thread list as a
cycle, and get stuck in an infinite loop.

The issue is that when a thread exits, its thread control structure in
glibc is moved from a "used" list to a "cache" list.  These lists are
simply circular linked lists where the "next/prev" pointers are
embedded in the thread control structure itself.  The "next" pointer
of the last element of the list points back to the list's sentinel
"head".  There's only one set of "next/prev" pointers for both lists;
thus a thread can only be in one of the lists at a time, not in both
simultaneously.

So when thread C exits, simplifying, the following happens.  A-C are
threads.  stack_used and stack_cache are the list's heads.

Before:

  stack_used -> A -> B -> C -> (&stack_used)
  stack_cache -> (&stack_cache)

After:

  stack_used -> A -> B -> (&stack_used)
  stack_cache -> C -> (&stack_cache)

td_ta_thr_iter starts by iterating at the list's head's next, and
iterates until it sees a thread whose next pointer points to the
list's head again.  Thus in the before case above, C's next points to
stack_used, indicating end of list.  In the same case, the stack_cache
list is empty.

For each thread being iterated, td_ta_thr_iter reads the whole thread
object out of the inferior.  This includes the thread's "next"
pointer.

In the scenario above, it may happen that td_ta_thr_iter is iterating
thread B and has already read B's thread structure just before thread
C exits and its control structure moves to the cached list.

Now, recall that td_ta_thr_iter is running in the context of GDB, and
there's no locking between GDB and the inferior.  From it's local copy
of B, td_ta_thr_iter believes that the next thread after B is thread
C, so it happilly continues iterating to C, a thread that has already
exited, and is now in the stack cache list.

After iterating C, td_ta_thr_iter finds the stack_cache head, which
because it is not stack_used, td_ta_thr_iter assumes it's just another
thread.  After this, unless the reverse race triggers, GDB gets stuck
in td_ta_thr_iter forever walking the stack_cache list, as no thread
in thatlist has a next pointer that points back to stack_used (the
terminating condition).

Before fully understanding the issue, I tried adding cycle detection
to GDB's td_ta_thr_iter callback.  However, td_ta_thr_iter skips
calling the callback in some cases, which means that it's possible
that the callback isn't called at all, making it impossible for GDB to
break the loop.  I did manage to get GDB stuck in that state more than
once.

Fortunately, we can avoid the issue altogether.  We don't really need
td_ta_thr_iter for live debugging nowadays, given PTRACE_EVENT_CLONE.
We already know how to map and lwp id to a thread id without iterating
(thread_from_lwp), so use that more.

gdb/ChangeLog:
2015-02-20  Pedro Alves  <palves@redhat.com>

	* linux-nat.c (linux_handle_extended_wait): Call
	thread_db_notice_clone whenever a new clone LWP is detected.
	(linux_stop_and_wait_all_lwps, linux_unstop_all_lwps): New
	functions.
	* linux-nat.h (thread_db_attach_lwp): Delete declaration.
	(thread_db_notice_clone, linux_stop_and_wait_all_lwps)
	(linux_unstop_all_lwps): Declare.
	* linux-thread-db.c (struct thread_get_info_inout): Delete.
	(thread_get_info_callback): Delete.
	(thread_from_lwp): Use td_thr_get_info and record_thread.
	(thread_db_attach_lwp): Delete.
	(thread_db_notice_clone): New function.
	(try_thread_db_load_1): If /proc is mounted and shows the
	process'es task list, walk over all LWPs and call thread_from_lwp
	instead of relying on td_ta_thr_iter.
	(attach_thread): Don't call check_thread_signals here.  Split the
	tail part of the function (which adds the thread to the core GDB
	thread list) to ...
	(record_thread): ... this function.  Call check_thread_signals
	here.
	(thread_db_wait): Don't call thread_db_find_new_threads_1.  Always
	call thread_from_lwp.
	(thread_db_update_thread_list): Rename to ...
	(thread_db_update_thread_list_org): ... this.
	(thread_db_update_thread_list): New function.
	(thread_db_find_thread_from_tid): Delete.
	(thread_db_get_ada_task_ptid): Simplify.
	* nat/linux-procfs.c: Include <sys/stat.h>.
	(linux_proc_task_list_dir_exists): New function.
	* nat/linux-procfs.h (linux_proc_task_list_dir_exists): Declare.

gdb/gdbserver/ChangeLog:
2015-02-20  Pedro Alves  <palves@redhat.com>

	* thread-db.c: Include "nat/linux-procfs.h".
	(thread_db_init): Skip listing new threads if the kernel supports
	PTRACE_EVENT_CLONE and /proc/PID/task/ is accessible.

gdb/testsuite/ChangeLog:
2015-02-20  Pedro Alves  <palves@redhat.com>

	* gdb.threads/multi-create-ns-info-thr.exp: New file.
2015-02-20 21:40:31 +00:00
..
.gitignore
acinclude.m4
aclocal.m4
ax.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
ax.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
ChangeLog GNU/Linux: Stop using libthread_db/td_ta_thr_iter 2015-02-20 21:40:31 +00:00
config.in Detect 64-bit-ness in PowerPC Book III-E 2015-01-14 22:28:27 +08:00
configure Move safe_strerror to common/ 2015-01-15 15:09:15 -05:00
configure.ac Move safe_strerror to common/ 2015-01-15 15:09:15 -05:00
configure.srv Move code to disable ASR to nat/ 2015-01-15 15:10:49 -05:00
debug.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
debug.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
dll.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
dll.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
event-loop.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
event-loop.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
gdb_proc_service.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
gdbreplay.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
gdbthread.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
hostio-errno.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
hostio.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
hostio.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
i387-fp.c Fix use of wrong struct i387_xsave field in i387_cache_to_xsave 2015-01-10 09:57:23 +04:00
i387-fp.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
inferiors.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
inferiors.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-aarch64-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-amd64-ipa.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-arm-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-bfin-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-cris-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-crisv32-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-i386-ipa.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-ia64-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-low.c fix gdbserver/linux-low'c's pending status handling 2015-02-20 19:52:51 +00:00
linux-low.h [gdbserver] linux-low.c: better starvation avoidance, handle non-stop mode too 2015-01-09 14:42:32 +00:00
linux-m32r-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-m68k-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-mips-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-nios2-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-ppc-low.c Detect 64-bit-ness in PowerPC Book III-E 2015-01-14 22:28:27 +08:00
linux-s390-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-sh-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-sparc-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-tic6x-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-tile-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
linux-x86-low.c [gdbserver] linux-low.c: better starvation avoidance, handle non-stop mode too 2015-01-09 14:42:32 +00:00
linux-xtensa-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
lynx-i386-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
lynx-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
lynx-low.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
lynx-ppc-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
Makefile.in btrace: add struct btrace_data 2015-02-09 09:21:44 +01:00
mem-break.c [gdbserver] linux-low.c: better starvation avoidance, handle non-stop mode too 2015-01-09 14:42:32 +00:00
mem-break.h [gdbserver] linux-low.c: better starvation avoidance, handle non-stop mode too 2015-01-09 14:42:32 +00:00
notif.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
notif.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
nto-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
nto-low.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
nto-x86-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
proc-service.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
proc-service.list Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
README
regcache.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
regcache.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
remote-utils.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
remote-utils.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
server.c record-btrace: add bts buffer size configuration option 2015-02-09 09:42:28 +01:00
server.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
spu-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
symbol.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
target.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
target.h record btrace: add configuration struct 2015-02-09 09:38:55 +01:00
tdesc.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
tdesc.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
terminal.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
thread-db.c GNU/Linux: Stop using libthread_db/td_ta_thr_iter 2015-02-20 21:40:31 +00:00
tracepoint.c gdbserver: Fix crash when QTinit is handled with no inferior process attached 2015-02-10 13:49:41 -05:00
tracepoint.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
utils.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
utils.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
win32-arm-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
win32-i386-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
win32-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
win32-low.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
wincecompat.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
wincecompat.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
x86-low.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
x86-low.h Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00
xtensa-xtregs.c Update year range in copyright notice of all files owned by the GDB project. 2015-01-01 13:32:14 +04:00

		   README for GDBserver & GDBreplay
		    by Stu Grossman and Fred Fish

Introduction:

This is GDBserver, a remote server for Un*x-like systems.  It can be used to
control the execution of a program on a target system from a GDB on a different
host.  GDB and GDBserver communicate using the standard remote serial protocol
implemented in remote.c, and various *-stub.c files.  They communicate via
either a serial line or a TCP connection.

For more information about GDBserver, see the GDB manual.

Usage (server (target) side):

First, you need to have a copy of the program you want to debug put onto
the target system.  The program can be stripped to save space if needed, as
GDBserver doesn't care about symbols.  All symbol handling is taken care of by
the GDB running on the host system.

To use the server, you log on to the target system, and run the `gdbserver'
program.  You must tell it (a) how to communicate with GDB, (b) the name of
your program, and (c) its arguments.  The general syntax is:

	target> gdbserver COMM PROGRAM [ARGS ...]

For example, using a serial port, you might say:

	target> gdbserver /dev/com1 emacs foo.txt

This tells GDBserver to debug emacs with an argument of foo.txt, and to
communicate with GDB via /dev/com1.  GDBserver now waits patiently for the
host GDB to communicate with it.

To use a TCP connection, you could say:

	target> gdbserver host:2345 emacs foo.txt

This says pretty much the same thing as the last example, except that we are
going to communicate with the host GDB via TCP.  The `host:2345' argument means
that we are expecting to see a TCP connection from `host' to local TCP port
2345.  (Currently, the `host' part is ignored.)  You can choose any number you
want for the port number as long as it does not conflict with any existing TCP
ports on the target system.  This same port number must be used in the host
GDBs `target remote' command, which will be described shortly.  Note that if
you chose a port number that conflicts with another service, GDBserver will
print an error message and exit.

On some targets, GDBserver can also attach to running programs.  This is
accomplished via the --attach argument.  The syntax is:

	target> gdbserver --attach COMM PID

PID is the process ID of a currently running process.  It isn't necessary
to point GDBserver at a binary for the running process.

Usage (host side):

You need an unstripped copy of the target program on your host system, since
GDB needs to examine it's symbol tables and such.  Start up GDB as you normally
would, with the target program as the first argument.  (You may need to use the
--baud option if the serial line is running at anything except 9600 baud.)
Ie: `gdb TARGET-PROG', or `gdb --baud BAUD TARGET-PROG'.  After that, the only
new command you need to know about is `target remote'.  It's argument is either
a device name (usually a serial device, like `/dev/ttyb'), or a HOST:PORT
descriptor.  For example:

	(gdb) target remote /dev/ttyb

communicates with the server via serial line /dev/ttyb, and:

	(gdb) target remote the-target:2345

communicates via a TCP connection to port 2345 on host `the-target', where
you previously started up GDBserver with the same port number.  Note that for
TCP connections, you must start up GDBserver prior to using the `target remote'
command, otherwise you may get an error that looks something like
`Connection refused'.

Building GDBserver:

The supported targets as of November 2006 are:
	arm-*-linux*
	bfin-*-uclinux
	bfin-*-linux-uclibc
	crisv32-*-linux*
	cris-*-linux*
	i[34567]86-*-cygwin*
	i[34567]86-*-linux*
	i[34567]86-*-mingw*
	ia64-*-linux*
	m32r*-*-linux*
	m68*-*-linux*
	m68*-*-uclinux*
	mips*64*-*-linux*
	mips*-*-linux*
	powerpc[64]-*-linux*
	s390[x]-*-linux*
	sh-*-linux*
	spu*-*-*
	x86_64-*-linux*

Configuring GDBserver you should specify the same machine for host and
target (which are the machine that GDBserver is going to run on.  This
is not the same as the machine that GDB is going to run on; building
GDBserver automatically as part of building a whole tree of tools does
not currently work if cross-compilation is involved (we don't get the
right CC in the Makefile, to start with)).

Building GDBserver for your target is very straightforward.  If you build
GDB natively on a target which GDBserver supports, it will be built
automatically when you build GDB.  You can also build just GDBserver:

	% mkdir obj
	% cd obj
	% path-to-gdbserver-sources/configure
	% make

If you prefer to cross-compile to your target, then you can also build
GDBserver that way.  In a Bourne shell, for example:

	% export CC=your-cross-compiler
	% path-to-gdbserver-sources/configure your-target-name
	% make

Using GDBreplay:

A special hacked down version of GDBserver can be used to replay remote
debug log files created by GDB.  Before using the GDB "target" command to
initiate a remote debug session, use "set remotelogfile <filename>" to tell
GDB that you want to make a recording of the serial or tcp session.  Note
that when replaying the session, GDB communicates with GDBreplay via tcp,
regardless of whether the original session was via a serial link or tcp.

Once you are done with the remote debug session, start GDBreplay and
tell it the name of the log file and the host and port number that GDB
should connect to (typically the same as the host running GDB):

	$ gdbreplay logfile host:port

Then start GDB (preferably in a different screen or window) and use the
"target" command to connect to GDBreplay:

	(gdb) target remote host:port

Repeat the same sequence of user commands to GDB that you gave in the
original debug session.  GDB should not be able to tell that it is talking
to GDBreplay rather than a real target, all other things being equal.  Note
that GDBreplay echos the command lines to stderr, as well as the contents of
the packets it sends and receives.  The last command echoed by GDBreplay is
the next command that needs to be typed to GDB to continue the session in
sync with the original session.