User Tools

Site Tools


gdb

gdb

Example code with a deadlock

Build your target with debug symbols. For example, with the testcode project, make with the target “debug”

make debug

If the deadlock was compiled in, then run it like so:

product/testcode &

Then you can run gdb and attach to the process in one of the following ways.

$ gdb -p <pid-of-testcode>

$ gdb product/testcode <pid-of-testcode>

$ gdb
(gdb) attach <pid-of-testcode>

Tips

Show all the backtraces:

(gdb) set pagination off
(gdb) thread apply all bt

Other useful tips:

(gdb) t a a f       # thread apply all frame
(gdb) t a a bt 3    # thread apply all backtrace bottom three frames
(gdb) t a a bt -3   # thread apply all backtrace top three frames

Detecting a Deadlock

Get high level info on the threads:

(gdb) info threads
  Id   Target Id         Frame
* 3    Thread 0x76c9d450 (LWP 24793) "testcode" 0x00013c38 in std::lock_guard<std::mutex>::lock_guard (this=0x76c9cd9c, __m=...) at /usr/include/c++/4.9/mutex:377
  2    Thread 0x7649d450 (LWP 24794) "testcode" 0x76ded780 in __lll_lock_wait (futex=futex@entry=0x31464 <(anonymous namespace)::left_chopstick>, private=<optimized out>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46
  1    Thread 0x76fdd000 (LWP 24792) "testcode" 0x76de7274 in pthread_join (threadid=<optimized out>, thread_return=0x0) at pthread_join.c:92

Make notes of those Thread IDs at LWP, and choose the thread you want, and choose a stack frame that shows the mutex of interest.

 (gdb) thread 3
 [Switching to thread 3 (Thread 0x76c9d450 (LWP 24793))]
 #0  0x76ded780 in __lll_lock_wait (futex=futex@entry=0x3147c <(anonymous namespace)::right_chopstick>, private=<optimized out>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46
 46      ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c: No such file or directory.
 (gdb) bt
 #0  0x76ded780 in __lll_lock_wait (futex=futex@entry=0x3147c <(anonymous namespace)::right_chopstick>, private=<optimized out>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46
 #1  0x76de81a4 in __GI___pthread_mutex_lock (mutex=0x3147c <(anonymous namespace)::right_chopstick>) at pthread_mutex_lock.c:79
 #2  0x00013868 in __gthread_mutex_lock (__mutex=0x3147c <(anonymous namespace)::right_chopstick>) at /usr/include/arm-linux-gnueabihf/c++/4.9/bits/gthr-default.h:748
 #3  0x00013b98 in std::mutex::lock (this=0x3147c <(anonymous namespace)::right_chopstick>) at /usr/include/c++/4.9/mutex:135
 #4  0x00013c38 in std::lock_guard<std::mutex>::lock_guard (this=0x76c9cd9c, __m=...) at /usr/include/c++/4.9/mutex:377
 #5  0x00019e4c in (anonymous namespace)::do_work_on_right (have_left=true, tick=0) at deadlock/deadlock.cpp:43
 ...
 #10 0x0001b37c in std::thread::_Impl<std::_Bind_simple<deadlock()::<lambda()>()> >::_M_run(void) (this=0x5f6084) at /usr/include/c++/4.9/thread:115
 #11 0x76f54348 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
 Backtrace stopped: previous frame identical to this frame (corrupt stack?)
 (gdb) f 4
 #4  0x00013c38 in std::lock_guard<std::mutex>::lock_guard (this=0x76c9cd9c, __m=...) at /usr/include/c++/4.9/mutex:377
 377           { _M_device.lock(); }
 (gdb) info args
 this = 0x76c9cd9c
 __m = @0x3147c: {<std::__mutex_base> = {_M_mutex = {__data = {__lock = 2, __count = 0, __owner = 24794, __kind = 0, __nusers = 1, {__spins = 0, __list = {__next = 0x0}}}, __size = "\002... }}, <No data fields>}
 

Note the __owner of the mutex that thread 3 is waiting on. It's 24794. That's thread 2. Likewise…

 (gdb) t 2
 (gdb) bt
 (gdb) f 3
 (gdb) p *this
 $8 = {<std::__mutex_base> = {_M_mutex = {__data = {__lock = 2, __count = 0, __owner = 24793, __kind = 0, __nusers = 1, {__spins = 0, __list = {__next = 0x0}}}, __size = "\002... }}, <No data fields>}
 

Note the __owner of the mutex that thread 2 is waiting on. It's 24793. That's thread 3. There's your deadlock.

Attaching to a remote target

  1. Deploy gdb server with the remote target. Launch remote target with gdb server.
  2. Untar remote libraries to a local dir. Eg., 487.72E04128A-2371582-rootfs.tar.gz in my ~/Downloads directory.
  $ /usr/local/arm/bin/arm-linux-gdb builds/myapp.sym
  GNU gdb (GDB) 7.5.1
  This GDB was configured as "--host=i686-build_pc-linux-gnu --target=arm-brcm-linux-gnueabi".
  Reading symbols from builds/myapp.sym...done.
  (gdb) set sysroot ~/Downloads/rootfs/firmware.obj/root/
  (gdb) set solib-search-path builds/myapp.dir/
  (gdb) target remote 10.15.24.54:5555
  Remote debugging using 10.15.24.54:5555
  ...
  (gdb) c
gdb.txt · Last modified: 2021/05/21 00:04 (external edit)