Build your target with debug symbols. For example, with the testcode project, make with the target “debug”

make debug

If the deadlock was compiled in, then run it like so:

product/testcode &

Then you can run gdb and attach to the process in one of the following ways.

$ gdb -p <pid-of-testcode>

$ gdb product/testcode <pid-of-testcode>

$ gdb
(gdb) attach <pid-of-testcode>

Show all the backtraces:

(gdb) set pagination off
(gdb) thread apply all bt

Other useful tips:

(gdb) t a a f       # thread apply all frame
(gdb) t a a bt 3    # thread apply all backtrace bottom three frames
(gdb) t a a bt -3   # thread apply all backtrace top three frames

Get high level info on the threads:

(gdb) info threads
  Id   Target Id         Frame
* 3    Thread 0x76c9d450 (LWP 24793) "testcode" 0x00013c38 in std::lock_guard<std::mutex>::lock_guard (this=0x76c9cd9c, __m=...) at /usr/include/c++/4.9/mutex:377
  2    Thread 0x7649d450 (LWP 24794) "testcode" 0x76ded780 in __lll_lock_wait (futex=futex@entry=0x31464 <(anonymous namespace)::left_chopstick>, private=<optimized out>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46
  1    Thread 0x76fdd000 (LWP 24792) "testcode" 0x76de7274 in pthread_join (threadid=<optimized out>, thread_return=0x0) at pthread_join.c:92

Make notes of those Thread IDs at LWP, and choose the thread you want, and choose a stack frame that shows the mutex of interest.

 (gdb) thread 3
 [Switching to thread 3 (Thread 0x76c9d450 (LWP 24793))]
 #0  0x76ded780 in __lll_lock_wait (futex=futex@entry=0x3147c <(anonymous namespace)::right_chopstick>, private=<optimized out>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46
 46      ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c: No such file or directory.
 (gdb) bt
 #0  0x76ded780 in __lll_lock_wait (futex=futex@entry=0x3147c <(anonymous namespace)::right_chopstick>, private=<optimized out>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46
 #1  0x76de81a4 in __GI___pthread_mutex_lock (mutex=0x3147c <(anonymous namespace)::right_chopstick>) at pthread_mutex_lock.c:79
 #2  0x00013868 in __gthread_mutex_lock (__mutex=0x3147c <(anonymous namespace)::right_chopstick>) at /usr/include/arm-linux-gnueabihf/c++/4.9/bits/gthr-default.h:748
 #3  0x00013b98 in std::mutex::lock (this=0x3147c <(anonymous namespace)::right_chopstick>) at /usr/include/c++/4.9/mutex:135
 #4  0x00013c38 in std::lock_guard<std::mutex>::lock_guard (this=0x76c9cd9c, __m=...) at /usr/include/c++/4.9/mutex:377
 #5  0x00019e4c in (anonymous namespace)::do_work_on_right (have_left=true, tick=0) at deadlock/deadlock.cpp:43
 ...
 #10 0x0001b37c in std::thread::_Impl<std::_Bind_simple<deadlock()::<lambda()>()> >::_M_run(void) (this=0x5f6084) at /usr/include/c++/4.9/thread:115
 #11 0x76f54348 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
 Backtrace stopped: previous frame identical to this frame (corrupt stack?)
 (gdb) f 4
 #4  0x00013c38 in std::lock_guard<std::mutex>::lock_guard (this=0x76c9cd9c, __m=...) at /usr/include/c++/4.9/mutex:377
 377           { _M_device.lock(); }
 (gdb) info args
 this = 0x76c9cd9c
 __m = @0x3147c: {<std::__mutex_base> = {_M_mutex = {__data = {__lock = 2, __count = 0, __owner = 24794, __kind = 0, __nusers = 1, {__spins = 0, __list = {__next = 0x0}}}, __size = "\002... }}, <No data fields>}
 

Note the __owner of the mutex that thread 3 is waiting on. It's 24794. That's thread 2. Likewise…

 (gdb) t 2
 (gdb) bt
 (gdb) f 3
 (gdb) p *this
 $8 = {<std::__mutex_base> = {_M_mutex = {__data = {__lock = 2, __count = 0, __owner = 24793, __kind = 0, __nusers = 1, {__spins = 0, __list = {__next = 0x0}}}, __size = "\002... }}, <No data fields>}
 

Note the __owner of the mutex that thread 2 is waiting on. It's 24793. That's thread 3. There's your deadlock.

1. Deploy gdb server with the remote target. Launch remote target with gdb server.
2. Untar remote libraries to a local dir. Eg., 487.72E04128A-2371582-rootfs.tar.gz in my ~/Downloads directory.

  $ /usr/local/arm/bin/arm-linux-gdb builds/myapp.sym
  GNU gdb (GDB) 7.5.1
  This GDB was configured as "--host=i686-build_pc-linux-gnu --target=arm-brcm-linux-gnueabi".
  Reading symbols from builds/myapp.sym...done.
  (gdb) set sysroot ~/Downloads/rootfs/firmware.obj/root/
  (gdb) set solib-search-path builds/myapp.dir/
  (gdb) target remote 10.15.24.54:5555
  Remote debugging using 10.15.24.54:5555
  ...
  (gdb) c

When using a define like this…

#define PRINT_BACKTRACE { void *stack[32];                                   \
    const int nptrs = ::backtrace(stack, sizeof(stack) / sizeof(stack[0]));  \
    if (nptrs) ::backtrace_symbols_fd(stack, nptrs, STDERR_FILENO);          \
}

You might get output like this:

  /lib/libUILib.so(_ZN12FakeSymbolDoesNotMatterEb+0x288)[0xb3831a78]
  Application[0x4c51d0]
  

You can get source code lines if you can calculate the offsets from each module. You can get those module base addresses from /proc/<pid>/maps

  target:$ egrep " r-xp .*/(Application|libUILib.so)" /proc/$(ps -a | awk '/Application/{print $1}')/maps
  00010000-00dfb000 r-xp 00000000 00:0d 135649148  /bin/Application
  b3578000-b3f45000 r-xp 00000000 00:0d 68148928   /lib/libUILib.so

So, for example, the first frame is at: libUILib.so + (0xb3831a78 - 0xb3578000). You can use addr2line to get that:

  $ addr2line -e path/to/libUILib.so 0x2b9a78
  path/to/MySource.cpp:5712