Build your target with debug symbols. For example, with the testcode project, make with the target “debug”
make debug
If the deadlock was compiled in, then run it like so:
product/testcode &
Then you can run gdb and attach to the process in one of the following ways.
$ gdb -p <pid-of-testcode> $ gdb product/testcode <pid-of-testcode> $ gdb (gdb) attach <pid-of-testcode>
Show all the backtraces:
(gdb) set pagination off (gdb) thread apply all bt
Other useful tips:
(gdb) t a a f # thread apply all frame (gdb) t a a bt 3 # thread apply all backtrace bottom three frames (gdb) t a a bt -3 # thread apply all backtrace top three frames
Get high level info on the threads:
(gdb) info threads Id Target Id Frame * 3 Thread 0x76c9d450 (LWP 24793) "testcode" 0x00013c38 in std::lock_guard<std::mutex>::lock_guard (this=0x76c9cd9c, __m=...) at /usr/include/c++/4.9/mutex:377 2 Thread 0x7649d450 (LWP 24794) "testcode" 0x76ded780 in __lll_lock_wait (futex=futex@entry=0x31464 <(anonymous namespace)::left_chopstick>, private=<optimized out>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46 1 Thread 0x76fdd000 (LWP 24792) "testcode" 0x76de7274 in pthread_join (threadid=<optimized out>, thread_return=0x0) at pthread_join.c:92
Make notes of those Thread IDs at LWP, and choose the thread you want, and choose a stack frame that shows the mutex of interest.
(gdb) thread 3
[Switching to thread 3 (Thread 0x76c9d450 (LWP 24793))]
#0 0x76ded780 in __lll_lock_wait (futex=futex@entry=0x3147c <(anonymous namespace)::right_chopstick>, private=<optimized out>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46
46 ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c: No such file or directory.
(gdb) bt
#0 0x76ded780 in __lll_lock_wait (futex=futex@entry=0x3147c <(anonymous namespace)::right_chopstick>, private=<optimized out>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46
#1 0x76de81a4 in __GI___pthread_mutex_lock (mutex=0x3147c <(anonymous namespace)::right_chopstick>) at pthread_mutex_lock.c:79
#2 0x00013868 in __gthread_mutex_lock (__mutex=0x3147c <(anonymous namespace)::right_chopstick>) at /usr/include/arm-linux-gnueabihf/c++/4.9/bits/gthr-default.h:748
#3 0x00013b98 in std::mutex::lock (this=0x3147c <(anonymous namespace)::right_chopstick>) at /usr/include/c++/4.9/mutex:135
#4 0x00013c38 in std::lock_guard<std::mutex>::lock_guard (this=0x76c9cd9c, __m=...) at /usr/include/c++/4.9/mutex:377
#5 0x00019e4c in (anonymous namespace)::do_work_on_right (have_left=true, tick=0) at deadlock/deadlock.cpp:43
...
#10 0x0001b37c in std::thread::_Impl<std::_Bind_simple<deadlock()::<lambda()>()> >::_M_run(void) (this=0x5f6084) at /usr/include/c++/4.9/thread:115
#11 0x76f54348 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb) f 4
#4 0x00013c38 in std::lock_guard<std::mutex>::lock_guard (this=0x76c9cd9c, __m=...) at /usr/include/c++/4.9/mutex:377
377 { _M_device.lock(); }
(gdb) info args
this = 0x76c9cd9c
__m = @0x3147c: {<std::__mutex_base> = {_M_mutex = {__data = {__lock = 2, __count = 0, __owner = 24794, __kind = 0, __nusers = 1, {__spins = 0, __list = {__next = 0x0}}}, __size = "\002... }}, <No data fields>}
Note the __owner of the mutex that thread 3 is waiting on. It's 24794. That's thread 2. Likewise…
(gdb) t 2
(gdb) bt
(gdb) f 3
(gdb) p *this
$8 = {<std::__mutex_base> = {_M_mutex = {__data = {__lock = 2, __count = 0, __owner = 24793, __kind = 0, __nusers = 1, {__spins = 0, __list = {__next = 0x0}}}, __size = "\002... }}, <No data fields>}
Note the __owner of the mutex that thread 2 is waiting on. It's 24793. That's thread 3. There's your deadlock.
$ /usr/local/arm/bin/arm-linux-gdb builds/myapp.sym GNU gdb (GDB) 7.5.1 This GDB was configured as "--host=i686-build_pc-linux-gnu --target=arm-brcm-linux-gnueabi". Reading symbols from builds/myapp.sym...done. (gdb) set sysroot ~/Downloads/rootfs/firmware.obj/root/ (gdb) set solib-search-path builds/myapp.dir/ (gdb) target remote 10.15.24.54:5555 Remote debugging using 10.15.24.54:5555 ... (gdb) c
When using a define like this…
#define PRINT_BACKTRACE { void *stack[32]; \
const int nptrs = ::backtrace(stack, sizeof(stack) / sizeof(stack[0])); \
if (nptrs) ::backtrace_symbols_fd(stack, nptrs, STDERR_FILENO); \
}
You might get output like this:
/lib/libUILib.so(_ZN12FakeSymbolDoesNotMatterEb+0x288)[0xb3831a78] Application[0x4c51d0]
You can get source code lines if you can calculate the offsets from each module. You can get those module base addresses from /proc/<pid>/maps
target:$ egrep " r-xp .*/(Application|libUILib.so)" /proc/$(ps -a | awk '/Application/{print $1}')/maps
00010000-00dfb000 r-xp 00000000 00:0d 135649148 /bin/Application
b3578000-b3f45000 r-xp 00000000 00:0d 68148928 /lib/libUILib.so
So, for example, the first frame is at: libUILib.so + (0xb3831a78 - 0xb3578000). You can use addr2line to get that:
$ addr2line -e path/to/libUILib.so 0x2b9a78 path/to/MySource.cpp:5712