====== gdb ======

===== Example code with a deadlock =====

Build your target with debug symbols. For example, with [[http://git.dlma.com/testcode.git/|the testcode project]], make with the target "debug"

  make debug

If the deadlock was compiled in, then run it like so:

  product/testcode &

Then you can run gdb and attach to the process in one of the following ways.

  $ gdb -p <pid-of-testcode>
  
  $ gdb product/testcode <pid-of-testcode>
  
  $ gdb
  (gdb) attach <pid-of-testcode>
  

===== Tips =====

Show all the backtraces:

  (gdb) set pagination off
  (gdb) thread apply all bt

Other useful tips:

  (gdb) t a a f       # thread apply all frame
  (gdb) t a a bt 3    # thread apply all backtrace bottom three frames
  (gdb) t a a bt -3   # thread apply all backtrace top three frames

===== Detecting a Deadlock =====

Get high level info on the threads:

  (gdb) info threads
    Id   Target Id         Frame
  * 3    Thread 0x76c9d450 (LWP 24793) "testcode" 0x00013c38 in std::lock_guard<std::mutex>::lock_guard (this=0x76c9cd9c, __m=...) at /usr/include/c++/4.9/mutex:377
    2    Thread 0x7649d450 (LWP 24794) "testcode" 0x76ded780 in __lll_lock_wait (futex=futex@entry=0x31464 <(anonymous namespace)::left_chopstick>, private=<optimized out>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46
    1    Thread 0x76fdd000 (LWP 24792) "testcode" 0x76de7274 in pthread_join (threadid=<optimized out>, thread_return=0x0) at pthread_join.c:92

Make notes of those Thread IDs at LWP, and choose the thread you want, and choose a stack frame that shows the mutex of interest.

   (gdb) thread 3
   [Switching to thread 3 (Thread 0x76c9d450 (LWP 24793))]
   #0  0x76ded780 in __lll_lock_wait (futex=futex@entry=0x3147c <(anonymous namespace)::right_chopstick>, private=<optimized out>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46
   46      ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c: No such file or directory.
   (gdb) bt
   #0  0x76ded780 in __lll_lock_wait (futex=futex@entry=0x3147c <(anonymous namespace)::right_chopstick>, private=<optimized out>) at ../ports/sysdeps/unix/sysv/linux/arm/nptl/lowlevellock.c:46
   #1  0x76de81a4 in __GI___pthread_mutex_lock (mutex=0x3147c <(anonymous namespace)::right_chopstick>) at pthread_mutex_lock.c:79
   #2  0x00013868 in __gthread_mutex_lock (__mutex=0x3147c <(anonymous namespace)::right_chopstick>) at /usr/include/arm-linux-gnueabihf/c++/4.9/bits/gthr-default.h:748
   #3  0x00013b98 in std::mutex::lock (this=0x3147c <(anonymous namespace)::right_chopstick>) at /usr/include/c++/4.9/mutex:135
   #4  0x00013c38 in std::lock_guard<std::mutex>::lock_guard (this=0x76c9cd9c, __m=...) at /usr/include/c++/4.9/mutex:377
   #5  0x00019e4c in (anonymous namespace)::do_work_on_right (have_left=true, tick=0) at deadlock/deadlock.cpp:43
   ...
   #10 0x0001b37c in std::thread::_Impl<std::_Bind_simple<deadlock()::<lambda()>()> >::_M_run(void) (this=0x5f6084) at /usr/include/c++/4.9/thread:115
   #11 0x76f54348 in ?? () from /usr/lib/arm-linux-gnueabihf/libstdc++.so.6
   Backtrace stopped: previous frame identical to this frame (corrupt stack?)
   (gdb) f 4
   #4  0x00013c38 in std::lock_guard<std::mutex>::lock_guard (this=0x76c9cd9c, __m=...) at /usr/include/c++/4.9/mutex:377
   377           { _M_device.lock(); }
   (gdb) info args
   this = 0x76c9cd9c
   __m = @0x3147c: {<std::__mutex_base> = {_M_mutex = {__data = {__lock = 2, __count = 0, __owner = 24794, __kind = 0, __nusers = 1, {__spins = 0, __list = {__next = 0x0}}}, __size = "\002... }}, <No data fields>}
   
Note the <nowiki>__owner</nowiki> of the mutex that thread 3 is waiting on. It's 24794. That's thread 2.  Likewise...

   (gdb) t 2
   (gdb) bt
   (gdb) f 3
   (gdb) p *this
   $8 = {<std::__mutex_base> = {_M_mutex = {__data = {__lock = 2, __count = 0, __owner = 24793, __kind = 0, __nusers = 1, {__spins = 0, __list = {__next = 0x0}}}, __size = "\002... }}, <No data fields>}
   
Note the <nowiki>__owner</nowiki> of the mutex that thread 2 is waiting on. It's 24793. That's thread 3. There's your deadlock.

==== Attaching to a remote target ====

  - Deploy gdb server with the remote target. Launch remote target with gdb server. 
  - Untar remote libraries to a local dir. Eg., 487.72E04128A-2371582-rootfs.tar.gz in my ~/Downloads directory.

    $ /usr/local/arm/bin/arm-linux-gdb builds/myapp.sym
    GNU gdb (GDB) 7.5.1
    This GDB was configured as "--host=i686-build_pc-linux-gnu --target=arm-brcm-linux-gnueabi".
    Reading symbols from builds/myapp.sym...done.
    (gdb) set sysroot ~/Downloads/rootfs/firmware.obj/root/
    (gdb) set solib-search-path builds/myapp.dir/
    (gdb) target remote 10.15.24.54:5555
    Remote debugging using 10.15.24.54:5555
    ...
    (gdb) c

===== Processing a backtrace from code =====

When using a define like this...

<code cpp>
#define PRINT_BACKTRACE { void *stack[32];                                   \
    const int nptrs = ::backtrace(stack, sizeof(stack) / sizeof(stack[0]));  \
    if (nptrs) ::backtrace_symbols_fd(stack, nptrs, STDERR_FILENO);          \
}
</code>

You might get output like this:

    /lib/libUILib.so(_ZN12FakeSymbolDoesNotMatterEb+0x288)[0xb3831a78]
    Application[0x4c51d0]
    
You can get source code lines if you can calculate the offsets from each module. You can get those module base addresses from /proc/<pid>/maps

    target:$ egrep " r-xp .*/(Application|libUILib.so)" /proc/$(ps -a | awk '/Application/{print $1}')/maps
    00010000-00dfb000 r-xp 00000000 00:0d 135649148  /bin/Application
    b3578000-b3f45000 r-xp 00000000 00:0d 68148928   /lib/libUILib.so

So, for example, the first frame is at: libUILib.so + (0xb3831a78 - 0xb3578000). You can use addr2line to get that:

    $ addr2line -e path/to/libUILib.so 0x2b9a78
    path/to/MySource.cpp:5712