Remote Debugging with GDB

Remote debugging:
Connecting to remote target
    a) (gdb) target remote /dev/ttyb   # serial-device
    b) (gdb) target remote host:port
    c) (gdb) target remote protocol:host:port
    d) (gdb) target remote | middleman-command   # gdb talks to a middleman command

Transfer files:
    remote put hostfile targetfile
    remote get targetfile hostfile
    remote delete targetfile

    (gdb) load filename    #   this command is used instead of 'run' in case of remote debugging
    (gdb) break
    (gdb) step
    (gdb) continue
    (gdb) detach
    (gdb) disconnect

a) Remote gdbserver on target with gdb on host (Need O.S. for this):
     target> gdbserver /dev/com1 emacs foo.txt   
    target> gdbserver host:2345 emacs foo.txt   
    target> gdbserver --attach /dev/com1 pid

    # Run a wrapper for the debugee program on the target-side
    target> gdbserver --wrapper env LD_PRELOAD=libtest.so -- :2222 ./testprog
   
    #Connect to the target gdbserver from my host gdb
    $ gdb myprogram
    (gdb) target remote myhost:9999
            0x00007f215893ba60 in ?? () from /lib64/ld-linux-x86-64.so.2
    (gdb) b main
    (gdb) continue

   # Send commands to gdbserver using the monitor command
    monitor help      # List the available monitor commands.
    monitor set debug 0 | 1      # Disable or enable general debugging messages   
    monitor set remote-debug 0|1    # Disable or enable debugging messages of the remote protocol
    monitor exit

   See Also: Remote Configuration
   More in-depth internal look at Remote Protocol

b) Remote Stub:  Used on Embedded devices without an O.S.

Blogs:
1) http://blog.flameeyes.eu/2010/02/20/remote-debugging-with-gdb-part-2-gdb
2) http://www.gentoo.org/proj/en/qa/backtraces.xml

Debugging Multiple Processes/Threads with GDB

Debugging multiple inferiors and programs
Gdb allows you to debug multiple programs at once without exiting the process you're currently running.
Gdb uses an inferior as a handle for a process. An inferior  is attachable/detachable from a process
    info inferiors    # display list of current inferiors.

     (gdb) info inferiors
       Num  Description       Executable
       2    process 2307      hello
     * 1    process 3401      goodbye

Switch to another inferior:
    inferior infno   # make inferior number infno the current inferior

Loading multiple inferiors:
1)    add-inferior [ -copies n ] [ -exec executable ] # n defaults to 1 and optional exec gives an empty inferior
2)    clone-inferior [ -copies n ] [ infno ]    # n defaults to 1 and infno defaults to current inferior number

Ending the inferiority complex:
remove-inferior infno        # Remove the empty inferior Note: Can't remove an inferior with a running assoc. process.
detach-inferior infno         # detaches from an inferior from assoc. process
kill-inferiror infno             # kills the process attached to the inferior. It can be removed or re-attached to some process.

---

Debugging Multi-threaded Programs:
    info threads    # displays thread info - thread number, thread-id,  stack frame summary
    thread threadno     # make threadno as active current thread and shows stack frame summary

Starting and Stopping Multi-thread Programs
2 modes of debugging:
1) All stop mode: aka 3 musketeers mode i.e. "All-for-one, One-for-all"
2) Non-stop mode:
    set target-async on|off    # Switch asynchronous mode.
    show target-async         # Show the current target-async setting.

    interrupt    # suspend execution of the running program (whole process in all-stop, current thread in non-stop modes)
    interrupt -a    # stop the whole program in non-stop mode

Howto do Reverse Debugging with GDB

Webinar on Reverse Debugging with DSF-GDB
This webinar introduces the new DSF-GDB debugger integration that is part of CDT 6.0. It illustrates the recent GDB additions such as Reverse debugging, Multi-Process and Non-stop multi-thread debugging.

GDB Reverse Debugging Wiki
GDB and Reverse Debugging
Presentation (PDF) on Internals of GDB Reverse Debug and Process Record Target
GDB Reverse Debugging Tutorial

Howto Debug with GDB

See Also:
Fun with strace and gdb - article on ibmdeveloperworks
GNU project debugging with gdb - article on ibmdeveloperworks

Debugging with GDB's Cheat-Sheet of Commonly Used Options

Gdb:
Compiling a program for debugging:
    g++ -ggdb -O0 hello.cpp   # 
    g++ -g3 -O0 hello.cpp       # -g3 compiles debugging info for preprocessor macros and no optimization

Running a program under the debugger without arguments
$> gdb myProgram

Setting the program arguments:
   a)  $> gdb --args myProgram  10 20 30 40
   b)  (gdb) start 10 20 30 40  # sets temporary breakpoint at main() and calls run.
   c)  (gdb) file myProgram
        (gdb) run 10 20 30 40
   d)   (gdb)  set args 10 20 30 40
         (gdb) show args
        (gdb) run
    e) $> gdb myProgram core
    f) $> gdb -tui myProgram core  # start vim/emacs ide for debug session