Remote Debugging with GDB

Remote debugging:
Connecting to remote target
    a) (gdb) target remote /dev/ttyb   # serial-device
    b) (gdb) target remote host:port
    c) (gdb) target remote protocol:host:port
    d) (gdb) target remote | middleman-command   # gdb talks to a middleman command

Transfer files:
    remote put hostfile targetfile
    remote get targetfile hostfile
    remote delete targetfile

    (gdb) load filename    #   this command is used instead of 'run' in case of remote debugging
    (gdb) break
    (gdb) step
    (gdb) continue
    (gdb) detach
    (gdb) disconnect

a) Remote gdbserver on target with gdb on host (Need O.S. for this):
     target> gdbserver /dev/com1 emacs foo.txt   
    target> gdbserver host:2345 emacs foo.txt   
    target> gdbserver --attach /dev/com1 pid

    # Run a wrapper for the debugee program on the target-side
    target> gdbserver --wrapper env LD_PRELOAD=libtest.so -- :2222 ./testprog
   
    #Connect to the target gdbserver from my host gdb
    $ gdb myprogram
    (gdb) target remote myhost:9999
            0x00007f215893ba60 in ?? () from /lib64/ld-linux-x86-64.so.2
    (gdb) b main
    (gdb) continue

   # Send commands to gdbserver using the monitor command
    monitor help      # List the available monitor commands.
    monitor set debug 0 | 1      # Disable or enable general debugging messages   
    monitor set remote-debug 0|1    # Disable or enable debugging messages of the remote protocol
    monitor exit

   See Also: Remote Configuration
   More in-depth internal look at Remote Protocol

b) Remote Stub:  Used on Embedded devices without an O.S.

Blogs:
1) http://blog.flameeyes.eu/2010/02/20/remote-debugging-with-gdb-part-2-gdb
2) http://www.gentoo.org/proj/en/qa/backtraces.xml

Debugging Multiple Processes/Threads with GDB

Debugging multiple inferiors and programs
Gdb allows you to debug multiple programs at once without exiting the process you're currently running.
Gdb uses an inferior as a handle for a process. An inferior  is attachable/detachable from a process
    info inferiors    # display list of current inferiors.

     (gdb) info inferiors
       Num  Description       Executable
       2    process 2307      hello
     * 1    process 3401      goodbye

Switch to another inferior:
    inferior infno   # make inferior number infno the current inferior

Loading multiple inferiors:
1)    add-inferior [ -copies n ] [ -exec executable ] # n defaults to 1 and optional exec gives an empty inferior
2)    clone-inferior [ -copies n ] [ infno ]    # n defaults to 1 and infno defaults to current inferior number

Ending the inferiority complex:
remove-inferior infno        # Remove the empty inferior Note: Can't remove an inferior with a running assoc. process.
detach-inferior infno         # detaches from an inferior from assoc. process
kill-inferiror infno             # kills the process attached to the inferior. It can be removed or re-attached to some process.

---

Debugging Multi-threaded Programs:
    info threads    # displays thread info - thread number, thread-id,  stack frame summary
    thread threadno     # make threadno as active current thread and shows stack frame summary

Starting and Stopping Multi-thread Programs
2 modes of debugging:
1) All stop mode: aka 3 musketeers mode i.e. "All-for-one, One-for-all"
2) Non-stop mode:
    set target-async on|off    # Switch asynchronous mode.
    show target-async         # Show the current target-async setting.

    interrupt    # suspend execution of the running program (whole process in all-stop, current thread in non-stop modes)
    interrupt -a    # stop the whole program in non-stop mode