Remote Debugging
================
Remote debugging refers to the act of debugging a process which is running on a
different system, than the debugger itself. We shall refer to the system
running the debugger as the local system, while the system running the debugged
process will be the remote system.
To enable remote debugging, LLDB employs a client-server architecture. The
client part runs on the local system and the remote system runs the server. The
client and server communicate using the gdb-remote protocol, usually
transported over TCP/IP. More information on the protocol can be found here and
the LLDB-specific extensions are documented in docs/lldb-gdb-remote.txt file
inside LLDB source repository. Besides the gdb-remote stub, the server part of
LLDB also consists of a platform binary, which is responsible for performing
advanced debugging operations, like copying files from/to the remote system and
can be used to execute arbitrary shell commands on the remote system.
In order to reduce code complexity and improve remote debugging experience LLDB
on Linux and macOS uses the remote debugging stub even when debugging a process
locally. This is achieved by spawning a remote stub process locally and
communicating with it over the loopback interface. In the case of local
debugging this whole process is transparent to the user. The platform binary is
not used in this case, since no file transfers are needed.
Preparation for Remote Debugging
---------------------------------
While the process of actual debugging (stepping, backtraces, evaluating
expressions) is same as in the local case, in the case of remote debugging,
more preparation is needed as the required binaries cannot started on the
remote system automatically. Also, if the remote system runs a different OS or
architecture, the server component needs to be compiled separately.
Remote system
*************
On Linux and Android, all required remote functionality is contained in the
lldb-server binary. This binary combines the functionality of the platform and
gdb-remote stub. A single binary facilitates deployment and reduces code size,
since the two functions share a lot of code. The lldb-server binary is also
statically linked with the rest of LLDB (unlike lldb, which dynamically links
to liblldb.so by default), so it does not have any dependencies on the rest of
lldb. On macOS and iOS, the remote-gdb functionality is implemented by the
debugserver binary, which you will need to deploy alongside lldb-server.
The binaries mentioned above need to be present on the remote system to enable
remote debugging. You can either compile on the remote system directly or copy
them from the local machine. If compiling locally and the remote architecture
differs from the local one, you will need to cross-compile the correct version
of the binaries. More information on cross-compiling LLDB can be found on the
build page.
Once the binaries are in place, you just need to run the lldb-server in
platform mode and specify the port it should listen on. For example, the
command
::
remote% lldb-server platform --listen "*:1234" --server
will start the LLDB platform and wait for incoming connections from any address
to port 1234. Specifying an address instead of * will only allow connections
originating from that address. Adding a --server parameter to the command line
will fork off a new process for every incoming connection, allowing multiple
parallel debug sessions.
Local system
************
On the local system, you need to let LLDB know that you intend to do remote
debugging. This is achieved through the platform command and its sub-commands.
As a first step you need to choose the correct platform plug-in for your remote
system. A list of available plug-ins can be obtained through platform list.
::
local% lldb
(lldb) platform list
Available platforms:
host: Local macOS user platform plug-in.
remote-freebsd: Remote FreeBSD user platform plug-in.
remote-linux: Remote Linux user platform plug-in.
remote-netbsd: Remote NetBSD user platform plug-in.
remote-windows: Remote Windows user platform plug-in.
remote-android: Remote Android user platform plug-in.
remote-ios: Remote iOS platform plug-in.
remote-macosx: Remote macOS user platform plug-in.
ios-simulator: iOS simulator platform plug-in.
darwin-kernel: Darwin Kernel platform plug-in.
tvos-simulator: Apple TV simulator platform plug-in.
watchos-simulator: Apple Watch simulator platform plug-in.
remote-tvos: Remote Apple TV platform plug-in.
remote-watchos: Remote Apple Watch platform plug-in.
remote-gdb-server: A platform that uses the GDB remote protocol as the communication transport.
The default platform is the platform host which is used for local debugging.
Apart from this, the list should contain a number of plug-ins, for debugging
different kinds of systems. The remote plug-ins are prefixed with "remote-".
For example, to debug a remote Linux application:
::
(lldb) platform select remote-linux
After selecting the platform plug-in, you should receive a prompt which
confirms the selected platform, and states that you are not connected. This is
because remote plug-ins need to be connected to their remote platform
counterpart to operate. This is achieved using the platform connect command.
This command takes a number of arguments (as always, use the help command to
find out more), but normally you only need to specify the address to connect
to, e.g.:
::
(lldb) platform connect connect://remote:1234
Platform: remote-linux
Triple: x86_64-gnu-linux
Hostname: remote
Connected: yes
WorkingDir: /tmp
Note that the platform has a working directory of /tmp. This directory will be
used as the directory that executables will be uploaded to by default when
launching a process from local.
After this, you should be able to debug normally. You can use the process
attach to attach to an existing remote process or target create, process launch
to start a new one. The platform plugin will transparently take care of
uploading or downloading the executable in order to be able to debug. If your
application needs additional files, you can transfer them using the platform
commands: get-file, put-file, mkdir, etc. The environment can be prepared
further using the platform shell command.
When using the "remote-android" platform, the client LLDB forwards two ports, one
for connecting to the platform, and another for connecting to the gdbserver.
The client ports are configurable through the environment variables
ANDROID_PLATFORM_LOCAL_PORT and ANDROID_PLATFORM_LOCAL_GDB_PORT, respectively.
Launching a locally built process on the remote machine
-------------------------------------------------------
Install and run in the platform working directory
*************************************************
To launch a locally built process on the remote system in the platform working
directory:
::
(lldb) file a.out
(lldb) run
This will cause LLDB to create a target with the "a.out" executable that you
cross built. The "run" command will cause LLDB to upload "a.out" to the
platform's current working directory only if the file has changed. The platform
connection allows us to transfer files, but also allows us to get the MD5
checksum of the file on the other end and only upload the file if it has
changed. LLDB will automatically launch a lldb-server in gdbremote mode to
allow you to debug this executable, connect to it and start your debug session
for you.
Changing the platform working directory
***************************************
You can change the platform working directory while connected to the platform
with:
::
(lldb) platform settings -w /usr/local/bin
And you can verify it worked using "platform status":
::
(lldb) platform status
Platform: remote-linux
Triple: x86_64-gnu-linux
Hostname: remote
Connected: yes
WorkingDir: /usr/local/bin
If we run again, the program will be installed into ``/usr/local/bin``.
Install and run by specifying a remote install path
***************************************************
If you want the "a.out" executable to be installed into "/bin/a.out" instead of
the platform's current working directory, we can set the platform file
specification using python:
::
(lldb) file a.out
(lldb) script lldb.target.module['a.out'].SetPlatformFileSpec("/bin/a.out")
(lldb) run
Now when you run your program, the program will be uploaded to "/bin/a.out"
instead of the platform current working directory. Only the main executable is
uploaded to the remote system by default when launching the application. If you
have shared libraries that should also be uploaded, then you can add the
locally build shared library to the current target and set its platform file
specification:
::
(lldb) file a.out
(lldb) target module add /local/build/libfoo.so
(lldb) target module add /local/build/libbar.so
(lldb) script lldb.target.module['libfoo.so'].SetPlatformFileSpec("/usr/lib/libfoo.so")
(lldb) script lldb.target.module['libbar.so'].SetPlatformFileSpec("/usr/local/lib/libbar.so")
(lldb) run
Attaching to a remote process
*****************************
If you want to attach to a remote process, you can first list the processes on
the remote system:
::
(lldb) platform process list
223 matching processes were found on "remote-linux"
PID PARENT USER TRIPLE NAME
====== ====== ========== ======================== ============================
68639 90652 x86_64-apple-macosx lldb
...
Then attaching is as simple as specifying the remote process ID:
::
(lldb) attach 68639
Codebase Browser
llvm