We built a vendor-neutral accelerator plugin architecture in lldb-server for debugging applications that span CPUs and GPUs. It separates shared debugger infrastructure from hardware-specific device control.
Samartha Chandrashekar & Nick Parker You click play on a Friday night, and in the second that follows, dozens of services answer in milliseconds. Most of them run on Titus, our internal container platform.
GitHub's new Git storage architecture is built so thousands of agents can push to a single repository at once. Here's how it works.
We're rebuilding GitHub's Git infrastructure while GitHub keeps running, creating a foundation for agent-scale software development.
A configuration mismatch in GLM 5.3 on Workers AI triggered an SGLang bug that exposed KV cache pages across requests. Here is how we addressed the issue.
We are launching a PHP Capture the Flag challenge integrated with our Bug Bounty program to test our WAF protections. Security researchers are invited to bypass our defenses and capture the flag to help make the Internet safer.
Naive byte-range caching can waste gigabytes of origin bandwidth on small requests. We rebuilt our cache architecture in Pingora to coordinate partial fetches automatically, cutting R2 p99 latency in half.
A month’s worth of security breaches in a single day – will things get worse before they improve?
We moved a copy of 99.99999% of GitHub's repositories to Azure. Here's how per-reference quorums will bring over the rest, and how object storage lets reads scale independently.
Modern AI applications rarely execute in one place. A workload can begin in a high-level framework and move back and forth between native CPU code and kernels running on a GPU. When a bug crosses those boundaries, developers need to understand the host application and the device together.
That is harder than attaching a conventional debugger to another process. The CPU and GPU may use different instruction sets, registers, memory spaces, and execution models. The GPU is also controlled through a runtime and driver inside the host process, so debugging the device often requires careful coordination with the CPU.
Historically, GPU vendors built device-specific debuggers, including AMD’s ROCgdb and NVIDIA’s CUDA-GDB. These tools provide deep support for their respective platforms, but common debugger capabilities must often be implemented and maintained separately.
At Meta, we have been developing a different approach for LLDB: a common accelerator plugin architecture in lldb-server. In this post, “GPU” refers to the concrete devices and examples, while “accelerator” refers to LLDB’s broader, vendor-neutral plugin abstraction. The architecture has evolved through contributions from engineers at Meta and across the accelerator ecosystem. It keeps the shared debugger independent of any one GPU while giving hardware vendors a focused place to implement device-specific behavior.
A traditional debugger generally assumes that one process has one architecture and a consistent view of threads, registers, and memory. GPU programs challenge each of those assumptions.
A host process might run on x86-64 while its kernels use a completely different instruction set. A CPU thread has a familiar register context, while a GPU may execute groups of lanes in lockstep. Host and device pointers can refer to different address spaces. A GPU can expose thousands of execution lanes, which do not map cleanly onto the thread model that most command-line and graphical debuggers present.
Combining all of this state into one artificial process model makes the debugger more complicated. Every operation must first determine whether it applies to the CPU or the GPU. Register handling, memory reads, breakpoints, stepping, and stop events all become conditional on the selected hardware. A single process model also forces the CPU and GPU into one unified memory map, even though their address spaces can be independent: The same address, such as 0x1000, can refer to different locations in the CPU and GPU processes.
Instead of hiding those differences, the LLDB accelerator architecture gives them a clear boundary.
LLDB uses a companion process called lldb-server to launch or attach to a program and perform low-level debugging operations. The LLDB client communicates with the server through the GDB remote protocol, a command protocol for operations such as reading memory, inspecting registers, controlling execution, and setting breakpoints.
Because lldb-server already controls the native process, it is also the right place to coordinate with an accelerator runtime. The new plugin interface lets a hardware-specific backend work alongside the normal CPU debugger. The backend translates a vendor’s device debugging API into concepts LLDB already understands, such as processes, threads, registers, memory, modules, and stop reasons.
This separation keeps the LLDB client vendor-neutral. Common improvements to the command line, remote protocol, symbol handling, and IDE integration can benefit every accelerator. At the same time, each vendor remains responsible for the details that are unique to its hardware and driver stack.
GPU debugging support is activated only when the debugged application uses a supported GPU runtime. For CPU-only applications, LLDB and lldb-server continue through the existing native debugging path without incurring runtime computational overhead.
The architecture represents the CPU and accelerator as separate LLDB targets.
Each target has its own architecture and is associated with a debugger process that maintains its threads, registers, address space, and execution state. The CPU continues to use LLDB’s established native debugging path. When the accelerator becomes available, the plugin provides a second connection that LLDB uses to create the device target.
To the debugger, the two targets remain technically independent. To the developer, they can appear in one LLDB or IDE session. This avoids forcing incompatible CPU and GPU concepts into a single process while still supporting a unified workflow.
A typical debugging session follows a small set of steps:
Structured actions coordinate the two sides. A plugin can request CPU breakpoints, provide connection information, report device modules, and synchronize host and device execution. The common protocol describes what the debugger needs to do without exposing vendor-specific API details to the LLDB client.
We also built a mock accelerator plugin that exercises this flow without specialized hardware. It lets the LLVM test suite verify discovery, breakpoint coordination, connection creation, and basic device state. This has been important for developing shared infrastructure that can be reviewed and tested independently of any particular GPU.
The vector dot-product HIP sample produced this abridged session:
The plugin boundary is intentionally based on debugger operations rather than concepts from a specific GPU platform. That makes it useful across devices with very different instruction sets, runtime APIs, and execution models.
The framework supports postmortem GPU debugging as well as live sessions. GPU core-dump plugins use a common base class for their LLDB integration, while vendor implementations provide the device-specific thread, register, memory, and module state.
GPU debugging efforts at Meta are already leveraging the architecture. Because we operate a heterogeneous fleet of GPUs with different hardware and software stacks, the fact that these efforts use the same model is an important validation of the design: the common LLDB layer can remain stable while each backend represents its own device accurately.
At the time of writing, we have upstreamed most of the common framework to LLVM. This includes the lldb-server plugin interface; GDB-remote packets for plugin discovery, initialization, runtime-event coordination, and GPU connection creation; support for associating CPU and GPU debugger processes while preserving their independent architecture; support for multiple address spaces, and hardware-independent mock tests. We expect to upstream the remaining pieces soon.
Next, we plan to finish upstreaming the remaining common infrastructure and expand the vendor integrations built on it. Our goal is that integrating new hardware with LLDB should require only a vendor-specific plugin, not a separate debugger to build and maintain.
We are grateful to Meta’s Developer Infrastructure leadership team, especially Shane Nay, Todd Fiala, and Ale Contenti, whose direct support made this work possible.
We also thank our Debugging & Tooling teammates Jacob Lalonde, David Young, Alexandre Perez, Zhiyuan Li, Piyush Jaiswal, and Philip DePetro for their collaboration and support.
Finally, we thank the open source community, especially Jonas Devlieghere, Jim Ingham, David Spickett, and Jason Molenda, for reviewing the RFCs and upstream pull requests.
The post Extending LLDB to Debug Applications That Span CPUs and GPUs appeared first on Engineering at Meta.
