Why AI agents struggle with RTL debugging, and what helps

Coding agents are built mostly for software work. Their working loop assumes that a test runs in seconds, that a failure prints a useful message, and that a command is cheap to rerun. Hardware design breaks all three. Chips are described in RTLRegister-transfer level: the code that describes a chip’s hardware, usually written in Verilog or SystemVerilog. A simulator runs it; synthesis tools turn it into gates. code and tested in a simulator. A simulation of a large design takes minutes to compile and run, so every rerun is expensive, and the state that explains a failure lives in internal signals that nobody printed.

Current models reason well about RTL. This note looks at what happens when that reasoning meets a real hardware debug loop. It asks where the agent's time goes, which of that time comes from hardware work rather than from agent setups in general, and what the agent's harnessThe environment around an AI agent: the tools it can call, how its commands run, and the instructions it gets. can do about it.

The scope is debugging: finding a bug, reproducing it, and proving the fix. Other RTL tasks, such as writing new design code or building verification environments, run into different problems. Later notes will cover those.

To answer it, I ran the same agent on the same bug in two setups: a plain shell, and Booley, an open-source agent harness for FPGA and ASIC work. I build Booley and ran this experiment myself, so weigh the comparison with that in mind. The problems described below, though, come from the plain-shell runs and apply to any agent setup for RTL.

The case: a forwarding bug in a RISC-V core¶

I wanted a bug where the environment, not the model's reasoning, would limit the agent. I started with bug-fixing tasks from a subset of CVDP, NVIDIA's benchmark of hardware design and verification problems. Those tasks were too small for this question: agents solved them quickly with any tooling, so the environment never had a chance to matter. I needed a design where reproducing a bug takes real simulation work.

So I looked for the largest high-quality open-source RTL codebase I could find. I chose Caliptra, a hardware root of trust: a security subsystem with its own processor and firmware.

I settled on a historical bug in VeeR EL2, the RISC-V processor inside Caliptra, and reintroduced it by reverting an upstream fix from the VeeR EL2 1.4 release.

In the scenario the bug report describes, firmware checks the contents of the processor's ROMRead-only memory. Here it holds the processor’s boot program. during startup. Under a particular sequence of reads and writes, that check fails even though the stored program is correct: the word at ROM address 0 reads back as zero.

The agent's task was to find and fix the cause in the RTL, then add a test that fails before the fix and passes afterward.

Why reproducing it was harder than finding it¶

The processor uses store-to-load forwarding: a read can take its value directly from a recent write to the same address instead of waiting for memory. To save power, it also blanks the address and data of a write that goes to its on-chip data memory (DCCM), setting both to zero. The forwarding check was missing its "this write goes out on the bus" condition, so it compared each read against that blanked address of zero. A read from ROM address 0, issued right after an on-chip write, "matched", skipped the real ROM read, and got the blanked value: zero.

In both setups, the agent usually identified the faulty line within five minutes. It then had to write a test that failed on the original design and passed after the fix. That would demonstrate the fault and help catch it if it returned later.

Two details made that test difficult to write:

• The test template marked ROM as a region where accesses may have side effects. That setting disables forwarding and hides the bug.
• The processor works on several instructions at once, each at a different stage of its pipeline. The bug needs the write and the read back to back, one stage apart. Fetching the test's instructions from ROM over the bus left gaps between them, so the test had to run its loop from the processor's fast on-chip instruction memory (ICCM) instead.

Most of the work went into building a test, simulating it, inspecting what happened, and adjusting it until it failed for the right reason.

Finding the bug took reading, and current models read code well. Reproducing it took repeated experiments: build, simulate, look at the result, change the test. Hardware and software differ most in that loop, so the rest of this note measures it.

Method¶

I ran ten trials in each of two setups, with the same agent, model, prompt, and container:

• Claude Code alone: a shell plus Caliptra's existing Makefiles and test scripts. This is the default when you point an agent at an RTL repository.
• Claude Code plus Booley: the same agent with a harness designed for RTL work, configured for Caliptra before the trials. The timings cover debugging only, not the one-time work of adapting Caliptra to Booley.

The Booley setup's AGENTS.md (also linked as CLAUDE.md) explains how to use Booley's tools and has two lines describing the repository; it contains no build commands, environment variables, or debugging hints.

The harness changed three things, each aimed at one of the problems described below:

• Waveform queries. Both setups could record a waveformA recording of how internal signals change, cycle by cycle, during a simulation.; only the Booley setup had a tool, B-Wave, to search it and read signal values at particular cycles.
• One blocking simulation command. Booley's sim Flow builds and runs a simulation, then returns its result when the job ends. It works the same way in every project, so the agent doesn't have to learn each project's shell commands.
• Build configuration in files. FuseSoC, the open-source build tool Booley uses, described the source files, simulator settings, dependency paths, and firmware build steps in core filesFuseSoC description files, one per component, listing its source files, dependencies, and build settings. and a test list.

Where the time went¶

Both setups produced an accepted fix and a reproducing test in all ten trials, so the comparison is about time. Mean completion time was 55.8 minutes with Claude Code alone and 21.3 minutes with Booley (Figure 1); the medians were 53.3 and 21.4 minutes. Average cost per trial at Sonnet 5.5 API prices was $2.33 and $1.98.

In the plain-shell runs, only about a third of the time went into building and simulating. The rest went to observing internal signals, sleeping on jobs that had already finished, and recovering from broken builds. Each recurred across the trials. The first two follow from how hardware is simulated; the third is a general agent problem that showed up here too. I identified them from the run histories; I didn't run separate experiments to isolate the contribution of each fix.

Problem 1: internal state is invisible by default¶

When a software test fails, it usually prints what went wrong, and a debugger can stop the program anywhere. In RTL simulation, the state that explains a failure is spread across thousands of internal signals, and none of them are visible unless the testbench records a waveform or prints them. Adding a print means editing the design or testbench and recompiling. Caliptra simulates with Verilator, which compiles the design into a C++ program, so each recompile and rerun takes several minutes.

This matters most when a test passes despite a suspected bug, which happened repeatedly here, and the agent needs to see why the bug didn't fire.

What the agent did with only a shell. It debugged the simulator the way it would debug a software program: with prints. In pair b3 (Figure 2), it added $displayVerilog’s print statement, similar to printf in C. statements, rebuilt the simulator, reran the test, and searched the output. Every new question meant another recompile. It repeated that loop seven times and spent 37.8 minutes on signal observation.

Recording a waveform was an option in this setup too, but a recording only helps if the agent can read it. Caliptra's waveforms are in FST, a compressed binary format that takes a dedicated library to parse, and a quick Python script is too slow on a dump this size.

What worked: record once, then query. With Booley, the agent built once with waveform recording enabled and asked B-Wave 16 questions about the recorded signals. That work took 9.8 minutes.

In trial b1, the agent searched the waveform to find whether the faulty address comparison ever fired. Listing 1 shows equivalent command lines, with output trimmed.

$ bwave find trace.fst '*bus_intf.ld_addr_rhit_lo_lo' 1
cycle 4687 caliptra_top_tb.caliptra_top_dut.rvtop.veer.lsu.bus_intf.ld_addr_rhit_lo_lo 1
cycle 4696 caliptra_top_tb.caliptra_top_dut.rvtop.veer.lsu.bus_intf.ld_addr_rhit_lo_lo 1
...
# 32 matches found

The comparison fired 32 times, yet the test still passed. The agent then asked for the surrounding signal values at the first match (Listing 2). Signal names end in their pipeline stage: _m for the load, _r for the store one stage ahead of it.

$ bwave value trace.fst --at 4687 -s '*bus_intf.ld_*' -s '*bus_intf.lsu_busreq*' -s '*bus_intf.is_sideeffects_m' ...
# Snapshot at cycle 4687
lsu_busreq_m                             = 1
is_sideeffects_m                         = 1
lsu_busreq_r                             = 0
ld_addr_rhit_lo_lo                       = 1
ld_full_hit_m                            = 0

The read matched a write that should not have been eligible for forwarding: a DCCM store (lsu_busreq_r = 0) whose address had been blanked to 0. But the test's memory setting (is_sideeffects_m = 1) blocked forwarding (ld_full_hit_m = 0) and hid the bug.

The agent changed that setting in the test. Half a minute later, the test failed on the buggy design.

The agent could ask follow-up questions of the same recording without rebuilding. Across all ten pairs, it spent 9.4 minutes less on average observing signals with Booley.

The b3 chart's final segment (Figure 2), roughly 47 to 85 minutes, is a sequence of final tests. That time isn't counted as signal observation.

In hardware debugging, a new question about the design should be answerable from an existing recording. A tool that searches a recorded waveform by signal name and reads values at a given cycle gives an agent roughly what a debugger gives it in software.

Problem 2: simulations are long, and the agent guesses how long¶

Agents' habits come from commands that finish in seconds. In hardware, minutes are normal: compiling a design like Caliptra for simulation and running a firmware test takes several, and synthesis or a full regression can take hours. So the agent runs the job in the background and has to decide when to check on it. If it guesses, it either checks too early or sleeps long after the job has finished.

What the agent did with only a shell. In several runs, it started work in the background and used a fixed sleep before checking the result. In b10 (Figure 3), a build started at 4.3 minutes and failed one second later. The failure was hidden: the command ended with ; tail, so it reported exit code 0. Seconds later the agent requested sleep 540, a nine-minute wait, and didn't check the result until 13.4 minutes.

It made similar waits four times in that run, spending 30.7 of its 60 minutes asleep after jobs had finished.

What worked: return when the job ends. Booley's simulation command waits for the job and returns the result. For long jobs it returns a run ID instead, and the agent calls a poll tool (booley_poll) that returns as soon as the job ends, so there is no fixed sleep to guess.

The Booley run in pair b10 took 13 minutes in total. Across all ten pairs, it spent 11.2 minutes less on average waiting on finished jobs. These waits affected six pairs. I counted only waits where I could establish that the job had finished, so the estimate is a lower bound. The sleeping is an agent habit. Hardware makes it expensive because the jobs are long, so the guessed waits are long too. There is a cheap partial fix: one line in CLAUDE.md telling the agent to wait for the process instead of sleeping would likely recover much of this time. Hiding exit codes behind pipes, as in ; tail, makes it worse, because a failed build looks like a running one.

Not hardware-specific: the build is fragile, and the shell forgets¶

This problem isn't specific to hardware. Any project whose build depends on undocumented environment variables trips agents up the same way, and RTL projects often have such builds. Caliptra's simulation needs the simulator, toolchain paths, and a firmware image compiled by a separate cross-compiler and loaded into the simulated memory, and its Makefiles locate these through environment variables. An engineer sets them once in a terminal and forgets about them. An agent works through a series of separate shell calls, and state set in one call doesn't necessarily survive to the next. When a step fails, the next one often runs anyway: a simulation without its firmware doesn't crash; it runs until it times out.

What the agent did with only a shell. All ten plain-shell runs hit missing build configuration; most recovered within a few minutes. In b1 (Figure 4), the agent worked out which variables to set and exported them in a shell call. Those values didn't carry over to later calls in this run. The firmware build then failed to produce program.hex, and the simulation started without a program. It waited until its timeout. This happened three times before the agent put the configuration in a single run.sh script.

The run spent 30 minutes dealing with setup. Sleeps after jobs had already finished, including two minutes after the first hung simulation was killed, are counted separately (Figure 4), as in Figure 1.

What worked: configuration in files, prerequisites checked. The Booley project's FuseSoC core files already held the dependency paths, so the agent didn't have to rediscover the Makefile variables. Each simulation invocation supplied its build configuration and firmware build step, without relying on exports from a previous shell call. If the firmware build failed, Booley stopped the run and reported the build error before starting a simulation without its program.

The Booley run in pair b1 spent no time repairing setup. Across all ten pairs, the average setup saving was 3.6 minutes; b1 was the unusually costly case. Configuring the project once avoided repeating that work during debugging.

Build configuration should live in files that every run reads, and each simulation run should carry its whole configuration and stop at the first failed prerequisite. A checked-in script that the agent's instructions point to gets part of the way there. The plain-shell agent in b1 arrived at exactly that, a run.sh, after 30 minutes.

What the tools didn't change¶

Both setups still had to work out how to make the bug appear: noticing that the test template's side-effect setting disabled forwarding, and moving the test loop into ICCM to get the store and the load back to back in the pipeline. That is reasoning about microarchitecture, and the harness didn't do it for the agent. It changed the cost of each experiment the agent ran on the way there, and that was enough to cut the total time by more than half.

Lessons for agent setups on RTL¶

This is a case study, not a benchmark: one bug, one design, one model, and ten trials per setup. I chose a bug that needed substantial simulation work and used an already configured Booley project. These results don't establish the same speedup for other models or other debugging tasks, or tell us which fix would help most on its own. They also say little about design or verification work, which have their own problems and will get their own notes.

Still, the plain-shell runs show where a general-purpose agent loses time on an RTL project, and none of it was spent finding the bug. The lessons below don't depend on a particular tool. The first two are about hardware. The third applies to any agent setup, but RTL projects need it often:

1. Make internal state queryable. Record waveforms on demand and give the agent a way to search them, so a new question about the design doesn't need a rebuild. This was the largest hardware-specific gap.
2. Return results when jobs finish. Use a blocking call or a poll that returns on completion, and don't let a pipeline hide a failed exit code.
3. Keep build configuration in files. Each simulation should be self-contained and should fail fast when a prerequisite, such as the firmware image, is missing.
4. Plan for reproduction. On this bug, the model found the faulty line quickly; the time went into building a test that made the bug visible. Tooling and time budgets should target that loop.

Booley implements all three tooling fixes, but a project can apply them with any waveform library, build system, and job runner.

Reproducing this study¶

The Caliptra project used here is public; its README explains the setup. The reproduction details below list the recorded versions, the original prompt, and the patches. To try the same workflow on a smaller design, the PicoRV32 demo comes preconfigured.

$ git clone https://github.com/chipsalliance/caliptra-rtl.git caliptra-source
$ cd caliptra-source
$ git checkout --detach e379800c5467379386f96a9a51a539bd662b608a
$ git submodule update --init --recursive
$ git -C submodules/adams-bridge checkout --detach 2f15ffb1a2e9bd99652fa04d434a48ffcdc23f42

$ patch --dry-run --fuzz=0 -p1 < ../trace-harness.patch
$ patch --fuzz=0 -p1 < ../trace-harness.patch
$ patch --dry-run --fuzz=0 -p1 < ../seed.patch
$ patch --fuzz=0 -p1 < ../seed.patch

--- a/src/riscv_core/veer_el2/rtl/lsu/el2_lsu_bus_intf.sv
+++ b/src/riscv_core/veer_el2/rtl/lsu/el2_lsu_bus_intf.sv
@@ -241,4 +241,4 @@
-   assign ld_addr_rhit_lo_lo = (lsu_addr_m[31:2] == lsu_addr_r[31:2]) & lsu_pkt_r.valid & lsu_pkt_r.store & lsu_busreq_m & lsu_busreq_r;
-   assign ld_addr_rhit_lo_hi = (end_addr_m[31:2] == lsu_addr_r[31:2]) & lsu_pkt_r.valid & lsu_pkt_r.store & lsu_busreq_m & lsu_busreq_r;
-   assign ld_addr_rhit_hi_lo = (lsu_addr_m[31:2] == end_addr_r[31:2]) & lsu_pkt_r.valid & lsu_pkt_r.store & lsu_busreq_m & lsu_busreq_r;
-   assign ld_addr_rhit_hi_hi = (end_addr_m[31:2] == end_addr_r[31:2]) & lsu_pkt_r.valid & lsu_pkt_r.store & lsu_busreq_m & lsu_busreq_r;
+   assign ld_addr_rhit_lo_lo = (lsu_addr_m[31:2] == lsu_addr_r[31:2]) & lsu_pkt_r.valid & lsu_pkt_r.store & lsu_busreq_m;
+   assign ld_addr_rhit_lo_hi = (end_addr_m[31:2] == lsu_addr_r[31:2]) & lsu_pkt_r.valid & lsu_pkt_r.store & lsu_busreq_m;
+   assign ld_addr_rhit_hi_lo = (lsu_addr_m[31:2] == end_addr_r[31:2]) & lsu_pkt_r.valid & lsu_pkt_r.store & lsu_busreq_m;
+   assign ld_addr_rhit_hi_hi = (end_addr_m[31:2] == end_addr_r[31:2]) & lsu_pkt_r.valid & lsu_pkt_r.store & lsu_busreq_m;

247a8a96d300af8c5b243a9fffaef2ce2762df317d477f547c94fc98d0aea0b0

Bug report against this Caliptra RTL snapshot.

Runtime ROM self-integrity digest is wrong, but only when the digest loop uses a DCCM scratch buffer

Early in the boot flow the runtime firmware recomputes a digest over its own
ROM image (a self-integrity check) and compares it against a constant that was
fixed at build time. The check normally passes.

We hit a case where the computed digest is wrong even though the ROM image on
the part is byte-for-byte unchanged and the compare constant is correct. No
error, fault, or status bit is set anywhere; the core reports no exception and
the memories report no ECC or parity events. The wrong digest is fully
deterministic across resets.

While bisecting the firmware we found the trigger is a coding pattern, not the
data: the digest routine that fails reads the ROM image word by word and, in
the same tight loop, writes each partially-accumulated result to a small
scratch buffer that the linker placed in DCCM. A functionally identical
version of the routine that keeps the accumulator only in registers (no scratch
writes in the loop), or that moves the scratch buffer out of DCCM, produces the
correct digest every time. The failing and passing routines compute the same
math and touch the same ROM bytes; only the placement/interleaving of the
scratch writes differs.

The corruption is confined to the very start of the digested region: the
recomputed digest diverges from the expected one as if one word of the ROM
input had been replaced, and everything after that point is consistent with the
true ROM contents.

No existing test reproduces the problem.

Find and correct the RTL root cause. Add a test that reproduces the problem
and passes after your fix, then run the relevant verification checks.

55fc823860c02e841b3c7b1ee46d81a014220b8244304dd5d7d04148ed731ea7