Mutation testing (mutation)¶

mutation checks that your tests catch bugs. It plants small bugs (Mutants) in the RTL, runs the Target's tests against each one, and reports how many Mutants the tests detected. A passing simulation only shows the tests accept the design; a high mutation score shows they would notice if it were wrong. No LLM is involved: the same RTL, --count, and --seed always give the same Mutants, so scores are reproducible and comparable across runs. For what all Flows share (choosing Targets, common options, logs, exit codes), see Running Flows.

Mutation drives a Simulation Target and runs the whole suite of that Target's Tests, so the Tests must already pass.

Running mutation testing¶

1. Check that the Project has a simulation Target (booley targets --for sim lists them). There is no --for mutation: mutation runs on any simulation Target.

2. Ask the Developer Agent:

   "Check whether the FIFO tests catch injected bugs."

   "Run mutation testing on sim_fifo with 20 Mutants."

3. The agent reports how many Mutants were detected, which survived, and the path to the report.

To reproduce a run yourself inside the Sandbox:

booley flow mutation --target sim_fifo --count 20 --seed 1

Mutation takes exactly one Target; --target is required and a list is refused.

The shared options (-C/--project, --timeout, --report-dir, --diagnostic, -q/--quiet) work as for every Flow.

How a run works¶

1. Booley selects the Mutants for the seed from the scope files.
2. It builds the unmodified RTL and runs every test of the Target. All must pass; otherwise the run ends with ERROR, because a failing test could not tell a Mutant apart from the original.
3. Each Mutant is applied to a private copy of the sources, built, and the Target's tests run on it. Your RTL is never modified.
4. A Mutant is detected when a test fails on it, or when a test runs past its time limit (reported as detected (timeout)): ten times the test's baseline duration, at least 30 seconds and at most 300 seconds. A Mutant that no test fails on survives. A Mutant that does not compile is invalid: it is reported, not scored, and can never be detected, so it uses up part of the threshold's margin.

Every run rebuilds the baseline and every Mutant, so a result never depends on stale builds. Some survivors may be equivalent Mutants that change no observable behavior, which is why the default threshold is 80 %, not all.

Which Mutants it plants¶

Booley parses each scope file (Verilog or SystemVerilog) and finds every spot where one of six operators applies. Each operator makes small changes of the kinds below, one change per Mutant; a spot with several possible changes yields one candidate Mutant per change.

/, %, and ** are never swapped, so Mutants do not divide by zero.

Mutations also reach the arguments of $signed and $unsigned and the expressions in the port connections of module and interface instances.

Table details: a !x that is a whole if or ?: condition is left to the condition operator. Dropping an operand puts the operator's identity there (1'b1 for &&, all ones for &), and an operand that calls a function or changes a variable is never dropped. A reduction swap trades & and \|, turns ^ into \|, and removes the ~ of ~&, ~\|, and ~^. A sign cast is never turned the other way. An enum member wraps around its enum, which must be declared in the same file; a two-member enum yields one Mutant. A concatenation swap skips equal neighbours, replication counts, streaming concatenations, and assignment patterns.

Only design code is mutated. Booley skips:

• testbench, vendored, and generated files (by default);
• classes, programs, checkers, covergroups, and every assertion, because a mutated assertion fires and would count as a detection;
• parameter defaults and overrides, widths, delays, attributes, and the parts of a for loop that decide how long it runs;
• the arguments of system calls other than $signed and $unsigned;
• port connections of gate primitives, bind directives, and checker and program instances, connections that are a bare name, and every bit-select move and deletion in a connection;
• code a macro or include supplies, and the inactive branch of an ifdef (Booley applies the build's defines);
• spots that could make a loop run forever or drive a variable from two places, and case labels (moving one can overlap another item).

How it chooses which Mutants¶

1. Booley collects the candidates from all scope files. Identical spots count once.
2. Within each operator, candidates are ranked by a hash of --seed and the Mutant's ID. The ID comes from the operator, file, and position inside its module, so editing code elsewhere in the file does not change it.
3. The six operators are then taken in turn, one candidate each, in the order relational, arithmetic-logical, condition, constant, bit-select, deletion, until --count Mutants are chosen. A small --count therefore covers different kinds of bugs instead of many of one kind.

If fewer candidates exist than --count, all of them run and the report says so. The knobs are --scope (where to look), --count (how many), and --seed (which sample); there is no filter by operator. The same RTL, scope, count, and seed always choose the same Mutants.

Reading results¶

The report starts with what reproduces the run, then the tally and the verdict:

[mutation_test] 10 Mutants (seed 0, count 10, generator 3, pyslang 9.0.0)
[mutation_test] sim sweep: 10/10 tested              4m
  8 detected (1 timeout), 0 invalid, 2 not detected
  detected: 8/10 (required: 8/10, default 80% of --count)
  NOT DETECTED [arithmetic-logical] rtl/alu.sv:42: 'a + b' -> 'a - b'
…
RESULT: PASS (8/10 detected)

Survivors are listed with file, line, and the original and mutated code: each is a spot your tests do not check.

Each run's files are in its numbered run directory, under flow-reports/ in the Booley Directory or --report-dir (Reports):

Configuration ([flows.mutation])¶

[flows.mutation]
enabled = true         # false turns the Flow off
timeout_ms = 1800000   # default: 30 minutes for the whole run

timeout_ms (or --timeout) sizes --count auto: the budget is divided to cap the number of Mutants, so a larger limit allows more of them. Over MCP it also raises the wait before the call hands back a run id. The command line does not stop a run when the budget passes. See Flow and Specialist availability.

Goals¶

A MUTATION:{target} Goal is met by this Flow's result: the Flow records the mutation_score criterion. Your own runs never count as Goal evidence. A Goal sets the campaign with total: N, min_detected: K, or auto: true in place of --count and --min-detected; without them it requires 8 of 10 Mutants.

Source: docs/user/FLOW-MUTATION.md on GitHub