What the numbers mean
We compare one focused Radforge run with one general-purpose mutation baseline on the same frozen repository revision, code change, and test command. The published numbers are wall-clock time, not summed worker time. Cold Radforge runs bypass result caches.
Comparison protocol
- Freeze the repository revision or commit range, code change, test command, runtime, tool versions, worker limits, and timeout policy before timing either tool.
- Where a scenario uses seeded bugs, add them before either arm runs and verify that the native test command remains green.
- Run Radforge and the baseline against the same code and test slice. Do not narrow the baseline test command after seeing results.
- Record elapsed time, completed mutation outcomes, failures, timeouts, cache state, and survivor-line recall.
- Mark unstable, mismatched, retuned, or incomplete comparisons as discarded. They are not used for external performance claims.
Correctness comes with speed
A survivor line is a source line where at least one mutation completed and the selected tests still passed. Survivor-line recall is the share of the baseline's survivor lines that Radforge also found in the frozen scenario. We report speed only with the corresponding recall or seeded-bug result, because a faster run that misses the relevant test gaps is not a useful win.
Results cited on this site
| Scenario | Radforge | Baseline | Outcome |
|---|---|---|---|
| Realistic pull-request change | Under 40 seconds | Over 6 minutes | 9.2× faster; both seeded bugs detected |
| Real-world build-tool change | 42.8 seconds | 77.5 seconds | 45% less time; every baseline test gap found |
| Larger application change | 148.5 seconds | 275.4 seconds | 46% less time; all 149 baseline test gaps found |
These are scenario-specific open-source replays against Stryker on the same code and test slices. Displayed timings are rounded to one decimal place where applicable.
Caveats
- These results support claims about the named frozen scenarios. They do not imply that Radforge is always faster on every repository, language, test runner, or change.
- Wall-clock results vary with hardware, available CPU, dependency state, test behavior, and tool versions.
- Survivor-line recall measures line-level test-gap discovery. It does not mean every possible fault on a recalled line was tested.
- A cache-assisted result must be labeled separately from a cold run. The homepage figures are cold comparisons.
Questions about a result?
We can walk design partners through the frozen scenario, comparison setup, and supporting run artifacts during evaluation.
Talk with Radforge