Mutation testing cheap enough to run while you are still writing the code.
It breaks your code on purpose, one operator at a time, and reports every mutation your tests let through. Install the command and `ditto staged` reads the git index, works out for itself which lines moved, and charges you only for those. Every claim it makes about speed ships with the counter that would disprove it.
┃ Releasing Ditto…┃ calc/calc.go — 7 mutants┃ calc/calc.go:9:12 → Arithmetic┃ baseline: the suite took <duration> on unmutated code, and every mutant runs it again.┃ calc/calc.go:12:11 → Arithmetic┃ calc/calc.go:16:41 → Arithmetic┃ calc/calc.go:4:41 → Comparison┃ calc/calc.go:8:8 → Comparison┃ calc/calc.go:4:40 → Comparison Invert┃ calc/calc.go:8:7 → Comparison Invert┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╍┅┃ 🧬 Survivors┠┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┃ calc/calc.go:12:11 → Arithmetic (- → +)┃ calc/calc.go:16:41 → Arithmetic (+ → -)┃ calc/calc.go:8:8 → Comparison (inserts =)┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╍┅┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╍┅┃ 🧬 Mutant survived: calc/calc.go:12:11 → Arithmetic┠┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┃ --- calc/calc.go (original)┃ +++ calc/calc.go (mutated with 'Arithmetic')┃ @@ -9,7 +9,7 @@┃ return a - b┃ }┃ ┃ - return b - a┃ + return b + a┃ }┃ ┃ // Uncovered is never called, so every mutant of it lives.┃ ┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╍┅ … two further survivor diffs elided … ┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓┃ • Total: 7 ┃┃ • Killed: 4 ┃┃ • Survived: 3 ┃┠┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┨┃ ✓ Score: 0.57 (minimum: 0.00) ┃┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛Starting the test command costs 750 to 950 milliseconds per mutant, whatever the suite does, and that toll dominates a run. Compile once and choose the mutant at run time and it is paid once for the whole run: the same twelve mutants come back in 0.4 to 1.3 seconds against 10 to 11. Both mechanisms below cut how often the command runs at all.
WithChangedRangesPay for the lines the change actually touched
Hand it the byte ranges a change covers, keyed by file, and the release mutates only those. Or run `ditto staged`, which reads them off the git index for you. On the reference fixture one changed function charges the four operators that fire on that line and nothing else in the repository.
It scales with your change: a changed function in each of two files charges eight runs, exactly twice what one file charges, because each file is only ever charged for its own ranges.
GatedOne compilation carries every mutant in the file
The file is instrumented so every mutant becomes a gate chosen at run time, the package is compiled once with `go test -c`, and each mutant is selected by environment variable. A file the instrumentation cannot carry goes back to the ordinary path on its own, and the run continues over the rest of the repository.
Where the two paths differ, the gated one is right: it scores a mutant the ordinary build cannot compile.
Gated| Verdict | Ordinary | Gated |
|---|---|---|
| Killed | 127 | 127 |
| Survivors | 8 | 8 |
What the gain is worth depends on your suite: the same mechanism produced 2.7× on a package whose suite takes 243 ms and 1.5× on one that takes 1155 ms, because the toll it removes is a fixed cost.
ditto reads a mutant’s fate from a test command that fails. If the suite is already red, every mutant looks killed and the run reports a perfect score. Everything below is what stands between you and that number.
One baseline run of your command on unmutated code opens every release, and a red one ends it there, carrying the command’s own output so you can read why without going into the sandbox. That run also announces what the suite cost, which is the price every mutant pays again.
It runs the instrumented file with no mutant selected, which is the file’s own suite: a measurement already bought, and now read. On the same mutants it separates a reported 4 killed of 4 from the true 1 of 4, and that reading is free.
`internal/verdict` reads it off the `go test -json` event stream the run already produces, and the four reasons it can return are the ones drawn at the top of this section.
A mutant that never became a program leaves the score entirely, numerator and denominator both, so what you read is the share of runnable mutants your tests caught. On a fixture where two of five mutants never compiled, that is 0.33.
When gating is on and one file will not carry the schema, that file goes back to the ordinary path by itself and the run keeps going over everything else. On ditto itself exactly one file needed that, and the rest kept their single compilation.
`path:line:col → Operator (what it replaced)`, listed before any diff, so a survivor can be jumped to. It is also what tells two survivors apart: over 135 mutants of four gofmt’d files, 129 are indistinguishable from another mutant on everything except their address and the text they replaced.
A golden test compares a complete release against a fixture that holds both kills and survivors, on the ordinary path and the gated one, so the two are held to printing the same thing.
Every number below is an exact integer: a count of parses, sandboxes, files and test-command invocations, identical on every machine. Each one is pinned, and the build fails if it moves in either direction, so what a release costs today is what it will cost you next week.
| Counter | Was | Is |
|---|---|---|
sourceParsesPerReleaseWithThreeVirusesOne parse per source file, shared across every operator: a file is read once however many of the fourteen defaults are on. | 12 | 4 |
sandboxesBuiltPerReleaseSandboxes are pooled and the mutated file is restored before one is handed back, so a sequential run builds exactly one and a parallel run builds as many as its peak concurrency. | 48 | 1 |
filesLinkedPerSandboxSix working files: a sandbox holds the tree without `.git`. The count is the same whichever way a file reaches it (copied, hard-linked or linked), at roughly 0.45 ms per file, paid once per release. | 11 | 6 |
laboratoryRunsForOneChangedFunctionFour runs for one changed function: the operators that fire on that line and nothing else in the repository. | 48 | 4 |
laboratoryRunsForOneChangedFunctionInEachOfTwoFilesA changed function in each of two files costs exactly twice one, because each file is charged only for its own ranges. | 48 | 8 |
mutantsPerReleaseOnThisRepositoryWhat one full run over ditto’s own repository pays for: 785 mutants. It is measured against the repository itself, so it is the figure that says what a run of that size costs. | 431 | 785 |
testCommandInvocationsPerReleaseWholeFixtureForty-eight mutants, plus the one baseline run of your suite on unmutated code that makes the score mean something. One baseline for the whole release. | 48 | 49 |
Wall clock is measured and reported: on a real development machine the identical workload has varied here by more than fifty percent between runs, while the counts above stayed exact.
Each one edits the syntax tree the way a real mistake would: an operator flipped, a constant nudged, a loop cut short. Fourteen are on before you configure anything, from the command or from `Release`; the fifteenth is added by naming it.
+↔-*↔/%↔*+=-=*=/=%=&=|=^=<<=>>=&^=→=+=↔-=*=↔/=%=↔*=&↔|^→&&^→&<<↔>><↔<=>↔>=>↔<=<↔>===↔!=&&operand→true||operand→falsex→x-1.0x→x+1.0n→n-1n→n+1break↔continuecondition→falserange→earlybreakcontext.CancelCauseFunc(err)→(nil)A virus is any struct satisfying the `viruses.Virus` interface, so a mutation that only means something in your codebase is a type and a call to `WithViruses`. The `dittotesting` package carries the helpers the shipped fifteen are tested with, so yours gets the same treatment.
The command is the shortest way in: install it once and `ditto staged` mutates only what your staged change justifies, with `--dry` to price it before you pay for it. The library is the same engine from inside a test binary you already have.
Each mutant is judged by the test command below, and `-json` is what lets ditto say why it died.
$ go install github.com/Disble/ditto/cmd/ditto@latest$ ditto staged --threshold 0.8$ go test -count=1 -json ./...One import and a build tag put the same engine inside the suite you already run, with `WithTestCommand` to name a different test command.
$ go get github.com/Disble/ditto$ go test -v -tags=mutation$ go test -count=1 -json ./...ditto — mutation testing for Go ditto run [flags] mutate a repository and report what survived ditto staged [flags] mutate only what a staged change justifies ditto changed [flags] mutate only what a committed change justifies ditto version the module version this binary was built from Run `ditto run -h` for its flags.//go:build mutation package main_test import ( "testing" "github.com/Disble/ditto") func TestMutation(t *testing.T) { ditto.Release(t)}The source, the dated learning log, and the pinned performance baseline.
Every option, every operator, the `viruses.Virus` interface and the `RunStaged` and `RunChanged` entry points, generated from the source.
The commit gate that hands its own staged Go changes straight to `ditto staged`.
ditto is a fork of gtramontina/ooze by Guilherme J. Tramontina. All the good ideas here are his, and the licence and copyright stay with him; it is MIT, as the original is.