Mutation testing
Find the changes to your code that no test detects with fallow mutation, and decide which test to add.
Your test suite passes, and the coverage report says that the tests run each line. But a test that runs a line does not always check its result. Somebody changes >= to > in that line, and every test still passes. Line coverage cannot show this gap. fallow mutation makes small changes to your code, runs your tests against each change, and reports the changes that no test detects.
npx fallow mutation
Mutation testing is opt-in. It runs your tests many times, so it is slower than the other analyses. Bare fallow, fallow audit and fallow health never run it.
What fallow finds
This example adds a file src/lib/pricing.ts with two functions to the demo project, and a test file src/lib/pricing.test.ts with one test for each function. It also updates Vitest to version 4, because fallow does not support the Vitest 2 of the demo. Run fallow on the new file:
// src/lib/pricing.ts
export const applyDiscount = (total: number, percent: number): number => {
if (percent <= 0) return total;
return Math.round(total * (1 - percent / 100));
};
export const shippingCost = (total: number): number => (total >= 5000 ? 0 : 495);
npx fallow mutation src/lib/pricing.ts
Mutation testing (ran the project tests)
Scope: the given paths
Score 69.2% killed 9 survived 4 timeout 0 complete
Coverage 92.9% no coverage 1
● Survived (4)
src/lib/pricing.ts
#2 2:7 applyDiscount negate-condition if (percent <= 0) return total; -> if (false) return total;
id f73493d4d103bf6f
covered by 1 test: src/lib/pricing.test.ts::discount
#3 2:15 applyDiscount relational-operator if (percent <= 0) return total; -> if (percent == 0) return total;
id 89b6132f47593839
covered by 1 test: src/lib/pricing.test.ts::discount
#4 2:15 applyDiscount relational-operator if (percent <= 0) return total; -> if (percent < 0) return total;
id cc6df8e855ee87fa
covered by 1 test: src/lib/pricing.test.ts::discount
#14 6:63 shippingCost relational-operator ... (total: number): number => (total >= 5000 ? 0 : 495); -> ... (total: number): number => (total > 5000 ? 0 : 495);
id c479a45156663278
covered by 1 test: src/lib/pricing.test.ts::shipping
● No coverage (1): no test reaches these mutants
src/lib/pricing.ts 1 mutant test reach: 1 direct
Next
Inspect a survivor (runs no code): fallow mutation inspect f73493d4d103bf6f
Each -> shows one small change to the code, for example >= to >. Fallow calls such a change a mutant. When a test fails, the test detected the change, and the mutant is killed. When each test passes, the mutant survived. Each survivor is a finding of the issue type mutation-survived.
| Part | Meaning |
|---|---|
Score 69.2% | The mutation score: the share of the tested mutants that a test detects. |
killed 9 survived 4 | The status counts. A timeout also counts as detected. |
complete | All planned mutants ran. A partial run lists its reason, for example a time budget. |
Coverage 92.9% | The share of the mutants that a test reaches. |
#3 2:15 applyDiscount relational-operator | One survivor: its rank, line and column, function, and kind of change. |
-> | The original line and the changed line. |
covered by 1 test | The tests that ran the changed code and still passed. |
The survivors show boundaries that no test checks. No test calls applyDiscount with a negative percent, and no test calls shippingCost with exactly 5000.
How fallow decides
A run on the demo took about one second, because it had 14 mutants and one test file. By default, fallow mutates only the code that your branch changes against the base branch, as fallow audit does. Add --all for each source file, or pass paths as above.
Fallow then runs these steps:
- It selects the mutants and ranks them.
fallow mutation --planshows this list and runs no code. - It runs your tests once without a mutant. This run finds the tests that reach each mutant.
- For each mutant, it runs only the tests that reach it. A survivor and a timeout run a second time, so that a slow host does not decide the result.
Fallow detects the test runner of your project: Vitest, node:test or bun test. For Vitest, fallow checks the installed version against a supported range, and a run stops when the version is outside it. fallow mutation --plan and fallow doctor report that version before a run.
The score counts only the mutants that ran: killed and timeout, divided by killed, timeout and survived. A mutant that no test reaches gets the status no_coverage. It does not lower the score, and the coverage value reports it.
Safety
A run executes your project tests, so fallow applies these rules:
- Fallow does not write to your project files. The changed copies of your files are in a run directory in the fallow cache.
- A run refuses a git worktree with uncommitted changes, unless you pass
--allow-dirty. After the run, fallow reports each file that your tests changed. - Fallow skips the mutants in a function that writes to files, starts a process, or writes to a network or a database. Such a mutant can remove a guard of a real side effect.
--allow-side-effectskeeps them. No config key can do this. - Fallow stops each test process when the run ends, also on Ctrl+C.
Do not write the report with > report.json. The shell creates the file first, and the worktree check then refuses the run. Use -o .fallow/mutation.json.
Cost and scope
On a real project, a run executes your tests many times. These options keep a run short:
- The default scope is the changed code. Run
fallow mutation --planto see the number of mutants before a run. --max-mutants <N>runs at most N mutants, in rank order.--time-budget 10mstops new mutants after 10 minutes.--concurrency <N>sets the number of test processes. The default is half of the logical cores.- A result cache keeps the result of each mutant. A second run on the same code and tests reuses it.
The GitHub Action and the GitLab template run mutation testing only when you turn it on. See CI integrations.
Act on a finding
Look at one survivor. This command runs no code:
npx fallow mutation inspect cc6df8e855ee87fa
Mutant cc6df8e855ee87fa fresh
src/lib/pricing.ts:2:15 applyDiscount relational-operator (report: survived)
- 2 | if (percent <= 0) return total;
+ 2 | if (percent < 0) return total;
● Covering tests (1)
src/lib/pricing.test.ts::discount
This mutant is different from the other three. For a percent of 0, the formula on line 3 also returns the total, so no input tells the two versions apart. Decide for each survivor:
| Situation | Action |
|---|---|
| The changed code gives a wrong result, and no test checks it. | Add an assertion for the input that tells the two versions apart. Here: applyDiscount(1000, -10) for #2 and #3, and shippingCost(5000) for #14. |
The changed code gives the same result for each input, as #4 does. | The mutant is equivalent. Record it in a verdict file, and pass the file with --verdicts. |
No test reaches the code (no_coverage). | Write a first test for the function. |
After you add tests, run fallow mutation --recheck. It runs only the open mutants of the last report (survived and no_coverage) against your new tests. With the two assertions above, the recheck ends with the equivalent mutant only:
Recheck of the earlier report
4 killed
1 still survived
The fallow-mutation agent skill lets a coding agent do these steps.
Fail a run
By default, survivors do not fail a run: it exits with code 0. A minimum score turns the score into a gate:
npx fallow mutation --min-score 80
The first run of the demo, with a score of 69.2 percent, then exits with code 1. Set the same value as mutation.minScore in your config file. A run with no tested mutant has no score and exits with code 0. The default scope already limits a run to the changed code; Changed code and baselines explains how fallow finds that code.
To show the mutation score next to the CRAP score of each complex function, pass a saved report to fallow health --mutation-report.
What fallow cannot see
- The score describes only the mutants that ran. A score of 100 percent on 20 mutants says nothing about the rest of the code.
- A survivor is not always a missing test. Some mutants are equivalent: no input tells them apart from the original code.
- By default, fallow does not type-check a mutant. A mutant that adds a type error can survive. Add
--typecheckwith the type-aware analysis to remove these mutants from the score. - Shared state between test processes, such as one database, can give wrong results. Run with
--concurrency 1to check. --workspaceis not supported. Pass the directory of the package as a path.
Reference
fallow mutation: each option, the statuses, the JSON output and the exit codes.mutationconfig: each key and its default.- CRAP scores with Vitest: measured coverage for the health report.