Weakened tests
What a weakened-tests finding of fallow audit means, why it matters and how to fix it.
What it reports
The change weakens tests or a coverage threshold.
Changed code explains how fallow finds it.
Why it matters
fallow audit compares the base and head text of each changed file. The kind field tells how the change weakened the tests: skip-only (a new .skip, .only, xit or fit marker), removed-tests (fewer test cases in a file, while the total of all changed test files also dropped), assertion-drop (fewer expect, assert or should calls in a test file with no test case removed), coverage-threshold (a coverage or quality threshold that is less strict), rule-severity (a rule severity in the Fallow config that is less strict, such as error to off, a removed rule whose default is weaker, or a new overrides entry that turns a rule off for some paths), or ignore-pattern (a new or broader entry in a config key that removes files or findings from the analysis or makes code count as used, or a removed ! exception of ignorePatterns or ignoreFindings). The keys are ignorePatterns, ignoreFindings, production.ignore, health.ignore, duplicates.ignore, duplicates.ignoredClones, entry, supportEntry, dynamicallyLoaded, publicPackages, usedClassMembers, ignoreDependencies, ignoreBinaries, ignoreUnresolvedImports, ignoreExports, ignoreCatalogReferences, ignoreDependencyOverrides and boundaries.coverage.allowUnmatched. The evidence names the key. A new entry that a base entry or a built-in default already covers does not report. The config check follows each relative extends file, so a change in an extended file or in the extends list reports too. A value that an npm: package or an https:// URL can set does not compare, because the audit cannot read it at the base. A rule-severity or ignore-pattern finding takes the stricter of the base and head weakened-tests severity, so a change that turns off weakened-tests still reports. A suppression comment in the head config does not hide it, and it has no suppression action. A drop does not report when the lost test lines name code that the change removed, such as a deleted function or a deleted fixture directory: the tests went with that code. A name of a deleted file that a changed file still holds, a name that a deleted file only imports, a directory that still exists, and a reserved word such as import do not count. An assertion drop reports only when assertions are at least half of the code lines that the file lost, so a test rewrite that moves setup checks into a helper does not report. The finding is a fact about the change, not a proof that coverage dropped. Defaults to warn.
Example
A change turns it('charges tax', ...) into it.skip('charges tax', ...), or lowers coverage.thresholds.lines from 90 to 70.
Fix
Decide if the weaker test or threshold is intended. If it is, suppress the finding with // fallow-ignore-next-line weakened-tests (or fallow-ignore-file for a file-level kind), or set the rule to off for the path in overrides.
Facts
| Fact | Value |
|---|---|
| Issue type | weakened-tests |
| Rule id in SARIF output | fallow/weakened-tests |
| Command | fallow audit |
| Default severity | warn |
Config key in rules | weakened-tests |
| Suppression comment | // fallow-ignore-next-line weakened-tests |
Fixable with fallow fix | No |
| Opt-in | No |