feat: exit non-zero from slither-mutate when mutants are uncaught#3018
Open
cats2101 wants to merge 1 commit intocrytic:masterfrom
Open
feat: exit non-zero from slither-mutate when mutants are uncaught#3018cats2101 wants to merge 1 commit intocrytic:masterfrom
cats2101 wants to merge 1 commit intocrytic:masterfrom
Conversation
Currently slither-mutate exits 0 even when mutants escape the test suite, so CI runs cannot tell whether the suite caught everything without parsing logs. Track campaign-wide totals across files and exit with EXIT_CODE_UNCAUGHT_MUTANTS (2) when any mutant survives, keeping the exit code distinct from 1 so CI can tell setup errors from uncaught mutants. Closes crytic#2772
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #2772.
slither-mutatecurrently exits 0 even when mutants escape the test suite, so CI runs cannot tell whether the suite caught everything without parsing logs. This adds campaign-wide accumulators across files and exits non-zero when any compiled mutant survives.Per the issue, the exit code is intentionally distinct from 1 (used by argparse / setup failures) so CI scripts can distinguish "test framework misconfigured" from "test suite failed to catch a mutant". The constant
EXIT_CODE_UNCAUGHT_MUTANTS = 2makes the contract explicit.The exit logic is factored into a small
_exit_for_campaign_resultshelper so it can be unit-tested without standing up a real Foundry/forge campaign.Test plan
pytest tests/tools/mutator/test_mutator.py -k exit_for_campaign— 3 new tests cover all-caught, no-mutants-at-all, and uncaught (assertsSystemExit.code == 2and a summary line is logged)ruff checkcleanruff format --checkclean