Tests that had never failed, written by an AI and passed by everyone: mutation testing with Stryker.NET

TL;DR

Continuing our series on Tendero, the AI-native ecommerce platform, in this article I ask whether the tests I already had would notice if the code were wrong, with mutation testing and Stryker.NET.

• Coverage says a line ran. Mutation testing asks whether anybody would notice if that line were wrong.
• Stryker.NET found real gaps in six minutes that my hand-written mutation lists had missed.
• It is slow and it cannot see past your fakes, so in Tendero it runs when a phase closes, never on every pull request.

One of my tests asked for page two of a search and checked that it got two results:

var page = await Hybrid(lexical).SearchAsync(
    new ProductSearchQuery("cafetera", "es", Page: 2, PageSize: 2), Ct);
Assert.Equal(2, page.Hits.Count);

→ The test as it was before this article’s story, shortened. At the article’s tag it already reads as it should: tests/Search.Tests/HybridSearchStrategyTests.cs, lines 230–247

It passed. It would also have passed if page two had been page one.

That is not a bad test by accident. It runs the paging code, so coverage counts that code as tested. It simply never looks at which products came back, and the line that decides them is this one:

var page = ranked.Skip((query.Page - 1) * query.PageSize).Take(query.PageSize);

→ Reference: src/Search/Strategies/HybridSearchStrategy.cs, line 155

Change that * to a / and page two becomes page one. The test still counts two results, and stays green.

I did not notice by reading the test. A tool made that change on purpose and then told me that nothing had failed. The test says what it should say now — Assert.Equal(["c", "d"], page.Hits.Select(hit => hit.ProductId)) — and that one line is the whole article in miniature.

Why this matters more now

Most of Tendero’s code is written with an AI coding agent, and that changes where the work is. Writing code became much faster. Reading it did not. A person can review a hundred lines with care, but not a thousand a day, and a review that tries ends up checking that the code looks right, not that it is right.

So the checks that never get tired matter more than they did, and they have to be checks that can fail:

• Integration tests against real services. With Testcontainers, a test starts a real PostgreSQL or Elasticsearch in Docker, so the query the code sends is answered by a real engine. A fake only behaves the way its author believed, and near the end of this article that is exactly what hid a bug.
• Load tests, which say whether the shop still answers when many people ask at once. Tendero does not have them yet; its traffic simulator, the subject of a later article, is the first step towards them.
• Mutation testing, which checks the tests themselves. When an agent writes the tests as well as the code, somebody has to ask whether those tests would fail if the code were wrong. That is what this article is about.

Tests became cheaper to write. That makes them worth more, not less, but only the ones that would notice a mistake.

What mutation testing is, in short

It is a test of the tests. Coverage answers «did this line run?». Mutation testing answers «would anybody notice if this line were wrong?».

The tool breaks the code on purpose, one small change at a time — a * becomes a /, a >= becomes a >, true becomes false — and runs the tests against each broken version. Each broken version is a mutant. If a test fails, the mutant is killed: your tests noticed. If every test stays green, the mutant survived, and nothing in your tests describes what that code should do.

I had been doing this by hand for a week: every new test has to fail against a deliberate change to the code, and the list of changes goes in the commit message. It works, and it has one obvious weakness. It only checks the mutations I thought of.

Stryker.NET is the tool for .NET. It picks the changes itself, builds them all in one go, and runs only the tests that reach each one. Two commands, because it is pinned in the repository’s tool manifest:

dotnet tool restore
dotnet tool run dotnet-stryker --config-file stryker-config.json --skip-version-check

Reading the report without being fooled by it

The first run over src/Search took six minutes and produced 983 mutants: 214 killed, 2 timed out, 171 survived, 273 with no test reaching them at all, plus 217 that did not compile and 106 excluded by configuration.

That gives two scores, and this is where everybody misreads the report:

• of total = detected ÷ (detected + undetected) = 216 ÷ 660 = 32.73 %
• of covered = detected ÷ (detected + survived) = 216 ÷ 387 = 55.81 %

The distance between the two is the diagnosis. The Qdrant adapter scored 12.60 % of total and 72.73 % of covered: almost all its mutants had no coverage, because that adapter is proven by integration tests against a real container, which this run did not include — and where unit tests do reach it, they killed 16 of 22. That is a strategy, not a gap. Elasticsearch.cs scored 15 % and 31 %. Both low. Unit tests run that code and do not notice when it changes. That was a gap, and it was real.

That is the report itself: the same four rows from the first run and from the third, which comes at the end of the next section. Qdrant’s of total barely moves, because its proof lives in the container tests, while its of covered climbs. Elasticsearch.cs more than doubles on both, because that gap was real and now has tests.

Survivors are a reading task, not a number

My first reaction to 171 survivors was the wrong one: that it meant writing 171 tests. I sorted them by reading the line each one sits on:

So eighteen, not a hundred and seventy-one. I wrote 17 test cases and strengthened two assertions, and then checked each one by hand against the exact mutation it was written for: 20 mutations, 20 kills.

The third run, with both test projects and logging excluded, took fifteen minutes over 1,008 mutants: of covered went from 55.81 % to 75.97 %, and survivors from 171 to 93.

The best line in that table is one that needed no new test. RelevanceMetrics.cs went from 14 survivors to 3 without a single change to its code or its tests — they had always existed, in a project the first run did not mutate against. A project measured against only some of its tests reports survivors that do not exist.

What we would not have found without it

Two real cases from this project, and they point in opposite directions.

The filter that could have gone missing. Tendero’s search only shows published products, so every lexical query carries one filter: status must be active. Stryker changed that word to an empty string, and every test stayed green. Reading the tests showed why: they checked that the request carried a status filter, and never checked its value.

Assert.Contains("\"status\"", body);   // the field is there
Assert.Contains("\"active\"", body);   // added after Stryker: so is the value

→ Reference: tests/Search.Tests/ElasticsearchRequestBodyTests.cs, lines 34–35, one of the tests that check it

Without that mutant, the day somebody changed that word the shop would have shown unpublished products in its search results, and every test would still have passed. The fix was one line per test, and the hand-made mutation now turns four of them red.

The page number nobody checked, at the top of this article, is the second one: the offset could have been a division, and the test counted results instead of reading them.

Both have the same shape. The code was covered. The tests ran it. Nobody had written down what the code should do, so nothing noticed when it did something else.

What it gives us, then:

• It finds what we did not think of. Six survivors of the first run were real gaps my hand-written lists had missed: the page offset, tie order after reranking, an empty-text fallback, two boundary conditions, one branch of a total.
• It tells a strategy apart from a gap, through the distance between its two scores.
• It catches a phase that arrives with holes. When the search-answers phase closed, src/Search came back at 70.75 % of covered, below the 75.97 % recorded before it. The new code had arrived with gaps, and the rule found them before anybody noticed by reading.

What it costs:

• Time. Six minutes with one test project, fifteen with two, for one project of the solution.
• A preview runner. With Microsoft.Testing.Platform, Stryker 5.0.0 counts kills inconsistently above concurrency 1 — 70 to 98 kills out of the same 337 mutants, with no change to the code (issue #3832). At concurrency 1, two runs came out identical, file by file. The committed configuration pins it there.
• Its compiler falls behind. Stryker 5.0.0 embeds Roslyn 5.9; C# 15 arrives in 5.11. (A month later, a phase close found the way past it: Stryker’s own language-version: Preview setting builds C# 15’s closed records.)
• Equivalent mutants. Some changes do not change behaviour at all, so no test can kill them. You mark them and move on.
• The percentage invites bad tests. Chasing a fixed 80 % means writing tests for log messages, and rebuilding with fakes what container tests already prove.

And one thing it cannot do. It cannot see past your fakes. The worst defect of that same week was in the Elasticsearch client: a second .Filter(...) call replaced the first instead of adding to it, so a query quietly lost its filter. That code was covered, and its mutants lived and died against a fake that behaved the way I believed the library behaved. No mutant could find it. What found it was a number that did not add up, and then reading the request the client really sent. Mutation testing measures your tests against your code, never your code against the world.

One dependency note, since the tool is one: dotnet-stryker 5.0.0 is Apache-2.0 at that tag, its dashboard is never contacted unless you ask, and its only outbound call checks NuGet for a new version — --skip-version-check removes it. For .NET there is no living alternative: Faultify is GPL-3.0 and last released in 2021, Fettle is archived.

How we do it in Tendero

Two levels, because they answer different questions.

Level 1 — by hand, on every new test. A test written for code that already exists passes the first time it runs, and that is exactly the moment it has proved nothing. So it earns its place by failing: the code is mutated one rule at a time, every mutation has to turn at least one test red, and the list goes in the commit message. Fast, deliberate, and blind to whatever nobody thought of.

Level 2 — Stryker, when a phase closes. It runs over every project the phase touched, with every test project that exercises them, at concurrency 1, with logging and telemetry excluded. Each mutated project has its own stryker-config.json committed next to it, so the run is repeatable.

The rule is a ratchet, not a target: a project’s of covered score may not fall below the number recorded last time, and every survivor is classified — a real gap, closed with a test or written down with why it waits; an equivalent mutant; or noise. A fixed 80 % would invite tests of log messages, and the first run of a project has nothing to compare against anyway. «Do not go down, and explain what survived» rewards the thing that matters: the code you touched did not lose its protection.

Never on every pull request, while that runner is in preview. Minutes per project, from a tool that only gives the same answer twice at concurrency 1, does not belong between a change and its merge. What does belong there is the fast half: the build with warnings as errors, the architecture rules, the tests, and the search relevance gate.

The frontend has only the first level, for now. Its Angular code is checked by Vitest specs and Playwright browser tests, and a spec written for code that already exists earns its place the same way: a hand-made mutation, listed in the commit. The nine mutations in How we know it works below are frontend ones, run against a design spec. What the frontend does not have yet is the second level: no tool mutates it.

The equivalent of Stryker.NET there is StrykerJS, so on the day this article went out I tried it on a throwaway branch, with rules written before the first run: no change to the application’s code, two development packages at most, no second test configuration to keep in step, and real gaps found, or it is not worth it. The result is split in two:

• The shared libraries: yes. Each one already runs Vitest with its own configuration, and StrykerJS read it as it was. One library, 146 mutants, 45 seconds, and about six real gaps among 12 survivors — an error body read as a problem without its code, a country code in lower case that was never tried.
• The applications: not yet. Angular’s own test builder runs Vitest inside itself and lets no other tool change its configuration or reach its runner. That is an open request on the Angular CLI (issue #32182), and mutation testing is the example it gives. It can be worked around: one store, 44 mutants, a little under two minutes, with real gaps in it too. But the workaround runs the tests from scratch for every mutant and edits the files in place, so it does not scale to a whole application, and a second Vitest configuration only for this tool is exactly the kind of complexity the rules exclude.

So the libraries can take the second level now, and the applications wait for Angular to open its runner. Until then, their tests are only as good as the mutations somebody thought of.

The repository is public and MIT licensed, and the code exactly as this article describes it is tagged blog/04-tests-that-had-never-failed. The Stryker configurations sit next to the projects they mutate: src/Search/stryker-config.json and src/Ucp/stryker-config.json.

Further reading

Where the idea comes from, and whether it holds:

• DeMillo, Lipton and Sayward, Hints on Test Data Selection: Help for the Practicing Programmer, IEEE Computer, 1978 — the paper usually named as the start of mutation testing.
• Jia and Harman, An Analysis and Survey of the Development of Mutation Testing, IEEE Transactions on Software Engineering, 2011 — thirty years of it in one survey.
• Just et al., Are Mutants a Valid Substitute for Real Faults in Software Testing?, FSE 2014 — over 357 real faults, catching mutants goes with catching real bugs, whatever the coverage.

How large companies use it:

• Petrović and Ivanković, State of Mutation Testing at Google, ICSE-SEIP 2018, and Petrović et al., Practical Mutation Testing at Scale: A view from Google, IEEE TSE 2021 — only the changed lines are mutated, and surviving mutants appear as comments in code review, with no global score.
• Beller et al., What It Would Take to Use Mutation Testing in Industry: A Study at Facebook, arXiv, 2021 — more than half of 15,000 mutants survived the existing tests, and almost half of the developers shown one said they would act on it.
• Foster et al., Mutation-Guided LLM-based Test Generation at Meta, FSE 2025 — a language model writes a few mutants aimed at one risk, then the tests that kill them.

The tools:

• Stryker.NET, its configuration, and StrykerJS’s incremental mode, which mutates only what changed since the last run.

How we know it works

• The test that fails: every test written for existing code is checked against a deliberate mutation, and the commit message lists them. The most recent set is the frontend’s accessibility work: nine mutations — a wordmark’s colour, its size, its weight, the token behind it — and all nine turn a test red.
• The check on every pull request: warnings as errors, the architecture rules, the backend and frontend tests, and the relevance gate. Stryker is deliberately not in that list.
• The mutation score: at this tag, src/Search is at 76.23 % of covered and src/Ucp at 69.34 %, both recorded in docs/testing.md with their survivors classified. Search’s number is marked stale on purpose: the code it measured moved to another project the same day, so the next phase close measures it again instead of comparing against a number that describes something else.

Next article: Seeing it run — the tests measure the code, but nothing yet measures the shop while it runs: a click in the browser, the span it caused three services away, and the one event that proves a feature is used at all.

Tendero — commerce for humans and agents.