Your build is green and your shop’s search is broken: NDCG as a CI gate in .NET

TL;DR

Continuing our series on Tendero, the AI-native ecommerce platform, in this article I put its search under regression tests: an offline relevance gate that runs in CI.

• A bad search result page looks exactly like a customer who was not interested, so search needs its own tests.
• An offline relevance gate (NDCG@10 and recall@50 against an annotated golden set) found a bug on its first run that every other test had missed.
• A gate that gives different numbers for the same code is worse than no gate: make it deterministic first.

Here is the uncomfortable thing about ecommerce search: a bad result page looks exactly like a customer who was not interested.

Nothing fails. The build stays green, the endpoint returns 200, the page renders, and the shopper leaves. Nobody gets an alarm, because the failure is silent. That is why so many shops have bad search that nobody fixes. In Baymard Institute’s 2026 benchmark of more than 170 ecommerce sites and apps, 56 % of them failed to support what their users searched for.

I built a quality gate for Tendero’s search before I changed the search itself. The first time I ran it, it found a bug that had been there since the first line of search code.

This article explains the gate from the start. First, why search matters so much in this project. Then the three terms the whole series uses to measure it: the golden set, recall and NDCG, plus BM25, the search they measure here. Then what the gate found, how other companies do the same, and why I think it matters more now, with AI.

Why search comes first

In a shop, search is not one feature among many. It is the floor that other features stand on.

In Tendero, at least three different users ask the same search for products:

• The shopper, who types in the search box.
• Tino, the shop’s AI assistant, which a later article builds. When a shopper asks Tino for «a coffee maker under 50 euros», Tino does not know the catalogue by heart. It calls the search, and then it explains what the search returned.
• External AI agents, which shop on someone’s behalf through MCP and UCP, also in later articles. They call the same search too.

So when search brings back the wrong products, everything above it is wrong as well. An AI assistant cannot recommend a product that search did not find. It will explain the wrong products very well. And an AI agent does not scroll like a person. It takes the first results and moves on. For an agent, the top of the list is the whole list.

That is the reason for this article. Search quality is the base of most of what comes next in the series: the shopper’s search and Tino’s answers. Later, Steward, the back-office assistant, gets a golden set of its own, built the same way. If I cannot measure search, I cannot measure anything built on it.

Why it matters more with AI

There is a second reason, and it is about how code is written today.

With AI tools, code is written much faster than before. I use them every day on this project. More code means more changes, and every change can break search without breaking anything else. A human reviewer cannot test thirty queries by hand on every pull request. A machine can, in under a minute.

So the gate gives me three things:

• It catches mistakes early. Mine and the AI’s. A change that makes search worse fails the build, before anybody sees it.
• It is deterministic. The same code gives the same number, every time. That matters even more when parts of the system use models, which are not deterministic. The gate is the fixed point that everything else is compared with.
• It turns opinions into numbers. «The search feels better» becomes «NDCG@10 went from 0.769 to 0.860». You can discuss a number in a pull request.

Fast building without a measure is just fast guessing. The gate is what lets me move fast and still know where I am.

First, the idea, explained with a shop

It took me three attempts to explain offline relevance evaluation well. What finally worked was dropping every technical word and using the shop the project is named after: tendero is Spanish for shopkeeper.

You have a shop with six products and a shop assistant. A customer asks for «running shoes». The assistant goes to the storeroom and comes back with a tray. The things on the tray are in an order: what he thinks you want most is on top. The only question is: is this assistant any good?

First, you write down the right answers. Before the test, you write, for each question, what he should bring back. It is like a teacher who writes the exam and the answers at the same time. For «running shoes»: the running shoe is a 3, the running t-shirt is a 1 (it is for running, but it is not a shoe), and the backpack is a 0. The grade is not yes or no. The t-shirt is not the answer, but it is not a silly answer either. That list of answers is the golden set.

Then you grade him, with two questions. Did he find it at all? That means: did it come out of the storeroom, anywhere on the tray? And did he put it where people look? Customers look at the top of the tray, and almost never at the bottom. The first question is recall. The second is NDCG.

Then you write the grade on the wall. Today he scores 8.6, so the wall says «never below 8.5». Every time someone changes how he searches, he takes the exam again.

Every part of the story matches one real thing. The tray is the result page. The order on the tray is the ranking. The list of answers is the golden set. The two questions are the two metrics. The grade on the wall is the threshold in CI. Nothing in it has to be unlearned later.

The next sections give each of these words its exact meaning.

The golden set: the answer sheet

This series uses the term golden set often, so here is the exact meaning.

A golden set is a fixed list of questions with the right answers already written down by a person. For search, each question is a query, and each answer is a product with a grade: how good that product is for that query. The grades are called judgments. The set is «golden» because it is the reference that everything else is measured against. It is the teacher’s answer sheet, written before anybody takes the exam.

Three things make it useful:

• It does not change when the code changes. The search changes, and the golden set grades it. If you change both at once, the number means nothing.
• A person writes it, not the system under test. A search engine cannot grade itself.
• It lives in the repository, next to the code, and changes go through review like code does.

The idea is older than the web. In the 1960s, Cyril Cleverdon’s Cranfield experiments in England tested library search systems against a fixed set of questions and documents judged by people. TREC (Text REtrieval Conference), the shared evaluation that search researchers have run since 1992, still works this way, and calls the judgments qrels. You will also see the same thing called a gold standard, a test collection or a judgment list. Here it is always golden set.

BM25: how the search finds products

At this point in the project, Tendero’s search is lexical: it looks for the words of the query inside the products. Elasticsearch does that, and it puts the results in order with a formula called BM25.

BM25 gives every product a score for a query. In plain words:

• A product scores more when the query’s words appear in it.
• A rare word counts more than a common one. «zapatillas» says much more about a product than «de» or «con».
• Repeating a word helps less and less. A product does not win by writing «shoe» twenty times.
• A word in a short field counts a little more than the same word lost in a long description.

The name is not very helpful. BM means Best Matching, and 25 is simply the number of the version of the formula that won. Stephen Robertson and his colleagues at City University London built it for their Okapi search system, and it first appeared at TREC in 1994. It grew from the probabilistic model of relevance that Robertson had developed with Karen Spärck Jones in the 1970s. Robertson and Zaragoza later described the whole idea in The Probabilistic Relevance Framework: BM25 and Beyond (2009). It has been the default ranking in Elasticsearch since version 5.0, in 2016.

BM25 is fast, needs no model and no GPU, and is still very hard to beat on exact words. Its limit is just as clear: it only sees words. If the shopper types «reading light» and the product says «desk lamp», BM25 cannot connect them. That limit comes back at the end of this article, and later articles add search by meaning to close it. BM25 stays in Tendero as the permanent fallback: when a model is not available, the shop still searches.

Recall@50: did the search find it at all?

Recall answers the first question from the shop: did the assistant bring it out of the storeroom?

For one query, it is a simple share:

recall@50 = (relevant products found in the first 50 results) ÷ (all relevant products in the golden set for that query)

A product counts as relevant when its grade is 1 or more. Its position inside the 50 does not matter. Only whether it is there.

An example. For «running shoes», the golden set has three relevant products: shoe A (grade 3), shoe B (grade 2) and the running t-shirt (grade 1). The search returns shoe A and the t-shirt somewhere in its first 50 results, but never shoe B. Recall@50 is 2 ÷ 3 = 0.67.

var relevant = judgments
    .Where(judgment => judgment.Value >= RelevantFrom)
    .Select(judgment => judgment.Key)
    .ToHashSet(StringComparer.Ordinal);
if (relevant.Count == 0)
    return null;
var found = rankedIds.Take(k).Count(relevant.Contains);
return (double)found / relevant.Count;

→ Reference: tools/SearchEval/RelevanceMetrics.cs, lines 55–71

Why 50, when a page shows far fewer? Because recall does not measure the page. It measures the first step of search: finding the candidates. Fifty is deep enough that order does not matter. So when recall falls, the problem is clear: the product was not found at all. It is not a product that was found and placed badly. This matters later in the series. Some steps only reorder the list that search found, for example a model that reads the first results again. Those steps can move a product up, but they cannot bring back a product that is not in the list. Recall@50 watches exactly that.

Recall is one of the oldest measures in search. The Cranfield experiments were already using it in the 1960s.

NDCG@10: did it put it where people look?

Recall says nothing about order. Two trays with the same products in a different order have the same recall. But a shopper sees them very differently. That is what NDCG measures: how good the order is, compared with the best order possible.

It has two simple ideas inside:

• The top counts more. A good product in ninth place is worth very little, because few people scroll that far.
• One perfect result is worth more than several average ones. A product with grade 3 is worth seven products with grade 1.

The score is divided by the score of a perfect order, so 1.0 means «impossible to order better» and 0 means nothing useful was shown. The @10 means «only look at the first ten results».

Here is the same example, with two different trays:

Both trays found the same products, so recall is 0.67 for both. But tray B puts the perfect shoe on top, and tray A hides it under a t-shirt. NDCG gives tray A 0.58 and tray B 0.81. That is exactly why the gate measures both: one number for «found», one number for «placed».

Where NDCG comes from

NDCG is not new, and it was not invented for shops. Two researchers at the University of Tampere, in Finland, Kalervo Järvelin and Jaana Kekäläinen, proposed it for document search. They first presented the idea at the SIGIR conference in 2000. The full version is their 2002 paper Cumulated gain-based evaluation of IR techniques, in ACM Transactions on Information Systems.

They had a simple problem. The usual measures of the time, precision and recall, treat every result as relevant or not relevant, with nothing in between. They also ignore where a good result appears. But people judge results in grades, and they read from the top. So the two researchers built a measure that rewards highly relevant results and the top of the list. They tested it on TREC data, judged on a four-point scale.

The name says what the measure does, one word at a time:

• Gain: what each result is worth, from its grade.
• Cumulative: the gains are added up, down the list.
• Discounted: each gain is made smaller the lower it sits, because fewer people get there.
• Normalized: the total is divided by the best total possible for that query, so every query gets a score from 0 to 1.

Today NDCG is the standard way to grade a ranked list. Search engines, recommender systems and learning-to-rank libraries all report it.

The formula, and why it has this shape

Here it is in code. Each result’s gain is 2^rel - 1, and that gain is divided by the log of its position:

private static double Gain(int relevance) => Math.Pow(2, relevance) - 1;
// position is 0-based; the first position is not discounted (log2(2) = 1).
private static double Discount(int position) => Math.Log2(position + 2);

→ Reference: tools/SearchEval/RelevanceMetrics.cs, lines 73–76

What 2^rel - 1 means. rel is the grade a person gave the product for that query, from 0 to 3. The gain is 2 to the power of that grade, minus one:

The «minus one» makes an irrelevant result worth exactly nothing. The power of two makes each grade worth about twice the one below it, so a 3 is worth seven 1s.

Why this formula and not another. The 2002 paper used the grade itself as the gain, so a 3 was worth three 1s. That is called a linear gain. The power-of-two form became common later, with Microsoft’s work on web search ranking (Burges et al., 2005). Today most search teams use it. The reason is about how people shop. A product that is exactly right ends the search. A product that is only related almost never does, however many of them you show. A linear gain says that three related products are as good as one perfect product. For a shop, that is false. The exponential gain is more honest.

Why the log of the position. The discount must grow with the position, but slowly. A result in second place is less useful than one in first place, but it is not worthless. The log does exactly that. Position 1 is divided by 1, position 2 by about 1.6, position 9 by about 3.3. Researchers later proved that this log discount still tells good rankings from bad ones as lists grow long (Wang et al., 2013). Discounts that fall much faster, such as dividing by the square of the position, do not.

The example, by hand. The perfect tray is shoe A, shoe B, t-shirt: 7 ÷ 1 + 3 ÷ 1.585 + 1 ÷ 2 = 9.39. Tray B is shoe A, t-shirt, backpack: 7 ÷ 1 + 1 ÷ 1.585 + 0 = 7.63, and 7.63 ÷ 9.39 = 0.81. Tray A is t-shirt, shoe A, backpack: 1 ÷ 1 + 7 ÷ 1.585 + 0 = 5.42, and 5.42 ÷ 9.39 = 0.58. The gate does the same sum for every query, and reports the average.

The golden set has reasons, and the reasons argue with you

This is what a few judgments look like:

{
  "query": "zapatillas running mujer",
  "judgments": [
    { "externalId": "B073WXYZ01", "relevance": 3, "why": "exact category + gender match" },
    { "externalId": "B07TUVW404", "relevance": 1, "why": "running shoe, wrong gender" },
    { "externalId": "B08JKLM202", "relevance": 0, "why": "backpack — brand noise" }
  ]
}

→ A simplified example, written for this article. The real golden set grows and changes in the next articles of the series, and each one links the version it describes. At this article’s tag, it is tools/SearchEval/golden/es.json and en.json.

Two details matter more than they seem to.

Judgments use externalId, the supplier’s id, not the internal product id. The internal id is a GUID v7, created on every import. A golden set that used it would stop working every time the catalogue was loaded again. The supplier’s id does not change, and it is easy to read in a diff.

The why field is not a comment. It is what lets somebody review a judgment. Without it, nobody can say whether a 3 is fair, and the gate slowly becomes a measure of the opinions of whoever wrote the answers. I come back to this later, because it caught me.

What the gate found on the first day

I built the gate first, measured, and only then changed the search. That order is the whole point, and I nearly lost it in a single commit.

Seven of the twenty-two Spanish queries returned zero results. The worst one was zapatillas running mujer, women’s running shoes. It matched nothing at all.

The cause was a single option. The query was a multi_match of type best_fields with operator: and. best_fields with and needs every word to appear in one field. «zapatillas running» is in the product’s name, and «mujer» (women) is in attributesText. No single field held all three words, so nothing matched.

The fix is not a different type. It is a should with two clauses. The first lets the words of a query spread across fields. The second accepts small typos: zapatilas still finds zapatillas. Elasticsearch calls this typo tolerance fuzziness, and that is the word you will see in the code:

.Must(m => m.Bool(alternatives => alternatives
    .Should(
        // cross_fields: terms may spread across fields.
        should => should.MultiMatch(mm => mm
            .Query(query.Text)
            .Fields(SearchableFields)
            .Type(TextQueryType.CrossFields)
            .Operator(Operator.And)),
        // best_fields with fuzziness: tolerates typos ("zapatilas").
        // Separate because cross_fields does NOT support fuzziness.
        should => should.MultiMatch(mm => mm
            .Query(query.Text)
            .Fields(SearchableFields)
            .Fuzziness(new Fuzziness("AUTO"))
            .Operator(Operator.And)))
    .MinimumShouldMatch(1)))

→ Reference: src/Search/Elasticsearch.cs, lines 300–320

Only the gate could have told me about the second clause. cross_fields cannot accept typos. Changing the type alone would have fixed three queries and quietly broken typo tolerance. zapatilas and runing shoes both score 1.000. Both would have dropped to zero, and nobody would have noticed, because nobody tests again the queries that already work.

The gate also removed a field. category was in the list of text fields to search. But it was stored as a keyword, so it could only match the exact code COFFEE_MAKER. It promised a match that could never happen.

The part worth remembering: the gate was worth building the first time it ran. The bug it found was invisible to every other kind of test. The build was green. The unit tests were green. The API returned 200. It just brought back an empty tray.

Two ways the numbers lied, both my fault

A quality gate that gives different numbers for the same code is worse than no gate, because you trust it. This happened to me twice. Both times I only noticed because I ran it more than once, out of habit.

Old documents in the index. Product ids are GUID v7, created on every run. So indexing again did not replace anything. It added to it. The documents from the previous run stayed in the index and competed in the ranking. Two runs of the same code gave 0.674, then 0.360. The fix is to delete the indexes before every run.

The refresh race. Elasticsearch makes new documents searchable about once a second. My first fix waited until some query returned a result, which only proves that one document is visible, not all six. Two runs gave 0.860, then 0.769. The fix is an explicit refresh after indexing, so nothing is left to timing:

public static Task RefreshAsync(
    string elasticsearch, IEnumerable<string> indexNames, CancellationToken cancellationToken) =>
    ForEachIndexAsync(elasticsearch, indexNames, HttpMethod.Post, "/_refresh", cancellationToken);

→ Reference: tools/SearchEval/IndexAdmin.cs, lines 27–29

The lesson is not «remember to refresh». It is that both bugs produced believable numbers. Nothing crashed, no test failed, and 0.769 looks like a normal score for a small catalogue. So test a measurement tool for giving the same answer every time, not only for giving a correct one. Run it five times and compare the output. Mine now gives the same numbers, to three decimals, across five runs. That is why the rest of this article can be trusted.

The why field checked my own work

I wrote this judgment for the Spanish golden set:

flexo con usb → relevance 3, «flexo is a synonym for desk lamp; lexical search cannot bridge that, this query exists to measure the gap»

It scores 1.000, and it always did. The Spanish product description starts with the words «Flexo LED».

I wrote the judgment from what I thought the catalogue said, not from what it says. Then I wrote the same wrong claim into the project documentation as a known gap. What caught it was the why field, which exists so that a person can check whether a judgment is fair.

It checked mine. A golden set without reasons is one person’s opinion with decimal places.

The gate in CI, and its honest limits

dotnet run --project tools/SearchEval -- --ci

→ Reference: .github/workflows/ci.yml, lines 113–118, where CI runs it

The tool deletes the indexes, indexes the seed catalogue through the same ports the application uses, refreshes, scores both languages, prints a Markdown report, and fails the build if a score falls below the committed thresholds. On the first day, those were:

Lowering a threshold or changing a judgment needs a reason in the pull request. That is the only protection against the weak point of this whole idea: lowering the threshold until the build passes.

It proved useful almost at once. Upgrading Elasticsearch from 9.1.0 to 9.5.0 is exactly the kind of change where «it still works» means nothing. What changes across four minor versions is not the API. It is the analyzers, stemming, stop words and BM25 details, and all of them live inside the ranking.

The same to three decimals, after two runs and no guessing. Without the gate, the honest report would have been «I searched for a few things and they looked fine».

One practical note, because I got it wrong once. The container is declared ContainerLifetime.Persistent, so changing the image tag does not give you the new version: .NET Aspire reuses the container it already has. Remove it first with docker rm -f, or you measure 9.1.0 twice and decide the upgrade was safe.

→ Reference: src/AppHost/AppHost.cs, line 76

What the queries scoring 0.000 were telling me

They are a diagnosis, not noise, and they were in the golden set on purpose:

1. Attribute values were not translated. Colour and gender were stored in Spanish. So «navy blue shoes» and «womens running shoes» could not match in the English index.
2. The category names were codes, not words people type. «induction cookware» failed because COOKWARE is a code, not a phrase anyone searches for.
3. Some words are simply not in the catalogue: «reading light with usb», «ropa deportiva», «gift for the kitchen». This is BM25’s limit from earlier: it only sees words. It is the gap that search by meaning later had to close.

The gate turned the first two into changes with a number attached. A few days later, attribute values got a label in each language. English NDCG@10 went from 0.720 to 0.811, and Spanish did not move at all. That was the test, because the change only added English text. Then the categories got real names in both languages, and NDCG@10 went to 0.943 in Spanish and 0.937 in English.

And the cutoff is a placeholder

@10 was a convention, not a choice based on Tendero. Ten is not a law: a Walmart paper on sponsored search reports NDCG@4 and NDCG@8. The cutoff should match what the storefront shows before you scroll, and mine had no real grid yet. Saying so is more honest than pretending it was reasoned.

Six products was not the end of it

Everything above was measured on six products and thirty judgments. By the code this article links, the catalogue had grown to 100 products. The golden set had grown to 268 judgments, over the same 22 queries per language. With 100 products, the old numbers stopped being comparable. Most of what search returned had never been judged, and a result nobody judged counts as irrelevant.

So the baseline was measured again, and the thresholds with it. At that code, the gate scores 0.813 (es) and 0.766 (en) on NDCG@10, against thresholds of 0.80 and 0.75. The scores are lower, and search did not get worse. The exam got harder and fairer. From outside, those two things look the same. That is why the reason goes into the pull request, and why the old numbers stay written down next to the new ones.

How other companies do it

«Add a relevance gate» sounds like standard practice. It is not, and it is worth being honest about where this sits.

• Most shops never measure search at all. They take the default search of their platform and never look at it again. Baymard’s 56 % from the start of this article is the result.
• Mid-sized shops buy a search service, such as Algolia, Klevu, Constructor or Bloomreach, and get dashboards. But those are business metrics: conversion, click-through and, above all, the zero-result rate, the most tracked search number in ecommerce. The zero-result rate is exactly what my gate found on the first day.
• Large retailers judge search the same way this article does, at a much bigger scale. Two of them published their golden sets:
  • Amazon released the Shopping Queries Dataset in 2022: about 130,000 queries and 2.6 million query-product judgments, in English, Spanish and Japanese. Each product gets one of four grades, Exact, Substitute, Complement or Irrelevant: the same idea as Tendero’s 0 to 3.
  • Wayfair released WANDS, presented at ECIR 2022: 480 queries and 233,448 judgments, graded Exact, Partial or Irrelevant, with the guidelines its annotators followed.

The idea is the same at every size: a fixed list of queries, products graded by people, and a number that says whether a change made search better or worse. Mine started with thirty judgments against six products. That is the same discipline, thousands of times smaller, and that is the honest way to describe it.

The difference that really matters is offline against online. An online A/B test shows a change to real customers and tells you what they did, but it takes weeks and needs traffic. An offline gate tells you in forty seconds whether you broke something, on a laptop, with no users. Serious teams use both: offline on every change, online on the changes that pass. A solo project only has the offline half, and that is not a weakness. It is the half that gives fast feedback.

Why I think this is so important

When I started the project, I thought of search as a box at the top of the page. I do not think that any more, for three reasons.

1. Search is invisible when it fails. No error, no alarm, just a customer who leaves. Without a number, nobody even knows there is a problem.
2. AI is built on top of search, not instead of it. Assistants and agents look products up first, and then they talk. A bad search makes a confident assistant that recommends the wrong things. Measuring the assistant without measuring the search under it measures the wrong thing.
3. Code changes faster than ever. With AI tools, a day brings more changes than a week used to. Only an automatic, deterministic check can keep up with that.

That is why the gate came before every other search feature in Tendero. The same tools come back later in the series, for the shopper’s search, for Tino and for Steward: golden sets with reasons, NDCG and recall, and a threshold in CI.

Remember how this article started: a bad result page looks exactly like a customer who was not interested. Without a gate, nothing warns you, and empty trays can go on for a year.

Summary and conclusions

• Search needs its own tests. A bad result page breaks no build and returns 200, so only a relevance measure can see it.
• Recall says whether search found a product. NDCG says whether it put it where people look. Measure both.
• Build the gate first, then change the search. On its first day, the gate found a bug that every other test had missed.
• Make the gate deterministic before you trust it. Both of my measurement bugs produced believable numbers.
• Give every judgment a reason. The why field caught my own wrong judgment.

The repository is public and MIT licensed, and the code exactly as this article describes it is tagged blog/03-ndcg-as-a-ci-gate. The golden sets, the thresholds and the evaluation tool are all in it.

How we know it works

• The test that fails: put the first-day bug back (change CrossFields to BestFields in the lexical query) and the gate fails at the tag: NDCG@10 falls to 0.671 (es) and 0.537 (en), below thresholds of 0.80 and 0.75, and the tool exits with an error.
• The check on every pull request: CI starts Elasticsearch 9.5.0 as a service container and runs SearchEval --ci against it, next to the build, the unit tests and the architecture rules.
• The mutation score: not yet. Stryker.NET comes in a later phase, and the next article is about it.

Next article: Tests that had never failed. A gate tells you when search gets worse, but who tells you that your own tests would notice a bug at all?

Tendero — commerce for humans and agents.