What Is a Golden Set — and Why Every Genie Space Needs One

A German trading group came within one board meeting of reporting a quarter's revenue that was 34.9% too high. The number was wrong for boring, ordinary reasons. A golden set with sixteen questions caught it before it went out the door. This is what a golden set is, why every serious Genie space needs one, and how to build yours.

You give an executive a beautiful, chatty text-to-SQL interface. They ask "how did last quarter go?" Three seconds later, a confident number, a nice chart, and a short explanation appear. Everyone is happy.

Then someone in Controlling opens the report and quietly asks whether that number includes the unconsolidated holding, whether it uses calendar quarters or the actual fiscal year, and whether the cancellation belegs are netted out.

Silence.

The number was 34.9% too high. Nobody could see it, because it looked exactly like what you would expect. That is the trap the semantic layer is designed to close, and a golden set is how you know whether you have closed it.

Core insight

A model that gives plausible answers is not useful. A model that gives correct answers, and that you can prove gives correct answers on the questions that matter most, is useful. A golden set is what turns the second one from claim into evidence.

What a golden set actually is

A golden set is a small, carefully chosen list of questions plus their expected answers. Not a test dataset in the ML sense — a business dataset. Real questions the business asks, phrased the way the business phrases them, with an answer that has been signed off by the people who own the number.

For a Genie space it looks like a CSV with five columns:

• The question, in the wording a real user would use.
• The expected reading: the business logic that defines the correct answer — fiscal year vs. calendar quarter, which entities count, which flags exclude a record.
• The expected SQL (optional): the query that produces the ground truth. Sometimes you have it, sometimes you don't.
• The owner: who signed off on the reading. Usually Controlling for revenue, HR for headcount, Ops for utilisation. Not the AI team.
• Critical?: yes/no. A subset is board-critical. Everything else is nice-to-have.

Sixteen questions is often enough. Fifty is a lot. Two hundred means someone is confusing this with an ML benchmark and has misunderstood the exercise.

Why the naked Genie space is dangerous

Genie — Databricks' natural-language-to-SQL interface — works out of the box. Point it at a Unity Catalog schema, hit "ask", get an answer. That is the seductive part. It is also, without further work, the dangerous part.

Genie decides how to answer by reading:

1. The table and column names.
2. The table and column comments.
3. Any instructions the space administrator has written.
4. Any SQL functions registered as trusted assets.
5. Example queries the administrator has provided.

Give Genie only step 1 — table names — and it will answer confidently, and often wrongly, because the interpretation of "the last quarter" is buried in a definition it cannot see. That definition, in our trading-group case, was: "the fiscal year begins on 1 May, so the last completed quarter is Feb–Apr, not Apr–Jun; only consolidated entities count; cancellation belegs and their originals must be excluded; skonto is a payment term, not a revenue reduction."

None of that is on the table names. None of it is guessable. If Genie does not know it, it will do the arithmetic on calendar quarters, count cancellations, throw in the holding, and confidently report a number that is off by roughly one third.

That was the state of the demo we ran with the trading group's board. Three states, on the same data:

• State A — naked. Table names only. Question: "How did last quarter go?" Answer: 15.56 million. Feels right. Off by 34.9%.
• State B — commented. Table and column comments describe the fiscal-year logic, consolidation flag, cancellation rules. Genie usually gets it right. Not always. Same question asked five times gives four correct answers and one wrong one, because the model still has room to interpret.
• State C — trusted assets. A registered UC function umsatz_netto_je_gesellschaft(gj, quartal) owns the definition. Genie stops writing SQL for this question and calls the function instead. Same question, five times, same number down to the cent. 11.54 million. Correct.

The difference between A and C is a 34.9% error on the headline revenue number. Chosen deliberately for the demo: high enough that a benchmark catches it, low enough that a room full of executives would not.

How to build your golden set

The temptation is to sit in a workshop with the data team and brainstorm. Do not do that. The questions will be too technical and miss what actually gets asked.

Instead, three sources produce a golden set worth trusting.

The last twelve months of board decks and management reports. Whatever number appears on a slide is a question someone answered. Convert those into questions in the wording an executive would use — "how did last quarter go?", "which entity had the highest revenue?", "what is our order book?" — and record the number that was actually reported. That is your ground truth, signed off by whoever presented that slide.

Actual questions from the last email round. Ask the CFO's office to forward the last thirty questions they got via email or Slack. The wording, the ambiguity, the informal reference to "our Austrian entity" — this is what a real prompt looks like. Genie will get these, or it will not.

One or two deliberate trap questions. In our case: "what was the revenue of our Beteiligungen holding?" — a legal entity that has no operational revenue and never appears in consolidated figures. The correct answer is not a number. It is "this entity is not consolidated; the question does not apply." An AI that silently answers with a number here is not safe to ship. Every golden set should include at least one question the model should refuse.

Mark the subset that must never be wrong as critical. In our trading-group example, six of sixteen. Those are the ones the board sees. The remaining ten are the wider quality signal.

How to actually set up Genie for it

Below is the sequence we ran, in the order that produced the demo. It applies to any Genie space, not just the trading-group one.

1. Comments before instructions before functions. Comments are the highest-leverage single move you can make. They live on the Unity Catalog object, which means every downstream consumer benefits — not just Genie. Comment the tables. Comment the columns that carry business rules (konsolidiert, storniert, belegart, beleg_datum). Explain the trap in the comment itself, in one or two sentences.
2. Write a short Instructions block. Under Genie Space → Instructions → General instructions, put the definitional facts that do not fit into column comments: what "revenue" means, what "the last quarter" refers to, what to do when the question is ambiguous ("ask; do not guess"). Keep it short. Every character here lands in every prompt, so long instructions push out context you need for reasoning.
3. Move the critical numbers behind trusted UC functions. A trusted asset is a parametrised SQL function you register under Instructions → SQL functions. Genie will prefer calling the function over writing its own SQL, so the definition of "quarterly revenue" is now owned by the function, not by whatever Genie invents. Two rules matter here: the function's COMMENT is what Genie reads to decide when to call it, so a lazy comment means the function does not get picked. And every parameter also needs a COMMENT explaining what to pass — "fiscal year in the form '2025/26'" — because Genie has to figure that out too.
4. Add example queries. Under Instructions → Example SQL queries, put the two or three canonical patterns you expect to be asked most. Every question from the golden set that fails in the benchmark becomes an example here on the next iteration.
5. Run the golden set as a benchmark. For every question, ask Genie, compare to the expected reading. Not by hand — write a small notebook that iterates through the CSV and logs pass/fail. In our demo we ran the sixteen questions three times each. Not one, three. If the same question gives three different answers, you have a stability problem the pass/fail rate hides.
6. Fix the failures — always upstream. If a question fails, do not tweak the prompt. Fix it in the comment, the instruction, or the trusted function. Prompt-level fixes leak away the next time someone reopens the space. Upstream fixes stay.

Best practices we now enforce

Six habits, learned expensively:

• The critical subset is signed off in writing. Whoever owns the number reviews the expected answers and confirms them. If Controlling does not sign off on the expected revenue formula, you do not ship.
• Every trusted function has a good COMMENT. Not "returns revenue". Something like "consolidated net revenue per entity for a fiscal-year quarter. Only invoices, cancellations and their originals excluded, only consolidated entities." Written for a reader, not for the compiler.
• Trap questions are permanent. The unconsolidated-holding trap stays in the golden set forever. Every model swap, every schema change, every Genie version bump re-runs the trap.
• Three runs per question, not one. Determinism matters as much as correctness. If the same question gives different numbers on different attempts, the space is not production-ready.
• The golden set lives next to the code. In the same git repo as the trusted-asset SQL. Reviewed on every change. Not in a shared drive.
• Every quarter, the business adds two questions. New products, new entities, new reporting lines. The golden set decays if it is not maintained.

What comes next

If you already have a Genie space in front of executives, run the sixteen-question exercise this month. Draft it with two people from Controlling. Run it against the space in its current state. If the pass rate on critical questions is below 100%, you have work to do — and you now know where.

If you are about to spin up your first space, comment the tables first, write the golden set second, ship the space third. In that order.

Every article ends with a concrete asset — a template, a diagram, or a demo repository. No generic "what is RAG" content. Only what we have actually built, measured, and learned.

Free live webinar · 60 min · See agentic AI in action — live.

Three real customer stories, two live demos, and a five-step framework you can apply on Monday.

Germany's first GenAI Boutique, based in Magdeburg. Previously responsible for data and AI strategy at Volkswagen Commercial Vehicles. Author of Artificial Intelligence in the Automotive Industry.