Contextflo is now SOC 2 Type II compliant. See more

Giving business context to AI agents - does it help? Does it save money? Let's find out

We took a realistic payments dataset built by Adyen and Hugging Face and ran the same AI agent on it with and without context.

Jump to a section

The takeaway

Giving an AI agent the company's docs was no better than giving it nothing: 40% of tries right without them, 36% with. What worked was four short notes, the kind of thing a teammate tells a new hire. They took it to 76% right, at about a quarter of the cost per answer. If your company has docs, it probably has the same gaps. The rules that answer questions live in people's heads.

Benchmark setup

We ran a Claude Haiku 5.5 agent on a public test dataset that mimics a payments company, with realistic, messy data, and compared it with and without context (the company's docs, plus notes we wrote).

ModelClaude Haiku 5.5
DatasetDABstep, by Adyen and Hugging Face
Exploration25 hard questions, picked by a script with fixed rules
Test set42 new questions, a mix of question kinds
Setups comparedOnly the tables, + the company's docs, + docs and four notes from exploration
Tries3 per question, in every setup

We started by exploring the data with a small set of hard questions. As expected, the agent did poorly with only the tables, and the docs didn't make it any better. So we read its wrong answers and wrote the four notes.

Then we froze the notes and tested on a fresh set of questions they were never written from, a wider mix where some turned out easier than others. The numbers in the takeaway come from that test.

You don't need to know anything about payments to follow this. Payments is just a stand-in for any company's data: tables, some docs, and rules only the team knows.

Why we picked this dataset

Most AI tests use clean, made-up puzzles. Real company data isn't like that. It has a price list nobody fully understands, docs that are mostly right, and things "everyone just knows".

DABstep is a public test built by Adyen, a large payments company, and Hugging Face. It describes a fake company with realistic data and realistic mess. The questions are the kind a finance or ops lead would ask. The right answers are kept secret, so nobody can cheat.

We loaded the data into a database (BigQuery) exactly as it comes. No cleaning, no renaming.

What the dataset contains

Picture a company that handles card payments for online shops. Shoppers pay by card, and the company charges the shop a small fee on every payment.

TableIn plain wordsRows
paymentsEvery card payment in 2023: which shop, how much, which card, how the shopper paid, and whether it turned out to be fraud138,236
feesThe price list: what each kind of payment costs the shop1,000
merchant_dataThe 30 shops: what they sell and how their account is set up30
two small lookup tablesShop categories and countriessmall

The shops have made-up names like Rafa_AI, Crossfit_Hanna and Golfclub_Baron_Friso. The card brands are made up too: GlobalCard, NexPay, TransactPlus, SwiftCharge.

The company also comes with two documents: a manual on how fees and fraud work, and a guide to the payments table. They're decent docs. In the setups with docs, the agent gets all of it, as written:

  • What the tables and columns mean: all five tables and most of their columns.
  • Seven business rules from the manual: how a fee is worked out (a fixed amount plus a percentage), that a blank (null) in a price row means "any", that monthly volume and fraud are counted per calendar month, and what local payments, account type, way of paying and capture delay mean.

That's real business context, not just a schema. Everything the agent was given is on GitHub.

The challenge of the price list

The price list has 1,000 rows. Each row describes a kind of payment and what it costs, like a line in a shipping rate table ("under 2 kg, to Europe, express: €12").

Here is one row, exactly as it sits in the data:

Loading code sample...

In plain words: a debit card payment on TransactPlus, at a shop in one of those eight categories, paid in one of two ways (C or B), costs 10 cents plus 0.19% of the amount.

To price a payment, you find the rows that describe it and use their prices. Notice the blanks ([] and null), though. What do they mean? The data doesn't say, and the manual only covers one of the two. Hold that thought.

The questions

The test has 450 questions. We used them in two rounds: an exploration round of 25 hard questions, where we read the wrong answers and wrote the notes, and a held-out test of 42 new hard questions the notes never saw (more on that below).

Neither set was picked by hand. A script chose both, using fixed rules (no questions where "not applicable" is the answer, a mix of question types), so anyone can rerun it and get the same questions. Each has a single right answer: a number, a letter, or a short list. Some examples, in plain words:

  • "If we changed this one price, how much more or less would Rafa_AI have paid in July?"
  • "For a €10 credit card payment on GlobalCard, which way of paying is the most expensive?"
  • "If Rafa_AI got its fraudulent shoppers to pay a different way, which way would cost the least?"

Every business asks questions like these about its own numbers: what did this cost, what if we changed that, where should we steer customers.

How we ran it

  • Same agent every time. Same model, instructions and tools. The agent connects to the database through Contextflo, which lets it see the tables, read any notes we give it, and run queries. It can't change anything.
  • In exploration, one note at a time. Six setups: only the tables, then the docs, then each note added in turn, so you can see what each one did (75 answers per setup).
  • Every try starts fresh, with no memory of the last one.
  • Every run is published, including the wrong ones.

Results from exploration (25 questions)

What the agent hadQuestions right on all 3 triesTries rightCost per try
Only the tables3 of 2514 of 75 (19%)~6 cents
+ the company's docs3 of 2512 of 75 (16%)~2 cents
+ note 1: a blank means "any"7 of 2523 of 75 (31%)~3.5 cents
+ note 2: add up every price that applies14 of 2544 of 75 (59%)~1.5 cents
+ note 3: "monthly" means calendar month13 of 2545 of 75 (60%)~1.4 cents
+ note 4: how to answer "which is cheapest"14 of 2547 of 75 (63%)~1.2 cents

How to read this: there are 25 questions, each tried 3 times. The percentages count tries, not questions. The 3 tries show whether the agent is repeatable, so they don't count as 75 separate tests. The first column is the stricter measure.

How it's scored: where someone had submitted the exact same answer to the official scoreboard, we use the scoreboard's verdict. Otherwise we do a strict check against an answer sheet we rebuilt from the scoreboard (the real answers are secret). More on that below.

Cost is what the AI model charged per try. The database bill is covered further down.

Docs alone: same accuracy, lower cost

Adding the manual didn't change accuracy. 19% vs 16% is too small a difference to count with this many questions. It did cut the cost per question by about two thirds.

Without docs, the agent spends most of its effort working out what the columns mean, running query after query to guess. With docs it stops guessing what the data is. But it still doesn't know how the business uses it. Knowing what a column means isn't the same as knowing how to answer the question.

The four notes

Each note came from reading the agent's wrong answers and asking: what would a teammate have told it? Two add something the docs don't have. Two clarify something the docs already say, because the agent had the rule and still got it wrong.

NoteNew or clarified?
1. A blank means "any"Clarified. The manual covers null, but not the empty lists [] the price list also uses
2. Add up every price that appliesNew. The docs never say one payment can match several prices
3. "Monthly" means calendar monthClarified. The manual says it; the note gives the exact steps
4. How to answer "which is cheapest"New. A how-to for one type of question, not a business rule

Does a blank mean "none" or "any"?

A blank in a price row means "any". A blank card type means any card, not no card.

This one clarifies the manual. The manual says it, but only for null. The card type, shop category and way-of-paying columns are never null: they use an empty list [] for "any". The note spells that out for both.

Without this note, the agent read blanks as "doesn't apply", so it ignored most of the price list and came up with fees that were too low. It went from 16% to 31% with it.

At your company: a subscription with an empty end date. Is it cancelled, or does it never end? Everyone on the team knows. The table doesn't say.

One price, or all of them?

One payment can match several rows of the price list, and it pays all of them.

This one is new. The manual explains how one row's price is worked out, but never says a payment can match more than one.

Without this note, the agent found one matching row and stopped. It's like a restaurant charging a table the service charge but forgetting the large-group charge that applies too. With it, accuracy went from 31% to 59%, the biggest jump, and runs got about 40% faster because the agent stopped going in circles.

At your company: does a partly refunded order still count as revenue? Does a customer with two discounts get both? One sentence, and almost every number changes.

Which month is "monthly"?

Some prices depend on how much a shop sold that month, like a bulk discount. "Month" means January, February and so on, not the last 30 days, and you count money, not number of payments.

This one clarifies the manual, which already says it. We restated it with the exact steps so it couldn't be misread. Accuracy barely moved (59% to 60%), but runs got cheaper.

At your company: is monthly revenue the calendar month or the last 30 days? Is an "active user" counted per day or per week?

How to answer "which option is cheapest?"

For questions like "if we moved shoppers to another way of paying, which is cheapest", work it out over the shop's actual payments, one option at a time.

This one is new, and it's more "how to answer this kind of question" than a fact about the business. It's the closest we came to giving away the answer, and its gain (60% to 63%) is too small to count. We kept it in and we're flagging it.

When context makes it worse

Note 2 ("a payment pays every price that applies") is right for nearly every question, and wrong for one.

Before the note, the agent got the "most expensive way to pay for a €10 GlobalCard payment" question right on all 3 tries. After it, 0 out of 3. In its work log, the agent works out the right answer, then changes it to follow the note.

Agents trust what you tell them more than their own reasoning. One note that's too broad can break something that used to work. Treat these notes like code: review them, test them, and watch what changes.

Our first number was wrong

Our scoring first said 77%. That was too generous.

The right answers are secret, so we built our own answer sheet from the public scoreboard: if lots of people submitted an answer and it was marked right, we counted it as right. That sheet was too forgiving on the "how much more would they pay" questions. It accepted -9.4166 when the real answer is -9.42, and the official scoring marks that wrong.

So we checked all 450 of our tries (6 setups, 75 tries each) against how the official scoreboard marked that exact answer, wherever someone had submitted it before. Under that check:

  • every setup drops by about the same amount
  • the gain from the notes holds (16% to 63%)
  • the headline is 63%, not 77%

Several of those "how much more" questions turn out to be wrong in every setup, in the same way. That points to a rule we haven't written a note for.

Testing on new questions

The notes came from reading wrong answers on these same 25 questions. Fair question: did we just teach it these 25?

So we ran a held-out test. Before running anything, we froze the notes, the docs, the instructions and the scoring rules. A script picked 42 new hard questions: 3 from each of the test's 14 question types. Ten of those types appeared in exploration; four never did. Same agent, 3 tries each, three setups.

What the agent hadQuestions right on all 3 triesTries rightCost per try
Only the tables13 of 4250 of 126 (40%)~5 cents
+ the company's docs9 of 4245 of 126 (36%)~1.6 cents
+ the four notes31 of 4296 of 126 (76%)~1.3 cents

Scored the same way as exploration. Most answers had already been submitted to the official scoreboard by someone, so we use its mark. The rest had never been submitted, so we checked them against our rebuilt answer sheet: 12 of 126 tries with only the tables, 19 with the docs, 2 with docs and notes. Even if we marked every one of those the wrong way, the result holds: docs and notes would still have at least 94 of 126 right, against at most 62 with only the tables and 64 with the docs.

The gain held, and it came out bigger than in exploration. The docs again didn't help accuracy but cut the cost. The notes nearly doubled the right answers.

Split by question type:

What the agent hadTypes from exploration (30 questions)Types it never saw (12 questions)
Only the tables26%75%
+ the company's docs26%61%
+ the four notes70%92%

The four new question types turned out easier, since the agent got 75% of them with nothing but the tables. Most of the gain is on the familiar types: same kinds of questions, but new shops, months and amounts the notes never saw. A note is a rule, so it works on the next question of that kind too.

What's still wrong

On the held-out test, with docs and notes, 11 of 42 questions weren't right on every try. They fall into a few groups:

  • "How much more would they pay if this price changed" (2 questions): wrong in every setup, every try. We saw the same thing in exploration. Something none of the four notes covers yet.
  • "Move the fraudulent shoppers to a different way of paying" (3 questions): wrong in every setup, every try. Note 4 was meant for this kind of question, and it isn't enough.
  • "What's the most expensive category / way of paying, in general" (4 questions): the notes made two of these worse. One went from 2 of 3 right with docs to 0 of 3, another from 2 of 3 to 0 of 3. It's the same kind of question that broke in exploration (above): the note about adding up every price is right for most questions and wrong for "in general" questions.
  • Two others slipped slightly once the docs were added, and the notes didn't win them back: 3 of 3 with only the tables, 2 of 3 after; and 1 of 3, then 0 of 3.

We stop at four notes on purpose. Getting the last third right would mean writing the answers into the notes, and that's cheating the test, not helping the agent.

What it costs

Every question comes with two bills: the AI model and the database.

  • AI model: from about 5 to 6 cents a question with nothing to about 1 cent with the docs and notes. Per right answer, on the held-out test: 12 cents with only the tables, under 2 cents with docs and notes.
  • Database: we measured what BigQuery billed for every query. On the held-out test, the average per try was about 43 MB with only the tables, 51 MB with the docs and 33 MB with docs and notes. That's a fraction of a cent on this small dataset. With notes, the agent ran fewer exploratory queries and read the least data. We didn't test a larger database, so we're not claiming how that scales.

If you're doing this at your company

Your docs say what the data is. The rules for answering questions live in people's heads, so write those down first. Start from the agent's wrong answers: each one points at something nobody wrote down. Review the notes like code. Good notes get you more right answers and a smaller bill.

Receipts

Every held-out question and every try is right below this post: what the agent ran, what came back, and how each answer was marked. The full, untrimmed runs, the code, the notes and how to rerun it are on GitHub.

  • Data: DABstep by Adyen and Hugging Face, free to reuse (CC BY 4.0), loaded as-is.
  • Model: Claude Haiku 5.5, the same in every setup.
  • Questions: 25 hard questions for exploration, picked by a script with fixed rules: hard only, no "not applicable" or empty answers, only questions where many scoreboard submissions agree on the answer, spread across question types. 42 more for the held-out test, picked the same way, 3 from each of 14 question types, none overlapping exploration.
  • Frozen before the held-out test: notes, docs, instructions, tools, model, question list and scoring rules, each pinned by a file hash. We snapshotted everything the agent could see at the start and end of every batch, and they matched.
  • Scoring: the official scoreboard's mark on each exact answer where one exists, otherwise a strict check against our answer sheet.