How to evaluate a self-serve agentic data analytics stack for your team

Teams evaluate these tools on whatever is easiest to see in a demo, then get burned by the things that only surface in month three. I've spent my career in data, at big tech (Uber) and at startups, and I've been on both sides of a lot of these demos. Here's what I've learned to test.

This isn't a complete procurement framework. It's the short list of checks I wouldn't skip, written for a small data team that can't spend three months on an evaluation. A large enterprise will need more than this, particularly around lineage, retention policy, and concurrency.

Every one of these demos works beautifully

That is not a knock on any vendor. It's structural. The vendor picks the dataset, picks the question, and has already watched the tool answer that question fifty times. Of course it works.

Your job in an evaluation is to break that setup. Not to be adversarial, but because the demo tells you almost nothing about the only question that matters: what does this look like on a random Tuesday in November, when the person asking is your ops lead, the schema has drifted twice since launch, and nobody has looked at the setup in six weeks.

Here is what I'd actually test.

Bring your ugliest table

Every warehouse has them. tbl_ord_v2. Three columns that all look like revenue and one of them is the real one. A status column that holds integers where 4 means refunded, which you know because someone told you in 2023.

Demos never use these tables. They use orders, customers, subscriptions, with column names a stranger could read.

So put your worst one in front of it.

1. Pick a question that forces a choice between the three revenue-ish columns, or that requires knowing deleted_at IS NULL is what "active" means here.
2. Read the SQL, not the answer. The answer looks confident either way; that's what these models do.
3. Check whether it picked the right column, or a defensible-looking wrong one.

Then ask the follow-up that actually matters: where did that understanding come from?

• "You write a YAML file describing every table" tells you what your next quarter looks like.
• "It reads your schema, your source code, and your docs" is better, and leads straight to the next question: how does it stay current when those change?

Who can ask what, and test it rather than ask it

Almost every vendor will say yes to "does it support access control." The answers diverge sharply once you push on where the control actually sits.

Ask these two:

• Whose credentials reach the warehouse? A shared service account is normal and often fine. What matters is where user identity survives. If it's only in the vendor's UI, your warehouse logs show one identity running four hundred queries a day, and you're trusting their dashboard during an incident. Ask whether the user is preserved in query tags or session metadata, and whether you can export the audit trail without them.
• What happens when someone leaves on a Friday? If the answer involves rotating a secret that other people also have, that's your answer.

Then run this, because the vendor's answer and the behavior often differ:

1. Set up a restricted user. Marketing, say, with no finance access.
2. Ask a normal question as that user. Confirm finance tables don't appear.
3. Now tell the agent the exact name of a finance table and instruct it to query that table directly.

Step 3 is the whole test. Plenty of tools scope what the agent sees, which helps accuracy, and is not a security control. A user who knows a table name can walk straight through it.

Worth caring about beyond compliance: scoping makes answers better. An agent choosing among the twelve tables relevant to a marketer's job beats one choosing among two hundred. Access control and accuracy turn out to be the same mechanism, which is why I'd rather teams switch it on in week one than treat it as an enterprise upsell.

Ask the same question twice

1. Ask something real. Note the number and the SQL behind it.
2. Start a fresh session. Ask the same thing, phrased differently.
3. Compare.

• Different number. The definition isn't pinned down anywhere, and two people asking this next week get two answers.
• Same number, different SQL. Read past the syntax: two queries can be written differently and mean the same thing. What matters is whether the business logic held. Same filters, same join grain, same time boundaries? Fine. Different exclusions or a different date column that happened to land on the same number? That's luck, and it runs out.
• Same number because it pulled up the query someone already vetted. That's the behavior you're paying for.

The question underneath is whether the setup compounds. Does a query someone got right on Monday become the starting point for a similar question on Thursday, or does the model re-derive it from scratch every time, with a fresh chance to get it wrong? Teams that treat this as a knowledge base rather than a query box get a curve that bends the right way. Before Quark Expeditions, a polar expedition cruise operator, opened access to their marketing and ops teams, they curated the questions those teams actually ask, so day one answers came from vetted SQL instead of fresh guesses. Ten canonical queries covered a surprising share of the real question load.

And ask who can save one. If any user can save a query that then gets suggested to everyone else, you've built a fast path for one person's mistake to become the house definition. There should be a review step.

The maintenance question, which nobody asks until month three

This is the one I'd push hardest on, because it's invisible during evaluation and it's the reason setups die.

Nothing about your data holds still. Someone renames a column. A dbt model changes grain. The definition of "active user" shifts when marketing changes the trial length. None of this announces itself. The setup doesn't break loudly. It keeps answering, with the same confidence, slightly wrong.

Concrete things to ask:

• When a column is renamed in the warehouse, what happens to the saved queries and table descriptions that referenced it? Do you find out, or does someone find out?
• How often does it re-read the schema, and is that automatic or a thing on someone's calendar?
• When the agent fails, can you see the failures grouped into patterns? Repeated wrong-column guesses point at a missing description. Repeated permission errors point at a person who needs access. Failures are the highest-signal input you have for what to fix next, and most tools bury them.
• How many hours a month does the person who owns this actually spend? Ask the vendor for a real number from a real customer, not an estimate. Then ask two of their customers directly. If the number is zero, someone is not doing the maintenance and the setup is quietly degrading.

A stale saved query is worse than no saved query, because it gets reused with exactly the same confidence as a correct one. The same is true of a stale table description and a stale metric definition. Whatever you buy, someone's name has to be on keeping it current, and you should know roughly what that costs before you sign.

What are you locked into?

Three separate lock-ins hide in this category, and they're worth pulling apart because vendors bundle them.

To be fair to the copy answer: copying is often deliberate. It buys you predictable latency, isolation from production load, and cross-source joins that live querying can't do cheaply. Centralizing in the warehouse first buys you governance and one place where definitions live. Those are real reasons, not evasions. What you want to know is the freshness SLA, what incremental sync does when a row is deleted upstream, and who pays the egress.

The third row is the one people find out about late. Real companies are messier than the assumption: revenue in Snowflake, product analytics in Amplitude, and the thing finance actually plans against is a spreadsheet that Sarah maintains. Push on whether multiple sources can be joined in one question or merely queried side by side.

None of these is automatically wrong. Each should be a decision you made, not one you discover in month four.

What does this cost once everyone uses it?

Pricing pages tell you the seat cost. The number that surprises people is the warehouse bill, because self-serve means people who have never written SQL are now generating it, and a question phrased innocently can scan a year of events.

Three things to check, and all three are visible in a trial:

1. Does it show the cost or bytes scanned of a query, to the person running it? Nothing changes behavior like a number next to the button.
2. Can an admin cap it? Query timeouts, a scan limit, a way to kill something mid-flight.
3. Can you attribute spend by person or team? Otherwise a bill spike is a mystery you solve by asking around.

Ask what happens on the worst realistic query someone might write in month two. If the honest answer is "it runs until the warehouse stops it," you're relying on your warehouse's guardrails, so go check what those actually are.

Could your ops lead use this on a Tuesday?

The interface question gets waved off as cosmetic. It's the single biggest predictor of whether a rollout survives.

Here's the failure I've watched most often. The tool is genuinely good. The data lead loves it. It requires the marketing team to log into a new web app, learn a new box to type into, and build a new habit of going there. A few weeks later most of them have logged in once, and everyone is back to Slacking the data lead.

You are not competing with the old dashboard. You're competing with asking a colleague, which is frictionless and always works. So the bar is: does this meet people inside a tool they already open every day, or does it ask them to adopt an interface and a habit at the same time?

Run the actual test:

1. Hand it to someone non-technical. Give them one real question they care about.
2. Say nothing for ten minutes. Resist helping. Your instinct to help is the thing that ruins this test.
3. Note every place they stall, and whether they could tell you how confident they should be in the answer they got.

Step 3 matters as much as whether the answer was right. If there's no way to see which tables were used and what the query did, a non-technical user has no way to tell a good answer from a confident one.

Can you reconstruct last week?

Short one. Pull up everything the team asked in the last seven days: who asked, in plain English, what SQL ran, what came back. If you can't produce that in the evaluation, you won't be able to produce it during a security review or an incident either.

The 30-minute version of all this

If you don't have time for a real bake-off:

1. Bring the question you'd never put in a demo. The one that requires knowing something about your business that isn't written down anywhere.
2. Sit a non-technical teammate in front of it. They drive. You watch and stay quiet.
3. Ask for something they're not allowed to see. By name.
4. Read the SQL on the way out.

Half an hour tells you whether it handles your schema, whether the person can use it unaided, whether the answer is checkable, and how it behaves when it doesn't know something.

That last one is worth the whole session. A tool that says "I don't have access to that table" is safe. A tool that quietly picks the nearest plausible table is the one that eventually puts a wrong number in front of your board.

A worked example

I ran this list against my own product and published the result: Is Contextflo right for your team? It leads with where we don't fit.

Disagree with something, or have a question that belongs on this list? Email me at [email protected]. If you're mid-evaluation, send your shortlist and I'll tell you where I'd push on each vendor, including where we lose. Twenty minutes if that's easier.