How to Tell If an AI Tool Actually Works Before You Trust It

I almost rolled out a tool that was, in plain terms, guessing. It passed my test perfectly: every example, right. Then I pointed it at actual, messy, day-to-day data, and it fell apart so badly I had to read the number twice. The tool wasn’t broken. My test was. That gap is the most expensive mistake I see people make with AI right now: trust the demo, roll it out, find out weeks later it only ever worked on the easy stuff. Here is how to catch it before it costs you.

TL;DR

• I wanted to swap an expensive AI service for a cheaper one doing the same routine job. On the test I wrote, both scored 100%.
• On my real data, the good tool stayed reliable; the cheap one dropped to roughly a coin flip. The test wasn’t wrong, it was easy: I’d unconsciously fed it the obvious cases.
• The fix costs almost nothing: test on a slice of your own real, messy data, not on the demo.

The setup

One small, boring task in a side project I run: take a piece of incoming text and sort it into a category, thousands of times a day. It doesn’t need a genius behind it, so a cheap tool should handle it. Replace the pricier service, keep the same quality, pocket the difference. Before flipping the switch, I did the responsible thing and built a little test.

The test that told me what I wanted to hear

I wrote out a few dozen examples by hand, with the right answer for each, and checked how often each tool got them right. The cheap tool scored 100%, identical to the expensive one I was replacing. Something nagged, though. A perfect score on a real-world job usually means one thing: the test was too easy. So before rolling it out, I ran one more check.

The test that told me the truth

Instead of my hand-picked examples, I pulled a slice of actual data, the real stuff flowing through the system every day. Both tools ran against it quietly in the background. Here is what came back.

Fifty-two percent on a two-way call is a coin flip. The tool that aced my test was, on real data, barely doing better than guessing.

I did not roll it out. I kept the reliable tool and later found a cheaper option that did pass the real test. But that’s another post.

Why the test lied

The test wasn’t broken. It was unrepresentative, which is worse, because it looks fine.

A test you write yourself measures how well you understand the easy cases. Nothing more.

The reason is simple: when you write test cases by hand, you write the clear ones without realising it. You end up testing whether the tool can do the easy version of the job, and almost anything can do the easy version. Your actual work is not the easy version. It’s the half-finished sentence, the message that’s about two topics at once, the one dripping with sarcasm, the form somebody filled in wrong. That’s exactly where a weak tool collapses and a good one earns its price. My hand-written test had none of that, so it couldn’t tell the two tools apart.

The habit I keep now

I don’t trust demos or hand-written tests for this anymore. The rule is short: before you trust an AI tool with real work, test it on a slice of your own real data, not on the examples the vendor picked, and not on the examples you picked either. In practice, four steps, none of them technical:

1. Grab a sample of your real work. A few dozen real cases from last month, the normal ones, not the tidy ones. Write down what the right answer should have been.
2. Run the tool against that sample quietly, where its answers don’t actually do anything yet.
3. Compare it to what you do today: your current tool, or your team doing it by hand.
4. Only roll it out if it holds up on the hard cases, not on the demo.

The cost of a real test is an hour. The cost of skipping it is weeks of cleanup.

What this means if you’re the one signing off

You don’t need to understand the model. You need to refuse one sentence: “It scored great in the demo.” Of course it did: the demo was built to score great. The only question that protects you is: how does it do on our data, on the cases we actually struggle with? If the vendor can’t or won’t run that test, that itself is your answer. The one number worth asking for: accuracy on your own data versus accuracy on the demo. If those two are far apart, you’ve found the risk before it found you.

The hard cases are the entire reason you’re paying for a good tool in the first place. So before you trust any AI tool with real work: when did you last check it against the messy, real thing, instead of the version built to impress you?