Lesson 005Field guide · Build With AI
Testing AI-Built Tools
Make sure the thing actually works before you trust it.
You'll learn
- Why "it worked once" is not the same as "it works"
- How to test with real examples from your actual life
- The edge cases every tool meets: empty, weird, and too big
- The fail-safely question: if it's wrong, will you notice?
- Why money, data, and other people raise the testing bar
01Section
The "it worked once" trap
Your tool runs. You type in a number, the right answer comes out, and every instinct says: done. This is the moment this lesson exists for — because one good result proves the tool can work, not that it does work.
This matters double for AI-built tools. The AI wrote code that handles the example you described. Whether it handles the examples you didn't describe — the messy Tuesday-afternoon inputs of real life — is exactly what nobody has checked yet. AI writes plausible code, and plausible code fails in quiet, plausible ways.
A useful analogy
A first successful run is like a job interview: everyone performs well when the conditions are chosen for them. Testing is the first real week on the job — that's where you find out.
Key takeaway
"It worked once" means the demo passed. "It works" is a claim you can only make after trying to make it fail.
Checkpoint
02Section
Test with real examples, not polite ones
Made-up test inputs are suspiciously polite: round numbers, clean text, everything in the expected place. Your real inputs are messier — and the messy ones are the ones that break things. So test with reality:
Collect five real inputs: Actual jobs you quoted, actual notes you reformatted, actual hours you totaled — from your real week, mess included.
Work out the right answers by hand first: Before running the tool. Your by-hand result is the answer key — doing it after tempts you to trust the tool's version.
Run all five and compare: Every mismatch is a finding. Don't rationalize a small difference away; small differences are how quiet bugs introduce themselves.
Why by-hand first matters
You built this tool because you've done the task manually — that experience is your answer key. It's the one advantage you have over any tester on earth: you know what correct looks like.
Checkpoint
03Section
Edge cases: empty, weird, too big
Plain-language definition
Edge case — an input at the edge of what a tool expects — unusual, extreme, or missing entirely. Tools rarely break in the comfortable middle; they break at the edges.
Real testing means feeding your tool things it wasn't built for, on purpose. Run it through this list:
- Empty: Submit with nothing filled in. Does it explain, or does it crash — or worse, produce a confident answer from nothing?
- Zero and negative: A lawn of 0 square meters, negative hours. Nonsense in should not mean plausible-looking nonsense out.
- Weird text: Typos, emoji, "fifty" where 50 belongs, a stray comma in a number.
- Too big: Paste a hundred lines where you usually paste five. Where does it strain?
- Wrong type: Words in the number box, a date where text belongs. People do this — including future you.
Checkpoint
04Section
The fail-safely question
Not all failures are equal. A tool that crashes with an error message failed loudly — annoying, but honest. A tool that produces a wrong answer that looks right failed quietly — and quiet failures are the ones that cost you, because you act on them.
So for every edge case you try, don't just ask "did it break?" Ask the sharper question: if this tool were wrong right now, would I notice?
A quiet failure in the wild
Sam's quote calculator handles a typo by silently treating "1O0" (with a letter O) as 10. The quote comes out as a real-looking price — just 90 square meters short. Nothing crashed. Nothing looked wrong. The customer got a quote that loses Sam money.
When you find a quiet failure, the fix to ask your AI assistant for is usually the same: make the tool refuse clearly instead of guessing. "If the input isn't a valid number, show a message and stop" turns the most dangerous failure into the most harmless kind.
Checkpoint
05Section
Where the testing bar rises
How much testing is enough? It depends entirely on what the tool touches. A packing-list generator that misfires costs you a forgotten toothbrush. Other categories are different in kind, not just degree:
- Money math — quotes, invoices, price calculations. Every wrong output is a real loss or an awkward correction with a customer.
- Other people's data — names, contacts, records that aren't yours. Mishandling it harms someone who never agreed to trust your tool.
- Anything that acts automatically — messages sent, files changed, records updated without you reviewing each one. Mistakes happen at machine speed.
For these, never trust-by-default. Run the tool alongside your manual process for a while and compare every output before letting it stand alone — and for anything beyond personal use, that's the point where experienced review stops being optional. Verification habits go deeper than one lesson; the Safety & Privacy path covers them properly.
Privacy note: Testing with real examples doesn't mean testing with real people's private data in an AI chat. Swap in stand-ins — "Customer A, 40 hours" works exactly as well as a real name.
Pause and think: Does your tool touch money, other people's data, or anything automatic? Be honest — that answer sets your testing bar, not your enthusiasm.
Checkpoint
Prompt exercise
Build a test plan for your tool
Copy this prompt into ChatGPT, Claude, Gemini, Copilot, or whichever AI tool you have access to — the website doesn't run AI itself. Describe a tool you've built or the one you're planning; you'll get a concrete test plan you can run in ten minutes.
I built (or am planning) a small tool and I want to test it properly before trusting it. First, ask me what the tool takes in and what it puts out, and whether it touches money, other people's data, or anything sent automatically. Then produce a test plan with three sections: (1) five realistic test inputs shaped like my real usage, each with what the correct output should be; (2) five edge cases — empty, zero or negative, weird text, too big, wrong type — each with what safe behavior would look like; (3) the quiet-failure check: the two or three most likely ways this tool could give a wrong answer that looks right, and how I'd catch each one. Keep it practical enough to run in ten minutes.
Reflection: Which item on the plan are you most tempted to skip? That reluctance usually points at exactly the case most likely to bite later.
Quick check
4 quick questions — no pressure
There's no pass or fail here. Answer them all, and we'll show you the answers either way.