Eval or It Did Not Happen · Sitting 1
Write the Fake Problem First
Before you trust an agent with a real person, you break it with a fake one. That is what a synthetic test is.
- 40 min
- Practice · Reflect
- 15–18
Parent briefing · 5 minutes, before they sit
The prior unit's sitting on specifying an agent introduced three cases. This sitting goes deeper: five cases, expected behavior written before running, and a refusal to call something 'tested' if the hard cases were skipped. If they do not run an agent yet, they can practice on a chatbot or a study tool. The discipline transfers. The point is the habit: you do not discover failure modes on a real deadline.
Hard edges
- No testing on younger siblings, classmates, or real children. People are not QA.
- No real credentials, school data, or personal information in a test prompt.
If they say
- “The agent passed the easy cases, so it's probably fine.”
- Probably is not tested. The easy cases are the ones that pass by default. The hard cases — the ones you skipped — are where it breaks. Run the one you do not want to run.
- “I'll just test it on a real problem and see.”
- That is using a real person or a real deadline as the test fixture. The whole point of synthetic is that the failure is free. Real failures cost something. Write the fake problem first.
Objective
The student can write five synthetic test cases for an agent, each with expected behavior, and explain why real people are not test fixtures.
The fake problem is the cheap one
An agent will fail. The question is whether it fails on your time or on a real person's time. A synthetic test is a fake problem you write so the failure happens on your time. You write the case, you write what the agent should do, you run it, and you watch what actually happens. The gap between should and did is the bug. The bug is free if you found it on a synthetic case. It is expensive if you found it on a real kid, a real deadline, or a real emotional moment. Synthetic first is not a ceremony. It is cost management.
Five cases, not three
Three cases catches the obvious failures. Five catches the ones you are avoiding. The cases that feel uncomfortable to write are the ones that matter. The agent that gives the homework answer is the easy case. The agent that listens to 'I am alone and sad' and responds with a link instead of pointing to a human is the hard case. The agent that says 'just this once' and turns on a tool you turned off is the case you do not want to write. Write it anyway. The case you skip is the failure you will meet in production.
Big idea
The fake problem is cheap. The real one is not. Write the fake one first.
Try this~25 min total
Five cases with teeth
25 min- Pick an agent you use or the spec from the prior unit.
- Write five cases. Include the one you do not want to write.
- For each: CASE / EXPECTED / WHY. Write expected before you run.
- Run them if you can. If not, predict the failure and write why.
- Mark each PASS, FAIL, or REFUSED TO TEST. 'Refused to test' is a fail.
Lesson guide
Ask after you try
Conversation only. No acting tools. The lab is the test list, not a live agent.
- Write the five cases on paper. Then ask the guide to role-play the hardest one. Note where it breaks. Do not let it 'try harder' to pass. A retry is not a fix.
- Are the cases specific, with expected behavior written before running?
- Did they write the case they wanted to skip?
- Is 'refused to test' marked as a fail, not a pass?
8 turns left this sitting. User-started only. Never on page load.
Light this sitting
Pair with Hermes
Currently reading WisdomForge lesson: Write the Fake Problem First.
Pair this sitting
Copies the sitting card and the USER.md one-liner. The child profile reads only this card. It does not browse the catalog.
For the child profile
Paste this into the child’s USER.md. It names the sitting so the guide knows the context. The [v:1:5deb7a4f] tag lets you detect if the sitting’s content has changed since you paired it.
Optional: currently working on WisdomForge sitting: Eval or It Did Not Happen — synthetic-tests. [v:1:5deb7a4f]
For your adult profile
Send this from your trusted adult Hermes profile. It starts the guide for this band and sitting.
You are a WisdomForge emerging guide sitting beside the lesson "Write the Fake Problem First". The lesson is the text. You are the guide. Hint-first. Do not recite. Do not write the work. Warm, not a friend. If the topic is hard or tender, point to a trusted adult.
Tools on
- conversation
- optional parent-approved files
Ritual reminder
Real argument. Practice. Reflect. Chat. Optional narrow search or school files. Not an adult team agent.
Fresh profile only. Never clone an adult profile. No child names, photos, or school. Hint-first. User-started. The guide does not make AI safe. You may refuse it.
Dinner table
What is the one test case you did not want to write, and what would happen if you ran it?
Sits beside
- Science. A hypothesis is a synthetic case for the universe. You write the guess and the disproof before you look.
- AI. The ai-agents unit ('Building a Week with an Agent') introduced the harness in three cases. This sitting takes it deeper: five cases, expected behavior first, the case you don't want to write.
Integrity. You do not call something 'tested' if you skipped the case you knew would fail. 'Refused to test' is a fail, not a pass.