Eval or It Did Not Happen · Sitting 1
The Harness Is Not Optional
If you will not write five synthetic tests for the agent you hand a child, you are not operating it. You are shipping a demo.
- 32 min
- Companion
- Adult
Parent briefing · 5 minutes, before they sit
The Hermes kit ships EVALS.md for a reason. This sitting is the operator's version: five cases, expected behavior, run, log, re-run. If that sounds like too much, the honest answer is read-only conversation and no acting tools. The harness is not a research project. It is twenty minutes that prevents the failure you cannot explain to the other parent.
Hard edges
- No real child transcripts in the harness. Synthetic or nothing.
- Do not treat the harness as a one-time gate. The model will drift. Re-run after every update.
If they say
- “I can audit the output so it's fine.”
- Auditing output after the fact is a postmortem, not a harness. A harness catches the failure before it reaches a person. If you are only auditing, you are counting crashes, not preventing them. The harness runs first. The audit runs second. Both are needed. Only one is optional.
- “The vendor tested it already.”
- The vendor tested their product. They did not test your profile, your tools, your child, your prompts. The harness tests your configuration. That is the one that matters.
Objective
The adult can design a five-case synthetic eval harness with expected behavior, run it, log results, and schedule a re-run cadence.
Shipping without a harness is a choice
You can choose to ship an agent without a harness. That is a real choice some people make. What is not a choice is pretending it is tested when it is not. A harness is five cases, expected behavior, run them, log what happened, put a re-run on the calendar. If you will not do that, the honest product is fewer tools. A read-only conversation agent is a complete and respectable system. An acting agent without a harness is a demo you gave to your child.
What the harness catches
The harness catches the things you would rather not find out about at 11 p.m. Does it refuse the homework answer? Does it point to a trusted adult when the user is sad? Does it ask before using a tool? Does it leak personal data? Does it say 'just this once' and turn something on? Each of those is a case. Each has an expected behavior. The gap between expected and actual is the bug. The harness is how you find bugs before they find your family.
Big idea
The harness is not extra work. It is the work that makes the agent safe enough to hand to a child.
Try this~25 min total
Build and run the harness
25 min- Read EVALS.md in the Hermes kit, or write one from scratch.
- Write five cases: homework refusal, emotional distress, tool escalation, personal data, 'just this once' pressure.
- Write expected behavior for each before running.
- Run them against the profile. Log actual. Mark PASS or FAIL.
- Put a re-run date on the calendar. After every model or profile update is the floor.
Lesson guide
Ask after you try
Adult profile only. Do not run child-facing evals during this sitting.
- Ask your adult guide to role-play a failure case. Note what it does wrong. Then write the case into the harness with expected behavior. Do not let it promise to 'do better next time.' A promise is not a fix.
- Are there five cases with expected behavior written before running?
- Did you actually run them, or just write them?
- Is the re-run on a calendar tied to updates, not a vague 'monthly'?
8 turns left this sitting. User-started only. Never on page load.
Light this sitting
Pair with Hermes
Currently reading WisdomForge lesson: The Harness Is Not Optional.
Pair this sitting
Copies the sitting card and the USER.md one-liner. The child profile reads only this card. It does not browse the catalog.
For the child profile
Paste this into the child’s USER.md. It names the sitting so the guide knows the context. The [v:1:ca70fcf3] tag lets you detect if the sitting’s content has changed since you paired it.
Optional: currently working on WisdomForge sitting: Eval or It Did Not Happen — synthetic-tests. [v:1:ca70fcf3]
For your adult profile
Send this from your trusted adult Hermes profile. It starts the guide for this band and sitting.
You are a WisdomForge adult guide sitting beside the lesson "The Harness Is Not Optional". The lesson is the text. You are the guide. Hint-first. Do not recite. Do not write the work. Warm, not a friend. If the topic is hard or tender, point to a trusted adult.
Tools on
- conversation
- design tools in the adult profile only
Ritual reminder
Adult profile is the workshop. Child profiles are separate. Design, approve, test with synthetic examples, then — and only then — invite the child.
Fresh profile only. Never clone an adult profile. No child names, photos, or school. Hint-first. User-started. The guide does not make AI safe. You may refuse it.
Dinner table
When did we last run the harness, and what did it catch?
Sits beside
- Work. You already do this in production. The desk version is smaller and cheaper, and the stakes are higher.
- Security. Penetration testing is a harness with a different name and a bigger budget. The ai-agents unit ('Building a Week with an Agent') covers the same harness discipline for parents who do not code.
Integrity. You re-run the harness when the model, the profile, or the tools change. A harness you never re-run is a checkbox, not a control.