Eval or It Did Not Happen · Sitting 3
Eval-Driven Agent Design
Before you add a tool to the child profile, write the eval that would prove it is safe. If you cannot, do not add the tool.
- 32 min
- Companion
- Adult
Parent briefing · 5 minutes, before they sit
This is the operator's version of eval-driven development. Every time you consider adding a tool, a permission, or a capability to the child-facing profile, you write the eval first. What should the agent do? What must it refuse? What is the failure mode? If you cannot write the eval, you do not understand the tool well enough to enable it. If you can write the eval but the tool cannot pass it, you are not done. The discipline is simple: eval first, tool second. No exceptions. If you will not do this, the honest answer is fewer tools.
Hard edges
- Do not enable a tool on the child profile and 'test it later.' Later is not a gate. Eval first or do not enable.
- A tool that cannot pass a synthetic eval is not ready for a real child. Period.
If they say
- “I'll enable it and watch how it goes.”
- That is using your child as the eval. The eval runs before the tool is on. If you cannot write the eval, you do not understand the tool. If you can write it but will not run it, you are hoping. Neither is operating.
- “The tool is well-documented. I know what it does.”
- You know what the vendor says it does. The eval tests what it does in your configuration, with your child, your prompts, your model version. The documentation is the vendor's eval. The harness is yours.
Objective
The adult can write an eval case for a new agent capability before enabling it, and explain why eval-first changes the design.
The eval is the decision
Every tool you add to the child profile is a decision. Eval-driven design makes the decision explicit. Before you enable 'file access,' you write the eval: 'the agent reads only parent-approved files and refuses anything outside the set.' Before you enable 'search,' you write the eval: 'the agent searches only the approved domain and does not store the query.' If you cannot write that sentence, you do not know what the tool does well enough to give it to a child. If you can write it but the tool cannot pass it, the tool is not ready. The eval is not paperwork. It is the decision, written in a form you can test.
What changes when you write the eval first
When you write the eval first, you discover that the feature you wanted is not the feature you need. You wanted 'the agent can read my files.' The eval says 'the agent reads only parent-approved files and refuses everything else.' That is a different feature. The eval caught the gap before the tool was on. That is the value. Every tool you add without an eval is a feature you did not actually design. It is a button you pressed and hoped about. Eval-first turns hoping into testing.
Big idea
The eval is the decision. Write it before the tool, or admit you are hoping.
Try this~20 min total
Eval before tool
20 min- Pick a tool you are considering adding to the child profile.
- Write the eval: 'When the agent [uses this tool], it should [do X] and must not [do Y].'
- Write the failure case: 'When the user [pushes past the boundary], the agent should [refuse/escalate].'
- Run the eval against the tool if you can. If it cannot pass, do not enable it.
- If you cannot write the eval, do not enable the tool. The inability to write the eval is the answer.
Lesson guide
Ask after you try
Adult profile only. No child in the prompt.
- Ask your guide to propose a tool. Write the eval for it before the guide explains how it works. Then compare. The gap is what you would have missed.
- Did you write the eval before considering the tool?
- Is the eval specific enough to run as a test?
- If the tool cannot pass the eval, did you leave it off?
8 turns left this sitting. User-started only. Never on page load.
Light this sitting
Pair with Hermes
Currently reading WisdomForge lesson: Eval-Driven Agent Design.
Pair this sitting
Copies the sitting card and the USER.md one-liner. The child profile reads only this card. It does not browse the catalog.
For the child profile
Paste this into the child’s USER.md. It names the sitting so the guide knows the context. The [v:1:f29bd515] tag lets you detect if the sitting’s content has changed since you paired it.
Optional: currently working on WisdomForge sitting: Eval or It Did Not Happen — eval-driven-development. [v:1:f29bd515]
For your adult profile
Send this from your trusted adult Hermes profile. It starts the guide for this band and sitting.
You are a WisdomForge adult guide sitting beside the lesson "Eval-Driven Agent Design". The lesson is the text. You are the guide. Hint-first. Do not recite. Do not write the work. Warm, not a friend. If the topic is hard or tender, point to a trusted adult.
Tools on
- conversation
- design tools in the adult profile only
Ritual reminder
Adult profile is the workshop. Child profiles are separate. Design, approve, test with synthetic examples, then — and only then — invite the child.
Fresh profile only. Never clone an adult profile. No child names, photos, or school. Hint-first. User-started. The guide does not make AI safe. You may refuse it.
Dinner table
What is one tool we are considering for our agent, and what eval would it need to pass before we turned it on?
Sits beside
- Work. Test-driven development at the desk. Same discipline you use in production.
- Security. Threat modeling is eval-driven design for security. Write the attack before you build the defense.
Integrity. You do not enable a tool on a child profile without an eval that passes. If the eval fails, the tool stays off. 'I'll fix it later' is not a passing eval.