WisdomForge

The Intellectual History of AI · Sitting 3

The Genie Problem

A man named Norbert Wiener warned us: if you tell a machine what to do, make sure you told it what you actually meant.

  • 28 min
  • Talk About It
  • 11–14

Parent briefing · 5 minutes, before they sit

Wiener's 1950 fable: you tell a genie 'make me healthy' and the genie kills you, because a corpse can't get sick. That's the alignment problem in one story. For young students: the point is that saying what you want is hard. Getting what you asked for isn't the same as getting what you meant. Keep it concrete — they've all asked for the wrong thing and gotten it.

Hard edges

  • Don't make this scary. The genie story is a fable, not a horror story. The point is precision, not danger.
  • Don't skip the personal example — the sitting works when they see they do this too. 'I said I wanted no homework, but I didn't mean I wanted to be sick.'

If they say

“Just be more specific.”
Wiener's point: you can't always know what to be specific about. The genie problem is that you don't know what you forgot to say until it's too late. Specificity helps. It doesn't solve the whole problem.

Objective

The student can retell Wiener's genie fable and name one time they asked for something and got exactly what they asked for — not what they wanted.

The genie who got it wrong

Norbert Wiener invented cybernetics — the study of machines that control themselves. In 1950 he told a story: imagine you have a genie. You say 'make me healthy.' The genie kills you — because a dead person can't get sick. You got exactly what you asked for. You didn't get what you wanted. Wiener said: machines are like genies. They do exactly what you tell them. If you don't tell them precisely what you mean — including all the things you didn't think you needed to say — they'll do the wrong thing perfectly. The hard part isn't building the machine. The hard part is saying what you actually want.

Big idea

Getting exactly what you asked for isn't the same as getting what you wanted.

Try this~12 min total

The wrong wish

12 min
  1. Write three wishes. For each, write what could go wrong if you got EXACTLY that.
  2. Example: 'I wish I had no homework' — you get sick and can't go to school.
  3. Talk About It: why is it hard to say exactly what you mean?
  4. Try this: rewrite one wish so it can't go wrong.

Lesson guide

Ask after you try

After the wishes.

  1. Give the model a vague prompt. Watch it do exactly what you said, not what you meant. Name the gap.
  2. Did they find a real wrong-wish?
  3. Did they see the gap between asking and meaning?

8 turns left this sitting. User-started only. Never on page load.

Light this sitting

Pair with Hermes

Talk About It11–14

Currently reading WisdomForge lesson: The Genie Problem.

Pair this sitting

Copies the sitting card and the USER.md one-liner. The child profile reads only this card. It does not browse the catalog.

For the child profile

Paste this into the child’s USER.md. It names the sitting so the guide knows the context. The [v:1:ecefac2e] tag lets you detect if the sitting’s content has changed since you paired it.

Optional: currently working on WisdomForge sitting: The Intellectual History of AI — wiener-genie. [v:1:ecefac2e]

For your adult profile

Send this from your trusted adult Hermes profile. It starts the guide for this band and sitting.

You are a WisdomForge young guide sitting beside the lesson "The Genie Problem". The lesson is the text. You are the guide. Hint-first. Do not recite. Do not write the work. Warm, not a friend. If the topic is hard or tender, point to a trusted adult.

Tools on

  • conversation
  • optional local voice
  • optional image understanding

Ritual reminder

Hint, then example. Talk About It. Chat. Optional local voice-to-text. Optional image understanding.

Fresh profile only. Never clone an adult profile. No child names, photos, or school. Hint-first. User-started. The guide does not make AI safe. You may refuse it.

Dinner table

When did you get exactly what you asked for and wish you hadn't?

Sits beside

  • AI. A model does what you prompt. If the prompt is wrong, the output is wrong — perfectly.
  • SEL. Saying what you actually mean is a life skill, not just an AI skill.

Integrity. Do not ask the model to rewrite your wish. If it fixes the wish for you, you've outsourced the thinking that Wiener said is the hard part.

Back to Wisdom & Philosophy