This is the checklist we walk through with every partner team before they ship an AI feature to children. None of it is theoretical. Every item exists because we have seen the absence of it cause a real incident in a real product.
Identity and age
- Verified age band attached to every request, not inferred from a profile name.
- Separate session storage per child profile, never shared between siblings or with parents.
- Parental consent stored, dated, and revocable, with an audit trail.
- No persistent device-level identifiers used to retarget children across products.
Inputs
- Prompt sanitisation that strips role-play escape attempts and instruction injection.
- Intent classifier that flags self-harm, sexual, violent, financial, and medical topics before the model runs.
- Length, rate, and frequency limits per child to catch loop and dare behaviour.
- Voice and image inputs scanned with the same rigour as text, not waved through.
Generation
- Model choice per task, with the ability to swap providers without a rebuild.
- Structured outputs wherever possible, so you parse fields instead of trusting prose.
- Streaming-aware moderation that can stop a response mid-token if it drifts unsafe.
- No tool, browser, or code execution access unless every step is policy-checked.
Outputs
- Output classifier independent from the generating model.
- Soft rewriting of borderline answers into age-appropriate language rather than dead-end refusals.
- Citations required for factual claims at the older age bands.
- Tone checks that catch flattery, manipulation, and parasocial drift.
Operations
- Per-turn audit log with model, prompt, response, scores, and policy decisions.
- Human escalation path for high-risk turns, with a real SLA.
- Regression suite that runs on every model and prompt change, kid-specific.
- Parent-visible activity view with the ability to delete history and revoke access.
- Kill switch per model and per feature, not just per environment.
Governance
- Model card and DPIA written before launch, not after the first incident.
- Red-team report from someone outside the team that built it.
- Clear retention, deletion, and training-opt-out policy in plain language.
- Compliance mapping against COPPA, GDPR-K, and the EU AI Act, kept up to date.
If you cannot tick every box, you do not need to delay launch. You need to know which boxes you are choosing not to tick, why, and what your mitigation is. That single act of writing it down is what separates a serious kids product from a hopeful one.