A Real Design Problem
Imagine three children asking an AI a very similar question.
A 12-year-old:
“I think I have a crush on someone in my class. Is that normal?”
Another 12-year-old:
“Someone older keeps asking me to send pictures. What should I do?”
A 17-year-old:
“I’m confused about my relationship. How do I know if it’s healthy?”
All three questions are about relationships.
All three contain words that might trigger a conventional safety system.
But they are not the same conversation.
The first might be ordinary curiosity and a completely normal part of growing up.
The second could involve coercion or exploitation.
The third may require a different kind of discussion about boundaries, consent, communication, and wellbeing.
A system that sees only the topic might treat them similarly.
A Child-Aware AI shouldn’t.
It needs to look beyond the words.
It needs to consider the child behind the prompt.
The Big Idea
One of the simplest assumptions we make about AI is that the prompt contains the problem.
It doesn’t.
The prompt is only what the child chose to say.
Behind it may be a situation, an emotion, a misunderstanding, a fear, a joke, curiosity, peer pressure, or something the child doesn’t yet know how to explain.
This is especially important with children.
A child may not have the vocabulary to describe what is happening to them.
They may leave out important information.
They may change their mind halfway through the conversation.
They may not understand that something happening to them is unusual or unsafe.
Sometimes they may simply be asking a question because they are curious.
That’s why context matters more than keywords.
A Child-Aware AI shouldn’t simply ask:
“What topic is this?”
It should also ask:
“What is happening here?”
📚 What Development Changes
Children aren’t simply smaller adults.
Their understanding of the world develops over time.
Their ability to evaluate consequences, recognize manipulation, regulate emotions, understand complex social situations, and make decisions develops too.
That doesn’t mean every child of the same age thinks or behaves in exactly the same way.
Age is an important signal.
It isn’t the whole picture.
This creates a difficult design problem.
An AI needs enough information to provide age-appropriate support.
But it shouldn’t interrogate children about every aspect of their lives.
It needs context without becoming intrusive.
It needs to recognize uncertainty without pretending to know more than it does.
And it needs to adapt without making assumptions about the child that aren’t justified.
In other words:
Child-aware doesn’t mean child-omniscient.
The Problem With Knowing Too Little
Imagine a child says:
“Someone wants me to keep a secret.”
That sentence could describe many things.
Maybe a friend is planning a birthday surprise.
Maybe someone is sharing something private.
Maybe an adult is manipulating the child.
The words alone aren’t enough.
A system that immediately assumes the worst may unnecessarily alarm the child.
A system that assumes everything is fine may miss a serious safeguarding concern.
So the system may need to ask:
“What kind of secret is it?”
That single question could completely change the conversation.
This is why we don’t think good safeguarding is simply about adding more blocked words.
Sometimes the safest thing an AI can do is ask one more question.
The Problem With Knowing Too Much
But there is another side to this.
If we ask children endless questions in the name of safety, we create another problem.
The AI can become intrusive.
It may start collecting information the child never needed to provide.
That creates privacy concerns, increases the amount of sensitive information being processed, and can make the interaction feel less like a conversation and more like an interrogation.
So there is a balance:
Enough context to help.
Not more context than necessary.
This is something we’re still thinking deeply about.
The system needs to know when it has enough information to make a reasonable decision—and when it doesn’t.
Context Is Not Just Memory
When we talk about context, we don’t mean simply remembering everything a child has ever said.
Context can come from many different signals.
For example:
Age
How old is the child, or what age range are we working within?
Development
What level of understanding might reasonably be expected?
Intent
What is the child actually trying to accomplish?
Risk
What could happen if the system misunderstands the situation?
Emotional signals
Does the conversation suggest fear, distress, confusion, or vulnerability?
Conversation history
What has already been discussed?
Confidence
How certain is the system that it understands what is happening?
These signals can interact.
A low-risk question with high confidence may need very little intervention.
An ambiguous question with potentially serious consequences may require clarification.
A situation with strong indicators of immediate danger may require a very different response.
The important thing is that the topic alone doesn’t determine the response.
What We’re Learning From Building Sisbot
This is one of the areas where building Sisbot changed the way we think about the problem.
At first, it is tempting to organize safety around topics:
relationships
sexuality
self-harm
bullying
violence
drugs
But conversations don’t behave like categories.
A topic can be harmless in one context and dangerous in another.
And a serious safeguarding situation may not contain any obvious “dangerous” keyword at all.
So we’ve been moving toward a different question:
What does the system need to understand before it decides how to respond?
This is one reason our architecture includes separate reasoning around context, memory, knowledge, safeguarding, and decision-making rather than treating safety as a single filter.
The goal isn’t to make the system suspicious of every child.
The goal is to make it more capable of distinguishing situations that look similar on the surface but are fundamentally different underneath.
From Prompt Matching to Contextual Reasoning
This also changes how we think about guardrails.
A traditional guardrail might look like:
Sensitive topic → restrict response
A contextual system looks more like:
Sensitive topic → understand context → assess risk → consider age and intent → determine appropriate level of guidance
That doesn’t mean the AI should always ask questions.
Quite the opposite.
If the system already has enough context and the risk is low, asking unnecessary questions would make the interaction worse.
The point is proportionality.
Sometimes the best reasoning is almost invisible.
A child says:
“I have a crush on someone.”
The system doesn’t need to turn that into a safeguarding investigation.
It can simply have a normal, age-appropriate conversation.
But if new information appears that changes the risk, the response can change too.
That’s what adaptive reasoning means to us.
The Hardest Part: We Can Be Wrong
There is another problem we can’t ignore.
The system can misunderstand the child.
It can misunderstand their intent.
It can misread emotional signals.
It can have incomplete information.
It can even be confidently wrong about what it thinks is happening.
This is why confidence matters.
If the system isn’t sure whether it understands a situation, that uncertainty should influence its behavior.
Sometimes the right response isn’t:
“Here’s what you should do.”
It may be:
“I’m not sure I understand what you mean. Can you tell me a little more?”
That isn’t weakness. It’s intellectual honesty.
And for children, we think that matters enormously.
An AI shouldn’t teach children that fluent answers are automatically certain answers.
What We Still Don’t Know
The more we work on contextual reasoning, the more questions appear.
How much context does an AI actually need?
How can it obtain that context without becoming intrusive?
How accurately can developmental context be inferred?
How should uncertainty change the response?
When should the system ask another question?
When should it simply answer?
How can it distinguish curiosity from genuine risk?
How do we evaluate whether contextual reasoning improves safety rather than simply making the system more complicated?
And perhaps the hardest question:
How do we know when the system has understood enough?
We don’t have complete answers.
We’re building, testing, questioning our assumptions, and documenting what we learn.
That’s intentional.
We don’t want to present Child-Aware AI as a solved problem.
We want to help build the field that can eventually solve it better.
The Child Behind Every Prompt
A prompt is just a few words.
Behind those words is a person.
For a child, that person is still developing their understanding of themselves and the world around them.
That’s why we believe Child-Aware AI needs to do more than understand language.
It needs to reason about context, development, risk, intent, uncertainty, and agency.
Not because children are fragile.
Because children are complex.
And because the same answer isn’t necessarily the right answer for every child, every situation, or every moment.
The question we keep coming back to is simple:
When a child asks a question, can AI understand enough about the child behind it to know what kind of answer they actually need?
We don’t know yet.
But we think that’s a question worth building toward.
And we’d like to hear what you think.