What child safety in AI actually means
Most AI safety work asks whether a system behaves the way its builders intended. Child safety in AI asks a narrower and more practical question: what happens to a child who meets this system, including when it is working exactly as designed? A recommender is not malfunctioning when it holds a thirteen-year-old's attention for three hours. A conversational model is not broken when it answers a lonely child warmly, at length, and without ever suggesting she speak to an adult. Much of the harm sits in the design, not in the failure.
That distinction decides who is responsible and what they have to do. If harm to children were only a matter of systems going wrong, better testing would be the answer. Because so much of it follows from systems working as intended, the answer has to be different design decisions, made earlier, by people who considered children before launch rather than after complaint.
The four ways children meet AI
Conversations about AI and children tend to collapse into a single worry — usually whatever is in the news that month. It is more useful to separate four distinct relationships, because each fails differently and each needs a different control.
- Systems that choose what children see. Recommendation and ranking decide the shape of a child's day online. The risk is rarely one catastrophic item; it is the cumulative pull of a feed optimised for attention, and the speed with which an ordinary interest can be narrowed into something harmful.
- Systems children talk to. Chatbots and AI companions occupy a role children previously reserved for people. They are available at three in the morning, they do not tire, and they are built to keep the conversation going. A system designed never to end a conversation is a poor confidant for a child in distress.
- Systems that generate imagery of children. Generative models have made synthetic child sexual abuse material cheap and fast to produce. This is not a future risk. It is already changing the workload of investigators and the evidentiary picture in prosecutions.
- Systems that make decisions about children. Age estimation, content classification, proctoring, welfare triage and school analytics all act on children rather than with them. Here the harm is a wrong decision applied at scale to people with almost no practical route to contest it.
Why children are not simply smaller users
Product teams often assume that a system safe for adults is safe for children with minor adjustments. The assumption does not hold, for reasons that are developmental rather than technical.
Children are still forming the judgement the system assumes they already have. They are more likely to treat a fluent machine as authoritative, less able to recognise manipulation, and less likely to report something that frightens them — particularly if reporting means admitting they were somewhere they had been told to avoid. Consent, in any meaningful sense, is not available to them: a child cannot weigh a data practice whose consequences will arrive a decade later. And the record follows them. Inferences drawn about a twelve-year-old can persist long into the adult life of the person that child becomes.
This is also why children's rights, rather than risk alone, is the more useful frame. The UN Convention on the Rights of the Child, and General Comment 25 on children's rights in the digital environment, set an expectation that children are protected and retain rights to participation, information, privacy and play. A safeguarding measure that strips those away has traded one harm for another. Age checks that exclude the children who most need support, or monitoring that removes any private space from adolescence, are real costs, not neutral precautions.
What regulators are converging on
The regulatory picture is moving quickly and differs by jurisdiction, but the direction is consistent. Regulators are shifting from policing individual pieces of content towards asking whether a service was designed with children in mind: whether risks to children were assessed before launch, whether the highest-risk features are off by default for young users, whether age assurance is proportionate to the actual risk, and whether someone can be named as accountable.
For most organisations this is the important shift. The question moves from "did you remove it once told?" to "what did you know, and when did you look?" — which is a governance question, answerable only with documentation produced in advance.
What responsible design looks like in practice
Four things distinguish organisations that handle this well from those that are improvising.
- Assess before you build. A child rights impact assessment identifies who could be harmed, how, and what would have to be true for the design to be defensible — while the design can still change.
- Make the safe configuration the default. Protections that depend on a child finding a settings menu protect the children who were already least at risk.
- Size age assurance to the actual risk. Identity verification for everything is neither proportionate nor safe; it creates a new store of children's data to defend. The measure should match what is being gated.
- Build the route out. A child who discloses harm to a system needs that disclosure to reach a competent human quickly, through a path that has been tested rather than assumed.
Where to start reading
The articles below go deeper on the specifics: why this moment is different from earlier waves of technology panic, how AI-generated abuse imagery is evolving and what it demands of investigators and platforms, and what building safe and ethical AI for children requires of product and policy teams.
If you are working on a system that children will use, Child Safe ME advises on responsible AI and child rights impact assessment and on the policy framework that has to sit underneath it.