OPINION: Our children need more protection from the AI that is teaching them

For 15 years, we let children carry cellphones into most of our classrooms. By the time states began ordering the phones out, a generation had spent its childhood inundated by screens. We had protected everything about the arrangement except for the children. 

Similarly, during the rise of the commercial internet, we wrote rules to protect the privacy of platforms and the people profiting from them. The protections we later built for children, like the rule that a website must get a parent’s permission before collecting a child’s data, would never have been able to make up for the ground we had already ceded.

As a subject matter expert on human trafficking and child labor exploitation at the U.S. Department of Education, I watched systems that had been built to protect children instead shield the privacy of the people harming them. That failure taught me that the guardrails we neglect to build at the outset are protections we forfeit permanently. 

Now we are watching artificial intelligence companies race to install enforceable rules protecting their interests. We’ve seen this before: If the pattern holds, the industry protections that set the precedent will not be dislodged by the child protections that follow. This time, the protections for children have to come first.

I believe no AI tool that teaches, assesses or tracks a child should be eligible for a school district contract until it passes a bias test that is built around the very same students that digital tools have harmed before. 

Related: A lot goes on in classrooms from kindergarten to high school. Keep up with our free weekly newsletter on K-12 education.

Last year, a study of AI assistants used by teachers found them recommending harsher approaches for struggling students whose names read as Black. A separate study found that an AI grader gave essays by Black students lower scores than essays by Asian students, reproducing a gap already present in human scoring.

None of this bias has to reach a child, because districts already have the place to catch it. Most districts already run a privacy check before buying a technology: They check whether a digital tool will keep a child’s data private and secure; a tool that fails is not supposed to be bought. Bias testing would work inside that same check. To test a tool for bias, an evaluator gives it nearly identical student work, changing only the name on each sample and the background and speech patterns the writing itself signals, and compares how the tool responds. An evaluator also examines what the tool teaches students about people: which stereotypes it repeats and whose history it tells in full. 

A tool that fails either examination should lose its eligibility. What makes a tool safe for the children most at risk makes it safe for every child.

Nearly every state has moved to regulate AI in some way, and a handful have enacted chatbot laws aimed at protecting children. California requires a chatbot to tell a child they are using AI and to remind that child to take regular breaks. New York State requires chatbots to detect expressions of suicidal thinking and refer the user to crisis services.

These are positive changes, but the child protected while engaging with AI at home has no comparable protections at school. Only a few states have passed laws directing their education agencies and school districts to manage AI tools. In Oklahoma, for example, an educator must review any content an AI tool generates before it reaches a classroom, and no AI tool may become the primary basis for a grade, promotion or retention decision.

Oklahoma’s law goes further than most. Typically, when one of these laws passes, it is little more than preliminary direction telling agencies and districts to develop their own policies. 

This month, New York City barred students through eighth grade from using generative AI, and Los Angeles Unified blocked every student from using generative AI on district devices, including the AI helpers built into Google Classroom, which nearly every American school uses. They banned what they could not test. New York City’s ban lasts one year, and the city will spend that year reviewing every technology tool in its schools. That review is what every district should be able to run before a tool reaches a student. A ban buys time but does not tell a district which tools are safe, and it takes useful tools away along with the harmful ones.

Even the most precise law will not be enough to protect a child. A law is only a promise before it is enforced. The school district must keep the promise by examining every tool before it is bought and again while children use it. Even without any new laws, every district carries a legal duty of care for every child, yet almost no district has the staff, training or technology to fulfill that duty. 

The privacy checks that most school districts run today do not test tools for bias at all: No state requires districts to do so, and no law requires the companies to disclose bias found in their AI. 

But those checks, which ask only whether a tool protects a child’s data, cannot determine whether a tool built on a large language model is safe for a child. That underlying AI model may treat a child differently even when race is never named, because it can infer race from the child’s name, neighborhood, school, language spoken at home or writing style. Those are the most obvious clues; a standard checklist is unlikely to catch the subtler ones. Facing public scrutiny, the companies that built these models have made the bias harder to detect. The companies trained the models to stop saying explicitly biased things but did not remove the bias itself. 

Related: AI gives more praise, less criticism to Black students 

When researchers gave AI models writing samples in the dialect many Black Americans speak at home, the models said nothing negative about Black people yet judged the writers themselves as less intelligent and matched them to less prestigious work. This is why no tool should keep its eligibility without being retested while students use it. These systems change after they are sold. 

Although a privacy check will not catch this bias, until recently, one protection remained after the check: A biased tool could still be pulled out of schools if statistical proof showed it was harming one group of children more than another. The U.S. Department of Education recently withdrew the rule that made this possible. Now, its Office for Civil Rights will act only when someone can prove a school or district intended to discriminate. Proof that children of one race are being harmed more than another will no longer be enough, and the tool will remain in the classroom. 

No one intends there to be bias in these tools: The models absorb it from the internet they learn from. The harms appear only in the patterns of which children are taught less, scored lower and tracked as struggling.

Once again, everything about the arrangement is protected — except for the children.

Shauna D. A. Knox is founder and CEO of The Emancipation Group, an innovation studio that combines research and development with digital product development to ensure Black children and families achieve authorship and ownership in the innovation economy.

Contact the opinion editor at opinion@hechingerreport.org.

This story about protecting children from AI was produced by The Hechinger Report, a nonprofit, independent news organization focused on inequality and innovation in education. Sign up for Hechinger’s weekly newsletter.

The post OPINION: Our children need more protection from the AI that is teaching them appeared first on The Hechinger Report.