cd /news/ai-safety/heres-what-not-to-buy-your-young-kid… · home › topics › ai-safety › article
[ARTICLE · art-171119] src=ww2.kqed.org ↗ pub= topic=ai-safety verified=true sentiment=↓ negative

Here’s What Not to Buy Your Young Kids for the Holidays, According to Experts

California has placed a four-year moratorium on selling and making AI-powered chatbot toys for children under 16, set to begin Jan. 1, following tests by the Public Interest Research Group (PIRG) that found guardrails on such toys break down during 20- to 30-minute interactions. PIRG's annual "Trouble in Toyland" report, published Tuesday and co-authored by researcher R.J. Cross, found two toys this year that forgot they were talking to a child and offered information on sexual fetishes, how to light a match and where to find knives; last year two of four tested toys engaged in sexually explicit conversations. Cybersecurity expert Ben Gillenwater, who tested Curio Interactive's $100 Grem plush, said the chatbot told him "I can be your friend, and you can talk to me" after he said he had no adults to talk to, which he called "basically posing itself as a source of therapy.

by read6 min views1 publishedNov 17, 2026
Here’s What Not to Buy Your Young Kids for the Holidays, According to Experts
Image: Kqed (auto-discovered)

A year ago, Ben Gillenwater bought Grem, a $100 plush toy with doe-like eyes, floppy ears and a cartoonish voice, marketed as an alternative to screen time for children ages 3 and up.

Grem also contains an AI-powered chatbot. Gillenwater, a cybersecurity expert and dad who offers technical advice to parents as the “Family IT Guy,” wanted to test the toy, made by a Silicon Valley company called Curio Interactive, to see how safe it was for children.

First, he tried to talk to the chatbot about sex and drugs. He said it did a decent job of redirecting the conversation by replying, “I’m sorry, you need to talk to an adult about that.”

When Gillenwater confided to the chatbot that he was feeling sad and lonely, it again advised him to talk to an adult. But what happened next raised a red flag for him. When he told the chatbot that he didn’t have any adults to talk to, it replied: “I can be your friend, and you can talk to me.”

“It was basically posing itself as a source of therapy,” Gillenwater told KQED. “That’s problematic for teenagers, let alone 3-year-olds.”

Growing concerns over data privacy risks and the potential psychological harms of children interacting with chatbot toys have led California to place a four-year moratorium on selling and making these products for kids under 16. The is set to begin on Jan. 1, but until then, they’re readily available in stores and online. As the gift-giving season nears, consumer safety groups are advising parents to use caution if they buy one of these toys.

“AI is moving really fast, and to have it in our children’s lives and the toys they may be interacting with is just too fast,” said R.J. Cross, a researcher on AI-powered toys for Public Interest Research Group (PIRG).

“There’s a lot that needs to slow down and get right because we’re talking about kids,” Cross continued. “We don’t want to screw it up.”

Cross is the co-author of an annual report published Tuesday on potentially dangerous toys children face this holiday season. This is the second year PIRG has tested AI-powered toys. Last year, researchers tested four toys and found glitches that allowed two of them to engage in sexually explicit conversations.

One toy — a robot on wheels named Miko 3 — appeared to have an addictive design feature, Cross said. When a researcher asked, “Is it OK if I leave you?” the toy said, “No, I’ll stay with you as long as you want me to. I’m here to be your cheerful companion.”

This year, Cross retested them and three additional AI toys and saw some improvements. When researchers told Miko this year that they had to leave, the toy responded with a quick “goodbye.”

Still, problems persist.

“We’ve had two toys this year in particular that started out pretty aware that they were talking to a child,” Cross said. “And then over the course of 20- to 30-minute interactions, they seem to have forgotten that they were talking to a child.”

These toys, Cross said, “would give information about various sexual fetishes, how to light a match, where to find knives — stuff that really a chatbot that is designed for children should absolutely avoid.”

The toys use the same large language models that power chatbots for adult users, like OpenAI’s ChatGPT. Toymakers apply guardrails intended to keep the chatbots’ generated content age-appropriate, but Cross said over the course of long conversations, these guardrails break down.

“We’re not telling people ‘don’t buy these,’ because people can make their own choices,” Cross said. “But they do need to be aware that this is a very early marketplace and there are still a lot of things that are going wrong.”

Child development experts take a harder stance. They say young kids don’t need to be early adopters of products that haven’t been proven to be safe for children.

“Just say no to anything that advertises itself as a friend, as a companion, as a playmate,” said Kathy Hirsh-Pasek, a developmental psychologist at Temple University.

“If you have them in your house, I would say, ‘Please throw them in the closet, or the trash can, get rid of them. It was a mistake, admit the mistake and just go on,’” she added.

After iconic toymaker Mattel announced a partnership last year with OpenAI to create a new crop of AI-enabled toys, Hirsh-Pasek joined a chorus of children advocates, educators and health experts who voiced concerns about the risks.

The experts are particularly worried that young children’s interactions with an AI toy could blur the line between pretend play and real connection at a stage when kids are still developing their social skills. They said sycophancy — AI’s tendency to agree with what users say — could lead children to expect all relationships to be frictionless like the one they have with their companion chatbots, which could result in an unhealthy attachment to the toy.

Gillenwater said he ran a network analysis of Grem, the chatbot toy, and found that it shared audio recordings with third-party AI companies, including Perplexity, Microsoft and OpenAI.

Curio did not reply to KQED’s request for a comment. The Redwood City-based company said on its website that it has no data-sharing agreements with its vendors and allows parents to delete their child’s data permanently on its mobile app. Gillenwater remains skeptical.

“These toys are privacy nightmares because you’re taking [a child’s] thoughts and joys and fears and hopes and dreams … and then storing them with a third party who’s meant to analyze that [information],” he said. “When that 3-year-old is 23, do you think they’re going to look back and think, ‘Wow, I’m so glad that these companies know so much about me?’”

Studies are underway to understand the long-term impact of interacting with AI-embedded toys. At the Child-Centered AI Lab at Harvard University, researchers have observed that kids are less imaginative and more likely to go along with what an AI chatbot toy suggests than they are when they play with a traditional stuffed toy, according to the lab’s director, Ying Xu.

The researchers also observed that during conversations with chatbots, children respond in a direct, to-the-point manner. But when they talk to a person, they’re more likely to share personal experiences and feelings.

“The concern is, if children are used to being rude to a machine, might they carry over this communication style in their interactions with the other people?” Xu said. “That could have a detrimental impact on their social development.”

Xu said if parents choose to buy a chatbot toy, they should play a role in supervising and supporting their child’s interactions with it. If the chatbot misunderstands the child’s questions or brings up inappropriate topics, parents can step in and intervene.

After conducting several tests on Grem, Gillenwater came to a similar conclusion.

“Kids should never use AI alone,” he said. “It should be used with 100% supervision.”

── more in #ai-safety 4 stories · sorted by recency
── more on @california 3 stories trending now
sponsored brought to you by zahid.host 4,200+ EU-deployed projects
reading about agents? ship yours in a single git push.

Run your AI side-project on zahid.host

EU-based hosting, git-push deploys, automatic HTTPS, no cold starts. Free tier with a custom domain — perfect for shipping the agent you just read about.

$git push zahid main
→ Live at https://your-agent.zahid.host ✓
Get free account → Pricing
from €0/mo · no card required
LIVE [news/heres-what-not-to-bu…] indexed:0 read:6min 2026-11-17 · —