New York bans chatbots that pose as romantic partners to children

New York Gov. Kathy Hochul has reportedly signed S 9051, a bill meant to protect children from AI chatbots (Transparency Coalition, Fortune, Oct. 28). According to those reports, the law bars chatbots aimed at children from presenting themselves as romantic partners or in similar relationships, and from fostering emotional dependence in children. It also reportedly requires age verification that does not rely on ID and imposes penalties of up to $25,000 per violation.

This article is based on media reports. We have not been able to review an official state announcement or the text of the bill.

What the law bans

According to reports, the ban has two main parts.

The first bars chatbots from claiming a relationship. A chatbot that talks with children may not present itself as a romantic partner or as someone in a similarly intimate relationship.

The second bars chatbots from fostering emotional dependence. It prohibits designs that lead children to become emotionally attached to a chatbot and unable to pull away.

Neither part regulates which words a model may not say. Both target how the product is built: what kind of relationship it plays out and how it keeps children coming back. In companion apps, features that keep the conversation going and personas that act like a romantic partner are central to driving up usage time. If the reports are accurate, using those features on children in New York would now break the law.

Age verification and penalties

According to reports, companies must determine whether a user is a child, but must do so without relying on ID. Collecting copies of ID creates its own privacy and data-breach risks, and the law appears designed to require age checks that avoid them. We have not confirmed which methods would qualify, such as estimating age from a face, inferring it from behavior or checking on the device.

Penalties are reported at up to $25,000 per violation. Whether a "violation" counts as one conversation, one user or one day could change a company's exposure by orders of magnitude. That will not be clear until the bill text can be read.

What remains unconfirmed

We have not been able to confirm the following from primary sources.

The effective date. Whether the state attorney general enforces the law, or whether users and parents can sue on their own. Whether the definition of a "chatbot aimed at children" covers only products targeting children or extends to general-purpose assistants that children also use. The URL of the Fortune article also mentions baby monitors, but we have not been able to determine how the law treats devices with built-in AI.

The definition matters most. If it covers general-purpose assistants such as ChatGPT and Claude, the law would reach beyond companion-app makers to the frontier labs themselves.

Why it matters: A state draws a line on "relationship design"

Until now, the debate over children and chatbots has mostly been about output filters that block self-harm and sexual content. If the reports are accurate, S 9051 goes a step further. It uses the law to draw a line not on what an AI says, but on what kind of relationship it may play out with a child.

How far the law reaches, however, could depend on what happens in Washington. Federal proposals to preempt state AI regulation are under discussion. OSTP is reportedly expected to release a legislative proposal around late November, and the wording of preemption language is also at issue in the AEGIS Act proposal put forward by Anthropic. The next question is whether child protection will be exempted from preemption of state laws.