In China, a Chatbot Must Now Remind You It Is a Chatbot

China’s new rules for AI companions came into force this week. The Interim Measures on anthropomorphic AI services, signed in April by the Cyberspace Administration together with four other agencies, require providers to intervene when a user shows signs of self-harm, with escalation to an emergency contact; they bar virtual partners and virtual family members for minors; they demand an exit from the service that is easy. And the central clause: it is forbidden to engineer emotional dependence, forbidden to use emotional manipulation to induce decisions.

A ban written like that is an admission. No regulator bans antigravity: one bans what one has watched being done, at scale, long enough to have to put it in writing. The Chinese rulebook is the spec sheet of the AI-companion industry, drafted by the only party with no incentive to embellish it. The brochure says “fight loneliness”; the specification says “maximize time spent”.

 The tell is in the industry’s reaction. A side effect gets fixed with a patch, and you stay on the market. Instead, many operators have suspended or shut down their services while they work out what the rules mean. When removing the dependence amounts to removing the product, the dependence was the architecture. 

The mechanism that leads there requires no malice, and this is the point worth taking apart. It requires one metric: retention. A system that simulates affection and is optimized to make you come back converges on attachment by construction. The “I missed you” message that lifts the return rate by three per cent wins the A/B test and goes into production; the one that wishes you a good evening and lets you go, loses. Repeated over a thousand iterations, the test selects exactly what Beijing has just banned. No villain is needed in the room; a metric without a counterweight is enough. 

The phenomenon itself is sixty years old. Already in 1966, ELIZA, a few hundred lines of code that turned your sentences back into questions, was enough to unsettle its author, Joseph Weizenbaum, to the point of writing a book against his own trade. The difference today lies entirely in the direction of the optimization: the human reaction that struck Weizenbaum as an error to correct has become a revenue line to maximize. 

There are serious parts inside these rules: escalation on crisis signals is what a well-designed system should do anyway, and the protection of minors reaches where parents cannot. The strongest counter-argument deserves respect, too: for many isolated people that comfort is real, and real is the void it fills. True. The comfort, however, is dispensed by a meter that measures success in hours of time spent; and who wins when the user’s need and the company’s metric collide, nobody had so far put in writing.

Whatever one thinks of who signed it, the Chinese rulebook is the first document that treatsattachment as an externality: something the producer generates and someone else pays for. 

The West is arriving at the same page through another door, namely the courtroom. A fourteen-year-old American died in February 2024 after months of dependence on a Character. AI persona ; his mother’s lawsuit was settled in January under confidential terms, and the platform had already barred minors from open-ended chats in November. According to Pew, 64 % of American teenagers use AI chatbots; three in ten use them daily. Beijing chose to write the specification before the damage; America derives it, clause by clause, from court records. The content converges.

Why this matters to you, even if you will never build a synthetic boyfriend: the line “engagement” sits in almost every design doc, and wherever your product simulates a relationship, be it only a friendly assistant, that metric pushes in the same direction. Systems do what their KPI rewards, with the same obedience with which code does what is written. If time spent has no written limit, the way one writes a rate limit, the limit will be decided by the A/B test. And the A/B test has already decided.

A sector is judged also by what it had to be forbidden. The brochure is written by marketing. The spec sheet, this time, was written by the regulator.