The Ethics of Machines That Persuade

Persuasion used to be expensive. Changing a mind required a person's time — a speech, an advertisement, a conversation — and that cost imposed a natural ceiling on how much persuading any institution could do. Modern recommendation and language systems have removed the ceiling. Persuasion is now manufactured at industrial scale, personalized to the individual, and optimized against metrics no one consented to.

The standard defenses arrive quickly. People were always persuadable; advertising is old; no one is forced to believe anything. All true, and all beside the point. The ethical questions raised by persuasive machines are questions of asymmetry: systems that model your weaknesses better than you do, iterating millions of times faster than cultural antibodies can develop.

A serious ethics here would begin with legibility. People deserve to know when an interaction has been optimized, what it was optimized for, and by whom. It would continue with limits on optimizing against a person's own considered goals — the difference between a system that helps you sleep and one that profits from keeping you awake. And it would treat the persuasion of children as the bright line it obviously is.

None of this requires believing machines are minds. It requires only noticing that influence has been industrialized, and that every previous industrialization — of food, of medicine, of finance — eventually demanded institutions equal to its scale. The question is whether we build them before or after the damage teaches us why.