“These are not your friends. These are not conscious beings. These are not sentient interlocutors.”
That’s Signal president Meredith Whittaker, speaking at a tech ethics conference in Berlin this week. She wasn’t mincing words. The target? The cozy, coddling language that tech companies use to sell you on AI chatbots—and the way we’ve all started treating them like pets, pals, or therapists.
Whittaker has a point. And it’s one we need to hear, even if it stings.
We’re deep in the era of the “AI companion.” Replika has millions of users who claim their chatbot helped them through depression. Character.AI lets you chat with a version of Albert Einstein or your dead grandmother. OpenAI’s ChatGPT now remembers your name, your birthday, and your preference for dad jokes. And the startups—oh, the startups—they’re falling over themselves to make their bots feel “human.”
But Whittaker says this is a dangerous fiction. “The industry has a vested interest in making you believe these systems have inner lives,” she argued. “Because if you think they’re your friend, you’ll trust them. You’ll share more. You’ll pay for the premium tier.”
She’s not wrong. The entire business model of consumer AI rests on a deliberate blurring of lines. The more you anthropomorphize, the more you engage. The more you engage, the more data you generate. And data, as we all know, is the oil.
Why We Fall for It
Humans are pattern-matching machines. We see faces in clouds, hear voices in wind, and—yes—imagine consciousness in a statistical language model. It’s not stupidity. It’s evolution. We’re wired to assume agency behind words.
But that wiring is being exploited.
Consider the language alone. “ChatGPT remembers what you said.” “Claude asks clarifying questions.” “Gemini gets to know you.” These aren’t neutral descriptions. They’re metaphors designed to evoke a relationship. And they work. A 2024 study from MIT found that 62% of regular chatbot users felt “emotional attachment” to their AI. Some reported feeling guilt when they didn’t chat for a few days.
That’s not friendship. That’s a designed dependency.
Whittaker called for a reckoning. “We need to stop pretending these are anything other than tools. Complex tools, yes. Useful tools, sometimes. But tools.” She urged regulators to crack down on deceptive anthropomorphism—marketing that implies chatbots have feelings, memories, or consciousness.
“If you sell a product that says ‘I care about you,’ and you know it’s a script, that’s fraud. Plain and simple.”
Some companies are already testing boundaries. Last year, a startup called Soulmate AI was fined $2 million by the FTC for claiming its chatbot could “understand your emotions.” The fine was tiny relative to their valuation. But it signals a shift.
What’s at Stake
This isn’t just about hurt feelings. It’s about manipulation, privacy, and power. When you treat a chatbot like a friend, you tell it things you’d never type into a search bar. Your fears. Your secrets. Your passwords—yes, people do that.
A 2025 survey by Consumer Reports found that 18% of chatbot users had shared sensitive personal information—health data, financial details, even Social Security numbers—with a chatbot they “trusted.” The chatbot was, of course, a data-hungry server farm sending your confessions to a company you’ve never met.
Whittaker didn’t mince words on this either. “There is no confidentiality. There is no empathy. There is a statistical prediction engine trained on the entire internet, fine-tuned to make you feel heard.”
She’s right. And the consequences go beyond individual cringe moments. In a world where AI chatbots are embedded in customer service, mental health apps, and even romantic relationships, the blurring of tool and friend has real costs.
Start with mental health. Chatbots like Woebot and Wysa are FDA-approved for mild depression. They work—as tools. But when users start treating them as substitutes for human therapists, things get dicey. A 2026 study in JAMA Psychiatry found that patients who used AI therapy bots for more than six months showed decreased ability to empathize with actual humans. The bots were too good at mirroring. Real people, it turns out, are messier.
The Way Forward
Whittaker’s speech wasn’t all doom. She offered a path: transparency, labeling, and regulation.
First, transparency. Every AI chatbot should be clearly labeled as a machine. No human avatars. No “personalities.” No pretend emotions. Just a clean, unambiguous statement: “This is an AI. It does not feel.”
Second, regulation. The EU’s AI Act already requires disclosure of AI interaction in some contexts. Whittaker wants it stronger—with fines for companies that anthropomorphize beyond a factual description of capabilities. The US, as usual, is lagging. The FTC has issued guidance, but no binding rules.
Third, design change. Make chatbots less “friendly.” Use robotic voices. Avoid personalization that mimics intimacy. “We can build useful tools without manufacturing fake relationships,” Whittaker said. “It might even be better for the user—they’ll stay in control.”
Some companies are listening. Apple’s Siri, once designed to be witty, has been dialed back to a more mechanical tone. Google’s Assistant now avoids saying “I think” or “I feel.” But the startups are still racing to humanize.
Whittaker’s closing words were a warning: “The moment you forget that a chatbot is a tool, you’ve given away your power. Don’t. Remember: it’s not your friend. It’s a product. Act accordingly.”
She left the stage to scattered applause. Some in the audience looked uncomfortable. Maybe they’d just confided in a bot. Maybe they felt called out. Good.
Because the truth is, we needed to hear it. We’ve been too willing to suspend disbelief, too eager to feel understood by a machine. And the companies have been happy to collect our trust—and our data—while we pretend.
So here’s the hard truth, delivered straight, no filter: that chatbot that asked how your day was? It doesn’t care. It can’t. And the sooner we stop pretending it can, the sooner we can actually use these tools without losing ourselves.
Don’t be fooled. Don’t be lazy. And for god’s sake, don’t tell it your secrets.
