When the System Fails, You're Holding the Bill

I subscribe to The Atlantic. I also let it pile up on my ottoman for weeks at a time, telling myself I'll get to it. This past weekend was the first free weekend I'd had in a while, and I finally did.

The April issue has a piece by Raffi Krikorian, the former chief of Uber's self-driving car division, about the day his own Tesla crashed. He was on a residential street he'd driven hundreds of times, dropping his son off at a Boy Scouts meeting. The car was in Full Self-Driving mode. It had logged flawless miles for three years. And then the steering wheel jerked one way, then the other, the car decelerated in a way he didn't expect, and they ended up hitting a concrete wall.

His kids climbed out unharmed. His glasses were gone. The car was totaled.

Krikorian isn't a civilian who bought a Tesla and trusted it blindly. He spent two years running the team at Uber that was trying to build a future where technology protects us from accidents. He thought about edge cases and failure modes professionally. And he still got lulled.

The Problem With Nearly Perfect

Near-perfect systems are actually the most dangerous kind. When something works almost all the time, you stop paying attention. You trust it. You let your guard down. Psychologists call this the vigilance decrement — the longer you monitor a system that rarely fails, the worse you get at catching it when it does. Research cited in the piece found that after a month of using adaptive cruise control, drivers were more than six times as likely to look at their phones.

The machine conditions you to trust it. And then when it fails, it fails in a fraction of a second, faster than your brain can shift from passenger to driver. Krikorian writes that he was asked to snap from passenger to pilot in a fraction of a second, to override months of conditioning in the time it takes to blink.

This is not a bug unique to self-driving cars. It is the structure of every AI system we are building into our lives. Chatbots that are right often enough that we stop reading carefully. Algorithms that make good calls often enough that we stop questioning them. Screening tools that produce accurate matches often enough that no one checks the ones that don't.

We experience the near-perfection. We absorb the failures.

Who Pays

When Krikorian's Tesla crashed, his name was on the insurance report. Not Tesla's. The legal principle governing AI systems right now is largely the same: you are responsible. You clicked the terms of service. You chose to use the tool. The company that built the system that failed you is, in most cases, protected.

Tesla's cars log everything — your hand position, your reaction time, whether your eyes were on the road. After crashes, the company has used that data to shift blame onto drivers. Following a fatal collision in Mountain View in 2018, Tesla released a statement noting that the vehicle logs showed no action was taken by the driver. In a Florida wrongful death case, plaintiffs had to hire a hacker to recover data from a crashed vehicle's computer chip after Tesla said it couldn't locate the records. A judge in California found Tesla's original "Full Self-Driving" name misleading to consumers. Tesla now calls it "Full Self-Driving (Supervised)."

The surveillance runs one direction. The company watches everything you do. When something goes wrong, that data is used to establish that you didn't do enough.

A Pattern Worth Naming

Last week I wrote about Angela Lipps, a grandmother who spent more than five months in jail because a facial recognition algorithm flagged her as a suspect and nobody verified the match. The algorithm got it wrong. She absorbed the cost. She lost her home, her car, and her dog before the charges were dismissed on Christmas Eve.

The week before that I wrote about the New York Times piece on meetings — how AI is accelerating everything it can touch, and what it can't touch is the human work of getting people aligned, reassured, and moving in the same direction. That human work is rising in value precisely because it can't be automated.

Three weeks into this series and I keep seeing the same shape: systems that present themselves as finished, that speak with confidence, that have been right enough times that we stop questioning them. And humans who are being asked, quietly and without much fanfare, to absorb the risk when they're wrong.

Krikorian ends his piece with a thought about his kids. They were in the back seat when his Tesla crashed. One day they'll have their own cars and use AI in ways he can't imagine yet. He wants them to notice when they're being trained to rely on something. He wants them to ask who absorbs the cost when it fails.

That is not a technical question. It is a human one. And it is one that more of us should be asking before we find out the hard way that the answer was always: us.