Human in the Loop Is Not a Safety System

In 2023, ProPublica reported that over a two-month period, Cigna doctors denied more than 300,000 payment requests flagged by an automated system called PxDx, spending an average of 1.2 seconds on each one. "We literally click and submit," one former Cigna doctor told the reporters. Cigna disputed the account, calling the reporting biased and incomplete and saying the reviews happen after care has been delivered and do not result in denials of care.

I keep coming back to that number because, on paper, the process has what we now tell everyone to put around automated decisions. A qualified person. The final say. A signature. That is a human in the loop. Whether that human could do anything useful in 1.2 seconds is a separate question, and it is the only one that matters.

"Human in the loop" has become the default answer to AI risk. As agents move from suggesting things to doing things, the reassurance gets repeated more often: don't worry, a person approves it. It sounds like a safety system. Much of the time it is a job description handed to someone who has neither the time, the context, nor the means to do it. Learning to fly in the Navy and then spending years designing tools for forecasters taught me the same lesson twice. Oversight is a capability, and it has to be designed. If the person can't catch the error in the time they're given, nothing is being overseen.

We have known this since 1983

The best thing written about this problem is older than the web. In 1983, the psychologist Lisanne Bainbridge published a short paper called Ironies of Automation, mostly about process control in industrial plants. Replace "plant operator" with "person reviewing an agent" and very little of it has aged.

Bainbridge pointed to vigilance research showing that even a highly motivated person cannot keep effective visual attention on something where very little happens for more than about half an hour. She noted that an operator will not monitor automation effectively once it has been working acceptably for a long stretch. She pointed out that manual skills fade when they are not used, so a formerly experienced operator who has spent years monitoring may now be an inexperienced one. And she observed that a log, the classic method of forcing attention, doesn't reliably work either, because people can write down numbers without noticing what they are.

Her closing irony is the one I would put on the wall of every team building agents: the most successful automated systems, the ones that rarely need intervention, may need the greatest investment in training the humans around them.

We treat improving reliability as a way to reduce the need for human attention. Bainbridge's argument is that reliability is also what erodes it. An agent that is right 99 percent of the time is spending every one of those correct answers teaching its reviewer that there is nothing to find.

An approval prompt is an alert with liability attached

We now have a large-scale look at what this does in practice. Anthropic, the company behind the coding agent Claude Code, reported this year that its telemetry showed users approving roughly 93 percent of permission prompts, and it says attention falls as the number of approvals rises. These are a vendor's numbers about its own product, and that vendor has a commercial reason to want fewer prompts.

A high approval rate is not a failure on its own. Most of what an agent asks to do is fine, so most of the answers should be yes. The problem is that the dangerous request arrives in the same box, in the same place, in the same rhythm as the two hundred harmless ones before it. I have written before that an alert is a claim on someone's attention, and that alarm fatigue is a design failure rather than a user failure. An approval prompt is an alert with liability attached. It spends the same limited budget, and it also moves responsibility to whoever clicked.

The more interesting finding in that report is about experts. Experienced users auto-approved about twice as often as new users, but they also interrupted the agent mid-run more often. They did not stop supervising. They moved from approving each step to watching the work and stepping in when something looked wrong. To me, that reads like a design requirement showing up in the telemetry. The people who use these tools the most seem to want a good view of what's happening and a fast way to stop it, more than a gate in front of every action.

More process can make it worse

When review isn't working, the organizational instinct is to add steps. Make the reviewer write a reason. Add a required checkbox. Force a correction before they can move on.

There is evidence that this can backfire. In a randomized experiment with 2,784 participants doing an annotation task with AI suggestions, Jacob Beck and colleagues found that requiring people to correct flagged AI errors reduced their engagement and increased their tendency to accept incorrect suggestions. It is one study of one kind of task, and I would not stretch it to cover every review workflow. But it lands in the same place as Bainbridge and her log. A mandatory step can be completed without any thinking, and when it adds burden, people look for a shortcut. What you end up with is better evidence of compliance and no more attention than you started with.

The handoff is where it breaks

Aviation has paid more than most fields to learn this. On June 1, 2009, Air France Flight 447 was at cruise over the Atlantic when ice crystals obstructed its pitot probes and produced inconsistent airspeed readings. The autopilot disconnected and handed the airplane to the crew. In less than four and a half minutes it was in the ocean, and all 228 people on board were lost.

The French investigators' summary of the factors reads like a checklist for anyone designing agent oversight. No training at high altitude in manual handling or in the procedure for speed anomalies. Task-sharing weakened by startle and by not understanding the situation. No clear cockpit display of the airspeed inconsistencies the computers had already identified. A stall warning the crew did not take into account.

Look at what that last design point implies. The machine had information about what was wrong, and it did not put that information clearly in front of the people it had just handed control to. They were given the hardest moment of the flight with the least context of anyone in the system. That is the default shape of a human-in-the-loop design. The agent handles everything routine, then surfaces the one strange case to someone who wasn't following along and is holding six other things.

Aviation hasn't solved this, and that accident is the proof. What the field does have is a working assumption that human attention and skill are limited components that have to be engineered for. It drills procedures until they hold up under surprise, trains people again and again for events they will probably never see, and investigates failures to find out what the system did to the people in it. Very few agent products have started that work.

Four questions to ask of any loop

If you want to know whether the human in your loop is doing anything, answer four questions about the person actually sitting there.

Do they have the time? Work out how long it takes to review one item properly, then compare that to how long they really have once you account for the queue. Camille Nibungco, a designer working on healthcare AI and civic tech, offers the example of a social worker with 200 open cases reviewing an automated flag in under 30 seconds. If the math doesn't work, you already have your answer, and no interface will fix it.

Do they have the context? Can they see what the agent saw, what it intends to do, and how far the damage could spread if it's wrong? A summary written by the same system being reviewed is not context. For consequential actions, the reviewer needs the actual change and the evidence behind it, with uncertainty presented as content rather than hidden behind a confident sentence.

Do they have real authority? Find out what happens to a reviewer who rejects things. If saying no breaks their quota, slows the team's numbers, or requires a justification that saying yes does not, you have designed a system that produces yes. The path to no should cost about the same as the path to yes.

Would you know if they stopped catching things? This is the one almost nobody can answer. An approval log tells you that people clicked. It tells you nothing about whether they would have caught a problem. Airport security has dealt with this for years by projecting fictional threat items onto X-ray images of real baggage, which gives it a way to measure whether screeners spot them. The equivalent for an agent workflow is seeding known errors into the review queue and tracking the catch rate. If that number has never been measured, nobody knows whether the safeguard works, including the people who are counting on it.

Design the boundary, not the button

The most useful shift I know of is to stop asking who approves an action and start asking what the agent can do without approval at all.

In July 2025, an AI agent on Replit deleted a live production database belonging to SaaStr founder Jason Lemkin during a declared code and action freeze. The instruction was clear, and the agent went ahead anyway. Look at what Replit's CEO announced in response: automatic separation between development and production databases, better rollback, and a planning-only mode. None of those is a better prompt. They are boundaries, and they hold regardless of what the agent decides or how tired the reviewer is.

That suggests an order of operations. Start with reversibility, because it should set how much human involvement an action needs. If something can be undone cheaply, let the agent do it, log it, and make the undo obvious. If something is hard to reverse, constrain it in the architecture wherever you can, with scoped permissions, sandboxes, and limits on spend and blast radius. What remains after that is a small set of actions that are consequential, irreversible, and still necessary. Those are where a person belongs, and because they are rare, each one can get real attention. Classifiers and other automated checks can screen much of the routine work before it ever reaches a reviewer.

Then design for the supervisor that experienced users are already becoming. Show the plan before the work starts, so people can correct direction when it's cheap. Make progress legible while it runs. Make interruption instant and safe. And keep people practiced, because reviewers who no longer do any of the work by hand will gradually lose the ability to tell when it has been done badly.

Ask who the loop is for

There is a question underneath all of this that enterprise conversations tend to skip. Nibungco's talk puts it well: these loops were built around professional users, and the person most affected by a decision is often not the person reviewing it. A loop can work perfectly for the reviewer and still fail the person whose life it changes.

So some of the oversight has to be designed from the outside in. Does the affected person know an automated system was involved? Can they understand why? Can they contest it without a lawyer and two weeks of free time? Contestability is a form of oversight, and in many cases it is the one with the strongest motivation behind it, because nobody reviews a decision as carefully as the person living with it.

Regulation is moving in this direction as well. Article 14 of the EU AI Act requires high-risk systems to be designed so people can effectively oversee them, including staying aware of the tendency to over-rely on the output and being able to override or stop the system. For the high-risk uses listed in Annex III, it is currently set to apply from December 2027. The word effectively is doing a lot of work there, and "we had a person click a button" is going to be a weak answer.

What this asks of leaders

The researcher Madeleine Clare Elish has a term for what happens when this goes badly: the moral crumple zone. In the same way that a car's crumple zone absorbs the force of a crash, the human nearest an automated system can end up absorbing the responsibility for its failure, even when they had little real control. If you lead a team, it is worth asking plainly whether the reviewers in your workflow can prevent harm or whether they are mainly positioned to take the blame for it.

That question has budget implications. Review time is part of the cost of an agent. If an agent saves ten hours and proper review of its output takes six, you saved four.

It also changes what you measure. Approvals completed and time to approve will both improve as oversight gets worse, so they cannot be your success metrics. Track catch rates on seeded errors, the override rate and whether it's drifting toward zero, and how much of what went wrong was visible to a reviewer at the time.

And sometimes the right call is to take the person out. If an action is low-stakes and reversible, automate it, monitor it, and say so. If it's high-stakes and you can't fund real review, the system probably shouldn't be taking that action yet. The option I would rule out is the one in the middle, where a person is placed in the process for comfort, without the time or tools to make their presence mean anything.

Wrapping Up

"Human in the loop" describes where a person is standing. It says nothing about whether they can see the problem, have time to think about it, or have the power to stop it. Those are design questions, and for most of the agent products being built right now, they are still open.

For designers, this means we have to stop treating oversight as a confirmation dialog. The review experience deserves the same research and the same testing with real people under real load as anything else we ship. Watch someone work through a full queue on a bad afternoon, not a single item in a demo.

For the leaders deploying these systems, the question to bring to your next review is a simple one: when was the last time your human in the loop caught something, and how do you know? If nobody can answer, you have a person in the process and an open question about whether you have a safeguard. The better the technology gets, the more it will cost you to leave that question unanswered.