If you’ve thought about AI in 2026, odds are that you’ve heard the phrase “human-in-the-loop”. On a high level, it’s the idea that we can manage the problems of AI systems (inaccuracies, inequities) by making sure that any important decisions are checked and decided by humans. It’s usually used as a shorthand for “Don’t worry! We still care about users,” and is a stand-in for making sure that automated systems aren’t punitive and inaccurate. I agree with this!
I cringe, though, when this many years into AI being a consumer product I see people bandying about the words “guardrails” and “human-in-the-loop” as catch-alls for accurate and fair AI services. They’re not.
The problem is that “human-in-the-loop” is often based on an outdated mental model of AI and failure states. (There’s a good case to be made for all of us being more precise about exactly what technologies and processes we mean by “AI”, but that’s a post for another day.)
Like everything else, we need to design them thoughtfully and describe them accurately to avoid creating a black box mess. The new processes behind the scenes need to support how the technologies we want to use actually work, to the best of our 2026 knowledge.
Before AI was cool
I started working with AI as a design material in London in 2018. Back then, there were a couple of design agencies in London doing interesting and heavily NDA’d work about emerging AI systems. I was at one of them. The brilliant people at Special Projects, another one of those agencies, spoke some of their insights from that time here.
The joy of working with emerging technology and ideas is stretching your mind to grasp something that’s not yet well-articulated. Looking back at it, there’s something deeply satisfying about realizing how little anyone knew, and how much easier it is to understand now. That Special Projects piece reminded me of how we have a subtly different relationship with AI from the received wisdom a decade ago, in part because we’re finding out more and more about how these technologies work.
At the time, it seemed like it would be pretty obvious when AIs were wrong. Like calling me Emma instead of Ella, or recommending a nighttime walking route through a public park for a nervous young woman. If you’re looking for a widely cited academic paper from that time that shows that brittleness, Jia and Liang’s 2017 paper on adversarial reading comprehension is a complicated read (sorry!) which argues that most inaccuracies would be easy to spot.
Human-in-the-loop, a Google Trend history
As far as I can tell, the idea of human-in-the-loop as a way of designing services and systems that manage the risks from AI came from that era.
Human-in-the-loop is a cybernetics idea (waves in Bateson), but mainly it has been used in aviation. The wikipedia page is great, if you want to burrow in to it.
It’s a great metaphor, which I think is why it caught on. If you don’t trust me, you can trust Google Trends:
I’m not being too critical about this. Human-in-the-loop makes sense as a way of thinking about managing risks and inaccuracies of a powerful new technology. We use airplanes to do something that wasn’t previously possible for a lot of people (hot air balloons and airships aside). Pilots aren’t just props or NPCs: they have real roles at critical times, and autopilots help them do their jobs better. Plus, they help reassure passengers that there is still a human in charge of the machine. I don’t know, but I deeply believe that “This is your pilot speaking” announcements are less about communications, and more about reassurance.
What’s wrong with human-in-the-loop?
Arguably, human-in-the-loop is still the main way enterprise and policy leaders are thinking about managing risk to individuals from AI systems. This would be fine if it were based on accurate, up-to-date understandings of how these technologies work.
What we know in 2026 is that while airplanes fail in catastrophic, obvious ways, LLMs often don’t. They often smear their inaccuracies across something: about 10% wrong but 90% right. To back me up on this, either think about the interactions you have with your own chatbots and agents. Or, if you’re looking for a paper about it, it’s worth checking out the Lost in the Middle one from a few years ago. If you’re less keen on academic papers but like HackerNews, “context rot” is probably your best shorthand.
I think this matters because a lot of the language and design approaches we developed in the late 2010s about humans-in-the-loop assumed that mistakes would be obvious and easy to catch. We now know that’s not really the case. The wrongness is in weird flabby code choices, in misreading tone, in plausible sounding sources that don’t actually contain what they seem to. Or in credible-sounding embarrassing, hallucinated sources in government reports (or, with chef’s kiss irony, in national AI strategies).
This kind of AI inaccuracy makes humans-in-the-loop a much more difficult design and organizational intervention. It’s not just that you need to be an actual expert to spot the mistakes: you need to be an actual expert who is on top of your game. Unlike being a pilot, this is an anxious game, with moments of intense, presumably high stakes activity. It will make these jobs radically more like being goalkeepers than outfielders.
That’s one thing if you’re choosing to run your operations as fully outsourced, exploitative businesses, much like how checking offensive images on social media has been done for the past decade. How is that going to work with existing, highly paid experts? Or heavily unionised government staff, who have rights to appeal when their work conditions risk giving them burnout? Leaving this as an implementation detail, to be figured out later (which I heard in a demo a while back), isn’t really a sound option.
It doesn’t work elsewhere either
Human-in-the-loop also has real problems in more predictable automated decision-making.
Since moving to LA last year, I’ve been surprised to find the UK’s Universal Credit program spoken of as a successful example of a redesigned service. Beyond being widely criticised by campaigning organizations for being exclusionary (here’s an example from 2023), it is well-documented that UC has long struggled with the problems of bias and suffering that human-in-the-loop seeks to avoid.
Source: AI was supposed to make the UK benefits system more efficient. Instead it’s brought bias and hunger. Image drafted with Claude.
What we can learn from Universal Credit, though, is the failings of a human-in-the-loop approach even when the underlying system is more predictable than an LLM one. Once human staffing cuts occur, and it’s clear that the KPI is about avoiding generosity and guarding against fraud, there are inevitably not enough humans in those loops. Sadly, those that remain are trained to be less humane in their decisions.
What should we try instead?
I think there’s less and less reason for us to base our organizational and service design on AI research and metaphors from a decade ago. We also can’t assume that super-human staff will save our public services. It’s one thing to ask people to spend time and effort helping other people. It’s quite another to expect them to be hypervigilant to avoid smearing damage across their user groups.
The potential wins of using AI are also vast. I’m not suggesting we don’t use it for public services. Nobody knows quite what good looks like, yet, but we have learned that LLMs break in smeared ways, not brittle ones. How we design services around this needs to reflect that. I have two main thoughts about this at the moment:
- We need to design clearer expectations into our services. A lot of the US public sector services I’ve used so far reduce friction up front, both by policy intent and through a shortage of standardized documents. For automated decisions, though, we need clear, underlying data schemas for what will be accepted, and why. Without this, it’s intensively difficult to get the benefit of automation. Those who don’t have these documents will continue to need other forms of support, throughout. That needs to be designed and staffed.
- Appeals need to be phenomenally easy, and entirely reimagined. It cannot simply be the case that users are redirected to a general call center, or asked to fill in a standardized form. AI systems already fail in unexpected and strange ways. As designers, product people and policy experts we need to acknowledge that we simply can’t anticipate all the ways an AI system will be inaccurate, even if it is only 10% of the time. Let’s make appeals the focus of our interface and org design work, with clear patterns for single click appeals, easy, well-resourced notifications and reassurance throughout that if you have appealed you will be heard.
A moratorium on hand-waving
I’m less interested in hearing “guardrails” and human in the loop, then, and more interested in metrics like: how you’ve used loops to do the first few accuracy checks; how the staffing has changed; whether staff satisfaction is up and sick days and other absences are down, and whether rework is reduced. Or if you’ve had to begin hiring different people. Systemic metrics, not simply how many more applicants can be processed an hour.
I think there’s a huge opportunity for all this to improve services and make work more satisfying for public sector staff. Let’s base that thinking on the reality of what today’s technology actually does, rather than base it on assumptions of the better part of a decade ago. Let’s be less invested in management memes, and more interested in designing and shipping efficient, fair services.