Who's hiding behind the human-in-the-loop?

Human oversight has been the default answer for questions of safety and risk in AI implementation. But effective oversight requires much more organisational capacity than the popular idea of human-in-the-loop suggests. This capacity gap will only grow larger as many organisations move from AI assistance on tasks to AI agents completing tasks on their own. The challenge in AI governance is not simply keeping humans in the loop, but making sure those humans matter. 

Human-in-the-loop extends familiar ideas 

A human-in-the-loop process uses a personal touch to manage the risks of computerised decisions. The concept is broad enough to describe someone spot-checking AI outputs, giving final authorisation for automated decisions, or managing complex cases. It’s essentially an attempt to get the best of both worlds in AI implementation. Combine the speed of a computer with the personal accountability of a human worker. 

The idea has quickly become the norm in AI governance across sectors. The Australian Government has named human-in-the-loop as a part of responsible AI use across the Transport and Safety Bureau, the Department of Health, Disability and Ageing, and Services Australia. Human-in-the-loop is a key part of AI risk management at Telstra and Commonwealth Bank of Australia. Without using the exact term, the Australian Institute of Company Directors emphasises human oversight and review of AI outputs. 

Perhaps human-in-the-loop is a popular idea because it is comfortingly familiar. Like a pilot in a cockpit managing many automated systems, a human-in-the-loop is ready and able to take charge of any task necessary if things go wrong. AI automation may accelerate mistakes and regulatory non-compliance to startling new speeds. Inscrutable models may risk untraceable new forms of bias. Content generation may risk new types of conflicts around credit and ownership. If only organisations could actually work around these risks by just deeming someone responsible for the results of a largely automated task. 

What our research shows 

Though it may sound comforting, the central promise of human-in-the-loop is dubious. Human accountability moves at the speed of human decisions, not computerised data processing. Organisations already need greater capacity to evaluate and manage AI assistance, not to mention more autonomous AI agents. This will mean hiring more people with domain expertise and investing in evaluation infrastructure. 

I led a recent study with colleagues at Harvard University that illustrates this challenge. We compared GPT models' assessments of health promotion messages with the judgements of health communication experts. In some cases, newer models clearly performed worse than older versions. Mostly, however, hundreds of data points were not enough to clearly determine whether performance had meaningfully changed with model updates. 

Our findings highlight a likely gap in human oversight without significant investment. Models should be evaluated against the tasks organisations care about. But meaningful evaluation requires large amounts of benchmark data and ongoing manual review. Organisations might fill this gap by asking frontend users to evaluate AI outputs as part of their existing roles. Or organisations might create dedicated, crosscutting AI evaluation teams. In reality, both layers of oversight (and then some) might be required to maintain individual accountability in individual cases while also assessing organisational impacts. This may very well be an acceptable cost for certain organisations, but it is a far cry from visions of AI purely as a tool of efficiency. 

Human-in-the-loop  or humanwashing? 

The gap between organisational capacity and AI implementation is not just theoretical. Atlassian uses the AI Fragmentation Tax to describe the cost of increased individual productivity for a team with high coordination needs. Stanford’s latest AI Index Report notes that responsible AI budgets and policies are not keeping pace with the scale of real-world AI applications. This gap creates an obvious incentive for overstating the scope of human oversight relative to the scope of AI implementation. AI regulation must address this risk of humanwashing to promote more responsible implementation. 

Health communication again illustrates what a meaningful focus on organisational capacity can look like in practice. Australia is a global leader in health literacy research and practice, which has moved from a focus on personal skills to organisational environments. This is not to say that individuals are not responsible for their own communication practices, but that health systems are responsible for workforce- and community-level outcomes. The National Health Literacy Framework highlights the importance of shared standards and coordinated action across the sector, as well as the vital role that organisational evaluations play. This approach works because there is a shared set of population-level goals. 

We can imagine a similar approach to AI governance that works toward a shared set of measurable goals around data sovereignty, economic opportunity, and human agency. This will ensure that human-in-the-loop doesn’t turn into humanwashing, and that oversight is measured by demonstrated organisational capacity rather than vague promises of personal accountability.