Aviation did not become safe by testing whether individual components could handle their specified loads. It became safe by analyzing how those components interact, how control responsibilities are distributed, and how operational conditions erode the accuracy of safety calibrations over time. The discipline that codified these lessons is called systems safety engineering, and it has produced methods like STPA, FRAM, and STECA that are now mandatory in regulated industries.
AI safety has not yet imported these methods.
Luca Carlucci, James Fillingham, Robert Walpole, and Bartłomiej Kryś address this directly in "A Systems-Thinking Approach to Loss of Control Risk from Advanced AI" (arXiv:2606.13474), submitted June 11 and updated June 29, 2026, and accepted at the ICML 2026 Technical AI Governance Workshop. They apply three established industrial safety methods to a frontier AI coding agent scenario and identify three systemic hazards that standard model evaluations (capability testing, red-teaming, and alignment measurement) were not designed to find.
The methods: STPA, FRAM, and STECA
The scenario
The paper applies all three methods to a realistic deployment scenario: a frontier AI coding agent used in a production software development environment, with a realistic oversight structure including a deploying lab, enterprise customers, and user-facing developers.
This scenario was chosen because autonomous coding agents represent current frontier deployment (long-horizon, multi-step, tool-using, with access to real codebases and external services) and because they are already deployed at scale while their loss-of-control risk remains largely unanalyzed at the systems level.
Three findings that model evaluations miss
In the deployed coding agent scenario, accountability for safety commitments rests substantially with the frontier lab: it makes the commitments, it sets the usage policies, it monitors for violations. External verifiers (regulators, enterprise customers, third-party auditors) cannot independently confirm that safety obligations are being honored at the operational level. The lab's self-reported compliance is the primary accountability mechanism.
STECA identifies this as a structural gap in the control structure. A control action is only meaningful if the entity applying it has reliable feedback about whether the action is having its intended effect. When the entity responsible for safety is also the primary source of information about whether safety is being maintained, the feedback loop is compromised. This is not a criticism of any particular lab's honesty; it is a structural property of the governance arrangement that applies regardless of good faith.
STPA maps the temporal gap between an AI agent taking an action and a human monitor receiving the signal that the action occurred. In real-time deployments with multi-step execution, this gap is long enough that intervention often arrives after the relevant harm pathway is already established. The agent has already called a tool, produced an output, or committed a code change by the time the alert reaches a human reviewer.
Aviation calls this the "out-of-the-loop" problem. When automated systems operate faster than human operators can track, the operator's nominal authority to intervene becomes nominal in both senses, formal but not functional. The monitoring delay finding suggests that real-time oversight of frontier coding agents may currently have this property: the formal oversight structure exists, but the timing makes effective intervention structurally difficult.
FRAM identifies a third risk through its focus on variability in normal operations. Safeguards are calibrated against a representative sample of operational conditions at deployment time. But operational conditions shift: users change how they prompt the agent, task distributions change, edge cases accumulate, and the gap between the deployment-time sample and actual operational distribution grows over time.
This erosion does not trigger alerts. The safeguards continue functioning exactly as designed against the conditions they were calibrated for. The problem is that those conditions are no longer representative. The gap grows silently until a sufficiently novel operational condition encounters a safeguard that was not calibrated for it, at which point it either fails to block a harmful action or incorrectly blocks a benign one, with no warning that calibration drift had been accumulating.
The gap in current AI safety practice
Model-level evaluation assesses what the model does in response to inputs. It does not assess the control structure within which the model operates, the distribution of accountability across that structure, or how operational variability interacts with safeguard calibration over time.
Carlucci, Fillingham, Walpole, Kryś, arXiv:2606.13474
The paper's central recommendation is methodological: frontier AI governance should pair model-level hazard analysis with systems-level hazard analysis as a mandatory component of safety assessment, not an optional supplement. The three methods applied in this paper provide a concrete starting point for what that analysis would include.
ICML (International Conference on Machine Learning) is one of the premier venues in machine learning research. Acceptance at its Technical AI Governance Workshop indicates that this systems-thinking approach is receiving serious attention from the research community, not only from safety-focused institutions. The paper's audience at ICML includes the researchers building the systems that need this kind of analysis.
Why this matters beyond AI labs
The three risks identified in this paper are not unique to coding agents. The governance verification gap applies wherever a deploying lab is the primary source of compliance information. The monitoring delay problem applies wherever AI agents operate faster than human oversight can track. The calibration drift problem applies wherever safeguards calibrated at deployment time encounter operationally shifting conditions over months or years of deployment.
These are structural features of how frontier AI is currently deployed, not quirks of any particular product or lab. The systems-thinking methods applied in this paper provide a framework for analyzing these structural features systematically, which is a prerequisite for regulating them meaningfully.
Current AI governance proposals focus primarily on model capabilities and model alignment: what can the model do, and is it behaving safely in response to inputs? This paper argues those are necessary but not sufficient evaluations. The systems-level analysis reveals hazards that exist in the gap between what a model is evaluated on and the conditions it actually operates under. Scaled up from a single deployment to a whole economy, that same blind spot is what makes gradual disempowerment possible: every model passes its evaluations while the system as a whole drifts out of human control.
The full paper is available at arxiv.org/abs/2606.13474.
Common questions.
What is STPA and why is it relevant to AI safety?
STPA (System Theoretic Process Analysis) is a hazard analysis method developed at MIT for complex sociotechnical systems. It maps control structures (who commands what, who monitors what, how information flows) rather than analyzing individual component failures. It is standard in aviation, nuclear power, and aerospace for identifying how a system can lose control even when all individual components work as designed. A June 2026 paper accepted at ICML applied STPA along with FRAM and STECA to a frontier AI coding agent and found that the control structure failures it identified were entirely invisible to standard model-level AI safety evaluation.
What is AI loss of control as defined in this paper?
The paper defines AI loss of control as the erosion of meaningful human oversight over AI system behavior, distinct from a single dramatic failure event. In the context of a frontier coding agent, loss of control occurs through three accumulating mechanisms: governance responsibilities that cannot be externally verified, monitoring delays that undermine the ability to intervene before harm occurs, and operational variability that erodes safeguard calibration accuracy over time. Each mechanism alone is manageable; all three interacting together constitute systemic loss of control.
What are the three risks found by the systems-thinking analysis?
Three risks that standard model evaluations do not detect. First, the governance verification gap: accountability relies substantially on self-reported compliance by the deploying lab, with no independent external verification mechanism. Second, the monitoring delay problem: the latency between an AI action and a human alert is long enough that intervention often arrives after the harm pathway is already established. Third, operational variability drift: safeguard calibrations accurate at deployment erode as operational conditions shift over months of deployment, without triggering any alert.
Why do model-level AI safety evaluations miss systemic risks?
Model-level evaluations test what a model does in response to inputs: whether it produces safe outputs, refuses harmful requests, and stays within constraints. They evaluate the model as a component. Systemic risks emerge from the structure connecting components together: how oversight responsibilities are distributed, what information flows where at what speed, and how operational context changes control effectiveness over time. These structural properties are not tested by evaluating model outputs because they exist at the deployment system level. The June 2026 ICML paper argues that model-level evaluation must be paired with systems-level hazard analysis as a mandatory component of frontier AI governance.