When a conversational AI gives a wrong answer, a user can notice and ask again. When an autonomous agent takes a wrong action, the consequences may compound across subsequent steps before anyone realizes something has gone wrong.
This is the central observation driving "Towards Trustworthy Agentic AI: A Comprehensive Survey of Safety, Robustness, Privacy, and System Security" (arXiv:2605.23989), submitted to arXiv on May 17, 2026 and published in Academia AI and Applications, vol. 2, 2026. Twelve researchers, led by Jinhu Qi and including Irwin King, systematically catalogued how agentic AI systems fail, organized the mitigation landscape, and identified where current defenses remain inadequate.
What makes agentic AI fundamentally different
A standard LLM takes a prompt and produces a response. An agentic AI system takes a goal and runs: planning, calling tools, storing intermediate results in memory, executing multi-step action sequences, and adapting based on what it observes along the way.
This architecture creates failure modes that prompt-response evaluation cannot catch. A single wrong tool call can corrupt all downstream steps. Memory can accumulate false beliefs that compound across a session. Long-horizon tasks can produce harmful effects from the interaction of individually harmless steps. The agent operates in an environment that changes as it acts, and those changes can interact with future behavior in ways neither designers nor users anticipate.
The survey covers all of this through two core dimensions: Safety and Robustness, and Privacy and System Security.
Safety and robustness
The safety section covers risks that emerge from the agent's own behavior across its multi-step trajectory. Adversarial robustness in agentic settings extends beyond standard prompt attacks to include prompt injection embedded in environmental observations. A malicious document the agent reads mid-task can redirect the agent's behavior for the remainder of the session without the user's knowledge.
Robustness under distribution shift is a particular concern. An agent trained and tested in a controlled environment may encounter tool behaviors, user requests, or environmental conditions at deployment that deviate from training in subtle ways. A conversational AI that encounters a distribution shift simply produces a suboptimal response. An agent in the same situation takes a sequence of confident, compounding actions that may progressively worsen the situation.
The survey introduces a unified metrics-and-benchmarks hub covering both outcome signals (did the task complete successfully?) and process signals (were constraints violated during execution, were action traces complete, what fraction of adversarial attempts succeeded?). Most current benchmarks measure only whether agents achieve their goals, not whether they respected safety constraints along the way. A system can complete tasks successfully while violating constraints on every run, and current evaluation will not catch it.
Privacy and system security
The security section covers what happens when an adversary actively controls part of the environment the agent operates in. The survey documents real-world security failure cases from open-source agentic systems, which provide concrete attack surfaces rather than theoretical ones.
Agents with persistent memory and long-horizon access create privacy exposure that conversational AI does not. An agent performing research tasks across a session may accumulate sensitive information in its working memory that can then be extracted through prompt injection or leaked through tool calls to third-party services. Memory poisoning (corrupting stored context to influence future behavior) represents an attack vector specific to agentic systems with no equivalent in single-turn AI deployment.
Multi-agent systems extend this attack surface further. One compromised agent can propagate an attack to others through the normal communication channels of the system, spreading influence without triggering individual agent safety filters that were designed to evaluate inputs from users, not from trusted peer agents.
What the survey identifies as unsolved
Why this survey matters for developers and deployers now
Agentic AI deployment is accelerating faster than the safety literature's ability to provide unified guidance. Teams building production systems with autonomous agents handling customer data, executing financial transactions, or coordinating external services are working from a fragmented literature that covers pieces of the risk surface without integrating them.
This survey provides that integration. The unified metrics hub in particular offers a starting point for evaluation programs that cover both behavioral safety and security resistance, rather than treating these as separate concerns handled by different teams with different frameworks. The real-world case studies from open-source systems connect theoretical attack categories to documented concrete failures.
The full paper is available at arxiv.org/abs/2605.23989.
Common questions.
What is the difference between AI agent safety and AI agent security?
Safety and robustness concerns focus on how an agent's own behavior can produce harmful outcomes: adversarial inputs causing unsafe outputs, distribution shift causing confident wrong actions, and constraint violations during multi-step execution. Security concerns focus on how external actors exploit the agent: prompt injection in environmental observations, memory poisoning to corrupt future behavior, tool interface exploitation, and multi-agent attacks where one compromised agent spreads attacks to others. Both are essential for trustworthy deployment, but require different mitigation approaches and are often siloed between teams.
What are the biggest unsolved challenges in agentic AI safety?
The May 2026 survey identifies three major open problems. Self-evolving agents that modify their own behavior during deployment can drift from safety properties in unmonitored ways. Runtime monitoring remains primitive, most deployed agents lack continuous verification that safety constraints are being respected during execution. Privacy-preserving personalization is unsolved at scale: agents need persistent memory to be useful, but mechanisms for doing this without creating exploitable memory stores or cross-session information leakage remain early-stage.
What is the self-evolving agent problem?
Self-evolving agents update their own behavior based on operational experience during deployment. Standard safety evaluation tests a fixed system at a point in time. A self-evolving agent that passes initial safety certification may drift from those properties as it accumulates experience, and current evaluation methods cannot continuously verify that safety properties are maintained as the system evolves. This is one of three primary unsolved challenges identified in the May 2026 survey.
Why does agentic AI have a fundamentally different failure surface?
Conversational AI fails in isolated, visible ways: a wrong response that a user can notice and correct. Agentic AI fails across time and causation: a single wrong tool call corrupts downstream steps; memory accumulates false beliefs that compound; long-horizon task failures emerge from the interaction of individually harmless steps; prompt injection embedded in environmental observations can redirect the agent mid-task without user knowledge. The agent also changes its environment as it acts, creating feedback loops that amplify initial errors in ways static response evaluation cannot model.