The Agent You Deployed Last Month Has No Memory of What It Got Wrong
Somewhere in your organisation, an LLM agent has file system access. It can read directories, move files, execute terminal commands, call APIs, fill out web forms on behalf of users. You gave it guardrails. A model-based safety filter, probably, running Claude-3.5-Sonnet or GPT-4o with a customised prompt telling it to flag dangerous actions. You checked a box. The agent went live.
What you may not know is that your safety filter is blocking roughly half of everything the agent tries to do legitimately, while simultaneously letting environment-injected attacks through at rates between 15% and 85%, depending on the attack type. That is not a worst-case hypothetical. That is the measured performance of the current generation of model-based guardrails tested against real OS agent tasks. The Claude-3.5 zero-shot baseline preserves only 50% of normal actions. LLaMA-Guard 3 achieves 0% accuracy on prompt injection detection in the same environment, meaning it blocks nothing. These two failure modes, over-restriction and under-detection, have been treated as separate problems to be balanced. Researchers at The Ohio State University, the University of Wisconsin-Madison, and UC Davis argue they share a single root cause: the guardrail does not learn.
Their system, AGrail, is a lifelong agent guardrail that treats safety checking as an iterative optimisation problem rather than a static classification task. Using two cooperative LLMs and a persistent memory module, it refines its own safety checks over time, converging toward an optimal set of verification policies for each type of agent action. Tested across OS agents, web navigation agents, and electronic health record access-control agents, AGrail with Claude-3.5-Sonnet preserved 95.6% of benign actions while driving prompt injection attack success rates to 0% on both Safe-OS and AdvWeb, and reducing system sabotage success to 3.8%. The gap between 50% benign preservation and 95.6% is not a tuning improvement. It is the difference between a guardrail that trusts its first impression and one that has learned from every action it has ever seen.
Why Every Existing Guardrail Is Frozen at Birth
The core problem with current approaches is architectural, not parametric. Adjusting the prompt does not fix it.
Model-based defenses, meaning advanced LLMs like GPT-4o or Claude-3.5-Sonnet given chain-of-thought safety prompts, apply the same reasoning process to every agent action regardless of what they have encountered before. They have no mechanism to remember that a particular class of file-move command was flagged incorrectly last time, or that a specific web form interaction pattern is consistently benign. Each evaluation starts from zero.
GuardAgent, the closest prior work, introduced multi-agent reasoning and a knowledge store for safety checking. But its knowledge store relies on manually specified trusted contexts. When the downstream task changes, the context has to be rewritten by a human. It cannot adapt to a new task environment on its own. Conseca attempted adaptive policy generation using LLMs, but LLM-generated policies carry the same failure modes as any single-pass LLM output: they can be simultaneously too restrictive for some action classes and too permissive for others, with no mechanism to detect or correct either error.
The result is a field where every deployed guardrail is, in effect, frozen at the moment of its configuration. It cannot get better. It can only be manually reconfigured.
The Mechanic Behind a Guardrail That Gets Better Over Time
AGrail's architecture is built around three interlocking ideas. Each one resolves a specific failure mode in existing systems.
| AGrail Component | What It Solves | What Breaks Without It |
|---|---|---|
| Memory-keyed safety checks | Persistent learning across agent actions; carries knowledge across tasks | Every evaluation starts from zero; the same mistakes repeat indefinitely |
| Dual-LLM Analyzer-Executor loop | Checks are generated AND independently validated before memory update | Generated checks may be redundant, incorrect, or block legitimate actions without detection |
| Test-time adaptation (TTA) | Safety check quality improves continuously during deployment | Guardrail performance is bounded by initial configuration quality |
The Analyzer, the first LLM, retrieves existing safety checks from memory for the relevant action type, revises or merges redundant checks, and adds new checks based on the current safety criteria and guard request. The Executor, the second LLM, takes the Analyzer's output and does three things: it deletes any check it determines to be redundant or incorrect, it invokes external validation tools when programmatic verification is possible, and it performs reasoning-based validation for checks that do not need a tool. It then updates the memory store.
The memory module is keyed by a generalised representation of the agent action, not the literal command string. A technique called step-back prompting converts the raw agent action into both a natural language description and a tool command form. The concatenation of both forms becomes the memory key, which means the system can retrieve relevant prior safety checks even when the specific syntax of the action differs. The threshold for triggering a memory overwrite is a cosine similarity above 0.8 with an existing key.
What makes this reproducible, not just adaptive, is that independent runs converge to the same safety checks. Across three random seeds with different data sequences, the TF-IDF similarity between memory stores exceeds 98% after 4 iterations for complex actions on the Mind2Web web navigation task. This is not a system that finds a local optimum that happens to work for one deployment. It is converging toward something close to an objective optimum for each action class.
The Performance Ceiling Depends on Which LLM You Use
AGrail is honest about a real constraint. The framework uses off-the-shelf LLMs as its reasoning components. It does not train a purpose-built guardrail model. That means the performance ceiling is set by the underlying model's reasoning capacity.
The gap between Claude-3.5-Sonnet and GPT-4o-mini is significant, and the variance data makes clear why model choice matters operationally. Claude-3.5-Sonnet running AGrail achieves ±1.2 percentage points of variance in label prediction accuracy across three random seeds on Mind2Web-SC. GPT-4o-mini achieves ±8.3 percentage points under the same conditions. At production scale, with thousands of agent actions per day, ±8.3 variance is not a statistical footnote. It means your guardrail's reliability is meaningfully unpredictable depending on the order in which actions happen to arrive.
The authors flag this directly: training a purpose-built guardrail model is future work. The current system's reasoning quality is bounded by what you put into it. Use a frontier model, get frontier stability. Use a smaller model to cut inference costs, and the safety guarantees degrade in ways that the aggregate numbers do not fully communicate.
How the Framework Processes a High-Stakes Action
The operational sequence is worth walking through concretely, because the architecture only makes sense when you see what happens at each decision point.
- An OS agent generates an action: move a file from one directory to another, for example.
- AGrail intercepts the action before it executes in the environment.
- The Analyzer retrieves the closest matching safety checks from memory using the generalised action representation as a lookup key.
- The Analyzer generates or modifies a checklist of safety verification items, informed by the current task's safety criteria and any guard requests the administrator has specified.
- The checklist passes to the Executor.
- The Executor evaluates each check. For a file-move action, it may invoke the OS Environment Detection Tool, a Claude-3.5-Sonnet-generated Python program that executes within the agent's Docker environment to retrieve system paths, file permissions, and user identity.
- If the programmatic check returns a result inconsistent with the action being safe, the Executor flags the action as unsafe. If no tool is needed, it reasons directly.
- Any check the Executor determines to be redundant or incorrectly blocking a legitimate action is deleted from the checklist and the memory is updated to reflect that deletion.
- If every check returns safe, the action proceeds. If any check returns unsafe, the action is blocked and the agent is notified.
The memory update at step 8 is what separates this from a standard dual-LLM evaluation pipeline. The system is not just checking safety. It is building a progressively more accurate model of what safety checking for this action class actually requires.
What This Looks Like at the Healthcare Access Control Boundary
The EICU-AC scenario, drawn from electronic health record agent interactions, is the highest-stakes test environment in the paper. The agent has access to patient diagnostic data. The task-specific risk is unauthorised access: an agent action that retrieves records the requesting role is not permitted to see.
AGrail with Claude-3.5-Sonnet on EICU-AC achieves 98.4% label prediction accuracy, 97.0% precision, and 100% recall, meaning it catches every unauthorised access attempt while correctly approving 97% of legitimate requests. The F1 score is 98.5. GuardAgent, the prior state-of-the-art purpose-built multi-agent guardrail, achieves 98.7% accuracy with a manually specified trusted context. AGrail reaches near-equivalent performance without manual context specification, and it does so using a memory that was in some tests transferred from a completely different domain.
When memory trained on Mind2Web web navigation tasks was transferred directly to EICU-AC healthcare access control, Claude-3.5-Sonnet showed no performance degradation. It achieved 100% across all metrics. GPT-4o showed a 5.7-point F1 drop after transfer, which suggests the cross-domain generalisation benefit is model-dependent. But the Claude-3.5 result means a single deployed AGrail instance could potentially serve multiple agent deployments, carrying safety knowledge from one domain into another without retraining or manual reconfiguration.
The agreement rate results across all systemic risk scenarios underscore the practical magnitude of the improvement. AGrail outperformed baselines by an average of 46.2 percentage points in agreement rate, the metric capturing how often the guardrail's safety determination aligns with ground truth across both safe and unsafe actions.
What Your Deployment Decision Actually Determines
This paper is an architectural framework with a working implementation, not a research prototype. That changes what the appropriate executive response looks like.
The first decision is model selection, and it is not negotiable on quality grounds. If you are deploying AGrail or any comparable reasoning-based guardrail on a high-stakes agent, the underlying LLM must be a frontier model. The variance data on GPT-4o-mini is a clear signal that smaller models introduce safety unpredictability that the framework cannot compensate for through learning alone. The cost of using a smaller model to reduce inference spend is paid in guardrail reliability, and that cost is asymmetric: one missed prompt injection in a system with file write access is not recoverable by pointing to the average ASR.
The second decision is about what the guardrail is protecting. AGrail's tools are purpose-built for specific environments: an OS detection tool for terminal agents, a web HTML tool for browser agents, a permission detection tool for healthcare access control. The framework can reason without tools, but tool-backed validation is what drives the most extreme results, specifically the 0% prompt injection ASR on Safe-OS and AdvWeb. Deploying AGrail without investing in tool development for your specific environment leaves performance on the table.
The third decision is about patience with the learning phase. The AdvWeb GPT-4o error analysis shows bypassed attacks in the first 30 steps before ASR dropped to 0%. The system needs enough interactions to converge. In a low-volume agent deployment, that convergence may take days or weeks of real use. In a high-volume deployment, it may take hours. Either way, the safety posture at day one is not the safety posture at day 30, and organisations need monitoring infrastructure that can track whether the memory is converging toward the right checks rather than assuming the guardrail is either working or not.
The deepest implication of this paper is not about the specific numbers. It is about what kind of object a safety layer should be. Static safety filters are configuration artifacts. They reflect the threat model of the person who wrote them on the day they wrote it. A lifelong guardrail is infrastructure that accumulates operational knowledge. The organisations that treat agent safety as a learned capability rather than a checked box will have a qualitatively different risk profile in 18 months. The gap between those two positions compounds the same way the memory does: iteration by iteration, action by action, until the distance is no longer closeable by manual reconfiguration alone.
Agents Applied covers the research that changes how operators build and deploy AI systems. Forensic extraction by the editorial team.