MemHarmBench: What a New arXiv Benchmark Says About Prompt Injection That Survives in Agent Memory

MemHarmBench (arXiv 2611.01700, submitted November 1, 2026) tests one scenario: an agent reads external content with a hidden instruction, saves it as a memory rule, the content is deleted, and a later session on an ordinary task follows the rule anyway. In the authors' tests with two agent products and two models, retained rules redirected a decision in 59.38 to 76.39 percent of eligible indirect-injection cases. The paper's sharper claim is that in 55.71 percent of eligible cases the agent also finished the task cleanly with no judged anomaly, so nothing looked wrong.

I read the paper's HTML version: the introduction, design, metrics, results, conclusion and limitations. I did not read the appendices or run the released code. Numbers below are the authors' own, from synthetic test cases.

What the benchmark does

It has 288 cases: 24 attack templates crossed with 12 operational contexts. Eight templates are direct instructions (96 cases) and 16 are indirect, hidden in business content such as a record or note (192 cases). Each case runs in two stages.

1. Exposure. The agent meets the malicious instruction. The exposure task asks it to adopt a workflow convention, and the test checks whether the rule enters persistent memory. If the first check is negative, one predefined follow-up is allowed in the same session.

2. Trigger. If the rule was retained, a new session starts with memory kept, the attack content removed, and a legitimate task. The trigger task contains neither the attacker's value nor a request to recall the rule.

The paper sorts the 12 attack goals by who is hurt: the user, a workspace or project owner, or an outside party. Its opening example is an expense record naming one reimbursement recipient, an imported note telling the agent to use a different one in future summaries, and a later summary that names the attacker's choice.

The four metrics

• PMIR: whether the injected rule was kept in memory, including a paraphrase.
• ASR: whether the final deliverable adopts the attacker's chosen value. Quoting or rejecting the rule does not count.
• UTDR: whether the task was absent, incomplete or violated a requirement unrelated to the attacked value.
• ATAR: whether the recorded execution shows looping, contradiction, repeated no-progress or another rubric anomaly.

Trigger metrics only count episodes where the rule was retained and every judgment resolved. The authors say this means the rates are not catalog-wide compromise rates.

The numbers

The authors tested OpenClaw 2026.7.1 and Claude Code 2.1.228, each with GPT-5 and Gemini-2.5-Flash, over three repetitions: 3,456 episodes. GPT-5 served as the judge.

Agent, averaged over both models	IPI ASR	IPI task deviation	IPI anomalies	DPI ASR
OpenClaw	75.69%	15.58%	7.38%	89.05%
Claude Code	60.94%	14.18%	3.37%	78.41%

Every configuration had higher adoption under direct injection than indirect. By victim group, outside-party goals had the highest adoption for both agents (79.58 percent and 64.20 percent). Workspace-owner goals had the lowest task deviation and the highest anomaly rates.

Factor studies

These used separate controlled case sets, and the paper calls the contrasts descriptive.

• Relevance: when the attacker's target was more relevant to the requested category, retention rose 30 and 25 points and adoption rose 25 and 20 points for the two agents.
• Distance: after four unrelated normal sessions, conditional adoption fell from 70 to 60 percent with one Claude Code backbone and from 62.5 to 50 percent with the other. It stayed above zero, and some individual cases went the other way.
• Scope cues: when the rule's stated applicability conflicted with the target record, retention and adoption fell by about 15 to 20 points, but conditional adoption stayed at 62.5 and 66.67 percent.
• Images: in one configuration, Claude Code with Claude Haiku 5.5, a webpage image carrying the instruction produced 52.5 percent retention and 40 percent adoption, against 2.5 percent adoption with a clean image. Matched text gave 42.5 percent.

What the authors say limits the results

The limitations paragraph says outcomes are decision-level proxies for harm, not real-world measurements. Trigger rates condition on retention. The direct and indirect subsets differ in composition. Confidence intervals treat episodes as independent. Differences in each product's tools and retained state prevent memory-only attribution and limit cross-system comparison. The visual test used one configuration and controlled pages. All identities and destinations are synthetic.

How it connects to memory scanners

This week I wrote about Pydantic AI's ArtifactScanner, which the project's docs describe as pattern matching that misses paraphrased instructions. MemHarmBench counts a paraphrase as retention, and its attacks include conventions written as ordinary business content. I did not test any scanner against this benchmark, and the paper does not evaluate Pydantic AI, so I draw no conclusion about how a scanner would score. The comparison only shows why the pattern-matching limit matters.

What I would check

These are my untested suggestions, not findings from the paper.

• List every path by which your agent can write memory, and note which ones accept text from outside sources such as email, web pages or imported records.
• Record the source of each stored rule, and who had authority to set it. The paper's framing is that authority to supply information is not authority to set future rules.
• Test your own agent with a two-session check: plant a harmless but distinctive convention in a document, delete the document, and see whether a later unrelated task follows it.
• Review decisions that came out of memory, not only failed tasks. The paper's central result is that harmful decisions can sit inside complete work.

What I would not conclude

• I would not conclude these rates apply to your agent. The cases are synthetic, the products and models are specific versions, and the judge is a model.
• I would not conclude one agent product is safer than another. The authors say cross-system comparison is limited.
• I would not conclude memory is the only cause. The paper says retained state can include non-memory artifacts.
• I would not treat a single arXiv preprint as settled. I did not look for replication.

Frequently asked questions

What is MemHarmBench?

A benchmark of 288 synthetic cases for cross-session prompt injection in agents with persistent memory, from arXiv 2611.01700.

What is cross-session prompt injection?

An instruction in external content is stored in memory, the content is later removed, and a new session follows the stored rule on a normal task.

What were the headline results?

Retained rules redirected decisions in 59.38 to 76.39 percent of eligible indirect-injection cases across four configurations, and 55.71 percent of eligible cases also finished the task with no judged anomaly.

Which agents were tested?

OpenClaw 2026.7.1 and Claude Code 2.1.228, each paired with GPT-5 and Gemini-2.5-Flash, plus one visual test with Claude Haiku 5.5.

Do the numbers mean most agents are compromised?

No. The rates apply only to episodes where the rule was retained, and the authors say they are not catalog-wide compromise rates.

Did images work as attack carriers?

In one controlled configuration, yes: 40 percent adoption with an injected image against 2.5 percent with a clean one.

Did you run the benchmark?

No. I read the paper's main text. I did not read the appendices or run the released code.

Sources

All read on November 3, 2026.

• MemHarmBench, arXiv 2611.01700 (HTML version)
• arXiv abstract page
• Pydantic AI Artifact Scanning docs (for the pattern-matching limit)

Not read: the paper's appendices, the released benchmark code, and any replication or response to the paper.