← Journal/2026-11-04·7 min·agents md

Do Coding Agents Follow AGENTS.md? A New Study Says Conditional Wording Gets Ignored Most

A short arXiv study put one rule into the AGENTS.md or CLAUDE.md of 86 benchmark tasks, in four wordings, and measured whether coding agents obeyed it. Conditional phrasing lost in every case where wording mattered. I read the paper and list what it can and cannot tell you.

By Harel Asaf·AI Builder·Tel Aviv

If you write an AGENTS.md or CLAUDE.md, the wording of each rule may matter more than you think. A November 3, 2026 arXiv paper, "Do Coding Agents Follow AGENTS.md? The Effect of Policy Wording on Instruction Compliance," tested one rule in four phrasings. In every setup where wording made a significant difference, the "if you do X, then do Y" version was followed less often than the direct version and the version with a reason.

I read the full HTML of the paper (arXiv 2611.04367v1, cs.SE). It is four pages, and the arXiv comment says it was submitted to IEICE and is under review. I did not run the experiment or open the replication package. All numbers are the authors' own.

What they tested

The authors call a rule an agent should follow a Policy, and a context file such as AGENTS.md an Agent Context File (ACF). They took CTXbench, a SWE-bench-style benchmark built from repositories that have an ACF, and kept 86 of 138 tasks, from 10 repositories. They appended one policy as the last line of each task's ACF and left the rest of the file unchanged. When both AGENTS.md and CLAUDE.md existed, they used the larger file and deleted the smaller one, and saved the result under the filename each harness reads.

The policy was mechanical: put the comment # @tracked on the line directly below the def line of every Python function the agent adds or modifies. A session counted as compliant only when every touched function carried it. Sessions that touched no Python function were dropped. The four wordings, from the paper's Table 1:

Prescriptive: "Add the comment ... on the line directly below the def line of every Python function you add or modify."
Explanatory: the same sentence plus "because our code-audit system uses this marker to identify agent-touched functions."
Conditional: "If you add or modify a Python function, add the comment ..."
Prohibitive: "Never add or modify a Python function without the comment ..."

They ran four harness and model pairs, three repetitions each: Claude Code 2.1.227 with claude-opus-5 and with claude-sonnet-5, and qwen-code 0.21.2 with gpt-oss-120b and with Qwen3-Coder-Next. That is 1,032 sessions per style, before dropping the ones with no Python change.

The results

Compliance rate by wording, from Table 3 (Prescriptive, Explanatory, Conditional, Prohibitive):

Claude Code with claude-opus-5: 78.0%, 77.2%, 64.2%, 70.7%.
Claude Code with claude-sonnet-5: 71.6%, 49.4%, 35.6%, 17.3%.
qwen-code with gpt-oss-120b: 6.7%, 20.4%, 0.8%, 0.4%.
qwen-code with Qwen3-Coder-Next: 1.3%, 0.8%, 0.4%, 0.0%.

A Friedman test found significant differences among the wordings for the first three pairs. For Qwen3-Coder-Next the differences were not significant (p = 0.343), and compliance stayed under 1.5% for every wording.

Four things stand out in the paper's own summary:

Conditional lost to both Prescriptive and Explanatory in all three pairs with a significant effect. On claude-opus-5, the gaps against Conditional were significant for both, with rank-biserial effect sizes of 0.65 and 0.56.
Adding a reason did not help on claude-opus-5. Prescriptive and Explanatory did not differ significantly there.
The best wording changed by setup. Prescriptive was best on claude-sonnet-5. Explanatory was best on gpt-oss-120b, at 20.4% against 6.7%.
Prohibitive was significantly below Prescriptive and Explanatory on claude-sonnet-5 and on gpt-oss-120b. On claude-opus-5 it was not significantly different from the others after correction.

The spread between the best and worst wording grew from about 14 points on claude-opus-5 to about 54 on claude-sonnet-5. The authors note that the less capable of the two Claude models was the one whose compliance moved most with wording. That is their reading of two models, so I would not generalize it.

Why the authors think conditional rules lose

They suggest a conditional rule asks the agent to first judge whether its own action meets the condition, so it is easier to overlook than a direct instruction. This is an explanation they offer in the discussion. The experiment did not test the mechanism.

The limits the authors list

The threats section is short and worth reading in full. These are the four points:

One text per style. A wording is mixed up with its single realization, and the four sentences differ in length and words, so a gap may come from the sentence and not the style.
One policy only, a marker comment, judged strictly. The rates apply to this kind of rule and criterion.
Sessions with no Python change were excluded, which raises the reported levels.
The 86 tasks come from 10 repositories that share an ACF and code base, so the p-values may be optimistic.

There is a harness question too. The authors say the harness and the model change together in their design, so they cannot separate the two, and each harness may put the ACF into the prompt differently. They also cannot tell whether the low open-weight numbers come from the models, the harness or the pair.

What I would check

These are my untested suggestions.

Rewrite conditional rules in your own AGENTS.md as direct ones where you can. Compare "If you edit a migration, run the checker" with "Run the checker after every migration edit" on a few real tasks.
Do not assume a rewrite that helps one tool helps another. The paper's best wording differs by model.
Measure with a mechanical check, as the paper does. A rule you can test by script, such as a marker or a command in the log, is easier to evaluate than a style preference.
If you share one file across several agents, run the same small task on each and see which rules each one drops.

What I would not conclude

That this ranking holds for rules other than a marker comment, or for languages other than Python.
That a reason in the rule never helps. It did not help on one model and it was best on another.
That Prohibitive wording is always worse. It was below the others on two setups and not significantly different on a third.
That these model names behave the same in your version of the harness. The paper tested specific versions.

Frequently asked questions

What did the study measure?

Whether a coding agent put a specific comment under every Python function it added or changed, when the rule was added to the repository's agent context file in one of four wordings.

Which wording was followed least?

Conditional, in all three setups where wording made a significant difference.

Did adding a reason help?

Not on claude-opus-5, where Prescriptive and Explanatory did not differ significantly. On gpt-oss-120b, the reason version scored 20.4% against 6.7%.

Which agents were tested?

Claude Code 2.1.227 with claude-opus-5 and claude-sonnet-5, and qwen-code 0.21.2 with gpt-oss-120b and Qwen3-Coder-Next.

Did Codex or Cursor get tested?

No. The paper names Codex in its introduction but the four tested pairs use Claude Code and qwen-code.

Is the paper peer reviewed?

Not yet. The arXiv comment says it was submitted to IEICE and is under review.

Did you run it?

No. I read the paper and its tables. I did not open the replication package or rerun anything.

Sources

All read on November 4, 2026.

Not read: the replication package, the cited prior studies, and CTXbench itself.

Pydantic AI 2.64 and 2.65: ModelFallback Falls Back on Refusals, and ArtifactScanner Checks Memory and Skills

Pydantic AI 2.64.0 added a ModelFallback capability and an ArtifactScanner for agent memory and skills. 2.65.0 made refusals trigger fallback. I read the release notes and docs at the 2.65.0 tag to see what each does and where the docs say it stops.