Long-document AI scores track how much it reads: new outer code lifted the same models 39 points on a diligence review

Key takeaways

On September 8, Harvey, a legal AI company, published a set of experiments. Seven AI models reviewed simulated M&A documents on a buyer's behalf, up to 5,000 per deal, far more than a model can take in at once. With the models unchanged and only the layer of code around them swapped (this piece calls it the harness), the average share of expert-listed review items that passed rose from 23.3% to 62.4%. The more of the material that got read, the higher the score.

After checking the sources, this publication adds two points. Two off-the-shelf AI coding agents scored low because, on default settings, they stopped reading too early; that does not mean they cannot do the job. And once the harness is right, models still differ widely: the open-weight model Harvey trained further still trails the best commercial model. So for an AI company serving a single industry, the hard-to-copy edge is not the harness code, and for now not a model the company trained further. This piece infers it is the standard that grades a review, plus the simulated data rooms built on it. All figures come from Harvey alone, and the documents are simulated.

One: First, a few terms: data room, agent, harness

A data room is the large set of documents a seller opens to a buyer for review during an acquisition. The buyer's lawyers search it for risks; this work is called due diligence. Harvey generated simulated data rooms with code, each with a checklist written by experts.

An agent is an AI program that can search, write code and call tools on its own to finish a task. The harness is the layer of code around a model that decides how the model reads, which tools it calls and when it stops.

Harvey compared three approaches:

• Simple tool loop: one agent searches on its own, picks documents to read, and writes the memo when it judges it has enough. This is the standard setup of Harvey's own legal AI benchmark.
• Recursive harness: the root agent does not read documents itself. It writes code that splits the data room by category and hands each slice to a sub-agent to read; the sub-agents return their findings, and the root agent compiles them into a memo. The approach comes from "Recursive Language Models", a late-2025 paper by Alex Zhang and colleagues at MIT; this piece calls it the original paper.
• Off-the-shelf general coding agents: Anthropic's Claude Code and OpenAI's Codex. Both were built for writing code but can also search large document sets with file tools.

The pass rate is the share of checklist items the memo passes: another AI acts as judge and checks the memo against the expert checklist item by item.

Post-training means taking an existing model and training it another round on data or rewards for a specific task. A frontier model is one of today's most capable commercial models. An open-weight model is one whose model files are public and can be run on one's own servers.

Two: The same seven models gain 39 points on average in a different harness

Harvey set aside 50 simulated data rooms purely for testing, unused when models were trained further (sections four and six); unless noted otherwise, every score here comes from this set. The seven models include Anthropic's Claude Opus 5 and OpenAI's GPT-5.6 Sol. In the recursive harness each takes a turn as the root agent, while the sub-agents always use the same small open-weight model. In the simple tool loop, the average pass rate was 23.3%; in the recursive harness, 62.4%, a gap of 39.1 percentage points. The top scorer was Opus 5, rising from 42.5% to 77.6%. Per-model scores are in the appendix.

The off-the-shelf general coding agents did worse still: Claude Code (running Opus 5) scored 24.6% and Codex (running GPT-5.6 Sol) 12.0%, 17.9 and 4.6 points below the same models in the simple tool loop.

Before reading these numbers: the data rooms are simulated, not real deals; the judge is an AI, and Harvey does not report how closely it agrees with lawyers; the reward used in post-training (sections four and six) also came from the same judge, which amounts to training the student on the examiner's own standard, so part of the post-training gains may simply be a better fit to the judge's taste; and Harvey calls its training experiments "initial explorations", with no claim of product deployment.

Three: Scores track how much gets read

In every run, Harvey measured the share of the data room's content that actually reached a model. The simple tool loop never exceeded 1%, and most runs read only 0.1% to 0.5%; the recursive harness read more than 10% in nearly every run, and most runs read close to all of it. Harvey's conclusion: the more read, the higher the pass rate.

That makes sense in due diligence. Some checklist items can be found only by combining several documents; others require confirming that a document that should exist does not: a cabinet nobody opened cannot be declared empty.

But there are two limits:

• Handing reading to sub-agents helps, but more layers are not always better. In a small extra experiment, Harvey let sub-agents delegate further, adding one more layer (root and sub-agents on the same open-weight model): of fourteen data rooms, four improved and ten got worse, and the average across all fourteen fell by 19 points; in four of the ten that got worse, the agent read the material but never wrote a report. So failures also happen at the writing stage; how much is read is not the only variable. Two March papers unrelated to Harvey point the same way. One reimplements the original paper (this piece calls it the reproduction paper; its tests are much smaller): adding a second layer made results worse, and run time jumped from 3.6 seconds to 344.5 seconds. The other (the self-check paper) drops recursion: the model writes several programs for searching the text, then picks one by how sure it is (whether repeated answers agree, its own stated confidence and so on), beating the original paper's recursive approach by up to 22% in the same time.
• Reading more and scoring higher only rise together; cause is not proven. Agents that divide the work well might read more and write better anyway; Harvey reports only that the two are correlated and has not ruled out this explanation.

Four: General agents lost this time on default settings, which does not prove they cannot do it

The numbers in section two invite a conclusion: treating a general coding agent as the all-purpose harness is the wrong choice. But look at what was measured.

Harvey gave Claude Code and Codex the same minimal instruction as the simple tool loop. In the traces, "both agents stop reading early and write shorter memos", and neither spawned sub-agents although both could. That is a reading of default behaviour, not of the ceiling of ability.

Supporting evidence: Z.ai's open-weight model GLM-5.2, as a root agent, also often gave up early and under-delegated at first. Harvey picked out GLM-5.2's runs that succeeded and read a lot, and used them to train the same model further. Before-and-after, on the same twenty data rooms: the score rose from 46.1% to 60.1%, and the number of sub-agents it spawned started to scale with the size of the data room. This set started lower: on section two's set the same untrained model already scored 65.4%, so only same-set comparisons hold. Harvey says such behaviour can be learned "from a relatively small number" of the model's own runs, with "no labeled legal knowledge" needed. This publication infers that general agents trained the same way by their makers could in principle learn it too; this has been verified only on GLM-5.2.

Model makers have already built heavy delegation into general agents. On May 28, Anthropic launched dynamic workflows in Claude Code, which spawn tens to hundreds of sub-agents in parallel; it is now generally available. Harvey does not say whether it was on during testing; the official examples are all coding, and this publication has seen no long-document measurement.

Third-party measurements are mixed, and none uses legal documents. The file-system paper (March 2026) has off-the-shelf coding agents treat long text as files and search it with their usual file tools, beating the best published results on several long-context tests. The parallel sub-agent paper (June 2026) lets a coding agent spawn sub-agents in parallel on Oolong, a synthetic data-aggregation test: it scores 81.36%, against 64.38% for the recursive approach. Those two papers favour general agents. The original paper (late 2025), by contrast, found the recursive approach ahead of the then-current Claude Code. Each paper's numbers and units are in the appendix.

So the fairer statement is: a general agent run on default settings reads too little on long documents; when explicitly asked to delegate it does not necessarily lose, and on other tests it won. This is extrapolation across tasks; Harvey did not test a general agent explicitly asked to delegate (section seven, test 1).

Five: Switching harness makes a big difference, and so does switching the root agent's model

Another conclusion the numbers invite: the gain from switching harness exceeds the gain from switching to a frontier model. Checked against Harvey's numbers, the two are about the same size, though one is an average gain and the other best minus worst.

• Switching harness: with the model fixed and only the harness changed, the seven models gained 39.1 points on average.
• Switching model: with the harness fixed as the recursive harness and only the model changed, the best of the seven, Opus 5, scored 77.6% and the worst 42.3%, a gap of 35.3 points. Harvey also paired four root agents with three sub-agents each: changing only the root agent moved the score by about 38 points on average between best and worst; changing only the sub-agent, by only about 8 points.

So once the harness is right, the root agent's model still has to be chosen well.

The root agent itself handles very little text: with Opus 5 as root, it accounted for only 3.8% of all tokens read in and 1.1% of tokens written out; the rest went to sub-agents reading documents. The grunt work of reading can go to cheap small models, but for the root agent, which decides how to split the room, whether to read all of it and how to turn hundreds of findings into a memo, the model makes a big difference.

On cost, six of the seven models became more expensive in the recursive harness, costing a few dollars to over twenty per data room after the switch; the exception was Opus 5, whose cost fell from about $18 to about $7 when it moved to the root role (approximations this publication read off Harvey's cost chart).

Six: A vertical AI company's hard-to-copy edge is neither the harness code nor a self-trained open-weight model

A vertical AI company serves only one industry, such as law. This piece's October 8 first version held that such a company's hard-to-copy edge is a harness built for the task, plus an open-weight model post-trained around it. Three points undercut that call:

1. The harness itself is public. The original paper's code library is open source under the MIT licence, so anyone may use and modify it for free. Harvey's data-room tuning is unpublished, but its post describes the approach fully.

2. Post-trained open-weight models currently trail frontier models. Harvey put Alibaba Cloud's open-weight model Qwen (not one of the seven models in section two) into the recursive harness as root agent and post-trained it; on the same 50 data rooms as section two, it rose from 29.9% to 63.0%. On that set, Opus 5 scored 77.6%, 14.6 points higher, and three more of the seven models scored above 63.0% without any post-training, including the likewise open-weight GLM-5.2. A post-trained open-weight model may win on cost and deployment control, but with no cost published by Harvey, that cannot be compared.

3. Harvey's post-training does not rely on customer data. Tenet, Harvey's own model announced in August, is built on Moonshot AI's open-weight model Kimi K3 and was post-trained together with Fireworks, a platform company that runs and trains open-weight models for businesses. Harvey states plainly: "We did not use any customer data in any of our post-training efforts."

What is left? The rest is this publication's inference, held with lower confidence. The delegating habit from section four is easy to learn, but picking which traces count as good, and how much reward to give, takes a yardstick: a grading standard that judges whether a due-diligence review is complete and correct, plus the simulated data rooms built around it. So the scarce asset may be the yardstick and the training environment, not the code that reads documents. That said, part of Harvey's benchmark question set is already open source, and this publication has established neither how much of the yardstick remains Harvey's alone nor how accurate it is (section two); channel advantages such as client trust are not measured in this piece either.

Seven: What to ask when evaluating long-document AI, and what would prove this piece wrong

Our call

When evaluating an AI product for long documents, ask not only which model the root agent uses, but also what share of the material it read and whether it handed the work out.

What would prove this wrong

All three tests run to March 31, 2027; the thresholds are observation lines set by this publication.

1. Someone reruns a general agent explicitly asked to delegate to sub-agents (or running dynamic workflows) on the same kind of due diligence, and it still trails the dedicated recursive harness by twenty points or more. Then the claim that the gap is mainly unfinished reading fails, and the conclusion at the start of section four holds.

2. A post-trained open-weight root agent matches the best frontier model. For example, GLM-5.3, a Z.ai open-weight model Harvey says it is now training, scores no lower than the best frontier model of the time (currently Opus 5's 77.6%) on data rooms not used in training. Then point 2 of section six must be rewritten.

3. A third party redoes the test with real data rooms or lawyer grading, and the link between how much is read and the score disappears. Then the claim in section three fails.

Appendix: how the numbers were obtained

Per-model scores

This publication read the labels off Harvey's per-model chart; the averages of the seven match the 23.3% and 62.4% in Harvey's text. The data rooms are the 50 not used in training; in the recursive harness the sub-agent is fixed as Alibaba Cloud's Qwen3.6-35B-A3B.

Other scores on the same data rooms: the post-trained Qwen3.5-122B-A10B as root agent, 29.9% before and 63.0% after post-training; Claude Code (running Opus 5) 24.6%, Codex (running GPT-5.6 Sol) 12.0%. In the simple tool loop the best and worst differ by 27.4 points (Opus 5's 42.5% against GLM-5.2's 15.1%).

Where each number comes from

The five papers

• Original paper: Alex L. Zhang, Tim Kraska and Omar Khattab, "Recursive Language Models", arXiv 2512.24601, first version December 31, 2025. With GPT-5 as the base, it beats Claude Code by 13% on the median of four long-context tasks.
• Reproduction paper: Daren Wang, "Think, But Don't Overthink: Reproducing Recursive Language Models", arXiv 2603.02615, March 3, 2026. Uses two test sets, S-NIAH and OOLONG.
• Self-check paper: Keivan Alizadeh and three co-authors, "Recursive Language Models Meet Uncertainty: The Surprising Effectiveness of Self-Reflective Program Search for Long Context", arXiv 2603.15653, March 2026. The abstract says recursion itself is not the main reason recursive language models perform well.
• File-system paper: Weili Cao and three co-authors, "Coding Agents are Effective Long-Context Processors", arXiv 2603.20432, March 20, 2026. On several long-context tests, it beats the best published results by 17.3% on average.
• Parallel sub-agent paper: Elias Lumer and three co-authors, "Recursive Agent Harnesses", arXiv 2606.13643, June 11, 2026. On Oolong, the coding agent Codex scores 71.75%, the recursive approach 64.38% and handing the whole text straight to the model 59.22%; these three are the paper's citations of the file-system paper's measurements. Letting Codex spawn sub-agents in parallel raises the score to 81.36%. The underlying model is fixed as GPT-5; with Claude Sonnet 4.5 it reaches 89.77%.

Approximate costs

These are approximations this publication read off Harvey's log-scale cost chart; Harvey states they are cache-aware estimates at list prices. In the recursive harness, cheap models went from under $1 per data room to about $3 to $6, and GPT-5.6 Sol to about $27. Opus 5 fell from about $18 to about $7.

Harvey's original wording

• How much the simple tool loop read: "In the standard tool-loop harness, no run reads more than 1% of the data room, and most read between 0.1% and 0.5%."
• How much the recursive harness read: "In the RLM harness, nearly every run reads more than 10% of the data room and most read close to all of it."
• Only a correlation: "Higher coverage is associated with higher rubric criteria pass rates across this range."
• General agents: "In the traces, both agents stop reading early and write shorter memos, and neither spawns sub-agents despite having the ability to."
• Same data rooms: "All other results use the 50-room hold-out set." (The exception is the GLM-5.2 further-training run, which used a separate set of 20 data rooms.)
• Learnable: "learnable from a relatively small number of on-policy traces, with no privileged information and no labeled legal knowledge"
• Tenet: "We did not use any customer data in any of our post-training efforts."

Other readings to watch

Follow-up posts on Harvey's blog and updates to Anthropic's dynamic workflows (test 1); whether the due-diligence task set in Harvey's benchmark is made public, and whether agreement between the AI judge and lawyers is disclosed. Once the yardstick is public, section six's claim that the yardstick is scarce must be withdrawn.

Measurement notes

• Harvey published these experiments together with Baseten, a company that runs AI models as a service.
• Web access was switched off for both general agents, Claude Code and Codex.
• Correlation coefficient: after GLM-5.2's further training, the correlation between the number of sub-agents spawned and data room size rose from 0.17 to 0.84 (1 means they move in perfect step).
• The extra-layer experiment was run with Qwen3.5-122B-A10B; Harvey writes that many GLM-5.2 runs did not finish within the time limit, so only Qwen is reported.
• The 17.3% (file-system paper), 13% (original paper) and 22% (self-check paper) are not marked in the papers as relative gains or percentage points.
• The October 8 first version of this piece made three calls: treating a general coding agent as the all-purpose harness is the wrong choice; switching harness pays more than switching to a frontier model; and the defensible position for a vertical AI company is a tailored harness plus a post-trained open-weight model. Sections four, five and six are the rewrites after re-testing.
• Tenet was announced on August 20, 2026.

The trend line this publication tracks

This publication charts changes in the shape of AI products as the tree below: from chat assistants, to assistants built into work software, to agents that deliver whole coding tasks, to general agent platforms that work across tools and whose underlying model can be swapped. Who ends up owning that last layer is still open: on one side, model makers' own platforms, such as Claude Code; on the other, vendor-neutral third-party outer layers that can switch between models.

Where this piece sits on the trend line: this piece's evidence lands on the final fork. On long-document work, the gap comes mainly from whether the work is handed out, and model makers are building delegation into their own general agents (Claude Code's dynamic workflows), so the model makers' own-platform side has moved a step forward; the fork itself is still open.

Sources

• Harvey Blog, 2026-09-08: Post-training RLM agents for end-to-end M&A Diligence
• Harvey Blog, 2026-08-20: Harvey Tenet Research Preview
• Anthropic, 2026-05-28: Introducing dynamic workflows in Claude Code
• arXiv, 2026-03-20: Coding Agents are Effective Long-Context Processors
• arXiv, 2026-06-11: Recursive Agent Harnesses
• arXiv, 2026-03-07: Recursive Language Models Meet Uncertainty: The Surprising Effectiveness of Self-Reflective Program Search for Long Context
• arXiv, 2026-03-03: Think, But Don't Overthink: Reproducing Recursive Language Models
• arXiv, 2025-12-31: Recursive Language Models
• GitHub: alexzhang13/rlm
• Latent Space, 2026-10-02: Academia is for Ambition — Alex Zhang, MIT
• Ahead of AI, 2026-06-27: Using Local Coding Agents
• arXiv, 2026-05-27: Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows
• arXiv, 2026-06-10: Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks

Written from the same research and judgments as the Traditional Chinese edition. Sources are linked; original documents are kept distinct from reporting about them, and unverified points are marked as such.