Does AI Agent Memory Actually Improve Performance — Why Task Order Decides the Answer (2026)

The two axes prior work left out

The setup is a re-evaluation rather than a new method. The authors took two existing memory-based methods — agents "that learn from an online stream of tasks and improve over time by maintaining a textual memory bank" — and widened the evaluation on two axes they describe as: "(1) including multiple runs to quantify variance, and (2) randomly shuffling the tasks to investigate the effect of task order."

Both of those are standard practice in most of empirical machine learning and both were missing here. That's the entire methodological contribution, and it's enough to change the results.

Finding one: the measurement was noisy before memory was added

Their first observation, verbatim: "agent evaluation is inherently noisy in complex environments and on multi-step tasks, and stacking a self-improving loop on top can further amplify this noise."

Read that twice, because it has two parts. Agent benchmarks on multi-step tasks are noisy on their own — the same agent, the same tasks, different run, different score. Then a self-improving loop takes that noisy signal and feeds it back into the memory bank, where it shapes the next run. Noise doesn't just persist; it compounds.

Practical consequence for anyone running their own evaluation: a single-run comparison of memory-on versus memory-off tells you very little. If you've ever shipped a memory change because the number went up once, this is the paper explaining why that was not evidence.

Finding two: task order was doing a lot of the work

The second observation is the striking one: "the agent's improvement is highly dependent on task order. Prior works often adopt default orderings that impose an implicit curriculum, acting as a hidden prerequisite for success."

An implicit curriculum means the default task sequence happened to go easy-to-hard, or happened to put the tasks that teach useful lessons before the tasks that need them. The agent learned in that order and improved. Shuffle the order and the improvement shrinks — the memory bank was benefiting from a teaching sequence nobody designed on purpose or reported.

This is a familiar failure mode dressed in new clothes. It isn't fraud and it isn't sloppiness; it's an uncontrolled variable that nobody thought to control because the field is young.

The hypothesis they tested, and only partly confirmed

The authors didn't stop at the negative result. They inspected the memory banks by hand and formed a hypothesis: "task and environment underspecification contribute to this fragility." In other words, the agent writes vague or wrong lessons because the task and the environment never told it clearly enough what good looked like.

Then they tested it, by "incorporating information that enables better specification, such as detailed rubrics and environment feedback, into the memory construction process."

The result is the most honest sentence in the abstract: "While this added information partially closes the performance degradation in previous experiments, significant gaps still remain, suggesting that other uncharacterized factors contribute to this fragility."

Partially. Significant gaps remain. Other factors uncharacterised. That's a research team declining to oversell its own fix, and it's the reason the paper is worth trusting.

Getting this boundary right matters more than the headline.

It studied two methods, not the category. The re-evaluation covers two memory-based self-improving methods. It is not a survey and does not claim that every approach to agent memory is fragile.

It's about agents that write their own memory unattended. The specific object of study is an agent maintaining a textual memory bank from an online stream of tasks, with no human in the loop. That is not the same thing as a person writing down a project's constraints, or a team maintaining conventions in a file. Nothing here says written-down context stops being useful.

It's a reliability critique, not a refutation. The finding is that reported gains are order-dependent and noisy — not that memory has no effect. A method whose benefit depends on task order still has a benefit under some orders.

It doesn't name products. These are research methods on benchmarks. Reading it as a verdict on any vendor's memory feature is a stretch the paper doesn't make, and neither should anyone else.

And it sits alongside results pointing the other way. Work published four days earlier measured real gains from injecting self-distilled guidelines, with the size depending on the model — covered in how much memory you should give an AI agent. Both can be true: gains exist, and the reported size of those gains is less stable than the literature implies. The reconciliation is better evaluation, which is exactly what this paper asks for — "reporting results across multiple runs and stress-testing them under challenging conditions."

"Agent memory doesn't work." Not what it says, and the authors go out of their way not to say it. Two methods, one evaluation protocol, order-dependent gains.

"So we should skip memory and use a bigger context window." A different question with its own answer, unaffected by this paper — the ground covered in why a large context window isn't memory.

"The fix is rubrics." Rubrics and environment feedback helped and did not close the gap. Treating them as the solution ignores the sentence about significant gaps remaining.

"Benchmarks are useless." The opposite. The paper is a demonstration that benchmarks are informative when you run them properly — multiple runs, shuffled order.

"Let the agent manage its own memory and it'll sort itself out." This is the belief the paper most directly undermines. An unattended self-improving loop amplifies evaluation noise into the memory bank.

"Our internal A/B showed a lift, so we're fine." If that A/B was a single run in a fixed task order, this paper is specifically about why it isn't evidence yet.

The paper closes with two recommendations, and they're the actionable part. The first is evaluation: report across multiple runs and stress-test under challenging conditions. The second is more interesting for anyone building on this: "our findings on underspecification call for systems and interfaces that enable effective human oversight, preventing agents from failing in unforeseeable ways."

Human oversight. Interfaces. Specification written into memory construction. Read together, the recommendation is not "give the agent less memory" — it's stop letting the memory be a black box the agent writes to unsupervised.

Three things follow for a working setup:

Evaluate with variance and shuffled order. Three runs minimum, and permute your task sequence. If the gain disappears under shuffling, you've learned that your task order was the curriculum.

Specify before you accumulate. The paper's partial fix was better task and environment specification feeding memory construction. In practice that means writing down what "done correctly" looks like, so an agent's distilled lesson has something to be right about.

Keep the memory readable and editable by a person. If you can't see what got written, you can't catch the vague or wrong entry that shapes the next fifty runs.

That last point is what MemoryLake is built around: memory as individual entries you can read, correct, and delete, rather than an opaque store that only grows. Setup is three steps.

Step 1: Create an API key

Sign in to MemoryLake and create an API key. One credential across the tools you connect.

Step 2: Upload your first memories

Write the specification the paper says agents lack, as short entries with one claim each:

What "correct" means for your recurring tasks. The rubric, in prose. This is the single category the paper found helped.

Constraints the environment doesn't announce. Rate limits, ordering requirements, the staging database with no recent rows. Environment feedback the agent can't infer.

Approaches already ruled out, with the reason. Under a shuffled task order an agent has no curriculum to lean on — written rejections are what replace the lucky sequence.

Corrections you've made more than once. If a person had to say it twice, an unsupervised loop was never going to derive it.

Step 3: Connect your AI & agents

Connect the tools you use. MemoryLake is reachable over MCP and over an API, so MCP-native agents — Claude Code, Codex, and OpenClaw among them — connect by pointing at the MCP server, while other assistants read the same memory through the API.

Three honest limits, and the first is the point of this article. MemoryLake was not part of this study and does not solve the fragility it describes. Nothing here is a claim that a curated memory layer makes self-improving agents reliable — the authors say other factors remain uncharacterised, and that applies to any store. What an inspectable layer gives you is the human oversight they call for, and the ability to fix a bad entry instead of discovering it in a regression. It holds only what you or your agents write into it. And it isn't an evaluation harness: the multiple-runs discipline is yours to implement.

Single-run comparisons stop counting as results. The cheapest change available, and the one with the highest return. Three runs and a shuffle.

"The agent learned it" becomes a claim you check. Open the memory, read the entries, and see whether the lesson it distilled is actually correct.

Writing rubrics becomes engineering work. Underspecification is named as a contributing cause. Specifying what good looks like is no longer a documentation chore.

Order-independence becomes a design goal. If your agent only improves in one task sequence, you have a curriculum, not a memory system.

Pruning matters more than accumulating. A wrong entry written under noise gets injected with the right ones — the general problem behind why RAG isn't memory.

Run it at least three times. Variance across runs was one of the two axes the original work omitted.

Shuffle the task order, deliberately. If the improvement is order-dependent, you want to know before you ship.

Report the spread, not the best number. The paper's request in one line.

Write the rubric before the memory. Better specification into memory construction is the intervention that measurably helped.

Read what the agent wrote. Manual inspection is how the authors found the underspecification hypothesis in the first place.

Date entries and delete stale ones. Under noise, an old wrong entry is worse than an empty store.

Don't claim a category result from two methods. Both directions: not "memory works," not "memory doesn't."

Keep a human in the loop by design. The paper's own closing recommendation is systems and interfaces enabling effective human oversight — the shape of it discussed in what AI memory is and isn't.

Does agent memory improve performance? On the evidence in this paper: sometimes, less than reported, and much of the measured difference in two prominent methods traced back to a task order nobody had controlled for. The authors are careful about what that means — agent evaluation is noisy before you add a self-improving loop, the loop amplifies the noise, better specification helps but leaves significant gaps, and other factors remain uncharacterised.

What it changes for practitioners is mostly discipline rather than architecture. Run your comparison more than once. Shuffle the order. Write down what correct looks like instead of hoping an agent infers it. And keep the memory somewhere a person can actually read, because the paper's own conclusion is a call for human oversight, not for a smarter unattended loop. Memory that you can inspect and correct is a weaker claim than memory that improves itself — and on this evidence, it's the one that holds up.