Does an AI agent remember my previous conversations?

Not by default. The model sees only what the software sends in the current request. Some products add memory by saving notes in a file or database and loading them later. For example, Anthropic's memory tool, as of October 2026, lets Claude store and retrieve information across conversations in a directory of memory files.

What is RAG in simple terms?

RAG means retrieval-augmented generation. Before the model answers, the software looks up relevant passages from a store of documents and gives them to the model along with the question. A 2020 paper by Lewis and colleagues described RAG models, which combined a language model with an index of Wikipedia.

Is a bigger context window always better?

No. Anthropic's context engineering post says that as the number of tokens in the window increases, the model's ability to accurately recall information from it decreases. Anthropic uses the term context rot for this. In our reading, larger windows help, but choosing what goes into the window still matters, because a crowded window can make answers worse.

How Do AI Agents Remember Things? Context Windows, Memory and RAG

Q: What is a context window?

A context window is all the text a language model can look at when it writes a response, including the response itself. Anthropic's documentation, as of October 2026, describes it as a kind of working memory for the model. It is different from the large body of data the model was trained on, and every model has a limit.

Imagine it is Tuesday at the Lagos fabric shop from the first post. On Monday the owner and the agent worked out that order 1042 was held at the depot because the address was incomplete. This morning the owner opens a new chat and types: “Did that address get fixed?” The agent has no idea what she means. Why? And how do agents get a memory at all?

(This is an illustrative scene, not a real business.)

What is a context window?

A context window is everything the model can look at when it writes one response. Anthropic’s documentation, as of October 2026, defines it as “all the text a language model can reference when generating a response, including the response itself”. It adds that this represents a “working memory” for the model, as distinct from the large body of data the model was trained on.

The window is measured in tokens, which are small pieces of text, often a word or part of a word. Every model has a limit. As of October 2026, Anthropic lists a 1M-token window for its current models, such as Claude Opus 5.5 and Claude Sonnet 5.5, and 200k tokens for some others. OpenAI’s guide, as of October 2026, defines the context window as “the maximum number of tokens that can be used in a single request”, counting input, output and reasoning tokens. Reasoning tokens are tokens a model uses to think before it answers. Sizes differ between models and change often, so check the model you use.

Does the model remember what we said before?

Not by itself. The software sends the earlier messages again each time. In the second post, we saw that the loop sends the earlier steps back to the model on every turn. Anthropic’s documentation, as of October 2026, describes this for its API: each turn’s input “contains all previous conversation history plus the current user message”. OpenAI’s guide, as of October 2026, says the same thing in other words: “To manually share context across generated responses, include the model’s previous response output as input, and append that input to your next request.”

As of October 2026, OpenAI also offers ways to let its service keep track of the thread, such as chaining responses with a previous_response_id. Its guide notes that “all previous input tokens for responses in the chain are billed as input tokens”. In our reading, the lesson is the same either way: the earlier conversation is part of every request, so the window fills up as the conversation grows.

That is why the shop’s agent knows nothing on Tuesday. The new chat starts with an empty window.

What happens when the window fills up?

It depends on the product. For the Anthropic API, the documentation says that, as of October 2026, if the input alone already exceeds the model’s context window, “the API returns a 400 invalid_request_error”, an error code that means the request was rejected. In our reading, an agent that runs for hours, with long tool results in every step, can get there.

A related problem can arrive earlier. Anthropic’s engineering team wrote on 29 September 2025: “Context, therefore, must be treated as a finite resource with diminishing marginal returns.” It describes context rot: “as the number of tokens in the context window increases, the model’s ability to accurately recall information from that context decreases.” It also says it is likely that, for the foreseeable future, “context windows of all sizes will be subject to context pollution and information relevance concerns”. So a bigger window helps, but it does not remove the need to choose what goes in.

What can an agent do about a full window?

Anthropic’s engineering post names several techniques. The first is compaction: “taking a conversation nearing the context window limit, summarizing its contents, and reinitiating a new context window with the summary”. The post warns that “overly aggressive compaction can result in the loss of subtle but critical context whose importance only becomes apparent later”. Anthropic’s documentation says that, as of October 2026, its server-side compaction is in beta.

A second idea is a sub-agent. A sub-agent is a helper agent given one focused task, which returns only a short summary to the main agent. Anthropic’s post says sub-agents can handle focused tasks “with clean context windows”. The ninth post in this series comes back to several agents working together.

How does an agent keep notes?

It writes them down outside the window. Anthropic’s post calls this “structured note-taking, or agentic memory”, “a technique where the agent regularly writes notes persisted to memory outside of the context window. These notes get pulled back into the context window at later times.” It gives examples such as a to-do list or a NOTES.md file.

Anthropic also offers a memory tool for this. Its documentation, as of October 2026, says: “The memory tool operates client-side: Claude requests file operations, and your application runs them.” This is the same pattern as the tool round trip in the fifth post. The model asks, and the developer’s code does the work, in storage the developer controls. The documentation adds that, when the memory tool is enabled, “Claude automatically checks its memory directory before starting a task.”

Illustrative example. Imagine the shop’s developer gives the agent a notes file. On Monday the agent writes: “Order 1042 held at depot, address incomplete. Customer to confirm the address.” On Tuesday, the new chat starts by reading that file, so “Did that address get fixed?” now makes sense. The agent then checks the courier tool again, because the note is only a record of Monday. This file and its contents are made up.

What is RAG?

RAG stands for retrieval-augmented generation. Retrieval means finding relevant information in a store, and generation means the model writing its answer. In RAG, the software retrieves passages first and puts them in the context window, so the model answers using them.

A 2020 paper by Patrick Lewis and colleagues (accepted at NeurIPS 2020) described RAG models. It explains the problem: language models “cannot easily expand or revise their memory, can’t straightforwardly provide insight into their predictions, and may produce ‘hallucinations’”. The paper proposed models “which combine pre-trained parametric and non-parametric memory for language generation”. In plain terms, parametric memory is what the model learned in training and holds in its own parameters, the numbers it adjusts during training. Non-parametric memory is an outside store that can be searched; in the paper, “a dense vector index of Wikipedia”, in plain terms a searchable index of Wikipedia passages. The authors argue that with a store like this, “knowledge can be directly revised and expanded, and accessed knowledge can be inspected and interpreted.”

Illustrative example. The shop has a returns policy and courier terms as documents. A customer asks about a return. The software searches those documents, finds the two relevant paragraphs, and passes them to the model with the question. The model has the relevant paragraphs in front of it when it answers. If the policy changes, the shop edits the document instead of changing the model.

The paper is from 2020, and it describes one particular design, so it is not a recipe for how every system works today. In our reading, an agent can do the same thing with a search tool, a version of the tool use from the fifth post.

How do agents find what they need?

Anthropic’s September 2025 post says that many applications use retrieval done in advance, and that teams are adding “just in time” strategies. With these, agents “maintain lightweight identifiers (file paths, stored queries, web links, etc.) and use these references to dynamically load data into context at runtime using tools”. It also says that, in certain settings, the most effective agents might use a hybrid: “retrieving some data up front for speed, and pursuing further autonomous exploration at its discretion”.

Which kind of memory fits which job?

In our reading, the three ideas answer different questions:

Context window: what can the model see right now? Everything for this request, up to the limit.
Notes or a memory tool: what should survive to a later session? Short facts the agent wrote down.
RAG or a search tool: what lives in a store too big to fit in the window? Documents and records, fetched when needed.

In our reading, all three have a catch. A note can be out of date, and a retrieved passage can be the wrong one. Stored notes may also contain private customer details, so the shop should decide where they are kept and who can see them. As in the earlier posts, anything risky should go past a person.

What to remember

A model sees only what is in its context window. The software sends earlier messages again each time, so a new chat starts empty.
A bigger window does not remove the need to choose what goes in. Notes outside the window and retrieval from a store help an agent work over a longer time.
RAG means fetching relevant passages and putting them in front of the model before it answers. Stored notes and retrieved passages can be stale or wrong, so check them.

If you want to add a language model to your own software, our LLM integration guide covers API patterns. To revisit earlier posts, see How Does an AI Agent Actually Work, Step by Step? and How Do AI Agents Use Tools and Call Functions?.

Next in the series: how to build a small AI agent yourself.