Buy vs. Build: A Decision Framework for Agent Memory

Key takeaways

• The decision turns on one question: is agent memory your product, or plumbing beneath your product?
• A production memory system has to do five things at once — ingestion, temporal correctness, budgeted retrieval, governance, and scale — and the cost lives in the interactions between them.
• Build estimates go wrong because they're written against a chat-history buffer, not the system that buffer grows into.
• Buy when enterprise requirements show up early; access control, audit, retention, and residency are the most expensive things to retrofit.

Every agent that does more than answer one-shot questions needs agent memory: what it knows across time about its users, the business, and the world it operates in. Without it, the agent starts from scratch every turn. With it, the agent can reason about the user and act on what it already knows.

Many teams build memory themselves. It starts as a chat-history buffer. Then a vector store gets added. Then a layer of glue code holds the two together. The whole thing works in the demo, so it ships. This page is about what happens after the demo: when the build path stops working, and how to decide before it costs you a rewrite. Building your own is sometimes the right call, and the framework below should hold up in a code review and a budget meeting.

What agent memory actually requires

The build-vs-buy math goes wrong in the same place almost every time: the problem gets under-scoped. A buffer of recent messages looks like memory, so the estimate gets written against that. The estimate is for a different, smaller problem. A production memory system has to do five things at once.

• Ingestion and extraction. A customer leaves a trail across more than chat: CRM records, support tickets, billing events, documents, product telemetry. All of it has to become structured facts the agent can query, unified per subject rather than scattered across stores. “The user switched from the Pro plan to Enterprise in March” is a fact with a subject, a relationship, and a timestamp. Pulling that out of free text and event streams reliably, across millions of subjects, is its own engineering problem.
• Temporal correctness. Facts change. A user's job title, their preferences, their account tier — all of it moves. A memory system has to track what is true now and what was true at any past moment. That means bi-temporal validity: the system records when a fact held in the world and when it learned the fact, and it invalidates the old fact when a new one arrives. Get this wrong and the agent acts on a preference the user abandoned three weeks ago.
• Retrieval that fits a token budget. You cannot dump a user's entire history into the prompt. Retrieval has to pick the most valuable context for the task in front of the agent and fit it to a budget. Too little and the agent misses something it needed. Too much and cost climbs while accuracy drops.
• Governance. The moment a second user exists, one user's data must be invisible to an agent acting for another. Add retention rules and audit logging, with provenance back to source. This is the part that turns a prototype into something you can put in front of an enterprise security review.
• Scale. All of the above, for millions of subjects, with retrieval fast enough to sit in the request path. Sub-200ms under concurrent load, without the cost curve bending the wrong way.

Each capability looks tractable on its own. The cost lives in the interactions. The hard part is making all of them hold together at once: temporal correctness that still respects access control while retrieval stays under 200ms. That is a much harder system than any single feature suggests.

The build path: what you are actually signing up for

The first version is a weekend of work: recent messages in a buffer, embeddings in a vector database, a similarity search at query time. For a prototype or a single-user tool, that is a reasonable place to stop. The trouble starts as usage grows, and it tends to arrive in a predictable order.

First, stale facts. The vector store has no concept of time, so it retrieves a fact the user has since contradicted. The agent acts on it. Now you are writing logic to detect and supersede outdated facts — the temporal problem you skipped earlier, back to collect its debt. Then context bloat. As history grows, similarity search returns more loosely related chunks. The prompt gets longer and inference costs rise; answer quality slips because the signal is buried under the noise. You start building re-ranking and summarization to compensate. Then access boundaries. A customer asks the obvious security question: can one user's data reach another user's agent? Nothing in a shared vector index prevents it, and retrofitting entity-level access control into a system built without it is expensive and risky. Then the queries the vector store cannot answer at all — “which of this user's projects share a stakeholder with the account that just churned?” is a graph question, and cosine similarity does not answer it.

The deeper cost is not any single fix. It is that you now own a system. Someone is on call for it. Someone maintains the eval harness, migrates it when the underlying models change, re-indexes when the schema moves, and debugs retrieval-quality regressions. That is a team working on memory infrastructure instead of the product you are actually trying to build. The initial build might be a few engineer-months; the maintenance is a standing tax with no end date. None of this means building is wrong. It means the build estimate has to include the second system — the one you grow into, not the buffer you start with.

The options in between

Between hand-rolling memory and adopting a dedicated platform sits a real field of options. Three categories matter, and each is the right pick for some teams.

• Open source. Two flavors matter here. mem0 is a memory store you self-host: a vector index and a key-value store, with a graph backend offered as a separate add-on. Facts are mutated in place, and you trigger and assemble retrieval yourself. Graphiti, Zep's own open-source layer, goes further and builds a bi-temporal context graph with entity extraction and fact invalidation, plus a hybrid retrieval API — the most capable open option. Either way, the system around the store is yours to run: scale, multi-tenant isolation, governance, deployment, and compliance.
• Framework-bundled memory. Build on an agent framework like Mastra and memory comes built in, and it is more than a scratchpad: working memory keeps a structured profile across threads, semantic recall pulls relevant past messages back by vector search, and background agents compress raw history into a dense, reflected log. The limit is the input. It works from the conversation alone — the same scope as mem0 and most memory layers — not the stream of touchpoints a customer leaves across the enterprise (CRM, support tickets, billing, documents, product events). Nor does it build a bi-temporal graph, so facts carry no validity windows and there are no point-in-time queries, and governance belongs to the framework rather than the memory layer.
• Hyperscaler primitives. Amazon Bedrock AgentCore Memory is the example: a managed service giving agents short-term session memory and long-term insights extracted asynchronously, inside the AWS and Bedrock environment. If your stack is committed to AWS, the native integration is the draw. The trade shows up in three places — lock-in (your agents' memory, among the stickiest data you own, is bound to one vendor's stack), governance (entity-level access control and retention sit outside the memory layer), and depth (an extraction-based store is shallower than a bi-temporal graph, with no way to ask what was true at a past moment or to trace a fact back to its source).

Each option is good enough for the job it was built for. None of them combines bi-temporal facts, entity-level governance on every query, ingestion across chat and business systems, and sub-200ms retrieval at scale on a neutral stack. That combination is the job of a dedicated platform.

The buy path: what you get and what you give up

Buying agent memory means the five capabilities above arrive as infrastructure rather than a roadmap. Ingestion, temporal facts, governed retrieval, and scale are someone else's maintenance burden.

Zep is agent memory at enterprise scale. The Context Lake is how we believe it should be done: a governed system of context graphs that manages, governs, and serves everything an agent needs to know. It ingests every source the agent touches and unifies them into one context graph per subject — the conversation is one input among many. Zep runs the open-source Graphiti framework inside a managed system and serves the result through its proprietary Context Graph Engine, with sub-200ms retrieval whether you have one context graph or millions. Every query is filtered by entity-level access control; facts carry validity windows and invalidate cleanly when they change; retrieval returns a prompt-ready context block that fits a token budget. SOC 2 Type II and HIPAA come with it, and it stays neutral across model and cloud — the opposite trade from a hyperscaler primitive. On the public benchmarks Zep reports 94.7% on LoCoMo at 155ms and 90.2% on LongMemEval at 162ms (benchmark results). The durable difference is architectural.

What you give up is real, and worth stating plainly. You take on a dependency and a pricing relationship. You have less control over the internals than you would with code you wrote. And adopting any memory system is integration work, not a switch you flip. Three objections usually decide this. Data control: where does the data live, and who can touch it? Deployment models include fully managed, plus bring-your-own-key and bring-your-own-cloud for keeping data in your own environment. Lock-in: the graph-construction layer, Graphiti, is open source, so the foundation is not a black box you can never leave. The “we're different” worry: customization lives in how graphs are built and how context is assembled, not in forking the storage engine.

How to decide

One question decides it. Is agent memory your product, or plumbing beneath your product?

Build it yourself when memory is the thing you sell, or a core differentiator you have to own end to end. Build when your requirements are narrow and stable: a single tenant, with no cross-user governance and a modest scale that is not going to move. Build when you have the team and the real appetite to carry the maintenance for years, not the launch.

Buy when memory sits beneath the product rather than being the product. Buy when enterprise requirements show up early, because access control, audit, retention, and data residency are the capabilities that are most expensive to retrofit. Buy when time-to-production and keeping your engineers on your core product matter more than owning the internals.

Take these questions to your team before you commit:

• Is memory a differentiator we must own, or a dependency we can rent?
• Will we need entity-level access control within 12 months?
• What does it cost us, concretely, if the agent acts on a stale fact?
• Who owns this system at 2am, 18 months from now?
• If our first version does not scale, what is the rewrite worth?

In the cost comparison, build is engineer-months plus a recurring maintenance tax; buy is a subscription plus integration effort. The line item that dominates is rarely the license. It is the maintenance.

Where to start

The right answer depends on whether memory is your product or your plumbing, and that is a question only your team can answer. If you want the open core and intend to run the surrounding system yourself, start with Graphiti. If memory is plumbing you would rather not build and operate, talk to us.

Related: What is agent memory? · What is a Context Lake? · What is a temporal knowledge graph? · How to give an AI agent long-term memory · AI agent memory guides

Frequently asked questions

Should I build or buy agent memory?

Build it when memory is the product you sell or a differentiator you must own end to end, your requirements are narrow and stable, and you have a team to carry the maintenance for years. Buy it when memory sits beneath the product, enterprise requirements (access control, audit, retention, residency) show up early, and time-to-production matters more than owning the internals.

When does building your own agent memory stop working?

Usually in a predictable order as usage grows: stale facts (the vector store has no concept of time), context bloat (similarity search returns more loosely related chunks), access boundaries (a shared index can't keep one user's data out of another's agent), and graph-shaped queries cosine similarity can't answer. The deeper cost is that you now own a system someone is on call for.

What does a production agent memory system actually need to do?

Five things at once: ingest and extract facts from every source (not just chat), track bi-temporal validity so it knows what's true now versus then, retrieve the most valuable context within a token budget, enforce governance (entity-level access control, retention, audit), and do all of it for millions of subjects with sub-200ms retrieval. The cost lives in the interactions between these, not in any single feature.

Is open source like Graphiti or mem0 a middle path?

Yes. Graphiti builds a bi-temporal context graph with entity extraction, fact invalidation, and hybrid retrieval — the most capable open option. mem0 is a self-hosted vector + key-value store with a graph add-on. Either way, the system around the store — scale, multi-tenant isolation, governance, and compliance — is yours to run.

What's the real cost of building agent memory in-house?

The initial build is a few engineer-months; the maintenance is a standing tax with no end date — an eval harness, model migrations, re-indexing on schema changes, and retrieval-quality debugging. In the cost comparison, build is engineer-months plus that recurring tax; buy is a subscription plus integration effort. The line item that dominates is rarely the license — it's the maintenance.

Does buying agent memory lock me in?

With Zep, the graph-construction layer is the open-source Graphiti, so the foundation isn't a black box you can never leave, and it stays neutral across model and cloud. Deployment models include fully managed plus bring-your-own-key and bring-your-own-cloud for keeping data in your own environment — the opposite trade from a hyperscaler primitive bound to one vendor's stack.