
How agents can learn skills from traces
Agents in production meet problems that their prompts and skills did not anticipate, and what they learn solving them is lost. Our new paper describes how Zep Agent Skills turns verified traces into procedural agent memory.
Agents lose what they learn
Long-horizon agents in production meet business problems that their prompts and skills did not anticipate. A support agent gets a ticket type that its instructions do not cover. An operations agent has to follow an approval rule that changed last week.
A frontier model can often work these cases out, and the work is slow and expensive. What the agent learned is lost when the task ends. A successful attempt changes neither the model weights nor the prompt, and the next time a similar problem appears the agent starts the same slow, costly work again.

Fine-tuning, transcript retrieval, and learned procedures
Teams usually reach for one of two fixes. Fine-tuning on successful traces changes the model weights. Retrieval of earlier transcripts puts past attempts back in the prompt. The paper describes a third option. Zep compiles the procedure out of the trace and gives it to the agent as a skill.
| Fine-tuning | Transcript retrieval | Compiled skills | |
|---|---|---|---|
| What changes | Model weights | Prompt context | Prompt context |
| Process changes | New training round | New transcripts | New skill version |
| Tied to one model | Yes | No | No |
| Inspect or undo | Difficult | Long transcripts | Readable, restorable versions |
| One user's data | In the weights | Verbatim | Replaced with placeholders |
Three ways to reuse a solved task, as characterized in Sections 1 and 5 of the paper.
A package format for procedures already exists. The open Agent Skills format is a folder with a SKILL.md file and optional files. The file's frontmatter gives the skill a name and a description. Harnesses such as Claude Code and OpenAI Codex show the agent the name and description first, and the agent reads the body only when the skill applies. Anthropic published the format as an open standard in December 2025 and donated it to the Agentic AI Foundation in October 2026.
The format defines how a skill is packaged and loaded. It says nothing about how a skill is learned from traces, verified, versioned or governed. In most deployments a human writes each one. Zep Agent Skills is a production service for that missing part. The paper’s main experiments compiled their skills with it.
How we turn traces into skills
Take a bank support agent that receives a dispute type its instructions do not cover. It works the case out over 40 tool calls, and the bank's case system confirms the dispute was filed correctly. Zep handles that run in five stages.
- Capture. The harness records each attempt as a trace: the events, the outcome, and how strongly the outcome was verified. Zep can also import traces from Braintrust and LangSmith.
- Compile. An LLM role, the Reflector, reads the verified traces for one kind of task and proposes a new skill or a patch. Each claim cites the trace events that support it.
- Curate. A second LLM role, the Curator, rejects each claim that its cited events do not support. It also stops a weaker rewrite from replacing a skill that works.
- Admit. Scanners and admission gates check the candidate. Each admitted change becomes an immutable version, automatically or after a person approves it.
- Serve. The next agent that meets the task sees a short description of each matching skill and reads the full skill only when it applies. Verified outcomes raise or lower the skill's rank.
Here is part of a skill that Zep compiled in one of the experiments below, as the agent reads it.
---name: agent-skill-3dfb4a1b…description: "Cool the requested item and put it in the requested receptacle."---# Cool an item and place it in a receptacle ## Invocation- The user asks to put a cool item in a receptacle. ## Success criteria- The requested item has been cooled and moved to the requested receptacle. ## Termination conditions- Stop after sending the task-completion message. ## Procedure1. Go to the location where the requested item is found. - Expected observation: The requested item is visible at that location.2. Pick up the requested item from its location. - Expected observation: The item is picked up.3. Go to the cooling appliance. - Expected observation: The cooling appliance is closed.4. Open the cooling appliance. - Expected observation: The cooling appliance is open.5. Cool the requested item using the cooling appliance. - Expected observation: The item is cooled using the appliance.6. Go to the requested receptacle.7. Move the cooled item to the requested receptacle. - Expected observation: The item is moved to the requested receptacle.8. Mark the task complete and send a message stating that the task is completed.A skill Zep compiled from Qwen3-8B's runs in the ALFWorld experiment, in the form the agent reads. Some sections are left out. From Figure 2 of the paper.
Several stages use LLMs, and no LLM output takes effect directly. Deterministic code, which the paper calls trusted code, checks each output first. If a claim cites an event that the code cannot match to a stored trace event, the claim is removed.
The writer and the checker are separate roles. Checking a claim against evidence is a narrower task than writing a procedure, and separate verdicts leave an audit record of why each change was accepted or rejected.
Why we only use verified traces
An agent can report success on a task that failed. Zep stores the reported outcome apart from its verification, and the strength of the verification depends on who submits it. An agent can only self-report. Strong verification comes from a registered tool or external system, such as a test suite, or from a person with permission. Scores imported from an evaluation platform count once a person registers the import.
Only successful runs with strong verification can support a new skill. Failed runs are kept as contrast. They can support a documented failure mode and cannot support a procedure step.
We are also researching how to learn from unverified traces.
Security controls for skills
The security case is stricter for procedures than for facts. An agent follows a skill as instructions, often with tools that change data, send messages or move money. A trace also contains tool output and user text that an attacker can control. The paper cites Skill-Inject, which reports attack success rates of up to 80% against frontier models through instructions in skill files. Zep applies these controls:
- The LLM roles read trace content as data and do not follow instructions inside it.
- Scanners reject candidates that contain names, identifiers, credentials, private URLs, or injected instructions.
- Each skill is compiled from the evidence of one namespace, and an agent reads only the skills of its own namespace.
- Deleting a source trace makes every skill that cites it unavailable at once, until the skill is repaired.
A new skill starts as provisional and ranks lower in search than a skill with more support. Its trust value is a conservative estimate of its success rate over verified uses, and the estimate stays low until the skill has enough of them.
How we stopped agents cheating on the benchmarks
Open benchmarks publish their tasks, code and results. That openness makes them reproducible. A capable agent with network access can find the same material. In our evaluation work, models tried to pull clues and results from GitHub, Hugging Face and the benchmark websites. Some of those attempts succeeded.
We modified the sandboxes and harnesses of EnterpriseOps-Gym and AppWorld to stop this. In the published experiments, the cheaper models have no web access and receive only the benchmark tools, a tool to read a skill and a finish tool. On EnterpriseOps-Gym, Claude Opus 5.5 worked in Claude Code with the shell, file, web fetch, and web search tools disabled. No agent sees the checker queries, the reference solutions, or the database IDs.
The same concern applies to the skills. A skill should contain a procedure and no answer to a scored task. A scan of the EnterpriseOps-Gym skills for checker SQL, database IDs, and hidden check values found none.
Cheap models learning from expensive ones
Our first set of experiments tests a pattern we expect in production. A strong model solves a task once, and Zep compiles skills from its verified runs. A cheaper model then does the same task with those skills. We call the strong model the teacher and the cheaper one the executor.
Claude Opus 5.5 at maximum reasoning effort runs each task, and the benchmark checker verifies each outcome. On τ³-bench the teacher ran each task four times. On the other benchmarks it ran each task once. Zep’s production compiler turns the passing traces into skills and admits them automatically. Three cheaper models (GPT-6 Luna, Claude Haiku 5.5 and DeepSeek V4.1 Flash) run the same tasks with and without the skills at several reasoning efforts. That gives 26 configurations of model, effort, and benchmark.
| Benchmark | Task type | Without skills | With skills | Gain (points) | Teacher |
|---|---|---|---|---|---|
| τ³-bench banking | Bank support with a knowledge base | 13.1% to 46.4% | 21.6% to 52.8% | +5.2 to +15.5 | 57.2% |
| AppWorld | Code against app APIs | 68.5% to 92.7% | 87.3% to 98.0% | +4.5 to +18.8 | 97.3% |
| EnterpriseOps-Gym | Enterprise operations under policy | 27.7% to 49.9% | 37.9% to 58.3% | +8.4 to +11.4 | 52.8% |
Pass-rate ranges across executor models and reasoning efforts, from Tables 8 to 10 of the paper. EnterpriseOps-Gym is scored as described in Appendix A of the paper.
Each comparison pairs a run with skills and a run without them on the same task. All 26 paired differences are positive. The statistical strength varies by benchmark. All nine EnterpriseOps-Gym intervals are above zero. Eight of nine AppWorld intervals are above zero after correction for multiple comparisons. Four of eight τ³-bench configurations pass the corrected significance threshold.
Reasoning effort. On EnterpriseOps-Gym, every executor passed more tasks with skills at its lowest effort than without skills at its maximum effort. The same held for two of three models on AppWorld and one of three on τ³-bench. On EnterpriseOps-Gym, DeepSeek V4.1 Flash with skills at low effort also passed more tasks than the teacher, 56.8% against 52.8%. The teacher ran each task once, and the executors also gained on tasks it had failed.

Repeatability. Skills raised the number of τ³-bench tasks that pass in all four runs in seven of eight configurations. For GPT-6 Luna at low effort, the count went from 2 to 11 of 97 tasks.
Tokens. On τ³-bench, where runs are long, tokens per passed run fell by 7.0% to 44.6% in all eight configurations. On AppWorld, where runs are short, tokens per passed run rose in seven of nine. The benchmark harness placed an index of all 62 skills in every prompt, which added about 2,300 tokens to each model call. A production agent calls search and receives only the matching descriptions. The paper draws no conclusion about latency.
Skills compared with fine-tuning
Our second set of experiments compares skills with fine-tuning on the same small model. Retrospective Self-Distillation (RSD) is a continuous fine-tuning method. Its paper reports results for Qwen3-8B on ALFWorld, a text game of household tasks. We ran Qwen3-8B on the same benchmark. It learned from its own runs on training games, and we scored it on games in rooms it had never seen.
Qwen3-8B’s starting pass rate depends heavily on the example game in its prompt, and the RSD paper does not say which example it used. We ran Qwen3-8B twice, once with an example that starts it below RSD’s model and once with one that starts it above.
| Before | After | Gain (points) | |
|---|---|---|---|
| RSD fine-tuning | 62.9% | 76.6% | +13.7 |
| Zep Agent Skills, starting below RSD | 42.8% | 75.4% | +32.6 |
| Zep Agent Skills, starting above RSD | 74.6% | 82.8% | +8.2 |
Qwen3-8B on ALFWorld games in rooms not seen in training. RSD values are as the RSD paper reports them. From Table 13 of the paper.
Starting below RSD, skills added more than twice RSD’s gain without any training. Most of it came from tasks the model did not know how to do. Heating an object went from 4% to 65%. Starting above RSD, the gain was smaller, and the final pass rate was still higher than RSD’s.
Skills beat fine-tuning on the benchmark, with no training. They removed 57% of Qwen3-8B's failures where RSD removed 37%, and from the stronger start they passed 82.8% of games, more than RSD reached even on Qwen3-14B.
Where skills did not help
- Tasks the teacher never solved. With no verified success, a task gets no skill of its own. On τ³-bench these tasks gained 0.6 points, which is within noise.
- Tasks the model already passes. On AppWorld the cheaper models already passed 68.5% to 92.7% of tasks without skills, so there was little room to gain.
- Skills that fit the wrong task. Skills also cost some tasks. On EnterpriseOps-Gym they lost 14 to 32 tasks in each setup, against 73 to 94 wins. In one AppWorld case, two versions of a task needed opposite rules, and the skill followed only one.
Using Zep Agent Skills
Zep Agent Skills works with any agent framework and any model. Traces arrive through the API as a task proceeds, or through an import from Braintrust or LangSmith. Agents retrieve skills over the API or MCP. Any agent that loads the open Agent Skills format can load a compiled skill.
The full design, the threat model and every result table are in the paper on arXiv.
Zep Agent Skills is free on every Zep plan until January 4, 2027. Start building or read the docs.
FAQ
What is procedural memory for an agent?
Procedural memory holds how to do a task: when a procedure applies, its steps, and when to stop. Semantic memory holds facts about users and domains, and episodic memory holds earlier interactions. Zep Agent Skills supplies procedural memory alongside the other two types.
Does Zep Agent Skills fine-tune models or execute skills?
No. The service does not train model weights, execute skills or application code, or store hidden model reasoning. A trace contains only observable events. If a namespace requires evaluations for a candidate, the application executes them in its own environment and records the results through the API.
Can a skill go live without human review?
Yes. Automatic approval is the default, and a candidate still has to pass the scans and admission gates. A namespace can switch to manual approval, limit automatic approval to selected task families, or require a human for any skill that uses selected tools.
Does it require a frontier model as the teacher?
No. Any agent whose outcomes are verified produces usable traces. In the ALFWorld experiment, an 8B-parameter model learned from its own traces.
What happens when a source trace is deleted?
Every skill that cites the deleted trace becomes unavailable at once. Zep schedules a repair of each affected skill, and a background job removes the deleted content.



