CHAPTERS
- 0:00 – 1:01
Why Masra optimized agent memory with LongMemEval
Sam Bhagwat introduces himself, Masra (a TypeScript agent framework), and the goal: using a benchmark to understand and improve agent memory performance. He frames the talk as an iteration loop—measure, diagnose, and optimize—then ship improvements into the framework so other teams benefit.
- •Masra’s memory layer was improved using a real benchmark
- •Frameworks encode opinionated defaults so teams don’t reinvent the wheel
- •Talk is structured around the performance-improvement loop (eval → iterate)
- 1:01 – 1:31
Memory as message compression + retrieval for the context window
Memory is defined as compressing a queue of chat messages and retrieving the right pieces in response to a query. The point is to place the most relevant information into the model’s context window so the LLM can answer correctly.
- •Memory = searchable compression of chat history
- •Retrieval quality determines whether the LLM gets the right context
- •Goal is meaningful search over prior messages, not just storage
- 1:31 – 2:02
LongMemEval benchmark: what it measures and how it’s organized
LongMemEval (released late the prior year) provides ~500 questions to evaluate agent memory. It categorizes questions into several subtask types that reflect real-world memory failures and requirements.
- •Benchmark specifically targets agent memory capabilities
- •~500 questions grouped by memory-relevant task types
- •Used as a diagnostic tool to guide implementation changes
- 2:02 – 2:32
Subtask 1: Single-session information extraction
The first and simplest subtask is accurately extracting details from within a single conversation session. The agent must find what the user or assistant said and bring it into context to answer the question.
- •Single-session lookup and extraction
- •Correct retrieval enables correct answers; misses cause failure
- •Focus on precision: pulling the exact needed detail
- 2:32 – 3:03
Subtask 2: Multi-session reasoning across long histories
Multi-session reasoning tests whether an agent can extract and combine information across multiple sessions. This mirrors long-running user relationships where key facts are scattered across time.
- •Cross-session retrieval and synthesis
- •Requires pulling multiple relevant facts into the context window
- •Failure mode: partial recall leads to wrong reasoning
- 3:03 – 4:04
Subtask 3: Temporal reasoning over time-based events
Temporal reasoning requires retrieving the right time-stamped events so the model can reason about ordering and recency. The agent must supply the relevant time context for correct comparisons (e.g., “today” vs. “five months ago”).
- •Time ordering and recency comparisons are core challenges
- •Retrieval must include the right temporal references
- •Incorrect timestamps or missing context breaks reasoning
- 4:04 – 5:35
Subtask 4–5: Knowledge updates and acknowledging missing info
Knowledge updates test whether memory correctly overwrites older user attributes with newer facts, rather than clinging to stale data. The benchmark also checks whether the agent can recognize when required information is absent and avoid hallucinating an answer.
- •User attributes may change; memory must update/overwrite correctly
- •Personal anecdote: ChatGPT mis-identified Sam due to bad updates
- •Agents should explicitly handle ‘not enough information’ cases
- 5:35 – 6:05
Masra’s two memory types: semantic recall vs. working memory
Sam explains Masra’s two main memory mechanisms used for the benchmark: semantic recall and working memory. These represent complementary approaches—retrieving relevant past messages vs. storing structured attributes.
- •Two core memory types in Masra: semantic recall + working memory
- •Benchmark performance depends heavily on these implementations
- •Sets up later optimizations and configuration choices
- 6:05 – 6:35
How semantic recall works: embeddings, vector search, and context expansion
Semantic recall embeds messages into a vector database and retrieves the top-K most similar items. Masra also pulls a “message range” around matches to capture surrounding context, similar to grabbing lines around a grep hit.
- •Embed messages → store in vector DB (pgvector/Chroma/etc.)
- •Top-K controls how many matches are retrieved
- •Message range expands context around a hit to preserve coherence
- 6:35 – 7:06
Working memory templates: what you store depends on the agent
Working memory is guided by templates that define which attributes to track. Sam emphasizes that different applications (personal trainer vs. legal research) need different structured memory fields, so the template strongly affects usefulness and accuracy.
- •Templates define which attributes are extracted/stored
- •Different domains require different memory schemas
- •Template quality directly impacts working-memory accuracy
- 7:06 – 7:36
Baseline results: below state of the art and motivation to iterate
Masra’s initial combined approach (working memory + semantic recall) achieved ~67% accuracy, while state-of-the-art was around ~72%. This gap motivated a systematic set of improvements guided by the benchmark.
- •State-of-the-art ~72% accuracy (as cited)
- •Masra initial combined result ~67%
- •Benchmark provided a concrete target and debugging lens
- 7:36 – 9:07
Improvements 1–2: question-tailored templates and targeted working-memory updates
They first generated templates tailored to each question, which improved working memory accuracy. They then changed working memory updates from rewriting the entire memory (error-prone) to targeted overwrites of specific fields, further boosting results to state-of-the-art levels.
- •LLM-generated, question-specific templates improved extraction
- •Avoided full rewrites; switched to targeted attribute updates
- •Reduced incorrect rewrites and improved overall accuracy
- 9:07 – 11:08
Improvements 3–4: fix date bugs and restructure message presentation for time reasoning
They found temporal bugs caused by incorrect dates being presented during benchmark runs, harming time-based reasoning. After correcting timestamps, they further improved results by restructuring retrieved messages—adding timestamps and grouping by day with hour-level detail to make reasoning easier.
- •Bug: benchmark date vs. run date mismatch led to wrong timestamps
- •Correct timestamps improved temporal reasoning scores
- •Grouping messages + adding timestamps improved temporal/session reasoning
- 11:08 – 12:09
Configuration matters: top-K tradeoffs and category-level evaluation
Sam shows that retrieval configuration (especially top-K) can significantly change results: too low (e.g., K=2) harms performance, while higher values (e.g., 5–20) improve it. He highlights the importance of breaking eval results down by category, since some changes help certain task types but not others (e.g., single-session tasks).
- •Top-K strongly affects semantic recall performance
- •Higher K (within reason) improved accuracy vs. very low K
- •Category breakdown reveals which changes move which metrics
- 12:09 – 13:39
Next steps: reranking, episodic memory, and the eval-driven iteration loop
He outlines future enhancements like adding reranking (used by other benchmark leaders) and shipping episodic memory to see how scores change. The conclusion reinforces the core methodology: build an eval scaffold, iterate repeatedly, and use data to find subtle bugs and presentation improvements.
- •Planned: episodic memory support and reranking for retrieval
- •Benchmarking reveals ‘weird bugs’ and data formatting wins
- •Main takeaway: eval scaffold + lots of iteration drives improvement
