RFC: Introduce a Working Memory layer in the AI SDK for low-latency local context retrieval

RFC: Introduce a Working Memory layer in the AI SDK for low-latency local context retrieval

motive

@supermemory/ai-sdk (packages/ai-sdk/src/tools.ts:34) creates supermemoryTools that hit api.supermemory.ai on every searchMemories call. In an agent loop (e.g., 10 tool calls in one turn), the same query — "user preferences", "project context" — is fetched repeatedly from the network. There is no local session cache, so:

• Repeated queries pay 80–150 ms RTT each
• Token cost is paid twice (retrieval → prompt expansion)
• Agent UX feels non-persistent within a session

This RFC proposes a Working Memory layer — Layer 1 of the Hierarchical Memory Pyramid — that keeps recent session memories locally and serves them before hitting the API.

Current retrieval flow

Agent tool call -> supermemoryTools.searchMemories -> client.search.execute (network) -> results

Every call is a network call. No in-process memoization, no deduplication, no stale-while-revalidate.

Proposed architecture

Agent tool call
  -> WorkingMemory.search(query)
       -> 1) probe local LRU+TTL cache (in-memory, per-SDK-instance)
       -> 2) on hit: return cached results (2-8 ms)
       -> 3) on miss: call client.search.execute (network), populate cache
       -> 4) concurrent callers for same query share one in-flight fetch
  -> results (with source: 'cache' | 'network' for observability)

WorkingMemory is a small, zero-dependency class:

class WorkingMemory {
  constructor(opts: { maxEntries?: number, ttlMs?: number })
  get(query: string, opts?: { limit?: number }): SearchResult[] | undefined
  set(query: string, results: SearchResult[], opts?: { pin?: boolean }): void
  has(query: string): boolean
  invalidate(query?: string): void   // single key or all
  pin(query: string): void
  unpin(query: string): void
  stats(): { hits, misses, size }
}

Integrated as an opt-in wrapper around supermemoryTools:

const { searchMemories, addMemory, workingMemory } = supermemoryTools(apiKey, {
  projectId: "...",
  workingMemory: { enabled: true, maxEntries: 100, ttlMs: 60_000 }
})
// or standalone:
import { WorkingMemory } from "@supermemory/ai-sdk/working-memory"

On addMemory, the new memory is inserted into WorkingMemory so a subsequent searchMemories for the same topic hits locally without a round-trip.

Cache policy

• LRU eviction at maxEntries (default 100) — bounded, prevents unbounded growth in long-lived agents.
• TTL per entry (default 60 s) — balances freshness vs. latency; stale entries are refreshed on next miss.
• Pin: pin(query) keeps high-value context (e.g., "user is allergic to peanuts") immune to LRU until unpin.
• Promise deduplication: concurrent searchMemories("preferences") calls share one client.search.execute promise.
• Source tagging: results annotated with _source: 'cache' | 'network' for debugging / evals.

API design

No breaking change — workingMemory is an optional field on SupermemoryToolsConfig. When omitted, behavior is identical to today (no cache). When enabled: true, searchMemories and addMemory automatically use the WorkingMemory layer.

export interface SupermemoryToolsConfig {
  baseUrl?: string
  containerTags?: string[]
  projectId?: string
  workingMemory?: {
    enabled?: boolean
    maxEntries?: number
    ttlMs?: number
  }
}

Expected latency / token savings

Scope (proposed PR)

• Single new file packages/ai-sdk/src/working-memory.ts (zero deps) + small wiring in packages/ai-sdk/src/tools.ts
• No backend change; no new package; no web/MCP/extension changes
• Out of scope: cross-process persistence, semantic dedup, embedding-level cache — deferred to phase 2

Questions for maintainers

1. Is an opt-in workingMemory field on SupermemoryToolsConfig the right surface, or would a standalone createWorkingMemory() wrapper be preferred?
2. Is 60 s TTL + 100-entry LRU the right default, or do agent sessions need a different bound?
3. Should addMemory auto-populate WorkingMemory, or should that be explicit (workingMemory.set(...) only)?

Verification

Manual verification via isolated parallel harness (10 scenarios, each in its own module instance, run concurrently as bun processes) covering: cache hit/miss, TTL expiry, promise dedup, pin immunity, invalidate (single + all), LRU eviction, concurrent callers, addMemory auto-populate, source tagging, and stats. No new test framework needed — packages/ai-sdk already has vitest.