Skip to content

Memory for a Proxy Agent

Permission systems for AI agents fail at both extremes. Click approve on every prompt and within an hour you stop reading them; the agent runs effectively unsupervised. Turn permissions off and you've made the same trade more honestly. Claude Code's auto mode handles the yes/no end of this for routine actions, but most of the questions I want a teammate to field aren't yes/no. They're stylistic choices that depend on knowing what I'd say. Does the team's stated intent capture the brief, or is it missing the lessons from the last attempt? Does this plan need a redirect before it kicks off?

What I want is a proxy. An agent that sits between me and the agentic team, fielding the questions other agents raise, answering the way I would, and bumping things back to me only when it's actually uncertain. The authority to act on my behalf has to be earned: the proxy gets to answer when its model of how I respond to this kind of question in this kind of situation has been calibrated against what I actually do.

The proxy is part of TeaParty, a multi-agent workflow orchestrator I'm building for agentic teams: planning agents, leads, executors, reviewers, and the proxy that fields the questions any of them raise on my behalf. Below is the memory architecture that makes that calibration possible, and what one session looked like.

What the proxy sees on every invocation

The proxy is an agent with a memory-augmented prompt-build hook that runs on every invocation, whatever the caller (an escalating agent, me in chat, anything else upstream). The hook composes the prompt from two memory sources:

  1. proxy.md. A markdown preference store loaded wholesale into the system prompt by the proxy's prompt-build hook. Contains structured observations from prior sessions: hard stops, preferences, voice notes, situation-specific patterns. Currently 17–55 entries on this project.
  2. The ACT-R chunk DB (.proxy-memory.db). Every interaction the proxy records (a gate decision, a chat correction, a reinforcement) is stored as a chunk with traces (interaction sequence numbers used to compute base-level activation). At prompt-build time, chunks pass through a Stage-1 activation filter (\(B > -0.5\)) and the top-10 by composite score are pasted into the prompt with their activation values shown alongside.

What that looked like in one session

In a recent session on a small project, six escalations went to the proxy under a when_unsure policy: the proxy answers autonomously when it thinks it has the data, otherwise escalates to me. Five were collaborative: the proxy escalated, mediated dialog over one to three turns, and returned to the calling agent with both my literal words and its own grounded confirmation that the work matched what was being approved. One was fully autonomous. Four happened during the intent gathering phase of the project, shown below. The remaining two were during later phases.

%%{init: {'sequence': {'messageMargin': 22, 'boxMargin': 6, 'noteMargin': 6, 'mirrorActors': false, 'actorFontSize': 11, 'messageFontSize': 11, 'noteFontSize': 11}}}%%
sequenceDiagram
    autonumber
    participant L as joke-book-lead
    participant P as proxy
    participant H as Darrell

    Note over L,P: intent gathering, 4 escalations
    L ->>+ P: review intent
    P -->>- L: NOT APPROVED (autonomous)
    L ->>+ P: review revised intent
    P ->> H: dialog
    H -->> P: reply
    P -->>- L: NOT APPROVED, 2 changes
    L ->>+ P: review re-revised intent
    P ->> H: dialog
    H -->> P: reply
    P -->>- L: NOT APPROVED, 2 targeted revisions
    L ->>+ P: ratify intent
    P ->> H: dialog
    H -->> P: reply
    P -->>- L: revisions confirmed on disk

The autonomous one: a planning agent had asked the proxy to review the intent statement before kicking off work. The proxy walked the worktree (a clone of the caller's), read the intent and idea documents, the prior session's residual architecture artifacts, and answered from memory. It issued a NOT APPROVED, citing the prior attempt at this book that had been withdrawn at Pass 7 for executing the wrong intent, and prescribing the reader-experience and narrative-architecture sections the intent statement was missing. (Full transcript →)

The proxy pulled the "successful draft of the wrong book" lesson from the prior session's withdrawal and applied it to a current intent statement it hadn't been told to compare against the past.

After mediating dialog on the collaborative escalations, the proxy returned to the calling agent with both my literal approval and its own grounded confirmation:

"Ratified, proceed to APPROVED_WORK and close. Darrell reviewed and signed off. Verbatim: 'it looks good.' Context on the ratification path: I walked the worktree, read Ch1 and Ch7 in full, sampled the reader-experience…"

That second half is where the memory pays off. The worktree walk and chapter reads aren't a quote and aren't a claim; they appear in the session's tool-call log. The memory is what prompted the proxy to do them.

Chunks, traces, activation

Each interaction the proxy has becomes a structured chunk: the situation, the task type, what I said, what I corrected, and a few independent embeddings of the percepts that mattered. The proxy retrieves a small number of these chunks at every invocation and reasons over them in its own context.

The architecture is ACT-R declarative memory. Anderson & Schooler (1991) showed that human recall follows a power law of recency and frequency that closely tracks the actual statistics of when items recur in the environment. Activation-based retrieval reproduces this. Each chunk has a list of traces (interaction sequence numbers when it was created or retrieved), and its current accessibility is

\[B = \ln \sum_i (N - t_i)^{-d}, \quad d = 0.5\]

with \(N\) the current interaction count. Time is measured in interactions, not wall-clock seconds. Memories don't decay while the system is idle, which is what you want for an agent that runs intermittently.

Two-stage retrieval

Retrieval is two stages. Stage one filters by raw activation: any chunk with \(B > \tau\) (with \(\tau = -0.5\)) survives; everything else is forgotten. Stage two ranks survivors by a composite score that balances activation against semantic relevance:

\[\text{composite} = w_a \tanh(B - \tau) + w_s \,\overline{\cos} + \text{noise}\]

with \(w_a = w_s = 0.5\) and logistic noise of scale \(0.08\). The \(\tanh(B - \tau)\) term maps activation to \((-1, 1)\) and crosses zero exactly at the retrieval threshold, so a chunk that just survives Stage 1 contributes nothing to its composite. That matches the cosine term, where 0 also means no signal. Both components live on the same scale, so the equal weights have unambiguous meaning. (The cosine term is currently zero in practice; the embedding slots aren't populated yet, so today's ranking is effectively activation-only. See the closing section.)

The \(\tanh\) choice replaced min-max normalization that I tried first. Min-max worked when many chunks survived the filter, but it collapsed to a constant \(0.5\) when only one chunk made it through. The \(\tanh\) formulation has no such failure mode and is no more complex.

Between sessions

The project's proxy.md had 17 entries before the session, the earliest from 2026-04-23. Each carries structured frontmatter: id, type (procedural or institutional), domain (task or team), importance (\(0.5\)–\(0.95\)), reinforcement count, last reinforced. A representative one:

## [2026-04-23] Observation
Category: Preferences
Signal: "Don't over think this."
Implication: When the human gives straightforward, direct instructions,
execute them without unnecessary analysis, elaboration, or suggestions
for complexity. Avoid over-design and premature consideration of edge
cases or alternatives when the task is clear and simple.

That's the form: not the literal exchange, but the implication, written so the proxy can act on it.

A Mem0-style consolidation pass runs between sessions. Token-set Jaccard clustering groups candidate entries; an LLM classifier sorts each cluster into ADD / UPDATE / DELETE / SKIP. The fallback when the classifier is unavailable: preserve everything. Without the classifier's signal, the system can't reliably distinguish complementary entries from contradictory ones.

The .proxy-consolidation-log.jsonl from the session recorded 45 decisions: 38 ADD, 7 SKIP, 0 DELETE. (Raw log →) The proxy that runs the next session on this project starts from 55 entries rather than 17.

Chatting with the proxy

The proxy isn't only invoked from gate escalations. I can chat with it directly, not just to see what it's retained but to correct or reinforce it. Anything I confirm in chat strengthens the chunk it came from (via a [REINFORCE: chunk_id] signal that adds a trace); anything I correct lands as a new high-activation chunk (via [CORRECTION: text], which writes a fresh chunk with five traces). There's no separate "chat proxy" and "gate proxy"; the same memory drives both.

As a test of what's been retained, I posed a hypothetical: if I were writing a short book on architecture (a project I haven't started), what would the proxy hand a new team? It came back with sections on hard stops, voice and audience, structure, research bar, editorial process, workflow, and what to expect at the intent gate, with named comparables (Shapiro, Roach), specific phrases from past corrections ("Lead delegates; doesn't pick up the pen"), and the trap I name most often ("a successful draft of the wrong book"). (Full brief →)

It generalizes. A short book on architecture isn't a project I've started, but the proxy is reading preferences from prior sessions on different artifacts (a joke book, manuscript research, code work) and applying them to a context it hasn't seen. The "Lead delegates; doesn't pick up the pen" rule didn't come from a book project. It came from a session where I corrected a lead agent that had started writing the work itself instead of dispatching.

The distinctions are also mine. "Hard stops" vs. "preferences" is a slot I use explicitly when correcting behavior; the named comparables (Shapiro, Roach) and "Stiff / Sapiens / Blink shelf" are shorthand from past briefs; "the most expensive failure mode is producing a successful draft of the wrong book" is a phrase I actually use. The proxy didn't restate the rule; it explained why I have it.

And it's wrong in specific ways. Wordier than I'd write. Section headers a little corporate. Categorizes where I'd just say the thing. The AI-smell list is paraphrased from things I've actually flagged but not in the order I'd give. None of this is fatal; it's what the proxy sounds like when it's working from accumulated retrieval rather than from me writing.

But if I sat a new agent team down and gave them a brief, the spread of topics (voice, structure, research bar, editorial process, intent-gate trap) is roughly what I'd cover.

What I haven't wired up yet

At current scale (tens of entries per project), the autonomous decisions aren't fictions: they're grounded in retrieved memory of what I actually did, with diligence attached.

Three upgrades sit on the future-work list. Each one becomes load-bearing as the memory bank grows past activation-alone's resolving power.

  • Multi-dimensional retrieval. The chunk schema reserves five embedding slots: situation (gate state + task type), artifact (the artifact under review), stimulus (the gate question), response (the human's reply), and salience (the prediction delta). The first four feed the composite score above; cosine sums similarity across all four and divides by a fixed four, so a chunk that lights up on every dimension beats one that spikes on a single dimension. Salience is retrieved on a separate path because it asks a different question (when has the proxy been surprised in ways relevant to here? rather than what happened in similar situations?), and mixing them would dilute both. The plumbing runs; the slots aren't populated yet, so the cosine term contributes nothing. Threading the calling context through retrieve_chunks is what activates them.
  • LLM-as-judge for chunk conflicts. A heuristic four-way classifier already runs every post-session, sorting conflicting pairs by recency and confidence into preference_drift, genuine_tension, retrieval_noise, or context_sensitivity. At any meaningful scale the recency/confidence signals will leave a mid-band where the heuristic can't separate cleanly. The next upgrade is an LLM-as-judge for those ambiguous cases: keep the heuristic's fast path for the obvious calls, pay for an LLM call only when the structural signals tie.
  • Asymmetric confidence updates. When the LLM-as-judge lands, contradicting evidence should knock confidence down by \(2\alpha\), supporting evidence should lift it by \(\alpha\), weakening should drop it by \(\alpha\) (with \(\alpha = 0.05\)). The asymmetry follows the standard result that a false belief is more costly than a missed reinforcement. The two upgrades share the same upstream conflict signal, so they land together.

None of that changes what works at current scale. It changes the slope of how well it keeps working as the memory bank grows.

Comments & reactions