The dangerous moment in an agent memory system is write-back.

An agent observes a pattern. It summarizes the pattern confidently. The summary lands in a durable store. The next agent reads the summary as reality, makes a decision based on it, and ships. That is how a guess becomes infrastructure.

Retrieval gets most of the attention because it is easy to demo. Ask a question, watch an old detail reappear, put a similarity score beside it. The harder problem starts after retrieval: deciding what kind of thing came back, how much authority it has, and whether it is allowed to change what future agents treat as true.

If the read path decides what the agent sees, the write path decides what the system becomes.

A governance swimlane showing observations becoming synthesis proposals and only entering canon after review.

Memory is five jobs hiding under one label

Most agent-memory tools pitch the same surface: durable context, continuity across sessions, retrieval, learning over time. Hindsight, Hippo, Honcho, AgentMemory, and GBrain do not own the same job, though. "Memory" is being used as one name for several objects with different authority, scope, and lifecycles.

ObjectJobAuthorityLifecycle
CanonReviewed facts, rules, and decisions about the world or projectSource of truthExplicit revision
RecallPrior material surfaced because it may be relevant nowCandidate evidenceAdaptive
Relationship statePreferences, roles, venue norms, and collaboration defaultsScoped influence, not world truthRevision and expiry by person, workspace, server, repo, task, and time
TracesCommands, diffs, tests, errors, rejected hypotheses, and review outcomesEvidence of what happenedTask and tool retention
SynthesisPatterns or mental models inferred from observationsProposalContradiction, revision, or rejection

The distinction is not academic. These objects need different write permissions. An agent can record a command it ran without asking anyone. It should not turn three similar commands into a permanent repo rule without showing the evidence and crossing a review gate.

Mechanisms sit on another axis. Sliding windows, summaries, vector indexes, keyword search, graphs, Markdown, and self-managed memory tools are ways to store or retrieve the objects above. They do not decide the objects' authority. A graph edge can be a reviewed fact or an extracted guess. A Markdown sentence can be canon or stale nonsense. An embedding hit can retrieve either one.

Confusing mechanism with job is how an architecture diagram ends up with five boxes labelled "memory" and no answer to the only question that matters: who is allowed to treat this as true?

A vector hit is a candidate

A semantic memory system can retrieve the perfect-looking sentence at exactly the wrong time.

Ask about authentication patterns and the store might surface a six-month-old design document describing the previous scheme. The paragraph is specific. It uses the same vocabulary as the current repo. The similarity score is excellent. It is also stale.

The search found nearby text, exactly what it was built to do. The failure happened when the pipeline shipped that text into the prompt without its source, date, or authority and let the model treat proximity as proof.

Exact details fail differently. Say the user tells the agent: preview servers for this repo use port 9473. A later summary compresses that into "discussed preview server setup" and drops the number.

Embeddings may find the original message when the query says "preview server port." Keyword search helps because 9473 is rigid. Reranking helps put the right result above unrelated port-shaped junk. The hybrid stack is better retrieval, but it still returns a candidate.

A useful result needs to answer more than "what text matched?" It should carry:

  • the source and object type
  • when it was written and last verified
  • its scope
  • what evidence supports it
  • what contradicts it
  • whether the current agent may act on it or should verify first

That metadata preserves uncertainty across the retrieval boundary. The agent can say, "This came from an old design note; I should check the current code before using it," instead of silently inheriting yesterday's confidence.

Memory can suggest. Canon decides.

Different objects should age differently

Storage without lifecycle turns into hoarding. Hippo's useful contribution is not a leaderboard claim; it is the framing that good memory includes forgetting. Observations can decay when stale. Retrieval can strengthen records that keep proving useful.

Repeated evidence can consolidate. Conflicts should be surfaced rather than quietly overwritten. Failed approaches may deserve longer retention because forgetting them makes the next agent pay to rediscover the same dead end.

Those rules do not apply evenly.

Canon should not decay because nobody asked about it last quarter. A reviewed rule remains the rule until the project changes it. Tool traces age quickly because the harness and code move. Relationship state ages by scope: "keep replies short in this deploy channel" should stay inside that channel instead of becoming a global personality law.

The useful scope for relationship state is closer to person × workspace × server × repo × task × time than a single user_id. The system may be storing inferences about a person, which demands a stricter privacy posture than codebase facts. Consent, visibility, and expiry belong in the data model from the start.

Lifecycle is therefore part of the write contract. Before storing anything durable, the system should know what kind of object it is, where it applies, and how it can become stale.

Observation is not synthesis

An observation is a tool trace, session event, test result, or captured message that records what happened. It sits below the table above. Synthesis interprets observations into a pattern; canon is the reviewed result that future agents may rely on.

These steps need separate write permissions because they make different epistemic promises. Observations can be captured freely. Synthesis should be deliberate and attributed. Canon requires a gate.

A concrete example walking up the ladder:

  • Observation: Three agents over two weeks failed the same Playwright test with EADDRINUSE on port 4173.
  • Synthesis: The local test flow may have unstable server teardown between runs.
  • Canon proposal: Update docs/testing.md with an explicit port check and add a pretest guard.

Reporting the observation is cheap. Turning it into a synthesis requires evidence, while the durable doc and script changes should ship through review.

The same boundary applies to reflective systems. Hindsight's reflect() operation is useful because it can form mental models from prior observations. The problem begins when the mental model lands wherever the next agent reads "facts." Reflection without provenance is hallucination at rest. A synthesized belief should point back to the observations that produced it and remain a proposal until something accepts it.

Proposal needs evidence

A useful agent does more than say, "I learned X." It says, "I think X should become canon, and here is why."

The "here is why" should be inspectable without replaying the whole session. Link to the traces, diffs, test results, accepted PRs, review comments, or current docs that support the proposal. A summary of the evidence is not the evidence.

The write path I keep landing on is boring on purpose:

  1. The agent captures observations while it works: commands, diffs, tests, failures, and rejected hypotheses.
  2. Recall surfaces related prior evidence without declaring it true.
  3. A reflective step notices a pattern and drafts a durable lesson.
  4. The proposal lands as a diff against canon, with links back to its evidence.
  5. A human, review agent, or automated gate checks the transition.
  6. Canon changes. Future agents read the reviewed rule rather than inheriting the proposal.

In my stack, canon usually lives in Markdown and git, including AGENTS.md, architecture docs, or a wiki that survives harness swaps. Git supplies history, authorship, diffs, and a familiar review surface. Markdown is not magic; reviewed SQL rows or graph changes could carry the same contract. The requirement is inspectability and an explicit transition from "an agent thinks" to "we agree."

Two canonical stores means no canonical store. Trace databases can retain what happened, vector indexes can retrieve related material, and relationship stores can shape tone and defaults. None of them should quietly become a second authority about the repo.

Graph writes make the hidden choices visible

Knowledge graphs make write-back look more deterministic than it is. Once stored, Mara → maintains → billing adapter resembles a clean fact. The path that created it was messy: a model extracted names, resolved identities, chose a relation, decided direction, and omitted uncertainty.

Every one of those choices can rot:

  • Mara, mara-k, and @mk may or may not be the same person.
  • maintains is not the same as reviewed once.
  • works_on probably expires; authored usually does not.
  • depends_on has direction, and reversing it poisons traversal.
  • a migration from REST to GraphQL contradicts an old edge rather than politely erasing its history.

Graph writes need the same contract as every other durable claim: source, timestamp, scope, confidence, relation type, and a way to mark contradiction or expiry. Otherwise the graph becomes a confident rumor mill with excellent traversal.

The gate is the moment of agreement

Review is usually criticized as the human bottleneck. That frames the gate incorrectly. The gate is not the human. It is the moment a claim moves from "an agent thinks" to "this system may rely on it." Whoever owns that transition owns the architecture.

The gate can be fast. It can be risk-tiered. A low-impact preference with explicit scope may write immediately. A repo-wide rule inferred from tool traces should arrive as a proposal. High-confidence changes with strong provenance may eventually pass automated review; noisy or consequential claims still need a person.

Agents are not locked out of writing under this model. They can capture observations, maintain scratch state, synthesize proposals, and prepare canon updates. What they cannot do is silently promote their own interpretation into infrastructure.

Provenance does not eliminate hallucination. It makes hallucination inspectable, and inspectable means contestable.

Read paths are projections

The inverse problem still matters: which memory should each agent read?

Planning wants broad history. Implementation wants deep local context. Review wants fresh evidence without the author's self-justifying plan. An orchestrator wants status and routing signals that the coding agent does not need. More context is not automatically better context; the projection should match the job.

That routing argument already has a fuller home in Context Is the Constraint. The detailed trace side, linking code, sessions, prompts, tools, and review outcomes, belongs in The code tells you what. The trace tells you why. Memory architecture should define the authority boundaries rather than duplicating either implementation guide.

One canon, many memory layers

The architecture is not one giant database and it is not seven products pretending to be one brain.

Recall finds candidates; relationship state shapes collaboration inside explicit scope; traces preserve evidence; reflection proposes patterns; and projections decide what each phase sees. Canon remains the reviewed place where durable claims land.

Read paths will keep changing as retrieval gets better. The write path is harder to replace because every unreviewed claim it persists becomes input to the next decision. Pay the cost there.

One canon. Many memory layers. Every durable claim earns a path back to evidence.