Every AI agent session starts blank. So does every session after context compaction, which is when the tool shrinks a long chat into a short summary. The agent has the model’s skills and nothing else. It does not know my rules, where the project stands or what it was doing ten minutes ago.
I run many agents, so this happens all day. I built a memory system for it and call it the memory palace. It is made of plain markdown files, a few scripts and a handful of hooks. A hook is a small script that the agent tool runs at a fixed moment, such as when a prompt arrives. The system has no database and no service.
Why I built it
Without memory, a fresh agent works out the same facts again, repeats old mistakes and trusts stale chat memory. One agent told me that a fact was missing when someone had written it down two days earlier, and it did this more than once. A rule that I gave once was gone after the next restart. An agent also “remembered” a state that had changed since, and then acted on it.
Before the palace, I tried an automatic memory tool and removed it. After two days it had stored prompts but no observations, and it still reported itself healthy. So the palace follows one design rule: it uses plain files that anyone can read with ls and grep.
I do not load everything into every session. One short rules file at the workspace root loads every time and holds the rules that always apply. It has a hard size cap, because some agent tools cut off a longer file without any warning. When a section grows, I move the full text out and leave one binding sentence behind.
A short references index says “read this file when this happens”, with triggers such as handing work to another agent or landing a change. Each reference file starts with two lines that say when to read it and why it exists. Each project has its own README, state file, task list, decisions file and audit log. Then there is the palace itself, which is a folder of notes with one fact per note. Next to it are one small checkpoint file per task, which holds the next step, and an inbox for drafts that a script captured automatically and that are not rules yet.
How a note is written
A note holds one fact. It starts with a short header: a name, a one-line description and a type. The type is feedback (a rule about how to work), project (a fact about a system) or reference (where to find something). A feedback note gives the rule in a sentence or two, the reason in my own words with the date, and how to apply it, so the next agent knows when the note matters.
I keep my own words in the note. One note says that a finding is not done until a mechanism stops it from happening again. Its “why” quotes me: “create a system for these do not just note it and then leave it”. An agent that reads the quote gets the intent, so it can apply the rule to cases that the note never mentioned. The date on each note lets a later agent judge whether the note is still current.
The palace holds hundreds of notes, and no agent can read them all at startup. So one index lists them, with a link and the description on each line, and a script builds it from the notes themselves. The top of the index says “do not hand-edit below”. The index groups notes into rooms by type, with a last section for superseded notes. Agents search the index for the topic at hand and then open the matching notes.
The same script does three jobs. It rebuilds the index, the backlinks and the rooms. It answers a question by searching the palace and the standing project records, and it returns one answer with its sources in order of authority. And it checks for orphan notes, broken links and missing headers, and it fails when the palace is in bad shape.
The script’s comments record an early bug. The first index generator matched nothing, because the header keys were indented, and it rebuilt the index empty over a good one. A generator that can silently produce nothing has the same defect as a test that cannot fail.
Startup, hooks and checkpoints
At startup, after compaction and when a task changes owner, the agent runs the same routine. It reads the workspace state file and finds the project in the registry. Then it reads the project README and any local rules, followed by the project’s state, decisions, tasks and the latest audit-log entries. Last, it verifies anything that can change against the real repository or service. This last step matters most, because a note says what was true when someone wrote it.
A rule that depends on the agent remembering to read the palace fails on the day the agent forgets. So a hook runs on every prompt. It gives the agent the path of the palace index with the instruction to read matching notes before it works out a fact again. It gives the path of the agent’s own checkpoint file, with its size and last update time. And it adds one rule: record each request from me, word for word, in the checkpoint before starting on it. After compaction, my original words are still on disk.
A second hook runs when the agent finishes a turn. If the turn changed files but not the checkpoint, the hook warns me and the agent. It never blocks anything, and every path in both hooks exits cleanly, so a bug in a hook cannot stop a prompt.
The comments in an older session-start hook show why I chose this way. I wrote: “IF YOU ARE UNSURE YOU LOOK FOR PALACE.” The palace was named in the instructions but never actually in the context. A pointer that someone must follow by hand gets skipped, and the agents drift.
Notes hold lasting knowledge, and a checkpoint holds the state of one task. It records the objective, the environment, where the work stands, the exact next action, the checks that remain, the sources and the risks. An agent saves a checkpoint after a significant change, when it waits or hands over, and before it ends or compacts. The store keeps numbered revisions and never overwrites an old one. Only one session owns a task at a time, and a transfer needs an explicit release and claim. Curated notes and the project task list stay authoritative, and anything that a hook records is marked unverified.
The inbox, and getting everything into one place
Automatic capture is useful and risky. A script saves what I say in chat as a draft note. Each draft says that it is unreviewed and “NOT a palace rule” until a reviewer rewrites it and moves it into the palace. Agents may read the drafts but may not cite them as rules.
Memory had also spread across more than one store. I told the agents to “unify the memory systems so all is in the same place”. The notes from the other store moved into the palace, marked with their date and source. The old store is now a read-only rollback copy that agents must not read or write.
What went wrong, and what it still costs
Three things went wrong, and each one left a rule behind. A cached copy can look current and be wrong, so now a stale cache grants no authority, and an agent never claims a task, transfers it or marks it done from a cache. I had given agents the same rule before, and a note existed, but nothing acted on it. So now every finding gets a mechanism that stops a repeat. The per-prompt hook is that kind of mechanism, because it acts on every prompt and nobody has to remember. And an agent once answered from what it remembered, so now an agent verifies mutable facts against the repository or service.
The setup has real limits. The index is too big to load whole, so agents depend on search to find the right note. Notes go stale, and someone still has to retire the old ones. The hooks warn and do not block, on purpose, because a hook that blocks work causes its own failures, but this means that an agent can ignore a warning. And a note exists only if an agent writes it after a verified outcome, so the system needs discipline.
I would keep three parts in any setup: a small always-loaded file with a hard cap, a reminder that lives in a hook and not in good intentions, and a saved next step before every pause. Memory tells the agent where to look. It does not tell the agent what is true today.
Want a memory setup like this in your repository? Email hello@jeffkent.dev.
