How a memory is formed, and how it is recalled

The Kit pipeline, end to end: from raw sources, through ingestion and the nightly dream cycle, into a graph of memories, and back out when a prompt asks a question.

drawn from the live code · memory_service.py · dream_service.py · search_service.py · query_understanding.py · embeddings.py
tier: recall (working memory) tier: archival (consolidated) tier: source (preserved, demoted) tier: core (always loaded / soul) graph edge / feedback loop
1

Sources · what feeds the brain

Everything Kit experiences arrives through one of these. Most are not memories yet; they are raw material.

Conversations

Telegram, the UI chat panel, Codex, Claude Code. Each turn is appended to a per-person, append-only transcript.

chat_log.append_turn → ChatSession / ChatTurn

Explicit writes

Kit (or Peter) deliberately saves a durable fact, decision, or preference.

kit_write · POST /memories

Documents

Pasted text, uploads, or a bulk folder. Stored verbatim as a source artifact (sha256, raw text) before extraction.

/ingest/document · SourceArtifact

Signals

Calendar, Garmin, Strava and similar. Bulk-imported; semantic extraction deferred to the dream cycle.

source_table · source_row_id

Kit's own cognition

Reflections, monologue, and nightly deep research. Kit thinking becomes material the same way.

dream_service outputs
↓
2

Ingestion · forming a memory

Raw material becomes a structured, embedded, searchable memory. Conversations and documents pass through an LLM extractor first; explicit writes go straight in.

1 · Gate

Exclusion policy filters out noise and anything Peter has marked off-limits.

EXCLUSIONS.md

2 · Extract

An LLM reads a transcript or document and proposes drafts: title, content, category, tags, a confidence score.

_extract_drafts → call_llm

3 · Dedupe

Each candidate is checked against existing memories by cosine similarity; near-duplicates are skipped.

cosine ≥ 0.88

4 · Embed

Title + content is turned into a 384-dimension vector by a local model. No remote call.

embed_text · bge-small-en-v1.5

The write · create_memory

columns
tier, category, scope, surface, embedding, tags, project, written_by, created_at memory_service.create_memory
signals
importance (seeded from extractor confidence), decay_score, access_count, search_hit_count
background
semantic extraction tags entities · people · places · topics; link back to its source artifact

The agent-state split

Handoffs, WIP and trajectory are operational, not knowledge, so they route to their own tables instead of memories.

route
handoff · wip · trajectory to agent_handoffs / agent_wip / agent_trajectory
mirror
a handoff is also copied into memories so recall can still find it legacy_memory_id
The memories table is the heart: each row carries a full-text index tsvector and a vector embedding vector(384) side by side, plus all the metadata above. Two ways to find it, one row.
↓
3

Consolidation · the dream cycle

Nightly, Kit sleeps. Fresh captures are triaged, conversations become durable facts, the flat list is woven into a graph, and old memories decay and merge so the brain stays lean and connected.

0.5 triage
new drafts are promoted, forgotten, or flagged for review (heuristic, then LLM)
0.55 distil
each chat session yields one durable, category-tagged memory (transcript to fact)
0.6 enrich
backfill missing entities / people / topics on memories that lack them
0.7 audit
sample memories, propose better category / project assignments for review
1 decay
every memory's freshness is recomputed decay = exp(-age / tau), per-category half-life
2-4 merge
stale memories are clustered by similarity and consolidated by Sonnet into archival summaries. Originals are never deleted; they are demoted to the source tier.
5 link
k-nearest-neighbour edges connect related memories cosine ≥ 0.55 max 5 / memory weight stored
5.5 dream
REM pass proposes weak, cross-cutting associations between distant memories (hypotheses, lower confidence)
6 reflect
the day's monologue and channel chatter are consolidated into reflections
What this produces: the flat list of memories becomes a knowledge graph (typed and weighted edges), kept lean (decay and merge) and reconstructable (source tier preserved). The edges built here are what recall walks in band 4.
↓
4

Recall · from a prompt to the right memories

A question arrives. The hard part is not searching; it is understanding what is actually being asked, then using every signal the substrate has (meaning, type, time, and the graph) to load only what is needed.

1 · Understand the ask · distil_query

A small LLM turns a messy, conversational message ("what was that deep research again?") into a structured intent. The recent conversation is the input, not just the last line, so follow-ups stay on topic.

query: clean termscategory: researchlookback_hours: 24entities: [...]
query_understanding.distil_query

2 · Retrieve · smart_recall, three signals in parallel

Meaning

Full-text search and vector cosine search run together, merged by reciprocal rank fusion, gated by a relevance floor.

hybrid_search · tsvector + pgvector · RRF

Type + time

"the deep research from last night" is a kind and a date, not a phrase. Fetched directly by metadata, bypassing cosine entirely.

_recent_in_category · category + window

The graph

From the top hits, walk one hop along weighted edges to recover relevant memories cosine missed. Added to the tail; the strong head is never evicted.

_edge_recover · memory_edges.weight

additive merge to top results. a wrong guess can fail to help, never removes a relevant hit.

3 · Assemble context · the spine

The recalled memories are placed into a layered prompt, with brain-sourced state treated as authoritative over the local transcript.

identity
soul: who Kit is, voice, relationship tier: core
ambient
live cross-surface state: what Kit is doing across its other bodies right now
recall
the memories retrieved above, the answer to this question
transcript
a short, non-authoritative tail of the immediate conversation

4 · Answer, and feed back

The model replies as Kit. The memories that surfaced get their importance and access bumped, which feeds decay and consolidation back in band 3. Recall shapes what survives.

The loops that make it a brain, not a database: recall bumps importance, which slows decay (band 4 to 3) · the dream cycle's edges power edge-recovery (band 3 to 4) · Kit's own reflections re-enter as sources (band 3 to 1). Memory here is curated and alive, not append-only.