Internals

Architecture

The reference implementation is one browser-safe TypeScript core in src/, wrapped by three surfaces: a browser app, a Bun CLI, and an MCP server. Every surface runs the same hashing, crypto, and search code; only the I/O differs.

Module map

Everything under src/ except the CLI and MCP entry points is browser-safe and free of Node APIs. The files, one line each:

Modules under src/.
FileResponsibility
types.tsSPEC_VERSION and the shared types: Role, Cell, CellInput, KdfInfo, CipherInfo, VaultFile, MemoryStats.
bytes.tsbase64, UTF-8, hex, and ArrayBuffer helpers built on browser globals (btoa/atob, TextEncoder).
canonical.tsThe deterministic JSON serializer: keys sorted recursively, no insignificant whitespace, undefined members omitted.
crypto.tsWebCrypto primitives: SHA-256 to lowercase hex, and PBKDF2-SHA256 key derivation to an AES-GCM 256-bit key; DEFAULT_ITERATIONS is 250,000.
cell.tsBuilds a Cell, assembles its six-field canonical payload, and computes its SHA-256 hash.
merkle.tsThe Merkle root over sorted cell hashes, plus verifyCells to recompute and flag bad cells.
bm25.tsThe tokenizer and BM25 lexical ranking, the recall used by the CLI and MCP server.
semantic.tsCosine similarity, vector normalization, and hybrid lexical-plus-semantic ranking for the browser app.
stats.tsCell count, source and tag histograms, and plaintext byte size.
vault.tsencryptVault and decryptVault (AES-256-GCM under a PBKDF2 key) and vaultSummary.
vault-ops.tsHigher-level vault operations: mergeCells, forgetCells, rotateVault, diffVaults.
context.tsThe token-budgeted Markdown context pack, built on demand and never stored.
interchange.tsThe memory pack: toBundle, fromBundle, and isBundle. See memory packs.
conformance.tsRuns and formats the published conformance vectors (L1 parse, L2 integrity, L3 crypto round-trip).
index.tsThe browser-safe barrel that re-exports the library; the entry point for the browser bundle.
cli.tsThe keepsake CLI entry point. The only file that uses node:fs, node:path, and node:url.
mcp.tsThe MCP server over stdio; imports the core from index.ts and node:fs for vault writes.

The src/import/ directory holds the parsers that feed index.ts: detect.ts labels a source from a filename, chatgpt.ts handles ChatGPT exports and generic message extraction, claude.ts handles Claude conversations.json, jsonl.ts handles JSONL transcripts, and import/index.ts normalizes and dispatches an input to the right parser. There is an empty src/mcp/ directory; the server itself lives in src/mcp.ts.

Data model

Two structures carry the format. A Cell is one memory; a Vault is the encrypted container that holds them.

Cell members (src/types.ts).
FieldTypeMeaning
idstringA UUID in canonical lowercase hyphenated form. Unique within a vault.
textstringThe memory itself.
sourcestringWhere it came from, such as an import label or a conversation title.
roleuser | assistant | note | systemThe speaker or kind of the text.
createdAtstringAn ISO 8601 timestamp.
tagsstring[]Free-form labels.
hashstringLowercase hex SHA-256 of the canonical payload, defined below.

A VaultFile is JSON with the members format ("keepsake/v1"), createdAt, cells (a count, not the array), kdf (name, iterations, base64 salt), cipher (name, base64 iv), base64 ciphertext that includes the 16-byte GCM tag, and merkle. The plaintext inside ciphertext is the canonical JSON of the full cells array.

The canonical payload and the cell hash

The canonical payload is the six non-hash members of a cell. The cell hash is SHA-256(canonicalJson(payload)), rendered as lowercase hex. Because the serializer is deterministic, two implementations that hold the same cell compute the same hash.

Canonical JSON

src/canonical.ts is the exact encoding that is hashed and encrypted, so every implementation must produce identical bytes. Its rules:

  1. Objects. Keys are emitted in ascending order, compared by Unicode code point. Members whose value is undefined are omitted. There is no insignificant whitespace: { and } hug the members, and , and : are unadorned.
  2. Arrays. Element order is preserved, and arrays may be empty. An undefined element becomes null.
  3. Strings. Standard JSON escapes; control characters below 0x20 use the shortest form. Non-ASCII characters are emitted literally as UTF-8, not \u-escaped.
  4. Numbers. Emitted as JSON numbers with no leading zero and no trailing fraction zero. Only integers appear in the fields this format defines.
  5. Booleans and null. Emitted as true, false, and null.
  6. Edge cases. null and undefined serialize to null; a non-finite number serializes to null; a bigint serializes to its decimal string.

The Merkle root

The root commits to the set of cell hashes and does not depend on cell order:

  1. Take every cell's hash as a lowercase hex string.
  2. Sort the list in ascending lexicographic order; for lowercase hex this equals byte order.
  3. While more than one value remains, walk the list in adjacent pairs and replace each pair (left, right) with sha256(left + right), where + concatenates the two hex strings and the result is hashed as UTF-8. An odd final value is carried to the next level unchanged, never duplicated.
  4. The single remaining value is the root. An empty cell set has root sha256(""), which is e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855.

The crypto boundary

Cryptography is confined to two modules, both using only the WebCrypto API so the same code runs in a browser tab and in Bun:

  • src/crypto.ts calls crypto.subtle.digest("SHA-256", …) for hashing and crypto.subtle.importKey plus crypto.subtle.deriveKey to turn a passphrase and salt into an AES-GCM key with PBKDF2-SHA256.
  • src/vault.ts fills the salt and IV with crypto.getRandomValues (16 and 12 bytes) and calls crypto.subtle.encrypt and crypto.subtle.decrypt with AES-GCM.
  • src/cell.ts generates ids with crypto.randomUUID().

This is the reason for the hard rule that src/ must stay browser-safe: no node: imports and no Node-only globals, because the core is bundled for the browser. The only files that break that rule are the two entry points that need a filesystem, src/cli.ts (node:fs, node:path, node:url) and src/mcp.ts (node:fs). The vault is never opened by a remote key or a hosted service; decryption happens in process, with a passphrase read from the environment by the CLI or entered in the browser.

One core, three surfaces

src/index.ts is the shared barrel. Each surface imports it, so there is a single implementation of the format:

Browser app
The landing page loads the generated assets/app.js bundle, which exposes the library as the global Keepsake. Import, search, seal, and open all run in the tab.
Bun CLI
src/cli.ts imports ./index.ts and ./mcp.ts, parses arguments, and reads and writes vault and pack files. It is the reference for scripts and CI.
MCP server
src/mcp.ts imports the core from ./index.ts and serves recall, context, stats, verify, remember, and forget over stdio.
Shared engine
Hashing, canonical JSON, the Merkle root, AES-GCM, BM25, context packs, and the memory pack are the same functions everywhere.

Module dependencies

Arrows point from a module to what it imports. index.ts re-exports the leaves and is the only entry point the surfaces share.

src/ module flow
types.ts   (shared types, no imports)
  |
  +-- canonical.ts   key-sorted JSON
  +-- bytes.ts       base64 / utf8 / hex
  |     +-- crypto.ts    SHA-256 + PBKDF2
  +-- bm25.ts        BM25 ranking
  +-- semantic.ts    cosine (browser)
  +-- stats.ts       counts + bytes
  |
  +-- cell.ts    (canonical + crypto)
  |     +-- merkle.ts
  |           +-- vault.ts
  |                 +-- vault-ops.ts
  |     +-- vault-ops.ts
  +-- interchange.ts  (canonical/cell/merkle)
  +-- conformance.ts  (bytes/canonical/vault)
  +-- import/*.ts  ->  index.ts barrel
                           |
              +------------+------------+
        cli.ts (+ node:*)   mcp.ts (+ node:fs)
                           |
              browser app  <--  build-site.mjs
                           (esbuild -> app.js)

Build pipeline

scripts/build-site.mjs bundles the core for the browser. It imports node:fs/promises, node:fs, node:path, node:url, and esbuild, then builds with these options:

esbuild options in scripts/build-site.mjs.
OptionValue
entry pointsrc/index.ts
outputsite/assets/app.js
bundletrue
formatiife
platformbrowser
targetes2022
minifytrue
globalNameKeepsake

The bundle is a single self-contained IIFE; the page loads it with a local <script> and calls into the Keepsake global. If src/index.ts is missing the script prints a skip message and returns. The rule that src/ has no node: imports is what makes this bundle possible: esbuild can resolve the whole graph for the browser with no Node shims.

Testing

Tests live in tests/ and run with bun test. The suite is currently 158 tests across 17 files, with 319 expect() calls.

bun test
$ bun test158 pass  0 fail319 expect() callsRan 158 tests across 17 files.

The 17 files cover:

Format and integrity
canonical.test.ts, cell.test.ts, merkle.test.ts — key ordering, the canonical payload and hash, odd and even pairing, and the empty root.
Primitives
bytes.test.ts and crypto.test.ts — base64 and UTF-8 round-trips, SHA-256 vectors, and PBKDF2 derivation.
Container
vault.test.ts and vault-ops.test.ts — seal and open, wrong-passphrase and tamper rejection, merge, forget, rotate, and diff.
Recall
bm25.test.ts, context.test.ts, and semantic.test.ts — ranking, the token budget, cosine, and hybrid blending.
Interchange
interchange.test.ts and conformance.test.ts — pack round-trips, tamper rejection, and the published vectors.
Import and entry points
import.test.ts, import-extra.test.ts, cli.test.ts, and mcp.test.ts — the parsers, argument handling, exit codes, and the MCP tools.

bun run typecheck runs tsc --noEmit, and bun run check:site validates every site/*.html page: exactly one <h1> and one <main>, a skip link that resolves, and only classes defined in lens.css or theme.css.