Internals
Architecture
The reference implementation is one browser-safe TypeScript core in src/, wrapped by three surfaces: a browser app, a Bun CLI, and an MCP server. Every surface runs the same hashing, crypto, and search code; only the I/O differs.
Module map
Everything under src/ except the CLI and MCP entry points is browser-safe and free of Node APIs. The files, one line each:
| File | Responsibility |
|---|---|
types.ts | SPEC_VERSION and the shared types: Role, Cell, CellInput, KdfInfo, CipherInfo, VaultFile, MemoryStats. |
bytes.ts | base64, UTF-8, hex, and ArrayBuffer helpers built on browser globals (btoa/atob, TextEncoder). |
canonical.ts | The deterministic JSON serializer: keys sorted recursively, no insignificant whitespace, undefined members omitted. |
crypto.ts | WebCrypto primitives: SHA-256 to lowercase hex, and PBKDF2-SHA256 key derivation to an AES-GCM 256-bit key; DEFAULT_ITERATIONS is 250,000. |
cell.ts | Builds a Cell, assembles its six-field canonical payload, and computes its SHA-256 hash. |
merkle.ts | The Merkle root over sorted cell hashes, plus verifyCells to recompute and flag bad cells. |
bm25.ts | The tokenizer and BM25 lexical ranking, the recall used by the CLI and MCP server. |
semantic.ts | Cosine similarity, vector normalization, and hybrid lexical-plus-semantic ranking for the browser app. |
stats.ts | Cell count, source and tag histograms, and plaintext byte size. |
vault.ts | encryptVault and decryptVault (AES-256-GCM under a PBKDF2 key) and vaultSummary. |
vault-ops.ts | Higher-level vault operations: mergeCells, forgetCells, rotateVault, diffVaults. |
context.ts | The token-budgeted Markdown context pack, built on demand and never stored. |
interchange.ts | The memory pack: toBundle, fromBundle, and isBundle. See memory packs. |
conformance.ts | Runs and formats the published conformance vectors (L1 parse, L2 integrity, L3 crypto round-trip). |
index.ts | The browser-safe barrel that re-exports the library; the entry point for the browser bundle. |
cli.ts | The keepsake CLI entry point. The only file that uses node:fs, node:path, and node:url. |
mcp.ts | The MCP server over stdio; imports the core from index.ts and node:fs for vault writes. |
The src/import/ directory holds the parsers that feed index.ts: detect.ts labels a source from a filename, chatgpt.ts handles ChatGPT exports and generic message extraction, claude.ts handles Claude conversations.json, jsonl.ts handles JSONL transcripts, and import/index.ts normalizes and dispatches an input to the right parser. There is an empty src/mcp/ directory; the server itself lives in src/mcp.ts.
Data model
Two structures carry the format. A Cell is one memory; a Vault is the encrypted container that holds them.
| Field | Type | Meaning |
|---|---|---|
id | string | A UUID in canonical lowercase hyphenated form. Unique within a vault. |
text | string | The memory itself. |
source | string | Where it came from, such as an import label or a conversation title. |
role | user | assistant | note | system | The speaker or kind of the text. |
createdAt | string | An ISO 8601 timestamp. |
tags | string[] | Free-form labels. |
hash | string | Lowercase hex SHA-256 of the canonical payload, defined below. |
A VaultFile is JSON with the members format ("keepsake/v1"), createdAt, cells (a count, not the array), kdf (name, iterations, base64 salt), cipher (name, base64 iv), base64 ciphertext that includes the 16-byte GCM tag, and merkle. The plaintext inside ciphertext is the canonical JSON of the full cells array.
The canonical payload and the cell hash
The canonical payload is the six non-hash members of a cell. The cell hash is SHA-256(canonicalJson(payload)), rendered as lowercase hex. Because the serializer is deterministic, two implementations that hold the same cell compute the same hash.
Canonical JSON
src/canonical.ts is the exact encoding that is hashed and encrypted, so every implementation must produce identical bytes. Its rules:
- Objects. Keys are emitted in ascending order, compared by Unicode code point. Members whose value is
undefinedare omitted. There is no insignificant whitespace:{and}hug the members, and,and:are unadorned. - Arrays. Element order is preserved, and arrays may be empty. An
undefinedelement becomesnull. - Strings. Standard JSON escapes; control characters below
0x20use the shortest form. Non-ASCII characters are emitted literally as UTF-8, not\u-escaped. - Numbers. Emitted as JSON numbers with no leading zero and no trailing fraction zero. Only integers appear in the fields this format defines.
- Booleans and null. Emitted as
true,false, andnull. - Edge cases.
nullandundefinedserialize tonull; a non-finite number serializes tonull; abigintserializes to its decimal string.
The Merkle root
The root commits to the set of cell hashes and does not depend on cell order:
- Take every cell's
hashas a lowercase hex string. - Sort the list in ascending lexicographic order; for lowercase hex this equals byte order.
- While more than one value remains, walk the list in adjacent pairs and replace each pair
(left, right)withsha256(left + right), where+concatenates the two hex strings and the result is hashed as UTF-8. An odd final value is carried to the next level unchanged, never duplicated. - The single remaining value is the root. An empty cell set has root
sha256(""), which ise3b0c44298fc1c14.9afbf4c8996fb924 27ae41e4649b934c a495991b7852b855
The crypto boundary
Cryptography is confined to two modules, both using only the WebCrypto API so the same code runs in a browser tab and in Bun:
src/crypto.tscallscrypto.subtle.digest("SHA-256", …)for hashing andcrypto.subtle.importKeypluscrypto.subtle.deriveKeyto turn a passphrase and salt into an AES-GCM key with PBKDF2-SHA256.src/vault.tsfills the salt and IV withcrypto.getRandomValues(16 and 12 bytes) and callscrypto.subtle.encryptandcrypto.subtle.decryptwith AES-GCM.src/cell.tsgenerates ids withcrypto.randomUUID().
This is the reason for the hard rule that src/ must stay browser-safe: no node: imports and no Node-only globals, because the core is bundled for the browser. The only files that break that rule are the two entry points that need a filesystem, src/cli.ts (node:fs, node:path, node:url) and src/mcp.ts (node:fs). The vault is never opened by a remote key or a hosted service; decryption happens in process, with a passphrase read from the environment by the CLI or entered in the browser.
One core, three surfaces
src/index.ts is the shared barrel. Each surface imports it, so there is a single implementation of the format:
assets/app.js bundle, which exposes the library as the global Keepsake. Import, search, seal, and open all run in the tab.src/cli.ts imports ./index.ts and ./mcp.ts, parses arguments, and reads and writes vault and pack files. It is the reference for scripts and CI.src/mcp.ts imports the core from ./index.ts and serves recall, context, stats, verify, remember, and forget over stdio.Module dependencies
Arrows point from a module to what it imports. index.ts re-exports the leaves and is the only entry point the surfaces share.
types.ts (shared types, no imports)
|
+-- canonical.ts key-sorted JSON
+-- bytes.ts base64 / utf8 / hex
| +-- crypto.ts SHA-256 + PBKDF2
+-- bm25.ts BM25 ranking
+-- semantic.ts cosine (browser)
+-- stats.ts counts + bytes
|
+-- cell.ts (canonical + crypto)
| +-- merkle.ts
| +-- vault.ts
| +-- vault-ops.ts
| +-- vault-ops.ts
+-- interchange.ts (canonical/cell/merkle)
+-- conformance.ts (bytes/canonical/vault)
+-- import/*.ts -> index.ts barrel
|
+------------+------------+
cli.ts (+ node:*) mcp.ts (+ node:fs)
|
browser app <-- build-site.mjs
(esbuild -> app.js)
Build pipeline
scripts/build-site.mjs bundles the core for the browser. It imports node:fs/promises, node:fs, node:path, node:url, and esbuild, then builds with these options:
| Option | Value |
|---|---|
| entry point | src/index.ts |
| output | site/assets/app.js |
bundle | true |
format | iife |
platform | browser |
target | es2022 |
minify | true |
globalName | Keepsake |
The bundle is a single self-contained IIFE; the page loads it with a local <script> and calls into the Keepsake global. If src/index.ts is missing the script prints a skip message and returns. The rule that src/ has no node: imports is what makes this bundle possible: esbuild can resolve the whole graph for the browser with no Node shims.
Testing
Tests live in tests/ and run with bun test. The suite is currently 158 tests across 17 files, with 319 expect() calls.
$ bun test158 pass 0 fail319 expect() callsRan 158 tests across 17 files.
The 17 files cover:
canonical.test.ts, cell.test.ts, merkle.test.ts — key ordering, the canonical payload and hash, odd and even pairing, and the empty root.bytes.test.ts and crypto.test.ts — base64 and UTF-8 round-trips, SHA-256 vectors, and PBKDF2 derivation.vault.test.ts and vault-ops.test.ts — seal and open, wrong-passphrase and tamper rejection, merge, forget, rotate, and diff.bm25.test.ts, context.test.ts, and semantic.test.ts — ranking, the token budget, cosine, and hybrid blending.interchange.test.ts and conformance.test.ts — pack round-trips, tamper rejection, and the published vectors.import.test.ts, import-extra.test.ts, cli.test.ts, and mcp.test.ts — the parsers, argument handling, exit codes, and the MCP tools.bun run typecheck runs tsc --noEmit, and bun run check:site validates every site/*.html page: exactly one <h1> and one <main>, a skip link that resolves, and only classes defined in lens.css or theme.css.