Native compression engine

For the opt-in Claude transport, cache behavior, output shaping, and limitations, see the experimental local proxy guide.

src/compress/index.ts exports a synchronous, local TypeScript engine. It uses gpt-tokenizer's bundled o200k_base BPE vocabulary; no network, Python, or remote compression service is used. Counts measure text tokens, including the expansion marker. They exclude chat framing and are approximate for Claude and other tokenizers, not billed usage. TOKENIZER_LABEL reports approx (o200k).

import { compress, OriginalStore } from "./src/compress/index.ts";

const store = new OriginalStore();
const result = compress(toolOutput, {
  source: "ci",
  minTokens: 300,
  store,
  ttlMs: 24 * 60 * 60 * 1000,
});
// result.text, kind, compressed, id?, originalTokens, compressedTokens, reason?
const original = result.id ? store.get(result.id) : result.text;

Routing and preservation#

• JSON and JSON lines: object arrays retain their first/last two items, every error-like item, and numeric outliers beyond the per-field 1.5×IQR fences. Retained rows remain complete, with original indices; summaries include total count, omitted count, and the union of field names/types. Error detection includes HTTP failure status, false success flags, diagnostic strings, and nested errors.
• String arrays: deduplicate and keep first/last three unique values and every diagnostic value. Numeric arrays: count, min/max, compensated mean, and linearly interpolated p50/p90/p99. Arrays smaller than 12 remain intact, with nested containers inspected up to depth eight. Mixed arrays remain intact.
• Logs/test/build output: collapse adjacent exact or near repeats with counts and first/last samples, repeating two-to-six-line cycles, complete successful TAP records, and passing test-line runs. Successful test titles are scenario names, not emitted diagnostics. Grouping normalizes numeric values, UUIDs, hex identifiers, and paths. Every recognized warning, error, failing test, exception, and stack frame survives verbatim with two neighboring lines and indented diagnostic payloads. The first/last five lines and all distinct lines survive. A summary reports retained/total lines and diagnostic count.
• Unified diffs: include per-file and total added/deleted line counts. Normal file hunks survive. Lockfiles, generated/minified/vendor/build output, binary patches, and files over 400 lines or 40,000 characters become named summaries. Expand to review omitted hunks. Header-like content inside a hunk is counted as content.
• Source code and prose pass through. Fenced code is conservatively treated as source, even when it contains a log. Callers can explicitly supply kind when the source is already known; this bypasses detection.

JSON containing non-finite or unsafe integer values passes through, avoiding rounding large numeric identifiers in retained rows.

All recognition is heuristic. When details are needed for an edit, diagnosis, or review, expand first. Callers must set protect: true for system prompts and code being worked on, including diffs of that code. The engine also protects sources named system or system-prompt. No engine can infer every edit target or prompt role from plain text alone.

Safety gates and expansion#

The default minimum is 300 tokens. Protected text, existing compressed markers (even nested in another message), source code, and prose pass through. Results also pass through if the complete output is not smaller or if any parse, count, or storage operation fails. store: false disables compression: an output must always be reversible. reason is short, protected, already, code, too-large, or no-gain for an unchanged result; no-gain also covers fail-open errors.

Every returned compressed result starts with:

[nid compressed log 3000→250 tokens · full text: expand("c_1234abcd")]

The original lives at $NID_HOME/compress/<id>.txt (default ~/.nid/compress), with a metadata JSON file recording source, kind, creation, and expiry. IDs are random, exclusive writes avoid collisions, and files request owner-only permissions where supported. This stores the same potentially sensitive text as the input; it is local plaintext, not encryption.

OriginalStore({ dir?, ttlMs?, now? }) allows isolated stores and clocks. put(text, { source?, kind?, ttlMs? }) returns an ID; get(id) returns the exact original or null for invalid/missing/expired data. The default TTL is 24 hours. Reads never extend TTL. prune() deletes expired records and returns their count. Foreground writes inspect at most 64 directory entries once per minute per directory (shared across store instances); this avoids rescanning every original for every compressed result. This opportunistic cleanup may leave other expired files until they are read or an explicit full prune() runs. Expiry is always enforced immediately on reads. compressStoreDir() exposes the default directory. Daemon/MCP wiring supplies the user-facing expand tool.

Synchronous work limits#

Compression runs on the daemon's event loop, so it fails open before tokenization when input exceeds 1 MiB of UTF-8, or has more than 512 consecutive non-whitespace code units. The latter catches long words/blobs that make BPE merging expensive. Minified JSON and long URLs can also hit this conservative guard. Returned text is byte-identical, reason is too-large, and tokensMeasured: false distinguishes its zero token counters from an actual measurement. No savings are recorded for a skipped input. Board rendering and uncompressed CI-log counting use the same preflight; all recorded token-saving benchmark fixtures remain eligible. When a board exceeds a work limit, its fallback includes every requested note; the performance guard does not reinstate the old 40-note history cutoff.

Proxy request bodies over 2 MiB of UTF-8 bypass rewriting and system shaping entirely. Smaller requests attempt at most 32 new result text blocks and 512 KiB of UTF-8 of new result text; remaining content is forwarded unchanged. Previously memoized summaries may still replay within the request size limit and do not consume the new-compression budget. Crossing that limit can cause a cache miss because the original request is forwarded unchanged. The transport's separate 8 MiB buffering limit remains in place.

expand always returns the full original. For legacy originals outside the new tokenizer guard, expansion accounting uses ceil(code units / 4) instead of BPE, so the daemon stays responsive. This is an estimate, not a token measurement, and can over- or under-count net savings. Newly stored compressed originals are checked against the guard before storage.

The engine reuses its body token count and normalizes each log line once. It still measures the final marker-bearing output with the real tokenizer; it does not estimate a compression saving to avoid counting work.

Run the isolated stress harness from the repository root:

node --import tsx packages/nid/test/bench/compression-performance.ts

Measured on the development Windows machine with Node 22.13.1, 2026-10-09 UTC; module loading and fixture creation are outside each timed interval:

These are latency observations, not hard wall-clock guarantees for arbitrary hardware or a blocked filesystem. The regression test runs each case in a separate process, allows 5 seconds of measured work for slower CI, and kills a child after 15 seconds. The standalone harness uses a 10-second child timeout.

nid's own traffic#

With compression.enabled on (the default), nid uses the engine on what it sends between agents. Each source has its own switch, resolved per session, and every saving is recorded under that source (/compression, nid compression, ls, status, the dashboard).

What is never compressed: the goal, nid's system prompt and protocol, the human's messages, source code and prose, text under minTokens (default 300; the board is the exception: its compact and incremental views pay off even on a small board, so they ignore minTokens), and anything that already contains a compressed block. When a compressed view would not be smaller, the original is sent. Agents get the MCP tool expand(id); an unknown or expired id (originals are kept ttlHours, default 24) returns a clear error.

With compression.enabled: false (or the source's switch off) the traffic is byte-for-byte what nid sent before compression existed.

Correctness audit#

The rule is that a compressed view must never make an agent fail where the full view would not. An adversarial audit (test/compress-audit.test.ts, test/e2e-compress-audit.test.ts) tried to break each crusher and source. It found the problems below, and each one is now fixed:

Checked and found safe: the human's messages and the goal are never compressed. Fenced code in a message passes through untouched. On the board, task reassignment, goal edits and released locks always show (locks and the team are listed in full on every read, and a changed owner or status marks a task "updated"). Notes cannot be edited. Handoff rotation and nid's own compaction reset the board view. The proxy never compresses file reads or edits, source-viewing commands or unknown tools.

Known limit, by design: the JSON crusher samples arrays of objects. It always keeps error-like items and outliers, the first and last two, the schema and the count of omitted items, plus an expand id. An unremarkable item in the middle is only reachable through expand. That is why JSON is compressed only in traffic that is output (logs, test reports, messages) and never in file contents or gh api data.

Verification#

test/compress-engine.test.ts covers routing, BPE counts, object/error/outlier retention, numeric summaries, JSONL, nested depth limits, log diagnostic/context preservation over varied positions, diff hunks/stats, all safety gates, storage failures, expansion roundtrips, and full-output token counts. test/compress-store.test.ts covers isolated local storage, provenance, invalid IDs, expiry, pruning, and persistence across store instances.

These are structural preservation tests, not evidence of task-outcome equivalence. On/off team measurements and recorded-fixture benchmarks are separate proof steps.

Recorded-fixture benchmark#

Run pnpm --filter nid bench:compress. The command exits nonzero if a preservation check fails, and its checks also run in the regular test suite. Fixtures and capture provenance are in undefined. The harness uses an isolated temporary original store and cleans it up afterward.

One measured run (2026-10-08):

Counts include the expansion marker. Random expansion IDs can change the measured output by a few tokens. The short build log has no net savings after marker overhead; the prose board remains byte-identical to the engine. No savings are attributed to these cases.

Board savings come from the board_read views (nid's own traffic), not the engine. The benchmark parses the recorded board back into board data and renders it as agents receive it (test/bench/board.ts), checking that every task, open-task detail, note, lock and the goal are still present:

Agents call board_read many times per task, so the incremental row is the common case. Configured transport behavior, billed tokens, cost, and agent task success require separate measurements.

The benchmark checks diagnostic line multiplicity and neighboring context, complete failing TAP records, JSON first/last and error objects, exact source hunks, complete per-file/total diffstats, board pass-through, and exact expansion. It rejects misrouting the recorded test/build logs as source code. Tests also deliberately damage outputs to verify that the checker rejects lost evidence.

Measuring a team run#

The fixture benchmark shows what the engine does to text; whether a team uses fewer tokens and still finishes its work needs a real run. scripts/ab/ holds an A/B kit for this repository: a nid.ab.yaml with two groups that differ only in compression: (ab-off, ab-on), a script that plants a small bug with a long failing test log in a scratch clone, and a README with the steps.

After both runs, compare them:

nid compression compare ab-off ab-on          # table, with the change from ab-off to ab-on
nid compression compare ab-off ab-on --json   # raw numbers (compression.compare RPC)

Per group it reports the compression switches it ran with, agents, turns, input tokens (with cache reads, cache writes and uncached input), output tokens, cost (billed, and estimated on a subscription), tokens compression saved (net of expand calls, with the number of expand calls and the tokens they returned) and by source, tasks done/blocked/open, review tasks and requested changes, automatic reassignments, wall time (group start to the lead's DONE note, else the last event) and whether the lead posted DONE. Any two or more groups can be compared, not only the A/B pair. Treat a single pair of runs as an anecdote: check task outcomes first and repeat the pair before trusting a token difference.

Measured: three arms on this repository (2026-10-09)#

One run per arm of scripts/ab/nid.ab.yaml (claude lead, claude worker, codex worker, review on, subscription billing), one after the other in an isolated NID_HOME daemon built from the compress branch. ab-off started from a slightly older integration commit than ab-on and ab-proxy; the three tasks were identical. Every arm's integration branch was re-checked independently afterwards: test file untouched, 281/281 percentile cases, typecheck, both --json flags with tests.

What this shows:

• No correctness cost. Every arm finished all tasks, with no requested changes, no blocked or reassigned tasks, and no case where an agent misread a compressed view. One agent expanded one block (555 tokens).
• nid's own traffic is a small share of a short run. Board reads and messages saved 4–9k tokens per arm, under 0.3% of input. The differences in turns, total tokens, cost and time between arms are run-to-run variation, not compression: ab-proxy cost more because its agents took more turns, while its proxy changed nothing. A single run per arm cannot resolve differences this small. Savings from board reads grow with the length of a run (incremental re-reads save about 85% each, see the board benchmark above).
• The proxy had nothing to compress here. All of Claude's large tool results in that arm were file Reads (never compressed), Grep hits and board_read output: the agents filtered their own command output (Select-String, tail). The largest tool results (up to 30k characters) came from the codex worker, which the proxy does not cover. The proxy pays off on unfiltered output: the live smoke test (docs/compression-proxy.md) cut each 8,040-token result to 188 with the cache intact.