CodeEraser applies deterministic computation to the non-deterministic output of language models.
A model writing into a long-lived repository stacks rather than
edits: the same function twice, a fact restated in a third file, an update appended. Instead of
auditing that drift with a second model, every verdict below is arithmetic over facts from the tree
(token fingerprints, tree edit distances, graph in-degrees, git-window counts) against thresholds
written down: no sampling temperature, no floating point in the judgment layer, no
model in the loop. The same tree yields the same bytes on any machine at any hour, so a disagreement
is settled by re-running the number and reading the file:line it points at.
Every heading links its full derivation in docs/reference/methodology.md, each constant traced to its implementing line.
Three counts ship across this site. nineteen booklets, one per judgment whose math earns a derivation: families 01–15 here, 16 on from the analysis track on a page of their own. sixteen judgment capabilities on the wire (the stack diagram's count, beside the handshake hello, the definition package tables/1, the report documents document/1 and eight stage capabilities the families and the write-time probe call: resolve/1, candidates/1, rank/1, docpairs/1, moves/1, bags/1, relocate/1, cover/1; the join and the split advisory ride inside verdict and structure requests, and the FPR tier ladder is a discipline, not a request). twenty-one read-only MCP tools follow the report surfaces, not the wire: churn and graph --sites, erase (its dry-run plan only), doctor (machine state), erase_log (the applied-erase trail), update_check (this build against the latest release), similar_units (ce similar's advisory document over similar/1), query and rules over query/1 (its own page), and flow, merge_suggestions and architecture over flow/1, merge/1 and arch/1.
Finds exact and parameterized duplicate token runs: renamed variables and changed constants are clones, changed syntax is not.
t = window + kgram - 1 = 26 + 25 - 1 = 50 tokens
Any common run of at least t normalized tokens holds
t - k + 1 = w consecutive k-grams, one complete window, whose minimum depends only on
its contents, so both copies select it: at least one shared fingerprint, always, the Schleimer et al.
SIGMOD'03 no-miss bound held as a correctness contract. The rolling hash is Rabin-Karp,
h = (h - t[i-k]*top)*BASE + t[i], over FNV-1a leaf hashes. Import and load statements give no
token, like comments; so do a package clause, a Haskell module header and a Python docstring that holds no code. Which verified runs are reported is the core's coverage rule: per file pair, diverse
runs first, longest first, each kept only with t tokens of its own on some side.
Judges units whose ASTs are almost the same but whose token streams are not: reshaped and rewritten copies.
TSED(a, b) = (max(n1, n2) - ted(a, b)) / max(n1, n2) clone ⇔ TSED >= 0.85
cloneDecidesWith (num, den) t n1 n2 = (mx - t) * den >= num * mx where mx = max n1 n2
ted is Zhang-Shasha with unit costs (delete = insert = 1,
relabel = 0 on matching kind codes, else 1), always integral, so the comparison is an exact
cross-multiplication decidable both ways: at max = 100, ted 15 is a clone
and ted 16 is not. Three admissible bounds come first: two O(1) tests, q · tsedDen < tsedNum · max for
q ∈ {min(n1,n2), I} with I = Σ_label min(c1, c2), then the edit distance between the
trees' label sequences; a pair that passes all three has its distance computed only up to the clone ceiling. They are admissible because they use the same 85/100.
Since 1.8.0 verified pairs ride ce check's clone axis and
ce join's similarity leg as sim kind 1, each verdict cached under the two trees' content
keys until the proto or a knob changes.
Decides whether two documentation blocks (markdown paragraphs, HTML text, plain-text paragraphs, comment blocks, docstrings) are near-duplicates.
dupDecidesWith num den inter union = inter * den >= num * union dupVerdictWith (num, den, vfloor) inter union run = dupDecidesWith num den inter union || run >= vfloor
Bound to (80, 100, 50): Jaccard ≥ 0.80 decided in integers,
or a verbatim run of at least 50 words. The core computes inter and
union itself from the ascending deduped shingle sets, so raw counts cross the wire, never
a ratio. MinHash/LSH is an RNG-free coarse filter (the permutation index is the salt); a run of
R shingles spans R + k - 1 words. A docstring is a Python body's first statement past any comment, when that statement is a bare string; a ~ line under a title, at least as long as the title, is a skeleton row, not a code fence.
Judges the tree rather than the file: directory geometry, naming distributions, reference locality, documentation coverage, doc staleness, redundancy, modularity.
tsallis2 cs = 1 - Σ (c/N)^2 -- and = 0 when N == 0
tsallis2Norm cs = tsallis2 cs / (1 - 1/n) -- n = nonzero bins, n > 1
chi2 pairs = Σ_{r > 0} (p - q)^2 / q -- p = o/Σo, q = r/Σr
perMille r = floor (r * 1000)
charge_i = floor(scale * v_i / (v_i + N)) -- v = flagged dirs, N = dir total raw = Σ_axes (charge_i * violCost) score = max 0 (scale - raw `div` (structViolCostNeutral * judgedAxisCount))
Shannon entropy and KL divergence need logarithms, not exactly decidable,
so the family uses their rational-closed relatives, Tsallis-2 diversity and the χ² f-divergence, over Data.Ratio. Observed mass on a zero-reference bin makes χ²
Nothing, a refusal that names those directories. An axis's mass v is its
flagged-directory count, charged into ‰ of scale and folded through
structViolCostNeutral = 10; judgedAxisCount is 5 to 8, by which optional
fact tables rode the wire.
Reads the share of the tree's lines that the seven axes' findings
name as one 0–1000 score (six count by default, the unreferenced axis only when [score.weights] deadcode names it), and gates it against a banked baseline that may only tighten.
score = max 0 (floor (scoreScale * (1 - k * E / L))) -- k = violCost / violCostNeutral L = the lines of every file a family reads (sized files, the scan-only arm included; prose-only files) E_f = min(n, max(T, n - S, churn)) -- one file of n lines E = sum_j (r_j - r_j+1) * E(axes with w_i/max w >= r_j) -- weights by layers
m = gmedian(x) -- log-space median: √(x_lo·x_hi) when the count is even r = gmedian( max(x/m, m/x) ) -- >= 1 by construction S = clamp(floor(m * r^k), [softMin, min softMax H])
tolerated(c) = max (c * tolNum `div` tolDen) (c + tolAbs) -- tolAbsCoc on a complexity row
added = current \ baseline -- non-empty => fail
removed = baseline \ current -- informational; drives the shrink
A file's T is the union of the lines its findings name — duplicate
member spans, every line of an unreferenced file (that axis weighs 0 unless
[score.weights] deadcode names it, so by default such a file is an ordinary file to the score:
its lines count in L and none is charged), reference lines inside a cycle, and for a function over
its cognitive ceiling C at complexity c its lines times 1 − C/c —
n − S its lines past the soft line, churn a rewrite-heavy file's excess; a line
two findings name counts once. Exact Rational, one floor. The soft line S is the repository's own frozen
LOC statistic, S = clamp(median + k·MAD, …) rewritten multiplicatively so no logarithm is
taken, derived at establish and frozen into the baseline. The fail bit is a disjunction of
seven named conditions, ratchet_over,
discrete_added, floor, dedup_budget,
knobs_digest, rows_dropped, dead_added, printed after FAIL in that order.
Since proto 3.1.0 a continuous row may carry its path class, the
1-based index of the first matching [[rules.class]] (0 for none), and is
measured against that class's soft line and CoC ceiling, the global line as fallback.
Beside the ceilings' codes 0 / 1 / 2, classKnobs adds 3
(proto 5.1.0), the class's ratchet allowance in lines, and 4 (proto 6.4.0), its
cognitive-complexity allowance; declared, either replaces both global legs, so
0 means any growth is over. The baseline stays three columns; names and globs never
cross the wire.
Answers "which files does nothing live reach?" over a reference graph whose node identity is the row index, with no text on the wire.
arcs = { (s,d) | [s,d,kind,rung] ∈ edges, rung <= minRung, kind ∉ inert }
reach = ⋃ { reachable(G, s) | s ∈ entries(entryMask, flags) }
public = testBit flags 0 referenced = indeg >= 1 over kept arcs judged = i ∉ reach code = 1 + public + 2*referenced -- the lookup table is the authority
Four codes keep an
exported-but-unreferenced API from collapsing into a private verdict: 1 unref_private,
2 unref_public, 3 unreach_private, 4 unreach_public. A site walks
its language's rungs in order and the first rung yielding exactly one in-scope candidate
wins; more is Unresolved, and External is a correct answer, not a miss.
Cycles are reported, never judged; a cyclic island with no entry seed is in the dead set by reachability alone.
The files no entry reaches are the dead set, and a face calls each one
unreferenced: no reference was found, which is not the claim that it is unused. A dead file
g supports f when a kept path leads from g to f;
f is a root when every dead file supporting it is supported by it, otherwise a
dependent under the roots that support it. The baseline's deadPoint gives every
unreferenced file, root or dependent, its state — standing (unreferenced then),
orphaned (there and referenced then), born (not there then); the point is keyed
by path, so a renamed file reads born. By default every unreferenced file is listed and fails
nothing. [deadcode] block, out of "orphaned" and "born"
("standing" is refused: the inventory never blocks), names a set M of unreferenced files;
the files that block are M's roots (the same root law, with M in place of the dead set), so
ce deadcode --check and ce check's dead_added fail exactly when M
is not empty, a new file under a standing root included; a declared block with no point refuses with
exit 2. Two blind audits back the default: 392 of 424 existing unreferenced files
were in use, and new ones were too few to judge (21 orphaned, 90 born, under the 100 a class needs).
inert = {3 asset, 5 unused ref-def}, excluded by the core:
an image link renders bytes and an unused reference definition renders nothing. Since proto 2.32.0
each unreferenced row carries a confidence judged from the per-language site ledger (0 unvouched, 1 vacuous,
2 vouched), the erase family's trust boundary. Import-edge precision: 38/40 = 0.95
over five pinned corpora against a ≥ 0.90 gate, on a sample frozen before any resolver existed. The six
languages plan v2.30 added each passed a frozen, blind-audited exam: C 73/73, C++ 15/15, Java 36/36,
Lua 86/86, R 18/18, HTML 75/75 (recall 73/74, 15/17, 36/36, 86/86, 18/18, 75/76), and a per-language
false-positive ledger (six corpora, 400 commits each) shows 0 false intercepts; see
EVAL-SET-LANGS and
FPR-REPLAY.
Combines similarity × graph position × churn into one of four candidate codes, then deliberately declines to act on it.
(1, sev 2, [1,2,3,4], []) -- merge_candidate: sim + graph + both referenced + distinct SCCs (2, sev 3, [1,2,5], [6]) -- delete_candidate: sim + graph + dead flank, RG10 guard clear (3, sev 1, [1,2,7,8], []) -- churn_hotspot: sim + graph + cochange + rewrite
rewriteHot = total > 0 && (rewrote_a + rewrote_b) * rewriteDen >= total * rewriteNum cochangeHot = cochange >= cochangeFloor legsMask = legSim .|. (if graphBoth then legGraph else 0) .|. legChurn
Priority is data, not guard order: the first row whose required bits hold and forbidden bits stay clear wins, else code 0, so the property battery can falsify the order by rotating the table. Every gating row requires the graph bit: mask 5 (graph leg absent) carries only code 0 rather than pretending indegree 0.
Since proto 2.33.0 each candidate row carries a leg-agreement
confidence, the severity face ships once as joinSeverity, and
ce join judges over the same verdict/1 road as the check gate. No join code enters the
fail bit: ce join exits 0 unless the run fails, and a missing or refusing core is a run
failure (exit 2).
Prices "is this long file worth splitting, and where?" as a number, an advisory outside the structure score and the fail bit.
benefitMilli(u) = max 0 (floor (1000 * (p(total) - p(end_u) - p(total - end_u))))
costMilli(u) = crossRefs(u) * roiRefMilli
+ cutClones(end_u) * roiCloneMilli
+ crossChurn(u) * roiChurnMilli
+ roiPhiMilli
viable ⇔ b >= c -- ROI >= 1, evaluated without division
Benefit is the graded-zone penalty a split returns, on the advisory's own
convex curve (CE.Structure.Zone, the score's size charge until protocol 10.0.0); p is convex with
p(0) = 0, hence superadditive, so the bracket is non-negative. The best seam is the exact
rational argmax over ROI, compared by cross-multiplied b % c; n top-level
units give n - 1 seams. Long-and-cohesive gets an exemption with numbers
attached, long-and-splittable a cut line.
Reduces every supervised edit to four integer counts per file pair (matched, novel, moved, deleted), so "stacked, not applied" becomes a measurement.
siteOpens s n = n * movedCost + s < n * plainCost
destFloor = least n with siteOpens siteCostCross n = 2
accepted = isStart && distinctEvidence >= destFloor && anchored
anchored = any (\(_,_,w) -> w >= anchorFloor) evidence
The cross-file evidence floor is derived: siteCostCross = 2
makes a single cross line a tie (1*1 + 2 = 3 = 1*3), ties do not open, so
destFloor is 2, and that tie is the coincidence rejection. Only
anchorFloor = 19 is decided: in the dual-corpus shadow ablation the invented station's
widest anchor measured 16 alnum characters and the thinnest real anchor 19, so 19 tops the window
that kills every measured coincidence and keeps every real site. The L2 delta moves one way:
plain lines may become moved, never the reverse.
Since proto 7.1.0 paired declRem / declAdd
tables of declaration keys that vanish on one side and appear on exactly one other open an edge when
the spans share declFloor = 1 leftover hash (the key pays
declCredit = siteCostCross). They append relocations with lines = 0; no line
class, block, suspicion or score moves.
Is the check score going up, flat or down across history, and by how much per day?
x_i = ts_i % 86400 -- seconds to days, exact ratio
y_i = (score_i * 1000000) % scale_i
slope = median{ (y_j - y_i) / (x_j - x_i) : x_i ≠ x_j } -- Theil-Sen
slope < -band → 2 (degrading) slope > band → 0 (improving) otherwise → 1 (flat) where band = floorMicro
% is Data.Ratio's exact-ratio constructor, not
modulo; y puts every commit on a fixed 10⁶ full-scale grid, so rows under different
scoreScale values compare. The slope is the median of pairwise slopes (trend/2): one wild
point drags a least-squares mean anywhere and cannot move the median past its neighbors.
The judged view sorts by timestamp and keeps the tsWindow most recent
points, so rebased commits are legal. The steepest single-step fall ships as cliff, the
longest strictly-falling run as declineRun, each by request index; hashes never cross.
Below minPoints, or with no timestamp-distinct pair, the slope is Nothing,
never a fabricated flat; at the default floor 0, degrading is reported and cannot fail.
A deterministic gate over a non-deterministic writer: on
PreToolUse for Write|Edit the guard answers each pending write with exact
arithmetic, and a rule class enforces only once it has paid for the right.
TIERS = ["observe", "warn", "ask", "deny"] PROMOTED_DEFAULT = "deny"
cap = lines_for(file).file_lines_fail // the file's class table; 750 for class 0 breach ⇔ cap != 0 && after > cap && after > before permille = (lines - S) * 1000 / (H - S) 0 ..= 249 → observe 250 ..= 750 → warn 751 .. → ask
| tier | permissionDecision | effect |
|---|---|---|
| observe | none, returns before printing | feed line only, no injected text |
| warn | allow | edit proceeds, reason surfaces as a visible warning |
| ask | ask | the user is prompted |
| deny | deny | write is refused, reason points at the existing file:line |
Determinism comes from replaying the write, once per hook call: the
exact post-write text and line count, and the line count before it. A file already past its line is
refused only a write that grows it, so splitting it can go one step at a time; a write that keeps or
shrinks it is recorded as carried. A call that would fail on its own (missing file,
no match, an ambiguous non-replace_all match) leaves the rules silent and the probe line
names why. An unrecognized [guard] mode resolves to
observe (ce.toml ERROR: …), never passed through.
A class earns deny by its record. M3:
≤ 1 mis-block in 500 real normal edits, N=1 demonstrations disallowed.
M4: FPR ≤ 1% over 500 real normal edits on a set pre-registered before
implementation, ≥ 200 edit samples, ≥ 50% from real agent transcripts, sampled only from observe-mode
and pre-guard sessions so the guard cannot certify itself. Two classes default to deny: T1/T2 exact duplicate
write, which earned it with the record below, and a write that grows a file past its hard line (750 by default, or the file's
[[rules.class]] line), deny since 1.0 without a record of its own until release 1.10.0 added one (this repository: 5 refusals in 9,707 write events, every one a crossing, 0.0515 %; the other seven corpora: 0 % to 28.1250 % counting every refusal), so the hook denies exactly where the CI wall fails.
Every other rule stays at observe for want of its own record.
Since CodeEraser 1.2.0 the duplicate-write rule charges novel
duplication only: the K-round replay of 2,761 real edit events (this repository's history plus the
pinned requests tail) separated genuine landings from re-fires on already-budgeted files
and split/fold mid-states, so matches the replaced content already carried (Write: the
on-disk file; Edit: old_string) are subtracted. Since 1.10.0 the subtraction counts copies:
the replaced content carries as many copies as it held, so a second copy written beside a carried
one denies. A rewrite carrying its debt is silent; an
introduction, or a split to a new file, denies and teaches the ordering that passes (trim the source
first). Both framings: FPR-REPLAY.md.
Replaying linear git history as an edit stream at the shipped knobs
t = 50 and min_distinct = 7: 630 events, 35 blocks, 0 false after
arbitration → 0.00 per 500 (M3), the 35 blocks remediated as the clone-block budget stepped
251 → 211 → 209 → 205 → 202. The standing instrument (cli/tests/it/fpr_replay.rs,
2026-11-14): requests 365 events / 0 intercepts; 5932 self-repo events with 171 landed introductions
and 105 write-first mid-states.
The graded zone reads S and H off the hard
budget's per-file table. [guard] zone_tiers is off by default: its
pre-registered rule armed only if every corpus read ≤ 1 %, and the first run read self
0.54 % but requests 2.46 % (variant B); the 2026-11 re-freeze, with the hook's own population and
tiers firing only on growth, reads self 1.90 % and requests 1.12 %, held by
cli/tests/it/fpr_zone_gate.rs. Warns are
rate-limited to once per (rule, file, session) and clipped at a token budget; a deny is not.
Judges erase-plan rows for three classes
(dead_file, verbatim_doc, t1_twin), erases only
byte-identical doc segments, and refuses every other row with a named reason code.
class = 0 (retired 4.0.0, refused by name) | 1 verbatim_doc | 2 t1_twin
3 dead_file (confidence road, 2.32.0)
reason = 0 eraseable | 1 language_unresolved | 2 not_full_segment
3 bytes_differ | 4 copy_not_dead | 5 unit_not_covered
6 public_surface (6.1.0) | 7 reachable_by_name (9.1.0)
8 linked_section (10.0.0)
Rust assembles integer facts from the three source families; Haskell applies a fixed first-failure predicate. Since proto 2.32.0 class 3 trusts the graph family's per-row confidence; since proto 6.1.0 a public dead verdict (2 unref_public, 4 unreach_public) is refused as public_surface before confidence is weighed, so an exported API is never eraseable; only then does an unvouched confidence refuse the row. Since 9.1.0 the two whole-file classes have no eraseable outcome: a row past every other bar is refused as reachable_by_name, because the graph says no reference site lands on the file, and a script run by name or a file a tool finds by convention leaves no such site. Since 10.0.0 a duplicated doc segment whose section a link's fragment lands on is refused as linked_section; of two copies the one fewer links reach is the candidate.
Class 0, the local-count road, retired at proto 4.0.0; the core refuses it by name. A degraded over-cap reply authorizes nothing, and the safety surface has no knobs.
Lists every judged declaration whose name no other file spells, with a visibility code: a negative instrument that never judges.
veto = another file spells it | fold (Rust, ≥2 segments ∧ ≥7 chars)
| the file's own exception regions spell it
row = [node, vis, conv] (integers only — a name never rides the wire)
emit ⇔ vis ⊇ {exported, scope-exported} ∧ conv has no exempt bit (0..10)
code = 1 private > 2 restricted > 3 reexported > 0 public
U is its own walk: hidden files enter, .git/.ce
are cut by name, an undeclared nested repository is cut whole, and a path the root's
.gitmodules declares stays as foreign (it can spell a name, never measured). Only
.gitignore and .ceignore are honoured, so one commit yields one U, and every
walk parameter is a MENTION_REV input; the store keeps 64-bit hashes, never text. Haskell reads only the visibility word and the mounts row
(private mod mounts, façade re-export, package-private). Both tables ride
graph.request together or not at all; without them the reply is the ten-key one byte for
byte and the dead set is unchanged. Past unmentionedCap the core drops the table and says so. Never a gate, never in
ce erase.
The families above read a tree; the fourteenth reads a change
(one edit, session or commit) at the PreToolUse hook, the Stop audit and
ce precommit / ce commitmsg.
Catches the change that erases X from the code and writes it back as an absence:
a heading (no X), an identifier without_x, a sentence
X is no longer needed.
R = ⋃ names(before) \ ⋃ alive(after) (structural positions only; keys, never text)
row = [kind, marks, names] kind ∈ {0 bracketed, 1 bare, 2 prose}
site ⇔ names ≥ 1 ∧ (kind ≠ 2 ∨ marks ≥ 1) (the conjunction, read per sentence)
over ⇔ budget declared ∧ |sites| > budget (spoken at [tombstone] tier; ships observe)
A name is what a text declares: an identifier outside comments and
literals, a unit name, a heading, a list lead; an inline code span only mentions, keeping a name alive. Rust sends three
integers per added surface; Haskell seats the sites and judges the budget. A changelog-role document is exempt whole, by path convention,
ledger shape or [tombstone] ledger; a quote run or section that is itself a ledger exempts
only itself. The class ships at observe; promotion would argue from the nine-round FPR
replay over two histories in docs/FPR-TOMBSTONE.md.
Clone families need two units to look alike; the fifteenth finds units that
play the same part with no line in common, as advice at
ce similar, the MCP tool similar_units, the GUI's similar screen
and one line of the Stop audit.
Ranks the units most like one unit (or a text) by integer BM25 over six-channel term bags from the index's facts, and asks the core which play the query's role: an offline "code RAG", no model, no float, advisory only.
bag = six channels N P C D S L (facts the parse already carries; hashes, never words)
score = Σ w · idf · 22·tf·avg / (10·tf·avg + 3·avg + 9·len) (k1 = 6/5, b = 3/4; integer fixed point)
widen = top-m PPMI neighbours at ≤ ½ weight (opt-in view; never evidence)
role ⇔ (N ≥ 1 ∧ C ≥ 1) ∨ (N ≥ 2 ∧ shapeEqual) (judged in Haskell over similar/1; advisory only)
Each T3-universe unit gets one bag (name pieces, shape, callee spellings,
its doc segment, node-kind histogram, literal kinds), every word stemmed and hashed with its
channel letter. Bags persist in the content-hash-gated refresh (bag is its own posting
list; a pair table measured 7–10× the cold index and was not built, so co-occurrence is derived at
query time). The in-memory instruments and the SQL reader feed one trait-written request and send the same
bytes on every unit of five corpora. Rust sends the query bag, the df and postings of the terms that
score and the seats' lengths; Haskell weights, widens, scores and cuts the top K over
rank/1, then orders exact rationals and applies the role conjunction over
similar/1. Floors from two arbitrated
oracle generations: 60 % on the first sample, 40 % on the holdout, which retired every tuning
candidate the first sample favoured. Never a gate, never in
ce erase.
PreToolUse shapes behavior; it is not a security
boundary: Bash: echo >> or sed -i bypasses it, and the backstops are
the Stop audit over git diff (write-tool agnostic) and the CI gate. The hook is
deliberately fail-open: an internal failure allows the edit and lands in the observe feed
instead of reading as "no duplicates". Both are written into the plan: a gate that lies about its reach is
worse than one that states it.
One request line to the core is at most 134,217,728 bytes (maxLineBytes,
the limits.caps.line_bytes the core states), and the reply reader stops at twice that. A family
whose request would pass the line splits it into batches; an item that alone would pass it is refused by
name, never truncated, so a verdict is never judged on part of its input.