# Mnema Derivation Model This document specifies how the Mnema reference node turns **evidence** into **claims**, how every claim carries a verifiable **provenance chain**, how **confidence** is computed and calibrated, how **corrections and refutations** cascade through the derivation graph, and how **explanation records** answer "why does the system believe this?". It is the normative reference for milestone #4 and the interoperability contract for alternative implementations of the derivation layer. Companion documents: * `docs/derivers.md` — reference for the four built-in derivers (routines, places, relationships, preferences). * The milestone #2 wire-format specification — canonical encoding, signing, and the operation-log envelope that everything below rides on. --- ## 1. Concepts and vocabulary | Term | Meaning | |---|---| | **Evidence** | A signed, append-only record imported from a source (calendar event, note, photo metadata, …). Evidence is never modified; it can only be superseded by newer evidence or tombstoned by its source adapter. | | **Claim** | A derived statement *about the user* (e.g. "user has a weekly gym routine on Tue/Thu mornings"). Claims are produced exclusively by derivers; they are never imported directly. | | **Deriver** | A pure, deterministic function from a set of input records (evidence and/or other claims) to zero or more candidate claims, each with a structured rationale and a confidence estimate. | | **Derivation graph** | The DAG whose nodes are evidence records and claims, and whose edges are "was derived from" links. Evidence nodes are sources (in-degree 0 from the derivation perspective). | | **Provenance chain** | For any claim, the transitive closure of its inputs in the derivation graph, down to raw evidence, together with the deriver identities and versions that produced each hop. | | **Correction** | A signed user operation that asserts a replacement value for a claim ("my office is actually at X, not Y"). | | **Refutation** | A signed user operation that asserts a claim is false, with no replacement ("I do not have a Tuesday gym routine"). | | **Invalidation** | The mechanical consequence of a correction, refutation, or evidence retraction: every downstream claim that depended on the affected node is marked invalid and queued for re-derivation. | | **Explanation record** | A structured, human-renderable artifact, generated on demand, that walks a claim's provenance chain and reproduces the rationale at each derivation hop. | ### 1.1 Design invariants 1. **Append-only.** Nothing in the log is ever rewritten. Corrections, refutations, and invalidations are *new* signed operations that reference prior operations by content ID. 2. **Determinism.** Given the same set of valid (non-invalidated) input records and the same deriver version, a deriver MUST emit the same claims with the same confidences. This is what makes re-derivation after a correction well-defined and what lets two nodes converge after sync. 3. **Claims never outrank the user.** A user refutation is terminal for a claim identity: the engine MUST NOT re-emit a claim whose identity key matches an active refutation, regardless of how much supporting evidence accumulates (see §6.4). 4. **No claim without provenance.** A claim operation with an empty `inputs` list is invalid and MUST be rejected at log-append time. 5. **Confidence is honest.** Confidence values are calibrated probabilities, not vibes. The regression suite (`tests/regression/`) pins golden confidence outputs so calibration drift is a reviewed, deliberate change. --- ## 2. Operation kinds used by the derivation layer The derivation layer adds the following operation kinds to the milestone #2 log schema. All of them are standard signed operations: canonically encoded (`mnema.core.canonical`), content-addressed (`mnema.core.ids`), and signed with a device key (`mnema.core.keys`). | Kind | Author | Purpose | |---|---|---| | `claim` | derivation engine (device key) | Asserts a derived statement with inputs, rationale, and confidence. | | `claim_correction` | user | Replaces a claim's value; carries the corrected payload. | | `claim_refutation` | user | Marks a claim identity as false; no replacement payload. | | `claim_invalidation` | derivation engine | Records that a claim was mechanically invalidated, and why (the triggering operation's ID). | | `evidence_retraction` | source adapter or user | Withdraws an evidence record (e.g. a deleted calendar event), triggering the same cascade as a refutation. | ### 2.1 The `claim` operation body ```json { "kind": "claim", "claim_type": "routine.weekly", "identity_key": "routine.weekly|gym|tue,thu|morning", "subject": "self", "payload": { "...claim-type-specific fields..." }, "inputs": [ {"op_id": "b3:9f2c…", "role": "supporting_event"}, {"op_id": "b3:1a7e…", "role": "supporting_event"} ], "deriver": {"name": "routines", "version": "1.2.0"}, "confidence": 0.87, "confidence_basis": { "prior": 0.30, "factors": [ {"name": "occurrence_count", "value": 9, "log_odds": 1.40}, {"name": "regularity", "value": 0.93, "log_odds": 1.10}, {"name": "recency", "value": 0.88, "log_odds": 0.35} ] }, "rationale": "9 matching calendar events over 5 weeks; inter-event spacing CV 0.07; most recent occurrence 2 days ago.", "supersedes": null, "valid_at": "2025-06-02T08:00:00Z" } ``` Field semantics: * **`claim_type`** — a dotted, namespaced type string. Built-in types are listed in `docs/derivers.md`. Third-party derivers MUST use a reverse-DNS prefix (e.g. `com.example.fitness.streak`) to avoid collisions. * **`identity_key`** — the *stable identity* of the claim, independent of its current payload or confidence. Two claim operations with the same identity key are versions of the same claim; the later one (by log order, ties broken by op ID) supersedes the earlier. The identity key is also the unit at which refutations apply. Each deriver documents its identity-key construction in `docs/derivers.md`; implementations MUST reproduce it byte-for-byte for interop. * **`inputs`** — content IDs of every record the deriver consumed *and that contributed to this claim*. Derivers MUST NOT list inputs they merely scanned and discarded; the inputs list is the cascade boundary, and over-listing causes spurious invalidations while under-listing causes stale claims to survive corrections. Each input carries a `role` string the explanation renderer uses. * **`deriver`** — name and semantic version of the producing deriver. Bumping a deriver's version invalidates nothing by itself, but the engine records it so explanations can say *which logic* produced a claim, and so goldens can be re-baselined per version. * **`confidence` / `confidence_basis`** — see §5. The basis is mandatory: a confidence number without its factor decomposition is not auditable. * **`supersedes`** — op ID of the prior claim version with the same identity key, if any. Maintained by the engine, not the deriver. * **`valid_at`** — the logical time at which the deriver ran, i.e. the high-water mark of evidence it had seen. Used to scope re-derivation. ### 2.2 Correction and refutation bodies ```json { "kind": "claim_correction", "target_claim": "b3:44de…", "target_identity_key": "place.frequent|office", "corrected_payload": { "label": "office", "location": {"lat": 52.52, "lon": 13.405} }, "note": "That's my old office; we moved in March." } ``` ```json { "kind": "claim_refutation", "target_claim": "b3:90aa…", "target_identity_key": "routine.weekly|gym|tue,thu|morning", "note": "I cancelled that membership." } ``` Both target a specific claim op **and** its identity key. The op ID pins exactly what the user saw and rejected; the identity key extends the user's intent to future re-derivations (§6.4). Corrections and refutations MUST be signed by a key holding the `correct` capability for the subject — in the single-user reference node that is any of the user's device keys. --- ## 3. The claim lifecycle A claim identity moves through these states (tracked per identity key in `mnema.derive.graph`): ``` ┌────────────────────────────────────────────────┐ │ │ (deriver) ───► ACTIVE ──correction──► CORRECTED (user value pinned) │ │ ▲ │ │ │ └── re-derivation emits superseding claim │ │ │ │ │ ├──refutation──────► REFUTED (terminal) │ │ │ │ │ └──upstream change─► INVALIDATED ─► (re-derive) └────────────────────────────────────────────────┘ ``` * **ACTIVE** — the latest claim op for this identity key, not corrected, refuted, or invalidated. This is what query surfaces and delegated views see. * **CORRECTED** — a user correction exists. The *user's corrected payload* is authoritative and is served with `confidence = 1.0` and `confidence_basis = {"source": "user_correction"}`. The original deriver output is retained in the log for audit but never served. If later evidence causes the deriver to re-derive a value *equal* (canonical-bytes equal) to the correction, the engine may note the agreement in the explanation; if the re-derived value *differs*, the correction still wins and the disagreement is surfaced as a low-priority review item (see `mnema.derive.sink`'s `pending_review` channel), never as a silent override. * **REFUTED** — terminal. The identity key is blocklisted for re-emission (§6.4). Downstream claims are invalidated (§6). * **INVALIDATED** — an input (evidence or upstream claim) was retracted, corrected, or refuted. The engine records a `claim_invalidation` op and schedules the responsible deriver to re-run over the surviving inputs. Re-derivation may produce: the same claim (new op, possibly different confidence), a different claim, or nothing. State is *derived* from the log, never stored as mutable truth: replaying the log from genesis reconstructs identical state on any node, which is what makes sync (milestone #6) a pure log-merge problem. --- ## 4. The derivation graph `mnema.derive.graph.DerivationGraph` is the in-memory index the engine maintains over the log. It is rebuilt by replay and updated incrementally on append. ### 4.1 Structure * **Nodes**: every evidence op and every claim op, keyed by op ID. * **Forward edges** (`inputs`): claim → each of its input op IDs. * **Reverse edges** (`dependents`): input op ID → set of claim op IDs that list it. This is the index that makes cascade invalidation O(affected subgraph) instead of O(log). * **Identity index**: identity key → ordered list of claim op versions, plus current lifecycle state and any pinning correction/refutation. ### 4.2 Acyclicity The graph MUST be a DAG. A claim's inputs are op IDs of operations that are already in the log when the claim is appended, and op IDs are content hashes; a cycle would require a hash collision or a forward reference, both of which append-time validation rejects. Layered derivation (claims derived from claims — e.g. a `preference` derived from a `routine`) is therefore safe and explicitly supported; the engine runs derivers in topologically ordered passes until fixpoint (§7.2), with a pass-count ceiling (`MAX_DERIVATION_DEPTH`, default 8) as a defense-in-depth bound against a misbehaving third-party deriver that emits ever-new identity keys. ### 4.3 Provenance queries `DerivationGraph.provenance(op_id)` returns the provenance chain: a deterministic, depth-first, deduplicated walk of forward edges down to evidence, yielding `(depth, op_id, role, deriver)` tuples. This is the raw material for explanation records (§8) and for the milestone #6 capability layer, which must be able to compute "what evidence is *reachable* from this claim" to enforce claims-without-evidence sharing. --- ## 5. Confidence: model and calibration Implemented in `mnema.derive.confidence`. ### 5.1 The combination rule Each deriver expresses its judgment as a **prior** plus independent **evidence factors**, combined in log-odds space: ``` logit(p) = logit(prior) + Σ_i w_i(value_i) p = σ( logit(prior) + Σ w_i ) ``` * The **prior** is the deriver's base rate for this claim type firing at all (e.g. `0.30` for weekly routines: most candidate event clusters are not real routines). Priors are per-claim-type constants documented in `docs/derivers.md` and were chosen against the sample corpus in `data/samples/`. * Each **factor** maps an observed quantity (occurrence count, regularity, recency, corroboration across sources, …) to a log-odds contribution via a documented monotone curve (typically a saturating function like `w_max · (1 − e^{−k·x})`, so that the tenth confirming event adds less than the second). Factor curves live next to their deriver and are pure functions — no hidden state. * The naive-Bayes independence assumption is wrong in detail and we say so: the saturating curves and clamping (below) exist precisely to keep the combined estimate sane when factors correlate. ### 5.2 Clamping and the confidence scale Combined confidence is clamped to `[0.02, 0.98]`. The derivation layer never asserts certainty; only a user correction yields `1.0`, and only a user refutation yields the terminal "false". Surfaces SHOULD render confidence in four bands — *speculative* (< 0.40), *probable* (0.40–0.70), *likely* (0.70–0.90), *strong* (≥ 0.90) — rather than raw decimals. ### 5.3 Calibration discipline "Calibrated" means: across the sample corpus and its labeled outcomes, claims emitted at confidence `p` should be true about `p` of the time per band. The reference implementation enforces this socially and mechanically: 1. **Golden regression tests** (`tests/regression/test_confidence_golden.py`) pin exact confidence outputs for every claim derived from the sample corpus. Any change to a prior, factor curve, or combination rule fails the suite and must be re-baselined via `scripts/update_goldens.py` — making every calibration change an explicit, diff-reviewed event. 2. **Monotonicity tests** (`tests/test_confidence.py`) assert that more supporting evidence never *decreases* confidence, that retracting a supporting input never *increases* it, and that all outputs respect the clamp. 3. **Band calibration check**: the regression suite buckets golden claims by band and asserts the labeled-true fraction in each band falls within the band's nominal range ± 0.10 tolerance on the sample corpus. This is a smoke check, not a proof — the corpus is small — but it catches gross miscalibration. ### 5.4 Confidence under re-derivation Re-derivation after an invalidation recomputes confidence from surviving inputs only — there is no "memory" of the prior confidence. This is a direct consequence of determinism (§1.1) and is tested in `tests/engine/test_corrections.py`: retracting 3 of 9 supporting events for a routine yields exactly the confidence that 6 events would have produced from scratch. --- ## 6. Corrections, refutations, and cascade invalidation This is the heart of the milestone. Implemented across `mnema.derive.graph` (the cascade walk) and `mnema.derive.engine` (re-derivation scheduling). ### 6.1 Triggers A cascade starts when any of these appends to the log: * `claim_correction` — the claim's *payload* changed; dependents that consumed the old payload may now be wrong. * `claim_refutation` — the claim is false; everything built on it is suspect. * `evidence_retraction` — an evidence input vanished. * A **superseding claim** — a re-derived claim version whose payload differs (canonical-bytes) from the version a dependent consumed. (If the payload is byte-identical and only confidence changed, dependents are scheduled for *confidence refresh* but not invalidated — their inputs are still semantically intact.) ### 6.2 The cascade algorithm ``` def cascade(trigger_op): frontier = {target of trigger_op} invalidated = [] seen = set() while frontier: node = pop frontier (deterministic order: log index) for dep in graph.dependents(node): if dep in seen: continue seen.add(dep) if state(dep.identity_key) in (CORRECTED, REFUTED): # User assertions are not invalidated by machine cascades. # Record a pending_review item instead (§6.3). continue append claim_invalidation(dep, cause=trigger_op) invalidated.append(dep) frontier.add(dep) schedule re-derivation for the derivers of invalidated claims, scoped to the affected identity keys, in topological pass order. return invalidated ``` Properties, all covered in `tests/test_graph_cascade.py` and `tests/engine/test_corrections.py`: * **Completeness** — every transitive dependent of the trigger target is visited (diamond and deep-chain topologies tested). * **Minimality** — nothing outside the reachable set is invalidated; unrelated claims are untouched. * **Idempotence** — replaying the same trigger produces no further invalidation ops (the `seen` set is effectively persisted as the invalidation records themselves; append-time validation rejects a duplicate invalidation of an already-invalidated claim version). * **Determinism** — invalidation ops are appended in log-index order of the invalidated claims, so two nodes replaying the same log produce identical invalidation sequences. * **User-assertion firewall** — corrected/refuted claims are never machine-invalidated (§6.3). ### 6.3 Cascades meeting user assertions If a cascade reaches a claim the user has CORRECTED, the machine does not get to undo the user. The engine stops the walk at that node, leaves the correction authoritative, and emits a `pending_review` item via `mnema.derive.sink` saying, in effect: *"Evidence underlying your correction about X changed; you may want to revisit it."* The user can then re-correct, withdraw the correction (a new op), or ignore it. REFUTED nodes likewise block the walk (there is nothing downstream of a refuted claim to protect — its dependents were already invalidated when the refutation landed — but a *new* cascade arriving at a refuted identity is simply a no-op). ### 6.4 Refutation permanence and the re-emission blocklist A refutation blocklists its identity key. During derivation, the engine filters every deriver's candidate claims against active refutations *before* appending. This is deliberate and tested: the alternative — letting fresh evidence resurrect a refuted claim — turns every refutation into a nag. If the user genuinely changes their mind, they withdraw the refutation with an explicit `refutation_withdrawal` op (supported by the engine; surfaced in the milestone #5 UI), which re-opens the identity key and triggers re-derivation. One subtlety: refutation applies to the *identity key*, not the claim type. Refuting "gym routine Tue/Thu mornings" does not blocklist a later, genuine "gym routine Sat mornings" — different identity key. Derivers therefore design identity keys to match the granularity at which a user would say "that's false" (see per-deriver notes in `docs/derivers.md`). ### 6.5 Re-derivation After a cascade, the engine re-runs the affected derivers scoped to the invalidated identity keys' input neighborhoods, in topological pass order, to fixpoint. Re-derived claims are new ops with `supersedes` pointing at the invalidated version, so the full history — believed, invalidated because of Y, re-believed at lower confidence — is readable straight off the log and rendered in explanations. --- ## 7. The engine `mnema.derive.engine.DerivationEngine` orchestrates everything. ### 7.1 Inputs and outputs * Reads: the operation log (`mnema.core.log`), via the typed loaders in `mnema.derive.loaders` that project evidence ops into deriver-friendly records. * Writes: `claim`, `claim_invalidation` ops (signed with the device key) and `pending_review` items, via `mnema.derive.sink`. * Never writes corrections or refutations — those are user-only. ### 7.2 Execution model 1. **Replay** the log; build the derivation graph and lifecycle states. 2. **Pass loop**: run all registered derivers over valid records. Pass 1 sees evidence only; pass *n* additionally sees claims emitted in passes < *n*. Repeat until a pass emits nothing new (fixpoint) or `MAX_DERIVATION_DEPTH` is hit. 3. **Filter** candidates against refutation blocklist and correction pins. 4. **Dedupe**: a candidate identical (identity key + canonical payload + confidence) to the current ACTIVE version is dropped — no append, no churn. A candidate with the same identity key but different payload or confidence supersedes (§6.1's superseding-claim rule then applies). 5. **Append** survivors in deterministic order (deriver name, then identity key, lexicographic). Incremental mode (the normal CLI path) replays from the engine's last `valid_at` watermark instead of genesis; `tests/engine/test_pipeline.py` asserts incremental and from-scratch runs converge to identical state. --- ## 8. Explanation records Implemented in `mnema.derive.explain`. An explanation answers, for one claim: **what is believed, why, from what, with what confidence, and what has changed**. `explain(graph, log, claim_op_id)` produces a structured record: ```json { "claim": {"op_id": "b3:44de…", "claim_type": "routine.weekly", "payload": {…}, "state": "active", "confidence": 0.87, "band": "likely"}, "because": [ {"deriver": {"name": "routines", "version": "1.2.0"}, "rationale": "9 matching calendar events over 5 weeks; …", "confidence_basis": {…}} ], "built_from": [ {"op_id": "b3:9f2c…", "role": "supporting_event", "kind": "evidence", "source": "calendar", "summary": "Event 'Gym' 2025-05-27 07:00 (60m)", "redacted": false}, … ], "history": [ {"op_id": "b3:1100…", "event": "derived", "at": "…", "confidence": 0.74}, {"op_id": "b3:2233…", "event": "invalidated", "cause": {"op_id": "b3:1f00…", "kind": "evidence_retraction"}}, {"op_id": "b3:44de…", "event": "re-derived", "at": "…", "confidence": 0.87} ], "user_actions": [] } ``` Rules: * Explanations are **generated, not stored** — they are a deterministic rendering of log facts, so they can never drift from the truth they explain. (Surfaces may cache them keyed by claim op ID; op IDs are content hashes, so the cache is trivially correct.) * `built_from` summaries come from per-source summarizer functions in the loaders; they MUST be derived only from the evidence record itself. * The `redacted` flag exists for the capability layer (milestone #6): a delegated node holding a claim *without* its evidence renders the same explanation structure with `redacted: true` entries — provenance *shape* is shared, provenance *content* is not. This keeps explanation records interoperable across trust boundaries. * `history` is the chronological lifecycle of the identity key, including the cause of every invalidation — this is the artifact that demos "you corrected X, therefore Y changed". The CLI (`mnema.derive.cli`) renders explanations as indented text trees (`mnema explain `); `tests/engine/test_explanations.py` covers both structured and rendered forms, including post-cascade histories. --- ## 9. Interoperability notes For a second implementation to interoperate at the derivation layer: 1. **Wire compatibility** — produce/consume the five operation kinds in §2 with canonical encoding per milestone #2. Unknown `claim_type`s MUST be preserved, synced, and renderable as opaque claims (type string + rationale + confidence); never dropped. 2. **Identity-key fidelity** — reproduce built-in identity-key construction byte-for-byte (specified per deriver in `docs/derivers.md`), or refutation permanence breaks across implementations. 3. **Cascade equivalence** — given the same log, you MUST reach the same set of invalidated claim versions (§6.2 determinism). The cascade scenarios in `tests/test_graph_cascade.py` are written as table-driven topologies and are intended to be portable as a conformance suite. 4. **Confidence is advisory across implementations** — you need not reproduce our factor curves for *your* derivers, but you MUST emit a `confidence_basis` decomposition for any claim you produce, and you MUST preserve ours verbatim when relaying. 5. **The user-assertion firewall (§6.3) and refutation permanence (§6.4) are normative.** An implementation that lets machine derivation override user corrections or resurrect refuted claims is non-conforming. ## 10. Explicit non-goals of the derivation layer * No ML-model inference in the reference derivers — rules and transparent heuristics only. The operation schema (`deriver`, `confidence_basis`, `rationale`) is designed so a model-backed deriver *could* slot in, but shipping one is out of scope and would compromise auditability of the reference. * No cross-user derivation. `subject` is always `self` in this milestone. * No automatic deletion of evidence on refutation — refuting a claim does not retract the evidence it was derived from; the user does that separately if desired. * No natural-language explanation generation — explanation records are structured + template-rendered, by design.