# 03 — Content Storage Format Specification

This document specifies the structured format for all educational content:
problems, hints, solutions, courses, and widget references. The format is
**MDX-in-JSON**: a JSON envelope (validated by the schemas in `schemas/`)
whose prose fields contain a constrained MDX dialect. Raw HTML is **never**
stored or rendered.

Design goals, in priority order:

1. **Safety** — content is untrusted input; the format must be sanitizable and
   renderable without script execution in the main origin.
2. **Portability** — documents must survive export/import (OER bundles),
   forking across instances, and offline rendering.
3. **Diff-ability** — reviewers need meaningful diffs between versions.
4. **Accessibility** — the format forces alt text, captions, and
   plain-language fallbacks at the schema level, not as an afterthought.

## 1. The MDX dialect ("PolyMDX")

PolyMDX is CommonMark + GFM tables/strikethrough/task-lists, plus:

- **Math:** `$...$` inline and `$$...$$` block (KaTeX). No raw `\( \)`.
- **A fixed registry of trusted components** (capitalized JSX tags). Anything
  not in the registry fails validation at save time.
- **No** raw HTML, no `import`/`export` statements, no JS expressions
  (`{...}` attribute values are limited to JSON literals), no `<script>`,
  no event-handler props. The compiler runs with expressions disabled and an
  allowlist acorn check; violations are hard validation errors.

### Trusted component registry (MVP)

| Component | Purpose | Required a11y props |
|---|---|---|
| `<Figure src alt caption?>` | images via MediaAsset ID (`media:` URI) | `alt` (may be `""` only with `decorative={true}`) |
| `<Diagram kind="tikz-like" source alt>` | server-rendered SVG diagrams (compiled in worker, served as static SVG) | `alt`, `longDescription?` |
| `<Plot spec alt>` | declarative function/data plots (JSON spec, rendered by trusted plotting component) | `alt` |
| `<Code lang src? children?>` | syntax-highlighted, non-executable code block | — |
| `<Callout kind="note|warning|theorem|definition|example">` | semantic admonitions (mapped to ARIA roles) | — |
| `<Spoiler summary>` | collapsed content (used for "reveal" pedagogy) | `summary` text |
| `<Tabs>` / `<Tab label>` | tabbed alternatives (e.g., Python/JS versions) | `label` |
| `<Term id>` | glossary term reference | — |
| `<ProblemRef slug>` | inline link card to another problem | — |
| `<Widget ref instance fallback>` | mount point for a sandboxed widget — see §5.9 and doc 07 | `fallback` MDX is **required** |

This registry is intentionally small. New trusted components require an ADR
because each one extends the trusted rendering surface.

## 2. Document envelope: common fields

Every content document shares this envelope (full schema:
`schemas/problem.schema.json`, `schemas/course.schema.json`):

```json
{
  "schemaVersion": 1,
  "kind": "problem",
  "language": "en",
  "title": "Sums of consecutive integers",
  "attribution": {
    "authors": [{"user": "ada", "role": "author"}],
    "forkedFrom": null,
    "license": "CC-BY-SA-4.0"
  }
}
```

- `schemaVersion` — integer, bumped on breaking schema changes; the API
  migrates old documents lazily on read and permanently on next version write.
- `language` — BCP-47. Translations are **separate linked documents**, not
  parallel fields (see doc 09 §3).
- `attribution` — embedded so documents remain attributable when exported;
  the database remains authoritative inside the platform (doc 04 §7).

## 3. Problem document

```json
{
  "schemaVersion": 1,
  "kind": "problem",
  "language": "en",
  "title": "Clock arithmetic",
  "statement": "<PolyMDX source>",
  "statementPlain": "optional plain-text/low-bandwidth fallback",
  "answer": { "$ref": "see §5 — AnswerSpec, stored in separate column" },
  "hints": [
    {"ordinal": 1, "body": "<PolyMDX>", "penaltyPercent": 10},
    {"ordinal": 2, "body": "<PolyMDX>", "penaltyPercent": 25}
  ],
  "solutions": [
    {"ordinal": 1, "kind": "official", "title": "Via modular arithmetic", "body": "<PolyMDX>"}
  ],
  "metadata": {
    "difficulty": 2,
    "estimatedMinutes": 5,
    "topics": ["math.number-theory.modular-arithmetic"],
    "tags": ["clocks", "remainders"],
    "prerequisites": [{"problem": "intro-to-remainders", "strength": "recommended"}],
    "learningObjectives": ["Compute residues mod n for small n"]
  },
  "assets": ["media:01J5...", "media:01J6..."],
  "widgets": [
    {
      "instanceId": "numberline-1",
      "package": "number-line",
      "version": "2.1.0",
      "props": {"min": 0, "max": 12, "marks": [3, 7]},
      "fallback": "A number line from 0 to 12 with marks at 3 and 7."
    }
  ]
}
```

Rules:

- `statement` MUST NOT contain or imply the answer in machine-readable form;
  the **AnswerSpec lives in a separate database column** (doc 02 §3) so the
  default API serializer physically cannot leak it.
- `assets` lists every `media:` URI referenced; validation cross-checks that
  the statement/hints/solutions reference only listed assets, enabling
  complete OER export and orphan detection.
- `hints` ordered, max 5; `penaltyPercent` feeds scoring and SRS.
- `solutions` — at least one `official` solution is required for the
  `submitted` transition (review gate, doc 06 §4).

## 4. Answer payload normalization

A learner submission is always normalized to
`{"type": "<answer type>", "value": ...}` before storage in
`ProblemAttempt.answer_payload`, with type-specific value shapes given below.
Grading is pure: `grade(answer_spec, payload) -> {correct, score, feedbackKey}`
— deterministic and unit-testable, no I/O except for `code` (runner call).

## 5. AnswerSpec — one shape per problem type

Full machine-readable definitions: `schemas/answer-spec.schema.json`. Every
spec has `"type"` plus type-specific fields, and optional shared fields
`feedback` (map of feedbackKey → PolyMDX shown after grading) and
`maxAttempts` (null = unlimited).

### 5.1 `multiple_choice`

```json
{
  "type": "multiple_choice",
  "choices": [
    {"id": "a", "body": "<PolyMDX>", "correct": false, "feedbackKey": "confused-perimeter"},
    {"id": "b", "body": "<PolyMDX>", "correct": true}
  ],
  "multiSelect": false,
  "shuffle": true,
  "partialCredit": false
}
```
`multiSelect: true` grades exact-set by default; `partialCredit: true` uses
`max(0, (hits − misses) / total_correct)`. Choice `id`s are stable across
versions for analytics. The serializer strips `correct` and `feedbackKey`
before sending choices to clients.

### 5.2 `numeric`

```json
{
  "type": "numeric",
  "accept": [
    {"value": 3.14159, "tolerance": {"kind": "absolute", "value": 0.001}},
    {"value": "355/113", "tolerance": {"kind": "relative", "value": 1e-6}}
  ],
  "unit": {"required": true, "canonical": "m/s", "accepted": ["m/s", "meters per second"]},
  "inputHint": "decimal"
}
```
Values may be decimal strings or exact rationals (`"355/113"`); grading is done
in exact rational/decimal arithmetic (Python `fractions`/`decimal`), never
binary floats. Tolerance kinds: `absolute`, `relative`, `sigfigs`. Unit
handling normalizes via a fixed unit table (Pint) when `unit` present.

### 5.3 `expression` (symbolic/math)

```json
{
  "type": "expression",
  "canonical": "2*x + sin(x)",
  "variables": [{"name": "x", "domain": "real"}],
  "equivalence": "symbolic",
  "constraints": {"forbidFunctions": ["diff", "integrate"], "requireSimplified": false},
  "sampling": {"points": 12, "range": [-10, 10], "tolerance": 1e-9}
}
```
Equivalence modes: `symbolic` (SymPy `simplify(expr − canonical) == 0` with a
CPU/time budget, falling back to sampling on timeout), `numeric_sampling`
(random-point evaluation; default fallback), `literal_normalized` (string
compare after canonical ordering — for "write the formula exactly" tasks).
Learner input is parsed from a restricted grammar (no function definitions, no
assignment), with an interactive "we parsed your input as $…$" echo before
submission. Symbolic grading runs in the worker with `multiprocessing` hard
timeouts — SymPy on untrusted parse trees is treated as semi-trusted (parse
with `sympy.parsing` whitelist, never `eval`).

### 5.4 `free_response` (proof-style)

```json
{
  "type": "free_response",
  "rubric": [
    {"id": "base-case", "points": 2, "description": "Establishes the base case"},
    {"id": "inductive-step", "points": 4, "description": "Valid inductive step"}
  ],
  "selfAssess": true,
  "minWords": 30,
  "modelAnswerRef": "solutions[0]"
}
```
MVP grading: after submission the learner sees the model solution and rubric
and **self-assesses** per rubric item; the attempt records
`{selfScores, finalized}`. Self-assessed completion counts toward progress but
awards reduced SRS confidence. Peer/reviewer grading is a post-MVP extension
(the rubric structure already supports it).

### 5.5 `code`

```json
{
  "type": "code",
  "languages": ["python", "javascript"],
  "starterCode": {"python": "def solve(xs):\n    ...\n"},
  "entrypoint": {"python": "solve", "javascript": "solve"},
  "testcases": [
    {"id": "t1", "visible": true,  "input": "[1,2,3]", "expected": "6", "points": 1},
    {"id": "t2", "visible": false, "input": "[]",      "expected": "0", "points": 1}
  ],
  "comparison": {"kind": "token_equal", "floatTolerance": null},
  "limits": {"cpuMs": 2000, "memoryMb": 128, "wallMs": 5000, "outputKb": 64}
}
```
Hidden testcase inputs/expected values never leave the server; clients receive
only pass/fail + visible-case details. Comparison kinds: `exact`,
`token_equal`, `float_tolerance`, `checker` (a reviewer-approved checker
program shipped in the spec, run in the same sandbox). Execution model in
doc 07 §6.

### 5.6 `ordering`

```json
{
  "type": "ordering",
  "items": [
    {"id": "i1", "body": "<PolyMDX>"},
    {"id": "i2", "body": "<PolyMDX>"},
    {"id": "i3", "body": "<PolyMDX>"}
  ],
  "correctOrder": ["i2", "i1", "i3"],
  "scoring": "kendall_tau"
}
```
Scoring: `exact` (all-or-nothing) or `kendall_tau` (normalized pairwise
agreement → partial credit). Items are shuffled server-side per attempt.

### 5.7 `matching`

```json
{
  "type": "matching",
  "left":  [{"id": "l1", "body": "$e^{i\\pi}$"}, {"id": "l2", "body": "$\\binom{4}{2}$"}],
  "right": [{"id": "r1", "body": "$-1$"}, {"id": "r2", "body": "$6$"}, {"id": "r3", "body": "$0$"}],
  "pairs": [["l1", "r1"], ["l2", "r2"]],
  "allowUnmatchedRight": true,
  "scoring": "per_pair"
}
```
`right` may include distractors. Scoring `exact` or `per_pair`
(correct pairs / total pairs).

### 5.8 `widget_task`

```json
{
  "type": "widget_task",
  "widgetInstance": "numberline-1",
  "resultSchema": {
    "type": "object",
    "required": ["selected"],
    "properties": {"selected": {"type": "integer", "minimum": 0, "maximum": 12}},
    "additionalProperties": false
  },
  "grader": {"kind": "json_logic", "rule": {"==": [{"var": "selected"}, 10]}}
}
```
The widget produces a result payload (via the postMessage protocol, doc 07
§4); the **server** validates it against `resultSchema` (JSON Schema) and
grades with a declarative grader: `json_logic` (a vetted JsonLogic evaluator —
pure data, no code) or `equals` (deep-equal to a canonical value). Widgets are
*input devices*, never grading authorities — a malicious widget can at worst
submit a wrong answer for the learner.

### 5.9 Widget instances & fallbacks

Every `<Widget>` in MDX must reference a `widgets[]` entry by `instanceId`;
every instance pins an exact reviewed `package@version` (immutable bundle,
doc 07); and `fallback` PolyMDX is **schema-required** so content degrades on
sandbox failure, JS-disabled clients, screen-reader "static mode", and
low-bandwidth mode.

## 6. Course document

Full schema: `schemas/course.schema.json`. Shape:

```json
{
  "schemaVersion": 1,
  "kind": "course",
  "language": "en",
  "title": "Modular Arithmetic from Scratch",
  "summary": "<PolyMDX>",
  "outcomes": ["..."],
  "modules": [
    {
      "slug": "foundations",
      "title": "Foundations",
      "lessons": [
        {
          "slug": "what-is-a-remainder",
          "kind": "lesson",
          "title": "What is a remainder?",
          "body": "<PolyMDX>",
          "problems": [
            {"problem": "clock-arithmetic", "version": 4, "required": true}
          ]
        },
        {"slug": "checkpoint-1", "kind": "quiz", "title": "Checkpoint",
         "problems": [{"problem": "...", "version": 2, "required": true}],
         "passingScore": 0.7},
        {"slug": "final", "kind": "project", "title": "Build a calendar cipher",
         "body": "<PolyMDX>", "rubric": [{"id": "...", "points": 4, "description": "..."}]}
      ]
    }
  ],
  "metadata": {"topics": ["math.number-theory"], "tags": [], "estimatedHours": 6,
               "prerequisites": [{"course": "arithmetic-review", "strength": "recommended"}]}
}
```

- Lesson kinds: `lesson` (prose + embedded practice), `problem_set`, `quiz`
  (scored, `passingScore`), `project` (rubric + free_response-style submission).
- Problem references pin `{problem, version}`; `"version": "published"` opts
  into tracking the live published version (doc 04 §6 explains the trade-off
  and the re-review trigger).
- Module/lesson `slug`s are stable across versions where content corresponds,
  enabling progress migration on version upgrade (doc 04 §6.2).

## 7. Validation & compilation pipeline (write path)

```mermaid
flowchart LR
    A[Author saves draft] --> B["JSON Schema validation<br/>(envelope, AnswerSpec)"]
    B --> C["PolyMDX compile<br/>(remark/MDX, expressions disabled)"]
    C --> D["Component allowlist check<br/>+ attribute JSON-literal check"]
    D --> E["Sanitization pass<br/>(no raw HTML nodes survive)"]
    E --> F["Cross-reference checks:<br/>assets ⊆ assets[], widget refs resolve,<br/>hints/solutions limits, KaTeX parse check"]
    F --> G["Store: MDX source + compiled AST (jsonb)<br/>+ content_hash"]
    F -- any failure --> X["422 with path-anchored errors<br/>(editor highlights exact node)"]
```

Both the **source MDX** and the **compiled AST** are stored per version: the
source is canonical (diffs, forks, export); the AST is a render-cache that can
always be recompiled. Compilation happens server-side in a Node sidecar
invoked by the API (the only Node dependency on the write path), pinned to the
same PolyMDX compiler version the frontend uses, with the compiler version
recorded on the document.

## 8. Diffing model for review

Reviews display three synchronized diffs: (1) structural JSON diff of the
envelope (metadata, hints added/removed, AnswerSpec changes — AnswerSpec diffs
visible to reviewers only), (2) word-level Markdown diff of each prose field,
(3) rendered side-by-side preview. The line/offset anchors used by
`ReviewComment.anchor` reference the MDX *source* of the immutable version,
so anchors never rot.