# Review Workflow & State Machine

**Status:** Stable draft for MVP implementation
**Related:** [04-versioning-and-forking.md](04-versioning-and-forking.md), [05-rbac-permissions.md](05-rbac-permissions.md), ADR-010

---

## 1. Goals

- Guarantee the platform invariant: **nothing is published without passing peer review.**
- Keep review latency low enough that contributors stay motivated (target: first reviewer response < 72 h at MVP scale).
- Make review *teachable*: structured checklists, inline comments anchored to content blocks, and visible rubrics raise community quality over time.
- Resist abuse: collusion, review-bombing, drive-by rejections, and self-approval.

## 2. Entities

| Entity | Purpose |
|---|---|
| `Review` | One review cycle for one submitted version. Holds state, checklist results, decision, timestamps |
| `ReviewAssignment` | A reviewer's claim on a review (supports multi-reviewer quorum) |
| `ReviewComment` | Threaded comments, optionally anchored to a content block id + text range |
| `ReviewChecklistResult` | Per-item rubric outcomes (pass/fail/n-a + note) |

A `Review` belongs to exactly one `ProblemVersion`/`CourseVersion`. Resubmission after `changes_requested` creates a **new version and a new Review**, linked via `previous_review_id` so the full conversation history forms a chain visible in the UI.

## 3. The state machine

```mermaid
stateDiagram-v2
    [*] --> submitted: author submits draft
    submitted --> in_review: first reviewer claims
    submitted --> withdrawn: author withdraws
    submitted --> expired: 30 days unclaimed (requeue w/ boost)
    in_review --> changes_requested: any reviewer requests changes
    in_review --> accepted: quorum of approvals
    in_review --> rejected: quorum of rejections
    in_review --> submitted: all reviewers unclaim / staleness timeout
    changes_requested --> [*]: author revises → NEW version,\nnew review (linked)
    changes_requested --> abandoned: 90 days inactive
    accepted --> [*]: → publish pipeline (doc 04 §3.2)
    rejected --> [*]: terminal for this version;\nauthor may revise & resubmit
    withdrawn --> [*]
```

### 3.1 Transition rules (authoritative table)

| From | To | Trigger | Actor | Guards |
|---|---|---|---|---|
| `submitted` | `in_review` | claim | Reviewer | topic scope match; not author/maintainer; COI check (< 25 % diff authorship); ≤ 5 concurrent claims per reviewer |
| `submitted` | `withdrawn` | withdraw | Author | — |
| `submitted` | `expired` | timer 30 d | System | auto-requeues with priority boost + broadcast to scope-matching reviewers |
| `in_review` | `changes_requested` | decision | Any assigned reviewer | requires ≥ 1 actionable comment or failed checklist item |
| `in_review` | `accepted` | quorum met | System (on last needed approval) | quorum rules §3.2; all *blocking* checklist items pass |
| `in_review` | `rejected` | quorum met | System | requires written rationale ≥ 100 chars from each rejecting reviewer |
| `in_review` | `submitted` | unclaim / 14 d reviewer inactivity | Reviewer / System | comments persist for the next claimant |
| any non-terminal | terminal override | admin override | Admin | justification required; high-severity audit event |

All transitions are executed in a single transaction with a row lock on the `Review`, emit an audit event, and enqueue notifications. Transitions are implemented as an explicit table-driven state machine (one function, `transition(review, action, actor)`), not scattered view logic, so the guard set above is testable in isolation.

### 3.2 Quorum rules

Configurable per instance; MVP defaults:

| Submission type | Approvals needed | Rejections needed |
|---|---|---|
| New problem, `minor`-class revision | 1 | 1 |
| New problem from author with < 3 published problems | 2 | 1 |
| `major`-class revision of published problem | 2 | 1 |
| New course / course `major` revision | 2 (≥ 1 with course-review grant) | 2 |
| Fast-track (hash-identical rollback, pin refresh) | 0 (auto) | — |

A mixed outcome (1 approve, 1 reject where quorum is 2/1) escalates: the review is flagged `contested`, a third reviewer is requested, and majority of three decides.

### 3.3 Review checklist (rubric)

Every review presents a structured checklist; blocking items (★) must pass for approval:

1. ★ **Correctness** — statement is unambiguous; answer spec is correct; numeric tolerance sensible; symbolic equivalence rules appropriate.
2. ★ **Solvability** — reviewer actually solved it (the review UI requires the reviewer to pass the problem's own answer check, or explicitly attest for proof-style content).
3. ★ **Originality/licensing** — no copied proprietary content; attribution chain valid for derivatives; passes the automated similarity pre-screen (§5).
4. ★ **Safety** — widgets reference only registered widget versions; no policy-violating material.
5. **Pedagogy** — hints scaffold rather than spoil; difficulty rating plausible; prerequisites accurate.
6. **Accessibility** — alt text on diagrams; math has speakable annotations where auto-derivation fails; color not sole information channel (per doc 09).
7. **Metadata** — tags, topic, difficulty, prerequisites sensible.

Items 5–7 failing produce `changes_requested` by convention but a reviewer may approve with notes; items 1–4 are hard blocks enforced by the transition guard.

### 3.4 Inline comments

`ReviewComment.anchor` = `{ blockId, textRange?: {from, to}, documentHash }`. Anchors store the `content_hash` they were made against; when the author resubmits, anchors are re-mapped to the new version by block id (stable across edits — see content format §03), and comments whose block was deleted render as "on removed content". Threads support `resolved` state toggled by author or commenter; unresolved threads are surfaced in the next review cycle.

## 4. Queue mechanics & reviewer experience

- **Queue ordering:** priority = f(wait time, author track record, expired-requeue boost, contested flag). Strictly no "skip the line by reputation of the author" — only *waiting time* dominates.
- **Claim TTL:** 14 days of reviewer inactivity auto-unclaims (with reminders at day 7 and 12).
- **Load protection:** reviewers cap at 5 concurrent claims; the queue UI shows estimated effort (diff size, problem type).
- **Review of reviews:** authors rate review helpfulness (not outcome agreement — the prompt explicitly asks "was this review clear and actionable?"); moderators see aggregate reviewer stats (response time, contested rate, helpfulness) when renewing grants.

## 5. Automated pre-screen (before human review)

On submission, a Celery pipeline runs and attaches results to the Review:

1. **Schema validation** — document validates against the JSON Schemas (docs/schemas/); hard fail blocks submission client-side and server-side.
2. **Render check** — MDX compiles, KaTeX parses all math, all asset references resolve.
3. **Answer self-test** — for auto-gradable types, the stored canonical answer is run through the grader and must pass; numeric problems must define tolerance; MCQ must have ≥ 1 correct option.
4. **Similarity screen** — shingled MinHash against the published corpus + recent submissions; > 0.85 similarity flags the review with side-by-side comparison (signal for reviewers, never an auto-reject).
5. **Link & widget audit** — external links checked against blocklist; widget references must resolve to registered, reviewed widget versions (doc 07).
6. **Profanity/spam heuristic** — flags only.

Failures of 1–3 bounce the submission back to draft with machine-readable errors; 4–6 annotate the review.

## 6. Notifications

| Event | Recipients |
|---|---|
| Submission enters queue | Scope-matching reviewers (digest, not instant — anti-spam) |
| Claimed / decision / comment | Author (instant) |
| Contested escalation | Scope reviewers + moderators |
| Stale claim warnings | Assigned reviewer |
| Published | Author, watchers of the entity, followers of the author |

All notifications respect per-user channel preferences (in-app always; email opt-out per category) and are delivered via the jobs queue with idempotency keys (`event_id`), so retries never double-send.

## 7. Reputation effects

| Event | Author Δ | Reviewer Δ |
|---|---|---|
| Version accepted & published | +15 (problem) / +40 (course) | +5 per completed review |
| Review marked helpful by author | — | +3 |
| Published problem upvote / downvote | +2 / −1 (floor: daily cap 100) | — |
| Submission rejected | 0 (no penalty — rejection must stay cheap to keep quality bars high) | — |
| Contested review where reviewer was in final minority | — | 0 (no penalty; disagreement is healthy) |
| Content retracted for plagiarism | −100 + contributor flag | — |
| Report upheld against user's content | −10 | — |

Topic-scoped reputation accrues to the primary topic of the content involved; global reputation is the sum. All reputation mutations are append-only ledger rows (`reputation_events`), never in-place counters, so totals are recomputable and auditable.

## 8. Anti-abuse summary

- **Self-approval:** structurally impossible (author/maintainer/COI guards on claim).
- **Sockpuppet farms:** reviewer grants are human-appointed; reputation from votes is daily-capped and vote-ring detection (mutual-vote graph analysis, weekly job) flags clusters to moderators.
- **Review-bombing a rival:** rejections require substantial written rationale tied to checklist items; contested escalation brings fresh eyes; reviewer aggregate stats expose serial rejectors.
- **Stalling (claim-and-sit):** claim TTL + concurrency caps.
- **Plagiarism laundering via forks:** the similarity screen runs against the whole corpus including the forker's own upstream; forks with near-zero diff from upstream are flagged "no meaningful change" for reviewers.