# Ratchet docs: full text > Ratchet is an OpenAI-compatible gateway that distills open teacher models into a small student. You end up with an adapter you own, learned only from open teachers, with a gated nightly release. --- # What is Ratchet Source: https://docs.ratchet.dualmindlab.tech/guides/what-is-ratchet/ Ratchet is a gateway you point your OpenAI client at. A small Nemotron model with your LoRA adapter on top answers the requests it is sure about. A large open model answers the rest, and its answers become training data. Every night a new small model is trained and is released only if it passes a held-out exam, a silent trial and a staged rollout. Your app is a busy restaurant. Today every dish is cooked by a famous head chef from an agency that charges per plate, even when the order is toast for the hundredth time. Ratchet puts an apprentice next to the head chef. The apprentice cooks each dish a few times. If the results agree and the apprentice is sure, the plate goes out. If not, the head chef cooks it, and the dish goes in a recipe book with the chef's name and a note that copying it is allowed. At night the apprentice studies the book. The next morning's apprentice cooks only if it passes a sealed exam first. Every model call on the live site runs on Nebius Token Factory, the adapter was [retrained on Nebius AI Cloud](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius), and the student was served from a Nebius Serverless Endpoint [in a measured test](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius). ## The problem, in numbers you can check - **The bill grows with traffic.** Enterprise spend on model APIs was $12.5B in 2025 ([Menlo Ventures](https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise/), report of Dec 9, 2025, read Oct 26, 2026). Much of it is narrow and repetitive: labelling, extraction, ticket triage, steps inside agents. - **The obvious fix is often not allowed.** Training a cheaper model on a closed model's answers breaks the terms of the largest vendors. [OpenAI Business Terms §3.3(e)](https://openai.com/policies/business-terms/) and [Anthropic Commercial Terms §D.4(a)](https://www.anthropic.com/legal/commercial-terms) (both pages read Oct 21, 2026) forbid using outputs to train competing models. A tool that distills that traffic puts its users at risk. Not legal advice. - **Doing it by hand is slow.** Logging, cleaning, fine-tuning, evaluating, deploying and watching for regressions are separate chores, and most tools cover one.
Every request, repeated or not, is a paid call to the teacher. Distilling by hand is slow, and closed models forbid training on their outputs.
A student with your adapter answers what it is sure about, the teacher answers the rest, and a better student is trained every night from open teachers only.
## The idea Learn from the traffic you already have, but only from teachers whose license allows it, and . Change one line, your base URL, and four parts start working: Sampled 3 times; its confidence decides. And checks a random 5% of the student's answers. Each with its teacher and license, after redaction. Golden set, shadow, canary 10%, 50%, 100%, rollback. In more detail: | Part | What it does | |---|---| | **The student** | NVIDIA Nemotron 3 Nano 4B with a LoRA adapter you own. It answers first and is sampled three times. Its confidence mixes agreement across samples with token probabilities | | **The teacher** | NVIDIA Nemotron 3 Ultra on Nebius Token Factory. It answers when the student is unsure, and checks a random share of the student's answers (audits) | | **The ledger** | Every teacher answer is stored, after personal data is redacted, with the teacher that wrote it and the license that permits training on it. Closed models are refused as teachers in code | | **Night School and the gate** | Each night a new adapter is trained from teacher answers only and scored on a held-out golden set. It then goes through shadow mode, a canary at 10%, 50% and 100%, and rolls back automatically if audited precision drops |
```mermaid flowchart TB app(["Your app"]) ==> gw["Ratchet gateway"] gw ==> st["Student
Nano 4B + your LoRA"] st ==>|"confident"| out(["Answer"]) st -->|"unsure"| te["Teacher
Nemotron 3 Ultra"] te --> out te --> led[("Ledger
answer, teacher, license")] led --> ns["Night School
trains a new adapter"] ns --> gate{{"Release gate
golden set, shadow, canary"}} gate -.->|"next student"| st class st student class te teacher class led store class gate gate class out ok ```
**Reading this diagram.** Start at the top: every request reaches the student first, and the thick path is the cheap one, where a confident student answers at once. Only unsure requests go to the teacher, and every teacher answer also flows down into the ledger. The dotted line closes the loop: what the ledger collects trains the next student, and the gate decides whether that student replaces the current one. That loop is why the share of rented answers can shrink over time.
The same flow for one request, step by step:
```mermaid sequenceDiagram participant A as Your app participant G as Gateway participant Student as Student participant Teacher as Teacher participant Ledger as Ledger A->>G: chat request G->>Student: sample 3 answers Student-->>G: answers, probabilities alt confident G-->>A: student answer else unsure G->>Teacher: same request Teacher-->>G: teacher answer G->>Ledger: store with its license G-->>A: teacher answer end opt random audit G->>Teacher: check the answer Teacher-->>G: agrees or not G->>Ledger: record the verdict end ```
**Reading this diagram.** Time runs downward. The gateway always asks the student first. The `alt` block is the fork: a confident answer returns at once and costs the teacher nothing, while an unsure one is answered by the teacher and stored with its provenance. The `opt` block at the bottom is the audit: on a random share of confident answers the teacher quietly checks the student, so wrong-but-sure answers show up in the numbers instead of staying hidden. For the full detail see [One request's journey](/guides/request-journey/).
Ratchet also reports the bill honestly: savings per token and by GPU-hours actually paid, and a plain warning when a GPU endpoint costs more than it saves. See [Why it matters](/guides/why-it-matters/). ## What is real today | Part | Status | |---|---| | Gateway, cascade, ledger, Night School, release gate, Scout, dashboard | Built, with 1,934 tests that need no network and no GPU | | Teacher calls | Real Nemotron 3 Ultra on Nebius Token Factory. Real teacher calls built the text-to-SQL data on Oct 22, 2026 | | Student endpoint | **Served by hand in two short sessions on a Nebius Serverless Endpoint for speed tests (27 and 28 Oct 2026, [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)); the endpoint script is tested offline, ready to run with `make nebius-up`. On 28 Oct 2026 the adapter from Night School's first end-to-end run, by hand, was served from a Nebius Serverless Endpoint too, and the golden gate rejected it (81.8% precision against the 95% target), so it was never made live ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)).** Ratchet's own adapter is not connected to the live site: every live model call runs on Nebius Token Factory, where Ultra teaches and a random 40% of eligible requests are answered by the hosted Nano 30B. If a trained version were live without a GPU endpoint, its calls would go to the hosted, untuned Nano 30B , and pages would say so | | A text-to-SQL adapter | The published adapters were trained on a borrowed GPU. The round-2 recipe was also retrained once on Nebius AI Cloud (a Serverless Job on one L40S, 27 Oct 2026); it scored 40.7% (70 of 172) on exam 1 and 34.5% (69 of 200) on exam 2, against 39.0% and 33.0% for the original adapter: the result reproduces (the row-by-row differences are within chance). The measured results of the published adapters are on [Results](/guides/results/) | | A tool-calling adapter | Trained on a borrowed GPU (not retrained on Nebius). Hard exam 136 of 150 (90.7%) against 94 of 150 (62.7%) untrained and 137 of 150 (91.3%) for Ultra; BFCL exam 241 of 300 (80.3%) against 232 (77.3%) untrained and 265 (88.3%) for Ultra. SQL stays the headline result. See [Results](/guides/results/) | | Model ids and Token Factory list prices in `ratchet/config.py` | Checked on 29 Oct 2026 against the price list Token Factory serves (`scripts/day1_check.py` passed, [evidence/day1](https://github.com/harshbhx/ratchet/tree/main/evidence/day1)); the GPU-hour rate is still unverified | | Token Factory Terms of Service §5(f) ("competing service") | **Open question.** Asked Nebius support in writing on Oct 19, 2026; no answer received as of 29 Oct 2026. Not legal advice | ## Where to go next Is the student as good as the teacher? It is held to agreement with the teacher on held-out rows, not to ground truth (see [Escalation and audits](/guides/escalation-and-audits/)). Does it always save money? No, see [Cost and savings](/guides/cost-and-savings/). What do you own? The adapter and its signed bill of materials; the Nano base stays under NVIDIA's license. More in the [FAQ](/guides/faq/). --- # Why it matters Source: https://docs.ratchet.dualmindlab.tech/guides/why-it-matters/ Ratchet matters for four kinds of reader: the team with a growing LLM bill, the team that must show where its data went, the team that must show a model is safe to ship, and the builder who wants to own the adapter and its license trail. Each section below says what you get and links to the page that backs it.
```mermaid flowchart TB A(["Paying for
LLM calls"]) --> A1["Cost and savings"] B(["Must know where
data went"]) --> B1["Ledger and
Security"] C(["Regulated
or careful"]) --> C1["Evidence and
Release gate"] D(["Open-source
builder"]) --> D1["Licenses and
Night School"] A1 --> P{{"Proof you can run"}} B1 --> P C1 --> P D1 --> P class P ok ```
**Reading this diagram.** The rounded boxes at the top are the four readers and the boxes under them are the pages that hold the evidence for each. Every path ends at the same place: a page of commands you can run yourself. The rest of this page follows the same four paths. Pages: [Cost and savings](/guides/cost-and-savings/), [Ledger](/guides/ledger/), [Security](/guides/security/), [Training on your data](/guides/training-on-your-data/), [Evidence](/guides/evidence/), [Release gate](/guides/release-gate/), [Licenses](/guides/licenses/), [Night School](/guides/night-school/), [Proof you can run](/guides/proof/).
## If you pay for LLM calls Your bill is price per token times traffic. Much of that traffic is the same narrow task again and again. Ratchet moves the easy share of it to a small model and keeps the hard share on the teacher. Every request records what it cost and what the teacher alone would have cost, so the saving can be measured, not promised (no live saving has been measured yet). The catch is that a self-hosted model runs on a GPU that bills by the hour, busy or idle. So Ratchet counts both views: | View | What it counts | Can it be negative? | |---|---|---| | Per token | The price of each call actually made | No | | By GPU-hours paid | Teacher spend, plus every hour the endpoint and the training jobs ran | **Yes**, and it is never clamped to zero | The break-even is the number of student answers per endpoint hour at which the GPU pays for itself: ```text break-even = GPU price per hour / teacher spend avoided per student answer ``` With estimated prices (an L40S at $1.35 a GPU-hour on demand, [Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026, and an assumed $0.001 of teacher spend avoided per ticket-triage answer), that is about 1,350 student answers an hour, roughly 32,000 a day. All figures here are **estimates**, not measurements. | Traffic | Student answers an hour, against break-even (about 1,350) | What Ratchet's report says | |---|---|---| | A small gateway, 1,500 requests a day | About 60: far below | A 24/7 endpoint loses money. Use the hosted fallback or stop the endpoint when idle | | 10M requests a month | About 14,000: about ten times above | Self-hosting costs less than the teacher, but an untuned hosted Nano may cost less per answer. It then recommends the hosted fallback and says the comparison assumes the hosted model covers as many requests | | Very high volume | Over about 45,000 (estimate): above the hosted-fallback crossover too | Self-host |
```mermaid flowchart TB S["Student answers
per endpoint hour"] --> Q1{{"Above break-even?"}} Q1 -->|"no"| N1["Skip the 24/7 endpoint:
hosted fallback or scale to zero"] Q1 -->|"yes"| Q2{{"Above the hosted-
fallback crossover?"}} Q2 -->|"no"| N2["Teacher spend falls, but the
hosted fallback costs less"] Q2 -->|"yes"| Y["Self-host: the endpoint
wins on both counts"] class Q1,Q2 gate class N1 danger class N2 outline class Y ok ```
**Reading this diagram.** The cost curve has two thresholds, not one. Below break-even the GPU costs more than the teacher spend it saves. Between break-even and the crossover the student does save money against the teacher, but a pay-per-token hosted model would save more. Only above both does owning a GPU win. Ratchet's cost report places your traffic on this chart and writes the answer in words, including when the answer is "this loses money". Full method: [Cost and savings](/guides/cost-and-savings/).
## If you must know where your data went Every example Ratchet keeps carries its history. The ledger stores, for each training example, the teacher that wrote the answer and the license that permits training on it, after personal data has been redacted. - **Redaction before storage.** Emails, phone numbers, card numbers (checksum-validated), Indian identifiers such as UPI and GSTIN, and similar patterns are replaced before anything is stored. Names and street addresses are **not** caught yet. - **Opt-outs only reduce storage.** A request can ask not to be stored (`store: false`), and a key, a task or an end user can be set to never train. Nothing in a request can undo an opt-out. - **End users are never stored as typed.** An end-user identifier is kept only as a keyed hash scoped to the API key. - **Erasure is final.** An erased example leaves only a hash, so it cannot be stored again. Retention windows purge old rows on a schedule. - **Export.** Stored data can be exported as NDJSON with its provenance.
```mermaid flowchart TB R(["Request"]) --> O{{"Opt-out set?"}} O -->|"yes"| NS["Served,
never stored"] O -->|"no"| RD["Redact personal data"] RD --> V{{"Complete, valid
teacher answer?"}} V -->|"no"| NS V -->|"yes"| ST[("Ledger: text,
teacher, license")] ST --> ER["Erase: only
a hash stays"] ST --> RT["Retention
purge by age"] ST --> EX["Export
as NDJSON"] class O,V gate class ST store ```
**Reading this diagram.** Read it as the checks a request passes before it is allowed to become training data. Two gates can stop it: an opt-out, and a check that the teacher's answer is complete and valid. Whatever passes is redacted and stored with its provenance. From the ledger, three exits exist for the data owner: erase it, let it age out, or export it. The student's own answers are never stored as training data. Details: [Ledger](/guides/ledger/), [Security](/guides/security/), [Training on your data](/guides/training-on-your-data/).
## If you are regulated or careful You need to show a model was tested before it served real users, and that you can reproduce the evidence. | Evidence | What it says | Where | |---|---|---| | Signed bill of materials | For each adapter: every training source, its license, the base model's license, a fingerprint of the dataset, signed with Ed25519 and checkable offline | [Evidence](/guides/evidence/) | | Held-out exam | Thresholds are calibrated on half of the golden set and measured on the other half. A candidate must reach 95% precision and not regress against the live model, re-scored the same night | [Release gate](/guides/release-gate/) | | Certified precision | An exact one-sided lower bound on the precision of what the version serves, with the assumptions stated | [Evidence](/guides/evidence/) | | Staged release | Shadow, then canary at 10%, 50% and 100%, with automatic rollback when the 95% upper bound of audited precision falls below 92% | [Release gate](/guides/release-gate/) | | Savings receipts | One signed receipt per day and API client, chained, that a third party can verify | [Evidence](/guides/evidence/) | **What this does not show.** Precision means agreement with the teacher, not ground truth. A signed bill of materials is the operator's signed statement: it proves nothing changed after signing, not that the weights were trained on that data. See [Escalation and audits](/guides/escalation-and-audits/).
```mermaid flowchart TB C["Candidate adapter"] --> G{{"Golden exam passed?"}} G -->|"no"| X["Rejected"] G -->|"yes"| SH["Shadow: runs silently"] SH --> C10["Canary 10%"] C10 --> C50["Canary 50%"] C50 --> L["Live 100%"] C10 -.->|"audits slip"| RB["Automatic rollback"] C50 -.->|"audits slip"| RB class C,SH,C10,C50 student class G gate class X,RB danger class L ok ```
**Reading this diagram.** A new student only moves down when evidence allows it, and each canary stage can send it back. The first gate is the held-out exam, then a silent trial on real traffic, then growing shares of live traffic judged by teacher audits. Because every step leaves a record in the ledger, a reviewer can ask which version served which request and why it was promoted.
## If you build on open models - **License-clean teachers.** Ratchet distills only from teachers whose license allows it. Nemotron 3 Ultra is under [OpenMDW-1.1](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16), which places no restrictions on outputs (license page checked Oct 21, 2026). Closed models are refused as teachers in code. Not legal advice. - **The adapter you own.** What Night School produces is a LoRA adapter plus its signed lineage. The base model, Nemotron 3 Nano 4B, stays under the NVIDIA Nemotron Open Model License. - **Licenses travel with the adapter.** The bill of materials names the base model's license, and a `NOTICE.md` next to the weights lists the licenses of the base and each teacher. - **Open code.** Apache-2.0, with a mock mode that runs the whole loop with no keys and no GPU. See [Quickstart](/guides/quickstart/). ## What is and is not shown today
The loop is built and tested offline. Real teacher calls on Nebius Token Factory built the text-to-SQL data on Oct 22, 2026; since Oct 24, 2026 the gateway's own live mode runs at live.ratchet.dualmindlab.tech, where every request is a real Token Factory call. Ratchet's own adapter is not connected to the live gateway: Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nano 30B. The text-to-SQL and tool-calling adapters were trained on a borrowed GPU. Open question: Token Factory Terms of Service §5(f), asked of Nebius support on Oct 19, 2026. See [What is Ratchet](/guides/what-is-ratchet/) and [Results](/guides/results/).
--- # The business case Source: https://docs.ratchet.dualmindlab.tech/guides/business/ Ratchet is for teams that pay frontier-model prices for narrow, repeated work. It moves that work to a small open model with an adapter they own, learns only from teachers whose license allows it, and ships each new model only if it passes a gate. Savings have not been measured on live traffic yet. ## The problem - **LLM bills grow with traffic.** Enterprise spend on model APIs was $12.5B in 2025 ([Menlo Ventures](https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise/), report of Dec 9, 2025, read Oct 26, 2026). Much of what teams send is narrow and repetitive: classification, extraction, ticket triage, steps inside agents. - **The obvious fix is often not allowed.** Training a cheaper model on a closed model's answers breaks the terms of the largest vendors: [OpenAI Business Terms §3.3(e)](https://openai.com/policies/business-terms/) and [Anthropic Commercial Terms §D.4(a)](https://www.anthropic.com/legal/commercial-terms) (both read Oct 21, 2026) forbid using outputs to train competing models. Not legal advice. - **Doing it by hand is slow.** Logging, cleaning, fine-tuning, evaluating, deploying and watching for regressions are separate chores, and most tools cover one. ## Who it is for Teams with steady, repetitive LLM traffic on tasks a small model can learn: support triage, field extraction, routing, classification, text-to-SQL, or the many small steps inside an agent. It is a poor fit for open-ended work where every request differs, and for very low volume, where a GPU endpoint costs more than it saves (see [Why it matters](/guides/why-it-matters/) and [Cost and savings](/guides/cost-and-savings/)). ## The value
Every narrow, repeated request is a paid call at frontier prices, the bill grows with traffic, and closed vendors forbid training a cheaper model on their answers.
A small open model with your adapter takes over the work it is sure about, learns only from teachers whose license allows it, and ships each new version only if it passes a gate.
```mermaid flowchart TB T(["Steady, narrow
LLM traffic"]) ==> R["Ratchet"] R --> V1["Lower bill
as the student
takes over"] R --> V2["An adapter you own:
signed lineage,
open base"] R --> V3["License-clean:
closed teachers
refused"] R --> V4["Safe releases:
gate, canary,
rollback"] V1 --> M[("Measured per request,
by token and GPU-hours")] class V2 student class V4 gate class M store ```
**Reading this diagram.** One input, steady narrow traffic, produces four separate benefits, and the first is measured rather than assumed. Each request records what it cost and what the teacher alone would have cost, in two views: per token and by the GPU-hours actually paid. The other three benefits are about risk: who owns the adapter, whether its training data was allowed, and whether a bad version can reach everyone.
- **Cost falls as the student takes over.** Real savings depend on the traffic mix and on GPU utilization. Once the student serves, the teacher is paid for only the requests it escalates; the [Cost and savings](/guides/cost-and-savings/) guide shows the arithmetic. - **You own the result.** A LoRA adapter on an open base model, plus a signed bill of materials listing the teacher and license of every training example. - **Releases are gated.** A new student needs at least 95% precision on held-out data and may fall no more than 2 points behind the live version, then passes shadow and canary stages. See [The release gate](/guides/release-gate/). ## How it could make money The plan is an open-source core (Apache-2.0: gateway, ledger, Night School, release gate, dashboard) that anyone can self-host, plus a managed offering for teams that do not want to run the nightly loop and GPUs themselves. The managed idea is to charge a share of savings that Ratchet can verify, with no fee when there are none. The code already produces signed daily savings receipts for this (see [Evidence](/guides/evidence/)). This is a plan: nothing is sold today. ## Why now - **A deadline for fine-tuning customers.** OpenAI lists no new self-serve fine-tuning jobs after Jan 6, 2027 ([OpenAI deprecations](https://developers.openai.com/api/docs/deprecations), checked Oct 21, 2026). Teams with fine-tunes there need somewhere else to run them. - **Small models are being used for narrow tasks.** An NVIDIA Research paper estimates that about 40 to 70% of the LLM queries in the three open agent systems it studied (MetaGPT, Open Operator and Cradle) could go to specialized small models ([arXiv 2506.02153v3](https://arxiv.org/abs/2506.02153v3), revised Sep 22, 2026, read Oct 26, 2026). It is an estimate for those three systems, not a measurement. ## How it differs from other categories | Category | What it does well | What it does not do | Ratchet | |---|---|---|---| | Routers and gateways | Pick a model per request, log and govern traffic | Learn from the traffic; every call stays rented | Learns from the traffic and trains a student | | Fine-tuning and distillation services | Train a model when you start a job | Run by themselves every night, or record each example's teacher license | Retrains nightly and signs the lineage into the adapter | | Reference blueprints and tutorials | Show the pattern and its parts | Run unattended; promotion is left to a person | Gates, canaries and rolls back automatically |
```mermaid flowchart TB subgraph Prior["Existing categories"] A["Routing and
gateways"] B["Distillation
services"] C["Blueprints
and papers"] end A --> X{{"One loop that runs by itself"}} B --> X C --> X X --> O["Per-request escalation,
nightly retraining,
gated release,
signed license lineage"] class X gate class O ok ```
**Reading this diagram.** Each category on top is real and useful, and Ratchet uses their ideas. The claim is only about the combination: routing, retraining and release safety joined in one loop, with each example's teacher license recorded. The README's [prior art section](https://github.com/harshbhx/ratchet#prior-art-and-what-is-new) names the work it builds on.
## Honest limits - **Not proven on live traffic.** Real teacher calls on Nebius Token Factory built the text-to-SQL data (Oct 22, 2026), and the live gateway runs in live mode, but Ratchet's own adapter is not connected to it (one was served in speed tests and deleted; Night School ran once on Nebius and the gate rejected its student). Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nano 30B, so there are no measured savings yet. The measured adapters, two text-to-SQL rounds and one tool-calling round, were trained on a borrowed GPU ([Results](/guides/results/)). - **Early stage.** There is no customer and no revenue. - **Savings depend on the traffic mix.** Per-token savings against the cheapest tiers of closed models are small. The large savings come from moving traffic off top-tier models, and only for tasks a small model can learn. - **A Token Factory terms question is open.** A question about its Terms of Service §5(f) was sent to Nebius support on Oct 19, 2026; no answer received as of 29 Oct 2026. This is not legal advice. ## Related Also: [What is Ratchet](/guides/what-is-ratchet/). --- # A text-to-SQL adapter you can check Source: https://docs.ratchet.dualmindlab.tech/guides/text-to-sql/ A small open model learned to write SQLite queries from a bigger open model's answers, but only from answers that were actually run on a database, and its score comes from whole databases it never trained on, so anyone can recompute that score. Think of a new analyst joining a company. People ask them questions like "which five customers spent the most last quarter?", and they must write the query that answers it. A senior analyst (the teacher) writes practice questions and model answers. Before an answer goes into the study pile, someone **runs it** on the practice database. If it crashes, changes data, or returns nothing, it is thrown away. Then comes the exam. It uses databases the new analyst has never seen, and some of them come from industries that were never in the study pile. The exam is marked by running the new analyst's query and the senior's query on the same database and comparing the rows they return. Anyone who gets the answer sheet can mark the exam again and must get the same score. Databases, business questions and SQL answers. Kept only if it runs and returns 1 to 100 rows. On the kept rows, on a borrowed GPU. Scored by running queries, on a CPU. ## The problem Teams that keep data in a SQL database get questions all day. A large model can answer them, but two things hurt:
**Cost.** Every question is a paid call to a big model. **Exposure.** Every call carries your table and column names to someone else's endpoint.
A small model on a GPU you control fixes both: no bill per question, and the schema stays with you. But only if it is good enough.
The honest question is "good enough at what, measured how?" This page and the [Results](/guides/results/) page answer that in a way you can repeat. ## What was built - A **teacher**, Nemotron 3 Ultra on Nebius Token Factory, that invents SQLite databases, business questions and SQL answers. See [Licenses](/guides/licenses/) for why this teacher is allowed. - A **checker** that runs every SQL answer and keeps only the ones that pass. - A **student**: Nemotron 3 Nano 4B with a small LoRA adapter, trained on the kept rows. - An **exam** made of databases that are never in training, and a **scorer** that anyone can rerun on a CPU. - A **confidence gate**: the adapter answers only when it is confident, and the rest go to the teacher. ## How it really works 1. **The teacher invents a database.** For each industry (online retail, retail banking, clinic appointments, freight and logistics and 26 others) the teacher is asked for 3 to 5 SQLite tables with keys and 10 to 15 realistic rows each. The script runs the teacher's `CREATE` and `INSERT` statements in memory and drops the database unless it ends up with 3 to 7 tables of at least 5 rows each. 2. **The teacher writes questions and SQL.** For each database it writes questions a business user might ask, mixed across four levels: `simple` (one table), `join`, `aggregate` and `hard` (subquery, `HAVING`, three tables, `CASE` or a date calculation). 3. **Every SQL is run.** The SQL goes into a sandbox on that database. It is kept only if it is one read-only `SELECT` (or `WITH`) that finishes inside 2 seconds and returns 1 to 100 rows. Duplicate questions on the same database are dropped. Everything else is counted as rejected and thrown away. 4. **The exam is kept apart by database.** Training uses 25 industries. The exam uses whole databases that never appear in training: about two thirds come from industries that are in training (but a different, new database) and 57 of the 172 questions (33%) from two industries that are never in training (airline operations and pharmacy inventory). The script stops if one database ends up on both sides. 5. **The student is trained.** QLoRA on a borrowed GPU: the Nano 4B base loaded in 4-bit, LoRA rank 16, fp16, two epochs, with the loss counted on the SQL only. 6. **The exam is answered, then scored on a CPU.** The base model alone and the base model with the adapter each answer every exam question, and the raw answers are saved. A separate step runs each student query and the teacher's query on the same database and compares the rows. 7. **The confidence gate is set and measured on different rows.** Half of the exam rows (the even ones) set the confidence threshold. The other half (the odd ones) measure it. Nothing is graded on rows that tuned it. 8. **The score is recomputed by `verify`.** It reads the raw answers again and fails if any number in `results.json` differs, or if an input file is not the one the results were computed from.
**Where it was trained.** On a borrowed GPU (both published rounds). On 27 Oct 2026 the round-2 recipe was retrained once on Nebius AI Cloud ([adapter on Hugging Face](https://huggingface.co/harshbhx/ratchet-nemotron-nano-4b-text2sql-nebius)), as a Serverless Job on one preemptible L40S (new adapter SHA-256 `68b9ed06d984a2ec38a32b58090c246a56ae12ba65f3277f66b5d74227ea25d5`; scores 40.7% (70 of 172) on exam 1 and 34.5% (69 of 200) on exam 2, against 39.0% and 33.0% for the original adapter: the result reproduces (the row-by-row differences are within chance)). The teacher ran on Nebius: real calls to Nemotron 3 Ultra on Token Factory built this data on Oct 22, 2026.
```mermaid flowchart TB T["Teacher invents a database,
questions and SQL"] --> C{{"Run on its database:
one read-only SELECT,
under 2 seconds,
1 to 100 rows?"}} C -->|"no"| X["Thrown away
and counted"] C -->|"yes"| S{{"Which database?"}} S -->|"training: 25 industries"| TR[("Training set")] S -->|"exam: new databases,
2 industries never in training"| EX[("Exam set")] TR --> L["Train the adapter
QLoRA, borrowed GPU"] L --> M["Adapter answers
every exam question"] EX --> M M --> R["Run student and teacher SQL
on the same database,
compare the rows"] class T teacher class C,S gate class X danger class TR,EX store class L,M student class R ok ```
**Reading this diagram.** Read from the top. The teacher writes everything, but nothing it writes is trusted: the first check runs each query on the database it was written for, and only queries that pass reach the training or exam pile. The split between training and exam happens by database, not by row, so the student cannot have seen the exam's tables. The last box is the only place a number is made, and it is made by running queries, not by asking a model whether they look right. Each database has 3 to 7 tables of at least 5 rows; the questions mix four levels: simple, join, aggregate and hard.
### The sandbox A model's SQL is untrusted code, so it runs in a locked room. - **Read only.** An SQLite authorizer allows reading, `SELECT`, functions and recursive queries, and denies every other action. An `INSERT`, `UPDATE`, `DELETE`, `DROP` or `ATTACH` is refused by the database itself. - **Fast.** A progress handler stops any query that runs longer than 2 seconds. - **In memory.** Each database is built from its script in memory and thrown away. - **Small answers.** Training rows must return 1 to 100 rows, so a query that returns everything or nothing is not a good example. The Ratchet gateway has the same idea built in for tasks of format `sql`: [`ratchet/sqlcheck.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/sqlcheck.py) runs two answers on a fixture database and calls them the same when they return the same rows. The data scripts for this run are standalone and enforce the same rules for the same reason. ### How an answer is scored Two queries are the same answer when they return the same rows on the same database. This is called execution match. - Column names are ignored, because `total` and `sum_spent` can mean the same thing. - Decimal numbers are compared after rounding to 6 places. - Row order matters only when the teacher's query has an `ORDER BY`. Otherwise the rows are compared as a set, sorted. - A query that does not run, is not a read-only `SELECT`, or takes more than 2 seconds counts as wrong.
```mermaid flowchart TB E(["One exam question
and its database"]) --> TQ["Teacher SQL"] E --> SQ["Student SQL"] TQ --> TR["Run it: rows, and
does it order?"] SQ --> SR["Run it: rows
or a failure"] TR --> CMP{{"Compare"}} SR --> CMP CMP -->|"failed or
different rows"| W["Wrong"] CMP -->|"same rows, same order
if the teacher orders"| OK["Match"] class TQ teacher class SQ student class CMP gate class W danger class OK ok ```
**Reading this diagram.** Both queries are run on the same database, so the only thing that can make them match is giving the same rows. The order rule sits in the comparison: if the teacher's query sorts, the student must sort the same way, and if it does not, order is ignored. A student query that crashes counts as wrong.
### The confidence gate The adapter's confidence in an answer is the average probability of its tokens. The threshold is the lowest confidence at which the answers on the calibration half reach the target precision (95% unless set otherwise). On the measured half, an answer above the threshold is served by the student, and the rest would go to the teacher. Three numbers come out: **coverage** (how much the student answers), **precision when it answers** (how often those answers match the teacher) and an **estimated cost** against sending everything to the teacher. If no threshold reaches the target on the calibration half, the report says so. In that case the adapter would answer nothing on its own at that target.
```mermaid sequenceDiagram participant U as Question participant Student as Student participant G as Gate participant Teacher as Teacher U->>Student: schema and question Student->>G: SQL and confidence alt at or above the threshold G-->>U: student SQL else below the threshold G->>Teacher: same prompt Teacher-->>U: teacher SQL end Note over Student,Teacher: threshold set on one half, measured on the other ```
**Reading this diagram.** The student always tries first, and its confidence is the average probability of the tokens it wrote. The gate compares that confidence with the threshold. A confident answer goes straight back, and an unsure one is handed to the teacher, which costs more but is the safer answer. The note at the bottom is the honesty rule: the rows that set the threshold are never the rows that grade it.
## What it scored The numbers below are read from the runs' results files when the site is built. Nothing here is typed by hand. The headline is the round 2 adapter on two exams: exam 1, written by the teacher, and exam 2, written by a different model on industries that appear in neither the training data nor exam 1. Every question is scored strictly by running the query.
Who answeredExam 1 ( questions)Exam 2 ( questions)
Base model (untrained Nano 4B) of = of =
Teacher (Nemotron 3 Ultra) of = of =
Adapter, round 2 of = of =
At the 95% precision target the round 2 adapter answers of questions on exam 1's measurement half and of on exam 2's, so it does not replace the teacher yet. The ranges, round 1 and the details are on the [Results](/guides/results/) page. ### Round 1 The round 1 adapter on the half of exam 1 that measures the confidence gate (86 of 172 questions). Round 2 and the second exam are in the table above and on the [Results](/guides/results/) page. More on how to read them is on the [Results](/guides/results/) page, and the full chain of checks is on [Proof you can run](/guides/proof/). ## Reproduce it and check it Where the code lives: the data and training scripts are in the repository under [`scripts/text2sql/`](https://github.com/harshbhx/ratchet/tree/main/scripts/text2sql) (`gen_sql_data.py` and the training script), and the scoring script with the SQL checks is `scripts/measure_adapter.py`. They are tools you run by hand, not part of the gateway. The three commands that matter: ```bash # 1. Build the data (needs a Token Factory key; the teacher is paid per token, capped by --max-usd) python scripts/text2sql/gen_sql_data.py --out out/sql --max-usd 3 # 2. Train on a GPU and answer the exam # the training script in scripts/text2sql/ trains the adapter, then runs: scripts/measure_adapter.py generate ... # 3. Score and verify (CPU only; this is the part anyone can rerun) python scripts/measure_adapter.py report --raw raw.jsonl --golden golden.jsonl --dbs dbs.json \ --adapter ./adapter --out results.json python scripts/measure_adapter.py verify --raw raw.jsonl --results results.json \ --golden golden.jsonl --dbs dbs.json # exact arguments for the published runs: evidence/text2sql-round2/README.md, "Check it yourself" ``` `verify` prints `OK: every number in results.json was recomputed from raw.jsonl and matches`, or exits with a line starting `FAIL:` that says which file is wrong or which numbers differ. The report holds the SHA-256 of the raw answers, the exam, the exam databases and the adapter, so a swapped file is caught. ## What it does not show - **The data is synthetic.** A model wrote the databases and the questions. Real schemas are messier. - **The exam is small, so the ranges are wide.** The 95% range shown on each bar is the honest size of the doubt. - **Only SQLite.** Other dialects are not tested. - **"Matches the teacher" is not "is what the user meant".** The teacher is a model too and can be wrong. The score measures how well the student copies the teacher, on queries that run. - **Small slices mean little.** The split by difficulty or by industry has few questions per cell. - **The cost is an estimate.** Token counts are measured. Prices are inputs. The exams were scored without a student endpoint (the Nebius endpoint of 27 to 28 Oct 2026 ran only for speed tests), so no GPU-hour bill is included. See [Cost and savings](/guides/cost-and-savings/). - **Trained on a borrowed GPU.** Both rounds were trained on a borrowed GPU with the training script in `scripts/text2sql/`; the teacher ran on Nebius Token Factory. Night School's Nebius Serverless Job is separate: it trains with `ratchet.nightschool.train` through `deploy/train_job.sh`, and ran once by hand on 28 Oct 2026 on Nebius (a spot L40S; the golden gate rejected that student). ## The license chain | Piece | License or terms | Where it is recorded | |---|---|---| | Base model, Nemotron 3 Nano 4B | NVIDIA Nemotron Open Model License | The adapter's model card and its `NOTICE.md` | | Labels (the SQL answers) | Written by Nemotron 3 Ultra, OpenMDW-1.1 | The data manifest (`teacher_manifest.json`) | | Other models' outputs | None used: no closed-model output is in the data | Ratchet refuses closed teachers, see [Licenses](/guides/licenses/) | ## For engineers | What | Where | |---|---| | Data generation and sandbox | `gen_sql_data.py`: `build_db`, `run_select`, `make_db`, `HELD_OUT_DOMAINS`, the `teacher_manifest.json` it writes | | Training | The training script in `scripts/text2sql/` (QLoRA, rank 16, fp16, 4-bit nf4, completion-only loss), run on a borrowed GPU | | Scoring | `scripts/measure_adapter.py`: `report` runs each answer and the reference query on the same database and compares the rows; `verify` recomputes every number from the raw answers | | The second exam | `gen_exam2.py` (a different model writes databases, questions and reference queries; every query is run before it is kept), `teacher_selfcheck.py` (the teacher answers an exam) | | The numbers on the pages | `exam_numbers.py` scores the saved answers of both exams and writes the data the landing page, the judges' page and these docs read; a test fails when they drift | | In-gateway SQL tasks | [`ratchet/sqlcheck.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/sqlcheck.py): `SqlSpec`, `Sandbox`, `parse_query` | | Statistics used for the ranges | Wilson 95% interval in `measure_adapter.py`; exact intervals in [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py) | ## Common questions ### Why run the SQL instead of asking a model if it looks right? A model that judges SQL can be fooled by SQL that looks right and returns the wrong rows. Running it on a database gives a fact. A query either returns those rows or it does not. ### Why are some exam databases from industries the student never saw? To test whether it learned to write SQL or just learned the shape of its training databases. The results page splits the score by seen and unseen industry for this reason. ### Can a query in the training data change a database or read a file? No. The sandbox's authorizer refuses anything that is not a read, and a query over 2 seconds is stopped. The databases are built in memory from the teacher's script. ### Why only half the exam measures the confidence gate? If the same rows set the threshold and graded it, the result would look better than it is. Even rows set the threshold and odd rows measure it. ### Is this the same code the gateway uses? Not the same file. The gateway's `sql` task format in `ratchet/sqlcheck.py` follows the same rules (read-only, time limit, same rows). The scripts for this run are standalone so that anyone can rerun the scoring without the gateway. ### Where do the numbers come from? From the runs' files in `evidence/`, copied into the site at build time. The Round 1 panel reads the `results.json` that `measure_adapter.py report` wrote; the headline table reads the exam scores that `scripts/text2sql/exam_numbers.py` computes from the saved answers (its `--check` fails when the copy differs). ## Related - [Results](/guides/results/) - [Proof you can run](/guides/proof/) - [Confidence](/guides/confidence/) - [Licenses](/guides/licenses/) - [Night School](/guides/night-school/) --- # Results Source: https://docs.ratchet.dualmindlab.tech/guides/results/ This page shows what the text-to-SQL adapters (the headline) and the tool-calling adapter measured on exams they never trained on, read straight from the runs' results files. A report card is only useful if you know what test it came from. Here the test is a set of questions about databases the student has never opened. The score is "how often does the student's query return the same rows as the teacher's query". Every score comes with a range, like "somewhere between 64% and 82%". A small exam gives a wide range. That is not a flaw in the report. It is the honest size of the doubt. ## The headline: text-to-SQL, round 2 Two exams, scored strictly (the scores come from `src/data/exams.json`, which `scripts/text2sql/exam_numbers.py --check` recomputes from the saved answers): a query is right only when running it returns the same rows as the exam's stored query on the same database. Exam 1 was written by the teacher; exam 2 by a different model, on industries that appear in neither the training data nor exam 1.
Who answeredExam 1 ( questions)Exam 2 ( questions)
Base model (untrained Nano 4B) of = of =
Teacher (Nemotron 3 Ultra) of = of =
Adapter, round 2 of = of =
- **The training shows on both exams.** The round 2 adapter scores times the base model on exam 1 and times on exam 2. - **It does not replace the teacher yet.** At the 95% precision target it answers of questions on exam 1's measurement half and of on exam 2's. - **The exams do not show it is better than the teacher.** Each exam has one stored query per question, so a correct query with other columns counts as a miss, and the teacher's own score shows it. The sections below give the ranges, round 1 and the details. The tool-calling adapter's results (the second result) are in [Tool calling, round 3](#tool-calling-round-3) below. ## Round 1, on the measurement half

Status:

Round 1, the half of the exam that measures the confidence gate (86 of 172 questions). Round 2 and the second exam are below. ## What the first real run showed Same exam, same scoring, all questions. A query is right when running it returns the same rows as the teacher's stored query on the same database.
Who answeredOriginal promptWith one extra line (prompt v2)
Prompt v2 adds "Select only the columns the question asks for." to the end of every question, for both models. The ranges are 95% Wilson ranges. - **The adapter is 2.6 to 3.1 times the base model's score** under both prompts, and the ranges do not overlap. - **The adapter is in the same range as the teacher, not above it.** The teacher itself returns the stored query's rows only of the time, because each question has one stored query, written together with the question. A correct query with other columns or another sort order counts as a miss. The adapter learned the stored queries' style, so we do not claim it is better than the teacher. - **Prompt v2 changed nothing that holds up.** Row by row the adapter gained a few questions and lost a few. Treat the two prompts as one result: about a third of the questions for the adapter, about an eighth for the base model. - **The adapter does not yet replace the teacher.** At the 95% precision target only of test questions pass the confidence gate and of those are right, so the cost saved at that coverage is %. Token probability does not tell the adapter's right answers from its wrong ones. - **No exam question leaked into training:** of exam prompts, questions, stored queries or table-name sets appear in the training data. Training: teacher-labelled rows, steps, minutes on a ; teacher labels cost $. The files behind every number, and the command that recomputes them, are in `evidence/text2sql-round1` in the repository. ## What the second run showed Round 2 trained a second adapter on rows (the first run's 821 plus new teacher-labelled rows) and answered the same -question exam, scored the same way.
Who answeredRight95% range
Base model of =
Teacher answering the exam prompt of =
Adapter, round 1 (821 rows) of =
Adapter, round 2 ( rows) of =
- **Better than round 1, but the exam cannot call it settled.** Row by row, questions went from wrong to right and from right to wrong. The two ranges overlap, and a test on those changes gives p = , so some of the gain could be chance. - **The gain is on familiar industries.** On industries seen in training the adapter went from to of . On the two industries it never saw it stayed at of . - **By difficulty:** simple of , join of , aggregate of , hard of . - **The point estimate is above the teacher's, but we do not claim it is better than the teacher.** The exam has one stored query per question, written together with the question, and the two 95% ranges still overlap a little (adapter 32.0 to 46.4%, teacher 19.6 to 32.6%). - **The confidence gate did not improve.** At the 95% precision target of test questions pass, so the cost saved at that coverage is %. The adapter still does not replace the teacher. - **No exam question leaked into training:** of for prompts, questions, stored queries and table-name sets. Six generated rows that shared a question or a query with the exam were removed before training. Training: steps over epochs in hours on a , loss to ; the extra teacher labels cost $. The files and the recompute command are in `evidence/text2sql-round2` in the repository. ## A second exam, written by a different model The first exam was written by the same teacher that wrote the training labels, which flatters the teacher and anything trained to copy it. So we built a second exam: questions on 10 industries that appear neither in training nor in the first exam, with the databases and the questions written by Claude Sonnet 5.5, used for evaluation only. Every reference query was run and kept only if it worked, nothing overlaps with the training data, and its fingerprint was recorded before any model answered. The same four models answered it, scored the same strict way.
Who answeredExam 1 (172, written by Ultra)Exam 2 (, written by Claude Sonnet 5.5 for evaluation only)
Base model = = ()
Teacher answering the exam prompt = = ()
Adapter, round 1 (821 rows) = = ()
Adapter, round 2 ( rows) = = ()
- **The training effect shows up on an exam the adapter's data never touched.** The base model scores about the same on both ( and ), and the round 2 adapter is about times the base model on the new exam. - **Round 2 beat round 1 on both exams,** by 10 questions each time. On exam 2, questions went from wrong to right and from right to wrong (p = ). The direction repeats, but neither exam alone settles it. - **Both adapters score 5 to 6 points lower than on exam 1,** as expected on an exam from a different author. - **By difficulty on exam 2** (base, round 1, round 2): simple , , of ; join , , of ; aggregate , , of ; hard , , of . Join questions improved most, again. - **The teacher scores low here for a reason of shape, not skill.** Of its answers, return the right number of rows with a different number of columns, which the strict exam marks wrong. We do not claim the adapter is better than the teacher. - **The confidence gate still does not pass at the 95% precision target:** of questions on the measurement half, for either adapter. The adapter does not yet replace the teacher. ### Who wrote what Three sets of questions are involved, and they are kept apart. The training examples teach the adapter. Exam 1 and exam 2 only test it. Exam 1 was written by the teacher, the same model that wrote the training examples, which flatters the teacher and anything trained to copy it. Exam 2 was written afterwards by a different model from a different lab, for testing only. ### The scores as charts Each bar is how many of that exam's questions the model got exactly right. The line on a bar is its 95% range: with 172 to 200 questions a score is good to about 7 points either way. **Exam 1, written by the teacher** **Exam 2, written by a different model** ### Where it does well and badly Join questions improved most from round 1 to round 2. Simple questions did not improve, and the aggregate questions were already the adapter's strongest. **Exam 1** **Exam 2** ### Round 1 against round 2, question by question The exam, its fingerprint, every answer and the command that recomputes the scores are in `evidence/text2sql-exam2` in the repository. ## Retrained on Nebius AI Cloud On 27 Oct 2026 the round-2 recipe (the same 4,322 examples and settings) was retrained on Nebius AI Cloud, by hand, as a Serverless Job on one preemptible L40S. The retrained adapter scored **40.7%** (70 of 172) on exam 1 and **34.5%** (69 of 200) on exam 2, against 39.0% and 33.0% for the original adapter that was trained on a borrowed GPU. Row by row the two are statistically indistinguishable (6 up and 3 down on exam 1, 7 up and 4 down on exam 2), so the result reproduces; it is not an improvement. The whole Nebius effort, failed and interrupted attempts included, used 2.73 GPU-hours, about $2.16 at list prices (the Nebius bill decides). Evidence: [`evidence/text2sql-round2-nebius`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius). The adapter is on Hugging Face: [ratchet-nemotron-nano-4b-text2sql-nebius](https://huggingface.co/harshbhx/ratchet-nemotron-nano-4b-text2sql-nebius). ## Tool calling, round 3
Strict tool-calling scores on a hard exam and a BFCL exam: untrained model, tool-calling adapter and teacher, with the 95% range and exact counts
On a small screen the chart scrolls sideways; the table below has the same numbers.
A tool-calling adapter on Nemotron 3 Nano 4B (2,940 training rows, about $2.33 of GPU time), scored strictly on two exams. These are measured results on real models (Oct 24, 2026). | | Hard exam (150) | BFCL exam (300) | |---|---|---| | Untrained Nano 4B | 94 of 150 (62.7%, 95% range 54.7 to 70.0%) | 232 of 300 (77.3%, 95% range 72.3 to 81.7%) | | Teacher (Nemotron 3 Ultra) | 137 of 150 (91.3%, 95% range 85.7 to 94.9%) | 265 of 300 (88.3%, 95% range 84.2 to 91.5%) | | Tool-calling adapter | 136 of 150 (90.7%, 95% range 84.9 to 94.4%) | 241 of 300 (80.3%, 95% range 75.5 to 84.4%) | On the hard exam the adapter is above the untrained model: 90.7% against 62.7%, and the 95% ranges do not overlap. It is about the same as the teacher: 90.7% against 91.3% (lower by point estimate, but the 95% ranges overlap, so the exam cannot tell them apart). On the BFCL exam the adapter is about the same as the untrained model: 80.3% against 77.3% (higher by point estimate, but the 95% ranges overlap, so the exam cannot tell them apart). It is lower than the teacher: 80.3% against 88.3% (241/300 against 265/300; the 95% ranges overlap only slightly, so part of the gap could be chance). Hard exam: question by question against the untrained model, the adapter fixed 42 and broke 0 (exact sign test p < 0.0001). BFCL exam: question by question against the untrained model, the adapter fixed 15 and broke 6 (exact sign test p = 0.0784). Strict scoring: an answer counts only if it is one JSON object with the right tool and arguments; a reply with text around the JSON is wrong. Free-text arguments are a known weak spot in the training data: the strict scorer cannot agree on free text, so tools such as create_ticket (15 training rows), search_files, send_payment and search_inbox are thin. The hard exam uses 20 tools that were never in training. The adapter was trained on a borrowed GPU, on synthetic data. The hard exam (150 questions, 20 new tools) was written by Claude Sonnet 5.5 for evaluation only and is never training data; the second exam (300 questions) is drawn from the Berkeley Function Calling Leaderboard (Apache-2.0). Every number comes from `evidence/toolcall/results.json`, which `python scripts/toolcall_results.py --check` recomputes from the saved answers. ## How it really works 1. **A run ends with a `results.json`.** `measure_adapter.py report` reads the raw answers and writes every number into that file, with the SHA-256 of each input file. 2. **The file is copied into the docs site.** The site reads `src/data/sql-results.json` when it is built, so no number on this page is typed by hand. If the file says the run is pending, the panel above says so. 3. **`verify` recomputes it.** Anyone with the raw answers, the exam and the exam databases can run `measure_adapter.py verify` and get `OK` or a `FAIL:` line. See [Proof you can run](/guides/proof/).
```mermaid flowchart TB RAW[("raw.jsonl
teacher, base and
adapter answers")] ==> REP["measure_adapter.py report
on a CPU"] GOLD[("golden.jsonl, dbs.json
the exam and
its databases")] ==> REP REP ==> RES[("results.json
numbers and the
inputs' SHA-256")] RES ==>|"copied in"| PAGE["This page,
at build time"] RAW -.-> VER{{"measure_adapter.py verify:
every number recomputed
and equal?"}} GOLD -.-> VER RES -.-> VER VER -->|"yes"| OK["OK"] VER -->|"no"| BAD["FAIL: names the file
or the numbers"] class RAW,GOLD,RES store class VER gate class OK ok class BAD danger ```
**Reading this diagram.** The thick arrows make the numbers and the dotted ones check them, and both start from the same three files. The page is fed from the results file by a build step (it is copied to `docs-site/src/data/sql-results.json`), not by hand. The check does not trust the results file at all: it recomputes everything from the raw answers and compares.
## How to read the numbers - **Match with the teacher** is execution match: the student's query and the teacher's query return the same rows on the same database. It is not "the answer is right". - **The 95% range** is a Wilson interval. If the exam were drawn again many times, about 19 of 20 such ranges would contain the true rate. With a small exam it is wide, and two bars whose ranges overlap should not be called different. - **Base model against adapter** is the point of the work. Both answer the same questions with the same prompt and the same limits. The gap is what the adapter added. - **The split by difficulty and by industry** shows where the adapter helps. Each cell has few questions, so read the ranges before the bars. "Industry never seen in training" is the fairest test of whether it learned SQL or memorised. - **Coverage and precision when it answers** describe the confidence gate. Coverage is the share of questions the student answers on its own. Precision is how often those answers match the teacher. A higher threshold raises precision and lowers coverage. - **The estimated cost** is an estimate: token counts are measured, the prices are inputs (Nemotron 3 Ultra at $1 and $3 per million tokens in and out, and a student price you set), and the exams were scored without a student endpoint (the Nebius endpoint of 27 to 28 Oct 2026 ran only for speed tests, on other prompts), so no GPU-hour bill is in it. See [Cost and savings](/guides/cost-and-savings/). - **The data is synthetic.** The teacher wrote the databases and the questions. These numbers say how well the adapter copies the teacher on queries that run, not how well it serves your real users. It cannot tell you how the adapter does on your schema, in another SQL dialect, or on questions a person would phrase badly. Run your own held-out questions through `measure_adapter.py` to find out. ## For engineers | What | Where | |---|---| | The panel | `docs-site/src/components/SqlResults.astro`, reads `docs-site/src/data/sql-results.json` | | Layout test data | `docs-site/src/data/sql-results.sample.json` (made up, never shown unless copied over, and then bannered `SAMPLE DATA`) | | Feed a real run | `cp run/out/results.json docs-site/src/data/sql-results.json`, then rebuild | | The numbers' code | `measure_adapter.py`: `agreement`, `wilson`, `calibrate`, `build_report` | ## Common questions ### Why not just show one accuracy number? One number hides the sample size and the slices. The range shows how much to trust it, and the slice by unseen industry shows whether it generalises. ### Why is the exam only partly used for scoring? Only the confidence gate uses half the exam: the even rows set the threshold and the odd rows grade it, so coverage and precision when it answers (the Round 1 panel's gate figures, and the gate numbers for round 2 and exam 2) come from the graded half. The Round 1 panel's base-against-adapter bars are scored on that graded half too (86 of 172 questions). Every table on this page, the headline included, scores all the questions of each exam (172 and 200), and the base model, the teacher and each adapter answer the same questions. ### Can I check these numbers without a GPU? Yes. Scoring and `verify` run on a CPU from the raw answers. Only generating the raw answers needs a GPU. ## Related - [A text-to-SQL adapter you can check](/guides/text-to-sql/) - [Proof you can run](/guides/proof/) - [Cost and savings](/guides/cost-and-savings/) - [Evidence you can check](/guides/evidence/) --- # Quickstart Source: https://docs.ratchet.dualmindlab.tech/guides/quickstart/ Install the gateway, give it a Token Factory key, point any OpenAI SDK at it and read the `ratchet` block in each answer. Every call here is a real model call and costs cents. Prefer not to install anything? Use the live gateway at [live.ratchet.dualmindlab.tech](https://live.ratchet.dualmindlab.tech). ## The one-minute path You need Python 3.10 or newer, `git` and a Nebius Token Factory key. ```bash git clone https://github.com/harshbhx/ratchet.git && cd ratchet pip install -e ".[dev]" export NEBIUS_API_KEY=... # your Token Factory key export RATCHET_API_KEYS=me:your-local-api-key-here # live mode needs a key for /v1/*; pick your own mkdir -p data RATCHET_MODE=live RATCHET_DB=./data/ratchet.db uvicorn ratchet.gateway:app --port 8000 ``` The gateway serves on `http://localhost:8000`. With no student endpoint, the teacher (Nemotron 3 Ultra on Token Factory) answers every request. In a second terminal, send it a request with the OpenAI SDK: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="your-local-api-key-here") resp = client.chat.completions.create( model="ratchet", messages=[{"role": "user", "content": "I was charged twice this month."}], extra_headers={"X-Ratchet-Task": "ticket-triage"}, ) print(resp.choices[0].message.content) print(resp.model_extra["ratchet"]["served_by"]) # teacher, until a trained student is serving ``` Then open `http://localhost:8000/` (the story), `/scout` (the research agents, which also need `TAVILY_API_KEY`) and `/dashboard` (releases, versions, costs). ## Install ```bash git clone https://github.com/harshbhx/ratchet.git && cd ratchet python -m venv .venv && source .venv/bin/activate make install ``` ```powershell git clone https://github.com/harshbhx/ratchet.git; cd ratchet python -m venv .venv .venv\Scripts\Activate.ps1 pip install -e ".[dev]" ruff ``` Windows has no `make`, so run the command `make install` stands for (the last line). In PowerShell, set variables with `$env:NEBIUS_API_KEY="..."; $env:RATCHET_API_KEYS="me:your-local-api-key-here"; $env:RATCHET_MODE="live"; $env:RATCHET_DB="./data/ratchet.db"` (after `mkdir data`) before `uvicorn`. `make install` runs `pip install -e ".[dev]" ruff`. The `dev` extra brings pytest, the OpenAI Python SDK (used by the example above), `cryptography` for signed BOMs, PyYAML and Jinja2. The gateway itself needs only FastAPI, uvicorn, httpx, pydantic and anyio. ## Send your own request The gateway speaks the OpenAI chat-completions format, so any OpenAI SDK works. Name the task with the `X-Ratchet-Task` header (or `metadata.ratchet_task`); without one, the request belongs to the task `default`. These are the examples from the [API reference](/reference/api/#examples). ```bash curl -s http://localhost:8000/v1/chat/completions \ -H 'content-type: application/json' \ -H 'authorization: Bearer your-local-api-key-here' \ -H 'X-Ratchet-Task: ticket-triage' \ -d '{"model":"ratchet","messages":[ {"role":"system","content":"You triage customer support tickets. Reply with JSON only ..."}, {"role":"user","content":"I was charged twice for my Pro plan this month. Order #48213."}]}' ``` ```js const client = new OpenAI({ baseURL: "http://localhost:8000/v1", apiKey: "your-local-api-key-here" }); const r = await client.chat.completions.create( { model: "ratchet", messages: [{ role: "user", content: "Where is my order #48213?" }] }, { headers: { "X-Ratchet-Task": "ticket-triage" } }, ); ``` Install the SDK first with `npm install openai`. Every answer is a standard `chat.completion` plus a `ratchet` block. The fields you will read most: | Field | Meaning | |---|---| | `served_by` | `student` or `teacher` | | `student_version` | The version that answered, or `null` when the teacher did | | `confidence` | The student's confidence, if it was tried | | `escalated` | `true` when the student was tried and was below its threshold | | `cost_usd`, `all_teacher_cost_usd` | What this request cost, and what the teacher alone would have cost | | `stored` | Whether a teacher answer of this request became, or matched, a training example | | `request_id`, `row_id` | The request's id and its ledger row | ## Try Scout from the command line ```bash export TAVILY_API_KEY=... python -m ratchet.scout "What is LoRA fine-tuning?" --base-url http://localhost:8000/v1 --api-key your-local-api-key-here ``` Scout runs as an outside client against the gateway: a planner and a writer pinned to the teacher, a searcher, and many small triage and reader calls. It also takes `--api-key` and `--max-results` (default 4). See [Scout](/guides/scout/). ## Common questions ### Do I need an API key, a GPU or Node? A Token Factory key, yes: every answer is a real model call. No GPU: the gateway does no inference. Node is needed only to rebuild the web app (`make web`); the build is committed. ### Where is my data? In the SQLite ledger at `RATCHET_DB`. Stopping the gateway keeps it. ### Port 8000 is taken. What now? Use another port, for example `--port 8001`, in the `uvicorn` line and in the URLs and examples. ### How do I get a student? A student appears only after Night School trains and releases one on your ledger, which needs a GPU endpoint and a training job: see [Host it on Nebius](/guides/host-on-nebius/). Those scripts are tested offline with mock models; one retraining job, one endpoint speed test and one Night School (rejected by the golden gate, never made live) have been run by hand on Nebius. The measured results are on [Results](/guides/results/) and [Evidence](/guides/evidence/). ## Next steps - Send a verdict back on an answer, and see every route: the [endpoint map](/api/endpoints/). - Run the loop on your own tasks: [Training on your data](/guides/training-on-your-data/) and the task registry (see the [glossary](/guides/glossary/)). - Read the other tools you have now: [Command line](/api/cli/) and [environment variables](/api/environment/). - Embed the cascade in a Python program: [Python API](/api/python/). - Move to a GPU student: [Host it on Nebius](/guides/host-on-nebius/). ## Related --- # For judges Source: https://docs.ratchet.dualmindlab.tech/guides/for-judges/ You have five minutes. Send one real request on the live site, read the exam results, run one command against an evidence folder, and watch the video. This page puts the proof, every link and the honest limits in one place.
**Built on NVIDIA Nemotron and Nebius: every model call on the live site runs on Nebius Token Factory, the adapter was [retrained on Nebius AI Cloud](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius), and the student was served from a Nebius Serverless Endpoint [in a measured test](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius).** The live site is not connected to that endpoint.
Ratchet is an OpenAI-compatible gateway for license-clean distillation on open models. A small NVIDIA Nemotron 3 Nano 4B student answers first. When it is unsure, the request goes to the teacher, Nemotron 3 Ultra on Nebius Token Factory, and the answer is logged with the teacher's name and license; closed models are refused as teachers, in code. Night School is built to train a new student from those answers and release it only if it passes a statistical gate. Measured: a Nano adapter taught by Ultra goes from 12.8% to 39.0% on a text-to-SQL exam it never saw, for about $17 on record, and every score recomputes from the saved answers. ## Check it in 5 minutes 1. **Use it.** Open [ratchet.dualmindlab.tech/live](https://ratchet.dualmindlab.tech/live) and send a real request. A badge says who answered. Send a few at most; your text is not stored. 2. **Read the exam results.** The proof tabs below, or [Results](/guides/results/). 3. **Check one result.** Clone the repository and run one command against an evidence folder: ```bash git clone https://github.com/harshbhx/ratchet.git && cd ratchet && pip install -e ".[dev]" python scripts/text2sql/exam_numbers.py --check ``` 4. **Watch the video** (2:59): [video.ratchet.dualmindlab.tech](https://video.ratchet.dualmindlab.tech/?from=docs). For the console version of this page, with live chapters read from the gateway, open the [judges page](https://ratchet.dualmindlab.tech/judges). A one-page copy for GitHub is [JUDGES.md](https://github.com/harshbhx/ratchet/blob/main/JUDGES.md). ## The proof, with links Every number is read from saved files in the repository. Commands are from [Check it yourself](https://github.com/harshbhx/ratchet/blob/main/docs/VERIFY.md) and need no GPU and no key. | What | Result | Evidence | Check it | |---|---|---|---| | Text-to-SQL exams (22 to 24 Oct) | Adapter **39.0%** / **33.0%**; untrained 12.8% / 12.5%; teacher (Ultra) 25.6% / 15.0% | [round 1](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round1), [round 2](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2), [exam 2](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-exam2) | `python scripts/text2sql/exam_numbers.py --check` | | Tool calling (24 Oct) | **90.7%** hard exam (Ultra 91.3%), **80.3%** BFCL | [evidence/toolcall](https://github.com/harshbhx/ratchet/tree/main/evidence/toolcall) | `python scripts/toolcall_results.py --check` | | Scout runs (23 to 24 Oct) | Three recorded live runs, about $0.02 each | [evidence/scout](https://github.com/harshbhx/ratchet/tree/main/evidence/scout), [replay](https://huggingface.co/spaces/harshbhx/ratchet-scout) | replay in the browser | | What | Result | Evidence | Check it | |---|---|---|---| | Retrain on Nebius AI Cloud (27 Oct) | **40.7% / 34.5%**: the original result reproduces; about **$2.16** | [evidence/text2sql-round2-nebius](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius) | `python evidence/text2sql-round2-nebius/score_nebius.py verify` | | Endpoint speed tests (27 to 28 Oct) | Cold start about **7.2 to 7.6 min**, restart **7.1 min**; 16 users: p50 **1.6 s** / **1.2 s** | [evidence/student-endpoint-nebius](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius) | the folder's `results/` | | Night School on Nebius (28 Oct) | Gate **rejected**: 81.8% against 95% | [evidence/nightschool-nebius](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius) | the folder's README and logs | | What | Result | Evidence | Check it | |---|---|---|---| | Signed BOM (signed 29 Oct, after the run) | Ed25519 key `81b0a086250a81d0` | [evidence/keys](https://github.com/harshbhx/ratchet/tree/main/evidence/keys), [signed BOM](https://github.com/harshbhx/ratchet/blob/main/evidence/nightschool-nebius/ratchet_bom.signed.json) | `python -m ratchet.bom verify evidence/nightschool-nebius/ratchet_bom.signed.json --public-key Gh68hnDDlRSjBRxmc5dTo4l+rvqkftUsH7B2Nm72gpc= --notice evidence/nightschool-nebius/NOTICE.md` | | Prices verified (29 Oct) | Token Factory list prices match the config | [evidence/day1](https://github.com/harshbhx/ratchet/tree/main/evidence/day1) | read `day1.json` | | Live smoke run (23 Oct) | 20 real requests, about $0.01 | [evidence/live-smoke-2026-10-23](https://github.com/harshbhx/ratchet/tree/main/evidence/live-smoke-2026-10-23) | `python -m ratchet.liverun verify evidence/live-smoke-2026-10-23` | ## All the links ### Live and video ### Docs ### Code ### Evidence folders [`text2sql-round1`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round1) · [`text2sql-round2`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2) · [`text2sql-exam2`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-exam2) · [`toolcall`](https://github.com/harshbhx/ratchet/tree/main/evidence/toolcall) · [`text2sql-round2-nebius`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius) · [`student-endpoint-nebius`](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius) · [`nightschool-nebius`](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius) · [`keys`](https://github.com/harshbhx/ratchet/tree/main/evidence/keys) · [`day1`](https://github.com/harshbhx/ratchet/tree/main/evidence/day1) · [`live-smoke-2026-10-23`](https://github.com/harshbhx/ratchet/tree/main/evidence/live-smoke-2026-10-23) · [`scout`](https://github.com/harshbhx/ratchet/tree/main/evidence/scout) · [`hosted-nano`](https://github.com/harshbhx/ratchet/tree/main/evidence/hosted-nano) ### Hugging Face ### Nebius and NVIDIA: what runs where | Where | What runs there | |---|---| | Nebius Token Factory | Every model call on the live site: the teacher (Nemotron 3 Ultra), the hosted Nano for the random split, the judge | | Nebius AI Cloud | The round-2 retrain (27 Oct) and the Night School training job (28 Oct), as Serverless Jobs on a spot L40S; the Serverless Endpoint for the speed tests and the Night School adapter; a CPU VM for the Night School gateway; an Object Storage bucket. By hand; the endpoint and VM were deleted afterwards | | NVIDIA Nemotron | Ultra (teacher), Nano 4B (student, with a LoRA adapter), hosted Nano 30B (random split) | | Cloudflare | The story site, the docs and the live gateway, CPU only | | A borrowed GPU | Training of the two original adapters | ## Hackathon checklist | | Requirement | Link | |---|---|---| | ✓ | Nebius Token Factory and AI Cloud, with NVIDIA open models | [Where everything runs](/reference/where-it-runs/) | | ✓ | Video, 2:59 | [video](https://video.ratchet.dualmindlab.tech/?from=docs) | | ✓ | Public repository with license and README | [repository](https://github.com/harshbhx/ratchet), [Apache-2.0](https://github.com/harshbhx/ratchet/blob/main/LICENSE), [README](https://github.com/harshbhx/ratchet/blob/main/README.md) | | ✓ | Working URL | [ratchet.dualmindlab.tech](https://ratchet.dualmindlab.tech), [/live](https://ratchet.dualmindlab.tech/live) | | ✓ | Feedback on Nebius and NVIDIA | [README, Feedback](https://github.com/harshbhx/ratchet#feedback-on-nebius-and-nvidia) | The track is the one chosen on the Devpost page. ## What is real and what is not yet
```mermaid %%{init: {"flowchart": {"rankSpacing": 24}}}%% flowchart TB subgraph Real["Real"] direction LR R1["Token Factory calls,
Scout search"] ~~~ R3["Adapters, trained
on a borrowed GPU"] ~~~ R4["Code and tests,
no GPU needed"] ~~~ R5["Nebius retrain, endpoint
speed tests, Night School:
each ran once by hand"] end subgraph Not["Not done"] direction LR N1["Nightly loop on
live traffic"] ~~~ N3["Live site connected
to a student endpoint"] ~~~ N2["Measured
live savings"] end Real ~~~ Not class R1,R3,R4,R5 ok class N1,N2,N3 pending class Real okZone class Not pendingZone ```
**Reading this diagram.** Two rows: what runs for real (green) and what has not been done (amber, dashed). The teacher side runs for real, and one hand-run Nebius training job, one hand-run speed test of the student and one hand-run Night School (a CPU VM, a spot L40S job and an endpoint, rejected by the golden gate) have run (green); the nightly loop on live traffic, the live site's connection to a student endpoint and a measured saving have not (amber); the one-command VM kit is scripted and tested offline. The two notes below say the same in words.
**Real, as of 29 Oct 2026.** Built on NVIDIA Nemotron and Nebius: every model call on the live site runs on Nebius Token Factory, the adapter was [retrained on Nebius AI Cloud](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius), and the student was served from a Nebius Serverless Endpoint [in a measured test](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius). Real calls to the teacher, Nemotron 3 Ultra on Nebius Token Factory, built the text-to-SQL data on Oct 22, 2026. The original published adapters were trained on a borrowed GPU, and their results are read from their own results files. On 27 Oct 2026 the round-2 text-to-SQL recipe (the same 4,322 examples and settings: 4-bit QLoRA, rank 16, 2 epochs, the same library versions) was retrained on Nebius AI Cloud, by hand, as a Nebius Serverless Job on one preemptible (spot) L40S, with checkpoints in a Nebius Object Storage bucket so it resumed after spot interruptions and with the Mamba kernels built for the L40S. Training loss went from 0.346 to 0.088 and the new adapter's SHA-256 is `68b9ed06d984a2ec38a32b58090c246a56ae12ba65f3277f66b5d74227ea25d5`. It scored 40.7% (70 of 172) on exam 1 and 34.5% (69 of 200) on exam 2, against 39.0% and 33.0% for the original adapter: the result reproduces (the row-by-row differences are within chance); training took 28.6 minutes of GPU time (58.7 minutes wall clock across two spot interruptions); the whole Nebius effort, failed and interrupted attempts included, used 2.73 GPU-hours, about $2.16 at list prices (the Nebius bill decides) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)). On 27 to 28 Oct the student was served by hand, in two short sessions, on a Nebius Serverless Endpoint (one spot L40S) for speed tests: cold start about 7.2 minutes to ready; with 16 users at once the median answer took 1.6 s (SQL adapter) and 1.2 s (tool-calling adapter); about $0.67 of GPU time across both sessions; speed only, no accuracy scored; the endpoint was deleted afterwards and the live gateway is not connected to it ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). On 28 Oct 2026 Night School ran end to end once on Nebius, by hand: the gateway on a Nebius CPU VM, the training job on a spot L40S, the adapter served from a Nebius Serverless Endpoint. The golden gate rejected it (81.8% precision against the 95% target), so it was never made live. The live site's own gateway is still on Cloudflare and is not connected to that endpoint, and the loop has not run nightly on live traffic ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). The code and its roughly 1,900 tests run with no GPU. Since Oct 24, 2026 the live gateway runs in live mode, and every request there is a real Token Factory call.
**Not yet.** Ratchet's own adapter is not connected to the live site, and no trained version exists on that ledger yet. On the live site a random 40% of eligible requests are answered by the hosted, untuned Nemotron 3 Nano 30B on Token Factory, a quarter of those re-checked by Ultra, and the pages label them as a random split. With a trained version and no GPU endpoint, student calls would go to that hosted model and the pages would say so with a badge. Night School has not run nightly on live traffic: it ran once, by hand, on Nebius, and the golden gate rejected that student. Live savings numbers do not exist yet and are not shown.
### Measured on Nebius: Night School, 28 Oct 2026
**Measured on Nebius, 28 Oct 2026, by hand, once.** Night School ran end to end once on Nebius, by hand (28 Oct 2026): the gateway on a Nebius CPU VM, the training job on a spot L40S, the adapter served from a Nebius Serverless Endpoint. The golden gate rejected it (81.8% precision against the 95% target), so it was never made live. The live site's own gateway is still on Cloudflare and is not connected to that endpoint, and the loop has not run nightly on live traffic. No saving is claimed.
| What | Result | |---|---| | Gateway | A Nebius CPU VM in `eu-west1` (2 vCPU), in live mode | | Data | 4,322 existing Ultra-labelled text-to-SQL rows imported (no new teacher calls, no exam rows): 3,864 to train on, 458 held out as the golden set | | Training | A spot L40S Serverless Job: 2 epochs, 966 steps, 1,478 s; loss about 0.15 to 0.08; token accuracy about 0.975 | | Serving | The adapter (SHA-256 `dc9ad4b9...`) hot-loaded into a Nebius Serverless Endpoint | | Golden gate | SQL compared as text: agreement 17.9%; at the gate's threshold the student would answer 4.8% of questions with 81.8% precision (9 of 11) against the 95% target. **Rejected**, never shadow, canary or live | The caveats, all of them, are in the [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius): four job submissions (an accidental on-demand job that was cancelled, a missing `einops`, a save that failed with `EPERM` and whose weights were recovered, then the job that saved); the night's own evaluation timed out at 15 s, so the gate result comes from running the eval and the gate again on the saved adapter with a 120 s timeout, through a one-off driver; five problems fixed by hand on the VM, now fixed in the repository (spot by default, `einops` pinned, the adapter saved locally and then copied, a clear stop when `aws` is missing, a separate student timeout for the night); and cost estimates (about 2.7 GPU-hours, at most about $4 at list prices) where the Nebius bill decides. Every answer on Scout and the dashboard carries a badge: | Badge | Means | |---|---| | | Nemotron 3 Nano 4B with the Night School adapter | | | The untuned Nemotron 3 Nano 30B on Token Factory, used while the GPU endpoint is off. Not covered by the adapter's precision certificate | | | Nemotron 3 Ultra on Nebius Token Factory | Scout has a daily run cap. If it is used up you get a polite refusal; evaluators receive an access code with their instructions, never on a public page. ## What it cost All of these are estimates recorded by the scripts, not invoices; the Nebius bill decides. | Item | About | |---|---| | Labelling by the teacher, round 1 / round 2 | $2.87 + $9.18 | | Nebius retrain, 27 Oct (2.73 GPU-hours, failed attempts included) | $2.16 | | Endpoint speed tests, 27 to 28 Oct | $0.67 | | Night School run, 28 Oct | about 2.7 GPU-hours, at most about $4 at list prices | | One real request on the live site | $0.0002 | | One Scout run | $0.02 | Request and Scout costs are from the [29 Oct day-1 check](https://github.com/harshbhx/ratchet/tree/main/evidence/day1). The GPU-hour rate used in estimates does not appear on the price list Token Factory serves and is unverified. ## Verify a result with two commands On the repository as cloned, these two commands recompute the published exam numbers from the saved answers: ```bash python scripts/text2sql/exam_numbers.py --check python scripts/toolcall_results.py --check ``` For a real signed bill of materials from your own gateway, `python -m ratchet.stats check bom.json --public-key ` recomputes its certificate from the signed counts, and `python -m ratchet.bom verify bom.json --public-key ` checks only the signature and the data lineage. The Night School BOM, signed on 29 Oct 2026, is published in the repository (see the proof tabs above). The [proof page](/guides/proof/) walks through every check; only the exam checks run on real files, and the others on documents the tests write with mock models. ## What to look at in the code | Claim | Where it is enforced | |---|---| | Only open teachers are distilled; closed models are refused | [`ratchet/licenses.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/licenses.py) `require_distillable` | | The student answers first and escalates when unsure | [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) `Ratchet.handle` | | Releases go shadow, canary 10%, 50%, 100%, with rollback | [`ratchet/release.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/release.py) `ReleaseGate` | | Each adapter has a signed bill of materials and a certificate | [`ratchet/bom.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/bom.py), [`ratchet/certificate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/certificate.py) | | The cost report says when self-hosting does not pay | [`ratchet/cost.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/cost.py) `cost_report` | ## Honest limits - **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)). - **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)). - **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). - **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). - **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate. - **The nightly schedule on live traffic.** Night School has not run nightly on live traffic; it runs by hand today and the timer is scripted. - **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline. Run every script with `DRY_RUN=1` first. Other limits: - No live savings figure exists yet; the measured results are the exam scores on [Results](/guides/results/). The original adapters were trained on a borrowed GPU. - Precision means agreement with the teacher, not ground truth. - The redactor is pattern-based: names and street addresses are not caught yet. - One gateway process per ledger (SQLite). - Token Factory Terms of Service §5(f) ("competing service") is an open question, asked of Nebius support on Oct 19, 2026; no answer received as of 29 Oct 2026. Not legal advice. ## Where to read more --- # The big picture Source: https://docs.ratchet.dualmindlab.tech/guides/big-picture/ Ratchet is , plus a few Nebius services it calls: Token Factory for the teacher, a GPU endpoint for the student, a nightly training job and a storage bucket that carries data and adapters between them. Picture the whole restaurant from above. - **The pass** (the gateway) is where every order arrives and every plate leaves. It decides who cooks each order. It never cooks anything itself, so it needs only a small room. - **The apprentice's station** (the student endpoint) is a rented kitchen with a stove, the GPU. The apprentice cooks there, reading from the **notebook of techniques** (the LoRA adapter you own). - **The agency** (Nebius Token Factory) sends the **head chef** (Nemotron 3 Ultra) for each hard dish and charges per plate. It also has a food critic (the judge, Nemotron 3 Super) and a spare, untrained cook (the hosted fallback, Nemotron 3 Nano 30B) for when the apprentice's station is closed to save money. - **The recipe book** (the ledger) sits at the pass. Every order, every dish the head chef cooks and every exam result is written in it. - **The storeroom** (the Object Storage bucket) holds the night's study pack and every notebook the apprentice ever wrote. - **Night school** (a Serverless Job on a GPU) opens once a night, reads the study pack from the storeroom and writes a new notebook back into it. - **The doorman** (Caddy, in the Nebius VM kit) checks the front door and keeps the connection to guests private (HTTPS). Three clocks run the place: one ticks for every order, one every few minutes to check how the newest notebook is doing, and one once a night for school. ## How it really works The design in one picture. It shows the intended flow; the original adapters were trained by hand on a borrowed GPU, the round-2 recipe was retrained once by hand on Nebius AI Cloud, and teacher calls run on Nebius Token Factory.
Architecture: your app calls the gateway, a small student model answers first, a teacher answers when the student is unsure, answers are logged in a provenance ledger, Night School trains a new adapter from the ledger, and release gates decide whether it becomes the student.
The detailed diagram, with the services each piece runs on:
```mermaid flowchart TB app(["Your app · any OpenAI SDK"]) ==>|"POST /v1/chat/completions"| caddy["Caddy in the VM kit
HTTPS, edge guard"] caddy ==> gw subgraph gw["Gateway · one process, no GPU"] direction LR router["Cascade
Ratchet.handle"] --> ledger[("SQLite ledger")] ledger <--> gate{{"Release gate"}} end subgraph tf["Token Factory, per token"] direction TB teacher["Nemotron 3 Ultra
teacher"] teacher ~~~ others["Judge: Super
Fallback: Nano
30B, untuned"] end subgraph ep["Serverless Endpoint"] student["Nano 4B + LoRA
on vLLM"] end subgraph ns["Night School"] direction TB night["ratchet.live night
cron, on the host"] --> job["Serverless Job
LoRA training"] end bucket[("Object Storage
bucket at /data")] gw ==>|"escalate, audit"| tf gw ==>|"3 samples"| ep gw -->|"rows"| ns ns -->|"adapter"| bucket ep ---|"loads adapters"| bucket class app,router,night,job neutral class caddy,others outline class gate gate class ledger,bucket store class student student class teacher teacher class ep studentZone class tf teacherZone ```
**Reading this diagram.** Start at the top. The thick arrows are the request path: your app's call passes Caddy (only in the Nebius VM kit), then the cascade asks the student endpoint for 3 samples, and the teacher on Token Factory when the student is unsure, an audit is drawn or no student may answer (a cold start, a pinned task). Every request writes one row to the SQLite ledger; the release gate reads the audits there and writes version states back. At night, Night School reads the rows, a Serverless Job trains a new adapter into the Object Storage bucket, and the endpoint loads adapters from that bucket at `/data`. Not drawn, to keep it readable: the night also loads the candidate into vLLM, grades it there (the judge scores free-text rows) and hands it to the release gate. The hosted fallback answers student calls only when no endpoint is set.
  • The student: Nano plus your adapter
  • Rented per token
  • Keeps data
  • Decides releases
  • Optional or fallback
  • Request path
### The components Sends a chat request with a task name. Admission checks, then the student first. Answers when it is confident. Answers the rest. | Component | What it does | Where it runs | Code | |---|---|---|---| | Your app | Sends OpenAI-style chat requests, with a task name | Anywhere | Any OpenAI SDK | | Gateway | OpenAI-compatible API, admission checks, pages, Scout endpoints, the release gate's tick | One process, one uvicorn worker. The live gateway runs on Cloudflare Workers with Containers; the Nebius VM kit (`2vcpu-8gb`) is scripted and tested offline, ready to run | [`ratchet/gateway.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/gateway.py) (`Runtime`) | | Cascade (router) | Student first, confidence, teacher on escalation, audits, shadow, cost | Inside the gateway | [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) (`Ratchet.handle`) | | Student | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` with the LoRA adapter you own, served by vLLM with runtime LoRA loading | A Nebius Serverless Endpoint (L40S by default) | [`deploy/student_endpoint.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/student_endpoint.sh) | | Teacher | `nvidia/Nemotron-3-Ultra-550b-a55b` answers escalations, cold starts, pinned tasks, audits and arbitrations | Nebius Token Factory | `RATCHET_TEACHER_MODEL` | | Judge | `nvidia/nemotron-3-super-120b-a12b` scores free-text golden rows in JSON mode (a diagnostic, `judge_pass_rate`) | Nebius Token Factory | `RATCHET_JUDGE_MODEL` | | Hosted fallback | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B`, untuned, plays the student when no endpoint is configured | Nebius Token Factory | `RATCHET_FALLBACK_STUDENT` | | Ledger | Requests, examples with provenance, versions, events, failures, receipts | One SQLite file on the gateway's local disk | [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py) | | Night School | Builds the train and golden sets, trains, evaluates, signs a bill of materials, calls the gate | Driven by cron on the gateway host; training on a Nebius Serverless Job (`gpu-l40s-a`, `1gpu-8vcpu-32gb`) with TRL and PEFT | [`ratchet/live.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/live.py), [`ratchet/nightschool/`](https://github.com/harshbhx/ratchet/tree/main/ratchet/nightschool) | | Bucket | Training rows, adapters, bills of materials | Nebius Object Storage, mounted at `/data` in the job and the endpoint | `RATCHET_BUCKET_ID`, `RATCHET_SYNC_CMD` | | Release gate | Golden gate, shadow, canary 10%, 50%, 100%, rollback | Inside the gateway (and the cron tick) | [`ratchet/release.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/release.py) (`ReleaseGate`) | | Web app | `/` (the story), `/scout`, `/dashboard`; no-build pages at `/lite*` | Served by the gateway from `ratchet/web/dist` | [`web/`](https://github.com/harshbhx/ratchet/tree/main/web) | | Scout | A multi-agent research app whose calls go through the cascade | Inside the gateway; Tavily for web search | [`ratchet/scout/`](https://github.com/harshbhx/ratchet/tree/main/ratchet/scout) | | Caddy | Automatic HTTPS, security headers, an optional token guard on `/v1/*` and `/ratchet/admin/*` | Next to the gateway in the Nebius VM kit (Docker Compose) | [`deploy/vm/Caddyfile`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/Caddyfile) | Token Factory listed no fine-tuning or LoRA serving for Nemotron when this design was chosen (research notes, Oct 19, 2026), and Nebius's [post on post-training in Token Factory](https://nebius.com/blog/posts/post-training-in-token-factory) names no Nemotron model to fine-tune (read Oct 26, 2026), so the student is trained and served on Nebius AI Cloud. The gateway itself does no inference, which is why a 2-vCPU machine is enough. ### How the pieces connect 1. **Your app calls the gateway** at `POST /v1/chat/completions` with an API key and a task name (`X-Ratchet-Task`). The gateway checks the body size, the key, rate limits, the request itself, the task and the quotas before any model is called. 2. **The cascade calls the student endpoint** over HTTPS with the OpenAI API. The model name it sends is the version's adapter name (`student-vN`), because vLLM serves each LoRA adapter under its own name. Three samples come back in one call. 3. **When the student is unsure, or for an audit, the cascade calls the teacher** on Token Factory. Each teacher answer that is complete and valid goes into the ledger as a training example, with the teacher's model id and license, after PII redaction. 4. **Every request writes one row** to the ledger: who served it, the route, confidence, cost and the all-teacher baseline. No prompt text is kept in that row; text is kept only in training examples, redacted. 5. **After requests, the release gate ticks**: it reads the audits and shadow verdicts from the ledger and moves versions between states (shadow, canary 10%, 50%, live, rolled back). 6. **At night, `python -m ratchet.live night` runs Night School**: it writes `night/train.jsonl` locally, uploads it to the bucket with `RATCHET_SYNC_CMD`, starts the training job with `RATCHET_TRAIN_CMD` (default `bash deploy/train_job.sh {version}`) and waits. 7. **The training job reads the bucket at `/data`** and writes the adapter to `/data/adapters/student-vN/`, with `NOTICE.md` and its training metadata beside the weights. 8. **The gateway host tells vLLM to load the new adapter** with `POST /v1/load_lora_adapter`, so no endpoint restart is needed. Night School then scores it on the golden set (the judge scores free-text rows), signs its bill of materials and hands the candidate to the gate. Last, the night writes the bill of materials and `NOTICE.md` into the adapter's folder and syncs the bucket again. 9. **The pages read the same ledger** through `GET /ratchet/stats`, `GET /ratchet/cost` and the other dashboard endpoints. Scout calls `Ratchet.handle` in the same process, once for each of its model calls. - **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)). - **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)). - **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). - **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). - **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate. - **The nightly schedule on live traffic.** Night School runs by hand today; the timer is scripted. - **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline. - **Measured savings.** They come from a published live run; none is published yet. The `nebius ai` commands in `deploy/` follow the CLI reference. Run every script with `DRY_RUN=1` first. ### What a request leaves behind Every request leaves exactly one row of metadata. Text is kept only when a teacher answer passes every check below, and then only after PII redaction.
```mermaid flowchart TB req(["A chat request"]) --> ans{{"A teacher answer?"}} req --> row[("Request row, always
metadata only")] ans -->|"no: the student served"| x1["Nothing to store"] ans -->|yes| only{{"Pinned, tool or
teacher-only?"}} only -->|yes| x2["Never stored"] only -->|no| opt{{"A storage opt-out?"}} opt -->|yes| x3["Not stored
the row says which"] opt -->|no| valid{{"Complete and valid?"}} valid -->|no| x4["Served, not stored
skip_example event"] valid -->|yes| red["PII redaction"] red --> ex[("Training example
teacher, license, split")] class req,red neutral class row,ex store class ans,only,opt,valid gate class x1,x2,x3,x4 outline ```
**Reading this diagram.** The short branch to the cylinder always happens: one request row with the route, cost, confidence and ids, never the prompt or the answer. The API key appears in it by name only, and an end user only as a keyed hash. The chain of checks down the middle decides whether any text is kept; each dashed box is a way out with nothing stored. A teacher answer exists after an escalation, a cold start, an audit or a shadow arbitration; the student's own answers are never stored. Pinned tasks, tool requests and other teacher-only requests are never stored (the one exception is an over-length request on a task that sets `store_ineligible`). Any storage opt-out stops it next: `store: false`, `X-Ratchet-Store: none`, `metadata.ratchet_store`, a key with `store=0`, a no-store task, a no-train end user or zero retention. Then the answer must be complete and valid under the task's spec. Only then are the messages, the stored conditions and the answer redacted and written as a training example with the teacher's model id and license, in the golden or the train split. An erased example's hash is refused from then on.
### The live runtime `RATCHET_MODE=live` builds the runtime in `Runtime.reset` in `ratchet/gateway.py`:
```mermaid flowchart TB l1[("SQLite file
at RATCHET_DB")] l1 --> l2["Teacher on
Token Factory"] l2 --> sb{{"RATCHET_STUDENT_BASE
set?"}} sb -->|yes| l3["Your adapter on
your vLLM endpoint"] sb -->|no| l4["Hosted fallback
Nano 30B, untuned"] class sb gate class l1 store class l2 teacher class l3 student class l4 outline ```
**Reading this diagram.** The runtime keeps a real ledger file and calls the teacher on Token Factory. Its student is your adapter on your own vLLM endpoint when `RATCHET_STUDENT_BASE` is set, and otherwise the untuned Nano 30B on Token Factory, drawn dashed because it is a stand-in, not the adapter you own.
| | Live mode | |---|---| | Ledger | SQLite file at `RATCHET_DB`, day 1 starts at the first request | | Teacher and student | Token Factory teacher; vLLM endpoint, or the hosted fallback when `RATCHET_STUDENT_BASE` is empty | | Night School | `python -m ratchet.live night` from cron, with a real training job (tested offline; ran once by hand on Nebius on 28 Oct 2026 and the golden gate rejected that student; not yet run on live traffic) | | Audit rate | `RATCHET_AUDIT_RATE`, default 0.05 | | Canary audits needed per stage | 20 (`ReleasePolicy.min_audits`) | | Gate tick from requests | At most once every 5 s (`RATCHET_TICK_INTERVAL_S`) | | Audits and shadow | Run after the response on 4 sidecar workers (`RATCHET_SIDECAR_WORKERS`) | | API keys | Required (`RATCHET_API_KEYS`), unless `RATCHET_ALLOW_UNAUTHENTICATED=1` with a guard in front | | Scout search | Tavily when `TAVILY_API_KEY` is set | With no `RATCHET_STUDENT_BASE`, a trained version's student calls go to the untuned hosted Nano 30B, held to the global threshold `RATCHET_CONFIDENCE_THRESHOLD` (0.80) instead of the version's calibrated one. Those answers are still reported as `served_by: student`; the response's `model` field names the model that wrote them, and the pages label them "Nano 30B · hosted", not as your adapter. ### Three loops at three speeds
```mermaid flowchart TB R["Every request
the cascade"] ==>|"rows, audits, examples"| L[("Ledger")] L ==>|"audits, shadow verdicts"| G{{"Seconds to minutes
the gate tick"}} G ==>|"version states"| R L -->|"train and golden rows"| N["Every night
Night School"] N -->|"a new candidate"| G class R,N neutral class L store class G gate ```
**Reading this diagram.** The thick loop turns fastest: requests write rows, the gate tick reads the audits and shadow verdicts in them, and the version states it writes decide which student the next request meets. The thin path turns once a night: Night School reads the training and golden rows from the same ledger and hands a new candidate to the gate. In live mode cron can also run the tick, so a quiet gateway still moves its releases.
| Loop | How often | What it does | Entry point | |---|---|---|---| | The request | Every call; seconds at most | Pick a version, sample the student, serve or escalate, maybe audit, maybe shadow, log one row | `Ratchet.handle` | | The gate tick | After requests (live: at most every 5 s) and from cron every 5 minutes | Shadow decisions, canary 10% to 50% to live, rollback, alerts | `ReleaseGate.tick`, `python -m ratchet.live tick` | | The night | Once a night (02:30 on the VM kit's cron line) | Skip if a release is in flight, training is paused or fewer than 50 new examples arrived; otherwise build, train, evaluate, gate | `run_night`, `python -m ratchet.live night` | Two more jobs run on the side: `python -m ratchet.live seal` seals the savings receipts of each ledger day that ended (hourly), and `deploy/vm/backup.sh` backs up the ledger (daily at 03:10 UTC). In the VM kit only the backup is switched on by default; the night, tick and seal lines in `/etc/cron.d/ratchet` are commented out until live mode works by hand. The loops share only the ledger. The gate reads what requests wrote, and Night School does not train while a version is in shadow or canary, so a release finishes or fails before the next candidate is trained. ## For engineers **`Runtime` wires it all.** [`ratchet/gateway.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/gateway.py) `Runtime.__init__` checks the whole configuration first (`config.check`, every error in one `ConfigError`), then `Runtime.reset` builds the ledger, the teacher provider, the student resolver (in live mode: the endpoint and `v.adapter_uri`, or the hosted fallback model), the `Ratchet` cascade and the `ReleaseGate`. **Settings that shape the picture** (defaults from [`ratchet/config.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/config.py); all of them in the [configuration reference](/reference/configuration/)): | Variable | Default | Meaning | |---|---|---| | `RATCHET_MODE` | `mock` | `mock` or `live` | | `RATCHET_DB` | `ratchet.db` | Ledger file (live mode) | | `RATCHET_TEACHER_BASE` | `https://api.tokenfactory.nebius.com/v1` | Token Factory, for the teacher, judge and hosted fallback | | `RATCHET_TEACHER_MODEL` | `nvidia/Nemotron-3-Ultra-550b-a55b` | Must be licensed for distillation, or startup fails | | `RATCHET_STUDENT_BASE` | empty | Your vLLM endpoint; empty means the hosted fallback | | `RATCHET_STUDENT_BASE_MODEL` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | The base every adapter is trained on | | `RATCHET_DATA_DIR` | `data` | Local folder laid out like the bucket | | `RATCHET_TRAIN_CMD` | `bash deploy/train_job.sh {version}` | Starts training and waits for it | | `RATCHET_ADAPTER_ROOT` | `/data/adapters` | Where vLLM finds adapters; also used to reload one vLLM forgot | **The bucket's layout**, as the training job and the endpoint see it at `/data`: - data - night - train.jsonl the rows Night School built tonight - adapters - student-v3 - adapter_model.safetensors written by the training job - ratchet_manifest.json base model, licenses, template arguments - NOTICE.md base model and teacher licenses - ratchet_bom.json the signed bill of materials - boms - student-v3.json a copy of the same bill of materials The ledger is not in the bucket. With the default WAL journal it must stay on the gateway host's local disk, because WAL needs shared memory between the processes that open the file (the gateway and the cron jobs); `RATCHET_DB_SYNC=FULL` is the setting for a network mount ([architecture](/reference/architecture/), section 11). In the VM kit it lives in a Docker volume. **Commands:** ```bash bash deploy/student_endpoint.sh create # once; DRY_RUN=1 first. Later: start, stop, status python -m ratchet.live night # Night School (cron) python -m ratchet.live tick # the release gate (cron, every 5 minutes) python -m ratchet.live seal # savings receipts of days that ended (cron, hourly) python -m ratchet.config check # the startup config check: exit 0, 1 or 2 DRY_RUN=1 bash deploy/vm/create_vm.sh # the Nebius VM kit, commands printed only ``` **Endpoints the pages use:** `GET /ratchet/stats`, `GET /ratchet/cost`, `GET /ratchet/tasks`, the Scout endpoints under `/scout/*`. See the [API reference](/reference/api/). ## Common questions ### Why does the gateway not need a GPU? It never runs a model. It sends the student's samples to the vLLM endpoint and the teacher's calls to Token Factory, and does the rest (scoring, logging, the gate) in plain Python. The Nebius VM kit's machine is 2 vCPU and 8 GB. ### Why one SQLite file and one process? At this scale (thousands to tens of thousands of requests a day) one file needs no setup, is easy to back up and inspect, and is fast enough. The price is one gateway process: run one uvicorn worker. A Postgres ledger for several workers is planned. ### What happens when the GPU endpoint is stopped? If `RATCHET_STUDENT_BASE` still points at it, student calls fail, and after 5 failures in a row the student's circuit breaker sends requests straight to the teacher (route `student_breaker`) until a probe succeeds. To keep a student answering while the endpoint is off, leave `RATCHET_STUDENT_BASE` empty: student calls then go to the hosted, untuned Nano 30B on Token Factory, held to the 0.80 default threshold and not covered by the adapter's precision certificate. Nebius's docs say a stopped endpoint is not billed for compute or its own storage; a mounted bucket still is. ### Why not train and serve the student on Token Factory? Token Factory listed no fine-tuning or LoRA serving for Nemotron when this design was chosen (research notes, Oct 19, 2026), and Nebius's [post on post-training in Token Factory](https://nebius.com/blog/posts/post-training-in-token-factory) names no Nemotron model to fine-tune (read Oct 26, 2026). So training runs as a Serverless Job and serving on a Serverless Endpoint, both on Nebius AI Cloud. ### What does running it cost? Estimates only. The Nebius VM kit, if you host the gateway on it: about $72.50 for Oct 29 to Dec 15, 2026, before live usage (prices as of Oct 20, 2026, [VM kit guide](/reference/deploy-nebius-vm/)). The student endpoint: about $32 per full day if left on, and a training run about $0.70 to $1.35 a night, at about $1.35 per L40S GPU-hour ([Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026; [deployment guide](/reference/deployment/#4-cost-estimates)). See [Cost and savings](/guides/cost-and-savings/). ### What runs on Nebius today? The teacher: it runs on Nebius Token Factory, and the live gateway calls it for real. As of Oct 29, 2026 the GPU side is covered by offline tests with fake transports and has run by hand: Night School ran end to end once on Nebius, by hand (28 Oct 2026): the gateway on a Nebius CPU VM, the training job on a spot L40S, the adapter served from a Nebius Serverless Endpoint. The golden gate rejected it (81.8% precision against the 95% target), so it was never made live; the loop has not run nightly on live traffic ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). The one-command host is ready to run with `make nebius-plan` and `make nebius-up` after the day-1 checks. The original text-to-SQL and tool-calling adapters were trained on a borrowed GPU; the round-2 recipe was retrained once by hand on a Nebius Serverless Job on 27 Oct 2026, and the student was served by hand, in two short sessions, on an endpoint for speed tests on 27 to 28 Oct 2026 ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). ## Related --- # One request's journey Source: https://docs.ratchet.dualmindlab.tech/guides/request-journey/ A request enters at `POST /v1/chat/completions`, passes the gateway's admission checks, goes through `Ratchet.handle` (the student first, the teacher when the student is unsure), is logged as one ledger row with a route that says why it was served that way, and comes back as a normal OpenAI response with a `ratchet` block. An order arrives at **the pass**. First a quick check: is the slip readable, is the guest allowed in, is this table over its limit tonight? A refused order is turned away before anyone touches a pan, and costs nothing. The slip names the dish (the task). Some dishes always go straight to **the head chef** (pinned tasks). For everything else **the apprentice** cooks the dish three times. If all three plates match and the apprentice is sure, one goes out, and now and then **the inspector** asks the head chef to cook the same dish to check it. If the apprentice is unsure, the head chef cooks instead, and that dish goes into **the recipe book** with the head chef's name and a note that copying it is allowed. Every order gets one line in the book: who served it, why, and what it cost. The plate goes out with a small slip on the side (the `ratchet` block) saying who cooked it and what the head chef alone would have cost. If the apprentice's stove breaks, the head chef simply cooks. If the head chef cannot come, the guest gets an apology (an error), unless you arranged a stand-in in advance. ## How it really works ### Every decision on the way
```mermaid flowchart TB adm{{"Admission checks pass?"}} adm -->|no| refused["Refused: 4xx or 503
no model call, no request row"] adm -->|yes| only{{"Teacher-only request?
pinned, tools, ineligible"}} only -->|no| ver{{"A version that serves
this task?"}} ver -->|yes| up{{"Student call works?"}} up -->|yes| conf{{"Confident enough
to serve?"}} conf -->|yes| serve["Student serves
route student"] serve --> audit{{"Audit drawn?
RATCHET_AUDIT_RATE"}} ver -->|"no: cold_start,
student_disabled, task_disabled"| teach up -->|"no: student_breaker,
student_error"| teach conf -->|"no: confidence,
n_choices, rule:i"| teach["Teacher answers
the escalation"] audit -->|"yes: the teacher checks it
at temperature 0"| store teach --> store{{"Storable? no opt-out,
complete and valid"}} store -->|yes| ex[("Training example
redacted, licensed")] only -->|"yes: the teacher
answers alone"| log audit -->|no| log store -->|no| log ex --> log(["Log one row, tick the gate,
send the response"]) class adm,only,ver,up,conf,audit,store gate class refused danger class serve student class teach teacher class ex store class log ok ```
**Reading this diagram.** Each hexagon is one check, in the order the code runs it. A refusal at admission costs nothing and calls no model. Teacher-only requests (a pinned task, a tool request, a request the student must not see) skip the student entirely and are not stored. Otherwise the student gets its chance, and each "no" on the way down sends the request to the teacher with a route that names the reason, the labels on the arrows. A confident student serves, and one random draw decides whether the teacher also checks that answer. Both teacher paths meet at the same storage check before anything becomes training data. Every path but a refusal ends in one ledger row; a teacher failure when its answer is needed ends in an error instead ([When something fails](#when-something-fails)).
  • A check
  • The student serves
  • The teacher answers
  • Kept as data
  • Refused
### An escalation, and an audit after the response
```mermaid sequenceDiagram autonumber participant Client participant Gateway participant Student as Student
vLLM participant Teacher as Teacher
Ultra participant Ledger Client->>Gateway: chat request Note over Gateway: admission, pick a version Gateway->>Student: 3 samples Student-->>Gateway: samples Note over Gateway: confidence vs threshold alt unsure: escalate Gateway->>Teacher: same request Teacher-->>Gateway: answer Gateway->>Ledger: example, if valid Gateway->>Ledger: row, route confidence Gateway-->>Client: teacher's answer else confident, audit drawn Gateway->>Ledger: row, audit pending Gateway-->>Client: student's answer rect rgba(42, 120, 214, 0.08) Note over Gateway,Ledger: after the response, on a sidecar worker Gateway->>Teacher: same request, temperature 0 Teacher-->>Gateway: answer Gateway->>Ledger: verdict, example, finish the row end end ```
**Reading this diagram.** Steps 1 to 3 are the same for both outcomes: one call to the student endpoint returns 3 samples at temperature 0.7 with token logprobs, and the gateway scores them. In the upper branch the student is unsure, so the teacher answers (4, 5), its answer is stored if it is complete and valid (6), the row is logged with route `confidence` (7) and the client gets the teacher's answer (8). In the lower branch the student is confident and the audit draw comes up. In live mode the row is logged with the audit pending (9) and the client gets the student's answer at once (10). The shaded part runs afterwards on a sidecar worker: the teacher answers the same request at temperature 0, and the verdict, the stored example and the costs finish the row (11 to 13). In mock mode, and for Scout, the same audit runs before the response.
A shadow's work runs after the response in live mode too. The audit draw itself is always made on the request path, so the same draws happen whether the audit runs before or after the response. See [Escalation and audits](/guides/escalation-and-audits/). ### Before the cascade: admission The gateway runs these checks in this order. A request refused here calls no model, writes no request row and costs nothing (a 401, 413, 429 or 503 leaves a row in the `failures` table; a 400 only a metric). | Order | Check | Refused with | Setting (default) | |---|---|---|---| | 1 | Declared body size | 413 `request_too_large` | `RATCHET_MAX_BODY_BYTES` (2 MiB) | | 2 | API key | 401 `invalid_api_key` | `RATCHET_API_KEYS` | | 3 | Requests a minute, per key (per client IP on a gateway without keys) | 429 `rate_limit_exceeded` | `RATCHET_API_KEY_RPM` (off), `RATCHET_IP_RPM` (120 in live mode) | | 4 | Body read and counted | 413 for a chunked body over the limit | | | 5 | Requests in flight: the gateway's cap, then the key's | 503 `overloaded`, 429 `rate_limit_exceeded` | `RATCHET_MAX_INFLIGHT` (32), `RATCHET_API_KEY_CONCURRENCY` (off) | | 6 | `Idempotency-Key` and JSON | 400; a finished answer for the same key and body is sent again with `Idempotent-Replayed: true` | `RATCHET_IDEMPOTENCY_TTL_S` (600 s) | | 7 | Validation and normalisation | 400 with `param` and `code` | `RATCHET_MAX_MESSAGES` (256), `RATCHET_MAX_INPUT_CHARS` (120,000), `RATCHET_MAX_N` (4) | | 8 | Task selection and the key's task scope | 400 `unknown_task`, 403 `task_not_allowed` | `RATCHET_UNKNOWN_TASKS` (`distill`) | | 9 | Daily request quota | 429 `daily_quota_exceeded` | `RATCHET_API_KEY_DAILY_REQUESTS` (off) | | 10 | US dollar budgets, per key and for the gateway | 429 `budget_exceeded`, `gateway_budget_exceeded` | `RATCHET_API_KEY_DAILY_USD`, `RATCHET_DAILY_BUDGET_USD` (off) | **Validation** ([`ratchet/openai_compat.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/openai_compat.py)) follows OpenAI's rules and answers in OpenAI's error envelope. Normalisation turns a `developer` message into `system` and text content parts into one string, so the cascade and the ledger see plain messages. The request's `model` field is ignored: Ratchet chooses the model. **Task selection.** The `X-Ratchet-Task` header wins over `metadata.ratchet_task`. With neither, a `response_format` json_schema request gets an auto task derived from its schema, and anything else is `default`. A name that is not registered follows `RATCHET_UNKNOWN_TASKS`: `distill` (the default) makes a task without fields that compares whole answers, `teacher` sends the request to the teacher alone, `reject` is a 400. A task whose `clients` list does not name the key is a 403. No header or field can pin a task; only an operator's spec does. After admission the rest runs in a worker thread, so the ledger's lock never blocks the event loop. ### Inside the cascade: `Ratchet.handle` Step through it one decision at a time, or read every rule in the numbered list under it. 1. **Read the switches and settle the request's terms.** The operator's runtime switches and task pins are read once (`Ledger.flags`). A `response_format` is bound to the task as a contract. The client's generation parameters are split by role (`GenPolicy`): what the student may receive, what the serving teacher may receive, and the audit's fixed parameters. Whether the request's teacher answers may be stored is settled now (`store_opt_out`: zero retention, a no-store task, the key's `store=0`, `store: false`, or a no-train end user). 2. **Teacher-only paths.** Some requests never reach the student. An unregistered task under `RATCHET_UNKNOWN_TASKS=teacher` (route `unknown_task`), a pinned task (route `pinned`), and a tool request, meaning tools offered or tool calls in the messages (route `tools`), all go to `_teacher_only`: the teacher answers, the request is logged, and nothing is stored as training data. A request the student must not see takes the same kind of path through `_ineligible` (route `ineligible:`): a content part only the teacher reads (`content_type`), or more input than the task's `student_max_input_tokens` (`input_tokens`); only an over-length request on a task with `store_ineligible` may be stored. No student, audit or shadow call on any of these. 3. **Pick a version** (`_pick_version`). If an operator turned the student off, none (route `student_disabled`). If a version is in canary, it is picked with probability `traffic_pct / 100` (10% or 50%); otherwise the live version. With neither (cold start) the teacher answers (route `cold_start`), and its answer is stored as usual. 4. **Skip a task the version cannot serve.** If the version's per-task record holds the task back (the release gate disabled it on that version, or there was no held-out row for it), the student is not called at all: the teacher answers (route `task_disabled`). 5. **Sample the student** (`_sample`). One call to the endpoint for `max(3, n)` samples at temperature 0.7 with token logprobs (`confidence.STUDENT_SAMPLING`), with only the client's shaping fields (`max_tokens`, `stop`, `seed`, `response_format`, `logit_bias`, `top_logprobs`), capped at 1,024 output tokens. The model name sent is the adapter's (`student-vN`). The call goes through the student's circuit breaker first. If the breaker is open the teacher answers (route `student_breaker`); if the call fails the teacher answers (route `student_error`), and a vLLM 404 naming the adapter queues an automatic reload. 6. **Score confidence** (`confidence.score`) on the first 3 samples, exactly as at calibration. A sample that is empty, cut off, unparseable or invalid under the task's spec gets no vote. Agreement is counted on the task's fields (for `ticket-triage`: `category` and `priority`). Confidence is `0.4 × p + 0.6 × agreement`, where `p` is the best agreeing sample's mean token probability; without logprobs it is agreement alone. See [Confidence](/guides/confidence/). 7. **Compare with the threshold** (`Ratchet.threshold`). Each version carries its own threshold per task, calibrated on half of its golden set. The hosted fallback model gets `RATCHET_CONFIDENCE_THRESHOLD` (0.80); a task whose spec changed since the version was scored gets 1.01, which nothing reaches. The student serves when at least one sample is valid, confidence is at or above the threshold, and, for `n` choices, at least `n` samples agree (route `student`). A confident student without `n` agreeing samples escalates (route `n_choices`); an unsure one escalates (route `confidence`). A task's `escalate_when` rules can still send a confident answer to the teacher (route `rule:`, for the rule that held). 8. **Maybe audit.** When the student serves, one random draw decides an audit: probability `RATCHET_AUDIT_RATE` (0.05; at least 0.10 in mock mode), or `RATCHET_CANARY_AUDIT_RATE` for a canary when set, or an operator's runtime rate. If drawn, the teacher answers the same request at temperature 0 and its answer is compared with the student's. The student's answer is served either way. 9. **Escalate otherwise** (`_call_teacher`). The teacher answers under the serving parameters: the client's temperature or 0, and 0.7 for `n` choices when the client set none, capped at 8,192 output tokens. The answer is cleaned (reasoning removed), served, and stored as a training example only when it is complete and valid. If the teacher fails, see [When something fails](#when-something-fails) below. 10. **Run the shadow.** If a version is in shadow, it is sampled and scored the same way, silently. Its verdict (would it serve, and does it agree with the best reference) goes into the row; a disagreement with the live student's answer is settled by the teacher (arbitration). A request answered by a stand-in runs no shadow. 11. **Log one row** (`Ledger.log_request`): task, who served, version, confidence, whether it escalated or was audited, the shadow's verdict, latency and `wall_ms`, `cost_usd` with its student and teacher parts, the all-teacher baseline, `stored`, `route`, the threshold used, `label_p` (the audit rate the answer was drawn at) and the request's ids. The row holds metadata only, never the prompt or the answer. 12. **Back in the gateway.** The spend meter adds the request's cost to its key, the release gate ticks (at most once every 5 s in live mode, `RATCHET_TICK_INTERVAL_S`), and the response is built from the router's `Result`. Scout's agents call `handle(messages, task)` directly. They skip the chat route's admission checks (Scout has its own budget guard), and their audits always run inline. ### Every route Every row gets a `route` (`router.ROUTES`). `/ratchet/stats` counts them per task, so you can see why each request went where it went. | Route | Served by | Why | Training example? | |---|---|---|---| | `student` | student | Confidence reached the threshold (and `n` samples agreed) | Only from an audit's teacher answer | | `confidence` | teacher | The student was unsure, had no valid sample, or the task's spec changed | Yes, if valid | | `n_choices` | teacher | Confident, but fewer than `n` samples agreed | Yes, choice 0 only | | `cold_start` | teacher | No live version, and the canary was not picked | Yes, if valid | | `student_disabled` | teacher | An operator turned the student off | Yes, as on a cold start | | `task_disabled` | teacher | The version's per-task record serves nothing, so no student call | Yes, if valid | | `student_error` | teacher | The student call failed (timeout, connection, error status, lost adapter) | Yes, if valid | | `student_breaker` | teacher | The student's circuit breaker was open | Yes, if valid | | `fallback_teacher` | teacher | The primary teacher was unavailable, so the fallback teacher answered | Only with `RATCHET_STORE_FALLBACK_TEACHER=1` | | `degraded` | student | Unsure student, unavailable teacher, `RATCHET_ON_TEACHER_FAILURE=serve_student` | Never | | `pinned` | teacher | The task is pinned by its spec or by an operator | Never | | `unknown_task` | teacher | Unregistered task under `RATCHET_UNKNOWN_TASKS=teacher` | Never | | `tools` | teacher | A tool request; the teacher answers with the tools | Never | | `rule:` | teacher | Rule `i` of the task's `escalate_when` held for the confident student's answer | Yes, if valid | | `ineligible:` | teacher | The student must not see the request: a content part only the teacher reads (`content_type`, `RATCHET_TEACHER_MODALITIES`) or too much input (`input_tokens`) | Only an over-length request on a task with `store_ineligible` | | `multimodal` | teacher | Set by a caller of `Ratchet.teacher_only` for a part only the teacher reads; the chat route uses `ineligible:content_type` | Never | "If valid" also means no storage opt-out applies. Rows written before v0.5 have no route and are counted as `unrecorded`. ### The response A standard `chat.completion` with every field OpenAI's has, plus a `ratchet` block. This example is from the [API reference](/reference/api/#post-v1chatcompletions) (illustrative values): ```json { "ratchet": { "served_by": "student", "student_version": 5, "confidence": 0.951, "escalated": false, "cost_usd": 0.0000312, "all_teacher_cost_usd": 0.00131, "latency_ms": 311.4, "usage_breakdown": { "student_samples": {"in": 104, "out": 93}, "served": {"in": 104, "out": 31}, "audit": null, "shadow": null, "arbitration": null, "repair": null }, "degraded": null, "stored": false, "store_opt_out": null, "request_id": "req_3f1c0a9b8e7d6c5b4a3f2e1d0c9b8a79", "row_id": 4512, "task": "ticket-triage", "schema_valid": null } } ``` | Field | Meaning | |---|---| | `served_by` | `student` or `teacher` | | `student_version` | The version that answered; `null` when the teacher answered | | `confidence` | The student's confidence when it was sampled, else `null` | | `escalated` | `true` when a version was picked but the teacher answered instead (unsure, too few agreeing samples, a failed or skipped student call, a task held back); stays `true` on a degraded answer | | `cost_usd` | What this request cost: every model call, the student priced per token. With `overhead_pending`, only what serving cost | | `all_teacher_cost_usd` | What the teacher alone would have cost: its own bill when it answered, else an estimate from the student's tokens plus 300 assumed reasoning tokens (`RATCHET_TEACHER_REASONING_TOKENS`) | | `usage_breakdown` | Every model call, in tokens: `student_samples`, `served`, `audit`, `shadow`, `arbitration`, `repair` | | `degraded` | `"teacher_unavailable"` on a degraded answer, else `null` | | `stored`, `store_opt_out` | Whether a teacher answer of this request became (or matched) a training example, and why not if opted out | | `request_id`, `row_id` | The request's id (`req_` and 32 hex, as in `x-request-id`; feedback names it) and its ledger row (a savings receipt proves it by this) | | `task`, `schema_valid` | The resolved task, and whether every choice meets the `response_format` | | `overhead_pending` | Present and `true` only when the audit or shadow runs after the response | `usage` is the served answer's own tokens, as if its model had answered alone; Ratchet's extra calls are in `usage_breakdown` and priced in `cost_usd`. `model` names who wrote the answer (the adapter `student-vN`, the hosted Nano 30B's id, the teacher's id, or the fallback teacher's), and `system_fingerprint` changes when a promotion or rollback puts another version in front. Headers: `X-Ratchet-Served-By`, `X-Ratchet-Request-Id` (the ledger row), `X-Ratchet-Degraded` on a degraded answer, and on every response `x-request-id` and `openai-processing-ms`. ### Streaming With `stream: true` the gateway sends OpenAI `chat.completion.chunk` events, a few words per chunk, and the `ratchet` block rides on the last finish chunk (or on a final usage chunk with `stream_options.include_usage`). But the cascade must see the student's whole answer to score it, so the answer is complete before the first chunk: time to first token equals the full latency. Streaming exists so streaming clients work unchanged. Errors come back as normal JSON before the stream starts. ### When something fails Only a teacher call whose answer is needed can fail a request. Everything else fails open. **The student fails: the teacher answers.** A student call on the request path waits 15 s to answer and 2 s to connect (`RATCHET_STUDENT_TIMEOUT_S`, `RATCHET_STUDENT_CONNECT_TIMEOUT_S`) and is never retried: the teacher is its retry. After 5 endpoint failures in a row (timeouts, connection errors, 5xx, 408, 429) the student's circuit breaker opens and requests go straight to the teacher without waiting:
```mermaid stateDiagram-v2 direction LR [*] --> closed closed --> open: 5 failures in a row open --> half_open: 30 s later, one probe half_open --> closed: probe succeeds half_open --> open: probe fails, wait doubles class closed ok class open danger class half_open pending ```
**Reading this diagram.** While the breaker is `closed`, every request tries the student. Five endpoint failures in a row (timeouts, connection errors, 5xx, 408, 429) open it, and requests go straight to the teacher with route `student_breaker`, without waiting for a timeout. After the wait exactly one request probes the endpoint (`half_open`): success closes the breaker and resets the wait to 30 s, failure opens it again for twice as long, up to 300 s. Each move writes a `circuit` event.
A shadow has its own breaker, so a bad staging adapter cannot open the one live traffic depends on. When vLLM forgets an adapter after a restart, its 404 queues a reload from `RATCHET_ADAPTER_ROOT` (at most once a minute per adapter). **The teacher fails on an audit or a shadow arbitration: the student's answer is served anyway**. The failure is counted and the audit simply does not count. **The teacher fails when its answer is needed** (an escalation, a cold start, a pinned or teacher-only request). The teacher gets up to 2 retries within 180 s (`RATCHET_TEACHER_RETRIES`, `RATCHET_TEACHER_DEADLINE_S`; a 120 s read timeout), on connection failures and 408, 409, 429 and 5xx. If it still fails, no request row is written; a `failures` row records what the student's samples spent, which also counts against the key's budget. The gateway maps the cause (`_map_upstream`): | Status | `code` | When | |---|---|---| | 400 | the teacher's, such as `context_length_exceeded` | The teacher refused the request itself (upstream 400, 413 or 422); `X-Should-Retry: false` | | 429 | `upstream_rate_limited` | Token Factory rate-limited the call; `Retry-After` is its wait, 1 to 60 s | | 502 | `teacher_unavailable` | The call failed after its retries | | 502 | `upstream_auth_error`, `upstream_not_found` | The gateway's own key or model setting is wrong; `X-Should-Retry: false` | | 502 | `tool_calls_unsupported` | The teacher answered a request without tools with a tool call | | 503 | `teacher_circuit_open` | The teacher's breaker is open after 5 failed calls in a row; answered at once with `Retry-After` | | 504 | `teacher_timeout` | The teacher did not answer in time | Two opt-in stand-ins answer instead of an error. A **fallback teacher** (`RATCHET_TEACHER_FALLBACK_MODEL`, licensed at startup like any teacher) answers, when the primary is unavailable, any request whose teacher answer will be served (route `fallback_teacher`); audits and arbitrations never use it. **Degraded mode** (`RATCHET_ON_TEACHER_FAILURE=serve_student`) serves the student's own valid, below-threshold answer to an escalation, marked `ratchet.degraded` and `X-Ratchet-Degraded` (route `degraded`). Neither covers a 400 about the request. A stand-in's answer is never audited or shadowed, and a degraded answer is never stored, so neither can move a release. ## For engineers | File | Functions | |---|---| | [`ratchet/gateway.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/gateway.py) | `chat` (key, rate, body, in-flight caps), `_complete` (validation, `_request_meta`, task, quota, budgets, cascade, response), `_map_upstream`, `_envelope` | | [`ratchet/openai_compat.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/openai_compat.py) | `validate_chat_request`, `normalize_messages`, `APIError` | | [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) | `Ratchet.handle`, `_pick_version`, `_sample`, `threshold`, `_audit`, `_call_teacher`, `_store_example`, `_shadow`, `_teacher_only`, `_may_degrade`, `ROUTES` | | [`ratchet/confidence.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/confidence.py) | `score`, `majority`, `STUDENT_SAMPLING` | | [`ratchet/resilience.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/resilience.py) | `Breaker`, `retry_call`, `endpoint_down`, `teacher_unavailable` | | [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py) | `log_request`, `add_example`, `flags` | | Setting | Default | Effect on the journey | |---|---|---| | `RATCHET_SAMPLES` | 3 | Student samples per request, and per golden row at calibration | | `RATCHET_CONFIDENCE_THRESHOLD` | 0.80 | Threshold for the hosted fallback, or a version without a calibrated one | | `RATCHET_AUDIT_RATE` | 0.05 | Share of student answers the teacher also checks (at least 0.10 in mock mode) | | `RATCHET_CANARY_AUDIT_RATE` | unset | A canary's own audit rate | | `RATCHET_STUDENT_TIMEOUT_S`, `RATCHET_STUDENT_CONNECT_TIMEOUT_S` | 15, 2 | Student call limits; never retried | | `RATCHET_TEACHER_TIMEOUT_S`, `RATCHET_TEACHER_RETRIES`, `RATCHET_TEACHER_DEADLINE_S` | 120, 2, 180 | Teacher read timeout, retries and overall deadline | | `RATCHET_STUDENT_BREAKER_FAILURES`, `RATCHET_STUDENT_BREAKER_COOLDOWN_S` | 5, 30 | Student breaker (0 failures turns it off) | | `RATCHET_TEACHER_BREAKER_FAILURES`, `RATCHET_TEACHER_BREAKER_COOLDOWN_S` | 5, 30 | Teacher breaker | | `RATCHET_TEACHER_FALLBACK_MODEL` | empty | Opt-in fallback teacher | | `RATCHET_ON_TEACHER_FAILURE` | `error` | `serve_student` turns on degraded mode | | `RATCHET_STUDENT_MAX_TOKENS`, `RATCHET_TEACHER_MAX_TOKENS` | 1,024, 8,192 | Output caps per call | | `RATCHET_SIDECAR_WORKERS` | 4 live, 0 mock | Audits and shadows after the response | | `RATCHET_TICK_INTERVAL_S` | 5 live, 0 mock | Least time between two gate ticks from requests | All of them are in the [configuration reference](/reference/configuration/). Look a request up afterwards with `GET /v1/ratchet/requests/{id}` (an API key, its own requests) or `GET /ratchet/requests/{id}` (the dashboard API), by the `req_` id or the `chatcmpl-` id. ```bash curl -si http://localhost:8000/v1/chat/completions \ -H 'content-type: application/json' \ -H 'X-Ratchet-Task: ticket-triage' \ -d '{"model":"ratchet","messages":[{"role":"user","content":"I was charged twice for my Pro plan this month."}]}' # look for X-Ratchet-Served-By, X-Ratchet-Request-Id and the "ratchet" block ``` ## Common questions ### Can a client force the student or the teacher? No. The request's `model` field is ignored, and no header or field can pin a task. Only an operator's task spec, or a runtime pin through the admin API, sends a task to the teacher alone. ### Does streaming make answers arrive sooner? No. The student's whole answer must be scored before Ratchet decides who serves, so the first chunk comes when the answer is complete. ### Why three samples, not one? Small models are poorly calibrated, so one token probability does not tell right from wrong. Agreement across three samples plus token probability is the signal the threshold is calibrated on. The cost is the student's output tokens three times, small next to one teacher call. ### Is my prompt stored? The request row holds metadata only. Prompt and answer text is stored only as a training example from a teacher answer, after PII redaction (emails, phone and card numbers and similar; names and street addresses are not caught yet), and not at all under a storage opt-out. Pinned, tool and multimodal requests are never stored. See [Security](/guides/security/). ### What does a request cost me? `ratchet.cost_usd` is the full price of every call the request made, at the Token Factory list prices checked on 29 Oct 2026 (the GPU-hour rate is still unverified). `usage` is only the served answer's tokens, so you can bill your own users for the answer they got. ### What if the student is wrong but confident? It is served. A random audit may catch it, and audits drive the canary's rollback and the live version's alert. Precision means agreement with Nemotron 3 Ultra, not ground truth. ### Has this path run against real endpoints? Partly. The teacher runs on Nebius Token Factory. The student side is tested offline and ran once by hand on Nebius on 28 Oct 2026 (the golden gate rejected that student): Ratchet's own adapter is not connected to the live gateway (a random 40% of eligible requests are answered by the hosted Nano 30B, a quarter of those re-checked by Ultra), and the paths through vLLM are tested with fake HTTP transports, ready to run with `make nebius-up`. Model ids and per-token prices were checked on 29 Oct 2026 (`scripts/day1_check.py` passed); the GPU-hour rate is still unverified. ## Related --- # Confidence Source: https://docs.ratchet.dualmindlab.tech/guides/confidence/ Confidence is the score that decides, on every request, whether the student's answer goes out or the teacher is asked, and each adapter version gets its own threshold, set on half of the golden set and checked on the other half. The apprentice chef gets an order. Before anything leaves the kitchen, the apprentice cooks the dish **three times**. If all three plates come out the same, and the apprentice felt sure at every step while cooking, the plate goes out. If the plates differ, or one of them is burnt, the head chef from the agency cooks the order instead. Why not just ask the apprentice "are you sure?" Because young cooks are often sure and wrong. So the pass looks at two things at once: did the three plates match, and how steady was the apprentice's hand while cooking. How sure is sure enough? That bar is set with **the sealed exam**: recipes the apprentice never studied. Half of the exam sets the bar: the lowest bar at which at least 97 of every 100 plates the apprentice would send out match the head chef's. The other half grades the result. The apprentice never saw the grading half while the bar was set, so the grade is honest. Every new notebook gets its own bar, and each kind of dish can get its own bar too: an apprentice can be great at salads and shaky at soufflés. ## How it really works
```mermaid sequenceDiagram participant C as Client participant G as Gateway participant Student as Student
Nano 4B + LoRA participant Teacher as Teacher
Ultra C->>G: chat request G->>Student: 3 samples, with logprobs Student-->>G: 3 answers, token logprobs Note over G: clean, key, agree,
one confidence number alt at or above the threshold G-->>C: the student's answer opt audit drawn G->>Teacher: same request, temperature 0 Teacher-->>G: answer, compared end else below the threshold G->>Teacher: escalate Teacher-->>G: answer G-->>C: the teacher's answer end ```
**Reading this diagram.** Time runs downward. The student is always asked first, three times in one call. Only the gateway looks at the samples: it cleans them, counts agreement and computes one confidence number, and that number alone decides who answers. When the student answers, the teacher may still be called for a random audit (the `opt` block), but in live mode the client never waits for it. When the student does not answer, the teacher's answer goes to the client and is stored as training data if it is valid.
  • The student: Nano plus your adapter
  • The teacher you rent
The student () answers first. The gateway samples it several times, scores the samples, and compares the score with a threshold that belongs to the version that answered and, since per-task calibration, to the task. 1. **Sample.** The router asks the student for `RATCHET_SAMPLES` answers (default 3) at temperature 0.7 with token logprobs on. These sampling settings are one constant, `confidence.STUDENT_SAMPLING`, shared by production and both golden evaluators, so the eval scores the student exactly the way production does. 2. **Check each sample.** Each answer is cleaned (`tasks.clean_answer`: reasoning stripped, a code fence around JSON unwrapped) and turned into a key by the task's spec (`Task.key`). A sample gets **no vote** when it does not parse, misses a required field, has a label outside the task's vocabulary, is empty, or did not end by itself (a `finish_reason` other than `stop` or `tool_calls`, so a cut-off answer never counts). If no sample is valid, confidence is 0 and nothing is served, whatever the threshold, even 0. 3. **Count agreement.** Valid samples are compared pairwise with the task's field comparators (`Task.match`). Each sample's votes are the valid samples it matches, itself included. The majority is the first sample with the most votes, and `agreement` is its votes divided by the number of samples. With a tolerance, agreement is not always "all equal": under a number field with `abs_tol` 1, the answers 0, 0.6 and 1.2 all match 0.6, so 0.6 is served with agreement 1.0. 4. **Read token probability.** Among the majority's samples with the top vote count, the gateway picks the one with the highest token probability, `p = exp(mean token logprob)`, from the logprobs vLLM returns. 5. **Combine.** `confidence = 0.4 × p + 0.6 × agreement`. If the endpoint returned no logprobs, confidence is the agreement alone. 6. **Compare.** If the answer is valid and confidence is at least the threshold, the student's answer is served (route `student`), and a random share of such answers is audited by the teacher. Otherwise the request escalates to the teacher (route `confidence`), whose answer is served and, if valid, stored as training data with its provenance.
```mermaid flowchart TB S["3 student samples"] --> K["Clean each answer
and compute its key"] K --> V{{"Any valid sample?"}} V -->|no| Z["Confidence 0
escalate"] V -->|yes| A["Agreement
majority votes / samples"] A --> P["Token probability p
of the best majority sample"] P --> C["Confidence =
0.4 × p + 0.6 × agreement"] C --> T{{"At least the threshold?"}} T -->|yes| Serve["Serve the student's answer"] T -->|no| Esc["Escalate to the teacher"] class S,Serve student class V,T gate class Z,Esc teacher ```
**Reading this diagram.** Read it top to bottom. The first check is validity: with no valid sample, confidence is 0 and the threshold is never consulted. Otherwise two signals are computed, agreement among the samples and the token probability of the best agreeing sample, and merged with weights 0.6 and 0.4. One comparison with the version's threshold for this task ends the path in either "serve" (the student, blue) or "escalate" (the teacher, orange).
A worked toy example (arithmetic only, not a result): three samples that all agree, best token probability 0.9, give `0.4 × 0.9 + 0.6 × 1.0 = 0.96`. If only two of three agree, the same sample scores `0.4 × 0.9 + 0.6 × 2/3 = 0.76`. Agreement carries more weight than probability on purpose. ### Why one signal is not enough Small models are poorly calibrated: a 4B model can put a high probability on a wrong answer. Token probability alone does not separate right from wrong, and agreement alone cannot tell three confident samples from three lucky ones. Ratchet combines them, because a single fixed bar was not enough. Validity is the third signal. A label outside the task's vocabulary, or a JSON answer missing a field, is not a vote at all, so the student cannot "agree with itself" on an answer the task would refuse. ### Precision and coverage, simply Two numbers describe any threshold: - **Precision**: of the answers the student would serve, the share that agree with the teacher () under the task's comparators. - **Coverage**: of all requests, the share the student would serve. Coverage is where the savings come from. Raising the threshold usually serves fewer answers at higher precision. Usually, not always: one confidently wrong answer at the top can make precision fall as the threshold rises (which rules out some textbook methods). The numbers below are made up to show the arithmetic. They are not results from Ratchet. Take 20 golden rows. The ten **calibration** rows, sorted by the student's confidence, with whether its answer agreed with the teacher: | Confidence | Agrees? | Served at this bar | Agreeing among served | Precision at this bar | |---|---|---|---|---| | 0.98 | yes | 1 | 1 | 100% | | 0.95 | yes | 2 | 2 | 100% | | 0.93 | yes | 3 | 3 | 100% | | 0.90 | yes | 4 | 4 | 100% | | 0.88 | yes | 5 | 5 | 100% | | 0.85 | no | 6 | 5 | 83% | | 0.80 | yes | 7 | 6 | 86% | | 0.72 | no | 8 | 6 | 75% | | 0.65 | yes | 9 | 7 | 78% | | 0.55 | no | 10 | 7 | 70% | The lowest bar where precision is still at least 97% is **0.88**. Now grade it on the ten **held-out** rows: confidences 0.99, 0.96, 0.94, 0.91 and 0.89 are at or above 0.88, and one of those five (the 0.89) disagrees with the teacher. So on held-out data this toy version has **coverage 50%** (5 of 10) and **precision 80%** (4 of 5), not the 100% the calibration half showed. That gap is why Ratchet never grades a threshold on the rows it was tuned on, and why this toy version would be rejected by the golden gate (80% is below the 95% floor). With only five served rows, its certified lower bound would also be weak: 34% (exact Clopper–Pearson at delta 0.05, from `ratchet/stats.py`). ## The threshold, per version Each trained adapter version carries its own threshold in its eval (`versions.eval.threshold`). Night School sets it the night the version is trained, in `cascade_eval` ([`ratchet/nightschool/evaluate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/evaluate.py)): 1. **Score every golden row** with the new adapter, sampled exactly as in production, under the generation conditions stored with the row. Each row becomes a point: (confidence, agreed with the teacher's stored answer). A row whose stored reference is no longer valid under the task's spec is skipped (`golden_invalid_refs`), and so is a row of a task that is pinned now (`excluded_pinned`). A row the student endpoint refuses (HTTP 400, 413 or 422) counts as confidence 0 and wrong, because production would escalate it. 2. **Split by position.** Even-indexed scored rows form the calibration half; odd-indexed rows form the held-out half. 3. **Calibrate on the even half.** By default (`RATCHET_CALIBRATION=point`), `calibrate_threshold` sorts the points by confidence, highest first, walks down in groups of equal confidence (ties included before checking) and keeps the lowest confidence at which cumulative precision is still at least the target: 0.97 (`calibration.DEFAULT_TARGET_PRECISION`). If no confidence reaches it, the result is infinity. 4. **Clamp.** `threshold = min(1.01, max(0.5, calibrated))`. The floor 0.5 keeps the student from serving coin-flip answers. Scores are at most 1, so **1.01 means "serve nothing"** (`SERVE_NOTHING`); it is a finite number so the eval stays valid JSON. Since v0.5 the threshold is stored exactly, so the router serves at the threshold the eval measured. 5. **Measure on the odd half.** Coverage and precision at that threshold, plus the certificate, the frontier, the per-task bounds and the per-row outcomes (`test_vector`). These held-out numbers are what the [release gate](/guides/release-gate/) reads.
```mermaid flowchart TB T["Teacher answers
stored as examples"] --> H{{"Hash of the request"}} H -->|"about 90%"| Train[("Training set")] H -->|"about 10%"| G[("Golden set
never trained on")] G -.->|"seen 3 or more times"| Train G --> Sc["Score each row
with the new adapter"] Sc --> E["Even rows
calibration half"] Sc --> O["Odd rows
held-out half"] E --> Thr["Pick the threshold"] Thr --> M["Measure precision
and coverage"] O --> M class T teacher class H gate class Train,G store class Sc student class M ok ```
**Reading this diagram.** Every teacher answer is first split by a hash of the request, so no model's opinion decides which rows are held out. The golden rows are scored with the new adapter and then split again by position: the even half picks the threshold and the odd half measures it, so the grade comes from rows the threshold never saw. The dotted arrow is the one exception: a request seen three or more times moves from golden to training.
Which requests are golden is decided by a hash of the request (`RATCHET_GOLDEN_FRACTION`, default 0.10), which acts like a coin flip: it does not look at difficulty or at any model's answer. A request seen 3 or more times (`FREQUENT`) moves to training and leaves the golden set, so a frequent question is never held out forever. Golden rows are teacher answers to escalations, audits and arbitrations, so they lean towards requests the student found hard. Precision measured on them means agreement with Nemotron 3 Ultra, not ground truth. The router reads the threshold on every request with a dictionary lookup (`Ratchet.threshold`). Thresholds differ between versions because each adapter's confidence scale is its own; that is also why the ledger replay never pools two versions. ## Per-task thresholds Several tasks can share one adapter: Scout's triage and reader calls next to support tickets, for example. One threshold calibrated over the pooled rows then serves the hardest task below its target (the easy tasks' right answers dilute its wrong ones) and holds the easy tasks back when the hard task pushes the bar up. So `cascade_eval` also writes one record per task, `eval['tasks'][name]`, split by the same global even/odd positions: | Situation | The task's threshold | |---|---| | At least `RATCHET_MIN_TASK_GOLDEN` (20) calibration rows | Its own, calibrated to its spec's `policy.target_precision` (default 0.97), never below 0.5 (`calibrated: true`) | | Fewer calibration rows | The higher of its own and the version's global threshold, so a few lucky rows cannot lower it (`calibrated: false`) | | No calibration rows | The version's global threshold | | No held-out row | 1.01: the teacher answers it until a night measures it | | No golden row at all | No record: the version's global threshold | | Disabled by the release gate | 1.01 for the life of that version | When a task's record serves nothing, the router does not call the student for it at all (route `task_disabled`): its samples could never be served, so calling would only add cost and latency. With one task the record equals the global numbers. Per-task calibration is on by default. `RATCHET_PER_TASK_CALIBRATION=0` makes Night School write no records, so new versions keep one pooled threshold, as in v0.4. Per-task calibration holds each task to its own target, at the price of the student serving somewhat less than with one pooled threshold. ### When a task's spec changes Each per-task record stores the `spec_hash` it was scored under. If an operator changes the task's spec (its fields, its vocabulary, or its `target_precision` or `min_precision`), the hash no longer matches and `calibration.threshold` returns 1.01 for that task: the student is still sampled, but every request of the task escalates (route `confidence`) until a night re-scores the version under the new spec. A night with too few new examples to train still re-scores the live version when its recorded hashes are stale (event `rescored`, or `rescore_failed`, after which the task stays on the teacher and the next night tries again). ## Certified thresholds and Learn-then-Test Every eval also carries a **certificate**: on the held-out half, how many answers the version would serve, how many of those agree with the teacher, and an exact one-sided Clopper–Pearson lower bound on its precision (`RATCHET_CERT_TARGET` 0.95, `RATCHET_CERT_DELTA` 0.05). [Evidence you can check](/guides/evidence/) explains how to read and recompute it. The default point rule picks the threshold and the certificate checks it afterwards. With `RATCHET_CALIBRATION=ltt` (opt-in), the threshold itself is chosen so that its precision is certified, by **Learn-then-Test**: 1. Take a fixed grid of thresholds, decided in advance: 0.95, 0.90, 0.85, down to 0.50 (`stats.CERT_GRID`). 2. From the top, test "precision of the calibration answers at or above this threshold is at most the target" with an exact binomial p-value. 3. Keep going while that test is rejected at `delta`; stop at the first threshold where it is not. 4. Serve at the lowest threshold that passed, or serve nothing (1.01) if the first one already fails. The held-out certificate then checks the choice independently. Testing a fixed order and stopping at the first failure spends the error budget once, so the chosen threshold has precision above the target with probability at least `1 − delta`. The grid must be fixed in advance: a grid built from the observed confidences would start with a one-row group, which cannot pass, and a grid picked after looking at the data breaks the guarantee. Learn-then-Test is opt-in because it is conservative early on. Ten calibration rows cannot certify 95%, since `0.95^10 = 0.60`, far above 0.05. In the toy table above only two calibration rows are at 0.95 or more, so Learn-then-Test would serve nothing. ## Watching the confidence scores over time: the drift alarm A threshold is set once per version, on a fixed golden set. Real traffic can change afterwards: new kinds of requests, a new customer, a changed prompt. The student's confidence scores often move before anyone has labels to show a drop in quality, so Ratchet also watches them without labels. For each (version, task) pair the gate keeps a **conformal test martingale** over the stream of confidence scores the student produced while it served. In plain words: each new score is ranked against the scores seen so far. If the traffic is unchanged, the ranks look random and the running evidence stays near 1. If the scores start to shift, the evidence grows. When it passes `RATCHET_DRIFT_ALARM` (default 100), the alarm fires. A stream whose traffic never changes fires with probability at most 1/100 over its whole life.
```mermaid flowchart TB L[("Ledger: the confidence
of each served answer")] --> R["Rank each score
against earlier ones"] R --> M["Test martingale
per version and task"] M --> Q{{"Evidence above 100?"}} Q -->|no| W["Keep watching"] Q -->|yes| D["Write a drift event"] D --> C["Suspend the version's
certificate"] D -.->|"opt-in"| B["Raise the
audit rate"] D -.->|"opt-in"| N["Start Night School
earlier"] class L store class Q gate class C danger class B,N outline ```
**Reading this diagram.** Data flows top to bottom from the ledger, never from the request path. The alarm has one fixed effect, the certificate marked suspended, and two opt-in ones on dotted arrows that are off by default. No arrow leads to a promotion, a rollback or a threshold change: a person or the next night decides what to do.
  • Keeps data
  • A check
  • Certificate suspended
  • Opt-in, off by default
What an alarm does and does not do: - It writes one `drift` event and marks the version's certificate as suspended, because the certificate was measured on traffic that may no longer look like the live traffic. - With `RATCHET_DRIFT_AUDIT_BOOST` above 1 (default 1, off), the alarmed version's audit rate is multiplied by that factor while it serves, up to 0.5. With `RATCHET_DRIFT_EARLY_NIGHT=1` (default 0), Night School needs half the usual number of new examples to train. - It never promotes, rolls back or changes a threshold by itself. It runs in the gate's tick, reads the ledger and never touches the request path. `GET /ratchet/drift` reports each stream and the chance that any stream alarms by luck (with K streams, at most K/100). A version holds at most 17 streams: 16 task streams and one shared stream for every later task name, so a client that sends many task names cannot grow memory. The code is in [`ratchet/drift.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/drift.py) (`Stream`, `DriftMonitor`), fed by `ReleaseGate._tick_drift`. ## What can go wrong | Failure | What the system does | Where you see it | |---|---|---| | The student endpoint returns no logprobs | Confidence is the agreement alone | `ratchet.confidence` in the response | | No valid sample | Confidence 0, the teacher answers | route `confidence` | | The task's spec changed after the version was scored | Every request of the task escalates until a night re-scores | route `confidence`, events `rescored` or `rescore_failed` | | A task's record serves nothing | The student is not called for it | route `task_disabled` | | The confidence scores of live traffic shift | A `drift` event and a suspended certificate; nothing is rolled back | `GET /ratchet/drift`, the event log | | Too few calibration rows for Learn-then-Test | It serves nothing (1.01) until there is more data | per-task `threshold` in `GET /ratchet/stats` | ## The hosted fallback's fixed threshold When the student's GPU endpoint is off (`RATCHET_STUDENT_BASE` unset), a trained version's calls go to an untuned, hosted model instead: Nemotron 3 Nano 30B on Token Factory (). In the restaurant, a different cook from the agency takes the easy orders, and the menu says so: the response's `model` field names the model that wrote the answer, though `served_by` still says `student`. That model never sat the exam, so no calibrated threshold or per-task record applies to it. `Ratchet.threshold` returns the fixed `Settings.confidence_threshold` (`RATCHET_CONFIDENCE_THRESHOLD`, default 0.80) for it, and its answers are **not covered by the adapter's certificate**. The same 0.80 is used for a version whose eval has no calibrated threshold. - **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)). - **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)). - **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). - **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). - **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate. - **The nightly schedule on live traffic.** Night School runs by hand today; the timer is scripted. - **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline. - **A threshold calibrated on a live ledger.** Night School's one Nebius run calibrated one on 458 golden rows, imported rather than served. The scoring, calibration and certificate code is tested offline with mock models. The text-to-SQL adapter, trained on a borrowed GPU, sets a similar gate on real Nano 4B outputs from its average token probability; its measured numbers are on [Results](/guides/results/). Run every script with `DRY_RUN=1` first. ## For engineers | What | Where | |---|---| | The score, validity, the sampling constant, `SERVE_NOTHING` | [`ratchet/confidence.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/confidence.py): `score`, `Scored`, `STUDENT_SAMPLING`, `majority` (the `n` choices) | | Point calibration | `confidence.calibrate_threshold` | | The golden eval | [`ratchet/nightschool/evaluate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/evaluate.py): `cascade_eval`, `_task_record`, `LiveEvaluator` | | Per-task records and spec changes | [`ratchet/calibration.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/calibration.py): `threshold`, `held_back`, `spec_changed`, `keep_disabled`, `target_precision` | | The threshold on the request path | [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py): `Ratchet.threshold`, `Ratchet._sample`, `Ratchet._held_back` | | Certificate and Learn-then-Test | [`ratchet/certificate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/certificate.py), [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py): `ltt_fixed_sequence`, `CERT_GRID`, `cp_lower` | | Tests | `tests/test_core.py`, `tests/test_calibration.py`, `tests/test_contract.py`, `tests/test_nightschool.py` | Settings (read in [`ratchet/config.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/config.py), documented in [Configuration](/reference/configuration/)): | Variable | Default | Effect | |---|---|---| | `RATCHET_SAMPLES` | `3` | Student samples per request and per golden row (1 to 16) | | `RATCHET_CONFIDENCE_THRESHOLD` | `0.80` | Used only for the hosted fallback student and for a version without a calibrated threshold | | `RATCHET_GOLDEN_FRACTION` | `0.10` | Share of new examples hashed into the golden set | | `RATCHET_PER_TASK_CALIBRATION` | `1` | `0`: one pooled threshold for new versions | | `RATCHET_MIN_TASK_GOLDEN` | `20` | Calibration rows a task needs before its threshold may go below the global one | | `RATCHET_CALIBRATION` | `point` | `ltt`: Learn-then-Test over the fixed grid | | `RATCHET_CERT_TARGET` | `0.95` | The precision a certificate must show, and Learn-then-Test's target | | `RATCHET_CERT_DELTA` | `0.05` | The certificate's error rate | | `RATCHET_DRIFT_ALARM` | `100` | Drift alarm threshold per stream; `0` turns the alarm off | | `RATCHET_DRIFT_AUDIT_BOOST` | `1.0` | Audit-rate multiplier after a drift alarm (`1` is off) | | `RATCHET_DRIFT_EARLY_NIGHT` | `0` | `1`: a drift alarm halves the new examples Night School needs | | `RATCHET_AUDIT_RATE` | `0.05` | Share of served student answers the teacher audits (mock mode raises it to at least 0.10) | Constants in the code: weights 0.4 (token probability) and 0.6 (agreement); student temperature 0.7; calibration target 0.97 (a task's `policy.target_precision` overrides it); floor 0.5; `SERVE_NOTHING` 1.01; `CERT_GRID` 0.95 down to 0.50 in steps of 0.05. Where to see it: - Every chat response carries `ratchet.confidence` and `ratchet.served_by`; `system_fingerprint` changes with the version and the threshold the answer was held to. - `GET /ratchet/stats` lists, per task, the threshold the router applies, the held-out precision and coverage, `calibrated` and `disabled` ([API](/reference/api/)). - `GET /ratchet/versions/{id}/certificate` serves the signed and the latest certificate. - `GET /ratchet/replay?threshold=0.4` replays a version's logged traffic at another threshold ([Evidence you can check](/guides/evidence/)). - Routes in `requests.route`: `student`, `confidence`, `cold_start`, `task_disabled`, `n_choices` (the student was sure, but fewer than the client's `n` samples agreed). - `GET /ratchet/drift` reports the drift alarm's streams. - Events: `rescored`, `rescore_failed`, `task_disabled`, `drift`. ```bash python -m ratchet.stats plan --target 0.95 --delta 0.05 # 0 errors: 59 rows, 1: 93, 2: 124, 3: 153 curl -s localhost:8000/ratchet/versions/2/certificate # the certificate behind a version's threshold ``` The full rules are in [Architecture §6](/reference/architecture/#6-confidence-and-calibration). ## Common questions ### What if the student is confidently wrong? It happens: the target is 97% agreement on the calibration half, not 100%. Three things catch it. A random share of served answers (`RATCHET_AUDIT_RATE`, 5% by default) is re-answered by the teacher, and those audits feed the canary's rollback rule and the live alert ([the release gate](/guides/release-gate/)). An audit's teacher answer is also stored as a training example when it is valid, so the next night learns from what the student got wrong. And the threshold is re-set for every new version on rows it never trained on. ### Why sample three times? Isn't that three times the cost? It triples the student's output tokens and adds a little latency. Ratchet accepts that as a small cost next to one teacher call. A judge model scoring every request was rejected because it costs a teacher-sized call every time. ### Can I set the threshold by hand? Not for a trained version. Thresholds change only through Night School's held-out calibration. `RATCHET_CONFIDENCE_THRESHOLD` applies only to the hosted fallback and to a version without a calibrated threshold. The ledger replay can suggest a lower threshold, but its suggestion is advice and nothing applies it. The runbook says never to edit a version's threshold by hand. ### Does "precision" mean the answer is correct? No. It means the student's answer agrees with Nemotron 3 Ultra's under the task's comparators. If the teacher is wrong, a student that copies it scores well. Precision is also measured on golden rows, which lean towards hard requests. ### What about free-text tasks with no fields? A task without fields compares the whole answer: its canonical JSON when it is JSON, otherwise its text with reasoning removed, lower-cased and whitespace collapsed. That is strict, so free-text tasks escalate more. The live evaluator can add a judge score (Nemotron 3 Super) for such tasks, which is reported in the eval but does not gate anything. ### What happens before any version exists? Nothing is scored. On a cold start there is no version, so every request goes to the teacher (route `cold_start`) and its answers become the first training and golden examples. ## Related --- # Escalation and audits Source: https://docs.ratchet.dualmindlab.tech/guides/escalation-and-audits/ Escalation sends a request to the teacher whenever the student cannot serve it safely, and audits send a random few of the student's own answers to the teacher as well, so Ratchet always has fresh, unbiased evidence of how often the student agrees with the teacher. **Escalation** is the apprentice saying "I'm not sure, chef". The apprentice cooked the dish three times and the plates did not match, or the stove broke, or this dish was taken off the apprentice's list. The **head chef** cooks it instead. The guest gets the head chef's plate, and the dish goes into the **recipe book**, so the apprentice can study it tonight. **Audits** are the **inspector**. The inspector does not wait for the apprentice to ask for help. Now and then, at random, the inspector takes a plate the apprentice already sent out and asks the head chef to cook the same dish. Do the two plates match? The guest has already eaten, so nothing changes for them. But the inspector's tally is how the restaurant knows whether the apprentice is really as good as the exam said. Why random? If the inspector only tasted plates the apprentice was unsure about, the tally would only show the hard dishes. Tasting at random shows how the apprentice does on the dishes it actually serves. One more thing: the inspector compares the apprentice with the head chef, not with a perfect cookbook. If the head chef gets a dish wrong, the inspector will not notice. ## How it really works ### When a request escalates
```mermaid flowchart TB A(["Request reaches the cascade"]) ==> B{{"1 · Version picked?"}} B ==>|yes| C{{"2 · Task held back?"}} C ==>|no| D{{"3 · Breaker open?"}} D ==>|no| E["4 · Sample the student
3 times"] E ==> F{{"5 · Valid, and at or
above the threshold?"}} F ==>|yes| G{{"6 · Enough agree
for n choices?"}} G ==>|yes| H["Student serves"] H --> I{{"Audit drawn?"}} I -->|yes| J["Teacher checks
the served answer"] B -->|no| T["Teacher answers
the route says why"] C -->|yes| T D -->|yes| T E -->|failed| T F -->|no| T G -->|no| T class A neutral class B,C,D,F,G,I gate class E,H student class T,J teacher ```
**Reading this diagram.** Follow one request down the thick path; the numbers are the order the checks run in. Each hexagon is a reason the student might not answer, and every way off the thick path ends at the teacher, with a route in the ledger that names the reason (the table below lists them). Only checks 5 and 6 happen after the student has produced something: confidence and `n` choices. If the student does serve, the only thing left is the audit draw, which sends a random few of its answers to the teacher as well.
  • A check
  • The student
  • The teacher, with the route
Every teacher box after the version check is an escalation: a version was picked, but the teacher answers (`escalated` is true in the row and the response). The route says why: | Trigger | Route | Student called? | |---|---|---| | Confidence below the version's threshold for this task | `confidence` | Yes | | No valid sample: every sample empty, cut off, unparseable or invalid under the task's spec (confidence 0) | `confidence` | Yes | | The task's spec changed since the version was scored: its threshold is 1.01 until a night re-scores it | `confidence` | Yes | | Confident, but fewer than `n` samples agree for a request with `n` choices | `n_choices` | Yes | | The student call failed (a vLLM 404 for the adapter also queues a reload) | `student_error` | Yes, it failed | | The student's circuit breaker is open | `student_breaker` | No | | The release gate disabled the task on this version, or it had no held-out rows | `task_disabled` | No | The teacher also answers, without an escalation, on a cold start (`cold_start`), while an operator has the student turned off (`student_disabled`), and on the teacher-only paths (`pinned`, `unknown_task`, `tools`, `multimodal`). See the full [routes table](/guides/request-journey/#every-route). If the gateway's daily budget is spent under `RATCHET_BUDGET_MODE=student_only`, an escalation is refused with a 429 `gateway_budget_exceeded` before any teacher call; confident student answers still go out. ### What the teacher's answer is used for 1. **It is served.** The client gets the teacher's answer, cleaned: reasoning removed (`` blocks and the like) and, for a structured task, bare JSON. The teacher answers under the serving parameters: the client's temperature, or 0 when it set none (0.7 when the client asked for several choices). 2. **It is checked before it may become training data** (`Ratchet._store_example`). It is stored only when all of these hold: no storage opt-out applies; it is the primary teacher's answer (a fallback teacher's only with `RATCHET_STORE_FALLBACK_TEACHER=1`); it was not written under label-altering client sampling (`logit_bias`, a non-zero penalty, a temperature above 1, a vendor field); its `finish_reason` is `stop` or `tool_calls`; it is not empty once reasoning is stripped; and it is valid under the task's spec (it parses, every required field is present, a label is in its vocabulary). Anything else is still served, but skipped, with a `skip_example` event on the 1st, 10th, 100th and 1000th skip of each reason. 3. **It is redacted, then checked again.** Emails, phone numbers, Luhn-checked card numbers, IBANs, secrets and similar identifiers are replaced with placeholders such as `[EMAIL]` in the messages, the answer and the stored conditions. The teacher saw the original. If a placeholder breaks a structured answer, it is skipped (`invalid after redaction`). 4. **It is stored with its provenance** (`Ledger.add_example`): the teacher's model id, the license that allows training on it and its URL, the number of placeholders, the conditions it was written under (`response_format`, `tools`, `tool_choice`, `stop`), the API client and the end user's hash. Its split, golden or train, is fixed from a hash of the task and messages: about 10% golden (`RATCHET_GOLDEN_FRACTION`). A repeat of the same request bumps its `seen` count instead (a client's retry does not). For `n` choices only choice 0 is stored. 5. **It labels the student's own answer.** When the student had a valid answer and the teacher wrote its answer under the audit's conditions (one choice, no client sampling), the row records whether they agreed (`student_agree`). The ledger replay uses it. 6. **It is the shadow's reference.** If a version is in shadow, its answer for this request is compared with the teacher's. The student's own answers are never stored as training targets, and pinned, tool and multimodal requests store nothing. ### Audits: what they are When the student serves, one random draw (`rng.random()`, exactly one per served answer) decides an audit. The probability is: - `RATCHET_AUDIT_RATE`: 0.05 by default; mock mode raises it to at least 0.10; - for answers a **canary** serves, `RATCHET_CANARY_AUDIT_RATE` when set (audit focusing: the canary's evidence arrives sooner); - an operator's runtime rate, when set, replaces both (`PUT /ratchet/admin/settings`, or `python -m ratchet.admin flag audit_rate 0.2`). The rate the answer was drawn at is logged as `requests.label_p` (1.0 for a student answer that escalated). That makes every teacher label's probability known by design, so the ledger replay and the savings receipts can weight audited rows by `1 / label_p`. If drawn, the teacher answers the same request under the audit's parameters: temperature 0, without the client's `top_p`, `seed`, penalties, `logit_bias` or logprobs (`_audit` → `_call_teacher(role="audit")`). Then: - **The verdict.** `task.agree(student answer, teacher answer)` compares them on the task's fields. For `ticket-triage` that is `category` and `priority`; a task without fields compares whole answers, as canonical JSON or as lower-cased text with whitespace collapsed. The row gets `audited = 1` and `audit_agree` 1 or 0. - **A training example too.** The audit's teacher answer goes through the same storage rules as an escalation's, so the money spent on the check also buys a label. - **A better baseline.** The request's all-teacher baseline in the ledger (`baseline_cost_usd`) becomes what the audit actually cost, instead of the token-formula estimate. Audited rows are how the savings receipts calibrate that estimate for unaudited ones. - **Not an audit** when the teacher's answer is cut off, empty or invalid for the task: it is priced, but measures nothing (`audited = 0`). Agreement with Nemotron 3 Ultra, not ground truth. If the teacher is wrong and the student copies it, the audit says "agree". Precision everywhere in Ratchet means this agreement. ### How audits drive the release gate Audits are the release gate's live signal. The gate counts, per version, the audits of answers that version served ([`ratchet/release.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/release.py), defaults from `ReleasePolicy`): | Decision | Rule (default) | |---|---| | Canary moves from 10% to 50%, then to live | At least `min_audits` (20; 15 in mock mode) audits **in the current stage**, and audited precision at or above `live_floor` (0.92) | | Canary holds | Enough audits, but precision below 0.92 without enough evidence to roll back | | Canary is rolled back | At least 20 audits and the 95% Wilson upper bound of audited precision below 0.92. Traffic returns to the live version, or to the teacher on a cold start | | Opt-in sequential rule (`RATCHET_SEQUENTIAL=1`) | Rollback when the e-value against "precision at least 0.92" over every audit of the canary's life reaches 20, with no minimum count; a stage with 400 audits and no decision retires the canary (`inconclusive`) | | Live version alert | The e-value against "precision at least 0.92" over its audits since it went live reaches 100 (`alpha_alert` 0.01): one `alert` event, no automatic action | | Per-task alert | The same rule per task for the live version and the canary: one `alert` per version and task | Only audits with a verdict count: a pending off-path audit counts in neither the number of audits nor the precision. See [The release gate](/guides/release-gate/) for the whole state machine. An operator can hold releases by setting the runtime audit rate to 0: no new canary starts, because it could neither advance nor roll back (setting 0 during a canary needs `force`). In live mode a configured rate below 1% is a startup warning, for the same reason. Human feedback is different. Verdicts sent to `POST /v1/ratchet/feedback` are reported and alerted on, but never promote or roll back a version: clients report what they notice, so that sample is biased. Audits are random by design. ### Audits off the request path An audit is a whole teacher call, seconds against a few hundred milliseconds for the student. In live mode the client should not wait for it, so the audit runs after the response on a small worker pool, the sidecar:
```mermaid sequenceDiagram autonumber participant C as Client participant G as Gateway participant Q as Sidecar participant Teacher as Teacher participant Ledger as Ledger Note over G: the student serves,
an audit is drawn G->>Ledger: row, audited 1,
verdict pending G-->>C: answer,
overhead_pending G->>Q: audit job Q->>Teacher: same request,
temperature 0 Teacher-->>Q: answer Note over Q: agree? valid? Q->>Ledger: the example, if valid Q->>Ledger: the verdict and the cost Note over Teacher,Ledger: the gate counts only
rows with a verdict ```
**Reading this diagram.** The client gets its answer at step 2, before the teacher is called at step 4. The row is written first with the verdict marked pending (1), then the sidecar fills it in once (6, 7). Because the gate counts only rows with a verdict, a request whose audit has not finished yet cannot move a release in either direction.
### When the teacher or the student is down Every upstream call goes through a circuit breaker. The student and the teacher each have their own, and the shadow has a third, so a failing candidate cannot close the live student's path.
```mermaid stateDiagram-v2 direction LR [*] --> Closed Closed --> Open: 5 failures in a row Open --> HalfOpen: after the wait, a probe HalfOpen --> Closed: probe succeeds HalfOpen --> Open: probe fails, wait doubles state "Half open" as HalfOpen class Closed ok class Open danger class HalfOpen pending ```
**Reading this diagram.** While a breaker is closed, calls go through, and a success resets the count of failures. Five failures in a row open it: calls are refused at once, without waiting for a timeout. After a wait, one probe call is let through. A good probe closes the breaker, a bad one reopens it with twice the wait. For the student the wait starts at 30 s and doubles up to 300 s. The teacher's breaker opens after 5 failures too, and answers requests that need the teacher with a 503 `teacher_circuit_open` and `Retry-After`.
  • Calls go through
  • Calls refused at once
  • One probe
What the request gets when its teacher is not available:
```mermaid flowchart TB A(["An escalation
needs the teacher"]) --> B{{"Teacher answers?"}} B -->|yes| OK["Serve the
teacher's answer"] B -->|"no: retries failed
or breaker open"| C{{"Fallback teacher set?"}} C -->|yes| F["Serve the fallback teacher
stored only if asked"] C -->|no| D{{"RATCHET_ON_TEACHER_FAILURE"}} D -->|"error, the default"| E["A mapped error
502, 503, 504
or 400, 429"] D -->|serve_student| G["Serve the student's answer
route degraded, never stored"] class A neutral class B,C,D gate class OK teacher class F,G outline class E danger ```
**Reading this diagram.** The three outcomes are chosen by the operator, not by the request. The default is an honest error. A licensed fallback teacher (`RATCHET_TEACHER_FALLBACK_MODEL`) can answer instead and keeps its own license in the provenance. Degraded mode serves the student's answer even though it was below the threshold; it is marked in the response (`ratchet.degraded`, `X-Ratchet-Degraded`) and in the route, and is never stored, audited or shadowed. Audits and arbitrations never use the fallback: they fail open as described below. A fallback teacher's answers are stored only with `RATCHET_STORE_FALLBACK_TEACHER=1`; a mapped error is a 502, 503 or 504, or a 400 or 429 when the cause is the client's.
  • The teacher answers
  • Opt-in stand-in
  • An error
### Retries do not pay twice Clients retry. Two mechanisms keep a retry from costing a second teacher call or from counting as a second sighting of the same example.
```mermaid sequenceDiagram autonumber participant C as Client participant G as Gateway participant K as Idempotency
cache C->>G: request,
Idempotency-Key abc G->>K: abc seen for this key? K-->>G: no Note over G: run the cascade once G->>K: keep the answer G-->>C: answer C->>G: retry, same key and body G->>K: abc seen? K-->>G: yes, same body G-->>C: the same answer,
Idempotent-Replayed: true Note over C,G: no cascade, no row, no quota ```
**Reading this diagram.** The first request runs the cascade once (1 to 5); the retry gets the kept answer back (6 to 9) with no model call, no ledger row and no quota used. The cache is per API key and lives in memory only, for `RATCHET_IDEMPOTENCY_TTL_S` (600 s by default), because answers contain request text and are never written to disk. A duplicate that arrives while the first is still running waits for the first's answer. The same key with a different body is a 400 `idempotency_key_reused`, and an error is never kept. Separately, a request marked as a retry (`x-stainless-retry-count` above 0) never adds to an example's `seen` count, and the same API client's repeat within `RATCHET_SEEN_DEDUP_S` (300 s live, 0 in mock mode) counts once, so retries cannot push a golden example to the frequent-request rule.
What keeps it honest: - **The draw stays on the request path**, in the same order (the canary pick, then the audit draw), so the same requests are audited however the work runs. - **A drawn audit is never dropped.** The last tenth of the queue is reserved for jobs that carry an audit; when even that is full, the audit runs before the response, as in mock mode. Dropping audits under load would leave the gate a sample of quiet hours. - **Pending is explicit.** The row is logged with `audited` 1 and `audit_agree` NULL, and `Ledger.update_request` writes the verdict once when the job ends. The response says `"overhead_pending": true`, and its `cost_usd` is the serving cost only; the ledger gets the rest. - **The audit keeps the teacher's full deadline** (180 s). If slow answers, often the hard requests, failed more often, the audited precision would be biased upward. - **At shutdown** the gateway waits up to `RATCHET_SIDECAR_DRAIN_S` (10 s) for queued jobs, then gives up; those rows stay pending, which every statistic ignores. Mock mode (0 sidecar workers by default) and Scout keep audits inline. ### Shadow arbitration While a version is in shadow, it is sampled and scored on every request without serving anything. The reference it is judged against is chosen in this order ([architecture](/reference/architecture/#shadow-reference-and-arbitration)): 1. the teacher's answer, if this request has one (the escalation's, or a counted audit's); 2. otherwise, if the shadow would serve and disagrees with the answer the live student served, the teacher **arbitrates**: it answers the request, that answer becomes the reference, and it is stored as a training example when valid; 3. otherwise the served answer itself, so agreements with the live student are accepted as they are. Disagreements are always settled by the teacher, or counted against the shadow when they cannot be. Once the shadow would have served 100 answers, it needs precision of at least 0.95 and coverage no more than 2 points below the live student's to move to canary. ### When an audit or arbitration fails These calls are optional: the client already has an answer. | Failure | What the client gets | What is recorded | |---|---|---| | The audit call raises | The student's answer, as usual | `audited = 0`, `audit_agree` NULL, nothing billed for the call; `audit_errors` + 1; an `audit_error` event on the 1st, 10th, 100th and 1000th failure, naming the error class and status, never the message; a `failures` row | | The teacher's breaker is open | The student's answer | The audit is skipped, not counted as an error | | The audit answer is cut off or invalid | The student's answer | Priced, not an audit, not stored | | The arbitration fails, or is skipped because this request's audit already failed | The live answer | `shadow_agree` NULL: the gate counts it as a disagreement, and the rejection reason says how many | A failed audit is simply not an audit. Audits are a random sample, so losing one shrinks the sample without biasing it. A failed arbitration is different: arbitration only runs when the shadow disagrees, so dropping those rows would hide exactly the cases where the shadow may be wrong. Counting them against the shadow means a flaky teacher can get a candidate rejected, never promoted. ### What audits cost Each audit is one extra teacher call at the teacher's price, added to the request's `cost_usd` and `teacher_cost_usd` and listed in `usage_breakdown.audit`. At the default rate that is one teacher call for about every 20 answers the student serves, so audits add roughly 5% of what the teacher would have charged for those answers. A shadow adds its own samples on every request, plus any arbitration. Ratchet counts all of this in its savings; nothing is left out of the bill. The teacher's list price in `ratchet/config.py` is $1.00 per million input tokens and $3.00 per million output tokens for Nemotron 3 Ultra (Token Factory list price, Oct 2026, unverified until the day-1 checks pass). ## For engineers | Where | What | |---|---| | [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) `Ratchet.handle` | The escalation branches, the audit draw, `label_p`, `student_agree` | | `Ratchet._audit` | One audit: the call, the verdict, failures (`AuditOutcome`) | | `Ratchet._call_teacher` | The teacher's answer in the `serve`, `audit` or `arbitrate` role, then `_store_example` | | `Ratchet._store_example` | The only caller of `Ledger.add_example` for teacher answers: the storage rules, redaction, provenance | | `Ratchet._shadow`, `Ratchet._arbitrate` | The shadow's samples, its verdict and the arbitration | | `Ratchet._finish_offpath` | The audit and shadow after the response, then `Ledger.update_request` | | `router.answer_problem` | Why a teacher answer can be neither an example nor a reference | | [`ratchet/sidecar.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/sidecar.py) | The bounded queue: `reserve`, `submit`, `drain`, `close` | | [`ratchet/release.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/release.py) | `ReleaseGate.tick`: the canary, rollback and alert rules over audit counts | | Setting | Default | Meaning | |---|---|---| | `RATCHET_AUDIT_RATE` | 0.05 | Audit probability for a served student answer; mock mode uses at least 0.10 | | `RATCHET_CANARY_AUDIT_RATE` | unset | A canary's own audit rate, more than 0 and at most 1 | | `RATCHET_MIN_AUDITS` | 20 (15 in mock mode) | Audits a canary stage needs before it can advance or roll back | | `RATCHET_LIVE_FLOOR` | 0.92 | Audited precision floor for canary and live | | `RATCHET_SEQUENTIAL` | 0 | 1 turns on the anytime-valid canary rule | | `RATCHET_STORE_FALLBACK_TEACHER` | 0 | 1 stores the fallback teacher's answers, under its own license | | `RATCHET_GOLDEN_FRACTION` | 0.10 | Share of new examples held out as golden | | `RATCHET_SIDECAR_WORKERS` | 4 live, 0 mock | Workers for audits and shadows after the response | | `RATCHET_SIDECAR_QUEUE` | 1000 | Jobs that may wait | | `RATCHET_SIDECAR_DRAIN_S` | 10 | Seconds shutdown waits for queued jobs | The `ReleasePolicy` fields read `RATCHET_` (see the [release policy settings](/reference/configuration/#release-policy-ratchetreleasepy)). The live-run harness expects the planned gate profile `RATCHET_SEQUENTIAL=1` and `RATCHET_CANARY_AUDIT_RATE=0.5`. Where to look: each row's `audited`, `audit_agree`, `label_p` and `stored` columns (`GET /ratchet/requests/{id}`); `audit_agreement` per day and each task's audits in `GET /ratchet/stats`; the `evidence` of the canary and the live version (counts, e-value, confidence sequence) in the same payload's `versions` list; events `audit_error`, `skip_example`, `rollback`, `alert`, `sidecar_inline` and `sidecar_dropped` in the release log. `Ratchet.health()` holds the in-process counters (`audit_errors`, `shadow_errors`). ## Common questions ### Is 5% enough to catch a bad model? It is enough to catch a badly broken one quickly, because a canary only serves 10% and then 50% of traffic and needs at least 20 audits per stage before it can move. To get evidence faster, raise `RATCHET_CANARY_AUDIT_RATE` for canaries only, or turn on the sequential rule, which can roll back after a couple of disagreeing audits. ### What if the teacher is wrong? Then the audit agrees with a wrong student, or flags a right one. Ratchet cannot see that: every precision number is agreement with Nemotron 3 Ultra. Human feedback (`POST /v1/ratchet/feedback`) is reported next to it so you can spot a gap, but it never moves a release. ### Do audits slow my requests down? Not in live mode: they run after the response, unless the queue is full, in which case a drawn audit runs before the response rather than being dropped. In mock mode they run inline. ### Why not audit only the answers the student was unsure about? Those already go to the teacher. The point of an audit is to check answers the student was confident about, at random, so the sample describes what users actually got. ### Can I turn audits off? You can set `RATCHET_AUDIT_RATE=0`, but then no canary can collect evidence, so nothing is promoted or rolled back on evidence; live mode warns about any rate below 1%. A runtime audit rate of 0 is treated as an operator hold on releases. ### Are audit answers stored even when a client opted out of storage? No. Under an opt-out the audit still runs (release safety does not depend on storage) and its verdict counts, but nothing is stored. ### Has the audit path run against Token Factory? Not yet. The teacher itself runs on Nebius Token Factory, but no trained student has served on a live ledger, so there has been nothing to audit. As of Oct 29, 2026 the audit path is tested offline (Night School's one run on Nebius, by hand on 28 Oct, graded a student on golden rows and rejected it; nothing was served), including the off-path sidecar, which a test compares row for row with an inline run. ## Related --- # The ledger Source: https://docs.ratchet.dualmindlab.tech/guides/ledger/ The ledger is one SQLite file that logs every request's metadata, keeps every stored teacher answer with its provenance (teacher, license, hash, redactions), and records every adapter version and release decision, so the training data behind an adapter can be checked later. The ledger is the kitchen's **recipe book**, plus an **order log**. The order log gets one line for every order: when it came in, who cooked it, how sure the apprentice was, and what it cost. It never says what the customer asked for. The recipe book only gets a page when the **head chef from the agency** cooks a dish. The page says what was ordered, what the head chef cooked, and a note: "this chef's contract lets you copy this recipe." Before the page is written, a clerk blacks out phone numbers, email addresses and card numbers. (The clerk does not recognise people's names yet, and the page says how many things were blacked out.) The **apprentice chef's** own dishes are never copied into the book. A customer can also say "please don't keep my order": the dish is cooked as usual, but no page is written. And a page can be torn out on request: a stub with a fingerprint stays behind, so the same recipe is never written again. ## How it really works ### One request on its way into the ledger 1. **The request arrives.** The gateway validates it and works out whether it may be stored at all. If the request names an end user (`user` or `safety_identifier`), that value is turned into a keyed hash at once (HMAC-SHA256 under the deployment's key, scoped to the API key's name, 16 hex digits). The raw value goes no further. The broadest storage opt-out that applies is settled here (see [opt-outs](#storage-opt-outs-and-no-train-end-users) below). 2. **The cascade runs** (student first, teacher when needed; see [the request journey](/guides/request-journey/)). Whatever happens, the request gets **one row in `requests`**: task, route, who served it, which version, confidence and the threshold it was held to, the audit and shadow flags, costs (with the student and teacher split), latency, the API key's *name*, the end user's hash, the opt-out reason and whether anything was stored. There is no message text and no answer text in this row. 3. **No teacher answer, nothing more.** When the student served the request and the teacher was not called (no audit drawn, no shadow disagreement to settle), the request row is all that is written. The student's text is not kept anywhere. 4. **Every teacher answer goes through one function**, `Ratchet._store_example`. That covers escalations, cold starts, audits and shadow arbitrations. It is the only caller of `Ledger.add_example` on the request path (feedback corrections and imports have their own single writer, `feedback.store_label`). It asks again whether an opt-out applies, so an end user who opted out while the request waited for the teacher is still kept out. 5. **The answer is checked.** It is skipped (served, but not stored) when its `finish_reason` is not `stop` or `tool_calls`, when it is empty once reasoning is stripped, when it is not valid under the task's spec (a missing field, a label outside the vocabulary), or when the client sent sampling that could choose the label (`logit_bias`, a non-zero penalty, a temperature above 1, a vendor field). Each skip writes a throttled `skip_example` event that names the task and the reason, never the answer. 6. **The text is redacted.** With `RATCHET_REDACT_PII=1` (the default) the messages, the answer and every string in the stored conditions get `[LABEL]` placeholders. The answer is checked once more: if a placeholder broke a structured answer, it is skipped as `invalid after redaction`. The teacher always saw the original request. 7. **The example is stored with its provenance**: the teacher model that wrote it, that teacher's license and license URL, the number of placeholders, the conditions it was answered under (`response_format`, `tools`, `tool_choice`, `stop`), the client and the end user's hash. Its golden or train split comes from a hash of the task and the redacted messages. An identical request bumps `seen` instead of adding a row (a client's retry does not count), and a hash that an erasure tombstoned is refused. 8. **Later, other parts write their own rows.** Night School reads `examples` every night and writes `versions` (with each adapter's bill of materials, signed when a key is set) and `version_leaves`. The release gate moves versions and writes `events`. Retention, erasure and receipts write their own tables. ### The privacy path, in one picture
```mermaid flowchart TB A(["A chat request"]) --> B{{"Did the teacher answer?"}} B -->|no| M[("requests row
metadata only")] B -->|yes| C{{"Opt-out, pinned
or tool request?"}} C -->|yes| M C -->|no| D{{"Complete and valid?"}} D -->|no| M D -->|yes| R["Redact the messages,
answer and conditions"] R --> E[("examples row
with provenance")] E --> S{{"Split by hash"}} S -->|"train, or seen
3 or more times"| T["night/train.jsonl"] S -->|golden| G["Golden eval only"] class A neutral class B,C,D,S gate class M,E store class R neutral ```
**Reading this diagram.** Start at the top with one chat request. Every path writes the metadata row; the arrows into it show the three ways a request ends there and nowhere else: the teacher never answered, an opt-out or a pinned or tool request applies, or the teacher's answer was incomplete or invalid. Only a request that passes all three checks reaches redaction, and only then does text enter the ledger. The last split is by a hash of the redacted text, so it never depends on any model's opinion.
### The main tables
```mermaid flowchart LR EC["example_clients"] --> EX TS["tombstones
and erasures"] --> EX LEAF["version_leaves
committed sets"] --> EX LEAF --> VER FB["feedback
verdicts"] --> EX FB --> REQ NT["no_train
end-user hashes"] -.-> REQ EX[("examples
redacted text")] REQ[("requests
metadata")] --> VER[("versions
adapters, BOMs")] REQ --> RU["daily_rollup
folded sums"] REQ --> RC["receipts
sealed days"] VER -.-> EV["events
release log"] class REQ,EX,VER store class RC,LEAF ok class TS danger ```
**Reading this diagram.** Cylinders are the three core tables, boxes the others, and arrows the column that links them (`request_id`, `example_hash`, `version_id`, `student_version`). `requests` is the busiest table and holds no text; `examples` is the only table with text, redacted. Three tables are special: `version_leaves` and `receipts` (green) can only be appended to, and `tombstones` (red) remembers erased hashes so they can never be stored again. Retention folds old request rows into `daily_rollup`, and each ledger day is sealed into `receipts`. Dotted arrows are not stored links: `no_train` is looked up for each request, and every move of a version is logged in `events`.
  • Core table
  • Append-only
  • Erased hashes
  • Not a stored link
### The evidence tables, append-only Two tables exist so that someone else can check Ratchet's claims later. `version_leaves` holds, for each adapter, one leaf per training and golden example (the example's hash and a hash of its label, teacher, license and redaction count). The adapter's signed BOM carries the Merkle roots over those leaves, so a leaf can be proven to be in, or absent from, the set the adapter was trained on. `receipts` holds one signed savings receipt per ledger day and API client, sealed once the day is over and chained to the previous one. SQLite triggers refuse UPDATE, DELETE and a replacing INSERT on both tables, on every connection, because a changed row would silently break the roots and chains built from it. [Evidence you can check](/guides/evidence/) shows how to verify them offline. ### Human labels: feedback and imports Teachers are not the only source of a stored answer. Every chat response carries `ratchet.request_id`, and the caller can send `POST /v1/ratchet/feedback` with a verdict on one of its own requests (any other id is the same 404 as an unknown one).
```mermaid flowchart TB V(["A verdict on a request id"]) --> K{{"Which verdict?"}} K -->|"correct, or incorrect
with no answer"| R["Counted toward human
agreement, nothing stored"] K -->|"incorrect, with
the right answer"| G{{"Opt-out, pinned task
or corrections off?"}} G -->|yes| R G -->|no| P["Check against the spec,
redact, hash"] I(["python -m ratchet.importer"]) --> A["Attested human labels, or
an open teacher's outputs"] A --> P P --> S[("store_label writes the
example, with provenance")] class V,I neutral class K,G gate class R outline class S store ```
**Reading this diagram.** A verdict is never a label: "correct" stores nothing, because that would be the student's own answer coming back as a training target. Only a correction, a human's own answer, can become an example, and it goes through the same checks and redaction as a teacher's answer. The importer enters at the same point with an attestation from the operator, or with an open teacher's outputs, and refuses closed-model families. A stronger label replaces a weaker one for the same prompt (teacher, then a client's correction, then an operator's import). Human corrections are stored with the provenance `human`; an open teacher's imported outputs keep that teacher and its license. Verdicts feed each task's human agreement and each version's human precision, which are reported and alerted on, and never promote or roll back a version.
  • A check
  • Counted, nothing stored
  • Stored as an example
### Where each kind of data lives, and who can read it
```mermaid flowchart TB A1["An API key
its own lookups,
receipts, feedback"] B1["A dashboard key
stats, cost, BOMs,
redaction preview"] C1["An admin key
erase, export, no-train,
proofs, all receipts"] D[("The ledger file, private
requests: metadata
examples: redacted text")] A1 --> D B1 --> D C1 --> D class D store ```
**Reading this diagram.** The three groups at the top are access levels, from narrowest to widest. An API key sees only what it sent. The dashboard key reads aggregates and never writes. Only an admin key can export, erase or open proofs. All three read from the same private file, which is why the file and its backups must be protected as much as the keys.
### What is stored | Table | One row per | Holds text? | |---|---|---| | `requests` | Gateway call | No. Metadata only: task, route, version, confidence, threshold, costs, flags, key name, end-user hash, opt-out reason | | `examples` | Distinct teacher-labelled request | Yes, **redacted**: the prompt, the teacher's answer and the stored conditions, with teacher, license, license URL, split, `seen`, redaction count, client, end-user hash and `origin` | | `versions` | Adapter | No. Base model, adapter name, train size, dataset hash, eval, state, traffic share and the adapter's signed BOM | | `events` | Release decision or notable event | No. A kind (`trained`, `skip`, `canary`, `rollback`, ...) and a short detail; never request text or keys | | `version_leaves` | Example in an adapter's training or golden set | No. The example hash and a hash of its label, teacher, license and redaction count; append-only | | `receipts` | Ledger day and API client | No. A signed savings receipt over request metadata; append-only | | `spend` | Spend outside requests | No. Scout's runs and Tavily credits keep `usd` 0 here (their model calls are already in `requests`) | | `failures` | Failed call | No. The stage, the exception's class and status, what it spent | | `example_clients`, `no_train`, `tombstones`, `erasures`, `erasure_items`, `daily_rollup`, `feedback` | Governance records | No. Hashes, key names, counts and flags | The `origin` of an example says where it came from. Teacher answers from the gateway are recorded without one and read as `live`; human labels carry `feedback` (a correction) or `import` (an attested file). Each example's provenance travels into Night School's rows and from there into the adapter's [signed BOM](/guides/licenses/), so the license of every training target can be checked offline. ### What is never stored

Teacher answers and human labels only. Training targets are teacher answers or customer-attested human labels, never the student's own answers.

| Never stored | Why | Where it is enforced | |---|---|---| | The student's answers, as training targets or at all | Training on a model's own outputs causes collapse | Only `_store_example` (teacher answers) and `feedback.store_label` (attested human labels) may call `add_example`; a test fails on any other caller | | Pinned tasks' answers | Open-ended tasks (Scout's planner and writer) stay on the teacher for good | `_teacher_only` never calls `add_example`, and Night School leaves out rows of a task that is pinned now | | Tool requests, multimodal requests | A tool request is answered by the teacher alone (route `tools`) until tool-call distillation exists | | | Opted-out requests | The client, its key, the task, the end user or the operator said so | `Ratchet.store_opt_out`, asked again before every write: opt-outs only reduce storage | | Incomplete or invalid teacher answers | A cut-off or malformed answer would teach the wrong thing | `Ratchet._store_example` checks `finish_reason` and the task's spec first | | Answers written under label-altering client sampling | A client may choose its own answer, not the label everyone's student learns | `GenParams.unstorable`, threat model T2 | | Raw end-user identifiers, API keys | Privacy and key safety | Key names and keyed hashes only | | Request text in `requests`, `failures`, logs or `/metrics` | Metadata is enough to run the system | `requests` holds metadata only; logs and metrics carry ids, names and numbers | A fallback teacher's answers are stored only with `RATCHET_STORE_FALLBACK_TEACHER=1`, and then under that model's own license. A request for `n` choices stores only the teacher's choice 0. ### PII redaction `ratchet/redact.py` replaces identifiers with placeholders before anything is stored. It runs in a fixed order, so the specific formats with checksums win over the broad phone pattern: | Placeholder | Matches | Extra check | |---|---|---| | `[SECRET]` | API secrets such as `sk-…`, `AKIA…`, `ghp_…`, `tvly-…`, `xox…-…`, and the token after `Bearer ` | | | `[EMAIL]` | `name@domain.tld` | | | `[IBAN]` | Country code, 2 check digits, then groups | mod-97 | | `[CARD]` | 13 to 19 digits with optional spaces or dashes | Luhn | | `[AADHAAR]` | 12 digits, first digit 2 to 9 | Verhoeff | | `[SSN]` | `123-45-6789` style, invalid areas excluded | | | `[PAN]` | Indian PAN, `ABCPE1234F` style | | | `[IP]` | IPv4 addresses | | | `[PHONE]` | Optional country code and area code, digit groups | 10 to 15 digits | For example, this text: ```text Hi, I am Priya. Email priya.k@example.com, phone +91 98765 43210, card 4111 1111 1111 1111, order 1234. ``` is stored as: ```text Hi, I am Priya. Email [EMAIL], phone [PHONE], card [CARD], order 1234. ``` with `redactions` = 3. The name stays: **names and street addresses are not detected yet**. That needs a named-entity model (Presidio is the planned option), which would add a large dependency and latency, so v0 uses patterns. Redaction covers message content (strings and text parts), `name`, an assistant's `refusal`, tool and function call arguments, and every string value in the stored conditions. It is **on by default**; `RATCHET_REDACT_PII=0` stores examples verbatim. You can test it on your own text with `POST /ratchet/redact` or the dashboard's **Try the redactor** box; nothing is stored by the preview. By default a placeholder is just the label (`[EMAIL]`, `RATCHET_REDACT_STYLE=label`). Extraction tasks need to know which of two addresses an answer names, so a task can ask for another style (`privacy.redact_style` in its spec, or the setting for all tasks): `numbered` writes `[EMAIL_1]` and `[EMAIL_2]`, one number per distinct value across the messages, the conditions and the answer; `surrogate` writes reserved look-alike values such as `person1@example.com`. Indian GSTINs and UPI ids are matched too (as `[GSTIN]` and `[UPI]`), and `RATCHET_REDACT_PLUGIN` (`module:function`) can add a detector for what patterns cannot find, such as names. If the plugin fails when an example is being stored, nothing is stored. Two consequences worth knowing. The example hash, the golden split and the frequent-request count are computed on the redacted text, so two requests that differ only in an email address become one example. And the student learns to write a placeholder where the teacher repeated an identifier from the request; for label tasks that does not matter, for free-text replies it can. ### Storage opt-outs and no-train end users Five opt-outs, broadest first. The first one that applies is logged in `requests.store_opt_out` and returned in the response as `ratchet.store_opt_out`: | Reason | Set by | |---|---| | `global` | `RATCHET_STORE_EXAMPLES=0` (zero retention; Night School then skips every night) | | `task` | `store: false` in the task's spec, or `RATCHET_NO_STORE_TASKS` | | `key` | `store=0` for the key's name in `RATCHET_KEY_LIMITS` | | `request` | `store: false`, the header `X-Ratchet-Store: none` (or `0`, `false`), or `metadata.ratchet_store` set to `"false"` (or `"0"`) | | `end_user` | A no-train registration: `POST /v1/ratchet/no-train` with `{"end_user": "..."}` | An opted-out request is handled exactly like any other: the student may serve it, audits still run and shadows are still arbitrated, so release decisions never depend on what was stored. Only the write to `examples` is skipped, and a duplicate does not bump `seen`. Nothing in a request can undo an opt-out: `store: true` is the default spelled out. An unknown header value is a 400, so a typo cannot store by accident. Ratchet's default is to store, the opposite of OpenAI's `store` default, because distilling is its purpose. A no-train registration keeps only the hash, scoped to the caller's API key name, so the same identifier from two clients is two people as far as the ledger can tell. An admin key can register an end user for every client at once (scope `*`), which also covers keys added later. `DELETE /v1/ratchet/no-train` removes a registration (an API key cannot remove one an admin made). Events record counts and names, never the end user or its hash. ### Retention, tombstones, erasure and export Opt-outs keep new data out; these remove or hand over what is there. - **Tombstones, not deletes.** A purge sets an example's `messages`, `completion` and `params` to NULL and keeps the row: hash, task, split, `seen`, teacher, license and client stay. That matters because Night School's rules read more than text: the frequent-request count adds `seen` over every variant of a prompt, and the new-data count compares with past versions. A purged row is in neither training set, but its `seen` still counts, so a purge never moves another variant of a frequent request back to golden. - **Retention windows.** `RATCHET_EXAMPLE_RETENTION_DAYS` purges examples not seen again for that many days (reason `retention`; a later storable duplicate brings the text back). `RATCHET_REQUEST_RETENTION_DAYS` folds old request rows into `daily_rollup` sums and deletes them; every statistic adds the sums back, so figures stay the same except the count of distinct end users. Rows that a release in flight still needs as evidence are never folded. Both default to 0 (keep forever). Retention runs at most hourly in the live gateway on a background thread, at the start of every live night, and on demand with `python -m ratchet.live purge`. - **Erasure is final.** `privacy.erase` selects by `client`, `end_user`, `hashes` or `contains` (each narrows the scope), purges the text and the end-user hash, and writes each hash to `tombstones`: `add_example` refuses it from then on, from any client. The record (`erasures`, `erasure_items`, an `erase` event) holds hashes, counts, names and a redacted reason, never text. `RATCHET_ERASURE_SHARED` decides what happens to an example another client also sent (`delete`, the default, or `keep` and unlink). Admin erasures are dry runs unless asked otherwise. - **Adapters are reported, never edited.** An erasure lists `affected_versions`: exactly, from a version's committed training set when it has one; otherwise every version created after the example was stored while it counted as a training row. Their signed BOMs stay as written. Ratchet cannot remove what an existing adapter memorised; the next night trains without the purged rows, and retiring an affected version is the operator's decision. - **Export.** `GET /ratchet/admin/export/{kind}` (admin key) or `python -m ratchet.admin export` streams NDJSON: examples that still have text with their provenance and every client that sent them, request metadata, versions with their BOMs, events, erasures. End-user hashes are left out. An erased example goes through a short life cycle:
```mermaid stateDiagram-v2 [*] --> Stored: a teacher answer Stored --> Purged: retention Purged --> Stored: a duplicate Stored --> Erased: erased by client,
end user, hash or text Purged --> Erased: erased note right of Erased add_example refuses the hash from any client end note class Stored ok class Purged pending class Erased danger ```
**Reading this diagram.** A purge is reversible in one way only: if the same request arrives again and may be stored, its text comes back. An erasure is not. From the moment the hash is a tombstone, no client can store it again, and the row keeps only hashes and counts. The adapters that may have learned from it are reported, but their signed BOMs are never rewritten.
  • Text kept
  • Text purged, may come back
  • Erased for good
### What can go wrong | Failure | What the system does | Where you see it | |---|---|---| | A teacher answer is cut off, empty or invalid for the task | Served, not stored | `skip_example` event, `requests.stored = 0` | | A placeholder breaks a structured answer | Served, not stored | `skip_example` with `invalid after redaction` | | The redaction plugin raises at store time | Nothing is stored for that request | `skip_example`, the request still succeeds | | An erased request is sent again | Served as usual, never stored | `requests.stored = 0` | | A name or street address is in the text | It is stored as written (not detected by patterns) | Use an opt-out, a retention window or a detector plugin | | Two gateway processes share one ledger | Not supported; caches and the gate are per process | Run one worker | | Power cut with `RATCHET_DB_SYNC=NORMAL` | The last commits may be lost, the file is intact | Use `FULL` if that matters | | Something tries to change a leaf or a receipt | The trigger refuses it | A SQLite error | ### SQLite, WAL and one process The ledger is a single SQLite file (`RATCHET_DB`, default `ratchet.db`; mock mode uses an in-memory ledger) behind the `Ledger` class. Every write commits at once. One re-entrant lock serialises writes, because Scout's agents call the router from several threads over one connection. A file ledger runs in SQLite's WAL journal with `synchronous=NORMAL` (`RATCHET_DB_SYNC=NORMAL`, the default). Readers no longer block the writer, dashboard reads use a second, read-only connection, and the gateway and a `python -m ratchet.live` cron job on the same file only contend writer against writer for one commit. The price: a power cut or OS crash can lose the last commits (the file is not corrupted). WAL needs every process on the same host with the file on a local disk; never put `RATCHET_DB` on the bucket mount or another network filesystem (if you must, set `RATCHET_DB_SYNC=FULL` for every process). Run the gateway with **one uvicorn worker**. The ledger's caches and the release gate are per process, and SQLite is not designed here for several writers. A Postgres ledger is planned. ### Backups The ledger is the source of truth for provenance, versions and training data. Back it up daily with SQLite's backup API, never `cp`: in WAL mode recent commits live in `ratchet.db-wal` until a checkpoint, and a copy of the main file alone misses them. ```bash sqlite3 /var/lib/ratchet/ratchet.db ".backup '/var/backups/ratchet-$(date +%F).db'" ``` In the Nebius VM kit, `deploy/vm/backup.sh` does the same with an integrity check, keeps 14 copies and can upload them to Object Storage. To restore: stop the gateway and cron jobs, remove any `ratchet.db-wal` and `ratchet.db-shm`, copy the backup over `RATCHET_DB`, start the gateway, then reload the live adapter. A backup taken before an erasure still holds the erased text, so run every later erasure again after a restore. See the [runbook](/reference/runbook/#backups). ## For engineers ### Code | File | What to read | |---|---| | [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py) | `SCHEMA` (every table), `MIGRATIONS`, `INDEXES`; `Ledger.add_example` (dedup, split, tombstone refusal, retries), `_split`, `log_request`, `purge_examples`, `record_erasure`, `fold_requests`, `export_page`, `bill_of_materials` | | [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) | `Ratchet._store_example` (the checks, redaction and provenance), `Ratchet.store_opt_out` | | [`ratchet/redact.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/redact.py) | `PATTERNS`, `redact`, `redact_messages`, `redact_json` | | [`ratchet/privacy.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/privacy.py) | `OPT_OUTS`, `EndUsers.hash`, `erase`, `apply_retention`, `export_lines` | | [`ratchet/feedback.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/feedback.py) | `store_label`, the only writer of human and imported labels | Schema changes add the column to `SCHEMA` **and** `MIGRATIONS`; an old file is migrated on open. `version_leaves` and `receipts` are append-only: triggers refuse UPDATE, DELETE and a replacing INSERT on every connection. ### Settings | Variable | Default | Meaning | |---|---|---| | `RATCHET_DB` | `ratchet.db` | The ledger file (live mode) | | `RATCHET_DB_SYNC` | `NORMAL` | `NORMAL`: WAL with `synchronous=NORMAL`. `FULL`: rollback journal with `synchronous=FULL` | | `RATCHET_GOLDEN_FRACTION` | `0.10` | Share of new examples assigned to the golden split (above 0, at most 0.5) | | `RATCHET_REDACT_PII` | `1` | Redact before storing; `0` stores verbatim | | `RATCHET_REDACT_STYLE` | `label` | `label`, `numbered` or `surrogate` placeholders | | `RATCHET_REDACT_PLUGIN` | empty | `module:function` detector for names and addresses | | `RATCHET_STORE_EXAMPLES` | `1` | `0` is zero retention: nothing stored, Night School skips | | `RATCHET_NO_STORE_TASKS` | empty | Comma-separated tasks whose answers are never stored | | `RATCHET_KEY_LIMITS` | empty | Per-key settings by key name; `store=0` keeps a key's requests out | | `RATCHET_END_USER_KEY` | empty | HMAC key for end-user hashes (16+ characters). Empty: a random key is generated once and kept in the ledger's `meta` table | | `RATCHET_EXAMPLE_RETENTION_DAYS` | `0` | Days an example is kept after it was last seen; 0 keeps it | | `RATCHET_REQUEST_RETENTION_DAYS` | `0` | Days a request row is kept before folding; 0 keeps it | | `RATCHET_ERASURE_SHARED` | `delete` | `delete` or `keep` for examples another client also sent | | `RATCHET_SEEN_DEDUP_S` | 300 s live, 0 mock | A repeat from the same client within this window counts once toward `seen` | | `RATCHET_STORE_FALLBACK_TEACHER` | `0` | `1` stores a fallback teacher's answers, under its own license | | `RATCHET_FEEDBACK_CORRECTIONS` | `1` | `0` keeps verdicts but stores no human correction | Full list: [Configuration, privacy and security](/reference/configuration/#privacy-and-security). ### Endpoints | Endpoint | Access | What | |---|---|---| | `GET /ratchet/bom` | admin or dashboard key in live mode; public with `RATCHET_PUBLIC_BOM=1` | Ledger-wide bill of materials: counts by teacher, license and split, `examples_sha256`, `purged_examples` | | `GET /ratchet/versions/{id}/bom` | as above | One adapter's signed BOM | | `POST /ratchet/redact` | dashboard | Preview redaction; stores nothing | | `POST` and `DELETE /v1/ratchet/no-train` | API key (admin for every client) | Register or remove a no-train end user | | `DELETE /v1/ratchet/examples` | API key (admin for any client) | Erase one end user's stored examples | | `POST /ratchet/admin/privacy/erase` | admin | Erase by client, end user, hashes or text; dry run by default | | `GET /ratchet/admin/export/{kind}` | admin | NDJSON export of `examples`, `requests`, `versions`, `events` or `erasures` | Details: [End users and storage](/reference/api/#end-users-and-storage) and [Erasure and export](/reference/api/#erasure-and-export). ### CLI ```bash python -m ratchet.admin erase --reason "contract ended" --client acme # dry run: counts and affected versions python -m ratchet.admin erase --reason "contract ended" --client acme --apply # do it python -m ratchet.admin export examples --client acme --out acme.ndjson python -m ratchet.live purge --dry-run # what the retention windows would remove ``` ### Events `skip_example` (a teacher answer not stored, with the reason), `no_train`, `no_train_unmatched`, `end_user_key`, `erase`, `purge`, `purge_failed`. Meanings are in the [runbook](/reference/runbook/). ## Common questions ### Does Ratchet keep my users' prompts? Only the prompts of requests the teacher answered (an escalation, a cold start, an audit or an arbitration), only when no opt-out applies, and only after redaction. Requests the student served without an audit leave a metadata row and nothing else. ### Is the student's answer ever stored? No. Not as a training target, and not as text anywhere. Training on the student's own outputs would make it collapse onto its own mistakes. ### What about names and street addresses? They are not caught by the redactor yet. If that matters for your traffic, use an opt-out for those requests or tasks, set a retention window, and erase anything that leaked ([PII leak playbook](/reference/runbook/#pii-leak)). Named-entity detection is planned. ### A customer asks us to delete their data. Can we? Yes. Erase by client or end user; the text is gone for good and its hash can never be stored again. The adapters that may have been trained on it are listed, but Ratchet cannot make an existing adapter forget. The next night trains without the rows, and taking an affected version out of service is your decision. Backups made before the erasure still hold the text. ### Can I run two gateway workers on one ledger? No. Run one gateway process with one worker. Night School's cron job may share the file on the same host. A Postgres ledger for several workers is planned, not built. ### Does request retention change my statistics? No, except the count of distinct end users. Folded rows become daily sums that every statistic adds back, and rows a release in flight still needs are never folded. ### Can someone tell whether a text was stored or erased? Not from a live gateway's public endpoints: the training-set proof endpoints need an admin key, and a keyless call to the erasure endpoint gets its counts as `null`. Whoever holds both the database file and a candidate text can test it against the stored hashes, which is also what makes erasures checkable. Keep the file and its backups private, and set `RATCHET_END_USER_KEY` outside the file. ## Related More detail: the [threat model](/reference/threat-model/) and the [architecture's data model](/reference/architecture/#4-data-model-sqlite). --- # Night School Source: https://docs.ratchet.dualmindlab.tech/guides/night-school/ Night School is the nightly job that trains from the base model on every teacher answer stored so far, grades it on a sealed golden set it never studied, and hands it to the release gate. Every night after the restaurant closes, the **apprentice chef** goes to **night school**. The apprentice does not just read today's new pages. They study the **whole recipe book** from the start, and write a brand new **notebook of techniques**. Starting fresh each night means yesterday's mistakes do not pile up. Some recipes are kept in a **sealed exam** envelope. The apprentice never studies them. After studying, the apprentice takes the exam: half of the exam decides how sure the apprentice must be before serving a dish, the other half grades them. One rule bends this: a dish customers order **three or more times** is too important to keep sealed, so it moves into the study pile. Night school is skipped when there is nothing new to learn, when the new apprentice from last night is still on a **trial shift**, or when the manager says "no class tonight". If the apprentice passes the exam, they start their trial shift. If not, the notebook goes in the bin. Training rows and the sealed golden rows, split by a hash of each request. A fresh adapter on every training row so far, never on student answers. Even rows set the threshold, odd rows it never saw grade it. A bill of materials, signed when a key is set, then shadow, canary or rejected. ## How it really works One night is `run_night()` in [`ratchet/nightschool/orchestrate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/orchestrate.py). In live mode a cron job starts it with `python -m ratchet.live night`. Each check below can end the night early with a `skip` event in the release log.
```mermaid flowchart TB start(["02:30 · ratchet.live night"]) --> pre{{"Paused, storage off,
or a release in flight?"}} pre -->|yes| skip(["skip event
nothing trained"]) pre -->|no| build["Build the dataset
golden split by hash,
seen 3+ times: train"] build --> enough{{"50+ new training rows
and golden rows?"}} enough -->|no| skip enough -->|yes| champ["Re-score the live version
on tonight's golden set"] champ --> train["Train a fresh LoRA
from the base, every row"] train --> evalx["Golden eval: even rows
calibrate, odd rows grade"] evalx --> admit{{"Sign the BOM,
then gate.admit"}} admit -->|"below the bar"| rej["rejected"] admit -->|"live version exists"| sh["shadow"] admit -->|"cold start"| can["canary 10%"] class pre,enough,admit gate class skip outline class start,build,champ,evalx neutral class train,sh,can student class rej danger ```
**Reading this diagram.** Two checks can end the night before any GPU time: the first (training paused, storage turned off, or a version still in shadow or canary) and the data check (fewer than 50 new training rows, or no golden rows yet). The dataset step is where the golden split happens: a row's hash puts about 10% of examples in the sealed golden set, and the FREQUENT rule moves a request seen 3 or more times into training. The live version, the champion, is graded again on tonight's golden rows before the challenger trains, so both are compared on the same data. The new adapter is trained from the base model on every training row, calibrated on the even golden rows and graded on the odd ones it never saw. The release gate's `admit` decides what happens next. Not drawn: the retention purge right after the pause check, the Merkle commitment before training, and the paperwork a live night writes back to the bucket.
1. **Training paused?** If an operator turned on `training_paused` (`POST /ratchet/admin/training/pause` or `python -m ratchet.admin flag training_paused on`), the night writes `skip` (`training paused by an admin; no training tonight`) and stops before anything else. 2. **Retention first.** In live mode, if a retention window is set, expired example text is purged now, so the night never reads text that should be gone ([the ledger](/guides/ledger/)). 3. **Storage off?** With `RATCHET_STORE_EXAMPLES=0` (zero retention) the night skips (`storage disabled`). Examples kept from before are not trained on. One thing still runs when no release is in flight: a task whose spec changed is re-scored on the stored golden rows, so it does not stay on the teacher for good. 4. **Release in flight?** If any version is in `shadow` or `canary`, the night skips (`vN still in shadow; no training tonight (saves GPU time)`). A candidate trained now could not be released anyway, and the GPU time would be wasted. 5. **Build the dataset** (`dataset.build`): the training rows and the golden rows, with the license check and every rule in [Rows left out of training](#rows-left-out-of-training) below. 6. **Enough new data?** New examples = training rows now, minus the most training rows any past version was trained on that are still stored (`Ledger.train_rows_kept`, so a deletion neither stops nor starts training). Below **50** (`min_new_examples`), the night skips (`only N new examples ... no training tonight`). With no golden rows at all, it skips too (`no golden examples yet`). A skipped night still re-scores the live version if a task's spec changed since it was scored. 7. **Re-score the champion.** The live version is evaluated again on **tonight's** golden set, under tonight's task specs, and its eval is overwritten. The candidate will be compared with this score, not with an old one (champion against challenger on the same data). 8. **Commit to the rows.** The training and golden sets are each turned into a Merkle root (`dataset.commitment`), before the paid job starts, so the signed BOM can later prove what was and was not trained on ([evidence](/guides/evidence/)). 9. **Train.** `trainer.train(train_rows, base_model, "student-vN")` trains a new adapter **from the base model** on **every** training row so far. In live mode the wall-clock time is recorded as a `training_job` compute interval for the cost report, also when training fails. 10. **Register the candidate.** A `candidate` version is created and its committed leaves are written once (append-only). 11. **Golden eval.** The candidate is sampled on each golden row exactly as production samples it (3 samples at temperature 0.7 with token logprobs). The confidence threshold is calibrated on the **even** rows (target precision 0.97), and precision and coverage are measured on the **odd** rows it never saw. The eval also carries an exact precision certificate, and, when a version is live, a row-by-row comparison with tonight's re-score. Free-text tasks also get a judge score from , recorded as `judge_pass_rate`; it does not gate the release. 12. **Sign the BOM.** The adapter's bill of materials is written from the exact rows: sources and licenses, the base model's license, the NOTICE.md text, the eval, the set roots. It is signed with Ed25519 when `RATCHET_BOM_SIGNING_KEY` is set. 13. **Admit.** A `trained` event is written, then `gate.admit(vN)` decides: **rejected**, **shadow** (a live version exists), or **canary 10%** (cold start, no live version). See [the release gate](/guides/release-gate/). 14. **Ship the paperwork (live mode).** `python -m ratchet.live night` writes the BOM and NOTICE.md into the adapter's folder and runs `RATCHET_SYNC_CMD` again, so the bucket copy of the adapter carries its licenses. A failed sync is a `sync_failed` event and a non-zero exit; the version still stands. ### One live night, step by step
```mermaid sequenceDiagram autonumber participant Night as Gateway host
live night participant Student as Student
endpoint participant Bucket as Bucket
at /data participant Job as Serverless
Job, L40S participant Judge as Judge
Super Note over Night: 02:30: checks,
then the rows Night->>Student: re-score live Night->>Bucket: night/train.jsonl Night->>Job: nebius ai job create Job->>Bucket: student-vN/ Night->>Student: load_lora_adapter Night->>Student: golden eval Night->>Judge: free-text rows Note over Night: candidate, BOM,
gate.admit Night->>Bucket: BOM, NOTICE.md ```
**Reading this diagram.** Everything starts on the gateway host, where cron runs `python -m ratchet.live night` next to the ledger and the release gate. The live version is graded again first, on the student endpoint (1). Then the night uploads the training file to the bucket (2) and starts a Serverless Job (3), which reads it from `/data` and writes the new adapter's folder back (4); `deploy/train_job.sh` waits until the weights are there. The night tells vLLM to load the adapter at runtime (5), grades it with 3 samples per golden row (6) and has the judge score free-text rows (7). It records the candidate, its eval and its signed bill of materials in the ledger and calls the gate. Last, it writes the BOM and the signed `NOTICE.md` into the adapter's folder and syncs the bucket again (8).
The live commands, file paths and polling are on [Training on your data](/guides/training-on-your-data/). ### The golden set: the sealed exam

Golden examples are never trained on. The one exception is a request seen 3 or more times, which moves to training and leaves the golden set.

- **Split by hash.** When an example is first stored, its split is fixed: `golden` if the first 8 hex digits of its hash, divided by `0xFFFFFFFF`, are below `RATCHET_GOLDEN_FRACTION` (default 0.10), else `train`. The hash is over the task and the redacted messages, so the split never changes and nothing leaks from re-splitting. - **Variants share a split.** The same prompt stored under other conditions (a `stop`, a JSON schema) is another example, but it lands in the same split. The trainer only sees the messages, so a golden prompt stored again in train would be trained on. - **Never trained on**, with one exception, the FREQUENT rule below. Golden rows are only ever used to calibrate and grade. - **Half calibrates, half grades.** `cascade_eval` picks the lowest threshold whose served answers reach the target precision on the even-indexed golden rows, and reports precision and coverage on the odd-indexed rows. Each task also gets its own threshold; with fewer than `RATCHET_MIN_TASK_GOLDEN` (20) calibration rows it is never below the global one. Thresholds are clamped between 0.5 and 1.01, and 1.01 means "serve nothing". - **Human-verified golden rows.** An operator can import a golden set that is held out for good, even past FREQUENT, with every variant of its prompt (`python -m ratchet.importer ... --golden-only`). The golden set comes from teacher-labelled traffic (escalations, audits, arbitrations), so it leans toward requests the student found hard. Its coverage is a conservative estimate of live coverage, and its precision means agreement with the teacher, not ground truth. ### The FREQUENT rule A request whose `seen` count reaches **3** (`dataset.FREQUENT`) is treated as a training row even if its hash put it in the golden set, and it leaves the golden set. Without this rule the most common questions, if they happened to hash into golden, would never be learned, and the student would keep escalating them. `seen` counts real traffic only. It is summed over every variant of a prompt. A client's retry does not count (the SDK's `x-stainless-retry-count`), the same client repeating a request within `RATCHET_SEEN_DEDUP_S` (300 s in live mode) counts once, and an imported label or a correction never adds an occurrence. Purged rows keep their count, so a deletion never moves a frequent request back to golden. ### What LoRA is **Simply:** the base model, NVIDIA Nemotron 3 Nano 4B, stays frozen. LoRA trains a small set of extra weights that sit beside some of its layers and nudge their output. Those extra weights are the adapter: a few files, cheap to train, quick to load and easy to swap. The adapter is yours; the base stays under NVIDIA's license. **Precisely:** for a frozen weight matrix W, LoRA learns two thin matrices B and A of rank r, and the layer computes W·x + (α / r)·B·A·x. With rank 16 and alpha 32, the update is scaled by 2. Ratchet's training script, [`ratchet/nightschool/train.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/train.py), uses Transformers, PEFT and TRL's `SFTTrainer` with these defaults: | Setting | Default | Flag | |---|---|---| | Base model | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | `--base` | | Epochs | 2 | `--epochs` | | Learning rate | 2e-4 | `--lr` | | LoRA rank | 16 | `--rank` | | LoRA alpha | 32 (twice the rank) | follows `--rank` | | Max sequence length | 2048 tokens | `--max-seq` | | Batch size per device | 8 | `--batch` | | Precision | bf16 | | | Loss | On the answer only (`completion_only_loss=True`) | | | Target modules | `q_proj, k_proj, v_proj, o_proj` (attention), `up_proj, down_proj` (MLP), `in_proj, out_proj` (Mamba) | `--targets` | Nemotron-H mixes attention, Mamba and MLP layers. The default targets include the Mamba projections. Whether vLLM loads and applies LoRA on them is checked on day 1; if it refuses, the fallback is `--targets q_proj,k_proj,v_proj,o_proj,up_proj,down_proj` (`TARGETS_ATTENTION_MLP`). The student endpoint is started with `--max-lora-rank 16`, so a higher rank needs that flag raised too. ### Retrain from the base every night Each night trains a **new** adapter from the **base model** on **all** training rows so far. Adapters are never stacked or continued. - No drift compounds from one adapter to the next, and each version is one dataset snapshot, named by its `dataset_hash`. - Replay of old data is built in, which limits forgetting. - Rollback is simple: the previous adapter is still there, unchanged. - The cost: training time grows with the dataset. Deduplication, sampling or recency-weighted replay are the later fixes. Training targets are teacher answers (and customer-attested human labels), never the student's own outputs. Training on its own outputs would make the student collapse onto its own mistakes. ### Train on the prompt the student is served Nemotron 3 Nano's chat template turns "thinking" on unless told otherwise. The student endpoint serves with thinking off, so its prompt ends in an empty think block. If training rendered the prompt with thinking on, the student would learn answers after a prompt it never sees, and TRL would only log a warning. So `train.jsonl` uses TRL's conversational **prompt-completion** format: - `prompt`: the stored request messages. - `completion`: one assistant message, the teacher's stored answer. - `chat_template_kwargs`: `{"enable_thinking": false}`, the same value the endpoint uses (`--default-chat-template-kwargs`). A test keeps the three copies of this value equal. Before the model loads, `train.py` checks every row (`check_rows`) and stops on one that is not prompt-completion, lacks the arguments, holds a think tag or has no provenance. After the tokenizer loads, it renders every row as TRL will and stops when the prompt is not a prefix of prompt plus answer, or differs from what the endpoint renders (`check_prefix`, `TemplateMismatch`). A drift stops the job in seconds instead of training a quietly worse adapter. ### Rows left out of training `dataset.rows` reads every stored example and applies these rules. The ledger keeps every row; only the training file changes. | Row | Left out of | Why | Counted as | |---|---|---|---| | Purged by retention or erasure | Both sets | No text left; its `seen` still counts toward FREQUENT | | | Task pinned now | Both sets | Pinned tasks never train the student, even when pinned after rows were stored | `excluded_pinned` | | Teacher unknown or not distillable | Both sets | Only distillable provenance is used ([licenses](/guides/licenses/)) | | | Stored with a client's `stop` | Training (stays golden) | `train.py` renders messages only, so the cut answer would teach the bare prompt to stop there | | | Answer invalid under the task's current spec | Training (stays golden; the eval skips an invalid reference) | The spec tightened since it was stored | `skipped_invalid`, `golden_invalid_refs` | | Answer holds a literal `` or `` | Training (stays golden) | The template renders such an answer without the empty think block | `skipped_think_tag` | | Golden-only import, and every variant of its prompt | Training | Held out for good | | Teacher answers that were incomplete or invalid when they arrived were never stored in the first place ([the ledger](/guides/ledger/)). ### Mock trainer and live trainer Both implement the same two interfaces, `Trainer.train(rows, base_model, name)` and `Evaluator.evaluate(adapter, golden_rows)`, and both grade with the same `cascade_eval` and the same student sampling (`confidence.STUDENT_SAMPLING`), so mock, live and production agree on how a student is scored. | | Mock: `sim.mock_models.MockFleet` | Live: `ratchet.live.NebiusTrainer` + `LiveEvaluator` | |---|---|---| | Used by | The tests and the mock gateway | `python -m ratchet.live night` | | Training | Counts each row's template and remembers its texts; no GPU, no weights | Writes `train.jsonl`, syncs it to the bucket, runs a Nebius Serverless Job, hot-loads the adapter into vLLM | | Evaluation | A deterministic mock student | The real adapter on the student endpoint, plus for free text | | Bad deploys | `corrupt_on_deploy` breaks a version on purpose, to test the rollback | Real | ### After admit: the release states Night School ends at `gate.admit`. From there the [release gate](/guides/release-gate/) moves the version on its own, one tick at a time:
```mermaid stateDiagram-v2 state "canary 10%" as c10 state "canary 50%" as c50 [*] --> candidate: a night trains candidate --> rejected: golden gate fails candidate --> shadow: passes, a live version exists candidate --> c10: passes, cold start shadow --> rejected: fails after 100 would-serve answers shadow --> c10: passes after 100 would-serve answers c10 --> c50: 20 audits, precision ≥ 0.92 c50 --> live: 20 audits, precision ≥ 0.92 c10 --> rolled_back: upper bound below 0.92 c50 --> rolled_back: upper bound below 0.92 live --> retired: a canary is promoted class candidate neutral class shadow, c10, c50 student class live ok class rejected, rolled_back danger class retired outline ```
**Reading this diagram.** A candidate is graded on the golden set first: it needs precision of at least 0.95, coverage of at least 0.25 and no regression against the live version's re-score. With a live version it then runs in shadow, answering every request silently, until it has 100 answers it would have served. A canary serves 10%, then 50% of requests; each stage needs 20 audits (15 in mock mode) with audited precision of at least 0.92, and the 50% stage then becomes live while the old live version is retired. A canary whose Wilson upper bound falls below 0.92 is rolled back, and traffic returns to the live version, or to the teacher on a cold start. Not drawn: a shadow or canary is also retired when a newer candidate is admitted (rare, because no night trains during a release), and the opt-in anytime-valid rule (`RATCHET_SEQUENTIAL=1`) decides the canary with an e-value instead. The release gate guide has every rule.
  • Your adapter on trial
  • Serving everyone
  • Stopped
  • Replaced
- **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)). - **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)). - **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). - **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). Live Night School is tested offline (`tests/test_live.py`, against fake endpoints and a fake `nebius` CLI), and it ran end to end once on Nebius, by hand, on 28 Oct 2026; the golden gate rejected that student, so it was never made live (see [It ran on Nebius](#it-ran-on-nebius)). It has not run nightly on live traffic. The three original adapters (two text-to-SQL, one tool-calling) were trained on a borrowed GPU, outside Night School. On 27 Oct 2026 the round-2 recipe was retrained once, by hand, as a Nebius Serverless Job on one preemptible L40S; the nightly loop did not start it. The day-1 checks (`scripts/day1_check.py`) come first. ## It ran on Nebius
**Measured on Nebius, 28 Oct 2026, by hand, once.** Night School ran end to end once on Nebius, by hand (28 Oct 2026): the gateway on a Nebius CPU VM, the training job on a spot L40S, the adapter served from a Nebius Serverless Endpoint. The golden gate rejected it (81.8% precision against the 95% target), so it was never made live. The live site's own gateway is still on Cloudflare and is not connected to that endpoint, and the loop has not run nightly on live traffic. No saving is claimed.
| What | Result | |---|---| | Gateway | A Nebius CPU VM in `eu-west1` (2 vCPU), in live mode | | Data | 4,322 existing Ultra-labelled text-to-SQL rows imported (no new teacher calls, no exam rows): 3,864 to train on, 458 held out as the golden set | | Training | A spot L40S Serverless Job: 2 epochs, 966 steps, 1,478 s; loss about 0.15 to 0.08; token accuracy about 0.975 | | Serving | The adapter (SHA-256 `dc9ad4b9...`) hot-loaded into a Nebius Serverless Endpoint | | Golden gate | SQL compared as text: agreement 17.9%; at the gate's threshold the student would answer 4.8% of questions with 81.8% precision (9 of 11) against the 95% target. **Rejected**, never shadow, canary or live | The caveats, all of them, are in the [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius): four job submissions (an accidental on-demand job that was cancelled, a missing `einops`, a save that failed with `EPERM` and whose weights were recovered, then the job that saved); the night's own evaluation timed out at 15 s, so the gate result comes from running the eval and the gate again on the saved adapter with a 120 s timeout, through a one-off driver; five problems fixed by hand on the VM, now fixed in the repository (spot by default, `einops` pinned, the adapter saved locally and then copied, a clear stop when `aws` is missing, a separate student timeout for the night); and cost estimates (about 2.7 GPU-hours, at most about $4 at list prices) where the Nebius bill decides. ## For engineers ### Code | File | What | |---|---| | [`ratchet/nightschool/orchestrate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/orchestrate.py) | `run_night`, `rescore`, `paired_evidence`, `spec_hashes`, the `Trainer` and `Evaluator` protocols | | [`ratchet/nightschool/dataset.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/dataset.py) | `rows` (every rule above), `FREQUENT`, `effective_split`, `counts_as_train`, `commitment`, `write_training`, `TRAIN_TEMPLATE_KWARGS`, `build` | | [`ratchet/nightschool/evaluate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/evaluate.py) | `cascade_eval` (calibrate on even rows, measure on odd rows, certificate, frontier, per-task records), `LiveEvaluator`, `REFUSED` | | [`ratchet/nightschool/train.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/train.py) | The GPU script: `parse_args`, `check_rows`, `check_prefix`, `columns`, `write_metadata`, `versions` | | [`ratchet/nightschool/notices.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/notices.py) | NOTICE.md rendering | | [`ratchet/live.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/live.py) | `main` (the `night` command), `NebiusTrainer`, `write_boms`, `load_adapter` | | [`sim/mock_models.py`](https://github.com/harshbhx/ratchet/blob/main/sim/mock_models.py) | `MockFleet` | The rules are tested one by one in `tests/test_nightschool.py` (skip reasons, held-out golden rows, variants sharing a split, stop rows held out, cumulative retraining, champion re-score, the certificate, the training rows' template arguments, `train.py`'s loss and checks, the pinned image). ### Settings | Setting | Default | Meaning | |---|---|---| | `min_new_examples` | 50 | Argument of `run_night`; no environment variable | | `RATCHET_GOLDEN_FRACTION` | `0.10` | Share of new examples that go to the golden split | | `RATCHET_SAMPLES` | `3` | Student samples per request, also in the golden eval | | `RATCHET_MIN_TASK_GOLDEN` | `20` | Calibration rows a task needs for its own threshold | | `RATCHET_PER_TASK_CALIBRATION` | `1` | `0` keeps one pooled threshold | | `RATCHET_CALIBRATION` | `point` | `ltt` picks the threshold by Learn-then-Test | | `RATCHET_CERT_TARGET`, `RATCHET_CERT_DELTA` | `0.95`, `0.05` | The certificate's target and error rate | | `RATCHET_JUDGE_MODEL` | `nvidia/nemotron-3-super-120b-a12b` | Judge for free-text tasks | | `RATCHET_STUDENT_BASE_MODEL` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | The base every adapter is trained from | | `RATCHET_BOM_SIGNING_KEY` | empty | Signs each BOM (Ed25519); empty means unsigned with a SHA-256 | | `RATCHET_TRAIN_GPU_USD_PER_HOUR`, `RATCHET_TRAIN_GPUS` | `1.55`, `1` | Prices the training interval for the cost report. The code's default; these docs use 1.35 USD per GPU-hour, Nebius's list price for an L40S. Set the rate on your bill | The live-mode variables (`RATCHET_DATA_DIR`, `RATCHET_SYNC_CMD`, `RATCHET_TRAIN_CMD`, `RATCHET_ADAPTER_ROOT`) are on [Training on your data](/guides/training-on-your-data/). The release policy's thresholds are on [the release gate](/guides/release-gate/). ### Commands ```bash python -m ratchet.live night # one live night (cron) python -m ratchet.admin flag training_paused on --reason "..." # skip nights until turned off ``` ### Events `skip` (with the reason), `rescored`, `rescore_failed`, `trained`, then the gate's `rejected`, `shadow` or `canary` (and `task_disabled` for a task that fails its own check), and in live mode `sync_failed`. Meanings: [runbook, event kinds](/reference/runbook/). ## Common questions ### Why retrain from scratch every night instead of continuing yesterday's adapter? Continuing piles up drift and forgetting, and makes rollback harder. Retraining from the base on the whole set means every adapter is one dataset snapshot and can be reproduced. The price is that training takes longer as the data grows. ### What does a night cost? An estimate from the repo's [deployment guide](/reference/deployment/#4-cost-estimates): an L40S at about $1.35 per GPU-hour on demand ([Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026) for 30 to 60 minutes is about $0.70 to $1.35 per training night. Skipped nights cost nothing. Night School also needs the student endpoint running while it loads and grades the new adapter. Real costs are recorded once it runs: `python -m ratchet.cost report`. ### What happens if training fails halfway? No version is created. The compute interval is closed with a "training failed" note (the GPU time was still paid), the error ends the command with a non-zero exit, and the next night tries again from scratch. ### Why not train while a canary is running? The new adapter could not be released until the current one finishes, and by then a newer night would replace it. Skipping saves GPU money. ### Can the student learn its own mistakes? No. Its answers are never stored. When the teacher audits a served answer, the teacher's answer is stored, not the student's. ### How big must the golden set be? Big enough for the certificate to mean something. With about 200 golden rows, precision is only known to a few points. `python -m ratchet.stats plan --target 0.95 --delta 0.05` says how many held-out rows a certificate needs (59 with no errors, 93 with one, per the README). Shadow and canary exist partly because the golden set is small. ### Has a real adapter been trained yet? Three, outside Night School: two text-to-SQL rounds and one tool-calling round, all trained on a borrowed GPU; the round-2 recipe was later retrained once by hand on Nebius AI Cloud (see [Text-to-SQL](/guides/text-to-sql/) and [Results](/guides/results/)). Night School's loop is tested with mock models and its live path offline; its first real night was a hand-run on Nebius on 28 Oct 2026 and the gate rejected it ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). ## Related The nightly loop in full: [Architecture, nightly loop](/reference/architecture/#3-nightly-loop-night-school). --- # The release gate Source: https://docs.ratchet.dualmindlab.tech/guides/release-gate/ The release gate decides, from held-out golden rows and teacher audits, whether a newly trained adapter may take traffic, how much, and when to send it back, so a bad night never quietly replaces a good student. After night school, the apprentice has a new notebook. Nobody lets it near customers yet. 1. **The sealed exam.** The new notebook is tested on recipes it never studied. At least 95 of every 100 dishes it chooses to send out must match the head chef's, it must be willing to cook at least a quarter of the orders, and it must not do worse than the current apprentice. To keep that fair, the current apprentice re-takes the very same exam the same night. 2. **The trial shift.** The new notebook cooks silently next to the current apprentice. Nothing it cooks leaves the kitchen. When the two disagree, the head chef decides who was right. After 100 dishes it would have sent out, it either passes or goes back. 3. **Serving a few tables first.** It serves 10% of the tables, then half, then all of them. The inspector tastes a few random plates at each step, and each step needs at least 20 tastings. 4. **Sending the apprentice back to class.** If the tastings show quality clearly below 92 of 100, the new notebook is taken off the line automatically and the previous one cooks again. "Clearly" matters. If the inspector tasted 20 plates and 17 were right, that is 85%, which looks bad. But 20 tastings is not many; maybe it was a few unlucky plates. So the inspector asks: **giving the apprentice the benefit of the doubt, how good could it really be?** With 17 of 20, that generous ceiling is about 95%, above 92, so the inspector keeps tasting. With 7 right out of 17, even the generous ceiling is 64%. Even being generous, the score is too low: back to class. The apprentice who already serves every table is never sent back automatically. If the inspector finds a problem there, an alarm goes off and a person decides. ## How it really works
```mermaid sequenceDiagram participant N as Night School participant G as Release gate participant Ledger as Ledger participant R as Request path participant Teacher as Teacher N->>Ledger: version, eval, BOM N->>G: admit Note over G: golden gate, against the
re-scored live version alt passes, a live version exists G->>Ledger: state shadow loop each request R->>G: shadow verdict,
nothing served G->>Teacher: arbitrate a disagreement end G->>Ledger: canary 10% else passes, cold start G->>Ledger: canary 10% end loop each request in the canary R->>Teacher: audit
an answer Teacher-->>G: agree or not end G->>Ledger: 50%, then live,
or rolled_back ```
**Reading this diagram.** Night School hands over a trained candidate and the gate does the rest. The first judge is the golden exam, run the same night for the candidate and the current live version. Then real traffic does the judging: first silently in shadow, then in the canary, where the teacher audits a random sample of what the canary served. Every state change is a write to the ledger, which is why the whole history can be replayed from the events table.
A version moves through a small set of states, stored in the ledger's `versions` table. The golden gate, the shadow decision, the default canary rule and every admin action write their moves with a compare-and-set, so two processes deciding at once make each decision only once.
```mermaid stateDiagram-v2 state "canary 10%" as c10 state "canary 50%" as c50 [*] --> candidate: trained tonight candidate --> rejected: fails the golden gate candidate --> shadow: passes, live exists candidate --> c10: passes, cold start shadow --> rejected: fails on traffic shadow --> c10: holds up c10 --> c50: enough audits c50 --> live: enough audits c10 --> rolled_back: clearly below c50 --> rolled_back: clearly below live --> retired: replaced class candidate neutral class shadow, c10, c50 student class live ok class rejected, rolled_back danger class retired outline ```
**Reading this diagram.** Every version is born as a candidate and ends in one of three resting places: `rejected` (never served anyone), `rolled_back` (served some, then failed its audits) or `retired` (replaced). Only a canary can become live, and the only way out of live is to be replaced: there is no automatic arrow out of `live`, on purpose. "Enough audits" means at least 20 in the stage with audited precision at or above 0.92; "clearly below" means the Wilson upper bound is under 0.92. Not drawn: a shadow or canary is also `retired` when it is replaced or cancelled, or when the opt-in sequential rule finds a stage inconclusive.
  • Your adapter on trial
  • Serving everyone
  • Stopped
  • Replaced
| State | Traffic | Meaning | |---|---|---| | `candidate` | 0 | Trained and scored tonight, not yet judged | | `rejected` | 0 | Failed the golden gate or the shadow | | `shadow` | 0 | Scores real traffic silently next to the live version | | `canary` | 10, then 50 | Serves a share of requests; the teacher audits a sample | | `live` | 100 | Serves every request the student is confident on | | `retired` | 0 | Replaced by a newer version, cancelled by an operator, or (sequential rule only) inconclusive | | `rolled_back` | 0 | Taken off the line because its audits were clearly below the floor | `inconclusive` is an event, not a state: under the opt-in sequential rule, a canary stage that reaches `max_stage_audits` without a decision is `retired` with an `inconclusive` event. Step through the rollout one stage at a time; the numbered list under it keeps every rule. 1. **Golden gate** (`ReleaseGate.admit`, right after training). Read the candidate's held-out numbers from its eval, in this order: - precision at least `min_precision` (0.95), else `rejected`; - coverage at least `min_coverage` (0.25), else `rejected`; - with `RATCHET_MIN_CERTIFIED_PRECISION` set (opt-in), the certified lower bound at least that floor, else `rejected`; - if a version is live: no regression against it. By default, the candidate is rejected if its precision or coverage is more than `tolerance` (0.02) below the live version's. With the opt-in paired gate, a row-by-row test instead (below); - per task: a task that fails its own check is disabled on the candidate (or, with `RATCHET_TASK_MODE=reject`, rejects it). The live version's numbers come from a re-score on the **same golden rows the same night** (champion against challenger), so both are graded on one exam. 2. **First stage.** With no live version (cold start), the candidate goes straight to canary at 10%: there is nothing to shadow against, so the teacher's audits judge it. Otherwise it enters **shadow**. Any older shadow or canary still in flight is retired first (rare, because Night School does not train while a release is in flight). 3. **Shadow** (`_tick_shadow`). On each request the shadow version is sampled and scored like a real student, against its own threshold. Nothing it says is served. Its answer is compared with the teacher's when the teacher answered that request (an escalation or an audit). Otherwise, when it would serve and disagrees with the served answer, the teacher arbitrates, and the arbitration is stored as training data; an arbitration that fails counts as a disagreement. When it agrees with the live student's served answer, that agreement is accepted as it is. Once the shadow would have served `min_shadow` (100) answers: - shadow precision below 0.95: `rejected`; - shadow coverage more than 0.02 below the live student's coverage on the same requests: `rejected`; - otherwise canary at 10%. 4. **Canary** (`_tick_canary`, default Wilson rule). The teacher audits a random share of the canary's served answers. With at least `min_audits` (20) audits **in the current stage**: - if the upper end of the 95% Wilson interval of audited precision is below `live_floor` (0.92): `rolled_back`, and traffic returns to the live version (or the teacher on cold start); - else if the point estimate is below 0.92: hold and keep auditing; - else advance 10 to 50, or 50 to live. The move starts a new stage, so the audit count starts again. 5. **Promotion.** Under the default rule, one transaction (`Ledger.promote_live`) makes the canary live at 100% and retires the old live version, with a `live_retired` event then a `live` event, so two versions are never live. 6. **Live watch** (`_tick_live`). The live version is never rolled back automatically. By default an e-process over its audits since it went live raises one `alert` event per version when the evidence that precision is below 0.92 reaches 100 (`alpha_alert` 0.01). The runbook says what to do. Ticks run from the request path at most every `RATCHET_TICK_INTERVAL_S` seconds (5 in live mode, every request in mock mode) and from cron (`python -m ratchet.live tick`).
```mermaid flowchart TB N(["Night School trains"]) ==> G{{"Golden gate"}} G -->|fail| R["rejected"] G ==>|"pass, live exists"| S["shadow"] G -->|"pass, first version"| C10 S -->|fail| R S ==>|"holds up"| C10["canary at 10%"] C10 ==>|"enough good audits"| C50["canary at 50%"] C50 ==>|"enough good audits"| L(["live at 100%"]) C10 -->|"clearly bad"| RB["rolled_back"] C50 -->|"clearly bad"| RB class N neutral class G gate class S,C10,C50 student class L ok class R,RB danger ```
**Reading this diagram.** The same story as the state diagram, as a pipeline from Night School at the top; the thick path is the way to live. The shadow stage is skipped only on a cold start, when there is no live student to compare with. Both canary stages can end in a rollback, but only the 50% stage can lead to live.
One canary tick under the default rule:
```mermaid flowchart TB A(["Audits in this stage"]) --> N{{"At least min_audits?"}} N -->|yes| W{{"Wilson upper bound
below 0.92?"}} W -->|yes| Back["Roll back"] W -->|no| P{{"Point estimate
below 0.92?"}} P -->|no| F{{"Releases frozen?"}} F -->|no| Up(["Advance a stage,
or promote"]) N -->|no| Hold["Hold:
keep auditing"] P -->|"yes: not enough
evidence"| Hold F -->|yes| Hold class A neutral class N,W,P,F gate class Back danger class Up ok class Hold neutral ```
**Reading this diagram.** Three of the exits lead to the same hold. The only way to move forward is to have enough audits, a point estimate at or above the floor and no operator freeze. A rollback needs the strongest evidence of all: not "the average looks bad" but "even the generous end of the interval is below the floor". A freeze is checked last, so it never blocks a rollback.
## The Wilson bound, precisely The Wilson score interval for `s` agreeing audits out of `n`, with `z = 1.96`: ```text p = s / n denom = 1 + z^2 / n centre = (p + z^2 / (2n)) / denom half = z * sqrt( p(1 - p)/n + z^2/(4n^2) ) / denom bounds = [max(0, centre - half), min(1, centre + half)] ``` It is the two-sided 95% interval; the rule uses its upper end, so a rollback needs strong evidence that the version is worse, not just a bad-looking average. Checked with `stats.wilson`: | Audits agreeing | Point estimate | Wilson upper bound | Decision at a 0.92 floor | |---|---|---|---| | 19 of 20 | 95% | 99.1% | Advance | | 17 of 20 | 85% | 94.8% | Hold: below the floor, but not clearly | | 10 of 20 | 50% | 70.1% | Roll back | A plain point-estimate rule was not used because at 20 audits, a healthy canary at 96% true precision shows a point estimate below 92% about 19% of the time. ## The anytime-valid rule (opt-in) The Wilson interval keeps its error rate for **one look** at a fixed sample. The gate looks after every request. When true precision sits exactly on the floor, checking Wilson after every audit falsely rolls back about 22% of such versions within 400 audits, against a nominal 2.5%. And `min_audits` lets a broken deploy keep serving until it has 15 to 20 audits. **Explained simply.** Think of a running bet against "quality is fine". You start with one coin. Before each audit you bet a little that the answer will be wrong. If quality really is fine, this bet cannot grow on average: it is a fair game or worse for you. So if your pile ever reaches 20 coins, that is strong evidence quality is not fine. Because a fair game cannot grow on average, it does not matter how often you look at your pile: checking after every single audit is allowed. That is what "anytime-valid" means. **Precisely.** With `RATCHET_SEQUENTIAL=1`, `_tick_canary` hands over to `_sequential_canary`, which reads two integers per version, audits `n` and agreements `s` (`Ledger.audit_counts`): 1. **Rollback.** The mixture e-value against "precision is at least the floor", over **every** audit of the canary's life, reaches `1/alpha_rollback` (20). It mixes likelihood ratios over a fixed grid of alternatives below the floor; under the null each audit multiplies it by a factor whose expected value is at most 1, so it is a nonnegative supermartingale and Ville's inequality gives P(it ever reaches 20) at most 5%, however often the gate looks. There is no minimum count: 0 agreeing out of 2 already gives 29.8. 2. **Promotion,** on the current stage's audits once there are `min_audits`: the point estimate at or above the floor, or with `RATCHET_PROMOTE_EVIDENCE=1` an e-value that precision is **above** the floor reaching `1/alpha_promote` (20). That needs 56 agreeing audits per stage at a 0.92 floor (55 give 19.996). 3. **Futility.** A stage with `max_stage_audits` (400) audits and no decision is retired, with an `inconclusive` event. It counts audits, not time. Every event carries the counts, the e-value and its threshold, and an anytime-valid 95% confidence sequence, for example `v3 rolled back: 3/7 audits agree; e-value 64.9 >= 20 (anytime-valid, alpha 5%); 95% CS [0.041, 0.908]; traffic back to live v1`. Anyone can recompute those numbers from the two counts. The sequential canary is opt-in. The **live alert** already uses the e-process by default (`RATCHET_ALERT_EPROCESS=1`, threshold 100). The live-run harness's planned gate profile turns the sequential rule on (`RATCHET_SEQUENTIAL=1`, `RATCHET_CANARY_AUDIT_RATE=0.5`). ## The paired gate (opt-in) The default regression check compares two point estimates within 0.02, but the candidate and the live version were scored on the same rows, so the comparison can be done row by row. With `RATCHET_PAIRED_GATE=1`, the gate reads the candidate's `paired` record against the re-scored live version: - **Coverage:** `b` rows only live serves, `c` rows only the candidate serves, and McNemar's exact one-sided p-value `p_cov = P(X at least b)` with `X ~ Binomial(b + c, 1/2)`. - **Wrong answers as a client sees them:** negative flips (the candidate serves a wrong answer where live was right, served or escalated) against positive flips, with `p_err`. - **Negative flip rate:** answers live served right that the candidate serves wrong, with a 95% Wilson interval. It rejects only a regression that is both large and significant: more than `tolerance` of the rows lost from coverage with `p_cov` below `RATCHET_REGRESSION_ALPHA` (0.05), more than `tolerance` of the rows newly answered wrong with `p_err` below it, or, with `RATCHET_MAX_NFR` below 1, a flip rate whose lower bound is above it. Without a paired record against the current live version it falls back to the default rule. In plain words: only rows where the two versions differ count, and four lost rows with none won is like four coin flips all landing heads, which happens one time in 16. ## Per-task checks When several tasks share one adapter, the golden gate also checks each task with at least `min_task_test_n` (10) held-out rows. A task fails when its held-out precision is below its floor (its spec's `policy.min_precision`, else 0.95), more than `tolerance` below the live version's precision on that task, or unmeasured because it answers none of its held-out rows. - `task_mode` `disable` (default): the task's threshold becomes 1.01 on that candidate, a `task_disabled` event is written, the teacher answers that task, and the version ships for its other tasks. The task stays disabled for the version's life, through every re-score. - `task_mode` `reject`: the candidate is rejected. - Either way, a candidate that would answer less than `min_coverage` of its held-out rows once its failing tasks are off is rejected: it would sit in shadow or canary forever, and no night would train meanwhile. On the live and canary versions, one `alert` per (version, task) fires when that task's audited precision has a Wilson upper bound below 0.92 with at least `min_audits` audits. Nothing is disabled automatically on a live version; pinning a task to the teacher is an operator's decision. ## Two watchers beside the gate: drift and the teacher The gate's tick has two more steps that never promote or roll back a version. They watch for changes the golden exam cannot see. **The drift alarm** (`_tick_drift`) watches each (version, task) stream of the student's confidence scores for a shift, with no labels needed. When it alarms (`RATCHET_DRIFT_ALARM`, default 100, so at most a 1% false alarm per stream over its life), it writes a `drift` event and suspends that version's certificate, because the certificate no longer describes the traffic. Two opt-in actions can follow: a higher audit rate for the alarmed version (`RATCHET_DRIFT_AUDIT_BOOST`) and an earlier night (`RATCHET_DRIFT_EARLY_NIGHT`). How the score is built is in [Confidence](/guides/confidence/#watching-the-confidence-scores-over-time-the-drift-alarm). **The teacher watch** (`_tick_watchers`) asks whether the teacher itself is steady. The golden set and the audits both measure agreement with the teacher, so a teacher whose answers wander would make every number meaningless. Each stored label gets at most one re-test per check epoch (a repeated request that the teacher labels again, or one of the `RATCHET_TEACHER_PROBES` probes Night School sends each night, 20 by default). A disagreement puts the label in quarantine (`contested`): it is left out of both the training and the golden split until Night School adjudicates it, by asking the teacher twice more and keeping the strict majority of three. When the evidence that the teacher disagrees with itself more than `RATCHET_TEACHER_NOISE_MAX` (0.05) reaches 1/`RATCHET_ALPHA_TEACHER` (100), the gate writes a `teacher_drift` event and Night School stops training until an operator acknowledges it with `POST /ratchet/teacher/ack`. A new model id in the teacher's responses starts a new epoch. `GET /ratchet/teacher` shows the state, and `RATCHET_TEACHER_WATCH=0` turns the actions off while the free checks are still recorded.
```mermaid stateDiagram-v2 direction LR [*] --> Steady Steady --> Quarantine: a re-test disagrees Quarantine --> Steady: 2 of 3 settle it Steady --> Frozen: evidence ≥ 1/alpha Frozen --> Steady: an operator acks class Steady ok class Quarantine pending class Frozen danger ```
**Reading this diagram.** One disagreement is not an emergency: it only quarantines one label until two more teacher answers settle it by a majority of 3. Training stops only when many disagreements add up to strong evidence (1/`RATCHET_ALPHA_TEACHER`, 100 by default), and it stays stopped until a person acknowledges it. The freeze affects training, not serving: the live student keeps answering. Not drawn: re-tests that agree leave the watch steady, and a new teacher model id starts a new check epoch.
  • Training as usual
  • One label held out
  • Training stopped
## No training while a release is in flight Night School skips the night while any version is in `shadow` or `canary`, with a `skip` event such as `v4 still in canary; no training tonight (saves GPU time)`. A new candidate would replace the one being judged and waste both the GPU time and the evidence gathered so far. An operator who wants tonight's training can retire the shadow first. ## Operator holds and the admin API Every manual change goes through `ratchet/admin.py`, shared by the HTTP admin API (`/ratchet/admin/*`, admin key from `RATCHET_ADMIN_KEYS`) and the CLI (`python -m ratchet.admin`). Each change needs a reason (3 to 500 characters), is checked against an explicit table of allowed moves, and writes the change, an `admin_actions` row (who, what, why, before and after) and an `admin` event in one transaction. | Switch or action | Effect on the gate | |---|---| | `releases_frozen` | No version starts taking traffic or takes more: no shadow to canary, no canary advance or promotion, and a cold-start candidate shadows the teacher instead of taking 10%. Rollbacks, rejections and the live alert keep running | | `audit_rate` set to 0 | Same hold: a canary without audits could neither advance nor be rolled back, so no new canary starts. Setting 0 during a canary needs `force` | | `student_disabled` | Kill switch: no student or shadow call; the teacher answers everything (route `student_disabled`). No version changes state | | `training_paused` | Every night writes `skip` and trains nothing | | `rollback`, `retire` | Roll a canary back; retire a canary, shadow or live version | | `promote` | A canary to another share or to live; a shadow to canary. A rejected, rolled-back or retired version (or a candidate) to canary only with `force`. Nothing becomes live except a canary, and a version without a golden eval is never promoted | A hold never blocks a rollback, a rejection or an alert. The admin's moves and the gate's (the golden gate, the shadow decision and the default canary rule) use the same compare-and-set (`Ledger.transition`, `Ledger.promote_live`): a move into shadow or canary is refused while another version is in flight, a decision made on a state someone else just changed is dropped, and a candidate an admin moved while it was being scored is left alone with a `conflict` event. In the current code, the opt-in sequential canary rule (`_sequential_canary`) moves versions with `Ledger.set_version`, an unconditional update, rather than the compare-and-set writers, and its promotion writes no `live_retired` event. The "each decision is made once" guarantee therefore covers the default rule, not the sequential one, until that path uses `transition` and `promote_live` too. ## The release log Every decision writes an event (the `events` table, shown in the dashboard's release log and in `GET /ratchet/stats`): | Event | Written when | |---|---| | `trained` | Night School trained and scored a new version | | `skip` | No training tonight (release in flight, too few new examples, paused, no golden set) | | `rejected` | Golden gate or shadow failed, with the reason | | `shadow` | Passed the golden gate; shadowing the live version (appends the paired counts and the certificate when there are any) | | `canary` | Started or advanced a canary stage | | `live_retired`, `live` | The old live version retired; the canary promoted to 100% | | `retired` | An in-flight version replaced by a newer candidate | | `rollback` | The canary rolled back, with the evidence | | `inconclusive` | Sequential rule only: a stage hit `max_stage_audits` without a decision | | `alert` | The live version (or one task of the live or canary version) is clearly below the floor | | `task_disabled` | One task of a candidate failed its own golden check | | `drift` | The drift alarm fired for a (version, task) stream | | `teacher_drift` | The teacher watch froze Night School | | `conflict` | A candidate changed by another process while the gate decided | | `admin` | An operator's change, with the actor and the reason | As of Oct 29, 2026, the release gate is tested with mock models and offline tests, and its golden step decided once on Nebius (28 Oct 2026: rejected). Ratchet's own adapter is not connected to the live gateway, so no shadow, canary or live release has run. Live mode, the cron tick and the admin CLI are tested offline. ## It ran on Nebius, and the gate said no
**Measured on Nebius, 28 Oct 2026, by hand, once.** Night School ran end to end once on Nebius, by hand (28 Oct 2026): the gateway on a Nebius CPU VM, the training job on a spot L40S, the adapter served from a Nebius Serverless Endpoint. The golden gate rejected it (81.8% precision against the 95% target), so it was never made live. The live site's own gateway is still on Cloudflare and is not connected to that endpoint, and the loop has not run nightly on live traffic. No saving is claimed.
| What | Result | |---|---| | Gateway | A Nebius CPU VM in `eu-west1` (2 vCPU), in live mode | | Data | 4,322 existing Ultra-labelled text-to-SQL rows imported (no new teacher calls, no exam rows): 3,864 to train on, 458 held out as the golden set | | Training | A spot L40S Serverless Job: 2 epochs, 966 steps, 1,478 s; loss about 0.15 to 0.08; token accuracy about 0.975 | | Serving | The adapter (SHA-256 `dc9ad4b9...`) hot-loaded into a Nebius Serverless Endpoint | | Golden gate | SQL compared as text: agreement 17.9%; at the gate's threshold the student would answer 4.8% of questions with 81.8% precision (9 of 11) against the 95% target. **Rejected**, never shadow, canary or live | The caveats, all of them, are in the [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius): four job submissions (an accidental on-demand job that was cancelled, a missing `einops`, a save that failed with `EPERM` and whose weights were recovered, then the job that saved); the night's own evaluation timed out at 15 s, so the gate result comes from running the eval and the gate again on the saved adapter with a 120 s timeout, through a one-off driver; five problems fixed by hand on the VM, now fixed in the repository (spot by default, `einops` pinned, the adapter saved locally and then copied, a clear stop when `aws` is missing, a separate student timeout for the night); and cost estimates (about 2.7 GPU-hours, at most about $4 at list prices) where the Nebius bill decides. The gate did its job: a student that is right 81.8% of the time on the few answers it would serve is below a 95% target, so nothing reached shadow, canary or live traffic. ## For engineers | What | Where | |---|---| | The gate | [`ratchet/release.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/release.py): `ReleaseGate.admit`, `tick` (six steps: `_tick_drift`, `_tick_live`, `_tick_tasks`, `_tick_shadow`, `_tick_canary`, `_tick_watchers`), `_sequential_canary`, `task_failures`, `_paired_regression`, `evidence`, `ReleasePolicy`, `CANARY_STEPS` | | Statistics | [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py): `wilson`, `log_e_bernoulli`, `e_bernoulli`, `confidence_sequence`, `mcnemar_one_sided`, `cp_lower` | | Paired comparison | [`ratchet/certificate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/certificate.py): `paired`, `paired_stats` | | Moves and counts | [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py): `transition`, `promote_live`, `transaction`, `audit_counts`, `version_live_quality`, `shadow_quality`, `flags` | | Manual changes | [`ratchet/admin.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/admin.py): `rollback`, `retire`, `promote` (`MOVES`), `set_flag`, `pin_task`, `tick` | | The night that feeds it | [`ratchet/nightschool/orchestrate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/orchestrate.py): `run_night` (champion re-score, paired evidence, BOM, then `admit`) | | Tests | `tests/test_release.py`, `tests/test_admin.py`, `tests/test_stats.py`, `tests/test_calibration.py` | `ReleasePolicy` fields, each set with `RATCHET_` ([Configuration](/reference/configuration/#release-policy-ratchetreleasepy)): | Variable | Default | Meaning | |---|---|---| | `RATCHET_MIN_PRECISION` | 0.95 | Golden gate and shadow precision floor | | `RATCHET_MIN_COVERAGE` | 0.25 | Golden gate coverage floor | | `RATCHET_TOLERANCE` | 0.02 | Largest drop against live (0 to 0.2) | | `RATCHET_MIN_SHADOW` | 100 | Would-serve shadow answers before a decision | | `RATCHET_MIN_AUDITS` | 20 (15 in mock mode) | Audits per canary stage before promotion, and before a Wilson rollback or alert | | `RATCHET_LIVE_FLOOR` | 0.92 | Audited precision floor; must be at most `RATCHET_MIN_PRECISION` | | `RATCHET_MIN_TASK_TEST_N` | 10 | Held-out rows a task needs before its own golden check | | `RATCHET_TASK_MODE` | `disable` | Or `reject` | | `RATCHET_SEQUENTIAL` | 0 | 1: the anytime-valid canary rule | | `RATCHET_ALPHA_ROLLBACK` | 0.05 | Sequential rollback threshold 1/alpha = 20 | | `RATCHET_PROMOTE_EVIDENCE` | 0 | 1: promote on evidence (needs `RATCHET_SEQUENTIAL=1`) | | `RATCHET_ALPHA_PROMOTE` | 0.05 | Evidence-based promotion threshold 20 | | `RATCHET_MAX_STAGE_AUDITS` | 400 | Sequential futility cap; 0 means never | | `RATCHET_ALERT_EPROCESS` | 1 | Live alert by e-process (0: Wilson) | | `RATCHET_ALPHA_ALERT` | 0.01 | Live alert threshold 100 | | `RATCHET_MIN_CERTIFIED_PRECISION` | 0.0 | Certified floor in the golden gate; 0 is off | | `RATCHET_PAIRED_GATE` | 0 | 1: row-by-row regression check | | `RATCHET_REGRESSION_ALPHA` | 0.05 | The paired gate's McNemar level | | `RATCHET_MAX_NFR` | 1.0 | Negative flip rate cap; 1 is off | The drift alarm and the teacher watch add `RATCHET_DRIFT_ALARM` (100), `RATCHET_DRIFT_AUDIT_BOOST` (1), `RATCHET_DRIFT_EARLY_NIGHT` (0), `RATCHET_TEACHER_WATCH` (1), `RATCHET_TEACHER_NOISE_MAX` (0.05), `RATCHET_ALPHA_TEACHER` (0.01), `RATCHET_TEACHER_PROBES` (20) and `RATCHET_ADJUDICATE_MAX` (50). A typo or an out-of-range value stops startup with a message naming the variable. Related settings: `RATCHET_AUDIT_RATE` (0.05; mock mode at least 0.10), `RATCHET_CANARY_AUDIT_RATE` (unset: a canary's own audit rate, so its evidence arrives sooner), `RATCHET_TICK_INTERVAL_S`. Endpoints ([API](/reference/api/)): - `GET /ratchet/stats`: `versions[].evidence` for the canary and live versions (`ReleaseGate.evidence`: the rule in force, counts over the life and the stage, e-values, the confidence sequence, the Wilson interval), `controls` (the switches), and the events. - `GET /ratchet/versions/{id}/certificate`: the golden certificate and the gate's `live` evidence. - `/ratchet/admin/*`: `versions`, `flags`, `audit-log`, `versions/{id}/rollback`, `.../retire`, `.../promote`, `releases/freeze`, `student/disable`, `training/pause`, `PUT settings`, `tasks/{name}/pin`, `tick` ([API: Admin](/reference/api/#admin)). ```bash python -m ratchet.live tick # run the gate now (cron does it every few minutes) python -m ratchet.admin versions # states, traffic, golden numbers, this stage's audits python -m ratchet.admin rollback 7 --reason "refund labels are wrong since 09:00" python -m ratchet.admin flag releases_frozen on --reason "holiday freeze" RATCHET_MODE=mock RATCHET_SEQUENTIAL=1 uvicorn ratchet.gateway:app --port 8000 # a mock gateway with the sequential canary ``` The state machine with every threshold is in [Architecture §5](/reference/architecture/#5-version-state-machine); what to do on each event is in the [runbook](/reference/runbook/), including [reading an e-value event](/reference/runbook/#reading-an-e-value-event) and [manual interventions](/reference/runbook/#manual-interventions). ## Common questions ### How many users get a bad canary's answers before it is rolled back? Some do: that is what a canary is. At 10%, roughly one request in ten can reach it, and only the ones the canary is confident on are served by it. Under the default rule it keeps serving until its stage has `min_audits` audits. A higher canary audit rate (`RATCHET_CANARY_AUDIT_RATE`) shortens that. ### Why is the live version never rolled back automatically? Automatic action on all traffic is a decision Ratchet leaves to a person. The live watch raises an alert; an operator can retire the live version, turn the student off with the kill switch, or restore an older version through the admin API, each in one command. ### How long does a release take? It depends on traffic and the audit rate. The arithmetic: at a 5% audit rate and a 10% canary, 20 audits take about 4,000 or more requests, before the 50% stage needs its own 20. The shadow first needs 100 would-serve answers. Low-traffic deployments should expect slow promotions; audit focusing (`RATCHET_CANARY_AUDIT_RATE`) helps. ### What does an audit cost? One teacher answer per audited request, on top of the student's samples. At the default 5% audit rate that is one teacher call for every 20 served student answers. Audited answers also become training data when valid. ### Could two processes promote the same version twice? Not under the default rule. The gateway's tick, the cron tick and an admin can run at the same time; each move is a compare-and-set inside one SQLite transaction, so the second decision changes nothing and writes nothing. The opt-in sequential canary rule does not use the compare-and-set writers yet (see the note above). ### Do customer complaints (feedback verdicts) roll a version back? No. Human verdicts are reported and can raise a `human_alert`, but the gate never reads them: clients report what they notice, so the sample is biased. Audited precision stays the release signal. ### Where has the gate run so far? On mock models and in offline tests. The whole gate, including the drift alarm and the teacher watch, is ready to run in live mode once a trained student is served. ## What can go wrong | Failure | What the system does | Where you see it | |---|---|---| | The candidate fails the golden gate | `rejected` with the reason; the live version stays | `rejected` event, `GET /ratchet/stats` | | The shadow's teacher arbitration fails | Counted as a disagreement, so a flaky teacher can reject a candidate but never promote one | rejection reason names the count | | The canary's audits are clearly below the floor | `rolled_back`, traffic returns to the live version (or the teacher) | `rollback` event with the evidence | | Two processes decide at once | One compare-and-set wins; the other changes nothing | `conflict` event when a candidate was moved meanwhile | | An operator freezes releases | No version takes more traffic; rollbacks and alerts keep running | `admin` event, `controls` in the stats | | The live version gets worse | An `alert`; no automatic action | `alert` event, then `python -m ratchet.admin rollback` or `student/disable` | | The teacher's labels turn unstable | Contested labels leave training; training freezes at strong evidence | `teacher_drift` event, `GET /ratchet/teacher` | ## Related --- # Training on your data with the student endpoint Source: https://docs.ratchet.dualmindlab.tech/guides/training-on-your-data/ The student endpoint only answers requests; training happens somewhere else: the gateway stores the teacher's answers to your traffic, and every night a Nebius Serverless Job trains , which the endpoint then loads from your bucket. - **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)). - **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)). - **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). - **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). - **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate. - **The nightly schedule on live traffic.** Night School runs by hand today; the timer is scripted. - **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline. - **Live results from your own data.** None are published yet. The three published adapters (two text-to-SQL, one tool-calling) were trained on a borrowed GPU. Run every script with `DRY_RUN=1` first. Run the first-run checks on Token Factory (`scripts/day1_check.py`) before relying on any of it. The **apprentice chef** has a cooking station: the GPU endpoint. The station only **cooks**. The apprentice never studies while cooking orders, and nothing the apprentice cooks is written into the recipe book. Every time the **head chef from the agency** cooks a dish, the pass writes it into the **recipe book** (the ledger), with phone numbers and emails blacked out. At night, a copy of the study pages goes into a **shared locker** (the storage bucket). A separate **classroom** is rented for about an hour (the training job). The apprentice studies the whole book there and writes a new **notebook of techniques** (the LoRA adapter), which is put back in the locker. Then the station picks the new notebook up from the locker and takes the **sealed exam**. If it passes, it starts a trial shift. When the station is **closed to save money**, a different, untrained cook from the agency takes the easy orders, and the menu says so. The notebook is yours: you can take it to another kitchen, with its certificate and the license notes for the apprentice's original training. ## How it really works ### The path from a request to a loaded adapter Checked, redacted, with its teacher and license. night/train.jsonl, uploaded to the bucket. A fresh LoRA from the base, written back to the bucket. Graded on the golden set, then handed to the gate. The thirteen steps in full: 1. **Your app sends a request** to the gateway's `/v1/chat/completions`, ideally with a task name (`X-Ratchet-Task` or `metadata.ratchet_task`). See [the request journey](/guides/request-journey/). 2. **The gateway stores teacher answers, never student answers.** When answers (an escalation, a cold start, a random audit or a shadow arbitration), the answer is checked, the prompt and answer are redacted, and both are stored in the ledger with their provenance: teacher, license, hash, redaction count. When serves an answer, only metadata is logged. Details: [the ledger](/guides/ledger/). 3. **Every night, cron runs `python -m ratchet.live night`** on the gateway's host. It runs Night School's checks (paused, storage off, a release in flight, fewer than 50 new examples, no golden rows) and skips the night if one applies. Otherwise it re-scores the live adapter on tonight's golden rows and goes on. See [Night School](/guides/night-school/). 4. **It writes the training file.** `RATCHET_DATA_DIR/night/train.jsonl` holds one prompt-completion row per training example: the stored messages, the teacher's stored answer, and `chat_template_kwargs` with thinking off, plus the row's task and provenance. Golden rows are not written there; they stay on the gateway's host. 5. **It uploads the file.** `RATCHET_SYNC_CMD` (for example `aws s3 sync ./data s3:// --endpoint-url ...`) copies `RATCHET_DATA_DIR` to the Object Storage bucket. It must only upload, never delete. 6. **It starts the training job.** `RATCHET_TRAIN_CMD` (default `bash deploy/train_job.sh {version}`, with `{version}` = `student-vN`) submits a Nebius Serverless Job: `nebius ai job create`, platform `gpu-l40s-a`, preset `1gpu-8vcpu-32gb`, the bucket mounted at `/data`, a 200 GiB disk and `--timeout 2h`. 7. **The job trains.** In the training image it runs `python -m ratchet.nightschool.train --data /data/night/train.jsonl --base nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --out /data/adapters/student-vN`. It checks every row, writes `versions.json`, checks the chat template, trains the LoRA adapter from the base model, and saves the adapter, the tokenizer files, `ratchet_manifest.json` and `NOTICE.md` into `/data/adapters/student-vN/`. 8. **The script waits for the adapter.** With `RATCHET_BUCKET_S3` set, `train_job.sh` polls the bucket for `adapters/student-vN/adapter_model.safetensors` every 30 seconds, for up to 2 hours, and exits when it appears. Without `RATCHET_BUCKET_S3` it returns right after submitting, so the adapter load that follows would run before the job has written anything: set it when the script is your `RATCHET_TRAIN_CMD`. 9. **The adapter is hot-loaded.** `ratchet.live` calls the endpoint's `POST /v1/load_lora_adapter` with `{"lora_name": "student-vN", "lora_path": "/data/adapters/student-vN"}` (the path is `RATCHET_ADAPTER_ROOT/`). vLLM reads the files from the same bucket, mounted at `/data` in the endpoint. No restart. An "already loaded" answer is fine. 10. **The golden eval runs on the endpoint.** `LiveEvaluator` sends each golden prompt to the new adapter (3 samples, as production does), calibrates the threshold on half the rows and measures precision on the other half. Free-text tasks also get a score from , which is recorded but does not gate. 11. **The adapter's BOM is signed and the gate decides.** The bill of materials (teachers, licenses, the base model's license, the NOTICE text, the dataset hashes, the eval, the set roots) is written once, and signed when `RATCHET_BOM_SIGNING_KEY` is set. Then the release gate admits the candidate to **shadow**, **canary 10%** on a cold start, or **rejects** it. See [the release gate](/guides/release-gate/). 12. **The licenses join the weights.** `ratchet.live` writes `ratchet_bom.json` and the BOM's `NOTICE.md` into `RATCHET_DATA_DIR/adapters/student-vN/` and runs `RATCHET_SYNC_CMD` again, so the bucket folder holds the weights and their licenses together. If this second sync fails, the version still stands, a `sync_failed` event is written, and the command exits non-zero so cron notices. 13. **The gateway serves the adapter by name.** As the gate moves the version through shadow and canary to live, the gateway sends student calls to the endpoint with `model` set to `student-vN`. vLLM serves each loaded adapter under its own name. ### The Nebius pieces
```mermaid flowchart TB subgraph Host["Your gateway host"] direction LR GW["Gateway"] --> DB[("SQLite
ledger")] NS["Night School
ratchet.live night"] --> DB end subgraph TF["Token Factory, per token"] TEA["Teacher: Ultra
Judge: Super"] end subgraph Neb["Nebius AI Cloud, ran by hand 27 to 28 Oct"] direction TB JOB["Serverless Job
LoRA on one L40S"] -->|"adapter"| BK[("Bucket
at /data")] BK -->|"loads"| EP["Serverless Endpoint
vLLM"] NIGHT["Nightly schedule and
live connection: not run"] end Host ==>|"escalations, audits,
judge"| TF Host ==>|"rows, job, student calls"| Neb class DB,BK store class TEA teacher class JOB,EP ok class NIGHT pending class TF teacherZone class Neb okZone ```
**Reading this diagram.** Three boxes, three jobs. Your host decides and remembers: it holds the gateway, the ledger and the night script. Nebius does the heavy lifting: a job trains, an endpoint serves, and one bucket is shared by both, which is how weights get from one to the other without a copy step. Token Factory is rented per token and holds the teacher and the judge. From the host, the night uploads the rows to the bucket, submits the job and tells the endpoint to load the adapter, and the gateway sends student calls to the endpoint. Nothing on Nebius ever writes to the ledger; only the host does. The job, the bucket and the endpoint are drawn green because each ran once by hand on Nebius (27 to 28 Oct 2026); the amber box is what has not run: the nightly schedule, and the live gateway's connection to an endpoint. There are no live results yet.
  • Rented per token
  • Keeps data
  • Ran by hand
  • Not run
### How one training example is made
```mermaid flowchart TB T(["A teacher answer"]) --> Q{{"Complete, valid,
no opt-out, not pinned?"}} Q -->|yes| RED["Redact the prompt,
answer and conditions"] RED --> V{{"Still valid?"}} V -->|yes| D{{"Hash of task and
redacted messages known?"}} D -->|new| SP{{"Hash bucket"}} D -->|"seen again"| B["seen + 1,
unless a retry"] B --> F{{"Seen 3 or more?"}} SP -->|"about 90%"| TR[("Training split")] SP -->|"about 10%"| GO[("Golden split")] F -->|yes| TR TR --> J["night/train.jsonl
teacher answer as target"] GO --> E["Golden eval only"] Q -->|no| X["Served, not stored"] V -->|no| X D -->|tombstoned| X class T teacher class Q,V,D,SP,F gate class TR,GO store class X outline ```
**Reading this diagram.** A teacher answer can fall out of the path at three places, always on the safe side: the answer is still served, only storage is skipped. After redaction the example gets a hash, and the hash does three jobs: it merges repeats, it blocks erased examples for good, and it fixes the split without anyone choosing it. The golden split is never trained on, with one exception: a prompt seen three or more times moves to training, so a question people ask all the time does not stay hidden forever.
### What each part of an example is used for | Part | Used by | Notes | |---|---|---| | Redacted messages | `prompt` in the training row, the golden prompt | Placeholders such as `[EMAIL]` replace identifiers | | Teacher answer | `completion`, the loss target, the golden reference | The student's own answers are never targets | | Stored conditions | The golden eval only (it samples under them) | Never reach the chat template | | Provenance: teacher, license, hash, redactions, origin | The signed BOM and NOTICE.md | Travels with the adapter | | Client and end-user hash | Erasure and opt-outs | Never in the training file | ### Where your data goes
```mermaid flowchart TB APP(["Your app"]) -->|"chat request"| GW["Gateway"] GW -->|"teacher answers, redacted"| LED[("Ledger: examples")] LED -->|"night/train.jsonl,
RATCHET_SYNC_CMD"| BK[("Object Storage bucket")] BK -->|"mounted at /data"| JOB["Serverless Job
train.py"] JOB -->|"adapters/student-vN"| BK BK -->|"load_lora_adapter"| EP["Student endpoint
vLLM"] EP -->|"answers as student-vN"| GW class LED,BK store class EP student ```
**Reading this diagram.** The loop is closed: requests and teacher answers go down the middle, and the adapter comes back up to the gateway through the endpoint on the right. The only arrow that carries your text out of the gateway is the one from the ledger to the training file, and by then it is redacted. The job and the bucket meet twice: it reads the rows from `/data` and writes the adapter back to the same place.
### One live night, with the real commands
```mermaid sequenceDiagram autonumber participant Live as ratchet.live
night participant Bucket as Bucket
at /data participant Job as Serverless
Job, L40S participant Student as Student
vLLM participant J as Judge
Super Note over Live: cron, 02:30 Live->>Student: re-score live
on golden rows Note over Live: write night/
train.jsonl Live->>Bucket: upload the rows Live->>Job: train_job.sh:
job create Job->>Bucket: read the rows,
write student-vN Live->>Bucket: poll for
the weights Live->>Student: load_lora_adapter Student->>Bucket: read student-vN Live->>Student: golden eval,
3 samples a row Live->>J: free-text rows Note over Live: signed BOM,
gate.admit Live->>Bucket: BOM, NOTICE.md,
sync again ```
**Reading this diagram.** Read it as a checklist in time order. Night School re-scores the old adapter before it trains (1), so the champion and the challenger sit the same exam, then uploads the rows with `RATCHET_SYNC_CMD` (2). `RATCHET_TRAIN_CMD` runs `deploy/train_job.sh`, which submits the job (3) and then polls the bucket for the weights file (5): the script, not the job, is what `ratchet.live` waits on. The last line matters for licensing: the weights are in the bucket from the job, and the BOM and `NOTICE.md` are added after the gate has decided (10), so the folder holds the adapter and its licenses together. Every arrow here is written and tested offline; none has run on Nebius yet.
### The student endpoint only answers The endpoint is a Nebius Serverless Endpoint running vLLM with runtime LoRA loading. It serves the base model `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` plus whichever adapters were loaded into it. It never trains, never writes to the ledger, and never sees the training file. `bash deploy/student_endpoint.sh create` (once) sets: | Setting | Value | Why | |---|---|---| | Image | `vllm/vllm-openai:v0.18.0-cu130` (`RATCHET_VLLM_IMAGE`) | `v0.18.0` without `-cu130` is the CUDA 12 build, for an older host driver | | Platform, preset | `gpu-l40s-a`, `1gpu-8vcpu-32gb` | One L40S | | Port, shared memory | `--container-port 8000`, `--shm-size 16Gi` | | | Storage | `--volume :/data` | vLLM reads adapters from the same bucket the job writes | | Auth | `--auth token --token $RATCHET_STUDENT_KEY` | The gateway sends this token; the script never prints it | | Runtime LoRA | `--env VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`, `--enable-lora --max-lora-rank 16 --max-loras 4` | New adapters load without a restart; up to four adapters in one batch, enough for live, canary and shadow | | Model code | `--trust-remote-code`, `--mamba_ssm_cache_dtype float32` | The second is the model card's flag (read Oct 21, 2026) | | Limits | `--max-model-len 8192`, `--max-num-seqs 64` | The card's example uses 262,144 tokens; ticket prompts are short, and a longer one gets vLLM's 400 and escalates to the teacher | | Thinking off | `--default-chat-template-kwargs '{"enable_thinking":false}'` | The student answers without reasoning, on the same prompt it was trained on | | Reasoning parser (optional) | `--reasoning-parser-plugin ... --reasoning-parser nano_v3` with `RATCHET_REASONING_PARSER_PLUGIN` | The card's parser, uploaded to the bucket | The other actions reuse the same endpoint (same URL, same token): `start` (the default; creates it when none has the name), `stop`, and `status`, which prints ` `. `start` and `stop` find the endpoint's id by name (`RATCHET_ENDPOINT_NAME`, default `ratchet-student`) in `nebius ai endpoint list --format json`, or use `RATCHET_ENDPOINT_ID`. The lookup never guesses: output that is not JSON, two endpoints with the name, or a name missing from a list with more pages stops the script before anything starts, stops or is created. Each `create`, `start` and `stop` also records the endpoint's uptime in the ledger for the cost report (`python -m ratchet.cost endpoint up` or `down`); that step is best effort and never fails the deploy. Full details: [Deployment, step 4](/reference/deployment/#step-4-student-endpoint-vllm-with-runtime-lora). ### What becomes training data Only teacher answers (and labels a person attested, from [feedback corrections or an import](/guides/ledger/)) become training targets. A teacher answer is stored only when it is complete (`finish_reason` `stop` or `tool_calls`), valid under the task's spec, not under a storage opt-out, not for a pinned task or a tool request, and after PII redaction. Each one records the teacher model, its license and license URL, its hash and the number of placeholders. You can keep any request, key, task or end user out of training with an [opt-out](/guides/ledger/), and switch storage off entirely with `RATCHET_STORE_EXAMPLES=0`. ### The training file `dataset.write_training` writes TRL's conversational prompt-completion format. One row, shown on several lines (the content is illustrative): ```json { "task": "ticket-triage", "params": null, "seen": 1, "provenance": {"teacher": "nvidia/Nemotron-3-Ultra-550b-a55b", "license": "OpenMDW-1.1", "hash": "…", "redactions": 1, "origin": "live"}, "prompt": [{"role": "user", "content": "Where is my refund for order 1234? Mail me at [EMAIL]"}], "completion": [{"role": "assistant", "content": "{\"category\": \"refund\", \"priority\": \"low\"}"}], "chat_template_kwargs": {"enable_thinking": false} } ``` `train.py` gives TRL only `prompt`, `completion` and `chat_template_kwargs`, so stored conditions never reach the template, and the loss covers the teacher's answer only. The same `{"enable_thinking": false}` lives in three places (the dataset writer, `train.py` and the endpoint script), and a test keeps them equal. Rows left out of training (a client's `stop`, a literal think tag, an answer the current spec refuses, a pinned task, a purged row) are listed on [Night School](/guides/night-school/#rows-left-out-of-training). ### The bucket One Object Storage bucket in the same region as the GPUs, mounted at `/data` in both the job and the endpoint. `RATCHET_DATA_DIR` (default `./data`) is the local mirror that `RATCHET_SYNC_CMD` uploads ([Deployment, step 2](/reference/deployment/#step-2-object-storage-bucket)): - data/ - night/ - train.jsonl rewritten by ratchet.live each night that trains - boms/ - student-v3.json the adapter's BOM, written after the night - adapters/ - student-v3/ - adapter_config.json written by the training job - adapter_model.safetensors the LoRA weights - versions.json Python, CUDA and library versions - ratchet_manifest.json base model, teachers and licenses, template arguments - NOTICE.md the licenses; replaced after the night by the text the BOM signed - ratchet_bom.json the BOM, synced after the night The job also saves the tokenizer files next to the weights. Keep every adapter a `live` version uses, for rollback. A mounted bucket is billed even while the endpoint is stopped. ### The training job `deploy/train_job.sh ` builds and runs this command ([Deployment, step 5](/reference/deployment/#step-5-training-job)): ```bash nebius ai job create --name ratchet- --image $RATCHET_TRAIN_IMAGE \ --platform gpu-l40s-a --preset 1gpu-8vcpu-32gb --volume $RATCHET_BUCKET_ID:/data \ --disk-size 200Gi --timeout 2h --container-command bash \ --args "-c 'python -m ratchet.nightschool.train --data /data/night/train.jsonl --base --out /data/adapters/'" # plus --subnet-id $RATCHET_SUBNET_ID and --registry-secret $RATCHET_REGISTRY_SECRET when set ``` - `nebius ai job create` is the scripted command; `nebius ai create` is the interactive wizard. - `--timeout 2h` is the job's own limit (1 h to 168 h, default 24 h), so a stuck job's GPU time is capped at the 2 hours the script waits. - The version name and base model are checked against a strict pattern first, because they go into the job's `bash -c` line. - `DRY_RUN=1` adds `--dry-run`: Nebius validates the request and submits nothing. Any value other than `0` or `1` is refused, so `DRY_RUN=true` can never submit a paid job. Training defaults in `train.py`: 2 epochs, learning rate 2e-4, LoRA rank 16 with alpha 32, max sequence 2048, batch 8, bf16, loss on the answer only, targets including the Mamba projections (with an attention-and-MLP fallback if vLLM refuses them on day 1). See [What LoRA is](/guides/night-school/#what-lora-is). **When the job fails.** A refused job request fails the script at once. A job that runs but writes no adapter makes the script time out after 2 hours and exit 1. Either way `ratchet.live` stops the night: no version is created, the training compute interval is closed with a "training failed" note (any GPU time the job used is still billed), and the command exits non-zero. Inside the job, `check_rows` and `check_prefix` stop a bad file or a chat-template mismatch within seconds, before the GPU trains anything (`row N: ...` or `TemplateMismatch` in the job's log). **When the endpoint is not set.** `NebiusTrainer` checks `RATCHET_STUDENT_BASE` before it syncs or starts the paid job: an adapter with no endpoint to load into cannot be evaluated. Without it, the night stops before any spend. ### The training image `deploy/Dockerfile.train` builds from `nvidia/cuda:12.8.0-devel-ubuntu24.04` and installs exact versions, listed again in `deploy/train-versions.txt` and read as pip constraints, so the two files disagreeing fails the build ([Deployment, step 3](/reference/deployment/#step-3-training-image)): | Package | Version | |---|---| | torch | 2.11.0 (CUDA 12.8 index) | | transformers | 4.57.6 | | trl | 1.16.0 | | peft | 0.21.2 | | accelerate | 1.15.0 | | datasets | 4.8.5 | | causal-conv1d | 1.6.2.post1 | | mamba-ssm | 2.3.2.post1 | | ninja, packaging, setuptools, wheel | 1.13.0, 26.2, 82.0.1, 0.47.0 | These are candidates read on Oct 21, 2026. The first-run smoke job (about 200 rows through `deploy/train_job.sh test`) confirms them, or its `versions.json` replaces them in both files. The last build step imports the Mamba kernels, so an image without them fails to build instead of failing on the GPU. There is no Unsloth: its Oct 19, 2026 release needs an older TRL. Build from the repository root: ```bash docker build -f deploy/Dockerfile.train -t /ratchet-train:0.1 . docker push /ratchet-train:0.1 export RATCHET_TRAIN_IMAGE=/ratchet-train:0.1 ``` If the image cannot build at all, the documented fallbacks are an NGC PyTorch base image with the same pins, or Nebius's fine-tuning tutorial image with this repository copied in. ### Loading the adapter, and after a restart `ratchet.live.load_adapter` posts to `{root}/v1/load_lora_adapter`, where `root` is `RATCHET_STUDENT_BASE` without its `/v1`, with the endpoint token and a 300-second timeout. The endpoint must have been started with `--enable-lora` and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`. vLLM **forgets runtime-loaded adapters when the endpoint restarts**, and answers 404 for them. The gateway handles that by itself: a 404 naming a version's adapter queues a reload on a background thread, at most once per adapter per minute, and writes `adapter_reloaded` or `adapter_reload_failed`. The request that hit the 404 is answered by the teacher, so it still succeeds, at teacher cost. If reloads keep failing, `python -m ratchet.live reload` loads every `live`, `canary` and `shadow` adapter by hand. The [runbook](/reference/runbook/#student-endpoint-down-or-adapters-lost) has the steps. ### When the endpoint is off: the hosted fallback With `RATCHET_STUDENT_BASE` unset, student calls for a trained version go to : the untuned `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` on Token Factory (`RATCHET_FALLBACK_STUDENT`), paid per token instead of per GPU-hour. - It is **not** the adapter you own and **not** covered by the adapter's precision certificate. It was never calibrated, so it is held to the global `RATCHET_CONFIDENCE_THRESHOLD` (default 0.80), not the version's calibrated threshold. Precision may be lower; the teacher's random audits will show it. - The ledger still records these answers as `served_by: student`; the response's `model` field names the hosted model, and the pages label them "Nano 30B · hosted", never as the adapter. - Night School cannot train or grade a new adapter in this state (see above). ### Stop the endpoint when idle, start it for the night A stopped Serverless Endpoint is not billed for compute or its own storage; a mounted bucket still is ([Nebius Serverless overview](https://docs.nebius.com/serverless/overview), read Oct 21, 2026). ```bash bash deploy/student_endpoint.sh stop # when idle bash deploy/student_endpoint.sh start # the same endpoint again: same URL, same token bash deploy/student_endpoint.sh status # prints " " ``` Night School needs the endpoint **running** while it re-scores the live adapter, loads the new one and grades it. With the endpoint stopped most of the time, start it before the night run (with `RATCHET_STUDENT_BASE` set in the night job's environment) and stop it after. Each start and stop is recorded, and answers in hours without uptime count as the hosted fallback's in the cost report. Whether running the endpoint pays at all depends on volume. With estimated prices (an L40S at about $1.35 per GPU-hour on demand, [Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026, and about $0.001 of teacher spend avoided per ticket-triage answer, an assumption), break-even is about 1,350 student answers an hour; at a small gateway's volume a 24/7 endpoint loses money, and `python -m ratchet.cost report` says so ([cost and savings](/guides/cost-and-savings/)). At the same price, estimates from the repo's [deployment guide](/reference/deployment/#4-cost-estimates): about $0.70 to $1.35 per training night, about $32 per full day if the endpoint is left on. ### Exporting the adapter you own You own the LoRA adapter and its signed training lineage. The base model stays under the NVIDIA Nemotron Open Model License. To hand the adapter over in one folder, with its licenses: ```bash aws s3 sync s3:///adapters/student-v3 ./data/adapters/student-v3 --endpoint-url $RATCHET_S3_ENDPOINT # weights, if not local python -m ratchet.live export --version 3 --out ./student-v3-export python -m ratchet.bom verify student-v3-export/ratchet_bom.json --public-key --notice student-v3-export/NOTICE.md ``` The export checks the stored BOM against its own SHA-256 and signature first, copies the adapter files when they are on this machine, writes the BOM and the exact `NOTICE.md` it signed, and prints the BOM's `sha256`. Without the weights it says where in the bucket to copy them from. The output folder must be new or empty and outside `RATCHET_DATA_DIR` (the next sync would upload it). Keep `NOTICE.md` with the adapter wherever it goes. See [licenses](/guides/licenses/). ### What can go wrong | Failure | What the system does | Where you see it | |---|---|---| | No student endpoint is set | The night stops before any paid call | `skip` or an error naming `RATCHET_STUDENT_BASE` | | The job request is refused, or the job writes no adapter | The script exits non-zero (after 2 hours when waiting); no version is created | the night's exit code, a "training failed" note in the compute log | | A bad training row or a chat-template mismatch | The job stops within seconds, before the GPU trains | `row N: ...` or `TemplateMismatch` in the job log | | The endpoint restarts and forgets adapters | A 404 queues a reload; that request goes to the teacher | `adapter_reloaded` or `adapter_reload_failed` events | | The second upload (BOM and NOTICE) fails | The version stands; the command exits non-zero | `sync_failed` event | | Too few new examples, a release in flight, or training paused | The night is skipped | `skip` event with the reason | | Someone erases examples after an adapter trained | The adapter is reported as possibly affected, never edited | the erasure's `affected_versions` | ### What is verified and what is not
**Checked offline, against fakes.** These parts run in the test suite, against fakes: - `ratchet.live` commands, `NebiusTrainer`, `LiveEvaluator`, the reloader and the export: tested in `tests/test_live.py` and `tests/test_licensing.py`, against fake endpoints. - `train_job.sh` and `student_endpoint.sh`: run in tests with a fake `nebius` CLI; the exact commands are asserted against the CLI reference as read. - `train.py`'s arguments, row checks, loss setting and chat-template prefix check: tested with the vendored Nano 4B chat template. Its `main` needs a GPU and is not run in tests.
Ready to run, or still to check: | Part | Status (29 Oct 2026) | |---|---| | The training image | Not built yet; pins are candidates | | A real training job, vLLM serving Nano 4B with LoRA, runtime loading, LoRA on the Mamba projections | Ran once by hand on 28 Oct 2026 (spot L40S job, endpoint, hot-load), and the golden gate rejected the result; the one-command host is scripted and tested offline. The three published adapters were trained on a borrowed GPU, outside this path | | The endpoint's arguments as the container splits them, thinking off | Not checked yet | | Serverless pricing, the S3 endpoint URL | Unverified (the S3 URL format in the docs is an assumption) | | Live numbers from this path | None yet; they come only from a published live run | ## For engineers ### Code | File | What | |---|---| | [`ratchet/live.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/live.py) | `main` (`night`, `tick`, `reload`, `purge`, `export`, `seal`), `NebiusTrainer.train`, `load_adapter`, `AdapterReloader`, `reload_adapters`, `write_boms`, `publish_bom`, `export` | | [`ratchet/nightschool/dataset.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/dataset.py) | `rows`, `training_record`, `write_training`, `TRAIN_TEMPLATE_KWARGS` | | [`ratchet/nightschool/train.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/train.py) | The job's script: `parse_args`, `check_rows`, `check_prefix`, `columns`, `write_metadata`, `versions` | | [`ratchet/nightschool/evaluate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/evaluate.py) | `LiveEvaluator`, `cascade_eval` | | [`deploy/train_job.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/train_job.sh) | Submits the job and waits for the adapter | | [`deploy/student_endpoint.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/student_endpoint.sh) | `create`, `start`, `stop`, `status` | | [`deploy/Dockerfile.train`](https://github.com/harshbhx/ratchet/blob/main/deploy/Dockerfile.train), [`deploy/train-versions.txt`](https://github.com/harshbhx/ratchet/blob/main/deploy/train-versions.txt) | The pinned training image | ### Settings read by the gateway and `ratchet.live` | Variable | Default | Meaning | |---|---|---| | `RATCHET_STUDENT_BASE` | empty | The endpoint's OpenAI base URL, `https:///v1`. Empty: student calls go to the hosted fallback, and Night School cannot train | | `RATCHET_STUDENT_KEY` | empty | The endpoint's token | | `RATCHET_STUDENT_BASE_MODEL` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | The base every adapter is trained from (also read by both scripts) | | `RATCHET_FALLBACK_STUDENT` | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` | The hosted, untuned model for student calls when the endpoint is not set | | `RATCHET_CONFIDENCE_THRESHOLD` | `0.80` | The threshold the hosted fallback is held to | | `RATCHET_DATA_DIR` | `data` | Local mirror of the bucket: `night/`, `boms/`, `adapters//` | | `RATCHET_SYNC_CMD` | empty | Uploads `RATCHET_DATA_DIR` to the bucket, before the job and after a night that trained. Upload only: a `--delete` would remove the weights the job wrote | | `RATCHET_TRAIN_CMD` | `bash deploy/train_job.sh {version}` | Runs training and **waits** for it; `{version}` and `{data}` are substituted | | `RATCHET_ADAPTER_ROOT` | `/data/adapters` | Where the endpoint sees adapters (`lora_path` = root + `/` + name); also used by the gateway's reloader | | `RATCHET_TRAIN_GPU_USD_PER_HOUR`, `RATCHET_STUDENT_GPU_USD_PER_HOUR` | `1.55` | Prices for the cost report. The code's default; these docs use 1.35 USD per GPU-hour, Nebius's list price for an L40S. Set the rate on your bill | `RATCHET_SYNC_CMD` and `RATCHET_TRAIN_CMD` are split with `shlex.split` and run without a shell, so pipes and `&&` do not work; wrap them in a script. ### Settings read by the deploy scripts | Variable | Used by | Default | |---|---|---| | `RATCHET_BUCKET_ID` | both | required (job; endpoint `create`) | | `RATCHET_BUCKET_S3` | `train_job.sh` | empty; set it (`s3://bucket[/prefix]`) so the script waits for the adapter | | `RATCHET_S3_ENDPOINT` | `train_job.sh` | empty; the Object Storage S3 URL (copy it from the console) | | `RATCHET_TRAIN_IMAGE` | `train_job.sh` | required | | `RATCHET_TRAIN_PLATFORM`, `RATCHET_TRAIN_PRESET` | `train_job.sh` | `gpu-l40s-a`, `1gpu-8vcpu-32gb` | | `RATCHET_SUBNET_ID` | both | empty (the project's default subnet) | | `RATCHET_REGISTRY_SECRET` | `train_job.sh` | empty; a MysteryBox secret for a private image | | `RATCHET_STUDENT_KEY` | `student_endpoint.sh` | required for `create` | | `RATCHET_ENDPOINT_NAME`, `RATCHET_ENDPOINT_ID` | `student_endpoint.sh` | `ratchet-student`, empty | | `RATCHET_SERVE_PLATFORM`, `RATCHET_SERVE_PRESET` | `student_endpoint.sh` | `gpu-l40s-a`, `1gpu-8vcpu-32gb` | | `RATCHET_VLLM_IMAGE` | `student_endpoint.sh` | `vllm/vllm-openai:v0.18.0-cu130` | | `RATCHET_SERVE_MAX_LEN`, `RATCHET_SERVE_MAX_SEQS` | `student_endpoint.sh` | `8192`, `64` | | `RATCHET_SERVE_TEMPLATE_KWARGS` | `student_endpoint.sh` | `{"enable_thinking":false}` | | `RATCHET_REASONING_PARSER_PLUGIN` | `student_endpoint.sh` | empty | | `RATCHET_PYTHON` | `student_endpoint.sh` | `python` | | `DRY_RUN` | both | `0` | Full descriptions: [Configuration, deploy scripts](/reference/configuration/#deploy-scripts-deploy). ### Setup, in order ```bash export NEBIUS_API_KEY=... # Token Factory key python scripts/day1_check.py # model ids, logprobs, JSON mode export RATCHET_BUCKET_ID=... RATCHET_BUCKET_S3=s3:// RATCHET_S3_ENDPOINT= export RATCHET_DATA_DIR=./data RATCHET_SYNC_CMD="aws s3 sync ./data s3:// --endpoint-url " export RATCHET_TRAIN_IMAGE=/ratchet-train:0.1 export RATCHET_STUDENT_KEY=$(openssl rand -hex 24) DRY_RUN=1 bash deploy/train_job.sh test # nebius validates the job, submits nothing DRY_RUN=1 bash deploy/student_endpoint.sh create bash deploy/student_endpoint.sh create # once export RATCHET_STUDENT_BASE=https:///v1 export RATCHET_MODE=live RATCHET_DB=/var/lib/ratchet/ratchet.db export RATCHET_API_KEYS="app:$(openssl rand -hex 24)" # live mode does not start without API keys python -m ratchet.config check # exit 0 clean, 1 warnings, 2 errors uvicorn ratchet.gateway:app --port 8000 --workers 1 ``` Cron, on the gateway's host (it needs the `nebius` and `aws` CLIs, the same `RATCHET_DB` file and network access to the endpoint): ```bash 30 2 * * * cd /opt/ratchet && . ./.env && python -m ratchet.live night >> /var/log/ratchet-night.log 2>&1 */5 * * * * cd /opt/ratchet && . ./.env && python -m ratchet.live tick >> /var/log/ratchet-tick.log 2>&1 ``` ### Events `trained`, `skip`, `sync_failed`, `adapter_reloaded`, `adapter_reload_failed`, `endpoint` (an uptime change recorded), `circuit` (the student breaker). Meanings: [runbook](/reference/runbook/). ## Common questions ### Does the endpoint learn from my traffic while it serves? No. The endpoint only answers. Training happens in a separate Serverless Job, at night, on teacher answers stored in the ledger. The student's own answers are never stored, so it never trains on them. ### Can I train on labels I already have? Yes, as attested human labels or as an open teacher's outputs: `python -m ratchet.importer labels.jsonl --task T --source human --attest "..."` (or `--source teacher:`; closed models are refused). Users can also send corrections with `POST /v1/ratchet/feedback`. Every label is redacted and checked against the task's spec. Human labels carry the provenance `human` and are listed apart from the teachers in the BOM; an imported teacher's outputs keep that teacher and its license. See [the ledger](/guides/ledger/) and [licenses](/guides/licenses/). ### What does it cost? Estimates only, at about $1.35 per L40S GPU-hour (Nebius's list price, read Oct 22, 2026): about $0.70 to $1.35 per night that trains (30 to 60 minutes), about $32 per full day for an endpoint left on, plus the bucket. Nights that skip cost nothing. Below about 1,350 student answers an hour, a 24/7 endpoint costs more than it saves; use the hosted fallback or stop the endpoint between uses. ### What if the endpoint restarts? It forgets its loaded adapters. The gateway reloads each one by itself on its first 404, at most once a minute per adapter. A few requests go to the teacher meanwhile and still succeed. ### Do I have to keep the endpoint running? No. Stop it when idle; the hosted fallback answers student calls meanwhile, labelled as such. But Night School needs it running during the night to load and grade a new adapter. ### Can I take the adapter somewhere else? Yes. `python -m ratchet.live export` gives you the weights with the signed BOM and `NOTICE.md`. The adapter runs on the Nemotron 3 Nano 4B base, which stays under NVIDIA's license; keep that license with any copy of the base model. ### What if the training job breaks on the first real run? That is what the day-1 checks are for. The script dry runs, the image's build-time kernel check, `check_rows` and `check_prefix` are all there to fail fast and cheaply. The fallbacks (another base image, LoRA on the attention and MLP layers only, merging the adapter into the base weights if runtime LoRA fails) are listed in `train.py`. ### What runs on Nebius today? The teacher: it runs on Nebius Token Factory. The training job and the endpoint are tested offline, and ran once by hand on 28 Oct 2026 (the golden gate rejected that student); the three published adapters were trained on a borrowed GPU. Treat every GPU command here as a plan until the day-1 checks pass. ## Related --- # Licenses: why only open teachers Source: https://docs.ratchet.dualmindlab.tech/guides/licenses/ Ratchet distills only from teachers whose license allows training other models on their outputs, refuses closed models in code, records the license on every training example, and ships the licenses with every adapter, because the terms of closed models such as GPT forbid using their outputs to train competing models. This page explains what the code does and what the license texts say, as the repo quotes them. Check each license yourself before you rely on it. Some agency chefs have a contract that says: "**you may not copy my recipes to train a rival kitchen**." If the apprentice learned from that chef, the restaurant would be breaking the contract. So Ratchet only hires head chefs whose contract says copying is allowed. The **head chef from the agency** that Ratchet uses, Nemotron 3 Ultra, has exactly such a contract. The pass checks every chef's contract **before the restaurant opens**. A chef with a "no copying" contract, or a chef nobody has checked, is turned away at the door, and the restaurant does not open until that is fixed. Every page of the **recipe book** says which chef cooked the dish and which contract allows copying it. When the apprentice finishes night school, the new **notebook** gets a **certificate on the wall** that lists every chef and contract it learned from, plus a slip of paper that goes everywhere the notebook goes, saying whose rules apply. The apprentice's own schooling came from NVIDIA, under NVIDIA's license. The notebook is yours; the apprentice's original training stays under NVIDIA's rules. ## How it really works 1. **The registry is built at startup.** `ratchet/licenses.py` holds the built-in teachers (`BUILTIN`, read-only). If `RATCHET_TEACHER_LICENSES` names a file, its entries are checked and merged into a copy, published as `licenses.REGISTRY`. A bad file stops startup with every error listed. This runs at every entry point: the gateway, `python -m ratchet.live` and `python -m ratchet.admin`. 2. **Every model that writes or judges answers must be distillable.** The startup check runs `licenses.require_distillable` on the teacher (`RATCHET_TEACHER_MODEL`), the judge (`RATCHET_JUDGE_MODEL`) and the fallback teacher (`RATCHET_TEACHER_FALLBACK_MODEL`, when set). An unknown model, or one in a closed family, raises `LicenseError` and the gateway does not start. 3. **Each stored answer records its license.** When a teacher answer is stored, the ledger row gets `teacher_model`, `license` and `license_url`: the license Ratchet holds for the configured model that answered (the teacher, or the fallback teacher), never one read from the response's `model` string. 4. **Night School re-checks provenance.** `dataset.rows` reads each row's license with `licenses.lookup_source` and drops any row whose provenance is unknown or not distillable. 5. **The adapter's bill of materials names every license.** The signed BOM lists each source (teacher, license, URL, `distill_ok`, number of examples, and `license_origin`: `builtin` or `config`), the base model's license, and the exact text of `NOTICE.md` with its SHA-256. Each set-commitment leaf binds an example's answer to its teacher, license and redaction count. 6. **The licenses travel with the weights.** The training job writes `NOTICE.md` and `ratchet_manifest.json` next to the adapter. After the night, `python -m ratchet.live night` replaces `NOTICE.md` with the text the BOM signed and syncs both into the adapter's bucket folder. `python -m ratchet.live export` bundles the adapter, the BOM and that `NOTICE.md`. ### The startup check
```mermaid flowchart TB A(["Teacher, judge or
fallback teacher id"]) --> B{{"A provenance source,
like human?"}} B -->|"yes"| X1["Refused:
not a teacher"] B -->|"no"| C{{"In the registry?"}} C -->|"no"| D{{"Starts with a
closed prefix?"}} D -->|"yes, like gpt"| X2["Refused:
the terms are quoted"] D -->|"no"| X3["Refused:
unknown teacher"] C -->|"yes"| E{{"distill_ok?"}} E -->|"no"| X2 E -->|"yes"| OK["Starts: the license is
kept for every example"] class B,C,D,E gate class X1,X2,X3 danger class OK ok ```
**Reading this diagram.** Every model that writes or judges answers passes these checks before the gateway starts. There is one way through: a registered model whose license allows distillation. Every other path ends in a refusal, and a refusal stops startup with the reason; for a closed family such as GPT the reason quotes the terms that forbid it.
  • A check
  • Refused: the gateway does not start
  • Starts
### From a teacher answer to the adapter's paperwork
```mermaid flowchart TB T["Teacher answer"] --> E[("Example row:
teacher, license, URL")] E --> R["Night School rows"] R --> L["Set leaf: answer,
teacher, license"] R --> J["Job writes NOTICE.md
and manifest"] L --> B["Signed BOM: sources,
base license, NOTICE text"] J -.->|"replaced by
the signed text"| N["NOTICE.md next
to the weights"] B --> N B --> X["ratchet.live export"] class T teacher class E store class B student ```
**Reading this diagram.** The license is written down once, when the teacher's answer is stored, and then carried forward: into the rows Night School trains on, into each leaf of the committed training set, and into the signed bill of materials. The training job writes a first `NOTICE.md`; after the night it is replaced by the exact text the BOM signed, and `python -m ratchet.live export` hands over the adapter with both.
  • The teacher's answer
  • The ledger row
  • The signed lineage you own
### Why closed teachers are refused The terms of the most used closed models forbid using their outputs to train competing models. OpenAI's [Business Terms §3.3(e)](https://openai.com/policies/business-terms/) (read Oct 21, 2026) forbid using Output "to develop artificial intelligence models that compete with OpenAI". A distillation tool that learns from GPT traffic would put its users in breach of terms like these. The README's [Why "license-clean"](https://github.com/harshbhx/ratchet#why-license-clean) section lists the other closed vendors' clauses. So the rule is hard, not a warning: the teacher must be in the registry with `distill_ok=True`, and an override needs a reviewed code change. The cost is real: traffic that runs on closed models today cannot become training data. It has to go through an open teacher first. ### The teacher Ratchet uses: Nemotron 3 Ultra under OpenMDW-1.1 is `nvidia/Nemotron-3-Ultra-550b-a55b` on Nebius Token Factory, under [OpenMDW-1.1](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16). In its own words, OpenMDW-1.1 "does not impose any restrictions or obligations with respect to any use, modification, or sharing of any outputs" ([license page](https://openmdw.ai/license/1-1/), last updated May 27, 2026; checked Oct 21, 2026). ### The license registry The built-in entries in `ratchet/licenses.py`: | Model id | Used as | License | Note in the code | |---|---|---|---| | `nvidia/Nemotron-3-Ultra-550b-a55b` | Teacher (default) | OpenMDW-1.1 | Places no restrictions or obligations on outputs | | `nvidia/nemotron-3-super-120b-a12b` | Judge (default) | NVIDIA Nemotron Open Model License | Commercial use and derivative works allowed; NVIDIA claims no ownership of outputs | | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` | Hosted fallback student | NVIDIA Nemotron Open Model License | As above | | `nvidia/Nemotron-3_5-Lightning` | Listed, not used by default | OpenMDW-1.1 | Verify on the model card | | `mock-teacher` | Tests and mock mode only | mock (tests only) | | `CLOSED` maps name prefixes of closed model families to the terms that forbid distillation. Every OpenAI family (`gpt`, `o1`, `o3`, `o4`) carries the Business Terms §3.3(e) note, so a refusal can quote it. For a model that is not in the registry, `lookup` reads the last part of the id after `/`, in lower case, and returns the closed terms when it starts with one of those prefixes. `require_distillable` then refuses it with the license name and the clause, and refuses any other unknown model as unknown. Matching on names is a heuristic, not legal review. ### Adding an open teacher by configuration An open teacher that Ratchet does not list yet can be added without a code change, but only within strict limits. `RATCHET_TEACHER_LICENSES` names a JSON file: ```json [ {"model": "example-org/open-model-7b", "license": "Apache-2.0", "url": "https://huggingface.co/example-org/open-model-7b", "note": "Checked on the model card"} ] ``` An entry is accepted only when: - its license is one of `ALLOWED_LICENSES`: **OpenMDW-1.1, Apache-2.0, MIT, NVIDIA Nemotron Open Model License** (any case; stored in the canonical spelling); - its model id belongs to **no closed family** (`closed_family`): the id is split at every character that is not a letter or digit, and no part may start with a closed prefix or with `o` and a digit, or contain a closed prefix of three letters or more such as `gpt` (so `chatgpt-4o-latest` is refused), a closed model's name such as `davinci` or `codex`, or a closed vendor's name such as `openai`. This is stricter than the startup lookup, and it also refuses an open model under a closed vendor's name, which has to be added in code after its license is read; - it is **no router alias** (a router's name such as `openrouter` anywhere in the id, or a last segment of `auto`): the model that answers an alias is chosen per request and may be closed; - it does not name a built-in teacher, does not repeat an earlier entry, has an `https://` URL, and has no other keys (at most 100 entries). Any problem stops startup with every error and its place, for example `[2].license: 'CC-BY-NC-4.0' is not an allowed license; ...`. Configured entries carry `origin` `config`: every BOM records it per source (`license_origin`), and NOTICE.md marks the license "(configured)". Configuration **cannot relabel a closed model Ratchet knows of**. But `closed_family` only knows the names it lists: a closed model from a vendor it does not list, under a new name, would pass the check and be admitted on the operator's claim alone. The `config` mark in the BOM and NOTICE is the safeguard there: it shows whoever receives the adapter that the license was the operator's claim, not Ratchet's reviewed registry. Widening the allowlist or loosening the check needs a reviewed code change. ### The student's base: the NVIDIA Nemotron Open Model License The student is : `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` plus a LoRA adapter. The base model is under the [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) (v. Dec 15, 2025), which asks you to give anyone you distribute it to a copy of that license. `licenses.BASE_MODELS` maps the known Nano bases (4B BF16 and FP8, and the 30B-A3B variants) to that license, so the BOM can name it (`base_model_license`). For a base model Ratchet does not know, the BOM and NOTICE carry a warning instead of a license name: then you find and ship its license yourself. ### What you own
**You own this.** The LoRA adapter and its signed bill of materials are yours; the Nano base keeps the NVIDIA Nemotron Open Model License.
- **The LoRA adapter** Night School trains: the files in `adapters/student-vN/` in your bucket. - **Its signed training lineage**: the bill of materials, signed with your Ed25519 key (`RATCHET_BOM_SIGNING_KEY`), which commits to the exact training and golden sets, every teacher and license, and the evaluation. You do **not** own the base model: Nemotron 3 Nano 4B stays under NVIDIA's license, and the adapter runs on top of it. The code is Apache-2.0. A signed BOM is your signed statement of the data and evaluation behind an adapter: it shows that nothing changed after signing, not that the weights were trained on that data. ### Licenses travel with the adapter Every adapter folder gets a `NOTICE.md` next to its weights: - the base model and its license, with the license URL (or a warning for an unknown base); - a table of teachers with their licenses, license URLs and example counts, configured ones marked; - the statement that the adapter was trained on teacher outputs only and that the student's own answers were never training targets (or, with human labels, that it was trained on teacher answers and customer-attested labels); - a **Human labels** section when there are any; - the Ratchet version, a pointer to the BOM, and "Not legal advice". The BOM carries the exact NOTICE text and its SHA-256 (`notices`), so a recipient can check that the file is the one written at training time: `python -m ratchet.bom verify ratchet_bom.json --public-key --notice NOTICE.md`. An export never re-renders the notice: it writes the text the BOM signed, so a later Ratchet that words the notice differently still ships the original. Keep `NOTICE.md` with the adapter wherever you copy, share or deploy it, and keep the base model's license with any copy of the base model. ### Human labels: a provenance source, not a teacher Labels a person wrote (a correction sent with `POST /v1/ratchet/feedback`, or a file imported with `python -m ratchet.importer --source human --attest "..."`) are stored with provenance `human`, license "customer-provided labels" and the note "Written or approved by the customer; the customer attests ownership." - `human` lives in `licenses.SOURCES`, not in the teacher registry. `require_distillable` refuses it by name, so no teacher can be called `human`, and a `RATCHET_TEACHER_LICENSES` entry may not use it. - Readers of stored rows (Night School, the BOM) use `lookup_source`, which knows both. - The BOM lists human labels apart from the teachers (`label_sources`), and NOTICE.md gets its Human labels section. - An import of a teacher's outputs (`--source teacher:`) must pass `require_distillable` and the closed-family check; a closed model is refused with its clause quoted, and Ratchet's own `train.jsonl` rows cannot be re-imported as human labels. ### A worked example: the text-to-SQL adapter The [text-to-SQL adapter](/guides/text-to-sql/) is the clearest case of the chain, because every link is a file you can read.
```mermaid flowchart TB TEA["Teacher: Nemotron 3 Ultra
on Token Factory
OpenMDW-1.1"] --> LAB["Labels: SQL the teacher
wrote that ran and was kept"] NONE["Closed-model
outputs"] -.->|"none used"| LAB BASE["Base: Nemotron 3 Nano 4B
NVIDIA Nemotron
Open Model License"] --> AD["LoRA adapter"] LAB --> AD LAB --> MAN[("teacher_manifest.json:
teacher, license, data hashes")] AD --> CARD["Model card and NOTICE.md
name both licenses"] class TEA teacher class NONE danger class AD student class MAN store ```
**Reading this diagram.** Two licenses meet in the adapter. The base model's license comes with the weights it sits on, so a copy of it ships with the adapter. The teacher's license covers the labels, and its terms place no restrictions on using the outputs. The dotted line is the rule that matters: no output from a closed model is in the data, so there is no closed-model clause to breach. The manifest and the model card are where a reader can check each claim.
The adapters were trained on a borrowed GPU. The teacher ran on Nebius Token Factory, so the Token Factory terms question below applies to the labels, not to the training. ### The open question: Token Factory's terms Token Factory's Terms of Service §5(f) is about using the service to "Develop a product or service that competes with the Service" (as quoted in the [threat model](/reference/threat-model/), T7). That wording is broad. The project asked Nebius support about it in writing on **Oct 19, 2026**; no answer received as of 29 Oct 2026, so treat the question as **open**. Ratchet's teacher runs on Token Factory and its student was trained and served on Nebius AI Cloud in hand-run tests, which is how it is positioned, but that is not a legal answer. Not legal advice. ## For engineers ### Code | File | What | |---|---| | [`ratchet/licenses.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/licenses.py) | `BUILTIN`, `REGISTRY`, `CLOSED`, `OPENAI_TERMS`, `SOURCES`, `BASE_MODELS`, `ALLOWED_LICENSES`, `closed_family`, `lookup`, `lookup_source`, `require_distillable`, `parse_entries`, `install` | | [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) | `Ratchet.__init__` licenses the teacher and fallback teacher; `_store_example` writes the license with each example | | [`ratchet/config.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/config.py) | The startup check: teacher, judge and fallback teacher must pass `require_distillable` | | [`ratchet/nightschool/notices.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/notices.py) | `render` (NOTICE.md), `bom_notice`, `write_adapter_files` | | [`ratchet/bom.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/bom.py) | The BOM manifest, Ed25519 signing, `verify`, `check_notice` | | [`ratchet/importer.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/importer.py), [`ratchet/feedback.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/feedback.py) | Human labels and imported teacher outputs | Tests: `tests/test_licensing.py` (licenses that travel with the adapter, `RATCHET_TEACHER_LICENSES`), `tests/test_contract.py` (the license rules clients rely on). ### Settings | Variable | Default | Meaning | |---|---|---| | `RATCHET_TEACHER_MODEL` | `nvidia/Nemotron-3-Ultra-550b-a55b` | Must pass `require_distillable` | | `RATCHET_JUDGE_MODEL` | `nvidia/nemotron-3-super-120b-a12b` | Must pass it too | | `RATCHET_TEACHER_FALLBACK_MODEL` | empty | Opt-in fallback teacher; must pass it | | `RATCHET_STORE_FALLBACK_TEACHER` | `0` | `1` stores the fallback teacher's answers, under its own license | | `RATCHET_TEACHER_LICENSES` | empty | Path to the JSON file of configured open teachers | | `RATCHET_STUDENT_BASE_MODEL` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | Its license goes into every BOM and NOTICE | | `RATCHET_BOM_SIGNING_KEY` | empty | Ed25519 seed (base64) that signs every BOM; empty means unsigned, with a SHA-256 | Details: [Configuration, teacher licenses](/reference/configuration/#teacher-licenses). ### Endpoints and commands | What | How | |---|---| | Ledger-wide BOM (teachers, licenses, counts, fingerprint) | `GET /ratchet/bom` | | One adapter's signed BOM | `GET /ratchet/versions/{id}/bom` | | The public key to publish | `GET /ratchet/bom/public-key` | | Make a signing key | `python -m ratchet.bom keygen` | | Check a BOM and its NOTICE | `python -m ratchet.bom verify bom.json --public-key --notice NOTICE.md` | | Hand an adapter over | `python -m ratchet.live export --version N --out DIR` | | Import labels | `python -m ratchet.importer FILE --task T --source human --attest "..."` or `--source teacher:` | The BOM endpoints need an admin or dashboard key in live mode unless `RATCHET_PUBLIC_BOM=1`. Without `--public-key`, `verify` only shows that the document is consistent with the key embedded in it; publish your public key somewhere you control. ## Common questions ### Can I use my GPT traffic as training data? No. Closed models such as GPT have terms that forbid using their outputs to train competing models, and Ratchet refuses them as teachers at startup. Route that traffic through an open teacher first; from then on, the teacher's answers are the training data. ### Can I add another open model as a teacher? Yes, if its license is OpenMDW-1.1, Apache-2.0, MIT or the NVIDIA Nemotron Open Model License, through `RATCHET_TEACHER_LICENSES`. Any other license, or a model whose name matches a closed family even though it is open, needs a reviewed change to `ratchet/licenses.py`. ### What stops someone from configuring a closed model? The closed-family check refuses every closed family, model name, vendor and router alias it knows. It cannot recognise names it does not list; such an entry is admitted on the operator's claim and marked `config` in every BOM and NOTICE, so the claim is visible to whoever receives the adapter. ### What exactly do I own? The LoRA adapter and its signed training lineage. Not the base model, which stays under the NVIDIA Nemotron Open Model License. ### Do I need to ship anything with the adapter? Yes: `NOTICE.md`, which Ratchet writes for you, and the base model's license with any copy of the base model. `python -m ratchet.live export` puts the adapter, the BOM and the notice in one folder. ### Does the judge's license matter? The judge only scores free-text answers in the golden eval and never writes training data, but the startup check still requires it to be distillable. Super is under the NVIDIA Nemotron Open Model License. ### Is Nebius itself fine with this? That is the open §5(f) question above: asked in writing on Oct 19, 2026; no answer received as of 29 Oct 2026. ### Is this legal advice? No. The registry matches names and records the licenses you configure; it is not a legal review. Check each license yourself. ## Related - [The ledger](/guides/ledger/): where each example's license is stored - [Night School](/guides/night-school/): where the BOM is written - [Training on your data](/guides/training-on-your-data/): the export and the bucket folder - [Evidence](/guides/evidence/): set commitments and certificates in the BOM --- # Cost and savings Source: https://docs.ratchet.dualmindlab.tech/guides/cost-and-savings/ Ratchet prices every request twice, per token and by the GPU-hours actually paid, works out the traffic at which a self-hosted student endpoint breaks even, and tells you when it does not. Every dish the head chef from the agency cooks shows up on the bill. When the apprentice chef cooks a dish instead, you skip that fee. That is the saving. But the apprentice needs a kitchen station, and the station is rented by the hour. You pay for it while it is open, whether orders come in or not. So the real question is not "is the apprentice cheaper per dish?" It is "does the apprentice cook enough dishes each hour to pay the station's rent?" Say the station costs $1.35 an hour and each dish the apprentice cooks saves about a tenth of a cent of head-chef fees. Then the apprentice has to cook about 1,350 dishes every hour just to break even. On a quiet day that never happens. Then it is cheaper to close the station and let the agency's cheaper, untrained cook take the easy orders, paid per dish. Ratchet writes this on the receipt in plain words, even when the answer is "you are losing money". ## How it really works 1. **Every request records what it cost, split in two.** Each model call is priced from its token counts at a price per million tokens (`config.DEFAULT_PRICES`). The **student part** (`student_cost_usd`) covers the student's samples, for the live version and for a shadow candidate. The **teacher part** (`teacher_cost_usd`) covers the escalation, the audit, the shadow arbitration and pinned calls. Their sum is the request's `cost_usd`, which the response also returns as `ratchet.cost_usd`. A failed call adds nothing. 2. **Every request also records an all-teacher baseline.** This is what the request would have cost if Nemotron 3 Ultra had answered alone (`ratchet.all_teacher_cost_usd`). When the teacher was called, the baseline is that call's price. When the student served, it is a formula: the teacher's price for the student's prompt tokens, plus the student's output tokens and an assumed 300 reasoning tokens (`RATCHET_TEACHER_REASONING_TOKENS`). Savings are baseline minus cost. 3. **The ledger records when GPUs ran.** A self-hosted endpoint bills while it runs, so per-token prices alone flatter the student. The `compute_intervals` table holds three kinds of interval: `student_endpoint`, `training_job` and `eval_job`. Each keeps the rate it was recorded at, so changing a setting never reprices history. 4. **The cost report prices the window two ways.** The per-token view sums `cost_usd`. The GPU-hour view adds teacher spend, the endpoint's hours, the training jobs' hours, and the per-token student spend in hours when no endpoint uptime was recorded. GPU-hour savings can be negative. They are never clamped to zero. 5. **It computes break-even and the hosted crossover.** Break-even is the number of student answers per endpoint hour at which the endpoint's hours cost exactly the teacher spend they avoid. The crossover is the volume above which the endpoint is cheaper per answer than the hosted fallback (Nemotron 3 Nano 30B on Token Factory). 6. **It recommends one of four things, in plain English.** `self_host`, `hosted_fallback`, `scale_to_zero` or `insufficient_data`, with an explanation that quotes the numbers and the price label ("unverified list prices" until an operator confirms them; the per-token prices were checked on 29 Oct 2026, the GPU-hour rate was not).
```mermaid flowchart TB R(["One request"]) --> S["Student samples
live and shadow"] R --> T["Teacher calls: escalation,
audit, arbitration, pinned"] R --> B["All-teacher baseline"] S --> SC["student_cost_usd"] T --> TC["teacher_cost_usd"] SC --> C["cost_usd"] TC --> C B --> V(["savings =
baseline − cost"]) C --> V class S,SC student class T,TC teacher ```
**Reading this diagram.** Every model call a request makes is priced and lands in one of two parts: the student's samples (for the live version and a shadow) or the teacher's calls. Their sum is the request's `cost_usd`. Next to it, the request records what the teacher alone would have cost, and the saving is the difference.
The response's `usage` field is not this bill. It counts only the tokens of the answer the client received, as if that model had answered alone. `ratchet.cost_usd` prices every call Ratchet made, which `ratchet.usage_breakdown` lists. ### Savings per token and savings by GPU-hours The two views answer different questions. | View | Total | What it assumes | |---|---|---| | Per token (`per_token`, the default headline) | Σ `cost_usd` | The student costs what the hosted Nano costs per token ($0.06 in, $0.24 out per 1M). Cheap per answer, but it ignores an idle GPU | | By GPU-hours (`gpu_hours`) | teacher spend + endpoint GPU cost + training GPU cost + student per-token spend in hours with no recorded uptime | You pay for every hour the endpoint and the training jobs ran, busy or idle |
```mermaid flowchart TB subgraph PT["Per token view"] direction LR P1["Each student answer
at the hosted Nano's
token price"] --> P2["Sum of
cost_usd"] P2 --> SAV1(["baseline − total"]) end subgraph GH["GPU-hours view"] direction LR G1["Teacher spend"] --> G5["Total paid"] G2["Endpoint and training
hours × rate, busy or idle"] --> G5 G4["Student tokens in hours
with no recorded uptime"] --> G5 G5 --> SAV2(["baseline − total
can be negative"]) end PT ~~~ GH class G2 student class G1 teacher ```
**Reading this diagram.** Both views start from the same baseline, what an all-teacher run would have cost, and subtract what you actually pay. They differ in what counts as paid. The per-token view charges only for tokens, so an idle GPU costs nothing in it. The GPU-hours view charges for every hour the endpoint and the training jobs ran, so at low traffic the same requests can show a saving in one view and a loss in the other. Neither is clamped to zero.
`RATCHET_STUDENT_COST_MODEL` picks which one is the report's `headline`. It changes nothing else: both views are always in the report, and each request's `cost_usd` stays per token, so history never moves. ### Where the GPU-hours come from The report only knows the hours somebody recorded. Four writers record them: - **The CLI**: `python -m ratchet.cost endpoint up` and `down` on the ledger file (`RATCHET_DB`). `close KIND` closes a job whose end was lost. - **The API**: `POST /ratchet/cost/endpoint` with an admin key. Each change writes an `endpoint` event that names the admin, never the key. - **The deploy script**: `deploy/student_endpoint.sh` runs `python -m ratchet.cost endpoint up|down --source deploy` after `create`, `start` and `stop`. It is best effort: run away from the ledger, the deploy still succeeds and prints the command to run there. - **Night School** in live mode wraps the training run in a `training_job` interval (wall-clock time, so an upper bound on the job's billed time). Overlapping records of one kind are merged and billed at the highest rate among them, so nothing is counted twice. An interval still open counts up to now. A training job open for more than 12 hours gets a warning, because it probably ended without anyone recording it. Nebius words the billing like this: a stopped endpoint is not billed for compute or its own storage, but a mounted Object Storage bucket still is ([Nebius Serverless overview](https://docs.nebius.com/serverless/overview), read Oct 21, 2026). ### Break-even, first in one line Here is the kid version again, with the numbers the docs use. The endpoint costs about $1.35 per GPU-hour (an L40S on demand, [Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026; an estimate until your bill confirms it). At list prices a typical ticket-triage call to Ultra costs about $0.0013 (about 150 input and 380 output tokens), so each such answer the student serves avoids about $0.001 of Ultra spend (an estimate). So: ```text break-even student answers per endpoint hour = GPU price per hour ÷ teacher spend avoided per student answer ≈ $1.35 ÷ $0.001 ≈ 1,350 ``` That is **about 1,350 student answers an hour, roughly 32,000 a day** (estimate). Below that line, a 24/7 endpoint costs more than the teacher spend it saves. The code's default rate is 1.55 USD per GPU-hour, which gives about 1,550 at the same $0.001 (estimate, from the formula below).
```mermaid flowchart TB R["GPU price per hour
about $1.35, estimate"] --> BE["Break-even answers per hour
= price per hour ÷ avoided
teacher spend per answer"] S["Teacher spend avoided
per student answer
about $0.001, estimate"] --> BE BE --> CMP{{"Student answers served
per endpoint hour"}} CMP -->|"at or above
break-even"| WIN["The endpoint pays
for its hours"] CMP -->|"below
break-even"| LOSE["The endpoint costs
more than it saves"] LOSE --> OPT["Hosted fallback, or run
the endpoint only
while traffic arrives"] WIN --> UP["Keep it, and scale
to zero when idle"] class CMP gate class WIN ok class LOSE danger ```
**Reading this diagram.** The top two boxes are the only inputs: what an hour of GPU costs and how much teacher money one student answer avoids. Dividing one by the other gives the number of answers an hour the student must serve before the endpoint is worth its rent. Everything after that is a comparison with the traffic you really have. Because the price sits on top of the division, a cheaper GPU lowers the bar in proportion. Both inputs are estimates until a bill confirms them.
### Break-even, precisely With `R` the endpoint's price per hour as recorded in the window (`endpoint_usd_per_hour`: its GPU cost divided by its hours, or the configured `RATCHET_STUDENT_GPU_USD_PER_HOUR × RATCHET_STUDENT_GPUS` while no uptime is recorded) and `S` the student-served requests: ```text s̄ = Σ over student-served requests (baseline − teacher spend) / S avoided teacher spend per student answer break-even = R / s̄ student answers per endpoint hour (null when s̄ ≤ 0) h̄ = fallback price × (all student calls' tokens) / S what the hosted fallback would bill per answer crossover = R / h̄ above it, the endpoint is cheaper per answer observed = student answers in buckets with uptime / endpoint hours idle hours = endpoint hours in buckets when no request reached the student ``` A bucket is an hour on a live ledger. Break-even prices the endpoint's hours only, not training. So a window can be above break-even and still lose money by GPU-hours once the training jobs are added, and the explanation then says exactly that. ### The four recommendations `ratchet/cost.py` has exactly four outcomes (`RECOMMENDATIONS`), decided in this order:
```mermaid flowchart TB A{{"Endpoint uptime recorded?"}} -->|yes| B{{"At or above break-even
and the crossover?"}} B -->|no| C{{"Idle more than
half its hours?"}} C -->|no| D{{"Has the student
served anything?"}} D -->|yes| E{{"Below the hosted
crossover?"}} B -->|yes| SH(["self_host"]) E -->|yes| HF(["hosted_fallback"]) C -->|yes| SZ(["scale_to_zero"]) E -->|"no: run fewer hours"| SZ A -->|no| I(["insufficient_data"]) D -->|no| I class A,B,C,D,E gate class SH ok class I outline ```
**Reading this diagram.** The checks run top to bottom in this order, and the first that settles it wins. Four answers are possible, and two of them can be reached in two ways: `insufficient_data` when nothing is recorded or the student has served nothing, and `scale_to_zero` when the endpoint is mostly idle or when neither it nor the hosted fallback pays at this volume. The table below says what to do for each.
| Recommendation | Means | What to do | |---|---|---| | `self_host` | The endpoint served at least the break-even volume per hour it ran, and more than the crossover | Keep it. If the explanation says it was often idle, scale it to zero between bursts | | `hosted_fallback` | Below the crossover: per answer, the untuned hosted Nano would cost less than the endpoint. Below break-even the explanation opens "Self-hosting does not pay at this volume" | Leave `RATCHET_STUDENT_BASE` empty so the hosted Nano serves the student role per token, or run the endpoint only while traffic arrives | | `scale_to_zero` | More than half the endpoint's hours (`IDLE_SHARE`, 0.5) saw no request reach the student, or neither the endpoint nor the hosted fallback pays at this volume | Stop the endpoint when idle (`bash deploy/student_endpoint.sh stop`) and start it for busy periods | | `insufficient_data` | No uptime recorded in the window, or the student has not served a request yet | Record uptime, or wait for a trained version | The explanation says two things separately so the words never contradict the figures: whether the observed volume covers the endpoint's hours, and whether the window lost money by GPU-hours. It never says "pays" when GPU-hour savings are negative. ### The hosted-fallback crossover, and its caveat The hosted fallback is Nemotron 3 Nano 30B on Token Factory, untuned, paid per token. Answers it gives carry the badge. With list prices and the token counts above it costs about $0.00003 per answer, so a $1.35-an-hour endpoint beats it per answer only above roughly **45,000 student answers an hour** ($1.35 ÷ $0.00003, an estimate). That gives a middle band. Take a customer sending 10M requests a month, about 14,000 an hour. That is far above break-even: a GPU running all month costs about $990 at $1.35 an hour (730 hours), against about $9,300 of avoided Ultra spend (an estimate). The GPU-hour savings are positive. The report still says `hosted_fallback`, worded "Self-hosting costs less than the teacher, but the hosted fallback would cost less per answer". The caveat is printed with every `hosted_fallback` explanation (`CROSSOVER_CAVEAT`): the crossover assumes the hosted fallback serves as many requests as the tuned student. It is untuned, so more of its requests may go to the teacher, and each of those costs about $0.001 of Ultra. Ratchet does not measure the hosted model's coverage, because the two are never served side by side. Before switching, run a day on each and compare the student's share of requests in `/ratchet/stats`. ### When self-hosting does not pay Plainly: **at low volume, a self-hosted student endpoint loses money.** - At a few thousand requests a day, a 24/7 endpoint runs at a small fraction of break-even. Per token the same traffic can look like a large saving; by GPU-hours it can be a loss of hundreds of dollars over ten days. Both numbers are true; only the second is what you would pay. - Below break-even the honest options are the hosted fallback (pay per token, no GPU running) or an endpoint that runs only while traffic arrives. - Above break-even, the levers are more traffic per GPU (several tasks or customers as LoRA adapters on one vLLM endpoint) and fewer idle hours. Ratchet never promises savings below break-even, and the cost report says so in its own words. ### Prices used Token prices are Token Factory list prices from `DEFAULT_PRICES` in `ratchet/config.py`, checked on 29 Oct 2026: the four teacher and judge models' per-token prices match the price list Token Factory serves, and `scripts/day1_check.py` passed ([evidence/day1](https://github.com/harshbhx/ratchet/tree/main/evidence/day1)). The GPU-hour rate is still **unverified**, so every report says "unverified list prices" until an operator sets `RATCHET_PRICES_VERIFIED=1`. | Model (key in `DEFAULT_PRICES`) | Role | Input, USD per 1M tokens | Output, USD per 1M tokens | |---|---|---|---| | `nvidia/Nemotron-3-Ultra-550b-a55b` | Teacher | 1.00 | 3.00 | | `nvidia/nemotron-3-super-120b-a12b` | Judge | 0.30 | 0.90 | | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` | Hosted fallback student | 0.06 | 0.24 | | `student` | Per-token proxy for the self-hosted student | 0.06 | 0.24 | GPU rates come from `RATCHET_STUDENT_GPU_USD_PER_HOUR` and `RATCHET_TRAIN_GPU_USD_PER_HOUR` (with `RATCHET_STUDENT_GPUS` and `RATCHET_TRAIN_GPUS`, 1 each). Both default to 1.55 in the code. These docs use $1.35 per GPU-hour, the L40S on-demand price on Nebius's [Compute pricing page](https://docs.nebius.com/compute/resources/pricing) when read on Oct 22, 2026; that is an estimate until it matches your bill. Break-even scales with the rate. Set the variables to what Nebius actually bills you. Estimates for running it, from the [deployment guide](/reference/deployment/#4-cost-estimates) (on-demand prices, read Oct 22, 2026): | Item | Assumption | Estimate | |---|---|---| | Training run | L40S at about $1.35 per GPU-hour, 30 to 60 min a night | about $0.70 to $1.35 a night | | Student endpoint | L40S at about $1.35 per GPU-hour | about $32 per full day if left on | | Teacher calls | Ultra $1 / $3 per 1M tokens, about 150 input and 380 output tokens per call | about $1.30 per 1,000 calls | - **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)). - **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)). - **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). - **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). - **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate. - **The nightly schedule on live traffic.** Night School runs by hand today; the timer is scripted. - **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline. - **Measured savings.** Every cost figure on this page is an estimate from list prices (the L40S at $1.35 per GPU-hour is from [docs.nebius.com/compute/resources/pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026). The text-to-SQL adapter's [results](/guides/results/) include an estimated token cost on its exam rows, and that estimate has no GPU-hour bill in it. Savings are reported from a published live run, and none is published yet. Run every script with `DRY_RUN=1` first. ### Spend and rate limits The cost report looks back. Spend limits stop money going out in the first place. Per API key, `RATCHET_KEY_LIMITS` (over the `RATCHET_API_KEY_*` defaults) sets a daily request quota, daily and monthly US dollar budgets, requests a minute, requests in flight, the tasks the key may send, and whether its traffic may become training data. For the whole gateway, `RATCHET_DAILY_BUDGET_USD` caps `/v1` spend per UTC day; `RATCHET_BUDGET_MODE=block` then refuses every chat request, while `student_only` keeps serving confident student answers and refuses only what needs the teacher. By default a `budget` event fires at 50, 80 and 100% of every cap (`RATCHET_BUDGET_ALERT_PCT`). The counts live in memory and are read from the ledger at startup, so a restart does not reset them. Two limits on this: the dollars are per-token list prices, so endpoint GPU-hours are not part of them; and a request in flight can overshoot a cap by its own cost. Scout has its own budget guard ([Scout](/guides/scout/)). ### Savings receipts The dashboard's savings come from the token formula, which assumes 300 reasoning tokens nobody measured. What would be billed comes from a **savings receipt** instead: one per ledger day for the gateway and one per API client, sealed after the day ends and never changed. The baseline of the answers the student served is calibrated on the audited answers' real teacher bills (a ratio estimator with a confidence bound), and the fee is `RATCHET_TAKE_RATE` (25% by default) of the one-sided 95% lower bound of the savings, never of the formula's estimate. A task with fewer than `RATCHET_RECEIPT_MIN_AUDITS` (30) audits that day is reported but not billed. Each receipt is a Merkle tree over the day's request metadata (no text), signed with the BOM key when one is set, so a client can prove any of its requests is in it and see any later edit. It cannot prove the costs logged at request time: reconcile those with the Token Factory bill. Details in [Evidence](/guides/evidence/). ## For engineers **Code** (links to `main`): | File | What to read | |---|---| | [`ratchet/cost.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/cost.py) | `cost_report` (both views, break-even, crossover, idle hours, daily rows, warnings), `segments` (merge intervals, highest rate wins), `split`, `_recommend`, `_explain`, `CROSSOVER_CAVEAT`, `RECOMMENDATIONS`, `IDLE_SHARE`, `record_endpoint` (shared by the CLI and the API), `record_simulated_day`, `stats_block`, `main` | | [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) | `Ratchet.handle` adds each call's price to `cost_usd` and to its part; `_call_cost`; `_formula_baseline` (the student-served baseline) | | [`ratchet/config.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/config.py) | `DEFAULT_PRICES`, `Price`, `GpuRate`, `CostPolicy`, `Settings.cost_policy()`, `COST_MODELS`, `MAX_USD_PER_GPU_HOUR` (1,000), `MAX_GPUS` (1,024) | | [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py) | `compute_start`, `compute_stop`, `compute_record`, `compute_intervals`, `cost_by_day`, `COMPUTE_KINDS` | | [`ratchet/nightschool/orchestrate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/orchestrate.py) | `run_night(train_gpu=...)`: the live training interval | | [`deploy/student_endpoint.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/student_endpoint.sh) | Records uptime after `create`, `start` and `stop` | **Settings** (from `ratchet/config.py` and [CONFIGURATION.md](/reference/configuration/)): | Variable | Default | Meaning | |---|---|---| | `RATCHET_STUDENT_COST_MODEL` | `per_token` | Headline view: `per_token` or `gpu_hours` | | `RATCHET_STUDENT_GPU_USD_PER_HOUR` | `1.55` | Student endpoint, USD per GPU-hour (0 to 1,000). The code's default; these docs use 1.35, Nebius's on-demand price for an L40S. Set the rate on your bill | | `RATCHET_STUDENT_GPUS` | `1` | GPUs the endpoint runs on | | `RATCHET_TRAIN_GPU_USD_PER_HOUR` | `1.55` | Training job, USD per GPU-hour. The code's default; set 1.35 or the rate on your bill | | `RATCHET_TRAIN_GPUS` | `1` | GPUs a training job runs on | | `RATCHET_PRICES_VERIFIED` | `0` | `1` once the day-1 checks pass and the rates match your Nebius bill | | `RATCHET_TEACHER_REASONING_TOKENS` | `300` | Reasoning tokens the student-served baseline assumes | A bad value stops startup with an error that names the variable. **Endpoints** ([API.md](/reference/api/#cost)): - `GET /ratchet/cost?days=N`: the whole report as JSON. Access level `dashboard`: in live mode it needs an admin or dashboard key unless `RATCHET_PUBLIC_DASHBOARD=1`. - `POST /ratchet/cost/endpoint`: body `{"state": "up"}` or `"down"`, with optional `usd_per_hour`, `gpus` and `note`. Admin key; admin key required. - `GET /ratchet/stats`: a compact `cost` block (`model`, `endpoint_hours`, `gpu_hours_savings_pct`, `break_even_student_requests_per_gpu_hour`, `recommendation`). **CLI**: ```bash python -m ratchet.cost report [--days 7] [--json] # both views, break-even, crossover, recommendation python -m ratchet.cost endpoint up --note "recording" # also --usd-per-hour, --gpus, --at TIME python -m ratchet.cost endpoint down python -m ratchet.cost close training_job --at 2026-10-22T03:10:00Z # a job whose end was lost ``` The CLI opens an existing ledger file only (`--db` or `RATCHET_DB`), exits 2 with `error: ...` on bad input, and refuses an `--at` in the future. **Events and warnings**: an `endpoint` event on every recorded change; report warnings for a job open more than 12 hours, for student answers in hours with no recorded uptime (priced per token as the hosted fallback's), and for rows from before v0.5 that have no student/teacher split (counted as teacher spend). ## Common questions ### Why do the dashboard and the cost report show different savings? The dashboard's figures are the per-token view: the student priced at the hosted Nano's token price. The cost report adds the GPU-hour view, which charges every hour the endpoint ran. At low volume the first is positive and the second negative. Both are correct; read the GPU-hour one before deciding to self-host. ### What does it cost to run Ratchet? The gateway does no inference, so it runs on a small CPU VM (about $72 for Oct 29 to Dec 15, 2026, estimate, [VM kit guide](/reference/deploy-nebius-vm/)). On top of that: teacher calls per token, a training job per night (about $0.70 to $1.35, estimate), and the student endpoint while it runs (about $32 per full day, estimate), both at about $1.35 per GPU-hour. All of these are estimates until a live run measures them. ### At what traffic does self-hosting pay? About 1,350 student-served answers per endpoint hour at about $1.35 per GPU-hour, roughly 32,000 a day (an estimate from list prices and the token counts above). Below that, use the hosted fallback or run the endpoint only while traffic arrives. ### Is the hosted fallback just cheaper, then? Per answer, often yes: below roughly 45,000 answers an hour it is (estimate). But it is untuned, so it may escalate more requests to Ultra, and those cost far more than the GPU-hours saved. The report always says the crossover assumes equal coverage. Compare a day on each before switching. ### What if nobody records the endpoint's uptime? Then the report does not know the GPU cost. Student answers in hours with no recorded uptime are priced per token and counted in a warning. Use `deploy/student_endpoint.sh` next to the ledger, or record `up` and `down` yourself, and compare `endpoint_hours` with the Nebius bill weekly. ### Are these real numbers? Not yet. The per-token prices were checked on 29 Oct 2026 against the price list Token Factory serves (`/v1/models?verbose=true`) and match; the GPU-hour rate is still unverified, and every figure on this page is an estimate. As of Oct 29, 2026 the teacher runs on Nebius Token Factory and Ratchet's own adapter is not connected to the live gateway (one was served in speed tests and deleted), where a random 40% of eligible requests are answered by the hosted Nano 30B; the live-run harness will publish measured numbers, including the Nebius bill. ### Does the baseline overstate savings? It can. The student-served baseline assumes the teacher would have written 300 reasoning tokens. Audits measure what the teacher really bills, and the savings receipts and the ledger replay correct the baseline with them. The dashboard's figures do not. ## Related The cost model in full: [Architecture, cost accounting](/reference/architecture/#7-cost-accounting). --- # Evidence you can check Source: https://docs.ratchet.dualmindlab.tech/guides/evidence/ Ratchet writes its claims about each adapter and each day's savings into signed documents whose numbers can be recomputed offline, so a customer or auditor can check them instead of trusting the gateway that served them. A restaurant can say anything about itself. Ratchet hands you things you can check yourself, at home, without asking the restaurant again. - **The certificate on the wall** (the signed bill of materials) lists which chefs taught the apprentice, under which licenses, how many recipes were studied and how the apprentice did on the sealed exam. It carries the restaurant's stamp. Anyone with a copy of the stamp's pattern can tell if a single word was changed after stamping. - **The exam score with a safety margin** (certified precision) does not say "the apprentice got 99%". It says "even if we were unlucky with the exam questions, the real score is at least this". - **The fingerprint of the recipe book** (a committed training set) lets you check that one recipe was in the study pile, or was not, without seeing the whole book. It also proves that no dish from the sealed exam was ever studied. - **The time machine** (the ledger replay) answers "what would last week have looked like with a different bar?". - **The receipt** (the savings receipt) is each day's bill, counted on the cautious side, with a fingerprint of every order on it. Each receipt includes yesterday's fingerprint, so changing an old receipt breaks the chain. Each piece also says plainly what it cannot prove. A stamp shows nobody changed the certificate; it does not show the certificate was true when it was written. ## How it really works The evidence is written at fixed moments, by code that draws no random numbers and calls no model, so it never changes a release decision. 1. **The night, before training.** Night School builds a set commitment (a sparse Merkle root) over the exact training rows and another over the golden rows, before the paid training job starts, so a problem fails the night before any GPU time. 2. **The night, after training.** Right after the new version is created, the leaves behind both roots are written once to the ledger's `version_leaves` table, which refuses any later change. The adapter is scored on the golden set (with its certificate), the live version is re-scored on the same rows, the paired comparison is added, and only then is the bill of materials written and signed. In live mode a copy goes into the adapter's folder in the bucket, with its `NOTICE.md`. 3. **Every request.** Each request the student answers logs the threshold it was held to, the probability that the teacher labels it (`label_p`), whether the student agreed with that label, and the token formula's baseline. This is what the replay and the receipts read. 4. **After each ledger day ends.** The day's savings receipts are sealed, once: one for the whole gateway and one per API client, each signed and chained to that client's previous receipt. 5. **Any time later, anywhere.** A customer downloads the documents and checks them with the command-line tools in this repository, using only the files and a public key the operator published. | Evidence | What it proves | What it does not prove | Check it with `python -m` | |---|---|---|---| | Signed BOM | The operator signed these claims about this adapter at training time, and nothing changed since | That the weights were trained on that data | `ratchet.bom verify` | | Certificate | With probability at least 95%, what this version serves agrees with the teacher at least `lcb` of the time, on traffic like the golden rows | Correctness, or behaviour on traffic unlike the golden rows | `ratchet.stats check` | | Committed training set | This example was, or was not, in the committed training or golden set | That the weights came from that set | `ratchet.proof verify` and `heldout` | | Ledger replay | An estimate of precision, coverage and cost at another threshold or audit rate | Anything signed: it reads the ledger as it is, and its interval is approximate | `ratchet.replay` | | Savings receipt | The day's leaves were not edited after sealing, and the bill follows from them | That the costs written at request time were right | `ratchet.receipt verify` |
```mermaid sequenceDiagram participant N as Night School participant Ledger as Ledger participant G as Gateway participant C as Auditor N->>Ledger: commit the rows N->>N: train and sign N->>Ledger: write leaves once G->>Ledger: log each request G->>Ledger: seal each day C->>G: fetch the files C->>C: check offline ```
**Reading this diagram.** The first three arrows happen once a night: Night School commits the training and golden rows (their Merkle roots), trains, scores the adapter with its certificate, re-scores the live version, signs the bill of materials, and writes the leaves once (append-only). The next two happen all day: the gateway logs every request with its label probability and seals each ended day into a signed receipt. So the evidence is made by the same system that does the work, but at fixed moments that cannot change a decision. The last two arrows are the point: a customer or auditor fetches the BOM, proofs, receipts and public key, and checks them offline without calling the gateway again. [Proof you can run](/guides/proof/) lists the one command for each check.
A separate, simpler check exists for the text-to-SQL adapter: its exam is scored by running queries, and `measure_adapter.py verify` recomputes every number from the raw answers. That one does not need a gateway at all. See [A text-to-SQL adapter you can check](/guides/text-to-sql/) and [Results](/guides/results/). All the arithmetic comes from two standard-library modules, [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py) and [`ratchet/merkle.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/merkle.py), checked against hand-computed values and seeded Monte Carlo runs in `tests/test_stats.py` and `tests/test_provenance.py`. ## Signed bills of materials **In one sentence for a 10-year-old:** a bill of materials is a list of everything that went into one adapter, signed so that anyone with the public key can tell if a single character was changed after it was written. **Precisely.** For every adapter it trains, Night School writes one document, schema `ratchet.bom/v2`, from the exact rows it used: - `adapter`: the version, the adapter's name, its base model (`nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16`) and the ledger day. - `training`: the number of examples, the dataset hash, examples per task, `targets` (`teacher answers only`, or with customer-attested human labels), how many examples had PII redacted, where the examples came from, and the training set's commitment (`{alg: ratchet.smt/v1, root, size}`). - `golden`: the number of golden rows, their hash, `held_out: true`, golden rows with a human reference, and the golden set's commitment. - `sources`: each teacher with its license, license URL, whether it allows distillation and whether the license entry is Ratchet's own (`builtin`) or added by the operator (`config`); `label_sources` lists human labels apart, because a person is not a teacher. - `base_model_license`: the NVIDIA Nemotron Open Model License and its URL for the Nano base (or a warning for a base model Ratchet does not know). - `notices`: the exact text of the `NOTICE.md` that sits next to the adapter's weights, with its SHA-256. - `redactor`: a SHA-256 over the redaction rules, how many there are, and whether redaction was on. - `eval`: the golden eval at training time: precision, coverage, threshold, and since v0.5 the certificate, the frontier, the per-task bounds, the per-row outcomes and the paired comparison with the live version. The manifest is serialised as canonical JSON (sorted keys, no whitespace, UTF-8). The document carries the SHA-256 of those bytes and, when `RATCHET_BOM_SIGNING_KEY` is set, an **Ed25519** signature with the public key and a key id. It is written once and never rewritten: a later re-check changes `/ratchet/stats`, not the signed document. Versions trained before v0.4 have no BOM; v1 documents still verify.
```mermaid flowchart TB TR[("Training rows")] --> R1["Training set root"] GR[("Golden rows")] --> R2["Golden set root"] F["Teachers and licenses,
base model license,
NOTICE.md text and hash,
redaction rules hash,
golden eval, certificate"] TR --> M["Manifest"] GR --> M R1 --> M R2 --> M F --> M M --> CJ["Canonical JSON"] CJ --> H["SHA-256"] CJ --> SG["Ed25519 signature"] H --> DOC["BOM document"] SG --> DOC M --> DOC class TR,GR store class DOC student ```
**Reading this diagram.** Everything at the top goes into one manifest: the counts and hashes of the exact rows, the roots that commit to each set, and the facts about the adapter. The manifest is written as canonical JSON, and that one byte string is both hashed and signed. The document you receive carries the manifest, its SHA-256 and the signature, so changing any byte of it breaks the check.
**Check it:** ```bash python -m ratchet.bom keygen # once: the signing key, and the public key to publish curl -s localhost:8000/ratchet/bom/public-key # the key the gateway signs with curl -s localhost:8000/ratchet/versions/2/bom > bom.json python -m ratchet.bom verify bom.json --public-key # OK: signature valid (Ed25519, key ..., trusted key) python -m ratchet.bom verify bom.json --public-key --notice NOTICE.md # and the NOTICE.md next to the adapter python -m ratchet.live export --version 3 --out ./student-v3-export # hand an adapter over with its BOM and NOTICE.md ``` Pass the public key you published somewhere you control (a README, a website). Without `--public-key`, `verify` uses the key embedded in the document, which only shows the document is consistent with itself. Without a signing key, the document carries only its SHA-256: `verify` reports it as unsigned. Rotating the key does not invalidate old BOMs; keep every public key you have published listed. **What it does not prove.** A signed BOM is the operator's signed statement of the data and evaluation behind an adapter. It shows that nothing changed after signing, not that the weights were trained on that data, and not that the claims were true: a compromised signing key can sign anything. When examples are erased later, the versions that may have been trained on them are reported, and their BOMs stay exactly as written. ## Certified precision **In one sentence for a 10-year-old:** a certificate is the exam score with a safety margin: "even if we were unlucky with the exam questions, the real score is at least this". **Precisely**. The golden eval picks the threshold on the even half of the golden rows and measures on the odd half (see [Confidence](/guides/confidence/)). The odd half played no part in choosing the threshold, so its count of agreeing answers among the `m` it would serve is one exact binomial experiment. The one-sided Clopper–Pearson lower bound is the precision `p` at which getting `s` or more agreeing answers out of `m` would happen only `delta` of the time: ```text lcb = cp_lower(s, m, delta) # the p with P(X >= s) = delta, X ~ Binomial(m, p) certified = lcb >= target # RATCHET_CERT_TARGET 0.95, RATCHET_CERT_DELTA 0.05 ``` "Certified at 95% (lcb 96.4%, delta 0.05)" means: with probability at least 95%, the answers this version serves at its threshold agree with the teacher at least 96.4% of the time. For example, 129 agreeing out of 130 served held-out rows gives a point estimate of 99.2% and a lower bound of 96.4% (checked with `stats.cp_lower`). The point estimate alone cannot tell 0.96 from 0.99 on that many rows; the bound can. The eval also carries: - **The frontier:** the same counts and bound at every threshold of the fixed grid 0.95, 0.90, down to 0.50, each bound at `delta / 10`, so all ten hold at once (Bonferroni) and an operator may read any row afterwards: a certified trade-off between precision and the share of traffic served. - **Per task:** the counts and bound per task, so a weak task cannot hide inside a pooled number. A diagnostic, not corrected for the number of tasks, and never used for routing. - **The test vector:** one letter per held-out row (`S` served and right, `W` served and wrong, `E` escalated), which the paired comparison with the live version reads, and which `check` uses to recount everything. How many rows a certificate needs, from `stats.required_rows`: | Target (delta 0.05) | 0 wrong answers | 1 | 2 | 3 | |---|---|---|---|---| | 0.90 | 29 | 46 | 61 | 76 | | 0.95 | 59 | 93 | 124 | 153 | | 0.97 | 99 | 157 | 208 | 257 | Only served held-out rows count, so certifying 95% with one wrong answer takes about `2 × 93 / coverage` golden rows. **Check it:** ```bash python -m ratchet.stats plan --target 0.95 --delta 0.05 # 0 errors: 59 rows, 1: 93, 2: 124, 3: 153 python -m ratchet.stats bound --successes 127 --n 128 # one bound by hand: 0.9635 curl -s localhost:8000/ratchet/versions/2/certificate # signed and latest certificate, the gate's evidence, assumptions curl -s localhost:8000/ratchet/versions/2/bom > bom.json python -m ratchet.stats check bom.json --public-key # recompute every number from the signed counts ``` `check` verifies the signature, then recomputes every bound, verdict, frontier row, per-task row, Learn-then-Test step and paired p-value from the signed counts, and the counts from the test vector. It exits 1 on any difference above 1e-9. The certificate endpoint's `status` is `certified` or `not certified` from the latest certificate (the nightly re-score's, else the signed one), `suspended` while a drift alarm is open on the version, or `no certificate` for a version evaluated before v0.5. **What it does not prove.** Precision means agreement with Nemotron 3 Ultra () under the task's comparators, not ground truth. The bound assumes the golden rows look like live traffic, and they lean towards requests the student found hard (they come from escalations, audits and arbitrations). The guarantee is about the traffic the golden rows were drawn from, when they were drawn; drift breaks it. Every certificate is served with these assumptions in plain words. ## Committed training sets **In one sentence for a 10-year-old:** a Merkle root is a fingerprint of a whole list that lets you prove one item is on it, or is not, without showing the list. **Precisely**. Every BOM since v0.5 signs one root per split, over exactly the rows Night School handed to the trainer and to the golden eval: - **Key:** the example's hash, a SHA-256 over the task, the redacted messages and any stored generation conditions. Anyone holding the request can recompute it. - **Value:** the SHA-256 of `{completion, teacher, license, redactions}`: the stored teacher answer the student trained on, who wrote it, its license and the number of PII placeholders. A proof therefore binds what an example was trained on, not only that its key was there. - **Tree:** a collapsed sparse Merkle tree split on the key's bits. An empty node is 32 zero bytes, a node with one key is `sha256(0x00 || key || value)`, any other node is `sha256(0x01 || left || right)`. The root depends on the set alone, never on insertion order. - **Proof:** the sibling hashes along the key's path, plus where the path ends: the key's own leaf (included), an empty node, or another key's leaf that shares the path (absent). The **held-out certificate** is the golden set's leaves plus, for every golden key, a proof of absence from the training root: two of the loop's rules (golden rows are never trained on; precision is measured on rows the model did not see) as something you verify, not something you are told. Absence is sound even against whoever built the tree: a key's path is fixed by its bits, so a proof of inclusion and a proof of absence for the same key against the same root would need a SHA-256 collision. A Merkle tree over a sorted list is not used for absence, because a verifier cannot check that the list was really sorted.
```mermaid flowchart TB Req(["The request:
task and messages"]) --> Key["Example key"] PF(["Proof: siblings
and terminal"]) --> Fold["Hash up from the terminal
along the key's bits"] Key --> Fold Fold --> CR["Computed root"] BOM(["The adapter's BOM"]) --> Sig{{"Signature valid with
your public key?"}} Sig -->|"no"| Fail["FAIL"] Sig -->|"yes"| SR["Signed training root"] CR --> Cmp{{"Same root?"}} SR --> Cmp Cmp -->|"no"| Inv["invalid"] Cmp -->|"yes, terminal
is the key"| Inc["included"] Cmp -->|"yes, terminal is
empty or another key"| Abs["absent"] class Sig,Cmp gate class Fail,Inv danger class Inc,Abs ok ```
**Reading this diagram.** Two roots meet at the bottom. On the left the verifier rebuilds a root from the request and the proof, hashing up from the terminal one level at a time, deepest sibling first. On the right it takes the training root from the BOM, but only after the BOM's signature checks out with your public key. If the two roots are the same, where the proof's path ends says whether the example was included or is absent.
  • A check
  • A valid answer: included or absent
  • The proof fails
The verifier never trusts the root, result or value written in the proof file; it checks against the root the signed BOM names. **Check it:** ```bash curl -s localhost:8000/ratchet/versions/2/bom > bom.json # ex.json: the body of a request that was stored as an example (task and messages) curl -s "localhost:8000/ratchet/versions/2/proof?split=train" -H 'content-type: application/json' -d @ex.json > proof.json python -m ratchet.proof verify proof.json --bom bom.json --expect included curl -s localhost:8000/ratchet/versions/2/heldout > heldout.json python -m ratchet.proof heldout heldout.json --bom bom.json # OK: all N golden examples provably absent from the training set python -m ratchet.proof key --task ticket-triage --messages request.json # the key a request is stored under ``` Add `--public-key ` to `verify` and `heldout` to trust the signer, not just the document. The proof (`POST /ratchet/versions/{id}/proof`) and held-out (`GET /ratchet/versions/{id}/heldout`) endpoints need an admin key in live mode (`Authorization: Bearer $ADMIN_KEY`); They are admin-only on purpose: for a text someone can guess, a yes-or-no answer about the training set is a leak. A proof is about log2 of the set's size in siblings. A held-out certificate for 133 golden rows against 1,288 training leaves is about 113 KB of JSON. **What it does not prove.** The proofs show the operator's binding commitment, signed at training time, to the dataset an adapter was trained and graded on. They do not show that the weights came from that dataset: an operator who signs one set and trains on another is not caught by a Merkle proof. A proof of absence also names the neighbouring training example's key and value hashes, so held-out certificates go to auditors, not to the public. ## Ledger replay **In one sentence for a 10-year-old:** because Ratchet decides which plates the inspector tastes by a coin flip with known odds, each tasted plate can stand in for the untasted plates like it, so last week can be replayed with a different bar. **Precisely**. Ratchet designs its own labelling, so it knows the chance of every teacher label it sees: 1 for a request the student answered and escalated, the audit rate for an answer it served. Each request logs that probability (`label_p`), the threshold it was held to, and whether the student's answer agreed with the teacher's when the teacher answered under an audit's conditions (`student_agree`). For one version, the replay then computes at any threshold `t` or audit rate `a`: - **coverage:** exact, from the logged confidences, no label needed; - **precision:** a weighted mean of `student_agree` over labelled served requests, each weighted by `1 / label_p` (Hajek), with an approximate interval (Wilson at Kish's effective sample size); null, with a reason, when nothing could label the answers it would count (an audit rate of 0); - **cost and savings:** student spend, plus the all-teacher cost of every request not served, plus the audit rate times that of every request served; - **a frontier** from 0.30 to 1.00 in steps of 0.01, and an advisory suggested threshold, which nothing applies: thresholds change only through Night School; - **a calibrated baseline:** the ratio of what the teacher really billed on audited answers to the token formula's estimate (which assumes 300 reasoning tokens, `RATCHET_TEACHER_REASONING_TOKENS`). **Check it:** ```bash curl -s "localhost:8000/ratchet/replay?threshold=0.4&audit_rate=0.05" # what happened, and what would have curl -s "localhost:8000/ratchet/replay/frontier?target=0.97" # thresholds 0.30 to 1.00, an advisory suggestion python -m ratchet.replay --db ratchet.db # the same offline, from a ledger file ``` **What it does not prove.** The replay is an analysis of the ledger as it stands, not a signed document. Its interval is approximate and computed on the same traffic the operator is looking at, so it is never a certificate and never a release decision. An escalation answered under a client's own sampling parameters, or for more than one choice, gives no label. A request whose escalation failed outright was never logged, so it is not in the replay. It replays one version at a time, because each adapter's confidence scale is its own. ## Verified savings receipts **In one sentence for a 10-year-old:** a receipt is the day's bill with a fingerprint of every order on it, and each receipt carries yesterday's fingerprint, so changing an old receipt breaks the chain. **Precisely**. The plan for a managed offering is to charge a share of verified savings (nothing is sold today), so the savings must be checkable by the customer. For each ledger day there is one receipt for the whole gateway and one per API client: 1. **Leaves.** Each request of the day becomes one leaf of metadata, never text: `id`, `ts_ms`, `day`, `task`, `client` (the API key's name, never the key), `served_by`, `route`, `student_version`, `escalated`, `audited`, `label_p_ppb` and the amounts (`cost_nusd`, `student_cost_nusd`, `teacher_cost_nusd`, `baseline_nusd`, `baseline_est_nusd`). Amounts are whole nano-USD, so sums are exact and the signed JSON holds no floating-point number. 2. **Groups.** A request is **measured** when the teacher answered it (its bill is the baseline), **served** when the student served it on the audit design, and **unverified** when a stand-in answered (route `degraded` or `fallback_teacher`). 3. **Calibration per task.** For each task, the audited served answers give a ratio `R` between the real teacher bills and the formula's estimates (weighted by `1 / label_p`), with a variance by the normal approximation. A task needs `RATCHET_RECEIPT_MIN_AUDITS` (30) audits that day; below that it is `insufficient_audits` and its savings are reported but not billed. 4. **The bill.** The billable savings (measured requests plus the calibrated tasks' served answers, minus any stand-in losses) less a margin of `ceil(1.645 × sqrt(sum of variances))` give the one-sided 95% **lower bound**. The fee is `RATCHET_TAKE_RATE` (0.25) of that bound, after the day's spend outside requests, when it is positive, rounded down to a nano-USD. With no calibrated task the basis is `formula` and the fee is 0. 5. **Seal.** The leaves form an RFC 9162 Merkle tree (the Certificate Transparency kind). The receipt names its root and size, its totals, the calibration and the bill, and `prev_sha256`, the hash of the same client's previous receipt (64 zeros for the first). It is signed with the BOM key and written once, after the day has ended (on a live ledger, 5 minutes after). The ledger refuses any later change.
```mermaid flowchart TB D1["Day 1 receipt"] -->|"its sha256 is"| D2["Day 2 receipt:
prev_sha256"] D2 -->|"its sha256 is"| D3["Day 3 receipt:
prev_sha256"] L[("Day 3
request leaves")] --> T["RFC 9162
tree root"] T --> D3 class L store ```
**Reading this diagram.** Each receipt names the hash of the same client's previous receipt, so the receipts form a chain: change day 1 and day 2's `prev_sha256` no longer matches. Each receipt also names the root of a Merkle tree over that day's request metadata, which is what lets one request be proved to be in it.
Every chat response carries its ledger row id in `X-Ratchet-Request-Id` and `ratchet.row_id`, so a client can later ask for a proof that its request is in that day's receipt. With API keys on, a key reads only its own receipts and proofs and an admin key reads all of them; live mode without any keys serves no receipts, because the leaves list every request's time, task and cost. The gateway-wide receipt also shows the day's savings by GPU-hours paid, which are often negative at low volume. That view is informational (billing uses the per-token view); the [cost report](/guides/cost-and-savings/) is what says when self-hosting does not pay. When the teacher's real reasoning differs from the formula's assumed 300 tokens, the receipts measure the ratio `R` from audited answers and correct the baseline. **Check it:** ```bash curl -s localhost:8000/ratchet/receipts/3 > r3.json # one day's receipt (your own, with an API key) curl -s localhost:8000/ratchet/receipts/2 > r2.json # the day before, for the chain curl -s localhost:8000/ratchet/receipts/3/proof/3006 > p.json # a row id you saw (ratchet.row_id) is in it curl -s localhost:8000/ratchet/receipts/3/leaves > leaves.jsonl # every leaf: recompute everything python -m ratchet.receipt verify r3.json --public-key --prev r2.json --proof p.json --rows leaves.jsonl python -m ratchet.live seal # live: seal every day that ended (hourly cron) ``` `verify` checks the hash and signature, that no floating-point number is in the document, the chain (it recomputes the previous receipt's hash rather than reading it), the request's inclusion against the signed root and size, and with `--rows` rebuilds the root and recomputes every total, estimate and the fee. It exits 0 when everything holds, 1 when anything fails, 2 on bad input. **What it does not prove.** Receipts make edits after sealing detectable and tie the baseline to audited teacher bills. They cannot prove the costs written at request time: reconcile the summed teacher spend (`totals.teacher_cost_nusd` over the days) with the Token Factory bill for the same API key. The interval is a normal approximation, and heavier-tailed real bills may need more than 30 audits. The chain shows that a receipt was changed; it does not stop an operator from withholding one, so clients should keep the receipts they receive. Request ids are the ledger's global row ids, so a client can infer the gateway's total volume between two of its own requests. As of Oct 29, 2026, no day has been sealed on a live gateway and no live-traffic BOM or receipt has been signed. The one Night School run on Nebius (28 Oct 2026) wrote an unsigned bill of materials for a student the gate rejected ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)); that BOM was signed afterwards, on 29 Oct 2026, with Ed25519 key `81b0a086250a81d0`, and its content is unchanged from the run's own record (check it: `python -m ratchet.bom verify evidence/nightschool-nebius/ratchet_bom.signed.json --public-key Gh68hnDDlRSjBRxmc5dTo4l+rvqkftUsH7B2Nm72gpc= --notice evidence/nightschool-nebius/NOTICE.md`); the GPU steps have otherwise run only by hand. Real teacher calls on Token Factory built the text-to-SQL data, but no BOM, certificate, proof, replay or receipt has been produced on live traffic. The formats, the signatures and the verifiers are tested offline. ## For engineers | What | Where | |---|---| | BOM, signing and verification | [`ratchet/bom.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/bom.py): `manifest`, `canonical`, `document`, `verify`, `check_notice`, `commitment`, `Signer`, `keygen`; `python -m ratchet.bom keygen` or `verify` | | NOTICE.md | [`ratchet/nightschool/notices.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/notices.py): `render`, `bom_notice`, `write_adapter_files` | | Certificates | [`ratchet/certificate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/certificate.py): `certificate`, `frontier`, `per_task`, `outcome_vector`, `paired`, `recheck`, `status`, `ASSUMPTIONS` | | Exact statistics | [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py): `cp_lower`, `required_rows`, `mcnemar_one_sided`, `ltt_fixed_sequence`, `CERT_GRID`, `hajek_mean`, `kish_n_eff`, `ratio_estimate`; `python -m ratchet.stats plan`, `bound`, `mcnemar`, `check` | | Merkle trees | [`ratchet/merkle.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/merkle.py): `KeySet`, `verify` (sparse tree), `LogTree`, `verify_inclusion` (RFC 9162) | | Set commitments | [`ratchet/nightschool/dataset.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/dataset.py): `leaf_value`, `commitment`; [`ratchet/proof.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/proof.py): `request_key`, `resolve_key`, `verify_proof`, `verify_heldout`; `python -m ratchet.proof key`, `verify`, `heldout` | | Replay | [`ratchet/replay.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/replay.py): `load`, `at`, `sweep`, `frontier`, `ratio`, `suggest`; `python -m ratchet.replay` | | Receipts | [`ratchet/receipts.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/receipts.py): `leaf`, `calibrate`, `compute`, `seal_due`, `seal`, `verify_receipt`, `verify_chain`, `verify_proof`, `recheck`; `python -m ratchet.receipt verify` or `seal` | | Append-only tables | [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py): `version_leaves` and `receipts`, refused UPDATE and DELETE by trigger | | Tests | `tests/test_stats.py`, `tests/test_provenance.py`, `tests/test_licensing.py`, `tests/test_receipts.py`, `tests/test_nightschool.py` | | Variable | Default | Effect | |---|---|---| | `RATCHET_BOM_SIGNING_KEY` | empty | Base64 of a 32-byte Ed25519 seed; signs every BOM and every receipt. Empty: unsigned documents | | `RATCHET_PUBLIC_BOM` | `0` | `1`: the BOMs stay public while the dashboard API is private, so auditors need no key | | `RATCHET_ADMIN_KEYS` | empty | Admin keys; needed for the proof and held-out endpoints and for every receipt in live mode | | `RATCHET_CERT_TARGET`, `RATCHET_CERT_DELTA` | `0.95`, `0.05` | The certificate's target and error rate | | `RATCHET_CALIBRATION` | `point` | `ltt`: the threshold itself is certified (Learn-then-Test) | | `RATCHET_REDACT_PII` | `1` | Redaction before storage; an example's key is computed on the redacted text (`python -m ratchet.proof key --no-redact` for a gateway with it off) | | `RATCHET_TAKE_RATE` | `0.25` | The fee's share of a receipt's billable lower bound | | `RATCHET_RECEIPT_MIN_AUDITS` | `30` | Audits a task needs in one receipt before its baseline is calibrated and billed | | `RATCHET_TEACHER_REASONING_TOKENS` | `300` | The formula's assumed teacher reasoning, which the replay and the receipts measure | Endpoints ([API](/reference/api/)): `GET /ratchet/bom` (the whole ledger's provenance), `GET /ratchet/versions/{id}/bom`, `GET /ratchet/bom/public-key` (always public), `GET /ratchet/versions/{id}/certificate`, `POST /ratchet/versions/{id}/proof?split=train` (admin), `GET /ratchet/versions/{id}/heldout` (admin), `GET /ratchet/replay`, `GET /ratchet/replay/frontier`, `GET /ratchet/receipts`, `GET /ratchet/receipts/{period}`, `GET /ratchet/receipts/{period}/proof/{request_id}`, `GET /ratchet/receipts/{period}/leaves`. See [API: committed training sets](/reference/api/#committed-training-sets), [ledger replay](/reference/api/#ledger-replay) and [savings receipts](/reference/api/#savings-receipts). Events: `integrity` (stored leaves or request rows no longer rebuild a signed root: the endpoint answers 500 and never serves a proof that would fail), `receipt_skipped` (a day's request rows were folded by retention before it was sealed). The runbook covers [checking a certificate](/reference/runbook/#checking-a-certificate) and [proving what an adapter was trained on](/reference/runbook/#proving-what-an-adapter-was-trained-on). ## Common questions ### What does a customer need to check all this? The documents (a BOM, a proof file, a receipt and its neighbours), the public key the operator published somewhere other than the gateway, and this repository's command-line tools. No access to the gateway, the ledger or the training data. ### Can the operator fake it? The operator can sign false claims: a signature shows who signed and that nothing changed afterwards, not that the claims are true. What the operator cannot do is change a document after signing without the change showing, prove both that an example was and was not in one committed set, or alter a sealed receipt without breaking the chain. The weakest links are named on this page: the BOM and the commitments are statements about data, not about the weights, and receipts trust the costs written at request time. ### What if no signing key is set? BOMs and receipts are still written, with their SHA-256 but no signature. `verify` reports them as unsigned. That shows a document is internally consistent, not who wrote it; set `RATCHET_BOM_SIGNING_KEY` before handing documents to anyone. ### Why are the training-set proofs admin-only? Because the proof endpoint is a membership oracle: for a text someone can guess (a customer's email template, say), "was this in the training set?" leaks it. Proofs go to auditors through an operator with an admin key. Held-out certificates also disclose neighbouring examples' hashes, so they are shared with auditors, not published. ### Does any of this cost money or slow requests down? Not on the request path beyond logging a few values per request: no model call, no extra random draw. The work happens at night (building the trees and signing) and when a day is sealed. Certificates add about 2 KB to each eval at the reference run's golden sizes. ### Has any of it run on real traffic? Not the signed documents on this page. As of Oct 29, 2026, the signed BOMs, proofs, certificates and receipts described here have been produced only by the tests, with mock models (the Nebius Night School run wrote an unsigned BOM, signed afterwards on 29 Oct 2026 and not at creation, and its gate certificate did not certify). The real-model measurements are the text-to-SQL and tool-calling adapters' exams, scored by standalone scripts (see [Results](/guides/results/)). The live-run harness (`python -m ratchet.liverun`) has not run its measured mode. ## Related - [Verify: the commands](/reference/verify/): a copy-and-run sheet for the checks, with the output to expect. - [Confidence](/guides/confidence/): where the threshold and the held-out half come from. - [The release gate](/guides/release-gate/): how the certificate and the paired comparison are used. - [Licenses](/guides/licenses/): why only open teachers, and what the NOTICE.md says. - [Cost and savings](/guides/cost-and-savings/): the formula baseline, break-even and when self-hosting does not pay. - The sections on [what a certificate means](https://github.com/harshbhx/ratchet/blob/main/docs/FEATURES.md#what-a-certificate-means), [proving what an adapter was trained on](https://github.com/harshbhx/ratchet/blob/main/docs/FEATURES.md#proving-what-an-adapter-was-trained-on), [replaying past traffic](https://github.com/harshbhx/ratchet/blob/main/docs/FEATURES.md#replaying-past-traffic-at-another-threshold), [savings you can check](https://github.com/harshbhx/ratchet/blob/main/docs/FEATURES.md#savings-you-can-check) and [limitations](https://github.com/harshbhx/ratchet/blob/main/README.md#limitations). --- # Proof you can run Source: https://docs.ratchet.dualmindlab.tech/guides/proof/ Each claim about an adapter or a bill has a document and a command that checks it on your own machine, so you do not have to trust the gateway that made it. "Trust me" is cheap. A better deal is "here is the receipt, here is how to add it up yourself". Ratchet gives you a chain of six such receipts. The first says how well the adapter did on a sealed exam. The next says who signed the list of ingredients. The next lets you check that one recipe was, or was not, in the study pile. The next gives a safety margin on the exam score. The next replays past orders at another setting, as an estimate rather than a signed receipt. The last shows each day's bill with a fingerprint of every order, chained to the day before. Each link has one command. When a link is broken, the command says `FAIL` and why. ## How it really works 1. **The exam.** The adapter answers questions about databases it never trained on, and a CPU-only script recomputes the score from the raw answers (`scripts/measure_adapter.py verify`, and `scripts/toolcall_results.py --check` for tool calling). These are the checks that have been run on real models, for the text-to-SQL and tool-calling adapters. 2. **The signed bill of materials.** When Night School trains a version it writes a document that lists the teachers and their licenses, the base model's license, the data counts and the eval, and signs it with an Ed25519 key. `python -m ratchet.bom verify` checks the signature and every hash in it. 3. **The committed training set.** The same document signs a Merkle root over the exact training rows and another over the golden rows. `python -m ratchet.proof verify` shows that one example is in a set, or is not, without the whole set. `python -m ratchet.proof heldout` shows that no golden example is in the training root. 4. **Certified precision.** The eval carries an exact Clopper-Pearson lower bound on how often the served answers match the teacher. `python -m ratchet.stats check` recomputes it from the counts in the signed BOM. 5. **The time machine.** `python -m ratchet.replay` replays one version's logged traffic at another threshold or audit rate and estimates precision, coverage and cost. It reads the ledger, so it is an estimate, not a signed document. 6. **The savings receipts.** After each ledger day, the gateway seals one receipt per client: a Merkle tree over request metadata (no text), signed and chained to the previous day. `python -m ratchet.receipt verify` checks the signature, the chain and, with a proof, that one request is in a receipt.
```mermaid flowchart TB DATA[("Training and
golden rows")] --> ROOTS["Merkle roots
over each set"] ROOTS --> PROOF["Membership and
held-out proofs
ratchet.proof"] ROOTS --> BOM["Signed bill of materials
ratchet.bom verify"] EXAM["Exam on unseen databases
measure_adapter.py verify"] BOM --> LIC["Licenses and
NOTICE.md"] BOM --> CERT["Certified precision
ratchet.stats check"] EXAM --> CERT CERT --> GATE{{"Release gate decision"}} GATE --> LEDGER[("Ledger of
served requests")] LEDGER --> REPLAY["Replay at another threshold
ratchet.replay"] LEDGER --> REC["Sealed daily receipts
ratchet.receipt verify"] REC --> CHAIN["Chained to the
previous day"] class DATA,LEDGER store class GATE gate class EXAM student ```
**Reading this diagram.** Follow the arrows from the data. The training and golden rows are committed first, then the signed document is built on top of those commitments, and the certificate is read from the signed document. Once the version is serving, the ledger fills with request metadata, and both the replay and the receipts read from the ledger. A command sits on every box (each `ratchet.*` one runs as `python -m ratchet.*`), so each step down the chain can be checked on its own, and a change anywhere above breaks the checks below it. The blue box is the one link that has run on a real model.
  • Stored data
  • A release decision
  • Run on a real model
## What exists today, and what has run for real Only the text-to-SQL and tool-calling adapters have been run on real models, and only their exams and check scripts (`python scripts/text2sql/exam_numbers.py --check`, `python scripts/toolcall_results.py --check`). Every other link on this page is code in the repository, tested offline with mock models. Night School itself ran once on Nebius on 28 Oct 2026 (dataset import, training job, hot-load, golden eval; [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)), but its bill of materials was written unsigned (no signing key was set), so no BOM, proof or receipt was signed by a real run as it happened. None of it has run on real production traffic. Signed on 29 Oct 2026 with Ed25519 key `81b0a086250a81d0`, after the run; the BOM content is unchanged from the run's own record. Verify it with `python -m ratchet.bom verify evidence/nightschool-nebius/ratchet_bom.signed.json --public-key Gh68hnDDlRSjBRxmc5dTo4l+rvqkftUsH7B2Nm72gpc= --notice evidence/nightschool-nebius/NOTICE.md`: it prints `OK` for the signature and for the notice. Only this BOM is covered. Receipts and BOMs from live traffic stay unsigned until the live gateway has the key. | Link | In the repository | Checked by | Run on a real model | |---|---|---|---| | Exam and `verify` | Standalone scripts in `scripts/` | The [Results](/guides/results/) page | Yes: the text-to-SQL and tool-calling adapters | | Signed BOM | `ratchet/bom.py` | Tests | No | | Committed sets and proofs | `ratchet/proof.py`, `ratchet/merkle.py` | Tests | No | | Certified precision | `ratchet/certificate.py`, `ratchet/stats.py` | Tests and seeded Monte Carlo runs | No | | Replay | `ratchet/replay.py` | Tests | No | | Savings receipts | `ratchet/receipts.py`, `ratchet/receipt.py` | Tests | No | ## The checks, one by one The short version, one claim at a time; the full text for each follows below. `, and add `--notice NOTICE.md` to check the notice next to the adapter.\n\nA failure starts with `FAIL:`: a signature that does not match, or a hash that differs.' }, { title: 'The training set', body: 'Run `python -m ratchet.proof verify` for one example, or `python -m ratchet.proof heldout` for every golden example.\n\nA failure starts with `FAIL:`, for example when a proof does not rebuild the signed root.' }, { title: 'Certified precision', body: 'Run `python -m ratchet.stats check bom.json --public-key `.\n\nIt recomputes the bound from the signed counts and prints a line for each number that differs from the signed one.' }, { title: 'The time machine', body: 'Run `python -m ratchet.replay --db ratchet.db --version 3`.\n\nIt is an estimate, not a signed document: a `null` column means the logged data cannot support an estimate at that setting.' }, { title: 'The savings receipts', body: 'Run `python -m ratchet.receipt verify r.json --public-key --prev p.json`.\n\nA changed receipt fails with a line such as `FAIL: chain: prev_sha256 is not the sha256 of day N\'s receipt`.' }, ]} /> ### 1. The exam: `measure_adapter.py verify` - **What it proves.** Every number in `results.json` is recomputed from the raw answers, the exam file and the exam databases, and each of those files is the one the results were computed from (SHA-256). - **The command.** ```bash python scripts/measure_adapter.py verify \ --raw evidence/text2sql-exam2/answers/raw_v2.jsonl \ --results evidence/text2sql-exam2/results/results_v2.json \ --golden evidence/text2sql-exam2/data/golden.jsonl \ --dbs evidence/text2sql-exam2/data/dbs.json ``` - **Success.** `OK: every number in results_v2.json was recomputed from raw_v2.jsonl and matches`. - **Failure.** `FAIL: raw.jsonl is not the file the results were computed from`, or `FAIL: golden.jsonl or dbs.json is not the file ...`, or `FAIL: recomputed numbers differ in [...]` with the names of the differing keys. - **What it does not prove.** That the teacher's answers are what a user wanted, or that the exam is like your data. ### 2. The signed bill of materials: `ratchet.bom verify` - **What it proves.** The operator signed these claims about this adapter at training time, and no byte changed since. With `--notice`, the `NOTICE.md` next to the adapter is the text the BOM signed. - **The command.** ```bash python -m ratchet.bom verify bom.json --public-key [--notice NOTICE.md] ``` - **Failure.** A line starting `FAIL:` with the reason, such as a signature that does not match or a hash that differs. A BOM written without a signing key verifies as unsigned: consistent, but not tied to anyone. - **What it does not prove.** That the weights were trained on the listed data. The BOM is a statement about the data. ### 3. Committed training sets: `ratchet.proof` - **What it proves.** An example's key is, or is not, a leaf of the root the BOM signs for the training or golden set. `heldout` proves every golden key is absent from the training root. - **The commands.** ```bash python -m ratchet.proof key --messages messages.json # the key a request is stored under python -m ratchet.proof verify proof.json --bom bom.json --public-key python -m ratchet.proof heldout heldout.json --bom bom.json --public-key ``` - **Failure.** One or more lines starting `FAIL:`, for example when the proof does not rebuild the signed root or the BOM itself does not verify. - **What it does not prove.** That the weights came from that set. The proof endpoints are admin only on a gateway, because they can reveal whether a guessable text was in the data. ### 4. Certified precision: `ratchet.stats check` - **What it proves.** The certificate in the BOM's eval follows from the counts it signs: an exact Clopper-Pearson lower bound, a Bonferroni frontier over a fixed grid of thresholds, per-task bounds and the paired comparison with the previous version. - **The command.** ```bash python -m ratchet.stats check bom.json --public-key python -m ratchet.stats plan --target 0.95 --delta 0.05 # how many golden rows a certificate needs ``` - **Failure.** `FAIL:` from the BOM check, or a line for each recomputed number that differs from the signed one. - **What it does not prove.** Correctness, or behaviour on traffic that does not look like the golden rows. ### 5. The time machine: `ratchet.replay` - **What it proves.** Nothing signed. It estimates what one version's logged traffic would have given at another threshold or audit rate, from the probabilities that were logged with each request. - **The command.** ```bash python -m ratchet.replay --db ratchet.db --version 3 --sweep 0.3:0.95:0.05 ``` - **Failure.** A column that is `null` where the data does not support an estimate (for example when no request could have been labelled at that setting). - **What it does not prove.** Anything about requests the ledger did not log. Its interval is approximate. ### 6. Savings receipts: `ratchet.receipt verify` - **What it proves.** The day's receipt was not edited after sealing, it follows the previous day's receipt, and with a proof file one request is in it. - **The command.** ```bash python -m ratchet.receipt verify r.json --public-key [--prev p.json] [--proof x.json] [--rows leaves.jsonl] ``` - **Failure.** Lines starting `FAIL:` such as `FAIL: chain: prev_sha256 is not the sha256 of day N's receipt (one of them was changed)`. - **What it does not prove.** That the costs written at request time were right. Compare them with the Token Factory bill. ## What a failure looks like, in one picture
```mermaid flowchart LR F(["A file is changed
after it was made"]) --> W{{"Which file?"}} W -->|"raw answers
or results"| E1["exam verify:
numbers differ"] W -->|"the BOM"| E2["bom verify: signature
or hash fails"] W -->|"training set"| E3["proof verify: root
does not rebuild"] W -->|"past receipt"| E4["receipt verify:
chain broken"] W -->|"request row"| E5["receipt proof:
leaf not in the root"] class W gate class E1,E2,E3,E4,E5 danger ```
**Reading this diagram.** Each file has its own alarm, and every alarm prints a line starting `FAIL:`. Changing a file never goes unnoticed by the check next to it, and because later documents commit to earlier ones, a change to an old receipt shows up in the next day's receipt as well. The failure text names the file and the reason, so you know where to look.
## For engineers | Check | Code | Tests | |---|---|---| | BOM and signing | [`ratchet/bom.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/bom.py) | `tests/test_provenance.py`, `tests/test_licensing.py` | | Merkle trees | [`ratchet/merkle.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/merkle.py) | `tests/test_provenance.py` | | Training-set proofs | [`ratchet/proof.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/proof.py) | `tests/test_provenance.py` | | Exact statistics and certificate | [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py), [`ratchet/certificate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/certificate.py) | `tests/test_stats.py` | | Replay | [`ratchet/replay.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/replay.py) | `tests/test_system.py`, `tests/test_core.py` | | Receipts | [`ratchet/receipts.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/receipts.py) | `tests/test_receipts.py` | Signing uses an Ed25519 key from `RATCHET_BOM_SIGNING_KEY`. Generate one with `python -m ratchet.bom keygen`, keep the seed secret and publish the public key somewhere other than the gateway. The tables that hold committed leaves and sealed receipts refuse `UPDATE` and `DELETE` by trigger. ## Common questions ### What do I need to check all of this? The documents, the operator's public key, and this repository's command-line tools. You do not need the gateway, the ledger or the training data. ### Which check should I run first? The exam's `verify`, since it is the one that touches a real model. Then the BOM, since the later checks start from it. ### If everything says OK, is the model good? It means the claims were not altered and the arithmetic follows from the inputs. Whether the inputs describe your use is a separate question. Read the "does not prove" line under each check. ### Why is there no real BOM or receipt for the SQL adapter yet? The SQL run used standalone scripts and a borrowed GPU rather than a nightly Night School run on a gateway, so the BOM, proofs and receipts have so far only been exercised by tests. ## Related - [Verify: the commands](/reference/verify/): the same checks as a copy-and-run sheet, with the output to expect - [Evidence you can check](/guides/evidence/) for the long form of each link - [A text-to-SQL adapter you can check](/guides/text-to-sql/) - [Results](/guides/results/) - [Licenses](/guides/licenses/) --- # Check it yourself Source: https://docs.ratchet.dualmindlab.tech/reference/verify/ Every claim Ratchet makes about a result can be rechecked from files in this repository. This page lists each claim, the command that checks it, what the command should print, and what it does **not** prove. This is the quick sheet: copy a command, run it, compare the output. The longer explanations are in [Proof you can run](https://docs.ratchet.dualmindlab.tech/guides/proof/) and [Evidence you can check](https://docs.ratchet.dualmindlab.tech/guides/evidence/). You need Python 3.10 or newer and a clone of the repository. Nothing here needs a GPU, a key or a network connection unless a row says so. ```bash git clone https://github.com/harshbhx/ratchet.git && cd ratchet pip install -e ".[dev]" ``` ## At a glance | Claim | One command | Needs | |---|---|---| | The exam scores are what the saved answers give | `python scripts/text2sql/exam_numbers.py --check` | nothing | | One adapter's score recomputes row by row | `python scripts/measure_adapter.py verify …` ([below](#2-one-adapters-score-recomputes-row-by-row)) | nothing | | The exam was frozen before any model answered it | `sha256sum evidence/text2sql-exam2/data/golden.jsonl` | nothing | | The adapters are the ones on Hugging Face | `sha256sum adapter_model.safetensors` ([below](#4-the-adapters-are-the-ones-on-hugging-face)) | a download | | The live run's numbers come from its records | `python -m ratchet.liverun verify evidence/live-smoke-2026-10-23` | nothing | | The live gateway calls Token Factory | `curl` the live gateway ([below](#5-the-live-gateway-calls-token-factory)) | a network | | The loop and the rollback behave as tested, on mock models (not a live result) | `python -m sim.simulate --days 10 --requests 2000 --bad-deploy-version 3` | nothing | | The tool-calling scores (round 3) recompute from the saved answers | `python scripts/toolcall_results.py --check` ([below](#tool-calling-round-3)) | nothing | | A signed BOM verifies offline | `python -m ratchet.bom verify …` ([below](#7-a-signed-bill-of-materials-per-adapter)) | nothing | | The hosted Nano's thresholds follow from real Ultra labels ([evidence/hosted-nano](https://github.com/harshbhx/ratchet/blob/main/evidence/hosted-nano/README.md)) | `python scripts/hosted_nano_calibration.py check` | nothing | | Every requirement maps to code and tests | `python scripts/reqtrace.py --check` | nothing | ## 1. The exam scores are what the saved answers give ```bash python scripts/text2sql/exam_numbers.py --check ``` Expected: `OK: both exams.json files match the saved answers`. This reruns the strict scoring over every saved answer of the untrained model, the [teacher](https://docs.ratchet.dualmindlab.tech/guides/glossary/#teacher) and both adapters on both exams, and compares the result with the numbers the website and the README chart show (`web/src/data/exams.json`). Scoring is strict: an answer is right only if running it on the exam's database returns exactly the reference rows. **It shows:** the published numbers are the saved answers' numbers, with no typed-in figure. **It does not show:** that the answers were produced the way the README says. For that, read the [exam 2 evidence](https://github.com/harshbhx/ratchet/blob/main/evidence/text2sql-exam2/README.md) and [exam 1 and round 1](https://github.com/harshbhx/ratchet/blob/main/evidence/text2sql-round1/README.md), which name each script and, where recorded, its cost. ## 2. One adapter's score recomputes row by row ```bash python scripts/measure_adapter.py verify \ --raw evidence/text2sql-exam2/answers/raw_v2.jsonl \ --results evidence/text2sql-exam2/results/results_v2.json \ --golden evidence/text2sql-exam2/data/golden.jsonl \ --dbs evidence/text2sql-exam2/data/dbs.json ``` Expected: `OK: every number in results.json was recomputed from raw.jsonl and matches`. This runs each of the 200 answers against the exam's [SQLite](https://docs.ratchet.dualmindlab.tech/guides/glossary/#sqlite) databases and recomputes the agreement, the coverage and the confidence-gate numbers. Swap `raw_v2` and `results_v2` for `raw_v1` and `results_v1` to check the round 1 adapter. ## 3. The exam was frozen before any model answered it ```bash sha256sum evidence/text2sql-exam2/data/golden.jsonl ``` Expected: `b7634971cf6b0d79e64e30e4c16d1253fad65ab9a501a06edc6c9f8e70d69058` (on Windows: `python -c "import hashlib;print(hashlib.sha256(open('evidence/text2sql-exam2/data/golden.jsonl','rb').read()).hexdigest())"`). The repository's `.gitattributes` keeps every file under `evidence/` byte for byte on checkout, so a Windows clone hashes the same as Linux. With an older clone, or a Git that ignores it, clone again with `git clone -c core.autocrlf=false https://github.com/harshbhx/ratchet.git`. That hash was written into `evidence/text2sql-exam2/data/exam_manifest.json` when the exam was generated. Exam 1's fingerprint is `6b9c1db49bbe2aecf4157ed27845fe024d1b4fcb540f93d81b6987b982b22864`, in the [round 1 manifest](https://github.com/harshbhx/ratchet/blob/main/evidence/text2sql-round1/MANIFEST.json). **It shows:** the exam file is the one that was frozen. **It does not show:** when the hash was first recorded: that rests on this repository's dated history and the saved generator output. ## 4. The adapters are the ones on Hugging Face Download an adapter's `adapter_model.safetensors` from the [model page](https://huggingface.co/harshbhx/ratchet-nemotron-nano-4b-text2sql) (the file at the top level is round 2, the one in `round1/` is round 1) and hash it: | Adapter | SHA-256 of the weights | |---|---| | Round 1 | `a883bcf6bbf4152efb5a874520b8fa1a09d9bead55c172507725dc255f18236a` | | Round 2 | `38f99d441e0527b75fdf8fd9d10acbf4473dcf70db7febe3ad73fccd631a6aa7` | The same values are in each evidence folder's `MANIFEST.json`. Round 2's first run is in [`evidence/text2sql-round2`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2). The weights themselves are 40.5 MB each and are not in git. ## 5. The live gateway calls Token Factory ```bash curl -s https://live.ratchet.dualmindlab.tech/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{"model":"ratchet","messages":[{"role":"user","content":"In one sentence, what is a LoRA adapter?"}]}' ``` The reply carries a `ratchet` block. Expect `served_by: "teacher"` and the model `nvidia/Nemotron-3-Ultra-550b-a55b`, or, for a random 40% of eligible requests, the hosted Nano 30B (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B`, route `demo_split`). Either way you get a small `cost_usd` (a fraction of a cent) and a `request_id`. It is a real call, so please keep it to a few. Ratchet's own adapter is not connected to the live gateway; every model call on it runs on Nebius Token Factory. **The run on record:** `python -m ratchet.liverun verify evidence/live-smoke-2026-10-23` recomputes the client-side numbers of a recorded 20-request run from its 21 per-request records. Expected: `recomputed 11 client-side values and 1 rounds from 21 records`. It also prints a line that starts `not recomputable here (ledger snapshots or entered by hand)`; those values are listed, not checked. Three recorded live Scout runs are in [`evidence/scout`](https://github.com/harshbhx/ratchet/tree/main/evidence/scout), and you can [replay](https://docs.ratchet.dualmindlab.tech/guides/glossary/#replay) them [in your browser](https://huggingface.co/spaces/harshbhx/ratchet-scout). ## 6. The loop and the rollback, on mock models ```bash python -m sim.simulate --days 10 --requests 2000 --bad-deploy-version 3 make demo # then open http://localhost:8000/dashboard make reference # three pinned workloads must reproduce the v0.4.0 behaviour ``` These run **mock models**. They show how the cascade, the release gate and the rollback behave, not real model quality, and every figure they print is a mock-mode run. ## 7. A signed bill of materials per adapter In the mock-mode gateway you can generate a key, sign an adapter's [bill of materials](https://docs.ratchet.dualmindlab.tech/guides/glossary/#bom) and verify it offline: ```bash python -m ratchet.bom keygen # a signing key and its public key RATCHET_BOM_SIGNING_KEY= make demo curl -s localhost:8000/ratchet/versions/2/bom | python -m ratchet.bom verify - --public-key ``` The public half of Ratchet's own signing key ([Ed25519](https://docs.ratchet.dualmindlab.tech/guides/glossary/#ed25519), key id `81b0a086250a81d0`) is in [evidence/keys](https://github.com/harshbhx/ratchet/blob/main/evidence/keys/README.md). The 28 Oct Night School [BOM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#bom) was signed with it afterwards, on 29 Oct 2026 (`evidence/nightschool-nebius/ratchet_bom.signed.json`; the content is unchanged from the run's own record). Check a document against it with `--public-key Gh68hnDDlRSjBRxmc5dTo4l+rvqkftUsH7B2Nm72gpc=` (and `--notice NOTICE.md` for the adapter's notice). A signed BOM is the operator's signed statement of the data and evaluation behind an adapter: it shows that nothing changed after signing, not that the weights were trained on that data. See Features in depth. ## 8. The test suite and the requirements matrix ```bash make test # 2014 tests, no network, no GPU (Linux, macOS or WSL) python -m pytest -q # the same suite without make python scripts/reqtrace.py --check ``` Run the suite on Linux, macOS or WSL: Windows has no `make`, and `tests/test_watch.py` needs `fcntl`, which Windows lacks. Expected for the last command: `reqtrace: OK: 228 requirements …`. Every requirement and every invariant maps to code and to tests in REQUIREMENTS.md. ## What it cost About $17 on record for the two text-to-SQL adapters. All are estimates recorded by the scripts, not invoices; check the billing pages before quoting them. | Item | USD | Where it is recorded | |---|---|---| | Teacher answers, round 1 data and exam 1 (273 calls) | 2.87 | `evidence/text2sql-round1/data/teacher_manifest.json` (`spent_usd_estimate`) | | Teacher answers, round 2 data (914 calls) | 9.18 | `evidence/text2sql-round2/data/teacher_manifest_round2.json` (`spent_usd_estimate`) | | Teacher answering exam 2 (200 rows) | 0.11 | `evidence/text2sql-exam2/results/teacher_selfcheck.json` | | GPU job, round 2 training and exam 1 answers | 4.80 | `evidence/text2sql-round2/logs/final_status.json` (`spent_usd`) | | Round 1 training | 0 | a borrowed GPU, no charge | Not costed: the GPU runs that answered exam 2, small teacher re-runs on exam 1, and any earlier probes. The tool-calling round (about $2.33 of GPU time, plus teacher spend) is separate and is not in this sum. So is the hosted Nano's [calibration](https://docs.ratchet.dualmindlab.tech/guides/glossary/#calibration) ($0.16 of Token Factory calls, `evidence/hosted-nano/collect.json`). ## What none of this proves - The exams are synthetic text-to-SQL, not your data. A strict one-reference exam also marks a correct query wrong when it returns extra columns; the teacher's own score shows that. - Agreement is with the teacher's answers, not with ground truth. - The adapters were trained on a borrowed GPU with the standalone scripts in `scripts/text2sql/`; Night School's Nebius job ran once by hand on 28 Oct 2026 ([`evidence/nightschool-nebius`](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)), and these exam results do not test it. - The hard tool-calling exam was written by one model (Claude Sonnet 5.5), so it may favour that model's style; the [BFCL](https://docs.ratchet.dualmindlab.tech/guides/glossary/#bfcl) exam is a sample of a public benchmark, which models may have seen. - On the BFCL exam the adapter's gain over the untrained model is not significant (sign test p = 0.0784), and its gain comes from the `irrelevance` category only. See [Tool-calling round 3](#tool-calling-round-3). - Scores come from one run of each model at one setting. The 95% ranges say how much a different sample of questions could move them; they do not cover a different training run. ## Tool-calling round 3 Every claim below recomputes from files in `evidence/toolcall/`. Nothing here needs a GPU, a key or a network connection. | Claim | One command | |---|---| | The scores, ranges and wording verdicts follow from the saved answers, and every saved verdict follows from its raw answer | `python scripts/toolcall_results.py --check` | | The hard exam is the frozen one | `python -c "import hashlib;print(hashlib.sha256(open('evidence/toolcall/exams/exam_hard.jsonl','rb').read()).hexdigest())"` prints `e26b6d020caedf2ce050b48641d5c2a55a96202826f51ca66ad81be9d04dada2` | | The BFCL exam is the frozen one | `python -c "import hashlib;print(hashlib.sha256(open('evidence/toolcall/exams/exam_bfcl.jsonl','rb').read()).hexdigest())"` prints `c6b2e9dd3abd6bb2a5e7b33105e0de372612be51e82e2117087e6c36faeccb37` | | The copied evidence files are unchanged | `python scripts/toolcall_results.py --check` (compares each file's SHA-256 with `evidence/toolcall/MANIFEST.json`) | | The drafts and the page data are the ones the numbers give | `python scripts/toolcall_results.py --render --web --check-render` | **It shows:** the published numbers are the saved answers' numbers, under the exam's own strict scorer. **It does not show:** that the adapter produced those answers on the borrowed GPU as described: that rests on the training and generation logs kept with the run. A 95% range on 150 or 300 questions is wide, which is why the wording says 'about the same' only when the ranges overlap and the gap is under 5 points, and 'lower' or 'higher' with a chance caveat when the ranges overlap and the gap is larger. Found something that does not match? Open an [issue](https://github.com/harshbhx/ratchet/issues/new/choose). A number that cannot be rechecked should not be on the page. --- # Scout Source: https://docs.ratchet.dualmindlab.tech/guides/scout/ Scout is a small multi-agent research app built on Ratchet: five agents answer a question from the web, the many short triage and reading calls move to the student, and the open-ended planning and writing stay pinned to the teacher. The restaurant takes a catering order: "find out what LoRA fine-tuning is and write it up". Five helpers work on it, and every dish still goes through the pass. - **The planner** decides what to shop for: three search queries. Only the head chef from the agency does this, because it needs judgement. - **The searcher** is the runner who goes to the market (a web search). Nobody cooks here, so nobody is paid by the dish. - **The triage helper** looks at each bag from the market and says "useful or not, and what kind of shop it came from". That is a dozen small, repetitive jobs per order. The apprentice chef learns these from the head chef's answers and soon does most of them. - **The reader** opens each useful bag and writes down the one fact that matters. Also small and repetitive, so the apprentice learns it too. - **The writer** plates the final answer with its sources. Only the head chef does this. The head chef's planning and plating notes are never copied into the recipe book, so the apprentice never studies them. And because the restaurant is open to anyone, it only takes so many catering orders a day, with a few held back for the food critics (the hackathon judges). ## How it really works 1. **A question arrives.** The browser opens `GET /scout/stream?q=...` (or a client calls `POST /scout/run`). The gateway checks it: not empty, at most 300 characters, and a Tavily key. Then the budget guard decides whether the run may start (`Guard.admit`). 2. **The planner writes three search queries.** One call with the task `research-plan`. The task is pinned, so Nemotron 3 Ultra answers it directly, with no student sample. Its JSON answer is `{"queries": [...]}`. 3. **The searcher runs the queries.** It is a tool, not a model. It calls the [Tavily Search API](https://docs.tavily.com) at basic depth (1 credit a search), up to `RATCHET_SCOUT_MAX_RESULTS` (4) results per query. Duplicate URLs are skipped, and at most 12 results go on. 4. **Triage labels every result.** One call per result (up to 12) with the task `source-triage`, run 6 at a time in a thread pool. The answer is `relevant`, `source_type` and a one-sentence `reason`. Only `relevant` and `source_type` are compared when Ratchet decides whether two answers agree. 5. **The reader pulls one fact from each relevant result.** One call per relevant result (up to 6) with the task `page-read`. The answer is `answers_question`, `fact_type` and a free-text `key_fact`. Only the first two are compared. 6. **The writer answers with citations.** One call with the task `research-write`, also pinned to Ultra. It writes 2 to 4 sentences from the numbered facts and cites them as `[n]`, or says the facts do not answer the question. 7. **Everything streams to the page, then the spend is counted.** Each finished call is sent as a `step` event, then a `done` event with totals. When the run ends, successfully or not, the guard adds its spend (`Guard.add_spend`).
```mermaid sequenceDiagram participant B as Browser participant G as Gateway
and cascade participant Teacher as Ultra
teacher participant Student as Nano
student participant S as Search B->>G: GET /scout/stream Note over G: Guard.admit G-->>B: start G->>Teacher: planner, pinned Teacher-->>G: 3 queries G->>S: up to 3 searches S-->>G: up to 12 results loop each result, 6 at a time G->>Student: triage, 3 samples opt not confident G->>Teacher: escalate end end loop each relevant result, up to 6 G->>Student: reader, 3 samples end G->>Teacher: writer, pinned G-->>B: step events,
then done Note over G: Guard.add_spend,
also on failure ```
**Reading this diagram.** The run is a pipeline with two fan-out stages. The budget guard is asked once at the start and told once at the end, and in between the gateway never calls a model without going through the cascade. Notice who answers: the planner and writer calls go straight to Ultra, while triage and reader calls try the student first, and the reader escalates the same way triage does. The page gets a `step` event as soon as each call finishes, which is why triage steps appear out of order.
  • Ultra, the teacher
  • Nano, the student
### The agents and what they pass on
```mermaid flowchart TB Q(["A question"]) --> PL["Planner
research-plan"] PL -->|"3 queries"| SE["Searcher
Tavily"] SE -->|"up to 12 results"| TR["Triage, up to 12 calls
source-triage"] TR -->|"the relevant ones"| RD["Reader, up to 6 calls
page-read"] RD -->|"numbered facts"| WR["Writer
research-write"] WR --> A(["An answer with citations"]) class PL,WR teacher class TR,RD student class SE outline ```
**Reading this diagram.** Orange calls are pinned to the teacher: never stored, never trained on. Blue calls are distilled: the student first, the teacher when it is unsure. The dashed box is a tool, not a model. Most calls are blue: up to 18 of up to 20 in a run. Each arrow is also a size: three queries become up to twelve results, then up to six facts, then one answer. Because only the planner and the writer are pinned, the savings a run shows are honest about the part that cannot move.
  • Pinned to the teacher
  • Distilled to the student
  • A tool
In the gateway each agent call is `Ratchet.handle` in the same process. From outside, the [Scout CLI](#for-engineers) sends the same calls to `/v1/chat/completions` with an `X-Ratchet-Task` header, like any OpenAI client. ### Pinned and distilled calls, and why Writes 3 search queries. Pinned to the teacher. Tavily Search API. Up to 18 short labelling calls, distilled to the student. 2 to 4 sentences with citations. Pinned to the teacher.
```mermaid flowchart TB A(["An agent call
with a task name"]) --> P{{"Pinned task?"}} P -->|"yes: planner, writer"| U["Ultra answers"] U --> L["Logged with its cost,
never stored"] P -->|"no: triage, reader"| S["The student samples
three times"] S --> C{{"Confident enough?"}} C -->|yes| SV["The student's
answer is served"] C -->|no| E["Ultra answers"] E --> T[("Stored as training data,
with provenance")] SV -.-> AU["A random few
audited by Ultra"] class P,C gate class U,E,AU teacher class S,SV student class T store ```
**Reading this diagram.** The first question decides everything: is the task pinned? A pinned call never touches the student and never reaches the training data. A distilled call is the normal cascade. Both lower branches end with Ultra involved: either because the student was unsure, or because an audit checks it afterwards (the dotted arrow). That is how the student's quality stays measured.
**Pinned** (`research-plan`, `research-write`): planning queries and writing an answer are open-ended. There are no label fields to compare, so the student and the teacher would almost never "agree", calibration would end up serving nothing, and every call would still pay for student samples. Training a 4B model on long free text would also dilute what it learns for the labelled tasks. So a pinned task goes straight to the teacher, with no student sample, audit or shadow. It is logged in the ledger like any other call, so the cost and the all-teacher baseline stay complete, but it is **never stored as a training example**, a rule the code and its tests enforce. Only an operator's task spec can pin a task, never a request header. **Distilled** (`source-triage`, `page-read`): one short labelling call per search result and one per relevant page. Each run makes up to 18 of them, they repeat in shape from question to question, and their label fields can be compared exactly. That is where the student takes over. Before a trained student is live (the cold start), these calls go to the teacher too, and its answers become the training data. The savings Scout shows include the pinned calls, so the percentage is honest about the part that cannot move. ### What live Scout uses | | Live Scout | |---|---| | Search | Tavily Search API (`TAVILY_API_KEY`); without a key, Scout answers 503 | | Questions | Any question up to 300 characters (`GET /scout/examples` lists examples) | | Models | Every call goes through the gateway to Token Factory: Ultra for the pinned calls and, until a trained student serves, for the rest; a trained student on the GPU endpoint or the hosted fallback once one exists | | Budget | The full budget guard below | The agents build their messages with the functions in `ratchet/scout/prompts.py`; never build Scout messages anywhere else. ### The event stream `GET /scout/stream` sends Server-Sent Events: one JSON object per `data:` line, in this order. | Event | When | Main fields | |---|---|---| | `start` | The run was admitted | `run_id`, `question`, `mode` | | `step` | An agent call finished (parallel steps arrive in any order) | `agent`, `task`, `served_by` (`student`, `teacher` or `tool`), `model`, `confidence`, `escalated`, `cost_usd`, `baseline_usd`, `latency_ms`, `detail` | | `done` | The writer finished | `answer`, `citations`, `totals` (calls, student and teacher calls, cost, baseline, savings), `search` | | `error` | Anything failed, before or during the run | `message`; it ends the stream | The stream always answers HTTP 200, because a browser's `EventSource` cannot read status codes; a refusal or a failure arrives as an `error` event. While waiting it sends a keep-alive comment every 15 seconds. `POST /scout/run` returns the same events in one JSON body and uses status codes instead: 400 for a bad question, 429 when the guard refuses, 503 without a Tavily key in live mode, 502 when a model call fails. The full contract is in [SCOUT.md](/reference/scout/#event-objects-sse-data-payloads-in-order). ### Who answered each call The page puts a badge on every step. A student step from a Night School adapter (`student-vN`) is . A teacher step is . When the GPU endpoint is off in live mode, student calls go to the untuned Nemotron 3 Nano 30B on Token Factory, and those steps say , held to the default confidence threshold and not covered by the adapter's precision certificate. Only the step's `model` tells the two student cases apart, so the page compares it with `fallback_student` from `GET /scout/status`. ### The budget guard Scout is a public path that spends money on every run, so live mode guards it, and the code never lets it run without the guard: every run, judge runs included, goes through `Guard.admit` before it starts and `Guard.add_spend` after it ends, also on failure. Live mode must never run without it. - **A daily spend cap**: `RATCHET_SCOUT_DAILY_BUDGET_USD` (code default $3.00). The live gateway, on Cloudflare Workers with Containers, sets $1.50 a day beside a $2.50 daily cap on the whole gateway (`deploy/cloudflare-live/src/index.ts`); the Nebius VM kit's `.env` sets $0.50, which would hold the 48 days from Oct 29 to Dec 15 to at most $24, plus runs in flight. Spend is the `cost_usd` of every step, audits and shadow calls included, so a run that fails halfway still counts what it spent. - **A daily run cap**: `RATCHET_SCOUT_MAX_RUNS_PER_DAY` (30), all visitors together. It also caps Tavily credits: about 3 searches a run. - **A per-IP hourly limit**: `RATCHET_SCOUT_RUNS_PER_HOUR` (6) per client address, in memory only, so no IP is stored. - **Kept across restarts**: each admitted run writes a `scout_run` row to the ledger's `spend` table, updated after every model call and closed with its outcome. At startup `Guard.restore()` rebuilds today's counters from those rows, so an update or a crash does not reset the caps while the ledger file survives. (The live gateway keeps its ledger on the container's own disk, so a container restart starts with empty counters; its low caps are set for that reason.) The rows hold no IP, question, URL, error message or code, and their `usd` stays 0 because the model calls are already counted in `requests`. - **A judges' reserve**: `RATCHET_SCOUT_JUDGE_RUNS_PER_DAY` (and an optional `RATCHET_SCOUT_JUDGE_BUDGET_USD`) holds part of each day back. With the day's total T, the reserve J, public use P and judge use U, a public run starts only while P + max(0, U − J) is below T − J (for spend, the largest finished run today is added to the left side as a margin, so public runs in flight cannot spill into the reserve), and a judge run while P + U is below T. Judges get at least their reserve and can use public room that is left. - **An access code**: a judge sends `RATCHET_SCOUT_JUDGE_CODE` once, in the body of `POST /scout/judge`, never in a URL or header (access logs record both). The gateway keeps only its SHA-256 digest and a derived key, allows 10 attempts per IP per hour, and answers with a signed `ratchet_judge` cookie (`HttpOnly`, `SameSite=Strict`, `Secure` in live mode, 7 days by default). Rotating the code ends every session. No refusal message hints that a code exists. The code goes in the submission's testing instructions only. - **Paced Tavily credits**: with `RATCHET_TAVILY_MONTHLY_CREDITS` set, every search reserves its credits first and settles them after the call. The public's allowance is spread evenly per UTC day until `RATCHET_TAVILY_PACE_UNTIL`, with a share held for judges. After Tavily answers 432 or 433 (plan or pay-as-you-go limit), searching pauses for an hour.
```mermaid flowchart TB A(["A Scout request
in live mode"]) --> J{{"Valid judge cookie?"}} J -->|yes| JP{{"P + U below T?"}} J -->|no| PB{{"P + max(0, U − J)
below T − J?"}} JP -->|yes| H{{"Under the IP limit
and the run cap?"}} PB -->|yes| H H -->|yes| CR{{"Tavily credits left?"}} CR -->|yes| OK["Admit, write a
scout_run row"] CR -->|no| NS["Run without search,
labelled unavailable"] OK --> RUN["Run the agents"] NS --> RUN RUN --> END(["add_spend in a finally
block, also on failure"]) JP -->|no| R["Refused, nothing spent"] PB -->|no| R H -->|no| R class J,JP,PB,H,CR gate class R danger class NS outline class END ok ```
**Reading this diagram.** T is the day's total, J the judges' reserve, P the public use so far and U the judge use so far. A visitor with the judge cookie is checked against the whole day, so the reserve is theirs. A public visitor is checked against what is left after the reserve, plus any reserve judges did not use. Credits are the last check, and failing it does not refuse the run: it only removes the search. A refusal starts nothing and spends nothing: it is an `error` event on the stream, or a 429 from `POST /scout/run`. Every admitted run ends in `add_spend`, even when it fails.
A worked example (made-up round numbers, not a measurement): with a daily budget T of $0.50 and a reserve J of $0.10, the public can spend up to $0.40 before public runs are refused. If judges have used nothing, a judge can still start a run until the whole $0.50 is spent. If judges have used $0.15 (more than their reserve), the extra $0.05 counts against the public side, so public runs stop at $0.35. The guard's limits: it covers Scout only, not `/v1/chat/completions` (API keys or the Caddy guard protect that, see [Security](/guides/security/)). Many IP addresses can still use up the public share early; the judges' reserve is held back from them. And the cap is checked when a run starts, so the worst case is the cap plus the runs already in flight (a few cents each). ### When search fails A search that fails (any HTTP or transport error, such as a revoked key's 401 or Tavily's 432) or that the credit meter refuses does not end the run: - The searcher step carries `detail.unavailable` with a fixed reason or the error's class and status, never its message or URL. No more queries are made. - Triage and reader run on whatever earlier queries found. With nothing found they make no calls. - The writer still answers, without (some) sources, and its step says `search_unavailable: true`. The `done` event's `search` is `{"available": false, "partial": ..., "reason": ...}`, and the page labels the run "search unavailable". - The planner, pinned to the teacher, ran before any search, so every admitted run makes a real Token Factory call in live mode, with or without search. - Failures (not credit refusals) are counted in `/scout/status` and written as a `search_error` event on the 1st, 10th, 100th and 1000th, so a broken key stays visible. ### What can go wrong | Failure | What the system does | Where you see it | |---|---|---| | The question is empty or over 300 characters | The stream sends an `error` event; `POST /scout/run` answers 400 | the `error` event or status | | The guard refuses (budget, runs, IP hour) | Nothing starts, nothing is spent | an `error` event; 429 from `POST /scout/run` | | No Tavily key in live mode | 503 before any model call | `GET /scout/status` | | Search fails or credits run out | The run goes on, labelled search unavailable; the planner's call is still paid | `detail.unavailable`, `search_error` or `tavily_exhausted` events | | A model call fails mid-run | The run ends with an `error` event; the spend so far is counted | `Guard.add_spend` runs in a `finally` block | | The gateway restarts | Today's counters are rebuilt from `scout_run` rows | `GET /scout/status` | | Someone guesses the judge code | 10 attempts per IP per hour, then refused; no message says a code exists | `judge` events only for success | ### What is measured Scout uses Tavily and the real cascade, with the teacher on Nebius Token Factory. Ratchet's own adapter is not connected to the live gateway, so student calls go to the hosted, untuned Nano 30B only through the random split, and carry its badge (its own calibration did not clear the bar, `evidence/hosted-nano`); the three recorded runs in `evidence/scout` (23 to 24 Oct, before the split) had all 57 calls answered by the teacher. As of Oct 29, 2026 no measured live Scout savings figures are published. ## For engineers **Code** (links to `main`): | File | What | |---|---| | [`ratchet/scout/agents.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/scout/agents.py) | `Scout.run(question, emit, mode, run_id)`: planner, searcher, triage, reader, writer; emits `start`, `step`, `done`. `Call`, `http_caller` (any gateway over HTTP). A `SearchUnavailable` or `httpx.HTTPError` from search ends the searching, not the run | | [`ratchet/scout/prompts.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/scout/prompts.py) | System prompts and message builders (`plan_messages`, `triage_messages`, `read_messages`, `write_messages`), `EXAMPLE_QUESTIONS`. Never build Scout messages anywhere else | | [`ratchet/scout/search.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/scout/search.py) | `TavilySearch`, `SearchUnavailable` | | [`ratchet/scout/guard.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/scout/guard.py) | `Guard`: `admit`, `progress`, `add_spend`, `restore`, the pool rule for the judges' reserve | | [`ratchet/scout/judge.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/scout/judge.py) | `JudgeAccess` (digest and derived cookie key, never the code), `AttemptLimiter` | | [`ratchet/scout/credits.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/scout/credits.py) | `CreditMeter` (monthly credits, daily pacing, judge share, `reserve` and `settle`), `MeteredSearch` | | [`ratchet/gateway.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/gateway.py) | `_scout_precheck` (calls `Guard.admit`), `_run_scout` (`Guard.add_spend` in a `finally`), `Runtime.scout`, the `/scout/*` routes | | [`ratchet/tasks.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/tasks.py) | The four built-in Scout tasks: `source-triage` and `page-read` with their label fields, `research-plan` and `research-write` with `pinned=True` | **Settings** (from `ratchet/config.py` and [CONFIGURATION.md](/reference/configuration/)): | Variable | Default | Meaning | |---|---|---| | `TAVILY_API_KEY` | empty | Tavily key; live Scout answers 503 without it | | `RATCHET_SCOUT_DAILY_BUDGET_USD` | `3.00` | Scout spend per UTC day | | `RATCHET_SCOUT_MAX_RUNS_PER_DAY` | `30` | Runs per UTC day, everyone together | | `RATCHET_SCOUT_RUNS_PER_HOUR` | `6` | Runs per client IP per rolling hour | | `RATCHET_SCOUT_MAX_RESULTS` | `4` | Tavily results per query | | `RATCHET_SCOUT_JUDGE_CODE` | empty | Judge access code, 16 to 256 characters; set it if and only if judge runs are above 0 | | `RATCHET_SCOUT_JUDGE_RUNS_PER_DAY` | `0` | Runs held back for judges | | `RATCHET_SCOUT_JUDGE_BUDGET_USD` | unset | USD held back for judges; unset means their share of the daily budget | | `RATCHET_SCOUT_JUDGE_SESSION_S` | `604800` | Judge cookie lifetime (7 days) | | `RATCHET_TAVILY_MONTHLY_CREDITS` | `0` | Credits per cycle; 0 counts without a cap | | `RATCHET_TAVILY_JUDGE_CREDITS` | unset | Credits held for judges | | `RATCHET_TAVILY_CYCLE_DAY` | `1` | Day of the month the credit cycle starts (00:00 UTC) | | `RATCHET_TAVILY_PACE_UNTIL` | unset | Date to spread the public allowance until, such as `2026-12-15` | **Endpoints** ([API.md](/reference/api/#scout-endpoints)): `GET /scout/examples`, `GET /scout/status` (budget, runs, judges' reserve, search and credits, `fallback_student`, `student_serving`), `GET /scout/stream?q=`, `POST /scout/run`, `POST /scout/judge`, `DELETE /scout/judge`. All are public; the guard protects them. **CLI**, Scout as an outside client: ```bash python -m ratchet.scout "What is LoRA fine-tuning?" --base-url http://localhost:8000/v1 # also --api-key (default $RATCHET_API_KEY or "unused") and --max-results (default 4) ``` Search runs on your machine with `TAVILY_API_KEY`. These calls are plain `/v1` traffic: they pass through API keys and spend limits, not through the Scout guard or the credit meter. The command exits 0 when the run finishes, also with search unavailable, and 1 when a model call fails. **Events** in the release log: `search_error`, `tavily_exhausted`, `judge` (a session opened, with no IP and no code). ## Common questions ### Why not let the student try the planner and writer too? Their answers are free text with nothing to compare exactly, so Ratchet could never certify the student on them, and every call would still pay for student samples. Pinning keeps those calls on the teacher and keeps their text out of the training data. ### What happens when the student is wrong on a triage call? If it is unsure, the call escalates to Ultra before anything is served. If it is confidently wrong, a random audit may catch it, and audited precision drives the release gate and rollback. Reader precision covers `answers_question` and `fact_type` only: the free-text `key_fact` the writer cites is not checked. That limit is stated in the README. ### What does a live run cost? A run makes 1 planner call, up to 12 triage calls, up to 6 reader calls and 1 writer call. At list prices that is roughly $0.02 to $0.03 per run before the student is trained, and under $0.01 after (estimate, [VM kit guide](/reference/deploy-nebius-vm/), prices as of Oct 2026, unverified). Tavily adds about 3 credits a run. ### Can someone run up my bill through Scout? Only up to the daily spend cap, plus the runs already in flight when it is reached. Runs per day and per IP are capped too, and the counters survive restarts. The guard does not cover `/v1/chat/completions`; protect that with API keys or the Caddy guard. ### Why is a judge code needed at all? The project gallery is public. Without a reserve, any visitor could use the day's runs before a judge arrived. A judge without the code still gets the public pool, and a run without search still calls Token Factory. ### Does Scout store the questions people ask? The ledger stores no question and no client IP for Scout runs: the spend rows hold only the pool, amounts and outcome. Questions do appear in the gateway's uvicorn access log, because `/scout/stream?q=` carries them in the URL, and in Caddy's access log in the Nebius VM kit, where those logs rotate. In live mode the search queries go to Tavily under Tavily's terms, so do not type personal data into the live gateway. ## Related --- # Scout: the multi-agent research tool Source: https://docs.ratchet.dualmindlab.tech/reference/scout/ Scout is a small team of five research agents (planner, searcher, triage, reader and writer) that answers a question from the web. Every agent is a plain OpenAI client pointed at Ratchet. Two of its agents make many narrow, repetitive calls. Ratchet learns those calls and moves them from [Nemotron](https://docs.ratchet.dualmindlab.tech/guides/glossary/#nemotron) 3 Ultra to the student. The planner and the writer make open-ended calls, which stay on Ultra; the searcher is a tool, not a model call. In the mock-mode gateway the student takes over most of the narrow calls. On the live gateway no trained student is served yet, so Ultra answers, and a random 40% of eligible calls go to the hosted Nano 30B (a quarter re-checked by Ultra); the three recorded live runs in `evidence/scout` show Scout working end to end, not savings. It exists to show Ratchet on real agent traffic: the kind of narrow call that, NVIDIA Research estimates, could go to small models, in the three open agent systems it studied (about 40%, 60% and 70% of their LLM queries; [arXiv 2506.02153](https://arxiv.org/abs/2506.02153), v3, Sep 22, 2026). ## Agents | Agent | Task name | Calls per question | Route | Output (JSON) | |---|---|---|---|---| | Planner | `research-plan` | 1 | **pinned** to the teacher | `{"queries": [str, str, str]}` | | Searcher | none (tool) | 1 per query | Tavily Search API (`search_depth: basic`, 1 credit) | results: title, url, content | | Triage | `source-triage` | 1 per search result (up to 12) | **distillable** | `{"relevant": bool, "source_type": "docs\|paper\|blog\|forum\|news\|other", "reason": str}` | | Reader | `page-read` | 1 per relevant result (up to 6) | **distillable** | `{"answers_question": bool, "fact_type": "definition\|number\|date\|name\|none", "key_fact": str}` | | Writer | `research-write` | 1 | **pinned** to the teacher | `{"answer": str (markdown, cites [n]), "used": [int]}` | - **Distillable** tasks compare only their label fields: `relevant` + `source_type`, and `answers_question` + `fact_type`. Free text (`reason`, `key_fact`) is ignored when deciding agreement, as with `ticket-triage`. So page-read's [precision](https://docs.ratchet.dualmindlab.tech/guides/glossary/#precision) covers its `answers_question` and `fact_type` fields only; the free-text `key_fact` the writer cites is not checked. - **Pinned** tasks (`Task.pinned = True`) always go to the teacher and are never stored as training examples. Open-ended planning and writing are where the big model earns its price. - Triage and Reader calls run in parallel (thread pool). The ledger is guarded by a lock. ## Modes - **mock** (no keys): the searcher returns mock results from `sim/research.py`, and the mock teacher and students answer. Only the example questions work (`GET /scout/examples`). The UI must show a "Mock search results" label in this mode. - **live**: [Tavily](https://docs.ratchet.dualmindlab.tech/guides/glossary/#tavily) (`TAVILY_API_KEY`) plus the real Ratchet cascade (Token Factory teacher, [vLLM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#vllm) student or hosted fallback). A budget guard protects the public live gateway. ## HTTP endpoints | Method | Path | Purpose | |---|---|---| | GET | `/`, `/scout`, `/dashboard` | The React app (`ratchet/web/dist`, built from `web/`). The hand-written pages are served at `/lite`, `/lite/scout` and `/lite/dashboard`, and when there is no build | | GET | `/scout/examples` | `{"mode": "mock"\|"live", "examples": [str, ...], "free_text": bool}`. `free_text` is false in mock mode | | GET | `/scout/stream?q=` | Server-Sent Events, one JSON object per `data:` line (below) | | POST | `/scout/run` | Body `{"question": str}`. Returns `{"events": [ ...same objects... ]}` once the run finishes | | GET | `/scout/status` | `{"mode", "budget_usd", "spent_today_usd", "runs_today", "runs_per_hour_per_ip", "max_runs_per_day", "tavily": bool, "fallback_student", "student_serving"}`. `budget_usd` and `max_runs_per_day` are `null` in mock mode; `fallback_student` and `student_serving` (v0.5) say which model answers student calls ([Who answered](#who-answered)). Since v0.5 also `public_runs_left_today`, `public_budget_left_usd`, `judge_runs_today`, `persistent`, `judge` and `search` (below) | | POST | `/scout/judge` | Body `{"code": str}`: opens a judge session (a signed `ratchet_judge` cookie). 400, 401, 404 (judge access off), 429 (10 attempts per IP per hour) | | DELETE | `/scout/judge` | Ends a judge session: clears the cookie | Request and response details for each endpoint are in [API.md](/reference/api/#scout-endpoints). ### Event objects (SSE `data:` payloads, in order) ```json {"type": "start", "run_id": "a1b2c3", "question": "What is LoRA fine-tuning?", "mode": "mock"} {"type": "step", "agent": "planner", "task": "research-plan", "served_by": "teacher", "model": "nvidia/Nemotron-3-Ultra-550b-a55b", "confidence": null, "escalated": false, "cost_usd": 0.0013, "baseline_usd": 0.0013, "latency_ms": 2400.0, "detail": {"queries": ["LoRA low-rank adaptation explained", "LoRA paper Hu 2021", "LoRA vs full fine-tuning"]}} {"type": "step", "agent": "searcher", "task": null, "served_by": "tool", "model": "tavily", "confidence": null, "escalated": false, "cost_usd": 0.0, "baseline_usd": 0.0, "latency_ms": 650.0, "detail": {"query": "LoRA paper Hu 2021", "results": [{"id": 3, "title": "...", "url": "...", "snippet": "..."}]}} {"type": "step", "agent": "triage", "task": "source-triage", "served_by": "student", "model": "student-v4", "confidence": 0.94, "escalated": false, "cost_usd": 0.00003, "baseline_usd": 0.0011, "latency_ms": 310.0, "detail": {"id": 3, "url": "...", "title": "...", "relevant": true, "source_type": "paper", "reason": "..."}} {"type": "step", "agent": "reader", "task": "page-read", "served_by": "teacher", "model": "nvidia/Nemotron-3-Ultra-550b-a55b", "confidence": 0.61, "escalated": true, "cost_usd": 0.0014, "baseline_usd": 0.0013, "latency_ms": 2710.0, "detail": {"id": 3, "url": "...", "answers_question": true, "fact_type": "date", "key_fact": "..."}} {"type": "step", "agent": "writer", "task": "research-write", "served_by": "teacher", "model": "nvidia/Nemotron-3-Ultra-550b-a55b", "confidence": null, "escalated": false, "cost_usd": 0.0021, "baseline_usd": 0.0021, "latency_ms": 3100.0, "detail": {"answer": "LoRA freezes the base weights and trains small low-rank matrices [1]. ...", "citations": [{"n": 1, "title": "...", "url": "..."}], "search_unavailable": false}} {"type": "done", "run_id": "a1b2c3", "answer": "...", "citations": [{"n": 1, "title": "...", "url": "..."}], "totals": {"calls": 17, "llm_calls": 15, "student_calls": 11, "teacher_calls": 4, "cost_usd": 0.0062, "baseline_usd": 0.0189, "savings_pct": 0.67, "wall_ms": 4200.0}, "search": {"available": true, "partial": false, "reason": null}} {"type": "error", "message": "The daily Scout budget is used up. Try again tomorrow (UTC), or run Ratchet yourself."} ``` - `served_by` is `student`, `teacher` or `tool`. `escalated` is true when a student was tried and was below its threshold. `model` is the id the call came back with: a student step from the hosted fallback carries `fallback_student` ([Who answered](#who-answered)). - `baseline_usd` is what the same call would cost on Ultra alone (Ratchet's `all_teacher_cost_usd`). - Steps for parallel agents may arrive in any order. Use `detail.id` to match a triage step and its reader step to a search result. - An `error` event ends the stream. `/scout/stream` always answers HTTP 200 and reports problems as an `error` event, because browsers' `EventSource` cannot read status codes. `/scout/run` uses HTTP codes instead: 400 (empty question, a question over 300 characters, or a non-example question in mock mode), 429 (rate limit, daily run cap or budget), 503 (live mode without `TAVILY_API_KEY`), 502 (a model call failed during the run). - Every step keeps exactly the top-level keys above; new fields go inside `detail`. - A call the live gateway's split sent to the hosted Nano at random (`RATCHET_DEMO_SPLIT`) has `detail.route = "demo_split"`, and `totals.demo_split_calls` counts them; the page labels such a call "[Random split](https://docs.ratchet.dualmindlab.tech/guides/glossary/#random-split)" (picked at random, not by the confidence gate). Absent otherwise. ### When search is unavailable (v0.5) A search that fails (any HTTP or transport error, for example a revoked key's 401 or [Tavily](https://docs.ratchet.dualmindlab.tech/guides/glossary/#tavily)'s 432 when credits run out) or that the credit meter refuses does not end the run: - The searcher step for that query has `detail = {"query": ..., "results": [], "unavailable": ""}` and no further queries are made. The reason is `public search credits for today are used up`, `search credits are used up`, or the error's class and status (`HTTPStatusError 401`, `ConnectError`), never its message or URL. - Triage and reader run on what earlier queries found. With nothing found they make no calls, and the writer gets an empty fact list (its prompt already says to say when the facts do not answer the question). The answer text is the model's own, never rewritten. - The writer step's `detail.search_unavailable` is `true`, and the `done` event's `search` is `{"available": false, "partial": , "reason": ""}` (`partial`: some results were found first). `/scout/run` answers 200: the run finished. - The planner is pinned to the teacher and runs before any search, so every admitted run makes a runtime Token Factory call, with or without search. - This is a fallback, not a hidden failure: `/scout/status.search` shows the last failure and the failures today, a broken key still needs fixing, and failures (not credit refusals) write a `search_error` event on the 1st, 10th, 100th and 1000th. ## Who answered Each step's badge names the model that answered it (`web/src/lib/who-model.ts`, `whoLabel`). A step's `served_by` says student, teacher or tool; for a student step only its `model` tells our adapter from the hosted fallback, which the live gateway sends student calls to when the GPU endpoint is off (no `RATCHET_STUDENT_BASE`). The page reads `fallback_student` and `student_serving` from `/scout/status` and compares: | Step | Badge | Tooltip | |---|---|---| | `student`, `model` is a Night School adapter (`student-vN`) | Nano + your adapter | "Nemotron 3 Nano 4B with our LoRA adapter v[N], built to run on a Nebius Serverless Endpoint" | | `student`, `model` is `fallback_student` | Nano 30B · hosted | "Untuned Nemotron 3 Nano 30B on Token Factory, held to confidence threshold [T]. Not covered by our adapter's precision certificate." [T] is `student_serving.threshold`. With daily endpoint windows (`student_serving.window`) it adds "Our adapter answers during the daily endpoint windows." | | `teacher` | Ultra · Token Factory | "Nemotron 3 Ultra on Nebius Token Factory" | | `tool` | Tavily, or Search | the tool's id | In mock mode the tooltips start with "Simulation: a mock model plays". When a student answer of the run came from the hosted fallback, or none came from the student while the hosted fallback serves, the totals read "Answered by the student" and the summary says "answered by the hosted Nano (untuned)" instead of "by your adapter". The lite page (`/lite/scout`) uses the same words. ## Budget guard (live mode) `ratchet/scout/guard.py`. It is checked once, when a run is admitted (: every run, judge runs included, goes through `Guard.admit` before it starts and `Guard.add_spend` when it ends). | Variable | Default | Meaning | |---|---|---| | `TAVILY_API_KEY` | empty | Tavily key. Without it, live Scout returns 503 | | `RATCHET_SCOUT_DAILY_BUDGET_USD` | `3.00` | Maximum Ratchet spend on Scout runs per UTC day. When reached, Scout returns 429 | | `RATCHET_SCOUT_RUNS_PER_HOUR` | `6` | Runs per client IP per rolling hour | | `RATCHET_SCOUT_MAX_RUNS_PER_DAY` | `30` | Runs per UTC day, all clients together. When reached, Scout returns 429. This caps Tavily credits: each run makes about 3 searches (1 credit each), so 30 runs use at most about 90 credits a day | | `RATCHET_SCOUT_MAX_RESULTS` | `4` | Tavily results per query | | `RATCHET_TRUST_PROXY` | `0` | `1` = take the client IP from the first `X-Forwarded-For` entry. Use it only behind a proxy that overwrites that header. The VM does not need it: uvicorn runs there with `--proxy-headers`, which already gives the gateway the real client IP from Caddy | How spend is counted: - Spend is added **per step**, from each step's `cost_usd` (which includes audit and shadow calls Ratchet made for it). A run that fails halfway still counts what its finished steps cost. - A run that is admitted counts against the hourly and daily run limits even if it fails. - The budget is checked when a run starts, not during it. A run that starts just under the cap can finish over it, so the real daily maximum is the cap plus the runs in progress at that moment (a few cents each). - In live mode the counters are kept in the ledger and restored at startup, so a restart, `update.sh` or a crash does not reset them (v0.5). This holds where the ledger is on a volume (the Nebius VM kit). On the Cloudflare Containers live gateway the ledger is on the container's disk and a restart (after 30 idle minutes) resets the counters; the low defaults in `deploy/cloudflare-live/src/index.ts` and the Token Factory balance are the limit there. Each admitted run writes a `scout_run` row to the `spend` table: `usd` stays 0 (the model calls are already in `requests.cost_usd`), and `detail` holds `{"judge", "llm_usd", "status"}`, updated after each model call and when the run ends (`done`, `error`, `search_unavailable`). A row still `admitted` (a process killed mid-run) counts its last recorded cost. Rows hold no IP, question, URL, error message or code. Counters reset at 00:00 UTC; a run counts against the day it started. - The per-IP windows stay in memory, so no IP is stored; they reset on restart. On the first restart after upgrading, runs from earlier that day have no rows and are not counted, once. Mock mode has no budget and no daily run cap, writes no spend rows, but the per-IP rate limit applies, at 10x the live rate. ### Judges' reserve `RATCHET_SCOUT_JUDGE_CODE` and `RATCHET_SCOUT_JUDGE_RUNS_PER_DAY` (with `RATCHET_SCOUT_JUDGE_BUDGET_USD`, [CONFIGURATION.md](/reference/configuration/#judge-access-and-tavily-credits)) hold back part of the day for evaluators, so visitors from the public gallery cannot use it up first. With the day's total T (runs, or USD), the judges' reserve J, public use P and judge use U today: - A public run is admitted only while P + max(0, U − J) + m < T − J. For runs m is 0. For spend, m is the largest finished run today (when a judge budget is reserved), so public runs already in flight cannot spill into the judges' share. - A judge run is admitted only while P + U < T. Judges get at least their reserve and can use public room that is left. - Runs stay at or under T. Spend stays at or under the budget plus the runs in flight when it is reached, as before. Judge runs and public runs from one address have separate hourly windows. On the Scout page, a small "I am a judge" link under the ask bar opens a password field and an Unlock button; it shows only when `GET /scout/status` reports `judge.enabled`, so it is absent in mock mode and on a gateway without a code. After a good code the page says "Judge access on" with the runs left today and offers Sign out (`DELETE /scout/judge`). A wrong code, too many tries (10 per IP per hour) and a gateway with judge access off each get a plain message. The page keeps the code in the input's memory only, sends it once in the POST body with `credentials: 'same-origin'`, clears it right after, and never stores or logs it ([invariant 18](https://github.com/harshbhx/ratchet/blob/main/AGENTS.md)'s rule, scanned by `test_dashboard_admin_key_stays_in_memory`). A judge sends the code once (`POST /scout/judge`, in the body: uvicorn and Caddy log URLs and custom headers) and gets a signed cookie (`ratchet_judge`, `HttpOnly`, `SameSite=Strict`, `Path=/scout`, `Secure` in live mode). Only the cookie makes a run a judge run. The gateway keeps the code's [SHA-256](https://docs.ratchet.dualmindlab.tech/guides/glossary/#sha-256) digest and a key derived from it, never the code; the cookie survives a restart, and rotating the code ends every session. A judge without the code is a public visitor: with search unavailable they still get a run that calls Token Factory, where a 429 would give them nothing. ### Tavily credits The free Tavily plan gives 1,000 API credits a month; a basic search costs 1 credit, an advanced one 2 ([Tavily docs](https://docs.tavily.com/documentation/api-credits), checked Oct 20, 2026). Tavily's monthly reset day is not documented (unverified), so the cycle day is a setting. With `RATCHET_TAVILY_MONTHLY_CREDITS` set, `ratchet/scout/credits.py` meters every search: - Each search reserves its credits first and is settled after the call with what it cost: `usage.credits` when Tavily reports it (the request sends `include_usage`), else the table amount; nothing for a 4xx (432 and 433 included) or a request never sent; the table amount for a 5xx or a failure after the request was sent, because whether Tavily bills those is not documented. Each call writes a `tavily` spend row (`usd` 0; `{"judge", "credits", "depth", "http", "counted"}`). - Public use is paced: today's allowance is what the public pool had left at 00:00 UTC, spread evenly over the days until `RATCHET_TAVILY_PACE_UNTIL` (while it falls in this cycle) or the cycle's end; unused credits roll forward. Judges draw on their reserve (`RATCHET_TAVILY_JUDGE_CREDITS`) first, then on today's allowance, so public and judge use together never pass the monthly credits. - After a 432 ("Key limit or Plan Limit exceeded") or 433 ("PayGo limit exceeded"), no search is sent for an hour (one `tavily_exhausted` event). - Credits never refuse a run: a public run whose allowance is used up runs without search, labelled (above). The run and USD caps bound the cost. - The meter restores the cycle's rows at startup. Reconcile it with the Tavily dashboard weekly ([RUNBOOK.md](/reference/runbook/)). Mock mode settings: | Variable | Default | Meaning | |---|---|---| | `RATCHET_MOCK_WARMUP_DAYS` | `0` | Mock days of `mixed` traffic, each followed by a night, run at startup (and on a mock-mode reset) so the mock-mode gateway opens with a trained student. `4` takes about 2 s and leaves a live student that answers about 80–90% of Scout's LLM calls (mock-mode run, mock models: 14–18 of 17–20 calls per example question, 72–86% cheaper than all-Ultra) | | `RATCHET_MOCK_WARMUP_REQUESTS` | `1500` | Requests per warm-up day | | `RATCHET_MOCK_PACE` | `0.25` | `/scout/stream` waits this share of each call's mock latency (Ultra 2.4 s → 0.6 s), so agents visibly work. `0` = no waiting | ## Mock research world (`sim/research.py`) - 8 example questions about well-established ML concepts ([LoRA](https://docs.ratchet.dualmindlab.tech/guides/glossary/#lora), knowledge distillation, model cascades, the [Wilson score](https://docs.ratchet.dualmindlab.tech/guides/glossary/#wilson-bound) interval, [vLLM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#vllm) PagedAttention, Mamba state-space models, [self-consistency](https://docs.ratchet.dualmindlab.tech/guides/glossary/#self-consistency), model collapse). The facts are from the original papers. - Each question has 3 planned queries and a pool of 72 mock results: 12 result templates (`r-*`) × 6 variants. The templates cover relevant papers, docs and blogs, off-topic news, near-miss forum posts, and SEO listicles. Each template has fixed labels for `source-triage` and `page-read`. - Result URLs use `example.org`, so nothing pretends to be a real page. - `ResearchStream` produces triage and read requests for the simulator and `POST /ratchet/demo/day?workload=research|mixed`. Users phrase each question 4 ways. Questions 7 and 8 (self-consistency, model collapse) appear only from the [drift](https://docs.ratchet.dualmindlab.tech/guides/glossary/#drift) day, as 20% of traffic. - The simulator and the agents build messages with the same functions in `ratchet/scout/prompts.py`, so the mock teacher and students can look up each request by its exact user text. ## Recorded runs The operator can save a real run as a file: the question, the run's events with their offsets and the totals. The gateway no longer serves these files (the `/scout/recorded` routes were removed); the three committed runs in `evidence/scout/` are evidence. ```bash python -m ratchet.scout record "What is LoRA fine-tuning?" --base-url https://demo.example [--judge-code C] [--out evidence/scout] [--allow-mock] ``` It reads `GET /scout/stream` and writes `/-.json` (schema `ratchet.scout-recording/v1`: question, host name, mode, events with `t_ms`, totals). It exits 1 and writes nothing when the run errors, does not finish, or the gateway is in mock mode and `--allow-mock` is not given. The judge code (or `$RATCHET_SCOUT_JUDGE_CODE`) is used for one `POST /scout/judge` and never written. The gateway never writes a recording, so no visitor's question is stored. ## Command line ```bash python -m ratchet.scout "What is LoRA fine-tuning?" --base-url http://localhost:8000/v1 ``` Runs Scout as an outside client: each agent call goes to the gateway's `/v1/chat/completions` with an `X-Ratchet-Task` header. Flags: `--base-url` (default `http://localhost:8000/v1`), `--api-key` (default `$RATCHET_API_KEY` or `unused`), `--max-results` (default 4). Search runs locally: Tavily when `TAVILY_API_KEY` is set, otherwise mock results, which only cover the example questions and only make sense against a mock-mode gateway. These calls do not pass through the budget guard or the credit meter; they are plain `/v1` traffic. The command exits 0 when the run finishes and 1 when a model call fails (the error, such as a 401 for a wrong key, goes to stderr). A failed search prints ` -> search unavailable ()` and a warning on stderr, and the run still finishes with exit code 0. --- # Where everything runs Source: https://docs.ratchet.dualmindlab.tech/reference/where-it-runs/ ![Where everything runs. Real on Nebius: Token Factory (every model call on the live site: the teacher, the hosted Nano for the random split, the judge), a Serverless Job (the round-2 retraining on 27 Oct 2026 and the Night School training job on 28 Oct 2026, by hand), a Serverless Endpoint (the student, served for measured speed tests on 27 to 28 Oct 2026 and the Night School adapter on 28 Oct 2026, deleted afterwards) and an Object Storage bucket. Real elsewhere: your laptop (development and tests, CPU), a borrowed GPU (the original adapters' training) and Cloudflare (the website, the docs and the live gateway, CPU). Ran once by hand on Nebius, 28 Oct 2026: Night School end to end (the gateway on a CPU VM, a spot L40S training job and the adapter served from the Serverless Endpoint; the golden gate rejected it, so it was never made live). Built and tested, not run live: the nightly loop on live traffic and the one-command CPU VM kit as a hosting option.](./img/where-it-runs.svg) **When the live gateway has slept.** The live gateway's container sleeps after 30 minutes idle (a Cloudflare setting), so the first visitor after a quiet spell waits about 20 seconds while it starts, and its [ledger](https://docs.ratchet.dualmindlab.tech/guides/glossary/#ledger) counters reset. Instead of an error, the main site shows a banner that the gateway is waking up and fills in by itself; the live address shows a splash page that reloads itself. This is a cost choice, not an outage. This page says, for every part of Ratchet, where it runs **today** and where it is **designed** to run on [Nebius AI Cloud](https://docs.ratchet.dualmindlab.tech/guides/glossary/#nebius-ai-cloud). Read the last column first: **Real** rows run today. **Scripted, ready to run** means the scripts exist and pass their tests against a fake `nebius` command, ready for `make nebius-plan` and `make nebius-up`. [Token Factory](https://docs.ratchet.dualmindlab.tech/guides/glossary/#token-factory) (the [teacher](https://docs.ratchet.dualmindlab.tech/guides/glossary/#teacher)'s calls), one hand-run [Serverless Job](https://docs.ratchet.dualmindlab.tech/guides/glossary/#serverless-job) (the round-2 retraining, 27 Oct 2026) and one hand-run [Serverless Endpoint](https://docs.ratchet.dualmindlab.tech/guides/glossary/#serverless-endpoint) (a speed test of the [student](https://docs.ratchet.dualmindlab.tech/guides/glossary/#student), 27 Oct 2026) and one hand-run Night School (a CPU VM, a spot [L40S](https://docs.ratchet.dualmindlab.tech/guides/glossary/#l40s) job and an endpoint, 28 Oct 2026) are the Nebius services that have run. Night School ran end to end once on Nebius, by hand (28 Oct 2026): the gateway on a Nebius CPU VM, the training job on a spot [L40S](https://docs.ratchet.dualmindlab.tech/guides/glossary/#l40s), the adapter served from a Nebius Serverless Endpoint. The [golden gate](https://docs.ratchet.dualmindlab.tech/guides/glossary/#golden-gate) rejected it (81.8% precision against the 95% target), so it was never made live ([`evidence/nightschool-nebius/`](https://github.com/harshbhx/ratchet/blob/main/evidence/nightschool-nebius/README.md)). The live site's own gateway is still on Cloudflare and is not connected to that endpoint, and the loop has not run nightly on live traffic. The one-command VM kit and the nightly schedule have not run. ## The table | Component | What it is | Runs today where | CPU or GPU | Designed to run on Nebius as | Status | |---|---|---|---|---|---| | Gateway and router | The FastAPI app: the OpenAI-compatible routes, the cascade (student first, teacher on doubt), the release gate | The live gateway on Cloudflare Workers with Containers | CPU | A small CPU VM with Docker Compose and Caddy ([`deploy/vm`](https://github.com/harshbhx/ratchet/tree/main/deploy/vm)) | Real today; the VM ran once by hand for the 28 Oct Night School, the one-command kit is Scripted, ready to run | | Ledger | One SQLite file: requests, stored teacher answers with their license, versions, signed bills of materials | On the live gateway container's disk (a restart resets the day's counters) | CPU | The VM's Docker volume, with a daily online backup | Real today; the VM ran once by hand for the 28 Oct Night School, the one-command kit is Scripted, ready to run | | Night School (training) | Builds the dataset, fine-tunes a LoRA adapter, grades it on the held-out golden set | On 27 Oct 2026 the round-2 text-to-SQL recipe (same 4,322 examples, same settings) was retrained on Nebius AI Cloud by hand, as a Serverless Job on one preemptible (spot) L40S, with checkpoints in an Object Storage bucket (evidence: [`evidence/text2sql-round2-nebius/`](https://github.com/harshbhx/ratchet/blob/main/evidence/text2sql-round2-nebius/README.md)). The retrained adapter ([on Hugging Face](https://huggingface.co/harshbhx/ratchet-nemotron-nano-4b-text2sql-nebius)) scored 40.7% (70 of 172) on exam 1 and 34.5% (69 of 200) on exam 2, against 39.0% and 33.0% for the original adapter: the result reproduces (the row-by-row differences are within chance). The original published adapters were trained by hand on a borrowed GPU. Night School then ran end to end once on Nebius, by hand, on 28 Oct 2026: a CPU VM imported 4,322 existing Ultra-labelled rows (no new teacher calls), a spot L40S job trained on 3,864 of them (2 epochs, 966 steps, 1,478 s), the adapter was hot-loaded into a Serverless Endpoint and the golden gate rejected it (81.8% precision against the 95% target on the 11 answers it would serve), so it was never made live; four job submissions, a re-run of the eval with a 120 s timeout through a one-off driver and five VM-only fixes are all stated in the evidence ([`evidence/nightschool-nebius/`](https://github.com/harshbhx/ratchet/blob/main/evidence/nightschool-nebius/README.md)). The nightly schedule has not run on live traffic | GPU | A Nebius Serverless Job on an L40S, with a bucket mounted at `/data` ([`deploy/train_job.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/train_job.sh)) | Real (the hand-run Nebius jobs and the borrowed-GPU training; one Night School run, rejected by the gate); Built and unit-tested, not run nightly on live traffic (the schedule) | | Student serving | Nemotron 3 Nano 4B with a LoRA adapter you own, served by vLLM | Ratchet's own adapter is not connected to a student endpoint on the live site: every live model call runs on Nebius Token Factory, where Ultra teaches and a random 40% of eligible requests are answered by the hosted Nano 30B, a quarter of those re-checked by Ultra. On 27 to 28 Oct 2026 the student was served by hand, in two short sessions, on a Nebius Serverless Endpoint (one spot L40S) for speed tests: cold start about 7.2 to 7.6 minutes to the first answer; a restart (stop, then start) about 7.1 minutes, with the adapters loaded again each time; 16 requests at once p50 1.6 s (SQL adapter) and 1.2 s (tool-calling adapter); about $0.67 of GPU time in all; the endpoint was deleted afterwards (evidence: [`evidence/student-endpoint-nebius/`](https://github.com/harshbhx/ratchet/blob/main/evidence/student-endpoint-nebius/README.md)) | GPU | A Nebius Serverless Endpoint on an L40S ([`deploy/student_endpoint.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/student_endpoint.sh)) | Real (the one hand-run speed test, not connected to the live gateway); Scripted, ready to run (the endpoint script and the live gateway's connection to it) | | Teacher | Nemotron 3 Ultra, an open model whose license allows training on its outputs | Nebius Token Factory, called per token from the live gateway | Hosted by Nebius; you do not size it | The same | Real | | Scout and Tavily | The research agents on the same gateway; Tavily is the web-search API they call (not a Nebius product) | The live gateway (real Token Factory and Tavily calls, under a spend guard) and recorded runs | CPU | The same gateway on the VM | Real. The story site's `/scout` page uses the live gateway (`SCOUT_ORIGIN`, DEPLOY_PAGES.md) | | Dashboard and web app | The React app the gateway serves at `/`, `/scout` and `/dashboard` | The live gateway, from its image | CPU | The VM, from the same image | Real | | Docs site and story site | Static pages built from `docs-site/` and `web/` | Cloudflare Pages | none (static files) | Nothing planned: static files stay where they are | Real | | Live gateway | `live.ratchet.dualmindlab.tech`: the gateway in live mode | Cloudflare Workers with Containers ([`deploy/cloudflare-live`](https://github.com/harshbhx/ratchet/blob/main/deploy/cloudflare-live/README.md)) | CPU | The VM runs the same image | Real | | Main site | `ratchet.dualmindlab.tech`: static pages on Cloudflare Pages whose worker sends every gateway-backed path (dashboard, judges, `/ratchet/*`, `/scout/*`) to the live gateway by default; `GATEWAY_ORIGIN` switches it (`DEPLOY_PAGES.md`) | Cloudflare Pages | none | the gateway serves the same pages | Real for gateway numbers | | Tests and CI | `pytest`, lint and the reference runs: no network, no GPU | Your laptop and GitHub Actions | CPU | Not applicable | Real | ## The router is just CPU The gateway does no model inference. It checks keys, routes a request, calls the student and teacher over HTTP, writes the [ledger](https://docs.ratchet.dualmindlab.tech/guides/glossary/#ledger) and serves the web pages, so a 2 vCPU, 8 GB machine is enough and costs about six cents an hour (estimate, below). That is why it already runs on Cloudflare's CPU-only containers and on a laptop. Only two things need a GPU: **training** (Night School, a job that runs for under an hour a night and then stops) and **serving the student** (an endpoint that bills for every hour it is on, so it should be stopped when idle). Without a student endpoint the whole system still works: the teacher answers every call except the live site's random 40% split to the hosted Nano 30B (a quarter of those re-checked by Ultra), which is how the live gateway runs today. ## One command to host it all on Nebius > **Scripted and tested offline, ready to run, apart from the hand-run steps.** Only these Nebius steps have run for real, all started by hand: on 27 Oct 2026 the round-2 text-to-SQL retraining, as a Serverless Job on one [preemptible](https://docs.ratchet.dualmindlab.tech/guides/glossary/#preemptible-gpu) L40S (evidence: [`evidence/text2sql-round2-nebius/`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)), and a speed test of the student on a Serverless Endpoint (evidence: [`evidence/student-endpoint-nebius/`](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)); and on 28 Oct 2026 one Night School on a CPU VM, a spot L40S job and an endpoint, which the golden gate rejected (evidence: [`evidence/nightschool-nebius/`](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). Everything else below, including the one-command VM kit, the nightly schedule on live traffic and the live gateway's connection to a student endpoint, has not run: the live site's own gateway is still on Cloudflare and is not connected to that endpoint. `deploy/nebius/nebius.sh` and the scripts it calls were written from the Nebius CLI reference and tested offline against a fake `nebius` command, a fake `ssh` and a stub smoke check. Nothing here is endorsed by or affiliated with Nebius. Do the [day-1 checks](/reference/deploy-nebius-vm/#what-is-confirmed-and-what-is-not) on the first real run. ```bash make nebius-plan # prints every command; changes nothing; calls no Nebius API RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1 make nebius-up YES=1 # the CPU VM only RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1 make nebius-up YES=1 ARGS="--with-student" # plus the GPU student endpoint make nebius-status # read-only: the VM, the endpoint, the URLs RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1 make nebius-down YES=1 # stop the endpoint, delete the VM ``` The same, without `make`: `bash deploy/nebius/nebius.sh plan|up|status|down [--yes] [--with-student] [--with-training] [--live]`. **A real run needs both `--yes` and `RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1`.** Without both, `up` and `down` are the same dry run as `plan`: a local stand-in replaces the `nebius` command, so no Nebius API is called and nothing is created, started, stopped or deleted. The GPU parts are opt-in: `--with-student` adds the bucket and the student endpoint, `--with-training` also submits one training job, and `--live` switches the VM's gateway to live mode. What `up` does, in order: 1. **Preflight.** The `nebius` CLI is installed and signed in, `jq` and `ssh` exist, the project has a subnet, the variables each flag needs are set (it names every missing one), and the platform fits the region (`cpu-e2` exists only in `eu-north1`). It prints the estimated hourly cost. 2. **The CPU VM** with the gateway and Caddy: [`deploy/vm/create_vm.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/create_vm.sh). 3. **With `--with-student`: a bucket** (unless `RATCHET_BUCKET_ID` is set) and **the student endpoint**: [`deploy/student_endpoint.sh start`](https://github.com/harshbhx/ratchet/blob/main/deploy/student_endpoint.sh), which creates it when no endpoint has the name. With `--with-training` it then submits [`deploy/train_job.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/train_job.sh) (the dataset must already be in the bucket at `night/train.jsonl`, and `RATCHET_TRAIN_IMAGE` must name an image built from `deploy/Dockerfile.train` and pushed where Nebius can pull it). It always prints the command that stops the GPU. 4. **Wiring.** `RATCHET_STUDENT_BASE` and `RATCHET_STUDENT_KEY` (and, with `--live`, `RATCHET_MODE=live` and the keys) are written to the VM's `.env` over `ssh` by [`deploy/vm/set_env.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/set_env.sh), with each value on stdin so it never appears in a command line, a log or the output; then [`deploy/vm/update.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/update.sh) rebuilds, checks the configuration and restarts. 5. **Smoke check** from outside with [`scripts/demo_check.py`](https://github.com/harshbhx/ratchet/blob/main/scripts/demo_check.py), then the URLs and the teardown command. `down` stops the student endpoint **first** (a stopped endpoint is not billed for compute or its own storage; its mounted bucket still is), then deletes the VM and its boot disk. It never deletes the endpoint or the bucket: it prints those commands, because deleting them also deletes your adapters. What you need before a real run: the Nebius CLI signed in, `NEBIUS_PROJECT_ID`, an SSH key, the repository public on GitHub (the VM clones it), and `RATCHET_STUDENT_KEY` for `--with-student` (`openssl rand -hex 24`). Every variable is in [CONFIGURATION.md](/reference/configuration/#deploy-scripts-deploy). ### Cost, as estimates These are estimates from Nebius list prices read on Oct 20 to 22, 2026, in US dollars before tax. Your bill decides; the prices were not confirmed by a run. | Part | Estimate | Source | |---|---|---| | CPU VM `cpu-e2`, `2vcpu-8gb` | about $0.060 an hour (about $43.80 a month) | [Compute pricing](https://docs.nebius.com/compute/resources/pricing), [DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/#cost-estimate) | | 30 GiB boot disk | about $0.003 an hour ($2.13 a month) | the same | | Student endpoint, one L40S | about $1.35 an hour while it runs, about $32 a day if left on | [DEPLOYMENT.md](/reference/deployment/#4-cost-estimates) | | One training job, one L40S | about $0.70 to $1.35 per run; the job's own 2 hour limit caps it near $2.70 | the same | | Bucket | per GiB-month, also while the endpoint is stopped | the same | | Teacher calls | about $1.30 per 1,000 calls at list prices | the same | At low traffic a 24/7 student endpoint costs more than it saves, and Ratchet's cost report says so ([Cost accounting](/reference/architecture/#7-cost-accounting)). Use `--with-student` when you want to measure the student, and stop it afterwards. More detail: [DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/) (the VM, monitoring, backups, teardown), [DEPLOYMENT.md](/reference/deployment/) (the endpoint and the job step by step) and the guide [Host it on Nebius](https://docs.ratchet.dualmindlab.tech/guides/host-on-nebius/). ## Run the tests on your own machine No keys, no GPU, no bill: ```bash pip install -e ".[dev]" make test # the offline test suite ``` Steps: [the README's quick start](https://github.com/harshbhx/ratchet/blob/main/README.md#use-it). ## Run the live gateway on a VM The live gateway (real Token Factory calls) runs on any machine that runs Docker. On a Nebius VM, the steps are in [DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/#switching-from-mock-to-live); `deploy/nebius/nebius.sh up --live` does the wiring for you. Anywhere else: `RATCHET_MODE=live` with `NEBIUS_API_KEY` and `RATCHET_API_KEYS` set (LIVE_MODE.md). The live gateway uses Cloudflare Workers with Containers instead ([`deploy/cloudflare-live`](https://github.com/harshbhx/ratchet/blob/main/deploy/cloudflare-live/README.md)), which is the host that has actually run. ## Unlock Scout runs as a judge Hackathon judges can hold back runs and spend for themselves on the live Scout page. The access code is on the Devpost page, never in this repository. The picture below is an illustration of the page, not a recording. ![Illustration, not a screen recording, of the Scout page's judge box in four steps: click "I am a judge", type the code (shown only as dots), press Unlock, and see "Judge access on" with "Sign out".](./img/anim/judge-box-howto.gif) How the access works: [Scout](/reference/scout/) and ADR 0120. --- # Host it on Nebius Source: https://docs.ratchet.dualmindlab.tech/guides/host-on-nebius/ Ratchet's Nebius deploy kit runs it on one small CPU VM that does no model inference, calls its teacher, judge and hosted fallback on Token Factory, and rents an L40S GPU only while you serve or train the student, so GPU money is spent only when you decide to spend it. Everything below is wrapped in one script. `make nebius-plan` prints every command and changes nothing; `deploy/nebius/nebius.sh up --yes` (with `RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1`) creates the CPU VM, adds the GPU student endpoint only with `--with-student`, wires the VM's `.env` and runs a smoke check, and `down` stops the endpoint first and then deletes the VM. It is scripted and tested offline, **ready to run with `make nebius-plan` and `make nebius-up`**. On 28 Oct 2026 Night School ran end to end once on Nebius, by hand, on a CPU VM with a spot L40S training job and a Serverless Endpoint; the golden gate rejected it, so it was never made live ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). Two other Nebius GPU steps have run so far, by hand: the round-2 retraining on 27 Oct 2026 (a Serverless Job on one preemptible L40S, checkpoints in Object Storage) and, the same day, one speed test of the student on a Serverless Endpoint (one spot L40S, deleted afterwards; [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). The one-command VM kit and the live site's connection to an endpoint have not run (a Nebius CPU VM did run once, by hand, for the 28 Oct Night School). The commands, the flags and the costs (estimates) are in [Where everything runs](/reference/where-it-runs/#one-command-to-host-it-all-on-nebius), which also says what runs where today. ```bash make nebius-plan RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1 make nebius-up YES=1 ARGS="--with-student" RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1 make nebius-down YES=1 ``` Think of the whole restaurant as a set of rented rooms. - **The front room** is a small CPU VM. It holds the pass (the gateway), the recipe book on a shelf (the SQLite ledger), a doorman who checks guests and puts a proper lock on the door (Caddy, with HTTPS), and an alarm clock on the wall (cron) that rings for the nightly backup. - **The head chef from the agency** works in another building (Token Factory). You phone each order in and pay per dish. The same agency sends the examiner who grades written answers (the judge) and the untrained cook who covers easy orders when the apprentice's station is closed (the hosted fallback). - **The apprentice's station** (a Serverless Endpoint with one GPU) is rented by the hour. While the lights are on the meter runs, even if no orders come in. So you switch it on for busy times and off afterwards. - **The night school classroom** (a Serverless Job) is rented only for the lesson, then handed back. - **The pantry** (an Object Storage bucket) is shared: the classroom puts each new notebook of techniques there, and the station reads it from there. The pantry's rent keeps running even while the station is closed. - Scout also phones a **librarian** (Tavily) for web searches. - **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)). - **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)). - **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). - **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). - **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate. - **The nightly schedule on live traffic.** Night School runs by hand today; the timer is scripted. - **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline. Run every script with `DRY_RUN=1` first. ## How it really works ### The pieces and how they connect | Piece | Nebius product and size | What runs there | How it bills | |---|---|---|---| | Front room | Compute VM, platform `cpu-e2`, preset `2vcpu-8gb`, region `eu-north1` | Docker Compose with Caddy (`caddy:2.11.7-alpine`), the gateway (one uvicorn worker), the SQLite ledger in the `ratchet-data` volume at `/data/ratchet.db`, and cron | While it runs. A stopped VM keeps billing for its disk | | Teacher, judge, hosted fallback | Token Factory | Nemotron 3 Ultra (`nvidia/Nemotron-3-Ultra-550b-a55b`), Nemotron 3 Super (`nvidia/nemotron-3-super-120b-a12b`), Nemotron 3 Nano 30B (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B`) | Per token | | Student | Serverless Endpoint, platform `gpu-l40s-a`, preset `1gpu-8vcpu-32gb` | vLLM `v0.18.0` serving `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16`, with LoRA adapters loaded at runtime | While it runs. A stopped endpoint is not billed for compute or its own storage | | Night School training | Serverless Job, platform `gpu-l40s-a` | `python -m ratchet.nightschool.train` (Transformers, PEFT and TRL), capped at 2 hours | While it runs | | Pantry | Object Storage bucket, mounted at `/data` in both the job and the endpoint | The training file, adapters, BOMs and `NOTICE.md`; optionally ledger backups | Per GiB-month, also while the endpoint is stopped | | Scout's web search | Tavily (not a Nebius product) | Scout's searcher, live mode only | Credits; the free plan has 1,000 a month ([Tavily credits](https://docs.tavily.com/documentation/api-credits), checked Oct 20, 2026) | The gateway does no inference. It routes requests, writes the ledger and serves the web pages, so a 2 vCPU / 8 GB VM is enough. The models run elsewhere: the teacher on Token Factory, the student on the endpoint, or the hosted fallback when the endpoint is off.
```mermaid flowchart TB user(["Judges, browsers, your app"]) --> dns["DNS"] dns --> pages["Cloudflare Pages
site and docs"] dns --> vm subgraph vm["Nebius CPU VM · 2 vCPU, 8 GB"] direction TB caddy["Caddy
HTTPS, edge guard"] ==> live["Live gateway
SQLite ledger"] end subgraph tf["Token Factory · in use"] teacher["Ultra: teacher
Super: judge
Nano 30B: fallback"] end tavily["Tavily
live Scout search"] subgraph gpu["Nebius AI Cloud · L40S · ran by hand 27 to 28 Oct"] direction TB ep["Serverless Endpoint
vLLM, only while on"] --- bucket[("Object Storage
bucket at /data")] job["Serverless Job
LoRA training"] --> bucket nightly["Nightly schedule and
live connection: not run"] end vm ==> tf tf ~~~ tavily tavily ~~~ gpu vm -.-> tavily vm -.-> gpu class user,dns,caddy,live neutral class pages,teacher ok class tavily outline class ep,job,bucket ok class nightly pending class gpu okZone class tf okZone ```
**Reading this diagram.** Green is in use today: Cloudflare Pages serves the static site and these docs (the live gateway runs on Cloudflare Workers with Containers; the Nebius VM in this guide ran once by hand, for the 28 Oct Night School; its one-command kit is tested offline), and the teacher runs on Nebius Token Factory, paid per token. The Nebius GPU side is green because each part ran once by hand: the round-2 retraining (27 Oct), the endpoint speed tests (27 to 28 Oct) and the Night School job and endpoint (28 Oct); the original adapters were trained on a borrowed GPU. The one amber box is what has not run: the nightly schedule and the live gateway's connection to an endpoint (the one-command script, `make nebius-up`, is tested offline). Plain boxes are the CPU VM this guide sets up: Caddy in front, the live gateway with its SQLite ledger. The thick arrow is the paid path every live request can take; the dotted ones are optional: Tavily only for live Scout runs, the endpoint only while you run it, and the training job only once the Night School cron is turned on.
  • In use
  • Not run
  • Optional
  • Set up by this guide
### What lives in the bucket The bucket is mounted at `/data` in the training job and in the endpoint, laid out like this: - night/ - train.jsonl - boms/ - student-vN.json - adapters/ - student-vN/ - adapter_config.json - adapter_model.safetensors - versions.json - ratchet_manifest.json - NOTICE.md - ratchet_bom.json - `python -m ratchet.live night` writes `night/train.jsonl` locally under `RATCHET_DATA_DIR` and uploads it with `RATCHET_SYNC_CMD`. That command must only upload: `aws s3 sync` with `--delete` would remove the weights the job wrote. - The training job writes the adapter weights, `versions.json` (Python, CUDA and every pinned package), `ratchet_manifest.json` and a first `NOTICE.md`. - After the golden eval, `ratchet.live` writes the adapter's signed BOM (`boms/student-vN.json` and `adapters/student-vN/ratchet_bom.json`) and the exact `NOTICE.md` text the BOM signed, then syncs again. A failed second sync writes a `sync_failed` event and exits non-zero; the night still counts. - vLLM reads `/data/adapters/student-vN` when the gateway asks it to load the adapter. ### The order of work 1. **Install the Nebius CLI and log in**, on macOS, Ubuntu, WSL or the console's Cloud Shell. 2. **Create the VM.** It starts with no keys, so it makes no model call and cannot run up a model bill until you configure it in step 6. 3. **Give it a web address and check the pages.** 4. **Run the day-1 checks** with a Token Factory key (`scripts/day1_check.py`, a few cents). 5. **Create the bucket, the training image, the student endpoint and one training job by hand**, each with `DRY_RUN=1` first. 6. **Configure live mode** with the edge guard and the Scout budget guard on. 7. **Keep it running** until judging ends: GPU only in planned windows, updates, backups, monitoring. 8. **Tear it down**: the VM, its boot disk, the endpoint and the keys. Each step in detail follows. ### 1. Install the Nebius CLI ```bash curl -sSL https://artifacts.nebius.cloud/cli/install.sh | bash exec -l $SHELL && nebius version export NEBIUS_PROJECT_ID= nebius profile create --parent-id $NEBIUS_PROJECT_ID # opens a browser login ``` The Nebius CLI runs on Ubuntu and macOS only. On Windows, use WSL (Ubuntu) or Cloud Shell in the Nebius console. The deploy scripts are bash scripts and also need `jq`, so WSL is the easier choice for the whole guide. You also need: - **`jq`** on the machine that runs `create_vm.sh` (it reads the public IP with it). - **An SSH key**, for example `~/.ssh/id_ed25519.pub` (`ssh-keygen -t ed25519`). - **The repo public** at `https://github.com/harshbhx/ratchet`, because the VM clones it on first boot. For a private repo, `RATCHET_REPO_URL` can carry a read-only token; it stays in the VM's cloud-init user data, so revoke it after first boot. - **A region.** The default platform `cpu-e2` exists only in `eu-north1`, which also has L40S GPUs. In other regions set `RATCHET_VM_PLATFORM=cpu-d3`. ### 2. Create the VM ```bash export NEBIUS_PROJECT_ID= export SSH_PUBLIC_KEY_PATH=~/.ssh/id_ed25519.pub DRY_RUN=1 bash deploy/vm/create_vm.sh # prints the commands, creates nothing bash deploy/vm/create_vm.sh ``` What [`create_vm.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/create_vm.sh) does: 1. Refuses to go on if a VM with the name already exists (`ratchet-host` by default). 2. Finds the first subnet in the project (`nebius vpc subnet list`), unless you set `NEBIUS_SUBNET_ID`. 3. Creates a 30 GiB `network_ssd` boot disk from the image family `ubuntu24.04-driverless`. The disk is created on its own, so it survives deleting the VM; you delete it yourself at teardown. 4. Creates the VM with that disk, a **static** public IP (kept while the VM is stopped) and [`cloud-init.yaml`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/cloud-init.yaml) as user data. 5. Prints the public IP, the VM and disk IDs, the next steps and the two teardown commands. On first boot, cloud-init installs Docker Engine and the Compose plugin, turns on unattended security upgrades and the `ufw` firewall (ports 22, 80 and 443), creates a `ratchet` service user that cannot log in, clones the repo to `/opt/ratchet`, writes `deploy/vm/.env` from `.env.vm.example` (mode 0600), and runs `docker compose up -d --build`. That takes 5 to 10 minutes: ```bash ssh ubuntu@ 'sudo tail -f /var/log/ratchet-bootstrap.log' ``` Optional settings for `create_vm.sh`: `RATCHET_VM_NAME`, `RATCHET_VM_PLATFORM`, `RATCHET_VM_PRESET` (for example `4vcpu-16gb`), `RATCHET_VM_DISK_GB`, `RATCHET_SSH_USER` (default `ubuntu`; Nebius reserves `root` and `admin`), `RATCHET_REPO_URL` and `RATCHET_DOMAIN`. ### 3. Give it a web address and check the pages - **Fastest: sslip.io.** If `RATCHET_DOMAIN` is not set, first boot sets it to `.sslip.io`. That name resolves to the IP inside it, so Caddy gets a real certificate without any DNS work. The VM finds its public IP by asking `api.ipify.org` (verify on day 1). - **Your own name.** Add a DNS A record pointing at the public IP, wait until `dig +short ` returns it, then on the VM: ```bash cd /opt/ratchet/deploy/vm sudo -u ratchet sed -i 's|^RATCHET_DOMAIN=.*|RATCHET_DOMAIN=ratchet.dualmindlab.tech|' .env sudo -u ratchet docker compose up -d ``` Set the DNS record first: repeated failed certificate requests can hit Let's Encrypt's rate limits. | URL | Expect | |---|---| | `https:///` | The story page | | `https:///scout` | Scout (needs the keys from step 6) | | `https:///dashboard` | The dashboard | | `https:///healthz` | `{"status":"ok",...}`, always public | | `https:///readyz` | 200 `ready` (or `degraded`, with the reason under `checks`); 503 `not_ready` when the ledger cannot be written or no teacher can answer | | `https:///v1/models` | The model list; 403 when the Caddy guard is on, 401 when API keys are on and you send none | ### 4. Run the day-1 checks Live mode spends real money on every Scout run and every chat request. Before it, check the model IDs and features with a real Token Factory key: ```bash export NEBIUS_API_KEY=... # a Token Factory key from its console python scripts/day1_check.py # or: make day1 ``` The script confirms that the four model IDs exist and that the teacher answers, returns logprobs and supports JSON mode. It also prints each model's `finish_reason` and whether the teacher takes the strict `json_schema` format. If an ID differs, set `RATCHET_TEACHER_MODEL` (and update `ratchet/licenses.py` and `DEFAULT_PRICES`). Model IDs and per-token prices in `ratchet/config.py` were checked this way on 29 Oct 2026 (it passed); the GPU-hour rate is still unverified. Every check, and how to run it, is described at the top of [`scripts/day1_check.py`](https://github.com/harshbhx/ratchet/blob/main/scripts/day1_check.py) (`teacher`, `choices`, `reasoning`, `student`, `tavily`, `cli` or `all`). The Nebius Builder Program's terms say $25 of Token Factory credit, expiring after 90 days (research notes, Oct 25, 2026; [DEPLOYMENT.md](/reference/deployment/)). Re-check the program terms before relying on it. ### 5. Bucket, training image, student endpoint and training job Do these from your laptop (or WSL), where the `nebius` and `aws` CLIs are. Stop the endpoint after each check. 1. **Bucket.** Create one in the same region as the GPUs (`eu-north1`), then: ```bash export RATCHET_BUCKET_ID= # for --volume export RATCHET_BUCKET_S3=s3:// # for polling in train_job.sh export RATCHET_S3_ENDPOINT=https://storage.eu-north1.nebius.cloud # verify on day 1: copy it from the console export RATCHET_DATA_DIR=./data export RATCHET_SYNC_CMD="aws s3 sync ./data s3:// --endpoint-url $RATCHET_S3_ENDPOINT" ``` 2. **Training image.** Build from the repository root and push it to a registry: ```bash docker build -f deploy/Dockerfile.train -t /ratchet-train:0.1 . docker push /ratchet-train:0.1 export RATCHET_TRAIN_IMAGE=/ratchet-train:0.1 ``` The image is CUDA 12.8 with exact versions of torch, Transformers, PEFT, TRL, Accelerate, datasets and the Mamba kernels, pinned in `deploy/train-versions.txt`. The versions are candidates read on Oct 21, 2026, to be confirmed by the first smoke job's `versions.json`. 3. **Student endpoint.** Create it once, then stop and start the same endpoint (same URL, same token): ```bash export RATCHET_STUDENT_KEY=$(openssl rand -hex 24) DRY_RUN=1 bash deploy/student_endpoint.sh create # nebius validates the request, creates nothing bash deploy/student_endpoint.sh create # once export RATCHET_STUDENT_BASE=https:///v1 bash deploy/student_endpoint.sh status # prints " ", for example RUNNING bash deploy/student_endpoint.sh stop # when idle bash deploy/student_endpoint.sh start # the same endpoint again ``` 4. **One training job by hand.** `deploy/train_job.sh ` submits `nebius ai job create` with the bucket at `/data`, a 200 GiB disk and a 2-hour timeout. With `RATCHET_BUCKET_S3` set it then waits, polling every 30 seconds for up to 2 hours, until `adapters//adapter_model.safetensors` exists, which is what lets Night School use it as `RATCHET_TRAIN_CMD`: ```bash DRY_RUN=1 bash deploy/train_job.sh test # nebius validates the request, submits nothing bash deploy/train_job.sh test # the day-1 smoke job, on about 200 rows ``` 5. **Smoke test the whole path.** With the day-1 checks passed and the endpoint up, `make live-smoke` sends 20 requests through a live gateway with about 5 control calls to Ultra and no nights (cents). What `student_endpoint.sh create` sets: the image `vllm/vllm-openai:v0.18.0-cu130` (`RATCHET_VLLM_IMAGE`), container port 8000, `--shm-size 16Gi`, the bucket at `/data`, token auth with `RATCHET_STUDENT_KEY`, `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`, and vLLM arguments from the model card: `--enable-lora --max-lora-rank 16 --max-loras 4 --mamba_ssm_cache_dtype float32 --max-model-len 8192 --max-num-seqs 64`, with thinking off by default. Training defaults (`train.py`): 2 epochs, learning rate 2e-4, LoRA rank 16 (alpha 32), max sequence 2048, batch 8, bf16, loss on the assistant's tokens only. Adapters loaded at runtime are lost when the endpoint stops. The gateway notices vLLM's 404 for an adapter and loads it again by itself, at most once per adapter per minute (`adapter_reloaded` events), and `/readyz` queues a reload for any routed adapter the endpoint no longer lists. Until then those student calls fall back to the teacher: they still succeed, at teacher cost. `python -m ratchet.live reload` reloads every live, canary and shadow adapter by hand. ### 6. Configure live mode The VM's settings live in `/opt/ratchet/deploy/vm/.env` (mode 0600, owned by `ratchet`, never in git). Keep comments on their own lines: Docker Compose and the scripts read this file. ```bash RATCHET_MODE=live NEBIUS_API_KEY= TAVILY_API_KEY= # Empty: the hosted fallback answers student calls. Set both only while the endpoint runs. RATCHET_STUDENT_BASE= RATCHET_STUDENT_KEY= # Caddy blocks /v1/*, /ratchet/admin/* without this token (hex, 32+ characters). RATCHET_API_GUARD=guarded RATCHET_ADMIN_TOKEN= # Only because Caddy guards /v1/*. Or set RATCHET_API_KEYS instead (or as well). RATCHET_ALLOW_UNAUTHENTICATED=1 # The Scout budget guard: keep it on in live mode. RATCHET_SCOUT_DAILY_BUDGET_USD=0.50 RATCHET_SCOUT_RUNS_PER_HOUR=6 RATCHET_SCOUT_MAX_RUNS_PER_DAY=30 RATCHET_AUDIT_RATE=0.05 RATCHET_PUBLIC_DASHBOARD=1 ``` Then apply it: 1. Edit: `sudo -u ratchet nano /opt/ratchet/deploy/vm/.env`. 2. Check every setting at once, in the image, without starting anything: ```bash cd /opt/ratchet/deploy/vm sudo -u ratchet docker compose run --rm --no-deps -T ratchet python -m ratchet.config check ``` Exit 0 is clean, 1 is warnings, 2 means the gateway would refuse to start. 3. Apply: `sudo -u ratchet docker compose up -d` (or `update.sh`, which runs the same check first). 4. Confirm: `curl -s https:///readyz` says `ready` (or `degraded`, with the reason). The ledger now lives at `/data/ratchet.db` in the `ratchet-data` volume and survives restarts and rebuilds. 5. Watch the Token Factory usage page and the Tavily dashboard daily for the first week, and check that `curl -s https:///scout/status | jq .search` shows `"available": true`. What each guard is for: - **`/v1/*` is a paid endpoint.** Without a guard anyone could send unlimited requests on your Token Factory credit. Live mode refuses to start without `RATCHET_API_KEYS`, unless `RATCHET_ALLOW_UNAUTHENTICATED=1` says that something in front checks callers. On the VM that something is Caddy's edge guard: `RATCHET_API_GUARD=guarded` with a hex `RATCHET_ADMIN_TOKEN` of at least 32 characters. An empty or shorter token blocks the guarded paths for everyone, and Caddy strips the header before it reaches the gateway. To use the API yourself: `curl -H "X-Ratchet-Admin-Token: $TOKEN" https:///v1/models`, or `default_headers` in the OpenAI SDK. - **The Scout budget guard stays on in live mode**. Every Scout run is admitted against a daily dollar cap, a daily run cap and a per-IP hourly limit, and its spend is added after it ends, also on failure. The counters are kept in the ledger, so a restart, a crash or `update.sh` does not reset them. The code's default cap is $3.00 a day; the VM example sets $0.50, so the worst case over the 48 days from Oct 29 to Dec 15 is about $24 (estimate, [DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/)). The budget guard does not cover `/v1/chat/completions`; the edge guard or API keys do. - **A reserve for judges.** `RATCHET_SCOUT_JUDGE_CODE` (generate it with `openssl rand -hex 16`) and `RATCHET_SCOUT_JUDGE_RUNS_PER_DAY` hold part of each day's runs, spend and Tavily credits back from the public. `RATCHET_TAVILY_MONTHLY_CREDITS` and `RATCHET_TAVILY_PACE_UNTIL=2026-12-15` spread the credits over the judging period. The code goes in the submission's testing instructions only, never on a public page, in the video or in a URL. - **The dashboard.** With `RATCHET_PUBLIC_DASHBOARD=1` judges see the live dashboard, and anonymous readers get client pseudonyms and events without URLs or error messages. - **Signed BOMs (optional).** Run `python -m ratchet.bom keygen` on your laptop, put the `RATCHET_BOM_SIGNING_KEY=` line in `.env`, and publish the public key it prints. Switching to live mode starts a new ledger file, so there is no trained version yet: every request is a cold start and the teacher answers it, at teacher prices. The hosted fallback stands in only for a version that exists. A student appears on the VM only after Night School trains and releases one on this ledger, and on the stock VM kit Night School does not run (step 7 below): the 28 Oct run needed the `nebius` and `aws` CLIs and the student endpoint already running on the VM, and the night stops with a message when `aws` is missing. ### 7. Keep it running until judging ends #### GPU windows The student endpoint is the largest cost: about $32 for every full day it runs (an estimate, below). So: - **Most of the time, keep the endpoint stopped** and leave `RATCHET_STUDENT_BASE` and `RATCHET_STUDENT_KEY` empty. Student calls for a trained version then go to the hosted fallback, Nemotron 3 Nano 30B on Token Factory, paid per token. Say so plainly: it is untuned, it is held to the default threshold (`RATCHET_CONFIDENCE_THRESHOLD`, 0.80) instead of a calibrated one, and no certificate covers it. The pages say so: student answers carry the badge, the dashboard header reads "Nano 30B · hosted on Token Factory (endpoint off)", and the live version's golden precision is hidden. `curl -s https:///ratchet/stats | jq .student_serving` shows what the pages read. - **Start the endpoint for fixed windows** (`bash deploy/student_endpoint.sh start`), such as recording the video or a judging window you announced, and stop it right after (`bash deploy/student_endpoint.sh stop`). A stopped endpoint is not billed for compute or its own storage; the mounted bucket still is. - **Never stop the endpoint while `RATCHET_STUDENT_BASE` still points at it.** Student calls would fail, the student's circuit breaker would open after 5 failures in a row, and the teacher would answer those requests at Ultra prices instead of the cheaper hosted fallback. `/readyz` reports `degraded` meanwhile. - **Record the uptime for the cost report.** `student_endpoint.sh` runs `python -m ratchet.cost endpoint up|down` after `create`, `start` and `stop`, but that needs the ledger file; on your laptop it prints a note instead. Record it where the ledger is, on the VM, from `/opt/ratchet/deploy/vm`: `sudo -u ratchet docker compose exec -T ratchet python -m ratchet.cost endpoint up --usd-per-hour ` (and `down` after), or `POST /ratchet/cost/endpoint` with an admin key. `GET /ratchet/cost` then reports savings by GPU-hours actually paid.
```mermaid sequenceDiagram autonumber actor You participant CLI as Nebius CLI participant Gateway as Gateway
on the VM participant Student as Student
endpoint You->>CLI: student_endpoint.sh start CLI->>Student: start the same endpoint You->>Gateway: set STUDENT_BASE and KEY, compose up You->>Gateway: record endpoint up Gateway->>Student: student calls, reload on a 404 Note over Gateway,Student: the window: your adapter answers You->>Gateway: clear both settings, compose up You->>CLI: student_endpoint.sh stop You->>Gateway: record endpoint down ```
**Reading this diagram.** A GPU window is eight manual steps, in this order. Start the endpoint you created once (1, 2), so its URL and token stay the same. Point the gateway at it with `RATCHET_STUDENT_BASE` and `RATCHET_STUDENT_KEY` and recreate the container (3), then record the uptime for the cost report (4). During the window the student calls go to your adapter, and an adapter vLLM lost on a restart is loaded again on its first 404 (5). To close the window, clear both settings before you stop the endpoint (6, 7), so student calls fall back to the hosted Nano 30B and not to the teacher at Ultra prices, and record the downtime (8).
Automatic daily endpoint windows (`student_serving.window`) are referenced in the web app's copy but not built yet: they are planned. Until then a window is the manual procedure above. #### The Night School cron (off by default) `/etc/cron.d/ratchet` on the VM runs the backup every day at 03:10 UTC. It also holds three commented-out lines: `python -m ratchet.live night` at 02:30 UTC, `... tick` every 5 minutes and `... seal` hourly, each through `docker compose exec -T ratchet`. Leave them off unless all of these hold: - live mode works, and the student endpoint and a training job have been run by hand (step 5); - the gateway image contains the `nebius` and `aws` CLIs, their credentials and the `deploy/` scripts, because `night` runs `RATCHET_TRAIN_CMD` (`deploy/train_job.sh`) inside the container. **The stock [`Dockerfile`](https://github.com/harshbhx/ratchet/blob/main/Dockerfile) does not include them yet**; - the endpoint is running during the night: without `RATCHET_STUDENT_BASE`, `night` stops before it syncs or starts a training job. To turn them on, remove the leading `# ` from those lines with `sudo nano /etc/cron.d/ratchet`. Logs go to `/var/log/ratchet/`. The ledger must stay on the VM's local disk: it runs in SQLite's WAL journal, so every process that opens it has to be on that one host, and never on the bucket mount. #### Updates ```bash sudo -u ratchet /opt/ratchet/deploy/vm/update.sh ``` [`update.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/update.sh) pulls (`git pull --ff-only`), builds the new image, and runs `python -m ratchet.config check` in it with your `.env`. On errors (exit 2) it stops and leaves the running gateway as it is, so a typo in `.env` never takes the gateway down. Then it runs `docker compose up -d`, waits for the Docker health check (`/healthz`) and for `/readyz` to answer 200 inside the container, fetches `/`, `/healthz`, `/readyz`, `/dashboard` and `/scout` over HTTPS (and `/ratchet/stats` when the dashboard is public), and exits non-zero if any of them fail. #### Backups [`backup.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/backup.sh) runs daily at 03:10 UTC. In live mode it copies the ledger with SQLite's online backup API inside the container, checks the copy with `PRAGMA integrity_check`, and keeps the newest 14 as `/var/backups/ratchet/ratchet-.db.gz`. To also copy each backup to Object Storage, install the `aws` CLI on the VM, configure it as the `ratchet` user, and set `RATCHET_BUCKET_S3` and `RATCHET_S3_ENDPOINT` in `.env`. To restore, stop the gateway, remove the old `ratchet.db-wal` and `ratchet.db-shm` files (a stale WAL file would be replayed onto the restored copy), copy the backup into the volume and start it again; the commands are in [DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/). Never copy the live file with `cp`. #### Monitoring ```bash cd /opt/ratchet/deploy/vm sudo -u ratchet docker compose ps # both running, ratchet healthy sudo -u ratchet docker compose logs -f --tail=100 ratchet sudo -u ratchet docker compose logs -f --tail=100 caddy # certificate problems show here curl -s https:///scout/status # live: spend today vs budget, judges' reserve, search, credits ``` - **Restarts use `/healthz`.** The ratchet container's Docker health check calls it every 30 seconds, and both containers restart automatically, also after a VM reboot. `/healthz` only says the process answers. - **Uptime monitors and load balancers use `/readyz`.** It is 200 while requests can be answered and 503 when they cannot. Never restart on `/readyz`: a teacher outage would become a restart loop. - Container logs rotate at 10 MB × 5 files each. Live mode logs one JSON line per chat request, with no prompt or answer text. - One gateway process per ledger: the compose file pins `--workers 1`. ### 8. Tear it down On Dec 16, or whenever judging ends. Deleting the VM does **not** delete its boot disk, because the disk was created on its own; delete both. ```bash # Optional: keep a last copy of the ledger scp ubuntu@:/var/backups/ratchet/*.db.gz ./ratchet-backups/ VM_ID=$(nebius compute instance get-by-name --parent-id $NEBIUS_PROJECT_ID --name ratchet-host --format json | jq -r .metadata.id) nebius compute instance delete --id $VM_ID nebius compute disk list --parent-id $NEBIUS_PROJECT_ID # find ratchet-host-boot (verify the flags on day 1) nebius compute disk delete --id ``` Then remove the DNS record and stop the student endpoint if it still runs. The bucket keeps billing for what it stores until you remove it, so export the adapters you want to keep first (`python -m ratchet.live export --version N --out DIR`). Rotate `NEBIUS_API_KEY`, `TAVILY_API_KEY`, `RATCHET_STUDENT_KEY`, `RATCHET_ADMIN_TOKEN`, `RATCHET_API_KEYS`, `RATCHET_ADMIN_KEYS` and `RATCHET_DASHBOARD_KEYS`, and revoke any GitHub token used for cloning. Keep `RATCHET_BOM_SIGNING_KEY` (or at least its public key) if old BOMs should stay verifiable. ### What it costs (estimates) Every figure here is an **estimate** from the repo docs or the Nebius pricing page, with its date. Check the pricing pages again before you rely on them. **The VM**, from [DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/#cost-estimate), prices as of Oct 20, 2026, USD before tax, for Oct 29 to Dec 15, 2026 (48 days, 1,152 hours): | Item | Rate | Oct 29 to Dec 15 | |---|---|---| | VM `cpu-e2` `2vcpu-8gb` (eu-north1) | $0.012 per vCPU-hour + $0.0045 per GiB-hour = $0.060 an hour | $69.12 | | Boot disk, 30 GiB network SSD | $0.071 per GiB per 730 hours | $3.36 | | Public IP, network egress | Free | $0 | | Backups in Object Storage (Standard) | $0.0147 per GiB-month | under $0.01 | | **Total** | | **about $72.50** (about $79.40 on `cpu-d3`) | **Usage in live mode**, on top of the VM ([DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/#cost-estimate)): | Item | How it is bounded | Estimate | |---|---|---| | Token Factory, Scout | `RATCHET_SCOUT_DAILY_BUDGET_USD` per UTC day | About $0.10 to $0.90 a day for 10 to 30 runs; worst case the cap × 48 days: $24 at $0.50, $144 at $3.00 | | Token Factory, `/v1/chat/completions` | Not covered by the Scout budget: the edge guard or API keys with a daily quota | $0 with the guard on and no shared keys | | Tavily | 3 basic searches a run, 1 credit each | 1,000 free credits a month, about 330 runs | **The GPU.** Nebius's [compute pricing page](https://docs.nebius.com/compute/resources/pricing) listed the L40S at about $1.35 per GPU-hour on demand when read on Oct 22, 2026. Every GPU figure below is an estimate at that price ([deployment guide](/reference/deployment/#4-cost-estimates)): | Item | At about $1.35 per GPU-hour | |---|---| | Student endpoint left on for a full day | about $32 | | One training run, 30 to 60 minutes | about $0.70 to $1.35 | The code still defaults `RATCHET_STUDENT_GPU_USD_PER_HOUR` and `RATCHET_TRAIN_GPU_USD_PER_HOUR` to 1.55. Set them (or pass `--usd-per-hour`) to the rate on your bill so the cost report prices the hours right, and set `RATCHET_PRICES_VERIFIED=1` only once the bill confirms them. A rough estimate for a short evaluation period is about **$35 to $75**: 6 to 8 training runs, the endpoint on a few hours a day, and $20 to $30 of teacher calls (estimate, [deployment guide](/reference/deployment/#4-cost-estimates)). Whether a 24/7 endpoint pays at all depends on volume; see [cost and savings](/guides/cost-and-savings/). ## For engineers **Files** | File | What it does | |---|---| | [`deploy/vm/create_vm.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/create_vm.sh) | Boot disk, VM with a static public IP, cloud-init user data. `DRY_RUN=1` prints the `nebius` commands (with the user data shown as a placeholder) and creates nothing | | [`deploy/vm/cloud-init.yaml`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/cloud-init.yaml) | First boot: Docker, `ufw`, unattended upgrades, the `ratchet` user, clone, `.env`, the sslip.io default, `docker compose up -d --build`, `/etc/cron.d/ratchet` | | [`deploy/vm/docker-compose.yml`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/docker-compose.yml) | The gateway (1 worker, `--proxy-headers`, health check on `/healthz`, port 8000 exposed only to Caddy) and Caddy (ports 80 and 443) | | [`deploy/vm/Caddyfile`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/Caddyfile) | HTTPS for `RATCHET_DOMAIN`, security headers, no buffering for `/scout/stream`, the opt-in edge guard, the admin token deleted from access logs | | [`deploy/vm/update.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/update.sh), [`deploy/vm/backup.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/backup.sh) | Pull, build, config check, restart, readiness wait; SQLite online backup with an integrity check | | [`deploy/vm/.env.vm.example`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/.env.vm.example) | Every VM setting with comments | | [`deploy/student_endpoint.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/student_endpoint.sh) | `create`, `start` (the default), `stop`, `status` for the vLLM endpoint; finds the id by name in `nebius ai endpoint list --format json` and never guesses | | [`deploy/train_job.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/train_job.sh) | `nebius ai job create` for one training run, then polls the bucket for the adapter | | [`deploy/Dockerfile.train`](https://github.com/harshbhx/ratchet/blob/main/deploy/Dockerfile.train), [`deploy/train-versions.txt`](https://github.com/harshbhx/ratchet/blob/main/deploy/train-versions.txt) | The training image and its pinned versions; change both together | | [`ratchet/live.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/live.py) | `python -m ratchet.live night`, `tick`, `reload`, `purge`, `export`, `seal` | | [`scripts/day1_check.py`](https://github.com/harshbhx/ratchet/blob/main/scripts/day1_check.py) | The Token Factory checks before live mode | **Settings that matter on Nebius** (full list in [CONFIGURATION.md](/reference/configuration/#deploy-scripts-deploy)) | Variable | Default | Meaning | |---|---|---| | `RATCHET_MODE` | `mock` | `live` (what this guide sets) calls Token Factory, the endpoint and Tavily | | `NEBIUS_API_KEY` | empty | Token Factory key: teacher, judge and hosted fallback | | `RATCHET_STUDENT_BASE`, `RATCHET_STUDENT_KEY` | empty | The endpoint's `/v1` URL and token; empty means the hosted fallback | | `RATCHET_FALLBACK_STUDENT` | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` | The hosted fallback model | | `RATCHET_API_GUARD`, `RATCHET_ADMIN_TOKEN` | `open`, empty | Caddy's edge guard and its hex token (32+ characters) | | `RATCHET_SCOUT_DAILY_BUDGET_USD` | `3.00` (the VM example sets `0.50`) | Scout's daily spend cap | | `RATCHET_AUDIT_RATE` | `0.05` | Share of student answers the teacher re-checks; each audit is a paid Ultra call | | `RATCHET_BUCKET_ID`, `RATCHET_BUCKET_S3`, `RATCHET_S3_ENDPOINT` | empty | The bucket for `--volume`, for polling and backups, and its S3 URL | | `RATCHET_SYNC_CMD`, `RATCHET_TRAIN_CMD`, `RATCHET_DATA_DIR` | empty, `bash deploy/train_job.sh {version}`, `data` | How a night uploads data and runs (and waits for) training | | `RATCHET_ENDPOINT_NAME`, `RATCHET_ENDPOINT_ID` | `ratchet-student`, empty | How `student_endpoint.sh` finds the endpoint | | `RATCHET_STUDENT_GPU_USD_PER_HOUR`, `RATCHET_TRAIN_GPU_USD_PER_HOUR` | `1.55` | GPU rates the cost report uses. The code's default; these docs use 1.35, Nebius's on-demand price for an L40S. Set the rate on your bill | | `DRY_RUN` | `0` | `1` validates without creating; any value but 0 or 1 is refused, so a typo never starts a paid GPU | **Endpoints**: `GET /healthz` and `/livez` (liveness), `GET /readyz` (readiness, [API reference](/reference/api/#get-readyz)), `GET /ratchet/stats` (`student_serving`, `cost`), `GET /ratchet/cost?days=N`, `POST /ratchet/cost/endpoint` (admin key), `GET /scout/status`. **Events to watch**: `adapter_reloaded`, `circuit` (breakers), `endpoint` (uptime recorded), `sync_failed`, `tavily_exhausted`, `search_error`, `budget`. **Tests**: `tests/test_live.py` runs `bash -n` on every script, `DRY_RUN=1 bash deploy/vm/create_vm.sh`, and both GPU scripts against a fake `nebius` on `PATH`, asserting the exact commands; `tests/test_cost.py` checks that `start` and `stop` record uptime. They check the commands against the CLI reference as read. ## Common questions ### What runs on Nebius today? Token Factory: the teacher runs there, and the live gateway calls it for real. The GPU endpoint and the training job are scripted and tested offline, ready to run with `make nebius-plan` and `make nebius-up`, and both ran by hand: the endpoint for speed tests on 27 and 28 Oct 2026 ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)), and the training job for the 27 Oct 2026 retrain ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)) and the 28 Oct 2026 Night School run (rejected by the golden gate; [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). The nightly schedule has not run; as of 29 Oct 2026 the published adapters were trained on a borrowed GPU. The scripts follow the Nebius docs and CLI reference (read Oct 20 and 21, 2026) and are tested offline, and the lines that could not be checked are marked "verify on day 1" in the scripts. Run each with `DRY_RUN=1` first and compare with `nebius ... --help`. ### Do I need a GPU to host the gateway? No. The gateway does no inference, so it runs on a 2 vCPU / 8 GB CPU VM. The teacher and the hosted fallback are paid per token on Token Factory. A GPU is needed only to serve the tuned student or to train it. ### What happens if I forget to stop the endpoint? It bills for every hour it runs: about $32 a day at about $1.35 per GPU-hour (an estimate). `bash deploy/student_endpoint.sh status` shows its state, and the cost report's `idle_hours` shows paid hours in which no request reached the student. ### Why would a live VM serve no student at all? A new live ledger has no trained version, so every request is a cold start answered by the teacher. A student appears only after Night School trains and releases one on that ledger, and the stock VM image cannot run Night School because it lacks the `nebius` and `aws` CLIs. Until that is wired up, a live VM is honest but all-teacher. ### Can I use a region other than eu-north1? For the VM, yes: set `RATCHET_VM_PLATFORM=cpu-d3`, which has the same `2vcpu-8gb` preset. The repo docs assume `eu-north1` for the GPUs (L40S and H100) and the bucket, which should sit in the same region as the GPUs. ### Why only one gateway worker? The ledger is one SQLite file behind one lock, and the release gate is not built for concurrent writers. A Postgres ledger for several workers is planned. ### Is the edge guard enough security? It closes the paid API and the admin API to anyone without the token, and it is enough for a small public gateway. For real clients use gateway API keys (`RATCHET_API_KEYS`) with quotas, and read the [security page](/guides/security/) and the [threat model](/reference/threat-model/). ## Related Reference: the [VM guide](/reference/deploy-nebius-vm/), [deployment](/reference/deployment/) and the [runbook](/reference/runbook/). --- # Deployment Source: https://docs.ratchet.dualmindlab.tech/reference/deployment/ There are two ways to run Ratchet: **mock** (local, no keys, about a minute) and **live** (real Nemotron on Nebius). The `nebius ai` commands in `deploy/` follow the Nebius CLI reference (auto generated on 19-Oct-2026, read Oct 21, 2026) and Nebius's [L40S](https://docs.ratchet.dualmindlab.tech/guides/glossary/#l40s) fine-tuning tutorial. They are tested offline, and the job and the endpoint each ran once by hand on Nebius (27 to 28 Oct 2026): run both scripts with `DRY_RUN=1` first, and check anything marked **verify on day 1** with `--help` before relying on it. `python scripts/day1_check.py` runs the checks. > **One command for the whole path:** `make nebius-plan` prints every step below (the CPU VM, the bucket, the student endpoint, the wiring and the smoke check) and changes nothing; `deploy/nebius/nebius.sh up --yes --with-student` runs them, with `RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1`. Same status as everything here: scripted, tested offline, ready to run. Three things have run on Nebius by hand: a training job (the round-2 retraining on 27 Oct 2026, evidence in `evidence/text2sql-round2-nebius/`), one speed test of the student on a Serverless Endpoint the same day (evidence in `evidence/student-endpoint-nebius/`; the endpoint was deleted afterwards), and one Night School end to end on 28 Oct 2026 (a CPU VM, a spot [L40S](https://docs.ratchet.dualmindlab.tech/guides/glossary/#l40s) job and an endpoint; the golden gate rejected the new student, so it was never made live; evidence in `evidence/nightschool-nebius/`). The wiring of the live gateway to an endpoint and a nightly schedule have not run. [Where everything runs](/reference/where-it-runs/) has the map and the costs. ## 1. Local mock mode ```bash git clone ratchet && cd ratchet python -m venv .venv && source .venv/bin/activate pip install -e ".[dev]" pytest -q # 2014 tests, ~120 s RATCHET_MODE=mock uvicorn ratchet.gateway:app --port 8000 # http://localhost:8000 ``` Mock mode answers with mock models and calls nothing: it is for developing and testing, and its numbers are not results. Real behaviour is in live mode (section 2). ## 2. Live mode on Nebius ``` Your app ──▶ Gateway (any host, RATCHET_MODE=live) ──▶ Token Factory (teacher/judge) │ └──▶ Serverless Endpoint (vLLM + LoRA) ◀── bucket /data/adapters cron: python -m ratchet.live night ──▶ Serverless Job (train) ──▶ bucket /data/adapters/ cron: python -m ratchet.live tick ``` ### Step 1: Token Factory key and checks 1. Join the Nebius Builder Program for credits. Its terms say $25 of Token Factory credit, expiring after 90 days (research notes, 2026-10-25; re-check the program terms before submission). 2. Create an API key in the Token Factory console. 3. Run the checks: ```bash export NEBIUS_API_KEY=... python scripts/day1_check.py ``` This confirms the four model IDs exist, and that the teacher answers, returns logprobs and supports JSON mode. If an ID differs, set `RATCHET_TEACHER_MODEL` (and update `ratchet/licenses.py` and `DEFAULT_PRICES`). ### Step 2: Object Storage bucket Create a bucket in the same region as your GPUs (eu-north1 has L40S and H100). It is mounted at `/data` in both the job and the endpoint, with this layout: ``` /data/night/train.jsonl written by ratchet.live (via RATCHET_SYNC_CMD): prompt-completion rows /data/boms/student-vN.json the adapter's bill of materials, written by ratchet.live after the night /data/adapters/student-vN/ written by the training job (weights, versions.json, ratchet_manifest.json), read by vLLM /data/adapters/student-vN/NOTICE.md the licenses that travel with the adapter (the job writes it; ratchet.live replaces it with the text the BOM signed) /data/adapters/student-vN/ratchet_bom.json the adapter's BOM, synced by ratchet.live after the night ``` `RATCHET_SYNC_CMD` runs twice on a night that trains: before the job (the dataset) and after the evaluation (the [BOM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#bom) and `NOTICE.md` into the adapter's folder). It must only upload: `aws s3 sync` without `--delete` leaves the weights the job wrote alone. Set: ```bash export RATCHET_BUCKET_ID= # for --volume export RATCHET_BUCKET_S3=s3:// # for polling in train_job.sh export RATCHET_S3_ENDPOINT=https://storage.eu-north1.nebius.cloud # verify on day 1 export RATCHET_DATA_DIR=./data export RATCHET_SYNC_CMD="aws s3 sync ./data s3:// --endpoint-url $RATCHET_S3_ENDPOINT" ``` The S3 endpoint URL format is an assumption; copy the real one from the console. ### Step 3: Training image ```bash docker build -f deploy/Dockerfile.train -t /ratchet-train:0.1 . docker push /ratchet-train:0.1 export RATCHET_TRAIN_IMAGE=/ratchet-train:0.1 ``` Build from the repository root (the image copies `deploy/train-versions.txt` and `ratchet/`). The image is CUDA 12.8 devel with exact versions of torch, Transformers, [PEFT](https://docs.ratchet.dualmindlab.tech/guides/glossary/#peft), TRL, Accelerate, datasets and the [Mamba](https://docs.ratchet.dualmindlab.tech/guides/glossary/#mamba) kernels (`causal-conv1d`, `mamba-ssm`) that Nemotron-H's layers run on: - `deploy/train-versions.txt` lists every pin, and each pip step after torch reads it as constraints, so a version that differs between the two files, or a dependency that would move a pinned package, fails the build. `tests/test_nightschool.py::test_dockerfile_is_pinned` checks the two files offline. - The [Mamba](https://docs.ratchet.dualmindlab.tech/guides/glossary/#mamba) kernels build with no `|| true`, and the last build step imports them (and checks torch's CUDA build), so an image without them never reaches a GPU. If no prebuilt wheel matches torch 2.11 and CUDA 12, they compile from source in the devel image, which takes a while. The import needs no GPU (torch's CUDA libraries load without a driver), but that was not checked on a real builder: if it fails with a driver error rather than a missing module, build on a GPU VM. - There is no Unsloth: its release of Oct 19, 2026 requires `trl<=1.15.0`, and the image pins TRL 1.16.0. `train.py` uses Transformers + [PEFT](https://docs.ratchet.dualmindlab.tech/guides/glossary/#peft) + TRL. - `train.py` checks every row and its rendering with the model's chat template before it trains: the student must be trained on the prompt the endpoint serves (thinking off). A `TemplateMismatch` or `row N: ...` error in the job's logs means the training file or the tokenizer does not match the endpoint; it stops the job before the GPU trains anything. - The job writes `versions.json` (Python, CUDA and every pinned package) next to the adapter before training starts. The versions are candidates read on Oct 21, 2026. To change one (or to adopt what the day-1 smoke job ran with, from its `versions.json`): edit `deploy/train-versions.txt` and the matching `"name==version"` in `deploy/Dockerfile.train` together (and `train.LIBRARIES` if you add a package), run `pytest -q tests/test_nightschool.py`, rebuild, push under a new tag and set `RATCHET_TRAIN_IMAGE` to it. If the image cannot build at all, the fallbacks in order are an NGC PyTorch base image (`nvcr.io/nvidia/pytorch`, a CUDA 12.8 tag) with the same pip steps, or Nebius's tutorial image `docker.io/axolotlai/axolotl:main-20260309-py3.11-cu128-2.9.1` ([fine-tuning tutorial](https://docs.nebius.com/serverless/tutorials/fine-tuning), read Oct 21, 2026) with `ratchet/` copied in; `versions.json` then records what that image holds. ### Step 4: Student endpoint (vLLM with runtime LoRA) ```bash export RATCHET_STUDENT_KEY=$(openssl rand -hex 24) DRY_RUN=1 bash deploy/student_endpoint.sh create # nebius validates the request (--dry-run), creates nothing bash deploy/student_endpoint.sh create # once export RATCHET_STUDENT_BASE=https:///v1 bash deploy/student_endpoint.sh status # prints " ", e.g. "aiendpoint-... RUNNING" bash deploy/student_endpoint.sh stop # when idle bash deploy/student_endpoint.sh start # the same endpoint again: same URL, same token ``` The four actions: | Action | What it runs | |---|---| | `create` | `nebius ai endpoint create ...` (below). Run it once: it stops with an error when an endpoint with the name exists | | `start` (the default) | `nebius ai endpoint start --id `. Nothing when the endpoint is already `RUNNING`, `STARTING`, `IMAGE_PULLING` or `PROVISIONING`; an error while it is `STOPPING` or `DELETING`. When no endpoint has the name, it runs `create` instead | | `stop` | `nebius ai endpoint stop --id `. Nothing when it is already `STOPPED` or `STOPPING`; an error when no endpoint has the name | | `status` | Prints ` ` (exit 1 when there is no such endpoint) | `nebius ai endpoint start` and `stop` take the endpoint's id, not its name (`--id`, required; there is no `--name`). The script finds the id by name (`RATCHET_ENDPOINT_NAME`, default `ratchet-student`) in `nebius ai endpoint list --format json`, read with `RATCHET_PYTHON`, or uses `RATCHET_ENDPOINT_ID` as given without the lookup. It never guesses: output that is not JSON, two endpoints with the name, an endpoint without an id, or a name missing from a list with more pages stops the script before anything starts, stops or is created; set `RATCHET_ENDPOINT_ID` then. (`nebius ai endpoint get-by-name` needs `--parent-id` and does not document what it does for a missing name, so the script reads the list.) With `DRY_RUN=1`, `create` passes `--dry-run`; `start` and `stop` have no `--dry-run`, so the script prints the command instead of running it. The read-only list lookup still runs, and no uptime is recorded. Every command is echoed on stderr first, with the token shown as ``. After `create`, `start` and `stop`, the script records the endpoint in the ledger (`python -m ratchet.cost endpoint up|down`, with `RATCHET_DB` pointing at the ledger file), so the cost report knows the GPU-hours you paid for ([Cost estimates](#4-cost-estimates)). Recording is best effort: run where the ledger is not, the deploy still succeeds and prints the command to run there (or use `POST /ratchet/cost/endpoint` with an admin key). A stopped endpoint is not billed for compute or its own storage, but the mounted bucket still is, also while the endpoint is stopped ([Nebius Serverless overview](https://docs.nebius.com/serverless/overview), read Oct 21, 2026). What `create` sets: - Image: `vllm/vllm-openai:v0.18.0-cu130` (`RATCHET_VLLM_IMAGE`; `vllm/vllm-openai:v0.18.0` is the CUDA 12 build, for a host driver too old for CUDA 13) - Platform and preset: `gpu-l40s-a`, `1gpu-8vcpu-32gb` - `--container-port 8000`, `--shm-size 16Gi` - `--volume $RATCHET_BUCKET_ID:/data`, and `--subnet-id $RATCHET_SUBNET_ID` when set - `--auth token --token $RATCHET_STUDENT_KEY` - `--env VLLM_ALLOW_RUNTIME_LORA_UPDATING=True` - [vLLM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#vllm) args: `--model nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --trust-remote-code --enable-lora --max-lora-rank 16 --max-loras 4 --mamba_ssm_cache_dtype float32 --max-model-len 8192 --max-num-seqs 64 --default-chat-template-kwargs '{"enable_thinking":false}'`. `--mamba_ssm_cache_dtype float32` is the model card's flag ([model card](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16), read Oct 21, 2026). The context cap (`RATCHET_SERVE_MAX_LEN`) is far below the card's 262,144-token example: ticket prompts are short, and a longer one gets [vLLM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#vllm)'s 400 and escalates. `--default-chat-template-kwargs` (vLLM v0.18.0, parsed as JSON) turns thinking off unless a request asks (`RATCHET_SERVE_TEMPLATE_KWARGS`, empty to leave it out). With `RATCHET_REASONING_PARSER_PLUGIN=/data/nano_v3_reasoning_parser.py` (the card's parser, uploaded to the bucket), it adds `--reasoning-parser-plugin /data/nano_v3_reasoning_parser.py --reasoning-parser nano_v3`. The vLLM arguments are one string that the container splits as a shell would (Nebius's fine-tuning tutorial relies on the same splitting for its `bash -c` script), so the JSON keeps its single quotes. `--dry-run` validates the request but cannot show the split: on day 1, check that the endpoint reaches `RUNNING` and that a request without `chat_template_kwargs` returns no reasoning. If vLLM refuses the flag, recreate the endpoint with `RATCHET_SERVE_TEMPLATE_KWARGS=` and rely on the per-request `chat_template_kwargs`. Checks once it is up: ```bash curl -s $RATCHET_STUDENT_BASE/models -H "Authorization: Bearer $RATCHET_STUDENT_KEY" # load an adapter by hand (ratchet.live does this automatically after training) curl -s ${RATCHET_STUDENT_BASE%/v1}/v1/load_lora_adapter -H "Authorization: Bearer $RATCHET_STUDENT_KEY" \ -H 'content-type: application/json' -d '{"lora_name":"student-v1","lora_path":"/data/adapters/student-v1"}' ``` > Adapters loaded at runtime are lost when the endpoint restarts. The gateway notices vLLM's 404 for an adapter and loads it again by itself (at most once per adapter per minute; `adapter_reloaded` events), so student calls fall back to the teacher for a few requests (they still succeed, at teacher cost). `python -m ratchet.live reload` re-loads every `live`, `canary` and `shadow` adapter by hand if that fails. See [RUNBOOK.md](/reference/runbook/#student-endpoint-down-or-adapters-lost). ### Step 5: Training job `deploy/train_job.sh ` submits (`DRY_RUN=1` adds `--dry-run`: nebius validates the request and submits nothing): ``` nebius ai job create --preemptible --follows-spot-price --name ratchet- --image $RATCHET_TRAIN_IMAGE --platform gpu-l40s-a --preset 1gpu-8vcpu-32gb --volume $RATCHET_BUCKET_ID:/data --disk-size 200Gi --timeout 2h --container-command bash --args "-c 'bash /app/install_kernels.sh && python -m ratchet.nightschool.train --data /data/night/train.jsonl --base --out /tmp/adapter-out && mkdir -p /data/adapters/ && cp -r /tmp/adapter-out/. /data/adapters//'" [--subnet-id $RATCHET_SUBNET_ID] [--registry-secret $RATCHET_REGISTRY_SECRET] ``` `nebius ai create` is the interactive wizard; scripts use `nebius ai job create` ([CLI reference](https://docs.nebius.com/cli/reference/ai/job/create)). `--timeout` is the job's own limit (1 h to 168 h, default 24 h), `--subnet-id` defaults to the project's default subnet, and `--registry-secret` names a MysteryBox secret with the credentials of a private image. With `RATCHET_BUCKET_S3` set, the script then polls for `adapters//adapter_model.safetensors` every 30 s for up to 2 h. That makes it suitable as `RATCHET_TRAIN_CMD`, which must block until the adapter exists; a refused job fails the script, so the night stops instead of waiting. What the 28 Oct 2026 run taught the script (evidence: `evidence/nightschool-nebius/`): the job is **spot by default** (`--preemptible --follows-spot-price`); `RATCHET_TRAIN_ON_DEMAND=1` is the only way to pay the on-demand price. The adapter is saved to the job's **local disk** and then copied into the bucket mount, because the mount refuses the in-place rename safetensors makes (`EPERM`). `einops` is pinned in `deploy/train-versions.txt` and installed by the image (rebuild it with the build-train-image workflow after this change). With `RATCHET_BUCKET_S3` set, the script and the night both need the **`aws` CLI on PATH** and stop with a message when it is missing, instead of waiting for an adapter they cannot see. The student endpoint must be **running before the [hot-load](https://docs.ratchet.dualmindlab.tech/guides/glossary/#hot-load)**: the night does not start or wake it, and the first call after a [hot-load](https://docs.ratchet.dualmindlab.tech/guides/glossary/#hot-load) into a cold LoRA endpoint is slow, so the [golden eval](https://docs.ratchet.dualmindlab.tech/guides/glossary/#golden-set)'s student calls have their own timeout, `RATCHET_NIGHT_STUDENT_TIMEOUT_S` (120 s by default; the request path keeps `RATCHET_STUDENT_TIMEOUT_S`). Training defaults (`train.py`): 2 epochs, learning rate 2e-4, LoRA rank 16 (alpha 32), max sequence 2048, batch 8, bf16, assistant-only loss. ### Step 6: Gateway Any small VM or container with Python 3.10+ works: ```bash pip install -e . export RATCHET_MODE=live RATCHET_DB=/var/lib/ratchet/ratchet.db export RATCHET_API_KEYS="app:$(openssl rand -hex 24)" RATCHET_DASHBOARD_KEYS="me:$(openssl rand -hex 24)" python -m ratchet.config check # exit 0 clean, 1 warnings, 2 errors: the gateway would not start uvicorn ratchet.gateway:app --host 0.0.0.0 --port 8000 --workers 1 ``` A bad setting stops startup with one error that lists every problem by variable ([CONFIGURATION.md](/reference/configuration/#configuration-checks)); `python -m ratchet.config check` gives the same verdict without starting anything, so run it after each change to the environment. Set `RATCHET_STRICT_CONFIG=1` to make warnings (an unknown or misspelled variable, an unpriced model) fatal too. Health checks ([API.md](/reference/api/#get-readyz)): - **Load balancers and deploy scripts: `GET /readyz`.** 200 (`ready`, or `degraded` while the teacher answers for a student endpoint that is down) keeps the instance in rotation; 503 `not_ready` (the ledger cannot be written, or no teacher can answer) takes it out. An instance taken out comes back by itself: once the teacher's breaker has waited out its cooldown and the teacher's `/models` answers, `/readyz` says 200 `degraded` (`probe_due`) and the next request probes the teacher. The probes are cached for 15 s and spend no tokens, so poll it as often as you like. The student endpoint gets one `GET /models` per 5 minutes from each gateway; if your platform scales an idle endpoint to zero and counts that request as traffic, it keeps the endpoint running. - **Restarts (Docker, systemd, a Kubernetes liveness probe): `GET /healthz` or `/livez`.** It only says the process answers, also with every worker thread busy. Never restart on `/readyz`: a teacher outage would turn into a restart loop. Use **one worker**. The ledger is a single SQLite connection inside the process, and the release gate is not designed for concurrent writers. Live mode does not start without `RATCHET_API_KEYS` (or `RATCHET_ALLOW_UNAUTHENTICATED=1` when a proxy checks callers instead), and its dashboard API (`/ratchet/stats`, `/ratchet/bom`, ...) answers only admin and dashboard keys unless `RATCHET_PUBLIC_DASHBOARD=1` ([API.md](/reference/api/#access-levels)). Put TLS in front with a reverse proxy. ### Step 7: Cron `python -m ratchet.live` must run on a machine that has the `nebius` CLI, the `aws` CLI (for sync and polling), the same `RATCHET_DB` file, and network access to the student endpoint. In v0.1 that means the same host as the gateway. The ledger runs in SQLite's WAL journal (`RATCHET_DB_SYNC=NORMAL`, the default), which needs every process that opens it on that one host with the file on a local disk: never put `RATCHET_DB` on the bucket mount or another network filesystem (if you must, set `RATCHET_DB_SYNC=FULL` for every process). ```cron # Night School at 02:30 local time 30 2 * * * cd /opt/ratchet && . ./.env && python -m ratchet.live night >> /var/log/ratchet-night.log 2>&1 # Release gate every 5 minutes (the gateway also ticks on every request) */5 * * * * cd /opt/ratchet && . ./.env && python -m ratchet.live tick >> /var/log/ratchet-tick.log 2>&1 # Savings receipts of every ledger day that has ended, hourly: a ledger day starts at the first request's time of day, # not at 00:00 UTC (the gateway also seals when a receipt is asked for) 20 * * * * cd /opt/ratchet && . ./.env && python -m ratchet.live seal >> /var/log/ratchet-seal.log 2>&1 ``` `python -m ratchet.live night` runs `run_night` with `NebiusTrainer` and `LiveEvaluator` (judge = `RATCHET_JUDGE_MODEL`), writes the adapter's BOM and `NOTICE.md` into `RATCHET_DATA_DIR/adapters//` and runs `RATCHET_SYNC_CMD` again, then prints the last 10 events. If that second sync fails, the night still counts (the version exists and is in the release gate), a `sync_failed` event is written and the command exits non-zero: run `RATCHET_SYNC_CMD` by hand. `tick` runs `ReleaseGate.tick()` once. `seal` seals the savings receipts of every day that has ended ([RUNBOOK](/reference/runbook/#savings-receipts)); `purge` seals them too before it folds request rows. Without `RATCHET_STUDENT_BASE`, `night` stops before syncing or starting the training job, and `reload` stops with the same message (nothing to reload is not an error). The same commands run offline in `tests/test_live.py` and `tests/test_licensing.py`, against fake endpoints, before they run here. ### Export an adapter with its licenses To hand an adapter to someone (another team, a customer, a model registry), export it with its licenses: ```bash aws s3 sync s3:///adapters/student-v3 ./data/adapters/student-v3 --endpoint-url $RATCHET_S3_ENDPOINT # weights, if not local python -m ratchet.live export --version 3 --out ./student-v3-export # exported student-v3 (v3) from data/adapters/student-v3 to student-v3-export # bill of materials: student-v3-export/ratchet_bom.json (signature valid (Ed25519, key 6cf6bc1a39c4491d, embedded key only; ...)) # notice: student-v3-export/NOTICE.md # bom sha256: 9b1f… ``` The folder gets the adapter files from `RATCHET_DATA_DIR/adapters//` when they are on this machine. With an upload-only `RATCHET_SYNC_CMD`, that folder holds only the `ratchet_bom.json` and `NOTICE.md` the night wrote, not the weights; the export then counts no adapter files and says so (`exported the licenses of student-v3 (v3) to student-v3-export; the adapter weights (adapter_config.json and adapter_model.safetensors) are not at data/adapters/student-v3: copy them from the bucket's adapters/student-v3/ into student-v3-export`). It also gets the BOM stored at training time (`ratchet_bom.json`, checked against its own `sha256` and signature first; a BOM that does not verify is refused) and the `NOTICE.md` that BOM signed: the base model's license (the NVIDIA Nemotron Open Model License for the Nano models), each teacher's license and URL, that the targets were teacher outputs only, and the Ratchet version. The output folder must be new or empty, and outside `RATCHET_DATA_DIR`, which the next sync would upload to the bucket. Give the recipient your public key: `python -m ratchet.bom verify student-v3-export/ratchet_bom.json --public-key --notice student-v3-export/NOTICE.md` prints `OK` for the BOM and `OK: NOTICE.md matches the BOM` for the notice (its SHA-256 equals the BOM's `notices[0].sha256`). Keep `NOTICE.md` with the adapter wherever it goes; a version trained before v0.4 has no BOM and is refused. ### Step 8: Measure The live-run harness (`python -m ratchet.liverun`) runs on the same host and with the same `.env` as `python -m ratchet.live`, because it runs the nights itself between its rounds and reads the gateway's ledger file. Add `liverun:` to the gateway's `RATCHET_API_KEYS` and set the run's gate profile (`RATCHET_SEQUENTIAL=1`, `RATCHET_CANARY_AUDIT_RATE=0.5`) on the gateway, restart it, pause the cron nights, then: ```bash export RATCHET_LIVERUN_KEY= # in this shell only, never a flag make live-plan && make live-smoke && make live-run MAX_USD=30 ``` On a host or image without `.git`, set `RATCHET_GIT_SHA` to the deployed commit (`git rev-parse HEAD` where you built it) so the results file names the code; it carries a digest of the sources either way. The [runbook](/reference/runbook/#live-measurement-run) has the prerequisites, the stop reasons, resuming, the repeat run, entering the Nebius bill and publishing. ## 3. Keeping the live gateway up A public gateway needs a working URL for as long as it is shown, and a GPU endpoint running nonstop is the biggest cost. **Where the live gateway runs today:** the live gateway (live.ratchet.dualmindlab.tech) runs on Cloudflare Workers with Containers, see [deploy/cloudflare-live/README.md](https://github.com/harshbhx/ratchet/blob/main/deploy/cloudflare-live/README.md). The live Worker limits each visitor to 3 "Try a real request" calls and 3 Scout runs per UTC day (a Durable Object counts them by a hash of the visitor's address, before the container wakes; `EDGE_DAILY_TRIES` and `EDGE_DAILY_SCOUT` change the numbers, `0` turns one off; a judge's cookie skips the Scout count), and the shared site key's own limit is 60 requests a minute. An optional Cloudflare Turnstile check ("verify you are human") sits in front of the browser buttons that spend money: it is off until the Worker secret `TURNSTILE_SECRET_KEY` is set, and the widget appears once the Worker variable `TURNSTILE_SITE_KEY` is set too (a token must also have been solved on one of the hostnames in `TURNSTILE_HOSTNAMES`, default the site and the live host); a visitor verifies once per visit, because the Worker answers a passed check with a 12-hour `rt_human` cookie (signed from the Turnstile secret, bound to the visitor's address, `GET /ratchet/human` says whether it is valid); a plain `curl` without a token keeps its 3 a day and shares `EDGE_DAILY_CURL_TOTAL` (default 100) a day across everybody (setup and limits in [deploy/cloudflare-live/README.md](https://github.com/harshbhx/ratchet/blob/main/deploy/cloudflare-live/README.md)). To know the visitor's address behind the main site's Pages proxy, both sides need the same secret `EDGE_TOKEN`: `npx wrangler secret put EDGE_TOKEN --name ratchet-live` in `deploy/cloudflare-live`, and `npx wrangler pages secret put EDGE_TOKEN --project-name ratchet` for the site (never in the repo); without it all main-site visitors count as one address. To give judges a key of their own, set the optional Worker secret `JUDGES_API_KEY` (Cloudflare dashboard, Workers & Pages, ratchet-live, Settings, Variables and Secrets, Secret `JUDGES_API_KEY`, or `npx wrangler secret put JUDGES_API_KEY` in `deploy/cloudflare-live`; limits 300 requests and $1 a day, 30 a minute, 2 at once; a dashboard `RATCHET_API_KEYS` or `RATCHET_KEY_LIMITS` replaces the built-in `demo` and `judges` entries). A judge calls `curl https://live.ratchet.dualmindlab.tech/v1/chat/completions -H "Authorization: Bearer " -H "Content-Type: application/json" -d '{"model":"ratchet","messages":[{"role":"user","content":"..."}]}'`, or with the OpenAI SDK `OpenAI(base_url="https://live.ratchet.dualmindlab.tech/v1", api_key="")`. The Nebius CPU VM kit below is the alternative host: it is written and tested offline, ready to run with `make nebius-plan`. **Alternative host: one Nebius CPU VM.** [DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/) sets up the gateway with Docker Compose and Caddy (automatic HTTPS) on a 2 vCPU / 8 GB VM, with the budget guard, an opt-in edge guard for `/v1/*` and `/openai/*`, backups, monitoring, updates and teardown. It starts in mock mode with a warmed-up student. The points below still apply to it. 1. **The live gateway at the root domain** (the URL for Stage One), in live mode with `RATCHET_STUDENT_BASE` **unset** when the GPU endpoint is off. Student calls then go to the hosted, untuned `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` on Token Factory, and you pay per token instead of per GPU-hour. Be honest that this is the fallback, not the tuned student. Note that a version's calibrated threshold was set for the tuned adapter, so precision on the fallback may be lower; audits will show it. 2. **Stop the vLLM endpoint when not recording or testing** (`bash deploy/student_endpoint.sh stop`, then `start` to bring the same endpoint back). A stopped endpoint is not billed for compute or its own storage; a mounted Object Storage bucket still is ([Serverless overview](https://docs.nebius.com/serverless/overview), read Oct 21, 2026). 3. **Cap teacher spend:** keep `RATCHET_AUDIT_RATE` at 0.05 or lower and watch the Token Factory usage page daily. In live mode, keep Scout's budget guard on (`RATCHET_SCOUT_DAILY_BUDGET_USD`, `RATCHET_SCOUT_MAX_RUNS_PER_DAY`, `RATCHET_SCOUT_RUNS_PER_HOUR`; see [CONFIGURATION.md](/reference/configuration/#scout-the-multi-agent-research-tool)) and turn on the Caddy guard (`RATCHET_API_GUARD=guarded`). ## 4. Cost estimates These are **estimates**: Serverless pricing was not confirmed (on-demand list prices assumed), and token counts depend on your prompts. Ratchet measures the real thing once it runs: `python -m ratchet.cost report` (or `GET /ratchet/cost`) prices the recorded endpoint and training hours at `RATCHET_STUDENT_GPU_USD_PER_HOUR` and `RATCHET_TRAIN_GPU_USD_PER_HOUR`, reports savings per token and by GPU-hours paid, the break-even volume, and whether self-hosting pays at your traffic ([RUNBOOK.md](/reference/runbook/#cost-and-the-self-host-recommendation)). At $1.35 per GPU-hour (an L40S on demand, [Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026; an estimate until your bill confirms it) a 24/7 endpoint breaks even near 1,350 student-served requests an hour; below that, use the hosted fallback or stop the endpoint when idle. Above it the endpoint costs less than the teacher, but the untuned hosted fallback is cheaper per answer below its crossover (roughly 52,000 student answers an hour at list prices), if it serves as many requests as the tuned student. | Item | Assumption | Estimate | |---|---|---| | Training run | L40S at $1.35/GPU-hour, 30–60 min per night | ~$0.70–1.35 per night (estimate) | | Student endpoint | L40S at $1.35/GPU-hour, only while testing or recording | ~$32 per full day if left on (estimate) | | Teacher calls | Ultra $1 / $3 per 1M tokens (list price in `ratchet/config.py`, checked against the Token Factory price list on 29 Oct 2026); ~150 input + ~380 output tokens per call | ~$1.30 per 1,000 calls | | Judge | Super $0.30 / $0.90 (same source); only for free-text tasks | small | | Short evaluation total | 6–8 training runs, endpoint on a few hours a day, ~$20–30 of teacher calls | **~$35–75** (estimate) | ## 5. Verification after deploy 1. `curl $GATEWAY/healthz` returns `{"status":"ok",...}`, `curl $GATEWAY/readyz` returns 200 with `"status": "ready"` (or `degraded`, with the reason under `checks`; a routed adapter the endpoint no longer lists is in `checks.student.missing_adapters` and queued for reload), and `curl -H "Authorization: Bearer $KEY" $GATEWAY/v1/models` returns `ratchet` (drop the header if `RATCHET_API_KEYS` is not set). 2. Send 20 real requests. `GET /ratchet/stats` (with `-H "Authorization: Bearer $DASHBOARD_KEY"` unless the dashboard is public) shows them with `served_by=teacher` (cold start), and `GET /ratchet/bom` shows the teacher with license `OpenMDW-1.1`. 3. Run `python -m ratchet.live night` once you have at least 50 new examples. Expect a `trained` event and either `canary` or `rejected`, and a bill of materials at `data/boms/student-v1.json` (also at `GET /ratchet/versions/1/bom`), with `ratchet_bom.json` and `NOTICE.md` in the bucket's `adapters/student-v1/` next to the weights. With `RATCHET_BOM_SIGNING_KEY` set, `python -m ratchet.bom verify data/boms/student-v1.json --public-key ` prints `OK`. 4. Check that `curl $RATCHET_STUDENT_BASE/models` lists `student-v1` after the night. 5. Watch the `canary` → `live` events over the next requests. Audits need traffic: at a 5% audit rate and 10% canary, reaching 20 audits takes roughly 4,000+ requests, so for a short test run raise `RATCHET_AUDIT_RATE`, or lower `RATCHET_MIN_AUDITS` (the default is 20). ## 6. The video link `video.ratchet.dualmindlab.tech` is a short, counted link to the video (`https://youtu.be/x9MpEdKyODc`): a small Cloudflare Worker with a D1 database, in `deploy/cloudflare-video/`. Only the manual workflow `deploy-cloudflare-video` deploys it (Actions, Run workflow, type `deploy-video`; the same two repository secrets as the other deploys). It runs the Worker's tests, applies the D1 migrations, deploys, and then checks the link from outside with HEAD requests only, so a deploy adds no click. - **Every GET or HEAD**, on any path, is a `302` to the video with `Cache-Control: no-store`. Both are counted, a HEAD (link checkers, the deploy smoke) always as a bot with the reason `head`; `/robots.txt`, `/favicon.ico` and `/.well-known/*` are not counted. The count is written after the answer has gone out (`ctx.waitUntil`), so a slow or failing D1 never delays or breaks the redirect. - **Where a link was posted** goes in `?from=`: `devpost`, `readme`, `site`, `live`, `demo`, `docs`, `hf`, `youtube`, `x`, `linkedin`, `discord`, `email`, `release`, `judges` or `other`. A link without it counts as `direct` and any other value as `other`; the raw query is never stored. Ratchet's own `/video` paths forward here with their source in a no-store 302: the story site (`site`, in `web/public/_worker.js`), the live gateway (`live`, answered by its Worker before the container is touched) and the docs (`docs`, `docs-site/public/_redirects`); the README's links use `readme`, the story page's `site` and the judges' page's `judges`. A test in `deploy/cloudflare-video/test/config.test.mjs` checks every such tag against the allowlist. - **Human or bot, and why.** Most "human" clicks on a link in a message are email and chat link scanners, previews and data-centre crawlers with a browser user agent, so the verdict has several rules (one small function, `verdictOf` in `src/visitor.ts`, tested in `test/visitor.test.mjs`). The first that matches sets `kind` to `bot` and records the reason; none matching is `human` with an empty reason: 1. `ua`: a user agent that names a bot, crawler, spider or preview (Slackbot, Twitterbot, Discordbot, facebookexternalhit, LinkedInBot, WhatsApp, TelegramBot, Googlebot, bingbot, Applebot and the like), a script or library (curl, python-requests, Go-http-client, ...), a headless browser, this repository's link check or the deploy smoke, an empty user agent, or a browser prefetch. 2. `verified-bot`: Cloudflare marks the request as a verified bot. 3. `head`: a HEAD request (a browser opening a link sends a GET). 4. `scanner`: the network is an email-security scanner: Proofpoint (AS22843, AS26211), Mimecast (AS30031), Barracuda (AS15324), or the network name says so. 5. `datacenter`: the network is a hosting or cloud provider: Amazon (AS16509, AS14618), Google (AS15169, AS396982), Microsoft (AS8075, which also runs Outlook and Defender link scanning), Cloudflare (AS13335), DigitalOcean (AS14061), OVH (AS16276), Hetzner (AS24940), Akamai/Linode (AS63949), Oracle (AS31898), Alibaba (AS45102), Tencent (AS132203), Contabo (AS51167), Vultr (AS20473), Scaleway (AS12876), Meta (AS32934); or its name has hosting, cloud, data center, server, vps or colocation in it. 6. `no-accept-language`: a GET without an Accept-Language header (every browser sends one). A chat app's link preview is a bot, and the person who then clicks the link is a human. A real person on a corporate Microsoft or Google network is counted as a bot by rule 5; that is the price of filtering the scanners. - **What is stored** in D1 (`ratchet-video`): a count per UTC day, source and kind (`clicks`); for each counted visit (`events`) its time, source, kind and bot reason, the country, city and region and the network's name and number that Cloudflare reports for the connection, three coarse words for the device (desktop, phone or tablet; Windows, macOS, Android, iOS, Linux, ChromeOS or other; Chrome, Safari, Firefox, Edge, Samsung or other) and a visitor hash; and how many alert emails went out per UTC day (`mail`). All of the per-visit data is deleted after 30 days by the daily cron. **Never stored:** the IP address, the full user agent, a cookie, the path or the raw query. The visitor hash is the first 16 hex characters of HMAC-SHA256 of the IP address, keyed with a salt derived from the `STATS_KEY` secret and the UTC day. The salt changes every day, so the hash cannot follow a person from one day to the next, and it cannot be turned back into the IP address. It counts unique humans per day and limits the alert emails to one per visitor per day. Workers Logs are off for this Worker, so request details are not kept there either; `npx wrangler tail ratchet-video` shows its failure lines live. Migration `0002_visitor_details.sql` adds these columns; rows from before it have empty fields. **Reading the stats.** The stats are a private endpoint, not a public page (it answers 404 without the key). `GET /stats` on the Worker's own address (`VIDEO_STATS_URL` below) with the Worker secret `STATS_KEY` as a bearer token returns totals per day and source (human and bot apart), per source, per country over the last 30 days, unique humans per day, bots by reason, the top networks of humans and of bots, the top cities and devices of humans, and the emails sent; add `?format=html` for the same as tables. Without the key, or with a wrong one, `/stats` is a 404. The key is kept outside the repository; to replace it, pipe a new random value into `npx wrangler secret put STATS_KEY --name ratchet-video` from `deploy/cloudflare-video/`. ```bash curl -s -H "Authorization: Bearer $(cat "$STATS_KEY_FILE")" "$VIDEO_STATS_URL" # the Worker's private /stats endpoint ``` **Email alerts.** With `EMAIL_ENABLED` set to `"1"` in `deploy/cloudflare-video/wrangler.json` (it is: the destination address is verified), each human visitor sends a short plain-text email from `EMAIL_FROM` to `EMAIL_TO` through the Worker's `send_email` binding (Cloudflare Email Routing), at most one per visitor per UTC day and never for a bot: the city and country, the network, the device (for example "Windows desktop, Chrome"), the source, the time in IST and "unique humans today: N", never an IP address or a user agent. At most 40 visit emails go out per UTC day; past that, visits are still counted. A daily cron at 03:30 UTC (09:00 IST) emails a summary of the previous UTC day (humans, unique humans and bots, bots by reason, the top 5 networks of humans and of bots, clicks per source and countries; nothing for a day without clicks) and prunes the old events, also while email is off. To switch email on for a new address: 1. The destination address must be verified in Cloudflare Email Routing (dashboard: the `dualmindlab.tech` zone, Email, Email Routing, Destination addresses; Cloudflare emails it a confirmation link). The sender is on `mail.dualmindlab.tech`, which already has Cloudflare's Email Routing and Email Sending records; the root domain's own mail records are not touched. 2. Set `EMAIL_ENABLED` to `"1"` in `wrangler.json`, merge, and run `deploy-cloudflare-video`. Setting it to `"0"` (also the behaviour when the variable is missing) stops every email at the next deploy. The redirect and the counts never depend on email: a missing binding, a bad address or a failed send only skips the email. --- # An alternative host: the Nebius CPU VM kit (ran once by hand; the one-command kit is tested offline) Source: https://docs.ratchet.dualmindlab.tech/reference/deploy-nebius-vm/ > **Where the live gateway runs today:** the live gateway (live.ratchet.dualmindlab.tech) runs on Cloudflare Workers with Containers, see [deploy/cloudflare-live/README.md](https://github.com/harshbhx/ratchet/blob/main/deploy/cloudflare-live/README.md). The Nebius CPU VM kit below is the alternative host: it is tested offline, and the VM ran once by hand on 28 Oct 2026 for the [Night School](https://docs.ratchet.dualmindlab.tech/guides/glossary/#night-school) run ([evidence](https://github.com/harshbhx/ratchet/blob/main/evidence/nightschool-nebius/README.md)); the one-command kit (`make nebius-plan`) has not run as one. Where this guide says a restart never resets the [Scout](https://docs.ratchet.dualmindlab.tech/guides/glossary/#scout) caps, that holds where the [ledger](https://docs.ratchet.dualmindlab.tech/guides/glossary/#ledger) is on a volume (this kit); on the Cloudflare Containers gateway the ledger is on the container's disk and a restart resets the day's counters. This guide puts the live Ratchet gateway on one small [Nebius AI Cloud](https://docs.ratchet.dualmindlab.tech/guides/glossary/#nebius-ai-cloud) virtual machine, so evaluators can use it if you choose this host. The VM runs the live gateway and [Caddy](https://caddyserver.com/), which gets HTTPS certificates automatically; the gateway is at your domain (the URL for Stage One). The kit also holds an optional keyless mock gateway, `ratchet-sim`, which is not a public page and which `RATCHET_SIM=off` leaves out. > **One command instead of the steps below:** `make nebius-plan` prints every command, and `deploy/nebius/nebius.sh up --yes` (with `RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1`) creates the VM, optionally the [student](https://docs.ratchet.dualmindlab.tech/guides/glossary/#student) endpoint, wires the VM's `.env` and runs a smoke check; `down` tears it down. It is scripted and tested offline like the rest of this kit, ready to run. See [Where everything runs](/reference/where-it-runs/#one-command-to-host-it-all-on-nebius). Files: [`deploy/vm/`](https://github.com/harshbhx/ratchet/tree/main/deploy/vm). Facts about Nebius below were checked on docs.nebius.com and nebius.com on **Oct 20, 2026** (items dated Oct 21 and 22 were re-checked then). Anything marked **verify on day 1** is checked on the first real run. ## Why a CPU VM The gateway does no model inference. It routes requests, keeps the [SQLite](https://docs.ratchet.dualmindlab.tech/guides/glossary/#sqlite) ledger and serves the web pages. The models run elsewhere: the [teacher](https://docs.ratchet.dualmindlab.tech/guides/glossary/#teacher) on Token Factory, the student on a Serverless Endpoint (or the hosted fallback). So a 2 vCPU / 8 GB VM is enough, and it costs about a tenth of the cheapest GPU. ``` judge's browser ──HTTPS──▶ Caddy :443 ─┬─ https:/// ──▶ ratchet:8000 (live gateway, 1 worker, SQLite in a volume) │ │ live mode only │ ├──▶ Token Factory (teacher, judge, fallback student) │ ├──▶ Serverless Endpoint (student, optional) │ └──▶ Tavily (Scout search) ``` The optional `ratchet-sim` container has its own environment (no `env_file`, no volume, no published port, never a live key, token or signing key) and is capped at 1 vCPU and 1 GiB. The live host keeps the app's 403 on every mock-only route and the optional edge guard. ## What you need 1. **The Nebius CLI**, installed and logged in ([CLI quickstart](https://docs.nebius.com/cli/quickstart)): ```bash curl -sSL https://artifacts.nebius.cloud/cli/install.sh | bash exec -l $SHELL && nebius version export NEBIUS_PROJECT_ID= nebius profile create --parent-id $NEBIUS_PROJECT_ID # browser login ``` The installer runs on Ubuntu and macOS. On Windows, run it in WSL, or use Nebius Cloud Shell, which has the CLI already. 2. **jq** on your laptop (the script reads the public IP with it). 3. **An SSH key**, for example `~/.ssh/id_ed25519.pub` (`ssh-keygen -t ed25519`). 4. **The repo public** at `https://github.com/harshbhx/ratchet`. The VM clones it on first boot. - Private repo instead: create a fine-grained GitHub token with read-only access to this one repo, and set `RATCHET_REPO_URL=https://x-access-token:@github.com/harshbhx/ratchet.git`. The VM removes the token from its git remote after cloning, but the token stays in the VM's cloud-init user data. Revoke it after the first boot. Later `git pull` (and `update.sh`) then needs the repo public, or a credential you add by hand. 5. **A region.** The default platform `cpu-e2` (Intel Ice Lake) exists only in `eu-north1`. In other regions use `RATCHET_VM_PLATFORM=cpu-d3` (AMD EPYC Genoa). Both have the `2vcpu-8gb` preset ([VM types](https://docs.nebius.com/compute/virtual-machines/types)). ## Step by step ### 1. Create the VM ```bash export NEBIUS_PROJECT_ID= export SSH_PUBLIC_KEY_PATH=~/.ssh/id_ed25519.pub DRY_RUN=1 bash deploy/vm/create_vm.sh # prints the commands, creates nothing bash deploy/vm/create_vm.sh ``` The script: 1. Finds the first subnet in the project (`nebius vpc subnet list`), unless you set `NEBIUS_SUBNET_ID`. 2. Creates a 30 GiB `network_ssd` boot disk from the image family `ubuntu24.04-driverless` (`nebius compute disk create`). 3. Creates the VM `ratchet-host` (`nebius compute instance create`) with that disk, a **static** public IP (kept while the VM is stopped), and [`cloud-init.yaml`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/cloud-init.yaml) as user data. 4. Prints the public IP, the VM and disk IDs, and the next steps. On first boot, cloud-init installs Docker Engine and the Compose plugin from Docker's apt repository, turns on unattended security upgrades and the `ufw` firewall (ports 22, 80, 443), creates a `ratchet` service user, clones the repo to `/opt/ratchet`, writes `deploy/vm/.env` from [`.env.vm.example`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/.env.vm.example), and runs `docker compose up -d --build`. This takes 5–10 minutes. ```bash ssh ubuntu@ 'sudo tail -f /var/log/ratchet-bootstrap.log' ``` Optional settings: `RATCHET_VM_NAME`, `RATCHET_VM_PRESET` (for example `4vcpu-16gb`), `RATCHET_VM_DISK_GB`, `RATCHET_SSH_USER` (default `ubuntu`; Nebius reserves `root` and `admin`), `RATCHET_DOMAIN`. ### 2. Pick the web address - **Fastest: sslip.io.** If you do not set `RATCHET_DOMAIN`, first boot sets it to `.sslip.io`, for example `203.0.113.7.sslip.io`. That name resolves to the IP, so Caddy gets a real certificate without any DNS work. The VM finds its public IP by asking `api.ipify.org` (verify on day 1). - **Your own name:** add a DNS **A record**, for example `gateway.example.com → `, wait until `dig +short gateway.example.com` returns the IP, then on the VM: ```bash cd /opt/ratchet/deploy/vm sudo -u ratchet sed -i 's|^RATCHET_DOMAIN=.*|RATCHET_DOMAIN=gateway.example.com|' .env sudo -u ratchet docker compose up -d ``` Set the DNS record **before** switching. Caddy retries failed certificate requests, but repeated failures can hit Let's Encrypt rate limits. - **The optional mock host's name** is `sim.` unless you set `RATCHET_SIM_DOMAIN`. `RATCHET_SIM=off` in `.env` drops the host (then add an empty `RATCHET_SIM_URL=`); with it on and your own domain, add a second A record (or a wildcard) for `sim.` before `docker compose up -d`. ### 3. Edit `.env` ```bash sudo -u ratchet nano /opt/ratchet/deploy/vm/.env # mode 0600, owned by ratchet cd /opt/ratchet/deploy/vm && sudo -u ratchet docker compose up -d # apply changes ``` The file starts in `RATCHET_MODE=mock` with `RATCHET_MOCK_WARMUP_DAYS=4`, so it starts on mock models until you switch to live (below). Every variable is explained in the file. ### 4. Check it works | URL | Expect | |---|---| | `https:///` | Landing page | | `https:///scout` | Scout UI. In mock mode only the example questions work | | `https:///dashboard` | Ratchet dashboard | | `https:///ratchet/stats` | JSON stats | | `https:///healthz` | `{"status":"ok",...}` (always public) | | `https:///readyz` | 200 with `"status": "ready"` (or `degraded`, with the reason under `checks`); 503 `not_ready` when the ledger cannot be written or no teacher can answer (always public; see [API.md](/reference/api/#get-readyz)) | | `https:///v1/models` | `{"object":"list","data":[{"id":"ratchet",...}]}` (403 if the Caddy guard is on, 401 if `RATCHET_API_KEYS` is set and you send no key; see Security) | ```bash curl -sN "https:///scout/stream?q=What%20is%20LoRA%20fine-tuning%3F" # events should arrive one by one ``` ## Switching from mock to live Only the root domain's gateway switches. The optional `ratchet-sim` container stays in mock mode by its own environment in `docker-compose.yml`, so it needs no change and holds no key. Live mode spends real money on every Scout run and every `/v1/chat/completions` call. Do it only after `python scripts/day1_check.py` passes. > **The public gateway should run in live mode.** Mock mode makes no Token Factory or Tavily call. Keep the VM running with the budget guard on, and make sure the Token Factory and Tavily credits last as long as the demo is meant to stay up. 1. In `.env`, set: - `RATCHET_MODE=live` - `NEBIUS_API_KEY` (Token Factory) and `TAVILY_API_KEY` - `RATCHET_STUDENT_BASE` and `RATCHET_STUDENT_KEY` only while the [vLLM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#vllm) endpoint is running. Leave both empty to use the hosted, untuned fallback student, which costs per token instead of per GPU-hour. Say so wherever you present it. With the endpoint off, the pages say the hosted Nano answers, at which threshold, and that no certificate covers it: student answers carry the "Nano 30B · hosted" badge, the landing strip and Scout say "Answered by the student", the dashboard header reads "Nano 30B · hosted on Token Factory (endpoint off)", and the live version's golden precision is hidden (`student_serving` in `/ratchet/stats`). `curl -s https:///ratchet/stats | jq .student_serving` shows what they read. - **Protect the paid API (`/v1/*` and the Azure-style `/openai/*`).** Either `RATCHET_API_KEYS=me:$(openssl rand -hex 24)` (gateway API keys; add `RATCHET_API_KEY_DAILY_REQUESTS=200` for a daily cap), or **`RATCHET_API_GUARD=guarded`** with `RATCHET_ADMIN_TOKEN=$(openssl rand -hex 32)` (Caddy edge guard), or both. Without one of them, anyone can send unlimited paid requests to `/v1/chat/completions` or `/openai/deployments//chat/completions`. The gateway refuses to start in live mode without `RATCHET_API_KEYS`; if Caddy's guard is your only protection, also set `RATCHET_ALLOW_UNAUTHENTICATED=1` to say so, and the gateway then starts only with `RATCHET_API_GUARD=guarded` and a `RATCHET_ADMIN_TOKEN` of at least 32 characters, so the one switch alone never opens the paid API. - **Decide who sees the dashboard.** In live mode the dashboard API (`/ratchet/stats`, `/ratchet/cost`, the task list, the [BOMs](https://docs.ratchet.dualmindlab.tech/guides/glossary/#bom)) needs an admin or dashboard key unless `RATCHET_PUBLIC_DASHBOARD=1`. `.env.vm.example` sets `1`, so judges see the live dashboard; anonymous readers then get client pseudonyms (`client-1`, ...) and events without URLs or error messages, while a request with your key gets everything. For a private dashboard set `0` and give yourself `RATCHET_DASHBOARD_KEYS=me:$(openssl rand -hex 24)`: read `/ratchet/stats` with it (`curl -H "Authorization: Bearer " ...`); the page itself sends no key yet and shows no data. `RATCHET_PUBLIC_BOM=1` keeps the [BOMs](https://docs.ratchet.dualmindlab.tech/guides/glossary/#bom) public either way. `/docs` and `/openapi.json` are off in live mode (`RATCHET_OPENAPI=1` turns them on). - Optional, for signed adapter BOMs: run `python -m ratchet.bom keygen` on your laptop, put the `RATCHET_BOM_SIGNING_KEY=` line in `.env`, and publish the public key it prints (for example in the README). This also works in mock mode, so you can check signed BOMs before going live. 2. Set the **budget guard** to what you can afford: - `RATCHET_SCOUT_DAILY_BUDGET_USD` caps Scout's Ratchet spend per UTC day (default 3.00). Over 48 days the worst case is the cap × 48, so 3.00 can reach **$144**. The Builder Program credit is $25 ([DEPLOYMENT.md](/reference/deployment/#step-1-token-factory-key-and-checks)), so **0.50/day ($24 worst case)** fits inside it. - `RATCHET_SCOUT_RUNS_PER_HOUR` (default 6) limits each client IP. - `RATCHET_SCOUT_MAX_RUNS_PER_DAY` (default 30) caps runs per UTC day for everyone, and with it Tavily credits (about 3 per run). - `RATCHET_AUDIT_RATE` (default 0.05): each audit is an extra Ultra call. Keep it at 0.05 or lower. - The counters are kept in the ledger and restored at startup, so `update.sh`, a crash or a recreate no longer resets the daily cap. The first restart after upgrading misses the runs from earlier that day, once. 3. Hold part of the day back for **judges**, and set the **Tavily credit budget** (the commented lines in `.env.vm.example`; details in [CONFIGURATION.md](/reference/configuration/#judge-access-and-tavily-credits)). The live gateway is public, so without a reserve any visitor can use up the day's runs before an evaluator arrives. An example that keeps runs and credits consistent: ```bash RATCHET_SCOUT_JUDGE_CODE= # share it with evaluators directly, never publish it RATCHET_SCOUT_MAX_RUNS_PER_DAY=22 RATCHET_SCOUT_JUDGE_RUNS_PER_DAY=7 # the public gets 15 runs a day RATCHET_TAVILY_MONTHLY_CREDITS=1000 # the free plan; more if the Builders Program adds credits (unverified) RATCHET_TAVILY_PACE_UNTIL=2026-12-15 # spread what is left until the last public day ``` - Credit math: Tavily's free plan gives **1,000 credits a month**, and a basic search costs 1 credit ([Tavily credits](https://docs.tavily.com/documentation/api-credits), checked Oct 20, 2026). The Nebius Builders Program may offer more (unverified; set the real number). Scout makes up to 3 searches a run. - The judges' credits default to `floor(1000 × 7 / 22) = 318`. In a calendar-month cycle that leaves `floor(682 / 15) = 45` public credits a day on Dec 1, which is 15 public runs of 3 searches; leftovers roll forward. The judges' budget defaults to their share of `RATCHET_SCOUT_DAILY_BUDGET_USD` (7/22 of $0.50 is about $0.16). - Tavily does not document its monthly reset day (unverified): set `RATCHET_TAVILY_CYCLE_DAY` to the day the Tavily dashboard shows. - When the public's credits for the day are used up, public runs still run, without web search and labelled "search unavailable": the planner has already called Token Factory, and the writer answers without sources. Judge runs keep searching from their reserve. 4. Check, then apply: `sudo -u ratchet docker compose run --rm --no-deps -T ratchet python -m ratchet.config check` lists every problem with `.env` at once (exit 2: the gateway would not start; it also warns that `RATCHET_MOCK_WARMUP_DAYS` does nothing in live mode, so remove that line), then `sudo -u ratchet docker compose up -d` (or `update.sh`, which runs the same check first) and `curl -s https:///readyz` should say `ready`. The ledger now lives in the `ratchet-data` volume at `/data/ratchet.db` and survives restarts and rebuilds. 5. Watch the Token Factory usage page and the Tavily dashboard daily for the first week, and check `curl -s https:///scout/status | jq .search` shows `"available": true`. ### Record one Scout run after each deploy A recorded run is evidence only: the gateway does not serve it. After the VM is live (and after each deploy that changes a model, a prompt or a price), you can record one run and commit it: ```bash python -m ratchet.scout record "What is LoRA fine-tuning?" --base-url https:// --judge-code git add evidence/scout && git commit -m "Record a live Scout run" ``` The command refuses a run that errors, does not finish or comes from a mock-mode gateway. The judge code is sent once in a POST body and is not written to the file. ### Night School cron (off by default) `/etc/cron.d/ratchet` contains two commented lines: `python -m ratchet.live night` at 02:30 UTC and `... tick` every 5 minutes, both through `docker compose exec -T ratchet`. Leave them off for the judging period unless all of these hold: - live mode works and the student endpoint and training job have been run by hand ([DEPLOYMENT.md](/reference/deployment/) steps 1–5); - the gateway image contains the `nebius` and `aws` CLIs, their credentials and the `deploy/` scripts. The stock [`Dockerfile`](https://github.com/harshbhx/ratchet/blob/main/Dockerfile) does not, so `night` cannot submit a training job from inside the container yet. To turn them on, remove the leading `# ` from those two lines with `sudo nano /etc/cron.d/ratchet`. Logs go to `/var/log/ratchet/`. ## Keeping it alive ### Cost estimate Prices from [Compute pricing](https://docs.nebius.com/compute/resources/pricing) and [nebius.com/prices](https://nebius.com/prices), as of **Oct 20, 2026**, USD, before tax. The public period is taken as 48 days (1,152 hours). Nebius prices disks per GiB per 730 hours, which is one month. | Item | Rate | Per month (730 h) | Judging period (48 days) | |---|---|---|---| | VM `cpu-e2` `2vcpu-8gb` (eu-north1) | $0.012/vCPU-h + $0.0045/GiB-h = **$0.060/h** | $43.80 | **$69.12** | | or VM `cpu-d3` `2vcpu-8gb` (other regions) | $0.015/vCPU-h + $0.0045/GiB-h = $0.066/h | $48.18 | $76.03 | | Boot disk, 30 GiB network SSD | $0.071 per GiB per 730 h | $2.13 | $3.36 | | Public IP | Free | $0 | $0 | | Network egress from the VM | Free | $0 | $0 | | Backups in Object Storage (Standard) | $0.0147 per GiB-month | < $0.01 | < $0.01 | | **Infrastructure total** | | | **≈ $72.50** (`cpu-e2`) or ≈ $79.40 (`cpu-d3`) | Notes: - The CPU rates above took effect on **Oct 1, 2026**. Before that they were $0.012/vCPU-h and $0.0032/GiB-h. Check the pricing page again before you rely on these numbers. - The VM bills while it runs. A stopped VM keeps billing for its disk; the static IP is kept while stopped. **Whether the gateway fits is unverified.** That the live gateway fits in 2 vCPU / 8 GB under judging load, or on a smaller VM, is **not verified**: measure it on day 1 with `docker stats --no-stream` and a few live requests. If it does not fit, `RATCHET_VM_PRESET=4vcpu-16gb` costs $0.120/h on `cpu-e2` ($0.048 for 4 vCPUs plus $0.072 for 16 GiB), $138.24 for the 1,152 hours of Oct 29 to Dec 15 against $69.12 now (`cpu-d3`: $0.132/h, $152.06; list rates checked Oct 21, 2026), or set `RATCHET_SIM=off`. Usage costs in live mode (on top of the infrastructure): | Item | How it is bounded | Expected | Worst case | |---|---|---|---| | Token Factory (Scout) | `RATCHET_SCOUT_DAILY_BUDGET_USD` per UTC day | A Scout run makes 1 planner + up to 12 triage + up to 6 reader + 1 writer call. With list prices in [CONFIGURATION.md](/reference/configuration/#prices-default_prices) that is roughly $0.02–0.03 per run before the student is trained, and under $0.01 after. 10–30 runs a day is about **$0.10–0.90/day** | cap × 48 days: $24 at 0.50, $144 at 3.00 | | Token Factory (`/v1/chat/completions`, `/openai/deployments//chat/completions`) | **Not covered by the Scout budget.** Gateway API keys with `RATCHET_API_KEY_DAILY_REQUESTS`, or the edge guard | $0 with `RATCHET_API_GUARD=guarded` or keys you do not share | Keys: the daily quota × your keys. Unbounded with both off | | Tavily search | 3 basic searches per run, 1 credit each ([Tavily credits](https://docs.tavily.com/documentation/api-credits)); `RATCHET_TAVILY_MONTHLY_CREDITS` with daily pacing | 1,000 free credits per month ≈ 330 runs a month | `RATCHET_TAVILY_MONTHLY_CREDITS` a cycle when set; otherwise `RATCHET_SCOUT_MAX_RUNS_PER_DAY` (30) × 3 = 90 credits a day; beyond the free credits, $0.008 per credit on pay-as-you-go | | Student endpoint (L40S) | Only while running | $0 if you use the hosted fallback | ~$32 per full day if left on ([DEPLOYMENT.md](/reference/deployment/#4-cost-estimates)) | To keep Tavily at $0, stay on the free plan without pay-as-you-go: once the 1,000 monthly credits are used, Tavily answers 432 and charges nothing. Scout then pauses searching for an hour (a `tavily_exhausted` event) and every run still finishes, labelled "search unavailable", with the planner's and the writer's Token Factory calls. With `RATCHET_TAVILY_MONTHLY_CREDITS` set, the meter usually stops before Tavily has to. ### Monitoring ```bash cd /opt/ratchet/deploy/vm sudo -u ratchet docker compose ps # all three "running", the gateways "healthy" sudo -u ratchet docker stats --no-stream # memory and CPU of ratchet and, if on, ratchet-sim (capped at 1 CPU, 1 GiB) sudo -u ratchet docker compose logs -f --tail=100 ratchet sudo -u ratchet docker compose logs -f --tail=100 caddy # certificate problems show here curl -s https:///scout/status # live mode: spend today vs budget, judges' reserve, search and credits ``` - `/scout/status` keeps today's runs and spend across `update.sh`: the counters are restored from the ledger. `search.available` false, a `search_error` event or a `tavily_exhausted` event: see [RUNBOOK.md](/reference/runbook/#tavily-credits-running-low-or-search-unavailable). - One gateway process per ledger: the compose file pins `--workers 1`. Detection of a second process (`gateway_lease`) waits for the leases table; until then see [RUNBOOK.md](/reference/runbook/#two-gateway-processes-on-one-ledger). - The live page's strip "Last Token Factory call" and the dashboard header read `/ratchet/stats` `upstream`: the newest real call per role, with its model id, latency and age, never content. An old age means no traffic since ([RUNBOOK.md](/reference/runbook/#reading-the-last-call-strip)). - The ratchet container (and ratchet-sim, if on) has a Docker health check on `/healthz` every 30 seconds (public, so it keeps working when `RATCHET_API_KEYS` protects `/v1/*`). Containers restart automatically (`restart: unless-stopped`), also after a VM reboot. - Container logs rotate at 10 MB × 5 files each. - For an outside check, point a free uptime monitor at `https:///healthz` ("is the process up"; always public) and, for "can it serve", `https:///readyz` (200 while requests can be answered, 503 when they cannot). A keyword monitor on `https:///scout/status` for `"state":"failing"` catches failing live model calls. `scripts/demo_check.py https://` (`make demo-check URL=https:// ARGS=--chat`) checks what a visitor sees from your laptop. A load balancer in front of the VM should use `/readyz` too; keep restarts on `/healthz`, so a teacher outage never becomes a restart loop. `/ratchet/stats` needs a key when the dashboard is private. ### Watchdog Stage One needs a runtime Token Factory call from this VM until judging ends (Dec 15), so a hung gateway or exhausted credit must not go unnoticed. [`python3 -m ratchet.watch`](https://github.com/harshbhx/ratchet/blob/main/ratchet/watch.py) runs on the VM host from `/etc/cron.d/ratchet-watch` (cloud-init installs it; on an older VM: `sudo install -m 0644 -o root -g root /opt/ratchet/deploy/vm/ratchet-watch.cron /etc/cron.d/ratchet-watch`). It uses the host's own Python 3 and no packages, so it is safe on a 1 GB VM. ADR: 0132. - **`tick`, every 2 minutes.** Asks the container's `/healthz` (through `docker compose exec`, which works while uvicorn is hung) and `/readyz`'s `checks.ledger`. After 3 failed liveness checks, or 3 failed ledger checks, in a row it runs `docker compose restart ratchet` (a start grace of 240 s, a 10 minute cooldown, at most 2 restarts an hour, never while `update.sh` runs or after `pause`). **It never restarts for the teacher or the student:** `/readyz` answers 503 when Token Factory is down, and a restart would fix nothing and drop the breakers' state. It never starts a container you stopped on purpose; it alerts instead. - **`probe`, hourly (live mode only).** One 16-token completion through `https:///v1/chat/completions` with the `watchdog` key on the pinned task `watchdog-probe`: always the teacher, never stored, about $0.0001 and at most 24 calls a day. It alerts on any non-200 answer, naming the gateway's error code and the upstream status (`upstream_402`, `upstream_auth` for a teacher 401 or 403, `upstream_rate_limited`, ...). **Which status Token Factory returns for exhausted credit is unverified**: the alert says "may be exhausted credit or a suspended account" and the Token Factory console decides. It sets the `live_check` flag on `/scout/status` and reads Scout's search state from it for free. - **Alerts** go to `RATCHET_ALERT_WEBHOOK_URL` as a signed JSON body (the gateway's own alerting is not built; this module sends its own). One alert when a check starts failing or changes code, a repeat every 6 h, one on recovery. Set up, once, on the VM: 1. Add a `watchdog:` entry to `RATCHET_API_KEYS` (`openssl rand -hex 24`), and, to cap it, `RATCHET_KEY_LIMITS` (room for 24 calls a day). 2. Set `RATCHET_TASKS_FILE=/config/tasks.json` in `.env` (compose mounts `deploy/vm/config` there read-only). It pins `watchdog-probe` for the `watchdog` client only; setting it without the `watchdog` key stops startup, by design. 3. Set `RATCHET_ALERT_WEBHOOK_URL` (and `RATCHET_ALERT_WEBHOOK_SECRET`), then `cd /opt/ratchet && python3 -m ratchet.watch test-alert`. Slack takes the body's `text`; whether it ignores the other keys, and Discord with `/slack` appended, are unverified. 4. Optionally set `RATCHET_WATCH_HEARTBEAT_URL` to a dead-man's-switch URL (period 1 h, grace 30 min): it alerts when the VM or cron is dead, which nothing on the VM can report. 5. `python3 -m ratchet.watch probe --once --force` after `update.sh`, then read `python3 -m ratchet.watch status`. If the probe reports `unreachable` or `probe_timeout` from the VM while the site works from outside, the VM may not reach its own public IP (hairpin NAT, unverified on Nebius): set `RATCHET_WATCH_CONNECT=local`. Commands: `status`, `pause --minutes N` (no restarts for N minutes, up to 1,440), `resume`, `test-alert`, `probe --once --no-flag --no-alert`. Switch it all off with `RATCHET_WATCH=off`. Subtract the probe's tokens from your credit estimate: about 24 calls a day. ### Updating ```bash sudo -u ratchet /opt/ratchet/deploy/vm/update.sh # flags: --skip-backup --no-rollback --no-probe DRY_RUN=1 /opt/ratchet/deploy/vm/update.sh # prints every command that changes something: rehearse it first ``` `update.sh` first takes a lock (the watchdog never restarts while it is held), saves the running git SHA, makes a ledger backup tagged `pre-update` (live mode) and tags the running image `ratchet-gateway:previous` (a tagged image survives `docker image prune -f`). If the new version is unhealthy, answers `/readyz` with a broken ledger, reports the wrong mode, or fails the live probe on the gateway side, it moves the checkout (`git reset --hard` to the old SHA; your `.env` is untouched) and the image back and recreates the container, then alerts. If Token Factory is the problem, it does not roll back: an older image would not fix it. It restores the ledger from the pre-update backup (`restore.sh`; requests logged after the backup are lost; the replaced file stays as `ratchet.db.before-restore-
`, an unterminated `` that opens what is left and what follows), and for a structured task such as `ticket-triage` a code fence around the whole JSON answer is removed, so the content is bare JSON. A structured answer that is one JSON object is returned whole, so a tag quoted in its strings stays; a `` after answer text is content too. A free-text answer keeps its code blocks. With a `response_format` the content is the JSON the answer holds ([Structured outputs](#structured-outputs)). A teacher answer that was cut off by its token limit is still returned (and is not stored as training data), with `finish_reason: "length"`. `finish_reason` is the served answer's: `stop`, `length` (cut off by `max_tokens` or a cap, or over the client's `max_tokens` with the reasoning headroom), `content_filter`, or `tool_calls` (`function_call` for a legacy request) when the message has tool calls ([Tools and function calling](#tools-and-function-calling)); any other value an upstream model reports is returned as `stop`, and `tool_calls` is never sent without `message.tool_calls`. A student answer is always one that ended by itself. Response headers: `X-Ratchet-Served-By: student|teacher`, `X-Ratchet-Request-Id` (the ledger row, as `ratchet.row_id`), `X-Ratchet-Task: `, `X-Ratchet-Degraded: teacher_unavailable` on a degraded answer (and only then; streamed responses too), `X-Ratchet-Model-Alias: ` when `RATCHET_MODEL_POLICY=lenient` served a `model` Ratchet does not know as `ratchet` (UTF-8 percent-encoded except for visible ASCII, so spaces and `%` are encoded too and one percent-decoding gives the id back; at most 256 characters), plus `X-RateLimit-Limit-Requests` and `X-RateLimit-Remaining-Requests` when a daily quota is set, `Idempotent-Replayed: true` on an answer sent again for an `Idempotency-Key` ([Idempotency and retries](#idempotency-and-retries)), and on every response `x-request-id` and `openai-processing-ms` ([Request ids](#request-ids)). #### Request ids Every request gets an id from the gateway, `req_` and 32 hex digits (a UUID4), returned as `x-request-id` on every response, errors and streams included, beside `openai-processing-ms` (the gateway's time until the response started, in whole ms), the two headers OpenAI's API sends. OpenAI's SDKs read the header as `response._request_id` and `error.request_id`. The same id is `ratchet.request_id`, `request_id` in every error envelope on `/v1/*`, the ledger's `requests.request_id` and `failures.request_id`, and the request's log line; the completion's `id` is `chatcmpl-` and the same digits. Look a request up with `GET /v1/ratchet/requests/{id}` (an API client, its own requests) or `GET /ratchet/requests/{id}` (the dashboard API); either takes the `req_` id or the `chatcmpl-` id. Your own id: send `X-Request-Id` (or `X-Client-Request-Id`, which OpenAI's API also takes) with 1 to 128 letters, digits, `.`, `_` or `-`, and it is stored as the request's `client_request_id` and logged beside the gateway's id. Any other value is ignored, never an error. The gateway's id is always its own; yours never replaces it. Tracing: a valid W3C `traceparent` (version `00`, lower-case hex, no all-zero ids) is accepted, its trace id is logged with the request, and every call to the teacher and the student (each HTTP attempt) carries it with a new span id and the same flags, with `tracestate` passed on when it is printable ASCII of at most 512 characters. A malformed `traceparent` is ignored. A request without one starts no trace. #### Usage `usage` is the served answer's, as if the model that wrote it had answered the request alone: | Field | Student answer | Teacher answer | |---|---|---| | `prompt_tokens` | The student's prompt tokens | The teacher's prompt tokens | | `completion_tokens` | The chosen sample's tokens (one of the samples; the endpoint's `completion_tokens` split over them by their logprob counts); with `n` > 1, every returned choice's, added up | The teacher's completion tokens, reasoning included, as OpenAI counts them for its reasoning models (every choice's, for `n` > 1) | | `total_tokens` | `prompt_tokens + completion_tokens` | The same | | `completion_tokens_details.reasoning_tokens` | The sample's share of the endpoint's reasoning tokens (0 for the tuned student), added up over the choices | What the teacher reported (Token Factory reports them; the mock teacher reports `RATCHET_TEACHER_REASONING_TOKENS`) | | `prompt_tokens_details.cached_tokens` | What the endpoint reported in `prompt_tokens_details.cached_tokens`, else 0 | The same | | `audio_tokens`, `accepted_prediction_tokens`, `rejected_prediction_tokens` | 0 | 0 | When the served model reported no prompt or completion count, Ratchet estimates it at about 4 characters a token (the messages as JSON; the answer as served), so `total_tokens` is never 0. Reasoning and cached tokens never exceed the counts they are part of. `usage` is **not** what the request cost Ratchet. The other student samples, an audit, a shadow, an [arbitration](https://docs.ratchet.dualmindlab.tech/guides/glossary/#arbitration) and a schema repair's unused answer are listed in `ratchet.usage_breakdown`, and `ratchet.cost_usd` prices all of them. After a repair, `prompt_tokens` and `cached_tokens` are the request's own prompt (the first call's), not the repair call's, whose prompt also held the rejected answer and the repair note ([Structured outputs](#structured-outputs)). So the same answer reports the same `usage` whether or not it was audited or shadowed, and a client that bills its own users from `usage` charges them for the answer they got. #### Choices and logprobs `n` asks for up to `RATCHET_MAX_N` (4) choices; `choices[i].index` numbers them from 0. - **The student** draws `max(RATCHET_SAMPLES, n)` samples in one call. Its confidence is scored on the first `RATCHET_SAMPLES` of them, exactly as for one choice, and it serves only when that confidence reaches the version's threshold **and** at least `n` samples give the majority answer. Then the choices are those samples: choice 0 is the one a request for one choice would get, the others follow by token probability. So all choices of a student answer agree on the task's label fields (`category` and `priority` for `ticket-triage`), and a request for more choices escalates more often than one for a single choice. - **Otherwise the teacher** writes `n` choices, at temperature 0.7 when the request sets none (at its default of 0 every choice would be the same). The route is `n_choices` when the student was confident but too few of its samples agreed, `confidence` when it was unsure. A teacher endpoint that returns fewer choices than asked gives fewer; Ratchet never pads them with copies. - **Only choice 0 is ever stored** as a training example, so one request adds or bumps at most one example. Audits, shadow comparisons and arbitrations judge choice 0. - `usage` adds up every choice's completion (the prompt once), `usage_breakdown.served` lists the same choices, and `cost_usd` prices every call. `message.content` is cleaned per choice as above, and `finish_reason` is each choice's own. With `"logprobs": true`, each choice has `logprobs: {"content": [...], "refusal": null}`, in OpenAI's format: one entry per token, `{"token", "logprob", "bytes", "top_logprobs"}`, with at most `top_logprobs` (0 to 20; none when not sent) alternatives per token. Without `logprobs`, or with `false`, `logprobs` is `null` as before. ```json "logprobs": {"content": [ {"token": "{\"category\"", "logprob": -0.012, "bytes": [123, 34, 99, 97, 116, 101, 103, 111, 114, 121, 34], "top_logprobs": [{"token": "{\"category\"", "logprob": -0.012, "bytes": [123, 34, 99, 97, 116, 101, 103, 111, 114, 121, 34]}]}, … ], "refusal": null} ``` - The tokens always spell `message.content` exactly. When reasoning (`…`), a fence around a structured answer, or whitespace at its edges was removed from the content, their tokens are removed too, and a token is cut only where the part removed is whitespace. Tokens a server lists for reasoning that its content already leaves out are removed the same way, so reasoning never reaches a client through logprobs. - `content` is `null` when the logprobs cannot be given: the model returned none (an operator can leave `logprobs` out of `RATCHET_TEACHER_PARAMS` for a teacher that refuses it), an entry was malformed, or the answer cannot be told apart from what was removed (a token holding both the end of the reasoning and the start of the answer, reasoning removed from the middle of an answer, or a character split over tokens that do not decode alone). Treat `null` as possible whenever you ask for logprobs. - A student answer has them whenever its endpoint returns them (the student is sampled with logprobs anyway); a teacher answer has them when the teacher returns them, and the teacher is asked for `logprobs` and `top_logprobs` only when the request sends them. A logprob of `-9999.0` stands for minus infinity, as [vLLM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#vllm) reports it (JSON has no `-Infinity`). The mock student returns tokens of about 4 characters with one logprob per sample; the mock teacher returns none (`content: null`). - Logprobs are the served model's own: a student choice's are the student's, a teacher choice's the teacher's. `exp` of a student choice's mean token logprob is the token probability its confidence score used. ### Streaming With `"stream": true` the response is `text/event-stream` in the format OpenAI SDKs expect: a first chunk with `delta.role`, content chunks of a few words each, a last chunk with the answer's `finish_reason` (`"stop"`, or `"length"` as above) and the same `ratchet` block as above, then `data: [DONE]`. Every chunk has the response's `id`, `created`, `model`, `system_fingerprint` and `service_tier`, and `logprobs: null` in its choice. With `n` > 1 the choices follow one another, each with its own `index`, role chunk, content chunks and finish chunk; the `ratchet` block rides on the last finish chunk. With `logprobs: true`, each content chunk carries a few tokens and their `logprobs: {"content": [...], "refusal": null}` (so the chunks' logprobs add up to the non-streamed response's; `content: null` when they cannot be given), and the role and finish chunks keep `logprobs: null`. ```text data: {"id":"chatcmpl-…","object":"chat.completion.chunk","created":1792400000,"model":"student-v5","system_fingerprint":"fp_6c1e0f3a9b2d","service_tier":"default","choices":[{"index":0,"delta":{"role":"assistant","content":"","refusal":null},"logprobs":null,"finish_reason":null}]} data: {"id":"chatcmpl-…","object":"chat.completion.chunk",…,"choices":[{"index":0,"delta":{"content":"{\"category\": \"billing\", "},"logprobs":null,"finish_reason":null}]} data: {"id":"chatcmpl-…","object":"chat.completion.chunk",…,"choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"ratchet":{"served_by":"student",…}} data: [DONE] ``` With `"stream_options": {"include_usage": true}`, as OpenAI documents it: every chunk above has `"usage": null`, the finish chunk has no `ratchet` block, and one more chunk comes before `[DONE]`, with no choices, the same `usage` as the non-streamed response and the `ratchet` block: ```text data: {"id":"chatcmpl-…","object":"chat.completion.chunk",…,"choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"stop"}],"usage":null} data: {"id":"chatcmpl-…","object":"chat.completion.chunk",…,"choices":[],"usage":{"prompt_tokens":104,"completion_tokens":31,"total_tokens":135,…},"ratchet":{"served_by":"student",…}} data: [DONE] ``` A choice with tool calls ([Tools and function calling](#tools-and-function-calling)) sends, after its role chunk (`"content": null` when it has no text) and any content chunks, one chunk per call with the whole call, then its finish chunk with `"tool_calls"`. This is what the OpenAI SDK's `ChatCompletionStreamState` (behind `client.chat.completions.stream`) and LangChain's chunk accumulators merge by `index`. A legacy request gets one `{"function_call": {"name", "arguments"}}` delta and `"function_call"`. ```text data: {…,"choices":[{"index":0,"delta":{"role":"assistant","content":null,"refusal":null},"logprobs":null,"finish_reason":null}]} data: {…,"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_paris01","type":"function","function":{"name":"get_weather","arguments":"{\"city\": \"Paris\"}"}}]},"logprobs":null,"finish_reason":null}]} data: {…,"choices":[{"index":0,"delta":{},"logprobs":null,"finish_reason":"tool_calls"}],"ratchet":{"served_by":"teacher",…}} data: [DONE] ``` The cascade has to see the student's whole answer to score its confidence before it decides who serves, so the answer is complete before the first chunk is sent: time to first token equals the full latency. Streaming exists so that clients built for streaming work unchanged. Errors (400, 401, 413, 429, 502, 504) are returned as normal JSON HTTP errors before the stream starts. ```python for chunk in client.chat.completions.create(model="ratchet", messages=msgs, stream=True, stream_options={"include_usage": True}): if chunk.choices: print(chunk.choices[0].delta.content or "", end="") else: print("\n", chunk.usage.total_tokens, "tokens") ``` ### Errors Every error on `/v1/*` (and `/openai/*`) has OpenAI's envelope, which OpenAI SDKs turn into typed exceptions (`openai.BadRequestError`, `AuthenticationError`, `NotFoundError`, `RateLimitError`, `InternalServerError`, …) with `.code`, `.param` and `.type`: ```json { "error": {"message": "Invalid value for 'temperature': expected a number from 0 to 2, but got 3 instead.", "type": "invalid_request_error", "param": "temperature", "code": "invalid_value"}, "detail": "Invalid value for 'temperature': expected a number from 0 to 2, but got 3 instead.", "request_id": "req_0b6f3a7c2d1e4f5a8b9c0d1e2f3a4b5c" } ``` `detail` repeats `error.message` until the release after v0.5, for clients written against earlier versions, then goes away: read `error`. `request_id` (since v0.5) is the request's id, also in the `x-request-id` header; `error` keeps exactly OpenAI's four fields. Routes outside `/v1` (`/ratchet/*`, Scout, pages) keep FastAPI's `{"detail": ...}`, with the id in `x-request-id`. | Status | `type` | `code` | When | Retried by OpenAI SDKs? | |---|---|---|---|---| | 400 | `invalid_request_error` | `invalid_json`, `invalid_type`, `invalid_value`, `missing_required_parameter`, `unsupported_value`, `context_length_exceeded`, `unsupported_content_type` | The request breaks a rule in [Request validation](#request-validation); `param` names the field | No | | 400 | `invalid_request_error` | `null` | Mock mode with `RATCHET_MOCK_STRICT=1` only: the request is not a synthetic test ticket or a Scout request from the mock research world (`mock mode only answers the mock world's own requests: '...'`). Without it the generic mock world answers every request ([Mock mode](#mock-mode-answers)) | No | | 400 | `invalid_request_error` | `reserved_task` | The request names the task `embeddings` (header, `metadata.ratchet_task`, `model` or Azure deployment), the name `/v1/embeddings` logs its requests under ([Embeddings](#post-v1embeddings)); `param` says where it came from. Checked before the quota | No | | 400 | `invalid_request_error` | `unknown_task` | `RATCHET_UNKNOWN_TASKS=reject` and the task is not registered (`unknown task "x"; registered: default, page-read, ...`, at most 20 names, only tasks this key may use); `param` is `metadata.ratchet_task` when the name came from there, and `response_format` for a json_schema request that named no task and whose schema gave no auto task ([Structured outputs](#structured-outputs)) | No | | 401 | `invalid_request_error` | `invalid_api_key` | API keys are on and the key is missing or wrong (`WWW-Authenticate: Bearer`) | No | | 403 | `permission_error` | `task_not_allowed` | The task's `clients` do not name this key, or the request has no key, or the key's `tasks` scope does not list the task (`task not allowed for this key`). Checked before the quota, so it uses none | No | | 404 | `not_found_error` | `unknown_url` | No such route: `Unknown request URL: POST /v1/foo.` | No | | 404 | `not_found_error` | `unsupported_endpoint` | An OpenAI endpoint this gateway knows but does not serve: `POST /v1/moderations`, a stored response (`/v1/responses/{id}`), or `/v1/embeddings` under `RATCHET_EMBEDDINGS_BASE=off`. The message says why and links [OpenAI endpoints](#openai-endpoints). Checked after the key | No | | 400 | `invalid_request_error` | `unsupported_parameter` | A field the gateway understands but cannot honour, so serving without it would answer another question: `suffix`, `echo` or `best_of` on `/v1/completions`; `previous_response_id`, `conversation`, `prompt`, `background`, `stream` or a built-in tool on `/v1/responses`. `param` names it | No | | 404 | `invalid_request_error` | `model_not_found` | `RATCHET_MODEL_POLICY=strict` and the `model` is not `ratchet`, `ratchet/` or an alias (`The model 'gpt-4o' does not exist.`, `param` `model`); under either policy, `ratchet/` with a name that is not a task name; on the Azure-style route, a deployment that is not a task name (`param` null). Checked before the quota, so it uses none and nothing is logged | No | | 405 | `invalid_request_error` | `null` | A route called with the wrong method (`Allow` names the right one) | No | | 413 | `invalid_request_error` | `request_too_large` | The body is larger than `RATCHET_MAX_BODY_BYTES` | No | | 400 | `invalid_request_error` | `invalid_idempotency_key` | The `Idempotency-Key` header is empty, longer than 255 characters or not printable ASCII (`Invalid Idempotency-Key header: ...`); `X-Should-Retry: false` ([Idempotency and retries](#idempotency-and-retries)) | No | | 400 | `invalid_request_error` | `idempotency_key_reused` | This API key used the `Idempotency-Key` for a different request (another body or `X-Ratchet-Task`) within `RATCHET_IDEMPOTENCY_TTL_S`; `X-Should-Retry: false` | No | | 409 | `invalid_request_error` | `idempotency_key_in_use` | The first request with this `Idempotency-Key` was still running after `RATCHET_TEACHER_DEADLINE_S`, or four requests were already waiting for it; `Retry-After: 1`. The retry gets its answer once it finished | Yes, after `Retry-After` | | 400 | `invalid_request_error` | the teacher's (`context_length_exceeded`, ...), `request_too_large` for an upstream 413 without one, or `null` | Live mode: the teacher refused the request with 400, 413 or 422, so it cannot succeed as sent (too long for the model, a content part it cannot read). `message` and `param` are the teacher's; `X-Should-Retry: false` | No | | 429 | `insufficient_quota` | `daily_quota_exceeded` | The key's daily quota is used up (`daily request quota of N used up for 'acme'; resets at 00:00 UTC`); `Retry-After` (seconds to 00:00 UTC), `X-RateLimit-Reset-Requests` and `X-Should-Retry: false` | No: the response tells them not to | | 429 | `insufficient_quota` | `budget_exceeded` | The key's daily or monthly US dollar budget is spent (`the daily budget of $5.00 is spent for 'acme'; it resets at 00:00 UTC`); `Retry-After` and `X-Ratchet-Budget-Reset` (until the cap that resets last resets), `X-Ratchet-Budget-Remaining-USD: 0.000000` and `X-Should-Retry: false` ([Limits and headers](#limits-and-headers)) | No: the response tells them not to | | 429 | `insufficient_quota` | `gateway_budget_exceeded` | The gateway's daily budget (`RATCHET_DAILY_BUDGET_USD`) is spent: every chat request with `RATCHET_BUDGET_MODE=block`, only a request that needs the teacher with `student_only`; `Retry-After` (to 00:00 UTC) and `X-Should-Retry: false` | No: the response tells them not to | | 429 | `rate_limit_error` | `rate_limit_exceeded` | The key's requests a minute or in flight, or on a gateway without keys the client address's requests a minute, are over their limit; `Retry-After`, and for a rate the `X-RateLimit-*-Requests` headers ([Limits and headers](#limits-and-headers)) | Yes, after `Retry-After` | | 429 | `rate_limit_error` | `upstream_rate_limited` | Live mode: Token Factory rate-limited the teacher call, after the gateway's own retries (`teacher rate limited: upstream 429; retry after 7 s`). `Retry-After` is the teacher's wait, clamped to 1 to 60 s (1 when it sent none) | Yes, after `Retry-After` | | 500 | `api_error` | `null` | A bug in the gateway (`The server had an error while processing your request.`). The message never includes request text; the ledger records a `gateway_error` event with the exception's class at the 1st, 10th, 100th, ... occurrence. In live mode this includes a `KeyError` from the cascade, which is a 400 only in strict mock mode (`RATCHET_MOCK_STRICT=1`) | Yes | | 502 | `api_error` | `teacher_unavailable` | Live mode: a teacher call whose answer was needed failed after its retries: an escalation, a cold start, a pinned task or a teacher-only request (`teacher unavailable: upstream 503`, `teacher unavailable: upstream 502 upstream_unreachable` when Token Factory could not be reached, `... bad_upstream_response` when its answer could not be read). Student failures do not cause errors: they fall back to the teacher. Neither do a failed audit or shadow arbitration: the student's answer is served. An upstream redirect or 4xx outside the other rows (402, for example) adds `X-Should-Retry: false`. When Token Factory sent a wait (`Retry-After` on a 503, for example), it is passed on as `Retry-After`, clamped to 1 to 60 s. See [RUNBOOK.md](/reference/runbook/#teacher-api-down-token-factory-errors-or-timeouts) | Yes (after `Retry-After` when present), unless `X-Should-Retry: false` | | 502 | `api_error` | `upstream_auth_error` | Live mode: Token Factory refused the gateway's own key (401 or 403): `NEBIUS_API_KEY` is wrong or lacks access. `X-Should-Retry: false` | No | | 502 | `api_error` | `upstream_not_found` | Live mode: Token Factory answered 404: `RATCHET_TEACHER_MODEL` or `RATCHET_TEACHER_BASE` is wrong. The request's `model` never selects the teacher, so this is the gateway's configuration, not the client's mistake. `X-Should-Retry: false` | No | | 502 | `api_error` | `tool_calls_unsupported` | Live mode: the teacher answered a request that offers no tools with a tool call (`The teacher answered with a tool call to a request that offers no tools.`). A request with tools gets the calls back ([Tools and function calling](#tools-and-function-calling)). Nothing is stored. `X-Should-Retry: false` | No | | 503 | `api_error` | `overloaded` | `RATCHET_MAX_INFLIGHT` chat requests are in flight (`the gateway has 32 chat requests in flight, its limit; retry in a second`): answered at once instead of queueing; `Retry-After: 1` | Yes, after `Retry-After` | | 503 | `api_error` | `teacher_circuit_open` | Live mode: the teacher failed `RATCHET_TEACHER_BREAKER_FAILURES` (5) calls in a row, so its circuit breaker is open and the gateway does not call it until the next probe (`teacher circuit open: recent teacher calls failed, so the gateway is not calling it; retry after 18 s`). Answered at once, for any request that needs the teacher's answer (an escalation, a cold start, a pinned task, a teacher-only request) that no fallback teacher or degraded answer covers. `Retry-After` is the time until the breaker lets a probe through, clamped to 1 to 60 s | Yes, after `Retry-After` | | 502, 504 | `api_error` | `upstream_unavailable`, `upstream_timeout` (and `upstream_auth_error`, `upstream_not_found`, `upstream_rate_limited` as above) | `/v1/embeddings` in live mode: the embeddings model failed after the provider's retries. Mapped by cause as a teacher failure is, with these codes, since no teacher was involved; an upstream 400 (an input over the model's length) is a 400 with its message and code | As for the teacher's | | 504 | `api_error` | `teacher_timeout` | Live mode: the teacher did not answer in time (`RATCHET_TEACHER_TIMEOUT_S`, retried once within `RATCHET_TEACHER_DEADLINE_S`), or Token Factory answered 504 (`teacher timeout: upstream 504 upstream_timeout`). Token Factory's `Retry-After`, if it sent one, is passed on (1 to 60 s) | Yes, after `Retry-After` when present | | 504 | `api_error` | `batch_deadline_exceeded` | `/v1/completions`: a batch of prompts ran for `RATCHET_TEACHER_DEADLINE_S` before its next prompt, so it stopped. The message names the prompts already answered, logged and billed (not returned); `X-Should-Retry: false`. Any other error of a prompt after the first also carries `X-Should-Retry: false` and that message ([Completions](#post-v1completions)) | No | OpenAI's Python and Node SDKs retry connection errors, 408, 409, 429 and 5xx responses twice by default, with backoff (or `Retry-After`, up to 60 s), unless the response carries `x-should-retry`. Set `max_retries=0` on the client to turn that off. Each retry of a request the student cannot answer runs the student's samples again, so the gateway marks the errors no retry can fix. The gateway retries the teacher itself before any of these errors ([CONFIGURATION.md](/reference/configuration/#upstream-timeouts-and-retries)): up to `RATCHET_TEACHER_RETRIES` (2) more attempts within `RATCHET_TEACHER_DEADLINE_S` (180 s), with full-jitter backoff or Token Factory's `Retry-After` (up to 8 s; a longer wait comes back to you at once, as the `Retry-After` of the 429, 502 or 504), on connection failures and 408, 409, 429, 500, 502, 503 and 504; a read timeout, a connection dropped mid-response or a 504 once at most in all, because the model may already have generated billed tokens. The student is never retried on the request path: a failing student escalates to the teacher at once. After 5 failed teacher calls in a row the teacher's breaker answers with the 503 above instead of waiting on Token Factory, and lets one request probe it every 30 s (doubling to 300 s while it keeps failing). Two opt-in settings answer instead of an error ([CONFIGURATION.md](/reference/configuration/#when-the-teacher-is-down)): a licensed fallback teacher (`RATCHET_TEACHER_FALLBACK_MODEL`, a 200 with its model id) and, for an escalated request, the student's own answer marked as degraded (`RATCHET_ON_TEACHER_FAILURE=serve_student`). Neither covers a 400 about the request or a tool-call answer. ```python try: client.chat.completions.create(model="ratchet", messages=msgs, temperature=3) except openai.BadRequestError as e: print(e.status_code, e.code, e.param) # 400 invalid_value temperature ``` ### Examples curl: ```bash curl -s http://localhost:8000/v1/chat/completions \ -H 'content-type: application/json' \ -H 'X-Ratchet-Task: ticket-triage' \ -d '{"model":"ratchet","messages":[ {"role":"system","content":"You triage customer support tickets. Reply with JSON only ..."}, {"role":"user","content":"I was charged twice for my Pro plan this month. Order #48213."}]}' ``` OpenAI Python SDK: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") # or your RATCHET_API_KEYS key resp = client.chat.completions.create( model="ratchet", messages=[{"role": "user", "content": "I was charged twice this month."}], extra_headers={"X-Ratchet-Task": "ticket-triage"}, ) print(resp.choices[0].message.content) print(resp.model_extra["ratchet"]) # served_by, confidence, cost_usd, ... ``` A client that can set only the model name (LangChain, a no-code tool) names the task there: ```python from langchain_openai import ChatOpenAI llm = ChatOpenAI(base_url="http://localhost:8000/v1", api_key="unused", model="ratchet/ticket-triage") ``` OpenAI Node SDK: ```js const client = new OpenAI({ baseURL: "http://localhost:8000/v1", apiKey: "unused" }); const r = await client.chat.completions.create( { model: "ratchet", messages: [{ role: "user", content: "Where is my order #48213?" }] }, { headers: { "X-Ratchet-Task": "ticket-triage" } }, ); ``` ### Mock mode answers In mock mode every request gets an answer from mock models (mock-mode run; nothing here measures a real model). The synthetic tickets generated by `sim/tickets.py` and Scout's requests for the example questions (`sim/research.py`) get their mock answers: use the mock-mode endpoints, take a ticket from `sim.tickets.TicketStream` in Python, or run the Scout CLI (`python -m ratchet.scout "What is LoRA fine-tuning?" --base-url http://localhost:8000/v1`). Any other request gets a deterministic answer from the generic mock world (`sim/generic.py`), the same every time for the same text: | The request | The mock teacher's answer | |---|---| | Plain text (`hello`) | `[mock teacher] You asked: "hello". ...`: a restatement of the prompt (at most 12 words) and one of four closing sentences | | `response_format` `json_schema` | JSON that validates against the schema: enum values and booleans drawn, integers and numbers within their bounds, `mock text` for strings (a date, time, UUID or URI for those formats), 1 or 2 array items, nested objects and local `$ref`s. The OpenAI SDK's `client.chat.completions.parse(response_format=SomeModel)` parses it | | `response_format` `json_object` | `{"answer": "[mock teacher] You asked: ..."}` | | Ending with a `tool` message | `[mock teacher] The tool returned: "...". ...` | | `tools` that reach the teacher | A tool call, returned as `message.tool_calls` ([Tools and function calling](#tools-and-function-calling)): `lookup_order(order_id)` or `refund(order_id, amount, reason)` (the mock catalogue, `sim.generic.TOOLS`) with the order number, amount and reason read from the prompt, or a call to your own tool with arguments built from its parameters schema. The call ids are stable, so the same request gets the same call. A request that carries tool turns or offers tools is answered by the mock teacher alone and never stored, as in live mode | The mock student learns these requests as it learns tickets, per schema name or system prompt (tool requests never reach it). `RATCHET_MOCK_STRICT=1` restores the earlier behaviour: anything but a test run request is a 400 `mock mode only answers demo requests`. --- ## End users and storage Every teacher answer of a request (an escalation, a cold start, an audit, a shadow arbitration) becomes training data unless an [opt-out](https://docs.ratchet.dualmindlab.tech/guides/glossary/#opt-out) applies. The broadest one that applies is recorded with the request (`requests.store_opt_out`) and returned in `ratchet.store_opt_out`: | Opt-out | Set by | Reason | |---|---|---| | Zero retention | `RATCHET_STORE_EXAMPLES=0` ([CONFIGURATION.md](/reference/configuration/#privacy-and-security)); Night School then skips every night | `global` | | A no-store task | `store: false` in the task's spec, or `RATCHET_NO_STORE_TASKS` | `task` | | A no-store API key | `store=0` for the key's name in `RATCHET_KEY_LIMITS` | `key` | | The request | `store: false`, `X-Ratchet-Store: none` (or `0`, `false`), or `metadata.ratchet_store` `"false"` (or `"0"`) | `request` | | A no-train end user | `POST /v1/ratchet/no-train` (below) | `end_user` | - An opted-out request is answered exactly as any other: the student may serve, the teacher still audits at the usual rate and still settles shadow disagreements, so release decisions never depend on what is stored. Nothing is inserted into the training set and no stored example's repeat count goes up; `ratchet.stored` is `false`. - An opt-out only ever removes storage. `store: true`, `X-Ratchet-Store: 1` or `metadata.ratchet_store: "true"` is the default spelled out and cannot undo another opt-out. An absent `store` means store (OpenAI's chat completions default to not storing; Ratchet exists to distill, so it defaults the other way). - Pinned tasks and teacher-only requests are never stored anyway; their rows still record the opt-out. Examples stored before an opt-out took effect stay until they are erased ([Erasure and export](#erasure-and-export)) or expire (`RATCHET_EXAMPLE_RETENTION_DAYS`). - **End users.** The request's `safety_identifier`, else its `user`, names its end user (any length, hashed whole; empty means none). Ratchet stores only HMAC-SHA256 of it under the deployment's key (`RATCHET_END_USER_KEY`, or a random key generated once in the ledger), scoped to the API key's name and cut to 16 hex digits, in the request row and in an example it stores. The same identifier sent by two API keys is two people as far as the ledger can tell, and two deployments cannot link their hashes. The raw value is never stored, logged or forwarded upstream. A request that sends both fields is attributed to `safety_identifier`; its `user` is only looked up for a no-train registration. - **Key names.** The scope is the key's *name* in `RATCHET_API_KEYS`, not the key: rotating a key under the same name keeps every hash, while renaming it starts new ones. A no-train registration scoped to a name that is no longer configured matches nothing (the gateway writes a `no_train_unmatched` event at startup and counts them in `/ratchet/stats` → `privacy.no_train_unmatched`) until the name is back or the end user registers again; an [erasure](https://docs.ratchet.dualmindlab.tech/guides/glossary/#erasure) can still name the old client. Keyless registrations match nothing while keys are on, the same way. - Examples also record the API key name of the request they were first stored from; a later duplicate from another key is recorded too, so a client's data can be found. ### POST `/v1/ratchet/no-train` ```bash curl -X POST localhost:8000/v1/ratchet/no-train -H "Authorization: Bearer $KEY" \ -H 'Content-Type: application/json' -d '{"end_user": "user-8812"}' ``` ```json {"object": "ratchet.no_train", "end_users": [{"client": "acme", "created": true}]} ``` From then on, requests from that end user are answered as usual but never stored (`store_opt_out` `end_user`), checked once at the start of each request and again just before anything would be stored (an opt-out found then is reported the same way). The value is any one the client sends as `user` or `safety_identifier`: a request is matched on each of the two it carries, so a client that registered `user` values and later also sends `safety_identifier` keeps its registrations. Only the hash is kept, and the response does not return it (the endpoint is not a way to compute hashes). Each entry names the client scope it was registered in (`null` for keyless requests, `"*"` for every client). - **Who.** Open wherever chat completions are: with `RATCHET_API_KEYS` set it needs an API key and registers that key's own end user; without keys it registers the end user of keyless requests. An admin key (`RATCHET_ADMIN_KEYS`) may add `"client": ""` to register that client's end user, or leave it out to register the end user for every client (scope `"*"`): every API key, also keys added later, and keyless requests, as for a data subject's request. - `created` is `false` when the end user was already registered in that scope. It is told only to an API key or an admin key: without API keys anyone may call this endpoint, so a keyless caller gets `null` either way and cannot learn whether an identifier was registered. A registration an admin makes cannot be removed by the client. - A client's registration is scoped to its key's name: renaming the key in `RATCHET_API_KEYS` unlinks it ([End users](#end-users-and-storage)); an admin's registration for every client does not depend on key names. - **Errors.** 400 for a body that is not an object, an unknown field (`unknown_parameter`), a missing, non-string or empty `end_user` (never quoted back), or a `client` that names no API key; 401 for a wrong key; 403 when an API key names another client; 409 when new registrations would pass 1,000 in mock mode (the shared mock-mode gateway; a reset clears them; one already registered does not count again). A live gateway has no cap: a cap anyone could fill would block real opt-outs. - The event log records a count and who registered (`no_train` events), never the end user or its hash. ### DELETE `/v1/ratchet/no-train` The same JSON body as `POST`, never a query string (access logs keep URLs). Removes the registration, so that end user's requests are stored again. The response lists each client scope with a `status`: `removed`, `kept` (the end user is still no-train there: an admin registered it in that scope, which an API key cannot remove, or an admin's registration for every client still covers it) or `absent`. An API key removes its own end users' registrations; an admin key removes any, for a named `client`, or with none in every scope: each configured key, the keyless scope, every scope a registration is stored in (also a renamed key's) and `"*"`. Without API keys, a live gateway answers 403 unless an admin key is sent, so nobody can switch storage back on for someone else's end users; the mock-mode gateway stays open, and its keyless callers get `status: null`, as they get `created: null` from `POST`. ## Feedback People who use the answers know when one is wrong. `POST /v1/ratchet/feedback` records their verdict on an answer the gateway served and, with a correction, stores the right answer as a training example whose [provenance](https://docs.ratchet.dualmindlab.tech/guides/glossary/#provenance) is `human`. The student's own text never becomes one (invariant 2): the gateway never kept it, and a `correct` verdict stores nothing. ### POST `/v1/ratchet/feedback` ```bash # A verdict only: counts toward the task's human agreement and the version's human precision. curl -X POST localhost:8000/v1/ratchet/feedback -H "Authorization: Bearer $KEY" \ -H "Content-Type: application/json" \ -d '{"request_id": "req_3f1c0a9b8e7d6c5b4a3f2e1d0c9b8a79", "verdict": "correct"}' # A correction: the right answer and the request's messages, exactly as the chat request sent them. curl -X POST localhost:8000/v1/ratchet/feedback -H "Authorization: Bearer $KEY" \ -H "Content-Type: application/json" \ -d '{"request_id": "req_3f1c0a9b8e7d6c5b4a3f2e1d0c9b8a79", "verdict": "incorrect", "corrected": "{\"category\": \"refund\", \"priority\": \"low\"}", "messages": [{"role": "user", "content": "Where is my refund for order 1234?"}]}' ``` | Field | Meaning | |---|---| | `request_id` | Required. The response's `ratchet.request_id` (`req_…`), or its `id` (`chatcmpl-…`) | | `verdict` | Required. `correct` or `incorrect` | | `corrected` | The right answer, with `incorrect` only. It is cleaned as an answer is served (`tasks.clean_answer`) and must be valid under the task's spec: JSON that parses, every required field, labels in their vocabulary (else **400** `invalid_correction`); at most `RATCHET_MAX_INPUT_CHARS` characters | | `messages` | Required with `corrected`, and only with it: the request's messages, because request rows hold no text. They are validated and normalised exactly as the chat route does (`developer` becomes `system`, text parts are joined), text parts only | | `response_format`, `tools`, `tool_choice`, `stop` | The request's conditions, when it sent any, so the correction has the same example hash as the teacher's answer to that request and replaces it. Without them, a correction of a request that had conditions is another variant of the prompt, in the same split | | `user`, `safety_identifier` | The end user the request named: every identifier it sent, both when it sent both. Required with a correction when the request named one (**400** `end_user_required`, also when it sent both and the correction sends one), and each must be the same (**400** `end_user_mismatch`): the request row keeps a keyed hash of each, and only with the values can every no-train registration of that end user, made with either identifier or by an admin for every client, keep the correction out. Hashed only, never stored. A request logged before the second hash was recorded needs only the identifier it was attributed to | The response: ```json {"object": "ratchet.feedback", "request_id": "req_3f1c…", "verdict": "incorrect", "updated": false, "correction": {"stored": true, "not_stored": null, "split": "train", "replaced": "teacher"}} ``` `updated` is `true` when an earlier verdict on the same request was replaced: there is one verdict per request, the latest. `correction` is `null` without one; otherwise `stored`, with `split` (the split the example's hash gives, invariant 3) and `replaced` (`teacher` when the correction replaced the teacher's stored answer to the same request, `human` when it replaced an earlier correction of the same request, else `null`), or `not_stored` saying why not: | `not_stored` | Why | |---|---| | `global`, `task`, `key`, `request`, `end_user` | A storage opt-out: the request's own (as it was recorded), or one in force now (zero retention, a no-store task, the key's `store=0`, an end user registered as no-train since) ([End users and storage](#end-users-and-storage)) | | `corrections_off` | `RATCHET_FEEDBACK_CORRECTIONS=0`: verdicts are kept, corrections are not | | `pinned` | The task is pinned: its answers are never training examples (invariant 10) | | `unknown_task`, `teacher_only` | The request was answered by the teacher alone and is never distilled: an unregistered task under `RATCHET_UNKNOWN_TASKS` `teacher` or `reject`, or a request with a part only the teacher reads | | `invalid after redaction` | A redaction placeholder breaks the structured answer (as for a teacher answer) | | `human_label_exists` | The example already has an equal or stronger human label: another client's correction, or an operator's import (`python -m ratchet.importer`). The first client's correction stands, and an import beats any correction | | `erased` | The example was erased and is never stored again | A correction that is stored becomes a training example like a teacher's, with `teacher_model` `human`, license `customer-provided labels` and `verified` 1, redacted when `RATCHET_REDACT_PII` is on, attributed to the API key (and the end user's hash). It replaces the teacher's answer on the same example hash (a human label beats a teacher's; a later teacher answer only bumps `seen`). The request counts once toward the frequent-request rule: a correction of a request whose teacher answer was already stored adds no occurrence, and when it lands on another variant of the prompt that example starts at `seen` 0 (invariant 3). A request is corrected once: the same messages and conditions again replace that correction (a fix); other messages are **409** `correction_exists`, and so is a `correct` verdict after a stored correction (erase the example to withdraw it). Rules: an API key reaches its own requests only; another key's request id, an unknown one and one whose row request retention folded are the same **404** `request_not_found`. With API keys off it is open, like chat completions, and reaches requests sent without a key. The volume is bounded by the requests a key may send: one verdict row and at most one stored correction per request. Errors use OpenAI's envelope (400, 401, 404, 409); a refused call records nothing. Human precision, the share of `correct` verdicts on the answers a version served itself (route `student`), is reported in `/ratchet/stats` → `feedback.by_version` with its 95% Wilson bounds (a fixed-sample interval, for reading at one moment). It is never used to promote or roll back a version, because clients report what they notice; the audited precision stays the release signal. The alert is checked after every verdict, so it uses an [anytime-valid](https://docs.ratchet.dualmindlab.tech/guides/glossary/#anytime-valid) rule instead of the Wilson bound: once a version has at least `RATCHET_MIN_AUDITS` verdicts and the mixture e-value against "precision is at least `RATCHET_LIVE_FLOOR`" reaches 40 (`feedback.ALERT_ALPHA` 0.025), one `human_alert` event says so (see the runbook). The false-alarm rate stays under 2.5% however many verdicts arrive. ## Erasure and export An opt-out keeps new data out; these remove what was stored, and hand it over. - **What an erasure does.** Each selected example loses its messages, answer, stored conditions and end-user hash for good. The row stays as a [tombstone](https://docs.ratchet.dualmindlab.tech/guides/glossary/#tombstone) (hash, task, split, seen count, teacher, license, client), so the counts Night School and the golden split rely on stay right, and its hash is never stored again: a later request with the same text is answered as usual and logged `stored` 0 (a `skip_example` event with reason `erased`), from any client. Erasing an end user also clears that end user's hashes from its request rows (both, when the request carried `user` and `safety_identifier` and either one is erased). - **Shared examples.** When the scope names a client or an end user, `RATCHET_ERASURE_SHARED` decides an example another client also sent: `delete` (default) purges it too; `keep` keeps it for the other client and unlinks the erased party. Erasure by hashes or by text purges whatever it selects. - **The record.** `erasures` holds who asked, the reason (redacted), the scope (end users as their HMACs, `contains` as its SHA-256), the counts, the versions that may hold the data (`tainted`) and a `status` (`pending` while such versions exist, else `complete`); `erasure_items` lists each purged hash with its task and split. No request text, raw end user or key is written. An `erase` event gives the counts. - **Trained adapters.** `affected_versions` lists every version that may have been trained on an erased example, with its state and how many: exactly from its committed training set when it has one, else every version created after the example was stored while it counted as a training row (this errs toward listing). They are reported, never changed: each adapter's signed BOM stays as written, and removing what an adapter may have memorised needs a retrain without the rows, which the next training night does (it never sees purged text). Unlearning from an existing adapter is out of scope ([THREAT_MODEL.md](/reference/threat-model/)). - **Retention.** Examples not seen for `RATCHET_EXAMPLE_RETENTION_DAYS` become tombstones the same way, except that a later duplicate that may be stored brings the text back; request rows older than `RATCHET_REQUEST_RETENTION_DAYS` fold into daily sums that every stats figure adds back ([CONFIGURATION.md](/reference/configuration/#privacy-and-security)). ### POST `/ratchet/admin/privacy/erase` Needs an admin key (`RATCHET_ADMIN_KEYS`; 401 without one, 403 for an API key, 403 in live mode without admin keys, open in mock mode as `mock-admin`). ```bash curl -X POST localhost:8000/ratchet/admin/privacy/erase -H "Authorization: Bearer $ADMIN_KEY" \ -H 'Content-Type: application/json' -d '{"client": "acme", "reason": "contract ended", "dry_run": false}' ``` ```json {"object": "ratchet.erasure", "id": 3, "dry_run": false, "matched": 412, "purged": 410, "kept": 2, "requests_unlinked": 0, "affected_versions": [{"id": 4, "state": "live", "adapter_uri": "student-v4", "examples": 371}], "scope": {"client": "acme", "shared": "keep"}} ``` | Field | Meaning | |---|---| | `client` | An API key name (also one whose key was since removed): the examples it sent, first or later | | `end_user` | The value requests are attributed to (`safety_identifier` when a request sent one, else `user`): hashed in `client`'s scope, or with no `client` in every configured API key's scope, every scope the ledger holds end users in (a renamed or removed key's too) and the keyless one; never quoted back | | `hashes` | Up to 10,000 example hashes (64 lowercase hex characters each, as in an export) | | `contains` | 1 to 1,000 characters matched with LIKE in the stored messages, answer and conditions (case-insensitive for ASCII letters; `%` and `_` are literal). Stored text is already redacted, so an identifier the redactor replaced is not there to find | | `reason` | Required, 1 to 500 characters; stored redacted | | `dry_run` | Default `true`: nothing changes, `id` is `null`, and the counts say what would happen | Every selector given narrows the scope; at least one is needed. A bad scope is a **400** that never quotes an end user or the text to match. The erasure is logged with the admin key's name as its actor (`python -m ratchet.admin erase --actor` takes a name of the same characters, never free text, because the `erase` event is public in `/ratchet/stats`). `matched` counts the examples selected (ones an earlier erasure took are left out), `purged` those whose text this erasure removed or whose retention tombstone it made final, `kept` the shared ones kept and unlinked, `requests_unlinked` the request rows whose end-user hash was cleared. ### DELETE `/v1/ratchet/examples` An API key erases the stored examples of one of its own end users, with the same JSON body as `/v1/ratchet/no-train` plus optional `reason` (default `end user request`) and `dry_run` (default `false`, unlike the admin call: a `DELETE` deletes). An admin key may name a `client`, or leave it out for every client. Open wherever chat completions are (it only removes data); keyless without API keys. The end user is read from the body only: a URL with `end_user` is a **400** (`end_user_in_url`), because access logs keep URLs (the item's spec proposed `?end_user=`). The response is the admin call's, except that a keyless caller (API keys off) gets `matched`, `purged`, `kept`, `requests_unlinked` and `affected_versions` as `null`, for a dry run too: an open endpoint must not tell anyone whether an identifier had anything stored (as no-train's `created`). Errors use OpenAI's envelope (400, 401, 403 as for no-train). ```bash curl -X DELETE localhost:8000/v1/ratchet/examples -H "Authorization: Bearer $KEY" \ -H 'Content-Type: application/json' -d '{"end_user": "user-8812"}' ``` ### GET `/ratchet/admin/export/{kind}` One table as NDJSON (`application/x-ndjson`, one JSON object per line, in id order), streamed a page at a time, so the ledger is never locked for the whole export; rows written meanwhile may appear (for a snapshot, back up the file). Needs an admin key, like the other management calls. | Kind | Each line | Filters | |---|---|---| | `examples` | An example that still has its text: `id`, `ts`, `day`, `task`, `hash`, `split`, `seen`, `messages`, `completion`, `params`, `provenance` (`teacher`, `license`, `license_url`, `hash`, `redactions`, `origin`), `client` (first sender), `clients` (every client that sent it), `last_seen_ts` | `task`, `split`, `since`, `client` | | `requests` | A request row's metadata (`requests` columns), without the end-user hashes (`end_user`, `end_user_secondary`) | `task`, `since`, `client` | | `versions` | A version with its `eval` and its BOM exactly as stored | `since` | | `events` | `id`, `ts`, `day`, `kind`, `detail` | `since` | | `erasures` | An erasure with its `scope` (end users as a count), counts, `affected_versions`, `status` and `items` (`hash`, `task`, `split`) | `since` | `since` is seconds since the epoch or an ISO 8601 date or date-time (UTC unless it names an offset), compared with the row's `ts` (`created_ts` for versions). A filter the kind does not take, an unknown kind or a bad value is a **400** before the first line. API keys are never stored and end-user hashes are left out, so no line holds either. `python -m ratchet.admin export [--task] [--split] [--since] [--client] [--out FILE]` writes the same lines. ## GET `/healthz` Liveness: the process answers. Always public, also when API keys are on. The VM's Docker health check uses it. It is a coroutine on the event loop, so it answers while every worker thread is busy (a full thread pool is not a dead process); it asks no dependency, so a restart loop never follows a slow teacher. For "can this gateway serve?", use [`/readyz`](#get-readyz). ```json {"status": "ok", "version": "0.5.0", "mode": "mock"} ``` ## GET `/livez` The same answer as `/healthz`, under the name Kubernetes probes use. Public. ## GET `/readyz` Readiness, for load balancers and deploy scripts. Public; the answer names states, statuses and exception classes only, never a URL, a key or an exception's message. `Cache-Control: no-store`. A 503 here is an answer, not a failed request: it writes no `failures` row, however often a load balancer asks. | Status | `status` | When | |---|---|---| | 200 | `ready` | Every check is `ok` | | 200 | `degraded` | Requests can be answered, but something is down or busy that the gateway works around: the student endpoint (its breaker open, its model list failing or slow, a routed adapter missing), the teacher's model list failing, slow or not listing the teacher model, the teacher's breaker open with its wait over (`probe_due`: the next request is its probe) while its endpoint answers, the teacher's breaker open while a fallback teacher's is closed or the student stands in (`RATCHET_ON_TEACHER_FAILURE=serve_student`, student up), or the ledger busy (another writer held it past 2 s) on one probe | | 503 | `not_ready` | The ledger cannot be written (closed, read-only, or busy on two probes in a row, 15 s apart), or no teacher can answer: its breaker is open and still waiting with no stand-in, its wait is over but its endpoint does not answer (a connection error, a timeout or a 5xx from `/models`), or its model list answers 401 or 403 (the key is refused) | What is checked: - `ledger` (both modes): `SELECT 1`, then `BEGIN IMMEDIATE`, a write of the header's `user_version` to its own value, and `ROLLBACK`. Nothing is committed. `error` is `closed`, `read_only`, `locked`, `lock_timeout` or the exception's class. `locked`, `lock_timeout` and a probe still waiting after 2 s (`timeout`) mean another writer holds the ledger (`python -m ratchet.live night`, a long write): requests wait for it rather than fail, so the first such probe is `degraded` and the second in a row `not_ready` (`busy_runs` counts them). - `teacher` (live mode): the teacher's breaker (and the fallback teacher's, `fallback_breaker`), and `GET {RATCHET_TEACHER_BASE}/models` with the teacher key (no tokens spent): `status`, `models` (`ok`, `http_`, `unreadable`) or `error` (`timeout`, `ConnectError`, ...), and `model_listed`. While the primary's breaker is not closed, `probe_due` says whether its wait is over (or it is half-open), so the next request that needs the teacher is the breaker's probe. `available`: some teacher can answer: its breaker is closed, or its probe is due and its endpoint answered `/models` below 500 (a fallback teacher on an endpoint of its own, `RATCHET_TEACHER_FALLBACK_BASE`, has no such check: its due probe is enough). Only a request moves an open breaker on, and a gateway a load balancer took out gets none, so a due probe must count, or one teacher blip would keep the gateway out for good. `ok`: the primary's breaker is closed and its model list lists the teacher model. - `student` (live mode): the student breakers by name (`breakers`; the shadow's is reported, not counted). With `RATCHET_STUDENT_BASE`, `GET {RATCHET_STUDENT_BASE}/models` must list the adapter of every live, canary and shadow version: `missing_adapters` names the ones it does not, and each is handed to the adapter reloader (`reload_queued`; at most once a minute per adapter). Without an endpoint, `model` is the hosted fallback student. - `config`: the number of startup warnings (in mock mode also their messages; in live mode read them in the log or with `python -m ratchet.config check`). - `flags`: operational switches that never change the status: `scout_budget_exhausted` (Scout's daily budget is spent). Mock mode checks the ledger only. The probes run on threads of their own (two for the endpoints, one for the ledger, never the request thread pool), each awaited at most 2 s, the teacher's at most 4.6 s because a `/models` call that times out is tried once more after 1.5 s, and the gateway is `degraded` only when that second try fails too (a slower one reports `"error": "timeout"` and goes on, and its result is kept), and a result is reused for 15 s, so the teacher sees at most one `/models` call per 15 s however often `/readyz` is asked. The student endpoint's result is reused for 5 minutes while it is `ok` (15 s when it is not, so a reloaded adapter shows soon): it never makes the gateway not ready, and each probe is a request to the endpoint, which on a platform that scales an idle endpoint to zero may count as traffic and keep it running. Breaker states are read fresh on every call. `ms` is how long a probe took. ```json { "status": "degraded", "mode": "live", "version": "0.5.0", "checks": { "ledger": {"ok": true, "ms": 1.2}, "teacher": {"breaker": "closed", "status": 200, "models": "ok", "ms": 84.0, "model_listed": true, "available": true, "ok": true}, "student": {"endpoint": true, "breakers": {"student": "closed"}, "ok": false, "status": 200, "models": "ok", "missing_adapters": ["student-v3"], "reload_queued": ["student-v3"], "ms": 41.5}, "config": {"ok": true, "warnings": 0} }, "flags": {"scout_budget_exhausted": false} } ``` ## GET `/v1/models` The model ids a client can send ([Task resolution](#task-resolution)), each a `model` object the OpenAI SDK's `Model` accepts. Needs a key when `RATCHET_API_KEYS` is set. - `ratchet` first (its task comes from each request); - `ratchet/` for every registered task the key may use (a task whose `clients` do not name it is left out; with keys on, an auto task only to a key whose own requests used it in the last 30 days, since its name is a digest of a client's system prompt), in registry order; - unless `RATCHET_UNKNOWN_TASKS=reject`, `ratchet/` for every unregistered name the key's own requests named in the last 30 days (with keys off, any request's), at most 500, the most recently used first; - every model alias whose task the key may use, with `alias_of`; - the embeddings model (`RATCHET_EMBEDDINGS_MODEL`) unless `RATCHET_EMBEDDINGS_BASE=off`, with `ratchet.endpoint: "embeddings"` and no task: the id `RATCHET_MODEL_POLICY=strict` accepts on `/v1/embeddings`. `ratchet/embeddings` is never listed for it: `embeddings` is the task name embeddings requests are logged under, not a chat task. ```json {"object": "list", "data": [ {"id": "ratchet", "object": "model", "created": 1792400000, "owned_by": "ratchet", "ratchet": {"task": null, "pinned": false, "format": null, "compare_keys": [], "live_version": 5}}, {"id": "ratchet/ticket-triage", "object": "model", "created": 1792310000, "owned_by": "ratchet", "ratchet": {"task": "ticket-triage", "pinned": false, "format": "json", "compare_keys": ["category", "priority"], "live_version": 5}}, {"id": "gpt-4o-mini", "object": "model", "created": 1792310000, "owned_by": "ratchet", "ratchet": {"task": "ticket-triage", "pinned": false, "format": "json", "compare_keys": ["category", "priority"], "live_version": 5, "alias_of": "ratchet/ticket-triage"}} ]} ``` `created` is when a request (this key's, with keys on) first named the task, else when the gateway process started. In `ratchet`: the task, whether it is pinned, its answer format, the answer fields that must agree, and the live version that serves it (`null` when the student never does: no live version, a pinned task, or an unregistered task under `RATCHET_UNKNOWN_TASKS=teacher`). ## GET `/v1/models/{id}` One object of the list above (`client.models.retrieve("ratchet/ticket-triage")`; the SDK sends the slash as `%2F`, which is fine; `ratchet:` finds `ratchet/`). It also finds `ratchet/` for an unregistered task name no request used yet, unless `RATCHET_UNKNOWN_TASKS=reject`, because a chat completion with that model would be served (under either `RATCHET_MODEL_POLICY`), so a client that checks its model first gets the request's answer; `created` is then the process start. Any other id is a 404 `model_not_found`: `{"error": {"message": "The model 'gpt-4o' does not exist.", "type": "invalid_request_error", "param": null, "code": "model_not_found"}}`. Needs a key when `RATCHET_API_KEYS` is set. ## Azure-style route `POST /openai/deployments/{deployment}/chat/completions?api-version=...` runs the same handler as `/v1/chat/completions` (the same validation, checks, quota, envelope, streaming and `ratchet` block) for clients written against Azure OpenAI. The deployment names the task: a model alias's task, `ratchet` (the other rules decide), `ratchet:`, or a task name, which follows `RATCHET_UNKNOWN_TASKS` when it is not registered; anything else is a 404 `model_not_found`. The body's `model` is not read (so `RATCHET_MODEL_POLICY=strict` does not refuse it), the header and `metadata.ratchet_task` still win, and `api-version` is accepted and ignored. The key may come in Azure's `api-key` header. ```python from openai import AzureOpenAI client = AzureOpenAI(azure_endpoint="http://localhost:8000", api_key="unused", api_version="2024-10-21") resp = client.chat.completions.create(model="ticket-triage", messages=[{"role": "user", "content": "I was charged twice."}]) ``` ## POST `/v1/embeddings` A pass-through: one call to `RATCHET_EMBEDDINGS_MODEL` at `RATCHET_EMBEDDINGS_BASE` (by default the teacher's Token Factory endpoint, with `NEBIUS_API_KEY`; another base gets `RATCHET_EMBEDDINGS_KEY`, or no key when that is empty, never the Token Factory key). No student, no teacher, no confidence, no audit, and nothing of the request is stored: vectors are not distilled. Mock mode answers locally with deterministic unit vectors (256 values, seeded from SHA-256 of each input). `RATCHET_EMBEDDINGS_BASE=off` turns the route into a 404 `unsupported_endpoint`. | Field | Type | Required | Notes | |---|---|---|---| | `input` | string, array of strings, array of token ids, or array of token-id arrays | yes | 1 to 2,048 items, none empty, all of one kind. Their text counts against `RATCHET_MAX_INPUT_CHARS` (a token as 4 characters) | | `model` | string | no | Never selects the upstream model: the configured one always answers. Under `RATCHET_MODEL_POLICY=lenient` any id is served (another id than `ratchet` or the configured one is echoed in `X-Ratchet-Model-Alias`); under `strict` only `ratchet` and the configured id, anything else a 404 `model_not_found`, because vectors from another model than an index was built with cannot be compared with it | | `encoding_format` | `float` or `base64` | no | What the client gets; the upstream is always asked for floats, and the gateway encodes base64 (little-endian float32, what the OpenAI SDKs ask for by default and decode) | | `dimensions` | integer, 1 to 8,192 | no | Sent upstream as is (the mock keeps the first that many values of the full vector and scales them back to length 1) | | `user` | string | no | Checked, never forwarded | | any other field | | no | Ignored, never forwarded | The response is OpenAI's (`openai.types.CreateEmbeddingResponse`) plus a `ratchet` block; `model` is the upstream's name for the model that served, and `usage` is the upstream's prompt tokens (about 4 characters a token when it reports none): ```json {"object": "list", "model": "Qwen/Qwen3-Embedding-8B", "data": [{"object": "embedding", "index": 0, "embedding": [0.0123, -0.0456, ...]}], "usage": {"prompt_tokens": 5, "total_tokens": 5}, "ratchet": {"served_by": "upstream", "route": "passthrough", "task": "embeddings", "cost_usd": 5e-08, "all_teacher_cost_usd": 5e-08, "latency_ms": 41.2, "requested_model": "text-embedding-3-small"}} ``` Each answered call is one request row (task `embeddings`, served_by `upstream`, route `passthrough`, `stored` 0) priced at the configured model's input price (`EMBEDDING_PRICES` in `ratchet/config.py`, checked on 29 Oct 2026; a model without a price there is priced as the teacher), so it counts against the key's quota and shows in the cost report and as the `embeddings` row of `/ratchet/stats` `tasks`, with its own cost as its all-teacher baseline. The headline figures (`days`, `totals`, `clients`) leave it out: they describe the distillation, which embeddings are not part of, so a client that embeds once per chat request does not halve its student share. Headers: `X-Ratchet-Served-By: upstream`, `X-Ratchet-Task: embeddings`. The task name `embeddings` is reserved for these rows: a chat, completions or responses request that names it (`X-Ratchet-Task`, `metadata.ratchet_task`, `model: "ratchet/embeddings"`, an Azure deployment) is a 400 `reserved_task`, and no task spec or model alias may use it. A failed call logs nothing; its error is mapped by cause ([Errors](#errors): `upstream_*` codes). The shared mock-mode gateway returns at most 131,072 values (inputs x dimensions) per request, about 3 MB of JSON. ```python vectors = client.embeddings.create(model="ratchet", input=["refund my order", "reset my password"]) ``` ## POST `/v1/completions` The legacy text API, for clients and tools that still call it. Each prompt becomes one user message, `[{"role": "user", "content": prompt}]`, and runs the chat path: the same validation, task resolution (`X-Ratchet-Task`, `metadata.ratchet_task`, `model: "ratchet/"`, ...), cascade, quota, storage rules and `ratchet` block as a `/v1/chat/completions` request with that message, so a prompt is stored as exactly that chat request's example. | Field | Notes | |---|---| | `prompt` | A string, or up to 16 strings (each its own cascade request, logged and counted against the quota; a batch the quota cannot finish is refused before its first prompt runs). Token arrays are a 400 `unsupported_value`: the gateway works in text | | `suffix` | A 400 `unsupported_parameter` unless empty: a chat model cannot fill in text before a suffix | | `echo` | A 400 `unsupported_parameter` when `true` | | `best_of` | A 400 `unsupported_parameter` unless 1 or `n`: the gateway returns the choices it samples, it does not rank more | | `logprobs` | 0 to 5: each choice's token logprobs in the legacy shape (`tokens`, `token_logprobs`, `top_logprobs` as `{token: logprob}` per token, `text_offset` counted from the prompt's length), from the chat path's token logprobs with that many alternatives ([Choices and logprobs](#choices-and-logprobs)); `null` when they cannot be given | | `model`, `max_tokens`, `temperature`, `top_p`, `n`, `stream`, `stream_options`, `stop`, `seed`, `presence_penalty`, `frequency_penalty`, `logit_bias`, `user`, `metadata`, others | The chat field of the same name, checked and sent exactly as on `/v1/chat/completions`. `max_tokens` has no default of 16 here: the per-call caps apply ([Generation parameters](#generation-parameters)) | ```json {"id": "cmpl-...", "object": "text_completion", "created": 1792400000, "model": "student-v3", "system_fingerprint": "fp_...", "choices": [{"text": "{\"category\": \"billing\", ...}", "index": 0, "logprobs": null, "finish_reason": "stop"}], "usage": {"prompt_tokens": 18, "completion_tokens": 21, "total_tokens": 39, ...}, "ratchet": {"served_by": "student", "task": "ticket-triage", ...}} ``` Prompt p's choice j is choice `p * n + j`, as OpenAI numbers them; `usage` adds up every prompt's. With one prompt `ratchet` is the chat path's block; with several it is `{"prompts": [block, ...]}`, and the headers are the last prompt's (`X-Ratchet-Served-By` is `mixed` when the prompts were not all served the same way). With `stream: true` the answer comes as `text_completion` chunks (`choices: [{text, index, logprobs, finish_reason}]`), each choice's text then its finish chunk, a last chunk with `usage` and no choices under `stream_options.include_usage`, and `[DONE]`; the `ratchet` block rides on the last finish chunk or on the usage chunk, as in a chat stream. The prompts run in order. If the first fails, the request returns its error, as a chat request would. If a later one fails (the teacher fails, or a concurrent request from the same key used up the quota the batch was checked against), or the batch has run for `RATCHET_TEACHER_DEADLINE_S` (180 s) before its next prompt, the request returns that prompt's error (or a 504 `batch_deadline_exceeded`) with `X-Should-Retry: false` and no `Retry-After`, and a message naming the prompts already answered (`The batch stopped at prompt[1] of 2: ... prompt[0] was answered, logged and billed, but this error does not return its answer; resend only the prompts from prompt[1] on.`). Those stay logged, billed and stored; their answers are not returned. OpenAI SDKs then do not resend the batch, which would run, bill and store the earlier prompts again (and count their examples as seen again); send only the prompts from the one named on. ```python resp = client.completions.create(model="ratchet/ticket-triage", prompt="I was charged twice this month.") print(resp.choices[0].text) ``` ## POST `/v1/responses` The stateless subset of OpenAI's Responses API. The body becomes a chat request, which runs the chat path (validation, task resolution, cascade, quota, storage, `ratchet` block); the answer comes back as a `response` object (`openai.types.responses.Response`). Ratchet keeps no response state: each response is built for its request and forgotten, so a conversation is sent whole every time. | Field | Becomes | |---|---| | `input` | A string: one user message. An array of items: `message` items keep their role (`user`, `assistant`, `system`, `developer`) and content (a string, or `input_text` / `output_text` / `refusal` parts joined as a chat request's text parts are; an `input_image` with an `image_url`, read by the teacher only under `RATCHET_TEACHER_MODALITIES`); consecutive `function_call` items the tool calls of one assistant message (`call_id` the call's id), joined to the assistant `message` item right before them when there is one, as a response's `output` lists its text and then its calls; a `function_call_output` a `tool` message. `reasoning` items are dropped (and split no calls). So a tool loop sent back as input is the chat conversation it would be, and is stored as that request. An `item_reference`, a built-in tool's item, a file, audio or an image by file id is a 400 `unsupported_value` | | `instructions` | A leading system message | | `text.format` | `response_format`: `json_schema` (its `name`, `schema`, `strict`, `description`), `json_object`, or `text` (none) | | `tools`, `tool_choice`, `parallel_tool_calls` | Function tools (`{type: "function", name, description, parameters, strict}`) and a function `tool_choice` become the chat ones and follow their rules ([Generation parameters](#generation-parameters)). A built-in tool (web search, file search, code interpreter, MCP, ...) or a tool choice naming one is a 400 `unsupported_parameter` | | `max_output_tokens`, `temperature`, `top_p`, `reasoning.effort` | `max_tokens`, `temperature`, `top_p`, `reasoning_effort` | | `model`, `metadata`, `user`, `safety_identifier` | The chat fields of the same name (`model: "ratchet/"` and `metadata.ratchet_task` name the task) | | `previous_response_id`, `conversation`, `prompt`, `background: true`, `stream: true` | A 400 `unsupported_parameter`: they need stored state, or a stream of response events, that the gateway does not keep or send. `stream: false` and `background: false` are fine | | `include`, `truncation`, `service_tier`, `store`, `top_logprobs`, `max_tool_calls`, `prompt_cache_key` | Accepted, no effect: no reasoning items or logprobs are returned, the input is never cut, and nothing is stored (`store` is `false` in the response) | | a chat-only field (`n`, `stop`, `messages`, ...) | Ignored | | any other field | An unknown chat field: kept, forwarded only when an allowlist names it ([Request](#request)) | Errors name the Responses field (`max_output_tokens`, `text.format.name`, `tools[0].name`, `input[2].call_id`). ```json {"id": "resp_...", "object": "response", "created_at": 1792400000, "status": "completed", "model": "nvidia/Nemotron-3-Ultra-550b-a55b", "output": [{"type": "message", "id": "msg_...", "status": "completed", "role": "assistant", "content": [{"type": "output_text", "text": "Refunds take 5 to 7 days.", "annotations": []}]}], "usage": {"input_tokens": 21, "input_tokens_details": {"cached_tokens": 0, "cache_write_tokens": 0}, "output_tokens": 9, "output_tokens_details": {"reasoning_tokens": 0}, "total_tokens": 30}, "instructions": "Be brief.", "store": false, "previous_response_id": null, "tool_choice": "auto", "tools": [], ..., "ratchet": {"served_by": "teacher", "task": "default", ...}} ``` `status` is `incomplete` with `incomplete_details.reason` `max_output_tokens` when the answer was cut off (`content_filter` when filtered). Tool calls in an answer become `function_call` items (`call_id`, `name`, `arguments`), which a client answers with a `function_call_output` item carrying the same `call_id`; a teacher answer that is a tool call follows the chat path's rules (`tool_calls_unsupported` for a tool call to a request that offered no tools). The settings the request sent are echoed as sent. ```python resp = client.responses.create(model="ratchet", instructions="Be brief.", input="How long do refunds take?") print(resp.output_text) ``` ## Unsupported endpoints `POST /v1/moderations` and the stored-response routes (`GET` and `DELETE /v1/responses/{id}`, `POST /v1/responses/{id}/cancel`, `GET /v1/responses/{id}/input_items`) answer 404 `unsupported_endpoint` after the key check, with a message that says why and links [OpenAI endpoints](#openai-endpoints), so a library that calls them fails with a clear reason (`openai.NotFoundError`, `e.code == "unsupported_endpoint"`). Any other path under `/v1/` is a 404 `unknown_url`. ## GET `/ratchet/counters` Access level `dashboard`, like `/ratchet/stats`. Cumulative aggregates of this gateway process, small enough to poll after every paid call. The ledger of the live gateway is on the container's own disk, so a restart empties it; the Worker in front of the container (`deploy/cloudflare-live`) therefore keeps running totals outside it, from the difference between two of these snapshots (`GET /ratchet/stats/persistent` on that Worker). ```json {"instance": "9f2c...", "started": 1793000000.0, "mode": "live", "requests": 41, "requests_teacher": 30, "requests_split": 4, "requests_student": 7, "student_tokens_in": 5120, "student_tokens_out": 1840, "cost_usd": 0.0412, "baseline_usd": 0.0518, "errors": 3, "server_errors": 1, "scout_runs": 3} ``` - `instance` names the process and `started` is when it began. Every number only grows while the instance is the same; a new instance counts from its own empty ledger, so a reader adds the difference of two snapshots of one instance and the whole first snapshot of a new one. - `requests` counts gateway calls (Scout's model calls included, embeddings pass-through left out), `requests_split` the live site's random split (route `demo_split`), `requests_student` the other student-served ones and `requests_teacher` the rest. `student_tokens_*` are the student calls' tokens only: the ledger keeps no teacher token counts. `cost_usd` is what requests and failed requests spent (Token Factory list prices). `baseline_usd` is what the same requests would have cost all-Ultra (the `baseline_usd` of `/ratchet/stats` totals), so a reader of the cumulative totals can compute a saving as 1 − `cost_usd` / `baseline_usd`, and only when `baseline_usd` is above zero. `scout_runs` counts the Scout runs this process admitted. `errors` counts every row of the failures log, so a refused key (a 401), a rate limit, a quota or a budget block is in it; it is kept for compatibility. `server_errors` counts only real failures: a failed teacher, student, audit, shadow or upstream call, and a 5xx from the gateway itself. Pages that show a number of errors show `server_errors`. - Counts and dollars only: no text, no address, no client name, no end user. ## GET `/ratchet/stats` Access level `dashboard` ([Access levels](#access-levels)): open in mock mode; in live mode an admin or dashboard key unless `RATCHET_PUBLIC_DASHBOARD=1`, and then an anonymous reader gets a scrubbed copy (client pseudonyms, also in `budget` events, no URLs or error messages in `events`, and no dollar caps: `clients[].daily_cap_usd`, `budget.daily_usd` and `budget.remaining_usd` are `null`, and a `budget` event names no amount). ```json { "mode": "mock", "day": 9, "teacher": "nvidia/Nemotron-3-Ultra-550b-a55b", "student_base": "nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16", "fallback_student": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B", "student_serving": {"kind": "mock", "model": "student-v5", "version": 5, "threshold": 0.76, "certified": true}, "days": [ {"day": 3, "requests": 1500, "student_share": 0.98, "escalation_rate": 0.02, "cost_per_1k": 0.16, "baseline_per_1k": 1.093, "savings_pct": 0.85, "avg_latency_ms": 352.0, "audit_agreement": 1.0, "degraded": 0, "fallback_teacher": 0, "fallback_served": 0, "p50_student_ms": 0.41, "p95_ms": 0.93} ], "versions": [ {"id": 5, "day": 7, "base_model": "nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16", "adapter_uri": "student-v5", "train_examples": 1801, "dataset_hash": "…", "eval": {"golden_n": 201, "golden_agreement": 0.94, "threshold": 0.76, "coverage": 0.89, "precision": 0.99, "test_n": 100, "rechecked_day": 9}, "state": "live", "traffic_pct": 100, "stage_since": 12034, "bom": "signed", "evidence": {"rule": "wilson", "promote_rule": "point", "alert_rule": "eprocess", "floor": 0.92, "n": 61, "s": 60, "e_rollback": 0.074, "n_stage": 38, "s_stage": 37, "e_promote": 2.0, "e_alert": 0.146, "thresholds": {"rollback": 20.0, "promote": 20.0, "alert": 100.0}, "cs": [0.879, 0.999], "wilson": [0.865, 0.995]}} ], "tasks": [ {"task": "ticket-triage", "pinned": false, "requests": 12000, "student_share": 0.84, "escalation_rate": 0.06, "cost_usd": 3.21, "baseline_usd": 13.1, "savings_pct": 0.75, "avg_latency_ms": 640.0, "routes": {"student": 10080, "cold_start": 1100, "confidence": 720, "student_error": 60, "student_breaker": 40}, "human_feedback": 40, "human_agreement": 0.925, "audits": 1008, "audit_agreement": 0.98, "threshold": 0.7412, "precision": 0.98, "coverage": 0.91, "calibrated": true, "disabled": null, "compare_keys": ["category", "priority"], "compare_source": "registry", "tool_requests": 0, "precision_kind": "exact", "judge_pass_rate": null}, {"task": "research-plan", "pinned": true, "requests": 3, "student_share": 0.0, "escalation_rate": 0.0, "cost_usd": 0.0039, "baseline_usd": 0.0039, "savings_pct": 0.0, "avg_latency_ms": 2400.0, "routes": {"pinned": 3}, "human_feedback": 0, "human_agreement": null, "audits": 0, "audit_agreement": null, "threshold": null, "precision": null, "coverage": null, "calibrated": null, "disabled": null, "compare_keys": [], "compare_source": "registry", "tool_requests": 0, "precision_kind": null, "judge_pass_rate": null} ], "events": [{"id": 57, "ts": 1792400000.0, "day": 9, "kind": "live", "detail": "v5 promoted to 100% (audited precision 94.1%, n=17)"}], "totals": {"requests": 13500, "cost_usd": 5.21, "baseline_usd": 14.58, "saved_usd": 9.37, "savings_pct": 0.64, "failed_cost_usd": 0.0, "cost_model": "per_token"}, "bill_of_materials": {"generated_at": 1792400000.0, "examples_sha256": "07ac2c…", "sources": [ … ], "label_sources": [ … ]}, "clients": [ {"client": "acme", "requests": 40, "student_share": 0.95, "cost_usd": 0.0044, "baseline_usd": 0.044, "savings_pct": 0.9, "end_users": 12, "stored_share": 0.05, "spent_today_usd": 0.0011, "daily_cap_usd": 5.0} ], "privacy": {"redaction": true, "examples": 2742, "redacted_examples": 12, "placeholders": 19, "not_stored_requests": 310, "not_stored_by_reason": {"request": 290, "end_user": 20}, "no_train_end_users": 3, "no_train_unmatched": 0, "store_examples": true, "no_store_tasks": ["hr-notes"], "no_store_clients": [], "end_user_key": "env", "example_retention_days": 180, "request_retention_days": 400, "erasure_shared": "delete", "purged_examples": 61, "purged_by_reason": {"retention": 58, "erasure": 3}, "erasures": 2, "erasure_refusals": 1, "rolled_up_requests": 120400, "redaction_style": "label", "redaction_task_styles": {"invoice-extract": "numbered"}, "redaction_plugin": false}, "security": {"api_keys": true, "daily_requests_per_key": 500, "bom_signing": true, "bom_key_id": "6cf6bc1a39c4491d", "key_limits": true, "ip_rpm": null, "max_inflight": 32}, "budget": {"daily_usd": 20.0, "mode": "student_only", "spent_today_usd": 3.41, "remaining_usd": 16.59, "exceeded": false, "alert_pct": [50, 80, 100]}, "cost": {"model": "per_token", "endpoint_hours": 216.0, "gpu_hours_savings_pct": -22.49, "break_even_student_requests_per_gpu_hour": 1582.0, "recommendation": "hosted_fallback"}, "drift": {"enabled": true, "threshold": 100.0, "family_bound": 0.03, "alarms": 1, "streams": 3}, "teacher_watch": {"enabled": true, "epoch": 1, "frozen": false, "checks": 273, "disagreements": 11, "self_agreement": 0.9597, "ceiling": 0.9886, "e_value": 96.38, "contested": 0}, "feedback": {"verdicts": 40, "correct": 37, "incorrect": 3, "corrections_sent": 3, "corrections_stored": 2, "human_examples": 152, "golden_only_examples": 120, "corrections": true, "by_version": [{"version": 5, "verdicts": 31, "correct": 29, "precision": 0.935, "lower": 0.793, "upper": 0.982}]}, "errors": {"last_hour": {"student": 3, "auth": 1}}, "sidecar": {"workers": 4, "max_queue": 1000, "audit_places": 100, "max_bytes": 67108864, "queued": 2, "running": 4, "bytes": 41210, "closed": false, "submitted": 5130, "done": 5124, "failed": 0, "full": 0, "refused": 0, "abandoned": 0, "by_version": [{"version": 5, "audits_inline": 0, "shadows_dropped": 0, "audits_abandoned": 0, "shadows_abandoned": 0}]}, "controls": {"student_disabled": false, "releases_frozen": false, "training_paused": false, "audit_rate": null, "pinned_tasks": []}, "kpis": {"mode": "mock", "simulated": true, "window_days": 3, "window": {"from": 7, "to": 9, "days": 3, "complete": true}, "metrics": {"SM-1": {"name": "Cost reduction vs all-teacher, steady state", "value": 0.84, "target": 0.7, "op": ">=", "unit": "fraction", "met": true, "detail": {"…": "…"}}, "…": "…"}, "summary": {"met": 5, "not_met": 1, "unknown": 0}} } ``` - `fallback_student` (v0.5): the hosted, untuned model that answers student calls in live mode without a student endpoint (`RATCHET_FALLBACK_STUDENT`). A response whose `ratchet.served_by` is `student` and whose `model` is this id came from it, not from our adapter. - `student_serving` (v0.5): which model answers a student call now, so a page can name it. Read from settings and the versions table; it calls no model. | Key | Meaning | |---|---| | `kind` | `mock` (mock mode: the mock fleet serves each version's own adapter); `endpoint` (live, `RATCHET_STUDENT_BASE` set: our adapter on the student endpoint); `hosted_fallback` (live, no student endpoint, a version serves: every student call goes to `fallback_student`); `none` (no live or canary version: the teacher answers everything). Later versions may add kinds; show an unknown one as it is | | `model` | The model id student calls are sent to: the version's `adapter_uri`, or `fallback_student` for `hosted_fallback`; `null` without a version | | `version` | The live version's id; on a cold start whose first student is in canary, the canary's; `null` for none | | `threshold` | The confidence threshold that model is held to (`Ratchet.threshold`): the version's calibrated one, or `RATCHET_CONFIDENCE_THRESHOLD` for the hosted fallback, which was never calibrated; `null` without a version | | `certified` | The version's eval carries a certificate that certifies (`certificate.certified`), and our adapter serves. Always `false` for `hosted_fallback`: the certificate was computed for our adapter, never for the hosted model | | `tasks` | Only while the hosted student answers (`RATCHET_HOSTED_STUDENT`): each task it answers with its measured threshold, `{"source-triage": 0.955}`; `version` is then its row (state `hosted`, never in `versions`), and `threshold` is the one task's threshold, or `null` for several. A hosted row that answers no task is reported as `none` | While `kind` is `hosted_fallback`, the pages say "Nano 30B · hosted" for a student answer (with the threshold, and "not covered by the certificate"), stop saying "your adapter", and hide the live version's golden precision ([SCOUT.md](/reference/scout/#who-answered)). - `demo_split`: the live gateway's split: `enabled`, `rate` (`RATCHET_DEMO_SPLIT`), `audit_rate`, and from the request rows still held `requests` (answered with route `demo_split`), `of_requests` (the requests of the tasks they came from), `share`, `audits` and `agreed` (audits that came back with a verdict, Ultra's answer against the Nano's) with `agreement`, and `cost_usd`, `baseline_usd`, `saved_usd`, `savings_pct` of those requests; null where there is nothing to divide. Counts and dollars only. `student_serving` gains `demo_split` (the rate) while the split is on. - `days`: one entry per `requests.day` (see `Ledger.stats_by_day`), including days whose rows retention folded into daily sums (`RATCHET_REQUEST_RETENTION_DAYS`); every figure in `days`, `tasks`, `clients` and `totals` adds those sums back. `/v1/embeddings` calls (route `passthrough`) are left out of `days`, `totals` and `clients`, and counted in `tasks` (the `embeddings` row) and in `cost`. `audit_agreement` is `null` on days without audits. `degraded` and `fallback_teacher` count that day's requests a stand-in answered because the teacher was unavailable (the routes of the same names below). `fallback_served` counts that day's student-served requests whose answer another model than their version's adapter wrote: the hosted fallback student, without `RATCHET_STUDENT_BASE` (v0.5). They are in `student_share` and `audit_agreement`, but count toward no version's release (days whose rows retention folded report 0). `avg_latency_ms` averages the providers' time (`requests.latency_ms`). `p50_student_ms` (the day's student-served requests) and `p95_ms` (all of them) are nearest-rank percentiles of `requests.wall_ms`, the whole cascade on the gateway's clock (v0.5): the numbers the latency target's target (p50 under 1 s for student-served requests) is checked against. Each is `null` for a day without such rows, such as days logged before v0.5. In mock mode the models answer at once, so these are Ratchet's own time, while `avg_latency_ms` is the mock model latency. - `tasks`: one entry per task name over all days, busiest first (`Ledger.stats_by_task`). `student_share` and `escalation_rate` are shares of that task's requests; `savings_pct = 1 − cost_usd / baseline_usd`. Pinned tasks always show `student_share` 0 and `savings_pct` 0. `pinned` is true for tasks that always go to the teacher and are never trained on (`research-plan`, `research-write`); the dashboard's **By task** table shows them as "pinned to teacher". `routes` counts the task's requests by how they were served (the ledger's `requests.route`), most frequent first, and adds up to `requests`: | Route | Served by | Why | |---|---|---| | `student` | student | Its confidence reached the version's threshold (and, for `n` > 1 choices, at least `n` samples agreed) | | `confidence` | teacher | The student was unsure (below the threshold) | | `demo_split` | student | The live gateway's split (`RATCHET_DEMO_SPLIT`) picked this request for the hosted Nano at random, not by confidence; the answer names the Nano and no version, is audited at `RATCHET_DEMO_SPLIT_AUDIT` and counts in no release | | `cold_start` | teacher | No version serves yet (no live version, and the canary was not picked) | | `pinned` | teacher | A pinned task (by its spec, or at runtime by an operator: [Admin](#admin)); before v0.5, also a request with a part only the teacher reads (now route `ineligible:content_type`) | | `student_error` | teacher | The student call failed (timeout, connection error, an error status, a lost adapter) | | `student_breaker` | teacher | The student's circuit breaker was open, so the student was not called ([ARCHITECTURE.md](/reference/architecture/#student-circuit-breaker)) | | `degraded` | student | The student was unsure and the teacher unavailable, so its own answer was served (`RATCHET_ON_TEACHER_FAILURE=serve_student`, `ratchet.degraded`); not audited, stored or shadowed ([ARCHITECTURE.md](/reference/architecture/#when-the-teacher-is-down)) | | `fallback_teacher` | teacher | The teacher was unavailable (its retries failed or its breaker was open), so the fallback teacher answered (`RATCHET_TEACHER_FALLBACK_MODEL`); an escalation, a cold start, a pinned task or a teacher-only request | | `n_choices` | teacher | The student was confident, but fewer than the request's `n` samples gave its answer, so it could not fill `n` choices ([Choices and logprobs](#choices-and-logprobs)) | | `unknown_task` | teacher | The task is not registered and `RATCHET_UNKNOWN_TASKS=teacher`: the student was not called and nothing was stored ([Tasks](#tasks)) | | `student_disabled` | teacher | An operator turned the student off (`POST /ratchet/admin/student/disable`, [Admin](#admin)): no student or shadow call; the teacher's answer is stored as on a cold start | | `tools` | teacher | A tool request (tools offered, or tool calls or results in its messages): the teacher answered alone, with the tools, and nothing was stored ([Tools and function calling](#tools-and-function-calling)) | | `multimodal` | teacher | Written only by a caller of `Ratchet.teacher_only` that passes it, for a request with a content part only the teacher reads: the student was not called and nothing was stored. The gateway sends such a request through the cascade, which logs it `ineligible:content_type` (below) | | `task_disabled` | teacher | The serving version's per-task record lets the student answer none of this task (the release gate disabled it on that version, or its eval had no held-out row of it), or the task has critical classes the version's record did not measure under the current spec, or the hosted untuned model would answer it, so the student was not called; never served in degraded mode ([per-task calibration](/reference/configuration/#per-task-calibration)) | | `rule:` | teacher | The student was confident, but rule `i` (from 0) of the task's `escalate_when` held for its answer (`action: teacher`, or a confidence below the rule's `min_confidence`), so the teacher answered and its answer was stored as for any escalation; never served in degraded mode ([risk-aware routing](/reference/configuration/#risk-aware-routing)) | | `ineligible:input_tokens` | teacher | The request holds more input than the task's `student_max_input_tokens` (characters of its message text / 4), so no version was picked and the student was not called; stored only with the task's `store_ineligible`; not counted as an escalation | | `ineligible:content_type` | teacher | The request has a content part the student does not read (an image or other part admitted by `RATCHET_TEACHER_MODALITIES`; the student reads text only), so the student was not called and nothing was stored; not counted as an escalation | | `unrecorded` | either | Requests logged before v0.5, when the ledger had no route column | Later versions add routes; a client should show an unknown route as it is. `human_feedback` counts the task's requests that got a verdict ([Feedback](#feedback)), the latest per request, and `human_agreement` is the share of them judged `correct` (`null` without any). `tool_requests` counts the task's tool requests, whatever their route (`tools`, `pinned`, `unknown_task`, `multimodal` or `fallback_teacher`); requests logged before the count existed count as none. each entry also says how well the student does the task and how it is held to it ([per-task calibration](/reference/configuration/#per-task-calibration)). `audits` and `audit_agreement` cover the teacher's audits of the task's student answers over every version (`audit_agreement` is `null` without audits). The rest come from the live version's eval and are `null` without a live version and for a pinned task: `threshold` is the confidence the live student needs to answer the task, as the router applies it (its per-task threshold, 1.01 when the task is disabled, has no held-out rows, its spec changed since the version was scored or it has critical classes the version's record did not measure, else the version's global threshold); `precision` and `coverage` are the task's held-out numbers measured when the version was scored, at the threshold it was calibrated to (for a disabled task, or one whose spec changed since, that measurement stays as it was while `threshold` is 1.01 and the student answers none of the task), `calibrated` whether the threshold is the task's own (enough calibration rows) or never below the global one, and `disabled` why the release gate disabled the task on the live version (a string) or `null`. These (all but `threshold`) are `null` when the live version has no per-task record for the task (scored before per-task calibration, or no golden rows of it). two more come from the same record ([free-text tasks](/reference/configuration/#free-text-tasks)): `precision_kind` says what `precision` means, `exact` (agreement under comparators that decide it) or `proxy` (a task with a `similarity` field: the share of served answers close enough to the teacher's, not checked correct), and `judge_pass_rate` is the judge's pass rate on the answers the live version would serve on the task's held-out rows (free-text tasks scored with a judge only, else `null`). Both are `null` where the others are, and for a record written before. `compare_keys` lists the paths of the fields the task's answers are compared on (empty for a task that compares whole answers, such as `default`), and `compare_source` says where they come from: `schema` for an auto task derived from a `response_format` schema ([Structured outputs](#structured-outputs)), `registry` for a built-in, file or API task, `null` for a name that is not registered. - `versions`: every version, oldest first. `eval` is the version's latest golden eval (the nightly re-score's for the live version); since v0.5 it also has `certificate`, `frontier`, `per_task`, `test_vector` and, for a candidate trained while a version was live, `paired` ([Certificates](#get-ratchetversionsidcertificate)), and its `threshold` is exact rather than rounded. `eval` is its golden evaluation (with one record per task in `eval.tasks`, [per-task calibration](/reference/configuration/#per-task-calibration)). A task with critical classes ([risk-aware routing](/reference/configuration/#risk-aware-routing)) has `critical` in its record: one entry per class, in the spec's order, with the condition as written (`field` and `equals` or `in`), `max_miss_rate`, `n` (its held-out rows whose teacher reference is in the class), `misses` (those the student would serve with an answer outside it, at the threshold the record stores and past its rules), `miss_rate` (`null` when `n` is 0) and `upper_bound` (the 95% Wilson upper bound, 1.0 when `n` is 0). `excluded_ineligible`, when present, counts golden rows left out because the task's routing block keeps such a request from the student. For a task with `escalate_when` rules, a record's `precision` and `coverage` (and the pooled ones) count only the answers no rule sends to the teacher. `other_tasks`, present only on an eval scored with `RATCHET_PER_TASK_CALIBRATION=0` that has records (the tasks with critical classes), holds the other tasks' held-out rows (`test_n`) and how many of them the pooled threshold serves (`served`). `bom` is `signed`, `unsigned`, or `null` for versions trained before v0.4. `evidence` (v0.5) is what the release gate reads for the version in `canary` or `live` state, recomputed from its audit counts on every call (`ReleaseGate.evidence`); `null` for every other state. Anyone can re-check a gate decision from it: | Key | Meaning | |---|---| | `rule` | The canary rule in force: `wilson` (the default) or `sequential` (`RATCHET_SEQUENTIAL=1`) | | `promote_rule` | `point` (point estimate at or above the floor) or `evidence` (`RATCHET_PROMOTE_EVIDENCE=1`) | | `alert_rule` | The live alert's rule: `eprocess` (the default) or `wilson` | | `floor` | `RATCHET_LIVE_FLOOR` | | `n`, `s` | Audits of answers the version served over its whole life, and how many agreed with the teacher | | `n_stage`, `s_stage` | The same in its current stage (for a live version: since it went live) | | `e_rollback` | e-value against "precision ≥ floor" over its life: the sequential rule rolls back at `thresholds.rollback` or more | | `e_alert` | The same over the current stage: the live alert fires at `thresholds.alert` or more | | `e_promote` | e-value against "precision ≤ floor" over the current stage: evidence-based promotion needs `thresholds.promote` | | `thresholds` | 1/alpha for each: `rollback` (`RATCHET_ALPHA_ROLLBACK`), `promote` (`RATCHET_ALPHA_PROMOTE`), `alert` (`RATCHET_ALPHA_ALERT`) | | `cs` | Anytime-valid 95% confidence sequence for precision over its life, on a 0.001 grid | | `wilson` | 95% Wilson interval of the current stage (what the Wilson rule reads; valid for one look only) | e-values are finite (capped at e^690). With no audits `n` is 0, the e-values are 1 and both intervals are `[0, 1]`. - `clients`: one entry per API key name that sent requests, busiest first (`Ledger.stats_by_client`). Empty when keys are off. `end_users` is how many distinct end users its requests named (by hash; only rows still kept count, since folded rows keep no end user); `stored_share` the share of its requests whose teacher answer was stored or matched a stored example (`null` when no request recorded it, as before v0.5). In a scrubbed copy `client` is a pseudonym. Since v0.5 each entry also has `spent_today_usd`, the key's chat-request dollars since 00:00 UTC (failed requests' spend included), and `daily_cap_usd`, its daily US dollar budget (`null` when it has none, when the name is no longer a key, and in a scrubbed copy; [Limits and headers](#limits-and-headers)). - `privacy`: whether redaction is on, how many stored examples had at least one identifier replaced, and how many placeholders were inserted in total (`Ledger.privacy_stats`); how many requests were under a storage opt-out, in all and by reason (`not_stored_requests`, `not_stored_by_reason`; `Ledger.opt_out_stats`), how many end users are registered as no-train (`no_train_end_users`) and how many of those registrations match no request under the current keys (`no_train_unmatched`: scoped to a key name no longer in `RATCHET_API_KEYS`, or keyless while keys are on); and the opt-outs in force: `store_examples` (false under zero retention), `no_store_tasks` (`RATCHET_NO_STORE_TASKS` and tasks whose spec says `store: false`), `no_store_clients` (keys with `store=0`) and where the end-user key comes from (`env` for `RATCHET_END_USER_KEY`, `ledger` for the generated one); the retention windows in force (`example_retention_days`, `request_retention_days`, 0 = forever) and `erasure_shared`; examples whose text was purged, in all and by reason (`purged_examples`, `purged_by_reason`: `retention`, `erasure`), erasures recorded, erased examples offered again and refused (`erasure_refusals`), and request rows folded into daily sums (`rolled_up_requests`; `Ledger.retention_stats`); how identifiers are written in stored examples (`redaction_style`: `label`, `numbered` or `surrogate`, from `RATCHET_REDACT_STYLE`; `redaction_task_styles`: the tasks whose spec overrides it; `redaction_plugin`: whether a detector plugin runs, never its name). `examples` counts the examples that still have their text. Counts and names only: no end-user hash is ever shown. - `security`: whether API keys are required, the daily quota per key (`null` = none), and whether adapter BOMs are signed (with the key id: the first 16 hex characters of the public key's SHA-256). Since v0.5 also whether admin keys and dashboard keys are set (`admin_keys`, `dashboard_keys`), whether the dashboard API is public (`public_dashboard`), and whether this payload is a scrubbed copy (`scrubbed`: client names are pseudonyms `client-1`, ..., and event details carry no URL or error message; [Access levels](#access-levels)); whether any key has a limit of its own (`key_limits`), the per-IP rate of a gateway without keys (`ip_rpm`, `null` when off) and the in-flight cap (`max_inflight`, `null` when off). - `budget` (v0.5): the gateway's own daily budget (`daily_usd`, `null` when `RATCHET_DAILY_BUDGET_USD` is 0), what happens past it (`mode`: `block` or `student_only`), what chat requests spent since 00:00 UTC (`spent_today_usd`; Scout is not counted), what is left (`remaining_usd`, `null` without a budget), whether it is spent (`exceeded`) and the shares that write a `budget` event (`alert_pct`). A scrubbed copy leaves the operator's budget out (`daily_usd` and `remaining_usd` are `null`), as the gateway's 429s do. - `events`: the last 40 events, oldest first. `id` is the ledger row id (stable, use it as a key); `ts` is Unix time. - `totals.cost_model` is `per_token`: `totals`, `days`, `tasks` and `clients` price the student per token (the hosted Nano price), as every figure since v0.1. The GPU-hour view is in `cost`. - `feedback`: human verdicts and labels ([Feedback](#feedback), `Ledger.feedback_stats`): verdicts in all, `correct` and `incorrect`, corrections sent and stored, examples that still have their text whose label a person wrote (`human_examples`, `verified` = 1: corrections and human imports) and golden-only examples (`golden_only_examples`), whether corrections may be stored (`corrections`, `RATCHET_FEEDBACK_CORRECTIONS`), and per version the human precision of the answers it served itself (route `student`) with its 95% Wilson bounds (`by_version`). Reported and alerted on (`human_alert`), never used to promote or roll back. Counts only, never a request id. - `totals.failed_cost_usd` (v0.5): what requests that failed spent without leaving a request row, such as the student's samples of an escalation whose teacher call then failed. It counts against the savings: `saved_usd = baseline_usd − cost_usd − failed_cost_usd` and `savings_pct = 1 − (cost_usd + failed_cost_usd) / baseline_usd`. Without failures both are what they were before. `days`, `tasks` and `clients` leave it out. - `errors.last_hour` (v0.5): the failures of the last hour by stage (`failures.stage`: `teacher`, `student`, `audit`, `shadow`, `quota`, `auth`, `budget`, `rate`, `upstream`, `gateway`), busiest first; `{}` when nothing failed. A failed student call the teacher covered counts here too, although the client got its answer. [RUNBOOK.md](/reference/runbook/#failures-and-request-ids) explains each stage. - `sidecar` (v0.5, [Off-path checks](#off-path-checks)): the queue that runs audits and shadows after the response. `workers` (0: every check runs before the response, mock mode's default), `max_queue`, `audit_places` (the places only a job with an audit may take) and `max_bytes` (the request text queued and running jobs may hold) are the settings; `queued` (places taken, jobs waiting included), `running` and `bytes` (the request text their jobs hold) are now; since start, `submitted`, `done`, `failed` (a job that raised; logged by class), `full` (a place refused because the queue or its byte budget was full), `refused` (a job handed over after shutdown began, which its request ran itself) and `abandoned` (still queued when the shutdown's drain ended: those rows stay pending). `by_version` lists every routed version, and any other version that lost something, with what could not go through the queue: `audits_inline` (drawn audits that ran before the response because the queue was full; an audit is never dropped), `shadows_dropped` (shadows that did not run for that reason), and `audits_abandoned` and `shadows_abandoned` (given up at shutdown). Counts are per process: a restart starts them at 0. - `controls` (v0.5): the operator's runtime switches ([Admin](#admin)): whether the student is turned off (`student_disabled`), releases are frozen (`releases_frozen`) and training is paused (`training_paused`), the runtime audit rate (`audit_rate`, `null` while `RATCHET_AUDIT_RATE` applies) and the tasks pinned to the teacher at runtime (`pinned_tasks`, sorted; such a task also shows `pinned: true` under `tasks`). - `drift` (v0.5): the [drift monitor](#get-ratchetdrift)'s summary: whether it is on (`RATCHET_DRIFT_ALARM` above 0), its threshold, the chance that any stream it holds ever alarms on unchanged traffic (`family_bound`, K / threshold), how many streams alarmed (`alarms`) and how many it holds (`streams`, a count; the list is in `GET /ratchet/drift`). Off: `enabled` false, `threshold` and `family_bound` `null`, both counts 0. - `teacher_watch` (v0.5): the [teacher watch](#teacher-watch)'s summary: whether it acts (`RATCHET_TEACHER_WATCH`), the check epoch, whether Night School is frozen, the epoch's checks and disagreements, the teacher's self-agreement and the agreement ceiling (`null` without checks), the freeze rule's e-value and the contested labels. The same keys as `GET /ratchet/teacher`, without the per-source and per-task rows. - `reasoning`: Nemotron's thinking mode by role (`student`, `fallback_student`, `serve`, `check`: `default`, `off`, `on` or `medium`) and the body `field` Token Factory-bound calls use ([Reasoning](/reference/configuration/#reasoning-nemotron-thinking)). - `cost`: the [cost report](#cost)'s headline numbers for all days: the headline model (`RATCHET_STUDENT_COST_MODEL`), the student endpoint's recorded hours, the savings share by GPU-hours paid (negative when self-hosting does not pay; `null` without a baseline), the break-even volume (`null` until the student has served) and the recommendation. - `kpis`: the [success metrics](#get-ratchetkpisdaysn) over the last `RATCHET_KPI_WINDOW_DAYS` days with traffic, exactly as `GET /ratchet/kpis` returns them. Compatibility: the keys the dashboard reads are listed with their JSON types in `tests/fixtures/stats_keys.json`, and `test_stats_contract` checks them on every run. New versions add keys; they do not remove or retype existing ones, so a client written against this page keeps working. Freshness: in live mode the payload is computed at most once every `RATCHET_STATS_TTL_S` seconds (default 2) by one request at a time, and every other read in between gets that payload; in mock mode (default 0) every read computes it. On a ledger file it is read from a read-only connection, so a dashboard never waits for requests nor slows them ([ARCHITECTURE §11](/reference/architecture/#11-hot-path-and-performance)). ## Request lookups What the ledger knows about one request, by the id the gateway gave it ([Request ids](#request-ids)): its `requests` row (metadata only; the table holds no prompt or answer text) and its `failures` rows. Either id form works: `req_<32 hex>` or the completion's `chatcmpl-<32 hex>`. ```json { "request_id": "req_3f1c0a9b8e7d6c5b4a3f2e1d0c9b8a79", "request": {"id": 1204, "ts": 1792400000.1, "day": 9, "task": "ticket-triage", "served_by": "teacher", "student_version": 5, "student_model": "student-v5", "shadow_model": null, "confidence": 0.41, "escalated": 1, "audited": 0, "route": "confidence", "latency_ms": 2711.0, "wall_ms": 2713.2, "cost_usd": 0.00133, "baseline_cost_usd": 0.00131, "client": "acme", "client_request_id": "order-77-retry-1", "request_id": "req_3f1c0a9b8e7d6c5b4a3f2e1d0c9b8a79", "...": "every other column of the requests table but the end user's hashes (end_user, end_user_secondary)"}, "failures": [{"id": 88, "ts": 1792400000.1, "day": 9, "request_id": "req_3f1c0a9b8e7d6c5b4a3f2e1d0c9b8a79", "client": "acme", "task": "ticket-triage", "stage": "student", "error": "UpstreamError 504 upstream_timeout", "status": 504, "latency_ms": 15002.4, "cost_usd": 0.0}] } ``` `student_version` is the version the request was routed to; `student_model` is the model its student samples came from (the version's adapter, or the hosted fallback student's id without `RATCHET_STUDENT_BASE`; `null` when no sample came back and in rows logged before v0.5), and `shadow_model` the model behind the shadow verdict (set with `shadow_version`). Only rows whose model is the version's adapter, or `null`, count toward that version's release. `request` is `null` for a request that failed before the cascade logged it (a teacher error after the student's samples, a quota 429, an unhandled error); then its `failures` row says where it failed, what it spent (`cost_usd`) and the status (`status`: the status of the stage that failed; for an upstream stage the upstream's, such as 504 for a timeout or 503 for an open breaker). `failures` also lists the optional calls that failed around an answered request: a student call the teacher covered, a failed audit, shadow sample or arbitration. `error` is the exception's class, status and code (`UpstreamError 502 upstream_unreachable`, `APIError 429 daily_quota_exceeded`), never a message. A failed request's rows are written before its response starts, so a lookup made as soon as the error arrives finds them; under a flood the gateway keeps at most 120 of its own rows a minute per stage (a row that carries spent money is always kept), so the id of a refusal past that is a 404. A request that is not found, and an id that is not one of the two forms, is a 404. ### GET `/ratchet/requests/{request_id}` Access level `dashboard`: any request, whoever sent it. A scrubbed read ([Access levels](#access-levels)) gets the `client` names as pseudonyms. 404 `{"detail": "no request has this id"}` when there is nothing to show. ### GET `/v1/ratchet/requests/{request_id}` Access level `api`: the caller's own requests only. With `RATCHET_API_KEYS` set, `request` and `failures` keep only what this key sent, and an id another key sent is a 404 `not_found_error` with code `request_not_found`, the same answer as for an id that never existed, so a key learns nothing about other clients. With keys off, it finds the requests sent without a key. ```python r = httpx.get(f"{BASE}/v1/ratchet/requests/{completion.id}", headers={"Authorization": f"Bearer {KEY}"}) print(r.json()["request"]["route"]) # student, confidence, cold_start, ... ``` ## GET `/metrics` Prometheus metrics in the text exposition format 0.0.4 (`Content-Type: text/plain; version=0.0.4; charset=utf-8`), from `ratchet/metrics.py`. Access level `metrics`: an admin or dashboard key, or `RATCHET_METRICS_TOKEN` as the bearer token (Prometheus' `authorization: {credentials: ...}` or `bearer_token`), or anyone when the dashboard is public (an anonymous scrape of a public live dashboard gets client pseudonyms in the `client` labels). | Metric | Type | Labels | Meaning | |---|---|---|---| | `ratchet_requests_total` | counter | `task`, `served_by`, `route`, `status` | Requests. The cascade counts each answered one with `status="200"` (chat requests, and Scout's agents, which call it in-process); the gateway counts each chat request that ended in an error, with `served_by="none"`, `route="none"` and its status. A replay by `Idempotency-Key` ran no cascade and is counted in `ratchet_idempotent_requests_total` only | | `ratchet_request_seconds` | histogram | `served_by` | Wall-clock time per request (`requests.wall_ms` for answered ones; `none`: until the error) | | `ratchet_upstream_requests_total` | counter | `upstream`, `outcome` | Model calls: `upstream` is `teacher`, `fallback_teacher`, `student`, `shadow`, `audit` or `arbitration`; `outcome` is `ok`, `error`, `timeout` or `circuit_open` (refused by the breaker, no call made) | | `ratchet_upstream_seconds` | histogram | `upstream` | Time per model call made, the provider's retries included | | `ratchet_cost_usd_total` | counter | `client` | US dollars spent, failed requests' spend included (per-token prices; mock in mock mode) | | `ratchet_failed_cost_usd_total` | counter | `client` | What requests that failed without an answer spent (the student's samples before a teacher failure); also in `ratchet_cost_usd_total` and in `/ratchet/stats`' `totals.failed_cost_usd` | | `ratchet_baseline_usd_total` | counter | `client` | What the answered requests would have cost with the teacher alone | | `ratchet_escalations_total` | counter | `task` | Requests the student tried and the teacher answered | | `ratchet_audits_total` | counter | `result` | Teacher audits of served student answers: `agree`, `disagree`, `invalid` (the teacher's answer measured nothing), `error`, `circuit_open` | | `ratchet_examples_stored_total` | counter | `task`, `split` | New training examples (`train` or `golden`); a duplicate's `seen` bumped is not one | | `ratchet_failures_total` | counter | `stage` | Failures by stage (`failures.stage`), every one: the table samples breaker refusals 1 in 10 and throttles the gateway's rows, this counter does not | | `ratchet_rejections_total` | counter | `reason` | Chat requests refused before the cascade: `auth`, `invalid_request`, `unknown_task`, `task_not_allowed`, `quota`, `budget` (a key's or the gateway's dollar budget), `rate` (a key's or an address's rate or in-flight limit), `overloaded` (the in-flight cap's 503), `too_large`, `disconnected`, `forbidden`, `idempotency` (a malformed or reused `Idempotency-Key`, or the 409 while its first request runs), `other` | | `ratchet_idempotent_requests_total` | counter | `outcome` | Chat requests with an `Idempotency-Key` ([Idempotency and retries](#idempotency-and-retries)): `first` (ran, answer kept), `uncached` (ran, answer not kept), `failed` (ran and failed; the key is released), `replayed` (sent from the cache), `joined` (waited for the first and got its answer), `reused` (400), `in_flight` (409), `invalid` (400) | | `ratchet_seen_skipped_total` | counter | `reason` | Repeats of a stored example not counted in its `seen`: `retry` (`x-stainless-retry-count` above 0) or `window` (the same API client within `RATCHET_SEEN_DEDUP_S`) | | `ratchet_sidecar_jobs_total` | counter | `outcome` | Off-path jobs (a request's audit, shadow samples and arbitration; [Off-path checks](#off-path-checks)): `queued`, `done`, `failed` (the job raised; logged by class), `abandoned` (still queued when the shutdown's drain ended) | | `ratchet_sidecar_full_total` | counter | `outcome` | Requests whose off-path work found the queue full: `audit_inline` (the drawn audit ran before the response) or `shadow_dropped` (the shadow did not run for this request) | | `ratchet_sidecar_job_seconds` | histogram | | Time from a job's queueing to its end: how long a row's checks stay pending | | `ratchet_sidecar_jobs` | gauge | `state` | Off-path jobs `queued` (places taken, waiting jobs included) and `running`, at scrape time | | `ratchet_circuit_open` | gauge | `breaker` | 1 while a breaker (`student`, `shadow`, `teacher`, `fallback_teacher`, ...) is open or half-open, else 0 | | `ratchet_version_traffic_pct` | gauge | `version`, `state` | Traffic share of the live, canary and shadow versions (`v5`, `live`, 100) | | `ratchet_spend_today_usd` | gauge | `client` | US dollars spent since 00:00 UTC, failed requests included | | `ratchet_inflight_requests` | gauge | | Chat requests in flight at the scrape; `RATCHET_MAX_INFLIGHT` refuses past it with a 503 ([Limits and headers](#limits-and-headers)) | | `ratchet_info` | gauge | `version`, `mode` | Always 1 | Label values stay a small set: `task` is a registered task's name, `other` for a name the registry does not hold, or `none` before a request named one; `client` is a `RATCHET_API_KEYS` name, `none` (no key: an open gateway, Scout) or `other`. Counters count from the start of the process (a test run reset keeps them); use `rate()` or `increase()`. Validation 400s are counted here (`ratchet_requests_total{status="400"}`, `ratchet_rejections_total`) and nowhere else. Alert examples are in [RUNBOOK.md](/reference/runbook/#alerts). ```yaml scrape_configs: - job_name: ratchet metrics_path: /metrics authorization: {credentials_file: /etc/prometheus/ratchet_token} # RATCHET_METRICS_TOKEN static_configs: [{targets: ["ratchet:8000"]}] ``` ## GET `/ratchet/kpis?days=N` The success metrics and the latency target, computed from the ledger by `ratchet/kpis.py` the same way as in the simulator and in CI. Access level `dashboard`, like `/ratchet/stats`, whose `kpis` block is this report. success metrics 1 to 3 and the latency target cover the last `days` ledger days with traffic (1 to 3,650, default `RATCHET_KPI_WINDOW_DAYS`, 3; anything else is **400**, a non-number **422**) cover the whole history. An example from the mock-mode gateway (numbers from mock models, `simulated: true`): ```json { "mode": "mock", "simulated": true, "window_days": 3, "window": {"from": 8, "to": 10, "days": 3, "complete": true}, "metrics": { "SM-1": {"name": "Cost reduction vs all-teacher, steady state", "value": 0.845, "target": 0.7, "op": ">=", "unit": "fraction", "met": true, "detail": {"cost_usd": 1.018, "failed_cost_usd": 0.0, "baseline_usd": 6.559, "requests": 6000, "cost_model": "per_token"}}, "SM-2": {"name": "Audited precision of student-served answers", "value": 0.985, "target": 0.95, "op": ">=", "unit": "fraction", "met": true, "detail": {"audits": 585, "agree": 576, "lower": 0.971, "upper": 0.992, "min_audits": 20}}, "SM-3": {"name": "Student share of traffic, steady state", "value": 0.969, "target": 0.8, "op": ">=", "unit": "fraction", "met": true, "detail": {"requests": 6000, "student_served": 5814}}, "SM-4": {"name": "Bad deploys reaching 100% traffic", "value": 0, "target": 0, "op": "==", "unit": "versions", "met": true, "detail": {"reached_full_traffic": 4, "bad_versions": []}}, "SM-5": {"name": "Time to recover from drift (escalations back to baseline)", "value": 4, "target": 3, "op": "<=", "unit": "nights", "met": false, "detail": {"days_measured": 9, "episodes": [{"start": 5, "end": 9, "nights": 4, "baseline": 0.022, "peak": 0.1405}]}}, "latency": {"name": "Median latency of student-served requests", "value": 327.1, "target": 1000.0, "op": "<", "unit": "ms", "met": true, "detail": {"requests": 5814, "column": "latency_ms", "quantile": 0.5, "served_by": "student"}} }, "summary": {"met": 5, "not_met": 1, "unknown": 0} } ``` - Every metric has `name`, `value`, `target`, `op` (`>=`, `<=`, `==`, `<`), `unit` (`fraction` for a share, else `versions`, `nights` or `ms`), `met` and `detail`. `met` is `null` when there is not enough data to say, with a `detail.reason`: no traffic in the window, fewer audits than `RATCHET_MIN_AUDITS`, no version at 100% yet, fewer than four days on which the student judged at least 50 requests, or a drift episode still open within its target. `value` is `null` when there is nothing to measure, and for success metric 5 while an episode is open (`detail.open_nights` counts it so far). - `simulated` is true in mock mode, where every number comes from mock models. `window` gives the first and last day used and whether the ledger had `window_days` days with traffic (`complete`). - **success metric 1** is `1 - (cost_usd + failed_cost_usd) / baseline_usd`: per token, as `totals`; `failed_cost_usd` is what failed requests spent without leaving a request row (0 on a ledger without a `failures` table). **success metric 2** pools every audit in the window, every version together, with a 95% Wilson interval (`lower`, `upper`); it is met on the point estimate. **success metric 3** is the student's share of the window's requests. - **success metric 4** counts versions that reached 100% of traffic (a `live` event, or live now) and were later named by an `alert` or a `rollback`, or are `rolled_back` now (`bad_versions`, each with its evidence and day). A canary rolled back before 100% does not count. - **success metric 5** is the longest drift episode in nights over the whole history. A day's escalation rate is the share of the requests the student judged (its samples came back) that it escalated; an episode starts on a day at least 5 points above the lowest of the 3 measured days before it and ends on the first later day at most 2 points above that baseline. - **the latency target** is the nearest-rank median latency of the window's student-served requests, from `requests.wall_ms` when the ledger records it, else `latency_ms` (`column`). Folded request rows keep no latencies. - New metrics may be added under `metrics`; a client should show an unknown one as it is. ## GET `/ratchet/bom` Access level `bom`: as `/ratchet/stats`, and public with `RATCHET_PUBLIC_BOM=1`. ```json { "generated_at": 1792400000.0, "examples_sha256": "07ac2ca35bac78959378cbabd98352540…", "sources": [ {"teacher_model": "nvidia/Nemotron-3-Ultra-550b-a55b", "license": "OpenMDW-1.1", "license_url": "https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16", "split": "train", "n": 2085} ], "label_sources": [ {"teacher_model": "human", "license": "customer-provided labels", "license_url": "", "split": "golden", "n": 120} ], "purged_examples": 61 } ``` `examples_sha256` is a SHA-256 over the hashes of the examples that still have their text, in sorted order, and `sources` counts those written by a teacher; `label_sources` counts those whose label a person wrote (provenance `human`: feedback corrections and imports), listed apart because they are not a teacher's; `purged_examples` counts the ones whose text a retention window or an erasure removed ([Erasure and export](#erasure-and-export)), so a purge changes the fingerprint. Two ledgers with the same kept examples give the same fingerprint. Each adapter's own BOM (below) is never changed. ## GET `/ratchet/versions/{id}/bom` Access level `bom`, like `/ratchet/bom`. The bill of materials of one adapter, written by Night School right after training and evaluation, from the exact rows it was trained and evaluated on. **404** for an unknown id or a version trained before v0.4. ```json { "manifest": { "schema": "ratchet.bom/v2", "generator": "ratchet 0.5.0", "created_at": "2026-10-20T00:15:12Z", "adapter": {"version": 1, "name": "student-v1", "base_model": "nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16", "day": 1}, "training": {"examples": 1029, "dataset_sha256": "478eef…", "tasks": {"ticket-triage": 420, "source-triage": 362, "page-read": 247}, "targets": "teacher answers only", "examples_with_pii_redacted": 0, "origins": {"live": 1029}, "commitment": {"alg": "ratchet.smt/v1", "root": "e8ab25…", "size": 1029}}, "golden": {"examples": 110, "sha256": "cbce0e…", "held_out": true, "human_labels": 0, "commitment": {"alg": "ratchet.smt/v1", "root": "1f6b2a…", "size": 110}}, "sources": [{"teacher": "nvidia/Nemotron-3-Ultra-550b-a55b", "license": "OpenMDW-1.1", "license_url": "https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16", "license_origin": "builtin", "distill_ok": true, "examples": 1029}], "label_sources": [], "base_model_license": {"name": "NVIDIA Nemotron Open Model License", "url": "https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/"}, "notices": [{"file": "NOTICE.md", "sha256": "5d0c3a…", "text": "# NOTICE\n\nThis folder holds a LoRA adapter…"}], "redactor": {"sha256": "7100b2…", "rules": 11, "redact_pii": true}, "eval": {"golden_n": 110, "golden_agreement": 0.89, "threshold": 0.65, "coverage": 0.96, "precision": 0.94, "test_n": 55} }, "sha256": "9b1f…", "signature": {"alg": "Ed25519", "key_id": "6cf6bc1a39c4491d", "public_key": "IiNlIP8Y…", "value": "k2Hc…"} } ``` - `sha256` and the signature cover the **canonical** manifest: JSON with sorted keys, no whitespace, UTF-8 (`ratchet.bom.canonical`). - `signature` is `null` when `RATCHET_BOM_SIGNING_KEY` is not set; the `sha256` still detects edits. - `training.dataset_sha256` equals the version's `dataset_hash`; `golden.sha256` fingerprints the golden rows the evaluation used. The `eval` block is the evaluation at training time; later re-checks update `/ratchet/stats` but not the signed BOM. Since v0.5 it also carries the certificate, the frontier, the per-task bounds, the per-row outcomes (`test_vector`) and, for a candidate trained while a version was live, the paired comparison with it ([Certificates](#get-ratchetversionsidcertificate)); `python -m ratchet.stats check bom.json` recomputes every one of those numbers from the signed counts. - Schema `ratchet.bom/v2` keeps every v1 field and adds `training.origins` (training rows by where they came from; `live` for rows that did not record it) and `redactor` (a SHA-256 over the redaction rules, the number of rules, and whether redaction was on; `null` when the writer did not say). Later features add optional sections: `teacher`, `backfill_jobs` and `erasures_honored`. Adapters trained before keep their v1 documents, which still verify; any other schema is refused as unknown. - `training.commitment` and `golden.commitment` (since v0.5): `{"alg": "ratchet.smt/v1", "root", "size"}`, the root of a sparse Merkle tree over the split's examples (each example's hash mapped to a hash of its label, teacher, license and redaction count) and how many there are. `size` always equals `examples`. A single example can be proved in or out of either set against these roots ([Committed training sets](#committed-training-sets)). In live mode a copy of the document is also written into the adapter's folder (`adapters//ratchet_bom.json`, with its `NOTICE.md`), so the weights carry their provenance. - The licenses travel with the adapter: `base_model_license` is the license of the base model the adapter runs on (`{name, url}` from `ratchet.licenses.BASE_MODELS`; for a base model Ratchet does not know, `name` and `url` are `null` and `warning` says to find and ship its license); `notices` holds the exact `NOTICE.md` that goes next to the adapter weights (the base model's license, each teacher's license and URL, teacher outputs only, the Ratchet version) with its SHA-256, so the file can be checked against the signed BOM; each source's `license_origin` is `builtin` (Ratchet's registry) or `config` (added by the operator in `RATCHET_TEACHER_LICENSES`). v2 documents written before these fields still verify. `python -m ratchet.live export --version N --out DIR` writes the adapter folder with this BOM and its `NOTICE.md` ([DEPLOYMENT.md](/reference/deployment/#export-an-adapter-with-its-licenses)). - Human labels are not teachers: `sources` lists teachers only, and `label_sources` lists the training rows whose label a person wrote (`[{"source": "human", "license": "customer-provided labels", "note", "distill_ok": true, "examples", "origins": {"feedback": n, "import": n}}]`; empty without any). With human labels among the targets, `training.targets` is `teacher answers and customer-attested human labels` and the NOTICE gets a **Human labels** section; an adapter without them gets the same text as before. `golden.human_labels` counts golden rows whose reference a person wrote (a human-verified golden set). v2 documents written before these fields still verify. Verify a downloaded BOM against the public key you published: ```bash curl -s https:///ratchet/versions/3/bom > bom.json python -m ratchet.bom verify bom.json --public-key "" # OK: signature valid (Ed25519, key 6cf6bc1a39c4491d, trusted key) python -m ratchet.bom verify bom.json --public-key "" --notice NOTICE.md # also the adapter's NOTICE.md # OK: NOTICE.md matches the BOM ``` Without `--public-key` the check uses the key embedded in the document, which only proves the document is consistent with itself. ## GET `/ratchet/versions/{id}/certificate` What a version's golden evals certify about the answers it serves. The threshold is chosen on the even half of the golden rows and measured on the odd half, so the odd half's counts at that threshold are one exact binomial test: with probability at least 1 − `delta`, the precision of what the version serves at its threshold is at least `lcb`. **404** for an unknown version. Access level `dashboard`, like `/ratchet/stats` ([Access levels](#access-levels)). ```json { "version": 3, "state": "live", "status": "certified", "training": { "certificate": {"method": "clopper-pearson", "calibration": "point", "delta": 0.05, "target": 0.95, "threshold": 0.5, "served": 119, "errors": 1, "precision": 0.9916, "lcb": 0.9608, "certified": true}, "frontier": [ {"threshold": 0.95, "coverage": 0.776, "served": 97, "errors": 0, "precision": 1.0, "lcb": 0.9468, "delta": 0.005}, {"threshold": 0.9, "coverage": 0.824, "served": 103, "errors": 0, "precision": 1.0, "lcb": 0.9499, "delta": 0.005} ], "per_task": { "page-read": {"n_test": 31, "served": 29, "errors": 0, "precision": 1.0, "lcb": 0.9019, "coverage": 0.9355}, "source-triage": {"n_test": 42, "served": 38, "errors": 0, "precision": 1.0, "lcb": 0.9242, "coverage": 0.9048}, "ticket-triage": {"n_test": 52, "served": 52, "errors": 1, "precision": 0.9808, "lcb": 0.912, "coverage": 1.0} }, "threshold": 0.5, "calibration": "point", "paired": {"n": 125, "cov_only_live": 5, "cov_only_cand": 8, "p_cov": 0.8666, "neg_flips": 1, "pos_flips": 3, "p_err": 0.9375, "neg_flips_served": 1, "live_served_ok": 113, "nfr": 0.0088, "nfr_95": [0.0016, 0.0484], "against": 2} }, "current": {"certificate": {"…": "the same keys"}, "frontier": ["…"], "per_task": {}, "threshold": 0.5, "calibration": "point", "paired": null, "rechecked_day": 5}, "bom": {"sha256": "dd71bf42…", "signed": false}, "live": {"rule": "wilson", "floor": 0.92, "n": 117, "s": 113, "cs": [0.89, 0.996], "…": "ReleaseGate.evidence"}, "drift": {"alarm": false, "threshold": 100.0, "streams": [{"version": 3, "task": "ticket-triage", "n": 2611, "log10_martingale": -2.1383, "…": "as in GET /ratchet/drift"}]}, "assumptions": ["Golden rows are assumed exchangeable with live traffic, …", "Precision means agreement with the teacher model …", "Which requests are golden is decided by a hash of the request, …"] } ``` (Mock gateway after five mock days of mixed traffic; numbers from mock models, rounded here.) - `status`: `certified` or `not certified` from the latest certificate (`current`'s when the version was re-scored, else `training`'s); `suspended` while a drift alarm is open for the version: its own traffic stopped looking like its earlier traffic, so the golden rows no longer stand for what it serves. The certificate is still shown. An alarm stays open for the version's life; the next version gets new streams. `no certificate` for a version evaluated before v0.5 or by an evaluator that writes none (also under an alarm). - `training`: from the signed BOM's `manifest.eval`, written once at training time; `null` without a BOM. `current`: from the version's eval, which the nightly champion re-score overwrites for the live version (with `rechecked_day`, `null` until then). Each holds `certificate`, `frontier`, `per_task`, `threshold`, `calibration` and `paired` (`null` when that eval lacks one). - `certificate`: the Clopper–Pearson certificate at the version's threshold, on the held-out half: `served` answers, `errors` among them, `precision`, the one-sided lower bound `lcb` at level `delta` (`RATCHET_CERT_DELTA`), and `certified` (`lcb ≥ target`, `RATCHET_CERT_TARGET`). `calibration` is how the threshold was chosen: `point` (v0.4's rule, 97% on the calibration half) or `ltt` (Learn-then-Test over a fixed grid, `RATCHET_CALIBRATION=ltt`), which adds `ltt_trace` (each grid threshold tested on the calibration half, `{threshold, n, k, p_value}`, highest first, stopping at the first that fails). - `frontier`: the same counts at each threshold of the fixed grid 0.95, 0.90, …, 0.50, each `lcb` at `delta / 10` (Bonferroni), so every row holds at once: an operator may pick any threshold on it afterwards and keep the guarantee. - `per_task`: the certificate's counts per task, each bound at `delta` on its own. A diagnostic that shows a weak task inside a pooled number; it is not corrected for the number of tasks and never changes routing. - `paired` (a candidate trained while a version was live): the exact comparison with that version (`against`) on the same held-out rows. `cov_only_live` and `cov_only_cand` are the rows only one of them serves, with `p_cov` the one-sided McNemar p-value for "the candidate serves less"; `neg_flips` are rows the candidate answers wrong where live's answer was right (served right, or escalated to the teacher), `pos_flips` the reverse, with `p_err`; `nfr` is `neg_flips_served / live_served_ok` (answers live served right that the candidate serves wrong), with its 95% Wilson interval `nfr_95`. The opt-in paired gate reads these ([Configuration](/reference/configuration/#release-policy-ratchetreleasepy)). - `bom`: the BOM's `sha256` and whether it is signed; `null` without one. `live`: the release gate's evidence from the version's audits (the same object as `/ratchet/stats`' `evidence`) for a version that serves or once served (`canary`, `live`, `rolled_back`, `retired`), else `null`. `drift`: the drift monitor's view of the version ([Drift](#get-ratchetdrift)): `alarm` (true while any of its streams has alarmed), the `threshold`, and its streams without trajectories; `null` when the monitor is off or the version never logged a confidence (a candidate or shadow). - `assumptions`: what every certificate assumes, in plain words. Precision is agreement with the teacher, not ground truth; golden rows over-represent hard requests, so a certificate is conservative for easy traffic only to the extent that exchangeability fails in that direction. Check a downloaded BOM's numbers yourself: ```bash curl -s https:///ratchet/versions/3/bom > bom.json python -m ratchet.stats check bom.json --public-key "" # OK: signature valid (Ed25519, key 6cf6bc1a39c4491d, trusted key) # OK: certificate: 118 of 119 served answers agree at threshold 0.5 (delta 0.05): lower bound 0.9608 vs target 0.95, certified # OK: frontier: 10 thresholds, each bound at delta 0.005 (Bonferroni) # OK: per_task: 3 tasks # OK: paired against v2: p_cov 0.8666, p_err 0.9375 # check: every number recomputed from the signed counts ``` It exits 1 when the signature or hash fails or when any recomputed number differs from the signed one by more than 1e-9, and 0 for a BOM written before v0.5 (no certificate to check). ## Committed training sets Every adapter's BOM signs one sparse Merkle root per split: the keys are the example hashes of exactly the rows it was trained on (`training.commitment`) and graded on (`golden.commitment`), each mapped to the SHA-256 of `{completion, teacher, license, redactions}` (the stored teacher answer, who wrote it, its license and the number of PII placeholders in it). The leaves are written once to the ledger (`version_leaves`, append-only) before the BOM is signed. From them the gateway answers whether one example is in a set, with a proof anyone can check offline against the signed root, and gives a certificate that no golden example was trained on (invariants 3 and 6). Both endpoints need an admin key ([Authentication](#authentication)): for a text an attacker can guess, a yes-or-no answer about the training set is a leak. **404** for an unknown version, and **404** `commitment not available: version N was trained before set commitments (v0.5)` for a version whose BOM has no commitment. What a proof shows: the operator's binding commitment, at training time, to the dataset the adapter was trained and graded on. It does not show that the weights came from that dataset. ### POST `/ratchet/versions/{id}/proof?split=train` `split` is `train` (the default) or `golden`; anything else is **400**. The body names the example in one of two ways: - `{"key": "<64 hex characters>"}`: the example's hash (`python -m ratchet.proof key` computes it). - The chat request itself: `messages` with its `task` (or `metadata.ratchet_task`; default `default`) and any other `/v1/chat/completions` field. It is validated, normalised and, when `RATCHET_REDACT_PII` is on, redacted exactly as the gateway stores a teacher answer to it, and its `stop`, `response_format`, `tools` and `tool_choice` are part of the key (other sampling fields are not). A request with a content part only the teacher reads is **400**: such requests are never stored, so no adapter was trained on one. Anything else (both forms, neither, a key that is not 64 hex characters, a request the gateway would refuse) is **400** with the reason; a body that is not JSON is **422**. ```bash curl -s "localhost:8000/ratchet/versions/3/proof?split=train" -H "Authorization: Bearer $ADMIN_KEY" \ -H 'content-type: application/json' -d '{"task": "ticket-triage", "messages": [...]}' > proof.json ``` ```json { "version": 3, "split": "train", "key": "009caefc0abdc8dd8033a716ed8e7861762d4a29d033afd88573f6af77967df9", "alg": "ratchet.smt/v1", "root": "e8ab2527eed2c08e…", "size": 1288, "result": "included", "value": "e43228f5b4cc42b4b61ef7fa6128142aa78b7eb2cd67b51e4858397ba615c460", "proof": {"siblings": ["92666e28efca7f96…", "c06a2dd0139ee1f1…", "b4715cc6b4fd8ce2…", "64a62adbe07cb66c…", "a5b2f8637ee380a5…", "d61a9c7573c0131a…"], "terminal": {"leaf": {"key": "009caefc0abdc8dd…", "value": "e43228f5b4cc42b4…"}}}, "bom_sha256": "301a48c00a468c60…" } ``` (Mock gateway after four mock days; hashes shortened here.) - `result` is `included` or `absent`. `value` (only when included) is the example's leaf value; anyone holding the example's stored answer, teacher, license and redaction count can recompute it. - `proof.siblings` are the sibling hashes from the root down (about log2 of `size`); `proof.terminal` is where the key's path ends: `{"leaf": {key, value}}` (the key itself when included; for an absent key, the one example that shares its path, whose key and value hashes the proof therefore discloses) or `{"empty": true}`. - `root` and `size` are the ones the version's BOM signs, and `bom_sha256` names that BOM. ### GET `/ratchet/versions/{id}/heldout` The held-out certificate: the golden set's leaves, and for each golden key a non-membership proof against the training root. ```json { "version": 3, "alg": "ratchet.smt/v1", "training": {"root": "e8ab2527eed2c08e…", "size": 1288}, "golden": {"root": "1f6b2a205ce04e6b…", "size": 133, "leaves": [["0001a3…", "5c9e10…"], ["…", "…"]]}, "proofs": {"0001a3…": {"siblings": ["…"], "terminal": {"leaf": {"key": "…", "value": "…"}}}, "…": {}}, "held_out": true, "bom_sha256": "301a48c00a468c60…" } ``` About 113 KB for 133 golden rows against 1,288 training leaves (mock gateway after four mock days). `held_out` is the gateway's own check; a verifier ignores it and checks the proofs. When the stored leaves no longer rebuild the root and size the BOM signs (the ledger file was edited outside Ratchet: SQLite's triggers refuse any edit through its API), both endpoints answer **500** and the gateway writes one `integrity` event per version and split; it never serves a proof that would fail against the signed root. ### Checking a proof offline ```bash curl -s https:///ratchet/versions/3/bom > bom.json python -m ratchet.proof verify proof.json --bom bom.json --public-key "" --expect included # OK: example 009caefc0abd IS in the training set of student-v3 (1,288 leaves, 6 siblings); BOM signature valid (Ed25519, key 6cf6bc1a39c4491d, trusted key) curl -s https:///ratchet/versions/3/heldout -H "Authorization: Bearer $ADMIN_KEY" > heldout.json python -m ratchet.proof heldout heldout.json --bom bom.json --public-key "" # OK: all 133 golden examples provably absent from the training set of student-v3 (1,288 leaves); BOM signature valid (...) python -m ratchet.proof key --task ticket-triage --messages request.json # the key a request is stored under ``` `verify` checks the BOM's hash and signature, that the proof is against the root the BOM signs for that split (never the root, `result` or `value` the file states, which only have to agree), and the proof itself. `heldout` rebuilds the golden root from the certificate's leaves, requires it and the size to match the BOM, and checks that every golden key is absent from the signed training root. Both exit 1 on any failure (a bad signature, a root mismatch, an invalid proof, an unexpected `--expect` result, a file that is not JSON) and 2 on bad input (a missing file). `key` reads a messages list or a request body (`--no-redact` for a gateway with `RATCHET_REDACT_PII=0`) and uses the gateway's settings from the environment. A BOM without commitments (v1, or written before v0.5) has nothing to check against: exit 1, `no set commitment`. ## GET `/ratchet/drift` The label-free drift monitor. For each version and task, the student's logged confidence is one stream. Each new score gets a conformal p-value (its rank among the stream's earlier scores, small when it is unusually low), and a gambler bets against "traffic has not changed" with those p-values. Its capital is a test martingale: while the traffic a version sees stays exchangeable, the chance that it ever reaches `threshold` is at most 1 / `threshold` (1% at the default 100), however long the stream runs and however often anyone looks. No labels and no teacher calls are needed; the release gate feeds it from the ledger in its tick, so the request path is untouched. Access level `dashboard`, like `/ratchet/stats` ([Access levels](#access-levels)). `?version=N` narrows `streams` to one version (**422** for a value that is not a whole number); the other numbers stay the monitor's. ```json { "enabled": true, "threshold": 100.0, "false_alarm_bound": 0.01, "family_bound": 0.03, "monitored": 3, "alarms": 1, "streams": [ {"version": 1, "task": "ticket-triage", "n": 3586, "log10_martingale": 14.7573, "peak_log10": 14.9509, "alarm": true, "alarm_at_request": 6129, "alarm_day": 5, "alarm_n": 3335, "last_request": 6600, "active": true, "trajectory": [[32, -0.6741], [64, -0.8717], [96, -0.9943], "…", [3584, 14.7458]]}, {"version": 3, "task": "ticket-triage", "n": 2611, "log10_martingale": -2.1383, "peak_log10": 0.0, "alarm": false, "alarm_at_request": null, "alarm_day": null, "alarm_n": null, "last_request": 9000, "active": true, "trajectory": [[16, -0.6724], [32, -0.7884], "…", [2608, -2.1378]]} ] } ``` (Mock gateway after six mock days of tickets with a product launch on day 5; numbers from mock models, one stream left out.) - `enabled`: false when `RATCHET_DRIFT_ALARM=0`; then `threshold`, `false_alarm_bound` and `family_bound` are `null`, the counts 0 and `streams` empty. - `false_alarm_bound`: 1 / `threshold`, for one stream. `family_bound`: min(1, K / `threshold`) with K = `monitored`, the chance that any stream the monitor holds ever alarms on unchanged traffic. It grows with every version and task; it is reported, not hidden. - A version has at most 16 task streams, opened in the order it first sees each task name; the rows of every later name share one more stream whose `task` is `*` (events call it "other tasks"). Task names come from clients (`X-Ratchet-Task`), so this keeps the monitor's memory and K bounded: at most 17 streams per version ([Threat model](/reference/threat-model/) T12). - `alarms`: streams that alarmed. Each stream alarms at most once (the alarm latches) and writes one `drift` event, once across restarts and processes. - Per stream: `n` confidences seen; `log10_martingale` and `peak_log10`, log10 of the capital now and at its highest (the alarm is at log10 `threshold`, 2 by default); `alarm_at_request` (the request id), `alarm_day` and `alarm_n` of the first crossing, or `null`; `last_request`, the last request id seen; `active`, false for an older version that no longer serves (only its final numbers are kept; should it serve again, its stream is replayed from the ledger first); `alarm_day` is the ledger day of the request that crossed, which is also the `drift` event's `day`, even when the monitor found it later by replaying the ledger; `trajectory`, at most 200 `[n, log10 capital]` points, evenly spaced. - The test is one-sided: confidence that rises makes the capital fall. An alarm says traffic changed for this version, not that the model got worse; audits measure that ([Runbook](/reference/runbook/#drift-event)). It suspends the version's certificate ([Certificates](#get-ratchetversionsidcertificate)) and, opt-in, boosts its audits (`RATCHET_DRIFT_AUDIT_BOOST`) and lets Night School train sooner (`RATCHET_DRIFT_EARLY_NIGHT`). ## Teacher watch Every number Ratchet reports is measured against the teacher, so the teacher is checked too. A request that repeats is labelled by the teacher again when it escalates or is audited; its first such re-test in a check epoch is compared with the stored label under the task's comparators, at no extra cost. Night School adds a few probes for traffic with few repeats (`RATCHET_TEACHER_PROBES`). Only labels that can be compared count: the same configured teacher at temperature 0, no redaction placeholders on either side, a registered task that names fields and is not pinned. A label a re-test disagreed with is **contested**: it is left out of training and the golden set until Night School's adjudication (two more teacher answers, a strict majority wins) settles it. When the epoch's first re-tests disagree too often, the release gate freezes Night School with a `teacher_drift` event: the freeze rule's e-value, a mixture over the epoch's days of `stats.e_bernoulli` against "a fresh label disagrees with an earlier one at most `RATCHET_TEACHER_NOISE_MAX` (5%) of the time", has reached 1 / `RATCHET_ALPHA_TEACHER` (100). A teacher within that noise freezes it with probability at most 1% per epoch, however often the gate looks. Training stays paused until an operator acknowledges ([Runbook](/reference/runbook/#teacher_drift-event)). A change in the model id the teacher's responses report starts a new epoch on its own (`teacher_model_changed`). ### GET `/ratchet/teacher` Access level `dashboard` and read-only, like `/ratchet/stats` ([Access levels](#access-levels)). ```json { "enabled": true, "epoch": 1, "epoch_started_day": 1, "frozen": false, "model": "nvidia/Nemotron-3-Ultra-550b-a55b", "noise_max": 0.05, "alpha": 0.01, "threshold": 100.0, "checks": 273, "disagreements": 11, "self_agreement": 0.9597, "interval": [0.9293, 0.9774], "ceiling": 0.9886, "e_value": 96.38, "strongest": {"from_day": 3, "checks": 48, "disagreements": 11}, "contested": 0, "sources": {"duplicate": {"checks": 193, "disagreements": 1}, "probe": {"checks": 80, "disagreements": 10}}, "tasks": [ {"task": "ticket-triage", "checks": 273, "disagreements": 11, "self_agreement": 0.9597, "interval": [0.9293, 0.9774], "ceiling": 0.9886, "e_value": 96.38, "contested": 0} ] } ``` (Mock gateway, four days of tickets, the mock teacher changing 15% of labels from day 3 (`POST /ratchet/demo/teacher-drift`); numbers from mock models, rounded here.) - `enabled`: `RATCHET_TEACHER_WATCH`. When 0 the checks are still recorded and reported, but nothing acts on them. - `epoch`, `epoch_started_day`: the check epoch (1 until the first ack or model change) and the ledger day it started. Each stored request gives at most one check per epoch: its first re-test. That is what keeps the e-value valid (comparing one stored label with several re-tests makes the checks dependent). - `frozen`: Night School skips training (`skip` event "teacher labels look unstable") until `POST /ratchet/teacher/ack`. - `model`: the model id the teacher's responses last reported (`null` before the first teacher call; at most 200 characters). - `noise_max`, `alpha`, `threshold`: the freeze rule (`RATCHET_TEACHER_NOISE_MAX`, `RATCHET_ALPHA_TEACHER`, 1 / alpha). - `checks`, `disagreements`: the epoch's first re-tests and how many disagreed. `self_agreement` is 1 − disagreements / checks, `interval` its 95% Wilson interval, and `ceiling` the square root of the interval's upper end: no student can agree with a fresh teacher label more often than that at full coverage. All three are `null` without checks. - `e_value`: the freeze rule's e-value (capped at e^690); `strongest`: the stretch of days from `from_day` on that weighs most in it, with its counts (`null` without checks). The `teacher_drift` event quotes these. - `contested`: stored labels quarantined now, of every epoch. - `sources`: the checks by where they came from: `duplicate` (a repeated request the teacher labelled again) and `probe` (Night School). - `tasks`: per task with checks or contested labels, the same numbers. A task's `e_value` is diagnostic only: the freeze tests the pooled checks. ### POST `/ratchet/teacher/ack` Starts a new check epoch, unfreezes Night School and writes a `teacher_ack` event naming the admin key (its name, never the key). Needs an admin key (`Authorization: Bearer ` or `X-API-Key`): **401** for a missing or wrong key, **403** for a regular API key, and **403** in live mode without `RATCHET_ADMIN_KEYS`. Open in mock mode without admin keys (the event names `mock-admin`). No body. Check the teacher's model id first ([Runbook](/reference/runbook/#teacher_drift-event)). ```json {"epoch": 2, "frozen": false} ``` ## GET `/ratchet/bom/public-key` ```json {"alg": "Ed25519", "public_key": "IiNlIP8YSYKDKqEHNzxBnn+m37imeLeykRmrZEEl5I4=", "key_id": "6cf6bc1a39c4491d"} ``` `public_key` and `key_id` are `null` when signing is off. Publish the key somewhere you control as well (README, website): a key served by the same gateway proves nothing if the gateway itself is compromised. ## POST `/ratchet/redact?style=` Runs the PII redactor ([ARCHITECTURE.md](/reference/architecture/#pii-redaction)) on a text and returns what would be stored: in the gateway's style (`RATCHET_REDACT_STYLE`) or the one `style` names (`label`, `numbered` or `surrogate`, any case). Nothing is logged or stored. Backs the dashboard's **Try the redactor** box. Access level `dashboard`; the detector plugin (`RATCHET_REDACT_PLUGIN`), which may be an NER model costing tens of milliseconds a call, runs only for a caller with a valid API or admin key (`Authorization: Bearer ` or `X-API-Key`). Anyone else, or a wrong key, gets the patterns alone. ```bash curl -s localhost:8000/ratchet/redact -H 'content-type: application/json' \ -d '{"text": "I am bob@example.com, card 4111 1111 1111 1111"}' ``` ```json {"text": "I am [EMAIL], card [CARD]", "found": {"EMAIL": 1, "CARD": 1}, "enabled": true, "style": "label", "plugin": false, "plugin_skipped": false} ``` ```bash curl -s 'localhost:8000/ratchet/redact?style=numbered' -H 'content-type: application/json' \ -d '{"text": "Sales: ana@x.io. Billing: bo@y.io, cc ana@x.io"}' ``` ```json {"text": "Sales: [EMAIL_1]. Billing: [EMAIL_2], cc [EMAIL_1]", "found": {"EMAIL": 3}, "enabled": true, "style": "numbered", "plugin": false, "plugin_skipped": false} ``` - `found` counts placeholders by label, every occurrence; `style` is the style used; `plugin` says whether the detector plugin ran; `plugin_skipped` is true when one is configured but did not run, because the caller sent no valid key. - `enabled` says whether this gateway redacts examples (`RATCHET_REDACT_PII`); the preview runs either way. A task whose spec sets `privacy.redact_style` stores in that style; preview it with `?style=`. - **400** when `text` is missing or longer than 5,000 characters, or `style` is not one of the three (`style must be one of label, numbered, surrogate`). **500** when the detector plugin fails (only callers with a key reach it), with a message that names the problem or the exception's class and never the text (`the redaction plugin raised RuntimeError`). --- ## Cost Utilization-aware cost accounting (formulas in [ARCHITECTURE.md](/reference/architecture/#7-cost-accounting)). The self-hosted student endpoint and the training jobs bill GPU-hours while they run; the ledger records when they ran (`compute_intervals`) and the report prices them. The per-token list prices were checked on 29 Oct 2026, the GPU-hour rate is still unverified; the report says so (`prices.label`) until `RATCHET_PRICES_VERIFIED=1`. Numbers from the mock-mode gateway are mock-model output. ### GET `/ratchet/cost?days=N` The example's GPU rate of $1.55 per hour is the setting's default, not Nebius's listed $1.35 (see [CONFIGURATION.md](/reference/configuration/#cost-accounting)); every price here is unverified. The cost report for every ledger day, or for the last `days` days (1 to 36,500; anything else is **400**, a non-number **422**). Access level `dashboard`, like `/ratchet/stats`. `python -m ratchet.cost report [--days N] [--json]` prints the same report. An example from the mock-mode gateway (numbers from mock models, nine days of 1,500 requests with the endpoint up 24 hours a day): ```json { "window": {"since_day": null, "until_day": 9, "now": 1792400000.0}, "model": "per_token", "headline": {"model": "per_token", "total_usd": 5.21, "savings_usd": 9.37, "savings_pct": 0.64}, "prices": {"verified": false, "label": "unverified list prices", "student_gpu_usd_per_hour": 1.55, "student_gpus": 1, "student_endpoint_usd_per_hour": 1.55, "train_gpu_usd_per_hour": 1.55, "train_gpus": 1, "teacher_model": "nvidia/Nemotron-3-Ultra-550b-a55b", "teacher_usd_per_m_tokens": {"input": 1.0, "output": 3.0}, "student_usd_per_m_tokens": {"input": 0.06, "output": 0.24}, "fallback_student_model": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B", "fallback_student_usd_per_m_tokens": {"input": 0.06, "output": 0.24}}, "requests": 13500, "student_served": 11300, "teacher_served": 2200, "teacher_usd": 4.69, "student_per_token_usd": 0.52, "hosted_student_usd": 0.0, "student_served_off_endpoint": 0, "baseline_usd": 14.58, "split_unknown_requests": 0, "split_unknown_usd": 0.0, "endpoint_hours": 216.0, "endpoint_usd": 334.8, "endpoint_usd_per_hour": 1.55, "idle_hours": 24.0, "idle_share": 0.11, "idle_granularity": "day", "training_hours": 1.9, "training_usd": 2.95, "views": { "per_token": {"total_usd": 5.21, "savings_usd": 9.37, "savings_pct": 0.64}, "gpu_hours": {"total_usd": 342.44, "savings_usd": -327.86, "savings_pct": -22.49} }, "avoided_teacher_usd_per_student_request": 0.00098, "break_even_student_requests_per_gpu_hour": 1582.0, "hosted_fallback_usd_per_request": 0.000029, "self_host_beats_hosted_above_requests_per_hour": 54070.0, "observed_student_requests_per_endpoint_hour": 52.3, "utilization_vs_break_even": 0.033, "recommendation": "hosted_fallback", "explanation": "Self-hosting does not pay at this volume. The endpoint served 52 student requests per hour it ran, below the break-even of 1,582; below 54,070 an hour the hosted fallback is cheaper per answer (1 GPU at $1.55 per GPU-hour; the GPU-hour rate is unverified). Use the hosted fallback student on Token Factory (leave RATCHET_STUDENT_BASE empty) or run the endpoint only while traffic arrives. The crossover assumes the hosted fallback serves as many requests as the tuned student; it is untuned, so more of its requests may go to the teacher. Savings by GPU-hours paid: -$327.86 (-2249%); per token: $9.37 (64%).", "open_intervals": [], "warnings": [], "daily": [ {"day": 9, "requests": 1500, "student_served": 1470, "teacher_usd": 0.12, "student_per_token_usd": 0.06, "endpoint_hours": 24.0, "endpoint_usd": 37.2, "training_usd": 0.33, "baseline_usd": 1.64, "savings_pct_per_token": 0.89, "savings_pct_gpu_hours": -22.0} ], "daily_truncated": false } ``` - `views.per_token` is what every other savings figure uses; `views.gpu_hours` prices the student by the GPU-hours paid: teacher spend + endpoint hours + training hours (each interval at the rate it was recorded at) + the student's per-token spend in hours (live ledger) or days (mock models) with no recorded endpoint uptime (`hosted_student_usd`, `student_served_off_endpoint` answers: the hosted fallback served, or nobody recorded the endpoint). Savings can be negative and are never clamped; `savings_pct` is `null` without a baseline. `headline` is the view `RATCHET_STUDENT_COST_MODEL` picks. - `avoided_teacher_usd_per_student_request` is the teacher spend a student answer saves on average (its all-teacher baseline less the teacher calls it still made); `break_even_student_requests_per_gpu_hour` is `endpoint_usd_per_hour` divided by it, so it counts student answers per hour of the whole endpoint. `endpoint_usd_per_hour` is the endpoint's GPU cost in the window over its hours (each interval at the rate it was recorded at, per-GPU price × GPUs), the same rate the GPU-hour view bills; the configured rate while no uptime is recorded. Break-even prices the endpoint's hours alone: training jobs can still leave the window at a loss above it, and the explanation then says so. `null` until the student has served, and when its answers save nothing. - `hosted_fallback_usd_per_request` is what the hosted fallback student (`RATCHET_FALLBACK_STUDENT`, per token) would bill for all the student calls, per student answer; `self_host_beats_hosted_above_requests_per_hour` (`endpoint_usd_per_hour` / that price) is the volume above which the endpoint is cheaper per answer. It compares price per answer only: the hosted model is untuned, so it may serve fewer requests than the tuned student and send more to the teacher. Every `hosted_fallback` explanation says so; compare the student's share of requests over a day on each before switching. - `observed_student_requests_per_endpoint_hour` counts student answers in buckets when the endpoint had uptime over its hours; `idle_hours` are endpoint hours in buckets when no request reached the student. `idle_granularity` names the bucket: `hour` on a live ledger (requests and uptime share the wall clock; an hour with any uptime is credited with all its answers), `day` on the simulator's synthetic clock (hours idle inside a busy mock day are not counted). - `recommendation` is `self_host`, `hosted_fallback`, `scale_to_zero` or `insufficient_data` (rules in [ARCHITECTURE.md](/reference/architecture/#the-report-costpy)), with a plain-English `explanation`. It says separately whether the observed volume covers the endpoint's hours and whether the window lost money by GPU-hours: between break-even and the crossover, `hosted_fallback` comes with "Self-hosting costs less than the teacher, but the hosted fallback would cost less per answer", and it never says "pays" when `views.gpu_hours.savings_usd` is negative. - `split_unknown_requests`: rows from before v0.5 have no student/teacher split; their whole cost counts as teacher spend in the GPU-hour view (`split_unknown_usd`), and a warning says so. `warnings` also names a training job open for more than 12 hours, which counts up to now until it is closed. - `daily`: one row per day with requests or compute, the latest 731 days at most (`daily_truncated`); the totals always cover the whole window. - The dashboard's **Economics** panel draws this report. The keys it reads are listed with their JSON types in `tests/fixtures/cost_keys.json` and checked by `test_cost_report_contract`; like `/ratchet/stats`, the report adds keys and never removes or retypes one. ### POST `/ratchet/cost/endpoint` Records the self-hosted student endpoint going up or down, as `python -m ratchet.cost endpoint up|down` does. Needs an admin key (open in mock mode without admin keys, recorded as `mock-admin`; **403** `admin API disabled: set RATCHET_ADMIN_KEYS` in live mode without them). ```bash curl -s localhost:8000/ratchet/cost/endpoint -H "Authorization: Bearer $ADMIN_KEY" -H 'content-type: application/json' \ -d '{"state": "up", "usd_per_hour": 1.55, "gpus": 1, "note": "recording a speed test"}' ``` ```json {"state": "up", "changed": true, "interval": {"id": 3, "kind": "student_endpoint", "start_ts": 1792400000.0, "end_ts": null, "usd_per_hour": 1.55, "gpus": 1, "version": null, "source": "api", "note": "recording a speed test"}, "summary": "student endpoint up at 2026-10-20 10:13:20 UTC: 1 GPU at $1.55 per GPU-hour (interval 3)"} ``` - `state` is required (`up` or `down`); `usd_per_hour` (per GPU, 0 to 1,000; default `RATCHET_STUDENT_GPU_USD_PER_HOUR`), `gpus` (0 to 1,024; default `RATCHET_STUDENT_GPUS`) and `note` (at most 200 characters, PII redacted) are optional. Any other key, or a bad value, is **400**. - `up` while the endpoint is up returns its open interval with `changed: false`; `down` while it is down returns `interval: null`, `changed: false`. `down` closes every open endpoint interval. A change writes an `endpoint` event naming the source and the admin's name, never the key. - In mock mode the time is on the live gateway's synthetic clock (the current mock day at the wall clock's time of day), and the shared mock-mode gateway holds at most 200 such records (**409**; a reset clears them). ## Ledger replay The ledger time machine: what one version's logged traffic would have looked like at another threshold or audit rate. Ratchet designs its own labelling, so every label's probability is logged (`requests.label_p`: 1.0 for a request the student answered and escalated, the audit rate for an answer it served), and precision is estimated at any threshold with weights 1 / `label_p` (Hajek), including thresholds below the live one. Both endpoints are read-only, access level `dashboard` like `/ratchet/stats` ([Access levels](#access-levels)) (aggregates only, never request text), in both modes. Numbers from the mock-mode gateway are mock-model output and say so (`label`). - `version`: one version id in ASCII digits (default: the live version; **400** `no live version: name one with ?version=` without one). A version that does not exist is **404**. `version` twice, a list (`1,2`), a word (`all`) or other digits (`²`) is **400**: confidence scales differ between versions, so a replay never pools them. - One replay runs at a time: it reads the version's whole history into memory, so a request while another replay runs is **429** with `Retry-After: 1`. - `since_day`, `until_day`: ledger days, both included (0 to 10,000,000; `since_day` after `until_day` is **400**). - `audit_rate`: 0 to 1 (**400** otherwise); prices the audits of every request served at the threshold. Default: each served answer's logged rate (a request that escalated in fact at the version's mean logged rate, `logged_audit_rate`). It changes cost only: precision comes from the labels that were logged. - A query value that is not a number is **422**, as elsewhere. ### GET `/ratchet/replay?version=&threshold=&audit_rate=&since_day=&until_day=` `threshold` is more than 0 and at most 1.01 (serve nothing; **400** otherwise), default: the version's threshold now (`current_threshold`: the one the router holds its answers to, which for the hosted fallback student, in live mode without a student endpoint, is `RATCHET_CONFIDENCE_THRESHOLD`). An example from the mock-mode gateway after four mock days (mock models): ```json { "version": 2, "state": "live", "label": "simulation (mock models)", "cost_model": "per_token", "n": 1697, "excluded": 0, "excluded_by": {"before_logging": 0, "no_student_answer": 0, "pinned_now": 0, "malformed": 0}, "current_threshold": 0.5, "logged_audit_rate": 0.1, "actual": {"threshold": null, "audit_rate": null, "n": 1697, "served": 1653, "coverage": 0.974, "labelled": 172, "precision": 0.983, "precision_reason": null, "n_eff": 172.0, "interval": [0.950, 0.994], "interval_method": "approximate (Wilson at Kish n_eff)", "served_errors_per_1k": 17.0, "cost_usd": 0.282, "baseline_usd": 1.886, "saved_usd": 1.604, "savings": 0.850}, "counterfactual": {"threshold": 0.4, "audit_rate": 0.05, "n": 1697, "served": 1680, "coverage": 0.990, "labelled": 199, "precision": 0.967, "precision_reason": null, "n_eff": 177.2, "interval": [0.930, 0.985], "interval_method": "approximate (Wilson at Kish n_eff)", "served_errors_per_1k": 32.3, "cost_usd": 0.162, "baseline_usd": 1.886, "saved_usd": 1.724, "savings": 0.914}, "calibrated_baseline": {"R": 1.0003, "se": 0.00026, "interval": [0.9998, 1.0008], "n": 172, "n_eff": 172.0, "min_n": 30, "status": "ok", "method": "ratio estimator (Cochran), audited served answers weighted by 1/label_p"} } ``` - `n`: requests the version's student answered with the replay's columns logged. `excluded_by` counts the version's other rows: written before the columns (`before_logging`), without a student answer (`no_student_answer`: the call failed or its breaker was open), of a task pinned now (`pinned_now`), or with out-of-range values (`malformed`, a ledger edited outside Ratchet). A request whose escalation failed outright was never logged, so it is not here. - `actual`: the cascade's threshold decision as logged (`threshold: null`): each request at the threshold it was logged with, and each audit priced at its logged rate times the request's all-teacher cost (an expected cost, not the audits drawn). Its `served` is exactly the requests the student's confidence served (route `student`); a degraded answer (the student standing in for an unavailable teacher, route `degraded`) is counted as not served and priced as if the teacher had answered it. Arbitration spend is left out, so `cost_usd` can be below the logged cost. - `counterfactual` at `threshold` and `audit_rate`: `coverage` is exact (no label needed); `precision` is Hajek's weighted mean of `student_agree` over labelled served requests, null with `precision_reason` when nothing is served, when a served request could never be labelled (audit rate 0: "audit rate was 0: precision above the live threshold cannot be identified") or when none has a label; `interval` is approximate (Wilson at Kish's `n_eff`), not a certificate. `cost_usd` = student spend + the all-teacher cost of every request not served + the audit rate times that of every request served; `baseline_usd` = the all-teacher cost of every request; `savings` = 1 − cost / baseline (per token). A request with no valid answer, one that could not fill its `n` choices (route `n_choices`), or one logged at 1.01 while its task's spec is still changed for the version (the router answers that task with the teacher at any threshold until a night re-scores the version) is never served. Not logged, and so counted as servable: a request for `n` > 1 choices that escalated below the threshold (route `confidence`) whose samples also had fewer than `n` in agreement; at a lower threshold it would escalate on route `n_choices`, so coverage below the logged threshold can be slightly high for such traffic. `student_agree` comes from an audit, or from an escalation the teacher answered under an audit's conditions (one choice, temperature 0, none of the client's `top_p`, `seed`, penalties or `logit_bias`); an escalation under the client's sampling parameters, or for `n` > 1 choices, has no label. - `calibrated_baseline`: the ratio `R` of what the teacher really cost to the token formula's estimate (which assumes `RATCHET_TEACHER_REASONING_TOKENS`), on audited answers, with its standard error and an approximate 95% interval; `insufficient` (R = 1) below 30 audited answers. The replay prices unaudited answers at R times the estimate. The mock teacher writes exactly the assumed tokens, so R is about 1 in the live gateway. ### GET `/ratchet/replay/frontier?version=&target=&audit_rate=&since_day=&until_day=` `target` is strictly between 0 and 1 (default 0.97; **400** otherwise). ```json { "version": 2, "state": "live", "label": "simulation (mock models)", "cost_model": "per_token", "n": 1697, "excluded": 0, "excluded_by": {"before_logging": 0, "no_student_answer": 0, "pinned_now": 0, "malformed": 0}, "current_threshold": 0.5, "target": 0.97, "rows": [{"threshold": 0.3, "...": "the counterfactual's keys"}, "... 71 rows, 0.30 to 1.00 every 0.01"], "suggested": {"threshold": 0.95, "target": 0.97, "lower": 0.974, "precision": 1.0, "coverage": 0.770, "savings": 0.667, "advisory": "approximate interval, not a certificate; thresholds change only through Night School", "method": "fixed-sequence scan of [0.95, 0.9, 0.85, 0.8, 0.75, 0.7, 0.65, 0.6, 0.55, 0.5], approximate (Wilson at Kish n_eff)"}, "money_left_on_table": null, "calibrated_baseline": {"R": 1.0003, "...": "as above"} } ``` - `rows`: one counterfactual per threshold; `coverage` never rises with the threshold. - `suggested`: the scan goes down `stats.CERT_GRID` from 0.95 and stops at the first threshold whose interval's lower bound is below `target`; it returns the last one that passed, or null when 0.95 already fails. It is advice: the interval is approximate and computed on the traffic being looked at, so it is not a certificate, and nothing applies it. Thresholds change only through Night School's held-out calibration. - `money_left_on_table`: `{threshold, current_threshold, savings, usd}`, what serving at the suggested threshold would have saved over the current one, when the suggestion is lower; null otherwise (above, the suggestion says the current threshold is not supported at the target by this approximate interval). `python -m ratchet.replay --db PATH [--version V] [--sweep 0.3:0.95:0.05] [--audit-rate A] [--target 0.97] [--since-day D] [--until-day D] [--json]` prints the frontier report as one table per version (labelled as mock-mode run when the simulator wrote the ledger); exit 2 on bad input. `python -m sim.simulate --db PATH` keeps a mock run's ledger in a new file for it. ## Savings receipts Verified savings receipts: for each ledger day, one receipt for the whole gateway and one per API client, sealed once the day is over and never changed (invariant 17). A receipt states what the day cost and saved, with the all-teacher baseline of the answers the student served **calibrated on audited teacher bills** and a one-sided 95% lower bound, and bills only on that bound. It is an RFC 9162 Merkle tree over each request's metadata (never request text; the client by its key's name), signed with the BOM key (`/ratchet/bom/public-key`) and chained to the same client's previous receipt. Every amount is whole nano-USD (1e-9 USD); numbers that are not whole are decimal strings, so the signed JSON has no floats. Who may read what is under [Authentication](#authentication). In the mock-mode gateway the numbers are mock-model output and say so (`label`). A day is sealed when it is over and someone asks: the mock-mode gateway after each mock night, any of these endpoints (lazily, every unsealed day up to the one asked for, oldest first), `python -m ratchet.live seal` from cron ([RUNBOOK](/reference/runbook/#savings-receipts)) or `python -m ratchet.receipt seal --db`. On a live ledger a day is over 5 minutes after it ended. A day that has not ended is **409**; a day on which the client sent no request (nobody did, for the gateway's receipt) is **404**; a `period` outside 0 to 10,000,000 is **400**. ### GET `/ratchet/receipts` ```json {"receipts": [{"period": 1, "global": true, "clients": ["acme"]}, {"period": 2, "global": true, "clients": ["acme", "globex"]}], "sealed_through": 2, "public_key": "YLHq1TAo…", "key_id": "9d2479f21890a44f", "label": "live traffic"} ``` Every sealed receipt the caller may read, by day, after sealing every day that has ended: an admin (or the open mock-mode gateway) sees the gateway's (`global`) and every client's; an API key sees only its own (`global: false`). `sealed_through` is the last day that has ended; `public_key` is null when receipts are unsigned. ### GET `/ratchet/receipts/{period}?client=` The signed document of day `period`: the gateway's, or `client`'s for an admin; an API key always gets its own (`?client=` naming another client is **403**). An example from the mock-mode gateway with `RATCHET_MOCK_TEACHER_REASONING_MEAN=420`, after four mock days (mock-mode run, mock models; shortened): ```json { "manifest": { "schema": "ratchet.receipt/v1", "generator": "ratchet 0.5.0", "created_at": "2026-10-21T11:39:44Z", "label": "simulation (mock models)", "period": {"day": 3}, "client": null, "first_id": 3001, "last_id": 4500, "n": 1500, "merkle": {"alg": "rfc9162-sha256", "root": "efd10ccd…", "size": 1500, "leaf_keys": ["id", "ts_ms", "…"]}, "totals": {"requests": 1500, "teacher_served": 29, "student_served": 1471, "unverified": 0, "audits": 147, "cost_nusd": 297406120, "student_cost_nusd": 40212120, "teacher_cost_nusd": 257194000, "baseline_recorded_nusd": 1703467000, "baseline_measured_nusd": 33954000, "baseline_calibrated_nusd": 2267342206, "missing_amounts": 0}, "calibration": {"method": "ratio estimator (Cochran, …) per task over audited student answers, weighted 1/label_p; …", "min_audits": 30, "z": "1.645", "tasks": {"ticket-triage": {"status": "ok", "n_served": 1471, "n_audits": 147, "n_eff": "147.0", "cost_nusd": 262666960, "recorded_nusd": 1669513000, "x_total_nusd": 1606891000, "ratio": "1.389882", "ratio_se": "0.029841", "point_nusd": 2233388206, "se_nusd": 45502963}}}, "savings": {"recorded_nusd": 1406060880, "calibrated_nusd": 1969936086, "stand_in_loss_nusd": 0, "billable_nusd": 1969936086, "margin_nusd": 74852375, "billable_lo_nusd": 1895083711, "spend_nusd": 0, "net_billable_lo_nusd": 1895083711}, "billing": {"share": "0.25", "fee_nusd": 473770927, "basis": "one-sided 95% lower bound", "billable": {"requests": 1500, "baseline_nusd": 2267342206, "cost_nusd": 297406120, "tasks": ["ticket-triage"]}}, "cost_views": {"billed": "per_token", "gpu_hours": {"total_nusd": 37743255111, "endpoint_seconds": 86400, "savings_calibrated_nusd": -35475912905, "basis": "informational: billing uses the per-token view", "…": "…"}, "gpu_hours_note": null}, "prev_sha256": "6c073996…", "units": "amounts in nano-USD (1e-9 USD); numbers that are not whole are decimal strings", "scope": "edits after sealing are detectable, …; costs written at request time are not verified: reconcile …" }, "sha256": "64008e19…", "signature": {"alg": "Ed25519", "key_id": "9d2479f21890a44f", "public_key": "YLHq1TAo…", "value": "uNVs30b5…"} } ``` - Requests are **measured** (the primary teacher answered: its bill is the baseline), **served** (the student served on the audit design) or **unverified** (a stand-in answer: route `degraded` or `fallback_teacher`, outside the audit design; counted at its recorded baseline, never billed, though what it lost counts: below). `totals.teacher_served` counts the measured ones. - `calibration.tasks`: per task with served answers, the ratio `R` of the audited answers' teacher bills to the token formula's estimates (weights 1 / `label_p`), the point estimate of the served answers' all-teacher cost (`point_nusd` = R × `x_total_nusd`) and its standard error (`se_nusd`, Cochran's variance with the finite population correction; normal approximation). `insufficient_audits` below `min_audits` (`RATCHET_RECEIPT_MIN_AUDITS`) and `unlogged` (a served answer written before the formula estimate was logged) have none and are not billed. - `totals.baseline_calibrated_nusd` = measured bills + each calibrated task's point + the recorded baseline of every other request. `savings.recorded_nusd` is the formula's savings, as `/ratchet/stats` reports them; `calibrated_nusd` uses the calibrated baseline. - `billing.billable`: the measured requests and the served answers of the calibrated tasks. `savings.stand_in_loss_nusd` (0 or less) = Σ min(0, baseline − cost) over the unverified requests: a stand-in's gain is never billed, but its loss (a fallback teacher's answer after the student's samples were paid for) counts against the bill, like a measured request's. `savings.billable_nusd` = the billable baseline − the billable cost + `stand_in_loss_nusd`, and `billable_lo_nusd` = that − `margin_nusd` (⌈1.645 √Σ se²⌉); `net_billable_lo_nusd` subtracts the day's spend outside requests (`spend_nusd`; a client's share by request count). `billing.fee_nusd` = ⌊`share` × max(0, net)⌋. With no calibrated task, `basis` is `formula` and the fee 0. - `cost_views.gpu_hours` (the gateway's receipt only; `gpu_hours_note` says why a client's has none): the day's cost by GPU-hours paid, informational. - `prev_sha256`: the `sha256` of the same client's previous receipt (64 zeros for its first). ### GET `/ratchet/receipts/{period}/proof/{request_id}?client=` ```json {"period": 3, "client": null, "request_id": 3006, "leaf": {"id": 3006, "ts_ms": 1792582781950, "day": 3, "task": "ticket-triage", "client": null, "served_by": "student", "route": "student", "student_version": 1, "escalated": 0, "audited": 0, "label_p_ppb": 100000000, "cost_nusd": 27240, "student_cost_nusd": 27240, "teacher_cost_nusd": 0, "baseline_nusd": 1093000, "baseline_est_nusd": 1093000}, "index": 5, "size": 1500, "path": ["273c76f7…", "… 11 hashes"], "root": "efd10ccd…", "alg": "rfc9162-sha256", "receipt_sha256": "64008e19…"} ``` The leaf (exactly the receipt's `leaf_keys`) and its RFC 9162 audit path, deepest sibling first, against the root and size the receipt signs. A request of another day or client is **404**. The gateway rebuilds the tree from the ledger and serves a proof only when it rebuilds the signed root; rows changed after sealing get **500** and one `integrity` event per day and client ([RUNBOOK](/reference/runbook/#savings-receipts)). ### GET `/ratchet/receipts/{period}/leaves?client=` The receipt's leaves as JSON lines (`application/x-ndjson`), in tree order: anyone holding them can rebuild the root and recompute every total, estimate and the fee. Same checks and errors as the proof. ### Checking a receipt offline ```bash curl -s localhost:8000/ratchet/bom/public-key # publish this once curl -s localhost:8000/ratchet/receipts/3 > r3.json; curl -s localhost:8000/ratchet/receipts/2 > r2.json curl -s localhost:8000/ratchet/receipts/3/proof/3006 > p.json; curl -s localhost:8000/ratchet/receipts/3/leaves > leaves.jsonl python -m ratchet.receipt verify r3.json --public-key --prev r2.json --proof p.json --rows leaves.jsonl # OK: signature valid (Ed25519, key 9d2479f21890a44f, trusted key) # OK: chain: prev_sha256 is the sha256 of day 2's receipt # OK: proof: request 3006 is leaf 5 of 1500 in the signed tree # OK: rows: 1500 leaves rebuild the signed root, and every total, estimate and the fee follow ``` Exit 0 when every check passes, 1 when one fails (an edited digit fails the hash), 2 on bad input. `--rows` recomputes amounts that come from the estimator within 2 nano-USD and the decimal strings within 1e-5 (relative), so another implementation's floating point may differ slightly. `python -m ratchet.receipt seal --db PATH [--period N [--client NAME]]` seals a ledger file offline (a simulator's ledger from `python -m sim.simulate --db`: every day it logged) and prints the receipt asked for. A receipt cannot prove the costs written at request time: reconcile `totals.teacher_cost_nusd` over the days with the Token Factory bill for the same API key. --- ## Tasks The task registry: the built-in tasks, the tasks of `RATCHET_TASKS_FILE`, the tasks written through these endpoints (API tasks, stored in the ledger and registered again at every start), and auto tasks, derived from a request's `response_format` json_schema and stored the same way ([Structured outputs](#structured-outputs)). The spec format is in [CONFIGURATION.md](/reference/configuration/#adding-a-task). Reads have the access level `dashboard` (open in mock mode; in live mode an admin or dashboard key unless `RATCHET_PUBLIC_DASHBOARD=1`, and then an anonymous reader sees `clients` allow-lists as the pseudonyms `/ratchet/stats` uses); writes and `validate` need an admin key ([Access levels](#access-levels)). Errors are `{"detail": ...}`; a bad spec is **422** `{"detail": "invalid task spec", "errors": [...]}`, every problem with its path. ### GET `/ratchet/tasks` ```json { "unknown_tasks": "distill", "max_tasks": 500, "tasks": [ {"name": "invoice-extract", "description": "Vendor and total from an invoice", "format": "json", "fields": {"vendor": {"type": "text", "strip_suffixes": ["inc", "ltd"]}, "total": {"type": "number", "abs_tol": 0.01}}, "sql": null, "pinned": false, "store": true, "privacy": {}, "match": {}, "routing": {}, "policy": {}, "tools": "teacher", "clients": ["acme"], "source": "api", "spec_hash": "1c7e77bb7041b4d8", "verifiable": true, "requests": 420, "student_share": 0.81, "recalibrating": false, "runtime_pinned": false} ], "unregistered": [{"name": "invoice-extrct", "requests": 3}] } ``` - `tasks`: every registered task, the built-in ones first, then the file's, then the API and auto tasks. The first twelve keys are the task's spec as a file would hold it (what `PUT` takes back); `sql` is null except for a text-to-SQL task, where it holds every option (`{"fixture": "shop.sql", "dialect": "sqlite", "ordered": "auto", "empty_result": "text", "max_rows": 1000, "max_steps": 200000}`, or `{"dialect": "postgres"}`; [CONFIGURATION.md](/reference/configuration/#text-to-sql-tasks)); `source` is `builtin`, `file`, `api` or `auto` (derived from a schema; never pinned); `verifiable` says it names fields that can be checked, or runs its SQL answers on a fixture; `requests` and `student_share` come from the ledger (the same numbers as `/ratchet/stats` `tasks`); `recalibrating` is true when the live version was scored under another spec of the task, so the teacher answers it until a night re-scores the version; `runtime_pinned` (v0.5) is true while an operator has pinned the task to the teacher at runtime ([Admin](#admin)), on top of its spec's `pinned`. - `unknown_tasks`: the `RATCHET_UNKNOWN_TASKS` policy. `max_tasks`: how many tasks the registry holds at most, built-in ones included. - `unregistered`: names requests used that are not registered (a typo, or a task to register or pin), with their request counts. - The dashboard's **Task registry** panel reads this list, `GET /ratchet/tasks/{name}` for its drawer, and the write results below. The keys it reads are listed with their JSON types in `tests/fixtures/tasks_keys.json` and checked by `test_task_registry_contract`; like `/ratchet/stats`, these responses add keys and never remove or retype one. The panel writes API tasks with an admin key the operator types in, sent only as `Authorization: Bearer` and kept in the page's memory. ### GET `/ratchet/tasks/{name}` One entry as above; for an API or auto task also `created_ts`, `updated_ts` and `updated_by` (the admin key's name; `null` for an auto task, which no admin wrote). **404** for a name that is not registered. ### POST `/ratchet/tasks/validate` A dry run, with no change to the registry, the ledger or the event log. The body is either one task spec (checked as `PUT` would take it, so a built-in or file task's name is an error) or a tasks file document `{"tasks": [...]}` (checked as startup reads `RATCHET_TASKS_FILE`: it may adjust or extend a built-in task and extend its own tasks, but not an API task or a task of the file running now, because startup registers the file before the stored API tasks; and its tasks, with the built-in ones it does not replace, must fit in `max_tasks`). A document that validates starts the gateway. Always **200**: ```json {"valid": false, "errors": ["tasks[1].fields.total.abs_tol: must be a number >= 0"], "tasks": []} {"valid": true, "errors": [], "tasks": [{"name": "invoice-extract", "spec_hash": "1c7e77bb7041b4d8", "registered": null}]} ``` `registered` is the source of the task that has the name now (`builtin`, `file`, `api`, `auto`), or null. ### PUT `/ratchet/tasks/{name}` Create or replace an API task (an auto task of that name becomes an API task). The body is one spec without `name` (or with the same name as the URL); `extends` may name any registered task, the task's own name included (a new version built on the current one). ```bash curl -X PUT localhost:8000/ratchet/tasks/invoice-extract -H "Authorization: Bearer $ADMIN_KEY" \ -d '{"description": "Vendor and total from an invoice", "fields": {"total": {"type": "number", "abs_tol": 0.01}}}' ``` | Status | When | |---|---| | 200 | `{"task": {...entry...}, "spec_hash": "...", "changed": true, "created": true}`. A spec identical to the stored one is a 200 with `changed: false` and changes nothing; for a sql task the fixture is read again, so the same body after its text changed is a change (a new `spec_hash`) | | 401, 403 | No admin key, a wrong one or an API key; the admin API is off (live mode without `RATCHET_ADMIN_KEYS`) | | 403 | The name is a built-in task (`task 'x' is built in; use another name (a spec may extend it)`) or a file task (`task 'x' is defined in RATCHET_TASKS_FILE; edit the file`) | | 409 | The registry holds `max_tasks` tasks, or (mock mode) 50 API tasks already | | 422 | The body is not strict JSON (NaN is refused), names another task, or breaks a spec rule; `clients` names a key that is not in `RATCHET_API_KEYS`; a sql task's fixture is outside `RATCHET_FIXTURES_DIR`, missing, over 5 MB, not UTF-8 or does not load (`sql.fixture: fixture '../x.sql' is outside RATCHET_FIXTURES_DIR`), or (Python 3.10) needs Python 3.11 | The event log records `task_registered` (`name (spec ) by `) or `task_changed` (`name: spec -> by `; for a change to `description`, `clients`, `match` or `privacy` only, the hash is kept). A changed spec does not touch stored examples: Night School judges them under the current spec, and the serving version keeps the task on the teacher until it is scored under the new one. ### DELETE `/ratchet/tasks/{name}` Remove an API task or an auto task (auto tasks include those made by `RATCHET_AUTO_TASK=system_prompt`). Removing an auto task frees its place under `RATCHET_MAX_AUTO_TASKS` (a request with its system prompt makes it again while there is room): `{"deleted": "invoice-extract", "spec_hash": "...", "examples_kept": 420}` and a `task_deleted` event. Its stored examples stay, for provenance (and stay trainable under the name's adhoc task unless it is registered again or pinned). Requests that name it then follow `RATCHET_UNKNOWN_TASKS`; the next request with an auto task's schema registers it again from that schema. **403** for a built-in or file task, **404** for a name that is not registered. ## Admin Manual interventions with an audit trail. Every route under `/ratchet/admin/`, its reads included, needs an admin key (`RATCHET_ADMIN_KEYS`; [Access levels](#access-levels)): **401** without a key or with a wrong one, **403** for an API key or a dashboard key, **403** `admin API disabled: set RATCHET_ADMIN_KEYS` in live mode without admin keys; open in mock mode without them (as `mock-admin`). `python -m ratchet.admin` does the same on the ledger file (see [CLI](#cli)). Conventions: - **Body.** A JSON object with only the keys each route names (**400** `unknown keys [...]` otherwise); an empty body is `{}`. Every call that can change something takes `reason`: one line of 3 to 500 characters saying why (**400** without it). It is redacted like a training example (an e-mail address becomes `[EMAIL]`) before it is stored, because it lands in the release log. `POST /ratchet/admin/tick` is the one route where it is optional. - **Answer.** `{"action", "target", "changed", "before", "after", "action_id"}`: what was acted on (`v7`, a switch, a task, `gate`), its state before and after, and the `admin_actions` row written (`null` when the call changed nothing: a switch already set, a canary already at that share; nothing is then written). - **Record.** The change, its `admin_actions` row (time, actor `api:` or `cli:`, action, target, reason, before, after, and the call's `x-request-id`) and an `admin` event (`v7 rolled back from canary 50% by api:ops: refund labels are wrong`) commit in one transaction. Keys are never stored, only names (invariant 13). - **Conflicts.** A move the table below does not allow, or a version that another admin or the release gate moved while the call ran (every move compares and sets), is **409** with what the version is now; nothing is written. **404** for an unknown version or task. - **Effect.** Every process that shares the ledger sees a change on its next request, tick or night without a restart: the switches and pins are read through a snapshot that any write invalidates, also another process's (`PRAGMA data_version`). | From | To | How | Needs `force` | |---|---|---|---| | `canary` | `rolled_back` | `rollback`: its traffic returns to the live version, or the teacher | No | | `canary` | `retired` | `retire` | No | | `canary` | `canary` (another share) | `promote` with `to: canary` | No | | `canary` | `live` | `promote` with `to: live`: the live version is retired in the same transaction | No | | `live` | `retired` | `retire`: the teacher answers what the canary (if any) does not | No | | `shadow` | `retired` | `retire`: cancels the shadow, so tonight's training can run | No | | `shadow` | `canary` | `promote` with `to: canary`: ends the shadow early | No | | `rejected`, `rolled_back`, `retired`, `candidate` | `canary` | `promote` with `to: canary`: the golden gate or the canary did not pass it (or never saw it) | Yes | Nothing becomes live except a canary, a version without a golden eval (no calibrated threshold) is never promoted, force or not, and a move into canary is refused while another version is in shadow or canary (one in flight at a time). With the runtime audit rate at 0, a new canary needs `force` (it would get no audits, so it could neither advance nor be rolled back). ### GET `/ratchet/admin/versions` ```json { "versions": [ {"id": 7, "day": 12, "state": "canary", "traffic_pct": 50, "adapter": "student-v7", "train_examples": 4210, "golden": {"precision": 0.97, "coverage": 0.91, "threshold": 0.74, "golden_n": 412}, "stage_audits": {"n": 31, "precision": 0.968}}, {"id": 6, "day": 9, "state": "live", "traffic_pct": 100, "adapter": "student-v6", "train_examples": 3890, "golden": {"precision": 0.96, "coverage": 0.9, "threshold": 0.76, "golden_n": 401}, "stage_audits": {"n": 640, "precision": 0.95}} ], "controls": {"student_disabled": false, "releases_frozen": false, "training_paused": false, "audit_rate": null, "pinned_tasks": []} } ``` Newest first. `stage_audits` (live and canary) counts the teacher's audits of what the version served in its current stage; a shadow has `shadow` instead (`Ledger.shadow_quality`: would-serve answers, precision, coverage, the live student's coverage, unarbitrated disagreements). ### GET `/ratchet/admin/flags` `controls` as above, plus `set_by` (each stored switch: its raw value, when and by whom it was last set) and `task_pins` (each runtime pin: task, when, by whom and why). ### GET `/ratchet/admin/audit-log?limit=50` `{"actions": [...]}`: the newest `limit` (1 to 1000, **400** otherwise) `admin_actions` rows, newest first, with `before` and `after` parsed. ### POST `/ratchet/admin/versions/{id}/rollback` and `.../retire` Body `{"reason": "..."}`. Roll a canary back, or retire a live, canary or shadow version. ```bash curl -X POST $GATEWAY/ratchet/admin/versions/7/rollback -H "Authorization: Bearer $ADMIN_KEY" \ -d '{"reason": "refund labels are wrong since 09:00"}' ``` ```json {"action": "rollback", "target": "v7", "changed": true, "action_id": 12, "before": {"v7": {"state": "canary", "traffic_pct": 50}}, "after": {"v7": {"state": "rolled_back", "traffic_pct": 0}}} ``` ### POST `/ratchet/admin/versions/{id}/promote` Body `{"reason", "to", "traffic_pct", "force", "load_adapter"}`: - `to`: `canary` or `live` (**400** otherwise). - `traffic_pct`: a canary's share, 1 to 99; by default the next stage (10% from outside canary, then 50%). Leave it out for `live` (100%). - `force`: allows the moves the table marks, and a canary while the audit rate is 0. The event says `(forced)`. - `load_adapter`: in live mode, first load the version's LoRA adapter into the student endpoint (`/v1/load_lora_adapter` at `RATCHET_STUDENT_BASE`, path `RATCHET_ADAPTER_ROOT/`), after the checks and before the move, outside the ledger's transaction. **409** without `RATCHET_STUDENT_BASE`; a failed load is **502** (`loading adapter student-v5 failed: HTTPStatusError 500; nothing changed`). Ignored in mock mode, whose fleet serves every version. A promotion to live answers both versions: `before` `{"v7": canary 50, "v6": live 100}`, `after` `{"v7": live 100, "v6": retired 0}`. ### POST `/ratchet/admin/student/disable`, `.../enable` Body `{"reason"}`. Sets `student_disabled`. While it is on, no request calls the student or the shadow candidate: the teacher answers (route `student_disabled`) and its answers are stored as on a cold start. A kill switch for a misbehaving endpoint or adapter that keeps every version's state as it is. ### POST `/ratchet/admin/releases/freeze`, `.../unfreeze` Body `{"reason"}`. Sets `releases_frozen`. While it is on the release gate starts no version taking traffic and gives none more: no shadow is promoted to canary, no canary advances or becomes live, and a cold-start candidate shadows the teacher instead of taking 10%. Rollbacks, rejections and the live alert run as usual, so a bad canary is still rolled back. Lifted, the gate goes on from the evidence gathered meanwhile. ### POST `/ratchet/admin/training/pause`, `.../resume` Body `{"reason"}`. Sets `training_paused`. While it is on, every night writes `skip` (`training paused by an admin; no training tonight`) and trains nothing. ### PUT `/ratchet/admin/settings` Body `{"audit_rate", "reason", "force"}`. `audit_rate` is required: a number from 0 to 1 that replaces `RATCHET_AUDIT_RATE` (the share of student answers the teacher audits), or `null` to go back to it. 0 while a canary is in flight is **409** unless `force`: with no audits it could neither advance nor be rolled back. With the rate at 0 the gate also starts no new canary. ### POST and DELETE `/ratchet/admin/tasks/{name}/pin` Body `{"reason"}`. Pin a registered task (built-in, file or API) to the teacher at runtime, or remove that pin. While pinned, its requests go to the teacher alone and nothing of them is stored (invariant 10), and Night School leaves its stored rows out of training and the golden eval from the next night on (`excluded_pinned`), as for a task its spec pins. The spec itself and its `spec_hash` do not change, so removing the pin needs no re-score. **404** for a name that is not registered; a task its spec pins answers `changed: false` to a pin and **409** to an unpin (change the spec instead). ### POST `/ratchet/admin/tick` Body `{"reason"}`, optional. Runs the release gate now, as the cron tick does, and records it (action `tick`, target `gate`). `before` and `after` are the live, canary and shadow versions around it (a version that left them, such as a rolled-back canary, is in both), `moved` lists the versions whose state changed, and `changed` says whether any did. The gate's own events say why. ### CLI `python -m ratchet.admin` works on `RATCHET_DB` (or `--db`; a missing file is an error, never created) with the actor `cli:`, through the same functions. Exit 0 on success (`nothing changed` on stderr when nothing did), 1 for a refused move (a 409), 2 for a bad input or an unknown version or task. ```bash python -m ratchet.admin versions [--json] python -m ratchet.admin rollback 7 --reason "refund labels are wrong since 09:00" python -m ratchet.admin retire 6 --reason "..." python -m ratchet.admin promote 5 --to canary [--traffic-pct 10] [--force] --reason "..." python -m ratchet.admin promote 5 --to live [--load-adapter] --reason "..." python -m ratchet.admin flag # the switches, who set them, the task pins (JSON) python -m ratchet.admin flag student_disabled on --reason "..." # on|off; also releases_frozen, training_paused python -m ratchet.admin flag audit_rate 0.2 --reason "..." # or `default`; --force for 0 during a canary python -m ratchet.admin pin ticket-triage --reason "..." # and unpin python -m ratchet.admin audit-log [--limit 50] [--json] ``` ## Test controls (mock mode only) All of these return **403** `these controls are only available in mock mode` when `RATCHET_MODE=live`. The mock-mode gateway is shared by every visitor, so the controls protect themselves: - **One mock-mode action at a time.** `day` and `reset` take a lock without waiting. If another `day` or `reset` is still running, the call returns **409** `another mock-mode action is running; try again in a moment`. (`break-next`, `student-outage` and `teacher-drift` do not take the lock.) - **Reset cooldown.** `reset` returns **429** `the mock-mode gateway was reset less than 30 seconds ago` within 30 seconds (`RESET_COOLDOWN_S`) of the last successful reset. The dashboard shows these messages under its buttons. ### `/ratchet/stats`: `links`, `demo` and `upstream` Three additive blocks, content-free, under `/ratchet/stats`'s access and scrubbing rules (a scrubbed anonymous copy passes them through unchanged). `tests/fixtures/stats_keys.json` lists them. | Block | Fields | |---|---| | `links` | `live`, `sim`: `RATCHET_LIVE_URL` and `RATCHET_SIM_URL` as set, else `null`. The menu link to the mock-mode gateway and the footer read them | | `demo` | `surface` (`simulation` in mock mode, else `live`), `controls` and `shared` (true in mock mode), `max_days`, `max_requests`, `idle_reset_s`, `actions_per_minute` (`null` when off), `auto_resets` (since the process started), `last_reset_at` (epoch seconds or `null`) | | `upstream` | `recording` (true in live mode), `since` (epoch seconds the record started, `null` in mock mode), `token_factory` (whether `RATCHET_TEACHER_BASE` is Token Factory's host, which decides if a page may say "Live on Nebius Token Factory"), `calls`: the newest call per role called since start, in the order teacher, fallback_teacher, student, tavily | A `calls` item: `role`, `host` (`token_factory`, `student_endpoint`, `tavily` or `other`), `requested_model`, `model` (the response's `model`, kept only when it looks like a model id, else `null`; Token Factory echoing the id is unverified), `ok`, `status` (the upstream's HTTP status, `null` without one), `error` (the exception's class name), `reason` (`timeout`, `unreachable`, `bad_response`, `http` or `other`), `latency_ms`, `at` (epoch seconds), `age_s` and `last_ok_age_s` (seconds since the last success of that role, `null` if none). It **never contains** prompt or answer text, URLs, keys, headers, error messages or the upstream's own error code. Breaker refusals, adapter loads, Night School's judge and other processes (the live-run harness) are not recorded. The hosted Nano is the `student` role on `token_factory` with the fallback model's id as `requested_model`; our adapter is the `student` role on `student_endpoint`. ### POST `/ratchet/demo/day?requests=1500&workload=tickets` Advances `Ledger.day` by one and sends `requests` mock requests through the cascade, calling `gate.tick()` every `requests/10`. Then it runs one night (`run_night`). `requests` is clamped to 100–5000. | Query param | Default | Values | |---|---|---| | `requests` | `1500` | Requests for the day (clamped to 100–5000) | | `workload` | `tickets` | `tickets` (support tickets), `research` (Scout's triage and reader calls), `mixed` (half each), `schema` (structured outputs: `response_format` json_schema requests over three schemas under the tasks `review-rating`, `ticket-route` and `meeting-extract`, and from day 5 `warranty-claim`, with nested records) or `tools` (support requests sent with the mock tool catalogue under the task `order-support`). Anything else returns **400** `workload must be one of [...]`; `schema` and `tools` return **400** under `RATCHET_MOCK_STRICT=1`, before the day starts, since only the generic mock world answers them | With `workload=tickets` the night trains on `ticket-triage` only, as in v0.1. With any other workload it trains on every stored task. Each workload has its own seeded stream, so the streams do not affect each other. The dashboard's **Traffic** selector sets `workload` (default there: `mixed`). A `tools` workload request is a tool request: the mock teacher answers it alone with a tool call, nothing is stored, and the student never learns it. The numbers are mock-model output from mock models, and the response says so (`simulation: true`). ```json {"day": 4, "workload": "tickets", "simulation": true, "auto_reset": null, "night": {"trained": false, "version": null, "admitted": false, "reason": "release in flight", "eval": null}} ``` `night.reason` is one of `admitted`, `rejected`, `release in flight`, `not enough new data`, `no golden set`. `auto_reset` is `null`, or `"max_days"` or `"max_requests"` when the shared mock-mode gateway reached `RATCHET_DEMO_MAX_DAYS` or `RATCHET_DEMO_MAX_REQUESTS`: it started over (a fresh ledger, the warm-up again, one `demo_reset` event) before this day ran, so `day` is then the first day after the warm-up. On a gateway with `RATCHET_DEMO_RPM` set, every non-GET `/ratchet/demo/*` call takes a token from the caller's IP bucket, and an empty bucket answers **429** `the shared simulation takes N actions a minute per visitor; try again in S s` with `Retry-After`. Live mode answers 403 before any of this. ### POST `/ratchet/demo/break-next` Marks the next version name (`student-v{count+1}`) to be served corrupted once it reaches canary or live. Its billing, refund and security answers become confidently wrong. This is used to demonstrate automatic rollback. It writes a `fault` event. ```json {"corrupt": "student-v4"} ``` ### POST `/ratchet/demo/student-outage?seconds=60` Fault injection for the student circuit breaker: for `seconds` (clamped to 5–300, default 60) every call to the production student endpoint, which serves the live and canary versions, fails as an unreachable endpoint does (`httpx.ConnectError`). Shadow candidates run on the mock staging endpoint and are not affected. It writes a `fault` event and returns when the outage ends (Unix time): ```json {"seconds": 60, "until": 1792400060.0} ``` Send traffic (Scout, or **Run 1 day + night**) and the release log shows the breaker at work: a `student_error` event for the first failure (`v2: ConnectError`; later ones are logged on the 10th, 100th and 1000th), then, after 5 failures in a row, `circuit` `student: closed->open after 5 failures (ConnectError)`; requests go straight to the teacher (route `student_breaker`); after 30 s one probe (`open->half_open`), which fails while the outage lasts and doubles the wait (`half_open->open for 60 s ...`); once the endpoint is back, a probe succeeds and `half_open->closed`. Calling it again starts a new outage of the given length from now. The dashboard's **Resilience** panel runs this drill (30 s, 1 min or 2 min) and reads the breaker's state from the `circuit` events and the outage's end from the `fault` event's `ts` plus its length; the wording of both events is parsed there, and `test_resilience_panel_contract` checks the parser against the gateway's events. ### GET `/ratchet/demo/sample-example?split=train&version=3` A committed example of `version` (default: the live version; **404** `no live version yet` before there is one) from its `split` set (`train` or `golden`, default `train`; anything else is **400**), as a body for [`POST /ratchet/versions/{id}/proof`](#post-ratchetversionsidproofsplittrain): its task, its stored (redacted) messages and the conditions it was stored under, never its answer. It is the first leaf, in key order, whose example is still stored; **404** for an unknown version or one with no such example. Read-only and lock-free. ```json {"version": 3, "split": "train", "task": "ticket-triage", "messages": [{"role": "system", "content": "You triage customer support tickets. …"}, {"role": "user", "content": "…"}]} ``` Post it back with `split=train` for an `included` proof, or take a `golden` sample and post it with `split=train` for an `absent` one. ### POST `/ratchet/demo/teacher-drift?noise=0.15` Fault injection for the [teacher watch](#teacher-watch): from now on the mock teacher changes the first label of an answer (a ticket's category) to another value with probability `noise` (0 to 1, default 0.15; **400** outside it). 0 makes it stable again; a reset does too. Each draw is seeded by the request text and a call counter, so a run is reproducible. It writes a `fault` event: ```json {"noise": 0.15} ``` Then run days: re-tests and Night School's probes start to disagree with the stored labels, contested labels are adjudicated each night (`teacher_watch` events), and once the e-value reaches 100 a `teacher_drift` event freezes Night School (`skip` events) until `POST /ratchet/teacher/ack`. At 0.15 that takes a few days of mock traffic, because a student that serves most requests leaves few re-tests to the teacher (mock models). ### POST `/ratchet/demo/reset` Rebuilds the runtime: new in-memory ledger, new mock fleet, all three traffic streams reset, new Scout budget guard. If `RATCHET_MOCK_WARMUP_DAYS` is set, the warm-up runs again before the call returns. ```json {"status": "reset"} ``` --- ## Scout endpoints Scout is the multi-agent research live gateway. The event contract (every field of `start`, `step`, `done` and `error` events) is in [SCOUT.md](/reference/scout/#event-objects-sse-data-payloads-in-order). This section covers the HTTP side. Before a run starts, both run endpoints check, in this order: | Check | `/scout/run` | `/scout/stream` | |---|---|---| | Question empty after trimming | 400 `Ask a question.` | `error` event | | Question longer than 300 characters | 400 | `error` event | | Mock mode and the question is not one of the example questions (exact match) | 400 | `error` event | | Live mode without `TAVILY_API_KEY` | 503 | `error` event | | Budget guard refuses: daily budget used, daily run cap reached, or per-IP hourly limit | 429 (message says which) | `error` event | A run that passes these checks counts against the per-IP hourly limit and the daily run cap, even if it fails later. Its spend is added to the daily total step by step, so a run that fails halfway still counts what it spent. In live mode the counters are kept in the ledger and restored at startup. **Judge runs.** A request that carries a valid `ratchet_judge` cookie (from `POST /scout/judge`) is a judge run: it draws on the runs and spend held back for judges (`RATCHET_SCOUT_JUDGE_RUNS_PER_DAY`, `RATCHET_SCOUT_JUDGE_BUDGET_USD`), and on public room that is left, and has its own per-IP hourly window. Only the cookie counts: a `code` query parameter or body field is ignored. When the public share is used up, the public gets 429 `The live gateway has reached its Scout run limit for today. ...` or `The daily Scout budget is used up. ...`; a judge run past the whole allowance gets `Today's Scout allowance, judges' share included, is used up.` None of them mentions a code. **Search unavailable.** A search that fails (any HTTP or transport error) or that the Tavily credit meter refuses does not end the run: the searcher step carries `detail.unavailable` (the reason), no more queries are made, triage and reader run on whatever earlier queries found, and the writer answers (without sources if nothing was found). The `done` event says so in `search`. The planner, pinned to the teacher, has called Token Factory before any search, so every admitted run makes a runtime Token Factory call. The client IP is the TCP peer address, or the first `X-Forwarded-For` entry when `RATCHET_TRUST_PROXY=1` ([CONFIGURATION.md](/reference/configuration/#scout-the-multi-agent-research-tool)). ### GET `/scout/examples` ```json {"mode": "mock", "examples": ["What is LoRA fine-tuning?", "What is knowledge distillation?", "..."], "free_text": false} ``` `examples` is `ratchet.scout.prompts.EXAMPLE_QUESTIONS` (8 questions). `free_text` is `true` only in live mode. ### GET `/scout/status` ```json {"mode": "live", "budget_usd": 3.0, "spent_today_usd": 0.0412, "runs_today": 3, "runs_per_hour_per_ip": 6, "max_runs_per_day": 22, "tavily": true, "fallback_student": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B", "student_serving": {"kind": "hosted_fallback", "model": "nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B", "version": 3, "threshold": 0.8, "certified": false}, "live_check": {"state": "unknown", "since": null, "checked": null, "search": null}, "public_runs_left_today": 13, "public_budget_left_usd": 2.0088, "judge_runs_today": 1, "persistent": true, "judge": {"enabled": true, "session": false, "reserved_runs_per_day": 7, "runs_today": 1, "runs_left_today": 19}, "search": {"provider": "tavily", "available": true, "last_error": null, "errors_today": 0, "credits": {"monthly": 1000, "used_this_cycle": 9, "left": 991, "public_left_today": 39, "judge_reserve": 318, "judge_left": 315, "cycle_start": "2026-12-01", "pace_until": "2026-12-15", "exhausted_until": null}}} ``` | Field | Meaning | |---|---| | `budget_usd` | `RATCHET_SCOUT_DAILY_BUDGET_USD`; `null` in mock mode (no budget) | | `spent_today_usd` | Ratchet spend on Scout runs this UTC day (`cost_usd` of every step, including audits) | | `runs_today` | Runs admitted this UTC day | | `runs_per_hour_per_ip` | Per-IP limit. Mock mode uses 10 × `RATCHET_SCOUT_RUNS_PER_HOUR` | | `max_runs_per_day` | `RATCHET_SCOUT_MAX_RUNS_PER_DAY`; `null` in mock mode (no cap) | | `tavily` | `true` when live search (Tavily) is configured | | `fallback_student` | v0.5: the hosted fallback's model id, as in `/ratchet/stats`. A `step` whose `served_by` is `student` and whose `model` is this id came from the untuned hosted Nano, not our adapter | | `student_serving` | v0.5: the same object as in [`/ratchet/stats`](#get-ratchetstats), so the Scout page labels each call without a stats call | | `live_check` | The live gateway watchdog's flag: `{"state", "since", "checked", "search"}`. `state` is `failing` only after the watchdog's hourly probe, through the public URL, got a teacher-side or gateway-side failure within `RATCHET_WATCH_FLAG_MAX_AGE_S`; `ok`; `stale` (the watchdog stopped); `unknown` (no row yet); `off` (mock mode). Never an error code or text, so a keyword monitor can match `"state":"failing"` and a page can show a banner | | `public_runs_left_today` | Runs the public can still start today: the run cap minus the judges' reserve, minus public runs and judge runs past the reserve; `null` without a cap (mock mode) | | `public_budget_left_usd` | The same for spend; `null` without a budget. A public run can be refused a little earlier, while a judge budget is reserved: the largest finished run today is kept as a margin | | `judge_runs_today` | Judge runs admitted today | | `persistent` | `true` when the counters are kept in the ledger (live mode) and survive a restart | | `judge` | `enabled` (judge access is on), `session` (this request carries a valid judge cookie), `reserved_runs_per_day`, `runs_today`, `runs_left_today` (the reserve plus public room left; `null` without a cap) | | `search` | `provider` (`tavily`, `mock-search` or `null`), `available` (configured, not paused after Tavily's 432 or 433, and the last search did not fail), `last_error` (the last failure's class and status, `null` once a search works again), `errors_today` (failures, not credit refusals) and `credits` (`null` without a meter; otherwise the cycle's credits, used (searches in flight included) and left, the public's allowance left today, the judge reserve and what is left of it, the cycle's first day, `RATCHET_TAVILY_PACE_UNTIL` and when a pause after a 432 or 433 ends; `null` limits when `RATCHET_TAVILY_MONTHLY_CREDITS` is 0) | Every v0.2 key keeps its name and type; new keys are only added. In live mode the counters are restored from the ledger at startup and reset at 00:00 UTC; in mock mode they live in memory. ### POST `/scout/judge` Opens a judge session. Body: `{"code": ""}`. The code travels only here, in a POST body, because uvicorn and Caddy write URLs and custom headers to their access logs. The body is parsed by hand, so no validation error can echo it. | Status | When | |---|---| | 200 | `{"judge": true, "expires": "2026-12-08T10:00:00+00:00", "runs_left_today": 19}` and `Set-Cookie: ratchet_judge=; Path=/scout; Max-Age=; HttpOnly; SameSite=Strict` (plus `Secure` in live mode). Writes the event `judge` `judge session opened`, with no IP and no code | | 400 | `Send {"code": "..."}.`: not a JSON object with a non-empty string `code` | | 401 | `That code is not valid.`, with no cookie | | 404 | `Judge access is not enabled on this gateway.` (`RATCHET_SCOUT_JUDGE_CODE` is empty) | | 429 | More than 10 attempts from this IP in a rolling hour (well-formed attempts, the right code included) | The cookie is `v1...`, keyed from the code: it survives a restart, and rotating the code ends every session. `SameSite=Strict` keeps another site from starting a judge's runs through `/scout/stream`, which is a GET. ### DELETE `/scout/judge` 204, and clears the cookie (`Max-Age=0`). Idempotent; needs no session. ### GET `/scout/stream?q=` Server-Sent Events (`text/event-stream`). Each event is one `data:` line holding a JSON object: `start`, then `step` events as agents finish, then `done`. Lines starting with `:` are keep-alive comments, sent every 15 seconds while waiting. This endpoint **always answers HTTP 200**, because the browser's `EventSource` cannot read status codes. Any failure, before or during the run, arrives as a single `error` event, which ends the stream. In mock mode the run waits `RATCHET_MOCK_PACE` × each call's mock latency, so the page shows agents working. ```bash curl -sN "http://localhost:8000/scout/stream?q=What%20is%20LoRA%20fine-tuning%3F" ``` ### POST `/scout/run` Request body: `{"question": "What is LoRA fine-tuning?"}`. Runs without pacing and returns once the run ends: ```json {"events": [{"type": "start", "...": "..."}, {"type": "step", "...": "..."}, {"type": "done", "...": "..."}]} ``` | Status | When | |---|---| | 200 | Run finished; the last event is `done`. Also when search was unavailable: `done.search.available` is `false` and the answer was written without (some) web sources | | 400 / 429 / 503 | Pre-run checks above | | 502 | A model call failed partway (`a model or search call failed: upstream 503`, or `upstream 502 upstream_unreachable`, `upstream 504 upstream_timeout`: the status and code, never the URL), or anything else raised (`run failed: : `). `detail` is the `error` event's message. A failed search is no longer a 502 (v0.5) | | 422 | Body is not JSON | --- ## Python API (embedding Ratchet without HTTP) ```python from ratchet.config import Settings from ratchet.ledger import Ledger from ratchet.providers import OpenAICompatProvider from ratchet.release import ReleaseGate from ratchet.router import Ratchet s = Settings(mode="live") ledger = Ledger(s.db_path, sync=s.ledger_sync()) # a file: the WAL journal unless RATCHET_DB_SYNC=FULL # The gateway's timeouts and retries: the teacher retried within its deadline, the student never (the teacher is its retry). teacher = OpenAICompatProvider(s.teacher_base, s.teacher_api_key, timeout=s.teacher_timeout(), retry=s.teacher_retry()) student = OpenAICompatProvider(s.student_base, s.student_api_key, timeout=s.student_timeout()) r = Ratchet(s, ledger, teacher, lambda v: (student, v.adapter_uri)) gate = ReleaseGate(ledger) result = r.handle([{"role": "user", "content": "..."}], "ticket-triage") gate.tick() print(result.served_by, result.text, result.cost_usd) # With the client's generation parameters (what the gateway passes) and what else the caller knows: from ratchet.providers import GenParams from ratchet.router import RequestMeta result = r.handle(msgs, "ticket-triage", client="acme", params=GenParams(max_tokens=200, stop=("\n\n",)), meta=RequestMeta()) print(result.finish_reason) # "stop", or "length" when the answer was cut off print(result.prompt_tokens, result.completion_tokens, result.total_tokens, result.fingerprint) # Two choices, with token logprobs: result = r.handle(msgs, "ticket-triage", params=GenParams(logprobs=True, top_logprobs=2), n=2) for choice in result.choices: print(choice.text, choice.finish_reason, choice.logprobs) # logprobs: OpenAI's content entries, or None # Wall-clock latency (requests.wall_ms, the whole cascade) by nearest rank, of one day or all days, and per day: print(ledger.latency_percentiles(day=None, served_by="student", qs=(0.5, 0.95))) # {0.5: ms, 0.95: ms}, None if no rows print([(d["day"], d["p50_student_ms"], d["p95_ms"]) for d in ledger.stats_by_day()]) ``` Tasks for an embedded `Ratchet` come from the same registry: `ratchet.taskspec.install(ledger, "tasks.json")` registers the built-in tasks, a tasks file and the API tasks a ledger stored, and `ratchet.tasks.register(taskspec.parse_spec(spec, "api"))` adds one. `r.task_for(name, client)` applies `RATCHET_UNKNOWN_TASKS` and a task's `clients` as the gateway does (`tasks.UnknownTask`, `tasks.TaskForbidden`); `handle` applies the unknown-task policy only. `handle(messages, task=None, client=None, params=None, meta=None, n=1)`: `task` is a task name or a `tasks.Task`; `n` is the number of choices ([Choices and logprobs](#choices-and-logprobs)), and `teacher_only` takes it too, with an optional `route` for its row (the gateway passes `multimodal`); a request whose `params.tools` or messages make it a tool request goes to the teacher alone (route `tools`, [Tools and function calling](#tools-and-function-calling)); `params` (`providers.GenParams`, `GenParams.from_request(chat_request)` in the gateway) are sent to each call as `Settings.gen_policy()` allows ([Generation parameters](#generation-parameters)); `meta` (`router.RequestMeta`) carries what else the caller knows: `RequestMeta(storable=False)` keeps the request's teacher answers out of the training set (with `store_opt_out` naming why, `request` by default), and `end_user` is the end user's hash (`privacy.EndUsers.hash`, never the raw value), with `end_user_aliases` the other hashes a no-train registration of that end user may use (`privacy.EndUsers.aliases`; looked up, never stored). `Result.stored` and `Result.store_opt_out` say what happened; zero retention, no-store tasks and no-train end users apply to every caller, Scout included. A provider's `complete(messages, model, n, temperature, logprobs, response_format, params=None, **_)` receives the role's `GenParams`; the explicit arguments win over the same fields in it. `result.route` says how the request was served (the routes table under [`/ratchet/stats`](#get-ratchetstats)); `result.text` is the answer after `tasks.clean_answer`. `result.choices` are the response's choices (`router.Choice`: `text`, `finish_reason`, `logprobs`, OpenAI's `logprobs.content` entries when `params.logprobs` was set and they could be given, else None, and `tool_calls`, the teacher's tool calls for a tool request, else None); `result.text` and `result.finish_reason` are choice 0's. `result.prompt_tokens`, `completion_tokens`, `reasoning_tokens`, `cached_tokens` and `total_tokens` are the response's [usage](#usage), `result.usage_breakdown` its `ratchet.usage_breakdown` (keys in `router.USAGE_PARTS`), and `result.fingerprint` its `system_fingerprint` (`router.fingerprint(...)`). Each `Completion` a provider returns carries `finish_reason` (`stop` when the server leaves it out or sends an empty string), `reasoning_tokens`, `cached_tokens` (from `usage.prompt_tokens_details`, 0 when absent) and `logprobs_content` (the choice's `logprobs.content` as the server sent it, or None; `providers.served_logprobs` turns it into what a client gets); only a teacher answer with `finish_reason` `stop` or `tool_calls` that is non-empty and valid for its task is stored (invariant 16), and `requests.stored` says whether a request stored one. `result.degraded` is `"teacher_unavailable"` for a degraded answer, else None. `result.task` is the resolved task's name and `result.schema_valid` whether every choice meets the request's `response_format` (None without one); `r.resolve(task, params)` is the task a request with those parameters gets (a registered task, or a schema's auto task, bound to the `response_format`), and `r.task_for(task, client, params)` is the same with the gateway's admission checks. `r.health()` returns the failure counters (`student_errors`, `audit_errors`, `shadow_errors`, `repair_errors` for schema repairs that raised), the answers a stand-in wrote for an unavailable teacher (`degraded_answers`, `fallback_answers`) and `breakers`: each breaker's state by name (`student`, `shadow`, `teacher`, `fallback_teacher`), as `{"state": "closed" | "open" | "half_open", "failures": n, "open_until": t, "open_s": s}`, where `open_until` is on the breaker's clock (`time.monotonic()` unless `Ratchet(clock=...)` was given). `Ratchet(..., on_adapter_missing=callback)` is called with the version when vLLM answers 404 for its adapter; the gateway passes `live.AdapterReloader.request`. `Ratchet(..., timer=time.perf_counter)` is the clock `requests.wall_ms` is measured on (tests pass a fake one). Off-path checks: a `Ratchet` builds its `sidecar` from `Settings.sidecar()` (live mode: 4 workers; mock mode: none), and a call that passes a `RequestMeta` then runs its audit and shadow after it returns: `result.overhead_pending` is True, `result.cost_usd` is the serving cost and `result.audited` False, and the ledger row is finished when the job ends. Call `r.close(drain_s)` before the process exits so queued jobs finish (it returns `r.sidecar_report()`: the queue's counts and, per version, `audits_inline`, `shadows_dropped`, `audits_abandoned`, `shadows_abandoned`). A call without `meta`, or with `RequestMeta(offpath=False)`, or a `Ratchet(..., sidecar=ratchet.sidecar.SideCar(0))`, keeps every check inline, as Scout's agents and the simulator do. When the teacher's answer was needed and the teacher failed after its retries (and no fallback teacher or degraded answer stood in), `handle` raises `ratchet.providers.UpstreamError` (an `httpx.HTTPError`): `status` (504 for a timeout, 502 when Token Factory could not be reached), and the teacher's `type`, `code`, `param`, `message` and `retry_after` when it sent them; while the teacher's breaker is open, `ratchet.resilience.CircuitOpen` (503, code `teacher_circuit_open`, `retry_after` until the next probe). It never carries the URL or headers. `Ratchet(..., fallback_teacher=provider)` sets the fallback teacher's provider (default: the teacher's); it is used only when `RATCHET_TEACHER_FALLBACK_MODEL` is set. --- # Endpoint map Source: https://docs.ratchet.dualmindlab.tech/api/endpoints/ The gateway serves 85 routes; this page groups them by purpose, says which key each one needs, and shows the three calls most people make first. The full schema of every route is in the generated [HTTP API reference](/api/http/). The long prose guide, with every field and error, is the [HTTP API guide](/reference/api/). This page is the map between them. ## The routes by purpose
```mermaid flowchart LR app(["Your app"]) watch(["Dashboard,
monitors, auditors"]) ops(["Operator"]) web(["Browsers"]) S["Serve
/v1/chat/completions
/v1/completions, /v1/responses
/v1/embeddings, /v1/models
/v1/ratchet/feedback, no-train,
examples"] P["Prove
/ratchet/bom
/ratchet/versions/id/bom,
certificate, proof
/ratchet/receipts"] O["Observe
/ratchet/stats, kpis, cost
/ratchet/drift, teacher, replay
/metrics, /healthz, /readyz"] D["Operate
/ratchet/admin/*
/ratchet/tasks
/ratchet/cost/endpoint
/ratchet/teacher/ack"] E["Pages
pages: / /dashboard /scout /judges
/scout/*"] app --> S app --> P watch --> O watch --> P ops --> D web --> E class S student class D gate class E outline ```
**Reading this diagram.** Read it from the left, from who calls to what they call. Your application only needs the first group, Serve: the OpenAI-shaped routes and the three Ratchet routes for feedback, no-train registrations and erasing its own examples. Prove holds the signed evidence: bills of materials, certificates, proofs and savings receipts. Observe is what the dashboard and your monitors read. Operate is for the person running the gateway: versions, switches, task pins and the task registry. Pages are the web pages and the Scout routes.
## Who may call a route Every route has one access level. A key goes in `Authorization: Bearer ` or in `X-API-Key: `.
```mermaid flowchart LR pub{{"public"}} --> r1(["Anyone"]) api{{"api"}} --> r2(["A valid API key, once
RATCHET_API_KEYS is set"]) dash{{"dashboard, bom,
metrics"}} --> r3(["Anyone on a public dashboard,
else an admin or dashboard key"]) adm{{"admin"}} --> r4(["An admin key,
with an audit trail"]) mock{{"mock"}} --> r5(["Mock mode only,
403 in live mode"]) class pub,api,dash,adm,mock gate class r1 ok class r5 outline ```
**Reading this diagram.** Each row is one access level, on the left, and who may call a route at that level, on the right. Public routes never ask for a key. API routes ask for a key only when `RATCHET_API_KEYS` is set. The read-only management routes (dashboard, bill of materials, metrics) are open on a public dashboard, which is the default in mock mode; in live mode they need an admin or dashboard key, and a gateway with neither configured refuses them. Admin routes always need an admin key in live mode. A route nobody has classified counts as dashboard level, so a new route starts private. On a public dashboard, a reader without a key still gets an answer, with client names replaced by pseudonyms. A missing or wrong key is a 401 or 403; on `/v1/*` a missing API key is a 401 `invalid_api_key`.
The bill of materials routes also open to anyone with `RATCHET_PUBLIC_BOM=1`, so auditors can read provenance from a private gateway. The metrics route also accepts `RATCHET_METRICS_TOKEN`, so a scraper needs no dashboard key. Savings receipts are API-level: a key reads its own, an admin key reads all. ## Every route The Method and Path columns link to the generated reference for that operation. ### OpenAI-compatible | Method | Path | Who | What it is for | |---|---|---|---| | GET | [`/v1/models`](/api/http/operations/models_v1_models_get/) | API key | The tasks as OpenAI model objects. | | GET | [`/v1/models/{model_id}`](/api/http/operations/retrieve_model_v1_models__model_id__get/) | API key | One model object. | | POST | [`/v1/chat/completions`](/api/http/operations/chat_v1_chat_completions_post/) | API key | The chat completion: student first, teacher when unsure. | | POST | [`/openai/deployments/{deployment}/chat/completions`](/api/http/operations/azure_chat_openai_deployments__deployment__chat_completions_post/) | API key | The same chat route under an Azure-style path. | | POST | [`/v1/embeddings`](/api/http/operations/embeddings_v1_embeddings_post/) | API key | Embeddings, passed through and never stored. | | POST | [`/v1/completions`](/api/http/operations/completions_v1_completions_post/) | API key | Legacy text completions, run as chat. | | POST | [`/v1/responses`](/api/http/operations/responses_v1_responses_post/) | API key | The stateless part of the Responses API, run as chat. | | POST | [`/v1/moderations`](/api/http/operations/moderations_v1_moderations_post/) | API key | Not served: a typed 404. | | GET | [`/v1/responses/{response_id}`](/api/http/operations/response_state_v1_responses__response_id__get/) | API key | Not served: a typed 404 (stored responses). | | POST | [`/v1/responses/{response_id}`](/api/http/operations/response_state_v1_responses__response_id__post/) | API key | Not served: a typed 404. | | DELETE | [`/v1/responses/{response_id}`](/api/http/operations/response_state_v1_responses__response_id__delete/) | API key | Not served: a typed 404. | ### End users and feedback | Method | Path | Who | What it is for | |---|---|---|---| | POST | [`/v1/ratchet/feedback`](/api/http/operations/feedback_post_v1_ratchet_feedback_post/) | API key | A verdict on an answer, with an optional correction. | | POST | [`/v1/ratchet/no-train`](/api/http/operations/no_train_add_v1_ratchet_no_train_post/) | API key | Register an end user whose requests are never stored. | | DELETE | [`/v1/ratchet/no-train`](/api/http/operations/no_train_remove_v1_ratchet_no_train_delete/) | API key | Remove that registration. | | DELETE | [`/v1/ratchet/examples`](/api/http/operations/erase_examples_v1_ratchet_examples_delete/) | API key | Erase the caller's stored examples (by end user, hashes or text). | | GET | [`/v1/ratchet/requests/{request_id}`](/api/http/operations/own_request_v1_ratchet_requests__request_id__get/) | API key | The caller's own request record. | ### Health and metrics | Method | Path | Who | What it is for | |---|---|---|---| | GET | [`/healthz`](/api/http/operations/healthz_healthz_get/) | Public | Liveness plus mode and version. | | GET | [`/livez`](/api/http/operations/livez_livez_get/) | Public | Liveness (alias of `/healthz`). | | GET | [`/readyz`](/api/http/operations/readyz_readyz_get/) | Public | Readiness: ledger, teacher and student probes. 503 when the gateway cannot serve. | | GET | [`/metrics`](/api/http/operations/metrics_endpoint_metrics_get/) | Dashboard key or metrics token | Prometheus metrics. | ### Stats and provenance | Method | Path | Who | What it is for | |---|---|---|---| | GET | [`/ratchet/stats`](/api/http/operations/stats_ratchet_stats_get/) | Dashboard key | The dashboard's numbers: routes, tasks, clients, versions, events. | | GET | [`/ratchet/requests/{request_id}`](/api/http/operations/request_lookup_ratchet_requests__request_id__get/) | Dashboard key | One request's record (metadata, never text). | | GET | [`/ratchet/kpis`](/api/http/operations/kpi_ratchet_kpis_get/) | Dashboard key | The success metrics computed from the ledger. | | GET | [`/ratchet/drift`](/api/http/operations/drift_ratchet_drift_get/) | Dashboard key | The label-free drift alarm's state per version and task. | | GET | [`/ratchet/bom`](/api/http/operations/bom_ratchet_bom_get/) | Dashboard key (public with `RATCHET_PUBLIC_BOM=1`) | The live adapter's signed bill of materials. | | POST | [`/ratchet/redact`](/api/http/operations/redact_preview_ratchet_redact_post/) | Dashboard key | Preview what the redactor would store for a text. | | GET | [`/ratchet/bom/public-key`](/api/http/operations/bom_public_key_ratchet_bom_public_key_get/) | Public | The public key that verifies bills of materials. | | GET | [`/ratchet/teacher`](/api/http/operations/teacher_ratchet_teacher_get/) | Dashboard key | The teacher watch: check epoch, noise evidence, freeze state. | | POST | [`/ratchet/teacher/ack`](/api/http/operations/teacher_ack_ratchet_teacher_ack_post/) | Admin key | Acknowledge a teacher drift freeze so training can resume. | ### Versions and evidence | Method | Path | Who | What it is for | |---|---|---|---| | GET | [`/ratchet/versions/{vid}/bom`](/api/http/operations/version_bom_ratchet_versions__vid__bom_get/) | Dashboard key (public with `RATCHET_PUBLIC_BOM=1`) | One version's signed bill of materials. | | GET | [`/ratchet/versions/{vid}/certificate`](/api/http/operations/version_certificate_ratchet_versions__vid__certificate_get/) | Dashboard key | One version's certified precision. | | POST | [`/ratchet/versions/{vid}/proof`](/api/http/operations/version_proof_ratchet_versions__vid__proof_post/) | Admin key | A proof that an example is, or is not, in a version's training set. | | GET | [`/ratchet/versions/{vid}/heldout`](/api/http/operations/version_heldout_ratchet_versions__vid__heldout_get/) | Admin key | A proof of absence from training for every golden row. | ### Cost | Method | Path | Who | What it is for | |---|---|---|---| | GET | [`/ratchet/cost`](/api/http/operations/cost_ratchet_cost_get/) | Dashboard key | Savings per token and by GPU-hours paid, break-even, recommendation. | | POST | [`/ratchet/cost/endpoint`](/api/http/operations/cost_endpoint_ratchet_cost_endpoint_post/) | Admin key | Record that the student endpoint went up or down. | ### Ledger replay | Method | Path | Who | What it is for | |---|---|---|---| | GET | [`/ratchet/replay`](/api/http/operations/replay_version_ratchet_replay_get/) | Dashboard key | Replay a version's traffic at another threshold or audit rate. | | GET | [`/ratchet/replay/frontier`](/api/http/operations/replay_frontier_ratchet_replay_frontier_get/) | Dashboard key | The same replay over a range of thresholds. | ### Savings receipts | Method | Path | Who | What it is for | |---|---|---|---| | GET | [`/ratchet/receipts`](/api/http/operations/list_receipts_ratchet_receipts_get/) | API key | The caller's sealed savings receipts. | | GET | [`/ratchet/receipts/{period}`](/api/http/operations/get_receipt_ratchet_receipts__period__get/) | API key | One day's receipt. | | GET | [`/ratchet/receipts/{period}/proof/{request_id}`](/api/http/operations/receipt_proof_ratchet_receipts__period__proof__request_id__get/) | API key | An inclusion proof for one request in a receipt. | | GET | [`/ratchet/receipts/{period}/leaves`](/api/http/operations/receipt_leaves_ratchet_receipts__period__leaves_get/) | API key | The leaves a receipt's root commits to. | ### Tasks | Method | Path | Who | What it is for | |---|---|---|---| | GET | [`/ratchet/tasks`](/api/http/operations/list_tasks_ratchet_tasks_get/) | Dashboard key | Every registered task. | | GET | [`/ratchet/tasks/{name}`](/api/http/operations/get_task_ratchet_tasks__name__get/) | Dashboard key | One task's spec. | | PUT | [`/ratchet/tasks/{name}`](/api/http/operations/put_task_ratchet_tasks__name__put/) | Admin key | Add or change an API task. | | DELETE | [`/ratchet/tasks/{name}`](/api/http/operations/delete_task_ratchet_tasks__name__delete/) | Admin key | Remove an API task. | | POST | [`/ratchet/tasks/validate`](/api/http/operations/validate_tasks_ratchet_tasks_validate_post/) | Admin key | Check task specs without saving them. | ### Admin | Method | Path | Who | What it is for | |---|---|---|---| | POST | [`/ratchet/admin/privacy/erase`](/api/http/operations/admin_erase_ratchet_admin_privacy_erase_post/) | Admin key | Erase stored examples for good (dry run by default). | | GET | [`/ratchet/admin/export/{kind}`](/api/http/operations/admin_export_ratchet_admin_export__kind__get/) | Admin key | A ledger table as NDJSON. | | GET | [`/ratchet/admin/versions`](/api/http/operations/admin_versions_ratchet_admin_versions_get/) | Admin key | Versions, runtime switches and task pins. | | GET | [`/ratchet/admin/flags`](/api/http/operations/admin_flags_ratchet_admin_flags_get/) | Admin key | The runtime switches. | | POST | [`/ratchet/admin/versions/{vid}/rollback`](/api/http/operations/admin_rollback_ratchet_admin_versions__vid__rollback_post/) | Admin key | Roll a canary back. | | POST | [`/ratchet/admin/versions/{vid}/retire`](/api/http/operations/admin_retire_ratchet_admin_versions__vid__retire_post/) | Admin key | Retire a version. | | POST | [`/ratchet/admin/versions/{vid}/promote`](/api/http/operations/admin_promote_ratchet_admin_versions__vid__promote_post/) | Admin key | Move a version to a canary stage or to live. | | POST | [`/ratchet/admin/student/disable`](/api/http/operations/admin_student_disable_ratchet_admin_student_disable_post/) | Admin key | Switch the student off. | | POST | [`/ratchet/admin/student/enable`](/api/http/operations/admin_student_enable_ratchet_admin_student_enable_post/) | Admin key | Switch the student on. | | POST | [`/ratchet/admin/releases/freeze`](/api/http/operations/admin_releases_freeze_ratchet_admin_releases_freeze_post/) | Admin key | Freeze releases (rollbacks and alerts keep running). | | POST | [`/ratchet/admin/releases/unfreeze`](/api/http/operations/admin_releases_unfreeze_ratchet_admin_releases_unfreeze_post/) | Admin key | Unfreeze releases. | | POST | [`/ratchet/admin/training/pause`](/api/http/operations/admin_training_pause_ratchet_admin_training_pause_post/) | Admin key | Pause Night School training. | | POST | [`/ratchet/admin/training/resume`](/api/http/operations/admin_training_resume_ratchet_admin_training_resume_post/) | Admin key | Resume training. | | PUT | [`/ratchet/admin/settings`](/api/http/operations/admin_settings_ratchet_admin_settings_put/) | Admin key | Set a runtime setting such as the audit rate. | | POST | [`/ratchet/admin/tasks/{name}/pin`](/api/http/operations/admin_pin_ratchet_admin_tasks__name__pin_post/) | Admin key | Pin a task to the teacher. | | DELETE | [`/ratchet/admin/tasks/{name}/pin`](/api/http/operations/admin_unpin_ratchet_admin_tasks__name__pin_delete/) | Admin key | Remove a runtime pin. | | POST | [`/ratchet/admin/tick`](/api/http/operations/admin_tick_ratchet_admin_tick_post/) | Admin key | Run the release gate's tick now. | | GET | [`/ratchet/admin/audit-log`](/api/http/operations/admin_audit_log_ratchet_admin_audit_log_get/) | Admin key | The newest admin actions. | ### Scout | Method | Path | Who | What it is for | |---|---|---|---| | GET | [`/scout/examples`](/api/http/operations/scout_examples_scout_examples_get/) | Public | Example questions for Scout. | | GET | [`/scout/status`](/api/http/operations/scout_status_scout_status_get/) | Public | The budget guard's status. | | POST | [`/scout/judge`](/api/http/operations/scout_judge_scout_judge_post/) | Public | Unlock judge access with a code. | | DELETE | [`/scout/judge`](/api/http/operations/scout_judge_end_scout_judge_delete/) | Public | End judge access. | | GET | [`/scout/stream`](/api/http/operations/scout_stream_scout_stream_get/) | Public | Run Scout and stream its events. | | POST | [`/scout/run`](/api/http/operations/scout_run_scout_run_post/) | Public | Run Scout and return the result. | ### Pages | Method | Path | Who | What it is for | |---|---|---|---| | GET | [`/`](/api/http/operations/landing__get/) | Public | The story page. | | GET | [`/dashboard`](/api/http/operations/dashboard_dashboard_get/) | Public | The dashboard page. | | GET | [`/scout`](/api/http/operations/scout_page_scout_get/) | Public | The Scout page. | | GET | [`/judges`](/api/http/operations/judges_page_judges_get/) | Public | The judges' console (React app only; without the build, a placeholder page with links to the others). | | GET | [`/lite`](/api/http/operations/lite_landing_lite_get/) | Public | The story page without a build step. | | GET | [`/lite/dashboard`](/api/http/operations/lite_dashboard_lite_dashboard_get/) | Public | The dashboard page without a build step. | | GET | [`/lite/scout`](/api/http/operations/lite_scout_lite_scout_get/) | Public | The Scout page without a build step. | Routes the schema leaves out (listed in `docs-site/generated/routes.json`): `/docs`, `/docs/oauth2-redirect`, `/redoc` and `/openapi.json` (off in live mode unless `RATCHET_OPENAPI` is set), `/favicon.svg`, and the static files under `/assets/`. ## Three examples The examples run against a gateway in mock mode, with no keys and no network: ```bash RATCHET_MODE=mock uvicorn ratchet.gateway:app --port 8000 ``` In live mode, add `Authorization: Bearer $KEY` with one of your API keys. ### 1. A chat completion with a task `X-Ratchet-Task` says which task the request belongs to. Without it the request uses the `default` task (see [task selection](/reference/api/#task-resolution) for the other ways to name one). ```bash curl -s localhost:8000/v1/chat/completions \ -H "Content-Type: application/json" \ -H "X-Ratchet-Task: ticket-triage" \ -d '{"model": "ratchet", "messages": [{"role": "user", "content": "Where is my refund for order 1234?"}]}' ``` The same call with the OpenAI Python SDK: ```python client = openai.OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed-in-mock-mode") msgs = [{"role": "user", "content": "Where is my refund for order 1234?"}] resp = client.chat.completions.create( model="ratchet", messages=msgs, extra_headers={"X-Ratchet-Task": "ticket-triage"}, ) print(resp.choices[0].message.content) ``` ### 2. Read `ratchet.request_id` and send feedback Every chat response carries a `ratchet` block next to the usual fields. `request_id` names the request; `served_by` says whether the student or the teacher answered. A verdict on a request goes back with that id. An `incorrect` verdict may also carry the right answer, which then becomes a human-labelled training example (when the task and your opt-outs allow it). ```python meta = resp.model_extra["ratchet"] print(meta["request_id"], meta["served_by"], meta["escalated"]) r = httpx.post( "http://localhost:8000/v1/ratchet/feedback", json={"request_id": meta["request_id"], "verdict": "correct"}, # add headers={"Authorization": ...} in live mode ) print(r.status_code, r.json()) ``` With curl, using the id from the response: ```bash curl -s localhost:8000/v1/ratchet/feedback \ -H "Content-Type: application/json" \ -d '{"request_id": "req_...", "verdict": "correct"}' ``` The answer is `{"object": "ratchet.feedback", "request_id": "req_...", "verdict": "correct", "updated": false, "correction": null}`. `updated` is true when an earlier verdict on the same request was replaced. A key can give verdicts on its own requests only; any other id is a 404. ### 3. The error envelope Errors on `/v1/*` use OpenAI's envelope, so the SDK raises its typed exceptions. Every response, errors included, carries an `x-request-id` header. ```python try: client.with_options(max_retries=0).chat.completions.create(model="ratchet", messages=msgs, temperature=3) except openai.BadRequestError as e: print(e.status_code, e.code, e.param, e.response.headers["x-request-id"]) # 400 invalid_value temperature req_... ``` The body of that response: ```json { "error": {"message": "Invalid value for 'temperature': expected a number from 0 to 2, but got 3 instead.", "type": "invalid_request_error", "param": "temperature", "code": "invalid_value"}, "detail": "Invalid value for 'temperature': expected a number from 0 to 2, but got 3 instead.", "request_id": "req_0b6f3a7c2d1e4f5a8b9c0d1e2f3a4b5c" } ``` `error` has OpenAI's four fields. `detail` repeats the message for older clients. Routes outside `/v1` answer `{"detail": ...}` and put the id in `x-request-id`.
```mermaid sequenceDiagram participant C as Your app participant G as Gateway participant Student as Student participant Teacher as Teacher participant Ledger as Ledger C->>G: POST /v1/chat/
completions alt the request breaks a rule G-->>C: 400 envelope,
x-request-id else the student is sure G->>Student: three samples Student-->>G: answers G-->>C: 200, request_id,
served_by student else the student is unsure G->>Teacher: escalate Teacher-->>G: answer G->>Ledger: store it, if valid G-->>C: 200, request_id,
served_by teacher end C->>G: POST feedback:
request_id, verdict G->>Ledger: record the verdict G-->>C: 200 ratchet.feedback Note over G,Ledger: a correction becomes
a human-labelled example ```
**Reading this diagram.** A request is checked first, and a broken rule is a 400 in OpenAI's envelope before anything is spent. Otherwise the student answers when it is sure, and the teacher answers and has its answer stored when it is not. Either way the response names the request in `ratchet.request_id`. Feedback comes later, as a separate call to `/v1/ratchet/feedback` that quotes that id. The verdict is recorded in the ledger. It never promotes or rolls back a version; it is reported as human precision.
## Where the reference pages come from The generated pages are not typed by hand, so they cannot drift from the code.
```mermaid flowchart TB code["gateway.py
app.openapi()"] --> script cfg["config.py
KNOWN_ENV"] --> script cli["each tool's
--help"] --> script mk["Makefile
targets"] --> script script["export-reference.py"] --> o[("openapi/
ratchet.json")] script --> e[("generated/
env.json")] script --> c[("generated/
cli.json")] o --> h["HTTP API
pages"] e --> ep["Environment
variables page"] c --> cp["Command
line page"] h --> site(["Docs site"]) ep --> site cp --> site hand["Endpoint map
and guides,
by hand"] --> site class script gate class o,e,c store class hand outline ```
**Reading this diagram.** The script at the top imports the gateway in mock mode, reads the settings registry, runs every command line tool with `--help` and reads the Makefile. It writes four committed data files: the OpenAPI schema, the environment variables, the command line tools and the routes the schema leaves out. The site turns those files into the HTTP reference, the environment variable page and the command line page. This page and the guides are written by hand and link to the generated pages. After you change a route, a setting or a command, run `python docs-site/scripts/export-reference.py` and commit the result.
## Next steps - [HTTP API guide](/reference/api/): every field, header and error code. - [Generated HTTP reference](/api/http/): the schema of every operation. - [Environment variables](/api/environment/) and [Command line](/api/cli/). - [Python API](/api/python/): call the cascade without HTTP. --- # Environment variables Source: https://docs.ratchet.dualmindlab.tech/api/environment/ Ratchet reads its settings from environment variables. This page lists all 244 rows of the configuration tables, grouped as in the [configuration guide](/reference/configuration/), which has the longer explanations. It is built from `docs/CONFIGURATION.md` and checked against the lists in [`ratchet/config.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/config.py) (`KNOWN_ENV`, `DEPLOY_ENV`, `MOCK_ONLY_ENV`, `TOOL_ENV`, `RESERVED_SETTINGS`) and `ReleasePolicy`'s variables, exported by `python docs-site/scripts/export-reference.py`. :::tip[Check before you start] Run `python -m ratchet.config check` after every change to the environment. It lists every problem at once and exits 0 (clean), 1 (warnings) or 2 (errors: the gateway would not start). An unknown `RATCHET_*` variable is a warning, so a typo is caught. See [Command line](/api/cli/#python--m-ratchetconfig). ::: ## What you need for each mode - **Mock mode** (the default, `RATCHET_MODE=mock`): nothing. No keys, no network, an in-memory ledger. - **Live mode** (`RATCHET_MODE=live`): `NEBIUS_API_KEY` (Token Factory), and `RATCHET_API_KEYS` (or `RATCHET_ALLOW_UNAUTHENTICATED=1` when a proxy such as the Caddy guard checks callers). Add `RATCHET_STUDENT_BASE` and `RATCHET_STUDENT_KEY` only while the GPU endpoint runs; without them the hosted fallback answers student calls. `TAVILY_API_KEY` gives Scout real web search. Keep the Scout budget guard on. - **Night School in live mode**: `RATCHET_SYNC_CMD`, `RATCHET_TRAIN_CMD` (or its default) and the deploy kit variables for the bucket, image and endpoint. The **Where** column says where a variable matters: | Where | Meaning | |---|---| | both | Read by the gateway in mock and live mode (some only change live behaviour; the meaning says so) | | mock only | Read only in mock mode; the startup check warns when one is set in live mode | | live | Read by live mode tools such as `python -m ratchet.live` | | deploy kit | Read only by `deploy/*.sh` and the VM, never by the package | | live-run tool | Read by the live-run harness (`python -m ratchet.liverun`) | | reserved | Declared for a feature not built yet: any value but the default stops startup | ## Gateway and cascade (ratchet/config.py) From [Gateway and cascade (ratchet/config.py)](/reference/configuration/#gateway-and-cascade-ratchetconfigpy) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_MODE` | `mock` | both | `mock`: offline test run with mock models and an in-memory ledger. `live`: real Token Factory teacher and a SQLite file | | `RATCHET_DB` | `ratchet.db` | both | SQLite ledger path (live mode only; mock mode always uses `:memory:`) | | `RATCHET_TEACHER_BASE` | `https://api.tokenfactory.nebius.com/v1` | both | OpenAI-compatible base URL for the teacher, judge and fallback student (and the fallback teacher, unless `RATCHET_TEACHER_FALLBACK_BASE` is set) | | `NEBIUS_API_KEY` | empty | both | Token Factory API key (sent as `Authorization: Bearer …`) | | `RATCHET_TEACHER_MODEL` | `nvidia/Nemotron-3-Ultra-550b-a55b` | both | Teacher model. Must be in the license registry and distillable, or startup fails with `LicenseError` | | `RATCHET_JUDGE_MODEL` | `nvidia/nemotron-3-super-120b-a12b` | both | Judge for free-text golden evaluation (`LiveEvaluator`): the answers the student would serve on free-text tasks' held-out golden rows, read by `RATCHET_MIN_JUDGE_PASS` ([free-text tasks](/reference/configuration/#free-text-tasks)) | | `RATCHET_FALLBACK_STUDENT` | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` | both | Hosted, untuned model used for student calls when `RATCHET_STUDENT_BASE` is empty and a trained version exists (on a fresh deployment the teacher answers). It answers under `RATCHET_CONFIDENCE_THRESHOLD`, and its answers count toward no version's release: a canary or shadow holds while it serves. Reported as `fallback_student` in `/ratchet/stats` and `/scout/status`; the pages label a student answer with this `model` "Nano 30B · hosted", never as our adapter | | `RATCHET_HOSTED_STUDENT` | `0` | both | `1`: in live mode, while no version is live, the hosted model above answers each task `ratchet/hosted.py` calibrated it for, at that task's measured threshold, and the teacher answers the rest. The gateway keeps one version row in state `hosted` for it (no adapter, never listed in `/ratchet/stats` `versions`); its answers are audited as drawn, never stored and counted by no release. Startup fails unless `RATCHET_FALLBACK_STUDENT`, `RATCHET_SAMPLES` and `RATCHET_FALLBACK_STUDENT_REASONING` are what the records were measured with (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B`, `3`, `off`). Ignored, with a warning, in mock mode. As measured on Oct 26, 2026 no task met the bar (95% held-out precision against Ultra on 20 served rows), so it changes no answer yet | | `RATCHET_DEMO_SPLIT` | `0` | both | A share from 0 to 1 (0 = off): in live mode with `RATCHET_HOSTED_STUDENT=1`, each request that reaches the cascade (never a pinned task, a tool request, a teacher-only request, `n` > 1, a task with critical classes) goes to the hosted Nano with this probability whatever its confidence, to show the economics on the live gateway. Such answers have route `demo_split` and are labelled "Random split" on every page; the draw is its own secure generator, never the router's. Another value, mock mode or no `RATCHET_HOSTED_STUDENT=1` stops startup | | `RATCHET_DEMO_SPLIT_AUDIT` | `0.25` | both | The share of forced (`demo_split`) answers Ultra audits, from 0 to 1, so the dashboard shows how often the Nano agreed. Only Ultra's answer is stored | | `RATCHET_STUDENT_BASE` | empty | both | Base URL of your vLLM endpoint, e.g. `https:///v1` | | `RATCHET_STUDENT_KEY` | empty | both | Token for the student endpoint | | `RATCHET_STUDENT_BASE_MODEL` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | both | Base model that adapters are trained on (also used by the deploy scripts) | | `RATCHET_CONFIDENCE_THRESHOLD` | `0.80` | both | Fallback threshold, used only when a version has no calibrated `threshold` in its eval (and for the hosted, untuned fallback student) | | `RATCHET_SAMPLES` | `3` | both | Student samples per request (self-consistency) and per golden row, for a task whose spec sets no `policy.samples` (a task's own count wins). A request for more choices (`n`) draws `n` samples, but its confidence is still scored on the first `RATCHET_SAMPLES` (or the task's count) of them | | `RATCHET_AUDIT_RATE` | `0.05` | both | Probability that a student-served answer is also sent to the teacher. Mock mode raises it to at least `0.10`. A task's `policy.audit_rate` replaces it for that task ([risk-aware routing](/reference/configuration/#risk-aware-routing)), and a canary's rate below replaces both. An admin can replace them all at runtime, without a restart (`PUT /ratchet/admin/settings` or `python -m ratchet.admin flag audit_rate 0.2`; [API.md](/reference/api/#admin)); `controls.audit_rate` in `/ratchet/stats` shows the override, `null` while this value applies | | `RATCHET_CANARY_AUDIT_RATE` | unset | both | Audit focusing: the audit probability for answers a **canary** serves, so its release evidence arrives sooner; live answers keep `RATCHET_AUDIT_RATE`. More than 0 and at most 1 (0 would leave a canary without audits, so it could never be promoted, rolled back or retired); anything else stops startup. Unset or empty: the canary is audited at `RATCHET_AUDIT_RATE`. Still one random draw per served answer; the rate used is logged as `requests.label_p` | | `RATCHET_GOLDEN_FRACTION` | `0.10` | both | Share of new examples assigned to the golden (held-out) split | | `RATCHET_MIN_TASK_GOLDEN` | `20` | both | Calibration rows (the even half of a task's golden rows) a task needs before its own threshold may go below the version's global one ([per-task calibration](/reference/configuration/#per-task-calibration)). With fewer it gets the higher of the two and `calibrated: false`. A whole number, 1 or more; anything else stops startup (gateway, simulator, `python -m ratchet.live`) | | `RATCHET_PER_TASK_CALIBRATION` | `1` | both | `0` switches [per-task calibration](/reference/configuration/#per-task-calibration) off: Night School writes no per-task records, so each new version keeps one pooled threshold and the golden gate checks only the pooled numbers, as in v0.4; only a task with critical classes still gets its record (its own threshold, checked by the per-task gate and the [critical gate](/reference/configuration/#risk-aware-routing)), and the eval's `other_tasks` gives the other tasks' held-out rows and how many the pooled threshold serves, so disabling the critical task leaves the version the coverage of its other tasks. Versions already scored with records keep them until a night re-scores them. Any other value leaves it on. The reference workloads (`make reference`) run with it off, so they reproduce v0.4.0 exactly | | `RATCHET_TEACHER_REASONING_TOKENS` | `300` | both | Extra output tokens assumed for the teacher in baseline estimates (and added by the mock teacher) | ## Request limits and OpenAI compatibility From [Request limits and OpenAI compatibility](/reference/configuration/#request-limits-and-openai-compatibility) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_MAX_BODY_BYTES` | `2097152` | both | Largest request body, in bytes (2 MiB), on every `POST`, `PUT` and `PATCH`. A larger body gets **413** `request_too_large`: at once when `Content-Length` says so, or as soon as a chunked body passes the limit. `0` = no limit | | `RATCHET_MAX_MESSAGES` | `256` | both | Most messages in one `/v1/chat/completions` request; more is a 400 with `param: messages`. `0` = no limit | | `RATCHET_MAX_INPUT_CHARS` | `120000` | both | Most characters of message text in one request: content strings, text and refusal parts, `name`, `refusal`, `tool_call_id`, and tool call ids, function names and arguments, added up (everything forwarded to the teacher except non-text parts, which only `RATCHET_MAX_BODY_BYTES` bounds); more is a 400 with code `context_length_exceeded`. About 30,000 tokens of English. `0` = no limit | | `RATCHET_MAX_N` | `4` | both | Most choices (`n`) one request may ask for, 1 to 128; more is a 400 `unsupported_value` with `param: n` ([API.md](/reference/api/#choices-and-logprobs)). Each choice past `RATCHET_SAMPLES` is one more student sample on every such request, and an escalated one asks the teacher for `n` choices, which it bills as `n` answers. `1` restores the v0.4 behaviour of one choice per request. A value outside 1 to 128 stops the gateway at startup with `ValueError` | | `RATCHET_MERGE_SYSTEM` | `0` | both | `1`: merge consecutive leading `system` messages (after `developer` becomes `system`) into one, joined by a blank line, for models whose chat template takes a single system message. It changes the stored form of such requests, so set it before collecting training data, not halfway | | `RATCHET_TEACHER_MODALITIES` | `text` | both | Content part types the teacher accepts besides text, comma-separated, e.g. `text,image_url` (also `input_audio`, `file`). A request with such a part goes to the teacher only (route `ineligible:content_type`, [risk-aware routing](/reference/configuration/#risk-aware-routing)): no student call, no audit, nothing stored as a training example. Any other non-text part is a 400 `unsupported_content_type`. Set it only when the teacher model reads those parts; otherwise the teacher's refusal comes back as a 400 with its own message ([API.md](/reference/api/#errors)) | ## Request limits and OpenAI compatibility: Other OpenAI endpoints From [Request limits and OpenAI compatibility: Other OpenAI endpoints](/reference/configuration/#other-openai-endpoints) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_EMBEDDINGS_BASE` | (empty) | both | OpenAI-compatible base URL that `/v1/embeddings` calls. Empty: `RATCHET_TEACHER_BASE` (Token Factory). `off` (any case): the route answers 404 `unsupported_endpoint` and `/v1/models` does not list the embeddings model. Anything that is not an `http(s)` URL stops startup. Mock mode embeds locally unless it is `off` | | `RATCHET_EMBEDDINGS_MODEL` | `Qwen/Qwen3-Embedding-8B` | both | The one model `/v1/embeddings` asks for, whatever `model` a client sends; listed in `/v1/models`, and the id `RATCHET_MODEL_POLICY=strict` accepts there besides `ratchet`. **Unverified until `scripts/day1_check.py` passes**, which checks it. Priced from `EMBEDDING_PRICES` ([Prices](/reference/configuration/#prices-default_prices)); a model without a price there is priced as the teacher. An empty id, one with spaces around it, a Ratchet model id or one that is also a model alias stops startup | | `RATCHET_EMBEDDINGS_KEY` | (empty) | both | API key for `RATCHET_EMBEDDINGS_BASE`. Empty: `NEBIUS_API_KEY` when the base is the teacher's (`RATCHET_TEACHER_BASE`), and no key at all for any other base, so the Token Factory key is never sent to another provider. Set it for a provider that needs one; a local server without auth needs none | ## Retries and idempotency From [Retries and idempotency](/reference/configuration/#retries-and-idempotency) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_IDEMPOTENCY_CACHE` | `10000` | both | Most answers kept by `Idempotency-Key`, 0 to 1,000,000; the least recently used goes first, and the answers kept never exceed 64 MiB together, each counted as its body plus 2 KiB for the entry that holds it (so at most about 32,000 small answers, whatever this says). `0` turns idempotency off: the header is ignored (not even checked) and every request runs | | `RATCHET_IDEMPOTENCY_TTL_S` | `600` | both | Seconds an answer is kept after it finished, 0 to 86,400; a request with the same key after that runs again. `0` keeps none: only a request that arrives while the first is still running shares its answer. Answers hold request-derived text, so they are never written to disk | | `RATCHET_SEEN_DEDUP_S` | unset: `300` in live mode, `0` in mock mode | both | Seconds within which the same API client's repeat of a stored example counts once toward the frequent-request rule (`seen`, `FREQUENT` = 3), 0 to 86,400; a repeat after the window counts again. Without API keys every caller is one client. Up to 10,000 (example, client) pairs are remembered, the least recently counted forgotten first. A request whose `x-stainless-retry-count` is above 0 never counts, whatever this says. `0`: every repeat counts, as in v0.4; mock mode's default, because the simulator's and the live gateway's repeats are traffic, not retries (internal calls, such as Scout's, the simulator's and the warm-up's, count every repeat in any mode). The VM's [`.env.vm.example`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/.env.vm.example) sets `300`, so the public mock-mode gateway counts a visitor's quick repeats once, as live mode does | ## Generation parameters and token caps From [Generation parameters and token caps](/reference/configuration/#generation-parameters-and-token-caps) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_TEACHER_PARAMS` | empty (every standard field) | both | Request fields the teacher gets when its answer is served (escalation, cold start, pinned, teacher-only), comma-separated, in any case: `max_tokens` (also for `max_completion_tokens`), `temperature`, `top_p`, `stop`, `seed`, `presence_penalty`, `frequency_penalty`, `logit_bias`, `response_format`, `tools`, `tool_choice`, `parallel_tool_calls`, `logprobs`, `top_logprobs`, `reasoning_effort`. Empty means the default: every standard field. `tools`, `tool_choice` and `parallel_tool_calls` are in it since v0.5: only a tool request has them, the teacher answers it alone and its tool calls come back to the client ([API.md](/reference/api/#tools-and-function-calling)); leave them out (name the other fields) and the teacher answers tool requests in prose, still teacher-only and never stored. `logprobs` and `top_logprobs` reach the teacher only when a client sends them, and its token logprobs come back in the response (v0.5); leave them out if the teacher refuses `logprobs` (`scripts/day1_check.py` reports whether it returns them), and teacher-served choices then carry `logprobs.content: null`. `none` forwards nothing; `all` stands for the whole list, and `all,top_k` adds a vendor field the client sends (otherwise unknown fields are never forwarded). Startup stops on `model`, `messages`, `n`, `stream`, `stream_options`, `user`, `safety_identifier`, `store`, `metadata`, `service_tier`, `functions` or `function_call`, on a name that is not an identifier, and on a near miss of a standard field (`max_token`, `stops`), which is a typo; any other name is taken as a vendor field. Audits and arbitrations take the same fields, minus `top_p`, `seed`, the penalties, `logit_bias` and logprobs, at temperature 0. A served answer written under a client's `logit_bias`, a non-zero penalty, a temperature above 1 or a vendor field is not stored | | `RATCHET_STUDENT_PARAMS` | empty (`max_tokens,stop,seed,response_format,top_logprobs`) | both | Request fields the student gets, same syntax. Its `n` (`RATCHET_SAMPLES`), temperature (0.7) and logprobs are fixed by the cascade whatever this says, so its confidence stays calibrated. `logit_bias` is not in the default on purpose: a client's bias would steer the student's answer while the audit runs without it, and OpenAI token ids mean other tokens to Nemotron; name it here to forward it. A request with tools never reaches the student (the teacher answers it alone), so `tools` and `tool_choice` here only matter for a golden row stored with them; add them only when the student's vLLM runs with `--enable-auto-tool-choice` and a `--tool-call-parser` (`deploy/student_endpoint.sh` does not), since without them vLLM answers 400. A golden row the student refuses this way scores as not served in the nightly eval | | `RATCHET_TEACHER_MAX_TOKENS` | `8192` | both | Most output tokens per teacher call (reasoning included, as Token Factory counts them). A client's larger `max_tokens` is capped to this; with none, this is sent. `0` = no cap | | `RATCHET_STUDENT_MAX_TOKENS` | `1024` | both | Most output tokens per student sample, the same way. `0` = no cap | | `RATCHET_TEACHER_REASONING_HEADROOM` | `0` | both | Tokens added to a client's `max_tokens` for the teacher (still at most `RATCHET_TEACHER_MAX_TOKENS`), for reasoning it writes before the answer: without it a reasoning teacher can spend a tight limit on its reasoning and be cut off. When the teacher gets more than the client asked for, a visible answer longer than the client's `max_tokens` (about 4 characters a token) is returned with `finish_reason: "length"` and not stored. Set it near `RATCHET_TEACHER_REASONING_TOKENS` for a reasoning teacher whose clients send small limits | ## Reasoning (Nemotron thinking) From [Reasoning (Nemotron thinking)](/reference/configuration/#reasoning-nemotron-thinking) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_STUDENT_REASONING` | `off` | both | The trained student on the vLLM endpoint (production, canary, shadow, both golden evals): `off` sends `{"enable_thinking": false}`, `default` nothing. `on` and `medium` stop startup: the student is trained with thinking off | | `RATCHET_FALLBACK_STUDENT_REASONING` | `off` | both | The hosted untuned student (`RATCHET_FALLBACK_STUDENT`) used while `RATCHET_STUDENT_BASE` is empty: `off`, `on` or `default`. Set `default` if Token Factory refuses the field (the student then fails fast and requests escalate, route `student_error`) | | `RATCHET_TEACHER_REASONING` | `default` | both | The teacher whose answer is served (escalation, cold start, pinned, teacher-only) and the fallback teacher: `default`, `off`, `on` or `medium` (Ultra's `{"enable_thinking": true, "medium_effort": true}`). Keep `default` until day 1 shows Token Factory accepts the field. With `off`, `RATCHET_TEACHER_REASONING_HEADROOM` is not needed | | `RATCHET_CHECK_REASONING` | empty (follows `RATCHET_TEACHER_REASONING`) | both | Audits and arbitrations, same values | | `RATCHET_TEACHER_REASONING_FIELD` | `chat_template_kwargs` | both | Where Token Factory-bound calls (teacher, fallback teacher, hosted fallback student) carry the kwargs: top level, or `extra_body` (nested as `extra_body.chat_template_kwargs`). **Unverified until day 1** (`scripts/day1_check.py`). The vLLM endpoint always gets the top-level field | ## Upstream timeouts and retries From [Upstream timeouts and retries](/reference/configuration/#upstream-timeouts-and-retries) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_TEACHER_TIMEOUT_S` | `120` | both | Read timeout for teacher and judge calls on Token Factory, in seconds: how long the model may take to start answering | | `RATCHET_TEACHER_RETRIES` | `2` | both | Extra attempts for a failed teacher or judge call (`0` = no retries). Retried: connection failures and HTTP 408, 409, 429, 500, 502, 503, 504, with full-jitter backoff (0.5 s base, 8 s cap) or the server's `Retry-After`. A read timeout, a connection dropped mid-response or a 504 is retried once at most in all, because the model may already have generated billed tokens; any other 4xx, and a 2xx body that cannot be read, never. A server wait over 8 s is not waited for: the 429 goes back to the client with it | | `RATCHET_TEACHER_DEADLINE_S` | `180` | both | Upper bound on one teacher call, attempts and waits together, in seconds. A retry whose wait would end past it is not started, and later attempts get only the time that is left as their timeout. A `/v1/completions` batch starts no further prompt once it has run this long (504 `batch_deadline_exceeded`, [API.md](/reference/api/#post-v1completions)) | | `RATCHET_STUDENT_TIMEOUT_S` | `15` | both | Read timeout for student calls (the vLLM endpoint, or the hosted fallback when `RATCHET_STUDENT_BASE` is empty), in seconds. On the request path a student call is never retried: a failure escalates to the teacher at once. Night School's golden eval retries it like the teacher, since no teacher stands behind it there | | `RATCHET_STUDENT_CONNECT_TIMEOUT_S` | `2` | both | Connect timeout for student calls, in seconds, so an endpoint that is down costs a request two seconds rather than the teacher's five | | `RATCHET_NIGHT_STUDENT_TIMEOUT_S` | `120` | both | Read timeout for the student's calls in `python -m ratchet.live night` (the golden eval), in seconds. The first call after an adapter hot-load into a cold LoRA endpoint can take far longer than a warm request-path call: the 28 Oct 2026 night timed out at 15 s. The request path keeps `RATCHET_STUDENT_TIMEOUT_S`. The night does not start or wake the student endpoint: it must be running before the hot-load | | `RATCHET_STUDENT_BREAKER_FAILURES` | `5` | both | Student circuit breaker: endpoint failures in a row (a timeout, a connection error, an unreadable or 5xx answer, 408, 429; another 4xx is the endpoint answering and resets the count) after which student calls are skipped and requests go straight to the teacher (route `student_breaker`), so they stop waiting on a dead endpoint. Live and canary on one endpoint share a breaker; shadow calls have their own. `0` = no breaker ([ARCHITECTURE.md](/reference/architecture/#student-circuit-breaker)) | | `RATCHET_STUDENT_BREAKER_COOLDOWN_S` | `30` | both | Seconds an open student breaker waits before one request probes the endpoint again. A failed probe doubles the wait, up to 300 s (or this value, if larger); a successful probe closes the breaker and resets it. It may be shorter than the student's timeouts: another request probes only once the probe has been out for the longer of the wait and the student call's longest time (connect, write, read and pool timeouts plus 5 s: 2 + 10 + `RATCHET_STUDENT_TIMEOUT_S` + 5 + 5 = 37 s by default), and a probe that fails after that still reopens the breaker and doubles the wait | ## When the teacher is down From [When the teacher is down](/reference/configuration/#when-the-teacher-is-down) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_TEACHER_BREAKER_FAILURES` | `5` | both | Teacher circuit breaker: failed teacher calls in a row, each counted after its retries (a timeout, a connection error, an unreadable or 5xx answer, 408, 429; a 400 such as `context_length_exceeded`, a 401 or a tool-call answer is the teacher answering and resets the count), after which the teacher is not called. While it is open, a request that needs the teacher's answer gets **503** `teacher_circuit_open` at once, with the time until the next probe as `Retry-After` (1 to 60 s), and audits and arbitrations are skipped. The fallback teacher has a breaker of its own with the same settings. `0` = no breaker. A negative or non-integer value stops the gateway at startup | | `RATCHET_TEACHER_BREAKER_COOLDOWN_S` | `30` | both | Seconds an open teacher breaker waits before one request probes the teacher again. A failed probe doubles the wait, up to 300 s (or this value, if larger); a successful one closes the breaker. Must be a positive number | | `RATCHET_ON_TEACHER_FAILURE` | `error` | both | What an escalated request gets when the teacher (and the fallback teacher, if set) is unavailable. `error`: the mapped error (502, 503 or 504). `serve_student`: the student's own answer, below its threshold, when it gave at least one valid sample, with route `degraded`, `ratchet.degraded: "teacher_unavailable"` and `X-Ratchet-Degraded: teacher_unavailable`; never stored, audited or shadowed, so it never counts toward a release. Never for a 400, 413 or 422 about the request, a tool-call answer, a request with `tools`, a pinned or teacher-only request, a cold start, or a request whose student call failed. Case and spaces are ignored, empty means `error`, anything else stops the gateway at startup | | `RATCHET_TEACHER_FALLBACK_MODEL` | empty | both | A second teacher that answers a call whose answer is served (escalation, cold start, pinned, teacher-only) when the teacher is unavailable: its retries failed or its breaker is open. Never for audits or arbitrations, a 400 about the request or a tool-call answer. Must be in the license registry and distillable, or startup fails with `LicenseError` (invariant 1). Its answers are served with route `fallback_teacher` and its own model id, priced with its own price (`Settings.price`). When it fails too, the client gets the teacher's error. Empty = none | | `RATCHET_TEACHER_FALLBACK_BASE` | empty (`RATCHET_TEACHER_BASE`) | both | OpenAI-compatible base URL for the fallback teacher. It always uses `NEBIUS_API_KEY` and the teacher's timeouts and retries. Empty or equal to `RATCHET_TEACHER_BASE`: the teacher's client is used | | `RATCHET_STORE_FALLBACK_TEACHER` | `0` | both | `1`: store the fallback teacher's answers as training examples (under the usual rules), each with the fallback model and its own license in its provenance and the BOM. `0`: serve them, store nothing, so golden labels stay the teacher's | ## Privacy and security From [Privacy and security](/reference/configuration/#privacy-and-security) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_REDACT_PII` | `1` | both | Replace emails, phone numbers, card numbers (Luhn-checked), IBANs (mod-97), Aadhaar (Verhoeff), US SSNs, Indian PANs and GSTINs, UPI ids, IPv4 addresses and API secrets with placeholders before a request and the teacher's answer are stored as a training example. The teacher still receives the original. `0` stores examples verbatim. Names and street addresses are found only by a detector plugin (`RATCHET_REDACT_PLUGIN`) | | `RATCHET_REDACT_STYLE` | `label` | both | How identifiers are written in stored examples: `label` (`[EMAIL]`, v0.4's output), `numbered` (`[EMAIL_1]`, `[EMAIL_2]`: the same value gets the same number in the prompt, the stored conditions and the answer, so an extraction task's answer still says which of two addresses it names) or `surrogate` (reserved values of the same shape: `person1@example.com`, `+12025550100`, a 16-digit card number that fails Luhn, a `ZZ` IBAN, `192.0.2.1`; the numbered placeholder for other identifiers). A task spec's `privacy.redact_style` overrides it for that task ([Tasks](/reference/configuration/#task-options)). Numbers start again in every example, and nothing original is kept. Changing it changes the stored text, so a request stored before and after is two examples (old rows are never rewritten); an erased example stays erased (each row stored otherwise than in `label` records its `label` key, which an erasure tombstones too). Case and spaces are ignored; empty means `label`; anything else stops startup | | `RATCHET_REDACT_PLUGIN` | empty | both | A detector for what the patterns cannot find, such as names and addresses: `module:function` (a dotted attribute path after the colon), importable by the gateway. The function takes a text and returns `(start, end, label)` spans (`0 <= start < end <= len(text)`, a label of 1 to 32 letters, digits and `_`); they are merged with the patterns' and written in the active style. Imported and called once on an empty text at startup; a missing module or attribute, or a detector that fails there, stops startup. It runs whenever an example is stored, on the request's thread, so its latency adds to escalated and audited requests, and it must be thread-safe. When it fails at store time the answer is served and nothing is stored (`skip_example` `redaction failed`). `POST /ratchet/redact` runs it only for callers with a valid API or admin key. a Presidio detector is a typical plug-in; the package bundles no NER model | | `RATCHET_STORE_EXAMPLES` | `1` | both | `0` is zero retention: every request is answered, audited and logged as usual, but no teacher answer is stored as a training example, and Night School skips every night (`storage disabled`), so examples kept from before are not trained on. A task whose spec changed is still re-scored on the stored golden rows when no release is in flight (that writes no example), so it does not stay on the teacher for good. Requests are logged with `store_opt_out` `global`. Accepts `1`/`0`, `true`/`false`, `yes`/`no`, `on`/`off` in any case; anything else stops startup, so a typo cannot mean "store" | | `RATCHET_NO_STORE_TASKS` | empty | both | Tasks whose teacher answers are never stored, comma-separated (`invoice-extract,hr-notes`): the same as `store: false` in their specs ([Tasks](/reference/configuration/#tasks)). Requests are logged with `store_opt_out` `task`. A name need not be registered, but one that is not a task name (a space-separated list) stops startup. Examples stored before stay | | `RATCHET_FEEDBACK_CORRECTIONS` | `1` | both | Whether a correction sent with an `incorrect` verdict (`POST /v1/ratchet/feedback`) may be stored as a human-labelled training example. `0` keeps every verdict (task agreement, human precision, the `human_alert` event) and stores no correction (`not_stored: corrections_off`), for deployments whose API clients should not write labels. The storage opt-outs above and pinned tasks apply either way. Accepts `1`/`0`, `true`/`false`, `yes`/`no`, `on`/`off` | | `RATCHET_END_USER_KEY` | empty | both | The HMAC key end users are hashed with, at least 16 characters (`openssl rand -hex 32`). The request's `safety_identifier`, else its `user`, is stored only as HMAC-SHA256 under this key, scoped to the API key name, cut to 16 hex digits. Empty: 32 random bytes are generated on first start and kept in the ledger's `meta` table, so hashes survive restarts, but anyone with a copy of the database file then has the key too; set this from a secrets manager to keep it out of the file and its backups. Changing it unlinks every earlier hash: no-train registrations stop matching (an `end_user_key` event says how many), so those end users must register again. A shorter value stops startup | | `RATCHET_EXAMPLE_RETENTION_DAYS` | `0` | both | Whole days a stored example is kept after it was last stored or seen again; `0` keeps it forever. Past the window its messages, answer and conditions are removed (golden examples too) and the row stays as a tombstone with its hash, so Night School's counts and an erasure still see it; a later duplicate that may be stored brings the text back with its own teacher and license. Applied hourly by the live gateway, at the start of every live night and by `python -m ratchet.live purge`. 0 to 36500; anything else stops startup | | `RATCHET_REQUEST_RETENTION_DAYS` | `0` | both | Whole days a request row (metadata only) is kept; `0` keeps it forever. Older rows are folded into the `daily_rollup` table (per day, task, client, route and opt-out: the sums every stats reader and the cost report add back; no end user) and deleted, except rows logged since a release stage in flight began and the rows a version in shadow or canary served or shadowed in an earlier stage, and the rows of a day whose savings receipt is not sealed yet (a day that ended less than 5 minutes ago keeps every row until the next purge seals it). Distinct end users per client then count only the rows still kept. Use at least 1, so today's rows (the daily quota counts them) stay. 0 to 36500; anything else stops startup | | `RATCHET_ERASURE_SHARED` | `delete` | both | What erasing a client or an end user does with an example another client also sent: `delete` purges it too (the text is the same, so the erased party's data was in it); `keep` keeps it for the other client and unlinks the erased party (the other client becomes its first sender, without an end user). Erasure by hashes or by text purges whatever it selects. Anything else stops startup | | `RATCHET_API_KEYS` | empty | both | `name:key,name:key`. When set, `/v1/*` and `/openai/*` need `Authorization: Bearer `, `X-API-Key: ` or `api-key: `; anything else gets 401. Names: letters, digits, `_`, `.`, `-` (up to 64). Keys: at least 16 characters; generate with `openssl rand -hex 24`. A bare key without `name:` is named `keyN`. The ledger stores the name per request, never the key. End-user hashes and no-train registrations are scoped to the name ([API.md](/reference/api/#end-users-and-storage)): rotate a key under the same name; renaming one unlinks its end users' earlier hashes, and its clients' no-train registrations match nothing until the name is back (a `no_train_unmatched` event at startup; an admin's registration for every client is not affected). A key also reads its own savings receipts and their proofs, never another client's ([API.md](/reference/api/#savings-receipts)). A malformed value stops the gateway at startup | | `RATCHET_API_KEY_DAILY_REQUESTS` | `0` | both | Requests per key per UTC day (chat completions, responses, embeddings, and completions at one per prompt); past it a 429 `insufficient_quota` (code `daily_quota_exceeded`) with `Retry-After` until 00:00 UTC and `X-Should-Retry: false` ([API.md](/reference/api/#authentication)). `0` = unlimited. `RATCHET_KEY_LIMITS` overrides it per key ([Spend and rate limits](/reference/configuration/#spend-and-rate-limits)). Counted by the spend meter, which reads today's requests from the ledger at startup, so it survives restarts in live mode | | `RATCHET_ADMIN_KEYS` | empty | both | Admin keys for management calls (`PUT` and `DELETE /ratchet/tasks/{name}`, `POST /ratchet/tasks/validate`, [API.md](/reference/api/#tasks); `POST /ratchet/cost/endpoint`; every route of the admin API under `/ratchet/admin/`, reads included, [API.md](/reference/api/#admin); and the training-set proofs, `POST /ratchet/versions/{id}/proof` and `GET /ratchet/versions/{id}/heldout`, [API.md](/reference/api/#committed-training-sets), which are a membership oracle for the training data; and every savings receipt, the gateway's and each client's, [API.md](/reference/api/#savings-receipts)), in the `RATCHET_API_KEYS` syntax: `name:key,name:key`, keys at least 16 characters. Sent as `Authorization: Bearer ` or `X-API-Key`; a missing or wrong key gets 401, an API key 403. The ledger and the event log record the name, never the key. A key may not be in both variables (startup error). Empty: the admin API is off in live mode (403 `admin API disabled: set RATCHET_ADMIN_KEYS`) and open in mock mode (the admin is `mock-admin`). Not the Caddy guard's `RATCHET_ADMIN_TOKEN` | | `RATCHET_DASHBOARD_KEYS` | empty | both | Read-only keys for the dashboard API ([API.md](/reference/api/#access-levels)), in the `RATCHET_API_KEYS` syntax. A dashboard key reads everything an admin key reads and writes nothing. A key may not also be an API key or an admin key (startup error), so each key has one role. Give one to whoever watches a live gateway's dashboard | | `RATCHET_PUBLIC_DASHBOARD` | `1` in mock mode, `0` in live mode | both | `1`: the dashboard API (`GET /ratchet/stats`, `GET /ratchet/cost`, `POST /ratchet/redact`, `GET /ratchet/tasks` and `/ratchet/tasks/{name}`, the BOM endpoints and the other reads [API.md](/reference/api/#access-levels) lists) needs no key. `0`: it needs an admin or dashboard key, as `Authorization: Bearer ` or `X-API-Key`; without one it is 401, with an API key 403, and with neither `RATCHET_ADMIN_KEYS` nor `RATCHET_DASHBOARD_KEYS` set every read is 403. The pages, `/healthz`, `/ratchet/bom/public-key` and Scout are always public. The dashboard page sends no key yet, so with `0` it shows no data: read the API with a key, or set `1` (the VM does) | | `RATCHET_PUBLIC_BOM` | `0` | both | `1`: `GET /ratchet/bom` and `GET /ratchet/versions/{id}/bom` stay public while the dashboard API is private, so auditors can check an adapter's provenance without a key | | `RATCHET_SCRUB_STATS` | unset | both | What a reader of the dashboard API sees. Unset: an anonymous read of a public dashboard in live mode is scrubbed: client names in `/ratchet/stats` and in task allow-lists (`GET /ratchet/tasks`) become `client-1`, `client-2`, ... (by sorted name, the same in both), event details lose every `http(s)://` URL (``), and an error message after an exception's class is cut (`ConnectError: ...` becomes `ConnectError`; v0.4 ledgers hold such events). A reader with an admin or dashboard key sees everything. `1`: every read is scrubbed, with a key or without, in either mode. `0`: none is. `security.scrubbed` in `/ratchet/stats` says which a reader got | | `RATCHET_OPENAPI` | `1` in mock mode, `0` in live mode | both | `1`: FastAPI serves `/docs`, `/redoc` and `/openapi.json`. `0`: they do not exist (404), so a live gateway does not hand anyone a map of its routes | | `RATCHET_ALLOW_UNAUTHENTICATED` | `0` | both | Live mode refuses to start without `RATCHET_API_KEYS`, because `/v1/*` would answer anyone, on your teacher credits. `1` lets it start for a gateway something else guards, but only with the guard configured: `RATCHET_API_GUARD=guarded` and a `RATCHET_ADMIN_TOKEN` of at least 32 characters (the VM's Caddy checks both), so one variable alone never opens the paid API. Otherwise startup and `python -m ratchet.config check` refuse, naming both variables and never the token. Mock mode ignores it | | `RATCHET_CSP` | empty | both | A `Content-Security-Policy` for every response. Empty sends none, so no page can break by default. Printable ASCII only (anything else stops startup). Recommended, which the React build and the lite pages work with: `default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; font-src 'self' data:; connect-src 'self'; object-src 'none'; base-uri 'none'; frame-ancestors 'none'; form-action 'self'` (the pages load nothing from other origins; `'unsafe-inline'` is for the theme script in the React page and the lite pages' inline scripts). It also blocks `/docs`, whose Swagger UI comes from a CDN | | `RATCHET_BOM_SIGNING_KEY` | empty | both | Ed25519 private key as the base64 of its 32-byte seed. Signs each adapter's bill of materials and every savings receipt (same key, same public key). Generate with `python -m ratchet.bom keygen`, which also prints the public key to publish. Needs the `sign` extra (`pip install "ratchet-distill[sign]"`, included in the gateway image). An invalid value stops the gateway at startup. Empty = BOMs and receipts are stored with a SHA-256 but unsigned | ## Spend and rate limits From [Spend and rate limits](/reference/configuration/#spend-and-rate-limits) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_KEY_LIMITS` | empty | both | Per-key overrides of the defaults below, by key name: `acme:daily_requests=10000,daily_usd=5,monthly_usd=100,rpm=60,concurrency=4,tasks=ticket-triage\|invoice-extract,store=1;globex:daily_usd=1`. Entries are separated by `;`, fields by `,`. `daily_requests`, `rpm` and `concurrency` are whole numbers, `daily_usd` and `monthly_usd` plain decimals (US dollars, at most 1,000,000,000), `0` turns a limit off for that key. `tasks` lists the task names the key may send, separated by `\|` (any other task, `default` included, is a 403 `task_not_allowed`; without `tasks` every task its own `clients` allow). `store=0` keeps the key's requests out of the training set: they are served, audited and logged, never stored (`store=1`, the default, stores as usual). A name that is not in `RATCHET_API_KEYS`, an unknown field, a field or a key given twice, or a bad value stops startup; the message lists the key names and never prints what may be a key pasted by mistake (`acme:`, in the `RATCHET_API_KEYS` syntax): an unknown name is not repeated, and an item, field or value is quoted only when it is shorter than any key (16 characters) | | `RATCHET_API_KEY_DAILY_USD` | `0` | both | Every key's US dollar budget per UTC day (`0` = none; at most 1,000,000,000). Past it the key gets 429 `insufficient_quota` (code `budget_exceeded`) until 00:00 UTC, with `X-Ratchet-Budget-Remaining-USD: 0`, `Retry-After`, `X-Ratchet-Budget-Reset` and `X-Should-Retry: false`. A request in flight can overshoot by its own cost, because the check reads what was spent before it. A key with a dollar budget gets `X-Ratchet-Spend-Today-USD` and `X-Ratchet-Budget-Remaining-USD` on every answer | | `RATCHET_API_KEY_MONTHLY_USD` | `0` | both | Every key's US dollar budget per UTC month (`0` = none), refused the same way until 00:00 UTC on the 1st. With a monthly budget set, the gateway reads this month's spend from the ledger at startup (a pass over the requests index; with none, only today's) | | `RATCHET_API_KEY_RPM` | `0` | both | Every key's requests a minute (`0` = no limit): a token bucket that holds `max(1, rpm // 6)` requests (ten seconds' worth) and refills continuously. Past it a 429 `rate_limit_error` (code `rate_limit_exceeded`) with `Retry-After` (seconds until one more request passes, which OpenAI SDKs honour) and `X-RateLimit-Limit-Requests`, `X-RateLimit-Remaining-Requests` and `X-RateLimit-Reset-Requests`. Checked before the body is read, so a request refused later (400, 403, 429) has still used its token | | `RATCHET_API_KEY_CONCURRENCY` | `0` | both | Every key's requests in flight (`0` = no limit), counted once a request's body has arrived. One more is refused at once (429 `rate_limit_exceeded`, `Retry-After: 1`), never queued | | `RATCHET_DAILY_BUDGET_USD` | `0` | both | The gateway's own US dollar budget per UTC day for `/v1` chat requests, all keys (and requests without a key) together (`0` = none; at most 1,000,000,000). A scrubbed `/ratchet/stats` read does not show it. What happens past it is `RATCHET_BUDGET_MODE`. Scout is not counted: it has `RATCHET_SCOUT_DAILY_BUDGET_USD` | | `RATCHET_BUDGET_MODE` | `block` | both | `block`: past the gateway's budget every chat request gets 429 `insufficient_quota` (code `gateway_budget_exceeded`) until 00:00 UTC (`/v1/models` and request lookups cost nothing and still answer). `student_only`: confident student answers are still served, with no audit and no shadow; a request that needs the teacher (an escalation, a cold start, a pinned or unregistered task, a request with `tools`) gets that 429 without a teacher call, and what its student samples cost is recorded. Without audits, canary promotion pauses; a rollback still acts on the audits already recorded. Any case; empty means `block` | | `RATCHET_BUDGET_ALERT_PCT` | `50,80,100` | both | The shares of a dollar cap (a key's daily or monthly budget, the gateway's daily budget) that write a `budget` event the first time spend reaches them in the cap's UTC day or month (`acme reached 80% of $5.00 daily budget`, `the gateway reached 100% of $20.00 daily budget`). Whole percentages from 1 to 100, comma-separated; empty = no alerts. The gateway reads the period's `budget` events at startup, so a restart does not write one again; the events go to the log like every other (`ledger_event`), where an alert rule can catch them | | `RATCHET_IP_RPM` | unset: `0` in mock mode, `120` in live mode | both | Requests a minute per client IP on `/v1/chat/completions` of a gateway without `RATCHET_API_KEYS` (with keys, each key has its own limit and this is off), as a token bucket like `RATCHET_API_KEY_RPM`. The IP is the connection's, or the first `X-Forwarded-For` hop with `RATCHET_TRUST_PROXY=1` (on the VM uvicorn already takes it from Caddy). At most 10,000 addresses are remembered; past that the least recently seen is forgotten. `0` = no limit | | `RATCHET_MAX_INFLIGHT` | `32` | both | Chat requests in flight across the gateway before the next gets a 503 `overloaded` with `Retry-After: 1` instead of waiting for a worker thread (the pool has 40, so the dashboard and health checks keep answering under a burst). A request counts once its body has arrived, so slow uploads hold no slot. `0` = no limit | ## Scout, the multi-agent research tool From [Scout, the multi-agent research tool](/reference/configuration/#scout-the-multi-agent-research-tool) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `TAVILY_API_KEY` | empty | both | Tavily Search key for Scout's searcher. In live mode without it, Scout returns 503. The Scout CLI (`python -m ratchet.scout`) also reads it; without it the CLI uses mock results | | `RATCHET_API_KEY` | `unused` | both | Scout CLI only (`python -m ratchet.scout`): the gateway API key it sends as `Authorization: Bearer`, for a gateway with `RATCHET_API_KEYS` set. `--api-key` overrides it. The gateway itself never reads this variable | | `RATCHET_SCOUT_DAILY_BUDGET_USD` | `3.00` | both | Live mode: most Ratchet spend on Scout runs per UTC day. Once reached, new runs get 429. Mock mode has no budget | | `RATCHET_SCOUT_RUNS_PER_HOUR` | `6` | both | Runs per client IP per rolling hour. Mock mode allows 10 × this value | | `RATCHET_SCOUT_MAX_RUNS_PER_DAY` | `30` | both | Live mode: most runs per UTC day, for all clients together. Caps Tavily credits (about 3 searches per run, so 90 credits a day). Mock mode has no cap | | `RATCHET_SCOUT_MAX_RESULTS` | `4` | both | Tavily results per query. With 3 queries, at most 12 results are triaged | | `RATCHET_TRUST_PROXY` | `0` | both | `1`: take the client IP for the per-IP limit from the first `X-Forwarded-For` entry. Only set it when a proxy you control overwrites that header; otherwise clients can fake their IP. Not needed on the VM, where uvicorn runs with `--proxy-headers` | ## Scout, the multi-agent research tool: Judge access and Tavily credits From [Scout, the multi-agent research tool: Judge access and Tavily credits](/reference/configuration/#judge-access-and-tavily-credits) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_SCOUT_JUDGE_CODE` | empty | both | The access code for evaluators; empty turns judge access off. 16 to 256 printable ASCII characters with no spaces: use `openssl rand -hex 16`. Set it if and only if `RATCHET_SCOUT_JUDGE_RUNS_PER_DAY` is above 0. Judges send it once (`POST /scout/judge`) and get a signed cookie; never put it in a URL, the project page or the video | | `RATCHET_SCOUT_JUDGE_RUNS_PER_DAY` | `0` | both | Runs per UTC day held back for judges, 0 to `RATCHET_SCOUT_MAX_RUNS_PER_DAY`. The public gets the run cap minus this; judges get their share plus whatever public room is left. Equal to the run cap: judges only, and the public gets 429 (allowed) | | `RATCHET_SCOUT_JUDGE_BUDGET_USD` | unset | both | USD per UTC day held back for judges, 0 to `RATCHET_SCOUT_DAILY_BUDGET_USD`. Unset: the daily budget × judge runs / run cap, rounded down to the cent (7 of 22 runs of $3.00: $0.95). Above 0 only with judge access on | | `RATCHET_SCOUT_JUDGE_SESSION_S` | `604800` | both | How long a judge cookie lasts, 60 s to 30 days (7 days: a judge who comes back does not need the code again). Rotating the code ends every session | | `RATCHET_TAVILY_MONTHLY_CREDITS` | `0` | both | Tavily credits per cycle (the free plan gives 1,000 a month; the Nebius Builders Program may add more, unverified). `0` counts credits without a cap. A basic search costs 1 credit, an advanced one 2; the meter charges what Tavily reports in `usage.credits` | | `RATCHET_TAVILY_JUDGE_CREDITS` | unset | both | Credits per cycle held back for judge runs, at most the monthly credits, above 0 only with judge access on. Unset: monthly × judge runs / run cap, rounded down (1,000 × 7 / 22 = 318) | | `RATCHET_TAVILY_CYCLE_DAY` | `1` | both | Day of the month (1 to 28) the credit cycle starts, at 00:00 UTC. Tavily's own reset day is not documented (unverified): set the day your dashboard shows | | `RATCHET_TAVILY_PACE_UNTIL` | unset | both | A date (`YYYY-MM-DD`, UTC). While it falls in the current cycle and is today or later, the public's daily credit allowance spreads what is left until that date instead of the cycle's end; unused credits roll forward. Recommended: the last day of judging (the plan assumes `2026-12-15`; unverified) | ## Mock mode From [Mock mode](/reference/configuration/#mock-mode) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_MOCK_WARMUP_DAYS` | `0` | mock only | Mock days of `mixed` traffic, each followed by a night, run at startup and on every mock-mode gateway reset, so the mock-mode gateway opens with a trained student. `4` takes about 2 s and leaves a live student that answers about 80–90% of Scout's LLM calls (mock models). Writes a `warmup` event | | `RATCHET_MOCK_WARMUP_REQUESTS` | `1500` | mock only | Requests per warm-up day | | `RATCHET_MOCK_PACE` | `0.25` | mock only | `/scout/stream` waits this share of each call's mock latency (mock Ultra 2.4 s → 0.6 s), so the page shows agents working. `0` = no waiting. `/scout/run` never waits | | `RATCHET_MOCK_STRICT` | `0` | both | `1`: the mock models answer only the mock world's own requests (synthetic tickets and Scout's research calls), and anything else gets 400 `mock mode only answers the mock world's own requests`, as before the generic mock world. `0`: any other request gets a deterministic answer from the generic world (`sim/generic.py`): free text, an instance of its `response_format` schema, or a tool call to the mock teacher. Night School's mock trainer and golden eval read stored rows either way. With `1`, `POST /ratchet/demo/day` refuses the `schema` and `tools` workloads (400), which only the generic world answers | | `RATCHET_MOCK_ENDPOINT_HOURS` | `24` | mock only | Hours a day the mock student endpoint runs, 0 to 24: each mock day (the mock-mode gateway, the warm-up, `python -m sim.simulate`) records one endpoint interval of this length from the start of the day, on the synthetic clock, for the cost report ([Cost accounting](/reference/configuration/#cost-accounting)). `0` = no endpoint (as with the hosted fallback). Anything outside 0 to 24 stops startup. Mock-mode run assumption, not a measurement | | `RATCHET_MOCK_TEACHER_REASONING_MEAN` | unset | mock only | The mock teacher's mean billed reasoning tokens. Unset, it bills exactly `RATCHET_TEACHER_REASONING_TOKENS` (300), the formula's assumption, so every number is as before. Set (more than 0, at most 100,000; anything else stops the gateway and `python -m ratchet.live` at startup, in live mode too, where nothing reads it), each request bills round(mean × exp(0.3 z − 0.045)) reasoning tokens, z a standard normal drawn from the request's text (the same request always costs the same), so the savings receipts have a gap between the formula and the bills to measure: with `420`, `ticket-triage`'s calibrated ratio is about 1.3 to 1.4 (mock-mode run; [Savings receipts](/reference/configuration/#savings-receipts)). The simulator takes `--teacher-reasoning-mean` instead | ## Mock-mode host settings From [Mock-mode host settings](/reference/configuration/#mock-mode-host-settings) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_LIVE_URL` | empty | mock only | The live gateway's public URL, for the pages' links to the live gateway (`/ratchet/stats` `links.live`). Empty, or `https://` with a host, at most 300 characters, no user info, no whitespace | | `RATCHET_SIM_URL` | empty | mock only | The mock-mode run's public URL (`links.sim`). Same rule. On the VM the compose file derives it from `RATCHET_SIM_DOMAIN` unless `.env` sets it (empty: no link) | | `RATCHET_DEMO_RPM` | `0` (off) | mock only | Test-control actions (non-GET calls to `/ratchet/demo/*`) a minute per client IP, 0 to 600. Over it: 429 with `Retry-After`. The burst is `rpm // 6`, at least 1 | | `RATCHET_DEMO_MAX_DAYS` | `0` (off) | mock only | Reset the shared mock-mode gateway before the day after this many mock days. 0, or above `RATCHET_MOCK_WARMUP_DAYS`, at most 3650. Enforced when a visitor clicks `POST /ratchet/demo/day` (`auto_reset: "max_days"`) | | `RATCHET_DEMO_MAX_REQUESTS` | `0` (off) | mock only | Reset past this many ledger requests, 0, or above warm-up days x `RATCHET_MOCK_WARMUP_REQUESTS`, at most 10,000,000. Checked on a click and by a background pass every minute | | `RATCHET_DEMO_IDLE_RESET_S` | `0` (off) | mock only | Reset after this many seconds with no mock-mode action and no request while the state differs from the last reset. 0 or 300 to 604,800 | ## VM: Caddy (deploy/vm/) From [VM: Caddy (deploy/vm/)](/reference/configuration/#vm-caddy-deployvm) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_DOMAIN` | required | both | Site name Caddy serves and gets a certificate for, e.g. `ratchet.dualmindlab.tech` or `.sslip.io`. First boot fills in the sslip.io name if it is still the placeholder | | `RATCHET_API_GUARD` | `open` | both | `open`: every path is public. `guarded`: `/v1/*`, `/openai/*`, `/ratchet/demo/*` and `/ratchet/admin/*` return 403 unless the request has a matching `X-Ratchet-Admin-Token` header (the admin API also needs an admin key, `RATCHET_ADMIN_KEYS`, in live mode). Pages, `/scout*`, `/ratchet/stats` and `/ratchet/bom` stay public at the edge; whether the gateway serves the dashboard API without a key is `RATCHET_PUBLIC_DASHBOARD`. In mock mode, `guarded` also disables the dashboard's test buttons. In live mode without `RATCHET_API_KEYS`, the gateway starts only with `RATCHET_ALLOW_UNAUTHENTICATED=1` | | `RATCHET_ADMIN_TOKEN` | empty | both | Token for `guarded`. Hex, at least 32 characters (`openssl rand -hex 32`). Shorter or empty blocks the guarded paths for everyone. Caddy strips the header before it reaches the gateway | | `RATCHET_SIM` | `on` | deploy kit | `on`: Caddy serves the `ratchet-sim` container at `RATCHET_SIM_DOMAIN`. `off`: no mock-mode run host (the container still runs, unreachable); then also set an empty `RATCHET_SIM_URL=` so the live pages link nothing. `update.sh` refuses anything else | | `RATCHET_SIM_DOMAIN` | `sim.` | deploy kit | The mock-mode run host's name. It needs DNS: `sim..sslip.io` resolves by itself; with your own domain add an A record (or a wildcard) first. Must differ from `RATCHET_DOMAIN` (`update.sh` refuses it before it builds anything) | | `RATCHET_SIM_BOM_SIGNING_KEY` | empty | deploy kit | Optional key that signs the mock-mode gateway's BOMs, handed to `ratchet-sim` as its `RATCHET_BOM_SIGNING_KEY`. Never the same as the live `RATCHET_BOM_SIGNING_KEY` (`update.sh` refuses it): a mock BOM must not verify against the live public key | | `RATCHET_VM_NAME` | `ratchet-host` | deploy kit | The VM's name (`create_vm.sh` refuses a name that exists) | | `RATCHET_VM_PLATFORM` | `cpu-e2` | deploy kit | Nebius platform; `cpu-e2` exists only in `eu-north1`, elsewhere use `cpu-d3` | | `RATCHET_VM_PRESET` | `2vcpu-8gb` | deploy kit | Nebius preset, for example `4vcpu-16gb` | | `RATCHET_VM_DISK_GB` | `30` | deploy kit | Boot disk size in GB | | `RATCHET_SSH_USER` | `ubuntu` | deploy kit | The login user cloud-init creates (Nebius reserves `root` and `admin`) | | `RATCHET_REPO_URL` | `https://github.com/harshbhx/ratchet.git` | deploy kit | What the VM clones into `/opt/ratchet` | | `RATCHET_BACKUP_DIR` | `/var/backups/ratchet` | deploy kit | `backup.sh`: the host folder for ledger backups | | `RATCHET_BACKUP_KEEP` | `14` | deploy kit | `backup.sh`: local copies to keep | | `RATCHET_BACKUP_TAG` | empty | deploy kit | `backup.sh`: 1 to 32 letters, digits or `-`, added to the file name (`ratchet--.db.gz`); `update.sh` sets `pre-update`. Anything else exits 1 | ## Live gateway watchdog (ratchet/watch.py) From [Live gateway watchdog (ratchet/watch.py)](/reference/configuration/#live-gateway-watchdog-ratchetwatchpy) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_WATCH` | `on` | both | `on` or `off`. `off`: every command but `status` prints one line and exits 0 | | `RATCHET_WATCH_CLIENT` | `watchdog` | both | The `RATCHET_API_KEYS` name the probe uses; the host takes that name's key from `RATCHET_API_KEYS` itself. The pinned task's `clients` list in `deploy/vm/config/tasks.json` must name it | | `RATCHET_WATCH_FAILURES` | `3` | both | Failed liveness checks, or failed ledger checks, in a row before a restart (2 to 20, one check every 2 minutes) | | `RATCHET_WATCH_RESTART_COOLDOWN_S` | `600` | both | Seconds after a restart before another (120 to 86,400) | | `RATCHET_WATCH_MAX_RESTARTS_PER_HOUR` | `2` | both | Restarts in a rolling hour; past it one alert and no restart (1 to 10) | | `RATCHET_WATCH_START_GRACE_S` | `240` | both | No check counts while the container started less than this ago (0 to 3,600) | | `RATCHET_WATCH_PROBE_TIMEOUT_S` | `200` | both | The probe's HTTP timeout, above the gateway's own teacher deadline (180 s) so its retries finish first (10 to 900) | | `RATCHET_WATCH_REALERT_S` | `21600` | both | While a check keeps failing, repeat its alert this often (600 to 604,800) | | `RATCHET_WATCH_CONNECT` | `public` | both | `public`: the probe resolves `RATCHET_DOMAIN` as anyone would. `local`: it connects to `127.0.0.1:443` with the domain as SNI and `Host` (certificates still checked), for a VM that cannot reach its own public IP. Whether Nebius VMs can (hairpin NAT) is unverified | | `RATCHET_WATCH_HEARTBEAT_URL` | empty | both | Secret. An `https://` URL the probe GETs after a healthy probe that follows a healthy tick (healthchecks.io or any dead-man's switch: it alerts when the VM, cron or the watchdog is dead, which nothing on the VM can report) | | `RATCHET_WATCH_STATE_DIR` | `/var/lib/ratchet` | both | State files and locks (`watch-tick.json`, `watch-probe.json`, `watch.paused`, `update.lock`), mode 0600 | | `RATCHET_WATCH_FLAG_MAX_AGE_S` | `10800` | both | Gateway: a banner flag older than this reads `stale` and shows nothing (600 to 604,800) | | `RATCHET_ALERT_WEBHOOK_URL` | empty | both | Secret. `https://` only. Alerts are POSTed here as JSON with a `text` field (Slack's incoming-webhook shape; whether Slack ignores the other keys, and Discord's `/slack` suffix, are unverified: `python3 -m ratchet.watch test-alert` is the check). Empty: alerts go to `/var/log/ratchet/watch.log` only | | `RATCHET_ALERT_WEBHOOK_SECRET` | empty | both | Secret. When set, each alert carries `X-Ratchet-Signature: sha256=` | | `RATCHET_ALERT_KINDS` | `alert,rollback,train_failed,circuit,budget,admin,task_disabled,probe,watchdog_restart` | both | Kinds to send; an unknown kind is an error naming it. The watchdog sends only `probe` and `watchdog_restart` | | `RATCHET_INSTANCE_NAME` | the host name | both | `gateway` in the alert body | | `RATCHET_WATCH_KEY` | empty | live-run tool | Secret. `scripts/demo_check.py --chat` only: the `watchdog` key for its one live completion (from the environment, never a flag) | | `RATCHET_DEMO_CHECK_KEY` | empty | live-run tool | Secret. `scripts/demo_check.py` only: the key for its free `GET /v1/models` check (the judge key's), unless `--key-env` names another variable | ## Live Night School (ratchet/live.py) From [Live Night School (ratchet/live.py)](/reference/configuration/#live-night-school-ratchetlivepy) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_DATA_DIR` | `data` | live | Local folder synced to the bucket, laid out as the training job sees the bucket at `/data`: `night/train.jsonl`, `boms/student-vN.json`, and `adapters//ratchet_bom.json` and `NOTICE.md` (written after a night that trained, next to where the job wrote the weights; an adapter name that is not one folder name gets the `boms/` copy only). `python -m ratchet.live export` copies `adapters//` from here when the weights are here too (a bucket mount, or synced down) | | `RATCHET_SYNC_CMD` | empty | live | Command that uploads `RATCHET_DATA_DIR` to the bucket, e.g. `aws s3 sync ./data s3://my-bucket --endpoint-url https://storage.eu-north1.nebius.cloud` (skipped if empty). Run before the training job and again after a night that trained, so the adapter's BOM and NOTICE.md reach its bucket folder; a failure of the second run is a `sync_failed` event and a non-zero exit. It must only upload: an option that deletes remote files missing locally (`--delete`) would delete the adapter weights | | `RATCHET_TRAIN_CMD` | `bash deploy/train_job.sh {version}` | live | Command that runs training and **waits** for it. `{version}` and `{data}` are substituted | | `RATCHET_ADAPTER_ROOT` | `/data/adapters` | live | Adapter folder as the vLLM endpoint sees it (`lora_path` = `/`). Also read by the gateway in live mode, which reloads an adapter from there when vLLM answers 404 for it | ## Live-run harness From [Live-run harness](/reference/configuration/#live-run-harness) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_LIVERUN_KEY` | empty (required) | live-run tool | The harness's gateway API key: the key of the `liverun:` entry in `RATCHET_API_KEYS` (`--client` names another entry). Export it in the harness's shell only; never a flag, so it stays out of shell history and the process list, and it is never written or printed (results carry the name) | | `NEBIUS_API_KEY_LIVERUN` | empty (`NEBIUS_API_KEY`) | live-run tool | A separate Token Factory key for the harness's own arms (the control arm and the untuned-Nano arm), so a runaway run cannot drain the key the gateway and the judge page serve with. The gateway's own teacher calls keep `NEBIUS_API_KEY`. Results record which variable was used, never its value | | `RATCHET_GIT_SHA` | empty | live-run tool | The code's commit when the image has no `.git` (`git rev-parse HEAD` comes first). 40 or 64 hex characters; anything else is ignored. Results carry a SHA-256 over the `ratchet` and `sim` sources either way | ## Deploy scripts (deploy/) From [Deploy scripts (deploy/)](/reference/configuration/#deploy-scripts-deploy) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `DRY_RUN` | `0` | deploy kit | Used by both. `1`: `train_job.sh` and `student_endpoint.sh create` pass `--dry-run` (nebius validates the request and creates nothing); `start` and `stop`, which have no `--dry-run`, print the command instead of running it; the read-only `nebius ai endpoint list` still runs, and no uptime is recorded. Any value but `0` or `1` is refused (exit 2), so `DRY_RUN=true` never starts a paid GPU | | `RATCHET_BUCKET_ID` | required (job; endpoint `create`) | deploy kit | Used by both. Nebius Object Storage bucket ID, mounted at `/data` | | `RATCHET_SUBNET_ID` | empty (the project's default subnet) | deploy kit | Used by both. `--subnet-id` for the job and for `create` | | `RATCHET_TRAIN_IMAGE` | required | deploy kit | Used by `train_job.sh`. Training image built from `deploy/Dockerfile.train` | | `RATCHET_WHEELS_DIR` | `/data` | deploy kit | Used by `install_kernels.sh` (inside the training job). Where the prebuilt Mamba kernel wheels are cached, as `wheels-/` on the bucket mount | | `RATCHET_TRAIN_PLATFORM` | `gpu-l40s-a` | deploy kit | Used by `train_job.sh`. Nebius GPU platform | | `RATCHET_TRAIN_PRESET` | `1gpu-8vcpu-32gb` | deploy kit | Used by `train_job.sh`. Nebius preset | | `RATCHET_TRAIN_ON_DEMAND` | `0` | deploy kit | Used by `train_job.sh`. `1` submits the training job on demand; the default is a spot job (`--preemptible --follows-spot-price`). Anything but `0` or `1` is refused. Spot jobs can be preempted: the 28 Oct 2026 run (evidence/nightschool-nebius/) used spot throughout | | `RATCHET_REGISTRY_SECRET` | empty | deploy kit | Used by `train_job.sh`. `--registry-secret`: a MysteryBox secret with `REGISTRY_USERNAME` and `REGISTRY_PASSWORD`, for a private training image (Nebius Container Registry) | | `RATCHET_BUCKET_S3` | empty | deploy kit | Used by `train_job.sh`. `s3://bucket[/prefix]`. When set, the script (and the night, before it writes anything) needs the `aws` CLI on PATH and stops with a message if it is missing; it then polls for `adapters//adapter_model.safetensors` (every 30 s, up to 2 h, the job's `--timeout`) so it can be used as `RATCHET_TRAIN_CMD` | | `RATCHET_S3_ENDPOINT` | empty | deploy kit | Used by `train_job.sh`. S3 endpoint URL for Nebius Object Storage | | `RATCHET_STUDENT_KEY` | required (`create`) | deploy kit | Used by `student_endpoint.sh`. Token clients must send to the endpoint. Passed to `--token`, never printed (`` in the echoed command) | | `RATCHET_ENDPOINT_NAME` | `ratchet-student` | deploy kit | Used by `student_endpoint.sh`. Name `create` gives the endpoint, and the name `start`, `stop` and `status` look up in `nebius ai endpoint list --format json` to find its id | | `RATCHET_ENDPOINT_ID` | empty | deploy kit | Used by `student_endpoint.sh`. The endpoint's id (`metadata.id`). When set, `start` and `stop` use it without the list lookup, and `status` looks this id up instead of the name. Set it when the lookup cannot decide (two endpoints with the name, or a list with more pages) | | `RATCHET_PYTHON` | `python` | deploy kit | Used by `student_endpoint.sh`. Python 3 that reads the endpoint list (standard library only, `-I`) and runs `python -m ratchet.cost endpoint up\|down` after `create`, `start` and `stop` to record the endpoint's uptime in the ledger at `RATCHET_DB` (best effort: when recording fails, the deploy still succeeds and prints the command to run where the ledger is) | | `RATCHET_SERVE_PLATFORM` | `gpu-l40s-a` | deploy kit | Used by `student_endpoint.sh`. GPU platform for vLLM | | `RATCHET_SERVE_PRESET` | `1gpu-8vcpu-32gb` | deploy kit | Used by `student_endpoint.sh`. Preset for vLLM | | `RATCHET_VLLM_IMAGE` | `vllm/vllm-openai:v0.18.0-cu130` | deploy kit | Used by `student_endpoint.sh`. vLLM image. `vllm/vllm-openai:v0.18.0` is the CUDA 12 build of the same release (both on Docker Hub, pushed Mar 20, 2026), for a host driver too old for CUDA 13 | | `RATCHET_SERVE_MAX_LEN` | `8192` | deploy kit | Used by `student_endpoint.sh`. vLLM `--max-model-len`. The model card's example uses 262,144 tokens; ticket prompts need far less, and a longer prompt gets vLLM's 400 and escalates to the teacher | | `RATCHET_SERVE_MAX_SEQS` | `64` | deploy kit | Used by `student_endpoint.sh`. vLLM `--max-num-seqs` (concurrent sequences) | | `RATCHET_SERVE_TEMPLATE_KWARGS` | `{"enable_thinking":false}` | deploy kit | Used by `student_endpoint.sh`. vLLM `--default-chat-template-kwargs` (a JSON object, checked before `create`; no single quote), so the student answers without reasoning unless a request asks. Set but empty to leave the flag out (requests then carry their own `chat_template_kwargs`) | | `RATCHET_REASONING_PARSER_PLUGIN` | empty | deploy kit | Used by `student_endpoint.sh`. Absolute path in the container of the model card's `nano_v3_reasoning_parser.py` (for example `/data/nano_v3_reasoning_parser.py`, uploaded to the bucket). When set, vLLM gets `--reasoning-parser-plugin --reasoning-parser nano_v3` | | `RATCHET_STUDENT_BASE_MODEL` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | deploy kit | Used by both. Base model | | `RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY` | unset | deploy kit | Used by `nebius.sh`. Must be `1`, together with `--yes`, for `up` or `down` to create or delete anything; without both, those commands are a dry run that calls no Nebius API. `plan` and `status` ignore it | | `RATCHET_TRAIN_VERSION` | `student-v1` | deploy kit | Used by `nebius.sh`. Version name of the training job `up --with-training` submits through `train_job.sh` | | `RATCHET_SMOKE_CMD` | empty (`scripts/demo_check.py`) | deploy kit | Used by `nebius.sh`. One executable called as ` https:// --mode mock\|live` for the smoke check after `up` (tests set a stub) | ## Cost accounting From [Cost accounting](/reference/configuration/#cost-accounting) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_STUDENT_COST_MODEL` | `per_token` | both | Which savings the cost report's `headline` (and `/ratchet/stats` `cost.model`) shows: `per_token` (the student priced per token at the hosted Nano price, as every figure since v0.1) or `gpu_hours` (the GPU-hours the student endpoint and the training jobs ran). The report always has both views, and each request's `cost_usd` stays per token in both, so history never moves. Case and spaces are ignored; empty means `per_token` | | `RATCHET_STUDENT_GPU_USD_PER_HOUR` | `1.55` | both | USD per GPU-hour of the student endpoint (default 1.55, the list price this setting was first written against; Nebius's pricing page lists an L40S at $1.35 on demand, [docs.nebius.com/compute/resources/pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026, so set 1.35 to match it; **unverified** until the day-1 checks pass; 0 to 1,000). The default for `python -m ratchet.cost endpoint up` and `POST /ratchet/cost/endpoint` when they name no price, for the simulator's intervals, and the price in the break-even | | `RATCHET_STUDENT_GPUS` | `1` | both | GPUs the student endpoint runs on (0 to 1,024) (`1gpu-8vcpu-32gb` in `deploy/student_endpoint.sh`). The endpoint costs this many times the per-GPU price per hour | | `RATCHET_TRAIN_GPU_USD_PER_HOUR` | `1.55` | both | USD per GPU-hour of a training job (the same L40S price: default 1.55, $1.35 on Nebius's pricing page; unverified). Prices the training interval `python -m ratchet.live night` records around each training run, and the simulator's | | `RATCHET_TRAIN_GPUS` | `1` | both | GPUs a training job runs on (`RATCHET_TRAIN_PRESET`) | | `RATCHET_PRICES_VERIFIED` | `0` | both | `1` once `scripts/day1_check.py` passed with a real key (it did on 29 Oct 2026, for the per-token prices) and the GPU rates above match your Nebius bill. Until then every report labels its prices "unverified list prices" (invariant 9); with `1` it says "prices confirmed by the operator". Ratchet never sets it by itself | ## Savings receipts From [Savings receipts](/reference/configuration/#savings-receipts) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_TAKE_RATE` | `0.25` | both | The fee's share of a receipt's billable lower bound (after spend outside requests), from 0 to 1. Stated in every receipt as a decimal string (`billing.share`). The fee is rounded down to a nano-USD and is 0 when the bound is negative or no task is calibrated | | `RATCHET_RECEIPT_MIN_AUDITS` | `30` | both | Audited answers a task needs, in one receipt, before its baseline is calibrated and its served answers are billed; a whole number from 2 to 1,000,000. Below it the task is `insufficient_audits`: its savings are reported at the formula and not billed. At a 10% audit rate, 30 audits take about 300 student-served answers of the task in a day (per client, for a client's receipt) | ## Certified precision (Settings.cert_policy()) From [Certified precision (Settings.cert_policy())](/reference/configuration/#certified-precision-settingscert_policy) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_CALIBRATION` | `point` | both | How the threshold is chosen on the calibration half of the golden rows. `point`: the lowest threshold whose calibration answers reach 97% precision (v0.4's rule; routing and the reference runs do not move). `ltt`: Learn-then-Test over the fixed grid 0.95, 0.90, …, 0.50, keeping the lowest threshold whose precision is above `RATCHET_CERT_TARGET` with probability at least 1 − `RATCHET_CERT_DELTA` (1.01, serve nothing, when the highest fails). Case is ignored; empty means `point` | | `RATCHET_CERT_TARGET` | `0.95` | both | The precision a certificate must show (`certified` when the lower bound reaches it), and Learn-then-Test's target. Strictly between 0 and 1 | | `RATCHET_CERT_DELTA` | `0.05` | both | The certificate's error rate: the one-sided Clopper–Pearson bound holds with probability at least 1 − delta (the frontier's rows at delta / 10 each). Strictly between 0 and 1 | ## Ledger and hot path From [Ledger and hot path](/reference/configuration/#ledger-and-hot-path) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_DB_SYNC` | `NORMAL` | both | How a ledger file is journaled, `NORMAL` or `FULL` (any case; empty means `NORMAL`). `NORMAL`: the WAL journal with `synchronous=NORMAL`. Readers never wait for the writer, the dashboard reads on a second, read-only connection, and a commit does not wait for the disk: a power cut or an operating-system crash (not a crash of the gateway process) can lose the last commits, never corrupt the file. `FULL`: the rollback journal with `synchronous=FULL`, v0.4's behaviour, for a ledger on a filesystem without shared memory (a network or bucket mount, where WAL does not work) or when every commit must reach the disk before the client gets its answer. Give the gateway, the cron jobs and the cost CLI the same value: `FULL` cannot take a file out of WAL while another process has it open, and stops startup instead. Mock mode's in-memory ledger has no journal to set (a bad value still stops startup) | | `RATCHET_TICK_INTERVAL_S` | unset: `5` in live mode, `0` in mock mode | both | Seconds between two release-gate ticks run from the request path (0 to 3600). A tick less than this after the previous one, or while another thread is ticking, is skipped, so requests pay for a tick at most once per interval and never wait for one. A rollback can wait this long; the cron `python -m ratchet.live tick` runs regardless. `0` ticks after every request (mock mode, so test runs and tests behave as before); the mock-mode gateway's mock days always tick | | `RATCHET_STATS_TTL_S` | unset: `2` in live mode, `0` in mock mode | both | Seconds `/ratchet/stats` serves one computed payload (0 to 3600). However many dashboards poll, it is computed at most once per TTL, by one caller at a time; the others get the previous payload. `0` computes it on every read | ## Off-path quality checks From [Off-path quality checks](/reference/configuration/#off-path-quality-checks) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_SIDECAR_WORKERS` | unset: `4` in live mode, `0` in mock mode | both | Threads that run requests' audits and shadows after the answer is sent, 0 to 64. `0`: every check stays on the request path, as before (mock mode's default, so test runs and the reference runs are unchanged). Each worker holds at most one teacher call at a time (an audit, or an arbitration bounded to 60 s), so more workers spend the teacher's rate limit faster, never more in total | | `RATCHET_SIDECAR_QUEUE` | `1000` | both | Jobs that may wait for a worker, 1 to 100,000. A tenth of the places (at least one, once there are two) is kept for jobs that carry an audit. Each job holds its request's messages until it ends, so the request text queued and running jobs hold is bounded too, at 64 MiB (`sidecar.MAX_BYTES`, a tenth of it also kept for audits; at `RATCHET_MAX_INPUT_CHARS`' default a job holds at most about 0.5 MB): past it the queue counts as full. A drawn audit is never dropped: with every place taken it runs on the request path instead (`audits_inline`). A shadow without a place does not run for that request (`shadows_dropped`), counted per version in `/ratchet/stats` `sidecar` and in `ratchet_sidecar_full_total` | | `RATCHET_SIDECAR_DRAIN_S` | `10` | both | Seconds the gateway's shutdown waits for queued jobs, 0 to 3600, after the server stopped taking requests. Jobs still queued then never run: their rows stay pending (`audited` 1 with `audit_agree` NULL, `shadow_version` with `shadow_serve` NULL), which the release gate ignores, and one `sidecar_dropped` event counts them per version. Keep it under the process manager's stop timeout: Docker sends SIGKILL 10 s after SIGTERM by default; the VM's Compose file gives the gateway 20 s (`stop_grace_period`), so raise both together | ## Observability From [Observability](/reference/configuration/#observability) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_LOG_FORMAT` | unset: `text` in mock mode, `json` in live mode | both | How the `ratchet` logger writes to stderr: `json` (one object per line: `ts`, `level`, `logger`, `event`, then the fields) or `text` (` key=value ...`), any case. Lines carry ids, names, numbers and statuses, never prompt or answer text, URLs or keys | | `RATCHET_LOG_LEVEL` | unset: `WARNING` in mock mode, `INFO` in live mode | both | `DEBUG`, `INFO`, `WARNING` or `ERROR`, any case. `INFO` writes one `chat` line per chat request (request id, client request id, trace id, client name, task, served_by, route, version, confidence, escalated, audited, latency_ms, wall_ms, cost_usd, status; `retry_count` on a retry, `idempotent_replay` and `replay_of` on an answer sent again for an `Idempotency-Key`, whose `cost_usd` is 0, and `overhead_pending` when its checks run after the answer), one `offpath` line when those checks end (row, audited, audit_agree, shadow_version, shadow_serve, shadow_agree, stored, the `cost_usd` they added, `pending_ms`; the same request id, so the two lines' costs add up to the row's) and every release-log event (`ledger_event`, scrubbed of URLs and error messages); `WARNING` keeps the `upstream_error` lines (role, exception class, status), so mock mode and the tests stay quiet | | `RATCHET_METRICS_TOKEN` | empty | both | A bearer token that reads `GET /metrics` (and nothing else), for a Prometheus scraper, besides the admin and dashboard keys. At least 16 characters; it may not also be an API, admin or dashboard key (startup error). Compared as a constant-time digest, never stored or logged. Empty: only admin and dashboard keys read a private `/metrics`. A public dashboard (`RATCHET_PUBLIC_DASHBOARD=1`) makes `/metrics` public too | ## Configuration checks From [Configuration checks](/reference/configuration/#configuration-checks) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_STRICT_CONFIG` | unset (off) | both | `1`: a warning stops startup like an error, and `python -m ratchet.config check` exits 2 on it. `0` or unset: warnings start the gateway; each is kept in `Runtime.config_warnings`, logged once as a `config_warning` line (the variable and the message, never the value) and counted under `checks.config` in `/readyz`. Anything but 0 or 1 is an error | ## Success metrics From [Success metrics](/reference/configuration/#success-metrics) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_KPI_WINDOW_DAYS` | `3` | both | The steady state the success metrics describe: success metrics 1 to 3 and the latency target cover the last this many ledger days with traffic (days without requests are skipped). A whole number from 1 to 3,650; anything else stops startup with `ValueError` naming the variable. `GET /ratchet/kpis?days=N` overrides it for one report | ## Teacher watch (Settings.teacher_watch_limits()) From [Teacher watch (Settings.teacher_watch_limits())](/reference/configuration/#teacher-watch-settingsteacher_watch_limits) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_TEACHER_PROBES` | `20` | both | Stored labels Night School asks the teacher about again each night (one call each, as an audit asks: temperature 0, under the example's stored conditions), so traffic with few repeats is checked too. Only labels that can be checked: the configured teacher's, no redaction placeholders, a task that names fields and is not pinned, not contested, not re-tested yet in the current epoch; in hash order. Each one is that request's first re-test of the epoch. 0: none | | `RATCHET_ADJUDICATE_MAX` | `50` | both | Contested labels Night School adjudicates each night, two more teacher answers each: a strict majority of the three becomes the label and ends the quarantine; without one the label stays quarantined and is tried again after the labels never adjudicated. 0: none (contested labels stay out of training) | ## Release policy (ratchet/release.py) From [Release policy (ratchet/release.py)](/reference/configuration/#release-policy-ratchetreleasepy) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_MIN_PRECISION` | 0.95 | both | Field `min_precision`. Golden gate and shadow: minimum precision of answers the candidate would serve | | `RATCHET_MIN_COVERAGE` | 0.25 | both | Field `min_coverage`. Golden gate: minimum share of golden traffic it would serve | | `RATCHET_TOLERANCE` | 0.02 | both | Field `tolerance`. Max allowed drop vs live (precision and coverage) in the golden gate, and coverage vs live in shadow | | `RATCHET_MIN_SHADOW` | 100 | both | Field `min_shadow`. Would-serve shadow answers needed before a shadow decision | | `RATCHET_MIN_AUDITS` | 20 | both | Field `min_audits`. Teacher audits needed in a canary stage before it is promoted, and before the Wilson rules roll back or alert. The anytime-valid rules have no minimum for a rollback or an alert | | `RATCHET_LIVE_FLOOR` | 0.92 | both | Field `live_floor`. Audited precision floor. The Wilson rule rolls back when the Wilson upper bound is below it; the e-processes test "precision at or above the floor". Strictly between 0 and 1 | | `RATCHET_MIN_TASK_TEST_N` | 10 | both | Field `min_task_test_n`. Held-out (odd-half) golden rows a task needs before the golden gate checks its own numbers ([per-task calibration](/reference/configuration/#per-task-calibration)). A whole number, 0 or more | | `RATCHET_TASK_MODE` | `disable` | both | Field `task_mode`. What the golden gate does with a task that fails its own check: `disable` it on the candidate (its threshold becomes 1.01, a `task_disabled` event; the version ships for its other tasks) or `reject` the candidate. Case and surrounding spaces are ignored; empty means the default | | `RATCHET_SEQUENTIAL` | 0 | both | Field `sequential`. 1 switches the canary to the anytime-valid rule: rollback when the e-value over every audit the canary has had reaches 1/`alpha_rollback`, with no minimum count; promotion on this stage's audits (point estimate, or evidence with `promote_evidence`); retirement at `max_stage_audits`. 0: the Wilson rule above. 0 or 1 | | `RATCHET_ALPHA_ROLLBACK` | 0.05 | both | Field `alpha_rollback`. Sequential rule: the chance of ever rolling back a version whose precision stays at or above the floor (threshold 1/alpha: 20). Strictly between 0 and 1 | | `RATCHET_PROMOTE_EVIDENCE` | 0 | both | Field `promote_evidence`. 1 promotes a canary stage only when the e-value that precision is above the floor reaches 1/`alpha_promote` (56 audits that all agree at a 0.92 floor and 5%), instead of on the point estimate. Needs `sequential=1`. 0 or 1 | | `RATCHET_ALPHA_PROMOTE` | 0.05 | both | Field `alpha_promote`. Sequential rule with `promote_evidence`: the chance that a version at or below the floor reaches live. Strictly between 0 and 1 | | `RATCHET_MAX_STAGE_AUDITS` | 400 | both | Field `max_stage_audits`. Sequential rule: a canary stage with this many audits and no decision is retired (event `inconclusive`); 0 = never. It counts audits, not time. Must be at least `min_audits`, and with `promote_evidence` at least the audits a promotion needs (startup says how many) | | `RATCHET_ALERT_EPROCESS` | 1 | both | Field `alert_eprocess`. 1: the live alert fires when the e-value against "precision at or above the floor" over the audits since the version went live reaches 1/`alpha_alert`, with no minimum count. 0: the Wilson rule (at least `min_audits` audits and the upper bound below the floor). 0 or 1 | | `RATCHET_ALPHA_ALERT` | 0.01 | both | Field `alpha_alert`. The live alert's error rate: a live version at or above the floor is alerted on with probability at most this (threshold 1/alpha: 100). Strictly between 0 and 1 | | `RATCHET_MIN_CERTIFIED_PRECISION` | 0.0 | both | Field `min_certified_precision`. Golden gate: reject a candidate whose certified lower bound on precision (recomputed from its certificate's counts) is below this, or that has no certificate. 0: off. From 0 to 1 | | `RATCHET_PAIRED_GATE` | 0 | both | Field `paired_gate`. 1: the golden gate's regression check compares the candidate with the re-scored live version row by row on the same held-out rows (the candidate's `paired` record), instead of the two point estimates within `tolerance`. It rejects only a regression that is both larger than `tolerance` (as a share of the rows) and significant (exact McNemar, `regression_alpha`), on coverage or on wrong answers, or a negative flip rate above `max_nfr`. Without a paired record against the live version it falls back to the unpaired rule. 0 or 1 | | `RATCHET_REGRESSION_ALPHA` | 0.05 | both | Field `regression_alpha`. The paired gate's error rate for each McNemar test: a candidate no worse than live is rejected with probability at most this. Strictly between 0 and 1 | | `RATCHET_MAX_NFR` | 1.0 | both | Field `max_nfr`. The paired gate: reject when the 95% Wilson lower bound of the negative flip rate (answers live served right that the candidate serves wrong) is above this. 1: off. More than 0 and at most 1 | | `RATCHET_MIN_JUDGE_PASS` | 0 (off) | both | Field `min_judge_pass`. Golden gate on free-text tasks ([free-text tasks](/reference/configuration/#free-text-tasks)): the judge's pass rate the answers a candidate would serve need. Below it overall, the candidate is rejected; below it on a task with `min_task_test_n` held-out rows, that task fails as under `task_mode`. With it on, a free-text task the candidate would answer that no judge graded fails too. From 0 to 1 | | `RATCHET_DRIFT_ALARM` | 100 | both | Field `drift_alarm`. The label-free drift alarm: the alarm threshold of each (version, task) stream's test martingale on the student's confidence. A stream that sees no traffic change alarms with probability at most 1/threshold over its whole life (1% at 100); with K streams, at most K/threshold for any of them (`GET /ratchet/drift` reports both). A version holds at most 17 streams: 16 task streams and one for every later task name. An alarm writes a `drift` event and suspends the version's certificate; it decides no release. On by default; it reads the ledger in the gate's tick and never touches the request path. 0 turns it off; otherwise more than 1 | | `RATCHET_DRIFT_AUDIT_BOOST` | 1.0 | both | Field `drift_audit_boost`. After a drift alarm, the alarmed version's audit rate is multiplied by this while it serves (canary or live), up to 0.5, and never lowered. The boosted rate is logged as `label_p`; a newer version going live clears it. 1: off. 1 or more, and needs the monitor on | | `RATCHET_DRIFT_EARLY_NIGHT` | 0 | both | Field `drift_early_night`. 1: when a drift alarm came from a request logged after the last `trained` event, Night School needs half the usual new examples (`max(1, min_new_examples // 2)`). Never while a release is in flight. 0 or 1, and needs the monitor on | | `RATCHET_TEACHER_NOISE_MAX` | 0.05 | both | Field `teacher_noise_max`. The teacher watch: the rate at which a fresh teacher label of a request may disagree with an independent earlier label of it before the watch calls the teacher unstable (H0 of the freeze rule). Strictly between 0 and 1. Raise it on purpose for a teacher you know is noisier (a sampling teacher, a free-text-heavy task mix); the runbook says when | | `RATCHET_ALPHA_TEACHER` | 0.01 | both | Field `alpha_teacher`. The freeze rule's error rate: a teacher within `teacher_noise_max` freezes Night School with probability at most this per check epoch, however often the gate looks (threshold 1/alpha: 100). Strictly between 0 and 1 | | `RATCHET_TEACHER_WATCH` | 1 | both | Field `teacher_watch`. 1: the teacher watch acts: contested labels are left out of training and the golden set, the freeze rule pauses Night School (`teacher_drift`), the golden gate warns about the agreement ceiling, and Night School probes and adjudicates. 0: none of that; the free checks are still recorded (one indexed lookup per stored duplicate) and shown by `GET /ratchet/teacher`, so turning it on later has history. 0 or 1 | ## Reserved settings From [Reserved settings](/reference/configuration/#reserved-settings) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_BATCH_API` | `operations` | reserved | Clean-room backfill: the Token Factory batch API (`operations` or `openai`) | | `RATCHET_BATCH_DISCOUNT` | `0.5` | reserved | Clean-room backfill: the batch price discount (unverified, like every price) | | `RATCHET_BACKFILL_MAX_TOKENS` | `2048` | reserved | Clean-room backfill: the output cap the worst-case budget is computed from | | `RATCHET_GATEWAY_LEASE` | `warn` | reserved | One gateway process per live ledger (waiting for the leases table): `warn` will write a `gateway_lease` event when a second process serves the same ledger and keep serving, `exclusive` will refuse to start, `off` will do nothing. Any other value stops startup now, also through `Settings.scout_policy()` | ## Tasks From [Tasks](/reference/configuration/#tasks) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_TASKS_FILE` | empty | both | A file of task specs ([Adding a task](/reference/configuration/#adding-a-task)), JSON, or YAML for a `.yaml` or `.yml` file (needs PyYAML: `pip install pyyaml`, or the `yaml` extra). Read when the gateway starts (and on a mock-mode gateway reset) and by `python -m ratchet.live`. A file task may replace a built-in one. A bad file stops startup with every error and its path (`tasks[2].fields.total.abs_tol: must be a number >= 0`); a YAML file without PyYAML is an error that says so | | `RATCHET_UNKNOWN_TASKS` | `distill` | both | What a request naming a task that is not registered gets. `distill` (v0.4): a task without fields made up for the name, distilled like any other. `teacher`: the teacher answers (route `unknown_task`), the student is not called and nothing is stored. `reject`: 400 `unknown_task`, listing the registered tasks the key may use. Under `teacher` and `reject` an `X-Ratchet-Task` header that is not a valid task name is a 400 `unknown_task` too (under `distill` it means `default`, as in v0.4). Case-insensitive; anything else stops startup | | `RATCHET_SCHEMA_REPAIR` | `1` | both | `0`: no repair call. Otherwise a complete teacher answer that breaks its request's `response_format` gets one more call to the same teacher, shown its answer and the schema errors; the repaired answer is served (unless it was cut off or is empty: the first answer is then kept) and, when valid, stored with the request's own messages. Both calls are paid for in `cost_usd`; the baseline and the client's `usage.prompt_tokens` count the request's own prompt, and `latency_ms` both calls. Read in `ratchet/config.py`, used by `ratchet/router.py` (`Ratchet._repair`) | | `RATCHET_MODEL_POLICY` | `lenient` | both | What a request whose `model` Ratchet does not know gets ([task resolution](/reference/configuration/#task-resolution)). `lenient`: it is served as `ratchet` (its task from the other rules) and the response carries `X-Ratchet-Model-Alias: ` (UTF-8 percent-encoded except for visible ASCII), so clients with a hard-coded model name work unchanged. `strict`: a 404 `model_not_found` ("The model 'gpt-4o' does not exist"), before anything is logged or billed, also when a header names the task. Under both, a missing or empty `model`, `ratchet`, `ratchet/` and the aliases are accepted (`ratchet/` for an unregistered task then follows `RATCHET_UNKNOWN_TASKS`, and `GET /v1/models/{id}` gives the same answer), and `ratchet/` with a name that is not a task name is a 404. Case-insensitive; anything else stops startup | | `RATCHET_MODEL_ALIASES` | empty | both | Model ids that name a task, as `model=task` pairs: `gpt-4o-mini=ticket-triage,openai/gpt-4o=default`. For clients that send a fixed model name. An alias is 1 to 256 characters without spaces, `,` or `=`, and must not be `ratchet` or start with `ratchet/` or `ratchet:`; at most 256. The task need not be registered yet: until it is, requests for the alias follow `RATCHET_UNKNOWN_TASKS`. Aliases are listed in `GET /v1/models` (with `alias_of`). A bad entry stops startup | | `RATCHET_AUTO_TASK` | `off` | both | `system_prompt`: a request no header, metadata, model, alias or match rule places gets an auto task of its own per first system message, `auto-sys-` and the first 10 hex digits of the message's SHA-256 (the name holds a digest, never the prompt). A new auto task compares whole answers like `default`, is never pinned, and is registered and stored in the ledger (so a restart and Night School find it) once its first request has passed the quota. A request without a system message gets `default`. An auto task is a registered task, so `RATCHET_UNKNOWN_TASKS` does not apply to it; the admin API may replace it with an API task. Case-insensitive; anything else stops startup | | `RATCHET_MAX_AUTO_TASKS` | `200` | both | The most auto tasks that may exist (at most 50 in mock mode, whose shared mock-mode gateway a reset clears). Past it, or when the registry has only its last 50 slots free (kept for file and API tasks, so request traffic cannot lock the admin API out), a request with a new system prompt gets `default`, and one `auto_task_cap` event is logged once such a request is admitted. `0`: none is ever made. An admin can delete an auto task (`DELETE /ratchet/tasks/{name}`) to free its place. A whole number from 0 to 250 (half the registry); anything else stops startup | | `RATCHET_FIXTURES_DIR` | `fixtures` | both | Where fixture files a task spec names must live (`taskspec.resolve_fixture`: a path that resolves outside it, through `..` or a symlink, is refused): the SQL scripts that [text-to-SQL tasks](/reference/configuration/#text-to-sql-tasks) run their answers on. Relative to the working directory; the gateway and `python -m ratchet.live` both read it | ## Teacher licenses From [Teacher licenses](/reference/configuration/#teacher-licenses) in the configuration guide. | Variable | Default | Where | Meaning | |---|---|---|---| | `RATCHET_TEACHER_LICENSES` | empty | both | A JSON file: a list of `{"model", "license", "url", "note"}` objects (`note` optional, at most 100 entries). Read at startup by the gateway (and on a mock-mode gateway reset), the simulator, `python -m ratchet.live` and `python -m ratchet.admin`, and merged into a copy of the built-in registry. The configured teachers can then be `RATCHET_TEACHER_MODEL` or `RATCHET_TEACHER_FALLBACK_MODEL`, Night School trains on their stored answers, and each adapter's BOM and NOTICE.md name them with `license_origin` `config`. A bad file stops startup with every error and its place (`[2].license: 'CC-BY-NC-4.0' is not an allowed license; ...`). Empty = the built-in registry only | ## Checked against the code Every one of the 241 variables the package and the deploy kit read (181 in `KNOWN_ENV`, 34 in `DEPLOY_ENV`, 26 release policy fields) has a row above. --- # Configuration Source: https://docs.ratchet.dualmindlab.tech/reference/configuration/ :::tip Every variable in one table per area, with where it matters, is on [Environment variables](/api/environment/). ::: All settings come from environment variables. Defaults live in `ratchet/config.py` (`Settings`), `ratchet/live.py`, the `deploy/` scripts and `deploy/vm/`. Model IDs and prices were read from the Nebius Token Factory catalog in October 2026 and checked on 29 Oct 2026 (`python scripts/day1_check.py` passed with a real key); the GPU-hour rates are still unverified. Terms (shadow, canary, [Wilson interval](https://docs.ratchet.dualmindlab.tech/guides/glossary/#wilson-bound), [e-value](https://docs.ratchet.dualmindlab.tech/guides/glossary/#e-value)) are in GLOSSARY.md; FR-, NFR- and INV- ids are requirement and invariant ids (REQUIREMENTS.md). A one-page overview is in [README.md](https://github.com/harshbhx/ratchet/blob/main/README.md). ## Gateway and cascade (`ratchet/config.py`) | Variable | Default | Meaning | |---|---|---| | `RATCHET_MODE` | `mock` | `mock`: offline test run with mock models and an in-memory ledger. `live`: real Token Factory teacher and a SQLite file | | `RATCHET_DB` | `ratchet.db` | SQLite ledger path (live mode only; mock mode always uses `:memory:`) | | `RATCHET_TEACHER_BASE` | `https://api.tokenfactory.nebius.com/v1` | OpenAI-compatible base URL for the teacher, judge and fallback student (and the fallback teacher, unless `RATCHET_TEACHER_FALLBACK_BASE` is set) | | `NEBIUS_API_KEY` | empty | Token Factory API key (sent as `Authorization: Bearer …`) | | `RATCHET_TEACHER_MODEL` | `nvidia/Nemotron-3-Ultra-550b-a55b` | Teacher model. Must be in the license registry and distillable, or startup fails with `LicenseError` | | `RATCHET_JUDGE_MODEL` | `nvidia/nemotron-3-super-120b-a12b` | Judge for free-text golden evaluation (`LiveEvaluator`): the answers the student would serve on free-text tasks' held-out golden rows, read by `RATCHET_MIN_JUDGE_PASS` ([free-text tasks](#free-text-tasks)) | | `RATCHET_FALLBACK_STUDENT` | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` | Hosted, untuned model used for student calls when `RATCHET_STUDENT_BASE` is empty and a trained version exists (on a fresh deployment the teacher answers). It answers under `RATCHET_CONFIDENCE_THRESHOLD`, and its answers count toward no version's release: a canary or shadow holds while it serves. Reported as `fallback_student` in `/ratchet/stats` and `/scout/status`; the pages label a student answer with this `model` "Nano 30B · hosted", never as our adapter | | `RATCHET_HOSTED_STUDENT` | `0` | `1`: in live mode, while no version is live, the hosted model above answers each task `ratchet/hosted.py` calibrated it for, at that task's measured threshold, and the teacher answers the rest. The gateway keeps one version row in state `hosted` for it (no adapter, never listed in `/ratchet/stats` `versions`); its answers are audited as drawn, never stored and counted by no release. Startup fails unless `RATCHET_FALLBACK_STUDENT`, `RATCHET_SAMPLES` and `RATCHET_FALLBACK_STUDENT_REASONING` are what the records were measured with (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B`, `3`, `off`). Ignored, with a warning, in mock mode. As measured on Oct 26, 2026 no task met the bar (95% held-out precision against Ultra on 20 served rows), so it changes no answer yet | | `RATCHET_DEMO_SPLIT` | `0` | A share from 0 to 1 (0 = off): in live mode with `RATCHET_HOSTED_STUDENT=1`, each request that reaches the cascade (never a pinned task, a tool request, a teacher-only request, `n` > 1, a task with critical classes) goes to the hosted Nano with this probability whatever its confidence, to show the economics on the live gateway. Such answers have route `demo_split` and are labelled "Random split" on every page; the draw is its own secure generator, never the router's. Another value, mock mode or no `RATCHET_HOSTED_STUDENT=1` stops startup | | `RATCHET_DEMO_SPLIT_AUDIT` | `0.25` | The share of forced (`demo_split`) answers Ultra audits, from 0 to 1, so the dashboard shows how often the Nano agreed. Only Ultra's answer is stored | | `RATCHET_STUDENT_BASE` | empty | Base URL of your vLLM endpoint, e.g. `https:///v1` | | `RATCHET_STUDENT_KEY` | empty | Token for the student endpoint | | `RATCHET_STUDENT_BASE_MODEL` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | Base model that adapters are trained on (also used by the deploy scripts) | | `RATCHET_CONFIDENCE_THRESHOLD` | `0.80` | Fallback threshold, used only when a version has no calibrated `threshold` in its eval (and for the hosted, untuned fallback student) | | `RATCHET_SAMPLES` | `3` | Student samples per request (self-consistency) and per golden row, for a task whose spec sets no `policy.samples` (a task's own count wins). A request for more choices (`n`) draws `n` samples, but its confidence is still scored on the first `RATCHET_SAMPLES` (or the task's count) of them | | `RATCHET_AUDIT_RATE` | `0.05` | Probability that a student-served answer is also sent to the teacher. Mock mode raises it to at least `0.10`. A task's `policy.audit_rate` replaces it for that task ([risk-aware routing](#risk-aware-routing)), and a canary's rate below replaces both. An admin can replace them all at runtime, without a restart (`PUT /ratchet/admin/settings` or `python -m ratchet.admin flag audit_rate 0.2`; [API.md](/reference/api/#admin)); `controls.audit_rate` in `/ratchet/stats` shows the override, `null` while this value applies | | `RATCHET_CANARY_AUDIT_RATE` | unset | Audit focusing: the audit probability for answers a **canary** serves, so its release evidence arrives sooner; live answers keep `RATCHET_AUDIT_RATE`. More than 0 and at most 1 (0 would leave a canary without audits, so it could never be promoted, rolled back or retired); anything else stops startup. Unset or empty: the canary is audited at `RATCHET_AUDIT_RATE`. Still one random draw per served answer; the rate used is logged as `requests.label_p` | | `RATCHET_GOLDEN_FRACTION` | `0.10` | Share of new examples assigned to the golden (held-out) split | | `RATCHET_MIN_TASK_GOLDEN` | `20` | Calibration rows (the even half of a task's golden rows) a task needs before its own threshold may go below the version's global one ([per-task calibration](#per-task-calibration)). With fewer it gets the higher of the two and `calibrated: false`. A whole number, 1 or more; anything else stops startup (gateway, simulator, `python -m ratchet.live`) | | `RATCHET_PER_TASK_CALIBRATION` | `1` | `0` switches [per-task calibration](#per-task-calibration) off: Night School writes no per-task records, so each new version keeps one pooled threshold and the golden gate checks only the pooled numbers, as in v0.4; only a task with critical classes still gets its record (its own threshold, checked by the per-task gate and the [critical gate](#risk-aware-routing)), and the eval's `other_tasks` gives the other tasks' held-out rows and how many the pooled threshold serves, so disabling the critical task leaves the version the coverage of its other tasks. Versions already scored with records keep them until a night re-scores them. Any other value leaves it on. The reference workloads (`make reference`) run with it off, so they reproduce v0.4.0 exactly | | `RATCHET_TEACHER_REASONING_TOKENS` | `300` | Extra output tokens assumed for the teacher in baseline estimates (and added by the mock teacher) | ## Request limits and OpenAI compatibility Read in `ratchet/config.py`; used by `ratchet/gateway.py` and `ratchet/openai_compat.py`. Every check runs before any model call, and a refused request is neither logged nor counted against a quota ([API.md](/reference/api/#request-validation)). | Variable | Default | Meaning | |---|---|---| | `RATCHET_MAX_BODY_BYTES` | `2097152` | Largest request body, in bytes (2 MiB), on every `POST`, `PUT` and `PATCH`. A larger body gets **413** `request_too_large`: at once when `Content-Length` says so, or as soon as a chunked body passes the limit. `0` = no limit | | `RATCHET_MAX_MESSAGES` | `256` | Most messages in one `/v1/chat/completions` request; more is a 400 with `param: messages`. `0` = no limit | | `RATCHET_MAX_INPUT_CHARS` | `120000` | Most characters of message text in one request: content strings, text and refusal parts, `name`, `refusal`, `tool_call_id`, and tool call ids, function names and arguments, added up (everything forwarded to the teacher except non-text parts, which only `RATCHET_MAX_BODY_BYTES` bounds); more is a 400 with code `context_length_exceeded`. About 30,000 tokens of English. `0` = no limit | | `RATCHET_MAX_N` | `4` | Most choices (`n`) one request may ask for, 1 to 128; more is a 400 `unsupported_value` with `param: n` ([API.md](/reference/api/#choices-and-logprobs)). Each choice past `RATCHET_SAMPLES` is one more student sample on every such request, and an escalated one asks the teacher for `n` choices, which it bills as `n` answers. `1` restores the v0.4 behaviour of one choice per request. A value outside 1 to 128 stops the gateway at startup with `ValueError` | | `RATCHET_MERGE_SYSTEM` | `0` | `1`: merge consecutive leading `system` messages (after `developer` becomes `system`) into one, joined by a blank line, for models whose chat template takes a single system message. It changes the stored form of such requests, so set it before collecting training data, not halfway | | `RATCHET_TEACHER_MODALITIES` | `text` | Content part types the teacher accepts besides text, comma-separated, e.g. `text,image_url` (also `input_audio`, `file`). A request with such a part goes to the teacher only (route `ineligible:content_type`, [risk-aware routing](#risk-aware-routing)): no student call, no audit, nothing stored as a training example. Any other non-text part is a 400 `unsupported_content_type`. Set it only when the teacher model reads those parts; otherwise the teacher's refusal comes back as a 400 with its own message ([API.md](/reference/api/#errors)) | ### Other OpenAI endpoints `/v1/completions` and `/v1/responses` run the chat path, so every setting above and below applies to them as to `/v1/chat/completions` (a completion's prompt counts against `RATCHET_MAX_INPUT_CHARS` on its own). `/v1/embeddings` is a pass-through to one model ([API.md](/reference/api/#post-v1embeddings)): | Variable | Default | Meaning | |---|---|---| | `RATCHET_EMBEDDINGS_BASE` | (empty) | OpenAI-compatible base URL that `/v1/embeddings` calls. Empty: `RATCHET_TEACHER_BASE` (Token Factory). `off` (any case): the route answers 404 `unsupported_endpoint` and `/v1/models` does not list the embeddings model. Anything that is not an `http(s)` URL stops startup. Mock mode embeds locally unless it is `off` | | `RATCHET_EMBEDDINGS_MODEL` | `Qwen/Qwen3-Embedding-8B` | The one model `/v1/embeddings` asks for, whatever `model` a client sends; listed in `/v1/models`, and the id `RATCHET_MODEL_POLICY=strict` accepts there besides `ratchet`. **Unverified until `scripts/day1_check.py` passes**, which checks it. Priced from `EMBEDDING_PRICES` ([Prices](#prices-default_prices)); a model without a price there is priced as the teacher. An empty id, one with spaces around it, a Ratchet model id or one that is also a model alias stops startup | | `RATCHET_EMBEDDINGS_KEY` | (empty) | API key for `RATCHET_EMBEDDINGS_BASE`. Empty: `NEBIUS_API_KEY` when the base is the teacher's (`RATCHET_TEACHER_BASE`), and no key at all for any other base, so the Token Factory key is never sent to another provider. Set it for a provider that needs one; a local server without auth needs none | Each answered embeddings call is one request row (task `embeddings`, served_by `upstream`, route `passthrough`, nothing stored) and counts against the key's daily quota, budgets, rate and in-flight caps ([Spend and rate limits](#spend-and-rate-limits)). The task name `embeddings` is reserved for these rows: a task spec (tasks file, admin API) or a `RATCHET_MODEL_ALIASES` entry that names it is refused, and a chat request that names it is a 400 `reserved_task`. Its calls get the teacher's timeouts and retries (`RATCHET_TEACHER_TIMEOUT_S`, `RATCHET_TEACHER_RETRIES`, `RATCHET_TEACHER_DEADLINE_S`) and go through no breaker. `/v1/embeddings` also takes `RATCHET_MAX_INPUT_CHARS` (a token id counts as 4 characters) and at most 2,048 inputs. ## Retries and idempotency A client's retry neither pays for a second cascade nor counts as a second occurrence of its request ([API.md](/reference/api/#idempotency-and-retries)). Read in `ratchet/config.py` (`Settings.idempotency()`, `Settings.seen_dedup()`); used by `ratchet/idem.py`, `ratchet/gateway.py` and `ratchet/ledger.py`. Both structures live in memory, in each gateway process (with several workers, a retry that reaches another process runs again), and a mock-mode gateway reset clears them. A value out of range stops startup, naming the variable. | Variable | Default | Meaning | |---|---|---| | `RATCHET_IDEMPOTENCY_CACHE` | `10000` | Most answers kept by `Idempotency-Key`, 0 to 1,000,000; the least recently used goes first, and the answers kept never exceed 64 MiB together, each counted as its body plus 2 KiB for the entry that holds it (so at most about 32,000 small answers, whatever this says). `0` turns idempotency off: the header is ignored (not even checked) and every request runs | | `RATCHET_IDEMPOTENCY_TTL_S` | `600` | Seconds an answer is kept after it finished, 0 to 86,400; a request with the same key after that runs again. `0` keeps none: only a request that arrives while the first is still running shares its answer. Answers hold request-derived text, so they are never written to disk | | `RATCHET_SEEN_DEDUP_S` | unset: `300` in live mode, `0` in mock mode | Seconds within which the same API client's repeat of a stored example counts once toward the frequent-request rule (`seen`, `FREQUENT` = 3), 0 to 86,400; a repeat after the window counts again. Without API keys every caller is one client. Up to 10,000 (example, client) pairs are remembered, the least recently counted forgotten first. A request whose `x-stainless-retry-count` is above 0 never counts, whatever this says. `0`: every repeat counts, as in v0.4; mock mode's default, because the simulator's and the live gateway's repeats are traffic, not retries (internal calls, such as Scout's, the simulator's and the warm-up's, count every repeat in any mode). The VM's [`.env.vm.example`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/.env.vm.example) sets `300`, so the public mock-mode gateway counts a visitor's quick repeats once, as live mode does | ## Generation parameters and token caps Read in `ratchet/config.py` (`Settings.gen_policy()`); used by `ratchet/router.py`, the [golden eval](https://docs.ratchet.dualmindlab.tech/guides/glossary/#golden-set) (`ratchet/nightschool/evaluate.py`, `sim/mock_models.py`) and `ratchet/providers.py` ([API.md](/reference/api/#generation-parameters)). A bad value (a field that cannot be forwarded, a name that is not an identifier, a negative number) stops the gateway, `python -m ratchet.live night` and the simulator at startup with `ValueError`. | Variable | Default | Meaning | |---|---|---| | `RATCHET_TEACHER_PARAMS` | empty (every standard field) | Request fields the teacher gets when its answer is served (escalation, cold start, pinned, teacher-only), comma-separated, in any case: `max_tokens` (also for `max_completion_tokens`), `temperature`, `top_p`, `stop`, `seed`, `presence_penalty`, `frequency_penalty`, `logit_bias`, `response_format`, `tools`, `tool_choice`, `parallel_tool_calls`, `logprobs`, `top_logprobs`, `reasoning_effort`. Empty means the default: every standard field. `tools`, `tool_choice` and `parallel_tool_calls` are in it since v0.5: only a tool request has them, the teacher answers it alone and its tool calls come back to the client ([API.md](/reference/api/#tools-and-function-calling)); leave them out (name the other fields) and the teacher answers tool requests in prose, still teacher-only and never stored. `logprobs` and `top_logprobs` reach the teacher only when a client sends them, and its token logprobs come back in the response (v0.5); leave them out if the teacher refuses `logprobs` (`scripts/day1_check.py` reports whether it returns them), and teacher-served choices then carry `logprobs.content: null`. `none` forwards nothing; `all` stands for the whole list, and `all,top_k` adds a vendor field the client sends (otherwise unknown fields are never forwarded). Startup stops on `model`, `messages`, `n`, `stream`, `stream_options`, `user`, `safety_identifier`, `store`, `metadata`, `service_tier`, `functions` or `function_call`, on a name that is not an identifier, and on a near miss of a standard field (`max_token`, `stops`), which is a typo; any other name is taken as a vendor field. Audits and arbitrations take the same fields, minus `top_p`, `seed`, the penalties, `logit_bias` and logprobs, at temperature 0. A served answer written under a client's `logit_bias`, a non-zero penalty, a temperature above 1 or a vendor field is not stored | | `RATCHET_STUDENT_PARAMS` | empty (`max_tokens,stop,seed,response_format,top_logprobs`) | Request fields the student gets, same syntax. Its `n` (`RATCHET_SAMPLES`), temperature (0.7) and logprobs are fixed by the cascade whatever this says, so its confidence stays calibrated. `logit_bias` is not in the default on purpose: a client's bias would steer the student's answer while the audit runs without it, and OpenAI token ids mean other tokens to Nemotron; name it here to forward it. A request with tools never reaches the student (the teacher answers it alone), so `tools` and `tool_choice` here only matter for a golden row stored with them; add them only when the student's vLLM runs with `--enable-auto-tool-choice` and a `--tool-call-parser` (`deploy/student_endpoint.sh` does not), since without them vLLM answers 400. A golden row the student refuses this way scores as not served in the nightly eval | | `RATCHET_TEACHER_MAX_TOKENS` | `8192` | Most output tokens per teacher call (reasoning included, as Token Factory counts them). A client's larger `max_tokens` is capped to this; with none, this is sent. `0` = no cap | | `RATCHET_STUDENT_MAX_TOKENS` | `1024` | Most output tokens per student sample, the same way. `0` = no cap | | `RATCHET_TEACHER_REASONING_HEADROOM` | `0` | Tokens added to a client's `max_tokens` for the teacher (still at most `RATCHET_TEACHER_MAX_TOKENS`), for reasoning it writes before the answer: without it a reasoning teacher can spend a tight limit on its reasoning and be cut off. When the teacher gets more than the client asked for, a visible answer longer than the client's `max_tokens` (about 4 characters a token) is returned with `finish_reason: "length"` and not stored. Set it near `RATCHET_TEACHER_REASONING_TOKENS` for a reasoning teacher whose clients send small limits | ## Reasoning (Nemotron thinking) Read in `ratchet/config.py` (`Settings.reasoning_policy()`); sent as the chat template's `chat_template_kwargs`. A bad value stops the gateway, the simulator and `python -m ratchet.live` at startup. `default` sends nothing. | Variable | Default | Meaning | |---|---|---| | `RATCHET_STUDENT_REASONING` | `off` | The trained student on the vLLM endpoint (production, canary, shadow, both golden evals): `off` sends `{"enable_thinking": false}`, `default` nothing. `on` and `medium` stop startup: the student is trained with thinking off | | `RATCHET_FALLBACK_STUDENT_REASONING` | `off` | The hosted untuned student (`RATCHET_FALLBACK_STUDENT`) used while `RATCHET_STUDENT_BASE` is empty: `off`, `on` or `default`. Set `default` if Token Factory refuses the field (the student then fails fast and requests escalate, route `student_error`) | | `RATCHET_TEACHER_REASONING` | `default` | The teacher whose answer is served (escalation, cold start, pinned, teacher-only) and the fallback teacher: `default`, `off`, `on` or `medium` (Ultra's `{"enable_thinking": true, "medium_effort": true}`). Keep `default` until day 1 shows Token Factory accepts the field. With `off`, `RATCHET_TEACHER_REASONING_HEADROOM` is not needed | | `RATCHET_CHECK_REASONING` | empty (follows `RATCHET_TEACHER_REASONING`) | Audits and arbitrations, same values | | `RATCHET_TEACHER_REASONING_FIELD` | `chat_template_kwargs` | Where Token Factory-bound calls (teacher, fallback teacher, hosted fallback student) carry the kwargs: top level, or `extra_body` (nested as `extra_body.chat_template_kwargs`). **Unverified until day 1** (`scripts/day1_check.py`). The vLLM endpoint always gets the top-level field | A client's `chat_template_kwargs` and `extra_body` are accepted and ignored, and `RATCHET_TEACHER_PARAMS` and `RATCHET_STUDENT_PARAMS` refuse them. The kwargs are never stored or hashed. ## Upstream timeouts and retries Read in `ratchet/config.py`; used by `ratchet/providers.py` and `ratchet/resilience.py`. Connect, write and pool timeouts are fixed at 5, 10 and 5 seconds (the student's connect timeout has its own variable). A bad value (zero, negative, not a number) stops live mode at startup with `ValueError`. | Variable | Default | Meaning | |---|---|---| | `RATCHET_TEACHER_TIMEOUT_S` | `120` | Read timeout for teacher and judge calls on Token Factory, in seconds: how long the model may take to start answering | | `RATCHET_TEACHER_RETRIES` | `2` | Extra attempts for a failed teacher or judge call (`0` = no retries). Retried: connection failures and HTTP 408, 409, 429, 500, 502, 503, 504, with full-jitter backoff (0.5 s base, 8 s cap) or the server's `Retry-After`. A read timeout, a connection dropped mid-response or a 504 is retried once at most in all, because the model may already have generated billed tokens; any other 4xx, and a 2xx body that cannot be read, never. A server wait over 8 s is not waited for: the 429 goes back to the client with it | | `RATCHET_TEACHER_DEADLINE_S` | `180` | Upper bound on one teacher call, attempts and waits together, in seconds. A retry whose wait would end past it is not started, and later attempts get only the time that is left as their timeout. A `/v1/completions` batch starts no further prompt once it has run this long (504 `batch_deadline_exceeded`, [API.md](/reference/api/#post-v1completions)) | | `RATCHET_STUDENT_TIMEOUT_S` | `15` | Read timeout for student calls (the vLLM endpoint, or the hosted fallback when `RATCHET_STUDENT_BASE` is empty), in seconds. On the request path a student call is never retried: a failure escalates to the teacher at once. Night School's golden eval retries it like the teacher, since no teacher stands behind it there | | `RATCHET_STUDENT_CONNECT_TIMEOUT_S` | `2` | Connect timeout for student calls, in seconds, so an endpoint that is down costs a request two seconds rather than the teacher's five | | `RATCHET_NIGHT_STUDENT_TIMEOUT_S` | `120` | Read timeout for the student's calls in `python -m ratchet.live night` (the golden eval), in seconds. The first call after an adapter hot-load into a cold LoRA endpoint can take far longer than a warm request-path call: the 28 Oct 2026 night timed out at 15 s. The request path keeps `RATCHET_STUDENT_TIMEOUT_S`. The night does not start or wake the student endpoint: it must be running before the hot-load | | `RATCHET_STUDENT_BREAKER_FAILURES` | `5` | Student circuit breaker: endpoint failures in a row (a timeout, a connection error, an unreadable or 5xx answer, 408, 429; another 4xx is the endpoint answering and resets the count) after which student calls are skipped and requests go straight to the teacher (route `student_breaker`), so they stop waiting on a dead endpoint. Live and canary on one endpoint share a breaker; shadow calls have their own. `0` = no breaker ([ARCHITECTURE.md](/reference/architecture/#student-circuit-breaker)) | | `RATCHET_STUDENT_BREAKER_COOLDOWN_S` | `30` | Seconds an open student breaker waits before one request probes the endpoint again. A failed probe doubles the wait, up to 300 s (or this value, if larger); a successful probe closes the breaker and resets it. It may be shorter than the student's timeouts: another request probes only once the probe has been out for the longer of the wait and the student call's longest time (connect, write, read and pool timeouts plus 5 s: 2 + 10 + `RATCHET_STUDENT_TIMEOUT_S` + 5 + 5 = 37 s by default), and a probe that fails after that still reopens the breaker and doubles the wait | ## When the teacher is down Read in `ratchet/config.py`; used by `ratchet/router.py` and `ratchet/gateway.py` ([RUNBOOK.md](/reference/runbook/#teacher-api-down-token-factory-errors-or-timeouts)). Every teacher call goes through a [circuit breaker](https://docs.ratchet.dualmindlab.tech/guides/glossary/#circuit-breaker); the other two features are off by default. On an escalation the order is: the teacher, then the fallback teacher (if set), then the student's own answer (with `serve_student`), then the error. | Variable | Default | Meaning | |---|---|---| | `RATCHET_TEACHER_BREAKER_FAILURES` | `5` | Teacher circuit breaker: failed teacher calls in a row, each counted after its retries (a timeout, a connection error, an unreadable or 5xx answer, 408, 429; a 400 such as `context_length_exceeded`, a 401 or a tool-call answer is the teacher answering and resets the count), after which the teacher is not called. While it is open, a request that needs the teacher's answer gets **503** `teacher_circuit_open` at once, with the time until the next probe as `Retry-After` (1 to 60 s), and audits and arbitrations are skipped. The fallback teacher has a breaker of its own with the same settings. `0` = no breaker. A negative or non-integer value stops the gateway at startup | | `RATCHET_TEACHER_BREAKER_COOLDOWN_S` | `30` | Seconds an open teacher breaker waits before one request probes the teacher again. A failed probe doubles the wait, up to 300 s (or this value, if larger); a successful one closes the breaker. Must be a positive number | | `RATCHET_ON_TEACHER_FAILURE` | `error` | What an escalated request gets when the teacher (and the fallback teacher, if set) is unavailable. `error`: the mapped error (502, 503 or 504). `serve_student`: the student's own answer, below its threshold, when it gave at least one valid sample, with route `degraded`, `ratchet.degraded: "teacher_unavailable"` and `X-Ratchet-Degraded: teacher_unavailable`; never stored, audited or shadowed, so it never counts toward a release. Never for a 400, 413 or 422 about the request, a tool-call answer, a request with `tools`, a pinned or teacher-only request, a cold start, or a request whose student call failed. Case and spaces are ignored, empty means `error`, anything else stops the gateway at startup | | `RATCHET_TEACHER_FALLBACK_MODEL` | empty | A second teacher that answers a call whose answer is served (escalation, cold start, pinned, teacher-only) when the teacher is unavailable: its retries failed or its breaker is open. Never for audits or arbitrations, a 400 about the request or a tool-call answer. Must be in the license registry and distillable, or startup fails with `LicenseError` (invariant 1). Its answers are served with route `fallback_teacher` and its own model id, priced with its own price (`Settings.price`). When it fails too, the client gets the teacher's error. Empty = none | | `RATCHET_TEACHER_FALLBACK_BASE` | empty (`RATCHET_TEACHER_BASE`) | OpenAI-compatible base URL for the fallback teacher. It always uses `NEBIUS_API_KEY` and the teacher's timeouts and retries. Empty or equal to `RATCHET_TEACHER_BASE`: the teacher's client is used | | `RATCHET_STORE_FALLBACK_TEACHER` | `0` | `1`: store the fallback teacher's answers as training examples (under the usual rules), each with the fallback model and its own license in its provenance and the BOM. `0`: serve them, store nothing, so golden labels stay the teacher's | ## Privacy and security Read in `ratchet/config.py`; used by `ratchet/router.py`, `ratchet/auth.py`, `ratchet/privacy.py`, `ratchet/bom.py`, `ratchet/live.py`, `ratchet/admin.py` and `ratchet/gateway.py`. Background in [ARCHITECTURE.md](/reference/architecture/#pii-redaction), [API.md](/reference/api/#authentication), [API.md](/reference/api/#end-users-and-storage) (opt-outs and end users), [API.md](/reference/api/#erasure-and-export) and [THREAT_MODEL.md](/reference/threat-model/). | Variable | Default | Meaning | |---|---|---| | `RATCHET_REDACT_PII` | `1` | Replace emails, phone numbers, card numbers (Luhn-checked), IBANs (mod-97), Aadhaar (Verhoeff), US SSNs, Indian PANs and GSTINs, UPI ids, IPv4 addresses and API secrets with placeholders before a request and the teacher's answer are stored as a training example. The teacher still receives the original. `0` stores examples verbatim. Names and street addresses are found only by a detector plugin (`RATCHET_REDACT_PLUGIN`) | | `RATCHET_REDACT_STYLE` | `label` | How identifiers are written in stored examples: `label` (`[EMAIL]`, v0.4's output), `numbered` (`[EMAIL_1]`, `[EMAIL_2]`: the same value gets the same number in the prompt, the stored conditions and the answer, so an extraction task's answer still says which of two addresses it names) or `surrogate` (reserved values of the same shape: `person1@example.com`, `+12025550100`, a 16-digit card number that fails Luhn, a `ZZ` IBAN, `192.0.2.1`; the numbered placeholder for other identifiers). A task spec's `privacy.redact_style` overrides it for that task ([Tasks](#task-options)). Numbers start again in every example, and nothing original is kept. Changing it changes the stored text, so a request stored before and after is two examples (old rows are never rewritten); an erased example stays erased (each row stored otherwise than in `label` records its `label` key, which an erasure tombstones too). Case and spaces are ignored; empty means `label`; anything else stops startup | | `RATCHET_REDACT_PLUGIN` | empty | A detector for what the patterns cannot find, such as names and addresses: `module:function` (a dotted attribute path after the colon), importable by the gateway. The function takes a text and returns `(start, end, label)` spans (`0 <= start < end <= len(text)`, a label of 1 to 32 letters, digits and `_`); they are merged with the patterns' and written in the active style. Imported and called once on an empty text at startup; a missing module or attribute, or a detector that fails there, stops startup. It runs whenever an example is stored, on the request's thread, so its latency adds to escalated and audited requests, and it must be thread-safe. When it fails at store time the answer is served and nothing is stored (`skip_example` `redaction failed`). `POST /ratchet/redact` runs it only for callers with a valid API or admin key. a Presidio detector is a typical plug-in; the package bundles no NER model | | `RATCHET_STORE_EXAMPLES` | `1` | `0` is zero retention: every request is answered, audited and logged as usual, but no teacher answer is stored as a training example, and Night School skips every night (`storage disabled`), so examples kept from before are not trained on. A task whose spec changed is still re-scored on the stored golden rows when no release is in flight (that writes no example), so it does not stay on the teacher for good. Requests are logged with `store_opt_out` `global`. Accepts `1`/`0`, `true`/`false`, `yes`/`no`, `on`/`off` in any case; anything else stops startup, so a typo cannot mean "store" | | `RATCHET_NO_STORE_TASKS` | empty | Tasks whose teacher answers are never stored, comma-separated (`invoice-extract,hr-notes`): the same as `store: false` in their specs ([Tasks](#tasks)). Requests are logged with `store_opt_out` `task`. A name need not be registered, but one that is not a task name (a space-separated list) stops startup. Examples stored before stay | | `RATCHET_FEEDBACK_CORRECTIONS` | `1` | Whether a correction sent with an `incorrect` verdict (`POST /v1/ratchet/feedback`) may be stored as a human-labelled training example. `0` keeps every verdict (task agreement, human precision, the `human_alert` event) and stores no correction (`not_stored: corrections_off`), for deployments whose API clients should not write labels. The storage opt-outs above and pinned tasks apply either way. Accepts `1`/`0`, `true`/`false`, `yes`/`no`, `on`/`off` | | `RATCHET_END_USER_KEY` | empty | The HMAC key end users are hashed with, at least 16 characters (`openssl rand -hex 32`). The request's `safety_identifier`, else its `user`, is stored only as HMAC-SHA256 under this key, scoped to the API key name, cut to 16 hex digits. Empty: 32 random bytes are generated on first start and kept in the ledger's `meta` table, so hashes survive restarts, but anyone with a copy of the database file then has the key too; set this from a secrets manager to keep it out of the file and its backups. Changing it unlinks every earlier hash: no-train registrations stop matching (an `end_user_key` event says how many), so those end users must register again. A shorter value stops startup | | `RATCHET_EXAMPLE_RETENTION_DAYS` | `0` | Whole days a stored example is kept after it was last stored or seen again; `0` keeps it forever. Past the window its messages, answer and conditions are removed (golden examples too) and the row stays as a tombstone with its hash, so Night School's counts and an erasure still see it; a later duplicate that may be stored brings the text back with its own teacher and license. Applied hourly by the live gateway, at the start of every live night and by `python -m ratchet.live purge`. 0 to 36500; anything else stops startup | | `RATCHET_REQUEST_RETENTION_DAYS` | `0` | Whole days a request row (metadata only) is kept; `0` keeps it forever. Older rows are folded into the `daily_rollup` table (per day, task, client, route and opt-out: the sums every stats reader and the cost report add back; no end user) and deleted, except rows logged since a release stage in flight began and the rows a version in shadow or canary served or shadowed in an earlier stage, and the rows of a day whose savings receipt is not sealed yet (a day that ended less than 5 minutes ago keeps every row until the next purge seals it). Distinct end users per client then count only the rows still kept. Use at least 1, so today's rows (the daily quota counts them) stay. 0 to 36500; anything else stops startup | | `RATCHET_ERASURE_SHARED` | `delete` | What erasing a client or an end user does with an example another client also sent: `delete` purges it too (the text is the same, so the erased party's data was in it); `keep` keeps it for the other client and unlinks the erased party (the other client becomes its first sender, without an end user). Erasure by hashes or by text purges whatever it selects. Anything else stops startup | | `RATCHET_API_KEYS` | empty | `name:key,name:key`. When set, `/v1/*` and `/openai/*` need `Authorization: Bearer `, `X-API-Key: ` or `api-key: `; anything else gets 401. Names: letters, digits, `_`, `.`, `-` (up to 64). Keys: at least 16 characters; generate with `openssl rand -hex 24`. A bare key without `name:` is named `keyN`. The ledger stores the name per request, never the key. End-user hashes and no-train registrations are scoped to the name ([API.md](/reference/api/#end-users-and-storage)): rotate a key under the same name; renaming one unlinks its end users' earlier hashes, and its clients' no-train registrations match nothing until the name is back (a `no_train_unmatched` event at startup; an admin's registration for every client is not affected). A key also reads its own savings receipts and their proofs, never another client's ([API.md](/reference/api/#savings-receipts)). A malformed value stops the gateway at startup | | `RATCHET_API_KEY_DAILY_REQUESTS` | `0` | Requests per key per UTC day (chat completions, responses, embeddings, and completions at one per prompt); past it a 429 `insufficient_quota` (code `daily_quota_exceeded`) with `Retry-After` until 00:00 UTC and `X-Should-Retry: false` ([API.md](/reference/api/#authentication)). `0` = unlimited. `RATCHET_KEY_LIMITS` overrides it per key ([Spend and rate limits](#spend-and-rate-limits)). Counted by the spend meter, which reads today's requests from the ledger at startup, so it survives restarts in live mode | | `RATCHET_ADMIN_KEYS` | empty | Admin keys for management calls (`PUT` and `DELETE /ratchet/tasks/{name}`, `POST /ratchet/tasks/validate`, [API.md](/reference/api/#tasks); `POST /ratchet/cost/endpoint`; every route of the admin API under `/ratchet/admin/`, reads included, [API.md](/reference/api/#admin); and the training-set proofs, `POST /ratchet/versions/{id}/proof` and `GET /ratchet/versions/{id}/heldout`, [API.md](/reference/api/#committed-training-sets), which are a membership oracle for the training data; and every savings receipt, the gateway's and each client's, [API.md](/reference/api/#savings-receipts)), in the `RATCHET_API_KEYS` syntax: `name:key,name:key`, keys at least 16 characters. Sent as `Authorization: Bearer ` or `X-API-Key`; a missing or wrong key gets 401, an API key 403. The ledger and the event log record the name, never the key. A key may not be in both variables (startup error). Empty: the admin API is off in live mode (403 `admin API disabled: set RATCHET_ADMIN_KEYS`) and open in mock mode (the admin is `mock-admin`). Not the Caddy guard's `RATCHET_ADMIN_TOKEN` | | `RATCHET_DASHBOARD_KEYS` | empty | Read-only keys for the dashboard API ([API.md](/reference/api/#access-levels)), in the `RATCHET_API_KEYS` syntax. A dashboard key reads everything an admin key reads and writes nothing. A key may not also be an API key or an admin key (startup error), so each key has one role. Give one to whoever watches a live gateway's dashboard | | `RATCHET_PUBLIC_DASHBOARD` | `1` in mock mode, `0` in live mode | `1`: the dashboard API (`GET /ratchet/stats`, `GET /ratchet/cost`, `POST /ratchet/redact`, `GET /ratchet/tasks` and `/ratchet/tasks/{name}`, the BOM endpoints and the other reads [API.md](/reference/api/#access-levels) lists) needs no key. `0`: it needs an admin or dashboard key, as `Authorization: Bearer ` or `X-API-Key`; without one it is 401, with an API key 403, and with neither `RATCHET_ADMIN_KEYS` nor `RATCHET_DASHBOARD_KEYS` set every read is 403. The pages, `/healthz`, `/ratchet/bom/public-key` and Scout are always public. The dashboard page sends no key yet, so with `0` it shows no data: read the API with a key, or set `1` (the VM does) | | `RATCHET_PUBLIC_BOM` | `0` | `1`: `GET /ratchet/bom` and `GET /ratchet/versions/{id}/bom` stay public while the dashboard API is private, so auditors can check an adapter's provenance without a key | | `RATCHET_SCRUB_STATS` | unset | What a reader of the dashboard API sees. Unset: an anonymous read of a public dashboard in live mode is scrubbed: client names in `/ratchet/stats` and in task allow-lists (`GET /ratchet/tasks`) become `client-1`, `client-2`, ... (by sorted name, the same in both), event details lose every `http(s)://` URL (``), and an error message after an exception's class is cut (`ConnectError: ...` becomes `ConnectError`; v0.4 ledgers hold such events). A reader with an admin or dashboard key sees everything. `1`: every read is scrubbed, with a key or without, in either mode. `0`: none is. `security.scrubbed` in `/ratchet/stats` says which a reader got | | `RATCHET_OPENAPI` | `1` in mock mode, `0` in live mode | `1`: FastAPI serves `/docs`, `/redoc` and `/openapi.json`. `0`: they do not exist (404), so a live gateway does not hand anyone a map of its routes | | `RATCHET_ALLOW_UNAUTHENTICATED` | `0` | Live mode refuses to start without `RATCHET_API_KEYS`, because `/v1/*` would answer anyone, on your teacher credits. `1` lets it start for a gateway something else guards, but only with the guard configured: `RATCHET_API_GUARD=guarded` and a `RATCHET_ADMIN_TOKEN` of at least 32 characters (the VM's Caddy checks both), so one variable alone never opens the paid API. Otherwise startup and `python -m ratchet.config check` refuse, naming both variables and never the token. Mock mode ignores it | | `RATCHET_CSP` | empty | A `Content-Security-Policy` for every response. Empty sends none, so no page can break by default. Printable ASCII only (anything else stops startup). Recommended, which the React build and the lite pages work with: `default-src 'self'; script-src 'self' 'unsafe-inline'; style-src 'self' 'unsafe-inline'; img-src 'self' data:; font-src 'self' data:; connect-src 'self'; object-src 'none'; base-uri 'none'; frame-ancestors 'none'; form-action 'self'` (the pages load nothing from other origins; `'unsafe-inline'` is for the theme script in the React page and the lite pages' inline scripts). It also blocks `/docs`, whose Swagger UI comes from a CDN | | `RATCHET_BOM_SIGNING_KEY` | empty | Ed25519 private key as the base64 of its 32-byte seed. Signs each adapter's bill of materials and every savings receipt (same key, same public key). Generate with `python -m ratchet.bom keygen`, which also prints the public key to publish. Needs the `sign` extra (`pip install "ratchet-distill[sign]"`, included in the gateway image). An invalid value stops the gateway at startup. Empty = BOMs and receipts are stored with a SHA-256 but unsigned | The switches above take `0` or `1`; any other value stops startup, and unset or empty takes the default. Every response also carries `X-Content-Type-Options: nosniff` and `Referrer-Policy: no-referrer`. Keep `RATCHET_API_KEYS`, `RATCHET_ADMIN_KEYS`, `RATCHET_END_USER_KEY`, `RATCHET_DASHBOARD_KEYS`, `RATCHET_METRICS_TOKEN` and `RATCHET_BOM_SIGNING_KEY` out of git, like every other key. Rotating the signing key does not invalidate old [BOMs](https://docs.ratchet.dualmindlab.tech/guides/glossary/#bom): each one embeds the public key and key id it was signed with, so keep a list of the public keys you have published. ## Spend and rate limits What each API key and the whole gateway may spend and send ([API.md](/reference/api/#limits-and-headers), [RUNBOOK.md](/reference/runbook/#cost-spike)). Read in `ratchet/config.py` (`Settings.limit_policy`); `RATCHET_KEY_LIMITS` is parsed by `ratchet/auth.py` against the key names; used by `ratchet/spend.py`, `ratchet/ratelimit.py` and `ratchet/gateway.py`. A bad value, or `RATCHET_KEY_LIMITS` naming a key that is not in `RATCHET_API_KEYS`, stops startup with a `ValueError` naming the variable. Dollars are per-token prices ([Prices](#prices-default_prices)), mock in mock mode; a key's spend counts answered requests and what failed requests spent (the student's samples before a teacher failure), for gateway API requests only: Scout's agents stay under the Scout budget (invariant 11). Limits are kept per process (one worker) and keyed by key name, never the key. | Variable | Default | Meaning | |---|---|---| | `RATCHET_KEY_LIMITS` | empty | Per-key overrides of the defaults below, by key name: `acme:daily_requests=10000,daily_usd=5,monthly_usd=100,rpm=60,concurrency=4,tasks=ticket-triage\|invoice-extract,store=1;globex:daily_usd=1`. Entries are separated by `;`, fields by `,`. `daily_requests`, `rpm` and `concurrency` are whole numbers, `daily_usd` and `monthly_usd` plain decimals (US dollars, at most 1,000,000,000), `0` turns a limit off for that key. `tasks` lists the task names the key may send, separated by `\|` (any other task, `default` included, is a 403 `task_not_allowed`; without `tasks` every task its own `clients` allow). `store=0` keeps the key's requests out of the training set: they are served, audited and logged, never stored (`store=1`, the default, stores as usual). A name that is not in `RATCHET_API_KEYS`, an unknown field, a field or a key given twice, or a bad value stops startup; the message lists the key names and never prints what may be a key pasted by mistake (`acme:`, in the `RATCHET_API_KEYS` syntax): an unknown name is not repeated, and an item, field or value is quoted only when it is shorter than any key (16 characters) | | `RATCHET_API_KEY_DAILY_USD` | `0` | Every key's US dollar budget per UTC day (`0` = none; at most 1,000,000,000). Past it the key gets 429 `insufficient_quota` (code `budget_exceeded`) until 00:00 UTC, with `X-Ratchet-Budget-Remaining-USD: 0`, `Retry-After`, `X-Ratchet-Budget-Reset` and `X-Should-Retry: false`. A request in flight can overshoot by its own cost, because the check reads what was spent before it. A key with a dollar budget gets `X-Ratchet-Spend-Today-USD` and `X-Ratchet-Budget-Remaining-USD` on every answer | | `RATCHET_API_KEY_MONTHLY_USD` | `0` | Every key's US dollar budget per UTC month (`0` = none), refused the same way until 00:00 UTC on the 1st. With a monthly budget set, the gateway reads this month's spend from the ledger at startup (a pass over the requests index; with none, only today's) | | `RATCHET_API_KEY_RPM` | `0` | Every key's requests a minute (`0` = no limit): a token bucket that holds `max(1, rpm // 6)` requests (ten seconds' worth) and refills continuously. Past it a 429 `rate_limit_error` (code `rate_limit_exceeded`) with `Retry-After` (seconds until one more request passes, which OpenAI SDKs honour) and `X-RateLimit-Limit-Requests`, `X-RateLimit-Remaining-Requests` and `X-RateLimit-Reset-Requests`. Checked before the body is read, so a request refused later (400, 403, 429) has still used its token | | `RATCHET_API_KEY_CONCURRENCY` | `0` | Every key's requests in flight (`0` = no limit), counted once a request's body has arrived. One more is refused at once (429 `rate_limit_exceeded`, `Retry-After: 1`), never queued | | `RATCHET_DAILY_BUDGET_USD` | `0` | The gateway's own US dollar budget per UTC day for `/v1` chat requests, all keys (and requests without a key) together (`0` = none; at most 1,000,000,000). A scrubbed `/ratchet/stats` read does not show it. What happens past it is `RATCHET_BUDGET_MODE`. Scout is not counted: it has `RATCHET_SCOUT_DAILY_BUDGET_USD` | | `RATCHET_BUDGET_MODE` | `block` | `block`: past the gateway's budget every chat request gets 429 `insufficient_quota` (code `gateway_budget_exceeded`) until 00:00 UTC (`/v1/models` and request lookups cost nothing and still answer). `student_only`: confident student answers are still served, with no audit and no shadow; a request that needs the teacher (an escalation, a cold start, a pinned or unregistered task, a request with `tools`) gets that 429 without a teacher call, and what its student samples cost is recorded. Without audits, canary promotion pauses; a rollback still acts on the audits already recorded. Any case; empty means `block` | | `RATCHET_BUDGET_ALERT_PCT` | `50,80,100` | The shares of a dollar cap (a key's daily or monthly budget, the gateway's daily budget) that write a `budget` event the first time spend reaches them in the cap's UTC day or month (`acme reached 80% of $5.00 daily budget`, `the gateway reached 100% of $20.00 daily budget`). Whole percentages from 1 to 100, comma-separated; empty = no alerts. The gateway reads the period's `budget` events at startup, so a restart does not write one again; the events go to the log like every other (`ledger_event`), where an alert rule can catch them | | `RATCHET_IP_RPM` | unset: `0` in mock mode, `120` in live mode | Requests a minute per client IP on `/v1/chat/completions` of a gateway without `RATCHET_API_KEYS` (with keys, each key has its own limit and this is off), as a token bucket like `RATCHET_API_KEY_RPM`. The IP is the connection's, or the first `X-Forwarded-For` hop with `RATCHET_TRUST_PROXY=1` (on the VM uvicorn already takes it from Caddy). At most 10,000 addresses are remembered; past that the least recently seen is forgotten. `0` = no limit | | `RATCHET_MAX_INFLIGHT` | `32` | Chat requests in flight across the gateway before the next gets a 503 `overloaded` with `Retry-After: 1` instead of waiting for a worker thread (the pool has 40, so the dashboard and health checks keep answering under a burst). A request counts once its body has arrived, so slow uploads hold no slot. `0` = no limit | ## Scout, the multi-agent research tool Read in `ratchet/config.py`; used by `ratchet/gateway.py` and `ratchet/scout/guard.py`. Details in [SCOUT.md](/reference/scout/). | Variable | Default | Meaning | |---|---|---| | `TAVILY_API_KEY` | empty | Tavily Search key for Scout's searcher. In live mode without it, Scout returns 503. The Scout CLI (`python -m ratchet.scout`) also reads it; without it the CLI uses mock results | | `RATCHET_API_KEY` | `unused` | Scout CLI only (`python -m ratchet.scout`): the gateway API key it sends as `Authorization: Bearer`, for a gateway with `RATCHET_API_KEYS` set. `--api-key` overrides it. The gateway itself never reads this variable | | `RATCHET_SCOUT_DAILY_BUDGET_USD` | `3.00` | Live mode: most Ratchet spend on Scout runs per UTC day. Once reached, new runs get 429. Mock mode has no budget | | `RATCHET_SCOUT_RUNS_PER_HOUR` | `6` | Runs per client IP per rolling hour. Mock mode allows 10 × this value | | `RATCHET_SCOUT_MAX_RUNS_PER_DAY` | `30` | Live mode: most runs per UTC day, for all clients together. Caps Tavily credits (about 3 searches per run, so 90 credits a day). Mock mode has no cap | | `RATCHET_SCOUT_MAX_RESULTS` | `4` | Tavily results per query. With 3 queries, at most 12 results are triaged | | `RATCHET_TRUST_PROXY` | `0` | `1`: take the client IP for the per-IP limit from the first `X-Forwarded-For` entry. Only set it when a proxy you control overwrites that header; otherwise clients can fake their IP. Not needed on the VM, where uvicorn runs with `--proxy-headers` | The budget counts the `cost_usd` of every Scout step, including audits and shadow calls. It does not cover `/v1/chat/completions`. In live mode the daily counters are kept in the ledger (`scout_run` rows in `spend`, `usd` 0) and restored at startup, so a restart or `update.sh` does not reset them. They reset at 00:00 UTC. The per-IP windows stay in memory and reset on restart, so no IP is stored. On the first restart after upgrading, runs from earlier that day have no rows and are not counted, once. ### Judge access and Tavily credits Read with `Settings.scout_policy()` in `Runtime.reset`; a bad value stops startup with a `ValueError` naming the variable, and no message ever shows the judge code. | Variable | Default | Meaning | |---|---|---| | `RATCHET_SCOUT_JUDGE_CODE` | empty | The access code for evaluators; empty turns judge access off. 16 to 256 printable ASCII characters with no spaces: use `openssl rand -hex 16`. Set it if and only if `RATCHET_SCOUT_JUDGE_RUNS_PER_DAY` is above 0. Judges send it once (`POST /scout/judge`) and get a signed cookie; never put it in a URL, the project page or the video | | `RATCHET_SCOUT_JUDGE_RUNS_PER_DAY` | `0` | Runs per UTC day held back for judges, 0 to `RATCHET_SCOUT_MAX_RUNS_PER_DAY`. The public gets the run cap minus this; judges get their share plus whatever public room is left. Equal to the run cap: judges only, and the public gets 429 (allowed) | | `RATCHET_SCOUT_JUDGE_BUDGET_USD` | unset | USD per UTC day held back for judges, 0 to `RATCHET_SCOUT_DAILY_BUDGET_USD`. Unset: the daily budget × judge runs / run cap, rounded down to the cent (7 of 22 runs of $3.00: $0.95). Above 0 only with judge access on | | `RATCHET_SCOUT_JUDGE_SESSION_S` | `604800` | How long a judge cookie lasts, 60 s to 30 days (7 days: a judge who comes back does not need the code again). Rotating the code ends every session | | `RATCHET_TAVILY_MONTHLY_CREDITS` | `0` | Tavily credits per cycle (the free plan gives 1,000 a month; the Nebius Builders Program may add more, unverified). `0` counts credits without a cap. A basic search costs 1 credit, an advanced one 2; the meter charges what Tavily reports in `usage.credits` | | `RATCHET_TAVILY_JUDGE_CREDITS` | unset | Credits per cycle held back for judge runs, at most the monthly credits, above 0 only with judge access on. Unset: monthly × judge runs / run cap, rounded down (1,000 × 7 / 22 = 318) | | `RATCHET_TAVILY_CYCLE_DAY` | `1` | Day of the month (1 to 28) the credit cycle starts, at 00:00 UTC. Tavily's own reset day is not documented (unverified): set the day your dashboard shows | | `RATCHET_TAVILY_PACE_UNTIL` | unset | A date (`YYYY-MM-DD`, UTC). While it falls in the current cycle and is today or later, the public's daily credit allowance spreads what is left until that date instead of the cycle's end; unused credits roll forward. Recommended: the last day of judging (the plan assumes `2026-12-15`; unverified) | A public run whose search credits for today are used up still runs: the planner (pinned to the teacher) has already called Token Factory, and the writer answers without web sources, labelled "search unavailable". Credits never refuse a run; the run and USD caps bound the cost. ## Mock mode Read in `ratchet/config.py`. Ignored in live mode. | Variable | Default | Meaning | |---|---|---| | `RATCHET_MOCK_WARMUP_DAYS` | `0` | Mock days of `mixed` traffic, each followed by a night, run at startup and on every mock-mode gateway reset, so the mock-mode gateway opens with a trained student. `4` takes about 2 s and leaves a live student that answers about 80–90% of Scout's LLM calls (mock models). Writes a `warmup` event | | `RATCHET_MOCK_WARMUP_REQUESTS` | `1500` | Requests per warm-up day | | `RATCHET_MOCK_PACE` | `0.25` | `/scout/stream` waits this share of each call's mock latency (mock Ultra 2.4 s → 0.6 s), so the page shows agents working. `0` = no waiting. `/scout/run` never waits | | `RATCHET_MOCK_STRICT` | `0` | `1`: the mock models answer only the mock world's own requests (synthetic tickets and Scout's research calls), and anything else gets 400 `mock mode only answers the mock world's own requests`, as before the generic mock world. `0`: any other request gets a deterministic answer from the generic world (`sim/generic.py`): free text, an instance of its `response_format` schema, or a tool call to the mock teacher. Night School's mock trainer and golden eval read stored rows either way. With `1`, `POST /ratchet/demo/day` refuses the `schema` and `tools` workloads (400), which only the generic world answers | | `RATCHET_MOCK_ENDPOINT_HOURS` | `24` | Hours a day the mock student endpoint runs, 0 to 24: each mock day (the mock-mode gateway, the warm-up, `python -m sim.simulate`) records one endpoint interval of this length from the start of the day, on the synthetic clock, for the cost report ([Cost accounting](#cost-accounting)). `0` = no endpoint (as with the hosted fallback). Anything outside 0 to 24 stops startup. Mock-mode run assumption, not a measurement | | `RATCHET_MOCK_TEACHER_REASONING_MEAN` | unset | The mock teacher's mean billed reasoning tokens. Unset, it bills exactly `RATCHET_TEACHER_REASONING_TOKENS` (300), the formula's assumption, so every number is as before. Set (more than 0, at most 100,000; anything else stops the gateway and `python -m ratchet.live` at startup, in live mode too, where nothing reads it), each request bills round(mean × exp(0.3 z − 0.045)) reasoning tokens, z a standard normal drawn from the request's text (the same request always costs the same), so the savings receipts have a gap between the formula and the bills to measure: with `420`, `ticket-triage`'s calibrated ratio is about 1.3 to 1.4 (mock-mode run; [Savings receipts](#savings-receipts)). The simulator takes `--teacher-reasoning-mean` instead | ## Mock-mode host settings The VM can run a second gateway in mock mode beside the live one ([DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/)). Read in `ratchet/config.py`; a bad value stops startup, naming the variable. The four `RATCHET_DEMO_*` settings act in mock mode only and only when set (a warning in live mode); the compose file sets them on `ratchet-sim`. | Variable | Default | Mode | Meaning | |---|---|---|---| | `RATCHET_LIVE_URL` | empty | both | The live gateway's public URL, for the pages' links to the live gateway (`/ratchet/stats` `links.live`). Empty, or `https://` with a host, at most 300 characters, no user info, no whitespace | | `RATCHET_SIM_URL` | empty | both | The mock-mode run's public URL (`links.sim`). Same rule. On the VM the compose file derives it from `RATCHET_SIM_DOMAIN` unless `.env` sets it (empty: no link) | | `RATCHET_DEMO_RPM` | `0` (off) | mock | Test-control actions (non-GET calls to `/ratchet/demo/*`) a minute per client IP, 0 to 600. Over it: 429 with `Retry-After`. The burst is `rpm // 6`, at least 1 | | `RATCHET_DEMO_MAX_DAYS` | `0` (off) | mock | Reset the shared mock-mode gateway before the day after this many mock days. 0, or above `RATCHET_MOCK_WARMUP_DAYS`, at most 3650. Enforced when a visitor clicks `POST /ratchet/demo/day` (`auto_reset: "max_days"`) | | `RATCHET_DEMO_MAX_REQUESTS` | `0` (off) | mock | Reset past this many ledger requests, 0, or above warm-up days x `RATCHET_MOCK_WARMUP_REQUESTS`, at most 10,000,000. Checked on a click and by a background pass every minute | | `RATCHET_DEMO_IDLE_RESET_S` | `0` (off) | mock | Reset after this many seconds with no mock-mode action and no request while the state differs from the last reset. 0 or 300 to 604,800 | The warm-up rules exist so a cap can never reset the mock-mode gateway right after its own warm-up. An automatic reset runs the warm-up again, writes a `demo_reset` event into the new ledger and clears everything visitors added. ## VM: Caddy (`deploy/vm/`) Read only by Docker Compose and Caddy on the VM ([DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/)), from `deploy/vm/.env`. The gateway does not read them. | Variable | Default | Meaning | |---|---|---| | `RATCHET_DOMAIN` | required | Site name Caddy serves and gets a certificate for, e.g. `ratchet.dualmindlab.tech` or `.sslip.io`. First boot fills in the sslip.io name if it is still the placeholder | | `RATCHET_API_GUARD` | `open` | `open`: every path is public. `guarded`: `/v1/*`, `/openai/*`, `/ratchet/demo/*` and `/ratchet/admin/*` return 403 unless the request has a matching `X-Ratchet-Admin-Token` header (the admin API also needs an admin key, `RATCHET_ADMIN_KEYS`, in live mode). Pages, `/scout*`, `/ratchet/stats` and `/ratchet/bom` stay public at the edge; whether the gateway serves the dashboard API without a key is `RATCHET_PUBLIC_DASHBOARD`. In mock mode, `guarded` also disables the dashboard's test buttons. In live mode without `RATCHET_API_KEYS`, the gateway starts only with `RATCHET_ALLOW_UNAUTHENTICATED=1` | | `RATCHET_ADMIN_TOKEN` | empty | Token for `guarded`. Hex, at least 32 characters (`openssl rand -hex 32`). Shorter or empty blocks the guarded paths for everyone. Caddy strips the header before it reaches the gateway | | `RATCHET_SIM` | `on` | `on`: Caddy serves the `ratchet-sim` container at `RATCHET_SIM_DOMAIN`. `off`: no mock-mode run host (the container still runs, unreachable); then also set an empty `RATCHET_SIM_URL=` so the live pages link nothing. `update.sh` refuses anything else | | `RATCHET_SIM_DOMAIN` | `sim.` | The mock-mode run host's name. It needs DNS: `sim..sslip.io` resolves by itself; with your own domain add an A record (or a wildcard) first. Must differ from `RATCHET_DOMAIN` (`update.sh` refuses it before it builds anything) | | `RATCHET_SIM_BOM_SIGNING_KEY` | empty | Optional key that signs the mock-mode gateway's BOMs, handed to `ratchet-sim` as its `RATCHET_BOM_SIGNING_KEY`. Never the same as the live `RATCHET_BOM_SIGNING_KEY` (`update.sh` refuses it): a mock BOM must not verify against the live public key | `deploy/vm/create_vm.sh` and `deploy/vm/backup.sh` have their own settings, listed at the top of each script and in [DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/). Like the variables above they are in `config.DEPLOY_ENV`, so the gateway's startup check does not call them unknown: | Variable | Default | Meaning | |---|---|---| | `RATCHET_VM_NAME` | `ratchet-host` | The VM's name (`create_vm.sh` refuses a name that exists) | | `RATCHET_VM_PLATFORM` | `cpu-e2` | Nebius platform; `cpu-e2` exists only in `eu-north1`, elsewhere use `cpu-d3` | | `RATCHET_VM_PRESET` | `2vcpu-8gb` | Nebius preset, for example `4vcpu-16gb` | | `RATCHET_VM_DISK_GB` | `30` | Boot disk size in GB | | `RATCHET_SSH_USER` | `ubuntu` | The login user cloud-init creates (Nebius reserves `root` and `admin`) | | `RATCHET_REPO_URL` | `https://github.com/harshbhx/ratchet.git` | What the VM clones into `/opt/ratchet` | | `RATCHET_BACKUP_DIR` | `/var/backups/ratchet` | `backup.sh`: the host folder for ledger backups | | `RATCHET_BACKUP_KEEP` | `14` | `backup.sh`: local copies to keep | | `RATCHET_BACKUP_TAG` | empty | `backup.sh`: 1 to 32 letters, digits or `-`, added to the file name (`ratchet--.db.gz`); `update.sh` sets `pre-update`. Anything else exits 1 | ## Live gateway watchdog (`ratchet/watch.py`) Read on the VM host by `python3 -m ratchet.watch` (standard library only), from `deploy/vm/.env` with the process environment winning, and by the gateway's `python -m ratchet.config check`, which refuses a bad value before `update.sh` replaces the container ([DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/#watchdog)). A bad value names its variable and never echoes a secret. Variables marked secret are never printed, logged or sent. | Variable | Default | Meaning | |---|---|---| | `RATCHET_WATCH` | `on` | `on` or `off`. `off`: every command but `status` prints one line and exits 0 | | `RATCHET_WATCH_CLIENT` | `watchdog` | The `RATCHET_API_KEYS` name the probe uses; the host takes that name's key from `RATCHET_API_KEYS` itself. The pinned task's `clients` list in `deploy/vm/config/tasks.json` must name it | | `RATCHET_WATCH_FAILURES` | `3` | Failed liveness checks, or failed ledger checks, in a row before a restart (2 to 20, one check every 2 minutes) | | `RATCHET_WATCH_RESTART_COOLDOWN_S` | `600` | Seconds after a restart before another (120 to 86,400) | | `RATCHET_WATCH_MAX_RESTARTS_PER_HOUR` | `2` | Restarts in a rolling hour; past it one alert and no restart (1 to 10) | | `RATCHET_WATCH_START_GRACE_S` | `240` | No check counts while the container started less than this ago (0 to 3,600) | | `RATCHET_WATCH_PROBE_TIMEOUT_S` | `200` | The probe's HTTP timeout, above the gateway's own teacher deadline (180 s) so its retries finish first (10 to 900) | | `RATCHET_WATCH_REALERT_S` | `21600` | While a check keeps failing, repeat its alert this often (600 to 604,800) | | `RATCHET_WATCH_CONNECT` | `public` | `public`: the probe resolves `RATCHET_DOMAIN` as anyone would. `local`: it connects to `127.0.0.1:443` with the domain as SNI and `Host` (certificates still checked), for a VM that cannot reach its own public IP. Whether Nebius VMs can (hairpin NAT) is unverified | | `RATCHET_WATCH_HEARTBEAT_URL` | empty | Secret. An `https://` URL the probe GETs after a healthy probe that follows a healthy tick (healthchecks.io or any dead-man's switch: it alerts when the VM, cron or the watchdog is dead, which nothing on the VM can report) | | `RATCHET_WATCH_STATE_DIR` | `/var/lib/ratchet` | State files and locks (`watch-tick.json`, `watch-probe.json`, `watch.paused`, `update.lock`), mode 0600 | | `RATCHET_WATCH_FLAG_MAX_AGE_S` | `10800` | Gateway: a banner flag older than this reads `stale` and shows nothing (600 to 604,800) | | `RATCHET_ALERT_WEBHOOK_URL` | empty | Secret. `https://` only. Alerts are POSTed here as JSON with a `text` field (Slack's incoming-webhook shape; whether Slack ignores the other keys, and Discord's `/slack` suffix, are unverified: `python3 -m ratchet.watch test-alert` is the check). Empty: alerts go to `/var/log/ratchet/watch.log` only | | `RATCHET_ALERT_WEBHOOK_SECRET` | empty | Secret. When set, each alert carries `X-Ratchet-Signature: sha256=` | | `RATCHET_ALERT_KINDS` | `alert,rollback,train_failed,circuit,budget,admin,task_disabled,probe,watchdog_restart` | Kinds to send; an unknown kind is an error naming it. The watchdog sends only `probe` and `watchdog_restart` | | `RATCHET_INSTANCE_NAME` | the host name | `gateway` in the alert body | | `RATCHET_WATCH_KEY` | empty | Secret. `scripts/demo_check.py --chat` only: the `watchdog` key for its one live completion (from the environment, never a flag) | | `RATCHET_DEMO_CHECK_KEY` | empty | Secret. `scripts/demo_check.py` only: the key for its free `GET /v1/models` check (the judge key's), unless `--key-env` names another variable | The probe also reads `RATCHET_MODE`, `RATCHET_DOMAIN`, `RATCHET_API_GUARD`, `RATCHET_ADMIN_TOKEN`, `RATCHET_API_KEYS` and `RATCHET_TEACHER_FALLBACK_MODEL` from the same file. ## Live Night School (`ratchet/live.py`) | Variable | Default | Meaning | |---|---|---| | `RATCHET_DATA_DIR` | `data` | Local folder synced to the bucket, laid out as the training job sees the bucket at `/data`: `night/train.jsonl`, `boms/student-vN.json`, and `adapters//ratchet_bom.json` and `NOTICE.md` (written after a night that trained, next to where the job wrote the weights; an adapter name that is not one folder name gets the `boms/` copy only). `python -m ratchet.live export` copies `adapters//` from here when the weights are here too (a bucket mount, or synced down) | | `RATCHET_SYNC_CMD` | empty | Command that uploads `RATCHET_DATA_DIR` to the bucket, e.g. `aws s3 sync ./data s3://my-bucket --endpoint-url https://storage.eu-north1.nebius.cloud` (skipped if empty). Run before the training job and again after a night that trained, so the adapter's BOM and NOTICE.md reach its bucket folder; a failure of the second run is a `sync_failed` event and a non-zero exit. It must only upload: an option that deletes remote files missing locally (`--delete`) would delete the adapter weights | | `RATCHET_TRAIN_CMD` | `bash deploy/train_job.sh {version}` | Command that runs training and **waits** for it. `{version}` and `{data}` are substituted | | `RATCHET_ADAPTER_ROOT` | `/data/adapters` | Adapter folder as the vLLM endpoint sees it (`lora_path` = `/`). Also read by the gateway in live mode, which reloads an adapter from there when vLLM answers 404 for it | Commands are split with `shlex.split` and run without a shell, so pipes and `&&` do not work. Wrap them in a script. ## Live-run harness `python -m ratchet.liverun` ([RUNBOOK](/reference/runbook/#live-measurement-run)) runs on the gateway's host with the gateway's `.env` (it reads the same `RATCHET_DB`, task registry, audit rates, release policy and prices), plus these variables, which only the harness reads. Its budget is a flag (`--max-usd`), never a variable. | Variable | Default | Meaning | |---|---|---| | `RATCHET_LIVERUN_KEY` | empty (required) | The harness's gateway API key: the key of the `liverun:` entry in `RATCHET_API_KEYS` (`--client` names another entry). Export it in the harness's shell only; never a flag, so it stays out of shell history and the process list, and it is never written or printed (results carry the name) | | `NEBIUS_API_KEY_LIVERUN` | empty (`NEBIUS_API_KEY`) | A separate Token Factory key for the harness's own arms (the control arm and the untuned-Nano arm), so a runaway run cannot drain the key the gateway and the judge page serve with. The gateway's own teacher calls keep `NEBIUS_API_KEY`. Results record which variable was used, never its value | | `RATCHET_GIT_SHA` | empty | The code's commit when the image has no `.git` (`git rev-parse HEAD` comes first). 40 or 64 hex characters; anything else is ignored. Results carry a SHA-256 over the `ratchet` and `sim` sources either way | The run's planned gate profile is set on the gateway, not here: `RATCHET_SEQUENTIAL=1` and `RATCHET_CANARY_AUDIT_RATE=0.5` ([Release policy](#release-policy-ratchetreleasepy)). `python -m ratchet.liverun run` refuses to start without them unless `--gate-profile as-set` is passed. ## Deploy scripts (`deploy/`) Read only by `deploy/train_job.sh`, `deploy/student_endpoint.sh` and the one-command script `deploy/nebius/nebius.sh` ([WHERE_IT_RUNS.md](/reference/where-it-runs/#one-command-to-host-it-all-on-nebius)) on the machine that runs the `nebius` CLI, never by the gateway (`config.DEPLOY_ENV` lists them so the traceability check holds each to a row here). The Nebius commands follow the CLI reference (auto generated on 19-Oct-2026, read Oct 21, 2026); [DEPLOYMENT.md](/reference/deployment/#step-4-student-endpoint-vllm-with-runtime-lora) has the actions. | Variable | Used by | Default | Meaning | |---|---|---|---| | `DRY_RUN` | both | `0` | `1`: `train_job.sh` and `student_endpoint.sh create` pass `--dry-run` (nebius validates the request and creates nothing); `start` and `stop`, which have no `--dry-run`, print the command instead of running it; the read-only `nebius ai endpoint list` still runs, and no uptime is recorded. Any value but `0` or `1` is refused (exit 2), so `DRY_RUN=true` never starts a paid GPU | | `RATCHET_BUCKET_ID` | both | required (job; endpoint `create`) | Nebius Object Storage bucket ID, mounted at `/data` | | `RATCHET_SUBNET_ID` | both | empty (the project's default subnet) | `--subnet-id` for the job and for `create` | | `RATCHET_TRAIN_IMAGE` | `train_job.sh` | required | Training image built from `deploy/Dockerfile.train` | | `RATCHET_WHEELS_DIR` | `install_kernels.sh` (inside the training job) | `/data` | Where the prebuilt Mamba kernel wheels are cached, as `wheels-/` on the bucket mount | | `RATCHET_TRAIN_PLATFORM` | `train_job.sh` | `gpu-l40s-a` | Nebius GPU platform | | `RATCHET_TRAIN_PRESET` | `train_job.sh` | `1gpu-8vcpu-32gb` | Nebius preset | | `RATCHET_TRAIN_ON_DEMAND` | `train_job.sh` | `0` | `1` submits the training job on demand; the default is a spot job (`--preemptible --follows-spot-price`). Anything but `0` or `1` is refused. Spot jobs can be preempted: the 28 Oct 2026 run (evidence/nightschool-nebius/) used spot throughout | | `RATCHET_REGISTRY_SECRET` | `train_job.sh` | empty | `--registry-secret`: a MysteryBox secret with `REGISTRY_USERNAME` and `REGISTRY_PASSWORD`, for a private training image (Nebius Container Registry) | | `RATCHET_BUCKET_S3` | `train_job.sh` | empty | `s3://bucket[/prefix]`. When set, the script (and the night, before it writes anything) needs the `aws` CLI on PATH and stops with a message if it is missing; it then polls for `adapters//adapter_model.safetensors` (every 30 s, up to 2 h, the job's `--timeout`) so it can be used as `RATCHET_TRAIN_CMD` | | `RATCHET_S3_ENDPOINT` | `train_job.sh` | empty | S3 endpoint URL for Nebius Object Storage | | `RATCHET_STUDENT_KEY` | `student_endpoint.sh` | required (`create`) | Token clients must send to the endpoint. Passed to `--token`, never printed (`` in the echoed command) | | `RATCHET_ENDPOINT_NAME` | `student_endpoint.sh` | `ratchet-student` | Name `create` gives the endpoint, and the name `start`, `stop` and `status` look up in `nebius ai endpoint list --format json` to find its id | | `RATCHET_ENDPOINT_ID` | `student_endpoint.sh` | empty | The endpoint's id (`metadata.id`). When set, `start` and `stop` use it without the list lookup, and `status` looks this id up instead of the name. Set it when the lookup cannot decide (two endpoints with the name, or a list with more pages) | | `RATCHET_PYTHON` | `student_endpoint.sh` | `python` | Python 3 that reads the endpoint list (standard library only, `-I`) and runs `python -m ratchet.cost endpoint up\|down` after `create`, `start` and `stop` to record the endpoint's uptime in the ledger at `RATCHET_DB` (best effort: when recording fails, the deploy still succeeds and prints the command to run where the ledger is) | | `RATCHET_SERVE_PLATFORM` | `student_endpoint.sh` | `gpu-l40s-a` | GPU platform for vLLM | | `RATCHET_SERVE_PRESET` | `student_endpoint.sh` | `1gpu-8vcpu-32gb` | Preset for vLLM | | `RATCHET_VLLM_IMAGE` | `student_endpoint.sh` | `vllm/vllm-openai:v0.18.0-cu130` | vLLM image. `vllm/vllm-openai:v0.18.0` is the CUDA 12 build of the same release (both on Docker Hub, pushed Mar 20, 2026), for a host driver too old for CUDA 13 | | `RATCHET_SERVE_MAX_LEN` | `student_endpoint.sh` | `8192` | vLLM `--max-model-len`. The model card's example uses 262,144 tokens; ticket prompts need far less, and a longer prompt gets vLLM's 400 and escalates to the teacher | | `RATCHET_SERVE_MAX_SEQS` | `student_endpoint.sh` | `64` | vLLM `--max-num-seqs` (concurrent sequences) | | `RATCHET_SERVE_TEMPLATE_KWARGS` | `student_endpoint.sh` | `{"enable_thinking":false}` | vLLM `--default-chat-template-kwargs` (a JSON object, checked before `create`; no single quote), so the student answers without reasoning unless a request asks. Set but empty to leave the flag out (requests then carry their own `chat_template_kwargs`) | | `RATCHET_REASONING_PARSER_PLUGIN` | `student_endpoint.sh` | empty | Absolute path in the container of the model card's `nano_v3_reasoning_parser.py` (for example `/data/nano_v3_reasoning_parser.py`, uploaded to the bucket). When set, vLLM gets `--reasoning-parser-plugin --reasoning-parser nano_v3` | | `RATCHET_STUDENT_BASE_MODEL` | both | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | Base model | | `RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY` | `nebius.sh` | unset | Must be `1`, together with `--yes`, for `up` or `down` to create or delete anything; without both, those commands are a dry run that calls no Nebius API. `plan` and `status` ignore it | | `RATCHET_TRAIN_VERSION` | `nebius.sh` | `student-v1` | Version name of the training job `up --with-training` submits through `train_job.sh` | | `RATCHET_SMOKE_CMD` | `nebius.sh` | empty (`scripts/demo_check.py`) | One executable called as ` https:// --mode mock\|live` for the smoke check after `up` (tests set a stub) | ## Prices (`DEFAULT_PRICES`) USD per 1M tokens (input / output). These are Token Factory list prices as read in October 2026 and checked on 29 Oct 2026 (`scripts/day1_check.py` passed); verify them in the console. | Key | Input | Output | Note | |---|---|---|---| | `nvidia/Nemotron-3-Ultra-550b-a55b` | 1.00 | 3.00 | Teacher | | `nvidia/nemotron-3-super-120b-a12b` | 0.30 | 0.90 | Judge | | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` | 0.06 | 0.24 | Fallback student | | `nvidia/Nemotron-3_5-Lightning` | 0.06 | 0.24 | | | `student` | 0.06 | 0.24 | Per-token **proxy** for the self-hosted student; real cost is GPU-hours ([Cost accounting](#cost-accounting)) | Embeddings models (`EMBEDDING_PRICES`, kept apart because they are never teachers and write no training data; merged into `Settings.prices`): | Key | Input | Output | Note | |---|---|---|---| | `Qwen/Qwen3-Embedding-8B` | 0.01 | 0 | `RATCHET_EMBEDDINGS_MODEL`'s default ([Other OpenAI endpoints](#other-openai-endpoints)) | `Settings.price(model)` checks for an exact key first, then uses `student` for any name starting with `student`. Anything else gets the teacher's price (or 1.00 / 3.00). To change prices, pass `Settings(prices={...})` in code; there are no price environment variables. A fallback teacher's answers (`RATCHET_TEACHER_FALLBACK_MODEL`) are priced with its own entry (the teacher's price when it has none); the all-teacher baseline of such a request is the teacher's price for the same tokens. ## Cost accounting Read in `ratchet/config.py` (`Settings.cost_policy()`); used by `ratchet/cost.py`, the gateway (`GET /ratchet/cost`, the `cost` block of `/ratchet/stats`, `POST /ratchet/cost/endpoint`), `python -m ratchet.cost`, `python -m ratchet.live night` and the simulator ([ARCHITECTURE.md](/reference/architecture/#7-cost-accounting)). A bad value (negative, not a number, above $1,000 per GPU-hour or 1,024 GPUs, not a whole number of GPUs, an unknown model) stops startup with `ValueError` naming the variable; the CLI, `POST /ratchet/cost/endpoint` and the ledger refuse the same values. Each interval stores the rate it was recorded at, so changing a rate later prices new intervals only; the [break-even](https://docs.ratchet.dualmindlab.tech/guides/glossary/#break-even) and the hosted-fallback crossover use the rate the window's intervals recorded (`endpoint_usd_per_hour`), and the configured student rate only while no uptime is recorded. The crossover compares price per answer with the hosted fallback (`RATCHET_FALLBACK_STUDENT`) and assumes that untuned model serves as many requests as the tuned student; the report says so. | Variable | Default | Meaning | |---|---|---| | `RATCHET_STUDENT_COST_MODEL` | `per_token` | Which savings the cost report's `headline` (and `/ratchet/stats` `cost.model`) shows: `per_token` (the student priced per token at the hosted Nano price, as every figure since v0.1) or `gpu_hours` (the GPU-hours the student endpoint and the training jobs ran). The report always has both views, and each request's `cost_usd` stays per token in both, so history never moves. Case and spaces are ignored; empty means `per_token` | | `RATCHET_STUDENT_GPU_USD_PER_HOUR` | `1.55` | USD per GPU-hour of the student endpoint (default 1.55, the list price this setting was first written against; Nebius's pricing page lists an L40S at $1.35 on demand, [docs.nebius.com/compute/resources/pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026, so set 1.35 to match it; **unverified** until the day-1 checks pass; 0 to 1,000). The default for `python -m ratchet.cost endpoint up` and `POST /ratchet/cost/endpoint` when they name no price, for the simulator's intervals, and the price in the break-even | | `RATCHET_STUDENT_GPUS` | `1` | GPUs the student endpoint runs on (0 to 1,024) (`1gpu-8vcpu-32gb` in `deploy/student_endpoint.sh`). The endpoint costs this many times the per-GPU price per hour | | `RATCHET_TRAIN_GPU_USD_PER_HOUR` | `1.55` | USD per GPU-hour of a training job (the same L40S price: default 1.55, $1.35 on Nebius's pricing page; unverified). Prices the training interval `python -m ratchet.live night` records around each training run, and the simulator's | | `RATCHET_TRAIN_GPUS` | `1` | GPUs a training job runs on (`RATCHET_TRAIN_PRESET`) | | `RATCHET_PRICES_VERIFIED` | `0` | `1` once `scripts/day1_check.py` passed with a real key (it did on 29 Oct 2026, for the per-token prices) and the GPU rates above match your Nebius bill. Until then every report labels its prices "unverified list prices" (invariant 9); with `1` it says "prices confirmed by the operator". Ratchet never sets it by itself | The mock endpoint's hours a day are `RATCHET_MOCK_ENDPOINT_HOURS` ([Mock mode](#mock-mode)). ## Savings receipts Read in `ratchet/config.py` (`Settings.receipt_policy()`); used by `ratchet/receipts.py`, the gateway's [receipt](https://docs.ratchet.dualmindlab.tech/guides/glossary/#receipt) endpoints and its mock days, `python -m ratchet.live seal`, `python -m ratchet.receipt seal` and `python -m sim.simulate --receipts` ([API](/reference/api/#savings-receipts)). A bad value stops startup with `ValueError` naming the variable, because a sealed [receipt](https://docs.ratchet.dualmindlab.tech/guides/glossary/#receipt) can never be corrected (invariant 17). Receipts are signed with `RATCHET_BOM_SIGNING_KEY` ([Privacy and security](#privacy-and-security)) when it is set; without it they carry their sha256 only. | Variable | Default | Meaning | |---|---|---| | `RATCHET_TAKE_RATE` | `0.25` | The fee's share of a receipt's billable lower bound (after spend outside requests), from 0 to 1. Stated in every receipt as a decimal string (`billing.share`). The fee is rounded down to a nano-USD and is 0 when the bound is negative or no task is calibrated | | `RATCHET_RECEIPT_MIN_AUDITS` | `30` | Audited answers a task needs, in one receipt, before its baseline is calibrated and its served answers are billed; a whole number from 2 to 1,000,000. Below it the task is `insufficient_audits`: its savings are reported at the formula and not billed. At a 10% audit rate, 30 audits take about 300 student-served answers of the task in a day (per client, for a client's receipt) | A day is sealed once it is over: on a live ledger 5 minutes after it ended (`receipts.SEAL_GRACE_S`; a live ledger's day 1 starts at its first request's time, not at 00:00 UTC, so run `python -m ratchet.live seal` hourly), on a mock one when the next day starts. The mock teacher's reasoning (`RATCHET_MOCK_TEACHER_REASONING_MEAN`) is under [Mock mode](#mock-mode). ## Certified precision (`Settings.cert_policy()`) How Night School's [golden eval](https://docs.ratchet.dualmindlab.tech/guides/glossary/#golden-set) picks each version's threshold and what its [certificate](https://docs.ratchet.dualmindlab.tech/guides/glossary/#certificate) claims. The gateway (mock mode's golden eval) and `python -m ratchet.live night` read them; a bad value stops startup with a `ValueError` naming the variable. The simulator takes `--calibration`, `--cert-target` and `--cert-delta` instead. The gate's opt-in rules that read certificates are in the [release policy](#release-policy-ratchetreleasepy). | Variable | Default | Meaning | |---|---|---| | `RATCHET_CALIBRATION` | `point` | How the threshold is chosen on the calibration half of the golden rows. `point`: the lowest threshold whose calibration answers reach 97% precision (v0.4's rule; routing and the reference runs do not move). `ltt`: Learn-then-Test over the fixed grid 0.95, 0.90, …, 0.50, keeping the lowest threshold whose precision is above `RATCHET_CERT_TARGET` with probability at least 1 − `RATCHET_CERT_DELTA` (1.01, serve nothing, when the highest fails). Case is ignored; empty means `point` | | `RATCHET_CERT_TARGET` | `0.95` | The precision a certificate must show (`certified` when the lower bound reaches it), and Learn-then-Test's target. Strictly between 0 and 1 | | `RATCHET_CERT_DELTA` | `0.05` | The certificate's error rate: the one-sided Clopper–Pearson bound holds with probability at least 1 − delta (the frontier's rows at delta / 10 each). Strictly between 0 and 1 | Golden rows a target needs (with 0, 1, 2 or 3 wrong answers among the served [held-out rows](https://docs.ratchet.dualmindlab.tech/guides/glossary/#held-out); `python -m ratchet.stats plan --target T --delta D`): 95% at delta 0.05 needs 59, 93, 124 or 153 served rows; 97% needs 99, 157, 208 or 257; 90% needs 29, 46, 61 or 76. Only the [held-out](https://docs.ratchet.dualmindlab.tech/guides/glossary/#held-out) half counts, so the golden set must be about twice that, divided by the coverage. ## Ledger and hot path How the ledger file is written and how often the request path does periodic work ([ARCHITECTURE §11](/reference/architecture/#11-hot-path-and-performance)). A bad value stops startup with a `ValueError` naming the variable (the gateway, `python -m ratchet.live` and `python -m ratchet.cost`). | Variable | Default | Meaning | |---|---|---| | `RATCHET_DB_SYNC` | `NORMAL` | How a ledger file is journaled, `NORMAL` or `FULL` (any case; empty means `NORMAL`). `NORMAL`: the WAL journal with `synchronous=NORMAL`. Readers never wait for the writer, the dashboard reads on a second, read-only connection, and a commit does not wait for the disk: a power cut or an operating-system crash (not a crash of the gateway process) can lose the last commits, never corrupt the file. `FULL`: the rollback journal with `synchronous=FULL`, v0.4's behaviour, for a ledger on a filesystem without shared memory (a network or bucket mount, where WAL does not work) or when every commit must reach the disk before the client gets its answer. Give the gateway, the cron jobs and the cost CLI the same value: `FULL` cannot take a file out of WAL while another process has it open, and stops startup instead. Mock mode's in-memory ledger has no journal to set (a bad value still stops startup) | | `RATCHET_TICK_INTERVAL_S` | unset: `5` in live mode, `0` in mock mode | Seconds between two release-gate ticks run from the request path (0 to 3600). A tick less than this after the previous one, or while another thread is ticking, is skipped, so requests pay for a tick at most once per interval and never wait for one. A rollback can wait this long; the cron `python -m ratchet.live tick` runs regardless. `0` ticks after every request (mock mode, so test runs and tests behave as before); the mock-mode gateway's mock days always tick | | `RATCHET_STATS_TTL_S` | unset: `2` in live mode, `0` in mock mode | Seconds `/ratchet/stats` serves one computed payload (0 to 3600). However many dashboards poll, it is computed at most once per TTL, by one caller at a time; the others get the previous payload. `0` computes it on every read | ## Off-path quality checks Where a request's teacher audit, shadow samples and [arbitration](https://docs.ratchet.dualmindlab.tech/guides/glossary/#arbitration) run ([ARCHITECTURE §15](/reference/architecture/#15-off-path-quality-checks)). With workers, a request the gateway serves returns as soon as its answer is ready, and a bounded sidecar finishes its ledger row afterwards; Scout's agents, the simulator and the mock warm-up keep every check on the request path in any case. Read in `ratchet/config.py` (`Settings.sidecar`); a bad value stops startup naming the variable. | Variable | Default | Meaning | |---|---|---| | `RATCHET_SIDECAR_WORKERS` | unset: `4` in live mode, `0` in mock mode | Threads that run requests' audits and shadows after the answer is sent, 0 to 64. `0`: every check stays on the request path, as before (mock mode's default, so test runs and the reference runs are unchanged). Each worker holds at most one teacher call at a time (an audit, or an arbitration bounded to 60 s), so more workers spend the teacher's rate limit faster, never more in total | | `RATCHET_SIDECAR_QUEUE` | `1000` | Jobs that may wait for a worker, 1 to 100,000. A tenth of the places (at least one, once there are two) is kept for jobs that carry an audit. Each job holds its request's messages until it ends, so the request text queued and running jobs hold is bounded too, at 64 MiB (`sidecar.MAX_BYTES`, a tenth of it also kept for audits; at `RATCHET_MAX_INPUT_CHARS`' default a job holds at most about 0.5 MB): past it the queue counts as full. A drawn audit is never dropped: with every place taken it runs on the request path instead (`audits_inline`). A shadow without a place does not run for that request (`shadows_dropped`), counted per version in `/ratchet/stats` `sidecar` and in `ratchet_sidecar_full_total` | | `RATCHET_SIDECAR_DRAIN_S` | `10` | Seconds the gateway's shutdown waits for queued jobs, 0 to 3600, after the server stopped taking requests. Jobs still queued then never run: their rows stay pending (`audited` 1 with `audit_agree` NULL, `shadow_version` with `shadow_serve` NULL), which the release gate ignores, and one `sidecar_dropped` event counts them per version. Keep it under the process manager's stop timeout: Docker sends SIGKILL 10 s after SIGTERM by default; the VM's Compose file gives the gateway 20 s (`stop_grace_period`), so raise both together | While a request's checks are pending, its row holds what serving cost (`cost_usd` and its split), the token formula's baseline and `audited` 1 with `audit_agree` NULL; the job adds the audit's, the shadow's and the arbitration's cost call by call, writes the verdicts and, after a teacher audit, the baseline the audit prices, so the finished row is the one an inline run writes. The response says so with `ratchet.overhead_pending: true` ([API.md](/reference/api/#off-path-checks)). ## Observability Logs, metrics and request ids ([API.md](/reference/api/#request-ids), [RUNBOOK.md](/reference/runbook/#failures-and-request-ids)). Read in `ratchet/config.py`; used by `ratchet/obs.py` and `ratchet/gateway.py`. A bad value stops startup with a `ValueError` naming the variable. | Variable | Default | Meaning | |---|---|---| | `RATCHET_LOG_FORMAT` | unset: `text` in mock mode, `json` in live mode | How the `ratchet` logger writes to stderr: `json` (one object per line: `ts`, `level`, `logger`, `event`, then the fields) or `text` (` key=value ...`), any case. Lines carry ids, names, numbers and statuses, never prompt or answer text, URLs or keys | | `RATCHET_LOG_LEVEL` | unset: `WARNING` in mock mode, `INFO` in live mode | `DEBUG`, `INFO`, `WARNING` or `ERROR`, any case. `INFO` writes one `chat` line per chat request (request id, client request id, trace id, client name, task, served_by, route, version, confidence, escalated, audited, latency_ms, wall_ms, cost_usd, status; `retry_count` on a retry, `idempotent_replay` and `replay_of` on an answer sent again for an `Idempotency-Key`, whose `cost_usd` is 0, and `overhead_pending` when its checks run after the answer), one `offpath` line when those checks end (row, audited, audit_agree, shadow_version, shadow_serve, shadow_agree, stored, the `cost_usd` they added, `pending_ms`; the same request id, so the two lines' costs add up to the row's) and every release-log event (`ledger_event`, scrubbed of URLs and error messages); `WARNING` keeps the `upstream_error` lines (role, exception class, status), so mock mode and the tests stay quiet | | `RATCHET_METRICS_TOKEN` | empty | A bearer token that reads `GET /metrics` (and nothing else), for a Prometheus scraper, besides the admin and dashboard keys. At least 16 characters; it may not also be an API, admin or dashboard key (startup error). Compared as a constant-time digest, never stored or logged. Empty: only admin and dashboard keys read a private `/metrics`. A public dashboard (`RATCHET_PUBLIC_DASHBOARD=1`) makes `/metrics` public too | Keep `RATCHET_METRICS_TOKEN` out of git, like the keys. ## Configuration checks The gateway checks every setting before it opens the ledger or calls anything: `Runtime` stops with one `ConfigError` (a `ValueError`, also a `LicenseError` when a teacher is refused) that lists **every** problem, each naming its variable, never a key's value. `python -m ratchet.config check` runs the same check without starting anything (plus `RATCHET_TASKS_FILE`, which the gateway checks itself when it builds the registry), so it exits 2 exactly when the gateway would not start, and exits **0** when clean, **1** with warnings only, **2** with errors (or with warnings under `RATCHET_STRICT_CONFIG=1`); `--json` prints `{"mode", "strict", "exit", "issues": [{"level", "var", "message"}]}`. `deploy/vm/update.sh` runs it in the new image before it replaces the running container, and CI runs it on the defaults. | Variable | Default | Meaning | |---|---|---| | `RATCHET_STRICT_CONFIG` | unset (off) | `1`: a warning stops startup like an error, and `python -m ratchet.config check` exits 2 on it. `0` or unset: warnings start the gateway; each is kept in `Runtime.config_warnings`, logged once as a `config_warning` line (the variable and the message, never the value) and counted under `checks.config` in `/readyz`. Anything but 0 or 1 is an error | Errors (the gateway does not start): | Rule | Variables | |---|---| | A number, whole number or 0/1 switch that does not parse (`5%`, `three`, `yes`) | every numeric setting and `RATCHET_PUBLIC_DASHBOARD`, `RATCHET_PUBLIC_BOM`, `RATCHET_SCRUB_STATS`, `RATCHET_OPENAPI`, `RATCHET_ALLOW_UNAUTHENTICATED`, `RATCHET_STRICT_CONFIG` | | `mock` or `live` (an empty or misspelled mode used to start live mode) | `RATCHET_MODE` | | From 0 to 1 | `RATCHET_AUDIT_RATE`, `RATCHET_CONFIDENCE_THRESHOLD` | | A whole number from 1 to 16 | `RATCHET_SAMPLES` | | More than 0 and at most 0.5 (more would hold most examples out of training) | `RATCHET_GOLDEN_FRACTION` | | 0 or more | `RATCHET_API_KEY_DAILY_REQUESTS`, `RATCHET_MAX_BODY_BYTES`, `RATCHET_TEACHER_REASONING_TOKENS`, the Scout limits and budget, `RATCHET_MOCK_WARMUP_DAYS`, `RATCHET_MOCK_WARMUP_REQUESTS`, `RATCHET_MOCK_PACE` | | A whole number from 0 to 1,000,000; seconds from 0 to 86,400 ([Retries and idempotency](#retries-and-idempotency)) | `RATCHET_IDEMPOTENCY_CACHE`; `RATCHET_IDEMPOTENCY_TTL_S`, `RATCHET_SEEN_DEDUP_S` | | A whole number from 0 to 64; from 1 to 100,000; seconds from 0 to 3600 ([Off-path quality checks](#off-path-quality-checks)) | `RATCHET_SIDECAR_WORKERS`; `RATCHET_SIDECAR_QUEUE`; `RATCHET_SIDECAR_DRAIN_S` | | An `http://` or `https://` URL with a host (the message never repeats the URL, which may hold a password) | `RATCHET_TEACHER_BASE` (required), `RATCHET_STUDENT_BASE`, `RATCHET_TEACHER_FALLBACK_BASE` (empty is fine) | | Distillable under the license registry | `RATCHET_TEACHER_MODEL`, `RATCHET_JUDGE_MODEL`, `RATCHET_TEACHER_FALLBACK_MODEL` | | Each setting's own rules, as before, now listed together (a rule that covers several variables, such as the cost settings, reports each bad one) | timeouts, retries, breakers, `RATCHET_ON_TEACHER_FAILURE`, `RATCHET_UNKNOWN_TASKS`, the request limits, the parameter allowlists and token caps, the cost settings, `RATCHET_DB_SYNC`, the intervals, the log settings, the access switches and `RATCHET_CSP`, the key rings and `RATCHET_METRICS_TOKEN`, the [reserved settings](#reserved-settings) | | The base64 of a 32-byte Ed25519 seed, with the `sign` extra installed, when set | `RATCHET_BOM_SIGNING_KEY` | | Live mode: a key for a Token Factory teacher (a teacher on another host may run without one: a warning) | `NEBIUS_API_KEY` | | Live mode: API keys, unless something in front checks callers | `RATCHET_API_KEYS` (or `RATCHET_ALLOW_UNAUTHENTICATED=1`) | | Live mode: a ledger file that can be written (not a directory, its directory exists and is writable) | `RATCHET_DB` | | Live mode: every `RATCHET_` parses and is in its range, each reported on its own (`ReleasePolicy.env_issues`); then 0 < `RATCHET_LIVE_FLOOR` <= `RATCHET_MIN_PRECISION` <= 1; `RATCHET_MIN_AUDITS` and `RATCHET_MIN_SHADOW` at least 1; `RATCHET_TOLERANCE` from 0 to 0.2 (`ReleasePolicy.validate`) | the [release policy](#release-policy-ratchetreleasepy) | Warnings (the gateway starts, unless `RATCHET_STRICT_CONFIG=1`): | Rule | Variables | |---|---| | A `RATCHET_*` variable Ratchet does not read, with the closest known name (`RATCHET_AUDIT_RAT`: did you mean `RATCHET_AUDIT_RATE`?). The deploy scripts' and the VM's own variables (`RATCHET_DOMAIN`, `RATCHET_API_GUARD`, `RATCHET_ADMIN_TOKEN`, ...) are known, because Docker Compose hands the VM's whole `.env` to the gateway | any | | A model without a price in `DEFAULT_PRICES`: its calls are costed as another model's | `RATCHET_TEACHER_MODEL`, `RATCHET_JUDGE_MODEL`, `RATCHET_TEACHER_FALLBACK_MODEL`, and `RATCHET_FALLBACK_STUDENT` in live mode without a student endpoint | | Live mode: a student endpoint without a token | `RATCHET_STUDENT_KEY` | | Live mode: an audit rate under 1%, at which a canary may never collect `RATCHET_MIN_AUDITS` audits | `RATCHET_AUDIT_RATE` | | Live mode: a variable only mock mode reads, set to anything but its default | `RATCHET_MOCK_WARMUP_DAYS`, `RATCHET_MOCK_WARMUP_REQUESTS`, `RATCHET_MOCK_PACE`, `RATCHET_MOCK_ENDPOINT_HOURS`, `RATCHET_MOCK_TEACHER_REASONING_MEAN` | | A 0/1 switch spelled otherwise (`true` reads as off, `false` as on for redaction) | `RATCHET_TRUST_PROXY`, `RATCHET_MERGE_SYSTEM`, `RATCHET_STORE_FALLBACK_TEACHER`, `RATCHET_PRICES_VERIFIED`, `RATCHET_REDACT_PII` | `config.KNOWN_ENV` lists every variable the package reads; `python scripts/reqtrace.py --check` fails when code reads one that is not listed (or lists one nothing reads), so a new variable goes in `KNOWN_ENV` and in this file. Variables the deploy scripts read go in `config.DEPLOY_ENV`. ## Success metrics Read in `ratchet/config.py` (`Settings.kpi_window()`); used by `ratchet/kpis.py` through `GET /ratchet/kpis`, the `kpis` block of `/ratchet/stats` and the simulator. success metric 2 needs the release gate's `RATCHET_MIN_AUDITS` audits in the window before it reports met or not met. | Variable | Default | Meaning | |---|---|---| | `RATCHET_KPI_WINDOW_DAYS` | `3` | The steady state the success metrics describe: success metrics 1 to 3 and the latency target cover the last this many ledger days with traffic (days without requests are skipped). A whole number from 1 to 3,650; anything else stops startup with `ValueError` naming the variable. `GET /ratchet/kpis?days=N` overrides it for one report | ## Teacher watch (`Settings.teacher_watch_limits()`) How much paid teacher work Night School spends each night checking the teacher. The checks that come free with traffic (a stored request's first re-test in each check epoch) need no setting. Read by the gateway (mock nights), the simulator and `python -m ratchet.live night`; a value that is not a whole number from 0 to 10,000 stops startup with a `ValueError` naming the variable. The freeze rule and the switch are [release policy](#release-policy-ratchetreleasepy) fields (`teacher_noise_max`, `alpha_teacher`, `teacher_watch`). | Variable | Default | Meaning | |---|---|---| | `RATCHET_TEACHER_PROBES` | `20` | Stored labels Night School asks the teacher about again each night (one call each, as an audit asks: temperature 0, under the example's stored conditions), so traffic with few repeats is checked too. Only labels that can be checked: the configured teacher's, no redaction placeholders, a task that names fields and is not pinned, not contested, not re-tested yet in the current epoch; in hash order. Each one is that request's first re-test of the epoch. 0: none | | `RATCHET_ADJUDICATE_MAX` | `50` | Contested labels Night School adjudicates each night, two more teacher answers each: a strict majority of the three becomes the label and ends the quarantine; without one the label stays quarantined and is tried again after the labels never adjudicated. 0: none (contested labels stay out of training) | At the defaults a night makes at most 120 teacher calls (20 probes, 50 adjudications of two calls each); at the teacher's list price (unverified) and about 200 tokens in and 300 out a call, that is about $0.13. The night's `teacher_watch` event states what it spent when anything happened; the cost report does not count it yet. ## Release policy (`ratchet/release.py`) `ReleasePolicy` defaults. The gateway (both modes) and `ratchet.live` read each field from the variable in the second column (`ReleasePolicy.from_env()` reads `RATCHET_` for every field). Integers stay integers, the rest are floats. Mock mode uses `min_audits=15` unless `RATCHET_MIN_AUDITS` is set, so mock mode can show the [anytime-valid](https://docs.ratchet.dualmindlab.tech/guides/glossary/#anytime-valid) gate with `RATCHET_SEQUENTIAL=1`. The simulator ignores them and takes its own flags (`--min-audits`, `--release`, `--promote-evidence`, `--canary-audit-rate`, `--paired-gate`, `--min-certified`, `--drift-alarm`, `--drift-boost`, `--drift-early-night`, `--teacher-watch`; `--teacher-noise` and `--teacher-noise-day` make its mock teacher unstable). | Field | Variable | Default | Used in | |---|---|---|---| | `min_precision` | `RATCHET_MIN_PRECISION` | 0.95 | Golden gate and shadow: minimum precision of answers the candidate would serve | | `min_coverage` | `RATCHET_MIN_COVERAGE` | 0.25 | Golden gate: minimum share of golden traffic it would serve | | `tolerance` | `RATCHET_TOLERANCE` | 0.02 | Max allowed drop vs live (precision and coverage) in the golden gate, and coverage vs live in shadow | | `min_shadow` | `RATCHET_MIN_SHADOW` | 100 | Would-serve shadow answers needed before a shadow decision | | `min_audits` | `RATCHET_MIN_AUDITS` | 20 | Teacher audits needed in a canary stage before it is promoted, and before the Wilson rules roll back or alert. The anytime-valid rules have no minimum for a rollback or an alert | | `live_floor` | `RATCHET_LIVE_FLOOR` | 0.92 | Audited precision floor. The Wilson rule rolls back when the Wilson upper bound is below it; the e-processes test "precision at or above the floor". Strictly between 0 and 1 | | `min_task_test_n` | `RATCHET_MIN_TASK_TEST_N` | 10 | Held-out (odd-half) golden rows a task needs before the golden gate checks its own numbers ([per-task calibration](#per-task-calibration)). A whole number, 0 or more | | `task_mode` | `RATCHET_TASK_MODE` | `disable` | What the golden gate does with a task that fails its own check: `disable` it on the candidate (its threshold becomes 1.01, a `task_disabled` event; the version ships for its other tasks) or `reject` the candidate. Case and surrounding spaces are ignored; empty means the default | | `sequential` | `RATCHET_SEQUENTIAL` | 0 | 1 switches the canary to the anytime-valid rule: rollback when the e-value over every audit the canary has had reaches 1/`alpha_rollback`, with no minimum count; promotion on this stage's audits (point estimate, or evidence with `promote_evidence`); retirement at `max_stage_audits`. 0: the Wilson rule above. 0 or 1 | | `alpha_rollback` | `RATCHET_ALPHA_ROLLBACK` | 0.05 | Sequential rule: the chance of ever rolling back a version whose precision stays at or above the floor (threshold 1/alpha: 20). Strictly between 0 and 1 | | `promote_evidence` | `RATCHET_PROMOTE_EVIDENCE` | 0 | 1 promotes a canary stage only when the e-value that precision is above the floor reaches 1/`alpha_promote` (56 audits that all agree at a 0.92 floor and 5%), instead of on the point estimate. Needs `sequential=1`. 0 or 1 | | `alpha_promote` | `RATCHET_ALPHA_PROMOTE` | 0.05 | Sequential rule with `promote_evidence`: the chance that a version at or below the floor reaches live. Strictly between 0 and 1 | | `max_stage_audits` | `RATCHET_MAX_STAGE_AUDITS` | 400 | Sequential rule: a canary stage with this many audits and no decision is retired (event `inconclusive`); 0 = never. It counts audits, not time. Must be at least `min_audits`, and with `promote_evidence` at least the audits a promotion needs (startup says how many) | | `alert_eprocess` | `RATCHET_ALERT_EPROCESS` | 1 | 1: the live alert fires when the e-value against "precision at or above the floor" over the audits since the version went live reaches 1/`alpha_alert`, with no minimum count. 0: the Wilson rule (at least `min_audits` audits and the upper bound below the floor). 0 or 1 | | `alpha_alert` | `RATCHET_ALPHA_ALERT` | 0.01 | The live alert's error rate: a live version at or above the floor is alerted on with probability at most this (threshold 1/alpha: 100). Strictly between 0 and 1 | | `min_certified_precision` | `RATCHET_MIN_CERTIFIED_PRECISION` | 0.0 | Golden gate: reject a candidate whose certified lower bound on precision (recomputed from its certificate's counts) is below this, or that has no certificate. 0: off. From 0 to 1 | | `paired_gate` | `RATCHET_PAIRED_GATE` | 0 | 1: the golden gate's regression check compares the candidate with the re-scored live version row by row on the same held-out rows (the candidate's `paired` record), instead of the two point estimates within `tolerance`. It rejects only a regression that is both larger than `tolerance` (as a share of the rows) and significant (exact McNemar, `regression_alpha`), on coverage or on wrong answers, or a negative flip rate above `max_nfr`. Without a paired record against the live version it falls back to the unpaired rule. 0 or 1 | | `regression_alpha` | `RATCHET_REGRESSION_ALPHA` | 0.05 | The paired gate's error rate for each McNemar test: a candidate no worse than live is rejected with probability at most this. Strictly between 0 and 1 | | `max_nfr` | `RATCHET_MAX_NFR` | 1.0 | The paired gate: reject when the 95% Wilson lower bound of the negative flip rate (answers live served right that the candidate serves wrong) is above this. 1: off. More than 0 and at most 1 | | `min_judge_pass` | `RATCHET_MIN_JUDGE_PASS` | 0 (off) | Golden gate on free-text tasks ([free-text tasks](#free-text-tasks)): the judge's pass rate the answers a candidate would serve need. Below it overall, the candidate is rejected; below it on a task with `min_task_test_n` held-out rows, that task fails as under `task_mode`. With it on, a free-text task the candidate would answer that no judge graded fails too. From 0 to 1 | | `drift_alarm` | `RATCHET_DRIFT_ALARM` | 100 | The label-free drift alarm: the alarm threshold of each (version, task) stream's test martingale on the student's confidence. A stream that sees no traffic change alarms with probability at most 1/threshold over its whole life (1% at 100); with K streams, at most K/threshold for any of them (`GET /ratchet/drift` reports both). A version holds at most 17 streams: 16 task streams and one for every later task name. An alarm writes a `drift` event and suspends the version's certificate; it decides no release. On by default; it reads the ledger in the gate's tick and never touches the request path. 0 turns it off; otherwise more than 1 | | `drift_audit_boost` | `RATCHET_DRIFT_AUDIT_BOOST` | 1.0 | After a drift alarm, the alarmed version's audit rate is multiplied by this while it serves (canary or live), up to 0.5, and never lowered. The boosted rate is logged as `label_p`; a newer version going live clears it. 1: off. 1 or more, and needs the monitor on | | `drift_early_night` | `RATCHET_DRIFT_EARLY_NIGHT` | 0 | 1: when a drift alarm came from a request logged after the last `trained` event, Night School needs half the usual new examples (`max(1, min_new_examples // 2)`). Never while a release is in flight. 0 or 1, and needs the monitor on | | `teacher_noise_max` | `RATCHET_TEACHER_NOISE_MAX` | 0.05 | The teacher watch: the rate at which a fresh teacher label of a request may disagree with an independent earlier label of it before the watch calls the teacher unstable (H0 of the freeze rule). Strictly between 0 and 1. Raise it on purpose for a teacher you know is noisier (a sampling teacher, a free-text-heavy task mix); the runbook says when | | `alpha_teacher` | `RATCHET_ALPHA_TEACHER` | 0.01 | The freeze rule's error rate: a teacher within `teacher_noise_max` freezes Night School with probability at most this per check epoch, however often the gate looks (threshold 1/alpha: 100). Strictly between 0 and 1 | | `teacher_watch` | `RATCHET_TEACHER_WATCH` | 1 | 1: the teacher watch acts: contested labels are left out of training and the golden set, the freeze rule pauses Night School (`teacher_drift`), the golden gate warns about the agreement ceiling, and Night School probes and adjudicates. 0: none of that; the free checks are still recorded (one indexed lookup per stored duplicate) and shown by `GET /ratchet/teacher`, so turning it on later has history. 0 or 1 | `ReleasePolicy` checks its values when it is built (`from_env` included), so a typo stops startup with a `ValueError` naming the variable: whole numbers for the integer fields, `min_precision`, `min_coverage`, `tolerance`, `min_judge_pass` and `min_certified_precision` from 0 to 1, `live_floor` and the alphas strictly between 0 and 1, `max_nfr` above 0 and at most 1, `drift_alarm` 0 or more than 1, `drift_audit_boost` 1 or more, `teacher_noise_max` and `alpha_teacher` strictly between 0 and 1, the switches 0 or 1, `task_mode` `disable` or `reject`. It also refuses settings that would decide nothing: `promote_evidence=1` without `sequential=1`, with `sequential=1` a `max_stage_audits` below `min_audits` or too small for `promote_evidence` to ever pass (the message names the audits needed), and a [drift](https://docs.ratchet.dualmindlab.tech/guides/glossary/#drift) action (`drift_audit_boost` above 1 or `drift_early_night=1`) with `drift_alarm=0`. The live gateway's startup check (and `python -m ratchet.config check`) also applies the rules across fields, which building a policy does not, so tests and test runs may still use any value: 0 < `live_floor` <= `min_precision` <= 1 (a rollback floor above the golden gate's bar would roll back versions the gate admitted), `min_audits` and `min_shadow` at least 1, `tolerance` at most 0.2 ([Configuration checks](#configuration-checks)). A new `ReleasePolicy` field needs a row here: `python scripts/reqtrace.py --check` derives the variable names from the dataclass and fails on an undocumented one. No policy field is reserved today ([Reserved settings](#reserved-settings)). Other constants: | Constant | Where | Value | |---|---|---| | `CANARY_STEPS` | `release.py` | `[10, 50]` then 100 | | `FREQUENT` | `nightschool/dataset.py` | 3: requests seen this often always go to training | | `min_new_examples` | `run_night()` | 50 new train rows needed to train | | `target_precision` | `cascade_eval()` / `LiveEvaluator` | 0.97 for threshold calibration (the `point` rule) (`calibration.DEFAULT_TARGET_PRECISION`); a task's `policy.target_precision` overrides it for that task | | `CERT_GRID` | `stats.py` | 0.95, 0.90, …, 0.50: the thresholds Learn-then-Test and the frontier use (fixed in advance, never from the data) | | `floor` | `cascade_eval()` | 0.5: lowest allowed threshold | | Drift monitor | `drift.py` | `BINS` 10,000 (a confidence's rank is taken to 4 decimals); `BETS` 0.05, 0.15, 0.3, 0.6, 1.0 (the power betting functions); switching rate 0.01 for a stream's first 100 confidences, then 1/n; `MAX_POINTS` 200 trajectory points per stream; seed 7 for the tie-breaking draws (never the router's generator) | | Confidence weights | `confidence.score()` | 0.4 × token probability + 0.6 × agreement | | Student sampling | `router._sample` | `temperature=0.7`, `logprobs=True` | | Teacher temperature for `n` > 1 | `providers.CHOICES_TEMPERATURE` | 0.7, when the client sets no temperature (at 0 every choice would be the same) | ## Reserved settings The evidence features are built one at a time. Their settings are declared already, with defaults that keep today's behaviour, so later items only fill in logic. **Until a feature is built, any value but its default stops startup** with a `ValueError` that names the variable and the feature: a setting that does nothing yet must not look like it protects anything. The gateway, the simulator and `python -m ratchet.live` all check (`Settings.check_reserved()` in `Ratchet`, and `ReleasePolicy` when it is built). Unset, empty, or the default spelled another way (`0.50`, ` Operations `) is fine. Each feature's item removes its rows from this section and documents them where they belong. `Settings` (`config.RESERVED_SETTINGS`): | Variable | Default | Feature that will read it | |---|---|---| | `RATCHET_BATCH_API` | `operations` | Clean-room backfill: the Token Factory batch API (`operations` or `openai`) | | `RATCHET_BATCH_DISCOUNT` | `0.5` | Clean-room backfill: the batch price discount (unverified, like every price) | | `RATCHET_BACKFILL_MAX_TOKENS` | `2048` | Clean-room backfill: the output cap the worst-case budget is computed from | | `RATCHET_GATEWAY_LEASE` | `warn` | One gateway process per live ledger (waiting for the leases table): `warn` will write a `gateway_lease` event when a second process serves the same ledger and keep serving, `exclusive` will refuse to start, `off` will do nothing. Any other value stops startup now, also through `Settings.scout_policy()` | `ReleasePolicy` (`release.RESERVED_POLICY`): none today. Every field declared for the evidence features is built; the check stays for the next feature that declares a field before it is built. ## Tasks A task tells Ratchet when two answers "agree" and which answers are valid (`ratchet/tasks.py`). Its `fields` list the parts of the answer that must match, each a path and a typed [comparator](https://docs.ratchet.dualmindlab.tech/guides/glossary/#comparator) (`ratchet/compare.py`); anything else in the answer, such as a free-text `reply`, is ignored. Tasks come from four places: the built-in ones below (`ratchet/tasks.py`), a tasks file, the admin API ([API.md](/reference/api/#tasks)), and auto tasks derived from a request's `response_format` json_schema ([Structured outputs](#structured-outputs)). The name `embeddings` is reserved for the `/v1/embeddings` pass-through's request rows and cannot be a task's ([Other OpenAI endpoints](#other-openai-endpoints)). Read in `ratchet/config.py`; used by `ratchet/taskspec.py`, `ratchet/router.py` and `ratchet/gateway.py`: | Variable | Default | Meaning | |---|---|---| | `RATCHET_TASKS_FILE` | empty | A file of task specs ([Adding a task](#adding-a-task)), JSON, or YAML for a `.yaml` or `.yml` file (needs PyYAML: `pip install pyyaml`, or the `yaml` extra). Read when the gateway starts (and on a mock-mode gateway reset) and by `python -m ratchet.live`. A file task may replace a built-in one. A bad file stops startup with every error and its path (`tasks[2].fields.total.abs_tol: must be a number >= 0`); a YAML file without PyYAML is an error that says so | | `RATCHET_UNKNOWN_TASKS` | `distill` | What a request naming a task that is not registered gets. `distill` (v0.4): a task without fields made up for the name, distilled like any other. `teacher`: the teacher answers (route `unknown_task`), the student is not called and nothing is stored. `reject`: 400 `unknown_task`, listing the registered tasks the key may use. Under `teacher` and `reject` an `X-Ratchet-Task` header that is not a valid task name is a 400 `unknown_task` too (under `distill` it means `default`, as in v0.4). Case-insensitive; anything else stops startup | | `RATCHET_SCHEMA_REPAIR` | `1` | `0`: no repair call. Otherwise a complete teacher answer that breaks its request's `response_format` gets one more call to the same teacher, shown its answer and the schema errors; the repaired answer is served (unless it was cut off or is empty: the first answer is then kept) and, when valid, stored with the request's own messages. Both calls are paid for in `cost_usd`; the baseline and the client's `usage.prompt_tokens` count the request's own prompt, and `latency_ms` both calls. Read in `ratchet/config.py`, used by `ratchet/router.py` (`Ratchet._repair`) | | `RATCHET_MODEL_POLICY` | `lenient` | What a request whose `model` Ratchet does not know gets ([task resolution](#task-resolution)). `lenient`: it is served as `ratchet` (its task from the other rules) and the response carries `X-Ratchet-Model-Alias: ` (UTF-8 percent-encoded except for visible ASCII), so clients with a hard-coded model name work unchanged. `strict`: a 404 `model_not_found` ("The model 'gpt-4o' does not exist"), before anything is logged or billed, also when a header names the task. Under both, a missing or empty `model`, `ratchet`, `ratchet/` and the aliases are accepted (`ratchet/` for an unregistered task then follows `RATCHET_UNKNOWN_TASKS`, and `GET /v1/models/{id}` gives the same answer), and `ratchet/` with a name that is not a task name is a 404. Case-insensitive; anything else stops startup | | `RATCHET_MODEL_ALIASES` | empty | Model ids that name a task, as `model=task` pairs: `gpt-4o-mini=ticket-triage,openai/gpt-4o=default`. For clients that send a fixed model name. An alias is 1 to 256 characters without spaces, `,` or `=`, and must not be `ratchet` or start with `ratchet/` or `ratchet:`; at most 256. The task need not be registered yet: until it is, requests for the alias follow `RATCHET_UNKNOWN_TASKS`. Aliases are listed in `GET /v1/models` (with `alias_of`). A bad entry stops startup | | `RATCHET_AUTO_TASK` | `off` | `system_prompt`: a request no header, metadata, model, alias or match rule places gets an auto task of its own per first system message, `auto-sys-` and the first 10 hex digits of the message's SHA-256 (the name holds a digest, never the prompt). A new auto task compares whole answers like `default`, is never pinned, and is registered and stored in the ledger (so a restart and Night School find it) once its first request has passed the quota. A request without a system message gets `default`. An auto task is a registered task, so `RATCHET_UNKNOWN_TASKS` does not apply to it; the admin API may replace it with an API task. Case-insensitive; anything else stops startup | | `RATCHET_MAX_AUTO_TASKS` | `200` | The most auto tasks that may exist (at most 50 in mock mode, whose shared mock-mode gateway a reset clears). Past it, or when the registry has only its last 50 slots free (kept for file and API tasks, so request traffic cannot lock the admin API out), a request with a new system prompt gets `default`, and one `auto_task_cap` event is logged once such a request is admitted. `0`: none is ever made. An admin can delete an auto task (`DELETE /ratchet/tasks/{name}`) to free its place. A whole number from 0 to 250 (half the registry); anything else stops startup | | `RATCHET_FIXTURES_DIR` | `fixtures` | Where fixture files a task spec names must live (`taskspec.resolve_fixture`: a path that resolves outside it, through `..` or a symlink, is refused): the SQL scripts that [text-to-SQL tasks](#text-to-sql-tasks) run their answers on. Relative to the working directory; the gateway and `python -m ratchet.live` both read it | | Task | Fields | `pinned` | Agreement rule | |---|---|---|---| | `default` | none | no | Whole answer: when both are a JSON object or array, canonical JSON (sorted keys; strings, keys included, NFKC-normalised, case-folded, whitespace collapsed; numbers as floats); otherwise the text after removing reasoning (`tasks.strip_reasoning`), lower-cased with whitespace collapsed. An empty answer never agrees | | `ticket-triage` | `category` (label over `billing`, `technical`, `account`, `shipping`, `refund`, `security`), `priority` (label over `low`, `medium`, `high`, `urgent`) | no | Parse JSON (tolerates reasoning, code fences and surrounding text) and compare both labels. Both are required: `{}`, a missing `priority` or a category outside the list is invalid. `reply` is ignored | | `source-triage` | `relevant` (bool), `source_type` (label over `docs`, `paper`, `blog`, `forum`, `news`, `other`) | no | Scout's triage agent. Same JSON rule; `reason` is ignored | | `page-read` | `answers_question` (bool), `fact_type` (label over `definition`, `number`, `date`, `name`, `none`) | no | Scout's reader agent. Same JSON rule; `key_fact` is ignored | | `research-plan` | none | **yes** | Scout's planner. Always answered by the teacher | | `research-write` | none | **yes** | Scout's writer. Always answered by the teacher | The built-in vocabularies are in `ratchet/vocab.py`; the prompts that ask for these labels (`sim/tickets.py`, `ratchet/scout/prompts.py`) import the same lists. **Valid answers.** Every field is required unless it says `required=False`, and may be null only when `nullable=True`. An answer is invalid when it is empty after cleaning, does not parse in the task's format, lacks a required field, or has a value its type cannot normalise (a label outside `values`, a date that is not one). An invalid answer never counts: a student sample gets no vote (all invalid: confidence 0, never served), a teacher answer is served but not stored (`skip_example` `invalid field `) and is not an audit, a golden reference is skipped (`golden_invalid_refs` in the eval), and a stored train row the current spec refuses is left out of training (`skipped_invalid` in the dataset manifest). An optional field that is missing or null agrees only with missing or null. **Agreement is pairwise.** `confidence.score` compares every pair of samples with the task's comparators, so a tolerance works: under `abs_tol=1`, samples 0, 0.6 and 1.2 agree through 0.6 (agreement 1.0), and 0.6 is the answer served ([ARCHITECTURE.md](/reference/architecture/#score-confidencescore)). ### Field types `FieldSpec(path, type, ...)`. `path` is `$` (the whole answer) or a dot path into nested objects (`customer.address.city`); a missing path is a missing field. | Type | Accepts | Normalised to | Agree when | Options | |---|---|---|---|---| | `exact` | any JSON value | strings stripped and lower-cased, anything else as it is (v0.4's rule) | equal as JSON text: `1` and `1.0` differ, so do `true` and `1` | none | | `label` | a string (or an integer) | NFKC, case-folded, trailing `.`, `,`, `;`, `:`, `!`, `?` dropped, runs of spaces, `_` and `-` as one `_` (`Very High.` is `very_high`), then `aliases` | equal; a label outside `values` is invalid | `values`, `aliases` (`{"P1": "urgent"}`; each alias must name a value and must not be one) | | `bool` | `true`/`false`, `yes`/`no`, `y`/`n`, `1`/`0`, as JSON or as a string | true or false | equal | none | | `number` | a JSON number, or a string with currency symbols (`$`, `€`, `₹`, any Unicode currency sign), an ISO 4217 code in capitals before or after (`USD 12`, `12.50 EUR`), thousands separators that fit `decimal` (`1,234.50`; `1.234,50` with `decimal=","`; spaces, apostrophes, Indian grouping `1,23,456.50`), `(12.00)` for a negative, a trailing `%`; Indian amounts: `Rs.`, `Rs` or `Re.` before or after, `rupees` after, a closing `/-`, and `lakh` (`lakhs`, `lac`, `lacs`), `crore` (`crores`) or `lakh crore` after (`Rs. 1.5 lakh` is 150000, scaled exactly; `Cr` and `L` are not read) | a float; NaN, infinities, ambiguous groupings (`1,5` under `decimal="."`) and strings over 100 characters are invalid | `abs(a − b) <= max(abs_tol, rel_tol × max(abs(a), abs(b)))`; exact with both tolerances 0 (`1` equals `1.0`) | `abs_tol`, `rel_tol`, `decimal` (`.` or `,`), `percent` (`number`: `12%` is 12; `fraction`: 0.12) | | `ordinal` | a label in `order` | its position | positions at most `tolerance` apart | `order` (required, lowest first), `tolerance`, `aliases` | | `text` | a string (or an integer) | `exact`: case-folded, whitespace collapsed; `loose` (default): NFKC, case-folded, punctuation read as a space between words (`Acme,Inc.` is `acme inc`, `hello-world` is `hello world`) except apostrophes and the dots of initials, which join (`O'Brien` is `obrien`, `U.S.A.` is `usa`), whitespace collapsed, trailing `strip_suffixes` tokens dropped while another token is left; `token_set`: the set of loose tokens | equal | `mode`, `strip_suffixes` (`("inc", "ltd", "llc", "gmbh", "pvt ltd")`) | | `date` | ISO date or datetime (`2026-10-03`, `2026-10-03T12:00:00Z`, offsets), `2026/10/03`, `3 Oct 2026`, `3 October 2026`, `Oct 3 2026`, `October 3rd, 2026`, `3.10.2026`, `10/03/2026` (month first) | the ISO date; with `granularity="datetime"`, the ISO datetime in UTC (naive times read as UTC) | equal | `day_first` (`03/10/2026` is 3 October), `granularity` (`date`: the day as written, even with a time zone; `datetime`) | | `language` | an ISO 639-1, 639-2 or 639-3 code, a locale tag (`en-US`, `pt_BR`, `zh-Hant-TW`, `es-419`: the whole value, subtags joined by `-` or `_`, led by a 639-1 code or a 3-letter individual language such as `cmn`), or an English or native name (`English`, `Deutsch`, `日本語`) of about 80 languages | the 639-1 code | equal; an unknown language, one outside `values`, or a sentence (`It is English`, `no idea`) is invalid | `values` | | `set` | a list, or a string split on `split` | the set of normalised elements, duplicates and `empty_values` removed | Jaccard overlap at least `min_jaccard` (default 1.0: equal); two empty sets agree | `item` (element spec or type name; default a label over the set's `values` and `aliases`), `min_jaccard`, `split`, `empty_values` (`("none", "n/a")`) | | `records` | a list of objects | each object's sub-`fields` | same length, and each record matches its partner: paired by the `key` fields, by position (`ordered=True`), or else in any order (a maximum matching, so the listed order never decides; quadratic in the records when sub-fields have a tolerance, so name a `key` when records have one) | `fields` (required), `key`, `ordered` | | `ranking` | a list of ids | the normalised ids; an id ranked twice is invalid | `prefix` (default): the first `k` in order; `set`: the first `k` as a set (`k=0`: all) | `k`, `mode`, `item` | | `similarity` | a string (free text: a summary, a short translation, a reply) | NFKC, case-folded, whitespace collapsed; an empty string is invalid | the longer is at most `max_len_ratio` times the shorter (in characters) and the `metric`'s score reaches `threshold`; equal texts always agree. A proxy: close to the other answer, not checked correct ([free-text tasks](#free-text-tasks)) | `metric` (`chrf`, the default; `token_f1`; `rouge_l`), `threshold` (default 0.7; more than 0, at most 1), `max_len_ratio` (default 2.0; 1 or more). The task needs `policy.samples` of at least 2. Not inside `records`, not a set element | `tool_calls` is a reserved type name for a later release, as is the format `tool_calls`; a spec that uses them is refused. The format `sql` takes a `sql` object instead of fields ([text-to-SQL tasks](#text-to-sql-tasks)). A set's elements and a record's `key` fields compare by their normalised value only, so an element type with a tolerance is refused (use `records`). Limits: 64 fields per task (counting the fields inside records), records nested at most 4 deep, at most 2,000 `values`, `aliases`, `order` entries, suffixes or empty values per field, at most 200 records in one answer (at any depth) and 2,000 elements in one set or ranking. A spec that breaks a rule raises `ValueError` when the task is defined, so a broken spec stops the gateway at startup; that includes an option the field's type does not use (`tolerance` on a `number`), which would otherwise do nothing. ### Task options | Option | Default | Meaning | |---|---|---| | `fields` | none | The `FieldSpec`s above. A task without fields compares whole answers (the `default` rule) | | `compare_keys` | none | The v0.4 shortcut: each key becomes `FieldSpec(key, "exact", required=False)`, so answers compare, validate and canonicalise exactly as in v0.4. Give `compare_keys` or `fields`, not both | | `format` | `json` with fields, `sql` with a `sql` object, else `text` | `json`: fields are paths into the JSON answer, and a code fence around the whole answer is unwrapped; with a `$` field the answer may be any JSON value (a list, a number). `text`: the answer is plain text and its one field is `$`, the text without trailing `.!,;:` (`Positive.` is the label `positive`). `sql`: the answer is one SQL query, compared by what it returns on a fixture ([text-to-SQL tasks](#text-to-sql-tasks)) | | `sql` | none | A sql task's options instead of `fields`: `fixture`, `dialect`, `ordered`, `empty_result`, `max_rows`, `max_steps` ([text-to-SQL tasks](#text-to-sql-tasks)). Only a task with format `sql` takes it, and such a task needs it | | `pinned` | false | Always the teacher, never training data (below) | | `store` | true | False: the task's teacher answers are served but never stored as training examples (requests are logged with `store_opt_out` `task`; `RATCHET_NO_STORE_TASKS` does the same without editing the spec) | | `privacy` | `{}` | `redact_style`: `label`, `numbered`, `surrogate` or null (the default, `RATCHET_REDACT_STYLE`), how identifiers in this task's stored examples are written. Use `numbered` or `surrogate` for extraction tasks whose answers repeat an identifier from the request. Not part of the spec's hash: it changes how new examples are stored, not how answers compare | | `policy` | `{}` | `samples` (1 to 16), `audit_rate`, `target_precision`, `min_precision` (0 to 1): part of the spec and its hash. `target_precision` is the precision the task's own threshold is calibrated to (default 0.97) and `min_precision` its floor in the golden gate (default `RATCHET_MIN_PRECISION`) ([per-task calibration](#per-task-calibration)); `audit_rate` replaces `RATCHET_AUDIT_RATE` for the task's served student answers ([risk-aware routing](#risk-aware-routing)); `samples` is how many student samples the task's confidence score rests on, in the router (the shadow too) and in the golden eval, instead of `RATCHET_SAMPLES`; a task with a `similarity` field must set it to 2 or more | | `routing` | `{}` | Eligibility, escalation rules and critical classes ([risk-aware routing](#risk-aware-routing)); part of the spec and its hash | | `tools` | `teacher` | `teacher` or `distill` (opt-in tool-call distillation, a later release). A tool request is answered by the teacher alone and never stored under both values until then ([API.md](/reference/api/#tools-and-function-calling)) | | `description` | empty | What the task is for, up to 1,000 characters. Shown by `GET /ratchet/tasks` | | `clients` | none | API key names (`RATCHET_API_KEYS`) that may use the task. Any other key gets 403 `task_not_allowed`, and so does a request without a key (with `RATCHET_API_KEYS` off, every request), before any quota or model call. A name must be a configured key when the spec is loaded or written. Empty or absent: every key | | `match` | `{}` | How a request that names no task finds this one ([task resolution](#task-resolution)): `system_prefix` (its first system or developer message starts with it), `schema_name` (its `response_format.json_schema.name` is it), `tools` (the names of its `tools` and legacy `functions` are exactly these). Every rule a task gives must hold; the first such task in registry order wins, among the tasks the key may use. Plain comparisons, never patterns | | `extends` | none | Spec files and the API only: start from another task's spec. In a tasks file, a built-in task or another task of the same file (before or after it; a task's own name extends the built-in task of that name), never an API task, because startup reads the file before it registers the stored API tasks. Through the API, any registered task (the task's own name extends its current version). `fields`, `sql`, `policy`, `match`, `privacy` and `routing` merge by key (`null` removes the base's entry; a routing block's `escalate_when` and `critical` lists are replaced whole); every other key replaces the base's; `name` and `description` are not inherited. Resolved when the task is defined, so later changes to the base do not reach it | `Task.spec_hash` is 16 hex digits of SHA-256 over the task's rules (`Task.spec()`: `format`, `fields` in their order, `pinned`, `store`, `policy`, `routing`, `tools`, and for a sql task its `sql` options and the SHA-256 of its fixture's text, not its path), so a changed vocabulary or tolerance changes it and a renamed task keeps it. `description`, `clients`, `match` and `privacy` are left out: they say who uses a task and how its examples are stored, not how its answers compare. Night School records each golden task's hash in every eval it writes (in the task's per-task record, and in `spec_hashes`), and a task whose hash changed since the serving version was scored goes to the teacher (threshold 1.01) until a night scores the version under the new spec; `GET /ratchet/tasks` shows it as `recalibrating`. Because `policy` is part of the hash, changing a task's `target_precision` or `min_precision` also sends it to the teacher until the next night calibrates it under the new target. ### Per-task calibration When several tasks share one adapter, each gets its own threshold. The golden eval scores every row once and splits the rows into the calibration and held-out halves by their global position, as before, then writes one record per task into the version's eval (`eval['tasks'][name]`): `golden_n`, `test_n`, `agreement`, `threshold`, `precision`, `coverage`, `spec_hash`, `target_precision`, `calibrated`, and `disabled` once the release gate disables the task. With one task the record holds exactly the global numbers. | Rule | Value | |---|---| | A task's threshold | Calibrated on its own calibration rows to its `policy.target_precision` (default 0.97), never below 0.5 | | Fewer than `RATCHET_MIN_TASK_GOLDEN` (20) calibration rows | The higher of its own threshold and the version's global one (`calibrated: false`); the global one when it has none | | No held-out row | 1.01: the teacher answers it until a night measures it | | No golden row at all | No record: the version's global threshold, as before per-task calibration | | A record that serves nothing (disabled, or no held-out row) | The student is not called for the task (route `task_disabled`), so it costs no student samples and is never served in degraded mode (`RATCHET_ON_TEACHER_FAILURE=serve_student`); the shadow still runs | | Spec changed since the version was scored | 1.01 until a night re-scores the version | | Golden gate, per task | Checked once it has `RATCHET_MIN_TASK_TEST_N` (10) held-out rows: it fails when its held-out precision (over the answers it would serve) is below its floor (`policy.min_precision`, else `RATCHET_MIN_PRECISION`), or below the live version's precision on it less `RATCHET_TOLERANCE`: point estimates, as for the global numbers. A task that answers none of its held-out rows fails too. `RATCHET_TASK_MODE=disable` disables it on the candidate; `reject` rejects the candidate. A candidate left answering under `RATCHET_MIN_COVERAGE` of its held-out rows at its per-task thresholds is rejected | | A disabled task | Stays disabled for that version's life, through every re-score | | Live alert, per task | One `alert` per (version, task) when its audited precision's Wilson upper bound is below `RATCHET_LIVE_FLOOR` with at least `RATCHET_MIN_AUDITS` audits, for the live and canary versions; a version whose audits are all one task's has only its own alert | A version scored before per-task calibration, or with `RATCHET_PER_TASK_CALIBRATION=0`, has no records and keeps its one threshold (with the switch off, a task with critical classes still gets its record: [risk-aware routing](#risk-aware-routing)). `/ratchet/stats` lists each task's threshold, precision and coverage from the live version ([API](/reference/api/#get-ratchetstats)). ### Free-text tasks Summaries, translations and replies are rarely word for word the same, so the exact types call every pair of good answers a disagreement and the student never earns a confident score. A `similarity` field compares them by a score instead: | `metric` | What it compares | Use for | |---|---|---| | `chrf` (default) | Character n-grams of 1 to 6 of the text without whitespace; precision and recall averaged over the orders, F-beta with beta 2; the lower of the two directions, so it is symmetric | Any language, Chinese, Japanese and Thai included; short summaries and translations | | `token_f1` | SQuAD's token F1: punctuation and the English articles dropped, the overlap of the two bags of words | Short English answers where word order does not matter | | `rouge_l` | ROUGE-L F1: the longest common subsequence of word tokens, at most 512 tokens of each answer | Answers where word order matters; languages that separate words with spaces | ```json {"tasks": [ {"name": "short-summary", "format": "text", "fields": {"$": {"type": "similarity", "metric": "chrf", "threshold": 0.7}}, "policy": {"samples": 3}}, {"name": "ticket-summary", "fields": {"category": {"type": "label", "values": ["billing", "bug", "account"]}, "summary": {"type": "similarity", "metric": "rouge_l", "threshold": 0.6}}, "policy": {"samples": 3}}, {"name": "long-form-writing", "pinned": true} ]} ``` What changes for such a task: - **Samples.** The confidence score counts how many samples agree, so a task with a similarity field needs `policy.samples` of at least 2 (with one sample it always agrees with itself); the spec is refused otherwise. The router and the golden eval both use the task's `samples`. - **The answer served** is the medoid: the sample that agrees with the most others, then the most probable among those. Similarity is not transitive (A close to B and B to C, A far from C), and the medoid is never an answer at the edge. - **Precision is a proxy.** On a similarity task, the golden eval's precision is the share of served answers close enough to the teacher's, labelled `precision_kind: proxy` in the task's record (and globally when such a task was scored). Audits and shadow comparisons use the same rule. Similar is not right: "the refund was approved" and "the refund was not approved" score 0.74 under chrF. - **The judge.** In live mode `LiveEvaluator` asks `RATCHET_JUDGE_MODEL` (JSON mode) whether each answer the student would serve on the held-out half of a free-text task's golden rows (no fields, or a similarity field) is as good as the teacher's: `judge_pass_rate` and `judge_n` in the eval and in each such task's record. The candidate is the answer chosen from the samples the eval already drew, so the judge grades what production would serve. Labels are never judged. The judge runs at night, never on the request path. The mock fleet's judge knows the stored truth, so the gate works offline. - **The judge gate.** `RATCHET_MIN_JUDGE_PASS` (default 0, off): a candidate whose overall `judge_pass_rate` is below it is rejected; a free-text task with at least `RATCHET_MIN_TASK_TEST_N` held-out rows whose own rate is below it fails like a task below its precision floor (`RATCHET_TASK_MODE`: disabled on the candidate, or the candidate rejected). With the gate on, a free-text task the candidate would answer that no judge graded fails it too. Recommended defaults. Pin long-form work (summaries of long documents, translations of whole pages, chat replies, planning, long writing): there the big model earns its price and a score cannot tell good from bad. Distill short free text (a one-paragraph summary, a short translation) with a similarity field, and turn on `RATCHET_MIN_JUDGE_PASS` (0.9 is a reasonable start). Set `RATCHET_UNKNOWN_TASKS=teacher` so that free text no operator registered is answered by the teacher and never stored, rather than distilled under the whole-answer rule. **Pinned tasks** (`Task.pinned = True`) skip the student entirely. `Ratchet.handle` sends them to the teacher, logs the request (`served_by=teacher`, cost equal to baseline) and does **not** store the answer as a training example. Use `pinned` for open-ended work (planning, long writing) that a small model should not take over. Every other task is **distillable**: its teacher answers become training data. Only an operator's spec pins a task (built-in, tasks file, admin API): no header, metadata key, `model` value or `response_format` can. Pinning a task that already has stored examples keeps them out of training and out of the golden eval from the next night on (`excluded_pinned` in the dataset manifest); the ledger keeps them, so unpinning brings them back. Every answer is cleaned the same way before it is compared, served or stored (`tasks.clean_answer`): reasoning is removed (`` blocks, anything before a lone ``, an unterminated `` that opens what is left and what follows), and for a JSON task (any task with fields, unless its format is `text`) a code fence around the whole answer is unwrapped when its body is JSON; an answer that is one JSON object is kept whole, tags quoted in its strings included. A sql task's answer loses one code fence around the whole of it (```` ```sql ... ``` ````). A teacher answer is stored as a training example only when it is complete (`finish_reason` `stop` or `tool_calls`), not empty, and valid under the task (invariant 16); otherwise it is served and logged as a `skip_example` event. So a JSON task's system prompt should ask for JSON only, and list the allowed values of each label. ### Risk-aware routing A task's `routing` block says which requests the student may take, which of its answers still go to the teacher, and which misses the release gate must rule out (`ratchet/routing.py`). It is checked when the task is defined, every error with its path (`tasks[0].routing.escalate_when[1].field: 'flaged' is not a field of this task; its fields: flagged, severity`): ```json { "name": "moderation", "fields": {"flagged": {"type": "bool"}, "severity": {"type": "label", "values": ["low", "medium", "high"]}}, "policy": {"audit_rate": 0.2}, "routing": { "student_max_input_tokens": 1800, "escalate_when": [ {"field": "flagged", "equals": true, "action": "teacher"}, {"field": "severity", "in": ["high"], "min_confidence": 0.97} ], "critical": [{"field": "flagged", "equals": true, "max_miss_rate": 0.1}] } } ``` | Key | Default | Meaning | |---|---|---| | `student_max_input_tokens` | none | A request whose message text holds more than this many tokens (characters / 4, as the gateway's input limit counts characters: content strings and text parts, names, refusals, tool call ids, names and arguments) goes to the teacher without a version being picked or the student called (route `ineligible:input_tokens`). 1 to 10,000,000. Set it at or below the student's training length (`train.py --max-seq`, 2048 tokens by default) less its answer: the student never learned longer prompts | | `student_content_types` | `["text"]` | Content part types the student reads. Only `text` today (`image_url`, `input_audio` and `file` are reserved for a student that reads them), so a request with another part, which the gateway admits only when `RATCHET_TEACHER_MODALITIES` lists it, goes to the teacher (route `ineligible:content_type`) | | `store_ineligible` | `false` | `true`: the teacher's answer to an over-length request is stored as a training example (under the usual rules: complete, valid, redacted). Off by default because `train.py` cuts every example at `--max-seq`, so a long prompt trains on its start only. A request with a part the student does not read is never stored | | `escalate_when` | `[]` | Rules checked in order, after the threshold, on the answer a confident student would serve: `{field, equals \| in, action: "teacher"}` sends an answer in the class to the teacher; `{field, equals \| in, min_confidence: x}` does unless the student's confidence is at least `x`. The first rule that holds names the route (`rule:`, from 0). The teacher's answer is served and stored as for any escalation; the request counts as escalated. At most 32 rules | | `critical` | `[]` | Classes whose misses are costly: `{field, equals \| in, max_miss_rate}` with `max_miss_rate` above 0 and below 1. The golden eval measures them and the release gate enforces them (below). At most 32 | A **condition** names one of the task's fields of a scalar type (`exact`, `label`, `bool`, `number`, `ordinal`, `text`, `date`, `language`) and holds when the field's comparator matches one of its values: `equals` (one value) or `in` (1 to 256 values). Values are normalised by the field when the task is defined, so `"HIGH"` holds for `high`, a label alias for its value, and a `number` condition keeps the field's tolerance; a value the field refuses (a label outside its vocabulary) is a spec error. On the request path the rules compare the normalised answer the cascade parsed already, so they cost microseconds, and the task's answer is never written into an event or a gate reason (only the spec's values are). | Rule | What happens | |---|---| | Order | pinned or teacher-only → eligibility → pick a version → sample → the task's threshold → `escalate_when` → serve, or escalate | | Ineligible request | The teacher answers it (or the fallback teacher, route `fallback_teacher`, when the primary is unavailable); no audit, no shadow, no degraded answer (the student never saw it); not counted as an escalation. The golden eval leaves such rows out (`excluded_ineligible`). It checks the stored prompt, which was redacted (invariant 12): a placeholder such as `[EMAIL]` is shorter than what it replaced, so a request stored with `store_ineligible` just over the limit can come back just under it and be scored. Only rows near the limit, and only with `store_ineligible` | | Rule escalation | Stored like any escalation. An answer a rule holds for never stands in for an unavailable teacher (`RATCHET_ON_TEACHER_FAILURE=serve_student`), also when the student was below the threshold. A candidate in shadow runs the same rules, so an answer a rule sends to the teacher does not count as one it would serve. The golden eval does the same: such an answer neither calibrates the task's threshold nor counts as served in the precision and coverage the gate checks (the task's record and the pooled numbers), though its row still counts in `test_n` and coverage's denominator | | Critical classes, eval | On the task's held-out rows: `n`, the rows whose teacher reference is in the class, and `misses`, those whose answer the student would serve (confidence at the threshold the record stores, the one the router applies, and no rule sends it to the teacher) while it is not in the class, with `miss_rate` and its 95% Wilson `upper_bound` (`eval['tasks'][task]['critical']`, [API](/reference/api/#get-ratchetstats)). Unsure, refused and rule-escalated rows are answered by the teacher, so they are not misses | | Critical classes, gate | A class passes when it has at least `RATCHET_MIN_TASK_TEST_N` (10) such rows and the upper bound is at most `max_miss_rate`. Otherwise the task fails at admit like any failing task (`RATCHET_TASK_MODE`: `disable` takes it off the candidate with a `task_disabled` event, `reject` rejects the candidate), and a night's re-score of the live version disables it in the same write (`task_disabled`, "re-scored"). With no misses, n rows show at most 3.84 / (n + 3.84): 10 rows 28%, 35 rows 10%, 73 rows 5%, 381 rows 1% | | Critical classes, router | The student answers such a task only on a version whose record measured its classes under the current spec (and passed the gate); without one (no golden rows of it yet, or scored before the classes were added) the task goes to the teacher (route `task_disabled`). The hosted untuned model (`RATCHET_FALLBACK_STUDENT`) never answers it, and is not sampled for it (route `task_disabled`) | | Per-task audit rate | `policy.audit_rate` (0 to 1) replaces `RATCHET_AUDIT_RATE` for the task's served student answers, with one random draw either way, so other tasks are audited on the same requests as before. The version-level release statistics (the canary's rollback and promotion, the live alert) pool audited answers without weights, so each task counts in them by its volume times its audit rate: a task audited at 0.2 weighs four times as much per request as one at 0.05, and a task at 0 is not in them at all (its answers are still served). The per-task alerts are per task, so unaffected | **Limits.** Rules catch costly *positive* predictions: they see the answer the student gives, never the one it should have given, so a student that answers `flagged: false` for a post that should be flagged is not caught by `flagged = true → teacher`. Such misses are what `critical` and the audits are for: the critical gate keeps a task on the teacher until the held-out rows show the miss rate is low enough, and audits measure the live answers. Choose `max_miss_rate` with the table above in mind: a strict limit needs many golden rows in the class before the student may answer the task at all. A routing block is part of the spec and its hash, so changing it sends the task to the teacher until a night re-scores the version. ### Task resolution A request's task is the first of these that names one (`openai_compat.resolve_task`); the response says which (`ratchet.task`, `ratchet.task_source`, `X-Ratchet-Task`): | # | Rule | `task_source` | |---|---|---| | 1 | The `X-Ratchet-Task` header | `header` | | 2 | `metadata.ratchet_task` | `metadata` | | 3 | A `model` of `ratchet/` or `ratchet:`; on the Azure-style route, the deployment in the URL instead (the body's `model` is not read there) | `model`, `deployment` | | 4 | A model alias (`RATCHET_MODEL_ALIASES`), for the `model` or a deployment | `alias` | | 5 | The first registered task whose `match` rules all hold, among those the key may use | `match` | | 6 | A `response_format` json_schema: its auto task ([Structured outputs](#structured-outputs)), made on first use (at most `RATCHET_MAX_AUTO_TASKS`) | `response_format` | | 7 | With `RATCHET_AUTO_TASK=system_prompt`, the auto task of the first system message | `auto_system` | | 8 | `default` | `default` | The `model` field only ever names a task: the teacher always gets `RATCHET_TEACHER_MODEL` (or the fallback teacher's model) and the student its adapter name, whatever a request sends, and the response's `model` is the model that served. A `model` Ratchet does not know follows `RATCHET_MODEL_POLICY`. With the default settings, a request that sends none of rules 1 to 4 is `default`, as before: the built-in tasks have no match rules and auto tasks are off. Clients pick a task with the `X-Ratchet-Task` header or `metadata.ratchet_task` (or the rules above). A name that is not registered follows `RATCHET_UNKNOWN_TASKS`: by default it gets a temporary task without fields with that name (`source` `adhoc`, not added to the registry, never pinned), so its data stays separate. Names must match `[A-Za-z0-9_.-]{1,64}`; a header that does not becomes `default` under `distill` (v0.4) and is a 400 `unknown_task` under `teacher` and `reject`, so it never becomes `default` training data there (a bad `metadata.ratchet_task` is always a 400). At most 500 tasks are registered, built-in ones included; a tasks file whose tasks, with the built-in ones it does not replace, come to more is a startup error. ### Structured outputs A request's `response_format` is a contract on every answer of that request ([API.md](/reference/api/#structured-outputs)): `json_object` asks for one JSON object, `json_schema` for JSON that matches the schema (`ratchet/jsonschema_lite.py`). An answer that breaks it is invalid under the request's task, as an answer that breaks the task's spec is: no vote for a student sample, never stored or used as a reference for a teacher answer (one repair first, `RATCHET_SCHEMA_REPAIR`). Which task a json_schema request gets: | The request names | It gets | |---|---| | An operator's task (built-in, tasks file or admin API; header or `metadata.ratchet_task`) | That task; the schema is forwarded and checked on top of its fields | | An auto task's name | That auto task when the request's schema derives the same fields; otherwise the request's own auto task (an `auto_task_collision` event) | | An unregistered name | Under `distill`, an auto task of that name derived from the schema; under `teacher` and `reject`, the unknown-task path for the name. A name starting with `auto-` counts as no name | | No name | The auto task `auto--<16 hex digits of SHA-256 over the canonical schema>`, registered on first use | An auto task stands for the fields its schema derives, so one client cannot choose what another client's answers are compared on by naming that client's auto task first with a weaker schema. If an auto name is held by an auto task of other fields (two schemas with one 64-bit digest, or one derived differently by another release), the request gets an unregistered task of that name that stores nothing, which `RATCHET_UNKNOWN_TASKS` decides about; delete the stale auto task (`DELETE /ratchet/tasks/`) to let the schema derive it again. An auto task's fields come from the schema (`ratchet/schema_tasks.py`, `derive`): | Schema | Field | |---|---| | `enum` | `label` over its values (`number`, exact, when they are all numbers, since the schema takes `2.0` for `2`; `exact` when they collide once normalised, are not all strings or integers, or are more than a label takes) | | `boolean` | `bool` | | `integer` | `number`, exact | | `number` | `number`, relative tolerance 1e-6 | | `string` with `format` `date` / `date-time` | `date` (day / datetime granularity) | | `string` with `format` `email`, `uuid`, `uri`, `ipv4`, `hostname` | `text`, exact | | any other `string`, a `const` | not compared (free text) | | `array` of enums, booleans or those strings | `set` (`ranking` with `x-ratchet: {"compare": "ordered"}`); other numbers in an array are not compared | | `array` of objects | `records` over their comparable properties (skipped when there is none) | | nested `object` | its properties as dot paths, at most 4 levels deep; 64 fields in all | Local `$ref`s and `$defs` are resolved; `"type": ["x", "null"]` or an `anyOf` with `{"type": "null"}` makes a field nullable; a field is required when the schema requires it and every object above it. A property's `x-ratchet` annotation changes its field: `{"compare": "text"}` compares a free-text string (normalised text), `{"compare": "ignore"}` leaves a field out, `{"compare": "ordered"}` makes an array a `ranking` (or ordered `records`), and `{"abs_tol": n}` gives a number a tolerance. Every key starting with `x-ratchet` is stripped from the schema before any model sees it or an example stores it, and the auto task's name hashes the schema with its annotations, since they change the fields. Auto tasks have source `auto`, are never pinned and are stored in the ledger's `tasks` table, so they are registered again at every start and by `python -m ratchet.live`. An admin may replace one with an API spec (`PUT`) or delete it (its examples stay). A schema with no comparable field, one whose derived spec holds personal data (an enum of email addresses), or one past `RATCHET_MAX_AUTO_TASKS` gets an unregistered task of its auto name, which `RATCHET_UNKNOWN_TASKS` decides about. `json_object` without a schema, and every schema with `RATCHET_MAX_AUTO_TASKS=0`, keeps the named task, or `default`, which compares canonical JSON. A teacher that answers a json_schema request with a 400 about the format is asked again with `json_object` and the schema in its system message, and that schema goes that way from then on. A refusal may be about the client's schema (a keyword the endpoint's grammar engine cannot compile), so after 3 refused schemas the endpoint is asked once with the smallest strict schema, and only when it refuses that too does every schema go as `json_object` (`json_schema_fallback` events). ### Text-to-SQL tasks Two correct queries are rarely the same string (`COUNT(*)` and `COUNT(id)`, a JOIN and an IN subquery, aliases, case), and two queries that differ by one word can mean opposite things (`city = 'Pune'` against `city = 'Oslo'`). A task with `format: "sql"` compares answers by what they return on a fixture: a small database the operator writes as a SQL script inside `RATCHET_FIXTURES_DIR` (`ratchet/sqlcheck.py`). ```json {"tasks": [ {"name": "shop-sql", "description": "Questions about the shop's customers and orders, answered with one SQLite query", "format": "sql", "sql": {"fixture": "shop.sql", "ordered": "auto", "empty_result": "text", "max_rows": 1000, "max_steps": 200000}}, {"name": "warehouse-sql", "format": "sql", "sql": {"dialect": "postgres"}} ]} ``` | `sql` option | Default | Meaning | |---|---|---| | `fixture` | required (sqlite) | A SQL script, relative to `RATCHET_FIXTURES_DIR` (a path that resolves outside it, through `..` or a symlink, is refused): a regular file of at most 5 MB of UTF-8. It is read and loaded when the spec is parsed (startup, the admin API, `validate`), so a missing, oversized, broken or refused fixture is a spec error with its path (`tasks[0].sql.fixture: fixture 'shop.sql' cannot be loaded: near ";": syntax error`) | | `dialect` | `sqlite` | `postgres`: nothing runs; two answers agree when their normalised texts are equal (below). Takes no other option | | `ordered` | `auto` | `auto`: rows compare in order when the outermost query has ORDER BY (not one in a subquery, a CTE or a window), else as a multiset, and an ordered result of two rows or more never equals an unordered one. `always`: every result in order. `never`: always as a multiset | | `empty_result` | `text` | Two empty results say little (two wrong filters both return nothing), so under `text` they agree only when the normalised queries are equal: comments and a trailing `;` dropped, unquoted words lower-cased, one space between tokens. `agree`: two empty results with the same column count agree | | `max_rows` | 1000 | 1 to 10,000. A query that returns more rows is invalid | | `max_steps` | 200000 | 1,000 to 10,000,000 SQLite VM steps. A query that needs more is invalid (a runaway recursive CTE stops in milliseconds) | **What agrees.** Two results agree when they have the same number of columns and the same rows: column names are ignored, floats are rounded to 9 significant digits and a whole float equals the integer (`SUM(qty)` and `TOTAL(qty)` agree), but `1` is not `'1'`. A duplicated row counts. **Valid answers.** One statement that starts with SELECT, WITH or VALUES, bare or in one code fence, with reasoning before it as for any task; a `;` may end it and comments may sit anywhere. Anything else is invalid, with a reason that never quotes the answer (so it can go into a `skip_example` event): `not one query` (a second statement, prose around the query, EXPLAIN, PRAGMA, a write, an unclosed string), `reads the clock` (the fixture is fixed, so the answer should be too: CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP; a date function without a time value, such as `date()`, `julianday()` or `strftime('%s')`; `'now'`, `'localtime'` or `'utc'` among a date function's arguments, also double-quoted; and, when the query runs, any date function given `now`, `subsec` or a `localtime`/`utc` modifier, however the value was built; write fixed dates such as `'2026-10-01'` instead), `sql error`, `sql denied` (an operation the sandbox refuses), `sql steps exceeded`, `sql time exceeded` (1 s), `sql memory exceeded` (128 MB, Linux), `sql too many rows`, `sql result too large` (8 MB). As for any task, an invalid student sample gets no vote and an invalid teacher answer is served but never stored. So the task's system prompt should give the schema and ask for one SQLite query and nothing else. **The fixture.** Write it as you would a test database, to tell the likely mistakes apart: NULLs, ties, duplicates, empty groups, rows on each side of every boundary a question asks about. Two queries that differ only on data the fixture lacks agree. `sqlite3 shop.db .dump > fixtures/shop.sql` makes one from a database (a `.dump`'s `PRAGMA foreign_keys`, BEGIN and COMMIT are fine). It must create a table or a view; it may set only the PRAGMAs `foreign_keys`, `defer_foreign_keys`, `recursive_triggers`, `user_version` and `application_id`, and may not ATTACH a database, call `load_extension`, `random`, `randomblob` or the `current_*` functions, read the clock while it loads (`date('now')`, `unixepoch()`, or a `DEFAULT CURRENT_TIMESTAMP` that a row takes; a `.dump` gives every value, so its defaults never run), define a view that reads the clock, hold a value over 100 KB, take more than 30 s or grow past 64 MB: its database must be the same every time it loads. `spec_hash` covers the SHA-256 of its text, so editing it sends the task to the teacher until a night re-scores the version (the stored API tasks read it again at every start, and a `PUT` of the same body after an edit is a change); moving it keeps the hash. Tasks with the same fixture text share one loaded copy. `RATCHET_FIXTURES_DIR` must be readable by the gateway and by `python -m ratchet.live`, which re-checks stored answers and runs the golden eval on the fixture. **Security.** The gateway never connects to a customer database: the fixture is the only data it runs SQL on, and the query comes from a model a client's prompt can steer, so it is run as hostile input ([THREAT_MODEL.md](/reference/threat-model/), T23): - in a private in-memory SQLite database per fixture text, never a file, with `temp_store = MEMORY`; - with `PRAGMA query_only = ON` and an authorizer that allows only reading the fixture's own tables and views, the schema and `json_each`/`json_tree` (not `sqlite_stmt`, which lists the statements other requests ran, nor `dbstat`), recursive CTEs and the deterministic functions in `sqlcheck.FUNCTIONS`, and denies writes, schema changes, ATTACH, PRAGMA (the `pragma_*` table-valued functions too), transactions, `random()`, `randomblob()`, `changes()`, `last_insert_rowid()`, `load_extension()`, `json_patch()`, `unhex()` and every function not on the list; - under a progress handler, every 100 VM steps, that stops a query after `max_steps`, 1 s or 128 MB of resident-memory growth (Linux), with results read 100 rows at a time up to `max_rows` and 8 MB; - with the work of one step and one row bounded, since SQLite calls the handler only where a loop jumps back: a value is at most 100 KB and a LIKE or GLOB pattern at most 100 bytes; `instr`, `replace` and the two-argument `trim`, `ltrim` and `rtrim`, which SQLite runs in quadratic time, are linear versions with the same results (one `trim` call took 15 s with 60,000 characters in each, measured); and LIKE, GLOB (up to about 25 ms a call), the date functions and those linear versions check the query's time on every call (a row whose WHERE called LIKE 200 times took 3.4 s before, measured); - one query at a time per fixture (Scout calls the gateway from several threads); a typical query on a small fixture takes about 0.1 ms (a call of LIKE or a date function adds a few microseconds), and a hostile one costs at most about 1.3 s and 128 MB before it is stopped. Capping a value's size needs `Connection.setlimit`, new in Python 3.11; without it one query can copy values of up to 1 GB in steps the progress handler never sees (3 GB and 18 s for a query that doubles a string, measured on 3.10). So on Python 3.10 a sqlite fixture is a spec error that says so; `dialect: postgres` still works. The Docker image runs Python 3.12. (A process-wide `PRAGMA hard_heap_limit` would also cap the ledger's SQLite and does not cap LIKE patterns;.) ### Adding a task Define tasks as data, in a tasks file or through the admin API; both take the same spec. A spec has the options above, with `fields` written as `{path: {type, ...options}}` (paths are `$` or dot-separated names of letters, digits and `_`). Unknown keys and options, an option the field's type does not use, a value of the wrong type or out of range are all errors, and every error is reported with its path, so one check finds them all. 1. Write the spec. In a file (`tasks.json`; or `tasks.yaml` with PyYAML): ```json {"tasks": [ {"name": "invoice-extract", "description": "Vendor, total, currency, due date and lines from an invoice", "fields": { "vendor": {"type": "text", "strip_suffixes": ["inc", "ltd", "llc", "gmbh"]}, "total": {"type": "number", "abs_tol": 0.01}, "currency": {"type": "label", "values": ["USD", "EUR", "INR"], "aliases": {"$": "USD", "€": "EUR"}}, "due_date": {"type": "date", "nullable": true}, "lines": {"type": "records", "key": ["sku"], "required": false, "fields": {"sku": {"type": "label"}, "qty": {"type": "number"}}} }}, {"name": "invoice-extract-eu", "extends": "invoice-extract", "fields": {"currency": {"type": "label", "values": ["EUR"]}}}, {"name": "contract-summary", "pinned": true} ]} ``` ```yaml tasks: - name: invoice-extract fields: total: {type: number, abs_tol: 0.01} currency: {type: label, values: [USD, EUR, INR]} ``` Then start the gateway with `RATCHET_TASKS_FILE=tasks.json`. Quote YAML strings that look like booleans or dates (`"yes"`, `"2026-10-03"`). Or, without a restart, through the admin API (`RATCHET_ADMIN_KEYS` set; the body is one spec, its name in the URL): ```bash curl -X POST localhost:8000/ratchet/tasks/validate -H "Authorization: Bearer $ADMIN_KEY" -d @invoice.json # a dry run curl -X POST localhost:8000/ratchet/tasks/validate -H "Authorization: Bearer $ADMIN_KEY" -d @tasks.json # a whole tasks file, before a restart curl -X PUT localhost:8000/ratchet/tasks/invoice-extract -H "Authorization: Bearer $ADMIN_KEY" -d @invoice.json ``` API tasks are stored in the ledger and registered again at every start. The API cannot change a built-in or file task (403): edit the file, or define a task under another name that `extends` it. A file task replaces a built-in one of the same name (`{"name": "ticket-triage", "extends": "ticket-triage", "policy": {"samples": 5}}` adjusts the built-in triage). 2. Make the system prompt ask for JSON with those keys, and list each label's allowed values. 3. Send traffic with `X-Ratchet-Task: invoice-extract`. `GET /ratchet/tasks` shows every task with its source, spec hash and traffic, and the unregistered names requests used. 4. Optional: for free-text tasks (no fields, or a `similarity` field), `LiveEvaluator` asks the judge model about the answers the student would serve (`judge_pass_rate`, `judge_n`), and `RATCHET_MIN_JUDGE_PASS` makes the release gate hold candidates to it ([free-text tasks](#free-text-tasks)). A built-in task is a `Task` in `tasks.BUILTIN` (`ratchet/tasks.py`), with `FieldSpec`s from `ratchet/compare.py`; put shared vocabularies in `ratchet/vocab.py`. Use that only for tasks the code itself depends on, such as Scout's. Choose fields that are verifiable (labels, IDs, numbers, dates). Typed comparison on these fields is what keeps evaluation free of judge bias; pick the loosest type that is still right (a `number` with a tolerance for amounts, a `set` for tags), because a field that is too strict escalates good answers and lowers measured precision. Changing a task's rules later is safe: the serving version keeps the task on the teacher until a night has scored it under the new spec. ## Teacher licenses `ratchet/licenses.py` holds the built-in registry (`BUILTIN`: known distillable teachers) and `CLOSED`, the ID prefixes that are always refused (`gpt`, `o1`, `o3`, `o4`, `claude`, `gemini`). Unknown teachers are refused until registered. See ADR 0001. An open teacher can also be added without a code change: | Variable | Default | Meaning | |---|---|---| | `RATCHET_TEACHER_LICENSES` | empty | A JSON file: a list of `{"model", "license", "url", "note"}` objects (`note` optional, at most 100 entries). Read at startup by the gateway (and on a mock-mode gateway reset), the simulator, `python -m ratchet.live` and `python -m ratchet.admin`, and merged into a copy of the built-in registry. The configured teachers can then be `RATCHET_TEACHER_MODEL` or `RATCHET_TEACHER_FALLBACK_MODEL`, Night School trains on their stored answers, and each adapter's BOM and NOTICE.md name them with `license_origin` `config`. A bad file stops startup with every error and its place (`[2].license: 'CC-BY-NC-4.0' is not an allowed license; ...`). Empty = the built-in registry only | Each entry is checked conservatively, so configuration cannot relabel a closed model Ratchet knows of (invariant 1): - `license` must be one of the **allowed licenses**: `OpenMDW-1.1`, `Apache-2.0`, `MIT`, `NVIDIA Nemotron Open Model License` (any case; stored in this spelling). Licenses with use restrictions or custom terms (a community license, a non-commercial one) need a reviewed change to `ratchet/licenses.py`. - `model` must not belong to a closed model family (`closed_family`). The id is split at every character that is not a letter or digit; it is refused when a part starts with a `CLOSED` prefix (`gpt`, `o1`, `o3`, `o4`, `claude`, `gemini`) or with `o` and a digit (`o5-mini`), a part contains `gpt`, `claude` or `gemini` anywhere (`chatgpt-4o-latest`), a closed model's name (`davinci`, `babbage`, `codex`; `CLOSED_NAMES`) or a closed vendor's name (`openai`, `anthropic`; `CLOSED_VENDORS`). `gpt-4o`, `azure-gpt-4o`, `chatgpt-4o-latest`, `davinci-002`, `openai/davinci-002` or `anthropic/claude-3-5-sonnet` claimed as `Apache-2.0` are refused. The rule errs toward refusing, so it also refuses open models with such a name (`openai/gpt-oss-120b`, `EleutherAI/gpt-j-6b`): add those in code, after reading their license. - `model` must not be a router alias: a part containing a router's name (`openrouter`; `ROUTER_VENDORS`) or a last path segment `auto` (`openrouter/auto`). The model that answers it is chosen per request and may be closed, so no license claim covers it; name the model itself. - `model` must not be a built-in teacher, in any case (configuration never relabels one), or appear twice; `url` must be an `https://` URL; no other keys. - `model` must not name a provenance source that is not a teacher (`licenses.SOURCES`: `human`, customer-provided labels), in any case. `RATCHET_TEACHER_MODEL=human` stops startup too: `require_distillable` never reads `SOURCES`. The model check knows only the closed families, model names, vendors and routers listed above. Any other id is admitted on your license claim alone: a closed model from another vendor under its own name (`x-ai/grok-4`, say) passes the check, so claiming an open license for it is on you. Every adapter trained on a configured teacher says so: its BOM records `license_origin` `config` and its NOTICE.md marks the license "(configured)", so whoever receives the adapter can tell your claim from Ratchet's reviewed registry. ```json [ {"model": "Qwen/Qwen3-235B-A22B-Instruct-2507", "license": "Apache-2.0", "url": "https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507", "note": "Checked on the model card, Oct 2026."} ] ``` A configured teacher has no entry in `DEFAULT_PRICES`, so it is priced as described in [Prices](#prices-default_prices). Removing an entry later makes the teacher unknown again: the gateway refuses it at startup and Night School leaves its stored rows out of training (they keep the license they were stored with). Not legal advice: check each license yourself. `BASE_MODELS` in the same file maps the student base models Ratchet knows (Nemotron 3 Nano 4B and 30B-A3B) to their license, the NVIDIA Nemotron Open Model License, which every adapter's BOM (`base_model_license`) and NOTICE.md carry. For another `RATCHET_STUDENT_BASE_MODEL` they carry a warning instead (never an error): find its license and ship it with the adapter, or add the model to `BASE_MODELS`. --- # Command line Source: https://docs.ratchet.dualmindlab.tech/api/cli/ Every command line tool in the repository, with a line on when to use it and its help text. The help text is captured from the code by `python docs-site/scripts/export-reference.py`, which runs each tool with `--help` (or reads its module docstring when it has no parser), so it matches the code it was exported from. Run commands from the repository root after `make install`. :::note[Mock mode first] Nothing here needs a key in mock mode except the live tools (`ratchet.live`, `ratchet.liverun`, `scripts/day1_check.py`). Commands that write take a ledger file with `--db` or `RATCHET_DB`; never point two gateways at one ledger. ::: ## Make targets From the [`Makefile`](https://github.com/harshbhx/ratchet/blob/main/Makefile). On Windows, run the command in the second column from WSL or Git Bash, or set the variables the PowerShell way (`$env:RATCHET_MODE="mock"`). | Target | Runs | What it is for | |---|---|---| | `make install` | `python -m pip install -e ".[dev]"` | Install the package with its dev dependencies and ruff. | | `make test` | `pytest -q` | Run the test suite (no network, no GPU). | | `make lint` | `ruff check .`
`ruff format --check .`
`python scripts/reqtrace.py --check` | Lint, check formatting and check the requirements matrix. Run it with `make test` before every commit. | | `make reference` | `python scripts/reference_check.py` | Three pinned workloads (tickets, mixed, research) must still reproduce v0.4.0's behaviour (about 25 s; CI runs it). | | `make demo-check` | `python scripts/demo_check.py $(URL) $(ARGS)` | The live gateway as a judge sees it, from outside (never part of test or lint): make demo-check URL=https:// ARGS=--chat | | `make nebius-plan` | `bash deploy/nebius/nebius.sh plan $(ARGS)` | One command to host Ratchet on Nebius (scripted and tested offline, ready to run; docs/WHERE_IT_RUNS.md). nebius-plan prints every command and changes nothing. nebius-up and nebius-down are dry runs unless you set RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1 and pass YES=1; GPU parts need ARGS="--with-student" (and --with-training). | | `make nebius-up` | `bash deploy/nebius/nebius.sh up $(if $(YES),--yes) $(ARGS)` | | | `make nebius-down` | `bash deploy/nebius/nebius.sh down $(if $(YES),--yes)` | | | `make nebius-status` | `bash deploy/nebius/nebius.sh status` | | | `make fmt` | `ruff format .`
`ruff check --fix .` | Format the code and apply safe lint fixes. | | `make sim` | `python -m sim.simulate --days 10 --requests 2000 --bad-deploy-version 3` | Developer tool: a multi-day run on mock models, used by the tests. | | `make demo` | `RATCHET_MODE=mock RATCHET_MOCK_WARMUP_DAYS=4 uvicorn ratchet.gateway:app --port 8000` | A mock gateway on `http://localhost:8000` (no keys, no GPU), for development. | | `make web` | `cd web && npm ci && npm run build` | React app (web/) -> ratchet/web/dist, which the gateway serves at /, /scout and /dashboard. | | `make web-dev` | `cd web && npm ci && npm run dev` | Hot-reload UI on :5173 against a gateway on :8000 (run `make demo` in another terminal). | | `make day1` | `python scripts/day1_check.py $(DAY1_CHECKS)` | Live day-1 probes (never in test, lint or CI). Default: the Token Factory teacher check. make day1 DAY1_CHECKS="all --json evidence/day1/day1.json" runs every probe whose key is set. | | `make live-plan` | `python -m ratchet.liverun plan --rounds $(ROUNDS) --requests $(REQUESTS) --max-usd $(MAX_USD) --workload $(WORKLOAD)` | Live-run harness: plan a run (requests per release stage, golden size, dollars per round). Free. | | `make live-smoke` | `python -m ratchet.liverun smoke --base-url $(BASE_URL)` | Live-run harness: 20 requests and a few control calls, no nights. Costs cents. | | `make live-run` | `python -m ratchet.liverun run --base-url $(BASE_URL) --rounds $(ROUNDS) --requests $(REQUESTS) --max-usd $(MAX_USD) --workload $(WORKLOAD)` | Live-run harness: the measured run, capped at `MAX_USD`. | | `make docker` | `docker build -t ratchet-gateway .` | Build the gateway image (`ratchet-gateway`). | | `make clean` | `rm -rf .pytest_cache .ruff_cache .coverage build dist *.egg-info sim_results.json ratchet.db`
`find . -name __pycache__ -type d -prune -exec rm -rf {} +` | Remove caches, build output, `sim_results.json` and `ratchet.db`. | ## Run it in live mode ### `python -m ratchet.live` **When to use it:** Live mode on Nebius: run Night School (night), advance the release gate (tick), reload adapters after an endpoint restart (reload), seal savings receipts (seal), apply retention now (purge) or export an adapter with its licenses (export). Help (the docstring of ratchet/live.py): ```text Live-mode Night School on Nebius: train on a Serverless Job, hot-load the adapter into vLLM, evaluate. Built to run inside a Nebius Serverless Job; scripted and tested offline. python -m ratchet.live night # run one night against the live ledger python -m ratchet.live tick # advance the release gate (run every few minutes via cron) python -m ratchet.live reload # re-load live/canary/shadow adapters after an endpoint restart (the gateway # also does this by itself when vLLM answers 404 for an adapter: AdapterReloader) python -m ratchet.live purge [--client NAME] [--older-than-days N] [--dry-run] # apply the retention windows now; the gateway does it hourly python -m ratchet.live export --version N --out DIR # an adapter with its licenses: the adapter folder (when synced locally), its # BOM and NOTICE.md in DIR; prints the BOM's sha256 python -m ratchet.live seal # seal the savings receipts of every day that has ended (run hourly via cron: # a ledger day starts at the first request's time of day, not 00:00 UTC; the # gateway also seals when a receipt is asked for) Environment (in addition to ratchet.config): RATCHET_DATA_DIR local folder synced to the bucket mounted in jobs/endpoints (default ./data) RATCHET_SYNC_CMD command that uploads RATCHET_DATA_DIR to the bucket, e.g. "aws s3 sync ./data s3://my-bucket/ratchet --endpoint-url https://storage.eu-north1.nebius.cloud" (run before training, and again after a night that trained, to upload the adapter's BOM into its folder; never with --delete, which would remove the weights the job wrote there) RATCHET_TRAIN_CMD command that runs the training job and waits for it; {version} and {data} are substituted, e.g. "bash deploy/train_job.sh {version}" RATCHET_ADAPTER_ROOT where the endpoint sees adapters (default /data/adapters) RATCHET_NIGHT_STUDENT_TIMEOUT_S read timeout of the student's calls in the night's golden eval (default 120 s; the first call after a hot-load is slow). The student endpoint must be RUNNING before the hot-load: the night does not start or wake it RATCHET_BUCKET_S3 when set with the default train command, `aws` must be on PATH (the night stops with a message if it is not) After a night that trained, the adapter's BOM and NOTICE.md go to RATCHET_DATA_DIR/adapters// (ratchet_bom.json, NOTICE.md) and RATCHET_SYNC_CMD runs again, so the bucket's adapter folder holds the weights and their licenses together. ``` ### `python -m ratchet.live export` **When to use it:** Hand an adapter to someone else with its bill of materials and NOTICE.md. Help (--help): ```text usage: python -m ratchet.live export [-h] --version VERSION --out OUT Copy an adapter with its licenses into a folder: the adapter files (when RATCHET_DATA_DIR/adapters/ holds them), its bill of materials (ratchet_bom.json) and NOTICE.md; prints the BOM's sha256. options: -h, --help show this help message and exit --version VERSION the version id (as in /ratchet/versions/{id}/bom) --out OUT the folder to write; it must not exist yet or be empty ``` ### `python -m ratchet.live purge` **When to use it:** Apply the retention windows now instead of waiting for the gateway's hourly run. Help (--help): ```text usage: python -m ratchet.live purge [-h] [--client CLIENT] [--older-than-days OLDER_THAN_DAYS] [--dry-run] Apply the retention windows now: purge the text of examples not seen for RATCHET_EXAMPLE_RETENTION_DAYS and fold request rows older than RATCHET_REQUEST_RETENTION_DAYS into daily sums. options: -h, --help show this help message and exit --client CLIENT only this API client's examples and request rows --older-than-days OLDER_THAN_DAYS this window, in whole days, for both examples and requests (instead of the settings) --dry-run count what would go and change nothing ``` ### `python -m ratchet.config` **When to use it:** Check the environment before you start or update a gateway: exit 2 means the gateway would not start. Help (--help): ```text usage: python -m ratchet.config [-h] {check} ... Check Ratchet's settings. positional arguments: {check} check list every configuration error and warning; exit 0, 1 (warnings) or 2 (errors) options: -h, --help show this help message and exit ```
python -m ratchet.config check --help ```text usage: python -m ratchet.config check [-h] [--json] options: -h, --help show this help message and exit --json print one JSON object ```
### `python -m ratchet.cost` **When to use it:** Print the cost report (savings per token and by GPU-hours paid) and record when the student endpoint was up. Help (--help): ```text usage: python -m ratchet.cost [-h] {report,endpoint,close} ... Ratchet cost accounting positional arguments: {report,endpoint,close} report print the cost report and the recommendation endpoint record the student endpoint going up or down close close an open interval of a kind (a job whose end was lost) options: -h, --help show this help message and exit ```
python -m ratchet.cost report --help ```text usage: python -m ratchet.cost report [-h] [--db DB] [--days DAYS] [--json] options: -h, --help show this help message and exit --db DB the live ledger file (default: RATCHET_DB) --days DAYS only the last N ledger days --json print the whole report as JSON ```
python -m ratchet.cost endpoint --help ```text usage: python -m ratchet.cost endpoint [-h] [--db DB] [--usd-per-hour USD_PER_HOUR] [--gpus GPUS] [--at AT] [--note NOTE] {up,down} positional arguments: {up,down} options: -h, --help show this help message and exit --db DB the live ledger file (default: RATCHET_DB) --usd-per-hour USD_PER_HOUR USD per GPU-hour (default: RATCHET_STUDENT_GPU_USD_PER_HOUR) --gpus GPUS GPUs (default: RATCHET_STUDENT_GPUS) --at AT when, if not now: seconds since the epoch or ISO 8601 (UTC unless it says) --note NOTE a note stored with the interval ```
python -m ratchet.cost close --help ```text usage: python -m ratchet.cost close [-h] [--db DB] [--at AT] {student_endpoint,training_job,eval_job} positional arguments: {student_endpoint,training_job,eval_job} options: -h, --help show this help message and exit --db DB the live ledger file (default: RATCHET_DB) --at AT when it ended: seconds since the epoch or ISO 8601 (default: now) ```
### `python -m ratchet.admin` **When to use it:** Operator actions from a shell, with an audit trail: erase an end user's data or export the ledger's tables. Help (--help): ```text usage: python -m ratchet.admin [-h] [--db DB] {versions,rollback,retire,promote,flag,pin,unpin,audit-log,erase,export} ... Manual interventions, with an audit trail; erase stored data and export tables. positional arguments: {versions,rollback,retire,promote,flag,pin,unpin,audit-log,erase,export} versions every version, the runtime switches and the task pins rollback roll a canary back retire retire a live, canary or shadow version promote promote a version to a canary stage or to live flag show the runtime switches, or set one pin pin a task to the teacher unpin remove a task's runtime pin audit-log the newest admin actions erase erase stored examples for good (a dry run without --apply) export one table as NDJSON options: -h, --help show this help message and exit --db DB the ledger file (default: RATCHET_DB) ```
python -m ratchet.admin versions --help ```text usage: python -m ratchet.admin versions [-h] [--json] options: -h, --help show this help message and exit --json ```
python -m ratchet.admin rollback --help ```text usage: python -m ratchet.admin rollback [-h] --reason REASON id positional arguments: id options: -h, --help show this help message and exit --reason REASON why, in 3 to 500 characters ```
python -m ratchet.admin retire --help ```text usage: python -m ratchet.admin retire [-h] --reason REASON id positional arguments: id options: -h, --help show this help message and exit --reason REASON why, in 3 to 500 characters ```
python -m ratchet.admin promote --help ```text usage: python -m ratchet.admin promote [-h] --to {canary,live} [--traffic-pct TRAFFIC_PCT] [--force] [--load-adapter] --reason REASON id positional arguments: id options: -h, --help show this help message and exit --to {canary,live} --traffic-pct TRAFFIC_PCT the canary's share, 1 to 99 (default: the next stage) --force allow a rejected, rolled-back or retired version or a candidate --load-adapter load the adapter into the student endpoint first --reason REASON why, in 3 to 500 characters ```
python -m ratchet.admin flag --help ```text usage: python -m ratchet.admin flag [-h] [--reason REASON] [--force] [--json] [{student_disabled,releases_frozen,training_paused,audit_rate}] [value] positional arguments: {student_disabled,releases_frozen,training_paused,audit_rate} value on|off for a switch; 0 to 1 or 'default' for audit_rate options: -h, --help show this help message and exit --reason REASON --force allow an audit rate of 0 while a canary is in flight --json ```
python -m ratchet.admin pin --help ```text usage: python -m ratchet.admin pin [-h] --reason REASON task positional arguments: task options: -h, --help show this help message and exit --reason REASON why, in 3 to 500 characters ```
python -m ratchet.admin unpin --help ```text usage: python -m ratchet.admin unpin [-h] --reason REASON task positional arguments: task options: -h, --help show this help message and exit --reason REASON why, in 3 to 500 characters ```
python -m ratchet.admin audit-log --help ```text usage: python -m ratchet.admin audit-log [-h] [--limit LIMIT] [--json] options: -h, --help show this help message and exit --limit LIMIT 1 to 1000 (default 50) --json ```
python -m ratchet.admin erase --help ```text usage: python -m ratchet.admin erase [-h] --reason REASON [--client CLIENT] [--end-user END_USER] [--hash HASHES] [--contains CONTAINS] [--apply] [--actor ACTOR] options: -h, --help show this help message and exit --reason REASON why (stored redacted in the erasure log) --client CLIENT an API key name: the examples it sent (and the scope of --end-user) --end-user END_USER the value a client sends as user or safety_identifier; '-' reads it from stdin --hash HASHES an example hash (repeat for more) --contains CONTAINS text in the stored (already redacted) messages, answer or conditions --apply erase; without it nothing changes --actor ACTOR who asks, for the erasure log: a name of letters, digits, '_', '.', '-' (default 'cli') ```
python -m ratchet.admin export --help ```text usage: python -m ratchet.admin export [-h] [--task TASK] [--split {train,golden}] [--since SINCE] [--client CLIENT] [--out OUT] {examples,requests,versions,events,erasures} positional arguments: {examples,requests,versions,events,erasures} options: -h, --help show this help message and exit --task TASK --split {train,golden} --since SINCE seconds since the epoch, or an ISO 8601 date or date-time (UTC) --client CLIENT an API key name --out OUT file to write (default standard output) ```
### `python -m ratchet.importer` **When to use it:** Import labelled examples (human labels or an allowed teacher's answers) from a JSONL file. Help (--help): ```text usage: python -m ratchet.importer [-h] --task TASK --source SOURCE [--attest ATTEST] [--golden-only] [--client CLIENT] [--db DB] [--dry-run] file Import labelled data into the ledger. positional arguments: file a JSONL file (OpenAI chat fine-tuning format, or Ratchet's train.jsonl); '-' reads stdin options: -h, --help show this help message and exit --task TASK the registered task the rows belong to --source SOURCE 'human' (with --attest) or 'teacher:' (an open teacher) --attest ATTEST where the labels come from and that you own them (required with --source human) --golden-only hold every row out of training for good (golden split) --client CLIENT the API key name to attribute the rows to (for erasure and export by client) --db DB ledger file (default RATCHET_DB) --dry-run import into an in-memory copy and report; write nothing ``` ### `python -m ratchet.liverun` **When to use it:** Measure a live run: rounds of synthetic traffic with Night School between them and a control arm sent straight to the teacher. Help (--help): ```text usage: python -m ratchet.liverun [-h] {plan,smoke,run,report,verify} ... Live-run harness positional arguments: {plan,smoke,run,report,verify} options: -h, --help show this help message and exit ```
python -m ratchet.liverun plan --help ```text usage: python -m ratchet.liverun plan [-h] [--base-url BASE_URL] [--client CLIENT] [--workload WORKLOAD] [--seed SEED] [--rounds ROUNDS] [--requests REQUESTS] [--max-usd MAX_USD] [--control-share CONTROL_SHARE] [--control-max-usd CONTROL_MAX_USD] [--concurrency CONCURRENCY] [--batch BATCH] [--until-settled | --no-until-settled] [--max-extend MAX_EXTEND] [--nights {inprocess,off}] [--on-night-failure {stop,continue}] [--drift-round DRIFT_ROUND] [--untuned-arm | --no-untuned-arm] [--scout-runs SCOUT_RUNS] [--plan-share PLAN_SHARE] [--plan-train-hours PLAN_TRAIN_HOURS] [--plan-night-minutes PLAN_NIGHT_MINUTES] [--out OUT] [--no-endpoint-check] [--gate-profile {planned,as-set}] options: -h, --help show this help message and exit --base-url BASE_URL --client CLIENT the harness key's name in RATCHET_API_KEYS --workload WORKLOAD tickets, research, mixed, or file:PATH --seed SEED --rounds ROUNDS --requests REQUESTS --max-usd MAX_USD --control-share CONTROL_SHARE --control-max-usd CONTROL_MAX_USD --concurrency CONCURRENCY --batch BATCH --until-settled, --no-until-settled --max-extend MAX_EXTEND --nights {inprocess,off} --on-night-failure {stop,continue} --drift-round DRIFT_ROUND --untuned-arm, --no-untuned-arm --scout-runs SCOUT_RUNS --plan-share PLAN_SHARE --plan-train-hours PLAN_TRAIN_HOURS --plan-night-minutes PLAN_NIGHT_MINUTES --out OUT --no-endpoint-check --gate-profile {planned,as-set} planned (run's default): refuse unless RATCHET_SEQUENTIAL=1 and RATCHET_CANARY_AUDIT_RATE=0.5 are in force ```
python -m ratchet.liverun smoke --help ```text usage: python -m ratchet.liverun smoke [-h] [--base-url BASE_URL] [--client CLIENT] [--workload WORKLOAD] [--seed SEED] [--rounds ROUNDS] [--requests REQUESTS] [--max-usd MAX_USD] [--control-share CONTROL_SHARE] [--control-max-usd CONTROL_MAX_USD] [--concurrency CONCURRENCY] [--batch BATCH] [--until-settled | --no-until-settled] [--max-extend MAX_EXTEND] [--nights {inprocess,off}] [--on-night-failure {stop,continue}] [--drift-round DRIFT_ROUND] [--untuned-arm | --no-untuned-arm] [--scout-runs SCOUT_RUNS] [--plan-share PLAN_SHARE] [--plan-train-hours PLAN_TRAIN_HOURS] [--plan-night-minutes PLAN_NIGHT_MINUTES] [--out OUT] [--no-endpoint-check] [--gate-profile {planned,as-set}] [--repeat-of RUN_DIR] options: -h, --help show this help message and exit --base-url BASE_URL --client CLIENT the harness key's name in RATCHET_API_KEYS --workload WORKLOAD tickets, research, mixed, or file:PATH --seed SEED --rounds ROUNDS --requests REQUESTS --max-usd MAX_USD --control-share CONTROL_SHARE --control-max-usd CONTROL_MAX_USD --concurrency CONCURRENCY --batch BATCH --until-settled, --no-until-settled --max-extend MAX_EXTEND --nights {inprocess,off} --on-night-failure {stop,continue} --drift-round DRIFT_ROUND --untuned-arm, --no-untuned-arm --scout-runs SCOUT_RUNS --plan-share PLAN_SHARE --plan-train-hours PLAN_TRAIN_HOURS --plan-night-minutes PLAN_NIGHT_MINUTES --out OUT --no-endpoint-check --gate-profile {planned,as-set} planned (run's default): refuse unless RATCHET_SEQUENTIAL=1 and RATCHET_CANARY_AUDIT_RATE=0.5 are in force --repeat-of RUN_DIR run the same plan and seed again ```
python -m ratchet.liverun run --help ```text usage: python -m ratchet.liverun run [-h] [--base-url BASE_URL] [--client CLIENT] [--workload WORKLOAD] [--seed SEED] [--rounds ROUNDS] [--requests REQUESTS] [--max-usd MAX_USD] [--control-share CONTROL_SHARE] [--control-max-usd CONTROL_MAX_USD] [--concurrency CONCURRENCY] [--batch BATCH] [--until-settled | --no-until-settled] [--max-extend MAX_EXTEND] [--nights {inprocess,off}] [--on-night-failure {stop,continue}] [--drift-round DRIFT_ROUND] [--untuned-arm | --no-untuned-arm] [--scout-runs SCOUT_RUNS] [--plan-share PLAN_SHARE] [--plan-train-hours PLAN_TRAIN_HOURS] [--plan-night-minutes PLAN_NIGHT_MINUTES] [--out OUT] [--no-endpoint-check] [--gate-profile {planned,as-set}] [--repeat-of RUN_DIR] [--resume RUN_DIR] options: -h, --help show this help message and exit --base-url BASE_URL --client CLIENT the harness key's name in RATCHET_API_KEYS --workload WORKLOAD tickets, research, mixed, or file:PATH --seed SEED --rounds ROUNDS --requests REQUESTS --max-usd MAX_USD --control-share CONTROL_SHARE --control-max-usd CONTROL_MAX_USD --concurrency CONCURRENCY --batch BATCH --until-settled, --no-until-settled --max-extend MAX_EXTEND --nights {inprocess,off} --on-night-failure {stop,continue} --drift-round DRIFT_ROUND --untuned-arm, --no-untuned-arm --scout-runs SCOUT_RUNS --plan-share PLAN_SHARE --plan-train-hours PLAN_TRAIN_HOURS --plan-night-minutes PLAN_NIGHT_MINUTES --out OUT --no-endpoint-check --gate-profile {planned,as-set} planned (run's default): refuse unless RATCHET_SEQUENTIAL=1 and RATCHET_CANARY_AUDIT_RATE=0.5 are in force --repeat-of RUN_DIR run the same plan and seed again --resume RUN_DIR ```
python -m ratchet.liverun report --help ```text usage: python -m ratchet.liverun report [-h] [--bill-usd BILL_USD] [--bill-note BILL_NOTE] [--repeat REPEAT_DIR] [--publish] [--allow-partial] run_dir positional arguments: run_dir options: -h, --help show this help message and exit --bill-usd BILL_USD --bill-note BILL_NOTE --repeat REPEAT_DIR attach a repeat run's comparison --publish --allow-partial ```
python -m ratchet.liverun verify --help ```text usage: python -m ratchet.liverun verify [-h] dir positional arguments: dir options: -h, --help show this help message and exit ```
## Check the evidence offline ### `python -m ratchet.bom` **When to use it:** Make a signing key, and verify an adapter's signed bill of materials and its NOTICE.md offline. Help (--help): ```text usage: python -m ratchet.bom [-h] {keygen,verify} ... Signed bill of materials for each trained adapter. positional arguments: {keygen,verify} keygen print a new signing key and its public key verify check a BOM document (file or - for stdin) options: -h, --help show this help message and exit ```
python -m ratchet.bom keygen --help ```text usage: python -m ratchet.bom keygen [-h] options: -h, --help show this help message and exit ```
python -m ratchet.bom verify --help ```text usage: python -m ratchet.bom verify [-h] [--public-key PUBLIC_KEY] [--notice NOTICE] path positional arguments: path options: -h, --help show this help message and exit --public-key PUBLIC_KEY the base64 public key you trust --notice NOTICE a NOTICE.md to check against the notice the BOM signed ```
### `python -m ratchet.stats` **When to use it:** Plan how many golden rows a certificate needs, compute exact bounds, and recheck every number in a signed BOM. Help (--help): ```text usage: python -m ratchet.stats [-h] {plan,bound,mcnemar,check} ... Exact statistics behind Ratchet's certificates. positional arguments: {plan,bound,mcnemar,check} plan golden rows needed to certify a precision, for 0 to --max-errors wrong answers bound one-sided Clopper-Pearson lower bound on a rate mcnemar exact one-sided McNemar p-value from the discordant pairs check verify a BOM and recompute its certificate from the signed counts options: -h, --help show this help message and exit ```
python -m ratchet.stats plan --help ```text usage: python -m ratchet.stats plan [-h] [--target TARGET] [--delta DELTA] [--max-errors MAX_ERRORS] options: -h, --help show this help message and exit --target TARGET --delta DELTA --max-errors MAX_ERRORS ```
python -m ratchet.stats bound --help ```text usage: python -m ratchet.stats bound [-h] --successes SUCCESSES --n N [--delta DELTA] options: -h, --help show this help message and exit --successes SUCCESSES --n N --delta DELTA ```
python -m ratchet.stats mcnemar --help ```text usage: python -m ratchet.stats mcnemar [-h] --b B --c C options: -h, --help show this help message and exit --b B pairs where the first side wins --c C pairs where the second side wins ```
python -m ratchet.stats check --help ```text usage: python -m ratchet.stats check [-h] [--public-key PUBLIC_KEY] path positional arguments: path a BOM document (GET /ratchet/versions/{id}/bom), or - for stdin options: -h, --help show this help message and exit --public-key PUBLIC_KEY the base64 Ed25519 public key you trust ```
### `python -m ratchet.proof` **When to use it:** Check offline that a request was, or was not, in an adapter's training set, and that no golden row was trained on. Help (--help): ```text usage: python -m ratchet.proof [-h] {key,verify,heldout} ... Committed training sets: prove from an adapter's signed BOM alone that an example was, or was not, in its training or golden set. positional arguments: {key,verify,heldout} key print the example key a request's teacher answer is stored under verify check a membership or non-membership proof against a BOM heldout check that every golden example is absent from the training set options: -h, --help show this help message and exit ```
python -m ratchet.proof key --help ```text usage: python -m ratchet.proof key [-h] --messages MESSAGES [--task TASK] [--no-redact] options: -h, --help show this help message and exit --messages MESSAGES a JSON file: a messages list or a chat request body (- for stdin) --task TASK the task (default: the body's metadata.ratchet_task, else 'default') --no-redact the gateway runs with RATCHET_REDACT_PII=0 ```
python -m ratchet.proof verify --help ```text usage: python -m ratchet.proof verify [-h] --bom BOM [--public-key PUBLIC_KEY] [--expect {included,absent}] path positional arguments: path the proof (POST /ratchet/versions/{id}/proof), or - for stdin options: -h, --help show this help message and exit --bom BOM the adapter's BOM (GET /ratchet/versions/{id}/bom) --public-key PUBLIC_KEY the base64 Ed25519 public key you trust --expect {included,absent} ```
python -m ratchet.proof heldout --help ```text usage: python -m ratchet.proof heldout [-h] --bom BOM [--public-key PUBLIC_KEY] path positional arguments: path the certificate (GET /ratchet/versions/{id}/heldout), or - for stdin options: -h, --help show this help message and exit --bom BOM the adapter's BOM (GET /ratchet/versions/{id}/bom) --public-key PUBLIC_KEY the base64 Ed25519 public key you trust ```
### `python -m ratchet.receipt` **When to use it:** Verify a savings receipt, its chain and an inclusion proof offline, or seal a day from a ledger file. Help (--help): ```text usage: python -m ratchet.receipt [-h] {verify,seal} ... Verify or seal verified savings receipts positional arguments: {verify,seal} verify check a receipt (file, or - for stdin) seal seal every day that has ended in a ledger file, offline options: -h, --help show this help message and exit ```
python -m ratchet.receipt verify --help ```text usage: python -m ratchet.receipt verify [-h] [--public-key PUBLIC_KEY] [--prev PREV] [--proof PROOF] [--rows ROWS] receipt positional arguments: receipt options: -h, --help show this help message and exit --public-key PUBLIC_KEY the base64 public key you trust (GET /ratchet/bom/public-key) --prev PREV the same client's previous receipt: check the chain --proof PROOF a request's inclusion proof (GET /ratchet/receipts/{period}/proof/{id}) --rows ROWS the receipt's leaves (GET /ratchet/receipts/{period}/leaves): recompute everything ```
python -m ratchet.receipt seal --help ```text usage: python -m ratchet.receipt seal [-h] [--db DB] [--period PERIOD] [--client CLIENT] options: -h, --help show this help message and exit --db DB the ledger file (default: RATCHET_DB) --period PERIOD seal through this ledger day and print its receipt --client CLIENT with --period: print this client's receipt (default: the gateway's) ```
### `python -m ratchet.replay` **When to use it:** Ask what past traffic would have cost and served at another threshold or audit rate, from a ledger file. Help (--help): ```text usage: python -m ratchet.replay [-h] [--db DB] [--version VERSION] [--sweep SWEEP] [--audit-rate AUDIT_RATE] [--target TARGET] [--since-day SINCE_DAY] [--until-day UNTIL_DAY] [--json] Replay a version's logged traffic under another threshold or audit rate options: -h, --help show this help message and exit --db DB the ledger file (default: RATCHET_DB) --version VERSION one version (default: every version with replayable traffic) --sweep SWEEP thresholds LO:HI:STEP (default 0.30:1.00:0.05) --audit-rate AUDIT_RATE price audits at this rate (default: as logged) --target TARGET precision the suggestion needs (default 0.97) --since-day SINCE_DAY first ledger day --until-day UNTIL_DAY last ledger day --json print the reports as JSON ``` ## Scout and developer tools ### `python -m ratchet.scout` **When to use it:** Run Scout from a terminal against any gateway, as an outside client would. Help (--help): ```text usage: python -m ratchet.scout [-h] [--base-url BASE_URL] [--api-key API_KEY] [--max-results MAX_RESULTS] question Scout: multi-agent research through Ratchet positional arguments: question options: -h, --help show this help message and exit --base-url BASE_URL --api-key API_KEY --max-results MAX_RESULTS ``` ### `python -m sim.simulate` **When to use it:** Run the whole loop offline with mock models (simulation): days of traffic, nights of training, releases and rollbacks. Help (--help): ```text usage: python -m sim.simulate [-h] [--days DAYS] [--requests REQUESTS] [--threshold THRESHOLD] [--audit-rate AUDIT_RATE] [--min-shadow MIN_SHADOW] [--min-audits MIN_AUDITS] [--min-new MIN_NEW] [--drift-day DRIFT_DAY] [--bad-deploy-version BAD_DEPLOY_VERSION] [--workload {tickets,research,mixed,schema,tools}] [--seed SEED] [--release {wilson,sequential}] [--promote-evidence] [--canary-audit-rate CANARY_AUDIT_RATE] [--calibration {point,ltt}] [--cert-target CERT_TARGET] [--cert-delta CERT_DELTA] [--paired-gate] [--min-certified MIN_CERTIFIED] [--drift-alarm DRIFT_ALARM] [--drift-boost DRIFT_BOOST] [--drift-early-night] [--teacher-noise TEACHER_NOISE] [--teacher-noise-day TEACHER_NOISE_DAY] [--teacher-watch {0,1}] [--compare-release] [--receipts] [--teacher-reasoning-mean TEACHER_REASONING_MEAN] [--teacher-reasoning-cv TEACHER_REASONING_CV] [--out OUT] [--db DB] Ratchet offline simulation options: -h, --help show this help message and exit --days DAYS --requests REQUESTS requests per day --threshold THRESHOLD --audit-rate AUDIT_RATE --min-shadow MIN_SHADOW --min-audits MIN_AUDITS --min-new MIN_NEW --drift-day DRIFT_DAY day a new product launches (0 = never) --bad-deploy-version BAD_DEPLOY_VERSION fault injection: this student version is pushed corrupted to production --workload {tickets,research,mixed,schema,tools} tickets, Scout research calls, both (mixed), structured outputs (schema) or tool requests (tools) --seed SEED --release {wilson,sequential} canary rule: the Wilson bound per stage, or the anytime-valid e-process --promote-evidence with --release sequential: promote a canary stage only on an e-value that precision is above the floor --canary-audit-rate CANARY_AUDIT_RATE the canary's own audit rate, more than 0 and at most 1 (default: --audit- rate) --calibration {point,ltt} threshold rule: the point rule at 97%, or Learn-then-Test over a fixed grid --cert-target CERT_TARGET the precision a certificate must show --cert-delta CERT_DELTA a certificate's error rate --paired-gate compare each candidate with the live version row by row on the same golden rows (exact McNemar tests) --min-certified MIN_CERTIFIED reject a candidate whose certified lower bound is below this (0 = off) --drift-alarm DRIFT_ALARM the drift monitor's alarm threshold: a false alarm has probability at most 1/threshold per stream (0 = off) --drift-boost DRIFT_BOOST after a drift alarm, multiply the version's audit rate by this, up to 0.5 (1 = off) --drift-early-night after a drift alarm, Night School trains on half the usual new examples --teacher-noise TEACHER_NOISE fault injection: from --teacher-noise-day, the mock teacher changes a label with this probability (0 = never) --teacher-noise-day TEACHER_NOISE_DAY the day the mock teacher becomes noisy --teacher-watch {0,1} the teacher watch: quarantine, freeze, ceiling warning, probes and adjudication (0 = off; checks still recorded) --compare-release run twice with the same seed, once per --release rule, and compare the rolled-back versions --receipts seal each day's savings receipt and print it next to the true all-teacher cost --teacher-reasoning-mean TEACHER_REASONING_MEAN the mock teacher bills this many reasoning tokens on average instead of exactly 300 (e.g. 420) --teacher-reasoning-cv TEACHER_REASONING_CV with --teacher-reasoning-mean: the spread (coefficient of variation, default 0.3) --out OUT --db DB write the run's ledger to this SQLite file (it must not exist yet), e.g. for python -m ratchet.replay ``` ## Build, train and check the repo ### `python -m ratchet.nightschool.train` **When to use it:** Inside the training job's container only: fine-tune a LoRA adapter on train.jsonl. Night School starts it for you. Help (--help): ```text usage: python -m ratchet.nightschool.train [-h] --data DATA [--base BASE] --out OUT [--epochs EPOCHS] [--lr LR] [--rank RANK] [--max-seq MAX_SEQ] [--batch BATCH] [--targets TARGETS] Ratchet Night School: LoRA fine-tune options: -h, --help show this help message and exit --data DATA --base BASE --out OUT --epochs EPOCHS --lr LR --rank RANK --max-seq MAX_SEQ --batch BATCH --targets TARGETS comma-separated LoRA target modules, or 'all-linear'. Nemotron-H mixes attention, Mamba (in_proj/out_proj) and MLP layers; the default, attention and MLP, is what the measured adapters used. To add the Mamba projections: q_proj,k_proj,v_proj,o_proj,up_proj,down_proj,in_proj,out_proj. ``` ### `python scripts/day1_check.py` **When to use it:** Before live mode: check the model IDs, logprobs and JSON mode on Token Factory with a real key (costs a few cents). Help (the docstring of scripts/day1_check.py): ```text Day-1 check: confirm the Token Factory pieces Ratchet depends on actually work, and probe the rest of the live setup. NEBIUS_API_KEY=... python scripts/day1_check.py # the teacher check (the default) python scripts/day1_check.py [CHECK ...] [--json PATH] [--adapter-path PATH] [--adapter-name NAME] [--query TEXT] CHECK: teacher | choices | reasoning | student | tavily | cli | all - teacher (the default): the model IDs exist, the teacher answers, logprobs come back, JSON mode works, and the teacher accepts the generation parameters the gateway sends it (`max_tokens` at RATCHET_TEACHER_MAX_TOKENS, `stop`, `seed`, `temperature`, `response_format`), whether it takes `response_format` json_schema (strict) with an answer that matches the schema (a refusal is a warning: the gateway then sends json_object with the schema in the system message), and whether the embeddings model behind `/v1/embeddings` answers (RATCHET_EMBEDDINGS_MODEL at RATCHET_EMBEDDINGS_BASE, skipped when that is `off`). Costs a few cents. - choices: one Ultra call with n=2 and logprobs, as an escalation asks for them. - reasoning: per listed Token Factory model, whether `chat_template_kwargs` (top-level or inside `extra_body`) and `reasoning_effort: none` are accepted, with status, reasoning tokens and latency. Facts only, no setting. - student (RATCHET_STUDENT_BASE, RATCHET_STUDENT_KEY): loads `--adapter-path` at runtime twice (the second is vLLM's 400 "already"), samples it as production does, compares the base model's and the adapter's answers, sends the thinking switch, asks for `student-v0` and runs the 404 through the router's own predicate (`resilience.adapter_missing`), then unloads. The adapter must be trained by train.py with the --targets it will be served with (the default is attention and MLP; TARGETS_WITH_MAMBA adds the Mamba in_proj and out_proj) and --rank at most the endpoint's --max-lora-rank (16): `bash deploy/train_job.sh test` trains that adapter. - tavily (TAVILY_API_KEY): one basic search (1 credit). Never Extract. - cli: the `nebius` flags the deploy scripts use, against the installed CLI's `--help` (warnings only, read-only). Exit codes: 0 no check failed, 1 a check failed, 2 bad input or a missing prerequisite. `--json PATH` writes every measured fact without keys, URLs or hosts; review it before committing it to the evidence pack (evidence/day1/). ``` ### `python scripts/reference_check.py` **When to use it:** Check (or rewrite) the pinned reference runs that hold the simulation's behaviour still. Help (--help): ```text usage: python scripts/reference_check.py [-h] [--only {tickets-seed7,mixed-seed1,research-seed7}] [--write] [--captured-from CAPTURED_FROM] Check (or rewrite) the three pinned reference runs of v0.4.0's behaviour. options: -h, --help show this help message and exit --only {tickets-seed7,mixed-seed1,research-seed7} check this workload only (repeatable) --write rewrite the fixture from fresh runs of every workload --captured-from CAPTURED_FROM with --write: the release or commit the runs reproduce, stored in the fixture ``` ### `python scripts/regen_sim_reference.py` **When to use it:** Regenerate (or check) the seed-7 reference simulation fixture the web app's numbers are pinned to. Help (--help): ```text usage: python scripts/regen_sim_reference.py [-h] [--check] Regenerate (or check) the pinned reference simulation fixture. options: -h, --help show this help message and exit --check compare a fresh run with the fixture and write nothing ``` ## Other commands ### `python -m ratchet.watch` **When to use it:** On the VM's host, from cron: restart a hung gateway, probe it through its public URL and send signed alerts. Not needed to run a gateway. ```text usage: python3 -m ratchet.watch [-h] [--env-file ENV_FILE] {tick,probe,status,pause,resume,test-alert,notify} ... The watchdog: keep the live gateway answering until judging ends. positional arguments: {tick,probe,status,pause,resume,test-alert,notify} options: -h, --help show this help message and exit --env-file ENV_FILE ``` ### `python scripts/demo_check.py` **When to use it:** Check a live gateway from the outside, as a visitor sees it: OK, WARN or FAIL per check. ```text usage: python scripts/demo_check.py [-h] [--mode {live,mock}] [--key-env KEY_ENV] [--chat] [--json] [--timeout TIMEOUT] url Check the live gateway from outside. positional arguments: url options: -h, --help show this help message and exit --mode {live,mock} --key-env KEY_ENV --chat --json --timeout TIMEOUT ``` --- # Python API Source: https://docs.ratchet.dualmindlab.tech/api/python/ The Python API is the cascade the gateway runs, `Ratchet.handle`, called from your own Python code, so a batch job, a test or an agent in the same process can use the student and the teacher without an HTTP hop. Most orders reach the kitchen through **the pass**, the counter where a waiter checks each guest. Does the guest have a membership card (an API key)? Has this table ordered too much today (quotas and rate limits)? Did the guest already order this exact dish a minute ago (idempotency)? Should the dish go out course by course (streaming)? If you own the restaurant and you are cooking for your own staff, you can skip the counter and hand the order straight to the kitchen. The kitchen does not change. **The apprentice** still tastes three times, **the head chef** still steps in when the apprentice is unsure, **the inspector** still tastes random plates, and every head chef's dish still goes into **the recipe book** with its license note, with emails, phone numbers and card numbers blacked out first. What you give up is the counter's checks. Nobody checks your card, because you are standing in the kitchen. So only bring your own orders this way. Two jobs the waiter used to do are now yours too: ask the manager to look at the inspector's notes now and then (tick the release gate), and close the kitchen properly at the end of the night, so plates still waiting for the inspector are finished (close the router). ## How it really works The gateway in [`ratchet/gateway.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/gateway.py) is a thin layer around a few Python objects. `Runtime.reset()` builds them once at startup, and every chat request ends in one call: `Ratchet.handle(messages, task, client=..., params=..., meta=..., n=...)`. Embedding Ratchet means building the same objects yourself and calling the same method. Scout's agents already do this inside the gateway process: each agent call is `Ratchet.handle(messages, task)`, with no HTTP in between. 1. **Build `Settings`.** `Settings()` in [`ratchet/config.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/config.py) reads every `RATCHET_*` variable (and `NEBIUS_API_KEY`) from the environment; keyword arguments override them, for example `Settings(mode="live")`. `Settings()` itself does not validate. `config.load()` returns the settings plus a list of variables that did not parse, and `config.check(settings)` lists every problem the gateway's startup would find, errors first. Run the check before you spend money. 2. **Open the ledger.** For live use, open it the way the gateway and `python -m ratchet.live` do: `Ledger(s.db_path, s.golden_fraction, auto_day=True, sync=s.ledger_sync())`. `auto_day=True` numbers days from the first logged request on the wall clock; without it the caller sets `ledger.day`, as the mock example below does. Then load the task registry with `taskspec.install(ledger, s.tasks_file)`: the built-in tasks, then `RATCHET_TASKS_FILE`'s, then the API and auto tasks stored in the ledger. 3. **Make the providers.** The teacher is an `OpenAICompatProvider` on Token Factory with the teacher's timeouts and retries (`s.teacher_timeout()`, `s.teacher_retry()`). The student is an `OpenAICompatProvider` on your vLLM endpoint with the student's timeouts and **no retry**: on the request path the teacher is the student's retry. 4. **Build `Ratchet` with a resolver for the student.** `Ratchet(s, ledger, teacher, resolver)`. The resolver is a function from a version (`ledger.Version`) to `(provider, model name)`. For vLLM with runtime LoRA the model name is the adapter's name, `v.adapter_uri`, so `lambda v: (student, v.adapter_uri)` is enough. The constructor installs the license registry, refuses a teacher that is not distillable (`licenses.require_distillable`), checks every setting it reads, and builds the off-path worker pool (`SideCar`) from `Settings.sidecar()`. A bad value raises here, before any request. 5. **Call `handle`.** `result = r.handle(messages, "ticket-triage")` runs the full cascade: task resolution, version pick (canary or live), three student samples, confidence against the threshold, escalation to the teacher, the random audit, the shadow, storage of valid teacher answers, and one ledger row. It returns a `Result`. On a teacher failure with nothing to stand in, it raises. 6. **Tick the release gate.** `gate = ReleaseGate(ledger, ReleasePolicy.from_env())`, then `gate.tick()` now and then. The tick decides shadow, canary stages, promotion and rollback from the audits in the ledger. The gateway ticks after every chat request and every Scout run, at most once per `RATCHET_TICK_INTERVAL_S` (5 s in live mode), and `python -m ratchet.live tick` can also run from cron. If nothing ticks, a canary never advances and never rolls back. 7. **Close.** Before the process exits, call `r.close(drain_s)` so queued off-path audits and shadows finish, then `ledger.close()`. The objects and who owns what:
```mermaid flowchart TB S["Settings"] --> R App(["Your program"]) --> R["Ratchet"] App --> G["ReleaseGate"] R --> T["Teacher
provider"] R --> V["Student
resolver"] V --> St["Student
provider"] R --> SC["SideCar"] R --> Reg["Task
registry"] R --> L[("Ledger
SQLite")] G --> L L <--> N["Night School"] class T teacher class V,St student class L store ```
**Reading this diagram.** Your program owns two objects, `Ratchet` and `ReleaseGate`. `Ratchet` is built from the settings and holds the teacher, the student resolver, the off-path worker pool (`SideCar`), the task registry and the ledger. The release gate and Night School never talk to `Ratchet` directly: they meet it in the ledger, which is why nothing breaks if the gate ticks in another process.
One `handle` call, with the default inline checks (no `RequestMeta` passed):
```mermaid sequenceDiagram participant A as Your code participant R as Ratchet participant Student as Student participant Teacher as Teacher participant Ledger as Ledger A->>R: handle(messages,
task) R->>Ledger: read the flags Note over R: resolve the task,
pick a version alt pinned, tool or unknown task R->>Teacher: the teacher alone Teacher-->>R: answer, never stored else no version yet R->>Teacher: cold start Teacher-->>R: answer R->>Ledger: store, if valid else a version serves R->>Student: 3 samples Student-->>R: samples Note over R: confidence
vs threshold alt unsure, or the student failed R->>Teacher: escalate Teacher-->>R: answer R->>Ledger: store, if valid else confident opt audit drawn R->>Teacher: temperature 0 Teacher-->>R: reference R->>Ledger: store, if valid end end end R->>Ledger: one request row R-->>A: Result Note over A: gate.tick() ```
**Reading this diagram.** Time runs downward. Before the student is asked, the call decides who may answer: a pinned, tool or unknown task goes to the teacher alone and is never stored, and a task with no trained version yet is a cold start. When a version serves, the student samples three times and the confidence score decides: unsure answers escalate to the teacher, confident ones are served, and a random share of those are audited by the teacher. One row is logged either way, and your code ticks the gate afterwards.
  • The student
  • The teacher
The [request journey](/guides/request-journey/) walks through the same cascade from the HTTP side, and [escalation and audits](/guides/escalation-and-audits/) covers what happens on each branch. - **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)). - **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)). - **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). - **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). - **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate. - **The nightly schedule on live traffic.** Night School runs by hand today; the timer is scripted. - **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline. - **Verified prices.** The model ids and list prices in `ratchet/config.py` were confirmed on 29 Oct 2026 (`scripts/day1_check.py` passed with a real key); the GPU-hour rate is still unverified. The live example below follows the gateway's own wiring and is tested offline with fake HTTP transports; the mock example further down is the one verified to run. Run every script with `DRY_RUN=1` first. ## For engineers ### What to import | Module | Names | Source | |---|---|---| | `ratchet.config` | `Settings`, `load`, `check`, `ERROR` | [config.py](https://github.com/harshbhx/ratchet/blob/main/ratchet/config.py) | | `ratchet.ledger` | `Ledger`, `Version` | [ledger.py](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py) | | `ratchet.providers` | `OpenAICompatProvider`, `GenParams`, `Completion`, `Provider`, `UpstreamError` | [providers.py](https://github.com/harshbhx/ratchet/blob/main/ratchet/providers.py) | | `ratchet.router` | `Ratchet`, `Result`, `Choice`, `RequestMeta`, `ROUTES`, `USAGE_PARTS` | [router.py](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) | | `ratchet.release` | `ReleaseGate`, `ReleasePolicy` | [release.py](https://github.com/harshbhx/ratchet/blob/main/ratchet/release.py) | | `ratchet.taskspec`, `ratchet.tasks` | `install`, `parse_spec`, `register`, `UnknownTask`, `TaskForbidden` | [taskspec.py](https://github.com/harshbhx/ratchet/blob/main/ratchet/taskspec.py), [tasks.py](https://github.com/harshbhx/ratchet/blob/main/ratchet/tasks.py) | | `ratchet.resilience` | `UpstreamError`, `CircuitOpen` | [resilience.py](https://github.com/harshbhx/ratchet/blob/main/ratchet/resilience.py) | | `ratchet.sidecar` | `SideCar` | [sidecar.py](https://github.com/harshbhx/ratchet/blob/main/ratchet/sidecar.py) | | `ratchet.nightschool.orchestrate` | `run_night`, `NightReport` | [orchestrate.py](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/orchestrate.py) | ### A live program This mirrors `Runtime.reset()` in live mode with a student endpoint. It needs `NEBIUS_API_KEY`, `RATCHET_STUDENT_BASE` and `RATCHET_STUDENT_KEY` in the environment ([environment variables](/api/environment/)). ```python from ratchet import config, taskspec from ratchet.config import Settings from ratchet.ledger import Ledger from ratchet.providers import OpenAICompatProvider from ratchet.release import ReleaseGate, ReleasePolicy from ratchet.router import Ratchet s = Settings(mode="live") # The gateway's startup check. An empty RATCHET_API_KEYS is an error for the gateway only: nothing here takes HTTP calls. errors = [i for i in config.check(s) if i.level == config.ERROR and i.var != "RATCHET_API_KEYS"] if errors: raise SystemExit("\n".join(i.message for i in errors)) ledger = Ledger(s.db_path, s.golden_fraction, auto_day=True, sync=s.ledger_sync()) taskspec.install(ledger, s.tasks_file) # built-in tasks, RATCHET_TASKS_FILE, API tasks stored in the ledger # Timeouts and retries per role: the teacher is retried within its deadline, the student never (the teacher is its retry). teacher = OpenAICompatProvider(s.teacher_base, s.teacher_api_key, timeout=s.teacher_timeout(), retry=s.teacher_retry()) student = OpenAICompatProvider(s.student_base, s.student_api_key, timeout=s.student_timeout()) r = Ratchet(s, ledger, teacher, lambda v: (student, v.adapter_uri)) # vLLM serves each adapter under its own name gate = ReleaseGate(ledger, ReleasePolicy.from_env()) try: result = r.handle([{"role": "user", "content": "I was charged twice for order 1234."}], "ticket-triage") gate.tick() print(result.route, result.served_by, result.text, result.cost_usd) finally: r.close(s.sidecar()[2]) # RATCHET_SIDECAR_DRAIN_S, 10 s by default ledger.close() ``` Without a student endpoint (`RATCHET_STUDENT_BASE` empty), the gateway does not call an empty URL. Its resolver sends every version's calls to the untuned hosted fallback on Token Factory, with the student's timeouts: ```python fallback = OpenAICompatProvider(s.teacher_base, s.teacher_api_key, timeout=s.student_timeout()) r = Ratchet(s, ledger, teacher, lambda v: (fallback, s.fallback_student_model)) # RATCHET_FALLBACK_STUDENT ``` `Ratchet.threshold` gives that model `RATCHET_CONFIDENCE_THRESHOLD` (0.80 by default) instead of a version's calibrated threshold, because the fallback was never trained or tested on your golden set. ### A runnable example in mock mode The mock models in [`sim/mock_models.py`](https://github.com/harshbhx/ratchet/blob/main/sim/mock_models.py) need no keys, no network and no GPU. `MockFleet` is the trainer, the golden evaluator and the student resolver at once, and `MockTeacher` plays Nemotron 3 Ultra. Everything this prints comes from mock models. Save it as `embed_mock.py` and run `python embed_mock.py` after `make install`: ```python """Ratchet embedded in mock mode: mock models, no keys, no network.""" from ratchet import taskspec from ratchet.config import Settings from ratchet.ledger import Ledger from ratchet.nightschool.orchestrate import run_night from ratchet.release import ReleaseGate, ReleasePolicy from ratchet.router import Ratchet from sim.mock_models import MockFleet, MockTeacher from sim.world import make_stream s = Settings(mode="mock") ledger = Ledger(":memory:", s.golden_fraction) taskspec.install(ledger, s.tasks_file) # built-in tasks, RATCHET_TASKS_FILE, API tasks stored in the ledger fleet = MockFleet(policy=s.gen_policy()) # the mock trainer, golden evaluator and student resolver r = Ratchet(s, ledger, MockTeacher(), fleet.resolve, seed=7) gate = ReleaseGate(ledger, ReleasePolicy.from_env(defaults={"min_audits": 15})) stream = make_stream("tickets", seed=7) # synthetic support tickets def day(n: int, requests: int = 300) -> None: ledger.day = n # a mock run sets the day; a live ledger (auto_day=True) counts days from the clock for _ in range(requests): t = stream.sample(n) result = r.handle(t.messages(), "ticket-triage") gate.tick() print(f"day {n}: last request route={result.route} served_by={result.served_by}") day(1) # no version yet: every request is a cold start, answered by the teacher and stored report = run_night(ledger, fleet, fleet, gate, s.student_base_model, redact_pii=s.redact_pii) print("night 1:", report.trained, report.version, report.admitted, report.reason) day(2) # version 1 starts as a 10% canary, so the student serves a small share for d in ledger.stats_by_day(): print(d["day"], d["requests"], round(d["student_share"], 2), d["p95_ms"]) print(r.close(0.0)["by_version"]) ledger.close() ``` On day 1 every request has route `cold_start`. Night 1 trains version 1 and admits it, and because there is no live version yet it goes straight to a 10% canary, so on day 2 only a small share of requests is served by the student. ### `Ratchet(...)` constructor `Ratchet(settings, ledger, teacher, student_resolver, seed=None, *, clock=time.monotonic, on_adapter_missing=None, fallback_teacher=None, timer=time.perf_counter, sidecar=None, tasks_lock=None)` | Argument | What it does | |---|---| | `settings` | The `Settings`. The constructor calls `check_reserved()` and the readers it needs (`unknown_task_policy`, `gen_policy`, `student_breaker`, `teacher_breaker`, `sidecar`, ...), so a bad value raises `ValueError` naming its variable | | `ledger` | The `Ledger` every request is logged to and every example stored in | | `teacher` | Any `Provider`. Checked against the license registry by its configured model (`RATCHET_TEACHER_MODEL`); a closed model raises `licenses.LicenseError` | | `student_resolver` | `Callable[[Version], tuple[Provider, str]]`: the provider and model name for a version | | `seed` | Seeds `Ratchet.rng`, which draws canary picks and audits. The gateway passes 7 | | `clock` | Drives the circuit breakers. Tests pass a fake clock | | `on_adapter_missing` | Called with the version when vLLM answers 404 for its adapter. The gateway passes `live.adapter_reloader(...).request`. A failing callback never fails the request | | `fallback_teacher` | Provider for `RATCHET_TEACHER_FALLBACK_MODEL`; default the teacher's own. Used only when that model is set, and also license-checked | | `timer` | The clock `Result.wall_ms` (`requests.wall_ms`) is measured on | | `sidecar` | The off-path worker pool. Default `SideCar(*Settings.sidecar()[:2])`: 4 workers in live mode, none in mock mode. `SideCar(0)` keeps every check inline | | `tasks_lock` | The lock auto-task registration takes. The gateway passes the lock its admin API takes, so the two never interleave | One more attribute matters when you meter spend: `r.on_offpath_spend`, a callable `(client, usd)` the router calls when an off-path audit or shadow spends after its request returned. The gateway points it at its spend meter. ### `handle(...)` arguments `handle(messages, task=None, client=None, params=None, meta=None, n=1)` returns a `Result`. | Argument | Default | What it does | |---|---|---| | `messages` | required | OpenAI chat messages. The gateway validates and normalises them first (`validate_chat_request`, `normalize_messages`); embedded callers pass them as they are, so keep them plain | | `task` | `None` | A task name or a `tasks.Task`. `None` or an invalid name means `default`. An unregistered name follows `RATCHET_UNKNOWN_TASKS`: `distill` (default) treats it like any task, `teacher` sends it to the teacher alone (route `unknown_task`), `reject` raises `tasks.UnknownTask` | | `client` | `None` | The API key name the request came from. Recorded on the row and the example; `None` for internal callers | | `params` | `None` | A `GenParams`: the client's generation parameters (`max_tokens`, `stop`, `temperature`, `response_format`, `tools`, `logprobs`, ...). Each role gets what its allowlist allows ([generation parameters](/reference/api/#generation-parameters)). A `response_format` binds the task to that schema; `tools` send the request to the teacher alone (route `tools`) | | `meta` | `None` | A `RequestMeta`. Passing one also lets audits and shadows run off the request path (below) | | `n` | `1` | Number of choices. The student draws at least `n` samples and serves only when `n` of them agree; otherwise route `n_choices` and the teacher writes them ([choices and logprobs](/reference/api/#choices-and-logprobs)) | `teacher_only(messages, task=None, client=None, params=None, meta=None, n=1, route=None)` is the pinned path: the teacher answers (or the fallback teacher), the row is logged, nothing is stored, and no student, audit or shadow call runs. The gateway uses it with `route="multimodal"` for requests with a content part only the teacher reads. Before a model call, two lookups help: `r.resolve(task, params)` returns the `tasks.Task` a request with those parameters would get, and `r.task_for(task, client, params)` adds the gateway's admission checks (the unknown-task policy, raising `UnknownTask`, and the task's `clients` allow-list, raising `TaskForbidden`). `handle` itself applies the unknown-task policy only, never the allow-list. ### `Result` | Field | Meaning | |---|---| | `text`, `finish_reason` | Choice 0's answer after `tasks.clean_answer` (reasoning removed, structured answers as bare JSON) and why it ended: `stop`, `length`, `content_filter` or `tool_calls` | | `choices` | Every choice, as `Choice` objects | | `served_by` | `student` or `teacher` | | `route` | Why it was served that way: one of `router.ROUTES` (`student`, `confidence`, `cold_start`, `pinned`, `student_error`, `student_breaker`, `degraded`, `fallback_teacher`, `n_choices`, `unknown_task`, `student_disabled`, `task_disabled`, `tools`, `multimodal`) | | `version` | The student version that served it, or `None` when the teacher did | | `confidence` | The student's confidence score, or `None` without student samples | | `escalated` | True when the request went past the student: unsure (`confidence`, `n_choices`), failed (`student_error`, `student_breaker`) or held back for the task (`task_disabled`); also on a `degraded` answer. False on a cold start and on the teacher-only routes | | `cost_usd`, `student_cost_usd`, `teacher_cost_usd` | Every model call of the request at configured prices, and its student and teacher parts | | `baseline_usd` | What the teacher alone would have cost for this request | | `latency_ms`, `wall_ms` | The providers' own time, and the whole cascade on the router's `timer`; the difference is Ratchet's overhead | | `model` | The model that wrote the served answer | | `prompt_tokens`, `completion_tokens`, `reasoning_tokens`, `cached_tokens`, `total_tokens` | The served answer's usage, as if its model had answered alone. `total_tokens` is a property | | `usage_breakdown` | Tokens per model call, keyed by `router.USAGE_PARTS`: `student_samples`, `served`, `audit`, `shadow`, `arbitration`, `repair`. `None` for a call not made | | `fingerprint` | The response's `system_fingerprint` | | `degraded` | `"teacher_unavailable"` when the student's below-threshold answer stood in for a failed teacher (`RATCHET_ON_TEACHER_FAILURE=serve_student`), else `None` | | `request_id` | The ledger row id (`requests.id`), what receipts and replays name the request by. Not `RequestMeta.request_id` | | `stored`, `store_opt_out` | Whether a teacher answer became (or bumped) a training example, and if not allowed, why (`global`, `task`, `request`, `key`, `end_user`) | | `audited` | Whether a teacher audit of the served answer counted | | `overhead_pending` | True when the audit or shadow was queued to run after `handle` returned | | `task`, `schema_valid` | The resolved task's name, and whether every choice meets the request's `response_format` (`None` without one) | ### `Choice` and `RequestMeta` `Choice` is frozen: `text`, `finish_reason` (default `stop`), `logprobs` (OpenAI's `logprobs.content` entries when `params.logprobs` was set and they could be given, else `None`) and `tool_calls` (the teacher's tool calls for a tool request, else `None`). `RequestMeta` is what the caller knows beyond the messages: | Field | Default | Meaning | |---|---|---| | `request_id`, `client_request_id` | `None` | Your own ids for the request; `request_id` is recorded in `requests.request_id` | | `retry_count` | `0` | The client's attempt number. Above 0, a stored duplicate's `seen` count is not bumped | | `storable` | `True` | `False` keeps every teacher answer of this request out of the training set; the cascade is otherwise unchanged | | `store_opt_out` | `None` | Why, when `storable` is `False` (`request` when not given) | | `end_user` | `None` | The end user's hash (`privacy.EndUsers.hash`), never the raw id. A hash registered as no-train stores nothing | | `end_user_aliases`, `end_user_secondary` | `()`, `None` | Other hashes of the same end user (looked up, never stored), and a second identifier's hash (logged only) | | `idempotency_key` | `None` | The gateway acts on it before the cascade; the router never stores it | | `offpath` | `True` | Lets this request's audit and shadow run after it returns, when the sidecar has workers | | `student_only` | `False` | Serve only what the student answers alone; a request that needs the teacher raises `spend.BudgetExceeded`. The gateway sets it when `RATCHET_BUDGET_MODE=student_only` and the daily budget is spent | Storage rules apply to every caller whether or not you pass a `RequestMeta`: `RATCHET_STORE_EXAMPLES=0`, `RATCHET_NO_STORE_TASKS`, pinned tasks and tool requests store nothing, and stored text is redacted first when `RATCHET_REDACT_PII` is on (the default). ### Errors | Raised | When | |---|---| | `providers.UpstreamError` (an `httpx.HTTPError`) | The teacher's answer was needed (escalation, cold start, pinned, teacher-only) and the teacher failed after its retries, with no fallback teacher or degraded answer to stand in. `status` is the upstream status, or 504 for a timeout and 502 when Token Factory could not be reached; `type`, `code`, `param`, `message` and `retry_after` when the teacher sent them. Never the URL or headers | | `resilience.CircuitOpen` (a subclass of `UpstreamError`) | The teacher's breaker is open: 503, code `teacher_circuit_open`, `retry_after` until the next probe. It opens after `RATCHET_TEACHER_BREAKER_FAILURES` (5) failed calls in a row | | `tasks.UnknownTask` | An unregistered task under `RATCHET_UNKNOWN_TASKS=reject` | | `tasks.TaskForbidden` | From `task_for` only: the task's `clients` do not name this client | | `spend.BudgetExceeded` | With `RequestMeta(student_only=True)`, a request that needs the teacher | | `ValueError`, `licenses.LicenseError` | From the constructor: a bad setting, or a teacher that is not distillable | A student failure never raises: it is counted, and the teacher answers instead (routes `student_error` or `student_breaker`). A failed audit, shadow sample or arbitration never fails the request either. The gateway maps `UpstreamError` to its HTTP error codes; see [errors](/reference/api/#errors) in the HTTP guide. ### Tasks The registry is module state in [`ratchet/tasks.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/tasks.py), shared by every `Ratchet` in the process. The built-in tasks are `default`, `ticket-triage`, `source-triage`, `page-read`, and the pinned `research-plan` and `research-write` (Scout's planner and writer). - `taskspec.install(ledger, path)` builds the registry from the built-in tasks, the tasks file at `path` and the API and auto tasks stored in the ledger, and publishes it in one step. A bad file raises `taskspec.SpecError` listing every error. It returns a line for each stored task it skipped. - `tasks.register(taskspec.parse_spec(spec, "api"))` adds one task in memory, for example `{"name": "invoice-fields", "format": "json", "fields": {"currency": {"type": "exact"}, "total": {"type": "number", "abs_tol": 0.01}}}`. It is not saved: put tasks that Night School (another process) must grade in `RATCHET_TASKS_FILE`, or create them through the [admin API](/reference/api/#put-ratchettasksname), which stores them in the ledger. - Tests that register tasks should call `tasks.reset_builtin()` afterwards, as `tests/conftest.py` does. See [training on your data](/guides/training-on-your-data/) for the task spec rules. ### Off-path checks and `close` A request's audit and shadow run **after** `handle` returns only when you pass a `RequestMeta` with `offpath=True` (the default) and the sidecar has workers. Then, if an audit was drawn or a shadow runs, `result.overhead_pending` is True, `cost_usd` covers serving only, `audited` is False, and the ledger row is finished when the job ends. A call without `meta`, with `RequestMeta(offpath=False)`, or on a `Ratchet(..., sidecar=SideCar(0))` keeps every check inline, so every cent is in the `Result`. Scout calls `handle(messages, task)` with no meta for exactly that reason: Scout's budget guard sums `cost_usd`. | Setting | Default | Meaning | |---|---|---| | `RATCHET_SIDECAR_WORKERS` | 4 in live mode, 0 in mock mode | Worker threads for off-path checks | | `RATCHET_SIDECAR_QUEUE` | 1000 | Jobs that may wait. When full, a drawn audit runs inline (an audit is never dropped) and a shadow is skipped | | `RATCHET_SIDECAR_DRAIN_S` | 10 | How long the gateway's shutdown waits; pass the same to `close` | `r.close(drain_s)` waits at most `drain_s` seconds, gives up on the rest and returns `r.sidecar_report()`: the queue's counts (`workers`, `queued`, `running`, `done`, `failed`, `full`, `abandoned`, ...) and, per version, `audits_inline`, `shadows_dropped`, `audits_abandoned` and `shadows_abandoned`. Rows of abandoned jobs stay pending, and the release gate ignores them. `r.health()` returns the failure counters (`student_errors`, `audit_errors`, `shadow_errors`, `repair_errors`), the stand-in counts (`degraded_answers`, `fallback_answers`) and `breakers`: each breaker's `state` (`closed`, `open` or `half_open`), `failures`, `open_until` and `open_s`, by name (`teacher`, `fallback_teacher`, and `student` and `shadow` once used). The gateway does not serve this over HTTP. ### Reading the ledger The read helpers the dashboard uses work from Python too: ```python print(ledger.latency_percentiles(day=None, served_by="student", qs=(0.5, 0.95))) # nearest rank over requests.wall_ms print([(d["day"], d["p50_student_ms"], d["p95_ms"]) for d in ledger.stats_by_day()]) print(ledger.stats_by_task()) # per task: requests, student share, audits, threshold, precision, coverage print(ledger.totals()) # whole-ledger totals print(ledger.events(last=10)) # the release log, oldest first print(ledger.live()) # the live Version, or None ``` What each table holds is on [the ledger](/guides/ledger/) page. ### Testing seams Never call real APIs in tests. Every live piece takes its network and clocks as arguments: - `OpenAICompatProvider(base_url, api_key, timeout=..., retry=..., transport=httpx.MockTransport(handler), sleep=lambda _: None, clock=..., rng=...)` runs live code paths against a fake server. - A fake provider is any object with `complete(messages, model, n=1, temperature=0.0, logprobs=False, response_format=None, params=None, **_)` that returns a list of `Completion`. Give it a `base_url` attribute: breakers are kept per role and endpoint, and the endpoint is the provider's `base_url` (else its class). - `Ratchet(..., seed=7)` fixes the random draws, `clock=` drives the breakers, `timer=` the wall time, and `sidecar=SideCar(0)` keeps checks inline. `Ledger(":memory:")` needs no file. - `sim.mock_models`: `MockTeacher`, `MockStudent`, and `MockFleet` (trainer, evaluator and resolver). A live-mode test with a fake teacher, verified to run: ```python from ratchet.config import Settings from ratchet.ledger import Ledger from ratchet.providers import OpenAICompatProvider from ratchet.router import Ratchet from ratchet.sidecar import SideCar def fake_teacher(request: httpx.Request) -> httpx.Response: body = json.loads(request.content) answer = {"category": "billing", "priority": "medium", "reply": "We are looking into the double charge."} return httpx.Response(200, json={ "model": body["model"], "choices": [{"index": 0, "message": {"role": "assistant", "content": json.dumps(answer)}, "finish_reason": "stop"}], "usage": {"prompt_tokens": 60, "completion_tokens": 20, "total_tokens": 80}, }) s = Settings(mode="live", teacher_api_key="test") # live code paths, fake network ledger = Ledger(":memory:") teacher = OpenAICompatProvider(s.teacher_base, s.teacher_api_key, transport=httpx.MockTransport(fake_teacher), sleep=lambda _: None) student = OpenAICompatProvider("http://student.invalid/v1", "", transport=httpx.MockTransport(lambda req: httpx.Response(503))) r = Ratchet(s, ledger, teacher, lambda v: (student, v.adapter_uri), sidecar=SideCar(0)) result = r.handle([{"role": "user", "content": "I was charged twice for order 1234."}], "ticket-triage") assert result.route == "cold_start" and result.served_by == "teacher" and result.stored r.close(0.0) ledger.close() ``` The test suite's own patterns are in [testing](/reference/testing/). ## Common questions ### When should I embed instead of calling the gateway over HTTP? Embed when the caller is your own trusted Python code in the same process: a batch job over a backlog, a test, a notebook, or an agent framework like Scout inside the gateway. You save the HTTP hop and get the `Result` object with every field. Use the gateway when callers are other services, other languages or other people, when you need API keys, quotas or streaming, or when you want any OpenAI SDK to work unchanged. If untrusted input can reach your program, put the gateway in front of it. ### Is `handle` thread-safe? Yes. Scout's triage and reader agents call `Ratchet.handle` from a pool of 6 threads, and the gateway runs each chat request on a worker thread. The ledger guards every write, and every read the request path or the release gate makes, with one re-entrant lock over a shared SQLite connection; the router's counters and breakers have their own locks. Two things to know: `Ratchet.rng` is shared, so the order of random draws (canary picks, audits) is not reproducible across threads; and `gate.tick()` should run from one thread at a time, as the gateway does with a lock around it. ### Can two processes share one ledger file? Run one serving process per ledger: one gateway, or one embedded program, never both on the same `RATCHET_DB`. The nightly job, the cron tick and the admin CLI may open the same file from other processes; the ledger's caches see their commits. A second serving process is not guarded against yet (a single-writer lease is planned): it would double every in-memory limit, and each process would keep its own breakers. Inside a process that already runs the gateway, use its `Ratchet` as Scout does instead of opening a second `Ledger` on the file. See [two gateway processes on one ledger](/reference/runbook/#two-gateway-processes-on-one-ledger). ### What do I lose compared with the gateway? Everything that lives in the HTTP layer of `ratchet/gateway.py`: API keys and per-key task scopes, daily quotas and spend budgets, rate limits and in-flight caps, idempotency keys, streaming, body limits, OpenAI request validation and the error envelope, the `ratchet` response block and headers, end-user hashing from `user` and `safety_identifier`, the throttled gate tick, the adapter reloader, the hourly retention purge, `/ratchet/stats` and the dashboard, and Scout's budget guard. What stays is everything in the router: licenses, redaction before storage, storage opt-outs, pinned tasks, the unknown-task policy, breakers, the fallback teacher, parameter allowlists, structured outputs, cost accounting and the ledger row. The full HTTP surface is in the [HTTP API guide](/reference/api/) and the generated [OpenAPI reference](/api/http/). ### How do I run Night School from Python? In live mode, run `python -m ratchet.live night` (see [command line](/api/cli/)): it opens the ledger, installs the tasks, builds the Nebius trainer and the golden evaluator, and calls `run_night`. From your own code, call `ratchet.nightschool.orchestrate.run_night(ledger, trainer, evaluator, gate, base_model, ...)` with any trainer and evaluator, as the mock example does with `MockFleet`. It returns a `NightReport` (`trained`, `version`, `admitted`, `reason`, `eval`). It never trains on student answers, keeps the golden set out of training, and skips while a release is in shadow or canary. See [Night School](/guides/night-school/) and [the release gate](/guides/release-gate/). ### Does the embedded path cost anything extra? No. The model calls are the same as over HTTP, and `cost_usd` prices them with the prices in `ratchet/config.py`, the per-token prices of which were checked on 29 Oct 2026 (`scripts/day1_check.py`); the GPU-hour rate is still unverified. With off-path checks, the audit and shadow cost lands in the ledger row after `handle` returns, not in that `Result`. Whether self-hosting the student pays at all depends on volume; see [cost and savings](/guides/cost-and-savings/). ### What if the student is confidently wrong? The same as over HTTP: the answer is served. A random audit (5% of student answers by default, `RATCHET_AUDIT_RATE`) may catch it, and audits drive canary promotion and rollback, but only if something calls `gate.tick()`. Precision means agreement with Nemotron 3 Ultra on held-out rows, not ground truth. See [confidence](/guides/confidence/). ## Related --- # Architecture Source: https://docs.ratchet.dualmindlab.tech/reference/architecture/ Ratchet is a gateway that sits between your app and a large "teacher" model. A small "student" model, NVIDIA's open Nano base with a [LoRA](https://docs.ratchet.dualmindlab.tech/guides/glossary/#lora) adapter you own, answers each request first. If it is unsure, the request goes to the teacher. Every teacher answer is stored with its license, and every night a new student is trained on those answers and released through a series of safety gates. This page describes the code in `ratchet/` as of v0.5.0 (which includes the work listed in [AGENTS.md](https://github.com/harshbhx/ratchet/blob/main/AGENTS.md#current-status-oct-27-2026)). The reasoning behind each choice is in [Design decisions](#9-design-decisions). Terms (shadow, canary, [Wilson interval](https://docs.ratchet.dualmindlab.tech/guides/glossary/#wilson-bound), [e-value](https://docs.ratchet.dualmindlab.tech/guides/glossary/#e-value)) are in GLOSSARY.md; FR-, NFR- and INV- ids are requirement and invariant ids (REQUIREMENTS.md). A one-page overview is in [README.md](https://github.com/harshbhx/ratchet/blob/main/README.md). ## 1. Components ![Architecture: your app calls the gateway, a small student model answers first, a teacher answers when the student is unsure, answers are logged in a provenance ledger, Night School trains a new adapter from the ledger, and release gates decide whether it becomes the student.](./img/architecture.svg) The picture shows the design. The two diagrams below name the files and services: the request path, then the nightly loop. ```mermaid flowchart TB app(["Your app"]) ==>|"/v1/chat/completions"| gw["Gateway · ratchet/gateway.py
router.py · ReleaseGate.tick in release.py
pages: / · /scout · /dashboard
Scout agents, Tavily search when live"] gw ==>|"teacher calls"| tf["Nebius Token Factory
Teacher: nvidia/Nemotron-3-Ultra-550b-a55b
Judge: nvidia/nemotron-3-super-120b-a12b
Fallback student: nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B"] gw -->|"student calls
model = adapter name"| endpoint["Nebius Serverless Endpoint · vLLM
nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16
+ LoRA adapters · scripted, ready to run"] gw ---> ledger[("SQLite ledger · ratchet/ledger.py
requests · examples · versions · events")] class tf teacher class endpoint student class ledger store classDef student stroke-width:1.5px classDef teacher stroke-width:1.5px classDef store stroke-width:1.25px classDef pending stroke-width:1.5px,stroke-dasharray:5 4 classDef outline stroke-width:1.25px,stroke-dasharray:4 3 ``` The nightly side, on Nebius AI Cloud (scripted and tested offline, ready to run with `make nebius-plan`): ```mermaid flowchart TB driver["Night School driver, cron
python -m ratchet.live night | tick"] --> job subgraph cloud["Nebius AI Cloud · scripted, ready to run"] job["Serverless Job, nightly
ratchet.nightschool.train · TRL + PEFT"] --> bucket[("Object Storage at /data
night/train.jsonl · adapters/VERSION/")] bucket -.->|"runtime LoRA load"| endpoint["Serverless Endpoint
the student"] end class job pending class bucket store class endpoint student class cloud pendingZone classDef student stroke-width:1.5px classDef teacher stroke-width:1.5px classDef store stroke-width:1.25px classDef pending stroke-width:1.5px,stroke-dasharray:5 4 classDef outline stroke-width:1.25px,stroke-dasharray:4 3 ``` Status: the Nebius Serverless Job and Endpoint scripts in `deploy/` are written and offline-tested, ready to run with `make nebius-plan`, and both ran by hand on 27 and 28 Oct 2026 ([endpoint](https://github.com/harshbhx/ratchet/blob/main/evidence/student-endpoint-nebius/README.md), [Night School run](https://github.com/harshbhx/ratchet/blob/main/evidence/nightschool-nebius/README.md)); the nightly schedule has not run on live traffic. The adapters behind the measured numbers in [evidence/](https://github.com/harshbhx/ratchet/tree/main/evidence) were trained on a borrowed GPU. The teacher and the judge do run on Nebius Token Factory. Numbers from mock models are labelled Mock-mode run. | Module | Responsibility | |---|---| | `ratchet/config.py` | `Settings` (read from environment variables) and the `Price` table | | `ratchet/providers.py` | `Completion`, the `Provider` protocol, and `OpenAICompatProvider` (httpx client for Token Factory and vLLM; every failure raises `UpstreamError`) | | `ratchet/resilience.py` | `RetryPolicy` and `retry_call` (full-jitter backoff, `Retry-After`, a deadline, a read timeout, dropped connection or 504 retried once at most in all), `UpstreamError`, per-role `timeouts`, and the student circuit breaker (`Breaker`, `CircuitOpen`, `AdapterMissing`, `endpoint_down`) | | `ratchet/tasks.py` | `Task` definitions: which answer fields must match for two answers to "agree", which answers are valid (`Task.key`, `match`, `check`), answer cleaning, the built-in tasks (`BUILTIN`) and the registry (`REGISTRY`, `register`, `get`) | | `ratchet/taskspec.py` | Task specs as data: parses `RATCHET_TASKS_FILE` (JSON or YAML) and admin API bodies into `Task`s, listing every error with its path; `extends`; `install()` builds the registry at startup from the built-in, file and stored API tasks | | `ratchet/jsonschema_lite.py` | A JSON Schema validator for OpenAI's strict structured-outputs subset, standard library only: errors name schema paths, never answer values; a step budget bounds a check | | `ratchet/schema_tasks.py` | Structured outputs: the `Contract` a `response_format` sets (`contract_for`), fields derived from a json_schema (`derive`), auto task names and tasks (`auto_name`, `auto_task`), `x-ratchet` stripping (`forwardable`), and the json_object fallback and repair conversations (`downgrade`, `repair_messages`) | | `ratchet/compare.py` | `FieldSpec`: typed field comparators (`label`, `number`, `date`, `set`, `records`, ...) that normalise a value once and compare two; standard library only | | `ratchet/sqlcheck.py` | Text-to-SQL tasks (format `sql`): `SqlSpec` (a fixture in `RATCHET_FIXTURES_DIR` and how results compare), `parse_query` (one statement), `Sandbox` (a private in-memory SQLite database per fixture text that runs read-only queries under an authorizer, step, time, memory and size limits); standard library only | | `ratchet/vocab.py` | The built-in tasks' label vocabularies, shared with the prompts in `sim/tickets.py` and `ratchet/scout/prompts.py` | | `ratchet/confidence.py` | `score()` (confidence from samples, agreement counted pairwise) and `calibrate_threshold()` | | `ratchet/calibration.py` | Reads a version's per-task calibration records (`eval['tasks']`): the threshold rule the router applies per task, spec-change checks, records a re-score must keep disabled ([§6](#per-task-calibration)) | | `ratchet/router.py` | `Ratchet.handle()`: the per-request cascade, audits, shadow, cost accounting | | `ratchet/release.py` | `ReleaseGate` (golden gate with the opt-in certified floor and paired gate, shadow, canary, rollback, the anytime-valid canary rule and live alert, `evidence()`) and `wilson()` | | `ratchet/ledger.py` | `Ledger`: SQLite storage, provenance, stats | | `ratchet/licenses.py` | Teacher license registry; refuses teachers that forbid distillation. `BUILTIN` (code) plus the open teachers of `RATCHET_TEACHER_LICENSES` (`install()` at startup: allowed licenses only, never a closed model family or router alias it recognises), and `BASE_MODELS`, the base models' licenses | | `ratchet/redact.py` | `redact()`, `redact_messages()`: PII placeholders before examples are stored | | `ratchet/auth.py` | `KeyRing` (named gateway API keys, admin keys for management writes, read-only dashboard keys) and `bearer()` | | `ratchet/access.py` | Who may call each route: one table of access levels (`route_access`: public, api, dashboard, bom, metrics, admin, mock; unknown routes count as dashboard) and the scrubbing of anonymous public reads (`pseudonyms`, `scrub_stats`, `scrub_task`, `scrub_detail`); standard library only | | `ratchet/obs.py` | Request ids, W3C trace context and structured logs without content ([§12](#12-observability)): `RequestContext` in the `request_ctx` context variable, `new_context`, `upstream_headers`, `log`, `upstream_failure`, `mirror_event`, the JSON and text formatters, `configure` | | `ratchet/metrics.py` | Prometheus metrics, standard library only: `Counter`, `Gauge`, `Histogram`, `Registry.render` (text format 0.0.4) and Ratchet's metrics with bounded labels (`client_label`) | | `ratchet/openai_compat.py` | `validate_chat_request()` (typed `ChatRequest`), `normalize_messages()`, `APIError` and OpenAI's error envelope; standard library only | | `ratchet/bom.py` | Per-adapter bill of materials: `manifest()`, `document()`, Ed25519 `Signer`, `verify()`, CLI (`keygen`, `verify`) | | `ratchet/cost.py` | Utilization-aware cost accounting ([§7](#7-cost-accounting)): `cost_report()` (savings per token and by GPU-hours paid, break-even, the hosted-fallback crossover, a recommendation), interval merging (`segments`), the simulator's compute (`record_simulated_day`), CLI (`report`, `endpoint up\|down`, `close`) | | `ratchet/kpis.py` | Success metrics ([§7](#success-metrics-kpispy)): success metrics 1 to 5 and the latency target from ledger rows (`report`, one pure function per metric, `SM_TARGETS`), for `GET /ratchet/kpis`, the `kpis` block of `/ratchet/stats` and the simulator | | `ratchet/nightschool/dataset.py` | Builds train and golden sets from the ledger | | `ratchet/nightschool/train.py` | LoRA fine-tune script, run inside the Serverless Job: checks the prompt-completion rows and their rendering, then trains with TRL + PEFT; saves `ratchet_manifest.json`, `versions.json` and `NOTICE.md` next to the weights | | `ratchet/nightschool/notices.py` | `NOTICE.md` for an adapter folder: the base model's license, each teacher's, teacher outputs only and the Ratchet version (`write_notices`, `render`); the BOM and NOTICE files of a version (`write_adapter_files`) | | `ratchet/nightschool/evaluate.py` | `cascade_eval()` and `LiveEvaluator`: golden-set precision, coverage and threshold (point rule or Learn-then-Test), and the certificate | | `ratchet/replay.py` | The ledger time machine ([§7](#calibrated-baseline-and-the-ledger-replay-replaypy)): one version's logged traffic replayed at another threshold or audit rate (`load`, `at`, `sweep`, `frontier`), Hajek precision weighted by 1 / `label_p`, the calibrated baseline (`ratio`), the advisory `suggest`, CLI | | `ratchet/receipts.py` | Verified savings receipts ([§7](#verified-savings-receipts-receiptspy)): leaves of request metadata (`leaf`), the per-task ratio estimator (`calibrate`) and the bill on its lower bound (`compute`), sealing once per day and client (`seal_due`, `seal`), the offline verifiers (`verify_receipt`, `verify_chain`, `verify_proof`, `recheck`) and the CLI `python -m ratchet.receipt verify\|seal` (`ratchet/receipt.py` runs it) | | `ratchet/certificate.py` | Certified precision ([§6](#certified-precision-ratchetcertificatepy)): the Clopper–Pearson certificate, the frontier, per-task bounds, the per-row outcomes, the paired comparison and `recheck()` (what `python -m ratchet.stats check` runs); standard library and `stats` only | | `ratchet/nightschool/orchestrate.py` | `run_night()`: one night of training and release; `paired_evidence()` (the candidate against the re-scored live version) | | `ratchet/live.py` | Live Night School on Nebius (`NebiusTrainer`, `load_adapter`, `write_boms` and `publish_bom`, CLI incl. `export`) and the gateway's `AdapterReloader` | | `ratchet/gateway.py` | FastAPI app: OpenAI-compatible API (with API keys, quotas, request validation, body limit, OpenAI error envelope and streaming), request ids and failures rows (`ObservabilityMiddleware`), request lookups, `/metrics`, stats, BOM and redaction endpoints, pages (`/`, `/scout`, `/dashboard`), mock-only test controls, Scout endpoints | | `ratchet/scout/agents.py` | `Scout`: planner → searcher → triage (parallel) → reader (parallel) → writer; `Call`, `http_caller` | | `ratchet/scout/prompts.py` | System prompts, message builders and example questions, shared with the simulator | | `ratchet/scout/search.py` | `TavilySearch` (live) and `MockSearch` (mock results from `sim/research.py`), `SearchUnavailable` | | `ratchet/scout/guard.py` | `Guard`: daily spend cap, daily run cap, per-IP hourly limit and the judges' reserve for the public live gateway; live mode keeps its counters in `spend` rows and restores them at startup | | `ratchet/scout/judge.py` | `JudgeAccess` (the judge code's digest and a derived cookie key) and `AttemptLimiter` | | `ratchet/scout/credits.py` | `CreditMeter` (monthly Tavily credits paced per UTC day, a judge share) and `MeteredSearch` | | `ratchet/scout/__main__.py` | CLI: `python -m ratchet.scout "" --base-url /v1` | | `ratchet/web/` | `dist/` (the committed build of the React app in `web/`) and the hand-written fallback pages `landing.html` and `scout.html`, served at `/lite*` and when the build is missing | | `sim/` | Synthetic tickets, the mock research world (`research.py`), the generic mock world for every other request (`generic.py`: free text, schema instances, tool calls), workloads (`world.py`), mock teacher/student for the offline mock-mode run and tests | The table lists the main modules. Every module is listed in the repo map in [AGENTS.md](https://github.com/harshbhx/ratchet/blob/main/AGENTS.md#repo-map). There are two runtime modes, chosen by `RATCHET_MODE`: - **mock** (default): in-memory ledger, mock teacher and students from `sim/`, test controls enabled, mock search for Scout. No network or keys. `RATCHET_MOCK_WARMUP_DAYS` runs mock days at startup. The mock models answer the live gateway's tickets and Scout's calls from their registries and every other request from the generic mock world (free text, an instance of its `response_format` schema, a tool call), deterministically from its text; `RATCHET_MOCK_STRICT=1` makes anything else a 400. - **live**: SQLite file at `RATCHET_DB`, real teacher on Token Factory, student on your [vLLM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#vllm) endpoint (or the hosted fallback model when `RATCHET_STUDENT_BASE` is empty), [Tavily](https://docs.ratchet.dualmindlab.tech/guides/glossary/#tavily) for Scout when `TAVILY_API_KEY` is set. ## 2. Request path `Ratchet.handle(messages, task, client, params, meta)` in `router.py`. Before it runs, the gateway checks the body size, the route's access level (`_access`, one dictionary lookup for `/v1/*`, whose level checks its own key), the API key (when `RATCHET_API_KEYS` is set), checks the gateway's rate limits (the in-flight cap, the key's requests in flight and a minute; [§17](#17-spend-and-rate-limits)), validates and normalises the request (`openai_compat.py`), checks the task (the key's scope included), the key's daily quota and the US dollar budgets, in that order; it passes the key's name as `client` and the request's generation parameters as `params` (`providers.GenParams`). A request refused by any of these is not logged as a request and costs nothing (a 401, 413, 429 or 503 leaves a `failures` row, a 400 only a metric; [§12](#12-observability)): ```mermaid sequenceDiagram participant App participant GW as Gateway participant S as Student (vLLM) participant T as Teacher (Ultra) participant SH as Shadow candidate participant L as Ledger App->>GW: POST /v1/chat/completions (X-Ratchet-Task, Bearer key) GW->>GW: body size (413), API key → client name (401), rate limits (429/503), validate + normalise (400), model (404 strict), resolve task, task scope (403), quota and budgets (429) opt task is pinned (by its spec, or at runtime by an admin) GW->>T: teacher answers (NOT stored as training data) GW->>L: log_request(served_by=teacher) GW-->>App: chat.completion end GW->>GW: pick version: canary with prob traffic_pct%, else live, else none (none while the student is turned off, §18) alt a version was picked GW->>S: n=3 samples, temperature 0.7, logprobs GW->>GW: score() -> confidence alt confidence >= version threshold GW-->>GW: serve student answer opt with probability audit_rate GW->>T: audit (answer stored as training example) end else below threshold GW->>T: escalate (answer stored as training example) end else cold start (no live, canary not picked) GW->>T: teacher answers (stored) end opt a version is in shadow GW->>SH: n=3 samples, score, would it serve? opt would serve AND no teacher answer yet AND disagrees with served answer GW->>T: arbitrate (stored) end end GW->>L: log_request(...) GW->>GW: ReleaseGate.tick() GW-->>App: chat.completion (or chunks when stream=true) + "ratchet" block, X-Ratchet-Served-By, X-Ratchet-Request-Id ``` With `RATCHET_SIDECAR_WORKERS` above 0 (live mode's default), the audit and the shadow above run after the response instead, on the sidecar: the audit is still drawn where the diagram draws it, the row is logged with the checks pending, and the sidecar finishes it ([§15](#15-off-path-quality-checks)). Details that matter: - **Validation and normalisation** (`openai_compat.py`): a malformed request gets a 400 in OpenAI's error envelope before any model call. `developer` becomes `system` and text content parts become one string, so the cascade, the mock models and the ledger see the same plain messages as before; plain messages pass through as the same objects, so example hashes do not change. A request with a non-text part listed in `RATCHET_TEACHER_MODALITIES` is ineligible for the student (risk-aware routing below, [CONFIGURATION.md](/reference/configuration/#risk-aware-routing); route `ineligible:content_type`): teacher only, never stored. The legacy `functions` and `function_call` become `tools` and `tool_choice` (`openai_compat.tool_params`), and a tool request's legacy turns become tool turns (`legacy_history`). Validation, the quota check and the cascade run in a worker thread. 0. **Task resolution** (`openai_compat.resolve_task`, in the gateway after validation): the header, then `metadata.ratchet_task`, then a `model` of `ratchet/` (or, on the Azure-style route `/openai/deployments/{deployment}/chat/completions`, the deployment), then a model alias, then the first registered task whose `match` rules all hold, then a `response_format` json_schema's auto task (`gateway._schema_auto` and `Ratchet.resolve`), then (opt-in, `RATCHET_AUTO_TASK=system_prompt`) an auto task per first system message, then `default`. Pure: one regex match in full, dict lookups and plain string comparisons. Before it, `check_model` refuses a `model` Ratchet does not know with a 404 under `RATCHET_MODEL_POLICY=strict` (under `lenient` it is served as `ratchet`). The request's `model` is read nowhere else: the teacher always gets `Settings.teacher_model` and the student its adapter name. A new auto task is registered and stored (`Runtime.keep_auto_task`, under the registry lock) only once the request has passed its quota. **Task admission** (`Ratchet.task_for`, in the gateway before the quota): an unregistered name under `RATCHET_UNKNOWN_TASKS=reject` is a 400 `unknown_task`, and a task whose `clients` do not name the request's API key is a 403 `task_not_allowed`; neither is logged. Under `teacher`, an unregistered task takes the teacher-only path below with route `unknown_task`. **Pinned tasks** (`Task.pinned`, `_teacher_only`): checked first in `handle`. The teacher answers, the request is logged with `served_by=teacher`, no student version, and `cost_usd = baseline_cost_usd`. The answer is **not** passed to `add_example`, so it never becomes training data. No student, audit or shadow call is made. Scout's planner and writer (`research-plan`, `research-write`) are pinned; see [§10](#10-scout-multi-agent-research-tool). **Tool requests** (`router.uses_tools`: `params.tools`, or tool calls or results in the messages) take the same path next, with route `tools`: the teacher gets the tools (`_ask_teacher(..., tool_calls=True)` lets its answer be tool calls, which `Choice.tool_calls` returns as the provider read them), nothing is stored, and no student, audit or shadow call is made. Everywhere else a teacher tool call is a 502 `tool_calls_unsupported`. 1. **Version choice** (`_pick_version`): if a version is in `canary`, it is chosen with probability `traffic_pct / 100`; otherwise the `live` version is used. With no live version (cold start) and the canary not picked, the request goes straight to the teacher. 2. **Student sampling** (`_sample`): one call with `n` = the task's `policy.samples`, else `Settings.samples` (default 3; `Task.sample_count`, so a task is served with the sample count it was calibrated on; the shadow draws the same) and `confidence.STUDENT_SAMPLING` (`temperature=0.7`, `logprobs=True`), the constant the [golden eval](https://docs.ratchet.dualmindlab.tech/guides/glossary/#golden-set) also uses, plus the request's generation parameters as the student's role allows (below). The model name sent is whatever the resolver returns. In live mode that is `version.adapter_uri`, which is the [LoRA](https://docs.ratchet.dualmindlab.tech/guides/glossary/#lora) adapter name. A request for `n` > `Settings.samples` choices draws `n` samples in the same call, but the confidence is scored on the first `Settings.samples` of them, the statistic the threshold was calibrated on ([n choices](#n-choices-and-token-logprobs)). When the version's current per-task record for the task serves nothing (the release gate disabled it, or the eval had no [held-out](https://docs.ratchet.dualmindlab.tech/guides/glossary/#held-out) row of it; `calibration.held_back`), the student is not called: the request escalates with route `task_disabled`, costs no student samples and has no student answer for [degraded mode](https://docs.ratchet.dualmindlab.tech/guides/glossary/#degraded-mode) to serve. The shadow still runs. 3. **Threshold**: `Ratchet.threshold(version, model, task)` (`calibration.threshold`) returns the task's own threshold when the version's eval has a per-task record for it under the task's current spec (`eval["tasks"][task]["threshold"]`, [per-task calibration](#per-task-calibration)); 1.01 when that record, or the older `eval["spec_hashes"]`, names another `spec_hash` than the task's current one (`Ratchet.spec_changed`): the student serves nothing for that task until a night scores the version under the new spec; otherwise the version's global `version.eval["threshold"]` (calibrated at training time), falling back to `Settings.confidence_threshold` (default 0.80). A record the release gate disabled, or one without [held-out rows](https://docs.ratchet.dualmindlab.tech/guides/glossary/#held-out), holds 1.01 too. The hosted untuned fallback student always gets `Settings.confidence_threshold`. A sample set with no valid answer (`Scored.valid` false: every sample unparseable, empty or cut off) is never served, whatever the threshold. For `n` > 1 choices, at least `n` samples must also share the majority answer (`confidence.majority`); a confident student that cannot fill them escalates with route `n_choices`. 4. **Audits** (`_audit`): when the student serves, the teacher is also called with probability `audit_rate` (default 0.05; mock mode raises it to at least 0.10). The audit result (`audit_agree`) is the live quality signal for canary and live versions, and the teacher's answer also becomes training data. A teacher answer that cannot be stored ([below](#storage-rules)) is not an audit: `audited=0`. With `RATCHET_DRIFT_AUDIT_BOOST` above 1, a version with an open [drift](https://docs.ratchet.dualmindlab.tech/guides/glossary/#drift) alarm is audited at its rate times the boost, up to 0.5 (`Ratchet.audit_boost`, which the release gate fills; [§6](#drift-monitor-ratchetdriftpy)); still one draw per served answer, and the boosted rate is its `label_p`. The draw is always made here; with sidecar workers the call itself runs after the response ([§15](#15-off-path-quality-checks)). 5. **Teacher calls** (`_call_teacher` → `_ask_teacher`, then `_store_example`): the teacher answer is cleaned (`tasks.clean_answer`), checked against the [storage rules](#storage-rules), the messages, the stored conditions and the answer are redacted with one placeholder mapping ([PII redaction](#pii-redaction)), and they are stored with `Ledger.add_example(task, messages, completion, teacher_model, license_name, license_url, redactions, params)`. The license comes from `Ratchet.licenses`, built when `Ratchet` is constructed with `licenses.require_distillable` for the teacher and the fallback teacher (if set), and looked up by the configured model that wrote the answer. A closed teacher raises `LicenseError` at startup. 6. **Shadow** (`_shadow`): see [state machine](#5-version-state-machine). The shadow candidate never serves. With sidecar workers it runs after the response, after the request's audit ([§15](#15-off-path-quality-checks)). 7. **Logging**: one row in `requests` per call (columns below), with the API client's name when keys are on, and what the ledger [replay](https://docs.ratchet.dualmindlab.tech/guides/glossary/#replay) reads: the threshold the student's answer was held to, `label_p` (the audit rate it was drawn at when served, 1.0 when it escalated), `student_agree` (from the escalation's or a random audit's primary-teacher answer, never from an [arbitration](https://docs.ratchet.dualmindlab.tech/guides/glossary/#arbitration), and from an escalation only when the teacher answered it under an audit's conditions: one choice at temperature 0, without the client's sampling parameters) and `baseline_est_usd` (the formula's all-teacher cost). Logging only: no extra model call and no random draw. 8. **Gate tick**: the gateway calls `rt.tick()` after every chat-path request (`/v1/chat/completions`, each prompt of `/v1/completions`, `/v1/responses`) and after every Scout run. It runs `gate.tick()` under a lock at most once per `RATCHET_TICK_INTERVAL_S` (live mode 5 s; mock mode every call) and skips, without waiting, while another thread is ticking ([§11](#11-hot-path-and-performance)). In live mode, `python -m ratchet.live tick` can also be run from cron. **Risk-aware routing** (`ratchet/routing.py`). A task's `routing` block is parsed once, when the task is defined (`Task.routing_rules`), so `handle` runs it in this order: pinned or teacher-only (step 0) → **eligibility** (`Routing.ineligible`: more input than `student_max_input_tokens`, or a content part the student does not read, such as an image admitted by `RATCHET_TEACHER_MODALITIES`; the teacher answers through `Ratchet._ineligible`, route `ineligible:`, before a version is picked, so no canary draw, no student call, no audit or shadow; stored only for an over-length request with `store_ineligible`, through `_store_example` and its rules) → version choice (step 1) → sampling (step 2) → the task's threshold (step 3) → **`escalate_when`** (`Routing.escalation` on the key `confidence.score` already computed for the served sample: the first rule that holds sends a confident answer to the teacher, route `rule:`, stored as any escalation, never answered in degraded mode) → serve, with the task's `policy.audit_rate` in place of `audit_rate` (step 4), or escalate. The shadow applies the same rules to its own answer. A task with **critical classes** is answered by the student only on a version whose per-task record measured them (`calibration.critical_measured`; else route `task_disabled`, and never by the hosted untuned model, which `_held_back` keeps from being sampled for it); the golden eval writes those measurements (`Routing.misses`, `eval['tasks'][task]['critical']`) and the release gate enforces them at admit (`ReleaseGate.critical_failure`, in `task_failures`) and at a night's re-score (`ReleaseGate.critical_holds`, applied by `orchestrate.rescore` in the same write). The gateway sends a request with a part only the teacher reads through `handle` too, so its route says why the student did not answer. **Generation parameters**. `Settings.gen_policy()` (`providers.GenPolicy`) turns the client's parameters into a `GenPlan` once per request (a request without any, from Scout or the simulator, reuses one built at startup): the student's (`RATCHET_STUDENT_PARAMS`: `max_tokens`, `stop`, `seed`, `response_format`, `top_logprobs`; never its temperature, `logit_bias`, logprobs or `n`), the serving teacher's (`RATCHET_TEACHER_PARAMS`: every standard field, the tool-call fields included since, at the client's temperature or 0, or 0.7 for `n` > 1 choices; `logprobs` and `top_logprobs` only when the client sends them) and the audit's and arbitration's (the same at temperature 0, without `top_p`, `seed`, the penalties, `logit_bias` and logprobs). Every call has an output cap: `RATCHET_STUDENT_MAX_TOKENS` (1,024) and `RATCHET_TEACHER_MAX_TOKENS` (8,192), and the teacher gets `RATCHET_TEACHER_REASONING_HEADROOM` (0) tokens above the client's `max_tokens` for reasoning; when it does, a visible answer longer than the client's limit is marked `length` (served whole, not stored). The model is never the client's: always `Settings.teacher_model` or the version's adapter. The [golden eval](https://docs.ratchet.dualmindlab.tech/guides/glossary/#golden-set) builds its student calls with the same policy from the conditions stored with each example. `Result.finish_reason` is the served answer's, and the gateway returns it (`tool_calls` only for a choice with tool calls, `function_call` for a legacy request; a teacher tool call to a request without tools is a 502 `tool_calls_unsupported`). **Upstream calls**. Each role has its own client settings. The teacher and judge: a 120 s read timeout (`RATCHET_TEACHER_TIMEOUT_S`), 5 s to connect, and up to two retries (`RATCHET_TEACHER_RETRIES`) within 180 s (`RATCHET_TEACHER_DEADLINE_S`). The student on the request path: 15 s to answer and 2 s to connect (`RATCHET_STUDENT_TIMEOUT_S`, `RATCHET_STUDENT_CONNECT_TIMEOUT_S`) and no retries, because the teacher is its retry; the hosted fallback model plays the student with the same settings. `resilience.retry_call` retries connection failures and 408, 409, 429 and 5xx, waiting full jitter (`uniform(0, min(8, 0.5 × 2^i))` seconds) or the server's `Retry-After` (a wait over 8 s, or one that would end past the deadline, returns the error at once); a read timeout is retried once, because those tokens may be billed; any other 4xx and an unreadable 2xx body never. The deadline also caps each attempt's timeouts. Whatever fails ends as an `UpstreamError` with a status, the endpoint's OpenAI `type`, `code` and `param`, and a clipped message, never the URL. **Failure handling**. A request fails only when the call whose answer it needs fails: the teacher on the escalation, cold-start, pinned or teacher-only path, when no opt-in stand-in answers instead ([below](#when-the-teacher-is-down)). That raises out of `handle`, and the gateway's `_map_upstream` answers by cause: 400 with the teacher's message and code when the request cannot succeed (an upstream 400, 413 or 422, such as `context_length_exceeded`), 429 with the teacher's `Retry-After` for a rate limit, 504 `teacher_timeout`, 502 `upstream_auth_error` or `upstream_not_found` for the gateway's own key or model setting, 503 `teacher_circuit_open` with the breaker's wait while the teacher's breaker is open, and 502 `teacher_unavailable` otherwise; `X-Should-Retry: false` where a retry cannot help. Nothing is logged for the request. Every other upstream call fails open: | Call | When it raises | Recorded | |---|---|---| | Student samples | Escalate to the teacher; a 404 for the adapter also queues its reload | `escalated=1`, route `student_error`; `student_errors`; `student_error` event; the student breaker counts it when the endpoint looks down ([below](#student-circuit-breaker)) | | Teacher audit | Serve the student's answer; no arbitration call for this request either, since the teacher just failed | `audited=0`, `audit_agree` NULL, no cost for the call; `audit_errors`; `audit_error` event (none of these while the teacher's breaker is open: the call is skipped, not made) | | Shadow samples | Serve as if no shadow ran | No shadow fields; `shadow_errors`; `shadow_error` event (`sampling`); counted by the shadow's own breaker | | Shadow arbitration | Serve the live answer | `shadow_serve=1`, `shadow_agree` NULL; `shadow_errors`; `shadow_error` event (`arbitration`); skipped without a count while the teacher's breaker is open | The counters are per process and thread-safe; `Ratchet.health()` returns them. Each event is written on the 1st, 10th, 100th and 1000th failure of its kind, so an outage cannot flood the release log, and an optional-call event names the exception class and HTTP status, never the message (an httpx message holds the endpoint URL). In the release gate, a failed audit is simply not an audit: audits are sampled at random, so dropping one shrinks the sample without biasing it. A failed arbitration is different, because arbitration only runs when the shadow disagrees with the served answer: `Ledger.shadow_quality` counts such a row in `n`, coverage and the would-serve evidence, and as a disagreement in precision (`unarbitrated` in its result), so a failing teacher can get a candidate rejected but never promoted. A row whose shadow verdict is unknown (`shadow_serve` NULL) counts toward nothing. ### Other OpenAI endpoints `gateway._serve` is the chat path after validation: normalisation, the model check, task resolution and admission, the quota, `Ratchet.handle`, `rt.tick()` and the `ratchet` block. Three routes call it: - `/v1/chat/completions` (and the Azure-style route): `validate_chat_request`, then `_serve`, then `_chat_response` builds the `chat.completion` or its chunks. - `/v1/completions`: `validate_completion_request` turns each prompt into a `ChatRequest` with one user message (`CompletionRequest.requests()`); `_legacy_complete` runs `_serve` once per prompt, in order, the first call checking the quota for the whole batch (`_quota(client, need)`), starting a prompt only while the batch has run for less than `RATCHET_TEACHER_DEADLINE_S` (on `Runtime.clock`); an error after the first prompt becomes `_batch_stopped` (the cause's status and code, a message naming the prompts already answered, `X-Should-Retry: false` in place of `Retry-After`), so an SDK does not resend prompts that were billed and stored; then `_completion_response` builds the `text_completion` (choice `p * n + j`) or its chunks with `_stream_pieces`, as the chat stream does. - `/v1/responses`: `validate_responses_request` turns `instructions`, `input` items, `text.format`, function tools and the sampling fields into one `ChatRequest` (errors renamed to the Responses field by `as_responses_error`); `_respond` runs `_serve` and builds the `response` object (`response_output`, `response_object`). Nothing is kept: no response id can be looked up again. So a completion or a response is answered, audited, logged and stored exactly as the chat request it becomes, and whatever changes the chat path changes them too. `/v1/embeddings` does not run the cascade: `_embed` validates, checks the model against `RATCHET_MODEL_POLICY` (`check_embeddings_model`; the configured model always answers), checks the quota, calls `Runtime.embedder` (`OpenAICompatProvider.embed` with the teacher's timeouts and retries, or `sim.mock_models.MockEmbedder`), and logs one row (task `embeddings`, reserved in `tasks.RESERVED_NAMES`; served_by `upstream`; route `passthrough`; cost = baseline in the teacher column; `stored` 0). `/v1/moderations` and `/v1/responses/{id}` answer 404 `unsupported_endpoint`; any other `/v1/*` path stays `unknown_url`. ### n choices and token logprobs `handle(..., n=k)` (the gateway passes the request's `n`, at most `RATCHET_MAX_N`) puts `k` in the `GenPlan` (`plan.n`). The student's call asks for `max(Settings.samples, k)` samples; `score` runs on the first `Settings.samples`, and `confidence.majority` picks the choices: the scored best sample, then the other samples with the same canonical answer by token probability, or None when fewer than `k` agree. When the teacher answers, `_ask_teacher` asks it for `k` choices (`CHOICES_TEMPERATURE`, 0.7, when the client set no temperature) and `_call_teacher` stores only choice 0 (`TeacherAnswer.choices` holds them all), so a request still adds or bumps at most one example. Audits and arbitrations are one answer at temperature 0 and judge choice 0, the shadow samples `Settings.samples` as calibrated, and a degraded answer needs `k` agreeing samples. `Result.choices` holds the `router.Choice` objects (`text`, `finish_reason`, `logprobs`); `usage` adds up the choices, `usage_breakdown.served` lists them, cost prices whole calls and the baseline is one teacher answer per choice. With `params.logprobs`, `providers.served_logprobs` turns each served completion's `logprobs_content` (the server's list, kept as it came) into OpenAI's entries: it keeps only the tokens that spell the served text, dropping the ones `clean_answer` removed (reasoning, a fence, edge whitespace; a token is cut only by whitespace), and gives None when they cannot be told apart. The gateway writes `logprobs: {content, refusal: null}` for a client that asked, and streams each choice in turn with a few tokens and their logprobs per content chunk. ### Structured outputs `Ratchet.resolve(task, params)` (the gateway calls it through `task_for`, before the quota) turns a request's `response_format` into a `schema_tasks.Contract` and binds it to the request's task (`Task.bound`: a copy with the same rules and `spec_hash`, so thresholds, per-task records and the registry see the same task). A registered name wins; otherwise a json_schema request gets its auto task, derived from the schema (`schema_tasks.derive`), registered and stored under the gateway's `tasks_lock` the first time (`_auto_task`; capped by `RATCHET_MAX_AUTO_TASKS`) and a dictionary lookup afterwards. `schema_tasks.forwardable` strips `x-ratchet` annotations before any model call and before the conditions are stored. Under a bound task an answer is invalid when it breaks the contract (`Task.check`: `invalid answer`, `schema mismatch at `), so the existing rules apply unchanged: a student sample gets no vote, a teacher answer is not stored, not an audit and not a reference. `tasks.clean_answer` serves the JSON the answer holds (`tasks.json_span`, a substring, never re-serialised). `Contract.problem` remembers each answer's result by its [SHA-256](https://docs.ratchet.dualmindlab.tech/guides/glossary/#sha-256), so the votes, the agreement checks and the served choice check one answer once; `jsonschema_lite.MAX_STEPS` bounds a check. `_call_teacher` and `_teacher_only` run `_repair` on a complete teacher answer that breaks the contract (`RATCHET_SCHEMA_REPAIR`): one more call to the same teacher (or fallback teacher) with the same parameters, its own answer as an assistant turn and a user turn naming the errors. The repair's answer replaces the first unless it was cut off or is empty. `TeacherAnswer.rejected` carries the unused answer, which is priced into `cost_usd` and `teacher_cost_usd` and listed in `usage_breakdown.repair`; `TeacherAnswer.first` is the first call's answer, whose prompt counts the baseline and the client's `usage.prompt_tokens` read (`router._asked`), so the repair conversation never inflates the savings; `latency_ms` adds both calls. A repair call that raises is optional (`repair_errors`, a throttled `schema_repair_error` event); an open breaker skips it. `_teacher_complete` handles a teacher that refuses json_schema: a 400 naming `response_format` or `json_schema` is retried at once, within the same breaker slot, as json_object with the schema appended to the leading system message (`schema_tasks.downgrade`); the refused schema's digest is remembered per endpoint, and after `JSON_SCHEMA_STRIKES` (3) different schemas the endpoint is probed once with `schema_tasks.PROBE_FORMAT` (`_probe_json_schema`); only a refused probe sets the provider's `supports_json_schema` False. `resolve` uses a registered auto task only for a schema that derives its `spec_hash` (`_derives`), so one client's schema never sets the fields of another's (`auto_task_collision`). Night School binds a stored row the same way from `examples.params` (`tasks.for_row`, in `dataset.rows` and `cascade_eval`), so the eval grades as production does. `Result.task` and `Result.schema_valid` reach the response's `ratchet` block. ### Student circuit breaker A student endpoint that hangs would hold every request for its timeouts (2 s to connect, 15 s to answer) before the teacher answers. `resilience.Breaker` stops that after a few failures: ```mermaid stateDiagram-v2 [*] --> closed closed --> open: 5 endpoint failures in a row open --> half_open: 30 s later, one probe goes through half_open --> closed: the probe succeeds (wait back to 30 s) half_open --> open: the probe fails (wait doubles, up to 300 s) ``` - **What counts.** `resilience.endpoint_down` reads each failure: a timeout, a connection or protocol error, an unreadable answer, a 5xx, 408 or 429 means the endpoint looks down and counts. Any other 4xx (a prompt too long for the student, a lost adapter) is the endpoint answering about this request, so it counts as a success for the endpoint's health. An error that is not an HTTP failure (a bug in a provider, the mock world not knowing a request) counts neither way. `RATCHET_STUDENT_BREAKER_FAILURES` (5) failures in a row open the breaker; `0` turns it off. - **Open.** `_sample` checks `allow()` before calling and raises `CircuitOpen` when the breaker is open. `handle` escalates to the teacher with `escalated=1`, `confidence` NULL and route `student_breaker`; nothing is billed for the student, and it is not another `student_error`. After `RATCHET_STUDENT_BREAKER_COOLDOWN_S` (30 s) exactly one request probes the endpoint; concurrent requests still see the breaker open. A failed probe doubles the wait, up to 300 s; a successful one closes the breaker and resets the wait. A probe that never reports back is presumed lost after the longer of the wait and the student call's own longest time (its pool, connect, write and read timeouts plus 5 s, 37 s by default), so a probe on a hanging endpoint is never replaced while it can still be running; a lost probe that reports while the breaker is still half-open decides as the probe would. Only a probe's own thread decides, so a late result from a call that began before the breaker opened is ignored. - **One breaker per failure domain.** Breakers are made on first use, keyed by role and endpoint: the provider's `base_url`, or its class when it has none (a resolver may build a new provider object per call, as the mock fleet does). Live and canary on one endpoint share the `student` breaker. Shadow calls use their own `shadow` breaker, so a bad staging adapter cannot open the breaker live traffic depends on; while it is open the shadow collects no evidence, as when its samples fail. Another endpoint in the same role gets `student-2`, and so on. - **Lost adapters.** [vLLM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#vllm) forgets adapters loaded at runtime when the endpoint restarts, and answers 404 ``The model `student-v3` does not exist.`` A 404 that names the version's adapter becomes `AdapterMissing`: the request escalates (route `student_error`), the breaker records the endpoint as healthy, and `Ratchet.on_adapter_missing(version)` is called. In live mode with an endpoint that is `live.AdapterReloader.request`, which loads the adapter again on one daemon thread (`live.load_adapter`, path `RATCHET_ADAPTER_ROOT/`), at most once per adapter per 60 s, and writes `adapter_reloaded` or `adapter_reload_failed`. `python -m ratchet.live reload` stays as the manual fallback. - **Events and health.** Transitions write `circuit` events (`student: closed->open after 5 failures (ReadTimeout)`, `student: open->half_open after 30 s; one probe`, `student: half_open->open for 60 s after a failed probe (...)`, `student: half_open->closed after a successful probe`), naming the failure's class and status, never its message. Once the wait is at 300 s, the repeated `open->half_open` / `half_open->open` pair is written only for the 1st, 10th, 100th and 1000th probe at that wait, so a long outage cannot push the release history out of the dashboard's last 40 events. `Ratchet.health()["breakers"]` has each breaker's `snapshot()`. `/readyz` reports each breaker's state too, but it is a public readiness probe and not part of `/ratchet/stats`, so the dashboard's Resilience panel reads each breaker's state from its latest `circuit` event (one probe behind at most, at the longest wait) and parses the wording above; `test_resilience_panel_contract` checks the panel's grammar against these events. - **Cost on the hot path.** While a breaker is closed, `allow()` and a successful `record()` read attributes without taking a lock; the lookup is one dictionary read. Locks are taken only while failures are being counted or the breaker is open. The mock-only `POST /ratchet/demo/student-outage` takes the mock production endpoint down for 5 to 300 s, to show the breaker open and close again ([API.md](/reference/api/#post-ratchetdemostudent-outageseconds60)). ### When the teacher is down Three layers, only the first on by default: 1. **The teacher's breaker.** `Ratchet._teacher_complete` sends every teacher call through a `resilience.Breaker` named `teacher` (the fallback teacher has its own, `fallback_teacher`), made in `Ratchet.__init__` with `Settings.teacher_breaker()` (`RATCHET_TEACHER_BREAKER_FAILURES` 5, `RATCHET_TEACHER_BREAKER_COOLDOWN_S` 30 s) and the student breaker's rules: a failed call counts once, after the provider's retries, when `endpoint_down` says the teacher looks down (a timeout, no connection, an unreadable or 5xx answer, 408, 429); a 400 about the request, a 401 or a tool-call answer is the teacher answering. While open, the call raises `CircuitOpen` (503 `teacher_circuit_open`, `retry_after` from `Breaker.retry_in()`) without calling: a request that needs the teacher gets a 503 with `Retry-After` at once instead of holding a worker thread for the 180 s deadline, and an audit or arbitration is skipped as a failure would be, without counting as one. One request per wait probes the teacher; transitions are `circuit` events (`teacher: closed->open after 5 failures (UpstreamError 503)`). 2. **A fallback teacher** (`RATCHET_TEACHER_FALLBACK_MODEL`, opt-in). `_serve_teacher` asks it only for an answer that will be served (escalation, cold start, pinned, teacher-only) and only when the primary is unavailable (`resilience.teacher_unavailable`: any HTTP failure or the open breaker, but not a 400, 413 or 422 about the request or a tool-call answer). Licensed at startup (invariant 1); served with route `fallback_teacher`, its own model id and fingerprint, priced at its own price; a throttled `fallback_teacher` event names the primary's failure. Its answers are stored only with `RATCHET_STORE_FALLBACK_TEACHER=1`, then with its own license. When it fails too, the primary's error is raised. Audits and arbitrations never use it: they are the release gate's evidence and the training labels, so they are the primary's or nothing. 3. **Degraded mode** (`RATCHET_ON_TEACHER_FAILURE=serve_student`, opt-in). When an escalation still has no teacher answer and the teacher is unavailable, `handle` serves the student's own answer if it had at least one valid sample (`_may_degrade`): route `degraded`, `served_by` `student`, `escalated` 1, `audited` 0, nothing stored, `Result.degraded` `teacher_unavailable` (the response's `ratchet.degraded` and `X-Ratchet-Degraded`), and a throttled `degraded` event. Never for a request with `tools`, a cold start, a request whose student call failed or was skipped, an answer an `escalate_when` rule holds for, or the pinned, teacher-only and ineligible paths, which have no student answer. A request a stand-in answered (route `degraded` or `fallback_teacher`, `router.STAND_IN_ROUTES`) runs no shadow: its only reference would be a below-threshold student answer or another model's. With `audited` 0 too, these requests never move a release. `stats_by_day` counts them per day (`degraded`, `fallback_teacher`), and `Ratchet.health()` has `degraded_answers`, `fallback_answers` and the teacher breakers' snapshots. ### Storage rules Every training example is a teacher answer that is complete and valid. `Ratchet._store_example` is the only caller of `Ledger.add_example`; it runs on every teacher call (escalation, cold start, audit, shadow arbitration), never on the pinned and teacher-only path. 1. **Clean** (`tasks.clean_answer`). `strip_reasoning` removes complete `…` blocks in any case, then everything up to the last remaining `
` (a chat template that opens the reasoning in the prompt, so the output has only the closing tag), then an opening `` still left at the start of what remains together with everything after it (an answer cut off while reasoning); an opening tag after answer text is content (a reply that mentions the tag), since a model reasons before it answers. For a JSON task (`Task.structured`: format `json`, the default for a task with fields), an answer that is one JSON object (bare or in one fence, as it is or after what follows its first ``) is kept whole, so a tag quoted in its strings survives; otherwise one code fence that is the whole answer is unwrapped when its body parses as JSON; a free-text answer keeps its fences, since a code block is content there. The provider never reads `reasoning_content` or `reasoning` (vLLM's and Token Factory's reasoning parsers), so reasoning cannot arrive another way. 2. **Check** (`GenParams.unstorable`, `router.answer_problem`). The answer is skipped when it was written under a client's `logit_bias`, a non-zero penalty, a temperature above 1 or a vendor field (`client logit_bias`, ...: the client may choose its answer, not the label; threat model T2), when its `finish_reason` is not `stop` or `tool_calls` (`finish_reason length`: cut off by the token limit), when nothing is left after cleaning (`empty answer`), when a JSON task's answer does not parse (`invalid answer`), or when a field of the task's spec is missing, null where it may not be, or does not normalise, such as a label outside its vocabulary (`invalid field category`; `Task.check`). String operations only, so the check costs microseconds. 3. **Redact**, then check once more (`Task.check`): a placeholder can break a structured answer (a card-like number outside quotes becomes `[CARD]`, an identifier inside a checked field becomes a label outside its vocabulary), and such an answer is skipped (`invalid after redaction`). 4. **Store** with its license and its conditions (`examples.params`: the `response_format`, `tools`, `tool_choice` and `stop` the teacher answered under, `ledger.STORED_PARAM_KEYS`; for an audit the audit's), or bump `seen` for a duplicate. The golden or train split comes from the task and messages alone, so every variant of a prompt shares it (`Ledger._split`). Under a storage opt-out nothing is stored, and that is not a skip (below). `requests.stored` is 1 for a request whose teacher answer was stored or bumped, 0 otherwise (including pinned requests), NULL for rows from before v0.5. **Storage opt-outs**. `Ratchet.store_opt_out(task, meta)` says why a request's teacher answers may not be stored, broadest first (`privacy.OPT_OUTS`): `global` (`RATCHET_STORE_EXAMPLES=0`), `task` (the spec's `store: false`, or `RATCHET_NO_STORE_TASKS`), the caller's verdict (`meta.storable` False with `meta.store_opt_out`: `key` for an API key with `store=0` in `RATCHET_KEY_LIMITS`, `request` for `store: false`, `X-Ratchet-Store: none` or `metadata.ratchet_store` `"false"`; the gateway decides these in `_request_meta`), or `end_user` (any hash the request's end user may be registered under is in `no_train`: its attributed hash, its `user` beside its `safety_identifier`, and the every-client scope; one query). `handle` settles it once at the start on its own copy of the meta (`_opted_out`), logs it in `requests.store_opt_out` and returns it in `Result.store_opt_out`; `_store_example` asks again before writing, so an end user registered while the request waited for the teacher is still kept out and reported as `end_user`, and an opted-out request never reaches the lookup twice. Everything else runs as it would: the student may serve, audits record `audit_agree`, shadows are arbitrated (release safety does not depend on storage), costs are counted. Nothing in a request can turn an opt-out back into storage (`store: true` is the default spelled out), and an opted-out duplicate neither bumps `seen` nor adds its client. **End users**. The gateway hashes the request's `safety_identifier`, else its `user`, with `privacy.EndUsers.hash`: HMAC-SHA256 under the deployment's key (`RATCHET_END_USER_KEY`, or 32 random bytes generated once into the ledger's `meta` table by `privacy.end_users`), over the API key's name, a NUL and the value, cut to 16 hex digits. Only that hash reaches the router (`RequestMeta.end_user`), `requests.end_user` and `examples.end_user`; the raw value never leaves `gateway._request_meta`. Scoping by key name keeps one client's end users apart from another's: the same identifier from two clients is two hashes, so the operator cannot link them and one client cannot opt out another's users. An admin's no-train registration for every client is one hash in the `*` scope (`privacy.EVERY_CLIENT`), which every request also looks up, so keys added later are covered. Renaming a key unlinks its client's registrations; at startup `privacy.check_no_train_scopes` counts registrations no current scope matches and writes a `no_train_unmatched` event once per change. A changed key unlinks earlier hashes, and `privacy.end_users` writes an `end_user_key` event with the number of no-train registrations that stopped matching. Examples record their first sender (`examples.client`, `examples.end_user`); a duplicate from another client adds a row to `example_clients`. A skipped answer is still served (the client asked for it, and with `finish_reason length` it is all there is), but it is not an audit (`audited=0`, counted neither for nor against the student, and the shadow is compared with the served answer as if no audit had run) and an escalation or arbitration answer like that settles no shadow disagreement (`shadow_agree` NULL, which the gate counts against the shadow, as for a failed arbitration). Each skip writes a `skip_example` event (`ticket-triage: finish_reason length`) on the 1st, 10th, 100th and 1000th skip of each reason, counted per process under the same lock as the failure counters; the detail never quotes the answer. On the student side, a sample whose `finish_reason` is not `stop` or `tool_calls` (the same rule) gets no vote in `confidence.score`, and an empty answer is invalid for every task. The response text is the served answer after `tasks.clean_answer`: no reasoning, and for a structured task bare JSON. With `stream: true` the gateway sends it as OpenAI `chat.completion.chunk` events after the cascade has decided; the cascade cannot stream the student's tokens as they arrive, because confidence is scored on the whole answer ([API.md](/reference/api/#streaming)). The response has every field of OpenAI's. `usage` is the served answer's, as if its model had answered alone: the chosen student sample's tokens, or the teacher's call with its reasoning tokens (part of `completion_tokens`) and cached prompt tokens; a count the model did not report is estimated at about 4 characters a token. `ratchet.usage_breakdown` lists every call of the request (`student_samples`, `served`, `audit`, `shadow`, `arbitration`, and `repair` for teacher answers a schema repair left unused), which `cost_usd` prices. `system_fingerprint` is a hash of what decides the answer (`router.fingerprint`: the student's adapter, version, training-set hash and threshold, or the teacher's model), so it changes when a promotion or rollback puts another version in front. The router computes all of it in `Result` from the completions it already holds; the gateway only shapes it (`openai_compat.usage_object`, `gateway._envelope`). With `stream_options.include_usage` the stream ends with a chunk that has no choices, the usage and the `ratchet` block. ### Human labels and feedback People's verdicts and labels reach the ledger without the student's text ever becoming a target. The gateway gives every chat request an id (`req_` + 32 hex, `requests.request_id`, returned as `ratchet.request_id`; the completion id has the same digits). `POST /v1/ratchet/feedback` (`feedback.submit`) records a verdict on one of the caller's requests in the `feedback` table, one row per request; human precision per version (route `student` only) is reported with Wilson bounds and alerted on with an anytime-valid e-value (`human_alert`, once per version), never read by the release gate. A correction (with the request's messages, conditions and every end-user identifier the request carried) goes through the chat route's validation and normalisation, then `feedback.prepare` (clean, `Task.check`, redact, hash) and `feedback.store_label`, the only writer of human and imported labels besides `Ratchet._store_example`: a pinned task, the opt-outs (`privacy.store_opt_out` and, for a correction, the request's recorded opt-out and `Ratchet.store_opt_out`) and an erased hash keep it out; on an existing example a label replaces the stored one only when it ranks higher (teacher < correction < import) or ranks the same and comes from the same writer. `store_label` reads the stored label, decides and writes in one write transaction (`Ledger.atomic`: the ledger's lock and `BEGIN IMMEDIATE`), so the importer and the gateway sharing the file cannot interleave. `python -m ratchet.importer` feeds files through the same `prepare` and `store_label`, as attested human labels or an open teacher's outputs (`licenses.require_distillable`, closed families refused); `--golden-only` rows are golden for good, each checked against the ledger as it is when that row is written (`Ledger.prompt_status`, in the same transaction). A label that is no occurrence of its request (an import, a correction of a request already counted) is inserted with `seen` 0, so only real requests move a prompt to FREQUENT. ### Retention and erasure . Stored data can be removed without stopping anything, and without breaking Night School's rules. - **Tombstones.** `Ledger.purge_examples` removes an example's `messages`, `completion` and `params` and keeps the row: hash, task, split, seen count, teacher, license, client and `canon_hash` stay, with `purged_ts`, `purge_reason` and `purged_train`. A row with conditions keeps its prompt key (`prompt_hash`), so `dataset.rows` still adds its `seen` to its request's frequency (: a purge never moves another variant of a frequent request back to golden) while leaving the row out of both sets. - **Erasure** (`privacy.erase`; `POST /ratchet/admin/privacy/erase`, `DELETE /v1/ratchet/examples`, `python -m ratchet.admin erase`). Selects by `client` (first sender or `example_clients`), end user (HMACs in the key's scope), `hashes` and `contains` (LIKE on the stored text, wildcards escaped), each narrowing the scope; under the ledger's lock it computes, per row, whether it counted as a training row (`dataset.counts_as_train`) and whether it is shared with another client (`RATCHET_ERASURE_SHARED`: `delete` purges, `keep` unlinks the erased party), then `Ledger.record_erasure` writes everything in one transaction: the purge (also clearing `end_user`), `tombstones`, `erasure_items`, the unlinking, the erased end users' request rows (`end_user` and `end_user_secondary` cleared) and the `erasures` row. `add_example` refuses a tombstoned hash from then on (returns None; the router skips it as `erased`). A row stored in another redaction style or with the detector plugin also has its key under the default redaction (`canon_hash`); the erasure tombstones it too and `add_example` checks both, so a style or plugin change never lets an erased example back. A dry run computes the same counts and writes nothing. - **Affected versions** (`Ledger.affected_versions`): a version with a committed training set (`version_leaves`) by membership; otherwise every version created after the example was stored, for examples that counted as training rows. Reported in the result, the `erasures.tainted` column and the `erase` event; versions and their signed BOMs are never changed. A night reads its rows once (`dataset.read_snapshot`): the manifest, the trainer's rows and the commitments come from that read. Right before training it drops the rows an erasure tombstoned since (`erasure_honoured`), and once the version's leaves are written, an erasure recorded since the read that purged one of them is marked pending with the version tainted (`Ledger.taint_late_erasures`, event `erasure_tainted`): an erasure during training cannot take its example out of that run, and its own list of affected versions could not name the version yet. - **Retention** (`privacy.apply_retention`). Examples whose `last_seen_ts` is older than `RATCHET_EXAMPLE_RETENTION_DAYS` are purged with reason `retention` (a later storable duplicate restores the text with its own provenance, `add_example`). Request rows older than `RATCHET_REQUEST_RETENTION_DAYS` are folded by `Ledger.fold_requests` into `daily_rollup` and deleted, `FOLD_CHUNK` rows per transaction, never past the `stage_since` of a version in shadow or canary, and never a row such a version served or shadowed in an earlier stage (the anytime-valid canary rule counts a version's whole life). `stats_by_day`, `stats_by_task`, `stats_by_client`, `totals`, `opt_out_stats` and `cost_by_day` add the folded sums back (`_merge`), and run their old queries unchanged while `daily_rollup` is empty. Runs hourly in the live gateway (`Runtime.maybe_purge`, on its own thread, started from the tick a request runs), at the start of every live night (`run_night(retention=)`), and from `python -m ratchet.live purge`. - **Night School after deletions.** `run_night` counts new data as the training rows now, less `Ledger.train_rows_kept` (for each version, its training rows less its own rows with `purged_train` stored before it and purged after it; the largest of those): a deletion neither stops training nor starts it, and without purges the count is v0.4's. - **Export** (`Ledger.export_page`, `privacy.export_lines`): NDJSON per table, a page of 500 rows per ledger call; examples with their provenance and clients, without end-user hashes. ### PII redaction `ratchet/redact.py` replaces identifiers with placeholders before anything is stored as training data. Patterns run in a fixed order, so specific formats win over the broad phone pattern: | Label | Matches | Extra check | |---|---|---| | `SECRET` | `sk-…`, `tvly-…`, `AKIA…`, `ghp_…`, `xox…-…`, the token after `Bearer ` | | | `EMAIL` | `name@domain.tld` | | | `UPI` | UPI id `name@okicici`, `9876543210@ybl`: a handle with no domain after it (v0.5) | not a package tag (`react@latest`) | | `IBAN` | country code, 2 check digits, 11–30 characters | mod-97 | | `CARD` | 13–19 digits with optional spaces or dashes | Luhn | | `AADHAAR` | 12 digits, first digit 2–9 | Verhoeff | | `SSN` | `123-45-6789` (excluding invalid areas) | | | `GSTIN` | Indian GSTIN `27AAPFU0939F1ZV`: state code, the holder's PAN, entity, `Z`, check character (v0.5) | | | `PAN` | Indian PAN `ABCPE1234F` | | | `IP` | IPv4 | | | `PHONE` | optional `+country`, optional `(area)`, digit groups | 10–15 digits | `redact.spans` runs each pattern once, in order, each seeing the earlier ones' matches masked, which is v0.4's replacement pattern by pattern. One addition: a match the `CARD` or `IBAN` checksum refuses is tried again shorter (ending at a space inside it, longest first) and kept when nothing joins it to the text before and either 1 to 4 characters of its run follow it, which nothing joins to more text (a CVV, an expiry month, a year), or the rest is the start of the next word; so a card typed with its CVV or expiry is found. Nothing else is retried, so a long reference number redacts exactly as in v0.4. A detector plugin (`RATCHET_REDACT_PLUGIN`, a `redact.Detector`, imported at startup) adds spans the patterns cannot find, such as names; overlapping spans merge (the earlier, then the longer, then the pattern's wins). Each span becomes a placeholder in the task's style (its spec's `privacy.redact_style`, else `RATCHET_REDACT_STYLE`): `label` (`[EMAIL]`, the default), `numbered` (`[EMAIL_1]`) or `surrogate` (`person1@example.com`). `_store_example` makes one `redact.Pseudonymizer` per example and redacts the messages (`redact_messages`), then every string value in the stored conditions, such as a stop string or a tool description (`redact_json`, which keeps object keys), then the answer (`redact`) with it, so one value gets one placeholder across the three and the example's hash never depends on the answer; it is dropped after the example is stored. A detector that fails stores nothing (`skip_example` `redaction failed`). The teacher itself receives the original. `examples.redactions` counts the placeholders per example, `Ledger.privacy_stats()` sums them for `/ratchet/stats` (with the style, the tasks that override it and whether a plugin runs), and each adapter BOM reports `examples_with_pii_redacted`. `POST /ratchet/redact` previews the result for any text, in any style. Without a plugin the redactor does not find names or street addresses. ## 3. Nightly loop (Night School) `run_night(ledger, trainer, evaluator, gate, base_model, min_new_examples=50, task=None, retention=None, relabel=None)` in `nightschool/orchestrate.py`: ```mermaid flowchart TB start(["run_night"]) --> retention["Retention windows set (live mode)?
apply them first: event purge"] retention --> paused{{"Training paused by an operator?"}} paused -->|"yes"| stop1(["skip, stop"]) paused -->|"no"| storage{{"Storage disabled?
RATCHET_STORE_EXAMPLES=0, zero retention"}} storage -->|"yes"| stop2(["skip, stop: storage disabled"]) storage -->|"no"| watch["Teacher watch, unless frozen:
probes, then adjudication · event teacher_watch"] watch --> frozen{{"Teacher watch frozen?
teacher_drift"}} frozen -->|"yes"| stop3(["skip, stop"]) frozen -->|"no"| inflight{{"Release in flight?
a version in shadow or canary"}} inflight -->|"yes"| stop4(["skip, stop"]) inflight -->|"no"| build["dataset.build()
train rows + golden rows"] build --> enough{{"At least 50 new train rows
and some golden rows?"}} enough -->|"no"| stop5(["skip, stop
re-score live first if a golden spec changed"]) enough ==>|"yes"| recheck["Champion re-check
score the live version on today's golden set"] recheck ==> train["Commit both sets, then train
trainer.train() → student-vN+1"] train ==> evaluate["Candidate version, its leaves, its eval
and the paired comparison with live"] evaluate ==> bom["Signed bill of materials
event trained"] bom ==> admit{{"gate.admit(version)"}} admit -->|"fails a check"| rejected(["rejected"]) admit ==>|"passes"| shadow(["shadow, or canary 10% on a cold start"]) class paused,storage,frozen,inflight,enough,admit gate class build,bom store class train student class rejected,stop1,stop2,stop3,stop4,stop5 danger class shadow ok classDef danger stroke-width:1.5px classDef ok stroke-width:1.5px classDef student stroke-width:1.5px classDef teacher stroke-width:1.5px classDef store stroke-width:1.25px classDef gate stroke-width:1.5px classDef pending stroke-width:1.5px,stroke-dasharray:5 4 classDef outline stroke-width:1.25px,stroke-dasharray:4 3 ``` In detail, in the order the code runs: 1. **Retention.** With windows set (live mode), apply them first (`privacy.apply_retention`): event `purge`. 2. **Operator pause.** `training_paused` (the admin API or CLI) skips the night before anything else. 3. **Storage disabled** (`RATCHET_STORE_EXAMPLES=0`, zero retention): event `skip`, stop (`storage disabled`). 4. **Teacher watch**, unless frozen: probes, then adjudication (`relabel`; this also runs during a release): event `teacher_watch`. Frozen (`teacher_drift`): event `skip`, stop. 5. **Release in flight** (any version in shadow or canary): event `skip`, stop. 6. **`dataset.build()`**: train rows and golden rows, license-checked, with contested rows left out. 7. **New data**: new = train_examples − max(train_examples of every earlier version) + that version's training rows purged since. Below `min_new_examples` (50): event `skip`, stop, after re-scoring the live version when a golden task's spec changed (`rescored`). No golden rows: event `skip`, stop. 8. **Champion re-check**: evaluate the LIVE version on today's golden set and overwrite its eval (with `spec_hashes`; tasks the gate disabled on it stay disabled). Examples erased while it ran are dropped from the training rows (`erasure_honoured`). 9. **Set commitments**: `dataset.commitment(train_rows)` and `dataset.commitment(golden_rows)` (root, size and sorted leaves of each split). A duplicate key stops the night here, before any GPU time. 10. **`trainer.train(train_rows, base_model, name="student-v{N+1}")`** → `adapter_uri`. 11. **`ledger.create_version(state="candidate")`**, then `ledger.set_version_leaves()`: the train and golden leaves, written once (append-only). 12. **`evaluator.evaluate(adapter_uri, golden_rows)`** → `{precision, coverage, threshold, tasks: {...}, certificate, ...}` + `spec_hashes`. 13. **With a live version, `paired_evidence()`**: the candidate's test vector against the re-check's → `eval["paired"]`. 14. **`ledger.update_eval()`, then `ledger.set_bom()`**: the bill of materials from the exact train and golden rows, with the eval (certificate, frontier, paired counts), both set commitments and the teacher's self-agreement, signed if a key is set. 15. **Event `trained`.** 16. **`gate.admit(version)`** → rejected, or canary 10% (cold start), or shadow (tasks that fail their own check are disabled). - **Mock mode**: `MockFleet` (in `sim/mock_models.py`) is both trainer and evaluator. It finds each row's template from its messages and stored conditions, so a generic row (a schema, free text) trains like a ticket. - **Teacher watch** (`teacher_watch.night`): first, unless the watch is frozen. Probes ask the teacher again about up to `RATCHET_TEACHER_PROBES` (20) stored labels not yet re-tested in the current check epoch (the configured teacher's, no redaction placeholders, a checkable task, not contested; in hash order), as an audit asks (temperature 0, the example's stored conditions); each answer that could be stored is compared with the stored label as that request's first re-test of the epoch. Adjudication gives up to `RATCHET_ADJUDICATE_MAX` (50) contested labels two more teacher answers each: a strict majority of the three becomes the label (the same teacher, so provenance and license stay), and without one the label stays quarantined as unresolved, behind the ones never adjudicated. Both are evidence, not training, so they run while a release is in flight too; the first teacher error stops them, not the night. Then the release gate's freeze rule runs once more (tonight's probes may tip it), and a frozen watch skips the night before anything else: only an operator lifts it (`POST /ratchet/teacher/ack`). With `RATCHET_TEACHER_WATCH=0` none of this happens and contested rows stay in the sets. - **Early night** (`early_night`, `drift_early_night=1`): after the in-flight check, so never during a release, Night School brings the drift monitor up to date and, when a `drift` event is newer than the last `trained` one, needs `max(1, min_new_examples // 2)` new examples instead of `min_new_examples`. The `skip` and `trained` events say so. - **Live mode**: `NebiusTrainer` (`ratchet/live.py`) first checks that `RATCHET_STUDENT_BASE` is set (an adapter with no endpoint cannot be evaluated, so it stops before any spend). Then it does four things. It writes `RATCHET_DATA_DIR/night/train.jsonl`, runs `RATCHET_SYNC_CMD` (if set), runs `RATCHET_TRAIN_CMD` (default `bash deploy/train_job.sh {version}`), then hot-loads the adapter into vLLM with `POST {root}/v1/load_lora_adapter` (an "already loaded" answer is fine). `LiveEvaluator` then scores it. After a night that trained, `python -m ratchet.live night` writes the BOM to `RATCHET_DATA_DIR/boms/student-vN.json` and, with the `NOTICE.md` it signed, into the adapter's folder `RATCHET_DATA_DIR/adapters//` (`ratchet_bom.json`, `NOTICE.md`; `live.write_boms` and `publish_bom`), then runs `RATCHET_SYNC_CMD` again, so the bucket's copy of the adapter carries its provenance and its licenses (a failed sync is a `sync_failed` event and a non-zero exit). ### Datasets (`nightschool/dataset.py`) - Every example gets a fixed split when first stored: `golden` if `int(sha256[:8], 16) / 0xFFFFFFFF < golden_fraction` (default 0.10), else `train`. The split never changes, so evaluation data cannot leak into training through re-splitting. - **Frequent requests:** an example whose `seen` count is at least `FREQUENT = 3` is treated as `train`, even if its stored split is `golden`. Requests that keep repeating are too valuable to hold out. - **One prompt, every task:** `dataset.rows` groups examples by the prompt itself (`ledger.content_key`: the messages, whatever the task or conditions). `seen` adds up over the group, and a prompt whose copy under another task is golden (and not yet frequent) is golden under this one too, so one adapter never trains on a prompt that any task holds out. - Rows whose provenance is unknown or not distillable (per `licenses.lookup_source`: a teacher in the registry, or `human`, a label a person wrote) are dropped. - **Golden-only rows** (`examples.golden_only`, imported with `--golden-only`) are golden whatever their split or `seen` count, and so is every other variant of their prompt (the same task and messages under other conditions), because the trainer sees only the messages. - Rows of a task that is pinned **now** are dropped from both sets and counted (`excluded_pinned` in `manifest.json`), and `cascade_eval` skips any it is handed (invariant 10 at read time): a task pinned after it had data neither trains the student nor calibrates its threshold. Unpinning brings the rows back. - Train rows whose answer is not valid under the task's current spec (`Task.valid`: a vocabulary shrank, a field became required) are dropped and counted (`skipped_invalid` in `manifest.json`); the ledger keeps them, so loosening the spec brings them back. Golden rows stay in the set, and `cascade_eval` skips the ones it cannot grade with (`golden_invalid_refs`). - Purged rows (retention or erasure) have no text and are in neither set; their `seen` still counts toward their request's frequency ([Retention and erasure](#retention-and-erasure)). - Training is cumulative: each night trains from the **base model** on every train row so far. - Train rows whose answer holds a literal `` or `` are dropped and counted (`skipped_think_tag`): Nemotron's chat template renders such an answer without the empty think block the served prompt ends in. They stay golden rows. - Contested rows (a later teacher label of the same request disagreed; the teacher watch) are dropped from both sets and counted (`excluded_contested` in `manifest.json`) until an adjudication settles their label. `exclude_contested=False` (`RATCHET_TEACHER_WATCH=0`) keeps them. - `build()` also writes `manifest.json`: train and golden counts, `dataset_sha256` (a hash of the train rows' example hashes), `skipped_invalid`, `skipped_think_tag`, `excluded_pinned`, `excluded_contested`, and the bill of materials. ### Training format (`nightschool/train.py`) The student must be trained on the prompt vLLM renders for it. Nemotron 3 Nano's chat template turns thinking on unless told otherwise (the generation prompt then ends in `\n`), while the endpoint serves with `--default-chat-template-kwargs '{"enable_thinking":false}'` (the prompt ends in ``), and TRL only warns when the two renderings disagree. So: - `train.jsonl` (`dataset.write_training`) holds TRL's conversational prompt-completion rows: `prompt` (the stored messages), `completion` (the teacher's stored answer as one assistant message) and `chat_template_kwargs: {"enable_thinking": false}` (`dataset.TRAIN_TEMPLATE_KWARGS`, equal to `train.TEMPLATE_KWARGS` and the endpoint's default, which a test keeps equal), plus `task`, `params`, `seen` and `provenance`. `golden.jsonl` and every in-memory consumer keep the `messages` rows. - `train.py` refuses a row that would train another prompt before the model loads (`check_rows`), then renders every row with the tokenizer as TRL will and stops when the prompt is not a prefix of prompt + answer or differs from the prompt rendered for the request (`check_prefix`). TRL gets the three columns only (`columns`), with `completion_only_loss=True`: the loss covers the teacher's answer and its end-of-turn token. - It writes `versions.json` (Python, CUDA and every pinned package) and `ratchet_manifest.json` (base model, licenses, template arguments) beside the adapter; neither is part of the signed BOM. - `deploy/Dockerfile.train` installs the exact versions in `deploy/train-versions.txt` (TRL 1.16.0, no Unsloth), read as pip constraints after torch, and fails its build when the Mamba kernels do not import. ## 4. Data model (SQLite) Created by `SCHEMA` in `ledger.py` (`CREATE TABLE IF NOT EXISTS`). ### `requests`: one row per gateway call | Column | Type | Meaning | |---|---|---| | `id` | INTEGER PK | Auto-increment. Also used as the "since" marker for stages | | `ts` | REAL | Unix time | | `day` | INTEGER | `Ledger.day` at the time (set by the simulator or live gateway; see note) | | `task` | TEXT | Task name, as resolved ([request path](#2-request-path)). Indexed as `(task, ts)` and `(client, task, ts)` for `GET /v1/models` (`Ledger.recent_tasks` and `task_seen` probe them one task name at a time) | | `served_by` | TEXT | `student` or `teacher`; `upstream` for a `/v1/embeddings` pass-through (task `embeddings`, route `passthrough`, [other OpenAI endpoints](#other-openai-endpoints)) | | `student_version` | INTEGER | Version that was tried first; NULL on cold start | | `confidence` | REAL | Student confidence; NULL if no student was tried | | `escalated` | INTEGER | 1 if the student was below threshold | | `audited` | INTEGER | 1 if the teacher double-checked a student answer (0 when the audit call failed or its answer could not be stored) | | `audit_agree` | INTEGER | 1/0 for audited rows, NULL otherwise. `audited` 1 with `audit_agree` NULL is an off-path audit still pending ([§15](#15-off-path-quality-checks)), which no statistic counts | | `shadow_version` | INTEGER | Shadow candidate that ran, if any | | `shadow_serve` | INTEGER | 1 if the shadow would have served; NULL with `shadow_version` set while an off-path shadow is pending | | `shadow_agree` | INTEGER | 1 if the shadow agreed with the best available reference; NULL when its disagreement could not be arbitrated, or the teacher's reference could not be stored | | `student_model`, `shadow_model` | TEXT | The model the resolver named for the student's samples (set when they came back) and for the shadow's (set with `shadow_version`): the version's adapter, or the hosted fallback student's id without `RATCHET_STUDENT_BASE`. A version's release statistics count only rows whose model is its adapter, or NULL (rows written before v0.5) ([§16](#16-fallback-attribution)) | | `latency_ms` | REAL | The providers' time: student latency (max over samples) plus teacher latency if called (as each provider measured or reported it) | | `cost_usd` | REAL | Actual cost of this request (student + teacher + audit + shadow + arbitration), the student priced per token. With off-path checks, the serving cost until they end, then the same total an inline run writes ([§15](#15-off-path-quality-checks)) | | `baseline_cost_usd` | REAL | Estimated cost if the teacher had answered alone | | `student_cost_usd`, `teacher_cost_usd` | REAL | `cost_usd`'s two parts: the student's samples (live and shadow) and the teacher's calls (escalation, audit, arbitration, pinned); their sum is `cost_usd` up to float rounding. NULL in rows written before v0.5 ([§7](#7-cost-accounting)) | | `student_in_tokens`, `student_out_tokens` | INTEGER | The student calls' tokens (the prompt once per call, every sample's output), which price the hosted fallback student in the cost report; 0 when the student was not sampled, NULL before v0.5 | | `client` | TEXT | API key name that sent the request; NULL when keys are off and for internal calls (Scout, live gateway). Indexed with `ts` for the daily quota | | `route` | TEXT | How the request was served (`router.ROUTES`): `student`, `confidence`, `cold_start`, `pinned`, `student_error`, `student_breaker`, `degraded`, `fallback_teacher`, `tools`, `multimodal`, ... ([API.md](/reference/api/#get-ratchetstats)). NULL in rows written before v0.5; `stats_by_task` reports those as `unrecorded` | | `stored` | INTEGER | 1 if a teacher answer from this request was stored as an example (inserted, or a duplicate: its `seen` bumped, or not for a retry, [§14](#14-retries-and-idempotency)); 0 if none was (student-served without audit, pinned, or a skipped answer, see [Storage rules](#storage-rules)); NULL in rows written before v0.5 | | `uses_tools` | INTEGER | 1 for a tool request (tools offered, or tool calls or results in its messages), whatever its route; 0 on the cascade; NULL in older rows and for internal rows that do not set it. `stats_by_task` counts it as `tool_requests` | | `label_p` | REAL | The designed probability that a teacher label of the student's answer is observed: the audit rate this request was drawn at when the student served (`RATCHET_CANARY_AUDIT_RATE` for a canary when set, else `RATCHET_AUDIT_RATE`, times a drift boost when one applies), 1.0 when the student answered and the request escalated, NULL without a student answer (cold start, a failed or skipped student call, pinned) and in rows written before v0.5; the ledger replay weights by 1 / `label_p` | | `threshold` | REAL | The threshold the student's answer was compared with (`Ratchet.threshold`: the version's, the global one for the hosted fallback model, 1.01 for a task whose spec changed). NULL when the student did not answer (no version, a failed call, an open breaker, pinned) and in older rows | | `student_agree` | INTEGER | 1/0: whether the student's answer agreed with the primary teacher's valid answer from this request's escalation or random audit (equal to `audit_agree` on audited rows). NULL when no such label exists: a served answer nobody audited, an arbitration (not a random draw), a fallback-teacher or degraded answer, an invalid student or teacher answer, an escalation the teacher answered under the client's sampling parameters or for n > 1 choices (not the audit's deterministic answer), no student answer, older rows | | `baseline_est_usd` | REAL | The all-teacher cost by the token formula (the student's prompt tokens, each choice's output tokens plus `RATCHET_TEACHER_REASONING_TOKENS`, at the teacher's price) for every student answer, also when the teacher was called; equal to `baseline_cost_usd` when the student served without a teacher call. The replay calibrates it on audited rows ([§7](#calibrated-baseline-and-the-ledger-replay-replaypy)). NULL without a student answer and in older rows | | `optional_failed` | INTEGER | 1 when an optional teacher call of the request (audit, arbitration) failed; NULL until live readiness writes it | | `end_user` | TEXT | The end user's 16-hex HMAC (`privacy.EndUsers.hash`, scoped to `client`), never the raw `user` or `safety_identifier`; NULL when the request named none and in older rows. Indexed with `ts` | | `request_id` | TEXT | The gateway's id for the request (`req_` + 32 hex, `ratchet.request_id`; the completion id has the same digits), which feedback names it by; NULL for internal calls (Scout, the simulator) and in older rows. Indexed | | `end_user_secondary` | TEXT | The 16-hex HMAC of the second end-user identifier the request carried (`user` beside `safety_identifier`), in the same scope as `end_user`; a correction must name it too, so a no-train registration made with it applies. NULL with one identifier or none and in older rows; never exported; an erasure of the end user by either identifier clears both hashes. Indexed | | `store_opt_out` | TEXT | Why the request's teacher answers could not be stored: `global`, `task`, `key`, `request` or `end_user` (`privacy.OPT_OUTS`, broadest first); NULL when no opt-out applied and in older rows | | `client_request_id` (with `request_id`, above) | TEXT | The gateway's id for the request (`req_` and 32 hex digits, also `x-request-id`; index `requests_request_id`) and the client's own `X-Request-Id` or `X-Client-Request-Id` when it was well formed. NULL for internal calls (Scout, the simulator) and rows written before v0.5 ([§12](#12-observability)) | | `wall_ms` | REAL | The whole cascade on the gateway's clock (`time.perf_counter`, from the start of `handle` or `teacher_only` to the ledger write), every model call included; `wall_ms - latency_ms` is Ratchet's own time. NULL in rows written before v0.5 ([§11](#11-hot-path-and-performance)) | | `retry_count` | INTEGER | The client SDK's attempt number (`x-stainless-retry-count`): 0 for a first attempt, an internal call, a client that does not send it, and rows written before v0.5 ([§14](#14-retries-and-idempotency)) | ### `examples`: teacher-written training data with provenance | Column | Type | Meaning | |---|---|---| | `id` | INTEGER PK | | | `ts`, `day`, `task` | | When and for which task it was first stored | | `hash` | TEXT UNIQUE | `sha256(json({"task","messages"}, sort_keys))`, with `"params"` added to the object only when the example has conditions (so every hash from before v0.5 is unchanged); deduplicates identical requests under identical conditions | | `messages` | TEXT | JSON request messages | | `completion` | TEXT | Teacher answer, reasoning removed | | `teacher_model` | TEXT | Model ID of the teacher | | `license`, `license_url` | TEXT | License that permits training on this output | | `split` | TEXT | `train` or `golden` (fixed from the hash) | | `seen` | INTEGER | Times this exact request was answered by the teacher (starts at 1, +1 per duplicate) | | `redactions` | INTEGER | PII placeholders inserted before storing (0 when none were found or redaction is off) | | `params` | TEXT | JSON: the conditions the teacher answered under that define the target (`response_format`, `tools`, `tool_choice`, `stop`), redacted like the messages; NULL when there were none and in rows from before v0.5. `dataset.rows` returns it as `params` | | `origin`, `job` | TEXT | Where the example came from (`escalation`, `audit`, `arbitration`, `backfill`; NULL: not recorded, read as `live`) and the backfill job that wrote it. NULL until that feature writes them | | `contested`, `checked_epoch` | INTEGER | The teacher watch. `contested`: 0, or 1 when a later teacher label of the same request disagreed (quarantined from training and the golden set until adjudicated), or 2 when an adjudication found no strict majority (still quarantined); indexed with `task`. `checked_epoch`: the check epoch of its one comparison (each label is compared at most once per epoch), 0 before any, or −1 (`ledger.UNCHECKABLE`) for a label that is not one independent greedy teacher answer (sampled above temperature 0 or under a reasoning effort, or chosen by an adjudication) and so is never compared. 0 by default and in older rows | | `client`, `end_user` | TEXT | The API key name and the end user's hash of the request it was first stored from; NULL for internal calls (Scout, the simulator) and in older rows. Later senders from other clients are in `example_clients` | | `last_seen_ts` | REAL | When it was stored or last seen again (a duplicate moves it); retention counts from it. Older rows get their `ts` on migration | | `purged_ts`, `purge_reason` | REAL, TEXT | When its text was removed, and why: `retention` (a later duplicate that may be stored brings it back) or `erasure` (never stored again); NULL while it is kept ([Retention and erasure](#retention-and-erasure)) | | `purged_train` | INTEGER | 1 when it counted as a training row as it was purged (`dataset.counts_as_train`); Night School's new-data count and an erasure's affected versions read it after the text is gone | | `verified` | INTEGER | 1 when a person wrote the label (`teacher_model` `human`: a feedback correction or a human import); 0 for teacher answers and in older rows | | `golden_only` | INTEGER | 1 for a row held out for good (an import with `--golden-only`): golden whatever its hash says, even past FREQUENT, with every variant of its prompt; sticky on a duplicate | | `prompt_hash` | TEXT | A purged row's prompt key (`ledger.prompt_key`: its task and messages without conditions), written by the purge for rows with conditions, so its `seen` still counts toward its request's frequency | | `content_hash` | TEXT | SHA-256 of the stored messages, the same whatever the task or conditions: groups one prompt across tasks for the golden split and the frequent rule; kept when a purge removes the text | | `canon_hash` | TEXT | The example's key under the default redaction (`label` style, no detector plugin) when it was stored in another style or with the plugin and that key differs from `hash`; NULL otherwise. An erasure tombstones it with the hash, so the example stays erased under any style or plugin setting | ### `versions`: student models | Column | Type | Meaning | |---|---|---| | `id` | INTEGER PK | Version number (`v{id}` in the UI) | | `created_ts`, `day` | | Creation time | | `base_model` | TEXT | Base model ID | | `adapter_uri` | TEXT | Adapter name, e.g. `student-v3` (also the vLLM model name) | | `train_examples` | INTEGER | Size of the train set used | | `dataset_hash` | TEXT | `dataset_sha256` from the manifest | | `eval` | TEXT (JSON) | `golden_n`, `golden_agreement`, `threshold`, `coverage`, `precision`, `test_n`, `refused_n`, `golden_invalid_refs`, `tasks` (one record per task, [§6](#per-task-calibration)), `spec_hashes`, `precision_kind`, optional `judge_n` and `judge_pass_rate`, `rechecked_day` | | `state` | TEXT | `candidate`, `rejected`, `shadow`, `canary`, `live`, `retired`, `rolled_back`; `hosted` for the hosted student's row ([§16](#16-fallback-attribution)), which no release move touches | | `traffic_pct` | INTEGER | 10 or 50 in canary, 100 when live, 0 otherwise | | `stage_since` | INTEGER | Max `requests.id` when the current stage began; a stage's audits are counted only after it (`Ledger.audit_counts(id, stage_since)`; the sequential rollback counts from 0) | | `bom` | TEXT (JSON) | This adapter's bill of materials document (`manifest`, `sha256`, `signature`), written at training time | ### `tasks`: tasks written through the admin API, and auto tasks `name` (primary key), `spec` (JSON: the task as a spec file writes it, `taskspec.document`, in its own key order, because the order of `fields` is part of the spec hash), `spec_hash`, `source` (`api`, or `auto` for a task made from requests: derived from a `response_format` schema, or one per system prompt with `RATCHET_AUTO_TASK=system_prompt`, whose `updated_by` is `auto`), `created_ts`, `updated_ts`, `updated_by` (the admin key's name, never the key; NULL for an auto task). Registered again at every start (`taskspec.install`); built-in and file tasks are never stored. A new table, so older files gain it on open with no migration. Deleting a task removes its row and keeps its examples. ### `compute_intervals`: when GPUs ran `id`, `kind` (`student_endpoint`, `training_job` or `eval_job`), `start_ts`, `end_ts` (NULL while it runs), `usd_per_hour` (per GPU, as recorded, so a later price change never reprices history), `gpus`, `version` (the version a training job produced), `source` (`cli`, `api`, `deploy`, `night` or `simulation`) and `note`. Timestamps are on the ledger's clock (`Ledger.now()`): wall-clock time in a live ledger, a synthetic clock in test runs and tests (day N starts at N x 86400 s). Indexed on `(kind, start_ts)`. A new table, so older files gain it on open. Written by `compute_start` (idempotent for a student endpoint that is up, in one transaction), `compute_stop` (every open interval of the kind, or one by id) and `compute_record` (a closed interval of its own, for the simulator; it never touches an open one); rates 0 to $1,000 per GPU-hour and 0 to 1,024 GPUs. Read by `compute_intervals(kind, since, until)` and [the cost report](#7-cost-accounting). ### Evidence tables Created by `SCHEMA` for the evidence features, so later items change no migration. `version_leaves` is written by committed training sets: `run_night` writes each new version's leaves once, right after `create_version`. The drift monitor, the teacher watch, the receipts, the Scout guard and erasure write the other tables; only the clean-room backfill (`backfill_jobs`) is still waiting for its feature. | Table | Columns | Feature | |---|---|---| | `version_leaves` | `version_id`, `split` (`train` or `golden`), `key` (the example hash, hex), `value` (sha256 of its label, teacher, license and redaction count); primary key (version, split, key), indexed by `key` | Committed training sets | | `teacher_checks` | `id`, `ts`, `day`, `epoch`, `task`, `source` (`duplicate` or `probe`), `agree`, `model` (the configured teacher); indexed by `epoch`. One row per comparison of a stored label with its first re-test in an epoch (`Ledger._check_label`) | Teacher watch | | `meta` | `key`, `value` | Small ledger-wide state. The teacher watch keeps `teacher_epoch` (1 when unset), `teacher_epoch_day`, `teacher_frozen` (`1` while Night School is frozen), `teacher_model` (the id the teacher's responses last reported, at most 200 characters) and `teacher_model_ids` (the ids reported in this epoch, a JSON list) here, each changed in one transaction with its event. The drift monitor records each alarm's event here (`drift::` → the event id, the alarm's request and its day), so it is written once across processes and restarts (`Ledger.event_once`) | | `receipts` | `period` (ledger day), `client` (`''` for the whole gateway), `created_ts`, `doc` (the signed JSON); primary key (period, client). Written once per day and client by `receipts.seal_due` ([§7](#verified-savings-receipts-receiptspy)) | Savings receipts | | `spend` | `id`, `ts`, `day`, `kind`, `ref`, `usd`, `realtime_usd`, `in_tokens`, `out_tokens`, `detail` | Money spent outside requests, e.g. backfill labelling. Scout's rows (`scout_run`: one per run, `{"judge", "llm_usd", "status"}`; `tavily`: one per search, `{"judge", "credits", "depth", "http", "counted"}`) keep `usd` 0, because their model calls are already in `requests`; indexed on `(kind, ts)` | | `backfill_jobs` | `job`, `ts`, `backend`, `teacher`, `license`, `prompts`, `unique_prompts`, `outputs_discarded`, `source_models` (JSON), `status`, `usd`, `realtime_usd` | Clean-room backfill | | `tombstones` | `key`, `erasure_id`, `ts`, `suppressed` | Example hashes an erasure removed, never stored again; `suppressed` counts the refusals (written by proves their absence) | | `erasures` | `id` (never reused), `ts`, `day`, `key`, `tainted` (JSON version ids that may hold the data), `status` (`pending` while there are some, else `complete`), `replacement`, `certificate` (JSON), and's `actor`, `reason` (redacted), `scope` (JSON: end users as HMACs, `contains` as its SHA-256), `purged`, `kept`; indexed by `status` | Erasure requests; `key`, `replacement` and `certificate` are for provable erasure | `version_leaves` and `receipts` are **append-only**: triggers refuse UPDATE, DELETE and an INSERT that would replace a row, on every connection, because the signed roots and receipt chains are built from those rows. Their key columns are NOT NULL. New indexes on `requests`: (`student_version`, `audited`, `id`) for a version's audits, `shadow_version` for the shadow's rows and `day` for per-day statistics; before them, each gate tick scanned the whole table. `versions_state` serves the routing snapshot's rebuild. ### `admin_actions`, `runtime_flags`, `task_overrides`: manual changes `admin_actions`: `id`, `ts`, `actor` (`api:` or `cli:`, never a key), `action` (`rollback`, `retire`, `promote`, `set_flag`, `pin_task`, `unpin_task`, `tick`), `target` (`v7`, a switch, a task, `gate`), `reason` (redacted), `before` and `after` (JSON: the target's state), `request_id` (the admin call's; NULL from the CLI). `runtime_flags`: `name` (primary key: `student_disabled`, `releases_frozen`, `training_paused`, `audit_rate`), `value`, `ts`, `actor`; a missing row is the default. `task_overrides`: `task` (primary key), `pinned`, `ts`, `actor`, `reason`. Written only through `ratchet/admin.py` ([§18](#18-admin-api-and-runtime-switches)); new tables, so an old file gains them on open. ### `failures`: what failed, and what it spent `id`, `ts`, `day`, `request_id`, `client`, `task`, `stage`, `error`, `status`, `latency_ms`, `cost_usd` ([§12](#12-observability)). `stage` is one of `ledger.FAILURE_STAGES`; `error` is the exception's class, status and code, never a message; `cost_usd` is what a request that left no `requests` row spent (the student's samples before a teacher failure), 0 for a failure around an answered request, whose cost is in its own row. `Ledger.totals()` adds it as `failed_cost_usd`. Indexes `failures_request_id`, `failures_ts` and `failures_client_ts`. Written by `Ledger.log_failure` only. ### Data governance tables New tables, so older files gain them on open. Hashes and key names only. | Table | Columns | Meaning | |---|---|---| | `example_clients` | `hash` (`examples.hash`), `client` (API key name); primary key (hash, client) | Other clients whose requests matched a stored example; `examples.client` holds the first and is never listed here (`Ledger.example_clients(hash)` returns all of them) | | `no_train` | `end_user_hash` (primary key), `ts`, `client` (the key name the hash is scoped to; NULL for keyless requests; `*` for an admin's registration for every client), `admin` (the admin's name when an admin registered it) | End users whose requests are never stored (`POST /v1/ratchet/no-train`). A row an admin made can only be removed by an admin | | `erasure_items` | `erasure_id`, `hash`, `task`, `split`; primary key (erasure, hash) | The examples each erasure purged: hashes, never text | | `daily_rollup` | `day`, `task`, `client`, `route`, `store_opt_out` and the sums of the request rows folded by retention: `requests`, `student_served`, `teacher_served`, `escalated`, `cost`, `baseline`, `teacher_cost`, `student_cost`, `split_unknown`, `split_unknown_cost`, `sampled`, `avoided`, the student tokens, `served_with_tokens`, `audited`, `audit_agree_sum`, `audit_n`, `wall_ms_sum`, `latency_n`, `stored_sum`, `stored_n`, `answered` and `answered_escalated` (requests the student judged, and of those the ones it escalated: success metric 5), `folded_ts`; indexed by `day` | Request rows older than `RATCHET_REQUEST_RETENTION_DAYS`. A fold may leave several rows per key; readers sum them. No end user is kept | Indexes for retention and erasure: `requests(ts)`, `examples(last_seen_ts)`, `examples(client)`, `examples(end_user)` and `example_clients(client)`. The `feedback` table holds one verdict per request: `request_id` (unique), `client` (the key name), `verdict` (`correct` or `incorrect`), `corrected_present`, the request's `task`, `version`, `served_by` and `route` (copied, so the statistics outlive the request row), `stored` and `example_hash` (the correction stored for it). Ids, names and flags only, never text. Indexes: `requests(request_id)`, `requests(end_user_secondary)`, `feedback(version)`. The `meta` table also holds `human_alert:v` once a version's `human_alert` event was written. The `meta` table also holds `day_origin` once request retention first folds rows: when day 1 started, so day numbers do not shift when the oldest rows go. The `meta` table also holds `end_user_key` (the generated HMAC key, hex, when `RATCHET_END_USER_KEY` is not set) and `end_user_key_id` (an HMAC of a fixed label under the key last used, to notice a change). ### `events`: release log `id`, `ts`, `day`, `kind`, `detail`. The main kinds written by the code are `trained`, `skip`, `rejected`, `shadow`, `canary`, `live`, `live_retired`, `retired`, `rollback`, `inconclusive`, `alert`, `drift`, `teacher_drift`, `teacher_model_changed`, `teacher_watch`, `teacher_ack`, `warning`, `student_error`, `audit_error`, `shadow_error`, `skip_example`, `circuit`, `adapter_reloaded`, `adapter_reload_failed`, `gateway_error`, `task_registered`, `task_changed`, `task_deleted`, `task_skipped`, `task_disabled`, `schema_repair_error`, `json_schema_fallback`, `auto_task_cap`, `auto_task_skipped`, `rescored`, `rescore_failed`, `endpoint`, `integrity`, `sync_failed`, `no_train`, `no_train_unmatched`, `end_user_key`, `erase`, `purge`, `purge_failed`, `human_alert`, `import`, `receipt_skipped`, `sidecar_inline`, `sidecar_dropped`, `budget`, `admin` (every manual change, [§18](#18-admin-api-and-runtime-switches)), `conflict`, `fault` (live gateway only), `warmup` (mock warm-up only), `erasure_honoured`, `judge`, `search_error`, `tavily_exhausted`, `unattributed`, `demo_reset` (mock-mode gateway only) and `liverun_start`, `liverun_round`, `liverun_stop` (the live-run harness). Their meanings are in [RUNBOOK.md](/reference/runbook/#event-kinds). > Note on concurrency: every public `Ledger` method that writes, or that the request path and the release gate read, holds one re-entrant lock (`_locked`), because Scout's triage and reader agents call `Ratchet.handle` from several threads at once and all share one SQLite connection (`check_same_thread=False`). Stats and report queries (`_reads`) use a second, read-only connection with its own lock on a WAL file ([§11](#11-hot-path-and-performance)). This makes one gateway process safe; it does not make several processes safe, so run one uvicorn worker. > Note on `day`: the simulator and the mock-mode gateway endpoint set `Ledger.day` themselves. In live mode the ledger is opened with `auto_day=True`: day 1 starts at the first logged request and a new day begins every 24 hours. Existing database files are migrated on open (`Ledger._migrate()` adds columns introduced after v0.1). ## 5. Version state machine Thresholds are the `ReleasePolicy` defaults in `release.py`: `min_precision=0.95`, `min_coverage=0.25`, `tolerance=0.02`, `min_shadow=100`, `min_audits=20`, `live_floor=0.92`. Canary steps are `CANARY_STEPS = [10, 50]`, then 100. The canary transitions below are the default (Wilson) rule; with `RATCHET_SEQUENTIAL=1` the anytime-valid rule decides them instead, as the notes in parentheses say. ```mermaid stateDiagram-v2 [*] --> candidate: run_night trains candidate --> rejected: golden precision < 0.95
or coverage < 0.25
or regresses vs live (>0.02) candidate --> canary: admit, no live version (cold start)
traffic 10% candidate --> shadow: admit, live version exists shadow --> rejected: ≥100 would-serve answers and
precision < 0.95 or coverage < live − 0.02 shadow --> canary: ≥100 would-serve answers,
precision ≥ 0.95, coverage ≥ live − 0.02
traffic 10% canary --> canary: ≥20 audits this stage,
audited precision ≥ 0.92 → 10% → 50% canary --> live: at 50%, ≥20 audits, precision ≥ 0.92
(old live → retired) canary --> rolled_back: ≥20 audits and Wilson upper bound < 0.92
(sequential: e-value over its life ≥ 20, any count) shadow --> retired: newer candidate admitted canary --> retired: newer candidate admitted
(sequential: 400 audits in a stage, no decision: inconclusive) live --> retired: a canary is promoted ``` ### Golden gate: `ReleaseGate.admit(vid)` Using the candidate's `eval` (calibrated on half of the golden set, measured on the other half): 1. Reject if `precision < 0.95`. 2. Reject if `coverage < 0.25`. 3. With `min_certified_precision` > 0 (`RATCHET_MIN_CERTIFIED_PRECISION`, opt-in), reject if the certified lower bound, recomputed from the certificate's counts, is below it, or if the eval has no certificate ([§6](#certified-precision-ratchetcertificatepy)). 4. If a live version exists, check for a regression against it. The live numbers were re-measured on the same golden set the same night ([§3](#3-nightly-loop-night-school)). - Default: reject if `precision < live.precision − 0.02` **or** `coverage < live.coverage − 0.02`. - With `paired_gate=1` and the candidate's `paired` record against this live version: reject only if more than `tolerance` of the rows are lost from coverage (`(b − c)/n`) with McNemar `p_cov < regression_alpha`, if more than `tolerance` of the rows are newly answered wrong (`(neg_flips − pos_flips)/n`) with `p_err < regression_alpha`, or if `max_nfr` < 1 and the negative flip rate's 95% Wilson lower bound is above it. Without such a record, the default rule. 5. With `min_judge_pass` above 0 (`RATCHET_MIN_JUDGE_PASS`, [free-text tasks](#free-text-tasks)), reject if the eval's `judge_pass_rate` is below it, or if no judge graded a free-text task the candidate would answer (`ReleaseGate.judge_problem`). 6. Per task (`ReleaseGate.task_failures`), for each per-task record with at least `min_task_test_n` (10) held-out rows that the candidate would answer: the task fails when its held-out precision (over the answers it would serve) is below its floor (`policy.min_precision`, else `min_precision`), or below the live version's precision on that task less `tolerance` (only where live answers the task under the same spec with as many rows): the global gate's point-estimate rule on the task's own rows. A task that answers none of its held-out rows fails too, since it has no precision to show. With the judge gate on, a free-text task with as many held-out rows fails when its own `judge_pass_rate` is below `min_judge_pass` or it was not judged (`ReleaseGate.judge_failures`). With `task_mode=reject` a failing task rejects the candidate. With `disable` (the default) it is disabled on the candidate: `Ledger.merge_task_eval` sets its threshold to 1.01 and `disabled` to the reason, and a `task_disabled` event is written. Either way, a candidate with per-task records is rejected when the share of its held-out rows it would still answer, each task at its own threshold and the failing ones off, is below `min_coverage`: a version that serves (almost) nothing would wait in shadow or canary forever, and no night trains meanwhile. With per-task calibration off only the critical tasks have records, and the eval's `other_tasks` (the other tasks' held-out rows and how many the global threshold serves) counts in that share too. A task with critical classes (`ReleaseGate.critical_failure`) is checked first, whatever its row count: each class needs at least `min_task_test_n` held-out rows whose teacher reference is in it and a miss rate whose 95% Wilson upper bound is at most its `max_miss_rate`, else the task fails the same way (a record that did not measure the current classes fails too). 7. Otherwise every version in `shadow` or `canary` is set to `retired` (event `retired`). Then: - no live version → `canary` at 10% (event `canary`, "first student"); - live version exists → `shadow` (event `shadow`; with the paired gate it appends `serves +c−b of n vs live vL (McNemar p=…); nf negative flips, pf positive flips`). Both events append `; certified >= 95% (lcb 96.4%, delta 0.05)` when the candidate's certificate holds (recomputed from its counts). In practice step 7's retire branch rarely runs, because `run_night` skips training while a shadow or canary exists. With the defaults (no certified floor, no paired gate) this is v0.4's gate exactly. The whole of `admit` is one transaction (`Ledger.transaction`), and it acts only on a `candidate`: one an admin moved while it was being scored is left as it is, with a `conflict` event. While an operator holds releases (`releases_frozen`, or a runtime audit rate of 0), a cold-start candidate goes to `shadow` against the teacher instead of canary. **Every move compares and sets**. The gate and the admin API move versions only with `Ledger.transition(vid, from_state, from_pct, to_state, to_pct, event)`, an `UPDATE ... WHERE id=? AND state=? AND traffic_pct=?` inside `BEGIN IMMEDIATE` that writes its event only when the row changed, and `Ledger.promote_live`, which makes a canary live and retires the live version the caller saw, both compared, in one transaction. A move into shadow or canary is refused while another version is in either. So the gateway's tick and the cron tick (two processes on one file) make each decision once, and at most one version is live and one in flight. `set_version` is for tests and migrations only. Before step 1 the teacher watch may write an advisory `warning` (`teacher_watch.ceiling_warning`): with at least 30 checks in the current epoch and `min_precision` above the agreement ceiling (the square root of the upper 95% Wilson bound of the teacher's self-agreement), no student can reach the floor at full coverage against this teacher. Per request a student agrees with a fresh teacher label at most max_y p_y ≤ sqrt(sum_y p_y²) of the time, and over requests E[max p] ≤ sqrt(E[sum p²]) by Jensen. It changes no decision. ### Ticks: `ReleaseGate.tick()` Run from the request path (at most once per `RATCHET_TICK_INTERVAL_S`: 5 s in live mode, every request in mock mode; [§11](#11-hot-path-and-performance)) and by `python -m ratchet.live tick`. It is six steps in a fixed order, each in its own method so each later feature edits only its own (`_tick_drift`, `_tick_live`, `_tick_tasks`, `_tick_shadow`, `_tick_canary`, `_tick_watchers`). The live version is read once, before the first. The teacher watch runs last. 0. **Drift** (`_tick_drift` → `check_drift`): one primary-key range query, `Ledger.confidence_stream(after_id)`, reads the student confidences logged since the last tick, and the drift monitor feeds each to its (version, task) stream ([§6](#drift-monitor-ratchetdriftpy)). A stream whose test martingale reaches `drift_alarm` (100) writes one `drift` event, once across processes and restarts (`Ledger.event_keyed`, keyed in `meta`), and its version's certificate reads `suspended`. With `drift_audit_boost` above 1, every serving version (canary or live) with an alarm is put in `audit_boost` and every other one taken out, so a newer version going live clears it. It decides no release. The gateway also runs it once at startup, so the replay of the ledger happens before the first request, and Night School runs it before deciding on an early night. 1. **Live watch:** take the audits of the live version since it became live (`audit_counts(id, stage_since)`: `s` agreements of `n`). By default (`alert_eprocess=1`), if `n > 0` and the e-value against "precision ≥ `live_floor`" reaches 1/`alpha_alert` (100), write one `alert` event per version, with the e-value and the 95% confidence sequence ([§6](#anytime-valid-evidence-statslog_e_bernoulli-statsconfidence_sequence)). With `alert_eprocess=0`: at least `min_audits` and the Wilson upper bound below `live_floor`. No automatic action is taken on the live version. 2. **Per-task watch**: for the live version and the canary, the same rule per task on the audits of the current stage (`Ledger.task_live_quality`, run only when a new audit arrived since the last check, `Ledger.last_audit_id`), one `alert` per (version, task) (`v3 task page-read: ...`). A version whose audits in the stage are all one task's is left to step 1 and the canary rollback, which say the same. Once-only is read from the events in the transaction that writes the alert (`Ledger.event_once`, `BEGIN IMMEDIATE`), so neither a new gate, as each `python -m ratchet.live tick` builds, nor the gateway's tick racing a cron tick on the same file repeats it. Nothing is rolled back or disabled; rollback stays per version. 3. **Shadow:** `shadow_quality(id)` returns `served` (would-serve count), `precision` (mean `shadow_agree` where `shadow_serve=1`, an unarbitrated NULL counting as 0), `coverage = served / n`, `live_coverage` (share of the same requests the live student actually served, leaving out those it never answered: route `student_breaker` or `student_error`) and `unarbitrated`, over the rows with a shadow verdict (`shadow_serve` not NULL). Once `served ≥ min_shadow` (100): - precision < 0.95 → `rejected` (the reason adds "N unarbitrated counted as disagreements" when N > 0); - coverage < live_coverage − 0.02 → `rejected`; - otherwise → `canary` at 10%. 4. **Canary:** `version_live_quality(id)` counts audits of answers this version's own adapter served since `stage_since` (never the hosted fallback's, [§16](#16-fallback-attribution)). With `n ≥ min_audits` (20) and audited precision `p`: - Wilson upper bound of `p` below `live_floor` (0.92) → `rolled_back` (event `rollback`). Traffic returns to the live version, or to the teacher on cold start. - else if `p < 0.92` → hold (no change; keep auditing). - else if releases are frozen (`releases_frozen`) → hold (a rollback above still runs). - else if traffic is 10% → 50% (event `canary`); the stage counter resets because the move (`Ledger.transition`) updates `stage_since`. - else → old live set to `retired`, canary set to `live` at 100% (event `live`). With `sequential=1` (`_sequential_canary`), in order; no audit means no decision: - **Rollback:** the e-value against "precision ≥ floor" over every audit the canary has had (`audit_counts(id)`, never reset per stage) reaches 1/`alpha_rollback` (20) → `rolled_back` (event `rollback` with the counts, the e-value and the confidence sequence). No minimum count: 0 of 2 agreeing is enough (e-value 29.8). - **Promotion,** on this stage's audits once there are `min_audits`: the point estimate ≥ floor, or with `promote_evidence=1` the e-value against "precision ≤ floor" over this stage reaching 1/`alpha_promote` (56 agreeing audits at a 0.92 floor). Then advance or promote as above; the event appends the evidence. - **Futility:** `max_stage_audits` (400) audits in this stage and no decision → `retired`, traffic back as on a rollback (event `inconclusive`). It counts audits, not time: a canary without audits holds. - else hold. 5. **Teacher watch** (`_tick_watchers` → `check_teacher`): two indexed lookups when up to date, the epoch in `meta` and the `teacher_checks` rows added since the last tick (a primary-key range), added to running counts by ledger day. When the freeze rule's e-value (§6, [Teacher watch](#teacher-watch-ratchetteacher_watchpy)) reaches 1/`alpha_teacher` (100), `Ledger.teacher_freeze` sets `teacher_frozen` and writes one `teacher_drift` event, in one transaction that re-checks the epoch, so the gateway and the nightly job write it once. A new epoch (an ack, a model change) resets the counts. It decides no release; Night School reads the freeze. ### Shadow reference and arbitration For each request while a shadow exists, the shadow candidate is sampled and scored like a real student. `shadow_serve = confidence ≥ shadow threshold`. The reference for agreement is chosen in this order: 1. The teacher's answer, if the teacher was called for this request (escalation or audit). An audit answer that breaks the [storage rules](#storage-rules) (cut off, empty, or invalid for the task) is not an audit, so it is no reference: the next steps apply as if no audit had run. 2. If the shadow would serve and disagrees with the served answer, the teacher is called to **arbitrate** (stored as training data) and its answer is the reference. If that call fails, or is skipped because the audit already failed in this request, there is no reference: `shadow_agree` stays NULL and the gate counts it as a disagreement. An escalation answer (step 1) or an arbitration answer (step 2) that breaks the storage rules is no reference either: `shadow_agree` stays NULL. 3. Otherwise the served answer (the live student's). So disagreements are always settled by the teacher (or counted against the shadow when it cannot be reached), while agreements with the live student are accepted as-is. ## 6. Confidence and calibration ### Score (`confidence.score`) Given `n` samples and a `Task`: 1. Each sample's key is computed once by `Task.key()`, after `tasks.clean_answer` ([Storage rules](#storage-rules)): each field of the task's spec normalised by its comparator (`compare.FieldSpec`), e.g. `ticket-triage` → `category` and `priority` as labels over their vocabularies. A task without fields keeps the whole answer: its canonical JSON when it is a JSON object or array, and its text with reasoning removed, lower-cased and whitespace collapsed. An answer that does not parse, lacks a required field, has a value that does not normalise (a label outside its vocabulary), is empty, or did not end by itself (`finish_reason` other than `stop` or `tool_calls`: `length`, `content_filter`, vLLM's `abort`) is invalid and gets no vote. 2. If every sample is invalid, confidence is 0 and the answer is never served (`Scored.valid` is false), even with a threshold of 0. 3. Votes are pairwise: each valid sample's votes are the valid samples whose key it matches (`Task.match`), itself included; comparators are symmetric, so each pair is compared once (at most n(n+1)/2 comparisons, no I/O). The majority is the first sample with the most votes, and `agreement = its votes / n`. 4. The best sample is, among the samples the majority matches that have as many votes, the one with the highest mean token probability, `p = exp(mean(token logprobs))` (the first on ties). 5. `confidence = 0.4 · p + 0.6 · agreement`, or `agreement` alone if no logprobs were returned. For comparators that are transitive (equality on normalised values, as in every built-in task) the matching samples are one group of equal answers, so steps 3 and 4 give exactly the old count of equal canonical answers, with `Counter.most_common`'s tie-break. A tolerance makes agreement non-transitive: for 0, 0.6 and 1.2 under `abs_tol=1`, 0.6 matches all three (agreement 1.0) while the ends do not match each other, and step 4 serves 0.6, the only sample with three votes. The served sample always has the top vote count itself, so `agreement` is its own; with four or more samples it need not match every member (0, 0.6, 1.2 and 1.8: 0.6 and 1.2 both have three votes, and 1.2 does not match 0). A sample whose key does not even match itself gets no vote, and when none has a vote the score is 0. `confidence.majority` picks a request's `n` choices the same way: the best sample, then the samples that match it. ### Threshold calibration (`calibrate_threshold`, `cascade_eval`) `cascade_eval(sample, golden_rows, samples=3, target_precision=0.97, floor=0.5, calibration='point', cert_target=0.95, cert_delta=0.05)`: 1. Score each golden row with the candidate, sampled as many times as its task is in production (`Task.sample_count(samples)`). Even-indexed rows go to a calibration set, odd-indexed rows to a test set; each point is `(confidence, agrees_with_teacher)`. 2. `calibration='point'` (the default): `calibrate_threshold(calib, 0.97)` sorts points by confidence (descending) and walks down **in groups of equal confidence** (ties included before checking). It returns the lowest confidence at which cumulative precision is still ≥ 0.97, or `inf` if none qualifies. `calibration='ltt'` (`RATCHET_CALIBRATION=ltt`): `stats.ltt_fixed_sequence(calib, CERT_GRID, cert_target, cert_delta)`, below. 3. `threshold = min(1.01, max(0.5, calibrated))`. The value 1.01 means "serve nothing". It is stored exactly (before v0.5 it was rounded to 4 decimals, which could round up past the confidence it was calibrated at), so the router serves at the threshold the eval measured. 4. On the **test** half: `coverage = share with confidence ≥ threshold`, and `precision = agreement rate among those`; then the certificate, the frontier, the per-task bounds and the test vector, below. A row whose reference (the stored teacher answer) is not valid under its task's current spec is skipped before the student is sampled; the halves alternate over the rows that are scored. For a task with `escalate_when` rules, a point whose answer a rule sends to the teacher is scored and counted in `test_n` and coverage's denominator, but left out of steps 2 and 4: production never serves it, so it neither calibrates the threshold nor counts as served. A task without rules is scored exactly as above. Returned keys: `golden_n`, `golden_agreement` (majority-answer agreement over all rows), `threshold`, `coverage`, `precision`, `test_n`, `refused_n`, `golden_invalid_refs` (rows skipped for an invalid reference), `certificate`, `frontier`, `per_task`, `test_vector`, `tasks` (below), plus `judge_pass_rate` from `LiveEvaluator` when tasks without fields exist, and `paired` added by `run_night` for a candidate trained while a version is live. ### Certified precision (`ratchet/certificate.py`) The threshold comes from the calibration half only, so the test half's counts at it are one exact binomial experiment. With `s` agreeing answers among `m` served test rows: ``` lcb = cp_lower(s, m, delta) # the p with P(X >= s) = delta, X ~ Binomial(m, p); bisection, rounded down certified = lcb >= target # RATCHET_CERT_TARGET (0.95), RATCHET_CERT_DELTA (0.05) frontier = for lam in CERT_GRID (0.95 .. 0.50): the same counts at lam, lcb at delta / 10 # Bonferroni: all hold at once per_task = the counts per task at the threshold, each at delta # diagnostic only test_vector = 'S' served and right, 'W' served and wrong, 'E' escalated, one per test row ``` - With probability at least 1 − delta, the precision of what the version serves at its threshold is at least `lcb`, assuming the golden rows are exchangeable with traffic (they over-represent hard requests), with precision meaning agreement with the teacher. 59 served rows with no error is the smallest 95% certificate (`python -m ratchet.stats plan`). - **Learn-then-Test** (`stats.ltt_fixed_sequence`): from 0.95 down the fixed grid, the exact p-value `precision_pvalue(k, n, target)` of the calibration answers at or above each threshold; keep going while it is ≤ delta, stop at the first that is not, serve at the lowest kept (1.01 when the first fails). Fixed-sequence testing needs no correction. The grid is fixed in advance: one built from the observed confidences would start with a one-point group and certify nothing. Conformal risk control does not apply, because selective precision is not monotone in the threshold. - **Paired comparison** (`certificate.paired`, `run_night`'s `paired_evidence`): the candidate's test vector against the re-scored live version's, same rows, same order. `b`/`c` rows only live / only the candidate serves, `p_cov = mcnemar_one_sided(b, c)`; `neg_flips` rows where the candidate serves a wrong answer and live's answer was right (served or escalated), `pos_flips` the reverse, `p_err = mcnemar_one_sided(neg_flips, pos_flips)`; the negative flip rate (live served right, the candidate serves wrong) with its 95% Wilson interval. The opt-in paired gate reads it ([§5](#golden-gate-releasegateadmitvid)). - `certificate.recheck(eval)` recomputes every number from the counts (and the counts from the test vector, the per-task rows' sums and Learn-then-Test's trace); `python -m ratchet.stats check bom.json` runs it on a signed BOM. `GET /ratchet/versions/{id}/certificate` serves the signed and the latest certificate with the gate's evidence and the assumptions. Reference run (seed 7, mock-mode run): the training certificates' lower bounds go from 0.875 (46/48, v1) to 0.964 (129/130, v7); v2 (70/70), v6 and v7 are certified at 95%, though the default gate rejects v2 for regressing against v1. Returned keys: `golden_n`, `golden_agreement` (majority-answer agreement over all rows), `threshold`, `coverage`, `precision`, `precision_kind` (`proxy` when a task with a similarity field was scored, else `exact`), `test_n`, `refused_n`, `golden_invalid_refs` (rows skipped for an invalid reference), `tasks` (below), plus `judge_n` and `judge_pass_rate` when a `judge` is passed and free-text tasks have golden rows ([free-text tasks](#free-text-tasks)). ### Per-task calibration Several tasks can share one adapter (Scout's triage and reader calls next to support tickets, UC-06), and the student is rarely equally good at each. One threshold calibrated over the pooled rows then serves the hardest task below the target precision, because the easy tasks' right answers dilute its wrong ones, and holds the easy tasks back when the hard one pushes it up. So `cascade_eval(..., min_task_n=RATCHET_MIN_TASK_GOLDEN)` also writes one record per task with golden rows that is not pinned: 1. Each scored row's point also goes into its task's calibration or test list, by the same global parity as above. The global numbers are computed exactly as before, and a single-task eval gives the global numbers in its record. 2. The task's own threshold is `calibrate_threshold` on its calibration points at its `policy.target_precision` (else `target_precision`), floored at 0.5 and capped at 1.01 as above. 3. With at least `min_task_n` (20) calibration points that is its threshold (`calibrated: true`). With fewer it is the higher of its own and the global one (`calibrated: false`), so a few lucky rows cannot lower it; with none, the global one. With no test point it is 1.01: a task that has a per-task record is never served without held-out rows, measured on rows the threshold was not tuned on. A task with no golden row has no record and uses the version's global threshold. 4. `precision` and `coverage` on the task's test points at its threshold; `precision_kind`; `golden_n`, `test_n`, `agreement`, `spec_hash` (the spec it was scored under), `target_precision`; for a free-text task scored with a judge, `judge_n` and `judge_pass_rate`. The router reads the records through `calibration.threshold` ([§2](#2-request-path)), the golden gate checks them ([§5](#golden-gate-releasegateadmitvid)), `run_night` keeps disabled tasks disabled through every re-score (`calibration.keep_disabled`), and `/ratchet/stats` shows the live version's per task. A version scored before per-task calibration, or with `RATCHET_PER_TASK_CALIBRATION=0` (`cascade_eval(per_task=False)`), has no `tasks` and keeps its one threshold (with the switch off a task with critical classes still gets its record, and `other_tasks` then holds the other tasks' held-out rows and how many the global threshold serves, for the gate's coverage check). The global numbers stay at the global threshold, as the gate's global checks read them; the records say what the version serves. ### Free-text tasks A `similarity` field (`ratchet/similarity.py`) makes two free-text answers agree when the longer is at most `max_len_ratio` times the shorter and a score reaches `threshold`: `chrf` (character n-grams 1 to 6, F-beta with beta 2, the lower of its two directions), `token_f1` (SQuAD) or `rouge_l` (bit-parallel LCS over at most 512 tokens). The normalised value (`compare.SimilarityText`) keeps each metric's features once built, so each sample's are built at most once per request. Agreement under a score is a proxy, so: 1. Each per-task record says `precision_kind` (`proxy` with a similarity field, else `exact`). 2. With a `judge` (`LiveEvaluator`: `RATCHET_JUDGE_MODEL` in JSON mode; `MockFleet.judge`: the stored truth), after calibration `cascade_eval` grades the held-out rows of free-text tasks (`Task.free_text`: no fields or a similarity field) whose answer the student would serve, at the threshold coverage is measured at (the task's own with per-task records, else the global one). The candidate is the scored answer from the samples already drawn, cleaned as served; no other student call is made. It writes `judge_n` and `judge_pass_rate` (None when nothing was served) to the eval and to each free-text task's record. 3. The golden gate reads them only when `min_judge_pass` is above 0 ([golden gate](#golden-gate-releasegateadmitvid), steps 5 and 6). The judge never runs on the request path. A task with a similarity field must set `policy.samples` to 2 or more, since one sample always agrees with itself. ### Text-to-SQL tasks . A task with format `sql` has a `sql` object instead of fields (`sqlcheck.SqlSpec`), and `Task.key` hands the cleaned answer to it: 1. `clean_answer` strips reasoning and one code fence around the whole answer (served and stored without it too). `parse_query` tokenises the rest in one linear pass and keeps it only when it is one statement starting with SELECT, WITH or VALUES (a `;` may end it, comments anywhere) that does not read the clock; it also notes whether the outermost query (parenthesis depth 0) has ORDER BY, and the normalised text (words lower-cased, one space between tokens). 2. For dialect `sqlite`, `Sandbox.run` executes it on the task's fixture under the sandbox's lock and returns the column count, the row count and two sha256 digests of the rows (floats to 9 significant digits, a whole float as the integer): in order and as a sorted multiset. Outcomes that do not depend on timing are cached by statement (256 per sandbox), so samples with the same text run once. 3. The key is a `SqlKey`: the in-order digest when the outermost query has ORDER BY and there are two rows or more (`ordered: auto`), else the multiset digest, and the normalised text. Two keys match when their digests are equal, or, both empty under `empty_result: text`, when their texts are. Dialect `postgres` runs nothing and matches texts only. 4. Anything else is `compare.INVALID` with a fixed reason (`not one query`, `reads the clock`, `sql error`, `sql denied`, `sql steps exceeded`, `sql time exceeded`, `sql memory exceeded`, `sql too many rows`, `sql result too large`), so the confidence score, `_store_example`, the golden eval and `dataset.rows` treat it like any invalid answer, and `skip_example` events carry the reason only. The sandbox (one per fixture text, held through a weak map by every task that uses it): the fixture is read once per parse (at most 5 MB, inside `RATCHET_FIXTURES_DIR`, a regular file opened without blocking or following a symlink) and loaded with `executescript` under a load authorizer (no ATTACH, DETACH, `load_extension`, non-deterministic functions or PRAGMAs outside a short list), within 30 s and 64 MB, with `Connection.setlimit` capping a value at 100 KB and attached databases at 0. Then `PRAGMA query_only = ON`, a statement limit, and the query authorizer: SQLITE_SELECT, SQLITE_READ, SQLITE_RECURSIVE and SQLITE_FUNCTION for the deterministic functions in `sqlcheck.FUNCTIONS`; SQLITE_DENY for everything else. Each query gets a progress handler every 100 VM steps that stops it after `max_steps`, 1 s or 128 MB of resident-memory growth (Linux), and rows are fetched 100 at a time and hashed as they arrive, up to `max_rows` and 8 MB. No handler runs inside one step, so its work is bounded as well: LIKE and GLOB patterns are capped at 100 bytes (`SQLITE_LIMIT_LIKE_PATTERN_LENGTH`), and the built-ins SQLite runs in quadratic time (`instr`, `replace`, the two-argument `trim`, `ltrim` and `rtrim`) are replaced on the query connection by linear Python versions with the same results (`sqlcheck._GUARDED`; other type mixes go to SQLite's own function on a helper connection, bounded by `NATIVE_WORK`); `json_patch` and `unhex` are not allowed. `setlimit` needs Python 3.11, so on 3.10 a sqlite fixture is a spec error. A typical query on a small fixture takes about 0.1 ms. `Task.spec()` holds the `sql` options and the fixture's sha256 (only for sql tasks, so every other hash is unchanged), so an edited fixture sends the task to the teacher until a night re-scores the version; `taskspec.install` reads fixtures again at every start, and `PUT /ratchet/tasks/{name}` treats the same body over a changed fixture as a change. An executing sql task is `verifiable` and not `free_text` (its precision is exact, measured on the fixture); a postgres one is free text, so the judge grades it at night. ### Wilson score interval (`stats.wilson`, also `release.wilson`) For `s` successes out of `n` and `z = 1.96`: ``` p = s / n denom = 1 + z² / n centre = (p + z² / (2n)) / denom half = z · sqrt( p(1−p)/n + z²/(4n²) ) / denom bounds = [max(0, centre − half), min(1, centre + half)] ``` Rollback requires the **upper** bound to be below the floor, which means strong evidence the version is worse. With 20 audits and a true precision of 96%, the chance of a false rollback is negligible. With a raw "point estimate < 92%" rule it would have been about 19%. Example: 19/20 gives roughly [0.76, 0.99], so no rollback; 10/20 gives roughly [0.30, 0.70], so rollback. The interval's 2.5% one-sided error holds for one look at a fixed sample. The gate looks after every request: at a true precision exactly on the floor, checked after every audit, it rolls back about 22% of such versions within 400 audits (mock-mode run, `tests/test_stats.py`). The Wilson rule stays the canary's default, so the reference runs do not move; the anytime-valid rule below fixes this when switched on (`RATCHET_SEQUENTIAL=1`). ### Anytime-valid evidence (`stats.log_e_bernoulli`, `stats.confidence_sequence`) The sequential canary rule and the live alert decide from two integers per version, audits `n` and agreements `s` (`Ledger.audit_counts`), with a mixture e-value: ``` alternatives q_j, w_j = stats.alt_grid(p0, side) # 'below': half the weight on p0 - 0.01 ... p0 - 0.10, # half on 20 points spread over (0, p0) log e(s, n) = logsumexp_j( ln w_j + s·ln(q_j / p0) + (n − s)·ln((1 − q_j) / (1 − p0)) ) # 0 when n = 0 ``` - `side='below'` tests H0 "precision ≥ p0" (rollback, alert); `'above'` tests H0 "precision ≤ p0" (promotion). The e-value depends on the counts only, not their order, so anyone can recompute it. - **Validity:** under H0 each audit multiplies every component by a factor whose conditional mean is at most 1 (for `'below'`: 1 + (q − p0)(q_j/p0 − (1 − q_j)/(1 − p0)) ≤ 1 when q ≥ p0 > q_j), with no independence assumption. So the mixture is a nonnegative supermartingale starting at 1, and by Ville's inequality P(it ever reaches 1/α) ≤ α however often the gate looks. - Thresholds: rollback 1/`alpha_rollback` = 20, promotion 1/`alpha_promote` = 20, live alert 1/`alpha_alert` = 100. Reported e-values are capped at e^690, so they stay finite in JSON. - **Confidence sequence:** the p on a 0.001 grid where neither `log e(s, n, p, 'below')` nor `'above'` reaches ln(2/0.05); the true precision stays inside at every `n` at once with probability ≥ 95%. Every gate event and `ReleaseGate.evidence()` report it as `95% CS [lo, hi]`; a scan costs about 20 ms and is cached by counts. - `ReleaseGate.evidence(v)` (in `/ratchet/stats` for the canary and the live version) recomputes everything from the counts: the rules in force, the floor, `n`/`s` over the version's life and `n_stage`/`s_stage` in its current stage, `e_rollback` (life, `'below'`), `e_alert` (stage, `'below'`), `e_promote` (stage, `'above'`), their thresholds, the confidence sequence over its life and the Wilson interval of its stage. Examples at a 0.92 floor: 0/2 agreeing → 29.8 (a sequential rollback); 0/3 → 268 (a live alert); 7/17 → 2.6 × 10^5; 20/20 → 2.57 against "precision ≤ 0.92" (no promotion yet); 60/60 → 27.6. ### Drift monitor (`ratchet/drift.py`) A label-free alarm for a change in the traffic a version sees. One stream per (version, task), fed in request-id order from `Ledger.confidence_stream` (rows with a confidence and a student version; pinned and teacher-only requests log none, and shadow confidences are not logged). A version gets at most 16 task streams, opened in the order it first sees each task name; the rows of every later name share one more stream of that version, task `*` ("other tasks"), so client-chosen `X-Ratchet-Task` values cannot grow the monitor ([Threat model](/reference/threat-model/) T12): ``` q = min(10000, max(0, round(confidence · 10000))) # the stream's bins, sorted, then in a Fenwick tree p_n = max(1e-12, (#{bin < q} + θ_n · #{bin = q}) / n), θ_n ~ U(0, 1) # over the stream's n so far, this one included J_n = min(0.01, 1/n) # the switching rate w_e ← (1 − J_n) · w_e + J_n / 5, e ∈ {0.05, 0.15, 0.3, 0.6, 1.0} M_n = Σ_e w_e · e · p_n^(e − 1); ln C += ln M_n; w_e ← w_e · e · p_n^(e − 1) / M_n alarm the first n with C ≥ drift_alarm (it latches) ``` - **Null hypothesis:** the version's requests are exchangeable (their order carries no information). The confidence is a fixed function of the request and the student's own sampling, so its stream is exchangeable too, and the smoothed conformal p-values p_n are independent and uniform on [0, 1]. - **Validity:** each betting function e · p^(e−1) integrates to 1 on [0, 1] and the weights w_e depend only on the past, so E[M_n | past] = 1 and C is a test martingale (the clamp at 1e-12 only lowers decreasing bets: a supermartingale). By Ville's inequality P(C ever ≥ c) ≤ 1/c per stream, however long it runs and however often the gate looks; across K streams, ≤ K/c, which `GET /ratchet/drift` reports. - **One-sided:** every bet with e < 1 pays on small p-values, which come from low confidence; e = 1 is "no bet". Confidence that rises makes the capital fall. - **Switching rate:** J = 0.01 (Vovk et al.'s Simple Jumper) for a stream's first 100 confidences, then 1/n. A constant rate moves 0.8% of the capital per request into bets that lose while nothing changes, so a version live for a long time would start a change many orders of magnitude down; with 1/n the capital shrinks only like a power of n. J depends on n alone, so validity is unchanged. - **Determinism:** θ comes from the stream's own `random.Random(f"{seed}:{version}:{task}")`, never `Ratchet.rng`. Replaying the ledger gives the same capital to the last bit, so a restarted gateway, `python -m ratchet.live tick` and the nightly job all agree, and the `drift` event is written once (`Ledger.event_keyed`). Which task names get a stream of their own depends only on the order of the version's rows, so a replay makes the same choice. - **Late alarms:** a monitor that replays the ledger (a restart, a lowered threshold, the nightly job in its own process) can find an alarm days after it happened. Its `drift` event is dated on the alarm's own ledger day and `meta` records its request and day; the early night counts an alarm only when its request was logged after the last `trained` event. An event the ledger refuses to write is queued and written by the next check. - **Cost:** about 5 µs per confidence. A stream keeps its bins in a sorted array (2 bytes each, about 3 KB for a new stream with its generator) until it has seen 4,096 confidences, then in one Fenwick tree (about 80 KB, two walks of 14 steps per rank); both give the same counts. A version therefore holds at most 17 streams of at most about 100 KB each. An up-to-date monitor costs one primary-key range query per tick; the versions' states are read again every 1,000 rows to drop the counts and generators of versions that stopped serving, keeping canary, live and the most recently retired one. A compacted stream that gets new rows is replayed from the ledger first. `/ratchet/stats` reads counts the monitor keeps, not a row per stream. ### Teacher watch (`ratchet/teacher_watch.py`) Every reported number is agreement with the teacher, so the teacher's own stability is measured. Each stored request's first re-test in a check epoch (a duplicate the teacher labels again during an escalation or audit, `Ledger.add_example`, or a Night School probe) is compared with its stored label under the task's comparators (`Task.key`, `Task.match`) and recorded in `teacher_checks`; a disagreement marks the label `contested`. Only comparable labels count: the same configured teacher, at temperature 0 without a reasoning effort (`router.comparable_label`), no redaction placeholders on either side, a registered task that names fields and is not pinned, both labels valid under its spec, and a stored label that is neither contested nor chosen by an adjudication. - **Freeze rule.** H0: a fresh label of any request disagrees with an independent earlier label of it with probability at most `teacher_noise_max` (ε, 5%). One check per request per epoch makes every check two independent draws, so given the earlier checks (of other requests) each disagrees with probability at most ε, and `stats.e_bernoulli(D, N, ε, 'above')` is a test supermartingale. Comparing one stored label with several re-tests would break this: a minority stored label makes every re-test disagree. The rule uses a mixture over the epoch's ledger days of that e-process started at each day's first check, with weight 1/(j(j+1)) for the j-th day (they sum to 1; a day not started counts 1), so a teacher that changes after weeks of agreeing is caught without first winning back what those weeks cost a single e-process. The start times are fixed calendar days, so the mixture is still a supermartingale starting at 1, and Ville's inequality bounds a false freeze by `alpha_teacher` per epoch, however often the gate looks. `evidence()` recomputes it from (day, checks, disagreements) triples; `first_alarm()` replays the rule for tests and test runs. - **Model ids.** `Ratchet._ask_teacher` compares each primary-teacher response's `model` with the last one (in memory, loaded from `meta`); a change reaches the ledger (`Ledger.teacher_model_seen`), which starts a new epoch with a `teacher_model_changed` event once per id new to the epoch. The fallback teacher's ids are not watched. - **Ceiling.** `ceiling(A) = sqrt(A)`; the golden gate's warning uses the self-agreement's upper Wilson bound (§5). - **Cost.** One indexed lookup per stored duplicate on the request path (the duplicate branch of `add_example`, in the same transaction); two indexed lookups per quiet tick; at most `RATCHET_TEACHER_PROBES` + 2 × `RATCHET_ADJUDICATE_MAX` teacher calls per night. The mock teacher at noise 0 draws nothing, so a seeded mock-mode run's numbers do not move. ## 7. Cost accounting Ratchet reports cost two ways. The per-token list prices were checked on 29 Oct 2026 (the day-1 check passed); the GPU-hour rate is still unverified, and the report says so until `RATCHET_PRICES_VERIFIED=1`. ### Per request (`router.py`) `Price(input, output)` is in USD per 1M tokens (`config.DEFAULT_PRICES`). Each call's price goes into `cost_usd` and into one of its two parts: - **Student** (`student_cost_usd`): `price("student").cost(in_tokens of the first sample, sum of out_tokens over samples)`, for the live samples and the shadow's. The prompt is counted once and every sample's output is counted. `student` is a per-token **proxy** at the hosted Nano price; the student's tokens are kept too (`student_in_tokens`, `student_out_tokens`). - **Teacher** (`teacher_cost_usd`): escalation, audit, arbitration and pinned calls, `price(teacher_model).cost(in, out)`, with `out` summed over the call's choices when the client asked for `n` > 1. The `out` count includes reasoning tokens, as Token Factory reports them in usage; the mock teacher adds `teacher_reasoning_tokens`. An answer from the fallback teacher is priced at `price(fallback model)`. - **Shadow:** the candidate's sampling cost, plus any arbitration, is added. Shadowing is not free. - **Baseline:** if the teacher was called, the baseline is that call's cost (for the fallback teacher's answer, the teacher's price for the same tokens; for an audit of an `n`-choice student answer, `n` times the audit's output). Otherwise (the student served, degraded answers included) it is estimated as `teacher price × (student in_tokens, student out_tokens + teacher_reasoning_tokens)` (default 300 reasoning tokens), summed over the served choices. `cost_usd` is summed call by call exactly as before v0.5, so it equals `student_cost_usd + teacher_cost_usd` up to float rounding and no stored number moved. A failed call adds nothing. The response's `usage` is not this bill: it counts only the served answer, while `cost_usd` prices every call in `ratchet.usage_breakdown`. `stats_by_day()` derives `cost_per_1k`, `baseline_per_1k` and `savings_pct = 1 − cost/baseline`; `stats_by_task()` gives the same per task over all days; `totals()` gives `saved_usd`. These are all the **per-token view** (`/ratchet/stats` names it in `totals.cost_model`). Pinned requests have cost equal to baseline, so they count fully in both and save nothing (unless the fallback teacher answered one at a lower price). ### Compute time (`compute_intervals`) A self-hosted endpoint bills GPU-hours while it runs, so the per-token view flatters it whenever it sits idle. The ledger records when GPUs ran ([§4](#compute_intervals-when-gpus-ran)): the student endpoint (`python -m ratchet.cost endpoint up|down`, `POST /ratchet/cost/endpoint`, `deploy/student_endpoint.sh`), Night School's training job (`run_night(train_gpu=...)` around `trainer.train()` in live mode: wall-clock time, closed also when training raises, an upper bound on the job's billed time), and in test runs one endpoint interval per day of `RATCHET_MOCK_ENDPOINT_HOURS` and one training job per trained version (600 s + 0.05 s per training example from 22:00 of the mock day; mock-mode run assumptions). Intervals of one kind are merged into disjoint segments (`cost.segments`); where records overlap the segment is billed at the highest rate among them, so nothing is counted twice. An open interval runs to now. ### The report (`cost.py`) `cost_report(ledger, settings, since, until, now, days)` covers ledger days `since` to `until` (or the last `days`): requests by their `day`, intervals clipped to the window's days on the ledger's clock. With `B` the all-teacher baseline, `T` the teacher spend (rows from before v0.5 count their whole `cost_usd` here), `E` and `J` the endpoint's and training jobs' GPU cost (hours × per-GPU price × GPUs, each interval at its recorded rate), `H` the student's per-token spend in buckets with no recorded endpoint uptime, `S` the student-served requests and `R` the endpoint's USD per hour as recorded (`E` / endpoint hours, `endpoint_usd_per_hour`; `RATCHET_STUDENT_GPU_USD_PER_HOUR × RATCHET_STUDENT_GPUS` while no uptime is recorded). A bucket (`idle_granularity`) is an hour on a live ledger, where requests and intervals share the wall clock, and a whole ledger day on the synthetic clock: ``` per-token view: total = Σ cost_usd savings = B − total GPU-hour view: total = T + E + J + H savings = B − total (negative below break-even; never clamped) s̄ = Σ over student-served requests (baseline − teacher spend) / S avoided teacher spend per student answer break-even = R / s̄ student answers per endpoint hour (null when s̄ ≤ 0; training not included) h̄ = fallback price × (Σ student tokens) / S the hosted fallback's bill per student answer (rows with tokens) crossover = R / h̄ above it, the endpoint is cheaper per answer (equal coverage assumed) observed = student answers in buckets with uptime / endpoint hours utilization = observed / break-even idle hours = endpoint hours in buckets when no request reached the student ``` The recommendation (`_recommend`): `self_host` when observed ≥ break-even and ≥ the crossover; otherwise `scale_to_zero` when more than half the endpoint hours were idle; `insufficient_data` with no recorded uptime or no student answer yet; `hosted_fallback` when observed is below the crossover; else `scale_to_zero` (below break-even with a hosted fallback that costs more, so the endpoint has to run fewer hours). The `explanation` gives the numbers and says separately whether observed covers the endpoint's hours (at or above break-even) and whether the window lost money by GPU-hours: between break-even and the crossover it says self-hosting costs less than the teacher while the hosted fallback costs less per answer, and it never says "pays" when the GPU-hour savings are negative (training jobs are not in the break-even). The crossover compares price per answer only; the untuned hosted model may serve fewer requests than the tuned student, and every `hosted_fallback` explanation says so (`cost.CROSSOVER_CAVEAT`). `RATCHET_STUDENT_COST_MODEL` picks the `headline` view. On the synthetic clock idle time is counted in whole ledger days, so idle hours inside a busy mock day are not counted; on a live ledger it is counted by hour, and an hour with any uptime is credited with all its answers. `GET /ratchet/cost`, the `cost` block of `/ratchet/stats` and `python -m ratchet.cost report` serve it ([API.md](/reference/api/#cost)). Worked example with estimated prices: an L40S at $1.35 per GPU-hour on demand ([Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026) and about $0.001 of Ultra spend avoided per ticket-triage answer (the mock-mode gateway) give a break-even of about 1,350 student answers per endpoint hour, roughly 32,000 a day. A 24/7 endpoint at the live gateway's 1,500 requests a day loses money; the report says `hosted_fallback`. At about 14,000 student answers an hour (10M requests a month) the endpoint costs far less than the teacher, and the report still says `hosted_fallback`, because per answer the hosted Nano (about $0.00003 with the mock-mode gateway's token counts) is cheaper below roughly 52,000 an hour, if it serves as many requests. ### Calibrated baseline and the ledger replay (`replay.py`) The baseline of a request the student served is an estimate: it assumes the teacher would have written `RATCHET_TEACHER_REASONING_TOKENS` (300) reasoning tokens. Audits measure it. On audited answers the student served, `requests.baseline_cost_usd` is what the teacher's audit really cost and `baseline_est_usd` the formula's estimate for the same request; the ratio estimator gives ``` R = Σ w·y / Σ w·x x = baseline_est_usd, y = baseline_cost_usd, w = 1 / label_p (audited served answers) se = sqrt(s² / n_eff) / x̄ s² = Σ w (y − R x)² / Σ w · n / (n − 1) (equal weights: sqrt(Σ (y − R x)² / (n (n − 1))) / x̄) ``` with R = 1 (`insufficient`) below 30 audited answers. `GET /ratchet/replay` reports it as `calibrated_baseline`. It does not change any stored baseline or `/ratchet/stats` figure; the replay prices unaudited served answers at R × `baseline_est_usd`, and the savings receipts use the same estimator. The replay prices a version's logged traffic at threshold t and audit rate a as Σ s_i + Σ over c < t of T_i + a × Σ over c ≥ t of T_i, with s_i the logged `student_cost_usd` and T_i the request's all-teacher cost (`baseline_cost_usd` when the teacher answered the escalation or an audit, else R × `baseline_est_usd`); the baseline is Σ T_i. Precision at t is Hajek's weighted mean of `student_agree` with weights 1 / `label_p`, null without positivity (a served row with `label_p` 0); its interval is approximate (Wilson at Kish's n_eff). Coverage is exact. A request that escalated in fact is priced at the version's mean logged audit rate when it would be served. One version at a time: confidence scales differ between versions. Like every figure here it is the per-token view. ### Verified savings receipts (`receipts.py`) What the fee is computed from. One receipt per ledger day for the gateway and one per API client, sealed once the day is over (`closed_through`: the day before the current one on a mock ledger, any day that ended 5 minutes ago on a live one) and never changed. ``` leaves = canonical JSON of each request's LEAF_KEYS (metadata, whole nano-USD), in id order; RFC 9162 tree over them per task R = Σ w·y / Σ w·x over its audited served answers (x = baseline_est, y = the audit's teacher bill, w = 1/label_p) point = R · Σ x over its N served answers; V = N² (1 − n/N) s_r² / n_eff; needs n ≥ RATCHET_RECEIPT_MIN_AUDITS billable = teacher-served requests + served answers of calibrated tasks (stand-ins, routes degraded and fallback_teacher, never) stand-ins = Σ min(0, baseline − cost) over the stand-ins: their losses count, a gain never does billable_lo = (billable baseline − billable cost + stand-ins) − ⌈1.645 √Σ V⌉ − the day's spend outside requests fee = ⌊RATCHET_TAKE_RATE × max(0, billable_lo)⌋ (0, basis "formula", when no task is calibrated) ``` The manifest (`ratchet.receipt/v1`) carries the totals, the calibration, the savings and the bill computed from the leaves alone (so `python -m ratchet.receipt verify --rows` recomputes them), the gateway-wide receipt's GPU-hours view of the day (informational, [§7](#the-report-costpy)), the tree's root and size, and `prev_sha256`, the previous receipt of the same client, so each client's receipts form a chain. `bom.document` signs it with the BOM key. Sealing runs after each mock night (`Runtime.simulate_day`), on `python -m ratchet.live seal`, and lazily on any receipt endpoint; it reads the ledger (a day in chunks, `Ledger.period_rows`) and writes receipts only, with no model call or random draw, so no other number moves. The proof endpoint rebuilds the day's tree from the ledger and serves a request's audit path only when the tree rebuilds the signed root. Request retention (`fold_requests`) deletes request rows, so every retention runner seals the days that have ended first (`live.seal_receipts`, `Runtime._purge`), a day folded before it was sealed is never sealed (`receipt_skipped`), and a sealed day's leaves and proofs answer 410 once its rows are folded; the receipt itself still verifies. A completion returns the row as `ratchet.row_id` and `X-Ratchet-Request-Id`; `ratchet.request_id` is the random id feedback takes. ### Measuring a live run (`liverun.py`) The audit-calibrated R above uses the check role at temperature 0. The live-run harness measures the baseline for the serving role instead, outside the gateway: it sends rounds of synthetic workload traffic through `/v1/chat/completions` under its own key, runs `python -m ratchet.live night` and `tick` in process between rounds, and sends a Bernoulli sample of the same requests straight to the teacher with the serving teacher's parameters (never stored). Over the student-served pairs (x the recorded `all_teacher_cost_usd`, y the control call's cost) the ratio estimator gives R_s, and the measured all-teacher baseline of a window is Σx over teacher-served rows + R_s × Σx over student-served rows. A round is an id range of `requests` filtered by the key's name (`Ledger.client_window`), because a live ledger's day is wall-clock time, not a round. Accuracy is scored against the workload's own labels with the task's spec, apart from agreement with the teacher and the ledger's audited precision. Everything lands in one `results.json` with `live.*` claim ids, which `python -m ratchet.liverun verify` recomputes from the per-request records; nothing in the request path changes. ### Success metrics (`kpis.py`) The success metrics come from the same ledger the cost report reads, by the same code in the simulator, the mock-mode gateway and a live gateway. `kpis.report(ledger, window_days, min_audits, mode)` reads per-day sums (`Ledger.kpi_days`, folded request rows included), the window's pooled audits (`audit_totals`), failed requests' spend (`failed_cost`, from the `failures` table when the ledger has one), the median latency of student-served requests (`latency_quantile`, nearest rank) and the release events (`events_of`), then applies one pure function per metric: ``` window = the last RATCHET_KPI_WINDOW_DAYS ledger days with requests metric 1 = 1 − (Σ cost + failed cost) / Σ baseline over the window, per token metric 2 = Σ agree / Σ audits, 95% Wilson interval over the window; met on the point estimate, n ≥ min audits metric 3 = Σ student-served / Σ requests over the window metric 4 = versions that reached 100% and were later alerted on or rolled back metric 5 = longest drift episode in nights; rate(d) = escalated among judged / judged, days with ≥ 50 judged start: rate(d) ≥ min(rate of the 3 measured days before) + 0.05; end: first later day ≤ that baseline + 0.02 latency = median latency of the window's student-served requests (wall_ms when recorded, else latency_ms) < 1,000 ms ``` Each metric is `{name, value, target, op, unit, met, detail}`, `met` `null` while the data cannot say. `SM_TARGETS` holds the targets; `scripts/reqtrace.py` checks them against the product requirements table. `GET /ratchet/kpis`, the `kpis` block of `/ratchet/stats` and `python -m sim.simulate` serve it ([API.md](/reference/api/#get-ratchetkpisdaysn)). ## 8. Provenance - `GET /ratchet/bom` returns `Ledger.bill_of_materials()`: counts grouped by teacher, license, license URL and split, plus `examples_sha256` (a SHA-256 over all example hashes in sorted order). - **Per adapter**: after training and evaluation, `run_night` builds a `ratchet.bom/v2` manifest from the exact rows used: sources (teacher, license, URL, count, `distill_ok`), tasks, `dataset_sha256`, a hash of the golden rows, the evaluation, the number of redacted examples, `training.origins` (rows by where they came from), `redactor` (a SHA-256 over the redaction rules and whether redaction was on) and a set commitment per split (below). Optional v2 sections come with later features: `teacher`, `backfill_jobs`, `erasures_honored`. Stored v1 documents stay as they are and still verify. `bom.document()` adds the SHA-256 of the canonical JSON and, when `RATCHET_BOM_SIGNING_KEY` is set, an Ed25519 signature with the public key and key id. It is stored in `versions.bom`, served at `/ratchet/versions/{id}/bom`, and in live mode written to `data/boms/student-vN.json` and into the adapter's folder (`data/adapters//ratchet_bom.json`, with the `NOTICE.md` it signed), both uploaded by `RATCHET_SYNC_CMD`. `python -m ratchet.bom verify` checks it offline. The v2 manifest, abridged (every v1 field is still there): ``` schema: ratchet.bom/v2 adapter: {version, name, base_model, day} training: {examples, dataset_sha256, tasks, targets, examples_with_pii_redacted, origins, commitment: {alg: ratchet.smt/v1, root, size}} golden: {examples, sha256, held_out, human_labels, commitment: {alg: ratchet.smt/v1, root, size}} sources: [{teacher, license, license_url, distill_ok, examples, license_origin}] label_sources: [{source: human, license, examples, origins}] (only with human labels) base_model_license: {name, url} | {warning} notices: [{file: NOTICE.md, sha256, text}] redactor: {sha256, rules, redact_pii} eval: {..., certificate, frontier, per_task, test_vector, paired} teacher | backfill_jobs | erasures_honored: later features, optional ``` - **Committed training sets**. Each split's `commitment` is the root of a `merkle.KeySet` (a collapsed sparse Merkle tree) over the rows the trainer and the golden eval were handed: key = the example hash (`examples.hash`), value = `dataset.leaf_value(row)`, the SHA-256 of the canonical JSON of `{completion, teacher, license, redactions}`. `dataset.commitment(rows)` builds it before training; `Ledger.set_version_leaves` writes the sorted leaves once, right after `create_version` (append-only; `version_leaves(vid, split)` reads them back, `versions_containing(key)` lists the versions trained on an example, on the `version_leaves_key` index); `bom.manifest(commitments=)` signs root and size, and refuses a size that is not the split's row count. - `POST /ratchet/versions/{id}/proof` (admin key) resolves the example key from `{"key"}` or from the chat request itself, validated, normalised and redacted as the gateway stores it (`proof.resolve_key`), and returns a membership or non-membership proof from a cached `KeySet` (`gateway._keyset`: per runtime, the last 8 trees, each cached only when it rebuilds the root and size the BOM signs, else a 500 and one `integrity` event). `GET /ratchet/versions/{id}/heldout` (admin key) returns the golden leaves and every golden key's proof of absence from the training root. Mock mode adds `GET /ratchet/demo/sample-example`, a committed example's prompt. - `python -m ratchet.proof verify|heldout|key` checks a proof or certificate against the root the BOM signs (never the file's own claims), with the BOM's signature, from the files alone (exit 0, 1 on a failed check, 2 on bad input). Non-membership is sound even against the operator who built the tree; a sorted-list tree would not be. A proof shows the operator's signed commitment to the dataset at training time, not that the weights came from it. - **Licenses travel with the adapter**: the v2 manifest also carries `base_model_license` (`licenses.BASE_MODELS`; a `warning`, never an error, for a base model Ratchet does not know), each source's `license_origin` (`builtin` or `config`) and `notices`, the exact `NOTICE.md` text (`nightschool/notices.render`: base model license, each teacher's license and URL, teacher outputs only, the Ratchet version) with its SHA-256. The training job writes a `NOTICE.md` next to the weights (`train.write_metadata`); live Night School overwrites it with the BOM's text and syncs; `python -m ratchet.live export --version N --out DIR` copies the adapter folder (when it is on the machine), the BOM and that `NOTICE.md` into one folder and prints the BOM's `sha256`. - **Configured teachers**: `RATCHET_TEACHER_LICENSES` (a JSON list of `{model, license, url, note}`) adds open teachers without a code change. `licenses.install()` checks the file and publishes a copy of the built-in registry with them, in one assignment, at every entry point's startup (`Ratchet.__init__`, the gateway's `Runtime.reset`, `python -m ratchet.live`, `python -m ratchet.admin`). An entry needs a license in `ALLOWED_LICENSES` (OpenMDW-1.1, Apache-2.0, MIT, NVIDIA Nemotron Open Model License) and a model id in no closed family (`closed_family`: no part starts with a `CLOSED` prefix or `o` and a digit, or contains `gpt`, `claude`, `gemini` or a `CLOSED_NAMES` or `CLOSED_VENDORS` name) that is no router alias (`ROUTER`: `openrouter/...`, `.../auto`); it can never relabel a built-in teacher; a bad file stops startup with every error listed. Configuration therefore cannot relabel a closed model Ratchet knows of; any other id rests on the operator's claim, which every BOM and NOTICE marks `config`. - **Human labels**: provenance `human` (`licenses.SOURCES`, never in the teacher registry) is listed apart: `label_sources` in `/ratchet/bom` and in each adapter's BOM (examples by origin), `training.targets` naming them when they are among the targets, `golden.human_labels`, and a Human labels section in NOTICE.md (absent, and the text unchanged, for an adapter without them). - `train.py` writes `ratchet_manifest.json` next to each adapter, listing the base model, example count, the (teacher, license) pairs and the chat-template arguments it trained under, and `versions.json` with the library versions. - `licenses.require_distillable()` refuses unknown teachers and any model whose ID (after the last `/`) starts with `gpt`, `o1`, `o3`, `o4`, `claude` or `gemini`. ## 9. Design decisions This table lists some of the decisions, not all of them. The full numbered index of ADRs is in adr/README.md. | # | Decision | |---|---| | 0001 | Only teachers whose license allows training on outputs | | 0002 | Train and serve the student on Nebius AI Cloud, not Token Factory | | 0003 | Per-request cascade with a calibrated, per-version threshold | | 0004 | Retrain from the base model on the cumulative dataset nightly | | 0005 | Shadow → canary → live with Wilson-bound rollback | | 0006 | SQLite ledger for v0 | | 0007 | Hash-based golden split; frequent requests always train | | 0008 | Pinned tasks stay on the teacher and are never training data | | 0009 | Pattern-based PII redaction before examples are stored | | 0010 | A signed bill of materials per adapter | | 0012 | Validate and normalise OpenAI requests at the gateway; OpenAI's error envelope on `/v1/*` | | 0013 | Optional teacher calls fail open for the request and count against the shadow in the release gate | | 0014 | Bounded retries, one upstream error type, and statuses a client can act on | | 0015 | A circuit breaker per student endpoint, automatic adapter reloads, and a route on every request | | 0016 | Store only complete, valid teacher answers | | 0017 | Generation parameters by role, token caps, and the conditions stored with an example | | 0018 | `usage` reports the served answer; Ratchet's own calls go in a breakdown | | 0019 | When the teacher is down: a teacher breaker, an opt-in degraded mode and an opt-in licensed fallback teacher | | 0020 | `n` choices and token logprobs without moving the calibration or the training set | | 0021 | Field comparators and task specs: typed fields, required by default, pairwise confidence | | 0022 | A task registry: specs as data, operator-only pinning, an unknown-task policy and admin keys | | 0023 | Utilization-aware cost accounting: a split per request, a compute-time ledger, savings by GPU-hours and a self-host recommendation | | 0060 | The hot path: a WAL ledger with a read-only stats connection, a version routing snapshot, incremental audit tallies, a throttled tick, cached stats and `wall_ms` | | 0070 | Configuration checks that list every bad variable before anything opens, a config check CLI, liveness apart from readiness, and `/readyz` with cached, bounded dependency probes | | 0040 | Per-task calibration when several tasks share one adapter: thresholds, a per-task golden gate, per-task alerts | | 0054 | Free-text tasks: a similarity comparator, per-task sample counts and a judge gate; long-form stays pinned | | 0055 | Text-to-SQL tasks compared by what their queries return, in a sandboxed in-memory SQLite fixture | | 0032 | A label-free drift alarm: a conformal test martingale on each version's own confidence stream, with opt-in actions | ## 10. Scout (multi-agent research tool) Scout shows Ratchet on agent traffic. The full description and event contract are in [SCOUT.md](/reference/scout/). ```mermaid flowchart LR q(["question"]) ==> planner["Planner
research-plan, pinned"] planner ==>|"3 queries"| searcher["Searcher
Tavily or MockSearch, a tool"] searcher ==>|"up to 12 results"| triage["Triage × results
source-triage, distillable
thread pool of 6"] triage ==>|"relevant, up to 6"| reader["Reader × results
page-read, distillable
thread pool of 6"] reader ==> writer["Writer
research-write, pinned"] writer ==> answer(["answer with citations"]) class planner,writer teacher class triage,reader student class searcher outline classDef student stroke-width:1.5px classDef teacher stroke-width:1.5px classDef store stroke-width:1.25px classDef gate stroke-width:1.5px classDef pending stroke-width:1.5px,stroke-dasharray:5 4 classDef outline stroke-width:1.25px,stroke-dasharray:4 3 ``` - Every agent call is `call(messages, task)`. In the gateway it is `Ratchet.handle` in-process (`Runtime.scout()`). From outside it is `http_caller`, a plain HTTP client for `/v1/chat/completions` with an `X-Ratchet-Task` header. - Scout emits events (`start`, `step`, `done`, or `error`) through a callback. `/scout/stream` runs it in a worker thread and forwards events as Server-Sent Events; `/scout/run` collects them. - Before a run, `Guard.admit(ip, judge=, run_id=)` applies the daily budget, the daily run cap, the judges' reserve and the per-IP hourly limit. After it, `Guard.add_spend` adds the summed `cost_usd` of the steps that finished, even if the run failed. In live mode each run's `scout_run` spend row is written at admission, updated after each model call (`Guard.progress`) and closed with the outcome, and `Runtime.reset` restores today's counters from those rows. A valid `ratchet_judge` cookie (`POST /scout/judge`) makes a run a judge run. - In live mode with Tavily, each run's searches go through `MeteredSearch`, which reserves credits from the `CreditMeter` before a call and settles them after it. A refused or failed search ends the searching, not the run: the writer answers from what was found, or without sources, and `done.search` says so. The planner has called Token Factory before any search. - In mock mode, `MockSearch` returns mock results from `sim/research.py`, and `MockTeacher` answers the planner and writer prompts by recognising their system prompts. The triage and reader requests are built with the same `prompts.py` functions the simulator uses, so the mock models can look each one up by its text. - The test controls (`/ratchet/demo/day?workload=tickets|research|mixed|schema|tools`) let the dashboard train the student on Scout's calls without running Scout itself, and on structured-output and tool requests from the generic mock world. ## 11. Hot path and performance What a request and a dashboard read cost, and why it does not grow with the ledger. **The journal.** A ledger file runs in SQLite's WAL journal with `synchronous=NORMAL` and a 5 s busy timeout (`RATCHET_DB_SYNC=NORMAL`, the default). A commit appends to the `-wal` file and does not wait for the disk, so logging a request takes about a tenth of a millisecond instead of an fsync's millisecond or more; readers never wait for the writer, and the writer never waits for readers. The trade-off: a power cut or an operating-system crash can lose the last commits (never corrupt the file; a crash of the gateway process loses nothing). WAL needs shared memory between the processes that open the file, so the gateway, the cron jobs and the cost CLI must run on one host with the ledger on a local disk, never on a network or bucket mount; there, `RATCHET_DB_SYNC=FULL` keeps v0.4's rollback journal with `synchronous=FULL`. Back up with SQLite's backup API (`deploy/vm/backup.sh`, `sqlite3 ... ".backup ..."`), never by copying the file, whose recent commits may still be in `ratchet.db-wal`; before restoring a copy, remove the old `-wal` and `-shm` files. **Two connections.** The writer (`Ledger.db`) serves every write and every read the request path and the release gate make, under the ledger's lock. Stats and report queries (`_reads`: `stats_by_day`, `stats_by_task`, `stats_by_client`, `totals`, `privacy_stats`, `bill_of_materials`, `bom_status`, `events`, `cost_by_day`, `latency_percentiles`) run on a read-only connection (`mode=ro`) under their own lock, each inside one read transaction, so a dashboard read sees one snapshot, never waits for a request and never makes one wait. An in-memory ledger (mock mode) and a `FULL` ledger have one connection, as before. **The routing snapshot.** Every request reads the live, canary and shadow versions (`live()`, `by_state()`). Before the hot-path caches each read loaded every version row and parsed its eval, three times a request, so a year of nightly versions cost milliseconds per request. Now the ledger keeps the newest row of each routed state and rebuilds it only when a version was written: on its own connection, a TEMP trigger on `versions` (which exists only on that connection and fires for every write through it, a Ledger method or raw SQL) bumps a generation; for another connection (the cron tick, the cost CLI) `PRAGMA data_version` changes. A read costs one PRAGMA and one `json.loads` of the eval; each caller gets a fresh `Version`. `versions()` still reads every row (stats, Night School). **Incremental audit tallies.** `version_live_quality` (the canary's and the live alert's evidence, read every tick) keeps, per version and stage, the audit count and sum up to the last request id it read, and each call reads only the rows above it through `requests_version_audited`. SQLite assigns ids in commit order, so no row can appear below a tally later; a deleted or rewritten request on this connection, or any commit by another connection, starts the tally again. The numbers are exactly `COUNT(*)` and `AVG(audit_agree)` over the stage, so release decisions and the reference mock-mode run do not move. `shadow_quality` keeps one aggregate over the shadow's rows, which stay few: the shadow is decided once it would have served `min_shadow` answers. **The throttled tick.** `Runtime.tick()` runs the gate at most once per `RATCHET_TICK_INTERVAL_S` (5 s in live mode) and skips, without waiting, while another thread ticks, so requests pay for a tick at most once per interval. A rollback can wait up to the interval; the cron tick runs regardless. Mock mode keeps an interval of 0, a tick per request, and the mock-mode gateway's mock days tick ten times a day with `force=True`, so test runs and tests behave as before. **Cached stats.** `/ratchet/stats` is computed at most once per `RATCHET_STATS_TTL_S` (2 s in live mode, 0 in mock) by one caller at a time (`SnapshotCache`); while one recomputes, the others get the previous payload. Within a computation, `bill_of_materials` hashes every example only when the examples' count, highest id or rewrite generation moved, and `stats_by_day` sorts a day's `wall_ms` for its percentiles only when that day's count or highest id moved: a read after a day of traffic sorts today's rows only. A cache key never includes a generation the writer may not have committed yet (`Ledger._settled`). The events list reads only the 40 newest rows. **Latency.** `requests.wall_ms` is the whole cascade on the gateway's clock; `latency_ms` stays the providers' time, so `wall_ms - latency_ms` is Ratchet's overhead. `Ledger.latency_percentiles(day, served_by, qs)` gives nearest-rank percentiles (`ORDER BY wall_ms LIMIT 1 OFFSET k`, no SQLite extension), and `stats_by_day` reports each day's `p50_student_ms` and `p95_ms`, the numbers the latency target's target is checked against. `tests/test_perf.py` holds a request's own time flat with 365 versions and 200,000 request rows (zero-latency models; under 5 ms a request and 20 ms a warm tick, several times what a laptop measures). ## 12. Observability How one request is followed, and what is counted. **Request ids.** `ObservabilityMiddleware`, the outermost of the gateway's middlewares, gives every HTTP request a `RequestContext` (`obs.new_context`): the gateway's id (`req_` and 32 hex digits), the client's `X-Request-Id` or `X-Client-Request-Id` when it matches `^[A-Za-z0-9._-]{1,128}$`, and the trace id and flags of a valid W3C `traceparent`. It sits in the `obs.request_ctx` context variable for the request's lifetime; `anyio.to_thread.run_sync` copies it into the worker thread the chat route runs in, so the router and the provider read it without the id being passed through every call (a plain `threading.Thread`, such as Scout's stream worker, starts without it). The gateway passes the ids to `handle` in `RequestMeta`, and the router writes them into the request's row. Every response gets `x-request-id` and `openai-processing-ms`; Starlette sends a 500 outside every middleware, so `_unhandled_error` adds them from `scope["state"]`. The completion's `id` is `chatcmpl-` and the same digits, so either finds the row (`GET /ratchet/requests/{id}`, `GET /v1/ratchet/requests/{id}`). **Trace context.** `OpenAICompatProvider.complete` asks `obs.upstream_headers()` for headers on every HTTP attempt: inside a request that came with a valid `traceparent`, its trace id with a new span id and the same flags (and the `tracestate`), so a collector can join Ratchet's teacher and student calls to the client's trace; otherwise nothing. No span is exported: the OpenTelemetry SDK is deferred. **Failures.** The router knows which failures it swallowed and what a failed request spent; the gateway knows which requests it refused. So the router writes a `failures` row for each student, audit and shadow failure it covered (a breaker's refusal one time in ten per stage) and for a teacher failure on the serving path, with the student's spend in `cost_usd`, and marks the request's context as decided. Only an HTTP failure is the teacher's: a bug on the teacher path leaves its spend on the context (`failed_cost_usd`) for the gateway's `gateway` row. Just before the response starts, `_failure_row` writes a row for any other 401, 413, 429 or 5xx, in a worker thread so the event loop never waits for the ledger's lock, at most 120 per stage a minute (`FailureThrottle`; a row with spend is never dropped), so a lookup by the response's id finds it. A failure never starts a live ledger's day 1. A 400 from validation writes nothing. All rows go through `Ledger.log_failure`; `errors.last_hour` in `/ratchet/stats` groups the last hour's rows by stage, and `totals.failed_cost_usd` sums their spend, which counts against the savings. **Logs.** `obs.configure` installs one handler on the `ratchet` logger (`RATCHET_LOG_FORMAT`, `RATCHET_LOG_LEVEL`; JSON and INFO in live mode, text and WARNING in mock mode). `obs.log(level, event, **fields)` adds the request's id and trace id from the context and costs one level check when the line would not be written. What is logged: the chat route's one `chat` line per request (written after the response, from the fields `_complete` left on the context), an `upstream_error` warning per failed model call (role, exception class, status; an httpx message names the URL, so it is never logged), and every ledger event through `Ledger.add_event_listener(obs.mirror_event)`, scrubbed by `access.scrub_detail`. A formatter shows an attached exception as its class only. Nothing logged is message or answer text, a URL or a key. **Metrics.** `ratchet/metrics.py` keeps process-wide counters, gauges and histograms, each behind its own lock (a few dictionary operations per update). `handle` gains counter increments only: `_count_request` after the request's row is written (requests, latency, cost and baseline by client, escalations), the upstream counters and latency around each model call in `_sample` and `_teacher_complete` (`circuit_open` when a breaker refused), audits by result, new examples by split, failures by stage (unsampled) and a failed request's spend. The gateway counts a chat request that ended in an error (`served_by="none"`) and a refusal before the cascade by reason. `GET /metrics` sets the scrape-time gauges from the runtime (breakers from `Ratchet.health()`, the routed versions, today's spend by client from `Ledger.spend_since`, one indexed range per key and table) and renders the registry. A task label is a registered name, `other` or `none` (`router.task_label`); a client label a key name, `none` or `other` (`metrics.client_label`); so no request can add a series. An anonymous read of a public live dashboard gets client pseudonyms (`Registry.render(relabel=...)`). **Version transitions.** Every `set_version` the release gate makes is followed by an event naming the version; the old live version's retirement on a promotion is `live_retired`, a kind the pinned reference runs skip, so v0.4's release log is unchanged. ## 13. Configuration checks and readiness How a gateway refuses a bad configuration, and how a load balancer learns whether it can serve. **Startup.** `Runtime.__init__` reads the settings with `config.load()`, which collects a variable that does not parse instead of raising at the first (the setting keeps its default meanwhile), then runs `config.check`: `Settings.validate` (ranges, URLs, the license registry for the teacher, judge and fallback teacher, every setting's own reader, the key rings, and in live mode the teacher key, the ledger file and the student token; then unknown, mock-only and loosely spelled variables in the environment) and, in live mode, `ReleasePolicy.from_env().validate()` (the rules across fields that building a policy does not check, so tests and test runs keep any value). Errors raise one `ConfigError` before `reset()` opens the ledger or builds a provider; it is also a `LicenseError` when a teacher is refused, so code that catches either still does. Warnings stay in `Runtime.config_warnings` and are logged once, by variable name; `RATCHET_STRICT_CONFIG=1` raises them too. `python -m ratchet.config check` runs the same functions in its own process, plus the tasks file, and exits 0, 1 or 2. `config.KNOWN_ENV` is the list the unknown-variable warning compares against; `scripts/reqtrace.py` checks it against every variable the package reads. **Liveness.** `/healthz` and `/livez` are `async def` routes on the event loop, so they answer while AnyIO's worker threads (which run every sync route and the chat cascade) are all busy, and they read nothing but the settings: a restart policy on them never follows a slow dependency. **Readiness.** `/readyz` awaits `Runtime.readiness.report()` (`ratchet/readiness.py`). Each probe is a blocking function run on a probe thread, never a request thread: the ledger's (`Ledger.probe`: `SELECT 1`, then `BEGIN IMMEDIATE`, a write of `user_version` to its own value and `ROLLBACK`, waiting at most 2 s for the ledger's lock and for SQLite's) on a thread of its own, and the teacher's and the student's `GET /models` (a fresh `httpx.Client` on the runtime's transport, a 2 s timeout) on a pool of two, so two hung endpoints can never make the ledger look unwritable. `ProbeCache` keeps each result for 15 s on the runtime's clock and shares one run between every caller that arrives while it runs; a caller waits at most 2 s (`asyncio.wait`, never a cancel, so a shared run is never cut short) and reports a slower probe as `timeout`, and the run's result is kept when it ends. Breaker states, the config warnings and the flags are read on the event loop on every call (each a short lock). The student probe compares the endpoint's model list with the adapters of the live, canary and shadow versions and hands each missing one to `AdapterReloader.request`, the same queue a vLLM 404 uses; its ok result is kept for 5 minutes (`STUDENT_PROBE_TTL_S`), a failed one for 15 s. A load balancer sends nothing to a 503 instance, so nothing that needs a request may be needed to leave `not_ready`: a teacher breaker that would let the next call through as its probe (open past its wait, or half-open; `_way_through`) counts as a way through while the teacher's endpoint answers `/models` below 500, and a busy ledger (`locked`, `lock_timeout`, `timeout`) is `degraded` until two probes in a row find it busy. `Runtime.reset()` makes a new `Readiness`, so a mock-mode gateway reset forgets the old ledger's results. A 503 marks its request's context as decided, so `_after_response` writes no `failures` row for it. ## 14. Retries and idempotency How a client's retry is kept from paying twice and from counting as a repeat. **Idempotency keys (`ratchet/idem.py`, `gateway._complete_idempotent`).** A chat request with an `Idempotency-Key` header (when `RATCHET_IDEMPOTENCY_CACHE` is above 0) goes through `IdemCache.begin(scope, key, fingerprint, wait_s)` after its body parsed and before validation, the quota and the cascade. The scope is the API key's name ('' with keys off), never the key; the fingerprint is the sha256 of the parsed body with sorted keys and the `X-Ratchet-Task` header. The cache is an `OrderedDict` in least-recently-used order under one lock: an entry is running (an owner and a `threading.Event`) or finished (the answer as sent: the JSON bytes, or every Server-Sent Event of a stream, with the headers that describe the answer, the task and the log line's fields). A fingerprint that cannot be computed (a body nested just under what `parse_body` accepts, which `json.dumps` cannot encode again) is the same 400 `invalid_json` as a body that does not parse, before the key is claimed. `begin` returns a claim the request owns (it then runs `_complete` and calls `finish` in a `finally`: the answer to keep, or None to release the key), or a finished answer to send again (`_replay`: `Idempotent-Replayed: true`, no cascade, ledger row, quota or tick); it raises `KeyReused` for another fingerprint and `InFlight` when a wait for a running owner (at most `RATCHET_TEACHER_DEADLINE_S` of real time, on the worker thread the request already holds) runs out, or at once when `MAX_WAITERS_PER_KEY` (4) requests already wait for that key (a waiter holds a worker thread and uses no quota). A waiter wakes when the owner finishes: with its answer even when it is not kept (TTL 0, larger than the 64 MiB budget), or, when the owner failed, to claim the key itself. Finished answers expire `RATCHET_IDEMPOTENCY_TTL_S` after they finished, on the runtime's clock (a sweep at most once per half TTL frees expired ones); the least recently used finished answer goes first when the count or the bytes (each answer's body plus `ENTRY_OVERHEAD`, 2 KiB for the entry itself) are over; a running entry is never dropped. A stream with a key is rendered before its first event (`BufferedStream`), as it always was in substance (the cascade decides before streaming), so the same events can be kept; a stream without a key is sent as before, and a response that is not a 200 JSON body or a buffered stream is never kept. A replay's log line repeats `served_by`, `route` and `version`, names the request that ran (`replay_of`) and has `cost_usd` 0. Everything is in memory (answers carry request-derived text) and per process; `Runtime.reset()` makes a new cache. **The seen window (`Ledger.add_example`, `idem.SeenWindow`).** `seen` counts how often a request occurred, and an example seen `FREQUENT` (3) times moves from the golden set to training (`dataset.rows`). `Ratchet._store_example` passes `count_seen=False` when `RequestMeta.retry_count` (the gateway's reading of `x-stainless-retry-count`) is above 0, and the sender: the API key's name ('' with keys off) of the request the gateway is serving (`Ratchet._own_context`), or None outside one (Scout's agents, the simulator, the mock warm-up). Under the ledger's lock, a new example records (hash, sender) in the window; a duplicate bumps `seen` only when `count_seen` is true and the window (when the sender is not None and RATCHET_SEEN_DEDUP_S is above 0) has no count of that pair within the window, which it then records. A skipped repeat does not restart the window, so a client retrying every few minutes still counts once per window. The window remembers at most 10,000 pairs, the least recently counted forgotten first; a refused INSERT's transaction is committed either way. The request itself is still logged, with `requests.retry_count`, and its answer served; only `seen` is protected. ## 15. Off-path quality checks How a request's optional teacher work leaves the request path without changing the release signals. **Why.** An audited student-served request waited for a whole teacher call (seconds, against a few hundred milliseconds for the student), and while a shadow was in flight every request also waited for its samples and any arbitration. So the p95 of student-served requests was the teacher's latency, against the latency target. None of that work changes the answer the client gets. **The sidecar (`ratchet/sidecar.py`).** `SideCar(workers, max_queue)` is a bounded queue on daemon threads that start on first use. A request takes a place before it logs its row (`reserve()`, never blocking: None when the queue is full, the pool is closed or it has no workers), then hands the job over (`Slot.submit`) or gives the place back (`Slot.cancel`, when logging the row raised). `max_queue` counts places taken and jobs waiting, not running ones; the last tenth of the places (`audit_places`, at least one once the queue has two) only a job that carries an audit may take (`reserve(priority=True)`), so shadow work alone can never fill the queue against audits. Each job keeps its request's messages until it ends, so `max_bytes` (64 MiB) bounds the request text queued and running jobs hold (`reserve(size=job_bytes(messages))`, the same share kept for audits); a reservation past it is refused like one for a full queue. Each job runs in a copy of the submitting thread's context variables, so its log lines, failures rows and the ledger's seen window name the request that queued it. A job that raises is counted and logged by its exception's class. `close(timeout_s)` takes no new places, waits for the queue to empty, then gives up on the jobs still queued and returns them counted by tag; a job handed over after that is refused (`submit` returns False) and its request runs it itself. **The request path (`Ratchet.handle`).** `_offpath(meta)` is true for a request the gateway serves (`RequestMeta.offpath`, default true) while the sidecar has workers and is open. Scout's agents, the simulator and the mock warm-up pass no meta, so their checks stay inline whatever the settings: Scout's budget guard sums `Result.cost_usd`, and the seeded test runs keep every call in its old order. Every random draw stays on the request path in the same order (the canary pick, then the audit draw), so the same requests are audited however the work runs. When the audit is drawn, the request reserves a place with priority; when no place is left even then, the audit runs inline as before (`audits_inline`, a `sidecar_inline` event on the 1st, 10th, 100th and 1000th time): an audit is never dropped, because a sample that thinned out under load would leave the release gate a sample of quiet hours. A shadow rides with the request's audit, or takes a place of its own; with none left it does not run for that request (`shadows_dropped`, a `sidecar_dropped` event). Shadow evidence is a pre-filter before the audited canary, whose Wilson bound still decides with every drawn audit. The row is logged with `audited` 1 and `audit_agree` NULL for a pending audit, and `shadow_version` set with `shadow_serve` NULL for a pending shadow; then the job is submitted, and the `Result` says `overhead_pending` with what serving cost (`cost_usd`, its split, the formula's baseline; `audited` false; no audit, shadow or arbitration in the breakdown). **The job (`Ratchet._finish_offpath`).** It makes the calls the request path would have made, with the same rules: `_audit` (a failure or an open breaker is no audit and stops the arbitration; an invalid answer measures nothing), then `_shadow` with the counted audit, or the escalation's teacher answer, as its reference, through the shadow's own breaker; the arbitration's retries end after `ARBITRATION_DEADLINE_S` (60 s; an unarbitrated disagreement counts against the shadow, so the shorter wait cannot flatter it), the audit keeps the teacher's deadline (a slow audit dropped more often than a quick one would bias the audited precision). Teacher answers are stored through `_store_example` as before. Then one `Ledger.update_request(row, ...)` writes the verdicts, adds the calls' costs and tokens one after the other in the request path's order (so the finished row holds the floats an inline run writes, to the last bit), raises `stored`, and after a teacher audit sets the baseline the audit prices. The job adds its cost to `ratchet_cost_usd_total` (and an audited request's baseline to `ratchet_baseline_usd_total`, which the request left out) and writes one `offpath` log line under the request's id. `tests/test_offpath.py` runs the same traffic inline and off the path and compares every row, example, event and failure. **The ledger.** `update_request` is `@_locked` and writes a verdict only while it is pending (`audited`=1 and `audit_agree` NULL; `shadow_version` set and `shadow_serve` NULL), so a verdict is written once; any other column raises before anything is written. Its writes rewrite no column a cache keys on, so they are left out of the `requests` generation the hot-path caches compare (`_offpath_bumps`, `_requests_gen`). `version_live_quality` counts audits with a verdict only (`COUNT(audit_agree)`, `AVG(audit_agree)`), so the count and the precision describe the same rows; its incremental tally keeps the ids of pending audits it read and settles them on each call (counted when their verdict is in, forgotten when the audit failed). `shadow_quality` already reads only rows with `shadow_serve` set, and `stats_by_day`'s audit agreement averages verdicts only. **Shutdown.** The app's lifespan calls `Runtime.shutdown()` after the server stopped taking requests: `Ratchet.close(RATCHET_SIDECAR_DRAIN_S)` waits for the queue, then gives up on the rest. Those rows stay pending for good, which every statistic ignores, and one `sidecar_dropped` event counts them per version; a job already running finishes in its thread. After `close`, requests run their checks inline again. A live gateway reset closes the old cascade's sidecar without waiting. A crash loses the queued jobs the same way (their rows stay pending). `/ratchet/stats` `sidecar` reports the queue and, per routed version, what could not go through it; `/metrics` has `ratchet_sidecar_jobs_total`, `ratchet_sidecar_full_total`, `ratchet_sidecar_job_seconds` and the `ratchet_sidecar_jobs` gauge. ## 17. Spend and rate limits What a key and the whole gateway may spend and send. Three parts, each per process (one worker) and keyed by key name, never the key (invariant 13): **Key limits.** `auth.parse_key_limits` turns `RATCHET_KEY_LIMITS` (`acme:daily_usd=5,rpm=60,tasks=a|b;globex:store=0`) over the `RATCHET_API_KEY_*` defaults into one `KeyLimits` per key name at startup; a name that is not a key, an unknown field or a bad value stops startup. `Runtime.limits_for(client)` is a dictionary lookup. **Rate limits (`ratchet/ratelimit.py`).** Both steps run on the event loop, so a refused request never takes a worker thread. `_rate`, after the API key and before the body is read: `Limiter.rate` takes a token from the key's bucket (`rpm`, a burst of `max(1, rpm // 6)` refilled continuously on the runtime's monotonic clock), or on a gateway without keys from the client address's bucket (`RATCHET_IP_RPM`; at most 10,000 addresses, least recently seen dropped first). `_hold`, once the body is in and just around the worker thread: `Limiter.hold` takes a slot of the global in-flight cap (`RATCHET_MAX_INFLIGHT`, 32, below the 40 threads; 503 `overloaded`), then one of the key's in-flight cap (`concurrency`; 429). A refusal gives back what it took. The caps protect the thread pool, so a client that sends its body slowly holds no slot. The slots go back when `chat` returns its response, which the cascade has finished by then (the stream only formats the answer). **Spend (`ratchet/spend.py`).** `SpendMeter` holds, in memory and behind one lock, the answered requests and US dollars per key per UTC day, the dollars per key per UTC month and the gateway's dollars per UTC day, for gateway API requests only (the rows with a gateway request id: Scout's agents and the simulator call `handle` without one, so their spend stays under their own guard, invariant 11). It reads the ledger once, at startup (`Ledger.api_usage_since`: today's requests and failed spend, this month's when a monthly cap is set, one pass over each table's `(client, ts)` index), so the quota and budget checks in `_complete` read memory only. After each request the gateway adds its `cost_usd`, or for a failed request what it spent before failing (`RequestContext.spent_usd`, which the router fills); a job whose cost for a request lands later adds it with `requests=0`. A new UTC day or month starts from zero (re-reading the ledger then could count twice a request logged before the read and added after it); a clock that moves back to an earlier day reads the ledger again. After an add, the meter writes a `budget` event for each `RATCHET_BUDGET_ALERT_PCT` share of a cap reached for the first time in its period, outside its lock; it reads the period's `budget` events at startup, so a restart repeats none. **The gateway's budget.** Past `RATCHET_DAILY_BUDGET_USD`, `block` refuses every chat request; `student_only` sets `RequestMeta.student_only` on each `/v1` request (a per-request flag, so Scout and the simulator are never affected), and `handle` then serves a confident student answer with no audit (the random draw is still made) and no shadow, and raises `spend.BudgetExceeded` before any teacher call for an escalation, a cold start, a pinned or unregistered task or a request with `tools`. What the student's samples cost by then gets a `budget` failures row and counts against the key, as for any failed request. The gateway maps `BudgetExceeded` to 429 `gateway_budget_exceeded`. ## 18. Admin API and runtime switches Manual interventions go through one module, `ratchet/admin.py`, which the gateway's `/ratchet/admin/*` routes (admin key; `admin` prefix in `access.py`) and `python -m ratchet.admin` (on the ledger file, actor `cli:`) share. Each function checks its reason and actor, checks the move against `MOVES` (canary → rolled back, retired, another share or live; live → retired; shadow → retired or canary; rejected, rolled back, retired or candidate → canary only with `force`; never a version without a golden eval), and writes the change (`transition` / `promote_live`, compare-and-set), an `admin_actions` row and an `admin` event in one `Ledger.transaction()`. A refused or stale move raises `AdminConflict` (409); a call that changes nothing writes nothing. A promotion's `load_adapter` (live mode) loads the LoRA adapter into vLLM after the checks and outside the transaction, so a slow endpoint never holds the ledger. **Runtime switches.** `Ledger.flags()` returns a `RuntimeFlags` snapshot of `runtime_flags` and `task_overrides`, rebuilt only when a write on this connection (TEMP triggers, `WATCHED`) or another process's commit (`PRAGMA data_version`) changed them, as the routing snapshot is ([§11](#11-hot-path-and-performance)); `handle` reads it once per request. Its effects: | Switch | Effect | |---|---| | `student_disabled` | `_pick_version` returns none: the teacher answers (route `student_disabled`), its answer is stored as on a cold start, no student or shadow call | | `releases_frozen` | The gate starts no version taking traffic and gives none more (no shadow → canary, no canary stage or promotion; a cold start shadows the teacher); rollbacks, rejections and the live alert run | | `training_paused` | `run_night` writes `skip` before anything else | | `audit_rate` | Replaces `Settings.audit_rate` for the audit draw (made either way, so the random stream is the same); at 0 the gate starts no new canary, and setting 0 during a canary needs `force` | | Runtime pins | `flags()` publishes the names to `tasks.OVERRIDES`; `tasks.get` and `tasks.effective` return a pinned copy of the registered task (its spec's `spec_hash` kept), so the router's pinned path (`_teacher_only`, nothing stored) and Night School's read-time exclusion (`excluded_pinned`) apply as for a spec pin (invariant 10) | A switch is seen by every process that shares the ledger on its next request, tick or night: no restart. `/ratchet/stats` shows them under `controls`. ## 16. Fallback attribution Which answers a version's release statistics count. **Why.** Without `RATCHET_STUDENT_BASE` the live resolver returns the hosted, untuned fallback model (`RATCHET_FALLBACK_STUDENT`) for every version, canary and shadow included. Its audited answers used to count as the routed version's, so a canary could be promoted or rolled back, and the live alert fire, on another model's answers; the shadow compared the fallback with itself. **The rule.** An answer is a version's only when the model the resolver named for it is the version's `adapter_uri` (`Ratchet._own`). Ratchet's resolvers (the vLLM endpoint, the mock fleet) name the adapter; the fallback names its own id. **The request path.** `handle` records the student's model (`requests.student_model`) beside the routed `student_version`. An answer another model wrote is served under the global threshold (`Ratchet.threshold`, as before) and audited as drawn, but its `Result` names no version (`version` None), `model` is the configured fallback id, and its fingerprint is `student||` (`_student_fingerprint`). The shadow is resolved once, before any sidecar place (`_shadow_student`): if it resolves to another model it does not run; if the resolver raises, that is a failed shadow sample. `shadow_model` is set exactly when `shadow_version` is, and an off-path shadow whose samples fail clears it with its verdict (`update_request` treats it as a verdict column). A throttled `unattributed` event names the version and the model. **The ledger.** `OWN_STUDENT` and `OWN_SHADOW` (`student_model IS NULL OR student_model = (SELECT adapter_uri FROM versions WHERE id=?1)`) are part of `AUDIT_TALLY_SQL`, `AUDIT_PENDING_SQL` and `SHADOW_QUALITY_SQL`: the adapter is read once per query by primary key, and the requests indexes are searched as before. `stats_by_day` adds `fallback_served`. **What follows.** While the fallback serves, no version gets evidence: the canary and the shadow hold, the live alert stays quiet, and Night School skips while that release is in flight. Rows from before v0.5 have no model and count as they did; the reference runs are unchanged. **The hosted student** (off by default). With `RATCHET_HOSTED_STUDENT=1` in live mode the same hosted model can answer before any version exists. The gateway keeps one version row in state `hosted` (`Ledger.ensure_hosted`: the hosted id as base model, `adapter_uri` empty, no training examples), whose eval holds the per-task records in `ratchet/hosted.py`, and writes a `hosted` event when it makes or changes it. `Ratchet._pick_version` returns that row only while no version is live (after the canary's draw); `Ratchet.threshold` holds it to its records (`calibration.threshold` with SERVE_NOTHING as the global threshold), and `_held_back` keeps it from being sampled for a task without a serving record (route `task_disabled`). The resolver always sends it to Token Factory. Its adapter is empty, so by the rule above its answers are never its own: audited as drawn, never stored, counted by no release. `/ratchet/stats` leaves the row out of `versions`; `student_serving` reports `hosted_fallback` with each task's threshold (`tasks`) only while it answers some task. The records come from `scripts/hosted_nano_calibration.py` over `evidence/hosted-nano/` (Ultra's labels, the Nano's samples, `cascade_eval`); a task serves only at 95% held-out precision against Ultra on 20 served rows. As measured on Oct 26, 2026 none does, so the switch changes no answer yet. ## 19. Hosting: the live and mock-mode gateways The public live gateway runs as a Cloudflare Worker with a Container (`deploy/cloudflare-live`). The Nebius VM kit, the alternative host (scripted and tested offline), serves the live gateway at the root domain from one image behind one Caddy, and can run `ratchet-sim` (a mock-mode test container, in-memory ledger, no keys, 1 vCPU and 1 GiB) at `sim.`. stays in the app, so the live host answers every test-control route 403 whatever Caddy does. ```mermaid flowchart LR caddy["Caddy"] ==>|"#lt;domain#gt;"| live["ratchet
live: Token Factory, student endpoint, Tavily"] caddy -->|"sim.#lt;domain#gt;"| sim["ratchet-sim
mock: test controls open, per-IP limit, self-reset"] class live teacher class sim outline classDef student stroke-width:1.5px classDef teacher stroke-width:1.5px classDef store stroke-width:1.25px classDef gate stroke-width:1.5px classDef pending stroke-width:1.5px,stroke-dasharray:5 4 classDef outline stroke-width:1.25px,stroke-dasharray:4 3 ``` - `ratchet/demo.py` (mock mode only): `DemoPolicy`, and the `DemoJanitor` that resets the shared mock-mode gateway after N mock days or requests (on a click) or idle time (a daemon thread). `gateway._demo_limit` is an app-level dependency after `_access`: a per-IP bucket (`ratelimit.RateLimiter`) on non-GET `/ratchet/demo/*` calls. - `ratchet/upstream.py`: the last real upstream call per role, content-free. The provider and the Tavily client report to an `observe` callback in a `finally`; the recorder sits beside the metrics (`ratchet_upstream_*`, counters by purpose) and keeps a model id, status, latency and time instead. `/ratchet/stats` carries it as `upstream`, with `links` and `demo`. --- # Testing Source: https://docs.ratchet.dualmindlab.tech/reference/testing/ ```bash pip install -e ".[dev]" pytest -q # 2014 tests (test_core, test_tasks, test_registry, test_nightschool, test_release, test_system, test_hardening, test_resilience, test_scout, test_contract, test_openai_compat, test_traceability, test_live, test_cost, test_stats, test_provenance, test_receipts, test_privacy, test_erasure, test_redaction, test_kpis, test_licensing, test_feedback, test_docs_claims, test_perf, test_security, test_observability, test_readiness, test_idempotency, test_offpath, test_limits, test_admin, test_calibration, test_mock_world, test_structured, test_tools, test_model_routing, test_routing, test_adjacent_endpoints, test_drift, test_teacher_watch, test_judges_page, test_livesim, test_attribution, test_exam_numbers, test_reasoning, test_toolcall_results, test_hosted_student, test_demo_split, test_watch), ~120 s pytest -q --runslow # also the full-size Monte Carlo runs marked slow (as CI does on main; pull requests run the quick versions) pytest -q --cov=ratchet --cov=sim --cov-report=term # with coverage (as in CI) python -m sim.simulate --days 10 --requests 2000 --bad-deploy-version 3 python -m sim.simulate --workload research --days 8 --requests 1500 # Scout's triage + reader calls python -m sim.simulate --bad-deploy-version 3 --compare-release # Wilson vs the anytime-valid gate, same seed python -m sim.simulate --calibration ltt --paired-gate # certified thresholds and the paired gate python -m sim.simulate --receipts --teacher-reasoning-mean 420 # savings receipts against the true cost python -m sim.simulate --workload schema --days 8 --requests 1500 # structured outputs (the generic mock world) python -m sim.simulate --drift-boost 3 --drift-early-night # the drift alarm's opt-in actions python -m sim.simulate --days 10 --teacher-noise 0.15 --teacher-noise-day 5 # an unstable teacher freezes Night School python scripts/regen_sim_reference.py --check # does the pinned reference simulation still match? (writes nothing) python scripts/reference_check.py # do tickets, mixed and research still reproduce v0.4.0? (make reference, CI) ``` | File | Tests | |---|---| | `tests/test_core.py` | 67 (44 functions; one has 19 cases, one has 5; 1 slow: skipped without `--runslow`) | | `tests/test_tasks.py` | 129 (44 functions; one has 37 cases, one has 20) | | `tests/test_registry.py` | 30 | | `tests/test_nightschool.py` | 31 (25 functions; one has 5 cases) | | `tests/test_release.py` | 43 (33 functions; one has 8 cases, three have 2) | | `tests/test_system.py` | 21 | | `tests/test_hardening.py` | 105 (70 functions; one has 11 cases, two have 8, one has 6, one has 4, two have 2) | | `tests/test_resilience.py` | 90 (50 functions; one has 22 cases, two have 7, one has 6, one has 3) | | `tests/test_scout.py` | 45 | | `tests/test_contract.py` | 59 (47 functions; one has 9 cases, one has 4) | | `tests/test_openai_compat.py` | 144 (37 functions; one has 87 cases, one has 22) | | `tests/test_live.py` | 83 (3 of them strict xfails for) | | `tests/test_nebius_script.py` | 12 | | `tests/test_livesim.py` | 19 | | `tests/test_cost.py` | 48 (30 functions; one has 11 cases, one has 9) | | `tests/test_stats.py` | 19 (2 of them slow: skipped without `--runslow`) | | `tests/test_provenance.py` | 26 | | `tests/test_receipts.py` | 44 (32 functions; one has 8 cases, one has 5) | | `tests/test_privacy.py` | 27 | | `tests/test_erasure.py` | 58 (37 functions; one has 13 cases, one has 4, three have 2) | | `tests/test_redaction.py` | 45 (27 functions; one has 9 cases, one has 7, one has 3) | | `tests/test_kpis.py` | 13 (9 functions; one has 5 cases) | | `tests/test_licensing.py` | 10 | | `tests/test_feedback.py` | 32 | | `tests/test_perf.py` | 25 | | `tests/test_security.py` | 12 | | `tests/test_observability.py` | 29 (28 functions; one has 2 cases) | | `tests/test_readiness.py` | 52 (21 functions; one has 31 cases, one 2) | | `tests/test_idempotency.py` | 25 | | `tests/test_offpath.py` | 32 | | `tests/test_attribution.py` | 10 | | `tests/test_limits.py` | 29 | | `tests/test_admin.py` | 39 (27 functions; one has 13 cases) | | `tests/test_calibration.py` | 28 (26 functions; one has 3 cases) | | `tests/test_mock_world.py` | 21 | | `tests/test_structured.py` | 45 | | `tests/test_tools.py` | 26 (20 functions; one has 7 cases) | | `tests/test_model_routing.py` | 26 | | `tests/test_routing.py` | 54 (22 functions; one has 33 cases) | | `tests/test_adjacent_endpoints.py` | 24 | | `tests/test_sqlcheck.py` | 25 (22 run SQL, which needs Python 3.11 or later, and one needs Linux: skipped elsewhere) | | `tests/test_drift.py` | 48 (42 functions; one has 5 cases, two have 2; 6 of them slow: skipped without `--runslow`) | | `tests/test_teacher_watch.py` | 49 (45 functions; one has 3 cases, two have 2; 1 of them slow: skipped without `--runslow`) | | `tests/test_judges_page.py` | 5 | | `tests/test_judges_pack.py` | 4 | | `tests/test_exam_numbers.py` | 4 | | `tests/test_story_real_numbers.py` | 4 | | `tests/test_reasoning.py` | 7 | | `tests/test_toolcall_results.py` | 11 | | `tests/test_hosted_student.py` | 11 | | `tests/test_demo_split.py` | 13 | | `tests/test_watch.py` | 70 (38 functions; counted on Linux, the file imports `fcntl` and does not collect on Windows) | | `tests/test_student_endpoint_evidence.py` | 4 | | `tests/test_nightschool_nebius_evidence.py` | 7 | | `tests/test_counters.py` | 6 | | `tests/test_measured_numbers.py` | 7 | | **Total** | **2014** | Tests without a table of their own yet: `test_model_routing`, `test_attribution`, `test_exam_numbers`, `test_reasoning`, `test_toolcall_results`, `test_hosted_student`, `test_demo_split` and `test_judges_page` (the file names and the docstrings of their tests say what each covers); `test_watch` has a paragraph below and no table. There are three layers: 1. **Unit tests** (`tests/test_core.py`, `tests/test_tasks.py`, `tests/test_nightschool.py`, `tests/test_release.py`, `tests/test_stats.py`, `tests/test_provenance.py`, `tests/test_receipts.py`, `tests/test_calibration.py`, `tests/test_sqlcheck.py`, most of `tests/test_drift.py` and `tests/test_teacher_watch.py`): the pure logic of each module (the exact statistics and the Merkle trees against hand-computed values, brute force and seeded Monte Carlo), and every Night School and release-gate rule tested directly against a small ledger with recording fakes (`tests/fakes.py`). 2. **System tests** (`tests/test_system.py`, `tests/test_resilience.py`, `tests/test_scout.py`, `tests/test_openai_compat.py`, `tests/test_live.py`, `tests/test_cost.py`, `tests/test_privacy.py`, `tests/test_erasure.py`, `tests/test_redaction.py`, `tests/test_kpis.py`, `tests/test_licensing.py`, `tests/test_feedback.py`, `tests/test_perf.py`, `tests/test_security.py`, `tests/test_observability.py`, `tests/test_readiness.py`, `tests/test_idempotency.py`, `tests/test_offpath.py`, `tests/test_limits.py`, `tests/test_admin.py`, `tests/test_mock_world.py`, `tests/test_structured.py`, `tests/test_adjacent_endpoints.py`): the whole loop over simulated days, the gateway API (including OpenAI request validation and the real OpenAI SDK in-process), the HTTP clients, Scout end to end, the live-mode and operational entry points run offline ([Live-path tests](#live-path-tests)), the hot path's caches, journal and time bounds ([Hot-path tests](#hot-path-tests)), who may call each route ([`tests/test_security.py`](#teststest_securitypy)), request ids, failures, logs and metrics ([`tests/test_observability.py`](#teststest_observabilitypy)), the startup configuration check, liveness and readiness ([`tests/test_readiness.py`](#teststest_readinesspy)), retry de-duplication ([`tests/test_idempotency.py`](#teststest_idempotencypy)), and audits and shadows run after the response ([`tests/test_offpath.py`](#teststest_offpathpy)), and the spend and rate limits ([`tests/test_limits.py`](#teststest_limitspy)). One of them pins the reference simulation exactly ([below](#the-pinned-reference-simulation)). 3. **Simulation** (`sim/`): a reproducible offline world with a mock teacher, mock students that learn, synthetic traffic, drift and fault injection. It is used for examples, and to find design bugs before spending on GPUs. Across the first two layers, the **contract tests** (`tests/test_contract.py`) pin what clients, the dashboard and the license rules depend on, so a rewrite of `Ratchet.handle` or `gateway.chat` cannot change it unnoticed ([Contract tests](#contract-tests)). Shared fixtures (`gw`, `gw_env`, `day1_check`, `reference_check`, `reference_run`) and the `slow` marker (`--runslow`) are in `tests/conftest.py`; `make_openai_client(app)` (the OpenAI SDK wired to the gateway in-process) is in `tests/sdk_bridge.py`; the recording provider, trainer and evaluator, `FakeClock`, the live-mode doubles (`FakeOpenAIServer`, `RecordingRun`, `AppTransport`) and the ledger builders (`_version`, `_audit`, `_shadow`, `_examples`) are in `tests/fakes.py`. [tests/AGENTS.md](https://github.com/harshbhx/ratchet/blob/main/tests/AGENTS.md) lists them. Every test names the requirements it proves with `@pytest.mark.req`. None of these touch Token Factory or a GPU. Live behaviour is checked with `scripts/day1_check.py` and the steps in [DEPLOYMENT.md](/reference/deployment/#5-verification-after-deploy); `tests/test_live.py` runs the same code paths offline first. ## What CI runs A push to `main` and a manual run (Actions, CI, Run workflow) run every check. A pull request runs only the checks for the areas it changed. The first job, `changes`, decides which (`.github/workflows/ci.yml`). | Area | Paths | Checks | |---|---|---| | backend | `ratchet/` (not `ratchet/web/dist/`), `sim/`, `tests/`, `scripts/`, any `.py` file, `pyproject.toml`, `Makefile`, `Dockerfile`, `.env.example` | `test`: ruff, pytest on Python 3.10 to 3.13 with `--runslow` (coverage on 3.12), reqtrace, the config check, the reference workloads, the simulation smoke test. Also `docs`, `docker`, `evidence` and `links`, which read the code | | web | `web/`, `ratchet/web/`, `ratchet/dashboard.html` | `web`: type check, `npm test`, build, the committed build is current. Also `workers`, whose tests check every link to the video counter. The Python tests that read the pages | | docs | `docs/`, `docs-site/`, any `.md` file, `design/`, `LICENSE`, `NOTICE`, `CITATION.cff` | `docs`, `links`, `workers` (every link to the video counter). The Python tests that read the docs | | evidence | `evidence/`, the exam and tool-calling scripts, the exam and tool-calling data files in `web/src/data/` and `docs-site/src/data/` | `evidence`. The Python tests that read `evidence/` | | deploy | `deploy/`, `Dockerfile`, `.dockerignore`, `.github/` | `docker`, `workers` (the video link Worker in `deploy/cloudflare-video/`: `npm test` against a fake D1 that runs its migrations, the type check, a Wrangler dry-run bundle; and the `/video` entry points' tests in `web/tests` for the Pages proxy and the live and simulation Workers), `workflows` (every workflow file parses). The Python tests that read the deploy kit and the workflows | - `secrets` always runs. - A change to `ci.yml` itself, or to a file no area names, runs everything. - A deleted or renamed file also runs `links`, since a link may point at it. - Without a backend change, reqtrace and the Python tests for the other areas run once, on Python 3.12, in the `area-tests` job. With one, the full matrix runs them. The lists of those tests are at the top of `ci.yml` (`AREA_TESTS_*`), and the matrix checks that every test they name still exists. - `ci-ok` fails when any job failed or was cancelled. A skipped job counts as passed, except on a full run, where every job must pass. It is the one check to require. ## What each test covers ### `tests/test_core.py` | Test | Covers | |---|---| | `test_task_agree_ignores_free_text_and_reasoning` | `Task.agree` compares only the task's fields (`category` and `priority` as labels), case-insensitively, and ignores `` blocks and code fences | | `test_invalid_json_never_agrees` | Two invalid answers never count as agreeing | | `test_strip_reasoning_cases` (19 cases) |: `strip_reasoning` removes a full `` pair, everything before a lone `` (also before a fence, which it keeps), an unterminated `` that opens what is left with what follows (`''`, also after whitespace, a pair or a lone ``), but keeps one after answer text (content, such as a reply that mentions the tag), tags in upper or mixed case, nested and repeated blocks (the last `` wins); text without tags is unchanged apart from surrounding whitespace; `None` is `''`; stripping twice changes nothing | | `test_clean_answer_unwraps_fences_only_for_structured_tasks` |: for `ticket-triage`, one fence around the whole answer (`json`, `JSON`, none, single-line, closing fence on the last line) is unwrapped after reasoning is stripped; a free-text task keeps it. Kept as is: a fence whose body is not JSON, two fences, prose around a fence, a four-backtick fence, a body nested too deeply to parse | | `test_answers_that_mention_think_tags_keep_their_text` |: a structured answer that is one JSON object quoting `` or `` in a string is kept whole, bare, fenced, after a reasoning pair or after a lone `
`; its canonical labels and `parse_json` value are right, and cleaning it twice changes nothing; a free-text answer that mentions `` keeps it; reasoning cut off while holding an object or quoting a tag is still `` (review finding) | | `test_empty_answers_are_invalid_for_every_task` |: an answer that is empty after cleaning (whitespace, unterminated or reasoning-only) is `` for free-text and structured tasks, never agrees, and three of them score 0 with `valid` false | | `test_unanimous_confident_samples_score_high` | 3 identical samples at p=0.95 give agreement 1.0 and confidence above 0.9 | | `test_disagreement_lowers_confidence` | 2-of-3 majority gives agreement 2/3, confidence below 0.85, and the majority answer is chosen | | `test_all_invalid_scores_zero` | All invalid samples give confidence 0 | | `test_length_samples_are_invalid_in_score` |: samples with `finish_reason` `length`, `content_filter`, `abort`, `unknown` or another value but `stop` and `tool_calls` get no vote even when their text parses (three of them: confidence 0, `valid` false); with two cut-off samples and one complete one, the complete one is the answer at agreement 1/3; `tool_calls` votes as usual | | `test_calibrate_threshold_hits_target_precision` | The calibrated threshold actually reaches the target precision, **including ties** (the regression test for bug 5 below) | | `test_closed_teachers_are_refused` (5 cases) | `gpt-5.6-sol`, `openai/gpt-4o`, `claude-opus-5-5`, `anthropic/claude-haiku`, `gemini-3.1-pro` all raise `LicenseError` | | `test_open_teacher_allowed_and_unknown_refused` | Ultra maps to OpenMDW-1.1; an unknown model is refused | | `test_examples_dedupe_and_count_and_stable_split` | Duplicate requests are stored once with `seen=2`; the split is the same in a fresh ledger | | `test_bill_of_materials_lists_licenses` | BOM groups by license and has a 64-character fingerprint | | `test_wilson_bounds` | 19/20 interval contains 0.95; n=0 gives (0, 1) | | `test_cold_start_goes_to_canary_and_bad_candidates_rejected` | Golden precision 0.80 is rejected; with no live version, a good candidate goes to canary at 10% | | `test_regression_vs_live_is_rejected_and_good_one_shadows` | A candidate worse than live on coverage is rejected; an equal-or-better one goes to shadow; live is untouched | | `test_canary_advances_then_goes_live` | 12 clean audits move canary 10 → 50; with no new audits it **holds** at 50 (per-stage counting); 12 more promote it to live and retire the old one | | `test_canary_rolls_back_on_strong_evidence` | 20 audits at 50% precision cause a rollback; live is unchanged; a `rollback` event is written | | `test_audit_counts_and_served_counts` |: `Ledger.audit_counts` over a version's life and after a request id counts only student-served audited rows (a teacher-served audited row is not one; an audit without a verdict counts against the version); `version_live_quality` agrees; `served_counts` includes a stand-in (degraded) answer and leaves teacher-served rows out | | `test_gen_params_from_a_validated_request` |: `GenParams.from_request` takes every generation field of a validated request (`max_completion_tokens` as `max_tokens`, `stop` as a tuple, an unknown field in `extra`) and `to_body` sends them by OpenAI's names, never `user`, `safety_identifier`, `store` or `metadata`; empty `tools` and `logit_bias` count as not sent; `only` keeps the object when nothing is dropped; an extra key never replaces a standard field or sets `model`, `n` or `messages` | | `test_param_allowlists_parse_and_refuse_what_ratchet_never_forwards` | (and: the default teacher list includes `logprobs` and `top_logprobs`: and the tool-call fields, so it is every standard field): empty is the default, `none` nothing, `all` the standard fields, `all,top_k` adds a vendor field, `max_completion_tokens` means `max_tokens`, names match in any case; `model`, `messages`, `n`, `stream`, `user`, `safety_identifier`, `store`, `metadata`, `service_tier`, `functions` and a name that is not an identifier raise, and so does a near miss of a standard field (`max_token`, `stops`, `temprature`, `tool`: "did you mean"), while vendor fields (`top_k`, `min_p`, `min_tokens`, `repetition_penalty`, `max_new_tokens`, `guided_json`) pass; the default lists leave out tool calls and the teacher's logprobs | | `test_gen_policy_gives_each_role_its_parameters_and_caps` |: the student gets the shaping fields capped at 1,024 and never the client's temperature or logprobs; the serving teacher gets every standard field by default, tools included (logprobs only when sent; all of them when allowed; a vendor field only when listed), at temperature 0 when none was sent, capped at 8,192; audits and arbitrations run at 0 without `CHECK_DROPS`, keeping limit, stops, format, tools and effort; the headroom adds to the teacher's limit under the cap and `visible_limit` says when an answer can exceed the client's; a field outside an allowlist is never sent but the caps apply; 0 means no cap; bad caps raise | | `test_stored_conditions_join_the_hash_only_when_set` |: no conditions, sampling fields only, or the default text format hash like v0.4; `response_format` and `stop` each give another hash; a tuple and a list of the same stops are one example; `GenParams.from_stored` reads stored conditions back and leaves out values of the wrong shape | | `test_v04_example_hash_unchanged` |: literals computed with the v0.4 code (a Unicode ticket and a plain request) still come out of `example_hash`, and `add_example` with only a seed stores that hash with no conditions | | `test_redact_json_redacts_string_values_and_keeps_keys` |: emails in a stop string and a tool description become `[EMAIL]` (counted), keys and numbers stay, the input is not changed, and 5,000 levels of nesting are fine | | `test_served_logprobs_spell_the_served_text_and_drop_reasoning` |: `providers.served_logprobs` returns the tokens as they are when they spell the served text, in OpenAI's shape, without changing the provider's list; drops `` reasoning, a fence and the whitespace `clean_answer` strips (trimming a token only by whitespace, bytes recomputed); returns only the answer's tokens when a server lists reasoning tokens its content leaves out; None when a token holds both the answer and a tag, when reasoning was removed from the middle, or when the tokens do not spell the answer (U+FFFD parts), and when the model sent none; `[]` for an empty answer | | `test_served_logprobs_check_every_entry_and_keep_the_clients_top_logprobs` |: `top_logprobs` cut to the client's count (none for 0 or unset), malformed alternatives left out; a malformed entry (no logprob, a string, a boolean, NaN, +Infinity, no token, an unpaired surrogate) makes the list None; -Infinity is -9999.0 (`MIN_LOGPROB`); the server's bytes are kept when they are bytes; the fakes' `token_entries` spell their text | | `test_majority_gives_n_agreeing_choices_with_the_best_sample_first` |: `confidence.majority` gives the best scored sample first, then the other agreeing samples by token probability, never the minority; None when fewer than n agree, a cut-off or invalid sample never fills a choice, and a repeated object counts twice | | `test_gen_plan_for_n_choices` |: `GenPolicy.plan(params, n)`: the serving teacher at 0.7 (`CHOICES_TEMPERATURE`) for n > 1 without a client temperature that reaches it (also when the allowlist leaves temperature out), the client's otherwise; audits at 0; the student's sampling unchanged; the teacher's logprobs only when asked, never in an audit; a bad n is a `ValueError` | | `test_certificate_frontier_and_per_task_from_held_out_points` |: 62 held-out points at threshold 0.85: 52 served, 2 errors, `lcb = cp_lower(50, 52, 0.05)` (0.884), certified at 0.85 and not at 0.9; serving nothing gives 0 and `ltt_trace` only for Learn-then-Test; the frontier's ten rows at delta / 10 with coverage never rising with the threshold; per-task counts and bounds; the outcome vector `S`/`W`/`E` | | `test_paired_counts_on_hand_built_vectors` |: ten hand-built rows give exact discordances, flips (an escalated row the candidate answers wrong is a negative flip), the served flip rate 2/5 with its Wilson interval and both McNemar p-values; 4 rows only live serves give 0.0625; bad vectors raise | | `test_recheck_recomputes_every_number_from_the_counts` |: a JSON round trip of a full eval rechecks clean (1e-12 of float noise is not a mismatch); an edited `lcb`, `errors` (also against the test vector and the per-task sums), target, frontier row, uncorrected frontier deltas, reordered rows, a paired p-value, impossible counts, a paired `n` or flip count the candidate's test vector rules out, a string count and a bad vector each give their line | | `test_recheck_follows_the_ltt_trace` |: Learn-then-Test's trace rechecks; a threshold below the lowest one it kept and a forged p-value are named | | `test_certificate_status_and_section` |: `section` of an eval with and without a certificate; the status follows the latest certificate, falls back to the signed one, is `no certificate` without one and `suspended` under an open drift alarm | | `test_versions_filter_by_state_in_order` |: `Ledger.versions(states)` selects in SQL in id order (a generator, an empty list), `version(vid)` reads one row, `live` and `by_state` the latest | | `test_replay_hajek_estimate_and_cost_on_a_hand_computed_example` |: six requests worked out by hand: Hajek precision 1/3 at the live threshold (weights 1 / `label_p`, an unaudited served answer has no label), 3/7 and 3/8 below it with the escalations' labels, Kish n_eff 1.8, the Wilson interval at it, cost 0.14 against a 0.158 baseline (0.126 at audit rate 0, which leaves precision alone), an escalation served at 0.35 audited at the version's mean logged rate, errors per 1,000, and `at(rows)` (each request at its logged threshold) equal to the replay at 0.6; finite JSON | | `test_replay_coverage_is_exact_and_the_frontier_never_gains_coverage` |: coverage at eight thresholds equals the count of requests at or above them, never counting a request with no valid answer or one that could not fill its n choices; the 71-row frontier never gains coverage and its rows equal `at` to the bit; `grid` counts in whole steps and refuses bad grids; a threshold of 0, above 1.01, NaN or a boolean is refused | | `test_replay_without_positivity_gives_no_precision_above_the_live_threshold` |: with audit rate 0 every threshold that serves a request gives a null precision with the positivity reason, while coverage and cost stay known; nothing served and no labels give their own reasons | | `test_replay_precision_is_unbiased_under_the_designed_labelling` |: 2,000 requests with known correctness, logged at threshold 0.6 (labelled with probability 1 below, 0.1 above), 120 seeded replicates of the labelling draw: the mean replayed precision at 0.4 and 0.8 is within 3 standard errors and 0.01 of the truth, while the unweighted mean of the labels is more than 4 points off | | `test_replay_precision_is_unbiased_under_the_designed_labelling_full_size` | The same with 500 replicates (`slow`) | | `test_replay_suggests_the_lowest_passing_grid_threshold_or_none` |: `suggest` scans `CERT_GRID` down from 0.95 and returns 0.80, the last threshold whose interval's lower bound reaches 0.97, with the advisory text; five rows at the top make it None although lower thresholds would pass (fixed sequence); `money_left_on_table` against 0.9 (nothing when every row was audited as logged, the savings and dollars at a 10% audit rate) and None when the current threshold is already lower; a target of 1 is refused | | `test_ratio_estimate_recovers_the_cost_ratio` |: `stats.ratio_estimate` recovers 1.3 within 3 standard errors on 400 noisy pairs, and `replay.ratio` from audited served answers weighted by 1 / `label_p` at mixed audit rates (escalations left out), with its interval; an unaudited served answer is priced at R times its estimate; `insufficient` (R 1.0) below 30 audited answers | | `test_a_client_logit_bias_never_steers_the_student_it_would_be_audited_against` |: `logit_bias` is not in the default student allowlist, so the student's body has none while the serving teacher still gets it; naming it in `RATCHET_STUDENT_PARAMS` restores it | ### `tests/test_tasks.py` Field comparators and task specs. Pure logic on `compare.FieldSpec` and `tasks.Task`, plus small ledgers and `RecordingProvider` for what the cascade, the golden eval and the dataset do with an answer the spec refuses. | Test | Covers | |---|---| | `test_exact_is_the_v04_rule` | `exact`: strings stripped and lower-cased; `1` and `1.0`, `true` and `1`, lists in another order differ | | `test_label_normalises_and_checks_its_vocabulary` | `label`: NFKC, case, trailing sentence punctuation and separators; a value outside `values` is invalid; `c#` is not `c` | | `test_label_aliases` | `P1` is `urgent`, `SEV-2` is `high`; an alias outside the vocabulary is refused | | `test_bool_reads_words_and_json` | `yes`/`no`/`y`/`n`/`1`/`0` and JSON booleans; `maybe` and `2` invalid | | `test_number_reads_money_separators_and_tolerances` | `1,234.50` == 1234.5 with `abs_tol` 0.01; `1.234,50 €` with `decimal=","`; `(5)` == -5; relative tolerance; percent as number or fraction | | `test_parse_number_cases` (20 cases) | Thousands groupings (also Indian, apostrophe, space), `US$`, ISO codes, exponents, signs; ambiguous `1,5`, lower-case codes, units, NaN, infinities, `1_000`, double signs refused | | `test_number_reads_indian_amounts` (29 cases) |: `Rs.`, `Rs`, `Re.` before or after (any case), `rupees`, `/-`, `INR` and `₹` with the Indian grouping, lakh, lakhs, lacs, crore and lakh crore scaled exactly (`1.15 lakh` is 115000), signs and `(...)`; `Rs` or `lakh` alone, `Cr`, `L`, `hrs`, `Rest 5`, a broken grouping and `1,5 lakh` refused; the comparator agrees with the plain number | | `test_indian_amounts_under_a_comma_decimal_and_with_tolerance` |: lakh and `Rs.` under `decimal=","` (the Indian grouping only with `.`); `abs_tol` 0.01 across `Rs.`, `INR`, `₹`, lakh and `/-`; lakh is not crore | | `test_number_refuses_non_finite_and_non_numbers` | Strings, booleans, lists, `1e999`, NaN and an integer too large for a float are invalid | | `test_ordinal_tolerance` | `low` and `medium` agree with tolerance 1, `low` and `high` do not; an unknown label is invalid; `order` is required | | `test_text_modes` | `loose` with suffixes (`Acme Corp.` == `ACME corp` == `Acme, Inc.` == `Acme,Inc.`, a two-token suffix), punctuation as a word break (`hello-world`), apostrophes and initials joined (`O'Brien`, `U.S.A.`), a suffix that is the whole name kept, `exact`, `token_set` | | `test_date_formats` | Six written forms == `2026-10-03`; month first by default, `day_first`; the local day of a datetime; UTC datetimes; impossible dates invalid | | `test_language_codes_and_names` | `English` == `en` == `eng` == `en-US`, `Deutsch`, `日本語`, locale tags with scripts, regions and variants; unknown languages invalid; `values` restricts; prose that starts with a code (`It is English`, `no idea`, `in progress`) and tags led by a 639-2 or retired code are invalid, so `It is English.` and `It is French.` do not agree | | `test_set_order_case_and_jaccard` | Order, case and duplicates ignored; Jaccard 0.8 boundary; `empty_values`; `split` with number elements; a string without `split` is invalid | | `test_records_unordered_keyed_and_ordered` | Unkeyed and keyed pairing across order, `ordered=True`, a changed field or length, duplicate keys and a missing sub-field invalid | | `test_records_without_a_key_pair_in_any_order` | Records with a tolerance pair by a maximum matching: the listed order never decides (`[0, 1]` and `[1, 0]` both agree with `[0.5, 2]`), a 50-record chain with one pairing, labels and nested records grouping first | | `test_ranking_top_k` | Top-3 in order and as a set; an id ranked twice is invalid | | `test_root_label_text_format` | A text task's `$` label: `Positive.` == `positive`, reasoning removed; other paths refused | | `test_top_level_array_with_dollar_path` | A `$` set over a top-level array (fenced, after a sentence, after reasoning), a `$` number, dot paths into nested objects | | `test_empty_json_is_invalid_when_fields_required` | `{}` is invalid for `ticket-triage` (score 0), but still valid for a `compare_keys` task, as in v0.4 | | `test_out_of_vocab_label_invalid` |: `check` reasons (`invalid field category`, `invalid field priority`, `invalid answer`, `empty answer`); every answer the mock world writes is valid | | `test_nullable_field_accepts_null` | Required, nullable and optional fields and how null and missing agree | | `test_bad_specs_fail_when_defined` (37 cases) | Reserved and unknown types, the `sql` format without its options, a similarity field's `metric`, `threshold` and `max_len_ratio` out of range, similarity options on another type, similarity inside records or as a set element, bad paths, options on the wrong type (`values`, `tolerance`, `abs_tol`, `mode`, `k`, `min_jaccard`, `day_first`), duplicate labels, an alias that is a label of the vocabulary or names two labels, tolerances on set elements, record keys, the value, field and format limits, `compare_keys` with `.` or `$`, policy keys and ranges, `tools` | | `test_records_nest_at_most_four_deep_and_hold_at_most_200` | Depth 4 allowed and 5 refused; 200 records valid and 201 invalid; the 200 count across nesting | | `test_spec_hash_covers_the_rules_not_the_name` | Same rules, same hash, whatever the name or source; a vocabulary or `pinned` changes it; `dataclasses.replace` recomputes it, also on a `compare_keys` task; `policy` is read-only | | `test_legacy_compare_keys_canonical_byte_identical` |: a `compare_keys` task's `canonical` and `agree` equal v0.4's on 25 answers with fences, ``, nulls, `1` vs `1.0`, nested values, NaN and deep nesting | | `test_pairwise_score_equals_counter_score_for_exact_tasks` |: on 600 seeded random sample sets (invalid and cut-off samples, ties, missing logprobs), the pairwise score equals v0.4's `Counter` score: the same answer object, confidence, agreement and token probability; `majority`'s choices agree | | `test_builtin_task_agreement_decisions_unchanged` | On mock answers for the three built-in tasks, the typed specs decide agreement, confidence and the best sample exactly as v0.4's exact keys | | `test_tolerance_non_transitive` | 0, 0.6, 1.2 with `abs_tol` 1: the middle sample, agreement 1.0, even at a lower token probability; `majority`; two groups tie and the first wins; with four samples the served sample's agreement is its own | | `test_score_never_raises_when_a_key_does_not_match_itself` |: a comparator that is not reflexive gives confidence 0 instead of raising on the request path | | `test_default_compares_json_canonically` | Key order, spacing (inside strings too, as in v0.4), case, NFKC and `1` vs `1.0` agree when both are JSON; list order and `true` vs `1` do not; the text rule otherwise; deep nesting compared as text; a `json` task without fields | | `test_comparators_are_linear_on_adversarial_answers` | Long labels, suffix runs, token sets, numbers, dates, languages and sets normalise at once; 200 records pair reversed, shifted within a tolerance, and with ten tolerant fields in the worst order, all under 2 s | | `test_comparators_never_raise_on_junk` |: 3,000 seeded junk values (odd strings, NaN, huge integers, nested lists and objects) never make a comparator raise, and every valid value matches itself | | `test_invalid_teacher_answer_served_but_not_stored` |: an out-of-vocabulary or incomplete teacher answer is served, not stored, `skip_example` names the field; an invalid student answer is never served; `store=False` stores nothing without a skip | | `test_invalid_audit_not_counted` |: an audit whose teacher answer the spec refuses is `audited=0`, not stored | | `test_cascade_eval_skips_invalid_golden_refs` |: rows with invalid references are not sampled and are counted in `golden_invalid_refs`; the halves alternate over the scored rows | | `test_dataset_rows_skip_invalid` |: train rows stored under a looser spec are left out and counted (`skipped_invalid`, also in `manifest.json` and in the night's `trained` or `skip` event); golden rows stay | | `test_chrf_identical_one_unrelated_low` |: chrF is 1 for equal texts (case and spacing aside) and below 0.25 for an unrelated one; `ab`/`ac` is 1/4 and `abc`/`ab` 14/33 by hand; on 300 seeded random strings it equals chrF written out from its definition and is symmetric; an unknown metric raises | | `test_token_f1_and_rouge_l` |: SQuAD token F1 (6/7 by hand; articles and punctuation dropped; nothing left is 0); ROUGE-L 10/12 by hand, order matters (2/8 where token F1 is 1); the bit-parallel LCS equals the textbook table on 300 seeded pairs, both ways; tokens after the 512th never change ROUGE-L | | `test_similarity_score_picks_medoid` |: a chain A~B~C with A far from C serves B (three votes) at the lowest token probability; between two samples with as many votes, the more probable; `majority` gives the medoid and its matches, never the outlier | | `test_len_ratio_blocks_match` |: a score at the threshold does not match when one answer is over `max_len_ratio` times longer; a larger ratio lets it; equal texts always match; a text task drops a trailing `.` | | `test_mixed_label_and_similarity_fields` |: `{category: label, summary: similarity}` agrees only when the label matches and the summaries are close; a missing, blank or non-string summary and an out-of-vocabulary label are invalid; `canonical` keeps the normalised text; the outlier gets one vote; `precision_kind`, `free_text` and `verifiable` for similarity, label and default tasks | | `test_similarity_task_requires_two_samples` |: a similarity task without `policy.samples` of at least 2 is a `ValueError` and a `SpecError` (in a document, a single spec and through `extends` dropping it), a bad value reported once by its own rule; `sample_count`; a similarity task round-trips through `document` | | `test_router_and_golden_eval_use_policy_samples_per_task` |: a task's `policy.samples` (5) is what the router draws for the live student and the shadow, at least the client's `n`; a task without it keeps `RATCHET_SAMPLES` (3); `cascade_eval` samples each row as its task; `precision_kind` per task and globally | | `test_similarity_is_bounded_on_long_answers` |: a 300,000-character answer normalises and compares under every metric in well under 2 s; features are built once per answer however many samples compare it, and never for a validity check | | `test_stats_report_proxy_precision_and_judge_pass_rate` |: `stats_by_task` gives `precision_kind` and `judge_pass_rate` from the live version's records; a hand-edited record gives None, never a wrong label | ### `tests/test_registry.py` The task registry: specs from `RATCHET_TASKS_FILE` and the admin API, the admin key, operator-only pinning, the unknown-task policy, client allow-lists, pinned rows left out of Night School, and spec changes. Gateway tests use `gw_env` (mock) or a live `Runtime` on a file database with the teacher behind `FakeOpenAIServer`; unit tests use `taskspec`, `tasks` and a small ledger. The registry is module state, so `tests/conftest.py` resets it after every test. | Test | Covers | |---|---| | `test_tasks_file_loads_and_registers` |: a file's tasks (one extending and adjusting the built-in `default`, one pinned) listed by `GET /ratchet/tasks` with their source and spec; a request distilled under the file task, the pinned one answered by the teacher and not stored | | `test_tasks_file_in_yaml` | The same in YAML (skipped without PyYAML); `clients` checked against the API key names; a syntax error with its line; bare `yes`/`no` read as booleans refused | | `test_tasks_file_unknown_key_fails_startup_with_path` | A bad file stops `Runtime` and `python -m ratchet.live` with every error and its path (`tasks[0].feilds`); a missing file, broken JSON and NaN refused | | `test_yaml_without_pyyaml_message` | A `.yaml` file without PyYAML is an error that says so; JSON needs nothing extra | | `test_spec_errors_list_every_problem_with_its_path` | One document with 36 mistakes (unknown keys and options, options for another type, bad paths, ranges, a reserved format (`tool_calls`), a similarity field's `metric` and `threshold`, policy, match, routing, clients, names, duplicates, `extends` cycles and missing bases, records nested too deep) gives exactly 36 errors, in file order | | `test_spec_parser_never_raises_anything_but_spec_error` | 3,000 seeded junk specs (wrong types, unhashable values, unknown types, nesting) in `parse_spec` and `parse_document` give `SpecError` or a task, never another exception | | `test_specs_round_trip_and_every_option_is_checked` | Every `FieldSpec` option has a check; a spec with every type, and every built-in task, survives `document` and `parse_spec` with the same hash; `description`, `clients` and `match` leave the hash alone, field order does not; the ledger keeps the order | | `test_extends_copies_and_merges` | `extends` within a file (forward too), from a built-in task, a file adjusting a built-in task of the same name, a copy of that, through the API on the task's own name; `fields`, `policy` and `match` merge, null removes, `name` and `description` are not inherited | | `test_registry_precedence_and_sources` | A file task replaces a built-in one; the API never replaces a built-in or file task; an auto task never shadows a registered one; adhoc and built-in tasks cannot be registered; removal and restore; `names_for` and allow-lists; an adhoc task is never pinned | | `test_registry_reads_never_see_a_half_made_registry` | Lock-free reads under 4 writer threads registering, removing and resetting | | `test_registry_cap` | `MAX_TASKS` refuses a new name but not a replacement; a file with too many tasks; 409 for the mock cap and the registry cap | | `test_put_task_requires_admin_key` |: 401 without or with a wrong admin key, 403 with an API key, for PUT, DELETE and validate; reads stay open; only the admin's name reaches the ledger and the events (the database dump holds no key) | | `test_admin_keys_are_their_own` |: a key in both `RATCHET_ADMIN_KEYS` and `RATCHET_API_KEYS` stops startup; errors name the right variable | | `test_admin_disabled_in_live_without_keys` | Live mode without admin keys: 403 `admin API disabled` for every write; mock mode: open, as `mock-admin` | | `test_put_builtin_task_forbidden` |: 403 for a built-in or file task (PUT and DELETE), 404 for an unknown one; `validate` says the same | | `test_client_cannot_pin_via_header_metadata_model_or_schema` |: pin-like headers, metadata, `model` and a `response_format` with `x-ratchet` never pin; the tasks are distilled; nor can a header unpin `research-write`; `Task` refuses a pinned adhoc or auto task | | `test_task_lifecycle_through_the_admin_api` | Create, an identical PUT (`changed: false`, no event), traffic, a changed spec (`task_changed` with both hashes), a metadata-only change, `recalibrating`, every 422 (bad JSON, NaN, an unpaired surrogate, another name, not an object, a bad option, 65 fields, a bad URL name), delete (examples kept), the name then following the unknown-task policy and listed as unregistered | | `test_api_task_persists_across_runtime_restart` | A live `Runtime` on a file database: an API task registered again after a restart with the same hash and `updated_by`; a change logs both hashes; a deleted task stays gone, its examples kept | | `test_stored_task_that_no_longer_parses_is_skipped` | A stored task that no longer parses (a similarity field without `policy.samples`), one whose name the file now has, and a garbled row are skipped with `task_skipped` events; the gateway starts | | `test_validate_endpoint_has_no_side_effects` | One spec and a document, valid and not (a similarity task without `policy.samples` is a spec error); a document may adjust a built-in task, a single spec may not; the registry, the events and the tasks table are unchanged | | `test_concurrent_admin_writes_keep_the_registry_and_the_table_in_step` | 8 threads creating, changing and deleting tasks: the registry, the table and the event counts agree | | `test_unknown_tasks_teacher_policy_stores_nothing` |: `teacher` answers an unregistered task with the teacher only (route `unknown_task`, no student call even with a live and a shadow version, nothing stored); registered tasks still distil; a typo stops startup; the same through the gateway | | `test_unknown_free_text_not_stored_under_teacher_policy` |: unregistered free text under `RATCHET_UNKNOWN_TASKS=teacher` is answered by the teacher (route `unknown_task`) and never stored; under `distill` it is stored; a pinned long-form task is never stored, a registered short-summary similarity task is | | `test_unknown_tasks_reject_policy_400` |: `reject` is a 400 `unknown_task` listing the registered tasks (`param` for metadata), logs nothing and uses no quota; direct callers get the same; at most 20 names listed | | `test_clients_allow_list_403` |: a `clients` name that is not an API key is a 422; another key gets 403 `task_not_allowed` before the quota; the named key is served; rejection messages list only tasks the key may use | | `test_pinning_task_excludes_existing_examples_from_dataset_and_eval` |: pinning a task with data leaves its rows out of `dataset.rows`, the manifest (`excluded_pinned`), `cascade_eval` and `run_night`'s trainer and evaluator; the eval records `spec_hashes`; unpinning brings them back | | `test_spec_change_forces_teacher_until_recalibrated` |: a changed spec makes the threshold 1.01 for that task only; the next night (even without enough data to train) re-scores the live version and records the new hash (`rescored`), and the student serves again; nothing to re-score the night after; a failed re-score keeps the task on the teacher (`rescore_failed`) | | `test_fixture_paths_resolve_inside_the_fixtures_dir` | `resolve_fixture` refuses `..`, absolute paths, empty and NUL paths and a symlink out of `RATCHET_FIXTURES_DIR` | ### `tests/test_nightschool.py` The invariant tests for Night School. Each runs `run_night` or `cascade_eval` on a small ledger with a `RecordingTrainer` and a `RecordingEvaluator`, so it can assert on exactly which rows reached the trainer and which golden rows were scored. The last six check that the student trains on the prompt it is served: the training file's rows, the vendored Nemotron 3 Nano chat template rendered offline with jinja2 (`tests/fixtures/nano4b_chat_template.jinja`), `train.py`'s checks and loss setup, and the pinned training image. | Test | Covers | |---|---| | `test_training_targets_are_teacher_answers_only` |: the teacher answers JSON that is valid for both tasks used ( would skip anything else); a student that answers `STUDENT-ONLY` with high confidence serves 200 requests with `audit_rate=1` and a shadow present, then 200 more without audits where a disagreeing shadow (`SHADOW-ONLY`) makes the teacher arbitrate and unparseable answers escalate. All 400 stored examples and dataset rows hold exactly the teacher's answer to their own request, and neither marker appears anywhere | | `test_add_example_called_only_from_allowed_callers` |: an AST scan of `ratchet/` finds every use of `add_example` (a call, a bound method passed on, or `getattr(..., "add_example")`); each must sit in a function in `ALLOWED_ADD_EXAMPLE_CALLERS` (today only `Ratchet._store_example`), and every listed caller must still use it (so moving a call site replaces its entry rather than adding one beside it) | | `test_add_example_guard_sees_indirect_uses` | The scanner itself (no requirement marker: it tests the tool, not), on a throwaway package: a direct call, `getattr` inside a nested function and a bound method in a module-level lambda are found and named; a docstring that mentions `add_example` is not | | `test_golden_rows_never_reach_the_trainer_except_frequent` |: the trainer gets the train split plus the golden requests seen at least `FREQUENT` (3) times; the evaluator gets the other golden rows; the two never overlap; a golden request seen twice stays held out; the stored split never changes | | `test_every_variant_of_a_prompt_shares_its_split_and_count` |: 200 prompts, each stored bare and under three sets of conditions: every prompt's four examples share one split (both splits occur), and the bare example's split is the one it gets alone (v0.4's); a request seen once bare and twice in JSON mode is frequent, so both variants move to training together | | `test_examples_cut_at_a_stop_are_held_out_of_training` |: `train.py` ignores stored conditions, so a row the client's stop cut is left out of the train rows (bare and JSON-mode rows stay); as a golden row it is kept, with its stop | | `test_closed_teacher_rows_are_dropped_at_dataset_build` |: rows from `openai/gpt-4o` and an unregistered teacher in the ledger never reach `dataset.rows`, `train.jsonl`, `golden.jsonl`, the trainer, the evaluator or the adapter's BOM | | `test_night_skips_with_reason` (5 cases) |: 49 new examples, 40 new since the last version, no golden set, a shadow in flight and a canary in flight each skip with their reason and a `skip` event; the trainer and evaluator are never called (not even to re-score the champion) and no version changes | | `test_retrain_from_base_on_cumulative_data` |: three nights (the first with exactly 50 new examples). Each candidate passes and is released to live through the gate, so nights 2 and 3 train while the previous night's adapter is the champion; every night still trains from the base model (each `TrainCall.base_model` and `Version.base_model`), never the live adapter. Each night's rows contain every earlier night's rows, and their counts (50, 100, 175) equal each version's `train_examples` | | `test_cascade_eval_measures_on_held_out_half` |: with right answers on even golden rows and wrong ones on odd rows at the same confidence, precision is 0.0 (graded on the odd half only), `test_n` is half of `golden_n` and all report keys are present; the mirror image calibrates on wrong answers and serves nothing (threshold 1.01) | | `test_cascade_eval_passes_row_params_to_sample` |: the sampler gets each golden row's stored conditions, `None` for a row without them | | `test_cascade_eval_scores_a_refused_row_as_not_served` |: three of six golden rows the sampler refuses with a 400 score as confidence 0 and wrong (`refused_n` 3, agreement 0.5, coverage 2/3 of the odd rows); a 401, 404, 429 or 503 still raises | | `test_golden_eval_runs_under_production_conditions` |: an example stored with a stop and JSON mode comes back from `dataset.rows` with them; the router (with `RATCHET_STUDENT_MAX_TOKENS=256`), `LiveEvaluator` and `MockFleet` send the student exactly the same arguments for those rows | | `test_champion_rescored_on_same_golden_set` |: the live version is evaluated first, on the same golden list as the candidate; its stored eval now holds the re-checked precision and coverage, with `rechecked_day` = the ledger's day; the candidate, which would regress against the stale eval, is admitted to shadow against the fresh one; the candidate is trained from the base model, not from the champion | | `test_cascade_eval_certifies_on_the_held_out_half` |: 80 rows of two tasks with seeded confidences: the threshold equals the pre-change point rule exactly, the certificate's counts are the odd rows' at it (its precision the eval's own), `lcb ≤ precision`, the test vector has `test_n` letters, the frontier's coverage never rises with the threshold, per-task counts add up, `recheck` is clean; another target and delta change only the certificate | | `test_cascade_eval_ltt_picks_a_grid_threshold` |: 120 right calibration rows certify every grid threshold (served at 0.5, `lcb` 0.05^(1/120)); 10 rows certify nothing (1.01, one trace step); confident right rows and unsure wrong ones stop at a grid threshold above the unsure ones | | `test_cascade_eval_refuses_bad_certification_settings` | An unknown calibration, a target of 1 and a delta of 0 or `True` raise | | `test_run_night_pairs_the_candidate_with_the_rescored_live_version` |: the candidate's eval and its signed BOM carry `paired` against the live version (the champion's re-score carries none); vectors of different lengths, a missing vector and empty vectors give none, and so does a cold start | | `test_paired_gate_admits_what_the_unpaired_rule_rejects_on_noise` |: the reference run's v2 (70 of 74 rows against live's 74, simulation): the unpaired rule rejects it, the paired gate admits it with `serves -4 of 74 vs live v1 (McNemar p=0.06)` | | `test_training_rows_carry_reasoning_off` |: every row of `train.jsonl` (from `dataset.build` and `dataset.write_training`, which the live trainer uses) has `prompt` (the stored messages), `completion` (the teacher's stored answer as the one assistant message), `chat_template_kwargs == {"enable_thinking": false}` and no `messages`, keeps `params` and `provenance`, and no row contains `` or ``; `train.check_rows` accepts them; `dataset.TRAIN_TEMPLATE_KWARGS` (read-only), `train.TEMPLATE_KWARGS` and the endpoint's `DEFAULT_TEMPLATE_KWARGS` are equal; answers with a literal tag are left out of training and counted (`skipped_think_tag`), stay golden rows, and the night's event says how many | | `test_training_prompt_is_a_prefix_and_matches_serving` | (skipped without jinja2, which the `dev` extra installs): three requests (one turn, a system prompt, a conversation with an assistant turn) escalate through the router and are stored; the vendored Nano 4B template (sha256 checked), compiled as transformers compiles it, renders each training row's prompt ending in `<\|im_start\|>assistant\n`, prompt + answer as exactly that prompt + the answer + `<\|im_end\|>\n`, and the prompt equal to the one rendered for the router's student call with the endpoint's default arguments. Without the argument the prompt ends in `\n` and is not a prefix (the bug), and `train.check_prefix` raises `TemplateMismatch` naming the row; so does an answer with a think tag. Messages filled with None keys (as `datasets` hands them to TRL) render the same prompt; a written prompt that differs, or a different row count, raises | | `test_train_args_have_no_assistant_only_loss` |: an AST scan of `train.py` finds one `SFTConfig` with `completion_only_loss=True` and no `assistant_only_loss` (or `False`), `main` calling `check_rows`, `check_prefix`, `columns` and `SFTTrainer`, and no Unsloth import; `train.columns` gives TRL only `prompt`, `completion` and `chat_template_kwargs` | | `test_dockerfile_is_pinned` |: `deploy/Dockerfile.train` has no `\|\| true` and `trl==1.16.0`; every package of every `pip install` carries `==`, and together they equal `deploy/train-versions.txt` (no Unsloth); every step after torch reads that file as constraints; the kernel import check comes after the Mamba install; `python-is-python3` is installed; `train.LIBRARIES` names exactly the pinned packages; the `train` extra in `pyproject.toml` agrees with the list | | `test_train_records_library_versions` |: `train.versions` records Python, CUDA and each pinned package (None when missing, from an injected lookup) in `LIBRARIES` order; `bom.EXTRA_KEYS` has no section for them, so `versions.json` stays beside the adapter; `ratchet_manifest.json` records the template arguments | | `test_train_refuses_rows_that_would_train_another_prompt` |: `train.check_rows` refuses, naming the row and never its text, an old `messages` row, missing or other template arguments, a completion that is not one assistant message with text, an answer with a think tag, an empty or malformed prompt, provenance without a license, a row that is not an object, and an empty file | | `test_the_version_is_recorded_on_the_rows_the_adapter_was_trained_on` |: an example stored while the dataset files are written is not trained on: rows, `train_examples`, the BOM's count, commitment size and `dataset_sha256` all describe the same 60 rows; the late one is in the next read | | `test_a_prompt_held_out_in_one_task_is_never_trained_on_through_another` |: the same messages under two tasks: golden under one, train under the other is golden under both; train under both stays train; golden under both stays golden; two occurrences in one task and one in the other make it frequent, so it trains in both; a prompt only one task has trains as before; `rows(task=)` returns that task's copy | ### `tests/test_release.py` | Test | Covers | |---|---| | `test_gate_rejects_low_coverage` | Golden coverage one point under `min_coverage` is rejected; an eval without coverage is rejected (fails closed); exactly `min_coverage` passes | | `test_gate_tolerance_boundary` (2 cases) | For precision and for coverage: a candidate exactly `tolerance` worse than live is admitted to shadow; a hair more is rejected as a regression | | `test_admit_retires_older_inflight` | A rejected candidate leaves the shadow and canary in flight alone; an admitted one retires both (traffic 0, one `retired` event each) and is the only version in flight | | `test_shadow_promotes_and_both_rejections` | 99 would-serve answers hold; at 100, precision exactly 95% and live's coverage it moves to canary 10%. The count is would-serve answers, not shadowed requests: 150 requests with 75 would-serve answers hold, 200 with 100 promote. 90% precision is rejected; 50% coverage against a live student at 100% is rejected, but passes against a live student that also serves 50% | | `test_shadow_coverage_is_not_compared_with_requests_the_live_student_never_answered` | (regression): rows where the live student never answered (route `student_breaker` or `student_error`) stay in the shadow's `n` and coverage but not in `live_coverage`, so a shadow serving 50% is rejected against a live student that served all it answered; the same pattern with the live student unsure (route `confidence`) is live coverage of 50% and promotes | | `test_shadow_coverage_counts_an_n_choices_escalation_as_live_served` |: a request the live student was confident about but escalated because fewer than n samples agreed (route `n_choices`) counts as served in `live_coverage`, so a shadow serving 50% is rejected against a live student that would have served them all | | `test_shadow_gate_counts_unarbitrated_disagreements_against_the_candidate` |: 100 would-serve answers whose arbitrated ones all agree, but 10 of which the teacher could not arbitrate (`shadow_agree` NULL), give precision 90%, not 100%, and are rejected with "(10 unarbitrated counted as disagreements)"; 2 of 100 give 98% and promote, and the event says so; 50 rows without a shadow verdict (`shadow_serve` NULL) change neither `n`, coverage, the evidence count nor precision | | `test_canary_holds_without_strong_evidence` | 18 of 20 audits (90%, under the 92% floor, Wilson upper bound above it) neither roll back nor advance, over several ticks; 20 more agreeing audits move it to 50%. The rollback boundary: 16 of 20 (Wilson upper bound 91.9%, just under the floor) rolls back to live, 17 of 20 (94.8%) only holds | | `test_rollback_without_live_returns_traffic_to_teacher` | A cold-start canary at 50% audited precision is rolled back with "traffic back to the teacher"; afterwards 50 requests are all answered by the teacher and the student is never called | | `test_shadow_arbitration_calls_teacher_once_and_stores_example` | A shadow that would serve a different answer than the live student triggers exactly one teacher call, whose answer is stored as the example, while the user keeps the live answer; agreeing or unsure shadows trigger none; with an audit, the audit is the reference and no second call is made; a shadow whose endpoint is down (`raise_on`) leaves the request untouched, with no shadow record and no extra cost. Reported cost includes the shadow's samples, the arbitration and the audit | | `test_canary_traffic_split` | `Ratchet._pick_version` with a seeded RNG: over 4,000 picks at 10% and at 50%, the canary's share is inside the 99% Wilson interval; 0% never picks it, 100% always does, and at cold start the rest of the traffic gets `None` (the teacher) | | `test_tick_runs_each_step_once_in_order_with_the_live_version_it_started_with` | `tick` calls `_tick_drift`, `_tick_live`, `_tick_shadow`, `_tick_canary` and `_tick_watchers` once each, in that order, the middle three with the live version read before any step (a canary promoted in the tick retires that one); with no confidence logged the drift step writes nothing, and the watcher step is not built | | `test_policy_refuses_values_out_of_range` (8 cases) | `ReleasePolicy` refuses a `live_floor` of 1 or 0, a precision above 1, a negative coverage or audit count, a fractional or boolean count and an infinite tolerance, naming the variable | | `test_sequential_canary_rolls_back_after_two_failures` |, US-6: with `sequential=1` and `min_audits=15`, no audit is no decision, 0 of 1 agreeing (e-value 4.22) holds, 0 of 2 (29.8 >= 20) rolls back to live with the exact event text (counts, e-value, threshold, alpha, `95% CS [0.001, 0.931]`); the Wilson rule holds on the same two audits | | `test_sequential_rollback_at_cold_start_returns_traffic_to_the_teacher` |: a cold-start canary rolled back by the e-process sends traffic to the teacher | | `test_sequential_rollback_counts_every_audit_of_the_canarys_life` |: rollback is one e-process over the canary's whole life: 40 agreeing audits at 10% still count at 50%, so 2 failures (which alone would reach 29.8) hold and it takes 11 (40/51, e-value 23.2) | | `test_sequential_promotion_needs_evidence` |: with `promote_evidence=1`, 20 agreeing audits (2.57) hold and 60 (27.6) advance to 50% with the old event text as a prefix; the count restarts per stage, 55 agreeing audits (19.996) hold and 56 (21.3) promote to live; the point rule promotes on 20 | | `test_futility_retires_inconclusive_canary` (2 cases) |: 45/50 under the point rule and 55/60 under evidence-based promotion: one audit short of `max_stage_audits` holds, at the cap the canary is `retired` with an `inconclusive` event (counts and confidence sequence), live untouched, no rollback | | `test_futility_counts_audits_not_time` |: a canary that serves 500 answers without an audit holds | | `test_sequential_policy_refuses_settings_that_decide_nothing` |: `promote_evidence` without `sequential`, a cap below `min_audits`, a cap of 50 for evidence-based promotion ("needs 56 audits that all agree") and the default cap at a 0.99 floor ("needs 450") are refused; 56, no cap, a cap equal to `min_audits` and a cap without the sequential rule are fine; alphas 0 and 1 and switches of 2 are refused, naming the variable | | `test_live_alert_uses_the_e_process` |: the default alert counts audits since the version went live (5 failures as a canary do not count), holds at 0/2 (29.8) and fires once at 0/3 (268 >= 100) with the exact text; a version above the floor is never alerted on over 400 audits; `alert_eprocess=0` keeps the Wilson rule (nothing before 20 audits) | | `test_evidence_is_recomputed_from_the_counts` |: `ReleaseGate.evidence` equals the `stats` functions of the life and stage counts, key by key; a live version's counts; finite JSON for 3,000 failures (e-value capped at e^690, a vanishing one still a number); no audits give 1 and `[0, 1]` | | `test_canary_audit_rate_focuses_audits_on_the_canary` |: with `RATCHET_CANARY_AUDIT_RATE=1` and an audit rate of 0, every canary answer is audited and no live one, `label_p` logs 1.0 and 0.0; exactly two random draws per request with and without it (a counting RNG), and the same requests reach the canary | | `test_label_p_is_the_designed_labelling_probability` |: `label_p` is NULL at cold start, 1.0 for an escalation the student answered, and the audit rate (0.25) for a served answer | | `test_stats_versions_carry_the_gates_evidence` |: `/ratchet/stats` gives the canary and the live version `evidence` equal to `ReleaseGate.evidence`, null for a retired one; the mock gateway reads `RATCHET_SEQUENTIAL` and `RATCHET_MIN_AUDITS` (15 when unset) | | `test_min_certified_precision_rejects_a_weak_certificate` |: 46/48 (bound 87.5%) is rejected at a 90% floor with the exact reason and admitted at 85% or off; a stored `lcb` of 0.99 is ignored (recomputed), no or a broken certificate fails the floor, and the precision floor still comes first | | `test_certified_versions_say_so_in_their_gate_events` |: the cold-start canary event and the shadow event append `; certified >= 95% (lcb 96.4%, delta 0.05)` (or 96.5% at another target and delta) for a certified candidate, and nothing for an uncertified one or a claim its counts do not back | | `test_paired_gate_rejects_only_large_significant_regressions` |: 2 of 63 rows within noise (p 0.42) are rejected by the unpaired rule and admitted by the paired gate with its event; a record against another version or with unreadable counts falls back; 10 rows lost (p 0.001) and 8 rows newly wrong of 100 (p 0.004) reject with exact reasons, the same 8 of 1,000 do not; 6 negative flips of 60 reject at `max_nfr` 0.02 (lower bound 4.7%) and pass at 0.05 or off | | `test_gate_rejects_below_min_judge_pass` |: off by default; with `min_judge_pass` 0.8 an overall judge pass rate of 70% on 20 answers is rejected (named in the event), 80% passes, a rate over no answers passes; the precision gate runs first; a free-text task the candidate would answer that no judge graded is refused (by `spec_hashes` or by records), labels and a task it serves nothing of need no judge; `RATCHET_MIN_JUDGE_PASS` is read and range-checked | | `test_judge_gate_disables_task` |: two free-text tasks and a label task; overall 90% clears 85% but one task's 60% does not: it is disabled on the candidate (threshold 1.01, `task_disabled`, the router holds it at 1.01) and the version ships, or `task_mode=reject` rejects it; a task under `min_task_test_n` held-out rows is left to the overall rate; an unjudged task with enough rows fails | | `test_mock_judge_catches_what_the_proxy_misses` |: simulation (mock models). An adapter that learned nothing shows 100% proxy precision under chrF on a free-text task, and only the mock judge, which knows the stored truth, sees most answers are wrong: rejected with `RATCHET_MIN_JUDGE_PASS` 0.9, admitted without it; a trained adapter passes; the judge rate is never above the proxy precision | | `test_judge_never_on_the_request_path` |: no module a chat request runs through (router, confidence, compare, similarity, tasks, calibration, providers) names a judge in its code or imports the golden eval, so a similarity task costs no judge call per request | ### `tests/test_system.py` | Test | Covers | |---|---| | `test_loop_learns_and_saves_money` | 6 days × 1,500 requests with no drift: day 1 is 0% student; final day is above 80% student; a version reaches `live`; over the last three days the run meets success metric 1 (savings ≥ 70%, also on the final day) (pooled audited precision ≥ 95%) and success metric 3 (student share ≥ 80%); no day's audited agreement falls below 85% | | `test_bad_deploy_is_rolled_back_and_users_return_to_live` | Fault-injects `student-v3`; v3 ends `rolled_back` and v1 stays `live` (skips if v3 was rejected before deploy for that seed). Shares the module fixture `bad_deploy` (the run under each canary rule) with the next test | | `test_sequential_gate_reduces_bad_deploy_exposure` |: the same corrupted v3 under the anytime-valid rule is rolled back too, after serving no more answers than under Wilson; each run's `exposure`, `rollbacks` and `served` agree; the rollback event carries an e-value and a confidence sequence (simulation, mock models) | | `test_every_version_proves_its_golden_rows_were_never_trained_on` |: over the six-day corrupted-deploy run (the module fixture `bad_deploy_runs`, which `bad_deploy` reads), every version's committed leaves rebuild the training and golden roots its BOM signs, the sets are disjoint (brute force), and a held-out certificate built from the leaves verifies with `proof.verify_heldout` from the signed roots alone (simulation, mock models) | | `test_replay_reproduces_each_versions_logged_coverage_and_audits` |: over the same run, for every version, the replay at each request's logged threshold serves exactly the requests the student served, each group of requests logged at one threshold reproduces its coverage at that threshold, and the replayed precision equals the audited precision the gate counted (`audit_counts`), inside its interval; nothing is left out (simulation, mock models) | | `test_label_p_and_student_agree_follow_the_design` |: on every row of the run, `label_p` is 0.10 on served answers and 1.0 on escalations; `student_agree` equals `audit_agree` on audited rows, is set on every escalation with a valid student answer, and is never set on a served row that only a shadow arbitration took to the teacher (some exist); `baseline_est_usd` is always set | | `test_drift_raises_escalations_then_recovers` | 9 days with a product launch on day 5: escalations rise on day 5 and are lower by day 9 | | `test_reference_simulation_unchanged` | The pinned reference run equals `tests/fixtures/sim_reference_seed7.json` exactly: the pinned inputs (a flag added later is not compared), daily numbers, release events and final versions ([below](#the-pinned-reference-simulation)) | | `test_reference_simulation_meets_success_metrics` | success metrics 1 to 4, US-9: on the reference run, `ratchet/kpis.py` over days 8 to 10 gives savings ≥ 70%, pooled audited precision ≥ 95% on at least the minimum audits (inside its Wilson interval), student share ≥ 80%, no bad deploy at 100% (the corrupted v3 was rolled back as a canary), the latency target met, and a drift episode that opens on day 5 and has closed; these back the `Met in simulation (seed 7)` statuses | | `test_sm5_drift_recovery_within_three_nights` | success metric 5, strict xfail: drift recovery within 3 nights; it takes 4 in the reference run (day 5 to day 9), so its status is `Not met`. It starts passing, and so fails the suite, once drift handling gets it within target | | `test_reference_workloads_reproduce_v040` | the tickets workload of `tests/fixtures/reference.json` (captured from v0.4.0) matches the session's reference run under `scripts/reference_check.py`'s rules, and every input the fixture pins for all three workloads is still the simulator's default | | `test_reference_check_runs_with_per_task_calibration_off` | `reference_check.capture` runs each workload with `RATCHET_PER_TASK_CALIBRATION=0` (`LEGACY_ENV`), whatever the caller's shell says, and puts the caller's `RATCHET_*` variables back afterwards | | `test_reference_check_rules` | `reference_check.compare` ignores float noise below a relative 1e-9, new day and totals keys, advisory event kinds and text appended to an event, and reports one line for a moved number, a removed key or day, reworded event text, an extra event, a changed version or a changed input default | | `test_provenance_only_lists_distillable_teachers` | Every BOM source is Ultra with OpenMDW-1.1 | | `test_gateway_openai_shape_and_demo_flow` | Mock gateway: `/` (landing page) returns 200; `/ratchet/demo/day` works; `/v1/chat/completions` returns a valid `chat.completion` with the right category; stats count 401 requests; break-next and reset work | | `test_replay_endpoints_on_the_mock_gateway` |: on the mock gateway, `/ratchet/replay` is a 400 without a live version; after two simulated days the canary's replay names its state, `simulation (mock models)` and `per_token`, has `actual` next to the counterfactual at the current threshold (and at 0.4 with audit rate 0, with more coverage), the calibrated baseline and finite JSON; the frontier has 71 rows from 0.30 and an advisory suggestion; bad thresholds, rates and days are 400s, an unknown version 404 | | `test_gateway_rejects_bad_requests` | Empty `messages` returns 400; a `stream: true` request the cascade refuses (the strict mock, `RATCHET_MOCK_STRICT=1`, does not know it) gets a normal HTTP 400 before any stream starts | | `test_openai_compat_provider_parses_logprobs_and_usage` | `OpenAICompatProvider` against an `httpx.MockTransport`: sends `logprobs` and `n`, sends the bearer header, parses logprobs per choice, splits completion tokens across choices (evenly here, because one choice has no logprobs), computes `mean_prob` | | `test_every_version_carries_a_certificate_in_its_eval_and_bom` |: 3 simulated days: every version's eval and signed `manifest.eval` carry a certificate with `lcb ≤ precision`, ten frontier rows, per-task bounds and a test vector of `test_n` letters, all clean under `recheck`; a version never re-scored has its signed eval | | `test_certificates_move_no_simulation_number` |: the same run with the certificate code taken out of the golden eval has the same days, totals, versions, thresholds and decisions; events only gain appended text | | `test_ltt_thresholds_come_from_the_fixed_grid` |: with `--calibration ltt --cert-target 0.9` every version's threshold is a grid value or 1.01 and its trace starts at 0.95 | ### `tests/test_hardening.py` Regression tests for the issues in [Issues found while documenting](#issues-found-while-documenting-and-their-fixes): `test_old_database_file_is_migrated`, `test_auto_day_counts_days_from_first_request`, `test_student_failure_falls_back_to_teacher`, `test_fallback_model_uses_default_threshold`, `test_teacher_failure_returns_502`, `test_unknown_tasks_do_not_grow_registry`, `test_live_alert_fires_once_per_version` (for the e-process alert, the default since, and the Wilson rule), `test_policy_from_env`. `test_anytime_valid_settings_parse` checks the anytime-valid gate's variables: `RATCHET_SEQUENTIAL`, `RATCHET_ALPHA_ROLLBACK` and `RATCHET_ALERT_EPROCESS` parse, `from_env(defaults=)` fills only unset fields, bad values name their variable, and `RATCHET_CANARY_AUDIT_RATE` is None unset, 0.5 when set, and stops `Ratchet` at 0, 1.5, -0.1 or NaN (and `Settings` at a non-number). `test_task_names_match_in_full` covers a bug found while building request validation: a task name with a trailing newline passed the `^...$` check. `test_live_night_checks_the_student_endpoint_before_paid_work` covers the issue found while testing the live entry points ([Live-path tests](#live-path-tests)). `test_stored_threshold_serves_what_the_eval_measured` covers a bug found while building certified precision: the golden eval stored its threshold rounded to 4 decimals, which could round up past the confidence it was calibrated at, so a version served less than its eval measured (nothing at all when every row had one confidence, as behind a constant-logprob endpoint). The answer-hygiene tests (v0.5) cover what a code reading found before any live run: reasoning before a lone `` was served and stored, an unterminated `` let the reasoning's JSON count as the answer, structured answers were stored inside code fences, a teacher answer cut off by the token limit (or empty, or not JSON for a structured task) became a training target and an audit, JSON nested too deeply crashed the request instead of escalating, and some long answers took seconds to clean. The v0.4 hardening tests (privacy, API keys, streaming, signed BOMs) are in the same file. The `gw_env` fixture (in `tests/conftest.py`) reloads the gateway in mock mode with extra environment variables and reloads it again with the defaults afterwards, so settings such as API keys never leak into other tests. | Test | Covers | |---|---| | `test_redact_finds_each_kind_and_checks_checksums` | Every label is found in one text (two phones, ten placeholders); a card number that fails Luhn is left alone | | `test_redact_leaves_ordinary_text_alone` | Order numbers, dates, times, prices, version strings, page counts, URLs and an ISBN are not touched | | `test_redact_messages_handles_content_parts` | OpenAI content-part lists are redacted part by part; non-text parts and the caller's copy are untouched | | `test_redact_messages_covers_names_refusals_and_call_arguments` | `name`, an assistant's `refusal`, tool-call and legacy function-call arguments are redacted and counted; a malformed tool call and the caller's copy are untouched | | `test_redaction_is_a_no_op_on_the_mock_world` | 2,000 requests from the `mixed` stream and 800 each from `schema` and `tools`, their stored conditions and their teacher answers (text and tool-call arguments) are unchanged by redaction, so mock lookups and simulation numbers cannot shift | | `test_examples_are_stored_redacted_but_the_teacher_sees_the_original` | The teacher receives the email and phone; the stored example has `[EMAIL]` and `[PHONE]` in messages and completion, `redactions == 3`, and `privacy_stats` agrees; with `redact_pii=False` the example is stored verbatim | | `test_keyring_parses_and_identifies` | Named and bare keys, lookup, short keys, bad names and duplicate names raise; `bearer()` reads `Authorization` and `X-API-Key` | | `test_api_keys_protect_v1_and_name_the_client` | No key or a wrong key gets 401 on `/v1/chat/completions` and `/v1/models`; Bearer and `X-API-Key` both work; `clients` in stats lists both names and never the keys; Scout stays public | | `test_daily_quota_per_key` | Quota 2: remaining-requests header counts down, the third call gets 429 with remaining 0, and another key is unaffected | | `test_streaming_matches_the_non_streaming_answer` | `stream: true` returns `text/event-stream`, a role chunk first, content chunks that join into exactly the non-streaming answer, a final chunk with `finish_reason: "stop"` and the `ratchet` block, then `[DONE]` | | `test_signed_bom_per_adapter_verifies_and_detects_tampering` | With a signing key, a simulated night stores a BOM that matches the schema, lists only distillable sources and verifies with the public key; editing it breaks the SHA-256, re-hashing still fails the signature, and another key fails; unknown versions get 404 | | `test_bom_is_stored_unsigned_without_a_key_and_cli_verifies` | Without a key, a simulated night stores an unsigned BOM whose dataset hash matches the version; the CLI `verify` and `keygen` commands work | | `test_bad_signing_key_fails_at_startup` | A key that is not base64, or not 32 bytes, raises | | `test_v03_database_gets_v04_columns` | A v0.3 database file gets `client`, `redactions` and `bom` on open | | `test_v04_database_gets_the_route_column` | A v0.4 database file with requests in it gets `requests.route` on open; `stats_by_task` reports the old rows as `unrecorded` next to a new `cold_start` row | | `test_v04_database_gets_the_examples_params_column` |: a v0.4 examples table gains `params`; its old row keeps its v0.4 hash and no conditions, a repeat bumps it, the same messages under JSON mode are a new example, and `dataset.rows` returns both | | `test_healthz_stays_public_when_keys_are_on` | `/healthz` answers 200 while `/v1/models` needs a key | | `test_redact_preview_endpoint` | `POST /ratchet/redact` returns the redacted text and counts; over-long or missing text gets 400 | | `test_live_night_checks_the_student_endpoint_before_paid_work` | Without `RATCHET_STUDENT_BASE`, `NebiusTrainer.train` raises before the sync, the training job or any file write; `reload` with adapters to load raises the same clear error (and with none, returns an empty list) | | `test_task_names_match_in_full` | `tasks.valid_name` matches the whole name: a trailing newline, 65 characters, an empty string, a space or a tab is not a name, and `tasks.get` falls back to `default` for each | | `test_teacher_reasoning_and_fences_are_not_stored_or_served` (2 cases) |: a teacher answer with a `` block, or with only a closing ``, and a fenced JSON answer is served and stored as the bare JSON, with `stored = 1` and no skip | | `test_served_text_is_clean_on_the_student_and_pinned_paths` |: a confident student's fenced answer with reasoning is served as bare JSON; a pinned (`research-plan`) and a teacher-only answer lose the reasoning but keep their fence (free text), store nothing and are no skip | | `test_truncated_teacher_answer_is_not_stored` |: through the real `OpenAICompatProvider` and a `FakeOpenAIServer` that reports `finish_reason: "length"`, a teacher answer whose visible part parses is served on a structured and a free-text task, but nothing is stored, both rows have `stored = 0`, and one `skip_example` event names the task and `finish_reason length` | | `test_empty_teacher_answer_not_stored` (4 cases) |: an empty, whitespace, unterminated-reasoning or reasoning-only answer is served as `''` on both kinds of task, not stored, `stored = 0`, `empty answer` | | `test_invalid_json_teacher_answer_for_ticket_triage_not_stored` |: prose, cut JSON and a JSON list are served but not stored for `ticket-triage` (`invalid answer`), while the same text is stored for the free-text `default` task | | `test_redaction_that_breaks_a_structured_answer_is_not_stored` |: a card-like number outside quotes becomes `[CARD]` and breaks the JSON, so the answer is served but not stored (`invalid after redaction`); with redaction off it is stored as is | | `test_unusable_teacher_answer_is_not_an_audit` |: an audit whose teacher answer was cut off gives `audited = 0`, `audit_agree` NULL, `stored = 0`, no audit for the version, and a cost that still includes the teacher's call; a complete answer is an audit and an example again | | `test_invalid_shadow_reference_leaves_shadow_agree_null` |: a shadow disagreeing with the live student, arbitrated with prose, gets `shadow_agree` NULL and one unarbitrated row, nothing stored; a valid arbitration settles the next one and is stored; an escalation whose teacher answer was cut off leaves `shadow_agree` NULL too | | `test_unusable_audit_answer_is_no_shadow_reference` |: with a cut-off audit answer, a shadow that agrees with the served answer gets `shadow_agree = 1` (no arbitration, precision 1.0), as without the audit; a disagreeing shadow is still arbitrated, and the cut-off ruling leaves NULL (review finding) | | `test_answers_that_quote_think_tags_are_served_and_stored_whole` |: a triage reply that mentions a stray `` tag is served and stored whole with no skip; a confident student quoting `` in its reply is served; a free-text answer that mentions `` is served and stored whole (review finding) | | `test_skip_example_events_are_throttled_per_reason` |: 200 cut-off answers from 8 threads write exactly 3 events (1st, 10th, 100th); `empty answer` and an unnamed finish reason (`other`) are counted apart; no event quotes the answer; every row has `stored = 0` | | `test_stored_column_marks_requests_that_stored_an_example` |: an escalation and its duplicate (`seen` 2) both have `stored = 1`; pinned and an unaudited student answer have 0; an audited one has 1 | | `test_v05_database_gets_the_stored_column` | A ledger from before answer hygiene gains `requests.stored`; its old row keeps NULL and a new escalation records 1 | | `test_answers_nested_too_deeply_are_invalid_not_a_crash` |: `parse_json` returns None instead of raising `RecursionError` on deeply nested JSON, so such a student answer is invalid and the request escalates (route `confidence`) instead of failing with a 500 | | `test_answer_cleaning_is_linear_on_adversarial_answers` |: 20,000 unclosed `` tags, 100,000 spaces, 100,000 `{`, a long space run before a fence, and stray closing tags followed by unclosed ones are each cleaned and judged invalid in well under 2 s (seconds each before, from regexes that rescanned every position); a fenced object after a sentence still parses | | `test_provider_parses_finish_reason_reasoning_tokens_and_per_choice_tokens` |: per choice, the text is `content` only (never `reasoning_content` or `reasoning`), `finish_reason` is read, output tokens follow the logprob counts and add up to `usage.completion_tokens` (in proportion when the counts do not, evenly with the remainder first when a choice has no logprobs, the counts as they are without usage), and reasoning tokens are split evenly; a single teacher choice gets the whole bill | | `test_finish_reason_values_are_read_safely` (11 cases) |: a missing, null, empty or blank `finish_reason` is `stop`; a word is lower-cased and trimmed; a number, a phrase or more than 40 characters is `unknown` | | `test_logprobs_too_large_for_a_float_are_ignored_not_a_crash` | (regression): a token logprob of 10^400, which `json.loads` accepts as an integer, made `math.isfinite` raise OverflowError while parsing (a 500 once the teacher returns logprobs); now it is left out of the confidence's logprobs and the served logprobs are None, and a normal value is served | | `test_provider_sends_params_and_its_explicit_arguments_win` |: `OpenAICompatProvider.complete(params=...)` adds the fields to the body; the explicit temperature, `n`, logprobs (off also drops `top_logprobs`) and `response_format` win; without params the body is v0.4's, and an unknown keyword argument is ignored; `allow` narrows what is sent | | `test_answers_under_label_altering_sampling_are_served_not_stored` (6 cases) |, threat model T2: a client's `logit_bias` of +100, a presence penalty of 2, a negative frequency penalty, temperature 2, a vendor field (`guided_choice`) or all three at once reach the serving teacher and its answer is returned, but nothing is stored (`stored` 0, one `skip_example` event naming the reason); neutral sampling (temperature 1, `top_p`, a seed, zero penalties) stores; an audit, which drops these fields, stores its answer | | `test_v04_file_gains_the_evidence_schema` | a ledger file with v0.4.0's schema and one row per table gains every new column (old rows: `contested` and `checked_epoch` 0, the rest NULL), the eight evidence tables, the append-only triggers and the new indexes, twice without change; its old data reads as before and new rows take the columns | | `test_every_index_exists_on_a_new_ledger` | every `INDEXES` entry exists on a fresh ledger (a misspelled column would make `_migrate` skip it everywhere), and the gate's audit count uses `requests_version_audited` | | `test_committed_leaves_and_sealed_receipts_are_append_only` |: `version_leaves` and `receipts` accept new rows and refuse UPDATE, DELETE, `INSERT OR REPLACE`, `INSERT OR IGNORE` and a duplicate INSERT (also on a migrated v0.4 file), a NULL key column and a split other than train or golden, leaving the rows as they were | | `test_reserved_settings_stop_startup_until_their_feature_lands` | each of the 9 reserved settings set away from its default stops `Settings.check_reserved()`, `Ratchet` and `python -m ratchet.live` (before the ledger is opened), naming the variable and the feature; the defaults spelled another way pass; a non-number names its variable | | `test_reserved_release_policy_fields_stop_startup` | each of the 5 reserved `ReleasePolicy` fields set through its variable stops `from_env`; the defaults, also set explicitly, pass | | `test_policy_from_env_names_a_bad_value` (8 cases) | a non-number, a fractional or negative count, a `live_floor` of 1 or 0, NaN and a negative tolerance each stop `from_env` with the variable named | | `test_certified_precision_settings_parse` |: `RATCHET_CALIBRATION` (any case; empty is `point`), `RATCHET_CERT_TARGET` and `RATCHET_CERT_DELTA` reach `Settings.cert_policy()` and the mock fleet; bad values stop `cert_policy`, the mock `Runtime` and `python -m ratchet.live` before a ledger is opened; the four `ReleasePolicy` fields parse and refuse values out of range | | `test_stored_threshold_serves_what_the_eval_measured` | (regression): every golden row at confidence 0.98049 measured 100% coverage, but the threshold stored rounded to 0.9805 served nothing; stored exactly, the router serves every row the eval counted | | `test_replay_migrates_an_old_file_and_counts_the_rows_it_cannot_replay` |: a v0.4.0 file migrates on open; its rows with a confidence are left out of the replay as `before_logging` and those without one counted as `no_student_answer`; rows the gateway writes afterwards are replayed at their logged threshold and audit rate; `since_day` leaves the old day out; two rows edited outside Ratchet (a probability of 1.5, a negative cost) are `malformed` and the report stays finite JSON | | `test_replay_endpoint_refuses_to_mix_versions` |: `version` twice, a list, a word, a negative, a fraction, or superscript and circled digits (`²`, `①`, `1²`, which `str.isdigit` accepts and `int` refuses) is a 400 on both replay endpoints, never a 500, saying versions are never pooled; an unknown version is 404; without `version` the live one is read | | `test_pinned_tasks_never_appear_in_replay_rows` |: a pinned task's requests have no replay row, and a task pinned after its traffic was logged is left out as `pinned_now` | | `test_replay_reads_a_long_history_in_chunks_up_to_the_id_it_started_at` |: with `REPLAY_CHUNK` at 2, `Ledger.replay_rows` returns exactly the rows of one read; another thread logs a request between each of the four chunks (the lock is free), and those requests are left out of this read and in the next | | `test_replay_endpoints_run_one_at_a_time` |: while a replay holds `rt.replay_lock`, both replay endpoints answer 429 with `Retry-After: 1`; afterwards 200 | | `test_replay_leaves_rows_held_by_a_changed_spec_unservable` |: rows logged at `SERVE_NOTHING` while the task's spec is changed for the version are never served at any replayed threshold, rows logged before the change are, and after a re-score under the new spec the held rows are servable again | | `test_replay_current_threshold_is_the_one_the_router_uses` |: the replay's `current_threshold` is the version's own in mock mode and the configured default (0.7, not the version's 0.9) for the hosted fallback student in live mode without a student endpoint, on both endpoints | | `test_receipt_verifiers_never_raise_on_nested_junk` | (8 cases): a receipt, or the previous receipt, whose `period` is a string, a number, a list or `{day: true}`, or whose `merkle` is a list or a string: `verify_chain` (both ways round), `verify_proof`, `verify_rows`, `recheck`, `check_manifest` and `verify_receipt` answer FAIL (they raised `AttributeError` before), and `python -m ratchet.receipt verify` with `--prev`, `--proof` and `--rows` exits 1 instead of printing a traceback | | `test_receipt_verifiers_never_raise_on_seeded_nested_mutations` |: 300 seeded trials, each replacing up to three fields anywhere in a receipt, the previous receipt, a proof or the leaves with junk of another type (or removing them): no verifier raises (72 raises in 2,000 trials before the fix) | | `test_a_stand_in_that_loses_money_lowers_the_fee` |: a fallback teacher's answer that cost $0.0002 more than its baseline adds -200,000 nano-USD to `savings.stand_in_loss_nusd`, the billable savings and the net bound, and lowers the fee to a quarter of the lower net; a cheaper fallback answer and a degraded answer add nothing (gains are never billed); the billable set is unchanged and the sealed receipt rechecks from its leaves | | `test_mock_reasoning_mean_out_of_range_stops_live_startup_too` |: `RATCHET_MOCK_TEACHER_REASONING_MEAN` of -5 stops a live `Runtime`, and 1e9 stops `python -m ratchet.live`, before the ledger is opened (it was checked in mock mode only); 420 is accepted | | `test_keyring_errors_name_the_entry_never_the_name_or_the_key` |: a pasted key that holds a ':' (the text before it is read as the name) is not quoted by any of the three errors; each says `entry N` | ### `tests/test_resilience.py` Upstream failures: around a request the student already answered, the provider's retries and the client statuses they end in, the student circuit breaker with automatic adapter reloads, and the teacher's breaker, degraded mode and the fallback teacher; the student falling back to the teacher is in `test_hardening.py`. Breaker tests run on a `FakeClock` (`Ratchet(clock=...)`, `Runtime(clock=...)`), so no test waits out a cooldown. The failing teacher puts its URL, with a query token, in each message, and every test checks the events, errors and responses never repeat it. Retry tests use the local `Upstream` fake (a scripted endpoint behind `httpx.MockTransport` that records each attempt's timeouts) and inject `sleep`, `clock` and `rng`, so nothing really waits. | Test | Covers | |---|---| | `test_audit_failure_still_serves_the_student` (3 cases) | With `audit_rate=1`, a teacher that times out, answers 503 or returns an unreadable body: the confident student's answer is served, the row has `audited=0` and `audit_agree` NULL, nothing is billed for the failed call, the baseline is the no-audit estimate, nothing is stored, one `audit_error` event (`ticket-triage: ReadTimeout`, `HTTPStatusError 503`, `ValueError`; never the URL), and `health()` counts one audit error | | `test_audit_errors_are_counted_and_throttled_and_escalation_still_fails` | With the teacher's breaker off (`teacher_breaker_failures=0`), 12 failed audits: 12 counted, events on the 1st and 10th only, no row counted as audited. The teacher comes back: audits and their training examples resume. An unsure student with the teacher down still raises (an error status at the gateway) and is not counted as an optional failure | | `test_failure_counters_are_exact_under_threads` | 8 threads, 200 failed audits (the teacher's breaker off): every request served by the student, exactly 200 counted, exactly three events (1st, 10th, 100th) | | `test_shadow_arbitration_failure_is_silent` | The shadow disagrees with the live student and the teacher times out only on that request: the user gets the live answer, the row keeps `shadow_serve=1` with `shadow_agree` NULL, no teacher cost and no example, one `shadow_error` (`v2 arbitration: ReadTimeout`); `shadow_quality` counts the row in `n` and coverage and as a disagreement (precision 1.0 → 0.5, `unarbitrated` 1) | | `test_shadow_sampling_error_counted` | With the breaker off (`student_breaker_failures=0`), a shadow endpoint that times out on 10 requests: all served by the live student at the samples' cost, no shadow record and no teacher call, events on the 1st and 10th, `shadow_errors` 10 | | `test_optional_teacher_failures_through_the_gateway` | Live `Runtime` with the real HTTP provider, its retry policy and a Token Factory that times out: audited requests (plain and streamed) are 200 with the student's answer, and the shadow's disagreement is not sent to the teacher again after the audit failed (one teacher call, two HTTP attempts, per request); unaudited, the failed arbitration is still a 200; the events read `UpstreamError 504 upstream_timeout`; an escalation is a 504 `teacher_timeout`. Live mode runs the audit and the shadow after the response, so the test waits for the sidecar before each check | | `test_student_error_event_never_quotes_the_endpoint` |: live `Runtime` with a student endpoint whose 400 quotes the request (`Invalid content: 'My SSN is 123-45-6789, email jane@corp.com'`) and its URL, as vLLM's validation errors can: the teacher answers (route `student_error`), and the `student_error` event, in the ledger and in the public `/ratchet/stats`, reads `v1: UpstreamError 400`, with no request text, message or URL | | `test_retry_backs_off_with_full_jitter_under_the_cap` | 503, 503, 200: three attempts, waits `uniform(0, min(cap, base·2^i))` that grow and stay under the cap; a long outage waits at the cap and returns the last `UpstreamError` (still an `httpx.HTTPError`); jitter's low end waits 0 | | `test_retry_after_is_honoured` | A 429 with `Retry-After: 2` waits exactly 2 s; `retry-after-ms` wins over `Retry-After`; a wait over the 8 s cap is not waited for: one attempt, and the error carries the wait | | `test_parse_retry_after_forms` | Seconds, fractions, milliseconds, an HTTP date, case-insensitive names, a bad ms value falling back; empty, zero, negative, infinite, NaN, past and malformed values are ignored | | `test_client_errors_are_not_retried` (6 cases) | 400, 401, 403, 404, 413 and 422: one attempt, no wait | | `test_deadline_stops_retries` | With a 5 s deadline and 1 s attempts, a wait that would end past it is not started (three attempts), each attempt's connect and read timeouts shrink to the time left (5, 3.5, 1.5 s), and a server wait counts too; with no policy, one attempt with the configured timeouts | | `test_read_timeout_is_retried_exactly_once` | A read timeout twice gives up after two attempts with 504 `upstream_timeout` (the cause kept, no URL); a 503 first does not use up that retry; connection failures before the model saw the request are retried up to `max_attempts`; write timeouts and read or write errors are not retried | | `test_bad_upstream_bodies_are_502_and_not_retried` (7 cases) | Empty, missing or null `choices`, a choice that is not an object, a body that is not an object or not JSON, JSON nested too deeply for the parser (a `RecursionError`): 502 `bad_upstream_response` after one attempt (before: an `IndexError`, so a 500) | | `test_malformed_optional_fields_count_as_absent` | Text parts joined, junk logprobs (strings, booleans, NaN) skipped, bad usage counted as 0, a missing model replaced by the requested one | | `test_upstream_error_bodies_are_parsed_without_the_url` (7 cases) | OpenAI's `{"error": {...}}`, vLLM's legacy body (integer `code` dropped), an error string, FastAPI's `detail` (string and validation list, with `param`), HTML (the reason phrase), and a long message with a control character and a lone surrogate (cleaned, clipped to 300); never the URL or the key; `raise_for_status()`'s error reads the same | | `test_upstream_errors_map_to_client_statuses` (20 cases) | Live `Runtime` at cold start, the teacher behind `MockTransport`: upstream 400 `context_length_exceeded` → 400 with the teacher's message, param and code; 413 → 400 `request_too_large`; 429 → 429 with `Retry-After` 7 (after two 7 s waits), 1 when none was sent, 60 for a 600 s wait (not retried); 500 and a connection error → 502 `teacher_unavailable` after three attempts; a 503 with `Retry-After` 2 → 502 with `Retry-After` 2 after two 2 s waits, with 30 or 600 → 502 with `Retry-After` 30 or 60 after one attempt (a wait over the 8 s cap goes to the client), a 504 with `Retry-After` 20 → 504 with it; a read timeout → 504 `teacher_timeout` after two; 401 and 403 → 502 `upstream_auth_error`; 404 → 502 `upstream_not_found`; 402 (also with a `Retry-After`, which is then not passed on) and a 301 redirect → 502 with `X-Should-Retry: false`; empty `choices` and JSON nested too deeply → 502, not 500. `X-Should-Retry: false` exactly where a retry cannot help; nothing logged, no `gateway_error` | | `test_live_runtime_passes_timeouts_and_retries` | The teacher gets its read timeout and retry policy (and the injected `sleep`), the student its own timeouts and no retries; the timeouts arrive on each request; the hosted fallback is a separate client with the student's settings; the variables reach `Settings`; zero, negative or NaN values stop live mode at startup | | `test_quota_429_says_when_it_resets` | At 23:00 UTC: `X-RateLimit-Reset-Requests: 1h0m0s` on the 200 and the 429; the 429 is `insufficient_quota` / `daily_quota_exceeded` with `Retry-After: 3600` and `X-Should-Retry: false`; 90.5 s before midnight rounds up to 91 s and `1m31s`; refusals are not counted; `_duration` formats | | `test_sdk_sends_an_exhausted_quota_once` | The real OpenAI SDK with its default two retries sends the quota-exhausted request once and raises `RateLimitError` with the type and code | | `test_sdk_sends_an_upstream_client_error_once` | The SDK raises `BadRequestError` with the teacher's `context_length_exceeded` and `param` after one request, and `InternalServerError` 502 `upstream_auth_error` without retrying | | `test_scout_error_event_names_the_upstream_status` | A failed Scout run says `a model or search call failed: upstream 503 service_unavailable` (or `upstream 502 upstream_unreachable`, `upstream 504 upstream_timeout`), never the URL | | `test_breaker_opens_probes_backs_off_and_closes` | `Breaker` on a fake clock: 4 failures and a success stay closed, 5 in a row open it (`open_until` 30 s on); exactly one probe after the wait; failed probes wait 60, 120, 240, 300, 300 s (at 300 s only the 1st probe's transitions are reported); a successful probe closes it and resets the wait; a released probe and a lost one (no report within the wait) let the next caller probe; a late result from another thread in half_open, and results while open, change nothing; threshold 0 never opens; a failing listener is swallowed; bad arguments and bad `RATCHET_STUDENT_BREAKER_*` settings raise `ValueError` when the cascade is built | | `test_half_open_lets_exactly_one_probe_through` | 25 rounds of 16 threads released by a barrier at the end of the cooldown: exactly one `allow()` is True each time | | `test_a_probe_slower_than_the_cooldown_still_counts_and_backs_the_breaker_off` | Regression (review of): a probe held past the cooldown on a hung endpoint used to be replaced and its failure dropped, so the breaker stayed half-open for the whole outage. Now a lost probe's failure reopens the breaker and doubles the wait (and its success closes it), and with `probe_timeout_s` nobody else probes while the probe can still be running. A 900 s hung-endpoint run (a request a second, calls held 15, 17 or 60 s, cooldowns 5 and 30 s) backs off to 300 s every time, with one probe in flight at most when `probe_timeout_s` is set; a bad `probe_timeout_s` raises | | `test_router_never_replaces_a_probe_that_can_still_be_running` | The router gives a breaker the student call's longest time (`OpenAICompatProvider.longest_call_s()`: pool, connect, write and read timeouts, 32 s by default, plus 5 s): with a 5 s cooldown, requests 5, 10 and 36.9 s into a probe that hangs on a `MockTransport` student skip the student (route `student_breaker`), and the probe's timeout reopens the breaker for 10 s | | `test_circuit_events_at_the_longest_wait_are_throttled` | 120 failed probes at 300 s report only the 1st, 10th and 100th pair of `circuit` transitions (each says which), every probe still counts in `failures`, closing is always reported, and the count starts again after the breaker closes | | `test_endpoint_failures_are_told_from_request_errors` | Timeouts, connection and protocol errors, 5xx, 408, 429 and an unreadable answer count as the endpoint down; 400, 401, 404 and `AdapterMissing` as the endpoint answering; a `KeyError` or `ValueError` as neither. `CircuitOpen` is a 503 `circuit_open` | | `test_open_student_breaker_sends_requests_straight_to_the_teacher` | A student that times out: 5 `student_error`s and `student: closed->open after 5 failures (ReadTimeout)`; the 6th request never reaches the student (call counter), is served by the teacher with route `student_breaker`, `escalated=1`, `confidence` NULL and only the teacher's cost, and is not another student error; a failed probe after 30 s waits 60 s; once the student is back the probe serves and closes it; the five `circuit` events in order; the teacher's breaker stays closed | | `test_shadow_breaker_is_its_own` | A shadow that times out opens the `shadow` breaker after 5 calls; the live student serves all 10 requests and its breaker stays closed; skipped shadow calls are not counted as errors and leave no shadow record | | `test_breakers_are_per_endpoint_and_role` | Live and canary on one endpoint share the `student` breaker (five failures between them open it for both); breakers are keyed by `base_url` (two clients of one URL share one), by class without one (a new provider object per call keeps one breaker), and by role; names `student`, `student-2`, `student-3`, `shadow` (and the teacher's own `teacher`) | | `test_lost_adapter_is_reported_and_does_not_open_the_breaker` | vLLM's 404 ``The model `student-v1` does not exist.``: 10 requests escalate, `on_adapter_missing` is called each time with the version, the breaker stays closed, the `student_error` detail says `AdapterMissing 404 adapter_missing`; a 404 that names no adapter or another one, and the hosted fallback model's 404, call nothing; a failing or missing callback never fails the request; a shadow's lost adapter is reported too and leaves the `shadow` breaker closed | | `test_errors_without_an_http_status_say_nothing_to_the_breaker` | 10 `KeyError`s from the student leave the breaker closed with no failures; a half-open probe that ends in a `KeyError` is handed back and the next request probes (and closes it); a 200 with no choices is a 502 `bad_upstream_response` that counts, so five of them open the breaker | | `test_adapter_reloader_rate_limits_and_reports` | `AdapterReloader` on a fake clock: one queued reload per adapter per 60 s, `adapter_reloaded` and `adapter_reload_failed` (`HTTPStatusError 503`, never the URL) events; the thread survives a failing ledger write; nothing is queued after `close()`; `adapter_reloader` refuses to build one without `RATCHET_STUDENT_BASE` | | `test_live_gateway_reloads_a_lost_adapter_once_a_minute` | Live `Runtime` with `FakeOpenAIServer(loaded=set())` as a restarted vLLM: 10 requests in 50 s are 200s from the teacher, one `load_lora_adapter` call with the adapter's path and the endpoint token, the breaker closed, rows routed `student_error`, `adapter_reload_failed` while loading fails; a minute later the next 404 reloads it and the student serves again. Mock mode and the hosted fallback have no reloader | | `test_demo_student_outage_opens_and_closes_the_breaker` | Mock gateway on a fake clock: `seconds` clamped to 5 and 300; a 60 s outage gives 5 `student_error`s, then `student_breaker`, a failed probe at 30 s, `student_breaker` again, and at 91 s a probe served by the student; the five `circuit` events and the `fault` event; `/ratchet/stats` routes add up to the task's requests | | `test_every_request_row_has_a_route` | Cold start, a pinned task, a teacher-only request, a confident student, one that is confident but cannot fill 4 choices, an unsure student, the fallback teacher for a down teacher, a degraded answer with both teachers down, a failing student, an open breaker and the student turned off by an operator give `cold_start`, `pinned`, `pinned`, `student`, `n_choices`, `confidence`, `fallback_teacher`, `degraded`, `student_error`, `student_breaker`, `student_disabled`, and a tool request and a part only the teacher reads give `tools` and `multimodal`: every route in `router.ROUTES`, no NULL route, `stats_by_task` routes per task, most frequent first, and `stats_by_day`'s `degraded` and `fallback_teacher` counts | | `test_open_teacher_breaker_answers_503_at_once_with_retry_after` |: live `Runtime` at cold start with a Token Factory that answers 503: five requests are 502 `teacher_unavailable` after three attempts each and open the teacher's breaker (`teacher: closed->open after 5 failures (UpstreamError 503)`); the 6th is a 503 `teacher_circuit_open` with `Retry-After: 30`, no `X-Should-Retry` and no teacher call; 12.5 s later the OpenAI SDK raises `InternalServerError` 503 with `Retry-After: 18`; nothing is logged; after the cooldown the probe finds the teacher back and closes the breaker. `_map_upstream` rounds the wait up and clamps it to 1 to 60 s; a student's `CircuitOpen`, or an upstream's own `teacher_circuit_open` code, stays a 502 | | `test_open_teacher_breaker_skips_audits_and_arbitration_without_counting_them` | Five escalations that time out open the breaker; the 6th raises `CircuitOpen` (503, `retry_after` 30) with no call and no row; then a confident student is served with the audit and the shadow's arbitration skipped (no teacher call, rows `audited=0`, `shadow_agree` NULL), no `audit_error` or `shadow_error`, only the `circuit` event; after the cooldown the next audit is the probe, succeeds and closes the breaker | | `test_teacher_breaker_counts_only_teacher_outages` | Seven 400 `context_length_exceeded` answers leave the breaker closed; four timeouts, then a tool-call answer (a success for the breaker) and a `ValueError` (no count), then four timeouts: still closed with 4 failures; `teacher_breaker_failures=0` never opens; bad `RATCHET_TEACHER_BREAKER_*` values stop startup; `Breaker.retry_in()` is None while closed or half-open, the time left while open, 0 once it is up | | `test_serve_student_mode_answers_degraded_with_a_flag_and_a_header` |: live `Runtime` with `RATCHET_ON_TEACHER_FAILURE=serve_student`, a student below its 0.99 threshold and a Token Factory that answers 503: a 200 with the student's answer, `ratchet.degraded: "teacher_unavailable"`, `X-Ratchet-Degraded` and `X-Ratchet-Served-By: student`; the row is route `degraded`, `audited=0`, `escalated=1`, `stored=0`, no shadow record (the shadow student is not called), billed for the samples only; a `degraded` event (`default: UpstreamError 503`); streamed the same; after the breaker opens, degraded answers come with no teacher call; no audits for the live version, no shadow evidence; `/ratchet/stats` counts them per day and per route; a confident answer has `degraded: null` and no header | | `test_error_mode_is_the_default_and_pinned_requests_never_degrade` |: by default an unsure student with the teacher down is still a 502 with no degraded header and no row; with `serve_student`, a pinned task (`research-plan`) is a 502 five times, then a 503 once the breaker is open, and the student is never called | | `test_degraded_mode_never_hides_a_client_error_or_serves_an_invalid_answer` | With `serve_student`, an unavailable teacher degrades (also with a client `max_tokens`), but these still raise: a request with `tools` (teacher-only since), the teacher-only path, a 400 `context_length_exceeded`, a non-HTTP error, a student whose samples are all invalid, a student that failed or whose breaker is open, a teacher tool-call answer and a cold start. `Serve_Student` with spaces is accepted, empty means `error`, other values stop startup | | `test_degraded_answers_are_counted_exactly_under_threads` | 8 threads, 200 degraded answers: every row route `degraded` with `audited` and `stored` 0, `health()["degraded_answers"]` exactly 200, three `degraded` events (1st, 10th, 100th) | | `test_n_choices_when_the_teacher_is_down` |: a degraded answer for n = 2 serves two agreeing student samples (usage of both); with only 2 of 3 agreeing, n = 3 gets the teacher's error while n = 1 still degrades; the fallback teacher writes n choices at 0.7 and stores nothing by default | | `test_fallback_teacher_answers_when_the_primary_is_unavailable` |: with the primary down, the fallback answers an escalation (route `fallback_teacher`, `served_by` teacher, the fallback's model id, its own price, the primary's price as the baseline, its own fingerprint), not stored, no shadow call, one `fallback_teacher` event; a 400 about the request is not sent to the fallback; five failures open the primary's breaker and the fallback then answers without a primary call; a pinned task gets the fallback, priced lower than its baseline; an audit is skipped, never sent to the fallback; with both down the primary's `CircuitOpen` is raised; no fallback-served row has a shadow record | | `test_stored_fallback_answers_carry_the_fallback_license` |: with `store_fallback_teacher`, the fallback's answer is stored with its own model and license (NVIDIA Nemotron Open Model License) and the primary's with OpenMDW-1.1, and the BOM lists both; by default nothing is stored, and without its own provider the fallback model is asked on the teacher's client; `RequestMeta(storable=False)` stores nothing, with no `skip_example` event | | `test_live_runtime_wires_the_fallback_teacher` |: `RATCHET_TEACHER_FALLBACK_BASE` gives the fallback its own client (the teacher's key, retries and timeouts); at cold start with Token Factory's host down the response's `model` is the fallback's, after three primary attempts and one fallback call, both with the teacher's key; without it, the fallback model is asked on the teacher's client; breakers `teacher` and `fallback_teacher`; the six variables reach `Settings`; `RATCHET_TEACHER_FALLBACK_MODEL=gpt-4o` stops the live `Runtime` | | `test_a_golden_row_the_student_refuses_never_stops_night_school` |: 60 examples, half stored with `tools`; a student behind `MockTransport` that answers 400 to `tools` as vLLM without a tool parser does; under an allowlist that forwards tools, `run_night` trains and reports `refused_n` equal to the golden rows with tools; the default allowlist sends no tools; a free-text row the student refuses escalates in production, so it is not judged (`judge_n` 0, no judge call); a 401 still stops the eval | ### `tests/test_scout.py` The `gw` fixture (in `tests/conftest.py`) reloads the gateway in mock mode with `RATCHET_MOCK_WARMUP_DAYS=0`. The guard tests use `FakeClock`. | Test | Covers | |---|---| | `test_examples_match_the_research_world` | `research.EXAMPLES` equals the UI's `EXAMPLE_QUESTIONS`; every planned query returns simulated results | | `test_scout_run_emits_the_documented_events` | A run emits `start` … `done`; steps have exactly the documented keys; planner first, writer last, all five agents present; one reader step per relevant triage; `llm_calls` excludes searcher steps; `cost_usd` equals the sum of the steps | | `test_pinned_tasks_go_to_teacher_and_are_never_training_data` | `research-plan` and `research-write` are pinned, `source-triage` is not; after a run the only stored example tasks are `source-triage` and `page-read`; `stats_by_task` shows 0% student share for the planner and one writer request | | `test_student_learns_scout_calls_over_simulated_days` | After 4 simulated days of `mixed` traffic a version is live, and a Scout run has at least 60% student calls and more than 30% savings | | `test_research_workload_simulation_learns` | `--workload research`, 6 days × 1,200: day 1 is 0% student; the last day is above 80% student and above 60% savings | | `test_gateway_scout_routes` | `/`, `/scout`, `/dashboard` return 200; `/scout/examples` in mock mode; `/scout/run` returns events ending in `done`; non-example and blank questions get 400; `/scout/stream` streams `start` … `done` with HTTP 200, and a bad question gives HTTP 200 with a single `error` event; `/scout/status`; `tasks` in `/ratchet/stats`; `/ratchet/demo/day?workload=research` works and an unknown workload gets 400 | | `test_rate_limit_returns_429` | With a guard of 2 runs per hour, the third `/scout/run` gets 429 | | `test_guard_budget_and_hourly_window` | Per-IP hourly limit, the window rolling over after an hour, the budget refusing new runs once spent, and a new UTC day resetting it (fake clock) | | `test_tavily_search_sends_bearer_and_parses_results` | `TavilySearch` against `httpx.MockTransport`: bearer header, `search_depth: basic`, `max_results`; results without a URL are skipped; a missing title falls back to the URL; snippet ≤ 500 and excerpt ≤ 1,500 characters with whitespace collapsed; an empty key raises | | `test_scout_survives_bad_json_from_models` | When every model call returns non-JSON, nothing is judged relevant, nothing is read, and the writer's raw text becomes the answer | | `test_guard_caps_runs_per_day` | `max_runs_per_day=2` refuses the third run of the day and resets the next UTC day | | `test_demo_controls_are_serialised_and_reset_has_a_cooldown` | A mock-mode action while another holds `_demo_lock` gets 409; a second reset within 30 s gets 429 | | `test_failed_run_returns_502_and_its_spend_still_counts` | When the writer's call fails, `/scout/run` returns 502 and the planner, triage and reader spend is still added to the budget | | `test_stats_mark_pinned_tasks` | `/ratchet/stats` `tasks` reports `pinned` for the planner and writer tasks only | | `test_react_app_is_served_with_lite_fallback` | With the committed build, `/`, `/scout` and `/dashboard` return the React shell and `/assets/*` is served; `/lite*` return the hand-written pages; without a build `/` falls back to the lite page | ### `tests/test_contract.py` The contract tests ([Contract tests](#contract-tests)). Gateway tests use `gw_env`; cascade tests build a small ledger with `_version` and script the models with `RecordingProvider`, with every setting they depend on passed explicitly. | Test | Covers | |---|---| | `test_v1_models_lists_ratchet` |: `GET /v1/models` returns a `list` with a `ratchet` model; with API keys on it needs a key (401) and answers the same with one | | `test_task_from_metadata_and_header_precedence` |: the ledger's `task` column is `default` with neither, the metadata task alone, the header alone, and the header when both are set (both orders); other metadata keys are ignored | | `test_missing_or_malformed_messages` (9 cases) |: missing, `null`, empty, string, object and number `messages`, an unknown role, a number as content and `null` user content get 400 and log nothing (the last three were 500s until v0.5) | | `test_messages_items_must_be_objects_400` |: `["hi"]`, `[null]`, `[1]` and a nested list as messages get 400 and log nothing | | `test_bad_metadata_is_400` |: `metadata` as a string, a list, or with a non-string `ratchet_task` gets 400 and logs nothing | | `test_text_content_parts_answered_in_mock` |: a test run ticket sent as OpenAI text content parts gets the same answer as the string form and the same stored example (`seen` 2) | | `test_bom_endpoint_counts_by_split_and_fingerprint_changes` |, US-7: `/ratchet/bom` counts per split equal the ledger's; every source names the configured teacher and its license; `examples_sha256` is the documented SHA-256 over the sorted example hashes, stable between reads and equal to the stats block; a repeated request changes nothing, a new one changes the fingerprint and adds one | | `test_cost_report_contract` |: every key in `tests/fixtures/cost_keys.json` (what the dashboard's Economics panel reads from `GET /ratchet/cost`), with its JSON type, on a fresh gateway (an empty report: no requests, hours or days) and after two simulated days and nights plus a training job left open for 13 hours (open intervals and warnings have items, each checked); the fixture still holds every key, with its type, it had when the panel shipped (`COST_MINIMUM`); each view's breakdown adds up to its total and savings are the baseline less the total; each day's dollars, as the panel derives them, match the report's daily shares; the `/ratchet/stats` `cost` summary matches the report | | `test_economics_panel_helpers` |: runs `web/tests/economics-model.test.mjs` under Node's test runner (skipped without Node 22.6+ or `web/node_modules`; CI's `web` job runs it with `npm test`): every gridline label on the break-even meter and the daily savings chart, over a sweep of values, prints the value it sits at (a 2.5 step labels 2.5 and 7.5, not 3 and 8; 0 is never `−$0.00`); both ends of the meter carry a tick and its log caption names the real ratio (100× over more than 6 decades); the empty state needs a report whose counts are present and zero, so one with only `views` and `recommendation` still shows them; a truncated daily list without `since_day` labels the window "Through day N"; idle time is described "in hours" or "on days" as the report counted it (`idle_granularity`) | | `test_task_registry_contract` |: every key in `tests/fixtures/tasks_keys.json` (what the dashboard's Task registry panel reads from `GET /ratchet/tasks`, `GET /ratchet/tasks/{name}`, `POST /ratchet/tasks/validate`, `PUT` (200 and 422) and `DELETE`), with its JSON type, on a fresh gateway and after a request naming an unregistered task, a refused and a saved API task (records, a set with an object `item`, a policy); the fixture keeps the keys this item named; every field spec at any depth names a comparator type the panel labels; the editor's `SPEC_KEYS` (read from `registry-model.ts`) are taskspec's without the name, and putting them back changes nothing; `extends` on a built-in task copies its fields and spec hash; a built-in task is 403; a 404 for a name that is gone | | `test_resilience_panel_contract` |: `CIRCUIT_RE`, `OUTAGE_RE` and `WAIT_RE`, read from the dashboard's source and compiled by Python, match every event of the outage drill (on a `FakeClock`) and of a teacher outage driven to the 300 s wait; the state the panel derives (each breaker's latest `circuit` event) equals `Ratchet.health()` at every step, except one probe behind at the longest wait, as the panel's footnote says; the parsed waits are the breaker's; the drill's `fault` time plus its length is the response's `until`; the fallback counts the panel shows come from `routes` | | `test_dashboard_admin_key_stays_in_memory` |: a scan of `web/src`: the theme is the only thing in browser storage, no file that holds the admin key writes a URL, and `api.ts` sends the key only through `adminHeaders` (a Bearer header) | | `test_story_simulation_numbers_match_reference_run` |: every simulation number the story page prints (`web/src/lib/sim.ts`) is the reference run's at the precision its literal is written with: the honest bill's `SIM_COST_REPORT` (both cost views, the break-even, the crossover, the recommendation and its explanation word for word, read as strict JSON up to its `// end` marker), "Watch it learn"'s ten days and totals, the command the page says reproduces them (parsed into the same simulator inputs) and the GPU-hour line that command prints (which `web/tests/story-model.test.mjs` pins `simulateLine` to); the four recommendations the chapter explains are `cost.RECOMMENDATIONS` in order, and its "idle more than half" is `IDLE_SHARE`. Uses the session's `reference_run` | | `test_story_page_helpers` |: runs `web/tests/story-model.test.mjs` under Node's test runner (skipped without Node 22.6+ or `web/node_modules`): the reference report and a live one render through the same `billOf`; a field the gateway did not send stays unknown, never zero; the break-even and crossover formulas hold for the rounded numbers the page prints them with (the divisor to three significant digits), and the formula divides the endpoint rate the report recorded (`endpoint_usd_per_hour`), not the configured price; idle time is described in the report's buckets (`idle_granularity`); the three bills share one dollar scale and a tiny part keeps a sliver; the meter's rows and log scale; the four recommendations with the report's marked; the simulator's GPU-hour line rebuilt exactly; the trust chapter's redaction list and clipped event text | | `test_story_trust_claims_match_code` |: the story's "Built to be checked" chapter claims only what the code does: every file it names exists; its redaction labels are `redact.PATTERNS` in order, and redaction is on by default as it says; its gate numbers (precision, regression tolerance, audits, floor) are `ReleasePolicy`'s defaults; the invariants it cites (4, 6, 12, 14) say what it says; every route it names is served; its BOM commands work as printed (a signed BOM fetched from a mock gateway with `RATCHET_BOM_SIGNING_KEY` verifies offline with the key from `/ratchet/bom/public-key` and prints the page's line; an edited one prints the page's failure); the BOM carries the Clopper–Pearson certificate the v0.5 note mentions and `python -m ratchet.stats check` passes on it; CI runs the traceability command it prints; its link lands on the dashboard's `#redactor`; it names v0.5 work only where its foundations exist, and no evidence tour before `ratchet/tour.py` exists | | `test_who_model_helpers` |: runs `web/tests/who-model.test.mjs` under Node's test runner (skipped without Node 22.6+ or `web/node_modules`): a student call whose model is `student-v3` is "Nano + your adapter" with its adapter version in the tooltip; one whose model is the gateway's `fallback_student` (case and spaces ignored) is "Nano 30B · hosted", its tooltip naming the threshold and "Not covered by our adapter's precision certificate", the window sentence only when `student_serving.window` is set; the teacher is "Ultra · Token Factory"; a tool is "Tavily" or "Search"; every text matches the label table; the badge context `whoContext` builds from `/ratchet/stats` or `/scout/status` (an older gateway's reads as before); the dashboard header's serving line for each kind; no precision figure while `hosted_fallback` serves; Scout counts hosted calls and a run they answered (or, serving hosted, a run with no student answer) stops saying "your adapter", while a run our adapter answered keeps it | | `test_student_serving_kinds_match_the_ui` |: the `StudentServingKind` union in `web/src/lib/api.ts` lists exactly `STUDENT_SERVING_KINDS`, and `/ratchet/stats` reports one of them with `fallback_student` equal to the setting the live resolver uses | | `test_stats_contract` |, US-9: every key in `tests/fixtures/stats_keys.json`, with its JSON type, on a fresh gateway (lists empty) and after two simulated days and nights plus a keyed request (every list has items, each item checked); the fixture still holds the keys this item named; totals, days, tasks and clients add up to the same requests and dollars, and the savings are positive; each task's `routes` add up to its requests and use only known routes | | `test_demo_endpoints_are_403_in_live_mode` |: in live mode `POST /ratchet/demo/day` (also with bad arguments), `break-next`, `student-outage` and `reset`, and `GET /ratchet/demo/sample-example`, get 403 and change nothing in the ledger | | `test_student_sampled_once_with_production_params` |: one student call per request with `n = Settings.samples` (3 and 5), temperature 0.7 and logprobs on, to the version's adapter with the request as sent; an escalated or audited request adds exactly one teacher call | | `test_confidence_formula_exact` |: two of three samples agreeing, the best majority sample at p = 0.9, give 0.4 × 0.9 + 0.6 × 2/3 (not the minority's 0.99, not the weaker majority sample's 0.6), from `score()` and from the router, which logs the same number | | `test_confidence_without_logprobs_is_agreement` |: without token logprobs the confidence is the agreement: 2/3 for a split, exactly 1.0 for a unanimous student whose endpoint ignores `logprobs=True` | | `test_threshold_boundary_serves_at_equality_and_escalates_below` |, US-3: for a calibrated threshold and for the global default, a confidence equal to the threshold is served by the student (no teacher call, nothing stored) and one a float step below escalates (teacher answers once, its answer stored, the version still logged as the one that tried) | | `test_audit_rate_zero_and_one` |, US-4: rate 0 never calls the teacher on served answers; rate 1 audits every one, logs each verdict, stores each answer and never changes what the user gets, and `version_live_quality` reads the result; an escalation is never counted as an audit | | `test_audit_rate_is_respected_statistically` |, US-4: at rate 0.2 over 2,000 served requests (seeded), the audited share is inside the 99% Wilson interval, with one teacher call and one stored example per audit | | `test_cost_includes_audit_shadow_and_arbitration` |: cost and all-teacher baseline to the token in 11 cases (served, audited, escalated, student endpoint down, cold start, pinned task, shadow agreeing, unsure and arbitrated, an audit or an escalation as the shadow's reference), with distinct token counts per model; the ledger row equals the result | | `test_replay_columns_are_logged_without_extra_calls` |: on 24 paths (served, audited, an audit or escalation that disagrees or is invalid, an invalid student answer, the student down, cold start, pinned, shadow agreeing, only an arbitration, an audit or escalation as the shadow's reference, degraded, the fallback teacher, and escalations under a client's temperature 0, logprobs, temperature 1, seed or `logit_bias`, for two choices, and an audit under temperature 1) the logged `threshold`, `label_p`, `student_agree` and `baseline_est_usd` are as designed and the teacher and student calls are exactly as many as before; `student_agree` is never set from an arbitration, a stand-in, an invalid answer, or an escalation the teacher answered under other conditions than an audit's (a sampled or client-steered answer: temperature above 0, a seed, `logit_bias`, n choices; logprobs alone keep the label); without a teacher answer `baseline_est_usd` is the baseline to the bit; for n choices the formula counts the choices served, or the student's answer once per choice | | `test_usage_is_the_served_answer_and_the_breakdown_every_call` |: on the 11 paths of the cost test, `Result`'s usage is the served answer's (the chosen student sample's 30 + 7 tokens, or the teacher's 120 + 45 with its 40 reasoning and 100 cached tokens), `usage_breakdown` lists exactly the calls made (student samples, served, audit, shadow, arbitration; `None` for the others), `cost_usd` equals the price of the breakdown to 1e-12, and the fingerprint is the live version's or the teacher's | | `test_usage_estimates_missing_counts_and_keeps_details_inside_totals` |: a teacher that reports no usage gets prompt tokens estimated from the messages as JSON (non-ASCII kept) and completion tokens from the answer as served (reasoning stripped), while the breakdown keeps its zeros; an empty answer has 0 completion tokens; reasoning and cached tokens above their totals are cut to them | | `test_concurrent_requests_keep_the_ledger_consistent` |: 8 threads × 40 requests through one `Ratchet`: no request lost or logged twice, one teacher call and one example per audit, each row's cost matches its own path, totals equal the sum | | `test_each_result_names_its_ledger_row` | `Result.request_id` is the id of the row `log_request` wrote, on the student, pinned, teacher-only and cold-start paths, and for each of 100 requests from 4 threads | | `test_each_call_gets_the_parameters_its_role_allows` |: with one client parameter set, the confident student gets `n` 3, 0.7, logprobs and its shaping fields, and the audit temperature 0 with the limit, stop and format; an escalation, a cold start, a pinned task and a teacher-only request send the teacher everything, logprobs included, at the client's temperature; a shadow samples like the live student and its arbitration is a check; without parameters both caps alone; the model is always the configured teacher; a `Task` object works as `task` | | `test_examples_store_the_conditions_the_teacher_answered_under` |: an escalation stores JSON mode and its stop (an email in it redacted and counted) as `params`, hashed with them; the same messages without conditions are a second example, and sampling fields that keep the teacher's own distribution (temperature 0.8, `top_p`, a seed, a zero penalty) bump it; the completion is the teacher's; an audit stores its own (check) conditions | | `test_reasoning_headroom_marks_answers_over_the_clients_budget` |: with 600 tokens of headroom the teacher gets 620 for a client's 20; an answer whose visible part is longer is served whole with `finish_reason` `length`, not stored, and a `skip_example` event names it; a short one after long reasoning is stored; an over-budget audit is not an audit; without headroom the teacher's own `stop` stands | | `test_a_request_that_is_not_storable_stores_nothing` |: `RequestMeta(storable=False)` on an audited student answer and an escalation stores and bumps nothing, still logs the audit's verdict, and writes no `skip_example` event | | `test_bad_generation_settings_stop_startup` |: a forbidden or malformed allowlist entry, or a negative cap or headroom, makes `Ratchet(...)` raise; `Settings.gen_policy()` parses a valid list | | `test_n_choices_are_scored_as_calibrated_and_choice_zero_is_the_one_choice_answer` |: with scripted samples, n = 1 to 4 draws `max(3, n)` samples and reports and logs the one-choice confidence; served choices are the best scored sample then the agreeing ones by token probability, with usage (prompt once, every choice), breakdown, cost and baseline to the token; a confident student that cannot fill n escalates with route `n_choices`, the teacher writes n choices at 0.7, cost and baseline price the whole call, and only choice 0 is stored; an unsure student escalates on `confidence` | | `test_teacher_writes_n_choices_and_only_choice_zero_is_stored` |: an escalation gets n choices at 0.7 or the client's temperature, and its example is bumped once per request; cold start, pinned and teacher-only requests get n choices, pinned ones stored never; a teacher returning fewer choices gives fewer; an audit of an n = 2 student answer is one answer at temperature 0 without logprobs, judged against choice 0, with a baseline of one audit-sized answer per choice; the shadow samples 3 while the live student draws 4, and its disagreement with choice 0 is arbitrated | | `test_logprobs_come_back_only_when_asked_and_match_the_score` |: choice 0's token logprobs spell its text and their mean probability is `Scored.token_prob`, with the client's `top_logprobs` alternatives (the student got `top_logprobs`); none without `logprobs`, the student sampled as before; each of n choices carries its own; the teacher gets `logprobs` and `top_logprobs` only when asked, an audit never; a teacher that sends none gives None | | `test_empty_json_answer_scores_zero` |: three `{}` answers to `ticket-triage` score 0 and `{}` never agrees with `{}` (a strict xfail until field comparators made the built-in fields required) | | `test_o_series_prefixes_refused` (4 cases) |, US-8: `o1-preview`, `o3-mini`, `o4-mini` and `openai/o3-mini` are refused under the OpenAI terms | | `test_closed_teacher_fails_at_startup` |: `Ratchet(...)` raises for `gpt-4o` and for an unknown teacher; the gateway's `Runtime` refuses to start with `gpt-4o` in live and mock mode | | `test_closed_fallback_teacher_fails_at_startup` |: `Ratchet(...)` raises `LicenseError` for a fallback teacher that is closed (`gpt-4o`, `openai/gpt-4o`, `o3-mini`, a Claude and a Gemini model) or unknown; the gateway's `Runtime` refuses to start with it in live and mock mode; a distillable fallback is licensed up front (`Ratchet.licenses`) | | `test_examples_record_license_url` |: for two registered teachers, the stored example has the teacher, license name and URL from the registry, the request hash, a split, and the BOM lists exactly that source | | `test_configured_models_are_licensed_and_day1_checked` | (offline proxy): every priced model in `config.py` except `student` is in `licenses.REGISTRY` and in `WANT` in `scripts/day1_check.py`, and the default teacher, judge and fallback student are priced | ### `tests/test_openai_compat.py` OpenAI compatibility of `/v1/chat/completions` (US-1): request validation, message normalisation, the error envelope, the limits, the rule that a client mistake is never a 500, and responses and stream chunks that pass the OpenAI SDK's own models, with `usage` and `system_fingerprint`. Gateway tests reload the mock gateway with every setting they depend on pinned; the SDK tests drive the real `openai` package through `tests/sdk_bridge.py` (skipped if it is not installed). | Test | Covers | |---|---| | `test_bad_request_gets_an_openai_error` (22 cases) | `["hi"]` as messages, `metadata: "x"`, a missing role, role `robot`, temperature 3, `n` 0 and 5 (above `RATCHET_MAX_N`), five stop strings, `top_logprobs` without `logprobs`, `json_schema` without a name, bad JSON (incl. `NaN`, 100,000 nested brackets, bad UTF-8, an empty body, an unpaired UTF-16 surrogate escaped or as UTF-8 bytes), a non-object body, a body over 2 MiB (413), 120,001 characters of input, an image part, and a bad field with `stream: true`: each gets the expected status, `code` and `param` in the envelope (with `detail` repeating the message), as JSON, with nothing logged | | `test_no_client_mistake_gives_500` | Every value of a request that uses every field (developer role, text parts, tool calls, metadata, `json_schema`, tools) swapped for one of eight kinds of junk, or deleted, each judged by `validate_chat_request` first: one it refuses gets that exact `code` and `param` from the gateway; one it accepts gets 200 from a live-mode gateway with the teacher behind `MockTransport` and from the mock gateway (the generic mock world answers it). Never a 500; refused requests are never logged | | `test_unpaired_surrogate_is_a_400_in_both_modes` | An unpaired surrogate in a system message sent with a test run ticket, in `name`, in a text part or in a key is 400 `invalid_json` with its JSON path in mock mode, and in live mode the teacher is never called; a valid escaped pair (an emoji) reaches the teacher as one character | | `test_unknown_v1_path_and_wrong_method` | Unknown paths under `/v1` and `/openai` get 404 `not_found_error` / `unknown_url` with `Unknown request URL: METHOD path.`; `GET /v1/chat/completions` gets 405 with `Allow: POST`; other paths keep `{"detail": "Not Found"}` | | `test_unhandled_error_is_a_500_envelope_and_an_event` | A crash in the cascade gives 500 `api_error` without the exception's message; the ledger gets `gateway_error` events (class name only) at the 1st and 10th crash; a crash outside `/v1` keeps the plain-text 500 and is counted but not written (11th); when the ledger itself fails at the 100th, the 500 still goes out | | `test_mock_keyerror_is_400_and_live_keyerror_is_500` |: with `RATCHET_MOCK_STRICT=1` an unknown request, a request without a user message and a Scout writer request without JSON facts (also facts nested too deeply to parse) get 400 `mock mode only answers demo requests` and log nothing; without it the generic mock world answers all four with a 200; in live mode a `KeyError` from the cascade is a 500 | | `test_framework_validation_errors_become_400_on_v1_only` | FastAPI's validation errors are 400 envelopes with `param` and `code` on `/v1`, and stay 422 `{"detail": [...]}` elsewhere | | `test_ratchet_routes_keep_detail` | `/ratchet/redact` (400, and 413 over a 1,000-byte limit), `/ratchet/versions/999/bom`, `/ratchet/demo/day` and `/scout/run` errors keep `{"detail": ...}` | | `test_checks_run_in_order_and_refusals_cost_nothing` | With a key and a quota of 1: no key and a bad body is 401 (`invalid_api_key`, `WWW-Authenticate` kept); a bad body with a key is 400 and uses no quota; the next good request is 200; the one after is 429 `rate_limit_error` with `Retry-After` and `X-Should-Retry: false`; a bad body at the quota is still 400; one ledger row | | `test_body_limit_declared_and_streamed` | A declared `Content-Length` over the limit is 413; a chunked body driven straight into the ASGI app is refused at the chunk that passes the limit, with the rest left unread; outside `/v1` the 413 is `{"detail"}`; a chunked body under the limit is answered; a client that disconnects halfway through its body gets a 400, not a 500; `0` turns the limit off | | `test_message_and_input_limits_come_from_settings` | `RATCHET_MAX_MESSAGES` and `RATCHET_MAX_INPUT_CHARS` (text parts added up) give 400; `0` turns both off (a 301-message, 200,000-character request is answered) | | `test_developer_role_reaches_the_teacher_as_system` | A `developer` message reaches the teacher, and the stored example, as `system`; `RATCHET_MERGE_SYSTEM=1` merges the leading system messages with a blank line | | `test_dropped_keys_reach_neither_the_teacher_nor_the_ledger` |: keys a message's role does not use (a vendor field, `annotations`, a tool call's `index`) never reach the teacher; a conversation with tool calls and results is a tool request, so its tool call's arguments reach the teacher as sent and nothing is stored (route `tools`, `uses_tools` 1); redaction of stored tool turns is in `tests/test_tools.py` | | `test_text_parts_ticket_matches_the_string_form` | A test ticket sent as text parts gets the same message back as the string form, and one stored example, seen twice, under the string form's hash | | `test_multimodal_parts_go_to_the_teacher_only` | With `RATCHET_TEACHER_MODALITIES=text,image_url` and a live student, an image request reaches the teacher with its parts as sent (route `ineligible:content_type`), never the student, stores nothing and logs one teacher row; a text request on the same gateway is still served by the student; audio stays refused; the mock teacher answers the image request from its text part, and nothing is stored | | `test_generation_parameters_reach_teacher_student_and_audit` |: a live gateway, both models behind `MockTransport`: for `max_completion_tokens` 50, a stop, seed 7, temperature 0.9, `model: gpt-4o` and a vendor `top_k`, the escalated teacher's body has `max_tokens` 50, the stop, the seed and 0.9 under the configured model, the student's has 0.7, `n` 3, logprobs, the stop, the seed and 50, and a forced audit's has 0 with the same stop and limit and no seed; `top_k` reaches the teacher only with `RATCHET_TEACHER_PARAMS=all,top_k` (the audit runs after the response in live mode, so the test waits for the sidecar first) | | `test_allowlists_and_token_caps_through_the_gateway` |: `max_tokens` 10⁶ is sent as 8,192 to the teacher and 1,024 to the student; without `logit_bias` in `RATCHET_TEACHER_PARAMS` it is not forwarded; `RATCHET_STUDENT_PARAMS=none` leaves the student its cap; `RATCHET_TEACHER_MAX_TOKENS=0` sends the client's value; a request without parameters still carries both caps; a forbidden allowlist entry or a negative cap stops `Runtime` | | `test_a_cut_off_teacher_answer_reaches_the_client_as_length` |: a teacher answer with `finish_reason` `length` is returned with `length` (and only the stream's last chunk carries it), and is not stored; an upstream value clients do not know is returned as `stop` | | `test_finish_reason_is_tool_calls_only_with_tool_calls` |: a request with `tools` goes to the teacher alone with `tools` and `tool_choice` (a student that would serve anything is never called, nothing stored); a prose answer ends with `stop`, and so does a teacher that says `tool_calls` but sends none, streamed or not; a teacher tool call to a request without tools (default or `RATCHET_TEACHER_PARAMS=all`) is a 502 `tool_calls_unsupported` with `X-Should-Retry: false`, also when streaming, and stores nothing, while the same call to a request with tools comes back as `message.tool_calls` with `finish_reason: "tool_calls"` | | `test_mock_models_honour_max_tokens_and_stop` |: in mock mode `max_completion_tokens` 3 cuts the teacher's answer to 12 characters with `length` and stores nothing; a stop string ends it with `stop` and stores it with that condition; the real OpenAI SDK reads `length` | | `test_openai_sdk_round_trip_and_errors` | The real OpenAI SDK: a `ChatCompletion` with the `ratchet` block, the same answer for text parts and the developer role, and `BadRequestError` with `code` and `param` for temperature 3 and for too much input | | `test_openai_sdk_auth_and_quota_errors` | The SDK raises `AuthenticationError` with code `invalid_api_key`, and `RateLimitError` for the daily quota without retrying it (two requests sent with `max_retries=2`) | | `test_responses_are_openai_chat_completions_with_usage` |, US-1: on a live gateway (both models behind `MockTransport`), a student-served and a teacher-served response pass the OpenAI SDK's `ChatCompletion` model with exactly OpenAI's top-level fields, `service_tier` `default`, a `fp_` fingerprint, `message.refusal` null, `annotations` empty and `logprobs` null; `usage` is one student sample's tokens (10 + 5, not the 3 samples' 15), or the teacher's 40 + 320 with 300 reasoning and 16 cached tokens; the breakdown lists the student's call and the served one; a cut-off teacher answer is `length` and still validates; the real SDK reads the same usage and fingerprint | | `test_mock_teacher_reports_its_reasoning_tokens` |: with `RATCHET_TEACHER_REASONING_TOKENS=120` the mock teacher's response reports 120 reasoning tokens inside `completion_tokens`, `total_tokens` is their sum and positive, the fingerprint is the teacher's; a pinned Scout planner request has usage from its one call and a breakdown with only `served` | | `test_system_fingerprint_follows_the_serving_configuration` |: equal for answers of one version (and the documented hash of adapter, version, training-set hash and threshold); another after a promotion; the old one again after a rollback; another when the threshold changes; the teacher's own, stable, also in every stream chunk | | `test_stream_chunks_validate_and_include_usage_ends_with_the_usage` |, US-1: student- and teacher-served, every chunk passes `ChatCompletionChunk` and carries the id, model, fingerprint, tier and null logprobs; with `include_usage` every chunk has `usage: null` and the last data chunk before `[DONE]` has no choices, the non-streamed usage and the `ratchet` block; without it (absent, false, `{}`) no chunk has `usage` and the block is on the finish chunk; `stream_options` without a stream changes nothing; the real SDK's last chunk has the usage | | `test_provider_reads_cached_prompt_tokens` |: `usage.prompt_tokens_details.cached_tokens` reaches every choice of a call; a string, negative or boolean count, or details that are not an object, count as 0 | | `test_n_choices_from_a_trained_mock_student` |, US-1: the mock fleet trained on 40 test ticket; n = 2 with logprobs gives two choices with the same labels, usage that adds up both (also in the breakdown), a valid `ChatCompletion`, each choice's tokens spelling its content with one alternative each, and choice 0's mean token probability equal to `Scored.token_prob` of the same seeded samples; streamed, choice 0's role, content and finish chunks come before choice 1's, every chunk validates as a `ChatCompletionChunk`, content and logprobs per index add up to the non-streamed ones, the role and finish chunks carry none, and `include_usage` ends with the same usage; without logprobs both are null; the real OpenAI SDK reads both choices and their logprobs | | `test_n_choices_escalated_over_http` |: escalated, the teacher behind `MockTransport` receives `n` 2 at temperature 0.7 without logprobs, the student `n` 3, the response has both choices (usage 12) and one example is stored; with `logprobs`, `top_logprobs` 2 and temperature 0.3 the teacher gets them all and each choice spells its content, the example bumped once; `n` 4 draws 4 student samples; a teacher-only image request gets `n` too; a reasoning teacher whose tokens include `` text returns only the answer's tokens and nothing of the reasoning; a teacher that returns no logprobs gives `{"content": null, "refusal": null}` per choice, also in every streamed content chunk, and the response and chunks still validate | | `test_max_n_comes_from_settings` |: `RATCHET_MAX_N=1` refuses `n` 2 (400 `unsupported_value`, nothing logged), `6` accepts 5 (five choices from the mock teacher), the default is 4, and 0, -1, 129, 2.5 or `True` stop `Limits` and the `Runtime` with a `ValueError` naming the variable | | `test_chat_request_fields` | `ChatRequest`: `max_completion_tokens` wins over `max_tokens`, a string `stop` becomes a list, `null` fields are unset, unknown fields are kept in `extra`; conversations with tool calls, tool answers, legacy function calls and assistant messages dumped from a response (unused fields `null`) pass; valid forms of every optional field come back as sent; a non-string `metadata` key from a Python caller is refused | | `test_validation_rules` (87 cases) | One case per rule in [API.md](/reference/api/#request-validation): the body, messages, roles, content and parts, tool messages, assistant tool calls in full (`id`, `type`, `function.name`, `function.arguments`), legacy `function_call` arguments, `refusal`, every sampling range, `n`, `stop`, `logit_bias`, `logprobs`, `metadata` (17 keys, long keys and values, task names incl. a trailing newline), `response_format` and `json_schema` (name, type, depth 17, over 64 KB, `strict`), `tools` (129, custom type, names, parameters, over 64 KB), `tool_choice`, the legacy function fields, the string and boolean fields and `stream_options.include_usage`: each gives a 400 with the documented `code` and `param` | | `test_limits_and_their_off_switch` | `Limits`: too many messages, too many characters in a string, in text parts, in a refusal part, and with `name`, `refusal`, `tool_call_id` or tool and function call arguments counted; keys a role does not use are not counted; exactly at the limit passes; `0` means no limit; `max_n` (4 by default) | | `test_parse_body_and_error_helpers` | Strict JSON (`Infinity`, `NaN`, bad UTF-8 refused); a valid surrogate pair is one character (bytes, text, UTF-16), and an unpaired surrogate (escaped, as bytes, in a key, CESU-8) is refused with its JSON path in an error that is itself valid UTF-8; the status-to-type table for plain HTTP errors (401 `invalid_api_key`, 403, 404 `unknown_url` only for the router's own 404, 409, 413, 429, 5xx `api_error`); headers kept; FastAPI validation errors turned into a JSON path | | `test_normalize_rewrites_only_what_it_must` | `developer` → `system` (other keys kept), text parts joined by newlines, a refusal part becomes text, keys a role does not use dropped (assistant `audio` and `annotations`, vendor fields, `tool_calls` on a user message), tool and function calls kept to their documented fields, plain messages returned as the same objects, the caller's list never changed, and merging leading system messages | | `test_non_text_parts_need_a_teacher_modality` | `image_url`, `input_audio` and `file` parts, and a refusal part from the user, are refused with their exact `param`; with `image_url` allowed the message comes back as sent and `multimodal` is set; `parse_modalities` | | `test_normalize_is_identity_on_the_mock_world` | Every mock ticket (every template, drift too) and every Scout request in the research world comes back as the same objects, with or without merging, and passes validation with its task, so example hashes, golden splits and the reference simulation cannot move | ### `tests/test_livesim.py` Live and simulation side by side, offline: live mode runs over `httpx.MockTransport` and the janitor over a `FakeClock`, with no thread and no server. The compose file, the Caddyfile and `update.sh` are checked in `tests/test_live.py` (`test_compose_and_caddy_run_a_keyless_simulation_beside_the_live_gateway`, `test_update_script_checks_both_hosts` with stub `git`, `docker` and `curl`), and with the limits set in `tests/test_contract.py` (`test_demo_limits_never_answer_before_the_mode_check`). | Test | Covers | |---|---| | `test_upstream_recorder_is_content_free` |: a sentinel in the prompt, in an upstream 400's message and code and in a URL-shaped `model` reaches nothing in `/ratchet/stats` `upstream`, read as an admin and as the scrubbed anonymous reader; the 400 is `UpstreamError`, `http`, 400; the URL-shaped model id becomes `null` | | `test_upstream_calls_shape_and_roles` | The call items' exact keys; teacher, endpoint student (`student_endpoint`), hosted Nano (`token_factory`, the fallback model's id) and the fallback teacher by role; `token_factory` false for another base | | `test_upstream_failures_record_class_status_and_last_success`, `test_classify_never_returns_a_message`, `test_age_grows_with_the_clock` | 429, timeout, connect error and a non-JSON body classified; the first success kept; an observer that raises and junk records change no call; ages follow the clock | | `test_tavily_calls_are_recorded_without_content`, `test_providers_take_observe_as_keyword_only` | The Tavily hook (200 and 432) holds no query, URL or key; `observe` is keyword-only and changes no request body or completion | | `test_demo_actions_are_limited_per_ip`, `test_demo_limit_off_by_default` | A second action from one IP is a 429 with `Retry-After` and writes no event; another IP passes; reads are never limited | | `test_demo_caps_reset_on_click`, `test_janitor_resets_when_idle_and_dirty_only`, `test_janitor_thread_starts_once_and_only_when_asked`, `test_janitor_stop_ends_its_thread_and_it_is_never_started_again`, `test_reloading_the_gateway_ends_the_janitor_it_replaces`, `test_janitor_thread_starts_in_mock_mode_on_a_demo_action_only`, `test_demo_lock_held_blocks_the_janitor_reset` | The day and request caps reset on a click with `auto_reset` and one `demo_reset` event; the idle reset runs once, only when dirty, and never while a test run action holds the lock; the thread starts once, in mock mode, on a test run action; `stop()` ends the real thread at once and for good (twice and without a thread are fine), and so do `gw_env`'s reload and the app's lifespan: a janitor that survived its test reset the next test's gateway (the scout simulation lost its live version) | | `test_link_and_demo_settings_are_checked`, `test_token_factory_host_is_the_default_teacher_base`, `test_stats_links_demo_and_upstream_blocks` | Bad URLs and `RATCHET_DEMO_*` values name their variable; the deploy variables are known; the stats blocks | ### `tests/test_live.py` The live-path tests ([Live-path tests](#live-path-tests)). Each runs offline: model endpoints are `FakeOpenAIServer` behind `httpx.MockTransport`, commands go to `RecordingRun`, and the Scout CLI reaches the gateway through `AppTransport`. | Test | Covers | |---|---| | `test_nebius_trainer_writes_syncs_trains_and_hot_loads` |, US-10: commands in order (`RATCHET_SYNC_CMD`, then `bash deploy/train_job.sh student-v1`, each with `check=True`); `night/train.jsonl` holds exactly the rows as prompt-completion training rows with the template arguments (`dataset.training_record`) and is on disk before the sync; then `POST {root}/v1/load_lora_adapter` with `lora_name`, `lora_path` under `/data/adapters` and the endpoint's bearer token. `{version}` and `{data}` are substituted, `RATCHET_ADAPTER_ROOT` moves the path, no sync command means no sync, a failed job stops before loading, and an empty `student_base` raises. `deploy/train_job.sh` mounts the bucket at `/data`, reads `/data/night/train.jsonl` and writes `/data/adapters/${VERSION}`, the paths the trainer uses | | `test_load_adapter_tolerates_already_loaded_and_raises_otherwise` |: the vLLM root is the base URL without `/v1` (with or without a trailing slash, or no `/v1` at all); no token, no header; a 400 saying "already" (any case) is tolerated; any other 400, a 401, a 404 (runtime LoRA loading off) and a 500 raise | | `test_bom_copies_never_leave_the_data_folder` |: an adapter name that is a path (`../../escaped`, an `s3://` URL, `..`) gets no copy beside weights (a message says so) but still its `boms/student-vN.json`, which is still uploaded; nothing is written outside `RATCHET_DATA_DIR`; a version without a BOM writes nothing | | `test_reload_adapters_loads_live_canary_shadow` |: only `live`, `canary` and `shadow` adapters are reloaded, in version order, from `RATCHET_ADAPTER_ROOT`, with the bearer token; one already loaded is skipped over; nothing to reload makes no request; a 503 raises | | `test_live_main_tick_reload_night_offline` |: `python -m ratchet.live tick`, `reload` and `night` against a ledger file with 60 train and 6 golden examples. The night records its training job as a closed `training_job` compute interval at the default GPU rate, linked to the version; a night that skips records none. `night` builds the student and judge clients with the configured bases and keys, the judge with the teacher's timeouts and retries and the student with its own timeouts and the teacher's retries (no teacher stands behind it in the eval), runs the training command, hot-loads `student-v1`, samples it on every golden row as in production, asks the judge in JSON mode about the 3 held-out answers it would serve (`judge_n` 3), admits it to canary and writes `data/boms/student-v1.json`, equal to the stored BOM and signed with `RATCHET_BOM_SIGNING_KEY`, and a copy in `data/adapters/student-v1/ratchet_bom.json`, which a second `RATCHET_SYNC_CMD` run uploads (the first, before training, saw none); the BOM's training commitment covers the 60 rows and their leaves are in the ledger. `reload` then loads the canary; no command means `night`, which skips while the release is in flight (no sync, no BOM); the next trained night whose upload fails keeps both copies locally, keeps the night's BOM, and exits non-zero naming `RATCHET_SYNC_CMD`; an unknown command exits with a message | | `test_live_runtime_wiring` |: the gateway's live `Runtime` with `transports`, over HTTP. A confident student answers from the adapter named by the version, with the endpoint token and `STUDENT_SAMPLING`; an unsure one escalates to the teacher with the Token Factory key and model, and the answer is stored. A 500 from the student falls back to the teacher (`student_error` event); a 500 from the teacher is a 502 `teacher_unavailable` after two retries (waits injected). Without `student_base`, the hosted Nano on Token Factory answers under the global threshold, not the version's. The mock warm-up never runs in live mode, and an unknown transport key is refused | | `test_stats_student_serving_kinds` |: `/ratchet/stats` `student_serving` per mode: `mock` (with the mock fleet's `student-vN` adapter, never the fallback id); live without `RATCHET_STUDENT_BASE`, `none` on a cold start, then `hosted_fallback` with the fallback model, the global threshold and `certified: false`; live with it, `endpoint` naming a cold start's canary, then the live version's adapter, its calibrated threshold and its certificate. Reading the stats calls no model | | `test_hosted_fallback_is_never_certified` |: a live version whose eval carries a certifying certificate is not `certified` while the hosted fallback serves, and its threshold is `Settings.confidence_threshold` (what `Ratchet.threshold` holds the fallback to); behind an endpoint it is `certified` with the version's threshold, and not once its certificate stops certifying | | `test_hosted_fallback_response_names_the_fallback_model` |: a live chat request the hosted fallback answers comes back with `ratchet.served_by == "student"` and `model` equal to the fallback id, the same id `/ratchet/stats` and `/scout/status` report as `fallback_student`; a Scout step built from the same result carries it | | `test_scout_status_has_fallback_student` |: `/scout/status` keeps the guard's keys and adds `fallback_student` and the same `student_serving` as `/ratchet/stats` | | `test_live_evaluator_uses_production_sampling` |: one cascade call per golden row with `n = samples`, exactly `STUDENT_SAMPLING` (read-only) and the student's token cap as `params` (rows without stored conditions), and no other student call: the judge's candidate is the answer chosen from those samples. The judge, in JSON mode with request, reference and candidate, grades only the free-text answer the student would serve on the held-out half (`judge_n`, `judge_pass_rate`, per task too, never for labels); a judge answer that is not a clear `{"score": 1}` (not JSON, a list, a string or 2, no score) scores 0, while fences and reasoning around valid JSON are fine; no judge or no free-text rows means no judge calls | | `test_live_evaluator_judge_pass_rate_covers_free_text_rows_only` |: tickets, a similarity task with `policy.samples` 5, the default task and a pinned task: each row sampled as its task (the pinned one never), the judge sees exactly the four held-out free-text answers that would be served, reasoning stripped, in row order; `judge_n` 4 and `judge_pass_rate` 0.75 overall, per task 2/1.0 and 2/0.5, none for tickets; without per-task records the same rows at the global threshold | | `test_production_and_both_evaluators_share_student_sampling` |: `Ratchet._sample`, `LiveEvaluator.evaluate` and `MockFleet.evaluate` send the same sampling arguments, `{"n": 3, **STUDENT_SAMPLING, "params": GenParams(max_tokens=1024)}` | | `test_train_parse_args_defaults_and_row_contract` |: `train.py` defaults (epochs, learning rate, rank, sequence length, batch, Mamba projections in the targets), required `--data` and `--out`, the same base model in `config.py`, `train_job.sh` and `student_endpoint.sh`; rows written by Night School (`dataset.write_training`) load back unchanged (blank lines skipped), pass `train.check_rows`, have the teacher's stored answer as their completion and carry teacher and license; `train.manifest` (saved as `ratchet_manifest.json`) names the base model, the licenses and the chat-template arguments | | `test_scout_cli_over_http` |: `python -m ratchet.scout` against the mock gateway in-process, with an API key: every agent prints a line, `--max-results` reaches search, citations and the cost summary print, every call is logged under the key's name with its task header (one planner and one writer request); the key also comes from `RATCHET_API_KEY`; a wrong key exits 1 with the 401 on stderr | | `test_sim_cli_writes_json_and_table` |: `python -m sim.simulate` (2 days × 200, mixed, `--drift-day 0`) prints one table row per day, the totals, the GPU-hour line with its recommendation (labelled simulation), the success metrics (labelled simulation, one line per metric with its target and verdict), every event and a line saying the run is a simulation with mock models, and writes the JSON results with the documented keys, `cost` included (48 endpoint hours), and `exposure`, `rollbacks`, `served` and `drift` | | `test_sim_cli_compares_release_rules` |: `--compare-release` (with `--promote-evidence` for the sequential run and `--canary-audit-rate`) writes both rules' results with the same requests and prints the comparison header; `compare_table` gives one row per rule for each version either rule rolled back ("-" for audits and day where that rule did not); `--promote-evidence` without `--release sequential`, a canary audit rate of 0 or 1.5 and an unknown rule are usage errors (exit 2) | | `test_mock_warmup_trains_a_student` |: `RATCHET_MOCK_WARMUP_DAYS=1` and `RATCHET_MOCK_WARMUP_REQUESTS=300` give day 1, 300 requests of mixed traffic, a `student-v1` in canary with a BOM and a `warmup` event, also in `/ratchet/stats`; the live gateway reset warms up again on a fresh ledger | | `test_trust_proxy_uses_forwarded_ip_only_when_enabled` |: with `RATCHET_TRUST_PROXY=1` the guard sees the first `X-Forwarded-For` address on `/scout/run` and `/scout/stream` (the peer address when the header is missing or empty); with it off the header is ignored; with it on, two forwarded clients each get their own hourly limit | | `test_day1_check_reports_missing_models_and_logprobs` |: `scripts/day1_check.py` against a fake Token Factory: no key exits 2 without a request; a missing model is reported (with the NVIDIA models that exist) and never called; missing logprobs and unparseable JSON are reported, and each model's `finish_reason` is printed ( stores only `stop` answers); an HTTP 500 fails the check (exit 1); every model request has the bearer key, `logprobs`, JSON mode and `max_tokens` 400; then the teacher gets `max_tokens` 8,192 with a stop, a seed, a temperature and JSON mode, and a 400 there fails the check; then the json_schema probe (`SCHEMA_FORMAT`), whose fenced answer is checked as the gateway checks it; all present exits 0 with "All good.".: the embeddings model is listed (`[MISSING]` otherwise) and embeds two inputs; an error or vectors of different lengths fail the check, `RATCHET_EMBEDDINGS_BASE=off` skips it, and another provider's base is not looked up in Token Factory's list and gets `RATCHET_EMBEDDINGS_KEY` or no key, never the Token Factory key | | `test_day1_check_probes_json_schema` |: the json_schema probe against a fake teacher: an answer inside the schema is `[ok]` (the body sends `SCHEMA_FORMAT`); a 400 refusing the format, an answer outside the schema (`$.category`) or one that is not JSON is a `[warn]` that still passes; a 500 or a body without an answer is `[FAIL]` | | `test_deploy_scripts_parse_and_dry_run` |: `bash -n` on every `deploy/**/*.sh`; `DRY_RUN=1 bash deploy/vm/create_vm.sh` prints the subnet, disk and instance `nebius` commands with the project id and `` in place of the user data, never the private-repo token; a given subnet skips the lookup; a missing project, missing key file, reserved user name or bad preset stops before any command; `backup.sh` exits 0 in mock mode. Skipped where bash is missing | | `test_sim_cli_certificate_flags` |: `--calibration ltt --cert-target 0.9 --cert-delta 0.1` reach every night's certificate and the table's certificates block (simulation); `--paired-gate` and `--min-certified` reach the gate's policy; bad values exit 2 | | `test_sim_db_and_the_replay_cli` |: `python -m sim.simulate --db` writes the run's ledger to a new file and refuses an existing one and `--compare-release` (exit 2); `python -m ratchet.replay --db` prints one table per version (never pooled), labelled as simulation, with its suggestion, and `--json` with `--audit-rate`; a missing file, a bad sweep, target, audit rate, version or day range exit 2 with `error:` and create nothing | | `test_live_evaluator_passes_its_certificate_settings` |: `LiveEvaluator(calibration='ltt', cert_target=0.6, cert_delta=0.2)` writes that certificate with its trace; the default is `point` | | `test_train_job_uses_job_create` | (W0): with a fake `nebius` on `PATH` (`FakeNebius`), `deploy/train_job.sh student-v3` makes exactly one call, `ai job create` (never the interactive `ai create`), with `--name ratchet-student-v3`, the image, `--platform gpu-l40s-a --preset 1gpu-8vcpu-32gb`, `--volume :/data`, `--disk-size 200Gi`, `--timeout 2h` and `--container-command bash`; `--args` splits (as a shell would) into `-c` and the training command with the dataset and adapter paths `ratchet.live` uses; `--subnet-id` and `--registry-secret` appear only when set; `DRY_RUN=1` adds `--dry-run` and returns without polling; `DRY_RUN=true`, a version or base model that would break out of `bash -c '...'` and a missing image stop before any call; a refused job fails the script. Skipped where bash is missing | | `test_endpoint_stop_and_start_use_the_id` | (W0): `stop` and `start` call `ai endpoint list --format json`, then `ai endpoint stop\|start --id `; `start` is the default; `RATCHET_ENDPOINT_ID` skips the lookup; with no endpoint of that name `start` creates one and `stop` exits 1; nothing to do in the state already reached; `status` prints ` ` (by name, or by `RATCHET_ENDPOINT_ID`); `DRY_RUN=1` prints start and stop instead of running them; a start or stop that nebius refuses fails the script. A stub `ratchet.cost` shows exactly which uptime records were made (`up` after create or start, `down` after stop, none in a dry run or after a refused call). Skipped where bash is missing | | `test_endpoint_lookup_never_guesses` | (W0): `create` refuses an existing name; output that is not JSON, `items` that is not a list, two endpoints with the name, an id that is empty or has a space, a name missing from a list with more pages, or a failing `list` stop `start` (and `stop` and `create` where it matters) before anything else runs; an empty list printed as nothing, `{}`, `{"items": null}`, `[]` and the like means `start` creates; `STOPPING` and `DELETING` refuse, a lower-case, numeric, missing or `ERROR` state lets Nebius decide; an unknown action or `DRY_RUN` value and a missing Python stop first. Skipped where bash is missing | | `test_endpoint_args_carry_the_model_card_flags` | (W0): `create`'s flags (image `vllm/vllm-openai:v0.18.0-cu130`, platform, preset, port 8000, `--shm-size 16Gi`, the bucket volume, token auth, runtime LoRA) and its vLLM arguments in order, split as the container does: the model card's `--mamba_ssm_cache_dtype float32`, `--max-model-len 8192`, `--max-num-seqs 64` and `--default-chat-template-kwargs` with JSON that parses to `{"enable_thinking": false}`; the token never printed; `RATCHET_VLLM_IMAGE`, `RATCHET_SERVE_MAX_LEN`, `RATCHET_SERVE_MAX_SEQS`, `RATCHET_REASONING_PARSER_PLUGIN` (adds `--reasoning-parser nano_v3`), `RATCHET_SUBNET_ID` and an empty `RATCHET_SERVE_TEMPLATE_KWARGS` take effect; `DRY_RUN=1` adds `--dry-run` and records nothing; values that would break the one `--args` string or that vLLM would refuse after the GPU is up (not a whole number, not a JSON object, a relative plugin path, a single quote) and a missing token or bucket stop before `create`. Skipped where bash is missing | | `test_deploy_scripts_never_say_cost_nothing` | (W0): neither deploy script, `deploy/AGENTS.md` nor `docs/DEPLOYMENT.md` says a stopped endpoint costs nothing (the mounted bucket is still billed); the endpoint script says so | | `test_deploy_variables_are_listed_and_documented` | (W0): every `RATCHET_*` variable the two Nebius scripts read is read by `config.py` or listed in `config.DEPLOY_ENV` (part of `KNOWN_ENV`, which the traceability check holds to a row in CONFIGURATION.md), with no stale or duplicate name; all of them and `DRY_RUN` are documented | | `test_liverun_plan_refuses_over_budget` |: `python -m ratchet.liverun plan --max-usd 0.01` exits 3 and makes no call at all; the plan lists requests per release stage at the configured canary audit rate, 0.5 and 1.0, the audit rates as set, the gate profile and whether it is in force, the golden-size line and a USD row per round; at the default rates the 10% canary stage needs 7,398 requests against 737 at 0.5; `plan_stage_requests` equals a brute-force scan with `binom_sf` | | `test_liverun_refuses_mock_gateway_and_wrong_ledger` |: a mock gateway exits 2 after `/healthz` alone (no live gateway route); `RATCHET_DB` on another file than the gateway's exits 2 after the one probe; a gateway without `RATCHET_API_KEYS` exits 2 naming `liverun:`; no `RATCHET_LIVERUN_KEY`, a training command not on PATH or a script that is not there, and no open endpoint interval exit 2 before any paid call | | `test_liverun_end_to_end_offline` |: the module run (`liverun_world`: a live `Runtime` on Token Factory and vLLM fakes, 3 rounds of 100 tickets, nights in process, a 50% control sample, the untuned arm): exit 0, two versions trained through `live.main`'s seams (canary, then shadow), only `/healthz` and `/v1/chat/completions` requested, every request with its `liverun--r-` id and the harness's key, `results.json` with the documented keys and `live.*` values each with a source, the three `liverun_*` events, the models as listed, no key or base URL in the settings, client and server-side latency, the cost total as its parts, LIVE_RESULTS.md with claim anchors | | `test_liverun_rounds_are_keyed_by_client_and_id_range` |: a request with another key between two batches of round 2 falls inside round 2's id range, is counted as `other_clients` and is left out of its counts | | `test_liverun_writes_no_text_and_no_key` |: no file in the run directory and nothing on stdout holds a key, a base URL or any 20-character piece of a request or answer the fakes saw; the `liverun_*` events neither | | `test_liverun_control_arm_is_never_stored` |: the ledger has one row per cascade request (none for the control calls); the same rounds with `--control-share 0` on a fresh world store exactly as many training and golden examples | | `test_liverun_control_call_matches_the_serving_teacher` |: for escalated requests the control arm's body equals the gateway's teacher call (model, messages, `max_tokens` cap, temperature, no logprobs, `n` 1), with the Token Factory key | | `test_liverun_control_ratio_and_reasoning_tokens` |: control reasoning tokens average 420 beside the formula's 300; `ratio_student` equals `stats.ratio_estimate` on the pairs read back from `requests.ndjson.gz` and is above 1; teacher-served pairs give 1; the measured reduction and its interval; nine student-served pairs are `insufficient` and leave the value out, ten give the exact ratio | | `test_liverun_accuracy_against_template_labels` |: with a teacher wrong on hard templates and invalid on one, cascade accuracy is 100% on easy and below on hard, invalid answers count as wrong, category and priority are reported apart, the control arm's accuracy too, labelled "template labels"; a cold start without nights has no student pairs, so no measured baseline | | `test_liverun_usd_cap_stops_between_batches` |: a teacher reporting 200,000 output tokens per call stops the run with `max_usd` (exit 1) after one batch, spend within the cap plus one batch, the control arm's own cap skipping the rest of its sample, results and state complete | | `test_liverun_server_cap_and_rate_limit_responses` |: a 429 `rate_limit_exceeded` waits its `Retry-After` and retries once; a 429 `budget_exceeded` stops with `server_cap`; a 401 with `auth`; three 503 `teacher_circuit_open` in a row with `teacher_down`; failures above 10% of a batch with `gateway_errors` | | `test_liverun_night_failure_stops_with_partial_results` |: a failing training command records `night_failed` (the class, never its output), exit 1, round 1's results written and the training interval closed; resumed once fixed, the night runs again and trains before round 2, the failure kept on record; `--on-night-failure continue` runs round 2 on the teacher | | `test_liverun_until_settled_extends_the_round` |: a canary still in flight after a round's requests gets extension batches until it settles; `--max-extend 0` records `settled: false`, and the next night skips by itself ("still in canary") | | `test_liverun_resume_continues_the_stream` |: a host that goes away after round 1's second batch stops the run; `run --resume` (with the base URL again: it is never written) sends every remaining index with the same id and template as an uninterrupted run, never one index twice, and only the lost batch is missing | | `test_window_percentiles_filter_by_route_client_and_ids` |: `Ledger.last_ids`, `client_window` (one client's rows in an id range, `other_clients` and `no_client`), `events_since`, and the per-route nearest-rank percentiles of `liverun.window_stats` on a hand-built ledger | | `test_liverun_report_bill_publish_and_verify` |: `report --bill-usd 12.34 --bill-note ...` sets `live.bill.nebius_usd`, redacts the note and shows the bill next to the break-even line; bad amounts and notes exit 2; `verify` exits 0, and 1 after one edited number or a missing source; `--publish` refuses a stopped run without `--allow-partial` and a smoke run always | | `test_live_targets_stay_out_of_test_and_ci` |: the Makefile's `test` and `lint` recipes and the CI workflows never mention `live-smoke`, `live-run` or `ratchet.liverun` | | `test_liverun_untuned_arm_paired` |: the hosted Nano (unsure on hard templates) covers less than the tuned adapter on the same golden rows; `arms.untuned.paired` equals `certificate.paired` of the two test vectors; the arm's calls went straight to Token Factory | | `test_liverun_scout_phase` |: `--scout-runs 1` with Tavily on a `MockTransport` runs Scout over `/v1` with the harness's key, one request id per agent call, its totals and Tavily credits recorded; no key or question text written; without `TAVILY_API_KEY` it is refused before any call | | `test_liverun_file_workload` |: a JSONL workload with three mislabelled rows scores 9 of 12 against the file's labels (priority 12 of 12), sends every row once in a pass, and a bad row names its line and field, never its text | | `test_liverun_gate_profile_and_the_arms_key` |: `run` without the planned gate profile exits 2 before any call; `--gate-profile as-set` runs and records which settings were in force; `NEBIUS_API_KEY_LIVERUN` carries the arms' calls while the gateway keeps `NEBIUS_API_KEY`, and is never written | | `test_liverun_control_budget_is_reserved_across_workers` |: eight workers reserving $1 against a $3 control cap at once: exactly three win | | `test_liverun_repeat_run_and_report` |: `smoke --repeat-of` runs the same plan and seed under a new run id (same texts, same order) and compares every value with the first run's; `report --repeat --publish` attaches the comparison and copies the repeat to `evidence/live/repeat/`, both verify, an edited repeat table fails `verify`; a smoke repeats only a smoke, a repeat of the wrong run is refused, `--resume` with `--repeat-of` exits 2 | | `test_caddy_guard_covers_every_paid_route` |: every gateway route under the OpenAI-compatible prefixes (`/v1/` and the Azure-style `/openai/`, `gateway.API_PREFIXES`) and every mock-mode gateway control matches a path the VM Caddyfile's `guarded` block protects, so `RATCHET_API_GUARD=guarded` keeps the whole paid API behind the admin token; each prefix is guarded as a whole, so a later route under it is covered too | | `test_a_night_after_an_endpoint_restart_reloads_the_adapters_before_it_scores_the_champion` |: `live.main(['night'])` against an endpoint that forgot its adapters (vLLM v0.18.0's 404s): the live adapter is loaded first, then the new one; the champion's re-score samples succeed; an `adapter_reloaded` event says `before the night` | | `test_a_failed_reload_before_the_night_is_an_event_and_the_evaluator_reloads_once_on_a_404` |: a 503 from the load endpoint is an `adapter_reload_failed` event with the error class and no key or URL; `LiveEvaluator` loads a forgotten adapter once per evaluation and asks again, and without the hook the 404 ends the evaluation | | `test_cloud_init_takes_an_ssh_key_with_quotes_and_waits_for_the_apt_lock` |: `create_vm.sh`'s substitution renders a cloud-init that PyYAML reads back with keys whose comments hold `'`, `"` and `''` unchanged; both apt calls carry `DPkg::Lock::Timeout=600` | | `test_update_names_the_settings_the_env_file_lacks_and_the_example_sets_prices_verified` |: `env_missing.sh` prints only the names `.env.vm.example` lists that `.env` lacks (a commented-out line is not a setting, no value is printed), nothing for a complete file; `update.sh` calls it; the example sets `RATCHET_PRICES_VERIFIED=0` | The React app itself is checked with `npx tsc --noEmit`, `npm test` (pure helpers in `web/tests/*.test.mjs` under Node's test runner, such as the Economics panel's axis labels and scales and the story page's honest bill) and a production build in CI (`web` job), and by Playwright screenshots during development (light, dark, 1440 and 390 px). During development the v0.4 features were also checked end to end against a running gateway with the official OpenAI Python SDK (401 on a wrong key, streaming text identical to the non-streaming answer, the quota cutting one key off at its limit while another kept working, a signed BOM verifying with `python -m ratchet.bom verify` and failing once edited), and the dashboard's Trust and privacy panel with Playwright in light and dark mode at desktop and phone widths. The dashboard's Economics panel was checked the same way at 1360 and 375 px against the mock gateway, and against crafted `/ratchet/cost` bodies routed in the browser (self-host with 90 days, scale to zero with warnings and an open job, not enough data in live mode, an empty report, a 404 that hides the panel, a 500 and a malformed body that fall back to the `/ratchet/stats` summary), with no horizontal scroll on phones; a refresh that fails keeps the last report with a notice, and a test run day updates it. The Task registry and Resilience panels were checked the same way at 1360 and 375 px: the outage drill run end to end in the browser (the countdown, the breaker opening during a day of traffic, the probe closing it the next day, the release log's breaker entries); the task editor's check, a refused spec with each error, bad JSON caught before sending, save, a no-change save, a copy of a built-in task and delete against the mock gateway; and a gateway with `RATCHET_ADMIN_KEYS`, where a save is a 401 without the key and recorded as `ops` with it, with nothing written to browser storage or cookies; no horizontal scroll on phones. The story page's honest bill and Built to be checked chapters were checked with Playwright in light and dark mode at 1360 and 375 px (full-page screenshots, no horizontal scroll), against the mock gateway (its own cost report, labelled Simulation (mock models)) and with every `/ratchet/*` call failing (the reference run, labelled Simulation); the source toggle, reduced motion (every bar and marker drawn at its final place), and the link to the dashboard's redactor, which lands on it both after an in-app navigation and on a direct load of `/dashboard#redactor`, also while panels above it are still loading. ### `tests/test_cost.py` Utilization-aware cost accounting. Hand-built ledgers run on the synthetic clock (day N starts at N × 86400 s) and every number is worked out in the test; the simulated days assert signs and trends only. A module fixture runs five simulated days of 1,500 tickets on a mock `Runtime` once (about 4 s). | Test | Covers | |---|---| | `test_split_is_exact_on_every_cost_path` |: the 11 cost paths of the contract test, each row's student part (live and shadow samples), teacher part (escalation, audit, arbitration, pinned), `cost_usd` equal to their sum and to the contract test's exact values, and the student tokens | | `test_pinned_and_teacher_only_requests_are_all_teacher_cost` |: a pinned task and a teacher-only request: no student call, every cent the teacher's, zero student tokens | | `test_split_holds_on_every_row_of_simulated_days` |: 7,500 simulated requests with canaries, shadows, audits and arbitrations: every row's parts add up (within 1e-12), one 24-hour endpoint interval per day and one training interval per trained version (600 s + 0.05 s per example from 22:00) | | `test_overlapping_and_nested_intervals_are_not_double_counted` |: nested, overlapping and separate intervals merge into hours once; a pricier overlapping record bills only its stretch at its rate; clipping to a window; empty records count nothing; the per-day split adds up | | `test_open_interval_counts_to_now_and_to_the_window` |: an open interval runs to `now`, never past the window, never negative; listed in `open_intervals` | | `test_endpoint_start_is_idempotent_and_stop_closes_it` |: a second `up` (also from a second connection to the file) returns the open interval; an end before the start is refused and closes nothing; `down` while down is None; jobs get an interval per start and close by id with their version and note | | `test_concurrent_endpoint_starts_keep_one_open_interval` |: eight connections to one file starting the endpoint at once leave one interval (only the write transaction can serialise them; it fails without `BEGIN IMMEDIATE`) | | `test_bad_compute_records_are_refused` |: an unknown kind, a negative or NaN rate, `True` or negative GPUs, an infinite time, a rate above $1,000 per GPU-hour or too large for a float, more than 1,024 GPUs, a closed record that ends before it starts (11 cases) | | `test_training_interval_is_closed_when_train_raises` |: `run_night(train_gpu=...)` closes the interval at the failure time with a note and re-raises; a successful night records the job's duration and version; without a rate it records nothing | | `test_report_math_on_a_hand_built_ledger` |: every total, both views (a negative GPU-hour saving, not clamped), s̄, break-even, the hosted-fallback price and crossover, observed volume, utilization, idle hours, the `hosted_fallback` recommendation, the daily rows, the headline following `RATCHET_STUDENT_COST_MODEL`, and the window arguments | | `test_recommendation_for_each_branch` |: `self_host` ("Self-hosting pays." with positive GPU-hour savings), `hosted_fallback`, `scale_to_zero` (idle days, and below break-even with a pricier hosted fallback), `insufficient_data` (no uptime; uptime but no student answer), an empty ledger with no division by zero or NaN | | `test_explanation_between_break_even_and_the_hosted_crossover` |: above break-even and below the crossover the recommendation is `hosted_fallback` and the explanation says self-hosting costs less than the teacher with the GPU-hour savings, the hosted fallback's crossover and the equal-coverage caveat, never "does not pay" or "below the break-even"; the same with the endpoint idle most hours (`scale_to_zero`) | | `test_break_even_uses_the_recorded_rate_and_never_says_pays_at_a_loss` |: break-even and the crossover at the rate the intervals recorded (2 GPUs against a setting of 1); an endpoint above break-even with a training job that leaves the window at a loss is kept without the words "Self-hosting pays" | | `test_live_ledger_days_count_from_the_first_request` |: a live ledger's days start at its first request; uptime before it is day 0; idle time in hours (49 of 51); a one-day window takes that day's hours; a reopened file reports the same days | | `test_live_ledger_counts_observed_volume_and_idle_time_by_hour` |: an endpoint up for one hour is credited with that hour's answers only, the later ones are priced per token as the hosted fallback's (with a warning), the daily row matches the totals, and a 24/7 endpoint with one request a day is told to scale to zero | | `test_last_days_before_the_first_live_request_keep_recorded_uptime` |: before the first request, `days=1` and `days=7` keep the 5 hours the endpoint has run; the first request then starts day 1 | | `test_simulated_day_leaves_an_endpoint_opened_through_the_api_alone` |: a simulated day writes its own closed interval, so an endpoint a visitor marked up after the day's simulated hours stays open (no error), and an overlapping one is counted once | | `test_student_answers_without_recorded_uptime_are_priced_per_token` |: student spend on a day with no recorded uptime stays in the GPU-hour view at its per-token price, stays out of the observed volume, and gets a warning | | `test_report_warnings_and_daily_cap` |: a training job open for 30 hours is named with the command that closes it; the daily breakdown keeps the latest `MAX_DAILY` days while the totals keep every day | | `test_mock_demo_with_a_24h_endpoint_does_not_pay` |: five simulated days of 1,500 requests: per-token savings positive, GPU-hour savings negative, never `self_host`, observed below break-even; `/ratchet/stats` `cost` and `totals.cost_model`; `?days=1`; 400 and 422 for bad `days` | | `test_simulated_endpoint_hours_follow_the_setting` |: `RATCHET_MOCK_ENDPOINT_HOURS=8` records 8-hour days without events; `0` records none and the report says `insufficient_data` | | `test_post_endpoint_records_uptime_with_an_admin_key` |: 401 without a key, 403 for an API key, 400 for each bad body (12, including rates above $1,000, 1e308 and a 401-digit integer, which was a 500), `up` with rate, GPUs and note, `up` again unchanged, `down`, `down` again; `endpoint` events with the admin's name and no key anywhere | | `test_post_endpoint_is_open_in_mock_capped_and_off_in_live` |: open in mock mode as `mock-admin` on the synthetic clock, the mock cap (409, never for `down`), 403 in live mode without admin keys, and the report public in live mode | | `test_cli_records_and_reports` |: `endpoint up` (with `--at` and `--gpus`), `up` again, `down` with an ISO time, `report` and `report --json` (the second `up` adds nothing), `--days 1` meaning today, `close training_job`; exit 2 for a missing ledger (never created), a bad or future `--at`, an end before the start, a rate above the cap or too large for a float, too many GPUs, `--days 0` and a bad setting | | `test_cli_runs_as_a_module_with_the_environment` |: `python -m ratchet.cost report` in a subprocess with `RATCHET_DB` | | `test_deploy_script_records_uptime_and_never_fails_on_it` |: `deploy/student_endpoint.sh` start (creating the endpoint, since the fake `nebius` lists none) and stop (by the id the list gives) with `FakeNebius` record the endpoint in a real ledger through the CLI; with no ledger the deploy still exits 0 and says so; an unknown action exits 2. Skipped where bash is missing | | `test_v04_database_gets_the_cost_columns_and_table` |: a v0.4 file gains the split and token columns and `compute_intervals`; its old rows count as teacher spend (`split_unknown_requests`, a warning) and give no hosted-fallback price | | `test_bad_cost_settings_stop_startup_naming_the_variable` |: a negative, non-numeric, NaN or 1e308 rate, fractional, negative or 4,096 GPUs, an unknown cost model and 25 endpoint hours a day each stop the runtime with the variable's name (9 cases) | | `test_cost_settings_defaults_and_spellings` |: the defaults, a cost model in any case with spaces, `RATCHET_PRICES_VERIFIED=1` changing the label, an empty model meaning `per_token`, and `--at` parsing | ### `tests/test_stats.py` The exact statistics of `ratchet/stats.py`. Monte Carlo checks are seeded, counts-only and sized to run in under a second; the two marked slow repeat them at full size. | Test | Covers | |---|---| | `test_binomial_tails_match_brute_force` | `binom_sf`, `binom_cdf` and `binom_logpmf` equal sums of `math.comb` terms for every k of small n, edge probabilities 0 and 1 and 60 random cases at n = 120 | | `test_binomial_tails_keep_tiny_values_and_refuse_bad_input` | 0.5^100 and 0.1^200 to relative 1e-12 and 1e-9 (summed directly, not as 1 minus almost 1); bad counts and probabilities raise | | `test_clopper_pearson_bounds` | 46/48 → 0.8746, 129/130 → 0.9640, 127/128 → 0.9635; s/s → delta^(1/s); for 50 random (s, n) the bound is where P(X ≥ s) = delta, from below, and the upper bound mirrors it | | `test_precision_pvalue_planner_and_mcnemar` | `required_rows`: 0.95 → 59, 93, 124, 153 rows for 0 to 3 errors; 0.97 → 99, 157, 208, 257; 0.90 → 29, 46, 61, 76; the p-value falls as agreement rises; McNemar 4 vs 0 is 0.0625 | | `test_ltt_fixed_sequence_stops_at_the_first_failure` | 10 correct points certify nothing (1.01); 80 correct at 0.97 and 40 wrong at 0.72 certify 0.75 and stop at 0.70 with its trace; grid order does not matter; bad grids raise | | `test_ltt_controls_the_error_rate` | Confidences U(0,1) with P(correct) = c: over 300 seeded draws of 400 rows, the chosen threshold has true precision below the 0.80 target in at most 5% (+3 SE) of draws | | `test_e_value_oracles_and_confidence_sequence_oracles` | e(0/2 below 0.92) = 29.77, e(0/1) = 4.22, e(20/20) = 0.2006, e(7/17) = 2.59e5, e(60/60 above) = 27.6, e(83/83 above) = 129.7, within 1%; CS(0/2) = (0.001, 0.931) and CS(35/38) = (0.710, 0.992) within 0.002 | | `test_alt_grid_components_are_supermartingales` | For p0 in 0.5, 0.8, 0.92, 0.97 and every true rate on the null side, each alternative's one-trial mean factor is at most 1 (the validity argument, checked exactly); weights sum to 1, also near the edges | | `test_e_values_edges_and_input_checks` | No trials: e = 1; a huge count stays finite (capped at exp(690)); the confidence sequence narrows with data; bad sides, probabilities and steps raise; the cached confidence sequence equals a fresh scan and still refuses bad counts (a bool, a list, a float, s > n) | | `test_e_process_false_rollback_rate_and_wilson_peeking` | At a true rate on the 0.92 floor, checked after every audit (120 runs of 200): the e-process rolls back at most 5% (+3 SE); Wilson's bound checked the same way does more than 10% (about 23%, simulation) | | `test_confidence_sequence_covers_the_truth_at_every_step` | At p = 0.95, the true rate leaves the 95% confidence sequence at any step in at most 5% (+3 SE) of 120 runs | | `test_e_process_and_confidence_sequence_error_rates_full` (slow) | The two checks above at 300 runs of 300 audits, at 0.92, 0.95 and 0.80 | | `test_clopper_pearson_coverage_full` (slow) | P(cp_lower ≤ p) ≥ 0.95 (−3 SE) for (20, 0.90), (50, 0.95), (100, 0.97) and (130, 0.99), 2,000 draws each | | `test_wilson_lives_in_stats_and_release_still_exports_it` | `release.wilson` is `stats.wilson`; 18/20 → (0.699, 0.972) | | `test_kish_and_hajek` | Kish's n_eff of equal and unequal weights; the Hajek mean of a hand-computed example; empty or zero weights give None; bad input raises | | `test_ratio_estimate` | R = 1.3 recovered from 400 synthetic pairs; the ratio and standard error equal the textbook formulas; equal weights change nothing; weighted case by hand; too few pairs or no x are `insufficient` (ratio 1.0, se None) | | `test_ratio_estimate_interval_covers_the_truth` | R ± 1.96 se covers 1.3 in at least 90% of 200 seeded samples of 150 rows | | `test_certificate_on_the_held_out_half_holds_at_its_rate` |: confidences U(0,1) with P(correct) = c, the threshold chosen on 200 calibration points by the point rule and certified at 0.80 on 200 fresh ones: over 300 seeded draws a certificate claims more than the truth in at most 5% (+3 SE), and some draws do certify | | `test_stats_cli_plans_bounds_and_tests` |: `plan` prints `0 errors: 59 rows, 1: 93, 2: 124, 3: 153`, `bound` 0.9635 and 0.8746, `mcnemar` 0.0625 and 1; impossible counts and a target of 1.5 exit 2; `python -m ratchet.stats` in a subprocess | ### `tests/test_provenance.py` Merkle trees and the BOM v2 scaffold, and committed training sets. | Test | Covers | |---|---| | `test_keyset_roots` | The empty set's root is 32 zero bytes, one key's root is its leaf hash, hex and bytes agree, and the root depends on the set, not the order | | `test_keyset_proves_members_and_non_members` | 1,000 random members verify `included` (with their value) and 1,000 random non-members `absent` | | `test_keyset_agrees_with_a_brute_force_set` | Sizes 0 to 40 over keys that share long prefixes: every key of a 120-key universe verifies `included` exactly when it is in the set | | `test_tampered_proofs_never_prove_inclusion` | Flipping a byte of any sibling or of the terminal value makes a proof `invalid`; another key or an absent key never reads as `included`; an inclusion proof never reads as `absent` | | `test_malformed_and_overlong_proofs_are_invalid` | A terminal leaf without the key's prefix, 257 siblings, missing fields, bad hex and wrong lengths are `invalid`, never an error; duplicate or short keys raise when building | | `test_keyset_builds_and_proves_quickly` | 2,500 keys built and 200 proofs in well under 0.5 s | | `test_log_tree_matches_rfc_vectors` | RFC 9162: the empty tree is sha256(''), the leaf hash of '' is 6e340b9c..., the Certificate Transparency 8-leaf vector's root is 5dc9da79..., and every prefix matches an independent implementation | | `test_every_inclusion_path_verifies_and_alterations_fail` | Sizes 1 to 64: every audit path verifies (bytes and hex); an altered leaf or index, another tree's root, a size the path cannot fit or an extra node fails; malformed input is False | | `test_bom_v2_manifest_sections` | Schema v2 with every v1 field, `redactor`, `training.origins`; commitments only when given (`ratchet.smt/v1`, size checked against the rows); `EXTRA_KEYS` merged; unknown sections, a wrong size and a bad root refused | | `test_bom_verify_accepts_v1_and_v2_and_refuses_unknown_schemas` | Signed v2 and v1 documents verify with the public key, an unsigned v1 document by its hash; an unknown schema and junk are refused | | `test_rows_report_their_origin` | `dataset.rows` reports `examples.origin`, or `live` when it is not recorded | | `test_night_writes_a_v2_bom_with_its_redaction_setting` | `run_night(redact_pii=False)` writes a verifiable v2 BOM with `redactor.redact_pii` false and the origins of its 60 rows | | `test_mock_gateway_bom_records_the_redaction_setting` | A mock `Runtime` with `redact_pii=False` writes it into its first adapter's BOM | | `test_certificate_endpoint_and_stats_check_on_a_signed_bom` |: on a mock gateway with a signing key, `GET /ratchet/versions/1/certificate` returns every key, the signed certificate, frontier and per-task block, the BOM's hash, the gate's evidence and three assumptions; 404 before training and for an unknown version; `drift` is the monitor's view of the version, with no alarm; a re-score changes `current` and the status, not `training`; a version from before v0.4 (no BOM) and v0.5 (no certificate) answers `no certificate`; `python -m ratchet.stats check` passes the BOM, fails it after `errors` is edited (hash and numbers), after a re-hash (numbers), without the key's signature, and passes a BOM from before v0.5 | | `test_commitment_binds_each_examples_label_teacher_license_and_redactions` |: `dataset.commitment` keys are the example hashes in order; each value is the SHA-256 of a canonical JSON written out independently in the test (non-ASCII kept); a different label, teacher, license or redaction count moves the root; the order of rows does not; the empty set is `EMPTY`; a duplicate key raises | | `test_version_leaves_are_written_once_and_found_by_key` |: `set_version_leaves` writes and reads back in key order, an empty split writes nothing, `versions_containing` finds versions by key and split; a second write to a split, an unknown version, a duplicate key, uppercase or short hex, a null value and an unknown split are refused with nothing written; SQL UPDATE is refused by the trigger | | `test_night_commits_the_exact_rows_it_trained_and_graded_on` |: two nights; each BOM's training and golden commitments equal `dataset.commitment` of exactly the rows the trainer and the evaluator got, size equal to `examples` and to the stored leaves; every golden key is `absent` from the signed training root; an example from night 1 is in both versions' training sets (cumulative retraining), one from night 2 only in the second | | `test_a_failed_training_job_commits_nothing` | A training job that raises leaves no version and no leaves | | `test_request_key_is_the_stored_hash` |: for a plain request and one with a developer message, text parts, PII, `stop`, `response_format` and a temperature, with redaction on and off, `proof.request_key` and `resolve_key` give exactly the hash `Ratchet._store_example` stored | | `test_resolve_key_validates_like_the_gateway` | A request without a task is `default`, normalised like the gateway's (developer role, text parts), `metadata.ratchet_task` and a body `task` (which wins); a key in any case; ten bad bodies refused with their reason; a part only the teacher reads refused because it is never stored | | `test_proofs_and_the_held_out_certificate_through_the_gateway` |: two mock days with a signing key: a v2 BOM whose commitments' sizes equal `examples`; the sample train example (prompt only) is `included` with its stored value and the signed root, size and BOM hash, the sample golden example `absent` from training and `included` in the golden set, by request or by key; both verify with `proof.verify_proof`; the default sample is the live version's (404 before there is one); the held-out certificate verifies with the exact success line; the CLI exits 0 for valid proofs and certificates and 1 for an unexpected result, another key's signature, a tampered sibling, a file that lies about its result, another root, BOM, version or split, an edited golden leaf, a missing proof and a file that is not JSON (2 for a missing file); 404 for an unknown version and `commitment not available` without a BOM, 400 for a bad split or body | | `test_an_edited_leaf_fails_against_the_signed_root` |: with the trigger dropped and one leaf edited, the proof and held-out endpoints answer 500 (twice) and write one `integrity` event; a proof built from the edited leaves verifies against its own root but fails against the BOM's (`root mismatch`, `invalid proof`) | | `test_a_v1_bom_still_verifies_and_has_nothing_to_prove_against` |: a signed v1 BOM verifies; `verify` and `heldout` exit 1 with `no set commitment`; `bom.commitment` is None for junk, another `alg` and a boolean size | | `test_verifiers_never_raise_on_malformed_files` |: 14 malformed proofs (non-list siblings among them, the regression) and 11 malformed certificates each give `FAIL:` lines and no exception; a value in another hex case still verifies | | `test_proof_endpoints_need_an_admin_key_in_live_mode` |: in live mode the proof and held-out endpoints are 401 without a key and 403 for an API key, admitted for an admin key (404: nothing trained), 403 `admin API disabled` without admin keys; the sample endpoint is 403 (mock-only) even for an admin | | `test_cli_key_prints_the_stored_key` |: `python -m ratchet.proof key` prints `request_key` for a messages file (with `--no-redact` too) and for a request body with `metadata.ratchet_task` and `stop`; junk, a missing file, a string and empty messages exit 2 | ### `tests/test_privacy.py` Data governance: storage opt-outs and pseudonymous end users. Its invariants are (opt-outs only reduce storage) and (end-user identifiers are never stored or logged); their proving tests carry those markers. | Test | Covers | |---|---| | `test_store_false_escalation_calls_the_teacher_and_stores_nothing` |: `RequestMeta(storable=False)` on an escalation: the teacher answers, `Result.stored` false, `store_opt_out` `request`, the row `stored` 0 with its reason; the storable request stores, an opted-out duplicate (reason `key`) does not bump it; no `skip_example` event | | `test_store_false_still_audits_and_arbitrates_without_storing` |: with `audit_rate=1` the audit still records `audit_agree` 1 and nothing is stored; a shadow that disagrees is still arbitrated (`shadow_agree` 0) and the arbitration is not stored, while the same disagreement on a storable request stores it | | `test_zero_retention_stores_nothing_for_any_caller` |: `store_examples=False` stores nothing for Scout's call shape (no meta), a request with an end user, or an audited student answer (still audited); every row `global`, `stored` 0 | | `test_zero_retention_mode_night_skips` |: `run_night(store_examples=False)` skips with `storage disabled` before the trainer or the evaluator is called, although 60 stored examples would train; the default trains; the mock `Runtime`'s simulated day under `RATCHET_STORE_EXAMPLES=0` serves 100 requests, stores nothing and skips | | `test_live_night_skips_under_zero_retention` |: `python -m ratchet.live night` with `RATCHET_STORE_EXAMPLES=0` prints `storage disabled`, runs no training command and creates no version | | `test_no_store_tasks_from_the_setting_and_the_spec` |: `RATCHET_NO_STORE_TASKS` (spaces trimmed, unregistered names allowed) and a spec's `store: false` give reason `task`; other tasks are stored; a space-separated list or an over-long name stops startup | | `test_examples_record_client` |: an example keeps its first client and end user; a duplicate from another client adds one `example_clients` row, its own duplicates, internal calls and opted-out requests add none; an example first stored without a client records the first client that sends it again | | `test_no_train_end_users_are_never_stored_even_when_registered_mid_request` |: a registered end user's escalation is served and not stored (`end_user`), another end user's is; a registration made while the request waits for the teacher still keeps it out (the store-time check) and is reported like one found at the start (`store_opt_out` `end_user` in the result and the row, counted in `not_stored_requests`), without changing the caller's `RequestMeta` | | `test_pinned_and_teacher_only_requests_record_the_end_user_and_opt_out` |: pinned and teacher-only rows record the end user's hash and the opt-out, and store nothing | | `test_opt_outs_hold_under_concurrent_requests` |: 80 requests from 8 threads, three quarters opted out three ways (request, key, no-train end user): exactly the 24 storable ones are stored, and every row's `stored` matches its `store_opt_out` | | `test_store_false_over_http_escalation` |: `store: false` over HTTP on an escalation: the student and the teacher are called once each, `ratchet.stored` false and `store_opt_out` `request`, nothing stored; `store: true` stores | | `test_store_header_metadata_and_key_opt_outs` |: `X-Ratchet-Store` `none`, `0` and ` FALSE `, `metadata.ratchet_store` `"false"` and `"0"`, and a key with `store=0` in `RATCHET_KEY_LIMITS` (also with `store: true`, `X-Ratchet-Store: 1` and `ratchet_store: "true"`) store nothing; explicit opt-ins store; any one opt-out wins; an unknown header or metadata value is a 400 before anything is logged; `/ratchet/stats` names the key and counts by reason; an empty `X-Ratchet-Store` is the default (stored), not a 400 | | `test_end_user_is_an_hmac_and_the_raw_value_is_stored_nowhere` |: with redaction off, `user` and `safety_identifier` (which wins) become 16-hex HMACs scoped per key in `requests` and `examples`; an empty `user` is none; a 309-character `user` is accepted and hashed whole (not as its first 128 characters); a non-string `safety_identifier` is a 400 that does not quote it; the raw values, a no-train value and the keys appear in no text value of any table; events carry no hash; `clients[].end_users` and `privacy.no_train_end_users` count | | `test_end_user_hash_survives_reopening_the_file_ledger` |: the generated key survives reopening the file (same hash, no event); another file gives another hash; `RATCHET_END_USER_KEY` gives the same hash in any file; switching keys writes one `end_user_key` event without the key; a short key or a corrupted stored key raises | | `test_after_no_train_registration_escalations_are_not_stored` |: `POST /v1/ratchet/no-train` with an API key registers the scoped hash without returning it (`created` true, then false); that client's end user is no longer stored, another client's same identifier is; 403 for another client, 200 for its own name; an admin registers once for every client (scope `*`), or for a named client (400 for an unknown one); a client cannot remove an admin's registration (`kept`), removes its own (`removed`, and storage resumes); an admin's removal for one client leaves the every-client registration (`kept`), its removal without a client clears it; `absent`; an admin registration over a client's own protects it; `no_train` events name who and never the end user or its hash; 401 without or with a wrong key | | `test_no_train_requests_are_validated` |: missing, non-string and empty `end_user`, an unknown field (`unknown_parameter`), a non-string `client`, a non-object body and bad JSON are 400s in OpenAI's envelope; a 5,000-character `end_user` is accepted and never echoed; a keyless caller cannot name a client (403), registers in the keyless scope and gets `created` and `status` `null` either way; the mock cap (409) counts new registrations only, and a refused batch writes nothing | | `test_a_no_train_registration_holds_whichever_identifier_a_request_sends` |: registered by `user`, a request that sends `user` and `safety_identifier` is still not stored; registered by `safety_identifier`, likewise; an admin's every-client registration by `user` holds for a request with both; another client's same values are stored; rows hold the `safety_identifier` hash and no alias | | `test_no_train_registrations_across_key_renames_and_keys_added_later` |: a live file ledger across five startups: keyless registrations become unmatched once keys are on; an admin's every-client registration covers a key added later and a renamed one; a client's registration lapses when its key is renamed, and a `no_train_unmatched` event (names and counts only, once per change) and `privacy.no_train_unmatched` say so; the name back restores it; an admin's removal without a client reaches the unmatched keyless scope | | `test_removing_a_registration_needs_a_key_in_live_mode` |: a live gateway without API keys registers for anyone but removes only for an admin key (403 otherwise) | | `test_zero_retention_still_rescores_a_task_whose_spec_changed` |: under `store_examples=False` the night skips (`storage disabled`, no training) but re-scores the live version on the stored golden rows of a task whose spec changed (new `spec_hashes`, a `rescored` event, no example written); nothing to re-score the next night; no re-score while a release is in flight | | `test_every_add_example_caller_honours_storage_opt_outs` |: every function in's `ALLOWED_ADD_EXAMPLE_CALLERS` calls `store_opt_out` before `add_example`, so a writer added at a merge (a backfill) must honour opt-outs | | `test_metadata_is_echoed_and_the_ratchet_block_says_whether_it_was_stored` |: `metadata` echoed in the response, on the stream's `ratchet` chunk (the finish chunk, or the usage chunk with `include_usage`) and nowhere else, absent when not sent, never stored | | `test_privacy_stats_report_opt_outs_and_clients` |: `/ratchet/stats` privacy keys on a fresh gateway (zero counts, `no_store_tasks` from the setting and specs, `no_store_clients`), `not_stored_by_reason` after traffic, `clients[].end_users` and `stored_share` (3 of 5), and `store_examples` false and `end_user_key` `env` when set | | `test_key_limits_parse_and_refuse_typos` |: `RATCHET_KEY_LIMITS` parsing (spaces, empty fields, several keys); an unknown name (not quoted, in case it is a key), an unknown field, a bad `store`, a key or field twice and malformed entries raise; with keys off every name is unknown | | `test_storage_settings_refuse_typos` |: `RATCHET_STORE_EXAMPLES` accepts 1/0, true/false, yes/no, on/off in any case (empty is the default) and refuses anything else; a non-bool in code stops `Ratchet`; a short `RATCHET_END_USER_KEY` and `RATCHET_KEY_LIMITS` without API keys stop the gateway | | `test_ledger_from_before_opt_outs_gains_the_columns_and_tables` |: a file from before this change gains `requests.end_user` and `store_opt_out`, `examples.client` and `end_user`, `example_clients`, `no_train` and the `(end_user, ts)` index (used by the query planner), twice without change; old rows read NULL | ### `tests/test_erasure.py` Retention, erasure and export. Its invariants are (an erased example is gone for good and leaves hashes only) and (request retention never removes a release's evidence, also across the stages of a canary); their proving tests carry those markers. Time is injected (`now=`, aged `ts` and `last_seen_ts`, a `FakeClock` for the gateway's schedule). | Test | Covers | |---|---| | `test_erase_a_client_purges_its_examples_and_shared_ones_by_policy` | (`delete`, `keep`): erasing `acme` purges its own examples and leaves globex's; under `delete` an example both sent is purged, under `keep` it stays with globex promoted to first sender (no end user) and acme unlinked, and a second erasure finds nothing; the `erasures` row and the `erase` event | | `test_erasure_records_hold_hashes_and_no_text` |: an erasure by client, end user and text purges one example and unlinks one request row; no table holds the text, the raw end user or the email in the reason (stored as `[EMAIL]`); the scope holds the HMAC and the SHA-256 of `contains`; `erasure_items` and `tombstones` hold the hash; events hold neither hash | | `test_dry_run_reports_and_writes_nothing` |: a dry run returns the counts and the affected version and changes no table or event | | `test_affected_versions_are_reported_and_their_signed_boms_stay_unchanged` |: two signed nights; erasing an early train row, an early golden row and a later train row reports v1 with 1 and v2 with 2 (`tainted`, status `pending`); both BOMs are unchanged and verify; a golden-only erasure reports none (`complete`) | | `test_affected_versions_use_committed_training_sets_when_a_version_has_them` |: a version with `version_leaves` is counted by membership, one without by time; leaves win over the time rule | | `test_bom_fingerprint_covers_kept_examples_and_counts_purged_ones` |: `/ratchet/bom` changes its fingerprint (the SHA-256 over the kept hashes), counts `purged_examples` and leaves the purged row out of `sources`; `/ratchet/stats` privacy counts | | `test_an_erased_hash_is_never_stored_again_and_a_retention_purged_one_is_restored` |: `add_example` refuses an erased hash from another client (None; nothing bumped; `suppressed` 1); a duplicate of a retention-purged one restores the text with its own teacher and license, clears the purge and records the client; `dataset.rows` sees it again | | `test_the_cascade_skips_a_tombstoned_answer_as_erased` |: after an erasure the same request is answered by the teacher and logged `stored` 0 with a `skip_example` `erased` event | | `test_erasure_makes_a_retention_tombstone_final` |: a retention tombstone keeps its end user, an erasure by that end user makes it final (reason `erasure`, end user cleared, purge time kept) and refused | | `test_contains_matches_the_stored_text_literally` |: `%`, `_`, quotes and backslashes are literal (look-alike rows do not match); ASCII case folds, `É` does not; the answer and the conditions are searched too | | `test_erase_refuses_a_bad_scope_without_quoting_it` | (13 cases): no selector, bad hashes (format, a string, a non-string, none), blank `contains`, a bad client name, a blank reason or actor, an end user without the key, empty or not a string, a bad `shared`; never quotes the end user; nothing recorded | | `test_erasure_holds_under_concurrent_stores` |: three threads storing duplicates while 150 examples are erased: no error, every row erased, none revived | | `test_admin_erase_endpoint_needs_an_admin_key_and_is_a_dry_run_by_default` |: 401 without a key, 403 for an API key; a dry run by default; eight bad bodies are 400s that never quote the end user; an end user without a client is erased in every key's scope and the keyless one; events | | `test_a_client_erases_its_own_end_users_examples` |: `DELETE /v1/ratchet/examples` erases the caller's end user's examples and unlinks its request rows, not another client's same identifier; 401 without a key; an `end_user` in the URL is a 400 that does not echo it; 403 for another client's scope; dry run; bad bodies; the actor and default reason; an admin erases across clients | | `test_keyless_erasure_callers_learn_nothing_and_admins_reach_removed_keys` |: with API keys off, `DELETE /v1/ratchet/examples` answers a keyless caller with null counts for an identifier with data and one without (dry run and real); an admin's erasure of an end user for every client, by `POST` and by `DELETE`, reaches a key no longer configured | | `test_retention_purges_examples_not_seen_within_the_window_golden_included` |: train and golden rows not seen for 30 days lose their text (reason `retention`), a recently seen duplicate keeps its row, one `purge` event; a second run purges nothing | | `test_retention_purges_examples_a_chunk_per_transaction` |: with chunks of 2, seven expired examples are purged in four transactions, each under the ledger lock and the lock free before each; the rows, the result and the one `purge` event equal a single pass; a chunk of 0 is refused | | `test_request_rollup_keeps_every_stat_identical` | (chunks of 20,000 and 7): folding half the rows (a day split across the cut), then the rest, leaves `stats_by_day`, `stats_by_task`, `stats_by_client` (but `end_users`), `totals`, the opt-out counts and `cost_by_day` (also a day window) equal within 1e-9 | | `test_fold_dry_run_counts_and_writes_nothing` |: a dry fold counts and writes nothing | | `test_retention_never_folds_rows_of_a_release_in_flight` |: a canary's rows older than the window stay (its audited precision unchanged) until it is promoted | | `test_retention_keeps_every_stage_of_a_canary_for_the_anytime_valid_rule` |: a canary's rows from its 10% stage (two disagreements) stay after it moved to 50%, so `ReleaseGate.evidence` under the anytime-valid rule (12 audits, 10 agree; 4 in the stage) is the same before and after a fold; only the live version's older rows go, and the canary's go once it is promoted | | `test_an_erased_committed_example_keeps_its_leaf_and_is_never_sampled` |: erasing the example behind a version's first committed leaf leaves the leaves unchanged, and `Ledger.leaf_example` (the mock-mode gateway's sample) moves to the next leaf that still has its text, and to None once every one is purged | | `test_day_numbers_survive_folding_the_rows_that_defined_day_one` |: a live file ledger keeps its day numbers and day 1's start after folding the oldest rows, and after folding all of them | | `test_run_night_applies_retention_before_reading_the_ledger` |: `run_night(retention=)` purges expired rows before the dataset is built: the trainer gets 60 of 70 | | `test_the_purge_thread_never_touches_the_release_gates_lock` |: the purge thread, started while a tick holds the gate lock, leaves it held and raises nothing; a tick still takes and gives back its own (a merge once left `tick`'s `finally: release()` in `_purge`) | | `test_live_gateway_purges_hourly_off_the_request_path` |: a live `Runtime` purges on its first tick on a background thread, not again within the hour, again after it; a failure is a `purge_failed` event with the class only; mock mode and zero windows never purge | | `test_bad_retention_settings_stop_startup` | (4 cases): a negative or too large window, a bool, or an unknown `RATCHET_ERASURE_SHARED` stop `Settings.retention()` and the `Runtime`; spaces and case are forgiven, empty is the default | | `test_retention_days_must_be_whole_numbers_in_the_environment` | A non-number in `RATCHET_REQUEST_RETENTION_DAYS` stops startup naming it | | `test_live_purge_cli` |: `python -m ratchet.live purge` with `--dry-run`, `--client`, `--older-than-days`, a bad window and no windows; a bad `--client` or a window past 36,500 days exits with a message, not a traceback | | `test_run_night_trains_again_after_deletions_once_50_new_examples_arrive` | (`erasure`, `retention`): after 60 of v1's 100 rows are purged, 49 new rows skip (`only 49 new examples`) and 50 train on 90 rows without the purged ones (v0.4's rule said -11); then three nights with no new data do not train (`only 0 new examples`; measured against v1's purges they retrained every night), a deletion of 10 of v2's rows is not new data either, and 50 new rows train on 130 | | `test_a_purged_variant_still_counts_toward_its_requests_frequency` |: a golden prompt seen three times over a bare and a stop variant trains; purging the stop variant keeps its prompt key and its count, so the bare variant stays in training | | `test_export_round_trips_with_provenance_and_holds_no_keys` |: every kind streams NDJSON as an attachment; examples equal the ledger's kept rows with their provenance and clients; requests have no end user; versions, erasures (items) and events; no API key, admin key, end-user key, end-user hash or raw end user in any line | | `test_export_filters_and_refusals` |: `split`, `task`, `client` and `since` (ISO date, epoch, date-time) filters; 401 and 403; an unknown kind, a filter the kind does not take, a bad `since`, `split` or client name are 400s | | `test_export_pages_through_large_tables` |: pages of 5 give all 23 rows in order, one line each | | `test_admin_cli_erases_and_exports` |: `python -m ratchet.admin erase` (a dry run, then `--apply` with the end user from stdin across every scope), its errors, and `export` to a file and to stdout, with a refused filter | | `test_admin_cli_erases_end_users_only_with_the_gateways_key_and_never_writes_otherwise` |: `python -m ratchet.admin erase --end-user` in a shell without the gateway's `RATCHET_END_USER_KEY`, with another key or a short one is refused (dry run and `--apply`) and the ledger's meta, events and erasures stay unchanged; with the right key and no `RATCHET_API_KEYS` it finds the end user in every scope the ledger holds (a removed key's too) and logs no key change; a gateway's generated key is used from the ledger; a ledger that never hashed an end user is refused; a free-text `--actor` is refused | | `test_a_ledger_from_before_retention_gains_its_columns_and_tables` |: a file with the old `erasures` table gains the new columns and tables twice without change; `last_seen_ts` is backfilled from `ts`, and retention works on it | | `test_an_erasure_during_the_champion_rescore_is_not_trained_on` |: an example erased while the live version is re-scored is not in the trainer's rows or the new version's leaves; the count and dataset hash match the rows; an `erasure_honoured` event, no `erasure_tainted` | | `test_an_erasure_while_the_adapter_trains_taints_the_new_version` |: an erasure recorded inside `trainer.train` names no version when it runs; afterwards it is `pending` with the new version in `tainted`, one `erasure_tainted` event, the leaf stays | | `test_an_erased_example_is_never_a_tombstone_hit_in_the_training_rows_of_a_quiet_night` |: an erasure before the night changes nothing the night writes (59 rows, no erasure event); `Ledger.tombstoned` answers by key | ### `tests/test_licensing.py` Licenses that travel with the adapter and configured teachers. It extends invariants 1 (configuration cannot relabel a known closed teacher) and 14 (the NOTICE is part of the signed BOM); the proving tests carry `` and ``. The live night runs through `live.main`'s seams (`run=`, `http=`, `provider_factory=`), offline. | Test | Covers | |---|---| | `test_write_notices_lists_base_and_teacher_licenses` |: `notices.sources` (URL, origin, distill_ok and count per teacher); NOTICE.md names the base model's license and URL, each teacher with its license, URL and count, "(configured)" for a configured one, teacher outputs only and the Ratchet version; the same bytes whatever the order; a warning for an unknown base model; no sources; cells that cannot break the table; `train.write_metadata` writes `ratchet_manifest.json` and NOTICE.md from the container's built-in registry | | `test_base_model_licenses_cover_the_configured_base_models` |: the default student base model, the hosted fallback and the 30B checkpoints map to the NVIDIA Nemotron Open Model License, in any case; a local path, an unlisted id or a bare name do not; `base_license`'s warning | | `test_bom_v2_has_base_model_license_and_v1_still_verifies` |: `base_model_license`, each source's `license_origin`, `notices` with the exact text and its SHA-256; a warning, not an error, for an unknown base; the signature covers the notice; v1 and early v2 documents still verify and get a rendered NOTICE; `run_night` writes it | | `test_live_night_puts_bom_and_notice_in_adapter_folder` |, US-10: `python -m ratchet.live night` syncs, trains, writes `data/adapters/student-v1/ratchet_bom.json` (the stored BOM's bytes, also in `data/boms/`) and the NOTICE.md the BOM signed, then syncs again (the second sync sees both files); it installs `RATCHET_TEACHER_LICENSES` itself, so the configured teacher is in the BOM and NOTICE; a night that does not train writes and syncs nothing | | `test_live_night_reports_a_failed_second_sync` |: a failed second sync leaves the files written, writes a `sync_failed` event with the class only, prints the report and exits non-zero; without `RATCHET_SYNC_CMD` the files are written and nothing runs after training | | `test_export_bundle` |, US-10: `export` copies the adapter folder (nested files too), writes the verified BOM and the NOTICE it signed over the job's copy, prints the BOM's sha256 and returns a summary; `python -m ratchet.bom verify --notice` accepts the exported NOTICE.md and refuses an edited or missing one; refuses a non-empty folder or a file, a folder inside the adapter folder or anywhere in `RATCHET_DATA_DIR`, an unknown version, a version without a BOM, a tampered BOM, an adapter name outside the adapters folder, a missing `--version`, writing nothing; without local weights it writes the licenses and says where the weights are; `adapter_files` counts the adapter's own files, not the job's NOTICE.md; `python -m ratchet.live export` prints only the export's lines | | `test_export_after_a_live_night_says_the_weights_are_in_the_bucket` |, US-10: after a live night with an upload-only sync the adapter folder holds only the BOM and NOTICE.md `publish_bom` wrote; `export` (and `python -m ratchet.live export`) then counts no adapter files, reports no weights and says where in the bucket they are, never that it exported the adapter; a folder with a stray file but no weights still gets the hint; once `adapter_config.json` and the weights are there, it ships them | | `test_config_teacher_license_allowed_and_closed_prefix_still_refused` |: configured teachers are licensed with origin `config` and canonical license names; the built-in registry is a read-only copy source; seventeen closed-family ids claimed as Apache-2.0 are refused (`azure-gpt-4o`, `chatgpt-4o-latest`, `anthropic/next-model`, `openai/gpt-oss-120b`, `my_gemini_mirror`, `davinci-002`, `o5-mini`...) and so are router aliases (`openrouter/auto`, `.../auto:floor`), while open ids pass the family check and an unrecognised closed id (`x-ai/grok-4`) is shown to pass it too; five license names outside the allowlist are refused; a built-in teacher cannot be relabelled; a closed id forced into the registry still raises; after a reset the teacher is unknown again | | `test_config_teacher_end_to_end` |: a configured teacher is refused without the file and licensed with it; its answer is stored with its license and URL; Night School trains on its rows and the BOM and NOTICE name it as configured; a cascade built without the file publishes the built-in registry, and the rows leave training | | `test_bad_teacher_license_file_fails_at_startup` |: bad JSON, NaN, not a list, too many entries, and twelve entry errors in one file, each listed with its place; a missing file; the registry left as it was; `Ratchet`, the gateway's `Runtime` in both modes, `python -m ratchet.live` and `python -m ratchet.admin` stop before opening the ledger; the fixed file starts | ### `tests/test_feedback.py` Ground truth from people: request ids, verdicts and corrections, human labels as a provenance source, labelled-data import and golden-only rows. It amends invariants 1, 2, 3, 10 and 16 (`human` is never a teacher; human labels are targets, student outputs never; golden-only rows; no human label for a pinned task; the second writer of examples) and adds (human verdicts never promote or roll back a version); the proving tests carry those markers. Gateway tests drive a live version whose student and teacher are `RecordingProvider`s. | Test | Covers | |---|---| | `test_response_has_request_id` |: `ratchet.request_id` (`req_` + 32 hex) on every response, the completion id with the same digits, stored in `requests.request_id`; a stream's chunks carry it; internal calls have none; `parse_request_id` takes either form and nothing else | | `test_feedback_only_on_own_requests` |: another key's request is the same 404 as an unknown one, without the id quoted; 401 without a key; one row per request, a later verdict replacing it; no text or key in the table | | `test_correct_verdict_never_creates_example` |: a `correct` verdict on a student answer stores nothing and calls no model; a `correct` verdict carrying the student's text is a 400; task agreement in `/ratchet/stats` | | `test_corrected_feedback_stored_with_human_provenance_redacted_and_verified` |: a correction stored as `human`, `customer-provided labels`, `verified`, origin `feedback`, with its client; messages and label redacted and counted; the teacher answer's hash and its split; Night School reads it | | `test_human_overrides_teacher_on_same_hash` |: a correction replaces the teacher's stored answer without counting the request twice; a later teacher answer only bumps `seen`; a JSON-mode request's correction needs its conditions to replace that variant, and other conditions are a 409 | | `test_teacher_model_human_refused_at_startup` |: `human` in any case as the teacher or fallback teacher stops `Ratchet`; never in the registry; a `RATCHET_TEACHER_LICENSES` entry named `Human` is refused; `lookup_source` | | `test_feedback_body_is_checked` |: eleven bad bodies, each a 400 with its `param` and `code` (missing fields, a bad verdict or id, an unknown key, a non-string or invalid correction, messages without a correction or a correction without messages, an image part); nothing recorded | | `test_correction_storage_rules_still_apply` |: the request's own opt-out, the key's `store=0`, a no-train end user registered after the request (the identifiers required and checked), a pinned task, `RATCHET_FEEDBACK_CORRECTIONS=0` and zero retention each keep the correction out with the reason, the verdict kept | | `test_correction_needs_every_end_user_identifier` |: a request that sent `user` and `safety_identifier` logs both keyed hashes; a correction with one of them is a 400 (`end_user_required`), a wrong second one `end_user_mismatch`; with both, a no-train registration made with `user` (the client's, or an admin's for every client) keeps it out; one identifier on the request needs one; the second hash is never exported and an erasure by either identifier clears both | | `test_correction_conflicts_and_fixes` |: the same request's correction again replaces its own label (`seen` unchanged); other messages and a `correct` verdict after a stored correction are 409s | | `test_label_precedence_human_teacher_import` |: teacher < correction < import; another client's correction never replaces one; a later import replaces an import; a teacher import never replaces a label; a retention-purged human label comes back as the teacher's; an erased example refuses a correction | | `test_store_label_is_atomic_across_processes` |: a second connection on the same file cannot write between `store_label`'s read and write (`Ledger.atomic`), so the import's label stands and the teacher's answer only bumps `seen` (from 0: the import was no occurrence); a refused golden-only row leaves no transaction open; an error rolls back | | `test_human_precision_alerts_and_never_moves_a_release` |: one `human_alert` once `min_audits` verdicts give anytime-valid evidence that precision is below the floor, never again (meta row); the version stays live and the gate writes nothing; teacher-served answers count toward task agreement only | | `test_human_alert_is_anytime_valid` |: the alert fires exactly when the mixture e-value against 'precision >= live_floor' reaches 1 / `ALERT_ALPHA`, checked after every verdict; 14 correct of 20 (Wilson upper bound under 0.92) does not alert | | `test_feedback_outlives_folded_request_rows` |: after request retention folds the row, the version's human precision and the task's agreement keep the verdict, and a new verdict on that request is a 404 | | `test_concurrent_verdicts_keep_one_row_and_one_label` |: eight identical corrections at once: one feedback row, one example, `seen` 1 | | `test_old_ledger_file_gains_feedback_schema` | an old file gains `request_id`, `end_user_secondary`, `verified`, `golden_only`, the `feedback` table and its indexes | | `test_import_openai_finetune_jsonl` |: OpenAI rows, a multi-turn row with weights and content parts (earlier turns kept, address redacted), in-file duplicates and conflicts, six kinds of invalid rows with line numbers, tools hashing like a live request; imported rows start at `seen` 0; the `import` event (counts, attestation, SHA-256, no row text); importing again counts nothing twice | | `test_import_refuses_closed_model_source` |: twelve closed-family ids and a router alias refused with the clause quoted, also by the CLI before reading the file; unknown teachers, `teacher:human` and malformed sources; an open teacher's rows keep its license, unverified | | `test_import_requires_attest_for_human` |: no, empty, multi-line or overlong attestation refused | | `test_import_refuses_pinned_task` |: pinned, no-store (spec and setting), zero retention and unregistered tasks refused before anything is stored; rows of a task pinned later leave Night School; `store_human_example` refuses a pinned task | | `test_golden_only_rows_never_trained` |: golden-only rows whose hash says train stay golden, past FREQUENT and with every variant of the prompt; prompts already trained on are refused (`trained`); a real `run_night` trains on none and its BOM counts `golden.human_labels` | | `test_labels_that_are_no_occurrence_never_reach_frequent` |: an imported variant of a twice-seen golden prompt (and a non-bumping correction) starts at `seen` 0 and leaves the prompt golden; the third real request makes it frequent; an imported prompt needs FREQUENT real requests; the export reports 0 | | `test_golden_only_check_reads_the_ledger_at_each_row` |: a train-split row the gateway stores while an import runs keeps that golden-only row out (`trained`); the check is per row, in the transaction that writes it | | `test_bom_lists_human_source` |: `sources` lists teachers only, `label_sources` the human labels by origin, `training.targets` and the NOTICE's Human labels section; an adapter without them gets the earlier NOTICE text; the ledger's `/ratchet/bom` lists them apart | | `test_import_ratchet_train_jsonl_roundtrip` |: Ratchet's `train.jsonl` imports back with the same hashes; a teacher's rows cannot come in as human labels or under another task | | `test_import_cli_dry_run_and_apply` |: `python -m ratchet.importer` dry run (refused for a ledger file that does not exist, which it does not create; an older file read without being migrated, byte for byte), apply (rows attributed to `--client`, one event), a missing file, a bad `--client`, a line over the body limit skipped | | `test_counts_as_train_reads_human_labels` | a human training row counts toward the new-data count, a golden-only row never | | `test_export_carries_label_flags` | the NDJSON export carries `verified`, `golden_only` and the provenance | | `test_a_correction_a_policy_keeps_out_is_never_redacted` |: a correction under the request's opt-out or a key's `store=0` records its verdict without `prepare` running; a kept one prepares once; an answer the spec refuses is still a 400 whatever the policy | | `test_import_refuses_a_client_whose_key_stores_nothing` |: `--client globex` with `globex:store=0` is refused before any row is read (nothing stored, no event); another key and a name that is no key import; a limits spec the gateway would not start with is `cannot read the key limits` | ### `tests/test_receipts.py` Verified savings receipts. Synthetic populations are seeded; simulation runs are a few days of mock traffic and assert trends. | Test | Covers | |---|---| | `test_leaves_hold_only_metadata_and_the_key_name` |: through a gateway with an API key, a request's leaf has exactly `LEAF_KEYS`, whole numbers and strings only, the key's name; neither the receipt, the leaves nor the proof holds the request's text, its answer or the key; no float anywhere in the manifest | | `test_leaf_turns_amounts_into_whole_nano_usd` | `leaf` rounds USD to nano-USD (half to even), `label_p` to parts per billion and `ts` to milliseconds; NaN and missing amounts are null; `''` clients are null; `leaf_bytes` is canonical JSON | | `test_ratio_estimator_recovers_the_ratio_and_its_lower_bound_covers` | y = 1.3 x (1 + 0.1 z) audited at 20%: the mean R over 200 seeded populations is within 0.01 of 1.3, and the one-sided 95% lower bound covers the true total in at least 180 of 200 (about 95% in 2,000) | | `test_weights_follow_label_p_and_the_finite_population_correction` | Answers audited at two rates are weighted 1 / `label_p` (R equals the hand-computed Horvitz-Thompson ratio, n_eff below n); with every answer audited the variance is 0 and the point is the exact total | | `test_insufficient_tasks_are_not_billed_and_the_fee_is_a_share_of_the_lower_bound` | A task under `min_audits` is `insufficient_audits` and only counts at its recorded baseline; the billable set is the teacher-served requests and the calibrated task's served answers; margin = ⌈1.645 se⌉; fee = ⌊share × net⌋ (share as a float or a string); spend outside requests comes off; a negative bound or no calibrated task bills 0 (`formula`); bad shares, minimums and spend raise | | `test_stand_in_answers_are_never_billed` | `UNVERIFIED_ROUTES` equals the router's `STAND_IN_ROUTES`; a degraded answer and a fallback teacher's answer change neither the calibration nor the bill, and count at their recorded baseline | | `test_unlogged_formula_estimates_are_never_calibrated` | Served answers without `baseline_est_usd` (before v0.5) make the task `unlogged`: no point, no bill | | `test_integer_sums_are_exact_over_100k_rows` | 100,000 synthetic rows: the totals equal the sum of the rounded amounts exactly, within half a nano-USD per row of the float sum | | `test_seal_is_idempotent_and_never_updates` |: a second `seal` and `seal_due` return the stored document and issue no INSERT, UPDATE, DELETE or REPLACE (`set_trace_callback`); `add_receipt` refuses an existing (period, client) | | `test_period_rows_are_read_in_chunks_with_the_lock_free_between_them` | `period_rows` reads `RECEIPT_CHUNK` rows per locked statement: another thread logs a request between chunks; the chunks equal the one read, in id order, a client's rows only for a client; `receipt_row` reads one row | | `test_concurrent_sealing_writes_each_receipt_once` | 8 threads sealing at once write each (day, client) receipt once with no error; a second connection to the same file that loses the race keeps the first writer's document | | `test_a_day_is_sealed_only_once_it_is_over` | The current day raises `PeriodOpen`; a day without requests has no receipt; on a live ledger a day is closed 5 minutes after it ended, not before | | `test_signature_verifies_and_edits_break_it_and_the_chain` | Three chained days: `prev_sha256` links each to the one before; the signature verifies with the public key and fails with another; an edited total fails, also re-hashed; an earlier receipt with a lowered fee, or the wrong day, breaks the chain check | | `test_receipts_are_chained_per_client_and_hold_only_that_clients_ids` | Two clients and anonymous traffic over two days: each client's receipt holds only its ids, rechecks from its own leaves and chains to its own previous receipt; the gateway's covers every request; GPU-view notes | | `test_spend_outside_requests_is_netted_and_shared_by_request_count` | A backfill spend row comes off the gateway's bound in full and off a client's by its share of the day's requests | | `test_recheck_recomputes_everything_from_the_leaves` | `recheck` passes on the stored leaves; a changed amount (root and totals), reversed order, a missing leaf and a fee beyond the tolerance are caught; an estimated amount within 2 nano-USD is not | | `test_verifiers_never_raise_on_malformed_input` | 8 junk documents: `verify_receipt`, `verify_chain`, `verify_proof`, `verify_rows` and `check_manifest` return failures, never an exception | | `test_gateway_returns_the_request_id_and_its_proof_verifies` | A signed mock gateway: `X-Ratchet-Request-Id` and `ratchet.row_id` (also on the last stream chunk; `ratchet.request_id` is the feedback id); the day is 409 until it ends, then sealed by the live gateway day; the listing, the signature, the GPU view; each request's proof verifies with `merkle.verify_inclusion` and `verify_proof`; unknown ids 404, bad periods 400; the leaves recheck; the next day chains to it | | `test_api_keys_read_only_their_own_receipts` | With API and admin keys: client B gets 403 on client A's receipt, proof and leaves, 404 for A's request in its own receipt; no key or a wrong key 401; an admin reads the gateway's and every client's receipt, and the listing per caller | | `test_live_mode_without_keys_serves_no_receipts` | Live mode without keys: 403; the mock-mode gateway without keys is open | | `test_rows_changed_after_sealing_are_caught` | A row slipped into a sealed day: proof and leaves answer 500 (twice) with one `integrity` event, and the receipt is unchanged; a row edited after its tree was cached is caught too (the proof reads its row again) | | `test_request_retention_never_breaks_a_receipt` |: a signed mock gateway; day 0 is sealed and a proof served (its tree cached); a fold takes day 0's rows and one of day 1's: day 0's receipt is unchanged and verifies, its proof (from the cache too) and leaves answer 410 `folded by request retention`, never the integrity 500; day 1, folded before it was sealed, is never sealed from the row left (410, one `receipt_skipped` event however often it is asked for) and day 2 chains to day 0 | | `test_live_purge_seals_every_ended_day_before_it_folds` |: `python -m ratchet.live purge` with a 2-day request window on a live ledger seals days 1 and 2 (the client's and the gateway's receipts, 2 and 1 leaves) before it folds their 3 rows; no `receipt_skipped` | | `test_bad_receipt_settings_stop_the_gateway` | 5 cases: a take rate above 1 or NaN, a minimum of 1 audit, a negative or huge mock reasoning mean stop the gateway with a message naming the variable | | `test_receipt_settings_are_read_and_checked` | Defaults 0.25 and 30 and no mock mean; the variables read and no longer reserved; `share_text` | | `test_mock_teacher_reasoning_varies_around_its_mean_deterministically` | The default bills exactly 300; with mean 420 and cv 0.3 the same request always costs the same, the mean is 395 to 445 over 400 requests, `out_tokens` follow; cv 0 bills the mean | | `test_sealing_moves_no_simulation_number` | `--receipts` on, off and with `seal_due` monkeypatched out: identical days, totals, versions, nights, events and serving; with the formula right (300 tokens) the bound held and R is about 1 | | `test_receipts_measure_and_correct_a_teacher_that_reasons_more` | `--teacher-reasoning-mean 420` (simulation): each calibrated day's `ticket-triage` R is above 1.2, the formula's baseline is below 95% of the true cost and the calibrated one within 10% of it, a fee is billed on the bound, and the bound is at or below the truth on all but one day at most | | `test_cli_verifies_a_bundle_and_fails_after_a_one_digit_edit` | `python -m ratchet.receipt verify` with the key, the previous receipt, a proof and the leaves: four OK lines, exit 0; one digit of the fee changed: exit 1 (`sha256 does not match`); not JSON: 1; a missing file: 2 | | `test_cli_module_entry_point` | The same as a subprocess (`python -m ratchet.receipt`), with an unchecked chain reported | | `test_offline_seal_of_a_simulated_ledger` | `python -m ratchet.receipt seal --db` on a `sim.simulate --db` ledger: every logged day sealed and labelled simulation, a day after the run 2, nothing left to seal 0, a missing file and `--client` without `--period` 2 | | `test_live_seal_command_seals_every_day_that_ended` | `python -m ratchet.live seal` on a live ledger seals every client's and the gateway's receipts of the days that ended (not today's), then says nothing is left; labelled live traffic | | `test_the_gateways_purge_never_folds_a_day_that_has_not_been_sealed` |: `Runtime._purge` with a day that ended 100 s ago (inside the 300 s grace) folds the earlier day and leaves the ended one whole; an hour later it is sealed from both of its rows, no `receipt_skipped` | ### `tests/test_perf.py` The hot path. See [Hot-path tests](#hot-path-tests) for how the timing tests stay stable. | Test | Covers | |---|---| | `test_file_ledger_uses_wal_and_memory_does_not` | A file ledger: WAL, `synchronous=NORMAL`, 5 s busy timeout, a read-only second connection that sees each commit at once and refuses writes; `:memory:` has none; an old rollback-journal file is switched on open and gains `wall_ms` | | `test_sync_full_keeps_the_rollback_journal_and_bad_values_stop_startup` | `RATCHET_DB_SYNC=FULL` (any case) takes a WAL file back to the rollback journal with `synchronous=FULL` and no second connection; anything else raises naming the variable; `config.SYNC_MODES` matches the ledger's | | `test_full_cannot_take_a_file_out_of_wal_while_another_process_holds_it` | `FULL` on a file another connection holds in WAL stops with an error naming the setting; `NORMAL` alongside is fine | | `test_gate_queries_search_their_indexes` | `EXPLAIN QUERY PLAN` of the ledger's own query constants: the audit tally searches `requests_version_audited` from the last id, the shadow verdicts `requests_shadow_version`, the routing rebuild `versions_state`, a day's percentiles `requests_day`; none scans `requests` | | `test_version_cache_sees_another_connections_writes` | Two ledgers on one file (gateway and cron tick): a new live version, canary steps, a rollback and a new eval written by one are seen by the other on its next read | | `test_version_cache_invalidates_on_state_change` | Every version write (create, eval, state, traffic, BOM, and SQL on `ledger.db` that bypasses the writers) is seen on the next read; each read is a fresh copy, so changing one changes nothing cached; `version(id)` agrees | | `test_routing_reads_no_version_rows_until_a_version_changes` | With 31 versions, ten routing rounds run only `PRAGMA data_version`; the next version write makes the next read rebuild once | | `test_live_quality_tally_matches_a_full_count` | `version_live_quality` equals one `COUNT(audit_agree)`/`AVG` over the stage after new audits (read above the last id), audits without a verdict (pending off-path audits, counted in neither number), other versions' and teacher rows, a new stage, a deleted row, and rows another connection commits | | `test_bom_digest_is_cached_until_examples_change` | The BOM digest is recomputed after an insert, a rewritten or deleted hash, or another connection's insert, not after a duplicate; never cached against an uncommitted write; always equal to a fresh computation | | `test_events_last_reads_only_the_newest` | `events(last=n)` is `events()[-n:]`; 0 gives none; a bad `n` raises | | `test_stats_reads_skip_the_writer_lock` | On a WAL file every stats query returns while another thread holds the writer's lock | | `test_reader_methods_use_only_the_reader_connection` |: no `@_reads` method touches `self.db` (the writer's connection, which a reader holds no lock for; `last_ids` did) | | `test_reads_and_an_offpath_writer_never_deadlock` |: in a child process (the freeze takes the interpreter with it), every `_reads` method that needs no arguments against a thread running `update_request` and logging requests on a WAL file for a second; a deadlock dumps every stack after 30 s and fails the test | | `test_tick_throttle_runs_the_gate_once_per_interval` | With a 5 s interval on a fake clock: one gate tick per interval, none while another thread ticks (without waiting), `force` and simulated days always tick, a failing tick is not retried on every request, interval 0 ticks every call | | `test_request_path_ticks_at_most_once_per_interval` | Through `/v1/chat/completions`: three requests within the interval share one tick; the next one after it ticks | | `test_snapshot_cache_ttl_and_single_flight` | `SnapshotCache`: one value per TTL; during a recompute other callers get the previous value at once, and the very first callers wait; a failed recompute raises and keeps the old value; TTL 0 computes every time | | `test_stats_ttl_returns_the_cached_payload` | `/ratchet/stats` with a 2 s TTL returns the same payload within it despite a request, the new one after it, and a fresh one after a test run reset | | `test_interval_settings_have_mode_defaults_and_bounds` | 5 s and 2 s in live mode, 0 in mock; 0 to 3600 s; negative, too long, NaN, infinite, bool and unparsable values raise naming the variable, also at gateway startup | | `test_latency_percentiles` | Nearest-rank p50 and p95 of `wall_ms`, exact, by day and `served_by`; q 0 and 1 are the minimum and maximum, 0.07 of 20 is rank 2; old rows without `wall_ms` count toward nothing; None without rows; bad quantiles raise | | `test_stats_by_day_reports_latency_percentiles_and_sorts_only_changed_days` | `stats_by_day`'s `p50_student_ms` and `p95_ms` equal `latency_percentiles`; a second read sorts nothing, a new request sorts only its day, a deleted row every day | | `test_wall_ms_logged_and_at_least_latency` | Over `OpenAICompatProvider` and a fake server, cold start, student-served, escalated, pinned and teacher-only rows each have `wall_ms >= latency_ms > 0` | | `test_wall_ms_is_the_timer_around_the_whole_cascade` | With `Ratchet(timer=...)`, `wall_ms` is exactly the two timer reads apart, on `handle`, a pinned task and `teacher_only`; `latency_ms` is the provider's | | `test_handle_overhead_does_not_grow_with_versions` | Zero-latency models: the mean request time with 365 retired versions is at most twice the one-version mean plus 0.5 ms (v0.4's lookups took about 25 times as long) | | `test_perf_guard_200k_requests` | A WAL ledger with 200,000 requests and a live, canary and shadow version: a cold tick under 250 ms, a warm tick under 20 ms (median), a request under 5 ms of its own time, the tally still exact | ### `tests/test_security.py` Endpoint access, scrubbed public stats, the OpenAPI switch, the live-mode auth guard and security headers. A private live gateway runs offline through `Runtime(transports=)` with three kinds of keys. | Test | Covers | |---|---| | `test_every_route_has_an_access_level` | Every route the app serves has a level in `access.ROUTES` or `access.PREFIXES`, and the Access column of docs/API.md shows the same one; HEAD reads as GET, the prefixes cover routes other packages add (`/ratchet/requests/...`, `/v1/ratchet/no-train`), `/livez`, `/readyz`, `/metrics` and `/ratchet/kpis` are pre-classified, and an unknown route is not guessed | | `test_private_live_dashboard_needs_an_admin_or_dashboard_key` | Every dashboard and BOM route of a private live gateway: 401 with `WWW-Authenticate: Bearer` without a key or with a wrong one, 403 for an API key, 200 for an admin or dashboard key (Bearer or `X-API-Key`); health, the BOM public key, the pages and Scout stay open; a dashboard key cannot write tasks (403) or call `/v1/*` (401); keyed readers see client names and no key reaches the payload or the database; `RATCHET_PUBLIC_BOM=1` opens only the BOM endpoints | | `test_private_dashboard_without_reader_keys_is_closed_to_everyone` | No admin or dashboard keys: every read is 403 naming `RATCHET_DASHBOARD_KEYS`, also with an API key | | `test_mock_dashboard_is_public_by_default` | Mock mode: the dashboard API is open with or without admin and API keys, and unscrubbed; `RATCHET_PUBLIC_DASHBOARD=0` closes it like a live one | | `test_public_live_stats_are_scrubbed_for_anonymous_readers` | A public live dashboard: anonymous readers get `client-1`, `client-2` by sorted name (the same pseudonyms in task allow-lists), a v0.4 `ConnectError: ...` event cut to the class, URLs as ``, no client name, address or key; admin and dashboard readers get the full payload from the same cached one (never changed by a scrubbed copy); `RATCHET_SCRUB_STATS` 1 and 0 | | `test_scrub_helpers` | `scrub_detail` on v0.4 and current event formats (today's class-and-status details are kept), `pseudonyms`, `scrub_stats` leaving its input unchanged, `scrub_task` | | `test_live_mode_without_api_keys_does_not_start` | A live `Runtime` without API keys raises before the ledger file exists; `allow_unauthenticated` lets it start; mock needs no keys; the same from the environment as uvicorn loads it | | `test_each_key_has_one_role` | A dashboard key that is an API or admin key, a short one, or an admin key that is an API key stops startup naming the variable | | `test_access_switches_resolve_by_mode_and_refuse_typos` | Mock and live defaults; explicit 0 and 1; set but empty is the default; `yes`, `true`, `2`, `on` stop startup naming the variable; a CSP with a line break is refused; `Access.scrubs` for every case | | `test_openapi_is_off_in_live_mode` | Live: `/openapi.json`, `/docs` and `/redoc` are 404, back with `RATCHET_OPENAPI=1`; mock serves the schema, `0` removes it | | `test_security_headers_on_every_response` | `nosniff` and `no-referrer` on a page, health, stats, a chat completion, a stream, a 400, a 404, a 413 from the body limit, Scout, and both 500s (sent outside the middleware); no CSP by default; `RATCHET_CSP` adds it | | `test_unauthenticated_live_mode_needs_the_edge_guard_it_relies_on` |: `allow_unauthenticated` alone, with the guard open, a missing token, a 31-character token or a misspelled guard is refused at startup (before the ledger opens) and by the config check, naming both variables and never the token; ` Guarded ` and 32 characters start; API keys need no guard | ### `tests/test_observability.py` Request ids, the failures table, structured logs, Prometheus metrics, trace forwarding and version-transition events. Gateways run in mock mode or as private live gateways through `Runtime(transports=)` with `FakeOpenAIServer` and `httpx.MockTransport` teachers; the cascade runs on `RecordingProvider`s. `parse_metrics` is a small parser of the text format 0.0.4 that fails on any malformed line or duplicate series. | Test | Covers | |---|---| | `test_request_id_echoed_and_client_id_stored` | `x-request-id` (`req_` + 32 hex) and `openai-processing-ms` on a completion; `id` is `chatcmpl-` + the same digits and `ratchet.request_id` the id; a valid `X-Request-Id` or `X-Client-Request-Id` lands in `requests.client_request_id`; spaces, 129 characters, `;` and non-ASCII bytes are ignored without an error; every page and 404 gets its own id, never the client's | | `test_streamed_answer_carries_the_request_id` | A stream's header, every chunk's `id` and the last chunk's `ratchet.request_id` | | `test_error_responses_carry_x_request_id_and_envelope_request_id` | A 400, 401, 413 and 500 carry the header and `request_id` beside OpenAI's untouched four-field `error`; the 400 writes no failures row, the 401 an `auth` row (`HTTPException 401 invalid_api_key`, no client), the 413 a `gateway` row, the 500 a `gateway` row naming the class only; a non-`/v1` 404 keeps `{"detail"}` with the header | | `test_request_context_parsing` | `new_context`: a padded client id, a valid `traceparent` and `tracestate`; the second header when the first is invalid; upper case, all-zero ids, version `ff`, extra fields on version `00` and a short parent id refused; a later version's extra fields accepted; a 600-character `tracestate` dropped; `lookup_hex`; no headers outside a request | | `test_traceparent_forwarded_to_the_teacher_with_a_new_span` | Each teacher attempt (a 503, then the retry) carries the trace id, flags and `tracestate` with its own new span id, never the client's; a malformed or missing `traceparent` sends none | | `test_request_lookup_own_request_and_another_clients` | `GET /v1/ratchet/requests/{id}` by either id form: the own request's metadata (no text); another key's request and an unknown id are the same 404 `request_not_found`; malformed ids 404; no key 401; a quota 429 is found by its failures row alone; `GET /ratchet/requests/{id}` reads any request | | `test_teacher_failure_row_carries_the_student_cost` | An escalation whose teacher fails: no requests row, one `teacher` row with the student's sample cost, status 502, the client, task, both ids and `UpstreamError 502 teacher_down`; `totals` counts it in `failed_cost_usd`, `saved_usd` and `savings_pct` | | `test_teacher_502_through_the_gateway_writes_one_row_with_the_student_cost` | The same through a live gateway: exactly one row (the router's), no URL or message in `error`, and `ratchet_cost_usd_total`, `ratchet_failed_cost_usd_total` and `ratchet_failures_total{stage="teacher"}` rise by it | | `test_quota_429_writes_a_quota_row_and_counts_a_rejection` | A quota 429: a `quota` row with the client, task and id (`APIError 429 daily_quota_exceeded`, cost 0), one `quota` rejection, still no requests row | | `test_teacher_429_is_a_teacher_failure_not_a_refusal` | A teacher 429 answered as a 429: one `teacher` row and no gateway `rate` row; no rejection counted; `ratchet_requests_total{status="429"}` once | | `test_a_bug_on_the_teacher_path_is_a_gateway_failure_not_the_teachers` | A `RuntimeError` in `_store_example` after the student was sampled: a 500 and one `gateway` row (500, `RuntimeError`, the samples' cost), no `teacher` failure counted, the failed spend in the metrics and totals; with a throttle of 0 a row that carries spend is still written; in process, on `handle` and `teacher_only`, the router writes `gateway` rows with no status | | `test_gateway_failure_row_exists_before_the_response_starts` | The app called as raw ASGI: when a wrong key's 401 starts, its `auth` row is already in the ledger under the response's `x-request-id` | | `test_swallowed_failures_get_rows_and_breaker_refusals_are_sampled` | 30 requests against a failing student: 5 `student` rows for the errors (502), 3 for 25 breaker refusals (503), `ratchet_failures_total` counts all 30, every request answered; a timed-out audit: an `audit` row (504, `ReadTimeout`, cost 0) and the student's answer served | | `test_stats_errors_last_hour_and_failed_cost` | `errors.last_hour` by stage leaves out a two-hour-old row; `totals.failed_cost_usd`; `log_failure` refuses an unknown stage or column | | `test_gateway_failure_rows_are_throttled_per_stage` | `FailureThrottle` per stage and minute; five wrong keys write two rows but count five rejections and five failures | | `test_json_log_lines_carry_no_content_and_no_keys` | A live gateway at INFO: no marker from a message or an answer, card number, API key, teacher key or endpoint host in any line; the `chat` lines have every expected key, the ids, the client name and trace id; the 502's line has `served_by` null; one `upstream_error` warning (role, class, status); a mirrored event cut to its exception class | | `test_log_settings_formats_and_handlers` | `log_settings` defaults by mode, case-insensitive values, typos stop startup; `config` and `obs` agree on the values; `configure` never doubles a handler; the text format quotes values with spaces or quotes; JSON lines; levels filter | | `test_event_listeners_see_every_event_and_cannot_fail_it` | A listener added twice runs once and gets the stored row; a failing listener does not stop the event | | `test_every_version_transition_writes_an_event` | Cold start to canary to live, shadow to canary to live over the old live, a candidate replaced by a newer one, a rolled-back canary and a rejected candidate: all 14 moves (9 kinds of transition) go through `transition` or `promote_live` (`set_version` raises if the gate calls it) and each writes, inside the same call, an event naming the version, one of them `live_retired` | | `test_metrics_parse_count_and_bound_their_labels` | `/metrics` parses, has `# TYPE` for every family and the 0.0.4 content type; a request counts under its task, an ad-hoc task as `other` (its name never a label), a 400 as `none`/`400` and an `invalid_request` rejection; an example counted by split; cost and baseline rise; `ratchet_info`; histogram buckets cumulative, ending at `+Inf` equal to `_count` | | `test_cascade_metrics_for_upstreams_audits_and_examples` | Upstream `ok`, `timeout` and `circuit_open` outcomes, audits by result, escalations, stored examples and upstream latency observations counted exactly; `task_label` | | `test_metrics_401_in_live_mode_without_a_key_or_token` | Live: no key or a wrong token 401, an API key 403; the metrics token, a dashboard key and an admin key 200 with key names in the labels; the token reads nothing else; without reader keys the token alone opens it, and without the token too it is 403 | | `test_metrics_token_is_checked_at_startup` | A short token, or one that is also an API, admin or dashboard key, stops startup | | `test_public_live_metrics_scrub_client_labels` | A public live dashboard: anonymous `/metrics` and request lookups show `client-2`, never a client name; a keyed reader sees `globex` | | `test_scrape_time_gauges` | Version traffic for the live and canary versions, the student breaker open after a test run outage and the teacher's closed, today's spend | | `test_metrics_module_format` | Label and help escaping, a counter refusing to go down or take the wrong label count, duplicate names refused, bad buckets refused, `Gauge.replace` dropping old series, NaN observations ignored, cumulative buckets, `number()` formatting, relabelled series added together, `client_label` | | `test_a_failure_never_starts_the_live_ledgers_day_one` | On a fresh live (`auto_day`) file a 401 leaves the origin unset (day 0); day 1 starts at the first request 20 h later; a later failure is filed under day 2; after reopening the file the origin is the same and the next request is still day 1 | | `test_old_ledger_gains_request_ids_and_the_failures_table` | An old file gains `request_id`, `client_request_id` and the `failures` table; the lookup uses `requests_request_id`; the stage list; `spend_since` adds answered and failed spend per client since a time | ### `tests/test_watch.py` The live gateway watchdog, offline through the seams of `ratchet.watch.Env` (`fetch`, `run`, `clock`, `sleep`, `out`): the module's import without site-packages; the settings and their checks; the signed alert, its retries and de-duplication; the classification table (a 502 with `upstream 402` is a different code from the same 502 with `upstream 503`); the restart rule on a fake `docker` (a teacher outage and a busy ledger never restart; a hung gateway and a broken ledger do; the grace, cooldown, hourly cap, pause and update lock; a stopped container is never started); the probe (the pinned-task preflight, the request's headers and size, retry once, the flag, no leak of a key or an upstream message); `flag_view`, `Runtime.live_check` and `inside flag|inspect` on a real ledger; `scripts/demo_check.py`; the cron file and cloud-init; and `update.sh`, `backup.sh` and `restore.sh` run with a fake `docker` and `sleep` first on `PATH` (the dry run, the backup-first order, the rollback, the restore only when the old image cannot open the ledger, no rollback for a teacher outage, a refused config check). ### `tests/test_readiness.py` The startup configuration check, the config check CLI, liveness and readiness. Live mode runs offline through `Runtime(transports=)`, `FakeOpenAIServer` and `httpx.MockTransport`; every test clears the shell's `RATCHET_*` variables first. | Test | Covers | |---|---| | `test_bad_setting_stops_startup_naming_the_variable` (29 cases) | Each bad value (an unparsable or out-of-range number, a bad mode, a non-http(s) URL, an empty Token Factory key, no API keys, a short admin key, release policy rules across fields, an unwritable or missing `RATCHET_DB`, a bad timeout, an unknown teacher failure mode, a bad 0/1 switch, a closed or unknown teacher, judge or fallback teacher) stops a live `Runtime` with a `ConfigError` (a `ValueError`) naming the variable, before the ledger file exists; with the variable fixed it starts | | `test_every_error_is_listed_at_once` | Six bad variables give one error listing all six (the unparsable one first, as `load()` read it); a refused teacher makes it a `LicenseError` too; outside `load()` a bad variable still raises at once with `var`, `value` and `expected`; `load()` keeps the default and collects the issue | | `test_readers_that_check_several_variables_report_each_one` | Sixteen bad variables across the release policy, the cost and generation policies, the logs, a breaker and the reserved settings, each reader of which stops at its first problem: one `ConfigError` names all sixteen; `ReleasePolicy.env_issues` alone; `config.defaults()` ignores the environment | | `test_config_check_and_startup_agree_on_every_variable` | In mock and live mode, every variable Ratchet reads set to `garbage` and to `-1`: `python -m ratchet.config check` exits 2 exactly when `Runtime` refuses to start, and a refusal is a `ConfigError` before the ledger file exists (a bad `RATCHET_TASKS_FILE` is the loader's `SpecError`) | | `test_misspelled_variable_warns_with_a_suggestion_and_strict_mode_stops` | `RATCHET_AUDIT_RAT` warns with "did you mean RATCHET_AUDIT_RATE?", an unrelated name without a suggestion, the VM's own variables not at all; the warning is kept and logged without the value; `RATCHET_STRICT_CONFIG=1` makes it fatal, a bad strict value is an error, strict and clean starts | | `test_warnings_for_settings_that_do_nothing` | A mock-only variable in live mode (not at its default), a 0/1 switch spelled otherwise, a student endpoint without a token, a live audit rate under 1%, an unpriced model, an empty key for a teacher off Nebius, an unpriced hosted fallback student: warnings, never errors, none in mock mode | | `test_release_policy_rules_across_fields` | `ReleasePolicy.validate`: the default and the edges are clean; a floor above the bar, 0 audits, 0 shadow answers and a tolerance over 0.2 are errors naming their variable, all four listed together; such policies can still be built | | `test_config_check_cli_exit_codes` | `python -m ratchet.config check`: 0 clean, 1 warnings, 2 errors (errors listed first, counts in the last line), 2 for warnings under strict; `--json`; usage errors; in its own process in live mode it reports every error and creates no ledger file | | `test_config_check_cli_loads_the_tasks_file` | The CLI reports a bad `RATCHET_TASKS_FILE` (every spec error), accepts a good one, refuses `clients` that name no API key, and reports a missing file | | `test_known_env_covers_every_variable_the_deploy_kit_sets` | Every `RATCHET_*` name in the deploy kit, the Dockerfiles, the Makefile, the scripts and the simulator is known, so the VM's `.env` never warns; mock-only and loose switches are in `KNOWN_ENV`; `DEPLOY_ENV` does not overlap it | | `test_update_script_checks_the_config_and_waits_for_readyz` | `deploy/vm/update.sh` runs the check before `docker compose up -d` and waits for `/readyz` after it; the Docker health check stays on `/healthz` | | `test_healthz_answers_with_every_worker_thread_busy` | `/healthz`, `/livez` and `/readyz` are coroutines and answer while AnyIO's thread limiter is fully borrowed | | `test_readyz_in_mock_mode_checks_the_ledger_only` | Mock mode: `ledger` and `config` (with the warning's message) only, the flags, `Cache-Control: no-store` | | `test_readyz_503_when_the_ledger_cannot_be_written` | A read-only connection and a closed file ledger are 503 with `read_only` and `closed`; a lock held past the probe's wait is 200 `degraded` (`lock_timeout`, `busy_runs` 1) and 503 when the next probe still finds it held, `ready` once released; a probe that outlasts its caller (`timeout`) counts as busy too; results are cached for 15 s on the runtime's clock; no failures rows | | `test_readyz_reports_missing_adapters_and_queues_their_reload` | Teacher `/models` 200 and a student `/models` without the canary's `student-v3`: 200 `degraded`, `missing_adapters=['student-v3']`, the reload sent to the endpoint; each key sent to its own endpoint; no URL or key in the answer; one probe per TTL, the student's failed result for 15 s; `ready` once the adapter is listed; an ok student result reused for 5 minutes | | `test_readyz_retries_a_timed_out_teacher_check_once` |: a first `/models` call that times out is tried once more; a second answer is `ready`, two timeouts are `degraded` (`ConnectTimeout`), a refused connection is not retried, and the teacher's wait stays under 5 s | | `test_readyz_follows_the_teacher` | An open teacher breaker is 503, unless a fallback teacher or a standing-in student keeps it 200 `degraded`; a 401 model list is 503; a 500, an unreadable list, a list without the teacher model and a connection error are `degraded` (the error named by class, no URL); an open student breaker is `degraded`; without a student endpoint the hosted fallback is reported | | `test_readyz_recovers_from_an_open_teacher_breaker_without_traffic` | An open teacher breaker is 503 while it waits; past its wait, with no request at all for up to a day, 200 `degraded` with `probe_due` while `/models` answers, 503 while the endpoint is unreachable; half-open is `degraded`; a failed probe is 503 for the doubled wait, a successful one `ready`; a fallback teacher's due probe counts on its own endpoint, and on the teacher's only when the endpoint answers | | `test_hung_endpoints_never_make_the_ledger_look_unwritable` | Teacher and student `/models` that hang hold both HTTP probe threads; the ledger's own thread still answers, 200 `degraded`, also once its result is stale; the hung runs are shared, not started again | | `test_probe_cache_shares_runs_times_out_and_names_classes` | `ProbeCache`: one run per name per TTL shared by concurrent callers, a caller's timeout that leaves the run going (and keeps its result), a raising probe named by class only, `clear()` | | `test_ledger_error_codes_and_model_lists` | `ledger_error`'s codes; `list_models` reads an OpenAI model list, sends no `Authorization` without a key and calls a `data` that is not a list unreadable | ### `tests/test_idempotency.py` Retry de-duplication: `Idempotency-Key` answers, the SDK's retry count and the ledger's seen window. The gateway tests run the mock gateway with a `RecordingProvider` teacher (cold start: the teacher answers and the example is stored); TTLs and windows run on a `FakeClock` (`Runtime(clock=)`); threads meet at known points by polling `IdemCache.stats()["waiting"]` or a `BlockingTeacher`'s events, never by sleeping for a guess. Metrics are process-wide, so the tests compare deltas. | Test | Covers | |---|---| | `test_keys_retry_counts_and_fingerprints_parse_strictly` | `check_key` (1 to 255 printable ASCII, trimmed), `retry_count` (a plain count, else 0), `fingerprint` (key order ignored; the task header and the body count) | | `test_idem_cache_keeps_expires_evicts_and_releases` | Replays, `KeyReused`, keys per client, a second `finish` ignored, the TTL on the cache's clock, a failure releasing the key, least-recently-used eviction that never drops a running entry, the byte budget (an oversized answer not kept), the sweep, bad arguments, `enabled` | | `test_idem_cache_byte_budget_counts_each_entry_not_only_its_body` | Each kept answer counts its body plus `ENTRY_OVERHEAD` (2 KiB), so tiny answers stop at the byte budget whatever `max_entries` allows; an answer whose body fits but whose entry does not is not kept | | `test_idem_cache_waiters_share_the_answer_or_take_over_after_a_failure` | A waiter gets the owner's answer even with TTL 0; another body refused while the first runs; after a failure one waiter takes over and the other gets its answer; a wait that runs out is `InFlight` | | `test_waiters_per_key_are_bounded` | Past `max_waiters` (default `MAX_WAITERS_PER_KEY`, 4) a duplicate is `InFlight` at once; another key is not affected; the slots are given back | | `test_seen_window_counts_once_per_window_per_sender` | `SeenWindow`: once per window per sender, a skipped repeat not restarting it, the oldest pair forgotten first | | `test_add_example_skips_retries_and_repeats_within_the_window` | `Ledger.add_example(count_seen=, sender=)`: no window counts every repeat; `count_seen=False` never counts and closes the refused INSERT's transaction; the window per sender ('' included), internal traffic (None) always counting, a repeat after the window, a new example starting its window; `ratchet_seen_skipped_total` | | `test_concurrent_retries_count_once` | Eight threads repeating one example for one client count once; eight clients count once each | | `test_old_ledger_gains_retry_count` | An old file gains `requests.retry_count` (old rows 0) | | `test_same_idempotency_key_replays_without_a_cascade` | One ledger row and one teacher call; the replay's body byte-identical (same completion id), `Idempotent-Replayed: true`, the answer's headers but no rate-limit headers, its own `x-request-id`; served after the daily quota is used up while a new key gets 429; another client's key space; outcome counts | | `test_concurrent_duplicates_call_the_teacher_once` | A duplicate that arrives while the teacher is blocked waits and gets the same answer (`joined`); one teacher call, one row | | `test_key_reused_with_another_body_or_malformed_is_a_400` | Another body or task header: 400 `idempotency_key_reused` with `X-Should-Retry: false`; malformed keys 400 `invalid_idempotency_key`; bad JSON never claims its key; `ratchet_rejections_total{reason="idempotency"}` | | `test_a_body_too_deep_to_fingerprint_is_a_400_and_claims_nothing` | Regression: a fingerprint that hits `RecursionError` is a 400 `invalid_json`, not a 500, with no key claimed, no teacher call and no failures row; around the real nesting limit (found by bisection without a key) every request with a key is a 400 | | `test_in_flight_wait_is_bounded_by_the_teacher_deadline` | After `RATCHET_TEACHER_DEADLINE_S` the duplicate gets 409 `idempotency_key_in_use` with `Retry-After: 1`; once the first finished, the key replays | | `test_idempotency_ttl_expiry_runs_again` | On a fake clock: a replay at 599 s, a second cascade at 601 s | | `test_streamed_answer_is_replayed_as_the_same_events` | A stream with `include_usage` replayed as the same events and headers; the same key without `stream` is reused | | `test_failed_first_request_releases_the_key` | A 502 is never kept: the retry with the key runs and answers | | `test_idempotency_off_ignores_the_header` | `RATCHET_IDEMPOTENCY_CACHE=0`: every request runs and the header is not checked | | `test_replay_log_line_says_it_replayed` | The chat log line's `retry_count` and `idempotent_replay`; the replay's `replay_of` names the request that ran, its `cost_usd` is 0; the key never logged | | `test_sdk_retries_do_not_drain_the_golden_set` | Three escalated retries with `x-stainless-retry-count: 1` leave a golden example at `seen` 1 and in `dataset.rows(ledger, "golden")`, rows recording the count; a keyless client's repeats within 300 s count once; another client's counts | | `test_the_same_request_minutes_later_still_counts` | Repeats ten minutes apart (fake clock) count, and the third moves the example to training (`FREQUENT`) | | `test_openai_sdk_retry_after_a_server_error_is_not_a_repeat` | The real OpenAI SDK's retry of a request that failed after its example was stored sends `x-stainless-retry-count: 1`, and `seen` stays 1 | | `test_internal_traffic_and_mock_mode_count_every_repeat` | Mock mode leaves the window off by default; calls outside a gateway request (no meta, or a meta whose request is not in flight) count every repeat; an explicit retry count is still honoured | | `test_retry_settings_are_checked_at_startup` | Mode defaults (300 s live, 0 mock) and each out-of-range value as one error naming its variable | | `test_bad_retry_setting_stops_the_gateway` | Unparsable and negative values stop a `Runtime` with one `ConfigError` naming both | ### `tests/test_offpath.py` Off-path quality checks: the sidecar, `Ledger.update_request` and the pending rows the gate ignores, and the router's audit and shadow after the response. Threads meet at known points through events (a `GatedProvider` teacher that answers only once released and says when a call started; a gated `FakeOpenAIServer` for the gateway), and every wait is bounded (`WAIT_S`), never a guess; `SideCar.drain` waits for the jobs. Metrics are process-wide, so the tests compare deltas. | Test | Covers | |---|---| | `test_sidecar_without_workers_runs_nothing` | `SideCar(0)` takes no places (and does not call itself full), drains and closes at once | | `test_sidecar_is_bounded_never_blocks_and_keeps_places_for_audits` | Refusals return at once; shadow work stops before the audits' places, an audit takes the last one; a cancelled place is free again; a slot ends once | | `test_sidecar_audit_places_scale_with_the_queue` | `audit_places` for queues of 1, 2, 10 and 1000 places and `reserve=0`; bad arguments | | `test_sidecar_bounds_the_request_text_its_jobs_hold` | `max_bytes` with its share kept for audits: a reservation past it is refused (and counted as full), a job larger than the whole budget is never taken, a running job holds its bytes, and a cancelled, finished, abandoned or refused job gives them back; `job_bytes` counts wide characters' storage | | `test_sidecar_jobs_run_in_the_submitting_context_and_failures_are_counted` | A job sees the submitter's context variables; a job that raises is counted and logged by its class, never its message | | `test_sidecar_close_drains_then_gives_up_and_refuses_late_jobs` | `close` waits, gives up on a queued job (returned by tag), refuses a job handed over after it, lets a running job finish; a second `close` gives up on nothing more | | `test_update_request_writes_each_verdict_once_and_adds_costs_call_by_call` | A verdict written only while pending, `stored` only raised, costs summed in order to the same float as Python's, a missing row, refused columns and values that write nothing | | `test_update_request_does_not_invalidate_the_hot_path_caches` | Finishing a row leaves the `requests` generation alone; any other rewrite moves it | | `test_the_ledger_connection_is_used_only_under_its_lock_once_a_sidecar_took_a_job` |: `LedgerGuard` (tests/conftest.py): no sidecar job, any use is fine; once one was queued, `ledger.db.execute` without `ledger.locked()` is refused with how to fix it (the writer connection's Python callback against the GIL deadlocks otherwise), under the lock or after `drain` it is not | | `test_live_quality_ignores_pending_audits_until_their_verdict` | Pending audits count in neither number; a verdict counts once written, a failed audit is forgotten; the day's audit agreement; a fresh tally reads the same | | `test_shadow_quality_ignores_pending_shadow_rows` | A pending shadow row counts once its verdict is written | | `test_an_audited_answer_returns_before_the_teachers_audit_ends` | With the teacher blocked, the answer returns with `overhead_pending`, the serving cost and no audit in the breakdown, the row pending; once released, the row has the verdict, the audit's cost and baseline, `stored`, and the gate counts it | | `test_offpath_checks_leave_the_rows_an_inline_run_writes` | 60 scripted requests inline and off the path (audits that agree, disagree, fail or measure nothing; shadows that agree, disagree and are arbitrated, fail to sample or meet a failed audit; escalations): every request row (costs to the last bit), example, failure and event, the failure counters, `version_live_quality` and `shadow_quality` are equal, and so is what `ratchet_cost_usd_total` and `ratchet_baseline_usd_total` count (once per cent: the ledger's totals) | | `test_a_drawn_audit_runs_inline_when_the_queue_is_full` | With every place held, the drawn audit runs before the response, counted per version, with a `sidecar_inline` event | | `test_a_long_request_past_the_byte_budget_audits_on_the_request_path` | A request whose text would pass the byte budget audits before the response; a short one still goes off the path; both rows end audited | | `test_a_shadow_queued_after_an_inline_audit_keeps_the_audit_as_its_reference` | The audit runs inline (queue full), a place frees up and the shadow goes off the path: judged against the counted audit's answer without an arbitration, and after a failed audit not arbitrated; one teacher call either way | | `test_a_shadow_without_a_place_is_skipped_and_counted_but_rides_with_an_audit` | A shadow without a place leaves no shadow record, counted per version with a `sidecar_dropped` event; with an audit drawn it rides in the audit's reserved place and gets its verdict | | `test_offpath_failures_follow_the_optional_call_rules` | A failed off-path audit: `audited` 0, no arbitration, the disagreement unarbitrated, `audit_errors`, an `audit_error` event and a failures row naming the request; a failed shadow sample: no shadow record, through the shadow's breaker | | `test_an_offpath_arbitration_gives_up_sooner_than_a_served_answer` | The off-path arbitration passes `deadline_s` 60; the inline one passes none | | `test_a_provider_deadline_shortens_its_retries_never_lengthens_them` | `OpenAICompatProvider.complete(deadline_s=)` on a fake clock: four attempts by default, two within 3 s, four with a longer deadline | | `test_requests_without_meta_keep_their_checks_on_the_request_path` | No meta (Scout, the simulator) or `offpath=False`: the checks run inline and `cost_usd` is the row's | | `test_a_ledger_error_gives_the_place_back` | A failing `log_request` cancels the reserved place | | `test_a_sidecar_that_stops_meanwhile_hands_the_job_back_to_the_request` | A job refused after shutdown began runs before `handle` returns | | `test_close_gives_up_on_queued_jobs_and_counts_them_per_version` | Jobs given up at shutdown: counted per version (audits and shadows), one `sidecar_dropped` event, rows left pending; the running job finishes; afterwards checks run inline and a closed sidecar is not counted as full | | `test_concurrent_requests_all_finish_their_checks` | Eight threads, 120 requests, four workers: every job done, no row pending, costs split consistently, the gate's count equal to the audited rows | | `test_sidecar_settings_defaults_and_bounds` | Mode defaults (4 live, 0 mock), every out-of-range value an error naming its variable (also through `Settings.validate`), an unparsable one collected by `load` | | `test_mock_mode_and_the_simulator_keep_every_check_on_the_request_path` | A mock-mode `Ratchet` has no workers; a live one has four, starts no thread before its first job, and keeps calls without meta inline | | `test_the_gateway_answers_first_and_its_shutdown_drains_the_queue` | A live gateway with a blocked teacher answers with `ratchet.overhead_pending`; `/ratchet/stats` `sidecar` and the `/metrics` gauge; leaving the `TestClient` runs the lifespan, whose drain completes the audit; the `chat` log line has `overhead_pending` and its cost plus the `offpath` line's is the row's | | `test_a_shutdown_past_the_drain_leaves_rows_pending_and_says_so` | `RATCHET_SIDECAR_DRAIN_S=0`: the queued job is given up with a `sidecar_dropped` event, its row stays pending and uncounted; the running one finishes | | `test_mock_gateway_answers_have_no_pending_overhead` | Mock mode: no `overhead_pending`, an idle `sidecar` block | | `test_a_demo_reset_stops_the_old_sidecar` | `Runtime.reset` closes the old cascade's sidecar and builds a new one | | `test_offpath_job_logs_one_line_without_content` | One `offpath` log line naming the row, its verdict and the cost it added, never the request's text; `ratchet_sidecar_jobs_total{outcome="done"}` | ### `tests/test_limits.py` Spend and rate limits. Gateways run as live gateways through `Runtime(transports=)` with a `FakeOpenAIServer` teacher whose answers cost a known amount (10 prompt and 5 completion tokens at the teacher's price; cold start, so the teacher answers every request), or in mock mode; the spend meter runs on its own with an injected wall clock (`FakeClock`), the rate limits on the runtime's monotonic `FakeClock`. Concurrency tests block a teacher on a `threading.Event` and send the second request from the main thread while the first waits in a background thread, both through one `TestClient`; the stalled-body test calls the ASGI app directly under `anyio.run`, with a `receive` that waits on an event between body chunks. Live `Settings` pass every limit explicitly (`_live`), because live mode's per-IP default would refuse a burst from one test address. | Test | Covers | |---|---| | `test_key_daily_usd_cap_refuses_once_spent_and_other_keys_go_on` | A cap of 2.5 requests' worth: three 200s with `X-Ratchet-Spend-Today-USD` and `X-Ratchet-Budget-Remaining-USD` counting, the third overshooting by its own cost; the fourth a 429 `insufficient_quota` / `budget_exceeded` with remaining 0, `Retry-After`, `X-Ratchet-Budget-Reset`, `X-Should-Retry: false`, a `budget` failures row and a `budget` rejection; another key without a cap gets a 200 and no spend headers; no requests row for the refusal | | `test_monthly_cap_resets_on_the_first_and_names_the_later_reset` | On Oct 30 a spent daily and monthly cap: `Retry-After` runs to Nov 1 (the later reset); still refused on the 31st, a new day; answered on the 1st | | `test_spend_survives_a_restart_on_the_same_ledger` | A second `Runtime` on the same file reads today's requests, dollars and failed spend (a `failures` row) per key and for the gateway, and refuses the spent dollar cap and the used request quota | | `test_failed_requests_spend_counts_against_the_key` | An unsure student, then a teacher 502: the samples' cost counts against the key's dollars (no request counted), at once and after a restart | | `test_utc_rollover_resets_the_counters_and_a_clock_moved_back_reads_the_ledger` | 23:59 on Oct 31, then 00:01 on Nov 1: day, month and gateway counters start from zero; a new day within the month keeps the month; a clock moved back to Oct 31 reads that day's gateway rows from the ledger, leaving out a row without a gateway request id; `month_bounds` | | `test_meter_ignores_bad_costs_and_counts_spend_without_a_request` | Negative, NaN and infinite costs count as 0 (the requests still count); `add(requests=0)` adds dollars only | | `test_budget_alerts_fire_once_per_share_also_after_a_restart` | 50, 80 and 100% of a key's $5.00 daily cap and 50% of the gateway's $10.00, each once and in order (two shares at once when one request crosses both); a restarted meter on the same ledger repeats none; one started with a lower gateway budget writes the three shares it already passed; the next UTC day starts over; `money` | | `test_alerts_name_a_tiny_cap_exactly_and_read_it_back` | `money` writes caps of $0.0000001 and $0.0000375 with every digit (never `$0.` or an exponent); a restarted meter reads those alerts back and writes none again | | `test_an_alert_that_cannot_be_written_is_tried_again` | A failing event write is retried on the next add, never raised | | `test_quota_path_issues_no_count_query` | With a sqlite trace callback: `SpendMeter.usage` issues no statement, and a chat request under a daily quota and dollar budget issues no `COUNT(` of any kind | | `test_student_only_serves_a_confident_student_and_refuses_what_needs_the_teacher` | The first request's audit spends the gateway's budget; then a confident student answer is a 200 with no audit and no shadow (route `student`, `audited` 0, no shadow version, one student call); an escalation is a 429 `gateway_budget_exceeded` that names no amount, with one `budget` failures row carrying the samples' cost; a pinned task and a request with `tools` get the 429 without any model call; a `budget` event at 100% | | `test_block_mode_refuses_every_chat_request_until_midnight` | Past the gateway's budget every key's chat request is a 429 `gateway_budget_exceeded`; `/v1/models` still answers; `/ratchet/stats` `budget` block | | `test_a_scrubbed_public_read_names_no_key_and_no_cap` | A public live dashboard read anonymously after a key's cap and the gateway's budget were reached: every `budget` event reads `client-1 reached N% of its daily budget` or `the gateway reached ...` with no amount, no key name or `$` anywhere, `daily_cap_usd`, `budget.daily_usd` and `remaining_usd` null; a dashboard key reads the events and caps as written; `public_alert` on a detail that is not an alert and on a name that is no longer a key | | `test_scout_spend_stays_under_its_own_guard` | A Scout run spends money that neither the meter nor a fresh one counts; `/v1` spends the budget, the next chat request is refused, Scout still runs | | `test_key_limits_override_the_defaults_and_bad_specs_stop_startup` | A per-key `daily_requests` override against the default; `parse_key_limits` with spaces and every field; 22 malformed specs, each naming `RATCHET_KEY_LIMITS` and never echoing a pasted key (`acme:` in `RATCHET_API_KEYS`' syntax, a key as a field, a value or a task list), and a 400-digit dollar cap that would read as infinite; a `Runtime` naming an unknown key does not start | | `test_task_scope_is_403_and_lists_only_the_scoped_tasks` | A pinned task and `default` outside the scope are 403 `task_not_allowed`; an unknown-task 400 lists only the scope's tasks; the refusals used no quota; another key sends the pinned task | | `test_store_zero_keeps_a_keys_requests_out_of_the_training_set` | `store=0`: the key's teacher answer is served with `stored` 0; another key's is stored | | `test_rpm_bucket_refuses_past_the_burst_and_refills_on_the_clock` | 60 a minute: ten 200s with the remaining count falling to 0; the 11th a 429 `rate_limit_exceeded` with `Retry-After: 1`, reset `10s`, no `X-Should-Retry`, a `rate` row and rejection; another key unaffected and without rate headers; one more a second later; the burst back a minute later | | `test_daily_quota_headers_win_over_the_rate_headers` | A key with both: the 200's `X-RateLimit-*-Requests` describe the daily quota | | `test_concurrency_one_refuses_a_second_request_while_the_first_runs` | `concurrency=1`: while the first request waits on the teacher, the key's second is a 429 with `Retry-After: 1`; another key is answered; after the first ends the slot is free again | | `test_max_inflight_answers_503_instead_of_queueing` | `RATCHET_MAX_INFLIGHT=1`: a second request is a 503 `overloaded` with `Retry-After: 1`, an `overloaded` rejection and a `rate` row; the in-flight count returns to 0 | | `test_a_stalled_body_holds_no_in_flight_slot` | Straight into the ASGI app with `RATCHET_MAX_INFLIGHT=1` and `concurrency=1` for one key: two requests send headers and 5 bytes of chunked body, then stall; the in-flight counts stay 0 and a whole request from each key is a 200; the stalled bodies, released one at a time, are 200s too (the code before the fix refused the second stalled request at once) | | `test_per_ip_rate_on_a_gateway_without_keys` | Live default 120, mock 0; an open gateway with 6 a minute: the second request from one address (the first `X-Forwarded-For` hop) is a 429 with `Retry-After: 10` that does not echo the address; another address is answered; ten seconds later the first is again; with API keys the per-IP rate is off | | `test_token_bucket_concurrency_and_limiter_units` | `TokenBucket` (burst, refill, a clock moved back), `burst`, `RateLimiter`'s LRU bound and a new bucket for a new rate, `Concurrency`, `Limiter.rate` (takes a token, holds no slot; refused when empty) and `Limiter.hold` (the in-flight cap, then the key's; a refusal gives back the slot it took) | | `test_limit_settings_are_checked_at_startup` | `limit_policy` reads the mode in any case and the shares sorted, once; thirteen bad values (a dollar default past a billion and a NaN gateway budget among them) each name their variable | | `test_stats_report_each_keys_spend_and_cap_and_the_security_switches` | `clients[].spent_today_usd` and `daily_cap_usd`, the three `security` keys and the `budget` block; key names only, never keys | | `test_router_refuses_teacher_work_under_student_only_without_a_gateway` | `RequestMeta(student_only=True)` on `Ratchet` itself: a pinned task raises `BudgetExceeded` (`pinned`) before any call and writes nothing; an unsure student raises it (`escalation`) after its samples with a `budget` row (429, the samples' cost) | | `test_daily_quota_holds_room_for_requests_in_flight` |: a quota of 3 with three requests held inside the teacher (none counted yet, `meter.held` 3): a fourth is 429 `daily_quota_exceeded` at once, another key still answers; when the three finish the count is 3, nothing is held and the next is refused | | `test_a_failed_request_gives_its_quota_room_back` |: a teacher error leaves nothing held and nothing counted, so a quota of 1 still admits the next request | ### `tests/test_admin.py` The admin API and CLI and compare-and-set release moves. HTTP tests run live gateways through `Runtime(transports=)` with `FakeOpenAIServer` teacher and student; the races run two `Ledger` instances on one file, as the gateway's tick and the cron tick do, and make the other process act between one gate's reads and its write (`_first_call_runs`), then repeat the race with real threads. | Test | Covers | |---|---| | `test_transition_moves_a_version_only_from_where_the_caller_saw_it` | A stale share or state moves nothing and writes no event; the right one moves, writes its event and starts a new stage; the same decision twice is refused; a move into shadow or canary while another version is in flight is refused; a NULL share reads as 0; a move to live, an unknown state and a share past 100 raise | | `test_promote_live_retires_the_old_live_version_in_the_same_transaction` | A caller that saw no live version promotes nothing; the promotion retires the old live version, `live_retired` before `live`; the same promotion again writes nothing; promoting a live version raises | | `test_a_transaction_commits_its_moves_and_events_together_or_not_at_all` | An exception inside rolls back the move and its event, the listener hears nothing and the routing snapshot built inside is dropped; nested transactions join; listeners hear events after the commit; a writer that commits by itself inside raises | | `test_two_ticks_on_one_ledger_file_advance_a_canary_once` | Gate B has read a ready canary when gate A advances it: one `canary 50%` event, 50% on both connections; six threaded races, one advance each | | `test_a_promotion_race_leaves_exactly_one_live_version` | Two gates promoting one canary: one live version, one `live` and one `live_retired` event; an admin (CLI, another connection) promoting while the gate is about to: one live version, no gate events, one `admin_actions` row | | `test_admit_leaves_a_version_an_admin_moved_alone` | A candidate an admin promoted to canary while it was scored stays a canary, with a `conflict` event | | `test_every_admin_route_needs_an_admin_key` | All 16 `/ratchet/admin/*` routes have the `admin` level; no key 401, an API key and a dashboard key 403, nothing changed; live mode without admin keys 403 `admin API disabled` on each; an admin key does not open the mock-only test controls; no key in the ledger | | `test_canary_rollback_over_http_writes_the_move_an_audit_row_and_an_event` | The answer's before and after; the canary `rolled_back`, live untouched; the `admin_actions` row (actor `api:ops`, reason, before, after, the response's `x-request-id`); the `admin` event; a second rollback 409 and nothing written; the audit log and its `limit` bounds | | `test_promote_to_live_over_http_retires_the_old_live_version_atomically` | The next canary stage by default, an unchanged share answered `changed: false` with no row, then live with both versions in before and after and exactly one live; the event; `GET /ratchet/admin/versions` with golden numbers and stage audits | | `test_refused_promotions` (13 cases) | A rejected version to live or canary without force, a rolled-back version or a candidate without force, live to canary, shadow to live (409); a share of 100 or a string, a share for live, an unknown `to`, a non-boolean `force` or `load_adapter`, an unknown key (400); nothing changed or recorded | | `test_the_transition_table` | Forced returns of a rejected, rolled-back, retired and candidate version to canary, refused while another is in flight; no eval, never; shadow to canary by default 10%, then 50%, then the last stage refused; rollback of live refused with a hint; retiring live; a retired version cannot be retired; 404 and 400 for a bad id | | `test_a_reason_is_required_checked_and_redacted` | Missing, short, blank, long, two-line and non-string reasons and malformed actors (one that looks like a key) refused with nothing changed; an e-mail address in a reason stored as `[EMAIL]`, nowhere in the ledger | | `test_load_adapter_before_a_live_promotion` | A failing `/v1/load_lora_adapter` is a 502 naming the class and status, nothing changed or recorded; a working one is called with the adapter's name, path and the endpoint key before the move; 409 without a student endpoint; ignored in mock mode | | `test_student_disabled_sends_every_request_to_the_teacher_and_stores_as_usual` | With a live version and a shadow: route `student_disabled`, no student or shadow resolution, the teacher's answer stored, no version on the row; switched back on, the student serves | | `test_student_kill_switch_over_http` | `disable` answers before and after, the next request is the teacher's with no student call, a second `disable` writes nothing, `controls` and the by-task routes in `/ratchet/stats`, `enable`, `GET /ratchet/admin/flags` with who set it | | `test_a_runtime_pin_keeps_a_task_on_the_teacher_and_out_of_training` | A pinned task (by name and as a `Task` object) takes route `pinned` with no student call and nothing stored; its stored rows leave the training set (`excluded_pinned`); its `spec_hash` is unchanged; a spec-pinned task's pin is a no-op and its unpin 409; unknown and invalid names; unpinned, the registered task is back and so are its rows | | `test_task_pins_over_http_show_in_stats_and_the_task_list` | `POST .../pin` sends the next request to the teacher; `controls.pinned_tasks`; `runtime_pinned` true with the spec's `pinned` false; 404 for an unregistered name; `DELETE` restores the student | | `test_put_audit_rate_changes_behaviour_without_a_restart` | Rate 0, then 1 (every answer audited), then null (the configured rate again); four bad values 400; `audit_rate` required; 0 during a canary 409, accepted with `force`; with audits off a new canary needs force | | `test_training_paused_skips_the_night_before_anything_else` | A paused night trains and evaluates nothing (reason `training paused`, a `skip` event) though there is plenty of data; resumed, it trains | | `test_frozen_releases_hold_every_promotion_but_never_a_rollback_or_an_alert` | Frozen: a cold start shadows the teacher, a ready shadow is held, then promoted on its evidence once unfrozen; a good canary does not advance; a bad canary is rolled back and the live alert fires while frozen; an audit rate of 0 holds a ready shadow too | | `test_a_frozen_gate_in_a_corrupt_deploy_simulation` | `sim.simulate.run` on a ledger file, freezing from another connection when v1 becomes a canary (simulation): deployed corrupted, v1 is still rolled back and nothing is promoted after the freeze; deployed sound, it stays at 10% instead of advancing, and the nights skip | | `test_the_tick_endpoint_runs_the_gate_and_records_who_ran_it` | A forced tick rolls a bad canary back and answers before, after and `moved`; the `rollback` then the `admin` event; a tick without a body; both recorded, the second without a reason | | `test_cli_changes_reach_a_running_ledger_through_data_version` | On one file, a running ledger with cached switches and routing sees the CLI's `flag`, `rollback` and `pin` at once; `audit-log --json` with actor `cli:` and no request id; `versions` output; exit 1 for a refused move, 2 for bad values (2, nan, a short reason); `default`; a missing ledger is refused and not created | | `test_bad_flag_rows_written_by_hand_never_fail_a_request` | Unparseable and unknown `runtime_flags` rows read as the defaults; `set_flag` refuses an unknown name and NaN; `admin_actions` refuses a boolean limit | | `test_requests_from_many_threads_while_an_operator_flips_switches` | Four threads, 100 requests, while the kill switch and a pin flip ten times: no error, every route `student`, `student_disabled` or `pinned`, 100 rows | | `test_admin_tables_exist_on_an_old_ledger_file` | An old file gains `admin_actions`, `runtime_flags` and `task_overrides` on open | ### `tests/test_calibration.py` Per-task calibration when several tasks share one adapter. Scripted golden rows set each answer's confidence and correctness exactly; the simulation numbers come from mock models. | Test | Covers | |---|---| | `test_cascade_eval_per_task_thresholds` |: an easy and a noisy task on one adapter. Pooled, the global threshold (0.85) serves the noisy task's wrong answers (97% overall); the records give the easy task 0.85 and the noisy one 0.95 (precision 1.0, coverage 0.85), with their row counts, agreement, spec hashes and `calibrated`. When nothing reaches its target the noisy task gets 1.01; a `policy.target_precision` of 0.8 gives it 0.85 and a new spec hash | | `test_single_task_eval_identical_to_global` (3 cases) |: with one task, its record holds exactly the global threshold, precision, coverage, row counts and agreement, for `min_task_n` 1, 20 and 1000 (`calibrated` true, true, false) | | `test_task_with_few_calibration_rows_is_never_below_global` |: with fewer than `min_task_n` calibration rows a task takes the higher of its own threshold and the global one (`calibrated: false`), the global one with no calibration row, 1.01 with no held-out row; enough rows make it its own | | `test_task_without_test_rows_is_teacher_only` |: a task whose one golden row calibrates (none held out) gets 1.01, and so does a task whose references are all invalid; the router escalates it without a student call (route `task_disabled`) while serving the other task | | `test_router_uses_task_threshold` |: at confidence 0.9 task A (0.7) is served, B (0.95) escalates and C (no record) uses the global 0.8; without a task the global threshold, for the fallback model `RATCHET_CONFIDENCE_THRESHOLD`; each task of a version has its own fingerprint | | `test_disabled_task_skips_the_student` |: a task the gate disabled on the serving version goes to the teacher without a student call (route `task_disabled`, no student cost, confidence null); with the teacher down and `RATCHET_ON_TEACHER_FAILURE=serve_student` another task degrades but the disabled one gets the teacher's error; the shadow still runs; the hosted untuned fallback model ignores the records; a failing resolver leaves the task held back | | `test_stale_spec_hash_threshold_ignored` |: a record under an old spec hash gives 1.01 (neither its own nor the global threshold), and `stale_tasks` names it; a current record wins over a stale `spec_hashes` entry | | `test_versions_without_task_eval_unchanged` |: a version without records keeps its one threshold for every task; `spec_hashes` alone still sends a changed task to the teacher; the gate checks only the global numbers | | `test_threshold_reads_malformed_records_safely` |: records that are not mappings are ignored; a non-numeric or NaN threshold, a missing hash or a disabled record give 1.01; a global threshold that is not a number serves nothing instead of failing the request | | `test_admit_disables_regressed_task_but_ships_version` |: 48 of 50 right (96%) is above the 95% floor but more than 2 points behind live's 100%: the task is disabled (threshold 1.01, the reason, a `task_disabled` event) and the version shadows; below the floor (70%) it is disabled with nothing live | | `test_per_task_gate_holds_each_task_to_its_floor` |: the global gate's point-estimate rule on each task's own rows: 36 of 40, 9 of 10, 18 of 20 and 91 of 100 right all fail a 95% floor next to a task at 100% (a Wilson upper bound passed them all), and a cold start ships the good task alone; exactly 95% passes; a task that answers none of its held-out rows fails as unmeasured | | `test_admit_reject_mode` |: with `task_mode="reject"` the same candidate is rejected, naming the task, and nothing is disabled | | `test_per_task_gate_needs_evidence` |: no failure under `min_task_test_n` rows, for a dip within the tolerance (49 of 50 against live's 100%), for a task already at 1.01, or against a live record that serves nothing, has another spec or too few rows; 97% passes the 95% floor but not a policy floor of 99% | | `test_admit_rejects_a_version_that_would_serve_too_little` |: disabling a task that leaves 12.5% of the held-out rows answered, or per-task thresholds that answer none, reject the candidate instead of leaving it in shadow or canary | | `test_disabled_task_survives_rescore` |: the champion re-check overwrites the eval but keeps the disabled tasks disabled, also one the re-score has no record for | | `test_per_task_live_alert_once` |: 12 of 20 audits on one task of the live version raise one alert, not repeated by later ticks or a new gate; `has_event` compares a prefix (a `_` in a task name is not a wildcard); the grouped query runs again only after a new audit; the canary gets its own alert | | `test_per_task_alert_is_written_once_across_processes` |: two `Ledger` connections on one file; the second checks for the alert while the first is writing it and waits on the write lock, then finds it: one alert, never two (`Ledger.event_once`: `BEGIN IMMEDIATE`, a check and the insert, `COMMIT`); another task or kind is another event; a detail outside its prefix is refused | | `test_per_task_alert_leaves_single_task_versions_to_the_version_alert` |: a version whose audits are all one task's gets only the version-level alert | | `test_stats_by_task_has_audit_agreement` |: `audits` and `audit_agreement` per task; `threshold`, `precision`, `coverage`, `calibrated` and `disabled` from the live version's records (the global threshold for a task without one, 1.01 for a disabled task or a changed spec, null for a pinned task or without a live version) | | `test_bom_eval_carries_task_records` |: a night's BOM carries the eval's per-task records, and `spec_hashes` matches them | | `test_task_calibration_settings` |: `RATCHET_MIN_TASK_GOLDEN` (a whole number, 1 or more, reaching `LiveEvaluator`), `RATCHET_TASK_MODE` (case-insensitive, empty is the default, a bad value names the variable) and `RATCHET_MIN_TASK_TEST_N` | | `test_per_task_calibration_can_be_switched_off` |: `RATCHET_PER_TASK_CALIBRATION=0` makes `cascade_eval` write no records and leaves the pooled numbers as they are; the simulator, the mock gateway and `python -m ratchet.live night` pass it on; only `0` switches it off | | `test_ledger_merges_a_task_record_atomically` |: `merge_task_eval` merges into one record, keeps the others, and rolls back on an unknown version | | `test_mixed_workload_calibrates_each_task` |, US-4: a 6-day mixed simulation (mock models) with `RATCHET_PER_TASK_CALIBRATION=1`: every trained version has a record for `ticket-triage`, `source-triage` and `page-read` under their current specs, and the student still takes over (over 80% on the last day) at audited agreement of 90% or more | ### `tests/test_mock_world.py` The generic mock world: every request the ticket and research worlds do not know gets a deterministic answer from mock models. Schemas are checked with a small validator in the test file (the subset the mock fills), since `jsonschema` is not a dependency; when it is installed, one test checks the same instances with it too. The simulation numbers are trends from mock models. | Test | Covers | |---|---| | `test_lookup_tries_tickets_then_research_then_the_generic_world` |: a test run ticket keeps its world whatever it is sent with (a schema too), a research request as well; `hello`, a system-only request, an image-only request and an empty list get a generic item that answers, and each raises `KeyError` under `strict` (RATCHET_MOCK_STRICT); a schema or tools pick the `schema` or `tools` sub-world | | `test_generic_items_are_deterministic_and_keyed_by_what_a_model_sees` |: the same request gives the same template, labels and answer whatever task it is sent under; another system prompt gives another template; 60 texts spread over all four framings | | `test_free_text_world_restates_the_prompt` |: `[mock teacher] You asked: "hello". ...`; a wrong framing differs; a long prompt quotes 12 words and `...`; a 1,000,000-character prompt in under 0.5 s; text parts joined; json_object gives `{"answer": ...}` | | `test_instance_from_schema_fills_the_structured_outputs_subset` |: 200 seeds of a schema with an enum, a boolean, bounded and exclusive (draft 4 and draft 6) integers and numbers, `multipleOf`, formats (date, date-time, UUID), min and max lengths, a nullable type, `anyOf` with null, `const`, arrays with prefixItems, minItems and uniqueItems, `allOf`, local `$ref`: every instance validates, the same seed gives the same instance, and enums, booleans and integers are drawn | | `test_instance_from_schema_ends_and_never_raises_on_hostile_schemas` |: a recursive `$ref` ends within `MAX_DEPTH`, a `$ref` loop and 50 × 50 × 50 nested arrays end, bounds of ±1.7e308 and 10^400 give finite JSON, all in under 1 s; 1,500 seeded junk schemas, also as a json_schema and as tool parameters (with `tool_choice` required), never raise, and every label space keeps a wrong value to draw | | `test_instance_from_schema_agrees_with_jsonschema` |: with `jsonschema` installed (skipped otherwise), the subset and the workload schemas validate under Draft 2020-12 for 100 seeds each | | `test_schema_answers_stay_valid_for_any_labels_and_wrong_ones_change_enums_first` |: 300 schema-workload items: the teacher's answer and a guessing student's both validate; labels list enums first; every schema task appears; a corrupted deploy's three samples agree on enum values that are all wrong; the mock teacher writes the item's answer | | `test_mock_teacher_calls_the_catalogue_with_arguments_read_from_the_prompt` |: `lookup_order` with `order_id` 48213, no content, finish `tool_calls`, a stable `call_` id; `refund` with the amount and a reason for each wording (`damaged`, `duplicate_charge`, `not_received`, `wrong_item`, `changed_mind`, a `$` amount too); a plain reply without an order number; after the tool's result a final text answer restating it (inside `{"answer": ...}` for json_object) | | `test_tool_choice_and_client_tools` |: a forced `tool_choice` wins, `none` gives free text, `required` calls a tool even without an order number; a client's own tool gets arguments that validate against its schema; an integer `order_id` gets an integer | | `test_mock_student_tool_calls_improve_with_training` |: on 300 tools-workload requests a student trained on 200 examples of the template makes the teacher's call (name and arguments) over 75% of the time, an untrained one under 60%; a tool-call sample lists no content tokens | | `test_mock_fleet_trains_and_evaluates_on_generic_and_ticket_rows` |: a ticket row, a free-text row, a schema row and a tool row with their stored conditions, a system-only row and a content-parts row train without raising (also with `strict`), every template is counted and the golden eval scores all six; a strict fleet's live student raises `KeyError` for `hello` and still answers a ticket; a lenient shadow student answers `hello` on staging | | `test_strict_teacher_and_unreadable_scout_writer_requests` |: a Scout writer request without readable facts gets a generic answer, and `KeyError` from the strict teacher, as `hello` does | | `test_mock_strict_setting` |: `RATCHET_MOCK_STRICT` `1` and `0` | | `test_schema_workload_student_takes_over` |: `--workload schema`, 4 days × 300 (seed 7, 5 audits and 40 shadow answers to move a release): no student on day 1, over 50% on day 4 and above day 2, audited agreement of 90% or more on the last day; every stored example is one of the three schema tasks and keeps its json_schema condition | | `test_tools_workload_is_answered_by_the_teacher_alone` |: `--workload tools`, 3 days × 300: every request goes to the teacher alone (route `tools`, `uses_tools` 1, nothing stored), the student share stays 0 and `stats_by_task` counts 900 `tool_requests` for `order-support` | | `test_demo_day_runs_the_schema_and_tools_workloads` |: `gateway.WORKLOADS` is `sim.world.WORKLOADS`; `/ratchet/demo/day?workload=schema` and `tools` answer 200 with `simulation: true`, their tasks appear in `/ratchet/stats`, and an unknown workload is a 400 | | `test_sdk_gets_answers_and_structured_outputs_from_the_mock_gateway` |, US-1: through the OpenAI SDK, `hello` gets a 200 restating it, and `client.chat.completions.parse(response_format=Claim)` (a Pydantic model with a `Literal`, a bounded integer, a date, a nested list and an optional field) returns a parsed `Claim` whose raw content validates against the schema the SDK sent | | `test_a_mock_tool_call_passes_through` |: with the default allowlist the mock teacher gets the tools and its call comes back as `message.tool_calls` (null content, `finish_reason: "tool_calls"`), the same call the mock teacher makes directly; with tools kept from the teacher (`RATCHET_TEACHER_PARAMS=max_tokens,stop`) it answers in prose; teacher-only and nothing stored either way | ### `tests/test_structured.py` Structured outputs: the JSON Schema check (`ratchet/jsonschema_lite.py`), fields derived from a schema and auto tasks (`ratchet/schema_tasks.py`), the contract a `response_format` binds to a request's task, one repair of a teacher answer that breaks it, the json_object fallback, and what clients see. Models are `RecordingProvider`s or `FakeOpenAIServer`s behind `httpx.MockTransport`. | Test | Covers | |---|---| | `test_validator_checks_required_enum_and_additional_properties` |: `required`, `enum` and `additionalProperties: false`, as schema paths that never quote a value or a key from the answer (a secret placed in both is absent); `max_errors`; an `additionalProperties` schema checked under `.*`; a key that is not an identifier written `$["a-b"]` | | `test_validator_types_and_nullable` |: type lists with null, JSON's integer (`1.0` is one, `true` is not), NaN and infinity refused, an unknown type name ignored, `anyOf` with null | | `test_validator_resolves_refs_and_guards_against_loops` |: a recursive `$defs` tree, a `$ref` loop that reaches no value, an unresolved or remote `$ref` (an error, never valid), `~1` escapes, `#` | | `test_validator_combinators_const_items_and_bounds` |: `oneOf` (an integer matches both `integer` and `number`), `allOf`, `const` and `enum` with JSON equality, `items`, `minItems`/`maxItems`, `maxLength`, exclusive and inclusive bounds on a 10^400 integer; `pattern`, `format` and `multipleOf` ignored | | `test_validator_tuples_and_pattern_properties` |: `prefixItems` with and without `items` (also `items: false`), the old list form ignored; `additionalProperties: false` not enforced next to `patternProperties` | | `test_validator_never_raises_and_is_bounded` |: `MAX_DEPTH` on a recursive schema; an exploding `anyOf` over 200 items stops at `MAX_STEPS` as a mismatch in under 20 times an ordinary 20,000-item check (timed in the same rounds, so a slow or traced run does not fail it); junk schemas never raise | | `test_json_equal_is_json_equality` |: `1 == 1.0`, `true != 1`, nested values | | `test_derive_takes_enums_and_ignores_free_text` |: the spec's example gives `category` and `priority` labels and ignores `reply`; required follows the schema; no object root, or nothing comparable, gives no fields | | `test_derive_resolves_refs_nullable_nested_records_and_formats` |: a Pydantic-shaped schema: `bool`, nullable `date`, `date-time`, exact `text` for `email`, `number` with its relative tolerance, `records` through a `$ref` (free text left out), sets of labels, booleans and dates, a nullable parent's children not required, an integer enum with null; consts, integer lists, bad names and two-way `anyOf` left out; the task accepts what the schema accepts; key order changes nothing | | `test_an_enum_of_numbers_takes_what_the_schema_takes` |: an enum of numbers (and an array of one) is an exact `number`, so `2.0` meets the field as it meets the schema; the schema still decides the allowed values; a mixed enum stays a label | | `test_derive_stops_at_four_levels_and_sixty_four_fields` |: four levels deep, 64 fields wide, a records field counted with its sub-fields | | `test_x_ratchet_annotations_include_exclude_and_are_stripped` |: `ignore`, `text`, `ordered` (a `ranking`, ordered `records`), `abs_tol`; unknown hints ignored; `strip_annotations` and `forwardable` remove every `x-ratchet` key at any depth but keep a property named that way, and copy nothing when there is none | | `test_derived_tasks_round_trip_through_a_spec` |: an auto task (source `auto`, never pinned) reads back from `taskspec.document` unchanged, for the test schemas, every mock-world schema and an annotated one; a pinned auto task is refused | | `test_auto_task_name_is_stable_across_key_order` |: `auto--<16 hex>` (at most 62 characters), the same for a reordered schema, another for another schema, 40 slug characters at most, `schema` without a name | | `test_contracts_and_bound_tasks` |: `contract_for` (cached per schema), `Task.bound` (same `spec_hash`, equal, not registrable), `check` reasons, `clean_answer` serving the JSON inside reasoning, a sentence and a fence, an array root | | `test_for_row_grades_a_stored_row_under_its_stored_format` |: `tasks.for_row` from a dict, JSON text, none and bad JSON, and on a ledger row | | `test_auto_task_is_registered_persisted_and_never_pinned` |: the first request registers and stores the auto task (`updated_by` null, the document and hash), one `task_registered` event, the example under its name with the client's format; a second request only looks it up | | `test_a_registered_task_beats_the_schema` |: `ticket-triage` named with a schema stays itself, the schema forwarded and still checked (an answer outside it is served with `schema_valid` false and not stored) | | `test_a_name_the_client_makes_up_becomes_an_auto_task_under_distill_only` |: an unregistered name becomes an auto task under `distill` (a later schema with other fields gets its own auto task and an `auto_task_collision` event), stays unknown under `teacher` (route `unknown_task`, nothing stored); no name gives the auto task under any policy | | `test_a_client_cannot_choose_the_fields_another_clients_answers_are_compared_on` |: a client naming another client's computable auto name first with a weaker schema gets its own schema's task; the victim's auto task compares the refund; naming it later with another schema is a collision; a made-up name held by a weaker schema is not used for a stronger one | | `test_an_auto_name_held_by_other_fields_stores_nothing` |: an auto name registered with other fields gives an unregistered task that stores nothing, an `auto_task_collision` event, and a 400 under `reject` | | `test_a_schema_without_comparable_fields_follows_the_unknown_task_policy` |: a free-text schema is not registered; `distill` stores it, `teacher` routes it `unknown_task`, `reject` raises `UnknownTask` | | `test_auto_tasks_stop_at_the_cap_and_say_so_once` |: past `RATCHET_MAX_AUTO_TASKS` new schemas get unregistered tasks; one `auto_task_cap` event, also across two processes on one ledger; 0 turns them off (`default` or the named task, held to the schema, never refused for it); a bad cap stops startup | | `test_an_auto_task_whose_schema_holds_personal_data_is_not_stored` |: an enum of email addresses never reaches the `tasks` table; an `auto_task_skipped` event | | `test_auto_tasks_come_back_at_startup_and_night_school_grades_under_the_schema` |: `taskspec.install` registers a stored auto task again; `dataset.rows` and `cascade_eval` hold each row to its stored schema (the student sampled under it), so an answer whose fields agree but which lacks a required free-text key gets no vote and is no training target (`skipped_invalid`) | | `test_concurrent_requests_register_one_auto_task` |: 8 threads on a new schema register one task, store one row and write one event | | `test_invalid_student_samples_get_no_vote_and_escalate` |: confident samples outside the enum have confidence 0 and escalate; matching ones are served; json_object refuses a non-object | | `test_json_object_answers_are_served_as_the_json_they_hold` |: json_object serves and stores the object inside reasoning and a sentence, as written; text with no JSON is served as written with `schema_valid` false and not stored | | `test_an_answer_is_checked_against_its_schema_once_per_request` |: three samples, an audit and the served choice with one text run one schema check (`Contract.problem`'s memo) | | `test_schema_mismatches_share_one_skip_counter` |: schema paths from clients count toward one throttle key; the 1st and 10th events each name their path, never the answer | | `test_a_repaired_teacher_answer_is_served_and_stored_with_the_original_messages` |: the repair call carries the request, the first answer without its reasoning and a user turn naming the error; the same parameters; one example with the original messages and the repaired answer; both calls in `cost_usd`, the replaced one in `usage_breakdown.repair`, the baseline one call | | `test_a_teacher_answer_invalid_twice_is_served_but_never_stored` |: served with `schema_valid` false, not stored, a `skip_example` naming the path; a cut-off answer and `n` = 2 are not repaired | | `test_repair_can_be_switched_off_and_a_failed_repair_serves_the_first_answer` |: `RATCHET_SCHEMA_REPAIR=0`; a repair call that raises serves the first answer, counts `repair_errors` and writes `schema_repair_error` with the class only | | `test_an_audit_answer_is_repaired_too` |: an audit's invalid answer is repaired, counted as an audit and stored | | `test_arbitration_and_pinned_answers_are_repaired_and_priced` |: an arbitration's repair settles the shadow disagreement and is priced; a pinned task's answer is repaired and priced (baseline one call) and never stored | | `test_after_a_repair_the_baseline_usage_and_latency_count_the_request_itself` |: with a teacher whose prompt tokens grow with what it is sent, the baseline, `usage.prompt_tokens` and `cached_tokens` count the request's own prompt on the escalation, pinned and audit paths; `cost_usd` and the breakdown pay both calls; `latency_ms` adds them | | `test_a_cut_off_or_empty_repair_answer_keeps_the_first_answer` |: a repair answer that was cut off, empty or only reasoning keeps the first answer and its `finish_reason`; nothing stored; the repair call is paid for | | `test_a_teacher_that_refuses_json_schema_gets_json_object_and_the_schema` |: a 400 naming `response_format` is retried once as json_object with the schema in the system message (added when there is none); the example keeps the client's format and messages; the schema goes straight to json_object afterwards, one event; another 400 is not retried | | `test_refused_client_schemas_never_turn_json_schema_off_for_the_others` |: refused client schemas go as json_object one by one; after `JSON_SCHEMA_STRIKES` the endpoint is probed once with `schema_tasks.PROBE_FORMAT`, takes it, and json_schema stays on for every other schema; no second probe; an inconclusive probe (503) is asked again | | `test_json_schema_is_turned_off_for_an_endpoint_that_refuses_the_probe_too` |: an endpoint that refuses every json_schema refuses the probe too: `supports_json_schema` False, every schema as json_object; two events, not repeated by a second process; another endpoint unaffected | | `test_annotations_never_reach_a_model_or_the_ledger` |: the teacher's request body and the stored conditions hold the schema without `x-ratchet` keys; the annotation decided the fields; the name hashes the annotated schema | | `test_fenced_teacher_json_is_served_as_bare_json` |: a live gateway serves bare JSON from a reasoning-and-fence answer (the SDK's `ChatCompletion` validates it), streamed too, with `task` and `schema_valid` in the `ratchet` block | | `test_responses_name_the_task_and_stats_name_the_compared_keys` |: `ratchet.task` and `schema_valid` for plain, named and schema requests; `compare_keys` and `compare_source` in `/ratchet/stats`; the auto task in `/ratchet/tasks` (`updated_by` null), deletable, a built-in one not; under `reject` a free-text schema is a 400 with `param` `response_format` | | `test_the_sdk_parse_helper_works_end_to_end_in_mock_mode` |: `client.chat.completions.parse(response_format=Claim)` against the mock gateway parses into the Pydantic model; the auto task compares `covered`, `decision` and `parts` | | `test_a_cut_off_answer_is_length_so_the_sdk_raises_its_length_error` |: a cut-off teacher answer is `finish_reason: "length"`, so `.parse()` raises `LengthFinishReasonError`; nothing stored | ### `tests/test_tools.py` Tool and function calling: what a tool request is, the legacy fields, the provider reading tool calls, the router answering tool requests with the teacher alone and storing nothing, the response and stream shapes the OpenAI SDK reads, and the multimodal route. Teachers are small providers in the file, `RecordingProvider`s, or `FakeOpenAIServer` / `ToolCallingServer` behind `httpx.MockTransport`; nothing calls a real API. | Test | Covers | |---|---| | `test_uses_tools_names_every_kind_of_tool_request` |: `tools`, `functions`, an assistant `tool_calls` or `function_call`, a `tool` or `function` message, and `tool_choice: "none"` with tools are tool requests; empty `tools` or `functions`, a `tool_choice` or `parallel_tool_calls` with nothing to call, and empty `tool_calls` / null `function_call` are not | | `test_tool_params_map_the_legacy_fields` |: `functions` become tools, `function_call` the tool choice (`auto`, `none`, `{name}`); legacy only when no current field is sent; a legacy request asks for one call (`parallel_tool_calls: false` unless set); both forms merge (the request's tools first, duplicates by name dropped); nothing about calling is sent without a tool; `GenParams.from_request` carries the mapped fields and never the legacy names | | `test_legacy_history_pairs_calls_and_results` |: an assistant `function_call` becomes a `tool_calls` entry with a stable `call_` id and the `function` reply a `tool` message with that id; an orphan reply and plain messages stay as they are; the caller's list is untouched; nothing to change returns the same list | | `test_provider_reads_tool_calls_in_openais_shape` |: vLLM's call with `index` (dropped, its id kept) and an OpenAI call; object arguments JSON-encoded (non-ASCII kept), missing arguments `{}`, a missing type a function, a missing or empty id `call_` and 24 hex digits (distinct); text beside the calls; a legacy `function_call` answer as one call; `[]` and null as no calls | | `test_a_tool_call_a_client_could_not_use_is_a_bad_upstream_response` |: 7 cases (not an array, an entry not an object, a `custom` type, no function, no name, a blank name, one bad call of two): each a 502 `bad_upstream_response` | | `test_tool_call_text_that_is_not_valid_unicode_is_refused` |: an unpaired surrogate in a name or in the arguments is refused at once, not after the teacher was paid for and the response failed to encode | | `test_a_tool_request_is_answered_by_the_teacher_alone_and_never_stored` |: route `tools`, the calls as made, `finish_reason` `tool_calls`, `schema_valid` null; the student never sampled and `add_example` never called; the configured teacher sent the tools, the choice and the parallel flag; `tool_choice: "none"` takes the same path with a prose answer; `n` = 2 choices; `tool_requests` per task | | `test_tool_history_without_tools_is_teacher_only` |: tool calls, a tool result, a legacy call and a `function` message without `tools` each go to the teacher alone with nothing stored; the same task without them is distilled | | `test_tool_routes_keep_the_operators_reason_first` |: a pinned task and an unregistered one under `teacher` keep their route (tool calls returned); `teacher_only` gives `tools` or the route passed; a task with `tools: distill` is still teacher-only; a pinned, generic or multimodal teacher-only request without tools whose teacher makes a call is a 502 `tool_calls_unsupported`; `tool_requests` per task; without tools the generic path gives `pinned` | | `test_a_tool_call_is_not_held_to_the_response_format` |: a tool call with a json_schema is not repaired and `schema_valid` is null; a prose answer to the same request is checked | | `test_tool_requests_use_the_fallback_teacher_and_map_teacher_failures` |: an unavailable teacher's tool request answered by the fallback teacher (route `fallback_teacher`, nothing stored, `uses_tools` 1); without one the error reaches the gateway | | `test_a_tool_request_gets_no_audit_and_no_shadow` |: at audit rate 1 with a shadow candidate a tool request makes one teacher call (`audited` 0, no shadow version on the row); the same request without tools runs both | | `test_usage_counts_a_tool_calls_tokens` |: with no completion count from the teacher, the estimate includes the calls' JSON | | `test_redact_messages_covers_tool_arguments_and_results` |: emails in tool call and function call arguments and in tool and function results (strings and text parts) become `[EMAIL]`, five counted; the caller's messages untouched | | `test_tool_calls_pass_through_the_gateway` |: a live gateway whose teacher (MockTransport) makes two calls: the SDK's `ChatCompletion` validates the response, the calls and ids are identical, content null, `finish_reason` `tool_calls`; the teacher got the tools, choice and parallel flag under the configured model although the client sent `gpt-4o`; the student endpoint got nothing; nothing stored; route `tools` in `/ratchet/stats`; a teacher 400 about the request is the client's 400 | | `test_a_malformed_teacher_tool_call_is_a_502_and_nothing_is_stored` |: a call without a function name makes the response a 502 `teacher_unavailable` naming `bad_upstream_response`; the student never asked, nothing stored | | `test_legacy_function_calling_round_trips` |: `functions` and `function_call` reach the teacher as `tools`, `tool_choice` and `parallel_tool_calls: false`, a legacy history as tool turns; the answer is `message.function_call` with `finish_reason` `function_call`, streamed as one `function_call` delta (each chunk validated); a teacher's second call is dropped in the legacy form; with `tools` too the current form (both calls); a server that answers the old way gets the current form for a current request | | `test_streamed_tool_calls_rebuild_with_the_sdks_accumulator` |: the role chunk, the text beside the calls, one delta per call with the whole arguments, the finish chunk; every chunk validated and `ChatCompletionStreamState` rebuilds the calls of the non-streamed response; the usage chunk equals its usage; calls alone give a null-content role chunk and no content chunk | | `test_multimodal_requests_are_teacher_only_with_their_own_route` |: with `RATCHET_TEACHER_MODALITIES=text,image_url` an image request skips the student, stores nothing and is logged with route `multimodal` (a pinned task keeps `pinned`); no image bytes anywhere in the ledger dump; an image with tools counts as a tool request | | `test_the_openai_sdk_runs_a_tool_loop_against_the_mock_gateway` |, US-1: the real SDK against the mock gateway: `lookup_order` with the order number, the tool result sent back, the final answer restating it, `.stream()` rebuilding the same call (stable mock ids), the legacy `functions` argument getting `function_call`; nothing stored; four `tools` routes (simulation, mock models) | ### `tests/test_routing.py` Risk-aware routing: a task's `routing` block. Most tests build a `Ratchet` with `RecordingProvider`s and a `moderation` task (`flagged` bool, `severity` label); the critical-class tests script the golden eval's student answers in each row's prompt. | Test | Covers | |---|---| | `test_escalate_when_flagged_forces_teacher` |: a confident `flagged: true` answer goes to the teacher with route `rule:0`, confidence logged, both costs counted, the teacher's answer stored (`stored` 1); an unflagged one is served by the student; below the threshold the route is `confidence` | | `test_min_confidence_rule` |: `min_confidence` 0.99 escalates at 0.98 and serves at 0.9996; rules run in order and the first that holds names the route; the label comparator applies (`HIGH` is `high`) | | `test_rule_answer_never_stands_in_for_the_teacher` |: with `serve_student` and the teacher down, a flagged answer raises the teacher's error on the rule route and after a low confidence; an unflagged one degrades as before; nothing stored | | `test_shadow_runs_the_rules_too` |: a shadow whose answer a rule holds for logs `shadow_serve` 0; with an unflagged answer it serves and agrees | | `test_long_input_skips_student_and_is_not_stored` |: a 101-token prompt over a 50-token limit: route `ineligible:input_tokens`, no student or shadow call, not an escalation, nothing stored and no `skip_example` event; 50 tokens exactly is eligible; with `store_ineligible` the answer is stored (an invalid one is not); `n` 2 gets two choices; a failing teacher raises | | `test_eligibility_runs_after_pinned_and_before_the_version` |: a pinned task's route is `pinned`; an over-length request on a task disabled on the version is `ineligible:input_tokens`, a short one `task_disabled`; with a canary in flight an ineligible request takes no draw from the canary split | | `test_image_part_skips_student` |: through the gateway with `RATCHET_TEACHER_MODALITIES=text,image_url`, an image request reaches the teacher as sent, never the student, stores nothing even with `store_ineligible`, and logs route `ineligible:content_type`; a text request is served by the student; `/ratchet/stats` shows both routes | | `test_ineligible_reads_every_message_shape` |: `input_chars` counts content strings, text parts, names, refusals, tool call ids, names and arguments and skips anything else; text parts and an assistant's refusal are text, a user's refusal part or an unknown part is not; the content check comes before the length check; `stores` only for an over-length request with `store_ineligible` | | `test_task_audit_rate_override` |: `policy.audit_rate` 1.0 audits every served answer of its task and 0 none, under a global 0.5; another task is audited on exactly the same requests whatever rate a neighbour task sets (one draw either way) | | `test_route_logged_and_in_stats_by_task` |: `stats_by_task` counts `rule:0`, `ineligible:input_tokens` and `student`; an ineligible request is not an escalation; `known_route` accepts the fixed routes and the two families and refuses `rule:`, `rule:x`, `rule:-1`, a non-ASCII digit and unknown reasons | | `test_cascade_eval_measures_critical_misses` |: 2 of 40 flagged held-out rows answered unflagged at the threshold are misses (and right for a second class on `severity`); answered below the threshold, sent to the teacher by a `min_confidence` rule, or refused by the endpoint, they are not; a miss just under the exact calibrated threshold but at the stored (rounded) one counts; an invalid answer is never served; no rows in the class give an upper bound of 1.0 | | `test_critical_miss_rate_disables_task_at_admit` |: a candidate whose pooled numbers pass but which misses 2 of 40 flagged rows (upper bound 16.5% > 10%) has the task disabled with the reason, a `task_disabled` event, and ships for its other task; the router then sends the task to the teacher without a student call; `RATCHET_TASK_MODE=reject` rejects the version | | `test_critical_gate_needs_enough_rows_in_the_class` |: with no misses 35 rows pass a 10% limit and 34 fail; 9 rows fail a 50% limit for want of 10 rows and 10 pass; `RATCHET_MIN_TASK_TEST_N` sets the minimum; the limit is the spec's; a record without measurements, with an empty or broken list, under another spec, or with a malformed class entry fails | | `test_critical_task_without_measurements_stays_on_teacher` |: without a record, or with one that lacks `critical`, the task is held back (route `task_disabled`, threshold 1.01) while other tasks keep the global threshold; a measured record serves; the hosted untuned model never serves the task and is not sampled for it (route `task_disabled`), while other tasks still sample it; a resolver that fails leaves the request to the student path (`student_error`); a shadow without a measured record does not serve it | | `test_per_task_off_still_records_critical_tasks` |: with `per_task=False` only the task with critical classes gets a record, with its measurements; without such a task there is no `tasks` | | `test_per_task_off_critical_gate_disables_only_that_task` |: with `per_task=False` the eval gives `other_tasks`; a failing critical task is disabled and the version ships for its other task (rejected under `reject`); a passing critical task with 20% coverage of its own does not reject a version whose other task is fully served; a version whose other task serves nothing is rejected once the critical task is disabled | | `test_rules_apply_to_task_precision_and_coverage` |: with every flagged answer sent to the teacher by a rule, the record (and the pooled numbers) show precision 0.9 and coverage 0.5 on what is served, and the task fails a 93% floor; wrong answers a rule always escalates do not keep the threshold from calibrating (0.9 instead of 1.01 without the rule) | | `test_rescore_disables_failing_critical_task` |: a training night's re-check of the live version that finds 3 misses in 40 disables the task in the same eval write with a `task_disabled` event ("re-scored"); `rescore` without a gate leaves it; passing, missing and pinned records give no holds | | `test_cascade_eval_skips_ineligible_rows` |: an over-length golden row is not sampled and is counted in `excluded_ineligible`; the task keeps its record; without such rows the key is absent | | `test_rule_must_reference_declared_field` |: a rule naming `flaged` is one `SpecError` listing the task's fields; a field-less task's critical class says `its fields: none`; a `Task` built in code raises `ValueError`; a field with its own error is reported once | | `test_routing_spec_errors_name_their_path` |: 33 bad blocks (unknown keys, `audit_rate` with a hint, limits, content types, `store_ineligible`, list sizes, condition shapes, values outside a vocabulary, `action`, `min_confidence`, `max_miss_rate`, keys of the other list) each give their error with its path, from the spec parser and from `Task` | | `test_routing_block_round_trips_and_extends_merges_it` |: a block keeps its spelling and hash through `document`, its values normalised for the router; `extends` merges routing by key (null removes); a child dropping the field an inherited class names is refused; a `set` field cannot be a condition; the parser refuses NaN, ignores an empty block, and checks nothing against fields that are themselves in error | ### `tests/test_adjacent_endpoints.py` The other OpenAI endpoints a drop-in base URL receives: the `/v1/embeddings` pass-through, `/v1/completions` and the stateless subset of `/v1/responses` on the chat path, and the typed 404s. Mock-mode tests use the `gw_env` gateway; live-mode tests build a `Runtime` over `FakeOpenAIServer`s. Responses are checked with the OpenAI SDK's own models (`CreateEmbeddingResponse`, `Completion`, `Response`) and clients (`make_openai_client`). | Test | Covers | |---|---| | `test_mock_embeddings_validate_with_the_sdk` |: deterministic unit vectors of 256 values in the SDK's shape, the same input giving the same vector; another `model` echoed in `X-Ratchet-Model-Alias`; `dimensions` keeps the full vector's direction; base64 (the SDK's default) decodes to the same floats; token arrays; the shared mock-mode gateway's cap on values per request; every call one row (task `embeddings`, served_by `upstream`, route `passthrough`, `stored` 0, cost = baseline = teacher column) and no example; the embeddings model in `/v1/models` and `GET /v1/models/{id}`, never `ratchet/embeddings` or an unregistered task | | `test_embeddings_stay_out_of_the_headline_figures` |: embeddings rows (route `passthrough`) leave `stats_by_day`, `totals` and `stats_by_client` unchanged (student share, latency and savings of the chat rows alone; a row without a route still counted), while the `embeddings` task row, the cost report and the key's quota count them; through the gateway, `/ratchet/stats` `days`, `totals` and `clients` are the same before and after three embeddings calls | | `test_embeddings_validation_names_the_field` |: 19 bad bodies (missing, empty, mixed or non-integer inputs, negative token ids, booleans, more than 2,048 items, over `RATCHET_MAX_INPUT_CHARS` with a token as 4 characters, `encoding_format`, `dimensions`, `model`) each a 400 naming the field; bad JSON; a refused request logs nothing | | `test_embeddings_model_policy_off_switch_and_quota` |: under `strict` only `ratchet`, no model and the configured id are served (anything else a 404 before anything is logged); `RATCHET_EMBEDDINGS_BASE=off` is a 404 `unsupported_endpoint` and drops the model from `/v1/models`; embeddings and chat completions share the key's daily quota | | `test_embeddings_settings_are_checked_at_startup` |: the teacher's base by default, `off` in any case, an http(s) URL; a bad base, an empty, padded or Ratchet-named model, or a model alias spelled like the embeddings model stops startup | | `test_embeddings_model_is_priced_and_day1_checked` |: every model in `EMBEDDING_PRICES` is in the day-1 check's list, has no output price, and is neither in `DEFAULT_PRICES` nor in the license registry | | `test_live_embeddings_forward_the_configured_model` |: whatever `model` a client sends, the upstream gets `RATCHET_EMBEDDINGS_MODEL` with the inputs and `dimensions` as sent and no `user` or `encoding_format`; usage prices the call; another base gets its own key and transport, an unpriced model the teacher's price, a keyless base no key at all (the teacher's own base, with a trailing slash, still gets `NEBIUS_API_KEY`); upstream 400, 401, 404, 429, 503, 504 and malformed bodies map to the client statuses and log nothing | | `test_embeddings_task_name_is_reserved` |: a chat, completions or responses request naming `embeddings` (header, metadata, `ratchet/` or `ratchet:` model, Azure deployment) is a 400 `reserved_task` naming where it came from, before the quota; `GET /v1/models/ratchet/embeddings` is a 404; the admin API, a tasks file and a model alias cannot define it | | `test_provider_parses_embeddings_strictly` |: vectors found by their `index`, base64 accepted, the upstream's model name and prompt tokens kept; 11 malformed bodies (wrong count, empty or uneven vectors, strings, booleans, values past float32, bad base64) are a 502 `bad_upstream_response` | | `test_completions_answer_and_store_as_the_chat_request` |: a prompt through `/v1/completions` validates as the SDK's `Completion`, is answered as task `ticket-triage` from `model: ratchet/ticket-triage`, and stores exactly the example the chat request with that user message stores; the chat request then gets the same answer and usage and bumps that example | | `test_completions_refuse_what_would_change_the_answer` |: `suffix`, `echo` and `best_of` that would change the answer are a 400 `unsupported_parameter`; token arrays, too many prompts, long prompts and bad `logprobs` are 400s naming the field; chat fields keep their names; an empty suffix, `echo: false` and `best_of` 1 or `n` are accepted; nothing refused is logged | | `test_completions_batches_and_streams` |: two prompts with `n: 2` give choices 0 to 3 in OpenAI's order, summed usage and one row per prompt; a batch the daily quota cannot finish is refused whole (`has 1 left`); the stream's `text_completion` chunks spell the non-streamed texts, end each choice with its finish reason, carry the usage chunk and the `ratchet` block, and end with `[DONE]` | | `test_completions_batch_that_stops_after_its_first_prompt_is_not_retried` |: in live mode, a 2-prompt batch whose second teacher call fails (503 with `Retry-After`) is a 502 with `X-Should-Retry: false`, no `Retry-After` and a message naming the answered prompt, so the OpenAI SDK with `max_retries=2` sends it once and the first prompt's example has `seen` 1 (below FREQUENT), one row and one quota request; the first prompt's own failure keeps its `Retry-After` and chat message; a later prompt refused by a quota a concurrent request used up is a 429 the same way; a batch past `RATCHET_TEACHER_DEADLINE_S` stops before its next prompt (504 `batch_deadline_exceeded`); mock mode's plain 400 for a later prompt carries the header too | | `test_completions_with_the_openai_sdk_and_legacy_logprobs` |: the SDK's completions client in live mode, streamed and not; `logprobs: 2` returns tokens that spell the text, their logprobs, at most 2 alternatives and offsets from the prompt's length; the teacher gets the user message, the configured model, `logprobs`, `top_logprobs` and `max_tokens`, never `prompt` or `echo` | | `test_legacy_logprobs_shape` |: the chat shape's token logprobs become `tokens`, `token_logprobs`, `top_logprobs` objects and `text_offset` | | `test_responses_string_input_validates_with_the_sdk` |: a string input with instructions validates as the SDK's `Response` with a non-empty `output_text`, echoes its settings (`store` false, no previous response), answers and stores exactly as the chat request [system, user]; the SDK's responses client reads it; a cut-off answer is `incomplete` with `max_output_tokens` | | `test_responses_refuse_what_needs_state` |: `previous_response_id`, `conversation`, a stored `prompt`, `background` and `stream` true, built-in tools and tool choices are a 400 `unsupported_parameter`; item references, built-in tool items, files and images by id a 400 `unsupported_value`; 25 cases in all, each naming the Responses field (`max_output_tokens`, not `max_tokens`), among them a `reasoning.effort` the SDK does not know; fields with no effect are accepted; every effort the installed SDK knows is echoed and parses as its `Response`; the stored-response routes are a 404 `unsupported_endpoint` | | `test_responses_input_items_become_chat_messages` |: message items with each role and part type, function calls and their outputs, reasoning items, instructions, `text.format`, function tools and choice, `max_output_tokens`, `temperature`, `reasoning.effort` and metadata become the chat request; chat-only fields are ignored and unknown ones kept; JSON-object and text formats; an image URL part | | `test_responses_tool_loop_becomes_the_chat_conversation` |: a response's output sent back as input (its text, then its calls, with reasoning between them) is one assistant message with both calls, as the chat request of the same loop; calls after a tool result or a user message start their own assistant message | | `test_response_output_carries_function_calls` |: a choice's tool calls become `function_call` items with their `call_id` (a generated one when empty, object arguments as JSON), after the message item when there is text; the object validates as the SDK's `Response`; usage details; `content_filter` is `incomplete` | | `test_moderations_and_unknown_paths_get_typed_404s` |: `/v1/moderations` is a 404 `unsupported_endpoint` with the docs link, which the SDK raises as `NotFoundError`; other `/v1/*` paths are `unknown_url`; a wrong method is a 405 in the envelope; with keys on every new route checks the key first | | `test_adjacent_endpoints_reach_the_student` |: with a live version that knows the ticket, a completion and a response are served by the student, the same cascade as the chat request | ### `tests/test_sqlcheck.py` Text-to-SQL tasks: answers compared by what their queries return on a fixture, in a sandboxed in-memory SQLite database. One fixture script (`SHOP`: customers and orders, written like a `.dump`) serves most tests; tasks with the same fixture text share one sandbox, so tests that count executions or change limits use a script of their own. Running SQL needs Python 3.11 (`Connection.setlimit`), so the tests that do are marked `needs_sql` and skipped on 3.10; the path, postgres and refusal tests run everywhere. | Test | Covers | |---|---| | `test_sql_equivalent_queries_agree` |: `COUNT(*)` and `count(id)`, a JOIN and an IN subquery (and EXISTS), aliases, SUM over integers and TOTAL, `0.1 + 0.2` and `0.3`, VALUES and UNION ALL agree; an executing sql task is verifiable, not free text, `exact` | | `test_sql_wrong_filter_disagrees` |: another filter, `>` against `>=`, another column count, `1` against `'1'`, a duplicated row (a multiset) disagree | | `test_sql_order_by_respected_otherwise_multiset` |: the outermost ORDER BY keeps rows in order; ORDER BY in a subquery, a CTE or a window does not; an ordered result never equals an unordered one of two rows or more; one row has one order; `ordered` `never` and `always` | | `test_sql_empty_results_fall_back_to_text` |: two empty results agree only as the same normalised query (case, whitespace, comments, `;`); empty against not empty disagrees; `normalize`; `empty_result: agree` with the column count | | `test_sql_multiple_statements_invalid` |: a second statement, prose after the query, unterminated strings, names and comments, EXPLAIN, PRAGMA, writes, a parenthesised start and a query over `MAX_QUERY_CHARS` are `not one query`; `;` and comments around one query, and `;` inside a string, are fine | | `test_sql_answer_cleaning` |: `` and a ```` ```sql ```` fence are stripped (cleaning is idempotent); a fence inside the fence is kept and invalid; an empty or unfinished answer is `empty answer` | | `test_sql_write_and_attach_denied_and_fixture_unchanged` |: writes that read as a query (`WITH ... INSERT`, `DELETE`, `UPDATE`) are `sql denied`; DELETE, INSERT, UPDATE, DROP, CREATE (TEMP too), ALTER, ATTACH, VACUUM INTO, BEGIN, SAVEPOINT, ANALYZE and PRAGMA run on the sandbox directly never return a result; no file is written; the row counts never change; with an authorizer that allows everything, `query_only` still refuses a write | | `test_sql_pragma_denied` |: PRAGMA statements and the pragma table-valued functions are denied; reading `sqlite_master`, `sqlite_schema`, `json_each` and `json_tree` is allowed; `sqlite_stmt` (other requests' statements) and `dbstat` are `sql denied`, also through a fixture's view | | `test_sql_runaway_recursive_cte_interrupted_quickly` |: an endless recursive CTE stops on `max_steps` in under 0.5 s, so does a nine-way cross join; a query within its steps but past `QUERY_SECONDS` stops too and is not cached (then runs with a generous time budget, so a busy machine cannot fail it) | | `test_sql_random_function_denied` |: `random()` (in ORDER BY too), `randomblob`, `changes`, `total_changes`, `last_insert_rowid`, `load_extension`, `sqlite_version`, `zeroblob` are `sql denied`; `'now'`, `'NOW'`, `'localtime'` in a date function (also inside a nested call, a subquery or a CASE among its arguments), CURRENT_DATE/TIMESTAMP, a date function without a time value (`date()`, `julianday()`, `unixepoch()`, `strftime('%s')`), a quoted function name and a double-quoted `"now"` `reads the clock` in the text; a value built at run time (`'n' \|\| 'ow'`, a subquery, `subsec` as the time value, a `localtime` modifier built from parts, text after a NUL, a blob) is caught when the date function runs; dates from the data, `'subsec'` as a modifier, NULL and the string `'now'` outside a date function are fine; the wrapped date functions give SQLite's results; a clock-reading query is invalid under any spelling, never a cached result | | `test_sql_rows_result_size_and_value_limits` |: more than `max_rows` rows is invalid, as many is fine; a result over `RESULT_BYTES` is `sql result too large` | | `test_sql_value_limit` |: on Python 3.11+ a doubling string stops at `VALUE_BYTES` (`sql error`); skipped on 3.10 | | `test_sql_quadratic_builtins_replaced_by_linear_ones` |: `trim`, `rtrim`, `instr` (text and blobs) and `replace` at `VALUE_BYTES` on inputs that are quadratic in SQLite all finish in under 2 s together (milliseconds each); a `replace` result over the cap and a large mixed-type `instr` are `sql error`; a LIKE pattern of `LIKE_PATTERN_BYTES` is refused and one just under it runs; `json_patch` and `unhex` are denied; a query that repeats the most expensive LIKE per row stops on `QUERY_SECONDS`, and so does one row whose WHERE calls it 400 times (LIKE checks the time on every call); LIKE with ESCAPE and GLOB give SQLite's results; on 3,000 random arguments per function (text, blobs, numbers, NULL, NUL characters, empty strings, long character sets) the Python versions give exactly SQLite's results and errors | | `test_sql_memory_guard` |: a sorter filled with 2 KB rows is stopped by the resident-memory guard (4 MB here) and the outcome is not cached; with a generous time budget, so a busy machine cannot turn it into a timeout; Linux only (skipped where `/proc` cannot be read) | | `test_sql_outcomes_cached_by_query_text` |: a result and an error run once for three checks; a statement over `CACHE_CHARS` runs every time; the cache holds at most `CACHE_SIZE` entries | | `test_sql_threads_share_one_sandbox_safely` |: two tasks with the same fixture text share one sandbox; eight threads checking 60 answers each get the single-threaded keys and no error | | `test_sql_spec_round_trip_and_spec_hash` |: `parse_spec` and `document` round-trip the `sql` options (also through the ledger); the hash changes with an option or the fixture's text, not with its path (one sandbox); no other task's spec has the key; `extends` merges `sql` by key; RATCHET_FIXTURES_DIR is the default directory | | `test_fixture_path_outside_dir_rejected` |: `..`, an absolute path, a symlink out and `sub/../..` are spec errors at `sql.fixture`; `SqlSpec.load` refuses them too | | `test_sql_spec_errors_have_paths` |: a document of 21 tasks with 27 mistakes (no `sql` for format sql, `sql` on a JSON task, fields on a sql task, every option out of range or unknown, postgres with a fixture, no fixture, not an object, a missing, broken, attaching, pragma-setting, random-using, tableless, Latin-1, oversized or directory fixture, a reserved format, a fixture whose load reads the clock, directly or through a `DEFAULT CURRENT_TIMESTAMP`, one whose DEFAULT calls `randomblob()`, one whose view reads the clock) gives exactly those errors in order; a FIFO is not read; `Task` and `SqlSpec` built in code refuse the same | | `test_fixture_load_limits` |: a `.dump` (PRAGMA foreign_keys, BEGIN, COMMIT), an unfinished transaction and clock-reading defaults that no row takes load; a value over `VALUE_BYTES`, a fixture over `MAX_FIXTURE_DB_BYTES` once loaded and one past `LOAD_SECONDS` are refused | | `test_sql_execution_needs_python_311` |: `CAN_LIMIT` is true from Python 3.11; without it a sandbox is refused and a sqlite spec is a `sql.fixture` error that says why and names `postgres`; a postgres task still works (runs on 3.10 too); uses an empty sandbox map, so no sandbox another test left alive answers for it | | `test_sql_postgres_dialect_compares_text_only` |: normalised text only, nothing runs (so no clock check); not verifiable, free text; parses and round-trips | | `test_sql_task_in_the_cascade` |: three texts of one query let the student answer, unfenced; disagreeing or failing samples escalate and the teacher's answer is stored without its fence; a teacher answer that is denied, fails or reads the clock (`date('now')`, `julianday()`, `date("now", ...)`) is served, not stored, and `skip_example` names only the reason | | `test_sql_golden_eval_and_dataset` |: `cascade_eval` skips references that fail on the fixture (`golden_invalid_refs`) and scores equivalent student queries as agreeing; `dataset.rows` leaves stored answers that fail out (`skipped_invalid`) | | `test_sql_task_through_the_admin_api_and_tasks_file` |: PUT creates a sql task (verifiable, its `sql` shown); the same body is no change until the fixture's text changes, then a new spec_hash and a `task_changed` event; a path outside the directory is a 422; validate; a chat request; a tasks file with a sql task starts, and one whose fixture is gone stops startup with the path | ### `tests/test_redaction.py` Redaction for extraction tasks, the detector plugin and Indian identifiers. Detector plugins for the tests are in `tests/redact_plugins.py` (a working one, ones that fail at store time or return bad spans, ones that fail the startup probe); nothing imports Presidio. | Test | Covers | |---|---| | `test_two_emails_distinct_placeholders` |: `label` loses which email is which; `numbered` and `surrogate` keep them apart, repeats and case included, every placeholder counted; numbers per label and per example; a phone with and without its country code, and a card with spaces or dashes, is one value; an unknown style refused | | `test_numbered_placeholders_consistent_between_prompt_and_answer` |: the teacher sees the original; the stored prompt has `[EMAIL_1]`, `[EMAIL_2]`, `[PHONE_1]` and the stored answer names `[EMAIL_2]`; the comparator tells the right placeholder from the wrong one | | `test_one_pseudonymizer_covers_messages_answer_and_conditions` |: content parts, `name`, `refusal`, tool-call arguments, the answer and the conditions share one numbering; the caller's messages are untouched | | `test_surrogate_values_stable_under_second_pass` |: `redact(redact(x)) == redact(x)` in every style for a text with one of each identifier; surrogates are `example.com`, 555-01xx in E.164, RFC 5737, valid `ZZ` IBANs and card numbers that fail Luhn and hold no Aadhaar; no original left | | `test_surrogates_are_reserved_and_distinct_until_they_run_out` |: 300 emails, 79,800 phones, 999 cards, IBANs and 762 IPs distinct and reserved, then the numbered fallback | | `test_label_style_matches_v04` |: the nine v0.4 rules' SHA-256; on 4,500 fuzzed texts exactly v0.4's spans and v0.4's sequential replacement with today's rules, and on those without GSTIN or UPI ids exactly v0.4's output with v0.4's rules; on 8,000 long reference numbers (5 to 8 groups of four digits, spaced, dashed or unspaced) exactly v0.4's output; numbered placeholders find the same spans | | `test_a_refused_match_no_longer_hides_a_card` |: a card with its CVV or expiry, two cards in a row, a card before a GSTIN, and a 16-character IBAN before a year or a short word are found where v0.4 left them; a card with digits or a word before it on its run, more than 4 digits after it, a tail that joins an IP address or a phone number, and a non-Luhn run stay as v0.4 left them; on 3,000 fuzzed texts, wherever the output differs from v0.4's it is that rule and nothing else; numbered and surrogate alike | | `test_redaction_is_idempotent_on_realistic_text` |: 3,000 texts of identifiers and words between ordinary separators, never two numbers one space apart (the known exception its docstring names), redacted twice, are unchanged in every style | | `test_default_settings_store_v04_examples` |: with the default settings the stored example is v0.4's byte for byte | | `test_gstin_and_upi_ids_are_redacted` |: GSTINs (also entity `A` and a company PAN) and UPI ids (`name@okicici`, `9876543210@ybl`, a sentence's full stop) replaced; package tags, `a+b@ybl`, one-letter handles, a space after `@` and a 14-character near-GSTIN kept | | `test_redact_plugin_detector_applied` |: a fake NER detector tags names, numbered with the emails; label and surrogate styles; an empty text never reaches the plugin; a dotted path, a lower-case label and offsets of an integer type that is not `int` (numpy's) | | `test_detector_spans_merge_with_patterns_earlier_then_longer` |: overlapping spans: the earlier wins, then the longer, then the pattern's, covering all it overlaps; touching spans stay apart | | `test_missing_plugin_fails_startup` (9 cases) |: a missing module or attribute, a non-callable, a malformed spec and a detector failing the empty-text probe stop `load_detector`, `Ratchet` and the gateway's `Runtime` | | `test_bad_style_fails_startup_and_style_is_read_loosely` |: an unknown style or a non-string plugin stops `Ratchet`; case and spaces are ignored and empty means `label`; both settings read from the environment | | `test_a_failing_detector_stores_nothing_and_never_leaks_the_text` (7 cases) |: a detector that raises, returns an out-of-range, empty or malformed span, bool or float offsets, or a bad label: the answer is served, nothing stored, one `skip_example` `redaction failed`, no table holds the text, and the error carries neither the text nor the plugin's message | | `test_original_values_never_stored` (3 cases) |: in each style, with the plugin and a stop string, no original value reaches any table; numbered placeholders in the answer match the prompt's | | `test_task_spec_privacy_redact_style_overrides_the_default` |: `privacy.redact_style` parsed, documented and round-tripped, outside `spec_hash`; bad values and keys listed with their paths; `extends` merges it (null removes it); the cascade stores that task numbered and another task labelled | | `test_admin_api_sets_a_tasks_style_without_recalibrating` |: a `PUT` that only sets `privacy.redact_style` changes the task, keeps its `spec_hash` (not recalibrating), shows in `/ratchet/stats` and writes a `task_changed` event saying the hash was kept; a bad style is a 422 with its path | | `test_changing_the_style_stores_a_new_example_once` |: the same request under `label` twice and `numbered` twice is two examples seen twice each | | `test_an_erasure_holds_under_every_style_and_plugin` |: an example erased under each of five style and plugin settings is refused under all five; both its hash and its key under the default redaction (`canon_hash`) are tombstoned; default rows record no `canon_hash` | | `test_long_reference_numbers_store_as_in_v04` |: 40 tracking numbers in a prompt and unspaced in the answer are stored exactly as v0.4 stored them, with v0.4's example hash, so the target never holds digits its prompt lacks | | `test_stored_conditions_are_numbered_in_key_order` |: conditions that differ only in key order get the same numbering and the same example hash; each copy keeps the client's key order | | `test_the_example_hash_never_depends_on_the_answer` |: three different answers to one request with a stop string are one example seen three times; the conditions are numbered before the answer | | `test_concurrent_examples_never_share_placeholders` |: 16 threads each store an example that starts at `[EMAIL_1]` and names its own `[EMAIL_2]` | | `test_redact_preview_shows_the_style_and_takes_style` |: `POST /ratchet/redact` in the gateway's style, with the plugin for an API or admin key and without it (`plugin_skipped`) for no key or a wrong one, `?style=` in any case, 400 for an unknown style; `/ratchet/stats` `privacy` style keys, a task's override included | | `test_redact_preview_reports_a_failing_plugin_without_the_text` |: a failing plugin is a 500 naming its exception class only, for a caller with a key; a keyless preview never reaches it; without a plugin, names are left | | `test_gateway_stores_numbered_examples_end_to_end` |: over HTTP under `RATCHET_REDACT_STYLE=numbered` the stored prompt and answer share placeholders | | `test_every_pattern_scans_adversarial_input_in_linear_time` |: every pattern in `PATTERNS`, and `redact` as a whole, finish 27 hostile inputs of 120,000 characters (runs of address, digit and separator characters) within 3 s (9 s for `redact`); the unbounded EMAIL pattern took 20 to 30 s | | `test_email_pattern_still_finds_addresses_and_cuts_a_long_local_part` |: plus-tags, dotted domains and two addresses in a row are found; a 64-character local part is an address, a 100-character one is not | ### `tests/test_kpis.py` Success metrics (). Each metric is checked on synthetic day rows built in the test, and the ledger reads against brute force over request rows the test logged; the reference run's metrics are in `test_system.py`. | Test | Covers | |---|---| | `test_kpis_unit` | success metric 1: the window is the last W days with traffic (a quiet day skipped, fewer days, bad sizes refused) with and without failed spend, a share at the target counting whatever the float rounding, no baseline as `None`'s pooled point estimate with `stats.wilson`'s bounds, `None` below the minimum audits or without audits, exactly 95% met, bad counts refused episodes (3 nights met, 4 not, open within the target `None`, open past it not met, the longest of two, none at all 0, too few days `None`) | | `test_drift_episode_boundaries_and_what_counts_as_a_day` | success metric 5: exactly 5 points above the baseline starts an episode and exactly 2 points above ends it; a slow climb never starts one; days with fewer than 50 judged requests are skipped and nights count calendar days; the rate ignores escalations the student never judged (an outage) | | `test_bad_deploys_count_versions_found_bad_at_full_traffic` | success metric 4: a canary rolled back before 100% does not count; an alert on a live version does (also in the anytime-valid wording, without a logged promotion, repeated); a manual rollback after 100% by its event or by the version's state; `None` before any version reached 100%; events that name no version ignored | | `test_report_reads_the_ledger_and_survives_folding` | success metric 1: `report` over six days of logged rows equals brute force (savings, share, pooled audits, nearest-rank median latency); `kpi_days`' judged and judged-escalated counts; folding days 1 to 2.5 and then everything into `daily_rollup` keeps every day's sums and the audits, while latency reads only the rows still kept; a `failures` table's spend counts against success metric 1 inside the window only | | `test_latency_quantile_is_nearest_rank` | the latency target: every quantile from 0 to 1 against the nearest-rank rule, per server and per day range, no rows as `None`, bad arguments refused; once `requests.wall_ms` exists it is read where a row in the range recorded it, `latency_ms` otherwise | | `test_report_on_an_empty_or_odd_ledger_stays_valid_json` |: an empty ledger (every metric unknown 0); a version live now and a promotion event count as reaching 100%, an alert then makes it a bad deploy; an infinite cost gives `None`, never invalid JSON; bad arguments refused | | `test_report_while_requests_are_logged` |: two threads build reports while four log 600 requests; no error, every report valid JSON with a request count between the bounds, and the final count exact | | `test_kpis_endpoint_shape` |: `GET /ratchet/kpis` on a fresh mock gateway (all unknown) and after three simulated days: `simulated`, the window, every metric's keys and types, the targets equal to `SM_TARGETS`'s minimum the gate's, a version at 100%; the same report as `/ratchet/stats`' `kpis` block; `?days=1` and a window larger than the ledger; 400 for 0, -2 and 3,651, 422 for a non-number | | `test_bad_kpi_window_stops_startup` (5 cases) |: `RATCHET_KPI_WINDOW_DAYS` of 0, -3, 3651, `two` and `1.5` stop the gateway's `Runtime` naming the variable; 7 and the default 3 are read | ### `tests/test_drift.py` The label-free drift alarm. Simulation numbers are from mock models. | Test | Covers | |---|---| | `test_fenwick_ranks_match_a_sorted_list_with_ties` | `Fenwick.prefix` equals `bisect` on a sorted list over 3,000 inserts with many values per bin (the quantised ties) and at both ends of the range; a bin outside the tree and an empty tree raise | | `test_bins_and_usable_confidences` | Confidence 0 maps to bin 0 and 1 to bin 10,000, out-of-range values clamp; NaN, infinities, booleans and strings are not usable and `observe` refuses them without counting; a threshold of 1 or less is refused | | `test_p_values_are_uniform_under_the_null` (2 cases) | 5,000 i.i.d. confidences give smoothed conformal p-values flat over [0, 1] (each decile within 4 binomial standard deviations of 500), also when the scores take only 9 values | | `test_rarely_alarms_under_the_null` | Ville's bound, quickly: 100 seeded i.i.d. streams of 300 points at threshold 10 alarm at most 10% of the time | | `test_rarely_alarms_under_the_null_full_size` | Slow: 200 seeded i.i.d. streams of 2,000 points at threshold 100: at most 3% ever alarm (the bound is 1%) | | `test_alarms_soon_after_confidence_drops` | A stream whose confidence drops at n = 1,000 (30% of requests new to the student) alarms within 500 points and never before the drop; the same stream without the drop never alarms | | `test_rising_confidence_makes_the_capital_fall` | One-sided: confidence that rises after n = 1,000 never alarms and the capital ends below where it was | | `test_a_late_change_is_caught_as_fast_as_an_early_one` | The switching rate falls as 1/n after 100 points: before a drop at 20,000 the capital stays above 10^-6, and the drop alarms within 1,000 points | | `test_alarm_latches_and_the_trajectory_stays_small` | Only the first crossing reports, with its request id and day; the trajectory keeps under 200 evenly spaced points by stride doubling; the summary's fields | | `test_ten_thousand_observations_take_under_150_ms` |: 10,000 `observe` calls in under 150 ms at the speed of the laptop the bound was set on: best of five rounds, each scaled by how much slower a fixed loop of the betting arithmetic ran just before (coverage's tracer and a slow runner made it 156 to 163 ms on CI) | | `test_boosted_rate_is_capped_and_never_lowers_a_rate` | `boosted_rate`: the rate times the boost, at most 0.5, never below the rate itself | | `test_confidence_stream_reads_only_student_confidences_in_id_order` | `Ledger.confidence_stream` returns rows with a confidence and a student version only (no cold start, no failed student call, no pinned task), in id order, after an id, up to a limit, and narrowed by version, task and an id bound; `version_states` | | `test_events_once_and_the_last_event` | `Ledger.event_keyed` writes an event under a key once, also from a second connection to the same file, and records its id in `meta` with extra entries; `day` dates it; `last_event` is `None` without one; `request_ts` | | `test_replaying_the_ledger_reproduces_every_capital_exactly` | A monitor fed tick by tick, one that replays the whole ledger at once and one that reads it in batches of 50 hold the same numbers to the last bit, alarms included; `rebuild` reproduces them; another seed gives other numbers | | `test_compaction_keeps_serving_versions_and_replays_a_restored_one` | Counts are kept for canary, live and the most recently retired version; an older retired or rolled-back one keeps only its final numbers and refuses to observe; restored to live by hand, its stream is replayed first and matches a fresh replay | | `test_compaction_keeps_the_most_recently_retired_version_not_the_newest` | A canary retired early has a higher id than the live version retired later; the later one (the one an operator would restore) keeps its counts | | `test_rows_without_a_finite_confidence_are_skipped` | Infinite confidences are counted as skipped and never observed | | `test_stream_memory_follows_what_it_has_seen` |: a stream that saw one confidence holds under 8 KB; the sorted array of bins and the Fenwick tree (forced from the first observation) give the same capital, alarm and trajectory to the last bit; `Fenwick.build` equals one insert per value; `compact` drops the counts and the generator | | `test_client_task_names_cannot_grow_the_monitor` |, threat model T12: 1,000 invented task names hold 17 streams (16 by first appearance and `*`) in under 250 KB; 5,000 more add no stream and under 150 KB; an early name keeps its stream; another version gets its own 16; a replay agrees; `max_streams=0` pools every task | | `test_a_compacted_shared_stream_replays_its_own_rows` | A compacted version restored by hand replays its `*` stream from its rows whose task has no stream of its own; a new name still goes to it; the result matches a fresh replay | | `test_tick_writes_one_drift_event_per_alarm_across_restarts` | An alarm writes one `drift` event naming the version, the task, the evidence and both false-alarm bounds, dated on the alarm's day with its request and day in `meta`; later ticks and a second gate replaying the same file write no other; `drift_status` and `open_alarm` | | `test_a_shift_under_invented_task_names_alarms_on_the_shared_stream` |: a drop in confidence spread over 1,500 invented names alarms on the version's `*` stream, and the event calls it "other tasks" | | `test_a_drift_event_the_ledger_refused_is_written_by_the_next_check` | A busy database refuses the event: the error propagates, the alarm stays latched, and the next check writes the event once, dated on the alarm's day | | `test_policy_turns_the_monitor_off_and_refuses_actions_without_it` | `RATCHET_DRIFT_ALARM=0` turns the monitor off (no events, no streams, the off status); thresholds of 1 or less, a boost below 1, an action with the monitor off and a switch other than 0 or 1 are refused; the three variables are read by `from_env` | | `test_certificate_is_suspended_while_a_drift_alarm_is_open` |: `GET /ratchet/versions/{id}/certificate` is `certified` with a quiet stream (its `drift` lists the streams without trajectories), `suspended` while an alarm is open (the certificate still shown), and `drift` is `null` for a version that never served | | `test_drift_endpoint_and_stats_block` | `GET /ratchet/drift` with every key, a stream's fields, `?version=` filtering and a 422 for a bad value; the `drift` block of `/ratchet/stats`; both with the monitor off | | `test_task_headers_sent_to_the_gateway_hold_a_bounded_number_of_streams` |, threat model T12: 40 unauthenticated requests to the mock gateway, each with its own `X-Ratchet-Task`, are distilled, yet the monitor holds 17 streams for the version (the 24 later names in `*`); `/ratchet/stats` reports 17 and a family bound of 0.17 | | `test_a_restarted_gateway_replays_the_ledger_at_startup` | A live `Runtime` on an existing ledger file replays it when it starts, writes no second event for an alarm already recorded, and shares its audit boosts with the router | | `test_sqlite_stores_no_nan` | A NaN confidence is stored as NULL, which `confidence_stream` leaves out | | `test_an_up_to_date_monitor_reads_one_query_per_update` |: after its replay an update is one range query; the versions' states are read again only every 1,000 rows | | `test_concurrent_checks_write_each_alarm_once` | Six threads checking drift at once observe each row once, write the alarm once and leave the capital of a plain replay and the boost in place | | `test_audit_boost_raises_only_the_alarmed_versions_rate_with_one_draw_per_answer` |: a boosted canary is audited at 0.3 (0.1 × 3) and at 0.5 with a boost of 10 (the cap), the live version at 0.1, with the same 800 draws and the same requests reaching each version as without a boost; `label_p` records each rate | | `test_gate_boosts_an_alarmed_serving_version_until_a_newer_one_goes_live` |: off by default; with a boost of 3 the alarmed live version is boosted in the dict the router holds, and cleared (in place) once a newer version is live | | `test_early_night_halves_the_new_examples_needed_only_after_an_alarm` |: 30 new examples do not train without an alarm, nor with the action off after one; with it on they train (the `trained` event says it was an early night); once a night has trained, 30 more need the usual 50 again | | `test_early_night_never_trains_while_a_release_is_in_flight` |: an alarm on a canary does not let Night School train | | `test_an_alarm_found_by_a_replay_keeps_its_day_and_counts_only_after_the_last_training` |: a replay finds an alarm from day 3, before the last training on day 9; its event is dated day 3 and brings no early night although its id is newer; a change after that training, found only by Night School's own check, halves the examples needed | | `test_sim_alarms_on_the_launch_day` |: the ticket launch on day 5 alarms on day 5 or 6, never before (the seed-7 reference run at 2,000 a day, whose corrupted v3 raises no alarm, and seed 11 at 1,000 a day) | | `test_sim_never_alarms_without_drift` (2 cases) | Seeds 7 and 11, 6 days × 1,000 requests without drift: no `drift` event | | `test_monitor_draws_nothing_from_the_router_rng` |: the same seeded run with the monitor on and off gives the same days, versions, totals and events, apart from the `drift` event | | `test_sim_alarm_sweep` (5 cases) | Slow: seeds 1, 2, 3, 7 and 11 at 2,000 a day alarm on day 5 or 6 after the launch and never in 6 days without one | | `test_sim_flags_reach_the_policy_and_the_router` | `--drift-alarm`, `--drift-boost` and `--drift-early-night` set the gate's policy and the router shares the gate's boosts; settings that would do nothing are refused; the table prints the drift line and the event | | `test_false_alarm_bounds_keep_three_significant_digits` | `drift.pct` keeps 3 significant digits, and the simulator's drift line at threshold 1000 says 0.1%, not 0% | ### `tests/test_teacher_watch.py` The teacher watch. Simulation numbers are from mock models. | Test | Covers | |---|---| | `test_a_disagreeing_label_is_checked_contested_and_left_out_of_both_splits` (2 cases) | A duplicate with another label writes one `teacher_checks` row (`agree` 0, source `duplicate`, epoch 1), marks the row contested and keeps the stored label; `dataset.rows` leaves it out of the train and the golden split (`excluded_contested`), and keeps it with `exclude_contested=False` | | `test_the_same_label_is_an_agreeing_check` | Other reply text with the same fields is an agreeing check; the row stays in training | | `test_free_text_pinned_and_unregistered_tasks_write_no_check` (3 cases) | `default` (no fields), `research-plan` (pinned) and an unregistered name are never checked or contested | | `test_labels_that_needed_redaction_are_never_checked` | A stored or a new label with placeholders writes no check; a clean re-test of a clean stored label still does | | `test_first_re_test_rule_one_check_per_request_per_epoch` | Ten re-labels in one epoch give one check; an ack and a model change each allow one more | | `test_a_contested_or_adjudicated_label_is_never_compared_again` | After an ack, a contested label and an adjudicated one (UNCHECKABLE) are compared by no duplicate and offered to no probe; only adjudication lifts a quarantine | | `test_labels_from_another_teacher_or_sampled_labels_are_not_compared` | Another configured teacher's label and a label sampled above temperature 0, stored or as a re-test, are not compared; sampled rows are UNCHECKABLE and never probe candidates | | `test_a_stored_label_the_current_spec_refuses_is_not_compared` | A stored label a tightened spec refuses is no evidence about the teacher | | `test_router_stores_sampled_teacher_answers_as_uncheckable` | `comparable_label`: temperature 0 (or unset) without a reasoning effort only; the router stores a sampled answer as UNCHECKABLE and checks two greedy ones | | `test_the_freeze_rule_rarely_alarms_when_the_teacher_is_within_its_noise` (2 cases) | Ville's bound by seeded Monte Carlo: 300 runs of 400 checks over 10 days at 5% and 3% disagreement alarm in at most 1% plus three standard errors | | `test_the_freeze_rule_catches_a_teacher_four_times_too_noisy_within_300_checks` | Power: at 20% every run alarms within 300 checks, the median under 100 | | `test_reusing_a_stored_label_breaks_the_guarantee_the_first_re_test_keeps` | Requests whose two draws disagree exactly 5% of the time: comparing the stored label with ten re-tests alarms in over 15% of runs (about 30%); the first re-test alone stays within the bound | | `test_day_restarts_catch_a_teacher_that_changes_mid_epoch` | After 400 agreeing checks over 14 days, 15% disagreement alarms within 150 checks with the day mixture; a single e-process does not within 200 | | `test_evidence_is_a_mixture_over_day_starts_whose_weights_sum_to_one` | The weights 1/(j(j+1)) sum to 1; `evidence` equals the hand-computed mixture (days without checks share the next day's counts; order and repeated days do not matter); a younger epoch weighs a later stretch more | | `test_the_ceiling_bounds_any_students_agreement_at_full_coverage` | `self_agreement` and `ceiling` on known distributions; max p ≤ sqrt(sum p²) on 2,000 random distributions and E[max p] ≤ sqrt(E[a]); `summary` | | `test_the_golden_gate_warns_when_the_floor_is_above_the_ceiling` |: 30 of 40 agreeing checks write an advisory `warning` naming the ceiling, the floor and the counts, and the candidate is still admitted; 29 checks, or the watch off, write none | | `test_the_gate_freezes_night_school_once_and_an_ack_resumes_it` |: 12 of 40 disagreeing checks freeze it with one `teacher_drift` event; a second process finds it frozen and writes none; a quiet tick reads two SELECTs; an ack from the other process starts epoch 2, unfreezes and resets the gate's counts | | `test_the_watch_switch_turns_off_every_action` | `teacher_watch=0`: no freeze, the checks still reported | | `test_policy_checks_the_teacher_watch_fields` | The alphas and the switch are refused out of range, naming the variable | | `test_settings_check_the_nightly_limits` | `RATCHET_TEACHER_PROBES` and `RATCHET_ADJUDICATE_MAX` from 0 to 10,000, whole numbers; `Ratchet` stops on a bad one | | `test_a_changed_teacher_model_id_writes_one_event_and_starts_an_epoch` | One `teacher_model_changed` event and epoch 2 when the reported id changes (empty ids ignored); a restarted router reads the id from `meta`; a 500-character id is kept to 200 | | `test_a_teacher_alternating_between_two_model_ids_starts_one_epoch` | Twenty responses alternating between two ids write one event; a third id one more; an ack keeps only the last id, so the other is new again | | `test_a_ledger_that_refuses_the_model_id_never_fails_the_teachers_answer` | A locked ledger in `teacher_model_seen` still serves the answer; the next response records the id | | `test_the_fallback_teachers_model_id_is_not_watched` | A fallback teacher's answer records no id and no event | | `test_concurrent_requests_see_a_model_change_once` | 16 threads that all see a new id write one event and one epoch | | `test_probes_re_ask_unchecked_stored_labels_in_hash_order_as_first_re_tests` |: only checkable rows are candidates, in hash order; two probes ask under the stored conditions, record `probe` checks, contest a disagreement and write the `teacher_watch` event; a later duplicate of a probed row is no check | | `test_a_probe_answer_that_could_not_be_stored_is_no_check` | `usable_label` (reasoning stripped; cut off, invalid or needing a placeholder refused); such a probe is counted unusable, writes no check and a quiet night no event | | `test_adjudication_two_of_three_resolves_and_one_one_one_stays_contested` |: the stored label kept, a 2-of-3 majority replacing it, and two without a majority left UNRESOLVED; resolved labels are UNCHECKABLE; provenance, license and redactions unchanged; `majority` | | `test_labels_never_adjudicated_go_before_unresolved_ones` | A fresh contested label is adjudicated before three unresolved ones | | `test_a_failing_teacher_stops_the_nights_watch_work_but_not_the_night` | A connection error stops probes and adjudication; the event names the class, never the message | | `test_a_frozen_watch_skips_the_night_before_any_teacher_call` |: a frozen night writes `skip` and makes no teacher call or training; after an ack it trains on every row that is not contested | | `test_probes_run_while_a_release_is_in_flight_but_nothing_trains` | Probes are evidence: they run during a shadow, and the night still skips training | | `test_the_bom_records_the_teachers_self_agreement_and_the_labels_left_out` | The BOM's `teacher` section; with the watch off no section and the contested rows trained on | | `test_unreadable_meta_values_fall_back_and_the_module_check_freezes` | An epoch, a day or a list of ids in `meta` that does not parse reads as its default; `teacher_watch.check` freezes once without a gate | | `test_concurrent_duplicates_give_one_check_per_epoch` | Eight threads adding 200 duplicates write one check; a probe and a duplicate from two connections to one file check an example once; a bad source is refused | | `test_an_old_file_gains_the_columns_and_its_rows_are_checked` | A v0.4 examples table gains `contested` and `checked_epoch` (0) and its rows are checked; the state defaults to epoch 1, not frozen, day 1 | | `test_redaction_placeholders_never_create_false_disagreements` | Two tickets that differ only in an email address share one stored row; their different labels write no check | | `test_mock_teacher_noise_is_reproducible_and_absent_at_zero` | Noise 0 draws nothing; 0.15 changes about 15% of first labels, the same in two instances; 1.0 always changes the category to another valid one and never the priority | | `test_teacher_endpoints_report_and_ack_in_mock_mode` | `GET /ratchet/teacher`'s keys and sources, the demo fault (400 or 422 outside 0 to 1), a noisy cold-start day freezing it, the `teacher_watch` block of `/ratchet/stats`, the ack (`mock-admin`) and a fresh epoch | | `test_teacher_ack_needs_an_admin_key_in_live_mode` |: in live mode the ack is 403 without admin keys, 401 without a key, 403 with an API key, and works with an admin key (its name in the event, never the key); the demo fault is mock-only; `GET /ratchet/teacher` is public | | `test_sim_a_teacher_noisy_from_day_3_freezes_night_school` |: seed 2, 6 days of 1,000 tickets, noise 0.15 from day 3: one `teacher_drift` on day 3 or later, no training after it, the strongest stretch from day 3 or later; the table's teacher line | | `test_sim_a_stable_teacher_moves_no_number` |: the same run at noise 0 with the watch on and off has the same days, versions and events, no teacher event, and more than 50 checks, none disagreeing | | `test_the_reference_run_has_checks_and_no_teacher_event` | The pinned reference run checks the teacher (over 100 checks, no disagreement, e-value below 1) and writes no teacher event | | `test_sim_never_freezes_a_stable_teacher_full_size` | Slow: five seeds of 10 days at 2,000 a day never freeze a stable teacher | ## Live-path tests Live mode, the training job, the day-1 check, the Scout CLI and the deploy scripts would otherwise run for the first time on paid Nebius resources. `tests/test_live.py` runs each of them offline first, through seams that are default arguments, so production code passes nothing and talks to the real services: | Seam | Where | Test double | |---|---|---| | `transport=` | `OpenAICompatProvider(base_url, api_key, timeout, retry, transport, sleep, clock=, rng=)` | `FakeOpenAIServer(...).transport` (an `httpx.MockTransport`); `sleep=`, `clock=` and `rng=` replace the waits between retries | | `transports=` | `Runtime(settings, transports={"teacher": ..., "student": ...})`, live mode only (other keys raise) | the same; later items drive live mode offline through it | | `client=` / `http=` | `load_adapter`, `NebiusTrainer`, `reload_adapters`, `live.main`, `http_caller`, the Scout CLI's `main(argv, client)`, `day1_check.main(client)` | an `httpx.Client` on a fake server, or on `AppTransport(gateway.app)` | | `run=` | `NebiusTrainer`, `live.main` (in place of `subprocess.run`) | `RecordingRun` | | `PATH` | `deploy/train_job.sh`, `deploy/student_endpoint.sh` (the `nebius` they call) | `FakeNebius`: logs each call's argv, prints `tests/fixtures/nebius_endpoint_list.json` for `ai endpoint list`; a stub `ratchet.cost` in the working directory logs the uptime records | | `provider_factory=` | `live.main` (builds the student and judge clients) | a factory returning providers on fake servers | | `argv=` | `sim.simulate.main(argv)`, the Scout CLI | a list | The student's sampling parameters are one read-only constant, `confidence.STUDENT_SAMPLING` (`temperature` 0.7, `logprobs` on), used by `Ratchet._sample`, `LiveEvaluator.evaluate` and `MockFleet.evaluate`, so the golden eval samples exactly as production does and a test fails if one of them drifts. The judge's free-text candidate is still sampled greedily (TASK-FREE-TEXT-SIMILARITY changes that). The live-run harness (`python -m ratchet.liverun`) is tested the same way: `LiveRunWorld` builds a live `Runtime` on Token Factory and vLLM fakes with its ledger in a file, and the harness gets its seams as default arguments (`client=` an `httpx.Client` on `AppTransport(gateway.app)` that records every path and header, `provider_factory=` over the fakes, `run=` a `RecordingRun`, `http=`, `tavily=`, `clock=` a `FakeClock`, `timer=` and `sleep=`). The module fixture `liverun_world` runs one 3-round run with nights in process once and most of the tests read its run directory. Tests that need the gateway's seeded audit draws in a fixed order pass `--concurrency 1`. These tests prove the wiring, the order of operations and the failure handling. They do not prove that the real services accept the requests: model ids, `nebius` CLI flags, vLLM's runtime LoRA loading and GPU training stay `untested (live)` until the first-run checks (`scripts/day1_check.py`) and a live run pass. Found while writing them: without `RATCHET_STUDENT_BASE`, `NebiusTrainer` ran the sync and the paid training job before noticing there was no endpoint to load the adapter into. It now checks first, and `python -m ratchet.live reload` gives the same clear error instead of an httpx URL error (`test_live_night_checks_the_student_endpoint_before_paid_work` in `test_hardening.py`). ## Hot-path tests `tests/test_perf.py` measures time, which a busy CI machine can stretch. Its timing tests stay stable by design: models answer with zero latency, so only Ratchet's own work is timed; each bound is a ratio between two runs on the same machine (365 versions against one) or a ceiling several times above what a laptop measures (a warm tick takes about 3 ms against 20, a request about 0.3 ms against 5); each figure is the best of three runs or a median; and the 200,000 rows go in with one `executemany`, so the whole file runs in about 3 s. The structural claims (which index a query searches, how many sorts a stats read does, that routing reads no version row) are checked exactly, with `EXPLAIN QUERY PLAN` and the connection's trace callback, so they do not depend on timing at all. ## Contract tests `tests/test_contract.py` was written before the v0.5 rewrites of `Ratchet.handle` and `gateway.chat` (request validation and normalisation, field comparators), as their regression net. Both have landed (`tests/test_openai_compat.py`, `tests/test_tasks.py`), and their five strict xfails now pass as ordinary tests. Each test pins behaviour that someone outside the module depends on: an OpenAI client (status codes, `/v1/models`, task selection), the dashboard (`/ratchet/stats`), an auditor (`/ratchet/bom`, license fields), or the operator's bill (cost to the token). Rules: - **The stats keys only grow.** `tests/fixtures/stats_keys.json` lists every key the dashboard reads from `/ratchet/stats` (it mirrors `Stats` in `web/src/lib/api.ts`) with its JSON type; a `[]` key is a list whose every item is checked. A change may add keys to the payload and to the fixture. It must not remove or retype one, so the React dashboard and other clients keep working. The test also fails if the fixture loses one of the keys this contract started with. - **The cost report keys only grow.** `tests/fixtures/cost_keys.json` does the same for `GET /ratchet/cost` and the dashboard's Economics panel (it mirrors `CostReport` in `web/src/lib/api.ts`), checked by `test_cost_report_contract`. - **The task API keys only grow.** `tests/fixtures/tasks_keys.json` does the same for the task API and the dashboard's Task registry panel (it mirrors `TaskRegistry`, `TaskEntry`, `FieldDoc` and the write results in `web/src/lib/api.ts`), checked by `test_task_registry_contract`. The Resilience panel's event grammar is not a fixture: the test reads it from the panel's source, so the event text and the parser change together. - **Known bugs are strict xfails that name the fixing item**, a v0.5 work item (none today: `API-VALIDATE-NORMALIZE` removed its four and `TASK-COMPARATORS` its one). When the fix lands the test passes, strict mode turns that into a failure, and the fix removes the marker in the same change. Write the mark literally (`@pytest.mark.xfail(strict=True, reason=...)`, or `marks=pytest.mark.xfail(...)` in a `pytest.param`): the traceability scan reads marks from the source, so a mark held in a variable would make an xfail count as proof. - Cost assertions are token-exact: each model gets distinct token counts, and the expected cost is built from `Settings.price(...)`, so a term billed with the wrong tokens, price or multiplicity fails. - The dashboard's rendering is still checked by hand ([web/AGENTS.md](https://github.com/harshbhx/ratchet/blob/main/web/AGENTS.md)); `test_stats_contract` proves the data it renders. (light and dark mode, phone width) stays a manual check. ## Simulation methodology `sim/simulate.py` runs `--days` days. Each day sends `--requests` requests from the chosen `--workload` through `Ratchet.handle()`, calls `ReleaseGate.tick()` ten times, then runs `run_night()`. It prints a day table and the event log, and writes JSON (`--out`, default `sim_results.json`). | Flag | Default | Meaning | |---|---|---| | `--days` | 8 | Days to simulate | | `--requests` | 2000 | Requests per day | | `--threshold` | 0.80 | Fallback threshold before calibration | | `--audit-rate` | 0.10 | Higher than live (0.05) so canaries move in a few days | | `--min-shadow` | 100 | `ReleasePolicy.min_shadow` | | `--min-audits` | 15 | `ReleasePolicy.min_audits` (live default 20) | | `--min-new` | 50 | `run_night(min_new_examples=…)` | | `--drift-day` | 5 | Product launch day (0 = never) | | `--bad-deploy-version` | 0 | Fault injection: this version is served corrupted once in canary/live | | `--workload` | `tickets` | `tickets` (support tickets), `research` (Scout's triage and reader calls), `mixed` (half each), `schema` (structured outputs) or `tools` (support requests with the mock tool catalogue). With `tickets` the night trains on `ticket-triage` only; otherwise on every task | | `--seed` | 7 | Seed for traffic and routing | | `--out` | `sim_results.json` | Results file | ### Assumptions (these drive the numbers; they are not measurements) **Traffic** (`sim/tickets.py`): - 20 templates across 6 categories (billing, technical, account, shipping, refund, security) and 4 priorities. 6 of them are marked `hard`. - Names, products and order numbers vary, so some templates produce many distinct texts and others very few (frequent repeats). - From `--drift-day`, 20% of traffic comes from 3 new "Ratchet Watch" templates the student has never seen (1 of them hard). **Research traffic** (`sim/research.py`, `--workload research`): - 8 questions about well-known ML ideas, each phrased 4 ways, each with a pool of 72 simulated search results (12 templates × 6 variants). 60% of requests are `source-triage` calls, 40% `page-read` calls. - Each template has fixed labels. 6 of the 12 are `hard` (for example a good forum answer next to an off-topic one, or a broad survey paper). - From `--drift-day`, 20% of traffic comes from questions 7 and 8, which the student has not seen. - `mixed` alternates between the ticket and research streams at random (50/50, its own seed), so both keep their own sequences. **Schema and tools traffic** (`sim/generic.py`, `--workload schema`, `--workload tools`): - `schema`: requests with `response_format` json_schema over three schemas, each its own task (`review-rating`: an enum, a 1 to 5 integer, a boolean; `ticket-route`: two enums and a boolean; `meeting-extract`: two enums, an integer and an array of enums). From `--drift-day`, 20% of traffic is a fourth schema the student has not seen (`warranty-claim`, with nested records through `$ref`). Prompts combine products, opinions and a four-digit reference, so most texts are distinct. - `tools`: support requests sent with the mock catalogue (`lookup_order`, `refund`) under `order-support`: 40% look-ups, 45% refunds with an amount and a reason, 15% questions no tool answers; new wordings from `--drift-day`. - The right answer is drawn from sha256 of the request's text, not read from it (the mock teacher is right by definition). A schema's labels are its enum, boolean and number fields; a template is a schema name (or a tool set, or a system prompt), and a schema with more than three labels is `hard`. - A tool request is answered by the teacher alone and never stored, so the `tools` workload never trains the student: its share stays 0 and it saves nothing (until tool-call distillation). **Mock teacher:** - Always right on the labels. - Bills `approx_tokens(prompt)` input plus answer and 300 reasoning output tokens. - 2.4 s latency. **Mock student skill** for a ticket of template `t`: - If its exact text was in training: skill = 0.985. - Otherwise: skill = max(0.35, CEIL × (1 − e^(−n/6))), where `n` is the number of distinct training examples of `t`, and CEIL = 0.985 (easy) or 0.86 (hard). - Whether the student "knows" a ticket is fixed per (version, text). **Sample behaviour:** - **Knows:** each sample is right with probability 0.97; token probability 0.80–0.97. - **Doesn't know, confidently wrong** (15% of unknown cases): all samples give the same wrong labels with probability 0.62–0.92. This deliberately overlaps with correct answers, so calibration cannot separate them perfectly. That is the case audits and shadow exist for. - **Doesn't know, otherwise:** random labels, probability 0.35–0.65. Research items use the same skill model, per result template. Scout's pinned planner and writer prompts are answered by the mock teacher only. **Fault injection:** - The corrupted version is fine on the staging copy, which golden eval and shadow use. - Once it reaches canary or live, its "fragile" slices become confidently wrong: billing, refund and security tickets; `paper` and `docs` triage results; `date` reads; in the generic world refund calls, schema answers whose first enum takes its first value, and free text framed as steps. This models pushing the wrong adapter file to production. **Costs:** Ultra $1/$3 per 1M tokens (the list price the simulator uses, see `ratchet/config.py`; research notes, 2026-10-25). The student uses the $0.06/$0.24 proxy, with 3 samples billed. ### Typical result (seed 7, 10 days, 2,000/day, drift day 5, bad deploy v3) This run is the pinned reference ([below](#the-pinned-reference-simulation)), so these numbers cannot drift silently. ``` day student escal. audit ok $/1k all-Ultra saved 1 0% 0% - 1.093 1.093 0% 2 44% 1% 100% 0.675 1.093 38% 3 98% 2% 100% 0.160 1.093 85% 4 98% 2% 93% 0.160 1.093 85% v3 canary → rolled back (41% audited precision) 5 86% 14% 97% 0.272 1.093 75% product launch 6 88% 12% 92% 0.246 1.093 77% 7 88% 12% 97% 0.255 1.093 77% 8 95% 5% 99% 0.200 1.093 82% 9 98% 2% 97% 0.162 1.093 85% 10 98% 2% 99% 0.147 1.093 87% total saved 69% ``` ### Seed robustness (10 days, 2,000/day, drift day 5, bad deploy v3) | Seed | Final student share | Final-day savings | Total savings | Min daily audit agreement | Bad v3 outcome | |---|---|---|---|---|---| | 1 | 98% | 85% | 69% | 97% | rejected before deploy | | 2 | 98% | 86% | 66% | 96% | rejected before deploy | | 3 | 98% | 87% | 67% | 93% | rolled back | | 11 | 98% | 85% | 66% | 92% | rolled back | "Total" includes the cold-start days, when all traffic goes to the teacher. ### The drift alarm (10 days, 2,000/day, ticket workload, no fault injection) Simulation, mock models. With the product launch on day 5, the drift monitor's first alarm, on the live version's `ticket-triage` stream; without it, none: | Seed | Launch on day 5: first alarm | Requests into the launch day | Without the launch | |---|---|---|---| | 1 | day 5 (v1) | 137 | no alarm | | 2 | day 5 (v2) | 246 | no alarm | | 3 | day 5 (v1) | 70 | no alarm | | 7 | day 5 (v1) | 379 | no alarm | | 11 | day 5 (v1) | 437 | no alarm | | 23 | day 5 (v1) | 190 | no alarm | The reference run (seed 7, with the corrupted v3) alarms on v1 on day 5 as well; the corrupted v3 is no traffic change and raises none (its rollback is the audits' job). At 600 requests a day seeds 7 and 11 do not alarm within two days of the launch, and the mixed and research workloads (seed 7, 1,500 a day) raise none in 8 days: power depends on the volume per stream and on how far confidence falls. The ticket numbers above are the same as in v0.1. Adding the research world did not change the random draws for tickets ([sim/AGENTS.md](https://github.com/harshbhx/ratchet/blob/main/sim/AGENTS.md)). ### The teacher watch (10 days, 2,000/day, ticket workload, launch on day 5) Simulation, mock models. `--teacher-noise 0.15 --teacher-noise-day 5`: from day 5 the mock teacher changes a ticket's category with probability 0.15. The freeze and what the epoch's checks saw by the end; without the noise, none: | Seed | `teacher_drift` (Night School frozen from) | Checks: duplicates / probes | Disagreements | Strongest stretch | Without the noise | |---|---|---|---|---|---| | 1 | day 7 | 244 / 140 | 17 | 17 of 95 from day 5 | no freeze, 0 of 445 disagree | | 2 | day 6 | 264 / 100 | 13 | 13 of 57 from day 5 | no freeze, 0 of 456 | | 3 | day 6 | 256 / 120 | 17 | 17 of 74 from day 5 | no freeze, 0 of 456 | | 7 | day 7 | 251 / 120 | 19 | 19 of 73 from day 5 | no freeze, 0 of 451 | | 11 | day 6 | 269 / 120 | 16 | 16 of 74 from day 5 | no freeze, 0 of 463 | | 23 | day 5 | 248 / 100 | 15 | 15 of 51 from day 5 | no freeze, 0 of 449 | Most checks come free from duplicates, but after the first days the student serves most repeats, so the probes (20 a night) find about as many disagreements as the traffic does. With the noise off, a run with the watch on has the same days, versions and events as one with it off (`test_sim_a_stable_teacher_moves_no_number`), and the reference run and the three pinned workloads do not move. ### Workloads: research and mixed Simulation, mock models. 8 days × 1,500 requests/day, drift day 5, no fault injection (selected days shown). Both workloads have several tasks on one adapter, so each task has its own threshold and its own golden check (per-task calibration, on by default): ``` python -m sim.simulate --workload research --days 8 --requests 1500 (seed 7) day student escal. audit ok $/1k all-Ultra saved 1 0% 0% - 1.133 1.133 0% 2 15% 3% 100% 0.987 1.131 13% 3 94% 6% 99% 0.205 1.132 82% 4 93% 7% 99% 0.218 1.132 81% 5 93% 7% 98% 0.197 1.132 83% 8 92% 8% 98% 0.211 1.132 81% total: 12000 requests, $5.11 vs $13.58 all-Ultra -> saved 62% python -m sim.simulate --workload mixed --days 8 --requests 1500 (seed 7) day student escal. audit ok $/1k all-Ultra saved 1 0% 0% - 1.112 1.112 0% 3 19% 2% 100% 0.932 1.112 16% 4 92% 8% 98% 0.226 1.111 80% 5 86% 14% 96% 0.265 1.111 76% 8 94% 6% 97% 0.202 1.112 82% total: 12000 requests, $6.48 vs $13.34 all-Ultra -> saved 51% ``` Seeds 1, 2, 3, 7, 11 (same settings): | Workload | Final student share | Final-day savings | Total savings | Lowest daily audit agreement | First version live | |---|---|---|---|---|---| | research | 65–97% | 56–86% | 19–65% | 97–98% | day 2–6 | | mixed | 90–96% | 78–85% | 47–60% | 96–98% | day 2–3 | | research, `RATCHET_PER_TASK_CALIBRATION=0` | 96–98% | 85–86% | 57–65% | 96–98% | day 2–3 | | mixed, `RATCHET_PER_TASK_CALIBRATION=0` | 92–98% | 79–86% | 54–63% | 93–96% | day 2–3 | On mixed traffic the first student comes a night later with seed 7 (v1 was rejected at 94.3% golden precision), so total savings are lower. With one pooled threshold and the pooled gate (`RATCHET_PER_TASK_CALIBRATION=0`, v0.4's behaviour) the student serves more at a lower worst day of audited agreement. Per task, each task's threshold and its golden check rest on a few dozen held-out rows, and the check is a point estimate, as the global gate's is: a good task dips under its 95% floor, or more than 2 points under live's 100% on it, now and then, and is disabled on that version (research, seed 11: the first version's `page-read` at 18 of 19 right; its canary, answering one task at 10% of traffic, then took until day 6 to collect its audits). A version with a disabled task that live still answers usually loses the shadow's coverage comparison too, so it is rejected a day later (research, seed 1, nights 3 to 5). ### Workloads: schema and tools Simulation, mock models. 8 days × 1,500 requests/day, drift day 5, no fault injection. Both come from the generic mock world; their right answers are drawn from each request's text, so these numbers show the loop on structured outputs, not how well a model reads a review. ``` python -m sim.simulate --workload schema --days 8 --requests 1500 (seed 7) day student escal. audit ok $/1k all-Ultra saved 1 0% 0% - 1.020 1.020 0% 2 15% 4% 100% 0.893 1.020 12% 3 77% 19% 100% 0.334 1.020 67% 4 85% 15% 97% 0.248 1.020 76% 5 80% 20% 96% 0.312 1.023 70% 8 93% 7% 97% 0.193 1.023 81% total: 12000 requests, $5.33 vs $12.26 all-Ultra -> saved 57% python -m sim.simulate --workload tools --days 8 --requests 1500 (seed 7) day student escal. audit ok $/1k all-Ultra saved 1 0% 0% - 1.068 1.068 0% 2 0% 0% - 1.069 1.069 0% 8 0% 0% - 1.069 1.069 0% total: 12000 requests, $12.83 vs $12.83 all-Ultra -> saved 0% ``` Seeds 1, 2, 3, 7, 11 (same settings): | Workload | Final student share | Final-day savings | Total savings | Lowest daily audit agreement | First version live | |---|---|---|---|---|---| | schema | 70–93% | 60–81% | 40–57% | 92–97% | day 3 | | tools | 0% | 0% | 0% | no audits | never (nothing stored) | | schema, `RATCHET_PER_TASK_CALIBRATION=0` | 86–94% | 75–82% | 55–61% | 92–96% | day 2–3 | On `schema` the drift schema (`warranty-claim`, five or more labels, `hard`) holds the share near 80% for two nights from day 5. Its three tasks share one adapter, so per-task calibration (on by default) applies, and with some seeds a task disabled on a few dozen held-out rows lowers the share, as on the research workload above; `tools` has one task and is the same either way. On `tools` every request offers tools, so the mock teacher answers it alone with its tool calls and nothing is stored: no night trains, the student never serves and the run saves nothing. It shows what the gateway does with tool requests until tool-call distillation; before both models answered these prompts in prose with the tools stripped, and that prose was learned within two nights. ## Bugs the simulation found (and the fixes) Running the loop end to end exposed design flaws that unit tests would not have caught. Each fix is in the code now. | # | Symptom in the simulation | Root cause | Fix | |---|---|---|---| | 1 | Every new candidate rejected as "regresses vs live" for days | Candidate measured on today's golden set, live measured on its own (older, easier) golden set from the day it was trained | **Champion vs challenger:** `run_night` re-evaluates the live version on the same golden set first and overwrites its eval (`rechecked_day`) | | 2 | Every candidate rejected in shadow at ~70% "agreement" | Shadow judged on all traffic, including requests it would have escalated; and an agreement with the live student was the only reference | Shadow runs the full cascade: records `shadow_serve` and judges **precision only on what it would serve**, plus **coverage** vs live. Disagreements with the live student are **arbitrated by the teacher** | | 3 | Student never learned the most common questions | The hash-based golden split held some frequently repeated requests out of training forever | Requests seen at least `FREQUENT = 3` times always go to training and leave the golden set | | 4 | Canary never finished; promotions stalled | Each night's new candidate replaced the version in canary | `run_night` **skips training while a shadow or canary is in flight**, which also saves GPU time | | 5 | Threshold promised 95% precision but delivered 94% | `calibrate_threshold` checked precision before including all points tied at the same confidence | Walk down in groups of equal confidence; return `inf` (serve nothing) if no threshold qualifies; `cascade_eval` caps it at 1.01 | | 6 | Golden coverage 94% vs real coverage 80% | Deduplication meant the golden set over-represented rare phrasings, and the threshold was calibrated and measured on the same rows | `seen` count stored per example; **calibrate on one half of the golden set, measure on the other**; frequent requests handled by fix 3 | Fixes 1, 3, 4 and 6 also have direct tests now: `test_champion_rescored_on_same_golden_set`, `test_golden_rows_never_reach_the_trainer_except_frequent`, `test_night_skips_with_reason` and `test_cascade_eval_measures_on_held_out_half`; fix 2 is covered by `test_shadow_promotes_and_both_rejections` and `test_shadow_arbitration_calls_teacher_once_and_stores_example`. Two more problems were fixed earlier in development. A fixed global threshold let about 10% wrong answers through, which led to per-version calibration. A point-estimate rollback rule would have rolled back healthy canaries about 19% of the time at n=20, which led to the Wilson upper bound. ## Issues found while documenting, and their fixes Writing these docs turned up ten gaps in v0.1. Each one is fixed in v0.1.1 and covered by `tests/test_hardening.py`. | # | Issue | Fix | Test | |---|---|---|---| | 1 | Live gateway never advanced `Ledger.day`, so every request landed in one bucket | `Ledger(auto_day=True)` in live mode: day 1 starts at the first logged request, one day per 24 hours | `test_auto_day_counts_days_from_first_request` | | 2 | A student HTTP error became a 500 | Student errors fall back to the teacher and log a `student_error` event (1st, 10th, 100th, 1000th). A teacher error returns **502** | `test_student_failure_falls_back_to_teacher`, `test_teacher_failure_returns_502` | | 3 | Runtime-loaded LoRA adapters are lost when the endpoint restarts | New command `python -m ratchet.live reload` re-loads live, canary and shadow adapters | `test_reload_adapters_loads_live_canary_shadow` (in `test_live.py`, against a fake vLLM); on a real endpoint, see [RUNBOOK.md](/reference/runbook/) | | 4 | The untuned fallback model was judged with the tuned version's threshold | `Ratchet.threshold(version, model)` uses the global default for `RATCHET_FALLBACK_STUDENT` | `test_fallback_model_uses_default_threshold` | | 5 | `alert` events repeated on every tick | One alert per live version | `test_live_alert_fires_once_per_version` | | 6 | No schema migration for existing database files | `Ledger._migrate()` adds missing columns on open (`MIGRATIONS` table in `ledger.py`) | `test_old_database_file_is_migrated` | | 7 | Stale docstring and column comment | Fixed | n/a | | 8 | Unknown `X-Ratchet-Task` values were stored forever in `tasks.REGISTRY` | Unknown names get a temporary whole-text task; names must match `[A-Za-z0-9_.-]{1,64}` | `test_unknown_tasks_do_not_grow_registry` | | 9 | `ReleasePolicy` could only be changed in code | `ReleasePolicy.from_env()` reads `RATCHET_` (e.g. `RATCHET_MIN_AUDITS`) | `test_policy_from_env` | | 10 | LoRA targets might miss Nemotron-H Mamba layers | `train.py --targets` (default attention and MLP, `TARGETS_ATTENTION_MLP`, as the measured adapters; `TARGETS_WITH_MAMBA` adds `in_proj,out_proj`, to confirm on day 1) | The default: `test_train_parse_args_defaults_and_row_contract`; the module names: manual (GPU) | Still open: `RATCHET_SYNC_CMD` and `RATCHET_TRAIN_CMD` run without a shell, so pipes and `&&` do not work; wrap them in a script. ## The pinned reference simulation `tests/fixtures/sim_reference_seed7.json` holds one run of the simulator with every input pinned (seed 7, 10 days × 2,000 requests, drift day 5, corrupted v3, mock models; the same run as `make sim`): the inputs, each day's requests, student share, escalation rate, savings and audited agreement rounded to 4 decimal places, the ordered Night School and release events as (day, kind, version), and each version's final state, traffic and training-set size. `test_reference_simulation_unchanged` runs it once per session (the session-scoped `reference_run` fixture, about 7 s) and compares exactly. The inputs compared are the ones the fixture pins: a simulator flag added later, whose default keeps the old behaviour, does not force a regeneration, while a changed or removed pinned input fails the test. It and the reference workloads below are the deliberate exceptions to "assert on trends, not numbers". Refactors and new features that claim to leave the legacy behaviour untouched prove it by keeping this test green. When a change is meant to move the simulation, run `python scripts/regen_sim_reference.py`, review the JSON diff (one day or event per line), commit it with the change and add a CHANGELOG line saying why; never edit the file by hand. `--check` compares without writing, with the same rule as the test. The run unsets every `RATCHET_*` variable while it runs, so settings in your shell cannot move it, and it depends only on seeded `random.Random` streams, SHA-256-derived seeds and SQLite arithmetic. When the fixture was made it was identical on CPython 3.10, 3.11, 3.12 and 3.13 (Linux, the CI matrix) and under different `PYTHONHASHSEED` values. If it fails on one platform only, compare the first differing day before regenerating anything. The numbers are simulation output, not measurements. ### The reference workloads `scripts/reference_check.py` pins three workloads, captured from v0.4.0 (commit 1f9b18c) in `tests/fixtures/reference.json`: `tickets-seed7` (the run above), `mixed-seed1` and `research-seed7` (8 days × 1,500 requests). All three run with `RATCHET_PER_TASK_CALIBRATION=0` (`LEGACY_ENV` in the script), because per-task calibration, on by default, changes what the multi-task runs serve by design; so the fixture is still v0.4.0's capture, byte for byte. The tickets run has one task, so it gives the same output with the switch on (the session's reference run, which the pytest check uses). It pins more than the reference simulation does: every input, every v0.4 key of each day's stats at full precision (`cost_per_1k`, `avg_latency_ms`, ...), the totals, each version's final state, training-set size and dataset hash, and the event log with its text. Every later feature defaults to v0.4's behaviour, so every change passes it: `make reference` (about 25 s) and a CI step run all three; the pytest suite checks the tickets workload against the session's reference run at no extra cost (`test_reference_workloads_reproduce_v040`). The rules: floats agree within a relative 1e-9 (an index may add the same numbers in another order); days and totals may gain keys; events compare as (day, kind, version) in order, and each pinned text must be a prefix of the fresh one, so a feature may append evidence to a message but never change it; the advisory kinds later features add (`drift`, `teacher_drift`, `teacher_model_changed`, `warning`, `student_healed`) are left out; a flag added later is not compared, a changed default is. When a change is meant to move these runs: `python scripts/reference_check.py --write --captured-from ""`, review the diff, and add a CHANGELOG line that says why. The numbers are simulation output from mock models, not measurements. ## Adding tests - Pure logic goes in `test_core.py` with small hand-built ledgers (`Ledger(":memory:")`); Night School rules in `test_nightschool.py`; release-gate and router release behaviour in `test_release.py`. Set up states with the builders in `tests/fakes.py` (`_version`, `_audit`, `_shadow`, `_examples`) and fake models with `RecordingProvider`, `RecordingTrainer` and `RecordingEvaluator`, rather than new local helpers. - A change to what clients or the dashboard see (a status code, a `/ratchet/stats` key, cost accounting) goes in `test_contract.py`. New stats keys also go in `tests/fixtures/stats_keys.json`, new cost report keys the dashboard reads in `tests/fixtures/cost_keys.json`, and new task API keys it reads in `tests/fixtures/tasks_keys.json`; existing ones stay. A change to the text of a `circuit` event or the outage drill's `fault` event also changes the dashboard's `CIRCUIT_RE` / `OUTAGE_RE` / `WAIT_RE` (`test_resilience_panel_contract`). - Scout, the research world and the guard go in `test_scout.py`. Use the `gw` fixture (mock gateway, no warm-up) and fake clocks for the guard. - Live mode, CLIs and deploy scripts go in `test_live.py`. Pass the seams ([Live-path tests](#live-path-tests)) instead of monkeypatching `httpx` or `subprocess`, and set every `Settings` value and environment variable the test depends on. - Loop behaviour goes in `test_system.py` through `sim(...)` with fixed seeds. Assert on trends (rising share, falling escalations), not exact numbers, so tuning the mock does not break tests. The exception is the pinned reference run; reuse the `reference_run` fixture instead of starting another 10-day simulation. - A new `Ledger.add_example` call under `ratchet/` needs a reviewed edit to `ALLOWED_ADD_EXAMPLE_CALLERS` in `test_nightschool.py`. - Never call real APIs in tests. Use `httpx.MockTransport` as in `test_openai_compat_provider_parses_logprobs_and_usage`. - Mark every new test with the requirement ids it proves (`@pytest.mark.req`), keep `parametrize` lists literal so the suite can be counted statically, and update the test counts above. --- # Glossary Source: https://docs.ratchet.dualmindlab.tech/guides/glossary/ This page gives every word Ratchet uses one plain line and a link to the page that explains it, so you can read any other page without getting stuck on a term. Every page in these docs uses the same restaurant. If you remember the cast, most terms explain themselves: - the **apprentice chef** is the student, and the **apprentice's notebook** is the adapter you own; - the **head chef from the agency** is the teacher; - **the pass** is the gateway, which decides who cooks each order; - **the inspector** tastes random plates (audits), and **the recipe book** is the ledger; - **night school** trains a new notebook, the **sealed exam** is the golden set, and the **trial shift**, **serving a few tables first** and **sending the apprentice back to class** are shadow, canary and rollback. ## Terms How the main terms connect, in one picture:
```mermaid flowchart TB req(["Request"]) ==> gw["Gateway"] gw ==> st["Student"] st -->|"escalation"| te["Teacher"] st -.->|"audit"| te te --> led[("Ledger")] led --> ns["Night School"] ns --> gate{{"Release gate"}} gate -.->|"shadow, canary, live"| st class st student class te teacher class led store class gate gate ```
**Reading this diagram.** A request reaches the gateway, which asks the student first. An unsure student's request is an escalation to the teacher; an audit is the teacher checking an answer the student already served. The teacher's answers go to the ledger, Night School trains a new adapter from them, and the release gate decides, through shadow and canary, whether it goes live. Each bold word is defined below.
- **Adapter**: The small set of LoRA weights one student version adds on top of the Nano base; it is the part you own, together with its signed training lineage. (Night School: /guides/night-school/) - **Admin key**: A key from `RATCHET_ADMIN_KEYS` for management calls (the admin API, task writes, training-set proofs); recorded by name only, never stored. (Security: /guides/security/) - **All-teacher baseline**: What a request would have cost if the teacher had answered it alone (`all_teacher_cost_usd` in each response); savings are measured against it. (Cost and savings: /guides/cost-and-savings/) - **Anytime-valid**: A statistical guarantee that holds however often you look at the data; the live alert uses it by default, and the canary gate with `RATCHET_SEQUENTIAL=1`. (Release gate: /guides/release-gate/) Learn more: https://en.wikipedia.org/wiki/Sequential_analysis - **API client**: The name of a gateway API key (`RATCHET_API_KEYS=name:key`); requests, cost, savings and receipts are reported per name, never per key. (Security: /guides/security/) - **Arbitration**: When a shadow candidate disagrees with the live student's served answer, the teacher decides who was right, and its answer becomes training data. (Release gate: /guides/release-gate/) - **Audit**: The teacher answers again a random share of the requests the student served, to measure live quality; a failed audit never fails the request. (Escalation and audits: /guides/escalation-and-audits/) - **Audit rate**: The share of student answers that are audited, `RATCHET_AUDIT_RATE`, 0.05 by default; each audit is a paid teacher call. (Escalation and audits: /guides/escalation-and-audits/) - **Auto task**: A task Ratchet derives from a request's `response_format` JSON schema when the request names no registered task. (Training on your data: /guides/training-on-your-data/) - **Base model**: The pretrained model an adapter sits on. For the student it is NVIDIA Nemotron 3 Nano 4B, which stays frozen. (The big picture: /guides/big-picture/) - **BFCL**: The Berkeley Function-Calling Leaderboard, a public benchmark that scores how exactly models call tools. (Results: /guides/results/) Learn more: https://gorilla.cs.berkeley.edu/leaderboard.html - **Bill of materials (BOM)**: The record of every training source and its license; `GET /ratchet/bom` covers the whole ledger, and each adapter gets its own, written once at training time and signed with Ed25519 when a key is set. (Evidence: /guides/evidence/) - **Break-even**: The number of student answers per endpoint hour at which a GPU endpoint pays for itself: the GPU price per hour divided by the teacher spend each student answer avoids. (Cost and savings: /guides/cost-and-savings/) - **Budget guard**: Scout's limits in live mode (a daily spend cap, a daily run cap and a per-IP hourly limit), kept in the ledger across restarts and always on in live mode. (Scout: /guides/scout/) - **Calibrated baseline**: The ratio of what the teacher really cost on audited answers to the token formula's estimate, used to correct the savings in receipts. (Cost and savings: /guides/cost-and-savings/) - **Calibration**: Choosing a version's threshold: the lowest confidence at which its served answers reach 97% precision on the calibration half of the golden set. (Confidence: /guides/confidence/) - **Canary**: A new version serving a slice of real traffic, 10% and then 50%, while audits check it, before it goes live. (Release gate: /guides/release-gate/) Learn more: https://en.wikipedia.org/wiki/Canary_release - **Cascade**: Try the cheap model first and send the request to the expensive one only when needed. (The request journey: /guides/request-journey/) - **Certificate**: An exact lower bound (one-sided Clopper–Pearson) on how often a version's served answers agree with the teacher, computed on held-out golden rows and signed in its BOM. (Evidence: /guides/evidence/) - **Champion and challenger**: The live version (champion) is scored again on the same golden set as the new candidate (challenger) before the two are compared. (Night School: /guides/night-school/) - **Checkpoint**: A saved snapshot of training progress, so an interrupted job can resume instead of starting over. - **Circuit breaker**: After 5 failed calls in a row the gateway stops calling the student endpoint (or the teacher) for 30 seconds, doubling up to 300 seconds while probes fail; student traffic then goes to the teacher. (The request journey: /guides/request-journey/) Learn more: https://en.wikipedia.org/wiki/Circuit_breaker_design_pattern - **Clopper–Pearson**: An exact method for a confidence bound on a rate measured on a sample; it errs on the side of caution. (Evidence: /guides/evidence/) Learn more: https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval - **Cold start**: The time before any student version exists, when the teacher answers every request. (The request journey: /guides/request-journey/) - **Comparator (field comparator)**: How one field of a task's answer is normalised and compared (`label`, `number`, `date`, `set`, `records`, ...); a missing required field makes the whole answer invalid. (Confidence: /guides/confidence/) - **Confidence**: A score from 0 to 1: 0.4 × the token probability of the best majority sample plus 0.6 × the agreement among the student's samples; an invalid sample gets no vote. (Confidence: /guides/confidence/) - **Confidence interval**: A range that probably contains the true value of a rate measured on a sample. Learn more: https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval - **Confidence sequence**: A range for a rate that holds at every sample size at once, so it can be checked after every audit; release events show a 95% one. (Release gate: /guides/release-gate/) - **Container**: A program packaged with everything it needs, so it runs the same anywhere. Nebius jobs and endpoints each start from one. - **Contested label**: A stored teacher answer that the teacher contradicted when it saw the same request again. It is kept out of training and out of the golden set until Night School asks the teacher again and two of three answers agree. (Evidence: /guides/evidence/) - **Coverage**: The share of requests a student serves, or would serve; more coverage means more savings. (Release gate: /guides/release-gate/) - **Cron**: A scheduler that runs a command at set times, such as every night or every hour. - **Daily limit**: How many requests one API key may send per day. Each key also has a spending limit. (Security: /guides/security/) - **Dashboard key**: A read-only key from `RATCHET_DASHBOARD_KEYS` that reads the dashboard API in live mode and writes nothing. (Security: /guides/security/) - **Day-1 checks**: The checks to run with a real Token Factory key before trusting live mode ([`scripts/day1_check.py`](https://github.com/harshbhx/ratchet/blob/main/scripts/day1_check.py)); model IDs and prices stay unverified until they pass. (Host it on Nebius: /guides/host-on-nebius/) - **Degraded mode**: An opt-in (`RATCHET_ON_TEACHER_FAILURE=serve_student`): when the teacher cannot answer an escalation, the student's own below-threshold answer is served, marked `degraded`, and never stored. (Escalation and audits: /guides/escalation-and-audits/) - **Distillation**: Training a small model to copy a larger model's answers. (What is Ratchet?: /guides/what-is-ratchet/) Learn more: https://arxiv.org/abs/1503.02531 - **Drift**: New kinds of requests the student has never seen, for example after a product launch; escalations rise, then fall as Night School learns them. (Night School: /guides/night-school/) - **Drift alarm**: A label-free alarm on each version's confidence scores. If the scores of a version's answers shift away from how they looked before, it raises one `drift` event and suspends the version's certificate. It needs no audits, and an unchanged stream triggers it with probability at most 1 in 100 over its whole life. `GET /ratchet/drift`. (Evidence: /guides/evidence/) - **Ed25519 signature**: A compact digital signature scheme. Ratchet signs BOMs, proofs and receipts with it so anyone can check them with a public key. (Evidence: /guides/evidence/) Learn more: https://ed25519.cr.yp.to/ - **Edge guard**: Caddy's opt-in `RATCHET_API_GUARD=guarded` in the Nebius VM kit, which blocks `/v1/*`, the mock-only controls and the admin API without the `X-Ratchet-Admin-Token` header. (Host it on Nebius: /guides/host-on-nebius/) - **E-process and e-value**: A running measure of evidence against "precision is at least the floor"; the sequential canary rolls back at an e-value of 20 and the live alert fires at 100, however often they look. (Release gate: /guides/release-gate/) - **Erasure**: Removing stored examples for good, by client, end user, hash or text; the example's hash stays behind as a tombstone so it is never stored again. (Security: /guides/security/) - **Escalation**: A request the student was not confident about, sent to the teacher; the teacher's answer is served and, if valid, stored as training data. (Escalation and audits: /guides/escalation-and-audits/) - **Exam**: A fixed set of questions with known answers, used to score a model before and after training. (Results: /guides/results/) - **Fallback student**: Another name for the hosted fallback (`RATCHET_FALLBACK_STUDENT`). (Host it on Nebius: /guides/host-on-nebius/) - **Fallback teacher**: An opt-in second licensed teacher (`RATCHET_TEACHER_FALLBACK_MODEL`) that answers when the primary teacher is unavailable; its answers are stored only with `RATCHET_STORE_FALLBACK_TEACHER=1`. (Escalation and audits: /guides/escalation-and-audits/) - **Fine-tuning**: Training an already-trained model a little more, on your own examples, so it does one job better. Learn more: https://en.wikipedia.org/wiki/Fine-tuning_(deep_learning) - **Freeze**: An operator's switch (`POST /ratchet/admin/releases/freeze`) that stops promotions and new stages. Rollbacks and alerts keep running. Also what the teacher watch does to training when the teacher's answers drift. (Release gate: /guides/release-gate/) - **FREQUENT rule**: A request seen 3 or more times leaves the golden set and goes to training, so frequent questions are never held out forever. (Night School: /guides/night-school/) - **Frontier**: The certificate at every threshold from 0.95 down to 0.50, corrected so all of them hold at once: the certified trade-off between precision and coverage. (Evidence: /guides/evidence/) - **Gateway**: Ratchet’s server: it receives each request, asks the student first, escalates when needed and records the result. (The big picture: /guides/big-picture/) - **Golden gate**: The first release check: at least 95% precision and 25% coverage on held-out golden rows, and no drop of more than 2 points against the live version. (Release gate: /guides/release-gate/) - **Golden set**: About 10% of unique requests (`RATCHET_GOLDEN_FRACTION`), chosen by hash, held out from training and used only to test versions. (Night School: /guides/night-school/) - **GPU**: A graphics processor, used to train and run models much faster than an ordinary processor could. - **Ground truth**: The known-correct answer. Ratchet usually measures agreement with the teacher, which is not the same thing. - **Hash**: A fixed-length fingerprint computed from some data. The same data always gives the same hash. - **Held-out**: Data a version never trained on; the even-numbered golden rows set its threshold and the odd-numbered rows measure its precision. (Night School: /guides/night-school/) - **Held-out certificate**: Every golden example's proof of absence from an adapter's committed training set, checkable from the BOM alone. (Evidence: /guides/evidence/) - **Hosted fallback**: The untuned Nemotron 3 Nano 30B on Token Factory that answers student calls while no student endpoint is set; paid per token, held to the default threshold of 0.80, and covered by no certificate. (Host it on Nebius: /guides/host-on-nebius/) - **Hot-load**: Loading a new adapter into a running vLLM server without restarting it. (Host it on Nebius: /guides/host-on-nebius/) - **Human label**: A customer's correction (`POST /v1/ratchet/feedback`) or an attested import, stored as a training example with the provenance `human`; a verdict on its own stores nothing. (Training on your data: /guides/training-on-your-data/) - **Human precision**: The share of `correct` verdicts on the answers a version served itself, reported next to the audited precision. It is never used to promote or roll back a version, because people report what they notice. (Training on your data: /guides/training-on-your-data/) - **Idempotency key**: An `Idempotency-Key` header that makes a retried request get the first answer back without running the cascade again. (The request journey: /guides/request-journey/) Learn more: https://en.wikipedia.org/wiki/Idempotence - **Inconclusive**: Under the sequential rule, a canary stage that reached 400 audits without enough evidence either way; the version is retired, not rolled back. (Release gate: /guides/release-gate/) - **Invariant**: One of the rules the code must never break, such as "never train on student outputs"; each one is mapped to code and tests. (Security: /guides/security/#the-safety-rules-the-loop-never-breaks) - **Judge**: Nemotron 3 Super on Token Factory, which scores free-text answers in the golden eval; tasks with typed fields are compared field by field instead, with no judge. (Night School: /guides/night-school/) - **Judges' reserve**: Scout runs, spend and Tavily credits held back each day for evaluators, unlocked by a code sent once to `POST /scout/judge`. (Scout: /guides/scout/) - **L40S**: An NVIDIA data-centre GPU with 48 GB of memory. Ratchet trains and serves the student on one. (Host it on Nebius: /guides/host-on-nebius/) - **label_p (labelling probability)**: The designed chance that the teacher labels a student answer (1 for an escalation, the audit rate for a served answer), logged with every request so the replay can weight each label. (Evidence: /guides/evidence/) - **Learn-then-Test**: An opt-in calibration (`RATCHET_CALIBRATION=ltt`) that picks the threshold from a fixed grid so its precision is certified. (Evidence: /guides/evidence/) - **Ledger**: The SQLite file that records every request's metadata, every stored training example with its provenance, every version and every event. (The ledger: /guides/ledger/) - **License-clean distillation**: Distilling only from teachers whose licenses allow training on their outputs; closed models whose terms forbid it are refused in code. (Licenses: /guides/licenses/) - **Live mode**: Running against real Token Factory, a real student endpoint and real web search; every teacher call costs money. (Host it on Nebius: /guides/host-on-nebius/) - **Live-run harness**: `python -m ratchet.liverun`, which sends rounds of traffic through a live gateway under a spending cap and writes the results file every live number must come from; its smoke mode has run once (20 requests on Oct 23, about $0.01, 0 students and 0 audits, so no savings number), the measured `run` mode has not. (Evidence: /guides/evidence/) - **Live version**: The version that serves all student traffic after its canary passed; it is alerted on, never rolled back automatically. (Release gate: /guides/release-gate/) - **llms.txt**: A plain-text file at /llms.txt that lists a site's pages for language models to read. These docs publish one. Learn more: https://llmstxt.org - **LoRA**: Low-Rank Adaptation: training a small set of extra weights (an adapter) on top of a frozen base model, cheap to train and to swap. (Night School: /guides/night-school/) Learn more: https://arxiv.org/abs/2106.09685 - **Mamba (hybrid Mamba)**: A model design that mixes attention layers with Mamba state-space layers, which handle long text cheaply. Nemotron Nano is built this way. (Night School: /guides/night-school/) Learn more: https://arxiv.org/abs/2312.00752 - **McNemar test**: A test that compares two models on the same rows by looking only at the rows where they disagree. (Release gate: /guides/release-gate/) Learn more: https://en.wikipedia.org/wiki/McNemar%27s_test - **Merkle root**: One hash that commits to a whole set; an adapter's BOM signs one over its training set and one over its golden set, and a savings receipt signs one over a day's request metadata. (Evidence: /guides/evidence/) Learn more: https://en.wikipedia.org/wiki/Merkle_tree - **Mock mode**: Running with mock models and search, with no keys and no GPU, for tests and development; its numbers say nothing about how good a real model is. (Quickstart: /guides/quickstart/) - **Nebius AI Cloud**: The cloud whose Token Factory, Serverless Jobs and Object Storage Ratchet uses for model calls and for training. (Host it on Nebius: /guides/host-on-nebius/) Learn more: https://nebius.com/ - **Nemotron**: NVIDIA's family of open language models: Nano 4B is the student, Super the judge and Ultra the teacher. (The big picture: /guides/big-picture/) Learn more: https://developer.nvidia.com/nemotron - **Nemotron Open Model License**: The license on NVIDIA's Nemotron base models. The base stays under it; the adapter you train is yours. (Licenses: /guides/licenses/) - **Night School**: The nightly job: build the dataset, retrain the adapter from the base on all teacher answers so far, evaluate it on the golden set, and hand it to the release gate. (Night School: /guides/night-school/) - **NOTICE.md**: The file next to each adapter's weights that names the base model's license and each teacher's license and says the targets were teacher outputs only; its SHA-256 is in the signed BOM. (Licenses: /guides/licenses/) - **No-train end user**: An end user registered with `POST /v1/ratchet/no-train`, whose requests are never stored as training examples. (Security: /guides/security/) - **NVIDIA small-model paper**: NVIDIA Research’s paper arguing that small language models suit agent work, with six steps for moving from a large model to a small one. (Why it matters: /guides/why-it-matters/) Learn more: https://arxiv.org/abs/2506.02153 - **Object Storage**: Nebius's S3-compatible storage; one bucket is mounted at `/data` in both the training job and the student endpoint. (Host it on Nebius: /guides/host-on-nebius/) Learn more: https://docs.nebius.com/object-storage - **Off-path checks**: In live mode, audits and shadow checks run after the answer is sent (`RATCHET_SIDECAR_WORKERS`, 4 by default there), so the client does not wait for the teacher. (Escalation and audits: /guides/escalation-and-audits/) - **OpenAI-compatible API**: An API that takes the same requests and returns the same responses as OpenAI’s chat endpoint, so existing client code works after a URL change. (What is Ratchet?: /guides/what-is-ratchet/) - **OpenMDW-1.1**: The license of Nemotron 3 Ultra; it places no restrictions on outputs, so its answers may be used to train a student. (Licenses: /guides/licenses/) Learn more: https://openmdw.ai/license/1-1/ - **Opt-out**: A way to keep a request's teacher answers out of training (`store: false`, `X-Ratchet-Store: none`, a key's `store=0`, no-store tasks, no-train end users); opt-outs only ever reduce storage. (Security: /guides/security/) - **Paired gate**: An opt-in golden-gate rule (`RATCHET_PAIRED_GATE=1`) that compares the candidate and the live version row by row on the same golden rows with exact McNemar tests. (Release gate: /guides/release-gate/) Learn more: https://en.wikipedia.org/wiki/McNemar%27s_test - **PEFT**: Parameter-efficient fine-tuning: training a small set of extra weights instead of the whole model. Also the name of Hugging Face’s library for it. (Host it on Nebius: /guides/host-on-nebius/) Learn more: https://github.com/huggingface/peft - **Per-task calibration**: When several tasks share one adapter, each gets its own threshold and held-out numbers, and a task that fails its golden check is disabled on that version. (Confidence: /guides/confidence/) - **PII redaction**: Replacing emails, phone numbers, card numbers, IBANs, national IDs, IP addresses and secrets with placeholders such as `[EMAIL]` before anything becomes a training example; names and street addresses are not caught yet. (Security: /guides/security/) Learn more: https://en.wikipedia.org/wiki/Personal_data - **Pinned task**: A task that always goes to the teacher and is never stored as training data, such as Scout's `research-plan` and `research-write`; only an operator's spec pins a task. (Scout: /guides/scout/) - **Precision**: Of the answers a student serves (or would serve), the share that agree with the teacher; agreement with Nemotron 3 Ultra, not ground truth. (Release gate: /guides/release-gate/) Learn more: https://en.wikipedia.org/wiki/Precision_and_recall - **Preemptible (spot) GPU**: A GPU rented at a discount on the condition that the provider can take it back at short notice, so training must resume. (Host it on Nebius: /guides/host-on-nebius/) - **Provenance**: Where a training example came from: which teacher wrote it and under which license, recorded when it is stored. (The ledger: /guides/ledger/) - **QLoRA**: LoRA training on a base model squeezed to 4-bit numbers, so a larger model fits on a small GPU. (Training on your data: /guides/training-on-your-data/) Learn more: https://arxiv.org/abs/2305.14314 - **Random split**: A share of eligible live calls sent on purpose to the hosted Nano 30B, labelled as such, so its answers can be measured. (FAQ: /guides/faq/) - **Readiness and liveness**: `/readyz` says whether the gateway can answer requests (for load balancers and uptime monitors); `/healthz` says only that the process is up (for restarts). (Host it on Nebius: /guides/host-on-nebius/) - **Receipt (savings receipt)**: A signed record per ledger day, for the gateway and for each API client, of what was saved, with a Merkle tree over the day's request metadata and the fee taken on the 95% lower bound of the savings. (Cost and savings: /guides/cost-and-savings/) - **Release gate**: The checks a new version must pass before it serves traffic: the golden gate, shadow, canary at 10% and 50%, then live, with automatic rollback of a bad canary. (Release gate: /guides/release-gate/) - **Replay (ledger replay)**: Recomputing one version's coverage, precision and cost from its logged traffic at another threshold or audit rate, with no new model call. (Evidence: /guides/evidence/) - **Request id**: `req_` and 32 hex digits, sent in every response (`x-request-id`, `ratchet.request_id`). It names the request in feedback calls and request lookups; it never holds text. (The request journey: /guides/request-journey/) - **Retention**: How long stored examples and request rows are kept (`RATCHET_EXAMPLE_RETENTION_DAYS`, `RATCHET_REQUEST_RETENTION_DAYS`; 0 keeps them forever). (Security: /guides/security/) - **Rollback**: Pulling a canary automatically when the 95% upper bound of its audited precision falls below 92% (or, with the sequential rule, when its e-value reaches 20). (Release gate: /guides/release-gate/) - **Route**: The recorded reason a request was answered the way it was: `student`, `confidence` (escalated), `cold_start`, `pinned`, `tools`, `student_breaker` and others. (The request journey: /guides/request-journey/) - **Scout**: The multi-agent research tool: a planner and a writer pinned to the teacher, a web searcher, and many small triage and reader calls the student learns. (Scout: /guides/scout/) - **Self-consistency (agreement)**: How many of the student's samples (3 by default) give the same answer, compared with the task's comparators. (Confidence: /guides/confidence/) - **Serverless Endpoint**: A Nebius service that serves a container at a URL; Ratchet runs vLLM with the student there on one L40S, billed while it runs. (Host it on Nebius: /guides/host-on-nebius/) Learn more: https://docs.nebius.com/serverless/overview - **Serverless Job**: A Nebius service that runs a container to completion; Night School trains each adapter in one, on one L40S. (Host it on Nebius: /guides/host-on-nebius/) Learn more: https://docs.nebius.com/serverless/overview - **Set commitment**: The Merkle root in an adapter's BOM that fixes its exact training (or golden) set, so a single example can be proved in or out of it. (Evidence: /guides/evidence/) - **SHA-256**: A hash function that turns any file into a 64-character fingerprint. Change one byte and the fingerprint changes. Learn more: https://en.wikipedia.org/wiki/SHA-2 - **Shadow**: A candidate version answering real traffic silently next to the live student; nothing it says is served, and disagreements go to the teacher. (Release gate: /guides/release-gate/) - **Spec hash**: A hash of the rules in a task's spec that decide agreement; when it changes, the task goes to the teacher until a night scores the version again. (Training on your data: /guides/training-on-your-data/) - **SQLite**: A database kept in a single file. The ledger is a SQLite file. (The ledger: /guides/ledger/) Learn more: https://www.sqlite.org - **Student**: The small model that answers first: NVIDIA Nemotron 3 Nano 4B with the adapter you own, served by vLLM. (The big picture: /guides/big-picture/) - **Take rate**: The share of the savings' billable lower bound that is the fee in a receipt (`RATCHET_TAKE_RATE`, 25% by default). (Cost and savings: /guides/cost-and-savings/) - **Task**: A named kind of request, such as `ticket-triage`, that says which answer fields must match, which answers are valid and whether it is pinned; set with the `X-Ratchet-Task` header. (Training on your data: /guides/training-on-your-data/) - **Task registry**: The tasks a request can name: the built-in ones, those in `RATCHET_TASKS_FILE` and those written through the admin API. (Training on your data: /guides/training-on-your-data/) - **Tavily**: A web-search API built for AI agents. Scout uses it to find sources. (Scout: /guides/scout/) Learn more: https://tavily.com - **Teacher**: The large model that answers what the student cannot: NVIDIA Nemotron 3 Ultra on Token Factory; only its answers and attested human labels become training data. (The big picture: /guides/big-picture/) - **Teacher watch**: A check that the teacher still gives the same labels. A repeated request, or one of Night School's probes, is compared with its stored label, once per request per check epoch. Enough disagreement freezes training until an operator acknowledges it (`POST /ratchet/teacher/ack`). A new model id in the teacher's responses starts a new epoch. `GET /ratchet/teacher`. (Evidence: /guides/evidence/) - **Text-to-SQL**: Turning a plain-English question into a SQL query. It is the task the published adapter is trained on. (Text-to-SQL adapter: /guides/text-to-sql/) - **Threshold**: The confidence at or above which a student answer is served, calibrated per version and per task. (Confidence: /guides/confidence/) - **Token**: A small piece of text, roughly a short word or part of one, that models read and that Token Factory bills by. - **Token Factory**: Nebius's OpenAI-compatible inference service for open models, billed per token; Ratchet's teacher, judge and hosted fallback run there. (Host it on Nebius: /guides/host-on-nebius/) Learn more: https://nebius.com/token-factory - **Tombstone**: The hash an erased example leaves behind, so the same example is refused from then on, from any client. (Security: /guides/security/) - **Tool calling**: A model replying with a structured call to a named function instead of prose, which code then runs. (Results: /guides/results/) - **Training example**: One request with the teacher’s answer to it, stored with its provenance and used to teach the student. (The ledger: /guides/ledger/) - **Version**: One trained student adapter (`student-v1`, `student-v2`, ...) with its own eval, threshold and state: `candidate`, `rejected`, `shadow`, `canary`, `live`, `retired` or `rolled_back`. (Release gate: /guides/release-gate/) - **vLLM**: The open-source model server that runs the student on the endpoint and loads each new LoRA adapter without a restart. (Host it on Nebius: /guides/host-on-nebius/) Learn more: https://github.com/vllm-project/vllm - **Weights**: The numbers a model learns in training. An adapter is a small set of extra weights. - **Wilson bound**: An interval for a proportion that behaves well on small samples; the default canary rollback uses its upper bound, and it is valid for one look at a fixed sample. (Release gate: /guides/release-gate/) Learn more: https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval ## For engineers Where the names live in the code (links go to GitHub): | Term | Where in the code | |---|---| | Cascade, route, escalation, audit | [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py): `Ratchet.handle`, `ROUTES`, `Ratchet.threshold` | | Confidence, calibration, sampling | [`ratchet/confidence.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/confidence.py): `score`, `calibrate_threshold`, `STUDENT_SAMPLING`; per task in [`ratchet/calibration.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/calibration.py) | | Release gate, shadow, canary, rollback, states | [`ratchet/release.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/release.py): `ReleaseGate`, `ReleasePolicy`, `CANARY_STEPS` | | Wilson, e-values, confidence sequences, certificates | [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py), [`ratchet/certificate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/certificate.py) | | Golden set, FREQUENT rule, held-out eval | [`ratchet/nightschool/dataset.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/dataset.py) (`FREQUENT`), [`ratchet/nightschool/evaluate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/evaluate.py) (`cascade_eval`) | | Night School | [`ratchet/nightschool/orchestrate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/orchestrate.py) (`run_night`), [`ratchet/live.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/live.py) | | Ledger, provenance, tombstones | [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py): `Ledger`, `add_example` | | Licenses, license-clean | [`ratchet/licenses.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/licenses.py): `require_distillable`, `BUILTIN`, `BASE_MODELS` | | BOM, NOTICE.md | [`ratchet/bom.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/bom.py), [`ratchet/nightschool/notices.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/notices.py) | | Merkle roots, set commitments, proofs | [`ratchet/merkle.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/merkle.py), [`ratchet/proof.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/proof.py) | | Receipts, take rate | [`ratchet/receipts.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/receipts.py) | | Replay, label_p | [`ratchet/replay.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/replay.py) | | Break-even, cost report | [`ratchet/cost.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/cost.py): `cost_report` | | Circuit breakers | [`ratchet/resilience.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/resilience.py): `Breaker`, `BreakerPolicy` | | Tasks, comparators, spec hash | [`ratchet/tasks.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/tasks.py), [`ratchet/taskspec.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/taskspec.py), [`ratchet/compare.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/compare.py) | | Redaction, opt-outs, erasure, retention | [`ratchet/redact.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/redact.py), [`ratchet/privacy.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/privacy.py) | | API, admin and dashboard keys | [`ratchet/auth.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/auth.py): `KeyRing` | | Readiness and liveness | [`ratchet/readiness.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/readiness.py), `/healthz` in [`ratchet/gateway.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/gateway.py) | | Scout, budget guard, judges' reserve | [`ratchet/scout/`](https://github.com/harshbhx/ratchet/tree/main/ratchet/scout): `guard.py`, `judge.py`, `credits.py` | | Settings and list prices | [`ratchet/config.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/config.py): `Settings`, `DEFAULT_PRICES` | | The rules the loop never breaks | Each one has code that enforces it and tests that check it; see [Security](/guides/security/#the-safety-rules-the-loop-never-breaks) | Environment variables are listed in [CONFIGURATION.md](/reference/configuration/) and endpoints in [API.md](/reference/api/). ## Common questions ### Is "the student" the base model or the adapter? Both together: the student is Nemotron 3 Nano 4B with one version's adapter on top. You own the adapter and its signed lineage; the Nano base stays under the NVIDIA Nemotron Open Model License. ### What is the difference between the hosted fallback and the fallback teacher? The hosted fallback stands in for the **student** when no GPU endpoint is set: an untuned Nano 30B, paid per token. The fallback teacher stands in for the **teacher** when it is down: a second licensed model you opt into. Neither is the tuned student, and the pages say so. ### Why are so many terms statistical? Because Ratchet makes release decisions by itself, and each decision rests on counts: audited answers that agree, held-out rows served correctly. Terms like Wilson bound, e-value and certificate are the rules that turn those counts into a decision you can recompute. The [release gate](/guides/release-gate/) and [evidence](/guides/evidence/) pages explain them with examples. ### A term is missing. Where else can I look? The [FAQ](/guides/faq/), the [architecture](/reference/architecture/) reference, and the search box at the top of every page. --- # FAQ Source: https://docs.ratchet.dualmindlab.tech/guides/faq/ This page answers the questions people ask most about Ratchet in a few honest sentences each, including what is not built or not proven yet, and links to the page that tells the whole story. Imagine the waiter's notebook of questions guests always ask. Is the restaurant allowed to copy the head chef's recipes? Is the apprentice already cooking for real guests? What if the apprentice gets a dish wrong? How much does all this cost? Each answer here is short, like a waiter's answer at the table. If you want the long version, every answer points to the page that has it. ## How it really works The whole loop in four steps; [the big picture](/guides/big-picture/) has every component. 1. **Your app sends a request** to the gateway, in the OpenAI chat-completions format. 2. **The student tries first** : it samples three answers, scores its confidence and serves the answer if the confidence reaches the version's threshold. 3. **Otherwise the teacher answers** , and its answer is stored with its provenance (which teacher, which license). A random 5% of the student's answers are also checked by the teacher (audits). 4. **Every night, Night School** trains a new adapter on the teacher's answers, tests it on held-out data, and releases it through shadow, canary and live, with automatic rollback of a bad canary.
```mermaid flowchart LR gw(["Gateway"]) ==> st["Student"] st -->|"unsure"| te["Teacher"] te --> led[("Ledger")] led --> ns["Night School"] ns -.-> st class st student class te teacher class led store ```
**Reading this diagram.** Every request reaches the gateway, which asks the student first; a confident student answers at once. Unsure requests go to the teacher, whose answers are stored in the ledger. Each night Night School trains a new student from them, and the dotted line is the gated release that may replace the current one.
- **Live site, as of 29 Oct 2026:** every model call runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)). - **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud (one spot L40S Serverless Job): 40.7% on exam 1 and 34.5% on exam 2, originally 39.0% and 33.0% ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)). - **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL, 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). - **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). Ratchet's own Nano 4B adapter is not connected to the live site; it goes live only after it passes the golden gate. Night School runs by hand today and its nightly timer is scripted. The one-command host (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. Run every script with `DRY_RUN=1` first. Every number on this site is an estimate or read from the text-to-SQL and tool-calling results files, and says which. ## For everyone ### What is Ratchet, in one sentence? An OpenAI-compatible gateway that answers each request with a small open model when it is confident, asks a large open model when it is not, and every night trains the small model on the large model's answers, so the share of traffic it can handle grows. [What is Ratchet?](/guides/what-is-ratchet/) ### Is this legal? Ratchet only learns from teachers whose licenses allow training on their outputs. The default teacher, Nemotron 3 Ultra, is under [OpenMDW-1.1](https://openmdw.ai/license/1-1/), which "does not impose any restrictions or obligations with respect to any use, modification, or sharing of any outputs" (license page last updated May 27, 2026; checked Oct 21, 2026). The student's base, Nemotron 3 Nano 4B, stays under the NVIDIA Nemotron Open Model License, which asks you to pass a copy of that license on when you distribute it. One question is open: Token Factory's Terms of Service §5(f) ("competing service") is broad, and it was put to Nebius support in writing on Oct 19, 2026, with no final answer yet. This is not legal advice; check the licenses yourself. [Licenses](/guides/licenses/) ### Can I use GPT as the teacher? No, and the code refuses it. Every teacher goes through `licenses.require_distillable` at startup, which rejects closed model families. Closed models such as GPT come with terms that forbid using their outputs to develop competing models ([OpenAI Business Terms §3.3(e)](https://openai.com/policies/business-terms/)). You can add another open teacher in `RATCHET_TEACHER_LICENSES`, but only under an allowed open license, and never for a closed family or router alias Ratchet recognises. The README's ["Why license-clean"](https://github.com/harshbhx/ratchet#why-license-clean) section has the full reasoning. [Licenses](/guides/licenses/) ### Do I own the model? No. You own the LoRA adapter Night School trains and its signed training lineage (its bill of materials). The student is NVIDIA's Nemotron 3 Nano 4B with that adapter on top, and the Nano base stays under NVIDIA's Nemotron Open Model License. When you hand an adapter on, `python -m ratchet.live export` writes it with its BOM and a `NOTICE.md` naming every license. [Licenses](/guides/licenses/) ### What if the student is wrong? Sometimes it will be, and Ratchet is built to keep that rare and visible: - The student serves only when its confidence reaches its version's threshold, which is set on half of a held-out golden set for 97% precision and measured on the other half. A new version must reach at least 95% there before it serves anyone. - 5% of the answers it serves are audited by the teacher (`RATCHET_AUDIT_RATE`), and every disagreement becomes training data. - A canary is rolled back automatically when the 95% upper bound of its audited precision falls below 92%. A live version is never rolled back automatically: an anytime-valid alert tells the operator, who can retire it or turn the student off in one admin command. "Precision" means agreement with Nemotron 3 Ultra, not ground truth. [Release gate](/guides/release-gate/), [escalation and audits](/guides/escalation-and-audits/) ### Does it work on real traffic yet? Not yet. As of Oct 29, 2026 the teacher side has run for real (Token Factory calls built the text-to-SQL data) and Night School ran once on Nebius (rejected by the gate), but the loop has not served real traffic and Ratchet's own adapter is not connected to the live gateway, so there are no live savings numbers. Live figures will come only from a published live-run results file. The measured results are the text-to-SQL and tool-calling adapters' exams, on [Results](/guides/results/); the loop's own numbers will come only from a published live run. [Evidence](/guides/evidence/) ### What does it cost? All of these are estimates, with their sources: - **A Nebius CPU VM, if you run one** (the live gateway is on Cloudflare today): about $72.50 for 48 days (`cpu-e2`, prices as of Oct 20, 2026). - **The teacher**: Ultra at $1 per million input tokens and $3 per million output tokens (Token Factory list price, checked 29 Oct 2026; the Nano 30B is $0.06 in and $0.24 out), roughly $1.30 per 1,000 teacher calls. - **The GPU endpoint**: about $32 for every full day it runs (an L40S at $1.35 a GPU-hour on [Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026; an estimate). [Cost and savings](/guides/cost-and-savings/), [Host it on Nebius](/guides/host-on-nebius/) ### When does self-hosting lose money? When too few requests reach the student. A GPU endpoint bills every hour it runs, whether requests arrive or not. Break-even is the GPU price per hour divided by the teacher spend each student answer avoids: with estimated prices ($1.35 a GPU-hour for an L40S, about $0.001 avoided per ticket-triage answer, from list prices), that is about 1,350 student answers an hour, roughly 32,000 a day. At a few thousand requests a day, a 24/7 endpoint loses money, and Ratchet's own cost report says so and recommends the hosted fallback or scaling the endpoint to zero. [Cost and savings](/guides/cost-and-savings/) ### What happens when the GPU endpoint is off? Leave `RATCHET_STUDENT_BASE` empty and the student's calls go to the hosted fallback : the untuned Nemotron 3 Nano 30B on Token Factory, paid per token. It is held to the default confidence threshold (0.80) and no certificate covers it. Responses still say `served_by: student`, but the `model` field names Nano 30B, and the pages show the hosted badge and hide the adapter's precision. [Host it on Nebius](/guides/host-on-nebius/) ### How is this different from a router or a distillation service? A cascade router saves money on each call but learns nothing: every call stays rented. Distillation services turn traffic into a fine-tune, but the student usually stays on the provider's platform, you start each job yourself, and none of the services we found records each example's teacher license when the example is captured. Ratchet does the loop by itself: the license is recorded when each example is captured and signed into the adapter's BOM, a new adapter is trained every night, a statistical gate decides each release, and the cost report says when the small model loses money. The pieces are prior art (FrugalGPT, Cache & Distil, NVIDIA's Data Flywheel Blueprint and others); putting them in one loop is what is new. [What is Ratchet?](/guides/what-is-ratchet/) ## For evaluators ### What should I look at first? The story page at https://ratchet.dualmindlab.tech, then `/scout` (ask an example question and watch each call get answered by the student or the teacher), then `/dashboard`. [For judges](/guides/for-judges/) ### Is the live gateway real? Yes. The gateway at live.ratchet.dualmindlab.tech runs in live mode and calls the real teacher on Token Factory. It has no GPU and Ratchet's own adapter is not connected to it. Ultra teaches, and a random split sends 40% of eligible calls to the hosted Nano 30B (a quarter of those re-checked by Ultra), labelled as such. `GET /healthz` reports the `mode` the gateway runs in. [Host it on Nebius](/guides/host-on-nebius/) ### Which numbers are real? The text-to-SQL results (the headline) and the smaller tool-calling results, both read from their results files on [Results](/guides/results/); none of the loop's numbers are live yet. Costs from the docs say "estimate" with their date. Live numbers will come only from a published, non-smoke live-run results file (`evidence/live/results.json`), which does not exist yet. [Evidence](/guides/evidence/) ### Which NVIDIA models and Nebius services does it use? | Role | Model or service | |---|---| | Teacher | `nvidia/Nemotron-3-Ultra-550b-a55b` on Token Factory | | Judge (free-text grading) | `nvidia/nemotron-3-super-120b-a12b` on Token Factory | | Hosted fallback student | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` on Token Factory | | Student | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` with a LoRA adapter, vLLM on a Serverless Endpoint | | Night School | Serverless Jobs on an L40S, with TRL and PEFT | | Datasets and adapters | An Object Storage bucket mounted in the job and the endpoint | | Live gateway host | Cloudflare Workers with Containers today; a Nebius CPU VM (`2vcpu-8gb`) is scripted and tested offline, ready to run with `make nebius-up` (`make nebius-plan` prints every step) | | Scout web search | Tavily Search API (not NVIDIA or Nebius) | Token Factory listed no fine-tuning or LoRA serving for Nemotron when this design was chosen (research notes, Oct 19, 2026), and Nebius's [post on post-training in Token Factory](https://nebius.com/blog/posts/post-training-in-token-factory) names no Nemotron model to fine-tune (read Oct 26, 2026), so the student is trained and served on Nebius AI Cloud. [Host it on Nebius](/guides/host-on-nebius/) ### Where are the Token Factory and Tavily calls? In live mode, every teacher call goes to Token Factory: cold starts, escalations, audits, shadow arbitrations, and Scout's planner and writer, which are pinned to the teacher. Scout's searcher calls the Tavily Search API. [Scout](/guides/scout/) ### Can I check the claims myself? Yes, offline. `make test` runs the 1,934 tests with no network and no GPU; `python -m ratchet.bom verify` checks an adapter's signed BOM and its `NOTICE.md`; `python -m ratchet.stats check` recomputes a certificate from its signed counts; `python -m ratchet.proof heldout` checks that no golden example was trained on; `python -m ratchet.receipt verify` checks a savings receipt. [Evidence](/guides/evidence/) ### Why does a Scout run say "search unavailable"? The day's public share of Tavily credits ran out, or a search failed. The run still finishes, labelled: the planner has already called Token Factory, and the writer answers without (more) sources. Runs held for judges keep searching from their own reserve. [Scout](/guides/scout/) ### Is anything held back for judges? Yes, on the server side. `RATCHET_SCOUT_JUDGE_CODE` and `RATCHET_SCOUT_JUDGE_RUNS_PER_DAY` hold part of each day's Scout runs, spend and Tavily credits back from the public; the code is shared with evaluators directly, never published, and opens a session through `POST /scout/judge` and a signed cookie. On the Scout page, an "I am a judge" link under the question box opens a field for the code; the page keeps it in memory only and shows how many judge runs are left today. [Scout](/guides/scout/) ## For engineers The design is in [ARCHITECTURE.md](/reference/architecture/) and a file-by-file map in the README's [repo map](https://github.com/harshbhx/ratchet#repo-map); endpoints are in [API.md](/reference/api/) and settings in [CONFIGURATION.md](/reference/configuration/). The answers below point to the guide pages. ### How do I point my app at Ratchet? Change the base URL of your OpenAI SDK to the gateway's `/v1` and use the model name `ratchet`: ```python from openai import OpenAI client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused") # or your RATCHET_API_KEYS key resp = client.chat.completions.create( model="ratchet", messages=[{"role": "user", "content": "I was charged twice this month."}], extra_headers={"X-Ratchet-Task": "ticket-triage"}, ) print(resp.model_extra["ratchet"]) # served_by, confidence, cost_usd, ... ``` [Quickstart](/guides/quickstart/) ### How do I tell Ratchet what kind of request this is? Name the task in the `X-Ratchet-Task` header or in `metadata.ratchet_task`. A task's spec says which answer fields must agree and which answers are valid; you can register your own in a tasks file (`RATCHET_TASKS_FILE`) or through the admin API. An unregistered name is distilled by default (`RATCHET_UNKNOWN_TASKS=distill`; `teacher` and `reject` are the other choices). [Training on your data](/guides/training-on-your-data/) ### Do streaming, tools and structured outputs work? Yes, with limits. Streaming works in the OpenAI chunk format, but the whole answer is ready before the first chunk, because confidence needs the whole answer. Tool calls pass through to the teacher alone and are never stored, so tool requests are not distilled yet. A `response_format` is checked on every answer, and a JSON schema without a registered task gets an automatic one. [The request journey](/guides/request-journey/) ### What happens when the teacher is down? The gateway retries the teacher (up to 2 more attempts within 180 seconds). After 5 failed calls in a row its circuit breaker opens, and requests that need the teacher get a 503 `teacher_circuit_open` with `Retry-After` at once instead of waiting. Two opt-in stand-ins exist: a second licensed teacher (`RATCHET_TEACHER_FALLBACK_MODEL`), and degraded mode (`RATCHET_ON_TEACHER_FAILURE=serve_student`), which serves the student's own answer, marked and never stored. Student failures never fail a request: they fall back to the teacher. [Escalation and audits](/guides/escalation-and-audits/) ### What data is stored? The `requests` table holds metadata only (task, route, confidence, cost, tokens), never prompt or answer text. Training examples hold the prompt and the teacher's answer, after PII redaction, with the teacher, its license and the number of placeholders. API keys are recorded by name only, end users only as keyed hashes, and live-mode logs carry no prompt or answer text. The student's own answers are never stored as training data. [The ledger](/guides/ledger/), [security](/guides/security/) ### Can I delete data? Yes. Stored examples can be erased for good by client, end user, hash or text (`DELETE /v1/ratchet/examples` for a client's own end users, `POST /ratchet/admin/privacy/erase` and `python -m ratchet.admin erase`, both dry runs unless told otherwise). An erased example leaves only a hash, so it is never stored again. Versions that may have trained on it are reported, and their signed BOMs are left as written; the next nights retrain from the base without it. Retention windows and storage opt-outs keep data from piling up or being stored at all. [Security](/guides/security/) ### Which key do I need for which route? Four kinds of key exist. An API key (`RATCHET_API_KEYS`) calls `/v1/*`. A dashboard key (`RATCHET_DASHBOARD_KEYS`) reads stats, costs and other management data. An admin key (`RATCHET_ADMIN_KEYS`) also writes: versions, switches, tasks, erasure. The [endpoint map](/api/endpoints/) lists every route with its level. ### How do I send feedback on an answer? Every chat response carries `ratchet.request_id`. Send it to `POST /v1/ratchet/feedback` with a verdict, `correct` or `incorrect`, and for `incorrect` optionally the right answer and the request's messages. A stored correction becomes a human-labelled training example. A verdict never promotes or rolls back a version. See the [examples](/api/endpoints/#three-examples). ### How do I retry a request safely? Send an `Idempotency-Key` header. The first answer is kept in memory for a short time (`RATCHET_IDEMPOTENCY_TTL_S`) and sent again, marked `Idempotent-Replayed: true`, without a second cascade, a second charge or a second row. Retries also never push an example toward the frequent-request rule that moves it out of the golden set. ### What is the difference between a pinned task and an auto task? A pinned task always goes to the teacher and is never stored as training data. Only an operator can pin one, never a request. An auto task is made by the gateway from a request's `response_format` JSON schema when the request names no registered task; it is stored and distilled like any other task. Both are listed in the [glossary](/guides/glossary/). ### How does Ratchet notice that things have changed? Three watchers, none of them a release signal on its own. The audits measure precision against the teacher. The drift alarm watches the student's confidence scores without any labels. The teacher watch compares the teacher's new answers with its stored ones and freezes training if they disagree too much, until an operator acknowledges it. See [evidence](/guides/evidence/) and the [release gate](/guides/release-gate/). ### Why SQLite? Zero setup, easy backups, easy to inspect, and fast enough for one gateway process at this scale. It runs in WAL mode so dashboard reads do not block writes. The price is one gateway process with one worker; a Postgres ledger is planned. [The ledger](/guides/ledger/) ### Why Nemotron? The teacher's license allows training on its outputs, which is the whole point. Nemotron 3 Nano 4B is an open model that fits a single L40S and that vLLM serves with LoRA adapters. Choosing the base is a manual decision: Ratchet does not compare base models. [Licenses](/guides/licenses/) ### Can I run more than one gateway process? Not on one ledger. Run one process with one uvicorn worker: the ledger is one SQLite file behind one lock, and the release gate is not built for concurrent writers. Detecting a second process is planned. [Host it on Nebius](/guides/host-on-nebius/) ### How do I run the tests, and do they call real APIs? `make test` (pytest) needs no network and no GPU and takes about two minutes; real APIs are never called (the tests use `httpx.MockTransport` and fakes). `make lint` and `make reference` are the other checks to run before a commit. [Quickstart](/guides/quickstart/) ## For buyers ### How is it priced? The core is open source under Apache-2.0 and free to self-host. A managed offering is a plan, not a product: the idea is a share of verified monthly savings (see [the business case](/guides/business/)), with no fee when there are no savings. In the code, a receipt's fee is `RATCHET_TAKE_RATE` (25% by default) of the one-sided 95% lower bound of the savings. ### Can I check the savings I would be billed for? Yes. Every ledger day gets a signed savings receipt for the gateway and for each API client. It calibrates the teacher baseline on audited teacher bills, bills only the lower bound, chains to the previous receipt, and proves every request id you received. `python -m ratchet.receipt verify` checks it offline. It cannot prove the costs written at request time, so reconcile its teacher spend with your Token Factory bill. [Cost and savings](/guides/cost-and-savings/) ### What do I take with me if I leave? The adapter you own: `python -m ratchet.live export --version N --out DIR` writes its files (when they are on that machine), its signed BOM and its `NOTICE.md`, and anyone with your public key can verify them. The ledger's tables export as NDJSON (`GET /ratchet/admin/export/{kind}`). The Nano base model stays under NVIDIA's license, wherever you take the adapter. [Licenses](/guides/licenses/) ### Do I need a GPU running all the time? No. Below break-even, the hosted fallback on Token Factory is cheaper: you pay per token and no GPU runs. Ratchet records the endpoint's uptime and says which option pays at your traffic. [Cost and savings](/guides/cost-and-savings/) ### Is it ready for production? Not yet, and it says so. Ratchet's own adapter is not connected to the live gateway (a random 40% of eligible requests go to the hosted Nano 30B); it runs as one gateway process; PII redaction is pattern-based, so names and street addresses are not caught; precision means agreement with the teacher, not ground truth; and Scout's reader is checked on its labels only, not on the free-text fact it extracts. The README's [limitations](https://github.com/harshbhx/ratchet#limitations) list is the full picture. [Security](/guides/security/) ### Where does my data go? When you host it on Nebius, everything runs in your own project: the gateway and ledger on your VM, adapters in your bucket, teacher calls to Token Factory with your key. The teacher sees the original request; redaction happens before anything is stored as training data. Tavily is called only by Scout. [Host it on Nebius](/guides/host-on-nebius/) ## Related - [Glossary](/guides/glossary/): every term in one line. - [Quickstart](/guides/quickstart/): run it locally against Token Factory. - [For judges](/guides/for-judges/): what to look at and how to check it.