# Ratchet docs: full text
> Ratchet is an OpenAI-compatible gateway that distills open teacher models into a small student. You end up with an adapter you own, learned only from open teachers, with a gated nightly release.
---
# What is Ratchet
Source: https://docs.ratchet.dualmindlab.tech/guides/what-is-ratchet/
Ratchet is a gateway you point your OpenAI client at. A small Nemotron model with your LoRA adapter on top answers the requests it is sure about. A large open model answers the rest, and its answers become training data. Every night a new small model is trained and is released only if it passes a held-out exam, a silent trial and a staged rollout.
Your app is a busy restaurant. Today every dish is cooked by a famous head chef from an agency that charges per plate, even when the order is toast for the hundredth time.
Ratchet puts an apprentice next to the head chef. The apprentice cooks each dish a few times. If the results agree and the apprentice is sure, the plate goes out. If not, the head chef cooks it, and the dish goes in a recipe book with the chef's name and a note that copying it is allowed. At night the apprentice studies the book. The next morning's apprentice cooks only if it passes a sealed exam first.
Every model call on the live site runs on Nebius Token Factory, the adapter was [retrained on Nebius AI Cloud](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius), and the student was served from a Nebius Serverless Endpoint [in a measured test](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius).
## The problem, in numbers you can check
- **The bill grows with traffic.** Enterprise spend on model APIs was $12.5B in 2025 ([Menlo Ventures](https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise/), report of Dec 9, 2025, read Oct 26, 2026). Much of it is narrow and repetitive: labelling, extraction, ticket triage, steps inside agents.
- **The obvious fix is often not allowed.** Training a cheaper model on a closed model's answers breaks the terms of the largest vendors. [OpenAI Business Terms §3.3(e)](https://openai.com/policies/business-terms/) and [Anthropic Commercial Terms §D.4(a)](https://www.anthropic.com/legal/commercial-terms) (both pages read Oct 21, 2026) forbid using outputs to train competing models. A tool that distills that traffic puts its users at risk. Not legal advice.
- **Doing it by hand is slow.** Logging, cleaning, fine-tuning, evaluating, deploying and watching for regressions are separate chores, and most tools cover one.
Every request, repeated or not, is a paid call to the teacher. Distilling by hand is slow, and closed models forbid training on their outputs.
A student with your adapter answers what it is sure about, the teacher answers the rest, and a better student is trained every night from open teachers only.
## The idea
Learn from the traffic you already have, but only from teachers whose license allows it, and .
Change one line, your base URL, and four parts start working:
Sampled 3 times; its confidence decides.
And checks a random 5% of the student's answers.
Each with its teacher and license, after redaction.
Golden set, shadow, canary 10%, 50%, 100%, rollback.
In more detail:
| Part | What it does |
|---|---|
| **The student** | NVIDIA Nemotron 3 Nano 4B with a LoRA adapter you own. It answers first and is sampled three times. Its confidence mixes agreement across samples with token probabilities |
| **The teacher** | NVIDIA Nemotron 3 Ultra on Nebius Token Factory. It answers when the student is unsure, and checks a random share of the student's answers (audits) |
| **The ledger** | Every teacher answer is stored, after personal data is redacted, with the teacher that wrote it and the license that permits training on it. Closed models are refused as teachers in code |
| **Night School and the gate** | Each night a new adapter is trained from teacher answers only and scored on a held-out golden set. It then goes through shadow mode, a canary at 10%, 50% and 100%, and rolls back automatically if audited precision drops |
```mermaid
flowchart TB
app(["Your app"]) ==> gw["Ratchet gateway"]
gw ==> st["Student Nano 4B + your LoRA"]
st ==>|"confident"| out(["Answer"])
st -->|"unsure"| te["Teacher Nemotron 3 Ultra"]
te --> out
te --> led[("Ledger answer, teacher, license")]
led --> ns["Night School trains a new adapter"]
ns --> gate{{"Release gate golden set, shadow, canary"}}
gate -.->|"next student"| st
class st student
class te teacher
class led store
class gate gate
class out ok
```
**Reading this diagram.** Start at the top: every request reaches the student first, and the thick path is the cheap one, where a confident student answers at once. Only unsure requests go to the teacher, and every teacher answer also flows down into the ledger. The dotted line closes the loop: what the ledger collects trains the next student, and the gate decides whether that student replaces the current one. That loop is why the share of rented answers can shrink over time.
The student: Nano plus your adapter
The teacher, rented per token
Stored data
A check that can stop a release
Main path
Once a night
The same flow for one request, step by step:
```mermaid
sequenceDiagram
participant A as Your app
participant G as Gateway
participant Student as Student
participant Teacher as Teacher
participant Ledger as Ledger
A->>G: chat request
G->>Student: sample 3 answers
Student-->>G: answers, probabilities
alt confident
G-->>A: student answer
else unsure
G->>Teacher: same request
Teacher-->>G: teacher answer
G->>Ledger: store with its license
G-->>A: teacher answer
end
opt random audit
G->>Teacher: check the answer
Teacher-->>G: agrees or not
G->>Ledger: record the verdict
end
```
**Reading this diagram.** Time runs downward. The gateway always asks the student first. The `alt` block is the fork: a confident answer returns at once and costs the teacher nothing, while an unsure one is answered by the teacher and stored with its provenance. The `opt` block at the bottom is the audit: on a random share of confident answers the teacher quietly checks the student, so wrong-but-sure answers show up in the numbers instead of staying hidden. For the full detail see [One request's journey](/guides/request-journey/).
Ratchet also reports the bill honestly: savings per token and by GPU-hours actually paid, and a plain warning when a GPU endpoint costs more than it saves. See [Why it matters](/guides/why-it-matters/).
## What is real today
| Part | Status |
|---|---|
| Gateway, cascade, ledger, Night School, release gate, Scout, dashboard | Built, with 1,934 tests that need no network and no GPU |
| Teacher calls | Real Nemotron 3 Ultra on Nebius Token Factory. Real teacher calls built the text-to-SQL data on Oct 22, 2026 |
| Student endpoint | **Served by hand in two short sessions on a Nebius Serverless Endpoint for speed tests (27 and 28 Oct 2026, [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)); the endpoint script is tested offline, ready to run with `make nebius-up`. On 28 Oct 2026 the adapter from Night School's first end-to-end run, by hand, was served from a Nebius Serverless Endpoint too, and the golden gate rejected it (81.8% precision against the 95% target), so it was never made live ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)).** Ratchet's own adapter is not connected to the live site: every live model call runs on Nebius Token Factory, where Ultra teaches and a random 40% of eligible requests are answered by the hosted Nano 30B. If a trained version were live without a GPU endpoint, its calls would go to the hosted, untuned Nano 30B , and pages would say so |
| A text-to-SQL adapter | The published adapters were trained on a borrowed GPU. The round-2 recipe was also retrained once on Nebius AI Cloud (a Serverless Job on one L40S, 27 Oct 2026); it scored 40.7% (70 of 172) on exam 1 and 34.5% (69 of 200) on exam 2, against 39.0% and 33.0% for the original adapter: the result reproduces (the row-by-row differences are within chance). The measured results of the published adapters are on [Results](/guides/results/) |
| A tool-calling adapter | Trained on a borrowed GPU (not retrained on Nebius). Hard exam 136 of 150 (90.7%) against 94 of 150 (62.7%) untrained and 137 of 150 (91.3%) for Ultra; BFCL exam 241 of 300 (80.3%) against 232 (77.3%) untrained and 265 (88.3%) for Ultra. SQL stays the headline result. See [Results](/guides/results/) |
| Model ids and Token Factory list prices in `ratchet/config.py` | Checked on 29 Oct 2026 against the price list Token Factory serves (`scripts/day1_check.py` passed, [evidence/day1](https://github.com/harshbhx/ratchet/tree/main/evidence/day1)); the GPU-hour rate is still unverified |
| Token Factory Terms of Service §5(f) ("competing service") | **Open question.** Asked Nebius support in writing on Oct 19, 2026; no answer received as of 29 Oct 2026. Not legal advice |
## Where to go next
Is the student as good as the teacher? It is held to agreement with the teacher on held-out rows, not to ground truth (see [Escalation and audits](/guides/escalation-and-audits/)). Does it always save money? No, see [Cost and savings](/guides/cost-and-savings/). What do you own? The adapter and its signed bill of materials; the Nano base stays under NVIDIA's license. More in the [FAQ](/guides/faq/).
---
# Why it matters
Source: https://docs.ratchet.dualmindlab.tech/guides/why-it-matters/
Ratchet matters for four kinds of reader: the team with a growing LLM bill, the team that must show where its data went, the team that must show a model is safe to ship, and the builder who wants to own the adapter and its license trail. Each section below says what you get and links to the page that backs it.
```mermaid
flowchart TB
A(["Paying for LLM calls"]) --> A1["Cost and savings"]
B(["Must know where data went"]) --> B1["Ledger and Security"]
C(["Regulated or careful"]) --> C1["Evidence and Release gate"]
D(["Open-source builder"]) --> D1["Licenses and Night School"]
A1 --> P{{"Proof you can run"}}
B1 --> P
C1 --> P
D1 --> P
class P ok
```
**Reading this diagram.** The rounded boxes at the top are the four readers and the boxes under them are the pages that hold the evidence for each. Every path ends at the same place: a page of commands you can run yourself. The rest of this page follows the same four paths. Pages: [Cost and savings](/guides/cost-and-savings/), [Ledger](/guides/ledger/), [Security](/guides/security/), [Training on your data](/guides/training-on-your-data/), [Evidence](/guides/evidence/), [Release gate](/guides/release-gate/), [Licenses](/guides/licenses/), [Night School](/guides/night-school/), [Proof you can run](/guides/proof/).
## If you pay for LLM calls
Your bill is price per token times traffic. Much of that traffic is the same narrow task again and again. Ratchet moves the easy share of it to a small model and keeps the hard share on the teacher. Every request records what it cost and what the teacher alone would have cost, so the saving can be measured, not promised (no live saving has been measured yet).
The catch is that a self-hosted model runs on a GPU that bills by the hour, busy or idle. So Ratchet counts both views:
| View | What it counts | Can it be negative? |
|---|---|---|
| Per token | The price of each call actually made | No |
| By GPU-hours paid | Teacher spend, plus every hour the endpoint and the training jobs ran | **Yes**, and it is never clamped to zero |
The break-even is the number of student answers per endpoint hour at which the GPU pays for itself:
```text
break-even = GPU price per hour / teacher spend avoided per student answer
```
With estimated prices (an L40S at $1.35 a GPU-hour on demand, [Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026, and an assumed $0.001 of teacher spend avoided per ticket-triage answer), that is about 1,350 student answers an hour, roughly 32,000 a day. All figures here are **estimates**, not measurements.
| Traffic | Student answers an hour, against break-even (about 1,350) | What Ratchet's report says |
|---|---|---|
| A small gateway, 1,500 requests a day | About 60: far below | A 24/7 endpoint loses money. Use the hosted fallback or stop the endpoint when idle |
| 10M requests a month | About 14,000: about ten times above | Self-hosting costs less than the teacher, but an untuned hosted Nano may cost less per answer. It then recommends the hosted fallback and says the comparison assumes the hosted model covers as many requests |
| Very high volume | Over about 45,000 (estimate): above the hosted-fallback crossover too | Self-host |
```mermaid
flowchart TB
S["Student answers per endpoint hour"] --> Q1{{"Above break-even?"}}
Q1 -->|"no"| N1["Skip the 24/7 endpoint: hosted fallback or scale to zero"]
Q1 -->|"yes"| Q2{{"Above the hosted- fallback crossover?"}}
Q2 -->|"no"| N2["Teacher spend falls, but the hosted fallback costs less"]
Q2 -->|"yes"| Y["Self-host: the endpoint wins on both counts"]
class Q1,Q2 gate
class N1 danger
class N2 outline
class Y ok
```
**Reading this diagram.** The cost curve has two thresholds, not one. Below break-even the GPU costs more than the teacher spend it saves. Between break-even and the crossover the student does save money against the teacher, but a pay-per-token hosted model would save more. Only above both does owning a GPU win. Ratchet's cost report places your traffic on this chart and writes the answer in words, including when the answer is "this loses money". Full method: [Cost and savings](/guides/cost-and-savings/).
A threshold
Loses money
The hosted fallback wins
Self-hosting wins
## If you must know where your data went
Every example Ratchet keeps carries its history. The ledger stores, for each training example, the teacher that wrote the answer and the license that permits training on it, after personal data has been redacted.
- **Redaction before storage.** Emails, phone numbers, card numbers (checksum-validated), Indian identifiers such as UPI and GSTIN, and similar patterns are replaced before anything is stored. Names and street addresses are **not** caught yet.
- **Opt-outs only reduce storage.** A request can ask not to be stored (`store: false`), and a key, a task or an end user can be set to never train. Nothing in a request can undo an opt-out.
- **End users are never stored as typed.** An end-user identifier is kept only as a keyed hash scoped to the API key.
- **Erasure is final.** An erased example leaves only a hash, so it cannot be stored again. Retention windows purge old rows on a schedule.
- **Export.** Stored data can be exported as NDJSON with its provenance.
```mermaid
flowchart TB
R(["Request"]) --> O{{"Opt-out set?"}}
O -->|"yes"| NS["Served, never stored"]
O -->|"no"| RD["Redact personal data"]
RD --> V{{"Complete, valid teacher answer?"}}
V -->|"no"| NS
V -->|"yes"| ST[("Ledger: text, teacher, license")]
ST --> ER["Erase: only a hash stays"]
ST --> RT["Retention purge by age"]
ST --> EX["Export as NDJSON"]
class O,V gate
class ST store
```
**Reading this diagram.** Read it as the checks a request passes before it is allowed to become training data. Two gates can stop it: an opt-out, and a check that the teacher's answer is complete and valid. Whatever passes is redacted and stored with its provenance. From the ledger, three exits exist for the data owner: erase it, let it age out, or export it. The student's own answers are never stored as training data. Details: [Ledger](/guides/ledger/), [Security](/guides/security/), [Training on your data](/guides/training-on-your-data/).
## If you are regulated or careful
You need to show a model was tested before it served real users, and that you can reproduce the evidence.
| Evidence | What it says | Where |
|---|---|---|
| Signed bill of materials | For each adapter: every training source, its license, the base model's license, a fingerprint of the dataset, signed with Ed25519 and checkable offline | [Evidence](/guides/evidence/) |
| Held-out exam | Thresholds are calibrated on half of the golden set and measured on the other half. A candidate must reach 95% precision and not regress against the live model, re-scored the same night | [Release gate](/guides/release-gate/) |
| Certified precision | An exact one-sided lower bound on the precision of what the version serves, with the assumptions stated | [Evidence](/guides/evidence/) |
| Staged release | Shadow, then canary at 10%, 50% and 100%, with automatic rollback when the 95% upper bound of audited precision falls below 92% | [Release gate](/guides/release-gate/) |
| Savings receipts | One signed receipt per day and API client, chained, that a third party can verify | [Evidence](/guides/evidence/) |
**What this does not show.** Precision means agreement with the teacher, not ground truth. A signed bill of materials is the operator's signed statement: it proves nothing changed after signing, not that the weights were trained on that data. See [Escalation and audits](/guides/escalation-and-audits/).
```mermaid
flowchart TB
C["Candidate adapter"] --> G{{"Golden exam passed?"}}
G -->|"no"| X["Rejected"]
G -->|"yes"| SH["Shadow: runs silently"]
SH --> C10["Canary 10%"]
C10 --> C50["Canary 50%"]
C50 --> L["Live 100%"]
C10 -.->|"audits slip"| RB["Automatic rollback"]
C50 -.->|"audits slip"| RB
class C,SH,C10,C50 student
class G gate
class X,RB danger
class L ok
```
**Reading this diagram.** A new student only moves down when evidence allows it, and each canary stage can send it back. The first gate is the held-out exam, then a silent trial on real traffic, then growing shares of live traffic judged by teacher audits. Because every step leaves a record in the ledger, a reviewer can ask which version served which request and why it was promoted.
The new student, on trial
The held-out exam
Live
Rejected or rolled back
## If you build on open models
- **License-clean teachers.** Ratchet distills only from teachers whose license allows it. Nemotron 3 Ultra is under [OpenMDW-1.1](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16), which places no restrictions on outputs (license page checked Oct 21, 2026). Closed models are refused as teachers in code. Not legal advice.
- **The adapter you own.** What Night School produces is a LoRA adapter plus its signed lineage. The base model, Nemotron 3 Nano 4B, stays under the NVIDIA Nemotron Open Model License.
- **Licenses travel with the adapter.** The bill of materials names the base model's license, and a `NOTICE.md` next to the weights lists the licenses of the base and each teacher.
- **Open code.** Apache-2.0, with a mock mode that runs the whole loop with no keys and no GPU. See [Quickstart](/guides/quickstart/).
## What is and is not shown today
The loop is built and tested offline. Real teacher calls on Nebius Token Factory built the text-to-SQL data on Oct 22, 2026; since Oct 24, 2026 the gateway's own live mode runs at live.ratchet.dualmindlab.tech, where every request is a real Token Factory call. Ratchet's own adapter is not connected to the live gateway: Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nano 30B. The text-to-SQL and tool-calling adapters were trained on a borrowed GPU. Open question: Token Factory Terms of Service §5(f), asked of Nebius support on Oct 19, 2026. See [What is Ratchet](/guides/what-is-ratchet/) and [Results](/guides/results/).
---
# The business case
Source: https://docs.ratchet.dualmindlab.tech/guides/business/
Ratchet is for teams that pay frontier-model prices for narrow, repeated work. It moves that work to a small open model with an adapter they own, learns only from teachers whose license allows it, and ships each new model only if it passes a gate. Savings have not been measured on live traffic yet.
## The problem
- **LLM bills grow with traffic.** Enterprise spend on model APIs was $12.5B in 2025 ([Menlo Ventures](https://menlovc.com/perspective/2025-the-state-of-generative-ai-in-the-enterprise/), report of Dec 9, 2025, read Oct 26, 2026). Much of what teams send is narrow and repetitive: classification, extraction, ticket triage, steps inside agents.
- **The obvious fix is often not allowed.** Training a cheaper model on a closed model's answers breaks the terms of the largest vendors: [OpenAI Business Terms §3.3(e)](https://openai.com/policies/business-terms/) and [Anthropic Commercial Terms §D.4(a)](https://www.anthropic.com/legal/commercial-terms) (both read Oct 21, 2026) forbid using outputs to train competing models. Not legal advice.
- **Doing it by hand is slow.** Logging, cleaning, fine-tuning, evaluating, deploying and watching for regressions are separate chores, and most tools cover one.
## Who it is for
Teams with steady, repetitive LLM traffic on tasks a small model can learn: support triage, field extraction, routing, classification, text-to-SQL, or the many small steps inside an agent. It is a poor fit for open-ended work where every request differs, and for very low volume, where a GPU endpoint costs more than it saves (see [Why it matters](/guides/why-it-matters/) and [Cost and savings](/guides/cost-and-savings/)).
## The value
Every narrow, repeated request is a paid call at frontier prices, the bill grows with traffic, and closed vendors forbid training a cheaper model on their answers.
A small open model with your adapter takes over the work it is sure about, learns only from teachers whose license allows it, and ships each new version only if it passes a gate.
```mermaid
flowchart TB
T(["Steady, narrow LLM traffic"]) ==> R["Ratchet"]
R --> V1["Lower bill as the student takes over"]
R --> V2["An adapter you own: signed lineage, open base"]
R --> V3["License-clean: closed teachers refused"]
R --> V4["Safe releases: gate, canary, rollback"]
V1 --> M[("Measured per request, by token and GPU-hours")]
class V2 student
class V4 gate
class M store
```
**Reading this diagram.** One input, steady narrow traffic, produces four separate benefits, and the first is measured rather than assumed. Each request records what it cost and what the teacher alone would have cost, in two views: per token and by the GPU-hours actually paid. The other three benefits are about risk: who owns the adapter, whether its training data was allowed, and whether a bad version can reach everyone.
- **Cost falls as the student takes over.** Real savings depend on the traffic mix and on GPU utilization. Once the student serves, the teacher is paid for only the requests it escalates; the [Cost and savings](/guides/cost-and-savings/) guide shows the arithmetic.
- **You own the result.** A LoRA adapter on an open base model, plus a signed bill of materials listing the teacher and license of every training example.
- **Releases are gated.** A new student needs at least 95% precision on held-out data and may fall no more than 2 points behind the live version, then passes shadow and canary stages. See [The release gate](/guides/release-gate/).
## How it could make money
The plan is an open-source core (Apache-2.0: gateway, ledger, Night School, release gate, dashboard) that anyone can self-host, plus a managed offering for teams that do not want to run the nightly loop and GPUs themselves. The managed idea is to charge a share of savings that Ratchet can verify, with no fee when there are none. The code already produces signed daily savings receipts for this (see [Evidence](/guides/evidence/)). This is a plan: nothing is sold today.
## Why now
- **A deadline for fine-tuning customers.** OpenAI lists no new self-serve fine-tuning jobs after Jan 6, 2027 ([OpenAI deprecations](https://developers.openai.com/api/docs/deprecations), checked Oct 21, 2026). Teams with fine-tunes there need somewhere else to run them.
- **Small models are being used for narrow tasks.** An NVIDIA Research paper estimates that about 40 to 70% of the LLM queries in the three open agent systems it studied (MetaGPT, Open Operator and Cradle) could go to specialized small models ([arXiv 2506.02153v3](https://arxiv.org/abs/2506.02153v3), revised Sep 22, 2026, read Oct 26, 2026). It is an estimate for those three systems, not a measurement.
## How it differs from other categories
| Category | What it does well | What it does not do | Ratchet |
|---|---|---|---|
| Routers and gateways | Pick a model per request, log and govern traffic | Learn from the traffic; every call stays rented | Learns from the traffic and trains a student |
| Fine-tuning and distillation services | Train a model when you start a job | Run by themselves every night, or record each example's teacher license | Retrains nightly and signs the lineage into the adapter |
| Reference blueprints and tutorials | Show the pattern and its parts | Run unattended; promotion is left to a person | Gates, canaries and rolls back automatically |
```mermaid
flowchart TB
subgraph Prior["Existing categories"]
A["Routing and gateways"]
B["Distillation services"]
C["Blueprints and papers"]
end
A --> X{{"One loop that runs by itself"}}
B --> X
C --> X
X --> O["Per-request escalation, nightly retraining, gated release, signed license lineage"]
class X gate
class O ok
```
**Reading this diagram.** Each category on top is real and useful, and Ratchet uses their ideas. The claim is only about the combination: routing, retraining and release safety joined in one loop, with each example's teacher license recorded. The README's [prior art section](https://github.com/harshbhx/ratchet#prior-art-and-what-is-new) names the work it builds on.
## Honest limits
- **Not proven on live traffic.** Real teacher calls on Nebius Token Factory built the text-to-SQL data (Oct 22, 2026), and the live gateway runs in live mode, but Ratchet's own adapter is not connected to it (one was served in speed tests and deleted; Night School ran once on Nebius and the gate rejected its student). Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nano 30B, so there are no measured savings yet. The measured adapters, two text-to-SQL rounds and one tool-calling round, were trained on a borrowed GPU ([Results](/guides/results/)).
- **Early stage.** There is no customer and no revenue.
- **Savings depend on the traffic mix.** Per-token savings against the cheapest tiers of closed models are small. The large savings come from moving traffic off top-tier models, and only for tasks a small model can learn.
- **A Token Factory terms question is open.** A question about its Terms of Service §5(f) was sent to Nebius support on Oct 19, 2026; no answer received as of 29 Oct 2026. This is not legal advice.
## Related
Also: [What is Ratchet](/guides/what-is-ratchet/).
---
# A text-to-SQL adapter you can check
Source: https://docs.ratchet.dualmindlab.tech/guides/text-to-sql/
A small open model learned to write SQLite queries from a bigger open model's answers, but only from answers that were actually run on a database, and its score comes from whole databases it never trained on, so anyone can recompute that score.
Think of a new analyst joining a company. People ask them questions like "which five customers spent the most last quarter?", and they must write the query that answers it.
A senior analyst (the teacher) writes practice questions and model answers. Before an answer goes into the study pile, someone **runs it** on the practice database. If it crashes, changes data, or returns nothing, it is thrown away.
Then comes the exam. It uses databases the new analyst has never seen, and some of them come from industries that were never in the study pile. The exam is marked by running the new analyst's query and the senior's query on the same database and comparing the rows they return.
Anyone who gets the answer sheet can mark the exam again and must get the same score.
Databases, business questions and SQL answers.
Kept only if it runs and returns 1 to 100 rows.
On the kept rows, on a borrowed GPU.
Scored by running queries, on a CPU.
## The problem
Teams that keep data in a SQL database get questions all day. A large model can answer them, but two things hurt:
**Cost.** Every question is a paid call to a big model. **Exposure.** Every call carries your table and column names to someone else's endpoint.
A small model on a GPU you control fixes both: no bill per question, and the schema stays with you. But only if it is good enough.
The honest question is "good enough at what, measured how?" This page and the [Results](/guides/results/) page answer that in a way you can repeat.
## What was built
- A **teacher**, Nemotron 3 Ultra on Nebius Token Factory, that invents SQLite databases, business questions and SQL answers. See [Licenses](/guides/licenses/) for why this teacher is allowed.
- A **checker** that runs every SQL answer and keeps only the ones that pass.
- A **student**: Nemotron 3 Nano 4B with a small LoRA adapter, trained on the kept rows.
- An **exam** made of databases that are never in training, and a **scorer** that anyone can rerun on a CPU.
- A **confidence gate**: the adapter answers only when it is confident, and the rest go to the teacher.
## How it really works
1. **The teacher invents a database.** For each industry (online retail, retail banking, clinic appointments, freight and logistics and 26 others) the teacher is asked for 3 to 5 SQLite tables with keys and 10 to 15 realistic rows each. The script runs the teacher's `CREATE` and `INSERT` statements in memory and drops the database unless it ends up with 3 to 7 tables of at least 5 rows each.
2. **The teacher writes questions and SQL.** For each database it writes questions a business user might ask, mixed across four levels: `simple` (one table), `join`, `aggregate` and `hard` (subquery, `HAVING`, three tables, `CASE` or a date calculation).
3. **Every SQL is run.** The SQL goes into a sandbox on that database. It is kept only if it is one read-only `SELECT` (or `WITH`) that finishes inside 2 seconds and returns 1 to 100 rows. Duplicate questions on the same database are dropped. Everything else is counted as rejected and thrown away.
4. **The exam is kept apart by database.** Training uses 25 industries. The exam uses whole databases that never appear in training: about two thirds come from industries that are in training (but a different, new database) and 57 of the 172 questions (33%) from two industries that are never in training (airline operations and pharmacy inventory). The script stops if one database ends up on both sides.
5. **The student is trained.** QLoRA on a borrowed GPU: the Nano 4B base loaded in 4-bit, LoRA rank 16, fp16, two epochs, with the loss counted on the SQL only.
6. **The exam is answered, then scored on a CPU.** The base model alone and the base model with the adapter each answer every exam question, and the raw answers are saved. A separate step runs each student query and the teacher's query on the same database and compares the rows.
7. **The confidence gate is set and measured on different rows.** Half of the exam rows (the even ones) set the confidence threshold. The other half (the odd ones) measure it. Nothing is graded on rows that tuned it.
8. **The score is recomputed by `verify`.** It reads the raw answers again and fails if any number in `results.json` differs, or if an input file is not the one the results were computed from.
**Where it was trained.** On a borrowed GPU (both published rounds). On 27 Oct 2026 the round-2 recipe was retrained once on Nebius AI Cloud ([adapter on Hugging Face](https://huggingface.co/harshbhx/ratchet-nemotron-nano-4b-text2sql-nebius)), as a Serverless Job on one preemptible L40S (new adapter SHA-256 `68b9ed06d984a2ec38a32b58090c246a56ae12ba65f3277f66b5d74227ea25d5`; scores 40.7% (70 of 172) on exam 1 and 34.5% (69 of 200) on exam 2, against 39.0% and 33.0% for the original adapter: the result reproduces (the row-by-row differences are within chance)). The teacher ran on Nebius: real calls to Nemotron 3 Ultra on Token Factory built this data on Oct 22, 2026.
```mermaid
flowchart TB
T["Teacher invents a database, questions and SQL"] --> C{{"Run on its database: one read-only SELECT, under 2 seconds, 1 to 100 rows?"}}
C -->|"no"| X["Thrown away and counted"]
C -->|"yes"| S{{"Which database?"}}
S -->|"training: 25 industries"| TR[("Training set")]
S -->|"exam: new databases, 2 industries never in training"| EX[("Exam set")]
TR --> L["Train the adapter QLoRA, borrowed GPU"]
L --> M["Adapter answers every exam question"]
EX --> M
M --> R["Run student and teacher SQL on the same database, compare the rows"]
class T teacher
class C,S gate
class X danger
class TR,EX store
class L,M student
class R ok
```
**Reading this diagram.** Read from the top. The teacher writes everything, but nothing it writes is trusted: the first check runs each query on the database it was written for, and only queries that pass reach the training or exam pile. The split between training and exam happens by database, not by row, so the student cannot have seen the exam's tables. The last box is the only place a number is made, and it is made by running queries, not by asking a model whether they look right. Each database has 3 to 7 tables of at least 5 rows; the questions mix four levels: simple, join, aggregate and hard.
The teacher, Nemotron 3 Ultra
The student and its adapter
A check
A data set
Rejected
### The sandbox
A model's SQL is untrusted code, so it runs in a locked room.
- **Read only.** An SQLite authorizer allows reading, `SELECT`, functions and recursive queries, and denies every other action. An `INSERT`, `UPDATE`, `DELETE`, `DROP` or `ATTACH` is refused by the database itself.
- **Fast.** A progress handler stops any query that runs longer than 2 seconds.
- **In memory.** Each database is built from its script in memory and thrown away.
- **Small answers.** Training rows must return 1 to 100 rows, so a query that returns everything or nothing is not a good example.
The Ratchet gateway has the same idea built in for tasks of format `sql`: [`ratchet/sqlcheck.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/sqlcheck.py) runs two answers on a fixture database and calls them the same when they return the same rows. The data scripts for this run are standalone and enforce the same rules for the same reason.
### How an answer is scored
Two queries are the same answer when they return the same rows on the same database. This is called execution match.
- Column names are ignored, because `total` and `sum_spent` can mean the same thing.
- Decimal numbers are compared after rounding to 6 places.
- Row order matters only when the teacher's query has an `ORDER BY`. Otherwise the rows are compared as a set, sorted.
- A query that does not run, is not a read-only `SELECT`, or takes more than 2 seconds counts as wrong.
```mermaid
flowchart TB
E(["One exam question and its database"]) --> TQ["Teacher SQL"]
E --> SQ["Student SQL"]
TQ --> TR["Run it: rows, and does it order?"]
SQ --> SR["Run it: rows or a failure"]
TR --> CMP{{"Compare"}}
SR --> CMP
CMP -->|"failed or different rows"| W["Wrong"]
CMP -->|"same rows, same order if the teacher orders"| OK["Match"]
class TQ teacher
class SQ student
class CMP gate
class W danger
class OK ok
```
**Reading this diagram.** Both queries are run on the same database, so the only thing that can make them match is giving the same rows. The order rule sits in the comparison: if the teacher's query sorts, the student must sort the same way, and if it does not, order is ignored. A student query that crashes counts as wrong.
### The confidence gate
The adapter's confidence in an answer is the average probability of its tokens. The threshold is the lowest confidence at which the answers on the calibration half reach the target precision (95% unless set otherwise). On the measured half, an answer above the threshold is served by the student, and the rest would go to the teacher. Three numbers come out: **coverage** (how much the student answers), **precision when it answers** (how often those answers match the teacher) and an **estimated cost** against sending everything to the teacher.
If no threshold reaches the target on the calibration half, the report says so. In that case the adapter would answer nothing on its own at that target.
```mermaid
sequenceDiagram
participant U as Question
participant Student as Student
participant G as Gate
participant Teacher as Teacher
U->>Student: schema and question
Student->>G: SQL and confidence
alt at or above the threshold
G-->>U: student SQL
else below the threshold
G->>Teacher: same prompt
Teacher-->>U: teacher SQL
end
Note over Student,Teacher: threshold set on one half, measured on the other
```
**Reading this diagram.** The student always tries first, and its confidence is the average probability of the tokens it wrote. The gate compares that confidence with the threshold. A confident answer goes straight back, and an unsure one is handed to the teacher, which costs more but is the safer answer. The note at the bottom is the honesty rule: the rows that set the threshold are never the rows that grade it.
## What it scored
The numbers below are read from the runs' results files when the site is built. Nothing here is typed by hand.
The headline is the round 2 adapter on two exams: exam 1, written by the teacher, and exam 2, written by a different model on industries that appear in neither the training data nor exam 1. Every question is scored strictly by running the query.
Who answered
Exam 1 ( questions)
Exam 2 ( questions)
Base model (untrained Nano 4B)
of =
of =
Teacher (Nemotron 3 Ultra)
of =
of =
Adapter, round 2
of =
of =
At the 95% precision target the round 2 adapter answers of questions on exam 1's measurement half and of on exam 2's, so it does not replace the teacher yet. The ranges, round 1 and the details are on the [Results](/guides/results/) page.
### Round 1
The round 1 adapter on the half of exam 1 that measures the confidence gate (86 of 172 questions). Round 2 and the second exam are in the table above and on the [Results](/guides/results/) page.
More on how to read them is on the [Results](/guides/results/) page, and the full chain of checks is on [Proof you can run](/guides/proof/).
## Reproduce it and check it
Where the code lives: the data and training scripts are in the repository under [`scripts/text2sql/`](https://github.com/harshbhx/ratchet/tree/main/scripts/text2sql) (`gen_sql_data.py` and the training script), and the scoring script with the SQL checks is `scripts/measure_adapter.py`. They are tools you run by hand, not part of the gateway. The three commands that matter:
```bash
# 1. Build the data (needs a Token Factory key; the teacher is paid per token, capped by --max-usd)
python scripts/text2sql/gen_sql_data.py --out out/sql --max-usd 3
# 2. Train on a GPU and answer the exam
# the training script in scripts/text2sql/ trains the adapter, then runs: scripts/measure_adapter.py generate ...
# 3. Score and verify (CPU only; this is the part anyone can rerun)
python scripts/measure_adapter.py report --raw raw.jsonl --golden golden.jsonl --dbs dbs.json \
--adapter ./adapter --out results.json
python scripts/measure_adapter.py verify --raw raw.jsonl --results results.json \
--golden golden.jsonl --dbs dbs.json
# exact arguments for the published runs: evidence/text2sql-round2/README.md, "Check it yourself"
```
`verify` prints `OK: every number in results.json was recomputed from raw.jsonl and matches`, or exits with a line starting `FAIL:` that says which file is wrong or which numbers differ. The report holds the SHA-256 of the raw answers, the exam, the exam databases and the adapter, so a swapped file is caught.
## What it does not show
- **The data is synthetic.** A model wrote the databases and the questions. Real schemas are messier.
- **The exam is small, so the ranges are wide.** The 95% range shown on each bar is the honest size of the doubt.
- **Only SQLite.** Other dialects are not tested.
- **"Matches the teacher" is not "is what the user meant".** The teacher is a model too and can be wrong. The score measures how well the student copies the teacher, on queries that run.
- **Small slices mean little.** The split by difficulty or by industry has few questions per cell.
- **The cost is an estimate.** Token counts are measured. Prices are inputs. The exams were scored without a student endpoint (the Nebius endpoint of 27 to 28 Oct 2026 ran only for speed tests), so no GPU-hour bill is included. See [Cost and savings](/guides/cost-and-savings/).
- **Trained on a borrowed GPU.** Both rounds were trained on a borrowed GPU with the training script in `scripts/text2sql/`; the teacher ran on Nebius Token Factory. Night School's Nebius Serverless Job is separate: it trains with `ratchet.nightschool.train` through `deploy/train_job.sh`, and ran once by hand on 28 Oct 2026 on Nebius (a spot L40S; the golden gate rejected that student).
## The license chain
| Piece | License or terms | Where it is recorded |
|---|---|---|
| Base model, Nemotron 3 Nano 4B | NVIDIA Nemotron Open Model License | The adapter's model card and its `NOTICE.md` |
| Labels (the SQL answers) | Written by Nemotron 3 Ultra, OpenMDW-1.1 | The data manifest (`teacher_manifest.json`) |
| Other models' outputs | None used: no closed-model output is in the data | Ratchet refuses closed teachers, see [Licenses](/guides/licenses/) |
## For engineers
| What | Where |
|---|---|
| Data generation and sandbox | `gen_sql_data.py`: `build_db`, `run_select`, `make_db`, `HELD_OUT_DOMAINS`, the `teacher_manifest.json` it writes |
| Training | The training script in `scripts/text2sql/` (QLoRA, rank 16, fp16, 4-bit nf4, completion-only loss), run on a borrowed GPU |
| Scoring | `scripts/measure_adapter.py`: `report` runs each answer and the reference query on the same database and compares the rows; `verify` recomputes every number from the raw answers |
| The second exam | `gen_exam2.py` (a different model writes databases, questions and reference queries; every query is run before it is kept), `teacher_selfcheck.py` (the teacher answers an exam) |
| The numbers on the pages | `exam_numbers.py` scores the saved answers of both exams and writes the data the landing page, the judges' page and these docs read; a test fails when they drift |
| In-gateway SQL tasks | [`ratchet/sqlcheck.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/sqlcheck.py): `SqlSpec`, `Sandbox`, `parse_query` |
| Statistics used for the ranges | Wilson 95% interval in `measure_adapter.py`; exact intervals in [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py) |
## Common questions
### Why run the SQL instead of asking a model if it looks right?
A model that judges SQL can be fooled by SQL that looks right and returns the wrong rows. Running it on a database gives a fact. A query either returns those rows or it does not.
### Why are some exam databases from industries the student never saw?
To test whether it learned to write SQL or just learned the shape of its training databases. The results page splits the score by seen and unseen industry for this reason.
### Can a query in the training data change a database or read a file?
No. The sandbox's authorizer refuses anything that is not a read, and a query over 2 seconds is stopped. The databases are built in memory from the teacher's script.
### Why only half the exam measures the confidence gate?
If the same rows set the threshold and graded it, the result would look better than it is. Even rows set the threshold and odd rows measure it.
### Is this the same code the gateway uses?
Not the same file. The gateway's `sql` task format in `ratchet/sqlcheck.py` follows the same rules (read-only, time limit, same rows). The scripts for this run are standalone so that anyone can rerun the scoring without the gateway.
### Where do the numbers come from?
From the runs' files in `evidence/`, copied into the site at build time. The Round 1 panel reads the `results.json` that `measure_adapter.py report` wrote; the headline table reads the exam scores that `scripts/text2sql/exam_numbers.py` computes from the saved answers (its `--check` fails when the copy differs).
## Related
- [Results](/guides/results/)
- [Proof you can run](/guides/proof/)
- [Confidence](/guides/confidence/)
- [Licenses](/guides/licenses/)
- [Night School](/guides/night-school/)
---
# Results
Source: https://docs.ratchet.dualmindlab.tech/guides/results/
This page shows what the text-to-SQL adapters (the headline) and the tool-calling adapter measured on exams they never trained on, read straight from the runs' results files.
A report card is only useful if you know what test it came from. Here the test is a set of questions about databases the student has never opened. The score is "how often does the student's query return the same rows as the teacher's query".
Every score comes with a range, like "somewhere between 64% and 82%". A small exam gives a wide range. That is not a flaw in the report. It is the honest size of the doubt.
## The headline: text-to-SQL, round 2
Two exams, scored strictly (the scores come from `src/data/exams.json`, which `scripts/text2sql/exam_numbers.py --check` recomputes from the saved answers): a query is right only when running it returns the same rows as the exam's stored query on the same database. Exam 1 was written by the teacher; exam 2 by a different model, on industries that appear in neither the training data nor exam 1.
Who answered
Exam 1 ( questions)
Exam 2 ( questions)
Base model (untrained Nano 4B)
of =
of =
Teacher (Nemotron 3 Ultra)
of =
of =
Adapter, round 2
of =
of =
- **The training shows on both exams.** The round 2 adapter scores times the base model on exam 1 and times on exam 2.
- **It does not replace the teacher yet.** At the 95% precision target it answers of questions on exam 1's measurement half and of on exam 2's.
- **The exams do not show it is better than the teacher.** Each exam has one stored query per question, so a correct query with other columns counts as a miss, and the teacher's own score shows it. The sections below give the ranges, round 1 and the details.
The tool-calling adapter's results (the second result) are in [Tool calling, round 3](#tool-calling-round-3) below.
## Round 1, on the measurement half
Status:
Round 1, the half of the exam that measures the confidence gate (86 of 172 questions). Round 2 and the second exam are below.
## What the first real run showed
Same exam, same scoring, all questions. A query is right when running it returns the same rows as the teacher's stored query on the same database.
Who answered
Original prompt
With one extra line (prompt v2)
Prompt v2 adds "Select only the columns the question asks for." to the end of every question, for both models. The ranges are 95% Wilson ranges.
- **The adapter is 2.6 to 3.1 times the base model's score** under both prompts, and the ranges do not overlap.
- **The adapter is in the same range as the teacher, not above it.** The teacher itself returns the stored query's rows only of the time, because each question has one stored query, written together with the question. A correct query with other columns or another sort order counts as a miss. The adapter learned the stored queries' style, so we do not claim it is better than the teacher.
- **Prompt v2 changed nothing that holds up.** Row by row the adapter gained a few questions and lost a few. Treat the two prompts as one result: about a third of the questions for the adapter, about an eighth for the base model.
- **The adapter does not yet replace the teacher.** At the 95% precision target only of test questions pass the confidence gate and of those are right, so the cost saved at that coverage is %. Token probability does not tell the adapter's right answers from its wrong ones.
- **No exam question leaked into training:** of exam prompts, questions, stored queries or table-name sets appear in the training data.
Training: teacher-labelled rows, steps, minutes on a ; teacher labels cost $. The files behind every number, and the command that recomputes them, are in `evidence/text2sql-round1` in the repository.
## What the second run showed
Round 2 trained a second adapter on rows (the first run's 821 plus new teacher-labelled rows) and answered the same -question exam, scored the same way.
Who answered
Right
95% range
Base model
of =
Teacher answering the exam prompt
of =
Adapter, round 1 (821 rows)
of =
Adapter, round 2 ( rows)
of =
- **Better than round 1, but the exam cannot call it settled.** Row by row, questions went from wrong to right and from right to wrong. The two ranges overlap, and a test on those changes gives p = , so some of the gain could be chance.
- **The gain is on familiar industries.** On industries seen in training the adapter went from to of . On the two industries it never saw it stayed at of .
- **By difficulty:** simple of , join of , aggregate of , hard of .
- **The point estimate is above the teacher's, but we do not claim it is better than the teacher.** The exam has one stored query per question, written together with the question, and the two 95% ranges still overlap a little (adapter 32.0 to 46.4%, teacher 19.6 to 32.6%).
- **The confidence gate did not improve.** At the 95% precision target of test questions pass, so the cost saved at that coverage is %. The adapter still does not replace the teacher.
- **No exam question leaked into training:** of for prompts, questions, stored queries and table-name sets. Six generated rows that shared a question or a query with the exam were removed before training.
Training: steps over epochs in hours on a , loss to ; the extra teacher labels cost $. The files and the recompute command are in `evidence/text2sql-round2` in the repository.
## A second exam, written by a different model
The first exam was written by the same teacher that wrote the training labels, which flatters the teacher and anything trained to copy it. So we built a second exam: questions on 10 industries that appear neither in training nor in the first exam, with the databases and the questions written by Claude Sonnet 5.5, used for evaluation only. Every reference query was run and kept only if it worked, nothing overlaps with the training data, and its fingerprint was recorded before any model answered. The same four models answered it, scored the same strict way.
Who answered
Exam 1 (172, written by Ultra)
Exam 2 (, written by Claude Sonnet 5.5 for evaluation only)
Base model
=
= ()
Teacher answering the exam prompt
=
= ()
Adapter, round 1 (821 rows)
=
= ()
Adapter, round 2 ( rows)
=
= ()
- **The training effect shows up on an exam the adapter's data never touched.** The base model scores about the same on both ( and ), and the round 2 adapter is about times the base model on the new exam.
- **Round 2 beat round 1 on both exams,** by 10 questions each time. On exam 2, questions went from wrong to right and from right to wrong (p = ). The direction repeats, but neither exam alone settles it.
- **Both adapters score 5 to 6 points lower than on exam 1,** as expected on an exam from a different author.
- **By difficulty on exam 2** (base, round 1, round 2): simple , , of ; join , , of ; aggregate , , of ; hard , , of . Join questions improved most, again.
- **The teacher scores low here for a reason of shape, not skill.** Of its answers, return the right number of rows with a different number of columns, which the strict exam marks wrong. We do not claim the adapter is better than the teacher.
- **The confidence gate still does not pass at the 95% precision target:** of questions on the measurement half, for either adapter. The adapter does not yet replace the teacher.
### Who wrote what
Three sets of questions are involved, and they are kept apart. The training examples teach the adapter. Exam 1 and exam 2 only test it. Exam 1 was written by the teacher, the same model that wrote the training examples, which flatters the teacher and anything trained to copy it. Exam 2 was written afterwards by a different model from a different lab, for testing only.
### The scores as charts
Each bar is how many of that exam's questions the model got exactly right. The line on a bar is its 95% range: with 172 to 200 questions a score is good to about 7 points either way.
**Exam 1, written by the teacher**
**Exam 2, written by a different model**
### Where it does well and badly
Join questions improved most from round 1 to round 2. Simple questions did not improve, and the aggregate questions were already the adapter's strongest.
**Exam 1**
**Exam 2**
### Round 1 against round 2, question by question
The exam, its fingerprint, every answer and the command that recomputes the scores are in `evidence/text2sql-exam2` in the repository.
## Retrained on Nebius AI Cloud
On 27 Oct 2026 the round-2 recipe (the same 4,322 examples and settings) was retrained on Nebius AI Cloud, by hand, as a Serverless Job on one preemptible L40S. The retrained adapter scored **40.7%** (70 of 172) on exam 1 and **34.5%** (69 of 200) on exam 2, against 39.0% and 33.0% for the original adapter that was trained on a borrowed GPU. Row by row the two are statistically indistinguishable (6 up and 3 down on exam 1, 7 up and 4 down on exam 2), so the result reproduces; it is not an improvement. The whole Nebius effort, failed and interrupted attempts included, used 2.73 GPU-hours, about $2.16 at list prices (the Nebius bill decides). Evidence: [`evidence/text2sql-round2-nebius`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius). The adapter is on Hugging Face: [ratchet-nemotron-nano-4b-text2sql-nebius](https://huggingface.co/harshbhx/ratchet-nemotron-nano-4b-text2sql-nebius).
## Tool calling, round 3
On a small screen the chart scrolls sideways; the table below has the same numbers.
A tool-calling adapter on Nemotron 3 Nano 4B (2,940 training rows, about $2.33 of GPU time), scored strictly on two exams. These are measured results on real models (Oct 24, 2026).
| | Hard exam (150) | BFCL exam (300) |
|---|---|---|
| Untrained Nano 4B | 94 of 150 (62.7%, 95% range 54.7 to 70.0%) | 232 of 300 (77.3%, 95% range 72.3 to 81.7%) |
| Teacher (Nemotron 3 Ultra) | 137 of 150 (91.3%, 95% range 85.7 to 94.9%) | 265 of 300 (88.3%, 95% range 84.2 to 91.5%) |
| Tool-calling adapter | 136 of 150 (90.7%, 95% range 84.9 to 94.4%) | 241 of 300 (80.3%, 95% range 75.5 to 84.4%) |
On the hard exam the adapter is above the untrained model: 90.7% against 62.7%, and the 95% ranges do not overlap. It is about the same as the teacher: 90.7% against 91.3% (lower by point estimate, but the 95% ranges overlap, so the exam cannot tell them apart).
On the BFCL exam the adapter is about the same as the untrained model: 80.3% against 77.3% (higher by point estimate, but the 95% ranges overlap, so the exam cannot tell them apart). It is lower than the teacher: 80.3% against 88.3% (241/300 against 265/300; the 95% ranges overlap only slightly, so part of the gap could be chance).
Hard exam: question by question against the untrained model, the adapter fixed 42 and broke 0 (exact sign test p < 0.0001). BFCL exam: question by question against the untrained model, the adapter fixed 15 and broke 6 (exact sign test p = 0.0784).
Strict scoring: an answer counts only if it is one JSON object with the right tool and arguments; a reply with text around the JSON is wrong. Free-text arguments are a known weak spot in the training data: the strict scorer cannot agree on free text, so tools such as create_ticket (15 training rows), search_files, send_payment and search_inbox are thin. The hard exam uses 20 tools that were never in training. The adapter was trained on a borrowed GPU, on synthetic data. The hard exam (150 questions, 20 new tools) was written by Claude Sonnet 5.5 for evaluation only and is never training data; the second exam (300 questions) is drawn from the Berkeley Function Calling Leaderboard (Apache-2.0).
Every number comes from `evidence/toolcall/results.json`, which `python scripts/toolcall_results.py --check` recomputes from the saved answers.
## How it really works
1. **A run ends with a `results.json`.** `measure_adapter.py report` reads the raw answers and writes every number into that file, with the SHA-256 of each input file.
2. **The file is copied into the docs site.** The site reads `src/data/sql-results.json` when it is built, so no number on this page is typed by hand. If the file says the run is pending, the panel above says so.
3. **`verify` recomputes it.** Anyone with the raw answers, the exam and the exam databases can run `measure_adapter.py verify` and get `OK` or a `FAIL:` line. See [Proof you can run](/guides/proof/).
```mermaid
flowchart TB
RAW[("raw.jsonl teacher, base and adapter answers")] ==> REP["measure_adapter.py report on a CPU"]
GOLD[("golden.jsonl, dbs.json the exam and its databases")] ==> REP
REP ==> RES[("results.json numbers and the inputs' SHA-256")]
RES ==>|"copied in"| PAGE["This page, at build time"]
RAW -.-> VER{{"measure_adapter.py verify: every number recomputed and equal?"}}
GOLD -.-> VER
RES -.-> VER
VER -->|"yes"| OK["OK"]
VER -->|"no"| BAD["FAIL: names the file or the numbers"]
class RAW,GOLD,RES store
class VER gate
class OK ok
class BAD danger
```
**Reading this diagram.** The thick arrows make the numbers and the dotted ones check them, and both start from the same three files. The page is fed from the results file by a build step (it is copied to `docs-site/src/data/sql-results.json`), not by hand. The check does not trust the results file at all: it recomputes everything from the raw answers and compares.
A file
The check
Makes the numbers
Checks them
## How to read the numbers
- **Match with the teacher** is execution match: the student's query and the teacher's query return the same rows on the same database. It is not "the answer is right".
- **The 95% range** is a Wilson interval. If the exam were drawn again many times, about 19 of 20 such ranges would contain the true rate. With a small exam it is wide, and two bars whose ranges overlap should not be called different.
- **Base model against adapter** is the point of the work. Both answer the same questions with the same prompt and the same limits. The gap is what the adapter added.
- **The split by difficulty and by industry** shows where the adapter helps. Each cell has few questions, so read the ranges before the bars. "Industry never seen in training" is the fairest test of whether it learned SQL or memorised.
- **Coverage and precision when it answers** describe the confidence gate. Coverage is the share of questions the student answers on its own. Precision is how often those answers match the teacher. A higher threshold raises precision and lowers coverage.
- **The estimated cost** is an estimate: token counts are measured, the prices are inputs (Nemotron 3 Ultra at $1 and $3 per million tokens in and out, and a student price you set), and the exams were scored without a student endpoint (the Nebius endpoint of 27 to 28 Oct 2026 ran only for speed tests, on other prompts), so no GPU-hour bill is in it. See [Cost and savings](/guides/cost-and-savings/).
- **The data is synthetic.** The teacher wrote the databases and the questions. These numbers say how well the adapter copies the teacher on queries that run, not how well it serves your real users.
It cannot tell you how the adapter does on your schema, in another SQL dialect, or on questions a person would phrase badly. Run your own held-out questions through `measure_adapter.py` to find out.
## For engineers
| What | Where |
|---|---|
| The panel | `docs-site/src/components/SqlResults.astro`, reads `docs-site/src/data/sql-results.json` |
| Layout test data | `docs-site/src/data/sql-results.sample.json` (made up, never shown unless copied over, and then bannered `SAMPLE DATA`) |
| Feed a real run | `cp run/out/results.json docs-site/src/data/sql-results.json`, then rebuild |
| The numbers' code | `measure_adapter.py`: `agreement`, `wilson`, `calibrate`, `build_report` |
## Common questions
### Why not just show one accuracy number?
One number hides the sample size and the slices. The range shows how much to trust it, and the slice by unseen industry shows whether it generalises.
### Why is the exam only partly used for scoring?
Only the confidence gate uses half the exam: the even rows set the threshold and the odd rows grade it, so coverage and precision when it answers (the Round 1 panel's gate figures, and the gate numbers for round 2 and exam 2) come from the graded half. The Round 1 panel's base-against-adapter bars are scored on that graded half too (86 of 172 questions). Every table on this page, the headline included, scores all the questions of each exam (172 and 200), and the base model, the teacher and each adapter answer the same questions.
### Can I check these numbers without a GPU?
Yes. Scoring and `verify` run on a CPU from the raw answers. Only generating the raw answers needs a GPU.
## Related
- [A text-to-SQL adapter you can check](/guides/text-to-sql/)
- [Proof you can run](/guides/proof/)
- [Cost and savings](/guides/cost-and-savings/)
- [Evidence you can check](/guides/evidence/)
---
# Quickstart
Source: https://docs.ratchet.dualmindlab.tech/guides/quickstart/
Install the gateway, give it a Token Factory key, point any OpenAI SDK at it and read the `ratchet` block in each answer. Every call here is a real model call and costs cents. Prefer not to install anything? Use the live gateway at [live.ratchet.dualmindlab.tech](https://live.ratchet.dualmindlab.tech).
## The one-minute path
You need Python 3.10 or newer, `git` and a Nebius Token Factory key.
```bash
git clone https://github.com/harshbhx/ratchet.git && cd ratchet
pip install -e ".[dev]"
export NEBIUS_API_KEY=... # your Token Factory key
export RATCHET_API_KEYS=me:your-local-api-key-here # live mode needs a key for /v1/*; pick your own
mkdir -p data
RATCHET_MODE=live RATCHET_DB=./data/ratchet.db uvicorn ratchet.gateway:app --port 8000
```
The gateway serves on `http://localhost:8000`. With no student endpoint, the teacher (Nemotron 3 Ultra on Token Factory) answers every request. In a second terminal, send it a request with the OpenAI SDK:
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="your-local-api-key-here")
resp = client.chat.completions.create(
model="ratchet",
messages=[{"role": "user", "content": "I was charged twice this month."}],
extra_headers={"X-Ratchet-Task": "ticket-triage"},
)
print(resp.choices[0].message.content)
print(resp.model_extra["ratchet"]["served_by"]) # teacher, until a trained student is serving
```
Then open `http://localhost:8000/` (the story), `/scout` (the research agents, which also need `TAVILY_API_KEY`) and `/dashboard` (releases, versions, costs).
## Install
```bash
git clone https://github.com/harshbhx/ratchet.git && cd ratchet
python -m venv .venv && source .venv/bin/activate
make install
```
```powershell
git clone https://github.com/harshbhx/ratchet.git; cd ratchet
python -m venv .venv
.venv\Scripts\Activate.ps1
pip install -e ".[dev]" ruff
```
Windows has no `make`, so run the command `make install` stands for (the last line). In PowerShell, set variables with `$env:NEBIUS_API_KEY="..."; $env:RATCHET_API_KEYS="me:your-local-api-key-here"; $env:RATCHET_MODE="live"; $env:RATCHET_DB="./data/ratchet.db"` (after `mkdir data`) before `uvicorn`.
`make install` runs `pip install -e ".[dev]" ruff`. The `dev` extra brings pytest, the OpenAI Python SDK (used by the example above), `cryptography` for signed BOMs, PyYAML and Jinja2. The gateway itself needs only FastAPI, uvicorn, httpx, pydantic and anyio.
## Send your own request
The gateway speaks the OpenAI chat-completions format, so any OpenAI SDK works. Name the task with the `X-Ratchet-Task` header (or `metadata.ratchet_task`); without one, the request belongs to the task `default`. These are the examples from the [API reference](/reference/api/#examples).
```bash
curl -s http://localhost:8000/v1/chat/completions \
-H 'content-type: application/json' \
-H 'authorization: Bearer your-local-api-key-here' \
-H 'X-Ratchet-Task: ticket-triage' \
-d '{"model":"ratchet","messages":[
{"role":"system","content":"You triage customer support tickets. Reply with JSON only ..."},
{"role":"user","content":"I was charged twice for my Pro plan this month. Order #48213."}]}'
```
```js
const client = new OpenAI({ baseURL: "http://localhost:8000/v1", apiKey: "your-local-api-key-here" });
const r = await client.chat.completions.create(
{ model: "ratchet", messages: [{ role: "user", content: "Where is my order #48213?" }] },
{ headers: { "X-Ratchet-Task": "ticket-triage" } },
);
```
Install the SDK first with `npm install openai`.
Every answer is a standard `chat.completion` plus a `ratchet` block. The fields you will read most:
| Field | Meaning |
|---|---|
| `served_by` | `student` or `teacher` |
| `student_version` | The version that answered, or `null` when the teacher did |
| `confidence` | The student's confidence, if it was tried |
| `escalated` | `true` when the student was tried and was below its threshold |
| `cost_usd`, `all_teacher_cost_usd` | What this request cost, and what the teacher alone would have cost |
| `stored` | Whether a teacher answer of this request became, or matched, a training example |
| `request_id`, `row_id` | The request's id and its ledger row |
## Try Scout from the command line
```bash
export TAVILY_API_KEY=...
python -m ratchet.scout "What is LoRA fine-tuning?" --base-url http://localhost:8000/v1 --api-key your-local-api-key-here
```
Scout runs as an outside client against the gateway: a planner and a writer pinned to the teacher, a searcher, and many small triage and reader calls. It also takes `--api-key` and `--max-results` (default 4). See [Scout](/guides/scout/).
## Common questions
### Do I need an API key, a GPU or Node?
A Token Factory key, yes: every answer is a real model call. No GPU: the gateway does no inference. Node is needed only to rebuild the web app (`make web`); the build is committed.
### Where is my data?
In the SQLite ledger at `RATCHET_DB`. Stopping the gateway keeps it.
### Port 8000 is taken. What now?
Use another port, for example `--port 8001`, in the `uvicorn` line and in the URLs and examples.
### How do I get a student?
A student appears only after Night School trains and releases one on your ledger, which needs a GPU endpoint and a training job: see [Host it on Nebius](/guides/host-on-nebius/). Those scripts are tested offline with mock models; one retraining job, one endpoint speed test and one Night School (rejected by the golden gate, never made live) have been run by hand on Nebius. The measured results are on [Results](/guides/results/) and [Evidence](/guides/evidence/).
## Next steps
- Send a verdict back on an answer, and see every route: the [endpoint map](/api/endpoints/).
- Run the loop on your own tasks: [Training on your data](/guides/training-on-your-data/) and the task registry (see the [glossary](/guides/glossary/)).
- Read the other tools you have now: [Command line](/api/cli/) and [environment variables](/api/environment/).
- Embed the cascade in a Python program: [Python API](/api/python/).
- Move to a GPU student: [Host it on Nebius](/guides/host-on-nebius/).
## Related
---
# For judges
Source: https://docs.ratchet.dualmindlab.tech/guides/for-judges/
You have five minutes. Send one real request on the live site, read the exam results, run one command against an evidence folder, and watch the video. This page puts the proof, every link and the honest limits in one place.
**Built on NVIDIA Nemotron and Nebius: every model call on the live site runs on Nebius Token Factory, the adapter was [retrained on Nebius AI Cloud](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius), and the student was served from a Nebius Serverless Endpoint [in a measured test](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius).** The live site is not connected to that endpoint.
Ratchet is an OpenAI-compatible gateway for license-clean distillation on open models. A small NVIDIA Nemotron 3 Nano 4B student answers first. When it is unsure, the request goes to the teacher, Nemotron 3 Ultra on Nebius Token Factory, and the answer is logged with the teacher's name and license; closed models are refused as teachers, in code. Night School is built to train a new student from those answers and release it only if it passes a statistical gate. Measured: a Nano adapter taught by Ultra goes from 12.8% to 39.0% on a text-to-SQL exam it never saw, for about $17 on record, and every score recomputes from the saved answers.
## Check it in 5 minutes
1. **Use it.** Open [ratchet.dualmindlab.tech/live](https://ratchet.dualmindlab.tech/live) and send a real request. A badge says who answered. Send a few at most; your text is not stored.
2. **Read the exam results.** The proof tabs below, or [Results](/guides/results/).
3. **Check one result.** Clone the repository and run one command against an evidence folder:
```bash
git clone https://github.com/harshbhx/ratchet.git && cd ratchet && pip install -e ".[dev]"
python scripts/text2sql/exam_numbers.py --check
```
4. **Watch the video** (2:59): [video.ratchet.dualmindlab.tech](https://video.ratchet.dualmindlab.tech/?from=docs).
For the console version of this page, with live chapters read from the gateway, open the [judges page](https://ratchet.dualmindlab.tech/judges). A one-page copy for GitHub is [JUDGES.md](https://github.com/harshbhx/ratchet/blob/main/JUDGES.md).
## The proof, with links
Every number is read from saved files in the repository. Commands are from [Check it yourself](https://github.com/harshbhx/ratchet/blob/main/docs/VERIFY.md) and need no GPU and no key.
| What | Result | Evidence | Check it |
|---|---|---|---|
| Text-to-SQL exams (22 to 24 Oct) | Adapter **39.0%** / **33.0%**; untrained 12.8% / 12.5%; teacher (Ultra) 25.6% / 15.0% | [round 1](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round1), [round 2](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2), [exam 2](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-exam2) | `python scripts/text2sql/exam_numbers.py --check` |
| Tool calling (24 Oct) | **90.7%** hard exam (Ultra 91.3%), **80.3%** BFCL | [evidence/toolcall](https://github.com/harshbhx/ratchet/tree/main/evidence/toolcall) | `python scripts/toolcall_results.py --check` |
| Scout runs (23 to 24 Oct) | Three recorded live runs, about $0.02 each | [evidence/scout](https://github.com/harshbhx/ratchet/tree/main/evidence/scout), [replay](https://huggingface.co/spaces/harshbhx/ratchet-scout) | replay in the browser |
| What | Result | Evidence | Check it |
|---|---|---|---|
| Retrain on Nebius AI Cloud (27 Oct) | **40.7% / 34.5%**: the original result reproduces; about **$2.16** | [evidence/text2sql-round2-nebius](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius) | `python evidence/text2sql-round2-nebius/score_nebius.py verify` |
| Endpoint speed tests (27 to 28 Oct) | Cold start about **7.2 to 7.6 min**, restart **7.1 min**; 16 users: p50 **1.6 s** / **1.2 s** | [evidence/student-endpoint-nebius](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius) | the folder's `results/` |
| Night School on Nebius (28 Oct) | Gate **rejected**: 81.8% against 95% | [evidence/nightschool-nebius](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius) | the folder's README and logs |
| What | Result | Evidence | Check it |
|---|---|---|---|
| Signed BOM (signed 29 Oct, after the run) | Ed25519 key `81b0a086250a81d0` | [evidence/keys](https://github.com/harshbhx/ratchet/tree/main/evidence/keys), [signed BOM](https://github.com/harshbhx/ratchet/blob/main/evidence/nightschool-nebius/ratchet_bom.signed.json) | `python -m ratchet.bom verify evidence/nightschool-nebius/ratchet_bom.signed.json --public-key Gh68hnDDlRSjBRxmc5dTo4l+rvqkftUsH7B2Nm72gpc= --notice evidence/nightschool-nebius/NOTICE.md` |
| Prices verified (29 Oct) | Token Factory list prices match the config | [evidence/day1](https://github.com/harshbhx/ratchet/tree/main/evidence/day1) | read `day1.json` |
| Live smoke run (23 Oct) | 20 real requests, about $0.01 | [evidence/live-smoke-2026-10-23](https://github.com/harshbhx/ratchet/tree/main/evidence/live-smoke-2026-10-23) | `python -m ratchet.liverun verify evidence/live-smoke-2026-10-23` |
## All the links
### Live and video
### Docs
### Code
### Evidence folders
[`text2sql-round1`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round1) · [`text2sql-round2`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2) · [`text2sql-exam2`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-exam2) · [`toolcall`](https://github.com/harshbhx/ratchet/tree/main/evidence/toolcall) · [`text2sql-round2-nebius`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius) · [`student-endpoint-nebius`](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius) · [`nightschool-nebius`](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius) · [`keys`](https://github.com/harshbhx/ratchet/tree/main/evidence/keys) · [`day1`](https://github.com/harshbhx/ratchet/tree/main/evidence/day1) · [`live-smoke-2026-10-23`](https://github.com/harshbhx/ratchet/tree/main/evidence/live-smoke-2026-10-23) · [`scout`](https://github.com/harshbhx/ratchet/tree/main/evidence/scout) · [`hosted-nano`](https://github.com/harshbhx/ratchet/tree/main/evidence/hosted-nano)
### Hugging Face
### Nebius and NVIDIA: what runs where
| Where | What runs there |
|---|---|
| Nebius Token Factory | Every model call on the live site: the teacher (Nemotron 3 Ultra), the hosted Nano for the random split, the judge |
| Nebius AI Cloud | The round-2 retrain (27 Oct) and the Night School training job (28 Oct), as Serverless Jobs on a spot L40S; the Serverless Endpoint for the speed tests and the Night School adapter; a CPU VM for the Night School gateway; an Object Storage bucket. By hand; the endpoint and VM were deleted afterwards |
| NVIDIA Nemotron | Ultra (teacher), Nano 4B (student, with a LoRA adapter), hosted Nano 30B (random split) |
| Cloudflare | The story site, the docs and the live gateway, CPU only |
| A borrowed GPU | Training of the two original adapters |
## Hackathon checklist
| | Requirement | Link |
|---|---|---|
| ✓ | Nebius Token Factory and AI Cloud, with NVIDIA open models | [Where everything runs](/reference/where-it-runs/) |
| ✓ | Video, 2:59 | [video](https://video.ratchet.dualmindlab.tech/?from=docs) |
| ✓ | Public repository with license and README | [repository](https://github.com/harshbhx/ratchet), [Apache-2.0](https://github.com/harshbhx/ratchet/blob/main/LICENSE), [README](https://github.com/harshbhx/ratchet/blob/main/README.md) |
| ✓ | Working URL | [ratchet.dualmindlab.tech](https://ratchet.dualmindlab.tech), [/live](https://ratchet.dualmindlab.tech/live) |
| ✓ | Feedback on Nebius and NVIDIA | [README, Feedback](https://github.com/harshbhx/ratchet#feedback-on-nebius-and-nvidia) |
The track is the one chosen on the Devpost page.
## What is real and what is not yet
```mermaid
%%{init: {"flowchart": {"rankSpacing": 24}}}%%
flowchart TB
subgraph Real["Real"]
direction LR
R1["Token Factory calls, Scout search"] ~~~ R3["Adapters, trained on a borrowed GPU"] ~~~ R4["Code and tests, no GPU needed"] ~~~ R5["Nebius retrain, endpoint speed tests, Night School: each ran once by hand"]
end
subgraph Not["Not done"]
direction LR
N1["Nightly loop on live traffic"] ~~~ N3["Live site connected to a student endpoint"] ~~~ N2["Measured live savings"]
end
Real ~~~ Not
class R1,R3,R4,R5 ok
class N1,N2,N3 pending
class Real okZone
class Not pendingZone
```
**Reading this diagram.** Two rows: what runs for real (green) and what has not been done (amber, dashed). The teacher side runs for real, and one hand-run Nebius training job, one hand-run speed test of the student and one hand-run Night School (a CPU VM, a spot L40S job and an endpoint, rejected by the golden gate) have run (green); the nightly loop on live traffic, the live site's connection to a student endpoint and a measured saving have not (amber); the one-command VM kit is scripted and tested offline. The two notes below say the same in words.
Real
Not done
**Real, as of 29 Oct 2026.** Built on NVIDIA Nemotron and Nebius: every model call on the live site runs on Nebius Token Factory, the adapter was [retrained on Nebius AI Cloud](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius), and the student was served from a Nebius Serverless Endpoint [in a measured test](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius). Real calls to the teacher, Nemotron 3 Ultra on Nebius Token Factory, built the text-to-SQL data on Oct 22, 2026. The original published adapters were trained on a borrowed GPU, and their results are read from their own results files. On 27 Oct 2026 the round-2 text-to-SQL recipe (the same 4,322 examples and settings: 4-bit QLoRA, rank 16, 2 epochs, the same library versions) was retrained on Nebius AI Cloud, by hand, as a Nebius Serverless Job on one preemptible (spot) L40S, with checkpoints in a Nebius Object Storage bucket so it resumed after spot interruptions and with the Mamba kernels built for the L40S. Training loss went from 0.346 to 0.088 and the new adapter's SHA-256 is `68b9ed06d984a2ec38a32b58090c246a56ae12ba65f3277f66b5d74227ea25d5`. It scored 40.7% (70 of 172) on exam 1 and 34.5% (69 of 200) on exam 2, against 39.0% and 33.0% for the original adapter: the result reproduces (the row-by-row differences are within chance); training took 28.6 minutes of GPU time (58.7 minutes wall clock across two spot interruptions); the whole Nebius effort, failed and interrupted attempts included, used 2.73 GPU-hours, about $2.16 at list prices (the Nebius bill decides) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)). On 27 to 28 Oct the student was served by hand, in two short sessions, on a Nebius Serverless Endpoint (one spot L40S) for speed tests: cold start about 7.2 minutes to ready; with 16 users at once the median answer took 1.6 s (SQL adapter) and 1.2 s (tool-calling adapter); about $0.67 of GPU time across both sessions; speed only, no accuracy scored; the endpoint was deleted afterwards and the live gateway is not connected to it ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). On 28 Oct 2026 Night School ran end to end once on Nebius, by hand: the gateway on a Nebius CPU VM, the training job on a spot L40S, the adapter served from a Nebius Serverless Endpoint. The golden gate rejected it (81.8% precision against the 95% target), so it was never made live. The live site's own gateway is still on Cloudflare and is not connected to that endpoint, and the loop has not run nightly on live traffic ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). The code and its roughly 1,900 tests run with no GPU. Since Oct 24, 2026 the live gateway runs in live mode, and every request there is a real Token Factory call.
**Not yet.** Ratchet's own adapter is not connected to the live site, and no trained version exists on that ledger yet. On the live site a random 40% of eligible requests are answered by the hosted, untuned Nemotron 3 Nano 30B on Token Factory, a quarter of those re-checked by Ultra, and the pages label them as a random split. With a trained version and no GPU endpoint, student calls would go to that hosted model and the pages would say so with a badge. Night School has not run nightly on live traffic: it ran once, by hand, on Nebius, and the golden gate rejected that student. Live savings numbers do not exist yet and are not shown.
### Measured on Nebius: Night School, 28 Oct 2026
**Measured on Nebius, 28 Oct 2026, by hand, once.** Night School ran end to end once on Nebius, by hand (28 Oct 2026): the gateway on a Nebius CPU VM, the training job on a spot L40S, the adapter served from a Nebius Serverless Endpoint. The golden gate rejected it (81.8% precision against the 95% target), so it was never made live. The live site's own gateway is still on Cloudflare and is not connected to that endpoint, and the loop has not run nightly on live traffic. No saving is claimed.
| What | Result |
|---|---|
| Gateway | A Nebius CPU VM in `eu-west1` (2 vCPU), in live mode |
| Data | 4,322 existing Ultra-labelled text-to-SQL rows imported (no new teacher calls, no exam rows): 3,864 to train on, 458 held out as the golden set |
| Training | A spot L40S Serverless Job: 2 epochs, 966 steps, 1,478 s; loss about 0.15 to 0.08; token accuracy about 0.975 |
| Serving | The adapter (SHA-256 `dc9ad4b9...`) hot-loaded into a Nebius Serverless Endpoint |
| Golden gate | SQL compared as text: agreement 17.9%; at the gate's threshold the student would answer 4.8% of questions with 81.8% precision (9 of 11) against the 95% target. **Rejected**, never shadow, canary or live |
The caveats, all of them, are in the [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius): four job submissions (an accidental on-demand job that was cancelled, a missing `einops`, a save that failed with `EPERM` and whose weights were recovered, then the job that saved); the night's own evaluation timed out at 15 s, so the gate result comes from running the eval and the gate again on the saved adapter with a 120 s timeout, through a one-off driver; five problems fixed by hand on the VM, now fixed in the repository (spot by default, `einops` pinned, the adapter saved locally and then copied, a clear stop when `aws` is missing, a separate student timeout for the night); and cost estimates (about 2.7 GPU-hours, at most about $4 at list prices) where the Nebius bill decides.
Every answer on Scout and the dashboard carries a badge:
| Badge | Means |
|---|---|
| | Nemotron 3 Nano 4B with the Night School adapter |
| | The untuned Nemotron 3 Nano 30B on Token Factory, used while the GPU endpoint is off. Not covered by the adapter's precision certificate |
| | Nemotron 3 Ultra on Nebius Token Factory |
Scout has a daily run cap. If it is used up you get a polite refusal; evaluators receive an access code with their instructions, never on a public page.
## What it cost
All of these are estimates recorded by the scripts, not invoices; the Nebius bill decides.
| Item | About |
|---|---|
| Labelling by the teacher, round 1 / round 2 | $2.87 + $9.18 |
| Nebius retrain, 27 Oct (2.73 GPU-hours, failed attempts included) | $2.16 |
| Endpoint speed tests, 27 to 28 Oct | $0.67 |
| Night School run, 28 Oct | about 2.7 GPU-hours, at most about $4 at list prices |
| One real request on the live site | $0.0002 |
| One Scout run | $0.02 |
Request and Scout costs are from the [29 Oct day-1 check](https://github.com/harshbhx/ratchet/tree/main/evidence/day1). The GPU-hour rate used in estimates does not appear on the price list Token Factory serves and is unverified.
## Verify a result with two commands
On the repository as cloned, these two commands recompute the published exam numbers from the saved answers:
```bash
python scripts/text2sql/exam_numbers.py --check
python scripts/toolcall_results.py --check
```
For a real signed bill of materials from your own gateway, `python -m ratchet.stats check bom.json --public-key ` recomputes its certificate from the signed counts, and `python -m ratchet.bom verify bom.json --public-key ` checks only the signature and the data lineage. The Night School BOM, signed on 29 Oct 2026, is published in the repository (see the proof tabs above). The [proof page](/guides/proof/) walks through every check; only the exam checks run on real files, and the others on documents the tests write with mock models.
## What to look at in the code
| Claim | Where it is enforced |
|---|---|
| Only open teachers are distilled; closed models are refused | [`ratchet/licenses.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/licenses.py) `require_distillable` |
| The student answers first and escalates when unsure | [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) `Ratchet.handle` |
| Releases go shadow, canary 10%, 50%, 100%, with rollback | [`ratchet/release.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/release.py) `ReleaseGate` |
| Each adapter has a signed bill of materials and a certificate | [`ratchet/bom.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/bom.py), [`ratchet/certificate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/certificate.py) |
| The cost report says when self-hosting does not pay | [`ratchet/cost.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/cost.py) `cost_report` |
## Honest limits
- **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)).
- **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)).
- **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)).
- **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)).
- **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate.
- **The nightly schedule on live traffic.** Night School has not run nightly on live traffic; it runs by hand today and the timer is scripted.
- **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline.
Run every script with `DRY_RUN=1` first.
Other limits:
- No live savings figure exists yet; the measured results are the exam scores on [Results](/guides/results/). The original adapters were trained on a borrowed GPU.
- Precision means agreement with the teacher, not ground truth.
- The redactor is pattern-based: names and street addresses are not caught yet.
- One gateway process per ledger (SQLite).
- Token Factory Terms of Service §5(f) ("competing service") is an open question, asked of Nebius support on Oct 19, 2026; no answer received as of 29 Oct 2026. Not legal advice.
## Where to read more
---
# The big picture
Source: https://docs.ratchet.dualmindlab.tech/guides/big-picture/
Ratchet is , plus a few Nebius services it calls: Token Factory for the teacher, a GPU endpoint for the student, a nightly training job and a storage bucket that carries data and adapters between them.
Picture the whole restaurant from above.
- **The pass** (the gateway) is where every order arrives and every plate leaves. It decides who cooks each order. It never cooks anything itself, so it needs only a small room.
- **The apprentice's station** (the student endpoint) is a rented kitchen with a stove, the GPU. The apprentice cooks there, reading from the **notebook of techniques** (the LoRA adapter you own).
- **The agency** (Nebius Token Factory) sends the **head chef** (Nemotron 3 Ultra) for each hard dish and charges per plate. It also has a food critic (the judge, Nemotron 3 Super) and a spare, untrained cook (the hosted fallback, Nemotron 3 Nano 30B) for when the apprentice's station is closed to save money.
- **The recipe book** (the ledger) sits at the pass. Every order, every dish the head chef cooks and every exam result is written in it.
- **The storeroom** (the Object Storage bucket) holds the night's study pack and every notebook the apprentice ever wrote.
- **Night school** (a Serverless Job on a GPU) opens once a night, reads the study pack from the storeroom and writes a new notebook back into it.
- **The doorman** (Caddy, in the Nebius VM kit) checks the front door and keeps the connection to guests private (HTTPS).
Three clocks run the place: one ticks for every order, one every few minutes to check how the newest notebook is doing, and one once a night for school.
## How it really works
The design in one picture. It shows the intended flow; the original adapters were trained by hand on a borrowed GPU, the round-2 recipe was retrained once by hand on Nebius AI Cloud, and teacher calls run on Nebius Token Factory.
The detailed diagram, with the services each piece runs on:
```mermaid
flowchart TB
app(["Your app · any OpenAI SDK"]) ==>|"POST /v1/chat/completions"| caddy["Caddy in the VM kit HTTPS, edge guard"]
caddy ==> gw
subgraph gw["Gateway · one process, no GPU"]
direction LR
router["Cascade Ratchet.handle"] --> ledger[("SQLite ledger")]
ledger <--> gate{{"Release gate"}}
end
subgraph tf["Token Factory, per token"]
direction TB
teacher["Nemotron 3 Ultra teacher"]
teacher ~~~ others["Judge: Super Fallback: Nano 30B, untuned"]
end
subgraph ep["Serverless Endpoint"]
student["Nano 4B + LoRA on vLLM"]
end
subgraph ns["Night School"]
direction TB
night["ratchet.live night cron, on the host"] --> job["Serverless Job LoRA training"]
end
bucket[("Object Storage bucket at /data")]
gw ==>|"escalate, audit"| tf
gw ==>|"3 samples"| ep
gw -->|"rows"| ns
ns -->|"adapter"| bucket
ep ---|"loads adapters"| bucket
class app,router,night,job neutral
class caddy,others outline
class gate gate
class ledger,bucket store
class student student
class teacher teacher
class ep studentZone
class tf teacherZone
```
**Reading this diagram.** Start at the top. The thick arrows are the request path: your app's call passes Caddy (only in the Nebius VM kit), then the cascade asks the student endpoint for 3 samples, and the teacher on Token Factory when the student is unsure, an audit is drawn or no student may answer (a cold start, a pinned task). Every request writes one row to the SQLite ledger; the release gate reads the audits there and writes version states back. At night, Night School reads the rows, a Serverless Job trains a new adapter into the Object Storage bucket, and the endpoint loads adapters from that bucket at `/data`. Not drawn, to keep it readable: the night also loads the candidate into vLLM, grades it there (the judge scores free-text rows) and hands it to the release gate. The hosted fallback answers student calls only when no endpoint is set.
The student: Nano plus your adapter
Rented per token
Keeps data
Decides releases
Optional or fallback
Request path
### The components
Sends a chat request with a task name.
Admission checks, then the student first.
Answers when it is confident.
Answers the rest.
| Component | What it does | Where it runs | Code |
|---|---|---|---|
| Your app | Sends OpenAI-style chat requests, with a task name | Anywhere | Any OpenAI SDK |
| Gateway | OpenAI-compatible API, admission checks, pages, Scout endpoints, the release gate's tick | One process, one uvicorn worker. The live gateway runs on Cloudflare Workers with Containers; the Nebius VM kit (`2vcpu-8gb`) is scripted and tested offline, ready to run | [`ratchet/gateway.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/gateway.py) (`Runtime`) |
| Cascade (router) | Student first, confidence, teacher on escalation, audits, shadow, cost | Inside the gateway | [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) (`Ratchet.handle`) |
| Student | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` with the LoRA adapter you own, served by vLLM with runtime LoRA loading | A Nebius Serverless Endpoint (L40S by default) | [`deploy/student_endpoint.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/student_endpoint.sh) |
| Teacher | `nvidia/Nemotron-3-Ultra-550b-a55b` answers escalations, cold starts, pinned tasks, audits and arbitrations | Nebius Token Factory | `RATCHET_TEACHER_MODEL` |
| Judge | `nvidia/nemotron-3-super-120b-a12b` scores free-text golden rows in JSON mode (a diagnostic, `judge_pass_rate`) | Nebius Token Factory | `RATCHET_JUDGE_MODEL` |
| Hosted fallback | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B`, untuned, plays the student when no endpoint is configured | Nebius Token Factory | `RATCHET_FALLBACK_STUDENT` |
| Ledger | Requests, examples with provenance, versions, events, failures, receipts | One SQLite file on the gateway's local disk | [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py) |
| Night School | Builds the train and golden sets, trains, evaluates, signs a bill of materials, calls the gate | Driven by cron on the gateway host; training on a Nebius Serverless Job (`gpu-l40s-a`, `1gpu-8vcpu-32gb`) with TRL and PEFT | [`ratchet/live.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/live.py), [`ratchet/nightschool/`](https://github.com/harshbhx/ratchet/tree/main/ratchet/nightschool) |
| Bucket | Training rows, adapters, bills of materials | Nebius Object Storage, mounted at `/data` in the job and the endpoint | `RATCHET_BUCKET_ID`, `RATCHET_SYNC_CMD` |
| Release gate | Golden gate, shadow, canary 10%, 50%, 100%, rollback | Inside the gateway (and the cron tick) | [`ratchet/release.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/release.py) (`ReleaseGate`) |
| Web app | `/` (the story), `/scout`, `/dashboard`; no-build pages at `/lite*` | Served by the gateway from `ratchet/web/dist` | [`web/`](https://github.com/harshbhx/ratchet/tree/main/web) |
| Scout | A multi-agent research app whose calls go through the cascade | Inside the gateway; Tavily for web search | [`ratchet/scout/`](https://github.com/harshbhx/ratchet/tree/main/ratchet/scout) |
| Caddy | Automatic HTTPS, security headers, an optional token guard on `/v1/*` and `/ratchet/admin/*` | Next to the gateway in the Nebius VM kit (Docker Compose) | [`deploy/vm/Caddyfile`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/Caddyfile) |
Token Factory listed no fine-tuning or LoRA serving for Nemotron when this design was chosen (research notes, Oct 19, 2026), and Nebius's [post on post-training in Token Factory](https://nebius.com/blog/posts/post-training-in-token-factory) names no Nemotron model to fine-tune (read Oct 26, 2026), so the student is trained and served on Nebius AI Cloud. The gateway itself does no inference, which is why a 2-vCPU machine is enough.
### How the pieces connect
1. **Your app calls the gateway** at `POST /v1/chat/completions` with an API key and a task name (`X-Ratchet-Task`). The gateway checks the body size, the key, rate limits, the request itself, the task and the quotas before any model is called.
2. **The cascade calls the student endpoint** over HTTPS with the OpenAI API. The model name it sends is the version's adapter name (`student-vN`), because vLLM serves each LoRA adapter under its own name. Three samples come back in one call.
3. **When the student is unsure, or for an audit, the cascade calls the teacher** on Token Factory. Each teacher answer that is complete and valid goes into the ledger as a training example, with the teacher's model id and license, after PII redaction.
4. **Every request writes one row** to the ledger: who served it, the route, confidence, cost and the all-teacher baseline. No prompt text is kept in that row; text is kept only in training examples, redacted.
5. **After requests, the release gate ticks**: it reads the audits and shadow verdicts from the ledger and moves versions between states (shadow, canary 10%, 50%, live, rolled back).
6. **At night, `python -m ratchet.live night` runs Night School**: it writes `night/train.jsonl` locally, uploads it to the bucket with `RATCHET_SYNC_CMD`, starts the training job with `RATCHET_TRAIN_CMD` (default `bash deploy/train_job.sh {version}`) and waits.
7. **The training job reads the bucket at `/data`** and writes the adapter to `/data/adapters/student-vN/`, with `NOTICE.md` and its training metadata beside the weights.
8. **The gateway host tells vLLM to load the new adapter** with `POST /v1/load_lora_adapter`, so no endpoint restart is needed. Night School then scores it on the golden set (the judge scores free-text rows), signs its bill of materials and hands the candidate to the gate. Last, the night writes the bill of materials and `NOTICE.md` into the adapter's folder and syncs the bucket again.
9. **The pages read the same ledger** through `GET /ratchet/stats`, `GET /ratchet/cost` and the other dashboard endpoints. Scout calls `Ratchet.handle` in the same process, once for each of its model calls.
- **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)).
- **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)).
- **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)).
- **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)).
- **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate.
- **The nightly schedule on live traffic.** Night School runs by hand today; the timer is scripted.
- **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline.
- **Measured savings.** They come from a published live run; none is published yet. The `nebius ai` commands in `deploy/` follow the CLI reference.
Run every script with `DRY_RUN=1` first.
### What a request leaves behind
Every request leaves exactly one row of metadata. Text is kept only when a teacher answer passes every check below, and then only after PII redaction.
```mermaid
flowchart TB
req(["A chat request"]) --> ans{{"A teacher answer?"}}
req --> row[("Request row, always metadata only")]
ans -->|"no: the student served"| x1["Nothing to store"]
ans -->|yes| only{{"Pinned, tool or teacher-only?"}}
only -->|yes| x2["Never stored"]
only -->|no| opt{{"A storage opt-out?"}}
opt -->|yes| x3["Not stored the row says which"]
opt -->|no| valid{{"Complete and valid?"}}
valid -->|no| x4["Served, not stored skip_example event"]
valid -->|yes| red["PII redaction"]
red --> ex[("Training example teacher, license, split")]
class req,red neutral
class row,ex store
class ans,only,opt,valid gate
class x1,x2,x3,x4 outline
```
**Reading this diagram.** The short branch to the cylinder always happens: one request row with the route, cost, confidence and ids, never the prompt or the answer. The API key appears in it by name only, and an end user only as a keyed hash. The chain of checks down the middle decides whether any text is kept; each dashed box is a way out with nothing stored. A teacher answer exists after an escalation, a cold start, an audit or a shadow arbitration; the student's own answers are never stored. Pinned tasks, tool requests and other teacher-only requests are never stored (the one exception is an over-length request on a task that sets `store_ineligible`). Any storage opt-out stops it next: `store: false`, `X-Ratchet-Store: none`, `metadata.ratchet_store`, a key with `store=0`, a no-store task, a no-train end user or zero retention. Then the answer must be complete and valid under the task's spec. Only then are the messages, the stored conditions and the answer redacted and written as a training example with the teacher's model id and license, in the golden or the train split. An erased example's hash is refused from then on.
### The live runtime
`RATCHET_MODE=live` builds the runtime in `Runtime.reset` in `ratchet/gateway.py`:
```mermaid
flowchart TB
l1[("SQLite file at RATCHET_DB")]
l1 --> l2["Teacher on Token Factory"]
l2 --> sb{{"RATCHET_STUDENT_BASE set?"}}
sb -->|yes| l3["Your adapter on your vLLM endpoint"]
sb -->|no| l4["Hosted fallback Nano 30B, untuned"]
class sb gate
class l1 store
class l2 teacher
class l3 student
class l4 outline
```
**Reading this diagram.** The runtime keeps a real ledger file and calls the teacher on Token Factory. Its student is your adapter on your own vLLM endpoint when `RATCHET_STUDENT_BASE` is set, and otherwise the untuned Nano 30B on Token Factory, drawn dashed because it is a stand-in, not the adapter you own.
| | Live mode |
|---|---|
| Ledger | SQLite file at `RATCHET_DB`, day 1 starts at the first request |
| Teacher and student | Token Factory teacher; vLLM endpoint, or the hosted fallback when `RATCHET_STUDENT_BASE` is empty |
| Night School | `python -m ratchet.live night` from cron, with a real training job (tested offline; ran once by hand on Nebius on 28 Oct 2026 and the golden gate rejected that student; not yet run on live traffic) |
| Audit rate | `RATCHET_AUDIT_RATE`, default 0.05 |
| Canary audits needed per stage | 20 (`ReleasePolicy.min_audits`) |
| Gate tick from requests | At most once every 5 s (`RATCHET_TICK_INTERVAL_S`) |
| Audits and shadow | Run after the response on 4 sidecar workers (`RATCHET_SIDECAR_WORKERS`) |
| API keys | Required (`RATCHET_API_KEYS`), unless `RATCHET_ALLOW_UNAUTHENTICATED=1` with a guard in front |
| Scout search | Tavily when `TAVILY_API_KEY` is set |
With no `RATCHET_STUDENT_BASE`, a trained version's student calls go to the untuned hosted Nano 30B, held to the global threshold `RATCHET_CONFIDENCE_THRESHOLD` (0.80) instead of the version's calibrated one. Those answers are still reported as `served_by: student`; the response's `model` field names the model that wrote them, and the pages label them "Nano 30B · hosted", not as your adapter.
### Three loops at three speeds
```mermaid
flowchart TB
R["Every request the cascade"] ==>|"rows, audits, examples"| L[("Ledger")]
L ==>|"audits, shadow verdicts"| G{{"Seconds to minutes the gate tick"}}
G ==>|"version states"| R
L -->|"train and golden rows"| N["Every night Night School"]
N -->|"a new candidate"| G
class R,N neutral
class L store
class G gate
```
**Reading this diagram.** The thick loop turns fastest: requests write rows, the gate tick reads the audits and shadow verdicts in them, and the version states it writes decide which student the next request meets. The thin path turns once a night: Night School reads the training and golden rows from the same ledger and hands a new candidate to the gate. In live mode cron can also run the tick, so a quiet gateway still moves its releases.
| Loop | How often | What it does | Entry point |
|---|---|---|---|
| The request | Every call; seconds at most | Pick a version, sample the student, serve or escalate, maybe audit, maybe shadow, log one row | `Ratchet.handle` |
| The gate tick | After requests (live: at most every 5 s) and from cron every 5 minutes | Shadow decisions, canary 10% to 50% to live, rollback, alerts | `ReleaseGate.tick`, `python -m ratchet.live tick` |
| The night | Once a night (02:30 on the VM kit's cron line) | Skip if a release is in flight, training is paused or fewer than 50 new examples arrived; otherwise build, train, evaluate, gate | `run_night`, `python -m ratchet.live night` |
Two more jobs run on the side: `python -m ratchet.live seal` seals the savings receipts of each ledger day that ended (hourly), and `deploy/vm/backup.sh` backs up the ledger (daily at 03:10 UTC). In the VM kit only the backup is switched on by default; the night, tick and seal lines in `/etc/cron.d/ratchet` are commented out until live mode works by hand.
The loops share only the ledger. The gate reads what requests wrote, and Night School does not train while a version is in shadow or canary, so a release finishes or fails before the next candidate is trained.
## For engineers
**`Runtime` wires it all.** [`ratchet/gateway.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/gateway.py) `Runtime.__init__` checks the whole configuration first (`config.check`, every error in one `ConfigError`), then `Runtime.reset` builds the ledger, the teacher provider, the student resolver (in live mode: the endpoint and `v.adapter_uri`, or the hosted fallback model), the `Ratchet` cascade and the `ReleaseGate`.
**Settings that shape the picture** (defaults from [`ratchet/config.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/config.py); all of them in the [configuration reference](/reference/configuration/)):
| Variable | Default | Meaning |
|---|---|---|
| `RATCHET_MODE` | `mock` | `mock` or `live` |
| `RATCHET_DB` | `ratchet.db` | Ledger file (live mode) |
| `RATCHET_TEACHER_BASE` | `https://api.tokenfactory.nebius.com/v1` | Token Factory, for the teacher, judge and hosted fallback |
| `RATCHET_TEACHER_MODEL` | `nvidia/Nemotron-3-Ultra-550b-a55b` | Must be licensed for distillation, or startup fails |
| `RATCHET_STUDENT_BASE` | empty | Your vLLM endpoint; empty means the hosted fallback |
| `RATCHET_STUDENT_BASE_MODEL` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | The base every adapter is trained on |
| `RATCHET_DATA_DIR` | `data` | Local folder laid out like the bucket |
| `RATCHET_TRAIN_CMD` | `bash deploy/train_job.sh {version}` | Starts training and waits for it |
| `RATCHET_ADAPTER_ROOT` | `/data/adapters` | Where vLLM finds adapters; also used to reload one vLLM forgot |
**The bucket's layout**, as the training job and the endpoint see it at `/data`:
- data
- night
- train.jsonl the rows Night School built tonight
- adapters
- student-v3
- adapter_model.safetensors written by the training job
- ratchet_manifest.json base model, licenses, template arguments
- NOTICE.md base model and teacher licenses
- ratchet_bom.json the signed bill of materials
- boms
- student-v3.json a copy of the same bill of materials
The ledger is not in the bucket. With the default WAL journal it must stay on the gateway host's local disk, because WAL needs shared memory between the processes that open the file (the gateway and the cron jobs); `RATCHET_DB_SYNC=FULL` is the setting for a network mount ([architecture](/reference/architecture/), section 11). In the VM kit it lives in a Docker volume.
**Commands:**
```bash
bash deploy/student_endpoint.sh create # once; DRY_RUN=1 first. Later: start, stop, status
python -m ratchet.live night # Night School (cron)
python -m ratchet.live tick # the release gate (cron, every 5 minutes)
python -m ratchet.live seal # savings receipts of days that ended (cron, hourly)
python -m ratchet.config check # the startup config check: exit 0, 1 or 2
DRY_RUN=1 bash deploy/vm/create_vm.sh # the Nebius VM kit, commands printed only
```
**Endpoints the pages use:** `GET /ratchet/stats`, `GET /ratchet/cost`, `GET /ratchet/tasks`, the Scout endpoints under `/scout/*`. See the [API reference](/reference/api/).
## Common questions
### Why does the gateway not need a GPU?
It never runs a model. It sends the student's samples to the vLLM endpoint and the teacher's calls to Token Factory, and does the rest (scoring, logging, the gate) in plain Python. The Nebius VM kit's machine is 2 vCPU and 8 GB.
### Why one SQLite file and one process?
At this scale (thousands to tens of thousands of requests a day) one file needs no setup, is easy to back up and inspect, and is fast enough. The price is one gateway process: run one uvicorn worker. A Postgres ledger for several workers is planned.
### What happens when the GPU endpoint is stopped?
If `RATCHET_STUDENT_BASE` still points at it, student calls fail, and after 5 failures in a row the student's circuit breaker sends requests straight to the teacher (route `student_breaker`) until a probe succeeds. To keep a student answering while the endpoint is off, leave `RATCHET_STUDENT_BASE` empty: student calls then go to the hosted, untuned Nano 30B on Token Factory, held to the 0.80 default threshold and not covered by the adapter's precision certificate. Nebius's docs say a stopped endpoint is not billed for compute or its own storage; a mounted bucket still is.
### Why not train and serve the student on Token Factory?
Token Factory listed no fine-tuning or LoRA serving for Nemotron when this design was chosen (research notes, Oct 19, 2026), and Nebius's [post on post-training in Token Factory](https://nebius.com/blog/posts/post-training-in-token-factory) names no Nemotron model to fine-tune (read Oct 26, 2026). So training runs as a Serverless Job and serving on a Serverless Endpoint, both on Nebius AI Cloud.
### What does running it cost?
Estimates only. The Nebius VM kit, if you host the gateway on it: about $72.50 for Oct 29 to Dec 15, 2026, before live usage (prices as of Oct 20, 2026, [VM kit guide](/reference/deploy-nebius-vm/)). The student endpoint: about $32 per full day if left on, and a training run about $0.70 to $1.35 a night, at about $1.35 per L40S GPU-hour ([Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026; [deployment guide](/reference/deployment/#4-cost-estimates)). See [Cost and savings](/guides/cost-and-savings/).
### What runs on Nebius today?
The teacher: it runs on Nebius Token Factory, and the live gateway calls it for real. As of Oct 29, 2026 the GPU side is covered by offline tests with fake transports and has run by hand: Night School ran end to end once on Nebius, by hand (28 Oct 2026): the gateway on a Nebius CPU VM, the training job on a spot L40S, the adapter served from a Nebius Serverless Endpoint. The golden gate rejected it (81.8% precision against the 95% target), so it was never made live; the loop has not run nightly on live traffic ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). The one-command host is ready to run with `make nebius-plan` and `make nebius-up` after the day-1 checks. The original text-to-SQL and tool-calling adapters were trained on a borrowed GPU; the round-2 recipe was retrained once by hand on a Nebius Serverless Job on 27 Oct 2026, and the student was served by hand, in two short sessions, on an endpoint for speed tests on 27 to 28 Oct 2026 ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)).
## Related
---
# One request's journey
Source: https://docs.ratchet.dualmindlab.tech/guides/request-journey/
A request enters at `POST /v1/chat/completions`, passes the gateway's admission checks, goes through `Ratchet.handle` (the student first, the teacher when the student is unsure), is logged as one ledger row with a route that says why it was served that way, and comes back as a normal OpenAI response with a `ratchet` block.
An order arrives at **the pass**. First a quick check: is the slip readable, is the guest allowed in, is this table over its limit tonight? A refused order is turned away before anyone touches a pan, and costs nothing.
The slip names the dish (the task). Some dishes always go straight to **the head chef** (pinned tasks). For everything else **the apprentice** cooks the dish three times. If all three plates match and the apprentice is sure, one goes out, and now and then **the inspector** asks the head chef to cook the same dish to check it.
If the apprentice is unsure, the head chef cooks instead, and that dish goes into **the recipe book** with the head chef's name and a note that copying it is allowed.
Every order gets one line in the book: who served it, why, and what it cost. The plate goes out with a small slip on the side (the `ratchet` block) saying who cooked it and what the head chef alone would have cost.
If the apprentice's stove breaks, the head chef simply cooks. If the head chef cannot come, the guest gets an apology (an error), unless you arranged a stand-in in advance.
## How it really works
### Every decision on the way
```mermaid
flowchart TB
adm{{"Admission checks pass?"}}
adm -->|no| refused["Refused: 4xx or 503 no model call, no request row"]
adm -->|yes| only{{"Teacher-only request? pinned, tools, ineligible"}}
only -->|no| ver{{"A version that serves this task?"}}
ver -->|yes| up{{"Student call works?"}}
up -->|yes| conf{{"Confident enough to serve?"}}
conf -->|yes| serve["Student serves route student"]
serve --> audit{{"Audit drawn? RATCHET_AUDIT_RATE"}}
ver -->|"no: cold_start, student_disabled, task_disabled"| teach
up -->|"no: student_breaker, student_error"| teach
conf -->|"no: confidence, n_choices, rule:i"| teach["Teacher answers the escalation"]
audit -->|"yes: the teacher checks it at temperature 0"| store
teach --> store{{"Storable? no opt-out, complete and valid"}}
store -->|yes| ex[("Training example redacted, licensed")]
only -->|"yes: the teacher answers alone"| log
audit -->|no| log
store -->|no| log
ex --> log(["Log one row, tick the gate, send the response"])
class adm,only,ver,up,conf,audit,store gate
class refused danger
class serve student
class teach teacher
class ex store
class log ok
```
**Reading this diagram.** Each hexagon is one check, in the order the code runs it. A refusal at admission costs nothing and calls no model. Teacher-only requests (a pinned task, a tool request, a request the student must not see) skip the student entirely and are not stored. Otherwise the student gets its chance, and each "no" on the way down sends the request to the teacher with a route that names the reason, the labels on the arrows. A confident student serves, and one random draw decides whether the teacher also checks that answer. Both teacher paths meet at the same storage check before anything becomes training data. Every path but a refusal ends in one ledger row; a teacher failure when its answer is needed ends in an error instead ([When something fails](#when-something-fails)).
A check
The student serves
The teacher answers
Kept as data
Refused
### An escalation, and an audit after the response
```mermaid
sequenceDiagram
autonumber
participant Client
participant Gateway
participant Student as Student vLLM
participant Teacher as Teacher Ultra
participant Ledger
Client->>Gateway: chat request
Note over Gateway: admission, pick a version
Gateway->>Student: 3 samples
Student-->>Gateway: samples
Note over Gateway: confidence vs threshold
alt unsure: escalate
Gateway->>Teacher: same request
Teacher-->>Gateway: answer
Gateway->>Ledger: example, if valid
Gateway->>Ledger: row, route confidence
Gateway-->>Client: teacher's answer
else confident, audit drawn
Gateway->>Ledger: row, audit pending
Gateway-->>Client: student's answer
rect rgba(42, 120, 214, 0.08)
Note over Gateway,Ledger: after the response, on a sidecar worker
Gateway->>Teacher: same request, temperature 0
Teacher-->>Gateway: answer
Gateway->>Ledger: verdict, example, finish the row
end
end
```
**Reading this diagram.** Steps 1 to 3 are the same for both outcomes: one call to the student endpoint returns 3 samples at temperature 0.7 with token logprobs, and the gateway scores them. In the upper branch the student is unsure, so the teacher answers (4, 5), its answer is stored if it is complete and valid (6), the row is logged with route `confidence` (7) and the client gets the teacher's answer (8). In the lower branch the student is confident and the audit draw comes up. In live mode the row is logged with the audit pending (9) and the client gets the student's answer at once (10). The shaded part runs afterwards on a sidecar worker: the teacher answers the same request at temperature 0, and the verdict, the stored example and the costs finish the row (11 to 13). In mock mode, and for Scout, the same audit runs before the response.
A shadow's work runs after the response in live mode too. The audit draw itself is always made on the request path, so the same draws happen whether the audit runs before or after the response. See [Escalation and audits](/guides/escalation-and-audits/).
### Before the cascade: admission
The gateway runs these checks in this order. A request refused here calls no model, writes no request row and costs nothing (a 401, 413, 429 or 503 leaves a row in the `failures` table; a 400 only a metric).
| Order | Check | Refused with | Setting (default) |
|---|---|---|---|
| 1 | Declared body size | 413 `request_too_large` | `RATCHET_MAX_BODY_BYTES` (2 MiB) |
| 2 | API key | 401 `invalid_api_key` | `RATCHET_API_KEYS` |
| 3 | Requests a minute, per key (per client IP on a gateway without keys) | 429 `rate_limit_exceeded` | `RATCHET_API_KEY_RPM` (off), `RATCHET_IP_RPM` (120 in live mode) |
| 4 | Body read and counted | 413 for a chunked body over the limit | |
| 5 | Requests in flight: the gateway's cap, then the key's | 503 `overloaded`, 429 `rate_limit_exceeded` | `RATCHET_MAX_INFLIGHT` (32), `RATCHET_API_KEY_CONCURRENCY` (off) |
| 6 | `Idempotency-Key` and JSON | 400; a finished answer for the same key and body is sent again with `Idempotent-Replayed: true` | `RATCHET_IDEMPOTENCY_TTL_S` (600 s) |
| 7 | Validation and normalisation | 400 with `param` and `code` | `RATCHET_MAX_MESSAGES` (256), `RATCHET_MAX_INPUT_CHARS` (120,000), `RATCHET_MAX_N` (4) |
| 8 | Task selection and the key's task scope | 400 `unknown_task`, 403 `task_not_allowed` | `RATCHET_UNKNOWN_TASKS` (`distill`) |
| 9 | Daily request quota | 429 `daily_quota_exceeded` | `RATCHET_API_KEY_DAILY_REQUESTS` (off) |
| 10 | US dollar budgets, per key and for the gateway | 429 `budget_exceeded`, `gateway_budget_exceeded` | `RATCHET_API_KEY_DAILY_USD`, `RATCHET_DAILY_BUDGET_USD` (off) |
**Validation** ([`ratchet/openai_compat.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/openai_compat.py)) follows OpenAI's rules and answers in OpenAI's error envelope. Normalisation turns a `developer` message into `system` and text content parts into one string, so the cascade and the ledger see plain messages. The request's `model` field is ignored: Ratchet chooses the model.
**Task selection.** The `X-Ratchet-Task` header wins over `metadata.ratchet_task`. With neither, a `response_format` json_schema request gets an auto task derived from its schema, and anything else is `default`. A name that is not registered follows `RATCHET_UNKNOWN_TASKS`: `distill` (the default) makes a task without fields that compares whole answers, `teacher` sends the request to the teacher alone, `reject` is a 400. A task whose `clients` list does not name the key is a 403. No header or field can pin a task; only an operator's spec does.
After admission the rest runs in a worker thread, so the ledger's lock never blocks the event loop.
### Inside the cascade: `Ratchet.handle`
Step through it one decision at a time, or read every rule in the numbered list under it.
1. **Read the switches and settle the request's terms.** The operator's runtime switches and task pins are read once (`Ledger.flags`). A `response_format` is bound to the task as a contract. The client's generation parameters are split by role (`GenPolicy`): what the student may receive, what the serving teacher may receive, and the audit's fixed parameters. Whether the request's teacher answers may be stored is settled now (`store_opt_out`: zero retention, a no-store task, the key's `store=0`, `store: false`, or a no-train end user).
2. **Teacher-only paths.** Some requests never reach the student. An unregistered task under `RATCHET_UNKNOWN_TASKS=teacher` (route `unknown_task`), a pinned task (route `pinned`), and a tool request, meaning tools offered or tool calls in the messages (route `tools`), all go to `_teacher_only`: the teacher answers, the request is logged, and nothing is stored as training data. A request the student must not see takes the same kind of path through `_ineligible` (route `ineligible:`): a content part only the teacher reads (`content_type`), or more input than the task's `student_max_input_tokens` (`input_tokens`); only an over-length request on a task with `store_ineligible` may be stored. No student, audit or shadow call on any of these.
3. **Pick a version** (`_pick_version`). If an operator turned the student off, none (route `student_disabled`). If a version is in canary, it is picked with probability `traffic_pct / 100` (10% or 50%); otherwise the live version. With neither (cold start) the teacher answers (route `cold_start`), and its answer is stored as usual.
4. **Skip a task the version cannot serve.** If the version's per-task record holds the task back (the release gate disabled it on that version, or there was no held-out row for it), the student is not called at all: the teacher answers (route `task_disabled`).
5. **Sample the student** (`_sample`). One call to the endpoint for `max(3, n)` samples at temperature 0.7 with token logprobs (`confidence.STUDENT_SAMPLING`), with only the client's shaping fields (`max_tokens`, `stop`, `seed`, `response_format`, `logit_bias`, `top_logprobs`), capped at 1,024 output tokens. The model name sent is the adapter's (`student-vN`). The call goes through the student's circuit breaker first. If the breaker is open the teacher answers (route `student_breaker`); if the call fails the teacher answers (route `student_error`), and a vLLM 404 naming the adapter queues an automatic reload.
6. **Score confidence** (`confidence.score`) on the first 3 samples, exactly as at calibration. A sample that is empty, cut off, unparseable or invalid under the task's spec gets no vote. Agreement is counted on the task's fields (for `ticket-triage`: `category` and `priority`). Confidence is `0.4 × p + 0.6 × agreement`, where `p` is the best agreeing sample's mean token probability; without logprobs it is agreement alone. See [Confidence](/guides/confidence/).
7. **Compare with the threshold** (`Ratchet.threshold`). Each version carries its own threshold per task, calibrated on half of its golden set. The hosted fallback model gets `RATCHET_CONFIDENCE_THRESHOLD` (0.80); a task whose spec changed since the version was scored gets 1.01, which nothing reaches. The student serves when at least one sample is valid, confidence is at or above the threshold, and, for `n` choices, at least `n` samples agree (route `student`). A confident student without `n` agreeing samples escalates (route `n_choices`); an unsure one escalates (route `confidence`). A task's `escalate_when` rules can still send a confident answer to the teacher (route `rule:`, for the rule that held).
8. **Maybe audit.** When the student serves, one random draw decides an audit: probability `RATCHET_AUDIT_RATE` (0.05; at least 0.10 in mock mode), or `RATCHET_CANARY_AUDIT_RATE` for a canary when set, or an operator's runtime rate. If drawn, the teacher answers the same request at temperature 0 and its answer is compared with the student's. The student's answer is served either way.
9. **Escalate otherwise** (`_call_teacher`). The teacher answers under the serving parameters: the client's temperature or 0, and 0.7 for `n` choices when the client set none, capped at 8,192 output tokens. The answer is cleaned (reasoning removed), served, and stored as a training example only when it is complete and valid. If the teacher fails, see [When something fails](#when-something-fails) below.
10. **Run the shadow.** If a version is in shadow, it is sampled and scored the same way, silently. Its verdict (would it serve, and does it agree with the best reference) goes into the row; a disagreement with the live student's answer is settled by the teacher (arbitration). A request answered by a stand-in runs no shadow.
11. **Log one row** (`Ledger.log_request`): task, who served, version, confidence, whether it escalated or was audited, the shadow's verdict, latency and `wall_ms`, `cost_usd` with its student and teacher parts, the all-teacher baseline, `stored`, `route`, the threshold used, `label_p` (the audit rate the answer was drawn at) and the request's ids. The row holds metadata only, never the prompt or the answer.
12. **Back in the gateway.** The spend meter adds the request's cost to its key, the release gate ticks (at most once every 5 s in live mode, `RATCHET_TICK_INTERVAL_S`), and the response is built from the router's `Result`.
Scout's agents call `handle(messages, task)` directly. They skip the chat route's admission checks (Scout has its own budget guard), and their audits always run inline.
### Every route
Every row gets a `route` (`router.ROUTES`). `/ratchet/stats` counts them per task, so you can see why each request went where it went.
| Route | Served by | Why | Training example? |
|---|---|---|---|
| `student` | student | Confidence reached the threshold (and `n` samples agreed) | Only from an audit's teacher answer |
| `confidence` | teacher | The student was unsure, had no valid sample, or the task's spec changed | Yes, if valid |
| `n_choices` | teacher | Confident, but fewer than `n` samples agreed | Yes, choice 0 only |
| `cold_start` | teacher | No live version, and the canary was not picked | Yes, if valid |
| `student_disabled` | teacher | An operator turned the student off | Yes, as on a cold start |
| `task_disabled` | teacher | The version's per-task record serves nothing, so no student call | Yes, if valid |
| `student_error` | teacher | The student call failed (timeout, connection, error status, lost adapter) | Yes, if valid |
| `student_breaker` | teacher | The student's circuit breaker was open | Yes, if valid |
| `fallback_teacher` | teacher | The primary teacher was unavailable, so the fallback teacher answered | Only with `RATCHET_STORE_FALLBACK_TEACHER=1` |
| `degraded` | student | Unsure student, unavailable teacher, `RATCHET_ON_TEACHER_FAILURE=serve_student` | Never |
| `pinned` | teacher | The task is pinned by its spec or by an operator | Never |
| `unknown_task` | teacher | Unregistered task under `RATCHET_UNKNOWN_TASKS=teacher` | Never |
| `tools` | teacher | A tool request; the teacher answers with the tools | Never |
| `rule:` | teacher | Rule `i` of the task's `escalate_when` held for the confident student's answer | Yes, if valid |
| `ineligible:` | teacher | The student must not see the request: a content part only the teacher reads (`content_type`, `RATCHET_TEACHER_MODALITIES`) or too much input (`input_tokens`) | Only an over-length request on a task with `store_ineligible` |
| `multimodal` | teacher | Set by a caller of `Ratchet.teacher_only` for a part only the teacher reads; the chat route uses `ineligible:content_type` | Never |
"If valid" also means no storage opt-out applies. Rows written before v0.5 have no route and are counted as `unrecorded`.
### The response
A standard `chat.completion` with every field OpenAI's has, plus a `ratchet` block. This example is from the [API reference](/reference/api/#post-v1chatcompletions) (illustrative values):
```json
{
"ratchet": {
"served_by": "student",
"student_version": 5,
"confidence": 0.951,
"escalated": false,
"cost_usd": 0.0000312,
"all_teacher_cost_usd": 0.00131,
"latency_ms": 311.4,
"usage_breakdown": {
"student_samples": {"in": 104, "out": 93},
"served": {"in": 104, "out": 31},
"audit": null, "shadow": null, "arbitration": null, "repair": null
},
"degraded": null,
"stored": false,
"store_opt_out": null,
"request_id": "req_3f1c0a9b8e7d6c5b4a3f2e1d0c9b8a79",
"row_id": 4512,
"task": "ticket-triage",
"schema_valid": null
}
}
```
| Field | Meaning |
|---|---|
| `served_by` | `student` or `teacher` |
| `student_version` | The version that answered; `null` when the teacher answered |
| `confidence` | The student's confidence when it was sampled, else `null` |
| `escalated` | `true` when a version was picked but the teacher answered instead (unsure, too few agreeing samples, a failed or skipped student call, a task held back); stays `true` on a degraded answer |
| `cost_usd` | What this request cost: every model call, the student priced per token. With `overhead_pending`, only what serving cost |
| `all_teacher_cost_usd` | What the teacher alone would have cost: its own bill when it answered, else an estimate from the student's tokens plus 300 assumed reasoning tokens (`RATCHET_TEACHER_REASONING_TOKENS`) |
| `usage_breakdown` | Every model call, in tokens: `student_samples`, `served`, `audit`, `shadow`, `arbitration`, `repair` |
| `degraded` | `"teacher_unavailable"` on a degraded answer, else `null` |
| `stored`, `store_opt_out` | Whether a teacher answer of this request became (or matched) a training example, and why not if opted out |
| `request_id`, `row_id` | The request's id (`req_` and 32 hex, as in `x-request-id`; feedback names it) and its ledger row (a savings receipt proves it by this) |
| `task`, `schema_valid` | The resolved task, and whether every choice meets the `response_format` |
| `overhead_pending` | Present and `true` only when the audit or shadow runs after the response |
`usage` is the served answer's own tokens, as if its model had answered alone; Ratchet's extra calls are in `usage_breakdown` and priced in `cost_usd`. `model` names who wrote the answer (the adapter `student-vN`, the hosted Nano 30B's id, the teacher's id, or the fallback teacher's), and `system_fingerprint` changes when a promotion or rollback puts another version in front.
Headers: `X-Ratchet-Served-By`, `X-Ratchet-Request-Id` (the ledger row), `X-Ratchet-Degraded` on a degraded answer, and on every response `x-request-id` and `openai-processing-ms`.
### Streaming
With `stream: true` the gateway sends OpenAI `chat.completion.chunk` events, a few words per chunk, and the `ratchet` block rides on the last finish chunk (or on a final usage chunk with `stream_options.include_usage`). But the cascade must see the student's whole answer to score it, so the answer is complete before the first chunk: time to first token equals the full latency. Streaming exists so streaming clients work unchanged. Errors come back as normal JSON before the stream starts.
### When something fails
Only a teacher call whose answer is needed can fail a request. Everything else fails open.
**The student fails: the teacher answers.** A student call on the request path waits 15 s to answer and 2 s to connect (`RATCHET_STUDENT_TIMEOUT_S`, `RATCHET_STUDENT_CONNECT_TIMEOUT_S`) and is never retried: the teacher is its retry. After 5 endpoint failures in a row (timeouts, connection errors, 5xx, 408, 429) the student's circuit breaker opens and requests go straight to the teacher without waiting:
```mermaid
stateDiagram-v2
direction LR
[*] --> closed
closed --> open: 5 failures in a row
open --> half_open: 30 s later, one probe
half_open --> closed: probe succeeds
half_open --> open: probe fails, wait doubles
class closed ok
class open danger
class half_open pending
```
**Reading this diagram.** While the breaker is `closed`, every request tries the student. Five endpoint failures in a row (timeouts, connection errors, 5xx, 408, 429) open it, and requests go straight to the teacher with route `student_breaker`, without waiting for a timeout. After the wait exactly one request probes the endpoint (`half_open`): success closes the breaker and resets the wait to 30 s, failure opens it again for twice as long, up to 300 s. Each move writes a `circuit` event.
A shadow has its own breaker, so a bad staging adapter cannot open the one live traffic depends on. When vLLM forgets an adapter after a restart, its 404 queues a reload from `RATCHET_ADAPTER_ROOT` (at most once a minute per adapter).
**The teacher fails on an audit or a shadow arbitration: the student's answer is served anyway**. The failure is counted and the audit simply does not count.
**The teacher fails when its answer is needed** (an escalation, a cold start, a pinned or teacher-only request). The teacher gets up to 2 retries within 180 s (`RATCHET_TEACHER_RETRIES`, `RATCHET_TEACHER_DEADLINE_S`; a 120 s read timeout), on connection failures and 408, 409, 429 and 5xx. If it still fails, no request row is written; a `failures` row records what the student's samples spent, which also counts against the key's budget. The gateway maps the cause (`_map_upstream`):
| Status | `code` | When |
|---|---|---|
| 400 | the teacher's, such as `context_length_exceeded` | The teacher refused the request itself (upstream 400, 413 or 422); `X-Should-Retry: false` |
| 429 | `upstream_rate_limited` | Token Factory rate-limited the call; `Retry-After` is its wait, 1 to 60 s |
| 502 | `teacher_unavailable` | The call failed after its retries |
| 502 | `upstream_auth_error`, `upstream_not_found` | The gateway's own key or model setting is wrong; `X-Should-Retry: false` |
| 502 | `tool_calls_unsupported` | The teacher answered a request without tools with a tool call |
| 503 | `teacher_circuit_open` | The teacher's breaker is open after 5 failed calls in a row; answered at once with `Retry-After` |
| 504 | `teacher_timeout` | The teacher did not answer in time |
Two opt-in stand-ins answer instead of an error. A **fallback teacher** (`RATCHET_TEACHER_FALLBACK_MODEL`, licensed at startup like any teacher) answers, when the primary is unavailable, any request whose teacher answer will be served (route `fallback_teacher`); audits and arbitrations never use it. **Degraded mode** (`RATCHET_ON_TEACHER_FAILURE=serve_student`) serves the student's own valid, below-threshold answer to an escalation, marked `ratchet.degraded` and `X-Ratchet-Degraded` (route `degraded`). Neither covers a 400 about the request. A stand-in's answer is never audited or shadowed, and a degraded answer is never stored, so neither can move a release.
## For engineers
| File | Functions |
|---|---|
| [`ratchet/gateway.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/gateway.py) | `chat` (key, rate, body, in-flight caps), `_complete` (validation, `_request_meta`, task, quota, budgets, cascade, response), `_map_upstream`, `_envelope` |
| [`ratchet/openai_compat.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/openai_compat.py) | `validate_chat_request`, `normalize_messages`, `APIError` |
| [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) | `Ratchet.handle`, `_pick_version`, `_sample`, `threshold`, `_audit`, `_call_teacher`, `_store_example`, `_shadow`, `_teacher_only`, `_may_degrade`, `ROUTES` |
| [`ratchet/confidence.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/confidence.py) | `score`, `majority`, `STUDENT_SAMPLING` |
| [`ratchet/resilience.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/resilience.py) | `Breaker`, `retry_call`, `endpoint_down`, `teacher_unavailable` |
| [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py) | `log_request`, `add_example`, `flags` |
| Setting | Default | Effect on the journey |
|---|---|---|
| `RATCHET_SAMPLES` | 3 | Student samples per request, and per golden row at calibration |
| `RATCHET_CONFIDENCE_THRESHOLD` | 0.80 | Threshold for the hosted fallback, or a version without a calibrated one |
| `RATCHET_AUDIT_RATE` | 0.05 | Share of student answers the teacher also checks (at least 0.10 in mock mode) |
| `RATCHET_CANARY_AUDIT_RATE` | unset | A canary's own audit rate |
| `RATCHET_STUDENT_TIMEOUT_S`, `RATCHET_STUDENT_CONNECT_TIMEOUT_S` | 15, 2 | Student call limits; never retried |
| `RATCHET_TEACHER_TIMEOUT_S`, `RATCHET_TEACHER_RETRIES`, `RATCHET_TEACHER_DEADLINE_S` | 120, 2, 180 | Teacher read timeout, retries and overall deadline |
| `RATCHET_STUDENT_BREAKER_FAILURES`, `RATCHET_STUDENT_BREAKER_COOLDOWN_S` | 5, 30 | Student breaker (0 failures turns it off) |
| `RATCHET_TEACHER_BREAKER_FAILURES`, `RATCHET_TEACHER_BREAKER_COOLDOWN_S` | 5, 30 | Teacher breaker |
| `RATCHET_TEACHER_FALLBACK_MODEL` | empty | Opt-in fallback teacher |
| `RATCHET_ON_TEACHER_FAILURE` | `error` | `serve_student` turns on degraded mode |
| `RATCHET_STUDENT_MAX_TOKENS`, `RATCHET_TEACHER_MAX_TOKENS` | 1,024, 8,192 | Output caps per call |
| `RATCHET_SIDECAR_WORKERS` | 4 live, 0 mock | Audits and shadows after the response |
| `RATCHET_TICK_INTERVAL_S` | 5 live, 0 mock | Least time between two gate ticks from requests |
All of them are in the [configuration reference](/reference/configuration/). Look a request up afterwards with `GET /v1/ratchet/requests/{id}` (an API key, its own requests) or `GET /ratchet/requests/{id}` (the dashboard API), by the `req_` id or the `chatcmpl-` id.
```bash
curl -si http://localhost:8000/v1/chat/completions \
-H 'content-type: application/json' \
-H 'X-Ratchet-Task: ticket-triage' \
-d '{"model":"ratchet","messages":[{"role":"user","content":"I was charged twice for my Pro plan this month."}]}'
# look for X-Ratchet-Served-By, X-Ratchet-Request-Id and the "ratchet" block
```
## Common questions
### Can a client force the student or the teacher?
No. The request's `model` field is ignored, and no header or field can pin a task. Only an operator's task spec, or a runtime pin through the admin API, sends a task to the teacher alone.
### Does streaming make answers arrive sooner?
No. The student's whole answer must be scored before Ratchet decides who serves, so the first chunk comes when the answer is complete.
### Why three samples, not one?
Small models are poorly calibrated, so one token probability does not tell right from wrong. Agreement across three samples plus token probability is the signal the threshold is calibrated on. The cost is the student's output tokens three times, small next to one teacher call.
### Is my prompt stored?
The request row holds metadata only. Prompt and answer text is stored only as a training example from a teacher answer, after PII redaction (emails, phone and card numbers and similar; names and street addresses are not caught yet), and not at all under a storage opt-out. Pinned, tool and multimodal requests are never stored. See [Security](/guides/security/).
### What does a request cost me?
`ratchet.cost_usd` is the full price of every call the request made, at the Token Factory list prices checked on 29 Oct 2026 (the GPU-hour rate is still unverified). `usage` is only the served answer's tokens, so you can bill your own users for the answer they got.
### What if the student is wrong but confident?
It is served. A random audit may catch it, and audits drive the canary's rollback and the live version's alert. Precision means agreement with Nemotron 3 Ultra, not ground truth.
### Has this path run against real endpoints?
Partly. The teacher runs on Nebius Token Factory. The student side is tested offline and ran once by hand on Nebius on 28 Oct 2026 (the golden gate rejected that student): Ratchet's own adapter is not connected to the live gateway (a random 40% of eligible requests are answered by the hosted Nano 30B, a quarter of those re-checked by Ultra), and the paths through vLLM are tested with fake HTTP transports, ready to run with `make nebius-up`. Model ids and per-token prices were checked on 29 Oct 2026 (`scripts/day1_check.py` passed); the GPU-hour rate is still unverified.
## Related
---
# Confidence
Source: https://docs.ratchet.dualmindlab.tech/guides/confidence/
Confidence is the score that decides, on every request, whether the student's answer goes out or the teacher is asked, and each adapter version gets its own threshold, set on half of the golden set and checked on the other half.
The apprentice chef gets an order. Before anything leaves the kitchen, the apprentice cooks the dish **three times**. If all three plates come out the same, and the apprentice felt sure at every step while cooking, the plate goes out. If the plates differ, or one of them is burnt, the head chef from the agency cooks the order instead.
Why not just ask the apprentice "are you sure?" Because young cooks are often sure and wrong. So the pass looks at two things at once: did the three plates match, and how steady was the apprentice's hand while cooking.
How sure is sure enough? That bar is set with **the sealed exam**: recipes the apprentice never studied. Half of the exam sets the bar: the lowest bar at which at least 97 of every 100 plates the apprentice would send out match the head chef's. The other half grades the result. The apprentice never saw the grading half while the bar was set, so the grade is honest.
Every new notebook gets its own bar, and each kind of dish can get its own bar too: an apprentice can be great at salads and shaky at soufflés.
## How it really works
```mermaid
sequenceDiagram
participant C as Client
participant G as Gateway
participant Student as Student Nano 4B + LoRA
participant Teacher as Teacher Ultra
C->>G: chat request
G->>Student: 3 samples, with logprobs
Student-->>G: 3 answers, token logprobs
Note over G: clean, key, agree, one confidence number
alt at or above the threshold
G-->>C: the student's answer
opt audit drawn
G->>Teacher: same request, temperature 0
Teacher-->>G: answer, compared
end
else below the threshold
G->>Teacher: escalate
Teacher-->>G: answer
G-->>C: the teacher's answer
end
```
**Reading this diagram.** Time runs downward. The student is always asked first, three times in one call. Only the gateway looks at the samples: it cleans them, counts agreement and computes one confidence number, and that number alone decides who answers. When the student answers, the teacher may still be called for a random audit (the `opt` block), but in live mode the client never waits for it. When the student does not answer, the teacher's answer goes to the client and is stored as training data if it is valid.
The student: Nano plus your adapter
The teacher you rent
The student () answers first. The gateway samples it several times, scores the samples, and compares the score with a threshold that belongs to the version that answered and, since per-task calibration, to the task.
1. **Sample.** The router asks the student for `RATCHET_SAMPLES` answers (default 3) at temperature 0.7 with token logprobs on. These sampling settings are one constant, `confidence.STUDENT_SAMPLING`, shared by production and both golden evaluators, so the eval scores the student exactly the way production does.
2. **Check each sample.** Each answer is cleaned (`tasks.clean_answer`: reasoning stripped, a code fence around JSON unwrapped) and turned into a key by the task's spec (`Task.key`). A sample gets **no vote** when it does not parse, misses a required field, has a label outside the task's vocabulary, is empty, or did not end by itself (a `finish_reason` other than `stop` or `tool_calls`, so a cut-off answer never counts). If no sample is valid, confidence is 0 and nothing is served, whatever the threshold, even 0.
3. **Count agreement.** Valid samples are compared pairwise with the task's field comparators (`Task.match`). Each sample's votes are the valid samples it matches, itself included. The majority is the first sample with the most votes, and `agreement` is its votes divided by the number of samples. With a tolerance, agreement is not always "all equal": under a number field with `abs_tol` 1, the answers 0, 0.6 and 1.2 all match 0.6, so 0.6 is served with agreement 1.0.
4. **Read token probability.** Among the majority's samples with the top vote count, the gateway picks the one with the highest token probability, `p = exp(mean token logprob)`, from the logprobs vLLM returns.
5. **Combine.** `confidence = 0.4 × p + 0.6 × agreement`. If the endpoint returned no logprobs, confidence is the agreement alone.
6. **Compare.** If the answer is valid and confidence is at least the threshold, the student's answer is served (route `student`), and a random share of such answers is audited by the teacher. Otherwise the request escalates to the teacher (route `confidence`), whose answer is served and, if valid, stored as training data with its provenance.
```mermaid
flowchart TB
S["3 student samples"] --> K["Clean each answer and compute its key"]
K --> V{{"Any valid sample?"}}
V -->|no| Z["Confidence 0 escalate"]
V -->|yes| A["Agreement majority votes / samples"]
A --> P["Token probability p of the best majority sample"]
P --> C["Confidence = 0.4 × p + 0.6 × agreement"]
C --> T{{"At least the threshold?"}}
T -->|yes| Serve["Serve the student's answer"]
T -->|no| Esc["Escalate to the teacher"]
class S,Serve student
class V,T gate
class Z,Esc teacher
```
**Reading this diagram.** Read it top to bottom. The first check is validity: with no valid sample, confidence is 0 and the threshold is never consulted. Otherwise two signals are computed, agreement among the samples and the token probability of the best agreeing sample, and merged with weights 0.6 and 0.4. One comparison with the version's threshold for this task ends the path in either "serve" (the student, blue) or "escalate" (the teacher, orange).
A worked toy example (arithmetic only, not a result): three samples that all agree, best token probability 0.9, give `0.4 × 0.9 + 0.6 × 1.0 = 0.96`. If only two of three agree, the same sample scores `0.4 × 0.9 + 0.6 × 2/3 = 0.76`. Agreement carries more weight than probability on purpose.
### Why one signal is not enough
Small models are poorly calibrated: a 4B model can put a high probability on a wrong answer. Token probability alone does not separate right from wrong, and agreement alone cannot tell three confident samples from three lucky ones. Ratchet combines them, because a single fixed bar was not enough.
Validity is the third signal. A label outside the task's vocabulary, or a JSON answer missing a field, is not a vote at all, so the student cannot "agree with itself" on an answer the task would refuse.
### Precision and coverage, simply
Two numbers describe any threshold:
- **Precision**: of the answers the student would serve, the share that agree with the teacher () under the task's comparators.
- **Coverage**: of all requests, the share the student would serve. Coverage is where the savings come from.
Raising the threshold usually serves fewer answers at higher precision. Usually, not always: one confidently wrong answer at the top can make precision fall as the threshold rises (which rules out some textbook methods).
The numbers below are made up to show the arithmetic. They are not results from Ratchet.
Take 20 golden rows. The ten **calibration** rows, sorted by the student's confidence, with whether its answer agreed with the teacher:
| Confidence | Agrees? | Served at this bar | Agreeing among served | Precision at this bar |
|---|---|---|---|---|
| 0.98 | yes | 1 | 1 | 100% |
| 0.95 | yes | 2 | 2 | 100% |
| 0.93 | yes | 3 | 3 | 100% |
| 0.90 | yes | 4 | 4 | 100% |
| 0.88 | yes | 5 | 5 | 100% |
| 0.85 | no | 6 | 5 | 83% |
| 0.80 | yes | 7 | 6 | 86% |
| 0.72 | no | 8 | 6 | 75% |
| 0.65 | yes | 9 | 7 | 78% |
| 0.55 | no | 10 | 7 | 70% |
The lowest bar where precision is still at least 97% is **0.88**. Now grade it on the ten **held-out** rows: confidences 0.99, 0.96, 0.94, 0.91 and 0.89 are at or above 0.88, and one of those five (the 0.89) disagrees with the teacher. So on held-out data this toy version has **coverage 50%** (5 of 10) and **precision 80%** (4 of 5), not the 100% the calibration half showed. That gap is why Ratchet never grades a threshold on the rows it was tuned on, and why this toy version would be rejected by the golden gate (80% is below the 95% floor). With only five served rows, its certified lower bound would also be weak: 34% (exact Clopper–Pearson at delta 0.05, from `ratchet/stats.py`).
## The threshold, per version
Each trained adapter version carries its own threshold in its eval (`versions.eval.threshold`). Night School sets it the night the version is trained, in `cascade_eval` ([`ratchet/nightschool/evaluate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/evaluate.py)):
1. **Score every golden row** with the new adapter, sampled exactly as in production, under the generation conditions stored with the row. Each row becomes a point: (confidence, agreed with the teacher's stored answer). A row whose stored reference is no longer valid under the task's spec is skipped (`golden_invalid_refs`), and so is a row of a task that is pinned now (`excluded_pinned`). A row the student endpoint refuses (HTTP 400, 413 or 422) counts as confidence 0 and wrong, because production would escalate it.
2. **Split by position.** Even-indexed scored rows form the calibration half; odd-indexed rows form the held-out half.
3. **Calibrate on the even half.** By default (`RATCHET_CALIBRATION=point`), `calibrate_threshold` sorts the points by confidence, highest first, walks down in groups of equal confidence (ties included before checking) and keeps the lowest confidence at which cumulative precision is still at least the target: 0.97 (`calibration.DEFAULT_TARGET_PRECISION`). If no confidence reaches it, the result is infinity.
4. **Clamp.** `threshold = min(1.01, max(0.5, calibrated))`. The floor 0.5 keeps the student from serving coin-flip answers. Scores are at most 1, so **1.01 means "serve nothing"** (`SERVE_NOTHING`); it is a finite number so the eval stays valid JSON. Since v0.5 the threshold is stored exactly, so the router serves at the threshold the eval measured.
5. **Measure on the odd half.** Coverage and precision at that threshold, plus the certificate, the frontier, the per-task bounds and the per-row outcomes (`test_vector`). These held-out numbers are what the [release gate](/guides/release-gate/) reads.
```mermaid
flowchart TB
T["Teacher answers stored as examples"] --> H{{"Hash of the request"}}
H -->|"about 90%"| Train[("Training set")]
H -->|"about 10%"| G[("Golden set never trained on")]
G -.->|"seen 3 or more times"| Train
G --> Sc["Score each row with the new adapter"]
Sc --> E["Even rows calibration half"]
Sc --> O["Odd rows held-out half"]
E --> Thr["Pick the threshold"]
Thr --> M["Measure precision and coverage"]
O --> M
class T teacher
class H gate
class Train,G store
class Sc student
class M ok
```
**Reading this diagram.** Every teacher answer is first split by a hash of the request, so no model's opinion decides which rows are held out. The golden rows are scored with the new adapter and then split again by position: the even half picks the threshold and the odd half measures it, so the grade comes from rows the threshold never saw. The dotted arrow is the one exception: a request seen three or more times moves from golden to training.
Which requests are golden is decided by a hash of the request (`RATCHET_GOLDEN_FRACTION`, default 0.10), which acts like a coin flip: it does not look at difficulty or at any model's answer. A request seen 3 or more times (`FREQUENT`) moves to training and leaves the golden set, so a frequent question is never held out forever.
Golden rows are teacher answers to escalations, audits and arbitrations, so they lean towards requests the student found hard. Precision measured on them means agreement with Nemotron 3 Ultra, not ground truth.
The router reads the threshold on every request with a dictionary lookup (`Ratchet.threshold`). Thresholds differ between versions because each adapter's confidence scale is its own; that is also why the ledger replay never pools two versions.
## Per-task thresholds
Several tasks can share one adapter: Scout's triage and reader calls next to support tickets, for example. One threshold calibrated over the pooled rows then serves the hardest task below its target (the easy tasks' right answers dilute its wrong ones) and holds the easy tasks back when the hard task pushes the bar up. So `cascade_eval` also writes one record per task, `eval['tasks'][name]`, split by the same global even/odd positions:
| Situation | The task's threshold |
|---|---|
| At least `RATCHET_MIN_TASK_GOLDEN` (20) calibration rows | Its own, calibrated to its spec's `policy.target_precision` (default 0.97), never below 0.5 (`calibrated: true`) |
| Fewer calibration rows | The higher of its own and the version's global threshold, so a few lucky rows cannot lower it (`calibrated: false`) |
| No calibration rows | The version's global threshold |
| No held-out row | 1.01: the teacher answers it until a night measures it |
| No golden row at all | No record: the version's global threshold |
| Disabled by the release gate | 1.01 for the life of that version |
When a task's record serves nothing, the router does not call the student for it at all (route `task_disabled`): its samples could never be served, so calling would only add cost and latency. With one task the record equals the global numbers.
Per-task calibration is on by default. `RATCHET_PER_TASK_CALIBRATION=0` makes Night School write no records, so new versions keep one pooled threshold, as in v0.4. Per-task calibration holds each task to its own target, at the price of the student serving somewhat less than with one pooled threshold.
### When a task's spec changes
Each per-task record stores the `spec_hash` it was scored under. If an operator changes the task's spec (its fields, its vocabulary, or its `target_precision` or `min_precision`), the hash no longer matches and `calibration.threshold` returns 1.01 for that task: the student is still sampled, but every request of the task escalates (route `confidence`) until a night re-scores the version under the new spec. A night with too few new examples to train still re-scores the live version when its recorded hashes are stale (event `rescored`, or `rescore_failed`, after which the task stays on the teacher and the next night tries again).
## Certified thresholds and Learn-then-Test
Every eval also carries a **certificate**: on the held-out half, how many answers the version would serve, how many of those agree with the teacher, and an exact one-sided Clopper–Pearson lower bound on its precision (`RATCHET_CERT_TARGET` 0.95, `RATCHET_CERT_DELTA` 0.05). [Evidence you can check](/guides/evidence/) explains how to read and recompute it.
The default point rule picks the threshold and the certificate checks it afterwards. With `RATCHET_CALIBRATION=ltt` (opt-in), the threshold itself is chosen so that its precision is certified, by **Learn-then-Test**:
1. Take a fixed grid of thresholds, decided in advance: 0.95, 0.90, 0.85, down to 0.50 (`stats.CERT_GRID`).
2. From the top, test "precision of the calibration answers at or above this threshold is at most the target" with an exact binomial p-value.
3. Keep going while that test is rejected at `delta`; stop at the first threshold where it is not.
4. Serve at the lowest threshold that passed, or serve nothing (1.01) if the first one already fails. The held-out certificate then checks the choice independently.
Testing a fixed order and stopping at the first failure spends the error budget once, so the chosen threshold has precision above the target with probability at least `1 − delta`. The grid must be fixed in advance: a grid built from the observed confidences would start with a one-row group, which cannot pass, and a grid picked after looking at the data breaks the guarantee.
Learn-then-Test is opt-in because it is conservative early on. Ten calibration rows cannot certify 95%, since `0.95^10 = 0.60`, far above 0.05. In the toy table above only two calibration rows are at 0.95 or more, so Learn-then-Test would serve nothing.
## Watching the confidence scores over time: the drift alarm
A threshold is set once per version, on a fixed golden set. Real traffic can change afterwards: new kinds of requests, a new customer, a changed prompt. The student's confidence scores often move before anyone has labels to show a drop in quality, so Ratchet also watches them without labels.
For each (version, task) pair the gate keeps a **conformal test martingale** over the stream of confidence scores the student produced while it served. In plain words: each new score is ranked against the scores seen so far. If the traffic is unchanged, the ranks look random and the running evidence stays near 1. If the scores start to shift, the evidence grows. When it passes `RATCHET_DRIFT_ALARM` (default 100), the alarm fires. A stream whose traffic never changes fires with probability at most 1/100 over its whole life.
```mermaid
flowchart TB
L[("Ledger: the confidence of each served answer")] --> R["Rank each score against earlier ones"]
R --> M["Test martingale per version and task"]
M --> Q{{"Evidence above 100?"}}
Q -->|no| W["Keep watching"]
Q -->|yes| D["Write a drift event"]
D --> C["Suspend the version's certificate"]
D -.->|"opt-in"| B["Raise the audit rate"]
D -.->|"opt-in"| N["Start Night School earlier"]
class L store
class Q gate
class C danger
class B,N outline
```
**Reading this diagram.** Data flows top to bottom from the ledger, never from the request path. The alarm has one fixed effect, the certificate marked suspended, and two opt-in ones on dotted arrows that are off by default. No arrow leads to a promotion, a rollback or a threshold change: a person or the next night decides what to do.
Keeps data
A check
Certificate suspended
Opt-in, off by default
What an alarm does and does not do:
- It writes one `drift` event and marks the version's certificate as suspended, because the certificate was measured on traffic that may no longer look like the live traffic.
- With `RATCHET_DRIFT_AUDIT_BOOST` above 1 (default 1, off), the alarmed version's audit rate is multiplied by that factor while it serves, up to 0.5. With `RATCHET_DRIFT_EARLY_NIGHT=1` (default 0), Night School needs half the usual number of new examples to train.
- It never promotes, rolls back or changes a threshold by itself. It runs in the gate's tick, reads the ledger and never touches the request path.
`GET /ratchet/drift` reports each stream and the chance that any stream alarms by luck (with K streams, at most K/100). A version holds at most 17 streams: 16 task streams and one shared stream for every later task name, so a client that sends many task names cannot grow memory. The code is in [`ratchet/drift.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/drift.py) (`Stream`, `DriftMonitor`), fed by `ReleaseGate._tick_drift`.
## What can go wrong
| Failure | What the system does | Where you see it |
|---|---|---|
| The student endpoint returns no logprobs | Confidence is the agreement alone | `ratchet.confidence` in the response |
| No valid sample | Confidence 0, the teacher answers | route `confidence` |
| The task's spec changed after the version was scored | Every request of the task escalates until a night re-scores | route `confidence`, events `rescored` or `rescore_failed` |
| A task's record serves nothing | The student is not called for it | route `task_disabled` |
| The confidence scores of live traffic shift | A `drift` event and a suspended certificate; nothing is rolled back | `GET /ratchet/drift`, the event log |
| Too few calibration rows for Learn-then-Test | It serves nothing (1.01) until there is more data | per-task `threshold` in `GET /ratchet/stats` |
## The hosted fallback's fixed threshold
When the student's GPU endpoint is off (`RATCHET_STUDENT_BASE` unset), a trained version's calls go to an untuned, hosted model instead: Nemotron 3 Nano 30B on Token Factory (). In the restaurant, a different cook from the agency takes the easy orders, and the menu says so: the response's `model` field names the model that wrote the answer, though `served_by` still says `student`.
That model never sat the exam, so no calibrated threshold or per-task record applies to it. `Ratchet.threshold` returns the fixed `Settings.confidence_threshold` (`RATCHET_CONFIDENCE_THRESHOLD`, default 0.80) for it, and its answers are **not covered by the adapter's certificate**. The same 0.80 is used for a version whose eval has no calibrated threshold.
- **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)).
- **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)).
- **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)).
- **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)).
- **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate.
- **The nightly schedule on live traffic.** Night School runs by hand today; the timer is scripted.
- **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline.
- **A threshold calibrated on a live ledger.** Night School's one Nebius run calibrated one on 458 golden rows, imported rather than served. The scoring, calibration and certificate code is tested offline with mock models. The text-to-SQL adapter, trained on a borrowed GPU, sets a similar gate on real Nano 4B outputs from its average token probability; its measured numbers are on [Results](/guides/results/).
Run every script with `DRY_RUN=1` first.
## For engineers
| What | Where |
|---|---|
| The score, validity, the sampling constant, `SERVE_NOTHING` | [`ratchet/confidence.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/confidence.py): `score`, `Scored`, `STUDENT_SAMPLING`, `majority` (the `n` choices) |
| Point calibration | `confidence.calibrate_threshold` |
| The golden eval | [`ratchet/nightschool/evaluate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/evaluate.py): `cascade_eval`, `_task_record`, `LiveEvaluator` |
| Per-task records and spec changes | [`ratchet/calibration.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/calibration.py): `threshold`, `held_back`, `spec_changed`, `keep_disabled`, `target_precision` |
| The threshold on the request path | [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py): `Ratchet.threshold`, `Ratchet._sample`, `Ratchet._held_back` |
| Certificate and Learn-then-Test | [`ratchet/certificate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/certificate.py), [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py): `ltt_fixed_sequence`, `CERT_GRID`, `cp_lower` |
| Tests | `tests/test_core.py`, `tests/test_calibration.py`, `tests/test_contract.py`, `tests/test_nightschool.py` |
Settings (read in [`ratchet/config.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/config.py), documented in [Configuration](/reference/configuration/)):
| Variable | Default | Effect |
|---|---|---|
| `RATCHET_SAMPLES` | `3` | Student samples per request and per golden row (1 to 16) |
| `RATCHET_CONFIDENCE_THRESHOLD` | `0.80` | Used only for the hosted fallback student and for a version without a calibrated threshold |
| `RATCHET_GOLDEN_FRACTION` | `0.10` | Share of new examples hashed into the golden set |
| `RATCHET_PER_TASK_CALIBRATION` | `1` | `0`: one pooled threshold for new versions |
| `RATCHET_MIN_TASK_GOLDEN` | `20` | Calibration rows a task needs before its threshold may go below the global one |
| `RATCHET_CALIBRATION` | `point` | `ltt`: Learn-then-Test over the fixed grid |
| `RATCHET_CERT_TARGET` | `0.95` | The precision a certificate must show, and Learn-then-Test's target |
| `RATCHET_CERT_DELTA` | `0.05` | The certificate's error rate |
| `RATCHET_DRIFT_ALARM` | `100` | Drift alarm threshold per stream; `0` turns the alarm off |
| `RATCHET_DRIFT_AUDIT_BOOST` | `1.0` | Audit-rate multiplier after a drift alarm (`1` is off) |
| `RATCHET_DRIFT_EARLY_NIGHT` | `0` | `1`: a drift alarm halves the new examples Night School needs |
| `RATCHET_AUDIT_RATE` | `0.05` | Share of served student answers the teacher audits (mock mode raises it to at least 0.10) |
Constants in the code: weights 0.4 (token probability) and 0.6 (agreement); student temperature 0.7; calibration target 0.97 (a task's `policy.target_precision` overrides it); floor 0.5; `SERVE_NOTHING` 1.01; `CERT_GRID` 0.95 down to 0.50 in steps of 0.05.
Where to see it:
- Every chat response carries `ratchet.confidence` and `ratchet.served_by`; `system_fingerprint` changes with the version and the threshold the answer was held to.
- `GET /ratchet/stats` lists, per task, the threshold the router applies, the held-out precision and coverage, `calibrated` and `disabled` ([API](/reference/api/)).
- `GET /ratchet/versions/{id}/certificate` serves the signed and the latest certificate.
- `GET /ratchet/replay?threshold=0.4` replays a version's logged traffic at another threshold ([Evidence you can check](/guides/evidence/)).
- Routes in `requests.route`: `student`, `confidence`, `cold_start`, `task_disabled`, `n_choices` (the student was sure, but fewer than the client's `n` samples agreed).
- `GET /ratchet/drift` reports the drift alarm's streams.
- Events: `rescored`, `rescore_failed`, `task_disabled`, `drift`.
```bash
python -m ratchet.stats plan --target 0.95 --delta 0.05 # 0 errors: 59 rows, 1: 93, 2: 124, 3: 153
curl -s localhost:8000/ratchet/versions/2/certificate # the certificate behind a version's threshold
```
The full rules are in [Architecture §6](/reference/architecture/#6-confidence-and-calibration).
## Common questions
### What if the student is confidently wrong?
It happens: the target is 97% agreement on the calibration half, not 100%. Three things catch it. A random share of served answers (`RATCHET_AUDIT_RATE`, 5% by default) is re-answered by the teacher, and those audits feed the canary's rollback rule and the live alert ([the release gate](/guides/release-gate/)). An audit's teacher answer is also stored as a training example when it is valid, so the next night learns from what the student got wrong. And the threshold is re-set for every new version on rows it never trained on.
### Why sample three times? Isn't that three times the cost?
It triples the student's output tokens and adds a little latency. Ratchet accepts that as a small cost next to one teacher call. A judge model scoring every request was rejected because it costs a teacher-sized call every time.
### Can I set the threshold by hand?
Not for a trained version. Thresholds change only through Night School's held-out calibration. `RATCHET_CONFIDENCE_THRESHOLD` applies only to the hosted fallback and to a version without a calibrated threshold. The ledger replay can suggest a lower threshold, but its suggestion is advice and nothing applies it. The runbook says never to edit a version's threshold by hand.
### Does "precision" mean the answer is correct?
No. It means the student's answer agrees with Nemotron 3 Ultra's under the task's comparators. If the teacher is wrong, a student that copies it scores well. Precision is also measured on golden rows, which lean towards hard requests.
### What about free-text tasks with no fields?
A task without fields compares the whole answer: its canonical JSON when it is JSON, otherwise its text with reasoning removed, lower-cased and whitespace collapsed. That is strict, so free-text tasks escalate more. The live evaluator can add a judge score (Nemotron 3 Super) for such tasks, which is reported in the eval but does not gate anything.
### What happens before any version exists?
Nothing is scored. On a cold start there is no version, so every request goes to the teacher (route `cold_start`) and its answers become the first training and golden examples.
## Related
---
# Escalation and audits
Source: https://docs.ratchet.dualmindlab.tech/guides/escalation-and-audits/
Escalation sends a request to the teacher whenever the student cannot serve it safely, and audits send a random few of the student's own answers to the teacher as well, so Ratchet always has fresh, unbiased evidence of how often the student agrees with the teacher.
**Escalation** is the apprentice saying "I'm not sure, chef". The apprentice cooked the dish three times and the plates did not match, or the stove broke, or this dish was taken off the apprentice's list. The **head chef** cooks it instead. The guest gets the head chef's plate, and the dish goes into the **recipe book**, so the apprentice can study it tonight.
**Audits** are the **inspector**. The inspector does not wait for the apprentice to ask for help. Now and then, at random, the inspector takes a plate the apprentice already sent out and asks the head chef to cook the same dish. Do the two plates match? The guest has already eaten, so nothing changes for them. But the inspector's tally is how the restaurant knows whether the apprentice is really as good as the exam said.
Why random? If the inspector only tasted plates the apprentice was unsure about, the tally would only show the hard dishes. Tasting at random shows how the apprentice does on the dishes it actually serves.
One more thing: the inspector compares the apprentice with the head chef, not with a perfect cookbook. If the head chef gets a dish wrong, the inspector will not notice.
## How it really works
### When a request escalates
```mermaid
flowchart TB
A(["Request reaches the cascade"]) ==> B{{"1 · Version picked?"}}
B ==>|yes| C{{"2 · Task held back?"}}
C ==>|no| D{{"3 · Breaker open?"}}
D ==>|no| E["4 · Sample the student 3 times"]
E ==> F{{"5 · Valid, and at or above the threshold?"}}
F ==>|yes| G{{"6 · Enough agree for n choices?"}}
G ==>|yes| H["Student serves"]
H --> I{{"Audit drawn?"}}
I -->|yes| J["Teacher checks the served answer"]
B -->|no| T["Teacher answers the route says why"]
C -->|yes| T
D -->|yes| T
E -->|failed| T
F -->|no| T
G -->|no| T
class A neutral
class B,C,D,F,G,I gate
class E,H student
class T,J teacher
```
**Reading this diagram.** Follow one request down the thick path; the numbers are the order the checks run in. Each hexagon is a reason the student might not answer, and every way off the thick path ends at the teacher, with a route in the ledger that names the reason (the table below lists them). Only checks 5 and 6 happen after the student has produced something: confidence and `n` choices. If the student does serve, the only thing left is the audit draw, which sends a random few of its answers to the teacher as well.
A check
The student
The teacher, with the route
Every teacher box after the version check is an escalation: a version was picked, but the teacher answers (`escalated` is true in the row and the response). The route says why:
| Trigger | Route | Student called? |
|---|---|---|
| Confidence below the version's threshold for this task | `confidence` | Yes |
| No valid sample: every sample empty, cut off, unparseable or invalid under the task's spec (confidence 0) | `confidence` | Yes |
| The task's spec changed since the version was scored: its threshold is 1.01 until a night re-scores it | `confidence` | Yes |
| Confident, but fewer than `n` samples agree for a request with `n` choices | `n_choices` | Yes |
| The student call failed (a vLLM 404 for the adapter also queues a reload) | `student_error` | Yes, it failed |
| The student's circuit breaker is open | `student_breaker` | No |
| The release gate disabled the task on this version, or it had no held-out rows | `task_disabled` | No |
The teacher also answers, without an escalation, on a cold start (`cold_start`), while an operator has the student turned off (`student_disabled`), and on the teacher-only paths (`pinned`, `unknown_task`, `tools`, `multimodal`). See the full [routes table](/guides/request-journey/#every-route).
If the gateway's daily budget is spent under `RATCHET_BUDGET_MODE=student_only`, an escalation is refused with a 429 `gateway_budget_exceeded` before any teacher call; confident student answers still go out.
### What the teacher's answer is used for
1. **It is served.** The client gets the teacher's answer, cleaned: reasoning removed (`` blocks and the like) and, for a structured task, bare JSON. The teacher answers under the serving parameters: the client's temperature, or 0 when it set none (0.7 when the client asked for several choices).
2. **It is checked before it may become training data** (`Ratchet._store_example`). It is stored only when all of these hold: no storage opt-out applies; it is the primary teacher's answer (a fallback teacher's only with `RATCHET_STORE_FALLBACK_TEACHER=1`); it was not written under label-altering client sampling (`logit_bias`, a non-zero penalty, a temperature above 1, a vendor field); its `finish_reason` is `stop` or `tool_calls`; it is not empty once reasoning is stripped; and it is valid under the task's spec (it parses, every required field is present, a label is in its vocabulary). Anything else is still served, but skipped, with a `skip_example` event on the 1st, 10th, 100th and 1000th skip of each reason.
3. **It is redacted, then checked again.** Emails, phone numbers, Luhn-checked card numbers, IBANs, secrets and similar identifiers are replaced with placeholders such as `[EMAIL]` in the messages, the answer and the stored conditions. The teacher saw the original. If a placeholder breaks a structured answer, it is skipped (`invalid after redaction`).
4. **It is stored with its provenance** (`Ledger.add_example`): the teacher's model id, the license that allows training on it and its URL, the number of placeholders, the conditions it was written under (`response_format`, `tools`, `tool_choice`, `stop`), the API client and the end user's hash. Its split, golden or train, is fixed from a hash of the task and messages: about 10% golden (`RATCHET_GOLDEN_FRACTION`). A repeat of the same request bumps its `seen` count instead (a client's retry does not). For `n` choices only choice 0 is stored.
5. **It labels the student's own answer.** When the student had a valid answer and the teacher wrote its answer under the audit's conditions (one choice, no client sampling), the row records whether they agreed (`student_agree`). The ledger replay uses it.
6. **It is the shadow's reference.** If a version is in shadow, its answer for this request is compared with the teacher's.
The student's own answers are never stored as training targets, and pinned, tool and multimodal requests store nothing.
### Audits: what they are
When the student serves, one random draw (`rng.random()`, exactly one per served answer) decides an audit. The probability is:
- `RATCHET_AUDIT_RATE`: 0.05 by default; mock mode raises it to at least 0.10;
- for answers a **canary** serves, `RATCHET_CANARY_AUDIT_RATE` when set (audit focusing: the canary's evidence arrives sooner);
- an operator's runtime rate, when set, replaces both (`PUT /ratchet/admin/settings`, or `python -m ratchet.admin flag audit_rate 0.2`).
The rate the answer was drawn at is logged as `requests.label_p` (1.0 for a student answer that escalated). That makes every teacher label's probability known by design, so the ledger replay and the savings receipts can weight audited rows by `1 / label_p`.
If drawn, the teacher answers the same request under the audit's parameters: temperature 0, without the client's `top_p`, `seed`, penalties, `logit_bias` or logprobs (`_audit` → `_call_teacher(role="audit")`). Then:
- **The verdict.** `task.agree(student answer, teacher answer)` compares them on the task's fields. For `ticket-triage` that is `category` and `priority`; a task without fields compares whole answers, as canonical JSON or as lower-cased text with whitespace collapsed. The row gets `audited = 1` and `audit_agree` 1 or 0.
- **A training example too.** The audit's teacher answer goes through the same storage rules as an escalation's, so the money spent on the check also buys a label.
- **A better baseline.** The request's all-teacher baseline in the ledger (`baseline_cost_usd`) becomes what the audit actually cost, instead of the token-formula estimate. Audited rows are how the savings receipts calibrate that estimate for unaudited ones.
- **Not an audit** when the teacher's answer is cut off, empty or invalid for the task: it is priced, but measures nothing (`audited = 0`).
Agreement with Nemotron 3 Ultra, not ground truth. If the teacher is wrong and the student copies it, the audit says "agree". Precision everywhere in Ratchet means this agreement.
### How audits drive the release gate
Audits are the release gate's live signal. The gate counts, per version, the audits of answers that version served ([`ratchet/release.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/release.py), defaults from `ReleasePolicy`):
| Decision | Rule (default) |
|---|---|
| Canary moves from 10% to 50%, then to live | At least `min_audits` (20; 15 in mock mode) audits **in the current stage**, and audited precision at or above `live_floor` (0.92) |
| Canary holds | Enough audits, but precision below 0.92 without enough evidence to roll back |
| Canary is rolled back | At least 20 audits and the 95% Wilson upper bound of audited precision below 0.92. Traffic returns to the live version, or to the teacher on a cold start |
| Opt-in sequential rule (`RATCHET_SEQUENTIAL=1`) | Rollback when the e-value against "precision at least 0.92" over every audit of the canary's life reaches 20, with no minimum count; a stage with 400 audits and no decision retires the canary (`inconclusive`) |
| Live version alert | The e-value against "precision at least 0.92" over its audits since it went live reaches 100 (`alpha_alert` 0.01): one `alert` event, no automatic action |
| Per-task alert | The same rule per task for the live version and the canary: one `alert` per version and task |
Only audits with a verdict count: a pending off-path audit counts in neither the number of audits nor the precision. See [The release gate](/guides/release-gate/) for the whole state machine.
An operator can hold releases by setting the runtime audit rate to 0: no new canary starts, because it could neither advance nor roll back (setting 0 during a canary needs `force`). In live mode a configured rate below 1% is a startup warning, for the same reason.
Human feedback is different. Verdicts sent to `POST /v1/ratchet/feedback` are reported and alerted on, but never promote or roll back a version: clients report what they notice, so that sample is biased. Audits are random by design.
### Audits off the request path
An audit is a whole teacher call, seconds against a few hundred milliseconds for the student. In live mode the client should not wait for it, so the audit runs after the response on a small worker pool, the sidecar:
```mermaid
sequenceDiagram
autonumber
participant C as Client
participant G as Gateway
participant Q as Sidecar
participant Teacher as Teacher
participant Ledger as Ledger
Note over G: the student serves, an audit is drawn
G->>Ledger: row, audited 1, verdict pending
G-->>C: answer, overhead_pending
G->>Q: audit job
Q->>Teacher: same request, temperature 0
Teacher-->>Q: answer
Note over Q: agree? valid?
Q->>Ledger: the example, if valid
Q->>Ledger: the verdict and the cost
Note over Teacher,Ledger: the gate counts only rows with a verdict
```
**Reading this diagram.** The client gets its answer at step 2, before the teacher is called at step 4. The row is written first with the verdict marked pending (1), then the sidecar fills it in once (6, 7). Because the gate counts only rows with a verdict, a request whose audit has not finished yet cannot move a release in either direction.
### When the teacher or the student is down
Every upstream call goes through a circuit breaker. The student and the teacher each have their own, and the shadow has a third, so a failing candidate cannot close the live student's path.
```mermaid
stateDiagram-v2
direction LR
[*] --> Closed
Closed --> Open: 5 failures in a row
Open --> HalfOpen: after the wait, a probe
HalfOpen --> Closed: probe succeeds
HalfOpen --> Open: probe fails, wait doubles
state "Half open" as HalfOpen
class Closed ok
class Open danger
class HalfOpen pending
```
**Reading this diagram.** While a breaker is closed, calls go through, and a success resets the count of failures. Five failures in a row open it: calls are refused at once, without waiting for a timeout. After a wait, one probe call is let through. A good probe closes the breaker, a bad one reopens it with twice the wait. For the student the wait starts at 30 s and doubles up to 300 s. The teacher's breaker opens after 5 failures too, and answers requests that need the teacher with a 503 `teacher_circuit_open` and `Retry-After`.
Calls go through
Calls refused at once
One probe
What the request gets when its teacher is not available:
```mermaid
flowchart TB
A(["An escalation needs the teacher"]) --> B{{"Teacher answers?"}}
B -->|yes| OK["Serve the teacher's answer"]
B -->|"no: retries failed or breaker open"| C{{"Fallback teacher set?"}}
C -->|yes| F["Serve the fallback teacher stored only if asked"]
C -->|no| D{{"RATCHET_ON_TEACHER_FAILURE"}}
D -->|"error, the default"| E["A mapped error 502, 503, 504 or 400, 429"]
D -->|serve_student| G["Serve the student's answer route degraded, never stored"]
class A neutral
class B,C,D gate
class OK teacher
class F,G outline
class E danger
```
**Reading this diagram.** The three outcomes are chosen by the operator, not by the request. The default is an honest error. A licensed fallback teacher (`RATCHET_TEACHER_FALLBACK_MODEL`) can answer instead and keeps its own license in the provenance. Degraded mode serves the student's answer even though it was below the threshold; it is marked in the response (`ratchet.degraded`, `X-Ratchet-Degraded`) and in the route, and is never stored, audited or shadowed. Audits and arbitrations never use the fallback: they fail open as described below. A fallback teacher's answers are stored only with `RATCHET_STORE_FALLBACK_TEACHER=1`; a mapped error is a 502, 503 or 504, or a 400 or 429 when the cause is the client's.
The teacher answers
Opt-in stand-in
An error
### Retries do not pay twice
Clients retry. Two mechanisms keep a retry from costing a second teacher call or from counting as a second sighting of the same example.
```mermaid
sequenceDiagram
autonumber
participant C as Client
participant G as Gateway
participant K as Idempotency cache
C->>G: request, Idempotency-Key abc
G->>K: abc seen for this key?
K-->>G: no
Note over G: run the cascade once
G->>K: keep the answer
G-->>C: answer
C->>G: retry, same key and body
G->>K: abc seen?
K-->>G: yes, same body
G-->>C: the same answer, Idempotent-Replayed: true
Note over C,G: no cascade, no row, no quota
```
**Reading this diagram.** The first request runs the cascade once (1 to 5); the retry gets the kept answer back (6 to 9) with no model call, no ledger row and no quota used. The cache is per API key and lives in memory only, for `RATCHET_IDEMPOTENCY_TTL_S` (600 s by default), because answers contain request text and are never written to disk. A duplicate that arrives while the first is still running waits for the first's answer. The same key with a different body is a 400 `idempotency_key_reused`, and an error is never kept. Separately, a request marked as a retry (`x-stainless-retry-count` above 0) never adds to an example's `seen` count, and the same API client's repeat within `RATCHET_SEEN_DEDUP_S` (300 s live, 0 in mock mode) counts once, so retries cannot push a golden example to the frequent-request rule.
What keeps it honest:
- **The draw stays on the request path**, in the same order (the canary pick, then the audit draw), so the same requests are audited however the work runs.
- **A drawn audit is never dropped.** The last tenth of the queue is reserved for jobs that carry an audit; when even that is full, the audit runs before the response, as in mock mode. Dropping audits under load would leave the gate a sample of quiet hours.
- **Pending is explicit.** The row is logged with `audited` 1 and `audit_agree` NULL, and `Ledger.update_request` writes the verdict once when the job ends. The response says `"overhead_pending": true`, and its `cost_usd` is the serving cost only; the ledger gets the rest.
- **The audit keeps the teacher's full deadline** (180 s). If slow answers, often the hard requests, failed more often, the audited precision would be biased upward.
- **At shutdown** the gateway waits up to `RATCHET_SIDECAR_DRAIN_S` (10 s) for queued jobs, then gives up; those rows stay pending, which every statistic ignores.
Mock mode (0 sidecar workers by default) and Scout keep audits inline.
### Shadow arbitration
While a version is in shadow, it is sampled and scored on every request without serving anything. The reference it is judged against is chosen in this order ([architecture](/reference/architecture/#shadow-reference-and-arbitration)):
1. the teacher's answer, if this request has one (the escalation's, or a counted audit's);
2. otherwise, if the shadow would serve and disagrees with the answer the live student served, the teacher **arbitrates**: it answers the request, that answer becomes the reference, and it is stored as a training example when valid;
3. otherwise the served answer itself, so agreements with the live student are accepted as they are.
Disagreements are always settled by the teacher, or counted against the shadow when they cannot be. Once the shadow would have served 100 answers, it needs precision of at least 0.95 and coverage no more than 2 points below the live student's to move to canary.
### When an audit or arbitration fails
These calls are optional: the client already has an answer.
| Failure | What the client gets | What is recorded |
|---|---|---|
| The audit call raises | The student's answer, as usual | `audited = 0`, `audit_agree` NULL, nothing billed for the call; `audit_errors` + 1; an `audit_error` event on the 1st, 10th, 100th and 1000th failure, naming the error class and status, never the message; a `failures` row |
| The teacher's breaker is open | The student's answer | The audit is skipped, not counted as an error |
| The audit answer is cut off or invalid | The student's answer | Priced, not an audit, not stored |
| The arbitration fails, or is skipped because this request's audit already failed | The live answer | `shadow_agree` NULL: the gate counts it as a disagreement, and the rejection reason says how many |
A failed audit is simply not an audit. Audits are a random sample, so losing one shrinks the sample without biasing it. A failed arbitration is different: arbitration only runs when the shadow disagrees, so dropping those rows would hide exactly the cases where the shadow may be wrong. Counting them against the shadow means a flaky teacher can get a candidate rejected, never promoted.
### What audits cost
Each audit is one extra teacher call at the teacher's price, added to the request's `cost_usd` and `teacher_cost_usd` and listed in `usage_breakdown.audit`. At the default rate that is one teacher call for about every 20 answers the student serves, so audits add roughly 5% of what the teacher would have charged for those answers. A shadow adds its own samples on every request, plus any arbitration. Ratchet counts all of this in its savings; nothing is left out of the bill.
The teacher's list price in `ratchet/config.py` is $1.00 per million input tokens and $3.00 per million output tokens for Nemotron 3 Ultra (Token Factory list price, Oct 2026, unverified until the day-1 checks pass).
## For engineers
| Where | What |
|---|---|
| [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) `Ratchet.handle` | The escalation branches, the audit draw, `label_p`, `student_agree` |
| `Ratchet._audit` | One audit: the call, the verdict, failures (`AuditOutcome`) |
| `Ratchet._call_teacher` | The teacher's answer in the `serve`, `audit` or `arbitrate` role, then `_store_example` |
| `Ratchet._store_example` | The only caller of `Ledger.add_example` for teacher answers: the storage rules, redaction, provenance |
| `Ratchet._shadow`, `Ratchet._arbitrate` | The shadow's samples, its verdict and the arbitration |
| `Ratchet._finish_offpath` | The audit and shadow after the response, then `Ledger.update_request` |
| `router.answer_problem` | Why a teacher answer can be neither an example nor a reference |
| [`ratchet/sidecar.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/sidecar.py) | The bounded queue: `reserve`, `submit`, `drain`, `close` |
| [`ratchet/release.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/release.py) | `ReleaseGate.tick`: the canary, rollback and alert rules over audit counts |
| Setting | Default | Meaning |
|---|---|---|
| `RATCHET_AUDIT_RATE` | 0.05 | Audit probability for a served student answer; mock mode uses at least 0.10 |
| `RATCHET_CANARY_AUDIT_RATE` | unset | A canary's own audit rate, more than 0 and at most 1 |
| `RATCHET_MIN_AUDITS` | 20 (15 in mock mode) | Audits a canary stage needs before it can advance or roll back |
| `RATCHET_LIVE_FLOOR` | 0.92 | Audited precision floor for canary and live |
| `RATCHET_SEQUENTIAL` | 0 | 1 turns on the anytime-valid canary rule |
| `RATCHET_STORE_FALLBACK_TEACHER` | 0 | 1 stores the fallback teacher's answers, under its own license |
| `RATCHET_GOLDEN_FRACTION` | 0.10 | Share of new examples held out as golden |
| `RATCHET_SIDECAR_WORKERS` | 4 live, 0 mock | Workers for audits and shadows after the response |
| `RATCHET_SIDECAR_QUEUE` | 1000 | Jobs that may wait |
| `RATCHET_SIDECAR_DRAIN_S` | 10 | Seconds shutdown waits for queued jobs |
The `ReleasePolicy` fields read `RATCHET_` (see the [release policy settings](/reference/configuration/#release-policy-ratchetreleasepy)). The live-run harness expects the planned gate profile `RATCHET_SEQUENTIAL=1` and `RATCHET_CANARY_AUDIT_RATE=0.5`.
Where to look: each row's `audited`, `audit_agree`, `label_p` and `stored` columns (`GET /ratchet/requests/{id}`); `audit_agreement` per day and each task's audits in `GET /ratchet/stats`; the `evidence` of the canary and the live version (counts, e-value, confidence sequence) in the same payload's `versions` list; events `audit_error`, `skip_example`, `rollback`, `alert`, `sidecar_inline` and `sidecar_dropped` in the release log. `Ratchet.health()` holds the in-process counters (`audit_errors`, `shadow_errors`).
## Common questions
### Is 5% enough to catch a bad model?
It is enough to catch a badly broken one quickly, because a canary only serves 10% and then 50% of traffic and needs at least 20 audits per stage before it can move. To get evidence faster, raise `RATCHET_CANARY_AUDIT_RATE` for canaries only, or turn on the sequential rule, which can roll back after a couple of disagreeing audits.
### What if the teacher is wrong?
Then the audit agrees with a wrong student, or flags a right one. Ratchet cannot see that: every precision number is agreement with Nemotron 3 Ultra. Human feedback (`POST /v1/ratchet/feedback`) is reported next to it so you can spot a gap, but it never moves a release.
### Do audits slow my requests down?
Not in live mode: they run after the response, unless the queue is full, in which case a drawn audit runs before the response rather than being dropped. In mock mode they run inline.
### Why not audit only the answers the student was unsure about?
Those already go to the teacher. The point of an audit is to check answers the student was confident about, at random, so the sample describes what users actually got.
### Can I turn audits off?
You can set `RATCHET_AUDIT_RATE=0`, but then no canary can collect evidence, so nothing is promoted or rolled back on evidence; live mode warns about any rate below 1%. A runtime audit rate of 0 is treated as an operator hold on releases.
### Are audit answers stored even when a client opted out of storage?
No. Under an opt-out the audit still runs (release safety does not depend on storage) and its verdict counts, but nothing is stored.
### Has the audit path run against Token Factory?
Not yet. The teacher itself runs on Nebius Token Factory, but no trained student has served on a live ledger, so there has been nothing to audit. As of Oct 29, 2026 the audit path is tested offline (Night School's one run on Nebius, by hand on 28 Oct, graded a student on golden rows and rejected it; nothing was served), including the off-path sidecar, which a test compares row for row with an inline run.
## Related
---
# The ledger
Source: https://docs.ratchet.dualmindlab.tech/guides/ledger/
The ledger is one SQLite file that logs every request's metadata, keeps every stored teacher answer with its provenance (teacher, license, hash, redactions), and records every adapter version and release decision, so the training data behind an adapter can be checked later.
The ledger is the kitchen's **recipe book**, plus an **order log**.
The order log gets one line for every order: when it came in, who cooked it, how sure the apprentice was, and what it cost. It never says what the customer asked for.
The recipe book only gets a page when the **head chef from the agency** cooks a dish. The page says what was ordered, what the head chef cooked, and a note: "this chef's contract lets you copy this recipe." Before the page is written, a clerk blacks out phone numbers, email addresses and card numbers. (The clerk does not recognise people's names yet, and the page says how many things were blacked out.)
The **apprentice chef's** own dishes are never copied into the book. A customer can also say "please don't keep my order": the dish is cooked as usual, but no page is written. And a page can be torn out on request: a stub with a fingerprint stays behind, so the same recipe is never written again.
## How it really works
### One request on its way into the ledger
1. **The request arrives.** The gateway validates it and works out whether it may be stored at all. If the request names an end user (`user` or `safety_identifier`), that value is turned into a keyed hash at once (HMAC-SHA256 under the deployment's key, scoped to the API key's name, 16 hex digits). The raw value goes no further. The broadest storage opt-out that applies is settled here (see [opt-outs](#storage-opt-outs-and-no-train-end-users) below).
2. **The cascade runs** (student first, teacher when needed; see [the request journey](/guides/request-journey/)). Whatever happens, the request gets **one row in `requests`**: task, route, who served it, which version, confidence and the threshold it was held to, the audit and shadow flags, costs (with the student and teacher split), latency, the API key's *name*, the end user's hash, the opt-out reason and whether anything was stored. There is no message text and no answer text in this row.
3. **No teacher answer, nothing more.** When the student served the request and the teacher was not called (no audit drawn, no shadow disagreement to settle), the request row is all that is written. The student's text is not kept anywhere.
4. **Every teacher answer goes through one function**, `Ratchet._store_example`. That covers escalations, cold starts, audits and shadow arbitrations. It is the only caller of `Ledger.add_example` on the request path (feedback corrections and imports have their own single writer, `feedback.store_label`). It asks again whether an opt-out applies, so an end user who opted out while the request waited for the teacher is still kept out.
5. **The answer is checked.** It is skipped (served, but not stored) when its `finish_reason` is not `stop` or `tool_calls`, when it is empty once reasoning is stripped, when it is not valid under the task's spec (a missing field, a label outside the vocabulary), or when the client sent sampling that could choose the label (`logit_bias`, a non-zero penalty, a temperature above 1, a vendor field). Each skip writes a throttled `skip_example` event that names the task and the reason, never the answer.
6. **The text is redacted.** With `RATCHET_REDACT_PII=1` (the default) the messages, the answer and every string in the stored conditions get `[LABEL]` placeholders. The answer is checked once more: if a placeholder broke a structured answer, it is skipped as `invalid after redaction`. The teacher always saw the original request.
7. **The example is stored with its provenance**: the teacher model that wrote it, that teacher's license and license URL, the number of placeholders, the conditions it was answered under (`response_format`, `tools`, `tool_choice`, `stop`), the client and the end user's hash. Its golden or train split comes from a hash of the task and the redacted messages. An identical request bumps `seen` instead of adding a row (a client's retry does not count), and a hash that an erasure tombstoned is refused.
8. **Later, other parts write their own rows.** Night School reads `examples` every night and writes `versions` (with each adapter's bill of materials, signed when a key is set) and `version_leaves`. The release gate moves versions and writes `events`. Retention, erasure and receipts write their own tables.
### The privacy path, in one picture
```mermaid
flowchart TB
A(["A chat request"]) --> B{{"Did the teacher answer?"}}
B -->|no| M[("requests row metadata only")]
B -->|yes| C{{"Opt-out, pinned or tool request?"}}
C -->|yes| M
C -->|no| D{{"Complete and valid?"}}
D -->|no| M
D -->|yes| R["Redact the messages, answer and conditions"]
R --> E[("examples row with provenance")]
E --> S{{"Split by hash"}}
S -->|"train, or seen 3 or more times"| T["night/train.jsonl"]
S -->|golden| G["Golden eval only"]
class A neutral
class B,C,D,S gate
class M,E store
class R neutral
```
**Reading this diagram.** Start at the top with one chat request. Every path writes the metadata row; the arrows into it show the three ways a request ends there and nowhere else: the teacher never answered, an opt-out or a pinned or tool request applies, or the teacher's answer was incomplete or invalid. Only a request that passes all three checks reaches redaction, and only then does text enter the ledger. The last split is by a hash of the redacted text, so it never depends on any model's opinion.
### The main tables
```mermaid
flowchart LR
EC["example_clients"] --> EX
TS["tombstones and erasures"] --> EX
LEAF["version_leaves committed sets"] --> EX
LEAF --> VER
FB["feedback verdicts"] --> EX
FB --> REQ
NT["no_train end-user hashes"] -.-> REQ
EX[("examples redacted text")]
REQ[("requests metadata")] --> VER[("versions adapters, BOMs")]
REQ --> RU["daily_rollup folded sums"]
REQ --> RC["receipts sealed days"]
VER -.-> EV["events release log"]
class REQ,EX,VER store
class RC,LEAF ok
class TS danger
```
**Reading this diagram.** Cylinders are the three core tables, boxes the others, and arrows the column that links them (`request_id`, `example_hash`, `version_id`, `student_version`). `requests` is the busiest table and holds no text; `examples` is the only table with text, redacted. Three tables are special: `version_leaves` and `receipts` (green) can only be appended to, and `tombstones` (red) remembers erased hashes so they can never be stored again. Retention folds old request rows into `daily_rollup`, and each ledger day is sealed into `receipts`. Dotted arrows are not stored links: `no_train` is looked up for each request, and every move of a version is logged in `events`.
Core table
Append-only
Erased hashes
Not a stored link
### The evidence tables, append-only
Two tables exist so that someone else can check Ratchet's claims later. `version_leaves` holds, for each adapter, one leaf per training and golden example (the example's hash and a hash of its label, teacher, license and redaction count). The adapter's signed BOM carries the Merkle roots over those leaves, so a leaf can be proven to be in, or absent from, the set the adapter was trained on. `receipts` holds one signed savings receipt per ledger day and API client, sealed once the day is over and chained to the previous one. SQLite triggers refuse UPDATE, DELETE and a replacing INSERT on both tables, on every connection, because a changed row would silently break the roots and chains built from it. [Evidence you can check](/guides/evidence/) shows how to verify them offline.
### Human labels: feedback and imports
Teachers are not the only source of a stored answer. Every chat response carries `ratchet.request_id`, and the caller can send `POST /v1/ratchet/feedback` with a verdict on one of its own requests (any other id is the same 404 as an unknown one).
```mermaid
flowchart TB
V(["A verdict on a request id"]) --> K{{"Which verdict?"}}
K -->|"correct, or incorrect with no answer"| R["Counted toward human agreement, nothing stored"]
K -->|"incorrect, with the right answer"| G{{"Opt-out, pinned task or corrections off?"}}
G -->|yes| R
G -->|no| P["Check against the spec, redact, hash"]
I(["python -m ratchet.importer"]) --> A["Attested human labels, or an open teacher's outputs"]
A --> P
P --> S[("store_label writes the example, with provenance")]
class V,I neutral
class K,G gate
class R outline
class S store
```
**Reading this diagram.** A verdict is never a label: "correct" stores nothing, because that would be the student's own answer coming back as a training target. Only a correction, a human's own answer, can become an example, and it goes through the same checks and redaction as a teacher's answer. The importer enters at the same point with an attestation from the operator, or with an open teacher's outputs, and refuses closed-model families. A stronger label replaces a weaker one for the same prompt (teacher, then a client's correction, then an operator's import). Human corrections are stored with the provenance `human`; an open teacher's imported outputs keep that teacher and its license. Verdicts feed each task's human agreement and each version's human precision, which are reported and alerted on, and never promote or roll back a version.
A check
Counted, nothing stored
Stored as an example
### Where each kind of data lives, and who can read it
```mermaid
flowchart TB
A1["An API key its own lookups, receipts, feedback"]
B1["A dashboard key stats, cost, BOMs, redaction preview"]
C1["An admin key erase, export, no-train, proofs, all receipts"]
D[("The ledger file, private requests: metadata examples: redacted text")]
A1 --> D
B1 --> D
C1 --> D
class D store
```
**Reading this diagram.** The three groups at the top are access levels, from narrowest to widest. An API key sees only what it sent. The dashboard key reads aggregates and never writes. Only an admin key can export, erase or open proofs. All three read from the same private file, which is why the file and its backups must be protected as much as the keys.
### What is stored
| Table | One row per | Holds text? |
|---|---|---|
| `requests` | Gateway call | No. Metadata only: task, route, version, confidence, threshold, costs, flags, key name, end-user hash, opt-out reason |
| `examples` | Distinct teacher-labelled request | Yes, **redacted**: the prompt, the teacher's answer and the stored conditions, with teacher, license, license URL, split, `seen`, redaction count, client, end-user hash and `origin` |
| `versions` | Adapter | No. Base model, adapter name, train size, dataset hash, eval, state, traffic share and the adapter's signed BOM |
| `events` | Release decision or notable event | No. A kind (`trained`, `skip`, `canary`, `rollback`, ...) and a short detail; never request text or keys |
| `version_leaves` | Example in an adapter's training or golden set | No. The example hash and a hash of its label, teacher, license and redaction count; append-only |
| `receipts` | Ledger day and API client | No. A signed savings receipt over request metadata; append-only |
| `spend` | Spend outside requests | No. Scout's runs and Tavily credits keep `usd` 0 here (their model calls are already in `requests`) |
| `failures` | Failed call | No. The stage, the exception's class and status, what it spent |
| `example_clients`, `no_train`, `tombstones`, `erasures`, `erasure_items`, `daily_rollup`, `feedback` | Governance records | No. Hashes, key names, counts and flags |
The `origin` of an example says where it came from. Teacher answers from the gateway are recorded without one and read as `live`; human labels carry `feedback` (a correction) or `import` (an attested file). Each example's provenance travels into Night School's rows and from there into the adapter's [signed BOM](/guides/licenses/), so the license of every training target can be checked offline.
### What is never stored
Teacher answers and human labels only. Training targets are teacher answers or customer-attested human labels, never the student's own answers.
| Never stored | Why | Where it is enforced |
|---|---|---|
| The student's answers, as training targets or at all | Training on a model's own outputs causes collapse | Only `_store_example` (teacher answers) and `feedback.store_label` (attested human labels) may call `add_example`; a test fails on any other caller |
| Pinned tasks' answers | Open-ended tasks (Scout's planner and writer) stay on the teacher for good | `_teacher_only` never calls `add_example`, and Night School leaves out rows of a task that is pinned now |
| Tool requests, multimodal requests | A tool request is answered by the teacher alone (route `tools`) until tool-call distillation exists | |
| Opted-out requests | The client, its key, the task, the end user or the operator said so | `Ratchet.store_opt_out`, asked again before every write: opt-outs only reduce storage |
| Incomplete or invalid teacher answers | A cut-off or malformed answer would teach the wrong thing | `Ratchet._store_example` checks `finish_reason` and the task's spec first |
| Answers written under label-altering client sampling | A client may choose its own answer, not the label everyone's student learns | `GenParams.unstorable`, threat model T2 |
| Raw end-user identifiers, API keys | Privacy and key safety | Key names and keyed hashes only |
| Request text in `requests`, `failures`, logs or `/metrics` | Metadata is enough to run the system | `requests` holds metadata only; logs and metrics carry ids, names and numbers |
A fallback teacher's answers are stored only with `RATCHET_STORE_FALLBACK_TEACHER=1`, and then under that model's own license. A request for `n` choices stores only the teacher's choice 0.
### PII redaction
`ratchet/redact.py` replaces identifiers with placeholders before anything is stored. It runs in a fixed order, so the specific formats with checksums win over the broad phone pattern:
| Placeholder | Matches | Extra check |
|---|---|---|
| `[SECRET]` | API secrets such as `sk-…`, `AKIA…`, `ghp_…`, `tvly-…`, `xox…-…`, and the token after `Bearer ` | |
| `[EMAIL]` | `name@domain.tld` | |
| `[IBAN]` | Country code, 2 check digits, then groups | mod-97 |
| `[CARD]` | 13 to 19 digits with optional spaces or dashes | Luhn |
| `[AADHAAR]` | 12 digits, first digit 2 to 9 | Verhoeff |
| `[SSN]` | `123-45-6789` style, invalid areas excluded | |
| `[PAN]` | Indian PAN, `ABCPE1234F` style | |
| `[IP]` | IPv4 addresses | |
| `[PHONE]` | Optional country code and area code, digit groups | 10 to 15 digits |
For example, this text:
```text
Hi, I am Priya. Email priya.k@example.com, phone +91 98765 43210, card 4111 1111 1111 1111, order 1234.
```
is stored as:
```text
Hi, I am Priya. Email [EMAIL], phone [PHONE], card [CARD], order 1234.
```
with `redactions` = 3. The name stays: **names and street addresses are not detected yet**. That needs a named-entity model (Presidio is the planned option), which would add a large dependency and latency, so v0 uses patterns. Redaction covers message content (strings and text parts), `name`, an assistant's `refusal`, tool and function call arguments, and every string value in the stored conditions. It is **on by default**; `RATCHET_REDACT_PII=0` stores examples verbatim. You can test it on your own text with `POST /ratchet/redact` or the dashboard's **Try the redactor** box; nothing is stored by the preview.
By default a placeholder is just the label (`[EMAIL]`, `RATCHET_REDACT_STYLE=label`). Extraction tasks need to know which of two addresses an answer names, so a task can ask for another style (`privacy.redact_style` in its spec, or the setting for all tasks): `numbered` writes `[EMAIL_1]` and `[EMAIL_2]`, one number per distinct value across the messages, the conditions and the answer; `surrogate` writes reserved look-alike values such as `person1@example.com`. Indian GSTINs and UPI ids are matched too (as `[GSTIN]` and `[UPI]`), and `RATCHET_REDACT_PLUGIN` (`module:function`) can add a detector for what patterns cannot find, such as names. If the plugin fails when an example is being stored, nothing is stored.
Two consequences worth knowing. The example hash, the golden split and the frequent-request count are computed on the redacted text, so two requests that differ only in an email address become one example. And the student learns to write a placeholder where the teacher repeated an identifier from the request; for label tasks that does not matter, for free-text replies it can.
### Storage opt-outs and no-train end users
Five opt-outs, broadest first. The first one that applies is logged in `requests.store_opt_out` and returned in the response as `ratchet.store_opt_out`:
| Reason | Set by |
|---|---|
| `global` | `RATCHET_STORE_EXAMPLES=0` (zero retention; Night School then skips every night) |
| `task` | `store: false` in the task's spec, or `RATCHET_NO_STORE_TASKS` |
| `key` | `store=0` for the key's name in `RATCHET_KEY_LIMITS` |
| `request` | `store: false`, the header `X-Ratchet-Store: none` (or `0`, `false`), or `metadata.ratchet_store` set to `"false"` (or `"0"`) |
| `end_user` | A no-train registration: `POST /v1/ratchet/no-train` with `{"end_user": "..."}` |
An opted-out request is handled exactly like any other: the student may serve it, audits still run and shadows are still arbitrated, so release decisions never depend on what was stored. Only the write to `examples` is skipped, and a duplicate does not bump `seen`. Nothing in a request can undo an opt-out: `store: true` is the default spelled out. An unknown header value is a 400, so a typo cannot store by accident. Ratchet's default is to store, the opposite of OpenAI's `store` default, because distilling is its purpose.
A no-train registration keeps only the hash, scoped to the caller's API key name, so the same identifier from two clients is two people as far as the ledger can tell. An admin key can register an end user for every client at once (scope `*`), which also covers keys added later. `DELETE /v1/ratchet/no-train` removes a registration (an API key cannot remove one an admin made). Events record counts and names, never the end user or its hash.
### Retention, tombstones, erasure and export
Opt-outs keep new data out; these remove or hand over what is there.
- **Tombstones, not deletes.** A purge sets an example's `messages`, `completion` and `params` to NULL and keeps the row: hash, task, split, `seen`, teacher, license and client stay. That matters because Night School's rules read more than text: the frequent-request count adds `seen` over every variant of a prompt, and the new-data count compares with past versions. A purged row is in neither training set, but its `seen` still counts, so a purge never moves another variant of a frequent request back to golden.
- **Retention windows.** `RATCHET_EXAMPLE_RETENTION_DAYS` purges examples not seen again for that many days (reason `retention`; a later storable duplicate brings the text back). `RATCHET_REQUEST_RETENTION_DAYS` folds old request rows into `daily_rollup` sums and deletes them; every statistic adds the sums back, so figures stay the same except the count of distinct end users. Rows that a release in flight still needs as evidence are never folded. Both default to 0 (keep forever). Retention runs at most hourly in the live gateway on a background thread, at the start of every live night, and on demand with `python -m ratchet.live purge`.
- **Erasure is final.** `privacy.erase` selects by `client`, `end_user`, `hashes` or `contains` (each narrows the scope), purges the text and the end-user hash, and writes each hash to `tombstones`: `add_example` refuses it from then on, from any client. The record (`erasures`, `erasure_items`, an `erase` event) holds hashes, counts, names and a redacted reason, never text. `RATCHET_ERASURE_SHARED` decides what happens to an example another client also sent (`delete`, the default, or `keep` and unlink). Admin erasures are dry runs unless asked otherwise.
- **Adapters are reported, never edited.** An erasure lists `affected_versions`: exactly, from a version's committed training set when it has one; otherwise every version created after the example was stored while it counted as a training row. Their signed BOMs stay as written. Ratchet cannot remove what an existing adapter memorised; the next night trains without the purged rows, and retiring an affected version is the operator's decision.
- **Export.** `GET /ratchet/admin/export/{kind}` (admin key) or `python -m ratchet.admin export` streams NDJSON: examples that still have text with their provenance and every client that sent them, request metadata, versions with their BOMs, events, erasures. End-user hashes are left out.
An erased example goes through a short life cycle:
```mermaid
stateDiagram-v2
[*] --> Stored: a teacher answer
Stored --> Purged: retention
Purged --> Stored: a duplicate
Stored --> Erased: erased by client, end user, hash or text
Purged --> Erased: erased
note right of Erased
add_example refuses
the hash from any client
end note
class Stored ok
class Purged pending
class Erased danger
```
**Reading this diagram.** A purge is reversible in one way only: if the same request arrives again and may be stored, its text comes back. An erasure is not. From the moment the hash is a tombstone, no client can store it again, and the row keeps only hashes and counts. The adapters that may have learned from it are reported, but their signed BOMs are never rewritten.
Text kept
Text purged, may come back
Erased for good
### What can go wrong
| Failure | What the system does | Where you see it |
|---|---|---|
| A teacher answer is cut off, empty or invalid for the task | Served, not stored | `skip_example` event, `requests.stored = 0` |
| A placeholder breaks a structured answer | Served, not stored | `skip_example` with `invalid after redaction` |
| The redaction plugin raises at store time | Nothing is stored for that request | `skip_example`, the request still succeeds |
| An erased request is sent again | Served as usual, never stored | `requests.stored = 0` |
| A name or street address is in the text | It is stored as written (not detected by patterns) | Use an opt-out, a retention window or a detector plugin |
| Two gateway processes share one ledger | Not supported; caches and the gate are per process | Run one worker |
| Power cut with `RATCHET_DB_SYNC=NORMAL` | The last commits may be lost, the file is intact | Use `FULL` if that matters |
| Something tries to change a leaf or a receipt | The trigger refuses it | A SQLite error |
### SQLite, WAL and one process
The ledger is a single SQLite file (`RATCHET_DB`, default `ratchet.db`; mock mode uses an in-memory ledger) behind the `Ledger` class. Every write commits at once. One re-entrant lock serialises writes, because Scout's agents call the router from several threads over one connection.
A file ledger runs in SQLite's WAL journal with `synchronous=NORMAL` (`RATCHET_DB_SYNC=NORMAL`, the default). Readers no longer block the writer, dashboard reads use a second, read-only connection, and the gateway and a `python -m ratchet.live` cron job on the same file only contend writer against writer for one commit. The price: a power cut or OS crash can lose the last commits (the file is not corrupted). WAL needs every process on the same host with the file on a local disk; never put `RATCHET_DB` on the bucket mount or another network filesystem (if you must, set `RATCHET_DB_SYNC=FULL` for every process).
Run the gateway with **one uvicorn worker**. The ledger's caches and the release gate are per process, and SQLite is not designed here for several writers. A Postgres ledger is planned.
### Backups
The ledger is the source of truth for provenance, versions and training data. Back it up daily with SQLite's backup API, never `cp`: in WAL mode recent commits live in `ratchet.db-wal` until a checkpoint, and a copy of the main file alone misses them.
```bash
sqlite3 /var/lib/ratchet/ratchet.db ".backup '/var/backups/ratchet-$(date +%F).db'"
```
In the Nebius VM kit, `deploy/vm/backup.sh` does the same with an integrity check, keeps 14 copies and can upload them to Object Storage. To restore: stop the gateway and cron jobs, remove any `ratchet.db-wal` and `ratchet.db-shm`, copy the backup over `RATCHET_DB`, start the gateway, then reload the live adapter. A backup taken before an erasure still holds the erased text, so run every later erasure again after a restore. See the [runbook](/reference/runbook/#backups).
## For engineers
### Code
| File | What to read |
|---|---|
| [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py) | `SCHEMA` (every table), `MIGRATIONS`, `INDEXES`; `Ledger.add_example` (dedup, split, tombstone refusal, retries), `_split`, `log_request`, `purge_examples`, `record_erasure`, `fold_requests`, `export_page`, `bill_of_materials` |
| [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) | `Ratchet._store_example` (the checks, redaction and provenance), `Ratchet.store_opt_out` |
| [`ratchet/redact.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/redact.py) | `PATTERNS`, `redact`, `redact_messages`, `redact_json` |
| [`ratchet/privacy.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/privacy.py) | `OPT_OUTS`, `EndUsers.hash`, `erase`, `apply_retention`, `export_lines` |
| [`ratchet/feedback.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/feedback.py) | `store_label`, the only writer of human and imported labels |
Schema changes add the column to `SCHEMA` **and** `MIGRATIONS`; an old file is migrated on open. `version_leaves` and `receipts` are append-only: triggers refuse UPDATE, DELETE and a replacing INSERT on every connection.
### Settings
| Variable | Default | Meaning |
|---|---|---|
| `RATCHET_DB` | `ratchet.db` | The ledger file (live mode) |
| `RATCHET_DB_SYNC` | `NORMAL` | `NORMAL`: WAL with `synchronous=NORMAL`. `FULL`: rollback journal with `synchronous=FULL` |
| `RATCHET_GOLDEN_FRACTION` | `0.10` | Share of new examples assigned to the golden split (above 0, at most 0.5) |
| `RATCHET_REDACT_PII` | `1` | Redact before storing; `0` stores verbatim |
| `RATCHET_REDACT_STYLE` | `label` | `label`, `numbered` or `surrogate` placeholders |
| `RATCHET_REDACT_PLUGIN` | empty | `module:function` detector for names and addresses |
| `RATCHET_STORE_EXAMPLES` | `1` | `0` is zero retention: nothing stored, Night School skips |
| `RATCHET_NO_STORE_TASKS` | empty | Comma-separated tasks whose answers are never stored |
| `RATCHET_KEY_LIMITS` | empty | Per-key settings by key name; `store=0` keeps a key's requests out |
| `RATCHET_END_USER_KEY` | empty | HMAC key for end-user hashes (16+ characters). Empty: a random key is generated once and kept in the ledger's `meta` table |
| `RATCHET_EXAMPLE_RETENTION_DAYS` | `0` | Days an example is kept after it was last seen; 0 keeps it |
| `RATCHET_REQUEST_RETENTION_DAYS` | `0` | Days a request row is kept before folding; 0 keeps it |
| `RATCHET_ERASURE_SHARED` | `delete` | `delete` or `keep` for examples another client also sent |
| `RATCHET_SEEN_DEDUP_S` | 300 s live, 0 mock | A repeat from the same client within this window counts once toward `seen` |
| `RATCHET_STORE_FALLBACK_TEACHER` | `0` | `1` stores a fallback teacher's answers, under its own license |
| `RATCHET_FEEDBACK_CORRECTIONS` | `1` | `0` keeps verdicts but stores no human correction |
Full list: [Configuration, privacy and security](/reference/configuration/#privacy-and-security).
### Endpoints
| Endpoint | Access | What |
|---|---|---|
| `GET /ratchet/bom` | admin or dashboard key in live mode; public with `RATCHET_PUBLIC_BOM=1` | Ledger-wide bill of materials: counts by teacher, license and split, `examples_sha256`, `purged_examples` |
| `GET /ratchet/versions/{id}/bom` | as above | One adapter's signed BOM |
| `POST /ratchet/redact` | dashboard | Preview redaction; stores nothing |
| `POST` and `DELETE /v1/ratchet/no-train` | API key (admin for every client) | Register or remove a no-train end user |
| `DELETE /v1/ratchet/examples` | API key (admin for any client) | Erase one end user's stored examples |
| `POST /ratchet/admin/privacy/erase` | admin | Erase by client, end user, hashes or text; dry run by default |
| `GET /ratchet/admin/export/{kind}` | admin | NDJSON export of `examples`, `requests`, `versions`, `events` or `erasures` |
Details: [End users and storage](/reference/api/#end-users-and-storage) and [Erasure and export](/reference/api/#erasure-and-export).
### CLI
```bash
python -m ratchet.admin erase --reason "contract ended" --client acme # dry run: counts and affected versions
python -m ratchet.admin erase --reason "contract ended" --client acme --apply # do it
python -m ratchet.admin export examples --client acme --out acme.ndjson
python -m ratchet.live purge --dry-run # what the retention windows would remove
```
### Events
`skip_example` (a teacher answer not stored, with the reason), `no_train`, `no_train_unmatched`, `end_user_key`, `erase`, `purge`, `purge_failed`. Meanings are in the [runbook](/reference/runbook/).
## Common questions
### Does Ratchet keep my users' prompts?
Only the prompts of requests the teacher answered (an escalation, a cold start, an audit or an arbitration), only when no opt-out applies, and only after redaction. Requests the student served without an audit leave a metadata row and nothing else.
### Is the student's answer ever stored?
No. Not as a training target, and not as text anywhere. Training on the student's own outputs would make it collapse onto its own mistakes.
### What about names and street addresses?
They are not caught by the redactor yet. If that matters for your traffic, use an opt-out for those requests or tasks, set a retention window, and erase anything that leaked ([PII leak playbook](/reference/runbook/#pii-leak)). Named-entity detection is planned.
### A customer asks us to delete their data. Can we?
Yes. Erase by client or end user; the text is gone for good and its hash can never be stored again. The adapters that may have been trained on it are listed, but Ratchet cannot make an existing adapter forget. The next night trains without the rows, and taking an affected version out of service is your decision. Backups made before the erasure still hold the text.
### Can I run two gateway workers on one ledger?
No. Run one gateway process with one worker. Night School's cron job may share the file on the same host. A Postgres ledger for several workers is planned, not built.
### Does request retention change my statistics?
No, except the count of distinct end users. Folded rows become daily sums that every statistic adds back, and rows a release in flight still needs are never folded.
### Can someone tell whether a text was stored or erased?
Not from a live gateway's public endpoints: the training-set proof endpoints need an admin key, and a keyless call to the erasure endpoint gets its counts as `null`. Whoever holds both the database file and a candidate text can test it against the stored hashes, which is also what makes erasures checkable. Keep the file and its backups private, and set `RATCHET_END_USER_KEY` outside the file.
## Related
More detail: the [threat model](/reference/threat-model/) and the [architecture's data model](/reference/architecture/#4-data-model-sqlite).
---
# Night School
Source: https://docs.ratchet.dualmindlab.tech/guides/night-school/
Night School is the nightly job that trains from the base model on every teacher answer stored so far, grades it on a sealed golden set it never studied, and hands it to the release gate.
Every night after the restaurant closes, the **apprentice chef** goes to **night school**.
The apprentice does not just read today's new pages. They study the **whole recipe book** from the start, and write a brand new **notebook of techniques**. Starting fresh each night means yesterday's mistakes do not pile up.
Some recipes are kept in a **sealed exam** envelope. The apprentice never studies them. After studying, the apprentice takes the exam: half of the exam decides how sure the apprentice must be before serving a dish, the other half grades them. One rule bends this: a dish customers order **three or more times** is too important to keep sealed, so it moves into the study pile.
Night school is skipped when there is nothing new to learn, when the new apprentice from last night is still on a **trial shift**, or when the manager says "no class tonight". If the apprentice passes the exam, they start their trial shift. If not, the notebook goes in the bin.
Training rows and the sealed golden rows, split by a hash of each request.
A fresh adapter on every training row so far, never on student answers.
Even rows set the threshold, odd rows it never saw grade it.
A bill of materials, signed when a key is set, then shadow, canary or rejected.
## How it really works
One night is `run_night()` in [`ratchet/nightschool/orchestrate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/orchestrate.py). In live mode a cron job starts it with `python -m ratchet.live night`. Each check below can end the night early with a `skip` event in the release log.
```mermaid
flowchart TB
start(["02:30 · ratchet.live night"]) --> pre{{"Paused, storage off, or a release in flight?"}}
pre -->|yes| skip(["skip event nothing trained"])
pre -->|no| build["Build the dataset golden split by hash, seen 3+ times: train"]
build --> enough{{"50+ new training rows and golden rows?"}}
enough -->|no| skip
enough -->|yes| champ["Re-score the live version on tonight's golden set"]
champ --> train["Train a fresh LoRA from the base, every row"]
train --> evalx["Golden eval: even rows calibrate, odd rows grade"]
evalx --> admit{{"Sign the BOM, then gate.admit"}}
admit -->|"below the bar"| rej["rejected"]
admit -->|"live version exists"| sh["shadow"]
admit -->|"cold start"| can["canary 10%"]
class pre,enough,admit gate
class skip outline
class start,build,champ,evalx neutral
class train,sh,can student
class rej danger
```
**Reading this diagram.** Two checks can end the night before any GPU time: the first (training paused, storage turned off, or a version still in shadow or canary) and the data check (fewer than 50 new training rows, or no golden rows yet). The dataset step is where the golden split happens: a row's hash puts about 10% of examples in the sealed golden set, and the FREQUENT rule moves a request seen 3 or more times into training. The live version, the champion, is graded again on tonight's golden rows before the challenger trains, so both are compared on the same data. The new adapter is trained from the base model on every training row, calibrated on the even golden rows and graded on the odd ones it never saw. The release gate's `admit` decides what happens next. Not drawn: the retention purge right after the pause check, the Merkle commitment before training, and the paperwork a live night writes back to the bucket.
1. **Training paused?** If an operator turned on `training_paused` (`POST /ratchet/admin/training/pause` or `python -m ratchet.admin flag training_paused on`), the night writes `skip` (`training paused by an admin; no training tonight`) and stops before anything else.
2. **Retention first.** In live mode, if a retention window is set, expired example text is purged now, so the night never reads text that should be gone ([the ledger](/guides/ledger/)).
3. **Storage off?** With `RATCHET_STORE_EXAMPLES=0` (zero retention) the night skips (`storage disabled`). Examples kept from before are not trained on. One thing still runs when no release is in flight: a task whose spec changed is re-scored on the stored golden rows, so it does not stay on the teacher for good.
4. **Release in flight?** If any version is in `shadow` or `canary`, the night skips (`vN still in shadow; no training tonight (saves GPU time)`). A candidate trained now could not be released anyway, and the GPU time would be wasted.
5. **Build the dataset** (`dataset.build`): the training rows and the golden rows, with the license check and every rule in [Rows left out of training](#rows-left-out-of-training) below.
6. **Enough new data?** New examples = training rows now, minus the most training rows any past version was trained on that are still stored (`Ledger.train_rows_kept`, so a deletion neither stops nor starts training). Below **50** (`min_new_examples`), the night skips (`only N new examples ... no training tonight`). With no golden rows at all, it skips too (`no golden examples yet`). A skipped night still re-scores the live version if a task's spec changed since it was scored.
7. **Re-score the champion.** The live version is evaluated again on **tonight's** golden set, under tonight's task specs, and its eval is overwritten. The candidate will be compared with this score, not with an old one (champion against challenger on the same data).
8. **Commit to the rows.** The training and golden sets are each turned into a Merkle root (`dataset.commitment`), before the paid job starts, so the signed BOM can later prove what was and was not trained on ([evidence](/guides/evidence/)).
9. **Train.** `trainer.train(train_rows, base_model, "student-vN")` trains a new adapter **from the base model** on **every** training row so far. In live mode the wall-clock time is recorded as a `training_job` compute interval for the cost report, also when training fails.
10. **Register the candidate.** A `candidate` version is created and its committed leaves are written once (append-only).
11. **Golden eval.** The candidate is sampled on each golden row exactly as production samples it (3 samples at temperature 0.7 with token logprobs). The confidence threshold is calibrated on the **even** rows (target precision 0.97), and precision and coverage are measured on the **odd** rows it never saw. The eval also carries an exact precision certificate, and, when a version is live, a row-by-row comparison with tonight's re-score. Free-text tasks also get a judge score from , recorded as `judge_pass_rate`; it does not gate the release.
12. **Sign the BOM.** The adapter's bill of materials is written from the exact rows: sources and licenses, the base model's license, the NOTICE.md text, the eval, the set roots. It is signed with Ed25519 when `RATCHET_BOM_SIGNING_KEY` is set.
13. **Admit.** A `trained` event is written, then `gate.admit(vN)` decides: **rejected**, **shadow** (a live version exists), or **canary 10%** (cold start, no live version). See [the release gate](/guides/release-gate/).
14. **Ship the paperwork (live mode).** `python -m ratchet.live night` writes the BOM and NOTICE.md into the adapter's folder and runs `RATCHET_SYNC_CMD` again, so the bucket copy of the adapter carries its licenses. A failed sync is a `sync_failed` event and a non-zero exit; the version still stands.
### One live night, step by step
```mermaid
sequenceDiagram
autonumber
participant Night as Gateway host live night
participant Student as Student endpoint
participant Bucket as Bucket at /data
participant Job as Serverless Job, L40S
participant Judge as Judge Super
Note over Night: 02:30: checks, then the rows
Night->>Student: re-score live
Night->>Bucket: night/train.jsonl
Night->>Job: nebius ai job create
Job->>Bucket: student-vN/
Night->>Student: load_lora_adapter
Night->>Student: golden eval
Night->>Judge: free-text rows
Note over Night: candidate, BOM, gate.admit
Night->>Bucket: BOM, NOTICE.md
```
**Reading this diagram.** Everything starts on the gateway host, where cron runs `python -m ratchet.live night` next to the ledger and the release gate. The live version is graded again first, on the student endpoint (1). Then the night uploads the training file to the bucket (2) and starts a Serverless Job (3), which reads it from `/data` and writes the new adapter's folder back (4); `deploy/train_job.sh` waits until the weights are there. The night tells vLLM to load the adapter at runtime (5), grades it with 3 samples per golden row (6) and has the judge score free-text rows (7). It records the candidate, its eval and its signed bill of materials in the ledger and calls the gate. Last, it writes the BOM and the signed `NOTICE.md` into the adapter's folder and syncs the bucket again (8).
The live commands, file paths and polling are on [Training on your data](/guides/training-on-your-data/).
### The golden set: the sealed exam
Golden examples are never trained on. The one exception is a request seen 3 or more times, which moves to training and leaves the golden set.
- **Split by hash.** When an example is first stored, its split is fixed: `golden` if the first 8 hex digits of its hash, divided by `0xFFFFFFFF`, are below `RATCHET_GOLDEN_FRACTION` (default 0.10), else `train`. The hash is over the task and the redacted messages, so the split never changes and nothing leaks from re-splitting.
- **Variants share a split.** The same prompt stored under other conditions (a `stop`, a JSON schema) is another example, but it lands in the same split. The trainer only sees the messages, so a golden prompt stored again in train would be trained on.
- **Never trained on**, with one exception, the FREQUENT rule below. Golden rows are only ever used to calibrate and grade.
- **Half calibrates, half grades.** `cascade_eval` picks the lowest threshold whose served answers reach the target precision on the even-indexed golden rows, and reports precision and coverage on the odd-indexed rows. Each task also gets its own threshold; with fewer than `RATCHET_MIN_TASK_GOLDEN` (20) calibration rows it is never below the global one. Thresholds are clamped between 0.5 and 1.01, and 1.01 means "serve nothing".
- **Human-verified golden rows.** An operator can import a golden set that is held out for good, even past FREQUENT, with every variant of its prompt (`python -m ratchet.importer ... --golden-only`).
The golden set comes from teacher-labelled traffic (escalations, audits, arbitrations), so it leans toward requests the student found hard. Its coverage is a conservative estimate of live coverage, and its precision means agreement with the teacher, not ground truth.
### The FREQUENT rule
A request whose `seen` count reaches **3** (`dataset.FREQUENT`) is treated as a training row even if its hash put it in the golden set, and it leaves the golden set. Without this rule the most common questions, if they happened to hash into golden, would never be learned, and the student would keep escalating them.
`seen` counts real traffic only. It is summed over every variant of a prompt. A client's retry does not count (the SDK's `x-stainless-retry-count`), the same client repeating a request within `RATCHET_SEEN_DEDUP_S` (300 s in live mode) counts once, and an imported label or a correction never adds an occurrence. Purged rows keep their count, so a deletion never moves a frequent request back to golden.
### What LoRA is
**Simply:** the base model, NVIDIA Nemotron 3 Nano 4B, stays frozen. LoRA trains a small set of extra weights that sit beside some of its layers and nudge their output. Those extra weights are the adapter: a few files, cheap to train, quick to load and easy to swap. The adapter is yours; the base stays under NVIDIA's license.
**Precisely:** for a frozen weight matrix W, LoRA learns two thin matrices B and A of rank r, and the layer computes W·x + (α / r)·B·A·x. With rank 16 and alpha 32, the update is scaled by 2. Ratchet's training script, [`ratchet/nightschool/train.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/train.py), uses Transformers, PEFT and TRL's `SFTTrainer` with these defaults:
| Setting | Default | Flag |
|---|---|---|
| Base model | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | `--base` |
| Epochs | 2 | `--epochs` |
| Learning rate | 2e-4 | `--lr` |
| LoRA rank | 16 | `--rank` |
| LoRA alpha | 32 (twice the rank) | follows `--rank` |
| Max sequence length | 2048 tokens | `--max-seq` |
| Batch size per device | 8 | `--batch` |
| Precision | bf16 | |
| Loss | On the answer only (`completion_only_loss=True`) | |
| Target modules | `q_proj, k_proj, v_proj, o_proj` (attention), `up_proj, down_proj` (MLP), `in_proj, out_proj` (Mamba) | `--targets` |
Nemotron-H mixes attention, Mamba and MLP layers. The default targets include the Mamba projections. Whether vLLM loads and applies LoRA on them is checked on day 1; if it refuses, the fallback is `--targets q_proj,k_proj,v_proj,o_proj,up_proj,down_proj` (`TARGETS_ATTENTION_MLP`). The student endpoint is started with `--max-lora-rank 16`, so a higher rank needs that flag raised too.
### Retrain from the base every night
Each night trains a **new** adapter from the **base model** on **all** training rows so far. Adapters are never stacked or continued.
- No drift compounds from one adapter to the next, and each version is one dataset snapshot, named by its `dataset_hash`.
- Replay of old data is built in, which limits forgetting.
- Rollback is simple: the previous adapter is still there, unchanged.
- The cost: training time grows with the dataset. Deduplication, sampling or recency-weighted replay are the later fixes.
Training targets are teacher answers (and customer-attested human labels), never the student's own outputs. Training on its own outputs would make the student collapse onto its own mistakes.
### Train on the prompt the student is served
Nemotron 3 Nano's chat template turns "thinking" on unless told otherwise. The student endpoint serves with thinking off, so its prompt ends in an empty think block. If training rendered the prompt with thinking on, the student would learn answers after a prompt it never sees, and TRL would only log a warning.
So `train.jsonl` uses TRL's conversational **prompt-completion** format:
- `prompt`: the stored request messages.
- `completion`: one assistant message, the teacher's stored answer.
- `chat_template_kwargs`: `{"enable_thinking": false}`, the same value the endpoint uses (`--default-chat-template-kwargs`). A test keeps the three copies of this value equal.
Before the model loads, `train.py` checks every row (`check_rows`) and stops on one that is not prompt-completion, lacks the arguments, holds a think tag or has no provenance. After the tokenizer loads, it renders every row as TRL will and stops when the prompt is not a prefix of prompt plus answer, or differs from what the endpoint renders (`check_prefix`, `TemplateMismatch`). A drift stops the job in seconds instead of training a quietly worse adapter.
### Rows left out of training
`dataset.rows` reads every stored example and applies these rules. The ledger keeps every row; only the training file changes.
| Row | Left out of | Why | Counted as |
|---|---|---|---|
| Purged by retention or erasure | Both sets | No text left; its `seen` still counts toward FREQUENT | |
| Task pinned now | Both sets | Pinned tasks never train the student, even when pinned after rows were stored | `excluded_pinned` |
| Teacher unknown or not distillable | Both sets | Only distillable provenance is used ([licenses](/guides/licenses/)) | |
| Stored with a client's `stop` | Training (stays golden) | `train.py` renders messages only, so the cut answer would teach the bare prompt to stop there | |
| Answer invalid under the task's current spec | Training (stays golden; the eval skips an invalid reference) | The spec tightened since it was stored | `skipped_invalid`, `golden_invalid_refs` |
| Answer holds a literal `` or `` | Training (stays golden) | The template renders such an answer without the empty think block | `skipped_think_tag` |
| Golden-only import, and every variant of its prompt | Training | Held out for good | |
Teacher answers that were incomplete or invalid when they arrived were never stored in the first place ([the ledger](/guides/ledger/)).
### Mock trainer and live trainer
Both implement the same two interfaces, `Trainer.train(rows, base_model, name)` and `Evaluator.evaluate(adapter, golden_rows)`, and both grade with the same `cascade_eval` and the same student sampling (`confidence.STUDENT_SAMPLING`), so mock, live and production agree on how a student is scored.
| | Mock: `sim.mock_models.MockFleet` | Live: `ratchet.live.NebiusTrainer` + `LiveEvaluator` |
|---|---|---|
| Used by | The tests and the mock gateway | `python -m ratchet.live night` |
| Training | Counts each row's template and remembers its texts; no GPU, no weights | Writes `train.jsonl`, syncs it to the bucket, runs a Nebius Serverless Job, hot-loads the adapter into vLLM |
| Evaluation | A deterministic mock student | The real adapter on the student endpoint, plus for free text |
| Bad deploys | `corrupt_on_deploy` breaks a version on purpose, to test the rollback | Real |
### After admit: the release states
Night School ends at `gate.admit`. From there the [release gate](/guides/release-gate/) moves the version on its own, one tick at a time:
```mermaid
stateDiagram-v2
state "canary 10%" as c10
state "canary 50%" as c50
[*] --> candidate: a night trains
candidate --> rejected: golden gate fails
candidate --> shadow: passes, a live version exists
candidate --> c10: passes, cold start
shadow --> rejected: fails after 100 would-serve answers
shadow --> c10: passes after 100 would-serve answers
c10 --> c50: 20 audits, precision ≥ 0.92
c50 --> live: 20 audits, precision ≥ 0.92
c10 --> rolled_back: upper bound below 0.92
c50 --> rolled_back: upper bound below 0.92
live --> retired: a canary is promoted
class candidate neutral
class shadow, c10, c50 student
class live ok
class rejected, rolled_back danger
class retired outline
```
**Reading this diagram.** A candidate is graded on the golden set first: it needs precision of at least 0.95, coverage of at least 0.25 and no regression against the live version's re-score. With a live version it then runs in shadow, answering every request silently, until it has 100 answers it would have served. A canary serves 10%, then 50% of requests; each stage needs 20 audits (15 in mock mode) with audited precision of at least 0.92, and the 50% stage then becomes live while the old live version is retired. A canary whose Wilson upper bound falls below 0.92 is rolled back, and traffic returns to the live version, or to the teacher on a cold start. Not drawn: a shadow or canary is also retired when a newer candidate is admitted (rare, because no night trains during a release), and the opt-in anytime-valid rule (`RATCHET_SEQUENTIAL=1`) decides the canary with an e-value instead. The release gate guide has every rule.
Your adapter on trial
Serving everyone
Stopped
Replaced
- **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)).
- **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)).
- **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)).
- **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)).
Live Night School is tested offline (`tests/test_live.py`, against fake endpoints and a fake `nebius` CLI), and it ran end to end once on Nebius, by hand, on 28 Oct 2026; the golden gate rejected that student, so it was never made live (see [It ran on Nebius](#it-ran-on-nebius)). It has not run nightly on live traffic. The three original adapters (two text-to-SQL, one tool-calling) were trained on a borrowed GPU, outside Night School. On 27 Oct 2026 the round-2 recipe was retrained once, by hand, as a Nebius Serverless Job on one preemptible L40S; the nightly loop did not start it. The day-1 checks (`scripts/day1_check.py`) come first.
## It ran on Nebius
**Measured on Nebius, 28 Oct 2026, by hand, once.** Night School ran end to end once on Nebius, by hand (28 Oct 2026): the gateway on a Nebius CPU VM, the training job on a spot L40S, the adapter served from a Nebius Serverless Endpoint. The golden gate rejected it (81.8% precision against the 95% target), so it was never made live. The live site's own gateway is still on Cloudflare and is not connected to that endpoint, and the loop has not run nightly on live traffic. No saving is claimed.
| What | Result |
|---|---|
| Gateway | A Nebius CPU VM in `eu-west1` (2 vCPU), in live mode |
| Data | 4,322 existing Ultra-labelled text-to-SQL rows imported (no new teacher calls, no exam rows): 3,864 to train on, 458 held out as the golden set |
| Training | A spot L40S Serverless Job: 2 epochs, 966 steps, 1,478 s; loss about 0.15 to 0.08; token accuracy about 0.975 |
| Serving | The adapter (SHA-256 `dc9ad4b9...`) hot-loaded into a Nebius Serverless Endpoint |
| Golden gate | SQL compared as text: agreement 17.9%; at the gate's threshold the student would answer 4.8% of questions with 81.8% precision (9 of 11) against the 95% target. **Rejected**, never shadow, canary or live |
The caveats, all of them, are in the [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius): four job submissions (an accidental on-demand job that was cancelled, a missing `einops`, a save that failed with `EPERM` and whose weights were recovered, then the job that saved); the night's own evaluation timed out at 15 s, so the gate result comes from running the eval and the gate again on the saved adapter with a 120 s timeout, through a one-off driver; five problems fixed by hand on the VM, now fixed in the repository (spot by default, `einops` pinned, the adapter saved locally and then copied, a clear stop when `aws` is missing, a separate student timeout for the night); and cost estimates (about 2.7 GPU-hours, at most about $4 at list prices) where the Nebius bill decides.
## For engineers
### Code
| File | What |
|---|---|
| [`ratchet/nightschool/orchestrate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/orchestrate.py) | `run_night`, `rescore`, `paired_evidence`, `spec_hashes`, the `Trainer` and `Evaluator` protocols |
| [`ratchet/nightschool/dataset.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/dataset.py) | `rows` (every rule above), `FREQUENT`, `effective_split`, `counts_as_train`, `commitment`, `write_training`, `TRAIN_TEMPLATE_KWARGS`, `build` |
| [`ratchet/nightschool/evaluate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/evaluate.py) | `cascade_eval` (calibrate on even rows, measure on odd rows, certificate, frontier, per-task records), `LiveEvaluator`, `REFUSED` |
| [`ratchet/nightschool/train.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/train.py) | The GPU script: `parse_args`, `check_rows`, `check_prefix`, `columns`, `write_metadata`, `versions` |
| [`ratchet/nightschool/notices.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/notices.py) | NOTICE.md rendering |
| [`ratchet/live.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/live.py) | `main` (the `night` command), `NebiusTrainer`, `write_boms`, `load_adapter` |
| [`sim/mock_models.py`](https://github.com/harshbhx/ratchet/blob/main/sim/mock_models.py) | `MockFleet` |
The rules are tested one by one in `tests/test_nightschool.py` (skip reasons, held-out golden rows, variants sharing a split, stop rows held out, cumulative retraining, champion re-score, the certificate, the training rows' template arguments, `train.py`'s loss and checks, the pinned image).
### Settings
| Setting | Default | Meaning |
|---|---|---|
| `min_new_examples` | 50 | Argument of `run_night`; no environment variable |
| `RATCHET_GOLDEN_FRACTION` | `0.10` | Share of new examples that go to the golden split |
| `RATCHET_SAMPLES` | `3` | Student samples per request, also in the golden eval |
| `RATCHET_MIN_TASK_GOLDEN` | `20` | Calibration rows a task needs for its own threshold |
| `RATCHET_PER_TASK_CALIBRATION` | `1` | `0` keeps one pooled threshold |
| `RATCHET_CALIBRATION` | `point` | `ltt` picks the threshold by Learn-then-Test |
| `RATCHET_CERT_TARGET`, `RATCHET_CERT_DELTA` | `0.95`, `0.05` | The certificate's target and error rate |
| `RATCHET_JUDGE_MODEL` | `nvidia/nemotron-3-super-120b-a12b` | Judge for free-text tasks |
| `RATCHET_STUDENT_BASE_MODEL` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | The base every adapter is trained from |
| `RATCHET_BOM_SIGNING_KEY` | empty | Signs each BOM (Ed25519); empty means unsigned with a SHA-256 |
| `RATCHET_TRAIN_GPU_USD_PER_HOUR`, `RATCHET_TRAIN_GPUS` | `1.55`, `1` | Prices the training interval for the cost report. The code's default; these docs use 1.35 USD per GPU-hour, Nebius's list price for an L40S. Set the rate on your bill |
The live-mode variables (`RATCHET_DATA_DIR`, `RATCHET_SYNC_CMD`, `RATCHET_TRAIN_CMD`, `RATCHET_ADAPTER_ROOT`) are on [Training on your data](/guides/training-on-your-data/). The release policy's thresholds are on [the release gate](/guides/release-gate/).
### Commands
```bash
python -m ratchet.live night # one live night (cron)
python -m ratchet.admin flag training_paused on --reason "..." # skip nights until turned off
```
### Events
`skip` (with the reason), `rescored`, `rescore_failed`, `trained`, then the gate's `rejected`, `shadow` or `canary` (and `task_disabled` for a task that fails its own check), and in live mode `sync_failed`. Meanings: [runbook, event kinds](/reference/runbook/).
## Common questions
### Why retrain from scratch every night instead of continuing yesterday's adapter?
Continuing piles up drift and forgetting, and makes rollback harder. Retraining from the base on the whole set means every adapter is one dataset snapshot and can be reproduced. The price is that training takes longer as the data grows.
### What does a night cost?
An estimate from the repo's [deployment guide](/reference/deployment/#4-cost-estimates): an L40S at about $1.35 per GPU-hour on demand ([Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026) for 30 to 60 minutes is about $0.70 to $1.35 per training night. Skipped nights cost nothing. Night School also needs the student endpoint running while it loads and grades the new adapter. Real costs are recorded once it runs: `python -m ratchet.cost report`.
### What happens if training fails halfway?
No version is created. The compute interval is closed with a "training failed" note (the GPU time was still paid), the error ends the command with a non-zero exit, and the next night tries again from scratch.
### Why not train while a canary is running?
The new adapter could not be released until the current one finishes, and by then a newer night would replace it. Skipping saves GPU money.
### Can the student learn its own mistakes?
No. Its answers are never stored. When the teacher audits a served answer, the teacher's answer is stored, not the student's.
### How big must the golden set be?
Big enough for the certificate to mean something. With about 200 golden rows, precision is only known to a few points. `python -m ratchet.stats plan --target 0.95 --delta 0.05` says how many held-out rows a certificate needs (59 with no errors, 93 with one, per the README). Shadow and canary exist partly because the golden set is small.
### Has a real adapter been trained yet?
Three, outside Night School: two text-to-SQL rounds and one tool-calling round, all trained on a borrowed GPU; the round-2 recipe was later retrained once by hand on Nebius AI Cloud (see [Text-to-SQL](/guides/text-to-sql/) and [Results](/guides/results/)). Night School's loop is tested with mock models and its live path offline; its first real night was a hand-run on Nebius on 28 Oct 2026 and the gate rejected it ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)).
## Related
The nightly loop in full: [Architecture, nightly loop](/reference/architecture/#3-nightly-loop-night-school).
---
# The release gate
Source: https://docs.ratchet.dualmindlab.tech/guides/release-gate/
The release gate decides, from held-out golden rows and teacher audits, whether a newly trained adapter may take traffic, how much, and when to send it back, so a bad night never quietly replaces a good student.
After night school, the apprentice has a new notebook. Nobody lets it near customers yet.
1. **The sealed exam.** The new notebook is tested on recipes it never studied. At least 95 of every 100 dishes it chooses to send out must match the head chef's, it must be willing to cook at least a quarter of the orders, and it must not do worse than the current apprentice. To keep that fair, the current apprentice re-takes the very same exam the same night.
2. **The trial shift.** The new notebook cooks silently next to the current apprentice. Nothing it cooks leaves the kitchen. When the two disagree, the head chef decides who was right. After 100 dishes it would have sent out, it either passes or goes back.
3. **Serving a few tables first.** It serves 10% of the tables, then half, then all of them. The inspector tastes a few random plates at each step, and each step needs at least 20 tastings.
4. **Sending the apprentice back to class.** If the tastings show quality clearly below 92 of 100, the new notebook is taken off the line automatically and the previous one cooks again.
"Clearly" matters. If the inspector tasted 20 plates and 17 were right, that is 85%, which looks bad. But 20 tastings is not many; maybe it was a few unlucky plates. So the inspector asks: **giving the apprentice the benefit of the doubt, how good could it really be?** With 17 of 20, that generous ceiling is about 95%, above 92, so the inspector keeps tasting. With 7 right out of 17, even the generous ceiling is 64%. Even being generous, the score is too low: back to class.
The apprentice who already serves every table is never sent back automatically. If the inspector finds a problem there, an alarm goes off and a person decides.
## How it really works
```mermaid
sequenceDiagram
participant N as Night School
participant G as Release gate
participant Ledger as Ledger
participant R as Request path
participant Teacher as Teacher
N->>Ledger: version, eval, BOM
N->>G: admit
Note over G: golden gate, against the re-scored live version
alt passes, a live version exists
G->>Ledger: state shadow
loop each request
R->>G: shadow verdict, nothing served
G->>Teacher: arbitrate a disagreement
end
G->>Ledger: canary 10%
else passes, cold start
G->>Ledger: canary 10%
end
loop each request in the canary
R->>Teacher: audit an answer
Teacher-->>G: agree or not
end
G->>Ledger: 50%, then live, or rolled_back
```
**Reading this diagram.** Night School hands over a trained candidate and the gate does the rest. The first judge is the golden exam, run the same night for the candidate and the current live version. Then real traffic does the judging: first silently in shadow, then in the canary, where the teacher audits a random sample of what the canary served. Every state change is a write to the ledger, which is why the whole history can be replayed from the events table.
A version moves through a small set of states, stored in the ledger's `versions` table. The golden gate, the shadow decision, the default canary rule and every admin action write their moves with a compare-and-set, so two processes deciding at once make each decision only once.
```mermaid
stateDiagram-v2
state "canary 10%" as c10
state "canary 50%" as c50
[*] --> candidate: trained tonight
candidate --> rejected: fails the golden gate
candidate --> shadow: passes, live exists
candidate --> c10: passes, cold start
shadow --> rejected: fails on traffic
shadow --> c10: holds up
c10 --> c50: enough audits
c50 --> live: enough audits
c10 --> rolled_back: clearly below
c50 --> rolled_back: clearly below
live --> retired: replaced
class candidate neutral
class shadow, c10, c50 student
class live ok
class rejected, rolled_back danger
class retired outline
```
**Reading this diagram.** Every version is born as a candidate and ends in one of three resting places: `rejected` (never served anyone), `rolled_back` (served some, then failed its audits) or `retired` (replaced). Only a canary can become live, and the only way out of live is to be replaced: there is no automatic arrow out of `live`, on purpose. "Enough audits" means at least 20 in the stage with audited precision at or above 0.92; "clearly below" means the Wilson upper bound is under 0.92. Not drawn: a shadow or canary is also `retired` when it is replaced or cancelled, or when the opt-in sequential rule finds a stage inconclusive.
Your adapter on trial
Serving everyone
Stopped
Replaced
| State | Traffic | Meaning |
|---|---|---|
| `candidate` | 0 | Trained and scored tonight, not yet judged |
| `rejected` | 0 | Failed the golden gate or the shadow |
| `shadow` | 0 | Scores real traffic silently next to the live version |
| `canary` | 10, then 50 | Serves a share of requests; the teacher audits a sample |
| `live` | 100 | Serves every request the student is confident on |
| `retired` | 0 | Replaced by a newer version, cancelled by an operator, or (sequential rule only) inconclusive |
| `rolled_back` | 0 | Taken off the line because its audits were clearly below the floor |
`inconclusive` is an event, not a state: under the opt-in sequential rule, a canary stage that reaches `max_stage_audits` without a decision is `retired` with an `inconclusive` event.
Step through the rollout one stage at a time; the numbered list under it keeps every rule.
1. **Golden gate** (`ReleaseGate.admit`, right after training). Read the candidate's held-out numbers from its eval, in this order:
- precision at least `min_precision` (0.95), else `rejected`;
- coverage at least `min_coverage` (0.25), else `rejected`;
- with `RATCHET_MIN_CERTIFIED_PRECISION` set (opt-in), the certified lower bound at least that floor, else `rejected`;
- if a version is live: no regression against it. By default, the candidate is rejected if its precision or coverage is more than `tolerance` (0.02) below the live version's. With the opt-in paired gate, a row-by-row test instead (below);
- per task: a task that fails its own check is disabled on the candidate (or, with `RATCHET_TASK_MODE=reject`, rejects it).
The live version's numbers come from a re-score on the **same golden rows the same night** (champion against challenger), so both are graded on one exam.
2. **First stage.** With no live version (cold start), the candidate goes straight to canary at 10%: there is nothing to shadow against, so the teacher's audits judge it. Otherwise it enters **shadow**. Any older shadow or canary still in flight is retired first (rare, because Night School does not train while a release is in flight).
3. **Shadow** (`_tick_shadow`). On each request the shadow version is sampled and scored like a real student, against its own threshold. Nothing it says is served. Its answer is compared with the teacher's when the teacher answered that request (an escalation or an audit). Otherwise, when it would serve and disagrees with the served answer, the teacher arbitrates, and the arbitration is stored as training data; an arbitration that fails counts as a disagreement. When it agrees with the live student's served answer, that agreement is accepted as it is. Once the shadow would have served `min_shadow` (100) answers:
- shadow precision below 0.95: `rejected`;
- shadow coverage more than 0.02 below the live student's coverage on the same requests: `rejected`;
- otherwise canary at 10%.
4. **Canary** (`_tick_canary`, default Wilson rule). The teacher audits a random share of the canary's served answers. With at least `min_audits` (20) audits **in the current stage**:
- if the upper end of the 95% Wilson interval of audited precision is below `live_floor` (0.92): `rolled_back`, and traffic returns to the live version (or the teacher on cold start);
- else if the point estimate is below 0.92: hold and keep auditing;
- else advance 10 to 50, or 50 to live. The move starts a new stage, so the audit count starts again.
5. **Promotion.** Under the default rule, one transaction (`Ledger.promote_live`) makes the canary live at 100% and retires the old live version, with a `live_retired` event then a `live` event, so two versions are never live.
6. **Live watch** (`_tick_live`). The live version is never rolled back automatically. By default an e-process over its audits since it went live raises one `alert` event per version when the evidence that precision is below 0.92 reaches 100 (`alpha_alert` 0.01). The runbook says what to do.
Ticks run from the request path at most every `RATCHET_TICK_INTERVAL_S` seconds (5 in live mode, every request in mock mode) and from cron (`python -m ratchet.live tick`).
```mermaid
flowchart TB
N(["Night School trains"]) ==> G{{"Golden gate"}}
G -->|fail| R["rejected"]
G ==>|"pass, live exists"| S["shadow"]
G -->|"pass, first version"| C10
S -->|fail| R
S ==>|"holds up"| C10["canary at 10%"]
C10 ==>|"enough good audits"| C50["canary at 50%"]
C50 ==>|"enough good audits"| L(["live at 100%"])
C10 -->|"clearly bad"| RB["rolled_back"]
C50 -->|"clearly bad"| RB
class N neutral
class G gate
class S,C10,C50 student
class L ok
class R,RB danger
```
**Reading this diagram.** The same story as the state diagram, as a pipeline from Night School at the top; the thick path is the way to live. The shadow stage is skipped only on a cold start, when there is no live student to compare with. Both canary stages can end in a rollback, but only the 50% stage can lead to live.
One canary tick under the default rule:
```mermaid
flowchart TB
A(["Audits in this stage"]) --> N{{"At least min_audits?"}}
N -->|yes| W{{"Wilson upper bound below 0.92?"}}
W -->|yes| Back["Roll back"]
W -->|no| P{{"Point estimate below 0.92?"}}
P -->|no| F{{"Releases frozen?"}}
F -->|no| Up(["Advance a stage, or promote"])
N -->|no| Hold["Hold: keep auditing"]
P -->|"yes: not enough evidence"| Hold
F -->|yes| Hold
class A neutral
class N,W,P,F gate
class Back danger
class Up ok
class Hold neutral
```
**Reading this diagram.** Three of the exits lead to the same hold. The only way to move forward is to have enough audits, a point estimate at or above the floor and no operator freeze. A rollback needs the strongest evidence of all: not "the average looks bad" but "even the generous end of the interval is below the floor". A freeze is checked last, so it never blocks a rollback.
## The Wilson bound, precisely
The Wilson score interval for `s` agreeing audits out of `n`, with `z = 1.96`:
```text
p = s / n
denom = 1 + z^2 / n
centre = (p + z^2 / (2n)) / denom
half = z * sqrt( p(1 - p)/n + z^2/(4n^2) ) / denom
bounds = [max(0, centre - half), min(1, centre + half)]
```
It is the two-sided 95% interval; the rule uses its upper end, so a rollback needs strong evidence that the version is worse, not just a bad-looking average. Checked with `stats.wilson`:
| Audits agreeing | Point estimate | Wilson upper bound | Decision at a 0.92 floor |
|---|---|---|---|
| 19 of 20 | 95% | 99.1% | Advance |
| 17 of 20 | 85% | 94.8% | Hold: below the floor, but not clearly |
| 10 of 20 | 50% | 70.1% | Roll back |
A plain point-estimate rule was not used because at 20 audits, a healthy canary at 96% true precision shows a point estimate below 92% about 19% of the time.
## The anytime-valid rule (opt-in)
The Wilson interval keeps its error rate for **one look** at a fixed sample. The gate looks after every request. When true precision sits exactly on the floor, checking Wilson after every audit falsely rolls back about 22% of such versions within 400 audits, against a nominal 2.5%. And `min_audits` lets a broken deploy keep serving until it has 15 to 20 audits.
**Explained simply.** Think of a running bet against "quality is fine". You start with one coin. Before each audit you bet a little that the answer will be wrong. If quality really is fine, this bet cannot grow on average: it is a fair game or worse for you. So if your pile ever reaches 20 coins, that is strong evidence quality is not fine. Because a fair game cannot grow on average, it does not matter how often you look at your pile: checking after every single audit is allowed. That is what "anytime-valid" means.
**Precisely.** With `RATCHET_SEQUENTIAL=1`, `_tick_canary` hands over to `_sequential_canary`, which reads two integers per version, audits `n` and agreements `s` (`Ledger.audit_counts`):
1. **Rollback.** The mixture e-value against "precision is at least the floor", over **every** audit of the canary's life, reaches `1/alpha_rollback` (20). It mixes likelihood ratios over a fixed grid of alternatives below the floor; under the null each audit multiplies it by a factor whose expected value is at most 1, so it is a nonnegative supermartingale and Ville's inequality gives P(it ever reaches 20) at most 5%, however often the gate looks. There is no minimum count: 0 agreeing out of 2 already gives 29.8.
2. **Promotion,** on the current stage's audits once there are `min_audits`: the point estimate at or above the floor, or with `RATCHET_PROMOTE_EVIDENCE=1` an e-value that precision is **above** the floor reaching `1/alpha_promote` (20). That needs 56 agreeing audits per stage at a 0.92 floor (55 give 19.996).
3. **Futility.** A stage with `max_stage_audits` (400) audits and no decision is retired, with an `inconclusive` event. It counts audits, not time.
Every event carries the counts, the e-value and its threshold, and an anytime-valid 95% confidence sequence, for example `v3 rolled back: 3/7 audits agree; e-value 64.9 >= 20 (anytime-valid, alpha 5%); 95% CS [0.041, 0.908]; traffic back to live v1`. Anyone can recompute those numbers from the two counts.
The sequential canary is opt-in. The **live alert** already uses the e-process by default (`RATCHET_ALERT_EPROCESS=1`, threshold 100). The live-run harness's planned gate profile turns the sequential rule on (`RATCHET_SEQUENTIAL=1`, `RATCHET_CANARY_AUDIT_RATE=0.5`).
## The paired gate (opt-in)
The default regression check compares two point estimates within 0.02, but the candidate and the live version were scored on the same rows, so the comparison can be done row by row. With `RATCHET_PAIRED_GATE=1`, the gate reads the candidate's `paired` record against the re-scored live version:
- **Coverage:** `b` rows only live serves, `c` rows only the candidate serves, and McNemar's exact one-sided p-value `p_cov = P(X at least b)` with `X ~ Binomial(b + c, 1/2)`.
- **Wrong answers as a client sees them:** negative flips (the candidate serves a wrong answer where live was right, served or escalated) against positive flips, with `p_err`.
- **Negative flip rate:** answers live served right that the candidate serves wrong, with a 95% Wilson interval.
It rejects only a regression that is both large and significant: more than `tolerance` of the rows lost from coverage with `p_cov` below `RATCHET_REGRESSION_ALPHA` (0.05), more than `tolerance` of the rows newly answered wrong with `p_err` below it, or, with `RATCHET_MAX_NFR` below 1, a flip rate whose lower bound is above it. Without a paired record against the current live version it falls back to the default rule.
In plain words: only rows where the two versions differ count, and four lost rows with none won is like four coin flips all landing heads, which happens one time in 16.
## Per-task checks
When several tasks share one adapter, the golden gate also checks each task with at least `min_task_test_n` (10) held-out rows. A task fails when its held-out precision is below its floor (its spec's `policy.min_precision`, else 0.95), more than `tolerance` below the live version's precision on that task, or unmeasured because it answers none of its held-out rows.
- `task_mode` `disable` (default): the task's threshold becomes 1.01 on that candidate, a `task_disabled` event is written, the teacher answers that task, and the version ships for its other tasks. The task stays disabled for the version's life, through every re-score.
- `task_mode` `reject`: the candidate is rejected.
- Either way, a candidate that would answer less than `min_coverage` of its held-out rows once its failing tasks are off is rejected: it would sit in shadow or canary forever, and no night would train meanwhile.
On the live and canary versions, one `alert` per (version, task) fires when that task's audited precision has a Wilson upper bound below 0.92 with at least `min_audits` audits. Nothing is disabled automatically on a live version; pinning a task to the teacher is an operator's decision.
## Two watchers beside the gate: drift and the teacher
The gate's tick has two more steps that never promote or roll back a version. They watch for changes the golden exam cannot see.
**The drift alarm** (`_tick_drift`) watches each (version, task) stream of the student's confidence scores for a shift, with no labels needed. When it alarms (`RATCHET_DRIFT_ALARM`, default 100, so at most a 1% false alarm per stream over its life), it writes a `drift` event and suspends that version's certificate, because the certificate no longer describes the traffic. Two opt-in actions can follow: a higher audit rate for the alarmed version (`RATCHET_DRIFT_AUDIT_BOOST`) and an earlier night (`RATCHET_DRIFT_EARLY_NIGHT`). How the score is built is in [Confidence](/guides/confidence/#watching-the-confidence-scores-over-time-the-drift-alarm).
**The teacher watch** (`_tick_watchers`) asks whether the teacher itself is steady. The golden set and the audits both measure agreement with the teacher, so a teacher whose answers wander would make every number meaningless. Each stored label gets at most one re-test per check epoch (a repeated request that the teacher labels again, or one of the `RATCHET_TEACHER_PROBES` probes Night School sends each night, 20 by default). A disagreement puts the label in quarantine (`contested`): it is left out of both the training and the golden split until Night School adjudicates it, by asking the teacher twice more and keeping the strict majority of three. When the evidence that the teacher disagrees with itself more than `RATCHET_TEACHER_NOISE_MAX` (0.05) reaches 1/`RATCHET_ALPHA_TEACHER` (100), the gate writes a `teacher_drift` event and Night School stops training until an operator acknowledges it with `POST /ratchet/teacher/ack`. A new model id in the teacher's responses starts a new epoch. `GET /ratchet/teacher` shows the state, and `RATCHET_TEACHER_WATCH=0` turns the actions off while the free checks are still recorded.
```mermaid
stateDiagram-v2
direction LR
[*] --> Steady
Steady --> Quarantine: a re-test disagrees
Quarantine --> Steady: 2 of 3 settle it
Steady --> Frozen: evidence ≥ 1/alpha
Frozen --> Steady: an operator acks
class Steady ok
class Quarantine pending
class Frozen danger
```
**Reading this diagram.** One disagreement is not an emergency: it only quarantines one label until two more teacher answers settle it by a majority of 3. Training stops only when many disagreements add up to strong evidence (1/`RATCHET_ALPHA_TEACHER`, 100 by default), and it stays stopped until a person acknowledges it. The freeze affects training, not serving: the live student keeps answering. Not drawn: re-tests that agree leave the watch steady, and a new teacher model id starts a new check epoch.
Training as usual
One label held out
Training stopped
## No training while a release is in flight
Night School skips the night while any version is in `shadow` or `canary`, with a `skip` event such as `v4 still in canary; no training tonight (saves GPU time)`. A new candidate would replace the one being judged and waste both the GPU time and the evidence gathered so far. An operator who wants tonight's training can retire the shadow first.
## Operator holds and the admin API
Every manual change goes through `ratchet/admin.py`, shared by the HTTP admin API (`/ratchet/admin/*`, admin key from `RATCHET_ADMIN_KEYS`) and the CLI (`python -m ratchet.admin`). Each change needs a reason (3 to 500 characters), is checked against an explicit table of allowed moves, and writes the change, an `admin_actions` row (who, what, why, before and after) and an `admin` event in one transaction.
| Switch or action | Effect on the gate |
|---|---|
| `releases_frozen` | No version starts taking traffic or takes more: no shadow to canary, no canary advance or promotion, and a cold-start candidate shadows the teacher instead of taking 10%. Rollbacks, rejections and the live alert keep running |
| `audit_rate` set to 0 | Same hold: a canary without audits could neither advance nor be rolled back, so no new canary starts. Setting 0 during a canary needs `force` |
| `student_disabled` | Kill switch: no student or shadow call; the teacher answers everything (route `student_disabled`). No version changes state |
| `training_paused` | Every night writes `skip` and trains nothing |
| `rollback`, `retire` | Roll a canary back; retire a canary, shadow or live version |
| `promote` | A canary to another share or to live; a shadow to canary. A rejected, rolled-back or retired version (or a candidate) to canary only with `force`. Nothing becomes live except a canary, and a version without a golden eval is never promoted |
A hold never blocks a rollback, a rejection or an alert. The admin's moves and the gate's (the golden gate, the shadow decision and the default canary rule) use the same compare-and-set (`Ledger.transition`, `Ledger.promote_live`): a move into shadow or canary is refused while another version is in flight, a decision made on a state someone else just changed is dropped, and a candidate an admin moved while it was being scored is left alone with a `conflict` event.
In the current code, the opt-in sequential canary rule (`_sequential_canary`) moves versions with `Ledger.set_version`, an unconditional update, rather than the compare-and-set writers, and its promotion writes no `live_retired` event. The "each decision is made once" guarantee therefore covers the default rule, not the sequential one, until that path uses `transition` and `promote_live` too.
## The release log
Every decision writes an event (the `events` table, shown in the dashboard's release log and in `GET /ratchet/stats`):
| Event | Written when |
|---|---|
| `trained` | Night School trained and scored a new version |
| `skip` | No training tonight (release in flight, too few new examples, paused, no golden set) |
| `rejected` | Golden gate or shadow failed, with the reason |
| `shadow` | Passed the golden gate; shadowing the live version (appends the paired counts and the certificate when there are any) |
| `canary` | Started or advanced a canary stage |
| `live_retired`, `live` | The old live version retired; the canary promoted to 100% |
| `retired` | An in-flight version replaced by a newer candidate |
| `rollback` | The canary rolled back, with the evidence |
| `inconclusive` | Sequential rule only: a stage hit `max_stage_audits` without a decision |
| `alert` | The live version (or one task of the live or canary version) is clearly below the floor |
| `task_disabled` | One task of a candidate failed its own golden check |
| `drift` | The drift alarm fired for a (version, task) stream |
| `teacher_drift` | The teacher watch froze Night School |
| `conflict` | A candidate changed by another process while the gate decided |
| `admin` | An operator's change, with the actor and the reason |
As of Oct 29, 2026, the release gate is tested with mock models and offline tests, and its golden step decided once on Nebius (28 Oct 2026: rejected). Ratchet's own adapter is not connected to the live gateway, so no shadow, canary or live release has run. Live mode, the cron tick and the admin CLI are tested offline.
## It ran on Nebius, and the gate said no
**Measured on Nebius, 28 Oct 2026, by hand, once.** Night School ran end to end once on Nebius, by hand (28 Oct 2026): the gateway on a Nebius CPU VM, the training job on a spot L40S, the adapter served from a Nebius Serverless Endpoint. The golden gate rejected it (81.8% precision against the 95% target), so it was never made live. The live site's own gateway is still on Cloudflare and is not connected to that endpoint, and the loop has not run nightly on live traffic. No saving is claimed.
| What | Result |
|---|---|
| Gateway | A Nebius CPU VM in `eu-west1` (2 vCPU), in live mode |
| Data | 4,322 existing Ultra-labelled text-to-SQL rows imported (no new teacher calls, no exam rows): 3,864 to train on, 458 held out as the golden set |
| Training | A spot L40S Serverless Job: 2 epochs, 966 steps, 1,478 s; loss about 0.15 to 0.08; token accuracy about 0.975 |
| Serving | The adapter (SHA-256 `dc9ad4b9...`) hot-loaded into a Nebius Serverless Endpoint |
| Golden gate | SQL compared as text: agreement 17.9%; at the gate's threshold the student would answer 4.8% of questions with 81.8% precision (9 of 11) against the 95% target. **Rejected**, never shadow, canary or live |
The caveats, all of them, are in the [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius): four job submissions (an accidental on-demand job that was cancelled, a missing `einops`, a save that failed with `EPERM` and whose weights were recovered, then the job that saved); the night's own evaluation timed out at 15 s, so the gate result comes from running the eval and the gate again on the saved adapter with a 120 s timeout, through a one-off driver; five problems fixed by hand on the VM, now fixed in the repository (spot by default, `einops` pinned, the adapter saved locally and then copied, a clear stop when `aws` is missing, a separate student timeout for the night); and cost estimates (about 2.7 GPU-hours, at most about $4 at list prices) where the Nebius bill decides.
The gate did its job: a student that is right 81.8% of the time on the few answers it would serve is below a 95% target, so nothing reached shadow, canary or live traffic.
## For engineers
| What | Where |
|---|---|
| The gate | [`ratchet/release.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/release.py): `ReleaseGate.admit`, `tick` (six steps: `_tick_drift`, `_tick_live`, `_tick_tasks`, `_tick_shadow`, `_tick_canary`, `_tick_watchers`), `_sequential_canary`, `task_failures`, `_paired_regression`, `evidence`, `ReleasePolicy`, `CANARY_STEPS` |
| Statistics | [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py): `wilson`, `log_e_bernoulli`, `e_bernoulli`, `confidence_sequence`, `mcnemar_one_sided`, `cp_lower` |
| Paired comparison | [`ratchet/certificate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/certificate.py): `paired`, `paired_stats` |
| Moves and counts | [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py): `transition`, `promote_live`, `transaction`, `audit_counts`, `version_live_quality`, `shadow_quality`, `flags` |
| Manual changes | [`ratchet/admin.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/admin.py): `rollback`, `retire`, `promote` (`MOVES`), `set_flag`, `pin_task`, `tick` |
| The night that feeds it | [`ratchet/nightschool/orchestrate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/orchestrate.py): `run_night` (champion re-score, paired evidence, BOM, then `admit`) |
| Tests | `tests/test_release.py`, `tests/test_admin.py`, `tests/test_stats.py`, `tests/test_calibration.py` |
`ReleasePolicy` fields, each set with `RATCHET_` ([Configuration](/reference/configuration/#release-policy-ratchetreleasepy)):
| Variable | Default | Meaning |
|---|---|---|
| `RATCHET_MIN_PRECISION` | 0.95 | Golden gate and shadow precision floor |
| `RATCHET_MIN_COVERAGE` | 0.25 | Golden gate coverage floor |
| `RATCHET_TOLERANCE` | 0.02 | Largest drop against live (0 to 0.2) |
| `RATCHET_MIN_SHADOW` | 100 | Would-serve shadow answers before a decision |
| `RATCHET_MIN_AUDITS` | 20 (15 in mock mode) | Audits per canary stage before promotion, and before a Wilson rollback or alert |
| `RATCHET_LIVE_FLOOR` | 0.92 | Audited precision floor; must be at most `RATCHET_MIN_PRECISION` |
| `RATCHET_MIN_TASK_TEST_N` | 10 | Held-out rows a task needs before its own golden check |
| `RATCHET_TASK_MODE` | `disable` | Or `reject` |
| `RATCHET_SEQUENTIAL` | 0 | 1: the anytime-valid canary rule |
| `RATCHET_ALPHA_ROLLBACK` | 0.05 | Sequential rollback threshold 1/alpha = 20 |
| `RATCHET_PROMOTE_EVIDENCE` | 0 | 1: promote on evidence (needs `RATCHET_SEQUENTIAL=1`) |
| `RATCHET_ALPHA_PROMOTE` | 0.05 | Evidence-based promotion threshold 20 |
| `RATCHET_MAX_STAGE_AUDITS` | 400 | Sequential futility cap; 0 means never |
| `RATCHET_ALERT_EPROCESS` | 1 | Live alert by e-process (0: Wilson) |
| `RATCHET_ALPHA_ALERT` | 0.01 | Live alert threshold 100 |
| `RATCHET_MIN_CERTIFIED_PRECISION` | 0.0 | Certified floor in the golden gate; 0 is off |
| `RATCHET_PAIRED_GATE` | 0 | 1: row-by-row regression check |
| `RATCHET_REGRESSION_ALPHA` | 0.05 | The paired gate's McNemar level |
| `RATCHET_MAX_NFR` | 1.0 | Negative flip rate cap; 1 is off |
The drift alarm and the teacher watch add `RATCHET_DRIFT_ALARM` (100), `RATCHET_DRIFT_AUDIT_BOOST` (1), `RATCHET_DRIFT_EARLY_NIGHT` (0), `RATCHET_TEACHER_WATCH` (1), `RATCHET_TEACHER_NOISE_MAX` (0.05), `RATCHET_ALPHA_TEACHER` (0.01), `RATCHET_TEACHER_PROBES` (20) and `RATCHET_ADJUDICATE_MAX` (50). A typo or an out-of-range value stops startup with a message naming the variable. Related settings: `RATCHET_AUDIT_RATE` (0.05; mock mode at least 0.10), `RATCHET_CANARY_AUDIT_RATE` (unset: a canary's own audit rate, so its evidence arrives sooner), `RATCHET_TICK_INTERVAL_S`.
Endpoints ([API](/reference/api/)):
- `GET /ratchet/stats`: `versions[].evidence` for the canary and live versions (`ReleaseGate.evidence`: the rule in force, counts over the life and the stage, e-values, the confidence sequence, the Wilson interval), `controls` (the switches), and the events.
- `GET /ratchet/versions/{id}/certificate`: the golden certificate and the gate's `live` evidence.
- `/ratchet/admin/*`: `versions`, `flags`, `audit-log`, `versions/{id}/rollback`, `.../retire`, `.../promote`, `releases/freeze`, `student/disable`, `training/pause`, `PUT settings`, `tasks/{name}/pin`, `tick` ([API: Admin](/reference/api/#admin)).
```bash
python -m ratchet.live tick # run the gate now (cron does it every few minutes)
python -m ratchet.admin versions # states, traffic, golden numbers, this stage's audits
python -m ratchet.admin rollback 7 --reason "refund labels are wrong since 09:00"
python -m ratchet.admin flag releases_frozen on --reason "holiday freeze"
RATCHET_MODE=mock RATCHET_SEQUENTIAL=1 uvicorn ratchet.gateway:app --port 8000 # a mock gateway with the sequential canary
```
The state machine with every threshold is in [Architecture §5](/reference/architecture/#5-version-state-machine); what to do on each event is in the [runbook](/reference/runbook/), including [reading an e-value event](/reference/runbook/#reading-an-e-value-event) and [manual interventions](/reference/runbook/#manual-interventions).
## Common questions
### How many users get a bad canary's answers before it is rolled back?
Some do: that is what a canary is. At 10%, roughly one request in ten can reach it, and only the ones the canary is confident on are served by it. Under the default rule it keeps serving until its stage has `min_audits` audits. A higher canary audit rate (`RATCHET_CANARY_AUDIT_RATE`) shortens that.
### Why is the live version never rolled back automatically?
Automatic action on all traffic is a decision Ratchet leaves to a person. The live watch raises an alert; an operator can retire the live version, turn the student off with the kill switch, or restore an older version through the admin API, each in one command.
### How long does a release take?
It depends on traffic and the audit rate. The arithmetic: at a 5% audit rate and a 10% canary, 20 audits take about 4,000 or more requests, before the 50% stage needs its own 20. The shadow first needs 100 would-serve answers. Low-traffic deployments should expect slow promotions; audit focusing (`RATCHET_CANARY_AUDIT_RATE`) helps.
### What does an audit cost?
One teacher answer per audited request, on top of the student's samples. At the default 5% audit rate that is one teacher call for every 20 served student answers. Audited answers also become training data when valid.
### Could two processes promote the same version twice?
Not under the default rule. The gateway's tick, the cron tick and an admin can run at the same time; each move is a compare-and-set inside one SQLite transaction, so the second decision changes nothing and writes nothing. The opt-in sequential canary rule does not use the compare-and-set writers yet (see the note above).
### Do customer complaints (feedback verdicts) roll a version back?
No. Human verdicts are reported and can raise a `human_alert`, but the gate never reads them: clients report what they notice, so the sample is biased. Audited precision stays the release signal.
### Where has the gate run so far?
On mock models and in offline tests. The whole gate, including the drift alarm and the teacher watch, is ready to run in live mode once a trained student is served.
## What can go wrong
| Failure | What the system does | Where you see it |
|---|---|---|
| The candidate fails the golden gate | `rejected` with the reason; the live version stays | `rejected` event, `GET /ratchet/stats` |
| The shadow's teacher arbitration fails | Counted as a disagreement, so a flaky teacher can reject a candidate but never promote one | rejection reason names the count |
| The canary's audits are clearly below the floor | `rolled_back`, traffic returns to the live version (or the teacher) | `rollback` event with the evidence |
| Two processes decide at once | One compare-and-set wins; the other changes nothing | `conflict` event when a candidate was moved meanwhile |
| An operator freezes releases | No version takes more traffic; rollbacks and alerts keep running | `admin` event, `controls` in the stats |
| The live version gets worse | An `alert`; no automatic action | `alert` event, then `python -m ratchet.admin rollback` or `student/disable` |
| The teacher's labels turn unstable | Contested labels leave training; training freezes at strong evidence | `teacher_drift` event, `GET /ratchet/teacher` |
## Related
---
# Training on your data with the student endpoint
Source: https://docs.ratchet.dualmindlab.tech/guides/training-on-your-data/
The student endpoint only answers requests; training happens somewhere else: the gateway stores the teacher's answers to your traffic, and every night a Nebius Serverless Job trains , which the endpoint then loads from your bucket.
- **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)).
- **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)).
- **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)).
- **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)).
- **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate.
- **The nightly schedule on live traffic.** Night School runs by hand today; the timer is scripted.
- **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline.
- **Live results from your own data.** None are published yet. The three published adapters (two text-to-SQL, one tool-calling) were trained on a borrowed GPU.
Run every script with `DRY_RUN=1` first. Run the first-run checks on Token Factory (`scripts/day1_check.py`) before relying on any of it.
The **apprentice chef** has a cooking station: the GPU endpoint. The station only **cooks**. The apprentice never studies while cooking orders, and nothing the apprentice cooks is written into the recipe book.
Every time the **head chef from the agency** cooks a dish, the pass writes it into the **recipe book** (the ledger), with phone numbers and emails blacked out.
At night, a copy of the study pages goes into a **shared locker** (the storage bucket). A separate **classroom** is rented for about an hour (the training job). The apprentice studies the whole book there and writes a new **notebook of techniques** (the LoRA adapter), which is put back in the locker. Then the station picks the new notebook up from the locker and takes the **sealed exam**. If it passes, it starts a trial shift.
When the station is **closed to save money**, a different, untrained cook from the agency takes the easy orders, and the menu says so. The notebook is yours: you can take it to another kitchen, with its certificate and the license notes for the apprentice's original training.
## How it really works
### The path from a request to a loaded adapter
Checked, redacted, with its teacher and license.
night/train.jsonl, uploaded to the bucket.
A fresh LoRA from the base, written back to the bucket.
Graded on the golden set, then handed to the gate.
The thirteen steps in full:
1. **Your app sends a request** to the gateway's `/v1/chat/completions`, ideally with a task name (`X-Ratchet-Task` or `metadata.ratchet_task`). See [the request journey](/guides/request-journey/).
2. **The gateway stores teacher answers, never student answers.** When answers (an escalation, a cold start, a random audit or a shadow arbitration), the answer is checked, the prompt and answer are redacted, and both are stored in the ledger with their provenance: teacher, license, hash, redaction count. When serves an answer, only metadata is logged. Details: [the ledger](/guides/ledger/).
3. **Every night, cron runs `python -m ratchet.live night`** on the gateway's host. It runs Night School's checks (paused, storage off, a release in flight, fewer than 50 new examples, no golden rows) and skips the night if one applies. Otherwise it re-scores the live adapter on tonight's golden rows and goes on. See [Night School](/guides/night-school/).
4. **It writes the training file.** `RATCHET_DATA_DIR/night/train.jsonl` holds one prompt-completion row per training example: the stored messages, the teacher's stored answer, and `chat_template_kwargs` with thinking off, plus the row's task and provenance. Golden rows are not written there; they stay on the gateway's host.
5. **It uploads the file.** `RATCHET_SYNC_CMD` (for example `aws s3 sync ./data s3:// --endpoint-url ...`) copies `RATCHET_DATA_DIR` to the Object Storage bucket. It must only upload, never delete.
6. **It starts the training job.** `RATCHET_TRAIN_CMD` (default `bash deploy/train_job.sh {version}`, with `{version}` = `student-vN`) submits a Nebius Serverless Job: `nebius ai job create`, platform `gpu-l40s-a`, preset `1gpu-8vcpu-32gb`, the bucket mounted at `/data`, a 200 GiB disk and `--timeout 2h`.
7. **The job trains.** In the training image it runs `python -m ratchet.nightschool.train --data /data/night/train.jsonl --base nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 --out /data/adapters/student-vN`. It checks every row, writes `versions.json`, checks the chat template, trains the LoRA adapter from the base model, and saves the adapter, the tokenizer files, `ratchet_manifest.json` and `NOTICE.md` into `/data/adapters/student-vN/`.
8. **The script waits for the adapter.** With `RATCHET_BUCKET_S3` set, `train_job.sh` polls the bucket for `adapters/student-vN/adapter_model.safetensors` every 30 seconds, for up to 2 hours, and exits when it appears. Without `RATCHET_BUCKET_S3` it returns right after submitting, so the adapter load that follows would run before the job has written anything: set it when the script is your `RATCHET_TRAIN_CMD`.
9. **The adapter is hot-loaded.** `ratchet.live` calls the endpoint's `POST /v1/load_lora_adapter` with `{"lora_name": "student-vN", "lora_path": "/data/adapters/student-vN"}` (the path is `RATCHET_ADAPTER_ROOT/`). vLLM reads the files from the same bucket, mounted at `/data` in the endpoint. No restart. An "already loaded" answer is fine.
10. **The golden eval runs on the endpoint.** `LiveEvaluator` sends each golden prompt to the new adapter (3 samples, as production does), calibrates the threshold on half the rows and measures precision on the other half. Free-text tasks also get a score from , which is recorded but does not gate.
11. **The adapter's BOM is signed and the gate decides.** The bill of materials (teachers, licenses, the base model's license, the NOTICE text, the dataset hashes, the eval, the set roots) is written once, and signed when `RATCHET_BOM_SIGNING_KEY` is set. Then the release gate admits the candidate to **shadow**, **canary 10%** on a cold start, or **rejects** it. See [the release gate](/guides/release-gate/).
12. **The licenses join the weights.** `ratchet.live` writes `ratchet_bom.json` and the BOM's `NOTICE.md` into `RATCHET_DATA_DIR/adapters/student-vN/` and runs `RATCHET_SYNC_CMD` again, so the bucket folder holds the weights and their licenses together. If this second sync fails, the version still stands, a `sync_failed` event is written, and the command exits non-zero so cron notices.
13. **The gateway serves the adapter by name.** As the gate moves the version through shadow and canary to live, the gateway sends student calls to the endpoint with `model` set to `student-vN`. vLLM serves each loaded adapter under its own name.
### The Nebius pieces
```mermaid
flowchart TB
subgraph Host["Your gateway host"]
direction LR
GW["Gateway"] --> DB[("SQLite ledger")]
NS["Night School ratchet.live night"] --> DB
end
subgraph TF["Token Factory, per token"]
TEA["Teacher: Ultra Judge: Super"]
end
subgraph Neb["Nebius AI Cloud, ran by hand 27 to 28 Oct"]
direction TB
JOB["Serverless Job LoRA on one L40S"] -->|"adapter"| BK[("Bucket at /data")]
BK -->|"loads"| EP["Serverless Endpoint vLLM"]
NIGHT["Nightly schedule and live connection: not run"]
end
Host ==>|"escalations, audits, judge"| TF
Host ==>|"rows, job, student calls"| Neb
class DB,BK store
class TEA teacher
class JOB,EP ok
class NIGHT pending
class TF teacherZone
class Neb okZone
```
**Reading this diagram.** Three boxes, three jobs. Your host decides and remembers: it holds the gateway, the ledger and the night script. Nebius does the heavy lifting: a job trains, an endpoint serves, and one bucket is shared by both, which is how weights get from one to the other without a copy step. Token Factory is rented per token and holds the teacher and the judge. From the host, the night uploads the rows to the bucket, submits the job and tells the endpoint to load the adapter, and the gateway sends student calls to the endpoint. Nothing on Nebius ever writes to the ledger; only the host does. The job, the bucket and the endpoint are drawn green because each ran once by hand on Nebius (27 to 28 Oct 2026); the amber box is what has not run: the nightly schedule, and the live gateway's connection to an endpoint. There are no live results yet.
Rented per token
Keeps data
Ran by hand
Not run
### How one training example is made
```mermaid
flowchart TB
T(["A teacher answer"]) --> Q{{"Complete, valid, no opt-out, not pinned?"}}
Q -->|yes| RED["Redact the prompt, answer and conditions"]
RED --> V{{"Still valid?"}}
V -->|yes| D{{"Hash of task and redacted messages known?"}}
D -->|new| SP{{"Hash bucket"}}
D -->|"seen again"| B["seen + 1, unless a retry"]
B --> F{{"Seen 3 or more?"}}
SP -->|"about 90%"| TR[("Training split")]
SP -->|"about 10%"| GO[("Golden split")]
F -->|yes| TR
TR --> J["night/train.jsonl teacher answer as target"]
GO --> E["Golden eval only"]
Q -->|no| X["Served, not stored"]
V -->|no| X
D -->|tombstoned| X
class T teacher
class Q,V,D,SP,F gate
class TR,GO store
class X outline
```
**Reading this diagram.** A teacher answer can fall out of the path at three places, always on the safe side: the answer is still served, only storage is skipped. After redaction the example gets a hash, and the hash does three jobs: it merges repeats, it blocks erased examples for good, and it fixes the split without anyone choosing it. The golden split is never trained on, with one exception: a prompt seen three or more times moves to training, so a question people ask all the time does not stay hidden forever.
### What each part of an example is used for
| Part | Used by | Notes |
|---|---|---|
| Redacted messages | `prompt` in the training row, the golden prompt | Placeholders such as `[EMAIL]` replace identifiers |
| Teacher answer | `completion`, the loss target, the golden reference | The student's own answers are never targets |
| Stored conditions | The golden eval only (it samples under them) | Never reach the chat template |
| Provenance: teacher, license, hash, redactions, origin | The signed BOM and NOTICE.md | Travels with the adapter |
| Client and end-user hash | Erasure and opt-outs | Never in the training file |
### Where your data goes
```mermaid
flowchart TB
APP(["Your app"]) -->|"chat request"| GW["Gateway"]
GW -->|"teacher answers, redacted"| LED[("Ledger: examples")]
LED -->|"night/train.jsonl, RATCHET_SYNC_CMD"| BK[("Object Storage bucket")]
BK -->|"mounted at /data"| JOB["Serverless Job train.py"]
JOB -->|"adapters/student-vN"| BK
BK -->|"load_lora_adapter"| EP["Student endpoint vLLM"]
EP -->|"answers as student-vN"| GW
class LED,BK store
class EP student
```
**Reading this diagram.** The loop is closed: requests and teacher answers go down the middle, and the adapter comes back up to the gateway through the endpoint on the right. The only arrow that carries your text out of the gateway is the one from the ledger to the training file, and by then it is redacted. The job and the bucket meet twice: it reads the rows from `/data` and writes the adapter back to the same place.
### One live night, with the real commands
```mermaid
sequenceDiagram
autonumber
participant Live as ratchet.live night
participant Bucket as Bucket at /data
participant Job as Serverless Job, L40S
participant Student as Student vLLM
participant J as Judge Super
Note over Live: cron, 02:30
Live->>Student: re-score live on golden rows
Note over Live: write night/ train.jsonl
Live->>Bucket: upload the rows
Live->>Job: train_job.sh: job create
Job->>Bucket: read the rows, write student-vN
Live->>Bucket: poll for the weights
Live->>Student: load_lora_adapter
Student->>Bucket: read student-vN
Live->>Student: golden eval, 3 samples a row
Live->>J: free-text rows
Note over Live: signed BOM, gate.admit
Live->>Bucket: BOM, NOTICE.md, sync again
```
**Reading this diagram.** Read it as a checklist in time order. Night School re-scores the old adapter before it trains (1), so the champion and the challenger sit the same exam, then uploads the rows with `RATCHET_SYNC_CMD` (2). `RATCHET_TRAIN_CMD` runs `deploy/train_job.sh`, which submits the job (3) and then polls the bucket for the weights file (5): the script, not the job, is what `ratchet.live` waits on. The last line matters for licensing: the weights are in the bucket from the job, and the BOM and `NOTICE.md` are added after the gate has decided (10), so the folder holds the adapter and its licenses together. Every arrow here is written and tested offline; none has run on Nebius yet.
### The student endpoint only answers
The endpoint is a Nebius Serverless Endpoint running vLLM with runtime LoRA loading. It serves the base model `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` plus whichever adapters were loaded into it. It never trains, never writes to the ledger, and never sees the training file. `bash deploy/student_endpoint.sh create` (once) sets:
| Setting | Value | Why |
|---|---|---|
| Image | `vllm/vllm-openai:v0.18.0-cu130` (`RATCHET_VLLM_IMAGE`) | `v0.18.0` without `-cu130` is the CUDA 12 build, for an older host driver |
| Platform, preset | `gpu-l40s-a`, `1gpu-8vcpu-32gb` | One L40S |
| Port, shared memory | `--container-port 8000`, `--shm-size 16Gi` | |
| Storage | `--volume :/data` | vLLM reads adapters from the same bucket the job writes |
| Auth | `--auth token --token $RATCHET_STUDENT_KEY` | The gateway sends this token; the script never prints it |
| Runtime LoRA | `--env VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`, `--enable-lora --max-lora-rank 16 --max-loras 4` | New adapters load without a restart; up to four adapters in one batch, enough for live, canary and shadow |
| Model code | `--trust-remote-code`, `--mamba_ssm_cache_dtype float32` | The second is the model card's flag (read Oct 21, 2026) |
| Limits | `--max-model-len 8192`, `--max-num-seqs 64` | The card's example uses 262,144 tokens; ticket prompts are short, and a longer one gets vLLM's 400 and escalates to the teacher |
| Thinking off | `--default-chat-template-kwargs '{"enable_thinking":false}'` | The student answers without reasoning, on the same prompt it was trained on |
| Reasoning parser (optional) | `--reasoning-parser-plugin ... --reasoning-parser nano_v3` with `RATCHET_REASONING_PARSER_PLUGIN` | The card's parser, uploaded to the bucket |
The other actions reuse the same endpoint (same URL, same token): `start` (the default; creates it when none has the name), `stop`, and `status`, which prints ``. `start` and `stop` find the endpoint's id by name (`RATCHET_ENDPOINT_NAME`, default `ratchet-student`) in `nebius ai endpoint list --format json`, or use `RATCHET_ENDPOINT_ID`. The lookup never guesses: output that is not JSON, two endpoints with the name, or a name missing from a list with more pages stops the script before anything starts, stops or is created. Each `create`, `start` and `stop` also records the endpoint's uptime in the ledger for the cost report (`python -m ratchet.cost endpoint up` or `down`); that step is best effort and never fails the deploy. Full details: [Deployment, step 4](/reference/deployment/#step-4-student-endpoint-vllm-with-runtime-lora).
### What becomes training data
Only teacher answers (and labels a person attested, from [feedback corrections or an import](/guides/ledger/)) become training targets. A teacher answer is stored only when it is complete (`finish_reason` `stop` or `tool_calls`), valid under the task's spec, not under a storage opt-out, not for a pinned task or a tool request, and after PII redaction. Each one records the teacher model, its license and license URL, its hash and the number of placeholders. You can keep any request, key, task or end user out of training with an [opt-out](/guides/ledger/), and switch storage off entirely with `RATCHET_STORE_EXAMPLES=0`.
### The training file
`dataset.write_training` writes TRL's conversational prompt-completion format. One row, shown on several lines (the content is illustrative):
```json
{
"task": "ticket-triage",
"params": null,
"seen": 1,
"provenance": {"teacher": "nvidia/Nemotron-3-Ultra-550b-a55b", "license": "OpenMDW-1.1",
"hash": "…", "redactions": 1, "origin": "live"},
"prompt": [{"role": "user", "content": "Where is my refund for order 1234? Mail me at [EMAIL]"}],
"completion": [{"role": "assistant", "content": "{\"category\": \"refund\", \"priority\": \"low\"}"}],
"chat_template_kwargs": {"enable_thinking": false}
}
```
`train.py` gives TRL only `prompt`, `completion` and `chat_template_kwargs`, so stored conditions never reach the template, and the loss covers the teacher's answer only. The same `{"enable_thinking": false}` lives in three places (the dataset writer, `train.py` and the endpoint script), and a test keeps them equal. Rows left out of training (a client's `stop`, a literal think tag, an answer the current spec refuses, a pinned task, a purged row) are listed on [Night School](/guides/night-school/#rows-left-out-of-training).
### The bucket
One Object Storage bucket in the same region as the GPUs, mounted at `/data` in both the job and the endpoint. `RATCHET_DATA_DIR` (default `./data`) is the local mirror that `RATCHET_SYNC_CMD` uploads ([Deployment, step 2](/reference/deployment/#step-2-object-storage-bucket)):
- data/
- night/
- train.jsonl rewritten by ratchet.live each night that trains
- boms/
- student-v3.json the adapter's BOM, written after the night
- adapters/
- student-v3/
- adapter_config.json written by the training job
- adapter_model.safetensors the LoRA weights
- versions.json Python, CUDA and library versions
- ratchet_manifest.json base model, teachers and licenses, template arguments
- NOTICE.md the licenses; replaced after the night by the text the BOM signed
- ratchet_bom.json the BOM, synced after the night
The job also saves the tokenizer files next to the weights. Keep every adapter a `live` version uses, for rollback. A mounted bucket is billed even while the endpoint is stopped.
### The training job
`deploy/train_job.sh ` builds and runs this command ([Deployment, step 5](/reference/deployment/#step-5-training-job)):
```bash
nebius ai job create --name ratchet- --image $RATCHET_TRAIN_IMAGE \
--platform gpu-l40s-a --preset 1gpu-8vcpu-32gb --volume $RATCHET_BUCKET_ID:/data \
--disk-size 200Gi --timeout 2h --container-command bash \
--args "-c 'python -m ratchet.nightschool.train --data /data/night/train.jsonl --base --out /data/adapters/'"
# plus --subnet-id $RATCHET_SUBNET_ID and --registry-secret $RATCHET_REGISTRY_SECRET when set
```
- `nebius ai job create` is the scripted command; `nebius ai create` is the interactive wizard.
- `--timeout 2h` is the job's own limit (1 h to 168 h, default 24 h), so a stuck job's GPU time is capped at the 2 hours the script waits.
- The version name and base model are checked against a strict pattern first, because they go into the job's `bash -c` line.
- `DRY_RUN=1` adds `--dry-run`: Nebius validates the request and submits nothing. Any value other than `0` or `1` is refused, so `DRY_RUN=true` can never submit a paid job.
Training defaults in `train.py`: 2 epochs, learning rate 2e-4, LoRA rank 16 with alpha 32, max sequence 2048, batch 8, bf16, loss on the answer only, targets including the Mamba projections (with an attention-and-MLP fallback if vLLM refuses them on day 1). See [What LoRA is](/guides/night-school/#what-lora-is).
**When the job fails.** A refused job request fails the script at once. A job that runs but writes no adapter makes the script time out after 2 hours and exit 1. Either way `ratchet.live` stops the night: no version is created, the training compute interval is closed with a "training failed" note (any GPU time the job used is still billed), and the command exits non-zero. Inside the job, `check_rows` and `check_prefix` stop a bad file or a chat-template mismatch within seconds, before the GPU trains anything (`row N: ...` or `TemplateMismatch` in the job's log).
**When the endpoint is not set.** `NebiusTrainer` checks `RATCHET_STUDENT_BASE` before it syncs or starts the paid job: an adapter with no endpoint to load into cannot be evaluated. Without it, the night stops before any spend.
### The training image
`deploy/Dockerfile.train` builds from `nvidia/cuda:12.8.0-devel-ubuntu24.04` and installs exact versions, listed again in `deploy/train-versions.txt` and read as pip constraints, so the two files disagreeing fails the build ([Deployment, step 3](/reference/deployment/#step-3-training-image)):
| Package | Version |
|---|---|
| torch | 2.11.0 (CUDA 12.8 index) |
| transformers | 4.57.6 |
| trl | 1.16.0 |
| peft | 0.21.2 |
| accelerate | 1.15.0 |
| datasets | 4.8.5 |
| causal-conv1d | 1.6.2.post1 |
| mamba-ssm | 2.3.2.post1 |
| ninja, packaging, setuptools, wheel | 1.13.0, 26.2, 82.0.1, 0.47.0 |
These are candidates read on Oct 21, 2026. The first-run smoke job (about 200 rows through `deploy/train_job.sh test`) confirms them, or its `versions.json` replaces them in both files. The last build step imports the Mamba kernels, so an image without them fails to build instead of failing on the GPU. There is no Unsloth: its Oct 19, 2026 release needs an older TRL. Build from the repository root:
```bash
docker build -f deploy/Dockerfile.train -t /ratchet-train:0.1 .
docker push /ratchet-train:0.1
export RATCHET_TRAIN_IMAGE=/ratchet-train:0.1
```
If the image cannot build at all, the documented fallbacks are an NGC PyTorch base image with the same pins, or Nebius's fine-tuning tutorial image with this repository copied in.
### Loading the adapter, and after a restart
`ratchet.live.load_adapter` posts to `{root}/v1/load_lora_adapter`, where `root` is `RATCHET_STUDENT_BASE` without its `/v1`, with the endpoint token and a 300-second timeout. The endpoint must have been started with `--enable-lora` and `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`.
vLLM **forgets runtime-loaded adapters when the endpoint restarts**, and answers 404 for them. The gateway handles that by itself: a 404 naming a version's adapter queues a reload on a background thread, at most once per adapter per minute, and writes `adapter_reloaded` or `adapter_reload_failed`. The request that hit the 404 is answered by the teacher, so it still succeeds, at teacher cost. If reloads keep failing, `python -m ratchet.live reload` loads every `live`, `canary` and `shadow` adapter by hand. The [runbook](/reference/runbook/#student-endpoint-down-or-adapters-lost) has the steps.
### When the endpoint is off: the hosted fallback
With `RATCHET_STUDENT_BASE` unset, student calls for a trained version go to : the untuned `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` on Token Factory (`RATCHET_FALLBACK_STUDENT`), paid per token instead of per GPU-hour.
- It is **not** the adapter you own and **not** covered by the adapter's precision certificate. It was never calibrated, so it is held to the global `RATCHET_CONFIDENCE_THRESHOLD` (default 0.80), not the version's calibrated threshold. Precision may be lower; the teacher's random audits will show it.
- The ledger still records these answers as `served_by: student`; the response's `model` field names the hosted model, and the pages label them "Nano 30B · hosted", never as the adapter.
- Night School cannot train or grade a new adapter in this state (see above).
### Stop the endpoint when idle, start it for the night
A stopped Serverless Endpoint is not billed for compute or its own storage; a mounted bucket still is ([Nebius Serverless overview](https://docs.nebius.com/serverless/overview), read Oct 21, 2026).
```bash
bash deploy/student_endpoint.sh stop # when idle
bash deploy/student_endpoint.sh start # the same endpoint again: same URL, same token
bash deploy/student_endpoint.sh status # prints ""
```
Night School needs the endpoint **running** while it re-scores the live adapter, loads the new one and grades it. With the endpoint stopped most of the time, start it before the night run (with `RATCHET_STUDENT_BASE` set in the night job's environment) and stop it after. Each start and stop is recorded, and answers in hours without uptime count as the hosted fallback's in the cost report.
Whether running the endpoint pays at all depends on volume. With estimated prices (an L40S at about $1.35 per GPU-hour on demand, [Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026, and about $0.001 of teacher spend avoided per ticket-triage answer, an assumption), break-even is about 1,350 student answers an hour; at a small gateway's volume a 24/7 endpoint loses money, and `python -m ratchet.cost report` says so ([cost and savings](/guides/cost-and-savings/)). At the same price, estimates from the repo's [deployment guide](/reference/deployment/#4-cost-estimates): about $0.70 to $1.35 per training night, about $32 per full day if the endpoint is left on.
### Exporting the adapter you own
You own the LoRA adapter and its signed training lineage. The base model stays under the NVIDIA Nemotron Open Model License. To hand the adapter over in one folder, with its licenses:
```bash
aws s3 sync s3:///adapters/student-v3 ./data/adapters/student-v3 --endpoint-url $RATCHET_S3_ENDPOINT # weights, if not local
python -m ratchet.live export --version 3 --out ./student-v3-export
python -m ratchet.bom verify student-v3-export/ratchet_bom.json --public-key --notice student-v3-export/NOTICE.md
```
The export checks the stored BOM against its own SHA-256 and signature first, copies the adapter files when they are on this machine, writes the BOM and the exact `NOTICE.md` it signed, and prints the BOM's `sha256`. Without the weights it says where in the bucket to copy them from. The output folder must be new or empty and outside `RATCHET_DATA_DIR` (the next sync would upload it). Keep `NOTICE.md` with the adapter wherever it goes. See [licenses](/guides/licenses/).
### What can go wrong
| Failure | What the system does | Where you see it |
|---|---|---|
| No student endpoint is set | The night stops before any paid call | `skip` or an error naming `RATCHET_STUDENT_BASE` |
| The job request is refused, or the job writes no adapter | The script exits non-zero (after 2 hours when waiting); no version is created | the night's exit code, a "training failed" note in the compute log |
| A bad training row or a chat-template mismatch | The job stops within seconds, before the GPU trains | `row N: ...` or `TemplateMismatch` in the job log |
| The endpoint restarts and forgets adapters | A 404 queues a reload; that request goes to the teacher | `adapter_reloaded` or `adapter_reload_failed` events |
| The second upload (BOM and NOTICE) fails | The version stands; the command exits non-zero | `sync_failed` event |
| Too few new examples, a release in flight, or training paused | The night is skipped | `skip` event with the reason |
| Someone erases examples after an adapter trained | The adapter is reported as possibly affected, never edited | the erasure's `affected_versions` |
### What is verified and what is not
**Checked offline, against fakes.** These parts run in the test suite, against fakes:
- `ratchet.live` commands, `NebiusTrainer`, `LiveEvaluator`, the reloader and the export: tested in `tests/test_live.py` and `tests/test_licensing.py`, against fake endpoints.
- `train_job.sh` and `student_endpoint.sh`: run in tests with a fake `nebius` CLI; the exact commands are asserted against the CLI reference as read.
- `train.py`'s arguments, row checks, loss setting and chat-template prefix check: tested with the vendored Nano 4B chat template. Its `main` needs a GPU and is not run in tests.
Ready to run, or still to check:
| Part | Status (29 Oct 2026) |
|---|---|
| The training image | Not built yet; pins are candidates |
| A real training job, vLLM serving Nano 4B with LoRA, runtime loading, LoRA on the Mamba projections | Ran once by hand on 28 Oct 2026 (spot L40S job, endpoint, hot-load), and the golden gate rejected the result; the one-command host is scripted and tested offline. The three published adapters were trained on a borrowed GPU, outside this path |
| The endpoint's arguments as the container splits them, thinking off | Not checked yet |
| Serverless pricing, the S3 endpoint URL | Unverified (the S3 URL format in the docs is an assumption) |
| Live numbers from this path | None yet; they come only from a published live run |
## For engineers
### Code
| File | What |
|---|---|
| [`ratchet/live.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/live.py) | `main` (`night`, `tick`, `reload`, `purge`, `export`, `seal`), `NebiusTrainer.train`, `load_adapter`, `AdapterReloader`, `reload_adapters`, `write_boms`, `publish_bom`, `export` |
| [`ratchet/nightschool/dataset.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/dataset.py) | `rows`, `training_record`, `write_training`, `TRAIN_TEMPLATE_KWARGS` |
| [`ratchet/nightschool/train.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/train.py) | The job's script: `parse_args`, `check_rows`, `check_prefix`, `columns`, `write_metadata`, `versions` |
| [`ratchet/nightschool/evaluate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/evaluate.py) | `LiveEvaluator`, `cascade_eval` |
| [`deploy/train_job.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/train_job.sh) | Submits the job and waits for the adapter |
| [`deploy/student_endpoint.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/student_endpoint.sh) | `create`, `start`, `stop`, `status` |
| [`deploy/Dockerfile.train`](https://github.com/harshbhx/ratchet/blob/main/deploy/Dockerfile.train), [`deploy/train-versions.txt`](https://github.com/harshbhx/ratchet/blob/main/deploy/train-versions.txt) | The pinned training image |
### Settings read by the gateway and `ratchet.live`
| Variable | Default | Meaning |
|---|---|---|
| `RATCHET_STUDENT_BASE` | empty | The endpoint's OpenAI base URL, `https:///v1`. Empty: student calls go to the hosted fallback, and Night School cannot train |
| `RATCHET_STUDENT_KEY` | empty | The endpoint's token |
| `RATCHET_STUDENT_BASE_MODEL` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | The base every adapter is trained from (also read by both scripts) |
| `RATCHET_FALLBACK_STUDENT` | `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` | The hosted, untuned model for student calls when the endpoint is not set |
| `RATCHET_CONFIDENCE_THRESHOLD` | `0.80` | The threshold the hosted fallback is held to |
| `RATCHET_DATA_DIR` | `data` | Local mirror of the bucket: `night/`, `boms/`, `adapters//` |
| `RATCHET_SYNC_CMD` | empty | Uploads `RATCHET_DATA_DIR` to the bucket, before the job and after a night that trained. Upload only: a `--delete` would remove the weights the job wrote |
| `RATCHET_TRAIN_CMD` | `bash deploy/train_job.sh {version}` | Runs training and **waits** for it; `{version}` and `{data}` are substituted |
| `RATCHET_ADAPTER_ROOT` | `/data/adapters` | Where the endpoint sees adapters (`lora_path` = root + `/` + name); also used by the gateway's reloader |
| `RATCHET_TRAIN_GPU_USD_PER_HOUR`, `RATCHET_STUDENT_GPU_USD_PER_HOUR` | `1.55` | Prices for the cost report. The code's default; these docs use 1.35 USD per GPU-hour, Nebius's list price for an L40S. Set the rate on your bill |
`RATCHET_SYNC_CMD` and `RATCHET_TRAIN_CMD` are split with `shlex.split` and run without a shell, so pipes and `&&` do not work; wrap them in a script.
### Settings read by the deploy scripts
| Variable | Used by | Default |
|---|---|---|
| `RATCHET_BUCKET_ID` | both | required (job; endpoint `create`) |
| `RATCHET_BUCKET_S3` | `train_job.sh` | empty; set it (`s3://bucket[/prefix]`) so the script waits for the adapter |
| `RATCHET_S3_ENDPOINT` | `train_job.sh` | empty; the Object Storage S3 URL (copy it from the console) |
| `RATCHET_TRAIN_IMAGE` | `train_job.sh` | required |
| `RATCHET_TRAIN_PLATFORM`, `RATCHET_TRAIN_PRESET` | `train_job.sh` | `gpu-l40s-a`, `1gpu-8vcpu-32gb` |
| `RATCHET_SUBNET_ID` | both | empty (the project's default subnet) |
| `RATCHET_REGISTRY_SECRET` | `train_job.sh` | empty; a MysteryBox secret for a private image |
| `RATCHET_STUDENT_KEY` | `student_endpoint.sh` | required for `create` |
| `RATCHET_ENDPOINT_NAME`, `RATCHET_ENDPOINT_ID` | `student_endpoint.sh` | `ratchet-student`, empty |
| `RATCHET_SERVE_PLATFORM`, `RATCHET_SERVE_PRESET` | `student_endpoint.sh` | `gpu-l40s-a`, `1gpu-8vcpu-32gb` |
| `RATCHET_VLLM_IMAGE` | `student_endpoint.sh` | `vllm/vllm-openai:v0.18.0-cu130` |
| `RATCHET_SERVE_MAX_LEN`, `RATCHET_SERVE_MAX_SEQS` | `student_endpoint.sh` | `8192`, `64` |
| `RATCHET_SERVE_TEMPLATE_KWARGS` | `student_endpoint.sh` | `{"enable_thinking":false}` |
| `RATCHET_REASONING_PARSER_PLUGIN` | `student_endpoint.sh` | empty |
| `RATCHET_PYTHON` | `student_endpoint.sh` | `python` |
| `DRY_RUN` | both | `0` |
Full descriptions: [Configuration, deploy scripts](/reference/configuration/#deploy-scripts-deploy).
### Setup, in order
```bash
export NEBIUS_API_KEY=... # Token Factory key
python scripts/day1_check.py # model ids, logprobs, JSON mode
export RATCHET_BUCKET_ID=... RATCHET_BUCKET_S3=s3:// RATCHET_S3_ENDPOINT=
export RATCHET_DATA_DIR=./data RATCHET_SYNC_CMD="aws s3 sync ./data s3:// --endpoint-url "
export RATCHET_TRAIN_IMAGE=/ratchet-train:0.1
export RATCHET_STUDENT_KEY=$(openssl rand -hex 24)
DRY_RUN=1 bash deploy/train_job.sh test # nebius validates the job, submits nothing
DRY_RUN=1 bash deploy/student_endpoint.sh create
bash deploy/student_endpoint.sh create # once
export RATCHET_STUDENT_BASE=https:///v1
export RATCHET_MODE=live RATCHET_DB=/var/lib/ratchet/ratchet.db
export RATCHET_API_KEYS="app:$(openssl rand -hex 24)" # live mode does not start without API keys
python -m ratchet.config check # exit 0 clean, 1 warnings, 2 errors
uvicorn ratchet.gateway:app --port 8000 --workers 1
```
Cron, on the gateway's host (it needs the `nebius` and `aws` CLIs, the same `RATCHET_DB` file and network access to the endpoint):
```bash
30 2 * * * cd /opt/ratchet && . ./.env && python -m ratchet.live night >> /var/log/ratchet-night.log 2>&1
*/5 * * * * cd /opt/ratchet && . ./.env && python -m ratchet.live tick >> /var/log/ratchet-tick.log 2>&1
```
### Events
`trained`, `skip`, `sync_failed`, `adapter_reloaded`, `adapter_reload_failed`, `endpoint` (an uptime change recorded), `circuit` (the student breaker). Meanings: [runbook](/reference/runbook/).
## Common questions
### Does the endpoint learn from my traffic while it serves?
No. The endpoint only answers. Training happens in a separate Serverless Job, at night, on teacher answers stored in the ledger. The student's own answers are never stored, so it never trains on them.
### Can I train on labels I already have?
Yes, as attested human labels or as an open teacher's outputs: `python -m ratchet.importer labels.jsonl --task T --source human --attest "..."` (or `--source teacher:`; closed models are refused). Users can also send corrections with `POST /v1/ratchet/feedback`. Every label is redacted and checked against the task's spec. Human labels carry the provenance `human` and are listed apart from the teachers in the BOM; an imported teacher's outputs keep that teacher and its license. See [the ledger](/guides/ledger/) and [licenses](/guides/licenses/).
### What does it cost?
Estimates only, at about $1.35 per L40S GPU-hour (Nebius's list price, read Oct 22, 2026): about $0.70 to $1.35 per night that trains (30 to 60 minutes), about $32 per full day for an endpoint left on, plus the bucket. Nights that skip cost nothing. Below about 1,350 student answers an hour, a 24/7 endpoint costs more than it saves; use the hosted fallback or stop the endpoint between uses.
### What if the endpoint restarts?
It forgets its loaded adapters. The gateway reloads each one by itself on its first 404, at most once a minute per adapter. A few requests go to the teacher meanwhile and still succeed.
### Do I have to keep the endpoint running?
No. Stop it when idle; the hosted fallback answers student calls meanwhile, labelled as such. But Night School needs it running during the night to load and grade a new adapter.
### Can I take the adapter somewhere else?
Yes. `python -m ratchet.live export` gives you the weights with the signed BOM and `NOTICE.md`. The adapter runs on the Nemotron 3 Nano 4B base, which stays under NVIDIA's license; keep that license with any copy of the base model.
### What if the training job breaks on the first real run?
That is what the day-1 checks are for. The script dry runs, the image's build-time kernel check, `check_rows` and `check_prefix` are all there to fail fast and cheaply. The fallbacks (another base image, LoRA on the attention and MLP layers only, merging the adapter into the base weights if runtime LoRA fails) are listed in `train.py`.
### What runs on Nebius today?
The teacher: it runs on Nebius Token Factory. The training job and the endpoint are tested offline, and ran once by hand on 28 Oct 2026 (the golden gate rejected that student); the three published adapters were trained on a borrowed GPU. Treat every GPU command here as a plan until the day-1 checks pass.
## Related
---
# Licenses: why only open teachers
Source: https://docs.ratchet.dualmindlab.tech/guides/licenses/
Ratchet distills only from teachers whose license allows training other models on their outputs, refuses closed models in code, records the license on every training example, and ships the licenses with every adapter, because the terms of closed models such as GPT forbid using their outputs to train competing models.
This page explains what the code does and what the license texts say, as the repo quotes them. Check each license yourself before you rely on it.
Some agency chefs have a contract that says: "**you may not copy my recipes to train a rival kitchen**." If the apprentice learned from that chef, the restaurant would be breaking the contract. So Ratchet only hires head chefs whose contract says copying is allowed. The **head chef from the agency** that Ratchet uses, Nemotron 3 Ultra, has exactly such a contract.
The pass checks every chef's contract **before the restaurant opens**. A chef with a "no copying" contract, or a chef nobody has checked, is turned away at the door, and the restaurant does not open until that is fixed.
Every page of the **recipe book** says which chef cooked the dish and which contract allows copying it. When the apprentice finishes night school, the new **notebook** gets a **certificate on the wall** that lists every chef and contract it learned from, plus a slip of paper that goes everywhere the notebook goes, saying whose rules apply.
The apprentice's own schooling came from NVIDIA, under NVIDIA's license. The notebook is yours; the apprentice's original training stays under NVIDIA's rules.
## How it really works
1. **The registry is built at startup.** `ratchet/licenses.py` holds the built-in teachers (`BUILTIN`, read-only). If `RATCHET_TEACHER_LICENSES` names a file, its entries are checked and merged into a copy, published as `licenses.REGISTRY`. A bad file stops startup with every error listed. This runs at every entry point: the gateway, `python -m ratchet.live` and `python -m ratchet.admin`.
2. **Every model that writes or judges answers must be distillable.** The startup check runs `licenses.require_distillable` on the teacher (`RATCHET_TEACHER_MODEL`), the judge (`RATCHET_JUDGE_MODEL`) and the fallback teacher (`RATCHET_TEACHER_FALLBACK_MODEL`, when set). An unknown model, or one in a closed family, raises `LicenseError` and the gateway does not start.
3. **Each stored answer records its license.** When a teacher answer is stored, the ledger row gets `teacher_model`, `license` and `license_url`: the license Ratchet holds for the configured model that answered (the teacher, or the fallback teacher), never one read from the response's `model` string.
4. **Night School re-checks provenance.** `dataset.rows` reads each row's license with `licenses.lookup_source` and drops any row whose provenance is unknown or not distillable.
5. **The adapter's bill of materials names every license.** The signed BOM lists each source (teacher, license, URL, `distill_ok`, number of examples, and `license_origin`: `builtin` or `config`), the base model's license, and the exact text of `NOTICE.md` with its SHA-256. Each set-commitment leaf binds an example's answer to its teacher, license and redaction count.
6. **The licenses travel with the weights.** The training job writes `NOTICE.md` and `ratchet_manifest.json` next to the adapter. After the night, `python -m ratchet.live night` replaces `NOTICE.md` with the text the BOM signed and syncs both into the adapter's bucket folder. `python -m ratchet.live export` bundles the adapter, the BOM and that `NOTICE.md`.
### The startup check
```mermaid
flowchart TB
A(["Teacher, judge or fallback teacher id"]) --> B{{"A provenance source, like human?"}}
B -->|"yes"| X1["Refused: not a teacher"]
B -->|"no"| C{{"In the registry?"}}
C -->|"no"| D{{"Starts with a closed prefix?"}}
D -->|"yes, like gpt"| X2["Refused: the terms are quoted"]
D -->|"no"| X3["Refused: unknown teacher"]
C -->|"yes"| E{{"distill_ok?"}}
E -->|"no"| X2
E -->|"yes"| OK["Starts: the license is kept for every example"]
class B,C,D,E gate
class X1,X2,X3 danger
class OK ok
```
**Reading this diagram.** Every model that writes or judges answers passes these checks before the gateway starts. There is one way through: a registered model whose license allows distillation. Every other path ends in a refusal, and a refusal stops startup with the reason; for a closed family such as GPT the reason quotes the terms that forbid it.
A check
Refused: the gateway does not start
Starts
### From a teacher answer to the adapter's paperwork
```mermaid
flowchart TB
T["Teacher answer"] --> E[("Example row: teacher, license, URL")]
E --> R["Night School rows"]
R --> L["Set leaf: answer, teacher, license"]
R --> J["Job writes NOTICE.md and manifest"]
L --> B["Signed BOM: sources, base license, NOTICE text"]
J -.->|"replaced by the signed text"| N["NOTICE.md next to the weights"]
B --> N
B --> X["ratchet.live export"]
class T teacher
class E store
class B student
```
**Reading this diagram.** The license is written down once, when the teacher's answer is stored, and then carried forward: into the rows Night School trains on, into each leaf of the committed training set, and into the signed bill of materials. The training job writes a first `NOTICE.md`; after the night it is replaced by the exact text the BOM signed, and `python -m ratchet.live export` hands over the adapter with both.
The teacher's answer
The ledger row
The signed lineage you own
### Why closed teachers are refused
The terms of the most used closed models forbid using their outputs to train competing models. OpenAI's [Business Terms §3.3(e)](https://openai.com/policies/business-terms/) (read Oct 21, 2026) forbid using Output "to develop artificial intelligence models that compete with OpenAI". A distillation tool that learns from GPT traffic would put its users in breach of terms like these. The README's [Why "license-clean"](https://github.com/harshbhx/ratchet#why-license-clean) section lists the other closed vendors' clauses.
So the rule is hard, not a warning: the teacher must be in the registry with `distill_ok=True`, and an override needs a reviewed code change. The cost is real: traffic that runs on closed models today cannot become training data. It has to go through an open teacher first.
### The teacher Ratchet uses: Nemotron 3 Ultra under OpenMDW-1.1
is `nvidia/Nemotron-3-Ultra-550b-a55b` on Nebius Token Factory, under [OpenMDW-1.1](https://huggingface.co/nvidia/NVIDIA-Nemotron-3-Ultra-550B-A55B-BF16). In its own words, OpenMDW-1.1 "does not impose any restrictions or obligations with respect to any use, modification, or sharing of any outputs" ([license page](https://openmdw.ai/license/1-1/), last updated May 27, 2026; checked Oct 21, 2026).
### The license registry
The built-in entries in `ratchet/licenses.py`:
| Model id | Used as | License | Note in the code |
|---|---|---|---|
| `nvidia/Nemotron-3-Ultra-550b-a55b` | Teacher (default) | OpenMDW-1.1 | Places no restrictions or obligations on outputs |
| `nvidia/nemotron-3-super-120b-a12b` | Judge (default) | NVIDIA Nemotron Open Model License | Commercial use and derivative works allowed; NVIDIA claims no ownership of outputs |
| `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` | Hosted fallback student | NVIDIA Nemotron Open Model License | As above |
| `nvidia/Nemotron-3_5-Lightning` | Listed, not used by default | OpenMDW-1.1 | Verify on the model card |
| `mock-teacher` | Tests and mock mode only | mock (tests only) | |
`CLOSED` maps name prefixes of closed model families to the terms that forbid distillation. Every OpenAI family (`gpt`, `o1`, `o3`, `o4`) carries the Business Terms §3.3(e) note, so a refusal can quote it. For a model that is not in the registry, `lookup` reads the last part of the id after `/`, in lower case, and returns the closed terms when it starts with one of those prefixes. `require_distillable` then refuses it with the license name and the clause, and refuses any other unknown model as unknown. Matching on names is a heuristic, not legal review.
### Adding an open teacher by configuration
An open teacher that Ratchet does not list yet can be added without a code change, but only within strict limits. `RATCHET_TEACHER_LICENSES` names a JSON file:
```json
[
{"model": "example-org/open-model-7b", "license": "Apache-2.0",
"url": "https://huggingface.co/example-org/open-model-7b", "note": "Checked on the model card"}
]
```
An entry is accepted only when:
- its license is one of `ALLOWED_LICENSES`: **OpenMDW-1.1, Apache-2.0, MIT, NVIDIA Nemotron Open Model License** (any case; stored in the canonical spelling);
- its model id belongs to **no closed family** (`closed_family`): the id is split at every character that is not a letter or digit, and no part may start with a closed prefix or with `o` and a digit, or contain a closed prefix of three letters or more such as `gpt` (so `chatgpt-4o-latest` is refused), a closed model's name such as `davinci` or `codex`, or a closed vendor's name such as `openai`. This is stricter than the startup lookup, and it also refuses an open model under a closed vendor's name, which has to be added in code after its license is read;
- it is **no router alias** (a router's name such as `openrouter` anywhere in the id, or a last segment of `auto`): the model that answers an alias is chosen per request and may be closed;
- it does not name a built-in teacher, does not repeat an earlier entry, has an `https://` URL, and has no other keys (at most 100 entries).
Any problem stops startup with every error and its place, for example `[2].license: 'CC-BY-NC-4.0' is not an allowed license; ...`. Configured entries carry `origin` `config`: every BOM records it per source (`license_origin`), and NOTICE.md marks the license "(configured)".
Configuration **cannot relabel a closed model Ratchet knows of**. But `closed_family` only knows the names it lists: a closed model from a vendor it does not list, under a new name, would pass the check and be admitted on the operator's claim alone. The `config` mark in the BOM and NOTICE is the safeguard there: it shows whoever receives the adapter that the license was the operator's claim, not Ratchet's reviewed registry. Widening the allowlist or loosening the check needs a reviewed code change.
### The student's base: the NVIDIA Nemotron Open Model License
The student is : `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` plus a LoRA adapter. The base model is under the [NVIDIA Nemotron Open Model License](https://www.nvidia.com/en-us/agreements/enterprise-software/nvidia-nemotron-open-model-license/) (v. Dec 15, 2025), which asks you to give anyone you distribute it to a copy of that license. `licenses.BASE_MODELS` maps the known Nano bases (4B BF16 and FP8, and the 30B-A3B variants) to that license, so the BOM can name it (`base_model_license`). For a base model Ratchet does not know, the BOM and NOTICE carry a warning instead of a license name: then you find and ship its license yourself.
### What you own
**You own this.** The LoRA adapter and its signed bill of materials are yours; the Nano base keeps the NVIDIA Nemotron Open Model License.
- **The LoRA adapter** Night School trains: the files in `adapters/student-vN/` in your bucket.
- **Its signed training lineage**: the bill of materials, signed with your Ed25519 key (`RATCHET_BOM_SIGNING_KEY`), which commits to the exact training and golden sets, every teacher and license, and the evaluation.
You do **not** own the base model: Nemotron 3 Nano 4B stays under NVIDIA's license, and the adapter runs on top of it. The code is Apache-2.0. A signed BOM is your signed statement of the data and evaluation behind an adapter: it shows that nothing changed after signing, not that the weights were trained on that data.
### Licenses travel with the adapter
Every adapter folder gets a `NOTICE.md` next to its weights:
- the base model and its license, with the license URL (or a warning for an unknown base);
- a table of teachers with their licenses, license URLs and example counts, configured ones marked;
- the statement that the adapter was trained on teacher outputs only and that the student's own answers were never training targets (or, with human labels, that it was trained on teacher answers and customer-attested labels);
- a **Human labels** section when there are any;
- the Ratchet version, a pointer to the BOM, and "Not legal advice".
The BOM carries the exact NOTICE text and its SHA-256 (`notices`), so a recipient can check that the file is the one written at training time: `python -m ratchet.bom verify ratchet_bom.json --public-key --notice NOTICE.md`. An export never re-renders the notice: it writes the text the BOM signed, so a later Ratchet that words the notice differently still ships the original. Keep `NOTICE.md` with the adapter wherever you copy, share or deploy it, and keep the base model's license with any copy of the base model.
### Human labels: a provenance source, not a teacher
Labels a person wrote (a correction sent with `POST /v1/ratchet/feedback`, or a file imported with `python -m ratchet.importer --source human --attest "..."`) are stored with provenance `human`, license "customer-provided labels" and the note "Written or approved by the customer; the customer attests ownership."
- `human` lives in `licenses.SOURCES`, not in the teacher registry. `require_distillable` refuses it by name, so no teacher can be called `human`, and a `RATCHET_TEACHER_LICENSES` entry may not use it.
- Readers of stored rows (Night School, the BOM) use `lookup_source`, which knows both.
- The BOM lists human labels apart from the teachers (`label_sources`), and NOTICE.md gets its Human labels section.
- An import of a teacher's outputs (`--source teacher:`) must pass `require_distillable` and the closed-family check; a closed model is refused with its clause quoted, and Ratchet's own `train.jsonl` rows cannot be re-imported as human labels.
### A worked example: the text-to-SQL adapter
The [text-to-SQL adapter](/guides/text-to-sql/) is the clearest case of the chain, because every link is a file you can read.
```mermaid
flowchart TB
TEA["Teacher: Nemotron 3 Ultra on Token Factory OpenMDW-1.1"] --> LAB["Labels: SQL the teacher wrote that ran and was kept"]
NONE["Closed-model outputs"] -.->|"none used"| LAB
BASE["Base: Nemotron 3 Nano 4B NVIDIA Nemotron Open Model License"] --> AD["LoRA adapter"]
LAB --> AD
LAB --> MAN[("teacher_manifest.json: teacher, license, data hashes")]
AD --> CARD["Model card and NOTICE.md name both licenses"]
class TEA teacher
class NONE danger
class AD student
class MAN store
```
**Reading this diagram.** Two licenses meet in the adapter. The base model's license comes with the weights it sits on, so a copy of it ships with the adapter. The teacher's license covers the labels, and its terms place no restrictions on using the outputs. The dotted line is the rule that matters: no output from a closed model is in the data, so there is no closed-model clause to breach. The manifest and the model card are where a reader can check each claim.
The adapters were trained on a borrowed GPU. The teacher ran on Nebius Token Factory, so the Token Factory terms question below applies to the labels, not to the training.
### The open question: Token Factory's terms
Token Factory's Terms of Service §5(f) is about using the service to "Develop a product or service that competes with the Service" (as quoted in the [threat model](/reference/threat-model/), T7). That wording is broad. The project asked Nebius support about it in writing on **Oct 19, 2026**; no answer received as of 29 Oct 2026, so treat the question as **open**. Ratchet's teacher runs on Token Factory and its student was trained and served on Nebius AI Cloud in hand-run tests, which is how it is positioned, but that is not a legal answer. Not legal advice.
## For engineers
### Code
| File | What |
|---|---|
| [`ratchet/licenses.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/licenses.py) | `BUILTIN`, `REGISTRY`, `CLOSED`, `OPENAI_TERMS`, `SOURCES`, `BASE_MODELS`, `ALLOWED_LICENSES`, `closed_family`, `lookup`, `lookup_source`, `require_distillable`, `parse_entries`, `install` |
| [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) | `Ratchet.__init__` licenses the teacher and fallback teacher; `_store_example` writes the license with each example |
| [`ratchet/config.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/config.py) | The startup check: teacher, judge and fallback teacher must pass `require_distillable` |
| [`ratchet/nightschool/notices.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/notices.py) | `render` (NOTICE.md), `bom_notice`, `write_adapter_files` |
| [`ratchet/bom.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/bom.py) | The BOM manifest, Ed25519 signing, `verify`, `check_notice` |
| [`ratchet/importer.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/importer.py), [`ratchet/feedback.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/feedback.py) | Human labels and imported teacher outputs |
Tests: `tests/test_licensing.py` (licenses that travel with the adapter, `RATCHET_TEACHER_LICENSES`), `tests/test_contract.py` (the license rules clients rely on).
### Settings
| Variable | Default | Meaning |
|---|---|---|
| `RATCHET_TEACHER_MODEL` | `nvidia/Nemotron-3-Ultra-550b-a55b` | Must pass `require_distillable` |
| `RATCHET_JUDGE_MODEL` | `nvidia/nemotron-3-super-120b-a12b` | Must pass it too |
| `RATCHET_TEACHER_FALLBACK_MODEL` | empty | Opt-in fallback teacher; must pass it |
| `RATCHET_STORE_FALLBACK_TEACHER` | `0` | `1` stores the fallback teacher's answers, under its own license |
| `RATCHET_TEACHER_LICENSES` | empty | Path to the JSON file of configured open teachers |
| `RATCHET_STUDENT_BASE_MODEL` | `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16` | Its license goes into every BOM and NOTICE |
| `RATCHET_BOM_SIGNING_KEY` | empty | Ed25519 seed (base64) that signs every BOM; empty means unsigned, with a SHA-256 |
Details: [Configuration, teacher licenses](/reference/configuration/#teacher-licenses).
### Endpoints and commands
| What | How |
|---|---|
| Ledger-wide BOM (teachers, licenses, counts, fingerprint) | `GET /ratchet/bom` |
| One adapter's signed BOM | `GET /ratchet/versions/{id}/bom` |
| The public key to publish | `GET /ratchet/bom/public-key` |
| Make a signing key | `python -m ratchet.bom keygen` |
| Check a BOM and its NOTICE | `python -m ratchet.bom verify bom.json --public-key --notice NOTICE.md` |
| Hand an adapter over | `python -m ratchet.live export --version N --out DIR` |
| Import labels | `python -m ratchet.importer FILE --task T --source human --attest "..."` or `--source teacher:` |
The BOM endpoints need an admin or dashboard key in live mode unless `RATCHET_PUBLIC_BOM=1`. Without `--public-key`, `verify` only shows that the document is consistent with the key embedded in it; publish your public key somewhere you control.
## Common questions
### Can I use my GPT traffic as training data?
No. Closed models such as GPT have terms that forbid using their outputs to train competing models, and Ratchet refuses them as teachers at startup. Route that traffic through an open teacher first; from then on, the teacher's answers are the training data.
### Can I add another open model as a teacher?
Yes, if its license is OpenMDW-1.1, Apache-2.0, MIT or the NVIDIA Nemotron Open Model License, through `RATCHET_TEACHER_LICENSES`. Any other license, or a model whose name matches a closed family even though it is open, needs a reviewed change to `ratchet/licenses.py`.
### What stops someone from configuring a closed model?
The closed-family check refuses every closed family, model name, vendor and router alias it knows. It cannot recognise names it does not list; such an entry is admitted on the operator's claim and marked `config` in every BOM and NOTICE, so the claim is visible to whoever receives the adapter.
### What exactly do I own?
The LoRA adapter and its signed training lineage. Not the base model, which stays under the NVIDIA Nemotron Open Model License.
### Do I need to ship anything with the adapter?
Yes: `NOTICE.md`, which Ratchet writes for you, and the base model's license with any copy of the base model. `python -m ratchet.live export` puts the adapter, the BOM and the notice in one folder.
### Does the judge's license matter?
The judge only scores free-text answers in the golden eval and never writes training data, but the startup check still requires it to be distillable. Super is under the NVIDIA Nemotron Open Model License.
### Is Nebius itself fine with this?
That is the open §5(f) question above: asked in writing on Oct 19, 2026; no answer received as of 29 Oct 2026.
### Is this legal advice?
No. The registry matches names and records the licenses you configure; it is not a legal review. Check each license yourself.
## Related
- [The ledger](/guides/ledger/): where each example's license is stored
- [Night School](/guides/night-school/): where the BOM is written
- [Training on your data](/guides/training-on-your-data/): the export and the bucket folder
- [Evidence](/guides/evidence/): set commitments and certificates in the BOM
---
# Cost and savings
Source: https://docs.ratchet.dualmindlab.tech/guides/cost-and-savings/
Ratchet prices every request twice, per token and by the GPU-hours actually paid, works out the traffic at which a self-hosted student endpoint breaks even, and tells you when it does not.
Every dish the head chef from the agency cooks shows up on the bill. When the apprentice chef cooks a dish instead, you skip that fee. That is the saving.
But the apprentice needs a kitchen station, and the station is rented by the hour. You pay for it while it is open, whether orders come in or not.
So the real question is not "is the apprentice cheaper per dish?" It is "does the apprentice cook enough dishes each hour to pay the station's rent?" Say the station costs $1.35 an hour and each dish the apprentice cooks saves about a tenth of a cent of head-chef fees. Then the apprentice has to cook about 1,350 dishes every hour just to break even.
On a quiet day that never happens. Then it is cheaper to close the station and let the agency's cheaper, untrained cook take the easy orders, paid per dish. Ratchet writes this on the receipt in plain words, even when the answer is "you are losing money".
## How it really works
1. **Every request records what it cost, split in two.** Each model call is priced from its token counts at a price per million tokens (`config.DEFAULT_PRICES`). The **student part** (`student_cost_usd`) covers the student's samples, for the live version and for a shadow candidate. The **teacher part** (`teacher_cost_usd`) covers the escalation, the audit, the shadow arbitration and pinned calls. Their sum is the request's `cost_usd`, which the response also returns as `ratchet.cost_usd`. A failed call adds nothing.
2. **Every request also records an all-teacher baseline.** This is what the request would have cost if Nemotron 3 Ultra had answered alone (`ratchet.all_teacher_cost_usd`). When the teacher was called, the baseline is that call's price. When the student served, it is a formula: the teacher's price for the student's prompt tokens, plus the student's output tokens and an assumed 300 reasoning tokens (`RATCHET_TEACHER_REASONING_TOKENS`). Savings are baseline minus cost.
3. **The ledger records when GPUs ran.** A self-hosted endpoint bills while it runs, so per-token prices alone flatter the student. The `compute_intervals` table holds three kinds of interval: `student_endpoint`, `training_job` and `eval_job`. Each keeps the rate it was recorded at, so changing a setting never reprices history.
4. **The cost report prices the window two ways.** The per-token view sums `cost_usd`. The GPU-hour view adds teacher spend, the endpoint's hours, the training jobs' hours, and the per-token student spend in hours when no endpoint uptime was recorded. GPU-hour savings can be negative. They are never clamped to zero.
5. **It computes break-even and the hosted crossover.** Break-even is the number of student answers per endpoint hour at which the endpoint's hours cost exactly the teacher spend they avoid. The crossover is the volume above which the endpoint is cheaper per answer than the hosted fallback (Nemotron 3 Nano 30B on Token Factory).
6. **It recommends one of four things, in plain English.** `self_host`, `hosted_fallback`, `scale_to_zero` or `insufficient_data`, with an explanation that quotes the numbers and the price label ("unverified list prices" until an operator confirms them; the per-token prices were checked on 29 Oct 2026, the GPU-hour rate was not).
```mermaid
flowchart TB
R(["One request"]) --> S["Student samples live and shadow"]
R --> T["Teacher calls: escalation, audit, arbitration, pinned"]
R --> B["All-teacher baseline"]
S --> SC["student_cost_usd"]
T --> TC["teacher_cost_usd"]
SC --> C["cost_usd"]
TC --> C
B --> V(["savings = baseline − cost"])
C --> V
class S,SC student
class T,TC teacher
```
**Reading this diagram.** Every model call a request makes is priced and lands in one of two parts: the student's samples (for the live version and a shadow) or the teacher's calls. Their sum is the request's `cost_usd`. Next to it, the request records what the teacher alone would have cost, and the saving is the difference.
The response's `usage` field is not this bill. It counts only the tokens of the answer the client received, as if that model had answered alone. `ratchet.cost_usd` prices every call Ratchet made, which `ratchet.usage_breakdown` lists.
### Savings per token and savings by GPU-hours
The two views answer different questions.
| View | Total | What it assumes |
|---|---|---|
| Per token (`per_token`, the default headline) | Σ `cost_usd` | The student costs what the hosted Nano costs per token ($0.06 in, $0.24 out per 1M). Cheap per answer, but it ignores an idle GPU |
| By GPU-hours (`gpu_hours`) | teacher spend + endpoint GPU cost + training GPU cost + student per-token spend in hours with no recorded uptime | You pay for every hour the endpoint and the training jobs ran, busy or idle |
```mermaid
flowchart TB
subgraph PT["Per token view"]
direction LR
P1["Each student answer at the hosted Nano's token price"] --> P2["Sum of cost_usd"]
P2 --> SAV1(["baseline − total"])
end
subgraph GH["GPU-hours view"]
direction LR
G1["Teacher spend"] --> G5["Total paid"]
G2["Endpoint and training hours × rate, busy or idle"] --> G5
G4["Student tokens in hours with no recorded uptime"] --> G5
G5 --> SAV2(["baseline − total can be negative"])
end
PT ~~~ GH
class G2 student
class G1 teacher
```
**Reading this diagram.** Both views start from the same baseline, what an all-teacher run would have cost, and subtract what you actually pay. They differ in what counts as paid. The per-token view charges only for tokens, so an idle GPU costs nothing in it. The GPU-hours view charges for every hour the endpoint and the training jobs ran, so at low traffic the same requests can show a saving in one view and a loss in the other. Neither is clamped to zero.
`RATCHET_STUDENT_COST_MODEL` picks which one is the report's `headline`. It changes nothing else: both views are always in the report, and each request's `cost_usd` stays per token, so history never moves.
### Where the GPU-hours come from
The report only knows the hours somebody recorded. Four writers record them:
- **The CLI**: `python -m ratchet.cost endpoint up` and `down` on the ledger file (`RATCHET_DB`). `close KIND` closes a job whose end was lost.
- **The API**: `POST /ratchet/cost/endpoint` with an admin key. Each change writes an `endpoint` event that names the admin, never the key.
- **The deploy script**: `deploy/student_endpoint.sh` runs `python -m ratchet.cost endpoint up|down --source deploy` after `create`, `start` and `stop`. It is best effort: run away from the ledger, the deploy still succeeds and prints the command to run there.
- **Night School** in live mode wraps the training run in a `training_job` interval (wall-clock time, so an upper bound on the job's billed time).
Overlapping records of one kind are merged and billed at the highest rate among them, so nothing is counted twice. An interval still open counts up to now. A training job open for more than 12 hours gets a warning, because it probably ended without anyone recording it.
Nebius words the billing like this: a stopped endpoint is not billed for compute or its own storage, but a mounted Object Storage bucket still is ([Nebius Serverless overview](https://docs.nebius.com/serverless/overview), read Oct 21, 2026).
### Break-even, first in one line
Here is the kid version again, with the numbers the docs use. The endpoint costs about $1.35 per GPU-hour (an L40S on demand, [Nebius pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026; an estimate until your bill confirms it). At list prices a typical ticket-triage call to Ultra costs about $0.0013 (about 150 input and 380 output tokens), so each such answer the student serves avoids about $0.001 of Ultra spend (an estimate). So:
```text
break-even student answers per endpoint hour = GPU price per hour ÷ teacher spend avoided per student answer
≈ $1.35 ÷ $0.001 ≈ 1,350
```
That is **about 1,350 student answers an hour, roughly 32,000 a day** (estimate). Below that line, a 24/7 endpoint costs more than the teacher spend it saves. The code's default rate is 1.55 USD per GPU-hour, which gives about 1,550 at the same $0.001 (estimate, from the formula below).
```mermaid
flowchart TB
R["GPU price per hour about $1.35, estimate"] --> BE["Break-even answers per hour = price per hour ÷ avoided teacher spend per answer"]
S["Teacher spend avoided per student answer about $0.001, estimate"] --> BE
BE --> CMP{{"Student answers served per endpoint hour"}}
CMP -->|"at or above break-even"| WIN["The endpoint pays for its hours"]
CMP -->|"below break-even"| LOSE["The endpoint costs more than it saves"]
LOSE --> OPT["Hosted fallback, or run the endpoint only while traffic arrives"]
WIN --> UP["Keep it, and scale to zero when idle"]
class CMP gate
class WIN ok
class LOSE danger
```
**Reading this diagram.** The top two boxes are the only inputs: what an hour of GPU costs and how much teacher money one student answer avoids. Dividing one by the other gives the number of answers an hour the student must serve before the endpoint is worth its rent. Everything after that is a comparison with the traffic you really have. Because the price sits on top of the division, a cheaper GPU lowers the bar in proportion. Both inputs are estimates until a bill confirms them.
### Break-even, precisely
With `R` the endpoint's price per hour as recorded in the window (`endpoint_usd_per_hour`: its GPU cost divided by its hours, or the configured `RATCHET_STUDENT_GPU_USD_PER_HOUR × RATCHET_STUDENT_GPUS` while no uptime is recorded) and `S` the student-served requests:
```text
s̄ = Σ over student-served requests (baseline − teacher spend) / S avoided teacher spend per student answer
break-even = R / s̄ student answers per endpoint hour (null when s̄ ≤ 0)
h̄ = fallback price × (all student calls' tokens) / S what the hosted fallback would bill per answer
crossover = R / h̄ above it, the endpoint is cheaper per answer
observed = student answers in buckets with uptime / endpoint hours
idle hours = endpoint hours in buckets when no request reached the student
```
A bucket is an hour on a live ledger. Break-even prices the endpoint's hours only, not training. So a window can be above break-even and still lose money by GPU-hours once the training jobs are added, and the explanation then says exactly that.
### The four recommendations
`ratchet/cost.py` has exactly four outcomes (`RECOMMENDATIONS`), decided in this order:
```mermaid
flowchart TB
A{{"Endpoint uptime recorded?"}} -->|yes| B{{"At or above break-even and the crossover?"}}
B -->|no| C{{"Idle more than half its hours?"}}
C -->|no| D{{"Has the student served anything?"}}
D -->|yes| E{{"Below the hosted crossover?"}}
B -->|yes| SH(["self_host"])
E -->|yes| HF(["hosted_fallback"])
C -->|yes| SZ(["scale_to_zero"])
E -->|"no: run fewer hours"| SZ
A -->|no| I(["insufficient_data"])
D -->|no| I
class A,B,C,D,E gate
class SH ok
class I outline
```
**Reading this diagram.** The checks run top to bottom in this order, and the first that settles it wins. Four answers are possible, and two of them can be reached in two ways: `insufficient_data` when nothing is recorded or the student has served nothing, and `scale_to_zero` when the endpoint is mostly idle or when neither it nor the hosted fallback pays at this volume. The table below says what to do for each.
| Recommendation | Means | What to do |
|---|---|---|
| `self_host` | The endpoint served at least the break-even volume per hour it ran, and more than the crossover | Keep it. If the explanation says it was often idle, scale it to zero between bursts |
| `hosted_fallback` | Below the crossover: per answer, the untuned hosted Nano would cost less than the endpoint. Below break-even the explanation opens "Self-hosting does not pay at this volume" | Leave `RATCHET_STUDENT_BASE` empty so the hosted Nano serves the student role per token, or run the endpoint only while traffic arrives |
| `scale_to_zero` | More than half the endpoint's hours (`IDLE_SHARE`, 0.5) saw no request reach the student, or neither the endpoint nor the hosted fallback pays at this volume | Stop the endpoint when idle (`bash deploy/student_endpoint.sh stop`) and start it for busy periods |
| `insufficient_data` | No uptime recorded in the window, or the student has not served a request yet | Record uptime, or wait for a trained version |
The explanation says two things separately so the words never contradict the figures: whether the observed volume covers the endpoint's hours, and whether the window lost money by GPU-hours. It never says "pays" when GPU-hour savings are negative.
### The hosted-fallback crossover, and its caveat
The hosted fallback is Nemotron 3 Nano 30B on Token Factory, untuned, paid per token. Answers it gives carry the badge. With list prices and the token counts above it costs about $0.00003 per answer, so a $1.35-an-hour endpoint beats it per answer only above roughly **45,000 student answers an hour** ($1.35 ÷ $0.00003, an estimate).
That gives a middle band. Take a customer sending 10M requests a month, about 14,000 an hour. That is far above break-even: a GPU running all month costs about $990 at $1.35 an hour (730 hours), against about $9,300 of avoided Ultra spend (an estimate). The GPU-hour savings are positive. The report still says `hosted_fallback`, worded "Self-hosting costs less than the teacher, but the hosted fallback would cost less per answer".
The caveat is printed with every `hosted_fallback` explanation (`CROSSOVER_CAVEAT`): the crossover assumes the hosted fallback serves as many requests as the tuned student. It is untuned, so more of its requests may go to the teacher, and each of those costs about $0.001 of Ultra. Ratchet does not measure the hosted model's coverage, because the two are never served side by side. Before switching, run a day on each and compare the student's share of requests in `/ratchet/stats`.
### When self-hosting does not pay
Plainly: **at low volume, a self-hosted student endpoint loses money.**
- At a few thousand requests a day, a 24/7 endpoint runs at a small fraction of break-even. Per token the same traffic can look like a large saving; by GPU-hours it can be a loss of hundreds of dollars over ten days. Both numbers are true; only the second is what you would pay.
- Below break-even the honest options are the hosted fallback (pay per token, no GPU running) or an endpoint that runs only while traffic arrives.
- Above break-even, the levers are more traffic per GPU (several tasks or customers as LoRA adapters on one vLLM endpoint) and fewer idle hours.
Ratchet never promises savings below break-even, and the cost report says so in its own words.
### Prices used
Token prices are Token Factory list prices from `DEFAULT_PRICES` in `ratchet/config.py`, checked on 29 Oct 2026: the four teacher and judge models' per-token prices match the price list Token Factory serves, and `scripts/day1_check.py` passed ([evidence/day1](https://github.com/harshbhx/ratchet/tree/main/evidence/day1)). The GPU-hour rate is still **unverified**, so every report says "unverified list prices" until an operator sets `RATCHET_PRICES_VERIFIED=1`.
| Model (key in `DEFAULT_PRICES`) | Role | Input, USD per 1M tokens | Output, USD per 1M tokens |
|---|---|---|---|
| `nvidia/Nemotron-3-Ultra-550b-a55b` | Teacher | 1.00 | 3.00 |
| `nvidia/nemotron-3-super-120b-a12b` | Judge | 0.30 | 0.90 |
| `nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B` | Hosted fallback student | 0.06 | 0.24 |
| `student` | Per-token proxy for the self-hosted student | 0.06 | 0.24 |
GPU rates come from `RATCHET_STUDENT_GPU_USD_PER_HOUR` and `RATCHET_TRAIN_GPU_USD_PER_HOUR` (with `RATCHET_STUDENT_GPUS` and `RATCHET_TRAIN_GPUS`, 1 each). Both default to 1.55 in the code. These docs use $1.35 per GPU-hour, the L40S on-demand price on Nebius's [Compute pricing page](https://docs.nebius.com/compute/resources/pricing) when read on Oct 22, 2026; that is an estimate until it matches your bill. Break-even scales with the rate. Set the variables to what Nebius actually bills you.
Estimates for running it, from the [deployment guide](/reference/deployment/#4-cost-estimates) (on-demand prices, read Oct 22, 2026):
| Item | Assumption | Estimate |
|---|---|---|
| Training run | L40S at about $1.35 per GPU-hour, 30 to 60 min a night | about $0.70 to $1.35 a night |
| Student endpoint | L40S at about $1.35 per GPU-hour | about $32 per full day if left on |
| Teacher calls | Ultra $1 / $3 per 1M tokens, about 150 input and 380 output tokens per call | about $1.30 per 1,000 calls |
- **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)).
- **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)).
- **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)).
- **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)).
- **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate.
- **The nightly schedule on live traffic.** Night School runs by hand today; the timer is scripted.
- **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline.
- **Measured savings.** Every cost figure on this page is an estimate from list prices (the L40S at $1.35 per GPU-hour is from [docs.nebius.com/compute/resources/pricing](https://docs.nebius.com/compute/resources/pricing), read Oct 22, 2026). The text-to-SQL adapter's [results](/guides/results/) include an estimated token cost on its exam rows, and that estimate has no GPU-hour bill in it. Savings are reported from a published live run, and none is published yet.
Run every script with `DRY_RUN=1` first.
### Spend and rate limits
The cost report looks back. Spend limits stop money going out in the first place. Per API key, `RATCHET_KEY_LIMITS` (over the `RATCHET_API_KEY_*` defaults) sets a daily request quota, daily and monthly US dollar budgets, requests a minute, requests in flight, the tasks the key may send, and whether its traffic may become training data. For the whole gateway, `RATCHET_DAILY_BUDGET_USD` caps `/v1` spend per UTC day; `RATCHET_BUDGET_MODE=block` then refuses every chat request, while `student_only` keeps serving confident student answers and refuses only what needs the teacher. By default a `budget` event fires at 50, 80 and 100% of every cap (`RATCHET_BUDGET_ALERT_PCT`). The counts live in memory and are read from the ledger at startup, so a restart does not reset them. Two limits on this: the dollars are per-token list prices, so endpoint GPU-hours are not part of them; and a request in flight can overshoot a cap by its own cost. Scout has its own budget guard ([Scout](/guides/scout/)).
### Savings receipts
The dashboard's savings come from the token formula, which assumes 300 reasoning tokens nobody measured. What would be billed comes from a **savings receipt** instead: one per ledger day for the gateway and one per API client, sealed after the day ends and never changed. The baseline of the answers the student served is calibrated on the audited answers' real teacher bills (a ratio estimator with a confidence bound), and the fee is `RATCHET_TAKE_RATE` (25% by default) of the one-sided 95% lower bound of the savings, never of the formula's estimate. A task with fewer than `RATCHET_RECEIPT_MIN_AUDITS` (30) audits that day is reported but not billed. Each receipt is a Merkle tree over the day's request metadata (no text), signed with the BOM key when one is set, so a client can prove any of its requests is in it and see any later edit. It cannot prove the costs logged at request time: reconcile those with the Token Factory bill. Details in [Evidence](/guides/evidence/).
## For engineers
**Code** (links to `main`):
| File | What to read |
|---|---|
| [`ratchet/cost.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/cost.py) | `cost_report` (both views, break-even, crossover, idle hours, daily rows, warnings), `segments` (merge intervals, highest rate wins), `split`, `_recommend`, `_explain`, `CROSSOVER_CAVEAT`, `RECOMMENDATIONS`, `IDLE_SHARE`, `record_endpoint` (shared by the CLI and the API), `record_simulated_day`, `stats_block`, `main` |
| [`ratchet/router.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/router.py) | `Ratchet.handle` adds each call's price to `cost_usd` and to its part; `_call_cost`; `_formula_baseline` (the student-served baseline) |
| [`ratchet/config.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/config.py) | `DEFAULT_PRICES`, `Price`, `GpuRate`, `CostPolicy`, `Settings.cost_policy()`, `COST_MODELS`, `MAX_USD_PER_GPU_HOUR` (1,000), `MAX_GPUS` (1,024) |
| [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py) | `compute_start`, `compute_stop`, `compute_record`, `compute_intervals`, `cost_by_day`, `COMPUTE_KINDS` |
| [`ratchet/nightschool/orchestrate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/orchestrate.py) | `run_night(train_gpu=...)`: the live training interval |
| [`deploy/student_endpoint.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/student_endpoint.sh) | Records uptime after `create`, `start` and `stop` |
**Settings** (from `ratchet/config.py` and [CONFIGURATION.md](/reference/configuration/)):
| Variable | Default | Meaning |
|---|---|---|
| `RATCHET_STUDENT_COST_MODEL` | `per_token` | Headline view: `per_token` or `gpu_hours` |
| `RATCHET_STUDENT_GPU_USD_PER_HOUR` | `1.55` | Student endpoint, USD per GPU-hour (0 to 1,000). The code's default; these docs use 1.35, Nebius's on-demand price for an L40S. Set the rate on your bill |
| `RATCHET_STUDENT_GPUS` | `1` | GPUs the endpoint runs on |
| `RATCHET_TRAIN_GPU_USD_PER_HOUR` | `1.55` | Training job, USD per GPU-hour. The code's default; set 1.35 or the rate on your bill |
| `RATCHET_TRAIN_GPUS` | `1` | GPUs a training job runs on |
| `RATCHET_PRICES_VERIFIED` | `0` | `1` once the day-1 checks pass and the rates match your Nebius bill |
| `RATCHET_TEACHER_REASONING_TOKENS` | `300` | Reasoning tokens the student-served baseline assumes |
A bad value stops startup with an error that names the variable.
**Endpoints** ([API.md](/reference/api/#cost)):
- `GET /ratchet/cost?days=N`: the whole report as JSON. Access level `dashboard`: in live mode it needs an admin or dashboard key unless `RATCHET_PUBLIC_DASHBOARD=1`.
- `POST /ratchet/cost/endpoint`: body `{"state": "up"}` or `"down"`, with optional `usd_per_hour`, `gpus` and `note`. Admin key; admin key required.
- `GET /ratchet/stats`: a compact `cost` block (`model`, `endpoint_hours`, `gpu_hours_savings_pct`, `break_even_student_requests_per_gpu_hour`, `recommendation`).
**CLI**:
```bash
python -m ratchet.cost report [--days 7] [--json] # both views, break-even, crossover, recommendation
python -m ratchet.cost endpoint up --note "recording" # also --usd-per-hour, --gpus, --at TIME
python -m ratchet.cost endpoint down
python -m ratchet.cost close training_job --at 2026-10-22T03:10:00Z # a job whose end was lost
```
The CLI opens an existing ledger file only (`--db` or `RATCHET_DB`), exits 2 with `error: ...` on bad input, and refuses an `--at` in the future.
**Events and warnings**: an `endpoint` event on every recorded change; report warnings for a job open more than 12 hours, for student answers in hours with no recorded uptime (priced per token as the hosted fallback's), and for rows from before v0.5 that have no student/teacher split (counted as teacher spend).
## Common questions
### Why do the dashboard and the cost report show different savings?
The dashboard's figures are the per-token view: the student priced at the hosted Nano's token price. The cost report adds the GPU-hour view, which charges every hour the endpoint ran. At low volume the first is positive and the second negative. Both are correct; read the GPU-hour one before deciding to self-host.
### What does it cost to run Ratchet?
The gateway does no inference, so it runs on a small CPU VM (about $72 for Oct 29 to Dec 15, 2026, estimate, [VM kit guide](/reference/deploy-nebius-vm/)). On top of that: teacher calls per token, a training job per night (about $0.70 to $1.35, estimate), and the student endpoint while it runs (about $32 per full day, estimate), both at about $1.35 per GPU-hour. All of these are estimates until a live run measures them.
### At what traffic does self-hosting pay?
About 1,350 student-served answers per endpoint hour at about $1.35 per GPU-hour, roughly 32,000 a day (an estimate from list prices and the token counts above). Below that, use the hosted fallback or run the endpoint only while traffic arrives.
### Is the hosted fallback just cheaper, then?
Per answer, often yes: below roughly 45,000 answers an hour it is (estimate). But it is untuned, so it may escalate more requests to Ultra, and those cost far more than the GPU-hours saved. The report always says the crossover assumes equal coverage. Compare a day on each before switching.
### What if nobody records the endpoint's uptime?
Then the report does not know the GPU cost. Student answers in hours with no recorded uptime are priced per token and counted in a warning. Use `deploy/student_endpoint.sh` next to the ledger, or record `up` and `down` yourself, and compare `endpoint_hours` with the Nebius bill weekly.
### Are these real numbers?
Not yet. The per-token prices were checked on 29 Oct 2026 against the price list Token Factory serves (`/v1/models?verbose=true`) and match; the GPU-hour rate is still unverified, and every figure on this page is an estimate. As of Oct 29, 2026 the teacher runs on Nebius Token Factory and Ratchet's own adapter is not connected to the live gateway (one was served in speed tests and deleted), where a random 40% of eligible requests are answered by the hosted Nano 30B; the live-run harness will publish measured numbers, including the Nebius bill.
### Does the baseline overstate savings?
It can. The student-served baseline assumes the teacher would have written 300 reasoning tokens. Audits measure what the teacher really bills, and the savings receipts and the ledger replay correct the baseline with them. The dashboard's figures do not.
## Related
The cost model in full: [Architecture, cost accounting](/reference/architecture/#7-cost-accounting).
---
# Evidence you can check
Source: https://docs.ratchet.dualmindlab.tech/guides/evidence/
Ratchet writes its claims about each adapter and each day's savings into signed documents whose numbers can be recomputed offline, so a customer or auditor can check them instead of trusting the gateway that served them.
A restaurant can say anything about itself. Ratchet hands you things you can check yourself, at home, without asking the restaurant again.
- **The certificate on the wall** (the signed bill of materials) lists which chefs taught the apprentice, under which licenses, how many recipes were studied and how the apprentice did on the sealed exam. It carries the restaurant's stamp. Anyone with a copy of the stamp's pattern can tell if a single word was changed after stamping.
- **The exam score with a safety margin** (certified precision) does not say "the apprentice got 99%". It says "even if we were unlucky with the exam questions, the real score is at least this".
- **The fingerprint of the recipe book** (a committed training set) lets you check that one recipe was in the study pile, or was not, without seeing the whole book. It also proves that no dish from the sealed exam was ever studied.
- **The time machine** (the ledger replay) answers "what would last week have looked like with a different bar?".
- **The receipt** (the savings receipt) is each day's bill, counted on the cautious side, with a fingerprint of every order on it. Each receipt includes yesterday's fingerprint, so changing an old receipt breaks the chain.
Each piece also says plainly what it cannot prove. A stamp shows nobody changed the certificate; it does not show the certificate was true when it was written.
## How it really works
The evidence is written at fixed moments, by code that draws no random numbers and calls no model, so it never changes a release decision.
1. **The night, before training.** Night School builds a set commitment (a sparse Merkle root) over the exact training rows and another over the golden rows, before the paid training job starts, so a problem fails the night before any GPU time.
2. **The night, after training.** Right after the new version is created, the leaves behind both roots are written once to the ledger's `version_leaves` table, which refuses any later change. The adapter is scored on the golden set (with its certificate), the live version is re-scored on the same rows, the paired comparison is added, and only then is the bill of materials written and signed. In live mode a copy goes into the adapter's folder in the bucket, with its `NOTICE.md`.
3. **Every request.** Each request the student answers logs the threshold it was held to, the probability that the teacher labels it (`label_p`), whether the student agreed with that label, and the token formula's baseline. This is what the replay and the receipts read.
4. **After each ledger day ends.** The day's savings receipts are sealed, once: one for the whole gateway and one per API client, each signed and chained to that client's previous receipt.
5. **Any time later, anywhere.** A customer downloads the documents and checks them with the command-line tools in this repository, using only the files and a public key the operator published.
| Evidence | What it proves | What it does not prove | Check it with `python -m` |
|---|---|---|---|
| Signed BOM | The operator signed these claims about this adapter at training time, and nothing changed since | That the weights were trained on that data | `ratchet.bom verify` |
| Certificate | With probability at least 95%, what this version serves agrees with the teacher at least `lcb` of the time, on traffic like the golden rows | Correctness, or behaviour on traffic unlike the golden rows | `ratchet.stats check` |
| Committed training set | This example was, or was not, in the committed training or golden set | That the weights came from that set | `ratchet.proof verify` and `heldout` |
| Ledger replay | An estimate of precision, coverage and cost at another threshold or audit rate | Anything signed: it reads the ledger as it is, and its interval is approximate | `ratchet.replay` |
| Savings receipt | The day's leaves were not edited after sealing, and the bill follows from them | That the costs written at request time were right | `ratchet.receipt verify` |
```mermaid
sequenceDiagram
participant N as Night School
participant Ledger as Ledger
participant G as Gateway
participant C as Auditor
N->>Ledger: commit the rows
N->>N: train and sign
N->>Ledger: write leaves once
G->>Ledger: log each request
G->>Ledger: seal each day
C->>G: fetch the files
C->>C: check offline
```
**Reading this diagram.** The first three arrows happen once a night: Night School commits the training and golden rows (their Merkle roots), trains, scores the adapter with its certificate, re-scores the live version, signs the bill of materials, and writes the leaves once (append-only). The next two happen all day: the gateway logs every request with its label probability and seals each ended day into a signed receipt. So the evidence is made by the same system that does the work, but at fixed moments that cannot change a decision. The last two arrows are the point: a customer or auditor fetches the BOM, proofs, receipts and public key, and checks them offline without calling the gateway again. [Proof you can run](/guides/proof/) lists the one command for each check.
A separate, simpler check exists for the text-to-SQL adapter: its exam is scored by running queries, and `measure_adapter.py verify` recomputes every number from the raw answers. That one does not need a gateway at all. See [A text-to-SQL adapter you can check](/guides/text-to-sql/) and [Results](/guides/results/).
All the arithmetic comes from two standard-library modules, [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py) and [`ratchet/merkle.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/merkle.py), checked against hand-computed values and seeded Monte Carlo runs in `tests/test_stats.py` and `tests/test_provenance.py`.
## Signed bills of materials
**In one sentence for a 10-year-old:** a bill of materials is a list of everything that went into one adapter, signed so that anyone with the public key can tell if a single character was changed after it was written.
**Precisely.** For every adapter it trains, Night School writes one document, schema `ratchet.bom/v2`, from the exact rows it used:
- `adapter`: the version, the adapter's name, its base model (`nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16`) and the ledger day.
- `training`: the number of examples, the dataset hash, examples per task, `targets` (`teacher answers only`, or with customer-attested human labels), how many examples had PII redacted, where the examples came from, and the training set's commitment (`{alg: ratchet.smt/v1, root, size}`).
- `golden`: the number of golden rows, their hash, `held_out: true`, golden rows with a human reference, and the golden set's commitment.
- `sources`: each teacher with its license, license URL, whether it allows distillation and whether the license entry is Ratchet's own (`builtin`) or added by the operator (`config`); `label_sources` lists human labels apart, because a person is not a teacher.
- `base_model_license`: the NVIDIA Nemotron Open Model License and its URL for the Nano base (or a warning for a base model Ratchet does not know).
- `notices`: the exact text of the `NOTICE.md` that sits next to the adapter's weights, with its SHA-256.
- `redactor`: a SHA-256 over the redaction rules, how many there are, and whether redaction was on.
- `eval`: the golden eval at training time: precision, coverage, threshold, and since v0.5 the certificate, the frontier, the per-task bounds, the per-row outcomes and the paired comparison with the live version.
The manifest is serialised as canonical JSON (sorted keys, no whitespace, UTF-8). The document carries the SHA-256 of those bytes and, when `RATCHET_BOM_SIGNING_KEY` is set, an **Ed25519** signature with the public key and a key id. It is written once and never rewritten: a later re-check changes `/ratchet/stats`, not the signed document. Versions trained before v0.4 have no BOM; v1 documents still verify.
```mermaid
flowchart TB
TR[("Training rows")] --> R1["Training set root"]
GR[("Golden rows")] --> R2["Golden set root"]
F["Teachers and licenses, base model license, NOTICE.md text and hash, redaction rules hash, golden eval, certificate"]
TR --> M["Manifest"]
GR --> M
R1 --> M
R2 --> M
F --> M
M --> CJ["Canonical JSON"]
CJ --> H["SHA-256"]
CJ --> SG["Ed25519 signature"]
H --> DOC["BOM document"]
SG --> DOC
M --> DOC
class TR,GR store
class DOC student
```
**Reading this diagram.** Everything at the top goes into one manifest: the counts and hashes of the exact rows, the roots that commit to each set, and the facts about the adapter. The manifest is written as canonical JSON, and that one byte string is both hashed and signed. The document you receive carries the manifest, its SHA-256 and the signature, so changing any byte of it breaks the check.
**Check it:**
```bash
python -m ratchet.bom keygen # once: the signing key, and the public key to publish
curl -s localhost:8000/ratchet/bom/public-key # the key the gateway signs with
curl -s localhost:8000/ratchet/versions/2/bom > bom.json
python -m ratchet.bom verify bom.json --public-key # OK: signature valid (Ed25519, key ..., trusted key)
python -m ratchet.bom verify bom.json --public-key --notice NOTICE.md # and the NOTICE.md next to the adapter
python -m ratchet.live export --version 3 --out ./student-v3-export # hand an adapter over with its BOM and NOTICE.md
```
Pass the public key you published somewhere you control (a README, a website). Without `--public-key`, `verify` uses the key embedded in the document, which only shows the document is consistent with itself. Without a signing key, the document carries only its SHA-256: `verify` reports it as unsigned. Rotating the key does not invalidate old BOMs; keep every public key you have published listed.
**What it does not prove.** A signed BOM is the operator's signed statement of the data and evaluation behind an adapter. It shows that nothing changed after signing, not that the weights were trained on that data, and not that the claims were true: a compromised signing key can sign anything. When examples are erased later, the versions that may have been trained on them are reported, and their BOMs stay exactly as written.
## Certified precision
**In one sentence for a 10-year-old:** a certificate is the exam score with a safety margin: "even if we were unlucky with the exam questions, the real score is at least this".
**Precisely**. The golden eval picks the threshold on the even half of the golden rows and measures on the odd half (see [Confidence](/guides/confidence/)). The odd half played no part in choosing the threshold, so its count of agreeing answers among the `m` it would serve is one exact binomial experiment. The one-sided Clopper–Pearson lower bound is the precision `p` at which getting `s` or more agreeing answers out of `m` would happen only `delta` of the time:
```text
lcb = cp_lower(s, m, delta) # the p with P(X >= s) = delta, X ~ Binomial(m, p)
certified = lcb >= target # RATCHET_CERT_TARGET 0.95, RATCHET_CERT_DELTA 0.05
```
"Certified at 95% (lcb 96.4%, delta 0.05)" means: with probability at least 95%, the answers this version serves at its threshold agree with the teacher at least 96.4% of the time. For example, 129 agreeing out of 130 served held-out rows gives a point estimate of 99.2% and a lower bound of 96.4% (checked with `stats.cp_lower`). The point estimate alone cannot tell 0.96 from 0.99 on that many rows; the bound can.
The eval also carries:
- **The frontier:** the same counts and bound at every threshold of the fixed grid 0.95, 0.90, down to 0.50, each bound at `delta / 10`, so all ten hold at once (Bonferroni) and an operator may read any row afterwards: a certified trade-off between precision and the share of traffic served.
- **Per task:** the counts and bound per task, so a weak task cannot hide inside a pooled number. A diagnostic, not corrected for the number of tasks, and never used for routing.
- **The test vector:** one letter per held-out row (`S` served and right, `W` served and wrong, `E` escalated), which the paired comparison with the live version reads, and which `check` uses to recount everything.
How many rows a certificate needs, from `stats.required_rows`:
| Target (delta 0.05) | 0 wrong answers | 1 | 2 | 3 |
|---|---|---|---|---|
| 0.90 | 29 | 46 | 61 | 76 |
| 0.95 | 59 | 93 | 124 | 153 |
| 0.97 | 99 | 157 | 208 | 257 |
Only served held-out rows count, so certifying 95% with one wrong answer takes about `2 × 93 / coverage` golden rows.
**Check it:**
```bash
python -m ratchet.stats plan --target 0.95 --delta 0.05 # 0 errors: 59 rows, 1: 93, 2: 124, 3: 153
python -m ratchet.stats bound --successes 127 --n 128 # one bound by hand: 0.9635
curl -s localhost:8000/ratchet/versions/2/certificate # signed and latest certificate, the gate's evidence, assumptions
curl -s localhost:8000/ratchet/versions/2/bom > bom.json
python -m ratchet.stats check bom.json --public-key # recompute every number from the signed counts
```
`check` verifies the signature, then recomputes every bound, verdict, frontier row, per-task row, Learn-then-Test step and paired p-value from the signed counts, and the counts from the test vector. It exits 1 on any difference above 1e-9. The certificate endpoint's `status` is `certified` or `not certified` from the latest certificate (the nightly re-score's, else the signed one), `suspended` while a drift alarm is open on the version, or `no certificate` for a version evaluated before v0.5.
**What it does not prove.** Precision means agreement with Nemotron 3 Ultra () under the task's comparators, not ground truth. The bound assumes the golden rows look like live traffic, and they lean towards requests the student found hard (they come from escalations, audits and arbitrations). The guarantee is about the traffic the golden rows were drawn from, when they were drawn; drift breaks it. Every certificate is served with these assumptions in plain words.
## Committed training sets
**In one sentence for a 10-year-old:** a Merkle root is a fingerprint of a whole list that lets you prove one item is on it, or is not, without showing the list.
**Precisely**. Every BOM since v0.5 signs one root per split, over exactly the rows Night School handed to the trainer and to the golden eval:
- **Key:** the example's hash, a SHA-256 over the task, the redacted messages and any stored generation conditions. Anyone holding the request can recompute it.
- **Value:** the SHA-256 of `{completion, teacher, license, redactions}`: the stored teacher answer the student trained on, who wrote it, its license and the number of PII placeholders. A proof therefore binds what an example was trained on, not only that its key was there.
- **Tree:** a collapsed sparse Merkle tree split on the key's bits. An empty node is 32 zero bytes, a node with one key is `sha256(0x00 || key || value)`, any other node is `sha256(0x01 || left || right)`. The root depends on the set alone, never on insertion order.
- **Proof:** the sibling hashes along the key's path, plus where the path ends: the key's own leaf (included), an empty node, or another key's leaf that shares the path (absent).
The **held-out certificate** is the golden set's leaves plus, for every golden key, a proof of absence from the training root: two of the loop's rules (golden rows are never trained on; precision is measured on rows the model did not see) as something you verify, not something you are told.
Absence is sound even against whoever built the tree: a key's path is fixed by its bits, so a proof of inclusion and a proof of absence for the same key against the same root would need a SHA-256 collision. A Merkle tree over a sorted list is not used for absence, because a verifier cannot check that the list was really sorted.
```mermaid
flowchart TB
Req(["The request: task and messages"]) --> Key["Example key"]
PF(["Proof: siblings and terminal"]) --> Fold["Hash up from the terminal along the key's bits"]
Key --> Fold
Fold --> CR["Computed root"]
BOM(["The adapter's BOM"]) --> Sig{{"Signature valid with your public key?"}}
Sig -->|"no"| Fail["FAIL"]
Sig -->|"yes"| SR["Signed training root"]
CR --> Cmp{{"Same root?"}}
SR --> Cmp
Cmp -->|"no"| Inv["invalid"]
Cmp -->|"yes, terminal is the key"| Inc["included"]
Cmp -->|"yes, terminal is empty or another key"| Abs["absent"]
class Sig,Cmp gate
class Fail,Inv danger
class Inc,Abs ok
```
**Reading this diagram.** Two roots meet at the bottom. On the left the verifier rebuilds a root from the request and the proof, hashing up from the terminal one level at a time, deepest sibling first. On the right it takes the training root from the BOM, but only after the BOM's signature checks out with your public key. If the two roots are the same, where the proof's path ends says whether the example was included or is absent.
A check
A valid answer: included or absent
The proof fails
The verifier never trusts the root, result or value written in the proof file; it checks against the root the signed BOM names.
**Check it:**
```bash
curl -s localhost:8000/ratchet/versions/2/bom > bom.json
# ex.json: the body of a request that was stored as an example (task and messages)
curl -s "localhost:8000/ratchet/versions/2/proof?split=train" -H 'content-type: application/json' -d @ex.json > proof.json
python -m ratchet.proof verify proof.json --bom bom.json --expect included
curl -s localhost:8000/ratchet/versions/2/heldout > heldout.json
python -m ratchet.proof heldout heldout.json --bom bom.json # OK: all N golden examples provably absent from the training set
python -m ratchet.proof key --task ticket-triage --messages request.json # the key a request is stored under
```
Add `--public-key ` to `verify` and `heldout` to trust the signer, not just the document. The proof (`POST /ratchet/versions/{id}/proof`) and held-out (`GET /ratchet/versions/{id}/heldout`) endpoints need an admin key in live mode (`Authorization: Bearer $ADMIN_KEY`); They are admin-only on purpose: for a text someone can guess, a yes-or-no answer about the training set is a leak.
A proof is about log2 of the set's size in siblings. A held-out certificate for 133 golden rows against 1,288 training leaves is about 113 KB of JSON.
**What it does not prove.** The proofs show the operator's binding commitment, signed at training time, to the dataset an adapter was trained and graded on. They do not show that the weights came from that dataset: an operator who signs one set and trains on another is not caught by a Merkle proof. A proof of absence also names the neighbouring training example's key and value hashes, so held-out certificates go to auditors, not to the public.
## Ledger replay
**In one sentence for a 10-year-old:** because Ratchet decides which plates the inspector tastes by a coin flip with known odds, each tasted plate can stand in for the untasted plates like it, so last week can be replayed with a different bar.
**Precisely**. Ratchet designs its own labelling, so it knows the chance of every teacher label it sees: 1 for a request the student answered and escalated, the audit rate for an answer it served. Each request logs that probability (`label_p`), the threshold it was held to, and whether the student's answer agreed with the teacher's when the teacher answered under an audit's conditions (`student_agree`). For one version, the replay then computes at any threshold `t` or audit rate `a`:
- **coverage:** exact, from the logged confidences, no label needed;
- **precision:** a weighted mean of `student_agree` over labelled served requests, each weighted by `1 / label_p` (Hajek), with an approximate interval (Wilson at Kish's effective sample size); null, with a reason, when nothing could label the answers it would count (an audit rate of 0);
- **cost and savings:** student spend, plus the all-teacher cost of every request not served, plus the audit rate times that of every request served;
- **a frontier** from 0.30 to 1.00 in steps of 0.01, and an advisory suggested threshold, which nothing applies: thresholds change only through Night School;
- **a calibrated baseline:** the ratio of what the teacher really billed on audited answers to the token formula's estimate (which assumes 300 reasoning tokens, `RATCHET_TEACHER_REASONING_TOKENS`).
**Check it:**
```bash
curl -s "localhost:8000/ratchet/replay?threshold=0.4&audit_rate=0.05" # what happened, and what would have
curl -s "localhost:8000/ratchet/replay/frontier?target=0.97" # thresholds 0.30 to 1.00, an advisory suggestion
python -m ratchet.replay --db ratchet.db # the same offline, from a ledger file
```
**What it does not prove.** The replay is an analysis of the ledger as it stands, not a signed document. Its interval is approximate and computed on the same traffic the operator is looking at, so it is never a certificate and never a release decision. An escalation answered under a client's own sampling parameters, or for more than one choice, gives no label. A request whose escalation failed outright was never logged, so it is not in the replay. It replays one version at a time, because each adapter's confidence scale is its own.
## Verified savings receipts
**In one sentence for a 10-year-old:** a receipt is the day's bill with a fingerprint of every order on it, and each receipt carries yesterday's fingerprint, so changing an old receipt breaks the chain.
**Precisely**. The plan for a managed offering is to charge a share of verified savings (nothing is sold today), so the savings must be checkable by the customer. For each ledger day there is one receipt for the whole gateway and one per API client:
1. **Leaves.** Each request of the day becomes one leaf of metadata, never text: `id`, `ts_ms`, `day`, `task`, `client` (the API key's name, never the key), `served_by`, `route`, `student_version`, `escalated`, `audited`, `label_p_ppb` and the amounts (`cost_nusd`, `student_cost_nusd`, `teacher_cost_nusd`, `baseline_nusd`, `baseline_est_nusd`). Amounts are whole nano-USD, so sums are exact and the signed JSON holds no floating-point number.
2. **Groups.** A request is **measured** when the teacher answered it (its bill is the baseline), **served** when the student served it on the audit design, and **unverified** when a stand-in answered (route `degraded` or `fallback_teacher`).
3. **Calibration per task.** For each task, the audited served answers give a ratio `R` between the real teacher bills and the formula's estimates (weighted by `1 / label_p`), with a variance by the normal approximation. A task needs `RATCHET_RECEIPT_MIN_AUDITS` (30) audits that day; below that it is `insufficient_audits` and its savings are reported but not billed.
4. **The bill.** The billable savings (measured requests plus the calibrated tasks' served answers, minus any stand-in losses) less a margin of `ceil(1.645 × sqrt(sum of variances))` give the one-sided 95% **lower bound**. The fee is `RATCHET_TAKE_RATE` (0.25) of that bound, after the day's spend outside requests, when it is positive, rounded down to a nano-USD. With no calibrated task the basis is `formula` and the fee is 0.
5. **Seal.** The leaves form an RFC 9162 Merkle tree (the Certificate Transparency kind). The receipt names its root and size, its totals, the calibration and the bill, and `prev_sha256`, the hash of the same client's previous receipt (64 zeros for the first). It is signed with the BOM key and written once, after the day has ended (on a live ledger, 5 minutes after). The ledger refuses any later change.
```mermaid
flowchart TB
D1["Day 1 receipt"] -->|"its sha256 is"| D2["Day 2 receipt: prev_sha256"]
D2 -->|"its sha256 is"| D3["Day 3 receipt: prev_sha256"]
L[("Day 3 request leaves")] --> T["RFC 9162 tree root"]
T --> D3
class L store
```
**Reading this diagram.** Each receipt names the hash of the same client's previous receipt, so the receipts form a chain: change day 1 and day 2's `prev_sha256` no longer matches. Each receipt also names the root of a Merkle tree over that day's request metadata, which is what lets one request be proved to be in it.
Every chat response carries its ledger row id in `X-Ratchet-Request-Id` and `ratchet.row_id`, so a client can later ask for a proof that its request is in that day's receipt. With API keys on, a key reads only its own receipts and proofs and an admin key reads all of them; live mode without any keys serves no receipts, because the leaves list every request's time, task and cost. The gateway-wide receipt also shows the day's savings by GPU-hours paid, which are often negative at low volume. That view is informational (billing uses the per-token view); the [cost report](/guides/cost-and-savings/) is what says when self-hosting does not pay.
When the teacher's real reasoning differs from the formula's assumed 300 tokens, the receipts measure the ratio `R` from audited answers and correct the baseline.
**Check it:**
```bash
curl -s localhost:8000/ratchet/receipts/3 > r3.json # one day's receipt (your own, with an API key)
curl -s localhost:8000/ratchet/receipts/2 > r2.json # the day before, for the chain
curl -s localhost:8000/ratchet/receipts/3/proof/3006 > p.json # a row id you saw (ratchet.row_id) is in it
curl -s localhost:8000/ratchet/receipts/3/leaves > leaves.jsonl # every leaf: recompute everything
python -m ratchet.receipt verify r3.json --public-key --prev r2.json --proof p.json --rows leaves.jsonl
python -m ratchet.live seal # live: seal every day that ended (hourly cron)
```
`verify` checks the hash and signature, that no floating-point number is in the document, the chain (it recomputes the previous receipt's hash rather than reading it), the request's inclusion against the signed root and size, and with `--rows` rebuilds the root and recomputes every total, estimate and the fee. It exits 0 when everything holds, 1 when anything fails, 2 on bad input.
**What it does not prove.** Receipts make edits after sealing detectable and tie the baseline to audited teacher bills. They cannot prove the costs written at request time: reconcile the summed teacher spend (`totals.teacher_cost_nusd` over the days) with the Token Factory bill for the same API key. The interval is a normal approximation, and heavier-tailed real bills may need more than 30 audits. The chain shows that a receipt was changed; it does not stop an operator from withholding one, so clients should keep the receipts they receive. Request ids are the ledger's global row ids, so a client can infer the gateway's total volume between two of its own requests.
As of Oct 29, 2026, no day has been sealed on a live gateway and no live-traffic BOM or receipt has been signed. The one Night School run on Nebius (28 Oct 2026) wrote an unsigned bill of materials for a student the gate rejected ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)); that BOM was signed afterwards, on 29 Oct 2026, with Ed25519 key `81b0a086250a81d0`, and its content is unchanged from the run's own record (check it: `python -m ratchet.bom verify evidence/nightschool-nebius/ratchet_bom.signed.json --public-key Gh68hnDDlRSjBRxmc5dTo4l+rvqkftUsH7B2Nm72gpc= --notice evidence/nightschool-nebius/NOTICE.md`); the GPU steps have otherwise run only by hand. Real teacher calls on Token Factory built the text-to-SQL data, but no BOM, certificate, proof, replay or receipt has been produced on live traffic. The formats, the signatures and the verifiers are tested offline.
## For engineers
| What | Where |
|---|---|
| BOM, signing and verification | [`ratchet/bom.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/bom.py): `manifest`, `canonical`, `document`, `verify`, `check_notice`, `commitment`, `Signer`, `keygen`; `python -m ratchet.bom keygen` or `verify` |
| NOTICE.md | [`ratchet/nightschool/notices.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/notices.py): `render`, `bom_notice`, `write_adapter_files` |
| Certificates | [`ratchet/certificate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/certificate.py): `certificate`, `frontier`, `per_task`, `outcome_vector`, `paired`, `recheck`, `status`, `ASSUMPTIONS` |
| Exact statistics | [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py): `cp_lower`, `required_rows`, `mcnemar_one_sided`, `ltt_fixed_sequence`, `CERT_GRID`, `hajek_mean`, `kish_n_eff`, `ratio_estimate`; `python -m ratchet.stats plan`, `bound`, `mcnemar`, `check` |
| Merkle trees | [`ratchet/merkle.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/merkle.py): `KeySet`, `verify` (sparse tree), `LogTree`, `verify_inclusion` (RFC 9162) |
| Set commitments | [`ratchet/nightschool/dataset.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/nightschool/dataset.py): `leaf_value`, `commitment`; [`ratchet/proof.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/proof.py): `request_key`, `resolve_key`, `verify_proof`, `verify_heldout`; `python -m ratchet.proof key`, `verify`, `heldout` |
| Replay | [`ratchet/replay.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/replay.py): `load`, `at`, `sweep`, `frontier`, `ratio`, `suggest`; `python -m ratchet.replay` |
| Receipts | [`ratchet/receipts.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/receipts.py): `leaf`, `calibrate`, `compute`, `seal_due`, `seal`, `verify_receipt`, `verify_chain`, `verify_proof`, `recheck`; `python -m ratchet.receipt verify` or `seal` |
| Append-only tables | [`ratchet/ledger.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/ledger.py): `version_leaves` and `receipts`, refused UPDATE and DELETE by trigger |
| Tests | `tests/test_stats.py`, `tests/test_provenance.py`, `tests/test_licensing.py`, `tests/test_receipts.py`, `tests/test_nightschool.py` |
| Variable | Default | Effect |
|---|---|---|
| `RATCHET_BOM_SIGNING_KEY` | empty | Base64 of a 32-byte Ed25519 seed; signs every BOM and every receipt. Empty: unsigned documents |
| `RATCHET_PUBLIC_BOM` | `0` | `1`: the BOMs stay public while the dashboard API is private, so auditors need no key |
| `RATCHET_ADMIN_KEYS` | empty | Admin keys; needed for the proof and held-out endpoints and for every receipt in live mode |
| `RATCHET_CERT_TARGET`, `RATCHET_CERT_DELTA` | `0.95`, `0.05` | The certificate's target and error rate |
| `RATCHET_CALIBRATION` | `point` | `ltt`: the threshold itself is certified (Learn-then-Test) |
| `RATCHET_REDACT_PII` | `1` | Redaction before storage; an example's key is computed on the redacted text (`python -m ratchet.proof key --no-redact` for a gateway with it off) |
| `RATCHET_TAKE_RATE` | `0.25` | The fee's share of a receipt's billable lower bound |
| `RATCHET_RECEIPT_MIN_AUDITS` | `30` | Audits a task needs in one receipt before its baseline is calibrated and billed |
| `RATCHET_TEACHER_REASONING_TOKENS` | `300` | The formula's assumed teacher reasoning, which the replay and the receipts measure |
Endpoints ([API](/reference/api/)): `GET /ratchet/bom` (the whole ledger's provenance), `GET /ratchet/versions/{id}/bom`, `GET /ratchet/bom/public-key` (always public), `GET /ratchet/versions/{id}/certificate`, `POST /ratchet/versions/{id}/proof?split=train` (admin), `GET /ratchet/versions/{id}/heldout` (admin), `GET /ratchet/replay`, `GET /ratchet/replay/frontier`, `GET /ratchet/receipts`, `GET /ratchet/receipts/{period}`, `GET /ratchet/receipts/{period}/proof/{request_id}`, `GET /ratchet/receipts/{period}/leaves`. See [API: committed training sets](/reference/api/#committed-training-sets), [ledger replay](/reference/api/#ledger-replay) and [savings receipts](/reference/api/#savings-receipts).
Events: `integrity` (stored leaves or request rows no longer rebuild a signed root: the endpoint answers 500 and never serves a proof that would fail), `receipt_skipped` (a day's request rows were folded by retention before it was sealed). The runbook covers [checking a certificate](/reference/runbook/#checking-a-certificate) and [proving what an adapter was trained on](/reference/runbook/#proving-what-an-adapter-was-trained-on).
## Common questions
### What does a customer need to check all this?
The documents (a BOM, a proof file, a receipt and its neighbours), the public key the operator published somewhere other than the gateway, and this repository's command-line tools. No access to the gateway, the ledger or the training data.
### Can the operator fake it?
The operator can sign false claims: a signature shows who signed and that nothing changed afterwards, not that the claims are true. What the operator cannot do is change a document after signing without the change showing, prove both that an example was and was not in one committed set, or alter a sealed receipt without breaking the chain. The weakest links are named on this page: the BOM and the commitments are statements about data, not about the weights, and receipts trust the costs written at request time.
### What if no signing key is set?
BOMs and receipts are still written, with their SHA-256 but no signature. `verify` reports them as unsigned. That shows a document is internally consistent, not who wrote it; set `RATCHET_BOM_SIGNING_KEY` before handing documents to anyone.
### Why are the training-set proofs admin-only?
Because the proof endpoint is a membership oracle: for a text someone can guess (a customer's email template, say), "was this in the training set?" leaks it. Proofs go to auditors through an operator with an admin key. Held-out certificates also disclose neighbouring examples' hashes, so they are shared with auditors, not published.
### Does any of this cost money or slow requests down?
Not on the request path beyond logging a few values per request: no model call, no extra random draw. The work happens at night (building the trees and signing) and when a day is sealed. Certificates add about 2 KB to each eval at the reference run's golden sizes.
### Has any of it run on real traffic?
Not the signed documents on this page. As of Oct 29, 2026, the signed BOMs, proofs, certificates and receipts described here have been produced only by the tests, with mock models (the Nebius Night School run wrote an unsigned BOM, signed afterwards on 29 Oct 2026 and not at creation, and its gate certificate did not certify). The real-model measurements are the text-to-SQL and tool-calling adapters' exams, scored by standalone scripts (see [Results](/guides/results/)). The live-run harness (`python -m ratchet.liverun`) has not run its measured mode.
## Related
- [Verify: the commands](/reference/verify/): a copy-and-run sheet for the checks, with the output to expect.
- [Confidence](/guides/confidence/): where the threshold and the held-out half come from.
- [The release gate](/guides/release-gate/): how the certificate and the paired comparison are used.
- [Licenses](/guides/licenses/): why only open teachers, and what the NOTICE.md says.
- [Cost and savings](/guides/cost-and-savings/): the formula baseline, break-even and when self-hosting does not pay.
- The sections on [what a certificate means](https://github.com/harshbhx/ratchet/blob/main/docs/FEATURES.md#what-a-certificate-means), [proving what an adapter was trained on](https://github.com/harshbhx/ratchet/blob/main/docs/FEATURES.md#proving-what-an-adapter-was-trained-on), [replaying past traffic](https://github.com/harshbhx/ratchet/blob/main/docs/FEATURES.md#replaying-past-traffic-at-another-threshold), [savings you can check](https://github.com/harshbhx/ratchet/blob/main/docs/FEATURES.md#savings-you-can-check) and [limitations](https://github.com/harshbhx/ratchet/blob/main/README.md#limitations).
---
# Proof you can run
Source: https://docs.ratchet.dualmindlab.tech/guides/proof/
Each claim about an adapter or a bill has a document and a command that checks it on your own machine, so you do not have to trust the gateway that made it.
"Trust me" is cheap. A better deal is "here is the receipt, here is how to add it up yourself".
Ratchet gives you a chain of six such receipts. The first says how well the adapter did on a sealed exam. The next says who signed the list of ingredients. The next lets you check that one recipe was, or was not, in the study pile. The next gives a safety margin on the exam score. The next replays past orders at another setting, as an estimate rather than a signed receipt. The last shows each day's bill with a fingerprint of every order, chained to the day before.
Each link has one command. When a link is broken, the command says `FAIL` and why.
## How it really works
1. **The exam.** The adapter answers questions about databases it never trained on, and a CPU-only script recomputes the score from the raw answers (`scripts/measure_adapter.py verify`, and `scripts/toolcall_results.py --check` for tool calling). These are the checks that have been run on real models, for the text-to-SQL and tool-calling adapters.
2. **The signed bill of materials.** When Night School trains a version it writes a document that lists the teachers and their licenses, the base model's license, the data counts and the eval, and signs it with an Ed25519 key. `python -m ratchet.bom verify` checks the signature and every hash in it.
3. **The committed training set.** The same document signs a Merkle root over the exact training rows and another over the golden rows. `python -m ratchet.proof verify` shows that one example is in a set, or is not, without the whole set. `python -m ratchet.proof heldout` shows that no golden example is in the training root.
4. **Certified precision.** The eval carries an exact Clopper-Pearson lower bound on how often the served answers match the teacher. `python -m ratchet.stats check` recomputes it from the counts in the signed BOM.
5. **The time machine.** `python -m ratchet.replay` replays one version's logged traffic at another threshold or audit rate and estimates precision, coverage and cost. It reads the ledger, so it is an estimate, not a signed document.
6. **The savings receipts.** After each ledger day, the gateway seals one receipt per client: a Merkle tree over request metadata (no text), signed and chained to the previous day. `python -m ratchet.receipt verify` checks the signature, the chain and, with a proof, that one request is in a receipt.
```mermaid
flowchart TB
DATA[("Training and golden rows")] --> ROOTS["Merkle roots over each set"]
ROOTS --> PROOF["Membership and held-out proofs ratchet.proof"]
ROOTS --> BOM["Signed bill of materials ratchet.bom verify"]
EXAM["Exam on unseen databases measure_adapter.py verify"]
BOM --> LIC["Licenses and NOTICE.md"]
BOM --> CERT["Certified precision ratchet.stats check"]
EXAM --> CERT
CERT --> GATE{{"Release gate decision"}}
GATE --> LEDGER[("Ledger of served requests")]
LEDGER --> REPLAY["Replay at another threshold ratchet.replay"]
LEDGER --> REC["Sealed daily receipts ratchet.receipt verify"]
REC --> CHAIN["Chained to the previous day"]
class DATA,LEDGER store
class GATE gate
class EXAM student
```
**Reading this diagram.** Follow the arrows from the data. The training and golden rows are committed first, then the signed document is built on top of those commitments, and the certificate is read from the signed document. Once the version is serving, the ledger fills with request metadata, and both the replay and the receipts read from the ledger. A command sits on every box (each `ratchet.*` one runs as `python -m ratchet.*`), so each step down the chain can be checked on its own, and a change anywhere above breaks the checks below it. The blue box is the one link that has run on a real model.
Stored data
A release decision
Run on a real model
## What exists today, and what has run for real
Only the text-to-SQL and tool-calling adapters have been run on real models, and only their exams and check scripts (`python scripts/text2sql/exam_numbers.py --check`, `python scripts/toolcall_results.py --check`). Every other link on this page is code in the repository, tested offline with mock models. Night School itself ran once on Nebius on 28 Oct 2026 (dataset import, training job, hot-load, golden eval; [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)), but its bill of materials was written unsigned (no signing key was set), so no BOM, proof or receipt was signed by a real run as it happened. None of it has run on real production traffic.
Signed on 29 Oct 2026 with Ed25519 key `81b0a086250a81d0`, after the run; the BOM content is unchanged from the run's own record. Verify it with `python -m ratchet.bom verify evidence/nightschool-nebius/ratchet_bom.signed.json --public-key Gh68hnDDlRSjBRxmc5dTo4l+rvqkftUsH7B2Nm72gpc= --notice evidence/nightschool-nebius/NOTICE.md`: it prints `OK` for the signature and for the notice. Only this BOM is covered. Receipts and BOMs from live traffic stay unsigned until the live gateway has the key.
| Link | In the repository | Checked by | Run on a real model |
|---|---|---|---|
| Exam and `verify` | Standalone scripts in `scripts/` | The [Results](/guides/results/) page | Yes: the text-to-SQL and tool-calling adapters |
| Signed BOM | `ratchet/bom.py` | Tests | No |
| Committed sets and proofs | `ratchet/proof.py`, `ratchet/merkle.py` | Tests | No |
| Certified precision | `ratchet/certificate.py`, `ratchet/stats.py` | Tests and seeded Monte Carlo runs | No |
| Replay | `ratchet/replay.py` | Tests | No |
| Savings receipts | `ratchet/receipts.py`, `ratchet/receipt.py` | Tests | No |
## The checks, one by one
The short version, one claim at a time; the full text for each follows below.
`, and add `--notice NOTICE.md` to check the notice next to the adapter.\n\nA failure starts with `FAIL:`: a signature that does not match, or a hash that differs.' },
{ title: 'The training set', body: 'Run `python -m ratchet.proof verify` for one example, or `python -m ratchet.proof heldout` for every golden example.\n\nA failure starts with `FAIL:`, for example when a proof does not rebuild the signed root.' },
{ title: 'Certified precision', body: 'Run `python -m ratchet.stats check bom.json --public-key `.\n\nIt recomputes the bound from the signed counts and prints a line for each number that differs from the signed one.' },
{ title: 'The time machine', body: 'Run `python -m ratchet.replay --db ratchet.db --version 3`.\n\nIt is an estimate, not a signed document: a `null` column means the logged data cannot support an estimate at that setting.' },
{ title: 'The savings receipts', body: 'Run `python -m ratchet.receipt verify r.json --public-key --prev p.json`.\n\nA changed receipt fails with a line such as `FAIL: chain: prev_sha256 is not the sha256 of day N\'s receipt`.' },
]}
/>
### 1. The exam: `measure_adapter.py verify`
- **What it proves.** Every number in `results.json` is recomputed from the raw answers, the exam file and the exam databases, and each of those files is the one the results were computed from (SHA-256).
- **The command.**
```bash
python scripts/measure_adapter.py verify \
--raw evidence/text2sql-exam2/answers/raw_v2.jsonl \
--results evidence/text2sql-exam2/results/results_v2.json \
--golden evidence/text2sql-exam2/data/golden.jsonl \
--dbs evidence/text2sql-exam2/data/dbs.json
```
- **Success.** `OK: every number in results_v2.json was recomputed from raw_v2.jsonl and matches`.
- **Failure.** `FAIL: raw.jsonl is not the file the results were computed from`, or `FAIL: golden.jsonl or dbs.json is not the file ...`, or `FAIL: recomputed numbers differ in [...]` with the names of the differing keys.
- **What it does not prove.** That the teacher's answers are what a user wanted, or that the exam is like your data.
### 2. The signed bill of materials: `ratchet.bom verify`
- **What it proves.** The operator signed these claims about this adapter at training time, and no byte changed since. With `--notice`, the `NOTICE.md` next to the adapter is the text the BOM signed.
- **The command.**
```bash
python -m ratchet.bom verify bom.json --public-key [--notice NOTICE.md]
```
- **Failure.** A line starting `FAIL:` with the reason, such as a signature that does not match or a hash that differs. A BOM written without a signing key verifies as unsigned: consistent, but not tied to anyone.
- **What it does not prove.** That the weights were trained on the listed data. The BOM is a statement about the data.
### 3. Committed training sets: `ratchet.proof`
- **What it proves.** An example's key is, or is not, a leaf of the root the BOM signs for the training or golden set. `heldout` proves every golden key is absent from the training root.
- **The commands.**
```bash
python -m ratchet.proof key --messages messages.json # the key a request is stored under
python -m ratchet.proof verify proof.json --bom bom.json --public-key
python -m ratchet.proof heldout heldout.json --bom bom.json --public-key
```
- **Failure.** One or more lines starting `FAIL:`, for example when the proof does not rebuild the signed root or the BOM itself does not verify.
- **What it does not prove.** That the weights came from that set. The proof endpoints are admin only on a gateway, because they can reveal whether a guessable text was in the data.
### 4. Certified precision: `ratchet.stats check`
- **What it proves.** The certificate in the BOM's eval follows from the counts it signs: an exact Clopper-Pearson lower bound, a Bonferroni frontier over a fixed grid of thresholds, per-task bounds and the paired comparison with the previous version.
- **The command.**
```bash
python -m ratchet.stats check bom.json --public-key
python -m ratchet.stats plan --target 0.95 --delta 0.05 # how many golden rows a certificate needs
```
- **Failure.** `FAIL:` from the BOM check, or a line for each recomputed number that differs from the signed one.
- **What it does not prove.** Correctness, or behaviour on traffic that does not look like the golden rows.
### 5. The time machine: `ratchet.replay`
- **What it proves.** Nothing signed. It estimates what one version's logged traffic would have given at another threshold or audit rate, from the probabilities that were logged with each request.
- **The command.**
```bash
python -m ratchet.replay --db ratchet.db --version 3 --sweep 0.3:0.95:0.05
```
- **Failure.** A column that is `null` where the data does not support an estimate (for example when no request could have been labelled at that setting).
- **What it does not prove.** Anything about requests the ledger did not log. Its interval is approximate.
### 6. Savings receipts: `ratchet.receipt verify`
- **What it proves.** The day's receipt was not edited after sealing, it follows the previous day's receipt, and with a proof file one request is in it.
- **The command.**
```bash
python -m ratchet.receipt verify r.json --public-key [--prev p.json] [--proof x.json] [--rows leaves.jsonl]
```
- **Failure.** Lines starting `FAIL:` such as `FAIL: chain: prev_sha256 is not the sha256 of day N's receipt (one of them was changed)`.
- **What it does not prove.** That the costs written at request time were right. Compare them with the Token Factory bill.
## What a failure looks like, in one picture
```mermaid
flowchart LR
F(["A file is changed after it was made"]) --> W{{"Which file?"}}
W -->|"raw answers or results"| E1["exam verify: numbers differ"]
W -->|"the BOM"| E2["bom verify: signature or hash fails"]
W -->|"training set"| E3["proof verify: root does not rebuild"]
W -->|"past receipt"| E4["receipt verify: chain broken"]
W -->|"request row"| E5["receipt proof: leaf not in the root"]
class W gate
class E1,E2,E3,E4,E5 danger
```
**Reading this diagram.** Each file has its own alarm, and every alarm prints a line starting `FAIL:`. Changing a file never goes unnoticed by the check next to it, and because later documents commit to earlier ones, a change to an old receipt shows up in the next day's receipt as well. The failure text names the file and the reason, so you know where to look.
## For engineers
| Check | Code | Tests |
|---|---|---|
| BOM and signing | [`ratchet/bom.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/bom.py) | `tests/test_provenance.py`, `tests/test_licensing.py` |
| Merkle trees | [`ratchet/merkle.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/merkle.py) | `tests/test_provenance.py` |
| Training-set proofs | [`ratchet/proof.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/proof.py) | `tests/test_provenance.py` |
| Exact statistics and certificate | [`ratchet/stats.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/stats.py), [`ratchet/certificate.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/certificate.py) | `tests/test_stats.py` |
| Replay | [`ratchet/replay.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/replay.py) | `tests/test_system.py`, `tests/test_core.py` |
| Receipts | [`ratchet/receipts.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/receipts.py) | `tests/test_receipts.py` |
Signing uses an Ed25519 key from `RATCHET_BOM_SIGNING_KEY`. Generate one with `python -m ratchet.bom keygen`, keep the seed secret and publish the public key somewhere other than the gateway. The tables that hold committed leaves and sealed receipts refuse `UPDATE` and `DELETE` by trigger.
## Common questions
### What do I need to check all of this?
The documents, the operator's public key, and this repository's command-line tools. You do not need the gateway, the ledger or the training data.
### Which check should I run first?
The exam's `verify`, since it is the one that touches a real model. Then the BOM, since the later checks start from it.
### If everything says OK, is the model good?
It means the claims were not altered and the arithmetic follows from the inputs. Whether the inputs describe your use is a separate question. Read the "does not prove" line under each check.
### Why is there no real BOM or receipt for the SQL adapter yet?
The SQL run used standalone scripts and a borrowed GPU rather than a nightly Night School run on a gateway, so the BOM, proofs and receipts have so far only been exercised by tests.
## Related
- [Verify: the commands](/reference/verify/): the same checks as a copy-and-run sheet, with the output to expect
- [Evidence you can check](/guides/evidence/) for the long form of each link
- [A text-to-SQL adapter you can check](/guides/text-to-sql/)
- [Results](/guides/results/)
- [Licenses](/guides/licenses/)
---
# Check it yourself
Source: https://docs.ratchet.dualmindlab.tech/reference/verify/
Every claim Ratchet makes about a result can be rechecked from files in this repository. This page lists each claim, the command that checks it, what the command should print, and what it does **not** prove. This is the quick sheet: copy a command, run it, compare the output. The longer explanations are in [Proof you can run](https://docs.ratchet.dualmindlab.tech/guides/proof/) and [Evidence you can check](https://docs.ratchet.dualmindlab.tech/guides/evidence/). You need Python 3.10 or newer and a clone of the repository. Nothing here needs a GPU, a key or a network connection unless a row says so.
```bash
git clone https://github.com/harshbhx/ratchet.git && cd ratchet
pip install -e ".[dev]"
```
## At a glance
| Claim | One command | Needs |
|---|---|---|
| The exam scores are what the saved answers give | `python scripts/text2sql/exam_numbers.py --check` | nothing |
| One adapter's score recomputes row by row | `python scripts/measure_adapter.py verify …` ([below](#2-one-adapters-score-recomputes-row-by-row)) | nothing |
| The exam was frozen before any model answered it | `sha256sum evidence/text2sql-exam2/data/golden.jsonl` | nothing |
| The adapters are the ones on Hugging Face | `sha256sum adapter_model.safetensors` ([below](#4-the-adapters-are-the-ones-on-hugging-face)) | a download |
| The live run's numbers come from its records | `python -m ratchet.liverun verify evidence/live-smoke-2026-10-23` | nothing |
| The live gateway calls Token Factory | `curl` the live gateway ([below](#5-the-live-gateway-calls-token-factory)) | a network |
| The loop and the rollback behave as tested, on mock models (not a live result) | `python -m sim.simulate --days 10 --requests 2000 --bad-deploy-version 3` | nothing |
| The tool-calling scores (round 3) recompute from the saved answers | `python scripts/toolcall_results.py --check` ([below](#tool-calling-round-3)) | nothing |
| A signed BOM verifies offline | `python -m ratchet.bom verify …` ([below](#7-a-signed-bill-of-materials-per-adapter)) | nothing |
| The hosted Nano's thresholds follow from real Ultra labels ([evidence/hosted-nano](https://github.com/harshbhx/ratchet/blob/main/evidence/hosted-nano/README.md)) | `python scripts/hosted_nano_calibration.py check` | nothing |
| Every requirement maps to code and tests | `python scripts/reqtrace.py --check` | nothing |
## 1. The exam scores are what the saved answers give
```bash
python scripts/text2sql/exam_numbers.py --check
```
Expected: `OK: both exams.json files match the saved answers`.
This reruns the strict scoring over every saved answer of the untrained model, the [teacher](https://docs.ratchet.dualmindlab.tech/guides/glossary/#teacher) and both adapters on both exams, and compares the result with the numbers the website and the README chart show (`web/src/data/exams.json`). Scoring is strict: an answer is right only if running it on the exam's database returns exactly the reference rows.
**It shows:** the published numbers are the saved answers' numbers, with no typed-in figure.
**It does not show:** that the answers were produced the way the README says. For that, read the [exam 2 evidence](https://github.com/harshbhx/ratchet/blob/main/evidence/text2sql-exam2/README.md) and [exam 1 and round 1](https://github.com/harshbhx/ratchet/blob/main/evidence/text2sql-round1/README.md), which name each script and, where recorded, its cost.
## 2. One adapter's score recomputes row by row
```bash
python scripts/measure_adapter.py verify \
--raw evidence/text2sql-exam2/answers/raw_v2.jsonl \
--results evidence/text2sql-exam2/results/results_v2.json \
--golden evidence/text2sql-exam2/data/golden.jsonl \
--dbs evidence/text2sql-exam2/data/dbs.json
```
Expected: `OK: every number in results.json was recomputed from raw.jsonl and matches`.
This runs each of the 200 answers against the exam's [SQLite](https://docs.ratchet.dualmindlab.tech/guides/glossary/#sqlite) databases and recomputes the agreement, the coverage and the confidence-gate numbers. Swap `raw_v2` and `results_v2` for `raw_v1` and `results_v1` to check the round 1 adapter.
## 3. The exam was frozen before any model answered it
```bash
sha256sum evidence/text2sql-exam2/data/golden.jsonl
```
Expected: `b7634971cf6b0d79e64e30e4c16d1253fad65ab9a501a06edc6c9f8e70d69058` (on Windows: `python -c "import hashlib;print(hashlib.sha256(open('evidence/text2sql-exam2/data/golden.jsonl','rb').read()).hexdigest())"`).
The repository's `.gitattributes` keeps every file under `evidence/` byte for byte on checkout, so a Windows clone hashes the same as Linux. With an older clone, or a Git that ignores it, clone again with `git clone -c core.autocrlf=false https://github.com/harshbhx/ratchet.git`.
That hash was written into `evidence/text2sql-exam2/data/exam_manifest.json` when the exam was generated. Exam 1's fingerprint is `6b9c1db49bbe2aecf4157ed27845fe024d1b4fcb540f93d81b6987b982b22864`, in the [round 1 manifest](https://github.com/harshbhx/ratchet/blob/main/evidence/text2sql-round1/MANIFEST.json).
**It shows:** the exam file is the one that was frozen. **It does not show:** when the hash was first recorded: that rests on this repository's dated history and the saved generator output.
## 4. The adapters are the ones on Hugging Face
Download an adapter's `adapter_model.safetensors` from the [model page](https://huggingface.co/harshbhx/ratchet-nemotron-nano-4b-text2sql) (the file at the top level is round 2, the one in `round1/` is round 1) and hash it:
| Adapter | SHA-256 of the weights |
|---|---|
| Round 1 | `a883bcf6bbf4152efb5a874520b8fa1a09d9bead55c172507725dc255f18236a` |
| Round 2 | `38f99d441e0527b75fdf8fd9d10acbf4473dcf70db7febe3ad73fccd631a6aa7` |
The same values are in each evidence folder's `MANIFEST.json`. Round 2's first run is in [`evidence/text2sql-round2`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2). The weights themselves are 40.5 MB each and are not in git.
## 5. The live gateway calls Token Factory
```bash
curl -s https://live.ratchet.dualmindlab.tech/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"ratchet","messages":[{"role":"user","content":"In one sentence, what is a LoRA adapter?"}]}'
```
The reply carries a `ratchet` block. Expect `served_by: "teacher"` and the model `nvidia/Nemotron-3-Ultra-550b-a55b`, or, for a random 40% of eligible requests, the hosted Nano 30B (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B`, route `demo_split`). Either way you get a small `cost_usd` (a fraction of a cent) and a `request_id`. It is a real call, so please keep it to a few. Ratchet's own adapter is not connected to the live gateway; every model call on it runs on Nebius Token Factory.
**The run on record:** `python -m ratchet.liverun verify evidence/live-smoke-2026-10-23` recomputes the client-side numbers of a recorded 20-request run from its 21 per-request records. Expected: `recomputed 11 client-side values and 1 rounds from 21 records`. It also prints a line that starts `not recomputable here (ledger snapshots or entered by hand)`; those values are listed, not checked. Three recorded live Scout runs are in [`evidence/scout`](https://github.com/harshbhx/ratchet/tree/main/evidence/scout), and you can [replay](https://docs.ratchet.dualmindlab.tech/guides/glossary/#replay) them [in your browser](https://huggingface.co/spaces/harshbhx/ratchet-scout).
## 6. The loop and the rollback, on mock models
```bash
python -m sim.simulate --days 10 --requests 2000 --bad-deploy-version 3
make demo # then open http://localhost:8000/dashboard
make reference # three pinned workloads must reproduce the v0.4.0 behaviour
```
These run **mock models**. They show how the cascade, the release gate and the rollback behave, not real model quality, and every figure they print is a mock-mode run.
## 7. A signed bill of materials per adapter
In the mock-mode gateway you can generate a key, sign an adapter's [bill of materials](https://docs.ratchet.dualmindlab.tech/guides/glossary/#bom) and verify it offline:
```bash
python -m ratchet.bom keygen # a signing key and its public key
RATCHET_BOM_SIGNING_KEY= make demo
curl -s localhost:8000/ratchet/versions/2/bom | python -m ratchet.bom verify - --public-key
```
The public half of Ratchet's own signing key ([Ed25519](https://docs.ratchet.dualmindlab.tech/guides/glossary/#ed25519), key id `81b0a086250a81d0`) is in [evidence/keys](https://github.com/harshbhx/ratchet/blob/main/evidence/keys/README.md). The 28 Oct Night School [BOM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#bom) was signed with it afterwards, on 29 Oct 2026 (`evidence/nightschool-nebius/ratchet_bom.signed.json`; the content is unchanged from the run's own record). Check a document against it with `--public-key Gh68hnDDlRSjBRxmc5dTo4l+rvqkftUsH7B2Nm72gpc=` (and `--notice NOTICE.md` for the adapter's notice).
A signed BOM is the operator's signed statement of the data and evaluation behind an adapter: it shows that nothing changed after signing, not that the weights were trained on that data. See Features in depth.
## 8. The test suite and the requirements matrix
```bash
make test # 2014 tests, no network, no GPU (Linux, macOS or WSL)
python -m pytest -q # the same suite without make
python scripts/reqtrace.py --check
```
Run the suite on Linux, macOS or WSL: Windows has no `make`, and `tests/test_watch.py` needs `fcntl`, which Windows lacks.
Expected for the last command: `reqtrace: OK: 228 requirements …`. Every requirement and every invariant maps to code and to tests in REQUIREMENTS.md.
## What it cost
About $17 on record for the two text-to-SQL adapters. All are estimates recorded by the scripts, not invoices; check the billing pages before quoting them.
| Item | USD | Where it is recorded |
|---|---|---|
| Teacher answers, round 1 data and exam 1 (273 calls) | 2.87 | `evidence/text2sql-round1/data/teacher_manifest.json` (`spent_usd_estimate`) |
| Teacher answers, round 2 data (914 calls) | 9.18 | `evidence/text2sql-round2/data/teacher_manifest_round2.json` (`spent_usd_estimate`) |
| Teacher answering exam 2 (200 rows) | 0.11 | `evidence/text2sql-exam2/results/teacher_selfcheck.json` |
| GPU job, round 2 training and exam 1 answers | 4.80 | `evidence/text2sql-round2/logs/final_status.json` (`spent_usd`) |
| Round 1 training | 0 | a borrowed GPU, no charge |
Not costed: the GPU runs that answered exam 2, small teacher re-runs on exam 1, and any earlier probes. The tool-calling round (about $2.33 of GPU time, plus teacher spend) is separate and is not in this sum. So is the hosted Nano's [calibration](https://docs.ratchet.dualmindlab.tech/guides/glossary/#calibration) ($0.16 of Token Factory calls, `evidence/hosted-nano/collect.json`).
## What none of this proves
- The exams are synthetic text-to-SQL, not your data. A strict one-reference exam also marks a correct query wrong when it returns extra columns; the teacher's own score shows that.
- Agreement is with the teacher's answers, not with ground truth.
- The adapters were trained on a borrowed GPU with the standalone scripts in `scripts/text2sql/`; Night School's Nebius job ran once by hand on 28 Oct 2026 ([`evidence/nightschool-nebius`](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)), and these exam results do not test it.
- The hard tool-calling exam was written by one model (Claude Sonnet 5.5), so it may favour that model's style; the [BFCL](https://docs.ratchet.dualmindlab.tech/guides/glossary/#bfcl) exam is a sample of a public benchmark, which models may have seen.
- On the BFCL exam the adapter's gain over the untrained model is not significant (sign test p = 0.0784), and its gain comes from the `irrelevance` category only. See [Tool-calling round 3](#tool-calling-round-3).
- Scores come from one run of each model at one setting. The 95% ranges say how much a different sample of questions could move them; they do not cover a different training run.
## Tool-calling round 3
Every claim below recomputes from files in `evidence/toolcall/`. Nothing here needs a GPU, a key or a network connection.
| Claim | One command |
|---|---|
| The scores, ranges and wording verdicts follow from the saved answers, and every saved verdict follows from its raw answer | `python scripts/toolcall_results.py --check` |
| The hard exam is the frozen one | `python -c "import hashlib;print(hashlib.sha256(open('evidence/toolcall/exams/exam_hard.jsonl','rb').read()).hexdigest())"` prints `e26b6d020caedf2ce050b48641d5c2a55a96202826f51ca66ad81be9d04dada2` |
| The BFCL exam is the frozen one | `python -c "import hashlib;print(hashlib.sha256(open('evidence/toolcall/exams/exam_bfcl.jsonl','rb').read()).hexdigest())"` prints `c6b2e9dd3abd6bb2a5e7b33105e0de372612be51e82e2117087e6c36faeccb37` |
| The copied evidence files are unchanged | `python scripts/toolcall_results.py --check` (compares each file's SHA-256 with `evidence/toolcall/MANIFEST.json`) |
| The drafts and the page data are the ones the numbers give | `python scripts/toolcall_results.py --render --web --check-render` |
**It shows:** the published numbers are the saved answers' numbers, under the exam's own strict scorer. **It does not show:** that the adapter produced those answers on the borrowed GPU as described: that rests on the training and generation logs kept with the run. A 95% range on 150 or 300 questions is wide, which is why the wording says 'about the same' only when the ranges overlap and the gap is under 5 points, and 'lower' or 'higher' with a chance caveat when the ranges overlap and the gap is larger.
Found something that does not match? Open an [issue](https://github.com/harshbhx/ratchet/issues/new/choose). A number that cannot be rechecked should not be on the page.
---
# Scout
Source: https://docs.ratchet.dualmindlab.tech/guides/scout/
Scout is a small multi-agent research app built on Ratchet: five agents answer a question from the web, the many short triage and reading calls move to the student, and the open-ended planning and writing stay pinned to the teacher.
The restaurant takes a catering order: "find out what LoRA fine-tuning is and write it up". Five helpers work on it, and every dish still goes through the pass.
- **The planner** decides what to shop for: three search queries. Only the head chef from the agency does this, because it needs judgement.
- **The searcher** is the runner who goes to the market (a web search). Nobody cooks here, so nobody is paid by the dish.
- **The triage helper** looks at each bag from the market and says "useful or not, and what kind of shop it came from". That is a dozen small, repetitive jobs per order. The apprentice chef learns these from the head chef's answers and soon does most of them.
- **The reader** opens each useful bag and writes down the one fact that matters. Also small and repetitive, so the apprentice learns it too.
- **The writer** plates the final answer with its sources. Only the head chef does this.
The head chef's planning and plating notes are never copied into the recipe book, so the apprentice never studies them. And because the restaurant is open to anyone, it only takes so many catering orders a day, with a few held back for the food critics (the hackathon judges).
## How it really works
1. **A question arrives.** The browser opens `GET /scout/stream?q=...` (or a client calls `POST /scout/run`). The gateway checks it: not empty, at most 300 characters, and a Tavily key. Then the budget guard decides whether the run may start (`Guard.admit`).
2. **The planner writes three search queries.** One call with the task `research-plan`. The task is pinned, so Nemotron 3 Ultra answers it directly, with no student sample. Its JSON answer is `{"queries": [...]}`.
3. **The searcher runs the queries.** It is a tool, not a model. It calls the [Tavily Search API](https://docs.tavily.com) at basic depth (1 credit a search), up to `RATCHET_SCOUT_MAX_RESULTS` (4) results per query. Duplicate URLs are skipped, and at most 12 results go on.
4. **Triage labels every result.** One call per result (up to 12) with the task `source-triage`, run 6 at a time in a thread pool. The answer is `relevant`, `source_type` and a one-sentence `reason`. Only `relevant` and `source_type` are compared when Ratchet decides whether two answers agree.
5. **The reader pulls one fact from each relevant result.** One call per relevant result (up to 6) with the task `page-read`. The answer is `answers_question`, `fact_type` and a free-text `key_fact`. Only the first two are compared.
6. **The writer answers with citations.** One call with the task `research-write`, also pinned to Ultra. It writes 2 to 4 sentences from the numbered facts and cites them as `[n]`, or says the facts do not answer the question.
7. **Everything streams to the page, then the spend is counted.** Each finished call is sent as a `step` event, then a `done` event with totals. When the run ends, successfully or not, the guard adds its spend (`Guard.add_spend`).
```mermaid
sequenceDiagram
participant B as Browser
participant G as Gateway and cascade
participant Teacher as Ultra teacher
participant Student as Nano student
participant S as Search
B->>G: GET /scout/stream
Note over G: Guard.admit
G-->>B: start
G->>Teacher: planner, pinned
Teacher-->>G: 3 queries
G->>S: up to 3 searches
S-->>G: up to 12 results
loop each result, 6 at a time
G->>Student: triage, 3 samples
opt not confident
G->>Teacher: escalate
end
end
loop each relevant result, up to 6
G->>Student: reader, 3 samples
end
G->>Teacher: writer, pinned
G-->>B: step events, then done
Note over G: Guard.add_spend, also on failure
```
**Reading this diagram.** The run is a pipeline with two fan-out stages. The budget guard is asked once at the start and told once at the end, and in between the gateway never calls a model without going through the cascade. Notice who answers: the planner and writer calls go straight to Ultra, while triage and reader calls try the student first, and the reader escalates the same way triage does. The page gets a `step` event as soon as each call finishes, which is why triage steps appear out of order.
Ultra, the teacher
Nano, the student
### The agents and what they pass on
```mermaid
flowchart TB
Q(["A question"]) --> PL["Planner research-plan"]
PL -->|"3 queries"| SE["Searcher Tavily"]
SE -->|"up to 12 results"| TR["Triage, up to 12 calls source-triage"]
TR -->|"the relevant ones"| RD["Reader, up to 6 calls page-read"]
RD -->|"numbered facts"| WR["Writer research-write"]
WR --> A(["An answer with citations"])
class PL,WR teacher
class TR,RD student
class SE outline
```
**Reading this diagram.** Orange calls are pinned to the teacher: never stored, never trained on. Blue calls are distilled: the student first, the teacher when it is unsure. The dashed box is a tool, not a model. Most calls are blue: up to 18 of up to 20 in a run. Each arrow is also a size: three queries become up to twelve results, then up to six facts, then one answer. Because only the planner and the writer are pinned, the savings a run shows are honest about the part that cannot move.
Pinned to the teacher
Distilled to the student
A tool
In the gateway each agent call is `Ratchet.handle` in the same process. From outside, the [Scout CLI](#for-engineers) sends the same calls to `/v1/chat/completions` with an `X-Ratchet-Task` header, like any OpenAI client.
### Pinned and distilled calls, and why
Writes 3 search queries. Pinned to the teacher.
Tavily Search API.
Up to 18 short labelling calls, distilled to the student.
2 to 4 sentences with citations. Pinned to the teacher.
```mermaid
flowchart TB
A(["An agent call with a task name"]) --> P{{"Pinned task?"}}
P -->|"yes: planner, writer"| U["Ultra answers"]
U --> L["Logged with its cost, never stored"]
P -->|"no: triage, reader"| S["The student samples three times"]
S --> C{{"Confident enough?"}}
C -->|yes| SV["The student's answer is served"]
C -->|no| E["Ultra answers"]
E --> T[("Stored as training data, with provenance")]
SV -.-> AU["A random few audited by Ultra"]
class P,C gate
class U,E,AU teacher
class S,SV student
class T store
```
**Reading this diagram.** The first question decides everything: is the task pinned? A pinned call never touches the student and never reaches the training data. A distilled call is the normal cascade. Both lower branches end with Ultra involved: either because the student was unsure, or because an audit checks it afterwards (the dotted arrow). That is how the student's quality stays measured.
**Pinned** (`research-plan`, `research-write`): planning queries and writing an answer are open-ended. There are no label fields to compare, so the student and the teacher would almost never "agree", calibration would end up serving nothing, and every call would still pay for student samples. Training a 4B model on long free text would also dilute what it learns for the labelled tasks. So a pinned task goes straight to the teacher, with no student sample, audit or shadow. It is logged in the ledger like any other call, so the cost and the all-teacher baseline stay complete, but it is **never stored as a training example**, a rule the code and its tests enforce. Only an operator's task spec can pin a task, never a request header.
**Distilled** (`source-triage`, `page-read`): one short labelling call per search result and one per relevant page. Each run makes up to 18 of them, they repeat in shape from question to question, and their label fields can be compared exactly. That is where the student takes over. Before a trained student is live (the cold start), these calls go to the teacher too, and its answers become the training data.
The savings Scout shows include the pinned calls, so the percentage is honest about the part that cannot move.
### What live Scout uses
| | Live Scout |
|---|---|
| Search | Tavily Search API (`TAVILY_API_KEY`); without a key, Scout answers 503 |
| Questions | Any question up to 300 characters (`GET /scout/examples` lists examples) |
| Models | Every call goes through the gateway to Token Factory: Ultra for the pinned calls and, until a trained student serves, for the rest; a trained student on the GPU endpoint or the hosted fallback once one exists |
| Budget | The full budget guard below |
The agents build their messages with the functions in `ratchet/scout/prompts.py`; never build Scout messages anywhere else.
### The event stream
`GET /scout/stream` sends Server-Sent Events: one JSON object per `data:` line, in this order.
| Event | When | Main fields |
|---|---|---|
| `start` | The run was admitted | `run_id`, `question`, `mode` |
| `step` | An agent call finished (parallel steps arrive in any order) | `agent`, `task`, `served_by` (`student`, `teacher` or `tool`), `model`, `confidence`, `escalated`, `cost_usd`, `baseline_usd`, `latency_ms`, `detail` |
| `done` | The writer finished | `answer`, `citations`, `totals` (calls, student and teacher calls, cost, baseline, savings), `search` |
| `error` | Anything failed, before or during the run | `message`; it ends the stream |
The stream always answers HTTP 200, because a browser's `EventSource` cannot read status codes; a refusal or a failure arrives as an `error` event. While waiting it sends a keep-alive comment every 15 seconds. `POST /scout/run` returns the same events in one JSON body and uses status codes instead: 400 for a bad question, 429 when the guard refuses, 503 without a Tavily key in live mode, 502 when a model call fails. The full contract is in [SCOUT.md](/reference/scout/#event-objects-sse-data-payloads-in-order).
### Who answered each call
The page puts a badge on every step. A student step from a Night School adapter (`student-vN`) is . A teacher step is . When the GPU endpoint is off in live mode, student calls go to the untuned Nemotron 3 Nano 30B on Token Factory, and those steps say , held to the default confidence threshold and not covered by the adapter's precision certificate. Only the step's `model` tells the two student cases apart, so the page compares it with `fallback_student` from `GET /scout/status`.
### The budget guard
Scout is a public path that spends money on every run, so live mode guards it, and the code never lets it run without the guard: every run, judge runs included, goes through `Guard.admit` before it starts and `Guard.add_spend` after it ends, also on failure. Live mode must never run without it.
- **A daily spend cap**: `RATCHET_SCOUT_DAILY_BUDGET_USD` (code default $3.00). The live gateway, on Cloudflare Workers with Containers, sets $1.50 a day beside a $2.50 daily cap on the whole gateway (`deploy/cloudflare-live/src/index.ts`); the Nebius VM kit's `.env` sets $0.50, which would hold the 48 days from Oct 29 to Dec 15 to at most $24, plus runs in flight. Spend is the `cost_usd` of every step, audits and shadow calls included, so a run that fails halfway still counts what it spent.
- **A daily run cap**: `RATCHET_SCOUT_MAX_RUNS_PER_DAY` (30), all visitors together. It also caps Tavily credits: about 3 searches a run.
- **A per-IP hourly limit**: `RATCHET_SCOUT_RUNS_PER_HOUR` (6) per client address, in memory only, so no IP is stored.
- **Kept across restarts**: each admitted run writes a `scout_run` row to the ledger's `spend` table, updated after every model call and closed with its outcome. At startup `Guard.restore()` rebuilds today's counters from those rows, so an update or a crash does not reset the caps while the ledger file survives. (The live gateway keeps its ledger on the container's own disk, so a container restart starts with empty counters; its low caps are set for that reason.) The rows hold no IP, question, URL, error message or code, and their `usd` stays 0 because the model calls are already counted in `requests`.
- **A judges' reserve**: `RATCHET_SCOUT_JUDGE_RUNS_PER_DAY` (and an optional `RATCHET_SCOUT_JUDGE_BUDGET_USD`) holds part of each day back. With the day's total T, the reserve J, public use P and judge use U, a public run starts only while P + max(0, U − J) is below T − J (for spend, the largest finished run today is added to the left side as a margin, so public runs in flight cannot spill into the reserve), and a judge run while P + U is below T. Judges get at least their reserve and can use public room that is left.
- **An access code**: a judge sends `RATCHET_SCOUT_JUDGE_CODE` once, in the body of `POST /scout/judge`, never in a URL or header (access logs record both). The gateway keeps only its SHA-256 digest and a derived key, allows 10 attempts per IP per hour, and answers with a signed `ratchet_judge` cookie (`HttpOnly`, `SameSite=Strict`, `Secure` in live mode, 7 days by default). Rotating the code ends every session. No refusal message hints that a code exists. The code goes in the submission's testing instructions only.
- **Paced Tavily credits**: with `RATCHET_TAVILY_MONTHLY_CREDITS` set, every search reserves its credits first and settles them after the call. The public's allowance is spread evenly per UTC day until `RATCHET_TAVILY_PACE_UNTIL`, with a share held for judges. After Tavily answers 432 or 433 (plan or pay-as-you-go limit), searching pauses for an hour.
```mermaid
flowchart TB
A(["A Scout request in live mode"]) --> J{{"Valid judge cookie?"}}
J -->|yes| JP{{"P + U below T?"}}
J -->|no| PB{{"P + max(0, U − J) below T − J?"}}
JP -->|yes| H{{"Under the IP limit and the run cap?"}}
PB -->|yes| H
H -->|yes| CR{{"Tavily credits left?"}}
CR -->|yes| OK["Admit, write a scout_run row"]
CR -->|no| NS["Run without search, labelled unavailable"]
OK --> RUN["Run the agents"]
NS --> RUN
RUN --> END(["add_spend in a finally block, also on failure"])
JP -->|no| R["Refused, nothing spent"]
PB -->|no| R
H -->|no| R
class J,JP,PB,H,CR gate
class R danger
class NS outline
class END ok
```
**Reading this diagram.** T is the day's total, J the judges' reserve, P the public use so far and U the judge use so far. A visitor with the judge cookie is checked against the whole day, so the reserve is theirs. A public visitor is checked against what is left after the reserve, plus any reserve judges did not use. Credits are the last check, and failing it does not refuse the run: it only removes the search. A refusal starts nothing and spends nothing: it is an `error` event on the stream, or a 429 from `POST /scout/run`. Every admitted run ends in `add_spend`, even when it fails.
A worked example (made-up round numbers, not a measurement): with a daily budget T of $0.50 and a reserve J of $0.10, the public can spend up to $0.40 before public runs are refused. If judges have used nothing, a judge can still start a run until the whole $0.50 is spent. If judges have used $0.15 (more than their reserve), the extra $0.05 counts against the public side, so public runs stop at $0.35.
The guard's limits: it covers Scout only, not `/v1/chat/completions` (API keys or the Caddy guard protect that, see [Security](/guides/security/)). Many IP addresses can still use up the public share early; the judges' reserve is held back from them. And the cap is checked when a run starts, so the worst case is the cap plus the runs already in flight (a few cents each).
### When search fails
A search that fails (any HTTP or transport error, such as a revoked key's 401 or Tavily's 432) or that the credit meter refuses does not end the run:
- The searcher step carries `detail.unavailable` with a fixed reason or the error's class and status, never its message or URL. No more queries are made.
- Triage and reader run on whatever earlier queries found. With nothing found they make no calls.
- The writer still answers, without (some) sources, and its step says `search_unavailable: true`. The `done` event's `search` is `{"available": false, "partial": ..., "reason": ...}`, and the page labels the run "search unavailable".
- The planner, pinned to the teacher, ran before any search, so every admitted run makes a real Token Factory call in live mode, with or without search.
- Failures (not credit refusals) are counted in `/scout/status` and written as a `search_error` event on the 1st, 10th, 100th and 1000th, so a broken key stays visible.
### What can go wrong
| Failure | What the system does | Where you see it |
|---|---|---|
| The question is empty or over 300 characters | The stream sends an `error` event; `POST /scout/run` answers 400 | the `error` event or status |
| The guard refuses (budget, runs, IP hour) | Nothing starts, nothing is spent | an `error` event; 429 from `POST /scout/run` |
| No Tavily key in live mode | 503 before any model call | `GET /scout/status` |
| Search fails or credits run out | The run goes on, labelled search unavailable; the planner's call is still paid | `detail.unavailable`, `search_error` or `tavily_exhausted` events |
| A model call fails mid-run | The run ends with an `error` event; the spend so far is counted | `Guard.add_spend` runs in a `finally` block |
| The gateway restarts | Today's counters are rebuilt from `scout_run` rows | `GET /scout/status` |
| Someone guesses the judge code | 10 attempts per IP per hour, then refused; no message says a code exists | `judge` events only for success |
### What is measured
Scout uses Tavily and the real cascade, with the teacher on Nebius Token Factory. Ratchet's own adapter is not connected to the live gateway, so student calls go to the hosted, untuned Nano 30B only through the random split, and carry its badge (its own calibration did not clear the bar, `evidence/hosted-nano`); the three recorded runs in `evidence/scout` (23 to 24 Oct, before the split) had all 57 calls answered by the teacher. As of Oct 29, 2026 no measured live Scout savings figures are published.
## For engineers
**Code** (links to `main`):
| File | What |
|---|---|
| [`ratchet/scout/agents.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/scout/agents.py) | `Scout.run(question, emit, mode, run_id)`: planner, searcher, triage, reader, writer; emits `start`, `step`, `done`. `Call`, `http_caller` (any gateway over HTTP). A `SearchUnavailable` or `httpx.HTTPError` from search ends the searching, not the run |
| [`ratchet/scout/prompts.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/scout/prompts.py) | System prompts and message builders (`plan_messages`, `triage_messages`, `read_messages`, `write_messages`), `EXAMPLE_QUESTIONS`. Never build Scout messages anywhere else |
| [`ratchet/scout/search.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/scout/search.py) | `TavilySearch`, `SearchUnavailable` |
| [`ratchet/scout/guard.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/scout/guard.py) | `Guard`: `admit`, `progress`, `add_spend`, `restore`, the pool rule for the judges' reserve |
| [`ratchet/scout/judge.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/scout/judge.py) | `JudgeAccess` (digest and derived cookie key, never the code), `AttemptLimiter` |
| [`ratchet/scout/credits.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/scout/credits.py) | `CreditMeter` (monthly credits, daily pacing, judge share, `reserve` and `settle`), `MeteredSearch` |
| [`ratchet/gateway.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/gateway.py) | `_scout_precheck` (calls `Guard.admit`), `_run_scout` (`Guard.add_spend` in a `finally`), `Runtime.scout`, the `/scout/*` routes |
| [`ratchet/tasks.py`](https://github.com/harshbhx/ratchet/blob/main/ratchet/tasks.py) | The four built-in Scout tasks: `source-triage` and `page-read` with their label fields, `research-plan` and `research-write` with `pinned=True` |
**Settings** (from `ratchet/config.py` and [CONFIGURATION.md](/reference/configuration/)):
| Variable | Default | Meaning |
|---|---|---|
| `TAVILY_API_KEY` | empty | Tavily key; live Scout answers 503 without it |
| `RATCHET_SCOUT_DAILY_BUDGET_USD` | `3.00` | Scout spend per UTC day |
| `RATCHET_SCOUT_MAX_RUNS_PER_DAY` | `30` | Runs per UTC day, everyone together |
| `RATCHET_SCOUT_RUNS_PER_HOUR` | `6` | Runs per client IP per rolling hour |
| `RATCHET_SCOUT_MAX_RESULTS` | `4` | Tavily results per query |
| `RATCHET_SCOUT_JUDGE_CODE` | empty | Judge access code, 16 to 256 characters; set it if and only if judge runs are above 0 |
| `RATCHET_SCOUT_JUDGE_RUNS_PER_DAY` | `0` | Runs held back for judges |
| `RATCHET_SCOUT_JUDGE_BUDGET_USD` | unset | USD held back for judges; unset means their share of the daily budget |
| `RATCHET_SCOUT_JUDGE_SESSION_S` | `604800` | Judge cookie lifetime (7 days) |
| `RATCHET_TAVILY_MONTHLY_CREDITS` | `0` | Credits per cycle; 0 counts without a cap |
| `RATCHET_TAVILY_JUDGE_CREDITS` | unset | Credits held for judges |
| `RATCHET_TAVILY_CYCLE_DAY` | `1` | Day of the month the credit cycle starts (00:00 UTC) |
| `RATCHET_TAVILY_PACE_UNTIL` | unset | Date to spread the public allowance until, such as `2026-12-15` |
**Endpoints** ([API.md](/reference/api/#scout-endpoints)): `GET /scout/examples`, `GET /scout/status` (budget, runs, judges' reserve, search and credits, `fallback_student`, `student_serving`), `GET /scout/stream?q=`, `POST /scout/run`, `POST /scout/judge`, `DELETE /scout/judge`. All are public; the guard protects them.
**CLI**, Scout as an outside client:
```bash
python -m ratchet.scout "What is LoRA fine-tuning?" --base-url http://localhost:8000/v1
# also --api-key (default $RATCHET_API_KEY or "unused") and --max-results (default 4)
```
Search runs on your machine with `TAVILY_API_KEY`. These calls are plain `/v1` traffic: they pass through API keys and spend limits, not through the Scout guard or the credit meter. The command exits 0 when the run finishes, also with search unavailable, and 1 when a model call fails.
**Events** in the release log: `search_error`, `tavily_exhausted`, `judge` (a session opened, with no IP and no code).
## Common questions
### Why not let the student try the planner and writer too?
Their answers are free text with nothing to compare exactly, so Ratchet could never certify the student on them, and every call would still pay for student samples. Pinning keeps those calls on the teacher and keeps their text out of the training data.
### What happens when the student is wrong on a triage call?
If it is unsure, the call escalates to Ultra before anything is served. If it is confidently wrong, a random audit may catch it, and audited precision drives the release gate and rollback. Reader precision covers `answers_question` and `fact_type` only: the free-text `key_fact` the writer cites is not checked. That limit is stated in the README.
### What does a live run cost?
A run makes 1 planner call, up to 12 triage calls, up to 6 reader calls and 1 writer call. At list prices that is roughly $0.02 to $0.03 per run before the student is trained, and under $0.01 after (estimate, [VM kit guide](/reference/deploy-nebius-vm/), prices as of Oct 2026, unverified). Tavily adds about 3 credits a run.
### Can someone run up my bill through Scout?
Only up to the daily spend cap, plus the runs already in flight when it is reached. Runs per day and per IP are capped too, and the counters survive restarts. The guard does not cover `/v1/chat/completions`; protect that with API keys or the Caddy guard.
### Why is a judge code needed at all?
The project gallery is public. Without a reserve, any visitor could use the day's runs before a judge arrived. A judge without the code still gets the public pool, and a run without search still calls Token Factory.
### Does Scout store the questions people ask?
The ledger stores no question and no client IP for Scout runs: the spend rows hold only the pool, amounts and outcome. Questions do appear in the gateway's uvicorn access log, because `/scout/stream?q=` carries them in the URL, and in Caddy's access log in the Nebius VM kit, where those logs rotate. In live mode the search queries go to Tavily under Tavily's terms, so do not type personal data into the live gateway.
## Related
---
# Scout: the multi-agent research tool
Source: https://docs.ratchet.dualmindlab.tech/reference/scout/
Scout is a small team of five research agents (planner, searcher, triage, reader and writer) that answers a question from the web. Every agent is a plain OpenAI client pointed at Ratchet. Two of its agents make many narrow, repetitive calls. Ratchet learns those calls and moves them from [Nemotron](https://docs.ratchet.dualmindlab.tech/guides/glossary/#nemotron) 3 Ultra to the student. The planner and the writer make open-ended calls, which stay on Ultra; the searcher is a tool, not a model call. In the mock-mode gateway the student takes over most of the narrow calls. On the live gateway no trained student is served yet, so Ultra answers, and a random 40% of eligible calls go to the hosted Nano 30B (a quarter re-checked by Ultra); the three recorded live runs in `evidence/scout` show Scout working end to end, not savings.
It exists to show Ratchet on real agent traffic: the kind of narrow call that, NVIDIA Research estimates, could go to small models, in the three open agent systems it studied (about 40%, 60% and 70% of their LLM queries; [arXiv 2506.02153](https://arxiv.org/abs/2506.02153), v3, Sep 22, 2026).
## Agents
| Agent | Task name | Calls per question | Route | Output (JSON) |
|---|---|---|---|---|
| Planner | `research-plan` | 1 | **pinned** to the teacher | `{"queries": [str, str, str]}` |
| Searcher | none (tool) | 1 per query | Tavily Search API (`search_depth: basic`, 1 credit) | results: title, url, content |
| Triage | `source-triage` | 1 per search result (up to 12) | **distillable** | `{"relevant": bool, "source_type": "docs\|paper\|blog\|forum\|news\|other", "reason": str}` |
| Reader | `page-read` | 1 per relevant result (up to 6) | **distillable** | `{"answers_question": bool, "fact_type": "definition\|number\|date\|name\|none", "key_fact": str}` |
| Writer | `research-write` | 1 | **pinned** to the teacher | `{"answer": str (markdown, cites [n]), "used": [int]}` |
- **Distillable** tasks compare only their label fields: `relevant` + `source_type`, and `answers_question` + `fact_type`. Free text (`reason`, `key_fact`) is ignored when deciding agreement, as with `ticket-triage`. So page-read's [precision](https://docs.ratchet.dualmindlab.tech/guides/glossary/#precision) covers its `answers_question` and `fact_type` fields only; the free-text `key_fact` the writer cites is not checked.
- **Pinned** tasks (`Task.pinned = True`) always go to the teacher and are never stored as training examples. Open-ended planning and writing are where the big model earns its price.
- Triage and Reader calls run in parallel (thread pool). The ledger is guarded by a lock.
## Modes
- **mock** (no keys): the searcher returns mock results from `sim/research.py`, and the mock teacher and students answer. Only the example questions work (`GET /scout/examples`). The UI must show a "Mock search results" label in this mode.
- **live**: [Tavily](https://docs.ratchet.dualmindlab.tech/guides/glossary/#tavily) (`TAVILY_API_KEY`) plus the real Ratchet cascade (Token Factory teacher, [vLLM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#vllm) student or hosted fallback). A budget guard protects the public live gateway.
## HTTP endpoints
| Method | Path | Purpose |
|---|---|---|
| GET | `/`, `/scout`, `/dashboard` | The React app (`ratchet/web/dist`, built from `web/`). The hand-written pages are served at `/lite`, `/lite/scout` and `/lite/dashboard`, and when there is no build |
| GET | `/scout/examples` | `{"mode": "mock"\|"live", "examples": [str, ...], "free_text": bool}`. `free_text` is false in mock mode |
| GET | `/scout/stream?q=` | Server-Sent Events, one JSON object per `data:` line (below) |
| POST | `/scout/run` | Body `{"question": str}`. Returns `{"events": [ ...same objects... ]}` once the run finishes |
| GET | `/scout/status` | `{"mode", "budget_usd", "spent_today_usd", "runs_today", "runs_per_hour_per_ip", "max_runs_per_day", "tavily": bool, "fallback_student", "student_serving"}`. `budget_usd` and `max_runs_per_day` are `null` in mock mode; `fallback_student` and `student_serving` (v0.5) say which model answers student calls ([Who answered](#who-answered)). Since v0.5 also `public_runs_left_today`, `public_budget_left_usd`, `judge_runs_today`, `persistent`, `judge` and `search` (below) |
| POST | `/scout/judge` | Body `{"code": str}`: opens a judge session (a signed `ratchet_judge` cookie). 400, 401, 404 (judge access off), 429 (10 attempts per IP per hour) |
| DELETE | `/scout/judge` | Ends a judge session: clears the cookie |
Request and response details for each endpoint are in [API.md](/reference/api/#scout-endpoints).
### Event objects (SSE `data:` payloads, in order)
```json
{"type": "start", "run_id": "a1b2c3", "question": "What is LoRA fine-tuning?", "mode": "mock"}
{"type": "step", "agent": "planner", "task": "research-plan", "served_by": "teacher",
"model": "nvidia/Nemotron-3-Ultra-550b-a55b", "confidence": null, "escalated": false,
"cost_usd": 0.0013, "baseline_usd": 0.0013, "latency_ms": 2400.0,
"detail": {"queries": ["LoRA low-rank adaptation explained", "LoRA paper Hu 2021", "LoRA vs full fine-tuning"]}}
{"type": "step", "agent": "searcher", "task": null, "served_by": "tool", "model": "tavily",
"confidence": null, "escalated": false, "cost_usd": 0.0, "baseline_usd": 0.0, "latency_ms": 650.0,
"detail": {"query": "LoRA paper Hu 2021", "results": [{"id": 3, "title": "...", "url": "...", "snippet": "..."}]}}
{"type": "step", "agent": "triage", "task": "source-triage", "served_by": "student", "model": "student-v4",
"confidence": 0.94, "escalated": false, "cost_usd": 0.00003, "baseline_usd": 0.0011, "latency_ms": 310.0,
"detail": {"id": 3, "url": "...", "title": "...", "relevant": true, "source_type": "paper", "reason": "..."}}
{"type": "step", "agent": "reader", "task": "page-read", "served_by": "teacher", "model": "nvidia/Nemotron-3-Ultra-550b-a55b",
"confidence": 0.61, "escalated": true, "cost_usd": 0.0014, "baseline_usd": 0.0013, "latency_ms": 2710.0,
"detail": {"id": 3, "url": "...", "answers_question": true, "fact_type": "date", "key_fact": "..."}}
{"type": "step", "agent": "writer", "task": "research-write", "served_by": "teacher", "model": "nvidia/Nemotron-3-Ultra-550b-a55b",
"confidence": null, "escalated": false, "cost_usd": 0.0021, "baseline_usd": 0.0021, "latency_ms": 3100.0,
"detail": {"answer": "LoRA freezes the base weights and trains small low-rank matrices [1]. ...", "citations": [{"n": 1, "title": "...", "url": "..."}],
"search_unavailable": false}}
{"type": "done", "run_id": "a1b2c3", "answer": "...", "citations": [{"n": 1, "title": "...", "url": "..."}],
"totals": {"calls": 17, "llm_calls": 15, "student_calls": 11, "teacher_calls": 4, "cost_usd": 0.0062,
"baseline_usd": 0.0189, "savings_pct": 0.67, "wall_ms": 4200.0},
"search": {"available": true, "partial": false, "reason": null}}
{"type": "error", "message": "The daily Scout budget is used up. Try again tomorrow (UTC), or run Ratchet yourself."}
```
- `served_by` is `student`, `teacher` or `tool`. `escalated` is true when a student was tried and was below its threshold. `model` is the id the call came back with: a student step from the hosted fallback carries `fallback_student` ([Who answered](#who-answered)).
- `baseline_usd` is what the same call would cost on Ultra alone (Ratchet's `all_teacher_cost_usd`).
- Steps for parallel agents may arrive in any order. Use `detail.id` to match a triage step and its reader step to a search result.
- An `error` event ends the stream. `/scout/stream` always answers HTTP 200 and reports problems as an `error` event, because browsers' `EventSource` cannot read status codes. `/scout/run` uses HTTP codes instead: 400 (empty question, a question over 300 characters, or a non-example question in mock mode), 429 (rate limit, daily run cap or budget), 503 (live mode without `TAVILY_API_KEY`), 502 (a model call failed during the run).
- Every step keeps exactly the top-level keys above; new fields go inside `detail`.
- A call the live gateway's split sent to the hosted Nano at random (`RATCHET_DEMO_SPLIT`) has `detail.route = "demo_split"`, and `totals.demo_split_calls` counts them; the page labels such a call "[Random split](https://docs.ratchet.dualmindlab.tech/guides/glossary/#random-split)" (picked at random, not by the confidence gate). Absent otherwise.
### When search is unavailable (v0.5)
A search that fails (any HTTP or transport error, for example a revoked key's 401 or [Tavily](https://docs.ratchet.dualmindlab.tech/guides/glossary/#tavily)'s 432 when credits run out) or that the credit meter refuses does not end the run:
- The searcher step for that query has `detail = {"query": ..., "results": [], "unavailable": ""}` and no further queries are made. The reason is `public search credits for today are used up`, `search credits are used up`, or the error's class and status (`HTTPStatusError 401`, `ConnectError`), never its message or URL.
- Triage and reader run on what earlier queries found. With nothing found they make no calls, and the writer gets an empty fact list (its prompt already says to say when the facts do not answer the question). The answer text is the model's own, never rewritten.
- The writer step's `detail.search_unavailable` is `true`, and the `done` event's `search` is `{"available": false, "partial": , "reason": ""}` (`partial`: some results were found first). `/scout/run` answers 200: the run finished.
- The planner is pinned to the teacher and runs before any search, so every admitted run makes a runtime Token Factory call, with or without search.
- This is a fallback, not a hidden failure: `/scout/status.search` shows the last failure and the failures today, a broken key still needs fixing, and failures (not credit refusals) write a `search_error` event on the 1st, 10th, 100th and 1000th.
## Who answered
Each step's badge names the model that answered it (`web/src/lib/who-model.ts`, `whoLabel`). A step's `served_by` says student, teacher or tool; for a student step only its `model` tells our adapter from the hosted fallback, which the live gateway sends student calls to when the GPU endpoint is off (no `RATCHET_STUDENT_BASE`). The page reads `fallback_student` and `student_serving` from `/scout/status` and compares:
| Step | Badge | Tooltip |
|---|---|---|
| `student`, `model` is a Night School adapter (`student-vN`) | Nano + your adapter | "Nemotron 3 Nano 4B with our LoRA adapter v[N], built to run on a Nebius Serverless Endpoint" |
| `student`, `model` is `fallback_student` | Nano 30B · hosted | "Untuned Nemotron 3 Nano 30B on Token Factory, held to confidence threshold [T]. Not covered by our adapter's precision certificate." [T] is `student_serving.threshold`. With daily endpoint windows (`student_serving.window`) it adds "Our adapter answers during the daily endpoint windows." |
| `teacher` | Ultra · Token Factory | "Nemotron 3 Ultra on Nebius Token Factory" |
| `tool` | Tavily, or Search | the tool's id |
In mock mode the tooltips start with "Simulation: a mock model plays". When a student answer of the run came from the hosted fallback, or none came from the student while the hosted fallback serves, the totals read "Answered by the student" and the summary says "answered by the hosted Nano (untuned)" instead of "by your adapter". The lite page (`/lite/scout`) uses the same words.
## Budget guard (live mode)
`ratchet/scout/guard.py`. It is checked once, when a run is admitted (: every run, judge runs included, goes through `Guard.admit` before it starts and `Guard.add_spend` when it ends).
| Variable | Default | Meaning |
|---|---|---|
| `TAVILY_API_KEY` | empty | Tavily key. Without it, live Scout returns 503 |
| `RATCHET_SCOUT_DAILY_BUDGET_USD` | `3.00` | Maximum Ratchet spend on Scout runs per UTC day. When reached, Scout returns 429 |
| `RATCHET_SCOUT_RUNS_PER_HOUR` | `6` | Runs per client IP per rolling hour |
| `RATCHET_SCOUT_MAX_RUNS_PER_DAY` | `30` | Runs per UTC day, all clients together. When reached, Scout returns 429. This caps Tavily credits: each run makes about 3 searches (1 credit each), so 30 runs use at most about 90 credits a day |
| `RATCHET_SCOUT_MAX_RESULTS` | `4` | Tavily results per query |
| `RATCHET_TRUST_PROXY` | `0` | `1` = take the client IP from the first `X-Forwarded-For` entry. Use it only behind a proxy that overwrites that header. The VM does not need it: uvicorn runs there with `--proxy-headers`, which already gives the gateway the real client IP from Caddy |
How spend is counted:
- Spend is added **per step**, from each step's `cost_usd` (which includes audit and shadow calls Ratchet made for it). A run that fails halfway still counts what its finished steps cost.
- A run that is admitted counts against the hourly and daily run limits even if it fails.
- The budget is checked when a run starts, not during it. A run that starts just under the cap can finish over it, so the real daily maximum is the cap plus the runs in progress at that moment (a few cents each).
- In live mode the counters are kept in the ledger and restored at startup, so a restart, `update.sh` or a crash does not reset them (v0.5). This holds where the ledger is on a volume (the Nebius VM kit). On the Cloudflare Containers live gateway the ledger is on the container's disk and a restart (after 30 idle minutes) resets the counters; the low defaults in `deploy/cloudflare-live/src/index.ts` and the Token Factory balance are the limit there. Each admitted run writes a `scout_run` row to the `spend` table: `usd` stays 0 (the model calls are already in `requests.cost_usd`), and `detail` holds `{"judge", "llm_usd", "status"}`, updated after each model call and when the run ends (`done`, `error`, `search_unavailable`). A row still `admitted` (a process killed mid-run) counts its last recorded cost. Rows hold no IP, question, URL, error message or code. Counters reset at 00:00 UTC; a run counts against the day it started.
- The per-IP windows stay in memory, so no IP is stored; they reset on restart. On the first restart after upgrading, runs from earlier that day have no rows and are not counted, once.
Mock mode has no budget and no daily run cap, writes no spend rows, but the per-IP rate limit applies, at 10x the live rate.
### Judges' reserve
`RATCHET_SCOUT_JUDGE_CODE` and `RATCHET_SCOUT_JUDGE_RUNS_PER_DAY` (with `RATCHET_SCOUT_JUDGE_BUDGET_USD`, [CONFIGURATION.md](/reference/configuration/#judge-access-and-tavily-credits)) hold back part of the day for evaluators, so visitors from the public gallery cannot use it up first. With the day's total T (runs, or USD), the judges' reserve J, public use P and judge use U today:
- A public run is admitted only while P + max(0, U − J) + m < T − J. For runs m is 0. For spend, m is the largest finished run today (when a judge budget is reserved), so public runs already in flight cannot spill into the judges' share.
- A judge run is admitted only while P + U < T. Judges get at least their reserve and can use public room that is left.
- Runs stay at or under T. Spend stays at or under the budget plus the runs in flight when it is reached, as before. Judge runs and public runs from one address have separate hourly windows.
On the Scout page, a small "I am a judge" link under the ask bar opens a password field and an Unlock button; it shows only when `GET /scout/status` reports `judge.enabled`, so it is absent in mock mode and on a gateway without a code. After a good code the page says "Judge access on" with the runs left today and offers Sign out (`DELETE /scout/judge`). A wrong code, too many tries (10 per IP per hour) and a gateway with judge access off each get a plain message. The page keeps the code in the input's memory only, sends it once in the POST body with `credentials: 'same-origin'`, clears it right after, and never stores or logs it ([invariant 18](https://github.com/harshbhx/ratchet/blob/main/AGENTS.md)'s rule, scanned by `test_dashboard_admin_key_stays_in_memory`). A judge sends the code once (`POST /scout/judge`, in the body: uvicorn and Caddy log URLs and custom headers) and gets a signed cookie (`ratchet_judge`, `HttpOnly`, `SameSite=Strict`, `Path=/scout`, `Secure` in live mode). Only the cookie makes a run a judge run. The gateway keeps the code's [SHA-256](https://docs.ratchet.dualmindlab.tech/guides/glossary/#sha-256) digest and a key derived from it, never the code; the cookie survives a restart, and rotating the code ends every session. A judge without the code is a public visitor: with search unavailable they still get a run that calls Token Factory, where a 429 would give them nothing.
### Tavily credits
The free Tavily plan gives 1,000 API credits a month; a basic search costs 1 credit, an advanced one 2 ([Tavily docs](https://docs.tavily.com/documentation/api-credits), checked Oct 20, 2026). Tavily's monthly reset day is not documented (unverified), so the cycle day is a setting. With `RATCHET_TAVILY_MONTHLY_CREDITS` set, `ratchet/scout/credits.py` meters every search:
- Each search reserves its credits first and is settled after the call with what it cost: `usage.credits` when Tavily reports it (the request sends `include_usage`), else the table amount; nothing for a 4xx (432 and 433 included) or a request never sent; the table amount for a 5xx or a failure after the request was sent, because whether Tavily bills those is not documented. Each call writes a `tavily` spend row (`usd` 0; `{"judge", "credits", "depth", "http", "counted"}`).
- Public use is paced: today's allowance is what the public pool had left at 00:00 UTC, spread evenly over the days until `RATCHET_TAVILY_PACE_UNTIL` (while it falls in this cycle) or the cycle's end; unused credits roll forward. Judges draw on their reserve (`RATCHET_TAVILY_JUDGE_CREDITS`) first, then on today's allowance, so public and judge use together never pass the monthly credits.
- After a 432 ("Key limit or Plan Limit exceeded") or 433 ("PayGo limit exceeded"), no search is sent for an hour (one `tavily_exhausted` event).
- Credits never refuse a run: a public run whose allowance is used up runs without search, labelled (above). The run and USD caps bound the cost.
- The meter restores the cycle's rows at startup. Reconcile it with the Tavily dashboard weekly ([RUNBOOK.md](/reference/runbook/)).
Mock mode settings:
| Variable | Default | Meaning |
|---|---|---|
| `RATCHET_MOCK_WARMUP_DAYS` | `0` | Mock days of `mixed` traffic, each followed by a night, run at startup (and on a mock-mode reset) so the mock-mode gateway opens with a trained student. `4` takes about 2 s and leaves a live student that answers about 80–90% of Scout's LLM calls (mock-mode run, mock models: 14–18 of 17–20 calls per example question, 72–86% cheaper than all-Ultra) |
| `RATCHET_MOCK_WARMUP_REQUESTS` | `1500` | Requests per warm-up day |
| `RATCHET_MOCK_PACE` | `0.25` | `/scout/stream` waits this share of each call's mock latency (Ultra 2.4 s → 0.6 s), so agents visibly work. `0` = no waiting |
## Mock research world (`sim/research.py`)
- 8 example questions about well-established ML concepts ([LoRA](https://docs.ratchet.dualmindlab.tech/guides/glossary/#lora), knowledge distillation, model cascades, the [Wilson score](https://docs.ratchet.dualmindlab.tech/guides/glossary/#wilson-bound) interval, [vLLM](https://docs.ratchet.dualmindlab.tech/guides/glossary/#vllm) PagedAttention, Mamba state-space models, [self-consistency](https://docs.ratchet.dualmindlab.tech/guides/glossary/#self-consistency), model collapse). The facts are from the original papers.
- Each question has 3 planned queries and a pool of 72 mock results: 12 result templates (`r-*`) × 6 variants. The templates cover relevant papers, docs and blogs, off-topic news, near-miss forum posts, and SEO listicles. Each template has fixed labels for `source-triage` and `page-read`.
- Result URLs use `example.org`, so nothing pretends to be a real page.
- `ResearchStream` produces triage and read requests for the simulator and `POST /ratchet/demo/day?workload=research|mixed`. Users phrase each question 4 ways. Questions 7 and 8 (self-consistency, model collapse) appear only from the [drift](https://docs.ratchet.dualmindlab.tech/guides/glossary/#drift) day, as 20% of traffic.
- The simulator and the agents build messages with the same functions in `ratchet/scout/prompts.py`, so the mock teacher and students can look up each request by its exact user text.
## Recorded runs
The operator can save a real run as a file: the question, the run's events with their offsets and the totals. The gateway no longer serves these files (the `/scout/recorded` routes were removed); the three committed runs in `evidence/scout/` are evidence.
```bash
python -m ratchet.scout record "What is LoRA fine-tuning?" --base-url https://demo.example [--judge-code C] [--out evidence/scout] [--allow-mock]
```
It reads `GET /scout/stream` and writes `/-.json` (schema `ratchet.scout-recording/v1`: question, host name, mode, events with `t_ms`, totals). It exits 1 and writes nothing when the run errors, does not finish, or the gateway is in mock mode and `--allow-mock` is not given. The judge code (or `$RATCHET_SCOUT_JUDGE_CODE`) is used for one `POST /scout/judge` and never written. The gateway never writes a recording, so no visitor's question is stored.
## Command line
```bash
python -m ratchet.scout "What is LoRA fine-tuning?" --base-url http://localhost:8000/v1
```
Runs Scout as an outside client: each agent call goes to the gateway's `/v1/chat/completions` with an `X-Ratchet-Task` header. Flags: `--base-url` (default `http://localhost:8000/v1`), `--api-key` (default `$RATCHET_API_KEY` or `unused`), `--max-results` (default 4). Search runs locally: Tavily when `TAVILY_API_KEY` is set, otherwise mock results, which only cover the example questions and only make sense against a mock-mode gateway. These calls do not pass through the budget guard or the credit meter; they are plain `/v1` traffic. The command exits 0 when the run finishes and 1 when a model call fails (the error, such as a 401 for a wrong key, goes to stderr). A failed search prints ` -> search unavailable ()` and a warning on stderr, and the run still finishes with exit code 0.
---
# Where everything runs
Source: https://docs.ratchet.dualmindlab.tech/reference/where-it-runs/

**When the live gateway has slept.** The live gateway's container sleeps after 30 minutes idle (a Cloudflare setting), so the first visitor after a quiet spell waits about 20 seconds while it starts, and its [ledger](https://docs.ratchet.dualmindlab.tech/guides/glossary/#ledger) counters reset. Instead of an error, the main site shows a banner that the gateway is waking up and fills in by itself; the live address shows a splash page that reloads itself. This is a cost choice, not an outage.
This page says, for every part of Ratchet, where it runs **today** and where it is **designed** to run on [Nebius AI Cloud](https://docs.ratchet.dualmindlab.tech/guides/glossary/#nebius-ai-cloud). Read the last column first: **Real** rows run today. **Scripted, ready to run** means the scripts exist and pass their tests against a fake `nebius` command, ready for `make nebius-plan` and `make nebius-up`. [Token Factory](https://docs.ratchet.dualmindlab.tech/guides/glossary/#token-factory) (the [teacher](https://docs.ratchet.dualmindlab.tech/guides/glossary/#teacher)'s calls), one hand-run [Serverless Job](https://docs.ratchet.dualmindlab.tech/guides/glossary/#serverless-job) (the round-2 retraining, 27 Oct 2026) and one hand-run [Serverless Endpoint](https://docs.ratchet.dualmindlab.tech/guides/glossary/#serverless-endpoint) (a speed test of the [student](https://docs.ratchet.dualmindlab.tech/guides/glossary/#student), 27 Oct 2026) and one hand-run Night School (a CPU VM, a spot [L40S](https://docs.ratchet.dualmindlab.tech/guides/glossary/#l40s) job and an endpoint, 28 Oct 2026) are the Nebius services that have run. Night School ran end to end once on Nebius, by hand (28 Oct 2026): the gateway on a Nebius CPU VM, the training job on a spot [L40S](https://docs.ratchet.dualmindlab.tech/guides/glossary/#l40s), the adapter served from a Nebius Serverless Endpoint. The [golden gate](https://docs.ratchet.dualmindlab.tech/guides/glossary/#golden-gate) rejected it (81.8% precision against the 95% target), so it was never made live ([`evidence/nightschool-nebius/`](https://github.com/harshbhx/ratchet/blob/main/evidence/nightschool-nebius/README.md)). The live site's own gateway is still on Cloudflare and is not connected to that endpoint, and the loop has not run nightly on live traffic. The one-command VM kit and the nightly schedule have not run.
## The table
| Component | What it is | Runs today where | CPU or GPU | Designed to run on Nebius as | Status |
|---|---|---|---|---|---|
| Gateway and router | The FastAPI app: the OpenAI-compatible routes, the cascade (student first, teacher on doubt), the release gate | The live gateway on Cloudflare Workers with Containers | CPU | A small CPU VM with Docker Compose and Caddy ([`deploy/vm`](https://github.com/harshbhx/ratchet/tree/main/deploy/vm)) | Real today; the VM ran once by hand for the 28 Oct Night School, the one-command kit is Scripted, ready to run |
| Ledger | One SQLite file: requests, stored teacher answers with their license, versions, signed bills of materials | On the live gateway container's disk (a restart resets the day's counters) | CPU | The VM's Docker volume, with a daily online backup | Real today; the VM ran once by hand for the 28 Oct Night School, the one-command kit is Scripted, ready to run |
| Night School (training) | Builds the dataset, fine-tunes a LoRA adapter, grades it on the held-out golden set | On 27 Oct 2026 the round-2 text-to-SQL recipe (same 4,322 examples, same settings) was retrained on Nebius AI Cloud by hand, as a Serverless Job on one preemptible (spot) L40S, with checkpoints in an Object Storage bucket (evidence: [`evidence/text2sql-round2-nebius/`](https://github.com/harshbhx/ratchet/blob/main/evidence/text2sql-round2-nebius/README.md)). The retrained adapter ([on Hugging Face](https://huggingface.co/harshbhx/ratchet-nemotron-nano-4b-text2sql-nebius)) scored 40.7% (70 of 172) on exam 1 and 34.5% (69 of 200) on exam 2, against 39.0% and 33.0% for the original adapter: the result reproduces (the row-by-row differences are within chance). The original published adapters were trained by hand on a borrowed GPU. Night School then ran end to end once on Nebius, by hand, on 28 Oct 2026: a CPU VM imported 4,322 existing Ultra-labelled rows (no new teacher calls), a spot L40S job trained on 3,864 of them (2 epochs, 966 steps, 1,478 s), the adapter was hot-loaded into a Serverless Endpoint and the golden gate rejected it (81.8% precision against the 95% target on the 11 answers it would serve), so it was never made live; four job submissions, a re-run of the eval with a 120 s timeout through a one-off driver and five VM-only fixes are all stated in the evidence ([`evidence/nightschool-nebius/`](https://github.com/harshbhx/ratchet/blob/main/evidence/nightschool-nebius/README.md)). The nightly schedule has not run on live traffic | GPU | A Nebius Serverless Job on an L40S, with a bucket mounted at `/data` ([`deploy/train_job.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/train_job.sh)) | Real (the hand-run Nebius jobs and the borrowed-GPU training; one Night School run, rejected by the gate); Built and unit-tested, not run nightly on live traffic (the schedule) |
| Student serving | Nemotron 3 Nano 4B with a LoRA adapter you own, served by vLLM | Ratchet's own adapter is not connected to a student endpoint on the live site: every live model call runs on Nebius Token Factory, where Ultra teaches and a random 40% of eligible requests are answered by the hosted Nano 30B, a quarter of those re-checked by Ultra. On 27 to 28 Oct 2026 the student was served by hand, in two short sessions, on a Nebius Serverless Endpoint (one spot L40S) for speed tests: cold start about 7.2 to 7.6 minutes to the first answer; a restart (stop, then start) about 7.1 minutes, with the adapters loaded again each time; 16 requests at once p50 1.6 s (SQL adapter) and 1.2 s (tool-calling adapter); about $0.67 of GPU time in all; the endpoint was deleted afterwards (evidence: [`evidence/student-endpoint-nebius/`](https://github.com/harshbhx/ratchet/blob/main/evidence/student-endpoint-nebius/README.md)) | GPU | A Nebius Serverless Endpoint on an L40S ([`deploy/student_endpoint.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/student_endpoint.sh)) | Real (the one hand-run speed test, not connected to the live gateway); Scripted, ready to run (the endpoint script and the live gateway's connection to it) |
| Teacher | Nemotron 3 Ultra, an open model whose license allows training on its outputs | Nebius Token Factory, called per token from the live gateway | Hosted by Nebius; you do not size it | The same | Real |
| Scout and Tavily | The research agents on the same gateway; Tavily is the web-search API they call (not a Nebius product) | The live gateway (real Token Factory and Tavily calls, under a spend guard) and recorded runs | CPU | The same gateway on the VM | Real. The story site's `/scout` page uses the live gateway (`SCOUT_ORIGIN`, DEPLOY_PAGES.md) |
| Dashboard and web app | The React app the gateway serves at `/`, `/scout` and `/dashboard` | The live gateway, from its image | CPU | The VM, from the same image | Real |
| Docs site and story site | Static pages built from `docs-site/` and `web/` | Cloudflare Pages | none (static files) | Nothing planned: static files stay where they are | Real |
| Live gateway | `live.ratchet.dualmindlab.tech`: the gateway in live mode | Cloudflare Workers with Containers ([`deploy/cloudflare-live`](https://github.com/harshbhx/ratchet/blob/main/deploy/cloudflare-live/README.md)) | CPU | The VM runs the same image | Real |
| Main site | `ratchet.dualmindlab.tech`: static pages on Cloudflare Pages whose worker sends every gateway-backed path (dashboard, judges, `/ratchet/*`, `/scout/*`) to the live gateway by default; `GATEWAY_ORIGIN` switches it (`DEPLOY_PAGES.md`) | Cloudflare Pages | none | the gateway serves the same pages | Real for gateway numbers |
| Tests and CI | `pytest`, lint and the reference runs: no network, no GPU | Your laptop and GitHub Actions | CPU | Not applicable | Real |
## The router is just CPU
The gateway does no model inference. It checks keys, routes a request, calls the student and teacher over HTTP, writes the [ledger](https://docs.ratchet.dualmindlab.tech/guides/glossary/#ledger) and serves the web pages, so a 2 vCPU, 8 GB machine is enough and costs about six cents an hour (estimate, below). That is why it already runs on Cloudflare's CPU-only containers and on a laptop. Only two things need a GPU: **training** (Night School, a job that runs for under an hour a night and then stops) and **serving the student** (an endpoint that bills for every hour it is on, so it should be stopped when idle). Without a student endpoint the whole system still works: the teacher answers every call except the live site's random 40% split to the hosted Nano 30B (a quarter of those re-checked by Ultra), which is how the live gateway runs today.
## One command to host it all on Nebius
> **Scripted and tested offline, ready to run, apart from the hand-run steps.** Only these Nebius steps have run for real, all started by hand: on 27 Oct 2026 the round-2 text-to-SQL retraining, as a Serverless Job on one [preemptible](https://docs.ratchet.dualmindlab.tech/guides/glossary/#preemptible-gpu) L40S (evidence: [`evidence/text2sql-round2-nebius/`](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)), and a speed test of the student on a Serverless Endpoint (evidence: [`evidence/student-endpoint-nebius/`](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)); and on 28 Oct 2026 one Night School on a CPU VM, a spot L40S job and an endpoint, which the golden gate rejected (evidence: [`evidence/nightschool-nebius/`](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). Everything else below, including the one-command VM kit, the nightly schedule on live traffic and the live gateway's connection to a student endpoint, has not run: the live site's own gateway is still on Cloudflare and is not connected to that endpoint. `deploy/nebius/nebius.sh` and the scripts it calls were written from the Nebius CLI reference and tested offline against a fake `nebius` command, a fake `ssh` and a stub smoke check. Nothing here is endorsed by or affiliated with Nebius. Do the [day-1 checks](/reference/deploy-nebius-vm/#what-is-confirmed-and-what-is-not) on the first real run.
```bash
make nebius-plan # prints every command; changes nothing; calls no Nebius API
RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1 make nebius-up YES=1 # the CPU VM only
RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1 make nebius-up YES=1 ARGS="--with-student" # plus the GPU student endpoint
make nebius-status # read-only: the VM, the endpoint, the URLs
RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1 make nebius-down YES=1 # stop the endpoint, delete the VM
```
The same, without `make`: `bash deploy/nebius/nebius.sh plan|up|status|down [--yes] [--with-student] [--with-training] [--live]`.
**A real run needs both `--yes` and `RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1`.** Without both, `up` and `down` are the same dry run as `plan`: a local stand-in replaces the `nebius` command, so no Nebius API is called and nothing is created, started, stopped or deleted. The GPU parts are opt-in: `--with-student` adds the bucket and the student endpoint, `--with-training` also submits one training job, and `--live` switches the VM's gateway to live mode.
What `up` does, in order:
1. **Preflight.** The `nebius` CLI is installed and signed in, `jq` and `ssh` exist, the project has a subnet, the variables each flag needs are set (it names every missing one), and the platform fits the region (`cpu-e2` exists only in `eu-north1`). It prints the estimated hourly cost.
2. **The CPU VM** with the gateway and Caddy: [`deploy/vm/create_vm.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/create_vm.sh).
3. **With `--with-student`: a bucket** (unless `RATCHET_BUCKET_ID` is set) and **the student endpoint**: [`deploy/student_endpoint.sh start`](https://github.com/harshbhx/ratchet/blob/main/deploy/student_endpoint.sh), which creates it when no endpoint has the name. With `--with-training` it then submits [`deploy/train_job.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/train_job.sh) (the dataset must already be in the bucket at `night/train.jsonl`, and `RATCHET_TRAIN_IMAGE` must name an image built from `deploy/Dockerfile.train` and pushed where Nebius can pull it). It always prints the command that stops the GPU.
4. **Wiring.** `RATCHET_STUDENT_BASE` and `RATCHET_STUDENT_KEY` (and, with `--live`, `RATCHET_MODE=live` and the keys) are written to the VM's `.env` over `ssh` by [`deploy/vm/set_env.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/set_env.sh), with each value on stdin so it never appears in a command line, a log or the output; then [`deploy/vm/update.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/update.sh) rebuilds, checks the configuration and restarts.
5. **Smoke check** from outside with [`scripts/demo_check.py`](https://github.com/harshbhx/ratchet/blob/main/scripts/demo_check.py), then the URLs and the teardown command.
`down` stops the student endpoint **first** (a stopped endpoint is not billed for compute or its own storage; its mounted bucket still is), then deletes the VM and its boot disk. It never deletes the endpoint or the bucket: it prints those commands, because deleting them also deletes your adapters.
What you need before a real run: the Nebius CLI signed in, `NEBIUS_PROJECT_ID`, an SSH key, the repository public on GitHub (the VM clones it), and `RATCHET_STUDENT_KEY` for `--with-student` (`openssl rand -hex 24`). Every variable is in [CONFIGURATION.md](/reference/configuration/#deploy-scripts-deploy).
### Cost, as estimates
These are estimates from Nebius list prices read on Oct 20 to 22, 2026, in US dollars before tax. Your bill decides; the prices were not confirmed by a run.
| Part | Estimate | Source |
|---|---|---|
| CPU VM `cpu-e2`, `2vcpu-8gb` | about $0.060 an hour (about $43.80 a month) | [Compute pricing](https://docs.nebius.com/compute/resources/pricing), [DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/#cost-estimate) |
| 30 GiB boot disk | about $0.003 an hour ($2.13 a month) | the same |
| Student endpoint, one L40S | about $1.35 an hour while it runs, about $32 a day if left on | [DEPLOYMENT.md](/reference/deployment/#4-cost-estimates) |
| One training job, one L40S | about $0.70 to $1.35 per run; the job's own 2 hour limit caps it near $2.70 | the same |
| Bucket | per GiB-month, also while the endpoint is stopped | the same |
| Teacher calls | about $1.30 per 1,000 calls at list prices | the same |
At low traffic a 24/7 student endpoint costs more than it saves, and Ratchet's cost report says so ([Cost accounting](/reference/architecture/#7-cost-accounting)). Use `--with-student` when you want to measure the student, and stop it afterwards.
More detail: [DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/) (the VM, monitoring, backups, teardown), [DEPLOYMENT.md](/reference/deployment/) (the endpoint and the job step by step) and the guide [Host it on Nebius](https://docs.ratchet.dualmindlab.tech/guides/host-on-nebius/).
## Run the tests on your own machine
No keys, no GPU, no bill:
```bash
pip install -e ".[dev]"
make test # the offline test suite
```
Steps: [the README's quick start](https://github.com/harshbhx/ratchet/blob/main/README.md#use-it).
## Run the live gateway on a VM
The live gateway (real Token Factory calls) runs on any machine that runs Docker. On a Nebius VM, the steps are in [DEPLOY_NEBIUS_VM.md](/reference/deploy-nebius-vm/#switching-from-mock-to-live); `deploy/nebius/nebius.sh up --live` does the wiring for you. Anywhere else: `RATCHET_MODE=live` with `NEBIUS_API_KEY` and `RATCHET_API_KEYS` set (LIVE_MODE.md). The live gateway uses Cloudflare Workers with Containers instead ([`deploy/cloudflare-live`](https://github.com/harshbhx/ratchet/blob/main/deploy/cloudflare-live/README.md)), which is the host that has actually run.
## Unlock Scout runs as a judge
Hackathon judges can hold back runs and spend for themselves on the live Scout page. The access code is on the Devpost page, never in this repository. The picture below is an illustration of the page, not a recording.

How the access works: [Scout](/reference/scout/) and ADR 0120.
---
# Host it on Nebius
Source: https://docs.ratchet.dualmindlab.tech/guides/host-on-nebius/
Ratchet's Nebius deploy kit runs it on one small CPU VM that does no model inference, calls its teacher, judge and hosted fallback on Token Factory, and rents an L40S GPU only while you serve or train the student, so GPU money is spent only when you decide to spend it.
Everything below is wrapped in one script. `make nebius-plan` prints every command and changes nothing; `deploy/nebius/nebius.sh up --yes` (with `RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1`) creates the CPU VM, adds the GPU student endpoint only with `--with-student`, wires the VM's `.env` and runs a smoke check, and `down` stops the endpoint first and then deletes the VM. It is scripted and tested offline, **ready to run with `make nebius-plan` and `make nebius-up`**. On 28 Oct 2026 Night School ran end to end once on Nebius, by hand, on a CPU VM with a spot L40S training job and a Serverless Endpoint; the golden gate rejected it, so it was never made live ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)). Two other Nebius GPU steps have run so far, by hand: the round-2 retraining on 27 Oct 2026 (a Serverless Job on one preemptible L40S, checkpoints in Object Storage) and, the same day, one speed test of the student on a Serverless Endpoint (one spot L40S, deleted afterwards; [evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)). The one-command VM kit and the live site's connection to an endpoint have not run (a Nebius CPU VM did run once, by hand, for the 28 Oct Night School). The commands, the flags and the costs (estimates) are in [Where everything runs](/reference/where-it-runs/#one-command-to-host-it-all-on-nebius), which also says what runs where today.
```bash
make nebius-plan
RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1 make nebius-up YES=1 ARGS="--with-student"
RATCHET_I_UNDERSTAND_THIS_COSTS_MONEY=1 make nebius-down YES=1
```
Think of the whole restaurant as a set of rented rooms.
- **The front room** is a small CPU VM. It holds the pass (the gateway), the recipe book on a shelf (the SQLite ledger), a doorman who checks guests and puts a proper lock on the door (Caddy, with HTTPS), and an alarm clock on the wall (cron) that rings for the nightly backup.
- **The head chef from the agency** works in another building (Token Factory). You phone each order in and pay per dish. The same agency sends the examiner who grades written answers (the judge) and the untrained cook who covers easy orders when the apprentice's station is closed (the hosted fallback).
- **The apprentice's station** (a Serverless Endpoint with one GPU) is rented by the hour. While the lights are on the meter runs, even if no orders come in. So you switch it on for busy times and off afterwards.
- **The night school classroom** (a Serverless Job) is rented only for the lesson, then handed back.
- **The pantry** (an Object Storage bucket) is shared: the classroom puts each new notebook of techniques there, and the station reads it from there. The pantry's rent keeps running even while the station is closed.
- Scout also phones a **librarian** (Tavily) for web searches.
- **Live site, as of 29 Oct 2026:** every model call on the live site runs on Nebius Token Factory. Nemotron 3 Ultra teaches, and a random 40% of eligible requests are answered by the hosted Nemotron 3 Nano 30B, with a quarter of those re-checked by Ultra ([live stats](https://live.ratchet.dualmindlab.tech/ratchet/stats)).
- **27 Oct 2026:** the text-to-SQL adapter was retrained on Nebius AI Cloud, by hand, as one spot L40S Serverless Job: 40.7% on exam 1 and 34.5% on exam 2 (originally 39.0% and 33.0%) ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/text2sql-round2-nebius)).
- **27 to 28 Oct 2026:** the student was served from a Nebius Serverless Endpoint with 16 users at once: median 1.6 s for SQL and 1.2 s for tool calling. The endpoint was deleted afterwards, so no GPU sits idle ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/student-endpoint-nebius)).
- **28 Oct 2026:** Night School ran end to end once, by hand. The golden gate rejected the candidate at 81.8% precision against the 95% target, so it was never made live: the safety check doing its job ([evidence](https://github.com/harshbhx/ratchet/tree/main/evidence/nightschool-nebius)).
- **Ratchet's own adapter on the live site.** The live site is not connected to a student endpoint. The fine-tuned Nano 4B adapter goes live only after it passes the golden gate.
- **The nightly schedule on live traffic.** Night School runs by hand today; the timer is scripted.
- **The one-command host** (`make nebius-plan`, `make nebius-up`) is ready and dry-run by default; it has not yet run as one command. The VM and GPU scripts follow the Nebius docs and CLI reference and are tested offline.
Run every script with `DRY_RUN=1` first.
## How it really works
### The pieces and how they connect
| Piece | Nebius product and size | What runs there | How it bills |
|---|---|---|---|
| Front room | Compute VM, platform `cpu-e2`, preset `2vcpu-8gb`, region `eu-north1` | Docker Compose with Caddy (`caddy:2.11.7-alpine`), the gateway (one uvicorn worker), the SQLite ledger in the `ratchet-data` volume at `/data/ratchet.db`, and cron | While it runs. A stopped VM keeps billing for its disk |
| Teacher, judge, hosted fallback | Token Factory | Nemotron 3 Ultra (`nvidia/Nemotron-3-Ultra-550b-a55b`), Nemotron 3 Super (`nvidia/nemotron-3-super-120b-a12b`), Nemotron 3 Nano 30B (`nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B`) | Per token |
| Student | Serverless Endpoint, platform `gpu-l40s-a`, preset `1gpu-8vcpu-32gb` | vLLM `v0.18.0` serving `nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16`, with LoRA adapters loaded at runtime | While it runs. A stopped endpoint is not billed for compute or its own storage |
| Night School training | Serverless Job, platform `gpu-l40s-a` | `python -m ratchet.nightschool.train` (Transformers, PEFT and TRL), capped at 2 hours | While it runs |
| Pantry | Object Storage bucket, mounted at `/data` in both the job and the endpoint | The training file, adapters, BOMs and `NOTICE.md`; optionally ledger backups | Per GiB-month, also while the endpoint is stopped |
| Scout's web search | Tavily (not a Nebius product) | Scout's searcher, live mode only | Credits; the free plan has 1,000 a month ([Tavily credits](https://docs.tavily.com/documentation/api-credits), checked Oct 20, 2026) |
The gateway does no inference. It routes requests, writes the ledger and serves the web pages, so a 2 vCPU / 8 GB VM is enough. The models run elsewhere: the teacher on Token Factory, the student on the endpoint, or the hosted fallback when the endpoint is off.
```mermaid
flowchart TB
user(["Judges, browsers, your app"]) --> dns["DNS"]
dns --> pages["Cloudflare Pages site and docs"]
dns --> vm
subgraph vm["Nebius CPU VM · 2 vCPU, 8 GB"]
direction TB
caddy["Caddy HTTPS, edge guard"] ==> live["Live gateway SQLite ledger"]
end
subgraph tf["Token Factory · in use"]
teacher["Ultra: teacher Super: judge Nano 30B: fallback"]
end
tavily["Tavily live Scout search"]
subgraph gpu["Nebius AI Cloud · L40S · ran by hand 27 to 28 Oct"]
direction TB
ep["Serverless Endpoint vLLM, only while on"] --- bucket[("Object Storage bucket at /data")]
job["Serverless Job LoRA training"] --> bucket
nightly["Nightly schedule and live connection: not run"]
end
vm ==> tf
tf ~~~ tavily
tavily ~~~ gpu
vm -.-> tavily
vm -.-> gpu
class user,dns,caddy,live neutral
class pages,teacher ok
class tavily outline
class ep,job,bucket ok
class nightly pending
class gpu okZone
class tf okZone
```
**Reading this diagram.** Green is in use today: Cloudflare Pages serves the static site and these docs (the live gateway runs on Cloudflare Workers with Containers; the Nebius VM in this guide ran once by hand, for the 28 Oct Night School; its one-command kit is tested offline), and the teacher runs on Nebius Token Factory, paid per token. The Nebius GPU side is green because each part ran once by hand: the round-2 retraining (27 Oct), the endpoint speed tests (27 to 28 Oct) and the Night School job and endpoint (28 Oct); the original adapters were trained on a borrowed GPU. The one amber box is what has not run: the nightly schedule and the live gateway's connection to an endpoint (the one-command script, `make nebius-up`, is tested offline). Plain boxes are the CPU VM this guide sets up: Caddy in front, the live gateway with its SQLite ledger. The thick arrow is the paid path every live request can take; the dotted ones are optional: Tavily only for live Scout runs, the endpoint only while you run it, and the training job only once the Night School cron is turned on.
In use
Not run
Optional
Set up by this guide
### What lives in the bucket
The bucket is mounted at `/data` in the training job and in the endpoint, laid out like this:
- night/
- train.jsonl
- boms/
- student-vN.json
- adapters/
- student-vN/
- adapter_config.json
- adapter_model.safetensors
- versions.json
- ratchet_manifest.json
- NOTICE.md
- ratchet_bom.json
- `python -m ratchet.live night` writes `night/train.jsonl` locally under `RATCHET_DATA_DIR` and uploads it with `RATCHET_SYNC_CMD`. That command must only upload: `aws s3 sync` with `--delete` would remove the weights the job wrote.
- The training job writes the adapter weights, `versions.json` (Python, CUDA and every pinned package), `ratchet_manifest.json` and a first `NOTICE.md`.
- After the golden eval, `ratchet.live` writes the adapter's signed BOM (`boms/student-vN.json` and `adapters/student-vN/ratchet_bom.json`) and the exact `NOTICE.md` text the BOM signed, then syncs again. A failed second sync writes a `sync_failed` event and exits non-zero; the night still counts.
- vLLM reads `/data/adapters/student-vN` when the gateway asks it to load the adapter.
### The order of work
1. **Install the Nebius CLI and log in**, on macOS, Ubuntu, WSL or the console's Cloud Shell.
2. **Create the VM.** It starts with no keys, so it makes no model call and cannot run up a model bill until you configure it in step 6.
3. **Give it a web address and check the pages.**
4. **Run the day-1 checks** with a Token Factory key (`scripts/day1_check.py`, a few cents).
5. **Create the bucket, the training image, the student endpoint and one training job by hand**, each with `DRY_RUN=1` first.
6. **Configure live mode** with the edge guard and the Scout budget guard on.
7. **Keep it running** until judging ends: GPU only in planned windows, updates, backups, monitoring.
8. **Tear it down**: the VM, its boot disk, the endpoint and the keys.
Each step in detail follows.
### 1. Install the Nebius CLI
```bash
curl -sSL https://artifacts.nebius.cloud/cli/install.sh | bash
exec -l $SHELL && nebius version
export NEBIUS_PROJECT_ID=
nebius profile create --parent-id $NEBIUS_PROJECT_ID # opens a browser login
```
The Nebius CLI runs on Ubuntu and macOS only. On Windows, use WSL (Ubuntu) or Cloud Shell in the Nebius console. The deploy scripts are bash scripts and also need `jq`, so WSL is the easier choice for the whole guide.
You also need:
- **`jq`** on the machine that runs `create_vm.sh` (it reads the public IP with it).
- **An SSH key**, for example `~/.ssh/id_ed25519.pub` (`ssh-keygen -t ed25519`).
- **The repo public** at `https://github.com/harshbhx/ratchet`, because the VM clones it on first boot. For a private repo, `RATCHET_REPO_URL` can carry a read-only token; it stays in the VM's cloud-init user data, so revoke it after first boot.
- **A region.** The default platform `cpu-e2` exists only in `eu-north1`, which also has L40S GPUs. In other regions set `RATCHET_VM_PLATFORM=cpu-d3`.
### 2. Create the VM
```bash
export NEBIUS_PROJECT_ID=
export SSH_PUBLIC_KEY_PATH=~/.ssh/id_ed25519.pub
DRY_RUN=1 bash deploy/vm/create_vm.sh # prints the commands, creates nothing
bash deploy/vm/create_vm.sh
```
What [`create_vm.sh`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/create_vm.sh) does:
1. Refuses to go on if a VM with the name already exists (`ratchet-host` by default).
2. Finds the first subnet in the project (`nebius vpc subnet list`), unless you set `NEBIUS_SUBNET_ID`.
3. Creates a 30 GiB `network_ssd` boot disk from the image family `ubuntu24.04-driverless`. The disk is created on its own, so it survives deleting the VM; you delete it yourself at teardown.
4. Creates the VM with that disk, a **static** public IP (kept while the VM is stopped) and [`cloud-init.yaml`](https://github.com/harshbhx/ratchet/blob/main/deploy/vm/cloud-init.yaml) as user data.
5. Prints the public IP, the VM and disk IDs, the next steps and the two teardown commands.
On first boot, cloud-init installs Docker Engine and the Compose plugin, turns on unattended security upgrades and the `ufw` firewall (ports 22, 80 and 443), creates a `ratchet` service user that cannot log in, clones the repo to `/opt/ratchet`, writes `deploy/vm/.env` from `.env.vm.example` (mode 0600), and runs `docker compose up -d --build`. That takes 5 to 10 minutes:
```bash
ssh ubuntu@ 'sudo tail -f /var/log/ratchet-bootstrap.log'
```
Optional settings for `create_vm.sh`: `RATCHET_VM_NAME`, `RATCHET_VM_PLATFORM`, `RATCHET_VM_PRESET` (for example `4vcpu-16gb`), `RATCHET_VM_DISK_GB`, `RATCHET_SSH_USER` (default `ubuntu`; Nebius reserves `root` and `admin`), `RATCHET_REPO_URL` and `RATCHET_DOMAIN`.
### 3. Give it a web address and check the pages
- **Fastest: sslip.io.** If `RATCHET_DOMAIN` is not set, first boot sets it to `.sslip.io`. That name resolves to the IP inside it, so Caddy gets a real certificate without any DNS work. The VM finds its public IP by asking `api.ipify.org` (verify on day 1).
- **Your own name.** Add a DNS A record pointing at the public IP, wait until `dig +short ` returns it, then on the VM:
```bash
cd /opt/ratchet/deploy/vm
sudo -u ratchet sed -i 's|^RATCHET_DOMAIN=.*|RATCHET_DOMAIN=ratchet.dualmindlab.tech|' .env
sudo -u ratchet docker compose up -d
```
Set the DNS record first: repeated failed certificate requests can hit Let's Encrypt's rate limits.
| URL | Expect |
|---|---|
| `https:///` | The story page |
| `https:///scout` | Scout (needs the keys from step 6) |
| `https:///dashboard` | The dashboard |
| `https:///healthz` | `{"status":"ok",...}`, always public |
| `https:///readyz` | 200 `ready` (or `degraded`, with the reason under `checks`); 503 `not_ready` when the ledger cannot be written or no teacher can answer |
| `https:///v1/models` | The model list; 403 when the Caddy guard is on, 401 when API keys are on and you send none |
### 4. Run the day-1 checks
Live mode spends real money on every Scout run and every chat request. Before it, check the model IDs and features with a real Token Factory key:
```bash
export NEBIUS_API_KEY=... # a Token Factory key from its console
python scripts/day1_check.py # or: make day1
```
The script confirms that the four model IDs exist and that the teacher answers, returns logprobs and supports JSON mode. It also prints each model's `finish_reason` and whether the teacher takes the strict `json_schema` format. If an ID differs, set `RATCHET_TEACHER_MODEL` (and update `ratchet/licenses.py` and `DEFAULT_PRICES`). Model IDs and per-token prices in `ratchet/config.py` were checked this way on 29 Oct 2026 (it passed); the GPU-hour rate is still unverified. Every check, and how to run it, is described at the top of [`scripts/day1_check.py`](https://github.com/harshbhx/ratchet/blob/main/scripts/day1_check.py) (`teacher`, `choices`, `reasoning`, `student`, `tavily`, `cli` or `all`).
The Nebius Builder Program's terms say $25 of Token Factory credit, expiring after 90 days (research notes, Oct 25, 2026; [DEPLOYMENT.md](/reference/deployment/)). Re-check the program terms before relying on it.
### 5. Bucket, training image, student endpoint and training job
Do these from your laptop (or WSL), where the `nebius` and `aws` CLIs are. Stop the endpoint after each check.
1. **Bucket.** Create one in the same region as the GPUs (`eu-north1`), then:
```bash
export RATCHET_BUCKET_ID= # for --volume
export RATCHET_BUCKET_S3=s3:// # for polling in train_job.sh
export RATCHET_S3_ENDPOINT=https://storage.eu-north1.nebius.cloud # verify on day 1: copy it from the console
export RATCHET_DATA_DIR=./data
export RATCHET_SYNC_CMD="aws s3 sync ./data s3:// --endpoint-url $RATCHET_S3_ENDPOINT"
```
2. **Training image.** Build from the repository root and push it to a registry:
```bash
docker build -f deploy/Dockerfile.train -t /ratchet-train:0.1 .
docker push /ratchet-train:0.1
export RATCHET_TRAIN_IMAGE=/ratchet-train:0.1
```
The image is CUDA 12.8 with exact versions of torch, Transformers, PEFT, TRL, Accelerate, datasets and the Mamba kernels, pinned in `deploy/train-versions.txt`. The versions are candidates read on Oct 21, 2026, to be confirmed by the first smoke job's `versions.json`.
3. **Student endpoint.** Create it once, then stop and start the same endpoint (same URL, same token):
```bash
export RATCHET_STUDENT_KEY=$(openssl rand -hex 24)
DRY_RUN=1 bash deploy/student_endpoint.sh create # nebius validates the request, creates nothing
bash deploy/student_endpoint.sh create # once
export RATCHET_STUDENT_BASE=https:///v1
bash deploy/student_endpoint.sh status # prints "", for example RUNNING
bash deploy/student_endpoint.sh stop # when idle
bash deploy/student_endpoint.sh start # the same endpoint again
```
4. **One training job by hand.** `deploy/train_job.sh ` submits `nebius ai job create` with the bucket at `/data`, a 200 GiB disk and a 2-hour timeout. With `RATCHET_BUCKET_S3` set it then waits, polling every 30 seconds for up to 2 hours, until `adapters//adapter_model.safetensors` exists, which is what lets Night School use it as `RATCHET_TRAIN_CMD`:
```bash
DRY_RUN=1 bash deploy/train_job.sh test # nebius validates the request, submits nothing
bash deploy/train_job.sh test # the day-1 smoke job, on about 200 rows
```
5. **Smoke test the whole path.** With the day-1 checks passed and the endpoint up, `make live-smoke` sends 20 requests through a live gateway with about 5 control calls to Ultra and no nights (cents).
What `student_endpoint.sh create` sets: the image `vllm/vllm-openai:v0.18.0-cu130` (`RATCHET_VLLM_IMAGE`), container port 8000, `--shm-size 16Gi`, the bucket at `/data`, token auth with `RATCHET_STUDENT_KEY`, `VLLM_ALLOW_RUNTIME_LORA_UPDATING=True`, and vLLM arguments from the model card: `--enable-lora --max-lora-rank 16 --max-loras 4 --mamba_ssm_cache_dtype float32 --max-model-len 8192 --max-num-seqs 64`, with thinking off by default. Training defaults (`train.py`): 2 epochs, learning rate 2e-4, LoRA rank 16 (alpha 32), max sequence 2048, batch 8, bf16, loss on the assistant's tokens only.
Adapters loaded at runtime are lost when the endpoint stops. The gateway notices vLLM's 404 for an adapter and loads it again by itself, at most once per adapter per minute (`adapter_reloaded` events), and `/readyz` queues a reload for any routed adapter the endpoint no longer lists. Until then those student calls fall back to the teacher: they still succeed, at teacher cost. `python -m ratchet.live reload` reloads every live, canary and shadow adapter by hand.
### 6. Configure live mode
The VM's settings live in `/opt/ratchet/deploy/vm/.env` (mode 0600, owned by `ratchet`, never in git). Keep comments on their own lines: Docker Compose and the scripts read this file.
```bash
RATCHET_MODE=live
NEBIUS_API_KEY=
TAVILY_API_KEY=
# Empty: the hosted fallback answers student calls. Set both only while the endpoint runs.
RATCHET_STUDENT_BASE=
RATCHET_STUDENT_KEY=
# Caddy blocks /v1/*, /ratchet/admin/* without this token (hex, 32+ characters).
RATCHET_API_GUARD=guarded
RATCHET_ADMIN_TOKEN=