The big picture

In one sentence

Ratchet is one small gateway process with one SQLite ledger, plus a few Nebius services it calls: Token Factory for the teacher, a GPU endpoint for the student, a nightly training job and a storage bucket that carries data and adapters between them.

In plain words

Picture the whole restaurant from above.

• The pass (the gateway) is where every order arrives and every plate leaves. It decides who cooks each order. It never cooks anything itself, so it needs only a small room.
• The apprentice’s station (the student endpoint) is a rented kitchen with a stove, the GPU. The apprentice cooks there, reading from the notebook of techniques (the LoRA adapter you own).
• The agency (Nebius Token Factory) sends the head chef (Nemotron 3 Ultra) for each hard dish and charges per plate. It also has a food critic (the judge, Nemotron 3 Super) and a spare, untrained cook (the hosted fallback, Nemotron 3 Nano 30B) for when the apprentice’s station is closed to save money.
• The recipe book (the ledger) sits at the pass. Every order, every dish the head chef cooks and every exam result is written in it.
• The storeroom (the Object Storage bucket) holds the night’s study pack and every notebook the apprentice ever wrote.
• Night school (a Serverless Job on a GPU) opens once a night, reads the study pack from the storeroom and writes a new notebook back into it.
• The doorman (Caddy on the demo machine) checks the front door and keeps the connection to guests private (HTTPS).

Three clocks run the place: one ticks for every order, one every few minutes to check how the newest notebook is doing, and one once a night for school.

One processthe gateway, with no GPUIt routes, scores and logs. A 2 vCPU, 8 GB VM is enough, because it never runs a model.

One filethe SQLite ledgerRequest metadata, stored teacher answers with their licenses, versions and the release log.

Three loopsat three speedsOne for every request, one every few seconds to minutes, and one once a night.

How it really works

flowchart TB
  app(["Your app · any OpenAI SDK"]) ==>|"POST /v1/chat/completions"| caddy["Caddy on the demo VM<br/>HTTPS, edge guard"]
  caddy ==> gw
  subgraph gw["Gateway · one process, no GPU"]
    direction LR
    router["Cascade<br/>Ratchet.handle"] --> ledger[("SQLite ledger")]
    ledger <--> gate{{"Release gate"}}
  end
  subgraph tf["Token Factory, per token"]
    direction TB
    teacher["Nemotron 3 Ultra<br/>teacher"]
    teacher ~~~ others["Judge: Super<br/>Fallback: Nano<br/>30B, untuned"]
  end
  subgraph ep["Serverless Endpoint"]
    student["Nano 4B + LoRA<br/>on vLLM"]
  end
  subgraph ns["Night School"]
    direction TB
    night["ratchet.live night<br/>cron, on the host"] --> job["Serverless Job<br/>LoRA training"]
  end
  bucket[("Object Storage<br/>bucket at /data")]
  gw ==>|"escalate, audit"| tf
  gw ==>|"3 samples"| ep
  gw -->|"rows"| ns
  ns -->|"adapter"| bucket
  ep ---|"loads adapters"| bucket

  class app,router,night,job neutral
  class caddy,others outline
  class gate gate
  class ledger,bucket store
  class student student
  class teacher teacher
  class ep studentZone
  class tf teacherZone

Reading this diagram. Start at the top. The thick arrows are the request path: your app’s call passes Caddy (only on the demo VM), then the cascade asks the student endpoint for 3 samples, and the teacher on Token Factory when the student is unsure, an audit is drawn or no student may answer (a cold start, a pinned task). Every request writes one row to the SQLite ledger; the release gate reads the audits there and writes version states back. At night, Night School reads the rows, a Serverless Job trains a new adapter into the Object Storage bucket, and the endpoint loads adapters from that bucket at /data. Not drawn, to keep it readable: the night also loads the candidate into vLLM, grades it there (the judge scores free-text rows) and hands it to the release gate. The hosted fallback answers student calls only when no endpoint is set.

• The student you own
• Rented per token
• Keeps data
• Decides releases
• Optional or fallback
• Request path

The components

1. Your app

   any OpenAI SDK

   Sends a chat request with a task name.

2. Gateway and cascade

   one process

   Admission checks, then the student first.

3. Student

   Nano 4B + LoRA

   Answers when it is confident.

4. Teacher

   Ultra on Token Factory

   Answers the rest.

Component	What it does	Where it runs	Code
Your app	Sends OpenAI-style chat requests, with a task name	Anywhere	Any OpenAI SDK
Gateway	OpenAI-compatible API, admission checks, pages, Scout endpoints, the release gate’s tick	One process, one uvicorn worker; in the demo, a Nebius CPU VM (2vcpu-8gb)	ratchet/gateway.py (Runtime)
Cascade (router)	Student first, confidence, teacher on escalation, audits, shadow, cost	Inside the gateway	ratchet/router.py (Ratchet.handle)
Student Nano · yours	nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16 with the LoRA adapter you own, served by vLLM with runtime LoRA loading	A Nebius Serverless Endpoint (L40S by default)	deploy/student_endpoint.sh
Teacher Ultra · Token Factory	nvidia/Nemotron-3-Ultra-550b-a55b answers escalations, cold starts, pinned tasks, audits and arbitrations	Nebius Token Factory	RATCHET_TEACHER_MODEL
Judge Super · judge	nvidia/nemotron-3-super-120b-a12b scores free-text golden rows in JSON mode (a diagnostic, judge_pass_rate)	Nebius Token Factory	RATCHET_JUDGE_MODEL
Hosted fallback Nano 30B · hosted	nvidia/NVIDIA-Nemotron-3-Nano-30B-A3B, untuned, plays the student when no endpoint is configured	Nebius Token Factory	RATCHET_FALLBACK_STUDENT
Ledger	Requests, examples with provenance, versions, events, failures, receipts	One SQLite file on the gateway’s local disk	ratchet/ledger.py
Night School	Builds the train and golden sets, trains, evaluates, signs a bill of materials, calls the gate	Driven by cron on the gateway host; training on a Nebius Serverless Job (gpu-l40s-a, 1gpu-8vcpu-32gb) with TRL and PEFT	ratchet/live.py, ratchet/nightschool/
Bucket	Training rows, adapters, bills of materials	Nebius Object Storage, mounted at /data in the job and the endpoint	RATCHET_BUCKET_ID, RATCHET_SYNC_CMD
Release gate	Golden gate, shadow, canary 10%, 50%, 100%, rollback	Inside the gateway (and the cron tick)	ratchet/release.py (ReleaseGate)
Web app	/ (the story), /scout, /dashboard; no-build pages at /lite*	Served by the gateway from ratchet/web/dist	web/
Scout	A multi-agent research demo whose calls go through the cascade	Inside the gateway; Tavily for web search in live mode	ratchet/scout/
Caddy	Automatic HTTPS, security headers, an optional token guard on /v1/*, /ratchet/demo/* and /ratchet/admin/*	Next to the gateway on the demo VM (Docker Compose)	deploy/vm/Caddyfile

Token Factory does not offer fine-tuning or LoRA serving for Nemotron yet (README, Oct 2026), so the student is trained and served on Nebius AI Cloud. The gateway itself does no inference, which is why a 2-vCPU machine is enough.

How the pieces connect

1. Your app calls the gateway at POST /v1/chat/completions with an API key and a task name (X-Ratchet-Task). The gateway checks the body size, the key, rate limits, the request itself, the task and the quotas before any model is called.
2. The cascade calls the student endpoint over HTTPS with the OpenAI API. The model name it sends is the version’s adapter name (student-vN), because vLLM serves each LoRA adapter under its own name. Three samples come back in one call.
3. When the student is unsure, or for an audit, the cascade calls the teacher on Token Factory. Each teacher answer that is complete and valid goes into the ledger as a training example, with the teacher’s model id and license, after PII redaction.
4. Every request writes one row to the ledger: who served it, the route, confidence, cost and the all-teacher baseline. No prompt text is kept in that row; text is kept only in training examples, redacted.
5. After requests, the release gate ticks: it reads the audits and shadow verdicts from the ledger and moves versions between states (shadow, canary 10%, 50%, live, rolled back).
6. At night, python -m ratchet.live night runs Night School: it writes night/train.jsonl locally, uploads it to the bucket with RATCHET_SYNC_CMD, starts the training job with RATCHET_TRAIN_CMD (default bash deploy/train_job.sh {version}) and waits.
7. The training job reads the bucket at /data and writes the adapter to /data/adapters/student-vN/, with NOTICE.md and its training metadata beside the weights.
8. The gateway host tells vLLM to load the new adapter with POST /v1/load_lora_adapter, so no endpoint restart is needed. Night School then scores it on the golden set (the judge scores free-text rows), signs its bill of materials and hands the candidate to the gate. Last, the night writes the bill of materials and NOTICE.md into the adapter’s folder and syncs the bucket again.
9. The pages read the same ledger through GET /ratchet/stats, GET /ratchet/cost and the other dashboard endpoints. Scout calls Ratchet.handle in the same process, once for each of its model calls.

The GPU steps have not run on Nebius yet

As of Oct 22, 2026, the teacher runs on Nebius Token Factory, but steps 2 and 6 to 8 (the student endpoint and the training job) have run only against mock models and fake HTTP transports in the tests: no student has been served from a Nebius GPU and no adapter trained in a Nebius job. The nebius ai commands in deploy/ follow the CLI reference but have not run against a real account, and live mode stays unverified until scripts/day1_check.py passes with a real key. See the deployment guide and Host on Nebius.

What a request leaves behind

Every request leaves exactly one row of metadata. Text is kept only when a teacher answer passes every check below, and then only after PII redaction.

flowchart TB
  req(["A chat request"]) --> ans{{"A teacher answer?"}}
  req --> row[("Request row, always<br/>metadata only")]
  ans -->|"no: the student served"| x1["Nothing to store"]
  ans -->|yes| only{{"Pinned, tool or<br/>teacher-only?"}}
  only -->|yes| x2["Never stored"]
  only -->|no| opt{{"A storage opt-out?"}}
  opt -->|yes| x3["Not stored<br/>the row says which"]
  opt -->|no| valid{{"Complete and valid?"}}
  valid -->|no| x4["Served, not stored<br/>skip_example event"]
  valid -->|yes| red["PII redaction"]
  red --> ex[("Training example<br/>teacher, license, split")]

  class req,red neutral
  class row,ex store
  class ans,only,opt,valid gate
  class x1,x2,x3,x4 outline

Reading this diagram. The short branch to the cylinder always happens: one request row with the route, cost, confidence and ids, never the prompt or the answer. The API key appears in it by name only, and an end user only as a keyed hash. The chain of checks down the middle decides whether any text is kept; each dashed box is a way out with nothing stored. A teacher answer exists after an escalation, a cold start, an audit or a shadow arbitration; the student’s own answers are never stored. Pinned tasks, tool requests and other teacher-only requests are never stored (the one exception is an over-length request on a task that sets store_ineligible). Any storage opt-out stops it next: store: false, X-Ratchet-Store: none, metadata.ratchet_store, a key with store=0, a no-store task, a no-train end user or zero retention. Then the answer must be complete and valid under the task’s spec. Only then are the messages, the stored conditions and the answer redacted and written as a training example with the teacher’s model id and license, in the golden or the train split. An erased example’s hash is refused from then on.

Mock mode and live mode

RATCHET_MODE picks one of two runtimes, built in Runtime.reset in ratchet/gateway.py:

flowchart TB
  mode{{"RATCHET_MODE"}}
  mode -->|"mock, the default"| m1[("In-memory ledger<br/>gone on restart")]
  m1 --> m2["Mock teacher and students<br/>from sim/, no network"]
  m2 --> m3["Demo controls on<br/>simulated search"]
  mode -->|live| l1[("SQLite file<br/>at RATCHET_DB")]
  l1 --> l2["Teacher on<br/>Token Factory"]
  l2 --> sb{{"RATCHET_STUDENT_BASE<br/>set?"}}
  sb -->|yes| l3["Your adapter on<br/>your vLLM endpoint"]
  sb -->|no| l4["Hosted fallback<br/>Nano 30B, untuned"]

  class mode,sb gate
  class m1,l1 store
  class m2,m3 neutral
  class l2 teacher
  class l3 student
  class l4 outline

Reading this diagram. One setting picks the runtime when the gateway starts (Runtime.reset). The left branch needs no network and no keys, and every number it shows is simulation. The right branch keeps a real ledger file and calls the teacher on Token Factory. Its student is your adapter on your own vLLM endpoint when RATCHET_STUDENT_BASE is set, and otherwise the untuned Nano 30B on Token Factory, drawn dashed because it is a stand-in, not the adapter you own.

	Mock mode	Live mode
Ledger	:memory:, gone on restart	SQLite file at RATCHET_DB, day 1 starts at the first request
Teacher and student	Mock models from sim/ (deterministic, no network)	Token Factory teacher; vLLM endpoint, or the hosted fallback when RATCHET_STUDENT_BASE is empty
Night School	MockFleet trains and scores in memory; a simulated day runs from the dashboard (POST /ratchet/demo/day) or at startup (RATCHET_MOCK_WARMUP_DAYS)	python -m ratchet.live night from cron, with a real training job
Audit rate	At least 0.10	RATCHET_AUDIT_RATE, default 0.05
Canary audits needed per stage	15 (unless RATCHET_MIN_AUDITS is set)	20 (ReleasePolicy.min_audits)
Gate tick from requests	After every request	At most once every 5 s (RATCHET_TICK_INTERVAL_S)
Audits and shadow	Run before the response (0 sidecar workers)	Run after the response on 4 sidecar workers (RATCHET_SIDECAR_WORKERS)
API keys	Optional	Required (RATCHET_API_KEYS), unless RATCHET_ALLOW_UNAUTHENTICATED=1 with a guard in front
Scout search	Simulated results	Tavily when TAVILY_API_KEY is set
Numbers it shows	Simulation	Live traffic

With no RATCHET_STUDENT_BASE, a trained version’s student calls go to the untuned hosted Nano 30B, held to the global threshold RATCHET_CONFIDENCE_THRESHOLD (0.80) instead of the version’s calibrated one. Those answers are still reported as served_by: student; the response’s model field names the model that wrote them, and the pages label them “Nano 30B · hosted”, not as your adapter.

Three loops at three speeds

flowchart TB
  R["Every request<br/>the cascade"] ==>|"rows, audits, examples"| L[("Ledger")]
  L ==>|"audits, shadow verdicts"| G{{"Seconds to minutes<br/>the gate tick"}}
  G ==>|"version states"| R
  L -->|"train and golden rows"| N["Every night<br/>Night School"]
  N -->|"a new candidate"| G

  class R,N neutral
  class L store
  class G gate

Reading this diagram. The thick loop turns fastest: requests write rows, the gate tick reads the audits and shadow verdicts in them, and the version states it writes decide which student the next request meets. The thin path turns once a night: Night School reads the training and golden rows from the same ledger and hands a new candidate to the gate. In live mode cron can also run the tick, so a quiet gateway still moves its releases.

Loop	How often	What it does	Entry point
The request	Every call; seconds at most	Pick a version, sample the student, serve or escalate, maybe audit, maybe shadow, log one row	Ratchet.handle
The gate tick	After requests (live: at most every 5 s) and from cron every 5 minutes	Shadow decisions, canary 10% to 50% to live, rollback, alerts	ReleaseGate.tick, python -m ratchet.live tick
The night	Once a night (02:30 on the demo VM’s cron line)	Skip if a release is in flight, training is paused or fewer than 50 new examples arrived; otherwise build, train, evaluate, gate	run_night, python -m ratchet.live night

Two more jobs run on the side: python -m ratchet.live seal seals the savings receipts of each ledger day that ended (hourly), and deploy/vm/backup.sh backs up the ledger (daily at 03:10 UTC). On the demo VM only the backup is switched on by default; the night, tick and seal lines in /etc/cron.d/ratchet are commented out until live mode works by hand.

The loops share only the ledger. The gate reads what requests wrote, and Night School does not train while a version is in shadow or canary, so a release finishes or fails before the next candidate is trained.

For engineers

Runtime wires it all. ratchet/gateway.py Runtime.__init__ checks the whole configuration first (config.check, every error in one ConfigError), then Runtime.reset builds the ledger, the teacher provider, the student resolver (in live mode: the endpoint and v.adapter_uri, or the hosted fallback model), the Ratchet cascade and the ReleaseGate. A mock reset (POST /ratchet/demo/reset) rebuilds everything.

Settings that shape the picture (defaults from ratchet/config.py; all of them in the configuration reference):

Variable	Default	Meaning
RATCHET_MODE	mock	mock or live
RATCHET_DB	ratchet.db	Ledger file (live mode)
RATCHET_TEACHER_BASE	https://api.tokenfactory.nebius.com/v1	Token Factory, for the teacher, judge and hosted fallback
RATCHET_TEACHER_MODEL	nvidia/Nemotron-3-Ultra-550b-a55b	Must be licensed for distillation, or startup fails
RATCHET_STUDENT_BASE	empty	Your vLLM endpoint; empty means the hosted fallback
RATCHET_STUDENT_BASE_MODEL	nvidia/NVIDIA-Nemotron-3-Nano-4B-BF16	The base every adapter is trained on
RATCHET_DATA_DIR	data	Local folder laid out like the bucket
RATCHET_TRAIN_CMD	bash deploy/train_job.sh {version}	Starts training and waits for it
RATCHET_ADAPTER_ROOT	/data/adapters	Where vLLM finds adapters; also used to reload one vLLM forgot

The bucket’s layout, as the training job and the endpoint see it at /data:

• Directorydata

  • Directorynight

    • train.jsonl the rows Night School built tonight
  • Directoryadapters

    • Directorystudent-v3

      • adapter_model.safetensors written by the training job
      • ratchet_manifest.json base model, licenses, template arguments
      • NOTICE.md base model and teacher licenses
      • ratchet_bom.json the signed bill of materials
  • Directoryboms

    • student-v3.json a copy of the same bill of materials

The ledger is not in the bucket. With the default WAL journal it must stay on the gateway host’s local disk, because WAL needs shared memory between the processes that open the file (the gateway and the cron jobs); RATCHET_DB_SYNC=FULL is the setting for a network mount (architecture, section 11). On the demo VM it lives in a Docker volume.

Commands:

make demo                                    # mock gateway with a 4-day warm-up on :8000 (simulation)
bash deploy/student_endpoint.sh create       # once; DRY_RUN=1 first. Later: start, stop, status
python -m ratchet.live night                 # Night School (cron)
python -m ratchet.live tick                  # the release gate (cron, every 5 minutes)
python -m ratchet.live seal                  # savings receipts of days that ended (cron, hourly)
python -m ratchet.config check               # the startup config check: exit 0, 1 or 2
DRY_RUN=1 bash deploy/vm/create_vm.sh        # the demo VM, commands printed only

Endpoints the pages use: GET /ratchet/stats, GET /ratchet/cost, GET /ratchet/tasks, the Scout endpoints under /scout/*, and, in mock mode only, /ratchet/demo/*. See the API reference.

Common questions

Why does the gateway not need a GPU?

It never runs a model. It sends the student’s samples to the vLLM endpoint and the teacher’s calls to Token Factory, and does the rest (scoring, logging, the gate) in plain Python. The demo VM is 2 vCPU and 8 GB.

Why one SQLite file and one process?

At demo scale (thousands to tens of thousands of requests a day) one file needs no setup, is easy to back up and inspect, and is fast enough. The price is one gateway process: run one uvicorn worker. A Postgres ledger for several workers is planned.

What happens when the GPU endpoint is stopped?

If RATCHET_STUDENT_BASE still points at it, student calls fail, and after 5 failures in a row the student’s circuit breaker sends requests straight to the teacher (route student_breaker) until a probe succeeds. To keep a student answering while the endpoint is off, leave RATCHET_STUDENT_BASE empty: student calls then go to the hosted, untuned Nano 30B on Token Factory, held to the 0.80 default threshold and not covered by the adapter’s precision certificate. Nebius’s docs say a stopped endpoint is not billed for compute or its own storage; a mounted bucket still is.

Why not train and serve the student on Token Factory?

Token Factory does not offer fine-tuning or LoRA serving for Nemotron yet (README, Oct 2026). So training runs as a Serverless Job and serving on a Serverless Endpoint, both on Nebius AI Cloud.

What does running it cost?

Estimates only. The demo VM: about $72.50 for Oct 29 to Dec 15, 2026, before live usage (prices as of Oct 20, 2026, demo VM guide). The student endpoint: about $32 per full day if left on, and a training run about $0.70 to $1.35 a night, at about $1.35 per L40S GPU-hour (Nebius pricing, read Oct 22, 2026; deployment guide). See Cost and savings.

Has any of this run on Nebius?

The teacher has: it runs on Nebius Token Factory. The GPU side has not, as of Oct 22, 2026: no student has been served from a Nebius endpoint and no adapter trained in a Nebius job (the text-to-SQL adapter was trained on a free Colab T4). Those paths are covered by offline tests with fake transports, and the first real run follows the day-1 checks.

Related

Every request

One request's journey

The request loop, step by step, from admission to the ledger row.

Every night

Night School

The nightly loop that trains a fresh adapter you own.

Every few minutes

The release gate

The tick loop: shadow, canary 10%, 50%, live, and rollback.

Deploy

Host it on Nebius

Putting the pieces on real infrastructure, with costs.