# Reasoning adaptation and verified sequence distillation
The recipes call established trainers rather than implement an optimizer. The
initial path is local teacher generation → final-answer filtering → SFT, followed
by an optional GRPO experiment on tasks with trusted reference answers. These
are runnable baselines, not evidence of improved model quality. The four bundled
training problems are synthetic handwritten fixtures, separate from evaluation
examples, and far too small to train a useful reasoning model.
For a completed GPU example using real data, follow the
[German Qwen SFT recipe](use-cases/german-reasoning.md) and its
[results](use-cases/german-reasoning-results.md). The next paths are
[native multi-turn agent RL](use-cases/german-agent-rl.md),
[existing-harness integration](agent-harness.md), and optional
[RL agent gyms](agent-gyms.md); each has separate qualification boundaries.
## Checkout and execution location
On our Determined workers, run these commands from
`/pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes` in workspace `JAIF`. See
[operations](operations.md) for allocation and mount instructions. Other teams
can use any checkout directory; the commands below assume its repository root.
CPU planning and fixture checks need no GPU. Actual Qwen training and evaluation
start only when their launch command runs on an allocated compatible GPU.
Recreate virtual environments and reinstall the editable package after moving a
checkout: scripts, interpreter links and editable installs may retain absolute
paths. Existing frozen run plans also bind absolute paths and source identities;
prepare a new plan for the new location instead of rewriting historical evidence.
Keep account credentials outside the checkout and out of model/run artifacts.
## Install and inspect
Use an isolated Python 3.11+ environment. The training extra intentionally pins
TRL 0.29.0, Transformers 5.3.0, Datasets 4.6.1, Accelerate 1.12.0, PEFT 0.18.1
and PyTorch 2.10.0. This is an API compatibility baseline, not a claim to track
the newest release. Choose the PyTorch wheel matching your actual hardware;
`torch==2.10.0+cpu` is accepted for CPU integration checks. These pins satisfy the
packages' declared dependency bounds, and the CPU smoke test exercises the real
trainer APIs. The observed full CPU environment, including development tools, is
captured in `requirements/training-cpu.txt`; use that snapshot to recreate the
validated CPU setup. It is not a GPU environment or a package-hash lock.
```bash
python -m venv .venv-training
.venv-training/bin/python -m pip install -e '.[training]'
.venv-training/bin/python -m hessian_recipes.training recipes/reasoning-sft.json --dry-run
.venv-training/bin/python -m hessian_recipes.training recipes/reasoning-grpo.json --dry-run
```
Planning uses the standard library, validates every row and training option, and
does not load a model, create output directories, or contact a service. All paths
inside a JSON recipe resolve relative to that recipe file. A Hub model requires
its immutable 40-character commit revision. The supplied revision identifies a
small model for wiring checks, not a strong teacher or a recommended production
reasoning model. Downloading is disabled unless you deliberately set
`model.allow_download` to `true`; otherwise only cached files are loaded.
Local checkpoint directories can be used with `{"name": "../models/checkpoint"}`.
Check model/data licenses before choosing a different teacher or student.
## Distillation without a hosted teacher
An offline fixture run needs no training dependencies:
```bash
python -m hessian_recipes.distillation prepare \
examples/training/teacher_sequences.jsonl runs/distilled.jsonl
```
This keeps four correct rows and rejects one intentionally incorrect row. The
output is directly suitable for SFT: copy the SFT recipe and set `dataset` to
the filtered output, using a path relative to the copied recipe. The supplied
SFT recipe points to a separate four-row curated fixture so that its dry run
works before generation or filtering.
For actual teacher generation, prepare JSONL rows with `prompt`, `solution` and
an optional `id`, then edit `recipes/teacher-generation.json` to identify your
teacher and output. Use a trusted local model or explicitly permit a pinned
download, then run:
```bash
python -m hessian_recipes.distillation generate recipes/teacher-generation.json --dry-run
python -m hessian_recipes.distillation generate recipes/teacher-generation.json
python -m hessian_recipes.distillation prepare runs/teacher-sequences.jsonl runs/distilled.jsonl
```
Generation uses Transformers locally, a tokenizer chat template, greedy decoding,
float32 and a bounded token budget. There is no external inference API. The
default device is CPU; `device: "cuda"` explicitly selects a GPU. Greedy decoding
does not guarantee identical output across hardware or library versions. Empty
or incorrect generations are rejected by filtering. Inputs must have trusted
references; a large teacher's output is not ground truth.
The verifier accepts either an exact whole response or one final
`value` block. It strips surrounding whitespace only. It rejects
multiple or malformed answer blocks, trailing text, and approximate or numerical
equivalence guesses. It never executes model code. It verifies the final answer
only; an accepted explanation can still be wrong. Domain experts must design
stronger task-specific verifiers before applying this to science or robotics.
Duplicate prompts are kept once, conflicting references fail before writing,
and existing outputs are never overwritten. Filtering reports counts and writes
an adjacent `.manifest.json` with source/output digests and verifier identity.
This is sequence distillation with rejection filtering, not logit matching,
online distillation, or a reproduction of an entire reasoning system. See
[TRL's versioned SFT data formats and completion loss](https://huggingface.co/docs/trl/v0.29.0/en/sft_trainer)
for the upstream dataset and masking contract.
## Run SFT or GRPO
After inspecting a recipe and provisioning its model:
```bash
python -m hessian_recipes.training recipes/reasoning-sft.json
python -m hessian_recipes.training recipes/reasoning-grpo.json
```
Training wraps string prompts as user messages and SFT completions as assistant
messages, using the checkpoint's own chat template. SFT uses completion-only
loss. Both recipes can train LoRA adapters through PEFT; setting `lora` to `null`
trains the full model. The saved `checkpoint/` contains an adapter when LoRA is
enabled; retain the base model identity and revision in the manifest when
loading it later. Replace the tiny defaults with measured token budgets and a
training corpus justified by your task.
For Qwen3, set the same explicit top-level mode in training and teacher recipes:
`"chat_template_kwargs": {"enable_thinking": true}`. Only this boolean switch is
accepted. TRL SFT receives it on every dataset row for both prompt and complete
conversation formatting; GRPO receives it through its native configuration.
Teacher generation forwards it when rendering the prompt. Keep the same mode
and tokenizer template during evaluation. With thinking enabled, reasoning SFT
targets should contain `German rationale` followed by the final
`…` block: a final-only target can teach the model to skip
thinking because Qwen's template inserts an empty thinking block. With thinking
disabled, targets containing nonempty or malformed thinking blocks are rejected
before training, avoiding a mismatch between the prompt prefix and complete
conversation. Empty thinking blocks and ordinary final responses are accepted.
The default local teacher uses greedy generation. Qwen recommends sampling for
its thinking mode; use the explicitly configured evaluation generator for
reasoning experiments rather than treating this simple teacher generator as a
Qwen reasoning-quality baseline. See the
[Qwen3 mode and decoding guidance](https://huggingface.co/Qwen/Qwen3-1.7B).
GRPO samples multiple completions and passes the dataset's `solution` column to
the same strict verifier. Its generation batch must be divisible by
`num_generations`. The adapter sets `steps_per_generation` to
`gradient_accumulation_steps`, and validates the effective batch using
`WORLD_SIZE`. For example, batch 2 with two generations works on one process.
`beta=0` uses no KL reference penalty. All-zero or identical rewards produce no
useful advantage signal; inspect reward spread, format failures and held-out
evaluation before increasing compute. The recipe's `loss_type` is explicit
because upstream defaults can change. Consult the
[versioned GRPO trainer and custom reward interface](https://huggingface.co/docs/trl/v0.29.0/en/grpo_trainer)
for the algorithm and runtime contract.
Real training requires the exact top-level dependency pins. Runs refuse a
nonempty output directory. They save a local checkpoint and
`recipe-manifest.json` containing the complete resolved plan, configuration and
data SHA-256, model revision, dependency versions, seed, status and train metrics.
Local model files are hashed at run start. A failed run records its failure
type. The adapters set `report_to="none"`, `push_to_hub=false`,
`trust_remote_code=false`, and use no vLLM server. No upload happens automatically.
The manifest helps reproduce inputs; a seed alone does not ensure deterministic
GPU training. A scheduler/cluster container still needs a real GPU validation
run and measurements of memory, throughput and checkpoint recovery.
### Merge a local LoRA adapter for evaluation
The evaluation adapter accepts standalone model checkpoints. Merge a trained
LoRA adapter into the exact local base used for training:
```bash
python -m hessian_recipes.checkpoints /absolute/path/to/base runs/reasoning-sft/checkpoint \
runs/reasoning-sft-merged --dtype bfloat16 --device cuda
```
`merge_adapter(base_dir, adapter_dir, output_dir, dtype="bfloat16", device="cpu")`
also exposes this operation in Python. CUDA is an explicit request and fails if
unavailable. CPU merging requires enough RAM for the model and temporary merge
buffers. The helper explicitly loads the supplied base, uses PEFT's
`merge_and_unload(safe_merge=True)`, saves full safetensors and the base tokenizer,
and never follows the adapter's stored base-model pointer or downloads files.
It hashes source directories before loading and verifies them after loading and
merging. The destination must not exist and must be separate from both sources.
An adjacent `