Reasoning adaptation and verified sequence distillation

The recipes call established trainers rather than implement an optimizer. The initial path is local teacher generation → final-answer filtering → SFT, followed by an optional GRPO experiment on tasks with trusted reference answers. These are runnable baselines, not evidence of improved model quality. The four bundled training problems are synthetic handwritten fixtures, separate from evaluation examples, and far too small to train a useful reasoning model.

For a completed GPU example using real data, follow the German Qwen SFT recipe and its results. The next paths are native multi-turn agent RL, existing-harness integration, and optional RL agent gyms; each has separate qualification boundaries.

Checkout and execution location

On our Determined workers, run these commands from /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes in workspace JAIF. See operations for allocation and mount instructions. Other teams can use any checkout directory; the commands below assume its repository root. CPU planning and fixture checks need no GPU. Actual Qwen training and evaluation start only when their launch command runs on an allocated compatible GPU.

Recreate virtual environments and reinstall the editable package after moving a checkout: scripts, interpreter links and editable installs may retain absolute paths. Existing frozen run plans also bind absolute paths and source identities; prepare a new plan for the new location instead of rewriting historical evidence. Keep account credentials outside the checkout and out of model/run artifacts.

Install and inspect

Use an isolated Python 3.11+ environment. The training extra intentionally pins TRL 0.29.0, Transformers 5.3.0, Datasets 4.6.1, Accelerate 1.12.0, PEFT 0.18.1 and PyTorch 2.10.0. This is an API compatibility baseline, not a claim to track the newest release. Choose the PyTorch wheel matching your actual hardware; torch==2.10.0+cpu is accepted for CPU integration checks. These pins satisfy the packages' declared dependency bounds, and the CPU smoke test exercises the real trainer APIs. The observed full CPU environment, including development tools, is captured in requirements/training-cpu.txt; use that snapshot to recreate the validated CPU setup. It is not a GPU environment or a package-hash lock.

python -m venv .venv-training
.venv-training/bin/python -m pip install -e '.[training]'
.venv-training/bin/python -m hessian_recipes.training recipes/reasoning-sft.json --dry-run
.venv-training/bin/python -m hessian_recipes.training recipes/reasoning-grpo.json --dry-run

Planning uses the standard library, validates every row and training option, and does not load a model, create output directories, or contact a service. All paths inside a JSON recipe resolve relative to that recipe file. A Hub model requires its immutable 40-character commit revision. The supplied revision identifies a small model for wiring checks, not a strong teacher or a recommended production reasoning model. Downloading is disabled unless you deliberately set model.allow_download to true; otherwise only cached files are loaded. Local checkpoint directories can be used with {"name": "../models/checkpoint"}. Check model/data licenses before choosing a different teacher or student.

Distillation without a hosted teacher

An offline fixture run needs no training dependencies:

python -m hessian_recipes.distillation prepare \
  examples/training/teacher_sequences.jsonl runs/distilled.jsonl

This keeps four correct rows and rejects one intentionally incorrect row. The output is directly suitable for SFT: copy the SFT recipe and set dataset to the filtered output, using a path relative to the copied recipe. The supplied SFT recipe points to a separate four-row curated fixture so that its dry run works before generation or filtering.

For actual teacher generation, prepare JSONL rows with prompt, solution and an optional id, then edit recipes/teacher-generation.json to identify your teacher and output. Use a trusted local model or explicitly permit a pinned download, then run:

python -m hessian_recipes.distillation generate recipes/teacher-generation.json --dry-run
python -m hessian_recipes.distillation generate recipes/teacher-generation.json
python -m hessian_recipes.distillation prepare runs/teacher-sequences.jsonl runs/distilled.jsonl

Generation uses Transformers locally, a tokenizer chat template, greedy decoding, float32 and a bounded token budget. There is no external inference API. The default device is CPU; device: "cuda" explicitly selects a GPU. Greedy decoding does not guarantee identical output across hardware or library versions. Empty or incorrect generations are rejected by filtering. Inputs must have trusted references; a large teacher's output is not ground truth.

The verifier accepts either an exact whole response or one final <answer>value</answer> block. It strips surrounding whitespace only. It rejects multiple or malformed answer blocks, trailing text, and approximate or numerical equivalence guesses. It never executes model code. It verifies the final answer only; an accepted explanation can still be wrong. Domain experts must design stronger task-specific verifiers before applying this to science or robotics. Duplicate prompts are kept once, conflicting references fail before writing, and existing outputs are never overwritten. Filtering reports counts and writes an adjacent .manifest.json with source/output digests and verifier identity.

This is sequence distillation with rejection filtering, not logit matching, online distillation, or a reproduction of an entire reasoning system. See TRL's versioned SFT data formats and completion loss for the upstream dataset and masking contract.

Run SFT or GRPO

After inspecting a recipe and provisioning its model:

python -m hessian_recipes.training recipes/reasoning-sft.json
python -m hessian_recipes.training recipes/reasoning-grpo.json

Training wraps string prompts as user messages and SFT completions as assistant messages, using the checkpoint's own chat template. SFT uses completion-only loss. Both recipes can train LoRA adapters through PEFT; setting lora to null trains the full model. The saved checkpoint/ contains an adapter when LoRA is enabled; retain the base model identity and revision in the manifest when loading it later. Replace the tiny defaults with measured token budgets and a training corpus justified by your task.

For Qwen3, set the same explicit top-level mode in training and teacher recipes: "chat_template_kwargs": {"enable_thinking": true}. Only this boolean switch is accepted. TRL SFT receives it on every dataset row for both prompt and complete conversation formatting; GRPO receives it through its native configuration. Teacher generation forwards it when rendering the prompt. Keep the same mode and tokenizer template during evaluation. With thinking enabled, reasoning SFT targets should contain <think>German rationale</think> followed by the final <answer>…</answer> block: a final-only target can teach the model to skip thinking because Qwen's template inserts an empty thinking block. With thinking disabled, targets containing nonempty or malformed thinking blocks are rejected before training, avoiding a mismatch between the prompt prefix and complete conversation. Empty thinking blocks and ordinary final responses are accepted.

The default local teacher uses greedy generation. Qwen recommends sampling for its thinking mode; use the explicitly configured evaluation generator for reasoning experiments rather than treating this simple teacher generator as a Qwen reasoning-quality baseline. See the Qwen3 mode and decoding guidance.

GRPO samples multiple completions and passes the dataset's solution column to the same strict verifier. Its generation batch must be divisible by num_generations. The adapter sets steps_per_generation to gradient_accumulation_steps, and validates the effective batch using WORLD_SIZE. For example, batch 2 with two generations works on one process. beta=0 uses no KL reference penalty. All-zero or identical rewards produce no useful advantage signal; inspect reward spread, format failures and held-out evaluation before increasing compute. The recipe's loss_type is explicit because upstream defaults can change. Consult the versioned GRPO trainer and custom reward interface for the algorithm and runtime contract.

Real training requires the exact top-level dependency pins. Runs refuse a nonempty output directory. They save a local checkpoint and recipe-manifest.json containing the complete resolved plan, configuration and data SHA-256, model revision, dependency versions, seed, status and train metrics. Local model files are hashed at run start. A failed run records its failure type. The adapters set report_to="none", push_to_hub=false, trust_remote_code=false, and use no vLLM server. No upload happens automatically. The manifest helps reproduce inputs; a seed alone does not ensure deterministic GPU training. A scheduler/cluster container still needs a real GPU validation run and measurements of memory, throughput and checkpoint recovery.

Merge a local LoRA adapter for evaluation

The evaluation adapter accepts standalone model checkpoints. Merge a trained LoRA adapter into the exact local base used for training:

python -m hessian_recipes.checkpoints /absolute/path/to/base runs/reasoning-sft/checkpoint \
  runs/reasoning-sft-merged --dtype bfloat16 --device cuda

merge_adapter(base_dir, adapter_dir, output_dir, dtype="bfloat16", device="cpu") also exposes this operation in Python. CUDA is an explicit request and fails if unavailable. CPU merging requires enough RAM for the model and temporary merge buffers. The helper explicitly loads the supplied base, uses PEFT's merge_and_unload(safe_merge=True), saves full safetensors and the base tokenizer, and never follows the adapter's stored base-model pointer or downloads files. It hashes source directories before loading and verifies them after loading and merging. The destination must not exist and must be separate from both sources. An adjacent <output>.merge-manifest.json records source/adapter/output hashes, declared and actually used base identities, tokenizer file hashes, dtype, device and dependencies. The manifest stays outside the checkpoint directory to avoid a self-referential content hash. Select this merged checkpoint for candidate evaluation, retaining one shared frozen tokenizer and the same thinking flag for baseline and candidate. See PEFT's checkpoint and merge contract.

For large distributed jobs, consider verl, OpenRLHF or NVIDIA NeMo RL. Keep their job orchestration and optimizer logic upstream. The own-harness boundary belongs at rollout and outcome verification; importing a trajectory does not itself establish an on-policy RL training loop. TRL's experimental rollout_func interface requires token IDs and rollout log probabilities. Do not replace those with arbitrary imported action logs.

What is actually verified

HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
  python examples/training/cpu_smoke.py
python -m unittest discover -s tests -p 'test_training.py'
python -m unittest discover -s tests -p 'test_distillation.py'

Pass --output runs/cpu-smoke to preserve the tiny base model, checkpoints and manifests for inspection or a subsequent local inference/evaluation check. The destination must be fresh or empty. Without --output, the temporary run is deleted on exit.

The offline smoke test builds a seeded random tiny GPT-2 model and a local WordLevel tokenizer, then runs teacher generation and one real optimizer step each with SFT and GRPO. It checks checkpoint and manifest creation and actual parameter changes for both methods. GRPO uses a 32-completion synthetic group and asserts varied rewards and a nonzero gradient. Its initial scalar loss can still be zero because group advantages average to zero. This verifies effective optimizer integration, without demonstrating a quality improvement. Unit tests cover recipe validation, immutable model identity, ambiguous-answer rejection, reward batch alignment, duplicate and conflicting data, filtering provenance and output protection. No GPU, cluster scale, scientific domain accuracy or model-quality improvement is validated by these checks.