German reasoning: first Qwen adaptation pilot

This pilot uses real translated math explanations, a source-disjoint development set, and an external human-translated German math test. It starts with supervised fine-tuning. RL should follow only if the reward, baseline and language behavior are sufficiently understood. Training requires a GPU allocation; data preparation and tokenizer checks can run on the current CPU node.

The completed A100 pilot results record the measured language and format changes, inconclusive accuracy difference, and saved artifacts.

Current cluster commands run from /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes in workspace JAIF; see operations. Other teams can use any checkout root and provide their own local model paths. The recorded pilot's identities and measurements remain historical evidence, not a new JAIF run.

Data preparation

From the repository root, using Python with optional pyarrow available:

PYTHONPATH=src python -m hessian_recipes.german_data \
  --output runs/german-qwen/data --train-size 1024 --dev-size 128

The sources and generated data are local ignored run artifacts. Source URLs, immutable revisions and SHA-256 hashes are fixed in the preparation module and recorded in data/manifest.json. Re-running into a nonempty output is rejected; --offline requires previously downloaded, checksum-matching sources.

MathOctopus is the released dataset associated with Breaking Language Barriers in Multilingual Mathematical Reasoning. This is a useful established baseline; its historical model results do not predict improvements on the selected Qwen checkpoint.

Alignment and filtering

The MathOctopus release contains 73,559 JSONL records despite its .json suffix. The reject field is a dummy refusal and is not used as a preference signal. German records are associated with the preceding English source block only when both its complete English question and explanation match one unique original GSM8K training record. Canonical test questions and ambiguous sources are excluded. The code never guesses source identity from a fixed multilingual block size.

For the first pilot, retain unambiguous integer answers whose last numeric token in the German explanation agrees with canonical gold. Reject missing or ambiguous numeric forms, preexisting structural tags, untranslated source questions, known audited explanation errors, and initially overlong text. Preserve the explanation as <think>...</think> and end with <answer>GOLD</answer>. A mismatching explanation is discarded, never rewritten to make its answer agree.

Split by canonical source ID using seed 42. train.jsonl contains 1,024 rows; dev.jsonl contains 128 rows from different source problems. The external mgsm-de.jsonl contains all 250 human-translated test problems and no explanation targets. Its prompts contain only the problem and the common German output instruction; the gold answer is a separate scorer field. MGSM's eight few-shot exemplars are not used as training examples.

The initial source-order inspection of 40 selected rows found an entirely English record (gsm8k/train/00098) and two explanations with incorrect intermediate reasoning despite correct final answers: incorrect calendar month lengths (00090) and contradictory egg counts (00237). Source-copy filtering and explicit audited exclusions remove those cases. This was an agent inspection, not a systematic native-speaker or mathematical-expert review. Outcome matching does not prove that every translated explanation is correct.

Model and experiment

Use an existing local Qwen snapshot and preserve its tokenizer and chat-template identity. Check the complete prompt and completion lengths with that tokenizer before training; the character limit in preparation is only a preliminary bound. Reject examples that do not fit rather than truncating away their final answer or reasoning. Keep Qwen's thinking mode and decoding settings fixed across the baseline and adapted checkpoints.

The first recorded allocation is one A100-SXM4-80GB in the existing Determined 42_Compute pool. The pinned PyTorch 2.10.0+cu126 environment passes an actual CUDA BF16 matrix multiplication check on that worker. The complete local Qwen3-1.7B snapshot identifies upstream commit 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e; content hashes bind the actual files consumed by the run.

The bounded experiment uses 1,024 training examples, 128 optimizer steps, batch 1 with 8-step accumulation, BF16, gradient checkpointing, and rank-16 LoRA on all linear layers. Its training examples are 116–646 tokens including the chat template, so the 1,024-token cutoff preserves all complete targets. The model's explicit thinking flag is true, and the targets contain German thinking blocks. The adapter is merged into the explicitly identified local base and saved as a full safetensors checkpoint with ancestry hashes.

Use a separate environment; the container's base Python packages are untouched:

uv venv .venv-gpu --python python3.11
uv pip install --python .venv-gpu/bin/python -r requirements/german-qwen-cu126.txt
uv pip install --python .venv-gpu/bin/python -e .
.venv-gpu/bin/python -m hessian_recipes.german_data --output runs/german-qwen/data
.venv-gpu/bin/python -m hessian_recipes.german_pilot prepare \
  --model /absolute/local/Qwen3-1.7B --data runs/german-qwen/data \
  --output runs/german-qwen/my-pilot --steps 128 --eval-size 64

The Determined shell configuration requests one GPU and the workspace's normal mounts. On the GPU worker:

python scripts/gpu-preflight.py --output runs/german-qwen/gpu-preflight.json
bash scripts/run-german-qwen.sh /absolute/.venv-gpu/bin/python \
  /absolute/runs/german-qwen/my-pilot/plan.json

The driver records each stage in run-manifest.json and keeps separate logs for baseline evaluation, SFT, merge and candidate evaluation. It refuses reused run directories and fails when frozen inputs change. A successful execution is not evidence of a useful language or reasoning improvement; inspect the comparison.

Generate a verified summary after all four stages complete:

python examples/german-reasoning/report.py \
  --run runs/german-qwen/my-pilot --output runs/german-qwen/my-pilot/summary

The summary validates the report snapshots, re-scores outputs, verifies the training/merge ancestry, and calculates paired item outcomes and a descriptive bootstrap interval. A completed run with a regressing candidate remains reportable and is marked as a regression.

Evaluation and limits

Evaluate the untouched baseline before training and the adapted model with the same prompt, tokenizer/template, thinking setting and generation budget. Keep the 128-source development set for debugging. The first bounded run uses a fixed, seeded 64-item subset of the 250-example MGSM German test; its sampled IDs and hash are frozen before any model evaluation. Use --eval-size 250 for the full test. Report strict final-answer protocol accuracy, unparseable outputs, outputs hitting the token cap, latency, token usage, and uncertainty on the paired accuracy difference. Small changes on these pilot samples may be inconclusive.

Thinking-mode decoding follows the local Qwen model card's recommended sampling: temperature 0.6, top-p 0.95, top-k 20 and min-p 0. The first pilot limits generation to 768 tokens per item, with one seeded trial, for a bounded compute run. This is a much smaller reasoning budget than the model card's full evaluation setup. Both checkpoints receive identical settings; report token-cap hits separately. This experiment does not reproduce the model card's benchmark numbers.

The strict score includes compliance with the requested terminal answer format. SFT explicitly teaches that format and shorter explanations, so a score change can reflect formatting and budget efficiency as well as arithmetic. Canonical numbers or one final <answer> block are accepted; other correct-looking prose is not automatically reinterpreted by the scorer. Hitting the token cap is reported as an observed budget limit, not proof that a particular answer would have been correct with additional tokens.

Audit German output behavior separately from answer accuracy. A model can answer the arithmetic correctly while writing English reasoning. Inspect a fixed sample for German explanations, language switching, instruction compliance and malformed answer tags. For a broader next step, the European IFEval repository contains a complete 541-prompt German track with upstream attribution and native checkers; its data/source.json points to the human-checked German IFEval translation. That track is a recommended follow-up, not an implemented pilot scorer. Preserve explicit requested response-language constraints rather than assuming every German prompt requires a German-only answer.

This is a narrow math-and-language adaptation experiment. It does not establish general German reasoning, transfer to professional domains, or improvements beyond possible benchmark familiarity. Add independently authored tasks before making deployment claims.