# German reasoning: first Qwen adaptation pilot
This pilot uses real translated math explanations, a source-disjoint development
set, and an external human-translated German math test. It starts with supervised
fine-tuning. RL should follow only if the reward, baseline and language behavior
are sufficiently understood. Training requires a GPU allocation; data preparation
and tokenizer checks can run on the current CPU node.
The [completed A100 pilot results](german-reasoning-results.md) record the measured
language and format changes, inconclusive accuracy difference, and saved artifacts.
Current cluster commands run from `/pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes`
in workspace `JAIF`; see [operations](../operations.md). Other teams can use any
checkout root and provide their own local model paths. The recorded pilot's
identities and measurements remain historical evidence, not a new JAIF run.
## Data preparation
From the repository root, using Python with optional `pyarrow` available:
```bash
PYTHONPATH=src python -m hessian_recipes.german_data \
--output runs/german-qwen/data --train-size 1024 --dev-size 128
```
The sources and generated data are local ignored run artifacts. Source URLs,
immutable revisions and SHA-256 hashes are fixed in the preparation module and
recorded in `data/manifest.json`. Re-running into a nonempty output is rejected;
`--offline` requires previously downloaded, checksum-matching sources.
| Input | Role | Pin and terms |
|---|---|---|
| [MathOctopus GSM8KInstruct Parallel](https://huggingface.co/datasets/Mathoctopus/GSM8KInstruct_Parallel) | German questions and existing German explanations | `dd8076b83bacd7c6ade2829317e8a500c6ffbec4`; dataset card Apache-2.0. |
| [Original GSM8K](https://github.com/openai/grade-school-math) | Exact training source identity and numeric gold; test used only for exclusion checks | `3101c7d5072418e28b9008a6636bde82a006892c`; MIT. |
| [MGSM German](https://huggingface.co/datasets/juletxara/mgsm) | External evaluation only: all 250 German test questions | `b2f13d426afe3be8d69a7e739b36724db8b66bbc`; mirror metadata CC-BY-SA-4.0. The card's body separately describes the original GSM8K MIT terms; retain the derivative attribution and license. |
MathOctopus is the released dataset associated with
[Breaking Language Barriers in Multilingual Mathematical Reasoning](https://arxiv.org/abs/2310.20246).
This is a useful established baseline; its historical model results do not predict
improvements on the selected Qwen checkpoint.
## Alignment and filtering
The MathOctopus release contains 73,559 JSONL records despite its `.json` suffix.
The `reject` field is a dummy refusal and is not used as a preference signal.
German records are associated with the preceding English source block only when
both its complete English question and explanation match one unique original
GSM8K training record. Canonical test questions and ambiguous sources are excluded.
The code never guesses source identity from a fixed multilingual block size.
For the first pilot, retain unambiguous integer answers whose last numeric token
in the German explanation agrees with canonical gold. Reject missing or ambiguous
numeric forms, preexisting structural tags, untranslated source questions, known
audited explanation errors, and initially overlong text. Preserve the explanation
as `...` and end with `GOLD`. A mismatching
explanation is discarded, never rewritten to make its answer agree.
Split by canonical source ID using seed 42. `train.jsonl` contains 1,024 rows;
`dev.jsonl` contains 128 rows from different source problems. The external
`mgsm-de.jsonl` contains all 250 human-translated test problems and no explanation
targets. Its prompts contain only the problem and the common German output
instruction; the gold answer is a separate scorer field. MGSM's eight few-shot
exemplars are not used as training examples.
The initial source-order inspection of 40 selected rows found an entirely English
record (`gsm8k/train/00098`) and two explanations with incorrect intermediate
reasoning despite correct final answers: incorrect calendar month lengths
(`00090`) and contradictory egg counts (`00237`). Source-copy filtering and
explicit audited exclusions remove those cases. This was an agent inspection,
not a systematic native-speaker or mathematical-expert review. Outcome matching
does not prove that every translated explanation is correct.
## Model and experiment
Use an existing local Qwen snapshot and preserve its tokenizer and chat-template
identity. Check the complete prompt and completion lengths with that tokenizer
before training; the character limit in preparation is only a preliminary bound.
Reject examples that do not fit rather than truncating away their final answer
or reasoning. Keep Qwen's thinking mode and decoding settings fixed across the
baseline and adapted checkpoints.
The first recorded allocation is one A100-SXM4-80GB in the existing Determined
`42_Compute` pool. The pinned PyTorch 2.10.0+cu126 environment passes an actual CUDA
BF16 matrix multiplication check on that worker. The complete local Qwen3-1.7B
snapshot identifies upstream commit `70d244cc86ccca08cf5af4e1e306ecf908b1ad5e`;
content hashes bind the actual files consumed by the run.
The bounded experiment uses 1,024 training examples, 128 optimizer steps, batch 1
with 8-step accumulation, BF16, gradient checkpointing, and rank-16 LoRA on all
linear layers. Its training examples are 116–646 tokens including the chat
template, so the 1,024-token cutoff preserves all complete targets. The model's
explicit thinking flag is `true`, and the targets contain German thinking blocks.
The adapter is merged into the explicitly identified local base and saved as a
full safetensors checkpoint with ancestry hashes.
Use a separate environment; the container's base Python packages are untouched:
```bash
uv venv .venv-gpu --python python3.11
uv pip install --python .venv-gpu/bin/python -r requirements/german-qwen-cu126.txt
uv pip install --python .venv-gpu/bin/python -e .
.venv-gpu/bin/python -m hessian_recipes.german_data --output runs/german-qwen/data
.venv-gpu/bin/python -m hessian_recipes.german_pilot prepare \
--model /absolute/local/Qwen3-1.7B --data runs/german-qwen/data \
--output runs/german-qwen/my-pilot --steps 128 --eval-size 64
```
The [Determined shell configuration](../../cluster/determined/german-qwen-shell.yaml)
requests one GPU and the workspace's normal mounts. On the GPU worker:
```bash
python scripts/gpu-preflight.py --output runs/german-qwen/gpu-preflight.json
bash scripts/run-german-qwen.sh /absolute/.venv-gpu/bin/python \
/absolute/runs/german-qwen/my-pilot/plan.json
```
The driver records each stage in `run-manifest.json` and keeps separate logs for
baseline evaluation, SFT, merge and candidate evaluation. It refuses reused run
directories and fails when frozen inputs change. A successful execution is not
evidence of a useful language or reasoning improvement; inspect the comparison.
Generate a verified summary after all four stages complete:
```bash
python examples/german-reasoning/report.py \
--run runs/german-qwen/my-pilot --output runs/german-qwen/my-pilot/summary
```
The summary validates the report snapshots, re-scores outputs, verifies the
training/merge ancestry, and calculates paired item outcomes and a descriptive
bootstrap interval. A completed run with a regressing candidate remains
reportable and is marked as a regression.
## Evaluation and limits
Evaluate the untouched baseline before training and the adapted model with the
same prompt, tokenizer/template, thinking setting and generation budget. Keep the
128-source development set for debugging. The first bounded run uses a fixed,
seeded 64-item subset of the 250-example MGSM German test; its sampled IDs and
hash are frozen before any model evaluation. Use `--eval-size 250` for the full
test. Report strict final-answer protocol accuracy,
unparseable outputs, outputs hitting the token cap, latency, token usage, and
uncertainty on the paired accuracy difference. Small changes on these pilot
samples may be inconclusive.
Thinking-mode decoding follows the local Qwen model card's recommended sampling:
temperature 0.6, top-p 0.95, top-k 20 and min-p 0. The first pilot limits generation
to 768 tokens per item, with one seeded trial, for a bounded compute run. This is
a much smaller reasoning budget than the model card's full evaluation setup.
Both checkpoints receive identical settings; report token-cap hits separately.
This experiment does not reproduce the model card's benchmark numbers.
The strict score includes compliance with the requested terminal answer format.
SFT explicitly teaches that format and shorter explanations, so a score change
can reflect formatting and budget efficiency as well as arithmetic. Canonical
numbers or one final `` block are accepted; other correct-looking prose
is not automatically reinterpreted by the scorer. Hitting the token cap is
reported as an observed budget limit, not proof that a particular answer would
have been correct with additional tokens.
Audit German output behavior separately from answer accuracy. A model can answer
the arithmetic correctly while writing English reasoning. Inspect a fixed sample
for German explanations, language switching, instruction compliance and malformed
answer tags. For a broader next step, the
[European IFEval repository](https://github.com/Telefonica-Scientific-Research/european-ifeval)
contains a complete 541-prompt German track with upstream attribution and native
checkers; its `data/source.json` points to the human-checked
[German IFEval translation](https://huggingface.co/datasets/jzhang86/de_ifeval).
That track is a recommended follow-up, not an implemented pilot scorer. Preserve
explicit requested response-language constraints rather than assuming every
German prompt requires a German-only answer.
This is a narrow math-and-language adaptation experiment. It does not establish
general German reasoning, transfer to professional domains, or improvements
beyond possible benchmark familiarity. Add independently authored tasks before
making deployment claims.