German Qwen pilot: measured results
The first GPU run completed on 15 October 2026. Adaptation changed the language and answer format as intended on the audited examples. Strict final-answer protocol accuracy changed from 28/64 to 30/64; this small experiment does not establish improved numerical reasoning.
Experiment
The base is the existing local Qwen3-1.7B snapshot, upstream revision 70d244cc86ccca08cf5af4e1e306ecf908b1ad5e. Rank-16 LoRA SFT used 1,024 filtered German MathOctopus/GSM8K explanations, 128 optimizer steps, batch size 1 with 8-step accumulation, learning rate 0.0001, BF16 and a 1,024-token training cutoff. All complete training targets fit. Training took 298.4 seconds on one NVIDIA A100-SXM4-80GB. All 196 LoRA-B tensors contain nonzero trained weights; all adapter tensors are finite. The adapter was merged into the identified local base and evaluated as a standalone checkpoint.
The executed code is commit a1d5e8c5cb6531b227cab6c2c61b4df7e2939ff6. Its 132 local tests and GitHub CI passed. The pinned environment includes PyTorch 2.10.0+cu126, Transformers 5.3.0, TRL 0.29.0 and PEFT 0.18.1.
Evaluation uses the same frozen 64-item subset of external MGSM German, tokenizer, thinking mode, prompt, seed and generation settings for both models. Each item has one trial: temperature 0.6, top-p 0.95, top-k 20, min-p 0, seed 42, and at most 768 generated tokens. This is a bounded pilot, not a replication of the base model's published benchmark conditions. The source-disjoint development set has 128 examples; the external test provides no training targets.
Paired evaluation
| Measure | Base | Adapted |
|---|---|---|
| Correct final answers | 28/64 | 30/64 |
| Strict final-answer protocol accuracy | 43.75% | 46.875% |
| Malformed outputs | 35 | 0 |
| Wrong answers in accepted format | 1 | 34 |
| Trials reaching the token cap | 33 | 0 |
| Total generated tokens | 40,919 | 8,904 |
| Generation time, excluding model loading | 942.4 s | 203.1 s |
The adapted model wins on 12 items and loses on 10, with 18 both correct and 24 both incorrect. The accuracy difference is +3.125 percentage points. A descriptive paired item bootstrap gives a 95% interval of −10.9 to +17.2 percentage points (10,000 resamples, seed 20261015). It excludes training-seed and generation-seed variance, so the numerical result is inconclusive.
The scorer accepts a canonical decimal answer or one terminal <answer> tag after closed thinking. Malformed and token-cap-hit outputs remain in the accuracy denominator. Learning shorter traces and the requested answer tags can change this score without improving mathematical reasoning. The malformed base outputs have not been rescored as mathematically correct or incorrect. Reaching the cap indicates possible clipping; the runner does not capture finish reasons. The generation-time reduction accompanies much shorter outputs and is not evidence of faster generation per token.
Qualitative language audit
An agent inspected the first ten frozen paired outputs, excluding quoted input when judging the dominant language of the thinking block. The item IDs and aggregate judgments are recorded in the accompanying JSON.
| Observation on ten fixed items | Base | Adapted |
|---|---|---|
| German-dominant thinking | 0/10 | 10/10 |
| English-dominant thinking | 10/10 | 0/10 |
| Valid terminal answer tags | 4/10 | 10/10 |
| Trials reaching the token cap | 6/10 | 0/10 |
| Mean generated tokens | 682.4 | 142.5 |
The adapted model's final segments are numeric-only and language-neutral. This small qualitative inspection supports a language and format change; it is not a native-speaker fluency assessment or a judgment of explanation correctness.
Artifacts and reproduction
The use-case protocol describes data preparation, GPU allocation and execution. Machine-readable results record aggregate metrics, source and model identities, evidence hashes and the audit item IDs. The full private run remains on shared storage under the ignored repository directory runs/german-qwen/pilot-02/:
training/checkpoint/: trained LoRA adapter and tokenizer.merged-checkpoint/: standalone model in safetensors format.model-card.md: measured behavior, usage, ancestry and limitations.summary/summary.jsonandsummary/summary.md: verified paired results.baseline/report.jsonandcandidate/report.json: full evaluation evidence.run-manifest.json,training/recipe-manifest.jsonand
merged-checkpoint.merge-manifest.json: execution and ancestry evidence.
release-candidate/: local adapter bundle with evaluation evidence; unpublished.
The temporary GPU allocation has been released; artifacts remain on shared storage. Data and weights are excluded from Git. This pilot does not establish general German reasoning ability or release readiness. Translated explanations have only a partial agent audit, and base-model exposure to the benchmarks is unknown. The cluster image tag and GPU driver are recorded, but the image's immutable digest was not captured.