# German Qwen pilot: measured results

The first GPU run completed on 15 October 2026. Adaptation changed the language
and answer format as intended on the audited examples. Strict final-answer
protocol accuracy changed from 28/64 to 30/64; this small experiment does not
establish improved numerical reasoning.

## Experiment

The base is the existing local Qwen3-1.7B snapshot, upstream revision
`70d244cc86ccca08cf5af4e1e306ecf908b1ad5e`. Rank-16 LoRA SFT used 1,024 filtered
German MathOctopus/GSM8K explanations, 128 optimizer steps, batch size 1 with
8-step accumulation, learning rate 0.0001, BF16 and a 1,024-token training cutoff.
All complete training targets fit. Training took 298.4 seconds on one
NVIDIA A100-SXM4-80GB. All 196 LoRA-B tensors contain nonzero trained weights;
all adapter tensors are finite. The adapter was merged into the identified local
base and evaluated as a standalone checkpoint.

The executed code is commit `a1d5e8c5cb6531b227cab6c2c61b4df7e2939ff6`.
Its 132 local tests and GitHub CI passed. The pinned environment includes
PyTorch 2.10.0+cu126, Transformers 5.3.0, TRL 0.29.0 and PEFT 0.18.1.

Evaluation uses the same frozen 64-item subset of external MGSM German, tokenizer,
thinking mode, prompt, seed and generation settings for both models. Each item
has one trial: temperature 0.6, top-p 0.95, top-k 20, min-p 0, seed 42, and at most
768 generated tokens. This is a bounded pilot, not a replication of the base
model's published benchmark conditions. The source-disjoint development set has
128 examples; the external test provides no training targets.

## Paired evaluation

| Measure | Base | Adapted |
|---|---:|---:|
| Correct final answers | 28/64 | 30/64 |
| Strict final-answer protocol accuracy | 43.75% | 46.875% |
| Malformed outputs | 35 | 0 |
| Wrong answers in accepted format | 1 | 34 |
| Trials reaching the token cap | 33 | 0 |
| Total generated tokens | 40,919 | 8,904 |
| Generation time, excluding model loading | 942.4 s | 203.1 s |

The adapted model wins on 12 items and loses on 10, with 18 both correct and 24
both incorrect. The accuracy difference is **+3.125 percentage points**. A
descriptive paired item bootstrap gives a 95% interval of **−10.9 to +17.2
percentage points** (10,000 resamples, seed 20261015). It excludes training-seed
and generation-seed variance, so the numerical result is inconclusive.

The scorer accepts a canonical decimal answer or one terminal `<answer>` tag
after closed thinking. Malformed and token-cap-hit outputs remain in the
accuracy denominator. Learning shorter traces and the requested answer tags can
change this score without improving mathematical reasoning. The malformed base
outputs have not been rescored as mathematically correct or incorrect. Reaching
the cap indicates possible clipping; the runner does not capture finish reasons.
The generation-time reduction accompanies much shorter outputs and is not
evidence of faster generation per token.

## Qualitative language audit

An agent inspected the first ten frozen paired outputs, excluding quoted input
when judging the dominant language of the thinking block. The item IDs and
aggregate judgments are recorded in the accompanying JSON.

| Observation on ten fixed items | Base | Adapted |
|---|---:|---:|
| German-dominant thinking | 0/10 | 10/10 |
| English-dominant thinking | 10/10 | 0/10 |
| Valid terminal answer tags | 4/10 | 10/10 |
| Trials reaching the token cap | 6/10 | 0/10 |
| Mean generated tokens | 682.4 | 142.5 |

The adapted model's final segments are numeric-only and language-neutral. This
small qualitative inspection supports a language and format change; it is not a
native-speaker fluency assessment or a judgment of explanation correctness.

## Artifacts and reproduction

The [use-case protocol](german-reasoning.md) describes data preparation, GPU
allocation and execution. [Machine-readable results](german-reasoning-results.json)
record aggregate metrics, source and model identities, evidence hashes and the
audit item IDs. The full private run remains on shared storage under the ignored
repository directory `runs/german-qwen/pilot-02/`:

- `training/checkpoint/`: trained LoRA adapter and tokenizer.
- `merged-checkpoint/`: standalone model in safetensors format.
- `model-card.md`: measured behavior, usage, ancestry and limitations.
- `summary/summary.json` and `summary/summary.md`: verified paired results.
- `baseline/report.json` and `candidate/report.json`: full evaluation evidence.
- `run-manifest.json`, `training/recipe-manifest.json` and
  `merged-checkpoint.merge-manifest.json`: execution and ancestry evidence.
- `release-candidate/`: local adapter bundle with evaluation evidence; unpublished.

The temporary GPU allocation has been released; artifacts remain on shared
storage. Data and weights are excluded from Git. This pilot does not establish
general German reasoning ability or release readiness. Translated explanations
have only a partial agent audit, and base-model exposure to the benchmarks is
unknown. The cluster image tag and GPU driver are recorded, but the image's
immutable digest was not captured.