# Pilot data

Generate the local ignored data from the repository root:

```bash
PYTHONPATH=src python -m hessian_recipes.german_data \
  --output runs/german-qwen/data --train-size 1024 --dev-size 128
```

The preparation command requires `pyarrow` only for reading MGSM's small Parquet
file. Training environments already normally provide it through `datasets`; it
is not a dependency of the core CLI. Source downloads use public immutable URLs,
require no credentials, and verify fixed SHA-256 hashes.

Files under `runs/german-qwen/data`:

| File | Rows | Purpose |
|---|---:|---|
| `train.jsonl` | 1,024 | German SFT prompt, preserved explanation and canonical numeric answer. |
| `dev.jsonl` | 128 | Source-disjoint development examples; not used for optimizer updates. |
| `mgsm-de.jsonl` | 250 | External German math evaluation only; prompt and gold stored separately. |
| `manifest.json` | — | Pins, source/output hashes, source IDs, rejection counts and license notes. |

Training fields are `id`, `source_id`, `prompt`, `completion`, `solution`, and
`source_row`. Evaluation fields are `id`, `source_id`, `prompt`, `answer`,
`scorer: "final_numeric"`, and `category: "German math"`. A generator must receive
only `prompt`; `answer` is for scoring after generation.

Sources live in the sibling `runs/german-qwen/sources` directory. The MathOctopus
dataset is Apache-2.0; original GSM8K is MIT; the MGSM derivative mirror is marked
CC-BY-SA-4.0. Preserve attribution to the original creators and the translation
work. Do not copy these upstream or derived dataset files into source control.

See [the full data and experiment protocol](../../docs/use-cases/german-reasoning.md)
for exact alignment, filtering, inspection findings and evaluation limits.