Pilot data
Generate the local ignored data from the repository root:
PYTHONPATH=src python -m hessian_recipes.german_data \
--output runs/german-qwen/data --train-size 1024 --dev-size 128The preparation command requires pyarrow only for reading MGSM's small Parquet file. Training environments already normally provide it through datasets; it is not a dependency of the core CLI. Source downloads use public immutable URLs, require no credentials, and verify fixed SHA-256 hashes.
Files under runs/german-qwen/data:
| File | Rows | Purpose |
|---|---|---|
train.jsonl | 1,024 | German SFT prompt, preserved explanation and canonical numeric answer. |
dev.jsonl | 128 | Source-disjoint development examples; not used for optimizer updates. |
mgsm-de.jsonl | 250 | External German math evaluation only; prompt and gold stored separately. |
manifest.json | — | Pins, source/output hashes, source IDs, rejection counts and license notes. |
Training fields are id, source_id, prompt, completion, solution, and source_row. Evaluation fields are id, source_id, prompt, answer, scorer: "final_numeric", and category: "German math". A generator must receive only prompt; answer is for scoring after generation.
Sources live in the sibling runs/german-qwen/sources directory. The MathOctopus dataset is Apache-2.0; original GSM8K is MIT; the MGSM derivative mirror is marked CC-BY-SA-4.0. Preserve attribution to the original creators and the translation work. Do not copy these upstream or derived dataset files into source control.
See the full data and experiment protocol for exact alignment, filtering, inspection findings and evaluation limits.