Provisional development tracks

These small, original synthetic fixtures exercise the evaluation pipeline. The two requested benchmark topics are now German RAG with cited sources and agent sequences in business communication. See the English benchmark designs and related work for their proposed scope, reusable upstreams and implementation gaps. Neither real task set nor topic-specific scorer is implemented yet.

The existing tracks below are not those two benchmarks, are not validated capability measures, and cannot establish release readiness. Their identities and fixtures remain unchanged. Public fixture replay is not a model evaluation and must not be reported as model performance.

Every track provides a frozen spec.json with synthetic baseline responses and a spec-candidate.json with synthetic improved responses. Their difference is an intentional demonstration, not measured post-training improvement. Responses have no token usage because no tokenizer or model ran. Repeated fixture trials replay the same answer and provide no additional statistical evidence.

The workflow track evaluates a single generated plan replayed against a toy state machine, not an interactive agent's ability to recover from live tool failures. No generated shell commands, Python, network access, or real tools are executed.

All fixture content was authored for this repository. No public data or repository license has been selected. Data and predictions are JSONL; hashes cover their exact UTF-8 bytes. Paths in specs resolve relative to the spec's directory, independently of the working directory. See evaluation guide.

Before a real benchmark release: agree the concrete tasks with the teams, document relevance and licensing, design held-out splits and contamination checks, validate scorers, measure agreement for subjective tasks, compare strong baselines with matched budgets, and register what each metric can and cannot establish. Do not train on the eventual test set.