# Provisional development tracks

These small, original synthetic fixtures exercise the evaluation pipeline. The two
requested benchmark topics are now **German RAG with cited sources** and **agent
sequences in business communication**. See the English
[benchmark designs and related work](../docs/benchmark-designs.md) for their
proposed scope, reusable upstreams and implementation gaps. Neither real task set
nor topic-specific scorer is implemented yet.

The existing tracks below are **not** those two benchmarks, are not validated
capability measures, and cannot establish release readiness. Their identities and
fixtures remain unchanged. Public fixture replay is not a model evaluation and
must not be reported as model performance.

| Track | Fixture scope | Items | Trials per item |
| --- | --- | ---: | ---: |
| `reasoning_reliability` | exact arithmetic, units, logic, constraints | 4 | 2 |
| `tool_workflow_reliability` | reservation state changes in a deterministic toy environment | 3 | 2 |

Every track provides a frozen `spec.json` with synthetic baseline responses and a
`spec-candidate.json` with synthetic improved responses. Their difference is an
intentional demonstration, not measured post-training improvement. Responses have
no token usage because no tokenizer or model ran. Repeated fixture trials replay
the same answer and provide **no** additional statistical evidence.

The workflow track evaluates a **single generated plan replayed against a toy state
machine**, not an interactive agent's ability to recover from live tool failures.
No generated shell commands, Python, network access, or real tools are executed.

All fixture content was authored for this repository. No public data or repository
license has been selected. Data and predictions are JSONL; hashes cover their exact UTF-8 bytes.
Paths in specs resolve relative to the spec's directory, independently of the
working directory. See [evaluation guide](../docs/evaluation.md).

Before a real benchmark release: agree the concrete tasks with the teams, document relevance and
licensing, design held-out splits and contamination checks, validate scorers,
measure agreement for subjective tasks, compare strong baselines with matched
budgets, and register what each metric can and cannot establish. Do not train on
the eventual test set.