# Two proposed benchmark tracks

Design status: **proposal, 20 October 2026**. The requested topics are now
**German RAG with cited sources** and **agent sequences in business communication**.
Neither has an implemented task set, expert labels, scorer or measured baseline
here. Documentation stays English. German business tasks also need native-language
and process review. Existing [synthetic fixtures](../benchmarks/README.md) remain
pipeline checks; their identities, data and results are unchanged.

## Implemented foundation and gaps

| Element | Available now | Remaining work |
| --- | --- | --- |
| Post-training | TRL SFT/GRPO, local distillation, merging; bounded German pilots. | Evidence on these actual tasks. Existing results establish no capability on them. |
| Local checkpoints | Frozen full/merged Transformers checkpoints, tokenizer/templates, sampling; exercised CPU/CUDA generation. | Retrieval and business-process environments. |
| Hosted endpoints | Trusted callable integration; optional Inspect provider execution. | Qualified generic HTTP adapter, identity/retry/timeout contract. Inspect logs are not imported into core comparisons. |
| Evaluation | Hashed evidence, replay scoring, coverage/category regression gates, usage/timing, experiment-specific paired analysis. | Retrieval/citation and communication scorers; current core scorers are exact/numeric/toy workflow. |
| Agent tools | Multi-turn lookup/calculator; native NeMo Gym/TRL and Verifiers integration, audited trajectories. | Inbox/calendar/CRM, permissions, approvals and communication gold. |
| Robustness/calibration | Integrity, malformed-input, bounded tool-state/infrastructure checks; descriptive uncertainty. | Topic robustness suites, confidence calibration and abstention evaluation. Accuracy intervals are not confidence calibration. |
| Audit/documentation | Private reports, hashes, candidate bundles and card templates; `release_eligible: false`. | Redaction, access/retention controls, tamper-evident operational logs and application-specific compliance documentation. |
| Service | Local kit and private recipe/source hub. | Managed evaluation execution; the hub is not a model inference endpoint. |

See [evaluation](evaluation.md), [training](training.md), [gyms](agent-gyms.md)
and [validation](validation.md).

## Closest upstreams to reuse

Pin commits, datasets and scorers before porting; repository licensing does not
settle every underlying artifact's rights. These are design choices, not installed
integrations.

| Upstream | Scope, reuse and limits |
| --- | --- |
| [MEMERAG](https://github.com/amazon-science/MEMERAG) | Human RAG faithfulness/relevance labels explicitly include German. Use for independent judge meta-evaluation, not end-to-end RAG gold. Upstream excludes model training as an intended use. Repository [CC-BY-NC-4.0](https://github.com/amazon-science/MEMERAG/blob/main/LICENSE): commercial-service reuse requires rights review. |
| [ALCE](https://github.com/princeton-nlp/ALCE) | English ASQA/QAMPARI/ELI5 citation-aware generation; reuse citation support/completeness methodology. [MIT code](https://github.com/princeton-nlp/ALCE/blob/main/LICENSE); inspect corpus rights separately. Older dependencies and automatic judges need German validation. Citation syntax alone proves no support. |
| [WorkBench](https://github.com/olly-styles/WorkBench) | English workplace tasks over sandbox email/calendar/CRM/project-management/analytics. Reuse terminal-state and unwanted-side-effect scoring, accepting alternative valid paths. Repository [MIT](https://github.com/olly-styles/WorkBench/blob/main/LICENSE). German processes/approvals need new expert work; current 2026 revisions change evaluation, so historical scores cannot be mixed. |
| [AgentDojo](https://github.com/ethz-spylab/agentdojo) | English prompt-injection tasks including workspace email/calendar/drive and Slack. Reuse adversarial messages and separate benign utility/attacked utility/attacker success. [MIT code](https://github.com/ethz-spylab/agentdojo/blob/main/LICENSE); evolving API. It does not validate complete business processes or German communication quality. |

Additional options: [RAGBench](https://huggingface.co/datasets/galileo-ai/ragbench)
(CC-BY-4.0 dataset; model annotations), [ARES](https://github.com/stanford-futuredata/ARES)
(Apache-2.0 judge methodology), and [BERGEN](https://github.com/naver/bergen)
(reproducible RAG components; CC-BY-NC-SA-4.0).
[CRUD-RAG](https://github.com/IAAR-Shanghai/CRUD_RAG) targets Chinese.
[τ-Bench](https://github.com/sierra-research/tau2-bench) supplies MIT customer-service
policy/tool simulation; current 2.0's six languages exclude German and its scores
are not comparable with 1.x. [ToolSandbox](https://github.com/apple-aiml-research/ToolSandbox)
and [AgentBench](https://github.com/THUDM/AgentBench) are less directly workplace-focused.
“WorkflowBench” identifies [different](https://workflowbench.theajaykumar.com/)
[projects](https://agentomy.com/workflowbench); select an exact repository first.

## German RAG design

Use an authorized, versioned German corpus with stable document/passage IDs,
source dates, offsets, hashes and rights/access labels. Experts annotate accepted
claims, evidence spans, contradictions and unanswerable questions. Split by
source/document family, not chunks. Freeze parsing, index/retriever/reranker,
context budgets and actual retrieved passages.

Separate oracle-evidence grounding from end-to-end retrieval. Measure retrieval
Recall@k/nDCG, answer correctness/coverage, citation precision/completeness,
unsupported claims and abstention. Compare BM25, dense/hybrid and oracle baselines
at matched budgets. Test paraphrases, tables/units, conflicting versions, passage
order and injected instructions. Resolve citations against frozen evidence;
source-grounded answers do not establish external scientific/legal truth.

## Business-communication design

Use an isolated inbox/calendar/CRM sandbox with explicit actors, recipients,
permissions, clock/time zone and approval state. Teams select representative
reply-drafting, scheduling, escalation and handoff tasks. Distinguish draft/send;
perform no real delivery. Annotate required terminal state and forbidden changes,
allowing valid alternative sequences and recovery.

Report completion, workflow validity, side effects, approval/wrong-recipient
violations, caps, tokens and latency separately from expert language/tone review.
Test ambiguous recipients, daylight-saving changes, duplicate requests, partial
failures and injected messages. Separate workflow/template families across splits.
Unexpected outages remain infrastructure failures, not agent mistakes.

## Acceptance work

Before release: secure rights; create blinded, independently annotated/adjudicated
gold; validate German judges; freeze splits/baselines/budgets; test leakage,
malformed evidence, outages and redaction; report paired item/source-clustered
uncertainty. If confidence is exposed, validate reliability/Brier and abstention
risk/coverage separately. No expert dataset or such validation exists here yet.

Reports can support documentation, not certify compliance. High-risk systems have
distinct technical-documentation and record-keeping requirements under
[Article 11](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-11) and
[Article 12](https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-12).
Applicability depends on intended use and responsible role; the kit supplies no
AI Act conformity assessment or release approval.