Two proposed benchmark tracks
Design status: proposal, 20 October 2026. The requested topics are now German RAG with cited sources and agent sequences in business communication. Neither has an implemented task set, expert labels, scorer or measured baseline here. Documentation stays English. German business tasks also need native-language and process review. Existing synthetic fixtures remain pipeline checks; their identities, data and results are unchanged.
Implemented foundation and gaps
| Element | Available now | Remaining work |
|---|---|---|
| Post-training | TRL SFT/GRPO, local distillation, merging; bounded German pilots. | Evidence on these actual tasks. Existing results establish no capability on them. |
| Local checkpoints | Frozen full/merged Transformers checkpoints, tokenizer/templates, sampling; exercised CPU/CUDA generation. | Retrieval and business-process environments. |
| Hosted endpoints | Trusted callable integration; optional Inspect provider execution. | Qualified generic HTTP adapter, identity/retry/timeout contract. Inspect logs are not imported into core comparisons. |
| Evaluation | Hashed evidence, replay scoring, coverage/category regression gates, usage/timing, experiment-specific paired analysis. | Retrieval/citation and communication scorers; current core scorers are exact/numeric/toy workflow. |
| Agent tools | Multi-turn lookup/calculator; native NeMo Gym/TRL and Verifiers integration, audited trajectories. | Inbox/calendar/CRM, permissions, approvals and communication gold. |
| Robustness/calibration | Integrity, malformed-input, bounded tool-state/infrastructure checks; descriptive uncertainty. | Topic robustness suites, confidence calibration and abstention evaluation. Accuracy intervals are not confidence calibration. |
| Audit/documentation | Private reports, hashes, candidate bundles and card templates; release_eligible: false. | Redaction, access/retention controls, tamper-evident operational logs and application-specific compliance documentation. |
| Service | Local kit and private recipe/source hub. | Managed evaluation execution; the hub is not a model inference endpoint. |
See evaluation, training, gyms and validation.
Closest upstreams to reuse
Pin commits, datasets and scorers before porting; repository licensing does not settle every underlying artifact's rights. These are design choices, not installed integrations.
| Upstream | Scope, reuse and limits |
|---|---|
| MEMERAG | Human RAG faithfulness/relevance labels explicitly include German. Use for independent judge meta-evaluation, not end-to-end RAG gold. Upstream excludes model training as an intended use. Repository CC-BY-NC-4.0: commercial-service reuse requires rights review. |
| ALCE | English ASQA/QAMPARI/ELI5 citation-aware generation; reuse citation support/completeness methodology. MIT code; inspect corpus rights separately. Older dependencies and automatic judges need German validation. Citation syntax alone proves no support. |
| WorkBench | English workplace tasks over sandbox email/calendar/CRM/project-management/analytics. Reuse terminal-state and unwanted-side-effect scoring, accepting alternative valid paths. Repository MIT. German processes/approvals need new expert work; current 2026 revisions change evaluation, so historical scores cannot be mixed. |
| AgentDojo | English prompt-injection tasks including workspace email/calendar/drive and Slack. Reuse adversarial messages and separate benign utility/attacked utility/attacker success. MIT code; evolving API. It does not validate complete business processes or German communication quality. |
Additional options: RAGBench (CC-BY-4.0 dataset; model annotations), ARES (Apache-2.0 judge methodology), and BERGEN (reproducible RAG components; CC-BY-NC-SA-4.0). CRUD-RAG targets Chinese. τ-Bench supplies MIT customer-service policy/tool simulation; current 2.0's six languages exclude German and its scores are not comparable with 1.x. ToolSandbox and AgentBench are less directly workplace-focused. “WorkflowBench” identifies different projects; select an exact repository first.
German RAG design
Use an authorized, versioned German corpus with stable document/passage IDs, source dates, offsets, hashes and rights/access labels. Experts annotate accepted claims, evidence spans, contradictions and unanswerable questions. Split by source/document family, not chunks. Freeze parsing, index/retriever/reranker, context budgets and actual retrieved passages.
Separate oracle-evidence grounding from end-to-end retrieval. Measure retrieval Recall@k/nDCG, answer correctness/coverage, citation precision/completeness, unsupported claims and abstention. Compare BM25, dense/hybrid and oracle baselines at matched budgets. Test paraphrases, tables/units, conflicting versions, passage order and injected instructions. Resolve citations against frozen evidence; source-grounded answers do not establish external scientific/legal truth.
Business-communication design
Use an isolated inbox/calendar/CRM sandbox with explicit actors, recipients, permissions, clock/time zone and approval state. Teams select representative reply-drafting, scheduling, escalation and handoff tasks. Distinguish draft/send; perform no real delivery. Annotate required terminal state and forbidden changes, allowing valid alternative sequences and recovery.
Report completion, workflow validity, side effects, approval/wrong-recipient violations, caps, tokens and latency separately from expert language/tone review. Test ambiguous recipients, daylight-saving changes, duplicate requests, partial failures and injected messages. Separate workflow/template families across splits. Unexpected outages remain infrastructure failures, not agent mistakes.
Acceptance work
Before release: secure rights; create blinded, independently annotated/adjudicated gold; validate German judges; freeze splits/baselines/budgets; test leakage, malformed evidence, outages and redaction; report paired item/source-clustered uncertainty. If confidence is exposed, validate reliability/Brier and abstention risk/coverage separately. No expert dataset or such validation exists here yet.
Reports can support documentation, not certify compliance. High-risk systems have distinct technical-documentation and record-keeping requirements under Article 11 and Article 12. Applicability depends on intended use and responsible role; the kit supplies no AI Act conformity assessment or release approval.