# Roadmap and acceptance criteria

The runnable foundation and several bounded GPU pilots are complete. Their
evidence qualifies execution paths; it does not establish general reasoning or
agent improvement. The survey motivates adaptation and evaluation. The supplied
programme proposal now names German RAG with cited evidence and business-
communication agent sequences; expert tasks, corpora and scoring remain pending.
The [documentation index](README.md) links each implemented path and its results.

| Workstream | Current evidence | Next acceptance target |
|---|---|---|
| Foundation | Local CLI, frozen evaluation/report contract, distillation preparation, training adapters, trajectory validator, candidate-bundle builder and private HF reading hub. More than300 core checks pass; the 73-test [foundation JSON](validation.json) is historical. | Representative partner inputs, actual model recovery and deployment-site qualification beyond local integration checks; see [validation](validation.md). |
| Reasoning and distillation | Tiny-model teacher/SFT/GRPO CPU checks and a real German Qwen SFT pilot with matched evaluation. Strict final-answer protocol accuracy changed from 28/64 to 30/64, an inconclusive result; see [measured SFT results](use-cases/german-reasoning-results.md). | A useful partner-task adaptation with repeated training/generation seeds, retained capabilities and explicit release thresholds. A substantive teacher-distillation experiment remains separate from the tiny-model wiring check. |
| Native multi-turn agent RL | Real local TRL GRPO completed 16 steps/64 rollouts with exact token/mask evidence and matched held-out evaluation. Strict workflow success changed from 4/32 to 5/32; MGSM retention changed from 30/64 to 29/64 and failed the zero-drop gate. See [results](use-cases/german-agent-rl-results.md). | Fresh evaluation data before further model selection; reliable workflow gains without unacceptable retention loss. One synthetic family and a single seed do not establish general agent ability. |
| Reusable agent gyms | Native NeMo Gym service/client checks and an eight-step/32-rollout A100 update completed. Native OpenEnv service/SDK/TRL bridge and Prime Verifiers' fixture pass CPU checks. Both 20 October longer-run GPU attempts stopped at their resume-qualification token audit; see [current status](programme-status.md) and [gym qualification](agent-gyms.md). | Verify the recreated JAIF environments and fresh plans, then complete actual resume qualification, 256-step training and matched held-out/MGSM assessment for each NeMo/OpenEnv arm, sequentially on one GPU. An actual Prime-RL-compatible pilot remains separate. CPU checks and the earlier NeMo update do not resolve the retention regression. |
| Existing team harness | A token-preserving contract, example harness and optional pinned Agent Lightning/verl launcher have source/API and mocked checks. No real Lightning GPU update has run. | One team harness with exact sampled tokens, tool-loss exclusion, policy/group consistency, failure recovery, a real update and matched held-out evidence; see [adoption pilot](agent-harness.md). |
| Two programme benchmarks | [Designs](benchmark-designs.md) now target German RAG with cited evidence and business-communication agent sequences. Two existing synthetic tracks exercise the kit; they are not these benchmarks. | Representative expert-reviewed tasks, task-specific scorers, held-out protocol, contamination/overlap review, agreement checks, repeated trials and release conditions. |
| Evaluation service | The local kit supplies specifications, reports and comparison gates; no hosted service exists. | Managed execution with isolation, access control, queue/retry semantics, private-data retention rules, measured costs and reproducibility checked with participating teams. |
| Domain pilots | Robotics, Bio/Med and materials are [design sketches](domains.md); no native domain training/container validation has run. | One committed team and dataset per selected domain, a native baseline and metrics, versioned artifacts and qualified execution environment. |
| Hugging Face recipe home | Private personal Space and links-only collection are live, with recipe pages, executable source and aggregate evidence. Weights, downloaded data and raw runs stay on the cluster. | Account storage restrictions currently block repository downloads; App/source pages work when authenticated. Institutional organization cards and public launch remain deferred. |

## Decisions for the next partner pilot

1. Turn the two proposed benchmark subjects into expert task specifications and
   deployment decisions; the supplied development tracks remain placeholders.
2. Select the team's model and adaptation target with an immutable revision and
   GPU budget. Use the completed German SFT and agent-RL paths as bounded examples;
   choose SFT, teacher distillation or RL from the task and available supervision.
3. Freeze train/development/test boundaries, scorer revision, decoding budget,
   baseline checkpoint, and evidence needed to call a run successful.
4. For agent RL, choose one existing team harness or reusable gym, one executable
   reward and one bounded task. Preserve actual token/context evidence and assess
   both task success and retention. Follow the [harness adoption pilot](agent-harness.md)
   or [gym protocol](agent-gyms.md).

Do not promote a sketch to a validated recipe because its command launches.
Promotion requires a completed, reproducible run with accessible evidence and
clearly stated resource requirements. The completed local GPU pilots meet that
execution requirement within their recorded scope; their quality limitations and
failed retention gate remain part of the evidence. Domain implementation,
distributed RL and hosted evaluation execution still require separate work.