Roadmap and acceptance criteria
The runnable foundation and several bounded GPU pilots are complete. Their evidence qualifies execution paths; it does not establish general reasoning or agent improvement. The survey motivates adaptation and evaluation. The supplied programme proposal now names German RAG with cited evidence and business- communication agent sequences; expert tasks, corpora and scoring remain pending. The documentation index links each implemented path and its results.
| Workstream | Current evidence | Next acceptance target |
|---|---|---|
| Foundation | Local CLI, frozen evaluation/report contract, distillation preparation, training adapters, trajectory validator, candidate-bundle builder and private HF reading hub. More than300 core checks pass; the 73-test foundation JSON is historical. | Representative partner inputs, actual model recovery and deployment-site qualification beyond local integration checks; see validation. |
| Reasoning and distillation | Tiny-model teacher/SFT/GRPO CPU checks and a real German Qwen SFT pilot with matched evaluation. Strict final-answer protocol accuracy changed from 28/64 to 30/64, an inconclusive result; see measured SFT results. | A useful partner-task adaptation with repeated training/generation seeds, retained capabilities and explicit release thresholds. A substantive teacher-distillation experiment remains separate from the tiny-model wiring check. |
| Native multi-turn agent RL | Real local TRL GRPO completed 16 steps/64 rollouts with exact token/mask evidence and matched held-out evaluation. Strict workflow success changed from 4/32 to 5/32; MGSM retention changed from 30/64 to 29/64 and failed the zero-drop gate. See results. | Fresh evaluation data before further model selection; reliable workflow gains without unacceptable retention loss. One synthetic family and a single seed do not establish general agent ability. |
| Reusable agent gyms | Native NeMo Gym service/client checks and an eight-step/32-rollout A100 update completed. Native OpenEnv service/SDK/TRL bridge and Prime Verifiers' fixture pass CPU checks. Both 20 October longer-run GPU attempts stopped at their resume-qualification token audit; see current status and gym qualification. | Verify the recreated JAIF environments and fresh plans, then complete actual resume qualification, 256-step training and matched held-out/MGSM assessment for each NeMo/OpenEnv arm, sequentially on one GPU. An actual Prime-RL-compatible pilot remains separate. CPU checks and the earlier NeMo update do not resolve the retention regression. |
| Existing team harness | A token-preserving contract, example harness and optional pinned Agent Lightning/verl launcher have source/API and mocked checks. No real Lightning GPU update has run. | One team harness with exact sampled tokens, tool-loss exclusion, policy/group consistency, failure recovery, a real update and matched held-out evidence; see adoption pilot. |
| Two programme benchmarks | Designs now target German RAG with cited evidence and business-communication agent sequences. Two existing synthetic tracks exercise the kit; they are not these benchmarks. | Representative expert-reviewed tasks, task-specific scorers, held-out protocol, contamination/overlap review, agreement checks, repeated trials and release conditions. |
| Evaluation service | The local kit supplies specifications, reports and comparison gates; no hosted service exists. | Managed execution with isolation, access control, queue/retry semantics, private-data retention rules, measured costs and reproducibility checked with participating teams. |
| Domain pilots | Robotics, Bio/Med and materials are design sketches; no native domain training/container validation has run. | One committed team and dataset per selected domain, a native baseline and metrics, versioned artifacts and qualified execution environment. |
| Hugging Face recipe home | Private personal Space and links-only collection are live, with recipe pages, executable source and aggregate evidence. Weights, downloaded data and raw runs stay on the cluster. | Account storage restrictions currently block repository downloads; App/source pages work when authenticated. Institutional organization cards and public launch remain deferred. |
Decisions for the next partner pilot
- Turn the two proposed benchmark subjects into expert task specifications and
deployment decisions; the supplied development tracks remain placeholders.
- Select the team's model and adaptation target with an immutable revision and
GPU budget. Use the completed German SFT and agent-RL paths as bounded examples; choose SFT, teacher distillation or RL from the task and available supervision.
- Freeze train/development/test boundaries, scorer revision, decoding budget,
baseline checkpoint, and evidence needed to call a run successful.
- For agent RL, choose one existing team harness or reusable gym, one executable
reward and one bounded task. Preserve actual token/context evidence and assess both task success and retention. Follow the harness adoption pilot or gym protocol.
Do not promote a sketch to a validated recipe because its command launches. Promotion requires a completed, reproducible run with accessible evidence and clearly stated resource requirements. The completed local GPU pilots meet that execution requirement within their recorded scope; their quality limitations and failed retention gate remain part of the evidence. Domain implementation, distributed RL and hosted evaluation execution still require separate work.