Post-training and evaluation programme: implementation status
Documentation and recipe instructions are in English. German is the language of the German-model use case and proposed German RAG tasks. This page maps the programme proposal to executable work and remaining validation, as of 21 October
- The main reading surface is the private Hugging Face recipe hub.
Our model weights, downloaded data and raw run artifacts remain on the cluster.
Publication update, 28 October 2026: the site is named hessian AI & JAIF and uses Protected visibility. Its public App includes the recipe pages and source downloads; the Hugging Face repository and GitHub collaboration repository remain private. Earlier publication records below describe the initial Private deployment.
| Programme element | Available now | Remaining work |
|---|---|---|
| Post-train a small reasoning model | Executable pinned TRL SFT/GRPO recipes, source checks, local checkpoint merging and before/after evaluation; real German Qwen3-1.7B SFT and bounded agent-RL GPU pilots. | Demonstrate useful adaptation with retained capabilities. Existing measured gains are inconclusive and the earlier RL candidate failed its observed MGSM retention gate. |
| Distill a large teacher | Local teacher generation, deterministic answer verification, filtering and SFT-ready sequence export. A tiny random-model CPU run checks wiring. | A substantive large-teacher/student experiment, source/output permissions and a matched quality/compute comparison. We do not claim demonstrated distillation benefit. |
| Train in an existing team's harness | Trajectory contract, callable example and optional Agent Lightning/verl launcher, with source/API and mocked checks. | An actual team-harness GPU update and matched held-out evaluation. The native TRL tool loop tested here is not an arbitrary team's deployed harness. |
| Reusable agent environments | Actual NeMo Gym resource server/client, stress checks, eight-step A100 qualification and a frozen longer-run driver. OpenEnv's native service/SDK and TRL bridge pass CPU lifecycle and failure checks. Native Verifiers scripted CPU checks also pass. | Complete the longer NeMo and matched OpenEnv GPU resume qualifications, training and held-out agent/MGSM assessments. No new quality result is available. |
| Evaluate own checkpoints | Core local Transformers adapter with frozen model/tokenizer/template/sampling identities, deterministic scorers and matched comparisons; exercised on actual CPU and GPU checkpoints. | Additional task-specific scoring and broader deployment qualification. |
| Evaluate hosted endpoints | Optional real Inspect task adapter exposes Inspect's provider ecosystem; trusted Python callables can wrap an existing team system. The recorded Inspect integration uses its mock provider. | A supported direct endpoint recipe, actual provider qualification, timeout/retry/rate-limit policy and import of Inspect results into the core evidence format. Hosted endpoints have not been qualified end to end here. |
| Robustness | Fixed specifications, repeated-trial support, category gates, strict workflow checks and adversarial lifecycle/tool-dispatch tests. | Representative perturbations, retrieved-content injection, distribution shifts and task-level robustness suites. Infrastructure stress tests do not establish model robustness. |
| Calibration | No calibration evaluator implemented. | Define an observable confidence/abstention contract, Brier/ECE or selective-risk metrics, and an independently reviewed target dataset. Verbal confidence alone is not probability calibration. |
| Audit evidence | JSON/Markdown reports bind configurations, source/data/artifact hashes, per-item outcomes, usage/errors and comparison conditions. RL additionally records sampled tokens, context and external-token masks. | Retention/access controls, signatures or append-only audit storage, approvals and the documentation process required by a deployment. Current reports are research evidence, not AI Act compliance certification. |
| Evaluation service | A working local kit and optional SDK integration. | Managed isolated execution, authentication, queues, costs, confidentiality policies and operational support. No shared hosted service exists. |
| Two open benchmarks | Designs and reuse map for German RAG with cited evidence and business-communication agent sequences. Existing synthetic fixtures exercise the kit. | Expert-reviewed corpora/tasks, permission review, dedicated scorers, hidden assessment protocol, strong baselines and release conditions. These two benchmarks have not been released. |
Harnesses, environments and checkpoints
A checkpoint contains model weights. A harness constructs model requests, executes tools and controls the agent loop. An environment owns task state, observations and verification. A benchmark supplies tasks and a comparison protocol. These are separately versioned parts of an experiment.
The FineEnvs multi-harness Space is a worked research article linked to runnable training/evaluation code. Our current integration uses TRL's multi-turn loop with NeMo Gym's resource server; it does not reproduce that multi-harness result. The framework comparison explains the alternatives and the exact qualification boundary here.
When reporting the cited FineEnvs compute figure, distinguish the authors' two runs: two H100s for roughly46 hours for multi-harness training is about92 H100 GPU-hours; the single-harness run reports roughly32 hours on two H100s. Evaluation, queueing and restarts are excluded. Those figures are author reports, not our measurements or a budget prediction for German Qwen.
Current experiment and publication
The longer NeMo protocol preregisters 256 updates, DEV-only checkpoint selection and final agent/MGSM comparisons from the original German SFT parent. The NeMo and matched OpenEnv GPU attempts on 20 October stopped during the initial continuous run of resume qualification: the audit detected disagreement between the native tool stream and sampled assistant tokens, conditioning context or tool masks. Neither arm obtained a successful GPU resume-equivalence proof or reached the 256-step run and held-out assessment. There is no new longer-run quality result. These are engineering qualification failures, not measurements of longer-run model quality.
A new JAIF attempt requires verified local inputs, recreated pinned environments, fresh path-bound plans and a successful full-four-step versus stop-two/resume-four qualification for each backend. Preserve the original failed-run plans and evidence. The operations guide distinguishes current JAIF submissions from legacy allocation controllers; no queued continuation is implied by these historical attempts.
The initial private HF App and a private links collection are live. The account's storage restriction currently blocks repository resolve downloads, so the GitHub checkout is the execution route. Authentication and byte/visibility checks are recorded in publication evidence. A public institutional organization card is deferred until a separately authorized public launch.