# Post-training and evaluation programme: implementation status

Documentation and recipe instructions are in English. German is the language of
the German-model use case and proposed German RAG tasks. This page maps the
programme proposal to executable work and remaining validation, as of 21 October
2026. The main reading surface is the [private Hugging Face recipe hub](https://huggingface.co/spaces/ctauchmann/hessian-recipes).
Our model weights, downloaded data and raw run artifacts remain on the cluster.

Publication update, 28 October 2026: the site is named **hessian AI & JAIF** and
uses Protected visibility. Its [public App](https://ctauchmann-hessian-recipes.static.hf.space/)
includes the recipe pages and source downloads; the Hugging Face repository and
GitHub collaboration repository remain private. Earlier publication records below
describe the initial Private deployment.

| Programme element | Available now | Remaining work |
| --- | --- | --- |
| Post-train a small reasoning model | Executable pinned TRL SFT/GRPO recipes, source checks, local checkpoint merging and before/after evaluation; real German Qwen3-1.7B SFT and bounded agent-RL GPU pilots. | Demonstrate useful adaptation with retained capabilities. Existing measured gains are inconclusive and the earlier RL candidate failed its observed MGSM retention gate. |
| Distill a large teacher | Local teacher generation, deterministic answer verification, filtering and SFT-ready sequence export. A tiny random-model CPU run checks wiring. | A substantive large-teacher/student experiment, source/output permissions and a matched quality/compute comparison. We do not claim demonstrated distillation benefit. |
| Train in an existing team's harness | Trajectory contract, callable example and optional Agent Lightning/verl launcher, with source/API and mocked checks. | An actual team-harness GPU update and matched held-out evaluation. The native TRL tool loop tested here is not an arbitrary team's deployed harness. |
| Reusable agent environments | Actual NeMo Gym resource server/client, stress checks, eight-step A100 qualification and a frozen longer-run driver. OpenEnv's native service/SDK and TRL bridge pass CPU lifecycle and failure checks. Native Verifiers scripted CPU checks also pass. | Complete the longer NeMo and matched OpenEnv GPU resume qualifications, training and held-out agent/MGSM assessments. No new quality result is available. |
| Evaluate own checkpoints | Core local Transformers adapter with frozen model/tokenizer/template/sampling identities, deterministic scorers and matched comparisons; exercised on actual CPU and GPU checkpoints. | Additional task-specific scoring and broader deployment qualification. |
| Evaluate hosted endpoints | Optional real Inspect task adapter exposes Inspect's provider ecosystem; trusted Python callables can wrap an existing team system. The recorded Inspect integration uses its mock provider. | A supported direct endpoint recipe, actual provider qualification, timeout/retry/rate-limit policy and import of Inspect results into the core evidence format. Hosted endpoints have not been qualified end to end here. |
| Robustness | Fixed specifications, repeated-trial support, category gates, strict workflow checks and adversarial lifecycle/tool-dispatch tests. | Representative perturbations, retrieved-content injection, distribution shifts and task-level robustness suites. Infrastructure stress tests do not establish model robustness. |
| Calibration | No calibration evaluator implemented. | Define an observable confidence/abstention contract, Brier/ECE or selective-risk metrics, and an independently reviewed target dataset. Verbal confidence alone is not probability calibration. |
| Audit evidence | JSON/Markdown reports bind configurations, source/data/artifact hashes, per-item outcomes, usage/errors and comparison conditions. RL additionally records sampled tokens, context and external-token masks. | Retention/access controls, signatures or append-only audit storage, approvals and the documentation process required by a deployment. Current reports are research evidence, not AI Act compliance certification. |
| Evaluation service | A working local kit and optional SDK integration. | Managed isolated execution, authentication, queues, costs, confidentiality policies and operational support. No shared hosted service exists. |
| Two open benchmarks | [Designs and reuse map](benchmark-designs.md) for German RAG with cited evidence and business-communication agent sequences. Existing synthetic fixtures exercise the kit. | Expert-reviewed corpora/tasks, permission review, dedicated scorers, hidden assessment protocol, strong baselines and release conditions. These two benchmarks have not been released. |

## Harnesses, environments and checkpoints

A checkpoint contains model weights. A harness constructs model requests,
executes tools and controls the agent loop. An environment owns task state,
observations and verification. A benchmark supplies tasks and a comparison
protocol. These are separately versioned parts of an experiment.

The [FineEnvs multi-harness Space](https://huggingface.co/spaces/FineEnvs/multi-harness-rl)
is a worked research article linked to runnable training/evaluation code. Our
current integration uses TRL's multi-turn loop with NeMo Gym's resource server;
it does not reproduce that multi-harness result. The
[framework comparison](environment-frameworks.md) explains the alternatives and
the exact qualification boundary here.

When reporting the cited FineEnvs compute figure, distinguish the authors' two
runs: two H100s for roughly46 hours for multi-harness training is about92 H100
GPU-hours; the single-harness run reports roughly32 hours on two H100s. Evaluation,
queueing and restarts are excluded. Those figures are author reports, not our
measurements or a budget prediction for German Qwen.

## Current experiment and publication

The [longer NeMo protocol](use-cases/german-nemo-long-run.md) preregisters 256
updates, DEV-only checkpoint selection and final agent/MGSM comparisons from the
original German SFT parent. The NeMo and
[matched OpenEnv](use-cases/german-openenv-comparison.md) GPU attempts on
20 October stopped during the initial continuous run of resume qualification:
the audit detected disagreement between the native tool stream and sampled
assistant tokens, conditioning context or tool masks. Neither arm obtained a
successful GPU resume-equivalence proof or reached the 256-step run and held-out
assessment. There is no new longer-run quality result. These are engineering
qualification failures, not measurements of longer-run model quality.

A new JAIF attempt requires verified local inputs, recreated pinned environments,
fresh path-bound plans and a successful full-four-step versus stop-two/resume-four
qualification for each backend. Preserve the original failed-run plans and
evidence. The [operations guide](operations.md) distinguishes current JAIF
submissions from legacy allocation controllers; no queued continuation is implied
by these historical attempts.

The initial private HF App and a [private links collection](https://huggingface.co/collections/ctauchmann/hessian-recipes-private-research-workspace-6ad7417d810d666542981033)
are live. The account's storage restriction currently blocks repository resolve
downloads, so the [GitHub checkout](https://github.com/ChTauchmann/hessian-recipes)
is the execution route. Authentication and byte/visibility checks are recorded
in [publication evidence](huggingface-publication.json). A public institutional
organization card is deferred until a separately authorized public launch.