# Validation evidence

The repository now includes completed CPU checks and bounded A100 training runs.
The initial JAIF CPU environment check on 21 October passed **500 tests**, with
eight optional checks skipped, using `.[dev,inspect]`. Subsequent deployment
checks are included in the [current CI runs](https://github.com/ChTauchmann/hessian-recipes/actions/workflows/ci.yml). Ruff, the offline demo,
package builds and the recipe website build also pass from the JAIF checkout.
Earlier checks in the pinned ML environments passed the upstream scalar
optimizer/RNG roundtrip and four saved BF16 tensor audit cases; **five optional
Verifiers tests** passed separately. Those earlier checks do not qualify the
recreated JAIF GPU environments. A fresh install and CPU demo were also checked
from the earlier HF source snapshot.
These counts describe software checks, not model accuracy or release readiness.
Use the [documentation index](README.md) to find protocols, results and limitations.

## Current measured scope

| Completed work | Evidence and practical limit |
| --- | --- |
| Private Hugging Face reading experience, 21 October | The research overview, methods index, source-bound interactive results and article reader, using official hessian.AI branding, were checked in Chromium at 62 desktop/mobile page-viewports. Final automated accessibility checks report zero violations; targeted 320/768px reflow, keyboard scrolling, code copy, search/filtering, mobile navigation, contents links and no-JavaScript reading pass. These are UI checks, with no model inference. See [interface evidence](huggingface-ui-validation.json). |
| German Qwen SFT | LoRA SFT completed 128 optimizer steps on 1,024 German explanation examples, followed by merge and matched evaluation on 64 MGSM German items. Strict final-answer protocol accuracy changed from 28/64 to 30/64; the paired result is inconclusive. A ten-item agent inspection supports a language/format change, not German fluency or improved arithmetic. See [results](use-cases/german-reasoning-results.md) and [aggregate evidence](use-cases/german-reasoning-results.json). |
| Local multi-turn agent RL | Native TRL GRPO completed 16 steps and 64 training rollouts with exact sampled-token/context checks and external-token loss masking. Matched strict workflow success changed from 4/32 to 5/32; MGSM retention fell from 30/64 to 29/64 and failed the zero-drop gate. This qualifies an executable path, not improved general agent ability or a release. See [results](use-cases/german-agent-rl-results.md) and [aggregate evidence](use-cases/german-agent-rl-results.json). |
| Native NeMo Gym online training | The actual pinned resource server and remote client completed eight A100 GRPO steps and 32 rollouts from the original German SFT checkpoint. All 392 trainable adapter tensors changed; independent replay checked 71 tool observations and masks for 7,955 assistant and 1,648 external tokens. Strict training success was 5/32. No held-out agent or MGSM retention evaluation was performed for this integration qualification. See [gym protocol and limits](agent-gyms.md) and [aggregate evidence](agent-gyms-validation.json). |
| Native gym CPU checks | NeMo passed 11 service and nine client checks. Prime Verifiers passed seven scripted native lifecycle scenarios and five optional tests. The Verifiers fixture does not perform model inference or training; its Prime-RL GPU path remains unqualified. See [gym evidence](agent-gyms-validation.json). |
| NeMo stress and recovery checks, 20 October | Actual pinned service: 32 concurrent isolated episodes, 260 sequential reset/solve episodes, the 256-active-session bound and cleanup recovery, service restart with sticky old-session failure and independent fresh recovery. Actual TRL tool dispatch was exercised with scripted outputs; invalid dispatch can retain partial reward while strict success fails. No model inference or learning. See [aggregate stress evidence](nemo-gym-stress-validation.json). |
| Native OpenEnv CPU qualification, 20 October | Twelve actual SDK/service checks passed, including 32 simultaneously open WebSocket sessions, rejection/recovery at the configured 32-session bound, 260 sequential resets, malformed requests and poisoned Python environment isolation. The actual SDK client passed strict tool/replay/isolation checks and sticky service-loss handling; 24 native tests and 27 client/wrapper tests pass separately. The 32-test matched driver and 21-test offline queue controller pass. The later 20 October GPU attempt stopped at the strict token audit; GPU resume equivalence and model assessment remain unqualified. See [aggregate evidence](openenv-validation.json) and [matched protocol](use-cases/german-openenv-comparison.md). |
| Longer NeMo/OpenEnv attempts, status 21 October | Fixed-horizon native resume, deterministic DEV selection and CPU optimizer/scheduler/RNG checks pass. Both 20 October GPU attempts failed during the initial continuous resume-qualification run at the strict sampled-token/context/tool-mask audit. Neither established GPU resume equivalence or reached 256-step training and held-out assessment. New JAIF plans and runtime qualification are required before another attempt. See [NeMo protocol](use-cases/german-nemo-long-run.md) and [matched OpenEnv arm](use-cases/german-openenv-comparison.md). |

The three completed GPU pilots retain private weights and raw evidence under ignored `runs/`
directories; committed results contain aggregate measurements and evidence hashes.
The NeMo qualification does not change the earlier agent-RL quality scores or
resolve its retention regression. No adapter or model has been published, no
Hugging Face organization has been established by this work, and no hosted
evaluation service is implemented. The private personal Hugging Face recipe App
and links-only collection are live; [publication evidence](huggingface-publication.json)
records access and source checks, including the repository-download quota limit.
No models or datasets are published by this workflow. The two development tracks remain provisional
synthetic fixtures; robotics, Bio/Med and materials remain [sketches](domains.md).

## Historical foundation checks

The initial foundation was checked at commit `8bc44ee` on 15 October 2026,
Linux x86-64, Python 3.11.9. Its **73-test** snapshot includes actual Inspect mock
execution, Ruff, package builds and dependency consistency checks. This historical
count is retained in [validation.json](validation.json); it is not the current
suite total or a record of the subsequent GPU runs.

| Check | Evidence and practical limit |
| --- | --- |
| Core CLI and unit tests | Offline replay, coverage/error handling, integrity checks, compatible comparisons, category regression rejection, filtering, trajectory validation, and candidate-bundle verification. |
| Real Inspect execution | The actual pinned Inspect SDK executes the task with a local mock provider; controlled outputs score correctly. No external model API is called. |
| Teacher generation | Transformers generates locally from a random tiny GPT-2 checkpoint and saved tokenizer. Outputs and input identities are recorded; no teacher-quality claim. |
| Actual SFT step | TRL completes a step and saves/reloads a checkpoint; parameters differ from the initial model. Recorded loss approximately 2.72 and gradient norm 3.14. |
| Actual GRPO step | TRL samples 32 one-token completions with varied rewards; reward mean 0.03125, reward standard deviation approximately 0.1768, gradient norm approximately 1.528, and parameters change. The initial scalar loss is zero because group advantages average to zero; it does not imply a zero gradient. |
| Local evaluation | Real CPU generation from baseline and SFT checkpoints uses one frozen tokenizer. Complete execution and compatible evidence are checked. Tiny random-model outputs are not useful task results. |
| Agent Lightning pilot | Actual pinned upstream checkout passes source/API checks; the bounded external tool loop passes mocked tests. No gateway, Ray training, GPU stack, or distributed optimizer run was executed. |
| Packaging | Python source distribution and wheel build successfully; the CLI is installed in isolated environments. |
| Slurm script | Bash syntax is checked. No cluster job is submitted. |

The machine-readable [foundation summary](validation.json) records that historical
checkpoint's counts, source identities and selected measurements. Model weights
and generated logs are not added to Git. The dependency snapshots in
`requirements/training-cpu.txt` and
`requirements/inspect.txt` describe the separate environments, including developer
tools. They are observed exact-version captures, not cryptographic package locks.

## Repeat the checks

Use a clean environment from the repository root:

```bash
python -m pip install -e '.[dev]'
python -m pytest -q
python -m ruff check .
hessian-recipes demo --output runs/replay-check
python -m build
bash -n scripts/slurm-train.sbatch
```

The core suite skips the Inspect integration when its optional dependency is not
installed. In a separate environment, install `.[dev,inspect]` and rerun the suite
to exercise the actual mock-model integration. To recreate the observed Inspect
environment, use `pip install -r requirements/inspect.txt -e .`.

For the actual CPU training and local inference path:

```bash
python -m venv .venv-training
.venv-training/bin/python -m pip install -r requirements/training-cpu.txt -e .
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
  .venv-training/bin/python examples/training/cpu_smoke.py --output runs/cpu-check
HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 \
  .venv-training/bin/python examples/evaluation_cpu_smoke.py \
  --baseline runs/cpu-check/tiny-model \
  --candidate runs/cpu-check/sft/checkpoint \
  --tokenizer runs/cpu-check/sft/checkpoint \
  --output runs/evaluation-cpu-check
```

Choose fresh output paths. The local evaluation example permits a full score drop
deliberately: it tests checkpoint/tokenizer compatibility and execution coverage,
not a quality threshold. Production thresholds require representative tasks.

The commands above reproduce foundation checks. The actual GPU protocols and
their separate dependency environments are documented in the
[German SFT use case](use-cases/german-reasoning.md),
[local agent-RL use case](use-cases/german-agent-rl.md) and [RL agent gyms](agent-gyms.md).
Use their bounded budgets and fresh output paths when reproducing those runs.

Still unqualified: the generic Slurm/container recipes at a deployment site,
actual model checkpoint recovery after interruption, multi-GPU or multi-node jobs, a real
Lightning/team-harness training update, Prime-RL training through Verifiers,
expert benchmark validity, native domain models and hosted evaluation services.
The GPU runs record image tags but lack immutable image digests.
Useful adaptation with retained capabilities and production release conditions
remain acceptance targets in the [roadmap](roadmap.md).