# Jobs and experiments on JAIF

Use short-lived commands for data preparation, tests, builds and evaluation.
Each command should finish on its own, retain its outputs under the JAIF project
and have a wall-clock limit. The retained interactive shell is for continuing
development and requests zero GPU slots.

The project now includes Determined Core entrypoints for managed CPU work and
native GRPO training. An actual zero-GPU CPU experiment passed checkpoint,
pause and restore checks. The GPU training entrypoint is implemented; its
end-to-end GPU qualification is still pending. The historical TRL pilot launchers
remain shell/command routes, with their original run evidence preserved.

## Bounded CPU utility command

[jaif-cpu-command.yaml](../cluster/determined/jaif-cpu-command.yaml) requests
`resources.slots: 0` from `42_Compute` in the Determined `JAIF` workspace. The
registered host mount and container mount both use
`/pfss/mlde/workspaces/mlde_wsp_JAIF`. The source checkout belongs directly
at `/pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes`.

Keep Determined's bootstrap working directory at `/run/determined/workdir`.
Change to the project directory in the command itself. This keeps bootstrap
authentication files out of the shared workspace root.

After the JAIF checkout exists, this launches the authored evaluation fixture
without a GPU, model, provider key or pre-existing Python virtual environment:

```bash
det command run --workspace-name JAIF \
  --config-file cluster/determined/jaif-cpu-command.yaml --detach \
  /usr/bin/timeout --signal=TERM --kill-after=30s 900s \
  /bin/bash -lc 'cd /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes &&
    PYTHONPATH=src exec python3 -m hessian_recipes demo \
      --output runs/jaif-cpu-fixture-01'
```

Run the submission command from a checkout containing the YAML. Use a fresh
output directory each time, and retain the command ID printed by `--detach`.
The task ends when the fixture exits, or after the timeout and termination grace
period. No source context is uploaded implicitly by `det command run`; execution
uses the explicit JAIF mount. For tests or optional tools, first create the
appropriate environment under the JAIF project and invoke its Python explicitly.
The command defaults disable GPUs and Hub downloads and bound common numerical
thread pools. Jobs using Hub download APIs must deliberately override the offline
flags.

This template and CLI were reviewed against Determined 0.35.0. This exact command
has not been submitted or execution-qualified as part of the template change.
The continuation shell's provisioning and environment setup are separate records.

## CPU experiments and the cluster's resource model

The installed Determined client and connected master reported **0.35.0** on
21 October 2026. Its [versioned resource schema](https://github.com/determined-ai/determined/blob/0.35.0/schemas/expconf/v0/resources.json)
allows `resources.slots_per_trial: 0`; the default is one slot. Experiment
configurations use `slots_per_trial`, while commands and shells use `slots`, as
explained in the [versioned configuration reference](https://github.com/determined-ai/determined/blob/0.35.0/docs/reference/experiment-config-reference.rst).

Pool inspection found an active CUDA pool, `42_Compute`, with auxiliary-container
capacity, and no pool reporting CPU compute slots. The zero-slot managed CPU
fixture ran in this pool without requesting a GPU. Such work uses auxiliary
capacity rather than reserving dedicated CPU compute slots. A separately
configured CPU pool can provide explicit CPU compute capacity; see the
[official master reference](https://hpe-mlde.determined.ai/reference/deploy/master-config-reference.html).

## Observed managed CPU lifecycle

On 21 October, managed **experiment 54512**, in JAIF's `hessian-recipes` project
(**project 363**), completed its eight-step deterministic CPU fixture. It paused
after step three, published a scheduler checkpoint, and continued in a new
allocation of the same **trial 76039**. The second allocation restored the exact
scheduler checkpoint UUID and finished at step eight with state matching the
uninterrupted reference. The scheduler recorded completion and eight checkpoints;
no GPU was requested.

The saved source snapshot, launch configuration and proof remain under
`runs/continuation-20261021/managed-cpu-lifecycle-01/`, including `proof.json`.
This is evidence for the observed Core lifecycle fixture. Native GPU optimizer
recovery, preemption and model-quality results require their own qualification.

The reusable [CPU experiment configuration](../cluster/determined/jaif-managed-cpu.yaml)
uses `resources.slots_per_trial: 0`, a single eight-step searcher operation and
bounded execution. Its entrypoint is
[`python3 -m hessian_recipes.managed_trial cpu`](../src/hessian_recipes/managed_trial.py).
From the JAIF checkout, submit it to the existing project with:

```bash
det experiment create --project_id 363 \
  cluster/determined/jaif-managed-cpu.yaml
```

No model-definition directory is passed: the entrypoint reads the explicit shared
JAIF mount. The saved live experiment used its own immutable source copy so
later implementation changes could not alter the running lifecycle check.

## Managed native training implementation

The [managed training configuration](../cluster/determined/jaif-managed-training.yaml)
requests one GPU in JAIF and invokes
[the managed training entrypoint](../src/hessian_recipes/managed_training.py):

```bash
.venv-gpu/bin/python -m hessian_recipes.managed_training \
  --plan /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes/runs/agent-gyms/jaif-managed-nemo-03/experiment-plan.json \
  --backend nemo \
  --service-python /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes/.venv-nemo-gym/bin/python \
  --max-wall-seconds 28800
```

This entrypoint runs inside a managed trial. Adapt the template's plan and service
Python to a freshly prepared, source-bound experiment. The OpenEnv route uses
`--backend openenv` and its qualified `.venv-openenv/bin/python`. Both routes
require their exact completed native GPU resume-qualification proof, followed by
a successful managed four-step recovery proof, before the main training run.

The managed qualification configurations run four optimizer steps with a
2400-second bound, using the verified native four-step configuration and changing
only its output directory. The original 256-step configuration stays frozen.

| Backend | Managed recovery qualification | Main training |
| --- | --- | --- |
| NeMo | [Four-step configuration](../cluster/determined/jaif-managed-qualification.yaml) | [256-step configuration](../cluster/determined/jaif-managed-training.yaml) |
| OpenEnv | [Four-step configuration](../cluster/determined/jaif-managed-qualification-openenv.yaml) | [256-step configuration](../cluster/determined/jaif-managed-openenv.yaml) |

Submit the relevant qualification configuration with `det experiment create`,
omitting a model-definition directory. Pause after the first optimizer-step
metric so the worker saves and stops at step two. Wait for its durable
`preempted_at_checkpoint` receipt and scheduler `PAUSED` state, then activate the
same experiment. The new allocation must restore the scheduler-selected step-two
checkpoint. Exact final adapter, optimizer, scheduler, RNG and rollout evidence
must match the native uninterrupted run. An uninterrupted managed run cannot
satisfy this recovery gate. The proof also binds the installed Core implementation.

After the managed trial finishes its 256 steps, continue the original backend
driver with `--resume` in a bounded evaluation job. It verifies and adopts the
retained checkpoints at 64, 128 and 256, then performs the original DEV selection
and final assessments. Scheduler snapshots retain earlier sealed checkpoints so
recovery preserves that selection protocol. Neither pause nor restart renews the
persistent deadline or original cumulative eight-hour GPU budget.

The pinned [managed GPU requirements](../requirements/german-qwen-managed-cu126.txt)
add Determined **0.35.0** to the existing native ML pins. The
`cluster` installation extra supplies the same SDK pin for other environments.
Both entrypoints explicitly select `TensorboardMode.MANUAL`. Core continues to
report trial metrics and checkpoints over HTTP, while the native GPU environment
requires no optional TensorBoard writer. A two-step zero-GPU trial in the exact
minimal image verified this path (experiment 54514, trial 76041).

Historical evidence from 2026-10-21: source c6288f6 passed native 02
qualification for both backends, and managed NeMo experiment **54515** passed.
Managed OpenEnv experiment **54516** was canceled after the scheduler selected
an earlier of two checkpoint UUIDs published at the same optimizer step, which
differed from the durable preemption receipt. The full failed artifacts are
preserved privately. These checks produced no long-run quality outcome. The
publication fix changes consumed source, so both backends require fresh 03
native and managed qualification before sustained training.

The implementation:

- Reports native training metrics and optimizer progress to the Determined trial.
- Publishes sealed native adapter, optimizer, scheduler and RNG state, together
  with the task/run identity and exact evidence prefixes, as scheduler snapshots.
- Restores the scheduler-selected checkpoint UUID and verifies its source,
  configuration and native seals before adopting local output.
- Publishes intermediate periodic saves after sealing. Preemption and final
  saves publish once after native cleanup, owned-service shutdown and the final
  manifest; only then can preemption be acknowledged. Repeated or decreasing
  checkpoint steps are rejected before creating another scheduler UUID.
- Reuses an already selected terminal checkpoint UUID after validation. Any
  reconstructed local manifest metadata is recorded locally, without publishing
  another snapshot at that optimizer step.
- Preserves policy RNG around Core reporting and storage calls, and retains the
  original deadline and cumulative GPU budget across allocations.

The [Core snapshot adapter](../src/hessian_recipes/managed_trial.py) and the
native trainer hooks implement these paths. DEV-only checkpoint selection,
held-out agent assessment and MGSM retention checks remain in the original
experiment driver; a managed training completion is not a model-quality result.

**GPU qualification remains pending.** Before sustained managed training, require
both the exact sampled-token/uninterrupted-versus-resumed native proof and an
actual managed GPU checkpoint/preemption/restart check. The completed CPU fixture
establishes neither guarantee for the GPU trainer.