Jobs and experiments on JAIF

Use short-lived commands for data preparation, tests, builds and evaluation. Each command should finish on its own, retain its outputs under the JAIF project and have a wall-clock limit. The retained interactive shell is for continuing development and requests zero GPU slots.

The project now includes Determined Core entrypoints for managed CPU work and native GRPO training. An actual zero-GPU CPU experiment passed checkpoint, pause and restore checks. The GPU training entrypoint is implemented; its end-to-end GPU qualification is still pending. The historical TRL pilot launchers remain shell/command routes, with their original run evidence preserved.

Bounded CPU utility command

jaif-cpu-command.yaml requests resources.slots: 0 from 42_Compute in the Determined JAIF workspace. The registered host mount and container mount both use /pfss/mlde/workspaces/mlde_wsp_JAIF. The source checkout belongs directly at /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes.

Keep Determined's bootstrap working directory at /run/determined/workdir. Change to the project directory in the command itself. This keeps bootstrap authentication files out of the shared workspace root.

After the JAIF checkout exists, this launches the authored evaluation fixture without a GPU, model, provider key or pre-existing Python virtual environment:

det command run --workspace-name JAIF \
  --config-file cluster/determined/jaif-cpu-command.yaml --detach \
  /usr/bin/timeout --signal=TERM --kill-after=30s 900s \
  /bin/bash -lc 'cd /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes &&
    PYTHONPATH=src exec python3 -m hessian_recipes demo \
      --output runs/jaif-cpu-fixture-01'

Run the submission command from a checkout containing the YAML. Use a fresh output directory each time, and retain the command ID printed by --detach. The task ends when the fixture exits, or after the timeout and termination grace period. No source context is uploaded implicitly by det command run; execution uses the explicit JAIF mount. For tests or optional tools, first create the appropriate environment under the JAIF project and invoke its Python explicitly. The command defaults disable GPUs and Hub downloads and bound common numerical thread pools. Jobs using Hub download APIs must deliberately override the offline flags.

This template and CLI were reviewed against Determined 0.35.0. This exact command has not been submitted or execution-qualified as part of the template change. The continuation shell's provisioning and environment setup are separate records.

CPU experiments and the cluster's resource model

The installed Determined client and connected master reported 0.35.0 on 21 October 2026. Its versioned resource schema allows resources.slots_per_trial: 0; the default is one slot. Experiment configurations use slots_per_trial, while commands and shells use slots, as explained in the versioned configuration reference.

Pool inspection found an active CUDA pool, 42_Compute, with auxiliary-container capacity, and no pool reporting CPU compute slots. The zero-slot managed CPU fixture ran in this pool without requesting a GPU. Such work uses auxiliary capacity rather than reserving dedicated CPU compute slots. A separately configured CPU pool can provide explicit CPU compute capacity; see the official master reference.

Observed managed CPU lifecycle

On 21 October, managed experiment 54512, in JAIF's hessian-recipes project (project 363), completed its eight-step deterministic CPU fixture. It paused after step three, published a scheduler checkpoint, and continued in a new allocation of the same trial 76039. The second allocation restored the exact scheduler checkpoint UUID and finished at step eight with state matching the uninterrupted reference. The scheduler recorded completion and eight checkpoints; no GPU was requested.

The saved source snapshot, launch configuration and proof remain under runs/continuation-20261021/managed-cpu-lifecycle-01/, including proof.json. This is evidence for the observed Core lifecycle fixture. Native GPU optimizer recovery, preemption and model-quality results require their own qualification.

The reusable CPU experiment configuration uses resources.slots_per_trial: 0, a single eight-step searcher operation and bounded execution. Its entrypoint is `python3 -m hessian_recipes.managed_trial cpu`. From the JAIF checkout, submit it to the existing project with:

det experiment create --project_id 363 \
  cluster/determined/jaif-managed-cpu.yaml

No model-definition directory is passed: the entrypoint reads the explicit shared JAIF mount. The saved live experiment used its own immutable source copy so later implementation changes could not alter the running lifecycle check.

Managed native training implementation

The managed training configuration requests one GPU in JAIF and invokes the managed training entrypoint:

.venv-gpu/bin/python -m hessian_recipes.managed_training \
  --plan /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes/runs/agent-gyms/jaif-managed-nemo-03/experiment-plan.json \
  --backend nemo \
  --service-python /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes/.venv-nemo-gym/bin/python \
  --max-wall-seconds 28800

This entrypoint runs inside a managed trial. Adapt the template's plan and service Python to a freshly prepared, source-bound experiment. The OpenEnv route uses --backend openenv and its qualified .venv-openenv/bin/python. Both routes require their exact completed native GPU resume-qualification proof, followed by a successful managed four-step recovery proof, before the main training run.

The managed qualification configurations run four optimizer steps with a 2400-second bound, using the verified native four-step configuration and changing only its output directory. The original 256-step configuration stays frozen.

Submit the relevant qualification configuration with det experiment create, omitting a model-definition directory. Pause after the first optimizer-step metric so the worker saves and stops at step two. Wait for its durable preempted_at_checkpoint receipt and scheduler PAUSED state, then activate the same experiment. The new allocation must restore the scheduler-selected step-two checkpoint. Exact final adapter, optimizer, scheduler, RNG and rollout evidence must match the native uninterrupted run. An uninterrupted managed run cannot satisfy this recovery gate. The proof also binds the installed Core implementation.

After the managed trial finishes its 256 steps, continue the original backend driver with --resume in a bounded evaluation job. It verifies and adopts the retained checkpoints at 64, 128 and 256, then performs the original DEV selection and final assessments. Scheduler snapshots retain earlier sealed checkpoints so recovery preserves that selection protocol. Neither pause nor restart renews the persistent deadline or original cumulative eight-hour GPU budget.

The pinned managed GPU requirements add Determined 0.35.0 to the existing native ML pins. The cluster installation extra supplies the same SDK pin for other environments. Both entrypoints explicitly select TensorboardMode.MANUAL. Core continues to report trial metrics and checkpoints over HTTP, while the native GPU environment requires no optional TensorBoard writer. A two-step zero-GPU trial in the exact minimal image verified this path (experiment 54514, trial 76041).

Historical evidence from 2026-10-21: source c6288f6 passed native 02 qualification for both backends, and managed NeMo experiment 54515 passed. Managed OpenEnv experiment 54516 was canceled after the scheduler selected an earlier of two checkpoint UUIDs published at the same optimizer step, which differed from the durable preemption receipt. The full failed artifacts are preserved privately. These checks produced no long-run quality outcome. The publication fix changes consumed source, so both backends require fresh 03 native and managed qualification before sustained training.

The implementation:

• Reports native training metrics and optimizer progress to the Determined trial.
• Publishes sealed native adapter, optimizer, scheduler and RNG state, together

with the task/run identity and exact evidence prefixes, as scheduler snapshots.

• Restores the scheduler-selected checkpoint UUID and verifies its source,

configuration and native seals before adopting local output.

• Publishes intermediate periodic saves after sealing. Preemption and final

saves publish once after native cleanup, owned-service shutdown and the final manifest; only then can preemption be acknowledged. Repeated or decreasing checkpoint steps are rejected before creating another scheduler UUID.

• Reuses an already selected terminal checkpoint UUID after validation. Any

reconstructed local manifest metadata is recorded locally, without publishing another snapshot at that optimizer step.

• Preserves policy RNG around Core reporting and storage calls, and retains the

original deadline and cumulative GPU budget across allocations.

The Core snapshot adapter and the native trainer hooks implement these paths. DEV-only checkpoint selection, held-out agent assessment and MGSM retention checks remain in the original experiment driver; a managed training completion is not a model-quality result.

GPU qualification remains pending. Before sustained managed training, require both the exact sampled-token/uninterrupted-versus-resumed native proof and an actual managed GPU checkpoint/preemption/restart check. The completed CPU fixture establishes neither guarantee for the GPU trainer.