Container and cluster qualification

The project default is the Determined JAIF workspace (ID 123), with code, models, datasets, environments and new run outputs under /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes. The YAML files use /pfss/mlde/workspaces/mlde_wsp_JAIF for both the host storage source and the container mount path. A compute workspace selection and a filesystem mount are separate settings; use both.

Verified JAIF checkout

The 21 October CPU migration passed the .[dev,inspect] suite, Ruff, offline demo, package build and website build in a fresh environment. After adopting the full registered JAIF path, the core .venv, .venv-gpu, .venv-nemo-gym and .venv-openenv environments were recreated there. Core and GPU Python use the persistent managed 3.11.17 interpreter; the native services use Python 3.13.14. Their import and interpreter paths bind the JAIF project. The old core environment was archived rather than reused at its former alias.

The recreated service environments passed 11 native NeMo service checks, nine NeMo client checks and 12 native OpenEnv checks. These CPU results and the package/path receipts stay under ignored runs/continuation-20261021/environment-setup-03/; original migration receipts stay under runs/migrations/. GPU training and recovery qualification remain separate requirements.

The local inputs for the next experiments are:

  • .cache/models/Qwen3-1.7B: the original pretrained snapshot.
  • runs/german-qwen/pilot-02/merged-checkpoint: the original German SFT parent

for both agent-RL arms.

  • runs/german-qwen/pilot-02/frozen-tokenizer, train.jsonl and dev.jsonl,

plus runs/german-qwen/data: frozen tokenizer and source-disjoint data.

Use the migration's byte/tree-hash checks before consuming those artifacts, and recreate environments from their documented pins when needed. Managed GPU execution adds the SDK pins in the managed environment snapshot; its installation retained the existing native ML package versions. The completed CPU checks do not qualify a new GPU training or recovery run. Keep UV_PYTHON_INSTALL_DIR and UV_CACHE_DIR under the JAIF project so managed interpreters and virtual environments remain usable on another worker. The private .env sets those paths and the existing account authentication, including the cluster's public CA trust configuration. Activate the core environment after sourcing it when running the local kit:

. .venv/bin/activate

JAIF shell and launch defaults

det shell start --workspace-name JAIF --detach \
  --config-file cluster/determined/jaif-shell.yaml

The saved configuration requests zero GPUs and gives fresh shells three mounts, using the same full path on host and container:

  • /pfss/mlde/workspaces/mlde_wsp_JAIF
  • /pfss/mlde/workspaces/mlde_wsp_HessianLLM
  • /pfss/mlde/workspaces/mlde_wsp_KIServiceCenter

A bounded CPU mount check confirmed all three were accessible and the short JAIF alias was absent; its receipt is runs/continuation-20261021/mount-check.json. Existing containers keep the mounts from their own launch configuration. Project commands, environments and new outputs use the JAIF checkout above. Keep the user's retained continuation shell running until they ask to stop it. Determined bootstraps its internal shell state in /run/determined/workdir; after connecting, run project commands from:

cd /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes
set -a
. /pfss/mlde/workspaces/mlde_wsp_JAIF/tauchmann/.env
set +a

The personal tauchmann directory and .env must be owned by the user with mode 0700, as requested. The .env stays outside the repository. Supply credentials over encrypted stdin; never put values in launch YAML, command arguments, logs, source, build artifacts or publication receipts.

The three GPU configurations also bootstrap in /run/determined/workdir. This keeps Determined's working-directory fallback for native authentication cache and lock files outside the checkout. After connecting, explicitly change to /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes; command entrypoints must do the same. Both legacy GPU workers already set that project working directory before training. Personal persistent credentials remain in tauchmann/.env outside the checkout; native task bootstrap state is separate from that file.

The SFT, agent RL and OpenEnv command configurations use the same JAIF mount and project root, and request one GPU. Launch them only after the local checkpoint, datasets and recreated pinned environments are verified in JAIF. They do not mount another project's storage. The OpenEnv launcher passes --workspace-name JAIF explicitly.

The two queue-controller scripts are legacy, opt-in recovery tools. They have no historical allocation default: a NeMo controller launch requires --shell-id (or HESSIAN_NEMO_SHELL_ID), and the chained OpenEnv controller requires --predecessor-shell-id (or the same environment variable). They refuse an unset allocation before cluster contact and reject predecessor shells outside JAIF. See JAIF commands and managed experiments for the current submission boundary. Never give the continuation shell to a GPU controller: those controllers release their selected GPU allocation when the bounded job ends. New standard submissions should use the JAIF command configuration or a qualified managed experiment launcher rather than replaying an old controller.

Both legacy controllers also require the untracked .cache/agent-rl-site.tar archive, pinned to SHA-256 289ecc42e53d3fa6ac5b8532b447977f53e571837fbc4fae133ba42cf5437cfe, and GPU libraries under .venv-gpu/lib/python3.11/site-packages/torch/lib, nvidia and triton. A clean source checkout and recreated environments do not recreate that historical archive. Until the exact archive and libraries are staged and verified in JAIF, those controllers are not ready to launch there. Do not weaken the archive check or copy old virtual environments to bypass it. The chained OpenEnv route additionally needs the matching predecessor NeMo controller's acknowledged queue-launcher-status.json; a different launcher does not supply that release journal automatically.

The historical pilots run TRL inside Determined shells or commands. The project now also includes managed Core entrypoints for trial metrics, scheduler snapshots, verified restoration and preemption. Managed CPU experiment 54512 (JAIF project 363, trial 76039) passed pause-after-three, scheduler-checkpoint restoration in a new allocation and completion at step eight. Its saved proof is runs/continuation-20261021/managed-cpu-lifecycle-01/proof.json.

The CPU experiment and one-GPU training configurations select JAIF explicitly. GPU recovery and preemption qualification remain pending for the managed training path; the CPU lifecycle result is not a GPU or model quality result. See the jobs guide for entrypoints and the exact qualification boundary.

Actual German SFT, agent RL, and NeMo Gym bridge runs used a single A100-SXM4-80GB. Those historical observations do not qualify a new JAIF run, the generic Docker build, or the Slurm script below. The observed container tag lacked a captured immutable digest. Preserve historical manifests and absolute-path-bound plans; prepare new plans for the JAIF root rather than editing old provenance. Recreate virtual environments there because interpreter links, editable installs and script shebangs can bind their original path. Confirm the exact-token and resume qualification before any further long training.

Core container

docker build -f containers/Dockerfile -t hessian-recipes:dev .
docker run --rm hessian-recipes:dev

The default image exercises offline fixture replay and runs as an unprivileged user. The /results directory holds reports. Bind-mount a directory writable by UID 10001 there to retain them. To include Inspect use --build-arg EXTRAS=inspect; --build-arg EXTRAS=training installs the heavier pinned training stack. Installing PyTorch dependencies does not qualify a GPU execution environment: validate the host driver, runtime, CUDA libraries, and selected model separately.

The Python base tag floats within 3.11. For a qualified deployment, replace it with the observed image digest, retain the build log and dependency freeze, scan that actual image, and attach its digest to the run evidence. The container is a packaging example until its build and execution are recorded in validation evidence.

Slurm example

sbatch --partition=YOUR_PARTITION \
  --export=ALL,HESSIAN_PYTHON=/absolute/venv/bin/python \
  scripts/slurm-train.sbatch /absolute/copied-recipe.json

The example requests one GPU and runs a normal installed Python recipe. Adapt the resource directives, account, partition, storage paths, and module setup to the site. It is not a distributed launcher or a site-tested cluster profile. Review hessian-recipes train CONFIG --dry-run before submission and stage all model files locally; worker downloads are disabled by default.

Before labelling a recipe cluster-supported, record: exact package environment, hardware/driver, model and data identities, successful run and checkpoint reload, matched-budget baseline and adaptation evaluation, restart behavior, peak memory, and throughput. Multi-GPU and multi-node support each need their own tests. Real tool-using agents additionally need environment isolation, resource limits, and a verifier inaccessible to the evaluated agent; the in-memory development fixture does not supply that deployment isolation.