# Container and cluster qualification The project default is the Determined **JAIF** workspace (ID 123), with code, models, datasets, environments and new run outputs under `/pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes`. The YAML files use `/pfss/mlde/workspaces/mlde_wsp_JAIF` for both the host storage source and the container mount path. A compute workspace selection and a filesystem mount are separate settings; use both. ## Verified JAIF checkout The 21 October CPU migration passed the `.[dev,inspect]` suite, Ruff, offline demo, package build and website build in a fresh environment. After adopting the full registered JAIF path, the core `.venv`, `.venv-gpu`, `.venv-nemo-gym` and `.venv-openenv` environments were recreated there. Core and GPU Python use the persistent managed 3.11.17 interpreter; the native services use Python 3.13.14. Their import and interpreter paths bind the JAIF project. The old core environment was archived rather than reused at its former alias. The recreated service environments passed 11 native NeMo service checks, nine NeMo client checks and 12 native OpenEnv checks. These CPU results and the package/path receipts stay under ignored `runs/continuation-20261021/environment-setup-03/`; original migration receipts stay under `runs/migrations/`. GPU training and recovery qualification remain separate requirements. The local inputs for the next experiments are: - `.cache/models/Qwen3-1.7B`: the original pretrained snapshot. - `runs/german-qwen/pilot-02/merged-checkpoint`: the original German SFT parent for both agent-RL arms. - `runs/german-qwen/pilot-02/frozen-tokenizer`, `train.jsonl` and `dev.jsonl`, plus `runs/german-qwen/data`: frozen tokenizer and source-disjoint data. Use the migration's byte/tree-hash checks before consuming those artifacts, and recreate environments from their documented pins when needed. Managed GPU execution adds the SDK pins in [the managed environment snapshot](../requirements/german-qwen-managed-cu126.txt); its installation retained the existing native ML package versions. The completed CPU checks do not qualify a new GPU training or recovery run. Keep `UV_PYTHON_INSTALL_DIR` and `UV_CACHE_DIR` under the JAIF project so managed interpreters and virtual environments remain usable on another worker. The private `.env` sets those paths and the existing account authentication, including the cluster's public CA trust configuration. Activate the core environment after sourcing it when running the local kit: ```bash . .venv/bin/activate ``` ## JAIF shell and launch defaults ```bash det shell start --workspace-name JAIF --detach \ --config-file cluster/determined/jaif-shell.yaml ``` The saved configuration requests **zero GPUs** and gives fresh shells three mounts, using the same full path on host and container: - `/pfss/mlde/workspaces/mlde_wsp_JAIF` - `/pfss/mlde/workspaces/mlde_wsp_HessianLLM` - `/pfss/mlde/workspaces/mlde_wsp_KIServiceCenter` A bounded CPU mount check confirmed all three were accessible and the short JAIF alias was absent; its receipt is `runs/continuation-20261021/mount-check.json`. Existing containers keep the mounts from their own launch configuration. Project commands, environments and new outputs use the JAIF checkout above. Keep the user's retained continuation shell running until they ask to stop it. Determined bootstraps its internal shell state in `/run/determined/workdir`; after connecting, run project commands from: ```bash cd /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes set -a . /pfss/mlde/workspaces/mlde_wsp_JAIF/tauchmann/.env set +a ``` The personal `tauchmann` directory and `.env` must be owned by the user with mode `0700`, as requested. The `.env` stays outside the repository. Supply credentials over encrypted stdin; never put values in launch YAML, command arguments, logs, source, build artifacts or publication receipts. The three GPU configurations also bootstrap in `/run/determined/workdir`. This keeps Determined's working-directory fallback for native authentication cache and lock files outside the checkout. After connecting, explicitly change to `/pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes`; command entrypoints must do the same. Both legacy GPU workers already set that project working directory before training. Personal persistent credentials remain in `tauchmann/.env` outside the checkout; native task bootstrap state is separate from that file. The [SFT](../cluster/determined/german-qwen-shell.yaml), [agent RL](../cluster/determined/german-qwen-agent-rl-shell.yaml) and [OpenEnv command](../cluster/determined/german-qwen-openenv-command.yaml) configurations use the same JAIF mount and project root, and request one GPU. Launch them only after the local checkpoint, datasets and recreated pinned environments are verified in JAIF. They do not mount another project's storage. The OpenEnv launcher passes `--workspace-name JAIF` explicitly. The two queue-controller scripts are legacy, opt-in recovery tools. They have no historical allocation default: a NeMo controller launch requires `--shell-id` (or `HESSIAN_NEMO_SHELL_ID`), and the chained OpenEnv controller requires `--predecessor-shell-id` (or the same environment variable). They refuse an unset allocation before cluster contact and reject predecessor shells outside JAIF. See [JAIF commands and managed experiments](jaif-jobs.md) for the current submission boundary. Never give the continuation shell to a GPU controller: those controllers release their selected GPU allocation when the bounded job ends. New standard submissions should use the JAIF command configuration or a qualified managed experiment launcher rather than replaying an old controller. Both legacy controllers also require the untracked `.cache/agent-rl-site.tar` archive, pinned to SHA-256 `289ecc42e53d3fa6ac5b8532b447977f53e571837fbc4fae133ba42cf5437cfe`, and GPU libraries under `.venv-gpu/lib/python3.11/site-packages/torch/lib`, `nvidia` and `triton`. A clean source checkout and recreated environments do not recreate that historical archive. Until the exact archive and libraries are staged and verified in JAIF, those controllers are not ready to launch there. Do not weaken the archive check or copy old virtual environments to bypass it. The chained OpenEnv route additionally needs the matching predecessor NeMo controller's acknowledged `queue-launcher-status.json`; a different launcher does not supply that release journal automatically. The historical pilots run TRL inside Determined **shells or commands**. The project now also includes [managed Core entrypoints](jaif-jobs.md#managed-native-training-implementation) for trial metrics, scheduler snapshots, verified restoration and preemption. Managed CPU experiment **54512** (JAIF project **363**, trial **76039**) passed pause-after-three, scheduler-checkpoint restoration in a new allocation and completion at step eight. Its saved proof is `runs/continuation-20261021/managed-cpu-lifecycle-01/proof.json`. The [CPU experiment](../cluster/determined/jaif-managed-cpu.yaml) and [one-GPU training](../cluster/determined/jaif-managed-training.yaml) configurations select JAIF explicitly. GPU recovery and preemption qualification remain pending for the managed training path; the CPU lifecycle result is not a GPU or model quality result. See [the jobs guide](jaif-jobs.md) for entrypoints and the exact qualification boundary. Actual [German SFT](use-cases/german-reasoning-results.md), [agent RL](use-cases/german-agent-rl-results.md), and [NeMo Gym bridge](agent-gyms.md) runs used a single A100-SXM4-80GB. Those historical observations do not qualify a new JAIF run, the generic Docker build, or the Slurm script below. The observed container tag lacked a captured immutable digest. Preserve historical manifests and absolute-path-bound plans; prepare new plans for the JAIF root rather than editing old provenance. Recreate virtual environments there because interpreter links, editable installs and script shebangs can bind their original path. Confirm the exact-token and resume qualification before any further long training. ## Core container ```bash docker build -f containers/Dockerfile -t hessian-recipes:dev . docker run --rm hessian-recipes:dev ``` The default image exercises offline fixture replay and runs as an unprivileged user. The `/results` directory holds reports. Bind-mount a directory writable by UID 10001 there to retain them. To include Inspect use `--build-arg EXTRAS=inspect`; `--build-arg EXTRAS=training` installs the heavier pinned training stack. Installing PyTorch dependencies does not qualify a GPU execution environment: validate the host driver, runtime, CUDA libraries, and selected model separately. The Python base tag floats within 3.11. For a qualified deployment, replace it with the observed image digest, retain the build log and dependency freeze, scan that actual image, and attach its digest to the run evidence. The container is a packaging example until its build and execution are recorded in validation evidence. ## Slurm example ```bash sbatch --partition=YOUR_PARTITION \ --export=ALL,HESSIAN_PYTHON=/absolute/venv/bin/python \ scripts/slurm-train.sbatch /absolute/copied-recipe.json ``` The example requests one GPU and runs a normal installed Python recipe. Adapt the resource directives, account, partition, storage paths, and module setup to the site. It is not a distributed launcher or a site-tested cluster profile. Review `hessian-recipes train CONFIG --dry-run` before submission and stage all model files locally; worker downloads are disabled by default. Before labelling a recipe cluster-supported, record: exact package environment, hardware/driver, model and data identities, successful run and checkpoint reload, matched-budget baseline and adaptation evaluation, restart behavior, peak memory, and throughput. Multi-GPU and multi-node support each need their own tests. Real tool-using agents additionally need environment isolation, resource limits, and a verifier inaccessible to the evaluated agent; the in-memory development fixture does not supply that deployment isolation.