Longer German Qwen agent RL with NeMo Gym

This experiment asks whether additional tool-workflow training improves the German Qwen3-1.7B SFT model while retaining its MGSM performance. The implementation and protocol were prepared on 20 October 2026. As of 21 October 2026, no longer-run quality result is available. The 20 October GPU attempt stopped in the initial continuous branch of resume qualification: the strict audit found disagreement between the native tool stream and sampled assistant IDs, conditioning context or tool masks. The four-step versus stop-two/resume-four proof did not complete, and the driver did not proceed to 256-step training or held-out assessment. Preserve that failed attempt's evidence; a new JAIF attempt needs fresh plans and successful runtime qualification.

Earlier agent-RL results failed the observed MGSM retention gate, so retaining capabilities is an explicit acceptance condition here. The later engineering failure does not provide another quality result.

Frozen experiment

Before the main run, an actual GPU engineering qualification compares an uninterrupted four-step run with stop-after-two/resume-to-four. Adapter, optimizer and scheduler states and raw rollout evidence must match exactly. CPU scalar optimizer/RNG tests and source review have passed; they cannot establish this GPU equivalence. The driver refuses to proceed when qualification fails.

The native trainer fixes full_determinism=True. The first JAIF token-fix check passed every sampled-token audit, but repeated updates from identical initial weights, rollout evidence and RNG state still produced different adapter and optimizer states. That attempt was stopped before long training. Deterministic CUDA settings must pass the same exact equality qualification in a fresh plan; the setting alone is not a qualification result.

Checkpoints are selected using DEV alone. Neither the final agent assessment nor MGSM may guide additional steps, hyperparameters or checkpoint choice. The earlier MGSM64 pilot was already inspected, so MGSM250 is not wholly unseen. A passing observed zero-drop gate would still not prove absence of harm.

Prepare and run

The current cluster checkout is /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes in workspace JAIF; see operations. Other teams can use their own checkout and scheduler. Following a workspace relocation, prepare a fresh plan with the copied model/data and newly qualified environment paths. Keep previous frozen plans, journals and measurement evidence unchanged; they are not portable launch configurations. Match the preregistered task and selection protocol when preparing the corresponding OpenEnv arm.

Use the separate GPU and native Gym environments, with the original SFT merged model, frozen tokenizer, source-disjoint SFT DEV data and full MGSM data already available. Preparation performs no model inference:

.venv/bin/python -m hessian_recipes.gym_experiment prepare \
  --baseline runs/german-qwen/pilot-02/merged-checkpoint \
  --tokenizer runs/german-qwen/pilot-02/frozen-tokenizer \
  --math-dev runs/german-qwen/pilot-02/dev.jsonl \
  --mgsm runs/german-qwen/data/mgsm-de.jsonl \
  --output runs/agent-gyms/my-jaif-nemo-experiment

The selected output must have no previous experiment inputs. Preparation freezes source and input hashes, generated episodes, seeds and the selection rule. Keep the original SFT train.jsonl beside dev.jsonl so source disjointness can be verified. Changing consumed runtime sources or input bytes invalidates the plan.

On an allocated compatible GPU, from the same repository checkout:

.venv-gpu/bin/python -m hessian_recipes.gym_experiment run \
  --plan runs/agent-gyms/my-jaif-nemo-experiment/experiment-plan.json \
  --gym-python "$PWD/.venv-nemo-gym/bin/python"

Use --qualification-only to stop after the real resume check, then --resume with the identical plan to continue. Interrupted execution requires --resume; the journal verifies completed evidence and the trainer restores sealed native checkpoints. Unsealed or conflicting partial outputs are preserved in recovery directories. Keep Gym's Python3.13 environment separate from the GPU Python3.11 stack; the driver owns and shuts down only its own Gym service.

The experiment driver, checkpoint wrapper and evaluation helper implement this protocol. Raw outputs, checkpoint weights and logs stay in ignored run directories. The Hugging Face hub contains recipe pages, upstream links and reviewed aggregate evidence; model weights, downloaded data and raw run artifacts stay on the cluster.

What the task can establish

The environment deliberately uses a narrow synthetic family with 80 ordered operand pairs, scaffolding, bounded lookup/calculator tools and independent strict workflow verification. Success would support this task family and sampling protocol, not broad German reasoning or general agent competence. Partial format reward is not strict success. Invalid native tool dispatches are recorded separately from errors visible to the environment.

The retained training token cap includes external observations and framing; a generation can consume and discard a longer tail. Record physical generation usage separately where the trainer exposes it. Small LoRA changes can disappear when merged into BF16 weights, so full merged-model identity and changed weights must be audited before interpreting evaluation results.

The first ten preselected agent and MGSM sample IDs at seed42 receive a documented language/format inspection, identifying an agent or human reviewer. That small audit does not measure German fluency or explanation correctness. One training seed and correlated synthetic tasks limit statistical conclusions; no run is automatically release eligible.