# Longer German Qwen agent RL with NeMo Gym

This experiment asks whether additional tool-workflow training improves the
German Qwen3-1.7B SFT model while retaining its MGSM performance. The implementation
and protocol were prepared on 20 October 2026. **As of 21 October 2026, no
longer-run quality result is available.** The 20 October GPU attempt stopped in
the initial continuous branch of resume qualification: the strict audit found
disagreement between the native tool stream and sampled assistant IDs, conditioning
context or tool masks. The four-step versus stop-two/resume-four proof did not
complete, and the driver did not proceed to 256-step training or held-out
assessment. Preserve that failed attempt's evidence; a new JAIF attempt needs
fresh plans and successful runtime qualification.

Earlier [agent-RL results](german-agent-rl-results.md) failed the observed MGSM
retention gate, so retaining capabilities is an explicit acceptance condition
here. The later engineering failure does not provide another quality result.

## Frozen experiment

| Element | Protocol |
| --- | --- |
| Starting policy | Original German SFT merged checkpoint, tree SHA-256 `b16dc15361e19c155df1f833f6ae9800d81b12091755a6f0d346c8c37527b108`; never the earlier RL adapter. |
| Environment | Actual pinned NeMo Gym service `720b5f318caee339bfd29c554c59a97edcedc846`, with the existing stateful German lookup/calculator task and verifier. |
| Optimizer | Native TRL 0.29.0 online GRPO, 256 optimizer steps, four rollouts per group, rank-16 LoRA, BF16, learning rate `5e-6`, KL coefficient `0.1`. These choices do not guarantee retention. |
| Training data | 256 fresh TRAIN episodes, seed `20261020`, fixed deterministic schedule. No DEV, MGSM or final agent-test examples enter training. |
| Recovery | Native optimizer, scheduler and RNG checkpoints every 64 steps, with atomic integrity seals and exact evidence prefixes. |
| DEV selection | Assess steps 64, 128 and 256 on 64 fresh agent episodes and 128 source-disjoint German math DEV examples. A checkpoint must improve strict agent success and match or exceed baseline math correctness. Rank by strict successes, fewer tool errors, then earlier step. Step192 is recovery only. |
| No eligible checkpoint | Retain the SFT baseline. Assess the preregistered step256 diagnostic once; final tests cannot promote a failed DEV candidate. |
| Final agent assessment | Baseline and the selected assessment candidate: 256 fresh agent episodes at each of three fixed generation seeds, `42`, `1729`, `20261020`. |
| Mathematical retention | Full German MGSM250 at generation seeds `42` and `20261020`, baseline versus candidate, only after DEV selection. Require no observed loss overall **and** at either seed. |
| Comparison | Matched tokenizer, thinking setting and sampling; item-paired wins/losses and 10,000 task-cluster bootstrap resamples, grouping repeated seeds by item. Record malformed answers, tool errors, caps, runtime and tokens separately. |
| Resource bound | One GPU; cumulative driver wall budget eight hours after allocation. Queueing is not training progress. |

Before the main run, an actual GPU engineering qualification compares an
uninterrupted four-step run with stop-after-two/resume-to-four. Adapter, optimizer
and scheduler states and raw rollout evidence must match exactly. CPU scalar
optimizer/RNG tests and source review have passed; they cannot establish this GPU
equivalence. The driver refuses to proceed when qualification fails.

The native trainer fixes `full_determinism=True`. The first JAIF token-fix check
passed every sampled-token audit, but repeated updates from identical initial
weights, rollout evidence and RNG state still produced different adapter and
optimizer states. That attempt was stopped before long training. Deterministic
CUDA settings must pass the same exact equality qualification in a fresh plan;
the setting alone is not a qualification result.

Checkpoints are selected using DEV alone. Neither the final agent assessment nor
MGSM may guide additional steps, hyperparameters or checkpoint choice. The earlier
MGSM64 pilot was already inspected, so MGSM250 is not wholly unseen. A passing
observed zero-drop gate would still not prove absence of harm.

## Prepare and run

The current cluster checkout is `/pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes`
in workspace `JAIF`; see [operations](../operations.md). Other teams can use
their own checkout and scheduler. Following a workspace relocation, prepare a
fresh plan with the copied model/data and newly qualified environment paths.
Keep previous frozen plans, journals and measurement evidence unchanged; they
are not portable launch configurations. Match the preregistered task and
selection protocol when preparing the corresponding OpenEnv arm.

Use the separate [GPU and native Gym environments](../agent-gyms.md), with the
original SFT merged model, frozen tokenizer, source-disjoint SFT DEV data and
full MGSM data already available. Preparation performs no model inference:

```bash
.venv/bin/python -m hessian_recipes.gym_experiment prepare \
  --baseline runs/german-qwen/pilot-02/merged-checkpoint \
  --tokenizer runs/german-qwen/pilot-02/frozen-tokenizer \
  --math-dev runs/german-qwen/pilot-02/dev.jsonl \
  --mgsm runs/german-qwen/data/mgsm-de.jsonl \
  --output runs/agent-gyms/my-jaif-nemo-experiment
```

The selected output must have no previous experiment inputs. Preparation freezes
source and input hashes, generated episodes, seeds and the selection rule. Keep
the original SFT `train.jsonl` beside `dev.jsonl` so source disjointness can be
verified. Changing consumed runtime sources or input bytes invalidates the plan.

On an allocated compatible GPU, from the same repository checkout:

```bash
.venv-gpu/bin/python -m hessian_recipes.gym_experiment run \
  --plan runs/agent-gyms/my-jaif-nemo-experiment/experiment-plan.json \
  --gym-python "$PWD/.venv-nemo-gym/bin/python"
```

Use `--qualification-only` to stop after the real resume check, then `--resume`
with the identical plan to continue. Interrupted execution requires `--resume`;
the journal verifies completed evidence and the trainer restores sealed native
checkpoints. Unsealed or conflicting partial outputs are preserved in recovery
directories. Keep Gym's Python3.13 environment separate from the GPU Python3.11
stack; the driver owns and shuts down only its own Gym service.

The [experiment driver](../../src/hessian_recipes/gym_experiment.py),
[checkpoint wrapper](../../src/hessian_recipes/gym_training_long.py) and
[evaluation helper](../../src/hessian_recipes/gym_experiment_evaluation.py)
implement this protocol. Raw outputs, checkpoint weights and logs stay in ignored
run directories. The [Hugging Face hub](../../examples/huggingface/README.md)
contains recipe pages, upstream links and reviewed aggregate evidence; model
weights, downloaded data and raw run artifacts stay on the cluster.

## What the task can establish

The environment deliberately uses a narrow synthetic family with 80 ordered
operand pairs, scaffolding, bounded lookup/calculator tools and independent
strict workflow verification. Success would support this task family and
sampling protocol, not broad German reasoning or general agent competence.
Partial format reward is not strict success. Invalid native tool dispatches are
recorded separately from errors visible to the environment.

The retained training token cap includes external observations and framing; a
generation can consume and discard a longer tail. Record physical generation
usage separately where the trainer exposes it. Small LoRA changes can disappear
when merged into BF16 weights, so full merged-model identity and changed weights
must be audited before interpreting evaluation results.

The first ten preselected agent and MGSM sample IDs at seed42 receive a documented
language/format inspection, identifying an agent or human reviewer. That small
audit does not measure German fluency or explanation correctness. One training
seed and correlated synthetic tasks limit statistical conclusions; no run is
automatically release eligible.