German Qwen: agentic RL after SFT

This recipe starts from the standalone checkpoint produced by the German reasoning SFT pilot. The agent must retrieve hidden order facts, use a calculator, and return the correct total in a German task. Native multi-turn GRPO learns from the current policy's actual tool interactions. The environment supplies a terminal reward; it supplies no target explanation.

This is a small synthetic development task for qualifying the training and evaluation path. It is not an open benchmark or evidence of general agent ability.

The completed A100 pilot ran 16 steps and 64 rollouts. Its small agent-score increase was inconclusive, and its mathematical retention gate failed. The reusable default below remains 32 steps.

Environment and reward

Each task has an opaque record identifier. Its quantity and unit price are absent from the prompt and available only through lookup(record_id). The next assistant turn must invoke calculator(expression) with the product of those retrieved values. A subsequent answer must end with <answer>INTEGER</answer>.

The common prompt supplies the exact first lookup JSON and explains that calls must appear outside closed thinking, one at a time, followed by a wait for real tool feedback. This scaffolds action format and the first tool choice. It does not provide the hidden values or calculator expression. The protocol was refined using training-only preflights; baseline and candidate use the same final prompt.

Strict success is one only when the observed environment state proves the lookup, calculation, turn ordering, and correct final answer. All other completed task outcomes receive zero success credit. A correct guessed answer, fabricated tool transcript, or calculator call containing only the final number does not pass. Tool arguments and arithmetic are bounded; an AST interpreter evaluates literal arithmetic without Python eval. A model mistake is a scored task failure; worker and inference failures stop the run and remain separate diagnostics.

Training uses disclosed milestone shaping because binary-only training preflights produced no successful rollouts and no parameter updates. Its reward is `clip(0.15 * verified_lookup + 0.25 * verified_calculation + 0.60 * strict_success

  • 0.05 * tool_errors, 0, 1)`. The lookup and calculation milestones are one-time

indicators from actual private environment state, not statements in model text. Repeated calls earn no additional milestone credit. A partially successful agent can earn reward while failing the task; report this training reward separately from held-out strict workflow success. No supervised tool-format warmup is used.

The generator uses separate task identities, seeds and wording templates for 128 training, 32 development and 32 test tasks. The numeric ranges overlap; this evaluates new records and wording within the same simple task family. Hidden facts are dataset metadata used by the environment, never model inputs. No German-fluency or explanation-quality reward is claimed.

Training and token boundaries

The pinned TRL 0.29.0 GRPO trainer owns generation, grouped advantages, reference probabilities and optimization. Its environment_factory creates private state per rollout. The reusable default uses four on-policy generations per group, 32 optimizer steps, rank-16 LoRA, BF16, learning rate 0.00001 and KL coefficient 0.02. The reference is the starting SFT checkpoint with the new adapter disabled. No rollouts are reused across policy updates.

With batch size four and four generations, each optimizer group contains one task. The manifest reports the actual unique tasks consumed from the 128-row pool; 32 steps do not imply that all 128 training tasks were visited.

The native tool loop can normalize parsed assistant JSON when formatting later turns. Our shared tokenizer helper preserves the raw assistant span instead. Runtime assertions compare the reconstructed policy spans with the actual sampled token IDs and verify that tool observations have zero loss mask. The trainer fails if those boundaries diverge. Both training and evaluation apply the same helper; its source and the installed upstream trainer source are hashed. This qualification applies to this Qwen recipe and environment.

The completion stream has a 768-token cap, including inserted observations and template continuations, with at most three tool-loop iterations. The policy's thinking mode is enabled. Sampling is temperature 0.6, top-p 0.95, top-k 20, min-p 0 and seed 42. Evaluation freezes those settings for both checkpoints. Evaluation samples only the remaining stream budget at each turn. Native training can generate a longer continuation and discard its excess before scoring; the retained cap is not a bound on all physically generated training tokens.

Run on a GPU worker

On our cluster, start from /pfss/mlde/workspaces/mlde_wsp_JAIF/hessian_recipes in workspace JAIF; other teams can run from their own checkout root. Follow operations for the allocation and mounts. Prepare a fresh plan after relocating the model or checkout; historical run manifests retain their original identities and paths.

Use the existing isolated CUDA environment or create one with UV:

uv venv .venv-gpu --python python3.11
uv pip install --python .venv-gpu/bin/python -r requirements/german-qwen-agent-rl-cu126.txt
uv pip install --python .venv-gpu/bin/python -e .

The Determined shell configuration requests one GPU with shared workspace mounts. Data preparation and tokenizer checks can run on a CPU node; training and model evaluation require the GPU. From the repository root:

.venv-gpu/bin/python -m hessian_recipes.agent_rl_pilot prepare \
  --model /absolute/german-sft/merged-checkpoint \
  --retention-run /absolute/german-sft/run \
  --output runs/german-qwen-agent-rl/my-pilot

# On the GPU worker, after scripts/gpu-preflight.py succeeds:
.venv-gpu/bin/python -m hessian_recipes.agent_rl_pilot run \
  --plan runs/german-qwen-agent-rl/my-pilot/plan.json

Preparation freezes the three splits, tokenizer, checkpoint, configuration, source hashes and prior mathematical-evaluation evidence. Execution refuses a CPU-only worker, changed inputs and reused run directories. Each model-consuming stage runs in a separate process: agent baseline, online GRPO, adapter merge, agent candidate, and mathematical retention evaluation. Logs and status remain available if a stage fails. Use a fresh output directory after correcting a failure; the recipe does not claim recovery of an interrupted optimizer state. The pilot requires mixed-reward groups, nonzero gradients and changed adapter parameters; a successful process is insufficient.

Evaluation and artifacts

The held-out agent comparison replays tool actions, re-scores outcomes, validates coverage and checks the task, tokenizer, harness and budget identities. Report full workflow success separately from correct final answers, malformed outputs, tool errors, cap hits, token usage and generation time. A descriptive paired item bootstrap summarizes uncertainty from the small item sample; it does not capture training-seed or generation-seed variance.

The same 64 frozen MGSM German items and decoding protocol from the SFT pilot provide a separate retention check without tools. Its pre-RL baseline is the already-recorded evaluation of the exact starting SFT checkpoint. Report any retention regression even when agent-task success improves. Neither check alone qualifies a model for deployment.

Run directories contain the frozen inputs, raw rollout evidence and token masks, training manifest, LoRA adapter, standalone merged checkpoint, paired reports, retention report and stage logs. They remain ignored by Git. The repository contains code, configuration and aggregate results. Model publication is a separate step.

Generate a verified summary after the run finishes:

python -m hessian_recipes.agent_rl_report \
  --run runs/german-qwen-agent-rl/my-pilot \
  --output runs/german-qwen-agent-rl/my-pilot/summary

This command replays training tools, recomputes milestone rewards, checks sampled tokens and loss masks, verifies checkpoint ancestry and both evaluation protocols, and writes aggregate summary.json and summary.md without raw tasks or outputs. Run it from a Git checkout that retains the recorded training commit; the summary verifies that commit's source files against the consumed hashes.

The Agent Lightning launcher remains an additional integration path for external harnesses. The bounded recipe here uses the existing pinned TRL CUDA environment and does not qualify that separate stack.