RL agent gyms

These optional integrations reuse three upstream environment systems for the German lookup/calculator task. NeMo Gym supplies an isolated HTTP resource server for online TRL training. Prime Intellect Verifiers supplies a native task, tool and reward lifecycle for its own rollout stack. Their environments are installed separately from the Python 3.11 Qwen GPU environment. OpenEnv supplies a native WebSocket service and isolated SDK for a matched second TRL arm; its CPU integration passes. As of 21 October 2026, both longer NeMo/OpenEnv GPU attempts stopped at the strict token audit during resume qualification; neither reached 256-step training or held-out assessment. See the current programme status.

Measured integration evidence is recorded in the validation summary. The native CPU checks passed 11 NeMo service checks, nine actual-client checks and seven Verifiers scenarios; five optional Verifiers tests also passed. These scripted checks are not policy accuracy measurements.

The NeMo Gym bridge also completed a real A100 qualification: eight GRPO steps, 32 on-policy rollouts and a saved adapter with all 392 trainable tensors changed. Seven groups had varied rewards and all eight steps had nonzero gradients. Independent replay verified 71 tool observations, 7,955 sampled assistant tokens and 1,648 external/framing tokens excluded from loss. Training took 251.4 seconds. The GPU worker has been released.

This bounded run started from the original German SFT checkpoint with seed 43, consuming eight unique tasks from a fresh 16-task training pool. Strict training success was 5/32; mean shaped training reward was 0.2171875. Neither is a held-out score. No new agent test or MGSM retention result is claimed. The validation-only adapter, model card, manifests and independent audits remain in the ignored shared-storage directory runs/agent-gyms/nemo-grpo-qualification-01/.

The frozen original agent-RL pilot remains the measured quality baseline. Adding a gym does not change those scores or resolve its retention regression. The integrations below distinguish framework execution checks from actual model training.

These integrations do not turn the synthetic German development task into an upstream benchmark. We reuse the frozen environment and verifier so that framework changes do not silently change what counts as success. The prompt still scaffolds the first lookup action, and hidden quantities remain private until tool execution.

NeMo Gym with online TRL

The resource server subclasses the actual upstream SimpleResourcesServer. It offers native session/verification routes plus an explicit bridge for the external TRL tool loop. It runs on the CPU and performs no model inference. Each rollout receives a fresh opaque session identifier, injected by the client rather than exposed as a model argument.

This pin requires Python 3.13.14 or later. Use a recent UV version that supports its upstream lock format:

git clone https://github.com/NVIDIA-NeMo/Gym.git .cache/upstreams/nemo-gym
git -C .cache/upstreams/nemo-gym checkout 720b5f318caee339bfd29c554c59a97edcedc846
uv venv --python 3.13.14 .venv-nemo-gym
UV_PROJECT_ENVIRONMENT="$PWD/.venv-nemo-gym" uv sync \
  --project .cache/upstreams/nemo-gym --no-dev --locked

.venv-nemo-gym/bin/python examples/agent_gyms/nemo_gym_server.py --port 18765

The two native CPU qualifications start and stop their own service processes:

.venv-nemo-gym/bin/python examples/agent_gyms/nemo_gym_smoke.py \
  --output runs/agent-gyms/my-nemo-service-check
.venv-nemo-gym/bin/python examples/agent_gyms/nemo_gym_client_smoke.py \
  --output runs/agent-gyms/my-nemo-client-check

Start the service on the same host as the trainer, using a separate terminal or owned background process. Both ends use loopback HTTP. There is no hosted service, external model API or provider credential in this path. The Gym stack does not install Ray, vLLM or another CUDA stack into the existing training environment.

Use a fresh frozen training configuration with the same schema accepted by agent_rl_training; the agent-RL preparation command writes training.json inside its prepared run directory. The Gym trainer accepts that configuration, not the whole pilot plan:

# From the existing pinned Python 3.11 GPU environment:
.venv-gpu/bin/python -m hessian_recipes.gym_training /absolute/training-config.json \
  --gym-url http://127.0.0.1:18765 --dry-run

# Requires one visible BF16-capable CUDA GPU:
.venv-gpu/bin/python -m hessian_recipes.gym_training /absolute/training-config.json \
  --gym-url http://127.0.0.1:18765

The dry run checks inputs and the live server identity; it performs no model inference. Training uses current-policy native multi-turn GRPO, the original raw sampled-token/context audit and external-token loss masking. Gym serves the actual lookup and calculator observations. Its strict verifier result must exactly match an independent local replay of those observed calls before training-only milestone reward shaping is applied. The model sees only lookup and calculator; service, verification and cleanup methods are private to the trainer.

HTTP failures, missing sessions, changed source identities and replay disagreement are infrastructure failures. They never become zero-reward model mistakes. A sticky error flag preserves this distinction when upstream catches a tool-call exception. Ordinary invalid arithmetic and wrong record requests remain scored task failures. Sessions close when reused or when the owned trainer finishes.

The training manifest records the gym revision/lock, server/client/environment source hashes, policy identity, consumed tasks, sampled-token evidence, gradient and parameter changes. A process exit alone does not qualify an update. Keep baseline/candidate and retention evaluation separate and matched, as in the evaluation protocol.

OpenEnv with the same TRL loop

The native server implements OpenEnv's typed environment contract. The SDK bridge owns a persistent WebSocket per rollout in a separate Python 3.13 process. The Qwen/TRL process remains on its existing Python 3.11 GPU stack. SDK/server children clear the inherited GPU package path, perform no model inference and expose only lookup and calculator to the model. Terminal verification stays private to the trainer and must agree with independent local replay.

uv venv .venv-openenv --python 3.13.14
uv pip sync --python .venv-openenv/bin/python requirements/openenv.txt
.venv-openenv/bin/python examples/agent_gyms/openenv_smoke.py \
  --output runs/agent-gyms/my-openenv-native-check

The smoke owns and releases its service. It checks native SDK state and reward parity, private facts, malformed calls, immutable terminal verification, concurrent session isolation, capacity recovery and 260 sequential resets. These are scripted CPU checks, not model scores. An additional native client test and forced service loss check qualify the Python ABI bridge and sticky infrastructure errors.

The matched German experiment provides preparation and sequential scheduling commands, frozen protocol details and the adaptation from Hugging Face's reasoning tutorial. It requires its own actual GPU uninterrupted-versus-resumed qualification before 256-step training, independent DEV selection and full MGSM assessment. Only one of the NeMo/OpenEnv arms requests a GPU at a time. Raw artifacts remain local.

Prime Intellect Verifiers

The native environment uses the pinned v1 APIs rather than the removed legacy ToolEnv interface. It defines the German taskset, per-rollout tool state and strict reward in Verifiers itself. The development smoke uses a scripted fixture policy through that real lifecycle; its outputs are not captured model samples and supply no evidence of learning.

Install the pinned source and MCP dependencies and run the native CPU fixture:

uv venv --python 3.11 .venv-verifiers
uv pip install --python .venv-verifiers/bin/python -r requirements/verifiers.txt
uv pip install --python .venv-verifiers/bin/python --no-deps -e .
CUDA_VISIBLE_DEVICES='' HF_HUB_OFFLINE=1 \
  .venv-verifiers/bin/python examples/agent_gyms/verifiers_smoke.py \
  --output runs/agent-gyms/my-verifiers-check

The environment defaults to the upstream real model harness. The smoke explicitly selects its separate fixture harness. Real evaluation can use an already running compatible local inference server:

LOCAL_MODEL_API_KEY=local PYTHONPATH="$PWD/examples/agent_gyms" \
  .venv-verifiers/bin/vf-eval verifiers_environment \
  --env.agent.harness.id null --env.agent.runtime.type subprocess \
  --env.agent.max-turns 6 --env.agent.max-output-tokens 768 \
  --env.agent.timeout.rollout 120 --model LOCAL_MODEL \
  --client.base-url http://127.0.0.1:8000/v1 \
  --client.api-key-var LOCAL_MODEL_API_KEY -n 4

The fixture needs no GPU or provider credentials. Real model inference through this second stack has not been qualified here. Its per-call output budget is not equivalent to the TRL recipe's cumulative completion-stream budget, so raw scores must not be compared as a matched experiment.

Native Prime-RL training requires its own compatible inference/training stack; the fixture does not manufacture token IDs, behavior log probabilities or a TRL training batch. The inspected Prime-RL snapshot pins a different Verifiers submodule, so a current Prime-RL checkout is not a tested drop-in training backend for this v0.4.0 environment. Its native training client uses token-in/token-out inference and retains loss masks; that path still needs an actual GPU pilot.

Keep reward and task identities pinned when comparing the two gyms. Preserve actual generated token IDs, conditioning context, policy versions and tool-loss exclusion before calling any additional training backend qualified. Current integration evidence and practical limits are recorded alongside the results.