# RL agent gyms

These optional integrations reuse three upstream environment systems for the German
lookup/calculator task. NeMo Gym supplies an isolated HTTP resource server for
online TRL training. Prime Intellect Verifiers supplies a native task, tool and
reward lifecycle for its own rollout stack. Their environments are installed
separately from the Python 3.11 Qwen GPU environment.
OpenEnv supplies a native WebSocket service and isolated SDK for a matched second
TRL arm; its CPU integration passes. As of 21 October 2026, both longer
NeMo/OpenEnv GPU attempts stopped at the strict token audit during resume
qualification; neither reached 256-step training or held-out assessment. See the
[current programme status](programme-status.md).

Measured integration evidence is recorded in
[the validation summary](agent-gyms-validation.json). The native CPU checks passed
11 NeMo service checks, nine actual-client checks and seven Verifiers scenarios;
five optional Verifiers tests also passed. These scripted checks are not policy
accuracy measurements.

The NeMo Gym bridge also completed a real A100 qualification: eight GRPO steps,
32 on-policy rollouts and a saved adapter with all 392 trainable tensors changed.
Seven groups had varied rewards and all eight steps had nonzero gradients.
Independent replay verified 71 tool observations, 7,955 sampled assistant tokens
and 1,648 external/framing tokens excluded from loss. Training took 251.4 seconds.
The GPU worker has been released.

This bounded run started from the original German SFT checkpoint with seed 43,
consuming eight unique tasks from a fresh 16-task training pool. Strict training
success was 5/32; mean shaped training reward was 0.2171875. Neither is a held-out
score. No new agent test or MGSM retention result is claimed. The validation-only
adapter, model card, manifests and independent audits remain in the ignored
shared-storage directory `runs/agent-gyms/nemo-grpo-qualification-01/`.

The frozen [original agent-RL pilot](use-cases/german-agent-rl-results.md) remains
the measured quality baseline. Adding a gym does not change those scores or
resolve its retention regression. The integrations below distinguish framework
execution checks from actual model training.

| Integration | Pinned upstream | Responsibility |
| --- | --- | --- |
| NVIDIA NeMo Gym | [`720b5f3`](https://github.com/NVIDIA-NeMo/Gym/tree/720b5f318caee339bfd29c554c59a97edcedc846) | Resource-server lifecycle, per-episode state, actual tool execution and verification; TRL retains generation and optimization. |
| Hugging Face OpenEnv | [`0.9.0 / 17c1edd`](https://github.com/huggingface/OpenEnv/tree/17c1edd22168fcb76b086def1ed90a7168fb8749) | Native per-session WebSocket reset/step/state, tools and verifier; the same pinned TRL loop retains policy generation and optimization. |
| Prime Intellect Verifiers | [`0.4.0`](https://github.com/PrimeIntellect-ai/verifiers/tree/6e69354ffe1b445603b493b722d0ecffbca6686d) | Native taskset, harness, MCP tools, environment and strict reward; its training integration is separate from TRL. |

These integrations do not turn the synthetic German development task into an upstream
benchmark. We reuse the frozen environment and verifier so that framework changes
do not silently change what counts as success. The prompt still scaffolds the
first lookup action, and hidden quantities remain private until tool execution.

## NeMo Gym with online TRL

The [resource server](../examples/agent_gyms/nemo_gym_server.py) subclasses the
actual upstream `SimpleResourcesServer`. It offers native session/verification
routes plus an explicit bridge for the external TRL tool loop. It runs on the
CPU and performs no model inference. Each rollout receives a fresh opaque session
identifier, injected by the client rather than exposed as a model argument.

This pin requires Python 3.13.14 or later. Use a recent UV version that supports
its upstream lock format:

```bash
git clone https://github.com/NVIDIA-NeMo/Gym.git .cache/upstreams/nemo-gym
git -C .cache/upstreams/nemo-gym checkout 720b5f318caee339bfd29c554c59a97edcedc846
uv venv --python 3.13.14 .venv-nemo-gym
UV_PROJECT_ENVIRONMENT="$PWD/.venv-nemo-gym" uv sync \
  --project .cache/upstreams/nemo-gym --no-dev --locked

.venv-nemo-gym/bin/python examples/agent_gyms/nemo_gym_server.py --port 18765
```

The two native CPU qualifications start and stop their own service processes:

```bash
.venv-nemo-gym/bin/python examples/agent_gyms/nemo_gym_smoke.py \
  --output runs/agent-gyms/my-nemo-service-check
.venv-nemo-gym/bin/python examples/agent_gyms/nemo_gym_client_smoke.py \
  --output runs/agent-gyms/my-nemo-client-check
```

Start the service on the same host as the trainer, using a separate terminal or
owned background process. Both ends use loopback HTTP. There is no hosted service,
external model API or provider credential in this path. The Gym stack does not
install Ray, vLLM or another CUDA stack into the existing training environment.

Use a fresh frozen training configuration with the same schema accepted by
`agent_rl_training`; the [agent-RL preparation command](use-cases/german-agent-rl.md)
writes `training.json` inside its prepared run directory. The Gym trainer accepts
that configuration, not the whole pilot plan:

```bash
# From the existing pinned Python 3.11 GPU environment:
.venv-gpu/bin/python -m hessian_recipes.gym_training /absolute/training-config.json \
  --gym-url http://127.0.0.1:18765 --dry-run

# Requires one visible BF16-capable CUDA GPU:
.venv-gpu/bin/python -m hessian_recipes.gym_training /absolute/training-config.json \
  --gym-url http://127.0.0.1:18765
```

The dry run checks inputs and the live server identity; it performs no model
inference. Training uses current-policy native multi-turn GRPO, the original raw
sampled-token/context audit and external-token loss masking. Gym serves the actual
lookup and calculator observations. Its strict verifier result must exactly match
an independent local replay of those observed calls before training-only milestone
reward shaping is applied. The model sees only `lookup` and `calculator`; service,
verification and cleanup methods are private to the trainer.

HTTP failures, missing sessions, changed source identities and replay disagreement
are infrastructure failures. They never become zero-reward model mistakes. A
sticky error flag preserves this distinction when upstream catches a tool-call
exception. Ordinary invalid arithmetic and wrong record requests remain scored
task failures. Sessions close when reused or when the owned trainer finishes.

The training manifest records the gym revision/lock, server/client/environment
source hashes, policy identity, consumed tasks, sampled-token evidence, gradient
and parameter changes. A process exit alone does not qualify an update. Keep
baseline/candidate and retention evaluation separate and matched, as in the
[evaluation protocol](use-cases/german-agent-rl.md#evaluation-and-artifacts).

## OpenEnv with the same TRL loop

The [native server](../examples/agent_gyms/openenv_server.py) implements OpenEnv's
typed environment contract. The [SDK bridge](../src/hessian_recipes/openenv_client.py)
owns a persistent WebSocket per rollout in a separate Python 3.13 process. The
Qwen/TRL process remains on its existing Python 3.11 GPU stack. SDK/server
children clear the inherited GPU package path, perform no model inference and
expose only `lookup` and `calculator` to the model. Terminal verification stays
private to the trainer and must agree with independent local replay.

```bash
uv venv .venv-openenv --python 3.13.14
uv pip sync --python .venv-openenv/bin/python requirements/openenv.txt
.venv-openenv/bin/python examples/agent_gyms/openenv_smoke.py \
  --output runs/agent-gyms/my-openenv-native-check
```

The smoke owns and releases its service. It checks native SDK state and reward
parity, private facts, malformed calls, immutable terminal verification, concurrent
session isolation, capacity recovery and 260 sequential resets. These are scripted
CPU checks, not model scores. An additional native client test and forced service
loss check qualify the Python ABI bridge and sticky infrastructure errors.

The [matched German experiment](use-cases/german-openenv-comparison.md) provides
preparation and sequential scheduling commands, frozen protocol details and the
adaptation from Hugging Face's
[reasoning tutorial](https://huggingface.co/docs/openenv/tutorials/end-to-end-walkthrough).
It requires its own actual GPU uninterrupted-versus-resumed qualification before
256-step training, independent DEV selection and full MGSM assessment. Only one
of the NeMo/OpenEnv arms requests a GPU at a time. Raw artifacts remain local.

## Prime Intellect Verifiers

The [native environment](../examples/agent_gyms/verifiers_environment.py) uses the
pinned v1 APIs rather than the removed legacy `ToolEnv` interface. It defines the
German taskset, per-rollout tool state and strict reward in Verifiers itself.
The development smoke uses a scripted fixture policy through that real lifecycle;
its outputs are not captured model samples and supply no evidence of learning.

Install the [pinned source and MCP dependencies](../requirements/verifiers.txt)
and run the native CPU fixture:

```bash
uv venv --python 3.11 .venv-verifiers
uv pip install --python .venv-verifiers/bin/python -r requirements/verifiers.txt
uv pip install --python .venv-verifiers/bin/python --no-deps -e .
CUDA_VISIBLE_DEVICES='' HF_HUB_OFFLINE=1 \
  .venv-verifiers/bin/python examples/agent_gyms/verifiers_smoke.py \
  --output runs/agent-gyms/my-verifiers-check
```

The environment defaults to the upstream real model harness. The smoke explicitly
selects its separate fixture harness. Real evaluation can use an already running
compatible local inference server:

```bash
LOCAL_MODEL_API_KEY=local PYTHONPATH="$PWD/examples/agent_gyms" \
  .venv-verifiers/bin/vf-eval verifiers_environment \
  --env.agent.harness.id null --env.agent.runtime.type subprocess \
  --env.agent.max-turns 6 --env.agent.max-output-tokens 768 \
  --env.agent.timeout.rollout 120 --model LOCAL_MODEL \
  --client.base-url http://127.0.0.1:8000/v1 \
  --client.api-key-var LOCAL_MODEL_API_KEY -n 4
```

The fixture needs no GPU or provider credentials. Real model inference through
this second stack has not been qualified here. Its per-call output budget is not
equivalent to the TRL recipe's cumulative completion-stream budget, so raw scores
must not be compared as a matched experiment.

Native Prime-RL training requires its own compatible inference/training stack;
the fixture does not manufacture token IDs, behavior log probabilities or a TRL
training batch. The inspected Prime-RL snapshot pins a different Verifiers
submodule, so a current Prime-RL checkout is not a tested drop-in training backend
for this v0.4.0 environment. Its native training client uses token-in/token-out
inference and retains loss masks; that path still needs an actual GPU pilot.

Keep reward and task identities pinned when comparing the two gyms. Preserve
actual generated token IDs, conditioning context, policy versions and tool-loss
exclusion before calling any additional training backend qualified. Current
integration evidence and practical limits are recorded alongside the results.