Environment frameworks: OpenEnv, NeMo Gym and FineEnvs
Upstream documentation checked on 20 October 2026. Both OpenEnv and NeMo Gym support custom models, existing agent harnesses, evaluation and training. The practical choice concerns interfaces, deployment and integration effort. Upstream capabilities and this repository's measured qualification are separate evidence.
| Framework | Environment contract | Deployment and composition |
|---|---|---|
| OpenEnv | Gymnasium-style reset(), step() and state(), with synchronous or asynchronous clients over WebSocket. | Docker environments can run locally or remotely. MCP exposes environment tools to agents; training integrations retain environment actions and rewards. |
| NeMo Gym | Composable FastAPI agent, model and resource servers communicating through asynchronous HTTP. | The model server standardizes inference behind the Responses API. Resource servers manage tools, isolated episode state and verification; an existing harness can manage model calls itself. |
These contracts come from the OpenEnv repository and NeMo Gym architecture. NeMo Gym does not require NVIDIA models or exclusive use of NeMo RL. Its architecture explicitly supports existing agents and local or provider model endpoints. An environment framework's vendor does not determine the model family.
OpenEnv offers two relevant routes: a trainer can own the tool loop through an environment adapter, or an existing harness can retain its loop through Harbor. The latter captures a typed TrainingTrace containing original engine tokens, log probabilities and explicit loss masks. Current documentation describes TRL's experimental AsyncGRPOTrainer with TRL 1.15 or later; that is a different runtime from our pinned TRL 0.29 recipe. OpenEnv therefore already addresses exact token capture for external harnesses. See its Harbor training integration.
NeMo Gym also documents training with an external harness that owns model calls. Its model-server capture associates calls with rollout identities and reconstructs their continuation relationships. Missing calls or unproven ancestry cause the rollout to be masked from training rather than guessed. Native token-bearing responses can instead be consumed directly. Both frameworks can serve this use case; their capture and trainer contracts need individual validation. See training on external harnesses.
Bringing our own model normally means configuring its inference endpoint and the harness that calls it, for example a self-hosted vLLM deployment inside the cluster. It does not inherently require uploading checkpoint weights to an environment website. Evaluation may operate on text, actions and trusted environment state. Policy-gradient training additionally needs the actual sampled tokens, conditioning contexts, compatible sampling log probabilities, policy identity and loss masks excluding external observations. Re-tokenizing a finished transcript does not establish that contract. Model calls must pass through the chosen training or capture path.
The FineEnvs multi-harness guide Space is an interactive article, not a checkpoint-submission or managed evaluation service. Its companion FineEnvs repository contains runnable environments and training projects, including examples across frameworks. We can adapt those projects to our model and harness and run them on our infrastructure. A public Space cannot automatically reach a cluster-private endpoint. Our model weights and datasets remain on the cluster; using the guide does not require publishing them. The article's historical results and current tutorial code are distinct; version pins matter when reproducing either.
Our qualified path is narrower: a real NeMo Gym CPU resource server supplied lookup/calculator state and verification while native TRL controlled generation and optimization. The A100 qualification completed eight steps and 32 on-policy rollouts, with all 392 trainable adapter tensors changed and independent observation/token-mask replay. Strict training success was 5/32. This establishes integration and an update, not held-out improvement or German-math retention. It does not qualify upstream external-harness capture, arbitrary black-box agents or transfer across harnesses. See local evidence and limits.
The matched OpenEnv arm now implements the same task, verifier and TRL loop. Native SDK CPU checks establish transport, state isolation, reward parity, capacity recovery and sticky infrastructure failure. The arm copies the frozen training data and evaluation protocol and starts from the original SFT parent. GPU resume qualification and held-out agent/MGSM assessment remain pending. The sequential controller waits for NeMo release before submitting one bounded GPU job; the original NeMo runtime stays frozen.
Hugging Face's reasoning walkthrough is a useful introductory example of Qwen3-1.7B, OpenEnv and GRPO with environment_factory. Its current main-version API and single-step arithmetic task differ from our pinned multi-turn recipe. The compatibility notes explain the adaptation and why training reward needs separate held-out tests.