Existing harnesses: trace contract and RL adoption pilot

Implemented: a strict, framework-neutral JSONL validator/exporter, an external callable instrumentation example, and an optional launcher for real upstream Agent Lightning / verl training with our bounded calculator harness. Source/API checks and mocked harness tests pass. The launcher has not been GPU-qualified; adopting an arbitrary team's harness still requires a pilot.

The German Qwen multi-turn RL recipe provides a separate local path using the pinned TRL environment, a stateful lookup/calculator harness, exact token assertions and matched agent/retention evaluation.

The optional RL agent gyms add native NeMo Gym and Prime Intellect Verifiers environment integrations. Their pins, setup and execution evidence are separate from the Lightning launcher and the original local TRL pilot.

Run the implemented example

After installing the package from the repository root:

python examples/agent_harness.py --output runs/calculator-trajectories.jsonl
hessian-recipes validate-trajectories runs/calculator-trajectories.jsonl

The callable external_harness keeps control of its tool loop. A wrapped policy captures each inference call while that loop executes. The policy is an explicitly synthetic byte-token fixture, so no model server or credentials are required. Its output carries source_kind: "fixture"; it is not a captured LLM rollout or a learning experiment.

`hessian.trajectory.v1`

One JSONL row is one scored episode. Unknown fields, duplicate JSON keys, duplicate episode/call IDs, invalid numbers, and token/mask mismatches are rejected. Group consistency is checked across the entire imported file.

Field	Meaning and constraint
`schema_version`	Exactly `hessian.trajectory.v1`; incompatible changes need a new version.
`episode_id`, `task_id`, `group_id`	Nonempty identifiers. A group contains comparable rollouts from one task/policy/harness/scorer.
`seed`	Nonnegative integer sample seed, recorded even for the fixture.
`source_kind`	`captured` for actual inference or `fixture` for synthetic data. This is provenance supplied by the producer, not independently authenticated.
`policy`	`model_id`, `revision`, `tokenizer_id`, `tokenizer_revision`, `chat_template_sha256`. Revisions cannot use obvious mutable aliases such as `main` or `latest`.
`harness`	`name`, `version`, `revision` identify orchestration and context processing.
`reward`	Finite `value`, named `scorer`, and scorer `revision`. Reward is episode-level; no credit assignment is performed.
`calls`	Nonempty ordered inference calls, defined below. At least one assistant response token must be trainable.

Every call contains:

Field	Meaning and constraint
`call_id`	Unique within the episode.
`prompt_token_ids`	Exact nonnegative integer input token IDs from that inference request.
`response_token_ids`	Exact nonnegative integer sampled assistant output IDs.
`response_role`	Must be `assistant`. Tool observations cannot be exported as training responses.
`response_loss_mask`	Integer 0/1 list aligned to output IDs; 1 marks a trainable model token. Prompt/context tokens have no trainable output mask.
`finish_reason`	`stop`, `length`, or `tool_calls`; truncation is explicit.
`context`	Audit messages with `role` and string `content`. Roles are `system`, `user`, `assistant`, or `tool`; this text is never substituted for captured tokens.
`behavior_logprobs`	Optional finite nonpositive per-output-token log probabilities, aligned to output IDs. A future RL adapter must require them if its algorithm needs them.

Capture the actual template-applied prompt and sampled output at inference time. Retokenizing displayed text can change token boundaries. Each call stores its own prompt, so a harness can rewrite or summarize context without pretending that all calls form a single continuous token sequence. Tool results appear in a subsequent prompt and audit context, never as sampled assistant responses. A sampler can still lie about those boundaries; validation checks structural consistency, not the truth of the producer's capture.

The neutral contract currently covers text and a single policy identity per episode. It does not encode multimodal tensors, per-subagent policies, retries, reward decomposition, advantages, reference-policy probabilities, or optimizer state. Infrastructure-failed/incomplete episodes need separate diagnostic records and must not be silently converted to scored task failures. A valid file is not a verl, TRL, Lightning, or Gym input format without an explicitly tested adapter.

Python APIs are validate_trajectory(record) -> dict, read_trajectories(path) -> list[dict], and write_trajectories(path, records) -> int. Export validates the complete batch before atomically replacing its destination. Malformed data raises ValueError.

Optional real Lightning launcher

examples/agent_lightning/launch.py checks the exact upstream Git revision and the actual build_config / run_ppo signatures without importing ML libraries. Its execution path calls those real functions, points the local controller at hessian_lightning_harness.Agent, and sends the harness's model requests through the gateway. The gateway captures token IDs/log probabilities; this code does not fabricate a neutral training trace. The harness retains its own three-turn tool loop and reports a scalar terminal reward using upstream's event endpoint.

Prepare a separate upstream environment on a suitable CUDA GPU host:

git clone https://github.com/microsoft/agent-lightning.git /absolute/agent-lightning
git -C /absolute/agent-lightning checkout 69f5a40b36d9bff646b8e7a5532e8b78bd41eaf9
cd /absolute/agent-lightning
uv sync
bash scripts/setup_verl.sh 0.8.0 cu130

Run these commands from the hessian-recipes repository:

# Offline source/API check and plan: starts no services and performs no training.
python examples/agent_lightning/launch.py --upstream /absolute/agent-lightning

# Actual one-step GPU pilot, using a pre-staged local model snapshot.
python examples/agent_lightning/launch.py --upstream /absolute/agent-lightning \
  --model /absolute/Qwen2.5-1.5B-Instruct-snapshot \
  --output runs/agent-lightning-pilot --execute

The recipe is recipes/agent-lightning.json. Its tool parser settings come from upstream Calc-X and target a compatible Qwen2.5-style model. Generic model compatibility is not claimed. Execution remains offline for model loading, disables external experiment logging, creates a new output directory, starts its own loopback gateway/controller processes, and stops only those owned process groups. It records model-file hashes, source hashes, resolved configuration, training/validation tasks, environment, and status; upstream writes checkpoints. It does not install or launch anything in the default repository environment.

The upstream setup script at this pin chooses verl==0.8.0, vllm==0.22.0, and a locally built FlashAttention. Treat the pinned script as authoritative; the upstream installation prose names a different vLLM version. This stack is isolated from this repository's TRL extra.

Source checking was run against an actual checkout of the pinned commit, and four local tests exercise tool observations, malformed-action feedback, reward event shape, and revision/API mismatch rejection. These checks do not establish CUDA compatibility, successful weight updates, or score improvements.

First partner adoption pilot

Use upstream snapshot `69f5a40b36d9bff646b8e7a5532e8b78bd41eaf9` as a source-review starting point, then freeze the exact dependency environment that succeeds on the selected GPU. The upstream v1.0 quick start uses a local controller with the verl/vLLM stack; follow those versioned upstream scripts rather than assuming an older Lightning API still works.

Reproduce upstream's small local example in an isolated environment and retain

its dependency lock, image digest, logs, and checkpoint. This is a GPU task, not covered by the fixture test suite.

Point one team's compatible model client at the rollout-specific gateway

endpoint. Keep the harness's tools and orchestration intact. Verify captured prompt/output token IDs and chosen-token log probabilities against inference. The gateway design describes these records and rollout identity.

Implement a version-pinned translation from upstream events into this audit

contract. Reject stale policy versions, cross-episode leakage, inconsistent GRPO groups, and trainable tool observations. Upstream retains responsibility for the actual training samples and updates.

Compare transition versus continuous-trajectory aggregation on a harness that

rewrites context. Preserve boundaries wherever token histories diverge. Follow the upstream trainer configuration for reward/advantage aggregation and loss normalization.

Demonstrate a real checkpoint update and reload, then evaluate the frozen

baseline and adapted checkpoint using matched task and decoding budgets. Record task failures separately from gateway, tool, and worker failures.

Acceptance requires at least one repeatable training update, exact-token audit, tool-loss exclusion, failed-worker recovery, consistent policy/group identity, and a held-out result with measured runtime/resources. Only then label this a supported agent-RL recipe. Compare NeMo Gym when environment/verifier reuse is the primary requirement; see related work.