# Existing harnesses: trace contract and RL adoption pilot

Implemented: a strict, framework-neutral JSONL validator/exporter, an external
callable instrumentation example, and an optional launcher for real upstream
Agent Lightning / verl training with our bounded calculator harness. Source/API
checks and mocked harness tests pass. The launcher has not been GPU-qualified;
adopting an arbitrary team's harness still requires a pilot.

The [German Qwen multi-turn RL recipe](use-cases/german-agent-rl.md) provides a
separate local path using the pinned TRL environment, a stateful lookup/calculator
harness, exact token assertions and matched agent/retention evaluation.

The optional [RL agent gyms](agent-gyms.md) add native NeMo Gym and Prime Intellect
Verifiers environment integrations. Their pins, setup and execution evidence are
separate from the Lightning launcher and the original local TRL pilot.

## Run the implemented example

After installing the package from the repository root:

```bash
python examples/agent_harness.py --output runs/calculator-trajectories.jsonl
hessian-recipes validate-trajectories runs/calculator-trajectories.jsonl
```

The callable `external_harness` keeps control of its tool loop. A wrapped policy
captures each inference call while that loop executes. The policy is an explicitly
synthetic byte-token fixture, so no model server or credentials are required. Its
output carries `source_kind: "fixture"`; it is not a captured LLM rollout or a
learning experiment.

## `hessian.trajectory.v1`

One JSONL row is one scored episode. Unknown fields, duplicate JSON keys,
duplicate episode/call IDs, invalid numbers, and token/mask mismatches are
rejected. Group consistency is checked across the entire imported file.

| Field | Meaning and constraint |
|---|---|
| `schema_version` | Exactly `hessian.trajectory.v1`; incompatible changes need a new version. |
| `episode_id`, `task_id`, `group_id` | Nonempty identifiers. A group contains comparable rollouts from one task/policy/harness/scorer. |
| `seed` | Nonnegative integer sample seed, recorded even for the fixture. |
| `source_kind` | `captured` for actual inference or `fixture` for synthetic data. This is provenance supplied by the producer, not independently authenticated. |
| `policy` | `model_id`, `revision`, `tokenizer_id`, `tokenizer_revision`, `chat_template_sha256`. Revisions cannot use obvious mutable aliases such as `main` or `latest`. |
| `harness` | `name`, `version`, `revision` identify orchestration and context processing. |
| `reward` | Finite `value`, named `scorer`, and scorer `revision`. Reward is episode-level; no credit assignment is performed. |
| `calls` | Nonempty ordered inference calls, defined below. At least one assistant response token must be trainable. |

Every call contains:

| Field | Meaning and constraint |
|---|---|
| `call_id` | Unique within the episode. |
| `prompt_token_ids` | Exact nonnegative integer input token IDs from that inference request. |
| `response_token_ids` | Exact nonnegative integer sampled assistant output IDs. |
| `response_role` | Must be `assistant`. Tool observations cannot be exported as training responses. |
| `response_loss_mask` | Integer 0/1 list aligned to output IDs; 1 marks a trainable model token. Prompt/context tokens have no trainable output mask. |
| `finish_reason` | `stop`, `length`, or `tool_calls`; truncation is explicit. |
| `context` | Audit messages with `role` and string `content`. Roles are `system`, `user`, `assistant`, or `tool`; this text is never substituted for captured tokens. |
| `behavior_logprobs` | Optional finite nonpositive per-output-token log probabilities, aligned to output IDs. A future RL adapter must require them if its algorithm needs them. |

Capture the actual template-applied prompt and sampled output at inference time.
Retokenizing displayed text can change token boundaries. Each call stores its own
prompt, so a harness can rewrite or summarize context without pretending that all
calls form a single continuous token sequence. Tool results appear in a subsequent
prompt and audit context, never as sampled assistant responses. A sampler can
still lie about those boundaries; validation checks structural consistency, not
the truth of the producer's capture.

The neutral contract currently covers text and a single policy identity per
episode. It does not encode multimodal tensors, per-subagent policies, retries,
reward decomposition, advantages, reference-policy probabilities, or optimizer
state. Infrastructure-failed/incomplete episodes need separate diagnostic records
and must not be silently converted to scored task failures. A valid file is not a
verl, TRL, Lightning, or Gym input format without an explicitly tested adapter.

Python APIs are `validate_trajectory(record) -> dict`,
`read_trajectories(path) -> list[dict]`, and
`write_trajectories(path, records) -> int`. Export validates the complete batch
before atomically replacing its destination. Malformed data raises `ValueError`.

## Optional real Lightning launcher

`examples/agent_lightning/launch.py` checks the exact upstream Git revision and
the actual `build_config` / `run_ppo` signatures without importing ML libraries.
Its execution path calls those real functions, points the local controller at
`hessian_lightning_harness.Agent`, and sends the harness's model requests through
the gateway. The gateway captures token IDs/log probabilities; this code does
not fabricate a neutral training trace. The harness retains its own three-turn
tool loop and reports a scalar terminal reward using upstream's event endpoint.

Prepare a **separate upstream environment** on a suitable CUDA GPU host:

```bash
git clone https://github.com/microsoft/agent-lightning.git /absolute/agent-lightning
git -C /absolute/agent-lightning checkout 69f5a40b36d9bff646b8e7a5532e8b78bd41eaf9
cd /absolute/agent-lightning
uv sync
bash scripts/setup_verl.sh 0.8.0 cu130
```

Run these commands from the hessian-recipes repository:

```bash
# Offline source/API check and plan: starts no services and performs no training.
python examples/agent_lightning/launch.py --upstream /absolute/agent-lightning

# Actual one-step GPU pilot, using a pre-staged local model snapshot.
python examples/agent_lightning/launch.py --upstream /absolute/agent-lightning \
  --model /absolute/Qwen2.5-1.5B-Instruct-snapshot \
  --output runs/agent-lightning-pilot --execute
```

The recipe is `recipes/agent-lightning.json`. Its tool parser settings come from
upstream Calc-X and target a compatible Qwen2.5-style model. Generic model
compatibility is not claimed. Execution remains offline for model loading,
disables external experiment logging, creates a new output directory, starts its
own loopback gateway/controller processes, and stops only those owned process
groups. It records model-file hashes, source hashes, resolved configuration,
training/validation tasks, environment, and status; upstream writes checkpoints.
It does not install or launch anything in the default repository environment.

The upstream setup script at this pin chooses `verl==0.8.0`, `vllm==0.22.0`,
and a locally built FlashAttention. Treat the pinned script as authoritative;
the upstream installation prose names a different vLLM version. This stack is
isolated from this repository's TRL extra.

Source checking was run against an actual checkout of the pinned commit, and
four local tests exercise tool observations, malformed-action feedback, reward
event shape, and revision/API mismatch rejection. These checks do not establish
CUDA compatibility, successful weight updates, or score improvements.

## First partner adoption pilot

Use upstream snapshot
[`69f5a40b36d9bff646b8e7a5532e8b78bd41eaf9`](https://github.com/microsoft/agent-lightning/tree/69f5a40b36d9bff646b8e7a5532e8b78bd41eaf9)
as a source-review starting point, then freeze the exact dependency environment
that succeeds on the selected GPU. The upstream v1.0
[quick start](https://microsoft.github.io/agent-lightning/stable/01-quick-start/)
uses a local controller with the verl/vLLM stack; follow those versioned upstream
scripts rather than assuming an older Lightning API still works.

1. Reproduce upstream's small local example in an isolated environment and retain
   its dependency lock, image digest, logs, and checkpoint. This is a GPU task,
   not covered by the fixture test suite.
2. Point one team's compatible model client at the rollout-specific gateway
   endpoint. Keep the harness's tools and orchestration intact. Verify captured
   prompt/output token IDs and chosen-token log probabilities against inference.
   The [gateway design](https://microsoft.github.io/agent-lightning/stable/05-basics/)
   describes these records and rollout identity.
3. Implement a version-pinned translation from upstream events into this audit
   contract. Reject stale policy versions, cross-episode leakage, inconsistent
   GRPO groups, and trainable tool observations. Upstream retains responsibility
   for the actual training samples and updates.
4. Compare transition versus continuous-trajectory aggregation on a harness that
   rewrites context. Preserve boundaries wherever token histories diverge. Follow
   the upstream [trainer configuration](https://microsoft.github.io/agent-lightning/stable/20-trainer-configuration/)
   for reward/advantage aggregation and loss normalization.
5. Demonstrate a real checkpoint update and reload, then evaluate the frozen
   baseline and adapted checkpoint using matched task and decoding budgets.
   Record task failures separately from gateway, tool, and worker failures.

Acceptance requires at least one repeatable training update, exact-token audit,
tool-loss exclusion, failed-worker recovery, consistent policy/group identity,
and a held-out result with measured runtime/resources. Only then label this a
supported agent-RL recipe. Compare NeMo Gym when environment/verifier reuse is the
primary requirement; see [related work](related-work.md).