# Evaluation evidence kit

For measured model experiments beyond the development fixtures below, see the
[German Qwen MGSM evaluation](use-cases/german-reasoning.md#evaluation-and-limits)
and [interactive agent plus retention evaluation](use-cases/german-agent-rl.md#evaluation-and-artifacts).
Their [SFT results](use-cases/german-reasoning-results.md) and
[agent-RL results](use-cases/german-agent-rl-results.md) retain separate metrics
and limitations. [Gym qualification](agent-gyms.md) establishes integration and
parameter updates, without a new held-out quality claim.

The kit implements a small offline runner and an optional real Inspect task adapter.
Its outputs support comparing experiments and reviewing a release candidate; they
never automatically certify a release. Every report has `release_eligible: false`.

The proposed programme topics are [German RAG with cited evidence and
business-communication agent sequences](benchmark-designs.md). Their real tasks
and specialized scorers are not implemented. The two
[provisional tracks](../benchmarks/README.md) are tiny synthetic development
fixtures for exercising this repository. Fixture results must not be described as
model capabilities or measured training gains.

## Run the complete offline demonstration

From the repository clone, after `pip install -e .`:

```bash
hessian-recipes demo --output runs/demo
hessian-recipes eval benchmarks/reasoning_reliability/spec.json --output runs/baseline
hessian-recipes eval benchmarks/reasoning_reliability/spec-candidate.json --output runs/candidate
hessian-recipes compare runs/baseline/report.json runs/candidate/report.json
```

Every evaluation writes `report.json` and `report.md`. The synthetic reasoning
baseline is deliberately 50% correct and its synthetic candidate 100%; the workflow
baseline is deliberately 1/3 correct and its synthetic candidate 100%. These values
verify wiring only. Each trial replays a public authored response, without model,
GPU, network, API credentials, or inferred token counts. Identical fixture trials
do not add statistical information.

Existing `report.json` or `report.md` files cause an error before model loading;
choose a fresh output directory for each experiment.

Evaluation exits successfully when execution coverage is complete, even if answers
are incorrect. A regression gate exits unsuccessfully when comparison fails.
Malformed manifests and hash failures are errors before generation. Runtime adapter
failures remain explicit per-trial infrastructure failures; they are never silently
removed from accuracy. Incomplete runs have no headline accuracy or confidence
interval and cannot pass comparison. `scored_accuracy` is a diagnostic on the
successfully executed subset, with its coverage shown separately.

## Frozen specification

The example `spec.json` files are executable references. All file and directory
paths resolve relative to **the specification file's parent**, independently of
current working directory. Required fields are:

| Field | Meaning |
| --- | --- |
| `schema_version` | Currently `1` |
| `benchmark` | `id`, `version`, `status`, `count`, `path` to JSONL, exact file `sha256` |
| `model` | `id`, immutable `revision`, `tokenizer_id`, immutable `tokenizer_revision`, full literal `chat_template` |
| `sampling` | `temperature`, `max_new_tokens`, `seed`; optional `top_p`, `top_k`, `min_p`; other knobs are rejected |
| `trials` | Positive repeats per sample, indexed from zero |
| `adapter` | Fixture file, local Transformers checkpoint, or trusted injected callable |
| `scorer_version`, `harness_version` | Both currently string `"1"`; implementation file SHA256 is also recorded |

Fixture-only revisions may use the explicit `fixture-v1` marker because no model
is executed. Nonfixture revisions must be immutable 40-character Git SHA1 or
64-character SHA256 identities. Do not use mutable `main` or `latest` revisions.

JSONL samples have unique `id`, `prompt`, `category`, and a `scorer`. `exact` and
`numeric` have an `answer`; exact comparison strips surrounding whitespace and
remains case-sensitive. Numeric scoring accepts exactly one finite decimal number,
including scientific notation, with exact decimal equality. It does not search a
reasoning transcript for a plausible answer. Prompts explicitly request the final
answer in that format.

The optional `final_numeric` scorer supports reasoning models: it accepts a whole
canonical decimal or exactly one terminal `<answer>number</answer>` block after
reasoning. It never extracts intermediate chain-of-thought numbers. Duplicate tags,
unclosed `<think>` sections, trailing prose, nonfinite numbers, leading plus signs,
leading zeros, and German decimal commas are rejected. Ask the model for a decimal
point and the explicit final-answer format. Decimal equality remains exact.

The `workflow` scorer takes `workflow.initial_state`, `target_state`, and
`max_steps`. Generated output must be exactly an `actions` object containing a list
of toy `read`, `reserve`, or `release` calls. A deterministic state machine validates
arguments, executes bounded in-memory state changes, and compares the final state.
Unknown tools, invalid quantities, extra arguments, unavailable stock, and excessive
steps are incorrect answers. A malformed plan is not an infrastructure failure.
This measures generated plan validity and terminal state in a toy environment;
interactive tool recovery and real agent harness behavior require separate tasks.
Generated Python, shell commands, network operations, and real tools are never run.

The runner records exact UTF-8 specification and benchmark snapshots, full
individual outputs, scores, error types, elapsed wall time,
category breakdowns, dataset/prediction hashes, frozen specification and its hash,
scorer source hash, and Python version. Token counts are summed only when an
adapter explicitly returns them, with reported/unreported trial counts. Fixture
latency is replay overhead; local generation latency excludes checkpoint loading.
The exact bytes parsed are also the bytes hashed. Reports contain benchmark prompts,
answer keys, and model outputs; keep them as private evidence when using confidential
data and review or redact them before any publication. No monetary cost, accelerator
throughput, or end-to-end training latency is inferred.
Local Transformers package versions are recorded when that adapter runs.

Wilson 95% intervals are descriptive summaries of item/trial outcomes. Correlation
between repeated trials, related items, and shared templates invalidates treating
these as independent evidence of generalization. The report states this caveat.
For substantial research comparisons, use held-out representative data and item-
clustered or paired uncertainty analysis appropriate to the benchmark design.

## Local Transformers adapter

Install the pinned optional stack with `pip install -e '.[training]'`. This adapter
accepts an existing local checkpoint and a separately pinned local tokenizer directory. It uses
`local_files_only=True`, `trust_remote_code=False`, and the literal chat template in
the specification. Only full or merged local model checkpoints are supported. A
standalone PEFT `adapter_config.json` directory is rejected, because automatic base
model resolution could load an unpinned external artifact. Model and tokenizer
directory hashes are checked again after loading. It defaults to CPU and float32. `adapter.device` may be `cpu` or `cuda`, and
`adapter.dtype` may be `float32`, `bfloat16`, or `float16`. CUDA requires an available
device; BF16 additionally requires hardware BF16 support. Model weights and inputs
are moved to the selected device, and runtime device/dtype, CUDA version, device
name, and package versions are recorded. Unknown adapter fields are rejected.
The adapter does not offer distributed inference or arbitrary provider configuration.

1. Materialize a checkpoint directory and a tokenizer-only directory. External
   symlinks are rejected to keep each artifact self-contained. Both baseline and
   adapted checkpoints can reference the same frozen tokenizer directory.
2. Compute the directory identity, hashing sorted relative filenames and SHA256
   content digests:

   ```python
   from hessian_recipes.evaluation import checkpoint_sha256
   digest = checkpoint_sha256("/absolute/path/to/checkpoint")
   print(digest)
   tokenizer_digest = checkpoint_sha256("/absolute/path/to/tokenizer")
   print(tokenizer_digest)
   ```

3. Copy an example spec beside your experiment. Keep a correct relative path to the
   desired benchmark and its original file hash. Set `adapter` to:

   ```json
   {"type": "transformers", "path": "checkpoint", "sha256": "<digest>",
    "tokenizer_path": "tokenizer", "tokenizer_sha256": "<tokenizer_digest>"}
   ```

4. Set model/tokenizer IDs to descriptive artifact identifiers, model revision to
   the checkpoint digest, and tokenizer revision to the tokenizer directory digest. Copy the checkpoint's actual chat template
   literally into `model.chat_template`; no default template is guessed. The
   adapter defaults `tokenizer_path` to `path` if omitted, but a separate directory
   lets tokenizers retain the same identity when model weights change.
5. For Qwen thinking mode, set `model.chat_template_kwargs` to
   `{"enable_thinking": true}` (or explicitly `false` for a separate experiment).
   Only a boolean `enable_thinking` is accepted. These arguments reach the actual
   frozen local tokenizer's `apply_chat_template`; no prompt-text substitute is
   used. Both baseline and candidate must use the same mode.
6. Run `hessian-recipes eval experiment.json --output runs/local`.

A GPU configuration can add `"device": "cuda", "dtype": "bfloat16"` to `adapter`.
Choose a completion budget that includes the thinking tokens. These changes do
not create or request GPU allocations; use an already allocated GPU environment.

The seed policy is `sampling.seed + trial`; this initializes the generator for each
item/trial. Temperature zero selects greedy decoding. Maximum generated tokens are
enforced by Transformers and reported counts are checked against the manifest.
For stochastic sampling, optional `top_p` must lie in (0, 1], `top_k` must be a
nonnegative integer (zero disables top-k), and `min_p` must lie in [0, 1]. The
defaults are top-p one, top-k zero, and min-p zero. For a Qwen3 thinking experiment,
use the model's documented sampling policy, for example temperature 0.6, top-p 0.95,
top-k 20, and min-p zero, with identical settings for baseline and candidate.
These stochastic controls apply when temperature is positive; temperature zero
selects greedy decoding. The full sampling specification is compared exactly. It
uses repetition penalty one, one decoding beam, and EOS/padding identities from
the shared frozen tokenizer; it does not inherit checkpoint sampling defaults.
Identical seeds and versions do not promise bitwise reproducibility across different
hardware or backend versions. This path is provided as a real local generation
adapter; the default test suite exercises fixture and Inspect mock generation and
checkpoint pinning without downloading a checkpoint or running a training job.

## Trusted Python harness adapter

Integrate an existing harness programmatically; JSON does not import arbitrary
Python symbols. Declare `adapter` as `{"type":"callable","name":"my-harness",
"revision":"<immutable implementation commit>"}`, with pinned model/tokenizer
revisions. Call:

```python
from hessian_recipes.evaluation import run

# Implement this trusted application function in your own integration.
def generate(sample, trial, spec):
    # sample contains only id, prompt, category; scoring labels are excluded.
    output, prompt_tokens, completion_tokens = your_harness(
        prompt=sample["prompt"],
        seed=spec["sampling"]["seed"] + trial,
        max_new_tokens=spec["sampling"]["max_new_tokens"],
        temperature=spec["sampling"]["temperature"],
    )
    return {"output": output, "usage": {
        "prompt_tokens": prompt_tokens, "completion_tokens": completion_tokens}}

report = run("experiment.json", "runs/harness", generator=generate)
```

A text-only return is allowed and leaves token usage explicitly unreported. Adapters
must honor the frozen configuration, enforce deadlines/resource limits, and record
the actual implementation revision. The small synchronous runner cannot interrupt
a hung trusted callable or verify its claimed model identity. Use the existing
harness or Inspect to enforce execution limits. Only inject code you trust; the
runner's safe toy-plan scoring does not sandbox application callbacks.

## Inspect integration

Install `pip install -e '.[inspect]'`. The adapter defines a genuine `@task`, using
the same hash-checked samples and deterministic scorers. Inspect handles model
execution, logs, errors, concurrency, and provider support:

```bash
inspect eval src/hessian_recipes/inspect_tasks.py@development_benchmark \
  -T spec_path=benchmarks/reasoning_reliability/spec.json \
  --model mockllm/model
```

The built-in mock model is an offline integration smoke check; its default response
will be scored incorrect. Use your configured local Inspect provider for substantive
evaluation. Core imports never load Inspect. Optional tests actually run the task
with Inspect's mock provider and controlled `ModelOutput` objects, without keys.

Inspect accepts model/generation overrides outside this task specification. The
initial adapter rejects explicit local `chat_template_kwargs` and `min_p` (which
the pinned Inspect configuration does not expose); use the core local
Transformers adapter when a frozen thinking mode is required. Inspect
logs contain the effective runtime configuration and must be reviewed; the task
metadata explicitly marks this boundary. **Inspect logs are not currently imported
into the core JSON comparison/bundle format.** Pin the actual model, template,
sampling, budgets, versions, and provider configuration in your Inspect run rather
than treating the development fixture's synthetic model labels as runtime evidence.

The adapter follows the official [task API](https://inspect.aisi.org.uk/tasks.html)
and [custom scorer API](https://inspect.aisi.org.uk/custom-scorers.html). This
repository reuses Inspect rather than adding another execution service.

## Regression gate and remaining release work

`compare` fails closed unless both reports are complete and their aggregate metrics
agree with individual sample/trial evidence. It validates stored spec/dataset
snapshot hashes, reconstructs exact item/category identities, and re-scores every
completed output using the pinned scorer implementation. It matches dataset identity/version/hash,
item/trial/category coverage, sampling budgets, trial counts, scorer/harness versions
and source hash, execution mode, tokenizer identity/revision, chat template and thinking kwargs, and execution device/dtype.
Model ID/revision may differ because comparing checkpoints is the intended use.
Moving a dataset file does not change identity. Changing tokenizer or template is a
separate experiment requiring explicit review; this initial gate rejects it.

The default permitted accuracy drop is zero; `--max-drop 0.01` permits one percentage
point in overall accuracy **and in every category**. A subgroup regression fails
even if overall accuracy improves. Category deltas are recomputed from individual
outcomes. The gate does not perform a significance test; review practical effect
sizes and set meaningful domain-specific thresholds separately. Authored synthetic candidates make it possible to test
both pass and regression paths. JSON evidence is local review material, not a signed
authenticity guarantee against deliberate forgery.

Before offering an evaluation service: select the actual scientific benchmark
questions with teams, create independently reviewed held-out datasets and scoring
protocols, establish category and task-specific release criteria, add provider and
harness limits, validate strong baselines, and define how confidential evaluation
inputs and outputs are stored. Domain-specific protocols remain sketches in this
repository. No shared evaluation service or external publication is started by this
kit.

## Optional real CPU generation smoke

After the optional [CPU training smoke](../examples/training/cpu_smoke.py) has saved its
artifacts, run the second example in the same training environment:

```bash
python examples/training/cpu_smoke.py --output runs/cpu-smoke
python examples/evaluation_cpu_smoke.py \
  --baseline runs/cpu-smoke/tiny-model \
  --candidate runs/cpu-smoke/sft/checkpoint \
  --tokenizer runs/cpu-smoke/sft/checkpoint \
  --output runs/evaluation-cpu-smoke
```

This executes four real local generation trials per checkpoint with pinned
model/tokenizer tree hashes and eight-token completion budgets, then checks
report compatibility while permitting all score changes (`max_drop=1.0`). That
full tolerance is deliberately a wiring check and must not be copied into a real
release policy. The tiny synthetic checkpoints are not trained to solve these
fixtures; measured scores here make no model-quality or training-improvement claim.