Evaluation evidence kit

For measured model experiments beyond the development fixtures below, see the German Qwen MGSM evaluation and interactive agent plus retention evaluation. Their SFT results and agent-RL results retain separate metrics and limitations. Gym qualification establishes integration and parameter updates, without a new held-out quality claim.

The kit implements a small offline runner and an optional real Inspect task adapter. Its outputs support comparing experiments and reviewing a release candidate; they never automatically certify a release. Every report has release_eligible: false.

The proposed programme topics are German RAG with cited evidence and business-communication agent sequences. Their real tasks and specialized scorers are not implemented. The two provisional tracks are tiny synthetic development fixtures for exercising this repository. Fixture results must not be described as model capabilities or measured training gains.

Run the complete offline demonstration

From the repository clone, after pip install -e .:

hessian-recipes demo --output runs/demo
hessian-recipes eval benchmarks/reasoning_reliability/spec.json --output runs/baseline
hessian-recipes eval benchmarks/reasoning_reliability/spec-candidate.json --output runs/candidate
hessian-recipes compare runs/baseline/report.json runs/candidate/report.json

Every evaluation writes report.json and report.md. The synthetic reasoning baseline is deliberately 50% correct and its synthetic candidate 100%; the workflow baseline is deliberately 1/3 correct and its synthetic candidate 100%. These values verify wiring only. Each trial replays a public authored response, without model, GPU, network, API credentials, or inferred token counts. Identical fixture trials do not add statistical information.

Existing report.json or report.md files cause an error before model loading; choose a fresh output directory for each experiment.

Evaluation exits successfully when execution coverage is complete, even if answers are incorrect. A regression gate exits unsuccessfully when comparison fails. Malformed manifests and hash failures are errors before generation. Runtime adapter failures remain explicit per-trial infrastructure failures; they are never silently removed from accuracy. Incomplete runs have no headline accuracy or confidence interval and cannot pass comparison. scored_accuracy is a diagnostic on the successfully executed subset, with its coverage shown separately.

Frozen specification

The example spec.json files are executable references. All file and directory paths resolve relative to the specification file's parent, independently of current working directory. Required fields are:

Fixture-only revisions may use the explicit fixture-v1 marker because no model is executed. Nonfixture revisions must be immutable 40-character Git SHA1 or 64-character SHA256 identities. Do not use mutable main or latest revisions.

JSONL samples have unique id, prompt, category, and a scorer. exact and numeric have an answer; exact comparison strips surrounding whitespace and remains case-sensitive. Numeric scoring accepts exactly one finite decimal number, including scientific notation, with exact decimal equality. It does not search a reasoning transcript for a plausible answer. Prompts explicitly request the final answer in that format.

The optional final_numeric scorer supports reasoning models: it accepts a whole canonical decimal or exactly one terminal <answer>number</answer> block after reasoning. It never extracts intermediate chain-of-thought numbers. Duplicate tags, unclosed <think> sections, trailing prose, nonfinite numbers, leading plus signs, leading zeros, and German decimal commas are rejected. Ask the model for a decimal point and the explicit final-answer format. Decimal equality remains exact.

The workflow scorer takes workflow.initial_state, target_state, and max_steps. Generated output must be exactly an actions object containing a list of toy read, reserve, or release calls. A deterministic state machine validates arguments, executes bounded in-memory state changes, and compares the final state. Unknown tools, invalid quantities, extra arguments, unavailable stock, and excessive steps are incorrect answers. A malformed plan is not an infrastructure failure. This measures generated plan validity and terminal state in a toy environment; interactive tool recovery and real agent harness behavior require separate tasks. Generated Python, shell commands, network operations, and real tools are never run.

The runner records exact UTF-8 specification and benchmark snapshots, full individual outputs, scores, error types, elapsed wall time, category breakdowns, dataset/prediction hashes, frozen specification and its hash, scorer source hash, and Python version. Token counts are summed only when an adapter explicitly returns them, with reported/unreported trial counts. Fixture latency is replay overhead; local generation latency excludes checkpoint loading. The exact bytes parsed are also the bytes hashed. Reports contain benchmark prompts, answer keys, and model outputs; keep them as private evidence when using confidential data and review or redact them before any publication. No monetary cost, accelerator throughput, or end-to-end training latency is inferred. Local Transformers package versions are recorded when that adapter runs.

Wilson 95% intervals are descriptive summaries of item/trial outcomes. Correlation between repeated trials, related items, and shared templates invalidates treating these as independent evidence of generalization. The report states this caveat. For substantial research comparisons, use held-out representative data and item- clustered or paired uncertainty analysis appropriate to the benchmark design.

Local Transformers adapter

Install the pinned optional stack with pip install -e '.[training]'. This adapter accepts an existing local checkpoint and a separately pinned local tokenizer directory. It uses local_files_only=True, trust_remote_code=False, and the literal chat template in the specification. Only full or merged local model checkpoints are supported. A standalone PEFT adapter_config.json directory is rejected, because automatic base model resolution could load an unpinned external artifact. Model and tokenizer directory hashes are checked again after loading. It defaults to CPU and float32. adapter.device may be cpu or cuda, and adapter.dtype may be float32, bfloat16, or float16. CUDA requires an available device; BF16 additionally requires hardware BF16 support. Model weights and inputs are moved to the selected device, and runtime device/dtype, CUDA version, device name, and package versions are recorded. Unknown adapter fields are rejected. The adapter does not offer distributed inference or arbitrary provider configuration.

1. Materialize a checkpoint directory and a tokenizer-only directory. External

symlinks are rejected to keep each artifact self-contained. Both baseline and adapted checkpoints can reference the same frozen tokenizer directory.

1. Compute the directory identity, hashing sorted relative filenames and SHA256

content digests:

``python from hessian_recipes.evaluation import checkpoint_sha256 digest = checkpoint_sha256("/absolute/path/to/checkpoint") print(digest) tokenizer_digest = checkpoint_sha256("/absolute/path/to/tokenizer") print(tokenizer_digest)

1. Copy an example spec beside your experiment. Keep a correct relative path to the

desired benchmark and its original file hash. Set adapter to:

``json {"type": "transformers", "path": "checkpoint", "sha256": "<digest>", "tokenizer_path": "tokenizer", "tokenizer_sha256": "<tokenizer_digest>"}

1. Set model/tokenizer IDs to descriptive artifact identifiers, model revision to

the checkpoint digest, and tokenizer revision to the tokenizer directory digest. Copy the checkpoint's actual chat template literally into model.chat_template; no default template is guessed. The adapter defaults tokenizer_path to path if omitted, but a separate directory lets tokenizers retain the same identity when model weights change.

1. For Qwen thinking mode, set model.chat_template_kwargs to

{"enable_thinking": true} (or explicitly false for a separate experiment). Only a boolean enable_thinking is accepted. These arguments reach the actual frozen local tokenizer's apply_chat_template; no prompt-text substitute is used. Both baseline and candidate must use the same mode.

1. Run hessian-recipes eval experiment.json --output runs/local.

A GPU configuration can add "device": "cuda", "dtype": "bfloat16" to adapter. Choose a completion budget that includes the thinking tokens. These changes do not create or request GPU allocations; use an already allocated GPU environment.

The seed policy is sampling.seed + trial; this initializes the generator for each item/trial. Temperature zero selects greedy decoding. Maximum generated tokens are enforced by Transformers and reported counts are checked against the manifest. For stochastic sampling, optional top_p must lie in (0, 1], top_k must be a nonnegative integer (zero disables top-k), and min_p must lie in [0, 1]. The defaults are top-p one, top-k zero, and min-p zero. For a Qwen3 thinking experiment, use the model's documented sampling policy, for example temperature 0.6, top-p 0.95, top-k 20, and min-p zero, with identical settings for baseline and candidate. These stochastic controls apply when temperature is positive; temperature zero selects greedy decoding. The full sampling specification is compared exactly. It uses repetition penalty one, one decoding beam, and EOS/padding identities from the shared frozen tokenizer; it does not inherit checkpoint sampling defaults. Identical seeds and versions do not promise bitwise reproducibility across different hardware or backend versions. This path is provided as a real local generation adapter; the default test suite exercises fixture and Inspect mock generation and checkpoint pinning without downloading a checkpoint or running a training job.

Trusted Python harness adapter

Integrate an existing harness programmatically; JSON does not import arbitrary Python symbols. Declare adapter as {"type":"callable","name":"my-harness", "revision":"<immutable implementation commit>"}, with pinned model/tokenizer revisions. Call:

from hessian_recipes.evaluation import run

# Implement this trusted application function in your own integration.
def generate(sample, trial, spec):
    # sample contains only id, prompt, category; scoring labels are excluded.
    output, prompt_tokens, completion_tokens = your_harness(
        prompt=sample["prompt"],
        seed=spec["sampling"]["seed"] + trial,
        max_new_tokens=spec["sampling"]["max_new_tokens"],
        temperature=spec["sampling"]["temperature"],
    )
    return {"output": output, "usage": {
        "prompt_tokens": prompt_tokens, "completion_tokens": completion_tokens}}

report = run("experiment.json", "runs/harness", generator=generate)

A text-only return is allowed and leaves token usage explicitly unreported. Adapters must honor the frozen configuration, enforce deadlines/resource limits, and record the actual implementation revision. The small synchronous runner cannot interrupt a hung trusted callable or verify its claimed model identity. Use the existing harness or Inspect to enforce execution limits. Only inject code you trust; the runner's safe toy-plan scoring does not sandbox application callbacks.

Inspect integration

Install pip install -e '.[inspect]'. The adapter defines a genuine @task, using the same hash-checked samples and deterministic scorers. Inspect handles model execution, logs, errors, concurrency, and provider support:

inspect eval src/hessian_recipes/inspect_tasks.py@development_benchmark \
  -T spec_path=benchmarks/reasoning_reliability/spec.json \
  --model mockllm/model

The built-in mock model is an offline integration smoke check; its default response will be scored incorrect. Use your configured local Inspect provider for substantive evaluation. Core imports never load Inspect. Optional tests actually run the task with Inspect's mock provider and controlled ModelOutput objects, without keys.

Inspect accepts model/generation overrides outside this task specification. The initial adapter rejects explicit local chat_template_kwargs and min_p (which the pinned Inspect configuration does not expose); use the core local Transformers adapter when a frozen thinking mode is required. Inspect logs contain the effective runtime configuration and must be reviewed; the task metadata explicitly marks this boundary. Inspect logs are not currently imported into the core JSON comparison/bundle format. Pin the actual model, template, sampling, budgets, versions, and provider configuration in your Inspect run rather than treating the development fixture's synthetic model labels as runtime evidence.

The adapter follows the official task API and custom scorer API. This repository reuses Inspect rather than adding another execution service.

Regression gate and remaining release work

compare fails closed unless both reports are complete and their aggregate metrics agree with individual sample/trial evidence. It validates stored spec/dataset snapshot hashes, reconstructs exact item/category identities, and re-scores every completed output using the pinned scorer implementation. It matches dataset identity/version/hash, item/trial/category coverage, sampling budgets, trial counts, scorer/harness versions and source hash, execution mode, tokenizer identity/revision, chat template and thinking kwargs, and execution device/dtype. Model ID/revision may differ because comparing checkpoints is the intended use. Moving a dataset file does not change identity. Changing tokenizer or template is a separate experiment requiring explicit review; this initial gate rejects it.

The default permitted accuracy drop is zero; --max-drop 0.01 permits one percentage point in overall accuracy and in every category. A subgroup regression fails even if overall accuracy improves. Category deltas are recomputed from individual outcomes. The gate does not perform a significance test; review practical effect sizes and set meaningful domain-specific thresholds separately. Authored synthetic candidates make it possible to test both pass and regression paths. JSON evidence is local review material, not a signed authenticity guarantee against deliberate forgery.

Before offering an evaluation service: select the actual scientific benchmark questions with teams, create independently reviewed held-out datasets and scoring protocols, establish category and task-specific release criteria, add provider and harness limits, validate strong baselines, and define how confidential evaluation inputs and outputs are stored. Domain-specific protocols remain sketches in this repository. No shared evaluation service or external publication is started by this kit.

Optional real CPU generation smoke

After the optional CPU training smoke has saved its artifacts, run the second example in the same training environment:

python examples/training/cpu_smoke.py --output runs/cpu-smoke
python examples/evaluation_cpu_smoke.py \
  --baseline runs/cpu-smoke/tiny-model \
  --candidate runs/cpu-smoke/sft/checkpoint \
  --tokenizer runs/cpu-smoke/sft/checkpoint \
  --output runs/evaluation-cpu-smoke

This executes four real local generation trials per checkpoint with pinned model/tokenizer tree hashes and eight-token completion budgets, then checks report compatibility while permitting all score changes (max_drop=1.0). That full tolerance is deliberately a wiring check and must not be copied into a real release policy. The tiny synthetic checkpoints are not trained to solve these fixtures; measured scores here make no model-quality or training-improvement claim.