Tink Skills evaluation loop

01

Problem

An agent-skill evaluation can show a difference for the wrong reason. The model alias can drift. The control can see the skill, or the treatment can miss it. A grader can accept a known-wrong answer. Copied artifacts can stop matching the run.

02

Constraints

• Task, fixtures, model, tools, harness, and trial count stay fixed while skill availability changes.
• Exact model identifiers are required; aliases are not silently inferred.
• References must pass every declared grader, and declared wrong answers must fail those same graders.
• Paid trials require an explicit invocation budget; provider call count and cost remain unknown unless the harness reports them.

03

My decisions

I designed a paired runner that alternates treatment and control order. For each condition, it records the response, trace, runtime attestation, grading result, and SHA-256. Harness adapters isolate skill payloads and reject symlinked entries.

The aggregator does not trust summary booleans from a copied manifest. It resolves the allowed artifact paths and rehashes them. The report keeps mechanism validity, runtime attestation, routing, task success, usage coverage, and cost coverage separate.

04

Difficult tradeoffs

I built four harness adapters so the system could run across different agent environments. Their enforcement and trace visibility differ, so the report names the limits of each harness instead of implying uniform isolation.

Alternating order reduces order bias, but it cannot eliminate temporal drift. A local paired result is not statistical significance, causal proof, distribution readiness, security approval, or evidence that a skill wins everywhere.

05

Implementation

The system includes a suite auditor, model recommender, paired runner, four runtime adapters, attestation parser, and fail-closed aggregator. It supports deterministic graders and model rubrics. Before a paid trial, it validates both correct and counter-reference answers and freezes the target. The artifacts remain addressable so the result can be checked again later.

• PlanExact model · harness · cases · invocation budget
• RunPaired conditions · counterbalanced order · isolated payloads
• AttestModel identity · treatment exposure · control isolation
• AggregateRehashed artifacts · layered verdicts · coverage limits

06

Verification

Tests cover model-alias rejection, target and judge attestation failures, payload symlinks, control and output isolation for each harness, interruption handling, alternating order, artifact drift, control exposure, and invalid grading summaries. The published main commit passed the validation workflow, including the evaluator health check, Python lint, package inventory, and link checks.

07

Outcome

The public Tink Skills repository includes three installable packages, one of them this evaluation loop. It breaks a skill comparison into checks that can be rerun, and it narrows the conclusion when the run does not prove the mechanism.