Agent Governance / Evidence / Inference Economics
A control score is not an evidence dossier.
A yes/no checklist can organize a review. It cannot show that a control worked. Two small, downloadable examples make the missing steps concrete: a tool authorization test and an inference-cost model.
Start with a falsifiable control claim
'Our agents use least privilege' is too broad to test. Narrow it to an actor, resource, action and environment: the support agent may read the assigned tenant's ticket; it may not read another tenant's ticket or initiate a refund. Attach the policy version and identify the enforcement point. A model instruction asking an agent not to do something is not the same as a permission check that rejects the action.
| Record | What makes it reviewable |
|---|---|
| Control claim | Actor, action, resource, tenant and expected decision |
| Test | Inputs, policy version, execution command and expected result |
| Observation | Actual decision, test time and environment |
| Limitation | Paths not exercised, missing integrations and unresolved exceptions |
| Ownership | Person responsible for the control and the next review |
Test the denial path before counting a control as evidenced
The sample dossier describes a synthetic support-agent scenario. Its companion script exercises allowed reads, denied cross-tenant access and restricted writes. The assertions make the intended boundary inspectable. They do not prove that a real tool server authenticates identities correctly, that every endpoint calls the check or that a production deployment uses the same policy.
Those are separate integration tests. A useful follow-up would send the same requests through the actual gateway, repeat them with revoked credentials and inspect the resulting redacted audit events. Store the correlation identifier, policy decision and relevant metadata. Avoid collecting raw prompts or secrets merely to make a dossier look thorough.
# Download the script linked below, then run with Node.js. node reproduce-samples.mjs # Assertions fail the process if an expected control decision changes.
Keep cost arithmetic separate from measured savings
The inference example uses hypothetical costs: 10,000 tasks per day, $0.0018 for a small-model call and $0.012 for a premium-model call. Route 70% to the small model and escalate 20% of those. That produces 7,000 small calls and 4,400 premium calls. The escalated tasks incur both costs.
baseline = 10_000 * 0.012 # $120/day routed = 7_000 * 0.0018 + 4_400 * 0.012 # $65.40/day reduction = 1 - routed / baseline # 45.5%
These inputs are assumptions, not provider quotes or a measured workload. An apparent reduction is useful only if task success, error severity and latency remain acceptable. Replay a representative evaluation set before a canary; measure escalation frequency and the combined latency of both calls. The example deliberately leaves those measurements unresolved.
Integrity is a narrower claim than truth
The script hashes a sample event, changes its decision and verifies that the digest changes. That demonstrates digest sensitivity. Someone able to rewrite both the event and the reference digest can still substitute a consistent pair. A production evidence chain needs a trustworthy collection path, access controls and a protected reference. Hashing an invented event faithfully preserves an invented event.
Make uncertainty part of the handoff
The controls scorecard gives equal weight to twenty self-reported answers. It withholds a percentage until every question is answered and exports the responses and gaps locally as Markdown or JSON. A 100% answer set is still self-report. The next artifact should connect each answer to an owner, a test and a dated observation.
For an audit or insurance discussion, agree on the recipient's evidence requirements first. There is no universal score here that predicts coverage, premium or legal compliance. The dossier helps a reviewer examine what is implemented and what remains untested; the reviewer retains the decision.
Key takeaways
- Separate a reported control from an observed test result and the scope of that test.
- Record denied actions as well as allowed actions; test the boundary the system actually enforces.
- Price escalations as two calls when both models run. Savings estimates need quality and latency gates.
- A digest detects a change against a trusted reference; it does not establish that the original event was true.
Frequently asked questions
Can I reproduce the examples?
Yes. Download the dependency-free Node.js script from the evidence samples and run it locally. It uses synthetic records and hypothetical costs; no account or uploaded data is required.
Does a complete score mean an agent is insurable?
No. The score summarizes answers and has not been validated as an underwriting model. An insurer or broker determines the relevant requirements.
What is still missing after the script passes?
Production integration tests, identity and deployment verification, representative quality and latency measurements, and a reviewer-approved scope. The samples list these limits explicitly.
Sources & references
Related notes
- How to publish infrastructure work you can't fully benchmark
Most infrastructure side projects are written on a laptop and benchmarked on a slide. A per-component verification column is cheaper than a bigger claim, and worth more.
- The Cost You Can't See
Itemising what one architectural component of a 284B model actually costs to serve - and why the published description of it was wrong by 2x.
Updates on new engineering notes: production AI infrastructure, with the arithmetic shown.
Request by emailn@nickcerutti.com · Include the question or project you have in mind.
Apply this to your system
Use the scorecard to locate reported gaps, then scope the tests and evidence needed for your actual system. The related incident case illustrates why spend controls need an enforcement point.