Skip to content

Agent Governance / Evidence / Inference Economics

A control score is not an evidence dossier.

A yes/no checklist can organize a review. It cannot show that a control worked. Two small, downloadable examples make the missing steps concrete: a tool authorization test and an inference-cost model.

By Nick Cerutti — AI ArchitectPublished 3 min read

Start with a falsifiable control claim

'Our agents use least privilege' is too broad to test. Narrow it to an actor, resource, action and environment: the support agent may read the assigned tenant's ticket; it may not read another tenant's ticket or initiate a refund. Attach the policy version and identify the enforcement point. A model instruction asking an agent not to do something is not the same as a permission check that rejects the action.

RecordWhat makes it reviewable
Control claimActor, action, resource, tenant and expected decision
TestInputs, policy version, execution command and expected result
ObservationActual decision, test time and environment
LimitationPaths not exercised, missing integrations and unresolved exceptions
OwnershipPerson responsible for the control and the next review

Test the denial path before counting a control as evidenced

The sample dossier describes a synthetic support-agent scenario. Its companion script exercises allowed reads, denied cross-tenant access and restricted writes. The assertions make the intended boundary inspectable. They do not prove that a real tool server authenticates identities correctly, that every endpoint calls the check or that a production deployment uses the same policy.

Those are separate integration tests. A useful follow-up would send the same requests through the actual gateway, repeat them with revoked credentials and inspect the resulting redacted audit events. Store the correlation identifier, policy decision and relevant metadata. Avoid collecting raw prompts or secrets merely to make a dossier look thorough.

# Download the script linked below, then run with Node.js.
node reproduce-samples.mjs
# Assertions fail the process if an expected control decision changes.

Keep cost arithmetic separate from measured savings

The inference example uses hypothetical costs: 10,000 tasks per day, $0.0018 for a small-model call and $0.012 for a premium-model call. Route 70% to the small model and escalate 20% of those. That produces 7,000 small calls and 4,400 premium calls. The escalated tasks incur both costs.

baseline = 10_000 * 0.012                   # $120/day
routed   = 7_000 * 0.0018 + 4_400 * 0.012   # $65.40/day
reduction = 1 - routed / baseline          # 45.5%

These inputs are assumptions, not provider quotes or a measured workload. An apparent reduction is useful only if task success, error severity and latency remain acceptable. Replay a representative evaluation set before a canary; measure escalation frequency and the combined latency of both calls. The example deliberately leaves those measurements unresolved.

Integrity is a narrower claim than truth

The script hashes a sample event, changes its decision and verifies that the digest changes. That demonstrates digest sensitivity. Someone able to rewrite both the event and the reference digest can still substitute a consistent pair. A production evidence chain needs a trustworthy collection path, access controls and a protected reference. Hashing an invented event faithfully preserves an invented event.

Make uncertainty part of the handoff

The controls scorecard gives equal weight to twenty self-reported answers. It withholds a percentage until every question is answered and exports the responses and gaps locally as Markdown or JSON. A 100% answer set is still self-report. The next artifact should connect each answer to an owner, a test and a dated observation.

For an audit or insurance discussion, agree on the recipient's evidence requirements first. There is no universal score here that predicts coverage, premium or legal compliance. The dossier helps a reviewer examine what is implemented and what remains untested; the reviewer retains the decision.

Key takeaways

  • Separate a reported control from an observed test result and the scope of that test.
  • Record denied actions as well as allowed actions; test the boundary the system actually enforces.
  • Price escalations as two calls when both models run. Savings estimates need quality and latency gates.
  • A digest detects a change against a trusted reference; it does not establish that the original event was true.

Frequently asked questions

Can I reproduce the examples?

Yes. Download the dependency-free Node.js script from the evidence samples and run it locally. It uses synthetic records and hypothetical costs; no account or uploaded data is required.

Does a complete score mean an agent is insurable?

No. The score summarizes answers and has not been validated as an underwriting model. An insurer or broker determines the relevant requirements.

What is still missing after the script passes?

Production integration tests, identity and deployment verification, representative quality and latency measurements, and a reviewer-approved scope. The samples list these limits explicitly.

Sources & references

Related notes

Updates on new engineering notes: production AI infrastructure, with the arithmetic shown.

Request by email

n@nickcerutti.com · Include the question or project you have in mind.

Apply this to your system

Use the scorecard to locate reported gaps, then scope the tests and evidence needed for your actual system. The related incident case illustrates why spend controls need an enforcement point.