# Reliability Foundation — SLOs and evals in CI

Every incident is argued over opinion instead of data.

Three weeks: reliability targets your team agrees on, error budgets tied to them, and evaluations that run in CI rather than in someone's head.

3 weeks · $12K-18K

Fixed for the agreed scope. Moves with the number of services instrumented.

## Who this is for

- Your team ships agent features and cannot agree on whether a release made things better or worse.
- QA is vibes-based: someone tries a few prompts and calls it fine.
- You want to say something specific about reliability to a customer or a board.

## Who this is not for

- You want an observability tool installed. Most of the tooling is open source and straightforward to stand up — the engagement is the layer that makes it mean something: objectives, error budgets, and evals wired into CI.
- You have no production traffic yet. There is nothing to set an objective against.

## What you get

- SLO definitions on task completion rate, mean tool calls, hallucination escape rate, and end-to-end latency.
- Error budgets tied to each objective, with the policy for what happens when one is spent.
- Agent trace instrumentation feeding dashboards your team will actually open.
- An evaluation suite running in CI, with regression diffs on every pull request.
- Alerting and runbooks for the failures that will actually page someone.

## How it runs

### Week 1: Define

What working means, agreed with the people who will be held to it. Objectives that cannot be measured get cut in this week, not defended for a quarter.

### Week 2: Instrument

Traces, dashboards, and the evaluation suite wired into CI so regressions surface on the pull request rather than in production.

### Week 3: Operate

Error budget policy, alerting thresholds, runbooks, and a dry run of an incident with your on-call.

## Reported experience

- 71% to 96%: SLO compliance across three client platforms, two quarters
- 99.9%: Uptime sustained across production containerized environments
- 900ms to 120ms: p99 query latency, through index and materialized-view redesign

## Evidence and limits

Meridian scheduling implementation (experimental, reviewed 2026-09-22): Synthetic replay and component checks are documented. The real-vLLM benchmark is marked unverified in this revision; no fleet-level latency or savings claim follows from it.

[Source](https://github.com/angelnicolasc/meridian/blob/f3e7cd2c127d6188c423bce18c9d4ace4454d323/README.md)

Anonymized client outcomes (author-reported, reviewed 2026-09-22): Reported by Nick Cerutti; client identities and raw operational records are not public. $200K/year is aggregate savings across engagements. $40K/day was projected exposure; 72 hours describes that incident, not a response or resolution guarantee. Zero-known-incident statements describe observations, not proof of absence.

[Source](https://nickcerutti.com/work)

[Inference economics: reproducible cost model](https://nickcerutti.com/evidence/inference-economics-sample.md) — Executable arithmetic with workload assumptions, quality gates and unmeasured latency.

## Questions

### What is a hallucination escape rate?

The share of responses that reach a user carrying a factual error the system should have caught. It is the hardest of the four objectives to measure and the one worth arguing about — the argument itself usually improves the product.

### Which tools do you use?

Whatever you already run, if it is adequate. OpenTelemetry, Prometheus and Grafana are the default because they are open, portable, and nobody has to renew a contract to keep their dashboards.

### Can this be part of the audit instead?

The audit tells you what your reliability posture is. This builds it. If you are unsure which you need, start with the audit and credit the fee.

## Contact

[Discuss this engagement](mailto:n@nickcerutti.com) · [Pricing and terms](https://nickcerutti.com/pricing)

Canonical page: https://nickcerutti.com/services/reliability-foundation