Error Budgets and SLOs: Build Reliable Software 2026

By Nikolay Sapunov, founder & CEO of Fora Soft — 20+ years shipping real-time and reliability-critical software. Reviewed and updated July 2026.

Why Fora Soft wrote this playbook

Fora Soft has shipped real-time video, e-learning, telemedicine and streaming products since 2005 — 250+ projects, 50 in-house engineers, and more than a thousand person-years of postmortems behind us. Every one of those products lives or dies by reliability: a frozen call costs a clinic visit, a dropped live-shopping stream costs a sales hour, a flaky lecture costs a school day. The failure patterns repeat across product categories, and so do the fixes.

This playbook collapses what we learned shipping BrainCert (a WebRTC-first LMS now at $3M ARR in 2024, 100K+ customers and 500M+ classroom minutes), Sprii (a live-shopping platform behind €365M+ in seller sales across 3,000+ brands), and ProVideoMeeting (an enterprise video-conferencing stack on Kurento and FreeSWITCH) into one decision framework. It’s written for founders, CTOs and engineering leads weighing “ship fast and patch later” against “invest in resilience now.” If you’d rather hand the whole thing to a team that has done it, that’s what our custom software development practice is for.

The numbers below are pulled from public sources: the Google SRE workbook, the AWS Well-Architected reliability pillar, ITIC and Gartner downtime surveys, the 2024 DORA report, and the CrowdStrike and Datadog postmortems. Where we cite our own client work we use ranges, not specifics, to respect NDAs.

Figure 1. Pick a reliability target and the allowable monthly downtime — the error budget — is just arithmetic.

What “reliable” actually means in 2026

Crash-proof software doesn’t exist. What exists is software that fails inside a budget, recovers faster than the user notices, and degrades gracefully instead of breaking outright. That’s what modern Site Reliability Engineering (SRE) means. The Google SRE book formalized the vocabulary a decade ago, and the rest of the industry converged on it.

Three terms anchor every reliability conversation. A Service Level Indicator (SLI) is the metric you measure — availability, p99 latency, error rate. A Service Level Objective (SLO) is the target for that metric over a window: “99.95% of requests succeed under 250 ms over 30 days.” A Service Level Agreement (SLA) is what you contractually owe customers if you miss the SLO, usually a credit. The error budget is what’s left over: a 99.95% SLO gives you 21.6 minutes of allowable downtime a month, and not a minute more before you stop shipping features and harden.

That single rule — spend the error budget on features while it’s healthy, spend it on hardening when it’s burning — replaces vague reliability rhetoric with a measurable trade-off a team can agree on without an argument. Figure 1 shows how each SLO tier converts into a monthly error budget.

What is an error budget?

An error budget is the amount of unreliability your SLO permits over a window. Subtract the SLO from 100%, multiply by the minutes in the period, and that’s the budget: a 99.95% monthly SLO allows 0.05% of 43,200 minutes, or 21.6 minutes of downtime a month. A 99.9% SLO allows 43.2 minutes; a 99.99% SLO, just 4.3 (Figure 1).

The number matters because of the decision it settles. While the budget is healthy you ship features at full speed; when you’re burning it — a bad deploy, a flaky dependency — you freeze launches and spend what’s left on hardening. The Google SRE workbook error-budget policy is the canonical write-up. The practical trick is wiring the burn-rate alert into the same pipeline that blocks releases, so the rule enforces itself instead of living on a wiki.

What downtime actually costs in 2026

The reliability conversation always comes back to money. A few reference numbers are worth memorising before the next budget meeting.

The reliability work that prevents a single hour of enterprise downtime usually pays for itself the same week. That’s the conversation a CFO needs framed in dollars, not engineering jargon.

The eight failure categories behind most outages

When we run a reliability audit on a client codebase, the same eight failure categories show up over and over. Pattern-matching against them at the start saves a week of investigation. Figure 3 maps each one to the patterns that contain it.

1. Memory leaks. Unclosed database connections, leaked event listeners, caches that only grow. Symptom: latency that creeps up and resets on restart. Fix: heap profiling and lifecycle audits.

2. Cascading failures and retry storms. One downstream API slows to 8 seconds; everyone retries; the upstream pool exhausts; the next hop falls over. Fix: circuit breakers, bulkheads, retry budgets.

3. Race conditions. Two requests update the same row, the wrong one wins, data corrupts silently. Fix: optimistic locking, transactional boundaries, idempotency keys.

4. Unhandled exceptions. A null pointer in a rarely-hit branch takes down the whole worker. Fix: structured error handling, panic budgets, sentinel-error monitoring.

5. Dependency and supply-chain failures. A third-party SDK ships a bad release at 3am and your CI pulls it in. The CrowdStrike shape. Fix: pinned versions, staged rollouts, a vendor incident channel.

6. Deployment-induced failures. The 2024 DORA data makes change-failure rate the single biggest reliability lever. Fix: progressive rollout, automated rollback gates, contract tests.

7. Infrastructure outages. AWS us-east-1 has had at least one major incident nearly every year since 2017. Fix: multi-AZ at minimum, multi-region for tier-1 services, runbooks for cloud failures.

8. Configuration drift. A flag set on one server, missed on another; staging works, production explodes. Fix: GitOps, infrastructure-as-code, config linting in CI.

Six patterns that prevent most crashes

There’s no single reliability framework you can install. Resilience comes from layering a small set of well-understood patterns. Six carry most of the weight.

Circuit breakers

A circuit breaker tracks the failure rate against a downstream service. When failures cross a threshold, the breaker trips open and your code fails fast for a cooldown instead of piling up requests. Production-grade libraries: resilience4j for the JVM, Polly for .NET, gobreaker for Go.

Bulkheads

Bulkheads isolate failure domains: one thread pool per downstream, one connection pool per database, one namespace per blast radius. When the payments service melts, the rest of the app stays up. Datadog’s March 2023 outage went total partly because a shared failure domain had no bulkhead between it and everything else.

Retry with exponential backoff and jitter

Retries without backoff turn one failure into a stampede. The standard recipe: 100 ms, then 200, 400, 800, with random jitter (±25%) to spread the herd. Cap at five attempts and a 30-second total. The AWS Architecture Blog’s “Exponential Backoff And Jitter” is the canonical reference implementation.

Idempotency keys

Mutating endpoints accept a client-supplied Idempotency-Key header. The server stores the result for 24 hours; retries return the same response with no duplicate side effects. Stripe popularised the pattern; every payments and reservation API ships it now.

Feature flags and progressive rollout

Every risky change ships behind a flag, rolled out 1% → 10% → 50% → 100% with a metric gate at each step. LaunchDarkly, GrowthBook (open source), and Unleash (open source) are the popular options. With progressive rollout, the CrowdStrike push would have been a 1% incident instead of 8.5M hosts worldwide.

OpenTelemetry-native observability

Emit logs, metrics and traces in the OpenTelemetry standard, then route them to whichever backend you prefer (Datadog, New Relic, Honeycomb, Grafana Cloud, Sentry). Vendor lock-in is dead. The real shift is alerting on SLO burn rate instead of raw thresholds — that one change does more for alert fatigue than any dashboard.

Figure 3. Eight failure modes we see in audits, mapped to the patterns that actually contain each one.

The reliability stack, layer by layer

The six patterns aren’t a menu you pick from at random. They stack. Each layer assumes the one beneath it already works, and skipping a layer quietly loads its risk onto everything above. Here’s the order we build in, bottom to top.

At the base sit the foundations — automated tests, CI/CD, infrastructure-as-code, multi-AZ by default. On top of that come SLOs and error budgets, the one number that decides when to ship and when to harden. Then observability, because an SLO you can’t measure is a wish. Then the resilience patterns from the last section, then progressive delivery so a bad change is a rollback rather than an incident, and finally process and culture: blameless postmortems, sane on-call, and disaster-recovery drills that actually run.

The order matters because the value flows upward. Circuit breakers you can’t observe hide their own tripping. Feature flags without an SLO have no signal to gate on. Build from the bottom and each layer makes the next one pay off.

Figure 2. The reliability stack. Each layer depends on the one beneath it; skip a layer and the ones above inherit the risk.

Five famous outages and the lesson each teaches

Public postmortems are the closest thing the industry has to free education. Five recent ones every founder should be able to summarise.

Mobile reliability: the metrics that move retention

For mobile-first products — calling apps, e-learning, streaming, telemedicine — the server-side SLO is only half the story. The other half lives on the device, and Firebase Crashlytics or Sentry is where it gets measured.

Crash-free user rate. The headline metric. Baseline is around 99% on Android and 99.5% on iOS; premium products run 99.9%+. A half-point drop usually shows up as a one-star wave on the store within two weeks.

ANR rate (Android). Application Not Responding under 0.47% of daily active users — that’s Google Play’s own “bad behaviour” threshold, and crossing it gets you down-ranked silently. The cause is almost always work that drifted onto the main thread: image decoding, deserialization in adapters, IO inside a Compose recomposition.

Foreground-service watchdog. Calling and streaming apps run an active FOREGROUND_SERVICE_TYPE_PHONE_CALL or FOREGROUND_SERVICE_TYPE_MEDIA_PLAYBACK. Since Android 14 (2023, tightened again in 15 and 16) you have about five seconds to call startForeground or the OS throws ForegroundServiceDidNotStartInTimeException — the call drops and the user blames the app. We wrote up the patterns in our foreground services and deep links on Android 14 guide.

Network resilience. Offline-first sync, retry budgets that defer to background workers, backoff on every API call, and Sentry breadcrumbs that capture the network class (LTE / 5G / WiFi / offline) at crash time.

OEM matrix testing. A stock Pixel doesn’t predict Xiaomi behaviour. We bake the top five OEMs into every Android client’s CI device matrix — the same posture in our custom Android call notifications playbook.

Real-time and AI workloads: a different bar

Standard SLOs were written for request/response services. Real-time video and LLM-powered features need extra metrics and extra defences.

Real-time video. The numbers that matter are mean opinion score (MOS), packet loss (target under 2%), and round-trip p95 (target under 200 ms). Size the SFU/MCU for two times peak load. If you’re new to how video is actually captured and encoded, our Learn primer on what digital video is is the ground floor; the trade-offs live in our custom video conferencing architecture guide.

LLM-powered features. Hallucination handling, token-cost guardrails, and multi-model fallback (Claude, OpenAI, Llama). Tools like Langfuse, LangSmith and Helicone give you observability tuned to model behaviour. Treat the model as another flaky dependency: circuit-break it, retry it, cap its blast radius. That’s the core of how we ship AI integration without betting the product on one vendor’s uptime.

Streaming. Player buffer-empty rate under 1%, time-to-first-frame under 2 seconds, and an ABR ladder tested on throttled connections. One comms platform we run, Nucleus, clears 600M+ AI phone minutes a month under SOC 2 and HIPAA — at that volume a 0.1% error rate is 600,000 failed minutes, so it ships a synthetic call every 60 seconds that pages on-call the moment MOS drops.

DORA 2024: how elite teams ship safely

Google’s annual DORA (DevOps Research and Assessment) report measures four metrics across thousands of engineering organisations and clusters them into elite, high, medium and low performers. The 2024 numbers below are the benchmark every reliability programme should track.

The counter-intuitive finding from a decade of DORA data: elite teams ship more often and fail less. Reliability and velocity are correlated, not opposed — the shared root cause is rigorous progressive rollout and observability, which give engineers the confidence to ship. The 2024 report added rework rate as a fifth, stability-focused metric, and reported an oddity you can see in Figure 4: for the first time the medium cluster posted a lower change-failure rate (around 10%) than the high cluster (around 20%). Elite still sits near 5%, low performers at 40%+.

Figure 4. DORA 2024: change failure rate by performance tier, with elite targets for the other three keys.

Organization and process: the human side of reliability

Tools alone don’t produce reliability. The same patterns deployed inside a punitive postmortem culture will fail; deployed inside a blameless on-call culture they thrive. Five process habits every reliable engineering org shares.

1. Blameless postmortems. The Etsy template is the public reference. Focus on the system, not the operator. Every postmortem produces three to five action items with owners and dates.

2. On-call rotations with handoff rituals. Twelve-hour shifts at most, a fixed weekly handoff with explicit ownership transfer, and pages counted as work, not heroics.

3. Pre-prod that matches prod. Same operator versions, same secrets manager, same network topology. Differences are what cause the “works on staging” bug that always surfaces at midnight.

4. Disaster-recovery drills. Quarterly. Restore from backup, fail over to the secondary region, kill the primary database on purpose. Untested DR is theoretical DR.

5. A test pyramid, not an ice-cream cone. Many fast unit tests, fewer integration tests, fewest end-to-end. The inverted shape is the second-most-common reliability anti-pattern we see, behind missing observability.

Mini case: 6-hour MTTR to 22 minutes

A US-based B2B SaaS client, similar in profile to BrainCert, came to us with two recent multi-hour outages and an enterprise customer threatening to invoke its SLA-credit clause. They had Sentry installed but no SLOs, no error budgets, and a single AWS region. Mean time to recovery sat at six hours.

Our four-week plan: define a 99.95% availability SLO with p95 latency under 350 ms; instrument every endpoint with OpenTelemetry into a Honeycomb backend with SLO burn-rate alerts; wrap the three external APIs behind both incidents in circuit breakers; and add progressive feature-flag rollout with GrowthBook. The final week ran two chaos exercises — kill the primary database, kill the auth service — against a written runbook.

Over the next 60 days, MTTR dropped from six hours to 22 minutes (elite on the DORA scale), change-failure rate from 31% to 12%, and customer-impacting incidents to zero. The enterprise customer renewed. Total effort was about 2.5 engineer-months across four calendar weeks; agent-assisted engineering let us reuse roughly 40% of the observability dashboards from earlier work. Want a similar assessment? Book a 30-minute call and we’ll scope it against your stack.

Figure 5. A four-week hardening sprint, before and after, on a real SaaS platform.

Five reliability pitfalls we see every week

1. SLOs without enforcement. The SLO lives on a wiki; nothing in the pipeline checks it; nobody stops shipping when the budget burns. Fix: burn-rate alerts that page on-call plus a release-blocker rule wired into CI.

2. Retries without backoff. Three retries with no delay turn one bad request into four. Add exponential backoff and jitter, capped by a retry budget.

3. Health checks that lie. The /healthz endpoint returns 200 while the database is unreachable, so the load balancer keeps sending traffic. Fix: deep health checks that probe critical dependencies.

4. Backups never tested. Daily backups, never restored. The first restore drill happens during the real incident. Fix: monthly automated restore to a sandbox, validated against a checksum.

5. Single-person knowledge silos. “Only Anya knows how the payments service works.” Anya is on holiday. Fix: paired on-call, a written runbook per service, and knowledge rotation as an explicit goal.

KPIs: what to measure once you harden

Quality KPIs. SLO compliance per service (target 100% over a 30-day window), p99 latency, error rate, and crash-free user rate on mobile clients (target above 99.5%).

Business KPIs. Customer-impacting incidents per month (target under 1), churn attributable to reliability complaints, and SLA credits paid out (target $0).

Reliability KPIs. Change-failure rate (target under 15%, elite is ~5%), MTTR (under 60 minutes), MTTD (under 5 minutes), and deployment frequency (multiple per week).

How much reliability to invest in: five questions

Q1. What is one hour of downtime worth in revenue? Anchor every reliability investment here. Under $5k, ship the basics: SLOs, observability, backups. Over $50k, ship the full pattern set including chaos engineering and multi-region.

Q2. Are you in a regulated industry? HIPAA, PCI, SOC 2 and GDPR all carry reliability minimums. Below 99.9% availability invites audit findings. Bake it into the SLO from day one.

Q3. Do you ship daily, weekly, or monthly? The faster you ship, the more you need progressive rollout and flags. A monthly cadence survives on canary deploys; a daily cadence needs full flag infrastructure.

Q4. How many third-party APIs do you depend on? Each is a potential cascade source. Wrap every one in a circuit breaker, cap its retry budget, and cache responses where consistency allows.

Q5. Who carries the pager today? If the answer is “the founder” or “everyone,” you need rotations before tools. People-shaped problems don’t respond to more dashboards.

When NOT to over-invest in reliability

Three cases where the cost of resilience exceeds the value. First, pre-product-market-fit startups under 100 active users — an SLO is irrelevant when you don’t yet know what you’re building. Ship CI, basic logging and Sentry; defer the rest.

Second, internal tools used by under 50 employees with a tolerated downtime window. Multi-region for an internal HR portal is theatre. Reach for the budget when the tool becomes load-bearing for the business.

Third, products in genuine MVP mode where every reliability hour competes with a feature the market is actively asking for. Cap the reliability backlog at 10–20% of capacity until you hit product-market fit, then re-evaluate at each funding milestone.

Adjacent topics worth a deep read

Reliability never lives alone. Three adjacent surfaces deserve attention.

QA testing is the upstream prevention layer. Our guide to QA testing in software development covers the test pyramid, contract tests and shift-left strategies.

Cost planning matters because resilience isn’t free. Our mobile app development cost guide models where a reliability budget fits inside a product’s P&L.

Build vs buy shapes the reliability question entirely. Our low-code/no-code vs. hiring developers piece is the right starting point if you’re still deciding on the team model.

FAQ

What uptime SLO should a SaaS startup commit to?

99.9% (43.2 minutes of downtime a month) is the standard early-stage commitment. Move to 99.95% (21.6 min/month) once you have enterprise customers, and 99.99% (4.3 min/month) only for finance, healthcare, or an explicit contract. Promising 99.99% without the chaos-engineering muscle to back it is a bigger reputational risk than a lower stated number.

What is an error budget, exactly?

An error budget is the amount of unreliability your SLO permits over a window. A 99.95% monthly SLO allows 0.05% of 30 days, or 21.6 minutes, of downtime. While the budget is healthy you ship features; when it’s burning you stop launching and harden. It turns “how reliable should we be?” into a number the whole team can act on.

Do microservices make software more reliable?

Microservices give you network-level bulkheading but add failure modes: network calls, distributed tracing, eventual consistency. For most products under 50 engineers, a well-structured modular monolith with internal bulkheads (separate thread pools, circuit-breaker libraries) is more reliable than premature microservices.

How much should a team budget for reliability work?

Google SRE’s heuristic is 50% of an SRE’s time on reliability projects. For product teams without dedicated SREs, 15–20% of capacity is a defensible long-term commitment. After a major incident, jump to 40–50% for a sprint or two and harden the specific failure mode.

Is chaos engineering needed on day one?

No. Chaos engineering pays off after you have observability, alerting, runbooks and a healthy on-call culture. Without those, chaos exercises produce panic, not learning. It correlates with elite reliability only when layered on top of mature SRE practice.

Which observability stack fits best in 2026?

Emit signals in OpenTelemetry from day one. The collector then routes to whatever fits your stage: Sentry for early-stage error tracking, Honeycomb or Datadog at scale, Grafana Cloud for self-hosted control. The key is the OTel instrumentation, not the vendor — that choice stays reversible.

What causes most major outages in 2024–2026?

Deployment-induced failures, by a wide margin. Public postmortems and the 2024 DORA report both put change-failure rate as the top reliability lever. CrowdStrike is the highest-profile example: a config push with no staged rollout took down 8.5M hosts.

How long does a reliability hardening engagement take?

A focused two-week sprint delivers SLOs, observability, basic circuit breakers and a written runbook. A full hardening (chaos exercises, multi-region, DR drills) usually runs six to eight weeks. Agent-assisted engineering compresses that timeline; a 30-minute call is enough to scope it against your codebase.

What to read next

Ready to ship software your users can trust?

Crash-proof software is an error budget, not a binary. Pick an SLO, instrument with OpenTelemetry, layer in circuit breakers and bulkheads, ship behind feature flags with progressive rollout, and run blameless postmortems when things still break. The pattern set is well understood; the work is operational, not heroic.

If you want a second pair of eyes on your stack, or a team that has shipped this pattern across 250+ projects over two decades, we’re a 30-minute call away.