How to Build Reliable, Crash-Proof Software in 2026: SLOs, Patterns, DORA Metrics

Why Fora Soft wrote this playbook

Fora Soft has shipped real-time video, e-learning, telemedicine and streaming products since 2005. Every one of those products lives or dies by reliability: a frozen call costs a clinic visit, a dropped live-shopping stream costs a sales hour, and a flaky LMS lecture costs a school day. We have run the postmortems. The patterns are remarkably consistent across product categories — and so are the fixes.

This playbook collapses what we have learned shipping BrainCert (a WebRTC-first LMS at $10M ARR), Sprii (a live-shopping platform that has hosted 72k+ live events generating €365M+ in seller revenue), and ProVideoMeeting (an enterprise video stack at 1k+ concurrent participants per room) into a single decision framework. It is written for founders, CTOs and engineering leads making the call between “ship fast and patch later” and “invest in resilience now.”

The numbers below are pulled from public sources (Google SRE workbook, AWS Well-Architected, ITIC and Gartner downtime reports, DORA 2024 metrics, CrowdStrike and Datadog postmortems). Where we cite our own client work we use ranges, not specifics, to respect NDAs.

What “reliable” actually means in 2026

Crash-proof software does not exist. What does exist is software that fails inside a budget, recovers faster than the user notices, and degrades gracefully instead of breaking outright. That is what modern Site Reliability Engineering (SRE) means. The Google SRE workbook formalized the vocabulary a decade ago and the rest of the industry has converged on it.

Three terms anchor every reliability conversation. A Service Level Indicator (SLI) is the metric you measure — availability percentage, p99 latency, error rate. A Service Level Objective (SLO) is the target for that metric over a window — “99.95% of requests succeed in under 250 ms over 30 days.” A Service Level Agreement (SLA) is what you contractually owe customers if you miss the SLO — usually a credit. The error budget is what is left over: an SLO of 99.95% gives you 21.6 minutes of allowable downtime per month.

When the error budget is healthy, you ship features fast. When you are burning through it, you stop launching and harden. That single decision rule replaces vague reliability rhetoric with a measurable trade-off engineering teams can agree on without arguments.

What downtime actually costs (industry data)

The reliability conversation always comes back to money. Industry surveys converge on a few reference numbers worth memorising before the next budget meeting.

The reliability investment that prevents a single hour of mid-market downtime usually pays for itself the same week. That is the conversation the CFO needs framed in dollars, not engineering jargon.

The eight failure categories that cause most outages

When we run a reliability audit on a client codebase, the same eight failure categories show up over and over. Pattern-matching against them at the start of an audit saves a week of investigation.

1. Memory leaks. Unclosed database connections, leaked event listeners, growing in-memory caches. Symptom: gradual latency increase that resets when the service restarts. Fix: heap profiling and lifecycle audits.

2. Cascading failures and retry storms. One downstream API slows to 8 seconds; everyone retries; the upstream pool exhausts; the next hop fails. Fix: circuit breakers, bulkheads, retry budgets.

3. Race conditions. Two requests update the same row, the wrong one wins, customer data corrupts silently. Fix: optimistic locking, transactional boundaries, idempotency keys.

4. Unhandled exceptions. A null pointer in a rarely-used branch crashes the whole worker. Fix: structured error handling, panic budgets, sentinel error monitoring.

5. Dependency and supply-chain failures. A third-party SDK ships a bad release at 3am; your CI pulls it in. The CrowdStrike pattern. Fix: pinned versions, staged rollouts, vendor incident channels.

6. Deployment-induced failures. The DORA 2024 report shows change-failure rate is the single biggest reliability lever. Fix: progressive rollout, automated rollback gates, contract tests.

7. Infrastructure outages. AWS us-east-1 has had at least one major incident every year since 2017. Fix: multi-AZ at minimum, multi-region for tier-1 services, runbooks for downstream cloud failures.

8. Configuration drift. A flag set on one server, missed on another; staging works, production explodes. Fix: GitOps, infrastructure-as-code, configuration linting in CI.

Six patterns that prevent the majority of crashes

There is no single “reliability framework” you can install. Resilience comes from layering a small set of well-understood patterns. Six of them carry most of the weight.

Circuit breakers

A circuit breaker tracks failure rates against a downstream service. When failures cross a threshold, the breaker trips open and your code fails fast for a cooldown period instead of piling up requests. Production-grade libraries: resilience4j for the JVM, Polly for .NET, gobreaker for Go.

Bulkheads

Bulkheads isolate failure domains. One thread pool per downstream, one connection pool per database, one Kubernetes namespace per blast radius. When the payments microservice melts, the rest of the app stays up. Datadog’s March 2023 outage went total because the bulkhead between control-plane and data-plane was missing.

Retry with exponential backoff and jitter

Retries without backoff turn one failure into a stampede. The standard recipe: 100 ms, then 200, then 400, then 800, with random jitter (±25%) to spread the herd. Cap at 5 attempts and a 30-second total. AWS’s reliability pillar has the canonical reference implementation.

Idempotency keys

Mutating endpoints accept a client-supplied Idempotency-Key header. The server stores the result for 24 hours; subsequent retries return the same response without side effects. Stripe popularised the pattern; every payments and reservation API ships it now.

Feature flags and progressive rollout

Every risky change ships behind a flag rolled out 1% → 10% → 50% → 100% with a metric gate at each step. LaunchDarkly, GrowthBook, and Unleash are the popular options. The CrowdStrike incident would have been a 1% incident with progressive rollout instead of 8.5M global hosts.

OpenTelemetry-native observability

Logs, metrics and traces emitted in the OpenTelemetry standard, then routed to whichever backend you prefer (Datadog, New Relic, Honeycomb, Grafana Cloud, Sentry). Vendor lock-in is dead. The point is to alert on SLO burn rate, not raw thresholds — that one shift is the single biggest improvement most teams can make to alert fatigue.

Five famous outages and the one lesson each teaches

Public postmortems are the closest thing the industry has to free education. Five recent ones every founder should be able to summarise.

Mobile reliability: the metrics that move retention

For mobile-first products — calling apps, e-learning apps, streaming, telemedicine — the server-side SLO conversation is only half the story. The other half lives on the device, and Firebase Crashlytics or Sentry Mobile is where it gets measured.

Crash-free user rate. The headline metric. Stock baseline is 99% on Android, 99.5% on iOS. Premium products run 99.9%+. A 0.5-point drop usually correlates with a one-star wave on the store within two weeks.

ANR rate (Android only). Application Not Responding under 0.5% of sessions. Above that and Google Play down-ranks the app silently. The fix is almost always background work that drifted onto the main thread — Compose recompositions doing IO, image decoding on the UI thread, deserialization in adapters.

Foreground-service watchdog. For calling and streaming products, an active FOREGROUND_SERVICE_TYPE_PHONE_CALL or FOREGROUND_SERVICE_TYPE_MEDIA_PLAYBACK with the new Android 14+ 5-second startForeground deadline. Miss it and the OS throws ForegroundServiceDidNotStartInTimeException — the call drops, the user blames the app. We covered the patterns in our foreground services and deep links on Android 14 guide.

Network resilience. Offline-first sync, retry budgets that defer to background workers, exponential backoff on every API call, and Sentry breadcrumbs that capture the network class (LTE / 5G / WiFi / no-net) at crash time.

OEM matrix testing. Stock Pixel does not predict Xiaomi behaviour. We bake the top five OEMs into every Android client’s CI device matrix — the same posture we recommend in our custom Android call notifications playbook.

Real-time and AI workloads: a different reliability bar

Standard SLOs were written for request/response services. Real-time video and LLM-powered features need additional metrics and additional defences.

Real-time video. The metrics that matter are mean opinion score (MOS), packet loss rate (target under 2%), and round-trip-time p95 (target under 200 ms). Use a SFU/MCU architecture sized for two times peak load. We covered the trade-offs in our custom video conferencing architecture guide.

LLM-powered features. Hallucination handling, token-cost guardrails, multi-model fallback (Claude ↔ OpenAI ↔ Llama). Tools like Langfuse, LangSmith, and Helicone provide observability tailored to model behaviour. Treat the LLM as another flaky dependency: circuit-break it, retry it, cap its blast radius.

Streaming. Player buffer-empty rate under 1%, time-to-first-frame under 2 seconds, ABR ladder tested under throttled connections. The Sprii live-shopping playbook ships with a synthetic stream that runs every 60 seconds and pages on-call when MOS drops.

DORA 2024: how elite teams ship safely

Google’s annual DORA (DevOps Research and Assessment) report measures four metrics across thousands of engineering organisations and clusters them into elite, high, medium and low performers. The 2024 numbers below are the benchmark every reliability programme should track.

The counter-intuitive finding from a decade of DORA data: elite teams ship more often AND have lower failure rates. Reliability and velocity are correlated, not opposed. The shared root cause is rigorous progressive rollout and observability, both of which give engineers the courage to ship.

Organisation and process: the human side of reliability

Tools alone do not produce reliability. The same set of patterns deployed inside a punitive postmortem culture will fail; deployed inside a blameless on-call culture they thrive. Five process habits every reliable engineering organisation shares.

1. Blameless postmortems. The Etsy template is the public reference. Focus on the system, not the operator. Every postmortem produces three to five action items with owners and dates.

2. On-call rotations with handoff rituals. Twelve-hour shifts at most. Wednesday handoff with explicit ownership transfer. Pages count as work, not heroics.

3. Pre-prod environments that match prod. Same Kubernetes operator versions, same secrets manager, same network topology. Differences cause “works on staging” bugs that always surface at midnight.

4. Disaster recovery drills. Quarterly. Restore from backup, fail over to the secondary region, kill the primary database. Untested DR is theoretical DR.

5. Test pyramid. Many fast unit tests, fewer integration tests, fewer end-to-end tests. The opposite shape (lots of slow E2E tests, few units) is the second-most-common reliability anti-pattern we see in audits, behind missing observability.

Mini case: cutting MTTR from 6 hours to 22 minutes on a SaaS platform

A US-based B2B SaaS client, similar in profile to BrainCert, came to us with two recent multi-hour outages and an enterprise customer threatening to invoke their SLA credit clause. They had Sentry installed but no SLOs, no error budgets, and a single AWS region. Mean time to recovery sat at six hours.

Our four-week plan: define an SLO of 99.95% availability and 99% p95 latency under 350 ms; instrument every endpoint with OpenTelemetry; route into a Honeycomb backend with SLO burn-rate alerts; ship circuit breakers around the three external APIs that caused both incidents; and add progressive feature-flag rollout via GrowthBook. Final week: two chaos exercises (kill the primary database, kill the auth service) with a written runbook.

Outcome over the following 60 days: MTTR dropped from 6 hours to 22 minutes, change-failure rate from 31% to 12% (high to elite per DORA), and zero customer-impacting incidents. The enterprise customer renewed. Engineering effort was 2.5 engineer-months spread across four calendar weeks. Agentic engineering let us reuse 40% of the Honeycomb dashboards from earlier client work.

Five reliability pitfalls we see in audit every week

1. SLOs without enforcement. The team has an SLO on a wiki page; nothing in the deploy pipeline checks it; nobody stops shipping when the budget burns. Fix: SLO burn-rate alerts that page on-call AND a release-blocker rule wired into the CI gate.

2. Retries without backoff. Three retries with no delay multiplies one bad request into four. Add exponential backoff and jitter, cap with a retry budget.

3. Health checks that lie. The /healthz endpoint returns 200 even when the database is unreachable. The load balancer keeps routing traffic. Fix: deep health checks that probe critical dependencies and surface real degradation.

4. Backups never tested. Daily database backups, never restored. The first restore drill happens during the actual incident. Fix: monthly automated restore to a sandbox environment, validated against a checksum.

5. Single-person knowledge silos. “Only Anya knows how the payments service works.” Anya is on holiday. Fix: pair on-call rotations, written runbooks per service, and a deliberate knowledge-rotation goal in OKRs.

KPIs: what to measure once you start hardening

Quality KPIs. SLO compliance per service (target 100% over 30-day window), p99 latency, error rate, and crash-free user rate on mobile clients (target >99.5%).

Business KPIs. Customer-impacting incidents per month (target under 1), churn attributable to reliability complaints, and SLA credits paid out (target $0).

Reliability KPIs. Change-failure rate (target under 15%), MTTR (target under 60 minutes), MTTD (target under 5 minutes), deployment frequency (target multiple per week).

A decision framework — how much reliability to invest in, in five questions

Q1. What is one hour of downtime worth in revenue? Anchor every reliability investment in this number. If it is under $5k, ship the basics (SLOs, observability, backups). If it is over $50k, ship the full pattern set including chaos engineering and multi-region.

Q2. Are you in a regulated industry? HIPAA, PCI, SOC 2, GDPR all have reliability minimums. Below 99.9% availability invites audit findings. Bake it into the SLO from day one.

Q3. Do you ship daily, weekly, or monthly? The faster you ship, the more you need progressive rollout and feature flags. Monthly cadence can survive on canary deploys; daily cadence needs full LaunchDarkly-class flag infrastructure.

Q4. How many third-party APIs do you depend on? Each one is a potential cascade source. Wrap every one in a circuit breaker; cap retry budgets; cache responses where consistency allows.

Q5. Who carries the pager today? If the answer is “the founder” or “everyone,” you need rotations before tools. People-shaped problems do not respond to more dashboards.

When NOT to over-invest in reliability

Three cases where the cost of resilience exceeds the value. First, pre-product-market-fit startups with under 100 active users — an SLO is irrelevant when you do not yet know what you are building. Ship CI, basic logging, Sentry; defer everything else.

Second, internal tools used by under 50 employees with a tolerated downtime window. Multi-region for an internal-only HR portal is theatre. Reach for the budget when the tool becomes load-bearing for the business.

Third, products in genuine MVP mode where every reliability hour competes with a feature hour the market is asking for. Set a hard reliability backlog cap (10–20% of capacity) until you have product-market fit, then re-evaluate at every funding milestone.

Adjacent topics worth a deep read

Reliability never lives alone. Three adjacent surfaces deserve attention.

QA testing is the upstream prevention layer. Our guide to QA testing in software development covers the test pyramid, contract tests, and shift-left strategies.

Cost planning matters because resilience is not free. Our 2025 mobile app development cost guide models where a reliability budget fits inside a typical product’s P&L.

Build-vs-buy shapes the reliability question completely. Our low-code/no-code vs. hiring developers piece is the right starting point if you are still deciding on the team model.

FAQ

What to read next

Ready to ship software your users can trust?

Crash-proof software is a budget, not a binary. Pick an SLO, instrument with OpenTelemetry, layer in circuit breakers and bulkheads, ship behind feature flags with progressive rollout, and run blameless postmortems when things still break. The pattern set is well understood; the work is operational, not heroic.

If you want a second pair of eyes on your stack, or a team that has shipped this pattern across 50+ real-time products over two decades, we are a 30-minute call away.