Reliability design

What does moving from 99.9 to 99.99 change?

The allowed downtime in the window drops by ten. The design, the dependencies, and the deploy practice all have to be able to live inside the smaller budget.

Five services in series are each 99.95 percent available. What is the path?

About 99.75 percent if they fail independently, because any one failure fails the request. The path is weaker than its weakest hop's headline.

A request waits for 100 shards. What dominates?

The slowest shard. The p99 of the fan-out is far worse than the p99 of one shard. Waiting for a quorum, or hedging, changes that.

How do you estimate ingest volume?

Events per second times bytes per event times seconds. The raw number decides the bus, the disk, and the bill.

The API slows, then every request times out. What keeps goodput?

A limit that rejects overflow quickly, so accepted requests still finish. A queue with no limit turns load into timeouts for everyone.

What is goodput?

Work that actually succeeds per second. Throughput that is mostly errors or timeouts is not goodput.

Why split tenants into cells?

A failure, a bad deploy, or a noisy tenant stays inside one cell. A shared everything fails everyone together.

What limits the size of a cell?

It has to be big enough to be efficient and small enough that losing it is inside the error budget.

A 1 percent canary on a rare error. What is the problem?

In a short window you may not see enough errors to tell the canary from the baseline. The step is too small for the signal you need.

Backups run nightly and have never been restored. What is missing?

A timed restore test. The objective you can meet is the one you have measured, not the one the backup vendor printed.

Auto-restart hides 200 crashes a day. Why does that matter?

Users see few errors, so nobody fixes the crash. The day the restart is slow, those crashes become the outage.

What is a shared-fate failure?

Parts you thought were independent die from one cause: one NAT, one certificate, one library. Redundancy that shares fate is not redundancy.

What is N+1 capacity?

Enough spare that one failure still leaves a full serving set. N with no spare means the first failure is overload.

How do you bound a queue?

A maximum depth and a timeout, with a defined drop or reject. An unbounded queue is a slow way to run out of memory.

Why pass a deadline down the stack?

So a hop does not start work that cannot finish in time, and so retries do not outlive the user.

What does a bulkhead isolate?

The concurrency one dependency may use. When that dependency stalls, the rest of the process still has workers.

How do you split a latency budget?

Across the hops, with a remainder. If the parts sum past the user's limit, the design cannot meet the SLO on the slow path.

How do you budget a dependency?

Your request's success needs its success. Either the dependency's SLO is tight enough, or you degrade when it fails.

What does degrading on purpose look like?

A cached or partial answer, flagged as such, for a feature that is not the user's main job.

What belongs off the critical path?

Anything the response does not need: analytics, a recommendation, a secondary write. Those move to async or fail open.

When do you fail open?

When serving without the check is safer than failing the request, and you have accepted the risk. A rate limiter may fail open and risk a flood. An auth check fails closed.

When do you fail closed?

When serving without the check is the incident: authentication, a safety limit, a payment confirmation.

What must be true before you retry?

The operation is idempotent, or you can tell it did not happen. A retry of a payment that might have committed needs a key.

Why jitter a scheduled job?

So a fleet does not kick off the same minute and stampede a dependency. Spread the start.

What is a lease?

Ownership that expires. If the holder dies, someone else can take the work after the lease, instead of waiting forever.

How do you run one job in a fleet?

A lease or a leader, with a timeout. A flag file on one disk becomes two jobs when a second host starts.

How do you design for a poison message?

A retry limit, then a side queue. Without it, one bad message blocks every message behind it on that partition.

How should a full consumer push back?

By slowing the producer or rejecting, with a visible lag. Dropping silently loses data the SLO thought was durable.

Where is a write durable?

When it has reached the copies you promised, and only then do you acknowledge. Acknowledging at the app and writing after is how a crash loses the write.

What does a quorum buy?

The write survives a minority of the copies being down, and a reader who also uses a quorum sees that write.

What do you decide for two regions?

Which region takes writes, how failover happens, and what a split would do. Two writers with no rule will conflict.

How does the RPO show up in the design?

As the replication lag you can tolerate. Synchronous replication can hit a zero RPO and will stall writes when the far side is slow.

How does the RTO show up in the design?

As how fast traffic can move, including DNS, warm capacity, and a restore if you need one. A cold start misses a short RTO.

What is a game day?

A planned failure in production or a faithful stage: kill a zone, restore a backup, and time it.

How does the budget constrain design?

A change that risks more than the remaining budget waits, or the SLO is explicitly loosened. The budget is the currency.

What is in an SLO document?

The SLI query, the target, the window, and who is paged. A target with no query is a slogan.

Why does the window matter?

A 30-day 99.9 percent budget is about 43 minutes. The same nines on a day are about a minute. The design lives in one of those, not in the word.

What does the design page on?

Budget burn of the user SLI. Host metrics are dashboards for the person who is already awake.

Which saturation metric do you design in?

The one that will hit first: connections, queue depth, or disk. You cannot alert on a limit you do not measure.

How do you limit deploy risk in the design?

A small first step, a gate on the SLI, and a rollback that does not need a data rewrite.

What is a kill switch?

A control, outside the bad deploy, that turns the feature off. It has to be tested before the day you need it.

Why design for certificate expiry?

A shared cert is a shared-fate outage on a clock. Renewal is automated, and the expiry is alerted with time to spare.

What breaks when clocks drift?

Leases, token expiry, and log order. The design assumes a bound on drift, and monitoring has to notice when the bound is broken.

A design has one coordination row everyone updates. What happens?

That row is the ceiling, no matter how many app servers you add. Split the coordination or cache the read.

What is a retry budget?

A cap on extra calls a service may generate. Past it, the service fails the request instead of multiplying load downward.

What is admission control?

Deciding at the door whether to accept work, based on what is already in flight. It is how you stay at a concurrency you measured.

What is steady-state load versus failover load?

Steady state is today's traffic. Failover load is that traffic on the remaining capacity. You size for the second.

What do you draw besides boxes?

For each arrow: whether the call is required, its timeout, and what the request returns when that call fails. Those three notes are what the box diagram leaves out.

How do you know the design meets the SLO?

You cause the failure you claim to survive, measure the SLI through it, and record how long recovery took. That run is the evidence the design meets the number.