Senior Reliability & Fault Tolerance Interview Questions (50 MCQs)

Question 1

A 30-second database slowdown has cleared, yet an hour later the service is still failing under the same normal traffic. What is this pattern called?

Accepted Answer

Metastable failure; a feedback loop such as retries keeps the system in the bad state

Answer

Gray failure; health checks pass while users see errors

Answer

Split-brain; two primaries accept writes at the same time

Answer

Fail-slow; a slow node lingers instead of a dead one

Question 2

What is usually the most effective way out of a metastable failure caused by a retry storm?

Accepted Answer

Sharply cutting incoming load (shedding or disabling retries) until it drops below capacity

Answer

Restarting every instance at once with the same settings

Answer

Raising timeouts so slow requests can also complete

Answer

Adding more retries at the edge to mask errors

Question 3

An overloaded service is shedding requests. To prevent callers from amplifying retries, what signal should it send?

Accepted Answer

An explicit 'overloaded, do not retry' error that upstream layers pass along without retrying

Answer

A `200 OK` with an empty body so callers don't notice

Answer

Dropping the TCP connection silently so callers time out

Answer

A `400 Bad Request` so the caller fixes its request

Question 4

How do 'hedged requests' cut tail latency, and what is their main risk?

Accepted Answer

A second copy goes to another replica if no reply by ~p95; the risk is extra load during a global slowdown

Answer

Every request goes to all replicas upfront; the risk is data inconsistency

Answer

Requests are queued until p50; the risk is a higher average

Answer

Responses are cached forever; the risk is stale data

Question 5

The API gateway times out at 2 seconds, but the service behind it uses a 5-second DB timeout with 2 retries. What is the core problem?

Accepted Answer

The service may work up to 15 seconds on a request the caller abandoned at 2 seconds

Answer

The gateway sends requests twice because of its shorter timeout

Answer

The database replies faster because of the longer timeout

Answer

No problem, because downstream timeouts should be longer

Question 6

Why does outlier detection (e.g., in Envoy) have a parameter like `max_ejection_percent`?

Accepted Answer

So a shared problem (like a broken dependency) can't eject every host and cause a total outage

Answer

So new hosts receive traffic faster than old hosts

Answer

So the percentage of retried requests is capped

Answer

So the cost of active health checks goes down

Question 7

When all targets fail health checks, how do load balancers (like AWS ALB) typically behave, and why?

Accepted Answer

They fail open and route to all targets, since the health check itself is likely broken

Answer

They reject all requests with `503` until targets recover

Answer

They automatically shift traffic to another region

Answer

They terminate and relaunch all the targets

Question 8

In shuffle sharding, each customer gets a random combination of 2 out of 8 workers. Why is this better than simple sharding against a poison tenant?

Accepted Answer

There are many combinations (28), so few customers share exactly the same two workers

Answer

Because every customer can use all 8 workers and load spreads out

Answer

Because the poison tenant is automatically removed

Answer

Because shuffle sharding removes the need for rate limiting

Question 9

What is the main goal of a cell-based architecture?

Accepted Answer

Limiting blast radius by splitting the system into independent copies each serving a subset of customers

Answer

Reducing service count by merging everything into a monolith

Answer

Running every service on one large shared cluster

Answer

Removing staged deploys by deploying all cells at once

Question 10

What does 'static stability' mean in multi-AZ design?

Accepted Answer

The system keeps working through an AZ loss without needing changes or new provisioning

Answer

The system's configuration never changes after initial setup

Answer

All instances have fixed (static) IP addresses

Answer

Traffic is split statically without a load balancer

Question 11

In a multi-region active-active system where every region accepts writes, which approach structurally reduces write conflicts for user accounts?

Accepted Answer

Assigning each user a 'home region' where their writes happen

Answer

Accepting writes everywhere and keeping the latest by wall clock

Answer

Disabling replication so no conflicts arise

Answer

Adding more regions so conflicts become less likely

Question 12

A three-node cluster requires a majority quorum (2 of 3), and each node is independently `99%` available. What is the system's availability, roughly?

Accepted Answer

About `99.97%`

Answer

About `99.9999%`

Answer

About `97.03%`

Answer

Exactly `99%`

Question 13

The `1 - (1-p)^n` calculation for parallel replicas is often optimistic in practice. What is the main reason?

Accepted Answer

Failures are correlated: a shared deploy, config, software bug, or dependency takes them all down together

Answer

The formula is only valid for an even number of replicas

Answer

Replicas always have higher availability than stated

Answer

The load balancer always doubles the availability

Question 14

A team has a standby database with automatic failover but has never tested failover in production. What is the biggest risk?

Accepted Answer

The failover path itself (config, DNS, permissions, capacity) fails when actually needed

Answer

The standby's hardware wears out from lack of use

Answer

Automatic failover raises licensing costs

Answer

The primary becomes slower because a standby exists

Question 15

A worker acquires a distributed lock, then hits a long GC pause and its lease expires. Another worker takes the lock. What prevents the first worker's stale write after it wakes?

Accepted Answer

A monotonically increasing fencing token that storage uses to reject older tokens

Answer

Extending the lease to hours so it never expires

Answer

Having the worker check the lock right before writing

Answer

Comparing write times using wall-clock timestamps

Question 16

Why does a leader in a lease-based system usually treat its lease as expired slightly before its nominal expiry?

Accepted Answer

To cover clock drift and network delay so two leaders never overlap

Answer

So it can receive more leases from the lock server

Answer

Because system clocks are always ahead of real time

Answer

So lease renewal costs are reduced

Question 17

A service runs active-active across 3 regions and must survive a full region loss without load shedding. What is the maximum normal utilization per region?

Accepted Answer

About 67%

Answer

About 33%

Answer

About 50%

Answer

About 90%

Question 18

After an outage is fixed, a million mobile clients whose WebSockets dropped reconnect at once and the server falls over again. What is the best combination of fixes?

Accepted Answer

Client backoff with jitter, plus server-side admission control and gradual traffic ramp-up

Answer

Increasing client connection timeouts to several minutes

Answer

Switching WebSockets to one-second polling

Answer

Repeatedly restarting servers until connections spread out

Question 19

After an outage, an interactive service has a large queued backlog. Why do some systems switch to LIFO in this state?

Accepted Answer

New requests still have waiting users, while old ones have likely timed out

Answer

Because LIFO reduces the queue's memory use to zero

Answer

Because FIFO is algorithmically slower on large queues

Answer

So the processing order of requests is preserved

Question 20

On what basis does the CoDel algorithm decide to drop queued requests?

Accepted Answer

If the minimum queueing delay stays above a target (e.g., 5ms) for an interval

Answer

If the queue length exceeds a fixed number

Answer

If the request came from a repeated IP address

Answer

If the server's CPU is above 80%

Question 21

What advantage does an adaptive concurrency limit (like Netflix's concurrency-limits) have over a static rate limit?

Accepted Answer

It tunes the limit automatically from latency changes and adapts as capacity shifts

Answer

It accepts every request without any limit

Answer

It removes the need for client timeouts

Answer

It works only for batch, non-interactive traffic

Question 22

A service handles 1,000 rps at 100ms latency with a 200-worker pool. What happens if a dependency pushes latency to 500ms?

Accepted Answer

Required concurrency hits 500, the pool saturates, and queueing and timeouts begin

Answer

Concurrency stays the same because request rate is unchanged

Answer

Throughput automatically increases fivefold

Answer

Required concurrency drops to 20

Question 23

Why should load shedding happen as early as possible in the request path?

Accepted Answer

If rejecting is costly (parsing, auth, queries), the server lacks capacity even to reject

Answer

Because early requests are always less important

Answer

Because late rejection requires a different status code

Answer

So downstream services produce more error logs

Question 24

In client-side adaptive throttling (Google SRE book), on what basis does a client locally reject requests before sending?

Accepted Answer

The ratio of requests sent to requests accepted by the backend over the last two minutes

Answer

The number of CPU cores on the client machine

Answer

The time of day and historical traffic patterns

Answer

The size of each request's body

Question 25

The Redis cluster backing a distributed rate limiter goes down. Which policy is most reasonable?

Accepted Answer

Fail open with a local-limit fallback for general APIs; be more cautious for sensitive ones like login

Answer

Always fail closed and reject everything until Redis returns

Answer

Queue all requests in memory until Redis returns

Answer

Remove rate limiting permanently since it's a SPOF

Question 26

A consumer of an at-least-once queue must update an account balance. Which design guarantees an 'exactly-once effect'?

Accepted Answer

Recording the message ID in a dedup table and updating the balance in one DB transaction

Answer

Acking the message before processing so it's never redelivered

Answer

Checking the message ID in Redis, then updating the balance in the DB

Answer

Raising the queue's visibility timeout so redelivery never happens

Question 27

A client resends a previous idempotency key with a different body (another amount). What should the server do?

Accepted Answer

Compare against the stored request hash and reject it with an error (e.g., `422`)

Answer

Return the previous response and ignore the new body

Answer

Execute the new request and overwrite the old result

Answer

Sum both amounts and perform a single transaction

Question 28

A canary ran on 1% of a low-traffic endpoint for 5 minutes with zero errors. Why isn't this result trustworthy?

Accepted Answer

The sample size is too small to statistically detect an error-rate increase

Answer

Because a canary must receive at least 50% of traffic

Answer

Because zero errors always means metrics are broken

Answer

Because canaries are only valid in staging

Question 29

A new version writes cache and queue data in a new serialization format the old version can't read. What is the safest rollout approach?

Accepted Answer

First ship a version that reads both formats, then enable writing the new format in a later release

Answer

Upgrade all servers simultaneously so no old version remains

Answer

Flush the entire cache and queues after deploying

Answer

Disable rollback since the new format is better

Question 30

Many major outages came from a global config change. What is the main lesson?

Accepted Answer

Config changes, like code, need validation, staged rollout, and fast rollback

Answer

Config should only be changed manually on each server

Answer

Config should never change after launch

Answer

Config should be hardcoded so it can't change

Question 31

The central feature-flag service goes down. What design keeps this failure from spreading to all services?

Accepted Answer

The SDK caches the last snapshot locally and has safe defaults in code

Answer

Every user request synchronously fetches flags from the central service

Answer

On error, turn every flag on by default

Answer

Services reject requests until the flag service returns

Question 32

A team wants a `99.99%` SLO but has a serial hard dependency with `99.9%` availability. Which statement is correct?

Accepted Answer

Without reducing the dependency (cache, fallback, redundancy), the SLO is unattainable

Answer

Adding retries to the dependency always makes the SLO attainable

Answer

A service's SLO is unrelated to its dependencies' availability

Answer

Adding more instances of your own service solves it

Question 33

Why do burn-rate alerts combine a long window (e.g., 1 hour) with a short window (e.g., 5 minutes)?

Accepted Answer

The long window confirms significance, and the short one lets the alert resolve quickly after recovery

Answer

The short window is for weekdays and the long one for holidays

Answer

Because storing metrics over a long window is cheaper

Answer

So every alert is sent to on-call twice

Question 34

Each of 20 hosts reports its own p99 latency. Why doesn't averaging those 20 numbers give the global p99?

Accepted Answer

Percentiles can't be summed or averaged; you need mergeable histograms or sketches

Answer

Because you should take the median of the p99s instead

Answer

Because each host's p99 always equals the global p99

Answer

Because the p99s should be summed, not averaged

Question 35

A developer added a `user_id` label to a Prometheus metric. What is the main problem?

Accepted Answer

Cardinality explosion; each user creates a separate time series, blowing up memory and cost

Answer

Prometheus doesn't accept labels named `user_id`

Answer

The metrics can no longer be shown in Grafana

Answer

Metric accuracy drops due to value rounding

Question 36

Why is tail-based sampling often better than 1% head-based sampling for tracing in high-traffic systems?

Accepted Answer

The decision is made after the trace completes, so error and slow traces are always kept

Answer

Because tail-based has no memory cost in the collector

Answer

Because tail-based doesn't need trace context propagation

Answer

Because head-based only works with gRPC

Question 37

Why do mature teams eventually run chaos experiments in production, and what makes it safe?

Accepted Answer

Only production has real traffic and config behavior; small blast radius and automatic abort make it safe

Answer

Because staging costs more than production; not telling the team makes it safe

Answer

Because production self-heals; running at peak hours makes it safe

Answer

Because it's legally required; disabling monitoring makes it safe

Question 38

An attacker has gained cloud admin credentials for the production account. Which backup design has the best chance of surviving?

Accepted Answer

Immutable backups (e.g., object lock) in a separate account with independent access

Answer

Daily snapshots in the same account and region

Answer

Synchronous replicas across AZs in the same account

Answer

Encrypted backups with a key stored in the same account

Question 39

All health checks are green, yet 5% of users time out due to a faulty NIC on one server. What is this called and what helps?

Accepted Answer

Gray failure; measuring from the client's view (like real per-host error rates), not just internal probes

Answer

Split-brain; adding quorum to the load balancer

Answer

Cascading failure; increasing the number of retries

Answer

Metastable failure; restarting the whole cluster

Question 40

Why is a 'slow' node said to be potentially more dangerous than a 'dead' one?

Accepted Answer

A dead node is detected and removed fast; a slow one ties up callers' resources and often stays in rotation

Answer

Because a slow node corrupts data but a dead one doesn't

Answer

Because a slow node uses more electricity

Answer

Because a dead node restarts automatically

Question 41

The order service gets a timeout calling the payment provider's `POST /charge`. What is the real payment state and the right behavior?

Accepted Answer

Unknown; retry with the same idempotency key or query the status and reconcile

Answer

It definitely failed; charge again with a new key

Answer

It definitely succeeded; confirm the order

Answer

It definitely failed; cancel the order and tell the user

Question 42

The checkout page depends on 8 services and fails if any one is down. What is the most effective architectural change for availability?

Accepted Answer

Turning non-essential dependencies (like recommendations, loyalty) into soft ones with timeouts and fallbacks

Answer

Adding 8 retries for each of the 8 dependencies

Answer

Merging all 8 services into one shared database

Answer

Raising every call's timeout to 30 seconds

Question 43

Why do some teams (e.g., in the AWS Builders' Library) warn against fallbacks that only activate during failures?

Accepted Answer

The rarely exercised fallback path creates bimodal behavior that may fail or add load in a crisis

Answer

Because fallbacks are always faster than the main path and cause unfairness

Answer

Because fallbacks require a separate license

Answer

Because fallbacks only work in compiled languages

Question 44

What reliability benefit does the 'constant work' pattern (e.g., pushing the full config every few seconds instead of only deltas) offer?

Accepted Answer

System load is the same in normal and crisis times, so a storm of changes can't overload it

Answer

Bandwidth use in normal operation drops to the minimum possible

Answer

The need for any monitoring disappears

Answer

Config changes are automatically rolled back

Question 45

A cache cluster with a 95% hit rate restarts, and the database, sized for only 2x normal miss traffic, collapses. What is the root design flaw?

Accepted Answer

The cache is effectively a capacity dependency; losing it raises DB load up to 20x, unplanned for

Answer

A 95% hit rate is too low for a cache

Answer

The database should run without indexes to be faster

Answer

The cache should have an infinite TTL

Question 46

During a spike the service autoscales from 20 to 60 instances; each has a 20-connection DB pool and the DB's `max_connections` is 500. What happens?

Accepted Answer

1,200 connections are needed; connections are refused and scaling itself worsens the outage

Answer

The database automatically raises `max_connections`

Answer

Each instance uses only an equal share (8 connections)

Answer

Nothing goes wrong because pools reuse connections

Question 47

Order placement synchronously calls email, analytics, and inventory services. From a reliability view, what does turning these into events on a durable queue gain?

Accepted Answer

Their failure or slowness no longer fails order placement, and the queue absorbs bursts

Answer

It guarantees every service processes the message exactly once, instantly

Answer

It removes the need for idempotency in consumers

Answer

It reduces consumer processing latency to zero

Question 48

Why is a `100%` SLO usually the wrong target?

Accepted Answer

Users can't perceive it (their network and devices are less reliable) and every change becomes forbidden

Answer

Because monitoring tools can't display the number `100`

Answer

Because legal SLAs forbid committing above `99%`

Answer

Because it's only achievable with a single server

Question 49

A service's liveness probe checks its database connection. The database slows down for 2 minutes. What happens?

Accepted Answer

Kubernetes restarts every pod, and simultaneous cold starts and reconnects worsen recovery

Answer

Only pods with active requests are removed from the Service

Answer

Kubernetes automatically restarts the database

Answer

Nothing, because liveness only runs at startup

Question 50

In designing a managed load balancer, why shouldn't the data plane depend on the control plane being available to serve traffic?

Accepted Answer

The control plane is more complex and less available; the data plane should keep running on the last known good config

Answer

Because the control plane is always faster than the data plane and must stay separate

Answer

Because the data plane receives no traffic without the control plane

Answer

Because the control plane is only available during business hours