Senior Replication & Sharding Interview Questions (50 MCQs)

Question 1

A leader holds a 10-second lease. A 15-second GC pause occurs and a new leader is elected, but the old leader resumes writing after the pause. What is the most robust defense?

Accepted Answer

A monotonically increasing fencing token (epoch) that storage uses to reject older tokens

Answer

Having the leader check its own lease validity right before each write

Answer

Shortening the lease to one second so pauses matter less

Answer

Tightly synchronizing all node clocks with NTP

Question 2

With N=3, W=2, R=2, a write is in flight and has reached only one replica. Two readers read one after another. What can happen?

Accepted Answer

The first reader sees the new value and the second the old one

Answer

W+R>N guarantees both readers see the new value

Answer

Both readers block until the write completes

Answer

Both readers receive a conflict error

Question 3

In a Dynamo-style store with version vectors, why must a client send the version context it read along with its next write?

Accepted Answer

So the server knows which siblings this write supersedes and which are concurrent

Answer

So the server can compute the write's exact timestamp

Answer

So the write is automatically routed to the key's leader

Answer

So tombstones are no longer needed for deletes

Question 4

For a 'like' counter in an active-active multi-region system, why is a G-Counter (one counter per replica) preferred over LWW on a single number?

Accepted Answer

Merging by per-slot max and summing never loses an increment

Answer

A G-Counter timestamps with a more precise clock, improving LWW

Answer

A G-Counter removes the need for cross-region replication

Answer

A G-Counter also allows decrements at no extra cost

Question 5

In an OR-Set (Observed-Remove Set), if one replica removes an item while another concurrently re-adds it, what is the result after merge?

Accepted Answer

The item remains, since the remove only cleared the tags it observed (add-wins)

Answer

The item is removed, because removes always beat adds

Answer

The result depends on which operation has the larger timestamp

Answer

The merge fails and the conflict is returned to the app

Question 6

The leader is in us-east with replicas in the EU. An EU user writes, and their next request may hit any region. Which design preserves read-your-writes with the least latency cost?

Accepted Answer

Return the commit position to the client; serve locally only if the replica reached it, else briefly wait or go to the leader

Answer

Reading all of the user's requests from the us-east leader forever

Answer

Sticking the user to one EU replica regardless of log position

Answer

Using the browser clock timestamp to choose a replica

Question 7

During resharding with dual-write and backfill, after cutover you find some records on the new shard have stale values. What is the most likely cause and fix?

Accepted Answer

The backfill overwrote newer dual-writes with an old snapshot; backfill writes must be conditional on version

Answer

Replication lag on old shards; raise the replication factor

Answer

The new shard uses a different hash function; switch it to MD5

Answer

The app cache was not invalidated; simply flush the cache

Question 8

In CDC-based resharding (like Vitess VReplication), why does cutover usually include a brief write pause?

Accepted Answer

So the stream fully drains and the target reaches the source's exact position before routing flips

Answer

So the new shard's indexes can be rebuilt from scratch

Answer

Because CDC only works when there are no writes

Answer

So the old shard's replicas can be deleted

Question 9

For messages in a chat app (1:1 chats and groups up to 5,000 members), which shard key fits the messages table best?

Accepted Answer

`conversation_id`, so reading a conversation's history is single-shard and ordered

Answer

`sender_id`, so all of a sender's messages sit together

Answer

A random `message_id`, for perfectly uniform distribution

Answer

`created_at`, so new messages are found quickly

Question 10

A user places an order and immediately opens 'my orders by status', served from an async global secondary index; the order is missing. What is the best fix?

Accepted Answer

Merge index results with the user's recent records read from the base table by partition key

Answer

Convert all indexes to local ones so reads become scatter-gather

Answer

Add a fixed 5-second delay before rendering the page

Answer

Increase the index table's replication factor

Question 11

A cluster runs 256 vnodes per node, but one partition key carrying 40% of write traffic saturates a node. Why won't adding more vnodes help?

Accepted Answer

Vnodes spread hash ranges, but one key always maps to one hash point and one replica set

Answer

Vnodes only spread read load, not write load

Answer

More than 256 vnodes is technically impossible

Answer

Vnodes only work if the hash function changes

Question 12

What problem does 'consistent hashing with bounded loads' (used at Google and Vimeo) solve?

Accepted Answer

With a per-node capacity cap, overflow goes to the next ring node so none gets overloaded

Answer

It reduces keys moved when adding a node to zero

Answer

It removes the need for vnodes by using two hash functions

Answer

It guarantees every key is replicated to every node

Question 13

Jump Consistent Hash gives near-perfect distribution with zero memory. What is its main limitation for a storage cluster?

Accepted Answer

Buckets are sequentially numbered, so removing an arbitrary middle node is not straightforward

Answer

Adding any bucket forces every key to move

Answer

It only works for string keys

Answer

It needs a central table lookup for every key

Question 14

How does rendezvous hashing (HRW) pick a key's node, and what is its advantage?

Accepted Answer

It computes hash(key, node) per node and picks the max; even spread and minimal movement without vnodes

Answer

It places the key on a ring and picks the first node clockwise

Answer

It uses a central directory to choose the node

Answer

It computes hash(key) mod N and picks that node

Question 15

CockroachDB splits hot ranges by load. Why doesn't this spread write load for a table with an ascending primary key (like `SERIAL`)?

Accepted Answer

All inserts always hit the end of the last range, so each new split is again the hot tail

Answer

Load-based splitting only applies to reads

Answer

A `SERIAL` key cannot be split at all in CockroachDB

Answer

Splitting a range causes replicas to be lost

Question 16

Your MySQL runs semi-sync with a 10-second `rpl_semi_sync_master_timeout`. During a network blip the replica stops acknowledging. What is the hidden risk?

Accepted Answer

The master silently falls back to async, and failover in that window can lose acknowledged writes

Answer

The master rejects all writes forever until the replica returns

Answer

The replica auto-promotes itself, causing split brain

Answer

The binlog is purged and the replica must be rebuilt

Question 17

A Raft group has five replicas: two in us-east (leader among them), two in us-west, one in the EU. Commit latency is roughly determined by what?

Accepted Answer

RTT to the nearest replicas forming a 3-node majority with the leader (here us-west)

Answer

RTT to the farthest replica, the EU one

Answer

Only the leader's own disk latency

Answer

The average RTT to all four followers

Question 18

A consensus cluster has 3 nodes in DC1 and 2 in DC2. What happens if DC1 is lost entirely, and what is the standard fix?

Accepted Answer

DC2 lacks a majority and the cluster stops writing; a third site (even just a witness) is needed

Answer

DC2 continues automatically since 2 nodes suffice for quorum

Answer

Adding a node to DC2 (making it 3+3) solves the problem

Answer

Replace consensus with async replication to solve it

Question 19

In MySQL with two active masters (multi-leader), how are duplicate `AUTO_INCREMENT` IDs avoided, and at what cost?

Accepted Answer

With `auto_increment_increment=2` and different offsets; but adding a third master requires reconfiguration

Answer

MySQL automatically detects the conflict and changes the ID

Answer

With sync replication between masters, at no cost

Answer

With LWW on the ID column, deleting the older row

Question 20

An active-active two-region system has two users registering the same username concurrently in different regions. What is the most correct design?

Accepted Answer

Route username claims through a strongly consistent path (single home leader or consensus)

Answer

Each region checks locally and LWW keeps one later

Answer

Merge both registrations with a CRDT set

Answer

Make cross-region replication faster to shrink the conflict window

Question 21

A rule says 'at least one doctor must be on call per shift', but doctor records are sharded by `doctor_id` across shards. Two doctors go off call concurrently. What best preserves the invariant?

Accepted Answer

Co-locate invariant data by `shift_id` so a single-shard serializable transaction protects it

Answer

Check the on-call count on a read replica before writing

Answer

Have each shard check the invariant independently under snapshot isolation

Answer

Run a nightly job after changes to fix violations

Question 22

A team replaces cross-shard 2PC with a saga. What is the most important thing they lose?

Accepted Answer

Isolation: intermediate states are visible and must be handled with compensations and semantic locks

Answer

Durability: saga steps are not persisted to disk

Answer

Availability: a saga halts the whole system if any shard fails

Answer

Retryability: saga steps can never be retried

Question 23

Each shard responds within 10 ms at p99. Roughly what p50 will a query fanned out to 100 shards have?

Accepted Answer

About the per-shard p99, since there's a ~63% chance at least one shard is slow

Answer

About the per-shard p50, since shards run in parallel

Answer

About 100 times the per-shard p50

Answer

Below the per-shard p50, since load is spread out

Question 24

An API using `ORDER BY created_at LIMIT 20 OFFSET 10000` over 50 shards has become slow. What is the root cause and fix?

Accepted Answer

Each shard must return 10,020 rows for a correct merge; use keyset (cursor) pagination

Answer

The shard indexes are corrupted; rebuild them

Answer

Replication lag causes it; read from the leader

Answer

LIMIT 20 is too small; raise it to 1,000

Question 25

EU user data must legally stay in the EU, yet emails must be globally unique. Which design satisfies both?

Accepted Answer

A global index holding only an email hash and region, with personal data kept in the user's region

Answer

Fully copying the users table to all regions for uniqueness checks

Answer

Checking uniqueness only within each region and accepting duplicates across regions

Answer

Storing all data in the EU for every user worldwide

Question 26

How do follower reads in CockroachDB or stale reads in Spanner (e.g., `AS OF SYSTEM TIME` a few seconds ago) allow local follower reads without breaking consistency?

Accepted Answer

The follower answers only at a timestamp it knows no new write can commit below (closed timestamp)

Answer

The follower returns its latest data and hopes it is current

Answer

Every follower read first round-trips to the leader for confirmation

Answer

Followers use sync replication so they are never behind

Question 27

The leader accepts concurrent writes on 32 threads, but the replica applies changes on a single thread and lag keeps growing. What is the most fundamental fix?

Accepted Answer

Parallel replication by writeset or commit group so independent transactions apply concurrently

Answer

Adding more read replicas to spread the apply load

Answer

Switching from row-based to statement-based replication

Answer

Increasing the failure-detection timeout on the replica

Question 28

You require RPO=0 even if an entire availability zone is lost, and the leader is in AZ-a. Which minimal configuration meets this?

Accepted Answer

Synchronous ack from at least one replica in another AZ before each commit is acknowledged

Answer

Two async replicas in AZ-b and AZ-c with sub-second lag

Answer

One sync replica in AZ-a on a different rack

Answer

Snapshots every five minutes to multi-region object storage

Question 29

In Cassandra you `DELETE` a row and immediately `INSERT` it from another app server, but the row is not visible. What is the most likely cause?

Accepted Answer

The second server's clock is behind, so the insert's smaller timestamp loses to the tombstone under LWW

Answer

The insert has not reached any replica yet due to replication lag

Answer

Cassandra forbids inserting after a delete on the same key

Answer

The tombstone permanently locks that partition key

Question 30

A Cassandra node was down for 15 days and `gc_grace_seconds` is 10 days. What is the risk after bringing it back and repairing?

Accepted Answer

Data deleted meanwhile, whose tombstones were purged, comes back from this node

Answer

All new data from the past 15 days gets wiped from the cluster

Answer

The node automatically becomes leader of all partitions

Answer

Repair fails on schema mismatch and has no effect

Question 31

Why does anti-entropy in systems like Cassandra and DynamoDB use Merkle trees?

Accepted Answer

Comparing high-level hashes locates and transfers only differing ranges, not all data

Answer

Merkle trees encrypt data so replica transfer is secure

Answer

Merkle trees determine the causal order of writes

Answer

Merkle trees replace tombstones for deletes

Question 32

In Cassandra with two datacenters (RF=3 each), what is the trade-off of `LOCAL_QUORUM` for reads and writes?

Accepted Answer

Low latency and tolerance of the other DC's outage, but clients in the other DC may read stale data

Answer

Strong cross-DC consistency but cross-DC latency on every operation

Answer

No data is replicated to the second DC at all

Answer

One local node failure halts all operations

Question 33

With `hash(key) % N`, why is doubling the shard count (N to 2N) better than adding one shard?

Accepted Answer

Each shard splits exactly in two, and every key either stays or moves to a known sibling shard

Answer

Doubling means no key moves at all

Answer

Doubling removes the need for a hash function

Answer

Doubling moves only 10% of the data

Question 34

A viral video gets 50k likes/sec on a single counter row. Which design both spreads write load and keeps reads cheap?

Accepted Answer

Split into N random sub-counters and keep a periodically refreshed cached total

Answer

Pessimistic locking on the counter row to prevent lost updates

Answer

Raising the counter row's replication factor to 10

Answer

Moving the counter row to the fastest shard in the cluster

Question 35

Orders are sharded by `region_id` and a user changes the delivery address to another region. What complexity arises?

Accepted Answer

Changing the shard key means moving the row across shards (delete + insert), not a single-shard atomic transaction

Answer

A plain `UPDATE` suffices and the DB moves the row at no cost

Answer

Changing a shard key triggers a full-cluster rebalance

Answer

All replicas of the old shard must be rebuilt

Question 36

In a 500-shard cluster, 'find user by phone number' runs thousands of times per second. Which index fits better and why?

Accepted Answer

A global index, since each lookup hits one partition and async write cost is acceptable

Answer

A local index, since scatter-gather over 500 shards per lookup is cheap

Answer

No index, since a parallel full scan is always faster

Answer

A local index on leaders and a global index on followers

Question 37

You want to upgrade PostgreSQL from 12 to 16 with only seconds of downtime. Which approach fits?

Accepted Answer

Logical replication to a v16 cluster, wait for catch-up, then switch over

Answer

Physical streaming replication to a v16 replica and promote it

Answer

A full `pg_dump` and restore while the service runs

Answer

In-place upgrade on the leader during peak hours

Question 38

Your automatic failover switched leaders four times in an hour due to flaky inter-AZ networking. Which fix is most appropriate?

Accepted Answer

Hysteresis and rate limits on failover (e.g., minimum interval) plus multi-vantage failure detection

Answer

Shortening the timeout so failover completes faster

Answer

Disabling replication until the network stabilizes

Answer

Adding more replicas in the leader's AZ

Question 39

To handle growth, you add three nodes during peak hours and suddenly cluster-wide p99 triples. What is the most likely cause?

Accepted Answer

Rebalancing data streaming is saturating disk and network on existing nodes

Answer

The new nodes run a different hash function version

Answer

The replication factor auto-dropped to 1

Answer

The new nodes are rejecting all writes

Question 40

For a global app, what drives the choice between 'active-active with CRDTs for everything' and 'each record has a home region where its writes are routed'?

Accepted Answer

Whether concurrent multi-region writes to one record are common and whether the data has mergeable semantics

Answer

Whether the database uses SQL or NoSQL

Answer

The total number of tables in the application

Answer

Whether servers run Linux or Windows

Question 41

In a partitioned async-replicated system, a comment and its reply sit in different partitions. Which solution guarantees consistent prefix for readers?

Accepted Answer

Track causal dependencies (e.g., the reply carries the comment's version) and show it only once the dependency is visible

Answer

Using LWW with server clock timestamps on both records

Answer

Raising the replication factor of both partitions

Answer

Reading from a random replica to spread load

Question 42

In Cassandra two concurrent requests read stock, decrement it, and write back (QUORUM). What happens and what is the right fix?

Accepted Answer

A lost update occurs; use a Paxos-based lightweight transaction (`IF stock = ?`)

Answer

QUORUM guarantees both decrements are applied correctly

Answer

Cassandra detects the conflict and rejects one with an error

Answer

Raising consistency to `ALL` solves the problem

Question 43

After an update, the app deletes the cache key. The next request misses and refills from a read replica. Why does stale data stay cached for hours?

Accepted Answer

The replica has not applied the update yet, so the old value is re-cached with a long TTL

Answer

Cache key deletion never actually happens

Answer

The leader rolled the update back

Answer

The cache automatically restores data from backup

Question 44

A ride-hailing service shards drivers by fixed-precision geohash cells. Downtown at rush hour saturates one shard. What is the best redesign?

Accepted Answer

Hierarchical cells (like S2) split into finer cells in dense areas and mapped to different shards

Answer

Hashing `driver_id` and scatter-gathering all shards on each geo search

Answer

Uniformly increasing geohash precision for the whole world

Answer

Moving that shard to a bigger server and keeping the design

Question 45

A 'total orders today' dashboard queries 200 shards every second and creates heavy load. What is the best approach?

Accepted Answer

Stream order events (CDC) to an aggregator that maintains a precomputed counter

Answer

Run `COUNT(*)` on all shards but with a shorter timeout

Answer

Add more replicas to all 200 shards

Answer

Move all of today's orders onto one separate shard

Question 46

A Kafka topic keyed by `order_id` guaranteed per-order event ordering. After increasing partitions from 12 to 24, consumers see out-of-order events. Why?

Accepted Answer

The `hash(key) % partitions` mapping changed, so new events for an order go to a different partition

Answer

Kafka shuffles ordering in all partitions after a partition increase

Answer

The topic's replication factor was automatically reduced

Answer

Consumer groups re-read from the start of the topic after the increase

Question 47

Service A writes to the leader and then publishes an event. Service B, on receiving it, reads from a read replica and cannot find the data. What is the best fix?

Accepted Answer

Include the needed data or commit position (LSN) in the event so B uses it or a caught-up replica

Answer

Adding a fixed `sleep` before reading in service B

Answer

Publishing the event before the transaction commits for speed

Answer

Adding more read replicas so lag shrinks

Question 48

In a ring with RF=3, each key's replicas go on the next 3 nodes clockwise. After one AZ outage, some keys become fully unavailable. What is the design flaw?

Accepted Answer

Replica selection is not AZ/rack-aware, so the next 3 nodes may share an AZ

Answer

RF=3 is never enough for any distributed system

Answer

Consistent hashing is inherently incompatible with multiple AZs

Answer

There are too many vnodes

Question 49

Spanner, using TrueTime, performs a 'commit wait' equal to clock uncertainty on each commit. What does this guarantee?

Accepted Answer

Timestamp order matches real-time commit order across regions (external consistency)

Answer

No replica will ever have any lag

Answer

Paxos is no longer needed for replication

Answer

Writes commit in all regions with zero delay

Question 50

Failover completes in 10 seconds, yet some services keep hitting the old (now read-only) leader for minutes and their writes fail. Most likely cause and fix?

Accepted Answer

DNS caching or long-lived connection pools; use a topology-aware proxy (e.g., ProxySQL/PgBouncer) or short TTLs plus closing old connections

Answer

The new leader hasn't applied the log yet; enable semi-sync

Answer

Fencing is too strict; disable it

Answer

The replication factor is too low; add more replicas