Consistency & Distributed Transactions: Understanding CAP, ACID, and When to Compromise

TLDR: You can't have strong consistency + availability + partition tolerance. The CAP theorem forces a choice. Distributed transactions (2PC) are slow and fragile. Most systems choose availability + eventual consistency (BASE). Consensus algorithms (Raft) work but have latency costs. CRDTs solve some problems without consensus.

Part 1: ACID vs BASE

ACID: The Database Dream

ACID transactions guarantee that operations are:

Atomic: All-or-nothing (no partial failures)
Consistent: Data moves from valid state to valid state
Isolated: Concurrent operations don't interfere
Durable: Committed data survives crashes

Sql

Transfer 100 from Account A to Account B:

BEGIN TRANSACTION;
  UPDATE accounts SET balance = balance - 100 WHERE id = 'A';
  UPDATE accounts SET balance = balance + 100 WHERE id = 'B';
COMMIT;

Guarantee: Either both updates happen, or neither.
           Money doesn't vanish or duplicate.
           Even if the database crashes mid-transaction.

Cost: Locks (serialization), reduced concurrency, single machine or complex consensus.

BASE: The Distributed Reality

BASE (Basically Available, Soft state, Eventually consistent) accepts relaxed guarantees:

Basically Available: System responds even if some replicas fail
Soft State: Data may be inconsistent at a moment in time
Eventually Consistent: System converges to consistency over time

Transfer 100 from Account A to Account B (distributed):

T=0:  
  POST /transfer {from: A, to: B, amount: 100}
  Account A immediately deducts: balance -= 100
  Event: TransferInitiated published

T=5ms:
  Account A: balance = 890
  Account B: balance = 910 (event not yet processed)
  Inconsistent, but...

T=50ms:
  Account B processes TransferInitiated event
  Account B: balance = 910 (updated)
  Consistent again

Guarantee: Eventually both accounts will be consistent.
           But there's a window where they're not.

Benefit: High availability, no locks, distributed systems work.

Part 2: The CAP Theorem

The Theorem (Stated Simply)

In a distributed system, you can have:

Consistency (all replicas see same data)
Availability (system responds to requests)
Partition tolerance (system works even if network partitions)

You can pick 2 out of 3.

CP system:
  ├─ Consistency + Partition tolerance
  └─ Trade: Unavailable during partitions
     Example: Consensus-based system (Raft)

AP system:
  ├─ Availability + Partition tolerance
  └─ Trade: Temporarily inconsistent
     Example: Dynamo, Cassandra

CA system:
  ├─ Consistency + Availability
  └─ Trade: Cannot tolerate partition (single machine or tight LAN)
     Example: Single-machine database, Google's Bigtable (no WAN partition)

CAP in Practice

The Catch: Partition tolerance isn't optional in distributed systems.

Network partitions happen. Full stop.

Reality in a 3-node cluster:

┌────────┐        ┌────────┐        ┌────────┐
│ Node 1 │        │ Node 2 │        │ Node 3 │
│        │───────→│        │───────→│        │
│ Replica│        │Replica │        │Replica │
│ v=10   │        │ v=10   │        │ v=10   │
└────────┘        └────────┘        └────────┘

Client writes to Node 1: v = 20

Client expects: All nodes eventually v=20

But then... network partition!

┌────────┐                          ┌────────┐
│ Node 1 │                          │ Node 2 │
│ Replica│ ════X════ (partition)    │Replica │
│ v=20   │                          │ v=10   │
└────────┘                          └────────┘
                                       │
                                       ↓
                                    ┌────────┐
                                    │ Node 3 │
                                    │Replica │
                                    │ v=10   │
                                    └────────┘

Now what?

CP choice: Reject writes to Node 2/3 (unavailable during partition)
           ├─ Node 1 can serve writes (has v=20)
           └─ Nodes 2/3 reject writes (consistency)

AP choice: Allow writes to both sides
           ├─ Partition 1 (Node 1): v becomes 25 (client writes 20+5)
           ├─ Partition 2 (Nodes 2/3): v becomes 15 (client writes 10+5)
           └─ After partition heals: CONFLICT (which v is right?)

The CAP Theorem is Misleading

Better framing: It's about partition recovery.

When a network partition heals, you choose:

Consistency: Merge conflicts carefully, may lose writes
Availability: Accept conflicts, merge via multi-value records (Vector Clocks)

Partition heals:

CP system (Raft, Consensus):
  ├─ One partition becomes leader (majority quorum)
  └─ Minority partition reverts writes (lost updates)
     ├─ Consistent (one value of truth)
     └─ Unavailable partition lost writes

AP system (Dynamo):
  ├─ Both partitions accept writes
  ├─ Partition heals → detect conflict
  └─ Options:
     ├─ Last-write-wins (client A wins, client B loses)
     ├─ Multi-value (return both, let client decide)
     └─ Application merge (CRDTs)

Part 2.5: The PACELC Theorem (CAP's Missing Half)

CAP only describes behaviour during a partition. But partitions are rare. What happens the rest of the time?

PACELC extends CAP:

If Partition → choose Availability or Consistency (same as CAP)
Else (no partition) → choose Latency or Consistency

PACELC:

              Partition?
             /          \
           YES           NO
          /                \
   A vs C               L vs C
(same as CAP)     (latency vs consistency
                   even when healthy)

Why this matters: A system can behave well during partitions but still be slow under normal conditions because it prioritizes consistency over latency.

PACELC Classification of Common Databases

Database        Partition choice   Else choice    Summary
──────────────────────────────────────────────────────────
MySQL/Postgres  PC (consistent)    EC (consistent) PC/EC
  └─ Single-node or synchronous replication
  └─ Prioritizes consistency always

DynamoDB        PA (available)     EL (low latency) PA/EL
  └─ Accepts writes during partition
  └─ Optimizes for low latency when healthy

Cassandra       PA (available)     EL (low latency) PA/EL
  └─ Tunable consistency (quorum reads/writes)
  └─ Default: availability + low latency

Google Spanner  PC (consistent)    EC (consistent) PC/EC
  └─ Global consensus (Paxos + TrueTime)
  └─ Correct always, expensive in latency

MongoDB         PC (consistent)    EC (consistent) PC/EC
  └─ Primary elected by Raft
  └─ Reads from primary only (default)

Riak            PA (available)     EL (low latency) PA/EL
  └─ CRDT-based, always available
  └─ Eventual consistency

Practical Implication

Choosing DynamoDB (PA/EL):
  ├─ Partition: still accepts writes (availability)
  └─ Normal: returns fast (low latency)
  → Accept: stale reads, eventual consistency

Choosing Spanner (PC/EC):
  ├─ Partition: minority nodes reject writes (consistency)
  └─ Normal: waits for global consensus (higher latency)
  → Accept: higher cost, higher latency, for guaranteed correctness

The PACELC framing forces you to ask:
  "Even when everything is working, am I okay
   trading latency for consistency?"

  YES → use PA/EL systems (Cassandra, DynamoDB)
  NO  → use PC/EC systems (Spanner, Postgres)

Part 3: Consistency Models (Beyond CAP)

Strong Consistency (Linearizability)

Every operation appears to happen at a single instant, in order.

Timeline:

Client A: Write(x=5) ━━→ OK
            ↓
         (instant in time)
            ↓
Client B: Read(x) ━━→ returns 5 (sees A's write)

Guarantee: If A's write succeeded, every read after must see 5.

Implementation: Single leader, synchronous replication.

┌───────────────┐
│   Leader      │
│ (x=5)         │
└───┬───────────┘
    │
    ├─→ Replicate to Follower 1
    ├─→ Replicate to Follower 2
    └─→ (wait for ack from majority before returning OK)

Latency: Depends on slowest replica.
         If replicas are in 3 regions → high latency.

Eventual Consistency

All replicas converge to same value, given no new writes.

Timeline:

T=0:  Writer A: Write(x=5)
      └─→ Write to Replica 1 ✓
      └─→ Returns OK immediately (not waiting for others)

T=5ms:  Replica 2 receives: x=5
        Replica 3 receives: x=5

T=10ms: All replicas have x=5 (consistent)

T=15ms: Reader B: Read(x)
        └─→ Gets 5 (from any replica)

Strength: Write doesn't block waiting for replicas. Read can use any replica.

Weakness: Between T=0 and T=10ms, replicas are inconsistent. Old writes propagate.

Causal Consistency

Cause-effect relationships are respected. If A's write caused B's read, B sees A's write.

Timeline:

T=0: User A logs in
     ├─ Auth service writes: user_session_id=12345
     └─ Publishes: UserLoggedIn event

T=5ms: User A views profile
       ├─ Profile service subscribes to UserLoggedIn
       └─ Reads: user_session_id=12345 (sees the session)
       
Guarantee: B's operation sees A's operation (causal chain).

But unrelated operations might be out of order.

Implementation: Vector clocks, happens-before relations.

Session Consistency (Read-Your-Writes)

Within a session, you always see your own writes.

User's browser session:

POST /update-profile {name: "Alice"}
  └─ Returns OK

GET /profile
  └─ Returns name: "Alice" (even if replicas haven't synced yet)
  
Guarantee: Client's subsequent reads see client's writes.

Implementation: Session-sticky routing (client always hits same server) or monotonic reads.

Monotonic Consistency (Monotonic Reads)

If you read value V1, later reads are >= V1.

T=0: Client reads user.followers=100
     (hits Replica A)

T=5ms: Replica B updates user.followers=105

T=10ms: Same client reads user.followers
        └─ Returns 105 (at least as much as last read)
        
What it prevents:
  ├─ Read=100 (Replica A, version 5)
  ├─ Read=95 (Replica B, version 3 - older!)
  └─ Seeing backwards-in-time values

Part 4: The 2PC Problem (and Why It Fails)

Two-Phase Commit (2PC)

Distributed transaction across multiple databases.

Transaction: Transfer 100 from DB1 to DB2

Phase 1: PREPARE
┌──────┐                    ┌──────┐
│ DB1  │────PREPARE──────→│ DB2  │
│      │  (Can you commit?)│      │
└──────┘                    └──────┘
   │                            │
   ├─ Lock row in DB1      ├─ Lock row in DB2
   ├─ Write undo log       ├─ Write undo log
   └─ ACK OK (promise)     └─ ACK OK (promise)

Coordinator hears: Both DBs ready to commit

Phase 2: COMMIT
┌──────┐                    ┌──────┐
│ DB1  │────COMMIT──────→│ DB2  │
│      │                    │      │
└──────┘                    └──────┘
   │                            │
   ├─ Release lock         ├─ Release lock
   ├─ Make changes durable ├─ Make changes durable
   └─ ACK DONE            └─ ACK DONE

Guarantee: Either both commit or both rollback. No partial success.

The 2PC Problems

Problem 1: Blocking Locks

Phase 1: DB1 locks row
         └─ Row unavailable for 500ms while waiting for DB2 response
         
Database becomes a bottleneck.
Other transactions wait.
Throughput plummets.

Problem 2: Coordinator Failure

Coordinator crashes after Phase 1 (prepare) but before Phase 2 (commit).

┌──────┐  Phase 1 done   ┌──────┐
│ DB1  │───────────────→│ DB2  │
│      │   ACK OK        │      │
└──────┘   (lock held)    └──────┘

Coordinator: CRASHED 💥

Now:
  ├─ DB1 is locked (waiting for commit/abort)
  ├─ DB2 is locked (waiting for commit/abort)
  ├─ Can't proceed
  ├─ Can't rollback (coordinator is dead)
  └─ Locks held indefinitely

Remedy: Coordinator recovery log, replay. Takes time.
        Meanwhile, deadlock.

Problem 3: Network Partition

Coordinator sends Phase 2 (commit) to DB2.
Network fails.
Coordinator doesn't hear ACK.

Question: Did DB2 commit?

├─ If coordinator assumes "yes" and tells DB1 to commit: Both committed ✓
├─ If coordinator assumes "no" and tells DB1 to rollback: DB1 rolled back, DB2 committed ✗
└─ Unknown = conflict

Problem 4: Scalability

Systems using 2PC:

Monolith + Monolith: Feasible (same data center, low latency)
Monolith + Monolith + Monolith: Getting hard
Monolith × 10: Disaster

Each additional service = exponential increase in failure modes.
2PC doesn't scale beyond ~3 systems.

Why 2PC is Still Used

Banks, financial systems, auditing:
  ├─ Correctness is non-negotiable
  ├─ Locks are acceptable (transaction is seconds, not minutes)
  ├─ Consistency > Availability
  └─ Use 2PC despite costs

E-commerce, social media:
  ├─ Availability is non-negotiable
  ├─ Inconsistency for milliseconds is okay
  ├─ User tolerance for delays is low
  └─ Avoid 2PC, use sagas + eventual consistency

Part 4.5: Idempotency & Deduplication

When you can't use 2PC (too slow) and sagas add complexity, idempotency is your safety net: calling the same operation multiple times produces the same result as calling it once.

The Problem: At-Least-Once Delivery

Client sends: POST /payments/charge {amount: 100}
  ├─ Server processes charge ✓
  ├─ Server sends response...
  └─ Network fails! Client never receives ACK.

Client retries: POST /payments/charge {amount: 100}
  └─ Server charges again → customer charged twice ✗

Without idempotency, retries cause duplicate operations. This is a real production incident waiting to happen.

The Solution: Idempotency Keys

Client generates a unique key per logical operation:
  idempotency_key = UUID v4 (client-generated, e.g. "txn-a1b2c3d4")

First call:
  POST /payments/charge
  Headers: Idempotency-Key: txn-a1b2c3d4
  Body: {amount: 100, account: "alice"}

  Server:
  ├─ Check idempotency store: key "txn-a1b2c3d4" → NOT FOUND
  ├─ Execute charge (debit account)
  ├─ Store result: {key: "txn-a1b2c3d4", result: {charge_id: "ch_123", status: "success"}}
  └─ Return: {charge_id: "ch_123", status: "success"}

Duplicate call (retry after network failure):
  POST /payments/charge
  Headers: Idempotency-Key: txn-a1b2c3d4
  Body: {amount: 100, account: "alice"}

  Server:
  ├─ Check idempotency store: key "txn-a1b2c3d4" → FOUND
  ├─ Skip execution (no double charge)
  └─ Return: {charge_id: "ch_123", status: "success"} (cached result)

Implementation

Java

// Idempotent payment handler (Spring Boot sketch)
@PostMapping("/payments/charge")
public ChargeResponse charge(
    @RequestHeader("Idempotency-Key") String idempotencyKey,
    @RequestBody ChargeRequest request) {

  // Check cache first
  Optional<ChargeResponse> cached = idempotencyStore.get(idempotencyKey);
  if (cached.isPresent()) {
    return cached.get(); // Return previous result, no side effects
  }

  // Execute once
  ChargeResponse result = paymentGateway.charge(request);

  // Store with TTL (24h typical for payment APIs)
  idempotencyStore.put(idempotencyKey, result, Duration.ofHours(24));
  return result;
}

Idempotency Store Options

Redis (recommended for most cases):
  ├─ SET idempotency:{key} {result} EX 86400
  ├─ Fast reads, automatic expiry
  └─ Risk: if Redis fails, lose idempotency cache

Database (for financial systems):
  ├─ INSERT INTO idempotency_log (key, result, expires_at)
  ├─ Transactional with the operation itself
  └─ Durable, but slower

Where Idempotency Is Non-Negotiable

Payment APIs:         Stripe uses idempotency keys on every write endpoint
Webhook delivery:     Exactly-once delivery = idempotent handler
Message consumers:    At-least-once queue delivery → idempotent processor
Distributed sagas:    Each saga step must be idempotent (compensations retry)
Form submissions:     Double-click prevention

Rule of thumb: Any operation that has side effects (charge money, send email, write to DB) and can be retried must be idempotent.

Part 5: Consensus Algorithms (Raft)

Consensus = all nodes agree on a value.

The Problem Consensus Solves

3 nodes, no consensus:

Node 1: x = 10
Node 2: x = 20
Node 3: x = 15

Which is correct? No way to know. Inconsistency.

With consensus:

All nodes must agree:
  ├─ x = 10 (agreed by majority)
  ├─ Minority forced to update to x=10
  └─ Consistent

Raft Consensus Algorithm

Raft is the practical consensus algorithm (Paxos is theoretically sound but complex).

Raft Overview:

1. ELECTION PHASE
   ├─ Random node declares itself leader (heartbeat)
   ├─ Other nodes vote
   └─ Winner with majority becomes leader for term T

2. LOG REPLICATION
   ├─ Leader receives write request
   ├─ Appends to its log
   ├─ Sends to followers
   ├─ Waits for majority ACK
   └─ Commits and replies to client: OK

3. SAFETY
   ├─ If leader crashes, election happens
   ├─ New leader elected from nodes with latest logs
   └─ Minority partitions have no majority, can't become leader

Visual:

3-node cluster:

┌──────────────┐
│   Leader     │
│   Term: 5    │
└──────┬───────┘
       │ Heartbeat
       │ (AppendEntries RPC)
       │
    ┌──┴───────┬──┐
    ▼          ▼  ▼
┌────────┐ ┌────────┐
│Follower│ │Follower│
│Term: 5 │ │Term: 5 │
└────────┘ └────────┘

Client writes to leader:
  ├─ Log: [entry at index 10]
  ├─ Send to followers
  ├─ Followers ACK
  └─ Leader commits (replicated to majority)
     └─ Notifies client: OK

Raft Consensus Cost

Write latency in 3-node cluster:

Write(x=5):
  ├─ Leader appends to log (1ms)
  ├─ Sends to Follower 1 (5ms network)
  │  └─ Follower 1 appends (1ms)
  │     └─ ACK to leader (5ms)
  ├─ Sends to Follower 2 (5ms network)
  │  └─ Follower 2 appends (1ms)
  │     └─ ACK to leader (5ms)
  ├─ Leader hears 2 ACKs (majority)
  │  └─ Commits to state machine (1ms)
  └─ Returns to client: OK

Total: ~20ms for a local write (5ms × 2 roundtrips + processing)

Compare to non-replicated write: 1ms
        Consensus adds 20x latency

For write-heavy workloads, this is brutal.

When to Use Consensus (Raft)

Use Raft:
  ├─ Small clusters (3-5 nodes, not 100)
  ├─ Strong consistency required
  ├─ Write frequency is low
  ├─ Acceptable latency is 50ms+
  └─ Example: Distributed lock service (etcd, Consul)

Avoid Raft:
  ├─ High-frequency writes
  ├─ Latency-sensitive (p99 < 10ms)
  ├─ Large clusters
  └─ Example: Real-time analytics, user-facing API

Part 6: CRDTs (Conflict-free Replicated Data Types)

What if we could avoid consensus and still get consistency?

The CRDT Idea

CRDTs are data structures that can be replicated across systems without coordination.

Property: Replicas can diverge, then merge automatically without conflict.

Example: Counter CRDT

Replica A increments: counter += 5
Replica B increments: counter += 3

No synchronization needed. When replicas merge:
  ├─ Merge is idempotent (safe to apply multiple times)
  ├─ Merge is commutative (order doesn't matter)
  └─ Result: 5 + 3 = 8 (automatic, no conflict resolution)

Common CRDTs

1. Counter CRDT

Structure:
  ├─ Inc counter (increments only)
  └─ Dec counter (decrements only)

Merge: Sum all Inc, sum all Dec → final count

Benefits:
  ├─ Can increment independently on any replica
  ├─ Merge is automatic
  └─ No conflicts (unlike registers)

2. Set CRDT (Add-Only Set)

Members: {Alice, Bob}

Replica A adds Charlie
  → Members: {Alice, Bob, Charlie}

Replica B adds Dave
  → Members: {Alice, Bob, Dave}

Merge:
  → Members: {Alice, Bob, Charlie, Dave}

Automatic union, no conflicts.

3. Register CRDT (Multi-Value)

Problem: Register can only hold one value.
         If two replicas set different values, conflict.

Solution: Return all concurrent writes as "siblings"

┌──────────────┐        ┌──────────────┐
│ Replica A    │        │ Replica B    │
│ name="Alice" │        │ name="Bob"   │
└──────────────┘        └──────────────┘

Merge:
  └─ name: ["Alice", "Bob"] (multi-value)
     └─ Application decides: Use "Bob" (last-write-wins, vector clock)

4. Text CRDT (Operational Transform)

Problem: Two users edit document simultaneously.

User A inserts "hello" at position 0
User B inserts "world" at position 0

Without CRDT:
  ├─ Result A: "hello..."
  ├─ Result B: "world..."
  └─ Conflict!

With CRDT (Operational Transform):
  ├─ A's insert(0, "hello") → metadata: Lamport clock
  ├─ B's insert(0, "world") → metadata: Lamport clock
  ├─ Merge: Reorder by clock → "helloworld" or "worldhello" (deterministic)
  └─ Same result on all replicas

Examples: Google Docs, Figma (both use operational transforms)

CRDT Trade-Offs

Advantages:

✓ No coordination (replicas are independent)
✓ Automatic merge (convergence guaranteed)
✓ High availability (always accept writes)
✓ Low latency (writes local, async replication)

Disadvantages:

✗ Memory overhead (tombstones, metadata)
✗ Merge overhead (complex merge logic)
✗ Limited data structures (not all types have efficient CRDTs)
✗ Semantic mismatch (you think "set value" but CRDT does "concurrent values")

When to Use CRDTs

Good fit:
  ├─ Collaborative editing (Google Docs)
  ├─ Real-time multiplayer (Figma)
  ├─ Counters, sets, lists
  └─ High availability > strong consistency

Bad fit:
  ├─ Relational data (complex merges)
  ├─ Financial transactions (conflicts unacceptable)
  ├─ Unique constraints (two CRDTs create duplicates)
  └─ Complex business logic

Part 7: Consistency in Practice

Decision Tree

                 ┌─ Consistency requirement?
                 │
        ┌────────┴────────┐
        │                 │
    STRONG            EVENTUAL
    (ACID)          (BASE)
        │                 │
    Transaction           │
    frequency?            │
        │                 │
    ┌───┴────┐           │
   LOW     HIGH          │
    │       │            │
    │    → 2PC        CRDTs?
    │      (bad!)        │
    │                    │
   Raft            ┌─────┴─────┐
   (consensus)    YES         NO
    │               │          │
    │         (Figma,     Sagas +
    │         Docs)      Eventual
    │                   Consistency
    │                   (standard)
    └─────→ Strong
           Consistency

Real-World Examples

LinkedIn (Availability-First)

Architecture:
  ├─ Replicated KV store (Espresso)
  ├─ Eventually consistent across datacenters
  ├─ Eventual consistency window: seconds
  
Why:
  ├─ Profile updates are not critical
  ├─ Users tolerate eventual updates
  ├─ Availability > consistency
  
SLA: 99.99% availability, inconsistency windows ≤ 5 seconds

Google Spanner (Consistency-First)

Architecture:
  ├─ TrueTime API (GPS + atomic clocks)
  ├─ Strong consistency across globe
  ├─ Consensus replication (Paxos)
  
Why:
  ├─ Financial data (Google's payments)
  ├─ Correctness non-negotiable
  
Cost: High latency (waits for consensus), high infrastructure cost

Stripe (Hybrid)

Architecture:
  ├─ Core ledger (PostgreSQL, ACID)
  │  └─ Payments must be transactionally consistent
  ├─ Replicated caches (eventual consistency)
  │  └─ Balances, statement history
  ├─ Async events (sagas)
  │  └─ Billing, analytics, notifications
  
Why:
  ├─ Money = strong consistency (2PC acceptable for correctness)
  ├─ Displays = eventual (cache can be seconds stale)
  ├─ Operations = async (notifications can fail, retry)
  
Strategy: Use strong consistency where it matters, eventual elsewhere

Part 8: The Trade-Off Matrix

System Property        Strong Consistency    Eventual Consistency
─────────────────────────────────────────────────────────────────
Write latency          High (wait for quorum) Low (write local)
Availability           Lower                  Very high
Partition tolerance    Loses availability      Loses consistency
Scalability (nodes)    Poor (consensus)       Excellent
Merge conflicts        Never                   Possible
Operational complexity Medium                 High (handling conflicts)
Data freshness         Immediate              Delayed
Use cases              Finance, locks,        Social media, caching,
                       elections              real-time collab

Part 9: Practical Consistency Guarantees

When Consistency Actually Matters

Strong Consistency Required:
  ├─ Money movement (Stripe, banks)
  ├─ Inventory (critical low stock)
  ├─ Permissions (access control)
  ├─ Unique constraints (email, username)
  └─ → Use: Raft, 2PC, or single source of truth

Eventual Consistency OK:
  ├─ Profile updates (name, bio)
  ├─ Social media feeds
  ├─ Recommendations
  ├─ Page views, likes, shares
  ├─ Recent comments
  └─ → Use: Event-driven, sagas, async

Time-Limited Inconsistency OK:
  ├─ Shopping cart (stale for 5 minutes)
  ├─ Product catalogs (stale for 1 minute)
  ├─ User balance display (eventual within 30 seconds)
  └─ → Use: Caching + TTL

Hybrid Approach (Best in Practice)

E-commerce order:

1. Strong consistency tier:
   ├─ Inventory (must not oversell)
   └─ Payments (must not double-charge)
   └─ Implementation: Transactional with locks

2. Eventual consistency tier:
   ├─ Order display (eventual within 1 second)
   ├─ User order history (eventual within 5 seconds)
   └─ Implementation: Event-driven, async replication

3. Cached tier:
   ├─ Product listings (cache TTL 5 minutes)
   ├─ User profile (cache TTL 1 minute)
   └─ Implementation: Redis with invalidation

Result:
  ├─ Core transactions are consistent
  ├─ Displays are eventually consistent
  ├─ Caches are fast
  ├─ High availability
  └─ Balanced trade-offs

Conclusion

Consistency is not binary. It's a spectrum.

Strong consistency guarantees correctness but costs latency and availability
Eventual consistency guarantees availability but requires handling conflicts
CRDTs provide automatic consistency for specific data types
Consensus (Raft) works but has latency costs

The goal: Use strong consistency where it matters (financial transactions), eventual consistency everywhere else (profiles, feeds, notifications).

The hard part: Distinguishing between "matters" and "doesn't."

Ask:

If this data is wrong, what's the impact? (financial loss vs. display delay)
How long can we tolerate inconsistency? (1ms vs. 1 second vs. 1 minute)
What's the cost of consistency? (latency, complexity, availability)
What's the cost of inconsistency? (conflicts, user confusion, lost data)

Choose accordingly.

Up next: API Design at Scale: REST, GraphQL, and gRPC — now that your services are consistent, how do they talk to each other and to clients? REST, GraphQL, and gRPC each make different trade-offs. We'll build a decision framework for choosing the right protocol per use case.