Online Auction (eBay)

Sharpened prompt. Design an auction platform where 200,000 people watch a single listing, thousands of bids land in the final five seconds, automatic proxy bids resolve correctly and instantly, the winner is unambiguous and legally defensible, and the auction closes at exactly the right moment even if a server dies one second before.

The defining constraint is that bid processing must be linearizable per auction — there is a single, total, auditable order of bids — while everything else (watching, browsing, notifications) must scale independently of that serialisation point.

1. Problem framing

Functional requirements

• List an item with a start price, reserve, and end time.
• Place a bid; support proxy bidding (bid up to a maximum automatically).
• Broadcast the current price to all watchers in real time.
• Close the auction and determine the winner; handle reserve-not-met.

Non-functional requirements

Property	Target	Consequence
Bid ordering	Linearizable per auction	One serialisation point per auction — an actor, not a lock
Bid latency	P99 < 200ms	In-memory processing; the database is written asynchronously
Close accuracy	Within 100ms of the stated end time	A reliable distributed timer, with fencing
Broadcast	200k watchers, ≤ 1s stale	Fan-out tree; the actor must not write 200k sockets
Auditability	Every bid reconstructable years later	Append-only event log is the source of truth

Back-of-the-envelope

Auctions:   100M active listings, ~1M ending per hour = ~280 closings/sec
Bids:       ~50M bids/day = 580/sec avg
            BUT: a hot auction takes 5,000 bids in its final 10 seconds = 500/sec
            on ONE auction. That single-key rate is the design constraint.
Watchers:   200k concurrent on a hot listing; ~10M concurrent across all listings
Broadcast:  price changes ~50/sec on a hot auction × 200k watchers = 10M msg/sec
            -> must be sampled/coalesced and distributed through a tree
Timers:     100M pending close events, ~280 firing/sec

500 bids/sec on a single key is what rules out optimistic database concurrency: at that rate, retry storms make throughput collapse rather than degrade.

2. High-level architecture

The whole design hangs off one actor per auction: a single-threaded mailbox that owns the auction's state. Bids are processed sequentially by construction, so there are no locks, no transactions, and no races — the hard problem is moved from concurrency control to actor placement and failover.

3. Component inventory

Component	Concrete choice	Why this one
Actor runtime	Akka/Pekko Cluster Sharding, Elixir/OTP, or Cloudflare Durable Objects	Entity-per-actor with location transparency and automatic rebalancing
Event log	Kafka partitioned by auction_id	Append-only truth, replayable for recovery and audit
Snapshot store	Postgres	Fast actor rehydration and queryable results
Timers	Hierarchical timing wheel in memory + durable schedule table	Same structure as the job scheduler
Fencing	etcd leases with monotonic revisions	Prevents a paused actor from acting after failover
Broadcast	Per-auction hub → relay tier	Same tree as live comments

4. The toughest parts

4.1 Linearizable bids without database locks

Why it's hard. Two bids of $100.01 and $100.02 arrive within a millisecond in the final seconds. With row locking, 500 bids/sec on one row means every transaction queues behind the previous one, holding a database connection while it waits; connection pools exhaust and the whole shard degrades. With optimistic concurrency, nearly every transaction fails its version check and retries, and the retry rate grows with contention — throughput collapses rather than plateauing.

Solution — an actor per auction: a single-threaded mailbox that owns the state.

class AuctionActor(auctionId: Long) extends PersistentActor {
  // In-memory authoritative state. No locks needed: the mailbox IS the lock.
  private var currentPrice: Long = 0        // integer cents, never floats
  private var highBidder: Long = 0
  private var maxProxy: Long = 0            // the leader's hidden maximum
  private var closesAt: Long = 0
  private var version: Long = 0

  def receiveCommand: Receive = {
    case PlaceBid(bidder, maxAmount, requestId) =>
      if (System.currentTimeMillis() > closesAt) { sender() ! Rejected("closed"); return }
      if (seenRequests.contains(requestId))      { sender() ! seenRequests(requestId); return }

      val outcome = resolveProxy(bidder, maxAmount)

      // Persist BEFORE acknowledging. The event log is the source of truth;
      // in-memory state is a projection of it.
      persist(BidPlaced(auctionId, bidder, maxAmount, outcome, version + 1)) { e =>
        applyEvent(e)
        seenRequests.put(requestId, outcome)     // idempotent retries
        sender() ! outcome
        context.system.eventStream.publish(PriceChanged(auctionId, currentPrice))
      }
  }
}

Three consequences worth stating. Ordering is free — the mailbox imposes a total order, so "who bid first" has an unambiguous answer. Throughput per auction is high — a single-threaded actor comfortably handles thousands of in-memory bids per second, far above the 500/sec peak. And the bottleneck moves to persistence, which you solve by batching event-log appends rather than by weakening the ordering guarantee.

The requestId deduplication is essential: a bidder's client retries after a timeout, and without it a retry could place a second, higher bid against themselves.

Placement and failover are the real work. Consistent-hash auction_id to a node (Akka Cluster Sharding does this for you); on node loss, the actor is recreated elsewhere and rehydrates from snapshot + events since snapshot — typically under 200 ms. Bids during that window are retried by the client. Add a fencing token from etcd so a paused old actor cannot append to the log after a new one has taken over, exactly as in the job scheduler.

4.2 Proxy bidding: the rule that is easy to get subtly wrong

Why it's hard. eBay-style proxy bidding means a bidder submits a maximum, and the system bids on their behalf only as much as needed to lead. The resulting price depends on the second-highest maximum, not the highest — and the edge cases (tied maximums, a new bid below the current leader's hidden max, bid increments) are where implementations quietly produce wrong outcomes that users notice and dispute.

Solution — implement it as an explicit second-price rule with a documented tie-break.

def resolve_proxy(state, bidder, new_max_cents) -> Outcome:
    inc = increment_for(state.current_price)          # e.g. $1 below $100, $2.50 above

    # Case 0: below the minimum acceptable bid.
    if new_max_cents < state.current_price + inc:
        return Rejected("below_minimum", minimum=state.current_price + inc)

    # Case 1: first bid on the auction.
    if state.high_bidder is None:
        return Leading(bidder=bidder, price=max(state.start_price, state.current_price),
                       hidden_max=new_max_cents)

    # Case 2: the challenger cannot beat the incumbent's hidden maximum.
    #         The incumbent stays in front, at the challenger's max plus one increment
    #         (capped at the incumbent's own max — never bid them above their limit).
    if new_max_cents <= state.max_proxy:
        # Tie: the EARLIER bid wins. Ties must be resolved by time, deterministically.
        new_price = min(state.max_proxy, new_max_cents + (0 if new_max_cents == state.max_proxy else inc))
        return Outbid(leader=state.high_bidder, price=new_price, bidder_max=new_max_cents)

    # Case 3: the challenger takes the lead, paying just enough to beat the
    #         incumbent's maximum — the second-price outcome.
    new_price = min(new_max_cents, state.max_proxy + inc)
    return Leading(bidder=bidder, price=new_price, hidden_max=new_max_cents,
                   outbid=state.high_bidder)

Three details that matter and are commonly wrong. Tied maximums go to the earlier bid — you must define this and apply it consistently, because it decides real money. Never raise the incumbent above their own maximum, hence the min(). And the hidden maximum must never leak, which means it cannot appear in any API response, any WebSocket payload, or any cache the challenger can read — a genuine security boundary, not just a UI concern.

Because this runs inside the actor, it is a pure function over in-memory state with no I/O, which makes it both fast and exhaustively unit-testable. Point that out: the actor model makes the tricky business logic trivially testable, which is a real engineering argument for it beyond concurrency.

4.3 Sniping and the soft close

Why it's hard. Sophisticated bidders place their real bid in the final second, leaving no time for anyone to respond. This is rational behaviour that produces a worse market: prices are lower, casual bidders lose repeatedly and disengage. It also concentrates the entire auction's load into a two-second window — thousands of bids, a broadcast storm, and a close event all at once.

Solution — soft close (automatic extension), implemented inside the actor so it is race-free.

SOFT_CLOSE_WINDOW = 120      # seconds
EXTENSION         = 120

def on_accepted_bid(state, now):
    if state.closes_at - now < SOFT_CLOSE_WINDOW:
        state.closes_at = now + EXTENSION
        state.extensions += 1
        # Reschedule the close timer with a FENCING TOKEN so the old timer,
        # if it fires anyway, is recognised as stale and ignored.
        state.close_token += 1
        timers.reschedule(state.auction_id, at=state.closes_at, token=state.close_token)
        broadcast(AuctionExtended(state.closes_at, state.extensions))

Because extension is decided inside the actor's serialised loop, there is no race between "the bid arrives" and "the timer fires" — both are messages in the same mailbox, and whichever is processed first wins deterministically. That is a concrete correctness benefit of the actor model worth naming explicitly.

The close token handles the reschedule race: a timer scheduled for the old close time may already be in flight when the extension happens. Tagging every close message with the token that was current when it was scheduled lets the actor discard stale ones — the same fencing idea applied to timers rather than leadership.

Soft close also flattens the load spike, which is a nice secondary benefit: instead of 5,000 bids in the last two seconds, activity spreads across several two-minute extensions.

4.4 The close event must fire exactly once, on time

Why it's hard. 100M pending auctions cannot be scanned. The close must happen within ~100 ms of the stated time — late is a legal and trust problem, early is worse. And it must fire exactly once: closing twice could notify two winners or double-charge, while not closing at all leaves an auction in limbo forever.

Solution — a two-tier timer (durable schedule + in-memory timing wheel) with fencing and an idempotent close handler.

# Tier 1: durable schedule, partitioned by minute bucket.
#   INSERT INTO auction_closes (minute_bucket, auction_id, closes_at, token)
# Tier 2: each shard's leader loads the next 2 minutes into a timing wheel.

async def load_window(shard, now):
    rows = await pg.fetch("""
        SELECT auction_id, closes_at, token FROM auction_closes
        WHERE minute_bucket IN ($1, $2) AND shard = $3 AND status = 'pending'
    """, minute(now), minute(now) + 1, shard)
    for r in rows:
        wheel.add(r.auction_id, at=r.closes_at, token=r.token)

async def on_fire(auction_id, token, fence):
    # Idempotent + fenced: only the current leader with the current token may close.
    n = await pg.execute("""
        UPDATE auctions SET status='closed', closed_at=now()
        WHERE auction_id=$1 AND status='active' AND close_token=$2 AND fence < $3
    """, auction_id, token, fence)
    if n == 0:
        return                              # stale timer, extended, or already closed
    await actor(auction_id).tell(CloseAuction(token))

The conditional UPDATE does three jobs at once: it is the idempotency guard (a second attempt matches zero rows), the staleness guard (close_token catches an extension that happened after scheduling), and the fencing guard (fence catches a paused old leader). One statement, three correctness properties — worth pointing out, because it is a compact example of making correctness structural rather than procedural.

Add a sweeper that finds auctions past their close time still marked active (a missed timer) and closes them, logging the miss. Defence in depth: the timing wheel is the fast path, the sweeper is the guarantee.

4.5 Broadcasting price to 200,000 watchers

Why it's hard. A hot auction's price changes ~50 times per second. Naively pushing each change to 200k watchers is 10M messages/sec from a single actor, which is impossible from one process and pointless anyway — no human perceives 50 updates per second.

Solution — coalesce at the source, then distribute through a relay tree.

// Hub: collapse a burst of price changes into one frame per interval.
type PriceHub struct {
    latest  map[int64]PriceUpdate      // auction_id -> most recent state only
    relays  []*Relay
}

func (h *PriceHub) OnChange(u PriceUpdate) { h.latest[u.AuctionID] = u }  // overwrite

func (h *PriceHub) Flush() {                                              // every 250ms
    if len(h.latest) == 0 { return }
    frame := encodeOnce(h.latest)          // serialise ONCE for all relays
    for _, r := range h.relays { r.Send(frame) }
    clear(h.latest)
}

Coalescing turns 50 updates/sec into 4, and the relay tree turns one 200k-socket write into ten 20k-socket writes across ten nodes. Same structure as live comments, and it is worth saying so — recognising that two superficially different problems share a solution is a strong signal.

Two auction-specific refinements. The final seconds get a faster cadence (100 ms instead of 250 ms) because that is when watchers are actually deciding — a dynamic flush interval based on time-to-close. And active bidders get a separate, unthrottled channel: there are dozens of them, not 200,000, so they can afford per-event delivery and they are the ones for whom latency actually matters.

4.6 Money: escrow, non-payment, and fraud

Why it's hard. Winning an auction creates an obligation, not a payment. The winner may not pay. The seller may not ship. Both may be fraudulent — shill bidding (the seller bidding up their own item through a second account) is the classic auction fraud and it is hard to detect from a single auction in isolation.

Solution — separate winning from paying, hold funds in escrow, and detect fraud on the graph rather than the transaction.

auction closed → winner obligated (invoice issued, payment window starts)
               → payment captured → funds held in escrow
               → item shipped + tracking → delivery confirmed (or window elapses)
               → funds released to seller, minus fees
               ↘ non-payment after N days → second-chance offer to the underbidder,
                                            strike against the winner's account
               ↘ dispute → escrow held, evidence-based resolution

The escrow hold is what makes the marketplace work: the seller ships because they can see funds are secured, and the buyer pays because funds are not released until delivery. Model it as a saga with an explicit state machine and compensations, exactly as in the payment system, using integer cents and a double-entry ledger.

For shill bidding, single-auction signals are weak; the detection lives in the graph:

• Accounts that bid frequently on one seller's items and rarely win.
• Bidders who consistently push the price to just below the eventual winner's maximum.
• Shared device fingerprints, payment instruments, IP subnets, or shipping addresses between bidder and seller.
• Accounts created shortly before, and active only during, a seller's auctions.

Run this as a graph-clustering job over the bid stream rather than a per-bid check. A per-bid check cannot see the pattern; that framing — "this is a graph problem, not a transaction problem" — is the useful thing to say.

4.7 Browsing 100 million live auctions

Why it's hard. The bidding core is a set of independent actors, which is perfect for writes and useless for "show me ending-soon cameras under $200 sorted by price." Search needs a global, queryable, continuously-changing view of prices that update thousands of times per second.

Solution — CQRS: the actors own writes, a projection owns reads, and the projection is allowed to lag.

# The actor's event stream feeds a search projection asynchronously.
async def project(event: BidPlaced):
    # Debounce: a hot auction's price changes 50×/sec, but the search index does
    # not need to see every one. Coalesce to at most one update per second.
    await debouncer.schedule(event.auction_id, delay=1.0, fn=reindex)

async def reindex(auction_id):
    a = await snapshot_store.get(auction_id)
    await es.update(index="auctions", id=auction_id, doc={
        "current_price": a.current_price, "bid_count": a.bid_count,
        "closes_at": a.closes_at, "category": a.category, "title": a.title,
    })

Two points. Debouncing is essential — indexing every price change on hot auctions would overwhelm Elasticsearch with updates for a handful of listings while the other 100M sit idle. And the projection is allowed to be one second stale, which is invisible in a search result list but would be unacceptable on the bid page — so the bid page reads from the actor directly, not from the projection.

That split — "search reads the projection, the auction page reads the actor" — is the practical face of CQRS, and stating it concretely is much better than naming the acronym.

5. What breaks first

Event	First failure	Mitigation
Hot auction final seconds	Actor mailbox depth, event-log append rate	Batch persistence; soft close spreads the spike
Actor node loss	That auction pauses ~200ms	Rehydrate from snapshot + events; clients retry; fencing prevents split-brain
200k watchers on one listing	Broadcast fan-out	Coalesce to 4 frames/sec; relay tree; separate channel for active bidders
Many auctions closing at :00	Timer thundering herd	Same deterministic jitter as the job scheduler; timing wheel handles the rest
Timer miss	An auction never closes	Sweeper finds past-due active auctions and closes them idempotently
Payment provider outage	Winners cannot pay within the window	Extend payment windows automatically; never strike an account for a provider outage
Search projection lag	Stale prices in listings	Bid page reads the actor directly; show "as of" on search results

6. Cheat sheet

• Actor per auction. The mailbox is the lock: linearizable bids, no database contention, and the tricky proxy logic becomes a pure testable function.
• Proxy bidding is second-price: winner pays the runner-up's max plus one increment; ties go to the earlier bid; the hidden max never leaves the actor.
• Soft close inside the actor removes the sniping race and flattens the load spike. Fence the close timer with a token.
• Close exactly once: durable schedule + timing wheel + a single conditional UPDATE that guards idempotency, staleness, and fencing at once. Plus a sweeper.
• Broadcast: coalesce to ~4 frames/sec, relay tree for 200k watchers, unthrottled channel for the few active bidders.
• Money: winning ≠ paying. Escrow with a saga; shill detection is a graph problem, not a per-bid check.
• Reads: CQRS — actors for the bid page, a debounced Elasticsearch projection for browse and search.
• The one-liner: "Serialise every auction through its own single-threaded actor so ordering and the proxy-bid rules are trivially correct, then push all the scale problems — watching, browsing, notifications — outside that serialisation point."