Online Chess (Chess.com / Lichess)

Sharpened prompt. Design an online chess platform hosting 500k concurrent games including bullet chess where 100ms of added latency decides the outcome, validating every move server-side against hacked clients, running clocks that are fair to a player on mobile data, matching players by strength in under 10 seconds, detecting engine assistance, and streaming a world-championship game to a million spectators.

Chess is small-data and hard-real-time: the state is 32 bytes, the rules are fixed, and the entire difficulty is latency, fairness, and trust.

1. Problem framing

Functional requirements

• Play real-time games at time controls from 1+0 bullet to 30+20 classical.
• Server-authoritative move validation, including the awkward rules.
• Clocks with increment/delay; flag detection; draw offers, resignation, takebacks.
• Matchmaking by rating and time control; ratings updated after each game.
• Spectating, analysis, and game archives.

Non-functional requirements

Property	Target	Consequence
Move round trip	P95 < 100ms regionally	Binary protocol, persistent connection, in-memory game state
Validation	100% server-side	Never trust a client, ever
Clock fairness	Lag compensation bounded and symmetric	Clock policy is a product decision with a precise implementation
Concurrency	500k games, 1M+ connections	One lightweight actor per game
Integrity	Engine use detectable	Statistical, not deterministic — and it must be defensible

Back-of-the-envelope

Games:     500k concurrent; a bullet game averages ~40 moves in ~2 minutes
Moves:     500k games × ~0.5 moves/sec (bullet-weighted) ≈ 250k moves/sec peak
Payload:   a move is ~4 bytes binary (from, to, promo, flags). Full position: 32 bytes.
           250k × 4 B = 1 MB/sec of actual game data. The DATA is nothing.
Conns:     1M+ sockets (players + spectators)
State:     500k games × ~1 KB (position, history, clocks) = 500 MB. Fits in RAM easily.
Archive:   10M games/day × 400 B (PGN compressed) = 4 GB/day. Trivial.
Spectate:  1M viewers on one game × 1 move/5s = 200k msg/sec for ONE game -> tree.

State the contrast plainly: the data volume is negligible and the latency requirement is brutal. That inverts the usual optimisation instincts — you are not sharding for throughput, you are eliminating hops.

2. High-level architecture

One actor per game again — the same structure as auctions and Google Docs. Serialising all events for one entity through one mailbox makes ordering, clock arithmetic, and validation trivially correct.

3. Component inventory

Component	Concrete choice	Why this one
Transport	WebSocket with binary frames (or WebTransport over QUIC)	JSON over HTTP wastes both bytes and a round trip; QUIC avoids head-of-line blocking
Serialisation	A hand-rolled 4-byte move encoding, or FlatBuffers	At 250k moves/sec, parse cost matters; zero-copy is worth it
Game runtime	Actor per game (Elixir/OTP, Akka, or Durable Objects)	Cheap isolation, sequential per-game semantics
Rules engine	Bitboard-based legal-move generator in Rust or C	Microseconds per validation; correctness is non-negotiable
Move log	Append-only, one partition per game	Crash recovery by replay; also the analysis and anti-cheat input
Matchmaking	In-memory pools per (time control, rating band)	Must match in seconds; a database query is the wrong tool
Ratings	Glicko-2	Models uncertainty; better than Elo for infrequent players
Anti-cheat	Offline engine correlation + behavioural models	Cannot be real-time; must be statistically defensible

4. The toughest parts

4.1 Getting a move across the world in under 100 milliseconds

Why it's hard. In 1+0 bullet, players make moves in 200–400ms. A 150ms round trip consumes a large fraction of their thinking time, and the perceived latency — the gap between playing a move and seeing the opponent's reply — determines whether the game feels playable. HTTP request/response adds a connection setup, headers, and JSON parsing to what should be four bytes.

Solution — persistent binary connections, regional edge termination, and a compact wire format.

Move encoding — 4 bytes, no parsing, no allocation:

  bits 0-5    from square (0-63)
  bits 6-11   to square   (0-63)
  bits 12-14  promotion piece (0 = none, 1-4 = N/B/R/Q)
  bits 15     is_drop (for variants)
  bits 16-31  client sequence number

  Server response adds: 8 bytes of clock state (both clocks in ms) + 1 byte flags.
  Total frame: 13 bytes + WebSocket framing. Compare to ~200 bytes of JSON.

// Client: optimistic application. Render the move immediately, reconcile on ack.
function playMove(from, to, promo) {
  if (!localRules.isLegal(board, from, to, promo)) return reject();  // instant feedback

  const seq = ++clientSeq;
  board.apply(from, to, promo);          // optimistic: the board updates NOW
  pending.set(seq, {from, to, promo});
  socket.send(encodeMove(from, to, promo, seq));
}

socket.onmessage = (frame) => {
  const msg = decode(frame.data);
  if (msg.type === ACK) {
    pending.delete(msg.seq);
    clocks.sync(msg.whiteMs, msg.blackMs);        // server clocks are authoritative
  } else if (msg.type === REJECT) {
    board.rollbackTo(msg.position);               // client and server disagreed
    pending.clear();
  }
};

Three levers. Optimistic local application removes the round trip from the player's own move — they see it instantly, and the server's ack only corrects the rare disagreement. Edge termination puts the socket in the player's region; the game actor lives in the region closest to both players (or the midpoint for intercontinental games), which is a real placement decision worth mentioning. Binary framing cuts bytes and, more importantly, parse time at 250k moves/sec.

WebTransport over QUIC is the modern upgrade worth naming: no head-of-line blocking (a lost packet does not stall subsequent moves), 0-RTT reconnection, and connection migration when a phone switches from Wi-Fi to cellular — which is exactly the scenario that ruins a bullet game today.

4.2 Never trusting the client

Why it's hard. The client is fully under the player's control. A modified client can send illegal moves, claim the opponent flagged, replay old messages, or attempt to move on the opponent's turn. Chess rules are also deceptively intricate: castling through check, en passant, the fifty-move rule, threefold repetition, insufficient material, and stalemate versus checkmate. A validator that gets any of them wrong produces games that are decided incorrectly.

Solution — the server maintains the authoritative position and generates the legal move set; a move is legal if and only if it is in that set.

// Bitboards: one u64 per piece type per colour. Move generation is bit manipulation,
// which makes full legal-move generation take microseconds.
pub struct Position {
    pieces:      [[u64; 6]; 2],   // [colour][piece_type]
    occupied:    [u64; 2],
    side_to_move: Colour,
    castling:    u8,              // 4 bits of rights
    ep_square:   Option<u8>,      // en passant target
    halfmove:    u16,             // for the fifty-move rule
    hash:        u64,             // Zobrist, for threefold repetition
}

impl Position {
    pub fn legal_moves(&self) -> MoveList {
        let mut list = self.pseudo_legal_moves();     // ignores self-check
        // Filter: a move is illegal if it leaves your own king attacked.
        list.retain(|m| {
            let next = self.make(*m);
            !next.is_attacked(next.king_square(self.side_to_move), !self.side_to_move)
        });
        list
    }

    pub fn validate(&self, m: Move) -> Result<(), IllegalMove> {
        if self.legal_moves().contains(&m) { Ok(()) } else { Err(IllegalMove) }
    }
}

Generating the full legal move set and checking membership is both the simplest correct approach and fast enough — a good bitboard generator produces all legal moves in a few microseconds, so 250k validations/sec is a fraction of one core.

Draw and termination conditions must also be server-side, and they are where naive implementations break:

fn terminal_state(pos: &Position, history: &[u64]) -> Option<Outcome> {
    if pos.legal_moves().is_empty() {
        return Some(if pos.in_check() { Outcome::Checkmate } else { Outcome::Stalemate });
    }
    if pos.halfmove >= 100 { return Some(Outcome::FiftyMove); }        // 50 full moves
    // Threefold: count Zobrist hashes. Cheap because the hash is maintained
    // incrementally as moves are made.
    if history.iter().filter(|&&h| h == pos.hash).count() >= 3 {
        return Some(Outcome::Repetition);
    }
    if insufficient_material(pos) { return Some(Outcome::InsufficientMaterial); }
    None
}

Zobrist hashing is the detail worth showing: an incrementally-maintained 64-bit position hash makes threefold-repetition detection a counter lookup rather than a board comparison.

Because the actor holds the position in memory, validation involves no I/O — the entire move handling path is CPU-only, which is what makes the latency budget achievable.

4.3 Clocks: fairness when the network is not

Why it's hard. In bullet chess the clock decides games. A player on a mobile connection with 200ms of latency loses 400ms per move pair to the network — over a 40-move game, more than 15 seconds off a 60-second clock. Charging them for network time is unfair; not charging them lets a cheater fake latency to gain time. Meanwhile clients cannot be trusted to report their own elapsed time, and client and server clocks drift.

Solution — the server owns the clock, deducts server-measured elapsed time, and grants bounded lag compensation.

class GameClock:
    def __init__(self, initial_ms, increment_ms):
        self.remaining = {WHITE: initial_ms, BLACK: initial_ms}
        self.increment = increment_ms
        self.turn_started_at = None            # server monotonic clock
        self.lag_credit = {WHITE: 0, BLACK: 0}

    def on_move_received(self, colour, rtt_estimate_ms):
        now = monotonic_ms()
        elapsed = now - self.turn_started_at

        # Lag compensation: refund up to half the measured round trip, capped.
        # Half, because only the inbound leg happened during THIS player's turn.
        compensation = min(rtt_estimate_ms / 2, MAX_LAG_CREDIT_MS)   # cap = 100ms
        charged = max(0, elapsed - compensation)

        self.remaining[colour] -= charged
        if self.remaining[colour] <= 0:
            return Flagged(colour)

        self.remaining[colour] += self.increment                     # Fischer increment
        self.turn_started_at = now
        return Ok(self.remaining)

The design points to defend:

• The server measures elapsed time, using a monotonic clock (never wall clock — NTP steps would corrupt game state). The client's opinion is irrelevant.
• Compensation is capped, typically at ~100ms per move. Uncapped compensation is exploitable: a cheater artificially inflates their measured RTT to buy thinking time. The cap bounds the exploit to something smaller than the noise.
• Half the RTT, not all of it. Only the inbound leg elapsed during the player's turn; refunding the full round trip over-compensates.
• RTT is measured continuously with lightweight pings, using a smoothed estimate (an EWMA, or a low percentile) rather than the instantaneous value, so one spike does not grant a windfall.

Flag detection has a subtle race worth raising: if White's clock hits zero while Black is disconnected, who wins? The rule (and the implementation) must handle "flagged but the opponent had insufficient material to mate," which is a draw, not a loss. The actor holds a timer that fires at the exact flag moment and evaluates that condition — another example of the actor making a race-prone situation deterministic.

4.4 Matching 500,000 players in under ten seconds

Why it's hard. Players want an opponent of similar strength, at their chosen time control, quickly. Those goals conflict: a tight rating window means a long wait for a 2400-rated player at 3am, and a loose one produces mismatched, unenjoyable games. There is also a pool-fragmentation problem — dozens of time controls × rating bands × rated/casual × variants splits a large population into many small, thin pools.

Solution — in-memory pools keyed by time control, with a rating window that widens over time.

class Matchmaker:
    def __init__(self):
        # One sorted structure per time control; players ordered by rating.
        self.pools = defaultdict(SortedList)     # (time_control, rated) -> [seeks]

    async def seek(self, player, tc, rated):
        s = Seek(player, tc, rated, queued_at=monotonic())
        pool = self.pools[(tc, rated)]

        while True:
            window = self.window_for(monotonic() - s.queued_at, player.rd)
            lo, hi = player.rating - window, player.rating + window
            for cand in pool.irange_key(lo, hi):
                if self.acceptable(s, cand):     # not a recent opponent, not blocked
                    pool.remove(cand)
                    return await create_game(s, cand)
            pool.add(s)
            await asyncio.sleep(0.5)             # re-evaluate with a wider window

    def window_for(self, waited_s, rating_deviation) -> int:
        # Start tight, widen steadily. Uncertain (high-RD) players start wider,
        # because their true strength is not yet known anyway.
        base = 50 + rating_deviation
        return min(int(base + 40 * waited_s), 600)

Two refinements worth mentioning. Merge thin pools — if the 5+3 pool is nearly empty, allow matching into 5+0 and 3+2 after a delay, since players care far more about playing than about an exact increment. And avoid immediate rematches with the same opponent unless both consent, which prevents a two-player pool at odd hours from becoming a single endless series.

Ratings use Glicko-2 rather than Elo because it models a rating deviation — the system's uncertainty about a player. A returning player after a year has high RD, so their rating moves quickly to find its level, and matchmaking can widen their window without unfairness. Explaining that RD serves both rating and matchmaking is a nice connection.

4.5 Detecting engine assistance

Why it's hard. A cheater consults a chess engine for some or all moves. There is no technical signal available — the moves arrive through a normal client, from a normal IP, at plausible times. Detection must be statistical, and the consequences of a false positive are severe (banning a genuinely strong player is a serious harm and a public-relations disaster). It must also be adversarial-robust: cheaters deliberately play weak moves sometimes, and only consult the engine in critical positions.

Solution — offline analysis combining move-quality correlation with behavioural signals, requiring multiple independent indicators before action.

async def analyse_game(game) -> CheatSignals:
    engine = await stockfish_pool.acquire(depth=20)
    matches, cp_losses, times = 0, [], []

    for i, (pos, played, spent_ms) in enumerate(game.positions()):
        best = await engine.best_move(pos)
        matches += (played == best)
        cp_losses.append(await engine.centipawn_loss(pos, played))
        times.append(spent_ms)

    return CheatSignals(
        # 1. Top-engine-move match rate, weighted by position difficulty. Matching
        #    in forced positions means nothing; matching in quiet ones means a lot.
        weighted_match_rate = weighted_match(matches, game.positions()),

        # 2. Average centipawn loss vs the player's rating expectation.
        #    A 1500 playing at 1900 accuracy for 40 moves is the core signal.
        accuracy_vs_rating = mean(cp_losses) / expected_cpl(game.player_rating),

        # 3. Time-usage anomalies: humans think longer in complex positions.
        #    Uniform timing, or long thinks on trivial moves, is machine-like.
        time_correlation = corr(times, [complexity(p) for p, _, _ in game.positions()]),

        # 4. The strongest signal: finding "only moves" — the single non-losing
        #    reply in a sharp position — repeatedly and quickly.
        only_move_hit_rate = only_moves_found(game),

        # 5. Cross-game consistency: real strength varies; engine use is oddly stable.
        rating_volatility = player_recent_volatility(game.player_id),
    )

The methodology points that matter as much as the signals:

• Weight by position difficulty. In a forced recapture every player plays the engine move; match rate is only informative in positions with several reasonable options.
• Require multiple independent signals over multiple games before any action. One suspicious game is noise; a consistent pattern across twenty is evidence.
• Human review for anything consequential. An automated ban on a titled player will occasionally be wrong and always be public.
• Do not reveal the thresholds. Publishing exactly what triggers detection tells cheaters precisely how to stay under it — the same reasoning as shadow-banning in Tinder.

For high-stakes events, add out-of-band measures: proctoring, a delayed broadcast (so a spectating accomplice cannot relay engine moves in time), and video verification. Recognising that some integrity problems are not solvable in software is itself a good answer.

4.6 Disconnections, abandonment, and the game that must end somehow

Why it's hard. A player's connection drops mid-game. Are they cheating (stalling for time), did their train go into a tunnel, or did they simply quit? The clock keeps running, which is correct, but a player who reconnects in five seconds should not have lost by abandonment. Meanwhile the opponent should not have to wait ten minutes for a player who has clearly gone.

Solution — separate connection state from game state, with graduated timeouts.

class GameActor:
    async def on_disconnect(self, colour):
        self.connected[colour] = False
        # The CLOCK KEEPS RUNNING. Disconnection is not a pause — otherwise
        # disconnecting becomes a free timeout.
        self.notify_opponent(OpponentDisconnected(colour))

        # Offer the opponent an early claim, but do not force it: many players
        # prefer to wait, and an automatic win feels unearned.
        await asyncio.sleep(min(15, self.clock.remaining[colour] / 1000))
        if not self.connected[colour]:
            self.notify_opponent(ClaimVictoryAvailable(after_s=self.claim_delay()))

    async def on_reconnect(self, colour, last_known_move):
        self.connected[colour] = True
        # Resume from the client's stated position: send everything since.
        await self.send_delta(colour, since=last_known_move)
        await self.send_clocks()          # authoritative clocks, always

    def claim_delay(self) -> int:
        # Proportional to the time control: a bullet game cannot wait 60 seconds.
        return clamp(self.initial_ms / 1000 * 0.2, 5, 60)

Three principles. The clock never pauses on disconnect, or disconnecting becomes a strategic resource. The opponent claims rather than the system awarding, which handles the case where a player is happy to wait. And reconnection is a delta — the client says what it last saw, the server sends the rest plus authoritative clocks — which is the same resumable-cursor pattern as Dropbox sync and WhatsApp inboxes.

For actor failover: the game actor's state is rebuildable from the move log, so a node loss means a ~200ms rehydration and both clients reconnect. Clocks are the subtle part — persist the clock state on every move, and on recovery charge the elapsed wall time since the last recorded move to whoever was on turn, which is exactly what would have happened had nothing failed.

4.7 A million spectators on one game

Why it's hard. A world championship game attracts an enormous simultaneous audience. The game actor cannot write a million sockets. But spectators also want more than the moves: evaluation bars, top engine lines, and commentary, all of which multiply the payload.

Solution — the same relay tree as everywhere else, plus a separate analysis stream and a deliberate broadcast delay.

Game actor  --(1 move, ~13 bytes)-->  Broadcast hub
                                          |
                        +-----------------+-----------------+
                        v                 v                 v
                  Regional relay    Regional relay    Regional relay
                        |                 |                 |
                  ~50k sockets each, encode the frame ONCE per relay

Separate, lower-priority stream:
  Analysis service (Stockfish) --> eval + top 3 lines --> relays --> spectators
  Sent at most once per move, and only to spectators who enabled it.

Two chess-specific details worth adding. A deliberate 15–30 second broadcast delay for high-stakes games prevents spectators from relaying engine analysis back to a player — an integrity measure that is invisible to the audience and impossible to achieve any other way.

And spectators must never touch the game actor. They subscribe to the relay tier only, so a million viewers add exactly zero load to the two players' latency path. This is the same read/write separation as the Ticketmaster seat map and the Google Docs viewer tier — protecting the latency-critical write path by pushing all read scale into a separate tier is the recurring pattern, and naming it as a pattern is worth more than solving it three times.

5. What breaks first

Event	First failure	Mitigation
Peak evening traffic	Connection tier, not game logic	Actors are cheap (~1 KB); scale connection nodes horizontally
Popular streamer's game	Spectator fan-out	Relay tree; spectators never reach the game actor
Game node loss	~200ms pause for those games	Rehydrate from the move log; charge elapsed time correctly on recovery
Player on poor mobile network	Unfair clock losses	Capped lag compensation; smoothed RTT; QUIC connection migration
Titled-player tournament	Anti-cheat analysis backlog	Prioritise by stakes; broadcast delay; offline analysis is not on the critical path
Matchmaking pool thin at 3am	Long waits for strong players	Widening window; merge adjacent time controls; bot opponents as an opt-in

6. Cheat sheet

• The inversion: tiny data, brutal latency. Optimise for hops, not throughput.
• Transport: binary WebSocket (or WebTransport/QUIC), ~13-byte frames, optimistic client application with server reconciliation.
• Validation: bitboard legal-move generation server-side; Zobrist hashes for threefold; every termination condition checked by the server.
• Clocks: server monotonic clock owns elapsed time; refund half the smoothed RTT, capped at ~100ms; never pause on disconnect.
• Matchmaking: in-memory pools per time control, window widens with wait time and rating deviation; Glicko-2 because RD serves both rating and matching.
• Anti-cheat: offline, statistical, difficulty-weighted engine correlation plus timing behaviour; multiple signals, human review, undisclosed thresholds.
• Spectators: relay tree, separate analysis stream, deliberate broadcast delay; they never touch the game actor.
• The one-liner: "An actor per game holding 1 KB of bitboards, reached over a binary socket in the players' own region — everything else, spectators and analysis and anti-cheat, is pushed off that path so the only thing between a move and its acknowledgement is the speed of light."