Robinhood / Stock Brokerage

Sharpened prompt. Design a retail brokerage supporting 20M users, 5M orders/day concentrated in the first minutes of market open, real-time quotes streamed to 2M concurrent clients, a matching engine with strict price-time priority at microsecond latency, buying power that is correct under T+1 settlement, and an audit trail that satisfies a regulator asking about one order from three years ago.

Two things make this different from a generic transactional system: the matching engine is a hard-real-time, single-threaded artifact, and regulation is a functional requirement, not a compliance checkbox bolted on afterwards.

1. Problem framing

Functional requirements

• Place, modify, and cancel orders: market, limit, stop, stop-limit.
• Route orders to venues (or match internally); report fills.
• Real-time quotes and order-book depth to clients.
• Track positions, cash, buying power, and P&L.
• Corporate actions: splits, dividends, symbol changes.

Non-functional requirements

Property	Target	Consequence
Order acknowledgement	P99 < 50ms (retail); matching itself in microseconds	In-memory book, no database on the hot path
Correctness	No lost, duplicated, or mis-sequenced orders	Idempotency + a sequenced event log
Market data	2M concurrent streams, sub-second	Fan-out tree with conflation
Availability at open	100% during 09:30–09:31 ET	The peak is the requirement; averages are irrelevant
Audit	Every event reconstructable for 7 years	Immutable, sequenced, timestamped event log

Back-of-the-envelope

Users:        20M, ~2M concurrent at market open
Orders:       5M/day, but ~35% arrive in the first 5 minutes
              = 1.75M orders / 300 s ≈ 6,000 orders/sec peak (vs ~90/sec average)
Market data:  ~10k symbols × ~50 updates/sec = 500k msg/sec from the feed
              × 2M subscribers if broadcast naively = 10^12 msg/sec. Impossible.
              -> conflation + per-client subscription sets are mandatory
Ledger:       5M orders × ~6 entries = 30M ledger rows/day
Matching:     a single-threaded engine handles millions of orders/sec in memory.
              6,000/sec is not remotely the bottleneck — the bottleneck is everything
              around it.

Point out the 6,000/sec versus millions/sec gap: the matching engine is not the scaling problem, which is counterintuitive and immediately reframes the discussion toward market data, buying power, and the event log.

2. High-level architecture

3. Component inventory

Component	Concrete choice	Why this one
Matching engine	Single-threaded C++/Rust, LMAX Disruptor-style ring buffer	Determinism and cache locality beat parallelism; locks are the enemy
Order book	Price-indexed array or B-tree of price levels, each an intrusive FIFO list	O(1) at the best price, which is where nearly all activity is
Sequencer	A single sequencing service per symbol group	Total order is what makes replay and audit possible
Event log	Aeron / Chronicle Queue locally, Kafka for downstream	Ultra-low-latency local persistence, durable fan-out downstream
Ledger	Double-entry, integer minor units	Same discipline as the payment system
Market data	UDP multicast internally, WebSocket to clients, conflated	Broadcast semantics are natural for market data
Risk	In-memory positions cache with a synchronous pre-trade check	Must be on the critical path — post-trade risk is not risk

4. The toughest parts

4.1 The matching engine: why single-threaded is faster

Why it's hard. Matching requires strict price-time priority: among orders at the same price, the earliest submitted must fill first. That is inherently a sequential guarantee. A multi-threaded engine needs locks around the book, and lock contention plus cache-line bouncing makes it slower than one thread, while introducing the possibility of subtly violating priority — which is a regulatory violation, not a bug.

Solution — one thread per symbol group, all state in memory, no I/O in the hot loop.

// Price levels in an array indexed by price ticks: O(1) access to the best bid/ask.
// Each level is an intrusive doubly-linked FIFO — insertion at the tail preserves
// time priority for free, and cancellation is O(1) via a pointer from an order map.
struct PriceLevel { Order* head; Order* tail; int64_t total_qty; };

class OrderBook {
  std::vector<PriceLevel> bids_, asks_;      // indexed by tick
  int32_t best_bid_, best_ask_;
  absl::flat_hash_map<OrderId, Order*> by_id_;   // O(1) cancel

public:
  void match(Order* incoming, std::vector<Fill>& out) {
    if (incoming->side == BUY) {
      while (incoming->qty > 0 && best_ask_ <= incoming->limit_tick) {
        PriceLevel& lvl = asks_[best_ask_];
        while (incoming->qty > 0 && lvl.head) {
          Order* resting = lvl.head;                    // FIFO: time priority
          int64_t q = std::min(incoming->qty, resting->qty);
          out.push_back({incoming->id, resting->id, best_ask_, q, now_ns()});
          incoming->qty -= q; resting->qty -= q; lvl.total_qty -= q;
          if (resting->qty == 0) { pop_front(lvl); by_id_.erase(resting->id); }
        }
        if (!lvl.head) advance_best_ask();
      }
    } // ... symmetric for SELL
    if (incoming->qty > 0 && incoming->type == LIMIT) rest(incoming);
  }
};

The mechanical sympathy points worth naming, because they are what the question is really probing:

• No allocation in the hot path. Orders come from a pre-allocated pool; a malloc in the matching loop is a latency spike waiting to happen.
• No I/O in the hot path. The engine writes to a ring buffer; a separate thread persists and publishes. The LMAX Disruptor pattern exists precisely for this handoff.
• No locks. Single-threaded means no synchronisation cost and no possibility of a priority violation.
• Determinism enables replay. Given the same input sequence, the engine produces exactly the same output. That is what makes the audit trail (4.6) and disaster recovery work: you rebuild state by replaying the log, not by restoring a backup.

Scale horizontally by symbol: partition symbols across engines. Cross-symbol atomicity is not required for equities (unlike options spreads), so this partitioning is clean.

4.2 Market data: 10^12 messages/sec is not a typo

Why it's hard. Naively, 500k market-data updates/sec × 2M subscribed clients is a trillion messages per second. Even after restricting each client to their watchlist, a popular symbol during volatility updates hundreds of times per second, and no phone needs — or can render — that.

Solution — conflate at the source, subscribe narrowly, and fan out through a tree.

// Conflation: for market data, only the LATEST value matters. Overwrite, never queue.
type Conflator struct {
    latest map[uint32]Quote        // symbol_id -> most recent quote
    dirty  map[uint32]struct{}
}

func (c *Conflator) OnQuote(q Quote) {
    c.latest[q.SymbolID] = q       // 300 updates/sec collapse into 1 per flush
    c.dirty[q.SymbolID] = struct{}{}
}

func (c *Conflator) Flush() {      // every 100-250ms, adaptive to client type
    for id := range c.dirty {
        frame := encode(c.latest[id])              // encode ONCE per symbol
        for _, relay := range c.subscribers[id] {  // subscription index, not broadcast
            relay.Send(frame)
        }
    }
    clear(c.dirty)
}

Three levers, in order of impact. Conflation collapses hundreds of updates per second into a few — and unlike chat messages, dropping intermediate quotes loses nothing, because only the current price matters. Per-symbol subscription means a client watching 20 symbols receives 20 streams, not 10,000. A relay tree (hub → regional relays → clients) means no single process writes 2M sockets, the same structure as live comments and auctions.

Two brokerage-specific details. Tier the cadence: a client with the app open and the symbol on screen gets 100 ms updates; a background client gets 1 s or push notifications only. And be careful what you conflate — for a chart you can drop intermediate ticks, but for the order book depth and for trade prints (time and sales) you cannot, because the sequence itself is the information. Conflate quotes, stream trades.

4.3 Buying power under T+1 settlement

Why it's hard. A user sells $10,000 of stock and immediately wants to buy something else. The cash from that sale does not actually settle for one business day (T+1). Meanwhile there are regulatory constraints: free-riding (buying with unsettled funds and selling before settlement) is prohibited, pattern day trader rules restrict accounts under $25,000 to three day trades in five business days, and margin adds a whole second accounting system. Getting buying power wrong means either blocking legitimate trades (users leave) or permitting trades that create regulatory violations (fines).

Solution — model cash in explicit buckets and compute buying power as a pure function over them, evaluated synchronously before every order.

@dataclass(frozen=True)
class CashState:
    settled_cash:      int          # minor units, available without restriction
    unsettled_credits: list[tuple[date, int]]   # (settlement_date, amount)
    pending_debits:    int          # orders placed but not yet filled
    margin_available:  int
    day_trades_5d:     int
    equity:            int

def buying_power(cs: CashState, order: Order, today: date) -> Decision:
    # 1. Cash available now = settled + credits whose settlement date has arrived.
    available = cs.settled_cash + sum(a for d, a in cs.unsettled_credits if d <= today)
    available -= cs.pending_debits            # reserve for in-flight orders

    # 2. Unsettled funds MAY be used to buy, but selling that position before
    #    settlement is a good-faith violation. Tag the position, don't block the buy.
    unsettled = sum(a for d, a in cs.unsettled_credits if d > today)

    # 3. PDT: below $25k equity, cap day trades at 3 per rolling 5 business days.
    if order.would_be_day_trade and cs.equity < 25_000_00 and cs.day_trades_5d >= 3:
        return Decision.reject("PDT_LIMIT", detail="3 day trades in 5 business days")

    need = order.notional_estimate()
    if need <= available:
        return Decision.accept(uses_unsettled=0)
    if need <= available + unsettled:
        return Decision.accept(uses_unsettled=need - available)   # tag for GFV tracking
    if need <= available + unsettled + cs.margin_available:
        return Decision.accept(uses_margin=True)
    return Decision.reject("INSUFFICIENT_BUYING_POWER",
                           shortfall=need - available - unsettled - cs.margin_available)

Three things to emphasise. pending_debits is essential — without reserving against unfilled orders, a user places ten $1,000 orders against $1,000 of cash and all ten pass the check. This is the same check-then-act race as everywhere else in this playbook, and the fix is the same: reserve atomically at order acceptance, release on cancel or fill.

The check must be synchronous and pre-trade. Post-trade risk is not risk; once the order is at the exchange it can fill in microseconds and you own the position.

And buying power is a pure function of an explicit state, which makes it unit-testable against regulatory scenarios — which is exactly what you will be asked to demonstrate during an audit.

4.4 Market open: 35% of the day in five minutes

Why it's hard. Retail order flow is extraordinarily peaked. Overnight, users queue orders that all release at 09:30:00. Notifications about pre-market moves drive a login surge at the same moment. Every component sees 60× its average load simultaneously — and this is the one minute of the day when failure is most visible and most costly.

Solution — design for the peak explicitly, and shed the right things.

# 1. Pre-open: warm everything. Nothing may be cold at 09:30:00.
async def pre_open_warmup():                      # runs 09:00-09:29
    await matching_engines.load_all_books()       # previous close + pre-market
    await positions_cache.warm(active_users())    # buying power needs positions
    await md_relays.scale_to(peak_capacity)       # provision, don't autoscale
    await connection_pools.prefill()

# 2. Queued market-on-open orders are released with deterministic jitter
#    across the first 2 seconds, not all at 09:30:00.000.
def release_at(order):
    base = market_open
    if order.type == MARKET_ON_OPEN:
        return base                                # these MUST go at the open
    return base + timedelta(milliseconds=hash(order.id) % 2000)

# 3. Load shedding ladder — shed reads long before you shed writes.
SHED_ORDER = [
    "historical_charts",      # nice to have
    "news_feed",
    "portfolio_analytics",
    "quote_update_frequency", # degrade 100ms -> 1s
    # NEVER shed: order placement, cancellation, fills, positions.
]

The load-shedding ladder is the most important part and the part candidates skip. State the principle plainly: degrade the read path to protect the write path. A user who cannot see a chart is annoyed; a user who cannot cancel an order during a crash is harmed, and that is what makes the news.

Provision, do not autoscale, for a known event. Market open happens at the same time every day. Reactive autoscaling is minutes too slow; scheduled scale-up at 09:00 is the correct answer, and saying so shows you distinguish predictable peaks from unpredictable ones.

4.5 Orders that must not duplicate, ever

Why it's hard. A user taps "Buy" and the network hangs. They tap again. If both requests become orders, they own twice the intended position — potentially tens of thousands of dollars of unwanted exposure in a volatile market. Unlike a duplicate social post, this cannot be quietly deleted; unwinding it means a real trade at a real price with a real loss.

Solution — client-generated idempotency keys enforced at the storage layer, plus a sequencer that assigns the authoritative order.

async def place_order(req: OrderRequest, client_order_id: str):
    # client_order_id is generated ON THE DEVICE before the first attempt, so
    # every retry of the same user intent carries the same key.
    try:
        async with pg.transaction():
            await pg.execute(
                "INSERT INTO orders (client_order_id, user_id, symbol, side, qty, "
                "limit_px, status) VALUES ($1,$2,$3,$4,$5,$6,'pending')",
                client_order_id, req.user_id, req.symbol, req.side, req.qty, req.limit_px)
            # Reserve buying power in the SAME transaction as order creation.
            await reserve_buying_power(req.user_id, req.notional)
    except UniqueViolation:
        existing = await pg.fetchrow(
            "SELECT * FROM orders WHERE client_order_id=$1", client_order_id)
        return OrderResponse.from_row(existing)          # replay, do not re-place

    seq = await sequencer.next(req.symbol)               # global total order
    await event_log.append(OrderAccepted(seq, client_order_id, req))
    await matching_engine.submit(seq, req)
    return OrderResponse(order_id=..., status="accepted")

Two properties beyond basic idempotency. Buying power reservation and order creation are one transaction — otherwise a crash between them either double-spends buying power or strands a reservation. And the sequencer assigns a monotonic number per symbol before the engine sees the order, which gives you a replayable total order: the engine is deterministic, so replaying the sequence reproduces every fill exactly. That is both your disaster recovery and your audit answer.

Cancellation has a symmetric race worth mentioning: a cancel can arrive after the order has already filled. The engine's response must be authoritative (CANCEL_REJECTED: already filled), and the client must display the fill rather than an optimistic "cancelled" — showing a cancellation that did not happen is how users end up with positions they believe they closed.

4.6 The audit trail a regulator will actually ask for

Why it's hard. Regulators (FINRA/SEC, and CAT reporting in the US) require you to reconstruct, years later, exactly what happened to a specific order: when it was received, every state change, every routing decision, every fill, with timestamps accurate to the millisecond or better and clock synchronisation you can prove. "We have logs" is not an answer; the requirement is a queryable, complete, tamper-evident record.

Solution — make the event log the system of record, not a byproduct.

Every state change is an immutable, sequenced, timestamped event:

  seq | ts_ns              | type            | order_id | payload
  ----+--------------------+-----------------+----------+---------------------------
  ... | 1738245600123456789| OrderReceived   | o_9f3a   | {symbol, side, qty, px, src}
  ... | 1738245600123501200| RiskChecked     | o_9f3a   | {bp_before, bp_after, rules}
  ... | 1738245600123512400| OrderRouted     | o_9f3a   | {venue: "NASDAQ", reason}
  ... | 1738245600124003100| PartialFill     | o_9f3a   | {qty: 50, px: 18734, venue}
  ... | 1738245600124119800| Filled          | o_9f3a   | {qty: 50, px: 18735}

Properties:
  - append-only, no updates, no deletes
  - globally sequenced per symbol; monotonic timestamps from a disciplined clock
  - hash-chained (each event includes the previous event's hash) -> tamper-evident
  - replaying the log reproduces every downstream state exactly

Three points that read as real-world experience. Clock discipline is a deliverable: PTP or GPS-disciplined NTP with continuous monitoring of offset, and the offset itself recorded, because a regulator may ask you to prove your timestamps. Hash-chaining makes the log tamper-evident at trivial cost, which turns "trust our database" into "here is the cryptographic evidence." And the log's retention is a hard 7 years in cheap object storage with an indexed hot tier for recent data, so the storage design is driven by a legal requirement rather than a performance one.

The strongest framing: "The event log isn't for debugging — it is the system of record. Positions, ledger, and the order book are all projections of it, which means audit, disaster recovery, and reconciliation are the same mechanism rather than three."

4.7 Corporate actions and the state that changes overnight

Why it's hard. A 4-for-1 split turns 25 shares at $600 into 100 at $150. A dividend credits cash. A symbol change rewrites every open order and position. A merger converts one security into another plus cash. These happen outside market hours, must apply atomically across positions, open orders, cost basis, and historical charts, and are a rich source of quiet, expensive bugs — a mis-applied split shows a user a 75% loss.

Solution — a nightly, transactional, idempotent corporate-actions pipeline with pre- and post-verification.

async def apply_split(symbol: str, ratio: Fraction, ex_date: date):
    action_id = f"split:{symbol}:{ex_date}"
    if await already_applied(action_id):          # idempotent: reruns are safe
        return

    async with pg.transaction():                  # one atomic unit for the symbol
        before = await checksum_positions(symbol) # invariant: total VALUE is preserved

        await pg.execute("""UPDATE positions
                            SET qty = qty * $1, cost_basis_per_share = cost_basis_per_share / $1
                            WHERE symbol = $2""", ratio, symbol)

        # Open orders must be adjusted too, or a $600 limit becomes unreachable.
        await pg.execute("""UPDATE orders
                            SET qty = qty * $1, limit_px = limit_px / $1
                            WHERE symbol = $2 AND status = 'open'""", ratio, symbol)

        # Fractional shares from the split: cash-in-lieu, booked through the ledger.
        await settle_fractional_residue(symbol, ratio)

        after = await checksum_positions(symbol)
        assert abs(after.total_value - before.total_value) < TOLERANCE, "split broke value"
        await mark_applied(action_id)

The value-preservation assertion is the detail to highlight: a split changes quantity and price but must not change total value, so asserting that invariant inside the transaction catches an entire class of arithmetic errors before anyone sees them. Corporate actions are exactly the kind of infrequent, high-consequence code path where an invariant check is worth far more than a test.

Also adjust historical price series for splits, or every chart shows a phantom crash on the ex-date — a support-ticket generator that is trivially avoidable.

5. What breaks first

Event	First failure	Mitigation
Market open	Everything at 60× average simultaneously	Scheduled pre-scaling, warm caches, deterministic release jitter, read-path shedding
Volatility event (a halt)	Market-data rate spikes, order rate spikes	Conflation absorbs MD; order path is protected by the shedding ladder
Matching engine node loss	That symbol group halts	Hot standby replaying the same sequenced input; deterministic replay makes takeover exact
Buying-power cache stale	Over-permitted orders	Reservation is transactional in Postgres, not cached; the cache is for display only
Exchange feed gap	Stale or wrong quotes shown	Sequence-number gap detection, request retransmit, mark data stale in the UI rather than showing a wrong price
Clock drift	Audit timestamps unprovable	PTP/GPS discipline with continuous offset monitoring and recording
Corporate action bug	Users see wrong positions overnight	Idempotent pipeline with a value-preservation invariant and a rollback path

6. Cheat sheet

• Matching: single-threaded per symbol group, in-memory price-level array + FIFO lists, no allocation or I/O in the hot loop, LMAX-style ring buffer handoff. Determinism enables replay.
• The counterintuitive number: 6,000 orders/sec peak against an engine capable of millions. The engine is not the bottleneck — market data and buying power are.
• Market data: conflate quotes (never trades or depth), per-symbol subscriptions, relay tree, tiered cadence by client state.
• Buying power: explicit cash buckets, pending-debit reservation in the same transaction as order creation, synchronous pre-trade, PDT and settlement rules as pure functions.
• Peak: provision on a schedule (open is predictable), release queued orders with jitter, shed reads to protect writes.
• Idempotency: device-generated client_order_id with a UNIQUE constraint; sequencer assigns the authoritative total order.
• Audit: the sequenced, hash-chained event log is the system of record; everything else is a projection. Clock discipline is a deliverable.
• The one-liner: "A deterministic single-threaded core wrapped in systems that must survive a 60× peak at a known time — and because it is deterministic and sequenced, audit, recovery, and reconciliation are all just replay."