LeetCode (Online Judge System)

Sharpened prompt. Design a coding judge where 50,000 contestants submit simultaneously at contest start, arbitrary untrusted code runs without escaping or exfiltrating anything, the same submission produces the same verdict every time regardless of which worker runs it, a leaderboard updates within seconds, and plagiarism is detected before prizes are awarded.

Two requirements sit at the centre: running hostile code safely and measuring it deterministically on shared hardware. Both are systems problems, not application problems, and that is what makes this question distinctive.

1. Problem framing

Functional requirements

• Submit a solution in one of ~20 languages; receive a verdict and diagnostics.
• Run against hidden test cases with per-problem time and memory limits.
• Contests: real-time leaderboard, penalties, tie-breaks, plagiarism detection.
• Show runtime and memory percentile against other submissions.

Non-functional requirements

Property	Target	Consequence
Isolation	No escape, no network, no cross-submission leakage	microVM or gVisor, seccomp, cgroups — layered
Determinism	Same code, same verdict, every time	CPU pinning and instruction-count measurement, not wall clock
Contest burst	50k submissions in the first minute	Queue plus elastic workers; submission acceptance decoupled from judging
Feedback latency	P95 < 5s outside contests	Warm runtimes, parallel test execution
Fairness	Identical resources for every submission	Reserved cores, no oversubscription during contests

Back-of-the-envelope

Users:       10M registered, ~200k daily submissions = ~2.3/sec average
Contest:     50k contestants × ~8 submissions over 90 min = 400k submissions
             But the START is spiked: ~15k submissions in the first 60 seconds = 250/sec
Judge cost:  avg 20 test cases × 200ms = 4s of CPU per submission
             250/sec × 4s = 1,000 CPU-seconds/sec  -> ~1,000 dedicated cores at peak
             vs ~10 cores at steady state. A 100× elastic range.
Storage:     400k submissions × 4 KB source = 1.6 GB per contest. Trivial.
Test data:   3,000 problems × ~5 MB of test cases = 15 GB. Fits on every worker's disk.

The 100× elastic range is the number that drives the architecture: you cannot provision for the contest peak, so submission acceptance must be decoupled from judging.

2. High-level architecture

The submission API does almost nothing: store the source, enqueue, return an ID. All the cost is behind the queue, which is what lets a 250/sec spike be absorbed rather than rejected.

3. Component inventory

Component	Concrete choice	Why this one
Sandbox	Firecracker microVM (or gVisor) per submission	Hardware-level isolation with ~125ms boot; containers alone share a kernel
Syscall filter	seccomp-bpf allowlist	Defence in depth inside the VM; blocks whole classes of attack
Resource limits	cgroups v2 (cpu, memory, pids)	Hard caps that the kernel enforces, not the runtime
Queue	Kafka with priority topics, or SQS with separate queues	Contest traffic must never queue behind practice traffic
Measurement	perf_event_open instruction counts + pinned cores	Wall clock on shared hardware is not reproducible
Test data	Baked into the worker image or on local NVMe	Fetching test data per submission adds latency and load
Leaderboard	Redis sorted sets	Same structure as Strava
Plagiarism	Winnowing over token streams (MOSS algorithm)	Robust to renaming and reformatting

4. The toughest parts

4.1 Running code that is actively trying to hurt you

Why it's hard. Submitted code is arbitrary and, in a competitive setting, sometimes hostile. The attack surface includes fork bombs, filling the disk, reading other submissions' data, making network connections to exfiltrate test cases, exploiting kernel bugs to escape a container, consuming all CPU on the host, and simply crashing the worker. A plain Docker container is not sufficient — it shares the host kernel, so any kernel vulnerability is a full escape.

Solution — layered isolation, with a hardware boundary as the outermost layer.

Layer 1  microVM (Firecracker)   separate kernel, KVM boundary. A kernel exploit
                                 inside the guest does not reach the host.
Layer 2  seccomp-bpf allowlist   only ~40 syscalls permitted; no socket, no ptrace,
                                 no mount, no clone with new namespaces.
Layer 3  cgroups v2              hard caps: CPU quota, memory.max, pids.max.
Layer 4  read-only rootfs        + a small tmpfs for scratch, wiped after each run.
Layer 5  no network device       not "firewalled" — the VM has no NIC at all.
Layer 6  one-shot VM             destroyed after a single submission. No reuse, ever.

# Firecracker VM config: minimal, disposable, and networkless by construction.
vm_config = {
    "boot-source": {"kernel_image_path": "/opt/vmlinux",
                    "boot_args": "console=none reboot=k panic=1 pci=off"},
    "drives": [
        {"drive_id": "rootfs", "path_on_host": "/opt/rootfs.ext4",
         "is_root_device": True, "is_read_only": True},          # immutable base
        {"drive_id": "scratch", "path_on_host": scratch_img,
         "is_root_device": False, "is_read_only": False},        # wiped after
    ],
    "machine-config": {"vcpu_count": 1, "mem_size_mib": 512, "smt": False},
    "network-interfaces": [],        # <- no NIC. Exfiltration is impossible, not blocked.
}

/* seccomp allowlist inside the guest: default-deny, permit only what a
   computation needs. Anything else kills the process immediately. */
static int allowed[] = {
    SCN(read), SCN(write), SCN(exit), SCN(exit_group), SCN(brk),
    SCN(mmap), SCN(munmap), SCN(mprotect), SCN(rt_sigaction),
    SCN(fstat), SCN(lseek), SCN(futex), SCN(clock_gettime),
    /* deliberately absent: socket, connect, execve, ptrace, clone,
       mount, unshare, setns, open (paths are pre-opened by the harness) */
};

Three points to emphasise. "No network device" beats "firewall rules" — a VM with no NIC cannot exfiltrate anything even if every other layer fails, and eliminating a capability is always stronger than restricting it. One-shot VMs eliminate cross-submission leakage entirely: no cache, no filesystem residue, no shared state, at the cost of ~125ms boot time (which is why Firecracker rather than a full VM). And layers are independent — an escape needs to defeat the seccomp filter and the guest kernel and KVM, and only the last one is a genuine host compromise.

Also defend the compilation step, which candidates routinely forget: a C++ template metaprogram can consume unbounded compiler memory and time, and #include of an unexpected path is an information leak. Compile inside a sandbox too, with its own time and memory limits.

4.2 Fifty thousand people pressing submit at once

Why it's hard. A contest opens. Everyone reads the first problem and submits within a few minutes. Peak submission rate is 100× the steady state, and the judging work is CPU-bound and cannot be cached. Provisioning 1,000 cores permanently for a weekly event is enormously wasteful; autoscaling reactively is minutes too slow.

Solution — decouple acceptance from judging, pre-scale for a known event, and prioritise ruthlessly.

@app.post("/submissions")
async def submit(req, user):
    # Accept in ~20ms. Everything expensive happens behind the queue.
    if not await rate_limit.allow(user.id, contest_aware=True):
        raise TooManyRequests(retry_after=10)

    sid = uuid7()
    await s3.put(f"src/{sid}", req.source)          # source is small; store it directly
    await queue.send(
        topic="judge.contest" if req.contest_id else "judge.practice",
        key=str(req.problem_id),                     # co-locate same-problem work
        value={"sid": sid, "lang": req.lang, "problem": req.problem_id,
               "user": user.id, "submitted_at": now_ms()})
    return {"submission_id": sid, "status": "queued"}

Four mechanisms make this work:

• Separate queues, separate worker pools. Contest judging must never queue behind someone's practice submission. Physical isolation, exactly as in the notification system's priority lanes.
• Pre-scale on a schedule. Contests are announced in advance; scale the fleet up 20 minutes before the start rather than reacting to queue depth. Reactive autoscaling on a known event is the wrong tool.
• Key by problem ID. Consecutive submissions for the same problem land on the same workers, so test data stays in the page cache and compilation caches hit.
• Push status, do not poll. 50k contestants polling every second is 50k QPS of "am I done yet?" — a WebSocket or SSE push costs nothing by comparison.

Compilation caching is the highest-leverage optimisation: many submissions differ only slightly, and identical source (retried submissions, or the same code submitted to multiple problems) can skip compilation entirely via a content hash. Cache the compiled artifact keyed by hash(source + language + compiler_version).

Under extreme load, degrade rather than fail: run a quick subset of test cases first to return an early "compile error" or "wrong answer on sample" verdict within a second, and complete the full suite afterwards. Contestants get the fast negative feedback that matters most, and the expensive full run is deprioritised.

4.3 The same code must get the same verdict

Why it's hard. A submission that takes 990ms against a 1000ms limit passes on an idle worker and fails on a busy one. The differences come from CPU frequency scaling, hyper-threading contention from a neighbouring process, NUMA memory placement, cache pollution, and noisy-neighbour I/O. Non-determinism here is not a minor annoyance — it decides contest outcomes and generates a flood of appeals.

Solution — remove the sources of variance, and measure something more stable than wall time.

# Host preparation: isolate cores from the kernel scheduler entirely.
# Kernel cmdline:  isolcpus=8-31 nohz_full=8-31 rcu_nocbs=8-31
# Disable SMT so a sibling hyper-thread cannot steal execution resources.
echo off > /sys/devices/system/cpu/smt/control
# Pin frequency: no turbo, no scaling. Predictability over peak speed.
cpupower frequency-set -g performance -d 2.4GHz -u 2.4GHz

def run_measured(vm, test_case, limits) -> Measurement:
    # One submission owns one isolated physical core for its entire run.
    core = core_pool.acquire_exclusive()
    try:
        vm.pin_vcpu(core)
        with perf_counters(core, ["instructions", "cache-misses"]) as pc:
            t0 = time.monotonic_ns()
            result = vm.run(test_case, timeout_ms=limits.wall_ms * 3)
            wall_ns = time.monotonic_ns() - t0

        return Measurement(
            wall_ms=wall_ns / 1e6,
            # INSTRUCTION COUNT is the deterministic measure: it does not vary
            # with clock speed, cache state, or neighbours. Same code + same
            # input = same count, every time.
            instructions=pc["instructions"],
            # Normalise to a reference machine so limits are hardware-independent.
            normalised_ms=pc["instructions"] / REFERENCE_IPS * 1000,
            peak_rss_kb=vm.cgroup_peak_memory() // 1024)
    finally:
        core_pool.release(core)

The key insight to articulate: instruction count is deterministic where wall time is not. Two runs of the same program on the same input retire the same number of instructions regardless of frequency, cache state, or what else is on the machine. Using it as the primary limit (with wall time as a generous secondary safety net, at 3× the limit, to catch pathological I/O or spin behaviour) removes almost all the appeals.

The trade-off to name: instruction count does not perfectly capture memory-bound performance — a cache-hostile algorithm retires few instructions but runs slowly. So report both, and set problem limits against a calibrated reference implementation rather than against an absolute number. Calibrating each problem's limit as a multiple of a known-good solution's measurement is the practical answer.

Memory measurement must come from the cgroup's peak (memory.peak), not from the process's self-reported usage — a program can lie about its own RSS, and the kernel cannot.

4.4 Test cases: hidden, large, and hard to keep secret

Why it's hard. Test cases must stay hidden (otherwise solutions hardcode outputs), be available instantly on every worker (fetching them per submission adds latency and load), and sometimes be large (a 100 MB input for a performance problem). Contestants actively try to extract them by printing what they read or by timing side channels.

Solution — bake test data into worker images, stream it into the sandbox, and never let the program see the expected output.

def run_test(vm, problem, case_idx):
    # 1. Input is streamed in via a pipe; the program never gets a filesystem path,
    #    so it cannot enumerate other cases or read them out of order.
    stdin_pipe = open_local(f"/data/{problem.id}/{case_idx}.in")

    # 2. Output goes to a SIZE-CAPPED pipe. A program printing 10 GB is killed,
    #    which also prevents disk exhaustion attacks.
    out = vm.run(stdin=stdin_pipe, stdout_limit_bytes=64 * 1024 * 1024)

    # 3. The expected output NEVER enters the VM. Comparison happens outside it.
    expected = open_local(f"/data/{problem.id}/{case_idx}.out")
    return compare(out, expected, problem.checker)   # exact, float-tolerant, or special

Three points. Never place expected outputs inside the sandbox — comparison happens on the host side, so even a full sandbox escape yields inputs but not answers. Cap output size at the pipe, which simultaneously prevents disk-filling and stops a program from dumping the input back as a side channel. And use a special checker program for problems with multiple valid answers, itself run in a sandbox because it is also code.

Against hardcoding attacks (submitting a solution that recognises the test input and prints a memorised answer), the defences are randomised test-case generation per contest, hidden cases distinct from the samples, and — as a detection signal — flagging submissions whose runtime is implausibly low for the problem's complexity class.

4.5 The contest leaderboard

Why it's hard. 50,000 contestants, each with per-problem state, ranked by problems solved then by penalty time, updating within seconds, and read by everyone constantly. Recomputing a full ranking on every submission is 400k recomputations over a contest; querying a database for "my rank" at 50k QPS is not viable either.

Solution — an incrementally-updated Redis sorted set with a composite score, plus a frozen tail.

def contest_score(user_state) -> float:
    """One float that sorts correctly by (problems desc, penalty asc).
       Redis sorted sets rank by a single score, so encode both into it."""
    solved  = user_state.problems_solved                 # 0..N
    penalty = user_state.total_penalty_minutes           # 0..~10^5
    # Higher is better: solved dominates; penalty breaks ties (inverted).
    return solved * 1e7 - min(penalty, 9_999_999)

async def on_accepted(contest_id, user_id, problem_id, minutes, wrong_before):
    st = await state.get(contest_id, user_id)
    if problem_id in st.solved:
        return                                            # already solved: no change
    st.solved.add(problem_id)
    st.total_penalty_minutes += minutes + 20 * wrong_before   # ICPC-style penalty
    await state.put(contest_id, user_id, st)
    await redis.zadd(f"lb:{contest_id}", {user_id: contest_score(st)})

# Reads are O(log N) for a rank, O(log N + k) for a page.
rank = await redis.zrevrank(f"lb:{contest_id}", user_id)
page = await redis.zrevrange(f"lb:{contest_id}", 0, 49, withscores=True)

Encoding two ranking dimensions into one float is the trick that makes a sorted set sufficient; it works as long as the secondary dimension's range is bounded, which penalty minutes are.

Two contest-specific behaviours worth mentioning. Freeze the leaderboard in the final 30–60 minutes (show submissions but not verdicts) — it preserves suspense and, more practically, prevents contestants from using the leaderboard as an oracle about which problems are solvable. And cache the top-N page aggressively, since almost everyone looks at the top and their own neighbourhood; those are two small queries, not a full ranking read.

4.6 Plagiarism, before the prizes

Why it's hard. Solutions leak to Discord and Telegram within minutes of a contest starting. Copies are trivially disguised: rename variables, reorder independent statements, change formatting, add dead code. Exact hashing catches nothing. And false accusations are serious — competitive programmers converge on similar solutions for a reason, because there is often one natural approach.

Solution — winnowing over a normalised token stream (the MOSS algorithm), then human review.

def fingerprint(source: str, lang: str, k: int = 12, w: int = 8) -> set[tuple[int, int]]:
    # 1. Normalise away everything cosmetic: identifiers, literals, whitespace,
    #    comments. What remains is the program's STRUCTURE.
    tokens = [canonical(t) for t in tokenize(source, lang)
              if t.kind not in (COMMENT, WHITESPACE)]
    # 'int n = 5;' and 'int count = 42;'  ->  TYPE IDENT ASSIGN NUM SEMI  (identical)

    # 2. Rolling hashes over k-grams of tokens.
    hashes = [(rolling_hash(tokens[i:i+k]), i) for i in range(len(tokens) - k + 1)]

    # 3. Winnowing: in each window of w hashes, keep the MINIMUM. This guarantees
    #    that any shared substring longer than (w + k - 1) tokens produces at least
    #    one shared fingerprint, while storing only ~1/w of the hashes.
    return {min(hashes[i:i+w]) for i in range(len(hashes) - w + 1)}

def similarity(a: set, b: set) -> float:
    fa, fb = {h for h, _ in a}, {h for h, _ in b}
    return len(fa & fb) / max(1, min(len(fa), len(fb)))

The winnowing guarantee is the part worth explaining: it is not a heuristic sample — the minimum-in-window rule provably retains at least one fingerprint from any sufficiently long shared passage, so you get bounded storage with a detection guarantee.

The methodology around it matters as much as the algorithm:

• Establish a baseline. For each problem, compute the similarity distribution across all submissions. Two independent solutions to an easy problem may legitimately score 0.7; the signal is being an outlier relative to that problem's distribution, not crossing an absolute threshold.
• Corroborate with behaviour. A submission that is highly similar to one posted publicly 40 seconds earlier, from an account with no partial attempts and an implausibly short editing session, is much stronger evidence than similarity alone.
• Human review before any penalty. Same principle as chess anti-cheat: statistical evidence plus judgement, never automation alone.
• Monitor the leak channels. Detecting a solution posted publicly during the contest lets you invalidate the problem for everyone, which is fairer than penalising the subset you happened to catch.

4.7 Twenty languages, each with its own way of being difficult

Why it's hard. Each language brings its own startup cost, memory profile, and pathologies. A JVM takes 100ms+ just to start and reports memory in a way that reflects the heap, not the algorithm. Python is 50× slower than C++, so a single time limit is either impossible for Python or trivial for C++. Compiled languages need a compilation step with its own limits. And each toolchain needs pinned versions, because a compiler upgrade silently changes verdicts.

Solution — per-language profiles, calibrated multipliers, and pre-warmed runtimes.

languages:
  cpp20:
    image: judge/cpp:gcc13.2            # pinned; changing it re-verifies problems
    compile: ["g++", "-O2", "-std=c++20", "-static", "main.cpp", "-o", "sol"]
    compile_limits: {wall_ms: 10000, memory_mb: 1024}
    time_multiplier: 1.0                # the reference
    startup_overhead_ms: 2
  python3:
    image: judge/python:3.12
    compile: null
    time_multiplier: 5.0                # calibrated from reference solutions
    startup_overhead_ms: 25
    preload: ["numpy", "sortedcontainers"]   # imported before the timer starts
  java21:
    image: judge/java:21
    compile: ["javac", "Main.java"]
    run: ["java", "-Xss64m", "-Xmx512m", "-XX:+UseSerialGC", "Main"]
    time_multiplier: 2.0
    startup_overhead_ms: 120            # subtracted from the measurement

Three details worth surfacing. Multipliers must be calibrated, not guessed — write a reference solution in each language for a sample of problems and measure. A guessed multiplier makes some languages unusable, which shows up as a complaint thread rather than a metric.

Subtract startup overhead from the measurement, or a Java solution spends 12% of a 1-second limit before main runs. Measure interpreter/JVM startup separately at worker boot and deduct it.

Pin toolchain versions and treat upgrades as a migration. A new compiler version changes optimisation behaviour and therefore verdicts; re-run a corpus of accepted submissions against the new image before rolling it out, and roll out per-problem rather than globally.

5. What breaks first

Event	First failure	Mitigation
Contest start	Judge queue depth explodes	Pre-scale on schedule; separate contest queue and pool; early-subset verdicts
Fork bomb / memory bomb	Worker degradation	cgroups pids.max and memory.max; one-shot VM destroyed regardless
Sandbox escape attempt	Potential host compromise	Layered isolation with a KVM boundary; workers run in a segmented network with no credentials
Popular problem	Test-data I/O on many workers	Bake data into images; key the queue by problem for cache locality
Measurement variance complaints	Appeals flood	Instruction-count limits, isolated cores, SMT off, fixed frequency
Solution leaked publicly mid-contest	Contest integrity	Monitor leak channels; invalidate the problem rather than selectively penalise
Toolchain upgrade	Silent verdict changes	Pinned versions; re-verify a corpus before rollout

6. Cheat sheet

• Isolation: Firecracker microVM (separate kernel) + seccomp allowlist + cgroups v2 + read-only rootfs + no NIC at all + one-shot destruction. Sandbox the compiler too.
• Burst: accept in 20ms and queue; separate contest lane and pool; pre-scale on schedule (contests are announced); key by problem for cache locality; push status, never poll.
• Determinism: isolcpus, SMT off, fixed frequency, exclusive core per run, and instruction count as the primary limit with wall time as a 3× safety net. Memory from the cgroup, not self-reported.
• Test data: streamed via pipes, expected output never inside the VM, output size capped at the pipe.
• Leaderboard: Redis sorted set with solved × 1e7 − penalty encoding both ranking keys into one score; freeze the final hour.
• Plagiarism: winnowing fingerprints over normalised token streams, compared against the per-problem similarity distribution, with human review.
• Languages: pinned images, calibrated time multipliers, startup overhead subtracted, upgrades re-verified against a corpus.
• The one-liner: "Two hard problems wearing one costume — safely executing hostile code, which needs a hardware isolation boundary, and measuring it reproducibly on shared hardware, which needs isolated cores and instruction counts instead of a stopwatch."