2 docs tagged with "distributed-systems"

Why exactly-once delivery is impossible, what exactly-once processing actually means, and the four mechanisms that make retries safe.

The Mercator frontier, per-domain politeness queues, Bloom-filtered URL dedup over billions of URLs, spider traps, recrawl scheduling, and near-duplicate content detection.