Web Crawler
The Mercator frontier, per-domain politeness queues, Bloom-filtered URL dedup over billions of URLs, spider traps, recrawl scheduling, and near-duplicate content detection.
News Aggregator
SimHash and MinHash LSH for wire-copy dedup, online clustering into evolving stories, source diversity and credibility, breaking-news latency, and personalisation without a filter bubble.
Price Tracking
Adaptive polling to make 500M products affordable, headless browser pools against anti-bot defences, percolator queries for alert fan-out, price extraction from heterogeneous pages, and time-series storage.