News Aggregator (Google News)

Sharpened prompt. Design a news aggregator ingesting 500k articles/day from 50,000 publishers, collapsing near-identical wire copies into one story, grouping related coverage into evolving clusters that live for days, surfacing breaking news within two minutes, balancing sources so one publisher does not dominate, and personalising without trapping anyone in a bubble.

The core technical problem is clustering a high-velocity text stream in real time, where the clusters themselves are the product and they must merge, split, and expire as a story develops.

1. Problem framing

Functional requirements

Ingest articles from RSS, sitemaps, partner APIs, and crawling.
Deduplicate identical and near-identical articles.
Cluster related articles into "stories" that grow over hours or days.
Rank stories for a homepage and for topic sections; personalise per user.
Support search, and "full coverage" views showing multiple perspectives.

Non-functional requirements

Property	Target	Consequence
Ingest-to-visible	< 2 min for breaking news	Streaming pipeline; no batch clustering
Dedup precision	Very high — a visible duplicate is an obvious failure	LSH plus a verification step
Cluster quality	Coherent stories, no merged unrelated events	Online clustering with a conservative merge threshold
Diversity	No single source dominating a story view	Explicit diversity constraints at slotting time
Scale	500k articles/day, 100M users	Modest ingest; the clustering state is the interesting part

Back-of-the-envelope

Articles:    500k/day = ~6/sec average, ~50/sec during major events
Text:        500k × 5 KB = 2.5 GB/day of text — small. This is not a volume problem.
Embeddings:  500k × 768 dims × 4 B = 1.5 GB/day; 30-day active window = 45 GB
             -> fits in a single vector index node, replicated
Clusters:    ~20k active stories at any time, each with 1-500 articles
Comparisons: naive clustering = 500k × 20k = 10^10 similarity computations/day.
             With LSH + an ANN index: ~50 candidate comparisons per article.
             That is the whole trick, again.
Users:       100M, ~10M DAU × 20 story impressions = 200M impressions/day

Note that this is a small-data, hard-algorithm problem — the opposite of most pages in this playbook. Saying so early reframes the discussion away from sharding and toward clustering quality.

2. High-level architecture

3. Component inventory

Component	Concrete choice	Why this one
Extraction	Readability / Trafilatura, plus per-publisher rules	Boilerplate destroys both dedup and clustering if left in
Near-dup	SimHash (64-bit) for near-identical; MinHash LSH for partial overlap	Two different similarity notions, two structures
Embeddings	A multilingual sentence transformer, distilled and quantised	Must run at ingest rate on CPU; quality matters more than size here
Vector index	Faiss HNSW (or Milvus) over a rolling 30-day window	ANN over 15M vectors with millisecond recall
Clustering	Online single-pass with centroid update, plus periodic re-clustering	Streaming requirement rules out batch k-means
Story store	Postgres or Cassandra, keyed by story ID	Small; needs transactional member updates
Ranking	Learned model over story features, with hard diversity constraints	Two different mechanisms for two different goals

4. The toughest parts

4.1 Wire copy: the same article from 400 publishers

Why it's hard. An AP or Reuters story is republished by hundreds of outlets, each with a different headline, a different lead paragraph, added local context, and different boilerplate. Exact hashing catches none of them. Full pairwise similarity against every recent article is 500k × 500k comparisons per day. And precision must be very high: showing the same story twice is the single most visible failure this product can have.

Solution — two complementary LSH schemes, chosen by the kind of similarity you need to detect.

# SimHash: near-IDENTICAL documents (a syndicated wire story with a new headline).
# Similar documents produce fingerprints within a small Hamming distance.
def simhash64(text: str) -> int:
    v = [0] * 64
    for shingle in shingles(normalise(text), n=5):          # 5-word shingles
        h = xxhash.xxh64(shingle).intdigest()
        for i in range(64):
            v[i] += 1 if (h >> i) & 1 else -1
    return sum(1 << i for i in range(64) if v[i] > 0)

def find_near_identical(fp: int, k: int = 3) -> list[int]:
    # Pigeonhole: two fingerprints within distance 3 must share one of 4 16-bit
    # blocks exactly. Four hash lookups instead of scanning millions.
    out = []
    for b in range(4):
        for cand in block_index[b].get((fp >> (16 * b)) & 0xFFFF, ()):
            if popcount(fp ^ cand) <= k:
                out.append(cand)
    return out

# MinHash + banded LSH: PARTIAL overlap (an article quoting several paragraphs).
# Estimates Jaccard similarity of shingle sets, which SimHash does not.
def minhash(text: str, perms: int = 128) -> np.ndarray:
    sh = {xxhash.xxh64(s).intdigest() for s in shingles(text, n=5)}
    return np.array([min((a * s + b) % P for s in sh) for a, b in HASH_PARAMS])

def lsh_bands(sig: np.ndarray, bands: int = 32) -> list[bytes]:
    # A candidate pair must match on at least one band. Tuning (bands, rows)
    # sets the similarity threshold: 32 bands × 4 rows ≈ threshold 0.5.
    rows = len(sig) // bands
    return [sig[i*rows:(i+1)*rows].tobytes() for i in range(bands)]

State the distinction, because using the wrong one is a common mistake: SimHash measures overall document similarity and is ideal for "is this the same article?" while MinHash estimates set overlap and is ideal for "does this share substantial passages with that?" Wire-copy detection wants SimHash; plagiarism and partial-quote detection want MinHash.

Always verify candidates before merging. LSH produces candidates, not decisions: compare the actual texts (cosine over TF-IDF, or a cross-encoder for the ambiguous band) before declaring a duplicate. The cost is negligible because there are only a handful of candidates, and the precision gain is large.

When a duplicate is found, do not discard it — attach it to the canonical article with its publisher, so the "also reported by" list and the source-diversity logic have something to work with. Choose the canonical by earliest publication timestamp (with the originating wire service preferred), which also gives you attribution for free.

4.2 Clustering a stream into stories that evolve

Why it's hard. Batch clustering (k-means, DBSCAN over a day's articles) cannot meet a two-minute latency target and produces unstable cluster IDs that change every run — so a story's URL changes, breaking links and history. Online clustering is required, but it is greedy: an early bad merge poisons a cluster permanently, and two genuinely distinct events with similar vocabulary (two separate earthquakes) can collapse into one.

Solution — single-pass assignment against centroids with a conservative threshold, plus periodic repair.

async def assign(article) -> StoryId:
    v = embed(article)                       # normalised sentence embedding

    # ANN over ACTIVE story centroids only — a few thousand, not all history.
    candidates = vector_index.search(v, k=20, filter={"active": True,
                                                      "lang": article.lang})

    best, best_score = None, 0.0
    for story in candidates:
        score = (0.60 * cosine(v, story.centroid)
               + 0.20 * entity_overlap(article.entities, story.entities)
               + 0.10 * time_proximity(article.published_at, story.last_update)
               + 0.10 * (1.0 if article.lang == story.lang else 0.0))
        if score > best_score:
            best, best_score = story, score

    # Conservative: when in doubt, create a new story. Splitting a story later is
    # easy and invisible; un-merging two events that were wrongly joined is not.
    if best_score >= MERGE_THRESHOLD:                       # ~0.82
        await best.add(article, v)                          # incremental centroid update
        return best.id
    return await create_story(article, v)

# Repair pass, every 15 minutes over recently-touched stories:
#   - MERGE two stories whose centroids have converged (coverage of one event
#     that started as two).
#   - SPLIT a story whose members form two well-separated sub-clusters.
#   - EXPIRE stories with no new articles in 48h (remove from the active set).

Three points to make. Asymmetric error cost justifies the conservative threshold: an over-split story shows as two similar entries (mildly redundant), while an over-merged story shows unrelated headlines together (obviously broken). Bias toward splitting.

Entity overlap is the signal that saves you. Pure embedding similarity confuses two earthquakes; named entities (locations, people, organisations) distinguish them sharply. Weighting entities alongside the embedding is the single highest-value addition to the score.

The repair pass is what makes greedy assignment acceptable. Greedy online clustering will make mistakes; a periodic merge/split pass over recently active stories corrects them within minutes, at a cost proportional to activity rather than to corpus size.

Story identity must be stable: when two stories merge, keep the older ID and redirect the newer, so links and user history survive.

4.3 Breaking news in under two minutes

Why it's hard. A major event is reported by one outlet at 14:03. Your pipeline must ingest, extract, dedup, embed, cluster, rank, and surface it — while the ranking model has almost no signal, because a two-minute-old story has no engagement history and no corroborating coverage. Rank it purely on quality signals and it loses to established stories; rank it purely on recency and every trivial post floods the homepage.

Solution — a fast lane with a separate ranking regime and corroboration as the promotion signal.

def story_score(story, now) -> float:
    age_min = (now - story.first_seen).total_seconds() / 60

    # Velocity: how fast is coverage accumulating? This is the breaking-news signal,
    # and it is available within minutes because it needs no user engagement.
    velocity = story.article_count / max(1.0, age_min)

    # Corroboration: independent outlets covering it. One outlet is a claim;
    # fifteen independent outlets within ten minutes is an event.
    corroboration = math.log1p(story.distinct_publishers)
    authority = mean(publisher_authority(p) for p in story.publishers)

    freshness = math.exp(-age_min / 180)              # 3-hour half-life
    engagement = math.log1p(story.clicks) if age_min > 30 else 0.0   # unavailable early

    return (2.0 * velocity * corroboration
          + 1.5 * authority
          + 2.0 * freshness
          + 1.0 * engagement
          + 1.0 * topic_importance(story))

Corroboration replaces engagement as the early-signal. Independent coverage by multiple credible outlets within a short window is both a strong importance signal and a strong veracity signal, and it is available in minutes rather than hours. Weighting velocity × corroboration together (rather than adding them) means a single outlet posting furiously does not score, but fifteen outlets posting once each does.

Operationally, the fast lane also needs prioritised ingestion: poll high-authority publishers every 30 seconds rather than every 10 minutes, prefer push (PubSubHubbub/WebSub, partner APIs) over polling, and give newly-created stories priority through the embedding and clustering stages so a breaking story is never stuck behind a batch of routine content.

4.4 Source diversity: not letting one publisher own a story

Why it's hard. Ranking each article independently by quality means the highest-authority publisher takes every slot in a story's coverage. Users see one perspective, smaller and local outlets never surface, and the product becomes a syndication channel for three large organisations. Meanwhile the opposite failure — mechanically balancing "both sides" — can elevate fringe sources to false equivalence with established reporting.

Solution — diversity as a constrained selection problem, with credibility as a floor rather than a balancing axis.

def select_coverage(story, n=8):
    # Credibility is a FLOOR, not a dimension to balance across. Sources below
    # the bar are excluded outright; above it, diversity is maximised.
    eligible = [a for a in story.articles if publisher_credibility(a.publisher) >= FLOOR]

    picked, seen_pub, seen_owner, seen_persp = [], set(), set(), Counter()
    while len(picked) < n and eligible:
        best = max(eligible, key=lambda a:
              article_quality(a)
            * (0.25 if a.publisher in seen_pub else 1.0)          # one per publisher
            * (0.50 if a.owner_group in seen_owner else 1.0)      # media conglomerates
            * (0.60 ** seen_persp[a.perspective_cluster])         # viewpoint spread
            * (1.30 if a.is_local_to(story.location) else 1.0)    # local outlets matter
            * (1.20 if a.is_original_reporting else 1.0))         # not aggregation
        picked.append(best); eligible.remove(best)
        seen_pub.add(best.publisher); seen_owner.add(best.owner_group)
        seen_persp[best.perspective_cluster] += 1
    return picked

Two things worth stating explicitly. Owner-group deduplication matters more than publisher deduplication: many nominally distinct outlets share a parent, and diversity that ignores ownership is cosmetic. And original reporting deserves a boost — much of what circulates is aggregation of someone else's work, and surfacing the source both improves the product and aligns incentives in the ecosystem you depend on.

On credibility: treat it as a threshold applied before selection, derived from durable signals (corrections policy, editorial standards, factual-accuracy track record, transparency of ownership) rather than from engagement. Using engagement as a proxy for credibility is precisely how aggregators amplify low-quality content, and being able to say why you would not do that is a substantive answer.

4.5 Personalisation that does not become a bubble

Why it's hard. Optimising click-through personalises hard: the model learns a user's preferences and narrows relentlessly, until they see only reinforcing coverage. That is bad for the user, bad for the product's long-term value, and in news specifically it has societal consequences. But no personalisation at all produces a generic homepage that serves nobody well.

Solution — personalise the ordering within a curated set, and reserve explicit slots for breadth.

def homepage(user, candidate_stories, n=20):
    slots = []

    # 1. Reserved slots for what everyone should see, regardless of interests.
    slots += top_by(candidate_stories, key=importance, n=3, tag="top_news")

    # 2. Personalised slots — but selected from stories that already passed the
    #    quality and importance bar, so personalisation reorders rather than
    #    determining what is eligible.
    eligible = [s for s in candidate_stories if s.importance > IMPORTANCE_FLOOR]
    personal = rank_by(eligible, key=lambda s: affinity(user, s), n=12)
    slots += personal

    # 3. Deliberate exploration: topics and perspectives outside the user's history.
    #    Not random — high-quality stories in adjacent areas.
    slots += explore(user, eligible, n=3, novelty_weight=0.7)

    # 4. Local news, which personalisation systematically under-serves because
    #    it has low engagement relative to national content.
    slots += local_news(user.location, n=2)

    return dedupe_by_story(slots)[:n]

The structural idea to articulate: personalisation chooses the order, editorial logic chooses the eligible set. That single constraint prevents the runaway narrowing, because a story cannot be filtered out purely for being outside a user's demonstrated preference — only for being low quality or unimportant.

Measure it. Track topic entropy and source entropy per user over time as first-class metrics, and alarm when they trend down. A personalisation system without a diversity metric will narrow, because narrowing improves the metric it is measured on.

4.6 Getting the text out of the page

Why it's hard. Publisher pages are 90% navigation, ads, related-links, cookie banners, and newsletter prompts. Boilerplate poisons everything downstream: dedup compares navigation menus, embeddings encode "subscribe to our newsletter," and clustering groups articles by their shared template rather than their content. There are 50,000 publishers, so per-site rules do not scale.

Solution — generic extraction with per-publisher overrides where volume justifies them, and structured data first.

def extract(html: str, url: str) -> Article:
    # 1. Structured data is authoritative when present — no heuristics needed.
    if (ld := parse_json_ld(html)) and ld.get("@type") in ("NewsArticle", "Article"):
        return Article(title=ld["headline"], body=ld.get("articleBody"),
                       published=parse_date(ld["datePublished"]),
                       authors=ld.get("author"), source="json-ld")

    # 2. Learned per-publisher template, once we've seen enough of their pages.
    if tmpl := template_store.get(registrable_domain(url)):
        if (a := tmpl.apply(html)).is_valid():
            return a

    # 3. Generic extraction: text-density and link-density heuristics.
    a = trafilatura_extract(html)

    # 4. Validate. A "successful" extraction of 40 words is a failure.
    if len(a.body.split()) < 100 or a.link_density > 0.3:
        metrics.inc("extraction.suspect", domain=registrable_domain(url))
        a.confidence = "low"                       # excluded from clustering
    return a

The validation step is what candidates usually omit and what actually keeps quality up: silently accepting a bad extraction poisons the cluster, whereas flagging low-confidence extractions and excluding them from clustering degrades gracefully. Feed the suspect-extraction rate per domain into a queue for learning a template — the highest-volume publishers get fixed first, which is exactly the right prioritisation.

Prefer structured data wherever it exists. JSON-LD NewsArticle gives you headline, body, publication time, and authors with no guessing, and most professional publishers emit it for their own SEO reasons.

5. What breaks first

Event	First failure	Mitigation
Major breaking event	10× article rate, clustering lag	Prioritise the fast lane; scale embedding workers on lag; conservative threshold prevents bad merges under load
A publisher redesigns	Extraction silently degrades	Per-domain extraction-quality monitoring; low-confidence articles excluded from clustering
Two similar events same day	Wrongly merged story	Entity overlap in the score; repair pass splits well-separated sub-clusters
Coordinated content farm	Fake corroboration inflates a story	Owner-group deduplication in the corroboration count; credibility floor
Vector index rebuild	Clustering unavailable	Blue/green index with alias swap, same as Yelp
Personalisation drift	Users narrow over time	Topic and source entropy as monitored metrics with alarms

6. Cheat sheet

This is a small-data, hard-algorithm problem. 2.5 GB/day of text; the difficulty is clustering quality, not scale.
Dedup: SimHash for near-identical wire copy (pigeonhole blocks for distance-3 lookup), MinHash LSH for partial overlap. Always verify candidates.
Clustering: online single-pass against active centroids via ANN, score = embedding + entity overlap + time + language, conservative merge threshold, periodic merge/split/expire repair pass.
Breaking news: velocity × corroboration replaces engagement as the early signal; prioritised polling and push feeds for the fast lane.
Diversity: credibility is a floor, not a balance axis; deduplicate by owner group, boost local and original reporting.
Personalisation reorders an editorially-curated eligible set; reserved slots for top news, exploration, and local. Monitor topic entropy.
Extraction: JSON-LD first, learned templates second, generic heuristics third, and always validate the result.
The one-liner: "Two LSH schemes collapse the wire copies, an entity-aware online clusterer turns the stream into stories, and a repair pass makes greedy clustering survivable — the ranking is comparatively easy once the clusters are right."

1. Problem framing​

Functional requirements​

Non-functional requirements​

Back-of-the-envelope​

2. High-level architecture​

3. Component inventory​

4. The toughest parts​

4.1 Wire copy: the same article from 400 publishers​

4.2 Clustering a stream into stories that evolve​

4.3 Breaking news in under two minutes​

4.4 Source diversity: not letting one publisher own a story​

4.5 Personalisation that does not become a bubble​

4.6 Getting the text out of the page​

5. What breaks first​

6. Cheat sheet​