Google Trends Embeddings

A 3D semantic map of what the world is searching for right now.

Polls Google Trends' "trending now" RSS feeds across 10 countries, embeds each trend (search term + its news headlines) with a multilingual model, and projects the result into 3D with UMAP. Colors encode topic (Politics, Sports, Entertainment, …), lines connect semantically-similar trends from different countries, size encodes peak search traffic.

Built as a tool for journalists to answer "is this story local or global?" at a glance — the same event trending in Dutch, German, and Japanese lands in the same cluster because the embeddings are multilingual.

Stack

• Python 3.11+, managed with uv
• BAAI/bge-m3 (sentence-transformers) for multilingual embeddings, CPU-only torch
• sklearn.cluster.HDBSCAN for cluster detection
• umap-learn for 3D projection
• FastAPI + a single /api/map endpoint
• Vanilla three.js frontend, no build step
• SQLite for storage

Run it locally

uv sync                              # first time pulls bge-m3 (~2.3 GB)
uv run python scripts/run_once.py    # fetch + embed + classify + UMAP once
uv run uvicorn app.web:app           # → http://127.0.0.1:8000

The web process runs the ingest loop in the background and re-polls every 30 minutes.

How it works

1. Fetch RSS for 10 geos → parse with lxml (handles the ht: namespace).
2. Upsert trends into SQLite keyed by (geo, title_normalized, pub_date). Peak traffic is max()-merged across polls.
3. For each trend, embed "{title}. {headline_1} · {headline_2} · …". Re-embed when the headline set changes.
4. Zero-shot topic classification by argmax cosine similarity against per-topic description embeddings (same encoder).
5. Fit 3D UMAP + HDBSCAN on the full corpus and write coords + cluster ids.
6. Frontend fetches /api/map, renders a three.js scatter, and offers click-to-filter by topic.

Prototype scope

10 countries (NL, DE, FR, GB, US, BR, JP, IN, ZA, AU). No filters, no search, no translation, no history UI. The map is the feature. See CLAUDE.md for the full design notes and future plans.

Layout

config.py            # countries, poll interval, model name
app/
  parse.py           # lxml RSS parser
  store.py           # SQLite repo
  fetch.py           # async feed fetcher
  embed.py           # sentence-transformers wrapper
  topics.py          # topic taxonomy + zero-shot classifier
  umap_job.py        # UMAP fit + HDBSCAN cluster
  loop.py            # background cycle
  web.py             # FastAPI app
  static/            # index.html + three.js frontend
scripts/run_once.py  # manual one-shot cycle