Storing News at the Edge with Cloudflare D1 and R2
Published 2026-10-07 by Web2News Systems Engineering
Aggregating over one hundred external RSS and Atom feeds on an edge runtime introduces an asymmetric data problem. Metadata changes slowly: the list of accredited news sources, their categories, and their HTTP ETag watermarks stay relatively compact. Syndicated item payloads, headline snippets, and raw XML responses change every few minutes and accumulate quickly over weeks.
Trying to store every parsed headline, description snippet, and raw feed document inside a single transactional SQL database creates unnecessary row-write volume, storage churn, and vacuum overhead. Web2News avoids this bottleneck by separating control-plane state in Cloudflare D1 from immutable and snapshot object payloads in Cloudflare R2, while baking the public reading interface into static HTML.
Why Naive SQL Aggregation Hits Limits
When an automated harvester polls 129 RSS and Atom endpoints every ten to thirty minutes, even a conservative twenty items per feed produces roughly 2,500 headline records per cycle, or more than 120,000 item updates per day. In a naive serverless relational schema:
- Write Amplification: Upserting thousands of items repeatedly triggers redundant index updates even when an upstream newsroom has not published a new article since the last poll.
- Read Multiplication: Rendering a homepage with 128 source boxes by querying live database tables per request wastes compute budgets and degrades Time to First Byte (TTFB) during traffic spikes.
- Pruning Complexity: Retaining rolling 48-hour caches alongside multi-week historical ledgers requires continuous
DELETEjobs and compaction passes.
Splitting Control Plane (D1) and Object Plane (R2)
Our storage topology assigns each Cloudflare primitive strictly to what it handles most efficiently:
- Cloudflare D1 (Relational Control Plane): D1 maintains the authoritative source registry (
id,name,category,kind,feed_url,site_url,column_order,is_active) alongside lightweight sync telemetry (last_etag,last_modified,last_fetch_status,last_sync_at). Because only feed-level watermarks update on a successful fetch, D1 row writes stay tiny and predictable. - Cloudflare R2 (Bulk Snapshot & Ledger Store): R2 stores normalized JSON feed snapshots (
feeds/{source_id}/latest.json), raw audit bundles, and dated daily ledger snapshots (ledgers/{YYYY-MM-DD}.json). Because R2 incurs zero egress charges inside the Cloudflare network, Workers and Pages Functions can fetch cached JSON snapshots or serve/api/feed/:idand/api/tickerresponses without touching SQL tables.
Conditional Fetching with HTTP ETags and Abort Timeouts
Responsible syndication requires respecting upstream publishers' bandwidth. During feed harvesting (scripts/harvest-live-feeds.ts and edge workers):
- Requests run inside bounded concurrency pools (batches of 10 sources) with strict 8-second
AbortControllertimeouts so one stalled publisher server never delays the rest of the wire. - HTML entities and malformed numeric character references returned by legacy CMS templates are sanitized deterministically before storage, preventing broken encoding or layout-breaking tags from reaching the UI.
- If a feed temporarily returns HTTP 429 or 5xx errors, the snapshot preserves the last known verified items cleanly while logging the fetch status rather than blanking out the bureau column.
Static Pre-Rendering at Build Time
Most readers on a headline portal want instant scanning, zero spinner states, and predictable keyboard navigation. Rather than waiting on client-side waterfall fetches to draw 128 cards, our static builder (scripts/build-static-content.ts) compiles:
- The master homepage grid and all 7 category hub pages directly into pre-rendered HTML.
- Individual Signal-to-Noise profile pages (
/source/<id>/), topic intelligence hubs (/topic/<slug>/), and daily historical ledgers (/news/YYYY-MM-DD/). - Standards-compliant RSS 2.0 XML feeds (
/feeds/<slug>.xml) and an importable OPML bundle (/feeds.opml) so readers can take our curated list to any desktop feed reader.
On top of that static baseline, optional interactive enhancements such as /api/ticker updates, keyword watchlists, and the Personal Media Diet tracker hydrate asynchronously in the browser without blocking first paint. Decoupling edge metadata, zero-egress object snapshots, and pre-rendered HTML delivers edge resilience with near-zero origin load.
← Back to all Web2News Editorial Essays | Editorial Methodology | Explore Live Analytics