Storing News at the Edge with Cloudflare D1 and R2

Published 2026-10-07 by Web2News Systems Engineering

Aggregating over one hundred external RSS and Atom feeds on an edge runtime introduces an asymmetric data problem. Metadata changes slowly: the list of accredited news sources, their categories, and their HTTP ETag watermarks stay relatively compact. Syndicated item payloads, headline snippets, and raw XML responses change every few minutes and accumulate quickly over weeks.

Trying to store every parsed headline, description snippet, and raw feed document inside a single transactional SQL database creates unnecessary row-write volume, storage churn, and vacuum overhead. Web2News avoids this bottleneck by separating control-plane state in Cloudflare D1 from immutable and snapshot object payloads in Cloudflare R2, while baking the public reading interface into static HTML.

Why Naive SQL Aggregation Hits Limits

When an automated harvester polls 129 RSS and Atom endpoints every ten to thirty minutes, even a conservative twenty items per feed produces roughly 2,500 headline records per cycle, or more than 120,000 item updates per day. In a naive serverless relational schema:

  1. Write Amplification: Upserting thousands of items repeatedly triggers redundant index updates even when an upstream newsroom has not published a new article since the last poll.
  2. Read Multiplication: Rendering a homepage with 128 source boxes by querying live database tables per request wastes compute budgets and degrades Time to First Byte (TTFB) during traffic spikes.
  3. Pruning Complexity: Retaining rolling 48-hour caches alongside multi-week historical ledgers requires continuous DELETE jobs and compaction passes.

Splitting Control Plane (D1) and Object Plane (R2)

Our storage topology assigns each Cloudflare primitive strictly to what it handles most efficiently:

Conditional Fetching with HTTP ETags and Abort Timeouts

Responsible syndication requires respecting upstream publishers' bandwidth. During feed harvesting (scripts/harvest-live-feeds.ts and edge workers):

Static Pre-Rendering at Build Time

Most readers on a headline portal want instant scanning, zero spinner states, and predictable keyboard navigation. Rather than waiting on client-side waterfall fetches to draw 128 cards, our static builder (scripts/build-static-content.ts) compiles:

On top of that static baseline, optional interactive enhancements such as /api/ticker updates, keyword watchlists, and the Personal Media Diet tracker hydrate asynchronously in the browser without blocking first paint. Decoupling edge metadata, zero-egress object snapshots, and pre-rendered HTML delivers edge resilience with near-zero origin load.


← Back to all Web2News Editorial Essays | Editorial Methodology | Explore Live Analytics