Cloudflare: five interventions in five years to slow data infrastructure growth
Cloudflare explains how it cut data pipeline traffic without increasing hardware: schema partitioning, compression, replication, sampling and aggregations.
UptimeMag editorial team · 14 October 2026 · 2 min read
Cloudflare published on 13 October 2026 on its technical blog the details of five interventions implemented between 2022 and autumn 2026 to contain hardware growth behind its data pipeline. The topic concerns anyone using Cloudflare Logs, the SQL API or analytics products: this is the infrastructure that collects and routes logs that those products read from.
Over 20% of the web runs behind Cloudflare's network, and the Data team owns the pipeline that carries that data from the global network to central data centres, across a distributed network spanning tens of thousands of servers. In May 2026 the pipeline delivered approximately two billion events per second, including customer traffic and internal data. For comparison: in December 2024 Cloudflare reported that the same pipeline ingested roughly 706 million events per second, according to an earlier blog post. In less than two years the volume nearly tripled.
Five interventions, five years
| Year | Intervention | Effect reported by Cloudflare |
|---|---|---|
| 2022 | Breaking down monolithic schema into product-specific schemas | −35–40% bandwidth on backbone |
| 2024 | Compression change from LZ4 to zstd | −40% compressed traffic, same pipe handles 1.5× the traffic |
| Spring 2026 | New replication architecture (load balancing) | eliminated approximately 43 GB/s of compressed transit traffic, adding only ~7 GB/s cross-data centre replication |
| Summer 2026 | Sampling moved upstream to Logfwdr | −25 GB/s across three main datasets, over 2 PB/day |
| Autumn 2026 | Aggregations inserted into pipeline | on HTTP dataset, drop from over 9 GB/s to under 2 GB/s, approximately −80% on the wire |
For load balancing, the first cut came on the HTTP analytics dataset with a reduction of around 20 GB/s (over 50%), whilst the Ready-Analytics dataset fell by 43%, from roughly 28 to 16 GB/s.
Why this matters for those managing their own logs
The techniques described are not exclusive to Cloudflare: product-specific schema instead of a single monolithic schema, zstd compression in place of LZ4, sampling applied as far upstream as possible, and aggregations computed in the pipeline rather than at query time are reusable patterns for any log stack built on buffers and ClickHouse (or similar). In Cloudflare's pipeline data arrives from centralised receivers, passes through the Buftee buffering system and from there gets distributed to final destinations such as ClickHouse. Anyone managing a log pipeline with a volume even several orders of magnitude smaller can apply the same logic to defer a bandwidth or storage upgrade.
Written with the help of artificial intelligence and checked by the editors (EU AI Act, art. 50).