Grafana Alloy as a central gateway: real-world sizing figures

Anyone managing an enterprise platform with dozens of application teams eventually hits the same problem: where to funnel metrics, logs and traces before sending them to the observability backend. The Grafana Labs blog published a technical report on 21 August, with real and anonymised data, on how a Kubernetes cluster of Alloy was sized and used as a central telemetry gateway for a large enterprise customer.

The central gateway problem

In the gateway model, all telemetry from application teams—metrics, logs and traces—converges on a shared fleet of Alloy via OTLP or native Prometheus/Loki write protocols. Alloy buffers, processes, groups and forwards everything to Grafana Cloud. The advantage is a single authentication point, rate limiting and cost attribution; the drawback is that this becomes a critical piece of infrastructure: when it struggles, the entire system feels it. For this reason it must be treated like any other production service: capacity planning first, then load testing before go-live.

The baseline numbers

For the customer followed by Grafana's Professional Services team, the expected traffic was around 17 million active series (via OTLP and RW), 1 TB/day of logs with a peak of 17.5 MB/s (via OTLP and LW), and around 1 TB/day of traces with a peak of about 23 MB/s (via OTLP). The figure was not the current volume but the target at the end of onboarding: Grafana explains that you need to size for that margin, not just the current baseline, because most large enterprises add teams gradually but still need to prepare full capacity upfront.

Resources and configuration

The planned resource budget was approximately 187 GB of memory and 7 CPU cores for metrics, 2.1 GB and 17.5 cores for logs, 4 GiB and 3 cores for traces: a total of around 195 GB of memory and 28 CPU cores. Pods were configured with 0.5 CPU in request, 6 GiB of memory in request and limit, but with no CPU limit, to avoid throttling which in Kubernetes generates hidden latency under high-throughput loads. HPA was set with minReplicas 30, maxReplicas 100, target at 70% CPU and 90% memory.

In production

Since go-live, the cluster has handled volumes exceeding plan: around 17 million active series, 20–30 MB/s of logs and up to 150 MB/s of traces. Among the operational lessons highlighted by Grafana, two directly concern cluster operators: Alloy's write-ahead log can saturate memory if not monitored under sustained pressure, and GOMEMLIMIT should be set to around 80% of the pod's memory limit (example: 4915MiB on a 6GiB limit) to trigger Go's garbage collector before the container's OOM kill.