How AWS Lambda logs every flow across thousands of microVMs per host with eBPF and Rust
The New Stack Prashant Kumar Singh
AWS Lambda rebuilt its network logging stack with eBPF and Rust. It had to track tiny, short-lived microVMs without missing packets or burning host resources.
Based on reporting by The New Stack, Prashant Kumar Singh — read the original for the full story.
Summary, retelling and take written by AI under human oversight; images are AI-generated illustrations. How we work · Report an error
AWS Lambda has swapped out an older network capture system for a new one built around eBPF and Rust, because the old setup ran into trouble once Lambda’s host density got high enough. The problem wasn’t abstract. On a single worker, thousands of Firecracker microVMs can live for only a few hundred milliseconds, then vanish. When a security alert, audit check, or billing question comes up, those logs may be the only record left.
The earlier design came from Lambda’s older EC2-era world. It used kernel-side packet counters and a userspace daemon that turned them into Amazon Ion records. That worked when the number of VMs was smaller. It stopped scaling for two reasons: iptables rule counts exploded as more microVMs were packed onto a host, and the kernel module never learned IPv6. Once dual-stack support was on the table, the old path was done.
The replacement splits the work cleanly. Small eBPF programs sit at traffic-control hooks on the ingress and egress sides of each network device, watch packets, and emit one compact event per packet into a ring buffer. A per-network Rust tagger drains that buffer, rolls events up into flow records, and writes the same Amazon Ion format the rest of the pipeline already understands. Above that, one privileged orchestrator per host loads the programs, wires up traffic control, and manages taggers as microVMs appear and disappear.
The design is careful about the parts that can quietly poison a log. The kernel code normalizes fields so “local” always means the sandbox side, handles IPv4 and IPv6, deals with fragments and offload packets, and writes events that are byte-compatible with the old records. The team also checks the eBPF code with CBMC during every build, because a logging bug that touches billing or compliance is the sort that waits until the worst possible day to show up.
Even the ring buffer size is worked out from packet rate rather than guessed. Lambda says the floor comes out to about 300 KB for a guest hitting 62,500 packets per second per direction with drains every 100 milliseconds, and the API requirement pushes the default to 512 KiB. In the running deployment, they’re using a few megabytes per network while tuning the final per-workload value. The broader point is simple: at Lambda’s scale, logging can’t be a side quest. It has to be cheap, exact, and invisible.
My take — AI-written commentary, not fact-checked reporting
This is the right move: logging systems should be boring, exact, and almost invisible, not heroic little tax collectors on every packet. The interesting bit is how much of modern cloud engineering is now just proving that the kernel, the serializer, and the billing stack all agree on reality. Also, if your security story can’t survive IPv6, it wasn’t a security story; it was a nice demo.
Read more about this at: The New Stack