Firecracker Snapshots for Coding Agents

Every frontier lab now runs coding agents, and every coding agent needs a computer: it installs packages, builds, runs tests, and breaks things. Reinforcement learning on coding tasks needs thousands of fresh environments an hour, each used once and thrown away. Platform teams converge on the same answer for starting them: don’t boot. Boot once, install the toolchain, warm every cache, and save the whole machine; every environment after that is a restore. On this page I want to go through how Firecracker snapshots work underneath, and the techniques that make restores fast at that scale.

What a snapshot is

Firecracker is the virtual machine monitor AWS built for Lambda. Snapshotting a paused microVM writes two files: a small vmstate file with vCPU registers and device state, and a memory file with every page of guest RAM. A restore loads them into a fresh Firecracker process and the guest continues mid-instruction.

The disk isn’t included. The guest’s block device has to be exactly as it was at the moment of the snapshot, or the guest’s in-memory view of its filesystem stops matching what’s on disk. In practice the disk is a copy-on-write layer over a read-only image, frozen alongside the memory file.

Lazy restore and its cost

With the default File backend, Firecracker maps the memory file into the new process privately. Nothing is read up front; a page is loaded when the guest first touches it, and a write gets a private copy. A 16 GiB guest resumes in milliseconds, and many guests restored from one file on one host share its untouched pages through the host page cache.

The cost moves to the first second after resume. The guest touches its working set all at once, each touch is a page fault, and each fault is a small random read. Against local NVMe that’s tolerable. Against anything remote it dominates. Most of the techniques below are ways of not paying for those faults one at a time.

Recording the working set

Restores of the same snapshot touch mostly the same pages in mostly the same order, because the guest is resuming the same code from the same state. So record it once and replay it.

The UFFD backend hands every page fault to a process you write, through Linux’s userfaultfd. On a first restore, the handler logs which pages were faulted and in what order. On later restores, it copies that set into guest memory in one sequential pass, before or alongside resume, and only falls back to on-demand faults for pages outside it. REAP (ASPLOS 2021) measured this on Firecracker and cut cold-start time by several times; FaaSnap (EuroSys 2022) went further by loading the set concurrently with execution instead of before it.⁠REAP is part of the vHive project. Both papers found the working set is small relative to guest memory and stable across invocations, which is what makes recording it worthwhile.

The working set also tells you how to lay out storage: write recorded pages contiguously, so the prefetch is one large read instead of thousands of small ones.

Shrinking what you save

A memory file is the size of guest RAM, most of which is free or holds data you could regenerate. Three things keep it small:

1. Inflate a balloon before snapshotting. The guest hands its free pages to the balloon driver, and they become zero pages in the file.
2. Store the file as content-addressed chunks. Zero chunks and chunks identical across snapshots, like an unchanged kernel or a shared toolchain, are stored once.
3. Decide deliberately about the guest’s page cache. Dropping it shrinks the snapshot; keeping it is often the reason you snapshotted, since a warm compiler cache or loaded model weights are what make the restored machine fast.

Diff snapshots and trees

With dirty page tracking on, KVM records every page the guest writes, and a diff snapshot saves only those. A diff is useless alone; it’s laid over the snapshot it came from, and Firecracker’s snapshot-editor can merge the two.⁠Tracking dirty pages costs a little on every first write to a page, so it’s off unless you ask for it.

Diffs turn snapshots into a tree. A base with the OS, a child with the toolchain, a child of that with a specific repository checked out at a specific commit. Each level shares everything above it.

This is where coding agent platforms differ most from serverless. An RL rollout can snapshot an environment mid-trajectory and restore it N times to explore N continuations; best-of-N sampling and tree search over a coding agent’s actions want the same thing. Branching a running computer becomes as cheap as writing a diff, and the environment, not just the model’s context, becomes something you can fork.

Tiered chunk caches

Snapshots live in tiers, from fastest to most durable: pages already in a host’s page cache, chunks on local NVMe, then object storage. AWS’s 2023 paper on loading Lambda container images describes the design most platforms now follow: split images into chunks named by content hash, deduplicate across customers, and put a shared cache tier between hosts and S3.

Two problems are particular to coding agents. The first is the thundering herd: an RL job starts, and a thousand hosts all fault on the same brand-new snapshot within seconds. Pushing hot chunks to hosts before the job begins, or letting hosts fetch chunks from each other, keeps object storage off the critical path. The second is that a UFFD handler can fetch from any tier, so a restore can begin before the snapshot has finished downloading. The guest runs on the working set while the long tail of pages streams in behind it.

Huge pages

Guest memory backed by 2 MiB huge pages cuts TLB misses and means one fault brings in 512 times as much memory. For restores that’s a trade: far fewer faults, but each one is larger and touches memory the guest may never use. Firecracker supports huge-page guests, and restoring them goes through the UFFD backend.

Where snapshots break

A snapshot is a frozen CPU. A guest that saw a set of CPU features at boot can crash if it resumes on a host without them, so fleets with mixed hardware use CPU templates to mask features to a common set. Snapshots are also tied to the Firecracker version that wrote them.

A snapshot is also a frozen secret. Every environment restored from one base starts with the same random number generator state, machine ID, and any keys in memory. Linux’s VM generation ID driver reseeds the kernel’s generator when the hypervisor signals a restore, and AWS documents the user-space side for SnapStart, but anything that drew randomness before the snapshot still has to be found and refreshed. For RL that cuts both ways: identical state is a uniqueness bug for networking and a feature for reproducible rollouts.

These are the systems I’m working on and learning as I go, and I’ll add to this page as I understand them better.