From jameshan.net

Snapshot Restore for Coding-Agent MicroVMs: Working-Set Prefetch, Diff Snapshots, and Tiered Caches

Coding agents require a full computer that installs packages, builds, and runs tests. Reinforcement learning (RL) on coding tasks consumes thousands of fresh environments an hour, each used once and discarded, so platforms boot once, install the toolchain, warm every cache, and save the entire machine; every later environment is a restore.

This note describes how Firecracker snapshots work and the techniques that make restoring them fast at this scale. A lazy restore resumes the guest in milliseconds but moves the cost into page faults during the first second of execution; most of the techniques below avoid paying for those faults one at a time.

Working-set prefetch. Restores of one snapshot touch largely the same pages, so a recorded set can be read sequentially. With a 100 µs fault, a page touched in more than about 2% of restores is worth prefetching.
Diff snapshots. Saving only dirty pages arranges snapshots into a tree, so forking a running environment for best-of-N sampling or tree search costs one diff.
Tiered chunk caches. Content-addressed chunks in memory, local NVMe, peers, and object storage keep the slowest tier off the critical path; peers reach a thousand hosts in 5 rounds.
Huge pages. A 2 MiB page is cheaper to restore than its 512 small pages once about a dozen are touched.

Snapshot Format

Firecracker is the virtual machine monitor that AWS built for Lambda. Snapshotting a paused microVM writes two files: a small vmstate file containing vCPU registers and device state, and a memory file containing every page of guest RAM. A restore loads both into a fresh Firecracker process, and the guest continues from the instruction at which it was paused.

A paused microVM's vCPU registers and device state are written to a small vmstate file and its guest RAM to a memory file; the block device's copy-on-write layer is frozen alongside; a restore starts a new Firecracker process, loads the vmstate, maps the memory file, and resumes the guest — Snapshot contents and restore sequence. The vmstate file holds vCPU and device state, the memory file holds guest RAM, and the block device is excluded; its copy-on-write layer is frozen alongside.

The disk is not part of the snapshot. The guest’s block device must be exactly as it was at the moment of the snapshot; otherwise the guest’s in-memory view of its filesystem no longer matches the contents of the disk. In practice the disk is a copy-on-write layer over a read-only image, frozen together with the memory file.

Lazy Restore

With the default File backend, Firecracker maps the memory file privately into the new process. Nothing is read in advance: a page is loaded when the guest first touches it, and a write produces a private copy. A 16 GiB guest therefore resumes in milliseconds, and many guests restored from one file on one host share its untouched pages through the host page cache.⁠ The memory file is read into the host page cache on first touch; two restored VMs share those pages read-only, and each page a VM writes becomes a private copy Lazy restore with the File backend. Untouched pages are shared through the host page cache; written pages become private copies.

The cost moves to the first second after resume. The guest touches its working set immediately, each first touch is a page fault, and each fault is a small random read. Against local NVMe this is tolerable; against remote storage it dominates.

Working-Set Prefetch

Restores of the same snapshot touch mostly the same pages in mostly the same order, because the guest resumes the same code from the same state. The access pattern can therefore be recorded once and replayed.

The UFFD backend delivers every page fault to a user-space handler through Linux’s userfaultfd. On a first restore, the handler logs which pages were faulted and in what order. On later restores, it copies that set into guest memory in one sequential pass, before or alongside resume, and falls back to on-demand faults only for pages outside the set. REAP (ASPLOS 2021) measured this approach on Firecracker and reduced cold-start time severalfold; FaaSnap (EuroSys 2022) improved on it by loading the set concurrently with execution rather than before it.⁠REAP is part of the vHive project. Both papers found that the working set is small relative to guest memory and stable across invocations, which is what makes recording it worthwhile.

Let $W$ be the set of pages a restore touches. A lazy restore pays one fault for each:

T_{\text{lazy}} \approx T_{\text{state}} + |W| \, t_{\text{fault}}

A prefetching restore reads the recorded set in one pass at sequential bandwidth $B$ , with page size $s$ , and faults only on the $m$ pages it missed:

T_{\text{prefetch}} \approx T_{\text{state}} + \frac{|W| \, s}{B} + m \, t_{\text{fault}}

When a fault is a round trip to remote storage, the $|W| \, t_{\text{fault}}$ term dominates, and prefetching replaces it with a single large read.

Scatter plots of page faults by guest physical address against time since resume on a log scale. Lazy restore shows a dense band of faults in each memory region from 6 ms to several seconds; prefetch shows a sequential read of the working set from 6 to 66 ms followed by a sparse set of missed pages — Page faults by guest physical address after resume. (a) Lazy restore faults on every page of the working set. (b) Prefetch reads the recorded working set sequentially, then faults only on pages outside it. Illustrative.

A recorded set is a prediction. If page $i$ is touched with probability $p_i$ across restores, prefetching it costs $s/B$ with certainty and saves $t_{\text{fault}}$ with probability $p_i$ . To first order, the page should be prefetched exactly when

p_i \, t_{\text{fault}} > \frac{s}{B} \quad\Longleftrightarrow\quad p_i > \frac{s}{B \, t_{\text{fault}}}

With 4 KiB pages, 2 GB/s of sequential bandwidth, and a 100 µs fault, the threshold is about 2%. A page touched in one restore out of twenty is still worth reading ahead, and the more expensive a fault, the more inclusive the set should be.

Cumulative distributions of time from restore request to first agent command completed, log scale: lazy restore with working-set prefetch near 100 ms, lazy restore near 3 s, cold boot near 7 s, and eager restore of all 16 GiB near 9 s — Time from restore request to completion of the first agent command for a 16 GiB guest on local NVMe, with a 100 µs fault and 2 GB/s sequential bandwidth. Eager restore reads the whole memory file before resuming. Illustrative.

The working set also determines the storage layout: recorded pages should be written contiguously, so that the prefetch is one large read rather than thousands of small ones.

Snapshot Size Reduction

A memory file is the size of guest RAM, most of which is free or holds data that could be regenerated. Three measures keep it small:⁠ Stacked bars of bytes stored per snapshot for a 16 GiB guest: 16 GiB raw, 6.5 GiB after ballooning and zero chunks, 4.7 GiB after cross-snapshot deduplication, and 1.2 GiB after dropping the page cache Bytes stored per snapshot for a 16 GiB guest after each reduction. The last step trades restore speed for size. Illustrative.

Inflate a balloon before snapshotting. The guest hands its free pages to the balloon driver, and they become zero pages in the file.
Store the file as content-addressed chunks. Zero chunks, and chunks identical across snapshots such as an unchanged kernel or a shared toolchain, are stored once.
Decide deliberately whether to keep the guest’s page cache. Dropping it shrinks the snapshot, but keeping it is often the reason for snapshotting, since a warm compiler cache or loaded model weights are what make the restored machine fast.

Diff Snapshots

With dirty page tracking enabled, KVM records every page the guest writes, and a diff snapshot saves only those pages. A diff cannot be used alone; it is applied over the snapshot from which it was taken, and Firecracker’s snapshot-editor can merge the two.⁠Tracking dirty pages adds a small cost to every first write to a page, so it is disabled unless requested.

Diffs arrange snapshots into a tree: a base containing the operating system, a child containing the toolchain, and below that a child with a specific repository checked out at a specific commit. Each level shares everything above it.

This is where coding-agent platforms differ most from serverless workloads. An RL rollout can snapshot an environment mid-trajectory and restore it N times to explore N continuations, and best-of-N sampling and tree search over a coding agent’s actions require the same operation. Branching a running computer becomes as cheap as writing a diff, so the environment, and not only the model’s context, can be forked.

If the guest makes $w$ page writes spread uniformly over $M$ pages, the expected number of distinct dirty pages is

E\,|D| = M\left(1 - \left(1 - \tfrac{1}{M}\right)^{w}\right) \approx M\left(1 - e^{-w/M}\right)

A diff grows almost linearly at first and then saturates toward all of memory, so a diff taken soon after its parent is small and one taken much later is nearly a full snapshot. Real writes concentrate in the heap and page cache, which makes diffs smaller than this model predicts, but the shape holds: snapshots should be taken often, or rebased onto a new full snapshot.

A tree stores each node once. Its total size is $|B| + \sum_v |D_v|$ over every diff $v$ in it, compared with $N\,|B|$ for $N$ independent full snapshots.⁠ $Two panels: expected diff size as a fraction of memory against page writes, saturating for uniform writes and growing more slowly for skewed writes; and total storage against number of snapshots, linear for independent full snapshots and far lower for a snapshot tree$ (a) Diff size under uniform and skewed writes. (b) Storage for a tree of diffs against independent full snapshots. Illustrative.

Tiered Chunk Caches

Snapshots are stored in tiers, from fastest to most durable: pages already in a host’s page cache, chunks on local NVMe, and object storage. AWS’s 2023 paper on loading Lambda container images describes the design most platforms now follow: split images into chunks named by content hash, deduplicate across customers, and place a shared cache tier between hosts and S3.

A single fault on a restored guest takes the following path. KVM traps the access and userfaultfd delivers it to the handler, which maps the page to the chunk containing it, ensures that only one fetch for that chunk is in flight, and queries the tiers in order until one holds it. The data is copied into guest memory with UFFDIO_COPY. In parallel, the prefetcher streams the recorded working set from local disk, so that most pages never fault.

With hit rates $h_1$ and $h_2$ for the first two tiers and latencies $t_1 < t_2 < t_3$ , the expected cost of a fault is

E[t] = h_1 t_1 + (1-h_1)\,h_2\,t_2 + (1-h_1)(1-h_2)\,t_3

Object storage is orders of magnitude slower than local NVMe, so even a small share of faults reaching it determines the average. The objective is therefore less a high hit rate than keeping the last tier off the critical path.⁠ Two panels: fetch latency by tier from 2 µs for the host page cache to 30 ms for object storage; and expected fault cost against the share of faults reaching object storage, dominated by the object-storage term above about 0.1% (a) Fetch latency per tier. (b) Expected fault cost as the share of faults reaching object storage grows, with a first-tier hit rate of 0.9. Illustrative.

Two problems are specific to coding agents. The first is the thundering herd: an RL job starts, and a thousand hosts fault on the same new snapshot within seconds. Without coordination, $N$ hosts restoring one snapshot with a working set of $c$ chunks send $N c$ requests to object storage at once. A regional cache that coalesces identical misses into one fetch sends $c$ . Peer distribution performs better still: if every host holding a chunk serves it to $k$ others, the number of hosts holding it multiplies by $k+1$ each round, and all $N$ hold it after

r = \left\lceil \log_{k+1} N \right\rceil

rounds. A thousand hosts with $k = 3$ are covered in 5 rounds. Pushing hot chunks before the job begins removes even those rounds from the critical path.

The second is that a UFFD handler can fetch from any tier, so a restore can begin before the snapshot has finished downloading. The guest runs on the working set while the remaining pages stream in behind it.

Huge Pages

Backing guest memory with 2 MiB huge pages reduces translation lookaside buffer (TLB) misses, and each fault brings in 512 times as much memory. For restores this is a trade-off: far fewer faults, but each is larger and loads memory the guest may never use.

Consider a 2 MiB region in which the guest touches $k$ of its 512 small pages. With 4 KiB pages the restore pays about $k \, t_{\text{fault}}$ ; with one huge page it pays $t_{\text{fault}} + 512\,s/B$ . Huge pages are cheaper when

k > 1 + \frac{512 \, s}{B \, t_{\text{fault}}}

which, with the parameters above, is about a dozen pages. Dense regions such as the heap favor huge pages, and sparse regions do not.⁠ Two panels: restore cost per 2 MiB region against pages touched, with 4 KiB pages crossing the huge-page cost at about 11.5; and a histogram of pages touched per region, sparse regions below the crossover and heap regions near 512 (a) Restore cost per region; 4 KiB pages become more expensive above about 11.5 touched pages. (b) Pages touched per region at restore. Illustrative. Firecracker supports huge-page guests, and restoring them requires the UFFD backend.

Limitations

A snapshot captures a frozen CPU. A guest that observed a set of CPU features at boot can crash if it resumes on a host without them, so fleets with mixed hardware use CPU templates to mask features to a common set. Snapshots are also tied to the Firecracker version that wrote them.

A snapshot also captures frozen secrets. Every environment restored from one base starts with the same random number generator state, machine ID, and any keys held in memory. Linux’s VM generation ID driver reseeds the kernel’s generator when the hypervisor signals a restore, and AWS documents the user-space side for SnapStart, but any state that drew randomness before the snapshot must still be found and refreshed. For RL this has two consequences: identical state is a uniqueness defect for networking and a benefit for reproducible rollouts.