Differential Privacy

What it is, plainly

Differential privacy is a way to share information about a group of people without revealing anything specific about any one person in the group. You add a small amount of random noise to your answers, just enough that anyone looking at the output can’t tell whether or not any specific person was in the dataset to begin with.

Imagine a teacher who wants to tell the class the average test score, but doesn’t want anyone to figure out what one specific student got. So she announces “the average is somewhere between 78 and 82” instead of the exact number. Differential privacy is that idea, made formal and applied to entire databases and machine learning models.

The threat it defends against

A trained machine learning model is a lossy compression of its training data — and sometimes the lossiness isn’t strong enough. The model itself can leak information about the people in its training set, even when the raw data never leaves the building.

• Membership inference: an attacker queries the model and figures out whether a specific person’s record was in the training set, by detecting that the model behaves slightly differently on examples it has seen during training.
• Memorization: large enough models literally regurgitate training data. Carlini et al. (2021) prompted GPT-2 to spit out verbatim phone numbers, emails, and addresses from its training corpus. Carlini (2023) extracted verbatim training images from Stable Diffusion.
• Outlier leakage: models overfit on unusual individuals more than on common ones, so the most privacy-sensitive people (rare conditions, unusual feature combinations) are the most exposed.

Locking the database in a vault doesn’t help. The trained model is the thing you ship, and it can act as a side channel for the records inside it. Differential privacy closes that channel by construction.

The formal promise

Pick any one person in the dataset — call her Alice. The differential privacy guarantee is:

Whether Alice’s record was included in the training set or excluded from it, the model behaves statistically almost identically.

Imagine two parallel universes — Universe A trains on all records including Alice’s, Universe B trains on the same records with Alice’s removed. The two resulting models are nearly indistinguishable to anyone querying them. Same predictions, same confidence scores, within some bounded factor.

The bound is a number called ε (epsilon). Smaller ε = stronger privacy. A relaxed version called (ε, δ)-DP adds an escape hatch δ: the bound holds most of the time, but can fail with probability up to δ (typically 10⁻⁵ to 10⁻⁹).

Formally:

Pr[M(D) ∈ S] ≤ e^ε · Pr[M(D′) ∈ S] + δ

for any neighboring datasets D and D′ (differing in one record) and any output set S.

What ε actually means

ε is on a log scale, not linear. The number you should compute in your head is e^ε:

When someone says “we deployed at ε=10,” they’re saying Alice’s presence vs. absence can shift the model’s output probability by 22,000×. Always mentally compute e^ε — the field’s biggest UX failure is quoting ε without context.

Reference deployments:

• Apple iOS keyboard telemetry (2016 launch): ε ≈ 4–14 per query per day.
• US Census 2020: ε ≈ 19 total across the entire decennial release.
• Research / strong-privacy literature: ε ≤ 1.

The unavoidable tradeoff

Privacy and learning-from-individuals are in direct mathematical opposition.

• ε = 0: perfect privacy. The model is literally identical whether Alice is in or out — which means it learned nothing from her. Useless.
• ε = ∞: no privacy. The model depends arbitrarily on individual records. Useful, no protection.
• ε in the middle: the real regime.

DP doesn’t prevent learning population patterns — if 50 people in the dataset share a rare genetic marker, the model still learns it, because removing any one of them barely changes anything. What dies is the ability to learn anything that only one specific person exemplifies.

DP is the right tool when the goal is to learn population patterns while keeping individuals invisible. It is the wrong tool when the individual is the signal.

The three trust models

Where the noise gets added determines the architecture. The math of DP is the same; the system around it is the consequential design choice.

Central DP — trust the curator

The server collects raw data from every user, holds it, computes the statistic, then adds noise before publishing. The privacy guarantee protects what gets released, not what gets stored. Trust required: the curator deletes the raw data, doesn’t leak it, doesn’t get breached, doesn’t comply with subpoenas.

• Lower noise needed per query (you noise once at the aggregator).
• Used by: US Census Bureau (2020), most internal analytics dashboards, Google’s central analytics.

Local DP — trust nobody

Each user adds noise to their own answer before sending. The server receives noisy reports, none of which is individually trustworthy, but aggregates across millions of users average out. The raw data never leaves the device. Trust required: none.

• Strictly stronger privacy than central DP — fewer assumptions.
• Pays in noise: requires ~√N more noise than central DP for the same final accuracy at scale of N users.
• Used by: Apple (iOS keyboard, Safari crash data, emoji telemetry, health analytics), Google RAPPOR (Chrome telemetry).

Shuffle DP — trust the shuffler

Each user adds a small amount of local noise (less than full local DP requires), then sends to a shuffler — a service distinct from the aggregator — which strips identifiers and randomizes order before forwarding. The shuffling amplifies privacy: hiding the user↔report link is equivalent to having added more noise. Sits between local and central.

• Trust required: the shuffler doesn’t collude with the aggregator.
• Used by: Google (Chrome telemetry pipelines), federated learning systems with anonymous aggregation.

How they compare

The right choice is downstream of the company’s threat model and brand posture. Apple optimizes for “no trust required” because the trust-me-with-your-data story is the entire iCloud + Apple Intelligence value prop. Google optimizes for “central DP plus shuffle where we can” because they already have the data and the question is what they publish.

Applying DP to machine learning training

The standard algorithm is DP-SGD (Differentially Private Stochastic Gradient Descent), introduced by Abadi et al. (2016). Two modifications to vanilla SGD:

1. Per-example gradient clipping — bound the L2 norm of every example’s gradient to a fixed threshold C. This caps how much any one record can influence the gradient.
2. Gaussian noise injection — add noise sized to C and the desired ε to the summed gradient before the optimizer step.

The privacy cost composes across training steps via a privacy accountant (RDP / GDP / moments accountant) that tracks how much ε is spent across the whole training run. Total privacy budget is a finite resource; longer training spends more.

Cost in practice: DP-SGD typically loses 1–5% accuracy vs. non-private SGD on standard benchmarks, sometimes much more on small datasets. Hyperparameter sensitivity is high (clip norm, noise multiplier, batch size, learning rate all interact).

Tools

Most production privacy ML work uses Opacus for training (because PyTorch dominates) and OpenDP / Tumult / PipelineDP for analytics.

Who works on this

What to read next

• Federated Learning — DP’s natural partner when data lives on user devices.
• On-Device Inference — the deployment story that makes local DP architecturally sensible.
• LLM Privacy — how DP and related techniques apply (and fail to apply) to foundation models.