From Machine Learning

Privacy and ML: Index

First created May 31, 2026 Last edited Jun 21, 2026

Notes on privacy and machine learning. I worked on Firefox’s privacy team for a while, shipping a model that runs on the user’s own device, and most of what I understand about this field I understand through that build and the things it forced me to read. These pages are that: what I ran into, what turned out to be harder than it looked, and the parts I am still working out. Not a survey, and not a textbook. Just notes.

The thing nobody tells you going in is that privacy in ML is not one technique with a clean guarantee. It is a pile of partial defenses against a pile of partly-overlapping attacks, and the words people use to talk about it (“private,” “on-device,” “anonymized”) paper over distinctions that turn out to decide everything. A few of the lessons that cost me the most:

“On-device” is not the same as “private,” and I had to learn where the seam is. Running the model on the user’s machine means the raw data never leaves, which kills a whole class of problems. It does nothing about what the model itself leaks, or what you do with the output once it exists. At Firefox the output was an aggregate the user saw, so this stayed simple. The moment you train on the device’s data and send anything back, even a gradient, the simplicity is gone and you are in the actual hard part of the field. On-device buys you a lot and then quietly hands the bill to the training story.

The privacy is in the deployment, not the proof. The clean version of every technique here is expensive: differential privacy adds noise that costs accuracy, local DP pays a √N tax, federated learning pays to ship gradients off weak devices. A guarantee you cannot afford to run protects no one. My whole Firefox project came down to this. The accurate approach was too large to ship inside a browser, and the version that actually protected users was the one small and cheap enough to fit the budget. The math was never the bottleneck. Fitting an honest answer inside the constraints was.

The trust models are usually drawn as a tidy spectrum, and the tidiness is a lie. Central, local, shuffle, federated, on-device. In a diagram they line up from “trust the platform” to “trust no one.” In practice they leak into each other and compose badly: shuffle DP’s amplification depends on the shuffler not colluding, federated learning leaks through the gradients it was supposed to protect, “anonymized” aggregates fall to reconstruction. Picking a box on the spectrum is the start of the problem, not the answer to it.

I have written the pieces I have a real opinion on so far. More as I build out the federated and gradient-DP side, which is where I am headed next.

Notes so far

  • Differential Privacy — the ε/δ guarantee and what it actually promises, the trust models and where they break, DP-SGD, and the Firefox case as the version of this where you never collect the data at all.
  • On-Device Inference — the column I actually built in. What “on-device” buys you and what it quietly does not, and why efficiency, not math, is what decides whether the private version ships.
  • Federated Learning — training without collecting the data, the direction past on-device inference. Why the gradients leak, the efficiency tax on real devices, and where I am headed next.
  • Privacy Attacks on ML Models — what a trained model leaks about its training set, and why it is the reason every defense here exists.

What I am reading

The papers I am working through to go from “shipped one on-device privacy system” to actually doing research on the federated and gradient-DP side. Roughly the order I would hand someone starting here.

Differential privacy, the foundations

  • Dwork, McSherry, Nissim, Smith, Calibrating Noise to Sensitivity in Private Data Analysis, TCC 2006. The paper that defines DP and the Laplace mechanism.
  • Dwork and Roth, The Algorithmic Foundations of Differential Privacy, 2014. The textbook. Composition, the mechanisms, the accounting.
  • Mironov, Renyi Differential Privacy, CSF 2017. The accounting framework most modern DP training actually uses.

DP for deep learning

  • Abadi et al., Deep Learning with Differential Privacy, CCS 2016. DP-SGD: per-example gradient clipping, Gaussian noise, the moments accountant. The thing the efficiency tax is paid on.
  • De, Berrada, Hayes, Smith, Balle, Unlocking High-Accuracy Differentially Private Image Classification through Scale, 2022. Evidence the privacy-utility gap shrinks with scale and tuning, not just more noise.

Federated learning

  • McMahan, Moore, Ramage, Hampson, Aguera y Arcas, Communication-Efficient Learning of Deep Networks from Decentralized Data, AISTATS 2017. FedAvg, the original.
  • Kairouz, McMahan, et al., Advances and Open Problems in Federated Learning, 2021. The survey that maps the whole problem space, including where it is unsolved.
  • Bonawitz et al., Practical Secure Aggregation for Privacy-Preserving Machine Learning, CCS 2017. The protocol that lets the server see only the sum of updates, never one device’s.

The attacks that make the defenses necessary

  • Shokri, Stronati, Song, Shmatikov, Membership Inference Attacks Against Machine Learning Models, S&P 2017. The shadow-model membership inference attack.
  • Carlini, Chien, Nasr, Song, Terzis, Tramer, Membership Inference Attacks From First Principles, S&P 2022. The likelihood-ratio (LiRA) attack, far stronger and the current bar.
  • Carlini et al., Extracting Training Data from Large Language Models, USENIX Security 2021. Verbatim memorized data pulled back out of a model.
  • Carlini et al., Extracting Training Data from Diffusion Models, USENIX Security 2023. The same, for image models.
  • Zhu, Liu, Han, Deep Leakage from Gradients, NeurIPS 2019. Reconstructing training examples from gradients alone, which is why federated learning is not private on its own.

Local DP and on-device, the deployed end

  • Erlingsson, Pihur, Korolova, RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response, CCS 2014. Google’s local-DP telemetry in Chrome.
  • Apple Differential Privacy Team, Learning with Privacy at Scale, 2017. Apple’s production local-DP deployment.
  • Erlingsson et al., Amplification by Shuffling, SODA 2019. The shuffle model that sits between local and central DP.

I will fold in the systems-side efficiency papers (MLSys / ASPLOS / S&P) as I work through them, since that intersection is the part I most want to contribute to.

Index

  • Differential Privacy. Reference page on differential privacy — what it is, the ε/δ definition, the three trust models (local / central / shuffle), and the tools used to apply it to machine learning.
  • On-Device Inference. Notes on running ML on the user device instead of a server, from building one. The privacy you get for free, the part you do not, and why efficiency is the whole game.
  • Federated Learning. Notes on training models without collecting the data, the direction past on-device inference, and the efficiency and leakage problems I am still working through.
  • Privacy Attacks on ML Models. Notes on what a trained model leaks about its training data, and why this is the thing that makes all the defenses necessary in the first place.