---
title: "Empirical Calibration and Metric Differential Privacy in Language Models"
license: CC-BY-4.0
license_url: https://creativecommons.org/licenses/by/4.0/
license_note: "arXiv abs page shows a CC BY 4.0 icon (by-4.0.png) and a view-license link to http://creativecommons.org/licenses/by/4.0/, confirmed 2026-10-18. This file is an original summary, not the abstract."
source: https://arxiv.org/abs/2503.13872
source_license: "CC BY 4.0"
attribution: "Summary by iinventedlight (AI-assisted). Source: Pedro Faustini, Natasha Fernandes, Annabelle McIver, Mark Dras, Empirical Calibration and Metric Differential Privacy in Language Models (CC BY 4.0)."
date: 2026-10-18
status: published
category: arxiv
tags: privacy, personal-data, language-models
arxiv_id: 2503.13872
checked_on: 2026-10-18
sources:
  - id: S1
    title: "Empirical Calibration and Metric Differential Privacy in Language Models"
    authors: "Pedro Faustini, Natasha Fernandes, Annabelle McIver, Mark Dras"
    url: https://arxiv.org/abs/2503.13872
    accessed: 2026-10-18
    license: CC BY 4.0
    attribution: "Pedro Faustini, Natasha Fernandes, Annabelle McIver, Mark Dras, Empirical Calibration and Metric Differential Privacy in Language Models (CC BY 4.0)."
---
# Empirical Calibration and Metric Differential Privacy in Language Models

License: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/). Source: https://arxiv.org/abs/2503.13872

## Summary
Faustini, Fernandes, McIver, and Dras look at how language-model papers report differential privacy. Training usually follows DP-SGD, and the privacy claim is a budget epsilon. They argue epsilon has no standalone meaning, so two methods cannot be ranked by comparing those numbers. Image work has used membership-inference attacks to put noise levels on a common scale. On text, they find membership inference barely helps that calibration, while attacks that try to reconstruct the hidden example are more informative. As a case study they define a directional, metric privacy mechanism from the von Mises-Fisher distribution: it perturbs the angle of a vector instead of adding the same Gaussian noise in every direction. Formal epsilons still cannot be lined up, but the empirical attacks show each mechanism winning on a different utility-versus-privacy slice. This note follows the abstract page only.

## Why it is here
"Keep personal data safe" is not only a chatbot habit. When a model is trained on people's text, a published epsilon is easy to over-read. This paper says the number is not comparable across methods, and names reconstruction attacks as a more useful check than membership inference for language models.