Privacy (in ML)
Concrete Problems in AI Safety — inherited
An adjacent research area concerned with preserving privacy when applying machine learning to sensitive data sources such as medical data.
Machine learning applied to sensitive records, medical data above all, raises a question of its own: what does a trained model give away about the people in its training set? Privacy research studies that leak, and Concrete Problems in AI Safety (2016) names it a neighbor to accident risk rather than a part of it.
First the prose covers what a model can reveal about its training data; then it explains why this hazard needs no adversary, unlike the field listed beside it.
Protecting what a model reveals about its training data
Privacy in machine learning is a research area concerned with ensuring that applying ML to sensitive data, medical records being the paper's own example, does not leak information about individuals back out through the model's behavior. This concept predates Concrete Problems; the broader field it names includes attacks such as membership inference, determining whether a specific record was in a model's training set from its outputs alone, and defenses such as differential privacy, which bounds how much a randomized computation's output distribution can change due to any single individual's data. Formally, a mechanism $M$ is $(\epsilon, \delta)$-differentially private if for any two datasets $D, D'$ differing in one record and any measurable output set $S$, $$P[M(D) \in S] \le e^{\epsilon}\, P[M(D') \in S] + \delta,$$ giving a quantitative guarantee that no output reveals much about any one person's presence in the data. The paper cites privacy only at the level of the open question, "How can we ensure privacy when applying machine learning to sensitive data sources such as medical data?" (concrete-problems, §"Privacy:", p. 21), without engaging this technical machinery directly.
Why it needs no adversary, unlike its neighbor in the list
Privacy sits in Drawing the boundary: what accident risk is not next to Security (attacks against ML systems), and Privacy (in ML) — needs no adversary unlike → Security (attacks against ML systems) notices a gap in the paper's own parallel one-line definitions: security asks "what can a malicious adversary do to a ML system?" while privacy's definition names no adversary at all. That gap tracks a real methodological difference: a privacy failure, a model leaking training-data details through its ordinary outputs, can happen with no attacker present, just an otherwise well-intentioned system trained carelessly on sensitive data, which is closer to Accidents in machine learning systems's own no-adversary premise than security research's adversarial framing is.