Security (attacks against ML systems)
Concrete Problems in AI Safety — inherited
An adjacent research area concerned with what a malicious adversary can do to attack a legitimate machine learning system.
What can a malicious adversary do to a legitimate machine learning system? That question defines security research, an established field Concrete Problems in AI Safety (2016) mentions precisely to set it outside its own scope, which concerns harms nobody intended.
The prose runs from a field defined by its adversary, through the sense in which it mirrors accident risk one step removed, to the contrast with a neighboring field, privacy, that needs no adversary at all.
An adjacent field defined by an adversary
Security research, in the sense Concrete Problems uses the term, predates the paper as an established area of machine learning research concerned with what a malicious adversary can do to attack a legitimate system, and the paper cites it only to place it outside its own scope: "What can a malicious adversary do to a ML system?" (concrete-problems, §"Security:", p. 21). Real ML security work covers concerns such as evasion attacks, crafting inputs that cause a deployed classifier to misbehave, and poisoning attacks, corrupting training data so the resulting model behaves badly, both of which presuppose an adversary actively working against the system.
A mirror image, one step removed
The paper's own footnote makes security and Abuse (of ML systems) a matched pair, defined by swapping which side of an attack the ML system sits on: security involves attacks against a legitimate ML system by an adversary, for instance a criminal trying to fool a face-recognition system, while abuse involves attacks by an ML system controlled by an adversary, for instance a criminal training a "smart hacker" system to break into a website (concrete-problems, §"Abuse:", p. 21). Both require an adversary and an ML system; what differs is whether the system is the target or the weapon. This is also what separates both most sharply from Accidents in machine learning systems itself, which the paper defines as requiring no adversary at all, only poor design, placing security and abuse at the far end of the adjacent-areas list from the paper's own subject.
Contrast with a neighbor that needs no adversary
Set beside Privacy (in ML), the difference in the paper's one-line definitions is instructive: privacy asks how to protect sensitive data with no adversary named, while security's definition names one explicitly (concrete-problems, §"Security:", p. 21; §"Privacy:", p. 21). A privacy failure, a model that leaks details of its training data through its ordinary outputs, can happen with no attacker present at all, just an otherwise well-intentioned system trained carelessly on sensitive data, which tracks a real methodological difference: security research typically models an opponent's strategy, while privacy and accidents research typically model what a benign system inadvertently does wrong. Security-ml belongs to the theme Drawing the boundary: what accident risk is not, the set of six fields the paper names and deliberately declines to develop.