Training on multiple distributions
Concrete Problems in AI Safety — inherited
An approach of training on several distinct training distributions in the hope that a model performing well across all of them will generalize to a novel test distribution.
Practitioners have long hedged against surprise by training on several different distributions at once, wagering that a model performing well across all of them will cope with a genuinely new one. Concrete Problems in AI Safety (2016) records the habit as one of the plainer responses to distributional shift.
The prose treats the practice first as an engineering wager rather than a statistical assumption, then contrasts it with the covariate shift approach.
An engineering wager instead of a statistical assumption
Training on multiple distributions is the practice of training a model on several distinct training distributions in the hope that a model performing well across all of them will generalize to a genuinely novel test distribution. It is not a technique this paper invented; it reflects a familiar engineering habit already used in production ML systems by the time the paper was written, and one of the authors reports finding it effective specifically in automated speech recognition (concrete-problems, §"Training on multiple distributions.", p. 18). Unlike the other members of its family, the approach asserts no explicit statistical claim about the data-generating process at all — it is offered as an engineering methodology rather than an identification result, with the paper suggesting it could in principle be combined with the more theoretically grounded approaches discussed alongside it.
Contrast with covariate shift
The edge Covariate shift assumption — single invariant contrasts with → Training on multiple distributions places this concept at the opposite end of a spectrum from Covariate shift assumption: covariate shift stakes everything on one explicit, checkable-in-principle claim, that $p(y \mid x)$ does not change, and inherits the paper's own warning that this claim can fail silently; training on multiple distributions makes no claim that could be true or false, and so cannot fail silently in the same way, but also offers no theoretical account of why or when it should work. The paper's own evidence for the approach is a practitioner's empirical report, not a proof. It stresses one methodological point that carries real safety weight regardless: it is important to detect when a new input falls outside the distributions actually trained on, and to stress-test the resulting model against distributions substantially different from anything it saw during training (concrete-problems, §"Training on multiple distributions.", p. 18). The concept is a child of Robustness to distributional shift, sits in the Weakening statistical assumptions to survive distribution shift theme and the One umbrella idea, aimed at different targets each time it's used connective theme, and belongs to the Distributional shift: relax the statistics, or rewrite the contract supertheme as the family's most assumption-free, most purely empirical member.