Covariate shift assumption
Concrete Problems in AI Safety — inherited
A prior assumption (Shimodaira 2000) that p(y|x) is unchanged between training and test distributions, allowing importance-weighted reuse of training data under a shifted p(x).
When the world a model faces drifts away from the world it trained on, one classical assumption offers a foothold: the inputs may change, but the relationship between input and label stays fixed. That is the covariate shift assumption, which Shimodaira formalized in 2000 and Concrete Problems in AI Safety (2016) inherits.
One section explains reweighting training data under a fixed conditional; a second explains why the paper treats the assumption warily.
Reweighting training data under a fixed conditional
The covariate shift assumption, due to Shimodaira (2000), addresses the case of prediction tasks where an input $x$ and output $y$ are drawn from a training distribution $p_0$ and a test distribution $p^*$ that differ. The assumption asserts that only the marginal input distribution changes while the conditional relationship between input and label is preserved: $p_0(y \mid x) = p^*(y \mid x)$. Under this assumption, each training example $(x,y)$ can be reweighted by the importance ratio $w(x) = p^*(x)/p_0(x)$, so that the expected loss under the test distribution can be estimated from training data alone, $$\mathbb{E}_{p^*}[\ell(x,y)] = \mathbb{E}_{p_0}\!\left[\frac{p^*(x)}{p_0(x)}\,\ell(x,y)\right],$$ and a model can even be retrained to directly minimize this reweighted objective (concrete-problems, §"Well-specified models: covariate shift and marginal likelihood.", p. 17). The technique requires estimating both $p_0(x)$ and $p^*(x)$ well enough to compute the ratio, and it degrades sharply — the paper notes the importance weights can have "very large or even infinite" variance — whenever $p_0$ and $p^*$ are far apart.
Why the paper treats it warily
Concrete Problems flags the covariate shift assumption as "very strong and also untestable," and calls the untestability "particularly problematic from a safety perspective, since it could lead to silent failures in a machine learning system" (concrete-problems, §"Well-specified models: covariate shift and marginal likelihood.", p. 17) — a system can satisfy every check available to it while resting on an assumption that happens to be false, and nothing in the pipeline will announce the fact. That single sentence motivates two edges attached to this concept. In Covariate shift assumption — untestability motivates retreat to → Partially specified models, the untestability itself is what pushes the paper toward Partially specified models, which commit to checkable assumptions about only part of a distribution rather than one sweeping, unverifiable invariant across all of it. In Covariate shift assumption — single invariant contrasts with → Training on multiple distributions, covariate shift is contrasted with Training on multiple distributions, which asserts no statistical claim at all and so cannot fail silently in the same way, at the cost of having no theoretical account of when it should work. Both edges sit in the Weakening statistical assumptions to survive distribution shift theme, and the concept is a child of Robustness to distributional shift within the Distributional shift: relax the statistics, or rewrite the contract supertheme, occupying the "maximal theoretical commitment" end of that theme's spectrum of responses.