Weakening statistical assumptions to survive distribution shift
A model trained on one distribution of data will eventually face another, and the honest response is to stop assuming so much about either one. This theme covers a family of remedies that work by relaxing exactly which statistical assumptions have to hold across training and deployment, trading certainty for a guarantee that still works when the world shifts.
Unsupervised risk estimation pushes the idea furthest, estimating a model's error on new data using no labels at all, and the page builds up to it from milder relaxations like the covariate shift assumption.
One family of responses to distributional shift works by relaxing exactly which statistical assumptions a model needs to hold across training and test. The covariate shift assumption licenses reusing training data under importance weighting by assuming only the input distribution moves while the conditional relationship between input and label stays fixed. Partially specified models, drawn mainly from econometrics, go further, requiring assumptions about only some aspects of a data distribution rather than a fully specified model, with the generalized method of moments identifying parameters from moment conditions alone. Unsupervised risk estimation pushes the relaxation to its limit, estimating a model's labeled risk on a test distribution using only unlabeled test data by positing structure in how errors are distributed. Training on multiple distributions takes an empirical rather than assumption-driven route, hoping a model that performs well across several distinct training distributions will generalize to a genuinely novel one. This theme's claim is that these are all variations on the same move: give up a little statistical certainty in exchange for validity under shift.