A new review paper shows that noise contrastive estimation, bridge sampling, and importance sampling are the same algorithm in disguise, and uses that insight to build better estimators for energy based models.
- Review paper unifies NCE, RLR, MIS, bridge sampling, and Bennett's acceptance ratio for energy based models .
- Shows these estimators are algebraically equivalent under specific bridge function and sample ratio choices.
- Explains NCE's practical success as an approximation to the variance minimizing bridge estimator.
- Numerical experiments across ideal and realistic regimes show no single estimator dominates across proposal widths.
- Theoretically optimal bridge estimator surprisingly posts highest MSE in ideal recursion free setting.
- MATLAB code released; framework enables new hybrid estimators with tunable bias variance trade offs.
Energy-based models assign each input an unnormalized score, often through fθ(x) = exp(-Eθ(x)). Converting that score into a probability requires a partition function that is usually too expensive to calculate. A review paper by Luca Martino shows that several established workarounds share the same mathematical structure.
Martino connects noise-contrastive estimation (NCE), reverse logistic regression (RLR), Bennett’s acceptance ratio, multiple importance sampling (MIS), and bridge sampling. The paper identifies conditions under which their estimators are algebraically equivalent, derives additional weighting schemes from the shared framework, and releases MATLAB code for its numerical experiments.
Why one integral stalls training #
An energy-based density has the form pθ(x) = fθ(x) / Zθ, where Zθ = ∫ fθ(x) dx. The model can evaluate fθ(x) at any chosen point, but computing Zθ requires integrating across the full input space. That integral becomes intractable for most high-dimensional models.
The unknown normalizer complicates several operations. Evaluating a normalized likelihood requires Zθ, while maximum-likelihood gradients require expectations under the model distribution, which developers usually approximate with Monte Carlo samples. Because Zθ changes with the model parameters, estimation errors can also propagate through training.
Several research communities address this problem through different formulations. NCE fits a classifier that distinguishes observed data from samples drawn from a known noise distribution. RLR uses logistic regression to estimate ratios of normalizing constants. Bridge sampling and Bennett’s acceptance ratio exploit overlap between a target and a reference distribution, while MIS combines weighted samples from one or more proposals.
Five names, one objective #
Martino expresses the methods as minimizers of a common contrastive cost built from weighted samples associated with two distributions. One distribution contains the unknown normalizer; the other acts as a tractable proposal, noise source, or reference. Changing the weighting rule recovers NCE, RLR, MIS, and established bridge estimators.
This story is for Pro members #
You've reached the end of the free preview. Upgrade to AlphaSignal Pro to read the full article - and everything else behind the paywall.