When developers first encounter deep generative models, it is easy to reduce them to one idea: models that generate new data.
That is only the visible outcome. The deeper problem is deciding how to represent a complex data distribution in a form that a model can compute and learn.
High-dimensional data such as images and audio can follow extremely complex distributions. A single simple probability model is not enough to represent them directly, so different generative model families restructure the problem in different ways.
An Autoregressive Model factorizes a joint distribution into conditional distributions. A Variational Autoencoder (VAE) introduces latent variables. A Flow-based Model starts from a simple distribution and transforms it through invertible mappings. A Generative Adversarial Network (GAN) takes a different route and learns a generation process without directly evaluating an explicit probability density.
A useful mental model is:
Deep generative modeling is not just about producing samples. It is about turning a difficult distribution-modeling problem into something the model can compute and learn.
One useful distinction is between Likelihood-based and Likelihood-Free approaches.
Likelihood-based models include:
GAN is a representative Likelihood-Free model.
This distinction matters because it reveals the structural choice behind each model family. A complex distribution can be decomposed into conditional distributions, represented indirectly through latent variables, transformed from a simpler distribution, or learned without directly computing its probability density.
Diffusion Models and Score-based Models also belong to the Likelihood-based side of this classification. Their detailed mechanisms, including Forward Diffusion, Reverse Diffusion, and Score Matching, are outside the scope here.
For Likelihood-based modeling, a natural goal is to make the model distribution close to the data distribution.
Let the true data distribution be pdata and the model distribution parameterized by θ be pmodel .
We can express that goal by minimizing KL Divergence:
Expanding the KL Divergence gives:
The first term depends only on the data distribution. If pdata is fixed, changing θ does not affect it.
The model-dependent part is therefore:
This is the Cross-Entropy term between the data distribution and the model distribution.
In practice, we do not know the full underlying data distribution, so the expectation is approximated using training samples. The resulting objective is Negative Log-Likelihood (NLL) minimization.
The optimization flow is:
Minimize KL Divergence
↓
Remove the entropy term independent of θ
↓
Minimize Cross-Entropy
↓
Minimize NLL on training data
The intuition is simple: assign high probability to regions where real data occurs.
This connection is the basic picture when the model provides a way to work explicitly with Likelihood. When p(x) becomes difficult to optimize directly, the model needs a different structure.
That is where the major generative model families begin to diverge.
An Autoregressive Model starts with a straightforward idea: if the full joint distribution is difficult to model at once, break it into a sequence of conditional distributions.
The Probability Chain Rule gives an exact factorization:
This is not an approximation introduced for convenience. It is an exact decomposition of the joint distribution.
The calculation proceeds through conditional terms such as p(x1) , then p(x2∣x1) , then p(x3∣x1,x2) , continuing until p(xn∣x1,…,xn−1) .
That changes the modeling problem. Instead of asking how to represent one complicated p(x) directly, we ask how well the network can model each conditional distribution.
GPT, RNN, PixelRNN, PixelCNN, WaveNet, NADE, and MADE use different data and network structures, but they share this conditional factorization principle.
From an implementation perspective, the important thing to inspect is the ordering and conditioning structure. Each prediction must depend on the variables that precede it in the chosen factorization.
A VAE restructures the problem differently.
Instead of factorizing the observed variables sequentially, it introduces an unobserved latent variable z and models the observed data x through that latent state.
The data distribution is written as:
Here, z is the latent variable, p(z) defines its distribution, and p(x∣z) describes how an observation is generated from a particular latent state.
The structural flow is:
Latent state
↓
Conditional generation
↓
Observed data
In probabilistic terms, the Decoder models p(x∣z) .
The VAE also needs a way to infer which latent states are plausible for a given observation. Its Encoder, also called the Recognition Network, does not simply map x to one fixed latent value. It constructs an approximate distribution over z given x .
The Decoder, or Generative Network, models the conditional distribution of x given z .
The difficulty is that the exact Posterior over z given an observation can be hard to compute. VAE addresses this with Variational Inference and trains by maximizing an ELBO.
The important point here is not the full ELBO derivation. It is the structural move: VAE combines latent-variable modeling with approximate inference so that a difficult p(x) becomes a learnable problem.
Normalizing Flow takes another approach.
Where an Autoregressive Model decomposes a complex p(x) into conditional distributions, a Flow-based Model begins with a simple distribution and changes its shape through a sequence of transformations.
The structure is:
Simple Distribution
↓
Invertible Transformation
↓
Invertible Transformation
↓
...
↓
Target Distribution
The key requirement is invertibility.
It is not enough to map values from a simple distribution into a more complicated one. The transformation must also allow the model to trace a transformed value back to where it came from.
That property keeps changes in probability density computable through the sequence of transformations.
This requirement also constrains the model architecture. A Flow-based Model cannot freely use arbitrary transformations if they violate invertibility.
The benefit is that the final data variable x has an explicit probability density. Its Likelihood can therefore be computed, and training can directly optimize NLL on the observed data.
A useful implementation-level mental model is:
Preserve invertibility so that probability density remains computable while the distribution becomes progressively more complex.
Saying that a Flow-based Model models the data distribution does not mean the learned distribution becomes identical to the true data distribution. It means the architecture defines an explicit model distribution p(x) whose Likelihood can be computed and optimized.
GAN changes the setup more substantially.
Instead of putting the explicit value of p(x) at the center of training, GAN introduces two competing models:
Their opposing objectives form a Minimax Problem.
The structural view is:
Generator
↓
Generated samples
↓
Discriminator
↙ ↘
Real Generated
data data
As the Generator produces more convincing samples, the Discriminator has a harder time separating generated data from real data. As the Discriminator improves, the Generator is pushed to produce better samples.
The important difference from an Autoregressive Model or a Flow-based Model is that GAN training does not center on directly computing an explicit p(x) for each observation.
Instead, the interaction between the Generator and Discriminator provides the learning signal that pushes the Generator toward producing samples similar to those from the real data distribution.
That is why GAN is treated as a Likelihood-Free approach in this comparison.
The most useful way to compare these models is not by memorizing architecture names, but by asking how each one turns a complex distribution into a problem the model can compute and learn.
| Model | Structural move | Core training view |
|---|---|---|
| Autoregressive Model | Factorize p(x) into conditional distributions | Model each conditional probability |
| VAE | Introduce latent variable z | Approximate inference and optimize ELBO |
| Flow-based Model | Apply invertible transformations | Explicit density and NLL |
| GAN | Avoid direct explicit density evaluation | Generator–Discriminator Minimax learning |
These approaches begin with the same fundamental difficulty: complex high-dimensional data distributions are hard to model directly.
What changes is the structure used to make learning possible.
When reading the implementation of a generative model, do not start with the layer names.
Start with the probability structure. Check whether the model factorizes the distribution, introduces latent variables and approximate inference, preserves density through invertible transformations, or learns through an adversarial objective without directly evaluating an explicit density.
Once that structural choice is clear, the architecture, objective, and training procedure become much easier to understand as parts of the same generative-modeling problem.
Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/deep-generative-models-course-en/