Generative Modeling starts with a simple shift in perspective: instead of memorizing individual observations, learn the probabilistic structure that produced them.
If a model can represent that structure, it can use the learned distribution to evaluate data and generate new samples that follow the same underlying patterns.
That gives us the core mental model:
Learn the distribution behind the data, then use that distribution for estimation or generation.
Suppose a dataset contains many dog images.
Storing those images is not enough to create a new dog image that was never part of the training set. To generate new samples, the model needs to capture structure shared across the observed examples and represent that structure as a probability distribution.
We can think of the training data as samples drawn from an unknown Data Distribution:
The exact form of pdata(x) is not known in advance. We only observe samples from it.
Generative Modeling therefore learns a Model Distribution, pmodel(x) , from those observations. New samples can then be drawn from the learned distribution:
Here, pdata(x) represents the distribution followed by real data, while pmodel(x) represents the distribution learned by the model.
The goal is for the Model Distribution to represent the Data Distribution well enough to support both probability evaluation and new sample generation.
A learned Model Distribution can be used in two closely related ways.
Density Estimation focuses on learning a distribution that can evaluate probabilities for data.
If the model captures the structure of real data well, samples similar to real data can receive higher probability, while samples that differ from that structure can receive lower probability.
The important point is that the model is not simply remembering particular training examples. It is representing the broader probabilistic structure shared by the data.
Sample Generation uses the learned Model Distribution to draw new samples.
The generated samples do not need to be identical to anything in the training set. What matters is that they follow the structure represented by pmodel(x) .
The overall flow is:
Observed training data
|
v
Learn the Model Distribution
|
v
p_model(x)
/ \
v v
Density Sample
Estimation Generation
Density Estimation and Sample Generation are therefore two ways of using the same learned distribution.
For a developer, the key structural idea is to identify the distribution being modeled and then see how that learned structure is used for probability evaluation or generation.
Images, text, and audio are represented in high-dimensional spaces. But not every point in those spaces corresponds to meaningful real data.
Consider images. If every pixel value is selected uniformly at random, the result is unlikely to resemble a real image.
The same applies to text. Randomly choosing characters rarely produces meaningful language.
The issue becomes clearer when we think about the full high-dimensional space:
High-dimensional data space
+--------------------------------------+
| |
| enormous set of |
| possible configurations |
| |
| +----------+ |
| | real data| |
| | structure| |
| +----------+ |
| |
+--------------------------------------+
The number of possible images or text sequences can be enormous, while realistic data occupies only a negligible portion of that space.
So generation is not simply a matter of selecting values from a large space. The model has to represent the restricted structure where real data actually exists.
The difficulty is not only that the data lives in a high-dimensional space. The model must represent the complex structure formed by real observations inside that space.
Realistic images are not arbitrary collections of pixel values. Meaningful text is not an arbitrary sequence of characters.
Real data exists in limited, structured regions rather than being spread uniformly across all possible configurations.
This perspective connects naturally to the kind of data structure considered in Manifold Learning.
A naive picture of generation would be:
choose values from a large space
A better mental model is:
learn where realistic data is structured
+
sample according to that learned structure
Representing complex high-dimensional distributions requires sufficient model capacity. Simple models may not be able to capture this complexity, which leads naturally toward Deep Generative Models for representing more complex data distributions.
Generative Modeling matters for more than producing new samples. Once a model has learned structure in the data distribution, that structure can also support other learning problems.
In reinforcement learning, possible futures can be simulated and used for Planning.
In Continual Learning, Generative Replay can reuse previously learned information through generated data.
Distribution learning can also help when part of the data is unobserved. The learned structure can support predictions about missing parts, connecting Generative Modeling with Imputation.
Another important connection is Latent Representation learning. A generative model can capture features shared across different observations in a latent form, and those representations can then serve as general features for other learning problems.
Across these cases, the common idea is that structure learned from the data can be reused beyond direct sample generation.
The distribution perspective becomes especially important when the output must resemble realistic data.
Examples include:
These problems look different, but they share a common requirement: the generated output must reflect the structure learned from real data.
In Super-Resolution, the model takes a lower-resolution input and produces a higher-resolution result. Producing additional values is not enough; the result must follow the learned structure of realistic data.
X-to-X Translation transforms one data representation into another while producing an output consistent with the structure learned from data.
Art Creation follows the same general principle: the model uses learned data structure to produce a new result rather than simply reproducing an observed sample.
The common flow is:
Real data
|
v
Learn its probabilistic structure
|
v
Model Distribution
|
+--> evaluate data probability
|
+--> generate new samples
From an implementation perspective, the central structure is straightforward: determine what distribution is being modeled, understand how the model represents the structured region occupied by high-dimensional data, and see how that learned distribution is used for estimation or generation.
Generative Modeling is fundamentally about learning the distribution behind observed data.
Training examples are sampled from pdata(x) , while generated samples come from the learned pmodel(x) .
Density Estimation and Sample Generation share that learned distribution as their common foundation.
The difficult part is that realistic images, text, audio, and other high-dimensional data occupy only a small, structured portion of the full space of possible values. A generative model must capture that structure rather than sample randomly from the full space.
Learn the structure of the data distribution, then use that structure to evaluate or generate data.
Originally published at zeromathai.com.
Original article: https://zeromathai.com/en/generative-modeling-en/