Generative Modeling: From Data Distributions to Deep Generative Models

A developer explains that generative models are best understood not as networks that create images but as solutions to a shared underlying problem: representing, learning, and inferring the probability structure of high-dimensional data. The post organizes the field around three interacting problems—representation, learning, and inference—and shows how autoregressive models, VAEs, flow-based models, GANs, and diffusion models all respond to the same tension. It also connects generative modeling to inverse problems and posterior inference, providing a unified mental model for developers.

If you approach generative models as ""networks that create images,"" the field quickly turns into a collection of disconnected architectures. A more useful developer mental model starts one level lower: What probability structure could have produced the data, and how can we represent, learn, and infer that structure without making the computation impossible? That question connects autoregressive models, VAEs, flow-based models, GANs, and diffusion models. Their architectures look very different, but they all respond to the same underlying tension: high-dimensional data distributions are difficult to represent, learn, normalize, sample from, and reason about. Generative modeling can therefore be organized around three interacting problems: Once these three pieces are connected, the major families of deep generative models become much easier to understand. A discriminative model usually begins with a prediction problem. Given an input x , predict the most likely output y : The model focuses directly on the conditional relationship required for prediction. A generative model asks a broader question. Instead of learning only the path from x to y , it models the probability structure from which the data arises. For class-conditional modeling, for example, we can model p x∣y together with the prior p y and recover the posterior using Bayes' rule: In unsupervised generative modeling, the target becomes the data distribution itself. We assume the training samples come from some unknown distribution: The model then constructs a parameterized distribution intended to approximate it. This perspective connects tasks that can otherwise seem separate: Density estimation asks how probable an observation is under the model. Generation draws new samples from the learned model distribution. Unsupervised representation learning can capture latent structure shared by the data while learning that distribution. All three begin with the same underlying goal: learning the probability structure of the data. Real data usually lives in a high-dimensional space. An image, for example, is described by many variables, while valid observations occupy only a limited and complicated region among all possible combinations. So the first problem is not simply ""use a neural network."" It is: How do we represent the joint distribution of many random variables compactly enough to compute with, while keeping enough flexibility to describe the data? This connects naturally to inverse problems. A forward problem moves from a cause x to an observation y . An inverse problem starts from an observed y and asks which possible x could have produced it. Conditional generation extends the same idea by generating possible x values consistent with a given condition. Forward: cause x ────────── observation y Inverse: possible cause x <────────── observation y Conditional generation: condition y ────────── possible samples x Recovering a high-resolution image from a low-resolution one, estimating color from grayscale data, reconstructing missing regions, or recovering a CT slice from a sparse-view sinogram all share this structure. The difficulty is that the inverse may not be unique. A single observation can be consistent with multiple possible causes. Inverse problems are therefore not simply about running a function backward; they require reasoning over possible causes that are compatible with the observation. Suppose a model contains an observed variable x and a hidden variable z . The generative direction models the relationship between z and x . Once x has been observed, the reverse question is which values of z could have produced it. That is posterior inference: Posterior inference can require both computing the posterior distribution and calculating expectations under it. The E-step of the EM algorithm is a direct example: Here the posterior over the unobserved z is used to compute the expected complete-data log-likelihood. As the latent space grows and interactions among hidden variables become more complicated, direct posterior computation can become intractable. Increasing model expressivity therefore leads naturally to another question: how can the required inference remain computationally feasible? more expressive representation ↓ more complicated latent structure ↓ harder posterior inference ↓ need for approximate inference Two major approaches are MCMC and Variational Inference. Both approximate a posterior that is difficult to compute directly, but the approximation arises in different ways. MCMC constructs an ergodic Markov chain whose stationary distribution is the target posterior. Samples generated by running the chain are then used to approximate the posterior and the expectations we need. target posterior ↓ construct Markov chain ↓ run the chain ↓ collect samples ↓ approximate posterior expectations The practical approximation comes from finite computation. In real problems, only a limited amount of time is available for generating samples, and exploring the relevant probability regions can be expensive in high-dimensional or complex models. MCMC therefore has a finite-time sampling approximation in practice. Variational Inference makes a different tradeoff. Instead of sampling directly from the posterior, it defines a tractable family of distributions Q and searches within that family for a distribution close to the true posterior: This turns posterior inference into an optimization problem. The tractability comes from restricting the search space. If the true posterior cannot be represented by the chosen variational family, more optimization time alone cannot eliminate the mismatch. The contrast is useful: MCMC approximation source: finite-time sampling Variational Inference approximation source: restricted distribution family This distinction becomes concrete in a VAE. Rather than computing a complicated latent posterior directly, the model uses a variational distribution, with the Encoder acting as a Recognition Network that constructs the approximation. The Decoder generates observations from the latent variable. An abstract inference problem has now become part of a trainable model architecture. Representation and computation collide when we try to model realistic high-dimensional distributions. A simple probability model can be easy to normalize and evaluate but may lack enough flexibility to describe complex data. A deep neural network can represent much richer functions, but its output is not automatically a valid probability density. A probability density must be nonnegative and integrate to one. One way to construct such a density is: where Exponentiation gives a positive quantity, while the normalizing constant Zθ ensures that the density integrates to one. For a complex neural function over a high-dimensional space, however, computing that integral can itself become intractable. simple probability model → tractable → limited flexibility deep neural representation → flexible → normalization or inference may become intractable This is the Tractability-Flexibility Tradeoff . It provides a useful way to compare deep generative models. Rather than asking only which network architecture a model uses, ask which difficult probability computation its structure makes manageable. In likelihood-based generative modeling, learning can be viewed as reducing the difference between the data distribution and the model distribution: Expanding this expression separates a term that depends only on the data distribution from a term containing the model log probability. The data-only term is independent of θ , so it does not change as the model parameters are optimized. The learning objective therefore connects to increasing the model's log probability on observed data. With a finite training set, this gives the progression: minimize KL divergence ↓ minimize cross-entropy ↓ minimize negative log-likelihood The challenge is to make pmodel x;θ expressive enough for high-dimensional data while keeping the required probability computations feasible. This is where the major generative model families diverge. An autoregressive model handles a high-dimensional joint distribution by decomposing it into sequential conditional distributions: Instead of directly modeling one complicated joint distribution, the model works with a product of conditional probabilities. The key structural idea is factorization . The model represents the high-dimensional joint distribution through a sequence of conditional distributions that can be modeled in a computationally manageable form. A latent-variable model can express the observed distribution as: This provides a latent space for representing hidden structure, but it also introduces the posterior problem p z∣x . A VAE connects the pieces this way: observed x ↓ Encoder / Recognition Network ↓ variational approximation to posterior over z ↓ latent z ↓ Decoder / Generative Network ↓ generated or reconstructed x Training connects this architecture to Variational Inference through maximizing the ELBO: latent-variable representation ↓ intractable posterior ↓ variational approximation ↓ Encoder ↓ ELBO-based learning This is why a VAE is more than an Encoder followed by a Decoder. Its structure turns posterior intractability and variational approximation into a trainable generative model. Flow-based models start from a simple distribution and apply a sequence of invertible transformations to reach a more complex target distribution. Invertibility is the critical structural constraint. Because the transformations can be reversed, probability density can be tracked through them, allowing the model to train with NLL. architectural restriction ↓ invertible transformation ↓ density remains computable ↓ likelihood-based training becomes possible The constraint is part of the solution to the tractability problem: it introduces a structural restriction while keeping density evaluation possible. GANs make a different choice. Instead of directly computing an explicit density function or normalizing constant, they learn the generation process itself. The Generator tries to produce samples resembling real data, while the Discriminator tries to distinguish generated samples from real ones. The two networks train with opposing objectives in a minimax problem. latent input ↓ Generator ↓ generated sample ──┐ ├── Discriminator real sample ───────┘ GANs use this adversarial minimax process to construct an implicit generative model rather than directly evaluating an explicit model density. Diffusion models take another route. Rather than directly handling the transformation between a simple distribution and a complex data distribution in one step, they connect the two through a sequence of noise levels. The forward process gradually adds Gaussian noise to real data: As the process continues, the original data structure gradually disappears and approaches a simple isotropic Gaussian distribution. Generation runs in the opposite direction: data ↓ add noise ↓ add more noise ↓ ... ↓ Gaussian-like noise generation: Gaussian noise ↓ learned reverse step ↓ learned reverse step ↓ ... ↓ data-like sample Instead of using one direct transformation, diffusion models learn a reverse process that progressively reconstructs data structure from noise. Score-based generative models can describe this process using the score function: The score indicates the direction in which the log probability increases. Two representative approaches are SMLD and DDPM. SMLD , or Score Matching with Langevin Dynamics, estimates the score at multiple noise scales and uses Langevin Dynamics to sample while the noise scale decreases. DDPM , or Denoising Diffusion Probabilistic Model, learns a probabilistic reverse model for the steps of the forward noise-corruption process. Their emphasis can be summarized as: SMLD: score estimation + Langevin Dynamics DDPM: forward corruption steps + learned reverse probability model The details differ, but both approaches connect a simple noise distribution to a complex data distribution through multiple noise levels or steps. That iterative structure also creates a practical tradeoff. Because generation requires multiple diffusion steps, sampling speed can become an important limitation. The major generative model families are easier to compare when viewed as different computational responses to the same high-dimensional distribution problem: The normalizing constant fits into the same picture. Energy-Based Models work with energy-based probability distributions in which the normalizing constant must be handled or approximated. Autoregressive models, flow-based models, and VAEs use computationally manageable structural constraints or model constructions. GANs avoid direct computation of explicit density and its normalizing constant by learning the generation process, while score-based diffusion uses the score rather than directly computing the normalizing constant. Different architectures make different choices, but the underlying problem is the same: preserve enough flexibility to represent complex data while keeping generative modeling computationally feasible. When you encounter a generative architecture, start with three questions rather than its layer diagram. Representation: What distribution or latent structure is the model trying to represent? Learning: What objective reduces the difference between the model and the observed data distribution? Inference: What hidden variables or generating processes must be inferred from observations, and can those quantities be computed directly? Then look at the structural choice the model uses to keep the relevant computations manageable: high-dimensional data distribution ↓ representation must be expressive ↓ probability or posterior computation becomes difficult ↓ choose a computational strategy ↓ factorize approximate restrict avoid explicit density or learn a multi-step reverse process No single family removes every tradeoff. In the comparison considered here, VAEs and flow-based models can show weaknesses in generation quality, GANs can face difficulties with generation diversity, and diffusion models can pay for their iterative generation process with slower sampling. So the useful comparison is not simply ""Which model is better?"" It is: Which computational problem does the model solve through its structural choice, and what does that choice cost in quality, diversity, or speed? Generative modeling is not fundamentally about producing realistic-looking samples. It is about learning an unknown high-dimensional data distribution while keeping Representation, Learning, and Inference computationally manageable. Autoregressive models, VAEs, flow-based models, GANs, and diffusion models look different because they make different structural choices around the same underlying problem. Once you identify whether an architecture factorizes a distribution, approximates an intractable quantity, imposes a structural constraint, avoids explicit density evaluation, or learns a multi-step reverse process, the landscape of deep generative models becomes much easier to reason about. Originally published at zeromathai.com. Original article: https://zeromathai.com/en/gan-en/ https://zeromathai.com/en/gan-en/