Generative Modeling: From Data Distributions to Deep Generative Models

If you approach generative models as ""networks that create images,"" the field quickly turns into a collection of disconnected architectures. A more useful developer mental model starts one level lower: What probability structure could have produced the data, and how can we represent, learn, and infer that structure without making the computation impossible? That question connects autoregressive models, VAEs, flow-based models, GANs, and diffusion models. Their architectures look very different, but they all respond to the same underlying tension: high-dimensional data distributions are difficult to represent, learn, normalize, sample from, and reason about. Generative modeling can therefore be organized around three interacting problems: Representation: How do we represent a complex high-dimensional joint distribution? Learning: How do we make the model distribution approach the data distribution? Inference: Given an observation, how do we reason backward about hidden variables or the process that generated it? Once these three pieces are connected, the major families of deep generative models become much easier to understand. From prediction to distribution learning A discriminative model usually begins with a prediction problem. Given an input x , predict the most likely output y : f(x)=yargmax​p(y∣x) The model focuses directly on the conditional relationship required for prediction. A generative model asks a broader question. Instead of learning only the path from x to y , it models the probability structure from which the data arises. For class-conditional modeling, for example, we can model p(x∣y) together with the prior p(y) and recover the posterior using Bayes' rule: p(y∣x)=p(x)p(x∣y)p(y)​ In unsupervised generative modeling, the target becomes the data distribution itself. We assume the training samples come from some unknown distribution: x1​,x2​,…,xN​∼i.i.d.pdata​(x) The model then constructs a parameterized distribution intended to approximate it. This perspective connects tasks that can otherwise seem separate: xtrain​∼pdata​(x),xnew​∼pmodel​(x) Density estimation asks how probable an observation is under the model. Generation draws new samples from the learned model distribution. Unsupervised representation learning can capture latent structure shared by the data while learning that distribution. All three begin with the same underlying goal: learning the probability structure of the data. Why high-dimensional data makes this hard Real data usually lives in a high-dimensional space. An image, for example, is described by many variables, while valid observations occupy only a limited and complicated region among all possible combinations. So the first problem is not simply ""use a neural network."" It is: How do we represent the joint distribution of many random variables compactly enough to compute with, while keeping enough flexibility to describe the data? This connects naturally to inverse problems. A forward problem moves from a cause x to an observation y . An inverse problem starts from an observed y and asks which possible x could have produced it. Conditional generation extends the same idea by generating possible x values consistent with a given condition. Recovering a high-resolution image from a low-resolution one, estimating color from grayscale data, reconstructing missing regions, or recovering a CT slice from a sparse-view sinogram all share this structure. The difficulty is that the inverse may not be unique. A single observation can be consistent with multiple possible causes. Inverse problems are therefore not simply about running a function backward; they require reasoning over possible causes that are compatible with the observation. Latent variables turn inversion into posterior inference Suppose a model contains an observed variable x and a hidden variable z . The generative direction models the relationship between z and x . Once x has been observed, the reverse question is which values of z could have produced it. That is posterior inference: p(z∣x)=p(x)p(z,x)​ Posterior inference can require both computing the posterior distribution and calculating expectations under it. The E-step of the EM algorithm is a direct example: Q(θ,θold)=z∑​p(z∣x;θold)logp(x,z;θ) Here the posterior over the unobserved z is used to compute the expected complete-data log-likelihood. As the latent space grows and interactions among hidden variables become more complicated, direct posterior computation can become intractable. Increasing model expressivity therefore leads naturally to another question: how can the required inference remain computationally feasible? When the posterior is intractable Two major approaches are MCMC and Variational Inference. Both approximate a posterior that is difficult to compute directly, but the approximation arises in different ways. MCMC: approximate with samples MCMC constructs an ergodic Markov chain whose stationary distribution is the target posterior. Samples generated by running the chain are then used to approximate the posterior and the expectations we need. The practical approximation comes from finite computation. In real problems, only a limited amount of time is available for generating samples, and exploring the relevant probability regions can be expensive in high-dimensional or complex models. MCMC therefore has a finite-time sampling approximation in practice. Variational Inference: approximate with optimization Variational Inference makes a different tradeoff. Instead of sampling directly from the posterior, it defines a tractable family of distributions Q and searches within that family for a distribution close to the true posterior: q∗(z)=q(z)∈Qargmin​DKL​![q(z)∥p(z∣x)] This turns posterior inference into an optimization problem. The tractability comes from restricting the search space. If the true posterior cannot be represented by the chosen variational family, more optimization time alone cannot eliminate the mismatch. The contrast is useful: This distinction becomes concrete in a VAE. Rather than computing a complicated latent posterior directly, the model uses a variational distribution, with the Encoder acting as a Recognition Network that constructs the approximation. The Decoder generates observations from the latent variable. An abstract inference problem has now become part of a trainable model architecture. The real bottleneck: flexibility versus tractability Representation and computation collide when we try to model realistic high-dimensional distributions. A simple probability model can be easy to normalize and evaluate but may lack enough flexibility to describe complex data. A deep neural network can represent much richer functions, but its output is not automatically a valid probability density. A probability density must be nonnegative and integrate to one. One way to construct such a density is: pθ​(x)=Zθ​efθ​(x)​ where Zθ​=∫efθ​(x)dx Exponentiation gives a positive quantity, while the normalizing constant Zθ​ ensures that the density integrates to one. For a complex neural function over a high-dimensional space, however, computing that integral can itself become intractable. This is the Tractability-Flexibility Tradeoff. It provides a useful way to compare deep generative models. Rather than asking only which network architecture a model uses, ask which difficult probability computation its structure makes manageable. Learning a model distribution In likelihood-based generative modeling, learning can be viewed as reducing the difference between the data distribution and the model distribution: DKL​![pdata​(x)∥pmodel​(x;θ)]Ex∼pdata![logpmodel​(x;θ)pdata​(x)​] Expanding this expression separates a term that depends only on the data distribution from a term containing the model log probability. The data-only term is independent of θ , so it does not change as the model parameters are optimized. The learning objective therefore connects to increasing the model's log