Generative models try to learn the underlying probability distribution of real-world data and then use it to make new samples — either to study that distribution or just to get fresh data.
Some models learn the distribution explicitly; others skip that and focus on generation. GANs are firmly in the second camp — they're built for sample generation, though you can coax them into estimating densities too. Before getting to GANs, it helps to see the bigger map.
A taxonomy of deep generative models
Explicit density models
I'd call these the "brute-force" models. The idea is simple: define a model that gives an estimate of a probability distribution, parameterised by θ, and call the likelihood the probability the model assigns to the training data. Maximum likelihood then says: pick the parameters that maximise the likelihood of the data — done in log-space in practice, which dodges the numerical underflow you'd hit from multiplying many tiny probabilities.
There's an equivalent view: maximum likelihood is the same as minimising the KL divergence between the true data-generating distribution and the model's. We don't have the true distribution, of course — only samples — so in practice we minimise the KL divergence between the empirical distribution of the data and the model.
Explicit models split further by whether that density is computationally tractable. Tractable examples include fully-visible belief networks, nonlinear ICA, and PixelRNN. Models that instead learn a bound on an intractable density include variational autoencoders (deterministic approximations) and Boltzmann machines (Markov-chain Monte Carlo).
Implicit density models
The trouble is that the tractable models tend to be slow, and the approximate ones often aren't great at high-quality samples. So if all you want is samples that look real, why learn the density at all? Drop it. (Fun fact: you can still make a GAN estimate a density — but vanilla GANs just generate.)
Some implicit models draw samples by running a Markov-chain transition operator many times — a generative stochastic network is the classic example — but Markov chains scale poorly to high dimensions and get expensive. GANs sidestep that entirely.
The GAN framework
It's a game between two players. A generator creates samples meant to look like the training distribution, and a discriminator inspects samples and labels them real or fake. During training the generator tries to fool the discriminator; the discriminator learns, alongside it, not to be fooled.
More precisely, both players are differentiable functions: a discriminator D(x) with parameters θd, and a generator G(z) with parameters θg, where z is random noise. Each has a cost defined in terms of the other's parameters, and each controls only its own parameters.
The solution is a Nash equilibrium — a point that is a local minimum of the discriminator's cost inθdand of the generator's cost inθg.
Training GANs
Each iteration samples two mini-batches — real x from the dataset and noise z from the prior — and takes two gradient steps at once: one on θd to lower the discriminator's cost, one on θg to lower the generator's. Adam is the usual update rule. People argue about whether to do k = 1 or several discriminator steps per generator step; some report k > 1 helping, but Goodfellow's original recommendation is simultaneous descent, one step each.
The cost function (and a crucial fix)
In the clean minimax version, the generator's cost is just the negative of the discriminator's. That's tidy for proving the game reaches equilibrium, but in practice it can stall: early on, when G is bad, D rejects its samples with high confidence, the generator's term saturates, and the gradient vanishes.
One small change fixes it. Instead of training G to minimise log(1 − D(G(z))), train it to maximise log D(G(z)). Same fixed point, but much stronger gradients early in training. The game is no longer zero-sum — and that's fine.
The workhorse architecture: DCGAN
DCGAN came from Radford et al. (2015), proposing a family of convolutional architectures for GANs and visualising the filters they learn. The key ingredients:
- An all-convolutional net (Springenberg et al., 2014): no pooling — use strided transposed convolutions to grow spatial dimensions.
- No fully-connected hidden layers in deeper architectures.
- Batch normalisation to stabilise learning — but not on the generator's output layer or the discriminator's input layer, or you get oscillation and instability.
- ReLU in the generator everywhere except the output (which uses tanh); LeakyReLU throughout the discriminator.
- Adam rather than SGD with momentum.
It worked well enough that most GANs in research and production today are loosely built on DCGAN's principles.
How good are they?
GANs were designed to dodge the weaknesses of other models:
- No Markov chains, so they scale to high dimensions where Boltzmann machines and GSNs struggle.
- No variational bound, so nothing inherently caps sample quality the way it can for VAEs.
- Very few restrictions on the generator's design — an edge over Boltzmann machines.
- Samples are generated in parallel, unlike the pixel-by-pixel generation of fully-visible belief nets and PixelCNNs.
- Subjectively, they tend to produce nicer samples.
The catch: training is less stable than for the other models — though years of architectural and training tricks have made that far more manageable.
Where to go next
Ian Goodfellow's NIPS 2016 GAN tutorial is a great two hours from the person who invented them. For the literature: the original Generative Adversarial Nets, DCGAN, Improved Techniques for Training GANs, Conditional GANs, and the ever-growing GAN Zoo.
GANs are a genuinely powerful family of networks — a real breakthrough in modelling the real world, and still producing astonishing results.