Two networks locked in a duel — a forger and a detective — and what happens when the detective gets too good. From the minimax loss, through Jensen–Shannon and Wasserstein, to StyleGAN.
The core idea of a GAN is almost mischievous: instead of telling the generator what good output looks like, you train a second network to catch fakes, and use its complaints as the only teaching signal.
Suppose you have a pile of real training examples — photographs of faces, say — written as . You would like a machine that can invent new faces that look like they came from the same source, even though they are not copies of any real photo. We call these invented examples , and we want them drawn from the same distribution as the real data.
The trick is to never look at a real face and a fake face side by side and measure their difference directly. (How would you even define "difference" between two faces?) Instead, a GAN sets up a game between two networks.
Generator : takes a random latent variable (drawn from a simple base distribution, e.g. a normal) and turns it into a sample . It is the forger.
Discriminator : takes an input and returns a scalar that is higher when it believes the input is real. It is the detective.
Generation works in two strokes: (1) choose a latent from the base distribution; (2) push it through the generator, . The training goal is to adjust the generator parameters until the generated samples are statistically indistinguishable from the real data — at which point even an excellent detective can do no better than a coin flip.
You cannot write down "looks like a real face" as a loss function. But you can train a classifier to separate real from fake — that is an ordinary supervised problem. The GAN insight is to use the classifier's success as the generator's loss. If the detective can tell them apart, the forger gets a gradient telling it how to improve. The forger only stops improving when the detective is fooled.
To build intuition, the slides use the simplest generator imaginable — it just adds a single parameter to the latent:
So the generated samples are just the latent samples shifted along the line by . Training amounts to sliding the orange (synthesized) cloud until it sits on top of the teal (real) cloud.
The discriminator faces a plain classification task. Label real examples and generated ones . With the logistic sigmoid, the standard binary cross-entropy objective is:
Now plug in the labels. Real examples carry (only the second term survives); generated samples carry (only the first term survives):
is the discriminator's estimated probability that is real. The first sum punishes the detective when it assigns high realness to fakes; the second punishes it when it assigns low realness to real data. Minimising it makes the detective a sharp judge.
Substitute the generator definition . Here is the pivotal observation: we want the generator to fool the detective, so we must maximise the same loss with respect to (push the generated samples toward being misclassified):
One objective, two opposed optimisers. The discriminator parameters minimise the loss (try to classify correctly); the generator parameters maximise it (try to make the discriminator wrong). This is more complex than any single-objective loss seen before.
The loss measures how well the discriminator separates real from fake. The discriminator wants this small error (good separation), so it minimises. The generator wants the discriminator to fail, i.e. it wants the separation error large, so it maximises the very same quantity. They are pulling on opposite ends of one rope — hence "minimax."
And yes, in practice we do split it into two losses (next slide). The point of writing the single expression is to make the adversarial structure explicit: it is one game, not two unrelated training problems.
GAN training is a minimax game, and to actually run gradient steps we split the joint objective into the part each network controls:
We alternate: take a step minimising (improve the detective), then a step on (improve the forger), and repeat. Note uses only the generated-sample term, because the real examples don't depend on .
The toy "add " generator becomes a deep convolutional network in practice. The Deep Convolutional GAN (DCGAN) turns a 100-dimensional latent vector into a image.
Don't mix them up. The generator uses fractionally-strided (a.k.a. transposed) convolutions to grow spatial size; the discriminator uses ordinary strided convolutions to shrink it. The generator ends in tanh; the discriminator ends in sigmoid.
No — a GAN never reconstructs a specific input and has no per-sample reconstruction loss. The generator maps random latents to images, and is trained only by an adversarial signal (a separate discriminator's success/failure), not by comparing its output to a target image. (Autoencoders, by contrast, have an encoder and a reconstruction objective — that distinction matters for the Week 1 autoencoder material and for VAEs next week.)
(1) Draw a mini-batch of latents , generate fakes . (2) Grab a mini-batch of real . (3) Update the discriminator by descending : it learns to score high and low. (4) Freeze ; update the generator by ascending the adversarial term (descending as written): the fakes shift toward regions the detective currently calls "real." Repeat. Early on the detective wins easily; ideally they converge to a stalemate where everywhere.
But there's a problem lurking in that last sentence — "early on the detective wins easily." When it wins too easily, the generator gets almost no usable gradient. The slides show Arjovsky et al. (2017) examples of this failure (muddy, mode-collapsed images). Understanding why is the whole of Part II.
Why does the original GAN train so badly? The answer is that its loss secretly measures a specific distance between distributions — the Jensen–Shannon divergence — and that distance gives no gradient when the two distributions don't overlap. The fix, the Wasserstein distance, needs a detour through linear programming and duality.
We start by rewriting the discriminator loss in a more revealing form. Divide the two sums by the counts (real) and (generated). Each averaged sum becomes an expectation, and in the limit an integral over the data space:
Here is the distribution over generated samples and the true distribution over real examples. We have turned a finite-sample loss into a statement about two probability distributions.
Consider a sample of unknown origin. Define the class-conditional densities and . By Bayes' theorem:
where and are the prior probabilities of a sample being real or generated. If we assume equal counts , the priors are equal:
By the law of total probability, . Substituting the priors and plugging into Bayes, the priors cancel:
That last constraint is a discrete Lipschitz condition: may not change too fast between neighbouring bins.
The primal optimises over the whole matrix (one variable per pair of bins — huge, and impossible in continuous space). The dual optimises over a single function (one value per bin), with a slope cap. In the continuous limit that "single function with a slope cap" is exactly a neural network constrained to be Lipschitz — which is buildable.
The primal generalises to a transport plan (mass moved from position to ):
and the dual generalises to an optimisation over a function :
subject to the constraint that the Lipschitz constant of is less than one.
Now make it trainable. Maximise over the function space by optimising the parameters of a network (now called a critic, not a classifier — there is no sigmoid). Approximate the integrals with samples — generated and real :
subject to the constraint that the critic has an absolute gradient norm of less than one at every position :
Three concrete differences. (1) No sigmoid/log — the critic outputs a raw score, the loss is a plain difference of means. (2) The critic is constrained to be 1-Lipschitz (gradient norm ). (3) Because the underlying distance is Wasserstein, the loss keeps decreasing smoothly as the clouds approach — even when they don't yet overlap — so the generator always has a gradient. This directly cures the vanishing-gradient disease from slide 19.
The dual identity is only valid when is 1-Lipschitz. Drop the constraint and the "max" is unbounded (you can scale up forever), so the loss is meaningless. Enforcing it — weight clipping in the original WGAN, or a gradient penalty in WGAN-GP — is not an optional regulariser; it is what makes the quantity equal the Wasserstein distance.
The vanilla discriminator outputs a probability via a sigmoid and minimises JS divergence; when the clouds are separated its output saturates and the generator gradient vanishes. The WGAN critic outputs an unbounded 1-Lipschitz score and estimates the Wasserstein distance; its gradient stays informative even for disjoint distributions. Consequence: the WGAN critic can be trained to (near-)optimality without starving the generator, whereas an over-trained vanilla discriminator kills generator learning.
(1) The discriminator's BCE loss, averaged, becomes expectations/integrals over and . (2) Solving for the optimal discriminator via Bayes (equal priors when ) gives . (3) Substituting it back shows the loss equals the Jensen–Shannon divergence between and . (4) JS saturates at for disjoint supports → vanishing gradients. (5) Replace JS with the Wasserstein distance (earth-mover's work), which is well-defined for disjoint supports and smooth. (6) Its transport definition is a hard primal LP, so we take its dual (justified by strong duality), which optimises a single 1-Lipschitz function. (7) Parameterise that function as a network → the WGAN critic loss with gradient-norm .
A better loss (Part II) stabilises training; these three architectural and sampling tricks push image quality and diversity. They are the engineering that made high-resolution GANs (e.g. faces at ) possible.
Training a GAN to emit a megapixel image from scratch is hopeless — the discriminator instantly spots the chaos and the generator never finds its footing. Progressive growing (Karras et al., 2018) sidesteps this by starting tiny and adding resolution in stages.
It is a curriculum: master the easy, low-frequency layout (where the head/sky/horizon go) before tackling hard, high-frequency detail (hair strands, skin pores). Each stage is a small, stable problem, and it starts from a good initialisation handed down by the previous stage. This both speeds training and dramatically improves final quality.
A mechanism designed to prevent mode collapse and ensure diversity in generated samples. The discriminator is allowed to look not just at one sample in isolation but at feature statistics across the whole mini-batch, comparing the spread of generated samples against the spread of real ones. These batch statistics are integrated into the feature maps and fed back as discriminator feedback.
Mode collapse is when the generator finds one image that fools the detective and produces near-copies of it forever. If the detective only ever sees one sample at a time, it cannot notice the lack of variety. Let it see a whole batch and compute "how diverse is this batch compared to a real batch?" — now a collapsed generator is instantly caught, because its batch is suspiciously uniform. The pressure to match real-batch diversity forces the generator to cover more modes.
This trick happens after training, when you sample. The latent is drawn from a base distribution; truncation shrinks it toward the mean by a factor before generating (Brock et al., 2019).
Mini-batch discrimination acts during training and increases diversity — it prevents mode collapse so the trained model can cover many modes. Truncation acts during sampling and decreases diversity on purpose — it sacrifices variety to raise per-image quality (samples near the latent mean are the ones the generator renders most reliably). One is a learning fix; the other is a knob you turn at inference to pick your point on the quality–diversity trade-off.
At full resolution from the start, the discriminator can trivially distinguish real from the generator's initial noise (high-res statistics are very informative), so the JS/adversarial signal saturates and the generator gets little usable gradient — the Part II problem, amplified. There is also an enormous, unstable optimisation landscape to navigate at once. Progressive growing turns one impossible problem into a sequence of easy, stable ones, each warm-started from the last, so coarse structure is locked in before fine detail is attempted.
A vanilla GAN generates something from the dataset, but you can't ask for "a 7" or "a bird." Conditional generation gives you a steering wheel — and the slides show three different ways to attach it.
All three variants feed extra information (a class label, an attribute vector) alongside the latent . They differ in where the conditioning is enforced and what the discriminator is asked to do.
(a) Conditional GAN (cGAN): the attribute is fed to both generator and discriminator. The discriminator scores the pair : is this image-plus-attribute combination real? This forces the image to match the requested attribute.
(b) Auxiliary classifier GAN (ACGAN): the discriminator has two heads — for "is it real?" and for "which class?". The class label is supervised, so the discriminator both judges realism and classifies. (Odena et al., 2017.)
(c) InfoGAN: the code is part of the latent input and is not labelled. A second head tries to recover from the generated image, maximising the mutual information between and the output. The result: discovers interpretable factors with no supervision. (Chen et al., 2016.)
Trained on unlabelled MNIST, InfoGAN's codes spontaneously align with meaningful factors: a discrete code controls digit identity (sweeping it walks 0→9); a continuous code controls rotation/slant; another continuous controls stroke width. Nobody told it what "rotation" or "width" was — it found them because they are the factors that carry the most information about the image.
In cGAN the code is a known label you supply — you tell the model "make a 7," and conditioning is enforced by having the discriminator judge the (image, label) pair. In InfoGAN the code is unsupervised — you don't know what it means in advance; the model is rewarded for making the code recoverable from the output, so it discovers interpretable factors on its own. Short version: cGAN imposes meaning; InfoGAN discovers it.
ACGAN's discriminator outputs two things from a single image input: (1) a sigmoid head giving the probability the image is real, and (2) a softmax head giving a probability distribution over the class the image belongs to. The generator still receives the class as input, but the discriminator is not shown the label — it must reconstruct it.
This is the structural contrast with cGAN: in cGAN the label is fed to both networks and the single discriminator head scores the pair ("does this image-and-label combination look real?"). In ACGAN the label is fed only to the generator; the discriminator's extra classification head supplies the conditioning pressure by being trained (on both real and generated images) to name the class. So cGAN verifies the label, ACGAN predicts it.
Conditioning on a label is one thing; conditioning on a whole image is image translation — turn a sketch into a photo, a low-res into high-res, a horse into a zebra. The recurring recipe: a content loss to stay faithful to the input, plus an adversarial loss to look real.
The generator is a U-Net that maps an input image to a prediction. Two losses train it:
Content loss alone (e.g. pixel ) gives blurry, "average" predictions — it can't decide between equally valid sharp details, so it hedges. Adversarial loss alone would let the generator produce a perfectly realistic image that ignores the input. Together: content loss keeps it faithful, adversarial loss keeps it sharp and realistic.
Same template, specialised. The generator is a convolutional network that takes a low-resolution input and predicts a high-resolution image. The content loss is a content/VGG loss (the prediction should agree with the real high-resolution image in the feature space of a pretrained VGG network, not just pixel-wise), plus the adversarial "looks real" loss. (Ledig et al., 2017; upscaling beats bicubic interpolation visibly.)
SRGAN's content loss is computed on VGG feature activations, not raw pixels. This is deliberate: pixel-wise loss for super-resolution produces over-smoothed results (the same blurring problem as Pix2Pix), whereas matching deep features rewards perceptually plausible high-frequency detail. If asked why SRGAN looks sharper than bicubic, "perceptual/VGG content loss + adversarial loss" is the answer.
Pix2Pix needs paired data (the same scene as sketch and photo). Often you only have two unpaired piles — horses here, zebras there. CycleGAN translates anyway, using a cycle-consistency loss.
Without paired data there is no ground-truth target, so no ordinary content loss. Cycle-consistency replaces it: "if you turn a horse into a zebra and back, you must recover that same horse." This stops the generator from ignoring the input or scrambling content — it must preserve enough structure to be reversible — while the adversarial loss handles the "look like a zebra" part.
Pix2Pix is trained on paired data, so for each input there is a known correct output to compute a content loss against. CycleGAN is trained on unpaired collections (e.g. a set of horse photos and an unrelated set of zebra photos) — there is no matching target image, so a direct content loss is impossible. It is replaced by the cycle-consistency loss: translating to the other domain and back must reproduce the original input, which preserves content without ever needing a paired target.
The adversarial loss is shared — a discriminator pushing outputs to look real. The second term differs because the available supervision differs:
Pix2Pix — paired data, so a direct content loss in pixel space: the prediction is compared to the known ground-truth target image.
SRGAN — paired (low-res in, high-res target), but pixel-wise comparison gives blurry results, so it uses a content/VGG loss: prediction and target are pushed to agree in the feature space of a pre-trained VGG network rather than raw pixels.
CycleGAN — unpaired, so no target exists; the second term is the cycle-consistency loss (translate across and back, recover the original). Exam trap: don't say all three use "content loss" — only Pix2Pix uses a plain pixel content loss; SRGAN uses a feature-space variant; CycleGAN uses cycle-consistency instead.
StyleGAN's signature move is to separate the things you'd intuitively want to control independently — coarse structure vs. fine texture vs. random detail — and inject them at different scales of the generator.
In a plain GAN the latent is shoved in at the bottom and everything is entangled — nudge and pose, identity, hair and lighting all move together. StyleGAN reorganises the generator into three subsystems: a style path, a noise path, and the main generative pipeline that consumes both.
Mapping network: latent () → fully-connected network → intermediate latent (). Disentangling happens here.
Style: linear transforms of produce styles (each : a scale and an offset). Each is applied as a per-channel scale and offset at a different layer — coarse layers control coarse attributes (pose, face shape), fine layers control fine attributes (hair, freckles, colour).
Noise: independent noise maps ( at , etc.) are added to every channel, supplying stochastic, non-semantic detail (exact hair placement, skin pores).
Two independent axes of control. Style = systematic, semantic content injected via (what the face is); noise = random texture (irrelevant micro-detail). And scale: inject at early/low-resolution layers to change coarse things, at late/high-resolution layers to change fine things. Slide 61 demonstrates exactly this — changing coarse/medium/fine styles independently, and increasing coarse vs fine noise independently, with the rest of the face held fixed.
comes from a fixed simple prior (e.g. a Gaussian), whose shape need not match the real distribution of face factors — forcing styles directly from entangles attributes (the manifold gets "warped"). The learned mapping network produces in a space free to be disentangled, where individual directions correspond to individual attributes. Controlling styles from is therefore far cleaner — this is why you can change "coarse style" without disturbing "fine style."
Objective: edit a real image by projecting it into the latent space, manipulating the latent variables, and re-projecting back to image space. Challenge: a GAN maps latent → image, not image → latent, so there is no built-in inverse. Methods: (i) encoder-based (train a network to predict the latent), (ii) optimisation-based (search for the latent whose output best matches the image), and for StyleGAN specifically (iii) pivotal tuning.
StyleGAN gives gorgeous, controllable generated faces — but to edit your photo you first need its latent code. Inversion finds the / that reproduces your photo; then you nudge that code along a meaningful direction (older, smiling, different hair) and regenerate. The hard part is that the inverse isn't unique or exact, which is why there are competing methods and why StyleGAN's pivotal tuning fine-tunes the generator slightly around the found code.
Everything in this week answers one question: how do you train a network to produce realistic data when you can't write down a loss for "realistic"? The answer was an adversary — and most of the week was about fixing the adversary's weaknesses.
The objective is to learn a generator network that transforms random noise into data indistinguishable from a training set. The mechanism has three moving parts: a generator that creates samples from random noise; a discriminator that tries to distinguish real examples from generated samples; and a training loop in which the generator is repeatedly updated to produce data that is increasingly "real" to the discriminator.
Weak training signal: the original GAN formulation suffers when it's easy to tell whether samples are real or generated. That's precisely early training — and precisely when JS divergence saturates and the generator's gradient vanishes.
Each later section is a solution to a different facet of the problem:
| Improvement | What it fixes / adds |
|---|---|
| Wasserstein | A more consistent training signal by measuring the Wasserstein (earth-mover's) distance — non-zero, smooth gradient even for disjoint distributions. |
| Convolutional tricks | Progressive growing, mini-batch discrimination and truncation enhance image quality and diversity (and enable high resolution). |
| Conditional | Control over output attributes (e.g. object class) and improved output relevance — cGAN, ACGAN, InfoGAN. |
| Image translation | Retains conditional information as images and uses the discriminator loss to favour realistic outputs — Pix2Pix, SRGAN, CycleGAN. |
| StyleGAN | Injects noise strategically to control style and noise at different scales, enhancing realism and controllability. |
Problem: the adversarial signal is weak/unstable. Fixes: II swaps JS for Wasserstein (smooth gradient); III stabilises and sharpens via progressive growing + mini-batch discrimination + truncation; IV adds controllability via conditioning; V extends conditioning to whole images (translation) with content + adversarial losses; VI disentangles control by scale (StyleGAN) and enables editing real images (inversion).
A GAN trains a forger against a detective; the detective's binary-cross-entropy loss is secretly the Jensen–Shannon divergence, which vanishes when fakes are obvious — so we replace it with the Wasserstein distance (computed via the LP dual as a 1-Lipschitz critic), then stack on progressive growing, mini-batch discrimination, truncation, conditioning, image-to-image translation, and StyleGAN's scale-separated style/noise to win quality, diversity and control.
Next lecture: Variational Autoencoders — a second route to generative modelling, this time with an explicit, optimisable likelihood instead of an adversary.