Unit 1 ready
Foundations of Generative AI
Discriminative vs generative models, and the six model families: autoencoders, VAEs, GANs, autoregressive models, diffusion, Transformers.
What you will be able to do
- tell a discriminative model p(y|x) from a generative model p(x), and say what each can and cannot do;
- write the training objective of an autoregressive model, an autoencoder, a VAE, a GAN and a diffusion model in one line each;
- derive the VAE evidence lower bound and the GAN optimal discriminator, and explain the reparameterisation trick;
- build a character-level text generator and read its perplexity on seen and unseen text;
- explain hallucination, bias, privacy and copyright risk as consequences of the training objective, with a measured example.
The big picture
Most machine learning you have met is discriminative: draw a boundary between classes. Generative AI asks for more: learn what the data looks like, so you can produce new examples that were never in the training set.
Six ways to model a distribution.
| family | idea |
|---|---|
| autoregressive | write one piece at a time, each conditioned on the earlier ones |
| autoencoder | squeeze to a short code and rebuild (a compressor, not yet a generator) |
| VAE | autoencoder with a fuzzy code and a prior, so you can sample new codes |
| GAN | a forger and a detective improve each other |
| diffusion | learn to remove noise that was added step by step |
| Transformer | the neural network inside most of the above (an architecture, not a family) |
One objective explains the risks. Generative models are trained to produce plausible data, not true data. That is why they hallucinate, why they mirror biases in their training data, and why a model with enough capacity can copy its training text. In the character model, 60% of 12-letter windows are copied verbatim at memory n = 6.
Key equations
Bayes' rule: a generative model can classify too.
The VAE bound: reconstruction minus KL, plus a gap that is never negative.
Closed-form KL used in the VAE loss.
Reparameterisation: gradients can pass through sampling.
The GAN optimal discriminator and what the generator is really minimising.
Diffusion forward process in one step.
Figures
Try it in the playground
Each link opens the widget with good starting settings. Every option has a plain-English label and a one-line help text.
Generative classifier
Move two Gaussian classes and a probe point; see Bayes' rule produce the probabilities.
Text generator (n-gram)
Write text letter by letter; watch it turn into copying as memory grows.
Run the code yourself
Everything is plain NumPy, including the neural networks (with hand-written backpropagation). After downloading and unzipping the All code (.zip):
pip install -r requirements.txt
python codes/genmodels/genmodels_np/genmodels_np.py # tiny n-gram demo
python codes/genmodels/genmodels_np/tests/test_genmodels_np.py # 7 tests incl. gradient checks, prints: ok
python figures/fig04_vae.py # trains the VAE (a few seconds) and reproduces Figure 4The module has CharNgram, MLP (manual backprop), train_autoencoder, train_vae, train_gan_1d and the classifiers.
What the experiments really showed. The textbook line “generative classifiers need less data” failed at 3 points per class here, because the Gaussian model estimates full covariance matrices. And the best GAN step was 300, not the last one. Both are discussed in the PDF.
Self-check
Answer in your head first, then click to reveal.
Which of p(y|x) and p(x) lets you create new data?
p(x), or p(x|y). p(y|x) alone only labels.
What are the two terms of the VAE loss?
A reconstruction error and a KL term that keeps the codes close to the prior N(0, I).
What does the optimal GAN discriminator output when the generator is perfect?
1/2 everywhere: it cannot tell real from generated.
Why does adding more context to an n-gram model first help and then hurt on new text?
More context fits the training text better but makes the model memorise it; held-out perplexity rises again after n = 4 here.
What does the reparameterisation trick do?
Writes z = mu + sigma * eps with eps independent of the parameters, so gradients can flow through the sampling step.
Name two ways a generative model can leak or misuse training data.
Memorisation (reproducing training text) and hallucination or bias inherited from the data; all follow from what the objective rewards.