Generative AI with Transformers

Executive Study Roadmap 🔴 Exam-core

Top exam-core topics

Exam Priority Dashboard 🔴 Exam-core

This dashboard is an evidence-based estimate, not a promise about the actual exam.

Language Modeling Foundations 🔴 Exam-core

🧭 This chapter teaches how simple language models assign probabilities to text, why the Markov assumption helps and hurts, and how model quality is measured.

📌 Why it matters: Transformers are powerful neural language models, but the basic question is already present in n-grams: given history, what token comes next?

🧠 Core intuition

A language model estimates a probability distribution over token sequences. Exact history is impossible to count for realistic text, so classical models use a short window. A bigram uses one previous token; a trigram uses two. More context usually helps until the state space explodes and counts become sparse.

📚 Clean explanation

For a sequence w₁...w_T, the chain rule says P(w₁...w_T)=∏ₜP(wₜ|w₁...wₜ₋₁). An n-gram model replaces the full history with the previous n-1 tokens. The estimate is usually a count ratio: how often did this next word follow this context?

🧪 Worked example: from corpus counts to perplexity

input: corpus tokens, order n, desired length L
build a table: context tuple -> counts of next tokens
choose an initial context
repeat until L tokens:
  sample next token from counts(context)
  append token
  context = last n-1 tokens
output generated sequence

⚠️ Common traps

• Higher n is not automatically better; sparse counts make estimates brittle.
• Perplexity is about predictive probability, not human factuality or usefulness.
• A Markov generator can sound locally coherent without understanding meaning.

✅ Exam memory

Be able to define the Markov property, write the n-gram approximation, compute a tiny count-based probability, explain state-space explosion, and define perplexity.

🧩 Connection: embeddings and neural LMs are attempts to escape brittle exact-count tables by learning smoother vector representations.

Text Representation and Embeddings 🔴 Exam-core

🧭 This chapter teaches how raw text becomes vectors: tokenization, sparse count vectors, TF-IDF, and dense learned embeddings.

📌 Why it matters: every neural model in the course starts by turning discrete token IDs into continuous vectors.

🧠 Core intuition

One-hot vectors identify tokens but say nothing about similarity. Bag-of-words and TF-IDF represent documents by token counts or weighted counts, but order and context are mostly lost. Dense embeddings compress distributional usage into a smaller vector space where nearby words tend to behave similarly.

📚 Clean explanation

Tokenization first defines the model's alphabet. Word-level tokenization suffers from huge vocabularies and unknown words. Subword methods such as BPE and WordPiece split rare words into reusable pieces, which is why modern Transformers can handle names, compounds, and morphology better than strict word vocabularies.

🧪 Worked example: TF-IDF counts vs Word2Vec training pairs

input: center word c, true context word o, negative words n_1...n_k
look up vectors v_c, u_o, u_n
maximize log σ(u_o · v_c)
maximize ∑ log σ(-u_n · v_c) for negatives
update vectors by gradient descent
output learned embedding matrix

⚠️ Common traps

• Distance in embedding space reflects training distribution, not guaranteed truth.
• Static Word2Vec gives one vector per word type, so ambiguity such as “Mars” remains hard.
• Subword tokenization solves out-of-vocabulary issues, not all semantic issues.

✅ Exam memory

Know one-hot vs BOW vs TF-IDF vs dense embeddings, explain CBOW vs skip-gram, and describe why embeddings are trainable lookup tables in neural models.

🧩 Connection: the Transformer embedding layer is the same core idea as learned word vectors, but trained end-to-end for the model objective.

Neural Language Models 🔴 Exam-core

🧭 This chapter teaches how neural networks turn context vectors into token probabilities, and why RNN/LSTM models were a stepping stone.

📌 Why it matters: Transformers still use softmax/cross-entropy next-token training; they mainly replace how context representations are computed.

🧠 Core intuition

A feed-forward neural language model embeds the last few tokens, concatenates them, passes them through hidden layers, then predicts the next token. RNNs remove the fixed window by carrying a hidden state through the sequence. LSTMs add gates to preserve useful information longer.

🧪 Worked example: logits → softmax → loss

input: token sequence x_1...x_T
initialize hidden state h_0
for each t:
  e_t = embedding[x_t]
  h_t = activation(W_e e_t + W_h h_{t-1} + b)
  logits_t = U h_t + c
  p_t = softmax(logits_t)
train with cross-entropy against x_{t+1}

⚠️ Common traps

• RNNs process sequentially; this limits parallelism compared with Transformers.
• LSTMs mitigate long-range gradient issues but do not make all long dependencies easy.
• Bidirectional RNNs are useful for representation tasks, but cannot directly generate left-to-right without respecting causality.

✅ Exam memory

Be able to draw a feed-forward LM, explain RNN hidden-state recurrence, state the purpose of LSTM gates, and compute/interpret softmax cross-entropy.

🧩 Connection: seq2seq uses RNN encoders/decoders; attention then removes the need to compress the whole source into one final hidden state.

Seq2Seq and Attention 🔴 Exam-core

🧭 This chapter teaches encoder-decoder models, teacher forcing, the bottleneck problem, and dot-product attention.

📌 Why it matters: attention was introduced to let the decoder look back at relevant source states instead of relying on one fixed vector.

🧠 Core intuition

An encoder reads the input sequence and produces hidden states. A decoder generates the output sequence. Without attention, the decoder depends heavily on the final encoder state, creating a bottleneck. Attention computes a custom context vector at every decoding step.

🧪 Worked example: attention chooses the relevant source word

input: previous target token, decoder state, encoder states h_1...h_n
q = current decoder state
for each source position i:
  score_i = q · h_i
weights = softmax(score)
context = weighted_sum(weights, h)
new decoder state uses previous token + context
predict next target token

⚠️ Common traps

• Attention weights are useful diagnostics but not perfect explanations.
• Teacher forcing is a training procedure, not an architecture component.
• Bidirectional encoders can see both directions, but autoregressive decoders must not see future target tokens.

✅ Exam memory

Know the encoder-decoder bottleneck, why attention fixes it, and how dot products become weights and a context vector.

🧩 Connection: Transformers remove recurrence and use attention among token representations inside each layer.

Transformer Architecture 🔴 Exam-core

🧭 This chapter teaches the Transformer block from token IDs to logits: embeddings, positional information, Q/K/V, scaled dot-product attention, multi-head attention, masks, FFN, residuals, normalization, training, and decoding.

📌 Why it matters: almost every modern LLM variation is a modification of this base block or its training/serving regime.

🧠 Core intuition

Self-attention lets each token build a context-aware representation by looking at other tokens. Q/K/V projections separate matching from information transfer. Multi-head attention repeats this in several smaller subspaces. The FFN then transforms each token independently. Residual and normalization paths keep deep stacks trainable.

🧪 Worked example: one decoder self-attention head

input: X with shape [sequence, d_model]
Q = X W_Q; K = X W_K; V = X W_V
scores = Q K^T / sqrt(d_k)
if decoder: add causal mask to scores
A = softmax(scores)
head = A V
MHA = concat(head_1...head_h) W_O
Y = LayerNorm(X + MHA)
Z = LayerNorm(Y + FFN(Y))
return Z

⚠️ Common traps

• Attention weights mix values, not keys.
• Multi-head attention is not just “more attention”; each head has its own projections and subspace.
• Masked vs non-masked attention is an architectural constraint, not a decoding preference.
• The FFN does not mix positions; attention does.

✅ Exam memory

Be able to draw the Transformer block, explain Q/K/V, write the attention formula, explain causal masking, and describe how logits become a next-token prediction.

🧩 Connection: model families differ mostly by which blocks and masks they use and what objective they train on.

Transformer Model Families 🔴 Exam-core

🧭 This chapter teaches how encoder-only, decoder-only, and encoder-decoder Transformers differ by mask, objective, and task shape.

📌 Why it matters: many exam answers fail by saying “Transformer” without specifying the family and objective.

📚 Clean explanation

🧪 MLM worked example: input tokens vs labels

input tokens: the cat sat on mat
choose masked position: sat
replace selected token with [MASK] or noise according to masking rule
labels = ignore everywhere except selected position
run bidirectional encoder
compute cross-entropy only at selected positions

✅ Exam memory

For each model family, memorize: allowed attention pattern, objective, output shape, and natural tasks.

🧩 Connection: modern LLM discussion mostly focuses on decoder-only and sparse/dense variants, but encoder-only models are still important for embeddings and classification.

Modern Transformer Techniques 🟠 Exam-practical

🧭 This chapter teaches what modern techniques change: context length, serving memory, attention speed, capacity, activation functions, and sparse routing.

📌 Why it matters: these topics explain why production LLMs are not just the 2017 Transformer copied at larger scale.

📚 Clean explanation

🧪 Worked example: top-2 MoE routing

⚠️ Common traps

• FlashAttention is not approximate attention.
• Long context at serving time is not the same as being trained to use long context well.
• GQA reduces K/V heads, not query heads.
• MoE specialization is useful only if routing is balanced and trained well.

✅ Exam memory

For modern techniques, focus on problem → mechanism → trade-off. That is more likely examinable than exact paper-specific benchmark numbers.

🧩 Connection: LoRA and RAG are not core Transformer block changes; they adapt or ground models around the architecture.

LLM Adaptation, RAG, and Limits 🟠 Exam-practical

🧭 This chapter teaches the system-level layer: dense vs sparse models, fine-tuning, LoRA, retrieval augmentation, hallucination, and evaluation.

📌 Why it matters: LLMs are not only architectures; useful systems combine a base model with data, adaptation, retrieval, and evaluation.

📚 Clean explanation

Full fine-tuning updates all model weights, which is powerful but expensive and storage-heavy. LoRA freezes the base weights and trains low-rank adapter matrices, making task adaptation cheaper. RAG tackles a different problem: instead of changing the weights, retrieve relevant documents and feed them into the prompt/context so the model can condition on external evidence.

🧪 Worked example: RAG answer with evidence

input: user question
embed/query the question
retrieve top-k relevant chunks from trusted course corpus
compose prompt with question + retrieved evidence
generate answer with citations to retrieved chunks
evaluate answer for grounding and missing evidence

⚠️ Common traps

• RAG is not a guarantee of correctness; retrieval can miss, rank poorly, or contain bad evidence.
• Fine-tuning changes model behavior; it does not automatically add reliable factual databases.
• RLHF/alignment improves preference-following behavior but does not prove truthfulness.
• Evaluation must match the task: perplexity, accuracy, human preference, factuality, latency, and cost measure different things.

✅ Exam memory

Be able to compare full fine-tuning vs LoRA, explain RAG as retrieval plus generation, and define hallucination without mystifying it.

🧩 Connection: these system ideas sit on top of the Transformer/model-family foundation; they do not replace the need to understand masks, objectives, and attention.

Practical Session Map 🟠 Exam-practical

🧭 This chapter maps the provided notebooks/scripts to the theory chapters.

Exam Toolkit 🔴 Exam-core

Comparison table

Explain in 3 sentences prompts

1. 🔴 Explain why n-grams suffer from state-space explosion.
2. 🔴 Explain Q, K, and V without using the words “magic” or “meaning”.
3. 🔴 Explain why decoder-only Transformers need a causal mask.
4. 🔴 Explain MLM labels and why ignored positions do not contribute to loss.
5. 🟠 Explain why FlashAttention is exact but faster.
6. 🟠 Explain why RAG can reduce hallucinations but cannot guarantee truth.

Likely short-answer questions

1. 🔴 Given a small corpus, compute a bigram probability and one-step generation distribution.
2. 🔴 Given logits, compute softmax probabilities conceptually and identify cross-entropy loss for the correct class.
3. 🔴 Draw the Transformer block and label Q/K/V, MHA, FFN, residual, and normalization.
4. 🔴 Decide whether a task should use encoder-only, decoder-only, or encoder-decoder architecture.
5. 🟠 Compare RoPE, GQA, FlashAttention, and MoE by problem solved.
6. 🟠 Explain LoRA parameter savings for a square weight matrix.

Conceptual traps

• 🔴 Attention scores are not values; values are what get mixed.
• 🔴 Positional encoding is needed because self-attention alone is permutation-insensitive.
• 🔴 Perplexity is not factual accuracy.
• 🟠 KV cache saves compute but costs memory.
• 🟠 MoE reduces active computation, not total parameter count.

Professor-specific weirdness

• 🟠 “De-embedding” means output projection / language modeling head.
• 🟠 “Procedural vectors” maps best to learned embeddings used by downstream procedures.
• 🗑️ Several slides are animation build slides and should not be studied as separate content.

Rapid revision checklist

Glossary 🔴 Exam-core

Source Traceability Appendix 🔴 Exam-core

Every relevant lecture PDF page is represented below. Practical files are listed after the page-level table. Low-text animation/title slides are preserved as traceability rows but often marked low priority or slide noise.

Missing / Weak Material Appendix 🟡 Context

Concepts the slides mention but do not fully explain

• Why attention scaling uses √dₖ and what happens without it.
• Why subword tokenizers are preferred in modern LLMs.
• Exact optimization details for Word2Vec negative sampling.
• How KV-cache tensors are laid out in production systems.
• How RAG systems should evaluate retrieval quality and answer grounding.

Ambiguous or weak slide material

• Several lecture 6 pages are animation build slides with only headings; neighboring pages provide the actual content.
• Lecture 3 has a repeated outline header that makes automatic title extraction weak, but body text was extracted and used.
• RLHF/alignment is visible only lightly in the extracted slides, so it is treated as context rather than deep exam-core.
• Public model landscape slides are useful context but model lists can age quickly and are not memorization targets.

Extraction uncertainty

The following pages had low text extraction and were checked through local rendering or inferred from neighboring pages:

• 1-presentation.pdf p.8: Problem: Models hallucinating (62 chars)
• 1-presentation.pdf p.10: Democratization of technology (63 chars)
• 2-presentation.pdf p.25: Btw: How old is this approach? (74 chars)
• 4-presentation.pdf p.8: Example: Sentiment classification (76 chars)
• 4-presentation.pdf p.13: Prof. Fischer, Neural Language Models 13/25 (43 chars)
• 4-presentation.pdf p.17: RNN unrolled (56 chars)
• 4-presentation.pdf p.20: Text generation with RNNs (69 chars)
• 6-presentation.pdf p.6: Reminder: Computing Dynamic Attention (42 chars)
• 6-presentation.pdf p.8: Attention is All You Need (30 chars)
• 6-presentation.pdf p.9: Attention is All You Need (30 chars)
• 6-presentation.pdf p.55: Attention Layer (21 chars)
• 6-presentation.pdf p.56: Attention Layer (21 chars)
• 6-presentation.pdf p.81: Visualization of Transformers (71 chars)
• 6-presentation.pdf p.85: MHA: Masked vs Non-masked (31 chars)
• 6-presentation.pdf p.86: MHA: Masked vs Non-masked (31 chars)
• 6-presentation.pdf p.104: Revision on Dimensions (28 chars)
• 6-presentation.pdf p.105: Revision on Dimensions (28 chars)
• 6-presentation.pdf p.122: Fin (9 chars)
• 7-presentation.pdf p.90: Fin (71 chars)
• 8-presentation.pdf p.107: Finis (73 chars)

Assumptions made

• Lecture order follows filenames `1-presentation.pdf` through `8-presentation.pdf`.
• Exam relevance is evidence-based: repetition, formulas, algorithms, practical exercises, and standard field importance.
• Generated SVG diagrams are original study aids, not copied slide screenshots.

External Verification Sources 🟡 Context

These public sources were used only to verify or clarify concepts. The professor slides and local practical files determine course scope.

1. Jurafsky & Martin, Speech and Language Processing. Used for: n-gram language models, perplexity, tokenization, and NLP terminology.
2. Bengio et al. (2003), A Neural Probabilistic Language Model. Used for: neural language model framing and embedding lookup intuition.
3. Mikolov et al. (2013), Efficient Estimation of Word Representations in Vector Space. Used for: Word2Vec, CBOW, skip-gram, and vector regularities.
4. Sennrich et al. (2016), Neural Machine Translation of Rare Words with Subword Units. Used for: subword/BPE motivation.
5. Hochreiter & Schmidhuber (1997), Long Short-Term Memory. Used for: LSTM and long-range dependency motivation.
6. Sutskever et al. (2014), Sequence to Sequence Learning with Neural Networks. Used for: encoder-decoder sequence learning.
7. Bahdanau et al. (2014), Neural Machine Translation by Jointly Learning to Align and Translate. Used for: attention as a fix for fixed-vector bottlenecks.
8. Vaswani et al. (2017), Attention Is All You Need. Used for: Transformer architecture, scaled dot-product attention, multi-head attention.
9. Harvard NLP, The Annotated Transformer. Used for: implementation-level Transformer verification.
10. Hugging Face Transformers, tokenizer summary. Used for: BPE, WordPiece, SentencePiece terminology.
11. Hugging Face Transformers, text generation strategies. Used for: greedy, beam, and sampling decoding distinctions.
12. Devlin et al. (2018), BERT. Used for: encoder-only pretraining and masked language modeling.
13. Brown et al. (2020), Language Models are Few-Shot Learners. Used for: decoder-only LLM scale and in-context learning context.
14. Lewis et al. (2020), Retrieval-Augmented Generation. Used for: RAG pipeline and knowledge-intensive task framing.
15. Ouyang et al. (2022), Training language models to follow instructions with human feedback. Used for: RLHF and instruction-following alignment.
16. Kaplan et al. (2020), Scaling Laws for Neural Language Models. Used for: scaling laws and loss trends.
17. Hoffmann et al. (2022), Training Compute-Optimal Large Language Models. Used for: compute-optimal parameter/data trade-offs.
18. Su et al. (2021), RoFormer / RoPE. Used for: rotary positional embeddings.
19. Dao et al. (2022), FlashAttention. Used for: IO-aware exact attention.
20. Ainslie et al. (2023), Grouped-Query Attention. Used for: GQA and KV-cache memory trade-offs.
21. Shazeer et al. (2017), Sparsely-Gated Mixture-of-Experts. Used for: MoE gating and conditional computation.
22. Fedus et al. (2021), Switch Transformers. Used for: top-1 MoE routing and sparse Transformer scaling.
23. Hu et al. (2021), LoRA. Used for: parameter-efficient fine-tuning with low-rank updates.