A self-contained studybook for Alex: concepts explained from first principles, every supplied exercise resolved, every calculation checked, and every priority tied back to the actual course evidence.
The exam note says to expect substantial mathematics and likely diagram questions, while the supplied work is coding-heavy. First build the dependency chain; then switch on “Exam-core only,” reproduce each diagram, and work the numerical exercises without looking.
If matrix shapes, conditional probability, or logarithms feel automatic, skim this. If not, master it before attention or state-space models: nearly every later “hard” formula is a composition of these pieces.
A vector is an ordered list; a matrix is a function from one vector space to another. Multiplication (m×n)(n×p)→(m×p) is legal only when the inner dimensions match. Each result entry is a row–column dot product.
Evidence: 6-presentation-architecture.pdf pp. 2–5; 6-practical.ipynb cell 3.
A language model asks: “given the prefix, how plausible is the next token?” Multiplying those conditional probabilities gives the probability of the whole sequence.
Evidence: 2-presentation-markov.pdf pp. 18, 34–39.
Probabilities are tiny; logs prevent numerical underflow and make products additive. The base changes the unit: natural log gives nats, log₂ gives bits. Perplexity exponentiates average negative log-likelihood back into an “effective branching factor.”
Evidence: Markov pp. 37–39; Word2Vec notebook Questions; log_10.png.
Exponentiate logits and normalize. Subtracting the largest logit leaves the result unchanged but avoids overflow.
Evidence: 4-presentation-rnn.pdf pp. 12–15; 6-presentation-architecture.pdf pp. 106–114.
A Markov model is a directed weighted graph over contexts. It is interpretable and easy to sample, but fixed memory creates state explosion and zero-probability failures.
An order-k model says the next token depends only on the previous k tokens. A bigram uses one-token context; a trigram normally uses two-token context. Count every context→next transition, then normalize outgoing counts.
Evidence: 2-presentation-markov.pdf pp. 3–24; notebook Tasks 2–6.
Perplexity is the inverse geometric mean probability per predicted token. Lower is better only on the same tokenization and evaluation set. A single unsmoothed zero-probability transition makes it infinite.
Evidence: Markov pp. 37–40; notebook Tasks 8–9.
Start at a seed context; sample a next token from its outgoing distribution; append it; shift the context window; repeat until an end condition. Temperature is not inherent to counts, but one can transform probabilities with pᵢ1/T and renormalize.
Evidence: Markov notebook Tasks 5–6; Markov pp. 20–24.
With vocabulary V and k retained context tokens, there are at most |V|k contexts and |V|k+1 directed next-token transitions. Sparse maps store only observed ones. Do not confuse “trigram” (three tokens per probability) with a state that itself stores three tokens.
Evidence: Markov pp. 34–36; notebook Task 7. The slides’ combination count is corrected in the discrepancy ledger.
A representation turns a symbolic token into numbers a model can compare and transform. Count vectors are interpretable but sparse; learned embeddings compress distributional similarity into dense coordinates.
One-hot vectors identify tokens; their dot product is zero for different words, so they encode no graded similarity. Bag-of-words sums token counts and discards order.
Embeddings pp. 9–23.
TF rewards frequency in a document; IDF suppresses terms appearing in many documents. The course uses a smoothed base-10 convention.
Embeddings pp. 24–31; Word2Vec notebook Questions.
CBOW predicts a center token from surrounding context; skip-gram predicts context from a center token. The embedding matrix row is the learned word vector; the output/unembedding matrix need not be identical.
Embeddings pp. 32–43; Word2Vec notebook cells 1–11.
An RNN reuses the same transition at every time step. That parameter sharing handles variable-length sequences; the compressed hidden state and difficult gradient path limit long-range memory.
Shapes: xₜ∈ℝd, hₜ∈ℝH, Wxh∈ℝH×d, Whh∈ℝH×H. Unrolling is a diagram of repeated computation, not separate parameter sets.
RNN pp. 16–21; RNN notebook.
The cell state provides an additive path. Forget gate near 1 retains a component; near 0 erases it. Gates are vectors, so memory is controlled componentwise.
RNN pp. 22–24; bidirectional LSTM notebook.
A bidirectional layer combines left-to-right and right-to-left states when the whole input is known—useful for classification or tagging. A standard autoregressive decoder cannot use future target tokens. The supplied prefix-based generator can be trained, but calling its recurrent layer “bidirectional” does not grant access to unknown future words at inference; it is a pedagogical architecture, not a causal decoder design.
Attention pp. 4–9; bidirectional-LSTM-generator.ipynb cells 1–15.
Instead of compressing a source sequence into one fixed vector, attention lets each query retrieve a weighted mixture of values. Keys determine addressability; values carry the retrieved content.
Scaling matters because a dot product of dₖ roughly unit-variance components has variance proportional to dₖ. Without √dₖ, softmax becomes overly sharp and gradients shrink.
Attention pp. 18–23; Architecture pp. 53–75.
A causal mask adds −∞ to scores for future key positions before softmax. Those probabilities become zero. A padding mask suppresses dummy tokens; it is a different mask for a different reason.
Architecture pp. 85–89.
The block alternates token mixing (attention) and channel mixing (the position-wise feed-forward network), stabilized by residual paths and normalization. Track the sequence axis n and model axis d throughout.
Lookup gives X∈ℝn×d. Add a position vector of the same shape; addition preserves n×d. Sinusoidal encoding alternates sine/cosine frequencies.
Architecture pp. 11–52.
Each head projects X into smaller Q,K,V spaces, retrieves a different relation, then concatenation returns n×(h·dᵥ). Wᴼ maps that back to d.
Architecture pp. 53–84.
The same two-layer MLP is applied independently to every position. Residual addition requires equal shapes. LayerNorm normalizes across each token’s feature dimension, then learns scale γ and shift β.
Architecture pp. 90–105.
The base equations stay recognizable; modern systems change position encoding, share attention projections, constrain connectivity, improve kernels, and activate only part of the feed-forward capacity.
Rotary position embedding rotates pairs of query/key coordinates by a position-dependent angle. Their dot product then depends on relative displacement; values are not rotated in the basic form.
Technique pp. 49–60.
Grouped-query attention keeps many query heads but shares fewer key/value heads across groups. It reduces KV-cache memory and decoding bandwidth between MHA and multi-query attention.
Technique pp. 61–69.
Sliding windows reduce the number of connected pairs. FlashAttention computes exact attention with tiled memory-efficient kernels; it is not an approximation and does not change the mathematical output.
Technique pp. 70–76.
GELU smoothly gates by input magnitude. SwiGLU splits the hidden projection: one branch is SiLU-gated and multiplied elementwise by another, then projected back.
Technique pp. 77–78.
A router scores experts per token, activates top-k, normalizes selected weights, and sums expert outputs. Sparse activation increases parameter capacity without proportional per-token compute; load-balancing loss fights expert collapse.
Technique pp. 79–89; Models pp. 55–84; 04_moe_layer.py.
Context-length charts and named model tables establish why efficiency matters, but memorizing volatile product numbers has little evidence value compared with deriving attention and routing.
Technique pp. 35–48; Models pp. 42–54.
Bidirectional self-attention produces contextual token representations. BERT-style corruption selects 15% of tokens: of those selected positions, 80% become [MASK], 10% a random token, and 10% remain unchanged; loss is computed only at selected positions.
Models pp. 14–33; practical Parts 1–3. Original reference: Devlin et al., BERT (2018).
A triangular mask permits each position to attend only leftward. Teacher forcing trains all next-token positions in parallel; generation is sequential because each new sample becomes part of the next prefix.
Models pp. 34–41; Architecture pp. 115–119.
Full fine-tuning updates all weights. LoRA freezes W and learns a low-rank update ΔW=BA with rank r≪d, reducing trainable parameters and optimizer memory. At inference the update can be merged.
Models pp. 98–107. Original reference: Hu et al., LoRA (2021).
Memory includes parameters, gradients, optimizer states, activations, and temporary buffers. Data parallelism replicates the model; tensor/pipeline parallelism partitions computation; ZeRO partitions optimizer state, gradients, then parameters by stage.
Models pp. 85–97.
SSMs start as dynamical systems: an input drives a latent state and the state produces an output. Discretization turns the continuous equation into a sequence recurrence; unrolling reveals a convolution for parallel training.
For a mass–spring–damper, choose state h=[position, velocity]ᵀ. Then velocity is the derivative of position and acceleration comes from mÿ+cẏ+ky=x(t).
SSM pp. 30–49.
Under zero-order hold with step Δ: Ā=eΔA and B̄=A−1(eΔA−I)B when A is invertible. Bilinear/Tustin is a different approximation: Ā=(I−ΔA/2)−1(I+ΔA/2).
SSM pp. 42–63. The slide shorthand is clarified here.
With h₋₁=0, unroll: y₀=CB̄x₀; y₁=CĀB̄x₀+CB̄x₁. Thus kernel K=[CB̄,CĀB̄,CĀ²B̄,…] convolves with x. Compute K once and train in parallel; step the recurrence for constant-memory inference.
SSM pp. 64–77.
A fixed linear time-invariant kernel cannot decide to store one token and ignore another based on content. Mamba makes discretization/input/output parameters input-dependent and uses a parallel associative scan. Selectivity improves content routing but gives up a single fixed convolution kernel.
SSM pp. 98–124. Original reference: Gu & Dao, Mamba (2023).
Chunk documents, preserve metadata, embed chunks, and index them. Dense nearest-neighbor search finds semantic candidates; HNSW trades exactness for fast graph traversal. A cross-encoder can rerank a small candidate set more accurately than a bi-encoder.
RAG pp. 11–25.
If the needed passage was not retrieved, fix chunking/query/index/search. If good evidence was retrieved but the answer is wrong, fix prompt grounding, context ordering, model behavior, and citation checks. Measure recall@k separately from answer faithfulness.
RAG pp. 26–31. Original reference: Lewis et al., RAG (2020).
Specify constraints and acceptance tests; ask for small reviewable changes; run linters, types, unit/integration/security tests; inspect the diff; verify dependencies and licenses; keep a human accountable for deployment.
AI programming pp. 15–29.
Product timelines show rapid capability change, but exact vendor features age quickly. The durable exam lesson is the workflow: prompts do not replace specifications, testing, provenance, or threat modeling.
AI programming pp. 1–14.
Numbering preserves the source notebook task labels or PDF page prompts. Where a source depends on an absent corpus or large model download, the solution gives executable reference code, deterministic shape/value checks, and the result that can be established without inventing a training run.
Vocabulary |V|=10,000; each token ID is a 64-bit integer (8 bytes).
In the standard convention, a trigram probability predicts the third word from the previous two. Therefore context states: |V|²=10⁸. Possible directed (context,next) transitions: |V|³=10¹². If the task literally calls each 3-word tuple a “state,” there are 10¹² such tuples.
Storing every 3-ID tuple explicitly costs 10¹²×3×8 = 24×10¹² bytes = 24 TB decimal (≈21.83 TiB), before hash-table counts/pointers. Storing all possible standard trigram transitions also needs 10¹² counts; at 8 bytes/count alone, 8 TB.
Using the course convention that includes the initial-word probability: P=½×¼×⅛=1/64. N=3, so PPL=(64)1/3=4.
The supplied graph has no transition from “like” to “you”; its probability is zero. Hence NLL=∞ and PPL=∞.
Load 2–5 UTF-8 books; lowercase/tokenize; train TF–IDF over book documents and Word2Vec over sentence-token lists; obtain dense vectors; reduce with UMAP; visualize. The source’s UMAP line must call fit_transform, and a downloaded gensim model must be assigned from api.load.
Word2Vec learns dense distributional similarity: words used in similar contexts can be close even if they are different tokens. TF–IDF is sparse, document-level, and primarily measures discriminative occurrence; it does not directly learn semantic neighborhoods. Word2Vec sacrifices direct interpretability and still gives one vector per word type.
D2 has five tokens and “listened” once: TF=1/5=0.2. It occurs in two of three documents: df=2, N=3. With the lecture’s formula, IDF=log₁₀(1+3/2)=log₁₀(2.5)=0.39794. Multiply: 0.2×0.39794=0.079588.
The class requires a text path, omitted in one notebook cell: rnn = RNNGenerator("input.txt", mode="word"). Train multiple seeds/epochs with fixed random seeds; compare held-out NLL/perplexity and generated coherence.
| Model | Memory | Strength | Failure |
|---|---|---|---|
| Markov | explicit fixed context | interpretable counts, cheap training | sparsity and state explosion |
| RNN | learned fixed-width hidden state | shares statistical strength across contexts | sequential training, vanishing/exploding gradients |
Use LSTM/GRU gates, gradient clipping, dropout, validation-based early stopping, pretrained subword tokenization, larger/better-cleaned corpus, and temperature/top-k sampling. Evaluate perplexity on the same held-out tokens.
Denominator = e¹+e⁰+e²+e¹ = (e+1)² ≈ 13.8256. Therefore softmax ≈ [0.196612, 0.072329, 0.534447, 0.196612]. Sum=1.
Concatenate [hₜ₋₁,xₜ], apply affine transform and sigmoid: fₜ=σ(Wf[hₜ₋₁,xₜ]+bf). Then retained old memory is fₜ⊙cₜ₋₁. Values are in (0,1); all vectors have cell-width H.
The tokenizer maps vocabulary strings to integer IDs and reserves 0 for padding; IDs are categorical labels, not magnitudes. A larger/cleaner corpus improves coverage but increases vocabulary/training cost. Training examples are prefixes→next-token; left padding makes equal batch lengths, and masking should prevent padding from affecting the sequence representation.
Sequence length fixes the maximum prefix window; output_word controls generated steps; tokenizer must match training. Temperature T<1 sharpens, T>1 diversifies. Compare model size/epochs with validation loss, held-out perplexity, repeated-seed generation, and overfitting gap.
For A=[[2,4,−1],[3,1,5]], A is 2×3, so AAᵀ is 2×2 and AᵀA is 3×3.
Example off-diagonal: row₁·row₂ = 2·3+4·1+(−1)·5=6+4−5=5.
Test identity, a hand-computed rectangular case, and incompatible shapes. For A above, compare against the two verified Gram matrices in the preceding solution.
For pos=0,d=4, PE=[0,1,0,1]. For pos=1, approximately [0.84147,0.54030,0.01000,0.99995].
Project attention output to d if needed, add X, then LayerNorm over the last dimension. Mean should be ≈0 and variance ≈1 per token before learned γ/β change it.
FFN(x)=W₂·ReLU(W₁x+b₁)+b₂, d→dff→d; apply independently to all n rows; add residual and normalize.
Unembedding WU∈ℝd×|V| maps each token state to logits; softmax along vocabulary yields probabilities. Decode greedy argmax or sample with temperature/top-k.
Sentence: “the small cat is outside”; n=5,d=4, vocabulary size 6. (1) Tokenize to five IDs. (2) Select the five corresponding rows of E∈ℝ6×4, giving X∈ℝ5×4. (3) Compute PE for positions 0…4, giving 5×4. (4) Add elementwise X′=X+PE. (5) Output stays 5×4.
Verified PE rows begin [0,1,0,1], [0.84147,0.54030,0.01000,0.99995], [0.90930,−0.41615,0.02000,0.99980]. The fifth position must also be present even though an incremental slide view visually truncates the table.
Pooling: CLS uses one designated token; mean pooling averages non-padding tokens; max pooling takes featurewise maxima after masking pads. MLM prevents a trivial copy identity because most selected targets are hidden or changed; unchanged/random cases reduce the train–test mismatch of seeing [MASK]. Only selected labels contribute to loss.
Obtain the static input embedding row for each token ID and the contextual last-hidden-state at its occurrence. Reduce a stacked array with PCA/UMAP, keeping labels and sentence contexts aligned.
Static vectors give one point per vocabulary item. Contextual vectors move with sentence meaning and layer. Dimensionality reduction can distort global distances; compare neighborhoods and multiple seeds, not just a pretty 2-D plot.
Freeze the encoder for a cheap baseline, then unfreeze with a smaller encoder learning rate. Use stratified train/validation splits, macro-F1 for imbalance, and early stopping. The classifier head maps hidden width→2 logits.
Top-k makes routing sparse; k=1 is cheapest but brittle, larger k improves mixture/redundancy. Expert collapse means a few experts receive most tokens; use an auxiliary importance/load loss, router noise, and capacity limits. Capacity factor sets buffer size relative to average tokens per expert; overflow tokens may be dropped or rerouted.
Stable examples, described by architectural/use distinctions rather than volatile leaderboard claims: (1) GPT-family decoder-only causal models—general generation/tool use; (2) BERT-family encoder-only masked models—representation/classification; (3) T5-family encoder–decoder denoising models—conditional sequence-to-sequence tasks. A valid three-LLM answer can instead name current systems, but must state objective, architecture/access, context/tool modality, and intended use with a dated source.
mÿ+cẏ+ky=x. Let h₁=y, h₂=ẏ. Then ḣ₁=h₂ and ḣ₂=−(k/m)h₁−(c/m)h₂+(1/m)x.
For zero-order-held input over Δ: Ā=eΔA, B̄=∫₀ΔeτAB dτ. If A invertible, B̄=A⁻¹(Ā−I)B.
With h₋₁=0: h₀=B̄x₀; h₁=ĀB̄x₀+B̄x₁; h₂=Ā²B̄x₀+ĀB̄x₁+B̄x₂. Multiply C to obtain the convolution coefficients.
If A=VΛV⁻¹, then Ak=VΛkV⁻¹. Transform B′=V⁻¹B and C′=CV. Each diagonal mode evolves independently: Kk=Σᵢ C′ᵢ λᵢkB′ᵢ. This replaces repeated dense matrix powers with elementwise powers of eigenvalues.
Embed query q with the same model as chunks, score cosine similarity q·d/(‖q‖‖d‖), retrieve top-k. For normalized vectors this is just dot product. Use HNSW for fast approximate search at scale.
Retrieve the smallest set of chunks that jointly contains sufficient evidence while fitting the model context. Tune chunk boundaries/overlap, k, filters, hybrid dense+sparse search, and reranking on labeled queries.
Log query, candidates, scores, reranked set, final context, answer, and cited spans. If gold evidence absent from candidates, retrieval failed; if present but unused/misrepresented, generation/grounding failed.
No separate formula sheet was supplied. This is an independently assembled cheat-sheet companion derived from repeated course evidence—compress it onto the permitted handwritten A4 page in your own notation.
P(sequence)=∏ conditional probabilities
NLL=−Σln ptrue
PPL=exp(NLL/N)
CE=−ln ptrue
softmax(z)ᵢ=ezᵢ−max z/Σezⱼ−max z
Q=XWQ, K=XWK, V=XWV
S=QKᵀ/√dₖ → n×n
A=row-softmax(mask(S)) → n×n
Y=AV → n×dᵥ
MHA=Concat(heads)Wᴼ
X←Embed(ids)+PE
X←LN(X+MHA(X))
X←LN(X+FFN(X))
logits=XWU; p=softmax(logits)
PE(pos,2i)=sin(pos/100002i/d)
hₜ=tanh(Wxₜ+Uhₜ₋₁+b)
cₜ=fₜ⊙cₜ₋₁+iₜ⊙ĝₜ
hₜ=oₜ⊙tanh(cₜ)
Gate values ∈(0,1)
ḣ=Ah+Bx; y=Ch+Dx
hₖ=Āhₖ₋₁+B̄xₖ
K=[CB̄,CĀB̄,CĀ²B̄,…]
train: convolution; infer: recurrence
Inner matrix shapes match.
Probability rows sum to 1.
Residual operands have equal shape.
Causal future weights are 0.
Gram matrices are symmetric.
Lower PPL requires same tokenization.
All physical pages are preserved because several decks use incremental-reveal exports. The full machine-auditable map is alongside the extraction workspace in .studybook_work/source_map.json; the table below is the human index.
| Source | Reviewed | Book destinations | Strongest evidence |
|---|---|---|---|
| 1-presentation-intro.pdf | 14/14 pages | Scope | course flow pp.13–14 |
| 2-presentation-markov.pdf | 43/43 | Markov, solutions | graphs pp.3–24; scaling/perplexity pp.34–40 |
| 3-presentation-word-embedding.pdf | 48/48 | Embeddings, solutions | TF–IDF pp.24–31; Word2Vec pp.32–43 |
| 4-presentation-rnn.pdf | 25/25 | RNN/LSTM | softmax pp.12–15; recurrence/gates pp.16–24 |
| 5-presentation-attention.pdf | 24/24 | Attention | encoder–decoder/attention pp.10–23 |
| 6-presentation-architecture.pdf | 122/122 | Transformer, solutions | numerical derivations pp.2–5, 27–119 |
| 7-presentation-technique.pdf | 90/90 | Modern techniques | end-to-end pp.3–34; RoPE/GQA pp.49–69; MoE pp.79–89 |
| 8-presentation-models.pdf | 107/107 | Families/adaptation | MLM pp.14–33; MoE pp.55–84; training/LoRA pp.85–107 |
| 9-presentation-ssm.pdf | 130/130 | SSM/Mamba, solutions | derivation pp.30–77; HiPPO/Mamba pp.85–124 |
| 10-presentation-rag.pdf | 32/32 | RAG, solutions | retrieval and pipeline pp.11–30 |
| 11-pres-ai-assisted-programming.pdf | 29/29 | AI programming | failure/risk/workflow pp.15–29 |
| 7 notebooks, including 2 checkpoints | 115/115 cells | Complete solutions | all Tasks, Questions, code, and visible outputs |
| 6 Python files | all lines | Complete solutions | support code and four Transformer practicals |
| 2 PNG + 1 draw.io | all | Markov/embeddings | simple graph and log plot |
| README.txt + practical README.md | all lines | scope/solutions | exam constraints and task definitions |
| 2 user-supplied mock-exam photographs | both full-resolution images | Mock cheat sheet | topic coverage and independently corrected calculations |
| Source locator | Observed issue | Resolution in this book |
|---|---|---|
| Architecture pp.32–34 | The displayed MHA result does not equal the displayed concatenated heads multiplied by the displayed Wᴼ. | Recomputed first row as [0.46,0.50,0.40,0.37], not [0.39,0.45,0.38,0.31]; full corrected matrix appears on Cheat Side C. |
| Architecture pp.2–5 | AAᵀ off-diagonal printed as 7. | Recomputed as 5; symmetry, trace, and NumPy confirm. |
| 6-practical §4 | Comments reverse attention-weight and attention-output shapes. | Weights n×n; output n×dₖ. |
| 6-practical §8 | def MyTransformer(nn.Module) is invalid class syntax. | Complete solution uses class. |
| Markov pp.34–36 / Task 7 | “trigram state” and combination-based transition counts mix conventions and overcount allowed directed transitions. | Standard context-state and literal tuple interpretations both stated with storage. |
| Markov Tasks 8–9 | Whether the initial word is predicted changes N and PPL. | Use the slide p.39 convention including the initial probability; alternate shown. |
| Embeddings p.29 | Course IDF is a nonstandard smoothed base-10 form. | Exercise uses it exactly and reports the unsmoothed alternative separately. |
| RNN slide p.18 | “No limited context problem” overstates vanilla RNN ability. | Clarified: formal recurrence is unbounded, but fixed state and gradients limit practical memory. |
| RNN notebook | Constructor call omits required textfile. | Corrected instantiation supplied. |
| Word2Vec notebook | Ignored api.load result; fictitious local load; UMAP uses fit rather than fit_transform. | Corrected reproducible pipeline supplied. |
| Bidirectional LSTM notebook | Bidirectionality can be mistaken for causal access to future output tokens. | Clarified its use over known prefixes and limitation for standard generation. |
| SSM pp.42–63 | Laplace derivative wording and discretization shorthand are misleading. | Separated Laplace derivative, z-delay, exact zero-order hold, and Tustin formulas. |
| Practical README | Part 3 says Part 4 covers LoRA; supplied Part 4 is MoE. | MoE practical solved; LoRA covered from model slides, not mislabeled as the practical. |
| Notebook experiments | Corpora and heavyweight pretrained downloads are not bundled. | Complete corrected code, deterministic calculations, shape invariants, and evaluation contracts supplied; no fabricated training metrics. |
Sides A–B mirror the topic coverage in the two photographed mock-exam sheets. Side C adds the highest-yield by-hand calculations from the supplied practicals and worked lecture exercises. All arithmetic and notation are independently checked.
V=vocabulary size; n=sequence length; d=model width; h=query heads; dₖ=d/h normally; B=batch size.
X: n×d W: d×r XW: n×rExam sequence: asked quantity → givens → symbol meanings → last formula → missing intermediates → numbers → range/shape check.
Order-k assumption: next token depends only on last k tokens.
Dense maximum: contexts=Vᵏ; context→next transitions=Vᵏ⁺¹.
pₜ=probability assigned to the actual token; N=number of predicted tokens in the chosen convention. Lower is better only with same data/tokenization.
|d|=tokens in document; N=documents; df=documents containing exact token t.
Checks: pᵢ∈(0,1); Σpᵢ=1; largest logit → largest p; loss≥0.
One-hot O: n×V; embedding table Wᴱ: V×d.
X=O·Wᴱ (n×V)(V×d)=n×dDifferent words have different rows. Synonyms may learn nearby vectors, never assume equality. Static Word2Vec gives one vector/word; contextual models give one vector/occurrence.
PE(pos,2i)=sin(pos/100002i/d)Checks: Gram matrix symmetric; trace(AAᵀ)=trace(AᵀA)=56.
Q,K:n×dₖ; S,A:n×n; V,Y:n×dᵥ
Meaning: Q asks; K is address; V is returned content. Softmax is across keys for each query row.
Causal: forbid j>i (future). Padding: forbid pad keys. Apply mask before softmax. Purpose: no future leakage; valid autoregressive generation.
Rotate paired Q/K coordinates; values unchanged. Dot products depend on relative position gap.
Same relative gap → same relative rotation; supports length extrapolation better than fixed learned absolute positions.
For head r: Qᵣ=XWQᵣ, Kᵣ=XWKᵣ, Vᵣ=XWVᵣ; headᵣ=Attention(Qᵣ,Kᵣ,Vᵣ).
H=Concat(head₁…headₕ) n×(h·dᵥ)Concatenate along feature axis, never sequence axis. Different heads can learn different relations.
LN mean/variance are across features of each row/token. Residual preserves signal/gradient path; FFN mixes channels independently per token.
For next-token generation use the last sequence row Z[n−1,:], then greedy argmax or sampling. Weight tying may use Wᵁ=(Wᴱ)ᵀ.
Select 15% token positions. Of selected:
Non-selected labels ignored (often −100). Bidirectional encoder sees left+right context; corruption prevents trivial copying and reduces [MASK] mismatch.
| Bi | Cross | |
|---|---|---|
| Input | q,d separately | [q;d] jointly |
| Score | cos/dot | learned pair score |
| Docs | precompute | runtime |
| Use | fast retrieval | accurate rerank |
Typical pipeline: bi-encoder top-k → cross-encoder rerank small candidate set.
ANN/HNSW approximates nearest neighbors faster; measure recall@k.
If gold passage absent: retrieval failure. If present but answer wrong: generation/grounding failure. Chunk size/overlap, k, filters, hybrid dense+sparse search are tunable.
hq query heads but fewer hkv K/V heads; group size=hq/hkv. Queries stay distinct; K/V cache shared within group.
MHA: hq=hkv. MQA: hkv=1. GQA lies between quality and cache efficiency.
f≈1 retain, f≈0 erase. LSTM additive cell path helps gradients. Bidirectional uses future known input; not valid future-token access for causal generation.
MoE: router → top-k experts → normalize selected gates → weighted sum. Risk: expert collapse; use load-balancing/capacity. LoRA freezes W and trains low-rank A,B; fewer trainable parameters/optimizer states.
Training: convolution K*x in parallel. Inference: recurrence with constant state. Mamba makes parameters input-dependent (selectivity) and uses parallel scan. Never mix continuous A,B with discrete Ā,B̄.
Probability: rows sum 1; loss≥0; PPL≥1.
Shapes: inner dimensions match; residual operands equal.
Mask: before softmax; forbidden weight 0.
Matrix: Gram symmetric.
LN: per-token features; population variance.
Answer: final value + interpretation + source convention.
Asked: symbol + shape/unit. Given: translate data to symbols. Formula: target on left. Work: substitute one dependency at a time. Answer: boxed value + meaning. Check: one shape/range/sum test.
givens → intermediate(s) → requested value → interpretationNever invent a learned matrix. If WQ/WK/WV/WO/WU values are absent: give shapes and leave the result symbolic unless identity is explicitly stated.
markov-generator.ipynb cell 17; Markov pp.34–36
Given: |V|=10,000; trigram predicts wₜ from two previous words; uint64 ID=8 B.
contexts=V²=10⁸Check: one more token multiplies dense possibilities by V=10⁴; observed sparse graph is far smaller.
6-practical §§Embedding–PE; Architecture exercise pp.20–24
Given: vocab V=6; “the small cat is outside”; n=5,d=4.
O:5×6; Wᴱ:6×4; E=OWᴱ:5×4One-hot multiplication selects rows: E[i,:]=Wᴱ[token_idᵢ,:]. PE rows start:
Exam answer: “Add elementwise; do not concatenate. Vocabulary controls Wᴱ rows; sequence length controls E rows.”
Architecture pp.35–37; 6-practical Attention (i–iv)
S=QKᵀ/√2 + M; A=row-softmax(S); Y=AVWith the supplied Q,K (5×2), the causal weight matrix is approximately:
Supplied v₁=[.5,.1],v₂=[.2,.7],v₃=[.4,.3]. Thus y₁=v₁=[.5,.1]; y₂≈.33v₁+.67v₂=[.30,.50]; y₃≈(v₁+v₂+v₃)/3=[.367,.367].
Checks: each A row sums 1; all j>i weights are 0; S,A are 5×5, Y is 5×2. Building S causes O(n²) attention storage.
Architecture pp.32–34; 6-practical Attention (vi)
Four supplied heads are each 5×2. Join the same token row across heads:
C=Concat(head₁…head₄):5×8Verified result from the displayed C and Wᴼ:
The slide’s displayed 5×4 result is arithmetically inconsistent. Use row×column multiplication above.
6-practical Residual/Norm + FFN; Architecture pp.38–41
R₁=X⁽⁰⁾+MHA(X⁽⁰⁾); N₁=LNrow(R₁)X,MHA,R₁,N₁,F,R₂,H⁽ᴸ⁾: n×d
W₁:d×dff; W₂:dff×d
Shorthand answer: “Residual operands match; LN uses μ,σ² over the d features of each token row; FFN changes d→dff→d independently per token.”
Architecture pp.42–45; 6-practical De-embedding (i–ii)
H⁽ᴸ⁾:5×4; Wᵁ:4×6; Z=H⁽ᴸ⁾Wᵁ:5×6Supplied final hidden row h=[.5,.2,.1,0]. Example token-ID 4 logit:
Wᵁ is learned unembedding/vocabulary projection; it produces V logits. Weight tying may set Wᵁ=(Wᴱ)ᵀ.
Technique walkthrough pp.3–34; Architecture pp.45–46
Lₜ=−ln pₜ(true next token); L=meanₜ LₜGreedy argmax can use logits directly; sampling requires softmax probabilities. Training compares every allowed position with its shifted true target.
SSM pp.64–77
h₋₁=0; h₀=B̄x₀Multiply by C: K=[CB̄,CĀB̄,CĀ²B̄,…], so y=K*x. Check: power counts delay; newest x has Ā⁰.
Coverage evidence: Sides A–B use two user-provided photographs from a friend who attended the mock exam, visually reviewed on 14 July 2026; the photographs are not instructor-endorsed solutions. Side C uses the cited course notebook cells and lecture pages. All displayed formula values were independently checked against the course sources and the studybook’s numerical verification ledger.
The calculator can execute arithmetic; it cannot decide what the symbols mean or which formula comes first. Use this page to perform that translation explicitly. Do not touch the calculator until steps 1–5 are written on paper.
| Line | What you write | Example: “Find the perplexity of I want coffee” |
|---|---|---|
| 1 · Asked | One output, with symbol and shape/unit. | PPL, one positive dimensionless number. |
| 2 · Given | Translate every supplied number/object into a symbol. | p₁=½, p₂=¼, p₃=⅛; N=3 predicted words. |
| 3 · Meaning | Write a one-line dictionary for every symbol. | pₜ is the probability assigned to word t; N is how many such probabilities are averaged. |
| 4 · Target formula | Formula with the asked symbol alone on the left. | PPL=(∏ₜpₜ)−1/N. |
| 5 · Dependency order | List intermediate values from givens to target. | product p → reciprocal → N-th root. |
| 6 · Substitute | Numbers in parentheses before calculating. | PPL=[(½)(¼)(⅛)]−1/3. |
| 7 · Check | Range, shape, sign, normalization, and interpretation. | Product=1/64; cube root of 64 is 4; PPL≥1, so plausible. |
Draw a tiny dependency chain. Every arrow means “must be known before.”
If an intermediate formula uses another unknown, add another step to the left. Never substitute all formulas into one enormous calculator expression until the chain works on paper.
Exam method derived from the repeated worked calculations in Markov pp.37–39, Architecture pp.48–119, Technique pp.3–34, and SSM pp.30–77.
| Formula family | Symbols in plain English | Calculation order | Fast check |
|---|---|---|---|
| TF–IDF TF·IDF | count(t,d): occurrences of term t in document d; |d|: token count in d; N: number of documents; df(t): documents containing t. | count and |d| → TF; N and df → IDF; TF×IDF. | Common-in-all-documents term should have smaller IDF than a rare term. |
| Softmax ezᵢ−m/Σezⱼ−m | zᵢ: logit for class i; m: largest logit in the same row; j: denominator index over all classes. | find m → subtract m from every logit → exponentiate → sum → divide each exponential by the same sum. | All outputs in (0,1), sum≈1, largest logit gets largest probability. |
| Cross-entropy −ln ptrue | ptrue: softmax probability at the correct class; “true” is an index, not the value 1. | logits → softmax → locate target class → select its probability → −ln. | Better correct-class probability means smaller non-negative loss. |
| Perplexity exp[−(1/N)Σln pₜ] | t: predicted-token position; pₜ: probability assigned to the actual token; N: number of predicted tokens included. | collect one pₜ per target → ln each → sum → divide by N → negate → exp. | A zero probability gives infinite unsmoothed PPL; lower is better only under the same tokenization. |
| Matrix product Cij=ΣₖAikBkj | i: output row; j: output column; k: paired position across A’s row and B’s column. | check inner shapes → choose output cell (i,j) → multiply paired entries → sum → repeat cells. | (m×n)(n×p)→m×p. A Gram matrix AAᵀ is symmetric. |
| Attention softmax(QKᵀ/√dₖ)V | n: tokens; d: model width; dₖ: key/query width; Q: what each token seeks; K: what each token offers as an address; V: content returned. | XWQ/XWK/XWV → QKᵀ → ÷√dₖ → add mask → row-softmax → weights·V. | scores/weights n×n; output n×dᵥ; each weight row sums to 1. |
| Position encoding | pos: position starting at 0; j: embedding column; i=⌊j/2⌋: frequency-pair index; d: total embedding width. | determine even/odd j → compute i → exponent 2i/d → denominator → angle → sin if even, cos if odd. | At pos=0, every sin entry is 0 and every cos entry is 1. |
| LayerNorm | xᵢ: one token’s feature i; d: feature count; μ: that token’s feature mean; σ²: population variance; ε: small stability constant; γ,β: learned scale/shift. | μ → deviations → squared deviations → average for σ² → add ε → square root → normalize each xᵢ → apply γ,β. | Before γ,β, normalized feature mean≈0 and population variance≈1. |
| Discrete SSM hₖ=Āhₖ₋₁+B̄xₖ; yₖ=Chₖ | k: time step; hₖ₋₁: old state; xₖ: current input; Ā: state transition; B̄: input injection; C: readout. Bars mean already discretized. | Āhₖ₋₁ → B̄xₖ → add for hₖ → Chₖ for yₖ → carry hₖ forward. | Never mix continuous A,B with discrete Ā,B̄ in one recurrence. |
| Cosine similarity | q,d: query/document vectors; q·d: paired-product sum; ‖q‖,‖d‖: Euclidean lengths. | dot product → each squared-length sum → roots → multiply lengths → divide → repeat candidates → rank. | For nonzero real vectors, result lies in [−1,1]. |
Question data: p=[½,¼,⅛], N=3. Paper sequence: product → inverse → cube root.
Calculate app entry:
Press EXE. Result: 4. A log-form entry gives the same result:
Which form should you use? Direct product is quickest for three clean fractions. Log form is safer for many tiny decimal probabilities. The alternate eˣ function is printed above ln; insert it with SHIFT then ln if using the keypad.
Question data: “listened” occurs once in a 5-token document; N=3 documents; df=2. The course uses log₁₀(1+N/df).
Paper sequence: TF=1/5; IDF=log₁₀(1+3/2); multiply.
In Calculate, the ordinary log function is base 10; press EXE → 0.079588…. Do not use ln here—the base is part of the course formula.
Question data: z=[1,0,2,1]. First write m=max(z)=2. Then shifted logits are [−1,−2,0,−1].
Result: [0.196612, 0.072329, 0.534447, 0.196612]. Add the four rounded values: approximately 1.
If class 3 is correct, select ptrue=0.534447 and enter (−)ln(0.534447) → loss ≈0.6265. If ptrue=0.206 in another task, (−)ln(0.206)≈1.58.
Question: pos=1,d=4. For columns j=0,1 use pair i=0 and angle 1/10000⁰=1. For j=2,3 use i=1 and angle 1/100002/4=0.01.
The fx‑991CW Matrix app supports matrices up to 4×4—enough for the course’s small hand examples, not real model tensors.
For A=[[2,4,−1],[3,1,5]], calculate MatA×Trn(MatA) → [[21,5],[5,35]]. Then calculate Trn(MatA)×MatA → the verified 3×3 result in the solution manual.
For one token vector only, you may use HOME → Statistics → 1-Variable. Enter its d feature values as observations.
Use Statistics only for the arithmetic. On paper you must still state that normalization is across the features of one token, not across tokens or the batch.
Calculator operations verified against Casio’s official fx‑570CW/fx‑991CW online user guide: Calculator Apps, Matrix Calculations, Function Analysis, Memory Functions, Result Format, and Calculation Priority Sequence. Menu labels may be faster and less error-prone than memorizing icon positions.