🔴 Exam relevance: high
Language model evaluation
Perplexity, calculated from corpus counts
Perplexity measures average surprise. Lower perplexity means the model assigns higher probability to the observed words.
1 Tiny training corpus
- <s> you like tea </s>
- <s> you like coffee </s>
- <s> you drink tea </s>
- <s> i like tea </s>
Test sequence W = "you like tea" with N = 3 predicted words.
2 Bigram rule for p
Model: use previous word as context.
P(next | context) = count(context, next) / count(context)
This is an unsmoothed maximum-likelihood estimate: just count from the corpus, then divide.
3 Where each probability comes from
| Needed probability | Count evidence in corpus | Calculation | p |
|---|---|---|---|
| P(you | <s>) | 4 sentences start after <s>; 3 start with "you". | count(<s>, you) / count(<s>) = 3 / 4 | 0.75 |
| P(like | you) | "you" appears 3 times; it is followed by "like" 2 times. | count(you, like) / count(you) = 2 / 3 | 0.667 |
| P(tea | like) | "like" appears 3 times; it is followed by "tea" 2 times. | count(like, tea) / count(like) = 2 / 3 | 0.667 |
4 Multiply the chain, then invert-root
you
3 / 4
from sentence starts
like
2 / 3
from words after "you"
tea
2 / 3
from words after "like"
P(W) = P(you|<s>) · P(like|you) · P(tea|like)
P(W) = (3/4) · (2/3) · (2/3) = 1/3 ≈ 0.333
ppl(W) = P(W)-1/N = (1/3)-1/3 = 31/3
5 Final answer
The model assigns probability 1/3 to the three-word sequence "you like tea". Its perplexity is therefore:
ppl("you like tea") ≈ 1.442
Interpretation: on average, the model is about as uncertain as choosing among 1.44 equally likely next words at each step.
1.44
On paper recipe: count the relevant transitions, divide to get probabilities, multiply them, then take the negative N-th root.
Important: if a needed bigram never appears, unsmoothed probability becomes 0 and perplexity blows up. Smoothing fixes this.
Boundary tokens: this card predicts only the 3 words. If the exam includes </s>, include that probability and adjust N.