Top-k Selection

Picture a leaderboard for 'what word comes next'. Instead of only ever crowning a single number-one word, or weighing every possible word in the language, you just keep the top handful of contenders, say the top 3, and ignore the long tail of unlikely options. This is top-k selection: a filtering step language models use before choosing the next word. You keep only the k highest-scoring candidates, then decide among just those.

1. 1Score every candidate, meaning every possible next word, with a number. A higher number means the model thinks that word is more likely to come next.
2. 2Sort the candidates by score, highest first.
3. 3Keep only the top k candidates and throw away the rest, no matter how good or bad they were.
4. 4Turn the kept candidates' scores back into probabilities, for example with softmax (see Softmax & Temperature), using only this smaller shortlist. Then pick from among them.

Use this to control text generation so a model cannot wander into extremely unlikely, nonsensical words. It is a common safety net used alongside temperature, often set to something like k=40 or k=50 in real systems.

• ⚠Don't set k too small, like k=1. That is the same as always picking the single highest-scoring word every time, which can make text repetitive and robotic.
• ⚠Don't set k too large, close to the whole vocabulary. Top-k then stops doing anything useful, since you are barely filtering out any of the unlikely candidates.
• ⚠Don't forget to re-score, or renormalize, the probabilities after cutting candidates out. The leftover scores no longer add up to 100% unless you recompute them over just the shortlist.