Your is more than a document—it is a rite of passage. It demystifies the black box. It proves that the foundations of large language models are accessible, teachable, and, most importantly, buildable.
(from the original "Attention is All You Need" paper) are a classic choice:
PE(pos, 2i) = sin(pos / 10000^(2i/d_model)) PE(pos, 2i+1) = cos(pos / 10000^(2i/d_model)) Your PDF should include a clear table showing how pos and i interact to give each time step a unique signature. This is where your LLM "thinks." For a sequence of tokens, self-attention computes a weighted sum of all previous tokens (causal means you cannot look into the future).
After attention, a simple feed-forward network (two linear layers with ReLU or GELU) processes each token independently. This is where most of the model’s parameters live.
def get_stats(ids): counts = {} for pair in zip(ids, ids[1:]): counts[pair] = counts.get(pair, 0) + 1 return counts A token is an integer. An embedding converts that integer into a dense vector of size d_model (e.g., 512). Since attention mechanisms are permutation-invariant, we must inject position information.
