Implement sinusoidal or learned positional encodings in PyTorch, explain the equations and tensor shapes, and say what you'd see if positional information were missing.

import math
import torch

def sinusoidal_pe(T, d):
    pos = torch.arange(T).float()[:, None]   # (T, 1)
    i = torch.arange(0, d, 2).float()        # (d/2,)
    freq = torch.exp(-math.log(10000.0) * i / d)
    pe = torch.zeros(T, d)                   # d even
    pe[:, 0::2] = torch.sin(pos * freq)
    pe[:, 1::2] = torch.cos(pos * freq)
    return pe

# x: (B, T, d); x = x + sinusoidal_pe(T, d).to(x)
# learned: x = x + nn.Embedding(max_len, d)(pos_ids)

Dimension pair 2i gets sin and cos of pos divided by 10000 to the power 2i/d. A learned table is simpler but can't go past max_len.

Without positions, attention is permutation-equivariant: an encoder sees a bag of words, so shuffled inputs give shuffled outputs and order-dependent tasks fail. A causal decoder recovers some position from its mask.