Your model needs to handle much longer sequences. How would you optimise attention for that?

First find which cost hurts: attention compute grows quadratically with length and dominates long prefills, while the KV cache grows linearly and caps decode batch size.

Exact fixes first. FlashAttention tiles the work in on-chip memory, never storing the n × n score matrix: same result, memory linear in n. For the cache: grouped-query attention, quantisation, paged allocation.

Then approximations that change the model: sliding-window or sparse patterns like Longformer's local plus global attention, or linear-attention and state-space layers. Past one GPU, ring attention splits the sequence across devices. Re-check long-context quality.