ByteDance interview questions

Project once, split into heads, attend in each head, merge.

def mhsa(x, w_qkv, w_o, h):            # x: (B, T, d)
    B, T, d = x.shape
    q, k, v = (x @ w_qkv).split(d, -1)
    heads = lambda t: t.view(B, T, h, -1).transpose(1, 2)
    q, k, v = heads(q), heads(k), heads(v)  # (B, h, T, d/h)
    s = q @ k.transpose(-2, -1) / (d // h) ** 0.5
    future = torch.ones(T, T, dtype=torch.bool,
                        device=x.device).triu(1)
    s = s.masked_fill(future, float("-inf"))
    y = s.softmax(-1) @ v
    return y.transpose(1, 2).reshape(B, T, d) @ w_o

Interviewers watch for the mask before softmax, scaling by the head size, not d, and the transpose before the final reshape.

import torch, torch.nn as nn, torch.nn.functional as F

class Block(nn.Module):
    def __init__(self, d, h):
        super().__init__()
        self.h = h
        self.ln1 = nn.LayerNorm(d)
        self.ln2 = nn.LayerNorm(d)
        self.qkv = nn.Linear(d, 3 * d)
        self.proj = nn.Linear(d, d)
        self.mlp = nn.Sequential(
            nn.Linear(d, 4 * d), nn.GELU(),
            nn.Linear(4 * d, d))

    def attn(self, x):
        B, T, C = x.shape
        q, k, v = self.qkv(x).split(C, dim=-1)
        q, k, v = (t.view(B, T, self.h, -1)
                   .transpose(1, 2) for t in (q, k, v))
        s = q @ k.transpose(-2, -1) / k.shape[-1] ** 0.5
        m = torch.ones(T, T, dtype=torch.bool,
                       device=x.device).tril()
        s = s.masked_fill(~m, float("-inf"))
        y = s.softmax(-1) @ v
        y = y.transpose(1, 2).reshape(B, T, C)
        return self.proj(y)

    def forward(self, x):
        x = x + self.attn(self.ln1(x))
        return x + self.mlp(self.ln2(x))

class Decoder(nn.Module):
    def __init__(self, vocab, d, h, n, max_len):
        super().__init__()
        self.tok = nn.Embedding(vocab, d)
        self.pos = nn.Embedding(max_len, d)
        self.blocks = nn.ModuleList(
            Block(d, h) for _ in range(n))
        self.ln = nn.LayerNorm(d)
        self.head = nn.Linear(d, vocab, bias=False)

    def forward(self, idx, targets=None):
        pos = torch.arange(idx.shape[1],
                           device=idx.device)
        x = self.tok(idx) + self.pos(pos)
        for b in self.blocks:
            x = b(x)
        logits = self.head(self.ln(x))
        if targets is None:
            return logits
        loss = F.cross_entropy(logits.flatten(0, 1),
                               targets.flatten())
        return logits, loss

Interviewers watch the causal mask, head reshapes that keep batch and heads apart, pre-norm residuals, and targets shifted by one token outside the model.

Ask which mistake costs more. Precision when a false alarm is expensive, like an agent that issues refunds or bans accounts. Recall when a miss is expensive, like a safety agent that must catch harmful content. F1 when both matter and positives are rare, so accuracy misleads. With known costs, weight errors directly.

Retrieval becomes tools the agent calls, a search per source, so it can reformulate and search again. Grounding: answers cite passages, a checker flags claims the passages don't support, and the agent abstains when evidence is missing.

Access control lives in retrieval, not the prompt: filter every search by the user's permissions, and run tools with that user's credentials and least privilege. Retrieved text is untrusted, a vector for indirect injection, so write actions need confirmation. Log every retrieval and action for audit, and evaluate retrieval, faithfulness and leakage separately.

Both maximise reward with a clipped probability ratio. PPO estimates advantages with a learned value model, often policy-sized. GRPO drops it: it samples a group of answers per prompt and uses each answer's reward, normalised by the group's mean and standard deviation, as the advantage.

GRPO also puts the KL penalty in the loss, not the reward. It saves memory and suits checkable rewards like maths, but needs several samples per prompt.

A full softmax over millions of items is too costly, so each positive competes against sampled negatives. In-batch negatives appear in proportion to popularity, so popular items get over-penalised. The logQ correction subtracts each item's log sampling probability from its logit, which makes the sampled loss approximate the full softmax.

import torch
import torch.nn.functional as F

def in_batch_loss(u, items, log_q):
    # u, items: [B, d]; row i of items is user i's
    # positive; log_q: [B] log sampling probability
    logits = u @ items.T - log_q[None, :]
    labels = torch.arange(len(u), device=u.device)
    return F.cross_entropy(logits, labels)

Loss: softmax cross-entropy when exactly one item is right among many; per-item binary cross-entropy for multi-label targets or calibrated click probabilities.

Generations gained from more and better data at compute-optimal scale, instruction tuning and preference training, then architecture tweaks like RoPE, grouped-query attention, MoE and RL for reasoning.

Multimodal LLMs run a vision encoder over image patches and project its outputs into the LLM's embedding space as extra tokens.

Normalisation keeps activations in range: pre-norm placement stabilises deep stacks, and RMSNorm drops mean-centring to save compute.

Profile first: preprocessing, vision encoder, prefill and decode each take their share. Then cut visual tokens, the big lever, by lowering resolution, pooling or merging tokens; quantise weights; cache image embeddings; and run encoder and LLM as separate stages with continuous batching, scaled independently.

For video, do the heavy work offline: split into shots, embed sampled keyframes with a CLIP-style model into a shared text-image space, and index them with timestamps. A query is one text embedding and an ANN lookup; rerank only the top candidates with a larger model.