NVIDIA interview questions

Profile first, then attack the binding limit. Training: BF16 or FP8 mixed precision, FlashAttention and fused kernels, activation checkpointing to trade compute for memory, ZeRO or FSDP sharding, and parallelism laid out to the topology, tensor parallel inside NVLink, pipeline and data across nodes, with communication overlapped. Track MFU.

Inference: decode is memory-bound, so cut bytes with quantised weights and KV cache and GQA, raise batch size with paged KV and continuous batching, use speculative decoding for latency, and add tensor parallelism only as far as latency needs.

A compiler turns the model graph into fewer, faster kernels. Kernel fusion matters most: chaining elementwise ops, norms or attention into one kernel keeps intermediates in registers or shared memory instead of round-tripping HBM, and most such ops are memory-bound. Others: constant folding, layout changes to what tensor cores want, lowering to vendor libraries, autotuning tile sizes per GPU, static memory planning, and CUDA graphs against launch overhead. Fitting the hardware means sizing tiles to SM resources and keeping matmuls in tensor-core shapes and precisions.

In eager PyTorch, each op dispatches to a kernel, often from cuBLAS or cuDNN, and the CPU launches it asynchronously on a CUDA stream. The GPU runs it as thread blocks across its SMs, reading HBM. A compiler like torch.compile instead captures a graph, fuses ops and generates Triton kernels; CUDA graphs replay launches with less CPU overhead. Small ops end up launch- or memory-bound, large matmuls compute-bound on tensor cores.

import numpy as np

def quantize(x, qmin=-128, qmax=127):
    lo, hi = min(x.min(), 0.0), max(x.max(), 0.0)
    scale = (hi - lo) / (qmax - qmin) or 1.0
    zp = int(np.clip(round(qmin - lo / scale),
                     qmin, qmax))
    q = np.round(x / scale) + zp
    return np.clip(q, qmin, qmax).astype(np.int8), scale, zp

def dequantize(q, scale, zp):
    return (q.astype(np.float32) - zp) * scale

Asymmetric quantization maps the observed range onto -128 to 127 with a scale and a zero point, keeping 0.0 exactly representable. Interviewers check rounding before adding the zero point, clipping, the all-zero case, and outliers: one huge value stretches the scale and crushes the rest, so clip to a percentile or use per-channel scales.

Benchmark the workload you'll serve: realistic prompt and output length distributions, arrival rates and concurrency, not one fixed prompt. Report time to first token, inter-token latency and end-to-end at p50 and p99, plus throughput at a latency SLO; peak throughput at unbounded latency means little.

Change one knob at a time, batch size, precision, tensor-parallel degree, engine, and rerun a quality eval after each, since quantization can quietly cost accuracy. Warm up, pin versions, repeat runs, and profile to explain each result.

Quantization stores weights or activations in fewer bits. Pruning removes weights, and only structured sparsity such as 2:4 speeds up GPUs. Distillation trains a small student on a big teacher's outputs. Fusion merges kernels to cut memory traffic.

Tensor parallelism shards each layer's matrices, with all-reduces every layer, so it wants NVLink. Pipeline parallelism assigns layers to stages, sends only activations, and suffers bubbles that micro-batches shrink.