There are four paths to bringing kernels into PyTorch pipelines: cuBLAS/cuDNN for library-backed operations, eager PyTorch with per-operator kernels, torch.compile with fused Triton kernels, and hand-written CUDA/Triton kernels, each offering different performance and flexibility trade-offs.

The roofline model shows when kernels are memory-bound versus compute-bound, with most deep-learning ops limited by data movement rather than arithmetic.

A GPU kernel is a function that runs on the GPU across many parallel threads, while Python or the CPU orchestrates the workflow but does not execute the kernel itself.

CUDA kernels operate at the thread level and require careful management of memory access, shared memory, and synchronization to reach high performance.

KernelBench is a benchmark framework for testing whether LLMs can produce production-grade kernels, using criteria like compilation success, correctness across inputs, and performance.

Numerical notes from the roofline model include ridge points for different precisions and the impact of TF32 on Tensor Cores’ performance relative to FP32 accuracy.

Triton offers a higher-level programming model by targeting blocks of data rather than individual threads, trading fine-grained control for easier readability and stronger auto-vectorization, and PyTorch torch.compile relies on Triton for fused kernels.

Four core optimizations are highlighted: fusion to reduce intermediate memory traffic, streaming online softmax to cut memory movement in attention, tiling and Tensor Cores with TF32 to reuse data, and epilogues to fuse post-matmul operations and tail processing.