How LinkedIn Made LLM Ranking 10× Faster for 1.3 Billion Members
November 18, 2026
LinkedIn scaled an LLM-based ranking system for search and recommendations serving 1.3 billion members, where strict latency requirements make every inference optimization count.
The ranker evaluates jobs and people by scoring many candidates per request, but unlike a chat system, it needs only the probabilities of “yes” and “no,” not generated text; requests must meet a P99 latency target of about 500 milliseconds.
To make the model practical, the team distilled a 7-billion-parameter teacher into a 400-million-parameter ranker, teaching smaller models separate relevance and engagement objectives before pruning layers and attention heads and fine-tuning to recover quality.
LinkedIn also shortened job-description context from about 2,000 tokens to roughly 150 through offline summarization, then represented each full description with a single embedding; these approaches increased throughput eight to 10 times with less than 2% quality loss.
The single-embedding approach processed about 22,000 items per second per GPU, versus about 2,200 with summaries and 290 with full descriptions, while delivering slightly better quality than summaries.
LinkedIn added a scoring API to the open-source SGLang engine and optimized inference with batched tokenization, asynchronous GPU copies and an Item-Aware Attention Mask that reuses query key-value pairs across item prompts.
The custom attention mask lets the system reuse the shared query-and-instructions cache and score multiple items in one pass, while preventing one job from affecting another job’s score.
The team also preserved batches across process communication and transferred only the needed yes/no scores, using asynchronous, non-blocking copies to avoid unnecessary data movement.
To eliminate recurring P99 latency spikes of about 300 milliseconds, LinkedIn used `gc.freeze` after model warm-up and ran multiple gRPC servicer and SGLang worker processes to ease Python’s Global Interpreter Lock bottleneck.
Ramachandran said the LLM ranker improved relevance metrics by roughly 5% to 10% over earlier small-MLP rankers; the feature launched first for Premium members, with LinkedIn planning to extend it to all users by year-end.
LinkedIn also reported a 5% to 10% increase in auction metrics, including relevance and engagement, from LLM-based reranking.
For reliable production results, the team recommended benchmarking recorded traffic or Poisson arrivals, keeping offline and online tokenizers and prompts aligned, and coordinating inference, model, prompt and product teams.
Summary based on 2 sources
Get a daily email with more AI stories
Sources

LavX News • Nov 17, 2026
LinkedIn shrinks a 7B ranking model and cuts prompts to one token to serve LLM ranking at scale