LinkedIn: building a GenAI product

01 · CONTEXT

An assistant inside the feed and job listings

LinkedIn describes an assistant that turns every feed post and every job listing into a starting point for questions. When the person clicks a suggested question, the system decides which AI agent is best suited to answer it.

That agent calls a combination of internal APIs and Bing to assemble a dossier that grounds the answer. With the material in hand, the agent writes the answer.

1. The person clicks a suggested question next to a post, and the question enters the pipeline.

2. Routing decides whether the question is in scope and which agent to send it to.

3. The agent decides which services to call and gathers material from the internal APIs and Bing.

4. Generation filters the noisy retrieved data and produces the final answer.

5. The synthesized answer is streamed to the client.

02 · PIPELINE

Routing, retrieval, and generation

The pipeline follows the pattern of retrieval-augmented generation. Routing decides whether the question is in scope and which agent to send it to; retrieval is a recall-oriented step, in which the agent decides which services to call and how; generation is a precision-oriented step, which filters the noisy data and produces the final answer.

Routing and retrieval looked like classification problems, so the team built development sets and tuned them with prompt engineering and their own models. Among the choices that worked are small models for routing and retrieval and larger models for generation.

Generation was different: it followed the 80/20 rule. Getting to 80% was fast, but the final 20% took most of the work.

The team had 80% of the basic experience in the first month and spent four more months trying to get past 95% completion of the full experience. Quality scores rose early on and soon plateaued, and the work felt more like tuning rules in expert systems.

03 · AGENTS AND TEAMS

Independent agents and horizontal pods

To move fast with several teams, the work was split into independent agents, developed by different people: general knowledge, job assessment, and post summarization. The speed gain came with fragmentation: keeping a uniform experience became hard when follow-up interactions could be handled by different models, prompts, or tools.

The response was organizational: a small horizontal pod takes care of the common components and the experience as a whole, and several vertical pods have autonomy over their agents.

04 · SKILLS AND STRUCTURED OUTPUT

Packaging APIs for the LLM

LinkedIn's exclusive data on people, companies, skills, and courses is not part of the LLMs' training. It sits behind RPC APIs, convenient for humans but not very LLM-friendly; the solution was to wrap them in skills, within a RAG pattern.

Each skill brings together an LLM-friendly description of what the API does and when to use it, the configuration for calling the RPC (endpoint, input and output schemas), LLM-friendly schemas with primitive values, and the logic that converts between those schemas and the RPC's.

1. The agent asks the LLM to decide which skill to use and with which parameters.

2. The parameters come back in YAML, in a structured form, and go through the defensive parser before being interpreted.

3. The skill applies the logic that converts between the LLM-friendly schemas and the real RPC schemas.

4. The skill calls the internal RPC API with the endpoint and schema configuration.

The prompts ask the LLM to decide which skill to use and, since the parameters must match the input schema, to return them in a structured form. The team chose YAML because it is less verbose and uses fewer tokens than JSON.

About 90% of the time the parameters came back in the right format, but in about 10% the LLM got it wrong, returning data invalid for the schema or even invalid YAML. Re-prompting the LLM adds latency and consumes GPU capacity, so the team wrote a defensive YAML parser, which detects and fixes common errors before parsing, and added hints about those errors to the prompts. With that, it reduced the errors to about 0.01%.

05 · EVALUATION

Measuring answer quality

Evaluating quality was harder than expected, on three fronts: guidelines, annotation scale, and automatic evaluation.

In job assessment, getting the answer that the person is a terrible fit is not very useful: the answer should be factual, but also empathetic, because some members need to understand the gaps and next steps. Consistent details were essential for uniform scores across annotators.

The in-house linguistics team built tools and processes to evaluate up to 500 conversations per day, with metrics such as overall quality, hallucination rate, responsible AI violation, coherence, and style. This evaluation became the main trend indicator, for iterating on prompts and making sure they were ready to go live.

Automatic evaluation is described as the holy grail, but still a work in progress. Without it, engineers end up eyeballing results and testing few examples, with a delay of more than a day to get the metrics. The team was building model-based evaluators and had some success in hallucination detection.

06 · LATENCY AND CAPACITY

Streaming, TTFT, TBT, and cost

Capacity and member-perceived latency were always front and center. Chain of Thought improves quality and reduces hallucinations, but requires tokens that the member never sees, increasing perceived latency.

All requests use streaming to reduce perceived latency, and this holds within the pipeline, end to end: the LLM response that decides which APIs to call is parsed progressively, and the calls fire as soon as the parameters are ready. The final synthesized answer is also streamed to the client, with incremental processing, for example, of the confidence and responsible AI classification.

A fully asynchronous, non-blocking pipeline was also built, without wasting resources on threads blocked on I/O.

At first only TTFT was bounded, since it mapped directly to member latency. With Chain of Thought in the prompts, the team overlooked that TBT would weigh much more: for a 200-token reasoning step, 10 ms more of TBT means 2 s of extra latency. Some tasks hit timeouts and the team had to scale up capacity quickly.

It is common to get 2x or 3x more tokens per second by sacrificing TTFT and TBT, and the team initially bounded them tightly. GPU clusters are hard to get and expensive, and early on it was necessary to set times when testing the product was allowed.