QuintoAndar: conversational search with LLM re-ranking

01 · CONTEXT

Why re-rank with an LLM

QuintoAndar's Concierge is a multi-agent assistant that handles property discovery, questions about listings, and visit scheduling through natural-language conversation.

According to the authors, these preference signals are essentially invisible to traditional faceted search, and retrieval with structured filters produces a reasonable candidate set but cannot distinguish properties that nominally satisfy the same constraints.

The paper's choice was to act at the re-ranking: it operates on a small, latency-tolerant set, integrates with the existing retrieval infrastructure, and can reason about hard constraints, soft preferences, and relative value. This way, the LLM improves search without requiring end-to-end generative retrieval or a rewrite of the serving stack.

02 · ARCHITECTURE

From the natural-language request to the final list

The pipeline follows a retrieval-then-re-ranking architecture: the first stage produces the candidates and the second refines the order with a pointwise LLM scoring function over query-property pairs.

1. An LLM-based planner is the entry point of every interaction: it classifies the user's intent and routes the request to the most suitable domain agent.

2. When the planner routes the query to the recommendation agent, the request enters the retrieval pipeline; the details and visit agents handle other flows.

3. Text2Filter converts the user's free text into structured search filters.

4. The resulting filters are used to retrieve and rank candidate properties in the existing search system, which serves as the baseline retrieval stage.

5. The re-ranking stage refines the order of the candidates with a pointwise LLM scoring function applied to query-property pairs.

6. Only the top five properties in the ranking reach the user.

Text2Filter is a two-stage LLM pipeline based on GPT-4o: in the first stage, the model identifies which filter attributes appear in the query; in the second, it generates the structured values for each identified attribute.

03 · THE RE-RANKER

Scoring one property at a time, with context

Scoring was done pointwise. The table summarizes what the authors say about the LLM ranking.

As an example of the quadratic cost, ranking twenty properties would require one hundred ninety pairwise comparisons.

1. For each request, a scoring context is assembled by combining the inputs the LLM will receive.

2. Set-level statistics make it possible to judge whether a candidate is comparatively expensive, better equipped than usual, or distinctive within the retrieved set.

3. The user's text profile is an LLM-generated summary of the user's historical interactions.

4. Candidate metadata includes amenities, facilities, property attributes, and home descriptions.

5. Each candidate is scored by the LLM, and the scores are computed asynchronously across all candidates.

6. Properties are sorted by descending score, and ties are broken by the original retrieval score.

The scoring prompt considers three complementary dimensions: satisfaction of hard constraints, alignment with stated and inferred preferences, and relative value within the set. The user profile, in turn, treats behavioral signals as soft preferences, not as hard constraints.

04 · OFFLINE EVALUATION

A custom dataset and an LLM judge

To evaluate offline, the authors built their own dataset. They started from a random sample of active listings, treated each listing as an anchor, and defined a taxonomy of six query types, generating one synthetic query per category for each property.

1. Property data is sampled from the data lake, and each listing serves as an anchor to generate synthetic queries.

2. Semantically similar candidates are retrieved with dense embeddings, to include hard negatives instead of trivial random negatives.

3. Relevance labels are assigned by an LLM acting as a judge.

4. The dataset is enriched with real production queries.

Neighbor retrieval expanded the dataset from sixty thousand to six hundred thousand pairs. The final dataset has nine hundred sixty thousand query-property pairs; the production portion comes from thirty thousand unique queries in the logs. The relevance distribution is 33% positive pairs and 67% negatives.

The binary relevance labels are assigned by an LLM (Claude Sonnet 4) acting as a juiz. To validate this use, the authors compared labels with human evaluation: on a sample of one thousand pairs, there was 94% agreement with the LLM-generated labels in the validation of synthetic queries and, on another sample of one thousand pairs, 96% agreement with the LLM's judgments.

05 · ONLINE EXPERIMENT

The measured gain and the cost paid

The re-ranker went to production in a randomized A/B experiment between April 7 and May 8, 2026, with a 50% treatment and 50% control split and more than 200 thousand real recommendation traces. The treatment group received the LLM-reordered results, and the control group received the baseline retrieval order.

An important detail: the version tested online was the second-best offline configuration, the one that did not use the user text profile, and the profile was not yet integrated into production serving at the time of deployment.

The authors consider that these costs stayed within acceptable production limits and were outweighed by the gain in engagement and scheduled visits, and they state that production latency remained stable during the experiment.

In parallel, an LLM judge compared, on samples of production traces, the order before and after re-ranking, in a pairwise preference format, with anonymized lists to reduce position and presentation biases. Across 3,944 traces, the judge preferred the reordered list in 95% of cases.

06 · WRAP-UP

Next steps and limits

Among the next steps, the authors mention A/B testing the addition of the LLM-generated user profile as a personalization signal.