Search is
test-time compute

ElasticON AI San Francisco, November 4, 2026
Han Xiao, VP of AI, Elastic
Search has mostly improved by training harder.
Encoders, data, indexes and rerankers all improve before the first query.

Recent LLMs found a new axis: test-time scaling

Noam Brown presenting test-time scaling curves
Having a bot think for just 20 seconds in a hand of poker got the same performance boost as scaling up the model by 100,000x and training it for 100,000 times longer.Noam Brown, OpenAI, 2024
Fun fact: search has always lived on that axis.
Embedding, retrieval, reranking, expansion and fusion all run after the user presses Enter.

Inference time is a currency, not a KPI

Until now: a KPI
Inference time is a KPI
Latency is a limit written into the SLA. Better accuracy requires a bigger model.
becomes
In this talk: a currency
Inference time is a currency
Keep the weights and raise the budget from 10 ms to 100 ms. What does it buy?
buys
Relevance
Better ranking on the same task.
buys
A new capability
Tasks the model was never trained for.

Four test-time compute examples

01
Score functions
Higher relevance from the same frozen encoder.
02
Open-vocabulary tagger
Annotate photos with any words.
03
Semantic highlighting
A reranker highlights the passages that match.
04
Reusable search pipelines
An agent builds small pipelines and reuses them in long-horizon tasks.

Three ways to spend compute budget

A
Deeper pass
Read every token or patch vector of the pass you already ran.
New information: from inside the pass
B
More passes
Run the encoder again on new views, such as sentences or image crops.
New information: from the input
C
Calibration
Transform existing vectors with whitening, label propagation or optimal transport.
New information: from a global prior
01
Score functions on a frozen encoder

Score function defines the relevance score

single-vector cosine
\( \cos(q,\,d) \)
one vector per document
sentence MaxSim
\( \max_{s}\ \cos(q,\,s) \)
same encoder, one vector per sentence
ColBERT multi-vector
\( \sum_i \max_j\ \cos(q_i,\,d_j) \)
max over document tokens for each query token

Agent finds new score function in a discovery loop

Proposer
LLM agent, Opus 4.6
→
Score function
Python over j-v5-nano
→
Evaluator
ΔnDCG, 14 tasks
→
Memory
JSONL file
→
Registry
144 score functions
memory conditions the next score function
writesone score function
per round
→
encoder239M
frozen
→
scores14 MMTEB tasks
ΔnDCG@10
→
logsΔ, cost, parent
and lesson
→
keeps144 score
functions
The evaluator rejects hyperparameters, task-specific routing, external models and learned parameters.

Score function needs to be generalizable

Discovery · the loop searches here
j-v5-nano
239M · Jina
discovery encoder
small and fast, so each round is quick
searched over14 MMTEB Tier-1 tasks · legal, financial, long-doc, general
→
the same
score function
Held-out · the loop never sees these
j-v5-small
568M · Jina
same family
gemma-300m
303M · Gemma 3
unseen family
qwen3-0.6b
600M · Qwen3
unseen family
unseen families test transfer
tested on19 MMTEB Tier-2/3 tasks · summarization, QA, fact-check, ...
Metric: ΔnDCG@10 against the cosine baseline. Cost c: encoder passes per query, relative to the baseline.

More compute, better relevance

Measured on the 14 discovery tasks.

Score function generalizes over models and tasks

1.2xcheap
Compute cost
14.7xcostly
19 held-out tasks →
−0.98+0.16

52 of 76 task-encoder pairs improve under at least one score function.

Final 4 winning score functions

Reciprocal Rank Fusion
appeared without a seed
fuse two rankings into one
Fisher Linear Discriminant
appeared without a seed
the axis that best separates relevant and non-relevant
Rocchio Pseudo-Relevance Feedback
operationalized from a seeded idea
pull the query toward the relevant centroid
Sentence-level MaxSim
operationalized from a seeded idea
score a document by its best sentence
02
A retrieval model tags images

Open-vocabulary tags from a frozen embedding model

mountainphoenixskyrooftoplocation
clockthermostatdialdstwifi
fiyatpagamentobrasilegratuitiutils
contienelitersdosagethirstconsumo
hargalcdsamsungrefurbishedwarrants
sunglasseswatchesbrandedbronzenecklace
tabletmotherboardwearablecelulardatastore
awardforwardingspeakingdefendantworkshop
gifttoycelebrationballoonscheerful
9 photos · top-5 tags each

Use jina-embeddings-v5-omni-nano for image tagging

input image
per image · test time
cat
coche
кот
× 128,260 tokens
vocabulary · encoded once
jina-embeddings-v5-omni-nano
Qwen3-VL
vision tower
16 px patches
2×2 merge
EuroBERT-12L
text tower
bidirectional · 768-d
vision tokens injected at
image-token positions
frozen
\(P\): one 768-d row per patch, attending to the whole image
\(g\): pooled image vector
\(E\): label matrix, 128,260 × 768

Any image patch can be scored against any word in the vocabulary.

The whole tagger is test-time compute

The label set is the tokenizer vocabulary: 128,260 tokens, gated to 25,465 whole words.

Offline · once
tokenizer vocab
128,260 tokens
encode_text
same frozen tower
label matrix \(E\)
128,260 × 768
background prior \(\mu\)
from neutral images
word-start gate
→ 25,465 words
Per image
image
one frozen forward
\(P,\;g\)
patches + global
Ascore vs \(E\)
patch-max + global
subtract \(\mu\)
remove base-rate bias
gate + NMS
words, deduped
furbearmuzzle
top-k tags
with crops
+
+
B14 crops → same model → per-label max
added to the score

Every box is the frozen model or arithmetic. A reads more of one pass, B adds passes on crops.

Image as single vector or multi-vector?

global pooled
\( s_\ell=\langle g,\,e_\ell\rangle \)
one vector, dominated by the most salient object
mAP 0.264
per-label max over patches
\( s_\ell=0.7\max_{p\in P}\langle p,e_\ell\rangle + 0.3\,\langle g,e_\ell\rangle \)
best patch per label, plus the global vector
mAP 0.635

Spend compute on crops

full image · 1 pass
3×3 · 9 crops
2×2 · 4 crops
center · 1 crop
0 / 14 crops
1 passwolfroarmuzzle
+ 14 crops · per-label maxfurbearmuzzle

+0.075 mAP for 14x the passes: 0.635 to 0.710.

P@1 goes from 0.753 to 0.813.

One pass reads wolf, roar, muzzle. With crops it reads fur, bear, muzzle.

Image tagging as test-time compute

Recursion: the same scoring function runs again on each crop as a new image
\(\displaystyle S(\ell)\)\(\displaystyle\;=\;\) \(\displaystyle 0.3\,\Big(\langle g,\,e_\ell\rangle-\mu^{g}_\ell\Big)\) \(\displaystyle\;+\;\) \(\displaystyle 0.7\,\Big(\max_{p\in P}\langle p,\,e_\ell\rangle-\mu_\ell\Big)\) \(\displaystyle\;+\;\) \(\displaystyle 1.3\,\Big(\max_{c=1..14}\,\)\(\displaystyle s_\ell(\mathrm{crop}_c)\)\(\displaystyle \,-\mu_\ell\Big)\)
CCalibration
APatch evidence
BMulti-crop re-encode
C
Calibration
The pooled vector minus its own background prior, at no extra cost.
A
Deeper pass
Arithmetic on patch rows the forward pass already produced, at no extra cost.
B
More passes
14 new forward passes over new pixels.

Accuracy climbs with more test-time compute

COCO-150 mAP: global → patch → + 14 crops

03
Reranker as semantic highlighter
GitHubQR code to the in-page search repo

Highlight the passages that answer the query

queryWhy is the sky blue?
document

p1The color of the sky puzzled people. p2Sunlight carries every rainbow color. p3Air molecules are tiny next to light. p4They scatter blue far more than red. p5That blue reaches us from all sides. p6At sunset, the blue scatters away. p7Clouds scatter all colors equally. p8On Mars, dust turns the sky tan.

inputquery q, long document D
splitD into passages p1 … pN
scores(q, pi) for every passage
outputtop passages, highlighted in place
p1
.21
p2
.35
p3
.52
p4
.94
p5
.90
p6
.41
p7
.12
p8
.08

Cross-encoder: score each passage alone

Cross-encoderbge-reranker-v2-m3

jina-reranker-v3/v3.5 new listwise reranker that scores all passages in one shot

Listwisejina-reranker-v3 · 0.6Bjina-reranker-v3.5
BEIR nDCG@10 · 0.6B class
jina-reranker-v3 (listwise)
61.94
bge-reranker-v2-m3
56.51
Qwen3-Reranker-0.6B
56.28

Where the extra compute goes

Cross-encoder
one pass: q + pi
query, re-read ×N
never computed
query timeO(N·(q+L)²) ≈ O(N·L²)
Listwise
same as cross-encoder
extra
passage ↔ passage
context across chunks
query timeO((q+N·L)²) ≈ O(N²·L²)
04
Build reusable agentic pipeline at test-time

Test-time compute is the pattern for long-horizon agents

2025 · Deep research
Query→
loop until budget
Search→Read→Reason
→Answer
one loop · web IO throughout
2026 · Long-horizon tasks (e.g. RSI)
Query→
research
Search→Read→Reason
→dataroom→
agent loop
Read→Run→Write
→Output
research with web IO  →  corpus  →  offline agent loop, no web IO
GitHubQR code to github.com/hanxiao/dataroom

dataroom: a token budget builds a corpus

query LOCAL pi GPU search read write edit pi + harness + local model dataroom 114 files .zip tokens 651,736
GitHubQR code to github.com/hanxiao/searchbox

searchbox: an agent builds its pipeline for each query

.zip no web dataroom/ local tools grep read embed rerank similarity cluster select_diverse pipelines q1 grep read q2 grep embed rerank q3 embed cluster select_diverse rerank reuse median per question, one turn: 6 tool calls · 20K tokens
GitHubQR code to github.com/hanxiao/knowledge-graph-extractor

knowledge-graph: hard questions as a private verifier

If one grep finds the answer, every method scores the same. Agentic search needs questions that force several hops.

dataroom LLM extract (s) -[p]-> (o) based_on trained_on XLM-RoBERTa ? hidden bridge jina-embeddings-v3 Common Crawl Q jina-embeddings-v3 is based on a model; what was that trained on? Common Crawl grep + read, one turn: answered for jina-embeddings-v3 private verifier 100 questions 1 hop 76% 2–3 hops 22% grep + read, one turn Scores every new pipeline.

Summary of four examples

Four examples
01
Score functions
Spend compute to build advanced score functions
02
Open-vocabulary tagger
Spend compute to unlock a new capability: image tagging
03
Semantic highlighting
Spend compute on more interactions between candidate documents
04
Reusable search pipelines
Spend compute to build reusable, verifiable pipelines
Three ways to spend compute
A
Deeper pass
Read every token or patch vector of the pass you already ran.
New information: from inside the pass
B
More passes
Run the encoder again on new views, such as sentences or image crops.
New information: from the input
C
Calibration
Transform existing vectors with whitening, label propagation or optimal transport.
New information: from a global prior
Search has always been test-time compute. Spend it where new information enters.

Thank you

Han Xiao, VP of AI, Elastic
Search is test-time compute
1 / 44