Training-free · No task head · Open vocabulary

Test-time compute of
jina-embeddings-v5-omni
for image tagging

Han Xiao

VP of AI, Elastic

@hxiao · in/hxiao87

this deck

The result, first

Training-free, open-vocabulary tagging from a frozen embedding model.

Setup

Problem formulation.

given

one embedding model, jina-embeddings-v5-omni-nano (~1B, image and text in one 768-d space), and one image.

output

words for every object in the image: multi-label, open vocabulary.

frozen

all weights. The model is a retrieval encoder; it has no tagging head, no classifier, and was never trained for this task.

allowed

inference-time computation over the model’s own outputs, and nothing else.

No trainingNo second modelNo WordNet / dictionaryNo regex / POS tagger

Whatever tagging ability appears must come from test-time compute of the embedding model.

A taxonomy

aie-sf talk

How test-time scaling works on frozen embedding models.

A frozen encoder’s inference-time budget takes roughly three forms:

A

Deeper pass

Read more of the one pass you already ran: every patch vector, not just the pooled one. The retrieval world’s late interaction.

new information: reclaimed from inside the pass

B

More passes

Run the frozen encoder again on new views: split a document, crop an image, re-encode.

new information: acquired from the input

C

Calibration

Transform the vectors you already have: whitening, propagation, optimal transport. Where most of the training-free literature lives.

new information: none

Same frozen encoder in all three; they differ only in what enters the computation.

Research questions

Two research questions.

RQ1

Can a frozen embedding model become an image tagger through test-time compute alone? Prior tagging work pays with training or resources: Tag2Text / RAM / RAM++ train over ~6,400 curated tags; TagCLIP (AAAI’24) and PIAA (“[CLS] is not enough”) are training-free but assume a given class list, on two-tower CLIP plus WordNet or POS tools.

RQ2

If it can: which test-time compute scales its accuracy, calibration or new information? The 2024–26 literature bets on calibration: whitening (PIAA), label propagation (ZLaP), optimal transport (OTTER), Bayesian adaptation (BCA).

The model · architecture

Image and text embedding space.

per image · test time

cat

coche

橛

× 128,260 tokens

vocabulary · encoded once

jina-embeddings-v5-omni-nano

Qwen3-VL
vision tower

16 px patches
2×2 merge

EuroBERT-12L
text tower

bidirectional · 768-d

vision tokens injected at
image-token positions

never modified, never retrained

\(P\): one 768-d row per patch (the tower’s internal 16 px unit), each vector shaped by attention over the whole image

\(g\): one vector for the whole image, pooled from the last token of the same sequence

\(E\): label matrix, 128,260 × 768; gated to 25,465 words at scoring

Both paths end in the same space, so \( \langle p,\,e_\ell\rangle \) is well-defined: any patch of any image, scored against any word the model knows.

The pipeline

The complete tagger workflow as test-time compute.

Offline

tokenizer vocab

128,260 tokens

encode_text

the same frozen tower

label matrix \(E\)

128,260 × 768

background prior \(\mu\)

from neutral images

word-start gate

→ 25,465 words

Per image

image

one frozen forward

\(P,\;g\)

patches + global

Ascore vs \(E\)

patch-max + global fuse

subtract \(\mu\)

remove base-rate bias

gate + NMS

words, deduped

top-k tags

kitty · cosy · paw

B--hq: 14 crops → same model → per-label max

adds into the score, weight 1.3

Every box is the frozen model or plain arithmetic: A reads more of one pass; B runs new passes on new pixels.

Step 1 · the label space

The label set is the tokenizer's own vocabulary.

No curated tag list. Take all 128,260 tokens and encode each one as text into a label matrix \[ E=\big[\,e_\ell\,\big]_{\ell\in V},\qquad e_\ell=\operatorname{encode\_text}(\ell)\in\mathbb{R}^{768},\quad |V|=128{,}260 \] One matmul now scores any image against the entire vocabulary; fragments are filtered later by a word-start gate (step 4).

wrong: image_vec · embed_tokens.weight

→ Tutor, avatar, PyTuple (garbage)

right: image_vec · encode_text(token)

→ kitty, cosy, plush, paw (aligned)

Why encode_text, not the embedding table? The model has tie_word_embeddings = false: the input-embedding table lives in a different space than the pooled output. You must encode each token as a text string to put labels in the image's space.

Step 2 · where to look

Score every patch.

global pooled

\( s_\ell=\langle g,\,e_\ell\rangle \)

one vector dominated by the most salient object

mAP 0.264

per-label max over patches

\( s_\ell=0.7\max_{p\in P}\langle p,e_\ell\rangle + 0.3\,\langle g,e_\ell\rangle \)

each label keeps its strongest patch match — late interaction, ColBERT-style

mAP 0.635

AThe patch vectors already exist in the forward pass. Scoring each label against all of them, and keeping the max, is one extra step over outputs you already have.

Step 3 · de-bias

Subtract the per-label prior.

Raw cosine has a base-rate bias: generic words ("bed", "cat") score high on almost any image because they sit near the image modality. Estimate a per-label prior once from neutral background images, then subtract it: \[ \tilde s_\ell \;=\; s_\ell-\mu_\ell,\qquad \mu_\ell=\mathbb{E}_{x\,\sim\,\text{background}}\big[s_\ell(x)\big] \]

No calibration set, no labels, one offline pass.

background:

μℓ per label

Step 4 · clean up

The tokenizer already knows what a word is.

Word-start gate. Byte-BPE tokenizers encode a leading space as the glyph Ġ; a token beginning with it starts a new word: Ġcat is a whole word, GetComponent is a fragment. That one built-in signal is the gate; no external dictionary.

Embedding-NMS. Synonyms and multilingual variants (猫 / Cat / кот / cat) collapse into one, using cosine in the model's own space. Walk labels by score; keep \(\ell\) only if \( \max_{k\,\in\,\text{kept}}\langle e_\ell, e_k\rangle < 0.6 \). Again: no lookup table, just the geometry.

raw vocabulary128,260 tokens

Ġkitty · Ġcat · GetComponent · _cpp · 猫 · quisites

word-start gate: keep tokens with the leading-space mark Ġ

whole words25,465

kitty · cat · kitten · chatte · kitt · cats · kittens · cosy · …

embedding-NMS: walk down by score; drop any word with cosine ≥ 0.6 to one already kept

distinct conceptstop-k

kitty · cat · kitten · chatte · cats · cosy · plush · crib

Step 5 · new views

Multi-crop re-encoding.

Class-Wise Reranking (CWR), the crop-and-rescore idea from TagCLIP (AAAI’24), extended to a full-coverage grid of 14 crops (overlaid right) through the same frozen model. A crop is a new input image, itself re-patchified by the tower; a small object fills whichever crop contains it, so weak signal becomes strong.

\[ S_\ell=\tilde s_\ell+1.3\big(\max_{c=1..14} s_{c,\ell}-\mu_\ell\big),\qquad s_{c,\ell}=\max\!\big(\max_{p\in P_c}\langle p,e_\ell\rangle,\;\langle g_c,e_\ell\rangle\big) \]

BPer-label max, not averaging: averaging hurt. The outlier crop is the evidence.

c1

c2

c3

c4

c5

c6

c7

c8

c9

c10

c11

c12

c13

c14

3×3 · 9 crops

2×2 · 4 crops

center · 1 crop

fast: wolf, roar, muzzle → --hq: fur, bear, muzzle
multi-crop re-encoding corrects the single-pass misreading

All five steps, assembled

The entire tagger is one scoring function.

\[ S(\ell)\;=\;\underbrace{0.3\,\Big(\langle g,\,e_\ell\rangle-\mu^{g}_\ell\Big)}_{\textbf{global context}} \;+\;\underbrace{0.7\,\Big(\max_{p\in P}\langle p,\,e_\ell\rangle-\mu_\ell\Big)}_{\textbf{patch evidence}} \;+\;\underbrace{1.3\,\Big(\max_{c=1..14}\, s_\ell(\mathrm{crop}_c)-\mu_\ell\Big)}_{\textbf{multi-crop re-encode}} \]

Global context: the pooled vector, de-biased by its own background prior. Free.

A

Patch evidence: algebra over rows the forward pass already produced. Free. +0.37 mAP.

B

Multi-crop: 14 fresh forward passes on new pixels. 14× the cost. +0.075 mAP.

then \( \ell \in \text{word-gated vocabulary} \rightarrow \text{embedding-NMS}\,(\tau=0.6) \rightarrow \text{top-}k \)

No head, no logits, no learned threshold: three fixed weights, two background priors, and max operations over frozen-model outputs.

The equation, visualized

The pipeline sharpens a distribution.

Real pipeline output, one image, all 128,260 tokens. Tagging is distribution sharpening.

Results

Results on COCO-150.

method	P@1	P@3	R@5	mAP

150 COCO-val images, real multi-label ground truth, avg 2.93 labels/image. softpool trades mAP for top-k precision; the ladder tracks the max-pool path.

mAP: global → patch → +CWR

The experiment

Does calibration scale accuracy at test time?

One example: OTTER re-balances scores toward a target class distribution, which only shuffles vectors it already has. Its open circle sits at the 5-crop baseline of 0.693, and it lands at 0.699.

The meta-conclusion

Only new information moves accuracy.

Calibration methods were designed for weakly calibrated, single-label, two-tower CLIP. This encoder's space is already aligned, calibrated, and multi-label: there is nothing left to recover.

CCalibration

whitening 0.06 · ZLaP 0.14 · EM-Dirichlet 0.17 · OTTER +0.006 over its own base. None improves the pipeline it modifies.

BMore passes

CWR crops → mAP 0.710, P@1 0.813. The only intervention that improves results.

The ceiling is the 1B model itself. Real test-time compute here means giving the model more to look at, not re-arranging what it already saw.

The frontier

Accuracy versus test-time compute.

The frontier is traced by reading more of the pass and re-encoding new views; calibration sits at the same cost, at or below it (<0.1 ms/img, measured).

One more mode · --ngram

Extend to n-grams.

Every tag so far is a single word. Can more test-time compute turn it into a grounded noun phrase, with no part-of-speech tagger and no grammar?

→

ttc tagging pipeline--ngram beam search

→

n = 1

kittycosyplushcrib

n = 2

couch kittyblack cosygrey plushgrey crib

n = 3

grey couch kittysleeping black cosygrey sleeping plushsleeps grey crib

--ngram · mechanics

Beam search over n-gram slots.

Each slot is filled one at a time; the encoder scores the full phrase against the region: \( \langle E(\text{phrase}),\text{region}\rangle-\langle E(\text{kitty}),\text{region}\rangle \). Beam width 4.

The winning path re-orders its own words: “grey couch kitty” beats “couch grey kitty”. Word order, resolved by an embedding model with no grammar.

Qualitative results

n-gram tagging in action.

default

onstageattendeeslivestconcertpresenter

crops

onstagevenueceilingprojectortheater

n=2

conference onstageprojector attendeesattendees livestprojector concertprojector presenter

n=3

conference crowd onstagelobby projector attendeesprojector attendees livestprojector lobby concertattendees projector presenter

default

drivrimsfordtractionsidew

crops

carrorimswindshieldsteeringbeach

n=2

windshield drivwindshield rimssail fordwindshield tractionblue sidew

n=3

blackjack windshield drivdealership windshield rimswindshield yacht fordyacht sedan tractionblue car sidew

default

attendeespassengersprotestorsmigrantscrowded

crops

attendeescommutersdemonstratorssynagoguecrowded

n=2

protester attendeesdemonstrators passengerspassengers protestorsattendees migrantsdemonstrators crowded

n=3

passengers protester attendeescrowded demonstrators passengerspassenger attendees protestorspassengers protester migrantspassenger demonstrators crowded

Default, crop re-encoding, beam-searched n-grams. Top-5 per mode, unstaged photos. Verbatim, unfiltered.

Relation to prior work

Versus the recent tagging literature.

line of work	their setup	our delta
Tag2Text / RAM / RAM++	trained tagging models, curated ~6.4k tag list	training-free, label space = tokenizer vocab
TagCLIP	CLIP two-tower, penultimate layer, DMAR+CWR	single omni model, last layer is the output space
PIAA	patch-level scoring (“[CLS] is not enough”) + GDA whitening	patch thesis confirmed (+0.37 mAP); whitening collapses (0.06)
recent calibration (OTTER, BCA)	calibration set or target prior to de-bias	one offline background prior, zero calibration

Both questions come out the way we hoped. A frozen embedding model does become a tagger with no training, and the only compute that scales it is the kind that adds new information.

Deployment

omni-macos

On-device deployment in Omni.

The Python study became a feature of Omni, a native on-device semantic-search app (Swift + MLX, same frozen model). The port reuses the architecture directly:

same forward pass

the patch rows already exist for the file's embedding; tagging adds one matmul.

~0.6 ms / image

about 4% overhead on the embed step. Tagging is effectively free at index time.

tags = snippets

tags land in the search snippet: media becomes keyword-searchable.

self-calibrating

the background prior \(\mu\) is estimated on-device from the first 64 images it sees, then frozen.

OMNI · ON-DEVICE SEARCH

search: carro steering

IMG_2214.jpg

carrosteeringrimswindshield

couch-nap.heic

kittycosyplushpaw

trail-cam-04.mp4 · segment 0:00–4:00

furbearmuzzle

tag chips as they appear in results; the third row is a video segment

One rule, every media shape

omni-macos

Images, video, and scanned documents.

Image

Tagged at index time, inside the embedding forward. Patch-max + global, prior-centered, gated, NMS-deduped.

HQ image

A background re-tag pass adds the production CWR variant: 5 crops, per-label max.

Video

32 frames per 240-second segment, one sequence through the tower: patch-max pools over space and time.

Scanned PDF

Pages render to images, same path: per-page tags ("invoice, table, signature") as the snippet.

The bigger picture

aie-sf talk

Test-time compute for embedding models.

A frozen embedding model can do more than its training reveals. You reach the rest by spending compute at inference, not by adding parameters. Both of my talks push on that same idea.

AIE SF Talk

Spend test-time compute to get more relevance, better retrieval on the task the encoder was already trained for.

This talk

Spend test-time compute to get a new capability, image tagging, a task the encoder was never trained for.

Scaling test-time compute of an
embedding model unlocks tasks
beyond retrieval.

Han Xiao

VP of AI, Elastic

@hxiao · in/hxiao87

github repo

Test-time compute ofjina-embeddings-v5-omnifor image tagging

Training-free, open-vocabulary tagging from a frozen embedding model.

Problem formulation.

How test-time scaling works on frozen embedding models.

Two research questions.

Image and text embedding space.

The complete tagger workflow as test-time compute.

The label set is the tokenizer's own vocabulary.

Score every patch.

Subtract the per-label prior.

The tokenizer already knows what a word is.

Multi-crop re-encoding.

The entire tagger is one scoring function.

The pipeline sharpens a distribution.

Results on COCO-150.

Does calibration scale accuracy at test time?

Only new information moves accuracy.

Accuracy versus test-time compute.

Extend to n-grams.

Beam search over n-gram slots.

n-gram tagging in action.

Versus the recent tagging literature.

On-device deployment in Omni.

Images, video, and scanned documents.

Test-time compute for embedding models.

Scaling test-time compute of anembedding model unlocks tasksbeyond retrieval.

Test-time compute of
jina-embeddings-v5-omni
for image tagging

Scaling test-time compute of an
embedding model unlocks tasks
beyond retrieval.