Encoder
A model that reads input text and outputs vector representations of it, rather than generating new text.
An encoder is a model that reads text and produces numbers that capture its meaning, instead of writing a reply. In search, encoders turn queries and documents into embeddings, or score how well a query matches a document.
Chat LLMs are decoders (they generate text); encoders are their read-only cousins, usually much smaller and faster.
A reader who summarises each document onto a compact index card, but never writes a new document.
Feeding text longer than the encoder's max length and not noticing it was silently truncated, so the end of every long chunk is invisible to search.
Classic encoders (BERT-style) use bidirectional Attention: every token sees every other token, which suits understanding tasks. The output is one contextual vector per token; for retrieval these are pooled (mean pooling or a [CLS] token) into one embedding (Sentence-BERT), or kept per token for late-interaction models (ColBERT). Several current top embedding models are decoder LLMs adapted with contrastive training, so the distinction is functional rather than architectural.
Two retrieval configurations: the Bi-encoder (query and document encoded independently; fast, indexable) and the Cross-encoder (pair encoded together; accurate, slow). Sizes run from 22M (all-MiniLM-L6-v2) to several billion parameters; maximum input runs from 256–512 tokens on older models to 8,192 on bge-m3 and nomic-embed-text-v1.5. Input beyond the maximum is silently truncated, which is the most common hidden bug in Chunking.
bge-m3 for multilingual or long chunks; all-MiniLM-L6-v2 when CPU-only and chunks are short; served with Text Embeddings Inference or Sentence-Transformers, batch 32–128, FP16 on GPU; an ingest assertion that token length ≤ max_seq_length.
Legal clause search: 2M contract clauses averaging 350 tokens embedded with bge-m3 on one L4 GPU. Assuming ~400 clauses/s at batch 64, the full embed takes ~1.4 hours; a single query embeds in ~8 ms. Swapping in bge-large-en-v1.5 (512-token limit) would have truncated the 14% of clauses longer than 512 tokens.
Commit: encoders do the reading in retrieval, and their max sequence length sets your chunk ceiling. Pick by golden-set recall, latency on your hardware and max length, in that order. Follow-up: 'why not use the chat LLM to embed?': cost and latency per vector, and a purpose-trained encoder usually wins on retrieval at 1/100th the size.
| Model | Params | Max tokens | Dims |
|---|---|---|---|
| all-MiniLM-L6-v2 | 22M | 256 | 384 |
| bge-large-en-v1.5 | 335M | 512 | 1,024 |
| nomic-embed-text-v1.5 | 137M | 8,192 | 768 (Matryoshka) |
| bge-m3 | 568M | 8,192 | 1,024 |
Numbers worth knowing
- Encoder sizes
- 22M–7B parametersproduction embedding models, Oct 2026
- Max input
- 256–8,192 tokensmodel-specific; beyond it input is truncated without error
- Query encode latency
- ≈5–15 ms100M–600M model on a GPU, single short query
How to decide
| When | Do this |
|---|---|
| Chunks exceed 512 tokens or the corpus is multilingual | Use an 8k-context multilingual encoder such as bge-m3. |
| CPU-only and latency budget under 20 ms | Small encoder (MiniLM class), shorter chunks. |
| Golden-set recall gap above 5 points vs a larger model | Pay for the larger encoder; ingestion is a one-off cost. |
Cost maths
Embedding time = chunks ÷ throughput. 2M chunks at an assumed ~400/s on one L4 (~$0.80/hour, illustrative) ≈ 1.4 hours ≈ $1.10 per full re-embed. Query-side: one GPU serving ~1,000 short queries/s covers most products; the cost driver is keeping a GPU warm 24/7 (~$580/month), so small CPU-served encoders win at low QPS.
Operating it
- Any non-zero truncation count at ingest
- Query encode p95 above 30 ms
- Recall gap to a larger encoder above 5 points on the golden set
- Chunks over max_seq_length at ingest: alert on any
- Encoder throughput (chunks/s) during backfills: alert below 50% of baseline
- Query encode p95: alert above 30 ms
When it goes wrong
Questions about indemnity carve-outs in long contracts returned the right contract but never the right clause.
Golden-set slice of 'long clause' questions at recall@10 0.38 vs 0.81 overall.
Chunks averaged 900 tokens; the encoder's limit was 512, so the second half of every long clause, where carve-outs sit, was never embedded.
Switched to an 8,192-token encoder and section-aware chunks capped at 800 tokens, plus an ingest assertion on token length.
Related terms
Sources
- BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding — Devlin et al., 2018paper
- Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — Reimers & Gurevych, 2019paper
Check yourself
Three questions. Every option is explained once you answer.