Sharding & partitioning
Splitting an index across machines so each holds a slice; queries fan out to all shards and merge. Scales size and write throughput, not per-query speed.
When an index is too big for one machine, sharding splits it into pieces (shards) and puts them on different machines. A query is sent to every shard, each returns its best results, and a coordinator merges them into one list. Splitting by customer instead (each tenant on its own shard) means most queries only touch one shard.
A library so large it's split across several buildings: a catalogue search asks every building and combines the answers.
Adding shards to make queries faster. More shards means more fan-out, more merge work and worse tail latency; per-query latency improves only if each shard was CPU-starved. To serve more QPS, add replicas.
Document-partitioned sharding (hash of id) is the default in Elasticsearch, OpenSearch, Qdrant, Milvus and Vespa: every query fans out to all shards and the coordinator merges top-k from each. Each shard must return k (or more) candidates, so ANN recall is preserved but work grows with shard count. Fan-out amplifies tail latency: if each shard has a 1% chance of a 100 ms stall, a query touching 50 shards hits at least one stall 39% of the time, 63% at 100 shards (Dean & Barroso, The Tail at Scale). Mitigations: fewer, larger shards; hedged/backup requests; replica selection by latency. Tenant/routing-key sharding (Elasticsearch _routing, Qdrant custom sharding / shard keys) makes single-tenant queries hit one shard — best for Multi-tenancy but creates hot shards for big tenants. Elasticsearch guidance: aim for 10–50 GB per shard and < 200M documents per shard; the primary shard count of an index is fixed (resizing needs split/shrink or reindex), so over-provision modestly or use time-/alias-based indices. Global statistics differ per shard for BM25 (IDF), which can skew rankings on small shards.
Elasticsearch/OpenSearch: 10–50 GB per shard, primaries sized for 12–18 months of growth, time- or alias-based indices for rollover. Qdrant/Milvus: shard count ≥ planned node count so you can scale out by moving shards. Tenant-routed shards for tenants > ~5% of data; hash sharding for the long tail.
Code-search platform, 900M chunks × 768-d int8 (~700 GB vectors + graph). 24 shards × ~30 GB on 12 nodes (2 shards each, 1 replica elsewhere). Query: coordinator fans out to 24 shards, each returns top 40 at ef=128 (3 ms), merge 960 → top 100, p50 9 ms, p99 48 ms. Most enterprise queries are scoped to one org: switching to org-routed shards for the 30 largest orgs cut their p99 to 11 ms; the long tail stayed hash-sharded.
Distinguish scale axes: "Shards buy capacity and write throughput; replicas buy QPS and availability. I size shards at 10–50 GB, route by tenant when queries are tenant-scoped, and watch fan-out tail latency." Expect the hot-shard follow-up: one huge tenant on a routing key — give it dedicated shards or a dedicated index.
Numbers worth knowing
- Shard size
- 10–50 GB, < 200M docsElasticsearch sizing guidance
- Tail amplification
- 1 − (1 − p)^np = per-shard slow probability; 63% at p=1%, n=100
- Merge cost
- n × k candidatese.g. 24 shards × 40 = 960 to merge for top 100
How to decide
| When | Do this |
|---|---|
| Index exceeds one node's RAM (vectors) or ~50 GB per shard | Add shards (hash by id). |
| Most queries are tenant-scoped | Route by tenant key; dedicated shards for the biggest tenants. |
| Need more QPS or availability | Add replicas, not shards. |
Cost maths
Nodes = total index size ÷ usable RAM per node × (1 + replicas). 900M × 768-d int8 + HNSW ≈ 700 GB + 130 GB graph ≈ 830 GB; at ~45 GB usable per 64 GB node → 19 nodes per copy, 38 with one replica. At ~$0.43/hr each (illustrative Oct 2026) ≈ $11,900/month. Binary quantization with rescoring from disk could cut it ~4× — size before you shard.
Operating it
- Shard > 50 GB or node RAM > 75% — add shards / nodes
- Query p99 rises with shard count while p50 is flat — tail amplification; hedge requests or consolidate shards
- One shard's CPU > 2× the median — hot tenant; route it to dedicated shards
- Per-shard size and doc count — alert above 50 GB / 200M docs
- Query p99 vs p50 ratio — alert if ratio > 5
- Hottest-shard CPU vs median — alert above 2×
When it goes wrong
Search p99 for a document platform tripled after a 'scale-up' that doubled shards from 32 to 64.
p99 latency SLO alert; p50 was unchanged.
Per-shard p99 stalls (GC and segment merges) now hit far more queries due to doubled fan-out; shards were only 6 GB each and CPU was never the bottleneck.
Shrunk back to 24 larger shards, added replicas for QPS, enabled adaptive replica selection; p99 fell below the original level.
Where it shows up
Sources
- The Tail at Scale — Dean & Barroso, 2013paper
- Elasticsearch reference: Size your shards — Elastic, 2026docs
- Qdrant documentation: Distributed deployment — Qdrant, 2026docs
Check yourself
Three questions. Every option is explained once you answer.