Chunk metadata
Structured fields stored with each chunk (tenant, ACL, source, date, type, page) used for filtering, permissions, citations and freshness.
Besides its text and Embedding, each chunk carries labels: which customer it belongs to, who may see it, which document and page it came from, when it was written, what kind of document it is. These labels are metadata.
Search uses them to filter (only this customer's contracts, only documents from 2025), to enforce permissions, and to show citations like 'Policy 4.2, page 7'.
The label on a file folder: date, client, department, confidentiality stamp. You can find and restrict folders by the label without reading what is inside.
Letting the LLM or the client choose the tenant or ACL filter. One prompt injection or bug and the search returns another customer's documents.
Fields that pay off: tenant_id, acl (groups or principals), doc_id, chunk_id, source, doc_type, title, section_path, page, bbox, language, effective_date / updated_at, version, embed_model, and domain fields (jurisdiction, product line, ICD codes). Index the ones you filter on; vector DBs need payload indexes for fast Metadata filtering (pre, post, filtered ANN) (Qdrant payload indexes, pgvector with B-tree/GIN indexes on columns, Elasticsearch/OpenSearch keyword fields).
Security fields are different from convenience fields: tenant_id and acl must be applied by the server on every query (Fail closed), never left to the client or the LLM. Pre-filtering (filter then search) keeps results correct but can hurt Approximate nearest neighbour (ANN) recall when the filter is very selective; modern engines (Qdrant filterable HNSW, pgvector 0.8 iterative scans, OpenSearch efficient filtering) mitigate this. For hard isolation, a collection or partition per large tenant beats a filter. Keep text that the model needs (title, section) in the chunk text too, so it helps the embedding; keep values that change often (price, stock, ACL) in metadata only, so updates do not need re-embedding.
Every chunk carries tenant_id, acl, doc_id, chunk_id, source, doc_type, section_path, page, updated_at and embed_model; payload indexes on tenant_id, acl and any field used in filters; tenant and ACL filters injected server-side from the authenticated principal; volatile fields (price, stock, ACL) in metadata only.
Financial filings assistant for an asset manager. Chunk 10k-acme-2025#item7-12 carries {tenant_id: 'fund-ops', acl: ['research', 'compliance'], doc_type: '10-K', ticker: 'ACME', fiscal_year: 2025, section_path: 'Item 7 › Liquidity', page: 48, embed_model: 'voyage-3.5@1024'}.
Query 'ACME liquidity risk FY2025' from a research analyst: server applies tenant_id = 'fund-ops' AND acl ∋ 'research', the query parser adds ticker = 'ACME' AND fiscal_year = 2025, then vector search runs over ~400 chunks instead of 9M. Citation shows 'ACME 10-K FY2025, Item 7, p.48'.
Split the fields by purpose: "Security fields (tenant, ACL) are applied server-side on every query and fail closed; relevance fields (type, date, ticker) come from a query parser; citation fields (doc, section, page) ride along. Volatile values live only in metadata so they update without re-embedding." Expect: "a filter matches 0.1% of the corpus and recall collapses?" Use filter-aware HNSW or iterative scans, or partition by that field.
| Field | Example | Purpose |
|---|---|---|
| tenant_id, acl | fund-ops; research | Security (server-side) |
| doc_type, ticker, year | 10-K; ACME; 2025 | Relevance filters |
| section_path, page | Item 7; p.48 | Citations |
| updated_at, version | 2026-03-02; v3 | Freshness |
| embed_model | voyage-3.5@1024 | Backfill tracking |
Numbers worth knowing
- Metadata per chunk
- ~200 B–1 KB15–20 fields, short ACL lists
- Selectivity where naive post-filtering breaks
- filters matching <1–5% of vectorsHNSW without filter-aware traversal (measure per engine)
- pgvector iterative index scans
- available from 0.8.0improves recall with selective WHERE clauses
How to decide
| When | Do this |
|---|---|
| Field controls who may see data (tenant, ACL) | Server-side filter on every query, fail closed if missing. |
| One tenant holds >10% of vectors or needs hard isolation | Separate collection or partition instead of a filter. |
| Field changes often (price, stock, permissions) | Metadata only, no re-embed; update the payload. |
Cost maths
Storage is small: 50M chunks × 600 B = 30 GB of payload, plus payload indexes. The cost lever is query work: a selective filter that cuts the candidate set from 9M to 400 makes vector search near-free; a filter the engine applies after ANN wastes the top-k and forces 5–10× oversampling, multiplying latency.
Operating it
- Filtered queries show recall well below unfiltered on the golden set: switch to filter-aware ANN or partition
- One tenant dominates the index: give it its own collection
- ACL changes lag behind the source by more than minutes: move ACL sync to CDC
- Vector queries executed without tenant filter: alert on any (should be zero)
- Recall@k on filtered vs unfiltered golden queries: alert if gap >5 points
- ACL sync lag from source to index: alert above 5 minutes
When it goes wrong
A legal-ops tool showed one client's contract clause in another client's search results.
Customer report; logs showed the request carried no tenant filter.
A new 'search all my matters' endpoint built filters client-side and omitted tenant_id when the matter list was empty, so the query ran unfiltered.
Tenant and ACL filters moved into the search service from the auth token, requests without a principal rejected (fail closed), and a test asserting every vector query carries a tenant filter.
Related terms
Where it shows up
Sources
- Qdrant filtering — Qdrant, 2025docs
- pgvector: open-source vector similarity search for Postgres — pgvector, 2024docs
Check yourself
Three questions. Every option is explained once you answer.