What AI-engineering interviewers test at each level
One design question, three scored answers (Mid, Senior, Staff), the rubric behind the scores, and the red flags that sink otherwise good candidates.
Two candidates in a staff loop get the same prompt: “Our legal team wants an assistant that answers questions across 40,000 signed contracts. How would you approach it?” The first draws a clean pipeline in four minutes: chunk, embed, hybrid search, rerank, generate with citations. The second spends those four minutes asking who uses it, what a wrong answer costs, and whether every lawyer may read every contract, then draws a pipeline that looks different. The first answer is correct and scores as Senior. The second scores as Staff.
This post is only about the interview: what is scored and what an answer at each level sounds like. How the levels map to daily work, scope and promotion is covered in the companion post, The AI engineering career ladder (/blog/ai-engineering-career-ladder/).
The rubric
Most loops score a design round on five to seven dimensions at 1–4. Labels vary between companies; the substance is consistent. Here is a six-dimension version, and what earns a 4 on each:
| Dimension | What a 4 looks like |
|---|---|
| Framing | Finds the question the obvious design can't answer |
| Design | Correct pipeline; justifies each stage with a number |
| Measurement | Golden set, metric per stage, judge calibrated to humans |
| Operations | Cost, latency, rollout, fallback, with figures |
| Risk & access | ACL in retrieval, fail closed, injection via documents |
| Org scope | Owners, build vs buy, reuse across teams |
Mid answer: 10 / 24
I'd parse the contracts, chunk them by clause, around 500 tokens with some overlap, embed them, and store them in a vector database. At query time I'd use hybrid search, BM25 plus dense, because legal text has exact terms like “indemnification” that embeddings can blur, then rerank the top 50 down to 8 and have the model answer with clause citations. To know it works I'd build a set of 100 questions with the legal team and measure recall@10 for retrieval and faithfulness for answers, separately, so I know which half is failing.
| Framing | Design | Measurement | Operations | Risk | Org | Total |
|---|---|---|---|---|---|---|
| 1 | 3 | 3 | 1 | 1 | 1 | 10 |
A strong Mid answer: the design is right, every choice has a reason, and measuring retrieval separately from generation is exactly the habit the level tests. It never questions the premise, and nothing in it would change if the users were customers instead of lawyers.
Senior answer: 15 / 24
Everything in that design, plus how it runs. Forty thousand contracts at roughly 25,000 tokens each is a billion tokens to ingest: a one-off batch job, and incremental after that as contracts are signed. Per question, about 8k input and 500 output tokens; at $3 / $15 per million that's about 3 cents, trivial at legal-team volume, so I'd optimise for quality over cost. Retrieval p95 under 1 s and answers streaming within 2 s. Ship to five lawyers first, collect thumbs-down with a reason, and do error analysis weekly. Version prompts and pin the model version so an upgrade is a deliberate, evaluated change. And lawyers can only see matters they're staffed on, so retrieval filters by matter ID.
| Framing | Design | Measurement | Operations | Risk | Org | Total |
|---|---|---|---|---|---|---|
| 2 | 3 | 3 | 4 | 2 | 1 | 15 |
Numbers that survive arithmetic (40,000 × 25,000 is one billion), a rollout plan and access control in the retrieval layer, not the prompt. Risk scores a 2 because it stops at the happy path of permissions: no mention of what happens when the matter lookup fails, or of instructions hidden inside a counterparty's contract.
Staff answer: 22 / 24
Before the pipeline: what do lawyers ask? If many questions are aggregates, like “which contracts auto-renew in Q1” or “where is indemnity uncapped”, top-k retrieval structurally cannot answer them, because the answer is spread across hundreds of documents. So I'd propose two paths. An offline extraction pass fills a clause table, about 30 fields per contract, run through a batch API: a billion input tokens at $3 per million is $3,000, about $1,500 batched, and it's queryable with plain SQL. Then a retrieval path for open-ended questions, with pinpoint citations. Access is enforced by matter in both paths and fails closed if the permission service is down. Contracts are untrusted input, since a counterparty wrote half of each one, so the model gets no tools that can send or write anything. Drafts go to lawyers only, never to clients. Two lawyers write 300 questions and we check their agreement before trusting the set. On ownership: legal ops owns the golden set and the field schema, and the platform team owns retrieval, which support search can reuse. I'd evaluate a legal-AI vendor for the extraction schema before building it; we build only the part that joins our matter system.
| Framing | Design | Measurement | Operations | Risk | Org | Total |
|---|---|---|---|---|---|---|
| 4 | 3 | 4 | 3 | 4 | 4 | 22 |
The decisive move is in the first sentence: find the class of question the obvious architecture fails on, then split the system around it. The rest follows that judgement: the batch cost for the extraction path, access control that fails closed, contracts treated as a prompt-injection surface, annotator agreement before trusting the golden set, and named owners with a build-versus-buy line. Design stays at 3 because the candidate never descended into chunking or reranking. A good interviewer pulls on that next, and the candidate must be able to go there.
Follow-ups that separate a 3 from a 4
- “Now it's 400,000 contracts.” A 3 scales the vector database. A 4 notes extraction is now about $15,000 batched and asks whether only the contracts still in force (often a minority) need it.
- “A regulated client requires EU data residency.” A 4 names what moves: the index, the extraction table, the model endpoint region and the logs. Logs are where residency usually breaks.
- “The model provider is down for two hours.” A 4 says the clause table still answers aggregates with no model at all; open questions degrade to search results with links.
- “How would you know the extraction is right?” A 4 samples 200 contracts per field, has a lawyer check them, and reports precision per field, because one bad field (say, renewal date) matters more than the average.
Red flags
- Naming a vector database before asking about the corpus, the users or the cost of a wrong answer.
- “We'll fine-tune it on the contracts” as the way to add knowledge. Facts in weights cannot be cited, filtered by matter or deleted.
- Access control as a prompt instruction (“tell the model not to reveal other matters”). Permissions belong in the retrieval filter.
- “We'll use an LLM as a judge” with no plan to check the judge against human labels.
- A number that does not survive one multiplication, or a cost given without its assumptions.
- Reaching for a multi-agent design for what is a lookup plus a table.
- At Staff, framing with no descent: unable to explain why the reranker helps when asked.
- In the behavioural round, “we” throughout with no “I”: interviewers probe until they find what you personally did.
Spending 45 minutes
Key takeaways
- Level is scored on what you choose to examine, not on how much you draw. The same pipeline scored 10 and 22 depending on what surrounded it.
- The fastest Staff signal is finding the question the obvious architecture cannot answer, then designing around it.
- Every figure you say aloud gets multiplied by the interviewer. State the assumptions and make sure the arithmetic holds.
- Framing without descent reads as hand-waving; descent without framing reads as Senior. Do both, in that order.
Sources
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection — Greshake et al., 2023paper
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena — Zheng et al., 2023paper
- Building effective agents — Anthropic, 2024blog
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Lewis et al., 2020paper