Card 1 of 50
Front. Launch-criteria. What are launch criteria?
These cards are original study material. They are not questions from a named employer or a live hiring loop.
All 50 cards in this deck
1.Launch-criteria
What are launch criteria?
The written conditions a model must meet before release, such as eval thresholds, safety review, cost, and a tested rollback.
2.Go-no-go
What is a go/no-go decision?
One named owner decides, against a checklist, whether the launch proceeds. A shared channel is not that owner.
3.Held-out-eval
What is a held-out eval set?
Examples kept out of training and prompt tuning, so the score shows performance on data the model was not fit to.
4.Demo-set-trap
Why is the demo script a bad eval set?
The model and team rehearsed on it. It measures memory of the script, not behavior on new questions.
5.Eval-leak
What is an eval leak?
Eval examples reaching training or tuning. The score rises without the model improving on new data.
6.Golden-set
What is a golden set?
Reference answers written before seeing the candidate model's outputs, used as the bar it must clear.
7.Offline-online-gap
What is the offline-online gap?
An offline metric improves while the user-facing metric does not. The launch decision follows what users get.
8.North-star-metric
What is a user-facing launch metric?
A measure of what the user actually gets, such as resolved questions without a wrong refund, rather than model loss.
9.Guardrail-metric
What is a guardrail metric?
A metric that must not get worse during a launch, such as harmful answers, latency, or cost, even if the main metric improves.
10.Human-eval
What makes a human eval trustworthy?
Raters who did not build the system, clear instructions, and enough cases to see the failures that matter.
11.Llm-judge
What is an LLM judge?
A model that scores outputs against a rubric. It needs checking against human labels before its scores decide a launch.
12.Red-team
What is red-teaming?
Deliberately trying to make the system fail or cause harm, such as prompt injection or unsafe advice, before users do.
13.Prompt-injection
What is prompt injection?
Text in user input or retrieved content that instructs the model to ignore its rules or take an action it should not.
14.Tool-permission
Why gate an agent's tools?
A tool that moves money or sends mail turns a bad answer into a real action. Risky tools need review and limits.
15.Safety-review
When should a safety or privacy review start?
When the design is chosen. A review that can block launch should not start in launch week.
16.Pii-training
What should happen if personal data reaches a training run?
Stop and discard the run, then screen the next dataset before training.
17.Model-card
What is a model card?
A written record of what a model is for, how it was evaluated, known failures, and limits on use.
18.Rollback
What is a rollback plan?
A tested way to return to the previous model or prompt quickly. 'Fix forward' is not a rollback.
19.Staged-rollout
What is a staged rollout?
Releasing to a small share of traffic first, with a metric that stops the rollout if it gets worse.
20.Shadow-mode
What is shadow mode?
The new model runs on real traffic, but its outputs are logged and compared instead of shown to users.
21.Feature-flag-debt
When does a feature flag become debt?
When it stays at 100 percent for months with no owner, leaving dead code paths and unclear rollback.
22.Error-budget
What is an error budget?
The amount of failure a service can absorb in a period. Once spent, risky launches wait.
23.Latency-budget
What is a latency budget?
The maximum response time for the question users actually ask. Missing it on the main path is a failed launch.
24.Serving-cost
Why put serving cost in a launch review?
A model that doubles the bill at expected traffic is a launch decision, not a later finance surprise.
25.Vendor-lock-in
What is vendor lock-in?
The cost and time to leave a provider. It belongs in the decision with the price you save.
26.Build-vs-buy
What goes into build versus buy for a model?
Quality on your eval, cost, time, ownership burden, and lock-in on both sides, not only the appeal of owning weights.
27.Sunk-cost
What is the sunk-cost trap?
Continuing a project because of what was already spent. Decide on whether the next month is still the best use of the team.
28.Scope-trade
What is a scope trade?
When new work is added, something named comes off, or the date moves. Absorbing it is not free.
29.Iron-triangle
What is the scope, time, and people trade-off?
With fixed people, adding scope moves the date, and holding the date means cutting scope.
30.Capacity-plan
What is capacity planning?
Counting the people actually available after on-call, interviews, and leave, not raw headcount.
31.New-hire-ramp
Why not count new hires as full capacity?
Unfilled seats and people in their first month do not deliver a pretrain or launch at full speed.
32.Wip-limit
What is a work-in-progress limit?
Capping how many projects run at once, so a few finish instead of many stalling at 20 percent.
33.Interrupt-budget
What is an interrupt budget?
A cap on unplanned requests per period. Past the cap, a planned item stops, and that cost is visible.
34.Estimate-vs-commitment
What is the difference between an estimate and a commitment?
An estimate is a best guess with uncertainty. A commitment is a date you plan around after cutting scope to fit.
35.Range-estimate
Why give a range instead of one date?
It shows real uncertainty, and the reason for the range points at the scope or risk to resolve.
36.Padding
Where does schedule padding belong?
On the risks you can name, with the reason stated. Uniform padding makes real tight dates unbelievable.
37.Spike
What is a spike?
A short, timeboxed investigation to answer an unknown, such as whether data can be retrieved, before committing to a build.
38.Early-slip-signal
When do you report a slip?
As soon as you know. Waiting for the review meeting makes future dates less believable.
39.Watermelon-status
What is a watermelon status?
Green on the outside and red inside: a report that hides a known blocker behind vague progress.
40.Dependency-owner
Why name an owner for shared services?
Without one, every incident becomes a negotiation between teams instead of a fix.
41.Stakeholder-tradeoff
How do you say no to a stakeholder usefully?
State the constraint and when the request could be considered again, so they do not route around you.
42.Priority-stack-rank
Why force a stack rank of priorities?
Two top priorities on one team means the loudest request wins. A ranking makes the trade explicit.
43.Decision-record
What is a decision record?
A short written note of the choice, the reason, the owner, and what was rejected, so the argument is not rerun.
44.Reopen-decision
When should a decision be reopened?
When new information changes the facts, such as a failed requirement. Missing the meeting is not a reason.
45.Legal-deadline
How do legal requirements affect a roadmap?
A required disclosure with a date outranks a locked plan. Replan and name what leaves.
46.Deprecation
How do you turn off an old model version safely?
Give callers a date and a working new path before shutdown. Turning it off silently is a scheduled outage.
47.Breaking-change
What is a breaking change in a model API?
Removing or changing a field callers depend on. It needs a new version and a migration window, not a patch.
48.Postmortem-actions
What happens to postmortem action items?
They are scheduled ahead of new scope. Unscheduled actions are a choice to repeat the incident.
49.Tech-debt-paydown
How do you use an early finish?
Pay down known debt or pull a slice you can finish, rather than starting a large project you cannot stop.
50.Progress-measure
What counts as progress in a status update?
What a user can newly do. GPU hours and checkpoints are cost, not progress.