Card 1 of 20
Front. Decision First. A team proposes a large fine-tuning sweep. What decision should the experiment plan identify first?
These cards are original study material. They are not questions from a named employer or a live hiring loop.
All 20 cards in this deck
1.Decision First
A team proposes a large fine-tuning sweep. What decision should the experiment plan identify first?
Which product or technical choice the result will change, the alternatives, and the evidence threshold. A sweep without a decision can consume budget without reducing relevant uncertainty.
2.Cheap Baseline
Why compare a proposed fine-tune with a strong prompt or retrieval baseline before allocating a large training budget?
The simpler system may meet the requirement at lower operational cost. Establish the actual quality, latency, and cost gap before funding a more complex approach.
3.Hypothesis Mechanism
How should 'more data will help' be made testable?
Specify which failure category the new data addresses, how it is selected, the expected metric change, and a controlled comparison. More examples can reinforce existing bias or add redundant content.
4.Stage Gates
How can a manager stage an uncertain research investment?
Use a small feasibility test, a reproducible offline comparison, and then a limited product experiment, each with explicit continuation criteria and a budget cap.
5.Primary Metric
Why agree on a primary success metric before inspecting experiment results?
It limits choosing a favorable metric after the fact. Keep guardrails and diagnostic metrics, but state which evidence actually supports the decision.
6.Practical Effect
A model's score improves significantly by 0.1 percentage points. What should determine whether that matters?
The practical value, uncertainty, error distribution, and added serving or development cost. Statistical significance alone does not establish a worthwhile product change.
7.Eval Readiness
Why can improving the evaluation set be a better first investment than launching more training runs?
If labels, coverage, or scoring are unreliable, additional runs optimize noise. Establish an evaluation capable of distinguishing the changes the team needs to decide between.
8.Data Budget
What should a training-data budget include beyond annotation price per row?
Collection, filtering, adjudication, deduplication, provenance, integration, and evaluation of usefulness. Low unit price can hide a high cost per usable example.
9.Compute Budget
How should a manager compare the cost of two experimental approaches fairly?
Include failed runs, tuning trials, preprocessing, evaluation, storage, and engineering effort, not only the final successful training run.
10.Selection Budget
Why is a new model selected from 100 trials not directly comparable to one baseline run?
The search procedure has a selection advantage. Allocate comparable tuning opportunities or report the different selection budgets explicitly.
11.Run Manifest
What should every experiment record so another engineer can reproduce it?
Code, data, model and tokenizer revisions, configuration, evaluation version, seeds, hardware-relevant details, and output artifacts. A run name and final score are insufficient.
12.Stopping Rule
Why should a team define stopping criteria before an expensive sweep begins?
It prevents endless continuation driven by sunk cost or one promising outlier. Specify success, futility, budget, and failure conditions, with any sequential statistical method planned appropriately.
13.Failed Experiments
How should a manager evaluate a well-run experiment that disproves the hypothesis?
Assess whether it reduced important uncertainty with credible evidence and stayed within scope. Rewarding only positive results encourages selective reporting and weak comparisons.
14.Null Result
A study finds no clear gain. What should the review distinguish?
Evidence of a negligible effect from a study too small or noisy to decide. Use the uncertainty interval and practical threshold to choose stopping, redesign, or more data.
15.Interaction Budget
When is a small factorial experiment worth more than testing improvements one at a time?
When components plausibly interact, such as data filtering and training objective. Testing combinations can reveal that an apparent individual failure is conditional on another component.
16.External Validity
What evidence is missing when a method wins on a polished demo but is proposed for broad rollout?
Representative users, difficult cases, missing evidence, latency, costs, and failure recovery. Define the intended operating population and test it directly.
17.Online Unit
Why must an online model experiment choose its randomization unit deliberately?
Users, sessions, or organizations can influence multiple requests and contaminate conditions. Select a unit matching exposure and interference, then analyze uncertainty accordingly.
18.Label Delay
How should delayed outcome labels affect an experiment schedule?
Budget time for outcomes to mature and define which interim signals are only proxies. Do not declare final quality from incomplete labels without addressing the resulting bias.
19.Portfolio
Why should an AI experiment portfolio include both incremental and uncertain approaches?
They serve different needs: dependable improvements and learning about larger opportunities. Allocate explicit budgets and decision checkpoints rather than treating all work as equally predictable delivery.
20.Decision Record
What should an experiment review produce besides a chart?
A decision, supporting evidence, uncertainty, scope limits, artifact links, and the next owner or stopping action. Results become useful when they change an explicit plan.