Card 1 of 50
Front. Hypothesis. What makes a research hypothesis testable?
These cards are original study material. They are not questions from a named employer or a live hiring loop.
All 50 cards in this deck
1.Hypothesis
What makes a research hypothesis testable?
It names a measurable outcome and a direction, written so that some possible result would count against it.
2.Preregistration
What is preregistration?
Writing down the hypothesis, data, metric, and analysis before seeing results, so later choices cannot be fitted to the outcome.
3.Primary-metric
Why fix one primary metric in advance?
With many metrics, some will improve by chance. Naming one up front makes the headline claim honest.
4.Secondary
What are secondary metrics for?
Context and diagnosis. Gains on them support the story but should not replace the primary result.
5.Train-val-test
What is each of the train, validation, and test splits used for?
Train fits weights, validation picks hyperparameters and checkpoints, and test is scored once at the end.
6.Sealed-test
What does it mean to keep a test set sealed?
Nobody looks at its scores while making design choices, so the final number is an unbiased estimate.
7.Test-tuning
What goes wrong if a threshold is tuned on the test set?
The test score is now optimistic, because the choice was fitted to it. It is no longer an untouched result.
8.Leakage
What is data leakage?
Information about the test labels reaching training or model selection, such as duplicates across splits or features computed after the outcome.
9.Temporal-split
What is a temporal split?
Training on earlier data and testing on later data, which mimics deployment and avoids future information leaking back.
10.Group-split
What is a group split?
Keeping all examples from one source, such as one user or document, in the same split, so the test measures generalization to new sources.
11.Baseline
What is a baseline?
An existing or simple method run under the same conditions, so the new method's gain has a reference point.
12.Strong-baseline
Why must baselines be tuned as hard as the new method?
An untuned baseline makes any method look better. Equal tuning effort keeps the comparison fair.
13.Budget-parity
What is compute or tuning parity?
Giving each method the same compute, data, and number of hyperparameter trials, so gains are not bought with extra resources.
14.One-change
Why change one thing at a time in a comparison?
If several things change together, the result cannot say which one caused the difference.
15.Leaderboard
Why is a gain over a leaderboard row weak evidence?
The rows may differ in data, compute, tuning, and scoring code. Only a controlled rerun isolates the method.
16.Ablation
What does an ablation show?
How much one component contributes, by removing it and holding everything else fixed.
17.Confounder
What is a confounder?
A factor that differs between the compared conditions and also affects the outcome, so it can explain the difference.
18.Seeds
Why run several random seeds?
Initialization and data order change results. Several seeds show whether a gain is larger than run-to-run noise.
19.Same-seeds
Why pair seeds across methods when possible?
Using the same seeds and data order for both methods removes some shared noise from the difference.
20.Variance-report
What should accompany a mean over seeds?
A spread, such as standard deviation, standard error, or a confidence interval, and the number of seeds.
21.Ci
What does a 95 percent confidence interval mean?
An interval built so that, over many repeated studies, 95 percent of such intervals would contain the true value.
22.Bootstrap
What is a bootstrap confidence interval?
Resample the test examples with replacement many times, recompute the metric each time, and take percentiles of those values.
23.Paired-test
Why use a paired test when two models share a test set?
Each example is scored by both models, so comparing per-example differences removes example difficulty from the noise.
24.P-value
What is a p-value?
The probability of a result at least this extreme if the null hypothesis were true. It is not the probability the hypothesis is true.
25.Significance-practical
What is the difference between statistical and practical significance?
A tiny gain can be statistically significant with enough data and still not matter. Report the effect size too.
26.Effect-size
What is effect size?
How large the difference is, in metric units or standardized units, separate from whether it is significant.
27.Power
What is statistical power?
The probability that a study detects a real effect of a given size. Small test sets have low power.
28.Test-size
Why does test set size limit the claims a paper can make?
On 500 examples, one example is 0.2 points, and noise can easily be a point or more. Small gains there are not resolvable.
29.Multiple-comparisons
What is the multiple comparisons problem?
Running many tests raises the chance that some look significant by luck alone.
30.Bonferroni
What is the Bonferroni correction?
Divide the significance threshold by the number of tests. It is simple and conservative.
31.Forking-paths
What is the garden of forking paths?
Many small analysis choices made after seeing data, each reasonable, that together make a chance result look real.
32.P-hacking
What is p-hacking?
Trying analyses, subsets, or metrics until one crosses the significance threshold, then reporting only that one.
33.Harking
What is HARKing?
Hypothesizing after the results are known: presenting a post-hoc explanation as if it were the original hypothesis.
34.Peeking
Why is peeking at results and stopping early a problem?
Stopping when the result looks good inflates false positives unless the stopping rule was planned for.
35.Cherry-pick
What is cherry-picking examples?
Showing only the outputs that support the claim. Qualitative examples should be random or chosen by a stated rule.
36.Noise-ceiling
What does annotator agreement set for a benchmark?
A rough ceiling. If annotators agree 94 percent of the time, scores near 94 cannot clearly show one system is better.
37.Kappa
What is Cohen's kappa?
Agreement between two annotators corrected for the agreement expected by chance.
38.Internal-validity
What is internal validity?
Whether the measured difference was really caused by the change being tested, not by a confounder or bug.
39.External-validity
What is external validity?
Whether the result holds outside the tested data, domain, or scale.
40.Construct-validity
What is construct validity?
Whether the metric actually measures the ability the paper claims, such as reasoning rather than pattern matching.
41.Proxy
What is a proxy metric?
A measurable stand-in for the real goal. Claims about the goal need evidence that the proxy tracks it.
42.Ood
What is an out-of-distribution test?
Evaluation on data from a different source, time, or population than training, to test robustness.
43.Causal-claim
What does a causal claim about a method require?
A controlled comparison where the method is the only difference. Correlation across published systems is not enough.
44.Scope-claim
What should a paper say about where its result applies?
Only the settings it tested: the datasets, model sizes, and conditions. Wider claims need wider evidence.
45.Error-analysis
What is error analysis?
Grouping a sample of the model's mistakes by type to see where and why it fails.
46.Negative-result
Why report negative results?
They save others from repeating dead ends and correct a literature that otherwise shows only successes.
47.Reproducibility
What is reproducibility?
Getting the same results from the same code and data. It needs fixed seeds, versions, and configs.
48.Replication
What is replication?
Getting a consistent result in a new study, with new data or a new implementation.
49.Contamination-check
How can a study check for test contamination?
Search the training data for test items or long n-gram overlaps, and compare scores on clean versus overlapping items.
50.Limitations
What belongs in a limitations section?
What the study did not test, the threats to validity it could not rule out, and what the result therefore cannot claim.