Metrics and experimentation

How should an activation event be chosen?

Identify an early behavior plausibly connected to delivered value and later success, then validate that relationship. A convenient click or completed form is not automatically activation.

Users who activate retain better. Does forcing that event improve retention?

Not necessarily. Motivation or account quality may cause both behaviors. Test an intervention that changes activation and measure downstream outcomes rather than treating correlation as a causal mechanism.

Why pair a leading indicator with a lagging outcome?

The leading measure gives faster feedback but may stop predicting value when optimized. The lagging outcome checks whether the intended benefit actually materialized over the relevant horizon.

What should a metric tree make explicit?

How measurable components contribute to an outcome, including denominators and assumptions. Algebraic decomposition is different from a causal claim that changing any component will improve the whole.

Revenue rose 20%. What decomposition helps explain it?

Separate customer count, purchase frequency, price, product mix, refunds, and timing under a consistent definition. Attribute changes carefully because components can interact and arithmetic explanation is not causal attribution.

Why can conversion rise while the business gets worse?

The denominator may shrink or become more selective, such as fewer low-intent visitors entering the funnel. Examine eligible traffic, absolute completions, quality, and acquisition mix alongside the rate.

A conversion rate rises from 4% to 5%. What are the absolute and relative changes?

The absolute change is one percentage point; the relative increase is 25%. Report both with denominators and uncertainty so a large relative number does not conceal a small base.

Why should overall conversion usually use total conversions divided by total eligible users?

A simple average of segment rates gives small and large groups equal weight. Use weights matching the intended population, and distinguish an observed mix from a standardized comparison.

Why compare retention at the same cohort age?

A new cohort has not had the same opportunity to churn or return as an older one. Align follow-up and eligibility before interpreting a difference as product improvement.

What retention measure fits a yearly tax-preparation product?

Return or successful reuse in the next relevant annual cycle, with eligibility defined. Monthly activity may describe support or browsing rather than the product's core recurring value.

What must a subscription churn rate state?

Whether it counts customers or revenue, which starting population is eligible, the period, and how pauses, reactivations, and involuntary payment failures are handled. Different definitions answer different questions.

How can net revenue retention exceed 100% while customers leave?

Expansion among retained accounts can exceed contraction and lost revenue in the starting cohort. Pair it with customer retention and concentration so expansion does not hide broad attrition.

Why is time spent not always a positive engagement measure?

More time may mean confusion, repeated failures, or inefficient work. Connect usage to successful outcomes and user intent; a productivity tool may improve by reducing time required.

What should accompany a growth metric for generated content?

Task quality, harmful or unsupported outputs, user correction burden, latency, and cost at relevant slices. More generated items can be a regression if users must repair or discard them.

What should an event tracking plan include?

Trigger semantics, unique IDs, required properties, schema version, responsible owner, and validation examples. Define whether the event records an attempt, server acceptance, or completed outcome.

Why distinguish button clicks from successful backend actions?

Clicks can be repeated, blocked, or followed by errors. Use authoritative completion events for outcomes and click events for interaction diagnostics, linked by a request or operation identifier.

How can retrying analytics delivery inflate a metric?

The same logical event may be received multiple times. Use stable event IDs and a documented deduplication window; deduplicating by user and timestamp can erase real repeated actions.

A metric drops after a frontend release. What should be checked first?

Event schema and delivery, consent behavior, identity joins, and backend outcomes. Separate a real behavior change from instrumentation loss before declaring product harm or success.

What can go wrong when anonymous and signed-in identities are merged?

Users may be double-counted or unrelated people merged on shared devices. Define identity rules and version them; historical counts can change when stitching logic changes.

Why should a dashboard label its completeness cutoff?

Delayed events and corrections can make recent periods partial. A visible cutoff and revision policy prevent comparing an incomplete current day with a fully matured historical day.

What is a useful experiment hypothesis?

A specified change for an eligible audience is expected to improve a defined outcome through a plausible mechanism, with guardrails and a decision rule. 'Test the new design' leaves success ambiguous.

Why choose the randomization unit before sizing an experiment?

User, account, store, and market assignment produce different dependence and spillovers. Sample-size and analysis calculations must reflect the independent units, not simply the number of clicks.

Why keep a user's experiment assignment stable when appropriate?

Switching experiences can create contamination, inconsistent workflows, and carryover. Persist assignment at the chosen unit and define behavior across devices, login, and experiment version changes.

Why distinguish assignment from actual exposure?

Assigned users may never reach the feature, while exposure can be affected by treatment. Primary assignment-based analysis preserves randomization; exposure-only comparisons need a justified design to avoid selection bias.

What should happen when an experiment has a sample-ratio mismatch?

Pause interpretation and investigate allocation, eligibility, logging, and filtering against the planned split. Do not explain a suspicious imbalance away merely because the outcome looks favorable.

What can an A/A experiment validate?

Assignment, event joins, metric implementation, and expected false-positive behavior under identical experiences. One significant result can occur by chance; assess diagnostics and repeated calibration rather than demanding every p-value be large.

How should a minimum detectable effect be chosen?

Connect it to a decision-relevant change and feasible sample size, rather than selecting the smallest number that makes a tool run. Distinguish the effect worth acting on from the effect the available data can detect.

What does an underpowered null result mean for a PM?

The experiment may not distinguish a useful effect from noise. Read the estimate and interval against the practical threshold; 'not significant' does not establish that the variants are equivalent.

Why can reaching the sample size in one day still be insufficient?

The result may omit weekly cycles, learning, delayed outcomes, or operational conditions. Plan sample size and observation horizon together, with a stopping rule that matches the analysis.

Why is stopping at the first favorable ordinary p-value risky?

Repeated opportunities to stop inflate false-positive risk under a fixed-horizon test. Use the planned endpoint or a valid sequential design, while retaining separate emergency safety stops.

A change is statistically significant but saves 0.1 seconds at high maintenance cost. What decides shipping?

Compare the plausible benefit with user value, operating cost, risk, and alternatives. Statistical significance describes evidence against a null, not whether the effect is worth building or maintaining.

An estimated lift is 2% with an interval from -1% to 5%. What is the honest conclusion?

The data remain compatible with a small decline and a useful improvement under the method's assumptions. Do not label it a proven win or no effect; decide whether more evidence is worth its cost.

Why define an acceptable guardrail deterioration before testing?

It turns 'do not hurt latency' into a measurable decision. A nonsignificant decline may still allow substantial harm; compare uncertainty with the prespecified margin rather than treating p > 0.05 as safety.

Why can choosing the best of many metrics create a false win?

Noise gives many chances to find a favorable result. Prespecify primary outcomes, handle multiplicity appropriately, and label unexpected improvements as exploratory until confirmed.

One segment is significant and another is not. Does that prove different treatment effects?

No. Test the difference or interaction directly and account for segment selection and sample size. Different p-values can arise from different precision even when estimated effects are similar.

What should be checked when an experiment's early lift fades?

Novelty, user learning, changing composition, delayed harms, and measurement maturity. Inspect prespecified time patterns and longer follow-up rather than choosing the strongest initial window.

Why can switching the same users rapidly between variants bias results?

Earlier treatment may change later behavior or state, so the control period is not untreated. Use an appropriate assignment design, washout where defensible, and analysis that accounts for temporal dependence.

Why might individual randomization fail in a marketplace?

Treated buyers can change outcomes for untreated sellers or other buyers. Consider market/cluster or switchback designs and define whether the desired effect includes equilibrium and spillover consequences.

How does a rollout canary differ from a product A/B test?

A canary primarily limits operational exposure and checks release health. An A/B test estimates an outcome contrast under a defined randomized design; a small healthy canary does not establish product value.

Why can comparing the first rollout users with everyone else be misleading?

Early access may select enthusiastic, high-value, or technically capable users. Their outcomes can differ before the feature; randomized assignment or another credible comparison is needed for causal attribution.

What can a longer-lived holdout reveal after launch?

Effects that compound or fade, such as retention, habit formation, or delayed cost. Maintain ethical and operational suitability, stable assignment, and sufficient sample size; indefinite holdouts are not automatically justified.

A feature increases clicks but decreases task completion. What does that reveal?

The proxy can move without the intended value, possibly through confusion or distraction. Revisit the metric relationship and optimize successful outcomes rather than preserving the attractive intermediate number.

Why is attributed revenue not necessarily incremental revenue?

Attribution allocates credit among observed touchpoints; customers may have purchased without the campaign. Incrementality requires a credible counterfactual, such as an appropriate randomized holdout.

How should a satisfaction score be interpreted alongside behavior?

Check who responded, timing, question wording, and uncertainty, then compare task outcomes and retention. A score shift can reflect respondent mix rather than a changed experience.

What happens when a metric becomes a target with no balancing checks?

Teams or users may optimize the measured proxy while degrading unmeasured value. Inspect behavior changes and pair outcomes with quality and integrity measures instead of assuming the metric's meaning stays fixed.

When should an inconclusive experiment be extended or rerun?

When additional valid evidence could change an important decision and the design supports the revised plan. Avoid extending repeatedly until significance; document new hypotheses and use suitable inferential rules.

What belongs in an experiment decision memo?

Assignment and data-quality checks, effect sizes and uncertainty, practical thresholds, guardrails, important limitations, and the decision. Separate prespecified findings from exploratory slices and record follow-up monitoring.

Why continue monitoring after a positive experiment?

Full rollout can change load, audience mix, interactions, and long-term behavior. Validate the expected benefit and operational constraints at scale, with owners and a recovery path if results do not transfer.

When should a product metric be changed or retired?

When its definition no longer reflects value, data quality is inadequate, or optimization has broken the proxy relationship. Version the definition and bridge old and new measures so a reporting change is not mistaken for product progress.