FreeExam
A closed notebook, a blank card, and a pencil on a wooden desk.

Signal research and backtest judgment

24 cards

Explain when a signal was knowable, how to test it on unseen data, and whether the edge survives realistic execution.

Card 1 of 24

Front. Research Hypothesis. A supplier-delay feed updates every Friday. What would you specify before looking at returns?

All 24 cards in this deck

  1. 1.Research Hypothesis

    A supplier-delay feed updates every Friday. What would you specify before looking at returns?

    State who might react slowly and why, which securities and return horizon should show the effect, and what result would undermine the mechanism. Record the comparison and metric before running the test.

  2. 2.Data Availability

    A broker report is stamped 08:00, but your feed usually receives it at 08:12. Which timestamp controls the signal?

    Use the first time the strategy could actually consume the report, including feed and processing delays. A report timestamp by itself does not establish tradability.

  3. 3.Point in Time Joins

    A vendor backfills corrected share counts each quarter. How would you join share count to a daily signal?

    Join each decision time to the latest version available by that time. Keep the historical releases and corrections so a later correction cannot silently change an earlier signal.

  4. 4.Historical Universe

    A database only exports tickers that are still listed today. How would you rebuild the universe for an old test?

    Use historical security identifiers and membership or eligibility records as of each decision date. Keep exits, delistings, and their realized outcomes so the test reflects the choices that existed then.

  5. 5.Missingness Drift

    A data vendor starts covering many more small companies halfway through your backtest. What would you check before attributing the result to the signal?

    Measure coverage by date and company size. Repeat the comparison on a consistently covered cohort and check whether the missing-data rule drives the result.

  6. 6.Signal to Fill Clock

    A signal uses an auction print that arrives after the order cutoff. How should the simulation time the trade?

    Start with the first eligible order after the strategy receives and processes the print. Price it at a feasible execution opportunity and measure returns from the resulting exposure.

  7. 7.Execution Aligned Labels

    The label is tomorrow's close versus today's close, but orders first fill tomorrow morning. How would you align the target with the trade?

    Measure from a feasible entry after the signal to the intended exit. Split overnight and intraday returns if you want to learn how much of the apparent edge occurs before entry.

  8. 8.Train Only Preprocessing

    A researcher imputes missing values with the median from the entire history before fitting a walk-forward model. How would you repair the pipeline?

    Fit the imputer inside each training window and apply its fitted values to the following validation window. Use the same rule for other learned transformations, including scaling and feature selection.

  9. 9.Walk Forward Validation

    The production model will retrain monthly. What should each validation step reproduce?

    Fit using only data and mature labels available on the historical retrain date. Freeze that fit for the next decision window, record out-of-sample outcomes, and advance the clock on the same schedule.

  10. 10.Overlapping Label Boundary

    Training ends Friday, but the last training examples use returns through the following Friday. Validation begins on Monday. What should change?

    Exclude training examples whose outcomes would be unknown at the fit time. Check the actual label intervals at each boundary so no training outcome contains returns from the validation period.

  11. 11.Holdout Reuse

    After seeing the holdout result, the team changes position limits and reruns the test. What evidence would you ask for?

    Treat the inspected period as development data for the revised strategy. Freeze the full rule and evaluate it on fresh, unseen data or collect forward observations.

  12. 12.Research Trial Log

    Only one factor from a long research project makes it into the final memo. What else belongs in the evidence?

    Record the candidate families, parameter variants, data cuts, and selection rule. A trial log gives reviewers the context to judge selection bias and plan a clean confirmation.

  13. 13.Parameter Stability

    A signal works at a 17-day window and fails at nearby windows. How would you investigate the spike?

    Check whether a mechanism predicts improvement near that horizon, inspect which observations create the peak, and check the neighborhood on fresh data. Record the windows already tried.

  14. 14.Serial Dependence

    Daily strategy returns cluster for weeks. What assumption should you revisit before reporting a confidence interval for mean return?

    Daily observations may be dependent. Estimate uncertainty with a method that allows serial dependence, such as a block bootstrap, and check how the interval changes with block length.

  15. 15.Factor Exposure

    The signal earned most of its return during a small-cap rally. How would you test for incremental predictive value?

    Measure size exposure and compare with a size-matched baseline or a portfolio constrained to similar size exposure. Recheck forward returns and trading costs after the adjustment.

  16. 16.Simple Benchmark

    A nonlinear model improves net returns over an equal-weight book. Which additional benchmark would help justify the model's complexity?

    Use a simple rule built from the same inputs. Match the universe, execution timing, and risk constraints so the comparison isolates the value of the model.

  17. 17.Rank Prediction and Pnl

    The daily rank correlation is positive, but the long-short portfolio keeps losing money. What would you inspect?

    Check whether predictive value reaches the tails actually traded. Compare gross and net returns and break out turnover, sizing, exposures, and execution costs.

  18. 18.Net Return Arithmetic

    A rebalance earns 14 basis points gross. The sum of absolute traded weights is 40% of capital, and cost is 20 basis points per dollar traded. What is the net return on capital?

    Trading costs 0.40 × 20 = 8 basis points of capital. The net return is 14 − 8 = 6 basis points before any other costs.

  19. 19.Passive Fill Assumption

    A simulation fills each resting buy order when the market first touches its limit price. What would you test before trusting the fills?

    Check queue position, traded volume, and partial fills using the available market data. Stress missed fills and compare the price moves following filled and unfilled orders.

  20. 20.Borrow Constraints

    The strongest short signals appear in names with scarce borrow. What should a realistic replay include?

    Use available borrow and fees at the trade time. Enforce locate constraints and model recalls or forced covers where material, then measure how the feasible short book changes.

  21. 21.Capacity Curve

    The strategy is profitable in a small pilot. What would you vary before claiming it has room for ten times the capital?

    Simulate a range of capital levels with position and participation limits, partial fills, and size-dependent impact. Report how net return and the ability to exit change as the book grows.

  22. 22.Regime Concentration

    One stress episode accounts for most of the strategy's cumulative gains. What should be in the research review?

    Show results with and without that episode and compare exposures and costs across periods defined by information available at the time. State whether the mechanism needs that regime to work.

  23. 23.Forward Shadow Test

    The historical test passes. What would you log in a live shadow run before turning on trading?

    Log data arrival, generated orders, feasible fills, and expected costs with timestamps. Compare those measurements with the simulator and document discrepancies before allocating risk.

  24. 24.Research Reproducibility

    A reviewer cannot reproduce last month's backtest after a vendor refresh. What should the research artifact identify?

    Record the input snapshots and their release times, code version, fitted settings, run dates, universe rules, and execution assumptions. Keep enough versioned data or replay instructions to recover the result.