Production Incident Diagnosis

1.An order trace shows 20 μs to validate, 480 μs waiting for a sender, and 25 μs to write. Which stage explains most of the delay?

2.The median acknowledgment stays at 2 ms while p99 rises from 5 to 40 ms. What should the incident report say?

3.A strategy records 5 ms from intent to acknowledgment. Gateway records show 4 ms before transmission and 1 ms afterward, both using a reliable time base. Where should you start?

4.A gateway receives 12,000 orders per second while the sender processes 10,000 per second for three seconds. Ignoring drops, how much does the queue grow?

5.After a burst, a queue has 9,000 orders. New arrivals average 6,000 per second and service averages 9,000. When should the backlog clear at those rates?

6.A chart combines all venue sessions and shows a latency jump. One session changed routes just before the jump. Which breakdown tests whether that path is implicated?

7.A gateway reports negative network latency when subtracting timestamps from two hosts. Which check comes first?

8.An outbound log says an order was queued to send. No wire observation or response exists. What does the log establish?

9.One order lacks an acknowledgment. The venue's status channel identifies it as live under the original client order ID. What should local state reflect?

10.A fill appears on the venue's execution copy feed, but the primary order session still shows pending acknowledgment. What is the best next step?

11.A strategy sees an acknowledgment timeout on one order. The next order on the same session is accepted normally. What can you infer about the first?

12.The last inbound message was sequence 811, and the next inbound message is 813. What should the operator investigate?

13.A venue execution report is present in the gateway journal, but the strategy's blotter lacks it. Where should the trace continue?

14.The gateway sends an order and captures a timely reply at its network interface. The application delivers the reply much later. What local evidence would narrow the delay?

15.A new deployment coincides with a latency spike. The undeployed control pool is stable at similar traffic. What is the strongest initial inference?

16.All strategies on one host stall together, while peer hosts on the same venue remain healthy. What is the best first scope?

17.Market data and order entry both slow on one rack. Venue sessions outside the rack remain healthy. What should be examined?

18.One venue's acknowledgment delay rises across multiple desks and hosts. Their local queue times remain stable. What evidence best tests a venue-side or shared-path issue?

19.Order entry latency rises after enabling per-order synchronous logging. CPU remains modest but disk flush times rise. What should the engineer examine?

20.A sender thread uses 100% of one core while host CPU averages 22%. Which metric is most relevant to its capacity?

21.Acknowledgments appear late only when large snapshots are processed in the same event loop. What would support head-of-line blocking?

22.A timestamp for the first slow acknowledgment predates the alleged deploy by five minutes. What should happen before asserting causality?

23.A timeout alert only counts completed orders. Many orders are still pending past the observation window. What metric is missing?

24.A service has recorded cumulative fill quantity of 120. A later report for the same order gives cumulative quantity of 200. No correction or bust applies. How much additional fill does the report imply?

25.A tracing change samples only successful acknowledgments. Which sample is needed to diagnose missing ones?

26.A latency alert includes deliberate pacing time for a new strategy, while the gateway service times are stable. What would clarify impact?

27.A monitoring server scrapes queue depth once a minute. Traders report brief spikes between scrapes. What measurement would help?

28.Two gateways share the same order ID suffix but use different sessions. An analyst joins logs on suffix alone. What is the problem?

29.An order is locally marked canceled after its cancel message was queued. A late fill arrives. What likely went wrong with the state transition?

30.A primary session reconnects, but the outbound journal is missing its last record. What state should that order have until verified?

31.A trace reports 12 ms between client and gateway using different clocks. Same-host spans total only 3 ms. Which is the defensible next measurement?

32.The market data sequence is current but quotes are visibly old for one instrument. What should you verify?

33.The first order after a period of inactivity is slow. Later orders on the same route are normal. Which test would isolate an idle-path issue?

34.Several strategies see similar latency spikes, but all alerts use the same central metrics collector. What distinguishes a collector issue from a trading issue?

35.An incident timeline includes a network alarm, a process restart, and a jump in unknown order states. What should the timeline record for each event?

36.A single venue session has unresolved orders. Other routes have independent risk budgets and healthy reconciliation. What is the most targeted immediate action?

37.The desk stops accepting new orders on an affected route, but venue orders are already resting. What additional action is needed?

38.A gateway cannot verify current risk limits because the risk service is unavailable. The desk wants to keep taking new orders. What is the safe default?

39.A strategy retries timed-out orders with fresh IDs. The original orders may be live. What action reduces the immediate risk?

40.A route has a severe backlog. Cancels and new orders use the same queue. The documented overload policy permits prioritizing cancels and supports separate queues. What should the operator do?

41.A desk sends a mass-cancel request and receives a session-level response saying the request was accepted. The protocol provides per-order outcomes separately. What must the desk verify?

42.A venue documents cancel-on-disconnect only when a session flag is enabled. The affected connection drops, and no one has verified its flag. What assumption is safe?

43.The standby gateway is ready, but the old primary might still be sending during a network partition. What is required before enabling the standby writer?

44.A release introduced tail latency. An older version reads a different order journal format. What should happen before rollback?

45.A trading lead asks for the first incident update while the cause is unknown. Which message is most appropriate?

46.A responder wants to restart a gateway to stop a worsening outage. Volatile diagnostics could disappear. Which approach fits?

47.An acknowledgment timeout triggers automated status queries for thousands of orders. Those queries worsen the queue. What change is appropriate?

48.A market data route becomes stale for one product group. Risk estimates depend on current prices for that group. What containment fits?

49.A gateway has a 2,000-share open buy and a separate unresolved 500-share buy. Its exposure ceiling is 3,000 shares. If both could fully fill and there are no other positions or orders, how much additional buy capacity remains?

50.A router stops submitting to an affected venue and reroutes to a healthy one. Why might it still reduce the rerouted quantity?

51.Two engineers begin separate restarts of shared components. What coordination change reduces further disruption?

52.A canary gateway causes slow acknowledgments at low volume. Its peers use the earlier build and have spare capacity. Which immediate change has the smallest expected scope?

53.The affected gateway is still responsive enough to cancel, but new orders are causing timeouts. Which temporary operating mode fits if supported by policy?

54.A cancel request returns an unknown-order response. The original order may have filled. What is the next step?

55.A temporary throttle limits a route to 600 orders per second. Three independent strategies request 250 each. How far does total demand exceed the throttle?

56.A production queue contains entry requests that are now too old under the strategy's policy. What should happen as the queue drains?

57.An outage degrades an automated hedge, while the primary strategy can still place new orders. What should containment consider?

58.A desk runs a documented kill switch, but its status panel is fed by the degraded gateway. What can confirm its effect?

59.Two control options are available. One removes traffic from a single unhealthy host. The other stops every market. How should the incident lead choose?

60.A gateway can log excessive per-message debug data during a slowdown. Which diagnostic plan minimizes extra risk?

61.A venue declares its session unreliable and gives an approved out-of-band status channel. What is a sensible use of that channel?

62.A strategy is paused, but a scheduler can start a second copy later in the day. What should the incident owner verify?

63.A recovered connection accepts new messages, but the system has a gap in prior executions. What should remain blocked?

64.A gateway outage creates an unknown position in a thin contract. The desk can hedge using a liquid proxy. What should determine the hedge size?

65.A backup feed appears healthy, but symbol mappings differ from the primary. What should happen before using it to reopen quoting?

66.The incident affects both entry and reporting. Which owner should confirm which trading activities are allowed during containment?

67.A protective throttle blocks new entries, but bulk status requests share its limit with cancels. What should the team check?

68.A queue is growing on one instrument after a malformed reference record caused repeated processing errors. What is a bounded containment step?

69.A trading service uses a backup for outbound order traffic. The backup depends on the same degraded risk store. What does this imply?

70.After a pause, three venues still show live orders. The team needs to choose where to start canceling. What is a defensible priority?

71.A gateway reconnects and heartbeats pass. One earlier order is still unresolved. What must be done before restoring new exposure on the affected route?

72.After replay, a venue shows 17 open orders and the local ledger shows 17. Why is that count insufficient?

73.A replay returns a fill already present on the execution copy feed. Both reports carry the same authoritative execution ID. What should happen?

74.A venue reports an order for 1,000 shares with 600 filled and 400 still open. Local state shows 1,000 open and no fill. What should reconciliation produce?

75.A failover writer reconstructed order state from a durable journal. What is the strongest external cross-check before it sends?

76.A standby has up-to-date risk limits, but its market data snapshot is stale. What should remain paused?

77.After a fix, the team submits a small authorized limit order and requests cancellation on one route. What evidence should it capture?

78.A readiness dashboard shows average acknowledgment latency below target, but p99 is still above the agreed threshold. What should happen under the stated gate?

79.Recovery tests pass at 5% traffic. The incident first appeared above 60% load. How should the next stage be planned?

80.During the ramp, one session's unresolved-order count rises above the agreed stop threshold. What should the operator do?

81.A strategy restarts with a persisted list of unsent orders from before the outage. What should it check before releasing them?

82.The gateway restored its order journal through sequence 500. The venue's latest confirmed replay watermark is 510. What remains before claiming complete coverage?

83.A per-route throttle was tightened during containment. It is still set below the minimum needed for expected production volume. How should normal capacity be restored?

84.A patch removes the visible latency spike. It was tested on a single route, but the same component serves three routes. What evidence is still needed?

85.An exchange sends a late trade correction after the local ledger initially reconciled. What should the recovery process do?

86.A desk defines safe resumption as zero unresolved orders for the affected session. Which evidence supports it?

87.One product resumes and passes entry checks. Cancel reports still arrive late. Which condition is unmet?

88.A resumed route has zero rejects during a period with no orders. What does that establish?

89.A recovery runbook requires 15 consecutive healthy minutes before raising limits. The latest breach was six minutes ago. What is the earliest allowed ramp if conditions stay healthy?

90.At resumption, four unresolved orders remain on an isolated product the team will keep paused. What should be checked before resuming other products?

91.A venue confirms cancel-on-disconnect was enabled. What evidence is still useful after reconnecting?

92.A local aggregate position matches the clearing report, but executions are allocated to the wrong strategy. What should the incident owner do?

93.A fix works in historical replay. The team has never exercised failover with an order in flight. Which additional test addresses that gap?

94.A rollout guide lists resume checks but no named approver. What should be added?

95.Production has resumed at limited size. Which observation best supports increasing order size?

96.An order entry fix improves acknowledgments, but the position updater is still processing old executions. What gate should remain in place?

97.During a canary test, the order and its cancel both succeed, but the risk reservation remains. What should happen?

98.A recovery plan allows fallback to the old route if latency rises. That route still has an unresolved session. What should be fixed in the plan?

99.An engineering lead proposes closing the incident as soon as order entry returns. What evidence would justify closure?

100.A postmortem identifies a queue alert that fired after the first missed acknowledgment. What preventive change follows most directly?

101.A team wants to verify a new latency dashboard before using it as a resume gate. What comparison is best?

102.Recovery found 950 of the 1,000 execution reports expected from an independent record. What fraction of the expected reports is still unaccounted for?

103.A report says recovery was safe because no duplicate executions appeared. There was no test of reconnect with replay. What evidence would address this gap?

104.After a canary passes, a dependency used only by full-size orders becomes degraded. What should the owner do?

105.A desk asks for a concise resume decision. Which evidence should the incident lead present?