Backend performance debugging

Why can average latency remain stable while users report timeouts?

A small slow tail can worsen while most requests improve. Inspect high percentiles, timeout counts, and distributions by route and payload size rather than relying on one average.

Can you average each replica's p99 to obtain the service p99?

No. Quantiles are not additive. Merge compatible histogram counts or raw observations with correct traffic weighting, then calculate the percentile.

Overall latency rises while every endpoint's latency is unchanged. What can explain it?

Traffic shifted toward slower endpoints. Separate within-endpoint regressions from changes in request mix before optimizing code.

The server span is 40 ms but the client sees 800 ms. Where should you look next?

DNS, connection acquisition, TLS, network transfer, proxy queues, and client scheduling. Confirm exactly which boundaries the server timer includes.

Why does summing all child span durations overestimate request latency during parallel fan-out?

Overlapping operations run concurrently. Reconstruct the critical path and waiting intervals instead of adding parallel durations as if they were serial.

A trace shows a 600 ms parent and only 30 ms of children. What does the gap establish?

Only that much time is unaccounted for by child spans. Instrument queueing, local computation, serialization, and waits; do not automatically attribute the gap to CPU.

Why can a dashboard of error-retained traces misrepresent normal latency?

The retained sample intentionally overrepresents failures. Use population metrics for rates and distributions, and use selected traces to explain individual failure paths.

What three questions make resource bottleneck inspection systematic?

How busy is the resource, how much work waits for it, and are errors occurring? Apply utilization, saturation, and error checks to CPU, disks, network, and constrained pools.

A container shows 20% host CPU yet suffers CPU throttling. How is that possible?

Its cgroup CPU quota can be much smaller than host capacity. Compare usage with allocated CPU and inspect throttled periods and duration, not just host-wide utilization.

Why can 12% CPU on an eight-core host coexist with a CPU-bound service?

One serial thread may saturate approximately one core. Inspect per-thread utilization and profiles; aggregate host utilization hides single-thread bottlenecks.

What question does a sampled CPU profile answer well?

Where on-CPU execution time is spent during the sampled workload. It does not directly explain time blocked on locks, network responses, or scheduling.

In a CPU flame graph, does a wide frame necessarily mean that function's own instructions are expensive?

No. Width usually includes time in descendants. Distinguish inclusive from self cost before rewriting a wrapper that merely calls expensive code.

Latency is high but CPU profiles show little activity. What evidence is useful next?

Off-CPU stacks, lock waits, scheduler delay, I/O waits, and pool acquisition timing. Identify what the request waits for rather than optimizing code that is rarely running.

Why should a profile be captured during the actual regression rather than a quiet period?

The expensive path may depend on load, contention, data distribution, or cache misses. A quiet profile can accurately describe an irrelevant workload.

How can memory allocation cause latency spikes without a growing live heap?

Short-lived objects can drive frequent garbage collection and CPU overhead. Compare allocation rate, collection pauses, and retained heap rather than diagnosing every GC issue as a leak.

What distinguishes a leak from a high but stable cache footprint?

A leak retains objects without a bounded useful lifetime and grows across comparable workload cycles. Inspect retained-object paths and cache limits after collections, not one heap snapshot alone.

Why can process RSS grow while the managed live heap stays flat?

Native allocations, thread stacks, allocator fragmentation, memory mappings, or runtime reservations may account for the difference. Compare process mappings and runtime metrics before blaming managed objects.

How can enabling detailed request logging create a performance regression?

Formatting, serialization, synchronization, disk I/O, or exporter backpressure can enter the request path. Measure overhead and bound buffering while retaining the diagnostics needed for failures.

Why can using request IDs as metric labels degrade an observability system?

Each distinct label set creates a time series, producing unbounded cardinality. Keep request identifiers in logs or traces and use bounded dimensions for metrics.

Queries execute in 15 ms, but database spans take 500 ms. What should be timed separately?

Connection-pool acquisition, execution, result transfer, and decoding. Pool wait can dominate even when the database reports fast query execution.

Why might doubling the database connection pool increase latency?

More concurrent queries can increase contention, memory pressure, and I/O queueing at the database. Tune against measured throughput and wait time rather than assuming more connections mean more capacity.

How does tracing reveal an ORM N+1 query problem?

One request issues a query for a list and then repeated similar queries per returned row. Check query count versus result size; batch or join deliberately while watching row multiplication.

A PostgreSQL plan predicts 10 rows but produces 500,000. Why does this matter?

Cardinality estimates influence join order and algorithm choice. Inspect statistics, correlated predicates, and parameter sensitivity before forcing a different plan.

Does a PostgreSQL plan cost of 5,000 mean the query takes 5,000 ms?

No. Cost is a planner estimate in relative units. Use measured execution and wait timing to assess elapsed latency.

A PostgreSQL node reports actual time 2 ms and loops 1,000. How should its work be interpreted?

Reported per-loop timing is averaged, so repeated execution can contribute substantial work. Avoid treating that node as a one-time 2 ms operation or blindly summing inclusive node times.

Why is EXPLAIN ANALYZE on an UPDATE different from plain EXPLAIN?

It executes the statement. Use an appropriate controlled environment; rollback does not undo every possible external effect or sequence advance.

A query suddenly takes seconds with the same plan and row counts. What database evidence can separate waiting from execution?

Inspect wait events, blocking sessions, transaction age, and lock relationships. A stable plan does not rule out time spent waiting for another transaction.

How can a sort become slow only for larger tenants?

The working set can exceed its memory budget and spill to temporary storage. Inspect spill indicators and row counts; raising memory globally may multiply memory use across concurrent operations.

Why is cache hit rate alone insufficient to judge whether a cache helps?

Hits have their own network and serialization cost, and misses may be expensive. Measure end-to-end hit and miss latency, freshness, memory cost, and backend load avoided.

A deployment causes a temporary latency surge. What cache-related hypothesis is testable?

New instances start cold and send more requests to dependencies. Compare miss rate and backend QPS during rollout; test controlled warmup or gradual traffic ramping.

What metric pattern suggests synchronized cache expiry is overloading the database?

Periodic miss and backend-query spikes near TTL boundaries. Coalesce fills, stagger expiry, or use explicitly permitted stale serving, then verify the spikes shrink.

Why can reducing JSON payload size improve more than network transfer time?

It can reduce database materialization, object allocation, encoding, decoding, and client processing. Measure each stage and remove unused fields rather than compressing unnecessary data blindly.

When can response compression make a small backend response slower?

CPU and buffering overhead can exceed transfer savings, especially on fast links and tiny payloads. Benchmark representative payload sizes and concurrency with both latency and CPU measurements.

Why can creating a new HTTPS connection for every request dominate a fast downstream call?

DNS, TCP, TLS, and connection setup can exceed the application work. Measure setup separately and use a correctly bounded reusable connection pool.

Why can packet loss increase latency without saturating link bandwidth?

Retransmissions and congestion-control responses delay delivery. Inspect retransmission and RTT evidence; low utilization does not prove the network is healthy.

Disk throughput is below its advertised maximum but requests wait. What else matters?

IOPS limits, operation size, random access, latency, queue depth, and shared-device contention. Peak sequential bandwidth is not a universal service-capacity measure.

A stable service completes 200 requests per second with mean time in system 0.25 seconds. What is average in-flight work?

L = lambda × W = 200 × 0.25 = 50 requests, using consistent boundaries and a stable system. This relates averages, not tail latency or a safe maximum concurrency.

What does steadily growing queue depth imply when input rate stays above completion rate?

The system is not in steady state and waiting time will keep increasing until a limit is reached. Reduce arrivals or increase bottleneck capacity; a larger queue only delays failure.

Why can latency rise sharply near maximum sustainable throughput?

Small arrival or service-time variations create queues when spare capacity disappears. Find the throughput-latency knee under representative load and retain headroom.

Why can a closed-loop load test hide overload behavior?

Clients wait for responses before sending more work, so offered load falls as the server slows. An arrival-driven test exposes queueing under a sustained external request rate, provided the generator keeps up.

What measurement error occurs when a load generator omits requests that would have arrived during a stall?

It underrepresents users' waiting times during the stall. Track intended arrival times or use an appropriate arrival-driven test rather than reporting only completed request service times.

How can you tell whether a benchmark plateau comes from the load generator?

Inspect generator CPU, network, connection limits, and actual versus intended request rate. Add independent generators and confirm the server is receiving the targeted workload.

Why can a benchmark on evenly distributed small records miss a production bottleneck?

Large tenants, skewed keys, oversized payloads, and rare expensive paths affect contention and tails. Preserve relevant distributions and report results by cohort.

Why should warmup and steady-state results be reported separately?

JIT compilation, connection setup, and caches alter early behavior. Cold starts may matter operationally, while steady-state throughput answers a different question.

If 40% of request time is a serial step, what is the best possible speedup from making everything else instantaneous?

At most 1 / 0.40 = 2.5 times. This assumes the measured fractions remain valid and excludes new overhead; optimize the critical bottleneck rather than only the easiest section.

What makes a proposed performance fix falsifiable?

A specific bottleneck hypothesis, predicted metric change, representative workload, and acceptance threshold. Change one relevant factor where practical and compare both latency and resource use.

What should a performance regression check preserve besides a single runtime threshold?

Input distribution, software configuration, hardware or normalized environment, warmup policy, and repeated measurements. A noisy one-off timing can fail good changes and accept bad ones.

p99 falls after a deploy while errors rise sharply. Why is that not evidence of an improvement?

Fast rejection can lower measured latency while reducing useful service. Compare successful-operation latency, completion rate, and correctness alongside the full request distribution.

When is a bottleneck fix verified beyond a local microbenchmark?

The representative end-to-end workload improves without unacceptable errors, resource cost, or correctness changes, and production rollout confirms the expected mechanism. Watch for the bottleneck moving elsewhere.