Card 1 of 20
Front. Utilization Meaning. Why does high GPU utilization not prove that a workload achieves high useful FLOPs?
These cards are original study material. They are not questions from a named employer or a live hiring loop.
All 20 cards in this deck
1.Utilization Meaning
Why does high GPU utilization not prove that a workload achieves high useful FLOPs?
A sampled busy metric can indicate kernels are active while they are memory-bound or inefficient. Correlate utilization with throughput, kernel profiles, memory traffic, and the workload's actual objective.
2.Idle Input
GPUs alternate between short busy bursts and long idle gaps during training. What should be inspected?
Data loading, tokenization, CPU scheduling, storage, host-to-device transfer, and synchronization. A timeline can show whether kernels wait for input rather than compute.
3.Reserved Memory
Why can nvidia-smi show more used memory than PyTorch reports as allocated tensors?
The caching allocator reserves memory beyond live tensor allocations, and CUDA contexts or other allocations also consume memory. Compare allocated, reserved, and device-level usage before diagnosing a leak.
4.Empty Cache
Why does torch.cuda.empty_cache() not solve memory held by live tensors?
It releases unused cached allocator blocks, not referenced tensor storage. Find retained tensors or reduce the live working set; cache clearing is not a general OOM fix.
5.Peak Phase
Why should GPU memory be measured across a full training step rather than only after model loading?
Activations, gradients, optimizer state, communication buffers, and temporary workspaces can peak later. Lazy initialization can also make the first optimizer step materially different.
6.Length Outlier
Why can one unusually long batch OOM a run whose average batch fits?
Activation and attention-related costs depend on sequence shape, not only average tokens. Track maxima, padding, and packing; bound or bucket lengths under an explicit data policy.
7.Async Timing
Why can timing a CUDA operation with only CPU timestamps underestimate its duration?
Kernel launches are asynchronous. Use CUDA events or appropriate synchronization around the measured region, while avoiding synchronization that changes the workload being studied.
8.Async Error
Why can a CUDA error appear at an operation after the kernel that caused it?
Asynchronous execution can delay error observation until a synchronization point. Use targeted debugging and execution traces to locate the initiating failure.
9.Rank Failure
One distributed rank exits, while others appear stuck in an all-reduce. What should be checked before tuning NCCL?
The earliest failure across all ranks, including OOM, data-loader errors, and mismatched control flow. A collective wait may be a downstream symptom of a missing participant.
10.Collective Order
Why must participating ranks call matching collectives in compatible order?
Different collective types, shapes, or execution paths can hang or fail the job. Trace rank progress and validate tensor metadata at the divergence point.
11.Fabric Baseline
How can you separate a model-code regression from a GPU-network problem?
Run a controlled collective or fabric benchmark on the same nodes and message sizes, then compare topology and logs. A healthy single-GPU test does not validate multi-node communication.
12.Topology
Why can tensor parallelism slow down when the same GPU count is spread across different hosts?
Frequent communication may move from fast local links to slower or higher-latency network paths. Map the communication pattern to GPU and NIC topology before choosing placement.
13.Numa
How can CPU and NUMA placement affect a GPU job even when GPU hardware is unchanged?
Input processing and transfers may cross remote memory or unfavorable PCIe paths. Inspect affinity and topology, then verify with measured transfer and step timings.
14.Compatibility
What must be checked when a new container fails to initialize CUDA on an existing node?
Driver compatibility, runtime libraries, device visibility, permissions, and GPU architecture support. Pin and test the complete compatibility set rather than assuming the container contains a usable host driver.
15.Health Event
A GPU repeatedly reports hardware errors on unrelated jobs. What is an appropriate operational response?
Correlate device identity and diagnostic events, drain the affected capacity, and follow the supported investigation or replacement procedure. Repeated job restarts can hide a failing device.
16.Mig Choice
When can MIG help a fleet serving several small workloads?
On supported hardware, it partitions defined compute and memory resources into isolated instances. Choose profiles against workload requirements; a smaller instance cannot borrow the full GPU's memory just because it is idle.
17.Time Sharing
Why should GPU time-sharing not be assumed to provide the same isolation as hardware partitioning?
Workloads can contend for memory and execution resources, and isolation guarantees differ. Test interference and failure behavior under the chosen scheduler and runtime.
18.Gang Capacity
Why can a distributed job remain queued despite several individually free GPUs?
It may require a compatible group with specific topology, memory, and simultaneous availability. Schedule the collective resource requirement rather than counting free devices alone.
19.Checkpoint Recovery
What must a preempted distributed training job validate when restoring a checkpoint?
Completeness of all required shards, model and optimizer compatibility, scheduler and RNG state, and data progress. Restore through the framework's supported process and compare resumed behavior.
20.Useful Cost
Why is dollars per GPU-hour insufficient for comparing two operational configurations?
Different configurations deliver different useful tokens, training progress, latency, and failure rates. Compare cost per accepted outcome under a fixed workload and quality requirement.