Peer comparison across GPU ranks reveals stragglers that nvidia-smi hides. eBPF traces show 20-35% of cluster compute wasted on cohorts that finish iterations slower than their peers, with no single GPU looking unhealthy.
Claude Code with MCP access to kernel-level GPU traces, pointed at a real performance anomaly. The full investigation session: what the agent called, what it found, and where it got things right and wrong.
A single straggling node held up a 4-node distributed training job. We found it by fanning out one eBPF-powered SQL query to all four nodes and getting the answer in under a second. No central service, no Prometheus — just the same single-binary agent on each machine.
CUDA graphs collapse hundreds of kernel launches into one opaque cudaGraphLaunch call, creating a blind spot for GPU observability. We traced graph lifecycle events with eBPF uprobes and found pool exhaustion, re-capture storms, and CPU contention hiding in plain sight.
PyTorch DataLoader with 8 workers: 124x slower than direct tensor indexing. Kernel-level tracing of the full execution path revealed CPU over-subscription, context switching storms, and CUDA kernel launch latency spikes up to 356x above baseline.
97% GPU utilization in nvidia-smi, but training throughput was a fraction of what benchmarks promised. CUDA API tracing and kernel scheduling data showed what the GPU was actually doing during that 97%.
CUDA OOM at 60% GPU utilization. nvidia-smi showed plenty of free memory, but PyTorch kept crashing. eBPF tracing of every cudaMalloc and cudaFree call exposed the real cause: memory fragmentation from misaligned allocation patterns.
