llm-serving

Diagram contrasting host-level DCGM metrics (per-GPU utilization, memory, power, temperature) with kernel-side eBPF signals (per-rank libnccl collective timestamps, kernel-launch overhead split, cgroup-attributed PCIe transfer cost, per-rank inter-node TCP retransmits) for multi-GPU inference observability

eBPF, GPU Debugging, GPU Observability, MLOps

What Inference-Platform Benchmark Posts Leave Out

Ingero Team / May 13, 2026

Inference-platform writeups optimize for p90 TTFT graphs. The dimensions that matter operationally – tail variance past p90, per-rank skew on multi-GPU, per-tenant attribution – are usually absent. Here’s why, and what eBPF on the host adds.

Abstract illustration of vLLM prefix-cache head-of-line blocking — 11-second time-to-first-token gradient bar with stalled short requests stuck behind heavy prefix-caching requests; GPU with idle SMs on one side, active SMs on the other

eBPF, GPU Debugging, GPU Observability, MLOps

11-Second Time to First Token on a Healthy vLLM Server

Ingero Team / April 21, 2026

A healthy vLLM server, normal nvidia-smi output, 11 seconds to first token. eBPF uprobes on the CUDA driver and kernel tracepoints on the scheduler traced it to prefix caching head-of-line blocking.