# Ingero

> Ingero is open-source software (Apache 2.0 user-space + GPL-2.0/BSD-3 eBPF kernel-space) that observes GPU workloads at the Linux kernel level via eBPF, with zero code changes and under 2% overhead. It traces CUDA Runtime + Driver + Graph API calls, kernel scheduler events, and memory pressure, then assembles causal chains that explain root causes for GPU stalls, OOMs, stragglers, and latency spikes. Used by ML and AI infrastructure teams running PyTorch, vLLM, and distributed training workloads on multi-GPU clusters.

## Core repositories and quickstarts

- [github.com/ingero-io/ingero](https://github.com/ingero-io/ingero): single-node agent. Statically linked Go binary, embedded SQLite, no external dependencies. Linux 5.15+, x86_64 + arm64. Apache 2.0 + GPL-2.0/BSD-3.
- [github.com/ingero-io/ingero-fleet](https://github.com/ingero-io/ingero-fleet): multi-node coordinator. OpenTelemetry Collector distribution with peer-relative straggler detection via Median Absolute Deviation. Apache 2.0.
- [Quick start (single-node)](https://github.com/ingero-io/ingero#try-it-in-60-seconds): install, trace, and explain in about 60 seconds.
- [Quick start (Kubernetes / Helm)](https://github.com/ingero-io/ingero-fleet/blob/main/docs/quickstart-k8s_fleet.md): 3-node cluster, straggler detected in ~20 minutes.
- [Quick start (bare-metal)](https://github.com/ingero-io/ingero-fleet/blob/main/docs/quickstart-binary_fleet.md): non-Kubernetes deployment.
- [Quick start (Docker)](https://github.com/ingero-io/ingero-fleet/blob/main/docs/quickstart-docker_fleet.md): single-host validation.
- [Zero-hardware demo (Codespaces)](https://github.com/ingero-io/ingero-fleet/tree/main/examples/cloud-demo): dashboards + Echo API in the browser, no GPU.
- [MCP server](https://github.com/ingero-io/ingero/tree/main/examples/integrations/mcp): the agent's `ingero mcp` stdio server, so Claude, Cursor, or any MCP-compatible LLM can query thousands of captured kernel events in natural language.

## What Ingero traces

- CUDA Runtime API: cudaMalloc, cudaFree, cudaLaunchKernel, cudaMemcpy, cudaMemcpyAsync, cudaStreamSync, cudaDeviceSynchronize.
- CUDA Driver API: cuLaunchKernel, cuMemcpy, cuMemcpyAsync, cuCtxSynchronize, cuMemAlloc. Catches kernel launches that bypass the runtime API, e.g. from cuBLAS and cuDNN.
- CUDA Graph lifecycle: cudaStreamBeginCapture, cudaStreamEndCapture, cudaGraphInstantiate, cudaGraphLaunch. Useful for torch.compile and vLLM workloads.
- Linux kernel tracepoints: sched_switch, sched_wakeup, mm_page_alloc, oom_kill, sched_process_exec/exit/fork.
- System context: CPU utilization, memory usage, load average, swap. Read from /proc, no eBPF or root needed for this layer.

## Investigation walkthroughs (blog)

- [CUDA Out of Memory at 60% Utilization: tracing PyTorch GPU memory fragmentation](https://ingero.io/gpu-problem-1-why-your-pytorch-training-runs-out-of-gpu-memory-and-how-to-actually-debug-it/): why nvidia-smi shows free memory but cudaMalloc still fails.
- [nvidia-smi reports 97% utilization while the GPU sits idle](https://ingero.io/your-gpu-is-97-utilized-but-your-training-is-3x-slower-than-expected/): kernel-level idle time hidden behind utilization metrics.
- [11-second time to first token on a healthy vLLM server](https://ingero.io/11-second-time-to-first-token-healthy-vllm-server/): prefix-cache head-of-line blocking found by tracing 10,869 CUDA events.
- [GPU incident response in 60 seconds with eBPF](https://ingero.io/gpu-incident-response-in-60-seconds-an-sres-guide-to-ebpf-based-gpu-observability/): SRE playbook for kernel-level GPU incident triage.
- [Production GPU training is 34% slower. Show me why.](https://ingero.io/gpu-stragglers-cluster-compute-waste/): straggler problem statement with FALCON, Llama 3, ByteRobust data.
- [Tracing torch.cuda.empty_cache on an RTX 4090](https://ingero.io/tracing-torch-cuda-empty-cache-rtx-4090/): where the 53 MB go when you call empty_cache.
- [124x slower: what PyTorch DataLoader actually does at the kernel level](https://ingero.io/124x-slower-pytorch-dataloader-kernel-level/): DataLoader latency traced to scheduler preemption.
- [Tracing a 13x PyTorch slowdown to a hidden NumPy synchronization](https://ingero.io/tracing-13x-pytorch-slowdown-hidden-numpy-synchronization/): cross-library latency root cause.
- [Catching a vLLM latency spike with eBPF and an open-weight LLM](https://ingero.io/debugging-vllm-latency-minimax-ollama-mcp/): MiniMax + Ollama investigation.
- [One query, four GPUs: tracing a distributed training stall across nodes](https://ingero.io/distributed-gpu-training-debugging-ebpf-fleet/): Fleet's fan-out query model.
- [MCP as observability interface: connecting AI agents to kernel tracepoints](https://ingero.io/mcp-observability-interface-ai-agents-kernel-tracepoints/): MCP architecture and 4-tool-call investigation example.
- [What happens when an AI agent gets kernel-level GPU traces](https://ingero.io/what-happens-when-an-ai-agent-gets-kernel-level-gpu-traces/): Claude diagnoses a PyTorch DataLoader bottleneck without shell access.
- [I gave Claude 10,869 GPU kernel events: root cause in 47 seconds](https://ingero.io/ebpf-trace-cuda-mcp-queryable/): full vLLM #37343 investigation walkthrough.
- [CUDA Graphs: the 8-year overnight success and the observability gap](https://ingero.io/cuda-graphs-observability-gap/): CUDA graph lifecycle visibility.
- [26 seconds to find a straggler: Fleet v0.10 end-to-end on A100 and GH200](https://ingero.io/fleet-v0-10-end-to-end-a100-gh200-straggler-detection/): Fleet v0.10 release announcement and Lambda Cloud reproduction kit.

## Common use cases

- Diagnose CUDA out-of-memory at sub-100% utilization (memory fragmentation).
- Map nvidia-smi 90%+ utilization to actual kernel-level idle time.
- Trace PyTorch DataLoader stalls to scheduler preemption or NumPy synchronization.
- Detect stragglers in multi-node distributed training via peer-relative threshold.
- Build AI agents that read GPU traces in natural language (MCP integration).
- Capture CUDA Graph lifecycle events for torch.compile and vLLM debugging.
- Correlate host-side events (CPU contention, OOM, swap) with GPU latency spikes.

## Project metadata

- License: Apache 2.0 (user-space components) + GPL-2.0/BSD-3 (eBPF kernel-space components).
- Current: agent v0.17.x, Fleet v1.x, Grafana plugin v1.x (2026). Multi-node Fleet collector, arm64 (GH200), external annotation ingest, a signed Grafana app plugin with bundled dashboards, and framework integrations (vLLM, HuggingFace Trainer, PyTorch Lightning, Ray, DeepSpeed, Accelerate).
- Maintainer: Ingero Team.
- Marketing site and blog: https://ingero.io/
- Twitter / X: https://x.com/ingero_io
- Hashnode: https://ingero.hashnode.dev/
- Dev.to: https://dev.to/ingero
- Medium: https://medium.com/@ingero
- Featured in: awesome-ebpf, awesome-observability, awesome-opentelemetry, Awesome-GPU, awesome-gpu-engineering, awesome-cloud-native, awesome-profiling, awesome-sre-tools, awesome-mcp-servers, awesome-devops-mcp-servers, MCP Registry, Glama.

## Optional

- [Lambda E2E reproduction kit](https://github.com/ingero-io/ingero-fleet/tree/main/examples/lambda-e2e): scripted Fleet v0.10 demo on Lambda Cloud A100 + GH200 (under $11 burn for both clusters, 1 hour each).
- [Grafana app plugin](https://github.com/ingero-io/ingero-grafana-app): bundled GPU dashboards + a native Echo datasource; also published as standalone dashboards on grafana.com/orgs/ingero.
- [Live public dashboard](https://ingero.grafana.net/public-dashboards/11d240020d394fa382c4b9facb9fde69): A100 and GH200 runs from the v0.10 release, both clusters overlaid.