Diagnoses performance issues on HyperPod clusters related to uneven NCCL bandwidth and poor filesystem throughput.
- GitHub stars
- 938
- Files
- 3
- Last commit
- Nov 10, 2026
- License
- Apache-2.0
- Allowed tools
- —
What does the hyperpod-performance-debugger skill do?
hyperpod-performance-debugger diagnoses performance issues on HyperPod clusters related to uneven NCCL bandwidth and poor filesystem throughput.
Summary generated from the skill's SKILL.md.
About the hyperpod-performance-debugger skill
Diagnosing performance bottlenecks on Amazon SageMaker HyperPod clusters is what this skill enables for machine learning engineers and infrastructure teams. It focuses specifically on two critical issues: uneven NCCL bandwidth across nodes and poor filesystem throughput that stalls training. By running the read-only scripts/perf-snapshot.sh command, the tool gathers host-side signals including GPU thermal readings, NVLink lane states, EFA port status, and FSx CloudWatch utilization metrics.
When an agent loads this skill, it acts as a diagnostic router. It identifies whether a problem is within its scope or requires a specialized sibling skill. For example, it directs hardware faults like ECC errors or Xid lines to the hyperpod-node-debugger, while NCCL timeouts or RDMA issues are routed to hyperpod-nccl. It provides the necessary data to determine if a workload is suffering from cross-AZ latency, topology mismatches, or saturated filesystem IOPS, helping users decide whether to adjust provisioning or investigate network layers.
What an agent can do with it
- 1Identify NCCL bandwidth varianceDetermine if specific node sets are causing training slowdowns due to uneven pairwise bandwidth.
- 2Diagnose filesystem bottlenecksInvestigate why training is stalled during data loading or why checkpointing is dominating step time.
Written from the skill's SKILL.md. Every use case is checked against a quote from it.
How to install the hyperpod-performance-debugger skill
Each command copies the skill folder into the client's skills directory. Restart the client or start a new session to load it.Install the hyperpod-performance-debugger skill in Plugin
/plugin marketplace add awslabs/agent-plugins /plugin install sagemaker-ai@agent-plugins-for-aws
Install the hyperpod-performance-debugger skill in Claude Code
~/.claude/skills/hyperpod-performance-debugger/SKILL.md
$ npx degit awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-performance-debugger ~/.claude/skills/hyperpod-performance-debuggerInstall the hyperpod-performance-debugger skill in Claude Code (project)
.claude/skills/hyperpod-performance-debugger/SKILL.md
$ npx degit awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-performance-debugger .claude/skills/hyperpod-performance-debuggerInstall the hyperpod-performance-debugger skill in Codex
~/.agents/skills/hyperpod-performance-debugger/SKILL.md
$ npx degit awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-performance-debugger ~/.agents/skills/hyperpod-performance-debuggerInstall the hyperpod-performance-debugger skill in Gemini CLI
~/.gemini/skills/hyperpod-performance-debugger/SKILL.md
$ npx degit awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-performance-debugger ~/.gemini/skills/hyperpod-performance-debuggerInstall the hyperpod-performance-debugger skill in Claude.ai
npx degit awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-performance-debugger hyperpod-performance-debugger && zip -r hyperpod-performance-debugger.zip hyperpod-performance-debugger # claude.ai → Customize → Skills → + → Create skill → Upload a skill → hyperpod-performance-debugger.zip
Automated checks
Ran Nov 15, 2026Full report →Passed our automated checks. That is not a security audit: read the scripts before you let an agent run them. How checks work
- Source repository linked and reachableawslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-performance-debugger is public and reachablegithub.com/awslabs/agent-plugins/tree/HEAD/plugins/sagemaker ↗Pass
- Open source license (OSI)Apache-2.0spdx.org/licenses/Apache-2.0.html ↗Pass
- Commit or release in the last yearLast commit 2026-11-10github.com/awslabs/agent-plugins/commits ↗Pass
- No download-and-run or credential theft in scripts1 script read; no download-and-run, remote eval, base64 execution or credential uploadPass
- No hidden instructions in SKILL.mdNo override phrases, safeguard switches, credential requests or hidden charactersgithub.com/awslabs/agent-plugins/blob/HEAD/plugins/sagemaker ↗Pass
- Pre-approved tools are scopedNo pre-approved tools; the agent asks before using onegithub.com/awslabs/agent-plugins/blob/HEAD/plugins/sagemaker ↗Pass
- SKILL.md follows the Agent Skills formatValid name and description; says when to use it; 186 linesgithub.com/awslabs/agent-plugins/blob/HEAD/plugins/sagemaker ↗Pass
SKILL.md
View on GitHubHyperPod Performance Debugger
- Uneven NCCL performance across nodes — workload faster on some node sets than others, pairwise bandwidth variance, suspected straggler.
- Poor filesystem performance — training stalled on data loading, checkpoint save/load dominating step time, FSx throughput saturated.
Scope and delegation
Route findings outside the two in-scope scenarios to the owner skill below.
| Concern observed | Route to |
|---|---|
| GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain/replace | hyperpod-node-debugger (§ F Hardware/Auto-Repair, § G GPU) |
Cannot allocate memory at os.fork(), root volume exhausted |
hyperpod-node-debugger (§ I Resource Exhaustion) |
| NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlock | hyperpod-nccl |
| EFA / NCCL / CUDA / NVIDIA driver version drift across nodes | hyperpod-version-checker |
| EFA self-referencing security-group rule missing — single node | hyperpod-node-debugger § A (EFA / Security Group) |
| EFA self-referencing security-group rule missing — cluster-wide | hyperpod-cluster-debugger § A (EFA Health Checks) |
| Slurm node state changes (drain / resume / reboot) | hyperpod-slurm-debugger |
| Diagnostic bundle for AWS Support | hyperpod-issue-report |
| Shell access on a node | hyperpod-ssm |
Operating policy
- Read-only. Print commands the customer runs; do not execute commands that modify state.
- Container vs host version comparisons go through
hyperpod-version-checker. - Xid lines, ECC counts, NVLink lane state, and thermal readings get surfaced; the catalog and verdict live in
hyperpod-node-debugger§ G.
Workflow
- Confirm the symptom is uneven NCCL or poor filesystem performance. If neither, route to the matching sibling skill above.
- Run
scripts/perf-snapshot.sh(read-only) to gather host-side signals for the suspect node and FSx filesystems mounted on it. - For each
[CONCERN]line in the script output, open the matching section below and read the supporting reference. - After the per-incident diagnosis, recommend the HyperPod platform health features in § Continuous health coverage so the customer gets ongoing protection.
Step 1: Run the snapshot
bash scripts/perf-snapshot.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>
# Scope to one suspect node:
bash scripts/perf-snapshot.sh --cluster <C> --region <R> --node <INSTANCE_ID>
The script samples one node by default. It collects host-side data via hyperpod-ssm: nvidia-smi output (temperature, SM clocks, PCIe link width, ECC, NVLink, topo -m), recent dmesg Xid lines, EFA port state and fi_info provider visibility, EFA installer + kernel module versions, CPU governor, NVL72 Fabric Manager state, FSx CloudWatch utilization, df -h / lfs df -h per mount, host iowait, /dev/shm size, and root-volume usage. All read-only.
Tags: [OK] healthy · [CONCERN] signal worth investigating (carries a → pointer to the owner skill) · [INFO] informational.
Host vs container scope. The script runs on the host via SSM and reports host-scope values. Many setups ship the EFA / libfabric / OFI-NCCL / CUDA stack inside the training container by design — a host value of unknown is not by itself a defect. What matters for performance is the stack the workload actually uses. Verify versions inside the container (and across nodes) via hyperpod-version-checker before drawing conclusions.
Step 2: Match signal → section
| Observation | Section |
|---|---|
| Pairwise NCCL bandwidth varies across node pairs / suspected straggler | A: Uneven NCCL Performance |
| Nodes spread across AZs / network-node-layer labels / UltraServer boundaries | A |
| EFA port not ACTIVE on a node, missing OFI plugin, or FI provider not visible | A + route to hyperpod-node-debugger § A; hyperpod-version-checker for cross-node version compare |
iostat shows high iowait, FSx CloudWatch utilization sustained near 100% |
B: Poor Filesystem Performance |
| DataLoader stalls, checkpoint dominates step time | B |
Xid line in dmesg, uncorrectable ECC, inactive NVLink lane, GPU ≥ 88°C |
Route to hyperpod-node-debugger § G |
| Container vs host version drift suspected | Route to hyperpod-version-checker |
Cannot allocate memory at os.fork(), root volume full, OOM events |
Route to hyperpod-node-debugger § I |
NCCL timeout, hang, TCP fallback (NET/OFI Using TCP), RDMA memlock |
Route to hyperpod-nccl |
A: Uneven NCCL Performance
The customer reports identical training jobs running with different step times on different node sets, pairwise bandwidth variance, or some allocations consistently slower than others despite identical code.
Per the official troubleshooting guide, the common contributing factors are network topology differences between nodes (cross-AZ, cross-rack, cross-UltraServer), degraded EFA performance on some nodes, mixed instance types or generations within an instance group, and CPU frequency scaling differences.
Diagnostic pass (read-only)
The host-side data points — GPU thermal/ECC/PCIe/clocks, Xid, NVLink lanes, EFA port state and provider visibility, CPU governor, EFA/OFI/driver versions, nvidia-smi topo -m — are all collected by scripts/perf-snapshot.sh (Step 1 above). The script tags [CONCERN] with thresholds and emits routing pointers; rerun it per suspect node via --node <INSTANCE_ID>.
For driver / CUDA / NCCL / EFA / OFI version drift across nodes, run hyperpod-version-checker skill.
Pairwise NCCL bandwidth test
Run the standard nccl-tests recipes from awslabs/awsome-distributed-training. For an N-node cluster, run all-reduce across every pair and record busbw for each pair. Pairs more than ~5% below the run mean (the threshold the AWS validation script flags) are problematic candidates.
Expected busbw per SKU is published in the AI-on-HyperPod NCCL test guide. Benchmark the specific instance type before relying on a number.
Pairwise scripts, HyperPod topology surfaces (HyperPod API, EKS labels, Slurm topology.conf), and GB200 NVL72 specifics are in references/perf-details.md § Uneven NCCL.
Topology verification
HyperPod exposes topology through three operator-visible surfaces:
- HyperPod API:
aws sagemaker describe-cluster-nodereturnsNodeDetails.Placement.AvailabilityZone/AvailabilityZoneIdandNodeDetails.UltraServerInfo.Id(UltraServer SKUs only). - EKS labels:
topology.kubernetes.io/zone,topology.k8s.aws/network-node-layer-{1,2,3}(highest-numbered = closest to instance),topology.k8s.aws/ultraserver-id. - Slurm: HyperPod auto-generates
topology.conf. Inspect viascontrol show topology.
Tightly coupled work shares the same AZ, the same highest-numbered network-node-layer label (EKS) or the same Slurm topology block, and — for NVL72 jobs — the same UltraServerInfo.Id / topology.k8s.aws/ultraserver-id. If the cluster is spread across AZs or layers, topology must be re-established at provisioning time. Route provisioning changes to hyperpod-cluster-debugger § B (Capacity & AZ).
B: Poor Filesystem Performance
The customer reports training bottlenecked on data loading, checkpoint save/load dominating step time, executables/scripts loading slowly, or iowait high.
Per the official troubleshooting guide, the resolution path follows this order:
- Check CloudWatch metrics on the filesystem.
- Check the provisioned performance configuration against workload requirements.
- Investigate which operations are causing the I/O — workload demand vs inefficient pattern.
- Consider upgrading provisioned performance.
- Choose the filesystem type that matches the I/O pattern.
This skill covers steps 1–3. Steps 4–5 are customer decisions; surface the data and let the customer pick.
Diagnostic pass (read-only)
scripts/perf-snapshot.sh (Step 1 above) covers the on-node side of this pass: it discovers FSx mounts, calls aws cloudwatch get-metric-statistics on DataReadBytes and (for OpenZFS) FileServerDiskIopsUtilization, prints df -h for /fsx /opt/dlami/nvme /opt/sagemaker, runs lfs df -h per Lustre mount, and reports iostat iowait. It tags [CONCERN] when OpenZFS IOPS utilization sustains ≥ 80% or iowait > 20%.
For longer windows or additional metrics (DataWriteBytes, Lustre DiskIopsUtilization, OpenZFS FileServerDiskThroughputUtilization), drive the query directly:
aws cloudwatch get-metric-statistics --region <REGION> \
--namespace AWS/FSx --metric-name DataReadBytes \
--dimensions Name=FileSystemId,Value=<FSID> \
--start-time "$(date -u -d '3 hours ago' +%Y-%m-%dT%H:%M:%S)" \
--end-time "$(date -u +%Y-%m-%dT%H:%M:%S)" \
--period 60 --statistics Sum Maximum
The full per-filesystem-type metric catalog is in references/perf-details.md § Filesystem.
Branches
Provisioned capacity is saturated. CloudWatch utilization sustained near 100% across the workload window. Customer decision: scale up the filesystem.
- FSx for Lustre throughput scales with
StorageCapacity × PerUnitStorageThroughput; capacity changes are non-disruptive. - FSx for OpenZFS — increase provisioned IOPS or throughput.
I/O pattern is inefficient. CloudWatch shows headroom but the workload is still I/O-bound. Customer decision: change the application.
- DataLoader: raise
num_workers, setpin_memory=True,persistent_workers=True. - Checkpointing: use async + sharded (
torch.distributed.checkpoint.async_saveplus FSDPSHARDED_STATE_DICT).FULL_STATE_DICTserializes through rank 0 and is a frequent root cause. - Small-file workloads: Lustre is optimized for large sequential I/O. For millions of small files, use WebDataset / tar shards, FSx for OpenZFS, or NVMe scratch.
Filesystem-selection guidance and the async-checkpoint pattern are in references/perf-details.md § Filesystem.
Continuous health coverage
Once the immediate incident is diagnosed, recommend HyperPod's built-in health features so problems are caught before the next training run rather than after another customer-reported regression.
-
Enable
NodeRecovery=Automaticon the cluster. The Health Monitoring Agent (HMA) continuously monitors GPU- and Trainium-based instances and marks instances unhealthy on detected failure. With auto-recovery enabled, HyperPod reboots or replaces the node — no operator intervention. -
Enable
OnStartDeepHealthCheckson every GPU instance group with both check categories:InstanceStress—stress-ngon CPU/memory/disk, GPU and PCI device count verification, DCGM level-4 diagnostics (memory test included), and EFA loopback bandwidth/latency.InstanceConnectivity— multi-node NCCL all-reduce.
Every newly provisioned or auto-replaced node passes the same hardware bar before accepting jobs.
-
Run on-demand deep health checks when this skill or any sibling surfaces a hardware concern but the cluster is mid-workload.
aws sagemaker start-cluster-health-checkruns the same checks against a specific instance group; nodes are placed in a Slurm maintenance reservation and the check is queued until any running job completes (not preempted). Console: HyperPod → Clusters → Instances → Run deep health checks.Not supported when
NodeProvisioningMode=Continuous; one on-demand request per cluster at a time. Requires the latest AMI — runUpdateClusterSoftwarefirst.
Logs land in CloudWatch at /aws/sagemaker/Clusters/<cluster_name>/<cluster_id> under DeepHealthCheckResults/<log_stream_id>, and on each node at /var/log/aws/clusters/sagemaker-deep-health-check.log.
References
- references/perf-details.md — pairwise NCCL test recipes, HyperPod topology check, GB200 NVL72 placement; CloudWatch metric catalog per filesystem type, async-checkpoint pattern, filesystem selection guide.
External:
- Amazon SageMaker HyperPod troubleshooting guide (official): https://github.com/aws/sagemaker-hyperpod-cluster-setup/blob/troubleshooting-doc-20250917/troubleshoot/index.md
- AI-on-HyperPod NCCL performance tests (expected
busbwper SKU): https://awslabs.github.io/ai-on-sagemaker-hyperpod/docs/slurm-orchestration/validation-and-testing/performance-testing/nccl-tests - Amazon SageMaker HyperPod resiliency (NodeRecovery, HMA, auto-resume): https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency.html
- Amazon SageMaker HyperPod deep health checks: https://docs.aws.amazon.com/sagemaker/latest/dg/sagemaker-hyperpod-resiliency-slurm-deep-health-checks.html
- StartClusterHealthCheck API: https://docs.aws.amazon.com/sagemaker/latest/APIReference/API_StartClusterHealthCheck.html
- Amazon EC2 instance topology / network-node-layer labels: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/how-ec2-instance-topology-works.html
- Amazon FSx for Lustre performance: https://docs.aws.amazon.com/fsx/latest/LustreGuide/performance.html
- Amazon FSx for OpenZFS metrics: https://docs.aws.amazon.com/fsx/latest/OpenZFSGuide/fsx-openzfs-metrics.html
- AWS Elastic Fabric Adapter and NCCL: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/efa-start-nccl.html
- awslabs/awsome-distributed-training NCCL tests: https://github.com/awslabs/awsome-distributed-training/tree/main/micro-benchmarks/nccl-tests
Files in this skill
3hyperpod-performance-debugger skill questions
How do I install the hyperpod-performance-debugger skill in Claude Code?
Run npx degit awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-performance-debugger ~/.claude/skills/hyperpod-performance-debugger. Claude Code loads skills from ~/.claude/skills (all projects) or .claude/skills (one project) and uses a skill when its description matches the task.
Which clients can use the hyperpod-performance-debugger skill?
Any client that reads the Agent Skills format: Claude Code, Codex, Gemini CLI and claude.ai (upload as a ZIP). Scripts in the skill run only where the client allows code execution.
What specific performance issues can this skill diagnose?
It focuses on uneven NCCL bandwidth across nodes and poor filesystem performance, such as training stalls during data loading or checkpointing.
Is this skill capable of modifying cluster settings?
No, the skill is read-only and only prints commands for the customer to run rather than executing state-modifying commands.
How can I verify if there is a version mismatch between the host and the container?
You should use the hyperpod-version-checker skill to compare versions between the container and the host.