hyperpod-node-debugger skill
✓ 공식scriptsEKS 또는 Slurm 기반 HyperPod 클러스터에서 GPU 하드웨어, EFA, 네트워킹 및 리소스 고갈을 포함한 노드별 문제를 진단하고 해결합니다.
- GitHub 스타
- 938
- 파일
- 7
- 마지막 커밋
- 2026년 11월 10일
- 라이선스
- Apache-2.0
- 허용된 도구
- —
hyperpod-node-debugger skill은 무엇을 하나요?
hyperpod-node-debugger은(는) agent skill입니다. EKS 또는 Slurm 기반 HyperPod 클러스터에서 GPU 하드웨어, EFA, 네트워킹 및 리소스 고갈을 포함한 노드별 문제를 진단하고 해결합니다.
skill의 SKILL.md에서 생성된 요약입니다.
hyperpod-node-debugger skill 정보
HyperPod 클러스터를 관리하는 엔지니어는 이 도구를 사용하여 EKS 또는 Slurm 환경에서 개별 노드에 영향을 미치는 문제를 식별하고 해결할 수 있습니다. 이 SKILL은 특정 인스턴스가 비정상적이거나 응답이 없거나 멈춘 원인을 진단하는 데 특화되어 있으며, XID 또는 ECC 문제와 같은 GPU 하드웨어 오류부터 EFA 및 VPC 설정을 포함한 네트워킹 문제까지 폭넓은 기술적 결함을 다룹니다. 이 SKILL은 triage-cluster.sh, check-efa-sg.sh, check-vpc-config.sh와 같은 스크립트를 사용하여 클러스터와 구성 요소의 상태를 점검하는 읽기 전용 모드로 작동합니다.
에이전트가 결함을 식별하면 근본 원인을 제공하고 조사, 재부팅, 교체라는 엄격한 조사 순서에 따라 해결을 위한 특정 명령을 제안합니다. NVLink 오류, 커널 패닉, 디스크 압박, 컨테이너 런타임 충돌 등 복잡한 하드웨어 및 소프트웨어 신호를 처리하도록 설계되었습니다. 사용자는 이 도구가 클러스터 전체의 프로비저닝이나 NCCL 또는 MFU 메트릭 관리용이 아닌, 노드별 트러블슈팅을 위한 용도임을 유의해야 합니다.
agent가 수행할 수 있는 작업
- 1GPU 하드웨어 결함 진단하드웨어 신호를 사용하여 노드에서 수정 불가능한 ECC 오류 또는 NVLink 문제가 발생하는지 식별합니다.
- 2EFA 네트워킹 문제 트러블슈팅보안 그룹 규칙이 EFA 상태 확인을 방해하고 있는지 확인합니다.
skill의 SKILL.md를 기반으로 작성되었습니다. 모든 사용 사례는 해당 문서의 인용구와 대조 확인되었습니다.
hyperpod-node-debugger skill 설치 방법
각 명령은 skill 폴더를 클라이언트의 skills 디렉터리에 복사합니다. 클라이언트를 재시작하거나 새 세션을 시작하여 로드하세요.플러그인에 hyperpod-node-debugger skill 설치
/plugin marketplace add awslabs/agent-plugins /plugin install sagemaker-ai@agent-plugins-for-aws
Claude Code에 hyperpod-node-debugger skill 설치
~/.claude/skills/hyperpod-node-debugger/SKILL.md
$ npx degit awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-node-debugger ~/.claude/skills/hyperpod-node-debuggerClaude Code (project)에 hyperpod-node-debugger skill 설치
.claude/skills/hyperpod-node-debugger/SKILL.md
$ npx degit awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-node-debugger .claude/skills/hyperpod-node-debuggerCodex에 hyperpod-node-debugger skill 설치
~/.agents/skills/hyperpod-node-debugger/SKILL.md
$ npx degit awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-node-debugger ~/.agents/skills/hyperpod-node-debuggerGemini CLI에 hyperpod-node-debugger skill 설치
~/.gemini/skills/hyperpod-node-debugger/SKILL.md
$ npx degit awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-node-debugger ~/.gemini/skills/hyperpod-node-debuggerClaude.ai에 hyperpod-node-debugger skill 설치
npx degit awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-node-debugger hyperpod-node-debugger && zip -r hyperpod-node-debugger.zip hyperpod-node-debugger # claude.ai → Customize → Skills → + → Create skill → Upload a skill → hyperpod-node-debugger.zip
자동 검사
2026년 11월 15일에 실행됨전체 보고서 →아직 검증되지 않았습니다. 스킬은 점수 80, 심각한 발견 없음, 검사에 충분한 데이터가 필요합니다. 검사 방식 · 점수를 높이기 위해 수정할 항목
- 소스 저장소 연결 및 접근 가능awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-node-debugger is public and reachablegithub.com/awslabs/agent-plugins/tree/HEAD/plugins/sagemaker ↗통과
- 오픈 소스 라이선스 (OSI)Apache-2.0spdx.org/licenses/Apache-2.0.html ↗통과
- 최근 1년 내 커밋 또는 릴리스Last commit 2026-11-10github.com/awslabs/agent-plugins/commits ↗통과
- 스크립트에 다운로드 후 실행이나 자격 증명 탈취가 없음scripts/triage-cluster.sh line 164 decodes base64 and executes itgithub.com/awslabs/agent-plugins/blob/HEAD/plugins/sagemaker ↗결함 발견
- SKILL.md에 숨겨진 지침 없음No override phrases, safeguard switches, credential requests or hidden charactersgithub.com/awslabs/agent-plugins/blob/HEAD/plugins/sagemaker ↗통과
- 사전 허용 도구가 제한됨No pre-approved tools; the agent asks before using onegithub.com/awslabs/agent-plugins/blob/HEAD/plugins/sagemaker ↗통과
- SKILL.md가 Agent Skills 형식을 따름the description does not say when to use the skillgithub.com/awslabs/agent-plugins/blob/HEAD/plugins/sagemaker ↗결함 발견
SKILL.md
GitHub에서 보기HyperPod Node Debugger
Operating policy. Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a Suggested command (run this yourself) block and wait for the customer. Destructive order: investigate → reboot → replace (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state, logs, or caches on speculation.
IaC note (always include with mutation commands). When you suggest any command that changes cluster, VPC, SG, subnet, or EKS configuration (e.g. authorize-security-group-*, modify-vpc-attribute, update-cluster, kubectl label/cordon/drain, create namespace, set env daemonset), ask the customer first whether the cluster / VPC / SG is managed by Infrastructure-as-Code (CloudFormation, CDK, Terraform, Pulumi). If yes, tell them: "Apply this change in your IaC source first, then deploy through the pipeline — running the command directly will drift from your template and the next stack update may overwrite it." If they need to fix the issue immediately and the IaC change will follow, flag the drift explicitly so they remember to reconcile.
Read-only triage. scripts/triage-cluster.sh (and helpers check-efa-sg.sh, check-node-reachability.sh, check-vpc-config.sh) read state and print each issue as [FAIL] ... → references/node-diagnostics-detail.md § <section>. Catalog of customer-ticket patterns: references/node-issue-catalog.md.
Workflow
- Collect cluster name, region, suspect instance ID, exact error string from logs.
- Run
scripts/triage-cluster.sh(add--node <INSTANCE-ID>to focus one node). - For every
[FAIL]/ issue entry,Readthe referenced section. - Present: what script detected (copy the line verbatim), root cause, exact command(s) with instance/SG IDs filled in, blast radius (e.g. "reboots i-xxx", "wipes volumes on replacement"). For any command that mutates cluster/VPC/SG/EKS state, ask whether the affected resource is IaC-managed and surface the drift warning from the operating-policy note above.
- Wait for explicit customer approval. Destructive order: investigate → reboot → replace.
- Re-run triage to confirm. Iterate if not cleared.
Step 1: Triage
bash scripts/triage-cluster.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION>
# Focus on one node:
bash scripts/triage-cluster.sh --cluster <CLUSTER_NAME_OR_ARN> --region <REGION> --node <INSTANCE_ID>
One pass collects: cluster status + NodeRecovery, events, per-node health (HyperPod + EKS labels, Slurm states), VPC/SG snapshot, CloudWatch availability, SSM readiness, on-node resource checks (disk, memory, /dev/shm, OOM, NVMe, time sync, SSM agent), Slurm node→instance mapping.
Tags: [PASS] passed · [FAIL] issue with a → references/... pointer · [WARN] advisory · [INFO] informational. Priorities: P0 blocks operation · P1 degraded · P2 informational.
Step 2: Match signal → section
Events (list-cluster-events) — provisioning-time:
| Event | Section |
|---|---|
"EFA health checks did not run successfully" (public-doc verbatim signal) |
A: EFA/SG |
| Instance bootstrap or network-misconfiguration event | A + B: VPC |
| Lifecycle-script failure or timeout | D: Lifecycle |
| Insufficient-capacity or AZ-mismatch failure at creation | C: Capacity |
Hardware failure / UnschedulablePendingReplacement |
F: Hardware |
EKS labels:
| Label | Section |
|---|---|
node-health-status: UnschedulablePendingReplacement |
F |
node-health-status: UnschedulablePendingReboot |
F |
deep-health-check-status: Failed |
G → F |
Symptoms:
| Symptom | Section |
|---|---|
| Training hangs at NCCL init / AllReduce | A → E |
Slurm node down / "Node unexpectedly rebooted" |
H: Slurm |
| Jobs stuck PENDING / COMPLETING | H |
| Auto-repair not triggering | F |
| GPU not visible / XID / ECC errors | G |
| GPU row-remap pending/failed / silent NaNs / DCGM Fail | G § G.1.a/b |
Disk full / OOM / "Cannot allocate memory" |
I: Resources |
| Wrong vCPU count (e.g. 96 instead of 192 on p5.48xlarge) | J: Config |
| Container CrashLoopBackOff / runtime crash | M: Container Runtime |
aws-node CrashLoopBackOff / gRPC 50051 refused |
O: CNI / Pod Networking |
| Pods stuck Pending with no IP / CNI error | O |
DNS resolution / enableDnsSupport |
B § B.2 |
| Public subnet / IGW misconfigured | B § B.3 |
| Missing VPC endpoints (ECR / STS / FSx) | B § B.4 |
| EKS VPC / SG mismatch with HyperPod | B § B.5 |
| Kernel panic / watchdog / hung task | N: Kernel |
| Need shell on a node | K: SSM |
| Collect logs for AWS Support | L: Log Collection |
A: EFA / Security Group
Per the HyperPod prerequisites doc, the SG must allow all inbound and outbound to itself. scripts/check-efa-sg.sh validates self-ref rules on every cluster SG. On-node EFA check via scripts/check-node-reachability.sh over SSM. Full: § A.
B: VPC / Routing
SG/subnet VPC mismatch, missing S3 Gateway endpoint, EKS auth mode, worker→controller routing, VPC DNS support, private-subnet + NAT / VPC endpoints, EKS↔HyperPod VPC alignment. scripts/check-vpc-config.sh. Full: § B.
C: Capacity / AZ
Insufficient-capacity failure at creation, or no subnets in the AZ where capacity is available. Check AZ offerings via describe-instance-type-offerings, then change subnet AZ or use Flexible Training Plans / ODCR. Full: § C.
D: Lifecycle Scripts
Surfaced in cluster events + CloudWatch under LifecycleConfig/<group>/<instance-id>. Common: S3 connectivity, IAM gaps, CRLF line endings, infinite loops, parameter-name mismatch. Full: § D.
E: Software Versions
Delegate to hyperpod-version-checker to compare NVIDIA driver, CUDA, NCCL, EFA installer, OFI NCCL, PyTorch across nodes. Ensure job env has FI_PROVIDER=efa, FI_EFA_USE_DEVICE_RDMA=1, NCCL_SOCKET_IFNAME=^lo,docker. Full: § E.
F: Hardware / Auto-Repair
Confirm NodeRecovery=Automatic, inspect the EKS health labels + sagemaker.amazonaws.com/fault-details annotation, and read the SagemakerHealthMonitoringAgent/<group>/<instance> CloudWatch stream. HMA runs passive background checks on GPU and Neuron state and reboots the node on count mismatch (per the HMA doc: "if there's a mismatch between the expected number of GPUs … and the count returned by nvidia-smi, then HMA reboots the node"; same for neuron-ls). Manual recovery order: reboot first, replace only if reboot fails; the preferred path is the batch APIs (BatchReboot/BatchReplaceClusterNodes). Full: § F · patterns: node-issue-catalog.md.
G: GPU / Accelerator
NVIDIA (p4d/p5/g5/g6): nvidia-smi + dmesg over SSM for Xid, ECC, thermal throttling. Xid classification per NVIDIA's catalog: 13 Graphics Engine Exception (application-level), 31 GPU memory page fault (application, can be driver/HW), 63 GPU memory remapping event (HW/ECC), 71 CE4 Error (HW copy engine), 74 NVLink Error (HW), 79 GPU has fallen off the bus (PCIe bus), 109 Context Switch Timeout Error (HW). Any uncorrectable ECC → drain and replace. Row-remap state is the authoritative silent-degradation signal (§ G.1.a).
Trainium / Inferentia (trn1/trn2/inf2): Neuron SDK — neuron-ls, neuron-top, neuron-monitor. nvidia-smi does not apply.
GPU / accelerator failures flow into § F for reboot / replace. Full: § G.
H: Slurm Node Management
Node down/unresponsive, unexpected reboots, stuck PENDING/COMPLETING jobs, Slurm-to-instance-ID translation. Primary access is SSM; diagnose slurmd first, fix the root cause, then start/resume the node per § H. Full: § H.
I: Resource Exhaustion
Disk full (HyperPod root volume defaults to 100 GB and is not intended to grow post-creation), OOM, os.fork() memory error, /dev/shm exhaustion, inode exhaustion. Fork-memory fix: export FI_EFA_USE_HUGE_PAGE=0. Redirect bulk data to /opt/sagemaker (secondary EBS) or /opt/dlami/nvme (instance store). Full: § I.
J: Configuration
p5.48xlarge reports 96 vCPU instead of 192 → set ThreadsPerCore=2 via update-cluster. Full: § J.
K: Node Access via SSM
No direct SSH on HyperPod. Target format sagemaker-cluster:<CLUSTER_ID>_<GROUP>-<INSTANCE_ID>. Failures: plugin missing, wrong prefix, IAM, VPC endpoints. Full: § K.
L: Log Collection
Delegate to hyperpod-issue-report for S3-stored bundles. Key CloudWatch streams: LifecycleConfig/<group>/<instance-id>, SagemakerHealthMonitoringAgent/<group>/<instance-id>. Full: § L.
M: Container Runtime
CrashLoopBackOff, OOMKilled, ImagePullBackOff, RunContainerError on EKS. kubectl describe pod + on-node crictl ps -a, journalctl -u containerd. Full: § M.
N: Kernel & System
Kernel panic, watchdog timeout, soft lockup, unexpected reboots not explained by HyperPod health monitoring. dmesg | grep -iE 'panic|watchdog|hung_task|NMI' + journalctl -b -1. nvrm-related signatures point at NVIDIA driver crashes. Full: § N.
O: CNI / Pod Networking
VPC CNI (aws-node) failures, IPAMD errors, gRPC 127.0.0.1:50051 refused, pods stuck Pending with FailedCreatePodSandBox. Script auto-checks aws-node, kube-proxy, CoreDNS. Full: § O.
Prerequisites
awsCLI v2, recent enough to support the HyperPod cluster commands (describe-cluster,list-cluster-nodes,batch-reboot-cluster-nodes,batch-replace-cluster-nodes)python3,bash4+ (associative arrays are required by the scripts)kubectlauthenticated to the EKS cluster (K8s checks skipped if absent)session-manager-pluginfor on-node hardware checksunbuffer(from theexpectpackage) — optional; if missing, SSM on-node probes are skipped while the rest of the triage still runs. Install viayum install expect/apt install expect.
Defaults
- Region — required: pass
--regionor set$AWS_DEFAULT_REGION. - Target scope — all nodes;
--node <ID>focuses one. - Event window — up to 500 most recent events (5 × 100, paginated).
- Node list cap — up to 20,000 nodes (200 × 100); warns on cap.
- SSM probes — 180 s per node with retry-on-throttle.
- Colors — auto-disabled on non-TTY;
--no-colorto force off.
Error handling
| Failure | Script | Tell the customer |
|---|---|---|
aws sts get-caller-identity fails |
Exit 1 | "Fix AWS credentials and rerun." |
describe-cluster fails |
Exit 1 after listing region's clusters | "Confirm cluster name and region." |
sagemaker:* / ec2:* / logs:* AccessDenied |
Warn, add Missing IAM permission for <API>, continue |
"Grant the listed IAM action and rerun." |
kubectl absent or unauthenticated |
Skip K8s checks | "Install/authenticate kubectl (see § K)." |
session-manager-plugin absent |
Skip on-node probes | "Install session-manager-plugin (see § K)." |
SSM start-session fails or times out (180s) |
Mark node unreachable with → § K pointer |
"Rerun with --node <ID> to isolate; verify SSM agent on the node." |
| Cluster > 20,000 nodes | First 20,000 paginated; warn | "Use --node to target specific nodes." |
Exit codes: 0 triage complete · 1 cluster not found or fatal prerequisite missing.
IAM permissions
Read-only diagnostic — covers triage-cluster.sh, check-efa-sg.sh, check-vpc-config.sh, and check-node-reachability.sh:
{
"Action": [
"sagemaker:DescribeCluster",
"sagemaker:DescribeClusterNode",
"sagemaker:ListClusterNodes",
"sagemaker:ListClusterEvents",
"sagemaker:ListClusters",
"eks:DescribeCluster",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeVpcs",
"ec2:DescribeVpcAttribute",
"ec2:DescribeVpcEndpoints",
"ec2:DescribeRouteTables",
"ec2:DescribeNetworkInterfaces",
"ec2:DescribeInstances",
"ec2:DescribeInstanceTypeOfferings",
"ec2:DescribeInstanceTypes",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:FilterLogEvents",
"ssm:StartSession",
"ssm:TerminateSession",
"service-quotas:GetServiceQuota"
]
}
sts:GetCallerIdentity is implicit — it requires no IAM action. SSM on HyperPod uses start-session against sagemaker-cluster:<cluster-id>_<group>-<iid> targets — not send-command against bare instance IDs. For remediation commands, grant the matching write permission (e.g. ec2:AuthorizeSecurityGroupIngress / Egress, ec2:RevokeSecurityGroupIngress / Egress, ec2:ModifyVpcAttribute, sagemaker:UpdateCluster, sagemaker:BatchRebootClusterNodes, sagemaker:BatchReplaceClusterNodes). Not needed for the diagnostic itself.
Skill delegation
| Need | Use |
|---|---|
| Cluster creation / deployment failures | hyperpod-cluster-debugger (§ A / B / C / H + --validate) |
| Cluster-wide SSM outage | hyperpod-cluster-debugger § F |
| Single-node SSM failure | stay here — § K |
| Cluster-wide EFA health-check failure at creation time | hyperpod-cluster-debugger § A |
| Single-node EFA failure post-provisioning | stay here — § A |
| NCCL AllReduce / collective-op timeouts (distributed) | hyperpod-nccl |
| Silent GPU NaNs on a specific node (row-remap / DCGM) | stay here — § G.1 (even if discovered by NCCL) |
| Post-deployment cluster-wide management | hyperpod-cluster-debugger |
| Shell / commands on nodes | hyperpod-ssm |
| CUDA / NCCL / EFA version comparison | hyperpod-version-checker |
| Diagnostic bundle for AWS Support | hyperpod-issue-report |
| Training performance / MFU degradation | hyperpod-mfu-debugger |
Escalate to AWS Support
Escalate when:
- SG rules correct and reachability passes but EFA still fails.
- VPC correct but K8s bootstrap fails — check VPC flow logs for REJECT.
- Hardware failure where replacement keeps failing (bad physical host).
- Node replacement fails with an insufficient-capacity signal despite a valid ODCR.
Before opening the case
# 1. Cluster identity + affected node status
aws sagemaker describe-cluster --cluster-name <CLUSTER> --region <REGION>
aws sagemaker list-cluster-nodes --cluster-name <CLUSTER> --region <REGION> \
--query "ClusterNodeSummaries[?InstanceId=='<INSTANCE_ID>']"
# 2. Triage bundle (scoped to the affected node where possible)
bash scripts/triage-cluster.sh --cluster <CLUSTER> --region <REGION> --node <INSTANCE_ID> > triage.txt
# 3. Per-node log/config bundle to S3 (delegates to hyperpod-issue-report)
# See skills/hyperpod-issue-report/SKILL.md for the exact invocation.
Include in the case
- Cluster name + ARN and AWS region
- Orchestrator (EKS or Slurm)
- Affected instance IDs / node names / instance-group names
- Timestamp window (UTC start / end) of the failure
- Exact error strings observed (copy verbatim from pod logs, CloudWatch, dmesg, events)
- XID numbers / ECC counts / DCGM output where hardware is implicated
triage.txtfrom step 2 above- S3 URI of the
hyperpod-issue-reportbundle from step 3
Patterns from real customer tickets: node-issue-catalog.md.
이 스킬의 파일
7hyperpod-node-debugger skill 질문
Claude Code에 hyperpod-node-debugger skill을 어떻게 설치하나요?
npx degit awslabs/agent-plugins/plugins/sagemaker-ai/skills/hyperpod-node-debugger ~/.claude/skills/hyperpod-node-debugger를 실행합니다. Claude Code는 ~/.claude/skills(모든 프로젝트) 또는 .claude/skills(한 프로젝트)에서 skills를 로드하며, 설명이 작업과 일치할 때 skill을 사용합니다.
hyperpod-node-debugger skill을 사용할 수 있는 클라이언트는 무엇인가요?
Agent Skills 형식을 읽는 모든 클라이언트(Claude Code, Codex, Gemini CLI, claude.ai(ZIP으로 업로드))가 사용할 수 있습니다. skill 내 스크립트는 클라이언트가 코드 실행을 허용하는 경우에만 실행됩니다.
결함이 있는 노드를 수정하기 위한 권장 작업 순서는 무엇인가요?
파괴적인 작업 순서는 조사, 재부팅, 마지막으로 노드 교체 순으로 진행합니다.
클러스터 또는 VPC 설정을 변경하는 명령은 어떻게 처리해야 하나요?
먼저 해당 리소스가 Infrastructure-as-Code(Terraform 또는 CloudFormation 등)로 관리되는지 확인해야 합니다. 만약 그렇다면, 구성 드리프트(configuration drift)를 방지하기 위해 IaC 소스에서 먼저 변경 사항을 적용하도록 제안하십시오.
GPU에서 수정 불가능한 ECC 오류가 보고되면 어떻게 해야 하나요?
수정 불가능한 ECC 오류가 감지되면 해당 노드를 드레인(drain)하고 교체해야 합니다.