
Amazon AWS
Amazon Web Services (AWS) released a diagnostic summary for HyperPod observability in Amazon SageMaker on Nov. 19 that flags suspect GPUs two ways: by comparing each card against its peers inside a training job, and by matching rules for ECC memory errors, Xid faults, network degradation on the Elastic Fabric Adapter (EFA), clock throttling and overheating. The catch for anyone planning to trust a clean result: AWS's guide on silent data corruption, published eight days earlier on Nov. 11, says that fault raises no Xid error and escapes ECC protection, the two signal types the new hardware rules are described as watching.
The summary works on clusters orchestrated by Amazon EKS or Slurm, in every AWS Region where HyperPod is offered. It targets the question operators ask when a training job slows: which GPU, node or network link is the problem. Operators gain a starting list of suspects, but they also need to know what that list cannot clear, and AWS's own documents plus outside research show where each of its two mechanisms breaks.
According to AWS's post, the first mechanism calls the CloudWatch DetectGroupOutliers API to compare GPUs on five measures: utilization, memory usage, temperature, power draw and SM clock frequency, the speed of the GPU's compute cores. A GPU that deviates significantly from its peers appears as an outlier card, and AWS says the comparison works for jobs on both EKS and Slurm.
Peer comparison fits how synchronous training behaves. All-reduce and barrier operations make every worker wait at each synchronization step, so one slow GPU can set the pace for the whole job without raising an error. Meta's Llama 3 authors wrote that a single straggler can slow thousands of other GPUs. AWS makes the same point, saying a single unhealthy accelerator can stall an entire distributed job.
Each flagged issue names the affected job, cluster, accelerator and rank. In the CloudWatch SageMaker Insights dashboard, findings appear as incident cards with a cross-layer explanation, alongside evidence charts drawn against the fleet average.
The second mechanism is a set of static rules for known failure patterns. AWS groups them into four rule categories:
| Category | What it catches |
|---|---|
| Hardware | GPU ECC memory errors, Xid faults and accelerator failures |
| Network | Throughput degradation, packet drops and retransmissions on EFA |
| Throttling | GPU clock throttling caused by heat, board or reliability limits |
| Overheating | GPU temperatures above configured limits |
Two of those signal types matter for what comes later. An Xid is a GPU error code, and NVIDIA's Xid error documentation exists to help operators interpret them when troubleshooting. ECC memory detects and corrects corrupted memory contents. Heat matters too: Meta's paper reports a 1% to 2% daily throughput swing from mid-day temperatures affecting GPU voltage and frequency scaling.
Read more: NVIDIA DSX OS Guide Says Slow GPU Nodes Pass Tests Until Operators Set Thresholds
Telemetry volume sets the engineering tradeoff. Default metrics are collected automatically every 30 seconds and cover cluster health, CPU, disk and OS usage, GPU utilization and temperature, and per-job resource use, at roughly 1% to 2% of CPU and 250 MB to 300 MB of memory. Advanced metrics are opt-in and add the full node exporter suite, all NVIDIA DCGM and AWS Neuron metrics (including clock speeds, power, PCIe activity and error indicators), and EFA metrics for throughput, drops, retransmissions and errors. AWS warns that they raise telemetry volume and cost.
The tier split raises a question AWS's post leaves open. The default list names GPU utilization and temperature, while clock speeds and power sit in the advanced list, yet SM clock and power draw are two of the five outlier measures. The post does not say whether the outlier comparison on those measures, or the ECC and throttling rules, requires advanced metrics to be switched on. AWS's separate silent-corruption guide tells operators to turn on advanced DCGM metrics if they want ECC, NVLink and throttling counters. That is an analytical reading of the two documents, not an AWS statement: operators who rely on the hardware and throttling rules may need the costlier tier.
Silent data corruption (SDC) is a hardware fault, typically in a GPU, that produces a wrong but plausible result without raising any error. In distributed training the bad value can spread into every replica's weights through the next all-reduce.
AWS's guide explains why the usual checks miss it. ECC protects memory, not arithmetic, so a tensor-core result carries no ECC. A GPU raises an Xid only when it detects a fault, so corruption that triggers no detection raises no Xid error. DCGM deep health checks run on idle nodes and may not trigger the defect. AWS's guide is blunt about hardware telemetry panels: "A clean panel doesn't clear the node."
The problem is documented at scale. Meta's Llama 3 reliability table lists silent data corruption as the cause of 6 of 419 unexpected interruptions, or 1.4%. AWS notes the figure covers only cases traced to silent data corruption. Google and Facebook researchers separately reported in 2021 that faulty processor cores can corrupt results, in work including the paper "Cores that Don't Count."
The Nov. 19 post does not mention silent data corruption, and its hardware category lists ECC, Xid and accelerator failures. Read together with AWS's own guide, a clean diagnostic summary should not be treated as clearance for a node. AWS's recommended detection runs inside the training loop instead, through a non-finite gradient check, ring replay audit and a weight fingerprint comparison across replicas, with the replay audit having each GPU recompute another node's work. The cost is step time: on a 953-million-parameter model across 32 H100 GPUs, deterministic audits every 100 steps added about 10% in AWS's tests, and every 10 steps added about 35%.
Researchers at New York University and ByteDance analyzed 3,079 large-model pretraining jobs of at least 128 GPUs, traced from January to May 2024 at ByteDance. They found that 42.5% of the jobs ran at least 10% slower because of stragglers, and that stragglers wasted up to 45% of allocated resources at the tail. The authors found hardware or software problems in a server were "not a significant cause" of the stragglers they observed. Instead they attributed computation stragglers mainly to workload partitioning imbalance, sequence-length imbalance and Python garbage-collection pauses, and communication stragglers to external issues such as switch or network-card flapping.
Meta's paper points the other way, noting that hardware issues may cause slow but still-functioning stragglers that are hard to detect. The two datasets differ, and the ByteDance jobs are not AWS clusters, so both findings can hold. The risk for peer comparison is specific, and this part is analysis rather than an AWS statement: utilization, memory and power move with how much work a GPU receives, so an unevenly loaded pipeline stage can look like an outlier. AWS's post does not address whether DetectGroupOutliers separates uneven workloads from faults, and it gives no deviation thresholds, minimum group size or false-positive handling. Anomaly-detection literature makes the same point in general terms: contextual anomalies depend on context, and the threshold choice drives the outcome. The static rules, which look for known fault signatures directly, should be less affected by uneven work.
Microsoft Research's gray failure work describes the broader problem class as faults that failure detectors may not notice even when applications are afflicted. Peer comparison is one attempt to catch those faults, and it carries the caveat above.
AWS's undated documentation for the HyperPod observability add-on lists preconfigured alerts, including a warning when GPU temperature exceeds 80°C (176°F) for five minutes and a critical alert above 85°C (185°F) for one minute. The same list makes any Xid error a critical alert that fires immediately.
The Nov. 19 post is vaguer about the new rules. It says only that the overheating rule fires on temperatures above configured limits, and it does not say whether those limits match the documented alerts. Left unpublished in the post are the outlier deviation thresholds, the limits behind the static rules, the minimum group size, detection latency, prices and the supported instance types, which it leaves to the HyperPod documentation.
Telemetry now goes to Amazon CloudWatch by default, where operators read it through CloudWatch SageMaker Insights and the SageMaker console. AWS says the route is priced lower than Amazon Managed Service for Prometheus, which was the only option before this release, but the post gives no figures. Teams that standardize on Prometheus and Grafana can keep that path, with prebuilt Amazon Managed Grafana dashboards, though first-time Grafana users must set up AWS IAM Identity Center.
The earlier HyperPod observability capability, announced July 10, 2025, was built on Amazon Managed Grafana and Amazon Managed Service for Prometheus and excluded two Regions, US West (N. California) and Asia Pacific (Melbourne). The new announcement covers every Region where HyperPod is offered.
The release follows another HyperPod operations update. On Nov. 17, AWS added cluster operations, which track scaling and on-demand Deep Health Check requests from submission to completion on the cluster's Events tab, at no additional cost for clusters using continuous provisioning. AWS's silent-corruption guide recommends confirming ambiguous alarms with an on-demand deep health check before replacing a node. Operators can also enable automatic instance recovery as a cluster setting.
Taken together, HyperPod now offers a way to find a suspect node, a way to run a health check on it and an audit trail for both. That synthesis is analysis, not an AWS claim.
The CloudWatch dashboard has four views. The diagnostic summary holds incident cards and evidence charts. The hardware view shows fleet topology by Active, Idle and Unhealthy nodes with a filterable device table. The workload view lists training jobs by type, status and duration with per-job EFA, GPU, CPU and memory use, flags abnormal network usage, and offers Profile CPU and Inspect NCCL actions on job pages. The network view shows fleet-wide EFA utilization, degraded nodes, retransmits, NCCL all-reduce latency and EFA health by instance type. The post describes profiling mainly through those actions and a host-profiling setting, not in detail.
On EKS the feature is an add-on named amazon-sagemaker-hyperpod-observability. On Slurm it is a cluster extension named sagemaker-observability, managed from the cluster's Extensions tab. Access is controlled through AWS IAM. With Quick setup, observability is enabled and cannot be turned off. With Custom setup, operators choose metric categories, advanced metrics, log groups, the metric store and a Grafana workspace. Existing clusters can be changed through the console or AWS command-line tools.
For operators of large GPU fleets, the diagnostic summary replaces a hand-built query with a ready list of suspects, and that is its value. It also comes with documented limits. It compares GPUs against peers on five measures and watches Xid, ECC, EFA, throttling and temperature signals, yet AWS publishes none of the numbers behind either mechanism. AWS's own guide says silent data corruption evades the Xid and ECC signals and that a clean panel does not clear a node. A study by researchers at New York University and ByteDance found many stragglers trace to uneven work distribution and Python pauses rather than failing hardware. A flagged GPU is worth checking against its pipeline stage and sequence-length share before anyone replaces it. A clean summary is worth pairing with in-loop corruption checks; deep health checks, which AWS says run on idle nodes and may not trigger the defect, are better used to confirm ambiguous alarms than to clear a node. The missing thresholds, minimum group size and prices are worth asking AWS about before committing to advanced metrics at scale.
AWS's Nov. 19 post does not say it can. AWS's separate guide relies on detection inside the training loop, and in its tests on 32 H100 GPUs, default-kernel audits, which compare only the loss, missed a fault confined to the backward pass for 50 steps, while deterministic audits that compare gradient fingerprints caught it. The guide reports zero false alarms across 15,320 fault-free audit steps.
CloudWatch is the default metric store, and AWS says it costs less than Amazon Managed Service for Prometheus. Advanced metrics add telemetry volume and cost, and the post lists no prices, pointing readers to the HyperPod documentation.
Yes. The diagnostic summary and outlier detection work for jobs on both orchestrators, and observability is available in every Region where HyperPod is offered.
An Xid is an error code reported for GPU problems. NVIDIA's Xid documentation exists to help operators interpret them when troubleshooting, and AWS's preconfigured alerts treat any Xid error as critical, firing immediately.
