Reliability, availability, serviceability (RAS)

Reliability, availability, serviceability (RAS)#

RAS aims to increase the robustness of a system by detecting hardware errors, recording them, and correcting them where possible. See Reliability, availability, serviceability (Linux kernel) for more general information.

ECC#

ECC (Error-Correcting Code) is a type of memory to automatically detect errors. Correctable 1-bit errors are handled by the ECC logic and logged by the hardware. Uncorrectable 2-bit errors can be detected but not reliably fixed; this is a more serious event that must be reported. See RAS Error Count sysfs Interface to learn how AMD SMI accesses error counts.

While ECC is a mechanism to handle different errors, CPER is the standard used to report that the event occurred.

CPER#

At its core, CPER (Common Platform Error Record) is a standard format included in the UEFI specification to report errors to the operating system. It works as a standard error report template that different hardware components can fill out when something goes wrong. It consists of a header, one or more section descriptors – and for each descriptor, an associated section containing error or informational data. See CPER (UEFI Specification) for more information.

A CPER record consists of vital information for diagnostics such as:

  • Error source

  • Error type

  • Error severity

    • 0 - Recoverable (also called non-fatal uncorrected)

    • 1 - Fatal

    • 2 - Corrected

    • 3 - Informational

  • Timestamp

  • Other data

A CPER record might contain an AFID in its data to help map a complex error to a more actionable service task.

AFID#

AFIDs (AMD Field ID) are unique numerical IDs associated with specific events or errors produced by AMD Instinct accelerators. It provides a specific identifier for a known condition, which helps facilitate root cause analysis. Each AFID is associated with category, type, and severity fields. See AFID Event List for more information.

From concept to action#

AMD SMI provides tools to programmatically monitor and manage these RAS features.

The AMD SMI library provides APIs to query ECC error counts and manage CPER records (list, decode, and clear).

See ECC information and RAS information for available APIs.

See related APIs:

See amd-smi ras --help for details and available options.

amd-smi ras --help

CLI examples#

The following sections summarize the amd-smi flags that drive each RAS feature. Global modifiers (--json, --csv, --file, --gpu, etc.) are documented in amd-smi --help and apply uniformly; the tabs below focus on the flags specific to ECC, CPER, and AFID.

ECC#

ECC counters are surfaced through amd-smi metric (one-shot snapshot) and amd-smi monitor (live tabular stream). There is no dedicated amd-smi ecc subcommand.

Print the per-GPU total correctable / uncorrectable / deferred ECC error counts (alias --ecc).

~$ amd-smi metric -e --gpu 0
GPU: 0
    ECC:
        TOTAL_CORRECTABLE_COUNT: 0
        TOTAL_UNCORRECTABLE_COUNT: 0
        TOTAL_DEFERRED_COUNT: 0
        CACHE_CORRECTABLE_COUNT: 0
        CACHE_UNCORRECTABLE_COUNT: 0

Break the counts down per IP block (UMC, SDMA, GFX, MMHUB, PCIE_BIF, HDP, …) with -k (alias --ecc-blocks).

~$ amd-smi metric -k --gpu 0
GPU: 0
    ECC_BLOCKS:
        UMC:
            CORRECTABLE_COUNT: 0
            UNCORRECTABLE_COUNT: 0
            DEFERRED_COUNT: 0
        SDMA:
            CORRECTABLE_COUNT: 0
            UNCORRECTABLE_COUNT: 0
            DEFERRED_COUNT: 0
        GFX:
            CORRECTABLE_COUNT: 0
            UNCORRECTABLE_COUNT: 0
            DEFERRED_COUNT: 0
        ...

Continuously monitor ECC single-bit (correctable), double-bit (uncorrectable), and PCIe replay error counts in a watch-style table. Press CTRL+C to stop.

~$ amd-smi monitor -e
GPU  XCP  SINGLE_ECC  DOUBLE_ECC  PCIE_REPLAY
  0    0           0           0            0
  1    0           0           0            0
  ...

CPER#

CPER retrieval is exposed through amd-smi ras --cper. Without --folder only a summary table is printed; with --folder each entry is also dumped as a matching .cper (raw binary) and .json (decoded metadata) pair.

List current CPER entries from the kernel driver as a summary table. No files are written; a warning is printed reminding the user to pass --folder to dump them.

~$ sudo amd-smi ras --cper
WARNING: No CPER files will be dumped unless --folder=<folder_name> is specified and cper entries exist.
timestamp            gpu_id  severity
2000/06/27 10:45:13  0       FATAL
2000/06/27 10:45:13  1       FATAL

Same listing as List, plus dump each entry to <DIR> as <severity>-<n>.cper (raw bytes) and <severity>-<n>.json (decoded metadata). The summary table gains file_name and list of afids columns.

~$ sudo amd-smi ras --cper --folder /tmp/cper_dump/
timestamp            gpu_id  severity             file_name         list of afids
2000/06/27 10:45:13  0       FATAL                fatal-1.cper      30
2000/06/27 10:45:13  1       FATAL                fatal-2.cper      30

Filter the listing by severity. Accepted values: nonfatal-uncorrected, nonfatal-corrected, fatal, all. Combine with any other CPER flag.

~$ sudo amd-smi ras --cper --severity fatal --folder /tmp/cper_dump/
timestamp            gpu_id  severity             file_name         list of afids
2000/06/27 10:45:13  0       FATAL                fatal-1.cper      30

After dumping, prune the oldest .cper/.json pairs in <DIR> so at most N .cper files remain.

~$ sudo amd-smi ras --cper --severity all --folder /tmp/cper_dump --file-limit 5
timestamp            gpu_id  severity             file_name         list of afids
2000/06/27 10:45:13  0       FATAL                fatal-1.cper      30
2000/06/27 10:45:13  1       FATAL                fatal-2.cper      30
2000/06/27 10:45:13  2       FATAL                fatal-3.cper      30
2000/06/27 10:45:13  3       FATAL                fatal-4.cper      30
2000/06/27 10:45:13  4       FATAL                fatal-5.cper      30

Continuously poll for new CPER entries until interrupted with CTRL+C. Press CTRL + C to stop. is printed once at startup. Combine with --folder to also dump new entries as they arrive.

~$ sudo amd-smi ras --cper --follow --severity all --folder /tmp/cper_dump
timestamp            gpu_id  severity             file_name         list of afids
Press CTRL + C to stop.
2000/06/27 10:45:13  0       FATAL                fatal-1.cper      30
...

AFID#

AFID extraction is a pure offline operation: it parses one or more CPER record files with amdsmi_get_afids_from_cper() and prints the AFIDs found in each. No driver or GPU access is required, so the source CPER(s) may have been captured on another system. Each CPER record carries up to 12 AFIDs.

--cper-file and --folder are mutually exclusive under --afid; exactly one must be supplied. Unlike the --cper --folder write path, the AFID --folder must already exist and contain at least one .cper file — it is not auto-created.

Parse one CPER record and print the space-separated list of AFIDs encoded in it. A single - is printed when the record contains no AFID payload (e.g. corrected entries).

~$ amd-smi ras --afid --cper-file /tmp/cper_dump/fatal-1.cper
30 31 32 33

Parse every *.cper in a pre-existing directory and print a file_name | list of afids table, one row per record. A record with no AFID payload shows -, and a file that cannot be parsed shows decode failed. Symlinks in the folder are skipped (they are never followed).

~$ amd-smi ras --afid --folder /tmp/cper_dump/
file_name                        list of afids
fatal-1.cper                     30 31 32 33
fatal-2.cper                     30 31 32 33
nonfatal-1.cper                  -
truncated.cper                   decode failed

Further reading#

Contents