.. meta:: :description: ROCm Compute Profiler performance model: L2 cache (TCC) :keywords: Omniperf, ROCm Compute Profiler, ROCm, profiler, tool, Instinct, accelerator, L2, cache, infinity fabric, metrics ************** L2 cache (TCC) ************** The L2 cache is the coherence point for current AMD Instinct™ MI-Series GCN™ and CDNA™ architecture-based GPUs, and is shared by all :doc:`CUs ` on the device. Besides serving requests from the :doc:`vector L1 data caches `, the L2 cache also is responsible for servicing requests from the :ref:`L1 instruction caches `, the :ref:`scalar L1 data caches ` and the :doc:`command processor `. The L2 cache is composed of a number of distinct channels (32 on AMD Instinct MI100 and :ref:`MI2XX ` Series CDNA architecture-based GPUs at 256B address interleaving) which can largely operate independently. Mapping of incoming requests to a specific L2 channel is determined by a hashing mechanism that attempts to evenly distribute requests across the L2 channels. Requests that miss in the L2 cache are passed out to :ref:`Infinity Fabric™ ` to be routed to the appropriate memory location. The L2 cache metrics reported by ROCm Compute Profiler are broken down into four categories: * :ref:`L2 Speed-of-Light ` * :ref:`L2 cache accesses ` * :ref:`L2-Fabric transactions ` * :ref:`L2-Fabric stalls ` .. _l2-sol: L2 Speed-of-Light ================= .. warning:: The theoretical maximum throughput for some of the listed metrics are currently computed with the maximum achievable clock frequency, as reported by ``rocminfo``, for a GPU. This may not be realistic for all workloads. The L2 cache's speed-of-light table contains a few key metrics about the performance of the L2 cache, aggregated over all the L2 channels, as a comparison with the peak achievable values of those metrics: .. tab-set:: .. tab-item:: CDNA .. jinja:: l2-sol-gfx908 :file: _templates/metrics_table.j2 .. tab-item:: CDNA 2 .. jinja:: l2-sol-gfx90a :file: _templates/metrics_table.j2 .. tab-item:: CDNA 3 .. jinja:: l2-sol-gfx942 :file: _templates/metrics_table.j2 .. tab-item:: CDNA 4 :selected: .. jinja:: l2-sol-gfx950 :file: _templates/metrics_table.j2 .. note:: The L2 cache on CDNA architecture-based AMD Instinct MI-Series uses a "hit-on-miss" approach to reporting cache hits. While satisfying a miss, if another request comes in that would hit on the same pending cache line, the subsequent request will be counted as a `hit`. As such, it is also important to consider the latency metric in the :ref:`L2-Fabric ` section when evaluating the L2 hit rate. .. _l2-cache-accesses: L2 cache accesses ================= This section details the incoming requests to the L2 cache from the :doc:`vL1D ` and other clients -- for instance, the :ref:`sL1D ` and :ref:`L1I ` caches. .. tab-set:: .. tab-item:: CDNA .. jinja:: l2-cache-accesses-gfx908 :file: _templates/metrics_table.j2 .. tab-item:: CDNA 2 .. jinja:: l2-cache-accesses-gfx90a :file: _templates/metrics_table.j2 .. tab-item:: CDNA 3 .. jinja:: l2-cache-accesses-gfx942 :file: _templates/metrics_table.j2 .. tab-item:: CDNA 4 :selected: .. jinja:: l2-cache-accesses-gfx950 :file: _templates/metrics_table.j2 .. note:: All requests to the L2 are for a single cache line's worth of data. The size of a cache line may vary depending on the GPU, however on a CDNA2 architecture-based AMD Instinct :ref:`MI2XX ` GPU, it is 128B, while it is 64B on MI100 GPUs. .. _l2-fabric: L2-Fabric transactions ====================== Requests/data that miss in the L2 must be routed to memory in order to service them. The backing memory for a request may be local to the GPU (i.e., in the local high-bandwidth memory), in a remote GPU's memory, or even in the CPU's memory. Infinity Fabric is responsible for routing these memory requests/data to the correct location and returning any fetched data to the L2 cache. The :ref:`l2-request-flow` describes the flow of these requests through Infinity Fabric in more detail, as described by ROCm Compute Profiler metrics, while :ref:`l2-fabric` give detailed definitions of individual metrics. .. _l2-request-flow: Request flow ------------ The following diagram illustrates how the `L2↔Fabric` requests are reported by ROCm Compute Profiler: .. figure:: ../../data/performance-model/l2perf_model.png :align: center :alt: L2-Fabric transaction flow on AMD Instinct MI-series GPUs :width: 800 Requests from the L2 Cache are broken down into two major categories, read requests and write requests (at this granularity, atomic requests are treated as writes). These requests can additionally be subdivided in a number of ways. First, these requests may be sent across Infinity Fabric as different transaction sizes, 32B or 64B on current CDNA architecture-based GPUs. .. note:: On current CDNA architecture-based GPUs, the 32B read request path is expected to be unused and so is disconnected in the flow diagram. In addition, the read and write requests can be further categorized as: * Uncached read/write requests, for instance: for access to :ref:`fine-grained memory ` * Atomic requests, for instance: for atomic updates to :ref:`fine-grained memory ` * HBM read/write requests OR remote read/write requests, for instance: for requests to the GPU's local HBM OR requests to a remote GPU’s HBM or the CPU’s DRAM These classifications are not necessarily exclusive. For example, a write request can be classified as an atomic request to the GPU’s local HBM, and an uncached write request. The request-flow diagram marks exclusive classifications as a splitting of the flow, while non-exclusive requests do not split the flow line. For example, a request is either a 32B Write Request OR a 64B Write request, as the flow splits at this point. However, continuing along, the same request might be an atomic request and an uncached write request, as reflected by a non-split flow. The :ref:`uncached ` read requests to :ref:`fine-grained memory ` are handled specially on CDNA-based GPUs, as indicated in the request flow diagram. These are expected to be counted as a 64B Read Request. However, requests to uncached memory (denoted by the dashed line), will be counted as two uncached read requests (that is, the request is split). .. _l2-fabric-metrics: Metrics ------- The following metrics are reported for the L2-Fabric interface: .. tab-set:: .. tab-item:: CDNA .. jinja:: l2-fabric-metrics-gfx908 :file: _templates/metrics_table.j2 .. tab-item:: CDNA 2 .. jinja:: l2-fabric-metrics-gfx90a :file: _templates/metrics_table.j2 .. tab-item:: CDNA 3 .. jinja:: l2-fabric-metrics-gfx942 :file: _templates/metrics_table.j2 .. tab-item:: CDNA 4 :selected: .. jinja:: l2-fabric-metrics-gfx950 :file: _templates/metrics_table.j2 .. _l2-detailed-metrics: Detailed transaction metrics ---------------------------- The following metrics are available in the detailed L2-Fabric transaction breakdown table: .. tab-set:: .. tab-item:: CDNA .. jinja:: l2-detailed-metrics-gfx908 :file: _templates/metrics_table.j2 .. tab-item:: CDNA 2 .. jinja:: l2-detailed-metrics-gfx90a :file: _templates/metrics_table.j2 .. tab-item:: CDNA 3 .. jinja:: l2-detailed-metrics-gfx942 :file: _templates/metrics_table.j2 .. tab-item:: CDNA 4 :selected: .. jinja:: l2-detailed-metrics-gfx950 :file: _templates/metrics_table.j2 .. note:: On CDNA4 architecture-based MI350 Series GPUs, the per-destination bandwidth metrics in the table above ("Read/Write/Atomic Bandwidth - HBM, - Infinity Fabric™, and - PCIe") are measured at the L2-Fabric interface and attribute traffic to a destination by request type, not by where the request was ultimately serviced. A request attributed to HBM may instead be served by the last level cache; a request attributed to Infinity Fabric may be served by the remote GPU's last level cache rather than its HBM; and a request attributed to PCIe may be served by host-side cache rather than host memory. .. _l2-fabric-stalls: L2-Fabric interface stalls ========================== When the interface between the L2 cache and Infinity Fabric becomes backed up by requests, it may stall, preventing the L2 from issuing additional requests to Infinity Fabric until prior requests complete. This section gives a breakdown of what types of requests in a kernel causes a stall (like read versus write), and to which locations (such as, to the GPU's local memory, or to remote GPUs or CPUs). .. tab-set:: .. tab-item:: CDNA .. jinja:: l2-fabric-stalls-gfx908 :file: _templates/metrics_table.j2 .. tab-item:: CDNA 2 .. jinja:: l2-fabric-stalls-gfx90a :file: _templates/metrics_table.j2 .. tab-item:: CDNA 3 .. jinja:: l2-fabric-stalls-gfx942 :file: _templates/metrics_table.j2 .. tab-item:: CDNA 4 :selected: .. jinja:: l2-fabric-stalls-gfx950 :file: _templates/metrics_table.j2 .. warning:: On current CDNA-based GPUs and GCN GPUs, these L2↔Fabric stalls can be undercounted in some circumstances. .. rubric:: Footnotes .. [#inf] In addition to being used for on-GPU data-traffic, AMD `Infinity Fabric `_ technology can be used to connect multiple GPUs to achieve advanced peer-to-peer connectivity and enhanced bandwidths over traditional PCIe connections. Some AMD Instinct MI-Series GPUs like the MI250X feature coherent CPU-to-GPU connections are built using AMD Infinity Fabric. For more information, see the `AMD CDNA2 white paper `_. .. rubric:: Disclaimer PCIe® is a registered trademark of PCI-SIG Corporation.