ROCm compiler reference#

ROCm includes compilers optimized for high-performance computing on AMD GPUs and CPUs supporting various heterogeneous programming models such as Heterogeneous-computing Interface for Portability (HIP), OpenMP, and OpenCL.

Important

The ROCm compilers only build the x86 and AMDGPU targets. Functionality and options that are relevant to other targets are not included within the ROCm compilers.

For more details, see:

ROCm compiler interfaces#

The recommended compiler for HIP programs is amdclang++, located at /opt/rocm/bin/amdclang++. It provides the same interface as clang++ and gives full control over compilation.

When invoking amdclang++ directly, specify the following:

amdclang++ -x hip --offload-arch=native my_kernel.cpp --hip-link -o my_kernel
  • -x hip — tells the compiler to treat the source as HIP (required for .cpp files; .hip files are detected automatically)

  • --offload-arch=<target> — specifies the GPU target (e.g. gfx1100). Use --offload-arch=native to auto-detect the GPUs on the current machine. To query available GPU architectures explicitly, use the offload-arch tool:

    offload-arch            # prints the GPU arch(es) on this machine
    amdclang++ -x hip --offload-arch=$(offload-arch) my_kernel.cpp --hip-link -o my_kernel
    

    See Offload-arch tool for full details on querying GPU architectures.

  • --hip-link — links to the HIP runtime library

For CMake-based projects, CMake’s native HIP language support handles all of the above automatically, including GPU target detection:

find_package(hip CONFIG REQUIRED)
enable_language(HIP)

add_executable(my_target my_kernel.hip)
target_link_libraries(my_target PRIVATE hip::device)

find_package(hip CONFIG) auto-detects the installed GPUs and adds the appropriate --offload-arch flags to any target that links hip::device.

Compiler options and features#

This section discusses compiler options and features.

AMD GPU compilation#

This section outlines commonly used compiler flags for hipcc and amdclang++.

Options

Description

-x hip

Compiles the source file as a HIP program.

-fopenmp

Enables the OpenMP support.

-fopenmp-targets=<gpu>

Enables the OpenMP target offload support of the specified GPU architecture, where <gpu> specifies the GPU architecture. For example: gfx908

--gpu-max-threads-per-block=<value>:

Sets the default limit of threads per block, also referred to as the launch bounds, where <value> specifies the default maximum amount of threads per block.

-munsafe-fp-atomics

Enables unsafe floating point atomic instructions (AMDGPU only).

-ffast-math

Allows aggressive, lossy floating-point optimizations.

-mwavefrontsize64, -mno-wavefrontsize64

Sets wavefront size to be 64 or 32 on RDNA architectures.

-mcumode

Switches between CU and WGP modes on RDNA architectures.

--offload-arch=<gpu>

HIP offloading target ID. |br| May be specified more than once, where <gpu> specifies the device architecture followed by target ID features delimited by a colon. |br| Each target ID feature is a predefined string followed by a plus or minus sign (e.g. gfx908:xnack+:sramecc-).

-g

Generates source-level debug information.

-fgpu-rdc,    -fno-gpu-rdc

Generates relocatable device code, also known as separate compilation mode.

-foffload-lto=<arg>

Link Time Optimization (LTO) is a compiler optimization technique that performs program-wide analysis and optimization at the linking stage, enabling more aggressive and comprehensive optimizations than those possible at the file level. The -foffload-lto option for Clang is associated with enabling LTO for offload targets. ThinLTO reduces the memory and time overhead associated with full LTO while still providing significant performance improvements. ThinLTO does not work for all code, so it is not enabled by default. If the device code contains any form of indirect function call (including C++ virtual functions), it is not expected to work. To enable full LTO for device code, use the -foffload-lto=full option, or simply -foffload-lto, for both the compile and linker flags. To enable ThinLTO, use -foffload-lto=thin for both the compile and linker flags. You can separately specify -flto for CPU LTO, and -foffload-lto for device LTO. For more information, see Clang Reference.

-flto-partitions=<num>

Equivalent to the --lto-partitions=<num> LLD option. Controls the number of partitions used for parallel code generation when using full LTO (including when using -fgpu-rdc). The number of partitions must be greater than 0, and a value of 1 disables the feature. The default value is 8. For more information, see Parallel code generation.

Parallel code generation#

The compiler driver will use parallel code generation by default when compiling using full LTO (including when using the -fgpu-rdc option) for HIP. This divides the optimized LLVM IR module into roughly equal partitions before instruction selection and lowering, which can help improve build times.

Each kernel in the linked LTO module can be put in a separate partition, and any non-inlined function it depends on can be copied alongside it. Thus, while parallel code generation can improve build time, it can also duplicate non-inlined, non-kernel functions across multiple partitions, potentially increasing the binary size of the final object file.

Developers are encouraged to experiment with different numbers of partitions using the -flto-partitions Clang command line option. Recommended values are 1 to 16 partitions, with especially large projects containing many kernels potentially benefitting from up to 64 partitions. It is not recommended to use a value greater than the number of threads on the machine. Smaller projects, or projects that contain only a few kernels might also not benefit at all from partitioning and could even see a slight increase in build time due to the small overhead of analyzing and partitioning the modules.

Controlling atomic code generation#

Generating code for atomic operations with AMDGPU targets can have complications related to accessing device/remote memory. You can control how atomic operations are lowered in LLVM IR by using the [[clang::atomic]] attribute. This attribute lets you provide additional metadata to the backend to specify options for atomic operations. For example, you can indicate that certain atomic operations are compatible with specific types of memory, or ensure correctness in denormal mode for floating-point operations. For additional information, see Extensions for controlling atomic code generation in the Clang documentation.

Inline ASM statements#

Inline assembly (ASM) statements allow a developer to include assembly instructions directly in either host or device code. The ROCm compiler supports ASM statements, however you should not use them for the following reasons:

  • The compiler’s ability to produce both correct code and optimize surrounding code is impeded.

  • The compiler does not parse the content of the ASM statements and cannot examine its contents.

  • The compiler must make conservative assumptions in an effort to retain correctness.

  • The conservative assumptions may yield code that is less performant compared to code without ASM statements. It is possible that a syntactically correct ASM statement may cause incorrect runtime behavior.

  • ASM statements are often ASIC-specific; code containing them is less portable and adds a maintenance burden for the developer if different ASICs are targeted.

  • Writing correct ASM statements is often difficult; thorough testing of any ASM statements is strongly encouraged.

Note

For developers who include ASM statements in the code, AMD is interested in understanding the use case and appreciates your feedback at ROCm/ROCm#issues

Miscellaneous OpenMP compiler features#

This section discusses features that have been added or enhanced in the OpenMP compiler.

Offload-arch tool#

An LLVM library and tool that is used to query the execution capability of the current system as well as to query requirements of a binary file. It is used by OpenMP device runtime to ensure compatibility of an image with the current system while loading it. It is compatible with target ID support and multi-image fat binary support.

Usage:

offload-arch [Options] [Optional lookup-value]

When used without an option, offload-arch prints the value of the first offload architecture found in the underlying system. This can be used by various clang front ends. For example, to compile for OpenMP offloading on your current system, invoke clang with the following command:

clang -fopenmp  -fopenmp-targets=``offload-arch`` foo.c

If an optional lookup-value is specified, offload-arch will check if the value is either a valid offload-arch or a codename and look up requested additional information.

The following command provides all the information for offload-arch gfx906:

offload-arch gfx906  - V

The options are listed below:

Options

Description

-h

Prints the help message.

-a

Prints values for all devices. Do not stop at the first device found.

-m

Prints device code name (often found in pci.ids file).

-n

Prints numeric pci-id.

-t

Prints clang offload triple to use for the offload arch.

-v

Verbose. Implies the following options -a -m -n -t

-f <file>

Prints offload requirements including offload-arch for each compiled offload image built into an application binary file.

-c

Prints offload capabilities of the underlying system. This option is used by the language runtime to select an image when multiple images are available. A capability must exist for each requirement of the selected image.

There are symbolic link aliases amdgpu-offload-arch and nvidia-arch for offload-arch. These aliases return 1 if no AMD GCN GPU or CUDA GPU is found. These aliases are useful in determining whether architecture-specific tests should be run or to conditionally load architecture-specific software.

Command-line simplification using offload-arch flag#

Legacy mechanism of specifying offloading target for OpenMP involves using three flags, -fopenmp-targets, -Xopenmp-target, and -march. The first two flags take a target triple (like amdgcn-amd-amdhsa or nvptx64-nvidia-cuda), while the last flag takes device name (like gfx908 or sm_70) as input.

Alternatively, users of the ROCm compiler can simply use the flag -offload-arch for a combined effect of the preceding three flags.

Example:

# Legacy mechanism
clang -fopenmp -target x86_64-linux-gnu \
-fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa \
-march=gfx906 helloworld.c -o helloworld

Example:

# Using offload-arch flag
clang -fopenmp -target x86_64-linux-gnu \
--offload-arch=gfx906 helloworld.c -o helloworld.

To ensure backward compatibility, both styles are supported. This option is compatible with target ID support and multi-image fat binaries.

Target ID support for OpenMP#

The ROCm compiler supports specification of target features along with the GPU name while specifying a target offload device in the command line, using -march or --offload-arch options. The compiled image in such cases is specialized for a given configuration of device and target features (target ID).

Example:

# compiling for a gfx908 device with XNACK paging support turned ON
clang -fopenmp -target x86_64-linux-gnu \
-fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa \
-march=gfx908:xnack+ helloworld.c -o helloworld

Example:

# compiling for a gfx908 device with SRAMECC support turned OFF
clang -fopenmp -target x86_64-linux-gnu \
-fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa \
-march=gfx908:sramecc- helloworld.c -o helloworld

Example:

# compiling for a gfx908 device with SRAMECC support turned ON and XNACK paging support turned OFF
clang -fopenmp -target x86_64-linux-gnu \
-fopenmp-targets=amdgcn-amd-amdhsa -Xopenmp-target=amdgcn-amd-amdhsa \
-march=gfx908:sramecc+:xnack- helloworld.c -o helloworld

The target ID specified on the command line is passed to the clang driver using target-feature flag, to the LLVM optimizer and back end using -mattr flag, and to linker using -plugin-opt=-mattr flag. This feature is compatible with offload-arch command-line option and multi-image binaries for multiple architectures.

Multi-image fat binary for OpenMP#

The ROCm compiler is enhanced to generate binaries that can contain heterogenous images. This heterogeneity could be in terms of:

  • Images of different architectures, like AMD GCN and NVPTX

  • Images of same architectures but for different GPUs, like gfx906 and gfx908

  • Images of same architecture and same GPU but for different target features, like gfx908:xnack+ and gfx908:xnack-

An appropriate image is selected by the OpenMP device runtime for execution depending on the capability of the current system. This feature is compatible with target ID support and offload-arch command-line options and uses offload-arch tool to determine capability of the current system.

Example:

clang -fopenmp -target x86_64-linux-gnu \
-fopenmp-targets=amdgcn-amd-amdhsa,amdgcn-amd-amdhsa \
-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx906 \
-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx908 \
helloworld.c -o helloworld

Example:

clang -fopenmp -target x86_64-linux-gnu \
--offload-arch=gfx906 \
--offload-arch=gfx908 \
helloworld.c -o helloworld

Example:

clang -fopenmp -target x86_64-linux-gnu \
-fopenmp-targets=amdgcn-amd-amdhsa,amdgcn-amd-amdhsa,amdgcn-amd-amdhsa,amdgcn-amd-amdhsa \
-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx908:sramecc+:xnack+ \
-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx908:sramecc-:xnack+ \
-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx908:sramecc+:xnack- \
-Xopenmp-target=amdgcn-amd-amdhsa -march=gfx908:sramecc-:xnack- \
helloworld.c -o helloworld

ROCmCC compilers create an instance of toolchain for each unique combination of target triple and the target GPU (along with the associated target features).

Unified shared memory#

The following OpenMP pragma is available on MI200, and it must be executed with xnack+ support.

omp requires unified_shared_memory

For more details on unified shared memory, see OpenMP support.