Using hipFORT with OpenMP offload#
Applications built with LLVM Flang (amdflang in ROCm) and OpenMP offload
often manage GPU data with map clauses instead of hipMalloc and
hipMemcpy. hipFORT routines can work directly on these mapped arrays. On a
GPU without unified memory, the library needs the device address of each array,
not its host address. The use_device_addr clause provides it.
The example on this page is adapted from one by Lynd Stringer.
The pattern#
The following program computes y := alpha * x + y three times with
hipblasSaxpy on arrays mapped to the device by OpenMP: twice with
use_device_addr, once through each interface, and once with
use_device_ptr:
program test_hipblas_saxpy_device_addr
use iso_c_binding
use hipfort
use hipfort_hipblas
use hipfort_check
implicit none
integer(c_int), parameter :: n = 1000
real(c_float), target :: alpha = 2.0
real(c_float), allocatable, target :: x(:), y(:)
type(c_ptr) :: handle = c_null_ptr
integer :: i
allocate(x(n), y(n))
x = [(real(i, c_float), i = 1, n)]
y = 10.0
call hipblasCheck(hipblasCreate(handle))
call saxpy_twice(n, x, y)
call saxpy_device_ptr()
call hipblasCheck(hipblasDestroy(handle))
! After three saxpy calls, y = 3*alpha*x + 10.
if (any(abs(y - (3*alpha*x + 10.0)) > 1.0e-5*abs(y))) stop 1
print *, "PASS"
contains
! Explicit-shape dummies have no descriptor, so use_device_addr only
! replaces their base addresses.
subroutine saxpy_twice(n, x, y)
integer(c_int) :: n
real(c_float), target :: x(n), y(n)
! Copy x and y to the device; y is copied back at the outer end.
!$omp target data map(to: x) map(tofrom: y)
! In this region, x and y refer to their device copies.
!$omp target data use_device_addr(x, y)
! Array overload: y = alpha*x + y.
call hipblasCheck(hipblasSaxpy(handle, n, alpha, x, 1, y, 1))
! Raw interface (Fortran 2003): pass the addresses with c_loc.
call hipblasCheck(hipblasSaxpy(handle, n, c_loc(alpha), c_loc(x), 1, &
c_loc(y), 1))
!$omp end target data
! hipBLAS is asynchronous: wait for it before y is copied back.
call hipCheck(hipDeviceSynchronize())
!$omp end target data
end subroutine saxpy_twice
! The allocatables themselves, without use_device_addr: use_device_ptr
! replaces C pointers to the mapped arrays by their device addresses.
subroutine saxpy_device_ptr()
type(c_ptr) :: px, py
px = c_loc(x)
py = c_loc(y)
!$omp target data map(to: x) map(tofrom: y)
!$omp target data use_device_ptr(px, py)
call hipblasCheck(hipblasSaxpy(handle, n, c_loc(alpha), px, 1, py, 1))
!$omp end target data
call hipCheck(hipDeviceSynchronize())
!$omp end target data
end subroutine saxpy_device_ptr
end program test_hipblas_saxpy_device_addr
The outer target data region copies x and y to the device and
copies y back to the host when it ends. In the inner region,
use_device_addr(x, y) makes x and y refer to their device copies,
so hipBLAS reads and writes device memory. The two regions can also be
written as one construct,
!$omp target data map(to: x) map(tofrom: y) use_device_addr(x, y).
Keep three points in mind:
Synchronize before the outer
end target data. hipBLAS routines are asynchronous: they return once the work is queued. Without a call tohipDeviceSynchronize(or a synchronization of the handle’s stream), the outerend target datacan copyyback before the GPU has written it. The example waits for the whole device for brevity; an application usually waits only for the handle’s stream (hipblasGetStream, thenhipStreamSynchronize).alphastays on the host. Only arrays go inuse_device_addr. The handle uses the default host pointer mode, so hipBLAS readsalphafrom host memory. To pass a device scalar instead, see Fortran interface variants.One generic, two interfaces. The first call uses the array overload:
xandyare passed as Fortran arrays and hipFORT takes their addresses. The second call uses the raw interface, which follows the C API and needs only Fortran 2003: insideuse_device_addr,c_loc(x)is the device address, passed as atype(c_ptr). Both are specific procedures of the generichipblasSaxpy.
The offload is done in a subroutine whose dummy arguments x(n) and
y(n) are explicit-shape, because of an amdflang limitation. An
allocatable or assumed-shape array carries a descriptor. With the amdflang
versions tested (ROCm 10.1 and 10.2), use_device_addr on such an array
maps the descriptor too, and the host then reads it from device memory, which
crashes on a discrete GPU. Explicit-shape dummies have no descriptor, so they
avoid the problem. The third call shows the other way, which works on the
allocatables themselves: use_device_ptr on type(c_ptr) variables set to
c_loc(x) and c_loc(y) before the region, passed to the raw interface.
Build the program with the OpenMP offload flags, for example
amdflang -fopenmp --offload-arch=gfx90a. With CMake, link it against
hipfort::hipblas; without CMake, Building your application without CMake lists the
options and the libraries to link. The hipFORT test suite builds and runs this example when
the Fortran compiler is LLVM Flang with OpenMP offload support; see
test/openmp/README.openmp.
If you link with -Wl,--gc-sections (see Linking only the routines you call), also
link with -Wl,-z,nostart-stop-gc. amdflang links with ld.lld, which
by default lets -Wl,--gc-sections remove the table of offload kernels, so
the program aborts at its first target construct.
Unified memory#
On a system with unified memory, such as the MI300A, a program built with
-fopenmp-force-usm can pass host arrays to the library directly. The
target data regions and use_device_addr are then not needed, but the
hipDeviceSynchronize call before reading the results on the host still is.
Run such a program with XNACK on (HSA_XNACK=1): with XNACK off, the GPU
cannot access host memory on demand, and the first library call faults. The
same holds for a program that passes host allocations to a library without
-fopenmp-force-usm. The test/openmp/<library>/test_*_host.f90 programs
of the hipFORT test suite do exactly that, one per library, and run only in the
XNACK matrix below.
To run the test suite in both XNACK modes, configure it with
-DBUILD_TESTING=ON -DHIPFORT_XNACK_MATRIX=ON. Every test then also runs with
HSA_XNACK=1, and with LLVM Flang, each OpenMP offload test also runs as a
-fopenmp-force-usm build. The option is off by default.