Environment variables#

2026-10-29

5 min read time

Applies to Linux

FlashInfer on ROCm reads the variables on this page at import time or when an operation runs. Set a variable that is read at import before import flashinfer. In a shell, export it for the process that starts Python:

export FLASHINFER_LOGGING_LEVEL=DEBUG

Build-time variables, including FLASHINFER_ROCM_ARCH_LIST, PYTORCH_ROCM_ARCH, FLASHINFER_JIT_VERBOSE, FLASHINFER_EXTRA_LDFLAGS, FLASHINFER_EXTRA_CFLAGS, FLASHINFER_EXTRA_CUDAFLAGS, FLASHINFER_OWN_HEADERS_NON_SYSTEM, MAX_JOBS, and FLASHINFER_JIT_DEBUG, are documented in CONTRIBUTING.md.

Compilation and kernel cache#

These variables control torch.compile, JIT compilation, and the on-disk kernel cache.

Environment variable

Default

Description

FLASHINFER_USE_TORCH_CUSTOM_OPS

0

Set to 1 before importing flashinfer to wrap kernels as PyTorch custom operations so torch.compile can trace them. Requires PyTorch 2.4 or later and adds a small dispatch cost on each call. When the variable is unset or 0, torch.compile raises an error if a trace enters a FlashInfer operation.

FLASHINFER_WORKSPACE_BASE

$HOME

Parent directory of the JIT cache, .cache/flashinfer/. Point this at fast local disk when the home directory is on a network filesystem. Use an absolute path. A relative path is resolved from the working directory, and a leading ~ is treated as a directory name.

FLASHINFER_DISABLE_JIT

Unset

Any non-empty value disables JIT compilation, including the string 0. FlashInfer then loads only ahead-of-time kernels and kernels already in the JIT cache. It will not compile a missing kernel, and it will not load a module that exists only under ~/.cache/flashinfer. AITER shims are not shipped prebuilt, so a call that selects AITER raises MissingJITCacheError.

FLASHINFER_DISABLE_VERSION_CHECK

Unset

Any non-empty value skips the version check between FlashInfer and the JIT-cache package.

FLASHINFER_DISABLE_AOT_ARCH_CHECK

Unset

Any non-empty value, including 0, allows a JIT-cache kernel to run when its architecture does not match the current GPU. By default, a mismatch is discarded with a warning and the kernel is JIT-compiled for the current GPU.

FLASHINFER_TRACE_APPLY

Unset

Set to 1 to apply the kernel substitution mapping named by FLASHINFER_TRACE_APPLY_PATH at import. Only 0, false, False, and the empty string leave substitution off; every other value, including off and FALSE, enables it. If substitution cannot be enabled, FlashInfer logs a warning and continues to import.

AITER#

These variables control the AITER backend and the kernels it selects. For operation coverage, see the kernel support matrix.

Environment variable

Default

Description

FLASHINFER_AITER_STRICT

0

Set to 1 to raise when an AITER kernel fails to probe or initialize for backend="auto", instead of falling back to another supported path. It does not force an error for every page size: a page size that is not routed through AITER’s native paged kernel still takes the flat-gather path, which copies K and V on each run.

FLASHINFER_AITER_ASM_PREFILL

1

Selects the single-prefill implementation inside AITER. 0 uses the CK Tile kernel. Any other value uses the assembly kernel. The value is read once per process. Some assembly-kernel failures stop the process. Set this variable to 0 to stay on CK Tile.

FLASHINFER_AITER_ASM_VERBOSE

0

Any value other than 0 writes to standard error which single-prefill implementation ran, assembly or CK Tile. The two implementations agree numerically.

FLASHINFER_ARCH_ALLOW_KNOWN_BAD

0

Set to 1 to run an operation and backend that the capability table marks as broken on the current toolchain. This release does not mark any combination that way. Soft-capped causal prefill on gfx950 (MI350X and MI355X) keeps its own routing.

FLASHINFER_HIP_FUSED_CASCADE

0

Applies to MultiLevelCascadeAttentionWrapper only. Set to 1 before importing flashinfer to pass each level’s partial state into the next prefill call. The value is read once at import. AITER levels and the shared-prefix cascade wrappers do not use this variable.

AITER_JIT_DIR

AITER default

Directory of AITER shared libraries that the FlashInfer C++ shim loads. Overrides the path recorded when FlashInfer was built.

FLASHINFER_AITER_VARIANT_DIR

FlashInfer stores

List of prebuilt AITER variant directories, separated by os.pathsep. FlashInfer loads these stores before the cache and the JIT-cache stores. AITER_JIT_DIR is still tried first.

AITER_ASM_DIR

Set by import aiter

Directory of AITER assembly code objects. When the variable is unset or empty, single prefill stays on the CK Tile kernel.

GPU_ARCHS

Autodetected

GPU architecture list for AITER JIT compilation. If you leave it unset, a shim build may set it from FLASHINFER_ROCM_ARCH_LIST for that build. If you set it yourself, FlashInfer restores your value after the build.

Logging#

FLASHINFER_LOGGING_LEVEL controls the FlashInfer logger. FLASHINFER_LOGLEVEL and FLASHINFER_LOGDEST control a separate API-call log.

Environment variable

Default

Description

FLASHINFER_LOGGING_LEVEL

INFO

Logger verbosity. Typical values are DEBUG, INFO, and WARNING. This level covers AITER fallback messages and JIT build messages.

FLASHINFER_LOGLEVEL

0

API-call log detail. 1 logs function names. 3 adds inputs and outputs. 5 adds tensor statistics. 10 also writes tensors to disk.

FLASHINFER_LOGDEST

stdout

Destination of the API-call log: stdout, stderr, or a file path. In a file path, %i is replaced with the process ID.

ROCm installation#

Environment variable

Default

Description

ROCM_PATH or ROCM_HOME

/opt/rocm

ROCm installation prefix used by flashinfer.rocm.hip_utils. Set one of these only when ROCm is installed somewhere else.

Runtime helpers#

Two helpers report whether the current process can use AITER and whether the PyTorch build matches the ROCm installation. Importing flashinfer already runs the ROCm compatibility check.

is_aiter_supported returns True when the device is gfx942 (MI300X or MI325X) or gfx950 (MI350X or MI355X) and PyTorch is a ROCm build. It does not import the aiter package. Catch ImportError around import aiter when you also need to confirm that the package is installed.

check_torch_rocm_compatibility warns when the ROCm version reported by PyTorch differs from the ROCm version on the system. It raises RuntimeError when PyTorch has no ROCm build, and ImportError when PyTorch is not installed.

PyTorch addresses AMD GPUs with the device string "cuda".

import torch

from flashinfer.rocm.aiter_utils import is_aiter_supported
from flashinfer.rocm.hip_utils import check_torch_rocm_compatibility

device = torch.device("cuda")
aiter_ready = is_aiter_supported(device)
check_torch_rocm_compatibility()