Kernel authoring guide#

This guide covers writing GPU kernels with FlyDSL: the @flyc.jit and @flyc.kernel decorators, expression API, launch configuration, shared memory, and synchronization. It documents the @flyc.kernel/@flyc.jit API from flydsl.compiler and flydsl.expr (python/flydsl/).

Quick reference#

Concept

API

Description

JIT host func

@flyc.jit

Emit host-side launcher with JIT compilation

GPU kernel

@flyc.kernel

Define GPU kernel function

Launch

kernel(...).launch(grid=, block=)

Configure and emit GPU launch

Thread ID

fx.gpu.thread_idx.x

Get thread index in workgroup

Block ID

fx.gpu.block_idx.x

Get block/workgroup index

Block dim

fx.gpu.block_dim.x

Get block dimension size

Compile-time

fx.Constexpr[int]

Compile-time constant parameter

Tensor arg

fx.Tensor

GPU tensor argument (via DLPack)

Stream arg

fx.Stream

CUDA/HIP stream argument

Barrier

fx.gpu.barrier()

Workgroup synchronization

Constants

fx.Int32 / fx.Int64 / fx.Float32

Create typed DSL constants (fx.Int64 for index/offset values; fx.Index is deprecated)

Range loop

range_constexpr(n)

Compile-time unrolled loop

Buffer load

buffer_ops.buffer_load(rsrc, off)

AMD buffer load intrinsic

Scalar buffer load

fx.rocdl.s_buffer_load(rsrc, off, count=1)

Wave-uniform AMD buffer load; off is in i32 elements and the result is fx.Int32 (or a four-word fx.Vector with count=4)


1. Basic kernel pattern#

1.1 @flyc.kernel + @flyc.jit#

import flydsl.compiler as flyc
import flydsl.expr as fx
from flydsl.expr import gpu

@flyc.kernel
def vec_add_kernel(
    A: fx.Tensor,
    B: fx.Tensor,
    C: fx.Tensor,
    N: fx.Constexpr[int],
):
    tid = gpu.thread_idx.x
    bid = gpu.block_idx.x
    idx = bid * 256 + tid
    # ... kernel body using fx.*, ArithValue, Vector, and buffer ops ...

@flyc.jit
def vec_add(
    A: fx.Tensor,
    B: fx.Tensor,
    C: fx.Tensor,
    N: fx.Constexpr[int],
    stream: fx.Stream = fx.Stream(None),
):
    vec_add_kernel(A, B, C, N).launch(
        grid=(N // 256,),
        block=(256,),
        stream=stream,
    )

# Usage:
import torch
A = torch.randn(1024, device="cuda", dtype=torch.float32)
B = torch.randn(1024, device="cuda", dtype=torch.float32)
C = torch.empty(1024, device="cuda", dtype=torch.float32)

vec_add(A, B, C, 1024)

1.2 How it works#

  1. @flyc.kernel wraps the function as a KernelFunction

  2. @flyc.jit wraps the function as a JitFunction

  3. On first call, JitFunction.__call__ triggers:

    • AST rewriting (Python loops/ifs → MLIR scf ops)

    • MLIR module creation with gpu.container_module

    • Tracing the jit function body to generate MLIR ops

    • Calling vec_add_kernel(...) captures a pending kernel call

    • .launch() emits the specialized gpu.func and gpu.launch_func

    • MlirCompiler.compile() runs the full pass pipeline

    • CompiledArtifact stores the compiled module and lazily owns the resulting ExecutionEngine

  4. Subsequent calls trace again and reuse the cached binary when the generated MLIR and compilation configuration match


2. Parameter types#

2.1 fx.Tensor#

Maps a PyTorch tensor to an MLIR memref descriptor via DLPack:

@flyc.kernel
def my_kernel(input: fx.Tensor, output: fx.Tensor):
    # input and output are Tensor wrappers around ir.Value (memref)
    ...

At the host boundary, torch.Tensor is converted through the registered PyTorch/DLPack argument adapter.

2.2 fx.Constexpr[T]#

Compile-time constant. The value is embedded directly in the generated IR:

@flyc.kernel
def my_kernel(data: fx.Tensor, N: fx.Constexpr[int], dtype: fx.Constexpr[str]):
    for i in range_constexpr(N // 64):  # unrolled at compile time
        ...

Different Constexpr values produce different compiled kernels (separate cache entries).

2.3 fx.Int32#

Runtime integer parameter (passed as i32):

@flyc.jit
def launch(data: fx.Tensor, size: fx.Int32, stream: fx.Stream = fx.Stream(None)):
    ...

Python int values are converted to Int32 via the JitArgumentRegistry.

2.4 fx.Stream#

CUDA/HIP stream for asynchronous kernel launch:

@flyc.jit
def launch(data: fx.Tensor, stream: fx.Stream = fx.Stream(None)):
    my_kernel(data).launch(grid=(1,), block=(256,), stream=stream)

# Launch on specific stream:
stream = torch.cuda.Stream()
launch(data, stream=fx.Stream(stream))

2.5 Custom argument types#

Register a raw Python type with a JitArgument adapter and the DslType that will appear inside the traced function:

from flydsl.compiler import JitArgumentRegistry

@JitArgumentRegistry.register(MyCustomType, dsl_type=MyDslType)
class MyCustomAdaptor:
    def __init__(self, value: MyCustomType):
        self.value = value

    def __get_ir_types__(self):
        return [...]  # MLIR types for this argument

    def __cache_signature__(self):  # optional; only cache_signature() reads it
        return (...)  # every property that can change generated code

    def __c_abi_spec__(self):
        return [...]  # ordered (ctypes storage type, fill(argument, storage)) slots

The adapter’s C-ABI slots may outnumber its MLIR types (for example, a dynamic memref has data and layout slots).


3. Thread / block hierarchy#

from flydsl.expr import gpu

# Thread index within workgroup (returns Int32)
tid_x = gpu.thread_idx.x
tid_y = gpu.thread_idx.y
tid_z = gpu.thread_idx.z

# Block (workgroup) index within grid
bid_x = gpu.block_idx.x
bid_y = gpu.block_idx.y

# Block dimensions
bdim_x = gpu.block_dim.x

# Grid dimensions
gdim_x = gpu.grid_dim.x

# Low-level (returns raw ir.Value)
raw_tid = gpu.thread_id("x")
raw_bid = gpu.block_id("x")

4. Expression API (flydsl.expr)#

4.1 Arithmetic and numeric types#

import flydsl.expr as fx

# Constants (prefer DSL numeric types)
c42 = fx.Int64(42)          # 64-bit integer constant (prefer over the deprecated fx.Index)
c3_14 = fx.Float32(3.14)    # f32 constant
mask = fx.Int32(0xFF)       # i32 constant

# Arithmetic (operator overloading via ArithValue / Numeric)
result = a + b
result = a * 2
result = a // 4
result = a % 16
largest = fx.max(a, b)
smallest = fx.min(a, b)
tiles = fx.ceildiv(count, tile_size)

# Cast (prefer DSL numeric constructors)
i64_val = fx.Int64(int_val) # cast to 64-bit integer (fx.Index is deprecated)
i32_val = fx.Int32(i64_val) # cast to i32

# Select
result = cond.select(true_val, false_val)  # when cond is an ArithValue

# Bitwise
result = a & b
result = a ^ b
result = a << 4

Use direct arith.*FOp(..., fastmath=...) only where explicit fastmath flags are performance-critical. Use fx.max / fx.min for type-dispatched extrema and fx.ceildiv for overflow-safe integer ceil division. fx.maxnumf / fx.minnumf intentionally retain non-NaN-wins semantics, and fx.ceil_div remains the layout/int-tuple API.

4.2 Vector values (Vector)#

from flydsl.expr.typing import Vector as Vec

# Build vector from elements
vec = Vec.from_elements([a, b, c, d], fx.Float32)

# Vector store to memref
vec.store(memref, [idx])

# Extract, bitcast, and convert
elem = vec[idx]
as_i32 = vec.bitcast(fx.Int32)
as_bf16 = vec.to(fx.BFloat16)

4.3 ROCm intrinsics (fx.rocdl)#

High-level helpers#

from flydsl.expr import rocdl

# Buffer tensor — wraps a Tensor with AMD buffer resource descriptor
A_buf = rocdl.make_buffer_tensor(A)

# RDNA bounded buffer — supplying the byte extent enables hardware OOB checks
A_buf = rocdl.make_buffer_tensor(
    A,
    num_records_bytes=problem_size * element_bytes,
)

# MFMA MMA atom constructor (CDNA3/CDNA4) — returns MmaAtomCDNA3_MFMAType
atom_type = rocdl.MFMA(m=16, n=16, k=32, elem_ty_ab=fx.Float8E4M3FNUZ)

# Buffer copy atom types
copy_op = rocdl.BufferCopy128b()   # 128-bit buffer copy
copy_op = rocdl.BufferCopy64b()    # 64-bit buffer copy
copy_op = rocdl.BufferCopy32b()    # 32-bit buffer copy

On RDNA, supplying num_records_bytes selects the raw-buffer descriptor mode where out-of-range reads return zero and writes are suppressed. The byte count may be dynamic. Omitting it keeps the default unchecked descriptor; passing max_size=False derives the count from the tensor layout and enables checking.

See gfx1250 WMMA & TDM atoms below for the gfx1250 WMMA (incl. MX-scaled) MMA atoms and the TDM async copy atom.

MFMA instructions#

Signature: (result_type, [a, b, c, cbsz, abid, blgp]) — trailing ints default to 0.

result = rocdl.mfma_f32_16x16x16f16(result_type, [a, b, acc])
result = rocdl.mfma_f32_16x16x32_fp8_fp8(result_type, [a, b, acc])
result = rocdl.mfma_i32_16x16x32_i8(result_type, [a, b, acc])
result = rocdl.mfma_f32_16x16x16bf16_1k(result_type, [a, b, acc])   # BF16 1K variant

# GFX950 scaled MFMA (MXFP4/FP6/FP8)
result = rocdl.mfma_scale_f32_16x16x128_f8f6f4(
    result_type, [a, b, acc, cbsz, blgp, opselA, scaleA, opselB, scaleB]
)

Instruction scheduling barriers#

Control instruction scheduling for performance tuning:

rocdl.sched_mfma(cnt)    # wait for cnt MFMA instructions to complete
rocdl.sched_vmem(cnt)    # wait for cnt VMEM reads to complete
rocdl.sched_dsrd(cnt)    # wait for cnt DS (LDS) reads to complete
rocdl.sched_dswr(cnt)    # wait for cnt DS (LDS) writes to complete

Math intrinsics#

Single-instruction hardware math (guaranteed 1 VALU cycle, lower precision than math.*):

# Base-2 exponential (v_exp_f32)
result = rocdl.exp2(T.f32, x)

# Reciprocal (v_rcp_f32)
result = rocdl.rcp(T.f32, x)

Generated upstream ROCDL builders and repository-only compatibility helpers are not part of the stable expression API and are intentionally omitted here. New kernels should prefer rocdl.make_buffer_tensor() plus fx.copy() or typed Tensor/Pointer access.

gfx1250 WMMA & TDM atoms (wave32)#

gfx1250 kernels run wave32 and use WMMA (not MFMA) for matrix math and the TDM (Tensor Data Mover) engine for async whole-tile Global↔LDS copies. All of the factories below live in flydsl.expr.rocdl (from flydsl.expr import rocdl).

WMMA MMA atom — rocdl.WMMA(m, n, k, elem_ty_ab, elem_ty_acc=None, **kwargs) is arch-dispatched (gfx11 v16 ABI; gfx12 / gfx1250 v8 ABI). On gfx1250 it builds MmaOpGFX1250_WMMAType (M=N=16). Supported K / dtypes:

dtype (A,B → Acc)

K

notes

f32 → f32

4

f16/bf16 → f32 or same

32

fp8/bf8 (E4M3FN / E5M2, any mix) → f32 or f16

64, 128

native OCP fp8

i8 → i32

64

sign_a / sign_b / clamp kwargs

i4 → i32

32

sign_a / sign_b / clamp kwargs

mma = fx.make_mma_atom(rocdl.WMMA(16, 16, 32, fx.Float16))          # f16 → f32
mma = fx.make_mma_atom(rocdl.WMMA(16, 16, 128, fx.Float8E4M3FN))    # fp8 → f32
# signed int4 with accumulator clamp:
mma = fx.make_mma_atom(rocdl.WMMA(16, 16, 32, T.i4, T.i32, sign_a=True, sign_b=True, clamp=True))

On RDNA4 (gfx1200 / gfx1201) the same factory builds MmaOpGFX120X_WMMAType instead. RDNA4 shares the v8 register ABI, while its supported floating-point forms use 16x16x16: f16/bf16 and every fp8(E4M3FN)/bf8(E5M2) A/B combination accumulating to f32. Pass a different B type with the keyword-only elem_ty_b; it defaults to the A type. The 16x16x32 BF16 and fp8 K=64/128 forms above are gfx1250-only and are rejected by the atom’s verifier. See kernels/gemm/rdna_f16_gemm.py for a full pipelined f16 example and kernels/gemm/rdna_fp8_preshuffle_gemm.py for an RDNA4 FP8 implementation.

mma = fx.make_mma_atom(rocdl.WMMA(16, 16, 16, fx.BFloat16, fx.Float32))  # RDNA4
mma = fx.make_mma_atom(rocdl.WMMA(16, 16, 16, fx.Float8E4M3FN, fx.Float32))  # RDNA4 FP8
mma = fx.make_mma_atom(rocdl.WMMA(16, 16, 16, fx.Float8E4M3FN, fx.Float32,
                                  elem_ty_b=fx.Float8E5M2))  # RDNA4 FP8 x BF8

MX-scaled WMMA — rocdl.WMMAScale(m, n, k, elem_ty_a, elem_ty_b=None, elem_ty_acc=None, *, opsel_a=0, opsel_b=0, mod_c=0, reuse_a=False, reuse_b=False, block_size=32) builds the E8M0 block-scaled WMMA (V_WMMA_SCALE / V_WMMA_SCALE16) for the unified f8/f6/f4 operand format. Shapes: 16x16x128 (f8/f6/f4) and 32x16x128 (fp4-only). Per-operand E8M0 scales are atom state (scale_a / scale_b); block_size 32 → i32 scale state, 16 → i64.

mma = fx.make_mma_atom(rocdl.WMMAScale(16, 16, 128, fx.Float8E4M3FN))
mma = fx.atom_set_value(mma, "scale_a", fx.Int32(scale_a))   # E8M0 scales
mma = fx.atom_set_value(mma, "scale_b", fx.Int32(scale_b))
fx.gemm(mma, frag_C, frag_A, frag_B, frag_C)

For per-tile scales, pass each operand as a list or tuple containing its data and scale fragments:

fx.gemm(tiled_mma, frag_D, [frag_A, scales_A], [frag_B, scales_B], frag_C)

If A has shape (V, M, K), its scale fragment has shape (1, M, K); B uses (1, N, K). Rank-2 operands omit the K mode. Scales use i32 for CDNA4 MFMA and block-32 WMMA, or i64 for block-16 WMMA. Use zero strides in tile modes to broadcast a scale. The compiler slices all tensors in each operand group together and fully expands the static M/N/K dimensions. All operands in each group must be Tensors.

The first tensor is always the primary operand. Further tensors are defined by the MMA atom; Single Tensor operands and scalar atom-state keywords remain supported for both bare atoms and tiled MMA. tiled_mma.set_value("scale_a", scale) returns a new tiled MMA with the same layout and permutation. Explicit scale tensors override the corresponding atom-state fields; omitted scale tensors use those fields. The current scaled atoms accept data and an optional scale; they do not consume sparsity metadata.

Tiled copies support the same state update API: tiled_copy.set_value("soffset", offset) returns a new TiledCopy with its original tile and thread/value layout. Passing it to fx.copy uses the updated atom state. Both tiled types also accept a dictionary of fields.

Use atom_callback to select a different atom for each static tile, for example to choose a byte from packed CDNA4 scale words explicitly:

fx.gemm(
    tiled_mma, frag_D, [frag_A, scales_A], [frag_B, scales_B], frag_C,
    atom_callback=lambda atom, mnk: fx.make_mma_atom(
        fx.rocdl.cdna4.MFMA_Scale(
            16, 16, 128, fx.Float8E4M3FN,
            opsel_a=mnk[0] % 2, opsel_b=mnk[1] % 4,
        )
    ),
)

When a callback is present, the callback receives the original atom with its runtime state and a tuple of integer tile indices (m, n, k), and returns the atom for that tile. Rank-2 operands use k=0; rank-1 calls use (0, 0, 0).

TDM tiled copy atom (preferred) — build the atom over the whole global tensor with rocdl.cdna5.make_tiled_tdm_atom; it returns the atom plus a coordinate tensor. fx.zipped_divide picks this block’s tile and rocdl.cdna5.tdm_partition cuts the LDS tile and the coordinate tile into one warp’s share. Padding is row slack in the LDS box layout, out-of-bounds comes from the tensor’s own extent plus boundary_check, and the K-loop advances by indexing the coordinate rest with an i32 tile number:

lds  = fx.SharedAllocator().allocate(fx.Array[fx.Float16, TM * TK]).peek()
box  = fx.make_layout((TM, TK), (TK, 1))          # row slack here IS the padding
smem = fx.make_view(lds.ptr, fx.make_layout(((TM, TK), 1), ((TK, 1), TM * TK)))
gA   = fx.make_view(fx.get_iter(A), fx.make_layout((M, K), (K, 1)))  # FULL tensor
atom, coord = rocdl.cdna5.make_tiled_tdm_atom(rocdl.TensorLoad(), gA, box, (TM, TK))
blk  = fx.zipped_divide(coord, (TM, TK))[None, (fx.block_idx.x, None)]
tAs, tAg = rocdl.cdna5.tdm_partition(atom, 0, fx.make_layout(1, 1), smem, blk)
fx.copy(atom, tAg[None, kt], tAs[None, 0])        # kt: i32 tile index, NOT a byte offset
rocdl.s_wait_tensorcnt(0)

For a store pass rocdl.TensorStore(), flip the fx.copy operands, and keep the LDS box packed — a store cannot de-pad LDS.

Older TDM atom — rocdl.make_tdm_atom still exists and still lowers, but it is not what new code should reach for. It takes the extents, strides and padding as explicit arguments and bumps the K loop with a byte imm_offset; the tiled atom reads all of that off the tensors. The in-tree gfx1250 kernels are on the tiled atom, and the only copies left on the old one are those it cannot express: a non-contiguous innermost dim, and gather/scatter (make_tensor_gather_descriptor / tensor_load_gather). See docs/api/dsl.rst for its signature if you have to read one of those.

4.4 GPU operations (fx.gpu)#

from flydsl.expr import gpu

# Barrier (workgroup synchronization)
gpu.barrier()

# Shared memory address space attribute
addrspace = gpu.smem_space()
addrspace_int = gpu.smem_space(int=True)

5. Control flow#

5.1 Python loops#

The ASTRewriter transforms Python for loops:

@flyc.kernel
def my_kernel(data: fx.Tensor, N: fx.Constexpr[int]):
    # Compile-time unrolled loop
    for i in range_constexpr(N):
        # This loop is fully unrolled in the generated IR
        ...

    # Runtime loop (lowered by the AST rewriter)
    for i in range(runtime_value):
        ...

fx.range is the runtime loop with keyword-only directives:

for i in fx.range(runtime_value, unroll=4):  # LLVM unrolls by 4
    ...
for k in fx.range(N, unroll_full=True):  # LLVM unrolls completely
    ...

5.2 const_expr()#

Mark a value as compile-time constant:

from flydsl.expr import const_expr

@flyc.kernel
def my_kernel(data: fx.Tensor, N: fx.Constexpr[int]):
    tile_size = const_expr(N // 4)
    for i in range_constexpr(tile_size):
        ...

5.3 if and return#

The ASTRewriter also rewrites if statements. What a branch becomes depends on the value of the condition at trace time:

  • A Python value, for example a Constexpr argument or const_expr(...): only the branch its truth value picks is traced, as in plain Python.

  • A dynamic value, such as a comparison of fx.Int32 values: both branches are traced into one scf.if.

A return inside a branch behaves as in Python when the condition is a Python value. When the condition is dynamic, a return that tracing reaches raises NotImplementedError, because an scf.if region cannot return from the enclosing function. The same holds for a return inside the body of a for ... in range(...) loop, which always becomes an scf.for, and of a dynamic while loop; a range(..., init=...) loop cannot contain a return either. Assign the value instead and return it after the branch or loop, and write an early exit as a branch around the code that follows:

def clamp(x, limit):  # a helper inside a @flyc.kernel / @flyc.jit body
    out = x
    if x > limit:  # dynamic condition: traced into one scf.if
        out = limit
    return out


@flyc.kernel
def scale(data: fx.Tensor, n: fx.Int32):
    tid = fx.thread_idx.x
    if tid < n:  # not `if tid >= n: return`
        ...

6. Shared memory (LDS)#

6.1 fx.SharedAllocator + @fx.struct#

New kernels declare their LDS layout as a @fx.struct storage type and allocate it with fx.SharedAllocator (from flydsl.expr.gpu, reached as fx.SharedAllocator) inside the kernel body. Each field is an fx.Array[elem, count, align]; .allocate(StorageStruct).peek() returns a handle whose fields expose typed views:

import flydsl.expr as fx

# Declare the LDS layout. Fields may be conditional on compile-time config.
@fx.struct
class SharedStorage:
    s_red: fx.Array[fx.Float32, red_slots, 16]   # red_slots elems, 16B aligned
    s_red2: fx.Array[fx.Float32, red_slots, 16]

@flyc.kernel
def my_kernel(...):
    # Allocate the storage struct in LDS (inside the @kernel body).
    lds = fx.SharedAllocator().allocate(SharedStorage).peek()

    # Get a logical view over each field and use it with the layout API.
    s_red = lds.s_red.view(fx.make_layout(red_slots, 1))
    s_red2 = lds.s_red2.view(fx.make_layout(red_slots, 1))

By default SharedAllocator is static=True: each leaf emits a per-leaf static LDS global that the compiler sizes, so launch(smem=...) is left unset. Use static=False (dynamic) mode to have the launch wrapper auto-infer smem from SharedAllocator.allocated_bytes when smem=None (an explicit smem must be >= that size). See kernels/gemm/preshuffle_gemm.py and kernels/norm/rmsnorm_kernel.py for real usage.

6.2 LDS capacity#

Architecture

LDS per CU

gfx942 (MI300X)

64 KB

gfx950 (MI350/MI355X)

160 KB

gfx1201 (Radeon AI PRO R9700)

64 KB

gfx1250

320 KB


7. Launch configuration#

7.1 KernelLauncher.launch()#

@flyc.jit
def launch(data: fx.Tensor, stream: fx.Stream = fx.Stream(None)):
    my_kernel(data).launch(
        grid=(num_blocks_x, num_blocks_y, num_blocks_z),
        block=(threads_x, threads_y, threads_z),
        smem=shared_mem_bytes,     # dynamic shared memory
        stream=stream,             # CUDA/HIP stream
    )

Grid and block dimensions accept:

  • int — static value

  • ir.Value — dynamic MLIR value

  • Tuple of 1–3 values — missing dimensions default to 1

7.2 Dynamic grid/block dimensions#

@flyc.jit
def launch(data: fx.Tensor, M: fx.Int32, stream: fx.Stream = fx.Stream(None)):
    grid_x = M // 256
    my_kernel(data, M).launch(
        grid=(grid_x, 1, 1),
        block=(256, 1, 1),
        stream=stream,
    )

8. Synchronization#

from flydsl.expr import gpu

# Workgroup barrier (s_barrier)
gpu.barrier()

9. Compilation and caching#

9.1 Automatic caching#

Every ordinary @flyc.jit call and every new flyc.compile(...) request traces the current Python launcher and its kernels before checking compiled artifacts. An unchanged trace reuses the artifact and skips lowering and code generation. Tracing still executes Python code, so its side effects and cost occur on cache hits too.

The artifact key is a SHA-256 over the complete prepared MLIR module, target and backend fingerprint, pass pipeline, LLVM options and codegen environment variables. The module includes host launches, all GPU kernels, constants and attributes. Normal builds omit source locations; debug builds include them. Disk entries additionally hash the compiler toolchain files, which cannot change inside a running process. Python argument values never select an artifact on their own; each traced call rebuilds its argument adapter.

The disk store holds one <key>.pkl per artifact in ~/.flydsl/cache/<function name>/, or under $FLYDSL_RUNTIME_CACHE_DIR. FLYDSL_RUNTIME_ENABLE_CACHE=0 disables disk caching; in-memory artifact reuse remains active and still requires a fresh trace.

9.2 Changes to Python code and compile-time state#

A helper rebind, changed global or closure value, mutable captured object, or different self instance is observed by the next ordinary JIT/compile request. If it changes the generated MLIR, the request selects a new artifact. This also works for helpers in other directories and calls inside nested Python functions; there is no Python dependency-source walker. Changes that produce identical MLIR and configuration can reuse the same artifact.

Tracing executes the currently loaded Python objects. Editing a file requires reloading/redefining its functions or starting a new process. A helper’s own memoization can still return an old value. Restart after rebuilding FlyDSL or replacing compiler libraries: compiler-file fingerprints are cached within a process and loaded native code is not hot-reloaded.

The toolchain fingerprint of disk entries hashes FlyDSL’s Python sources at run time and takes the native libraries’ SHA-256 from _mlir/_mlir_libs/_native_digest.py, which every build regenerates after the libraries (wheels ship it as built). If that file is missing or lists other files, the libraries are hashed at run time with a warning. Change libraries only by rebuilding: a library replaced by hand under its old name keeps the old digest.

Artifacts using external libraries or Python post-load initializers are reused only in-process and never written to disk. Post-load callbacks are absent from MLIR, so the in-process key also includes their ordered identities. Python bound methods use the receiver and function identities. Different callbacks get separate loaded modules; the same callbacks reuse a module and run only once per module. Provide a new callback object when its initialization configuration changes: mutating the same callback’s captured state does not reinitialize a cached module. Rebuilding a linked library in place requires a new process. Ktrace buffer configuration enters the key. Event IDs are assigned at compilation from this process’s name table and stay out of the key; a cached artifact whose IDs conflict with a live binary is rebuilt.

9.3 Fixed compiled callables#

For repeated launches without frontend tracing, keep the CompiledFunction returned by flyc.compile(...):

compiled = flyc.compile(launch, *specialization_args)
compiled(*runtime_args)

The returned callable retains the selected specialization. Its calls do not retrace or detect changes to Python helpers/captures; request a new callable with flyc.compile(...) when those changes should take effect. Arguments must remain compatible with that specialization. With live device arguments, the initial flyc.compile(...) request also launches once; compile placeholders defer launch.

FlyDSL does not record Python dependencies or add specialization guards as part of artifact caching.

Anything that launches the same specialization repeatedly — benchmark loops, per-step runtime wrappers, graph-capture setup — must hold a CompiledFunction instead of calling the @flyc.jit function, or it pays a full trace per launch and benchmarks measure tracing. _run_compiled in kernels/common/tensor_shim.py keeps one handle per launcher object, which is valid when the launcher’s factory fixes every compile-time input. @autotune times a @jit function through one handle per config; a plain callable it wraps must dispatch through handles itself.

When one launcher serves arguments that specialize differently (dtype, rank, static layout), key the handles with cache_signature of each argument’s registered JitArgument adapter:

from flydsl.compiler import JitArgumentRegistry
from flydsl.compiler.protocol import cache_signature

handles = {}

def signature(value):
    adapter, _ = JitArgumentRegistry.get(type(value))
    return cache_signature(adapter(value) if adapter else value)

def run(tile, *runtime_args):
    key = (tile, *map(signature, runtime_args))
    compiled = handles.get(key)
    if compiled is None:
        handles[key] = flyc.compile(launch, tile, *runtime_args)  # Compiles and launches once.
    else:
        compiled(tile, *runtime_args)

A hit costs the signatures plus the handle call, tens of microseconds. Constexpr values have no runtime slot, so put them into the key yourself, as tile above. Changes to captured Python state are not detected; start a new cache when they should take effect.


10. Debugging#

10.1 Dumping IR#

FLYDSL_DUMP_IR=1 FLYDSL_DUMP_DIR=./my_dumps python my_script.py

Dumping bypasses the disk store, so each process dumps its first compilation of every artifact; later requests in the process reuse the in-memory artifact.

10.2 Printing IR#

Use the supported debug controls to print IR during compilation:

FLYDSL_DEBUG_PRINT_ORIGIN_IR=1 python my_script.py
FLYDSL_DEBUG_PRINT_AFTER_ALL=1 python my_script.py

The internal CompiledArtifact object also stores compiled and source IR, but it is not a public return value of @flyc.jit or flyc.compile.

10.3 AST diff#

FLYDSL_DEBUG_AST_DIFF=1 python my_script.py

Shows the diff between original and rewritten AST for debugging control flow transformations.


11. Complete example: Preshuffle GEMM#

From kernels/gemm/preshuffle_gemm.py:

import flydsl.compiler as flyc
import flydsl.expr as fx
from flydsl.expr import gpu, range_constexpr, rocdl
from flydsl.expr.typing import T

def compile_preshuffle_gemm(*, N, K, tile_m, tile_n, tile_k,
                             in_dtype="fp8", out_dtype="bf16",
                             epilogue="none", lds_stage=2, ...):
    a_lds_elems = tile_m * tile_k

    # Declare the LDS layout as a storage struct.
    @fx.struct
    class SharedStorage:
        a0: fx.Array[layout_elem, a_lds_elems, 16]
        if lds_stage == 2:
            a1: fx.Array[layout_elem, a_lds_elems, 16]

    @flyc.kernel
    def kernel_gemm(
        arg_c: fx.Tensor, arg_a: fx.Tensor, arg_b: fx.Tensor,
        arg_scale_a: fx.Tensor, arg_scale_b: fx.Tensor, arg_bias: fx.Tensor,
        i32_m: fx.Int32, i32_n: fx.Int32,
        tiled_mma_arg: fx.TiledMma, tiled_copy_g2s: fx.TiledCopy,
    ):
        tid = fx.thread_idx.x
        bid_x, bid_y, _ = fx.block_idx

        gA = fx.rocdl.make_buffer_tensor(arg_a, ...)
        gB = fx.rocdl.make_buffer_tensor(arg_b)
        gC = fx.rocdl.make_buffer_tensor(arg_c, ...)

        # Allocate LDS and take typed views over the storage fields.
        lds = fx.SharedAllocator().allocate(SharedStorage).peek()
        # ... GEMM implementation using MFMA, LDS, tiling ...

    @flyc.jit
    def launch_fn(
        arg_c: fx.Tensor, arg_a: fx.Tensor, arg_b: fx.Tensor,
        arg_scale_a: fx.Tensor, arg_scale_b: fx.Tensor, arg_bias: fx.Tensor,
        M_val: fx.Int32, N_val: fx.Int32,
        stream: fx.Stream = fx.Stream(None),
    ):
        kernel_gemm(arg_c, arg_a, arg_b, arg_scale_a, arg_scale_b, arg_bias,
                    M_val, N_val, tiled_mma, tiled_copy_g2s).launch(
            grid=(grid_x, grid_y), block=(256,), stream=stream,
        )

    return launch_fn

M is a runtime argument (i32_m / M_val), not a compile-time parameter, so one compiled kernel serves any M. Select output post-processing with epilogue= ("none", "bias", "bias_relu", "bias_silu", "bias_gelu").


12. Decision tree#

Writing a new kernel?
│
├── Simple element-wise?
│   ├── Use @flyc.kernel + @flyc.jit
│   ├── fx.gpu.thread_idx.x for thread indexing
│   └── See tests/kernels/test_vec_add.py
│
├── Reduction (norm, softmax)?
│   ├── Reductions are inline (wave/block reduce helpers within the kernel)
│   └── See kernels/norm/rmsnorm_kernel.py, kernels/norm/softmax_kernel.py
│
├── Matrix multiply (GEMM)?
│   ├── Use @flyc.kernel + fx.SharedAllocator + MFMA
│   ├── B-preshuffle layout from kernels/common/mma/mfma_preshuffle_pipeline.py
│   └── See kernels/gemm/preshuffle_gemm.py
│
├── Need shared memory?
│   ├── Declare a @fx.struct storage layout
│   ├── Allocate with fx.SharedAllocator().allocate(Storage).peek()
│   └── Take .view(...) over each field inside @kernel
│
└── Need compile-time specialization?
    ├── Use Constexpr[T] parameters
    └── Use range_constexpr() for unrolled loops

13. Source files#

File

Description

python/flydsl/compiler/__init__.py

Public API: jit, kernel, from_dlpack

python/flydsl/compiler/jit_function.py

@jit decorator, MlirCompiler, JitCacheManager

python/flydsl/compiler/kernel_function.py

@kernel decorator, KernelFunction, KernelLauncher

python/flydsl/compiler/jit_executor.py

CompiledArtifact (compiled IR and lazy ExecutionEngine wrapper)

python/flydsl/compiler/jit_argument.py

JitArgumentRegistry and tensor/pointer argument adapters

python/flydsl/compiler/ast_rewriter.py

ASTRewriter — Python AST → MLIR control flow

python/flydsl/expr/typing.py

Types (T), Tensor, Stream, Constexpr

python/flydsl/expr/arith.py

Arithmetic operations

python/flydsl/expr/gpu.py

GPU operations (thread indices, barriers, …)

python/flydsl/expr/rocdl/

ROCm dialect intrinsics (MFMA/WMMA, buffer, TDM, cluster)

python/flydsl/expr/primitive.py

Layout algebra primitives (make_shape, crd2idx, etc.)

python/flydsl/expr/gpu.py

SharedAllocator, GPU ops (thread indices, barriers, …)

kernels/gemm/preshuffle_gemm.py

Preshuffle GEMM kernel example

tests/kernels/test_vec_add.py

Vector add kernel test

tests/kernels/test_preshuffle_gemm.py

Preshuffle GEMM test