.. meta::
  :description: The hipThreads execution model
  :keywords: hipThreads, ROCm, scheduler, persistent kernel, fiber, width, yield

.. _execution-model:

**********************************
hipThreads execution model
**********************************

hipThreads provides a threads library that runs on AMD GPUs. This threads library is different from typical GPU programming models.

In a typical GPU programming model, each unit of parallel work is expressed as a kernel launch. For workloads that create many short-lived parallel tasks, such as iterative algorithms that spawn and join threads in a loop, kernel launch overhead is incurred on every iteration.

hipThreads replaces this pattern with a submit-to-scheduler model:

- A single persistent kernel runs for the lifetime of the application's ``hip::wthread`` usage.
- Each ``hip::wthread`` construction submits a work item to the scheduler, which dispatches it to an available virtual core (vcore).
- Joining a ``hip::wthread`` waits for that work item to complete without tearing down the kernel.

The execution model for hipThreads relies on the persistent scheduling kernel. When the first ``hip::wthread`` object is created, a scheduler kernel is launched on a dedicated stream.

Each subsequent ``hip::wthread`` object is submitted to this persistent scheduler kernel as a work item. The scheduler loops, polling work items in the queue, and running their workload.

The scheduler kernel persists until the last ``hip::wthread`` object is destroyed.

The scheduler manages a fixed grid of vcores. Each workgroup processor (WGP) on the GPU hosts a configurable number of vcores. That count is set by the :ref:`scheduler concurrency settings <tune-scheduler-concurrency>`.

The logical thread executes across multiple single instruction, multiple data (SIMD) lanes called fibers within a single GPU wavefront. All fibers in a thread execute in lockstep.

A single ``hip::wthread`` can run as multiple fibers, with one fiber per hardware lane. The workload runs on each active lane, which lets the thread partition SIMD-style work across lanes.

Logical threads are scheduled cooperatively. ``hip::this_thread::pseudo_yield()`` runs the next work item nested inside the current one. Once the nested work item completes, the original workload can continue. There is no preemption or hardware blocking in this model. Synchronization primitives such as ``condition_variable`` spin and yield rather than block. Those constraints, and the resulting deadlocks, are covered in :ref:`Limitations <limitations>`.

.. note::

  Because the scheduler kernel persists as long as any ``hip::wthread`` object exists, any call that waits for GPU work to finish also waits for the scheduler to finish. As a result, calls such as ``hipDeviceSynchronize()`` or a synchronous ``hipMemcpy()`` deadlock.

Stack space for callables
=========================

The submit-to-scheduler model also determines how much stack a ``hip::wthread`` callable has available, which is a design consideration when deciding what a callable keeps in local variables.

For an ordinary HIP kernel, the compiler sees every function the kernel calls, adds up the local variables along the deepest call path, and records the total in the kernel. The runtime then allocates exactly that much stack per thread.
The persistent scheduler kernel is compiled once, ahead of time, and runs whichever callable is submitted to it later, so it invokes the callable indirectly through a function pointer.
Because the callable is not known at that call site when the kernel is compiled, there is no frame size to compute, and the runtime instead applies the HIP stack limit ``hipLimitStackSize`` to the whole call chain.
This limit defaults to 1024 bytes per thread.

That budget covers every local variable in the callable and in anything it calls.
Objects with inline storage dominate it, and what matters is the storage class/location rather than the container type: a ``hip::std::inplace_vector`` and a plain array of the same element count occupy roughly the same stack space.
A callable that spawns child threads and keeps their handles in a local array is a common case.
Since ``sizeof(hip::wthread)`` is 16 bytes on the device, roughly 63 handles fit in the default budget, independently of ``hip::wthread::hardware_concurrency()``.

There are two ways to design around this. The first is to keep large objects off the stack by allocating them on the device heap, so the callable's own stack usage stays constant no matter how much data it manages.

.. code-block:: cpp

   hip::wthread coordinator(1, [=] __device__ {
       // On the device heap: the callable's stack holds only a pointer.
       hip::wthread* workers = new hip::wthread[count];
       for (int i = 0; i < count; ++i)
           workers[i] = hip::wthread(1, [=] __device__ { /* ... */ });
       for (int i = 0; i < count; ++i)
           workers[i].join();
       delete[] workers;
   });

The second is to raise ``hipLimitStackSize``. Set it before constructing the first ``hip::wthread``, because that construction is what launches the scheduler kernel, and the limit is applied at launch.
Size the value against actual usage: the budget applies per thread, so raising it increases the scratch memory reserved for the scheduler.
Raising it to 2 KB, for example, only moves the handle ceiling above from roughly 63 to roughly 127.

.. code-block:: cpp

   hipDeviceSetLimit(hipLimitStackSize, 128 * 1024);

   // Any hip::wthread created from here on runs with the larger budget.

.. note::

  Exceeding the budget is undefined behavior with no dedicated diagnostic, and the symptom depends on how much scratch memory the runtime reserved, which varies with occupancy and GPU architecture.
  It can appear as an abort with ``HSA_STATUS_ERROR_EXCEPTION`` (surfacing as ``an illegal memory access was encountered`` or ``unspecified launch failure``), or as a workload that runs correctly for a while and then hangs or produces wrong results after corrupting another thread's scratch memory.
  The latter is easy to mistake for a scheduling problem, but it is unrelated to scheduler capacity: a workload can exhaust its stack budget while running far below ``hardware_concurrency()``.
  Re-running with a substantially larger ``hipLimitStackSize`` confirms which one it is.
