Text-to-video generation with ComfyUI and an AMD Radeon GPU

Text-to-video generation with ComfyUI and an AMD Radeon GPU#

Author: Xun Wang
Knowledge level: Beginner

With the advancement of Artificial Intelligence Generated Content (AIGC) technology, the fields of text-to-video and image-to-video generation have garnered widespread attention from visual designers, art enthusiasts, and media creators.

ComfyUI is a ‌node-based graphical interface‌ designed for diffusion models, enabling users to visually construct AI image/video generation workflows through modular operations. Its modular node design, efficiency, compatibility, and workflow advantage make it a perfect choice for media creators to ‌boost productivity.

This tutorial guides you through setting up and running ComfyUI on AMD Radeon™ GPUs using ROCm™ software. Learn how to configure your environment, install the ComfyUI tool, and generate video from text on AMD consumer GPUs.

Note: AMD provides a special implementation of ComfyUI for ROCm (on Instinct™ GPUs only). For more information, see the ComfyUI on ROCm documentation.

Prerequisites#

This tutorial was developed and tested using the following setup.

Operating system#

  • Ubuntu 22.04/24.04: Ensure your system is running Ubuntu 22.04 or 24.04.

Hardware#

  • AMD Radeon GPUs: This tutorial was tested on an AMD Radeon RX 7900 XTX GPU. Ensure you are using an AMD Radeon GPU or compatible hardware with ROCm support and that your system meets the official requirements.

Software#

  • ROCm 6.3: Install and verify ROCm by following the ROCm install guide. After installation, confirm your setup using:

    amd-smi

    This command lists your AMD GPUs with relevant details.

    Note: For ROCm 6.4 and earlier, use the rocm-smi command instead.

ComfyUI setup#

To set up the ComfyUI inference environment, follow the steps below.

Prepare the inference environment#

Create and activate a virtual environment using conda or venv.

conda create -n comfyui_env_test python=3.12
conda activate comfyui_env_test

PyTorch installation#

Install the PyTorch ROCm wheels in the virtual environment. See Install PyTorch for Radeon GPUs for installation instructions.

Install and launch Jupyter#

Inside the Python virtual environment, install Jupyter using the following command:

pip install jupyter

Then start the Jupyter server:

jupyter-lab --ip=0.0.0.0 --port=8888 --no-browser --allow-root

Note: Ensure port 8888 is not already in use on your system before running the above command. If it is, you can specify a different port by replacing --port=8888 with another port number, for example, --port=8890.

Verify the PyTorch installation#

Verify that PyTorch is correctly installed.

Step 1: Verify PyTorch is installed and can detect the GPU compute device.

!python3 -c 'import torch' 2> /dev/null && echo 'Success' || echo 'Failure'

The expected result is Success.

Step 2: Confirm the GPU is available.

!python3 -c 'import torch; print(torch.cuda.is_available())'

The expected result is True.

Step 3: Display the installed GPU device name.

!python3 -c "import torch; print(f'device name [0]:', torch.cuda.get_device_name(0))"

The expected result should be similar to: device name [0]: Radeon RX 7900 XTX

device name[0]: <Supported AMD GPU>

ComfyUI installation#

Install ComfyUI from source on the system with the AMD GPU.

Clone the following repo:

%%bash
set -e
[ -d ComfyUI ] || git clone --branch v0.42.1 https://github.com/comfyanonymous/ComfyUI.git

Ensure that PyTorch will not be reinstalled with the CUDA version:

import os
os.chdir("ComfyUI")
print(os.getcwd())
!sed -i.bak -E '/^(torch|torchaudio|torchvision)([<>=~!0-9.]*)?$/s/^/# /' requirements.txt

Install the dependencies:

%%bash
set -e
pip3 install -r requirements.txt

Running text-to-video generation#

Follow these steps to generate video from your text.

Model preparation#

LTX Video is an efficient video model from Lightricks. Ltx-video-2B-v0.9.5 is used for this tutorial.

Download the LTX Video model ltx-video-2b-v0.9.5.safetensors and text encoder model t5xxl_fp16.safetensors files. Place these model files in the ComfyUI/models/checkpoints and ComfyUI/models/text_encoders folders. Use the following code to automate the process. It downloads the files with huggingface_hub, which resumes interrupted downloads and keeps them in the Hugging Face cache (~/.cache/huggingface), and links them into those folders:

%pip install -q huggingface_hub
import os
from huggingface_hub import hf_hub_download

# (repository, file in the repository, ComfyUI model folder). hf_hub_download
# resumes interrupted downloads and reuses files already in the Hugging Face
# cache. If your environment provides the models in MODELS_DIR (as
# MODELS_DIR/<repository>/<file>), those copies are used instead.
models = [
    ("Lightricks/LTX-Video", "ltx-video-2b-v0.9.5.safetensors", "models/checkpoints"),
    ("Comfy-Org/mochi_preview_repackaged", "split_files/text_encoders/t5xxl_fp16.safetensors",
     "models/text_encoders"),
]
models_dir = os.environ.get("MODELS_DIR")
for repo, filename, folder in models:
    local = models_dir and os.path.join(models_dir, repo, filename)
    if local and os.path.isfile(local):
        cached = local
    else:
        print(f"Downloading {repo}/{filename}...")
        cached = hf_hub_download(repo, filename)
    os.makedirs(folder, exist_ok=True)
    target = os.path.join(folder, os.path.basename(filename))
    if os.path.lexists(target):
        os.remove(target)
    os.symlink(cached, target)  # ComfyUI follows links into the cache
    print(f"✅ {target}")

Different diffusion model pipelines might contain different submodels that should be placed into the proper subfolders under ComfyUI/models.

Launch the server#

Launch the ComfyUI server in the background and wait until it answers:

Hide code cell source

 
import os
import signal
import subprocess
import time
import urllib.request


def server_is_up(url, headers=None):
    try:
        with urllib.request.urlopen(urllib.request.Request(url, headers=headers or {}),
                                    timeout=5):
            return True
    except Exception:
        return False


def ensure_server(command, url, log_path, timeout=1200, headers=None):
    """Start `command` in the background (its output goes to log_path) unless
    url already answers, then wait until it does. Returns the server process,
    or None if a server was already running. Raises, with the end of the log,
    if the server exits or is not ready within timeout seconds."""
    if server_is_up(url, headers):
        print(f"Already running: {url}")
        return None
    log = open(log_path, "w")
    server = subprocess.Popen(command, shell=True, stdout=log, stderr=subprocess.STDOUT,
                              start_new_session=True)
    deadline = time.time() + timeout
    while time.time() < deadline and server.poll() is None:
        if server_is_up(url, headers):
            print(f"Ready after {timeout - int(deadline - time.time())}s: {url}")
            return server
        time.sleep(5)
    tail = "".join(open(log_path).readlines()[-40:])
    state = f"exited with code {server.returncode}" if server.poll() is not None else "timed out"
    raise RuntimeError(f"Server {state} before {url} answered. Last log lines:\n{tail}")


def stop_server(server):
    """Stop a server started by ensure_server, and everything it started."""
    if server is not None and server.poll() is None:
        os.killpg(server.pid, signal.SIGTERM)
        try:
            server.wait(timeout=60)
        except subprocess.TimeoutExpired:
            os.killpg(server.pid, signal.SIGKILL)

Start the server (using the helpers above) and wait until it responds:

comfyui_server = ensure_server("python3 main.py --listen 127.0.0.1 --port 8188",
                               "http://127.0.0.1:8188/system_stats",
                               "comfyui_server.log", timeout=600)

The server log (comfyui_server.log) shows these entries once it is up:

Starting server

To see the GUI go to: http://127.0.0.1:8188

Copy the address http://127.0.0.1:8188 and paste it into your browser. The resulting webpage displays a default workflow which looks like this:

default workflow

Load the workflow#

The ComfyUI workflow defines the full pipeline and parameters used to generate an image or video. It’s formatted as a JSON file or encoded in a WebP animated image (*.webp). You can construct your own workflow from scratch or customize the workflow from third party.

Download the LTX Video text-to-video workflow (in *.json or *.webp format) from the example page, then load it or drag it on the ComfyUI GUI. The loaded workflow looks like the image below:

ltx workflow

The workflow is composed of different nodes, each of which has different functionality.

  • The Load Checkpoint node handles diffusion model loading. The ckpt_name field can be changed if there are other diffusion models available.

  • The Load Image node handles loading the input image. Upload the required image using this node.

  • The CLIP TEXT Encode (Positive Prompt) node is a placeholder for the positive prompt. You can change it to your own prompt. Here is the recommended prompt for this tutorial:

    A drone quickly rises through a bank of morning fog, revealing a pristine alpine lake surrounded by snow-capped mountains. The camera glides forward over the glassy water, capturing perfect reflections of the peaks. As it continues, the perspective shifts to reveal a lone wooden cabin with a curl of smoke from its chimney, nestled among tall pines at the lake’s edge. The final shot tracks upward rapidly, transitioning from intimate to epic as the full mountain range comes into view, bathed in the golden light of sunrise breaking through scattered clouds.

  • The EmptyLTXVLatentVideo node is used to control the frame size and length for the generated video.

For more information about the ComfyUI nodes, see the ComfyUI wiki.

Run inference#

After the workflow is ready, click Run to initiate the whole pipeline to process. The output log for the inference will look like this:

 got prompt
 model weight dtype torch.bfloat16, manual cast: None
 model_type FLUX
 VAE load device: cuda:0, offload device: cpu, dtype: torch.float32
 no CLIP/text encoder weights in checkpoint, the text encoder model will not be loaded.
 CLIP/text encoder model load device: cuda:0, offload device: cpu, current: cpu, dtype: torch.float16
 Requested to load MochiTEModel_
 loaded completely 23280.8 9083.38671875 True
 Requested to load LTXV
 loaded completely 10667.71279296875 3667.902587890625 True
 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 30/30 [00:43<00:00,  1.46s/it]
 Requested to load VideoVAE
 loaded completely 5763.6171875 4756.450138092041 True
 Warning: Ran out of memory when regular VAE decoding, retrying with tiled VAE decoding.
 Prompt executed in 66.29 seconds

The final output video is shown in the SaveAnimatedWEBP node, while the *.webp file output can be found in the ComfyUI/output/ folder.

ltx output

Optional: run the same workflow without the browser#

The same LTX-Video workflow can also run from Python, through ComfyUI’s HTTP API: queue it with /prompt, then wait on /history until it completes. The next cells do this with the recommended prompt above, so you can skip them if you ran the workflow in the GUI.

import json
import time
import urllib.request
from pathlib import Path

from PIL import Image

# LTX-Video text-to-video, in ComfyUI's API format: the same graph as the example
# workflow above, with each node's inputs spelled out.
workflow = {
    "38": {"class_type": "CLIPLoader",
           "inputs": {"clip_name": "t5xxl_fp16.safetensors", "type": "ltxv"}},
    "44": {"class_type": "CheckpointLoaderSimple",
           "inputs": {"ckpt_name": "ltx-video-2b-v0.9.5.safetensors"}},
    "6": {"class_type": "CLIPTextEncode",
          "inputs": {"clip": ["38", 0],
                     "text": "A drone quickly rises through a bank of morning fog, revealing a pristine "
                             "alpine lake surrounded by snow-capped mountains. The camera glides forward "
                             "over the glassy water, capturing perfect reflections of the peaks. As it "
                             "continues, the perspective shifts to reveal a lone wooden cabin with a curl "
                             "of smoke from its chimney, nestled among tall pines at the lake's edge. The "
                             "final shot tracks upward rapidly, transitioning from intimate to epic as the "
                             "full mountain range comes into view, bathed in the golden light of sunrise "
                             "breaking through scattered clouds."}},
    "7": {"class_type": "CLIPTextEncode",
          "inputs": {"clip": ["38", 0],
                     "text": "low quality, worst quality, deformed, distorted, blurry"}},
    "69": {"class_type": "LTXVConditioning",
           "inputs": {"positive": ["6", 0], "negative": ["7", 0], "frame_rate": 25}},
    "70": {"class_type": "EmptyLTXVLatentVideo",
           "inputs": {"width": 768, "height": 512, "length": 97, "batch_size": 1}},
    "71": {"class_type": "LTXVScheduler",
           "inputs": {"steps": 30, "max_shift": 2.05, "base_shift": 0.95, "stretch": True,
                      "terminal": 0.1, "latent": ["70", 0]}},
    "73": {"class_type": "KSamplerSelect", "inputs": {"sampler_name": "euler"}},
    "72": {"class_type": "SamplerCustom",
           "inputs": {"model": ["44", 0], "add_noise": True, "noise_seed": 42, "cfg": 3,
                      "positive": ["69", 0], "negative": ["69", 1], "sampler": ["73", 0],
                      "sigmas": ["71", 0], "latent_image": ["70", 0]}},
    "8": {"class_type": "VAEDecode", "inputs": {"samples": ["72", 0], "vae": ["44", 2]}},
    "41": {"class_type": "SaveAnimatedWEBP",
           "inputs": {"images": ["8", 0], "filename_prefix": "ltxv_text_to_video", "fps": 24,
                      "lossless": False, "quality": 90, "method": "default"}},
}

request = urllib.request.Request("http://127.0.0.1:8188/prompt",
                                 data=json.dumps({"prompt": workflow}).encode(),
                                 headers={"Content-Type": "application/json"})
prompt_id = json.load(urllib.request.urlopen(request))["prompt_id"]
print("Queued", prompt_id)

# Wait for the job to finish (the first run also loads the models).
deadline = time.time() + 1800
while time.time() < deadline:
    history = json.load(urllib.request.urlopen(f"http://127.0.0.1:8188/history/{prompt_id}"))
    if prompt_id in history and history[prompt_id].get("status", {}).get("completed"):
        break
    time.sleep(5)
else:
    raise TimeoutError("The workflow did not finish within 30 minutes")

Check the result: the workflow saves an animated WebP video.

# The video is saved under ComfyUI/output; check it exists and has frames.
saved = [Path("output") / img.get("subfolder", "") / img["filename"]
         for node in history[prompt_id]["outputs"].values()
         for img in node.get("images", [])]
assert saved, "The workflow finished without saving a video"
video = Image.open(saved[0])
frames = getattr(video, "n_frames", 1)
assert frames > 1, f"{saved[0]} has {frames} frame(s)"
print(f"{saved[0]}: {video.size[0]}x{video.size[1]}, {frames} frames")
video

Stop the ComfyUI server when you are done:

stop_server(comfyui_server)
Contents