Fine-tune Llama-3-8B with Llama-Factory

Fine-tune Llama-3-8B with Llama-Factory#

Author: Alex He
Knowledge level: Intermediate

This tutorial demonstrates how to fine-tune the Llama-3 8B large language model (LLM) on AMD ROCm GPUs by leveraging Llama-Factory. Efficient fine-tuning is vital for adapting large language models (LLMs) to downstream tasks. However, non-trivial efforts are required to implement these methods on different models.

Llama-Factory is a unified framework that integrates a suite of cutting-edge, efficient training methods.

It provides a solution for flexibly customizing the fine-tuning of over 100 LLMs without the need for coding through the built-in web UI LLAMABOARD.

Prerequisites#

This tutorial was developed and tested using the following setup.

Operating system#

Ubuntu 22.04: Ensure your system is running Ubuntu version 22.04.

Hardware#

AMD Instinct™ GPUs: This tutorial was tested on an AMD Instinct MI300X GPU. Ensure you are using an AMD Instinct GPU or compatible hardware with ROCm support and that your system meets the official requirements.

Software#

ROCm 6.3: Install and verify ROCm by following the ROCm install guide. After installation, confirm your setup using:
```
amd-smi
```
This command lists your AMD GPUs with relevant details.

Note: For ROCm 6.4 and earlier, use the rocm-smi command instead.
Docker: Ensure Docker is installed and configured correctly. Follow the Docker installation guide for your operating system.

Note: Ensure the Docker permissions are correctly configured. To configure permissions to allow non-root access, run the following commands:
```
sudo usermod -aG docker $USER
newgrp docker
```
Verify Docker is working correctly:
```
docker run hello-world
```

Hugging Face API access#

Obtain an API token from Hugging Face for downloading models.
Ensure the Hugging Face API token has the necessary permissions and approval to access the Meta Llama checkpoints.

Data preparation#

This tutorial uses a sample dataset from Hugging Face, which is prepared during the setup steps.

Prepare the training environment#

1. Pull the Docker image#

Ensure your system meets the system requirements.

Pull the Docker image required for this tutorial. LLaMA-Factory 0.9.5 needs Python 3.11 or later; this image has Python 3.12:

docker pull rocm/pytorch:rocm10.0_ubuntu24.04_py3.12_pytorch_release_2.13.0

2. Launch the Docker container#

Launch the Docker container and map the necessary directories. Replace /path/to/notebooks with the full path to the directory on your host machine where these notebooks are stored.

docker run -it --rm \
  --network=host \
  --device=/dev/kfd \
  --device=/dev/dri \
  --group-add=video \
  --ipc=host \
  --cap-add=SYS_PTRACE \
  --security-opt seccomp=unconfined \
  --shm-size 8G \
  -v $(pwd):/workspace \
  -w /workspace/notebooks \
  rocm/pytorch:rocm10.0_ubuntu24.04_py3.12_pytorch_release_2.13.0

Note: This command mounts the current directory to the /workspace directory in the container. Ensure the notebook file is either copied to this directory before running the Docker command or uploaded into the Jupyter Notebook environment after it starts. Save the token or URL provided in the terminal output to access the notebook from your web browser. You can download this notebook from the AI Developer Hub GitHub repository.

3. Install and launch Jupyter#

Inside the Docker container, install Jupyter using the following command:

pip install jupyter

Start the Jupyter server:

jupyter-lab --ip=0.0.0.0 --port=8888 --no-browser --allow-root

Note: Ensure port 8888 is not already in use on your system before running the above command. If it is, you can specify a different port by replacing --port=8888 with another port number, for example, --port=8890.

4. Install the Required Libraries and Frameworks#

Run the cell below to install the libraries and frameworks required for this tutorial.

DeepSpeed – Optimizes large-model training by reducing memory usage and enabling efficient distributed training.
LLaMA-Factory – An open-source framework that simplifies the training and fine-tuning of Large Language Models (LLMs).

%%bash
set -e
# Install Llama-Factory from source at release v0.9.5, with its metrics extras
# and DeepSpeed.
rm -rf LLaMA-Factory
git clone -q --depth 1 --branch v0.9.5 https://github.com/hiyouga/LLaMA-Factory.git
cd LLaMA-Factory
pip install -e . -r requirements/metrics.txt deepspeed==0.18.4
# This notebook is verified under llamafactory==0.9.5 deepspeed==0.18.4

Verify the installation:

# Verify the installation and version of the required libraries
!pip list | grep llamafactory

Here is the expected output:

llamafactory            0.9.5

5. Verify Llama-Factory#

To confirm the package is installed correctly, run the following command:

!llamafactory-cli help

You should see the following output:

----------------------------------------------------------------------
| Usage:                                                             |
|   llamafactory-cli api -h: launch an OpenAI-style API server       |
|   llamafactory-cli chat -h: launch a chat interface in CLI         |
|   llamafactory-cli export -h: merge LoRA adapters and export model |
|   llamafactory-cli train -h: train models                          |
|   llamafactory-cli webchat -h: launch a chat interface in Web UI   |
|   llamafactory-cli webui: launch LlamaBoard                        |
|   llamafactory-cli env: show environment info                      |
|   llamafactory-cli version: show version info                      |
| Hint: You can use `lmf` as a shortcut for `llamafactory-cli`.      |
----------------------------------------------------------------------

⚠️ Important: ensure the correct kernel is selected

If the verification process fails, ensure the correct Jupyter kernel is selected for your notebook. To change the kernel, follow these steps:

Go to the Kernel menu.
Select Change Kernel.
Select Python 3 (ipykernel) from the list.

Important: Failure to select the correct kernel can lead to unexpected issues when running the notebook.

6. Provide your Hugging Face token#

You’ll require a Hugging Face API token to access Llama-3.1. Generate your token at Hugging Face Tokens and request access for Llama-3.1 8B Instruct. Tokens typically start with “hf_”.

Run the following interactive block in your Jupyter notebook to set up the token:

Note: Uncheck the “Add token as Git credential” option.

from huggingface_hub import notebook_login, HfApi

# Prompt the user to log in
notebook_login()

Verify that your token was accepted correctly:

from huggingface_hub import HfApi

try:
    api = HfApi()
    user_info = api.whoami()
    print(f"Token validated successfully! Logged in as: {user_info['name']}")
except Exception as e:
    print(f"Token validation failed. Error: {e}")

Fine-tuning the model#

This section covers the process of setting up and running fine-tuning for the Llama-3 model using Llama-Factory. The following steps describe how to set up GPUs, import the required libraries, configure the model and training parameters, and run the fine-tuning process.

⚠️ Important: ensure the correct kernel is selected

Ensure the correct Jupyter kernel is selected for your notebook. To change the kernel, follow these steps:

Go to the Kernel menu.
Select Change Kernel.
Select Python 3 (ipykernel) from the list.

Important: Failure to select the correct kernel can lead to unexpected issues when running the notebook.

Set and verify the GPU availability#

Begin by specifying the GPUs available for fine-tuning. Verify that they are properly detected by PyTorch.

import os
import torch
gpus= [0, 1] # Rank 0 is for MI300x single device finetune, and Rank 0/1 for full 
os.environ.setdefault("CUDA_VISIBLE_DEVICES", ','.join(map(str, gpus)))
# Ensure PyTorch detects the GPUs correctly
print(f"PyTorch detected number of available devices: {torch.cuda.device_count()}") 

Download the model#

Run the following command to download the weights to your local machine. This also downloads the tokenizer model and a responsible use guide.

To download Llama-3.1-8B-Instruct, use the following command:

!huggingface-cli download --resume-download meta-llama/Llama-3.1-8B-Instruct

Run the fine-tuning recipes#

Now write the fine-tuning recipe llama3_full_sft.yaml for distributed training. It follows the full fine-tuning example LLaMA-Factory ships (examples/train_full/qwen3_full_sft.yaml), set up for Llama-3.1 8B Instruct with the llama3 chat template, and uses DeepSpeed ZeRO-3. The recipe uses the dataset identity and alpaca_en_demo JSON files.

Note: These datasets are only for evaluation purposes. You might need to change them to the actual ones.

The recipe saves a checkpoint every 500 steps. Before running the training command, configure the appropriate save_only_model and save_steps values as required.

%%writefile LLaMA-Factory/examples/train_full/llama3_full_sft.yaml
### model
model_name_or_path: meta-llama/Llama-3.1-8B-Instruct
trust_remote_code: true

### method
stage: sft
do_train: true
finetuning_type: full
deepspeed: examples/deepspeed/ds_z3_config.json  # choices: [ds_z0_config.json, ds_z2_config.json, ds_z3_config.json]

### dataset
dataset: identity,alpaca_en_demo
template: llama3
cutoff_len: 2048
max_samples: 1000
overwrite_cache: true
preprocessing_num_workers: 16
dataloader_num_workers: 4

### output
output_dir: saves/llama3.1-8b/full/sft
logging_steps: 10
save_steps: 500
plot_loss: true
overwrite_output_dir: true
save_only_model: false
report_to: none  # choices: [none, wandb, tensorboard, swanlab, mlflow]

### train
per_device_train_batch_size: 1
gradient_accumulation_steps: 2
learning_rate: 1.0e-5
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
ddp_timeout: 180000000
resume_from_checkpoint: null

The following command fine-tunes the Llama-3.1 8B model using the available GPUs. The recipe trains for three epochs. To try it quickly, set MAX_STEPS to the number of training steps to run; -1 keeps the recipe’s three epochs:

MAX_STEPS = -1  # -1 trains the recipe's full three epochs

Expose MAX_STEPS as an environment variable so the shell training cell below can read it:

Run the fine-tuning: launch distributed training with the recipe across the available GPUs:

%%bash
set -e
cd LLaMA-Factory/
FORCE_TORCHRUN=1 llamafactory-cli train examples/train_full/llama3_full_sft.yaml max_steps=$MAX_STEPS

Check the results#

Training writes the fine-tuned model, its tokenizer and the training metrics to the recipe’s output_dir:

import json

output_dir = "LLaMA-Factory/saves/llama3.1-8b/full/sft"
with open(f"{output_dir}/train_results.json") as f:
    results = json.load(f)
print(f"Final training loss: {results['train_loss']:.4f}")
print(sorted(os.listdir(output_dir)))
assert os.path.exists(f"{output_dir}/config.json"), "the fine-tuned model was not saved"

Monitoring GPU memory#

To monitor GPU memory during training, run the following command in the terminal.

watch -n0 rocm-smi

Note: For ROCm 6.5 and later, use the amd-smi command instead.

This command displays memory usage and other GPU metrics to ensure your hardware resources are being optimally used.

Conclusion#

LLaMA-Factory + DeepSpeed + ROCm provides a simple and efficient workflow for fine-tuning LLMs on AMD GPUs.