Building AI pipelines for voice assistants using ROCm, LlamaIndex, and RAG#
Author: Victoria Godsoe
Knowledge level: Beginner
The following notebook demonstrates how to use AMD GPUs with LlamaIndex and Retrieval-Augmented Generation (RAG). It takes an input audio recording, transcribes it to text, sends the transcribed text to the RAG model, and generates a response in text format, which is then converted to speech and saved as an audio file.
Prerequisites#
This tutorial was developed and tested using the following setup.
Operating system#
Ubuntu 22.04: Ensure your system is running Ubuntu version 22.04.
Hardware#
AMD GPUs: This tutorial was tested on an AMD Instinct™ MI300X and an AMD Radeon™ W7900. Ensure you are using an AMD GPU with ROCm support and that your system meets the official requirements.
Software#
This tutorial was tested on both AMD Radeon and AMD Instinct GPUs using the following setup:
ROCm 6.2.0
Python 3.10
PyTorch 2.3.0
Objectives#
After completing this tutorial, you should understand the following concepts:
Multi-model pipeline
LlamaIndex with ROCm on AMD GPUs
Prepare the inference environment#
To set up the inference environment, follow these steps:
Create a conda environment:
conda create -n rocm python=3.10
Activate the environment:
conda activate rocm
Install the PyTorch for ROCm software:
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2
Include an audio file (for example,
summarize_question.wav). Place it in the current working directory.
The notebook installs its Python packages itself (LlamaIndex with its Ollama and Hugging Face integrations, openai-whisper, transformers, ChatTTS and soundfile), installs ffmpeg for Whisper if it is missing, starts Ollama and pulls the Llama 3 model.
Install and launch Jupyter#
Inside the Docker container, install Jupyter using the following command:
pip install jupyter
Then start the Jupyter server:
jupyter-lab --ip=0.0.0.0 --port=8888 --no-browser --allow-root
Note: Ensure port 8888 is not already in use on your system before running the above command. If it is, you can specify a different port by replacing --port=8888 with another port number, for example, --port=8890.
Install the Python packages#
Install the tutorial’s Python packages: LlamaIndex with its Ollama and Hugging Face integrations, Whisper for speech recognition, transformers, ChatTTS for text-to-speech, and soundfile to write the audio:
%pip install llama-index llama-index-llms-ollama llama-index-embeddings-ollama llama-index-embeddings-huggingface openai-whisper transformers ChatTTS soundfile
Import the packages#
Import the following packages:
os: Operating-system-dependent functionality.
whisper: A speech-recognition library.
torch: A PyTorch library for tensor computations and deep learning.
llama_index.core: Core functionality for the Llama Index.
llama_index.embeddings.huggingface: Support for embedding HuggingFace.
llama_index.llms.ollama: Functionality for the Ollama language model.
ChatTTS: A text-to-speech conversion library.
soundfile: Writes the generated audio to a WAV file.
IPython.display: For displaying audio in Jupyter notebooks.
# System Imports
import os
import numpy as np
import re
# Imports for Speech to Text
import whisper
import torch
# Imports for RAG Model
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.llms.ollama import Ollama
# Imports for Text to Speech
import ChatTTS
import soundfile as sf
from IPython.display import Audio
Set up the environment#
Optionally, set the environment variables to enable experimental features in PyTorch ROCm.
Verify the PyTorch version and GPU availability.
Select the computation device:
Use the GPU if available and print its properties.
Fall back to the CPU otherwise.
# Set the environment variable for experimental features (optional)
os.environ['TORCH_ROCM_AOTRITON_ENABLE_EXPERIMENTAL'] = '1'
os.environ['HIP_VISIBLE_DEVICES'] = "0"
print(f"Torch version: {torch.__version__}")
print(f"CUDA available: {torch.cuda.is_available()}")
# Check GPU availability and properties
device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
if device.type == "cuda":
print(f"Using GPU: {torch.cuda.get_device_name(0)}")
else:
print("Using CPU (no GPU detected)")
Transcribe speech to text#
The following section performs speech-to-text transcription using the Whisper model.
First, download the sample audio file for this tutorial.
!curl -L https://raw.githubusercontent.com/ROCm/gpuaidev/main/docs/notebooks/assets/summarize_question.wav -o summarize_question.wav
Load the audio file and transcribe the speech content into text.
AUDIO_FILE = "summarize_question.wav"
Audio(AUDIO_FILE, rate=24_000, autoplay=True)
Whisper decodes audio with ffmpeg. Install it if it is missing:
%%bash
set -e
command -v ffmpeg >/dev/null || (apt-get update -qq && apt-get install -y -qq ffmpeg)
Now transcribe the speech content into text.
# Speech-to-Text with Whisper
try:
model = whisper.load_model("base")
result = model.transcribe(AUDIO_FILE)
input_text = result["text"]
print(f"Transcribed text: {input_text}")
except Exception as e:
raise RuntimeError(f"Error in speech-to-text: {e}") from e
Add the RAG model#
To use a RAG model, provide the context that you’d like the LLM to use for the queries. This example is configured to use the documents in the data folder. If you don’t have any documents yet, you can add your own or download the one below.
DATA_DIR = "./data"
# Check if the data directory exists, and create it if it doesn't
if not os.path.exists(DATA_DIR):
os.makedirs(DATA_DIR)
print(f"Data directory '{DATA_DIR}' created. Add a file of your choosing, or use the cell below to download sample text.")
else:
# Check if data directory is empty
if not os.listdir(DATA_DIR):
print(f"Data directory '{DATA_DIR}' is empty. Add a file of your choosing, or use the cell below to download sample text.")
If the data directory is empty, run the following cell:
## OPTIONAL - Run this cell if your data directory is empty
!mkdir -p data && curl -L https://www.gutenberg.org/cache/epub/11/pg11.txt -o data/pg11.txt
Verify the data file now exists in the data directory.
# View the files in your data directory
print("Files in data directory:", os.listdir("data"))
documents = SimpleDirectoryReader(DATA_DIR).load_data()
Start Ollama#
The RAG step uses Llama 3 served by Ollama. Install Ollama with its official installer from a terminal (it needs sudo):
curl -fsSL https://ollama.com/install.sh | sh
The next cell checks that Ollama is installed. If it is not and the notebook runs as root, for example in a container, the cell installs the Ollama release this tutorial was tested with.
%%bash
set -e
# Install Ollama only if it is missing and this notebook runs as root (for example
# in a container); otherwise install it from a terminal as described above.
OLLAMA_VERSION=0.40.5 # the release this tutorial was tested with
if ! command -v ollama >/dev/null; then
if [ "$(id -u)" != 0 ]; then
echo "Ollama is not installed: install it from a terminal (see above), then rerun this cell." >&2
exit 1
fi
command -v zstd >/dev/null || (apt-get update -qq && apt-get install -y -qq zstd)
for archive in ollama-linux-amd64 ollama-linux-amd64-rocm; do
curl -fsSL "https://github.com/ollama/ollama/releases/download/v$OLLAMA_VERSION/$archive.tar.zst" \
| zstd -d | tar -x -C /usr/local
done
fi
ollama --version
Next, start the Ollama server and pull the model. These helpers start a server in the background, wait until it answers (or stop with its log if it fails), and stop it again:
Start the Ollama server in the background (or reuse one that is already running):
ollama_server = ensure_server("ollama serve", "http://localhost:11434/api/version",
"ollama_server.log", timeout=120)
Pull the Llama 3 model that Ollama serves:
%%bash
set -e
ollama pull llama3
For the embedding model, use “bge-base” from HuggingFaceEmbedding. The Ollama server started above supplies Llama-3 for the LLM.
Next, create a VectorStoreIndex from the loaded documents and initialize a query engine with the index. Then issue your query using the text output from the Whisper model. Print the response so you can compare it against the audio output in the next step.
# Initialize embedding and LLM models
Settings.embed_model = HuggingFaceEmbedding(model_name="BAAI/bge-base-en-v1.5")
try:
Settings.llm = Ollama(model="llama3", request_timeout=360.0)
except Exception as e:
raise RuntimeError(f"Error connecting to Ollama server: {e}") from e
# Build and query the vector index
index = VectorStoreIndex.from_documents(documents)
query_engine = index.as_query_engine(streaming=True, response_mode="compact", similarity_top_k=3)
response = query_engine.query(input_text)
# Function to convert StreamingResponse to string
def streaming_response_to_string(streaming_response):
text = ""
for chunk in streaming_response.response_gen:
if isinstance(chunk, dict) and "text" in chunk:
text += chunk["text"]
else:
text += str(chunk)
return text
# Convert response to string
response_text = streaming_response_to_string(response)
print(f"Generated response: {response_text}")
Perform text-to-speech conversion#
The following example performs text-to-speech conversion using the ChatTTS library and saves the output audio to a file.
This example uses the following constants:
OUTPUT_AUDIO_FILE(str): The name of the output audio file.SAMPLE_RATE(int): The sample rate for the output audio file.
It provides the following functionality:
Initializes a
ChatTTS.Chatobject.Loads the chat model without compilation for faster loading. (Set
compile=Truefor better performance.)Converts the response text from the previous step to speech.
Saves the generated audio to the specified output file using soundfile.
OUTPUT_AUDIO_FILE = "voice_pipeline_response.wav"
SAMPLE_RATE = 24000
# Text cleanup function for TTS
def sanitize_input(text):
sanitized_text = text.replace('-', '') # Remove hyphens
sanitized_text = sanitized_text.replace('(', '').replace(')', '') # Remove parentheses
return sanitized_text.strip()
# Text-to-Speech processing
try:
sanitized_response = re.sub(r"[^a-zA-Z0-9.,?! ]", "", response_text) # Remove special characters
print(f"Sanitized response for TTS: {sanitized_response}")
sanitized_response = [sanitized_response]
chat = ChatTTS.Chat()
chat.load(compile=False) # Set to True for better performance
params_infer_code = ChatTTS.Chat.InferCodeParams(
spk_emb = chat.sample_random_speaker(),
)
wavs = chat.infer(
sanitized_response,
params_infer_code=params_infer_code,
)
# soundfile writes the WAV directly; torchaudio.save now needs TorchCodec, which
# isn't available for ROCm PyTorch
sf.write(OUTPUT_AUDIO_FILE, np.asarray(wavs[0]).squeeze(), SAMPLE_RATE)
except Exception as e:
raise RuntimeError(f"Error in text-to-speech: {e}") from e
finally:
if 'chat' in locals():
chat.unload()
Play the following cell to hear the generated speech.
Audio(wavs[0], rate=24_000, autoplay=True)
When you’re done, stop the Ollama server:
stop_server(ollama_server)