AINIC bundle versions#
AMD published training images ship a fixed AINIC bundle. The bundle version is
baked in at build time through the AINIC_BUNDLE_VERSION build argument, and the
image installs the userspace half of the AINIC stack — the libionic library —
from https://repo.radeon.com/amdainic/pensando/ubuntu/<bundle>/.
Sometimes the bundle in the image is not the one your cluster needs. This guide shows how to rebuild a published image against an arbitrary bundle, and how to confirm the change landed. Upgrades and downgrades are both normal operations here; the correct bundle for your cluster is often an older one.
Scope. This guide covers making the change to
libionicinside the container, and verifying it took effect. Deciding which bundle you need is your responsibility —libionicin the container has to be compatible with theionicdriver on the host. See section 3, which includes host commands that often answer the question directly.This guide does not change the host driver, and does not change how AINIC is enabled at runtime — for that, see Multi-node networking.
This guide assumes a published bundle throughout. If you have been given a bundle as a tarball rather than a repository, see section 7.
What this was tested on. The MaxText JAX training images —
rocm/jax-training:maxtext-v26.3.2,-v26.4, and-v26.6. The rebuild was exercised on all three; the runtime check in section 6 onv26.3.2. It has not been tested on Primus’s ownrocm/primus:*images. Nothing here is specific to MaxText — it operates onlibionicand the apt configuration, not on the framework — but a different image may enable different bundle repositories or ship a differentlibionicto begin with, so confirm with section 6 rather than assuming the versions quoted here.
Quickstart#
If you already know which bundle you need, this is the whole procedure. Save the
block below as Dockerfile in an otherwise empty directory — the whole directory
is sent to the Docker daemon as build context, so keeping it empty keeps the
build fast.
ARG BASE_IMAGE
FROM ${BASE_IMAGE}
ARG AINIC_BUNDLE_VERSION
RUN set -eux; \
codename="$(. /etc/os-release; echo "$VERSION_CODENAME")"; \
rm -f /etc/apt/sources.list.d/*amdainic*; \
add-apt-repository -y "deb https://repo.radeon.com/amdainic/pensando/ubuntu/${AINIC_BUNDLE_VERSION} $codename main"; \
apt update --allow-insecure-repositories; \
ver="$(apt-cache madison libionic1 \
| awk -F'|' -v b="/${AINIC_BUNDLE_VERSION} " 'index($3,b){gsub(/ /,"",$2); print $2; exit}')"; \
test -n "$ver"; \
echo "installing libionic $ver from ${AINIC_BUNDLE_VERSION}"; \
apt install -y --allow-unauthenticated --allow-downgrades \
"libionic-dev=$ver" "libionic1=$ver"; \
test "$(dpkg-query -W -f='${Version}' libionic1)" = "$ver"; \
test "$(dpkg-query -W -f='${Version}' libionic-dev)" = "$ver"; \
rm -rf /var/lib/apt/lists/*
# Structural check: the package is installed consistently and the provider
# symlink the fabric actually loads points at the installed object.
RUN set -eux; \
dpkg-query -W -f='${Package} ${Version}\n' libionic1 libionic-dev; \
dpkg -C; \
test -e /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so; \
readlink -f /usr/lib/x86_64-linux-gnu/libionic.so.1
Then, from that same directory:
BASE=rocm/jax-training:maxtext-v26.7
BUNDLE=1.117.5-a-147
docker build --network host \
--build-arg BASE_IMAGE=$BASE \
--build-arg AINIC_BUNDLE_VERSION=$BUNDLE \
-t ${BASE##*/}-ainic-$BUNDLE .
--network host is needed when the daemon’s default bridge network cannot
resolve repo.radeon.com; it is harmless otherwise, so it is included by
default.
Neither build argument has a default, so omitting one fails the build rather than
silently producing an unintended image. Tagging the image after the bundle is
worth the small effort: the bundle is otherwise invisible without running
dpkg-query inside the image.
A clean build does not mean the image works on your host. It means the version you asked for was installed. See section 6 before trusting it, and section 5 for why the apparently boilerplate lines are load-bearing.
1. What is actually in the image#
An AINIC-enabled training image contains only the userspace half of the pair:
Package |
Role |
|---|---|
|
User space ionic provider driver for |
|
Development headers for the ionic provider |
The driver-side packages of the bundle — ionic-dkms, pds-dkms, nicctl,
ainic-monitor — are not installed in the container. They belong to the host.
This is why bundle mismatches show up as a host/container skew rather than as a
missing package.
Note that the bundle name and the libionic version are different numbering
schemes, and they are easy to confuse when discussing versions. The excerpt
below is ordered by bundle version, newest first; the libionic column does not
follow, because a higher bundle version does not imply a higher libionic
version. See section 4 to list every published
bundle.
Bundle |
|
|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
Most published bundles use a 54.0-NNN scheme and ship one libionic version
for all distributions. That is not universal: 1.117.5-a-196 (and currently
1.117.5) ship a dated, per-distro version instead. Always read the version
out of the installed package rather than assuming the scheme.
2. Check what you have#
# libionic version actually installed in the image
docker run --rm <image> dpkg-query -W -f='${Package} ${Version}\n' libionic1 libionic-dev
# which bundle repositories the image was built against
docker run --rm <image> grep -rh amdainic /etc/apt/sources.list.d/
The repository list is not a reliable indicator of the installed version. An
image can carry several AINIC repositories, and apt resolves libionic to the
highest version across all enabled repositories. An image whose Dockerfile names
one bundle can therefore ship the library of a different one. Always trust
dpkg-query, not the sources.list entries or the build argument.
On the host, check the other half of the pair:
lspci | grep -i pensando
modinfo ionic | grep -E '^version|^srcversion'
ls /sys/class/infiniband/ # expect ionic_* entries
3. Choosing a bundle#
libionic in the container must be compatible with the ionic driver on the
host. Start by asking the host, which frequently answers the question outright:
# Often reports the AINIC bundle version directly, e.g. 1.117.5-a-77
cat /sys/class/infiniband/*/fw_ver
# Same information from the network device side
ethtool -i <ionic_netdev> | grep -i firmware
# The host driver build, which tracks the bundle's linux-ionic component
modinfo ionic | grep '^version'
If fw_ver reports a bundle version, that is the bundle the host is running and
the natural target for the container. Where it does not, or where the host is
managed by someone else, bring your cluster’s operators the output above together
with:
cat /sys/class/infiniband_verbs/uverbs*/abi_version
This is the kernel side of the uverbs ABI, the interface libionic has to speak.
There is no matching number you can read out of the container to compare it
against — a provider library declares the ABI range it supports internally, not
as a queryable version. Note also that this value is coarse: hosts running
different bundles can report the same ABI, so it will not distinguish two
candidate bundles from each other. Treat it as an input to the conversation with
your operators, not as a selection test. The authoritative test is the runtime one
in section 6.
One thing worth knowing before you pick: newer is not automatically better. Compatibility is not monotonic in the bundle version, so the correct target may well be older than what the image already ships.
If you install a bundle the host cannot work with, the build still succeeds and every package-level check in section 6 still passes. Package-level checks cannot see a host/container skew at all, which is why section 6 ends with a runtime check. Note that the converse also holds: a skewed pair is not guaranteed to fail visibly — userspace/firmware compatibility spans a range of versions rather than requiring an exact match, so a mismatched pair may enumerate devices and reach the fabric normally. Matching the bundle to the host remains the supported configuration, and the runtime check tells you which situation you are in.
4. List published bundles#
curl -s https://repo.radeon.com/amdainic/pensando/ubuntu/ | grep -oE 'href="[0-9][^"]+/"'
The [0-9] anchor keeps the page header and parent-directory links out of the
result, so every line is a bundle.
To see the exact package versions a bundle ships before committing to it:
BUNDLE=1.117.5-a-147
curl -s "https://repo.radeon.com/amdainic/pensando/ubuntu/${BUNDLE}/dists/noble/main/binary-amd64/Packages" \
| awk '/^Package: /{p=$2} /^Version: /{print p"="$2}'
Some published directories are empty placeholders — they contain only conf/
and db/, with no dists/ or pool/, and apt cannot install from them. The
Packages query above returns nothing for these, which is the quickest way to
spot one before a build fails.
Check for this even if you are deliberately picking the newest bundle: the
newest entries in the listing are currently placeholders, so choosing the
highest version is exactly the case that hits it. A build against a placeholder
fails at apt update with 404 Not Found on the Packages file, before the
bundle version is ever evaluated.
This query also only works for published bundles. To install from a pre-downloaded tarball instead, see section 7.
5. How the rebuild works#
Four lines in the quickstart Dockerfile look like boilerplate and are not.
Deriving
$codenamefrom/etc/os-release— the repository path is per-distribution. Hardcodingnobleworks only for a noble base image and silently produces a404on any other, so it is read from the base image instead.rm -f /etc/apt/sources.list.d/*amdainic*— the base image already has one or more AINIC repositories enabled. This matters less for the build itself than for what happens afterwards: any laterapt installorapt upgradein a downstream layer can pulllibionicback to whichever bundle offers the highest version, silently undoing the rebuild. Removing the repositories makes the change durable.Selecting
$verfrom the requested bundle specifically — theindex($3, "/${AINIC_BUNDLE_VERSION} ")filter takes the version offered by that repository rather than the highest across all of them. Without the filter, a stale repository offering a higher version turns the pin into a request for the version you already have.Pinning
libionic-dev=$ver libionic1=$ver— an unpinnedapt installis a no-op when the requested bundle is older than what the base image carries; apt reportsalready the newest versionand exits 0.--allow-downgradesonly permits a downgrade, it does not request one.
The pinned version must be read with apt-cache madison, which reports only what
the repository offers. apt-cache policy ... Candidate cannot be used here: apt
refuses to nominate a downgrade as the candidate at any pin priority, so for an
older bundle it reports the installed version and the pin silently becomes a
no-op again.
The version assertion is what makes a silent failure loud#
test "$(dpkg-query -W -f='${Version}' libionic1)" = "$ver"; \
test "$(dpkg-query -W -f='${Version}' libionic-dev)" = "$ver";
These two lines are the only thing in the Dockerfile that compares what was
installed against what was requested. The structural checks in the second
RUN — dpkg -C, the provider symlink, readlink — all pass happily on an image
whose libionic never moved, because that image is perfectly self-consistent; it
is just built against the wrong bundle. Without the assertion, an unpinned install
that apt reported as already the newest version produces a successful build and
a wrong image.
No uninstall is required#
Upgrades and downgrades both apply in place. dpkg unpacks the new version over
the old one with 0 to remove, and dpkg -C stays clean afterwards.
This is worth stating explicitly because the library filename embeds its version
(libionic.so.1.1.54.0-187), which normally suggests old files would accumulate.
They do not: dpkg removes the previous versioned object and repoints both
libionic.so.1 and the libibverbs provider symlink libionic-rdmav34.so at the
new one. This holds across repeated changes, including moving out to another
bundle and back again.
6. Verify#
docker run --rm --privileged --network host --cap-add=IPC_LOCK \
-v /dev/infiniband:/dev/infiniband <image> bash -c '
dpkg-query -W -f="libionic1 \${Version}\n" libionic1
readlink -f /usr/lib/x86_64-linux-gnu/libionic.so.1
readlink -f /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so
dpkg -C && echo "dpkg consistent"
ibv_devices
'
Check that the package version and the provider symlink agree, and that
ibv_devices lists ionic devices.
Three traps when automating this check:
Treat “no
ionicdevices” as a failure, not a warning. An image whoselibioniccannot attach to the fabric passes every package-level check above. If the node does have AINIC hardware, an empty enumeration means the library is incompatible — see section 3.A successful enumeration does not mean the bundle matches the host. A container whose
libioniccomes from a different bundle than the host firmware will still enumerate every device normally. Enumeration detects an absent or grossly incompatible provider; it has no power to detect a version skew.Do not compare versions by parsing the
.sofilename. The soname scheme is not stable across bundles, so a literal prefix match rejects images that are in fact correct. Check package ownership instead, which is what you actually want to know and also catches a stale file:so=$(readlink -f /usr/lib/x86_64-linux-gnu/libionic.so.1) dpkg -L libionic1 | grep -qxF "$so" && echo "owned by installed libionic1"
Confirm the fabric is actually used#
The checks above prove the library you asked for is installed. They cannot prove it works with your host. Do not skip this step on the grounds that the build was clean and training runs — when RCCL cannot use the NIC it falls back to TCP, and the job still completes every step and converges to the same loss. The throughput cost of that fallback is severe, but nothing else in the training output distinguishes the two runs, so the transport has to be checked directly.
Run the job with RCCL logging enabled:
export NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET
Then confirm in the log that an ionic device was selected and that the transport
is not NET/Socket:
grep -oE 'NET/IB|NET/Socket' rank0.log | sort | uniq -c
grep -oiE 'using network [A-Za-z-]+' rank0.log | sort -u
grep -oE 'ionic_[0-9]+' rank0.log | sort -u
Any NET/Socket in the selected path is a failure, regardless of whether the
job completes.
NCCL_NET_PLUGIN is set for you and the expected success signature depends on
which value is in effect, so check the one that applies:
|
Expected log line |
Notes |
|---|---|---|
|
|
RCCL’s built-in verbs path. Also emits a |
|
|
The external ANP plugin. May emit |
Primus selects between these in
runner/helpers/hooks/03_enable_ainic.sh, which sets NCCL_NET_PLUGIN=none on
ROCm builds where ANP is compiled into RCCL. Let the hook decide — an
explicitly exported NCCL_NET_PLUGIN always wins and will override it. See
Multi-node networking for
the full picture.
7. Installing from a pre-downloaded bundle#
If the bundle is on repo.radeon.com, use the apt path in
section 5. That is the supported way to rebuild
against a published bundle: apt selects the packages that directory actually
ships.
Use this section when you only have a tarball. The helpers install libionic
from that file, but tarball layout is not a stable contract. Bundles handed
around as ainic_bundle_*.tar* are often pre-release (or mixed) artifacts:
inner archives may be misnamed, gzip despite a .tar.xz suffix, or include
both a release-style rdma-core-debs archive and an extra dated
libionic-debs copy that apt would not install. Expect sharp edges; if the
same version is published, prefer apt.
Do not unpack the tarball on the host. Copy it into the image and let the
build unpack it: that picks the distribution-matching .deb files from
inside the container.
The helpers live in tools/ainic-bundle-rebuild/ (Dockerfile, build.sh,
test.sh). Those files are the source of truth; this page only reproduces the
Dockerfile so the install RUN can be explained. From the Primus repository
root:
./tools/ainic-bundle-rebuild/build.sh \
rocm/jax-training:maxtext-v26.7 \
ainic_bundle_1.117.5-a-147.tar.gz
./tools/ainic-bundle-rebuild/test.sh \
jax-training:maxtext-v26.7-ainic-1.117.5-a-147
Symptoms of the sharp edges (wrong inner archive, Pre-Depends, gzip named
.xz) are recorded in Appendix A. You
should not need it to follow this section, only to recognise a failure.
Dockerfile#
Use tar -xf rather than tar -xJf throughout: an inner archive is sometimes
gzip regardless of its .tar.xz name, and -xf detects the format either way.
ARG BASE_IMAGE
FROM ${BASE_IMAGE}
ARG AINIC_BUNDLE
RUN mkdir /tmp/ainic
COPY ${AINIC_BUNDLE} /tmp/ainic/bundle.tar.gz
RUN set -ex; \
rm -f /etc/apt/sources.list.d/*amdainic*; \
cd /tmp/ainic; \
tar -xf bundle.tar.gz; \
cd */; \
tar -xf host_sw_pkg.tar*; \
cd host_sw_pkg/ionic_driver/deb; \
tar -xf rdma-core-debs.tar* 2>/dev/null || tar -xf libionic-debs.tar*; \
REL_NAME="$(. /etc/os-release; echo "$VERSION_CODENAME")"; \
dpkg -i --force-overwrite "$REL_NAME"/libionic1_*.deb; \
dpkg -i --force-overwrite "$REL_NAME"/libionic-dev_*.deb; \
cd /tmp; \
rm -rf /tmp/ainic
# Structural check: the package is installed consistently and the provider
# symlink the fabric actually loads points at the installed object.
RUN set -eux; \
dpkg-query -W -f='${Package} ${Version}\n' libionic1 libionic-dev; \
dpkg -C; \
ls /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav*.so; \
readlink -f /usr/lib/x86_64-linux-gnu/libionic.so.1
The install RUN is doing four things that are easy to flatten incorrectly:
Unpack inside the image, then install from
$VERSION_CODENAME/. The tarball extracts to one directory per distribution. Installing from the codename that matches this image avoids handingdpkgconflicting versions fromjammy/andnoble/at once. The archive carryinglibionicis not named consistently: we have seenrdma-core-debs.tar*,libionic-debs.tar*, or both. The Dockerfile prefersrdma-core-debswhen present so a mixed tarball does not silently install a leftover dated copy; it falls back tolibionic-debsif that is all there is. That is a heuristic, not a guarantee that the next tarball will look the same.linux-debs.tar.xzandperftest-debs.tar.xzin the same directory are host driver and benchmark packages and must not be installed in the container.Two separate
dpkg -iinvocations,libionic1first. Wherelibionic-devPre-Dependson an exactlibionic1version, a singledpkg -igiven both files cannot satisfy it: a pre-dependency must be met by an already-configured package, which one pass over both archives cannot arrange. It fails withpre-dependency problem - not installing libionic-devafter having already replacedlibionic1, leaving the layer half-converted. Installing in two steps costs nothing when there is no pre-dependency.--force-overwrite. The barelibionic.sosymlink is not always owned by the same package — it has been seen shipped bylibionic1and bylibionic-dev— and the packages declare noReplacesfor each other. Where the owning package changes, the install fails withtrying to overwrite '/usr/lib/x86_64-linux-gnu/libionic.so', which is also in package .... The flag is harmless when ownership does not change.rm -f /etc/apt/sources.list.d/*amdainic*. More important here than on the apt path. A bundle installed from local files exists in no repository, and its version may well sort below what the base image’s repositories offer, in which case apt regards the bundle you just replaced as an upgrade. Any laterapt installorapt upgradein a downstream layer then quietly reverts your bundle, anddpkg -Cstill reports a consistent system afterwards. There is no diagnostic signal at all. Removing the repositories is the only thing that makes a local install durable.
dpkg -i resolves nothing. If the .deb Depends/Pre-Depends are not already
satisfied in the base image — some builds pin a narrow ibverbs-providers range
against a newer rdma-core — the install fails rather than pulling
dependencies. Inspect with dpkg-deb -f on the extracted packages if you need
to see why.
There is no $ver from a repository on this path, so the version assertion from
section 5 does not carry over. The structural RUN still applies: it prints
whatever libionic landed. Trust that output, not the tarball filename.
Build and test helpers#
AINIC_BUNDLE is a path relative to the Docker build context, so the bundle
file has to be visible to the daemon. tools/ainic-bundle-rebuild/build.sh
creates a temporary context containing only that tarball, then tags the
image from the bundle filename. tools/ainic-bundle-rebuild/test.sh wraps
the package-level half of section 6. It does not replace
the runtime ibv_devices / RCCL log check on a node with AINIC hardware;
ibv_devices will be empty if you are not on such a node.
8. Troubleshooting#
Symptom |
Cause |
|---|---|
|
Target bundle is older than the installed |
Build fails on |
Working as intended — the install did not move |
Built image still has the old |
Version assertion missing from the install step. See section 5. |
|
Old bundle repositories left enabled. See section 5, and section 7 for why this is worse after a tarball install. |
|
Expected for a downgrade; add |
|
Bundle directory is an empty placeholder, the name is wrong, or the distribution codename in the repository line does not match the base image. See sections 4 and 5. |
Inner |
Tarball inner archive is gzip despite its name. Use |
Installed |
Tarball contained more than one |
|
Both |
|
The two bundles disagree about which package owns that symlink. Add |
|
Not AINIC hardware; the host |
Training runs fine but throughput is far below expectation |
RCCL fell back to |
The requested |
The installed bundle is not compatible with the host. Bundle selection is a cluster question — check |
Appendix A: packaging variants#
Not required reading. Section 7’s helpers already paper over the cases below. This list is so you can recognise a failure; it is not a spec of how the next tarball will be packaged.
Published apt repos are the consistent interface. Tarballs may be
pre-release or mixed: they have shown more than one inner archive name, gzip
files named .tar.xz, extra dated .debs next to the packages apt would
install, Pre-Depends, and per-distro version strings. Some of those same
package traits also appear in published apt (for example 1.117.5-a-196).
The difference is that apt only offers one libionic1 per bundle, while a
tarball can contain leftovers. Verify with dpkg-query / dpkg-deb -f.
|
Dated style (e.g. |
|
|---|---|---|
|
|
|
Apt pool path |
|
|
Tarball archive |
|
|
|
typically |
|
|
none |
|
A tarball can contain both archives. 1.117.5-a-147 does: rdma-core-debs
matches the published 54.0-197-1 packages, while libionic-debs is a dated
copy that apt does not install. Section 7 prefers rdma-core-debs when
present so the leftover copy is not the one that lands. That heuristic can
still be wrong on a tarball we have not seen.
What each one does to you#
Dated version scheme. A dated version sorts below
54.0-NNN, so apt regards whatever the base image’s repositories offer as an upgrade. This is the mechanism behind the silent revert described in section 7 — and it is why removing the repositories matters more for a local install than for an apt one. Do not expect the version to resemble the bundle name; read it withdpkg-queryordpkg-deb -f.A different version per distribution. Both styles extract to one directory per codename. The
54.0-NNNstyle puts the same version in every directory; the dated style does not. Installing everylibionic1_*.debtherefore handsdpkggenuinely conflicting versions, and all but one will fail the base image’sibverbs-providersconstraint — noble’s dated packages pinibverbs-providers (>= 50.0), (<< 51). Section 7 installs from$VERSION_CODENAME/inside the image so only the matching pair is used.Pre-Dependsrather thanDepends. Producesdpkg: error processing archive ... pre-dependency problem - not installing libionic-devfrom a singledpkg -igiven both files, afterlibionic1has already been replaced. Fixed by the two-invocation form in section 7.libionic.soowned by a different package. Producesdpkg: trying to overwrite '/usr/lib/x86_64-linux-gnu/libionic.so', which is also in package libionic-dev. Neither package declares aReplacesfor the other, sodpkgis right to refuse. Fixed by--force-overwritein section 7.Archive misnamed
.tar.xzwhile being gzip.tar -xJffails withxz: (stdin): File format not recognized. Usetar -xf, which detects either.
The dated scheme also appears as a dbgsym next to 54.0-NNN packages (for
example on 1.117.5-a-147). On 1.117.5-a-196 the dated form is the
published libionic1. Treat the tarball filename as a hint, not as proof of
which of those you installed.