# AINIC bundle versions

AMD published training images ship a **fixed AINIC bundle**. The bundle version is
baked in at build time through the `AINIC_BUNDLE_VERSION` build argument, and the
image installs the userspace half of the AINIC stack — the `libionic` library —
from `https://repo.radeon.com/amdainic/pensando/ubuntu/<bundle>/`.

Sometimes the bundle in the image is not the one your cluster needs. This guide
shows how to rebuild a published image against an arbitrary bundle, and how to
confirm the change landed. Upgrades and downgrades are both normal operations
here; the correct bundle for your cluster is often an older one.

> **Scope.** This guide covers **making the change** to `libionic` inside the
> container, and verifying it took effect. **Deciding which bundle you need is
> your responsibility** — `libionic` in the container has to be compatible with
> the `ionic` driver on the host. See [section 3](#3-choosing-a-bundle), which
> includes host commands that often answer the question directly.
>
> This guide does not change the host driver, and does not change how AINIC is
> *enabled* at runtime — for that, see
> [Multi-node networking](./multi-node-networking.md#4-ainic-amd-ai-nic).
>
> This guide assumes a **published** bundle throughout. If you have been given a
> bundle as a tarball rather than a repository, see
> [section 7](#7-installing-from-local-deb-files).
>
> **What this was tested on.** The MaxText JAX training images —
> `rocm/jax-training:maxtext-v26.3.2`, `-v26.4`, and `-v26.6`. The rebuild was
> exercised on all three; the runtime check in [section 6](#6-verify) on
> `v26.3.2`. It has **not** been tested on Primus's own `rocm/primus:*` images.
> Nothing here is specific to MaxText — it operates on `libionic` and the apt
> configuration, not on the framework — but a different image may enable
> different bundle repositories or ship a different `libionic` to begin with, so
> confirm with section 6 rather than assuming the versions quoted here.

---

## Quickstart

If you already know which bundle you need, this is the whole procedure. Save the
block below as `Dockerfile` in an otherwise empty directory — the whole directory
is sent to the Docker daemon as build context, so keeping it empty keeps the
build fast.

```dockerfile
ARG BASE_IMAGE
FROM ${BASE_IMAGE}

ARG AINIC_BUNDLE_VERSION

RUN set -eux; \
    codename="$(. /etc/os-release; echo "$VERSION_CODENAME")"; \
    rm -f /etc/apt/sources.list.d/*amdainic*; \
    add-apt-repository -y "deb https://repo.radeon.com/amdainic/pensando/ubuntu/${AINIC_BUNDLE_VERSION} $codename main"; \
    apt update --allow-insecure-repositories; \
    ver="$(apt-cache madison libionic1 \
            | awk -F'|' -v b="/${AINIC_BUNDLE_VERSION} " 'index($3,b){gsub(/ /,"",$2); print $2; exit}')"; \
    test -n "$ver"; \
    echo "installing libionic $ver from ${AINIC_BUNDLE_VERSION}"; \
    apt install -y --allow-unauthenticated --allow-downgrades \
        "libionic-dev=$ver" "libionic1=$ver"; \
    test "$(dpkg-query -W -f='${Version}' libionic1)" = "$ver"; \
    test "$(dpkg-query -W -f='${Version}' libionic-dev)" = "$ver"; \
    rm -rf /var/lib/apt/lists/*

# Structural check: the package is installed consistently and the provider
# symlink the fabric actually loads points at the installed object.
RUN set -eux; \
    dpkg-query -W -f='${Package} ${Version}\n' libionic1 libionic-dev; \
    dpkg -C; \
    test -e /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so; \
    readlink -f /usr/lib/x86_64-linux-gnu/libionic.so.1
```

Then, from that same directory:

```bash
BASE=rocm/jax-training:maxtext-v26.7
BUNDLE=1.117.5-a-147

docker build --network host \
  --build-arg BASE_IMAGE=$BASE \
  --build-arg AINIC_BUNDLE_VERSION=$BUNDLE \
  -t ${BASE##*/}-ainic-$BUNDLE .
```

`--network host` is needed when the daemon's default bridge network cannot
resolve `repo.radeon.com`; it is harmless otherwise, so it is included by
default.

Neither build argument has a default, so omitting one fails the build rather than
silently producing an unintended image. Tagging the image after the bundle is
worth the small effort: the bundle is otherwise invisible without running
`dpkg-query` inside the image.

**A clean build does not mean the image works on your host.** It means the
version you asked for was installed. See [section 6](#6-verify) before trusting
it, and [section 5](#5-how-the-rebuild-works) for why the apparently boilerplate
lines are load-bearing.

---

## 1. What is actually in the image

An AINIC-enabled training image contains only the **userspace** half of the pair:

| Package | Role |
|---------|------|
| `libionic1` | User space ionic provider driver for `libibverbs` |
| `libionic-dev` | Development headers for the ionic provider |

The driver-side packages of the bundle — `ionic-dkms`, `pds-dkms`, `nicctl`,
`ainic-monitor` — are **not** installed in the container. They belong to the host.
This is why bundle mismatches show up as a host/container skew rather than as a
missing package.

Note that the bundle name and the `libionic` version are **different numbering
schemes**, and they are easy to confuse when discussing versions. The excerpt
below is ordered by bundle version, newest first; the `libionic` column does not
follow, because a higher bundle version does not imply a higher `libionic`
version. See [section 4](#4-list-published-bundles) to list every published
bundle.

| Bundle | `libionic` version |
|--------|--------------------|
| `1.125.0-a-187` | `54.0-192-1` |
| `1.117.5-a-147` | `54.0-197-1` |
| `1.117.5-a-77` | `54.0-187-1` |
| `1.117.5-a-56` | `54.0-184` |
| `1.117.1-a-63` | `54.0-149.g3304be71` |

Every published bundle uses this `54.0-NNN` scheme and ships one `libionic`
version for all distributions. A bundle delivered as a tarball may not — read the
version out of the `.deb` itself rather than assuming the scheme
([section 7](#7-installing-from-local-deb-files)).

## 2. Check what you have

```bash
# libionic version actually installed in the image
docker run --rm <image> dpkg-query -W -f='${Package} ${Version}\n' libionic1 libionic-dev

# which bundle repositories the image was built against
docker run --rm <image> grep -rh amdainic /etc/apt/sources.list.d/
```

**The repository list is not a reliable indicator of the installed version.** An
image can carry several AINIC repositories, and `apt` resolves `libionic` to the
highest version across *all* enabled repositories. An image whose Dockerfile names
one bundle can therefore ship the library of a different one. Always trust
`dpkg-query`, not the `sources.list` entries or the build argument.

On the **host**, check the other half of the pair:

```bash
lspci | grep -i pensando
modinfo ionic | grep -E '^version|^srcversion'
ls /sys/class/infiniband/          # expect ionic_* entries
```

## 3. Choosing a bundle

`libionic` in the container must be compatible with the `ionic` driver on the
host. Start by asking the host, which frequently answers the question outright:

```bash
# Often reports the AINIC bundle version directly, e.g. 1.117.5-a-77
cat /sys/class/infiniband/*/fw_ver

# Same information from the network device side
ethtool -i <ionic_netdev> | grep -i firmware

# The host driver build, which tracks the bundle's linux-ionic component
modinfo ionic | grep '^version'
```

If `fw_ver` reports a bundle version, that is the bundle the host is running and
the natural target for the container. Where it does not, or where the host is
managed by someone else, bring your cluster's operators the output above together
with:

```bash
cat /sys/class/infiniband_verbs/uverbs*/abi_version
```

This is the kernel side of the uverbs ABI, the interface `libionic` has to speak.
There is no matching number you can read out of the container to compare it
against — a provider library declares the ABI range it supports internally, not
as a queryable version. Note also that this value is **coarse**: hosts running
different bundles can report the same ABI, so it will not distinguish two
candidate bundles from each other. Treat it as an input to the conversation with
your operators, not as a selection test. The authoritative test is the runtime one
in [section 6](#6-verify).

One thing worth knowing before you pick: **newer is not automatically better.**
Compatibility is not monotonic in the bundle version, so the correct target may
well be older than what the image already ships.

If you install a bundle the host cannot work with, the build still succeeds and
every package-level check in section 6 still passes. Package-level checks cannot
see a host/container skew at all, which is why section 6 ends with a runtime
check. Note that the converse also holds: a skewed pair is **not** guaranteed to
fail visibly — userspace/firmware compatibility spans a range of versions rather
than requiring an exact match, so a mismatched pair may enumerate devices and
reach the fabric normally. Matching the bundle to the host remains the supported
configuration, and the runtime check tells you which situation you are in.

## 4. List published bundles

```bash
curl -s https://repo.radeon.com/amdainic/pensando/ubuntu/ | grep -oE 'href="[0-9][^"]+/"'
```

The `[0-9]` anchor keeps the page header and parent-directory links out of the
result, so every line is a bundle.

To see the exact package versions a bundle ships before committing to it:

```bash
BUNDLE=1.117.5-a-147
curl -s "https://repo.radeon.com/amdainic/pensando/ubuntu/${BUNDLE}/dists/noble/main/binary-amd64/Packages" \
  | awk '/^Package: /{p=$2} /^Version: /{print p"="$2}'
```

Some published directories are **empty placeholders** — they contain only `conf/`
and `db/`, with no `dists/` or `pool/`, and `apt` cannot install from them. The
`Packages` query above returns nothing for these, which is the quickest way to
spot one before a build fails.

Check for this even if you are deliberately picking the newest bundle: **the
newest entries in the listing are currently placeholders**, so choosing the
highest version is exactly the case that hits it. A build against a placeholder
fails at `apt update` with `404 Not Found` on the `Packages` file, before the
bundle version is ever evaluated.

This query also only works for published bundles. To read the version out of a
local `.deb` instead, see [section 7](#7-installing-from-local-deb-files).

## 5. How the rebuild works

Four lines in the quickstart Dockerfile look like boilerplate and are not.

1. **Deriving `$codename` from `/etc/os-release`** — the repository path is
   per-distribution. Hardcoding `noble` works only for a noble base image and
   silently produces a `404` on any other, so it is read from the base image
   instead.

2. **`rm -f /etc/apt/sources.list.d/*amdainic*`** — the base image already has one
   or more AINIC repositories enabled. This matters less for the build itself than
   for what happens afterwards: **any later `apt install` or `apt upgrade` in a
   downstream layer can pull `libionic` back to whichever bundle offers the
   highest version**, silently undoing the rebuild. Removing the repositories
   makes the change durable.

3. **Selecting `$ver` from the requested bundle specifically** — the
   `index($3, "/${AINIC_BUNDLE_VERSION} ")` filter takes the version offered by
   *that* repository rather than the highest across all of them. Without the
   filter, a stale repository offering a higher version turns the pin into a
   request for the version you already have.

4. **Pinning `libionic-dev=$ver libionic1=$ver`** — an unpinned `apt install` is a
   no-op when the requested bundle is *older* than what the base image carries;
   apt reports `already the newest version` and exits 0. `--allow-downgrades` only
   *permits* a downgrade, it does not request one.

The pinned version must be read with `apt-cache madison`, which reports only what
the repository offers. `apt-cache policy ... Candidate` cannot be used here: apt
refuses to nominate a downgrade as the candidate at any pin priority, so for an
older bundle it reports the *installed* version and the pin silently becomes a
no-op again.

### The version assertion is what makes a silent failure loud

```dockerfile
    test "$(dpkg-query -W -f='${Version}' libionic1)" = "$ver"; \
    test "$(dpkg-query -W -f='${Version}' libionic-dev)" = "$ver";
```

These two lines are the only thing in the Dockerfile that compares what was
*installed* against what was *requested*. The structural checks in the second
`RUN` — `dpkg -C`, the provider symlink, `readlink` — all pass happily on an image
whose `libionic` never moved, because that image is perfectly self-consistent; it
is just built against the wrong bundle. Without the assertion, an unpinned install
that apt reported as `already the newest version` produces a successful build and
a wrong image.

### No uninstall is required

Upgrades and downgrades both apply in place. `dpkg` unpacks the new version over
the old one with `0 to remove`, and `dpkg -C` stays clean afterwards.

This is worth stating explicitly because the library filename embeds its version
(`libionic.so.1.1.54.0-187`), which normally suggests old files would accumulate.
They do not: `dpkg` removes the previous versioned object and repoints both
`libionic.so.1` and the `libibverbs` provider symlink `libionic-rdmav34.so` at the
new one. This holds across repeated changes, including moving out to another
bundle and back again.

## 6. Verify

```bash
docker run --rm --privileged --network host --cap-add=IPC_LOCK \
  -v /dev/infiniband:/dev/infiniband <image> bash -c '
    dpkg-query -W -f="libionic1 \${Version}\n" libionic1
    readlink -f /usr/lib/x86_64-linux-gnu/libionic.so.1
    readlink -f /usr/lib/x86_64-linux-gnu/libibverbs/libionic-rdmav34.so
    dpkg -C && echo "dpkg consistent"
    ibv_devices
  '
```

Check that the package version and the provider symlink agree, and that
`ibv_devices` lists `ionic` devices.

Three traps when automating this check:

- **Treat "no `ionic` devices" as a failure, not a warning.** An image whose
  `libionic` cannot attach to the fabric passes every package-level check above.
  If the node does have AINIC hardware, an empty enumeration means the library is
  incompatible — see section 3.
- **A successful enumeration does not mean the bundle matches the host.** A
  container whose `libionic` comes from a different bundle than the host firmware
  will still enumerate every device normally. Enumeration detects an absent or
  grossly incompatible provider; it has no power to detect a version skew.
- **Do not compare versions by parsing the `.so` filename.** The soname scheme is
  not stable across bundles, so a literal prefix match rejects images that are in
  fact correct. Check package ownership instead, which is what you actually want
  to know and also catches a stale file:

  ```bash
  so=$(readlink -f /usr/lib/x86_64-linux-gnu/libionic.so.1)
  dpkg -L libionic1 | grep -qxF "$so" && echo "owned by installed libionic1"
  ```

### Confirm the fabric is actually used

The checks above prove the *library you asked for is installed*. They cannot prove
it works with your host. Do not skip this step on the grounds that the build was
clean and training runs — when RCCL cannot use the NIC it falls back to TCP, and
the job still completes every step and converges to the same loss. The throughput
cost of that fallback is severe, but nothing else in the training output
distinguishes the two runs, so the transport has to be checked directly.

Run the job with RCCL logging enabled:

```bash
export NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,NET
```

Then confirm in the log that an `ionic` device was selected and that the transport
is not `NET/Socket`:

```bash
grep -oE 'NET/IB|NET/Socket' rank0.log | sort | uniq -c
grep -oiE 'using network [A-Za-z-]+' rank0.log | sort -u
grep -oE 'ionic_[0-9]+' rank0.log | sort -u
```

**Any `NET/Socket` in the selected path is a failure**, regardless of whether the
job completes.

`NCCL_NET_PLUGIN` is set for you and the expected success signature depends on
which value is in effect, so check the one that applies:

| `NCCL_NET_PLUGIN` | Expected log line | Notes |
|---|---|---|
| `none` | `Using network IB` | RCCL's built-in verbs path. Also emits a `NET/Plugin: Could not find: none librccl-net-none.so.` line, which is **expected and harmless**. |
| `librccl-anp.so` | `Using network RCCL` | The external ANP plugin. May emit `Failed to find ncclCollNetPlugin_vN symbol` warnings, which are benign — the library provides a net plugin, not a collnet plugin. |

Primus selects between these in
`runner/helpers/hooks/03_enable_ainic.sh`, which sets `NCCL_NET_PLUGIN=none` on
ROCm builds where ANP is compiled into RCCL. **Let the hook decide** — an
explicitly exported `NCCL_NET_PLUGIN` always wins and will override it. See
[Multi-node networking](./multi-node-networking.md#rccl-network-plugin-anp) for
the full picture.

## 7. Installing from local `.deb` files

Occasionally a bundle is delivered as a tarball rather than published to
`repo.radeon.com`. **Prefer the apt path in [section 5](#5-how-the-rebuild-works)
whenever the bundle you need is published** — it handles either direction with no
extra flags and none of the care below.

Tarball-delivered bundles have been seen to differ from published ones in how
their packages are built, so the steps here are deliberately written to work
either way rather than assuming a particular layout. Read the version and the
dependency fields out of the files themselves instead of assuming them.

If the packaging of the bundle you were given does differ, the specific
differences observed in one internal build — and the error each produces — are
recorded in [Appendix A](#appendix-a-packaging-differences-observed-in-an-internal-build).
You should not need it to follow this section, only to recognise a symptom.

### Getting to the two `.deb` files

A bundle tarball does not contain loose `.deb` files. Extract in three stages, and
use `tar -xf` rather than `tar -xJf` throughout — an inner archive is sometimes
gzip regardless of its `.tar.xz` name, and `-xf` detects the format either way:

```bash
tar -xf ainic_bundle_<version>.tar*    # or .tar.gz
tar -xf host_sw_pkg.tar*
ls host_sw_pkg/ionic_driver/deb/       # find the archive carrying libionic
tar -xf host_sw_pkg/ionic_driver/deb/<name>-debs.tar.xz
```

**List that directory rather than guessing the filename.** It also holds
`linux-debs.tar.xz` and `perftest-debs.tar.xz`, which are the driver and
benchmark packages and belong on the host, not in the container. The archive
carrying `libionic` has been seen named both `rdma-core-debs.tar.xz` and
`libionic-debs.tar.xz`.

The result may be one directory per distribution codename. **Copy only the two
files matching your base image**, and name them explicitly — where several
variants are present, a wildcard `COPY` picks up all of them and hands `dpkg`
conflicting versions:

```bash
codename=$(docker run --rm <base-image> sh -c '. /etc/os-release; echo $VERSION_CODENAME')
cp "$codename"/libionic1_*.deb "$codename"/libionic-dev_*.deb .
```

To read the version out of the files, since the `Packages` query in section 4 does
not apply:

```bash
dpkg-deb -f libionic1_*.deb Package Version
```

Also check the package's own prerequisites against your base image before
building, because `dpkg -i` resolves nothing and will simply fail if they are not
already satisfied:

```bash
dpkg-deb -f libionic1_*.deb Pre-Depends Depends
```

Some builds pin a narrow `ibverbs-providers` range, which a base image with a
newer `rdma-core` will not satisfy.

### The install step

Replace **only** the first `RUN` of the quickstart Dockerfile. Keep the structural
verification `RUN` unchanged; it applies however the packages got in. The version
assertion from section 5 does not carry over — there is no `$ver` from a
repository — so substitute the expected version explicitly:

```dockerfile
COPY libionic1_<version>_amd64.deb libionic-dev_<version>_amd64.deb /tmp/
RUN set -eux; \
    rm -f /etc/apt/sources.list.d/*amdainic*; \
    dpkg -i --force-overwrite /tmp/libionic1_*.deb; \
    dpkg -i --force-overwrite /tmp/libionic-dev_*.deb; \
    test "$(dpkg-query -W -f='${Version}' libionic1)" = "<version>"; \
    rm /tmp/*.deb
```

Three things differ from a naive `dpkg -i` of both files at once, and all three
are required:

1. **Two separate invocations, `libionic1` first.** Where `libionic-dev`
   **`Pre-Depends`** on an exact `libionic1` version, a single `dpkg -i` given
   both files cannot satisfy it: a pre-dependency must be met by an
   already-*configured* package, which one pass over both archives cannot
   arrange. It fails with `pre-dependency problem - not installing libionic-dev`
   *after* having already replaced `libionic1`, leaving the layer half-converted.
   Check with `dpkg-deb -f libionic-dev_*.deb Pre-Depends`; installing in two
   steps costs nothing when there is no pre-dependency.

2. **`--force-overwrite`.** The bare `libionic.so` symlink is not always owned by
   the same package — it has been seen shipped by `libionic1` and by
   `libionic-dev` — and the packages declare no `Replaces` for each other. Where
   the owning package changes, the install fails with
   `trying to overwrite '/usr/lib/x86_64-linux-gnu/libionic.so', which is also in package ...`.
   The flag is harmless when ownership does not change.

3. **`rm -f /etc/apt/sources.list.d/*amdainic*`.** More important here than on the
   apt path. A bundle installed from local files exists in no repository, and its
   version may well sort *below* what the base image's repositories offer, in
   which case apt regards the bundle you just replaced as an upgrade. Any later
   `apt install` or `apt upgrade` in a downstream layer then quietly reverts your
   bundle, and `dpkg -C` still reports a consistent system afterwards. There is no
   diagnostic signal at all. Removing the repositories is the only thing that
   makes a local install durable.

## 8. Troubleshooting

| Symptom | Cause |
|---------|-------|
| `apt install` reports `already the newest version` and nothing changes | Target bundle is older than the installed `libionic` and the version was not pinned. See section 5. |
| Build fails on `test "$(dpkg-query ...)" = "$ver"` | Working as intended — the install did not move `libionic` to the requested version. Check the pin and the repository filter. See section 5. |
| Built image still has the old `libionic`, build reported success | Version assertion missing from the install step. See section 5. |
| `libionic` reverts to another version after a later `apt` command | Old bundle repositories left enabled. See section 5, and section 7 for why this is worse after a local `.deb` install. |
| `E: Packages were downgraded and -y was used without --allow-downgrades` | Expected for a downgrade; add `--allow-downgrades`. |
| `apt update` fails with `404 Not Found` on `Packages` | Bundle directory is an empty placeholder, the name is wrong, or the distribution codename in the repository line does not match the base image. See sections 4 and 5. |
| `xz: (stdin): File format not recognized` while extracting a bundle | Inner archive is gzip despite its `.tar.xz` name. Use `tar -xf`. See section 7. |
| `dpkg: pre-dependency problem - not installing libionic-dev` | Both `.deb` files given to one `dpkg -i`. Install `libionic1` first, in its own invocation. See section 7. |
| `dpkg: trying to overwrite '.../libionic.so', which is also in package ...` | The two bundles disagree about which package owns that symlink. Add `--force-overwrite`. See section 7. |
| `ibv_devices` lists no `ionic` device | Not AINIC hardware; the host `ionic` driver is not loaded; or `libionic` is ABI-incompatible with the host driver. See sections 2 and 3. |
| Training runs fine but throughput is far below expectation | RCCL fell back to `NET/Socket`. The job still completes and converges normally, so check the transport rather than the loss. See section 6. |
| The requested `libionic` is installed but the fabric is unreachable | The installed bundle is not compatible with the host. Bundle selection is a cluster question — check `fw_ver` on the host, and note that the required version may be older than what the image shipped. See section 3. |

---

## Appendix A: packaging differences observed in an internal build

**Not required reading.** Section 7's commands already handle everything here, and
they are written so that none of it has to be known in advance. This appendix
exists so that if you hit one of these errors, you can recognise it rather than
debug it from scratch.

**Sample size is one.** These are observations from a single internal build,
compared against the published bundles available at the time of writing. They are
**not** a specification of how internal builds are packaged, and there is no reason
to assume the next one will look the same. Verify against your own files with
`dpkg-deb -f` and `dpkg-deb -c`.

| | Published bundles (all inspected) | The internal build inspected |
|---|---|---|
| `libionic` version scheme | `54.0-NNN` | dated, `<maj>.<min>.YY.MM.DD.<build>-1~<distro>` |
| Per-distro directories | yes — `bookworm`, `buster`, `jammy`, `noble`, all shipping the *same* version | yes — six, each shipping a *different* version string |
| `libionic-dev` → `libionic1` | `Depends` | `Pre-Depends` |
| `libionic1` `Replaces:` | none | `ibverbs-providers, libibverbs1` |
| Owner of bare `libionic.so` | `libionic1` | `libionic-dev` |
| Archive carrying `libionic` | `rdma-core-debs.tar.xz`, real xz | `libionic-debs.tar.xz`, gzip despite the name |

### What each one does to you

- **Dated version scheme.** The consequence is not cosmetic: a dated version sorts
  *below* the `54.0-NNN` scheme, so apt regards whatever the base image's
  repositories offer as an **upgrade**. This is the mechanism behind the silent
  revert described in section 7 — and it is why removing the repositories matters
  more for a local install than for an apt one. Do not expect the version to
  resemble the bundle name; read it with `dpkg-deb -f`.
- **A different version per distribution.** Both kinds of bundle extract to one
  directory per codename, so that layout alone is unremarkable — but where the
  published bundle puts the *same* version in every directory, this one puts a
  different version string in each (`50.0.…-1~ubu24.04` under `noble`,
  `39.0.…-1~ubu22.04` under `jammy`, and so on). A wildcard `COPY libionic1_*.deb`
  therefore hands `dpkg` genuinely conflicting versions rather than duplicates, and
  all but one will fail the base image's `ibverbs-providers` constraint — `noble`'s
  pins `ibverbs-providers (>= 50.0), (<< 51)`. Copy the two files from the
  directory matching your base image's codename, by name.
- **`Pre-Depends` rather than `Depends`.** Produces
  `dpkg: error processing archive ... pre-dependency problem - not installing libionic-dev`
  from a single `dpkg -i` given both files, *after* `libionic1` has already been
  replaced. Fixed by the two-invocation form in section 7.
- **`libionic.so` owned by a different package.** Produces
  `dpkg: trying to overwrite '/usr/lib/x86_64-linux-gnu/libionic.so', which is also in package libionic-dev`.
  Neither package declares a `Replaces` for the other, so `dpkg` is right to
  refuse. Fixed by `--force-overwrite` in section 7.
- **Archive misnamed `.tar.xz` while being gzip.** `tar -xJf` fails with
  `xz: (stdin): File format not recognized`. Use `tar -xf`, which detects either.

### One hint that the dated scheme is a build-pipeline artifact

The dated scheme is visible in the published repositories too, but only on a
debug-symbol package rather than a shipped one:

```bash
curl -s "https://repo.radeon.com/amdainic/pensando/ubuntu/1.117.5-a-147/dists/noble/main/binary-amd64/Packages" \
  | awk '/^Package: /{p=$2} /^Version: /{print p"="$2}' | grep libionic
```

```
libionic-dev=54.0-197-1
libionic1=54.0-197-1
libionic1-dbgsym=22.1.26.07.10.001-1~deb10
```

The deliverables carry release versions; the `dbgsym` carries a dated one of the
same shape. So the dated form appears to be what the build pipeline emits before
release versioning is applied, which is consistent with seeing it on a build that
was never published — and is a reason to treat it as a signal that you are holding
a pre-release artifact rather than a released bundle.

---

## Related documentation

- [Multi-node networking](./multi-node-networking.md): enabling AINIC, ANP plugin, and NCCL/RCCL variables
- [Preflight diagnostics](../02-user-guide/preflight.md)
- [Deployment](../05-operations/deployment.md)
- [Troubleshooting](../05-operations/troubleshooting.md)