byte_pair_encode

2026-03-11

19 min read time

Applies to Linux

class pylibhipdf.nvtext.byte_pair_encode.BPEMergePairs

Bases: object

The table of merge pairs for the BPE encoder.

For details, see cudf::nvtext::bpe_merge_pairs.

pylibhipdf.nvtext.byte_pair_encode.byte_pair_encoding(Column input, BPEMergePairs merge_pairs, Scalar separator=None, Stream stream=None) → Column

Byte pair encode the input strings.

For details, see cpp:func:cudf::nvtext::byte_pair_encoding

Parameters

inputColumn

Strings to encode.

merge_pairsBPEMergePairs

Substrings to rebuild each string on.

separatorScalar

String used to build the output after encoding. Default is a space.

streamStream | None

CUDA stream on which to perform the operation.

Returns

Column

An encoded column of strings.