ngrams_tokenize

2026-03-11

19 min read time

Applies to Linux

pylibhipdf.nvtext.ngrams_tokenize.ngrams_tokenize(Column input, size_type ngrams, Scalar delimiter, Scalar separator, Stream stream=None) → Column

Returns a single column of strings by tokenizing the input strings column and then producing ngrams of each string.

For details, see ngrams_tokenize()

Parameters

inputColumn

Input strings

ngramssize_type

The ngram number to generate

delimiterScalar

UTF-8 characters used to separate each string into tokens. An empty string will separate tokens using whitespace.

separatorScalar

The string to use for separating ngram tokens

streamStream | None

CUDA stream on which to perform the operation.

Returns

Column

New strings columns of tokens