hipdf.core.wordpiece_tokenize.WordPieceVocabulary

19 min read time

Applies to Linux

class hipdf.core.wordpiece_tokenize.WordPieceVocabulary(vocabulary: Series)

Bases: object

A vocabulary object used to tokenize input text.

Parameters

vocabularycudf.Series

Strings column of vocabulary terms

__init__(vocabulary: Series) → None

Methods

__init__(vocabulary)
tokenize(text[, max_words_per_row])	Produces tokens for the input strings.

__init__(vocabulary: Series) → None

tokenize(text, max_words_per_row: int = 0) → Series

Produces tokens for the input strings. The input is expected to be the output of NormalizeCharacters or a similar normalizer.

Parameters

textcudf.Series

Normalized strings to be tokenized.

max_words_per_rowint

Maximum number of words to tokenize per row. Default 0 tokenizes all words.

Returns

cudf.Series

Token values