hipdf.core.character_normalizer.CharacterNormalizer

19 min read time

Applies to Linux

class hipdf.core.character_normalizer.CharacterNormalizer(do_lower: bool, special_tokens: Series | None = None)

Bases: object

A normalizer object used to normalize input text.

Parameters

do_lowerbool

If True, the normalizer should also lower-case while normalizing.

special_tokenscudf.Series

Series of special tokens.

__init__(do_lower: bool, special_tokens: Series | None = None) → None

Methods

__init__(do_lower[, special_tokens])
normalize(text)	Parameters text cudf.Series The strings to be normalized.

__init__(do_lower: bool, special_tokens: Series | None = None) → None

normalize(text: Series) → Series

Parameters

textcudf.Series

The strings to be normalized.

Returns

cudf.Series

Normalized strings