tokenizer

The tokenizer property determines the strategy used for tokenizing text and estimating token counts within a message-processing feature.

This property allows overriding the default tokenization behavior by specifying a custom Tokenizer implementation. By default, the NoTokenizer instance is used, which effectively disables token counting by always returning zero.

Tokenizers play a key role in scenarios involving large language models (LLMs), where accurate token counting can be essential for understanding and managing resource usage or request limits.