tokenizer

Provides access to the PromptTokenizer instance used within the AI agent's context.

This property retrieves the tokenizer from the agent's storage using the MessageTokenizer.Feature, which must be initialized in the pipeline's features. The PromptTokenizer allows for tokenization operations on prompts and messages during the agent's execution.

It facilitates operations such as calculating token counts for messages and prompts, which are critical in managing and optimizing interactions with language models.

Throws an exception if the MessageTokenizer.Feature is not available in the context.