Package-level declarations

The MessageTokenizer feature is responsible for handling tokenization of messages using a provided Tokenizer implementation. It serves as a feature that can be installed into an AIAgentPipeline. The tokenizer behavior can be configured with caching or on-demand tokenization based on the provided configuration.

Configuration class for message tokenization settings.

Provides access to the PromptTokenizer instance used within the AI agent's context.