agents-features-tokenizer/ai.koog.agents.features.tokenizer.feature/MessageTokenizer

MessageTokenizer

class MessageTokenizer(val promptTokenizer: PromptTokenizer)(source)

The MessageTokenizer feature is responsible for handling tokenization of messages using a provided Tokenizer implementation. It serves as a feature that can be installed into an AIAgentPipeline. The tokenizer behavior can be configured with caching or on-demand tokenization based on the provided configuration.

Constructors

MessageTokenizer

constructor(promptTokenizer: PromptTokenizer)

Types

Feature

object Feature : AIAgentFeature<MessageTokenizerConfig, MessageTokenizer>

Companion object implementing the AIAgentFeature interface for the MessageTokenizer feature. This feature integrates a message tokenizer into the agent pipeline, allowing for tokenization of input messages. It supports both caching and non-caching tokenization strategies based on the configuration.

Properties

promptTokenizer

val promptTokenizer: PromptTokenizer

An instance of PromptTokenizer used to process tokenization of messages and prompts.