agents-features-tokenizer/ai.koog.agents.features.tokenizer.feature/MessageTokenizer

MessageTokenizer

class MessageTokenizer(val promptTokenizer: PromptTokenizer)(source)

The MessageTokenizer feature is responsible for handling tokenization of messages using a provided Tokenizer implementation.

Constructors

MessageTokenizer

constructor(promptTokenizer: PromptTokenizer)

Types

object Feature : AIAgentGraphFeature<MessageTokenizerConfig, MessageTokenizer> , AIAgentFunctionalFeature<MessageTokenizerConfig, MessageTokenizer>

Companion object implementing agent feature, handling MessageTokenizer creation and installation.

Properties

promptTokenizer

val promptTokenizer: PromptTokenizer

An instance of PromptTokenizer used to process tokenization of messages and prompts.