MessageTokenizer

class MessageTokenizer(val promptTokenizer: PromptTokenizer)(source)

The MessageTokenizer feature is responsible for handling tokenization of messages using a provided Tokenizer implementation.

Constructors

Link copied to clipboard
constructor(promptTokenizer: PromptTokenizer)

Types

Link copied to clipboard

Companion object implementing agent feature, handling MessageTokenizer creation and installation.

Properties

Link copied to clipboard

An instance of PromptTokenizer used to process tokenization of messages and prompts.