prompt-tokenizer/ai.koog.prompt.tokenizer.tiktoken/TiktokenEncoder/countTokens

countTokens

open override fun countTokens(text: String): Int(source)

Counts the number of tokens in the given text.

The method utilizes the encodeAsIds function to tokenize the input string and determines the number of resulting tokens.

Return

The total number of tokens in the input text.

Parameters

text

The input text to be tokenized and counted.