prompt-tokenizer/ai.koog.prompt.tokenizer/Tokenizer/countTokens

countTokens

abstract fun countTokens(text: String): Int(source)

Counts the number of tokens in the given text.

Return

The estimated number of tokens in the text

Parameters

text

The text to tokenize and count