countTokens

open override fun countTokens(text: String): Int(source)

Counts tokens by splitting on whitespace and common punctuation.

The implementation adds a small overhead factor (1.1x) to account for special tokens and tokenization differences in actual LLM tokenizers.

The estimated number of tokens in the text

text

The text to tokenize