Package-level declarations

A caching implementation of the PromptTokenizer interface that optimizes token counting by storing previously computed token counts for messages. This reduces redundant computations when the same message is processed multiple times.

A tokenizer implementation that always returns 0.

An implementation of the PromptTokenizer interface that delegates token counting to an instance of the Tokenizer interface. The class provides methods to estimate the token count for individual messages and for the entirety of a prompt.

An interface that provides utilities for tokenizing and calculating token usage in messages and prompts.

A simple regex-based tokenizer that splits text on whitespace and common punctuation.

Interface for tokenizing text and counting tokens.