OnDemandTokenizer

An implementation of the PromptTokenizer interface that delegates token counting to an instance of the Tokenizer interface. The class provides methods to estimate the token count for individual messages and for the entirety of a prompt.

This is useful in contexts where token-based costs or limitations are significant, such as when interacting with large language models (LLMs).