A tokenizer implementation that always returns 0.
A simple regex-based tokenizer that splits text on whitespace and common punctuation.
Interface for tokenizing text and counting tokens.