Interface Tokenizer<T>

Type Parameters:
T - given type
All Known Implementing Classes:
RegexTokenizer

interface Tokenizer<T>
A tokenizer. Can produce arrays of tokens from a given type.
Since:
1.0
  • Method Summary

    Modifier and Type
    Method
    Description
    T[]
    Returns an array of tokens.
  • Method Details

    • tokenize

      T[] tokenize(CharSequence text)
      Returns an array of tokens.
      Parameters:
      text - input text
      Returns:
      array of tokens