Interface Tokenizer<T>
-
- Type Parameters:
T
- given type
- All Known Implementing Classes:
RegexTokenizer
interface Tokenizer<T>
A tokenizer. Can produce arrays of tokens from a given type.- Since:
- 1.0
-
-
Method Summary
All Methods Instance Methods Abstract Methods Modifier and Type Method Description T[]
tokenize(java.lang.CharSequence text)
Returns an array of tokens.
-
-
-
Method Detail
-
tokenize
T[] tokenize(java.lang.CharSequence text)
Returns an array of tokens.- Parameters:
text
- input text- Returns:
- array of tokens
-
-