Package com.google.protobuf
Class TextFormat.Tokenizer
java.lang.Object
com.google.protobuf.TextFormat.Tokenizer
- Enclosing class:
- TextFormat
Represents a stream of tokens parsed from a
String.
The Java standard library provides many classes that you might think would be useful for implementing this, but aren't. For example:
java.io.StreamTokenizer: This almost does what we want -- or, at least, something that would get us close to what we want -- except for one fatal flaw: It automatically un-escapes strings using Java escape sequences, which do not include all the escape sequences we need to support (e.g. '\x').java.util.Scanner: This seems like a great way at least to parse regular expressions out of a stream (so we wouldn't have to load the entire input into a single string before parsing). Sadly,Scannerrequires that tokens be delimited with some delimiter. Thus, although the text "foo:" should parse to two tokens ("foo" and ":"),Scannerwould recognize it only as a single token. Furthermore,Scannerprovides no way to inspect the contents of delimiters, making it impossible to keep track of line and column numbers.
Luckily, Java's regular expression support does manage to be useful to us. (Barely: We need
Matcher.usePattern(), which is new in Java 1.5.) So, we can use that, at least.
Unfortunately, this implies that we need to have the entire input in one contiguous string.
-
Field Summary
FieldsModifier and TypeFieldDescriptionprivate intprivate booleancontainsSilentMarkerAfterCurrentTokenindicates if there is a silent marker after the current token.private booleanprivate Stringprivate static final Patternprivate static final Patternprivate static final Patternprivate intprivate final Matcherprivate intprivate intprivate intprivate final CharSequenceprivate static final Patternprivate static final Pattern -
Constructor Summary
ConstructorsModifierConstructorDescriptionprivateTokenizer(CharSequence text) Construct a tokenizer that parses tokens from the given text. -
Method Summary
Modifier and TypeMethodDescription(package private) booleanatEnd()Are we at the end of the input?(package private) voidIf the next token exactly matchestoken, consume it.booleanIf the next token is a boolean, consume it and return its value.(package private) ByteStringIf the next token is a string, consume it, unescape it as aByteString, and return it.private voidconsumeByteString(List<ByteString> list) LikeconsumeByteString()but adds each token of the string to the given list.doubleIf the next token is a double, consume it and return its value.floatIf the next token is a float, consume it and return its value.(package private) StringIf the next token is an identifier, consume it and return its value.(package private) intIf the next token is a 32-bit signed integer, consume it and return its value.(package private) longIf the next token is a 64-bit signed integer, consume it and return its value.If the next token is a string, consume it and return its (unescaped) value.(package private) intIf the next token is a 32-bit unsigned integer, consume it and return its value.(package private) longIf the next token is a 64-bit unsigned integer, consume it and return its value.private TextFormat.ParseExceptionConstructs an appropriateTextFormat.ParseExceptionfor the givenNumberFormatExceptionwhen trying to parse a float or double.(package private) int(package private) boolean(package private) boolean(package private) intgetLine()(package private) int(package private) intprivate TextFormat.ParseExceptionConstructs an appropriateTextFormat.ParseExceptionfor the givenNumberFormatExceptionwhen trying to parse an integer.(package private) booleanReturnstrueif the current token's text is equal to that specified.(package private) booleanReturnstrueif the next token is an integer, but does not consume it.(package private) voidAdvance to the next token.(package private) TextFormat.ParseExceptionparseException(String description) Returns aTextFormat.ParseExceptionwith the current line and column numbers in the description, suitable for throwing.(package private) TextFormat.ParseExceptionparseExceptionPreviousToken(String description) Returns aTextFormat.ParseExceptionwith the line and column numbers of the previous token in the description, suitable for throwing.private voidSkip over any whitespace so that the matcher region starts at the next token.(package private) booleantryConsume(String token) If the next token exactly matchestoken, consume it and returntrue.(package private) booleanIf the next token is a string, consume it and return true.booleanIf the next token is a double, consume it and returntrue.booleanIf the next token is a float, consume it and returntrue.(package private) booleanIf the next token is an identifier, consume it and returntrue.(package private) booleanIf the next token is a 64-bit signed integer, consume it and returntrue.booleanIf the next token is a 64-bit unsigned integer, consume it and returntrue.
-
Field Details
-
text
-
matcher
-
currentToken
-
pos
private int pos -
line
private int line -
column
private int column -
previousLine
private int previousLine -
previousColumn
private int previousColumn -
WHITESPACE
-
TOKEN
-
DOUBLE_INFINITY
-
FLOAT_INFINITY
-
FLOAT_NAN
-
containsSilentMarkerAfterCurrentToken
private boolean containsSilentMarkerAfterCurrentTokencontainsSilentMarkerAfterCurrentTokenindicates if there is a silent marker after the current token. This value is moved tocontainsSilentMarkerAfterPrevTokenevery time the next token is parsed. -
containsSilentMarkerAfterPrevToken
private boolean containsSilentMarkerAfterPrevToken
-
-
Constructor Details
-
Tokenizer
Construct a tokenizer that parses tokens from the given text.
-
-
Method Details
-
getPreviousLine
int getPreviousLine() -
getPreviousColumn
int getPreviousColumn() -
getLine
int getLine() -
getColumn
int getColumn() -
getContainsSilentMarkerAfterCurrentToken
boolean getContainsSilentMarkerAfterCurrentToken() -
getContainsSilentMarkerAfterPrevToken
boolean getContainsSilentMarkerAfterPrevToken() -
atEnd
boolean atEnd()Are we at the end of the input? -
nextToken
void nextToken()Advance to the next token. -
skipWhitespace
private void skipWhitespace()Skip over any whitespace so that the matcher region starts at the next token. -
tryConsume
If the next token exactly matchestoken, consume it and returntrue. Otherwise, returnfalsewithout doing anything. -
consume
If the next token exactly matchestoken, consume it. Otherwise, throw aTextFormat.ParseException.- Throws:
TextFormat.ParseException
-
lookingAtInteger
boolean lookingAtInteger()Returnstrueif the next token is an integer, but does not consume it. -
lookingAt
Returnstrueif the current token's text is equal to that specified. -
consumeIdentifier
If the next token is an identifier, consume it and return its value. Otherwise, throw aTextFormat.ParseException.- Throws:
TextFormat.ParseException
-
tryConsumeIdentifier
boolean tryConsumeIdentifier()If the next token is an identifier, consume it and returntrue. Otherwise, returnfalsewithout doing anything. -
consumeInt32
If the next token is a 32-bit signed integer, consume it and return its value. Otherwise, throw aTextFormat.ParseException.- Throws:
TextFormat.ParseException
-
consumeUInt32
If the next token is a 32-bit unsigned integer, consume it and return its value. Otherwise, throw aTextFormat.ParseException.- Throws:
TextFormat.ParseException
-
consumeInt64
If the next token is a 64-bit signed integer, consume it and return its value. Otherwise, throw aTextFormat.ParseException.- Throws:
TextFormat.ParseException
-
tryConsumeInt64
boolean tryConsumeInt64()If the next token is a 64-bit signed integer, consume it and returntrue. Otherwise, returnfalsewithout doing anything. -
consumeUInt64
If the next token is a 64-bit unsigned integer, consume it and return its value. Otherwise, throw aTextFormat.ParseException.- Throws:
TextFormat.ParseException
-
tryConsumeUInt64
public boolean tryConsumeUInt64()If the next token is a 64-bit unsigned integer, consume it and returntrue. Otherwise, returnfalsewithout doing anything. -
consumeDouble
If the next token is a double, consume it and return its value. Otherwise, throw aTextFormat.ParseException.- Throws:
TextFormat.ParseException
-
tryConsumeDouble
public boolean tryConsumeDouble()If the next token is a double, consume it and returntrue. Otherwise, returnfalsewithout doing anything. -
consumeFloat
If the next token is a float, consume it and return its value. Otherwise, throw aTextFormat.ParseException.- Throws:
TextFormat.ParseException
-
tryConsumeFloat
public boolean tryConsumeFloat()If the next token is a float, consume it and returntrue. Otherwise, returnfalsewithout doing anything. -
consumeBoolean
If the next token is a boolean, consume it and return its value. Otherwise, throw aTextFormat.ParseException.- Throws:
TextFormat.ParseException
-
consumeString
If the next token is a string, consume it and return its (unescaped) value. Otherwise, throw aTextFormat.ParseException.- Throws:
TextFormat.ParseException
-
consumeByteString
If the next token is a string, consume it, unescape it as aByteString, and return it. Otherwise, throw aTextFormat.ParseException.- Throws:
TextFormat.ParseException
-
tryConsumeByteString
boolean tryConsumeByteString()If the next token is a string, consume it and return true. Otherwise, return false. -
consumeByteString
LikeconsumeByteString()but adds each token of the string to the given list. String literals (whether bytes or text) may come in multiple adjacent tokens which are automatically concatenated, like in C or Python.- Throws:
TextFormat.ParseException
-
parseException
Returns aTextFormat.ParseExceptionwith the current line and column numbers in the description, suitable for throwing. -
parseExceptionPreviousToken
Returns aTextFormat.ParseExceptionwith the line and column numbers of the previous token in the description, suitable for throwing. -
integerParseException
Constructs an appropriateTextFormat.ParseExceptionfor the givenNumberFormatExceptionwhen trying to parse an integer. -
floatParseException
Constructs an appropriateTextFormat.ParseExceptionfor the givenNumberFormatExceptionwhen trying to parse a float or double.
-