module ToknInternal
Namespace to encompass the portions of the Tokn
gem used only internally
Constants
- CODEMAX
One plus the maximum code represented
- CODEMIN
Minimum code possible (e.g., indicating a token id)
- EPSILON
Code for epsilon transitions
- UNKNOWN_TOKEN
Token id if text didn't match any tokens in the DFA
Public Class Methods
edgeLabelToTokenId(edgeLabel)
click to toggle source
Convert an edge label value ( < 0) to a token id (>=0)
# File lib/tokn/tokn_const.rb, line 31 def self.edgeLabelToTokenId(edgeLabel) EPSILON-1-edgeLabel end
tokenIdToEdgeLabel(tokenId)
click to toggle source
Convert a token id (>=0) to an edge label value ( < 0)
# File lib/tokn/tokn_const.rb, line 25 def self.tokenIdToEdgeLabel(tokenId) EPSILON-1-tokenId end