module ToknInternal

Namespace to encompass the portions of the Tokn gem used only internally

Constants

CODEMAX

One plus the maximum code represented

CODEMIN

Minimum code possible (e.g., indicating a token id)

EPSILON

Code for epsilon transitions

UNKNOWN_TOKEN

Token id if text didn't match any tokens in the DFA

Public Class Methods

edgeLabelToTokenId(edgeLabel) click to toggle source

Convert an edge label value ( < 0) to a token id (>=0)

# File lib/tokn/tokn_const.rb, line 31
def self.edgeLabelToTokenId(edgeLabel)
  EPSILON-1-edgeLabel
end
tokenIdToEdgeLabel(tokenId) click to toggle source

Convert a token id (>=0) to an edge label value ( < 0)

# File lib/tokn/tokn_const.rb, line 25
def self.tokenIdToEdgeLabel(tokenId)
  EPSILON-1-tokenId  
end