TokenizeWithRules {fastai} | R Documentation |
A wrapper around 'tok' which applies 'rules', then tokenizes, then applies 'post_rules'
TokenizeWithRules(tok, rules = NULL, post_rules = NULL)
tok |
tokenizer |
rules |
rules |
post_rules |
post_rules |
None