.process_wp_vocab {wordpiece}R Documentation

Process a Wordpiece Vocabulary for Tokenization

Description

Process a Wordpiece Vocabulary for Tokenization

Usage

.process_wp_vocab(v)

## Default S3 method:
.process_wp_vocab(v)

## S3 method for class 'wordpiece_vocabulary'
.process_wp_vocab(v)

## S3 method for class 'integer'
.process_wp_vocab(v)

## S3 method for class 'character'
.process_wp_vocab(v)

Arguments

v

An object of class wordpiece_vocabulary.

Value

A character vector of tokens for tokenization.


[Package wordpiece version 2.1.3 Index]