create_longformer_model {aifeducation} | R Documentation |
This function creates a transformer configuration based on the Longformer base architecture and a vocabulary based on Byte-Pair Encoding (BPE) tokenizer by using the python libraries 'transformers' and 'tokenizers'.
create_longformer_model(
ml_framework = aifeducation_config$get_framework,
model_dir,
vocab_raw_texts = NULL,
vocab_size = 30522,
add_prefix_space = FALSE,
trim_offsets = TRUE,
max_position_embeddings = 512,
hidden_size = 768,
num_hidden_layer = 12,
num_attention_heads = 12,
intermediate_size = 3072,
hidden_act = "gelu",
hidden_dropout_prob = 0.1,
attention_probs_dropout_prob = 0.1,
attention_window = 512,
sustain_track = TRUE,
sustain_iso_code = NULL,
sustain_region = NULL,
sustain_interval = 15,
trace = TRUE,
pytorch_safetensors = TRUE
)
ml_framework |
|
model_dir |
|
vocab_raw_texts |
|
vocab_size |
|
add_prefix_space |
|
trim_offsets |
|
max_position_embeddings |
|
| |
| |
num_attention_heads |
|
intermediate_size |
|
| |
| |
attention_probs_dropout_prob |
|
attention_window |
|
sustain_track |
|
sustain_iso_code |
|
sustain_region |
Region within a country. Only available for USA and Canada See the documentation of codecarbon for more information. https://mlco2.github.io/codecarbon/parameters.html |
sustain_interval |
|
trace |
|
pytorch_safetensors |
|
This function does not return an object. Instead the configuration and the vocabulary of the new model are saved on disk.
To train the model, pass the directory of the model to the function train_tune_longformer_model.
Beltagy, I., Peters, M. E., & Cohan, A. (2020). Longformer: The Long-Document Transformer. doi:10.48550/arXiv.2004.05150
Hugging Face Documentation https://huggingface.co/docs/transformers/model_doc/longformer#transformers.LongformerConfig
Other Transformer:
create_bert_model()
,
create_deberta_v2_model()
,
create_funnel_model()
,
create_roberta_model()
,
train_tune_bert_model()
,
train_tune_deberta_v2_model()
,
train_tune_funnel_model()
,
train_tune_longformer_model()
,
train_tune_roberta_model()