rt_get_fields_worker {robotstxt} | R Documentation |
extracting robotstxt fields
Description
extracting robotstxt fields
Usage
rt_get_fields_worker(txt, type = "all", regex = NULL, invert = FALSE)
Arguments
txt |
content of the robots.txt file |
type |
name or names of the fields to be returned, defaults to all fields |
regex |
subsetting field names via regular expressions |
invert |
field selection |
[Package robotstxt version 0.7.15 Index]