rt_get_fields_worker {robotstxt} | R Documentation |
extracting robotstxt fields
rt_get_fields_worker(txt, type = "all", regex = NULL, invert = FALSE)
txt |
content of the robots.txt file |
type |
name or names of the fields to be returned, defaults to all fields |
regex |
subsetting field names via regular expressions |
invert |
field selection |