rt_get_fields {robotstxt} | R Documentation |
extracting permissions from robots.txt
Description
extracting permissions from robots.txt
Usage
rt_get_fields(txt, regex = "", invert = FALSE)
Arguments
txt |
content of the robots.txt file |
regex |
regular expression specify field |
invert |
invert selection made via regex? |
[Package robotstxt version 0.7.15 Index]