is_valid_robotstxt {robotstxt} | R Documentation |
function that checks if file is valid / parsable robots.txt file
is_valid_robotstxt(text, check_strickt_ascii = FALSE)
text |
content of a robots.txt file provides as character vector |
check_strickt_ascii |
whether or not to check if content does adhere to the specification of RFC to use plain text aka ASCII |