module Spider

Author

Robert Dormer (rdormer@gmail.com)

Copyright

Copyright © 2016 Robert Dormer

License

MIT

¶ ↑

Class to encapsulate the crawl delay being used.
Clamps the value to a maximum amount and implements
an exponential backoff function for responding to
rate limit requests

¶ ↑