Difference between revisions of "Website classification crawler"
Revision as of 08:45, 4 September 2020
As part of the URL Classification service we scan the web on a daily bases, we use a web crawler to classify web sites, it checks each site about once every three days, and will have: KomodiaBot in the User-Agent field.
The user agent is:
* Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KStandBot/1.0 * Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0
And will come from these IPs:
Any issues, questions, complaints should be sent to: sales [at] url-classification.io