Website classification crawler

From URL Classification
(Redirected from URL server crawler)
Jump to navigation Jump to search
The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

As part of the URL Classification service we scan the web on a daily bases, we use a web crawler to classify web sites, it checks each site about once every three days, and will have: KomodiaBot in the User-Agent field.

The user agent is: 

* Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KStandBot/1.0
* Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0

And will come from these IPs:

  • 142.44.138.126
  • 207.182.136.170
  • 78.46.82.170
  • 78.110.173.202
  • 223.252.24.17

Any issues, questions, complaints should be sent to: sales [at] url-classification.io

Retrieved from "https://url-classification.io/wiki/index.php?title=Website_classification_crawler&oldid=30"