Website classification crawler

From URL Classification
Revision as of 23:14, 18 December 2024 by Barak (talk | contribs)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

As part of the URL Classification service we scan the web on a daily bases, we use a web crawler to classify web sites, it checks each site about once every three days, and will have: KomodiaBot in the User-Agent field, it can also make requests with other User-Agent.

The user agent are: 

* Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KStandBot/1.0
* Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0

And will come from these IPs:

  • 54.39.104.60
  • 144.217.71.170
  • 149.56.20.160
  • 207.182.136.170
  • 147.135.44.125
  • 78.46.82.170
  • 78.110.173.202
  • 223.252.24.17
  • 159.89.102.112
  • 165.227.197.119
  • 146.190.222.181
  • 83.229.67.114
  • 83.229.70.228
  • 15.204.140.134

Any issues, questions, complaints should be sent to: sales [at] url-classification.io

Retrieved from "https://url-classification.io/wiki/index.php?title=Website_classification_crawler&oldid=191"