Website classification crawler: Difference between revisions

From URL Classification
Jump to navigation Jump to search
(Created page with "As part of the URL Classification service we scan the web on a daily bases, we use a web crawler to classify web sites, it checks each site about once every three days, and wi...")
 
No edit summary
 
(2 intermediate revisions by the same user not shown)
Line 4: Line 4:


<pre>
<pre>
* Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KomodiaBot/1.0
* Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KStandBot/1.0
* Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://www.komodia.com/newwiki/index.php/URL_server_crawler) KomodiaBot/1.0
* Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0
</pre>
</pre>


And will come from these IPs:
And will come from these IPs:


* 207.182.140.210
* 142.44.138.126
* 207.182.140.211
* 207.182.140.212
* 207.182.140.213
* 207.182.140.214
* 207.182.136.170
* 207.182.136.170
* 78.46.82.170
* 78.46.82.170

Latest revision as of 08:46, 4 September 2020

As part of the URL Classification service we scan the web on a daily bases, we use a web crawler to classify web sites, it checks each site about once every three days, and will have: KomodiaBot in the User-Agent field.

The user agent is:

* Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KStandBot/1.0
* Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0

And will come from these IPs:

  • 142.44.138.126
  • 207.182.136.170
  • 78.46.82.170
  • 78.110.173.202
  • 223.252.24.17

Any issues, questions, complaints should be sent to: sales [at] url-classification.io