Website classification crawler: Difference between revisions
Jump to navigation
Jump to search
No edit summary |
No edit summary |
||
| Line 1: | Line 1: | ||
As part of the URL Classification service we scan the web on a daily bases, we use a web crawler to classify web sites, it checks each site about once every three days, and will have: ''KomodiaBot'' in the User-Agent field. | As part of the URL Classification service we scan the web on a daily bases, we use a web crawler to classify web sites, it checks each site about once every three days, and will have: ''KomodiaBot'' in the User-Agent field, it can also make requests with other User-Agent. | ||
The user agent | The user agent are: | ||
<pre> | <pre> | ||
| Line 10: | Line 10: | ||
And will come from these IPs: | And will come from these IPs: | ||
* | * 54.39.104.60 | ||
* 144.217.71.170 | |||
* 149.56.20.160 | |||
* 207.182.136.170 | * 207.182.136.170 | ||
* 147.135.44.125 | |||
* 78.46.82.170 | * 78.46.82.170 | ||
* 78.110.173.202 | * 78.110.173.202 | ||
* 223.252.24.17 | * 223.252.24.17 | ||
* 159.89.102.112 | |||
* 165.227.197.119 | |||
* 146.190.222.181 | |||
* 83.229.67.114 | |||
* 83.229.70.228 | |||
* 15.204.140.134 | |||
Any issues, questions, complaints should be sent to: sales [at] url-classification.io | Any issues, questions, complaints should be sent to: sales [at] url-classification.io | ||
Latest revision as of 23:14, 18 December 2024
As part of the URL Classification service we scan the web on a daily bases, we use a web crawler to classify web sites, it checks each site about once every three days, and will have: KomodiaBot in the User-Agent field, it can also make requests with other User-Agent.
The user agent are:
* Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KStandBot/1.0 * Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0
And will come from these IPs:
- 54.39.104.60
- 144.217.71.170
- 149.56.20.160
- 207.182.136.170
- 147.135.44.125
- 78.46.82.170
- 78.110.173.202
- 223.252.24.17
- 159.89.102.112
- 165.227.197.119
- 146.190.222.181
- 83.229.67.114
- 83.229.70.228
- 15.204.140.134
Any issues, questions, complaints should be sent to: sales [at] url-classification.io