Website classification crawler: Difference between revisions
Jump to navigation
Jump to search
(Created page with "As part of the URL Classification service we scan the web on a daily bases, we use a web crawler to classify web sites, it checks each site about once every three days, and wi...") |
No edit summary |
||
| (2 intermediate revisions by the same user not shown) | |||
| Line 4: | Line 4: | ||
<pre> | <pre> | ||
* Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) | * Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KStandBot/1.0 | ||
* Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http:// | * Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0 | ||
</pre> | </pre> | ||
And will come from these IPs: | And will come from these IPs: | ||
* | * 142.44.138.126 | ||
* 207.182.136.170 | * 207.182.136.170 | ||
* 78.46.82.170 | * 78.46.82.170 | ||
Latest revision as of 08:46, 4 September 2020
As part of the URL Classification service we scan the web on a daily bases, we use a web crawler to classify web sites, it checks each site about once every three days, and will have: KomodiaBot in the User-Agent field.
The user agent is:
* Mozilla/5.0 (compatible; MSIE 8.0; Windows NT 5.1) KStandBot/1.0 * Mozilla/5.0 (Windows NT 6.1; Win64; x64; +http://url-classification.io/wiki/index.php?title=URL_server_crawler) KStandBot/1.0
And will come from these IPs:
- 142.44.138.126
- 207.182.136.170
- 78.46.82.170
- 78.110.173.202
- 223.252.24.17
Any issues, questions, complaints should be sent to: sales [at] url-classification.io