Offline URL Database 1st January 2023 changelog

From URL Classification
Jump to navigation Jump to search

Category change

  • Added Parked domain category
  • Added Domain not available category
  • Added Private site available category
  • Added Earth science category
  • Added Fringe theory category
  • Added Conspiracy theory category
  • Added Alt right category
  • Added Alt left category

Database change

  • Removed 1.6m invalid domains (they were classified under DNS not found category)
  • Improved detection of Captcha protected sites
  • Improved the ability to classify captcha protected sites
  • Improved detection for parked domains

Domain count and statistics

ID Name Domain count
1 Abortion and pregnancy 103,122
2 Ads 81,349
3 Adult 433,537
4 Adult dating 18,822
5 Alcohol 407,089
6 Astrology 63,652
7 Blogs 3,467,789
8 Books or comics 491,035
9 Business and services 6,355,917
10 CDN 1,048,973
11 Chats 98,172
12 Classifieds 111,450
13 Clothing and fashion 1,111,631
14 Cooking and food 1,799,018
15 Dating 89,236
16 Drugs 78,046
17 Education 1,247,376
18 Empty site 1,747,224
19 Entertainment 749,283
20 Error site can't be accessed 5,844,831
21 DNS error site can't be resolved 41,923,049
22 File hosting 78,577
23 Finance (Banks_Real estate_Insurance) 645,657
24 Gambling 394,582
25 Computer games 544,632
26 General (safe) 9,925
27 Government 133,134
28 Hacking 12,210
29 Hate 17,532
30 Hobbies 373,352
31 Home improvement 465,905
32 Humor 29,447
33 Hunting and fishing 109,552
34 Image and video hosting 112,941
35 Jobs 834,226
36 Kids safe 237
37 Legal 446,981
38 Life style and beauty 761,550
39 Link farm (no real content) 7,174,545
40 Local information 73,426
41 Medical 2,650,605
42 Military 31,264
43 Mixed adult 1,043,067
44 Mobile 282,692
45 Models 57,687
46 Music 1,118,131
47 News 1,498,895
49 Not child safe 22,942
50 Online drugs 41,389
51 Online marketing 581,405
52 Online video or audio 327,634
53 Pets and animals 424,470
54 Phishing 1,474
55 Politics 348,562
56 Porn 852,102
57 Internet portals 295,704
58 Profanity 32,824
59 Proxy 16,021
60 Religion 536,742
62 Science 344,082
63 Search engines 81,401
64 Sects 13,029
65 Semi nude and pin up 27,048
66 Sex education 67,498
67 Shopping 2,491,562
68 Social networks 631,912
69 Spam or scam 3,652
70 Sports 1,361,544
71 Suicide 293
72 Technology and computers 937,929
73 Site under construction or not available 5,231,976
74 General - not porn 9,445,557
75 URL shortener 7,157
76 Vacation and travel 1,800,131
77 Vehicles 1,277,162
78 Violence 32,506
79 Virus or Spyware 1,755
80 Warez 38,300
81 Weapons 150,389
82 Web hosting 867,988
83 Web mail 114,425
84 Reference 201,620
85 Community and society 235,593
86 Software repository 34,049
87 Philosophy and psychology 246,067
88 History 301,236
89 Nonsense and time waste 73,668
90 Local IP 70
91 Swimsuits and underwear 110,587
92 Arts 749,980
94 Login page 144,551
95 Surveys and polls 35,298
96 Tobacco 56,205
97 ISP 19,293
98 Tracker 1,144,480
99 Bypass protection 3,067
100 Search block 52
101 Image search block 171,739
102 Desktop themes 9,389
103 Depress domain 71,347
104 Depress page 1,839
105 Parenting and family 71,638
106 School cheats 4,588
107 Celebrity gossip 122,298
108 How to 100,124
109 Illegal 3,134
110 Recreation 211,185
111 Bookmarks 17,684
112 Money spam 5,633
113 Directories 4,843
114 Site_Graphic design 368,814
115 Weather 80,739
116 Knowledge 62,116
117 Kids entertainment 16,005
118 Nature 107,232
119 Diet 119,703
120 Eating disorders 5,342
121 Self help 94,318
122 Relationships 21,186
123 Paranormal 37,595
124 Images CDN 2,952
125 Images CDN Unknown 65,694
126 LGBT 16,219
127 Marketing or ads 33,601
128 Magazines 153,333
129 Forums 128,074
130 User generated content 41,146
131 Tele Communications 60,070
132 Gender specific 2,334
134 Crime 37,866
135 Person 59,620
136 Grief 2,431
137 Retirement 50,478
138 Other drugs 69,238
139 Pharmacy 42,976
140 Songs download 18,585
141 Lookup 45,445
142 Movies download 112,313
143 Consumer reviews 19,530
144 API 87,711
145 Legal movies 641
146 Legal songs 1,639
147 Bitcoin 75,757
148 Victim help 3,443
149 Rewards 9,676
150 Photography 380,737
151 Toys 124,686
152 Restaurant 375,699
153 Fitness 276,768
154 Contact sport 126,870
155 Dancing 50,443
156 Software 428,950
157 Programming 258,516
158 Registry cleaner 324
159 Online security 125,113
160 Real estate 763,346
161 Loans 130,508
162 Forex 26,322
163 Banks 250,261
164 Insurance 279,614
165 SEO 84,320
166 PPC 7,298
167 Supplements 47,769
168 URL Classification 455
170 Tatoo 30,615
172 Tactical equipment 5,977
173 Alternative medicine 230,771
174 AI 16,115
175 Comics 20,342
176 Manga 7,348
177 Captcha 487,782
178 Racism education 1,310
179 Payment 14,989
180 Brokerage 18,924
181 SAAS 13,334
182 Anime 14,552
183 Cartoon 5,936
184 Grownups 19
185 Stream converter 3,195
186 File converter 2,811
187 TV Personality 839
188 Radio Personality 634
189 Youtube Personality 7,985
190 Prevent crime or violence 3,519
191 Collaboration software 8,045
193 Internet support 2,956
194 Computer support 24,520
195 Home appliance 4,380
196 Disposable mail 414
197 Parked domain 315,396
198 Domain not available 83,876
199 Private site 135,361
200 Earth science 27,311
201 Fringe theory 45
202 Conspiracy theory 185
203 Alt right 106
204 Alt left 97

Total domains: 126,012,685

Total classified domains: 78,244,805 (Total domains excluding DNS resolve errors and other errors)

Total unique domains: 115,219,367

Total unique classified domains: 67,451,725 (Total unique domains excluding DNS resolve errors and other errors)

TLD Distribution (all domains)

# TLD Domain count
1 com 58,746,836
2 net 6,434,278
3 de 4,347,180
4 org 4,263,895
5 uk 3,503,529
6 ru 2,375,815
7 cn 1,899,847
8 info 1,729,539
9 nl 1,388,483
10 fr 1,196,513
11 pl 1,128,700
12 it 1,118,404
13 jp 1,104,209
14 br 1,037,378
15 tk 917,956
16 ca 716,416
17 tw 705,894
18 eu 641,022
19 top 605,562
20 co 604,706
21 in 594,994
22 ch 547,632
23 biz 547,415
24 us 530,790
25 es 488,538
26 cz 466,159
27 se 429,430
28 be 408,253
29 at 389,258
30 ga 361,392
31 cf 344,705
32 ir 340,061
33 dk 322,437
34 za 318,612
35 hu 302,208
36 club 291,850
37 ml 282,107
38 io 280,960
39 me 271,191
40 gq 270,862
41 nz 270,813
42 ua 269,832
43 ro 259,773
44 cc 255,628
45 tr 251,823
46 home 251,553
47 mx 221,564
48 ar 216,032
49 online 213,020
50 il 209,699
51 gr 205,441
52 icu 200,112
53 kr 197,749
54 xyz 195,489
55 no 184,838
56 live 180,119
57 loan 174,224
58 site 172,896
59 fi 169,519
60 pw 167,069
61 vn 161,463
62 tv 150,487
63 cl 147,461
64 sk 145,794
65 edu 140,498
66 id 129,276
67 pt 111,654
68 pro 109,381
69 ws 105,826
70 ie 102,981
71 su 98,101
72 vip 79,255
73 nu 79,003
74 shop 78,439
75 lt 76,968
76 space 73,487
77 kz 71,895
78 my 71,557
79 gdn 68,475
80 by 66,774
81 sg 63,217
82 win 62,439
83 asia 61,691
84 hk 61,515
85 bid 60,156
86 website 60,107
87 si 58,499
88 to 56,567
89 hr 54,871
90 ee 53,026
91 mobi 52,551
92 rs 52,385
93 tech 52,315
94 app 49,790
95 gov 47,819
96 lv 46,565
97 domain 46,245
98 work 45,362
99 fun 45,205
100 All other 5,597,896

TLD Distribution (excluding DNS resolve errors and other errors)

# TLD Domain count
1 com 37,300,408
2 net 3,766,009
3 de 2,987,387
4 org 2,530,378
5 uk 2,528,466
6 ru 1,320,466
7 nl 1,021,809
8 fr 788,261
9 it 781,646
10 jp 777,437
11 pl 719,634
12 br 624,499
13 ca 527,891
14 cn 507,153
15 info 440,798
16 ch 417,061
17 cz 370,561
18 eu 364,051
19 co 327,177
20 es 323,874
21 in 320,848
22 se 306,929
23 be 293,779
24 at 291,725
25 dk 232,536
26 hu 228,054
27 za 226,850
28 us 215,532
29 io 200,616
30 ga 190,811
31 cf 187,682
32 biz 186,318
33 nz 175,385
34 ua 173,415
35 ml 172,104
36 tr 166,641
37 ro 165,723
38 ir 161,217
39 gq 150,990
40 gr 147,176
41 no 143,592
42 il 141,349
43 me 136,554
44 mx 135,424
45 fi 135,069
46 kr 126,860
47 ar 124,527
48 sk 113,717
49 edu 109,395
50 tw 104,169
51 vn 103,675
52 cc 95,787
53 cl 91,639
54 pt 81,711
55 ie 77,138
56 tv 76,522
57 id 73,907
58 ws 63,332
59 online 62,485
60 lt 59,030
61 top 57,701
62 club 49,982
63 my 47,511
64 xyz 45,974
65 site 45,551
66 by 44,767
67 pro 43,313
68 ee 42,050
69 si 41,946
70 sg 40,926
71 hr 39,182
72 rs 38,169
73 hk 38,082
74 nu 36,689
75 tk 36,588
76 lv 33,330
77 ph 32,519
78 app 30,983
79 kz 30,503
80 shop 30,287
81 su 27,578
82 th 27,157
83 bg 26,841
84 gov 25,798
85 cat 24,337
86 live 23,455
87 ae 22,213
88 blog 21,982
89 pe 21,644
90 pk 21,533
91 mobi 18,619
92 is 17,580
93 store 16,876
94 space 16,839
95 lu 16,491
96 to 16,341
97 asia 15,845
98 link 15,134
99 tech 14,761
100 All other 1,579,464