Blog
SEO bots that block ~ 140 million websites the most
May
Have you ever wondered which SEO bots are most blocked? This can affect the quality of the data provided by the tools.
Blocking these bots will mainly affect the tool link index. They will not be able to crawl the pages, so they cannot check where these pages are connected. It does not matter for traffic estimates, keyword rankings, the best pages, etc. They are made of various data sources.
In the case of Ahrefs, this would also affect the function of the page history that shows changes on your pages in time, which you may need at some point. Ahrefsbot also feeds our search engine index, Yep.comSo blocking Ahrefsbot means that you would not show yourself in YEP search results.
We looked at ~ 140 million websites to see how often SEO bots were blocked. I want to thank our scientist very much XIBEIJIA Guan to download this data.
Here are the 3 best most blocked SEO bots:
- MJ12Bot (Majestic). Blocked by 6.49% of all websites.
- Semrushbot. Blocked by 6.34% of all websites.
- Ahrefsbot. Blocked by 6.31% of all websites.
We looked at the total number of websites blocking bots. There are many ways to block bots with robots.txt, and this includes all, including:
- Clear blockswhere the bot is replaced and not allowed
- General blockswhere all bots can be blocked
- All cases in which the directive allowed the botAfter blocking all bots
Reservations: This does not include any other types of blocks, such as firewalls or IP blocks.
As I mentioned earlier, the most blocked bot is MJ12Bot from Majestic. I suspect there are several reasons.
- They are distributed crazy, which means that they cannot be looked or blocked according to IPS, which makes them less trusted.
- They crawl longer.
- They have a smaller user base than more popular SEO tools, and therefore a smaller lever to remove all blocks.
Here are the most blocked SEO bots:
And total websites blocking SEO bots:
Here are the data:
| Bot name | To count | Percentage | Bot operator |
|---|---|---|---|
| MJ12BOT | 9081205 | 6.49 | Majestic |
| Semrushbot | 8868486 | 6.34 | Semrush |
| Ahrefsbot | 8831316 | 6.31 | Ahrefs |
| dotbot | 8569766 | 6.13 | May |
| Blasbot | 8374216 | 5.99 | Seo poweruite |
| Serpstatbot | 7878935 | 5.63 | Sertstat |
| DataFordersobot | 7872939 | 5.63 | DataFordeo |
| Semrushbot-c | 7855400 | 5.62 | Semrush |
| Barrrowler | 7804425 | 5.58 | Huge |
| Semrushbot-ba | 7796785 | 5.57 | Semrush |
| Semrushbot-Swa | 7789812 | 5.57 | Semrush |
| Semrushbotter | 7789062 | 5.57 | Semrush |
| Seokicks | 7758904 | 5.55 | Seokicks |
| Screaming frog seo spider | 7711108 | 5.51 | Shouting frog |
| LinkDexBOT | 7704425 | 5.51 | Linkdex |
| DomainStatsbot | 7696944 | 5.5 | DomeainStats |
| Zoombot | 7669495 | 5.48 | Seozoom |
| Sitecheckerbotcrawler | 7666545 | 5.48 | Sitechecker |
| Cocolyzebot | 7666233 | 5.48 | Cocolize |
| SeobilityBot | 7664228 | 5.48 | Seobility |
| Senutobot | 7655145 | 5.47 | Senuto |
| Hypestate | 7648671 | 5.47 | Hypestate |
| Online-Webceo-Bot | 7648444 | 5.47 | Webceo |
| Brightdge crawler | 7648139 | 5.47 | Brightdge |
| Seolizer | 7648112 | 5.47 | Seolizer |
The analysis is a little more complicated. In the case of the above, we looked at the main file robots.txt for the website, but each subdomen can have its own set of instructions. If we look at ~ 461, a meter robots. Here is the best 5:
- Semrushbot: 5.76%
- Dotbot (MO): 5.34%
- MJ12Bot (Majestic): 4.96%
- Blasbot: 4.88%
- Ahrefsbot: 4.67%
In this regard, we only look at cases in which a given bot is not allowed. It does not include any general statements or cases in which only some bots may be allowed. In such cases, site owners tried to lock some bots specially.
Majestic Bot is the most targeted, and then Bot Moz.
Here are the most blocked SEO bots according to clear references:
Here is the number of websites clearly blocking SEO bots:
Here are the data:
| Bot name | To count | Percentage | Bot operator |
|---|---|---|---|
| MJ12BOT | 2000372 | 1.43 | Majestic |
| dotbot | 1402305 | 1 | May |
| Ahrefsbot | 1350771 | 0.97 | Ahrefs |
| Semrushbot | 1285857 | 0.92 | Semrush |
| Blasbot | 861184 | 0.62 | Seo poweruite |
| Serpstatbot | 354683 | 0.25 | Sertstat |
| DataFordersobot | 284694 | 0.2 | DataFordeo |
| Barrrowler | 276332 | 0.2 | Huge |
| Seokicks | 219961 | 0.16 | Seokicks |
| Semrushbot-c | 211895 | 0.15 | Semrush |
| LinkDexBOT | 166405 | 0.12 | Linkdex |
| DomainStatsbot | 157053 | 0.11 | DomeainStats |
| Semrushbot-ba | 154349 | 0.11 | Semrush |
| Semrushbotter | 147999 | 0.11 | Semrush |
| Semrushbot-Swa | 146261 | 0.1 | Semrush |
| Zoombot | 125310 | 0.09 | Seozoom |
| Sitecheckerbotcrawler | 122574 | 0.09 | Sitechecker |
| Cocolyzebot | 121737 | 0.09 | Cocolize |
| SeobilityBot | 117558 | 0.08 | Seobility |
| Screaming frog seo spider | 87673 | 0.06 | Shouting frog |
| Senutobot | 54978 | 0.04 | Senuto |
| Hypestate | 861 | 0 | Hypestate |
| Senutobot | 54978 | 0.04 | Senuto |
| Hypestate | 861 | 0 | Hypestate |
| Online-Webceo-Bot | 659 | 0 | Webceo |
| Brightdge crawler | 289 | 0 | Brightdge |
| Seolizer | 253 | 0 | Seolizer |
We looked at the best places 1m by DR, which adapt to places with DR> 45. Semrush is the most blocked, and then Majestic and MO.
Here’s how it breaks down for each bot in various categories of websites. Top 3 is:
- Autos_and_vehicles: 39%
- Books_and_literature: 27%
- Property: 17%
Passing through the Bot’s requests in Cloudflare radarAhrefs is by far the fastest worm in the SEO space. ~ 4.6x faster than I ~ 6.7x faster than Semrush.
