Blog
AI bots are locking in search engines
Jun
The network is torn more than ever, but it is not just a search engine when crawling. AI bots absorb all the data they can train and power their AI assistants.
The problem is that AI bots are recent to crawl on the web. They still make stupid creeping decisions, fall into the area and waste many resources. We have already seen that AI bots are starting to be blocked by much more sites with the most commercial.
I have no doubt that AI bots will come up with everything, but it will probably be a painful process. I don’t think any of them even started rendering the content.
In the past, there was a lot of explosion to Bing because of their aggressive creep. They fixed it, they started IndexialAnd now they can be the most effective internet pike.
Check Indexnow
If you haven’t configured Indexnow for your site, you should.
If you are CDN, CMS, AI BOT or Google, you should also take care of it. Save resources, save money and save the planet. It is good for you, it is good for your users, it is good for website owners, it is good for websites and is good for network and planet.
It is estimated that 53% of the feces movement is wastedAnd the indexing can assist reduce this number to almost nothing.
Looking at the data from Cloudflare radarWhich reflects a huge piece of internet traffic, we can see how AI bots compare themselves to Seo search engines and bots. Cloudflare handles ~ 20% of all internet trafficSo it is a enormous and representative set of data for general internet creep.
Let’s immerse ourselves inside.
The bots in the search engine crawl the most. AI bots are relatively recent on the Internet, but they are very second, even though there were fewer of them. If more AI assistants are launched and their bots will start crawling on the internet, they will probably go through the search engine in the crawling. They grow at a rate, it can happen in the next few years.
If we break some individual bots, you can see that Googlebot eat the Internet much more than other bots.
Note that Bot Ai Google also has quick creep indicators, as is Google images. There are many more Googlebots that perform various tasks.
Amazonbot and GPTBOT are on an equal footing with Bing.
Ahrefsbot is by far the most busy bot in the SEO industry. Some other platforms like to say that they are crawling faster, but all evidence says that this is not true. Ahrefs makes almost as many demands as all 20 other SEO bots in total. We beat the combination of 19 of them.
Despite our indexing coefficient, I guarantee that we are also the most productive side of every SEO bot. We have solved many problems that I still see present in the other working ones. We also receive data from Indexnow via our search engine, YesTo assist us be even more productive. At this point, maybe instead we should be classified as a search engine bot.
AI bots have increased this year. They were almost 25% of all bots in May, only 10% below search bots. It is amazing for me, considering that Googlebot is as dominant and there are about half as many AI bots as there are bots in the search engine.
Googlebot is by far the most dominant bot on the Internet. GPTBOT downloads this year. I am curious if they will close the gap in the future.
Final thoughts
Bots suspending sites have costs, and between search engines and website owners there is a social agreement in which search engines escalate value by sending a recommendation traffic to websites. Google sends traffic (for now), so they are not blocked.
This agreement covers AI assistants. Send traffic on websites, provide data owners’ details about searching and visibility of their website in your reporting system, otherwise you will be blocked at the pace you are crawling. It is already happening.
No wonder that the AI search movement is only 0.1% of the general traffic on websites. Send more traffic to websites, provide website owners with greater value.
