Page Rank Obsession Killed the Interweb

The Interweb is dying, these days it's all about the money. Forget websites being about the subject of the website, it's mostly about the clicks. Among the worst people working the web in that direction are those who run SEO/marketing analytic bots. These bots scrape the web and attempt to make money using other people's content.

Their "bot info" page will often try to scare webmasters, especially those obsessed with being "first on Google". These are not search engines that help the general public find web pages. These are paid for tools sold to those who want reports on how their websites and online reputations are doing.

Semrush charges their clients between 70 and 550 USD per month to use their services. Their product is entirely based on Google results and your website's cooperation.

Well, the real question is why do you not want the bot visiting your page?

Well, the real question is, why would you want this bot visiting your website ? It's a waste of resources.

They generally claim to respect robots.txt directives. I allow myself to doubt. For example, Semrush accessed the robots.txt file twice in over 3 months while visiting over 4000 times during this period.

Same goes for Ahrefs, they claim to visit robots.txt at least once a week, my logs beg to differ. But even if they did, I woudln't care. Like any SEO business they just use your content to sell a service to those way too concerned by page rank.

Some of these SEO/marketing companies do respect the robots.txt, like Trendiction, which is good. However, I don't want to participate in their "Social media monitoring and social analytics", unless the data is equally accessible to everyone I don't see the point in participating.

There are many others too. If their crawling serves the general public, I will usually let them do their thing, I might even take time to send them feedback if there are issues. However, bots that only serve the purpose of monetisation can get bent.

These bots are getting out of control, on some websites there are more visits from scrapers than actual people. This growing obsession with page rank is making good and useful content harder and harder to find. Too many websites are trying to be first on as many possible search queries and these tools are helping them.