At the beginning of 2014, we wrote about the negative impact that bots have on the web, noting that the number of bots — software web users like crawlers and spam bots — outstripped the number of human web visitors by a considerable margin. In its annual bot report for this year, Incapsula has revealed what at first seems like a cheery reversal of the domination of the bots — there are more human that bot users, but the report isn’t quite the good news that it seems.
Bots come in many guises. Some are so-called good bots — bots that contribute positively to the online ecosystem. Google’s Googlebot falls into this category. Annoying as it might be for site owners when Googlebot comes along and gobbles through their bandwidth allowance, Googlebot is crucial to the functioning of the world’s most popular search engine, and search engines are necessary if we’re to find anything online.
But then there are the bad bots. Bad bots are the software agents of online criminals and other groups with malicious intent. A typical example of a bad bot is the XOR botnet — a botnet that searches for servers with weak SSH passwords to infect with malware. Other bad bot species include spam bots, which harvest emails for spamming, and bots that take part in distributed denial of service attacks against websites.
Incapula’s study breaks down the bot vs. human user base of the web like this:
- Humans – 51%
- Good bots – 19%
- Bad bots – 29%
There has been an increase in the relative amount of human traffic from 38.5% in 2013 to 51.5% in 2015. Good bots have decreased by around 11% this year. Bad bots have stayed at about the same.
That sounds like good news for humans and web hosts — bots use up a lot of bandwidth and bad bots in particular make no positive contribution to the online economy.
In fact, the situation isn’t quite so rosy, as Incapsula points out. Humans have increased as a proportion of web use and web use as a whole has increased. More people and more bots use the web. As the number of human web visits rises, there isn’t an associated rise in the number of good bots because most of the increase in human use is clustered around high-traffic sites. Good bot visits don’t increase as human traffic increases on a site, so the proportion of good bot visits drops as a whole.
Bad bots, on the other hand, hold steady as a proportion because they’re rising along with web use as a whole. Even though the number of human visits now outstrips the number of bot visits, most of the bot decline comes from the good bots, while the bad bots rampage undeterred.
The lesson from this study is, unfortunately, much the same as last year. Bad bots are a huge problem for the online ecosystem and economy, and there seems to be no indication that web hosting companies, security experts, and big web service providers are making much headway in the fight against malicious bot traffic.