Bot traffic on your site: reading it from AWStats and logs without blocking Googlebot

Not all bot traffic should be blocked. In fact, in most cases it's worth letting it through: without Googlebot and without AI assistant crawlers, a site won't be indexed and won't be accessible to people searching through new conversational tools. Problems arise when content scrapers, brute-force attempts or bots spamming contact forms slip into automated requests. An article published on 21 October 2026 on the Serverplan blog explains how to distinguish between the two categories using tools already available on hosting with cPanel.

Three families of bots

The source divides bots into three groups: those that are useful and necessary (Googlebot, Bingbot, GPTBot, OAI-SearchBot, uptime monitoring systems), those that are neutral and consume only a bit of bandwidth (third-party SEO tools, data aggregators), and those that are harmful, designed to search for vulnerabilities, force access, send spam or download content.

Context helps explain why the issue is no longer marginal. According to Cloudflare Radar data released in June 2026, automated requests now account for 57.5% of HTML traffic to web content, against 42.5% generated by humans—the first time in internet history that machines are in the majority. The figure refers to HTML traffic monitored by Cloudflare and does not include video, email or gaming.

Where to look: AWStats and raw logs

On cPanel hosting with Apache web server, as noted by the source, AWStats automatically separates traffic identified through known User-Agent strings, showing visit volume, data transferred and the date of last access for bots such as Googlebot, Baiduspider or YandexBot. For those wanting more detail, raw logs remain the most comprehensive source: they can be downloaded from cPanel, Metrics section, Raw Access entry, in .gz format. From there you can read the complete User-Agent strings and identify suspicious patterns such as repeated requests from the same IP within seconds or sequential URL scans.

What to do with unwanted bots

The robots.txt file remains the first-level tool: it indicates scanning rules but doesn't block a malicious actor bent on ignoring them. For real filtering you need WAF solutions like Cloudflare or Sucuri, which recognise signatures of known bots and apply CAPTCHAs to suspicious IPs, possibly combined with honeypots using invisible links for users but visible only to scanning bots.

Serverplan is a hosting provider; AWStats is included by default in cPanel plans with Apache.