Bots and crawlers eating your server: reading logs and rate-limiting

The command to run immediately

If your server is under load and you suspect a bot is responsible, the first thing to do is count requests by user agent in the access log. On a system with Nginx the default path is logs/access.log, but on Ubuntu and Debian it is usually found at /var/log/nginx/access.log, as indicated by the official documentation which explains that the access log is located by default in logs/access.log with combined format.

To count requests by IP address from the latest portion of the log:

awk '{print $1}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

To count by user agent (the field depends on the log format used; in combined format the user agent is the last field in quotes):

awk -F'\"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -20

On cPanel the raw logs for each domain are located in the account's domlogs folder (standard path /usr/local/apache/domlogs/<user>/<domain>); on Plesk it's better to go through the panel, section Websites and Domains > Logs, because the physical path changes depending on the version and distribution chosen during installation. Those without direct filesystem access can still download the log from the relevant panel section and run the same awk command on a local file.

If a single IP address appears at the top of the list with thousands of requests in just a few minutes, it's almost always an aggressive bot or scraper, not human traffic.

Common causes in order of frequency

1. SEO or AI crawlers that index aggressively (Googlebot, Bingbot, GPTBot, Bytespider, ClaudeBot and similar), often with many parallel requests to URLs with different parameters (filters, pagination, internal search).
2. Undeclared scrapers that spoof the user agent to look like a browser or known search engine, bypassing superficial blocks.
3. Bad bots impersonating Googlebot in the user agent but arriving from networks that don't belong to Google: this is the trickiest case because a crude block based only on user agent won't stop them, and an overly aggressive block risks catching the real Googlebot too.

How to distinguish a real search engine from a fake one

The user agent alone proves nothing: anyone can write "Googlebot" in an HTTP request. The method that Google itself recommends in its developer documentation is two-step cross-verification via DNS.

First step, reverse DNS lookup on the IP address that appears in the log:

host 66.249.66.1

If the IP really belongs to Googlebot, the result must be a hostname ending in googlebot.com, google.com or googleusercontent.com, as stated in the Google documentation describing how to confirm whether a domain name is googlebot.com, google.com or googleusercontent.com after the reverse DNS lookup.

Second step, forward DNS lookup on the hostname obtained, to check that it resolves back to the same starting IP:

host crawl-66-249-66-1.googlebot.com

If the result doesn't match the original IP, or the first lookup doesn't return a hostname in those domains at all, the request is not from real Googlebot, regardless of what the User-Agent header claims. This technique is called forward-confirmed reverse DNS and is the only method published by Google itself to validate one of its crawlers from the command line, without needing third-party tools.

The same principle applies to Bingbot (Microsoft publishes its own IP ranges) and other declared crawlers: if the hostname doesn't resolve to a domain owned by the operator, it's an impostor.

Three ways to limit them, in order of severity

1. Crawler rules (robots.txt)

This is the lightest and most polite intervention: you ask the crawler to slow down or not visit certain sections. However, it's worth noting that Google does not support the crawl-delay directive: the official documentation lists the fields supported by Google in robots.txt (user-agent, allow, disallow, sitemap), specifying that other fields like crawl-delay are not supported. So for Googlebot, crawl-delay in robots.txt is simply ignored: writing it won't help you slow down Google.

It does work for other crawlers that respect it (though not all): a typical robots.txt file to completely block access to a non-essential aggressive bot is:

User-agent: Bytespider
Disallow: /

This only blocks bots that read and respect the file. A malicious bot or scraper that doesn't follow the rules simply ignores it: you need the next level for those.

2. Rate limiting on the web server

On Nginx the ngx_http_limit_req_module module allows you to set a ceiling on requests per second for a given key, typically the IP address or user agent. A minimal example to insert in the http {} block:

limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;

server {
 location / {
 limit_req zone=perip burst=20 nodelay;
 limit_req_status 429;
 }
}

This limits each individual IP to 10 requests per second, with a margin of 20 requests in burst, returning code 429 ("Too Many Requests") to anyone exceeding the threshold.

When aggressive traffic comes from a declared bot but distributed across hundreds of different IPs (this happens with some crawlers from ad platforms), limiting by IP won't help: you need to group requests by user agent with a map directive, so all requests from that bot share the same counter regardless of the originating IP:

map $http_user_agent $limit_bot {
 default "";
 ~*GPTBot "gptbot";
}

limit_req_zone $limit_bot zone=gptbot:1m rate=10r/m;

location / {
 limit_req zone=gptbot burst=20;
 limit_req_status 429;
}

On Apache the equivalent is the mod_ratelimit module or, more often in cPanel/Plesk environments, a WAF-level rule (ModSecurity) or a limit enforced by the panel itself: it's worth checking your panel's documentation to see which module is active before writing manual rules, because a conflict between two rate-limiting systems can give unpredictable results.

3. Total blocking

The last resort, to be used only when the first two levels aren't enough and the IP has been verified as not belonging to a legitimate crawler. On Nginx, at server or http level:

deny 203.0.113.45;

Or for an entire network block:

deny 203.0.113.0/24;

A block should always be verified after applying it, by sending a test request from the blocked IP (or using curl --resolve from another host) and checking that the response is 403 or the connection drops, and not that the bot continues to get through because the rule is in a position in the configuration file that isn't read for that location.

What never to block

Before blocking an IP based on its hostname ("looks like Googlebot"), always perform the two-step verification described above. Accidentally blocking real Googlebot, real Bingbot or other active search engine crawlers means ceasing to be indexed: your site gradually disappears from search results, causing damage that becomes visible weeks later, when it's hard to connect to the cause. When in doubt, it's better to apply a soft rate limit first (high burst, nodelay disabled) rather than a hard block, and observe the log effects for a couple of days before tightening further.

If after applying all three levels the load remains high, the problem might not be the bot but page generation itself (unoptimised queries, missing caching): in that case it's worth measuring the backend response time upstream of the bot, because that's where you can tell if the page is slow or the bot is too aggressive.