DB-유

Blocking Bad Bots with nginx

· linux, nginx, security

Open the access log of any public web server and a big share of the requests aren't from people. Scrapers, scanners hunting for wp-login.php on a site that has never run WordPress, "SEO tools" crawling at a rate no search engine would. I watched one of them use more CPU than my real users combined. Polite crawlers respect robots.txt; the trouble ones don't. nginx can turn most of them away before they reach the application.

What follows is a layered filter: a user-agent blocklist with map, per-client rate limiting, denial of probe paths, and an IP blocklist for repeat offenders. It works on nginx 1.14+ on Ubuntu 18.04/20.04 or CentOS 7/8.

Look at your own log first, so you block what's actually hitting you: awk -F'"' '{print $6}' /var/log/nginx/access.log | sort | uniq -c | sort -rn | head -30 lists the most frequent user agents.

User-agent blocklist with map

The map directive evaluates a variable once per request and is far cheaper than a chain of if statements. It must live in the http context, so put it in /etc/nginx/conf.d/badbots.conf:

map $http_user_agent $bad_bot {
    default                       0;
    ~*(ahrefsbot|semrushbot|mj12bot|dotbot|blexbot)   1;
    ~*(python-requests|go-http-client|curl|wget)      1;
    ~*(masscan|nikto|sqlmap|nmap)                     1;
    ""                            1;
}

~* makes the match case-insensitive. The empty string entry blocks requests with no user agent at all, which legitimate browsers never send. Be careful with the second line: curl and python-requests are also what your own monitoring and API clients may use, so remove them or add exceptions for your own hosts (see the probe-paths section below). I forgot this once and my own uptime monitor got blocked.

Applying the map

server {
    listen 443 ssl http2;
    server_name example.com;

    if ($bad_bot) {
        return 444;
    }

    # ... rest of the configuration
}

return 444 is nginx-specific: it closes the connection without sending any response, which wastes the least of your resources and gives the client nothing to parse. Use return 403 instead if you want a proper response in the logs of whoever is on the other end. This is one of the few places where if inside server is safe, because it only ever returns.

Rate limiting

Bots that rotate user agents still tend to come from one address at a time. Define a shared memory zone in http:

limit_req_zone $binary_remote_addr zone=perip:10m rate=10r/s;
limit_req_status 429;

Then apply it where it matters, typically to dynamic content rather than static assets:

location / {
    limit_req zone=perip burst=20 nodelay;
    try_files $uri $uri/ /index.php?$args;
}

location ~* \.(css|js|png|jpg|gif|svg|woff2)$ {
    expires 30d;
}

Ten requests per second sustained with a burst of twenty is generous for a human and restrictive for a scraper. nodelay serves the burst immediately rather than queuing it. The 10 MB zone holds about 160,000 addresses.

Probe paths, and whitelisting your own tools

Requests for files that do not exist on your site are a strong signal. Deny the usual suspects:

location ~* /(wp-login\.php|xmlrpc\.php|\.env|\.git/|phpmyadmin|cgi-bin/) {
    return 444;
}

Only include paths your site genuinely does not use. To exempt your own monitoring from the user-agent block, layer a second map on the client address and combine them:

geo $trusted_ip {
    default        0;
    10.0.0.0/8     1;
    203.0.113.5    1;
}

map "$bad_bot:$trusted_ip" $block_request {
    default   0;
    "1:0"     1;
}

Then use if ($block_request) instead of if ($bad_bot) in the server block.

IP blocklist

For addresses that keep coming back, a plain include file is easy to manage by script:

# /etc/nginx/conf.d/blocklist.conf
geo $blocked_ip {
    default        0;
    include        /etc/nginx/blocked-ips.txt;
}

/etc/nginx/blocked-ips.txt contains one entry per line in 198.51.100.23 1; form. A cron job can append to it from log analysis and run nginx -s reload. For automated banning based on log patterns, fail2ban with its nginx-limit-req and nginx-botsearch jails does the same thing at the firewall level.

Testing

sudo nginx -t && sudo systemctl reload nginx
curl -A "AhrefsBot/7.0" -sI https://example.com/            # expect: curl: (52) Empty reply from server
curl -A "" -sI https://example.com/                          # same
curl -A "Mozilla/5.0" -sI https://example.com/ | head -1     # expect: HTTP/2 200
ab -n 100 -c 20 https://example.com/ 2>/dev/null | grep 'Non-2xx'

The last command (from apache2-utils) should show a number of 429 responses once the burst is exhausted, and /var/log/nginx/error.log will contain limiting requests, excess: ... by zone "perip".

Side effects

  • Legitimate users hit 429. Everyone behind one corporate NAT shares an address. Raise rate or burst, or key the zone on $binary_remote_addr$http_user_agent to spread them out slightly.
  • Your uptime monitor reports the site down after enabling the block. Its user agent matched the list. Either whitelist its IP with the geo map or remove the generic library names from the pattern.
  • nginx: [emerg] "map" directive is not allowed here. map, geo and limit_req_zone must be in http, not server. Move them to conf.d/.
  • Googlebot blocked and rankings drop. A pattern like bot on its own is far too broad. Match specific names, and verify search engine crawlers by reverse DNS rather than user agent if you must treat them specially.

None of this stops a determined attacker, and it doesn't need to. It just makes the logs readable again. Review the list every few months, because bot names change.