Bot Management

Bots generate nearly half of all internet traffic. While many bots serve legitimate purposes like search engine crawling and content aggregation, others originate from malicious sources. Bot management encompasses both observing and controlling all bot traffic. A key component of this is bot protection, which focuses specifically on mitigating risks from automated threats that scrape content, attempt unauthorized logins, or overload servers.

Bot management systems analyze incoming traffic to identify and classify requests based on their source and intent. This includes:

• Verifying and allowing legitimate bots that correctly identify themselves
• Monitoring bot traffic patterns and resource consumption
• Detecting and challenging suspicious traffic that behaves abnormally
• Enforcing browser-like behavior by verifying navigation patterns and cache usage

To effectively manage bot traffic and protect against harmful bots, various techniques are used, including:

• Signature-based detection: Inspecting HTTP requests for known bot signatures
• Rate limiting: Restricting how often certain actions can be performed to prevent abuse
• Challenges: Using JavaScript checks to verify human presence
• Behavioral analysis: Detecting unusual patterns in user activity that suggest automation

With Vercel, you can use:

• Managed rulesets to challenge specific bot traffic
• Rate limiting and challenge actions with WAF custom rules to prevent bot activity from reaching your application
• DDoS protection to defend your application against bot driven attacks
• Observability and Firewall to monitor bot patterns, traffic sources, and the effectiveness of your bot management strategies

With Vercel, you can use the bot protection managed ruleset to challenge non-browser traffic from accessing your applications. It filters out automated threats while allowing legitimate traffic.

• It identifies clients that violate browser-like behavior and serves a javascript challenge to them.
• It prevents requests that falsely claim to be from a browser such as a curl request identifying as Chrome.
• It automatically excludes verified bots, such as Google's crawler, from evaluation.

To learn more about how the ruleset works, review the Challenge section of Firewall actions. To understand the details of what get logged and how to monitor your traffic, review Firewall Observability.

You can apply the ruleset to your project in log or challenge mode. Learn how to configure the bot protection managed ruleset.

Vercel's AI bots managed ruleset allows you to control traffic from AI bots that crawl your site for training data, search purposes, or user-generated fetches.

• It identifies and filters requests from known AI crawlers and bots.
• It provides options to log or deny these requests based on your preferences.
• The list of known AI bots is automatically maintained and updated by Vercel.

When new AI bots emerge, they are automatically added to Vercel's managed list and will be handled according to your existing configured action without requiring any changes on your part.

You can apply the ruleset to your project in log or deny mode. Learn how to configure the AI bots managed ruleset.

Vercel maintains and continuously updates a comprehensive directory of known legitimate bots from across the internet. This directory is regularly updated to include new legitimate services as they emerge. Attack Challenge Mode and bot protection automatically recognize and allow these bots to pass through without being challenged. You can block access to some or all of these bots by writing WAF custom rules with the User Agent match condition or Signature-Agent header. To learn how to do this, review WAF Examples.

To prove that bots are legitimate and verify their claimed identity, several methods are used:

• IP Address Verification: Checking if requests originate from known IP ranges owned by legitimate bot operators (e.g., Google's Googlebot, Bing's crawler).
• Reverse DNS Lookup: Performing reverse DNS queries to verify that an IP address resolves back to the expected domain (e.g., an IP claiming to be Googlebot should resolve to *.googlebot.com or *.google.com).
• Cryptographic Verification: Using digital signatures to authenticate bot requests through protocols like Web Bot Authentication, which employs HTTP Message Signatures (RFC 9421) to cryptographically verify automated requests.

Submit a bot request if you are a SaaS provider and would like to be added to this list.