This post is also available in 简体中文, 繁體中文, 日本語, Bahasa Indonesia, ไทย.
Almost half of the Internet’s traffic is powered by bots. Bots have scoured the net for years, relentlessly hacking into bank accounts, scooping up Bruno Mars tickets, and scraping websites for data. The problem is so widespread that we launched Bot Fight Mode in 2019 to fight back. Since then, over 150,000 individuals and small businesses have used the product, and we’ve received countless requests for more functionality. More analytics, more detections, and more controls.
Introducing Super Bot Fight Mode.
Beginning immediately, any Cloudflare user with a Pro or Business site can take new action against bots. We’ve added advanced features in the dashboard and some exciting updates to analytics. Free customers will retain all the benefits they've enjoyed with Bot Fight Mode, and our Enterprise Bot Management product will continue to push the needle on innovation.
In the Dashboard
Our bot solutions have a new home. The features we discuss in this blog post go beyond a single toggle, so we created a hub for bot protection. Head to the Firewall app and select the “Bots” subtab to get started.
The new hub is live for all users, including those with Enterprise Bot Management.
Pro Plan Features
First up: we’re bringing our popular Bot Report to the Pro plan. Here, you can see a breakdown of your bot traffic, updated in real time to help you spot attacks.
The Bot Report includes three traffic types:
- Likely automated traffic may have come from bad bots. We use heuristics, machine learning, and other techniques to spot these requests. In most cases, this traffic will hurt your site without providing anything useful in return.
- Likely human traffic is legitimate and important. Ideally, the vast majority of your traffic matches this type.
- Verified bot traffic comes from good bots on the Internet. We have verified search crawlers like Google and payment notification services like PayPal. Most users choose to allow this traffic.
All of this data is available via GraphQL as well. So if you are looking to routinely monitor bot traffic, the API will help you do so.
Pro users can also do more to stop bots — select “Configure Super Bot Fight Mode” to add protection. Highlights include:
- The option to challenge or block traffic from “definitely automated” sources. Note that this will only affect the traffic we are most confident comes from bots.
- The option to enable JavaScript Detections to identify headless browsers and other actors on the Internet.
- The option to include or exclude verified bots from protection.
If your site interacts with Slack, for example, you can exclude verified bots to help Slackbot do its job. Or if you notice an increase in ad fraud, try challenging automated traffic and watch the results.
Business Plan Features
Bot Analytics is now included with the Business plan.
We originally launched Bot Analytics to give our Enterprise users more visibility. Since the launch, however, Business users have asked us for many of the same insights. And because Cloudflare has always tried to democratize technology (as we’ve done with Firewall Events and other products), this is something we had to do.
Business users can access a new version of Bot Analytics; one that is designed to work with the mitigation tools described below. Users can view traffic by type, adjust the time frame, and filter by different attributes like IP address or user agent.
Another perk: Bot Analytics shows how we categorize traffic. Scroll to “requests by detection source” to understand which engine flagged a particular request. If you want to learn more about our detection engines, check out our blog post on the topic.
Of course, we also added new mitigation features. While Pro users can defend against “definitely automated” traffic, Business users can also target “likely automated” traffic. What’s the difference? The latter includes requests scored by our machine learning engine. These requests often come from sophisticated bots — the ones that evade simple security tools by rotating IPs or convincingly imitating humans.
Perhaps your site suffers from inventory hoarding. You list items for sale, but they are almost immediately claimed by bots. Understandably, your customers are upset (and so are you!). Go ahead and use Bot Analytics to pinpoint the attacker, and if the attack falls under “likely automated,” consider blocking this traffic.
We also realize that different sites may have different sensitivities to bot traffic. Users can respond appropriately by issuing a challenge, blocking entirely, or doing nothing at all.
These features are all included in the Cloudflare Business plan. Once you enable mitigation, check your Firewall Events tab to watch traffic get blocked or challenged.
Enterprise Bot Management
For those with more advanced security needs, Bot Management remains the gold standard. And it’s only getting better.
Unlike Bot Fight Mode, Bot Management is built directly into the Firewall. This means that users can restrict their bot protection to a particular path (like a /login endpoint). Bot Management also includes granular bot scores, which users can pair with other attributes to produce more powerful protection. It even includes Anomaly Detection, which we use to recognize outlier patterns on your site.
We also continue to improve Bot Management. For example, just moments ago, we announced early access to API Abuse Detection. This announcement follows months of research and development. We’re using unsupervised learning to map out APIs, identify legitimate user flows, and keep out bad bots. The end result: Cloudflare will be able to protect your mobile apps (without an SDK) and secure your API endpoints (without any provided schema). Read more about the early access period.
These features (and countless others) will continue to guard the Internet’s largest sites. If you think you need Bot Management, let us know.
Helping to Build a Better Internet
Cloudflare’s goal has always been to help build a better Internet. This mission extends to every part of the Internet — and to every person who uses it.
Today’s introduction of Super Bot Fight Mode was born from this mission, particularly from the idea that we are stronger as a united front against bots. Each website we protect is one that bots will waste their resources on. At Cloudflare, we are actively fighting back, and unleashing new challenges that will disincentivize bot operation with tarpitting.
We encourage you to enable Super Bot Fight Mode today. Cloudflare now offers bot protection with every plan (including Free), so there’s no excuse not to try it! Test the new features and let us know what you think.