What is rate limiting? | Rate limiting and bots

What is rate limiting?

Rate limiting is a strategy for limiting network traffic. It puts a cap on how often someone can repeat an action within a certain timeframe – for instance, trying to log in to an account. Rate limiting can help stop certain kinds of malicious bot activity. It can also reduce strain on web servers. However, rate limiting is not a complete solution for managing bot activity.

What kinds of bot attacks are stopped by rate limiting?

Rate limiting is often employed to stop bad bots from negatively impacting a website or application. Bot attacks that rate limiting can help mitigate include:

• Brute force attacks
• DoS and DDoS attacks
• Web scraping

Rate limiting also protects against API overuse, which is not necessarily malicious or due to bot activity, but is important to prevent nonetheless.

How does rate limiting work?

Rate limiting runs within an application, rather than running on the web server itself. Typically, rate limiting is based on tracking the IP addresses that requests are coming from, and tracking how much time elapses between each request. The IP address is the main way an application identifies who or what is making the request.

A rate limiting solution measures the amount of time between each request from each IP address, and also measures the number of requests within a specified timeframe. If there are too many requests from a single IP within the given timeframe, the rate limiting solution will not fulfill the IP address's requests for a certain amount of time.

Essentially, a rate-limited application will say, "Hey, slow down," to unique users that are making requests at a rapid rate. This is comparable to a police officer who pulls over a driver for exceeding the road's speed limit, or to a parent who tells their child not to eat so much candy in such a short span of time.

What is an IP address?

An IP address is the unique numerical (or, in IPv6, alphanumerical) identifier assigned to any device that connects to the Internet. Every device will have its own IP address for as long as it's online, and like a physical street address or a phone number, this enables devices to send messages back and forth. A traditional (IPv4) address looks like this: 198.41.129.1

For user devices, IP addresses are typically not permanent, because there are not enough IP addresses to go around in IPv4. Instead, the user's Internet service provider (ISP) will dynamically assign addresses as devices connect to the Internet.

A rate limiting solution may use an IP address as a basis for determining which devices are making too many requests and should be temporarily blocked.

How does rate limiting work with user logins?

Users may find themselves locked out of an account if they unsuccessfully attempt to log in too many times in a short amount of time. This occurs when a website has login rate limiting in place.

This precaution exists, not to frustrate users who have forgotten their passwords, but to block brute force attacks in which a bot tries thousands of different passwords in order to guess the correct one and break into the account. If a bot can only make 3 or 4 login attempts an hour, then such an attack is statistically unlikely to be successful.

Rate limiting on a login page can be applied according to the IP address of the user trying to log in, or according to the user's username. Ideally it would use a combination of the two, because:

• If rate limiting is only applied by IP address, brute force attackers could bypass this by attempting logins from multiple IP addresses (perhaps by using a botnet).
• If it's only done by username, any attacker that has a list of known usernames can try a variety of commonly used passwords with those usernames and is likely to successfully break into at least a few accounts, all from the same IP address.

Because rate limiting is necessary to prevent these brute force attacks, users who can't remember their passwords may be rate limited along with malicious bots. Users will likely see a "too many login attempts" message of some sort and be prompted to try again within a specified timeframe, or be advised that they are locked out of their accounts altogether.

How does rate limiting work for APIs?

An API, or application programming interface, is a way to request functionality from a program. APIs are invisible to most users, but they're extremely important for applications to function properly. For example, a restaurant's website could rely upon the API of a table reservation service to enable customers to make reservations online. Or, an ecommerce platform could integrate a shipping company's API to provide users with accurate shipping costs.

Every time an API responds to a request, the owner of that API has to pay for compute time: the server resources required for code to run and produce a response to that API request. In the example above, the restaurant's API integration will cause the table reservation service to pay for compute time whenever a restaurant customer makes a reservation.

For this reason, any application or service that offers an API for developers will have limitations on how many API calls can be made per hour or day by each unique user. In this way, third-party developers don't overuse an API.

Rate limiting can also motivate developers to pay more for leveraging the API: often they can only make so many API calls before paying more for the API service.

Rate limiting for APIs helps protect against malicious bot attacks as well. An attacker can use bots to make so many repeated calls to an API that it renders the service unavailable for anyone else, or crashes the service altogether. This is a type of DoS or DDoS attack.

How do social media platforms like X (Twitter) and Instagram use rate limiting?

Social media platform rate limiting is often similar to API rate limiting. Any third-party application that integrates X (formerly known as Twitter), for instance, can only refresh to look for new posts or messages a certain amount of times per hour. Instagram has similar limits for third-party apps. This is why users may occasionally encounter "rate limit exceeded" messages.

What is the difference between bot management and rate limiting?

Rate limiting is fairly one dimensional: While useful, it can only stop very specific types of bot activity. Additionally, rate limiting is not just for bots, but for limiting usage in general. Cloudflare Rate Limiting, for instance, protects against DDoS attacks, API abuse, and brute force attacks, but it doesn't necessarily mitigate other forms of malicious bot activity, and it doesn't distinguish between good bots and bad bots.

In contrast, bot management can holistically detect bot activity in general. For instance, Cloudflare Bot Management uses machine learning to identify likely bots, which enables it to block a wider variety of bot attacks (like credential stuffing, spam posting, inventory hoarding, etc.).Smaller organizations can also block bad bots with Super Bot Fight Mode, available on the Cloudflare Pro and Business plans.