How is an Internet bot constructed?

What does 'bot' mean?

An Internet bot is a computer program that runs on a network. Bots are programmed to automatically do certain actions, such as crawling webpages, chatting with users, or attempting to break into user accounts.

Unlike the manufacturing robots used in factories, or the "battle bots" built by robotics hobbyists, a bot is really just a few lines of code with a database. Another way to put it is that an Internet bot is a set of instructions for computers, plus a collection of information. Most bots are fairly simple in design, but some bots are more complex and use artificial intelligence (AI) in an attempt to imitate human behavior.

Writing a bot is fairly easy for most developers, and sometimes even for non-developers. This is part of the reason why bots are so widespread on the Internet. In some cases, it isn't even necessary to write actual lines of code in order to create a bot – for instance, Twitter provides a visual interface for users to create bots that tweet, retweet, like, and perform other actions on the social network.

What are the main components of an Internet bot?

A bot's architecture usually includes the following:

• Application logic
• Database
• API integrations

The application logic is the executable, machine-readable code that the bot developer writes and a computer executes. The chatbot code example above fits in this category.

The database is the collection of data that the bot draws from in order to know what actions to take. A bot can save additional information to its database, such as when a web scraper bot downloads content from a website.

APIs allow the bot to use external functionalities without the developer needing to write them. All the developer has to do is add the right commands into the code, and the bot will call an API as needed.

(An API is a way to incorporate complex software functionality that someone else has already built. Think of an API as a way to avoid "reinventing the wheel" when programming an application. For example, a chatbot could use a weather app's API to provide users with detailed information about the weather if they ask for it. That way the chatbot doesn't need to track the weather itself – instead it just calls the external weather app's API.)

Unlike applications that users are more used to interacting with, most bots don't have a user interface. This is because bots on the Internet usually interact with webpages, apps, and APIs, not users (although they can interact with users via chat, social media, and other channels).

How can websites and apps deal with excessive bot traffic?

Because bots are relatively simple to create, they are extremely common on the Internet – about half of all Internet traffic is from bots, both good bots and bad bots.

Some bots, like web crawler bots and chatbots, are essential for helping the Internet work properly and allowing users to find the information they need. However, excessive bot traffic can overwhelm a web property's origin servers, and malicious bots can carry out a variety of cyber attacks. To prevent these occurrences, websites and web apps can use robots.txt files strategically, implement rate limiting, and leverage bot management solutions.

Want to dive deeper into bot development? See this blog post on how to write a serverless chatbot.