Bot traffic: What it is and why you should

Today, bots are an important part of the online world. They help us order food, play music on our Slack channel, and pay our coworkers back for the delicious smoothies they bought us. The internet is also full of bots that do what they were made to do. What does this mean for people who own websites? And, what does this mean for the environment (which may be even more important)? Read on to learn what you need to know about bot traffic and why it matters.

What are bots?

First, let's talk about the basics: A bot is a piece of software that can do things automatically on the internet. Bots can act like real users or even take their place. They are very good at doing boring jobs over and over again. They are also quick and effective, which makes them a great choice if you need to do something on a big scale.

How do bots get traffic?

Bot traffic is any traffic to a website or app that isn't from real people. Which is something that happens a lot on the internet. If you have a website, there's a good chance that a bot has been there. In fact, bot traffic makes up almost 30% of all traffic on the internet right now.

Is traffic from bots bad?

You've probably heard that having bots visit your site is bad for it. And that's true a lot of the time. But there are also good bots that aren't scams. It depends on what the bots are meant to do and what their makers want. Some bots are necessary for digital services like search engines and personal helpers to work. But some hackers want to get into your website by force and steal private information. So, which bots are 'good' and which are 'bad'? Let's learn more about this subject.

The 'good' robots

Bots that are "good" do things that don't hurt your website or computer. They let you know who they are and what they do on your website. Search engine robots are the most common 'good' bots. Search engines can't give you information when you're looking for something unless crawlers visit your website and find content. So, when we talk about "good" bot traffic, we mean this type of bot.

Other good internet bots besides search engine robots are:

SEO crawlers: If you work in SEO, you've probably used tools like Semrush or Ahrefs to study keywords or find out more about your competitors. For these tools to give you information, they also need to send out bots to crawl the web and collect data.

Commercial bots: These bots are used by businesses to crawl the web and find information. For example, research companies use them to keep track of market news, ad networks use them to keep track of and improve the performance of display ads, and "coupon" websites use them to collect discount codes and sales programs for their users.

Site-monitoring bots: help you keep track of how long your site is up and other data. They check and share data like the status of your server and how long it has been up for. This lets you know when something is wrong with your site and lets you fix it.

Feed/aggregator bots: They find newsworthy content and put it all together so it can be sent to your site viewers or email subscribers.

The "bad" bots

"Bad" bots are made with the purpose of doing harm. You've probably seen spam bots that flood your website with useless comments, backlinks that don't go anywhere, and awful ads. You may also have heard that bots take people's places in online raffles or buy all the good seats at shows.

Because of these harmful bots, bot traffic has a bad name, which is fair. There are a lot of bad bots on the internet these days, which is a shame.

Here are some bots you don't want on your website:

Email scrapers collect email addresses and send bad emails to those people.
Comment spam bots Flood your website with comments and links that send people to a bad website. Most of the time, they spam your website to try to sell or get backlinks to their own sites.
Scraper bots These bots go to your page and download everything they can find. This can include your writing, images, HTML files, and even videos. Then, people who run bots will use your information again without your permission.
Bots for credential stuffing or brute force attacks These bots will try to get into your website to steal private information. They try to do this by logging in as a real person.
Botnets and zombie computers are networks of devices that have been hacked and are used to launch DDoS attacks. Distributed loss of service is what DDoS stands for. During a DDoS attack, the attacker uses a network of devices like this to send a lot of bot traffic to a website. This sends too many requests to your web server, making your website slow or useless.
Inventory and ticket bots They go to websites to buy up tickets for entertainment events or to buy a lot of newly released goods. Brokers use them to make money by reselling tickets or goods at a higher price.

Why bot traffic should bother you

Now that you know what bot traffic is, let's talk about why it matters to you.

For how well your page works

Malicious bot traffic puts stress on your web server and can even cause it to crash. With their requests, these bots use up your server's bandwidth. This slows down your website or, in the case of a DDoS attack, makes it impossible to get to at all. In the meantime, you might have lost customers and traffic to other businesses.

Malicious bots also try to look like normal human traffic, so you might not see them when you look at your website's data. What happened? You might see odd traffic spikes but not know why. Or, you might not understand why your site gets visitors but no sales. You can probably guess that this could hurt your business choices because you don't have the right information.

For the safety of your site

Bots that do bad things are also bad for the security of your site. They will try different username/password combinations to get into your site, or they will look for weak entry points and tell their handlers about them. If your website has security holes, these bad actors might even try to put bugs on it and spread them to your users. And if you run an online store, you'll have to handle private information like credit card numbers, which hackers would love to steal.

To the environment

Did you know that the environment is affected by bot traffic? When a bot comes to your site, it sends an HTTP request to your server asking for information. Your server needs to reply and then send back the information that was asked for. When this happens, your computer has to use a little bit of power to finish the request. Think about how many bots are on the internet. You can probably guess that bot traffic takes up a huge amount of energy.

In this way, it doesn't matter if a good or bad bot sees your site. The same steps are still taken. Both need energy to do their jobs, and both affect the world in some way.

Search engines are an important part of the internet, but they also waste a lot of time. They might go to your site too often and still miss the right changes. Checking your server log to see how often robots and bots visit your site is a good idea. Google Search Console also has a report called "crawl stats" that tells you how often Google crawls your site. Some of the facts there might surprise you.

What can you do about 'bad' bots?

You can try to find bad bots and stop them from getting in. This will save you a lot of bandwidth and make your computer less busy, which will save energy. The easiest way to do this is to stop a single IP address or a group of IP addresses. If you see strange data coming from an IP address, you should block it. This method does work, but it takes a lot of work and time.

You can also use a bot control service from a company like Cloudflare. These companies keep a large list of both good and bad bots. They also use AI and machine learning to find and stop bad bots before they can do damage to your site.

Security plugins

If you're running a WordPress site, you should also install a security tool. Some of the most popular security plugins, like Sucuri Security and Wordfence, are kept up to date by companies that hire security developers to keep an eye out for problems and fix them as they come up. Some security plugins will block certain "bad" bots for you immediately. Others let you see where strange traffic is coming from and then decide how to handle it.

What about the bots that are 'good'?

We've already said that "good" bots are good because they do things that are necessary and clear. Still, they can use a lot of energy. Not only that, but these bots might not help you at all. Even though what they do is thought to be "good," it could still hurt your website and the world. So, what can you do to help the bots that are good?

1. Don't let them through if they don't help.

You must decide if you want these "good" bots to crawl your site or not. Does it help you that they crawl your site? Do they help you more than it hurts your servers, their servers, and the world when they crawl your site?

2. Slow down the crawl speed

If the crawl delay in robots.txt is supported by bots, you should try to slow down how often they crawl. So, they won't come back every 20 seconds to crawl the same links over and over again. Because, let's face it, you probably won't change the text of your website 100 times in a single day. Even if your website is bigger than most.

You should try out different crawl rates and keep an eye on how they affect your website. Start with a small delay and then raise it when you're sure it won't hurt anything. Also, you can give crawlers from different sources a different crawl delay rate. You can't use this for Google bots because Google doesn't support craw delay.

3. Teach them how to crawl better

There are many places on your website where bots shouldn't go. Like, say, your personal search results. So, you should use the robots.txt file to stop them from getting in. Not only does this save energy, but it also helps you get the most out of your crawl budget.

Next, you can make it easier for bots to crawl your site by getting rid of the extra links that your CMS and apps make automatically. For example, WordPress makes an RSS feed for your website comments on its own. This RSS feed has a link, but not many people look at it, especially if there aren't many comments. So, it's possible that this RSS feed won't help you in any way. It just gives bots another link to follow over and over, which wastes energy.