SEMrush keeps taking down my site

My site is being absolutely bombarded by page requests originating from semrush.com. Its a bot that is indexing and downloading links to my entire site, which would be fine if it were a small website, except that I have content going back to 1996, which is rather a lot, and they are hitting the server with so many requests that Apache2 and/or MySQL Server 5.7 is crashing.

It’s utterly frustrating, and they have taken the site down 3 times in the past couple of hours. Their bot should be rate-limited so that it doesn’t cause issues like this! The site can cope with 100+ simultaneous visitors as it is running on an AWS T3 micro instance, which should be more than capable, but unfortunately, its not capable of dealing with the SEMrush bot.

In the coming hours I will be working on ways to permanently blacklist their services, and I suggest that other website owners do the same so that the same misfortune does not befall their site too.

Setting up the LAMP Stack from scratch on Ubuntu

When migrating servers to AWS, its great to be able to set them up easily and quickly. This is effectively a lovely cheat-sheet. By running the scripts below you can have a full LAMP Stack running on an EC2 instance running Ubuntu in less than 5 minutes.

/* Set locale to UK - V IMPORTANT!! */
sudo locale-gen en_GB.UTF-8


/* Run update and upgrades */
sudo apt-get update
sudo apt-get upgrade


/* Install Apache 2 */
sudo apt-get install apache2


/* Server internal firewall config */
sudo ufw allow OpenSSH
sudo ufw allow in "Apache Full"
sudo ufw enable


/* Install MySQL */
sudo apt-get install mysql-server


mysql_secure_installation

(No,No,Yes,Yes,Yes,Yes)

/* Install php and associated plugins as needed */
sudo apt-get install php libapache2-mod-php php-mcrypt php-mysql
sudo apt-get install php-curl php-gd php-mbstring php-mcrypt php-xml php-xmlrpc


/* Restart services */
sudo systemctl restart apache2


/* Update Apache2 config to allow .htaccess files and modified file permissions */
sudo nano /etc/apache2/apache2.conf

<Directory /var/www/html/>
AllowOverride (Change None to All)
</Directory>

/* Turn on rewrite rules for Apache2 */
sudo a2enmod rewrite
sudo systemctl restart apache2


/* Update permissions so that FTP can write to the web root folder */
sudo chown -R ubuntu:www-data /var/www/html


/* Reboot server */
sudo reboot