Posted on

SEMrush keeps taking down my site

My site is being absolutely bombarded by page requests originating from semrush.com. Its a bot that is indexing and downloading links to my entire site, which would be fine if it were a small website, except that I have content going back to 1996, which is rather a lot, and they are hitting the server with so many requests that Apache2 and/or MySQL Server 5.7 is crashing.

It’s utterly frustrating, and they have taken the site down 3 times in the past couple of hours. Their bot should be rate-limited so that it doesn’t cause issues like this! The site can cope with 100+ simultaneous visitors as it is running on an AWS T3 micro instance, which should be more than capable, but unfortunately, its not capable of dealing with the SEMrush bot.

In the coming hours I will be working on ways to permanently blacklist their services, and I suggest that other website owners do the same so that the same misfortune does not befall their site too.

Posted on

Ubuntu – set the time zone so that it takes BST into account

On Ubuntu, if you set the time manually using

sudo locale-gen en_GB.UTF-8

then the problem is that it does not take BST (British Summer Time) into account, so during BST the server time is out by 1 hour. This is obviously an issue if you have time restrictions on logging on to whatever system is hosted on the server.

The solution is to run:

sudo dpkg-reconfigure tzdata

Follow through the on-screen prompts to set the locale to Europe and then London this then solves the issue, and the server automatically stays at the correct time when the clocks change.