My website is plagued by two robots that have no name, no IP nothing at all.
The first one is the well known
Unknown robot identified by bot slash*
and the other one is the
Unknown robot (identified by hit on robots.txt)
I used to block the first one that is the most persistent on my .htaccess file but that stopped working at some point.
I don’t want also to add yet another plugin just for blocking these damn bots. My website seems to be just fine as it is now. It has the golden ratio of plugins.
Looking up and down online I found this discussion.
Is it of any valid, I mean can I block these bots from index.php that is modified for CP or any attempt to edit it, it will bring the universe down??
This is the best (most efficient) way to block small numbers of bots.
You shouldn’t put this code in index.php because it will be overwritten the next time ClassicPress is updated.
You can put it in e.g. wp-content/mu-plugins/block-bad-bots.php, and it looks like the code you linked should work fine there. Don’t forget the opening <?php tag on the first line.
All bots (all visitors to your site in general) do have an IP address. They should also be sending a User-Agent header which is one way that bot detection works. You can find the raw values for both of these pieces of information in the server logs provided by your web host.
If you’re able to, try using modsec. It’s what I use and has cut down on the number of bad bots significantly (and I include the likes of ahrefs in that list). It may be that it’s something you need to discuss with your hosting company.
modsec will also provide an IP address for each bad bot.
But be warned, if it’s not something you’ve used before, it can be a bit difficult (and somewhat dangerous) to set up.
Thank you for the suggestion, but I will not try it. If something goes wrong I will not be able to fix it and I can’t even think to spend time fixing my website. I already spend a considerable amount of time and effort to update it, time that I could use it to paint.
This website is there to show my art. And my aim is to improve my art not the website.
I think that it is more than good the way it is if you think that I made it from scratch and by reading online tutorials.
I’m very proud for the final result but I don’t want to start messing up with it. It works. Let it be. LOL
@Marialena.S Found an interesting plugin at Blackhole for Bad Bots that sets a trap for bots. It adds an entry in the robots.txt file and if a Bot follows it, you can configure what happens.
I set it to display a message and send me an email. Using a VPN, I went to the trap page and got greeted with the red message. Refreshing the page then displays my custom message. The most important Bots are already whitelisted by the plugin and do not get blocked.
I often read Jeff Star’s blog at Perishable Press. Some of the “tricks” he suggests are quite useful and amusing. Here’s a short article that gives a bit of an insight to the Blackhole plugin.
@Aussie I think that I have used this plugin at some point in the past. The thing that I don’t understand is why that plugin leaves a message for the bot. Can the bot read it perhaps?? lol @1stepforward Thank you … I’ll check it out and I’ll be back.
In practice it makes very little difference. This type of “security through obscurity” has little to no benefit, and even some drawbacks that are worth considering. (It is a similar story with things like hiding your version number - many common practices related to web security are unhelpful or wrong.)
Some of the users that will see your error message are actually real users. These techniques always have false positives so it is pretty unfriendly to have the site just show a blank page or a generic error. This is also the reason why blocking entire countries or IP ranges is usually a bad idea - you will block legitimate users too.
Many bots are just misbehaved rather than malicious, in which case if anyone does happen to see the error message and act on it, then the most likely action is they will fix their bot.
For the bots that are malicious (harvesting for email addresses and other spammable info), most of them are “dumb” in the sense that they just look for patterns and don’t bother reporting anything else back to the owner.
For the few bots that are malicious and smart about what they are doing, there are hundreds of available detection methods and there is basically no way to hide the fact that you are using a security plugin, or even which one you are using. At that point you probably have a determined and resourceful attacker that is looking at your site specifically, and you are playing a completely different game.
tl;dr the default behavior of showing an error message is probably the best choice.
I think the trick is to use multiple systems. I consider a plugin to be the last line of defence and hardening the server to be the most important consideration. Yes, it may slow down your website very slightly but not enough to make any noticeable difference. Not in my experience anyway.
Regarding the protection of my ClassicPress sites (but this is valid for non-CP sites as well), I have recently moved all my sites to CleanTalk. And I am quite pleased with it. Moreover, I had recently issues of odd spam getting through, and their support proved to be very committed for adjusting to very specific needs with a third-party contact form. They seem to be a great team.
I do not know if you have already tried them, but I would consider that, as least for testing. What is great with their spam firewall is that: “Spambots are blocked before they get access to the website, it prevents the loading of pages of a website spam bots, so your web server doesn’t need to run all the scripts on these pages. This can reduce the load on the database and web server.”
On two of my ClassicPress sites, I have Shield installed along with CleanTalk, and I have not come across issues until now.
An update to this thread.
Thank you all for your suggestions. I have tested up until now countless plugins and firewall, I edited also the robots.txt and .htaccess files but nothing seems to be able to block this post.
@jfmayer I haven’t test yet this plugin but I don’t have problem with spam comments but with that bot that consumes a huge amount of the bandwidth.
And though the problem with this bot is known for more than a decade nobody seems to know or gives online an applicable solution on how to block it.
P.S. Sorry for my absence all this time but I have joined two group gallery exhibitions and I was lost in outer space. And now I want to join another one with a new artwork and my website does its own things. …
