Fighting the continuing evil of AI bot scraping

[overscan (PaulMM)]

Well, digging around in forum settings I found that a plugin I installed to track 404 (page not found) errors and set up redirects for them, which I haven't actually checked in a while, was falling foul of some bot or other spamming the forum with hundreds of requests a second of requests for URLs that don't exist.

This is a directly an extra load on the server, but the sheer quantity of requests being logged to the database, into the millions, expanded the database by several gigabytes and slowed things down in the process.

I've cleared the log and will disable the plugin. This should improve server performance.

 

[Archibald]

Oh for f*ck sake...

View: https://www.youtube.com/watch?v=QNrqCulYkbU

The only bottom A.I should scrap is that of a septic tank.

 

[overscan (PaulMM)]

Viewing the log files is an insight into stupidity. Some automated process is trying and failing to download hundreds of images from the site every minute with a totally incorrect URL format. I can block, but the beast of stupidity, like Hydra, just spawns more heads. Hard to win the battle.

 

[overscan (PaulMM)]

Well lookie here, when I changed up my security rules, site traffic went from 900GB/day to 77GB/day.

 

[GeorgeA]

This explains a lot, thanks Paul. It is indeed a long twilight struggle.

I was noticing degraded performance last week but didn’t want to bitch about it. Looks much better today.

 

[dannydale]

Smells blatantly like a DDOS.

 

[DWG]

No matter how hard you try, the universe can always breed a dumber class of idiot.

 

[EmoBirb]

Any idea what could be the reasoning behind this?

 

[dannydale]

Any idea what could be the reasoning behind this?

Education enables populations to resist harmful agendas.

 

[GeorgeA]

I am, however, getting this a lot: