Site slow due to mass scraping / DDOS from Alibaba Singapore datacenter

overscan (PaulMM)

overscan (PaulMM)

Administrator
Staff member
Joined
27 December 2005
Messages
19,230
Reaction score
35,696
Hi all - noticed site was sluggish and have discovered thousands of open connections emanating from Alibaba's Singapore datacentre.

Not sure if its intended to take down the forum, or they are cloning the forum posts or something,

I have completed a planned future transition to using Cloudflare WAF/DDOS protection for the site earlier than intended. This will progressively kick in over the next 24 hours.

As a temporary measure users from Singapore will have to solve a challenge to view the site. This is hopefully temporary while I figure out what is going on.
 
Last edited:
Its actually stabilised and back to okay speed even with 2500 connections open, though the Cloudflare protection isn't in effect yet. I'm still going ahead with it, because this kind of thing is likely to reoccur and I need to have tools to restrict traffic if needed.
 
I was wondering why the site was loading so slowly today.
 
The Starship Modeler forum has been having what I expect is a similar issue.

If y'all will allow the liberty of sharing what I posted there in an attempt to perhaps increase general understanding,

Fri May 09, 2025 3:50 pm

...

That 'All' brings to mind things such as,

Google and OpenAI Are Slowing Down Your Website
A Smarter Hosting Fix
March 1, 2025
3 minutes read
https://aijourn.com/google-and-openai-are-slowing-down-your-website/
The Silent Strain of AI-powered web crawlers.

Artificially intelligent web crawlers are also called bots or spiders and search the web to index content and gather data. Search engine bots like Googlebot have always existed on the internet but AI giants like OpenAI have pushed up crawler activity. These AI bots skim the surface too – they also scrape large amounts of data to feed machine learning models. This deep crawling can unintentionally cause problems.

Increased Bandwidth Consumption

Each visit by a crawler uses bandwidth. Some visits from search engine bots are manageable, AI crawlers are more aggressive. This excess traffic may soon exhaust your bandwidth limits – especially on shared hosting plans – and cause throttled performance or overage charges.

Overloading servers and performance lag

AI bots may send thousands of requests in a row and overload servers. That leads to slower page load times / timeouts / and sometimes full site outages. For small to mid-sized websites on shared or virtual private server hosting, this is disastrous.
Click to expand...


‘Grey bots’ inundating websites to feed genAI
Real users crowded out as automated agents swarm.
By David Braue on Apr 10 2025 11:43 AM
https://ia.acs.org.au/article/2025/-grey-bots--inundating-websites-to-feed-genai.html
Between December and the end of February alone, Barracuda reports, the Anthropic-owned ClaudeBot lodged up to 2.5 million requests for data in a day – with one application seeing more than 9.7 million requests in a month, and another fielding over 500,000 in a day.

The volume of requests remained relatively consistent over the course of a day, with an average of around 17,000 requests per hour, suggesting that the grey bots were running at a steady pace to find and download whatever data they could come across.

This is far more measured than the traffic floods created by bad bots, which a recent analysis found account for over a third of Australia’s Internet traffic as they pummel popular sites to snag choice concert tickets, harvest personal data, and commit ad fraud.

Yet “both scenarios – constant bombardment or unexpected, ad hoc traffic surges – present challenges for web applications,” Barracuda senior principal software engineer Rahul Gupta noted, with copyright, privacy and other legal issues only the tip of the iceberg.
Click to expand...
...
With AI giants and startups introducing new genAI scrapers on a regular basis – and many using sneaky tactics to avoid detection – managing their impact remains a cat-and-mouse game.

Grey bots threaten to crowd out legitimate users as they inundate servers with requests for data, often ignoring website owners’ requests, using a widespread method called the Robots Exclusion Protocol (REP) and its robots.txt file, to move the bots on.
Click to expand...


Bots now account for over half of all internet traffic
News
By Sead Fadilpašić
published 16 April 2025
https://www.techradar.com/pro/security/bots-now-account-for-over-half-of-all-internet-traffic
With the proliferation of Generative AI, things are only going to get worse, Imperva further states. ByteSpider Bot alone is apparently responsible for more than half (54%) of all AI-enabled attacks. Other significant contributors include AppleBot (26%), ClaudeBot (13%), and ChatGPT User Bot (6%).

Not all bot traffic is malicious, though. There are many useful, and often essential bots, such as search engine crawlers, monitoring bots, social media bots, or data scraping bots. They are used to index websites for search engines, check websites for performance or downtime, schedule posts or respond automatically, or to aggregate sites and scrape valuable data.

Still, bad bots take up a hefty portion of all bot traffic, presenting a real challenge for the cybersecurity community.

These tools, whose popularity exploded roughly three years ago with the introduction of Chat-GPT, have simplified the creation and scaling of malicious bots, Imperva noted.
Click to expand...


Mike Elgan
by Mike Elgan
Contributing Columnist

Inside the war between genAI and the internet
opinion
Mar 28, 20256 mins
https://www.computerworld.com/article/3854642/inside-the-war-between-genai-and-the-internet.html
While much digital ink has been spilled decrying the taking of content, it’s also important to know that the chatbot companies are overwhelming many of the sites they’re copying content from, much like a daily DDOS attack.
Click to expand...
 
So - they were able to evade my protection by accessing the site directly by the old IP. I've now closed this loophole down (hopefully most people switched to Cloudflare IPs by now).

Its quite possible its a new AI scraping data rather than a deliberate attack but the quantity of requests was overwhelming.
 
Last edited:
overscan (PaulMM) said:
Hi all - noticed site was sluggish and have discovered thousands of open connections emanating from Alibaba's Singapore datacentre.

Not sure if its intended to take down the forum, or they are cloning the forum posts or something,
Click to expand...

Hmm, wonder if that Alibaba are the same as this Alibaba who appearo to have the Alibaba sales website?

Building a Custom Chatbot with Web-Scraping and Alibaba Cloud Model Studio
JwdShah
December 16, 2024

https://www.alibabacloud.com/blog/601863
Chatbots are transforming how businesses interact with customers, offering instant and accurate responses tailored to specific needs. With Alibaba Cloud Model Studio and web-scraping techniques, creating a domain-specific chatbot has never been easier. This blog walks you through the process of building a chatbot that uses web-scraped data and advanced prompt engineering to deliver precise and relevant answers.
Click to expand...
...
Step 1: Web-Scraping for Data Collection

In this blog, we will use web-scrapping that allow us to extract information directly from websites. This data forms the chatbot’s knowledge base. Using a simple Python script, you can scrape text content and save it for later use.

View the web-scraping code here.

All you need is to replace the "target_url" with your desire website URL. Once you run this file, it will create a text file containing the data of website.

The extracted data ensures that the chatbot's responses are not only accurate but also relevant to the latest information available online.
Click to expand...

See for connection reference:

https://www.alibabacloud.com/en/about?_p_lc=1
 
Training_Dummy said:
Every Perplexity Pro search related to obscure aerospace and defense subjects hits this site. Would be interesting to know if the posts from really high impact / knowledgeable users gets a heavier weight in the models.
Click to expand...

Lol good luck to the A.I trying to make sense of my posts. It will blow a few gaskets, I tell you.
 
I lost access to the forum for most of the day. I've only just been able to log in. Every other attempt I made got a 'server is down' message.
 
Graham1973 said:
I lost access to the forum for most of the day. I've only just been able to log in. Every other attempt I made got a 'server is down' message.
Click to expand...
Yeah the nameserver records take up to 24 hours to update in worst case. You must have been one of those unfortunate people.

I wouldn't have closed down access to the previous IP addresses this morning, except the bad traffic was still arriving and seriously overloading the server, so I decided it was better for 75% of people to get good speed access than 100% of people to get super slow access.

I actually think its improved the site speed generally, but more importantly I now have the ability to rate limit, block and otherwise defend the forum from such traffic.
 
overscan (PaulMM) said:
Hi all - noticed site was sluggish and have discovered thousands of open connections emanating from Alibaba's Singapore datacentre.

Not sure if its intended to take down the forum, or they are cloning the forum posts or something,

I have completed a planned future transition to using Cloudflare WAF/DDOS protection for the site earlier than intended. This will progressively kick in over the next 24 hours.

As a temporary measure users from Singapore will have to solve a challenge to view the site. This is hopefully temporary while I figure out what is going on.
Click to expand...
Good to know you're appreciated. ;)
 
overscan (PaulMM) said:
Hi all - noticed site was sluggish and have discovered thousands of open connections emanating from Alibaba's Singapore datacentre.

Not sure if its intended to take down the forum, or they are cloning the forum posts or something,

I have completed a planned future transition to using Cloudflare WAF/DDOS protection for the site earlier than intended. This will progressively kick in over the next 24 hours.

As a temporary measure users from Singapore will have to solve a challenge to view the site. This is hopefully temporary while I figure out what is going on.
Click to expand...
Yikes!
 
overscan (PaulMM) said:
Yeah the nameserver records take up to 24 hours to update in worst case. You must have been one of those unfortunate people.

I wouldn't have closed down access to the previous IP addresses this morning, except the bad traffic was still arriving and seriously overloading the server, so I decided it was better for 75% of people to get good speed access than 100% of people to get super slow access.
Click to expand...
I lost access from 19:45 UK time yesterday til midday-ish today, but if that let you keep the forum up in the face of a DDOS, intentional or not, then that works fine for me.
 
Many many years ago I was working on a very busy website that had a similar problem, before the terms "DDoS" or "botnet" were coined. It got to the point where the clients were difficult to block by IP, they would just start coming from someplace else. I realized that all of the clients were coming from Windows machines and appeared to be using some of the built-in software to make the requests.

After getting approval from the company owner I wrote a script that reliably identified the pattern of an attack. It then sent the client an HTTP redirect that went to a file URL on the local machine - \LPT1\LPT1 , the printer port. Their software would "open" the printer port like it was a file, which triggered a Windows Protection Error (crashing their Windows machines).

That stopped the attacks quickly and got the site running again. It took the bad guys a few months to figure out what was happening to them.
 
aim9xray said:
Then what happened?
Click to expand...

They changed tactics, so did I. Eventually I found that they were coordinating through IRC and was able to join them there. Users in the channel could request attacks. So I requested they attack us, watched the attacks roll in, developed countermeasures. Eventually they blacklisted our sites for attacks. If you requested it in IRC a bot would deny the request and not trigger the attack.
 
quellish said:
Many many years ago I was working on a very busy website that had a similar problem, before the terms "DDoS" or "botnet" were coined. It got to the point where the clients were difficult to block by IP, they would just start coming from someplace else. I realized that all of the clients were coming from Windows machines and appeared to be using some of the built-in software to make the requests.

After getting approval from the company owner I wrote a script that reliably identified the pattern of an attack. It then sent the client an HTTP redirect that went to a file URL on the local machine - \LPT1\LPT1 , the printer port. Their software would "open" the printer port like it was a file, which triggered a Windows Protection Error (crashing their Windows machines).

That stopped the attacks quickly and got the site running again. It took the bad guys a few months to figure out what was happening to them.
Click to expand...
I worked for Yellow Pages and an engineer used to block by IP for users scraping the White Pages website. He got tired of them swapping IPs in the same subnet and started using IP bans on entire network addresses. When he left, entire ISPs were blocked and we had loads of users complaining they couldn't access the site. It seemed like we were blocking a significant portion of NZ IP addresses.

We got rid of all the IP blocks and went with a more sophisticated Web Application Firewall approach instead.
 
Wow.
I thought it was a planned move to a new website as had be said.a while back, but I'd forgotten the date and I'd lost the web address for the new.
Access was zero from Thursday evening.

Another 'benefit' of AI then.
 
The URL isn't changing. The site will move to a new IP address, but the URL will stay the same.

Due to the new arrangement using Cloudflare in front of the forum, during the migration there'll be a "maintenance page" up for the duration which I'll keep updated and unlike the old way I did it, I can bypass it for testing the new forum. I'll only drop it when everything is ready for users to login.

It will likely be next weekend for the migration.
 
I'm down from 3000 users online to a bit over 500, which is much more normal, but slightly sad to lose even if they were fake...

It reminds of a certain CTO talking to us about page hits and when I raised the idea that was a misleading stat to give his customers, and they should switch to unique visits like Google analytics he went a bit pale and said "Christ no, if they knew how few people were involved they'd never pay us".
 
Last edited:
lancer21 said:
Well, maybe there might be a slight positive to this, the AI could learn to design better planes and stuff, the kind of designs AI comes with nowadays looks like were done by a disturbed schizophrenic mind...
Click to expand...
i find it reassuring myself. IA is still a poor about planes, can't make the difference between a MiG-21 F-13 and a MiG-21PFM late production, ahahaha poor noob.

Anyway, other forums I visit have been hit too by lot of IA scraping coming from Brazil (?!). Dunno what they are up to...
 

Similar threads

Back
Top Bottom