Loss of service for several minutes

[spot]

Forumgarden has seen higher than usual web page access from illegitimate bots over the last 20 hours. For the time being I've put a stop to it. I can't tell whether it will develop into a DDoS attack - I've no reason to think we've done anything to warrant one. It had us repeatedly peaking at over 20MB/s disk accesses though, and touching 100% CPU, corresponding to consistently higher than 10 reads a second for hours on end. Insignificant figures for larger sites but not this one.

I'll sleep on it and consider how to deal with this more long-term. Geo-blocking, perhaps.

If anyone else noticed, or notices, do please add to the thread with a description.

[spot]

There we are, I got six hours sleep after sorting that - the graph indicates a successful bodyslam of a minor DoS in the tradition of that old-time operator Mick McManus.

Screenshot_2024-08-29_13-21-28.png (108.24 KiB) Viewed 4695 times

What you're looking for as confirmation is /var/log/apache2/error.log showing authz_core:error announcements like (and note the attacks are continuing, this is now):

Code: Select all

[Thu Aug 29 12:25:40.063345 2024] [authz_core:error] [pid 19432:tid 19432] [client 43.133.38.100:59988] AH01630: client denied by server configuration: /var/www/xx/forums/viewforum.php, referer: https://google.com
[Thu Aug 29 12:25:40.215427 2024] [authz_core:error] [pid 19518:tid 19518] [client 43.133.43.227:49500] AH01630: client denied by server configuration: /var/www/xx/forums/viewforum.php, referer: https://google.com
[Thu Aug 29 12:25:40.328066 2024] [authz_core:error] [pid 19374:tid 19374] [client 43.153.5.20:60008] AH01630: client denied by server configuration: /var/www/xx/forums/viewforum.php, referer: https://google.com
[Thu Aug 29 12:25:40.404979 2024] [authz_core:error] [pid 19458:tid 19458] [client 43.159.146.48:34278] AH01630: client denied by server configuration: /var/www/xx/forums/viewforum.php

and the code to make it happen is in the virtual host directory configuration, such as:

Code: Select all

      <Directory /[directory masked for privacy reasons]>
            DirectoryIndex index.php
            AllowOverride All

# New syntax for Apache 2.4 and later
            <RequireAll>
                  Require all granted
                  Require not ip 43.0.0.0/8
                  Require not ip 49.51.0.0/16
                  Require not ip 66.249.64.0/16
                  Require not ip 85.208.96.0/16
                  Require not ip 94.74.0.0/16
                  Require not ip 101.32.0.0/16
                  Require not ip 101.44.0.0/16
                  Require not ip 110.238.0.0/16
                  Require not ip 111.119.0.0/16
                  Require not ip 114.119.0.0/16
                  Require not ip 119.8.0.0/16
                  Require not ip 119.13.0.0/16
                  Require not ip 124.156.0.0/16
                  Require not ip 124.243.0.0/16
                  Require not ip 129.226.0.0/16
                  Require not ip 150.109.0.0/16
                  Require not ip 154.54.249.0/16
                  Require not ip 159.138.0.0/16
                  Require not ip 166.108.0.0/16
                  Require not ip 170.106.0.0/16
                  Require not ip 185.191.171.0/16
                  Require not ip 190.92.0.0/16
                  Require not ip 199.167.138.0/16
                  Require not ip 216.244.66.0/16
                  Require not ip 217.113.194.0/16
            </RequireAll>
      </Directory>

It's my guess that the spiky nature as the attack builds is a timing effort to avoid fail2ban triggers for a frequency and volume trap, it allows the timers to expire before the next push instead of a potential 24 hour or one week or permanent ban.

And they all originate in:

ISP: Tencent Building, Kejizhongyi Avenue
Organization: Tencent Building, Kejizhongyi Avenue

and
ISP: Shenzhen Tencent Computer Systems Company Limited
Organization: 16 COLLYER QUAY # 18-29 INCOME AT RAFFLES (tencent.com)

in which case I may be slightly out of my league here.

Or it may be a rogue unwanted spider which ignores polite requests to go away.

[spot]

It's been back for 12 hours now and the fix appears to be holding:

Screenshot_2024-08-29_18-01-11.png (128.81 KiB) Viewed 4681 times

[spot]

That's more like it:

Screenshot_2024-08-30_10-46-35.png (137.54 KiB) Viewed 4667 times

For anyone watching who needs to fend off this particular bunch of antisocial data thieves, here's my blanket suppression of their current spider hosts. I'm aware my ranges are too broad but I was annoyed when I specified them.

Code: Select all

      <Directory /var/www/xx>
            DirectoryIndex index.php
            AllowOverride All

# New syntax for Apache 2.4 and later
            <RequireAll>
                  Require all granted

                  Require not ip 43.0.0.0/8

                  Require not ip 3.224.0.0/16
                  Require not ip 23.22.0.0/16
                  Require not ip 34.230.0.0/16
                  Require not ip 49.0.0.0/16
                  Require not ip 49.51.0.0/16
                  Require not ip 52.70.0.0/16
                  Require not ip 54.36.0.0/16
                  Require not ip 74.201.0.0/16
                  Require not ip 85.208.0.0/16
                  Require not ip 94.74.0.0/16
                  Require not ip 101.32.0.0/16
                  Require not ip 101.44.0.0/16
                  Require not ip 110.238.0.0/16
                  Require not ip 111.119.0.0/16
                  Require not ip 114.119.0.0/16
                  Require not ip 119.13.0.0/16
                  Require not ip 119.28.0.0/16
                  Require not ip 119.8.0.0/16
                  Require not ip 124.156.0.0/16
                  Require not ip 124.243.0.0/16
                  Require not ip 129.226.0.0/16
                  Require not ip 150.109.0.0/16
                  Require not ip 159.138.0.0/16
                  Require not ip 166.108.0.0/16
                  Require not ip 170.106.0.0/16
                  Require not ip 182.43.0.0/16
                  Require not ip 185.191.0.0/16
                  Require not ip 190.92.0.0/16
                  Require not ip 199.167.0.0/16
                  Require not ip 216.244.0.0/16

            </RequireAll>
      </Directory>
      

[spot]

The rogue bot stopped accessing the site at 8am today. Until then they'd continued trying to hammer the site from a thousand or so IP addresses, but all their requests were rejected. So was Alexa's bot which got caught up in the rejection list.

I've removed some of the constraints for today to see what happens.

I've left a couple in place though, requiring reverse proxy lookup success before responding. I'll take that off on Monday.

If anyone has found ForumGarden inaccessible this weekend, that will be the reason. PM me and I'll leave reverse proxy lookup permanently disabled. Otherwise if nobody speaks and the problem returns I may reinstate it.

[spot]

Start: 2025-04-02 20:00 UTC
End: 2025-04-02 23:00 UTC

Hello,

During the above window, our Networking team will be making changes to the core networking infrastructure to improve performance and scalability in the LON1 region.

Expected impact:

During the maintenance window, users may experience delays or failures with event processing for a brief duration on Droplets and Droplet-based services, including Droplets, Managed Kubernetes, Load Balancers, Container Registry, and App Platform. We will endeavor to keep this to a minimum for the duration of the change.

If you have any questions related to this issue, please send us a ticket from your cloud support page. https://cloudsupport.digitalocean.com/s/createticket

Thank you,
Team DigitalOcean

[spot]

Hello,

We are reaching out again to inform you that the Network maintenance in LON1 region which was previously scheduled to be complete on 2025-04-02 20:00 UTC has been rescheduled to the following window:

Start: 2025-04-08 20:00 UTC
End: 2025-04-08 23:00 UTC

We apologize for any inconvenience this short notice causes and thank you for your understanding. You may find the initial maintenance notice along with a description of any expected impact related to this work included at the bottom of this message.

If you have questions or concerns about this maintenance, please reach out to us by opening up a ticket on your account.

Thank you,

Team DigitalOcean

[spot]

The site is visibly slow at the moment. Google is indexing the entire site by the look of it. Eventually I might intervene but it's potentially a benefit. Bear with, bear with...

[spot]

A helpful PM, thank you. That sod is using an Agent Name to which he he not entitled.

The last 24 hour graph is exactly as you'd expect. The 3am archive runs, the bad actor arrives, I take the site down for quarter of an hour's analysis, and it's straight back in when I restart.

I might leave it a bit and see if it sates itself, otherwise I'll put significant blocks in place.

Screenshot_2025-04-28_17-32-03.png (86.42 KiB) Viewed 325 times

[Bryn Mawr]

And ignoring robots.txt which is bad form

[spot]

All out of China, like last time, these associated with Alibaba Cloud (Aliyun) rather than tencent. Blocked again now, the CPU is more modest, it may still feel less immediately responsive than it ought because the pummelling is still there. This evening I'll consider bricking up the port rather than a gatekeeper on the database.

[spot]

The backlog of vibrations settled overnight, response time is back to speedy. Do add observations to this thread if you find it sluggish again, it helps us focus on potential problems faster.

Speedy ought to be significantly less than a second.

[spot]

The Chinabots came back 15 minutes ago. I've tightened the firewall and added rate limiting, and rebooted. It may again take a while to shake out the delay, I'll keep an eye on it.

[Bryn Mawr]

Still up around 9 seconds as at 23:35 UTC

[spot]

Still up around 9 seconds as at 23:35 UTC

It is, isn't it. And I'm exploring reasons. I'm moderately puzzled at the moment.

[spot]

Sorted.