I've had to turn on Cloudflare's full proxy with nearly 10's of thousands of requests coming out of CN IPs, in the last hour making ~7000 requests:

there are ~250,000 requests from china in the last ~20 hours

Cloudflare lets you block requests by location. I just have China and Russia permanently blocked

I felt like blocking the whole 36.x.x.x block this morning

We deployed Anubis, which helps, but we are still getting hammered with requests to the point that Lemmy has been unstable. I've been trialling moving away from Cloudflare, but I think the ASN blocks are pretty critical so I'll need to work out how to do that without Cloudflare.

Good luck, it's hard work fighting the bots.

I get them evenly split across IPs in countries across Asia and South America. I'm guessing it's the thing I've read about on Lemmy where people install game apps on their TVs which have inbuilt proxies to proxy scraping traffic through residential internet connections 😭

The storm seems to have abated over night, I guess they are logging blocked requests

I'm going to switch off cloudflare full proxy and revert back to good old DNS with these block in Caddy:

You're using Caddy? Have you considered Iocaine? It's like Anubis with more of a focus on data poisoning. I know their example uses Caddy which made me think of it: https://iocaine.madhouse-project.org/documentation/3/getting-started/

I used caddy on the deploy as it is what I was used to for all my homelab stuff

I might look at iocaine

I use Traefik for the same reason 😆. There's an Iocaine Traefik middleware but I haven't looked too far into it.

The problem with iocaine, by the looks, is it's use of ai.robots.txt. The bots/scrapers from yesterday all looked to have the agent set to "node". I could of cause add that string to the robots file and send any node.js agent to purgatory

The robots.txt is for well behaved bots. I already deployed that list quite some time back but it didn't help, because the problem are the not well behaved bots.

Iocane should be showing hidden links that only bots can see, and leading then into a maze of gibberish. It has other features too, but the core function is to keep them busy on easy to serve pages that poison their training and are inexpensive for your site to serve (as opposed to the lemmy pages, that have expensive database calls on every page load).

it has started up again just before I went to turn CF off, I think I'll leave it in and install Iocaine

CF bot filter (blocks about 3K/24h) => Caddy proxy => Iocaine => Caddy IP filter => Lemmy

Installing Anubis was certainly a big help for us, Iocaine should do something similar for you, fingers crossed!

I have been toggling the Cloudflare proxying trying to handle bots without it. old.lemmy.nz has had over 100k hits in the last 12 hours, this is just the ones that got past anubis. Something tells me that's not legit.

Yeah, I had about 600k + the ones I blackholed in Caddy in the 20h yesterday (I blackholed the whole 36.0.0.0/8 block in the end on the server)

3 requests (/api/v3/site /api/v3/post? /api/v3/comment/list?) in 50ms, from a IP then that IP would go silent for ages, then 500ms latter it was a fresh IP

I remember investigating and finding all the problematic IPs had requests evenly spread. Dozens of IPs from China, Singapore, and various South American countries. All of them had approx 80k requests in the period I was looking at, as if they send a few requests with one, then the next, then the next, rotating through then back to the start. Avoids rate limits I guess.

I am considering adding Iocaine as a first line of defence before Anubis, and seeing if that helps

CF ASN and bot block
│
▼
Caddy/Iocaine
│
├── response other than 421 ──► return Iocaine response
│
└── 421 ──────────────────────► continue to normal Caddy routing
                                  │
                                  ▼
                               Caddy - CN block
                                  │
                                  ▼
                                Lemmy

Caddyfile:

{$LEMMY_HOSTNAME} {
    import caddy-common

    ######################################
    ########## log files        ##########
    log cn_blocked {
            output file /data/cn-blocked.log
            format json
            no_hostname
    }

    ########## iocaine  code ##########
    @read method GET HEAD
            reverse_proxy @read host.docker.internal:42069 {
            @fallback status 421
            handle_response @fallback
    }

    ########## cn blocking code ##########
    @blocked {
            remote_ip 1.0.1.0/24 ...
    }

    route @blocked {
            log_name cn_blocked
            respond "<h1>Access Denied</h1>" 403
    }
    ##########   cn block ends  ##########
    ######################################  

Iocaine metrics with CF still on:

$ curl -s http://127.0.0.1:42042/metrics | grep '^qmk_'
qmk_garbage_generated{host="127.0.0.1:42069"} 3888
qmk_garbage_generated{host="no.lastname.nz"} 251440
qmk_requests{host="127.0.0.1:42069"} 2
qmk_requests{host="172.18.0.1:42069"} 2
qmk_requests{host="no.lastname.nz"} 404
qmk_ruleset_hits{outcome="default",ruleset="default"} 223
qmk_ruleset_hits{outcome="default",ruleset="trusted-path"} 1
qmk_ruleset_hits{outcome="garbage",ruleset="ai.robots.txt"} 11
qmk_ruleset_hits{outcome="garbage",ruleset="major-browsers"} 172
qmk_ruleset_hits{outcome="garbage",ruleset="unwanted-visitors"} 1

How's it holding up, making a difference? I reckon the bot hits are trending down from where they were a few days ago

470k blocked at CF in the past 24h, another 80+k tarpited in iocaine since 12h and still some getting through

Edit: Moved from mostly from CN to a mix of SE Asia and South America now

Damn, you're getting similar levels to us. Got about 520k requests to the UIs in the past 24 hours (this excludes federation or app usage that hit APIs directly, I have Anubis placed after those are split off).

Of those 520k, Anubis flat denies about half. and virtually all the rest are challenged, with somewhere around 7% actually being completed. So we are talking about 18k of the 520k getting through now. Most of those are probably still bots (logged in users get a straight pass through, and as mentioned federation or app users don't follow this path), but the server is much happier now than it was a few days ago.

I have Cloudflare disabled now, trying that out. I've had to get Grafana up to scratch as I don't get to use Cloudflare stats to track anymore.

not getting any better. Peaked at 25 requests per second blocked by CF just from China.

Damn. I've got Cloudflare off so should go install a geoip database so I can check what country things are coming from.

Not to out do you but we actually hit 50 requests per second a few hours back! That's through the reverse proxy so includes federation and other traffic not hitting anubis.

Baseline is about 3.5-4.5k per 15min (4/s), this China traffic is 3x base traffic across 24h

For the last 3 days (since turning on the block) there has been a dead spot from China from 4:00 to 14:00 NZST.

@Dave@lemmy.nz you seeing the same drop?

It's just crazy, these indescriminate scrapers are the worst. They think they are getting more data but they are just scraping the same data over and over via different instances 🙄

For the last 3 days (since turning on the block) there has been a dead spot from China from 4:00 to 14:00 NZST.

@Dave@lemmy.nz you seeing the same drop?

Ah i missed your edit. I don't have per country data like that, I've only very recently played with removing Cloudflare from the equation and haven't set up any geo-location stuff yet. My requests are coming pretty evenly over the day though, up and down a bit bit depending on time but nothing like what you're describing. I'm looking at all traffic though, not just China.

It's was getting worse, near 1,000,000 requests blocked in only 14 hours, peaking at 67/s blocked

When I get home I'll gig into the peak

Damn, good luck! At least those are all blocked rather than killing your server.

It does look like that since 9am it's back to not much above background sort of levels, although at 6k/15min it would be more that doubling the server load.That 60K/15min may have made it struggle

This would have been a hectic 5 min.

Can you tell when I turned CF off?

vs.

Do you know how many of those are being caught by iocaine?

These are my stats for anubis (this is only UI requests - LemmyUI and Mlmym mostly, no federation or app usage in this). A total of about 600k requests. Weigh and Challenge are the options given a proof of work challenge to solve (weigh just means the rules for the user agent or whatever mean the challenge difficulty was adjusted, often to make it harder for suspicious requests).

Out of the challenges issued in this time period, around 11% were validated (so 89% of the weigh and challenge traffic was also denied in addition to "deny" which was denied without issuing a challenge due to rules).

I've not set up nice graphics for iocaine (or any of the host stats) bu this is the last 5 days and that is with the asn-blocks and AI block turned on in CF

curl -s http://127.0.0.1:42042/metrics | grep '^qmk_'
qmk_garbage_generated{host="127.0.0.1:42069"} 3888
qmk_garbage_generated{host="no.lastname.nz"} 1147955554
qmk_requests{host="127.0.0.1:42069"} 2
qmk_requests{host="172.18.0.1:42069"} 6
qmk_requests{host="no.lastname.nz"} 1386448
qmk_ruleset_hits{outcome="default",ruleset="default"} 490771
qmk_ruleset_hits{outcome="default",ruleset="trusted-agent"} 2
qmk_ruleset_hits{outcome="default",ruleset="trusted-path"} 134
qmk_ruleset_hits{outcome="garbage",ruleset="ai.robots.txt"} 1689
qmk_ruleset_hits{outcome="garbage",ruleset="major-browsers"} 48928
qmk_ruleset_hits{outcome="garbage",ruleset="poisoned-url"} 844868
qmk_ruleset_hits{outcome="garbage",ruleset="unwanted-visitors"} 64

Hmm it looks like it's working, right? Like a billion (!) garbage pages generated. Still 500k getting through but over 5 days that seems minimal.