Hi,

We had a DDOS attack against old.lemmy.today which overloaded the entire server for about 20 minutes. I had to turn off old.lemmy.today temporarily and will look through logs and probably block more chinese ip ranges, since thats where it came from.

Always something to do. :) Keeping this as sticky for a while to inform everyone.

Edit: I added more ip blocks for Chinese networks and hopefully it will keep future bots away. Bringing up the old.lemmy.today user interface again...

Edit 2: Had to take it down again due to massive bot scraping. Currently down.

Edit 3: Decided to require login for old.lemmy.today instead of using anubis.

What is "old.lemmy"?

dang it . interesting.
I wonder if this is shady reddit operation

Maybe someone hid some JavaScript on some page somewhere.

https://old.lemmy.today/

When I was running a web server this was a common occurrence. Most of the DDOS attacks I experienced came from China.

We have some alternative user interfaces that you can use when you use the browser (info in the sidebar). But now they are down temporarily. :)

Yeah its very common. Some instances block the entire Chinese internet for this reason...

This is a genuine question, but why doesn't this instance? Is it because we would miss out on some content? I mean you are much more in touch with the technicalities, but do we have a mandarin speaking user base you're not trying to alienate?

I understood that reference.

I don't have an account there but I am always happy when fediverse instance admins are transparent about their problems. This will help other instances too.

are you sure it wasnt the tesseract dev slipping in some javascript to DDOS your instance?

I think we are unfortunantly heading in that direction of blocking more and more of them...

And no, no mandarin user base. :)

Alright, https://old.lemmy.today/ is back up, as well as the other alternative user interfaces mentioned in the sidebar.

I have blocked more chinese ip ranges found in the log files, so hopefully it helps.

Absolutely!

I dont think so since the traffic came from China and came from 951,466 unique ips. :)

Thanks for keeping the instance running. I happened to get a error message smth about the instance being broken and remembering that disk upgrade post from a couple days ago I feared the DB got nuked for a sec hehe.

While you're here, I published a small update yesterday (26.3.4) that fixes a few more bugs I've found :)

The db is like the holy grail for a lemmy instance. Heavily backed up and fortified. :)

😆🤜🤛

Very good, I pulled the latest version.

Thanks for your work on this, and the code you added to forward ip numbers was super useful today to see where these botnets are coming from. Really appreciate it.

I think there is only one Holy Grail and it belongs in a museum.

Thanks to both of you for keeping mlmym alive. It's the best way to use Lemmy and I moved to lemmy.today primarily because they have it as an available interface.

Thanks for the heads up. I got a brief error message a while ago and figured that's what happened. Good job keeping the instance going!

Thank you! These ddos attacks are annoying but not so common after all. Last time was like 3 months ago or something.

It's a front end for lemmy instances that mimics old reddit.

Thanks to both of you!

Now that you’re both here, I have a question/request. What I love most from the “old Reddit” interface is using RES to navigate through posts with the keyboard: “h” to hide, j/k as up/down, “l” to open in a new tab, etc. Makes browsing super fast and efficient.

Is this something you’d consider? I’m not asking for a full RES reimplementation, of course. Just the keyboard navigation.

thanks a lot for keeping this instance running without cloudflare stuff, really appreciate it..

Thank you!

I dont want cloudflare either. :)

I know of the lemmy keyboard navigation addon that reimplements this feature for lemmy-ui and mlmym. I haven't tested it myself though, and it's been last updated over a year ago, so I don't know if it still works. I have thought about including the feature directly in mlmym (could actually make use of the keyboard navigation setting that lemmy already has to toggle it), but if I do, that'll only happen after I'm done with the refactor for the v4 API. So it will probably take a while 😅

edit: opened an issue for it (#27) so I don't forget

Protected by a rabbit with nasty big pointy teeth.

Look into pow protection. The Chinese botnets come from a practically infinite range at the moment

Are we running a HIPS like IPBan Pro or are you doing this manually via iptables?

Thank you, I was reading about pow protection and finding out more about it. Is there some local software you could recommend? I'm not too hot on cloudflare or other big tech services unless absolutely needed.

It's manual at this point since it hasn't been a huge problem so far, but we will see how it evolves. Perhaps it gets worse as the instance grows bigger.

What is your reverse proxy software?

First, thank you for all the hard work you do.

Second, as someone still struggling to understand lemmy, one of the reasons I'm not a fan of old.lemmy.world is that I have to turn on cloudflare.

I have noscript running and I have cloudlflare blocked.

Your response implies otherwise that you don't have cloudflare running either.

So why am I always greeted with cloudflare when I visit old.lemmy.world?

Edit: Oh I misread the title. Its old.lemmy.today that you're talkinga bout, not old.lemmy.world.

My goof. Please ignore my post, but do take my gratitude.

Edit 2: So I went over to old.lemmy.today. It is tempting to switch. I like the default interface and it has content from a lot of places (on quick glance). One of the reasons I'm not a huge fan of old.slrpnk.net is that I can't figure out how to access content from other instances. I seem tempted to switch.

You are very welcome if you do. Its the same content in old.lemmy.today as lemmy.today, just a different look. :)

I dont like cloudflare so doing my best to avoid it if I can.

Just nginx.

You can look into Anubis then.

Much appreciated sir. :)

Thank you so much!

Thanks for the heads up and keeping us updated!

Hi I just wanted to follow up on something

Since last evening, I can't access old.lemmy.today (I can still access lemmy.today but that's not an interface I like).

Are you having bot issues again or is there something I messed up on my end?

One of the reasons I'm not a huge fan of old.slrpnk.net is that I can't figure out how to access content from other instances.

old.slrpnk.net should still have posts from other instances, as far as I can tell. They only show local stuff by default, so you have to set “listing” to “all” to see them (the small toggle under the yellow tagline). If you want to primarily interact with posts from other instances, lemmy.today is probably the better choice though either way. slrpnk.net looks like it mostly focuses on local communities.

I have noscript running and I have cloudlflare blocked.

The good thing about mlmym (the software behind old.lemmy.today, old.lemmy.world, and so on) is that it's entirely usable without JavaScript :)

So as long as there isn't something like cloudflare in the way that does require JS, it should work fine to use it that way.

Looking at it right now though, @mrmanager@lemmy.today it looks like old.lemmy.today is erroring with 502?

seems like it's working again but with anubis.. maybe can we just have something like http basic auth with a public password? (like have the password prompt as "type lemmy")

i don't think anubis is a good solution. it's very trivially bypassed though maybe good enough to block stupid generic bots..

see also https://lock.cmpxchg8b.com/anubis.html

Its not really working yet, it keeps popping up the "seeing if you are a bot" message all the time.

But yeah, working on it. If it can prevent dumb bots, its a win for sure.

It was down overnight and today im working on trying to get anubus to work with it, so it will be unstable... :)

Yeah, I had to take it down overnight and currently experimenting with anubis for it... will be a bit unstable today.. :)

Ok now its working and seemingly blocking bots, but we will see over time how well this works. Like you said, im sure its possible to bypass but maybe it improves the situation somewhat still.

Now it should work well again. I put it behind anubis and we will see if it improves the situation with the bots.