Looks like our servers are on the fritz again. Usually we aren't out of commission for too long. Hopefully we are back soon.

@Kris@feddit.org give the server a smack with the old convincer, will ya?

Which one is this - the slr or the pnk? 🫠🤪

Me? I'm the punk...I think? Definitely a lot of solar chops, but I think a lot differently than my illustrious server members.

Usually we aren’t out of commission for too long.

Bad news on that front, from Lemmy General Matrix chat:

Every time this happens, I think to myself, I should back up and restore settings to my alts, so it's not as irritating next time. And then I forget.

Well, fortunately, it's the Fediverse, so you can make a second account on one of the many alternate instances out there and keep on trucking until they come back up.

I really need to cool it on the shitposting...

That's why I have this account. I only have to break it out once or twice a year, honestly. For the most part out server is pretty stable, but then every now and then, a rare Azorian moss creeps in though the window hosting the server, and begins to colonize. This results in much delicate work, as @ProdigalFrog@slrpnk.net and @poVoq@slrpnk.net carefully scrape the moss off the motherboards, using hori-hori knives imported directly from Japan. They then coax the moss onto several 9x13 Pyrex dishes loaded with a home made agar and begin the trek up Mount Pico, to relocate the moss to its natural habitat. This is only done under light or a full moon and with much chanting. If done right, a Satyr emerges from the bushes (erect phallus and all), playing melodic flute music, and thanks them for restoring the nomadic moss back to the wild. After a few moments of awkward silence for both the admins walk back down the slope, wondering what actually just fucking happened, and how this became part of the routine maintenance of the server.

Bad luck for Ecospheres community

I'm pretty sure the downtime is just for enforced grass-touching.

Lol yeah. I'm almost done reposting the original posts from the old community.

Who are we to question a process that works?

I really like that idea, yet here I am circumventing it.

Get off the internet and go outside!

No no no, silly. It's PHP, remember? P H P

poVoq here. I am starting to think I am cursed, because everything works fine (or is at least fixable with a remote server restart) until the second I go on a longer business trip and can't easily access the hardware.

Remote access is mostly lost, but I was able to triangulate the issue and of course it is something I had on my list of potential issues, but as a low priority item 🫠 At least it isn't the same problem again as in 2025.

What exactly triggered the issue is unclear though. I was still able to access the reverse-proxy server yesterday (shortly before it went down) and tried to do some updates for that php security problem in a VM that runs on the same machine. I thought I had those covered, but there is a slight chance that it is related and someone chained another exploit to escape the VM.

Anyways, it looks like the realistic / not overly optimistic scenario is that the server will stay down until christmas. I did have someone who could restart the servers on site until this summer, but they had to relocate to mainland Portugal for job reasons (tourism sector) and so they will also only return for christmas.

So yeah, not great at all, and I will have to think more seriously about how compatible this kind of self-hosting is with my current job, which limits my access to the servers for a few weeks at least once a year. It was easier during the pandemic times when I started all this, as deployments were shorter and more seldom back then.

That'd be funny if it came back up as PieFed/PyLova.

Shut up Trelane!

Be really nice to build some local community so that you can both self host and have this kind of support if you need it

You really just need a comically long stick to flick the power switch from the mainland. Simple as.

Or perhaps pay that Satyr, I mention in another comment.

In seriousness, is physically moving the server possible? I know absolutely nothing about servers at all. Or would we be able to host it virtually? I realize that probably defeats the idea of a self hosted instance. What about as a back up or something?
I'm completely just throwing spaghetti at a wall here, and it isn't even half cooked.

Due to the LLM scrapers I have been thinking that I might have to move to a hybrid system of going through a VPS with only some storage and compute services on my home servers remaining. It is an annoying expense of about 20€/month that I was hoping to avoid, but it will give some other benefits like a fully fixed IP, IPv6 support and DDOS protection.

In such a setup the service would degrade better, like images breaking, but the main service staying up when my home server goes down.

But it is not something I can setup now that the service is already inaccessible.

Would be nice, yes. But easier said than done 😓

I think a more realistic option is to move my self-hosting ambitions down a notch and only focus on more redundand backend services that are too costly on a rented VPS.

Yeah fair enough. I agree, annoying on your end about the expense.

Bummed that you can't seem to catch a break with the sever suffering from separation anxiety and throwing a fit every time you leave for work.

This website is currently offline due to an hardware issue that can't be resolved quickly.

It is possible that it will only be restored in December 2026. Sorry.

For urgent questions please contact info@f-hub.org

Bummed that you can’t seem to catch a break with the server suffering from separation anxiety and throwing a fit every time you leave for work.

Artist's rendering. (had to use catbox since lemmy.cafe doesn't seem to allow image uploads?)

Base image courtesy of @undthr@horp.sciops.net

Scraping the moss with the hori-hori knives is lowkey relaxing though, ngl.

We've also built up a bit of a rapport with Gary the Satyr (we're on a first name basis now), and the little dude will jam out some Checkfield on his flute upon request (the forest weed he offered us really enhanced the experience, too).

First time was a little awkward, but by the third time, it's honestly a chill time, and a good excuse to get some nighttime hiking in. We don't even notice Gary's fully erect schlong anymore :)

I'm so glad you liked my elaborate stoy :D

You have no idea how much I wish I could fit this in my profile picture

I loved it! 😄

poVoq here. I am starting to think I am cursed, because everything works fine (or is at least fixable with a remote server restart) until the second I go on a longer business trip and can’t easily access the hardware.

Have you tried leaving for a couple hours and then coming back and giving the server a couple of bytes of a treat? Ease the server into you leaving with a bit of training.

Thank you for your service, and thanks for the update. Would an IP KVM help here? Know some of them can cycle power remotely, and they're getting a lot cheaper

I bought a chinese PiKVM copy last time it happened, but ran into issues with the usb otg connection which I procastrinated on to fix, so it currently only shares the screen but not the input. But I also connected it to the main router which is supposed to run the ssh tunnel and reverse-proxy eventually but right now those run on a seperare server and that one went down 🫠

So yeah, I am victim of my own half finished stuff as well as security hardening that now locked myself out pretty much...

Happens to the best of us. I'm just glad it's a glitch not a crisis, and temporary not permanent :)

So you're saying this might all be because of AI (which found the security flaw in PHP, which caused the maintenance, which happened sometime around when the server went down). Sounds fair to me.

💕 Seriously, thanks for the update and I know sysadmining can be a stressy job. Take care of yourself.

I don't think it was actually the php issue that took it down, but I don't wan't to rule it out entirely. But I had some issues with spotty remote connection for a while now that that is likely due to LLM scraper load.

Oh yeah, I was teasing about the php. From what I've read it's wicked hard to real-world implement an attack using it.

Scrapers, I'd believe. They've been fierce lately.