Ahoy! It would seem that the Hetzner Helsinki Datacenter is having an outage that caught our little corner of the webs...

Luckily, our backups were updated just before their systems went down, and I was able to spin up a replacement and move DNS over. While things all cleared on my scopes, if you see trouble, please reach out in a DM.

Alternately, find us via matrix:

https://matrix.to/#/#faf-flotilla-lounge:anarchist.nexus

Apologies for the troubles!

The outage lasted long enough for it to be worth to move the entire server? Come one, don't make us other instance admins look bad 😅

In proper IT tradition... as soon as the new server was up and running and settled down, Hetzner got the issue with the previous node solved.

Because of course.

The little "good sysadmin" guy on my left shoulder is chastising me for not having failover and a floating IP already set up for this kind of thing.

The little "devops burnout" guy on my right shoulder is chuckling at me and whispering something about just writing a BASH script to do it in a janky way and forget about it.

I am caught in the middle and will probably do something with haproxy and keepalived. Such is the life of Ops.

Or, you know, people would have survived with the website being down for a few hours 🤷

for sure... but what about me and MY doom scrolling? 😁