Only the lucky few have a test env separate to the production env.

Tested in the school of hard knocks.

test? just do it and let your uptime kuma alert tell you if something's wrong.

that's the test

My current job provided me with a test env that has half the resourse allocation of production, and it immediately runs out of RAM when you try to use it...

I use a sample for tests, not the full set

Production? That's just our pool of thousands of testers who pay for the privilege!

Check for security vulnerabilities? But Internet hackers will do that for free.

Every team has a QA environment. The fancy ones have a separate production environment.

Mine is like 3 years out of date to prod.

As a sysadmin, nothing makes me want to bitchslap devs harder. It's shocking how many don't even ask if we have a dev enviro. (The answer is yes)

I literally live between ops sentiment and this sentiment

I stand shit up as a test server and then threat model before cloning a production server to a separate vlan and lock it down proper, with the added Intel of a snapshot on how the test server got tested.

I stood up a test, staging, and prod pair of web server/dbs, got the Drupal 8 site stable on Drupal 10, hand it over, and yeah. No one uses anything but prod.

Thanks for 3x the work, ya dicks.

My team made the "mistake" that all new projects need to be built and approved on our dev clusters before we allow them to be deployed to the prod clusters. We're very explicit that dev is ephemeral, and that everything should be as code, so everything heals itself...

Despite this, whenever we reprovision the dev env, we always get emails from several different people saying that we wiped their prod databases and such. It's crazy.

this. the test environments we have need to be built manually and they haven't been rebuilt in ages.

apparently it was a multi month project to get it setup in the first place. nothing like trying to recreate a bug in dev and finding out there's 1000 lines of different code in the sandbox (that's also misbehaving but in new and fascinating ways)

The hero we deserve

Stay borked, my friends.

That sounds like a fun frequent task to introduce!

Get your BoFH on!

Vibe coders: "I don't get it."

Test environment? Everyone’s got a test environment! Although, some folks are even lucky enough to get a whole separate environment to do production in.

Damn, beat me by 3 hours; that’s what I get for not reading the thread first

We don't usually do it for fun, but I'm sure my teammate who has petitioned deploying a "chaosmonkey"-thingy to our prod clusters would love it 😁

Your teammate has the right approach here. People will quickly learn to stop relying on the dev environment in production deployments when the environment becomes unreliable.

I really wish we could get to that level of maturity (cries in healthcare)

Not sure if we should take the approach of biting the bullet and forcing it on ourselves or slowly growing to it (under a hostile govt that sacked ~30% of the countries ICT team)

back when my team had proper coverage, we used to do chaosmonkey style tests for hosted health systems. We'd schedule an event with the hospital IT staff. Say... 1 hour on a specific day near midnight. The local IT staff would put out a notice that for the hour the system may be inaccessible. Then we'd start a test suite, as ungracefully as possible. We'd run a abort shutdown command on one server, see how RHCS would handle it. Bring it back up stable. Silo a database instance. See how that handled. Yank out the Red1 cable from the switch, see how the network balanced the new traffic over Red2/Blue. That kind of stuff. We caught a few problems that way.

The clients hated it, but it really made the system more bulletproof for when a new recruit ran a command in the wrong window and took down an unrelated resource.

Actual footage of me testing in prod:

Dang I haven't seen this meme format in a while.

What do you mean I shouldn’t have plugged Claude into the prod database?

My manager quotes this to me every time we demo a bunch of random untested stuff to our clients.

Better than the new devs here finding out that the half the tools in staging touch production, giving me 5 hours of impatient sales texts

Jealous!

A-ha! I knew I was suave.

And why is all the stuff from my desk in that box?