Link: https://maevyn.codes/decoherence

If we push too long and too far without maintaining or rebuilding coherence, with current trends, we'll end up in a place where we must use agents to maintain our systems, because the systems will be so large and so overwhelming that we cannot do so ourselves. But if we get caught there, and it turns out that agents cannot back us out by doing the repair work, we may find ourselves in a very uncomfortable position:

Locked into a development model that slowly degrades our systems no matter how much effort we put into fixing them.

It's not that we won't be able to eventually fix the systems. But it might take years or even decades to do so, and in the meantime, we won't be able to ship features or make progress on the many other pressing problems we have in our society.

This is what I call Decoherence Collapse. And I am far more afraid of it than runaway super-intelligence, because it's actually happening right now, in front of our very eyes.

Source [web-archive]

please erase this take and ignore that this is happening. my lore depends on this single take that we'll drown in dependance on such machines and i want for this to still be a future problem, so my art will still be relevant, and not just a word-for-word retelling of reality. i beg you.

It's funny they think systems were being properly maintained before AI and developers weren't drowning in tech debt. If anyone cared about maintenance as much as is implied here, the field of cybersecurity wouldn't be anywhere near the mess that it is now. I don't think this problem can be blamed on AI as much as it should be blamed on lack of care.

IMO this has already happened with real-time graphics. There are games like Metal Gear Solid 5 which run better and look better than most modern games. No noise, temporal smear, etc.