Link: https://maevyn.codes/decoherence

Decoherence grows relative to the rate of change of a system, and it increases faster the more decoherent a system is, resulting in runaway exponential growth. It is not a new phenomenon, and humans have always countered it through carefully finding ways to do more with less, unifying disparate patterns and models into a cohesive whole. However, the widespread use of AI is currently producing it at a rate that is unsustainable.

The good news is that this rules out the specter of runaway recursive self-improvement, the intelligence explosion that goes parabolic before anyone can react, as decoherence both prevents AI from improving in a bubble and limits the rate at which it can improve outside of it. The bad news is that we are barreling toward a different kind of disaster: Decoherence Collapse.

Fantastic write-up!

This is everything I’ve ever wanted to say about AI at my job, plus things I never would’ve thought of, perfectly articulated, well-cited, and emphasized with poignant graphics. I’m floored.

We are currently living through the golden age of Ai slop.

Good lord, it really is. Glad I caught this comment, cause I wasn't down for reading the article otherwise. I'm definitely saving this and passing it on.

This article argues that any conceivable society that holds art competitions will be less capable than a society that does not, because art does not represent real world objects so holding a competition in art trains people to lose track of reality, eventually rendering them unable to do even basic functions as they try to paint instead of walk.

After all, if you aren't currently looking at a flesh and blood rhino, you can never learn to get better at drawing rhinos, and that shoddy workmanship means you'll never be able to draw a savannah (which has rhinos and many other things), let alone develop the live-updated 3D model you need to walk across one.

Soon, people who aren't currently looking at rhinos will be unable to drag themselves across the floor, not knowing how to draw the hands and feet that would propel them.

Did you read the article? It's long, but worth it. It describes analytical problem solving and software development.

Did I miss anything supporting your claim that the article argues about art?

The article makes general assertions about systems. It gives plumbing, software, and machine learning as examples, and I simply add the examples of human learning and human societies.

An agent specifying the criteria of a test inspired by the real world and then evaluating the performance of a bunch of other agents is equally information-incestuous when those agents are LLMs as when they are humans. If the observations of the article must hold true for LLM training, then they must hold true for humans. If they do not hold true for humans, then they do not prohibit the existence of LLM training programs that likewise avoid the decoherence trap.

Given the fact that evolution was able to find human creativity and art as forms of cohering enrichment of human performance beyond the limits of the training data of a world without art, I would expect that there is a good chance tens of thousands of engineers and billions of dollars of compute will be able to find some form of abstract expression and learning as a form of cohering enrichment of LLM performance beyond the limits of the training data of a world without that abstract LLM expression.