The catch is: this is 90% of what they will make for the next three years, now.

Why can’t the master race take these guys down?

The answer — as it is for most things — is money.

Similar to HBM, HBF relies on multiple memory dies that have been stacked together, but instead of DRAM, it utilizes NAND flash to increase storage capacity.

What's the lifetime on the flash memory in such a setup? I mean, if you're constantly shoveling 3TB/sec into said flash memory, even if you have a lot of memory behind that interface, I'd think that it'd have a limited life.

thinks

Maybe if the stuff is rarely-written? Like, say you use HBF for static data that you mostly just read, like the model edge weights, and HBM-accessed DRAM just for the dynamic stuff?

reads more

Oh. Maybe this is really more a replacement for M.2 or whatever is currently used for flash on compute servers, not as an HBM alternative:

Its primary role will be to complement capacity limits rather than outright replace HBM. LLMs can be processed through HBM, which is immediately required for computing, with HBF storing data in a larger capacity.

Might eliminate the need to load a model from flash to DRAM, if it can just be used directly from flash.

considers

That might legitimately reduce demand for DRAM. Like, a lot of AI DRAM usage is (presently, though might change for more-advances AI) for data that doesn't change much. Like, you're only putting it in DRAM so that the parallel compute hardware has fast access to it, not because it needs to be written to much.

EDIT: It also might provide parallel compute manufacturers a way to price-discriminate between more-price-sensitive GPU (gaming) users and AI (often much less price-sensitive) users via a route other than capping on-card DRAM for gaming users. It'd kind of suck for gaming users if texture memory gets capped because GPU makers don't want to cannibalize their AI market. If you make your parallel compute cards have a ton of onboard flash, that's something that gaming users don't care about --- they need their memory to be mutable --- but AI users do.

3tb/s is insanity. No one man should have all that bandwidth

No worries. I'm sure they'll be fleecing as many as they can in a shared access scheme.