I've found that all the web archiving software I've encountered are either manual (you have to archive everything individually in a separate application) or crawler-based (which can end up putting a lot of extra load on smaller web server, and could even get your ip blocked).

Are there any solutions that simply automatically archive web pages as you load them in your browser? If not, why aren't there?

I could also see something like that being useful as a self-hosted web indexer, where if you ever go "I think I've seen this name before", you can click on it, and your computer will say something like "this name appeared in a news headline you scrolled past two weeks ago"

Huh. This seems like one of those "this must exist" situations, but I can't think of anything that does this, and a brief search suggests there may not be. The closest I could find was The Internet Archive's Archive-IT, though it's not an exact match. Otherwise, Archive Webpage , a pricey paid-for option (which seems like a terrible idea) appears to be the closest. OSS/self-host like Archivebox and Linkwarden don't really do this (though you can save/send a current tab to them), and apart from that... I don't really see anything.

Yeah, this is exactly what I was thinking, that "surely this must already be a thing"?

But yeah. I can't think of something. I mean, its like, you're already downloading the data. Just write it down somewhere else.

I think a squid proxy can do something like that, or could be tweaked to do that, if you really wanted to.

https://www.squid-cache.org/

web pages used to sort of operate that way with the 'Temporary Internet Files' folder. i'm not sure how it's changed i just know this was how i used to circumvent websites that disabled right-clicking to save their images.

How interesting. Ive never seen this before.

The Firefox extension for archive.org has an option to archive the page you visit if said page hasn't been archived recently. Its not exactly what you're asking for, but similar

Check out archive warrior
It’s dead simple to set up on Docker, and will run in the background while you help literally save the internet. Ignore the steps about watchtower, as that has been deprecated

Wait, watchtower is deprecated? Noooooo.

https://github.com/containrrr/watchtower/discussions/2135

Dang. Its still working fine for me for now (just like that long deprecated trailer downlaoder for the arrs).

Maybe offpunk could fit? I’ve never used it but I read the blog post about it

wget is the command line program to do what you're saying. Or what I use, anyway. Not tied to a browser, though. Maybe you could export your history and pipe it into wget if you're using Linux or have a Linux-like command line?

I also use the FF SingleFile plugin. Again, not automatic, though.