yeah, most slop content already existed before chatgpt and was written by humans. kind of like a student is asked to write "a minimum of 500 words for their homework assignment", too many bloggers copied that attitude to their blogs.

You might try to give one of the alternative search providers a try. There's duck duck go for the USA, and Qwant for the EU.

To my old ears, 'bloggers' sounds way too much like the self-employed people with a web-site about their hobby, that they are trying to make a living through.

But there were already content farms, that paid an army of underpaid people to write something about a subject they had no clue about and plaster it with SEO keywords in order to attract ad revenue.

I use Kagi too, it's great.

The other commenter corrected your index note.

What I found most illuminating was a comment by the CEO in an interview, perhaps The Verge? The TLDR of it was that Google makes money when you dwell on their site (so they want the highest ad serves per search), and a paid search engine loses money every page they have to serve you (so they want to give you your answer fast).

So, the user and Google are at odds, and the user and Kagi have the same goal.

Those sites train their models for SEO is how they get top results.

And I agree, somtimes even the domains look like they are generated on the fly to suit your search keywords.

"Number go up" is all that counts.

That's kinda true for any online-based service. Having the the software FOS is great, but someone somewhere has got to run the servers.

I wouldn't be able to write this, if there weren't people who set up the instances for my user account and this community here.
The difference with FOSS and the fediverse especially is, that any of us could be one of these people.

Because "Fuck you, pay me"

Because I bought metager first and have a bunch of tokens left 😅

In Cory Doctorows latest book he describes that it was revealed that google not only neglected the search function, but deliberately made it worse. If you have to search several times to find a good result, they can show you several ads.

Wow, what a great site. I like the cut of this guy's jib.

Because data is never free. Google, duckduckgo, kagi, they all cost money one way or another whether that be a subscription service or by forcing websites to be shit so your only option is using their LLMs. Yes the reason why the internet has gone to shit is because of LLMs

I've went through hundreds of search results on duckduckgo, even when I make it intentionally broad and genetic that wikipedia would cover it, it still doesn't come as the first search result. Search engines have gone to shit because LLMs make the most money in terms of data surveillance. Meaning (take for example google) they have a profile of you, they have queries to train, they have data that can be sold to advertisers. The only reason LLMs are being pushed so much isn't because its intelligent, it's because it makes the most money.

There is a "search engine" that hasn't gone to shit, and that's called marginalia search, It's not exactly a search engine where you can type a url and it will bring you to a page, Instead it compliments your general search engine as a research aid and you can find some golden sites.

I've personally used it for data sheets on IC electronics and their implementations. Unfortunately I've found it insufferably difficult on duckduckgo (like multiple pages deep for a datasheet, something that should pop up on the first page), and I don't trust the output of LLMs (large lying models).

So yea, hopefully this helps.

Just check their Wikipedia page.

Startpage is owned and operated by Surfboard Holding BV, which is majority owned by System1, an American internet advertising company.

I'll have to check that out. Thanks

Others have addressed what I'm about to say, but there is an answer older than LLMs: advertising.

Google made an amazing search product, one used by so so many people from all over. This made them a perfect platform for advertising, and as time passed and it became their primary focus, they realized they could make even more money people spent more time on their own site.

How do you do that, when the product people are there to use is aimed to redirect you elsewhere? Make it take longer to find the thing they are looking for. Search was incentived to work less well so that users saw more ads and for longer.

Integrating AI into search results introduced new ways to keep people on Google itself, including by presenting the answers themselves directly. But it started earlier.

Episode of the Better Offline podcast: The Man That Destroyed Google Search

Important to note that Qwant still uses the Bing search index everywhere except for France and Germany, where they use the Staan search index, which is being developed together with Ecosia.

I'm sure the more people use it, the faster they can expand, though.

Using it from Finland myself, quite satisfied with it even with this Bing nonsense. Only have to use Google for VERY RARE cases.

Money.

Anytime you think it's not about the money, it's about the money.

Yeah, that's why I'd just like to write now. I'm done with it all. I'll wrestle with a fucking word processor, and everyone else can eat a dick.

I've been using metasearch engines (engines that pass your query to multiple search engines so you can aggregate them easily) and the two I've been using were pretty good in my experience. 4get.ca and degoog.org.

Doctorow is awesome. He's a great author too. I'd recommend https://en.wikipedia.org/wiki/Walkaway_(Doctorow_novel) and https://en.wikipedia.org/wiki/Little_Brother_(Doctorow_novel)

I think they intentionally made search engines worse to push people into using LLM's

On DDG (can't say for Google as I don't use it) search results include a date & I automatically disregard anything dated very recently - almost always slop

Lmao no it wasnt I did search for it too

Sheesh, the description sounds like a Behind the Bastards episode.

Can't wait to give it a listen. Thanks

I'll check them out, thanks

Ever since the deployment of machine generated results for search engines, there has always been a competition between the search engines and various others using search engine optimization to get to the top of the results. It isn't worth it for Alphabet or Microsoft to fix their search engines if they can push you to AI, so their non-AI results naturally degrade.

Search quality has been degrading for years, even before AI was involved.

There are leaked emails from Google (from many years ago) where it was proposed to purposely degrade search quality to increase ad clicks. While some at Google responded with dismay as this, it is not a great leap to assume that this faction eventually won and they did in fact deliberately do that. Advertising is Google's main revenue source, they are clearly motivated to do so.

Which are amazing now, but not to worry, LLMs will be complete dog shit too once they pull the rug.

People with bad info put more effort into SEO than people with good info. Google changing focus from getting people good results to getting people to stay on their site. Powers that be filling the web with misinformation. There's always been conspiracies and misinformation on the internet, but now there's major money in pushing it.

But it feels it got much worse much faster over the last 2 years.

Open source: am I a joke to you?

They can still re-rank Google's results to surface more relevant information.

SEO. Everybody is blaming corporations but nobody today can develop a search engine that can't be spammed. If they could, they would.

True, but that reeks of victim blaming. The spammers are to blame here, not the mathematical inevitability of antispammers being behind on the arms race. Also, fucking AI.

The one thing "AI" truly excels at.

If only we (the collective users) spent more time creating a framework and vision of how the Internet should develop and aggressively pushed back on attempts to deviate from it...

When the people refuse to convene a governing body with teeth, the vacuum is filled with opportunists.

In a sense we are suffering from our own success. A place where anybody can build anything they want digitally. Left a ripe field for unrestrained commercial interests to infest the whole thing. Now it is bloated with human and AI slop to such a degree it smothers everything else and every major tech push is tuned towards data scraping every morsel.

They already are dogshit.

If your LLM prompt isn't about something common or mainstream is has an incorrect response most of the time. Unfortunately due to how they work, they will present the incorrect answers with the same confidence as correct answers.

LLMs seem like their only use is writing reports or resumes that no one needs to read. Everything else is better done by hand.

SEO.

Sure, it is kind of true for any service. But the resources needed to scrape and index the entire web are so enormous that, AFAIK, only 2* companies are doing it. Every other search engine goes through them. Kagi only indexes the "small web". It's such a large difference in degree that it becomes a difference in kind.

*There was something about Qwant and Ecosia building a third. Haven't read up recently.

I have a bunch of Search engines where they say different things about your search (I combine it with noscript because fuck JS)

Startpage: you need JS to verification, basicaly is an API from google.

Mojeek: also JS verification, their results are focus in local websites (.es .nl .de).

Searx.be: you need write a code before start use it, index all the fediverse, opensource frontends and scientific works in an anti-massmedia site, also can fail in specific searchs.

I trying different ones, because as you say is not realiable anymore so I have to act as if I asking help to a bunch of greddy friends with their own interests

I have not had good luck with any search engine I've used lately, even Kagi. I pretty much have to go directly on the website I expect to find what I'm looking for and search there now. Also I suspect that with AI people might be posting in forums less now.

Ironically I think I've gotten better results from AI-optimized search engines than from DuckDuckGo

As others have mentioned, it was engineered to be worse for the capitalists to make profits from ads and the statistics to look better to shareholders. This in turn has made people to create a shit ton of garbage articles and blogs that say a ton while saying nothing, so even if you used a new search engine with the funding of Google but the algorithm of back then it would still be shit probably, because the volume of shit on the internet has now reached a level where finding anything in it is impossible.

The explosion of AI has come right at this moment where search engines have been lobotomized and gives people an option to be able to find things again, which is also another factor contributing to the profits of the AI companies, but even that is limited to what an LLM can average out from the totality of information that it has been fed and has the obvious danger of making stuff up and treating its biased training data as truth.

Generally, I think we're fucked. I really don't know if we're ever going to get back the internet experience that once existed and what it's going to take to get there, but it would probably involve a massive societal change to come before it.

yeah, most slop content already existed before chatgpt and was written by humans.

Sure, I'd say most topics already had slop content. However, LLMs can generate entire websites of slop content in minutes if not less. It's pointlessly inefficient to employ people to generate slop by hand. Entire news sites are now just LLM generated slop because it was determined to be cheaper than human generated slop. Most slop content is now, and will going forward be, LLM generated.

I'd add that traditional search can't reasonably hope to compete with AI in this environment. If you're running a traditional search engine right now, there's a real argument that the best business decision is to accelerate the enshitification to capture as much value as you can from your remaining audience while you still have it.

This is the usual cycle with these sorts of technologies - once a replacement arrives, the natural choice is to shift from growth to exploitation.

This is going to get worse before it gets better.

No. It’s a challenge to all the brainiacs out there hacking high security servers with a freaking TV remote. I hope that they’ll read “Nobody today can develop a search engine that can’t be spammed,” and say to themselves, “I accept that challenge.”

Also, AI has been spamming forever. One of the tricks they did in the 2000s was to spam forums with links to websites so that Google would think those sites were more important than they actually were. In 2009, I hosted a forum that went nowhere, left it unattended for a month and came back to find bots had spammed it into oblivion with links. AI isn’t a new problem.

At this point I think the only way forward is for enough like-minded people to sort of stake their claim on some semi-walled off portion of the internet and greatly limit access from swaths of IP addresses.

Sort of like web rings 2.0. Everyone, or enough people, will have to chip in on costs and provide either money or hardware.

There will have to be some sort of regulatory body with voting members.

To be 'in' the site has to abide by certain advertising restrictions and privacy agreements.

A decent use (finally) for llms would be to audit the private index and remove slop. I think we're just at the beginning of what will be a resource hogging AI arms race.

It's sort of what we've been trying to avoid since the beginning but it seems inevitable at this point.