I shared a link from NPR Tiny Desk but there was no image preview or video embed. After investigating this I learned that when you do that, each instance picks up the link and individually retrieves the link preview nearly simultaneously. For gargantuan sites like YT or Reddit, this is a trivial amount of traffic and not a problem, but for smaller sites like NPR this results in a sort of botnet DDOS attack on the site's servers. More here. Another example here . For that reason, NPR has blocked Fediverse servers like Mastodon.

I have experienced this on my personal server as well (DDOS), but at the time didn't know why, exactly.

This isn't the end of the world, but certainly is not great and hurts engagement with the content. Is there a solution for this? Perhaps using some sort of webtorrent service like PeerTube where they just fetch the information from another fediverse server instead of the source server? Maybe fetching embeds client-side? At least that would stagger the requests.

E: Many people have replied comparing this to the Reddit "hug of death" or "The Slashdot effect". This is not the same at all. Those are legitimate site visits from humans that happen over the course of several hours. This is the result of several thousand servers all automatically fetching the same image simultaneously , and without any human involvement, and potentially next to zero human traffic to the site, which is why I compared it to a botnet or DDOS attack.

they could include the link preview data as part of the federation

smaller sites like NPR

There are two proposed solutions I have seen. I do not claim that this is exhaustive, just only limited to the proposals that I have stumbled across.

FASP - Fediverse Auxiliary Service Provider

The idea that mastodon has is to have a third-party service that does something like generate a link preview once and then share that to fediverse instances that need it. They have a whole spec they wrote up about these kinds of services that can be run to help out fediverse instances with different things like this, search, or discoverability. They call these FASPs - the repo with the spec is here.

I don't yet know of any FASP that is live and running, so it is kind of a paper spec at the moment. I don't do too much on the microblogging side of the fediverse though, so I very well could be wrong.

Federate link metadata and the post at the same time

This comes from an issue on PieFed's repo. It is basically just an idea at this point, but it is a pretty basic one; have the originating instance fetch the link preview information and simply federate it out alongside the post itself.

That second solution is what I thought of as doing as I read this. It doesn’t rely on a third party service being up. And that instance has to do that work anyway. So why not just share the results with others?

For gargantuan sites like YT or Reddit

Lemmy does this to itself, too - when you create a post it federates out and then every Lemmy instance that receives it makes a GET request to retrieve the post from the original server. Even though it just received the exact same data. Some other fedi software is even worse - Akkoma does a GET of the id of every activity it receives, not just for posts but for Likes too.

It's insane.

Anyway, proper caching at the reverse proxy / Cloudflare level solves it. The same kind of caching setup would do the same for sites serving HTML, like NPR too.

the people have spoken

Well they didn't seem to have much to say?

Norwalk Public Radio. From Norwalk, Iowa.

smaller sites like NPR

A slight delay would be perfectly sufficient.

how would you know the results havent been tampered with?

If they're all delayed the same it would have the same effect. In order to prevent that it would require some form of coordination amongst them. And even then would have the same complaint about missing link previews. Better than nothing but not really a "solution", per se.

The same way you know the rest of the data hasn’t been tampered with - you don’t.
And when you find untrustworthy instances, you defederate from them.

They can't afford a $5 CDN?

I’m trusting the user who submitted the link and/or the instance that user is on. I’m doing that anyway by clicking on their link right?

Is this why the automated title retrieval never works for NPR?

Yep

Random back off is an ancient and very well understood algorithm. This is not exactly rocket science.

Random numbers exist.

I'm a fan of the second one. My thought is, if you trust a server enough to accept messages from it, then you should also trust it enough to let it scrape the page title, article image, etc. for you, too.

No one said anything about random.

Whoa!

wait 3 //randomlydecidedby fair dice roll

This probably why my place of work blocked it behind the firewall

This was my first thought too.

Seems like a no-brainer! They could even leave the old behavior as an option for instance admins to re-enable, but if they default it to not DDoSing, that would make a huge difference.

Do they block the useragent or the URL? NPR apparently blocks the former.

"Random back off" is the name of a technique to avoid this exact problem.

Each instance generates a random number of (say) seconds between (say 15 to 120) and schedules the fetch for then. It is extremely unlikely that any two instances select the same second.

Yes I understand. I just don't appreciate the demeaning tone this information was delivered in.

It was my first thought too but there are downsides to it. Trust is a small one given the nature of the system but compute and bandwidth costs could be an issue also, especially for larger instances.

The first solution is more fragile but could be CDN'd and those are cheap.

I remember reading the same thing about reddit, slashdot, fark, and other sites. But I like the FASP idea as a way to relieve this.

Randomly stagger the times of each instance's request.

+1-300 seconds

Nobody said the delay had to be fixed instead of random either…

Why is that? Those are centralized platforms, they should only need to retrieve the preview once each?

I mean, just how many instances are gonna ping the URL? Several hundred, low thousands at worst?

Unless it happens to be some very large media content or a dynamic page with no caching, I'd think the automated federation hits shouldn't be too bad.

The influx of users that might come after a viral post though, that's rough....

That's basically what's now implemented, iirc. At least in masto

Yes that's the problem...?

So I receive a post from someone presumably with some comment, along with a preview of a link, and I'm supposed to trust one but not the other ?

I don't really get it. The image is federated, there's no need for instances to refetch it. Just checked out some posts and indeed it's there, in the image property. With the URL pointing to the Lemmy instance.

So if this is actually an issue, that's just bad programming and needs to be fixed.

Edit: didn't notice OP uses Piefed. Unlike Lemmy and Mbin, it seems like Piefed doesn't federate it.

Edit: actually I'm blind, Piefed does federate it too, sorry piefed

https://en.wikipedia.org/wiki/Slashdot_effect but people were going in, looking at the link, sometimes overwhelming the server

Also https:// itsfoss.com/news/mastodon-link-problem/

Really the only metadata we are interested in is the thumbnail image url. Afaik PieFed already retrieves this from the og:image meta tag before sending the URL along with the other post content to the instance hosting the community.

Another idea - those meta tags are also retrieved during post composition in an ajax call, in order to prepopulate the title and body fields so the image url could be stashed in a hidden field at that point...

The image is not federated.

That's not at all the same thing. That's just the inability to handle large amounts of unexpected normal user traffic.

This is essentially bot traffic driven by servers all independently but simultaneously fetching the same image, without even a single human necessarily looking at it.

From server's perspective the only difference between those is the user-agent. Traffic is the same

the image should be federated

But what OP is talking about is a technical detail that can be fixed.

I don't think compute resources should be an issue, if anything it saves other instances resources.

As far as I understand, right now, all instances are fetching this data individually, so they all have the same compute load from this. Doing it on one and sending it along with federation should keep resource useage pretty much the same on the originating instance, and less for all others.

This would still increase bandwidth of course, but at least with all VPS providers I have used, bandwith is pretty cheap.

That's weird. Is that how it's supposed to work, or did they just implement it in a bad way?