The origin and purpose of the traffic is not the same. That's why they block one and not the other.

Did I say "shouldn't be"? Could have sworn i said "is"...

You're talking about 2 different kinds of blocking here. NPR can't block a URL, that concept doesn't really exist at this level. They could block ip addresses, or user agents as you mentioned.

A workplace can't block by user agent (at least not on any website using TLS), as this header is not readable by any intermediary devices.

Are you sure? In your post you're talking about fetching an image for a link preview, not an image post.

Usually, when I see an image post, the source is actually the pictrs of the source instance, so in that case I think it is federated.

That's not true, you know the data hasn't been tampered with because of HTTP signatures. The fediverse isn't just "yolo, just trust me bro". There are some parts that are a bit more towards that side (particularly with groups on Lemmy) but I don't think this should be.

How else can you be sure the content is valid though? Lemmy posts are sent as Announces from the community, not from the actor posting the post. The community could just fake a post if you just trust it.

How do you know that data is authentic?

NPR can't block a URL

I wasn't talking about NPR, I was talking about their employer...

You mentioned both in the same comment...

[...] NPR apparently blocks the former

I'm just saying, their employer can't block like NPR does, because they are fundamentally different things. The one is a server receiving a request, the other is a firewall seeing encrypted packets.

Edit: the other could also be a DNS resolver that just does not resolve certain domains.

No one was talking about an image post...

Yes, but then a comment mentioned the "hug of death" - high spike of requests resulting from a link being shared on some site, and OP asked why centralized sites were also creating problems

There was another post regarding previews some months/(knowing me probably a year) ago and personally I would go with federating previews. With a button for mods to refresh it from source for when something goes fucky

But social networks have been creating random bursts of requests, this way or another, even in the times of centralized socials

I figured a post with a link with preview image wouldn't be the same in terms of event as an image post. Turns out it pretty much is, but also, I checked a few posts and cross-posts across instances, all of them have an image url of the original instance, not the one I'm viewing on. So if they are individually fetching this image, they're not using it.

Edit: I was wrong, two of the crossposts just didn't respond to the request for activity, so I was looking at the wrong response data

It's almost like the software that powers the internet was designed for a different time and is only still used because there's monied interest in keeping it around. We need GNUnet.

The great thing about this if you run a server, as soon as a user posts a link to some kiddy porn, you not only download it automatically, you distribute it as well. Go-go-gadget-jailtime.

(Actually already a problem with Mastodon federation of image posts, and why I would strongly encourage against anyone in a strict liability jurisdiction from running their own Mastodon server tbh.)

I just took this random post from my frontpage: https://sh.itjust.works/post/68605733
and made a manual request for the federated JSON-LD.

{
  "@context": [
    "https://join-lemmy.org/context.json",
    "https://www.w3.org/ns/activitystreams"
  ],
  "type": "Page",
  "id": "https://sh.itjust.works/post/68605733",
  "attributedTo": "https://sh.itjust.works/u/derrickoswald",
  "to": [
    "https://lemmy.world/c/technology",
    "https://www.w3.org/ns/activitystreams#Public"
  ],
  "name": "Stratospheric airship flew from US to Japan and back",
  "cc": [],
  "content": "<p>It seems with an unmanned airship they should be using hydrogen for the ~8% increased payload capacity.</p>\n",
  "mediaType": "text/html",
  "source": {
    "content": "It seems with an unmanned airship they should be using hydrogen for the ~8% increased payload capacity.",
    "mediaType": "text/markdown"
  },
  "attachment": [
    {
      "href": "https://arstechnica.com/gadgets/2026/10/trans-pacific-journey-tested-stratospheric-cell-tower-concept-in-japan/",
      "mediaType": "text/html; charset=utf-8",
      "type": "Link"
    }
  ],
  "image": {
    "type": "Image",
    "url": "https://sh.itjust.works/pictrs/image/df608ab3-b644-489d-b217-9224aa09e203.jpeg"
  },
  "sensitive": false,
  "published": "2026-10-19T20:16:37.383385Z",
  "language": {
    "identifier": "en",
    "name": "English"
  },
  "audience": "https://lemmy.world/c/technology",
  "tag": [
    {
      "href": "https://sh.itjust.works/post/68605733",
      "name": "#technology",
      "type": "Hashtag"
    }
  ]
}

Notice the

"image": {
    "type": "Image",
    "url": "https://sh.itjust.works/pictrs/image/df608ab3-b644-489d-b217-9224aa09e203.jpeg"
  }

Yes the image itself isn't sent over (iirc they never are?), but it's stored on sh.itjust.works. Other instances grab it from there, not the website the post links to.

I did notice just now though that some Lemmy instances seem to just proxy the image through their servers, which I assume is to avoid CORS. But that's an instance choice, not a general fediverse issue.

Mbin does the same as Lemmy, but it also includes the image in the attachment property. Probably for compatibility with some other platform.

One thing I didn't notice before was that you're using Piefed. That's the one software I didn't check before writing my previous comment. It does seem like Piefed is the outlier here. I tried a post from piefed.ca and the image is nowhere to be seen. The image property is not present, it's not in the attachment array, and I can't see any other property that contains it.

Edit: A piefed.social post I just tried did have an image, so I assume I'm just blind.

Rimu posted a parent comment elsewhere regarding PieFed and Lemmy.

It's just bad design. What happens when/if the fediverse scales up to many more instances?

Perhaps a modification to the ActivityPub specification?

It could have been FAPS, what a missed opportunity.

It's an access control method, called "authorized fetch". Rather than broadcasting the data you just announce the source of it, and then any client that wants to show it needs to negotiate with the originating server to request the data (And then blocklists and such come into play)

Say you've got 3 users, A, B, and C. A has blocked C, B follows A, and C follows B. Now A makes a post, B sees it in their feed and reboosts it, what does C see?

If B's server sends the full response along, then C can see the contents of A's post even though they've been blocked by A. But if all it sends is the URL of the post, C's server needs to make a request for it (providing authentication), and then A's server can reject it, so C sees nothing.

GNUnet helps building a new Internet

GNUnet is a network protocol stack for building secure, distributed, and privacy-preserving applications.

With strong roots in academic research, our goal is to replace the old insecure Internet protocol stack.

GNUnet provides privacy-by design

It provides [improving addressing, routing, naming and content distribution](https://www.w3.org/2014/strint/papers/65.pdf) in a technically robust manner - as opposed to ad-hoc designs in place today.

I have my blog address (which goes to my home server) in my Mastodon profile, and every time I post on Masto, my server gets a request from every Mastodon site that federates with my instance - i.e. hundreds of requests. It's no problem for my server, because my blog is a static site, but for anything that dynamically generates every single page impression (like worpress or joomla), it could be a problem. Caching settings might help, I suppose.

Truth is, though, I get far more traffic from vulnerability scanners and AI crawlers that don't honour robots.txt (or aren't yet listed).

GNUnet provides privacy-by design

Money doesn't want us to have privacy.

Thanks for bringing this up. I think in general we need to work on optimisation of Activitypub and Threadiverse implementations of it.

But what happens when a link target changes? For example, I've seen news articles change as the author gets new information. Might be an edge case, but a solution like this would always show the preview at the time of the initial post.

Now this is interesting. Thanks for sharing

There's no reason to believe that this mechanism doesn't check for CSAM (I am also pretty stumped that this is actually a problem with Mastodon according to your comment?) . Lemmy had a phase where messed up stuff got federated, which led to @db0@lemmy.dbzer0.com's implementation of an anti-CSAM tool: https://github.com/db0/fedi-safety, which is based on the CSAM detection tool for the AI Horde: https://dbzer0.com/blog/ai-powered-anti-csam-filter-for-stable-diffusion/ - it's probably not so hard to adapt for Mastodon too.

I liked writing efficient code but every place I worked didn't care at all about it. All they wanted was a finished feature and a kanban note moved to finished.

is there something that makes this different to 17 other proposals for a new internet or is it just a case of

Is there a reason why the publishing instance is not downloading the preview image, and then submits it with the post to the other instances? This way they would not have to fetch it themselves.

Yeah, thats how it should be: one download, everyone else downloads from that server, not the original source.

Seems abusable though

If you want to trust a hypothetical "detects CSAM honest" feature with something like this, well, you go ahead.

I mean; just imagine for one moment that lemmy.ml actually is the hotbed of revolutionaries that is going to usher in the Glorious Revolution that it thinks it is; it starts actually moving mountains, politically speaking.

And, some bright spark has implemented the "automatically download and distribute content from unverified sources, protected only by the open-source Kiddy-Porno-Matic Verifier v0.1.23" feature.

If I were Peter Thiel it would take me 5 minutes to start posting links that caused lemmy.ml's servers to start downloading and then distributing kiddy porn, about 20 minutes to get some tame journalists parked outside the owners' houses and lemmy.ml's datacentre, and another 10 minutes to call up the friendly neighbourhood goons in wherever they are located for the photogenic armed raid. And once the "revolutionary leaders in court for making and distributing child pornography" headlines had finished writing themselves, I wouldn't even worry about whether the court would accept the "but we ran the Kiddy-Porno-Matic Verifier v0.1.23!" defence, because their lives would be over already. No smoke without fire, after all...

Nah. "Automatically download and distribute content" features need rather more thinking through than historically Mastodon et al have given them. They certainly don't need to create any more.

(And to be clear - maybe Mastodon has fixed some of their underlying problems - I ran an instance a few years ago, and the constant whack-a-mole of blocking actual-Nazi and actual-Pedo instances, and the risk that it could just go ahead and download something to my servers because a user followed the wrong person had me shutting it down not long after. As I say, maybe they fixed that - but this mooted feature would be bringing back the same problem.)

Of course, maybe Lemmy.ml is (a) hosted in a jurisdiction where they don't give a fuck, and (b) can afford to defend themselves. But this kind of feature will actively work against people who otherwise might host a small instance, and I thought that is what we wanted, not just a few massive instances that can afford to lawyer up.

All, I should say, IMHO of course.

that don’t honour robots.txt

That's essentially all of them in my experience.

The image isn't federated, but [I believe] it should be.

But I am already thinking it shouldn't be because of hosting concerns.

Add metadata with a checksum of the og one with signature and let each client check that up.

Seems abusable though

Exactly

Seems abusable though

If one instance doesn't trust another, they shouldn't federate in the first place.

Yeah but we can't say we want to encourage people to have/build/share independent smaller websites then beat them up when they are linked. That's a huge mismatch of morality. Either we want to encourage people off the big platforms and should make it easier for the little guys or we are blowing magic smoke.

It’s obviously bad enough that major news orgs are blocking it.

Thing is, Fediverse is not a product

HTTP signatures are not a default feature, the baseline is ECDHE key exchange with forward secrecy and deniability. Headers are not typically signed.

There isn't even an official standard for redistributable signed page metadata. There's just prototypes. I absolutely hope one of those prototypes becomes official, but we're not there yet.

The activitypub fediverse is literally very extremely Yolo trust me bro, it works because you can see the originating server with everything and misbehaving ones gets banned.

300 seconds is a lot.

If it's a signed checksum then maybe. But if the og instance is doing the abusing then it won't help.

Yeah there must be some trust indeed of the og instance. A scoring system could help with that.

Is this the same thing as the “Reddit hug of death”?

Is that really why they’re blocking it I wonder…

Or being slashdotted.

that don’t honour robots.txt (or aren’t yet listed).

Throw'em in the tar pit.