cross-posted from: https://r1b.it/post/12935

As I have browsed the Fediverse through r1b.it, I've noticed lots of missing content from remote communities. Some of that has to do with actual federation problems (which I have been in talks with other instance owners to fix), but most of it is a result of completely intentional behavior. Regardless of that, though, I feel as if missing content detracts from the r1b.it user experience. So, I decided to flex my programming muscles a bit and devised a solution.

For the more technically inclined bunch, you can keep reading to see how it works. For everyone else, you can just be happy to hear that you should now be getting all new content from your subscribed communities! If you have any further federation issues, please let me know. You can always @ me in !croak :)

::: spoiler The nerd stuff
Lemmy relies on an event-driven feed, meaning it doesn't always automatically pull posts from remote servers unless someone is actively browsing or querying them. So, if a r1b.it user doesn't actively interact with a community, the content won't get pulled. If no new content is getting pulled, no one will interact with the community. To break this cycle, I've set up a lightweight Python script to automatically sync every subscribed community in the background. This script is set up as a cron job that runs every 6 hours.

This script first queries r1b.it's local PostgreSQL database to gather a unique list of every remote community local users are subscribed to. For each community it finds, the script targets Lemmy's resolve_object API endpoint using the community's full actor URL to refresh its metadata and maintain connection health.

To keep this script from turning into a DDOS as r1b.it grows, it shuffles the community list to spread instances out and has a polite randomized pause between each request. Additionally, I plan to remove this script entirely if r1b.it ever gets big enough to federate itself naturally.

#!/usr/bin/env python3
import subprocess
import urllib.parse
import urllib.request
import time
import random

LOCAL_DOMAIN = "https://r1b.it/"
DB_CONTAINER = "REDACTED"
DB_USER = "REDACTED"
DB_NAME = "REDACTED"

SQL_QUERY = """
SELECT DISTINCT c.actor_id 
FROM community_follower cf
JOIN person p ON cf.person_id = p.id
JOIN community c ON cf.community_id = c.id
WHERE p.local = true 
  AND c.local = false;
"""

def get_subscriptions():
    cmd = [
        "docker", "exec", "-i", DB_CONTAINER,
        "psql", "-U", DB_USER, "-d", DB_NAME, "-t", "-A", "-c", SQL_QUERY
    ]
    try:
        res = subprocess.run(cmd, capture_output=True, text=True, check=True)
        return [line.strip() for line in res.stdout.splitlines() if line.strip()]
    except Exception:
        return []

def resolve_community(actor_id):
    encoded = urllib.parse.quote(actor_id, safe="")
    api_url = f"{LOCAL_DOMAIN}/api/v3/resolve_object?q={encoded}"
    req = urllib.request.Request(
        api_url, 
        headers={"Accept": "application/json", "User-Agent": "LemmyAutoSync/1.0"}
    )
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            return resp.status
    except urllib.error.HTTPError as e:
        return e.code
    except Exception:
        return 500

def main():
    communities = get_subscriptions()
    if not communities:
        return

    print(f"Found {len(communities)} unique remote subscriptions")
    
    random.shuffle(communities)

    for actor_id in communities:
        status = resolve_community(actor_id)
        if status == 200:
            print(f"{actor_id} FEDERATED")
        elif status == 400:
            print(f"{actor_id} ALREADY FEDERATED")
        else:
            print(f"{actor_id} FAILED ({status)")
        
        # Polite randomized pause
        time.sleep(random.uniform(1, 5))

    print("Sync complete.")

if __name__ == "__main__":
    main()

:::

the content won’t get pulled. If no new content is getting pulled, no one will interact with the community.

This is a feature, not a bug.

You are trying to solve the issue of discovery by replicating the whole fediverse to your instance local database.

Relays, my guy. Get your instance into relays.

I looked into relays a bit, but I don't think they're what I want for my tiny little instance. Relays are a firehose; they ingest everything from every instance connected to the same relay, which can quickly take up storage on my low-end setup. My little Python script is more precise. It only targets individual communities that my users are actually interested in.

That's all assuming I'm actually understanding how relays work. If I'm missing something, please let me know, and I can change gears without a hitch

I know it's a feature -- that's what I meant by "most of it is a result of completely intentional behavior". However, I then went on to say that it is still detrimental to my users' browsing experience if they want to "daily drive" r1b.it. Like I said in another comment, I'm not replicating the entire fediverse -- that's why I didn't opt for a relay. I'm only replicating what my users want to see

Edit: I also want to make it clear that this optimization is actually super beneficial for most instance. However, for us little guys that are just getting off the ground with only a handful of users, it can be hard to deal with