I mean, I haven't really started discussing it outside IRL friends, so not surprised you haven't heard of it 😅

In essence, the problem is twofold:

1. LLMs tend to use CLIs more effectively than MCP, at least in the domains I use. However, installing all the CLI tools and configuring them can be cumbersome and not easily repeated. CLIs also cannot be modified, and are often quite verbose.
2. LLMs will repeat very similar commands over and over with just a few tweaks. This often happens when there is a gap in a tool or poor documentation.

So fcli is intended to be an easily extended, configured, and authenticated CLI. The "one CLI" to do everything your LLM needs. Ideally, you set up a personal repo to store 'ucks' (f packages) in, and you set up credentials for that repo so your LLM can push any new ucks to it. You can then share that repo with other users if they could also benefit from any of that functionality.

As you go along in a session, when you notice the LLM repeatedly running similar verbose commands, you can have it create an uck for it instead and save those tokens.

Likewise, if, say, an MCP server doesn't have full functionality, you can have your LLM import the SDK from npm into an uck and write a basic fcli wrapper around it to start using.

The other nice thing is that it makes your tool configuration repeatable between LLMs and harnesses. You don't need to set up your MCP servers in both Claude and Codex and Pi, just in f and tell them to use f.

There's also a creds uck to make your secrets more portable and (a bit) safer.

In general, I'm trying to model it a bit like pi.dev, but as part of the tooling instead of the harness itself. I think they work quite well together :)

Interesting paper comparing MCP and CLI

Might wanna try a ternary model, which are a lot denser in comparison.

Ternary Bonsai 27b is 7.17GB in size, which might fit on your 3070 depending on how much free VRAM you have (and with CPU offloading it should work in all cases). You can also try their edge-computing model designed to be working on smart phones here (iPhones seems to have a limit of 6Gb RAM max for such things) - that one is less than 4GB in size, which should fit in any case. In my experience they do work very well compared to other models in that size class.

Be aware that they need a specific llama.cpp build found here

I wouldn’t say I ‘run’ it, but I mess around with it

I have a base level Mac Mini I got before the price increase a few years ago, also from when models were smaller and my ambitions were less. I’ve contemplated upgrading it, but I have other things to spend my money on.

M4 processor, 16 gb ram, 256 gb storage, and a 4tb external HDD drive (an old NAS drive).

It’s fine. It’s mostly a docker machine, but I do throw AI tasks at it a few times a week. Mostly stuff that I just let it chew on for 5-10 minutes and come back to.

I run openwebui, and have several models I bounce between. I mostly run Gemma4 (Google) and Granite4.2:8B (IBM).

Editing to add: the self hosted version of all these models, via llama, just in case that was unclear.

I just got qwn3-4b-instruct working on an Intel A750 with 8 gigs of vram.

I use it to power Home Assostant/Voice Assistant on a few M5Stack echos around the house to replace cloud based smart speakers.

Acronyms, initialisms, abbreviations, contractions, and other phrases which expand to something larger, that I've seen in this thread:

4 acronyms in this thread; the most compressed thread commented on today has 19 acronyms.

[Thread #135 for this comm, first seen 17th Oct 2026, 13:50]
[FAQ] [Full list] [Contact] [Source code]

I'm just glad that this is both Top & Hot comment. 🧠✊

So one thing that's really helpful to automate is checking all the repos for my self hosted stacks and testing/validating/deploying new versions when they come out. They're also good at writing helper scripts in Python to automate things deterministically so I don't have to keep using LLMs for it. There's an agent that is focused on learning and managing my wife's school schedule, ingests her grades and syllabi and reviews her homework and makes helpful 10 minute podcasts on concepts she's struggling with in her engineering program (not notebookLM, a self hosted lookalike), there's a domestic agent that accepts uploads of grocery receipts and does our meal planning and schedules time for meal prep and chores and divides it up between us and the kids depending on everyone's calendars, there's a home media agent that manages qbittorrent and my ratios on private torrent trackers and files new manga or TV shows or movies ebooks audiobooks etc in the right library when new things come out we might like, viewing/reading/watching metadata tells the agent what content is popular to queue up in the next sweep - there's a home assistant front door agent that can make automations or report sensor state or trigger scenes or entities in the general text channel of our home discord server and can tell me where my phone or keys are thanks to ESPresence being able to use cheap ATOM esp boards throughout the house (tiny little USB powered things with Bluetooth and wifi) to triangulate the location of any watched Bluetooth device or tracker tag in 3 dimensions through floors and walls - what else..there's a home security red-team agent that looks for security holes in my cloudflare tunnel/kv worker/tailscale/caddy/protonVPN routes and monitors a cyber security RSS feed, I have a job/market research bot that ingests another RSS feed for news relevant to my industry and organization and gives me email and voice briefings on new regulations and corporate leadership signals, stuff like that. What else.. there's a game dev bot where my kids can talk to a discord bot that routes to a game dev agent that has Godot MCP and Unity MCP and renpy MCP and lots of other fun tools like Roblox MCP and Minecraft to make custom games we can deploy on Google cloud for their friends to play (my daughter is really into visual novels for some reason I don't get it) - all sorts of fun stuff. Discord is kind of the family shared interface and then the privileged control planes are routed to my phone through tailscale and the home media server stacks are exposed through cloudflared.

Hardware is a 5070 12GB VRAM, an AMD 5950x and 64G of DDR4 running on Nobara (Fedora spinoff).

I use 3 models for most things: Ternary Bonsai 27b and Gemma 4 E4B Uncensored Q8_0 for everyday stuff, and Qwen Agentworld 35B A3B Q8_0 with CPU offloading for tasks that require to keep the model on track for long periods of time.

Everyday performance is best with Bonsai for me - it comfortably fits in my VRAM and is noticeably better with staying on task than similarily sized models. But if i want it to work overnight on a task without issue, the lower speed of the agentic Qwen is outweighed by its capability to do this.

They are allowed tool calls for websearch and reading webpages and have a sandboxed directory for R/W operations, but anything regarding command line and R/W access in other areas has to be manually authorized by me (i'm not willing to play with fire here).

They are currently not reachable from outside, but i am currently building a smaller machine as a home server, which will then be using a 3070Ti and 32G RAM to provide probably the Bonsai Model for me on the go per VPN.

The smaller models are here for quickly creating documents i do not care to do myself, RAG for technical documentation, creating quick python scripts for automating small tasks and teaching me RegEx and Python(mostly by analyzing the LLM output and playing around with it) and also some RP as NPC in a fantasy setting. The agentic model is currently working on cataloguing my games collection (around 36000 titles) by looking up games, their genres and descriptions and filling in blanks in the database. (Funnily, this task was refused by any model freely available because of the size of the database. Even the local models moan and bitch that it takes so long. No shit, this is why i outsource this!)

I'm not advertising any service. I added an image because Lemmy wouldn't let me post the question without one. I think I may have chosen the wrong post type or something. Sorry for the confusion.

Played around with it on my desktop machine. Someone's distilled variant of Qwen 3.8-27B-Q4_K_M, on a intel12700k + RTX3090 24GB is what I use.

I haven't adjusted the reasoning down from very high but I mostly use it for trying tweaks to half-finished coding projects I've lost motivation to complete many years ago. I used it to develop a very simple JS-free personal website. I have secure shell which I can access from Termux or PuTTY, but sometimes I have the llama.cpp webui insecurely accessible online for one day at a time just for fun.

Qwen 3.8 flash next on a 4090, using strata instead of llama.cpp, gives me about 45-55 tokens/sec. Mainly used for coding tasks. The model size is perfect. If the qwen 4.0 models can be a bit less wordy in reasoning but the same size and quality they’ll be really strong.

Does Immich sell/use/keep your ML-based FR? Or is this totally local a never leaves your instance?

That was always the question that bordered me... I run Leeche its a very simple and minimal photo album service and I really like it ! But my family want something that's more like immich but I'm very cautious about what i'm selfhosting and something that collects data points on my behalf is a no-go !

My server's old GPU (8gb) can't run anything useful LLW wise I don't think.

Have run some stuff locally on my main computer, but nothing setup as a proper service.

Is that the only community? I see the last activity was 2 months ago.

I'm confused about some of these. For example everything you mentioned in the torrent agent (maintaining ratios, watching for new media, seeing what's popular) are already settings in the *arr apps that use significantly fewer resources, no?

there’s a home assistant front door agent that can make automations or report sensor state or trigger scenes or entities

This is all basic functionality of Home Assistant, yes? I'm confused about what is the agent is doing?

I recently gave Strata a shot for my local AI. I'm very impressed with the performance they have eeked out of my hardware. I've got a rather beefy gaming desktop I built as an AI server back when the current craze started, before prices started going up, but models up until this point really failed to take full advantage of that hardware. Strata (which uses a Qwen model) both simplified the setup process, and it maxes out the hardware when running.

Strata acts like an OpenAI compatible API server, I then have Hermes as a "harness" to make an "agent", which I interact with on "buzz", which is like discord, but made for bots. Although Hermes also supports discord and several other chat platforms.

If you have the hardware to run Strata, I recommend setting it up, it comes with a basic chat interface, which you can then grill to assist you with setting up Hermes and whatever else.

One thing I didn't mention, is security. I highly recommend you treat AI as unpredictable, unreliable, and capable of being taken over by bad actors. This isn't a problem for a pure chat interface, but Hermes gives the AI tools to do things, and those tools could be used to take over your computer, scan your network, and create all kinds of havoc. For my part, Hermes is running in a virtual machine which is on an isolated network. I keep tight control on what it does and doesn't have access to in my network main network. That may not be something you have the hardware or knowledge to do, at the very least I recommend running Hermes in a docker container, and not giving it access to your personal desktop nor credentials to things like your email account.

As a service? No, not really.

I use llms for specific tasks, usually small models, for things like identification and tagging (immich, paperless), translation or transcription (whisper), etc. I have a bunch of larger models on a Mac mini and my work laptop for various testing purposes, and have used them to make summaries and other such things.

They aren't great for search IMO, and search summaries can be quite bad.

I do find them useful to interrogate an unknown rs232 protocol or similar, though, which is where larger models come in handy.

Corporate LLMs are generally garbage to me not only because they are just data farming, but also because they really aren't good at everything. They might claim it, but the reality is a tailored model is often the better approach.

Using openwebui as a front end for llama.cpp is a decent strategy, combined with sub-agent routing (so generic model hands off to tailored models then returns the right info). This can be useful for some scenarios.

A caveat on coding - IMO, they are mostly bad at it. If you are using it to generate a small part of something that needs to be repeated they are handy, but at scale they aren't great, meaning for the entire project (or even an entire plugin or module, depending on complexity). Maintainability becomes a serious issue. While they have improved somewhat, there are limits to what they can do.

Macs with M series chips are good options for a hardware base since the memory is shared, and IMO is one of the most cost effective options over a dgpu with a ton of memory. A dgpu with a decent bit of memory (say 8-12GB) is a solid choice for a model thats larger (27b with 3 or 4b active as a mixture-of-experts model) especially if you have an iGPU for regular graphics tasks. Since its all shared on the M series macs, just get something that fits entirely in memory there.

Your use cases are quite similar to what I've been trying to action.

One question I keep wondering about is the "security" aspect of it. And how the inputs to the model might work.
Yes it can read your calendar and you could "Jeeves, please remind me to hang out with Wooster next week" but it's easy to forget and I have a hard time logging spontaneous events.

Any ideas on how you imagine the friend/activity tracker could work?

Was thinking maybe it could read my smartwatch location history. Or scan the photos I take. But that gets dystopian fast (yes it'll be local but then I'm worried about security).

Fun to ponder!

What a timely comment.

I recently gave my kid a gaming PC for his birthday...which I used as an excuse to upgrade my GPU from a 1660 Super to a 5060ti 16GB

Combined with a 5600x3d and 32gb of system memory.

I just started messing with local LLMs yesterday. "Getting started" is rather easy. pacman -S lm-studio (I use arch, btw), launch it, download a model that your hardware supports, and start talking.

Right now I'm working on getting it to talk to vscode.

Only 33% downtime? Damn, that must be nice. I really need to get a hardware update.

24GB GPU

Qwen 3.8 ( 2 variants )

Pangolin>OpenWebUI>lama.cpp for general use and available outside the homelab

Zoo Code>lama.cpp for coding , mostly scripts and html

Qwen 3.8 was a game changer for me.

If you haven't seen it already, take a look at !localllama@sh.itjust.works

Not me. Call me old-fashioned, but I prefer organic.

Random Question, I've been trying to offload some of my stuff handled by claud to basically this exact setup. However one of the things I manage has a pair of MCP servers that are required. I might not be smart enough, but how do you get Qwen to use an MCP?

I think having a chat interface is part of it so it's friendly and convenient to use. I think someone suggested a matrix bot and that would continue the theme of privacy. You could have different channels or groups for different personas. Maybe one is your Majordomo, another is a simpler calendar bot, another is for journaling, etc.

Then, wire in some cron jobs to invoke the AI to prompt you on various subjects. "Anything you want to share?" every day at 9pm. If you miss a day or two because you were busy or asleep, you can catch it up later or not at all. You could even have it ask what you did at work if your life is like mine and they kinda blend together and you don't know how many hours you worked.

Come to think of it you could use it as a note taker. It could separate work notes from personal notes.

I do not run Immich but their privacy policy states:

Locally Stored Data: Immich stores all your photos, albums, settings, and other metadata locally on your device. We do not have access to this data, nor do we transmit or store it on any of our servers.

https://immich.app/privacy-policy

Additionally in their docs I can find no mention of connecting to outside services. I do see references to downloading hugging face models.

https://docs.immich.app/features/facial-recognition#how-face-detection-works

Not for complex tasks, really. Gemma4b or similar might be just enough as the backend for a voice assistant replacing Alexa (which I honestly hope, I need to get family members off of that fucking spy devices…). Small web searches might just fit. Or image recognition for Nextcloud Memories, stuff like that.

Thank you very much ! I didn't expected someone doing my part to RTFM o_O !

Without doing again the heavy reading part, what should I expect from hugging face models?

If my reading skills are correct it should be local, but I'm really paranoid about this stuff ^^ Even though I read your posted links, I'm still not sure if I should trust them...

Something written in a manual and what the code realy does are sometimes far appart ^^'

Posts from 16 minutes and 2 days ago here. Maybe your instance is not federating with theirs.

I see the last activity was 2 months ago.

that means no one from your instance is subscribed, if you subscribe then you'll get new posts

you can see that there are new posts here https://sh.itjust.works/c/localllama

32GB RAM, RTX 2080 8GB VRAM, I use Tiel Coder 35b-a3b (it's a fine tune of Qwen 3.6) with Zoo Code

• 96gb ddr5
• intel i7 12th gen
• RTX 3060

it also doubles as my media ripper. I initially built it specifically for that before AI was even a thing.

I run too many models to count really. everything through CLI because I'm dissatisfied with all the sloppy UIs.

Immich is entirely free and open-source software and got a grant from the FUTO Foundation, which is fiercely pro privacy. As such, it does not engage in any data sharing. Development is financed through the grant and purchases of the server licence (I got one; it's ridiculously good software).

The downtime isn't the biggest issue, you can plan around that. The real problem is the random lag on some queries and volatile memory being lost (usually when there's a context change).

oh yea feddit blocks shitjustworks https://feddit.online/instance/sh.itjust.works/posts

maybe @xSikes@feddit.online could ask their admins to convert the defederation to a "silenced" or whatever that feature is on PieFed

No worries. Yeah, the user interface is difficult. Thanks anyways for the post, it motivated me to self host my own agent, and it works quite well actually.

In the case of HA, the agent is receiving ambiguous natural language requests and converting them into HA MCP calls. I can open the HA app and press buttons, sure, but the agent can reason over time about patterns of water/electricity use, provide natural language feedback, push back on my automation ideas with counterexamples from sensor logs - and all the sensors in the home and on the phones carrying the companion app are rich signal sources for an agent. It unlocks neat tricks like when our babysitter comes over the front door agent can alert her if there's a water leak and walk her through shutting off the water to the house or locate the backup key for her. It can DM me when someone rings the doorbell and carry on a different conversation with STT depending on if it is a person wearing a uniform or holding a clipboard for example thanks to Frigate and some custom vision ingestion with Qwen 3 VL.

Regarding the torrents, that agent keeps track of my ratios on the private trackers and I can give it natural language feedback about what I did or didn't like about a show or movie and it will suggest alternatives, find them on a private tracker, download them, unpack them, and ingest them in the library for me, keeps track of the families like and dislikes and suggests media to watch together, watch private trackers for open invites or interview slots and take the knowledge interviews for me on IRC, deduplicate downloads and hunt for releases with minimum quality standards etc. one of my first tasks for it was replacing my music collection that got damaged when my external hard drive got dropped with no backup and I had to get it extracted in a clean room. Lots of rare lifetime accumulated mp3s gone - mostly recovered now. Searching for replacements would have been a long slog so I automated it.

To your point some of these things could be done by deterministic python scripts and in fact the agents write those to save inference costs later and most of this stuff I could do by siting on my computer an clicking on stuff and reading - the point is that the agents constitute a natural language control surface to trigger all of those actions for me automatically so I can just send a couple chats to a discord bot and it will be done a few minutes later. I'm a busy person I dunno if you can relate. Busy career, got a family to support, parenting obligations, no time to hang out playing video games any more sadly. Building stuff takes time and effort and you have to learn and read about how to do it or pay someone else to do it for you so I was stuck on the easy convenience of windows and Netflix and Amazon/kindle etc. Not anymore! It started with having to install docker desktop to get OpenClaw working and then pretty fast I figured out pretty much everything I wanted to do now that was not work related is better done on Linux and better done self hosted. I would never have gotten over that hump without LLMs.

96gb ddr5

Look at the billionaire flaunting their wealth

Been subscribed. Still shows two months ago

I think that’s it. Even though I’ve been on Lemmy then PieFed for over a year. I still end up not understanding any of this but I think this is one of my core issues here. I would have never know.

Thank you for confirming. I didn’t know I was even missing out on posts.
Your follow up, I don’t even understand. Maybe I should go back to Lemmy (I think my account is on sh.itjust.works) and do my own filtering.

PieFed allows admins different levels of blocking instances, they can do defederation but there are levels softer than that

Looks like Reddit.online blocks a lot.

Replaced to other on top but look at feddit.online vs Lemmy sh.itjust.works.
I think I’m going back to lemmy

Are you saying that the admins of feddit.online can do that only to my account or is this a instance as a whole?

Immich's image recognition works fine off the CPU. I did get it running an LMM, but it was useless.

Epyc 7551? 256gb ddr4, r9700 + mi50. Running two models depending on what I’m doing . Llama.cpp + comfyUI or two llama.cpp for model and advisor in OMP. Main model is Qwen 3.8 27b abliterated

Ollama, openwebui, running the ~20GB Qwen models on a 3090

in my defense I bought the majority of that in 2024 for around $300.

I was surprised to see it's currently listed at over $2000.

Me also, and I can barely afford to run that