Yeah, I have a fairly useable setup although it's only accessible on my lan. I have a web server but it doesn't host the llm as it's not powerful enough.

Rx7600 8gb vram GPU
5700x CPU
64gb 3600mts ram

Hosting ornith 35b a3b iq4_xs
Llama.cpp server
Pi harness or llama.cpp webui
Searxng web search
Coding sandbox
Restricted filesystem access

Get around 40 tok / sec and it's really useable. Not as good as Claude sonnet, but good enough for most things.

Yes, I installed an LLM at home. I use it for many useful things:

1. reading novels by translating them from English to my language on the fly via machine
2. for communicating on lemmy by translating posts and comments
3. For HomeAssistant
4. And very rarely for development and searching the internet

I have a RX9070XT and llama-swap to switch models on the fly. I'm thinking about installing a second RX9070XT so that I don't take away the translation capability from my lemmy instance while I stress the other neural network with other tasks.

You can setup a reverse proxy on the web server to direct a subdomain over to the ai server if you'd like to have it accessible outside your lan

In case you are unaware, friends don't let friends use ollama. The llama.cpp CLI has become pretty much equally easy to use, plus since you use Docker there's no need to compile anything.

Edit: typos and missing words

Yeah default blocks or silencing or something like that

Very similar, but I'm using Gemma4 12b. I could run 26b but that would exhaust my vram after a short interaction. And ollama. Open WebUI + Home Assistant hooked up with it, a couple very basic scripts I threw together, and connected to Kagi.

I'm trying to get my folks off the Gemini/Google bandwagon for the privacy aspect, but also because it's just cool from a tech geek perspective. For me, any questions throughout the day that I want info on, but not enough to do a deep-dive for an hour+, gets fed to Steve, my models name. Or I'll use it as a springboard and then properly read the citations for more info. And I feed it HA automations when I get stuck scratching my head on an edge case or whatever - it's alright but helps me explain what I want from the automation, and when it understands my goal, great, and when it doesn't I reflect on what I have setup and can usually go 'ah ha' and fix the issue. It's like a rubber ducky/pair programmer that you can both think is an idiot sometimes, but it doesn't judge you when you're the idiot.

Oh shit I wasn't aware! Thanks for the heads up, I'll swap to llama.cpp

Yes, I run Ollama on a NVIDIA 1080 Ti at home. I run granite4:3b for my Home Assistant assistant for my voice assistant, paperless-ngx, OpenWebUI for chat, and a few other things. For speech to text, I run a custom-built hybrid speech-to-phrase + parakeetTTS (requests that match speech-to-phrase take priority over parakeet.) I run GOModel to route requests to my local Ollama instance, or DeepInfra (cloud-based for larger models), or Tinfoil (cloud-based for encrypted LLM chats.)

As much as I love to use local-everything, only my smarthome+basic chats go local. Trying to fit both the STT and LLM models on the same GPU at the same time while also using this computer as a media console is challenging. I've recently started looking at the Hailo m.2 accelerator board to see if I can shift the STT to it.

Hardware:

2 x ASUS GX10 128GB(each)

2 x Mac Mini M4 Pro 24GB(each)

The GX10s are clustered and run DeepSeek-V4-Flash-0731-Abliterated. The M4 Pros run various utility LLMs such as Qwen 3.8

Software:

OpenWebUI

Hermes Agent

LiteLLM

Llama.cpp

vLLM

StoryBinder - custom web app that I use for story/script writing when in the mood

LASM - Local AI Stack Manger - custom web interface to manage pretty much all aspects of my local stack. Agents run on the 4 nodes as well as my NAS(Download GGUF/HF snapshots once and copy out to the nodes)

Have yet to connect my Home Assistant into the stack.

For mobile access I use Conduit https://github.com/cogwheel0/conduit

Yes, I have a desktop with a radeon 9700 ai pro with 32GB of VRAM. Its more than doubled since I bought it but its basically a down clocked 9070xt with double the VRAM. I also have a little arc b50 in my server.

Mostly yeah use it for privacy reasons. I run llama.CPP directly and access via open webui. I host a searx instance for it to use as web search. Mostly run qwen3.6/3.8 or gemma4 on the radeon and almost exclusively gptoss 20b on the B50 along with a second small model for tags and titles in open webui. Yeah I just use it mostly as a basic chatbot or as an agent in obsidian using the engram plugin. Helps search and summarize years of notes and works pretty well.

I've tried to use it as a coding assistant in zed without much luck so usually just post my code into open webui for feedback and it works alot better for me.

Yeah I can access it with other machines on my network including phone with conduit app. I can get to it on the go by connecting to home network with wireguard and sending a magic packet to wakeup desktop.

I havent done any image gen nor have much use for it outside of maybe using it to make app icons in the style of papirus icons for programs they don't and won't support.

It definitely feels more comprable to first ai wave than current frontier models but the privacy and price trade off is worth it to me. I'd say using it in obsidian has been the best use of it so far and one I certainly wouldn't trust to an external AI.

1070ti and some hexacore something cpu that's alright with around 24gb ddr4, I think one of the slower speeds. I also got a faster PC with another llama.cpp instance and I vibe coded a quick and dirty proxy which I can use to switch between the different backends. It's genuinely useful because I got code editors but also my Open-WebUI using it.

Sounds elaborate but generally it's all pretty primitive.

Also fyi long preloading steps because my vram is obv not good enough to hold the full model and kv cache and stuff, generation at about 15t/s tops.

My experience is right now we're at the stage where it's a good assistant but not great for actually taking over bigger tasks completely.

Am looking forward to:

• experimenting with existing non-uniform quants; I believe those quants are the next step to more bang for your buck on those low-grade consumer rigs
• hoping for new kinds of quants to come out and be more widely adopted
• maybe a more universal or consistent harness that is less prone to broken tool-calls etc.
• better models, specifically a new version for 35b a3b.

MCP is a really dumb server that just takes requests and sends back data.

Somewhere in your frontend settings (Open-WebUI has it under "integrations" I believe, some backends can set these things too) you see a section called "MCP server". You need to keep your MCP server running somewhere (it's very slim, so not a huge deal usually) and then in those settings you put your IP and port for the server, for example "http://192.168.1.15:8080/". Usually if you did it right (sometimes you can check if it works in the settings), you can ask the model if it has access to the tools and it will tell you.

What's happening in the background is that every conversation the AI does, it gets a short preamble with some context about what it can do, and if you connect web search, custom skills or an MCP server (it's just skills but they are run through their own server basically), the AI sees all that in this preamble if it's set up correctly. And then when it feels like it needs to use those tools, it will check the description on how to do that, and then it uses them.

Oh damn what a useful link. Gonna check out those orchestrators, maybe I can replace my janky ass proxy.

That's gonna be quick af with the same models.

Yeah running in the background type beat. Not gonna trust this with anything important but it's great to have someone just do shit for you or get a second opinion.

That's a pretty good way to describe those kinds of models. Somewhere between rubber duck debugging and a really lazy intern. Better than an algorithm, faster than a human, but less trustworthy than an intern. It's great if you know how and when to use it. Maybe it'll become an intern soon enough, but for now let's focus on decompiling adobe into bankruptcy.

Not only do I run and share it. I've built an entire service to allow everyone else to do so safely. https://aihorde.net/

https://piefed.social/comment/13343280

PieFed has the concept of silencing another instance. A silenced instance is not defederated from but their posts do not show in the Popular or All feeds and their communities are not shown during onboarding. Their communities can still be found in the communities list (if you unblock the instance first) and joined in the normal way. Once joined, posts in there show up in the Subscribed feed of the joiner as usual.

Since I added silencing functionality I feel like there’s almost no need for (more) defederation. I can offer protection to newbies who don’t yet know which places to avoid and they can explore the dark corners when they’re ready. It’s a low-drama way to put a bit of distance between groups that rub each other the wrong way.

It is, i also use the gemma4b model on it for description generation for my immich photos and its super fast.

Thank you, I'll give this a go

Scheduled most of the time, anyway.

Yes! I’ve got quite a few projects going right now. Probably too many of them to keep track

I am the only user, so it’s relatively easy to run multiple models for what ever task. Ollama is pretty good at balancing which models are working at what time to not bog down they system

I’m running gpt-oss 120 for a general purpose Hermes agent. Out of the box, it was ok. Not very snappy, but it was one of the best ones for orchestrating other agents.

Glm flash (4.something?) does pretty good as well.

I’ve found a hybrid of cloud and local is the best solution. Use local for smaller or more defined tasks. Use cloud for orchestrating, planning, and…fixing the local agents mistakes lol

I’ve got an LLM maintained Obsidian wiki that is written and edited by a few Ollama agents based on chats via my own self hosted matrix instance. I’ve taken it to a point where I’m pretty happy with the results. But I think if I had enough time and energy, I could really dial it in.

One of my favorite use cases for this is when I’m researching topics like UFOs/high strangeness/conspiracy theories. I have my very own LLM maintained Obsidian vault filled with curated notes and links to sources.

Literally, no one cares about this and they think it’s stupid and I’m stupid and I should feel bad for spending time on this or having the audacity of sharing it… they just don’t appreciate the amount of work and knowledge it took to get what I have set up.

I think it’s cool as fuck. I geek out every time that it works.

Oh, forgot to mention I am running it on a Mac studio with 96 gig of unified memory. I think it’s an M3 ultra chip.

I have an LLM running object detection on a b580 that’s also handling camera rotation in frigate. It’s hanging out 95-120 watts

If you update the model json you can pass through almost any parameters you want

It is an extra step though

Do what makes you happy mate! Tell the naysayers to fk off.

What variants? And why two?

Good to know, it's been a while since I played with it and the docs were sparse.

It's local. You can tell by how your computer dies from inference, when you upload new assets 😅. And you can run the models on another computer separately and watch that one die then, when you upload new assets.

Yeah, I know how to do that very easily, but I don't want to have my pc running 24/7 just so I can use the llm when I'm out of the house. My heaviest use is at home. It's a large power requirement for minimal use.

Can you elaborate on LLMs using CLI more effectively than MCP? Do you have any sense of how likely it is just your domain or the MCPs you're using? Do you have any theories about why that may be?

Tried running a decent Qwen on a 3060 TI but I only have 16g ram so it kept getting stuck and not completing tasks.

Ollama container, 2070 super, 3-7b, meta and qwen

I use it to ingest tons of news rss, find dupes, summarize, and then post that output to mesh a mesh network.

I also use it for a self hosted mealie to figure out recipe->ingredient.

I also use it for receipts. I translate pdf -> txt, feed the text to qwen, ask it who is the vendor, what is the price, what it a reasonable memo for the receipt, and give a dozen categories, which is best.

I self host both at home and at work.

At work (for completion and generation) we have a now quite old A6000 running mainly Qwen 3.6 (though I'm trialling 3.8 from Monday) with OpenClaw as the harness. The use case is mainly triage and pre-work on bugs. We have an agent which can be given an exception report with a bunch of information like stack trace and local variable values and it will go off and gather the required code, order files, and data to analyse the issue and come up with a fair idea of the cause and possible solution. This saves devs and support team time. Also looking at adding a PA agent to this which will be able to read emails and calendars and be able to set up (with approval) meetings and move things around and generally make someone's life a bit less admin-y without having to hire a full PA role.

Also technically in the AI space we have a fleet of Pis with Hailo AI hats running object and anomaly detection in visible and LWIR spectrums for QC

You don't need an image, and the one you picked makes your post look like an advertisement. You can edit your post to remove it.

I don't want to create an extra post, so I ask here.

Where do you guys get your hardware from, for an affordable price?

I'd like to run at least a 27b model but my hardware is too weak. I'm on a 16GB unified mem, M4. Tried running Bonsai 27b (it's Qwen architecture) but I had to close all other apps and it is not in a state that I could work with it.

I run Gemma4 12B on my RX9070 XT. it's not permanently on, I just turn it on when I want to use it. Access it through a browser. Does what I want out of it. Nothing "agentic".

I use Qwen 3.8 27b Q4 with 100k context (llama.cpp and the Mistral AI harness) on a 24 GB AMD Radeon 7900 XTX and it's been great. It can handle pretty much anything I throw at it and I can let it work indepedently overnight / over days on long tasks.

I am actually very interested in the news feed thing. How do you find duplicates in the news feeds? Wouldn't you have to pull the actual articles and then have MB of data to shift through which would exceed the window? Do you use a vector DB? Do you just have it use curl or do you pay for an Internet API service?

I've thought about bodging something together with n8n and Tasker so I can just screenshot something or send it a voice note and have it add an event on my calendar.

That being said my server is an 8th gen i3 w/ 12GB of RAM and I'm not willing to leave my desktop running 24/7 so idk if I can run a model good enough to handle this over there. Even if it is a supposedly menial "OCR and call tool" workflow.

Which Qwen?

I can't remove it because there is no delete image option (in my client at least; I am using Voyager), only a replace image. I suppose I could replace it with a single pixel image but I think at this point it doesn't matter.

I write my own code, skid.

Are you planning on using a specific package for your PA agent? Or is this something you plan on developing in house?

I used Cline as my VSCode chat plugin for models in LMStudio. Worked pretty well but I have moved my local LLM stack to Hermes in the meantime.

Using openclaw for the harness and then attaching office 365 mcp with an Entra ID App token to give it access to email, calendar, and teams. The MCP format means that we can explicitly control what it reads and writes

Last night I managed to run Qwen Flash Next iq2 XS using Strata in docker on a VM

I passed through my 3090,
64GB of DDR4,
12 vcores of my 5960x,

Managed 90t/s

I'll probably try a larger version soon but QFN already seems a lot more capable than 3.6 35b A3B, and runs better than Qwen 3.8 dense in llama.cpp

I have nearly the same setup, except OpenCode for the harness. I run 200k context for anything that doesn’t need vision, or 150k otherwise. Typically get about 30 t/s decode
With MTP, starts at close to 800 t/s for prefill and drops to the 200s for a long context. I’d imagine higher decode speeds are possible with some tinkering, but I haven’t gotten around to it.

I tried using zoo with qwen3.6 and 2.5 but it failed horrendously. My card could only handle like a 16k context window with those models, and the prompt alone was like 11k.

Tried opencode and that was fast...but can't get it to interact with system yet. More playing to do.

MCP in general:

1. MCP is very verbose (json tool declarations)
2. Every tool incurs a prefix cost, which can be in the thousands of tokens across all tools
3. Tool outputs are usually dumped straight into the prompt. Sometimes they are truncated and dumped into a file with a warning to the LLM, but usually after they reach thousands of tokens
4. Changing any tools breaks your prefix cache.
5. Tool discovery can help with 1 a bit, 2 a lot, and 4 a lot, but isn't a cure-all. It does have its own (much smaller) token overhead too.

CLI in comparison:

1. Fairly concise, by nature - short names for frequently used commands, spaces as implicit delimiters, implied outputs.
2. Tools are generally implied to be present. A short "this environment contains x,y,z," system prompt can be very effective. Very minimal prefix cost.
3. Tool outputs can be written to a file directly, formatted, head/tailed, grepped, or used in any other way.
4. Doesn't break the prefix cache, but can "confuse" the LLM.
5. Tool discovery is implicit in the system. This does have its own overhead, but for many things is implied - ls, find, grep, sed, cat - usually an llm will just assume they're present, and use them to trim down tool outputs to only the required context. In f, I also added search functionality to avoid parsing 'help' messages.

CLI does of course have its own downsides, I think the biggest being security. Give an LLM an MCP server, and all it can do is what that MCP server supports. Give an LLM a Turing complete shell, and it can do anything.

MCP also tends to be harder to set up and harder still to fix when it doesn't work. I've watched Claude fall back to curl commands when a connector doesn't work many times. I've never seen it try curl commands, fail, and then try to set up MCP.

I think this is all fairly domain dependent, but I think my domain (software development and general agenetic assistance) is very common. If you're using an LLM in a product that you're selling with a fixed range of features, MCP is probably the way to go. If you're using an LLM for dynamic functionality with expanding capabilities, I think CLI might fare better.

My main reasoning for all of this is flexibility. In the history of computing, a program has always had a fixed output for a fixed input. LLMs are at the point where they can handle what is effectively an arbitrary input and give an arbitrary output. Giving an LLM a fixed set of tools negates that newfound ability to process the unknown. It's like giving someone a sledgehammer to bust rocks, but tiring one hand behind their back; the work might get done, but not as efficiently.

No, I prefer using my brain or other things that usually work 100x more efficient without hallucinations and don't required a shit ton of RAM that are way to expensive to buy right now.