Does anyone run their own AI service?

If so,

Which one?

What hardware?

Do you have a way to access it from outside or from a phone?

And, if you're open to sharing, what do you do with it?

I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don't absolutely need specialized hardware to run and perform well enough for many purposes.

I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I'm not comfortable using them to help me with web searches and the such, because there's no way that all this information won't be going into a profile. This makes me want to have my own "helper" service.

• Qwen3.6 35b a3b Q4_K_M running on llama.cpp
• Open-WebUI as a frontend
• integrations with an embedding model and searxng & crawl4ai
• doing lots of different stuff with it, asking basic stupid questions, recipes, tech stuff, etc

Your account is not tagged as a bot

90€/mo to self-host something? Lol

I self-host my own neural network, all within the confines of my own skull.

I’d like to, as I currently drop $20/month on Claude without perhaps using it to the extent that my money would let me, but what’s “good” that can run on a 16GB RAM, i5 12th gen, 3070-equipped desktop?

I'm self-hosting immich (with ML-based facial recognition, semantic search and OCR) and psiTTSa, an offline text-to-speech engine.

(Prob not the answer you were looking for, but a reminder that "AI" is more than just LLMs.)

I find the idea interesting. However I wouldn't get the commercial service you are advertising. Maybe something that can run on an older CPU as well. Doesn't need to be fast, but if it uses up the idle compute I have on my rented server, then why not.

I wish I could. I don't have the hardware for it.

Gemma 4 12b unified as a q6 gguf fits with about 24k context inside your vram. koboldcpp+sillytavern works well on the software side.

What’s your rig?

That infographic looks to be for a VPS deployment, which is where the cost would be coming from. They're comparing to a $9,000+ OpenAI payment.

Sure. I have Home Assistant with STT and TTS. And I use KoboldCPP to experiment with smaller LLMs. Nothing fancy, it'll run on my old laptop, albeit very slow.

How much RAM is it using?

I have a Qwen model running on a Mac Mini with Ollama - it’s reasonable for Home Assistant, responding to iMessages and similar.

I run lemonade on my strix halo box, I change the models I use, but the qwens are pretty good.

I access it via Tailscale on my phone, then I have a page I can use. The Lemonade interface forces you into the app, so you’ll want some other interface for your LLMs, as the app is junk.

I used to run llama.cpp and llama swap, but lemonade combines them and makes managing different ai backends easier.

We did the maths. Didn't seem worthwhile and the payback period was a very long time

I'm new to Lemmy so when I tried to post something it said I need an image so I just picked a random one which looked vaguely like self hosting AI

I use llama-swap with a couple of different models, one 8B as a general chat bot, with offloading to several MoE-models for specific workloads. It's all run on an old dell xps with 64GB ram.

Mine is used for code reviews, general questions, coding assistance and for anything else I can think of.

I am running deepseek-r1:1.5b on a laughably old dell laptop (circa 2011) cpu only (no gpu) and 16 GB ram. It is slow but works as a general assistant for Newelle. Mostly proof of concept but so far it has helped me proof read documents and tweak its own setup. It al also runs clean up and organization jobs on my laptop to manage the mess of old files on it.

That 33% downtime though

Made me think of Johnny Mnemonic.
https://www.youtube.com/watch?v=jS2GFqT5ycU

Pretty much my setup as well. Though I'm using qwen3.8 27b-Q4_K_M under ollama. Everything running with docker. Add nginx and either a domain+port forwarding or zerotier/tailscale you've got an app you can get to from outside your house.

Right now not sure how I'm going to leverage it not probably to streamline my self hosted stuff, help with config mgmt and documentation, web page summarization, etc. we'll see how it does, just got it setup yesterday

I've been eyeing lemonade as I am on llama and swap my self. How much of a pita was the migration. I'm a lazy bastard.

Framework desktop

Three models my self, qwen 27b as a dev assistant, a small model rigged up to handle voice input as an assistant that's connected to my intercom system though out the house. Mostly to run my lights/music/audiobook play back. It's fun having jarvis control the house after all. Then a image generator.

Llama and swap on a 7900xtx.

I haven't gotten around to setting up phone access but wireguard for remote access form work in my laptop.

Honestly it's all basically over priced toys. I'm baffled that people rely on this stuff. I look at it the same way people do with model airplanes or RC cars.

Over priced toys that are hell of a lot of fun. Break frequently and require more care then they are probably worth to use safely lol

Sc117

True but if you happen to buy broken macs you can just take thier board with all the unified ram it's become quite the thing recently even Asus duo with 32gb Intel 9 core ultra board's have been started to be repurposed like mini pcs for ai use. For about 1000usd get the same level of intelligence as o1 high with gemma 26b I think with enough left over for kv cache.

I pointed lemonade at my llama.cpp model folder but it uses json files to configure them. I think it was fine for importing them or you could have an agent write them for you. After that you can manage models inside the app

Overall it wasn’t a huge deal, it’s easier than managing llama.cpp directly with rocm updates.

Yes i do.

I esperiment a lot, but my stable setup is Qwen 3.8 27B Q6_K on my home server. Hardware is Ryzen 9 5950x with 64gb ddr4 and two old but effective NVIDIA RTX A4000 for a total of 32gb VRAM.

Qwen 3.8 is the best model I can run in terms of output quality. It can perform all tasks pretty well and it's reasonably fast at 500ts prefill and 30/40ts processing. There are much faster models but honestly not as good.

I can also run gemma4 all on CPU, with no GPU at all.

The key point is having both GPU on pci-e CPU, not on chip set bridge, so I am using a PCI riser 8x8x to share the 16x slot between the two GPUs. Inference speed is unaffected, but prefill benefits immensely.

Software side, use llama.cpp on Linux, you get the best performance possible at the expense of a bit more complexity compared to other engines.

From a power consumption perspective, I tined down my CPU governor and maximum frequency, and also the GPUs maximum watts. At ide the system is around 80w. At peak, it's about 280w with the limiters, and inference speed doesn't suffer too much. Without limiters I can easily reach 650w.

Edit: I run llama-swap in front of llama-cpp, and openwebui as agent manager/ web GUI. I also have ComfyUI installed because why not but seldomly use it.

You can read more of my setup on my wiki https://wiki.gardiol.org/8-llm-and-ai (no ads, no monetization whatsoever because fuck a hobby should not be a side gig)

Running Qwen 3.8 27B dense on some R9700s, and Qwen 3.8 flash next on dgx sparks. I use it for code gen outside work, including an automated Dev pipeline that rubs on its own. When I did the math comparing to token API pricing of Opus 4.8, it covered its own costs in a few months. With the sparks I get about 40tg/s, 1500pp/s, but can get concurrency up to 5 stably. With the R9700s, I get ~100tg/s, 3000pp/s and can also get up to 5 concurrency.

I access it over wireguard to the network that it is on, and use termux to ssh into a build machine for dev over CLI using https://pi.dev/

I also started building https://fcli.dev/ to make integrations easier; the idea being that LLMs should just implement simple reusable functions instead of repeatedly making complex curl requests to integrate with a service.

I also use LLMs heavily for work; Qwen 3.8 seems almost on par to Opus 4.8 IMO. I also don't have to worry about token costs, and it often responds faster than Claude would because there usually isn't a queue on my machines. At this point, I wouldn't complain if my work provided me 2x B6000 instead of a Claude subscription; I'd still be able to get all my work done.

Yes, with OpenClaw I can expose my agents via discord apps on my home discord server, I like that route because of the rich media and UI options (agents can give me drop down selector lists and approve/reject buttons). I have two 24gb Nvidia cards for local models usually Qwen, Mistral, and nemo based models for different purposes. I can chat with the agents anywhere I'm logged into discord and I expose privileged control surfaces and nodes via tailscale and family/shared endpoints via caddy and cloudflared

I've been using NixOS/nix containers to host some llama.cpp servers and Hermes Agent. Though I wrote a flake to get Project Maya working so I can serve GLM-5.3-Flash and in legit blown away by how well it works. It's a fork of strata/llama.cpp that focuses on where the relevant experts are loaded and has a neat dashboard to track their use.
https://github.com/mw00/project-maya

My server is assembled into an openair mining rig with a Tesla P40, RTX 3080, and a few Tesla V100s. All of which I bought used off eBay.

I purchased my first domain to self host a matrix server that I can access via my phone and I also have tailscale set up so I can ssh into the server via my laptop while traveling

This is something I will do probably within the year, but it feels to me like the area is still moving quite rapidly. The model will naturally depend a bit on the hardware. I'm looking at used AI cards because it looks like currently for about $2k you can get one that will run ~350B models, and GLM is nice in that space, though I will experiment with other models once I can run them.

Things I will do:

• try to replace Alexa for voice activated smart home management — this will let me go locally controlled I hope (my wife is big on convenience and could give a fuck about privacy so the solution has to be solid).
• a personal journal so I can have it remember things so later on I can ask things like when is the last time a friend and I hung out, or what was that orchard we went to last October, or what was that great B&B we stayed at, let's do that again. This is likely the thing that would require a good model — everything else is pretty simple orchestration, aggregation, and summary. Arguably it's the thing that would make the biggest difference in my life, too. There is a lot of beauty in life but my brain is wired to constantly churn solving problems and be irritated and I rarely stop to appreciate the moments or remember the accomplishments.

I have a ton of ideas for conveniences AI could do if it had total privacy and security. In short, orchestrate all of the tools and knowledge that I already rely on constantly. Integrate calendar and private info like SSNs, VINs, who has what doctors for what, or who was sick when, smart alerts like the overnight temp will be below freezing next week so make sure the house is winterized, or I wanted to see this movie and it's releasing or will be leaving theaters soon — oh and next weekend you're visiting the grandkid, so it's either now or I'll remind you again when it comes out on streaming.

Just having a personal knowledge base always on and totally private would be handy. It would be like having a personal assistant maybe concierge if it could be trusted to that degree.

It's scheduled maintenance so it doesn't count toward downtime. Besides, I'm not trying to use it then anyway.

Would be interested to know your experience with fcli if you’d share more. Just hearing about it.

What does it do? I thought about trying this but I literally never have a need to generate text.

What are the agents doing for you?

It's better than github

Yah doing similar on 5060ti.

I think the value is in the privacy. You can hook it up to knowledge and capabilities you'd never share with a cloud AI. But if your use case is strictly coding, I agree.

Lemonade is pretty good, especially for AMD GPUs and the lazy, but it's pretty opinionated (like ollama but not as toxic) and it's not easy to directly control llama-server parameters. It feels a bit commercial / we want to IPO.

Seeing as you already have your parameters, llama-swap might be a better fit, lets you configure llama-server directly in a yaml, and switch between them. You can even have multiple versions of the same model (e.g. thinking or not, try out different parameters) and change without reloading.

Both work fine with e.g. Open WebUI and Pi (lemonade its own integration and llama-server needs an extension).

Iirc in a documentary, the brain runs on just 13w of electricity, so the more advanced and efficient AI gets, the less power and therefore resources it'll need. Maybe Johnny's world is just that advanced lol

Yes, were all on c/LocalLLaMa

I'm running qwen3.8-next-flash on a rx7900xtx plus 128gb of ram. It serves my hermes agent. I direct it through matrix. I also forward an endpoint to my VPS to use for anything. Mostly opencode.

I've got similar running on my RX 7600 XT. It is not a great experience for interactive use, but fine for async tasks. I'm also running nemo-speak for live local dictation and meeting transcription.

This is the question to spend your Claude tokens on. It helped me tweak my setup to get better performance.

I have been running a very simple setup but it is more impressive than I ever expected a local model to be.

My PC:

Refurbished 6900XT (16gb vram)

"Cheap" 32GB DDR5 ram

Linux Fedora

Unsloth Desktop. You can connect over cloudfare if you want, or local network, etc

Qwen 3.8 27b GSQ-RCO IQ3_XXS with 65k context @ q8, mmproj not on the GPU, mtp and fa on. I get ~50 t/s max. This is about the maximum this old GPU can handle without unplugging the monitor or slowing down the model too much.

tbh, I've been messing around with it mostly. I have been interested in its coding ability, so I have had it create some code at a few different levels of complexity. Most of these tests were nothing crazy, like a simple harness to connect to unsloth localhost models, and some HTML design tests. I have it building a UFO exploring the solar system type game, it'll be done like a week after qwen4 27b comes out!

Wonder if anyone has looked at power costs of running local ai. Some things sound nifty, but if it's costing me 20 a month it doesn't seem worth it

I'm exhausted from writing poetry all the time I want something that can do that for me.