MCP in general:

1. MCP is very verbose (json tool declarations)
2. Every tool incurs a prefix cost, which can be in the thousands of tokens across all tools
3. Tool outputs are usually dumped straight into the prompt. Sometimes they are truncated and dumped into a file with a warning to the LLM, but usually after they reach thousands of tokens
4. Changing any tools breaks your prefix cache.
5. Tool discovery can help with 1 a bit, 2 a lot, and 4 a lot, but isn't a cure-all. It does have its own (much smaller) token overhead too.

CLI in comparison:

1. Fairly concise, by nature - short names for frequently used commands, spaces as implicit delimiters, implied outputs.
2. Tools are generally implied to be present. A short "this environment contains x,y,z," system prompt can be very effective. Very minimal prefix cost.
3. Tool outputs can be written to a file directly, formatted, head/tailed, grepped, or used in any other way.
4. Doesn't break the prefix cache, but can "confuse" the LLM.
5. Tool discovery is implicit in the system. This does have its own overhead, but for many things is implied - ls, find, grep, sed, cat - usually an llm will just assume they're present, and use them to trim down tool outputs to only the required context. In f, I also added search functionality to avoid parsing 'help' messages.

CLI does of course have its own downsides, I think the biggest being security. Give an LLM an MCP server, and all it can do is what that MCP server supports. Give an LLM a Turing complete shell, and it can do anything.

MCP also tends to be harder to set up and harder still to fix when it doesn't work. I've watched Claude fall back to curl commands when a connector doesn't work many times. I've never seen it try curl commands, fail, and then try to set up MCP.

I think this is all fairly domain dependent, but I think my domain (software development and general agenetic assistance) is very common. If you're using an LLM in a product that you're selling with a fixed range of features, MCP is probably the way to go. If you're using an LLM for dynamic functionality with expanding capabilities, I think CLI might fare better.

My main reasoning for all of this is flexibility. In the history of computing, a program has always had a fixed output for a fixed input. LLMs are at the point where they can handle what is effectively an arbitrary input and give an arbitrary output. Giving an LLM a fixed set of tools negates that newfound ability to process the unknown. It's like giving someone a sledgehammer to bust rocks, but tiring one hand behind their back; the work might get done, but not as efficiently.

No, I prefer using my brain or other things that usually work 100x more efficient without hallucinations and don't required a shit ton of RAM that are way to expensive to buy right now.

A Tesla V100 is fairly affordable but older and won't run newer quants like FP8 or attention kernel routines. But it will run a pile of models, even most new models are compatible or usable.

Hell yeah. I totally agree.

With my NVidia Geforce 3070 card with 8GB of VRAM I can get 40 tokens per second on smallish models such as llama3.1 or qwen 3.8. I use it mostly for NLP processing of game server logs to create match summaries and I use whisper for audio-to-text transcription of videos.

While I do not have enough vram to put large logfiles into the LLM directly I was able to have it create a python script that generates a summary file at the end of each match. I feed the LLM the summary file and it generates color commentary that gets posted to Discord. Here is an example:

[🌟]Color Commentary: A Display of Skill on dod_napoli_b1!
[📈]Multi-Player Action: On 08/05/2026 at 19:33:25, the action was intense on dod_napoli_b1, with key moments including the [Loft] capture by Inspector Lag (Allies) and [Axis Barn] capture by THE ROCK/BionicTwin (Allies).
[🏆]Impact Players: Allies: Inspector Lag (Kills: 37), [415]Drago (Caps: 10). Axis: The Notorious C.U.M. (Kills: 39), [415]IronHeaD (Caps: 8).

$ollama list
NAME ID SIZE MODIFIED
qwen3:8b 500a1f067a9f 5.2 GB 5 months ago
llama3.1:8b 46e0c10c039e 4.9 GB 5 months ago

$ollama run llama3.1:8b

what is the best AI model to run on an Nvidia 3070 with 8gb of vram?
The best AI model to run on an NVIDIA GeForce RTX 3070 with 8 GB of VRAM depends on several factors, such as:

1. Specific task or application: Are you interested in tasks like image classification, object detection, natural language processing (NLP), or generative models?
2. Desired level of complexity: Do you want to run small, simple models or larger, more complex ones that require more computational resources?
3. Computational requirements: What's your expected performance target? Do you need high-throughput, low-latency, or a balance between the two?

Considering these factors and the specifications of your GPU, here are some popular AI model examples suitable for an NVIDIA GeForce RTX 3070 with 8 GB VRAM:

Computer Vision:

1. YOLOv5: A lightweight yet powerful object detection model that's well-suited for real-time applications.
2. EfficientNet-B3: A compact and efficient image classification model, part of the EfficientNet family.

Natural Language Processing (NLP):

1. BERT-base-uncased: A popular language representation model for various NLP tasks like text classification, sentiment analysis, and question answering.
2. RoBERTa-base: A transformer-based language model that outperforms BERT on several tasks.

Generative Models:

1. StyleGAN2: A state-of-the-art generative adversarial network (GAN) for image synthesis and manipulation.
2. DDPM (Denoising Diffusion Probabilistic Model): A flexible and efficient generative model suitable for various image generation tasks.

Keep in mind that these models have varying computational requirements, memory footprints, and performance characteristics. You may need to experiment with different hyperparameters or architectures to find
the best match for your specific needs.

To estimate which models will run smoothly on your hardware, consider the following:

• Memory usage: Make sure the model fits within your GPU's VRAM (8 GB). Larger models might require downsampling or other memory-saving techniques.
• FLOPS (Floating-Point Operations Per Second): Estimate the computational requirements of each model. The RTX 3070 has a peak FLOPS rating of around 12 TFLOPS, but actual performance may vary depending
  on the specific workload.

For more detailed estimates and comparisons, I recommend exploring resources like:

• TensorFlow Model Garden: A collection of pre-trained models for various tasks.
• PyTorch Hub: A library of pre-trained models and implementations.
• NVIDIA Deep Learning SDK: Offers a range of tools, including cuDNN-accelerated deep learning primitives.

Feel free to provide more specific requirements or task details if you'd like me to help with model selection!

total duration: 13.256527877s
load duration: 71.80054ms
prompt eval count: 466 token(s)
prompt eval duration: 62.91049ms
prompt eval rate: 7407.35 tokens/s
eval count: 572 token(s)
eval duration: 12.889617619s
eval rate: 44.38 tokens/s

I'm building toward exactly this, so take it with that bias. The cheap tier works if you lower expectations. An N150 mini PC with 16GB runs small models through Ollama fine, but 8B models are slow on that class of chip, and you'll feel it on anything chat-shaped. Where it shines is async work: summarize this, tag that, answer when it's ready. For phone access I'd keep it LAN-only and use WireGuard or Tailscale to get home rather than opening a port.

Disclosure: I'm working on a small open-source home box that bundles local AI with offline Wikipedia and media. What would make you trust a box like that over a setup you built yourself?

I run the AI service (OpenwebUi) myself, but buy the compute on OpenRouter. Self hosting models is not an economic viability for me.

For coding with large context: Qwen3.8-27B-GSQ-RCO-IQ3_S-mtp
Anything else: Qwen3.8-27B-GGUF:Q5_K_M

What game servers ?
Have you looked into qwen3.5 9b ? Gemma e4b ?
There is plenty of new small models after yours.
There is r/lowendlocalai, a community of people with low computing resources but it is in reddit.

MiniCPM5 may be good.

I will ask Santa Claus for that GPU <3

I tried, but didn't want to invest into it, so I used my 12-GB-vram gpu but no matter what I did it was worse than Lumo from proton,which is basically only one LLM from 3rd party that I am okay to use on occasions and since it has really good free tier I gave up on self hosted LLM.

How much electricity does it burn? And how much does it cost to let it run overnight? I feel like this is the same problem that the companies have, its way too expensive to use it on a lots of things and it is way cheaper to use it on someone else's server..

Unless your brain is hallucinating this whole reality 😀

Does OpenRouter have full access to your prompts? What is their privacy policy?

I get about 45 t/s after some tweaking, getting the thermals under control helped a lot; mainly by reducing the power consumption to 261 W (and downclocking a little bit).

What vision model do you use?

It uses 261 W, probably more expensive than getting a 2nd Mistral subscription with the cost of power where I live (Belgium), but it will be free heat once I have solar panels and a battery.

i run k2 7b, it's delightful!

Qwen 3.8 27B, running on a RTX 5090. Works so well.

They have a flag Zero Data Retention that cones with the promise that after a call no data is kept on any servers. I'm using that.

It's contract level trust; I wouldn't bet my life on it, but overall it's a compromise I'm willing to make.

If you are playing around at home I suggest seeking a MoE model. You get less penalty from having less vram and offloading to normal ram. I run Qwen 3.6 quant 6 on 250k context on a 16gb vram 64gb ram ddr4 PC. I get about 31tps which is enough for my use. After a little tweaking I have put results of it's output into Claude Pro for vetting and it hasn't found any flaws in the simple scripts and things I've asked it to make. I can only run Qwen 3.8 which is a dense model and not MoE. At low quant with not as much context. It spits out accurate short context answers if that's all you need. So I prefer 3.6. Tiel is faster but Claude picked up more errors in what I was doing. Orinith kept getting into thinking loops for me so I stopped using it. Anyway I'm just a little guy.

Thank you, that is not as bad as I expected. 👍

A quick glance shows that I'm in the minority. I grabbed a Bosgames M5 with 128GB UMA RAM when I discovered openclaw (at the time claudebot). I'm running Lemonade, which is a wrapper for llama.cpp, sd-cpp, flux, and others. I'm happy to have paid a "mere" $2400 compared to what similar systems cost now, and now I'm toying with fine tuning my own models. Though, I primarily run Hermes Agent paired with Qwen3.6 35b A3B (using a heretic ablitieration for 0 refusals).

Hermes Agent does a lot of random stuff for me right now. It manages my calendar and to-do list in nextcloud via MCP server. It created and maintains an llm-wiki based on my Obsidian Vault. Then it creates a report every morning about both and the weather.

The server has tailscale installed, so yes I can access it anywhere.

OpenCode recently got hacked and advises changing API keys fyi.

I wish

Even ignoring all of the functional issues with LLMs/"AI" related to their design to never admit to not knowing and preferring to make stuff up instead (hallucinate), I have an ethical qualms about the way they are trained. Two primary issues are that they aggressively scrape data from sources that explicitly do not consent to sharing their work and the data centers which have made hardware impossibly expensive for everyone but them and are so impactful to our underfunded electrical grids that they cause brown outs that keep people from heating their homes among other things, cause major spikes that cause house fires and destroy home appliances, and create an excuse to generate power using the worst polluting generators that we had finally gotten shut down to slow climate change (personally I'm tired of the wildfires and extreme storms from climate change).

So, I personally consider it extremely unethical to use the technology for that and several other reasons which are hard to touch on exhaustively in a post like this.

Pardon the mess, this turned out to be quite a lot of text.

No vector DB, no paid API, no full articles. Plain Python pulling free RSS feeds, plus Ollama running on my own 2070 Super. RSS already gives you the headline and a short blurb, so there are never megabytes to sift through. Every Ollama call is one story at a time, so the context window isn't an issue. Llama3.1:8b for summaries and scoring. Gemma4:e4b for clickbate work, nomic-embed-text for headline duplicate vectors.

Find duplicates:

Python: strip each headline down to keywords (drop words like "the", "says", "update") and only compare headlines that share an uncommon word. Two headlines match if they share enough keywords.

Python: sends batches of 128 headlines to Ollama

Ollama: turns each headline into a vector using nomic-embed-text

Python: compares the vectors, and anything over the config threshold counts as the same story. The vectors just live in memory for the run, no database needed. If Ollama is down, the keyword match still works on its own.

Regen short summaries:

Ollama: Generate a 2 sentence summary from the feed text if available

Get rid of clickbait:

Ollama: Detect clickbait from the headline only.
Classify this news headline.
"informative": it states the news itself (who did what, what happened, what was found).
Dramatic or emotional wording is fine if the fact is stated.
"teaser": it withholds the key fact to make you click: e.g. "The Surprising Reason...",
"The Class That...", "Why X...", "How X became...", "What we know about...", "Is X about to...?",
"Then things got weird."

Unwrap clickbait and rewrite the headline if possible, otherwise drop the story

Ollama:
A clickbait headline hides a key fact. Question it leaves open: {withheld}

Does the text below explicitly answer that exact question? Mentioning the topi
talking around it, or answering a different question does not count.

"answer": the answer quoted or closely paraphrased from the text, or "" if the text doesn't give it.
"headline": if answered, a plain, factual replacement headline (at most 90 cha
states that answer, using only facts stated in the text. Keep the text's certainty: ...

Respond only with JSON: {"answer": "", "headline": ""}

Headline: {title}
Text: {text}

Score each headline

Ollama: return a summary and an importance score in the same call:

SCALE = ("1-3 niche, local or trivial; 4-6 notable; 7-8 major national or inte
"9-10 historic events or emergencies affecting millions right now.")

Python: rank = importance + a bonus for more outlets carrying it - points for age

Don't put US articles in World unless they have non-US countries involved

Ollama:
Which countries is this news story about? List the country where it
takes place and every other country that is a real party to it (acting, targeted,
negotiating, fighting, trading, sanctioned). Use common English country names,
e.g. "United States", "United Kingdom". Ignore countries only mentioned in passing
or as background. Always name at least one country: if the text doesn't say wh
give the country where it most likely takes place.

Respond only with JSON: {"countries": ["..."]}

Headline: {title}
Text: {text}

Python: if the only country is the US, it's left out of World

Everything gets cached per story, so each one only goes through Ollama once.

Wow, this is an absolutely wonderful write-up, thank you!

Llama.cpp running qwen 3.8 on a mini PC with 32GB ddr5 ram and 8GB vram. Not the most speedy but it works. I only really access it while at home but I'm also set up with tailscale.

I agree with you for common CLIs that the model will natural have in its training set, but when building your own CLI it seems like the self-documenting nature and progressive disclosure of MCP would have a huge advantage.
With a cli you don't have the prefix cost, but your agent also doesn't know that the tool exists, what it's for, how to use it, etc. once you give it that information, then you have the same issues with prefix cost, right? Changing tools does break the prefix cache, but changing tools is a thing you can do. You can't change (model awareness of) CLI, and if you could then it'd also break the prefix cache, or you'd have tool instructions scattered throughout the context without structure.
Dumping to file is a harness thing, not a MCP thing. A harness will do the same with large terminal outputs.
Likewise setting up a new CLI can be complicated. I never see an agent try to install an mcp, but I also don't typically see it try to install a new CLI.

I don't see how mcp is any more a fixed set of tools than CLI is. It's just presented to the agent in a structured way, which is important for discovery and instruction following.

But yeah ease of chaining is certainly useful wrt CLI. That's a huge win for sure.

I wouldn't mind running a model at home, mostly to help document my setup. But I just don't trust it. So no.

I just bought a 32GB M6 Mac Mini that I'll also use for local ai. Will probably use Qwen3.8-27B. Accessing it from outside my lan would happen via Tailscale.

Why wouldn't you trust a local model? I mean sure they are less intelligent but if you dont expose it publicly you should be good to go

So you're using CPU offloading? Or the model's Bonsai2 variation?

I don't trust it to be accurate; I'd have to reference the sources it pulled from. Extra work for little payoff.

Got it, makes sense. You'd probably have to do some benchmarking first. Depending on how large the source catalog is.

Downtime is the friends made along the way

It depends on your hardware. But right now the "cheapest" hardware you can buy for a usable setup, is the strix halo (128GB), and run gufo with qwen 3.8 flash next (Q4XL). It is sonnet 5 level. Very good speed (both prefill 1000+ and decode 50+), with up to 256k context.

I have it set up with omp + searxng. Sometimes with hermes. Really good! It changes the game for local setup.

Try gufo with qwen 3.8 flash next. Much much better than 3.6-35B.

strix halo (128GB)

googles

Jesus christ

Just get the cheapest OEM, Bosgame M5. I know "cheapest" is in quotes. But again, it is the cheapest 128GB unified vram device you can get right now. I got mine at pre-orders, so yes, more than a year and a half ago, prices were a steal. Also, great desktop (mini) PC overall and very efficient, even if you don't want to do AI.

How well does it handle concurrency? I could see this as a vast improvement on llamacpp, but I was already speccing out vLLM.

I have like 150GB of DDR3 spread over multiple 2/4/8/16GB sticks, I wonder if I can get that working for something like this 🤔

I run Gemma-4 4B.

It's not the best model, but it runs with CPU only and with 6 ~ 9 G of RAM.