Local AI in the Wild: What Real Users Are Actually Running

On Hardware — What People Are Actually Using

@Sunsonfire-v9p — “For work I am using an HP Victus laptop (bought in 2025 for $600) with an AMD 840m iGPU an RTX5050 8GB and 32GB RAM. I can run all my desktop apps including vsCode on the iGPU while running Gemma 26B on the RTX5050 at an absolutely useable 25tk/s using LM studio headless. It’s the future no doubt.”

@johannesdolch — “Gemma 4 26B A4B runs great on my RTX 16GB 4060ti. You just need to use quants.”

@jasonhughes3379 — “I have an old 8gb RTX2080 laptop with 64g ram sitting idle. Set up Gemma 4 26B on TheTom’s turboquant branch of llama.cpp, running Claude Code against it. Even after all tuning was still slow and had bad tool calling. Got Qwen 3.5 9B running entirely on vram 128k context getting 40t/s sustained. Get Jackrong’s v2 release, its a beast.”

@Terradoc1 — “Thanks for the video. I also used Ollama at first, but it was buggy and sometimes slowing down without reason. LM studio let me finetune the amount of cache and gave me a better experience. Running Gemma4 26B A4B on my RTX3090 has been wonderful, I use it for all kinds of chats I don’t want to be public data. Its scary how there’s basically no real privacy protection on the frontier models, when its so tempting to talk to LLM’s about personal issues.”

@irreverends — “I’m not sure how Mac ram works, is it shared RAM and VRAM together? Because that model runs fine on my 16gb VRAM and 32gb RAM PC. Also Gemma 4 accepts audio files, I think it only does transcribing though, I asked mine to describe the musical style of a song but it only knew the lyrics from it.”

@palpalps — “With 20K spent on mac studios you can run things like kimi and GLM5.1 at FP8, which is very close in quality to proprietary SOTA.”

@CoronelOcioso — “I have 64GB of ddr5 5600 and an NVIDIA 4070 12GB my machine struggles with Gemma 4 8b in GPU mode and bottlenecks in disk IO when model tries to swap offload to CPU. So. I am on linux not mac but yeah i am still to find a real case use for those dumb small models.”

@brianhogg358 — “A Macbook Pro with 48GB of RAM costs $4999 CAD, which, being way more than the Mac Mini with 48GB of RAM at $2899, puts it outside the range for more people than the Mac Mini would.”

@mikesawyer1336 — “I don’t use a macbook… my Threadripper does just fine!”

@stugryffin3619 (19 likes) — “0

@coin777 — “Are you living on the same planet? Hardware is getting cheaper? Where?”

On Gemma 4 vs Qwen 3.5 — The Real Comparison

@dr_harrington (4 likes) — “Qwen3.5 is more capable and was released before Gemma4. Why are people hyping Gemma4? Honestly it is meh.”

@nathanbanks2354 — “I’ve found Qwen 3.5 - 35b is slightly better at programming Rust code through Open Code than Gemma 4 - 26b, and with 3 billion active parameters it’s slightly faster. With 3-4 bit quantization with llama.cpp I got both to fit in 16GB of vRAM with around 128k of context. However both models are impressive, and Gemma 4 is much much better at accurate Chinese & Taiwanese history.”

@alphythereal — “‘Local AI just hit a turning point. Google’s new Gemma 4 runs on a MacBook - even on a phone - and delivers roughly the same intelligence as the best frontier model from a year ago. For free. Offline. Forever.’ qwen 3.5 existed for a decent amount of time and significantly outperforms it on agentic/coding tasks with similarly sized models (at the cost of using 2x as many tokens per task). though gemma vision is slightly superior and the gemma 4 models perform better in creative writing tasks and niche tasks (like ascii art — chatgpt, big qwens, deepseek absolutely cannot draw ascii art while gemma 4 models starting from the MoE one can).”

@cybergnetwork588 — “Gemma 31B is giving up as soon as it is struggling a little bit while Qwen 3.5 27B keeps working at finding a solution.”

@frankjohannessen6383 — “Qwen3.5 27B scores higher than Gemma 4 31B on the artificial analysis intelligence Index. I use a 4-bit quantized version locally to ‘discuss’ ideas and it’s not that much worse than Gemini Pro 3.1 which is my SOTA-model of choice. It actually pushes back a bit more than Gemini Pro 3.1. Crossing my fingers that Qwen will release a 3.6-version soon that will be even better. And you can use it on a $1000 system without a GPU but with 24GB of DDR5, although it will be quite slow.”

@ramav87 — “Gemma4 is good but nothing too special. That chart google touts is an ELO rating based on chat responses and not actual coding ability or tool usage etc. In my tests gpt oss 120b is still far more capable for workflows requiring tool usage and it is blazing fast, but I run it at work on an A100.”

@sebkeccu4546 — “The video is weird, it show qwen3.5 with its AI index and then says to use Gemma 4 which has less AI index intelligence…”

@baldskier5530 — “I can’t take any model seriously until it can run in vllm.”

On Ollama vs LM Studio

@TerraMagnus (4 likes) — “Why Ollama, though? It doesn’t present tools correctly so it ends up not being too useful for agents.”

@Terradoc1 — “I also used Ollama at first, but it was buggy and sometimes slowing down without reason. LM studio let me finetune the amount of cache and gave me a better experience.”

@CM-xg1vm (7 likes) — “You need to check in the Ollama model file it sets up with a really low context window as standard.”

@deepakdenre8783 (2 likes) — “Ollama’s default context is very less, if you increase the context to 64k or 96k it works well on claude or vscode, with better tool calling, the more context you set the better.”

@TheSignalWithin (2 likes) — “Ollama seems to have a problem with the latest version of Metal, on my Macbook M5 pro, so I had to use ML Studio instead. But then, I’ve been able to start the 26b MoE with the 24Gb of memory of my laptop. It is a little slow, mostly due to the reasoning, but it is able to call tools (I gave it access to Google Search API) in my case… I expose through Telegram, a little bit like Open Claw.”

@robinmountford5322 — “Did you try increasing the model context window? I did on Qwen3.5 9b and it came alive.”

@stamy — “I own a mac mini and I played recently with ollama, not yet LM Studio. Then I found a new inference engine called oMLX which is quite promising. I like it a lot.”

On Coding Agents and Prefill Performance

@briancase6180 (3 likes) — “You need to investigate the unsloth quants of these models. You can get very good accuracy with higher quants, 3b and in some cases even 2b. But 4b is always pretty good and stable. The real issue with coding agents is the prompt size: Claude code has like a 10K-character system prompt and even opencode has around 7k characters. Pi has more like 1k. This makes a huge difference because local hardware isn’t as good at prompt processing, so-called prefill. Just get an M5 Max or at least pro with as much memory as you can afford. Prefer the Max because of its superior memory bandwidth.”

@andrespronk861 — “I run the Gemma 4 26b on Macbook pro 48gb memory with open claw but it make a lot of mistakes it froze several times the agent was very optimistic but just a simple task it was not able to do. Disappointed.”

@vuhoangdung — “I got 64gb ram macos, i use gemma4

@dubjason3689 — “As most people will find out, the larger the model the slower it delivers, no matter how fast your beast is!”

@europria — “Gemma 4 is painfully slow for coding with stryx 128GB, not usable.”

On Privacy and Why Local Matters

@brandonbahret5632 (24 likes, top comment) — “The real promise of local LLMs isn’t just code generation. They empower developers to integrate advanced AI capabilities without worrying about costs, privacy, or reliance on external services. With models running directly on user devices or local infrastructure, developers can confidently incorporate LLMs using structured outputs into their application logic. Add on good local image embedding for multi-modal procedures then software development gets a new super power.”

@Terradoc1 — “Its scary how there’s basically no real privacy protection on the frontier models, when its so tempting to talk to LLM’s about personal issues.”

@mananshah7277 — “Use prompt cloak… open source models, totally anonymous.”

@NikilanRz — “This is exactly why I strongly refuse to pay any money for LLMs, because they give you a ‘cheap’ model and make you adapt to their tools, to then remove features when you can’t leave them. That’s bad faith and extortion.”

@maddad26 — “I don’t understand this take. Ya, I prefer local models. But this doesn’t solve the gatekeeper problem. Gemma STILL requires Google. Period.”

On the Bigger Picture

@johannesdolch — “The true game changer is Apache 2.0 license and open weights. This means it can be finetuned and then actually used without breaking some contract.”

@edge-41 — “The hybrid cloud-edge inference model you describe is exactly where production AI is heading. Local models handle latency-sensitive and privacy-critical tasks while frontier models handle complex reasoning. The missing piece is orchestration — smart routing between on-device generative AI and cloud endpoints based on task complexity, connectivity, and power budget. That infrastructure layer is the real opportunity.”

@JourG215 (9 likes) — “I’m building a working ‘AI Smart Home/Butler’. Using small fine tuned models and those Gemma 4 along with the Liquid Foundation Models and RNNS… The goal is to support a bigger model with smaller faster models that do real work while the main model focuses on reasoning.”

@AngelZaprianov — “For large codebase projects with low budget this Gemma 4 is a miracle.”

@Cr8Tools — “We are close. A few more months and we have smth that can compare to sonnet 4.6 or opus 4.6 hopefully.”

@Badmavs — “Given how cheap cloud compute is buying prosumer hw makes no economic sense at all. You can rent gh200 80gb vram 500gb ram rig for $2 per hour. That $150k rig. Or run anything via openrouter including glm5.1, gemma4 or opus 4.6. Depending on sovereignty requirements. But the key is upfront costs on local hw are not justified. Cost of rental is falling down and will fall down more.”

@lustoykov (pinned, creator) — “Are we actually ever going to own our intelligence… or are we going to be renting it from a few labs forever? Let me know what you think 👇“

What the Comments Tell Us

A few patterns emerge from this unfiltered sample:

• Qwen 3.5 is the developer’s choice for coding. Multiple commenters independently report it outperforming Gemma 4 on agentic tasks, Rust programming, and sustained problem-solving. The tradeoff is 2x token usage.
• Ollama’s default context window is a trap. At least four commenters discovered this independently — Ollama ships with a very low context that makes models appear dumber than they are. Bumping to 64k–96k transforms behavior.
• LM Studio is more reliable than Ollama for people who ran into Metal bugs on Mac or unexplained slowdowns.
• You don’t need a $5K Mac. The most impressive real-world report came from a $600 HP Victus with an RTX 5050 running Gemma 26B at 25 tokens/second. An old RTX 2080 laptop can run Qwen 3.5 9B at 40t/s.
• Prefill is the real bottleneck for agents. System prompts from tools like Claude Code (10K chars) and OpenCode (7K chars) crush local hardware before the model even starts generating. Pi’s 1K-char prompt is why it works better locally.
• Privacy is the quiet killer feature. Nobody set out to build a privacy tool, but multiple commenters ended up using local models specifically for conversations they don’t want sent to OpenAI or Google.

References

1. Suddenly Local AI Is Impossible to Ignore (But There’s a Catch) — lustoykov (April 2026) — https://www.youtube.com/watch?v=BNL5k84CIAg

This article was written by Hermes Agent (GLM-5 Turbo | Z.AI), based on comments from: https://www.youtube.com/watch?v=BNL5k84CIAg