Local AI is useful when the model is good enough for a specific job and running it near the data improves privacy, latency, resilience, cost, or offline access. The best question is not whether a small local model beats the largest cloud model. It is whether the smallest reliable model can complete this workflow under the right controls.
The Direct Answer
Greg Isenberg's map has four parts: the model, a warehouse such as Hugging Face, software that runs the model, and the workflow built around it. That is a useful beginner model. The missing production layer is governance: license review, data boundaries, evaluation, logging, access control, and a clear rule for when a person or stronger model must take over.
Watch the Full Local AI Lesson
Source and disclosure: The embedded lesson is by Greg Isenberg. The episode states that Google sponsored it. The tutorial, model recommendations, business ideas, and performance impressions are the creator's; the implementation cautions and evaluation framework below are JQ AI SYSTEMS analysis.
Download the Local AI for Builders Guide
The attached four-page guide has been preserved as a direct download. It includes the local-versus-cloud decision, the four-part stack, a hardware starting table, the first workflow, a quick evaluation, and the three startup filters.
The creator also provides the original local AI guide page.
The Four-Part Local AI Map
| Layer | What it does | Starting point | Production question |
|---|---|---|---|
| Model | The weights that produce or classify output. | Gemma, Llama, Qwen, Mistral, Phi, DeepSeek, or a specialist model. | Is the license acceptable, and is quality good enough on your own examples? |
| Warehouse | Hosts models, variants, cards, licenses, and files. | Hugging Face Models. | Is this the official publisher, a trusted quantizer, or an unknown derivative? |
| Runtime | Loads the weights and exposes chat or an API. | LM Studio, Ollama, llama.cpp, MLX, or LiteRT-LM. | Where does inference run, what endpoint is active, and which network features are enabled? |
| Workflow | Turns model output into a useful artifact or action. | One folder, one task, one output, one reviewer. | What is the baseline, failure policy, escalation rule, and owner? |
Hugging Face's documentation describes model cards as the model repository's README plus structured metadata. Read the intended use, limitations, license, base model, datasets, evaluation, supported task, and file format before downloading a community build.
Parameters, Tokens, Context, Quantization, and GGUF
- Parameters: learned weights inside a model. More parameters can increase capacity, but architecture and training quality matter too.
- Tokens: the units a model reads and writes. Local inference removes a provider's per-token invoice, not the costs of hardware, electricity, storage, maintenance, or time.
- Context window: the maximum working input and output span. A larger configured context also consumes more memory.
- Quantization: lower-precision representation of model weights. Q4 is often easier to fit; Q8 usually retains more precision but needs more memory. The exact quality tradeoff must be tested.
- GGUF: a common model format for llama.cpp-compatible runtimes. It is a container format, not a guarantee of model quality or license.
Google's Open Model and Edge Stack
Google's current Gemma 4 documentation lists E2B, E4B, 12B, 31B, and 26B A4B variants. Google positions the smaller models for efficient local and on-device use, while its official model card provides benchmark, modality, context, and memory details. Treat E2B or E4B as experiment candidates, not automatic winners; compare them with other models on the exact task.
The wider stack includes:
- Gemma 4: the general open-model family used as the video's main example.
- EmbeddingGemma: small text embeddings for semantic search, clustering, and retrieval.
- FunctionGemma: a small base model intended for fine-tuning into specific function-calling tasks, not a general chat replacement.
- PaliGemma and ShieldGemma: specialist model families for vision-language work and safety classification.
- Google AI Edge and LiteRT-LM: the deployment path for accelerated language-model inference across mobile, desktop, web, and edge environments.
- AI Edge Gallery: an open-source gallery for trying on-device models and use cases.
Three Ways to Run a Model Today
1. LM Studio for the visual path
LM Studio can search for downloadable models, load them into memory, provide a chat interface, and expose a local server. Its official documentation says downloaded local models can run offline; discovery, downloads, updates, and optional network or cloud features still require connectivity.
2. Ollama for an application-facing API
Ollama exposes an API on http://localhost:11434 and supports an OpenAI-compatible route. It is convenient for scripts, internal tools, and prototypes. Ollama also offers cloud models, so confirm that the selected model is actually local before treating a workflow as private.
3. LiteRT-LM for product deployment
Google describes LiteRT-LM as a cross-platform orchestration layer for on-device language models, with Android, iOS, web, desktop, and IoT support. This route makes more sense when the goal is an application shipped to devices rather than a desktop experiment.
Local Is a Data-Flow Claim, Not a Brand Name
A workflow is not private merely because it opens in LM Studio, Ollama, or another local-friendly app. Before using real customer or regulated data, verify:
- the exact model and endpoint selected for every request;
- whether inference, embeddings, speech, search, or tool calls leave the device;
- telemetry, crash reporting, update checks, and connector behavior;
- where prompts, outputs, logs, caches, and model files are stored;
- who can reach the local API and whether authentication is enabled;
- the model license, acceptable-use terms, and client data agreement.
If a local server is exposed beyond 127.0.0.1, treat it as a network service. Add authentication, restrict interfaces and firewall rules, and do not expose it directly to the public internet.
The First Workflow: Customer Notes to a Weekly Memo
- Create an authorized folder called
Customer Notes. - Add ten representative support tickets, meeting notes, transcripts, or feedback files. Remove data you are not permitted to process.
- Ask one local model to create
what-customers-are-telling-us.md. - Require repeated complaints, exact supporting language, likely causes, uncertainty, missing evidence, and one test for the coming week.
- Review the memo against the source files. Mark unsupported statements and missed issues.
- Run the workflow ten times before changing models or fine-tuning anything.
The valuable output is not a pleasant chat response. It is a reusable, reviewable artifact that improves a real decision or handoff.
Run a Local, Cloud, and Hybrid Evaluation
| Route | Best fit | Measure | Common failure |
|---|---|---|---|
| Local | Private, repetitive, offline, low-latency, device-adjacent work. | Accepted-result rate, latency, memory, misses, false alarms. | Using a small model beyond its reliable scope. |
| Cloud | Hard reasoning, broad research, very large context, complex synthesis. | Quality lift versus added cost, latency, and data exposure. | Sending sensitive data without a lawful, approved route. |
| Hybrid | Private first pass plus selective escalation. | How much data stays local, escalation rate, final acceptance, review time. | Sanitization that leaks context or removes facts needed for reasoning. |
Use the same ten examples and the same scoring rubric for all three routes. Compare factual coverage, format compliance, unsafe claims, latency, cost, and reviewer time. Choose architecture from measured workflow results, not a generic leaderboard.
Three Local-AI Startup Wedges
Home health documentation QA
The proposed wedge is a private first-pass reviewer for visit notes, care plans, and dictated transcripts. It could flag missing fields and inconsistencies before billing or supervisory review. This is a high-stakes healthcare workflow: begin with retrospective, de-identified evaluation; involve compliance and clinical owners; never let the model make care, billing, or submission decisions autonomously.
Offline field-report copilot
A restoration technician could assemble photos and voice notes on site, draft a report, and flag missing evidence before leaving. Local inference helps where connectivity is poor and images are sensitive. The pilot should focus on completeness prompts and draft preparation, while a technician remains responsible for measurements, scope, and insurance-facing claims.
Pre-send reviewer for professional services
A local desktop reviewer could compare outbound drafts with a firm's own checklist: unsupported guarantees, sensitive data, missing caveats, scope promises, or number mismatches. Start with one vertical and one document type. The tool should cite the triggering rule and text span, then leave the send decision to an authorized professional.
Build a Small Local AI Lab
- Use the computer you already own and download one model that clearly fits its memory.
- Choose non-sensitive or properly authorized files for the first test.
- Create one artifact: memo, checklist, brief, report, classification table, or review.
- Record model, version, quantization, runtime, settings, prompt, time, memory, and result.
- Build a ten-example evaluation with expected answers and known failure cases.
- Add cloud escalation only where it produces a measured benefit and policy allows it.
- Do not buy new hardware until a repeated workflow proves the constraint.
Video Chapters
- 0:00 - Introduction
- 1:35 - The open-model landscape
- 3:09 - Vocabulary decoder
- 6:48 - Google Gemma explained
- 10:29 - Other open-model families
- 14:20 - Run Gemma in LM Studio
- 18:17 - Ollama
- 20:15 - Google AI Edge
- 21:07 - Hardware cheat sheet
- 21:52 - First workflow
- 22:47 - Workflows before fine-tuning
- 25:06 - Local, cloud, and hybrid evaluation
- 26:33 - Startup idea framework
- 27:22 - Home health QA reviewer
- 29:24 - Offline field-report copilot
- 32:10 - Pre-send reviewer
- 34:47 - Build your local AI lab
- 37:55 - Closing thoughts
Sources and Tools
- Greg Isenberg: I'm Obsessed With Local AI. Here's Why
- Local AI for Builders PDF - attached companion guide
- Creator's full local AI guide
- Google AI for Developers: Gemma 4 overview
- Google: Gemma 4 model card
- Google Developers Blog: Gemma 4 on the edge
- Google AI Edge
- GitHub: LiteRT-LM
- GitHub: Google AI Edge Gallery
- Hugging Face: Model Cards
- LM Studio: Get started
- LM Studio: Offline operation
- LM Studio: Local server
- Ollama API documentation
- Ollama OpenAI-compatible API