Needle Local AI: Build Private Automation Directly on Device

Needle makes a useful trade: it gives up general chat ability so a tiny local model can choose from a developer-defined set of actions, fill structured arguments, and refuse requests that do not match the available tools.

What Needle Actually Is

Needle is an open-source automation model from Cactus Compute. It accepts natural language plus a set of functions or a structured schema. It then returns a tool name and typed arguments that the surrounding application can inspect and execute.

The repository currently describes three core capabilities: tool calling, structured extraction, and text embeddings. A phone can translate “start a 20-minute timer” into a timer function. A smart-home controller can map “turn off the kitchen lights” to a bounded light-control call. A document workflow can extract invoice fields into a declared record.

Needle is deliberately not a general assistant. Unsupported input should return an empty call rather than a plausible paragraph. That narrower contract is what makes an 8 to 29 MB deployment target useful on phones, wearables, robots, smart-home hardware, cars, and small computers.

Current-version note: the video demonstrates the project as it existed on 22 September 2026. The repository now documents Needle 3, including deployable ladder sizes from 8 to 29 MB. Its coding-assistant reference also describes a roughly 35 MB base archive and an engine under 1 MB. Use the current repository and device guide when estimating an actual package.

Watch the Full Demonstration

Source and credit: Better than Jev - because you can build with it, published by The Next New Thing on 22 September 2026. Andrew Warner interviews Cactus Compute co-founder Henry Ndubuaku and demonstrates smart-home controls, reminders, extraction, phone actions, a robot-vacuum scenario, and confidence gating.

The Model Proposes; Your Software Executes

The most important architectural detail in the video is easy to miss: Needle does not directly switch on a light or move a photograph. It returns a structured instruction. The application owns the function, permissions, validation, execution, logging, and recovery.

This division is why the project is buildable. A coding agent can read the repository's llms.txt, install the package, define tools, and wire the returned call into an application. The model stays narrow; the product supplies the useful behavior.

Needle Versus Jev: Different Deployment Decisions

There is no need to force a winner. A product could use Needle for immediate on-device commands and escalate uncertain or complex work to a hosted decision model or frontier LLM. That hybrid route preserves a fast private path while keeping a stronger fallback for cases the tiny model cannot safely resolve.

Where Reliability Comes From, and Where It Stops

Needle's byte-level grammar constrains generation to the declared schema, so the output should parse. That solves a real engineering problem. It does not prove that the selected action is semantically correct. “Turn on” and “turn off” can both be valid enum values, and the model can still choose the wrong one.

The project documentation recommends one tool per action, names that users would naturally say, explicit formats, constrained values, and small tool sets. With more than five tools, the current runtime retrieves the five most relevant before generation, making unselected tools unreachable for that turn.

Needle also returns a confidence score for its base model. The current documentation defines the score as the lower of a calibrated post-hoc head and the call-token probability. The engine suppresses very-low-confidence calls, but product teams still need their own thresholds: act, ask for confirmation, escalate, or refuse.

Use destructive and costly tools differently from reversible ones. A timer may execute automatically above a tested threshold. Deleting photographs, unlocking a door, buying something, or changing temperature beyond a safe range should require stronger deterministic checks and explicit human approval.

What “Local,” “Private,” and “Free” Mean Here

• Local inference: prompts and tool selection can remain on the device after the model files are installed.
• Initial download: the standard Python package fetches and caches the engine and weights on first use. Cactus documents explicit downloads and air-gapped packaging for deployments that cannot connect.
• Telemetry: the current repository says anonymous usage counts are enabled by default and exclude prompts, outputs, and data. It documents NEEDLE_TELEMETRY=0 or DO_NOT_TRACK=1 for opting out.
• Open-source license: the repository uses Apache 2.0. Review third-party dependencies, model files, and distribution obligations before shipping a commercial product.
• Cost: no per-request cloud inference bill does not mean zero cost. Hardware, integration, testing, support, optional hosted fine-tuning, and updates still need an owner.

A Practical Build Path

1. Pick one reversible action. Start with a timer, reminder, media control, or local search. Define a visible success state.
2. Write one narrow tool. Use explicit argument types, enums, limits, and a description that distinguishes it from every other action.
3. Install and bind Needle. The current Python entry point is pip install cactus-needle. Construct one agent for one stable tool set.
4. Add a policy gate. Validate arguments again in code, check device state and permissions, then act, confirm, escalate, or refuse.
5. Build an acceptance suite. Include clear requests, paraphrases, negations, unsupported commands, ambiguous wording, multiple actions, and adversarial phrasing.
6. Test on the target hardware. Measure latency, memory, battery, cold start, offline startup, and failure recovery on the actual device.
7. Log decisions without collecting unnecessary private content. Keep the selected tool, sanitized arguments, confidence state, approval, result, and software version.

The repository's llms.txt is useful context for Claude Code or Codex, but it is documentation, not a substitute for product tests. Ask the coding agent to implement the contract and its tests together.

Where Needle Fits Best

A Seven-Day Needle Pilot

1. Day 1: choose one action and write its safety boundary.
2. Day 2: define the tool schema and deterministic validation.
3. Day 3: integrate the base model and handle empty or suppressed calls.
4. Day 4: create at least 30 positive, negative, ambiguous, and out-of-scope tests.
5. Day 5: choose confidence and approval behavior from observed results, not intuition.
6. Day 6: run on target hardware with the network disabled and measure the full interaction.
7. Day 7: review wrong actions, false refusals, latency, memory, battery, and recovery before expanding the tool set.

Video Chapters

Sources and Links

• The Next New Thing: Better than Jev - because you can build with it
• Cactus Compute: Needle repository (Apache 2.0)
• Needle coding-assistant reference
• Needle interactive demo and current model information
• Cactus Compute: Designing tools for Needle
• Cactus Compute: Leveraging Needle confidence
• Cactus Compute: Supported devices and deployment targets
• Vercel: Jev availability through AI Gateway