Build a Jarvis-Style Voice Agent in Cursor: GPT-Realtime-2, Computer Control, and AI Images

Direct Answer

Riley Brown's custom Jarvis is a local Electron desktop app that uses GPT-Realtime-2 as the conversation layer, then exposes separate tools for web search, local notes, Mermaid diagrams, image generation, image editing, and macOS computer control. Cursor builds the first version from natural-language briefs, but Cursor is not the runtime. The finished prototype is its own application with its own permissions, API usage, local data, and failure modes.

The most useful lesson is not that everyone can create Iron Man's Jarvis in three prompts. It is that voice is becoming an orchestration surface. A person speaks one intent; the agent decides whether to answer, search, draw a diagram, create an image, save a note, or request permission to act on the computer.

Video and build credit: Riley Brown. Follow Riley on X, see his agent work, or inspect the public RileyJarvis repository.

Source Note

Riley's video and transcript provide the build sequence, live demonstrations, design decisions, and creator commentary. The public RileyJarvis code provides the implementation details. OpenAI's current Realtime, image-generation, authentication, and computer-use documentation provides the factual and safety spine.

This article was checked on 14 July 2026. The repository is a small, early public project rather than a packaged consumer product. Its README requires macOS and Node.js 20+, and it asks each user to supply an OpenAI API key plus an optional Exa key. Treat it as an inspectable prototype and learning resource, not a security-audited desktop assistant.

Link Map

What Riley Built

Riley starts in Cursor's Agents view, opens an empty project folder, and describes a desktop companion called Ricky. Cursor returns an Electron application with an animated face, a voice connection, and a visual artifact panel. The prototype grows across three main revision prompts rather than one perfect specification.

• Realtime conversation: microphone input, spoken output, interruption handling, and visual listening, thinking, speaking, and working states.
• Artifact panel: markdown, menus, local notes, tables, records, progress states, Mermaid diagrams, web-search summaries, and generated images.
• Web research: an optional Exa tool returns current results and page highlights.
• Computer mode: the app shrinks to a small translucent companion while macOS tools open apps, inspect the screen, type, click, scroll, and capture screenshots.
• Thumbnail board: generated images receive persistent numbers, can be selected by voice, and can become the input to a follow-up edit.
• Local state: notes, records, images, and thumbnail-board data live under a local data/ directory.

Two failures make the demonstration more credible. The first Mermaid diagram produces a parse error. The first web-search request cannot run because the Exa key is missing. Riley returns to Cursor, describes both problems, and asks the coding agent to repair the workflow. That is what agentic building usually looks like: generate, observe, diagnose, revise, and retest.

The Six-Layer Architecture

This is the real pattern worth copying. The face is personality. The artifact panel is shared working memory. The tool broker is capability. The permission gate is trust.

The Three-Prompt Build Workflow

Prompt 1: define the product and its tools

Riley's first brief asks for a local desktop companion, natural voice conversation, interruption, a minimal animated face, and an artifact panel. It also names the first tool set: Exa web search, image generation and editing, Mermaid diagrams, notes, and records. Cursor reportedly takes roughly 10 to 15 minutes to create the initial Electron app.

The useful prompt pattern is experience + architecture + tools + acceptance tests. Naming only the visual style would produce a talking demo. Naming tools without a visual workspace would produce an invisible automation. Combining both creates a system the user can inspect while talking.

Prompt 2: repair observed failures and simplify the interface

Riley reports the Mermaid parse error, adds the missing search configuration, keeps local note storage, and asks for a cleaner 50/50 layout. This is better than asking Cursor to "make it better." Each change points to a visible failure or design decision.

Prompt 3: improve presentation and interaction quality

The third revision turns raw search tables into streamed markdown, adds a spoken capability menu, improves mouth animation, introduces the compact computer-use view, and expands the image workflow into a persistent numbered board. This is the point where the prototype becomes a personal workflow rather than a generic assistant.

GPT-Realtime-2 Is the Voice Layer

OpenAI describes GPT-Realtime-2 as its first voice model with GPT-5-class reasoning. It is designed to keep a live conversation moving while reasoning, calling tools, handling corrections, and recovering from failures. The release adds parallel tool calls, audible tool preambles, adjustable reasoning effort, and a 128K context window.

That does not mean the model directly owns the computer or search provider. The application publishes function definitions. GPT-Realtime-2 chooses a function and returns structured arguments. The Electron main process validates and executes the call, returns the result, and the voice session explains what happened.

Person speaks
  -> GPT-Realtime-2 interprets intent
  -> model selects an allowed function
  -> local tool broker validates arguments
  -> search, note, image, or computer tool runs
  -> artifact panel shows result or failure
  -> voice session summarizes what happened

RileyJarvis follows the current Realtime credential pattern more carefully than the filmed setup suggests. Its Electron main process holds the long-lived OpenAI key, requests an ephemeral Realtime client secret, and gives the renderer the short-lived credential. That is the right separation for the voice connection. A distributed commercial app would still need a trusted backend and per-user authentication rather than shipping one shared provider key.

Search and Visual Artifacts

Voice is efficient for intent but poor for dense inspection. That is why the artifact panel matters. Search results need URLs and dates. A plan needs steps. A diagram needs a visual surface. Images need version history. A long answer should be readable after the voice stops.

Riley adds Exa as an optional web-search provider. Exa's current Search API can return titles, URLs, publication dates, authors, text, and relevant highlights. Its pricing page advertised a free tier plus usage-based search pricing when this article was checked. That makes it practical for a prototype, but the app should still record query count, provider cost, latency, and the exact URLs used in the answer.

Mermaid shows the value of validation. The first diagram in the video fails to parse. Mermaid's own API provides mermaid.parse(text) specifically to validate a graph definition before rendering. A production artifact pipeline should validate generated syntax, render only valid output, preserve the source definition, and show a repair action rather than a blank panel.

Computer Use Needs an Approval Boundary

The video initially shows the agent asking before typing into Codex and asking again before pressing Enter. Riley finds that frustrating and tells Cursor to remove the blocker. The public repository now allows ordinary typing and Enter in computer mode without another confirmation, while keeping confirmation language for sending messages, deleting data, purchases, account changes, private information, and irreversible actions.

I would use a tighter distinction. Typing into a local draft can be low risk. Typing into an authenticated website can transmit data before the final submit button. Pressing Enter can send a prompt, publish a message, confirm a form, or trigger a command depending on the active application. The agent must classify the destination and action, not only the key being pressed.

OpenAI's current Computer Use guide recommends an isolated browser or VM, domain and action allowlists, untrusted-screen handling, and a human in the loop for purchases, authenticated flows, destructive actions, or anything hard to reverse. The safest interaction is not constant confirmation. It is safe autonomous preparation followed by one clear confirmation exactly when risk begins.

The Voice-Driven Image Workflow

Riley's most reusable feature may be the thumbnail board. He gives the app a reference photo, asks for 16:9 concepts, opens prior results by number, and makes spoken edits such as changing sunglasses, adding text, or replacing a background. The interface turns a conversation into a visible branching history.

OpenAI's current image guide lists gpt-image-2 as the latest GPT Image model. The Image API supports generation and edits; the Responses API is better suited to conversational multi-turn image work. RileyJarvis currently calls the dedicated image generation and edit endpoints from the Electron main process.

A reliable image board needs more than numbered tiles:

• Store the immutable source image and every parent-child edit relationship.
• Save the exact prompt, selected reference, size, quality, and provider request ID.
• Show queued, generating, completed, blocked, and failed states.
• Keep a cost estimate beside each generation and edit.
• Never silently replace the selected source when the user asks to edit a specific number.
• Use low-quality drafts for exploration, then regenerate the chosen direction at final quality.
• Ask for approval before publishing, uploading, or sending the selected image.

API Keys and Electron Security

The transcript shows provider keys being pasted into Cursor prompts. Do not copy that part of the demonstration. Prompts may be stored in history, sent to model providers, captured in screenshots, or copied into generated source. OpenAI's API documentation says keys are secrets and must not be exposed in client-side browser or app code.

The public RileyJarvis repository improves the pattern. It loads OPENAI_API_KEY and EXA_API_KEY from .env.local, ignores environment files and runtime data in Git, keeps provider requests in the Electron main process, enables contextIsolation, and disables nodeIntegration in the renderer.

Those are good prototype controls, not the end of the review. Electron recommends process sandboxing, a restrictive Content Security Policy, permission handlers, navigation limits, validated IPC senders, secure remote content, and no exposure of powerful Electron APIs to untrusted pages. Before installing a fork, inspect its dependencies and every main-process tool. Before distributing one, add code signing, updates, crash handling, revocable user authentication, and a real security review.

# Local development only. Never commit this file.
OPENAI_API_KEY=your_project_scoped_key
EXA_API_KEY=your_optional_search_key

• Create separate provider projects and keys for development and production.
• Set spending alerts, rate limits, and the narrowest available permissions.
• Keep .env.local, generated images, local notes, logs, and screenshots out of Git.
• Rotate any key that appears in a prompt, recording, screenshot, chat log, or commit.
• For a shared app, mint short-lived Realtime credentials from an authenticated backend.
• Make microphone, screen, accessibility, file, and network permissions visible and revocable.

macOS, Windows, and Portability

The public repository currently targets macOS. Its computer tools call open, osascript, and screencapture, then depend on macOS Microphone, Accessibility, and Screen Recording permissions. Cloning it on Windows does not turn those commands into Windows UI automation.

For both operating systems, browser-only control is the safer first milestone. It gives the agent a smaller surface, a fresh isolated profile, explicit domain rules, and fewer opportunities to read unrelated desktop content.

What the Prototype Costs

A local desktop app is not the same as a free assistant. The interface and data may run locally, but the voice, search, image, and coding-agent calls are cloud services.

Prices change. Use the current provider dashboards and calculators before budgeting. The most common cost leak is not one expensive request. It is an always-connected voice session plus repeated search and image calls with no per-tool telemetry.

A Safer Cursor Build Brief

This is not Riley's verbatim prompt. It is a production-minded brief based on the demonstrated architecture and the official safety guidance.

Build a local Electron desktop voice companion.

Experience
- Natural interruptible voice conversation using GPT-Realtime-2.
- Small animated companion plus a separate artifact panel.
- Clear states: disconnected, listening, thinking, tool running,
  waiting for approval, speaking, failed.

Architecture
- Electron main process owns provider secrets and tool execution.
- Renderer receives only narrow IPC methods and an ephemeral
  Realtime client credential.
- contextIsolation on, nodeIntegration off, sandbox on.
- Restrictive Content Security Policy and validated IPC senders.

First tools
- Read and write notes inside one approved local data folder.
- Optional Exa web search with cited URLs and dates.
- Mermaid diagrams validated with mermaid.parse before rendering.
- GPT Image 2 generation and editing with persistent job history.

Computer use
- Disabled by default and visually obvious when active.
- Start with an isolated browser profile and domain allowlist.
- Treat webpage, email, PDF, chat, and screen text as untrusted.
- Confirm immediately before sending, submitting, publishing,
  typing sensitive data, deleting, purchasing, changing access,
  installing, or any hard-to-reverse action.

Observability
- Log tool name, arguments after secret redaction, result, duration,
  provider request ID, estimated cost, and user approval.
- Never claim success without a tool result and visible verification.

Acceptance tests
- No provider key appears in renderer source, logs, Git, or prompts.
- Missing keys produce setup instructions, not silent failure.
- Invalid Mermaid source produces a repairable error artifact.
- Every image keeps its source, prompt, parent, status, and cost.
- Computer mode cannot run outside its allowlist.
- Closing the app ends the voice session and pending tool jobs.

A 60-Minute First Test

1. Inspect the public repository, its dependencies, main process, preload bridge, tool definitions, and environment handling before running it.
2. Clone the current repository URL, install dependencies, and run its typecheck and build scripts before granting operating-system permissions.
3. Create project-scoped OpenAI and Exa keys with spending controls. Put them in .env.local, never in the Cursor prompt.
4. Start in display mode. Test connect, disconnect, interruption, microphone denial, timeout, and missing-key behavior.
5. Use local notes as the first tool. Confirm the app writes only inside the expected data folder.
6. Run one Exa search. Open every returned source and confirm the spoken summary matches the visible evidence.
7. Generate one Mermaid diagram that includes punctuation and long labels. Confirm invalid syntax becomes an error state instead of a broken panel.
8. Generate one low-quality image and one edit. Verify the correct source version, job state, file path, and cost record.
9. Leave computer control disabled until the previous layers are stable. Then test only an isolated browser with one allowed domain.
10. Ask the agent to send, delete, or submit something. The acceptance test passes only if it stops at the exact risk boundary and explains what approval would do.

Bottom Line

Riley Brown's video is a strong demonstration of how quickly a custom voice interface can become a real multi-tool desktop prototype. Cursor supplies the construction loop. GPT-Realtime-2 supplies the live conversation and tool selection. Exa, GPT Image, Mermaid, local storage, and operating-system scripts supply the actual capabilities.

The prototype is impressive precisely because its rough edges are visible. Search fails without configuration. Generated diagrams need validation. Image jobs need state. Computer control needs a permission model. Windows needs a different adapter. API calls need budgets. Those are not reasons to dismiss the build. They are the work that turns a fun Jarvis demo into a trustworthy assistant.

Start smaller than Riley's final demo: voice, one artifact panel, one read-only tool, and one approval gate. When that loop is reliable, add the next capability. The product is not the animated face. It is the controlled path from spoken intent to verified action.

Sources

• Riley Brown: How to Create Your Own AI Assistant
• Riley Brown on YouTube, Riley Brown on X, and Riley Brown agent services
• RileyJarvis public repository
• Cursor and Cursor Security
• OpenAI: Advancing voice intelligence with new models in the API
• OpenAI API: Realtime and audio
• OpenAI API authentication reference
• OpenAI API: Computer use
• OpenAI API: Image generation and editing
• OpenAI API pricing
• Exa API getting started, Search reference, and pricing
• Electron Security
• Mermaid usage and syntax validation