How to Manage a Team of AI Agents: Ryan Carson's Cloud Software Factory Playbook

Direct Answer

A world-class agent operator does not keep the largest number of sessions running. They keep a small portfolio of well-scoped tasks moving through isolated environments, collect evidence from each one, and make the few decisions that only a person should make. The core loop is simple: scope, delegate, verify, decide, and learn.

Ryan Carson's interview with Greg Isenberg shows the high-output version of that system. Ryan says he runs five to ten cloud agents, handles about half his work from a phone, and ships 22 to 40 pull requests on busy days while building Untangle. Those are his operating results, not promises. The transferable value is the architecture beneath them: fresh cloud environments, pinned priority threads, scheduled checks, child sessions for triage, model routing, and controlled production access.

Watch: Most Valuable Skill of 2026, Managing AI Agents

Video credit: Greg Isenberg with entrepreneur and Untangle founder Ryan Carson. Follow Greg on X and Ryan on X.

What Is Confirmed, and What Is Ryan's Field Report?

One important correction to the episode's cloud-only rhetoric: local development is not obsolete. Local agents remain better for private or regulated data, hardware-bound builds, offline work, low-latency pairing, and tasks where uploading a repository or production context is inappropriate. The mature system routes work between local and cloud environments based on risk and task shape.

The New Job: Manager of Agents

Ryan's central idea is that knowledge workers are becoming managers of agents. That does not mean writing a vague goal and waiting for finished work. It means translating intent into a contract an agent can execute and a reviewer can evaluate.

This is why good agent management makes people more technical rather than less. Operators need enough understanding of branches, databases, migrations, environments, authentication, observability, and tests to judge whether an agent's confidence is deserved.

Cloud Agents Versus Local Agents

Devin describes each session environment as a fresh Linux virtual machine created from a reusable snapshot with repositories, dependencies, and tools already configured. That isolation is the main operational advantage. Independent tasks do not compete for the same worktree, ports, database state, or package installation.

Do not parallelize dependent work just because you can. Five agents editing the same subsystem can create merge conflict, duplicated implementation, and inconsistent architecture. Build a dependency map first. Parallelize only the branches that can be reviewed and integrated independently.

Ryan's Management System, Made Reusable

Ryan pins the most important threads, lets lower-risk work continue in the background, and checks his priority queue on an approximately 25-minute cadence. The useful pattern is not the exact timer. It is batching decisions so the operator does not spend the entire day refreshing agent sessions.

1. Keep a maximum of three priority outcomes. These are the changes that matter today.
2. Separate priority work from maintenance. Bugs and small cleanup can run in another queue with stricter permissions.
3. Use scheduled review windows. Check the queue when you can make a complete decision, not every time an agent emits progress.
4. Require a decision packet. Every agent should return what changed, evidence, risk, cost, unresolved questions, and the requested decision.
5. Stop starting work when review backs up. A growing decision queue is a work-in-progress signal, not a reason to launch more agents.

Do Not Optimize for Pull-Request Count

Ryan's 22 to 40 daily PRs are striking, but PR count is an activity metric. It can rise because work was decomposed well, or because changes became fragmented and noisy. Google Cloud's summary of the DORA Four Keys balances speed with stability:

• Deployment frequency: how often accepted value reaches users.
• Lead time for changes: how long useful work takes to reach production.
• Change failure rate: how often releases create degradation or require remediation.
• Time to restore service: how quickly the system recovers after failure.

Add three agent-specific measures: accepted result rate, human review minutes per accepted change, and cost per accepted change. Together, these reveal whether more agent activity is creating leverage or merely producing more material to inspect.

Three Automations Worth Building

1. The end-to-end signup test

Ryan says a Devin playbook browser-tests the complete signup flow three times a week, records evidence, and triggers a child triage session when it finds a failure. He estimates the run costs about $60 in tokens. That is a creator-reported cost, not current Devin pricing.

The responsible version runs against staging first, uses a dedicated synthetic account, records each step, checks analytics and database side effects, and opens an issue when evidence fails. A production smoke test should be narrower, read-only wherever possible, and designed so repeated execution cannot create real charges, messages, or records that confuse customers.

2. The production watchdog

Every morning, Ryan's watchdog summarizes important customer events and links each observation to the relevant internal UI. This is a strong human-in-the-loop design because the agent compresses logs while the founder still uses judgment to notice what feels unusual.

Give the watchdog read-only access, redact personal data, define the customers or events that deserve attention, and require links back to source evidence. It should distinguish confirmed failures from unusual patterns and include an explicit "nothing material found" state. In legal, medical, financial, or employment contexts, summaries should minimize sensitive content and preserve access controls.

3. The self-improvement loop

Ryan's third automation samples conversations handled by Untangle's agent, Grace, grades them against a rubric, and opens a child session for low-scoring cases. He says this produces about three small fixes per day.

The safe architecture is: sample, redact, grade, explain, cluster, propose, test, and request review. Do not let an evaluator rewrite its own rubric or automatically ship policy, legal guidance, safety behavior, pricing, permissions, or customer communications. A self-improvement loop without an independent acceptance test can optimize for its own grader instead of the customer.

Production Safety: Better Than Pasting a Key

Ryan correctly keeps production write credentials away from ordinary agent sessions. In the interview, he describes retrieving a key from 1Password only when a production action is necessary. The safer production design goes one step further: do not paste a broad, long-lived key into the conversation at all.

1Password's developer guidance recommends scoped service accounts and runtime secret injection. GitHub environments can delay access to environment secrets until approval conditions pass, restrict deployment branches, and require reviewers. Combine those patterns:

1. Use read-only production access by default.
2. Create a separate machine identity for each automation.
3. Grant only the specific secret, resource, action, and environment required.
4. Prefer short-lived credentials and runtime injection.
5. Require a protected deployment environment and a non-agent reviewer.
6. Log secret access, actions, diffs, test evidence, and deployment identity.
7. Make revocation and rollback one step, not a recovery project.

For production databases, agents should usually generate a migration or administrative command for review rather than execute an open-ended write session. Destructive changes need backups, dry runs, row limits, transactions, and an explicit rollback path.

Model Routing and Token Economics

Ryan says an early version of his system spent about $20,000 in one month and argues that model routing is necessary once real engineering usage grows. His estimate of $5,000 per employee per month is a personal forecast, not a universal budget. Current Devin documentation says usage depends on task actions, browser and code execution, virtual-machine time, and networking. Self-serve plans can also draw on paid on-demand credits after included quota.

Measure the complete accepted task, not the price of one million tokens. A cheaper model that causes three retries can cost more than a stronger model that succeeds once. A frontier model used for every log summary is equally wasteful.

What Managing Agents by Phone Is Good For

Cloud agents make remote management possible because the work continues outside the operator's laptop. Devin is web-based, and OpenAI now documents Codex remote access from the ChatGPT mobile app for connected macOS hosts. The phone is useful for:

• checking a concise decision packet;
• answering one blocking question;
• approving a clearly bounded next step;
• reviewing screenshots, test status, and small diffs;
• pausing a session that is going off course.

It is a poor surface for reviewing broad migrations, security-sensitive changes, large diffs, complex visual behavior, or production-data operations. Mobile access should reduce latency on decisions, not normalize approving risky work without enough context. Avoid entering credentials on the move or from untrusted networks.

Copy-Ready Agent Operator Brief

OUTCOME
State one user or business result. Do not describe activity.

SCOPE
Repository, files, systems, and behavior that may change.
List explicit out-of-scope areas.

ACCEPTANCE EVIDENCE
Tests:
Browser or visual checks:
Performance or data checks:
Artifacts to return:

PERMISSIONS
May read:
May write:
May open a PR:
May not merge, deploy, send, spend, delete, or change production data.

BUDGET AND STOP CONDITIONS
Maximum model or compute spend:
Maximum runtime:
Maximum diff size:
Stop after two failed approaches or any unexpected permission request.

DECISION PACKET
Return:
1. What changed
2. Evidence
3. Remaining risk
4. Cost and runtime
5. Exact decision needed from the operator

This brief creates the management layer Ryan describes without requiring his exact tools. It works with Devin, Codex, Claude Code, Cursor, Factory, Amp, or an internal harness.

A 30-Day Adoption Plan

Week 1: One agent, one repository, one safe task

• Choose a repository with reliable tests and a reversible deployment.
• Create the operator brief and run two small tasks.
• Measure acceptance, review time, cost, and defects.
• Keep production credentials and deployment access out of scope.

Week 2: Two parallel sessions

• Choose independent tasks with no shared files or migrations.
• Use isolated branches or cloud environments.
• Create a decision queue and two review windows per day.
• Stop if integration or review time exceeds the saved implementation time.

Week 3: One read-only automation

• Build a production watchdog, log summary, documentation drift check, or staging E2E report.
• Require source links, explicit uncertainty, and a clean "no issue" state.
• Send failures to one visible channel and retain an audit trail.
• Do not auto-fix or auto-merge yet.

Week 4: Add bounded action and routing

• Allow the automation to open an issue or draft PR.
• Add deterministic tests and independent review.
• Route repetitive steps to a cheaper model.
• Set weekly budgets and compare cost per accepted change.

At day 30, expand only if lead time improved without worsening change failure, recovery time, security, or customer experience.

Build Reputation by Publishing Evidence

Ryan's final recommendation is to share what you learn publicly. The useful version is not posting inflated concurrency or revenue claims. Publish the task, constraints, failed attempt, evidence, cost, and what changed in your playbook. That attracts people who value judgment rather than spectacle.

A compact weekly field note can include one workflow, one screenshot or diagram, one measured result, one limitation, and one reusable template. X can be one distribution channel, but a durable article, repository, newsletter, or case study gives the work a longer life and makes the evidence easier to verify.

Video Chapters

Bottom Line

Ryan Carson's real advantage is not an eight-screen monitor or a specific agent vendor. It is a management design that keeps work isolated, makes priority visible, turns repeated tasks into playbooks, and escalates exceptions with evidence. The scalable unit is not the prompt. It is the verified handoff.

Start with two agents and one read-only automation. Track accepted outcomes, review load, failures, recovery, and cost. Add concurrency only when your tests, permissions, and decision process can absorb it. That is how a software factory becomes leverage instead of a faster way to create incidents.

Sources and Useful Links

• Most Valuable Skill of 2026: Managing AI Agents, Greg Isenberg with Ryan Carson.
• Ryan Carson: official biography.
• Untangle: company story and team.
• Devin introduction and documented limitations.
• Devin environment configuration.
• Devin guidance for task scope and parallel sessions.
• Devin scheduled sessions.
• Devin self-serve plans and spending controls.
• How Devin usage is metered.
• OpenAI Codex use cases.
• OpenAI: Codex remote access from the ChatGPT mobile app.
• 1Password: load secrets at runtime with scoped service accounts.
• GitHub: protected branches and required review.
• GitHub: deployment environments, reviewers, and secrets.
• Google Cloud: DORA Four Keys software-delivery metrics.