Direct Answer
The secret to an AI agent workforce is not more agents. It is moving management into the system. Goals, context, permissions, routing, review thresholds, logs, and escalation rules should carry the routine coordination. The human stays responsible for priorities, sensitive decisions, exceptions, and the quality bar.
Allie K. Miller describes a personal workforce of 34 agents led by Simon, an AI chief of staff, with six directors and task-specific agents below them. The number is interesting, but her more transferable rule is better: increase the breadth of work while holding the risk tier steady. Her agents may identify more opportunities, but she still reviews outbound email and other consequential actions.
Watch Greg Isenberg and Allie K. Miller
Episode credit: Greg Isenberg. Guest and framework credit: Allie K. Miller. Watch the original video on YouTube.
From Managing Agents to Enabling Them
Allie compares her role to an SVP sitting three levels above execution. She is not assigning every task. She creates infrastructure, defines goals, and waits for decisions or exceptions that require judgment. That is a useful aspiration, but only after the system can answer six questions:
- Objective: What measurable outcome is the workforce pursuing?
- State: What does it know, where did that information come from, and when was it updated?
- Authority: What may each agent read, draft, change, send, buy, or delete?
- Routing: Which agent owns the next step, and why is a handoff needed?
- Escalation: Which uncertainty, cost, conflict, or external side effect requires a person?
- Evidence: What artifact, source, test, or log proves the work was done?
Without those answers, "enablement" can become unsupervised delegation. With them, it becomes a control plane that lets a small team supervise more parallel work without reviewing every intermediate thought.
Why "Do Smart Things" Is Not Really a Three-Word Prompt
Allie's memorable instruction is "do smart things." The transcript also explains why it works: her workforce can inspect business and personal goals, context documents, meeting transcripts, email, calendar, Notion, Stripe, Supabase, GitHub, and a growing diary. It has a chief-of-staff router, defined roles, and an established review pattern.
The true prompt is therefore closer to this:
Vagueness at the final instruction can be productive when the foundation is precise. Vagueness everywhere is just missing specification.
| Proactivity level | Agent behavior | Appropriate authority |
|---|---|---|
| 1. Observe | Reports a problem. | Read-only access. |
| 2. Diagnose | Finds likely causes and evidence. | Read plus analysis tools. |
| 3. Propose | Offers options, tradeoffs, and a recommendation. | Drafting and sandbox tools. |
| 4. Prepare | Builds the reversible change and presents the diff. | Branch, draft, staging, or pending state. |
| 5. Execute | Acts, verifies, reports, and has a recovery plan. | Only low-risk, proven, monitored actions. |
The five-level framing is credited in the episode to Alex Lieberman. The essential governance addition is that proactivity and authority are separate. An agent can reason at level five while still waiting for approval before an irreversible action.
Make the Company Queryable Without Making Everything Accessible
Meetings, email, Slack, and project tools do not capture every decision. Allie closes that gap with a daily dictated diary. The prompt asks her to record what changed, what she learned, what remains uncertain, and what the agents would otherwise miss. More than 80 entries had accumulated when the episode was recorded.
A useful diary entry should be structured enough to retrieve:
- Date, project, people, and source
- Observation versus decision versus hypothesis
- What changed and why
- Open questions, commitments, and due dates
- Who may access the note
- When the information should expire or be reviewed
Do not interpret "queryable" as "every agent reads everything." Separate personal notes, client data, credentials, finance, legal, HR, and public knowledge. Use least privilege, retention rules, source citations, access logs, and deletion procedures. A context system should reduce re-explanation without turning one compromised agent into a route across the whole company.
Design Roles Around Missing Capabilities
Simon's six directors roughly map to business functions, but Allie's most interesting roles do not. Phoebe is a "chief dreaming officer" that asks how work could become ten times more ambitious. Toby watches the workforce itself, records friction, and identifies repeated corrections or missing access.
This suggests a better rule than copying a human org chart: add an agent when it owns a distinct transformation or control.
| Role pattern | What it owns | Why it may deserve separation |
|---|---|---|
| Router | Classifies requests, selects workflows, and tracks state. | Keeps the human from coordinating every handoff. |
| Specialist | Uses one domain, toolset, or output contract. | Needs focused context or different permissions. |
| Skeptic | Challenges assumptions, evidence, and edge cases. | Independent checking reduces self-approval. |
| Dreamer | Generates higher-ambition alternatives. | Exploration can remain separate from execution. |
| Watchdog | Observes friction, conflicts, duplication, and drift. | Oversight needs broad visibility but little write authority. |
| Operator | Executes a bounded, tested action. | Write permissions and rollback need tight isolation. |
A name and a Markdown biography do not create a useful agent. A separate role earns its place through a different objective, input, tool, permission boundary, model, schedule, or independent evaluation.
The Four-Stage Workforce Ladder
| Stage | Build | Promotion test |
|---|---|---|
| 1. One bounded agent | One recurring task, approved sources, explicit output, no external side effects. | Five accepted runs with measurable time or quality improvement. |
| 2. One proactive agent | A schedule or trigger that proposes useful work from current goals. | Most proposals are relevant; false positives and review time remain tolerable. |
| 3. Two-agent handoff | A router and specialist, or producer and independent checker. | The handoff improves accepted output enough to justify latency and cost. |
| 4. Multiplayer workforce | Shared state, parallel work, permissions, budgets, logs, escalation, and a human-facing control surface. | The system handles multiple real workflows without lost state, duplicate work, or unowned failures. |
Allie routes cheaper models to many sub-agents and reserves more capable models for difficult work. Apply that after measuring quality: classify by task risk and complexity, record cost per accepted output, and maintain a fallback when a smaller model fails. "Cheapest model" is not routing; it is a price preference without an acceptance test.
Start With AI Watchdogs
Watchdogs are a strong first proactive use case because they can create value while remaining read-only. Allie proposes watching Slack for duplicate work, calendars for conflicts, meetings for disagreements, and the workforce for repeated corrections or missing access.
- Choose one observable failure. Example: two teammates unknowingly prepare the same artifact.
- Define evidence. Require links to both tasks, owners, dates, and the overlapping deliverable.
- Set a confidence threshold. Low confidence goes into a digest; high confidence creates a review item.
- Keep the first version read-only. It may notify, but it may not cancel meetings, close work, or message clients.
- Measure precision. Track true positives, false positives, missed incidents, reviewer minutes, and value saved.
A watchdog with broad visibility can become a surveillance risk. Tell the team what it reads, what it infers, how long records remain, who can inspect them, and how people correct a bad conclusion. Meeting disagreement detection should support follow-up, not score personalities.
Build the Factory, Then Reuse It
For the AI First Index, Allie says the team invested in reusable product primitives: login, payments, social sharing, newsletters, and related foundations. Once one product earned revenue, adjacent products could reuse the factory.
The same principle applies to agent workforces. Reusable primitives include:
- Identity, role, and permission templates
- Approved connectors and credential isolation
- Shared task state and artifact storage
- Source and citation contracts
- Model routing, budgets, retries, and timeouts
- Human approval, audit logs, rollback, and shutdown
- Evaluation sets and accepted-output scorecards
There is an important order-of-operations caveat. Build the smallest product that proves demand before building a generalized factory. Extract a primitive after the second real use, not merely because it might be reusable. Otherwise infrastructure becomes an elegant way to delay contact with customers.
Greg and Allie also make a useful market distinction. Enterprise buyers still value ownership, support, integration, procurement, and a vendor who is accountable when something breaks. Consumer software has a lower implementation barrier, so taste, trust, community, and distribution can matter more. Neither market eliminates software; each changes where defensibility sits.
Measure Outcomes, Not Agent Headcount
| Metric | Question it answers |
|---|---|
| Accepted-result rate | How often does the output pass without substantive correction? |
| Human attention per result | Did orchestration remove review work or move it elsewhere? |
| Cost per accepted result | Are model, tool, retry, and operator costs economically useful? |
| Escalation precision | Does the workforce surface decisions that actually need a person? |
| Proactive hit rate | What share of self-generated tasks create accepted value? |
| Watchdog precision | How many alerts are real, useful, and timely? |
| Permission exceptions | How often does work request access outside its role? |
| Recovery rate | Can failed actions be stopped, rolled back, and explained? |
Thirty-four agents can be powerful, wasteful, or merely decorative. The scorecard tells you which.
A Safe First Step You Can Finish in Three Hours
- Write one outcome. Example: deliver a verified weekly competitor-change brief that saves two hours.
- Create a context pack. Add the goal, audience, approved sources, examples, definitions, and a short decision log. Exclude credentials and unrelated data.
- Ask the model to interview you. Have it identify missing context, failure modes, the output schema, and where approval belongs.
- Define a read-only agent. It may retrieve, compare, draft, and cite. It may not send, publish, buy, delete, or modify source systems.
- Give it one proactive trigger. Run on a schedule or when a known source changes. Send the result to a review queue.
- Score the first run. Record accuracy, useful findings, false positives, missing evidence, cost, latency, and review time.
Video Chapters
| Time | Topic |
|---|---|
| 00:00 | Introduction |
| 02:29 | Become a great agent manager |
| 04:49 | The three-word prompt |
| 08:12 | The pyramid of proactivity |
| 12:23 | Making the company queryable |
| 19:14 | How to design an AI workforce |
| 22:25 | AI as a watchdog |
| 24:56 | Startup opportunities |
| 26:29 | Build the factory, then the product |
| 30:09 | The SaaS question |
| 34:53 | Consumer software as art |
| 37:12 | High-value bottlenecks |
| 44:56 | Closing thoughts |
Final Verdict
Allie K. Miller's system is valuable because it exposes the work behind proactive agents: a maintained context layer, quarterly goals, unusual but purposeful roles, model routing, watchdogs, human escalation, and months of iteration.
The transferable lesson is not to reproduce Simon and 33 colleagues. Build one agent that reliably completes a bounded job. Let it propose work before it performs work. Add a second agent only when separation improves evidence, cost, speed, or control. Turn repeated coordination into shared infrastructure. Keep consequential actions behind approval until the system earns broader authority through observed performance.
That is how an agent workforce underpromises and overdelivers: ambition in the search space, restraint at the action boundary.