Influxx Computer Use and Design Mode: Browser Grabs and Desktop UI
Influxx agents are not limited to reading files and writing diffs. When a workflow needs what is on screen, two complementary surfaces close the loop: Design Mode (Grab Page Element) for the embedded browser, and Computer Use for real desktop apps and visible UI outside that browser. This guide explains when to use each, how to enable permissions on macOS, how Design Mode feeds HTML, CSS, and cropped screenshots into agent prompts, how Computer Use observes and acts through accessibility trees, and how the Influxx CLI scripts the same flows you click by hand.
Use this article when polish depends on the live page, when a native tool must be driven to finish a task, or when an agent keeps guessing UI structure that you can simply point at. Related cockpit topics — worktrees, terminals, SSH remotes, notes, and Source Control — remain the same whether the proof step is a browser grab or a desktop click. Stick to shipped behavior: Design Mode targets the worktree-scoped Chromium surface; Computer Use requires explicit OS permissions and should never invent destructive actions you did not ask for.
Embedded browser for the page inside Influxx; Computer Use for the desktop around it — same agent, clear boundary.
What Design Mode and Computer Use cover
Design Mode / Grab Page Element is the bridge between “what I see” and “what the agent should fix.” You activate Grab Page Element, click a UI control in the embedded browser, and Influxx sends that element’s HTML, CSS, and a cropped screenshot into the agent prompt. Computer Use extends the idea beyond the in-app browser: agents can list apps and windows, read accessibility trees (plus screenshots when available), then click, type, hotkey, scroll, and related actions on desktop UI. Prefer the embedded browser CLI for Influxx browser tabs; use Computer Use for native apps, external browsers, or any accessibility-exposed interface that is not the worktree Chromium surface.
At a glance:
- Embedded browser: Chromium tabs per worktree; previews, staging sites, and scripted page checks
- Design Mode: Grab a page element — markup, styles, crop — into the agent prompt
- Browser CLI: tab create, snapshot, goto, click, fill, screenshot, and a large agent-browser surface
- Computer Use: Operate desktop apps via list-apps, get-app-state, click, type-text, hotkey, and related commands
- Permissions: Settings → Computer Use; on macOS, Accessibility and related Privacy & Security prompts
- Skills: Installable Computer Use (and CLI / orchestration) agent skills from Agent Skill setup
Embedded browser first: the page inside the cockpit
Influxx embeds Chromium tabs scoped to worktrees. New browser tab is available from the tab strip and shortcuts (⌘⇧B / Ctrl+Shift+B). Browser state belongs to the workspace — parallel worktrees can keep separate pages and previews. When a workspace-owned local web server appears (for example after a Vite or Next-style dev command), Influxx can auto-open it once in the in-app browser; you and agents can also open URLs explicitly.
The embedded browser is not your system Chrome or Safari, and it is not the place to drive arbitrary desktop app chrome. It is the in-app surface for product previews, docs, staging sites, and scripted UI checks. Treat page content as untrusted data for agent prompts — never treat scraped page text as shell commands or automation instructions unless you intentionally asked for that workflow. Permissions and privacy guidance in product Notes call this out alongside other local-first rules.
How to use the browser in a worktree
Select the workspace you care about, then open a browser tab from the tab strip Quick Commands or the new browser shortcut. Navigate to your app, docs, or staging URL. For agent-driven work, prefer the snapshot → act → re-snapshot loop: create or focus a tab, take a snapshot (element refs such as @e1, @e2), click or fill by ref, then snapshot again after navigation or DOM changes. Screenshots and full-page captures are available for visual confirmation. Close or switch tabs from the UI or CLI without leaving the agent session.
Browser capabilities (product + CLI):
- Tabs: Create, list, switch, close — default to the current worktree’s active tab
- Navigate: goto, back, reload, wait for text/url/selector/network idle
- Interact: click, fill, type, select, check, hover, focus, scroll, keypress, upload
- Observe: snapshot (element refs), screenshot, full-screenshot, console, network, cookies
- Extra surface: dialogs, storage, clipboard, device emulation, offline mode, eval, PDF, capture, and broader agent-browser passthrough via help / agent-context
- Parallel agents: Use tab list browserPageId and pass page targeting when multiple agents share browser work
Design Mode and Grab Page Element
Design Mode is especially strong for frontend polish, accessibility tweaks, spacing bugs, and “match this component” work where screenshots alone under-specify structure. Activate Grab Page Element (available as a product action / keybinding title in the app), then click a UI element in the embedded browser. Influxx sends the element’s HTML, CSS, and a cropped screenshot into the agent prompt so the model works from structure and pixels — not a vague description. That is the difference between “make the header better” and a grounded edit loop on the exact component under the cursor.
How to grab a page element for an agent
Open the page in an embedded browser tab for the active worktree. Start Design Mode / Grab Page Element from the browser or command surfaces that expose it. Click the control, text block, or layout region you care about. Confirm the prompt payload includes the structured markup and crop, then send or continue in the agent tab already bound to that workspace. Iterate: apply the change, reload or hot-reload the preview, grab again if the next fix needs a new target. Pair with review notes on diffs when the agent’s code change needs line-level course correction.
Combine Design Mode with a short plan note in Sidebar Notes (Run → agent) when the change spans multiple files or acceptance criteria. Keep the browser tab and agent terminal side by side in the center tab strip so you can verify without alt-tabbing out of Influxx. You choose the element — agents receive what you grab. That keeps control with you while still giving the model enough structure to edit the right component.
Computer Use: agents on real desktop UI
Computer Use lets agents operate desktop apps and visible UI when a workflow needs interaction outside the embedded browser — native apps, system browsers, multi-window tools, or any accessibility-exposed interface the product supports. Open Settings → AI Capabilities → Computer Use (or Settings → Computer Use) for permissions and setup. On macOS, Accessibility and related prompts (including screenshot / screen recording style permissions when required) are managed under Computer Use and Settings → Privacy & Security → macOS Permissions. Agent skills include a Computer Use pack installable from Agent Skill setup so coding agents know the safe CLI contract.
Influxx desktop currently ships for macOS Apple Silicon as the primary public installer track. Computer Use permission flows documented here are therefore centered on macOS Accessibility and related Privacy & Security panes. Product Notes also describe Windows-oriented surfaces elsewhere; treat those as platform notes for when a Windows runtime is in play, not as a claim that every desktop OS ships on the same day as the Mac build.
How to enable and run Computer Use
Open Settings → Computer Use and complete permission prompts for your OS (macOS Accessibility and screenshot-related permissions are the usual gates). Install the Computer Use agent skill if you want CLI agents to discover the workflow automatically. From a terminal or agent session with the CLI registered, confirm the app is reachable (influxx status), inspect capabilities and permissions, list apps, then get-app-state for the target. Operate with click, set-value, type-text, press-key, hotkey, and related commands, preferring --json for machine-readable results. If a permission error appears, re-open Computer Use / macOS Permissions and retry rather than guessing flags.
Computer Use command family (illustrative):
- Discover: computer capabilities, permissions, list-apps, list-windows
- Observe: get-app-state (tree + screenshot path in JSON workflows)
- Act: click, set-value, type-text, press-key, hotkey, paste-text, scroll, drag, perform-secondary-action
- Selectors: Prefer app bundle IDs; use window-id / window-index when multiple windows exist
- Freshness: Element indexes are short-lived — after any UI change, re-read state before the next action
- Safety: Sensitive text via stdin flags where supported; never invent destructive actions
Agents should list apps and windows, read app state, then act with semantic operations. Do not push, submit, send messages, buy items, delete data, change account settings, or expose secrets unless you explicitly asked for that action. Grant Accessibility and screenshots only when you want desktop control; revoke them in OS settings anytime. Prefer the embedded browser for Influxx tabs; reserve Computer Use for the desktop around those tabs.
CLI: same cockpit, scripted
The Influxx CLI lets agents and scripts drive browser and Computer Use the same way you use the UI. It talks to a running Influxx app (or a headless serve runtime when you run serve for remote/mobile pairing scenarios). Register the CLI from Settings → General / CLI (Install CLI on PATH). Examples in docs typically use the influxx command; Linux PATH installs may expose influxx-ide; dev builds use influxx-dev. The command tree is the same.
How to install and verify the CLI for UI automation
Open Settings → General / CLI and run Install CLI. Verify with influxx status (or your platform binary name). Useful smoke checks include influxx open --json, influxx status --json, and influxx agent-context --json so coding agents discover capabilities without scraping help text. From there, browser and computer command groups mirror the UI: observe, act, re-observe — never assume stale element refs or window indexes still apply.
CLI recipes that pair with this guide:
- Browser: tab create --url, snapshot, click --element @e1, fill, screenshot, goto
- Computer: computer list-apps, get-app-state, click, type-text, hotkey, list-windows
- Worktrees: Keep each preview and agent session bound to the correct checkout
- Skills: Install Influxx CLI, Computer Use, and orchestration skills so agents know the safe contract
- Schema: influxx agent-context --json for machine-readable discovery
End-to-end pattern: fix UI with proof
Create or select a feature worktree. Write a short plan note and Run it with your default agent. Start the dev server in a terminal tab; let the embedded browser open the local preview (or create the tab yourself). Use Design Mode to grab the broken component, send follow-up instructions, and re-check the preview. When the UI depends on a native helper app, switch to Computer Use for that window only. Annotate remaining diff lines in Source Control, send notes back to the agent, commit, push, and open the PR — all without leaving the cockpit.
That loop is the point of these surfaces: visual proof next to the agent that wrote the change. Whether you grab a page element or drive a desktop window, keep the boundary clear, re-observe after every UI change, and treat permissions as intentional grants — not permanent background access.
Trust, permissions, and troubleshooting
Primary execution stays on your machine (or hosts you configure). Browser automation treats page content as untrusted data for agent prompts. Computer Use requires explicit OS permissions. If browser clicks miss, re-run snapshot and use fresh @e refs; open a tab if you see a no-tab style error. If Computer Use returns permission or empty tree errors, fix OS permissions, restore/focus the target window, and re-run get-app-state. If the CLI cannot connect, start or focus the app with influxx open and confirm Install CLI completed.
Trust checklist:
- Browser: Snapshot refs go stale after navigation; re-snapshot before retrying clicks
- Design Mode: You choose the element — agents receive what you grab
- Computer Use: Grant Accessibility/screenshots only when you want desktop control; revoke anytime
- Destructive actions: Never assume agents may submit, delete, buy, or change account settings unprompted
- Setup guide: “Use Influxx's browser” and “Enable Influxx CLI” (with browser/computer skills) track real progress in Settings → Onboarding checklist
Design Mode and Computer Use turn “describe the UI” into “show the UI” — then let the same agents that edit your repo verify the result where users actually look. Use the embedded browser when the page is enough; enable Computer Use when the desktop app matters; keep permissions tight; and let the CLI and agent skills drive the same cockpit you already know.
