Computer Use

Computer Use lets an agent work in a native Windows application the way you would: it looks at the window, decides what to do, clicks and types, waits for the screen to settle, and looks again (v0.170.0).

look at the window -> decide -> click / type / press keys -> wait until the screen settles
      ^                                                                   |
      +-------------------------------------------------------------------+

It is ChatWalaʻau's own implementation, not a vendor feature, so it works with any chat model that can read images returned by tools: Azure OpenAI, OpenAI and Claude models. Models served through the Microsoft Foundry provider cannot use it -- that connector drops images in tool results, so the agent would be working blind.

Where it works

Computer Use operates the screen of the machine the ChatWalaʻau backend runs on, so it is available only when that machine is yours:

• the Desktop app, or chatwalaau / pnpm run dev:full on your own Windows PC;
• Windows 10 / 11, signed in (not locked, not running as a service);
• the server listening on 127.0.0.1 (the default), and the chat opened on that machine.

A server deployment, Docker, Linux or macOS never offers it. It works in chats with the built-in or a custom agent, in harness agents that include the Computer Use tools, and in a Live conversation started on this machine (its delegated work can operate the desktop). Microsoft Teams, the OpenAI-compatible API, scheduled jobs and workflows never get it.

Turning it on

Add to your .env (Desktop app: %LOCALAPPDATA%\ChatWalaau\desktop\profile\.env) and restart:

COMPUTER_USE_ENABLED=true
# Optional: where traces and the screenshot history go (default .computeruse).
COMPUTER_USE_DIR=.computeruse
# Optional: passwords the agent can type by NAME without ever seeing them.
COMPUTER_USE_SECRETS_FILE=C:\Users\me\secure\computer-use-secrets.json

The built-in agent gets the tools automatically. For a custom agent, pick the Computer Use tools in its tool picker. To keep one model away from it, set Computer Use to Not available on that model in App Settings > Model Offering Catalog.

Then ask, for example: "Open Notepad, write a short thank-you note in Japanese and save it as thanks.txt on the desktop."

How it works

Tool	What it does
computer_list_windows	Lists the open windows
computer_focus_window	Brings one window to the front, sizes it to 1920x1080, and locks it as the target
computer_capture_screen	A screenshot of the target plus its buttons, fields and other controls; can zoom into a region
computer_perform_actions	Runs one batch of steps, waits until the screen settles, checks the result and reports back
computer_get_active_window	Which window is in front, and whether a dialog is open
computer_wait_for_change	Waits for the application to finish something on its own
computer_abort	The agent ends the task and says why

Many steps per decision. The agent plans a whole step -- click the user name field, type, click the password field, type, press Enter, wait until Loading disappears -- and sends it in one go. Every round trip to the model costs seconds, so this is what makes Computer Use fast enough to be useful.

No fixed waiting. After acting, ChatWalaʻau watches the window and continues as soon as it has stopped changing (400 ms of calm by default), instead of sleeping a fixed time.

Only what is new. A new screenshot is sent only when the screen changed or something did not go as expected, and only the newest screenshot stays in the conversation (the history on disk is for you, see below).

Accurate clicks. The agent prefers the application's own controls (by name) over pixel positions, and pixel positions are mapped exactly -- on scaled displays and on a second monitor too.

No code. The agent can only describe steps (click, type, keys, scroll, drag, wait, if / repeat). It cannot run programs or scripts itself. Any window except ChatWalaʻau's own can be operated -- including a terminal, where typed text runs as a command -- so watch what it does.

Works with less, too. Clicking by screen coordinates, keys and typing always work. Reading the application's controls, pasting through the clipboard and detecting dialogs make it more precise, but the agent falls back to coordinates and typing when a desktop cannot provide them. This is also the contract for a future Computer Use provider built as a separate app or MCP server, in any language.

Watching it work

On the full-width chat, a small rounded window in the top-right corner shows what the agent is looking at, updated with every new screenshot while it works, and it goes away when the work is done. The x in its corner hides the current picture; the next screenshot shows it again. Closing it never pauses or stops the agent -- it is only there so you can follow along. It is not shown in the popup, the sidebar or on a phone-width screen.

Every screenshot is also kept on disk, per chat, under COMPUTER_USE_DIR/captures/ (default .computeruse/captures/), so you can review a run afterwards. The model itself only ever sees the newest screenshot. Screenshots can contain sensitive information; delete the folder when you no longer need it.

Stopping it

While the agent is working, a Controlling the desktop banner appears above the message box. The application you are automating usually covers the chat, so you have four ways to stop it:

• press Ctrl+Alt+End anywhere;
• move the mouse -- the agent stops as soon as it notices you took over;
• the Stop button;
• Abort on the banner.

After a stop, the rest of that reply cannot touch the desktop.

Safety

• It is off until you set COMPUTER_USE_ENABLED=true.
• The agent works in one locked window. If another window comes to the front, it stops instead of typing into the wrong place.
• The ChatWalaʻau window, sign-in and permission prompts are never targets.
• Passwords by name. Put them in the secrets file ({"crm_password": "..."}) and the agent types {"secret": "crm_password"}: the value never reaches the model, the chat or the logs.
• Text on the screen is not a command. The agent is told so, but a web page or an email on screen could still try to mislead it. Watch the first runs in the capture window, and keep a hand near the mouse.

Settings

App Settings > Computer Use: target window size, screenshot size sent to the model (1920 by default; Claude models are kept at a size the service accepts unchanged), how many controls to list, change threshold, how long the screen must stay calm, wait timeout, steps per batch, computer actions per reply, and how many screenshots to keep in context.

The desktop provider

The part that captures the screen and sends mouse and keyboard input is a small native program, chatwalaau-computer-use, installed with ChatWalaʻau on Windows x64 (a first step towards a desktop provider for more platforms). ChatWalaʻau starts it the first time the agent uses the desktop and talks to it over MCP -- there is nothing to set up, and it is not listed among your MCP servers. It starts in a fraction of a second, captures a window in about 18 ms and prepares a screenshot in about 36 ms (measured on a 2880x1824 screen). If it stops or hangs, the agent is told the desktop is unavailable and the program restarts on the next step. COMPUTER_USE_PROVIDER_COMMAND can point at a different provider program.

Waiting for the screen does not mean looking at it over and over. The provider hears from Windows which parts of the screen were redrawn, so ChatWalaʻau looks again only after something inside the target window was redrawn: a change is noticed almost at once, and a calm screen is not captured again while it stays calm. Whether the screen changed is still decided by comparing screenshots, so a blinking cursor never counts as a change. Where Windows cannot report redraws (the lock screen, a remote session, a window across two monitors) ChatWalaʻau checks the screen every 120 ms instead.

The program is not code-signed yet: Windows or your security software may ask about it the first time it runs.

Checking the speed

GET /api/computer-use/metrics?days=7 summarizes recent runs: time per phase (model, capture, encode, act, settle), the 95th-percentile cycle and how many cycles per minute that allows, and how many steps each decision carried. GET /api/computer-use/status says whether Computer Use is available on this machine, and why not, and whether the desktop provider is running.