Gemini API Computer Use Is Generally Available: Intents, Safety Decisions and the Client Loop You Still Build

On October 30, 2026 the Gemini API release notes list: "Computer Use generally available (GA): Released the Computer Use tool to general availability (GA) across the Interactions and GenerateContent APIs." The docs page explains what that means in practice. The model does not control a computer. It looks at a screenshot you send and returns a proposed UI action, and your code has to execute it, check it and send back the next screenshot.

I read the Computer Use docs page and the Gemini API changelog. I did not run any code or test the tool. Everything below is what Google's documents say.

What the docs say it is

The page describes a tool for building browser, mobile and desktop control agents. Using screenshots, the model can "see" a screen and "act" by generating UI actions such as mouse clicks and keyboard input. Like function calling, you implement the client-side execution environment that receives and runs the actions.

For Gemini 4 and Gemini 3.x models, the page lists these capabilities:

• Multi-environment support for browser, mobile and desktop.
• Actions that include an intent field explaining the model's reasoning for each step.
• Configurable safety policies with built-in categories and overrides.
• Opt-in prompt injection detection that scans screenshots for hidden adversarial instructions.

A minimal call in the docs passes a tool with type computer_use and an environment, for example browser, to the Interactions API. The prompt-injection option is a flag on that tool, enable_prompt_injection_detection. The examples use the gemini-3.8-flash model string, and the Python walkthrough says to use the google-genai SDK version 2.7.0 or higher.

The loop you build

The docs describe four steps, repeated until the task ends:

1. Your app sends the request with the tool, your settings, the prompt and a screenshot.

2. The model returns a suggested function_call for a UI action, with an intent on Gemini 4 and 3.x. The response may also include a safety_decision that classifies the action as regular or allowed, require_confirmation (needs user approval) or blocked.

3. If the action is allowed, or the user confirms, your code parses it, scales the coordinates and runs it with something like Playwright. If it is blocked, your client should halt or handle the interruption.

4. Your app captures a new screenshot and sends it back in a function_result.

Coordinates are normalized to 0 to 999 on each axis in the action tables, and the page says the model predicts positions scaled to the screen, so you do not specify display size in the request. Your code converts them to real pixels.

Actions by environment

The page lists the actions per environment.

• Browser: click variants, move, type, drag and drop, wait, key presses, hotkey, screenshot, scroll, back, forward and navigate to a URL.
• Mobile: open_app, list_apps, click, type, drag and drop, long_press, wait, back, key press and screenshot.
• Desktop: OS-level cursor commands: click variants, move, type, drag and drop, wait, key presses, hotkey, screenshot and scroll.

You can also exclude predefined actions and add your own functions. The docs show a human-in-the-loop example that removes click and registers a custom yield_to_user function so the model can hand control back, for example when a second factor code is needed.

What Google tells you to do about risk

The page carries a plain warning: Computer Use "may contain errors and security vulnerabilities". It recommends close supervision for important tasks and says to avoid it for critical decisions, sensitive data, or actions where serious errors cannot be corrected. It also tells you to run the agent in a sandboxed VM or container to isolate it from your host, and it mentions a reference implementation with a Docker-based sandbox. In the Playwright setup code, a comment says to use a sandboxed environment in production.

What the changelog adds

The same changelog has earlier Computer Use entries. One says a public preview for Gemini 3.5 Flash included simplified actions with intents, built-in support for browser, mobile and desktop environments, configurable safety policies and advanced prompt injection detection. Another entry says gemini-3.5-flash is deprecated and requests are routed to gemini-3.6-flash. Check the current model strings in the docs before copying an example, because the page I read used a different model name from those entries.

What I would check

These are my untested suggestions.

1. Turn on prompt injection detection in a test and feed it a page with hidden text, then see what comes back.

2. Log every safety_decision and make sure require_confirmation reaches a human, not a default yes.

3. Run the loop only in a disposable container with no real credentials.

4. Record the intent field next to each action so a reviewer can compare the stated reason with what happened.

5. Set a step limit and a time limit on the loop in your own code.

6. Verify the current supported model list before choosing a model string.

What I would not conclude

• That GA means the tool is safe for sensitive work. The docs page still warns against it for critical decisions and sensitive data.
• That the intent field proves why the model acted. The docs call it an explanation of reasoning. I did not test how reliable it is.
• That the model executes actions. Your client does.
• That prompt injection detection catches everything. It is documented as opt-in screenshot scanning, and I have no detection numbers.

Frequently asked questions

When did Computer Use become generally available?

The Gemini API changelog lists it under October 30, 2026, across the Interactions and GenerateContent APIs.

Which environments does it support?

Browser, mobile and desktop, per the docs page.

Does Gemini run the actions?

No. The model proposes a function call and your client-side code runs it.

What is the safety decision?

A field the response may include that marks the action as allowed, requiring confirmation, or blocked.

What is the intent field?

A short explanation of why the model chose each step, on Gemini 4 and 3.x models.

How do I detect prompt injection?

The docs describe an opt-in flag, enable_prompt_injection_detection, that scans screenshots for hidden adversarial instructions.

Did you test it?

No. I read the docs page and the changelog.

Sources

All read on November 1, 2026.

• Computer use, Gemini API docs
• Gemini API release notes

Not read: the Safety best practices page, the Model versions page, the Vertex and Enterprise Agent Platform computer use page, pricing, and the reference implementation.