Finest Computer Use

From intent to action.

An integration and control layer that lets AI agents operate macOS interfaces within explicit boundaries.

My role

Design and implementation of the Python integration and control layer.

Technologies

Python · MCP · Gemini · macOS

View source

Intent.
Action.
Verified.

Autonomy needs judgment. And boundaries.

Python · MCP · macOS

The problem

An agent can understand an instruction and still execute the wrong action. In a graphical interface, an ambiguous click can produce an effect nobody authorized. The ability to act needs boundaries, observation and verification.

What I built

Integrated Gemini/Antigravity agents with Peekaboo MCP to operate macOS interfaces.
Implemented per-task authorization, action budgets, isolated state and post-mutation barriers.
Added diagnostics, recovery and an installer with backup and rollback.

How it works

User intent
Authorization and budget
Execution via Peekaboo MCP
Observation and verification

Engineering decisions

Intent sets the boundary.

Authorization stays tied to the task. It does not silently expand just because an action is convenient for the agent.

Observe before acting again.

Mutation barriers require checking the resulting state. The next action should not rely on a view of the interface that is already out of date.

Integrate, rather than reinvent.

Peekaboo provides macOS interaction. My contribution focuses on the integration, policies and effect controls around those tools.

Results and evidence

The repository contains hooks, an effect firewall, regression tests and installation tests. I have tested the integration with Gemini. Synthetic checks and real executions are different forms of evidence.