Macos Gui Automation

macOS GUI Automation Skill

Capabilities

Tools Available

cliclick (Mouse/Keyboard)

# Click at coordinates
cliclick c:x,y
Double click
cliclick dc:x,y
Right click
cliclick rc:x,y
Move mouse
cliclick m:x,y
Drag from to
cliclick dr:x1,y1:x2,y2
Type text
cliclick t:"text"
Press key (Enter, Tab, etc.)
cliclick kp:enter

screencapture + tesseract (Screen Reading)

# Capture region to file
screencapture -R x,y,w,h /tmp/screen.png
Capture full screen
screencapture /tmp/screen.png
OCR from image
tesseract /tmp/screen.png stdout
OCR with Chinese support
tesseract /tmp/screen.png stdout -l chi_sim+eng

osascript (AppleScript - Window/App Control)

# List all windows
osascript -e 'tell application "System Events" to get name of every process'
Get window position/size
osascript -e 'tell application "Finder" to get bounds of window of front window'
Click menu item
osascript -e 'tell application "System Events" to click menu item "Save" of menu "File" of process "TextEdit"'

Usage Patterns

Read Screen Text

# 1. Capture screen
screencapture -R 100,100,800,600 /tmp/region.png
2. OCR
tesseract /tmp/region.png stdout

Click Button at Position

cliclick c:500,300

Type in Field

# Click field first
cliclick c:400,200
Then type
cliclick t:"hello world"
cliclick kp:enter

Find and Click (OCR + Click)

# 1. Capture and OCR
screencapture /tmp/screen.png
text=$(tesseract /tmp/screen.png stdout)
2. Parse coordinates from OCR result or use image recognition
3. Click
cliclick c:x,y

Limitations

Security Notes