Peekaboox

---

name: linux-desktop description: > Control and automate the Linux desktop GUI on X11. Use this skill to take screenshots, find and click UI elements, type text, send keyboard shortcuts, scroll, manage windows (focus, minimize, maximize, close, move, resize), and use your own vision to locate elements on screen. Requires X11 (not Wayland). Use for desktop automation, GUI testing, remote desktop control, and any task requiring interaction with graphical applications. metadata: clawdbot: emoji: "\U0001F5A5" os: ["linux"] requires: bins: ["xdotool", "wmctrl", "scrot"] env: ["DISPLAY"] install: - id: apt kind: shell command: bash install.sh label: Install dependencies (apt/dnf/pacman)

---

Linux Desktop GUI Automation

Automate any X11 Linux desktop: capture screens, find and click elements, type, use hotkeys, manage windows.

Preferred screenshot interpretation path: capture with capture.sh and interpret the image directly in your OpenClaw chat (existing image-capable model connection).

Prerequisites

X11 session running (XFCE, GNOME on X11, KDE on X11, i3, openbox, etc.)

DISPLAY environment variable set (usually :0)

Run bash install.sh once to install dependencies

No extra key needed for screenshot interpretation when using OpenClaw's image-capable chat path

Quick Reference

| Task | Command | |------|---------| | Take screenshot | bash capture.sh | | Screenshot of window | bash capture.sh --window "Firefox" | | List windows | bash inspect.sh | | Active window info | bash inspect.sh --active | | Find window by name | bash inspect.sh --window "Firefox" | | Click at coordinates | bash click.sh --x 500 --y 300 | | Right-click | bash click.sh --x 500 --y 300 --button right | | Double-click | bash click.sh --x 500 --y 300 --double | | Click relative to window | bash click.sh --window "Firefox" --x 200 --y 150 | | Type text | bash type.sh "hello world" | | Type into window | bash type.sh --window "Terminal" "ls -la" | | Send hotkey | bash hotkey.sh "ctrl+c" | | Send Enter | bash hotkey.sh "Return" | | Scroll down | bash scroll.sh --direction down --amount 3 | | Scroll up at position | bash scroll.sh --x 500 --y 300 --direction up --amount 3 | | Focus window | bash window.sh --action focus --window "Firefox" | | Minimize window | bash window.sh --action minimize --window "Firefox" | | Maximize window | bash window.sh --action maximize --window "Firefox" | | Close window | bash window.sh --action close --window "Firefox" | | Move window | bash window.sh --action move --window "Firefox" --x 100 --y 50 | | Resize window | bash window.sh --action resize --window "Firefox" --width 1280 --height 800 |

Typical Automation Workflow

For most GUI automation tasks, follow this pattern:

1. Capture a screenshot with capture.sh — note the file path printed 2. Look at the screenshot yourself to understand what's on screen 3. Find the target element by examining the screenshot and estimating its pixel coordinates 4. Act using the coordinates: click.sh --x X --y Y 5. Verify by capturing another screenshot and checking the result

Example: Click the Save button in a dialog

# Step 1: Capture the screen
SCREENSHOT=$(bash capture.sh | tail -1)
Step 2: Look at the screenshot (read the image file with your vision)
Examine the image and identify the Save button's position
Step 3: Click at the coordinates you identified
bash click.sh --x 450 --y 320

Example: Type into a specific application

# Focus the terminal window and type a command
bash type.sh --window "Terminal" "ls -la"
bash hotkey.sh "Return"

Example: Window management

# Maximize Firefox, then focus a terminal
bash window.sh --action maximize --window "Firefox"
bash window.sh --action focus --window "Terminal"

JSON Output

All tools support a --json flag for machine-readable output:

{"success": true, "output": "...", "error": null}

On failure:

{"success": false, "output": null, "error": "Error description"}

Environment Setup

If DISPLAY is not set (e.g., running over SSH), set it before calling any tool:

export DISPLAY=:0

For headless servers with a virtual display:

Xvfb :0 -screen 0 1920x1080x24 &
export DISPLAY=:0

Hotkey Reference

Key names follow X11 conventions:

| Key | Name | |-----|------| | Enter | Return | | Tab | Tab | | Escape | Escape | | Backspace | BackSpace | | Delete | Delete | | Home | Home | | End | End | | Page Up | Page_Up | | Page Down | Page_Down | | F1-F12 | F1 through F12 | | Super/Win | super | | Ctrl | ctrl | | Alt | alt | | Shift | shift |

Combine with +: ctrl+c, ctrl+shift+t, alt+F4, super+d

Limitations

X11 only — does not work on Wayland sessions

Cannot interact with Wayland-native apps in a Wayland session

Some apps with custom rendering (games, Electron apps with security flags) may resist automation

Screenshot quality depends on compositor; disable compositing if captures look wrong

X11 session running (XFCE, GNOME on X11, KDE on X11, i3, openbox, etc.)

DISPLAY environment variable set (usually :0)

Run bash install.sh once to install dependencies

No extra key needed for screenshot interpretation when using OpenClaw's image-capable chat path