Resilient Claude Agent

Resilient Coding Agent

Long-running coding tasks are vulnerable to interruption: orchestrator restarts, process crashes, network drops. This skill decouples Claude Code from the orchestrator using tmux, enabling fire-and-forget execution with automatic resume on interruption. The orchestrator specifies which model to use; the skill handles session lifecycle, crash recovery, and output capture.

Trust Model

This skill runs Claude Code with --dangerously-skip-permissions and dispatches commands into tmux via send-keys. These are required capabilities, not bugs — the whole point is fire-and-forget execution with no human in the loop to approve prompts. Read this section before installing so you understand exactly what you are enabling.

What this skill can do at runtime:

What this skill does NOT do:

Guarantees the skill enforces:

Deployment expectations:

If you need a version with an explicit, auditable consent mechanism instead of --dangerously-skip-permissions, this skill is not the right choice — the permission bypass is fundamental to fire-and-forget execution.

Placeholders

| Brain sends | CLI receives | |-------------|-------------| | opus | claude-opus-4-6 | | sonnet | claude-sonnet-4-6 |

Full model names are used for determinism. Aliases auto-resolve to the latest version, which could change behavior unexpectedly.

Temp Directory and Prompt Safety

Each task uses a secure temp directory created with mktemp -d. Store this path and use it for all task files (prompt, events, session state). This avoids predictable filenames and symlink/race conditions.

TMPDIR=$(mktemp -d)
chmod 700 "$TMPDIR"

Prompt safety: Task prompts are never interpolated into shell commands. Instead, write the prompt to a temp file using the orchestrator's write tool (no shell involved). The wrapper reads the prompt via stdin redirection (claude -p --model "$MODEL" - < "$TASK_TMPDIR/prompt"), so the prompt content never appears in process arguments (invisible to ps/top) and is not subject to shell interpolation. This depends on the orchestrator's write tool not invoking a shell; OpenClaw's built-in write tool meets this requirement.

Sensitive output: tmux scrollback and log files may contain secrets or API keys from agent output. On shared machines, restrict file permissions (chmod 600) and clean up temp directories after task completion.

When to Use This

Use this skill when the orchestrator explicitly delegates a coding task. Typical use cases include:

The orchestrator decides when to delegate through this skill. Do not self-invoke.

Task Directory Schema

Every task operates within a secure temp directory. The following layout is the canonical specification -- all phases build on this convention.

$TMPDIR/                         # mktemp -d, chmod 700
  prompt                         # Task instructions
                                 #   Written by: orchestrator write tool
                                 #   Read by: Claude Code via $(cat)
  pid                            # Claude Code child process PID
                                 #   Written by: wrapper.sh ($!, atomic write)
                                 #   Read by: monitor.sh (kill -0)
  output.log                     # Continuous output capture
                                 #   Written by: tmux pipe-pane
                                 #   Read by: Brain (tail -n 50), monitor (mtime)
  manifest                       # Structured task state (key=value)
                                 #   Written by: orchestrator (initial) + wrapper.sh (PID, completion)
                                 #   Read by: Brain (grep '^status=' | cut -d= -f2-), monitor
  done                           # Completion marker (presence = complete)
                                 #   Written by: wrapper.sh on exit
                                 #   Read by: monitor.sh ([ -f done ])
  exit_code                      # Process exit code (numeric string)
                                 #   Written by: wrapper.sh (echo $?)
                                 #   Read by: monitor.sh, Brain
  resume                         # Resume signal (written by monitor, consumed by wrapper)
                                 #   Written by: monitor.sh (dispatch_resume)
                                 #   Read by: wrapper.sh (mode detection)
                                 #   Deleted by: wrapper.sh on resume

The resume file is a transient signal used by the monitor to tell the wrapper to resume rather than start fresh. All other files persist after task completion for result retrieval.

Start a Task

Create a tmux session and launch Claude Code with the appropriate model. The launch sequence uses scripts/wrapper.sh which handles PID capture, manifest updates, completion notification, and the done-file protocol. The same wrapper handles both first-run and resume modes.

# Step 1: Create secure temp directory
TMPDIR=$(mktemp -d) && chmod 700 "$TMPDIR"
Step 2: Write prompt to file (use orchestrator's write tool, not echo/shell)
File: $TMPDIR/prompt
Step 3: Create initial manifest
model accepts short aliases (opus, sonnet) or full names (claude-opus-4-6, claude-sonnet-4-6)
cat > "$TMPDIR/manifest" << EOF
task_name=
model=
project_dir=
session_name=claude-
pid=0
tmpdir=$TMPDIR
started_at=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
status=running
EOF
Step 4: Create tmux session (pass TMPDIR via env)
tmux new-session -d -s claude- -e "TASK_TMPDIR=$TMPDIR"
Step 5: Start output capture with ANSI stripping (BEFORE send-keys)
tmux pipe-pane -t claude- -O \
  "perl -pe 's/\x1b\[[0-9;]*[mGKHfABCDJsu]//g; s/\x1b\][^\x07]*\x07//g; s/\x1b\(B//g; s/\r//g' >> $TMPDIR/output.log"
Step 6: Launch via wrapper script
tmux send-keys -t claude- \
  'bash /scripts/wrapper.sh' Enter

Step 3 -- manifest creates manifest with all fields before the tmux session exists. PID is set to 0 (placeholder) because the real PID is not known until after background launch.

Step 5 -- pipe-pane is set BEFORE send-keys to guarantee no output is missed. The -O flag captures only pane output (not input). The perl chain strips four categories of ANSI escapes: CSI sequences (colors, cursor movement), OSC sequences (window titles), charset selection, and carriage returns (progress bar overwrites).

Step 6 -- wrapper invokes scripts/wrapper.sh, which reads TASK_TMPDIR from the tmux session environment (set in Step 4) and model/project_dir from manifest (created in Step 3). The wrapper detects its mode from the filesystem: if done exists, it exits early; if resume exists (written by the monitor), it runs claude -c; otherwise it runs claude -p with the prompt. In all active modes, the wrapper: (1) launches Claude Code in background, (2) writes the real PID to $TASK_TMPDIR/pid, (3) updates manifest with running status, (4) waits for completion, (5) writes exit_code and completion manifest atomically, (6) fires openclaw system event (fire-and-forget), and (7) touch done as the last operation. This ensures resume operations get the same full lifecycle management as first runs.

Replace with the full model name from the mapping table:

Replace with the absolute path to this skill's directory. The Brain already resolves this path when launching scripts/monitor.sh in Step 9.

The wrapper invokes Claude Code with --dangerously-skip-permissions -p --model . -p enables non-interactive (print) mode for fire-and-forget execution. --model selects the model tier. --dangerously-skip-permissions is required because fire-and-forget execution inside tmux cannot handle interactive permission prompts -- without it, the agent stalls on the first file write. This skill is designed for trusted, orchestrator-delegated coding tasks where permission prompts would block the entire flow.

Monitor Progress

Continuous output is captured to $TMPDIR/output.log via pipe-pane (set up in Step 4 of the launch sequence). This is the preferred way to read task output:

# Read recent output from continuous log (preferred)
tail -n 50 $TMPDIR/output.log

Both output.log and manifest persist after the tmux session is killed -- $TMPDIR is created outside the session and is not deleted by monitor cleanup or tmux kill-session. This means result retrieval via tail -n 50 $TMPDIR/output.log works even after the session is gone.

For ad-hoc checks or manual debugging, tmux capture-pane is still available:

# Check if the session is still running
tmux has-session -t claude- 2>/dev/null && echo "running" || echo "finished/gone"
Read recent output (last 200 lines) via tmux
tmux capture-pane -t claude- -p -S -200
Read the full scrollback via tmux
tmux capture-pane -t claude- -p -S -

Check progress when:

Health Monitoring

Use the active monitor script for every task. The monitor runs continuously with configurable intervals and handles its own timing -- no cron or external scheduler needed.

nohup bash /scripts/monitor.sh claude- "$TMPDIR" \
  >"$TMPDIR/monitor.log" 2>&1 &

nohup is critical: it makes the monitor ignore SIGHUP so it survives the orchestrator's shell context ending. Without it, the monitor can be killed when its launching shell exits, leaving tasks without a watchdog. Output goes to $TMPDIR/monitor.log for post-hoc inspection.

The monitor uses a layered detection flow, checked in this exact priority order every iteration:

1. Done-file check -- If $TASK_TMPDIR/done exists, the task completed. Read $TASK_TMPDIR/exit_code for the result. Exit monitor. 2. PID liveness check -- Read PID from $TASK_TMPDIR/pid and test with kill -0 $PID. If the process is dead, the monitor waits 2 seconds and re-checks the done-file (closing the race where the wrapper is still writing completion markers after Claude exits). If done appears during the grace window, treat as success. Otherwise the task crashed: update manifest to status: "crashed" with retry_count and last_checked_at, create a resume marker file, remove the stale pid file, and dispatch scripts/wrapper.sh into the tmux session. The wrapper detects the resume marker and runs claude -c with full lifecycle management. 3. Pane state classification (advisory) -- Capture the last 30 lines of the tmux pane and match against known patterns. Detects waiting_for_input (approval/trust prompts that should not appear with --dangerously-skip-permissions), crash_text (panic/segfault markers), and upgrade_nag. The classified state is written to manifest.pane_state for observability. waiting_for_input is the only actionable state: the monitor abandons immediately with abandon_reason: "waiting_for_input" because nothing is going to answer the prompt. 4. Output staleness check -- If the process is alive but output.log mtime exceeds the staleness threshold (3x base interval, default 90 seconds), the monitor enters a grace period. On the first stale detection, no action is taken -- only a timestamp is recorded. If output remains stale for the full grace period duration, the monitor treats it as a hang: updates the manifest to status: "hung" and dispatches scripts/wrapper.sh for resume (same flow as crash recovery).

The done-file is checked FIRST because a completed task may have a dead PID (expected). Only if done-file is absent does a dead PID indicate a crash. Pane classification and staleness checks only run for live processes.

On consecutive failures, the monitor doubles the polling interval (exponential backoff) and resets when the agent produces fresh output. The monitor stops after the configured deadline (default 5 hours wall-clock).

Configuration

Override monitor behavior by setting environment variables before launching the monitor:

| Variable | Default | Purpose | |----------|---------|---------| | MONITOR_BASE_INTERVAL | 30 (seconds) | Base polling interval; doubles on each consecutive failure | | MONITOR_MAX_INTERVAL | 300 (5 minutes) | Maximum polling interval cap | | MONITOR_DEADLINE | 18000 (5 hours) | Wall-clock deadline; monitor exits after this | | MONITOR_GRACE_PERIOD | 30 (seconds) | Grace period before acting on stale output | | MONITOR_MAX_RETRIES | 10 | Maximum resume attempts before abandoning task | | MONITOR_DISPATCH_WAIT | 10 (seconds) | Post-resume wait before next monitor check |

The staleness threshold is derived as 3x MONITOR_BASE_INTERVAL (default: 90 seconds). To adjust hang detection sensitivity, change MONITOR_BASE_INTERVAL -- the staleness threshold scales automatically.

Cleanup and Abandonment

When the deadline is reached, max retries exceeded, or the monitor is terminated (signal, manual kill), an EXIT trap fires automatically:

1. Manifest update -- Sets manifest status to "abandoned" with an abandoned_at timestamp, unless the task already completed (done-file exists) or was already marked abandoned by the max-retry path. This guard prevents overwriting a completed task's manifest. 2. Notification -- Fires openclaw system event to notify the Brain that the task was abandoned. 3. Session cleanup -- Disables pipe-pane and kills the tmux session, preventing orphan processes.

When max retries are exceeded, the manifest is updated with status: "abandoned", abandon_reason: "max_retries_exceeded", and the final retry_count before the EXIT trap fires.

All exit paths (deadline, max retries, signal, error) trigger the same cleanup sequence.

Recovery After Interruption

For automated crash detection and retries, use Health Monitoring above. Keep this section as a manual fallback when you need to intervene directly:

# Resume the most recent Claude Code session in the working directory
tmux send-keys -t claude- 'claude -c' Enter

claude -c continues the most recent conversation in the current working directory. This is the correct resume command for Claude Code sessions running inside tmux, where only one conversation exists per session.

Cleanup

After a task completes, disable pipe-pane before killing the session. This prevents orphan perl processes that would otherwise hold stale file descriptors:

tmux pipe-pane -t claude-  # Disable pipe-pane (no command = disable)
tmux kill-session -t claude-

List all coding agent tmux sessions:

tmux list-sessions 2>/dev/null | grep -E '^claude-'

Naming Convention

Tmux sessions use the pattern claude-:

Keep names short, lowercase, hyphen-separated. The claude- prefix identifies sessions managed by this skill.

Checklist

Before starting a task:

1. Create secure temp directory (mktemp -d + chmod 700) 2. Write prompt to $TMPDIR/prompt via orchestrator write tool 3. Create initial manifest (all fields, pid=0 placeholder) 4. Create tmux session with TASK_TMPDIR env var 5. Set up pipe-pane output capture with ANSI stripping 6. Launch Claude Code with wrapper (PID capture + manifest updates + done-file protocol) 7. Verify pipe-pane is capturing output (ls -la $TMPDIR/output.log) 8. Notify user: task content, session name (claude-), model used 9. Launch monitor: nohup bash /scripts/monitor.sh claude- "$TMPDIR" >"$TMPDIR/monitor.log" 2>&1 & (mandatory for every task; nohup ensures the monitor survives parent shell exit)

Limitations

Prerequisites

This skill requires:

The orchestrator must be configured to delegate coding tasks through this skill instead of attempting them directly. SKILL.md is the orchestrator's interface -- it reads this document and follows the instructions.

Override monitor behavior by setting environment variables before launching the monitor:

| Variable | Default | Purpose | |----------|---------|---------| | MONITOR_BASE_INTERVAL | 30 (seconds) | Base polling interval; doubles on each consecutive failure | | MONITOR_MAX_INTERVAL | 300 (5 minutes) | Maximum polling interval cap | | MONITOR_DEADLINE | 18000 (5 hours) | Wall-clock deadline; monitor exits after this | | MONITOR_GRACE_PERIOD | 30 (seconds) | Grace period before acting on stale output | | MONITOR_MAX_RETRIES | 10 | Maximum resume attempts before abandoning task | | MONITOR_DISPATCH_WAIT | 10 (seconds) | Post-resume wait before next monitor check |

The staleness threshold is derived as 3x MONITOR_BASE_INTERVAL (default: 90 seconds). To adjust hang detection sensitivity, change MONITOR_BASE_INTERVAL -- the staleness threshold scales automatically.

Cleanup and Abandonment

When the deadline is reached, max retries exceeded, or the monitor is terminated (signal, manual kill), an EXIT trap fires automatically:

1. Manifest update -- Sets manifest status to "abandoned" with an abandoned_at timestamp, unless the task already completed (done-file exists) or was already marked abandoned by the max-retry path. This guard prevents overwriting a completed task's manifest. 2. Notification -- Fires openclaw system event to notify the Brain that the task was abandoned. 3. Session cleanup -- Disables pipe-pane and kills the tmux session, preventing orphan processes.

When max retries are exceeded, the manifest is updated with status: "abandoned", abandon_reason: "max_retries_exceeded", and the final retry_count before the EXIT trap fires.

All exit paths (deadline, max retries, signal, error) trigger the same cleanup sequence.