Node Transfer

node-transfer

High-speed, memory-efficient file transfer between OpenClaw nodes using native Node.js streams.

📋 Table of Contents

Problem Solved

Architecture

Requirements

Installation

Usage

API Reference

Troubleshooting

---

🎯 Problem Solved

The Original Problem

When transferring large files between OpenClaw nodes using the standard nodes.invoke mechanism, we encountered several critical issues:

| Issue | Impact | |-------|--------| | Base64 Encoding Overhead | 33% larger payload, slower transfers | | Memory Exhaustion (OOM) | Loading multi-GB files into memory crashes the process | | Transfer Latency | JSON serialization/deserialization adds significant delay | | 9-Minute Deployments | Re-deploying scripts on every transfer |

The Solution

node-transfer uses native HTTP streaming with Node.js streams, providing:

✅ Zero memory overhead - Files stream directly from disk to network

✅ No Base64 encoding - Raw binary transfer

✅ Speed - Line-speed limited only by network bandwidth

✅ Install Once, Run Many - Scripts persist on nodes after first deployment

Performance Comparison

| Metric | Base64 Transfer | node-transfer | Improvement | |--------|----------------|---------------|-------------| | 1GB file transfer time | ~15-30 min | ~8 sec | ~150x faster | | Memory usage | 1GB+ | <10MB | 99% reduction | | First transfer overhead | N/A | ~30 sec (one-time install) | - | | Subsequent transfers | ~15-30 min | <1 sec check + ~8 sec transfer | ~200x faster |

---

🏗️ Architecture

How It Works

┌──────────────┐     HTTP Stream      ┌──────────────┐
│  send.js     │ ◄──────────────────► │ receive.js   │
│  (Source)    │   (Token-protected)  │ (Destination)│
└──────────────┘                      └──────────────┘
       │                                     │
       ▼                                     ▼
┌──────────────┐                      ┌──────────────┐
│  Read Stream │                      │ Write Stream │
│  (fs.create  │                      │ (fs.create   │
│   ReadStream)│                      │  WriteStream)│
└──────────────┘                      └──────────────┘
       │                                     │
       ▼                                     ▼
┌──────────────┐                      ┌──────────────┐
│  File on     │                      │  File on     │
│  Disk        │                      │  Disk        │
└──────────────┘                      └──────────────┘

Security Model

1. One-time Token: 256-bit cryptographically random token (64 hex chars) 2. Single Connection: Only one download allowed per token 3. Auto-shutdown: Server closes after transfer completes or disconnects 4. Token Validation: Every request must include the correct token

Data Flow

1. Sender (send.js): - Generates random port and security token - Starts HTTP server on ephemeral port - Streams file directly from disk to HTTP response - Auto-shutdown after transfer or timeout (5 min default)

2. Receiver (receive.js): - Connects to sender URL with token - Streams HTTP response directly to disk - Reports progress, speed, and completion status - Validates received bytes match expected size

---

📦 Requirements

Node.js: 14.0.0 or higher

Network: TCP connectivity between nodes (any port 1024-65535)

Firewall: Must allow outbound connections and inbound on ephemeral ports

Disk Space: Sufficient space on destination for received files

---

🚀 Installation

The "Install Once" Pattern

Instead of deploying scripts on every transfer, we deploy them once per node and use a fast version check for subsequent transfers.

Method 1: Using deploy.js (Recommended)

# Generate deployment script for a target node
node deploy.js E3V3
This outputs a PowerShell script that you can execute via nodes.invoke()

Method 2: Manual Deployment

On each target node, create the directory and copy files:

# Create directory
mkdir C:/openclaw/skills/node-transfer/scripts -Force
Copy these files (ensure UTF-8 without BOM encoding):
- send.js
- receive.js
- ensure-installed.js
- version.js

Method 3: Via OpenClaw Agent

// 1. Check if already installed (< 100ms)
const check = await nodes.invoke({
    node: 'E3V3',
    command: ['node', 'C:/openclaw/skills/node-transfer/scripts/ensure-installed.js', 
              'C:/openclaw/skills/node-transfer/scripts']
});
const checkResult = JSON.parse(check.output);
if (!checkResult.installed) {
    // 2. Deploy if needed (one-time, ~30 seconds)
    // Use the deploy.js output or manually copy files
    console.log('Deploying node-transfer to E3V3...');
    // ... deployment code ...
}

---

💡 Usage

Basic Transfer Workflow

const INSTALL_DIR = 'C:/openclaw/skills/node-transfer/scripts';
const SOURCE_NODE = 'E3V3';
const DEST_NODE = 'E3V3-Docker';
// Step 1: Check installation on both nodes (fast!)
const [sourceCheck, destCheck] = await Promise.all([
    nodes.invoke({
        node: SOURCE_NODE,
        command: ['node', ${INSTALL_DIR}/ensure-installed.js, INSTALL_DIR]
    }),
    nodes.invoke({
        node: DEST_NODE,
        command: ['node', ${INSTALL_DIR}/ensure-installed.js, INSTALL_DIR]
    })
]);
// Deploy if needed (usually only once per node ever)
// ... deployment code if not installed ...
// Step 2: Start sender on source node
const sendResult = await nodes.invoke({
    node: SOURCE_NODE,
    command: ['node', ${INSTALL_DIR}/send.js, 'C:/data/large-file.zip']
});
const { url, token, fileSize, fileName } = JSON.parse(sendResult.output);
// Step 3: Start receiver on destination node
const receiveResult = await nodes.invoke({
    node: DEST_NODE,
    command: ['node', ${INSTALL_DIR}/receive.js, url, token, '/incoming/file.zip']
});
const result = JSON.parse(receiveResult.output);
console.log(Transferred ${result.bytesReceived} bytes in ${result.duration}s at ${result.speedMBps} MB/s);

Using the Command Line

#### Sender

node send.js /path/to/file.zip

Output:

{
  "url": "http://192.168.1.10:54321/transfer",
  "token": "a1b2c3d4e5f6789...",
  "fileSize": 1073741824,
  "fileName": "file.zip",
  "sourceIp": "192.168.1.10",
  "port": 54321,
  "version": "1.0.0"
}

Options:

node send.js /path/to/file.zip --port 8080 --timeout 10
node send.js --help
node send.js --version

#### Receiver

node receive.js "http://192.168.1.10:54321/transfer" "token-here..." /path/to/save.zip

Output:

{
  "success": true,
  "bytesReceived": 1073741824,
  "totalBytes": 1073741824,
  "duration": 8.42,
  "speedMBps": 121.5,
  "outputPath": "/path/to/save.zip"
}

Options:

node receive.js   
 --timeout 60 --no-progress
node receive.js --help
node receive.js --version

---

📚 API Reference

send.js

Starts an HTTP server to stream a file.

Usage: node send.js [options]

Arguments:

filePath (required): Path to the file to send

Options:

--port : Use specific port (default: random ephemeral)

--timeout : Timeout in minutes (default: 5)

Output (JSON): | Field | Type | Description | |-------|------|-------------| | url | string | HTTP URL for receiver to connect to | | token | string | Security token (64 hex chars) | | fileSize | number | File size in bytes | | fileName | string | Original filename | | sourceIp | string | IP address of sender | | port | number | TCP port used | | version | string | Version of send.js |

Exit Codes:

0: Success (transfer completed or info displayed)

1: Error (check stderr for JSON error details)

Error Output (JSON):

{
  "error": "ERROR_CODE",
  "message": "Human-readable description"
}

Error codes: FILE_NOT_FOUND, NOT_A_FILE, SERVER_ERROR, TIMEOUT, READ_ERROR, RESPONSE_ERROR

---

receive.js

Connects to a sender and downloads a file.

Usage: node receive.js [options]

Arguments:

url (required): URL from send.js output

token (required): Security token from send.js output

outputPath (required): Path to save the received file

Options:

--timeout : Connection timeout in seconds (default: 30)

--no-progress: Suppress progress updates

Output (JSON): | Field | Type | Description | |-------|------|-------------| | success | boolean | Always true on success | | bytesReceived | number | Actual bytes received | | totalBytes | number | Expected bytes (from Content-Length) | | duration | number | Transfer time in seconds | | speedMBps | number | Average speed in MB/s | | outputPath | string | Absolute path to saved file |

Progress Updates (when not using --no-progress):

{
  "progress": true,
  "receivedBytes": 536870912,
  "totalBytes": 1073741824,
  "percent": 50,
  "speedMBps": 125.4
}

Exit Codes:

0: Success

1: Error (check stderr for JSON error details)

Error codes: INVALID_ARGS, INVALID_URL, CONNECTION_ERROR, HTTP_ERROR, TIMEOUT, WRITE_ERROR, SIZE_MISMATCH, FILE_EXISTS, NO_DATA

---

ensure-installed.js

Fast check if node-transfer is installed on a node.

Usage: node ensure-installed.js

Arguments:

targetDir (required): Directory to check

Output (JSON):

Installed:

{
  "installed": true,
  "version": "1.0.0",
  "message": "node-transfer is installed and up-to-date"
}

Needs installation:

{
  "installed": false,
  "missing": ["send.js"],
  "mismatched": [],
  "currentVersion": null,
  "requiredVersion": "1.0.0",
  "action": "DEPLOY",
  "message": "Installation needed: 1 missing, 0 outdated"
}

Exit Codes:

0: Already installed and up-to-date

1: Needs installation/update

2: Error (invalid directory, etc.)

---

deploy.js

Generates deployment scripts for the main agent.

Usage: node deploy.js [targetDir]

Output: JSON with:

script: PowerShell script to deploy files

escapedScript: Escaped version for command-line use

usage: Example code for JavaScript and CLI usage

---

🔧 Troubleshooting

"Connection timeout"

Cause: Network connectivity issue or firewall blocking connection.

Solutions:

Verify both nodes can reach each other

Check firewall rules allow outbound connections

Try specifying a specific port with --port

Increase timeout with --timeout

"403 Forbidden: Invalid or missing token"

Cause: Token mismatch or URL manipulation.

Solutions:

Use the exact token from send.js output

Don't modify the URL

Ensure the token hasn't expired (sender times out after 5 minutes)

"409 Conflict: Transfer already in progress"

Cause: Multiple connections attempted with same token.

Solutions:

Each sender URL/token can only be used once

Start a new sender if you need to retry

"FILE_NOT_FOUND" or "NOT_A_FILE"

Cause: Invalid file path on sender.

Solutions:

Use absolute paths

Verify file exists

Check file permissions

"SIZE_MISMATCH"

Cause: Connection interrupted or network error.

Solutions:

Retry the transfer

Check network stability

The partial file is automatically cleaned up

"Hash mismatch" during ensure-installed

Cause: Files were modified or corrupted.

Solutions:

Re-deploy scripts using deploy.js

Ensure files are copied without modification

Check encoding (must be UTF-8 without BOM)

Slow transfers on subsequent runs

Cause: Not using ensure-installed.js check pattern.

Solutions:

Always check installation first (< 100ms)

Only deploy if installed: false

Follow the "Install Once, Run Many" pattern

---

📄 Files

| File | Purpose | |------|---------| | send.js | HTTP server that streams files to receivers | | receive.js | HTTP client that downloads files from senders | | ensure-installed.js | Fast version/integrity check for deployment | | version.js | Version manifest for update detection | | deploy.js | Generates deployment scripts for agents |

---

🤝 Contributing

See CONTRIBUTING_PROPOSAL.md for information on how this could be integrated into OpenClaw core.

---

*Built for OpenClaw - No Base64, No OOM, No Waiting.*