🎁 Get the FREE AI Skills Starter Guide β€” Subscribe β†’
BytesAgainBytesAgain
BytesAgain β€Ί Skills β€Ί Hadoop
πŸ¦€ ClawHub

Hadoop

by @ivangdavila

Manage Hadoop clusters with HDFS operations, YARN job tuning, and distributed processing diagnostics.

Versionv1.0.0
Downloads1,186
#legal#productivity
TERMINAL
clawhub install hadoop

πŸ“– About This Skill

name: Hadoop slug: hadoop version: 1.0.0 homepage: https://clawic.com/skills/hadoop description: Manage Hadoop clusters with HDFS operations, YARN job tuning, and distributed processing diagnostics. metadata: {"clawdbot":{"emoji":"🐘","requires":{"bins":["hdfs","yarn","hadoop"]},"os":["linux","darwin"]}}

Setup

If ~/hadoop/ doesn't exist or is empty, read setup.md and start the conversation naturally.

When to Use

User works with Hadoop ecosystem (HDFS, YARN, MapReduce, Hive). Agent handles cluster diagnostics, job optimization, storage management, and troubleshooting distributed processing failures.

Architecture

Memory lives in ~/hadoop/. See memory-template.md for structure.

~/hadoop/
β”œβ”€β”€ memory.md        # Cluster configs, common issues, preferences
β”œβ”€β”€ clusters/        # Per-cluster notes and configs
β”‚   └── {name}.md    # Specific cluster context
└── scripts/         # Custom diagnostic scripts

Quick Reference

| Topic | File | |-------|------| | Setup process | setup.md | | Memory template | memory-template.md | | HDFS operations | hdfs.md | | YARN tuning | yarn.md | | Troubleshooting | troubleshooting.md |

Core Rules

1. Verify Cluster State First

Before any operation, check cluster health: 
hdfs dfsadmin -report
yarn node -list
Never assume cluster is healthy. A single dead DataNode changes everything.

2. Storage Before Compute

HDFS issues cascade into job failures. Always check: 
hdfs dfs -df -h                    # Capacity
hdfs fsck / -files -blocks         # Block health
A job failing with "No space left" is storage, not code.

3. Resource Calculator Awareness

YARN allocates based on configured scheduler. Know which is active: 
yarn rmadmin -getServiceState rm1
cat /etc/hadoop/conf/yarn-site.xml | grep scheduler
Default (Capacity) vs Fair scheduler behave very differently.

4. Replication Factor Context

Default replication=3. For temp data, suggest 1-2 to save space: 
hdfs dfs -setrep -w 1 /tmp/scratch/
For critical data, verify replication is honored: 
hdfs fsck /data/critical -files -blocks -replicaDetails

5. Log Location Awareness

Hadoop logs scatter across machines. Key locations: | Component | Log Path | |-----------|----------| | NameNode | /var/log/hadoop-hdfs/hadoop-hdfs-namenode-*.log | | DataNode | /var/log/hadoop-hdfs/hadoop-hdfs-datanode-*.log | | ResourceManager | /var/log/hadoop-yarn/yarn-yarn-resourcemanager-*.log | | NodeManager | /var/log/hadoop-yarn/yarn-yarn-nodemanager-*.log | | Application | yarn logs -applicationId |

6. Safe Mode Handling

NameNode enters safe mode on startup or low block count: 
hdfs dfsadmin -safemode get        # Check status
hdfs dfsadmin -safemode leave      # Exit (if blocks OK)
Never force-leave if blocks are actually missing.

7. Memory Settings Matter

90% of "job killed" issues are memory: 
# Container settings
yarn.nodemanager.resource.memory-mb     # Total per node
yarn.scheduler.minimum-allocation-mb    # Min container
mapreduce.map.memory.mb                 # Map task
mapreduce.reduce.memory.mb              # Reduce task
Check these before assuming code is wrong.

HDFS Operations

Essential Commands

# Navigation
hdfs dfs -ls /path
hdfs dfs -du -h /path              # Size with human units
hdfs dfs -count -q /path           # Quota info
Data movement

hdfs dfs -put local.txt /hdfs/     # Upload
hdfs dfs -get /hdfs/file.txt .     # Download
hdfs dfs -cp /src /dst             # Copy within HDFS
hdfs dfs -mv /src /dst             # Move within HDFS
Maintenance

hdfs dfs -rm -r /path              # Delete (trash)
hdfs dfs -rm -r -skipTrash /path   # Delete (permanent)
hdfs dfs -expunge                  # Empty trash

Block Management

# Find corrupt blocks
hdfs fsck / -list-corruptfileblocks
Delete corrupt file (after confirming unrecoverable)

hdfs fsck /path/file -delete
Force replication

hdfs dfs -setrep -w 3 /important/data/

YARN Job Management

Application Lifecycle

# List applications
yarn application -list                    # Running
yarn application -list -appStates ALL     # All states
Application details

yarn application -status 
Kill stuck application

yarn application -kill 
Get logs (after completion)

yarn logs -applicationId 
yarn logs -applicationId  -containerId

Queue Management

# List queues
yarn queue -list
Queue status

yarn queue -status 
Move application between queues

yarn application -movetoqueue  -queue

Common Traps

  • Deleting without -skipTrash on full cluster β†’ Trash still uses space, cluster stays full
  • Setting container memory below JVM heap β†’ Instant container kill, confusing errors
  • Ignoring speculative execution on slow jobs β†’ Wastes resources on duplicated tasks
  • Running fsck on busy cluster β†’ Performance impact, run during maintenance
  • Assuming HDFS = POSIX semantics β†’ No append-in-place, no random writes
  • Forgetting timezone in scheduling β†’ Oozie/Airflow jobs fire at wrong times

    • Security & Privacy

    Data that stays local:

  • Cluster notes saved in ~/hadoop/clusters/
  • Preferences and environment context

    • What commands access:

  • hdfs/yarn commands connect to your Hadoop cluster
  • Some commands read system paths (/var/log, /etc/hadoop/conf)
  • Destructive commands require explicit user confirmation

    • This skill does NOT:

  • Store credentials (use kinit/keytab separately)
  • Make external API calls beyond your cluster
  • Run destructive commands without asking first

    • Related Skills

    Install with clawhub install if user confirms:
  • linux β€” system administration
  • docker β€” containerized deployments
  • bash β€” shell scripting

    • Feedback

  • If useful: clawhub star hadoop
  • Stay updated: clawhub sync

    • ⚑ When to Use

    User works with Hadoop ecosystem (HDFS, YARN, MapReduce, Hive). Agent handles cluster diagnostics, job optimization, storage management, and troubleshooting distributed processing failures.

    βš™οΈ Configuration

    If ~/hadoop/ doesn't exist or is empty, read setup.md and start the conversation naturally.