Pandas Skill

Pandas Data Processing Skill

English | 简体中文

This skill provides comprehensive pandas data processing capabilities through executable scripts and reference documentation. Use this skill whenever tasks involve data manipulation, cleaning, analysis, or transformation of tabular data.

When to Use This Skill

Activate this skill when the user requests:

Core Capabilities

1. Data Cleaning (scripts/data_cleaner.py)

Handles common data cleaning tasks with a single command:

Usage:

python scripts/data_cleaner.py input.csv output.csv [options]

Available Options:

Example:

python scripts/data_cleaner.py data.csv cleaned_data.csv \
    --remove-duplicates \
    --handle-missing mean \
    --remove-outliers \
    --standardize-columns

2. Data Analysis (scripts/data_analyzer.py)

Generates comprehensive data analysis reports:

Usage:

python scripts/data_analyzer.py input.csv [options]

Available Options:

Report Includes:

Example:

python scripts/data_analyzer.py sales_data.csv -o report.json --format json

3. Data Transformation (scripts/data_transformer.py)

Performs various data transformation operations through subcommands:

#### Convert Format

python scripts/data_transformer.py convert input.csv output.xlsx

#### Merge Files

python scripts/data_transformer.py merge file1.csv file2.csv file3.csv \
    --output merged.csv \
    --how outer \
    --on key_column

#### Filter Data

python scripts/data_transformer.py filter data.csv \
    --query "age > 18 and city == 'Beijing'" \
    --output filtered.csv

#### Sort Data

python scripts/data_transformer.py sort data.csv \
    --by sales quantity \
    --descending \
    --output sorted.csv

#### Select Columns

python scripts/data_transformer.py select data.csv \
    --columns name age city \
    --output selected.csv

Reference Documentation

The references/ directory contains detailed documentation:

references/common_operations.md

Comprehensive reference covering:

When to use: When Claude needs to understand pandas syntax or find the right method for a specific operation.

references/data_cleaning_best_practices.md

Best practices guide covering:

When to use: When designing a data cleaning workflow or deciding on the best approach for specific data quality issues.

Workflow Guidelines

Step 1: Initial Assessment

python scripts/data_analyzer.py input_file.csv -o analysis_report.json

Step 2: Plan Cleaning Strategy

Step 3: Execute Cleaning

python scripts/data_cleaner.py input.csv cleaned.csv [options]

Step 4: Transform as Needed

python scripts/data_transformer.py [subcommand] [options]

Step 5: Validate Results

python scripts/data_analyzer.py cleaned.csv -o final_report.json

Common Patterns

Pattern 1: Quick Data Quality Report

python scripts/data_analyzer.py data.csv --format text

Pattern 2: Standard Cleaning Pipeline

python scripts/data_cleaner.py raw_data.csv clean_data.csv \
    --standardize-columns \
    --remove-duplicates \
    --handle-missing median \
    --remove-outliers

Pattern 3: Excel to CSV with Filtering

# Convert
python scripts/data_transformer.py convert data.xlsx data.csv
Filter
python scripts/data_transformer.py filter data.csv \
    --query "status == 'active'" \
    --output filtered.csv

Pattern 4: Merge Multiple CSVs

python scripts/data_transformer.py merge *.csv \
    --output combined.csv

Dependencies

Ensure pandas is installed:

pip install pandas numpy openpyxl

Optional for specific formats:

pip install pyarrow  # For Parquet support
pip install xlrd     # For older Excel files (.xls)

Tips for Effective Use

1. Start with analysis: Always run the analyzer first to understand the data 2. Incremental cleaning: Apply cleaning operations step by step, verify each step 3. Preserve originals: Never overwrite original data files 4. Check references: Consult reference docs for complex operations or best practices 5. Validate results: Use the analyzer to verify cleaning effectiveness 6. Memory efficiency: For large files, consider using the data type optimization techniques in the reference docs 7. Combine operations: Chain multiple transformer commands for complex workflows

Limitations

Troubleshooting

Import errors: Ensure pandas and dependencies are installed Memory errors: Process data in chunks or optimize dtypes (see references) Encoding issues: Add encoding='utf-8' parameter when loading CSVs Date parsing issues: Use pd.to_datetime() with explicit format string

For detailed pandas operations and troubleshooting, always refer to references/common_operations.md and references/data_cleaning_best_practices.md.

Import errors: Ensure pandas and dependencies are installed Memory errors: Process data in chunks or optimize dtypes (see references) Encoding issues: Add encoding='utf-8' parameter when loading CSVs Date parsing issues: Use pd.to_datetime() with explicit format string

For detailed pandas operations and troubleshooting, always refer to references/common_operations.md and references/data_cleaning_best_practices.md.