🎁 Get the FREE AI Skills Starter Guide β€” Subscribe β†’
BytesAgainBytesAgain
BytesAgain β€Ί Skills β€Ί Agent Scorecard
πŸ¦€ ClawHub

Agent Scorecard

by @theshadowrose

Configurable quality evaluation for AI agent outputs. Define criteria, run evaluations, track quality over time. No LLM-as-judge, no API calls, pattern-based...

Versionv1.0.6
#clawhub
⚑ When to Use
TriggerAction
- **Model comparison:** Same task, different models β€” which scores higher?
- **Agent regression testing:** Catch quality degradation before it ships
- **Team quality standards:** Define shared rubrics for consistent evaluation
- **Continuous monitoring:** Track quality trends over days/weeks/months
- **A/B testing:** Quantified before/after comparisons
πŸ’‘ Examples

# 1. Configure
cp config_example.json scorecard_config.json

Edit dimensions, thresholds, and weights for your use case
2. Evaluate a response

python3 scorecard.py --config scorecard_config.json --input response.txt
3. Evaluate and save to history

python3 scorecard.py --config scorecard_config.json --input response.txt --save history.jsonl
4. Manual scoring mode

python3 scorecard.py --config scorecard_config.json --input response.txt --manual --save history.jsonl
5. View trends

python3 scorecard_track.py --history history.jsonl --summary
6. Compare before/after (last 10 vs previous 10)

python3 scorecard_track.py --history history.jsonl --compare 10
7. Generate a report

python3 scorecard_report.py --config scorecard_config.json --history history.jsonl

βš™οΈ Configuration

See config_example.json for the complete reference. Key areas:

  • DIMENSIONS β€” Quality dimensions with rubrics, weights, thresholds, and auto-checks
  • AUTO_CHECKS β€” Tuning for each pattern-based check (markers, thresholds, penalties)
  • AGGREGATE_METHOD β€” How to combine dimension scores ("weighted_average", "minimum", "geometric_mean")
  • HISTORY_FILE β€” Where to store evaluation history
  • REPORT_OUTPUT_DIR β€” Where reports are saved

    • View on ClawHub
    TERMINAL
    clawhub install agent-scorecard

    πŸ§ͺ Use this skill with your agent

    Most visitors already have an agent. Pick your environment, install or copy the workflow, then run the smoke-test prompt above.

    Manus
    Task-oriented agent. Great for testing AI skills end-to-end.
    Try Manus β†’
    OpenClaw
    Local-first agent. Install skills via ClawHub CLI.
    Set up OpenClaw β†’
    Claude Code
    Anthropic's coding agent. Paste the prompt or SKILL.md into your session.
    Claude Code docs β†’
    Cursor
    AI-powered IDE. Use the smoke-test prompt in Cursor Agent.
    Open Cursor β†’
    Continue.dev
    Open-source AI code assistant. Add SKILL.md as a custom tool.
    Continue docs β†’
    Windsurf
    Agentic IDE by Codeium. Paste the prompt into Cascade.
    Try Windsurf β†’
    Cline
    VS Code extension for autonomous coding with MCP tools.
    Cline on GitHub β†’
    Copilot Workspace
    GitHub's AI dev environment. Suitable for code-generation skills.
    Copilot Workspace β†’

    πŸ” Can't find the right skill?

    Search 60,000+ AI agent skills β€” free, no login needed.

    Search Skills β†’