🎁 Get the FREE AI Skills Starter Guide β€” Subscribe β†’
BytesAgainBytesAgain
BytesAgain β€Ί Skills β€Ί Llm Evaluator
πŸ¦€ ClawHub

Llm Evaluator

by @aiwithabidi

LLM-as-a-Judge evaluation system using Langfuse. Score AI outputs on relevance, accuracy, hallucination, and helpfulness. Backfill scoring on historical trac...

Versionv1.0.0
#finance#legal
TERMINAL
clawhub install llm-evaluator

πŸ“– About This Skill

name: llm-evaluator description: LLM-as-a-Judge evaluation system using Langfuse. Score AI outputs on relevance, accuracy, hallucination, and helpfulness. Backfill scoring on historical traces. Uses GPT-5-nano for cost-efficient judging. Use when evaluating AI quality, building evals, or monitoring output accuracy. homepage: https://www.agxntsix.ai license: MIT compatibility: Python 3.10+, Langfuse instance, OpenRouter API key metadata: {"openclaw": {"emoji": "\u2696\ufe0f", "requires": {"env": ["OPENROUTER_API_KEY"]}, "primaryEnv": "OPENROUTER_API_KEY", "homepage": "https://www.agxntsix.ai"}}

LLM Evaluator βš–οΈ

LLM-as-a-Judge evaluation system powered by Langfuse. Uses GPT-5-nano to score AI outputs.

When to Use

  • Evaluating quality of search results or AI responses
  • Scoring traces for relevance, accuracy, hallucination detection
  • Batch scoring recent unscored traces
  • Quality assurance on agent outputs

    • Usage

    # Test with sample cases
python3 {baseDir}/scripts/evaluator.py test
    Score a specific Langfuse trace
    
python3 {baseDir}/scripts/evaluator.py score 
    Score with specific evaluator only
    
python3 {baseDir}/scripts/evaluator.py score  --evaluators relevance
    Backfill scores on recent unscored traces
    
python3 {baseDir}/scripts/evaluator.py backfill --limit 20

    Evaluators

    | Evaluator | Measures | Scale | |-----------|----------|-------| | relevance | Response relevance to query | 0–1 | | accuracy | Factual correctness | 0–1 | | hallucination | Made-up information detection | 0–1 | | helpfulness | Overall usefulness | 0–1 |

    Credits

    Built by M. Abidi | agxntsix.ai YouTube | GitHub Part of the AgxntSix Skill Suite for OpenClaw agents.

    πŸ“… Need help setting up OpenClaw for your business? Book a free consultation

    ⚑ When to Use

    TriggerAction
    - Scoring traces for relevance, accuracy, hallucination detection
    - Batch scoring recent unscored traces
    - Quality assurance on agent outputs

    πŸ’‘ Examples

    # Test with sample cases
python3 {baseDir}/scripts/evaluator.py test
    Score a specific Langfuse trace
    
python3 {baseDir}/scripts/evaluator.py score 
    Score with specific evaluator only
    
python3 {baseDir}/scripts/evaluator.py score  --evaluators relevance
    Backfill scores on recent unscored traces
    
python3 {baseDir}/scripts/evaluator.py backfill --limit 20