🎁 Get the FREE AI Skills Starter Guide β€” Subscribe β†’
BytesAgainBytesAgain
BytesAgain β€Ί Skills β€Ί Peer Review
πŸ¦€ ClawHub

Peer Review

by @staybased

Multi-model peer review layer using local LLMs via Ollama to catch errors in cloud model output. Fan-out critiques to 2-3 local models, aggregate flags, synthesize consensus. Use when: validating trade analyses, reviewing agent output quality, testing local model accuracy, checking any high-stakes Claude output before publishing or acting on it. Don't use when: simple fact-checking (just search the web), tasks that don't benefit from multi-model consensus, time-critical decisions where 60s lat

Versionv1.0.0
Downloads1,447
Installs20
#crypto-defi#devops#legal#gaming#productivity
TERMINAL
clawhub install peer-review

πŸ“– About This Skill

name: peer-review description: | Multi-model peer review layer using local LLMs via Ollama to catch errors in cloud model output. Fan-out critiques to 2-3 local models, aggregate flags, synthesize consensus.

Use when: validating trade analyses, reviewing agent output quality, testing local model accuracy, checking any high-stakes Claude output before publishing or acting on it.

Don't use when: simple fact-checking (just search the web), tasks that don't benefit from multi-model consensus, time-critical decisions where 60s latency is unacceptable, reviewing trivial or low-stakes content.

Negative examples: - "Check if this date is correct" β†’ No. Just web search it. - "Review my grocery list" β†’ No. Not worth multi-model inference. - "I need this answer in 5 seconds" β†’ No. Peer review adds 30-60s latency.

Edge cases: - Short text (<50 words) β†’ Models may not find meaningful issues. Consider skipping. - Highly technical domain β†’ Local models may lack domain knowledge. Weight flags lower. - Creative writing β†’ Factual review doesn't apply well. Use only for logical consistency. version: "1.0"

Peer Review β€” Local LLM Critique Layer

> Hypothesis: Local LLMs can catch β‰₯30% of real errors in cloud output with <50% false positive rate.

Architecture

Cloud Model (Claude) produces analysis
        β”‚
        β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚   Peer Review Fan-Out  β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚  Drift (Mistral 7B)   │──► Critique A
β”‚  Pip (TinyLlama 1.1B) │──► Critique B
β”‚  Lume (Llama 3.1 8B)  │──► Critique C
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
        β”‚
        β–Ό
  Aggregator (consensus logic)
        β”‚
        β–Ό
  Final: original + flagged issues

Swarm Bot Roles

| Bot | Model | Role | Strengths | |-----|-------|------|-----------| | Drift 🌊 | Mistral 7B | Methodical analyst | Structured reasoning, catches logical gaps | | Pip 🐣 | TinyLlama 1.1B | Fast checker | Quick sanity checks, low latency | | Lume πŸ’‘ | Llama 3.1 8B | Deep thinker | Nuanced analysis, catches subtle issues |

Scripts

| Script | Purpose | |--------|---------| | scripts/peer-review.sh | Send single input to all models, collect critiques | | scripts/peer-review-batch.sh | Run peer review across a corpus of samples | | scripts/seed-test-corpus.sh | Generate seeded error corpus for testing |

Usage

# Single file review
bash scripts/peer-review.sh  [output_dir]
Batch review

bash scripts/peer-review-batch.sh  [results_dir]
Generate test corpus

bash scripts/seed-test-corpus.sh [count] [output_dir]

Scripts live at workspace/scripts/ β€” not bundled in skill to avoid duplication.

Critique Prompt Template

You are a skeptical reviewer. Analyze the following text for errors.
For each issue found, output JSON:
{"category": "factual|logical|missing|overconfidence|hallucinated_source",
 "quote": "...", "issue": "...", "confidence": 0-100}
If no issues found, output: {"issues": []}
TEXT:


{cloud_output}

Error Categories

| Category | Description | Example | |----------|-------------|---------| | factual | Wrong numbers, dates, names | "Bitcoin launched in 2010" | | logical | Non-sequiturs, unsupported conclusions | "X is rising, therefore Y will fall" | | missing | Important context omitted | Ignoring a major counterargument | | overconfidence | Certainty without justification | "This will definitely happen" on 55% event | | hallucinated_source | Citing nonexistent sources | "According to a 2024 Reuters report..." |

Discord Workflow

1. Post analysis to #the-deep (or #swarm-lab) 2. Drift, Pip, and Lume respond with independent critiques 3. Celeste synthesizes: deduplicates flags, weights by model confidence 4. If consensus (β‰₯2 models agree) β†’ flag is high-confidence 5. Final output posted with recommendation: publish | revise | flag_for_human

Success Criteria

| Outcome | TPR | FPR | Decision | |---------|-----|-----|----------| | Strong pass | β‰₯50% | <30% | Ship as default layer | | Pass | β‰₯30% | <50% | Ship as opt-in layer | | Marginal | 20–30% | 50–70% | Iterate on prompts, retest | | Fail | <20% | >70% | Abandon approach |

Scoring Rules

  • Flag = true positive if it identifies a real error (even if explanation is imperfect)
  • Flag = false positive if flagged content is actually correct
  • Duplicate flags across models count once for TPR but inform consensus metrics

    • Dependencies

  • Ollama running locally with models pulled: mistral:7b, tinyllama:1.1b, llama3.1:8b
  • jq and curl installed
  • Results stored in experiments/peer-review-results/

    • Integration

    When peer review passes validation:

  • Package as Reef API endpoint: POST /review
  • Agents call before publishing any analysis
  • Configurable: model selection, consensus threshold, categories
  • Log all reviews to #reef-logs with TPR tracking

    • πŸ’‘ Examples

    # Single file review
bash scripts/peer-review.sh  [output_dir]
    Batch review
    
bash scripts/peer-review-batch.sh  [results_dir]
    Generate test corpus
    
bash scripts/seed-test-corpus.sh [count] [output_dir]

    Scripts live at workspace/scripts/ β€” not bundled in skill to avoid duplication.