🎁 Get the FREE AI Skills Starter Guide β€” Subscribe β†’
BytesAgainBytesAgain
BytesAgain β€Ί Skills β€Ί Eidolon Search
πŸ¦€ ClawHub

Eidolon Search

by @dev-jslee

AI Agent memory search using SQLite FTS5. 90%+ token reduction (10x+) compared to reading full files. Use when the agent needs to search through markdown mem...

Versionv0.0.1
Downloads816
#crypto-defi#database#legal
TERMINAL
clawhub install eidolon-search

πŸ“– About This Skill

name: eidolon-search description: AI Agent memory search using SQLite FTS5. 90%+ token reduction (10x+) compared to reading full files. Use when the agent needs to search through markdown memory files, daily notes, or any text corpus efficiently. Triggers on memory search, file search, knowledge retrieval, or when context window is limited and full-file reading is too expensive.

Eidolon Search

FTS5-based memory search for AI Agents. Index markdown files once, search with 90%+ token savings.

Quick Start

1. Index memory files (once)

python3 scripts/build-index.py

Example: 

python3 scripts/build-index.py ./memory ./memory.db

This creates a SQLite database with FTS5 full-text index of all .md files in the directory (recursive).

2. Search

python3 scripts/search.py  [limit] [db_path]

Example: 

python3 scripts/search.py "Physical AI roadmap" 5
python3 scripts/search.py "project timeline" 10 ./memory.db

Default limit: 10. Default db_path: ./memory.db

Output: matching snippets with file paths and relevance scores.

3. Re-index when files change

Run build-index.py again. It rebuilds the index from scratch (fast, <1 second for typical workspaces).

When to Use

  • Memory search: Find specific information across many daily notes or memory files
  • Token-limited contexts: When reading all files would exceed context limits
  • Repeated searches: Index once, search many times
  • Large workspaces: 10+ markdown files with cumulative size >50KB

    • When NOT to Use

  • Single small file (<5KB): just read it directly
  • Need semantic/meaning-based search: FTS5 is keyword-based only
  • See references/QUALITY.md for known limitations

    • Search Tips for Agents

    FTS5 is keyword-based. Improve results by:

  • Use specific terms: "Jetson Orin" not "hardware plans"
  • Use OR for synonyms: "car OR vehicle OR automobile"
  • Use quotes for phrases: "Physical AI"
  • Try multiple queries if first attempt returns nothing
  • Check file paths in results to read full context when needed

    • Benchmarks

  • Token savings: 90%+ (measured 93-98.9%)
  • Speed: 15x faster (measured 10-20x)
  • Details: references/PERFORMANCE.md

    • Run benchmarks yourself: 

    python3 scripts/benchmark-recall.py    # Recall@5, Recall@10
python3 scripts/benchmark-cache.py     # Warm vs cold cache

    DB Schema

    CREATE VIRTUAL TABLE memory_fts USING fts5(path, content);

    Direct SQL access: 

    sqlite3 memory.db "SELECT path, snippet(memory_fts, 1, '', '', '...', 32) FROM memory_fts WHERE memory_fts MATCH 'query' ORDER BY rank LIMIT 5;"

    ⚑ When to Use

    TriggerAction
    - **Token-limited contexts**: When reading all files would exceed context limits
    - **Repeated searches**: Index once, search many times
    - **Large workspaces**: 10+ markdown files with cumulative size >50KB

    πŸ’‘ Examples

    1. Index memory files (once)

    python3 scripts/build-index.py

    Example: 

    python3 scripts/build-index.py ./memory ./memory.db

    This creates a SQLite database with FTS5 full-text index of all .md files in the directory (recursive).

    2. Search

    python3 scripts/search.py  [limit] [db_path]

    Example: 

    python3 scripts/search.py "Physical AI roadmap" 5
python3 scripts/search.py "project timeline" 10 ./memory.db

    Default limit: 10. Default db_path: ./memory.db

    Output: matching snippets with file paths and relevance scores.

    3. Re-index when files change

    Run build-index.py again. It rebuilds the index from scratch (fast, <1 second for typical workspaces).