🎁 Get the FREE AI Skills Starter Guide β€” Subscribe β†’
BytesAgainBytesAgain
BytesAgain β€Ί Skills β€Ί arxivkb
πŸ¦€ ClawHub

arxivkb

by @camopel

Local arXiv paper manager with semantic search. Crawls arXiv categories, downloads PDFs, chunks content, and indexes with FAISS + Ollama embeddings. No cloud...

Versionv1.0.1
Downloads1,313
#crypto-defi#devops#research#writing#legal
TERMINAL
clawhub install arxivkb

πŸ“– About This Skill

name: arxivkb description: "Local arXiv paper manager with semantic search. Crawls arXiv categories, downloads PDFs, chunks content, and indexes with FAISS + Ollama embeddings. No cloud API keys required β€” everything runs locally." metadata: {"openclaw":{"requires":{"bins":["python3","ollama"]}}}

ArXivKB β€” Science Knowledge Base

Why This Skill?

🏠 100% local β€” crawls arXiv's free API, embeds with Ollama (nomic-embed-text), indexes in FAISS + SQLite. No cloud cost.

πŸ” Semantic search on paper content β€” FAISS indexes PDF chunks (not just abstracts), so you find papers by what they contain.

πŸ“‚ arXiv category-based β€” tracks official arXiv categories (155 available, 8 groups). No free-text queries.

🧹 Auto-cleanup β€” configurable expiry deletes old papers, PDFs, and chunks.

Install

python3 scripts/install.py

Works on macOS and Linux. Installs Python deps (faiss-cpu, pdfplumber, tiktoken, arxiv, numpy), pulls nomic-embed-text via Ollama, creates data directories and DB.

Prerequisites

  • Ollama β€” must be installed and running (ollama serve)
  • Python 3.10+

    • Quick Start

    # 1. Add arXiv categories to track
akb categories add cs.AI cs.CV cs.LG
    2. Browse all available categories
    
akb categories browse
    3. Ingest recent papers (last 7 days)
    
akb ingest
    4. Check stats
    
akb stats

    Categories

    akb categories list                    # Show enabled categories
akb categories browse                  # Browse all 155 arXiv categories
akb categories browse robotics         # Filter by keyword
akb categories add cs.AI cs.RO         # Enable categories
akb categories delete cs.AI            # Disable a category

    Categories are official arXiv codes (e.g. cs.AI, eess.IV, q-fin.ST). The full taxonomy is built in.

    Ingestion

    akb ingest                    # Crawl, download PDFs, chunk, embed
akb ingest --days 14          # Look back 14 days
akb ingest --dry-run          # Preview only
akb ingest --no-pdf           # Index abstracts only (faster)

    Pipeline: arXiv API β†’ PDF download β†’ text extraction (pdfplumber) β†’ chunking (tiktoken, 500 tokens, 50 overlap) β†’ embedding (Ollama nomic-embed-text) β†’ FAISS + SQLite.

    Paper Details

    akb paper 2401.12345    # Show title, abstract, categories, PDF status

    Statistics

    akb stats   # Papers, chunks, categories, DB size

    Expiry & Cleanup

    akb expire               # Delete papers older than 90 days (default)
akb expire --days 30     # Override: delete papers older than 30 days
akb expire --days 30 -y  # Skip confirmation

    Configuration

    No config file needed. Defaults:

    | Setting | Default | Override | |---------|---------|----------| | Data directory | ~/workspace/arxivkb | ARXIVKB_DATA_DIR env or --data-dir | | Ollama endpoint | http://localhost:11434 | β€” (hardcoded) | | Embedding model | nomic-embed-text (768d) | β€” (hardcoded) | | Chunk size | 500 tokens, 50 overlap | β€” | | Expiry | 90 days | --days flag |

    Data Layout

    ~/workspace/arxivkb/
β”œβ”€β”€ arxivkb.db           # SQLite: papers, chunks, translations, categories
β”œβ”€β”€ pdfs/                  # Downloaded PDF files ({arxiv_id}.pdf)
└── faiss/
    └── arxivkb.faiss    # FAISS IndexFlatIP (chunk embeddings)

    DB Schema

  • papers: id, arxiv_id, title, abstract, categories, published, status, created_at
  • chunks: id, paper_id, section, chunk_index, text, faiss_id, created_at
  • translations: paper_id, language, abstract, created_at (PK: paper_id+language)
  • categories: code, description, group_name, enabled, added_at (155 entries)

    • πŸ’¬ Chat Commands (OpenClaw Agent)

    When this skill is installed, the agent recognizes /akb as a shortcut:

    | Command | Action | |---------|--------| | /akb list | Show enabled categories | | /akb add cs.AI cs.RO | Enable categories for crawling | | /akb remove cs.AI | Disable a category | | /akb browse | Browse all 155 arXiv categories | | /akb browse robotics | Filter categories by keyword | | /akb stats | Show paper/chunk/category counts | | /akb help | Show available commands |

    The agent runs these via the akb CLI internally.

    πŸ“± PrivateApp Dashboard

    A companion PWA dashboard is available. Provides:

  • Semantic search across paper content
  • Paper detail with abstract translation (on-demand via LLM)
  • Inline PDF viewing
  • Category browser
  • Stats (papers, chunks, categories)

    • Architecture

    scripts/
β”œβ”€β”€ cli.py             # CLI β€” categories, ingest, paper, stats, expire
β”œβ”€β”€ db.py              # SQLite schema + CRUD
β”œβ”€β”€ arxiv_crawler.py   # arXiv API search + PDF download
β”œβ”€β”€ arxiv_taxonomy.py  # Full arXiv category taxonomy (155 categories)
β”œβ”€β”€ pdf_processor.py   # PDF text extraction + tiktoken chunking
β”œβ”€β”€ embed.py           # Ollama nomic-embed-text (768d, normalized)
β”œβ”€β”€ faiss_index.py     # FAISS IndexFlatIP manager
β”œβ”€β”€ search.py          # Semantic search: query β†’ FAISS β†’ group by paper
└── install.py         # One-command installer

    πŸ’‘ Examples

    # 1. Add arXiv categories to track
akb categories add cs.AI cs.CV cs.LG
    2. Browse all available categories
    
akb categories browse
    3. Ingest recent papers (last 7 days)
    
akb ingest
    4. Check stats
    
akb stats

    βš™οΈ Configuration

    No config file needed. Defaults:

    | Setting | Default | Override | |---------|---------|----------| | Data directory | ~/workspace/arxivkb | ARXIVKB_DATA_DIR env or --data-dir | | Ollama endpoint | http://localhost:11434 | β€” (hardcoded) | | Embedding model | nomic-embed-text (768d) | β€” (hardcoded) | | Chunk size | 500 tokens, 50 overlap | β€” | | Expiry | 90 days | --days flag |