Llm Chain

LLM Chain

An AI toolkit for configuring, benchmarking, comparing, prompting, evaluating, fine-tuning, analyzing, and optimizing LLM workflows. Each command logs timestamped entries to local files with full export, search, and statistics support.

Commands

Core AI Operations

| Command | Description | |---------|-------------| | llm-chain configure | Record a configuration change (or view recent configs with no args) | | llm-chain benchmark | Log a benchmark run and its results | | llm-chain compare | Record a model or output comparison | | llm-chain prompt | Log a prompt template or prompt engineering note | | llm-chain evaluate | Record an evaluation result or metric | | llm-chain fine-tune | Log a fine-tuning session or parameters | | llm-chain analyze | Record an analysis observation | | llm-chain cost | Log cost tracking data (tokens, dollars, etc.) | | llm-chain usage | Record API usage metrics | | llm-chain optimize | Log an optimization attempt and outcome | | llm-chain test | Record a test case or test result | | llm-chain report | Log a report entry or summary |

Utility Commands

| Command | Description | |---------|-------------| | llm-chain stats | Show summary statistics across all log files | | llm-chain export | Export all data in json, csv, or txt format | | llm-chain search | Search all entries for a keyword (case-insensitive) | | llm-chain recent | Show the 20 most recent activity log entries | | llm-chain status | Health check: version, entry count, disk usage, last activity | | llm-chain help | Display full command reference | | llm-chain version | Print current version (v2.0.0) |

How It Works

Every core command accepts free-text input. When called with arguments, LLM Chain:

1. Timestamps the entry (YYYY-MM-DD HH:MM) 2. Appends it to the command-specific log file (e.g. benchmark.log, cost.log) 3. Records the action in a central history.log 4. Reports the saved entry and running total

When called with no arguments, each command displays the 20 most recent entries from its log file.

Data Storage

All data is stored locally in plain-text log files:

~/.local/share/llm-chain/
├── configure.log     # Configuration changes
├── benchmark.log     # Benchmark results
├── compare.log       # Model comparisons
├── prompt.log        # Prompt templates & notes
├── evaluate.log      # Evaluation metrics
├── fine-tune.log     # Fine-tuning sessions
├── analyze.log       # Analysis observations
├── cost.log          # Cost tracking
├── usage.log         # API usage metrics
├── optimize.log      # Optimization attempts
├── test.log          # Test cases & results
├── report.log        # Report entries
├── history.log       # Central activity log
└── export.{json,csv,txt}  # Exported snapshots

Each log uses pipe-delimited format: timestamp|value.

Requirements

When to Use

1. Tracking LLM experiments — log benchmark results, prompt variations, and evaluation scores as you iterate on model configurations 2. Cost monitoring — record token usage, API costs, and billing data to keep spending under control across multiple models 3. Comparing models side-by-side — use compare and benchmark to log performance differences between GPT-4, Claude, Gemini, etc. 4. Fine-tuning documentation — capture fine-tuning parameters, dataset info, and results for reproducibility 5. Generating operational reports — export all logged data to JSON/CSV for dashboards, audits, or stakeholder reviews

Examples

# Log a configuration change
llm-chain configure "switched to gpt-4o, temperature=0.7, max_tokens=2048"
Record a benchmark result
llm-chain benchmark "gpt-4o MMLU=87.2% latency=1.3s cost=$0.012/req"
Track a cost entry
llm-chain cost "2024-03-18: 142k tokens, $4.26 total (gpt-4o)"
Compare two models
llm-chain compare "claude-3.5 vs gpt-4o: claude wins on reasoning, gpt wins on speed"
Log a prompt engineering note
llm-chain prompt "added chain-of-thought prefix: 'Let me think step by step...'"
Search all logs for a keyword
llm-chain search "gpt-4o"
Export everything to JSON
llm-chain export json
Check health and disk usage
llm-chain status

Configuration

Set the DATA_DIR variable in the script or modify the default path to change storage location. Default: ~/.local/share/llm-chain/

Powered by BytesAgain | bytesagain.com | hello@bytesagain.com

# Log a configuration change
llm-chain configure "switched to gpt-4o, temperature=0.7, max_tokens=2048"
Record a benchmark result
llm-chain benchmark "gpt-4o MMLU=87.2% latency=1.3s cost=$0.012/req"
Track a cost entry
llm-chain cost "2024-03-18: 142k tokens, $4.26 total (gpt-4o)"
Compare two models
llm-chain compare "claude-3.5 vs gpt-4o: claude wins on reasoning, gpt wins on speed"
Log a prompt engineering note
llm-chain prompt "added chain-of-thought prefix: 'Let me think step by step...'"
Search all logs for a keyword
llm-chain search "gpt-4o"
Export everything to JSON
llm-chain export json
Check health and disk usage
llm-chain status

Set the DATA_DIR variable in the script or modify the default path to change storage location. Default: ~/.local/share/llm-chain/

Powered by BytesAgain | bytesagain.com | hello@bytesagain.com