Memory Pipeline
by @joe-rlo
Complete agent memory + performance system. Extracts structured facts, builds knowledge graphs, generates briefings, and enforces execution discipline via pre-game routines, tool policies, result compression, and after-action reviews. Includes external knowledge ingestion (ChatGPT exports, etc.) into searchable memory. Use when working on memory management, briefing generation, knowledge consolidation, external data ingestion, agent consistency, or improving execution quality across sessions.
Install
clawdhub install memory-pipeline
Setup
bash skills/memory-pipeline/scripts/setup.sh
The setup script will detect your workspace, check dependencies (Python 3 + any LLM API key), create the memory/ directory, and run the full pipeline.
Requirements
OPENAI_API_KEY or ~/.config/openai/api_key)
- Anthropic (ANTHROPIC_API_KEY or ~/.config/anthropic/api_key)
- Gemini (GEMINI_API_KEY or ~/.config/gemini/api_key)Run Manually
# Full pipeline
python3 skills/memory-pipeline/scripts/memory-extract.py
python3 skills/memory-pipeline/scripts/memory-link.py
python3 skills/memory-pipeline/scripts/memory-briefing.py
Automate via Heartbeat
Add to your HEARTBEAT.md for daily automatic runs:
### Daily Memory Pipeline
Frequency: Once per day (morning)
Action: Run the memory pipeline:
1. python3 skills/memory-pipeline/scripts/memory-extract.py
2. python3 skills/memory-pipeline/scripts/memory-link.py
3. python3 skills/memory-pipeline/scripts/memory-briefing.py
{
"enabled": true,
"briefing": {
"maxChars": 6000,
"checklist": [
"Restate the task in one sentence.",
"List constraints and success criteria.",
"Retrieve only the minimum relevant memory.",
"Prefer tools over guessing when facts matter."
],
"memoryFiles": ["memory/IDENTITY.md", "memory/PROJECTS.md"]
},
"tools": {
"deny": ["dangerous_tool"],
"maxToolResultChars": 12000
},
"afterAction": {
"writeMemoryFile": "memory/AFTER_ACTION.md",
"maxBullets": 8
}
}
Hook Details
| Hook | What it does |
|------|-------------|
| before_agent_start | Loads memory files, builds bounded briefing packet, injects into system prompt |
| before_tool_call | Checks tool against deny list, prevents unsafe calls |
| tool_result_persist | Head (60%) + tail (30%) compression of large results |
| agent_end | Appends session summary to memory file with tools used and outcomes |
| Problem | Fix | |---------|-----| | No facts extracted | Check that daily notes or transcripts exist; verify API key | | Low-quality links | Add OpenAI key for embedding-based similarity; adjust threshold | | Briefing too long | Reduce facts in template or let LLM generation handle it (auto-constrained to 2000 chars) |
clawhub install memory-pipeline