Self-Improving AI
by @jose-compu
Captures learnings about GenAI/LLM configuration, model selection, inference optimization, fine-tuning, RAG pipelines, prompt engineering, multimodal process...
1. Pin model versions β never use "latest" in production 2. Benchmark before switching models β gut feeling is not evaluation 3. Measure cost per task, not just per token β task-level cost captures retries and multi-turn 4. Cache aggressively β same prompt + same input = same output 5. Use structured output (JSON mode) when downstream parsing is needed 6. Separate system prompt from user context for cleaner caching 7. Test multimodal inputs with edge cases β blurry images, noisy audio, long video 8. Keep eval sets versioned alongside fine-tuned models 9. Log temperature/parameters with every issue β reproducibility matters 10. Set up fallback chains β provider outages happen 11. Monitor embedding drift after source data changes 12. Review guardrail logs monthly for false positive trends
clawhub install self-improving-ai