A notebook demonstrating how to monitor, analyze, and optimize Retrieval-Augmented Generation (RAG) agents in production using Contextual AI's Metrics API. Learn to track performance metrics, analyze user feedback, and ensure consistent, high-quality responses.
This example showcases how to build a complete monitoring and analytics dashboard for RAG agents that can:
- Extract Real-time Metrics from the Contextual AI Metrics API
- Analyze User Feedback across conversations
- Monitor Response Quality with content analysis
- Visualize Usage Patterns through interactive time series analysis
- Generate Actionable Insights for performance optimization and capacity planning
📁 RAG Agent Monitoring/
├── 📁 data/ # Sample monitoring data
│ └── 📄 synthetic_data.csv # Demo dataset with 441 conversations
├── 📓 monitoring_intro.ipynb # Main monitoring notebook
└── 📄 README.md # This file
- API Key: Contextual AI API key with metrics access permissions
- Python Environment: Google Colab or Jupyter with internet access
- Dependencies:
contextual-client,pandas,plotly,matplotlib,seaborn - Active RAG Agent: At least one deployed agent with conversation history (optional for demo)
- 🔗 RAGAS Evaluation: 07-evaluation-ragas
- 🔗 Retrieval Analysis: 11-retrieval-analysis
- 🔗 LMUnit Evaluation: 03-standalone-api/01-lmunit
- 🔗 Agent Performance: 06-improve-agent-performance
- Contextual AI Documentation: docs.contextual.ai
- Metrics API Reference: API Documentation
- RAGAS Framework: GitHub Repository
Ready to start monitoring? Open monitoring_intro.ipynb and build your RAG agent analytics dashboard in under 30 minutes!