Chatbot Performance Evaluator

A modular framework for benchmarking, validating, and monitoring chatbot behavior.

Github repo https://github.com/andalenavals/chatbot_performance_evaluator/

Documentation

• Overview
  • Why this project exists
  • What the framework provides
  • Why FAQ evaluation matters
  • Business value
  • Evaluation philosophy
• Installation
• Testing
• Demo
  • Hosted demo
  • Local demo
• Metrics and Mathematical Foundations
  • Semantic search in the strict semantic-match bot
  • Full-context prompting
  • Deterministic metrics
  • LLM-as-a-judge metrics
  • Interpreting metric families together
• API reference
  • Core modules
  • Bot modules
  • Configuration modules
  • Metric modules
  • I/O and utility modules
• Deployment
  • Hugging Face Spaces demo
  • GitHub Pages documentation
• Adding a New Bot
  • Steps to add a new bot
• Adding a New LLM Judge
  • Structure
  • Prompt Example
  • Key Notes
• Default Prompts
  • Bot Prompts
  • Judge Prompts
  • Notes