Adding a New LLM Judge

LLM judges evaluate outputs using a prompt.

Structure

A judge config:

{
  "name": "safety_judge",
  "model_config": "configs/models/deepseek-r1.json",
  "prompt_path": "configs/judges/prompts/safety.txt",
  "debug": true
}

Prompt Example

You are an evaluator.

Question: $question
Expected answer: $expected_answer
Generated answer: $generated_answer

Score from 1-5.

Key Notes

Prompts use string.Template
No JSON escaping needed
Judges return structured outputs: - score - reason