Skip to main content
Evaluation helps you measure the quality of your RAG pipeline and identify areas for improvement. LlamaIndex provides evaluators for faithfulness, relevancy, and correctness.

​
Overview

All evaluators implement the BaseEvaluator interface: 
interface BaseEvaluator {
  evaluate(params: EvaluatorParams): Promise<EvaluationResult>;
  evaluateResponse?(params: EvaluatorResponseParams): Promise<EvaluationResult>;
}
Evaluators return an EvaluationResult: 
type EvaluationResult = {
  query?: string;
  contexts?: string[];
  response: string;
  score: number;        // Numeric score (0-1 or 0-5)
  passing: boolean;     // Whether evaluation passed
  feedback: string;     // Detailed feedback from evaluator
};

​
Faithfulness

What it measures: Whether the response is grounded in the provided context. Faithfulness checks if the answer contains hallucinations or makes claims not supported by the source documents. 
import { FaithfulnessEvaluator } from "llamaindex/evaluation";

const evaluator = new FaithfulnessEvaluator({
  raiseError: false, // Don't throw on failing evaluation
});

const result = await evaluator.evaluate({
  query: "What is LlamaIndex?",
  response: "LlamaIndex is a data framework for LLM applications.",
  contexts: [
    "LlamaIndex is a data framework for building LLM applications.",
    "It provides tools for data ingestion, indexing, and querying.",
  ],
});

console.log(result.passing); // true
console.log(result.score);   // 1.0
console.log(result.feedback); // "Yes"

​
Evaluate Response Objects

Directly evaluate query engine responses: 
const response = await queryEngine.query({
  query: "What is LlamaIndex?",
});

const result = await evaluator.evaluateResponse({
  query: "What is LlamaIndex?",
  response: response,
});

​
Custom Prompts

import { PromptTemplate } from "@llamaindex/core/prompts";

const faithfulnessPrompt = new PromptTemplate({
  template: `Context: {context}

Response: {query}

Is the response faithful to the context? Answer yes or no.`,
});

const evaluator = new FaithfulnessEvaluator({
  faithfulnessSystemPrompt: faithfulnessPrompt,
});

​
Relevancy

What it measures: Whether the response actually answers the question. Relevancy checks if the response addresses the user’s query. 
import { RelevancyEvaluator } from "llamaindex/evaluation";

const evaluator = new RelevancyEvaluator();

const result = await evaluator.evaluate({
  query: "What is the capital of France?",
  response: "Paris is the capital of France.",
  contexts: ["Paris is the capital and largest city of France."],
});

console.log(result.passing); // true
console.log(result.score);   // 1.0

​
How It Works

Relevancy uses an LLM to determine if the response answers the question:
  1. Formats query and response together
  2. Queries a SummaryIndex of the contexts
  3. LLM answers “yes” or “no”
  4. Returns score (1.0 for yes, 0.0 for no)

​
Correctness

What it measures: How correct the response is compared to a reference answer. Correctness requires a reference (ground truth) answer: 
import { CorrectnessEvaluator } from "llamaindex/evaluation";

const evaluator = new CorrectnessEvaluator({
  scoreThreshold: 4.0, // Passing score threshold
});

const result = await evaluator.evaluate({
  query: "What is 2+2?",
  response: "2+2 equals 4",
  reference: "The answer is 4",
});

console.log(result.score);   // 5.0 (scale of 1-5)
console.log(result.passing); // true (>= 4.0)
console.log(result.feedback); // Reasoning for the score

​
Score Scale

Correctness uses a 1-5 scale:
  • 5 - Perfect match
  • 4 - Correct with minor differences
  • 3 - Partially correct
  • 2 - Mostly incorrect
  • 1 - Completely incorrect

​
Custom Parser

Parse LLM responses differently: 
function customParser(response: string): [number, string] {
  // Extract score and reasoning from response
  const scoreMatch = response.match(/Score: (\d+)/);
  const score = scoreMatch ? parseInt(scoreMatch[1]) : 0;
  const reasoning = response.split("\n").slice(1).join("\n");
  return [score, reasoning];
}

const evaluator = new CorrectnessEvaluator({
  parserFunction: customParser,
});

​
Batch Evaluation

Evaluate multiple queries: 
import {
  FaithfulnessEvaluator,
  RelevancyEvaluator,
  CorrectnessEvaluator,
} from "llamaindex/evaluation";

const testCases = [
  {
    query: "What is LlamaIndex?",
    reference: "LlamaIndex is a data framework for LLMs",
  },
  {
    query: "How do I create an index?",
    reference: "Use VectorStoreIndex.fromDocuments()",
  },
];

const faithfulness = new FaithfulnessEvaluator();
const relevancy = new RelevancyEvaluator();
const correctness = new CorrectnessEvaluator();

const results = [];

for (const testCase of testCases) {
  const response = await queryEngine.query({
    query: testCase.query,
  });

  const [faithResult, relResult, corrResult] = await Promise.all([
    faithfulness.evaluateResponse({ query: testCase.query, response }),
    relevancy.evaluateResponse({ query: testCase.query, response }),
    correctness.evaluate({
      query: testCase.query,
      response: response.toString(),
      reference: testCase.reference,
    }),
  ]);

  results.push({
    query: testCase.query,
    faithfulness: faithResult.score,
    relevancy: relResult.score,
    correctness: corrResult.score,
    passing: faithResult.passing && relResult.passing && corrResult.passing,
  });
}

// Calculate averages
const avgFaithfulness = results.reduce((sum, r) => sum + r.faithfulness, 0) / results.length;
const avgRelevancy = results.reduce((sum, r) => sum + r.relevancy, 0) / results.length;
const avgCorrectness = results.reduce((sum, r) => sum + r.correctness, 0) / results.length;

console.log({
  avgFaithfulness,
  avgRelevancy,
  avgCorrectness,
  passRate: results.filter(r => r.passing).length / results.length,
});

​
Rate Limiting

Avoid API rate limits: 
const results = [];

for (const testCase of testCases) {
  const result = await evaluator.evaluate({
    query: testCase.query,
    response: testCase.response,
    contexts: testCase.contexts,
    sleepTimeInSeconds: 1, // Wait 1 second between calls
  });
  results.push(result);
}

​
Evaluation Pipeline

Create a comprehensive evaluation workflow: 
class RAGEvaluationPipeline {
  constructor(
    private queryEngine: any,
    private evaluators = {
      faithfulness: new FaithfulnessEvaluator(),
      relevancy: new RelevancyEvaluator(),
      correctness: new CorrectnessEvaluator(),
    }
  ) {}

  async evaluate(testCases: Array<{
    query: string;
    reference?: string;
  }>) {
    const results = [];

    for (const testCase of testCases) {
      const response = await this.queryEngine.query({
        query: testCase.query,
      });

      const evalPromises = [
        this.evaluators.faithfulness.evaluateResponse({
          query: testCase.query,
          response,
        }),
        this.evaluators.relevancy.evaluateResponse({
          query: testCase.query,
          response,
        }),
      ];

      if (testCase.reference) {
        evalPromises.push(
          this.evaluators.correctness.evaluate({
            query: testCase.query,
            response: response.toString(),
            reference: testCase.reference,
          })
        );
      }

      const [faithfulness, relevancy, correctness] = await Promise.all(evalPromises);

      results.push({
        query: testCase.query,
        response: response.toString(),
        scores: {
          faithfulness: faithfulness.score,
          relevancy: relevancy.score,
          correctness: correctness?.score,
        },
        passing: {
          faithfulness: faithfulness.passing,
          relevancy: relevancy.passing,
          correctness: correctness?.passing ?? true,
        },
        feedback: {
          faithfulness: faithfulness.feedback,
          relevancy: relevancy.feedback,
          correctness: correctness?.feedback,
        },
      });
    }

    return this.summarize(results);
  }

  private summarize(results: any[]) {
    return {
      results,
      summary: {
        total: results.length,
        passed: results.filter(r => 
          r.passing.faithfulness && 
          r.passing.relevancy && 
          r.passing.correctness
        ).length,
        avgScores: {
          faithfulness: this.average(results.map(r => r.scores.faithfulness)),
          relevancy: this.average(results.map(r => r.scores.relevancy)),
          correctness: this.average(
            results.map(r => r.scores.correctness).filter(Boolean)
          ),
        },
      },
    };
  }

  private average(numbers: number[]) {
    return numbers.reduce((sum, n) => sum + n, 0) / numbers.length;
  }
}

// Usage
const pipeline = new RAGEvaluationPipeline(queryEngine);

const evaluation = await pipeline.evaluate([
  { query: "What is LlamaIndex?", reference: "A data framework" },
  { query: "How do I use it?", reference: "Import and create an index" },
]);

console.log(evaluation.summary);

​
Best Practices

Test Set Creation:
  • Create diverse test cases covering different query types
  • Include edge cases and common failure modes
  • Use real user queries when possible
  • Maintain reference answers for correctness evaluation
Metric Selection:
  • Faithfulness - Critical for preventing hallucinations
  • Relevancy - Ensures responses answer the question
  • Correctness - Requires reference answers, best for regression testing
Iteration:
  1. Establish baseline scores
  2. Make changes (prompts, retrievers, etc.)
  3. Re-run evaluation
  4. Compare scores to baseline
  5. Keep improvements, discard regressions
Performance:
  • Run evaluations in parallel when possible
  • Cache LLM responses to avoid redundant calls
  • Use rate limiting to avoid API errors

​
Next Steps

Postprocessors

Improve retrieval quality with filtering and reranking

Memory

Manage conversation context and history