Test Suites

A Test Suite is the primary testing unit in Evalion that orchestrates all components into executable test cases. Test suites define what will be tested, how it will be tested, and what constitutes success by combining agents, scenarios, personas, and metrics into comprehensive evaluation frameworks.

Test Suite Components

Each test suite combines the following elements:

1. Agent Selection

During the test run, the AI agent, including its configuration, connection method, and behavioral guidelines, will be evaluated.

2. Scenario Sets

Collections of related scenarios that define the situations and use cases your agent will encounter during testing.

3. Personas

User personality types that will interact with your agent, ensuring evaluation across different communication styles and behavioral patterns.

4. Metrics

Success criteria used to evaluate agent performance, including custom semantic and built-in technical metrics.

Test Suite Configuration

When creating a test suite, define these key parameters:

Suite Information

• Name: Descriptive identifier for the test suite (e.g., "Booking Flow Validation").
• Description: Brief overview of the testing objectives and scope.

Component Selection

• Scenario sets: Choose one or more scenario collections relevant to your testing goals.
• Personas: Select multiple personality types to ensure diverse interaction coverage.
• Metrics: Include custom semantic metrics alongside automatic technical measurements.

Test Suite Role in Testing

Test suites serve as the execution framework that transforms individual testing components into comprehensive evaluation processes. When a test suite runs, it generates multiple simulations by combining each scenario with each selected persona, creating a matrix of test cases that thoroughly evaluate your agent's performance across diverse conditions.