Scenarios

A Scenario in Evalion defines a specific testing situation or use case your AI agent will encounter during evaluation. Scenarios represent realistic user interactions and business contexts, providing structured frameworks for measuring how well your agent handles different conversation types and user needs.

Scenarios serve as the foundation for creating comprehensive test coverage by simulating the variety of situations your agent will face in production environments.

Scenario Sets

Evalion organizes related scenarios into Scenario Sets for efficient management and testing:

• Grouping related situations: Keep similar scenarios together (e.g., all booking-related scenarios in one set)
• Batch testing capabilities: Run multiple related scenarios simultaneously
• Organized test management: Easily find and modify scenarios by category
• Reusable scenario collections: Apply the same scenario set across different test suites

Important: Test suites execute entire scenario sets, not individual scenarios.

Scenario Components

Each scenario consists of several key elements that define the testing context:

1. Scenario Name

A descriptive identifier that clearly indicates the situation being tested (e.g., "New Customer Onboarding" or "Appointment Rescheduling").

2. Scenario Description

This brief overview provides a general context and purpose of the scenario, helping you understand the simulated user situation.

3. User Behavior (Definition)

Detailed description of what the simulated user persona is required to do during the interaction, such as:

• The initial request or need that the user will express.
• Information the user knows or doesn't know at the start.
• User goals and motivations driving the conversation.

4. Success Criteria (Expected Output)

Clear definition of what constitutes a successful interaction for your AI conversational agent, specifying:

• Required information that the agent must collect.
• Actions the agent should complete (e.g, bookings, transfers, confirmations) .
• Communication standards that the agent must maintain.
• Problem resolution expectations.

Scenario Role in Testing

Scenarios work with personas and metrics to create comprehensive test cases. During test runs, personas follow scenario guidelines as they interact with your agent, and the results are measured against the scenario's defined success criteria.

This combination ensures that your agent is evaluated across realistic, business-relevant situations using consistent, measurable standards that reflect real user needs and expectations.