Monitor Agent System

Monitor Agents are external voice AI agents from platforms like VAPI or ElevenLabs that are integrated into Evalion purely for observability purposes. Unlike testing agents, monitor agents cannot initiate conversations but allow you to track and evaluate real conversations from external platforms.

Core Workflow

1. Agent Creation

When creating a monitor agent, Evalion performs the following steps:

1. Validates Credentials: Verifies the provided API key with the external platform
2. Fetches Agent Details: Retrieves agent information (name) from the platform API
3. Stores Configuration: Saves the external agent ID and API credentials securely
4. Links to Project: Associates the monitor agent with your Evalion project for organization

2. Historical Data Import

Monitor agents support importing past conversations from the external platform:

Preview Mode

• Fetches calls from the external platform without importing
• Allows you to review conversation metadata before importing
• Useful for understanding data volume and quality

Import Mode

• Downloads historical calls from the external platform
• Includes duplicate detection (idempotency based on external call ID)
• Supports date range filtering for targeted imports

Key Features:

• Automatic deduplication prevents importing the same conversation twice
• Batch processing for efficient large-scale imports
• Preserves original timestamps and metadata from the external platform

3. Real-time Monitoring

Once configured with webhooks, monitor agents automatically process new conversations:

1. Webhook Reception: External platform sends event notifications for new calls
2. Data Processing: Evalion receives and validates the conversation data
3. Automatic Evaluation: Applies your configured metrics to evaluate the conversation

Supported Integration Platforms

Evalion supports multiple integration types:

1. VAPI

Platform: Voice AI platform focused on telephony and voice interactions

Capabilities:

• Full transcript access with speaker labels
• Detailed cost breakdown per call
• Recording URLs for audio playback
• Message-level analysis and timestamps
• Call summary and analysis data

Data Includes:

• Transcript with timing information
• Cost metrics (per-minute, total)
• Message array with role and content
• Recording URLs
• Call metadata (duration, status, outcome)

2. ElevenLabs

Platform: Conversational AI platform with advanced voice synthesis

Capabilities:

• Conversation summaries and metadata
• Agent configuration details
• Call ratings and quality metrics
• Conversation direction (inbound/outbound)

Data Includes:

• Call duration and timing
• Agent details and configuration
• Rating information
• Conversation summary
• Direction (inbound/outbound)

Note: ElevenLabs uses a conversation-based structure rather than VAPI's call-based structure.

3. Custom

Platform: Custom data import and bulk audio upload

Purpose: Import custom conversation data or bulk upload audio recordings for evaluation

Capabilities:

• Import custom conversation data from any source
• Bulk upload audio recordings for analysis
• Automatically creates or associates with an agent behind the scenes
• Flexible data structure to accommodate various conversation formats

Use Cases:

• Migrating historical conversation data from legacy systems
• Importing recordings from call centers or custom telephony solutions
• Batch processing of audio files for evaluation
• Integrating with platforms not natively supported

Data Requirements:

• Audio files (for bulk upload scenarios)
• Conversation metadata (timestamps, participants, etc.)
• Optional transcript data if available

4. Evalion

Platform: Internal Evalion agents

Purpose: Agents created natively within Evalion for testing purposes

Characteristics:

• Not monitor-only (used for active testing)
• Don't require external API integration
• Skip import processing (data already exists internally)
• Full control over agent behavior and configuration

Monitor Agents vs Testing Agents

Best Practices

Security

• Store API keys securely and rotate them regularly
• Use read-only API keys when possible
• Restrict access to monitor agents based on team roles

Data Management

• Import historical data in batches to avoid rate limits
• Use date range filters to import relevant time periods
• Monitor storage usage for high-volume integrations

Evaluation

• Apply consistent metrics across testing and monitoring agents
• Set up alerts for critical metric thresholds
• Review conversation samples regularly to validate metric accuracy

Limitations

• External monitor agents cannot be used in test suites for active testing
• They are read-only and cannot modify agent behavior on external platforms
• Historical import is limited by the external platform's API rate limits and data retention policies

Next Steps

• Set up VAPI Integration
• Set up ElevenLabs Integration
• Learn about Observability Metrics