Evaluations

Evaluations help teams compare agent behavior using repeatable scenarios instead of relying only on ad-hoc test calls. They are useful after instruction, model, knowledge, or workflow changes.

Create a useful evaluation

The evaluation workspace has four tabs:

1. Personas: define caller age, nationality, tone, background, and other traits.
2. Scenarios: describe the call context and the outcomes the evaluator should expect.
3. Run Tests: select a use case plus persona and scenario combinations.
4. Reports: inspect completed runs and scoring results.

Create personas that reflect real customer constraints and objections. Scenarios should have a clear objective and measurable expected behavior. Run enough combinations to expose inconsistent behavior, then review failures before promoting the change to a larger lead segment.

Design the test set

Use personas to vary communication style and customer context, not to encode the expected answer. Put the business situation and success criteria in the scenario. Include normal conversations, common objections, missing information, interruption, refusal, escalation, and requests the agent must not fulfill.

Keep a small stable regression set for every important use case. Add a new scenario when a production conversation reveals a behavior that should remain fixed. This turns one-off incident review into a repeatable release check.

Interpret reports

Treat evaluation output as evidence for review rather than an automatic production approval. Compare failures by scenario and persona, inspect the underlying interaction, and decide whether the cause is the instruction, knowledge, selected agent configuration, integration behavior, or the evaluation criterion itself.

Run the same set before and after a material change. If a change improves one scenario while regressing another, refine the instruction or split workflows with genuinely different objectives.

Open Evaluations to manage scenarios and runs.