Create an Evaluation Task

An evaluation task runs a selected agent version and scores its results with evaluators. Read the Agent Evaluation overview before creating a task.

Select an evaluation mode

In the Evaluation Tasks tab, select Create Evaluation and choose a mode:

Mode	Description
Evaluation with a set	Runs questions from an existing evaluation set; suitable for regression and release validation
Evaluation without a set	Has AI generate test questions from the agent configuration before running the agent; suitable for quick exploration

Create the task

1. In the left navigation, select Agent Development > Agent Evaluation, then open Evaluation Tasks.
2. Select Create Evaluation and choose an evaluation mode.
3. Under Agent, select the agent and version to evaluate. The version must be published. If no version is specified, Nexent uses the latest published version.
4. Under Evaluation Configuration, select a Judge model. With-set mode also requires an evaluation set; no-set mode requires a question count (1–50).
5. Under Evaluators, select one or more published evaluators, up to five.
6. Review the configuration and select Start Evaluation.

The task is queued in the background and appears in the Evaluation Tasks list. It stores the agent version, Judge model, and evaluator configuration used at creation time; later edits do not change historical tasks.

Evaluator inputs

The system prepares these standard inputs for evaluators:

Field	Source
query	Evaluation-set Question / query
expected	Evaluation-set Answer / answer; empty in no-set mode
actual	Actual answer generated by the agent
runtime_stats	Runtime statistics from the agent execution

The current page prepares these standard fields automatically. Use these field names when creating a custom evaluator.

Trial Run

In with-set mode, select Trial Run to check one question, the agent answer, and evaluator results. A trial run does not create an evaluation history record.

Start with a representative question to verify the agent version, Judge model, and evaluator configuration before running the full set.

Task states

State	Meaning
Pending	The task was created and is waiting for background execution
Running	The agent and evaluators are running
Completed	The task finished and results are available
Failed	The task or a key execution stage failed; use the error details to correct the configuration and create a new task

For result details, see View Results and Annotations.