Skip to main content
Human review tells you what’s broken. An evaluator is how you check for it automatically, at scale, on every request. Building an evaluator and running it are separate steps. This page covers building one; Run evals on your data covers pointing it at your data.
Let Alyx build the eval for you. Press Cmd+L (macOS) or Ctrl+L (Windows/Linux) to open Alyx and try: “Build an eval to check if the response answers the question”. It drafts the prompt template and maps variables to your data.

​
What is an evaluator

An evaluator reads your data and returns a label (e.g. correct, incorrect), a numeric score, and an explanation. Two settings matter when you create one:
  • Scope - the unit of data the evaluator sees, and where its results land.
  • Optimization direction - maximize when higher scores are better, minimize when lower is. Arize colors results accordingly, so failures are visible at a glance.
Evaluators are versioned, so every change is tracked.
Arize AX Evaluators UI showing an LLM-as-a-judge evaluator with name and span scope, judge model and prompt template comparing human ground truth to model output, aligned and not aligned choice labels with scores, optimization direction set to maximize, and version history in the sidebar

Evaluator detail

The anatomy of an evaluator showing the inputs it consumes, prompt, model, and data, and the outputs it emits, label, score, and explanation

The anatomy of an evaluator

For what an evaluator is made of conceptually - its inputs and the anatomy of its output - see Anatomy of an evaluator.

​
Which type do I need?

For each failure mode, ask two questions: is the judgment subjective or deterministic, and should Arize or your own infrastructure run it? Most teams layer several types: a code eval for the objective rules, an LLM judge for the subjective ones. See Run evals on your data to point each at the slice of data it should score. The two most common types score your data like this:
Diagram of an LLM-as-a-judge evaluator showing metadata including scope of span, trace, or session, prompt template with query reference and output variables, data injection into the template, and structured output with score label and explanation after Run eval

How an LLM evaluator scores your data

An LLM evaluator’s prompt template references variables like `{query}`, `{reference}`, and `{output}`, which are mapped to your data at runtime.
Diagram of a code evaluator showing metadata with eval column name and scope of span, trace, or session, Python CodeEvaluator class and evaluate method, dataset_row span attribute keys as data inputs, and structured output with score label and explanation after Run eval

How a code eval scores your data

A code evaluator runs a Python class with an evaluate method. Its dataset_row input is a dictionary of span attributes - common keys are attributes.output.value, attributes.input.value, and attributes.llm.token_count.total.
For the conceptual comparison of the evaluator families and their trade-offs, see Evaluator types.

LLM-as-a-judge

Score outputs with an LLM against criteria you write. The default choice for subjective quality.

Code

Deterministic Python checks. Faster, cheaper, and more consistent for objective rules.

Agent-as-a-judge

A Claude Code harness that reads your traces at run time, with no column mapping.

Remote

Host the scoring logic yourself; Arize handles orchestration, retries, and writing results back.

​
Eval scope: span, trace, and session

Scope sets what unit of data the evaluator sees. Pick the narrowest one that can answer your question: a session eval passes far more context to the judge than a span eval, which costs more and gives the judge more text to weigh. All evaluators on one task share a scope, so each task runs at a single level: span, trace, or session. To cover several scopes on the same project, create a task for each. Trace and session evals add their own rules: how values concatenate across spans, the session-only {conversation} and {turn_data} variables, and truncation limits. See Trace and session evals for those and the setup steps for both scopes.

​
Where evaluators live

You can start an evaluator from wherever you already are: the Evaluator Hub, a trace or span in Tracing, a dataset or experiment, or the Prompt Playground. Wherever you start, it saves to the Hub. The Evaluators page has two tabs:
  • Eval Hub - where evaluators are defined. Build one, attach it to any task (online monitoring, offline batch, dataset experiments) without rewriting prompts or reconfiguring models. Version history and commit messages track every change.
  • Running Tasks - where tasks execute those evaluators against your data. See Run evals on your data.
Attaching an evaluator to a task means mapping its template variables to that source’s columns. That mapping is what makes evaluators portable: one judge works across projects with different schemas.
Evaluators page with Evaluator Hub tab selected, showing a table of LLM evaluators with scope, judge model, maintainer, and usage

Evaluator Hub lists saved LLM judges and their configuration

​
Eval best practices

Binary vs Score Evals

Should I Use the Same LLM for my Eval as My Agent?

Eval Guides

​
Further reading