> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Build evals

> The failure you need to catch decides which kind of evaluator you build, and the unit it scores - a span, a trace, a session, or an experiment.

[Human review](/docs/ax/evaluate/human-review) tells you what's broken. An evaluator is how you check for it automatically, at scale, on every request.

Building an evaluator and running it are separate steps. This page covers building one; [Run evals on your data](/docs/ax/evaluate/run-evals) covers pointing it at your data.

<Tip>
  **Let Alyx build the eval for you.** Press **Cmd+L** (macOS) or **Ctrl+L** (Windows/Linux) to open [Alyx](/docs/ax/alyx) and try: *"Build an eval to check if the response answers the question"*. It drafts the prompt template and maps variables to your data.
</Tip>

<h2 id="what-is-an-evaluator">
  What is an evaluator
</h2>

An evaluator reads your data and returns a **label** (e.g. correct, incorrect), a numeric **score**, and an **explanation**.

Two settings matter when you create one:

* **[Scope](#scope)** - the unit of data the evaluator sees, and where its results land.
* **Optimization direction** - maximize when higher scores are better, minimize when lower is. Arize colors results accordingly, so failures are visible at a glance.

Evaluators are versioned, so every change is tracked.

<Frame caption="Evaluator detail">
  <img src="https://storage.googleapis.com/arize-phoenix-assets/assets/images/arize-docs-images/evaluate/eval%20detail.png" alt="Arize AX Evaluators UI showing an LLM-as-a-judge evaluator with name and span scope, judge model and prompt template comparing human ground truth to model output, aligned and not aligned choice labels with scores, optimization direction set to maximize, and version history in the sidebar" />
</Frame>

<Frame caption="The anatomy of an evaluator">
  <img src="https://storage.googleapis.com/arize-phoenix-assets/assets/images/arize-docs-images/concepts/evaluators/anatomy-of-an-evaluator.png" alt="The anatomy of an evaluator showing the inputs it consumes, prompt, model, and data, and the outputs it emits, label, score, and explanation" />
</Frame>

<Note>
  For what an evaluator is made of conceptually - its inputs and the anatomy of its output - see [Anatomy of an evaluator](/docs/ax/concepts/evaluators/anatomy-of-an-evaluator).
</Note>

## Which type do I need?

For each failure mode, ask two questions: is the judgment subjective or deterministic, and should Arize or your own infrastructure run it?

|                          | **LLM-as-a-judge**                                         | **Code**                                          | **Agent-as-a-judge**                                                       | **Remote**                                                  |
| :----------------------- | :--------------------------------------------------------- | :------------------------------------------------ | :------------------------------------------------------------------------- | :---------------------------------------------------------- |
| **How it judges**        | An LLM scores against criteria you write in plain language | Deterministic Python logic                        | A Claude Code harness that reads trace context at run time                 | Your own scoring endpoint, called by Arize                  |
| **Best for**             | Subjective or nuanced criteria at high volume              | Objective, rule-based checks                      | Nuanced or multi-step criteria where upfront column mapping is impractical | Logic you already own, or runtimes code evals don't support |
| **Examples**             | Helpfulness, tone, correctness, hallucination              | JSON validation, regex matching, keyword presence | Agent trajectory quality, multi-step reasoning                             | Proprietary scoring models, in-house classifiers            |
| **Needs column mapping** | Yes - stable variable mappings                             | Yes - span attribute keys                         | No - reads the trace itself                                                | Yes - payload you define                                    |
| **Runs on**              | Arize AX                                                   | Arize AX                                          | Arize AX                                                                   | Your infrastructure                                         |

Most teams layer several types: a code eval for the objective rules, an LLM judge for the subjective ones. See [Run evals on your data](/docs/ax/evaluate/run-evals) to point each at the slice of data it should score.

The two most common types score your data like this:

<Frame caption="How an LLM evaluator scores your data">
  <img src="https://storage.googleapis.com/arize-phoenix-assets/assets/images/arize-docs-images/evaluate/llm_eval.png?v=20260415" alt="Diagram of an LLM-as-a-judge evaluator showing metadata including scope of span, trace, or session, prompt template with query reference and output variables, data injection into the template, and structured output with score label and explanation after Run eval" />
</Frame>

An LLM evaluator's prompt template references variables like `` `{query}` ``, `` `{reference}` ``, and `` `{output}` ``, which are mapped to your data at runtime.

<Frame caption="How a code eval scores your data">
  <img src="https://storage.googleapis.com/arize-phoenix-assets/assets/images/arize-docs-images/evaluate/code_eval_1.png?v=20260415" alt="Diagram of a code evaluator showing metadata with eval column name and scope of span, trace, or session, Python CodeEvaluator class and evaluate method, dataset_row span attribute keys as data inputs, and structured output with score label and explanation after Run eval" />
</Frame>

A code evaluator runs a Python class with an `evaluate` method. Its `dataset_row` input is a dictionary of span attributes - common keys are `attributes.output.value`, `attributes.input.value`, and `attributes.llm.token_count.total`.

<Note>
  For the conceptual comparison of the evaluator families and their trade-offs, see [Evaluator types](/docs/ax/concepts/evaluators/evaluator-types).
</Note>

<CardGroup cols={2}>
  <Card title="LLM-as-a-judge" icon="scale-balanced" href="/docs/ax/evaluate/evaluators/llm-as-a-judge">
    Score outputs with an LLM against criteria you write. The default choice for subjective quality.
  </Card>

  <Card title="Code" icon="code" href="/docs/ax/evaluate/evaluators/code-evaluations">
    Deterministic Python checks. Faster, cheaper, and more consistent for objective rules.
  </Card>

  <Card title="Agent-as-a-judge" icon="robot" href="/docs/ax/evaluate/agent-as-a-judge">
    A Claude Code harness that reads your traces at run time, with no column mapping.
  </Card>

  <Card title="Remote" icon="server" href="/docs/ax/evaluate/remote-evaluators">
    Host the scoring logic yourself; Arize handles orchestration, retries, and writing results back.
  </Card>
</CardGroup>

<h2 id="scope">
  Eval scope: span, trace, and session
</h2>

Scope sets what unit of data the evaluator sees. Pick the narrowest one that can answer your question: a session eval passes far more context to the judge than a span eval, which costs more and gives the judge more text to weigh.

| Scope       | Unit evaluated                     | Questions it answers                                                       | Results written to                                                  |
| :---------- | :--------------------------------- | :------------------------------------------------------------------------- | :------------------------------------------------------------------ |
| **Span**    | One span                           | Was the right tool selected? Was this retrieval relevant?                  | `eval.<name>` on the evaluated span                                 |
| **Trace**   | Every span in one request          | Was the sequence of tool calls correct? Did the agent loop or skip a step? | `trace_eval.<name>` on the trace's root span                        |
| **Session** | Every trace sharing a `session.id` | Did the agent keep context across turns? Did the user reach their goal?    | `session_eval.<name>` on the root span of the session's first trace |

All evaluators on one task share a scope, so each task runs at a single level: span, trace, or session. To cover several scopes on the same project, create a task for each.

Trace and session evals add their own rules: how values concatenate across spans, the session-only `{conversation}` and `{turn_data}` variables, and truncation limits. See [Trace and session evals](/docs/ax/evaluate/trace-and-session-evals) for those and the setup steps for both scopes.

<h2 id="evaluator-hub">
  Where evaluators live
</h2>

<span id="llm-as-a-judge" />

<span id="tutorial-create-a-custom-llm-as-a-judge-eval" />

<span id="tutorial-run-pre-built-evals-on-your-traces" />

<span id="step-2-create-an-evaluation-task" />

<span id="code-evaluations" />

You can start an evaluator from wherever you already are: the **Evaluator Hub**, a trace or span in [Tracing](/docs/ax/observe/tracing/view-and-manage-traces), a [dataset or experiment](/docs/ax/develop/datasets-and-experiments), or the [Prompt Playground](/docs/ax/prompts/prompt-playground). Wherever you start, it saves to the Hub.

The **Evaluators** page has two tabs:

* **Eval Hub** - where evaluators are defined. Build one, attach it to any task (online monitoring, offline batch, dataset experiments) without rewriting prompts or reconfiguring models. Version history and commit messages track every change.
* **Running Tasks** - where tasks execute those evaluators against your data. See [Run evals on your data](/docs/ax/evaluate/run-evals).

Attaching an evaluator to a task means mapping its template variables to that source's columns. That mapping is what makes evaluators portable: one judge works across projects with different schemas.

<Frame caption="Evaluator Hub lists saved LLM judges and their configuration">
  <img src="https://storage.googleapis.com/arize-phoenix-assets/assets/images/arize-docs-images/evaluate/eval%20hub.png" alt="Evaluators page with Evaluator Hub tab selected, showing a table of LLM evaluators with scope, judge model, maintainer, and usage" />
</Frame>

## Eval best practices

<CardGroup cols={3}>
  <Card title="Binary vs Score Evals" href="https://arize.com/blog/testing-binary-vs-score-llm-evals-on-the-latest-models/" />

  <Card title="Should I Use the Same LLM for my Eval as My Agent?" href="https://arize.com/blog/should-i-use-the-same-llm-for-my-eval-as-my-agent-testing-self-evaluation-bias/" />

  <Card title="Eval Guides" href="/docs/ax/cookbooks/evaluate/overview" />
</CardGroup>

## Further reading

* [Evaluator best practices](/docs/ax/concepts/evaluators/evaluator-best-practices) - what good evaluators do, and the five common mistakes
* [Run online evals on traces](/docs/ax/evaluate/run-evals-on-traces)
* [Run offline evals on experiments](/docs/ax/evaluate/run-evals-on-experiments)
* [Code evals reference](/docs/ax/evaluate/evaluators/code-evaluations)