> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Agent-as-a-judge

> Run evaluations with a Claude Code harness that reads your traces at runtime - no column mapping, better for nuanced or multi-step criteria.

<Info>
  Agent-as-a-Judge is available in closed Enterprise beta. Contact your Arize account team for access.
</Info>

**Agent-as-a-Judge** runs evaluations with a **harness**, not as a single LLM API call. **Claude Code** is supported today; **Cursor**, **Codex**, **Hermes**, and **OpenClaw** are coming soon. You describe what to score in plain language; the agent pulls span and trace data from your project at run time, applies your criteria, and writes results back as eval columns - same as [LLM-as-a-judge](/docs/ax/evaluate/create-evaluators#llm-as-a-judge) and [code evaluators](/docs/ax/evaluate/evaluators/code-evaluations).

<Frame caption="An agent evaluator in Arize AX">
  <img src="https://storage.googleapis.com/arize-assets/doc-images/agent%20as%20a%20judge/agent-as-a-judge-evaluator.png" alt="An agent evaluator in Arize AX" />
</Frame>

Use it when a fixed prompt and column mapping are too rigid for the judgment you need.

## Why use it

|                   | **LLM-as-a-judge**                                           | **Agent-as-a-Judge**                                                                               |
| ----------------- | ------------------------------------------------------------ | -------------------------------------------------------------------------------------------------- |
| **How it scores** | One judge prompt per span/trace; variables mapped to columns | Agent explores exported trace data and scores from your instructions                               |
| **Setup**         | Template + column mappings                                   | Scoring instructions only - no column mapping required                                             |
| **Best for**      | High-volume, repeatable checks with stable inputs            | Nuanced criteria, multi-field reasoning, or evals that benefit from reading context across a trace |

**Agent-as-a-Judge** is for subjective or complex quality checks where you want an **agent** to interpret production data - not just fill a template. Examples:

* Relevance or helpfulness when the right answer depends on full trace context
* Agent trajectory quality (tool choice, recovery, multi-step reasoning)
* Custom rubrics that are easier to describe in prose than to wire into `{variable}` mappings

For deterministic rules (JSON shape, regex, keyword checks), use a [code evaluator](/docs/ax/evaluate/evaluators/code-evaluations). For simple, high-throughput evals use [LLM-as-a-judge](/docs/ax/evaluate/create-evaluators#llm-as-a-judge). Many teams use all three on the same project.

## How it works

**Configure the evaluator** in the [Evaluator Hub](/docs/ax/evaluate/create-evaluators#evaluator-hub) - **Evaluators → Create → Agent-as-a-Judge**. Select a harness (**Claude Code** is supported today; **Cursor**, **Codex**, **Hermes**, and **OpenClaw** are coming soon), pick an Anthropic model (or **Auto**), then write scoring instructions in plain language. Optional placeholders like `{attributes.output.value}` are filled from span data. Optionally define fixed labels or let the harness decide each run.

**Attach to an online eval task** on an LLM project - date range, query filter, and sampling rate - same flow as [Run online evals on traces](/docs/ax/evaluate/run-evals-on-traces).

**On each run**, the platform starts the selected harness. The harness reads exported spans for the task window, scores them from your instructions, and publishes `eval.<name>.*` columns on the spans.

**View results** on traces, in dashboards, and in task run history. See [View eval results](/docs/ax/evaluate/results-and-costs).

The harness gets read access to traces on the bound project automatically. Add the optional [Arize skill](/docs/ax/agents/connect-your-harness) only if you need broader API access in the harness.

## Create an Agent-as-a-Judge evaluator

<Frame>
  <video
    src="https://storage.googleapis.com/arize-assets/doc-images/agent%20as%20a%20judge/agent-as-a-judge-create.mp4"
    alt="Creating an agent evaluator in Evaluator Hub and selecting the evaluation agent"
    width="100%"
    height="100%"
    style={{
  display: 'block',
  objectFit: 'fill',
  backgroundColor: 'transparent',
}}
    controls
    autoPlay
    muted
    loop
  />
</Frame>

<Steps>
  <Step title="Open Evaluator Hub">
    Go to **Evaluators** in the space sidebar, then **Create** and choose **Agent-as-a-Judge**.
  </Step>

  <Step title="Select harness">
    Choose a harness. **Claude Code** is supported today; **Cursor**, **Codex**, **Hermes**, and **OpenClaw** are coming soon.
  </Step>

  <Step title="Select model">
    Pick an [Anthropic AI integration](/docs/ax/security-and-settings/integrations-playground/overview) and model, or **Auto**.
  </Step>

  <Step title="Write scoring instructions">
    Describe what good and bad look like - for example, whether the assistant's response is relevant to the user input given the full trace.

    The agent reads traces at run time; you do not map template variables to columns upfront.
  </Step>

  <Step title="Configure labels (optional)">
    Leave **Let agent decide labels** on for open-ended rubrics, or turn it off to define fixed labels and scores (for example `relevant` / `irrelevant`).
  </Step>

  <Step title="Save to Evaluator Hub">
    The evaluator is versioned like LLM and code evaluators - reuse it across tasks.
  </Step>
</Steps>

## Run on production traces

An agent evaluator attaches to an online eval task like any other evaluator - see [Run online evals on traces](/docs/ax/evaluate/run-evals-on-traces).

What is specific to the harness: each run provisions it, exports up to the task's span limit for that window, and runs it with your scoring instructions. Cancelling a run from task history tears the harness down.