> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Run online evals on traces
> Online evals answer whether quality is holding up right now. Ground what you automate in trace review and a clear failure taxonomy first.
An online eval runs continuously against your production traces: you take an evaluator you already built and attach it to a task pointed at a live project. Any evaluator works here - an [LLM judge](/docs/ax/evaluate/evaluators/llm-as-a-judge), a [code evaluator](/docs/ax/evaluate/evaluators/code-evaluations), an [agent-as-a-judge](/docs/ax/evaluate/agent-as-a-judge), or a [remote evaluator](/docs/ax/evaluate/remote-evaluators). For the full anatomy of a task - evaluators, data source, variable mapping, cadence and sampling - see [How tasks work](/docs/ax/evaluate/run-evals#what-is-a-task).
You'll need traces flowing into Arize AX and an LLM provider configured. See [AI Provider Integrations](/docs/ax/security-and-settings/integrations-playground/overview).
Most teams start with a one-time backfill over historical data to set a baseline, then switch on a continuous task.
## Start from real traces
Before automating, read real traces in your [tracing project](/docs/ax/observe/tracing/view-and-manage-traces) to understand where things go wrong. Group failure patterns into a taxonomy - each category can map to an evaluator or filter. To capture those categories as structured labels, see [Human review](/docs/ax/evaluate/human-review).
Create a task
Create a task in any of these ways.
**Let Alyx set up the task for you.** Press **Cmd+L** (macOS) or **Ctrl+L** (Windows/Linux) to open [Alyx](/docs/ax/alyx) and try: *"Create an online task that runs this eval on 10% of traces"*
Use the [arize-evaluator skill](https://github.com/Arize-ai/arize-skills/blob/main/skills/arize-evaluator/SKILL.md) to create and trigger tasks via the `ax` CLI without leaving your editor. Install the [Arize skills plugin](/docs/ax/skills/overview) in your coding agent if you have not already. Then ask your agent:
* "Create a continuous task to run my hallucination evaluator on my project"
* "Trigger a backfill eval run on my project for the last 7 days"
* "Set up a task that only evaluates LLM spans"
Ask [Alyx](/docs/ax/alyx/meet-alyx) to create a task and run your evaluator on your traces:
* "Run my correctness evaluator continuously on my production traces"
* "Backfill my hallucination eval on the last 7 days of spans"
* "Set up a task to score only LLM spans with my relevance evaluator"
You can create a task from several places in Arize AX: from the **Evaluators** page in the left sidebar, from the **Projects** page, or directly from within a span. To start from the evaluator instead, find it in the **Eval Hub** tab and click **Use Evaluator** - task creation opens with it already selected.
1. **Click New Task** from any of the entry points above.
2. **Name your task** and select your project as the data source.
3. **Click Add Evaluator** and pick an evaluator **From Hub**, **Create from Template**, or **Create Custom**. You can add multiple evaluators to a single task.
4. **Configure column mappings** to map template variables to your data. The evaluator definition itself stays read-only here - see [Column mapping](/docs/ax/evaluate/run-evals#column-mapping).
5. **Set evaluation granularity:** span, trace, or session.
6. **Choose cadence:** run continuously on new data or run once on historical data.
7. **Set sampling rate** and any filters.
8. **Click Create Task.**
Once created, results appear automatically in the Tracing view attached to each span. To check on a task, go to the **Running Tasks** tab, open any task, and click **View Logs**. From the logs you can also click **View Traces** to jump directly to the spans that were evaluated with the same filters applied.
Use this approach when you need to run evals on large datasets, incorporate external data sources, or want full control over execution and cost. Export your spans, run evals using Phoenix Evals, and log results back to Arize AX via the Python SDK.
### 1. Export spans
From the Tracing page, click **Export** and select **Export to Notebook** to get prefilled export code. Or export programmatically:
```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import os
from datetime import datetime
from arize import ArizeClient
client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
primary_df = client.spans.export_to_df(
space_id=os.environ["ARIZE_SPACE_ID"],
project_name="your-project-name",
start_time=datetime.fromisoformat(''), # prefilled by export
end_time=datetime.fromisoformat(''), # prefilled by export
)
```
### 2. Run evals
Check which attributes are present with `primary_df.columns`, then map your input and output columns:
```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
primary_df["input"] = primary_df["attributes.input.value"]
primary_df["output"] = primary_df["attributes.output.value"]
from phoenix.evals import create_classifier
from phoenix.evals.evaluators import async_evaluate_dataframe
from phoenix.evals.llm import LLM
MY_SAMPLE_TEMPLATE = '''
You are evaluating the positivity or negativity of the responses to questions.
[BEGIN DATA]
************
[Question]: {input}
************
[Response]: {output}
[END DATA]
Please focus on the tone of the response.
Your answer must be single word, either "positive" or "negative"
'''
llm = LLM(provider="openai", model="gpt-5")
sample_evaluator = create_classifier(
name="sample-eval",
llm=llm,
prompt_template=MY_SAMPLE_TEMPLATE,
choices={"correct": 1.0, "incorrect": 0.0},
)
results_df = await async_evaluate_dataframe(
dataframe=primary_df,
evaluators=[sample_evaluator],
)
```
It is easier to iterate on your evaluator in a Python script or Colab notebook first - see [Test evaluator code locally](/docs/ax/evaluate/run-evals#test-evaluator-code-locally). For the in-product **Create Evaluator** layout (imports, class, and sample-data mapping), see [Create evaluators](/docs/ax/evaluate/create-evaluators#code-evaluations).
### 3. Log results back to Arize AX
Results require four columns: `eval..label`, `eval..score`, `eval..explanation`, and `context.span_id`. For trace or session evals use the prefixes `trace_eval.` and `session_eval.`.
```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import os
from arize import ArizeClient
from phoenix.evals.utils import to_annotation_dataframe
client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
sample_eval_df = to_annotation_dataframe(results_df)
sample_eval_df = sample_eval_df.rename(columns={
"label": "eval.correctness.label",
"score": "eval.correctness.score",
"explanation": "eval.correctness.explanation"
})
client.spans.update_evaluations(
space_id=os.environ["ARIZE_SPACE_ID"],
project_name="your-project-name",
dataframe=sample_eval_df,
)
```
Evals can be applied to spans up to 14 days prior to the current day. For older spans contact [support@arize.com](mailto:support@arize.com).
## Task configuration
### Sampling rate
| Rate | When to use |
| ---------- | -------------------------------------------------------------------------- |
| **100%** | Low-volume or critical applications where you want to evaluate every trace |
| **10–50%** | High-volume applications balancing cost and coverage |
| **1–5%** | Very high-volume applications where representative sampling is enough |
Start at **10–20%** and increase once you have validated your evaluator is working correctly.
Sampling applies at the **highest evaluator scope** on the task (`session` > `trace` > `span`). Evaluators at lower scopes then run on all matching data inside that sampled set. The same ordering sets the ceiling on a one-time backfill: the maximum number of items is based on the highest scope on the task.
### Filters
Use filters to target specific subsets of your data:
* **Span kind:** Only evaluate specific span types (for example LLM spans)
* **Model name:** Only evaluate spans from a specific model
* **Metadata:** Only evaluate spans with certain metadata tags
* **Span attributes:** Filter on any span attribute
A filter matches spans directly, and admits the traces or sessions that contain a matching span.
#### Multi-span queries (trace and session evals)
For **trace** and **session** evaluators, you can use a **multi-span query** (MSQ) - the same filter model as [multi-span filters](/docs/ax/observe/tracing/view-and-manage-traces#multi-span-filters) on the Tracing page. Named span subqueries (A, B, C, …) are combined with operators (`AND`, `OR`, `NOT`, `->`, `=>`) to express patterns like "retrieval happened, then an LLM answered" or "agent directly called a tool."
> *Find the traces or sessions where a **pattern** of spans occurred - then evaluate only the spans that matter.*
**Example - RAG audit**
| Subquery | Filter |
| :------- | :------------------------------------------------------------------ |
| **A** | `attributes.openinference.span.kind = 'RETRIEVER'` |
| **B** | `attributes.openinference.span.kind = 'LLM' AND status_code = 'OK'` |
**Expression:** `A => B` - retriever is the direct parent of the LLM span.
| Evaluation level | MSQ behavior |
| :--------------- | :-------------------------------------------------------------------------------------- |
| **Span** | Single subquery only - complex operators add little value |
| **Trace** | Full MSQ; one score per trace, written to the trace **root span** (`trace_eval.`) |
| **Session** | Full MSQ; scores written to each turn's root span (`session_eval.`) |
**Subquery-aware variable mapping:** For each template variable, specify which **column** to read and which **subquery(ies)** may supply it.
| Variable | Column | Subqueries |
| :----------- | :------------------------------- | :--------- |
| `{question}` | `input` | A |
| `{answer}` | `output` | B |
| `{context}` | `attributes.retrieval.documents` | A |
**`{conversation}` (session evals only):** A virtual variable - not a column. The platform builds a formatted transcript from each root span (`parent_id IS NULL`) in the session and injects it into the prompt.
MSQ does not replace sampling or cadence - it replaces the filter half of the task. Sampling applies after admission, at the trace/session unit level.
| Use MSQ when… | A simple span filter is enough when… |
| :------------------------------------------------- | :-------------------------------------- |
| Span order or parent-child structure matters | You only need "session contains span X" |
| Different variables come from different span roles | All variables live on the same spans |
For operator syntax (`->`, `=>`, parentheses), see [Multi-span filters](/docs/ax/observe/tracing/view-and-manage-traces#multi-span-filters). For span vs trace vs session scope, see [Evaluation levels](/docs/ax/concepts/evaluators/evaluation-levels).
## Run evals continuously
For tasks that use **Run continuously on new data**, evaluators from the Eval Hub (including pre-built LLM judge templates) run on incoming traces on a rolling schedule. When you [create a task](#create-a-task) and add an evaluator, you can pick a template from the hub before mapping columns and saving.
On the **Evaluators** page, the **Running Eval Tasks** tab lists every task, its target and evaluators, a snapshot of the last few runs, and **View Logs** when you need execution details.
## Viewing results
Once a task runs, results attach automatically to the spans it scored. See [Results and costs](/docs/ax/evaluate/results-and-costs) for where scores appear in the tracing UI, how to chart them over time, and how to read task logs when a run fails.