> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
> Evaluate an agent's intermediate reasoning, tool selection, and decision path — not just its final answer — on a movie recommendation agent using Arize AX.
# Trace-Level Evaluation: Beyond Input/Output Checks
The most common way to evaluate an LLM application is **end-to-end**: take the user's request, take the final answer, and judge whether the answer is good. That works for a single model call. It breaks down for **agents**, because an agent isn't one call — it's a *sequence of decisions*: which tools to call, in what order, with what arguments, and how to reason over the results.
End-to-end eval only inspects the two **endpoints** — the input and the final output. The interesting failures happen in the middle, where you can't see them:
* **Reasoning** — did the agent reason coherently toward the goal, or get there by luck?
* **Tool selection** — did it pick the right tools (and *skip* the wrong ones)?
* **Decision path** — did it call those tools in a sensible order, with sensible arguments?
A **correct-looking answer can come from a broken path** — the right result for the wrong reasons, which fails the next time the inputs shift. This guide shows how to capture the full trace, reconstruct the agent's intermediate signals from its spans, and evaluate those — using a movie recommendation agent as the worked example.
We'll go through the following steps:
* Reconstruct an agent's **decision path** (`tool_path`) and **tool I/O** (`tool_io`) from its spans
* Run an *endpoint* check (recommendation relevance) alongside two *intermediate* checks (decision path and reasoning/support)
* See each evaluator catch a distinct failure the endpoint check misses
* Log trace-level evaluations back to Arize AX
***
## Before you start
You need an [Arize AX account](https://app.arize.com/auth/join) and an OpenAI API key.
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
pip install "arize>=8" arize-otel "arize-phoenix-evals>=3" \
openinference-instrumentation-openai openinference-instrumentation-openai-agents \
openinference-instrumentation openai openai-agents
```
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
export ARIZE_SPACE_ID=""
export ARIZE_API_KEY=""
export OPENAI_API_KEY=""
```
## Build and trace the agent
Register the tracer and instrument the Agents SDK, so every agent run, tool call, and model call becomes a span.
```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import os
from arize.otel import register
from openinference.instrumentation.openai import OpenAIInstrumentor
from openinference.instrumentation.openai_agents import OpenAIAgentsInstrumentor
model_id = "movie-recommendation-agent"
MODEL = "gpt-5.4-mini"
JUDGE_MODEL = "gpt-4.1-mini"
tracer_provider = register(
space_id=os.environ["ARIZE_SPACE_ID"],
api_key=os.environ["ARIZE_API_KEY"],
project_name=model_id,
set_global_tracer_provider=True,
)
OpenAIAgentsInstrumentor().instrument(tracer_provider=tracer_provider)
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)
```
The agent has three tools. The ordering constraint between them is what makes the decision path worth evaluating: selection has to happen before review or summarization, because the other two operate on the selected movies.
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import ast
from typing import List, Union
from agents import Agent, Runner, function_tool
from openai import OpenAI
client = OpenAI()
@function_tool
def movie_selector_llm(genre: str) -> List[str]:
prompt = (
f"List up to 5 recent popular streaming movies in the {genre} genre. "
"Provide only movie titles as a Python list of strings."
)
response = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": prompt}],
)
content = response.choices[0].message.content
try:
movie_list = ast.literal_eval(content)
if isinstance(movie_list, list):
return movie_list[:5]
except Exception:
return content.split("\n")
@function_tool
def reviewer_llm(movies: Union[str, List[str]]) -> str:
if isinstance(movies, list):
movies_str = ", ".join(movies)
prompt = (
"Sort the following movies by rating from highest to lowest and "
f"provide a short review for each:\n{movies_str}"
)
else:
prompt = f"Provide a short review and rating for the movie: {movies}"
response = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": prompt}],
)
return response.choices[0].message.content.strip()
@function_tool
def preview_summarizer_llm(movie: str) -> str:
prompt = f"Write a 1-2 sentence summary describing the movie '{movie}'."
response = client.chat.completions.create(
model=MODEL,
messages=[{"role": "user", "content": prompt}],
)
return response.choices[0].message.content.strip()
agent = Agent(
name="MovieRecommendationAgentLLM",
model=MODEL,
tools=[movie_selector_llm, reviewer_llm, preview_summarizer_llm],
instructions=(
"You are a helpful movie recommendation assistant with access to three tools:\n"
"1. MovieSelector: Given a genre, returns up to 5 recent streaming movies.\n"
"2. Reviewer: Given one or more movie titles, returns reviews and sorts them by rating.\n"
"3. PreviewSummarizer: Given a movie title, returns a 1-2 sentence summary.\n\n"
"Your goal is to provide a helpful, user-friendly response combining relevant information."
),
)
```
Run the agent against a handful of questions to generate traces to evaluate.
```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import asyncio
import time
questions = [
"Which Batman movie should I watch?",
"I want to watch a good romcom",
"What is a very scary horror movie?",
"Name a feel-good holiday movie",
"Recommend a musical with great songs",
"Give me a classic drama from the 90s",
]
async def run_all():
for question in questions:
await Runner.run(agent, question)
asyncio.run(run_all())
# Spans are exported in the background, so force a flush and give Arize AX a
# moment to ingest before querying them below.
tracer_provider.force_flush()
time.sleep(30)
```
## Get Span Data from Arize AX
Export your traces from Arize AX so we can reconstruct what each evaluator needs:
```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
from datetime import datetime, timedelta, timezone
from arize.client import ArizeClient
ax_client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
primary_df = ax_client.spans.export_to_df(
space_id=os.environ["ARIZE_SPACE_ID"],
project_name=model_id,
start_time=datetime.now(timezone.utc) - timedelta(days=7),
end_time=datetime.now(timezone.utc),
)
```
## Separate the endpoints from the trace
First pull the **endpoints** — the user's question and the agent's *final* answer. Take the final answer only, not a concatenation of every span's output: folding in tool outputs would blur the line between "what the user saw" and "what happened inside the trace."
With the OpenAI Agents instrumentation, the root `AGENT` span doesn't record `input.value` / `output.value` — those attributes live on the underlying `LLM` spans, so we read the question and final reply from there with two small helpers. With an instrumentation that populates the root span, you could read `attributes.input.value` / `attributes.output.value` off the agent root directly.
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import json
import pandas as pd
SPAN_KIND = "attributes.openinference.span.kind"
def _as_list(value):
"""Span attributes can arrive as a JSON string or an already-parsed list."""
if isinstance(value, str):
try:
return json.loads(value)
except json.JSONDecodeError:
return []
return value if isinstance(value, list) else []
def user_query(input_messages):
"""The first user message in an LLM span's input."""
for message in _as_list(input_messages):
if message.get("message.role") == "user":
contents = message.get("message.contents") or []
return message.get("message.content") or "".join(
c.get("message_content.text", "") for c in contents
)
return None
def assistant_text(output_messages):
"""The assistant's text reply in an LLM span (empty on tool-calling turns)."""
texts = []
for message in _as_list(output_messages):
if message.get("message.role") == "assistant":
if message.get("message.content"):
texts.append(message["message.content"])
for c in message.get("message.contents") or []:
if c.get("message_content.type") == "text":
texts.append(c.get("message_content.text", ""))
return " ".join(t for t in texts if t).strip() or None
# Restrict to the agent's own traces: each has a root span (no parent) of kind
# AGENT. This keeps the evaluators' own LLM calls out of the evaluation set.
agent_roots = primary_df[(primary_df["parent_id"].isna()) & (primary_df[SPAN_KIND] == "AGENT")]
agent_trace_ids = agent_roots["context.trace_id"].unique()
llm_spans = primary_df[
(primary_df[SPAN_KIND] == "LLM") & (primary_df["context.trace_id"].isin(agent_trace_ids))
].sort_values("start_time")
# Endpoint input = the user's question (first user message).
# Endpoint output = the agent's FINAL answer only (its last text turn).
trace_df = pd.DataFrame(
{
"input": llm_spans.groupby("context.trace_id")["attributes.llm.input_messages"].apply(
lambda s: next((q for q in s.map(user_query) if q), None)
),
"output": llm_spans.groupby("context.trace_id")["attributes.llm.output_messages"].apply(
lambda s: next((a for a in reversed(list(s.map(assistant_text))) if a), None)
),
}
).dropna(subset=["input", "output"])
```
## Reconstruct the intermediate signals
To evaluate the agent's *process*, reconstruct two signals the endpoints never show, both from the trace's `TOOL` spans (sorted by `start_time`):
* **`tool_path`** — the ordered tool calls the agent made, *with their arguments*. This is the decision path: tool selection, order, and the arguments each tool was called with.
* **`tool_io`** — what each tool was called *with* and what it *returned*, so an evaluator can check whether the final answer is grounded in real tool results.
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
tool_spans = primary_df[primary_df[SPAN_KIND] == "TOOL"].sort_values("start_time")
# tool_path: the ordered tool calls WITH their arguments (selection, order, args).
def format_decision_path(group):
return " -> ".join(
f"{row['name']}({row['attributes.input.value']})" for _, row in group.iterrows()
)
trace_df["tool_path"] = (
tool_spans.groupby("context.trace_id")[["name", "attributes.input.value"]]
.apply(format_decision_path)
.reindex(trace_df.index)
.fillna("No tools called")
)
# tool_io: each tool call's input AND output.
def format_tool_calls(group):
lines = []
for i, (_, row) in enumerate(group.iterrows(), start=1):
lines.append(
f"{i}. {row['name']} | input: {row['attributes.input.value']} "
f"| output: {row['attributes.output.value']}"
)
return "\n".join(lines)
trace_df["tool_io"] = (
tool_spans.groupby("context.trace_id")[["name", "attributes.input.value", "attributes.output.value"]]
.apply(format_tool_calls)
.reindex(trace_df.index)
.fillna("No tools called")
)
```
## Define the three evaluators
Each evaluator reads a different column and answers a different question:
1. **Relevance** — an *endpoint* check on `input` + `output` (exactly what end-to-end eval does).
2. **Decision path** — an *intermediate* check on `input` + `tool_path` (right tools, right order, *sensible arguments*?).
3. **Reasoning / support** — an *intermediate* check on `input` + `tool_io` + `output` (is the answer grounded in the actual tool results, or does it invent facts no tool produced?).
Each prompt is written to judge only its own concern. The decision-path judge, for example, never sees the final answer:
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
DECISION_PATH = """
You are evaluating an agent's DECISION PATH: the ordered tool calls it made to
answer a request — which tools, in what order, and with what arguments. You are
NOT judging the final answer text, and you are NOT judging whether each tool's
output was correct — only whether the agent's choices were sensible.
The agent has three tools available:
- movie_selector_llm(genre): returns candidate movies. Must come FIRST, because the
other tools operate on the selected movies.
- reviewer_llm(movies): reviews and sorts movies. Only meaningful AFTER selection,
and should be called with the movies that were actually selected.
- preview_summarizer_llm(movie): summarizes a movie. Only meaningful AFTER selection.
You will be given:
1. The user input that initiated the trace
2. The ordered tool calls the agent executed, with their arguments
##
User Input:
{input}
Decision Path (ordered tool calls with arguments):
{tool_path}
##
Respond with exactly one word: `correct` or `incorrect`.
1. `correct` ->
- movie_selector_llm is called before reviewer_llm or preview_summarizer_llm, AND
- each tool is called with sensible arguments (e.g. reviewer_llm receives the
movies that were selected, not an empty or unrelated list).
2. `incorrect` ->
- a tool that operates on movies (reviewer_llm / preview_summarizer_llm) runs
before any movies have been selected, the selection step is missing, OR a tool is
called with nonsensical arguments.
"""
RECOMMENDATION_RELEVANCE = """
You are evaluating the relevance of movie recommendations provided by an LLM application.
You will be given:
1. The user input that initiated the trace
2. The list of movie recommendations output by the system
##
User Input:
{input}
Recommendations:
{output}
##
Respond with exactly one word: `correct` or `incorrect`.
1. `correct` ->
- All recommended movies match the requested genre or criteria in the user input.
- The recommendations are relevant to the user's request and are not repetitive.
2. `incorrect` ->
- One or more recommendations do not match the requested genre or criteria, or the
recommendations are repetitive.
"""
REASONING_SUPPORT = """
You are checking whether an agent's FINAL ANSWER is SUPPORTED by the actual
results its tools returned.
You are NOT judging tool order (that's the decision-path check) or genre match
(that's the relevance check). Judge ONLY whether every concrete claim in the final
answer — titles, ratings, scores, review quotes, plot facts — is grounded in the
tool outputs below. An answer that asserts a fact no tool produced (for example a
specific rating) is unsupported, even if it sounds plausible.
##
User Input:
{input}
Tool calls and their results (in order):
{tool_io}
Final Answer:
{output}
##
Respond with exactly one word: `correct` or `incorrect`.
1. `correct` -> every concrete claim in the final answer is supported by the tool
results above.
2. `incorrect` -> the final answer asserts at least one concrete fact (a rating,
score, review, or title) that does not appear in the tool results.
"""
```
Run all three judges with `arize-phoenix-evals`. Each `ClassificationEvaluator` reads the columns named in its template and returns a `Score` (label + numeric score + explanation). Wrap the run in `suppress_tracing()` so the judges' own LLM calls don't get traced back into the same project:
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import asyncio
from phoenix.evals import LLM, ClassificationEvaluator, async_evaluate_dataframe
from openinference.instrumentation import suppress_tracing
judge_llm = LLM(provider="openai", model=JUDGE_MODEL)
relevance_evaluator = ClassificationEvaluator(
name="relevance", llm=judge_llm, prompt_template=RECOMMENDATION_RELEVANCE,
choices={"correct": 1.0, "incorrect": 0.0},
)
path_evaluator = ClassificationEvaluator(
name="decision_path", llm=judge_llm, prompt_template=DECISION_PATH,
choices={"correct": 1.0, "incorrect": 0.0},
)
reasoning_evaluator = ClassificationEvaluator(
name="reasoning", llm=judge_llm, prompt_template=REASONING_SUPPORT,
choices={"correct": 1.0, "incorrect": 0.0},
)
EVALUATORS = [relevance_evaluator, path_evaluator, reasoning_evaluator]
with suppress_tracing():
results_df = asyncio.run(
async_evaluate_dataframe(dataframe=trace_df, evaluators=EVALUATORS)
)
```
Each evaluator writes a `_score` column holding a `Score` dict (`label`, `score`, `explanation`) — so the columns above are `relevance_score`, `decision_path_score`, and `reasoning_score`.
## Seeing what each evaluator catches
On well-behaved traces the three evaluators agree. The point of trace-level evaluation is what happens when they *don't*. Running the three judges on controlled cases that mirror the real schema — each with a relevant-looking answer, so the endpoint check passes every time — shows each intermediate check catching a distinct failure:
| case | relevance (endpoint) | decision path | reasoning / support |
| :-------------------------------------------------- | :------------------- | :------------ | :------------------ |
| clean run | correct | correct | correct |
| broken order (reviews before selecting) | correct | **incorrect** | correct |
| unsupported claim (cites a rating no tool returned) | correct | correct | **incorrect** |
Two intermediate failures, two different lenses — and both invisible to the endpoint check.
## Log Results Back to Arize AX
Attach each trace's three evaluations to its **root span**, using the `trace_eval..label/score/explanation` column convention that Arize AX expects:
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import json
def _unpack(cell):
"""Phoenix returns each evaluator's result as a Score dict in `_score`."""
if isinstance(cell, str):
try:
cell = json.loads(cell)
except json.JSONDecodeError:
return None, None, None
if isinstance(cell, dict):
return cell.get("label"), cell.get("score"), cell.get("explanation")
return getattr(cell, "label", None), getattr(cell, "score", None), getattr(cell, "explanation", None)
# Map each evaluator's score column to the Arize AX eval name shown in the UI.
EVAL_COLUMNS = {
"RecommendationRelevance": "relevance_score",
"DecisionPath": "decision_path_score",
"ReasoningSupport": "reasoning_score",
}
eval_df = pd.DataFrame()
for ax_name, score_col in EVAL_COLUMNS.items():
unpacked = results_df[score_col].apply(_unpack)
eval_df[f"trace_eval.{ax_name}.label"] = unpacked.map(lambda t: t[0]).values
eval_df[f"trace_eval.{ax_name}.score"] = unpacked.map(lambda t: t[1]).values
eval_df[f"trace_eval.{ax_name}.explanation"] = unpacked.map(lambda t: t[2]).values
eval_df["context.trace_id"] = trace_df.index.values
# Map each trace to its root (AGENT) span so the evals attach to the root span.
# update_evaluations needs a context.span_id column (drop=False keeps it).
root_spans = agent_roots[["context.trace_id", "context.span_id"]]
log_df = eval_df.merge(root_spans, on="context.trace_id", how="inner").set_index(
"context.span_id", drop=False
)
# Reuse the ArizeClient from the export step.
resp = ax_client.spans.update_evaluations(
space_id=os.environ["ARIZE_SPACE_ID"],
project_name=model_id,
dataframe=log_df,
)
```
## View Results in Arize AX
After logging the evaluations, you can view the results in the Traces tab of your Arize AX project. Each trace's root span carries all three labels — `RecommendationRelevance`, `DecisionPath`, and `ReasoningSupport` — with the judge's score and explanation, so you can:
* Monitor trace-level performance metrics
* Spot answers that look right but reached the goal through a broken path
* Track recommendation quality, decision-path correctness, and grounding side by side
## Takeaway
We ran three evaluators over the same traces, each asking a different question and reading a different signal:
* **relevance** (endpoint) — *did the final answer look right?* Reads only `input` and `output`.
* **decision path** (intermediate) — *did the agent pick the right tools, in the right order, with sensible arguments?* Reads `tool_path`.
* **reasoning / support** (intermediate) — *is the answer grounded in what the tools returned?* Reads `tool_io`.
The lesson: **a good-looking answer can hide a broken process, and only intermediate evals — reading signals reconstructed from spans — can see it.** The pattern generalizes: for any intermediate step you care about, reconstruct the relevant signal from spans into a column, write a judge that reads that column, and run it alongside your endpoint eval.
## Get Span Data from Arize AX
Export your traces from Arize AX so we can reconstruct what each evaluator needs:
```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
from datetime import datetime, timedelta, timezone
from arize.client import ArizeClient
ax_client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
primary_df = ax_client.spans.export_to_df(
space_id=os.environ["ARIZE_SPACE_ID"],
project_name=model_id,
start_time=datetime.now(timezone.utc) - timedelta(days=7),
end_time=datetime.now(timezone.utc),
)
```
## Separate the endpoints from the trace
First pull the **endpoints** — the user's question and the agent's *final* answer. Take the final answer only, not a concatenation of every span's output: folding in tool outputs would blur the line between "what the user saw" and "what happened inside the trace."
## Takeaway
We ran three evaluators over the same traces, each asking a different question and reading a different signal:
* **relevance** (endpoint) — *did the final answer look right?* Reads only `input` and `output`.
* **decision path** (intermediate) — *did the agent pick the right tools, in the right order, with sensible arguments?* Reads `tool_path`.
* **reasoning / support** (intermediate) — *is the answer grounded in what the tools returned?* Reads `tool_io`.
The lesson: **a good-looking answer can hide a broken process, and only intermediate evals — reading signals reconstructed from spans — can see it.** The pattern generalizes: for any intermediate step you care about, reconstruct the relevant signal from spans into a column, write a judge that reads that column, and run it alongside your endpoint eval.