Trace and session evals

​

How trace and session evals work

​

Session-only variables

• {conversation} is the session transcript Arize AX builds as a JSON array with one input / output pair per turn, taken from each trace’s root span in chronological order. It needs no variable mapping, and it is valid only at session scope.
• {turn_data} is the string Arize AX produces by rendering a Turn Definition template you write once per turn, then joining the rendered turns with TURN 1, TURN 2, and so on as separators. Use it when the default input/output transcript leaves out something the judge needs, such as tool results or retrieved context.

​

How variables are filled in

​

Where results land

​

Set up a trace or session eval

• Trace eval
• Session eval

1

Open the New Evaluator form

Open Evaluators in the left navigation and click New Evaluator, or open a trace from the traces table and click Add Trace Eval in the trace header, an action that pre-sets the scope for you.

2

Name it and set Scope to Trace

In the left pane, enter a Name and set Scope to Trace. The Result Column Name is auto-populated from the name and becomes the trace_eval.<name> column.

3

Pick the judge model and write the template

Choose the judge model above the template, then write the template, referencing the variables the judge needs, for example {question}, {tool_calls}, and {output}. Because the eval sees the whole trace, write criteria about the path the agent took.

4

Define Choices and Optimization Direction

Below the template, define Choices and their scores, then set the Optimization Direction so Arize AX knows whether a higher or a lower score is better.

5

Point it at data and configure the task

Choose Project in the Configuration column - a dataset is scored one example at a time whatever the scope, so trace grouping applies only to a project. Filtering here accepts a multi-span query, available only at trace and session scope. Then map each variable and set sampling and cadence, as in Run online evals on traces.

6

Create the evaluator

Click Create Evaluator to save it to the Evaluator Hub for reuse, or Create and Run Evaluator to save it and start the task you just configured.

1

Open the New Evaluator form

Open Evaluators in the left navigation and click New Evaluator, or use the Add Session Eval action next to the session ID in the session header. Sessions exist only when your application sets session.id on its spans; see OpenInference context managers if that attribute is missing.

2

Name it and set Scope to Session

In the left pane, enter a Name and set Scope to Session. The Result Column Name is auto-populated from the name and becomes the session_eval.<name> column.

3

Choose Conversation or Turn data

Pick Conversation to have Arize AX build the transcript from each turn’s input and output. Picking it inserts {conversation} into your template. Pick Turn data when the judge needs more than input and output; that option inserts {turn_data} and reveals the Turn Definition field.

4

Write the Turn Definition (Turn data only)

Lay out one turn using {variable} references, for example:

User: {user_msg}
Assistant: {assistant_msg}
Tool result: {tool_result}

Arize AX renders this template once per turn and joins the results with TURN 1, TURN 2 markers. Variables you leave unmapped render empty rather than failing the turn.

5

Write the template, then define Choices and Optimization Direction

Choose the judge model, write the template around {conversation} or {turn_data}, then define Choices and their scores and set the Optimization Direction so Arize AX knows whether a higher or a lower score is better.

6

Point it at data

Choose Project in the Configuration column - a dataset is scored one example at a time whatever the scope, so session grouping applies only to a project. Filtering here accepts a multi-span query, available only at trace and session scope.

7

Map variables and check the Data Preview

A {conversation} eval maps nothing: Arize AX assembles the transcript from the matched session spans. For a {turn_data} eval, map each Turn Definition variable to a column. Each variable is read only from the turn it belongs to, so turn 2’s values never appear in turn 1. Data Preview shows the exact prompt the judge receives for the latest matching session. Then set sampling and cadence, as in Run online evals on traces.

8

Create the evaluator

Click Create Evaluator to save it to the Evaluator Hub for reuse, or Create and Run Evaluator to save it and start the task you just configured.

​

View results

​

Log trace and session evals from code

# Trace eval: one row per trace, keyed on that trace's root span id
trace_eval_df["context.span_id"] = trace_eval_df["root_span_id"]
trace_eval_df = trace_eval_df.rename(columns={
    "label": "trace_eval.trajectory.label",
    "score": "trace_eval.trajectory.score",
    "explanation": "trace_eval.trajectory.explanation",
})

# Session eval: one row per session, keyed on the root span of the session's first trace
session_eval_df["context.span_id"] = session_eval_df["first_root_span_id"]
session_eval_df = session_eval_df.rename(columns={
    "label": "session_eval.correctness.label",
    "score": "session_eval.correctness.score",
    "explanation": "session_eval.correctness.explanation",
})

​

Further reading