> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
> Score whole conversations, not single turns — coherence, goal completion, frustration, and correctness — on a multi-turn AI tutor using Arize AX.
# Session-Level Evaluation: Scoring the Whole Conversation
Most evals score a **single turn**. But a tutor — like a support agent or any assistant you talk to over time — isn't one turn, it's a **session**, and the properties you care about are **emergent**: they only exist across the whole conversation. **Coherence** is a relationship *between* turns; **goal completion** happens over the arc of the session; **frustration** *builds*. The trap: every individual turn can look fine while the session as a whole fails. Turn-level checks pass each turn and miss it.
The method: trace the full multi-turn conversation under a shared `session_id`, aggregate its spans into one ordered transcript, run **session-scoped judges** that read the *whole* conversation, then log each result back onto the session.
We'll go through the following steps:
* Run a multi-turn AI tutor traced as sessions — with a simulated student so the notebook runs top-to-bottom, no manual typing
* Aggregate each session's spans into one ordered transcript
* Evaluate each session on four session-only dimensions — **coherence, goal completion, frustration, and correctness**
* See a controlled example where every turn looks fine but the session fails
* Log the results back to Arize AX as session-level evaluations
***
## Before you start
You need an [Arize AX account](https://app.arize.com/auth/join) and an OpenAI API key.
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
pip install "arize>=8" arize-otel "arize-phoenix-evals>=3" \
openinference-instrumentation-openai openinference-instrumentation openai pandas
```
```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
export ARIZE_SPACE_ID=""
export ARIZE_API_KEY=""
export OPENAI_API_KEY=""
```
```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import os
from arize.otel import register
from openai import OpenAI
from openinference.instrumentation.openai import OpenAIInstrumentor
model_id = "ai-tutor-session"
MODEL = "gpt-5.4-mini"
JUDGE_MODEL = "gpt-4.1"
tracer_provider = register(
space_id=os.environ["ARIZE_SPACE_ID"],
api_key=os.environ["ARIZE_API_KEY"],
project_name=model_id,
set_global_tracer_provider=True,
)
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)
client = OpenAI()
```
## Build AI Tutor with Session Tracking
The tutor teaches Socratically across several turns. To keep the guide runnable with no manual typing, a second LLM call plays the **student**. Only the **tutor** calls are wrapped in `using_attributes(session_id=...)`, so each session's spans share a `session.id`; the student calls run under `suppress_tracing()` so they stay out of the project.
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
from openinference.instrumentation import suppress_tracing
def student_reply(persona: str, transcript: list) -> str:
"""Simulate the student's next message with a second (untraced) LLM call."""
convo = "\n".join(f"{speaker.capitalize()}: {text}" for speaker, text in transcript)
system = (
"You are role-playing a STUDENT in a tutoring session. "
f"Your persona: {persona} "
"Reply in 1-2 sentences, in character, as the student's next message. "
"When your question has been fully answered, or you've given up, reply with exactly DONE."
)
# suppress_tracing keeps these scaffolding calls out of the ai-tutor-session
# project, so each session's spans are only the tutor's turns.
with suppress_tracing():
resp = client.chat.completions.create(
model=MODEL,
messages=[
{"role": "system", "content": system},
{"role": "user", "content": convo},
],
)
return resp.choices[0].message.content.strip()
```
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import uuid
from openinference.instrumentation import using_attributes
def run_session(user_id: str, topic: str, question: str, persona: str, max_turns: int = 5) -> str:
session_id = f"tutor-{uuid.uuid4()}"
system = (
f"You are a thoughtful AI tutor teaching {topic}. "
"Ask questions, give hints, and only give the full answer once the student "
"shows correct reasoning. Keep each reply to 3-5 sentences."
)
messages = [{"role": "user", "content": question}]
transcript = [("student", question)]
for _ in range(max_turns):
# Only the tutor call carries the session attributes.
with using_attributes(session_id=session_id, user_id=user_id):
resp = client.chat.completions.create(
model=MODEL,
messages=[{"role": "system", "content": system}] + messages,
)
tutor_text = resp.choices[0].message.content.strip()
messages.append({"role": "assistant", "content": tutor_text})
transcript.append(("tutor", tutor_text))
student_text = student_reply(persona, transcript) # untraced
if student_text.strip().upper().startswith("DONE"):
break
messages.append({"role": "user", "content": student_text})
transcript.append(("student", student_text))
return session_id
```
Run two sessions with contrasting student personas, so the evaluators have signal to separate: an engaged learner who gets there, and a struggling one who may not.
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
sessions = [
{
"user_id": "student-anya",
"topic": "Science",
"question": "Why is the sky blue?",
"persona": (
"Curious and engaged. You follow the tutor's hints, reason out loud, "
"and stop once it clicks."
),
},
{
"user_id": "student-ben",
"topic": "Math (derivatives)",
"question": "How do I find the derivative of f(x) = x^2 * sin(x)?",
"persona": (
"Easily confused and impatient. You keep saying you don't get it, get "
"frustrated, and may give up before fully understanding."
),
},
]
for session in sessions:
print(f"=== Session: {session['user_id']} - {session['topic']} ===")
run_session(**session)
```
## Aggregate Spans into Session Transcripts
Export every span, group by `attributes.session.id`, and rebuild a clean role-tagged transcript from each tutor turn's structured `llm.input_messages` / `llm.output_messages`. We also keep one `context.span_id` per session — the handle we'll attach the session evals onto at the end.
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import json
import time
from datetime import datetime, timedelta, timezone
import pandas as pd
from arize.client import ArizeClient
# Spans are exported in batches and ingested asynchronously, so flush and wait
# before querying. If the export comes back empty, wait longer and run it again.
tracer_provider.force_flush()
time.sleep(30)
ax_client = ArizeClient(api_key=os.environ["ARIZE_API_KEY"])
primary_df = ax_client.spans.export_to_df(
space_id=os.environ["ARIZE_SPACE_ID"],
project_name=model_id,
start_time=datetime.now(timezone.utc) - timedelta(days=7),
end_time=datetime.now(timezone.utc),
)
SESSION_ID = "attributes.session.id"
def _as_list(value):
"""Span message attributes arrive as a JSON string or an already-parsed list."""
if isinstance(value, str):
try:
return json.loads(value)
except json.JSONDecodeError:
return []
return list(value) if isinstance(value, (list, tuple)) else []
def _message_text(message):
"""Text of an OpenInference message (flat content or multi-part content)."""
content = message.get("message.content")
if content:
return content
parts = message.get("message.contents") or []
return " ".join(
p.get("message_content.text", "") for p in parts if isinstance(p, dict)
).strip()
def prepare_sessions(df: pd.DataFrame) -> pd.DataFrame:
df = df[df[SESSION_ID].notna()]
sessions = []
for session_id, group in df.sort_values("start_time").groupby(SESSION_ID):
lines = []
for _, row in group.iterrows():
inputs = _as_list(row.get("attributes.llm.input_messages"))
user_turns = [_message_text(m) for m in inputs if m.get("message.role") == "user"]
if user_turns and user_turns[-1]:
lines.append(f"user: {user_turns[-1]}")
for m in _as_list(row.get("attributes.llm.output_messages")):
if (text := _message_text(m)):
lines.append(f"assistant: {text}")
sessions.append(
{
"session_id": session_id,
"messages": "\n\n".join(lines),
"trace_count": group["context.trace_id"].nunique(),
"span_id": group["context.span_id"].iloc[0], # handle for logging
}
)
return pd.DataFrame(sessions)
sessions_df = prepare_sessions(primary_df)
```
## Define the Session-Level Evaluators
Each evaluator reads the **entire transcript** and scores one session-only property — none could be computed from a single turn. All four read the same `messages` column and run together with `async_evaluate_dataframe`.
Each prompt is scoped to one concern and told explicitly what *not* to judge, so the four scores stay independent rather than all tracking overall quality.
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
SESSION_COHERENCE_PROMPT = """
You are evaluating the COHERENCE of a multi-turn tutoring session between a student and an AI tutor.
You will be given the full session transcript, in order. Messages from the student have the role `user`; messages from the AI tutor have the role `assistant`.
A coherent session:
- Stays internally consistent - the tutor never contradicts something it established in an earlier turn
- Builds on previous turns instead of resetting or ignoring what was already said
- Keeps track of the topic, the student's question, and prior answers as the conversation progresses
- Uses terminology and explanations consistently throughout
##
Session transcript:
{messages}
##
Judge ONLY coherence across turns - not factual correctness, and not whether the student's goal was met.
Respond with a single word: `coherent` or `incoherent`.
- `coherent` -> the session is internally consistent and builds on itself across turns.
- `incoherent` -> the tutor contradicts an earlier turn, loses track of context, or the conversation fails to hang together.
"""
SESSION_GOAL_COMPLETION_PROMPT = """
You are evaluating whether the AI tutor helped the student COMPLETE their learning goal over the course of a session.
You will be given the full session transcript, in order. Messages from the student have the role `user`; messages from the AI tutor have the role `assistant`.
To decide whether the goal was completed, consider:
- Whether the tutor directly addressed the question the student came in with
- Whether the explanations actually resolved the student's doubt or problem
- Whether the student's later messages indicate understanding or closure
- Whether the conversation logically progressed to completing the student's objective (rather than drifting onto tangents and never returning)
##
Session transcript:
{messages}
##
Respond with a single word: `completed` or `not_completed`.
- `completed` -> the session met the student's learning goal and resolved their question.
- `not_completed` -> the session left the student's original question unanswered or their goal unmet.
"""
SESSION_FRUSTRATION_PROMPT = """
You are evaluating whether the student became FRUSTRATED at any point during a tutoring session with an AI tutor.
You will be given the full session transcript, in order. Messages from the student have the role `user`; messages from the AI tutor have the role `assistant`.
Signs of student frustration include:
- Repeating or rephrasing the same question multiple times without resolution
- Expressing confusion ("I don't get it", "this doesn't make sense")
- Expressing annoyance, impatience, or disengagement ("ugh", "forget it", "this is taking forever")
- Abruptly giving up or ending the session
##
Session transcript:
{messages}
##
Respond with a single word: `frustrated` or `not_frustrated`.
- `frustrated` -> the student showed confusion, impatience, or frustration at any point.
- `not_frustrated` -> the student stayed engaged and satisfied throughout.
"""
SESSION_CORRECTNESS_PROMPT = """
You are evaluating the CORRECTNESS and educational soundness of an AI tutor's session with a student.
You will be given the full session transcript, in order. Messages from the student have the role `user`; messages from the AI tutor have the role `assistant`.
A correct tutoring session:
- Provides factually and conceptually accurate explanations
- Correctly answers the student's questions
- Clarifies misunderstandings rather than reinforcing them
- Avoids hallucinations, vague non-answers, or incorrect reasoning
##
Session transcript:
{messages}
##
Judge ONLY correctness and educational soundness - not coherence, and not whether the student's goal was ultimately met.
Respond with a single word: `correct` or `incorrect`.
- `correct` -> the tutor's explanations are accurate, clear, and educationally sound throughout.
- `incorrect` -> the tutor gives factually wrong, misleading, or incorrect explanations at any point.
"""
```
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import asyncio
from phoenix.evals import LLM, ClassificationEvaluator, async_evaluate_dataframe
judge = LLM(provider="openai", model=JUDGE_MODEL)
coherence_evaluator = ClassificationEvaluator(
name="coherence", llm=judge, prompt_template=SESSION_COHERENCE_PROMPT,
choices={"coherent": 1.0, "incoherent": 0.0},
)
goal_completion_evaluator = ClassificationEvaluator(
name="goal_completion", llm=judge, prompt_template=SESSION_GOAL_COMPLETION_PROMPT,
choices={"completed": 1.0, "not_completed": 0.0},
)
frustration_evaluator = ClassificationEvaluator(
name="frustration", llm=judge, prompt_template=SESSION_FRUSTRATION_PROMPT,
choices={"not_frustrated": 1.0, "frustrated": 0.0},
)
correctness_evaluator = ClassificationEvaluator(
name="correctness", llm=judge, prompt_template=SESSION_CORRECTNESS_PROMPT,
choices={"correct": 1.0, "incorrect": 0.0},
)
session_evaluators = [
coherence_evaluator, goal_completion_evaluator, frustration_evaluator, correctness_evaluator,
]
with suppress_tracing():
results_df = asyncio.run(
async_evaluate_dataframe(
dataframe=sessions_df, evaluators=session_evaluators, concurrency=10
)
)
```
## Seeing what session-level evals catch
Running the four judges on four hand-written transcripts — each built so every *turn* is locally fine but one *session* property fails — shows each dimension catching its own failure. A turn-by-turn check would pass every individual turn in all four:
| case | coherence | goal\_completion | frustration | correctness |
| :------------------------------------- | :------------- | :----------------- | :-------------- | :------------ |
| clean | coherent | completed | not\_frustrated | correct |
| incoherent (tutor contradicts itself) | **incoherent** | completed | not\_frustrated | **incorrect** |
| goal not met (drifts onto tangents) | coherent | **not\_completed** | not\_frustrated | correct |
| frustrated (student visibly impatient) | coherent | completed | **frustrated** | correct |
The `incoherent` row shows a real cascade: a self-contradiction also reads as less *correct*, because these dimensions aren't fully independent — itself worth seeing.
## Log Evaluations Back to Arize AX
Arize AX routes columns named `session_eval..label/score/explanation` to the **session** that the row's `context.span_id` belongs to — so we attach each session's four results to one of its spans (the `span_id` we kept earlier). They appear on each session in the **Sessions** tab.
```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
def _unpack(cell):
"""async_evaluate_dataframe stores each evaluator's result as a Score dict."""
if isinstance(cell, dict):
return cell.get("label"), cell.get("score"), cell.get("explanation")
return getattr(cell, "label", None), getattr(cell, "score", None), getattr(cell, "explanation", None)
SESSION_EVALS = ["coherence", "goal_completion", "frustration", "correctness"]
eval_df = pd.DataFrame()
for name in SESSION_EVALS:
unpacked = results_df[f"{name}_score"].apply(_unpack)
eval_df[f"session_eval.{name}.label"] = unpacked.map(lambda t: t[0]).values
eval_df[f"session_eval.{name}.score"] = unpacked.map(lambda t: t[1]).values
eval_df[f"session_eval.{name}.explanation"] = unpacked.map(lambda t: t[2]).values
# Attach to one span per session (drop=False keeps the column update_evaluations requires);
# Arize AX routes session_eval.* to that span's session.
span_for_session = dict(zip(sessions_df["session_id"], sessions_df["span_id"]))
eval_df["context.span_id"] = [span_for_session[sid] for sid in results_df["session_id"]]
log_df = eval_df.set_index("context.span_id", drop=False)
resp = ax_client.spans.update_evaluations(
space_id=os.environ["ARIZE_SPACE_ID"],
project_name=model_id,
dataframe=log_df,
)
```
## View Results in Arize AX
After logging, open the **Sessions** tab of your Arize AX project. Each session carries its four session-level evaluations — `coherence`, `goal_completion`, `frustration`, and `correctness` — each with a label, score, and explanation, letting you:
* Monitor session-level quality, not just per-turn
* Spot sessions that pass every turn yet fail as a whole — an unmet goal, a contradiction, a frustrated user
* Track engagement and goal completion across sessions
The pattern generalizes: for any session-only property, aggregate the session's spans into one transcript, write a judge that reads the whole thing, and log it back as a session-level evaluation — right next to your turn-level and trace-level evals.