Setting Up Online Evals

​

What is an Evaluator?

1. LLM-as-a-Judge Evaluators:
   • An LLM template (prompt that instructs the LLM how to evaluate)
   • Rails (the possible output values/scoring scale)
   • LLM configuration (model, provider, and parameters—configured when creating the evaluator) LLM-as-a-Judge evaluators are stored in the Eval Hub and can be reused across multiple tasks.
2. Code Evaluators:
   • Code definition (Python code that performs the evaluation)
   • Column mappings (which trace/span attributes to use as inputs) Code evaluators are defined inline when creating a task.

• Relevance: Does the response answer the question?
• Toxicity: Is the content safe and appropriate?
• Factual Accuracy: Are the claims in the response correct?
• Helpfulness: Is the response useful to the user?
• Your success metrics: Whatever success means for your agent or organization—evaluators can measure any quality dimension that matters to you

💡 Tip: Template evaluators are great for subjective quality measures, while code evaluators excel at objective checks (e.g., checking if a response contains required keywords or follows a specific format).

​

What is a Task?

• Continuously evaluates your traces/spans
• Applies filters to target specific data (e.g., only LLM spans)
• Runs evaluators at a specified sampling rate
• Attaches evaluation results directly to your traces

​

Prerequisites

• Data source: You need either:
  • Traces flowing into an Arize AX Project: Your application should be sending traces/spans to Arize. If you haven’t set this up yet, see our Tracing documentation.
  • OR a Dataset with an Experiment: A dataset with an associated experiment to evaluate.
• LLM Integration configured: You’ll need an LLM provider integration set up for your evaluators. See AI Provider Integrations.

​

Step 1: Create an Evaluator

​

1. LLM-as-a-Judge Evaluator

1

Arize provides pre-built evaluation templates tested against benchmarked datasets:

• Hallucination: Detects when outputs contain information not in the reference
• Relevance: Measures if responses are relevant to the input
• Toxicity: Detects harmful or inappropriate content
• Helpfulness: Assesses how useful the response is
• Q&A Correctness: Evaluates if answers are correct given reference documents
• Summarization: Checks if summaries accurately capture the input
• User Frustration: Detects signs of user frustration in conversations
• Code Generation: Evaluates code correctness and readability
• SQL Generation: Checks SQL query correctness
• Tool Calling: Validates function calls and extracted parameters

Select a template and configure:

• Name: Give your evaluator a descriptive name
• LLM Config: Select the model, provider, and parameters for evaluation

2

Build a custom evaluator from scratch:

1. Name: Give it a descriptive name
2. Template: Write your evaluation prompt template with variables (e.g., {question}, {response}, {context})
3. Rails: Define the possible output values/scoring scale
4. LLM Config: Select which LLM to use for evaluation (model, provider, parameters)

3

Alyx can generate custom evaluators from plain language descriptions:

1. Click the ✨ Alyx icon in the upper right corner
2. Use a prompt like:
   Copy

   Ask AI

   Write a custom evaluation that checks if customer support responses 
   are empathetic, address the customer's concern, and provide actionable 
   next steps. Score from 1-5.
   
3. Review and customize the generated evaluator as needed

​

2. Code Evaluator

• Matches Regex: Check if text matches a regex pattern
• JSON Parseable: Validate JSON format
• Contains any Keyword: Check for presence of keywords
• Contains all Keywords: Ensure all specified keywords are present

​

Step 2: Use the Evaluator in a Task

1. From the Evaluators page, click New Task
2. Select LLM-as-a-Judge
3. Click Add Evaluator and select your evaluator from the Eval Hub

1. Navigate to the Eval Hub tab on the Evaluators page
2. Find your evaluator and click the Use Evaluator button
3. This opens the task creation flow with your evaluator pre-selected

​

Step 3: Configure the Task

​

Select Data Source

• Project: Evaluate traces from a specific project. Your application should be sending traces/spans to Arize.
• Dataset: Evaluate examples from a dataset with an associated experiment.

​

Select Cadence (⚠️ only when Data Source is Project)

• Run continuously on new incoming data (recommended for production): Task runs automatically every 2 minutes on new traces. Best for ongoing monitoring.
• Run once (historical): Task runs on a batch of existing data. Best for evaluating historical data or one-time assessments.

​

Configure Data Filters (⚠️ only when Data Source is Project)

• Span Kind: Only evaluate specific span types (e.g., LLM spans)
• Model Name: Only evaluate specific models
• Metadata: Only evaluate spans with certain metadata tags
• Span Attributes: Filter on any span attribute

​

Configure Sampling Rate (⚠️ only when Data Source is Project)

• 100%: Evaluate every trace (useful for low-volume, critical applications)
• 10-50%: Common for high-volume applications to balance cost and coverage
• 1-5%: For very high-volume applications where you want representative sampling

💡 Tip: Start with a lower sampling rate (10-20%) and increase it once you’ve validated your evaluators are working correctly.

​

Step 4: Configure Evaluator Settings

​

Evaluation Granularity (⚠️ only when Data Source is Project)

• Span: Evaluate individual spans (e.g., a single LLM call)
• Trace: Evaluate entire traces (the full request flow)
• Session: Evaluate across sessions (multi-turn conversations)

​

Column Mappings

1. The column mappings panel displays all variables from the evaluator template
2. For each variable, select the corresponding column from your datasource
3. The output variable is special—it refers to the LLM output

​

Step 5: Create Task and Monitor

1. Review your task configuration and click Create Task
2. Navigate to the Running Tasks tab to find your task
3. Click View Logs to see detailed logs for each task run, including:
   • Run timing and status
   • Number of successes, errors, and skips
   • Any error messages that occurred