Code evals

Why use code evals

Use a code evaluator when the check is objective: a keyword appears, a URL is valid, a format follows a rule. Arize AX includes evaluators for the common checks, and you can write your own for custom logic. Both are saved to the Eval Hub, where they are versioned and reusable across tasks. See Where evaluators live.

Creating a code evaluator

You can create a code evaluator directly in the UI, or have Alyx or Arize Skills do it for you.

By Arize Skills
By Alyx
By UI
By Code

Use the Arize skills plugin in your coding agent and the arize-evaluator skill to create code evaluators and tasks via the ax CLI without leaving your editor. See the skill doc for supported commands. Then ask your agent:

“Create a code evaluator that checks if the output is valid JSON”
“Set up a regex evaluator that checks for a phone number in the response”

A custom code evaluator is a Python class that extends CodeEvaluator and implements a single evaluate method:

# Note: This example uses Python SDK v7
from typing import Any, Mapping, Optional
from arize.experimental.datasets.experiments.evaluators.base import (
    EvaluationResult,
    CodeEvaluator,
    JSONSerializable,
)

class ContainsHelloEvaluator(CodeEvaluator):
    def evaluate(
        self,
        *,
        dataset_row: Optional[Mapping[str, JSONSerializable]] = None,
        **kwargs: Any,
    ) -> EvaluationResult:
        output = dataset_row.get("attributes.output.value") if dataset_row else None
        text = str(output or "").lower()

        if "hello" in text:
            return EvaluationResult(
                label="pass",
                score=1.0,
                explanation="Output contains 'hello'"
            )

        return EvaluationResult(
            label="fail",
            score=0.0,
            explanation="Output does not contain 'hello'"
        )

For the editor template, static input parameters, and the supported package list, see Writing your own evaluator.

Pre-built evaluators

Arize manages a set of ready-to-use evaluators. Pick one from the drop-down and its Python source appears read-only in the editor, so you can see exactly what it does.

A pre-built code evaluator template in Arize AX showing its read-only Python source in the editor alongside its configurable parameters — A pre-built code evaluator, with its Python source shown read-only

Examples of the pre-built code evals Arize offers:

Eval	Description	Parameters
Matches Regex	Checks whether the text matches a specified regex pattern	span attribute: which span attribute to look at pattern: The regex pattern used for matching against the span attribute value.
JSON Parseable	Checks whether the LLM data is a valid JSON-parsable string	span attribute: which span attribute to look at
Contains any Keyword	Checks whether any specified keywords are present in the LLM data	span attribute: which span attribute to look at keywords: A list of keyword strings to search for in the span attribute. If any keyword matches, the evaluator will flag the data as a match.
Contains all Keywords	Checks that all specified keywords are present in the LLM data	span attribute: which span attribute to look at keywords: A list of keyword strings; the evaluator flags a match only when every keyword is present.
Exact Match	Checks whether the output exactly matches the expected output	span attribute: which span attribute to look at expected output: The reference string to compare against.

After you pick an evaluator:

Provide a unique Eval Column Name for the evaluator in plaintext. Ensure that this name is distinct from other evaluators across all tasks. Here, you can also set Evaluator Scope and Filters.
Define any required parameters for the selected code evaluator.

Writing your own evaluator

Select Create Custom to write your own evaluation logic in Python. The editor opens with a default template.

Custom Code Evaluators are only available in Arize AX Enterprise. Request a demo here.

The custom code evaluator editor in Arize AX with a Python evaluate method, named input parameters, and sample data mapping — The custom code evaluator editor

from typing import Any, Optional
from arize.experimental.datasets.experiments.evaluators.base import (
    EvaluationResult,
    CodeEvaluator,
)

class MyEvaluator(CodeEvaluator):
    """Custom evaluator -- edit this class to define your evaluation logic."""

    def evaluate(
        self,
        *,
        input1: Optional[str] = None,
        input2: Optional[str] = None,
        **kwargs: Any,
    ) -> EvaluationResult:
        is_valid = input1 is not None and input1.strip() != ""
        return EvaluationResult(
            label="valid" if is_valid else "invalid",
            score=float(is_valid),
            explanation=(
                "input1 is non-empty."
                if is_valid
                else "input1 is empty or missing."
            ),
        )

Replace input1 and input2 with named arguments that describe the data your evaluator needs. Each named keyword argument in the evaluate() method signature becomes a variable - when you use the evaluator in a task, you’ll map each variable to a span attribute or dataset column. You can name variables anything you want (e.g., user_query, assistant_response, ground_truth). self, dataset_row, and **kwargs are excluded from mapping. The evaluate() method must return an EvaluationResult with:

label - A categorical string (e.g., "pass", "fail")
score - A numeric value quantifying the result
explanation - A brief rationale for the result

Static input parameters

In addition to variables (which change per row), you can define static input parameters - configuration values set once that stay the same for every row. This makes evaluators reusable without editing code. For example, a regex evaluator can be reused for different patterns just by changing its pattern parameter. Static parameters are accessed via self.param_name and can be typed as String, StringArray (comma-separated list), or Regex.

class MyEvaluator(CodeEvaluator):
    def evaluate(self, *, output=None, **kwargs) -> EvaluationResult:
        score = len(output) / 100 if output else 0
        return EvaluationResult(
            label="pass" if score >= self.threshold else "fail",
            score=score,
            explanation=f"Score {score} vs threshold {self.threshold}",
        )

In this example, threshold is a static parameter configured in the UI when creating or editing the evaluator.

Accessing data via `dataset_row`

For evaluators that need access to more data than the named variables provide, include a dataset_row parameter:

def evaluate(self, *, output=None, dataset_row=None, **kwargs) -> EvaluationResult:
    # `output` is a normal variable (mapped via column mapping)
    # `dataset_row` is a dict containing ALL mapped span attributes
    user_id = dataset_row.get("attributes.metadata.user_id") if dataset_row else None

The dataset_row dictionary contains span attributes. Common keys include attributes.output.value, attributes.input.value, and attributes.llm.token_count.total. Access values using .get() to handle missing keys gracefully. When the system detects dataset_row in your method signature, the UI displays an Additional Span Attributes section where you can add extra attributes to include in the dict.

Use named variables when you know exactly which data points your evaluator needs - they are cleaner and self-documenting. Use dataset_row when you need access to a dynamic or large set of attributes that may vary between use cases.

Supported packages

Custom evaluators run in a sandboxed environment with the following packages available:

numpy
pandas
scipy
arize[Datasets]==7.25.7
pydantic==2.11.7
jellyfish==1.2.0

If you need an additional package, contact the Arize support team.

Editor features

Real-time validation: As you write code, the system validates it server-side and extracts variable names from your evaluate() method signature automatically. Errors are shown inline.
Expand-to-modal: Click the expand button to open a full-screen editor for complex evaluator code.
Split-pane layout: The left panel contains your code and configuration; the right panel shows variable mappings and a live data preview.

Next step

Once your evaluator is in the Hub, see Run evals on your data to attach it to a task, map its variables to your data, and run it.

Quickstart

Instrument

Observe

Agents

Evaluate

Improve

Machine Learning

Settings

Security

Why use code evals

Creating a code evaluator

Pre-built evaluators

Writing your own evaluator

Static input parameters

Accessing data via `dataset_row`

Supported packages

Editor features

Next step

​Why use code evals

​Creating a code evaluator

​Pre-built evaluators

​Writing your own evaluator

​Static input parameters

​Accessing data via dataset_row

​Supported packages

​Editor features

​Next step

Why use code evals

Creating a code evaluator

Pre-built evaluators

Writing your own evaluator

Static input parameters

Accessing data via `dataset_row`

Supported packages

Editor features

Next step