Evaluators

The evaluators functions are currently in BETA. The API may change without notice. A one-time warning is emitted on first use.

List Evaluators

import { listEvaluators } from "@arizeai/ax-client";

const { data: evaluators, pagination } = await listEvaluators({
  space: "my-space",  // space name or ID (optional)
  name: "Relevance",  // substring filter on evaluator name (optional)
  limit: 10,
});

Create a Template (LLM-as-Judge) Evaluator

import { createTemplateEvaluator } from "@arizeai/ax-client";

const evaluator = await createTemplateEvaluator({
  name: "Relevance",
  space: "my-space",  // space name or ID
  commitMessage: "Initial version",
  templateConfig: {
    name: "Relevance",
    template: "Is the following response relevant to the query?\nQuery: {{query}}\nResponse: {{response}}",
    includeExplanations: true,
    useFunctionCallingIfAvailable: true,
    classificationChoices: { relevant: 1, irrelevant: 0 },
    direction: "MAXIMIZE",
    llmConfig: {
      aiIntegrationId: "your_ai_integration_id",
      modelName: "gpt-4o",
      invocationParameters: { temperature: 0 },
      providerParameters: {},
    },
  },
});

Create a Code Evaluator

Use a built-in MANAGED code check (e.g. JSON_PARSEABLE, MATCHES_REGEX, CONTAINS_ANY_KEYWORD, CONTAINS_ALL_KEYWORDS, EXACT_MATCH) or a CUSTOM user-supplied Python evaluator.

import { createCodeEvaluator } from "@arizeai/ax-client";

const evaluator = await createCodeEvaluator({
  name: "JSON Parseable",
  space: "my-space",
  commitMessage: "Initial version",
  codeConfig: {
    type: "MANAGED",
    name: "json_parseable",
    managedEvaluator: "JSON_PARSEABLE",
    variables: ["output"],
  },
});

Create a Remote Evaluator

Remote evaluators call a customer-hosted HTTP endpoint on each evaluation run. The endpoint must already exist as an EVALUATOR integration created via createIntegration (or POST /v2/integrations). The enableRemoteEvalTasks feature flag must be enabled on the account.

import { createRemoteEvaluator } from "@arizeai/ax-client";

const evaluator = await createRemoteEvaluator({
  name: "My Remote Evaluator",
  space: "my-space",  // space name or ID
  integrationId: "your_evaluator_integration_id",  // global ID of the EVALUATOR integration
  commitMessage: "Initial version",
  description: "Calls our hosted scoring endpoint",  // optional
});

Get an Evaluator

import { getEvaluator } from "@arizeai/ax-client";

// Get with the latest version (by name + space)
const evaluator = await getEvaluator({
  evaluator: "Relevance",
  space: "my-space",  // required when resolving by evaluator name
});

// Get with a specific version
const byVersion = await getEvaluator({
  evaluator: "Relevance",
  space: "my-space",
  versionId: "your_version_id",
});

Get an Evaluator Version

import { getEvaluatorVersion } from "@arizeai/ax-client";

const version = await getEvaluatorVersion({ versionId: "your_version_id" });

Update an Evaluator

At least one of name or description must be provided.

import { updateEvaluator } from "@arizeai/ax-client";

const updated = await updateEvaluator({
  evaluator: "Relevance",
  space: "my-space",  // required when resolving by evaluator name
  name: "Updated Evaluator Name",
  description: "Updated description",
});

Delete an Evaluator

This operation is irreversible — it deletes the evaluator and all its versions.

import { deleteEvaluator } from "@arizeai/ax-client";

await deleteEvaluator({
  evaluator: "Relevance",
  space: "my-space",  // required when resolving by evaluator name
});

Manage Versions

List Versions

import { listEvaluatorVersions } from "@arizeai/ax-client";

const { data: versions, pagination } = await listEvaluatorVersions({
  evaluator: "Relevance",
  space: "my-space",  // required when resolving by evaluator name
  limit: 10,
});

Create a New Template Version

Versions are immutable once created. To change configuration, create a new version.

import { createTemplateEvaluatorVersion } from "@arizeai/ax-client";

const version = await createTemplateEvaluatorVersion({
  evaluator: "Relevance",          // evaluator name or ID
  space: "my-space",               // required when resolving by evaluator name
  commitMessage: "Updated prompt template for better precision",
  templateConfig: {
    name: "Relevance",
    template: "Rate the relevance of the response on a scale of 0-1.\nQuery: {{query}}\nResponse: {{response}}",
    includeExplanations: true,
    useFunctionCallingIfAvailable: true,
    classificationChoices: { relevant: 1, irrelevant: 0 },
    direction: "MAXIMIZE",
    llmConfig: {
      aiIntegrationId: "your_ai_integration_id",
      modelName: "gpt-4o",
      invocationParameters: { temperature: 0 },
      providerParameters: {},
    },
  },
});

Create a New Code Version

import { createCodeEvaluatorVersion } from "@arizeai/ax-client";

const version = await createCodeEvaluatorVersion({
  evaluator: "JSON Parseable",     // evaluator name or ID
  space: "my-space",               // required when resolving by evaluator name
  commitMessage: "Updated managed evaluator",
  codeConfig: {
    type: "MANAGED",
    name: "json_parseable",
    managedEvaluator: "JSON_PARSEABLE",
    variables: ["output"],
  },
});

Create a New Remote Version

The new version becomes the latest immediately. The integration is shared across evaluator versions — updating it affects all versions that reference it.

import { createRemoteEvaluatorVersion } from "@arizeai/ax-client";

const version = await createRemoteEvaluatorVersion({
  evaluator: "My Remote Evaluator",  // evaluator name or ID
  space: "my-space",                 // required when resolving by evaluator name
  integrationId: "your_evaluator_integration_id",  // global ID of the EVALUATOR integration
  commitMessage: "Switch to new endpoint",
});

Delete Versions

Deletes a batch of evaluator versions by their IDs (1-100 per call). This operation is irreversible but partial-tolerant: versions that belong to the evaluator are deleted, requested IDs that were not deleted are reported back, and re-submitting already-deleted IDs is safe.

import { deleteEvaluatorVersions } from "@arizeai/ax-client";

const result = await deleteEvaluatorVersions({
  evaluator: "Relevance",  // evaluator name or ID
  space: "my-space",       // required when resolving by evaluator name
  versionIds: ["your_version_id_1", "your_version_id_2"],
});
console.log(result.deletedVersionIds);

Returns a DeleteEvaluatorVersionsResult with completed (boolean), deletedVersionIds (string[]), and notDeletedVersionIds (string[]).

Skills

MCP server

API reference

API clients

List Evaluators

Create a Template (LLM-as-Judge) Evaluator

Create a Code Evaluator

Create a Remote Evaluator

Get an Evaluator

Get an Evaluator Version

Update an Evaluator

Delete an Evaluator

Manage Versions

List Versions

Create a New Template Version

Create a New Code Version

Create a New Remote Version

Delete Versions

​List Evaluators

​Create a Template (LLM-as-Judge) Evaluator

​Create a Code Evaluator

​Create a Remote Evaluator

​Get an Evaluator

​Get an Evaluator Version

​Update an Evaluator

​Delete an Evaluator

​Manage Versions

​List Versions

​Create a New Template Version

​Create a New Code Version

​Create a New Remote Version

​Delete Versions

List Evaluators

Create a Template (LLM-as-Judge) Evaluator

Create a Code Evaluator

Create a Remote Evaluator

Get an Evaluator

Get an Evaluator Version

Update an Evaluator

Delete an Evaluator

Manage Versions

List Versions

Create a New Template Version

Create a New Code Version

Create a New Remote Version

Delete Versions