> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Pydantic Evals

1. Use Pydantic Evals to evaluate your LLM app for a simple question-answering task.
2. Log your results to Arize AX to track your experiments and traces.

## Install dependencies

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
pip install pydantic-evals "arize[Tracing]" arize-otel openai \
  openinference-instrumentation-openai
```

## Set your credentials

Copy your **Space ID** and **API key** from your Arize AX Space Settings page and set them as environment variables alongside your OpenAI key.

```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
export ARIZE_SPACE_ID="<your-space-id>"
export ARIZE_API_KEY="<your-api-key>"
export OPENAI_API_KEY="<your-openai-api-key>"
```

## Setup Arize AX

Add our auto-instrumentation for OpenAI using arize-otel.

```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
import os

from openai import OpenAI
from pydantic_evals import Case, Dataset

from arize.otel import register

tracer_provider = register(
    space_id=os.environ["ARIZE_SPACE_ID"],
    api_key=os.environ["ARIZE_API_KEY"],
    project_name="pydantic-evals-tutorial",
)

from openinference.instrumentation.openai import OpenAIInstrumentor
OpenAIInstrumentor().instrument(tracer_provider=tracer_provider)
```

## Define the Evaluation Dataset

Create a dataset of test cases using Pydantic Evals for a question-answering task.

1. Each Case represents a single test with an input (question) and an expected output (answer).
2. The Dataset aggregates these cases for evaluation.

```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
cases = [
    Case(name="capital of France", inputs="What is the capital of France?", expected_output="Paris"),
    Case(name="author of Romeo and Juliet", inputs="Who wrote Romeo and Juliet?", expected_output="William Shakespeare"),
    Case(name="largest planet", inputs="What is the largest planet in our solar system?", expected_output="Jupiter")
]
dataset = Dataset(cases=cases)
```

## Setup LLM task to evaluate

```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

def evaluate_case(case):
    response = client.chat.completions.create(
        model="gpt-5.4-mini",
        messages=[{"role": "user", "content": case.inputs}]
    )
    output = response.choices[0].message.content
    print(output)
    is_correct = case.expected_output.lower() in output.strip().lower()
    return is_correct
```

## Run your experiment and evaluation

```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}}
results = [evaluate_case(case) for case in dataset.cases]

for case, result in zip(dataset.cases, results):
    print(f"Case: {case.name}, Correct: {result}")
```

## View results in Arize AX

<Frame>
  ![](https://storage.googleapis.com/arize-phoenix-assets/assets/images/arize-docs-images/cookbooks/image-8.png)
</Frame>