> ## Documentation Index
> Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt
> Use this file to discover all available pages before exploring further.
# Evaluator Types — Human, Code, and LLM-as-a-Judge
> Three kinds of evaluators — human, code-based, and LLM-as-a-judge. What each is good at, what each isn't, and how to choose between them.
export const AskAlyx = ({children}) => {
const gradientId = `askAlyxGradient-${Math.random().toString(36).slice(2)}`;
return ;
};
The level of an evaluator tells you what slice of data it looks at. The **type** tells you how it makes its judgment. Three kinds of evaluators exist, and they aren't interchangeable — each is suited to a different class of question.
# The three types
| Type | How it works | Where it shines | Where it doesn't |
| :----------------- | :------------------------------------------------------------- | :------------------------------------------------------------------------------------------------------------------ | :--------------------------------------------------------------------- |
| **Human** | A reviewer scores outputs against a rubric. | Capturing nuance and subject-matter expertise; defining golden datasets that calibrate other evaluators. | Slow, expensive, inconsistent across reviewers, doesn't scale. |
| **Code-based** | Deterministic Python (or built-in templates) check the output. | Anything that can be programmatically checked — string contains, JSON shape, regex match, deterministic comparison. | Anything subjective — tone, helpfulness, factuality. |
| **LLM-as-a-judge** | An LLM reads the data and emits a label and explanation. | Subjective qualities at scale, with minimal up-front data; flexible across domains. | Non-deterministic; requires prompt design; vulnerable to known biases. |
# Human evaluation
Human evaluation is the calibration target for everything else. When you want to know whether your other evaluators are *correct*, you compare them to human-labeled data.
What humans are good at:
* **Nuance and context.** A human reviewer with domain knowledge can spot subtle factual errors, tone problems, or context-specific failures that pattern-matching can't.
* **Defining ground truth.** Golden datasets — small, carefully-labeled collections of input/output pairs — are how every other evaluator gets validated. They almost always start with human labeling.
* **Edge cases.** Rare failure modes that no automated system was designed to catch.
What humans aren't good at:
* **Scale.** A human reviewer can read maybe a few hundred traces a day; a production application generates orders of magnitude more.
* **Speed.** Human turnaround is hours-to-days; automated turnaround is seconds-to-minutes.
* **Consistency.** Two reviewers, given the same rubric, will disagree on a non-trivial fraction of cases. One reviewer, given the same trace twice on different days, will sometimes disagree with themselves.
* **Bandwidth.** Humans cannot evaluate against four million tokens of context. Pattern-matching tasks like "find spans that match X criteria across the last month of data" are not human tasks.
Use humans to **define and validate** golden datasets, and to spot-check production. Don't use humans as a continuous evaluator for any meaningful volume.
# Code-based evaluators
Code-based evaluators run pure Python (or a built-in template) against the output. They are fast, cheap, deterministic, and repeatable — which makes them the right choice whenever the question can be expressed as a check.
What code is good at:
* **Deterministic criteria.** "Does the output contain a competitor's name?" "Is the JSON shape correct?" "Did the function call use the right parameter names?" Anything where a correct/incorrect answer is fully determined by the data.
* **Reliability.** A code evaluator gives the same answer every time. No retry loops, no rate limits, no model drift.
* **Speed and cost.** Running a code evaluator is essentially free compared to an LLM call.
What code isn't good at:
* **Anything subjective.** Tone, factuality, helpfulness, coherence — these can't be checked by pattern-matching.
* **Anything that needs reasoning.** "Is the response consistent with the retrieved context?" requires understanding both pieces, which code can't do.
A common pattern: combine code evaluators (for the deterministic checks) with LLM-as-a-judge evaluators (for the subjective ones) and use the code ones as cheap pre-filters before paying for the LLM ones.
# LLM-as-a-judge
LLM-as-a-judge uses a pre-trained LLM to read the data and emit a label, score, and explanation.
What LLM-as-a-judge is good at:
* **Subjective evaluation at scale.** Tone, helpfulness, factuality, coherence — questions that need reasoning.
* **Flexibility.** A new evaluator is a new prompt, not a new model.
* **No training data required up front.** You'll want a small golden dataset to validate the judge, but you don't need thousands of examples to get started.
What LLM-as-a-judge isn't good at:
* **Determinism.** The same judge, run twice on the same input, can disagree with itself.
* **Cost-free operation.** Every eval is an LLM call. At production volume, this matters — see [Filters, scope, and cadence](/docs/ax/concepts/evaluators/filters-scope-and-cadence) for the sampling levers.
* **Resistance to bias.** Verbosity bias, position bias, self-enhancement bias — see [Evaluator best practices](/docs/ax/concepts/evaluators/evaluator-best-practices) for the catalog and how to guard against them.
For the canonical LLM-as-a-judge survey paper, see Gu et al., 2024 — [A Survey on LLM-as-a-Judge](https://arxiv.org/abs/2411.15594).
# Choosing between types
A decision tree:
1. **Is the question deterministic?** (Yes = code-based. Done.)
2. **Do you need to evaluate at production scale?** (No = human is fine for small samples.)
3. **Default for everything else** = LLM-as-a-judge.
**Ask Alyx** to suggest the right evaluator for a check you describe in plain language.
Most production setups end up running multiple types in parallel — code evaluators for the cheap deterministic checks, LLM-as-a-judge evaluators for the subjective questions, periodic human review to validate the judges. The three types complement each other; they don't compete.
***
## Next step
Whichever type you pick, every evaluator has the same shape — inputs, outputs, and metadata. The next page covers that anatomy: