> ## Documentation Index > Fetch the complete documentation index at: https://arize-ax.mintlify.site/docs/llms.txt > Use this file to discover all available pages before exploring further. # Prompt Experimentation For Summarization Task Summarization is a good task for prompt experimentation because the output is hard to eyeball at scale: two summaries can both look fine and still score very differently against a human reference. This guide builds a summarization task, scores it with ROUGE, and then runs the *same* dataset and evaluators against three different prompts so you can see which one actually wins in Arize AX. ## Before you start You need an [Arize AX account](https://app.arize.com/auth/join) and an OpenAI API key. ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} pip install "arize[Datasets]<8" openai datasets pyarrow rouge ``` ```bash theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} export ARIZE_SPACE_ID="" export ARIZE_API_KEY="" export ARIZE_DEVELOPER_KEY="" export OPENAI_API_KEY="" ``` ## Create a dataset Connect to Arize AX with the Arize SDK and upload a dataset. We use ten rows from [cnn\_dailymail](https://huggingface.co/datasets/abisee/cnn_dailymail), which pairs each news article with a human-written summary. Those reference summaries are what the evaluators score against, so the dataset needs both an `article` and a `summary` column. ```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} # Note: This example uses Python SDK v7 import os from uuid import uuid1 from datasets import load_dataset from arize.experimental.datasets import ArizeDatasetsClient from arize.experimental.datasets.utils.constants import GENERATIVE SPACE_ID = os.environ["ARIZE_SPACE_ID"] hf_ds = load_dataset("abisee/cnn_dailymail", "3.0.0") df = ( hf_ds["test"] .to_pandas() .sample(n=10, random_state=0) .rename(columns={"highlights": "summary"})[["article", "summary"]] .reset_index(drop=True) ) arize_client = ArizeDatasetsClient( developer_key=os.environ.get("ARIZE_DEVELOPER_KEY"), api_key=os.environ["ARIZE_API_KEY"], ) dataset_name = "summarization-" + str(uuid1())[:5] dataset_id = arize_client.create_dataset( space_id=SPACE_ID, dataset_name=dataset_name, dataset_type=GENERATIVE, data=df, ) print(f"created dataset '{dataset_name}' (id: {dataset_id})") ``` ## Define our experiment Now let's define our experiment. The task summarizes one article per dataset row. Taking the prompt template and the model as arguments is what lets us reuse the identical task across prompt variants later — only the template changes. ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} from typing import Any, Dict from openai import OpenAI client = OpenAI() MODEL = "gpt-5.5" def summarize_article(dataset_row: Dict[str, Any], prompt_template: str, model: str) -> str: # dataset_row is a dictionary which contains every field for each row in your dataset article_text = dataset_row.get("article") response = client.chat.completions.create( model=model, messages=[ { "role": "user", "content": prompt_template.format(article=article_text), }, ], ) return response.choices[0].message.content ``` ## Define our evaluators Next, we can run evaluations on the results. Evaluators take the output of a task (in this case, a string) and grade it, often with the help of an LLM. In this case, we will create ROUGE evaluators to compare the LLM-generated summaries with the human reference summaries you uploaded as part of your dataset. There are several variants of ROUGE, but we'll use ROUGE-1 F1 score for simplicity: * ROUGE-1 precision is the proportion of overlapping tokens (present in both reference and generated summaries) that are present in the generated summary (number of overlapping tokens / number of tokens in the generated summary) * ROUGE-1 recall is the proportion of overlapping tokens that are present in the reference summary (number of overlapping tokens / number of tokens in the reference summary) * ROUGE-1 F1 score is the harmonic mean of precision and recall, providing a single number that balances these two scores. Higher ROUGE scores mean that a generated summary is more similar to the corresponding reference summary. Scores near 1 / 2 are considered excellent, and a [model fine-tuned on this particular dataset achieved a rouge score of \~0.44](https://huggingface.co/datasets/abisee/cnn_dailymail#supported-tasks-and-leaderboards). Logging precision and recall alongside F1 costs nothing extra and shows you *why* a prompt scored the way it did: a terse prompt tends to raise precision and lower recall, a verbose one does the reverse. Each evaluator returns an `EvaluationResult`. Return the full object rather than a bare float: `label` and `explanation` are reserved columns that cannot be null, so a float-only return fails the upload. ```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} from arize.experimental.datasets.experiments.types import EvaluationResult from rouge import Rouge def _rouge_1(dataset_row, output) -> Dict[str, float]: # you can access the output of your task using the keyword output # you can also access any attribute of your dataset_row here expected = dataset_row.get("summary") return Rouge().get_scores(output, expected)[0]["rouge-1"] def _result(score: float, metric: str) -> EvaluationResult: return EvaluationResult( score=score, label=f"{score:.2f}", explanation=f"ROUGE-1 {metric} against the reference summary", ) def rouge_1_f1_score(dataset_row, output) -> EvaluationResult: return _result(_rouge_1(dataset_row, output)["f"], "F1") def rouge_1_precision(dataset_row, output) -> EvaluationResult: return _result(_rouge_1(dataset_row, output)["p"], "precision") def rouge_1_recall(dataset_row, output) -> EvaluationResult: return _result(_rouge_1(dataset_row, output)["r"], "recall") EVALUATORS = [rouge_1_f1_score, rouge_1_precision, rouge_1_recall] ``` Before spending a full experiment run, check the task and one evaluator on a single row: ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} from functools import partial BASE_TEMPLATE = """ Summarize the article in two to four sentences: ARTICLE ======= {article} SUMMARY ======= """ test_row = df.iloc[0].to_dict() test_output = summarize_article(test_row, prompt_template=BASE_TEMPLATE, model=MODEL) print("Generated:", test_output) print("Reference:", test_row["summary"]) print("ROUGE-1 F1:", rouge_1_f1_score(test_row, test_output).score) ``` ## Run the experiment Now you can run an experiment with this task, evaluator, and dataset. ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} experiment = arize_client.run_experiment( space_id=SPACE_ID, dataset_id=dataset_id, task=partial(summarize_article, prompt_template=BASE_TEMPLATE, model=MODEL), evaluators=EVALUATORS, experiment_name="initial-template", ) ``` ![](https://storage.googleapis.com/arize-phoenix-assets/assets/images/arize-docs-images/cookbooks/summarization.png) ## Compare prompt variants The point of holding the dataset and evaluators fixed is that any score change is attributable to the prompt. Run the same experiment again with a stricter instruction: ```python theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} CONCISE_TEMPLATE = """ Summarize the article in two to four sentences. Be concise and include only the most important information. ARTICLE ======= {article} SUMMARY ======= """ arize_client.run_experiment( space_id=SPACE_ID, dataset_id=dataset_id, task=partial(summarize_article, prompt_template=CONCISE_TEMPLATE, model=MODEL), evaluators=EVALUATORS, experiment_name="concise-template", ) ``` Then try few-shot. The examples are drawn from the dataset's *train* split, so they never overlap with the ten test rows you are scoring against. ```python expandable theme={"theme":{"light":"github-light-default","dark":"github-dark-default"}} EXAMPLE_TEMPLATE = """ ARTICLE ======= {article} SUMMARY ======= {summary} """ train_df = ( hf_ds["train"] .to_pandas() .sample(n=5, random_state=42) .rename(columns={"highlights": "summary"}) ) examples = "\n".join( EXAMPLE_TEMPLATE.format(article=row["article"], summary=row["summary"]) for _, row in train_df.iterrows() ) FEW_SHOT_TEMPLATE = ( """ Summarize the article in two to four sentences. Be concise and include only the most important information, as in the examples below. EXAMPLES ======== """ + examples + """ Now summarize the following article. ARTICLE ======= {article} SUMMARY ======= """ ) arize_client.run_experiment( space_id=SPACE_ID, dataset_id=dataset_id, task=partial(summarize_article, prompt_template=FEW_SHOT_TEMPLATE, model=MODEL), evaluators=EVALUATORS, experiment_name="few-shot-template", ) ``` Open the dataset's **Experiments** tab to compare the three runs. Each experiment is a row and each evaluator its own score column, so you can read F1 against precision and recall together and see which prompt traded which for which.