GitHub Copilot

Trace GitHub Copilot sessions in VS Code and the Copilot CLI in Arize AX for full observability.

GitHub Copilot is GitHub’s AI pair programmer for VS Code and the command line. There are two ways to get Copilot activity into Arize AX: the Arize Coding Harness Tracing, which installs hooks and exports OpenInference spans directly, or Copilot’s own OpenTelemetry export, which Arize AX normalizes on ingestion.

Choose an approach

	Arize Coding Harness Tracing	Native OpenTelemetry export
Setup scope	Once per repository, since the hooks file lives in each workspace	Once per user, device, or enterprise
Central rollout	Each developer runs the installer	Through Copilot managed settings (MDM, server-managed, or file-based)
Credentials	`~/.arize/harness/config.json`, or environment variables	Managed headers stay inside the extension’s exporter and never enter environment variables
Content capture	On by default, with three separate `ARIZE_LOG_*` redaction flags	One `captureContent` toggle, off by default, lockable by an admin
Span fidelity	Native OpenInference, no convention mapping in between	Normalized from `gen_ai.*` on ingestion, so hook spans arrive unclassified

For a team or an enterprise, prefer the native OpenTelemetry export. It scales without touching every repository, and an administrator can mandate it centrally so developers configure nothing. Use the harness when you need per-category redaction control, or when you want OpenInference spans with no convention mapping in between. Both paths need an Arize AX Space ID and API key.

Launch Arize AX

To get started, sign up for a free Arize AX account and get your Space ID and API Key:

Log in at app.arize.com
Click Settings and copy the Space ID
Open the API Keys tab and create or copy an API key

Install

This installs the Arize Coding Harness. Copilot hooks are installed at the project level under .github/hooks/, so run the installer from the root of each repository where you want tracing. To avoid a per-repository install entirely, use the native OpenTelemetry export instead, which needs no installer.

Curl installer

macOS / Linux:

cd /path/to/your/project
curl -sSL https://raw.githubusercontent.com/Arize-ai/coding-harness-tracing/main/install.sh | bash -s -- copilot

Windows (PowerShell):

cd C:\path\to\your\project
iwr -useb https://raw.githubusercontent.com/Arize-ai/coding-harness-tracing/main/install.bat -OutFile $env:TEMP\install.bat
& $env:TEMP\install.bat copilot

Local clone

git clone https://github.com/Arize-ai/coding-harness-tracing.git
cd /path/to/your/project
/path/to/coding-harness-tracing/install.sh copilot         # macOS / Linux
\path\to\coding-harness-tracing\install.bat copilot        # Windows

The installer prompts for your Arize AX credentials and project name, writes them to ~/.arize/harness/config.json, and registers VS Code (.github/hooks/*.json) and Copilot CLI (.github/hooks/hooks.json) hooks in the current project.

Configuration

Credentials live in ~/.arize/harness/config.json and apply across all projects. Per-project environment variables override config.json and can be set in your shell profile.

export ARIZE_API_KEY="<your-api-key>"
export ARIZE_SPACE_ID="<your-space-id>"
export ARIZE_PROJECT_NAME="copilot"
export ARIZE_TRACE_ENABLED="true"

Redaction controls

Each ARIZE_LOG_* flag accepts "true" or "false" and defaults to "true". Set to "false" to opt out per category:

export ARIZE_LOG_PROMPTS="false"
export ARIZE_LOG_TOOL_DETAILS="false"
export ARIZE_LOG_TOOL_CONTENT="false"

Flag	Redacts
`ARIZE_LOG_PROMPTS`	User prompt and assistant response text
`ARIZE_LOG_TOOL_DETAILS`	Tool names and arguments
`ARIZE_LOG_TOOL_CONTENT`	Tool call output content

Observe

Once tracing is enabled, Copilot activity from both VS Code and the CLI streams to Arize AX. You’ll see:

Turn traces — each prompt → response cycle as a parent span
Tool spans — nested spans for each tool call with inputs, outputs, and duration
Subagent spans — activity from any subagents Copilot spawns (VS Code)
Error spans — errorOccurred events from the CLI
Session grouping — all turns from the same session grouped by session_id

Hooks Captured

Copilot in VS Code and on the CLI use different event names. The Agent Kit handles both with one install command.

Surface	Hook File	Events
VS Code	`.github/hooks/session-start.json`	`SessionStart`
VS Code	`.github/hooks/user-prompt.json`	`UserPromptSubmit`
VS Code	`.github/hooks/pre-tool.json`	`PreToolUse`
VS Code	`.github/hooks/post-tool.json`	`PostToolUse`
VS Code	`.github/hooks/stop.json`	`Stop`
VS Code	`.github/hooks/subagent-stop.json`	`SubagentStop`
CLI	`.github/hooks/hooks.json`	`sessionStart`, `userPromptSubmitted`, `preToolUse`, `postToolUse`, `errorOccurred`, `sessionEnd`

Native OpenTelemetry export

Copilot Chat emits its own OpenTelemetry traces following the OTel GenAI semantic conventions, with no hooks and no installer. Arize AX normalizes gen_ai.* spans into OpenInference on ingestion, so you can point Copilot straight at the Arize AX OTLP endpoint. Enabling OTel instruments every agent type at once: the foreground agent, Copilot CLI sessions, Claude agent sessions, and any subagents, with trace context propagated across them.

Set up a single developer

In settings.json:

{
  "github.copilot.chat.otel.enabled": true,
  "github.copilot.chat.otel.otlpEndpoint": "https://otlp.arize.com",
  "github.copilot.chat.otel.captureContent": true
}

There is no user-level setting for OTLP headers, so credentials and project routing go through environment variables:

export OTEL_EXPORTER_OTLP_HEADERS="space_id=<your-space-id>,api_key=<your-api-key>"
export OTEL_RESOURCE_ATTRIBUTES="openinference.project.name=copilot"

Without openinference.project.name, spans have no project to land in. OTEL_SERVICE_NAME does not route them.

The exported URL must resolve to https://otlp.arize.com/v1/traces. VS Code appends /v1/traces to the endpoint you configure, the same way it does for the http://localhost:4318 collector default. If no spans arrive, set otlpEndpoint to the full path instead. Posting to the bare host returns an HTTP success and silently discards the spans.

Leave captureContent off and you get metadata only: model names, token counts, durations, and tool names, but no prompts, responses, or tool arguments.

Roll out across an enterprise

Administrators can mandate the whole configuration through the telemetry block in Copilot managed settings, delivered by MDM (Intune, Jamf, Group Policy), server-managed settings from your .github-private repository, or a file-based managed-settings.json. Developers configure nothing.

{
  "telemetry": {
    "enabled": true,
    "endpoint": "https://otlp.arize.com",
    "protocol": "http/protobuf",
    "captureContent": true,
    "lockCaptureContent": true,
    "resourceAttributes": {
      "openinference.project.name": "copilot"
    },
    "headers": {
      "space_id": "<your-space-id>",
      "api_key": "<your-api-key>"
    }
  }
}

A managed value always wins, taking precedence over environment variables and user settings. lockCaptureContent prevents developers from changing whether prompt and tool content is captured.

Managed telemetry.headers are applied only to the Copilot Chat extension’s exporter and are deliberately never passed through environment variables, so an API key cannot leak into the tool subprocesses the agent host spawns. As a consequence, managed headers do not currently reach the agent host, so Copilot CLI terminal sessions still need OTEL_EXPORTER_OTLP_HEADERS set in the environment.

Spans Captured

Copilot’s operation names normalize to OpenInference span kinds as follows:

Copilot span	`gen_ai.operation.name`	Arize AX span kind
`invoke_agent`	`invoke_agent`	`AGENT`
`chat`	`chat`	`LLM`
`execute_tool`	`execute_tool`	`TOOL`
`execute_hook`	`execute_hook`	Unclassified

On the agent and chat spans you get prompts and responses as llm.input_messages and llm.output_messages, token counts, model name, and computed cost. Tool spans carry tool.name, tool.parameters, and the tool result as output.value. gen_ai.conversation.id becomes session.id, so turns group into sessions. The raw gen_ai.*, github.copilot.*, and copilot_chat.* attributes are all preserved on the span, so you can still filter on things like github.copilot.git.branch.

What does not carry over

Hook spans are unclassified. execute_hook is not a recognized GenAI operation, so those spans arrive with no span kind. Their github.copilot.hook.* attributes are preserved and still queryable.
Metrics and events are dropped. Arize AX ingests traces. The copilot_chat.* metrics (edit acceptance, edit survival, time to first token) and the OTel events are not spans and are not stored.
The provider stays unmapped. gen_ai.provider.name is github, which is not in the provider table, so it passes through to llm.system unchanged and llm.provider is left unset.

Reference

For the full list of environment variables, default file paths, and troubleshooting steps, see the Copilot tracing README. For the native export, see Monitor agent usage with OpenTelemetry in the VS Code documentation.

Uninstall

Run from the project root where you installed the hooks:

curl -sSL https://raw.githubusercontent.com/Arize-ai/coding-harness-tracing/main/install.sh | bash -s -- uninstall copilot

To turn off the native export, set github.copilot.chat.otel.enabled to false, or set telemetry.enabled to false in managed settings.

OpenTelemetry

LLM Providers

Agent Frameworks

Coding Agents

Platforms

Orchestration

Evaluation

Choose an approach

Launch Arize AX

Install

Curl installer

Local clone

Configuration

Redaction controls

Observe

Hooks Captured

Native OpenTelemetry export

Set up a single developer

Roll out across an enterprise

Spans Captured

What does not carry over

Reference

Uninstall

Resources

Arize Coding Harness Tracing

OpenInference

GitHub Copilot

​Choose an approach

​Launch Arize AX

​Install

​Curl installer

​Local clone

​Configuration

​Redaction controls

​Observe

​Hooks Captured

​Native OpenTelemetry export

​Set up a single developer

​Roll out across an enterprise

​Spans Captured

​What does not carry over

​Reference

​Uninstall

​Resources

Arize Coding Harness Tracing

OpenInference

GitHub Copilot

Choose an approach

Launch Arize AX

Install

Curl installer

Local clone

Configuration

Redaction controls

Observe

Hooks Captured

Native OpenTelemetry export

Set up a single developer

Roll out across an enterprise

Spans Captured

What does not carry over

Reference

Uninstall

Resources