Arize Experiment Skill
Concepts
Experiment
= a named evaluation run against a specific dataset version, containing one run per example
Experiment Run
= the result of processing one dataset example -- includes the model output, optional evaluations, and optional metadata
Dataset
= a versioned collection of examples; every experiment is tied to a dataset and a specific dataset version
Evaluation
= a named metric attached to a run (e.g.,
correctness
,
relevance
), with optional label, score, and explanation
The typical flow: export a dataset → process each example → collect outputs and evaluations → create an experiment with the runs.
Prerequisites
Three things are needed:
ax
CLI, an API key (env var or profile), and a space ID. A project name is also needed but usually comes from the user's message.
Install ax
Verify
ax
is installed and working before proceeding:
Check if
ax
is on PATH:
command -v ax
(Unix) or
where ax
(Windows)
If not found, check common install locations:
macOS/Linux:
test -x ~/.local/bin/ax && export PATH="$HOME/.local/bin:$PATH"
Windows: check
%APPDATA%\Python\Scripts\ax.exe
or
%LOCALAPPDATA%\Programs\Python\Scripts\ax.exe
If still not found, install it (requires shell access to install packages):
Preferred:
uv tool install arize-ax-cli
Alternative:
pipx install arize-ax-cli
Fallback:
pip install arize-ax-cli
After install, if
ax
is not on PATH:
macOS/Linux:
export PATH="$HOME/.local/bin:$PATH"
Windows (PowerShell):
$env:PATH = "$env:APPDATA\Python\Scripts;$env:PATH"
If
ax --version
fails with an SSL/certificate error:
macOS:
export SSL_CERT_FILE=/etc/ssl/cert.pem
Linux:
export SSL_CERT_FILE=/etc/ssl/certs/ca-certificates.crt
Windows (PowerShell):
$env:SSL_CERT_FILE = "C:\Program Files\Common Files\SSL\cert.pem"
(or use
python -c "import certifi; print(certifi.where())"
to find the cert bundle)
ax --version
must succeed before proceeding. If it doesn't, stop and ask the user for help.
Verify environment
Run a quick check for credentials:
macOS/Linux (bash):
ax
--version
&&
echo
"--- env ---"
&&
if
[
-n
"
$ARIZE_API_KEY
"
]
;
then
echo
"ARIZE_API_KEY: (set)"
;
else
echo
"ARIZE_API_KEY: (not set)"
;
fi
&&
echo
"ARIZE_SPACE_ID:
${ARIZE_SPACE_ID
:-
(not set)}
"
&&
echo
"--- profiles ---"
&&
ax profiles show
2
>
&1
Windows (PowerShell):
ax
--
version
;
Write-Host
"--- env ---"
;
Write-Host
"ARIZE_API_KEY:
$
(
if
(
$env
:ARIZE_API_KEY
)
{ '(set)' } else { '(not set)' })"
;
Write-Host
"ARIZE_SPACE_ID:
$env
:ARIZE_SPACE_ID"
;
Write-Host
"--- profiles ---"
;
ax profiles show 2>&1
Read the output and proceed immediately
if either the env var or the profile has an API key. Only ask the user if
both
are missing. Resolve failures:
No API key in env
and
no profile →
AskQuestion: "Arize API key ( https://app.arize.com/admin

API Keys)" Space ID unknown → AskQuestion , or run ax projects list -o json --limit 100 and search for a match Project unclear → ask, or run ax projects list -o json --limit 100 and present as selectable options Space ID and Project Both are needed for most commands. Resolve each: User provides it in the conversation -- use directly via --space-id / --project flags. Env var is set ( ARIZE_SPACE_ID , ARIZE_DEFAULT_PROJECT ) -- use silently. If missing, AskQuestion once. Tell the user: Space ID is in the Arize URL: /spaces/{SPACE_ID}/... Project is the project name as shown in the Arize UI. For convenience, recommend setting env vars so they don't get asked again: export ARIZE_SPACE_ID="U3BhY2U6..." and export ARIZE_DEFAULT_PROJECT="my-project" Prefer asking the user over searching or iterating through projects and API keys. If you get a 401 Unauthorized , tell the user their API key may not have access to that space and ask them to verify. List Experiments: ax experiments list Browse experiments, optionally filtered by dataset. Output goes to stdout. ax experiments list ax experiments list --dataset-id DATASET_ID --limit 20 ax experiments list --cursor CURSOR_TOKEN ax experiments list -o json Flags Flag Type Default Description --dataset-id string none Filter by dataset --limit, -l int 15 Max results (1-100) --cursor string none Pagination cursor from previous response -o, --output string table Output format: table, json, csv, parquet, or file path -p, --profile string default Configuration profile Get Experiment: ax experiments get Quick metadata lookup -- returns experiment name, linked dataset/version, and timestamps. ax experiments get EXPERIMENT_ID ax experiments get EXPERIMENT_ID -o json Flags Flag Type Default Description EXPERIMENT_ID string required Positional argument -o, --output string table Output format -p, --profile string default Configuration profile Response fields Field Type Description id string Experiment ID name string Experiment name dataset_id string Linked dataset ID dataset_version_id string Specific dataset version used experiment_traces_project_id string Project where experiment traces are stored created_at datetime When the experiment was created updated_at datetime Last modification time Export Experiment: ax experiments export Download all runs to a file. By default uses the REST API; pass --all to use Arrow Flight for bulk transfer. ax experiments export EXPERIMENT_ID

-> experiment_abc123_20260305_141500/runs.json

ax experiments export EXPERIMENT_ID --all ax experiments export EXPERIMENT_ID --output-dir ./results ax experiments export EXPERIMENT_ID --stdout ax experiments export EXPERIMENT_ID --stdout | jq '.[0]' Flags Flag Type Default Description EXPERIMENT_ID string required Positional argument --all bool false Use Arrow Flight for bulk export (see below) --output-dir string . Output directory --stdout bool false Print JSON to stdout instead of file -p, --profile string default Configuration profile REST vs Flight ( --all ) REST (default): Lower friction -- no Arrow/Flight dependency, standard HTTPS ports, works through any corporate proxy or firewall. Limited to 500 runs per page. Flight ( --all ): Required for experiments with more than 500 runs. Uses gRPC+TLS on a separate host/port ( flight.arize.com:443 ) which some corporate networks may block. Agent auto-escalation rule: If a REST export returns exactly 500 runs, the result is likely truncated. Re-run with --all to get the full dataset. Output is a JSON array of run objects: [ { "id" : "run_001" , "example_id" : "ex_001" , "output" : "The answer is 4." , "evaluations" : { "correctness" : { "label" : "correct" , "score" : 1.0 } , "relevance" : { "score" : 0.95 , "explanation" : "Directly answers the question" } } , "metadata" : { "model" : "gpt-4o" , "latency_ms" : 1234 } } ] Create Experiment: ax experiments create Create a new experiment with runs from a data file. ax experiments create --name "gpt-4o-baseline" --dataset-id DATASET_ID --file runs.json ax experiments create --name "claude-test" --dataset-id DATASET_ID --file runs.csv Flags Flag Type Required Description --name, -n string yes (prompted) Experiment name --dataset-id string yes (prompted) Dataset to run the experiment against --file, -f path yes (prompted) Data file with runs: CSV, JSON, JSONL, or Parquet -o, --output string no Output format -p, --profile string no Configuration profile Required columns in the runs file Column Type Required Description example_id string yes ID of the dataset example this run corresponds to output string yes The model/system output for this example Additional columns are passed through as additionalProperties on the run. Delete Experiment: ax experiments delete ax experiments delete EXPERIMENT_ID ax experiments delete EXPERIMENT_ID --force

skip confirmation prompt

Flags Flag Type Default Description EXPERIMENT_ID string required Positional argument --force, -f bool false Skip confirmation prompt -p, --profile string default Configuration profile Experiment Run Schema Each run corresponds to one dataset example: { "example_id" : "required -- links to dataset example" , "output" : "required -- the model/system output for this example" , "evaluations" : { "metric_name" : { "label" : "optional string label (e.g., 'correct', 'incorrect')" , "score" : "optional numeric score (e.g., 0.95)" , "explanation" : "optional freeform text" } } , "metadata" : { "model" : "gpt-4o" , "temperature" : 0.7 , "latency_ms" : 1234 } } Evaluation fields Field Type Required Description label string no Categorical classification (e.g., correct , incorrect , partial ) score number no Numeric quality score (e.g., 0.0 - 1.0) explanation string no Freeform reasoning for the evaluation At least one of label , score , or explanation should be present per evaluation. Workflows Run an experiment against a dataset Find or create a dataset: ax datasets list ax datasets export DATASET_ID --stdout | jq 'length' Export the dataset examples: ax datasets export DATASET_ID Process each example through your system, collecting outputs and evaluations Build a runs file (JSON array) with example_id , output , and optional evaluations : [ { "example_id" : "ex_001" , "output" : "4" , "evaluations" : { "correctness" : { "label" : "correct" , "score" : 1.0 } } } , { "example_id" : "ex_002" , "output" : "Paris" , "evaluations" : { "correctness" : { "label" : "correct" , "score" : 1.0 } } } ] Create the experiment: ax experiments create --name "gpt-4o-baseline" --dataset-id DATASET_ID --file runs.json Verify: ax experiments get EXPERIMENT_ID Compare two experiments Export both experiments: ax experiments export EXPERIMENT_ID_A --stdout

a.json ax experiments export EXPERIMENT_ID_B --stdout

b.json Compare evaluation scores by example_id :

Average correctness score for experiment A

jq '[.[] | .evaluations.correctness.score] | add / length' a.json

Same for experiment B

jq '[.[] | .evaluations.correctness.score] | add / length' b.json Find examples where results differ: jq -s '.[0] as $a | .[1][] | {example_id, b_score: .evaluations.correctness.score, a_score: ($a[] | select(.example_id == .example_id) | .evaluations.correctness.score)}' a.json b.json Download experiment results for analysis ax experiments list --dataset-id DATASET_ID -- find experiments ax experiments export EXPERIMENT_ID -- download to file Parse: jq '.[] | {example_id, score: .evaluations.correctness.score}' experiment_*/runs.json Pipe export to other tools

Count runs

ax experiments export EXPERIMENT_ID --stdout | jq 'length'

Extract all outputs

ax experiments export EXPERIMENT_ID --stdout | jq '.[].output'

Get runs with low scores

ax experiments export EXPERIMENT_ID --stdout | jq '[.[] | select(.evaluations.correctness.score < 0.5)]'

Convert to CSV

ax experiments export EXPERIMENT_ID --stdout | jq -r '.[] | [.example_id, .output, .evaluations.correctness.score] | @csv' Troubleshooting Problem Solution ax: command not found Check ~/.local/bin/ax ; if missing: uv tool install arize-ax-cli (requires shell access to install packages) 401 Unauthorized API key may not have access to this space. Verify the key and space ID are correct. Keys are scoped per space -- get the right one from https://app.arize.com/admin

API Keys. No profile found Run ax profiles show --expand to check; set ARIZE_API_KEY env var or write ~/.arize/config.toml Experiment not found Verify experiment ID with ax experiments list Invalid runs file Each run must have example_id and output fields example_id mismatch Ensure example_id values match IDs from the dataset (export dataset to verify) No runs found Export returned empty -- verify experiment has runs via ax experiments get Dataset not found The linked dataset may have been deleted; check with ax datasets list Save Credentials for Future Use At the end of the session , if the user manually provided any of the following during this conversation (via AskQuestion response, pasted text, or inline values) and those values were NOT already loaded from a saved profile or environment variable, offer to save them for future use. Credential Where it gets saved API key ax profile at ~/.arize/config.toml Space ID macOS/Linux: shell config ( ~/.zshrc or ~/.bashrc ) as export ARIZE_SPACE_ID="..." . Windows: user environment variable via

Skip this entirely if: The API key was already loaded from an existing profile or ARIZE_API_KEY env var The space ID was already set via ARIZE_SPACE_ID env var The user only used base64 project IDs (no space ID was needed) How to offer: Use AskQuestion : "Would you like to save your Arize credentials so you don't have to enter them next time?" with options "Yes, save them" / "No thanks" . If the user says yes: API key — Check if ~/.arize/config.toml exists. If it does, read it and update the [auth] section. If not, create it with this minimal content: [ profile ] name = "default" [ auth ] api_key = "THE_API_KEY" [ output ] format = "table" Verify with: ax profiles show Space ID — Persist the space ID as an environment variable: macOS/Linux — Detect the user's shell config file ( ~/.zshrc for zsh, ~/.bashrc for bash). Append: export ARIZE_SPACE_ID = "THE_SPACE_ID" Tell the user to run source ~/.zshrc (or restart their terminal) for it to take effect. Windows (PowerShell) — Set a persistent user environment variable: System.Environment ::SetEnvironmentVariable ( 'ARIZE_SPACE_ID' , 'THE_SPACE_ID' , 'User' ) Tell the user to restart their terminal for it to take effect.

arize-experiment

安装