Arize에서 LLM-as-judge 평가 워크플로우를 관리하며, 평가자 생성, span 또는 experiment 대상 작업 실행, 지속적인 모니터링을 지원합니다.
- GitHub 스타
- 41,038
- 파일
- 3
- 마지막 커밋
- 2026년 11월 11일
- 라이선스
- MIT
- 허용된 도구
- —
arize-evaluator skill은 무엇을 하나요?
arize-evaluator은(는) agent skill입니다. Arize에서 LLM-as-judge 평가 워크플로우를 관리하며, 평가자 생성, span 또는 experiment 대상 작업 실행, 지속적인 모니터링을 지원합니다.
skill의 SKILL.md에서 생성된 요약입니다.
arize-evaluator skill 정보
이 SKILL은 AI 애플리케이션을 구축하는 개발자가 Arize에서 LLM-as-judge 평가 워크플로우를 관리할 수 있도록 지원합니다. 사용자는 gpt-4o 또는 claude-sonnet-4-5와 같은 모델을 사용하여 환각(hallucination), 충실도(faithfulness), 관련성(relevance) 등의 품질을 점수화하는 evaluator를 설계, 생성 및 실행할 수 있습니다. 특정 프롬프트 템플릿과 분류 옵션으로 evaluator를 정의하면 모델 출력 평가를 자동화할 수 있습니다.
Agent는 이 SKILL을 사용하여 라이브 프로젝트 trace 또는 특정 experiment 데이터셋에 대해 task를 실행할 수 있습니다. span, trace, session 레벨을 포함한 다양한 데이터 단위(granularity)를 지원하므로, 개별 Q&A의 정확성부터 멀티턴 대화 품질 분석까지 모두 가능합니다. 이 SKILL은 ax CLI를 기반으로 하며, 구성된 Arize profile과 LLM 제공업체 자격 증명을 위한 AI integrations가 필요합니다. 라벨, 수치 점수, 선택적 설명을 포함한 구조화된 평가 결과가 생성되며, 이는 프로덕션 모델의 지속적인 모니터링에 활용될 수 있습니다.
agent가 수행할 수 있는 작업
- 1judge 프롬프트 반복 개선LLM judge의 품질을 향상시키기 위해 새로운 evaluator 프롬프트를 개발하고 버전을 관리합니다.
skill의 SKILL.md를 기반으로 작성되었습니다. 모든 사용 사례는 해당 문서의 인용구와 대조 확인되었습니다.
arize-evaluator skill 설치 방법
각 명령은 skill 폴더를 클라이언트의 skills 디렉터리에 복사합니다. 클라이언트를 재시작하거나 새 세션을 시작하여 로드하세요.Claude Code에 arize-evaluator skill 설치
~/.claude/skills/arize-evaluator/SKILL.md
$ npx degit github/awesome-copilot/skills/arize-evaluator ~/.claude/skills/arize-evaluatorClaude Code (project)에 arize-evaluator skill 설치
.claude/skills/arize-evaluator/SKILL.md
$ npx degit github/awesome-copilot/skills/arize-evaluator .claude/skills/arize-evaluatorCodex에 arize-evaluator skill 설치
~/.agents/skills/arize-evaluator/SKILL.md
$ npx degit github/awesome-copilot/skills/arize-evaluator ~/.agents/skills/arize-evaluatorGemini CLI에 arize-evaluator skill 설치
~/.gemini/skills/arize-evaluator/SKILL.md
$ npx degit github/awesome-copilot/skills/arize-evaluator ~/.gemini/skills/arize-evaluatorClaude.ai에 arize-evaluator skill 설치
npx degit github/awesome-copilot/skills/arize-evaluator arize-evaluator && zip -r arize-evaluator.zip arize-evaluator # claude.ai → Customize → Skills → + → Create skill → Upload a skill → arize-evaluator.zip
자동 검사
2026년 11월 15일에 실행됨전체 보고서 →자동 검사를 통과했습니다. 보안 감사는 아닙니다. 에이전트가 실행하기 전에 스크립트를 읽어 보세요. 검사 방식 · 점수를 높이기 위해 수정할 항목
- 소스 저장소 연결 및 접근 가능github/awesome-copilot/skills/arize-evaluator is public and reachablegithub.com/github/awesome-copilot/tree/HEAD/skills/arize-eva ↗통과
- 오픈 소스 라이선스 (OSI)MITspdx.org/licenses/MIT.html ↗통과
- 최근 1년 내 커밋 또는 릴리스Last commit 2026-11-11github.com/github/awesome-copilot/commits ↗통과
- 스크립트에 다운로드 후 실행이나 자격 증명 탈취가 없음No scripts in the skill folder통과
- SKILL.md에 숨겨진 지침 없음No override phrases, safeguard switches, credential requests or hidden charactersgithub.com/github/awesome-copilot/blob/HEAD/skills/arize-eva ↗통과
- 사전 허용 도구가 제한됨No pre-approved tools; the agent asks before using onegithub.com/github/awesome-copilot/blob/HEAD/skills/arize-eva ↗통과
- SKILL.md가 Agent Skills 형식을 따름SKILL.md is 674 lines; the spec suggests under 500github.com/github/awesome-copilot/blob/HEAD/skills/arize-eva ↗결함 발견
SKILL.md
GitHub에서 보기Arize Evaluator Skill
SPACE— All--spaceflags and theARIZE_SPACEenv var accept a space name (e.g.,my-workspace) or a base64 space ID (e.g.,U3BhY2U6...). Find yours withax spaces list.
This skill covers designing, creating, and running LLM-as-judge evaluators on Arize. An evaluator defines the judge; a task is how you run it against real data.
Prerequisites
Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront.
If an ax command fails, troubleshoot based on the error:
command not foundor version error → see references/ax-setup.md401 Unauthorized/ missing API key → runax profiles showto inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys- Space unknown → run
ax spaces listto pick by name, or ask the user - LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run
ax ai-integrations list --space SPACEto check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the arize-ai-provider-integration skill - Security: Never read
.envfiles or search the filesystem for credentials. Useax profilesfor Arize credentials andax ai-integrationsfor LLM provider keys. If credentials are not available through these channels, ask the user. - CRITICAL — Never fabricate evaluation results: If an evaluation task fails, is cancelled, or produces no scores, report the failure clearly and explain what went wrong. Do NOT perform a "manual evaluation," invent quality scores, estimate percentages, or present any agent-generated analysis as if it came from the Arize evaluation system. Instead suggest: (1) fix the identified issue and retry, (2) try running from the Arize UI, (3) verify integration credentials with
ax ai-integrations list, (4) contact support at https://arize.com/support
Concepts
What is an Evaluator?
An evaluator is an LLM-as-judge definition. It contains:
| Field | Description |
|---|---|
| Template | The judge prompt. Uses {variable} placeholders (e.g. {input}, {output}, {context}) that get filled in at run time via a task's column mappings. |
| Classification choices | The set of allowed output labels (e.g. factual / hallucinated). Binary is the default and most common. Each choice can optionally carry a numeric score. |
| AI Integration | Stored LLM provider credentials (OpenAI, Anthropic, Bedrock, etc.) the evaluator uses to call the judge model. |
| Model | The specific judge model (e.g. gpt-4o, claude-sonnet-4-5). |
| Invocation params | Optional JSON of model settings like {"temperature": 0}. Low temperature is recommended for reproducibility. |
| Optimization direction | Whether higher scores are better (maximize) or worse (minimize). Sets how the UI renders trends. |
| Data granularity | Whether the evaluator runs at the span, trace, or session level. Most evaluators run at the span level. |
Evaluators are versioned — every prompt or model change creates a new immutable version. The most recent version is active.
What is a Task?
A task is how you run one or more evaluators against real data. Tasks are attached to a project (live traces/spans) or a dataset (experiment runs). A task contains:
| Field | Description |
|---|---|
| Evaluators | List of evaluators to run. You can run multiple in one task. |
| Column mappings | Maps each evaluator's template variables to actual field paths on spans or experiment runs (e.g. "input" → "attributes.input.value"). This is what makes evaluators portable across projects and experiments. |
| Query filter | SQL-style expression to select which spans/runs to evaluate (e.g. "span_kind = 'LLM'"). Optional but important for precision. |
| Continuous | For project tasks: whether to automatically score new spans as they arrive. |
| Sampling rate | For continuous project tasks: fraction of new spans to evaluate (0–1). |
Data Granularity
The --data-granularity flag controls what unit of data the evaluator scores. It defaults to span and only applies to project tasks (not dataset/experiment tasks — those evaluate experiment runs directly).
| Level | What it evaluates | Use for | Result column prefix |
|---|---|---|---|
span (default) |
Individual spans | Q&A correctness, hallucination, relevance | eval.{name}.label / .score / .explanation |
trace |
All spans in a trace, grouped by context.trace_id |
Agent trajectory, task correctness — anything that needs the full call chain | trace_eval.{name}.label / .score / .explanation |
session |
All traces in a session, grouped by attributes.session.id and ordered by start time |
Multi-turn coherence, overall tone, conversation quality | session_eval.{name}.label / .score / .explanation |
How trace and session aggregation works
For trace granularity, spans sharing the same context.trace_id are grouped together. Column values used by the evaluator template are comma-joined into a single string (each value truncated to 100K characters) before being passed to the judge model.
For session granularity, the same trace-level grouping happens first, then traces are ordered by start_time and grouped by attributes.session.id. Session-level values are capped at 100K characters total.
The {conversation} template variable
At session granularity, {conversation} is a special template variable that renders as a JSON array of {input, output} turns across all traces in the session, built from attributes.input.value / attributes.llm.input_messages (input side) and attributes.output.value / attributes.llm.output_messages (output side).
At span or trace granularity, {conversation} is treated as a regular template variable and resolved via column mappings like any other.
Multi-evaluator tasks
A task can contain evaluators at different granularities. At runtime the system uses the highest granularity (session > trace > span) for data fetching and automatically splits into one child run per evaluator. Per-evaluator query_filter in the task's evaluators JSON further narrows which spans are included (e.g., only tool-call spans within a session).
Basic CRUD
AI Integrations
AI integrations store the LLM provider credentials the evaluator uses. For full CRUD — listing, creating for all providers (OpenAI, Anthropic, Azure, Bedrock, Vertex, Gemini, NVIDIA NIM, custom), updating, and deleting — use the arize-ai-provider-integration skill.
Quick reference for the common case (OpenAI):
# Check for an existing integration first
ax ai-integrations list --space SPACE
# Create if none exists
ax ai-integrations create \
--name "My OpenAI Integration" \
--provider openAI \
--api-key $OPENAI_API_KEY
Copy the returned integration ID — it is required for ax evaluators create --ai-integration-id.
Evaluators
# List / Get
ax evaluators list --space SPACE
ax evaluators get ID # accepts name or ID
ax evaluators get NAME --space SPACE # required when using name instead of ID
ax evaluators list-versions NAME_OR_ID
ax evaluators get-version VERSION_ID
# Create (creates the evaluator and its first version)
ax evaluators create \
--name "Answer Correctness" \
--space SPACE \
--description "Judges if the model answer is correct" \
--template-name "correctness" \
--commit-message "Initial version" \
--ai-integration-id INT_ID \
--model-name "gpt-4o" \
--include-explanations \
--use-function-calling \
--classification-choices '{"correct": 1, "incorrect": 0}' \
--template 'You are an evaluator. Given the user question and the model response, decide if the response correctly answers the question.
User question: {input}
Model response: {output}
Respond with exactly one of these labels: correct, incorrect'
# Create a new version (for prompt or model changes — versions are immutable)
ax evaluators create-version NAME_OR_ID \
--commit-message "Added context grounding" \
--template-name "correctness" \
--ai-integration-id INT_ID \
--model-name "gpt-4o" \
--include-explanations \
--classification-choices '{"correct": 1, "incorrect": 0}' \
--template 'Updated prompt...
{input} / {output} / {context}'
# Update metadata only (name, description — not prompt)
ax evaluators update NAME_OR_ID \
--name "New Name" \
--description "Updated description"
# Delete (permanent — removes all versions)
ax evaluators delete NAME_OR_ID
Key flags for create:
| Flag | Required | Description |
|---|---|---|
--name |
yes | Evaluator name (unique within space) |
--space |
yes | Space name or ID to create in |
--template-name |
yes | Eval column name — alphanumeric, spaces, hyphens, underscores |
--commit-message |
yes | Description of this version |
--ai-integration-id |
yes | AI integration ID (from above) |
--model-name |
yes | Judge model (e.g. gpt-4o) |
--template |
yes | Prompt with {variable} placeholders (single-quoted in bash) |
--classification-choices |
yes | JSON object mapping choice labels to numeric scores e.g. '{"correct": 1, "incorrect": 0}' |
--description |
no | Human-readable description |
--include-explanations |
no | Include reasoning alongside the label |
--use-function-calling |
no | Prefer structured function-call output |
--invocation-params |
no | JSON of model params e.g. '{"temperature": 0}' |
--data-granularity |
no | span (default), trace, or session. Only relevant for project tasks, not dataset/experiment tasks. See Data Granularity section. |
--direction |
no | Optimization direction: maximize or minimize. Sets how the UI renders trends. |
--provider-params |
no | JSON object of provider-specific parameters |
Tasks
PROJECT_NAME,DATASET_NAME, andevaluator_idall accept a name or base64 ID.
# List / Get
ax tasks list --space SPACE
ax tasks list --project PROJECT_NAME
ax tasks list --dataset DATASET_NAME --space SPACE
ax tasks get TASK_ID
# Create (project — continuous)
ax tasks create \
--name "Correctness Monitor" \
--task-type template_evaluation \
--project PROJECT_NAME \
--evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"input": "attributes.input.value", "output": "attributes.output.value"}}]' \
--is-continuous \
--sampling-rate 0.1
# Create (project — one-time / backfill)
ax tasks create \
--name "Correctness Backfill" \
--task-type template_evaluation \
--project PROJECT_NAME \
--evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"input": "attributes.input.value", "output": "attributes.output.value"}}]' \
--no-continuous
# Create (experiment / dataset)
ax tasks create \
--name "Experiment Scoring" \
--task-type template_evaluation \
--dataset DATASET_NAME --space SPACE \
--experiment-ids "EXP_ID_1,EXP_ID_2" \ # base64 IDs from `ax experiments list --space SPACE -o json`
--evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"output": "output"}}]' \
--no-continuous
# Trigger a run (project task — use data window)
ax tasks trigger-run TASK_ID \
--data-start-time "2026-03-20T00:00:00" \
--data-end-time "2026-03-21T23:59:59" \
--wait
# Trigger a run (experiment task — use experiment IDs)
ax tasks trigger-run TASK_ID \
--experiment-ids "EXP_ID_1" \ # base64 ID from `ax experiments list --space SPACE -o json`
--wait
# Monitor
ax tasks list-runs TASK_ID
ax tasks get-run RUN_ID
ax tasks wait-for-run RUN_ID --timeout 300
ax tasks cancel-run RUN_ID --force
Time format for trigger-run: 2026-03-21T09:00:00 — no trailing Z.
Additional trigger-run flags:
| Flag | Description |
|---|---|
--max-spans |
Cap processed spans (default 10,000) |
--override-evaluations |
Re-score spans that already have labels |
--wait / -w |
Block until the run finishes |
--timeout |
Seconds to wait with --wait (default 600) |
--poll-interval |
Poll interval in seconds when waiting (default 5) |
Run status guide:
| Status | Meaning |
|---|---|
completed, 0 spans |
The eval index lags 1–2 hours — spans ingested recently may not be indexed yet. Shift the window to data at least 2 hours old, or widen the time range to cover more historical data. |
cancelled ~1s |
Integration credentials invalid |
cancelled ~3min |
Found spans but LLM call failed — check model name or key |
completed, N > 0 |
Success — check scores in UI |
Workflow A: Create an evaluator for a project
Use this when the user says something like "create an evaluator for my Playground Traces project".
Step 1: Confirm the project name
ax spans export accepts a project name directly — no ID lookup needed. If you don't know the project name, list available projects:
ax projects list --space SPACE -o json
Find the entry whose "name" matches (case-insensitive) and use that name as PROJECT in subsequent commands. If you later hit a validation error with a name, fall back to using the project's "id" (a base64 string) instead.
Step 2: Understand what to evaluate
If the user specified the evaluator type (hallucination, correctness, relevance, etc.) → skip to Step 3.
If not, sample recent spans to base the evaluator on actual data:
ax spans export PROJECT --space SPACE -l 10 --days 30 --stdout
Inspect attributes.input, attributes.output, span kinds, and any existing annotations. Identify failure modes (e.g. hallucinated facts, off-topic answers, missing context) and propose 1–3 concrete evaluator ideas. Let the user pick.
Each suggestion must include: the evaluator name (bold), a one-sentence description of what it judges, and the binary label pair in parentheses. Format each like:
- Name — Description of what is being judged. (
label_a/label_b)
Example:
- Response Correctness — Does the agent's response correctly address the user's financial query? (
correct/incorrect) - Hallucination — Does the response fabricate facts not grounded in retrieved context? (
factual/hallucinated)
Step 3: Confirm or create an AI integration
ax ai-integrations list --space SPACE -o json
If a suitable integration exists, note its ID. If not, create one using the arize-ai-provider-integration skill. Ask the user which provider/model they want for the judge.
Step 4: Create the evaluator
Use the template design best practices below. Keep the evaluator name and variables generic — the task (Step 6) handles project-specific wiring via column_mappings.
ax evaluators create \
--name "Hallucination" \
--space SPACE \
--template-name "hallucination" \
--commit-message "Initial version" \
--ai-integration-id INT_ID \
--model-name "gpt-4o" \
--include-explanations \
--use-function-calling \
--classification-choices '{"factual": 1, "hallucinated": 0}' \
--template 'You are an evaluator. Given the user question and the model response, decide if the response is factual or contains unsupported claims.
User question: {input}
Model response: {output}
Respond with exactly one of these labels: hallucinated, factual'
Step 5: Ask — backfill, continuous, or both?
Recommended approach: Always start with a small backfill (~100 historical spans) to validate the evaluator before turning on continuous monitoring. This lets you catch column mapping errors, wrong span kinds, and template issues on known data before scoring all future production spans. Only enable continuous after a backfill confirms correct scoring.
Before creating the task, ask:
"Would you like to: (a) Run a backfill on historical spans (one-time)? (b) Set up continuous evaluation on new spans going forward? (c) Both — backfill first to validate, then keep scoring new spans automatically? (recommended)"
Step 6: Determine column mappings from real span data
Do not guess paths. Pull a sample and inspect what fields are actually present:
ax spans export PROJECT --space SPACE -l 5 --days 7 --stdout
For each template variable ({input}, {output}, {context}), find the matching JSON path. Common starting points — always verify on your actual data before using:
| Template var | LLM span | CHAIN span |
|---|---|---|
input |
attributes.input.value |
attributes.input.value |
output |
attributes.llm.output_messages.0.message.content |
attributes.output.value |
context |
attributes.retrieval.documents.contents |
— |
tool_output |
attributes.input.value (fallback) |
attributes.output.value |
Validate span kind alignment: If the evaluator prompt assumes LLM final text but the task targets CHAIN spans (or vice versa), runs can cancel or score the wrong text. Make sure the query_filter on the task matches the span kind you mapped.
query_filter only works on indexed attributes: The query_filter in the evaluators JSON is evaluated against the eval index, not the raw span store. Attributes under attributes.metadata.* or custom keys may not be indexed and will silently match nothing. Use well-known indexed attributes like span_kind or attributes.llm.model_name for filtering. If a filter returns 0 spans despite data existing, try removing the filter as a diagnostic step.
Full example --evaluators JSON:
[
{
"evaluator_id": "EVAL_ID",
"query_filter": "span_kind = 'LLM'",
"column_mappings": {
"input": "attributes.input.value",
"output": "attributes.llm.output_messages.0.message.content",
"context": "attributes.retrieval.documents.contents"
}
}
]
Include a mapping for every variable the template references. Omitting one causes runs to produce no valid scores.
Step 7: Create the task
Backfill only (a):
ax tasks create \
--name "Hallucination Backfill" \
--task-type template_evaluation \
--project PROJECT \
--evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"input": "attributes.input.value", "output": "attributes.output.value"}}]' \
--no-continuous
Continuous only (b):
ax tasks create \
--name "Hallucination Monitor" \
--task-type template_evaluation \
--project PROJECT \
--evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"input": "attributes.input.value", "output": "attributes.output.value"}}]' \
--is-continuous \
--sampling-rate 0.1
Both (c): Use --is-continuous on create, then also trigger a backfill run in Step 8.
Step 8: Trigger a backfill run (if requested)
Eval index lag: The eval index is built asynchronously from the primary trace store and can lag 1–2 hours. For your first test run, use a time window ending at least 2 hours in the past. If you set
--data-end-timeto "now" on spans ingested in the last hour, the run will complete successfully but score 0 spans.
First find what time range has data:
ax spans export PROJECT --space SPACE -l 100 --days 1 --stdout # try last 24h first
ax spans export PROJECT --space SPACE -l 100 --days 7 --stdout # widen if empty
Use the start_time / end_time fields from real spans to set the window. For the first validation run, cap --max-spans at ~100 to get quick feedback:
ax tasks trigger-run TASK_ID \
--data-start-time "2026-03-20T00:00:00" \
--data-end-time "2026-03-21T23:59:59" \
--max-spans 100 \
--wait
Review scores and explanations before widening to the full backfill or enabling continuous.
Workflow B: Create an evaluator for an experiment
Use this when the user says something like "create an evaluator for my experiment" or "evaluate my dataset runs".
If the user says "dataset" but doesn't have an experiment: A task must target an experiment (not a bare dataset). Ask:
"Evaluation tasks run against experiment runs, not datasets directly. Would you like help creating an experiment on that dataset first?"
If yes, use the arize-experiment skill to create one, then return here.
Step 1: Find the dataset and experiment names
ax datasets list --space SPACE
ax experiments list --dataset DATASET_NAME --space SPACE -o json
Note the dataset name and the experiment name(s) to score. These accept names or IDs in subsequent commands — names are preferred.
Step 2: Understand what to evaluate
If the user specified the evaluator type → skip to Step 3.
If not, inspect a recent experiment run to base the evaluator on actual data:
ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE --stdout | python3 -c "import sys,json; runs=json.load(sys.stdin); print(json.dumps(runs[0], indent=2))"
Look at the output, input, evaluations, and metadata fields. Identify gaps (metrics the user cares about but doesn't have yet) and propose 1–3 evaluator ideas. Each suggestion must include: the evaluator name (bold), a one-sentence description, and the binary label pair in parentheses — same format as Workflow A, Step 2.
Step 3: Confirm or create an AI integration
Same as Workflow A, Step 3.
Step 4: Create the evaluator
Same as Workflow A, Step 4. Keep variables generic.
Step 5: Determine column mappings from real run data
Run data shape differs from span data. Inspect:
ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE --stdout | python3 -c "import sys,json; runs=json.load(sys.stdin); print(json.dumps(runs[0], indent=2))"
Common mapping for experiment runs:
output→"output"(top-level field on each run)input→ check if it's on the run or embedded in the linked dataset examples
If input is not on the run JSON, export dataset examples to find the path:
ax datasets export DATASET_NAME --space SPACE --stdout | python3 -c "import sys,json; ex=json.load(sys.stdin); print(json.dumps(ex[0], indent=2))"
Step 6: Create the task
ax tasks create \
--name "Experiment Correctness" \
--task-type template_evaluation \
--dataset DATASET_NAME --space SPACE \
--experiment-ids "EXP_ID" \ # base64 ID from `ax experiments list --space SPACE -o json`
--evaluators '[{"evaluator_id": "EVAL_ID", "column_mappings": {"output": "output"}}]' \
--no-continuous
Step 7: Trigger and monitor
ax tasks trigger-run TASK_ID \
--experiment-ids "EXP_ID" \ # base64 ID from `ax experiments list --space SPACE -o json`
--wait
ax tasks list-runs TASK_ID
ax tasks get-run RUN_ID
Best Practices for Template Design
1. Use generic, portable variable names
Use {input}, {output}, and {context} — not names tied to a specific project or span attribute (e.g. do not use {attributes_input_value}). The evaluator itself stays abstract; the task's column_mappings is where you wire it to the actual fields in a specific project or experiment. This lets the same evaluator run across multiple projects and experiments without modification.
이 스킬의 파일
3arize-evaluator skill 질문
Claude Code에 arize-evaluator skill을 어떻게 설치하나요?
npx degit github/awesome-copilot/skills/arize-evaluator ~/.claude/skills/arize-evaluator를 실행합니다. Claude Code는 ~/.claude/skills(모든 프로젝트) 또는 .claude/skills(한 프로젝트)에서 skills를 로드하며, 설명이 작업과 일치할 때 skill을 사용합니다.
arize-evaluator skill을 사용할 수 있는 클라이언트는 무엇인가요?
Agent Skills 형식을 읽는 모든 클라이언트(Claude Code, Codex, Gemini CLI, claude.ai(ZIP으로 업로드))가 사용할 수 있습니다. skill 내 스크립트는 클라이언트가 코드 실행을 허용하는 경우에만 실행됩니다.
Arize에서 evaluator와 task의 차이점은 무엇인가요?
evaluator는 프롬프트 템플릿과 모델 설정을 포함하는 LLM-as-judge 정의이며, task는 프로젝트나 데이터셋의 실제 데이터에 대해 하나 이상의 evaluator를 실행하는 작업입니다.
평가 시 어떤 데이터 단위(granularity)를 지원하나요?
개별 단위, 전체 호출 체인, 또는 멀티턴 대화 중 무엇을 점수화하느냐에 따라 span, trace, 또는 session 레벨에서 평가를 실행할 수 있습니다.
LLM 제공업체의 인증은 어떻게 처리하나요?
환경 파일을 직접 읽는 대신 ax ai-integrations 명령을 사용하여 OpenAI 또는 Anthropic과 같은 제공업체의 자격 증명을 관리해야 합니다.