Otimiza prompts de LLM usando dados de traces de produção, avaliações e anotações por meio de um loop orientado a dados utilizando a CLI ax.
- Estrelas no GitHub
- 41.038
- Arquivos
- 3
- Último commit
- 11 de nov. de 2026
- Licença
- MIT
- Ferramentas permitidas
- —
O que a skill arize-prompt-optimization faz?
Otimiza prompts de LLM usando dados de traces de produção, avaliações e anotações por meio de um loop orientado a dados utilizando a CLI ax.
Resumo gerado a partir do SKILL.md da skill.
Sobre a skill arize-prompt-optimization
Melhorar a qualidade da saída de LLMs por meio de ciclos orientados a dados é o que esta SKILL permite para desenvolvedores e engenheiros de prompt. Ela utiliza a CLI ax para extrair prompts de dados de traces de produção, permitindo que os usuários identifiquem exatamente onde os modelos falham ao inspecionar mensagens de chat estruturadas, templates de prompt e substituições de variáveis. Ao acessar sinais de performance, como anotações humanas, scores numéricos de qualidade e explicações de LLM-as-judge, a ferramenta ajuda a identificar padrões específicos em erros de modelos.
Os usuários podem executar um loop de otimização completo exportando spans, datasets e resultados de experimentos para analisar a lacuna entre as respostas reais do modelo e o ground truth. A SKILL facilita a reconstrução de prompts e a preparação de registros de performance para uso com meta-prompts de otimização. Esse processo depende da CLI ax e de um perfil Arize configurado para gerenciar traces, projetos e spaces, garantindo que o ajuste de prompts seja baseado em feedback real de produção em vez de intuição.
O que um agente pode fazer com ela
- 1Extrair prompts de traces de produçãoRecupere as mensagens exatas de sistema e usuário usadas em spans específicos de LLM para entender as estruturas atuais de prompt.
- 2Analisar falhas de performance do modeloIdentifique spans onde o modelo produziu respostas incorretas ou recebeu scores de avaliação baixos de juízes automatizados.
- 3Preparar dados para prompt tuningMescle datasets e execuções de experimentos em um único formato JSON para fornecer contexto para a geração de prompts aprimorados.
- 4Depurar variáveis de templates de promptInspecione como valores específicos são substituídos em templates de prompt durante o runtime para garantir a injeção correta de contexto.
Escrito a partir do SKILL.md da skill. Cada caso de uso é verificado com uma citação dele.
Como instalar a skill arize-prompt-optimization
Cada comando copia a pasta da skill para o diretório de skills do cliente. Reinicie o cliente ou inicie uma nova sessão para carregá-la.Instalar a skill arize-prompt-optimization no Claude Code
~/.claude/skills/arize-prompt-optimization/SKILL.md
$ npx degit github/awesome-copilot/skills/arize-prompt-optimization ~/.claude/skills/arize-prompt-optimizationInstalar a skill arize-prompt-optimization no Claude Code (project)
.claude/skills/arize-prompt-optimization/SKILL.md
$ npx degit github/awesome-copilot/skills/arize-prompt-optimization .claude/skills/arize-prompt-optimizationInstalar a skill arize-prompt-optimization no Codex
~/.agents/skills/arize-prompt-optimization/SKILL.md
$ npx degit github/awesome-copilot/skills/arize-prompt-optimization ~/.agents/skills/arize-prompt-optimizationInstalar a skill arize-prompt-optimization no Gemini CLI
~/.gemini/skills/arize-prompt-optimization/SKILL.md
$ npx degit github/awesome-copilot/skills/arize-prompt-optimization ~/.gemini/skills/arize-prompt-optimizationInstalar a skill arize-prompt-optimization no Claude.ai
npx degit github/awesome-copilot/skills/arize-prompt-optimization arize-prompt-optimization && zip -r arize-prompt-optimization.zip arize-prompt-optimization # claude.ai → Customize → Skills → + → Create skill → Upload a skill → arize-prompt-optimization.zip
Verificações automatizadas
Executado em 15 de nov. de 2026Relatório completo →Passou nas nossas verificações automáticas. Não é uma auditoria de segurança: leia os scripts antes de deixar um agente executá-los. Como as verificações funcionam
- Repositório de origem vinculado e acessívelgithub/awesome-copilot/skills/arize-prompt-optimization is public and reachablegithub.com/github/awesome-copilot/tree/HEAD/skills/arize-pro ↗Passou
- Licença de código aberto (OSI)MITspdx.org/licenses/MIT.html ↗Passou
- Commit ou release no último anoLast commit 2026-11-11github.com/github/awesome-copilot/commits ↗Passou
- Sem baixar e executar nem roubo de credenciais nos scriptsNo scripts in the skill folderPassou
- Sem instruções ocultas no SKILL.mdNo override phrases, safeguard switches, credential requests or hidden charactersgithub.com/github/awesome-copilot/blob/HEAD/skills/arize-pro ↗Passou
- Ferramentas pré-aprovadas são restritasNo pre-approved tools; the agent asks before using onegithub.com/github/awesome-copilot/blob/HEAD/skills/arize-pro ↗Passou
- O SKILL.md segue o formato Agent SkillsValid name and description; says when to use it; 458 linesgithub.com/github/awesome-copilot/blob/HEAD/skills/arize-pro ↗Passou
SKILL.md
Ver no GitHubArize Prompt Optimization Skill
SPACE— All--spaceflags and theARIZE_SPACEenv var accept a space name (e.g.,my-workspace) or a base64 space ID (e.g.,U3BhY2U6...). Find yours withax spaces list.
Concepts
Where Prompts Live in Trace Data
LLM applications emit spans following OpenInference semantic conventions. Prompts are stored in different span attributes depending on the span kind and instrumentation:
| Column | What it contains | When to use |
|---|---|---|
attributes.llm.input_messages |
Structured chat messages (system, user, assistant, tool) in role-based format | Primary source for chat-based LLM prompts |
attributes.llm.input_messages.roles |
Array of roles: system, user, assistant, tool |
Extract individual message roles |
attributes.llm.input_messages.contents |
Array of message content strings | Extract message text |
attributes.input.value |
Serialized prompt or user question (generic, all span kinds) | Fallback when structured messages are not available |
attributes.llm.prompt_template.template |
Template with {variable} placeholders (e.g., "Answer {question} using {context}") |
When the app uses prompt templates |
attributes.llm.prompt_template.variables |
Template variable values (JSON object) | See what values were substituted into the template |
attributes.output.value |
Model response text | See what the LLM produced |
attributes.llm.output_messages |
Structured model output (including tool calls) | Inspect tool-calling responses |
Finding Prompts by Span Kind
- LLM span (
attributes.openinference.span.kind = 'LLM'): Checkattributes.llm.input_messagesfor structured chat messages, ORattributes.input.valuefor a serialized prompt. Checkattributes.llm.prompt_template.templatefor the template. - Chain/Agent span:
attributes.input.valuecontains the user's question. The actual LLM prompt lives on child LLM spans -- navigate down the trace tree. - Tool span:
attributes.input.valuehas tool input,attributes.output.valuehas tool result. Not typically where prompts live.
Performance Signal Columns
These columns carry the feedback data used for optimization:
| Column pattern | Source | What it tells you |
|---|---|---|
annotation.<name>.label |
Human reviewers | Categorical grade (e.g., correct, incorrect, partial) |
annotation.<name>.score |
Human reviewers | Numeric quality score (e.g., 0.0 - 1.0) |
annotation.<name>.text |
Human reviewers | Freeform explanation of the grade |
eval.<name>.label |
LLM-as-judge evals | Automated categorical assessment |
eval.<name>.score |
LLM-as-judge evals | Automated numeric score |
eval.<name>.explanation |
LLM-as-judge evals | Why the eval gave that score -- most valuable for optimization |
attributes.input.value |
Trace data | What went into the LLM |
attributes.output.value |
Trace data | What the LLM produced |
{experiment_name}.output |
Experiment runs | Output from a specific experiment |
Prerequisites
Proceed directly with the task — run the ax command you need. Do NOT check versions, env vars, or profiles upfront.
If an ax command fails, troubleshoot based on the error:
command not foundor version error → see references/ax-setup.md401 Unauthorized/ missing API key → runax profiles showto inspect the current profile. If the profile is missing or the API key is wrong, follow references/ax-profiles.md to create/update it. If the user doesn't have their key, direct them to https://app.arize.com/admin > API Keys- Space unknown → run
ax spaces listto pick by name, or ask the user - Project unclear → ask the user, or run
ax projects list -o json --limit 100and present as selectable options - LLM provider call fails (missing OPENAI_API_KEY / ANTHROPIC_API_KEY) → run
ax ai-integrations list --space SPACEto check for platform-managed credentials. If none exist, ask the user to provide the key or create an integration via the arize-ai-provider-integration skill - Security: Never read
.envfiles or search the filesystem for credentials. Useax profilesfor Arize credentials andax ai-integrationsfor LLM provider keys. If credentials are not available through these channels, ask the user.
Phase 1: Extract the Current Prompt
Find LLM spans containing prompts
# Sample LLM spans (where prompts live)
ax spans export PROJECT --filter "attributes.openinference.span.kind = 'LLM'" -l 10 --stdout
# Filter by model
ax spans export PROJECT --filter "attributes.llm.model_name = 'gpt-4o'" -l 10 --stdout
# Filter by span name (e.g., a specific LLM call)
ax spans export PROJECT --filter "name = 'ChatCompletion'" -l 10 --stdout
Export a trace to inspect prompt structure
# Export all spans in a trace
ax spans export PROJECT --trace-id TRACE_ID
# Export a single span
ax spans export PROJECT --span-id SPAN_ID
Extract prompts from exported JSON
# Extract structured chat messages (system + user + assistant)
jq '.[0] | {
messages: .attributes.llm.input_messages,
model: .attributes.llm.model_name
}' trace_*/spans.json
# Extract the system prompt specifically
jq '[.[] | select(.attributes.llm.input_messages.roles[]? == "system")] | .[0].attributes.llm.input_messages' trace_*/spans.json
# Extract prompt template and variables
jq '.[0].attributes.llm.prompt_template' trace_*/spans.json
# Extract from input.value (fallback for non-structured prompts)
jq '.[0].attributes.input.value' trace_*/spans.json
Reconstruct the prompt as messages
Once you have the span data, reconstruct the prompt as a messages array:
[
{"role": "system", "content": "You are a helpful assistant that..."},
{"role": "user", "content": "Given {input}, answer the question: {question}"}
]
If the span has attributes.llm.prompt_template.template, the prompt uses variables. Preserve these placeholders ({variable} or {{variable}}) -- they are substituted at runtime.
Phase 2: Gather Performance Data
From traces (production feedback)
# Find error spans -- these indicate prompt failures
ax spans export PROJECT \
--filter "status_code = 'ERROR' AND attributes.openinference.span.kind = 'LLM'" \
-l 20 --stdout
# Find spans with low eval scores
ax spans export PROJECT \
--filter "annotation.correctness.label = 'incorrect'" \
-l 20 --stdout
# Find spans with high latency (may indicate overly complex prompts)
ax spans export PROJECT \
--filter "attributes.openinference.span.kind = 'LLM' AND latency_ms > 10000" \
-l 20 --stdout
# Export error traces for detailed inspection
ax spans export PROJECT --trace-id TRACE_ID
From datasets and experiments
# Export a dataset (ground truth examples)
ax datasets export DATASET_NAME --space SPACE
# -> dataset_*/examples.json
# Export experiment results (what the LLM produced)
ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE
# -> experiment_*/runs.json
Merge dataset + experiment for analysis
Join the two files by example_id to see inputs alongside outputs and evaluations:
# Count examples and runs
jq 'length' dataset_*/examples.json
jq 'length' experiment_*/runs.json
# View a single joined record
jq -s '
.[0] as $dataset |
.[1][0] as $run |
($dataset[] | select(.id == $run.example_id)) as $example |
{
input: $example,
output: $run.output,
evaluations: $run.evaluations
}
' dataset_*/examples.json experiment_*/runs.json
# Find failed examples (where eval score < threshold)
jq '[.[] | select(.evaluations.correctness.score < 0.5)]' experiment_*/runs.json
Identify what to optimize
Look for patterns across failures:
- Compare outputs to ground truth: Where does the LLM output differ from expected?
- Read eval explanations:
eval.*.explanationtells you WHY something failed - Check annotation text: Human feedback describes specific issues
- Look for verbosity mismatches: If outputs are too long/short vs ground truth
- Check format compliance: Are outputs in the expected format?
Phase 3: Optimize the Prompt
The Optimization Meta-Prompt
Use this template to generate an improved version of the prompt. Fill in the three placeholders and send it to your LLM (GPT-4o, Claude, etc.):
You are an expert in prompt optimization. Given the original baseline prompt
and the associated performance data (inputs, outputs, evaluation labels, and
explanations), generate a revised version that improves results.
ORIGINAL BASELINE PROMPT
========================
{PASTE_ORIGINAL_PROMPT_HERE}
========================
PERFORMANCE DATA
================
The following records show how the current prompt performed. Each record
includes the input, the LLM output, and evaluation feedback:
{PASTE_RECORDS_HERE}
================
HOW TO USE THIS DATA
1. Compare outputs: Look at what the LLM generated vs what was expected
2. Review eval scores: Check which examples scored poorly and why
3. Examine annotations: Human feedback shows what worked and what didn't
4. Identify patterns: Look for common issues across multiple examples
5. Focus on failures: The rows where the output DIFFERS from the expected
value are the ones that need fixing
ALIGNMENT STRATEGY
- If outputs have extra text or reasoning not present in the ground truth,
remove instructions that encourage explanation or verbose reasoning
- If outputs are missing information, add instructions to include it
- If outputs are in the wrong format, add explicit format instructions
- Focus on the rows where the output differs from the target -- these are
the failures to fix
RULES
Maintain Structure:
- Use the same template variables as the current prompt ({var} or {{var}})
- Don't change sections that are already working
- Preserve the exact return format instructions from the original prompt
Avoid Overfitting:
- DO NOT copy examples verbatim into the prompt
- DO NOT quote specific test data outputs exactly
- INSTEAD: Extract the ESSENCE of what makes good vs bad outputs
- INSTEAD: Add general guidelines and principles
- INSTEAD: If adding few-shot examples, create SYNTHETIC examples that
demonstrate the principle, not real data from above
Goal: Create a prompt that generalizes well to new inputs, not one that
memorizes the test data.
OUTPUT FORMAT
Return the revised prompt as a JSON array of messages:
[
{"role": "system", "content": "..."},
{"role": "user", "content": "..."}
]
Also provide a brief reasoning section (bulleted list) explaining:
- What problems you found
- How the revised prompt addresses each one
Preparing the performance data
Format the records as a JSON array before pasting into the template:
# From dataset + experiment: join and select relevant columns
jq -s '
.[0] as $ds |
[.[1][] | . as $run |
($ds[] | select(.id == $run.example_id)) as $ex |
{
input: $ex.input,
expected: $ex.expected_output,
actual_output: $run.output,
eval_score: $run.evaluations.correctness.score,
eval_label: $run.evaluations.correctness.label,
eval_explanation: $run.evaluations.correctness.explanation
}
]
' dataset_*/examples.json experiment_*/runs.json
# From exported spans: extract input/output pairs with annotations
jq '[.[] | select(.attributes.openinference.span.kind == "LLM") | {
input: .attributes.input.value,
output: .attributes.output.value,
status: .status_code,
model: .attributes.llm.model_name
}]' trace_*/spans.json
Applying the revised prompt
After the LLM returns the revised messages array:
- Compare the original and revised prompts side by side
- Verify all template variables are preserved
- Check that format instructions are intact
- Test on a few examples before full deployment
Phase 4: Iterate
The optimization loop
1. Extract prompt -> Phase 1 (once)
2. Run experiment -> ax experiments create ...
3. Export results -> ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE
4. Analyze failures -> jq to find low scores
5. Run meta-prompt -> Phase 3 with new failure data
6. Apply revised prompt
7. Repeat from step 2
Measure improvement
# Compare scores across experiments
# Experiment A (baseline)
jq '[.[] | .evaluations.correctness.score] | add / length' experiment_a/runs.json
# Experiment B (optimized)
jq '[.[] | .evaluations.correctness.score] | add / length' experiment_b/runs.json
# Find examples that flipped from fail to pass
jq -s '
[.[0][] | select(.evaluations.correctness.label == "incorrect")] as $fails |
[.[1][] | select(.evaluations.correctness.label == "correct") |
select(.example_id as $id | $fails | any(.example_id == $id))
] | length
' experiment_a/runs.json experiment_b/runs.json
A/B compare two prompts
- Create two experiments against the same dataset, each using a different prompt version
- Export both:
ax experiments export EXP_Aandax experiments export EXP_B - Compare average scores, failure rates, and specific example flips
- Check for regressions -- examples that passed with prompt A but fail with prompt B
Prompt Engineering Best Practices
Apply these when writing or revising prompts:
| Technique | When to apply | Example |
|---|---|---|
| Clear, detailed instructions | Output is vague or off-topic | "Classify the sentiment as exactly one of: positive, negative, neutral" |
| Instructions at the beginning | Model ignores later instructions | Put the task description before examples |
| Step-by-step breakdowns | Complex multi-step processes | "First extract entities, then classify each, then summarize" |
| Specific personas | Need consistent style/tone | "You are a senior financial analyst writing for institutional investors" |
| Delimiter tokens | Sections blend together | Use ---, ###, or XML tags to separate input from instructions |
| Few-shot examples | Output format needs clarification | Show 2-3 synthetic input/output pairs |
| Output length specifications | Responses are too long or short | "Respond in exactly 2-3 sentences" |
| Reasoning instructions | Accuracy is critical | "Think step by step before answering" |
| "I don't know" guidelines | Hallucination is a risk | "If the answer is not in the provided context, say 'I don't have enough information'" |
Variable preservation
When optimizing prompts that use template variables:
- Single braces (
{variable}): Python f-string / Jinja style. Most common in Arize. - Double braces (
{{variable}}): Mustache style. Used when the framework requires it. - Never add or remove variable placeholders during optimization
- Never rename variables -- the runtime substitution depends on exact names
- If adding few-shot examples, use literal values, not variable placeholders
Workflows
Optimize a prompt from a failing trace
- Find failing traces:
ax traces list PROJECT --filter "status_code = 'ERROR'" --limit 5 - Export the trace:
ax spans export PROJECT --trace-id TRACE_ID - Extract the prompt from the LLM span:
jq '[.[] | select(.attributes.openinference.span.kind == "LLM")][0] | { messages: .attributes.llm.input_messages, template: .attributes.llm.prompt_template, output: .attributes.output.value, error: .attributes.exception.message }' trace_*/spans.json - Identify what failed from the error message or output
- Fill in the optimization meta-prompt (Phase 3) with the prompt and error context
- Apply the revised prompt
Optimize using a dataset and experiment
- Find the dataset and experiment:
ax datasets list --space SPACE ax experiments list --dataset DATASET_NAME --space SPACE - Export both:
ax datasets export DATASET_NAME --space SPACE ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE - Prepare the joined data for the meta-prompt
- Run the optimization meta-prompt
- Create a new experiment with the revised prompt to measure improvement
Debug a prompt that produces wrong format
- Export spans where the output format is wrong:
ax spans export PROJECT \ --filter "attributes.openinference.span.kind = 'LLM' AND annotation.format.label = 'incorrect'" \ -l 10 --stdout > bad_format.json - Look at what the LLM is producing vs what was expected
- Add explicit format instructions to the prompt (JSON schema, examples, delimiters)
- Common fix: add a few-shot example showing the exact desired output format
Reduce hallucination in a RAG prompt
- Find traces where the model hallucinated:
ax spans export PROJECT \ --filter "annotation.faithfulness.label = 'unfaithful'" \ -l 20 --stdout - Export and inspect the retriever + LLM spans together:
ax spans export PROJECT --trace-id TRACE_ID jq '[.[] | {kind: .attributes.openinference.span.kind, name, input: .attributes.input.value, output: .attributes.output.value}]' trace_*/spans.json - Check if the retrieved context actually contained the answer
- Add grounding instructions to the system prompt: "Only use information from the provided context. If the answer is not in the context, say so."
Troubleshooting
| Problem | Solution |
|---|---|
ax: command not found |
See references/ax-setup.md |
No profile found |
No profile is configured. See references/ax-profiles.md to create one. |
No input_messages on span |
Check span kind -- Chain/Agent spans store prompts on child LLM spans, not on themselves |
Prompt template is null |
Not all instrumentations emit prompt_template. Use input_messages or input.value instead |
| Variables lost after optimization | Verify the revised prompt preserves all {var} placeholders from the original |
| Optimization makes things worse | Check for overfitting -- the meta-prompt may have memorized test data. Ensure few-shot examples are synthetic |
| No eval/annotation columns | Run evaluations first (via Arize UI or SDK), then re-export |
| Experiment output column not found | The column name is {experiment_name}.output -- check exact experiment name via ax experiments get |
jq errors on span JSON |
Ensure you're targeting the correct file path (e.g., trace_*/spans.json) |
Arquivos desta skill
3Perguntas sobre a skill arize-prompt-optimization
Como instalo a skill arize-prompt-optimization no Claude Code?
Execute npx degit github/awesome-copilot/skills/arize-prompt-optimization ~/.claude/skills/arize-prompt-optimization. O Claude Code carrega skills de ~/.claude/skills (todos os projetos) ou .claude/skills (um projeto) e usa uma skill quando a descrição dela corresponde à tarefa.
Quais clientes podem usar a skill arize-prompt-optimization?
Qualquer cliente que leia o formato Agent Skills: Claude Code, Codex, Gemini CLI e claude.ai (envie como ZIP). Scripts da skill são executados apenas onde o cliente permite execução de código.
Como posso encontrar prompts dentro dos meus dados de trace?
Os prompts são armazenados em atributos de span; para LLMs baseados em chat, use 'attributes.llm.input_messages' ou 'attributes.input.value' como fallback.
Quais informações estão disponíveis nas colunas de sinais de performance?
Os sinais de performance incluem anotações humanas (labels, scores e texto) e avaliações de LLM-as-judge (labels, scores e explicações).
Como solucionar uma falha no comando 'ax'?
Solucione o problema com base no erro: use 'ax profiles show' para erros 401 Unauthorized, 'ax spaces list' para espaços desconhecidos e 'ax projects list' se o projeto não estiver claro.