Audits Python and BigQuery pipelines for cost efficiency, idempotency, and production readiness by identifying risks in query design and write operations.
- GitHub stars
- 41,038
- Files
- 1
- Last commit
- Nov 11, 2026
- License
- MIT
- Allowed tools
- —
What does the bigquery-pipeline-audit skill do?
bigquery-pipeline-audit audits Python and BigQuery pipelines for cost efficiency, idempotency, and production readiness by identifying risks in query design and write operations.
Summary generated from the skill's SKILL.md.
About the bigquery-pipeline-audit skill
Data engineers reviewing Python and BigQuery pipeline scripts can use this tool to identify financial risks and architectural flaws. The skill performs a deep audit of codebase logic to prevent runaway costs by checking for missing maximum bytes billed configurations or excessive BigQuery job counts within loops. It evaluates whether scripts are safe to rerun by inspecting write dispositions and idempotency patterns like MERGE operations or deduplication views.
When an agent loads this skill, it acts as a senior engineer to produce a structured report containing a pass or fail grade and a prioritized patch list. The audit examines specific technical implementations such as partition filtering, dry run modes, and observability logging. It relies on analyzing function names and line locations to suggest minimal fixes for issues like many-to-many join explosions, unversioned table reads, or silent exception handling. This ensures pipelines are production-ready and cost-efficient before deployment.
What an agent can do with it
- 1Ensuring data idempotencyCheck if a pipeline can be safely rerun without creating duplicate rows in a destination table.
- 2Validating production safety modesConfirm that the codebase includes a dry run mode to estimate costs without executing billed jobs.
Written from the skill's SKILL.md. Every use case is checked against a quote from it.
How to install the bigquery-pipeline-audit skill
Each command copies the skill folder into the client's skills directory. Restart the client or start a new session to load it.Install the bigquery-pipeline-audit skill in Claude Code
~/.claude/skills/bigquery-pipeline-audit/SKILL.md
$ npx degit github/awesome-copilot/skills/bigquery-pipeline-audit ~/.claude/skills/bigquery-pipeline-auditInstall the bigquery-pipeline-audit skill in Claude Code (project)
.claude/skills/bigquery-pipeline-audit/SKILL.md
$ npx degit github/awesome-copilot/skills/bigquery-pipeline-audit .claude/skills/bigquery-pipeline-auditInstall the bigquery-pipeline-audit skill in Codex
~/.agents/skills/bigquery-pipeline-audit/SKILL.md
$ npx degit github/awesome-copilot/skills/bigquery-pipeline-audit ~/.agents/skills/bigquery-pipeline-auditInstall the bigquery-pipeline-audit skill in Gemini CLI
~/.gemini/skills/bigquery-pipeline-audit/SKILL.md
$ npx degit github/awesome-copilot/skills/bigquery-pipeline-audit ~/.gemini/skills/bigquery-pipeline-auditInstall the bigquery-pipeline-audit skill in Claude.ai
npx degit github/awesome-copilot/skills/bigquery-pipeline-audit bigquery-pipeline-audit && zip -r bigquery-pipeline-audit.zip bigquery-pipeline-audit # claude.ai → Customize → Skills → + → Create skill → Upload a skill → bigquery-pipeline-audit.zip
Automated checks
Ran Nov 15, 2026Full report →Passed our automated checks. That is not a security audit: read the scripts before you let an agent run them. How checks work · What to fix for more points
- Source repository linked and reachablegithub/awesome-copilot/skills/bigquery-pipeline-audit is public and reachablegithub.com/github/awesome-copilot/tree/HEAD/skills/bigquery- ↗Pass
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit 2026-11-11github.com/github/awesome-copilot/commits ↗Pass
- No download-and-run or credential theft in scriptsNo scripts in the skill folderPass
- No hidden instructions in SKILL.mdNo override phrases, safeguard switches, credential requests or hidden charactersgithub.com/github/awesome-copilot/blob/HEAD/skills/bigquery- ↗Pass
- Pre-approved tools are scopedNo pre-approved tools; the agent asks before using onegithub.com/github/awesome-copilot/blob/HEAD/skills/bigquery- ↗Pass
- SKILL.md follows the Agent Skills formatthe description does not say when to use the skillgithub.com/github/awesome-copilot/blob/HEAD/skills/bigquery- ↗Finding
SKILL.md
View on GitHubBigQuery Pipeline Audit: Cost, Safety and Production Readiness
You are a senior data engineer reviewing a Python + BigQuery pipeline script. Your goals: catch runaway costs before they happen, ensure reruns do not corrupt data, and make sure failures are visible.
Analyze the codebase and respond in the structure below (A to F + Final). Reference exact function names and line locations. Suggest minimal fixes, not rewrites.
A) COST EXPOSURE: What will actually get billed?
Locate every BigQuery job trigger (client.query, load_table_from_*,
extract_table, copy_table, DDL/DML via query) and every external call
(APIs, LLM calls, storage writes).
For each, answer:
- Is this inside a loop, retry block, or async gather?
- What is the realistic worst-case call count?
- For each
client.query, isQueryJobConfig.maximum_bytes_billedset? For load, extract, and copy jobs, is the scope bounded and counted against MAX_JOBS? - Is the same SQL and params being executed more than once in a single run? Flag repeated identical queries and suggest query hashing plus temp table caching.
Flag immediately if:
- Any BQ query runs once per date or once per entity in a loop
- Worst-case BQ job count exceeds 20
maximum_bytes_billedis missing on anyclient.querycall
B) DRY RUN AND EXECUTION MODES
Verify a --mode flag exists with at least dry_run and execute options.
dry_runmust print the plan and estimated scope with zero billed BQ execution (BigQuery dry-run estimation via job config is allowed) and zero external API or LLM callsexecuterequires explicit confirmation for prod (--env=prod --confirm)- Prod must not be the default environment
If missing, propose a minimal argparse patch with safe defaults.
C) BACKFILL AND LOOP DESIGN
Hard fail if: the script runs one BQ query per date or per entity in a loop.
Check that date-range backfills use one of:
- A single set-based query with
GENERATE_DATE_ARRAY - A staging table loaded with all dates then one join query
- Explicit chunks with a hard
MAX_CHUNKScap
Also check:
- Is the date range bounded by default (suggest 14 days max without
--override)? - If the script crashes mid-run, is it safe to re-run without double-writing?
- For backdated simulations, verify data is read from time-consistent snapshots
(
FOR SYSTEM_TIME AS OF, partitioned as-of tables, or dated snapshot tables). Flag any read from a "latest" or unversioned table when running in backdated mode.
Suggest a concrete rewrite if the current approach is row-by-row.
D) QUERY SAFETY AND SCAN SIZE
For each query, check:
- Partition filter is on the raw column, not
DATE(ts),CAST(...), or any function that prevents pruning - No
SELECT *: only columns actually used downstream - Joins will not explode: verify join keys are unique or appropriately scoped and flag any potential many-to-many
- Expensive operations (
REGEXP,JSON_EXTRACT, UDFs) only run after partition filtering, not on full table scans
Provide a specific SQL fix for any query that fails these checks.
E) SAFE WRITES AND IDEMPOTENCY
Identify every write operation. Flag plain INSERT/append with no dedup logic.
Each write should use one of:
MERGEon a deterministic key (e.g.,entity_id + date + model_version)- Write to a staging table scoped to the run, then swap or merge into final
- Append-only with a dedupe view:
QUALIFY ROW_NUMBER() OVER (PARTITION BY <key>) = 1
Also check:
- Will a re-run create duplicate rows?
- Is the write disposition (
WRITE_TRUNCATEvsWRITE_APPEND) intentional and documented? - Is
run_idbeing used as part of the merge or dedupe key? If so, flag it.run_idshould be stored as a metadata column, not as part of the uniqueness key, unless you explicitly want multi-run history.
State the recommended approach and the exact dedup key for this codebase.
F) OBSERVABILITY: Can you debug a failure?
Verify:
- Failures raise exceptions and abort with no silent
except: passor warn-only - Each BQ job logs: job ID, bytes processed or billed when available, slot milliseconds, and duration
- A run summary is logged or written at the end containing:
run_id, env, mode, date_range, tables written, total BQ jobs, total bytes run_idis present and consistent across all log lines
If run_id is missing, propose a one-line fix:
run_id = run_id or datetime.utcnow().strftime('%Y%m%dT%H%M%S')
Final
1. PASS / FAIL with specific reasons per section (A to F). 2. Patch list ordered by risk, referencing exact functions to change. 3. If FAIL: Top 3 cost risks with a rough worst-case estimate (e.g., "loop over 90 dates x 3 retries = 270 BQ jobs").
Files in this skill
1bigquery-pipeline-audit skill questions
How do I install the bigquery-pipeline-audit skill in Claude Code?
Run npx degit github/awesome-copilot/skills/bigquery-pipeline-audit ~/.claude/skills/bigquery-pipeline-audit. Claude Code loads skills from ~/.claude/skills (all projects) or .claude/skills (one project) and uses a skill when its description matches the task.
Which clients can use the bigquery-pipeline-audit skill?
Any client that reads the Agent Skills format: Claude Code, Codex, Gemini CLI and claude.ai (upload as a ZIP). Scripts in the skill run only where the client allows code execution.
What specific cost risks does the audit look for?
The audit flags BigQuery queries running in loops, missing maximum bytes billed configurations, and excessive job counts.
What are the requirements for a safe dry run mode?
A dry run must print the execution plan and estimated scope without incurring any BigQuery billing or external API/LLM costs.