data-engineering-skills プラグイン

Write production-grade Apache Airflow DAGs using the TaskFlow API — idempotent tasks, correct scheduling and catchup, retries/SLAs, connections/variables, and avoiding top-level code. Use when creating or reviewing Airflow DAGs, scheduling pipelines, wiring task dependencies, configuring retries/backfills, or fixing non-idempotent tasks.

Build Dagster pipelines using software-defined assets — asset dependencies, partitions, resources and IO managers, asset checks, and schedules/sensors. Use when creating Dagster assets or jobs, modeling data as assets, adding partitions or backfills, wiring resources/IO managers, or migrating from task-based orchestration to assets.

Build well-structured dbt models — staging/intermediate/marts layers, ref() and source(), materializations, and incremental models with the right strategy. Use when creating or refactoring dbt models, choosing table vs view vs incremental, structuring a dbt project, or writing incremental logic.

Build ML feature pipelines and feature stores — point-in-time-correct joins to avoid label leakage, offline/online parity, feature freshness and backfills, and materialization with tools like Feast. Use when engineering features for ML, preventing train/serve skew or data leakage, building a feature store, or backfilling historical features for training.

Design and operate Apache Iceberg tables — partitioning and hidden partitioning, partition/schema evolution, snapshots and time travel, compaction and small-file cleanup, and MERGE/upsert for lakehouse tables on Spark, Flink, Trino, or Snowflake. Use when creating or maintaining Iceberg tables, choosing partitioning, evolving schema/partitions, or fixing small-file and metadata bloat.

Build batch and incremental data ingestion (extract-load) pipelines — full vs incremental extraction, change data capture (CDC), watermarks and high-water marks, API pagination and rate limits, and choosing managed EL tools (Fivetran, Airbyte) vs custom code. Use when ingesting data from databases, APIs, files, or SaaS into a warehouse/lake, or designing incremental extraction and CDC.

Build reliable Apache Kafka consumers and producers — consumer groups and partition assignment, offset commit strategy, at-least-once vs exactly-once, idempotent/transactional producers, rebalancing, and dead-letter handling. Use when writing Kafka consumers/producers, configuring offset commits or consumer groups, tuning throughput, or handling rebalances and poison messages.

Diagnose failing or stuck Apache Airflow pipelines — task failures and retries, scheduler/executor problems, XCom errors, zombie/queued tasks, dependency deadlocks, and pools/concurrency limits. Use when an Airflow task fails or is stuck queued, the scheduler is not running tasks, a DAG will not trigger, or tasks become zombies.

Systematically root-cause data pipeline failures and data incidents — job errors, wrong or missing data, duplicates, and freshness misses — by tracing lineage upstream, isolating the failing stage, reconciling against source, and planning a safe fix and backfill. Use when a pipeline fails, numbers look wrong, data is missing or duplicated, a dashboard is stale, or a stakeholder reports a data discrepancy.

Diagnose and fix failing dbt runs — Jinja/compilation errors, ref/dependency and DAG issues, incremental models producing wrong or duplicate rows, full-refresh needs, and state/deferral problems. Use when dbt run or dbt build fails, a model compiles wrong, an incremental model is stale or duplicated, or a CI dbt job errors.

Plan and run safe data backfills and replays — idempotent reprocessing of historical windows, partition-by-partition execution, isolating backfill compute from production, verifying results, and avoiding double-counting or changed history. Use when backfilling a new or fixed model, reprocessing after a bug, replaying events, or loading history for a new pipeline without corrupting existing data.

Define and enforce data contracts between producers and consumers — explicit schema, semantics, ownership, SLAs, and versioning — to prevent silent upstream changes from breaking downstream pipelines. Use when a producer schema change could break consumers, defining an interface between teams/services and the warehouse, or adding schema enforcement at ingestion.

Structure a lakehouse with the medallion architecture — bronze (raw), silver (cleaned/conformed), and gold (business/aggregated) layers — with clear responsibilities, idempotent layer transitions, and where to put quality checks and modeling. Use when organizing a data lakehouse, defining bronze/silver/gold layers, deciding what logic belongs in each layer, or refactoring a flat pipeline into layers.

Document a dbt project — model and column descriptions in schema.yml, source definitions, exposures, doc blocks, and generated docs/lineage. Use when adding descriptions to dbt models, documenting sources or dashboards as exposures, setting up dbt docs, or improving data catalog coverage.

Build reliable Databricks pipelines on the lakehouse — Delta Lake tables, MERGE and time travel, Delta Live Tables / Lakeflow declarative pipelines, Unity Catalog governance, Photon, Auto Loader ingestion, and cluster/job sizing. Use when building Databricks jobs or DLT pipelines, working with Delta tables, ingesting with Auto Loader, or organizing Unity Catalog.

Generate realistic synthetic data for testing data pipelines — deterministic seeded fixtures, referential integrity across tables, edge cases (nulls, duplicates, late/out-of-order events), volume for load tests, and privacy-safe stand-ins for production. Use when creating test data for pipeline/dbt tests, seeding dev environments, load testing, or replacing PII with safe synthetic data.

Evolve data schemas safely over time — backward/forward compatibility, additive vs breaking changes, column adds/renames/type changes, and evolution in Avro, Parquet, Iceberg, Delta, and warehouse tables. Use when changing a table or event schema, adding or renaming columns, changing types, or preventing a schema change from breaking readers or pipelines.

Capture database changes with Debezium change data capture — connector setup for Postgres/MySQL/SQL Server, snapshot vs streaming phases, handling inserts/updates/deletes and tombstones, schema changes, and applying the change stream idempotently to a warehouse/lake. Use when setting up CDC, replicating an OLTP database, capturing deletes, or consuming a Debezium change stream.

Set up CI/CD for data pipelines — SQL/dbt linting (SQLFluff), compilation and test gates, dbt Slim CI with state:modified and deferral, environment promotion (dev/staging/prod), and running only changed models on pull requests. Use when adding CI checks to a dbt or SQL project, automating pipeline tests on PRs, speeding up CI, or promoting changes across environments.

Add data quality checks to pipelines — freshness, volume/row-count anomalies, schema drift, null/uniqueness/referential integrity, and value distributions — using dbt tests, Great Expectations, or Soda, and deciding warn vs block. Use when adding data quality validation, catching bad data before it reaches consumers, setting up freshness/volume checks, or defining expectations.

Instrument data pipelines with observability — freshness and volume SLAs, run success/latency metrics, data quality monitors, anomaly detection, lineage, and actionable alerting that avoids fatigue. Use when pipelines fail silently, incidents are found by stakeholders instead of alerts, setting up monitoring/SLAs for data, or reducing noisy alerts.

Capture and use data lineage with OpenLineage and Marquez — emitting run/job/dataset events from Airflow, dbt, and Spark, column-level lineage, and using lineage for impact analysis, debugging, and backfill scoping. Use when setting up data lineage, integrating OpenLineage, tracing what a change breaks downstream, or scoping the blast radius of an incident.

Protect personally identifiable information in data pipelines — classifying PII, choosing masking vs tokenization vs hashing vs encryption, dynamic data masking and column-level access control, and handling deletion/right-to-be-forgotten. Use when handling sensitive data, masking or anonymizing PII, meeting GDPR/CCPA/HIPAA requirements, or restricting column access in a warehouse.

Plan and execute migrations of legacy ETL and data warehouses — stored procedures, SSIS/Informatica, or on-prem warehouses to modern stacks (dbt, Spark, cloud warehouses) — using strangler-fig phasing, parallel runs, and row/aggregate reconciliation. Use when migrating legacy pipelines or warehouses, rewriting stored procedures into dbt/Spark, or validating a migration against the source system.

Design analytics data models using dimensional modeling — star and snowflake schemas, fact and dimension tables, grain declaration, surrogate keys, and slowly changing dimensions (SCD Type 1/2/3). Use when designing a warehouse schema, building marts, choosing a table grain, tracking history, or deciding fact vs dimension.

Reduce Google BigQuery cost and runtime — partitioning and clustering, minimizing bytes processed, avoiding SELECT * and full scans, slot usage and reservations, approximate functions, and materialized views. Use when BigQuery queries are expensive or slow, bytes billed are high, a query scans full tables, or you need to size slots.

Optimize columnar Parquet storage for analytics — file and row-group sizing, compression codecs (Snappy/ZSTD), partitioning and file layout, column pruning and predicate pushdown, dictionary encoding, and fixing the small-files problem. Use when Parquet reads are slow or costly, files are too small/large, choosing compression or partitioning, or improving scan pruning on a data lake.

Optimize slow or failing PySpark and Spark SQL jobs — partitioning and repartitioning, data skew, shuffles, broadcast joins, caching, Adaptive Query Execution, and avoiding driver collects and Python UDFs. Use when a Spark job is slow, spills, OOMs, has skewed tasks, runs a huge shuffle, or a stage hangs on a few straggler tasks.

Reduce Snowflake cost and latency — right-size and auto-suspend warehouses, use multi-cluster for concurrency, apply clustering keys, read the Query Profile, exploit result/warehouse caching, and control credit spend. Use when Snowflake queries are slow or expensive, warehouses spill or queue, credits are high, or you need to size a warehouse.

Optimize slow analytical SQL by reading query/EXPLAIN plans, cutting scanned data, fixing join strategy, and using partitioning, clustering, and indexes across Postgres, Snowflake, BigQuery, Databricks/Spark SQL, and Redshift. Use when a query is slow, times out, costs too much, scans too many rows/bytes, or spills to disk.

Build Prefect workflows — flows and tasks, retries and caching, parameters, concurrency limits, deployments and schedules, blocks for connections/secrets, and idempotent task design. Use when writing or debugging Prefect flows, scheduling runs, configuring retries/caching, or migrating scripts to Prefect orchestration.

Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.

Review data engineering pull requests with a data-specific checklist — idempotency, correct grain, incremental logic, cost impact, data quality tests, PII handling, and backward compatibility — that generic code review misses. Use when reviewing a dbt/SQL/Spark/Airflow PR, a data pipeline change, or a new model, and you want to catch data-correctness and cost issues before merge.

Provision data infrastructure with Terraform — warehouses, buckets, IAM/roles, orchestration, and streaming resources — using modules, remote state, workspaces/environments, and safe plan/apply workflows. Use when writing Terraform for Snowflake/BigQuery/Redshift, S3/GCS, IAM, Airflow/MWAA, or Kafka, or structuring reusable data-infra modules across environments.

Add data quality tests to a dbt project — generic tests (unique, not_null, accepted_values, relationships), singular tests, unit tests, dbt-utils and dbt-expectations packages, and source freshness. Use when adding tests to dbt models, catching data quality regressions, validating assumptions, or setting up source freshness checks.

Write data transformations and loads that produce the same result no matter how many times they run — using MERGE/upsert, deterministic partition overwrites, deduplication, and stable keys. Use when a retry could duplicate data, a job is not safe to re-run, a pipeline needs exactly-once effects, or loads must be backfill-safe.