Skill processing-streaming-data
Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.
- Estrellas en GitHub
- 27
- Archivos
- 1
- Último commit
- 30 ago 2026
- Licencia
- —
- Herramientas permitidas
- —
¿Qué hace la skill processing-streaming-data?
processing-streaming-data es una agent skill. Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.
Cómo instalar la skill processing-streaming-data
Cada comando copia la carpeta de la skill al directorio de skills del cliente. Reinicia el cliente o inicia una nueva sesión para cargarla.Instalar la skill processing-streaming-data en Plugin
/plugin marketplace add Unknown-333/awesome-data-engineering-skills /plugin install data-engineering-skills@awesome-data-engineering-skills
Instalar la skill processing-streaming-data en Claude Code
~/.claude/skills/processing-streaming-data/SKILL.md
$ npx degit Unknown-333/awesome-data-engineering-skills/skills/processing-streaming-data ~/.claude/skills/processing-streaming-dataInstalar la skill processing-streaming-data en Claude Code (project)
.claude/skills/processing-streaming-data/SKILL.md
$ npx degit Unknown-333/awesome-data-engineering-skills/skills/processing-streaming-data .claude/skills/processing-streaming-dataInstalar la skill processing-streaming-data en Codex
~/.agents/skills/processing-streaming-data/SKILL.md
$ npx degit Unknown-333/awesome-data-engineering-skills/skills/processing-streaming-data ~/.agents/skills/processing-streaming-dataInstalar la skill processing-streaming-data en Gemini CLI
~/.gemini/skills/processing-streaming-data/SKILL.md
$ npx degit Unknown-333/awesome-data-engineering-skills/skills/processing-streaming-data ~/.gemini/skills/processing-streaming-dataInstalar la skill processing-streaming-data en Claude.ai
npx degit Unknown-333/awesome-data-engineering-skills/skills/processing-streaming-data processing-streaming-data && zip -r processing-streaming-data.zip processing-streaming-data # claude.ai → Customize → Skills → + → Create skill → Upload a skill → processing-streaming-data.zip
SKILL.md
Ver en GitHubProcessing Streaming Data
When to use
- Building or debugging Kafka / Spark Structured Streaming / Flink pipelines.
- Choosing delivery semantics and making consumers idempotent.
- Windowing on event time and handling late/out-of-order events.
- Do NOT use for batch ELT (use
building-ingestion-pipelines).
Workflow
- [ ] Choose delivery semantics; make effects idempotent regardless
- [ ] Use event time (not processing time) with watermarks
- [ ] Configure checkpointing for recovery
- [ ] Decide late-data policy (allowed lateness -> update, drop, or side output)
- [ ] Size partitions/parallelism to the throughput
- Delivery semantics. At-least-once is the common default (duplicates possible on retry). Exactly-once needs transactional sinks/offsets. Either way, make the downstream effect idempotent (upsert by key) so duplicates don't corrupt state — this is more robust than relying on exactly-once alone.
- Event time + watermarks. Window by when the event happened, not when it was processed; a watermark bounds how long to wait for stragglers.
- Checkpoint so a failed job resumes from the last committed offset/state instead of reprocessing everything or losing data.
- Late data policy — allowed lateness updates windows; beyond it, drop or route to a side output/dead-letter for reconciliation.
Patterns
Idempotent consumer — key the sink write on a stable event id so replays upsert rather than duplicate:
# Structured Streaming: exactly-once-ish via idempotent upsert in foreachBatch
def upsert(batch_df, batch_id):
(delta_table.alias("t")
.merge(batch_df.dropDuplicates(["event_id"]).alias("s"), "t.event_id = s.event_id")
.whenNotMatchedInsertAll().execute())
(stream.writeStream.foreachBatch(upsert)
.option("checkpointLocation", "/chk/events").start())
Event-time windowing with watermark:
(events
.withWatermark("event_time", "10 minutes")
.groupBy(window("event_time", "5 minutes"), "user_id")
.count())
Kafka consumer basics — consumer group for scale-out; commit offsets after processing (not before) to avoid data loss; partition count caps parallelism.
Common pitfalls
- Relying on exactly-once instead of idempotent sinks — any at-least-once hop reintroduces duplicates; upsert by key.
- Processing-time windows — produce wrong results when events arrive late or the job lags; use event time + watermark.
- Committing offsets before processing — a crash then loses those messages.
- No checkpointing — recovery either reprocesses everything or loses state.
- Unbounded state (no watermark/TTL) — keyed state grows until the job OOMs.
- Too few partitions — caps consumer parallelism; you can't scale past partition count.
- Ignoring dead-letter/late paths — bad or very late events silently vanish.
Archivos de esta skill
1Preguntas sobre la skill processing-streaming-data
¿Cómo instalo la skill processing-streaming-data en Claude Code?
Ejecuta npx degit Unknown-333/awesome-data-engineering-skills/skills/processing-streaming-data ~/.claude/skills/processing-streaming-data. Claude Code carga las skills desde ~/.claude/skills (todos los proyectos) o .claude/skills (un proyecto) y usa una skill cuando su descripción coincide con la tarea.
¿Qué clientes pueden usar la skill processing-streaming-data?
Cualquier cliente que lea el formato Agent Skills: Claude Code, Codex, Gemini CLI y claude.ai (subir como ZIP). Los scripts de la skill se ejecutan solo donde el cliente permite la ejecución de código.