processing-streaming-data skill
Build reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.
- GitHub stars
- 27
- Files
- 1
- Last commit
- Aug 30, 2026
- License
- —
- Allowed tools
- —
What does the processing-streaming-data skill do?
processing-streaming-data builds reliable streaming data pipelines with Kafka, Spark Structured Streaming, or Flink — delivery semantics (at-least-once vs exactly-once), idempotent consumers, event-time windowing and watermarks, handling late/out-of-order data, and checkpointing. Use when building or debugging streaming pipelines, configuring consumer groups, choosing delivery guarantees, or handling late events.
How to install the processing-streaming-data skill
Each command copies the skill folder into the client's skills directory. Restart the client or start a new session to load it.Install the processing-streaming-data skill in Plugin
/plugin marketplace add Unknown-333/awesome-data-engineering-skills /plugin install data-engineering-skills@awesome-data-engineering-skills
Install the processing-streaming-data skill in Claude Code
~/.claude/skills/processing-streaming-data/SKILL.md
$ npx degit Unknown-333/awesome-data-engineering-skills/skills/processing-streaming-data ~/.claude/skills/processing-streaming-dataInstall the processing-streaming-data skill in Claude Code (project)
.claude/skills/processing-streaming-data/SKILL.md
$ npx degit Unknown-333/awesome-data-engineering-skills/skills/processing-streaming-data .claude/skills/processing-streaming-dataInstall the processing-streaming-data skill in Codex
~/.agents/skills/processing-streaming-data/SKILL.md
$ npx degit Unknown-333/awesome-data-engineering-skills/skills/processing-streaming-data ~/.agents/skills/processing-streaming-dataInstall the processing-streaming-data skill in Gemini CLI
~/.gemini/skills/processing-streaming-data/SKILL.md
$ npx degit Unknown-333/awesome-data-engineering-skills/skills/processing-streaming-data ~/.gemini/skills/processing-streaming-dataInstall the processing-streaming-data skill in Claude.ai
npx degit Unknown-333/awesome-data-engineering-skills/skills/processing-streaming-data processing-streaming-data && zip -r processing-streaming-data.zip processing-streaming-data # claude.ai → Customize → Skills → + → Create skill → Upload a skill → processing-streaming-data.zip
SKILL.md
View on GitHubProcessing Streaming Data
When to use
- Building or debugging Kafka / Spark Structured Streaming / Flink pipelines.
- Choosing delivery semantics and making consumers idempotent.
- Windowing on event time and handling late/out-of-order events.
- Do NOT use for batch ELT (use
building-ingestion-pipelines).
Workflow
- [ ] Choose delivery semantics; make effects idempotent regardless
- [ ] Use event time (not processing time) with watermarks
- [ ] Configure checkpointing for recovery
- [ ] Decide late-data policy (allowed lateness -> update, drop, or side output)
- [ ] Size partitions/parallelism to the throughput
- Delivery semantics. At-least-once is the common default (duplicates possible on retry). Exactly-once needs transactional sinks/offsets. Either way, make the downstream effect idempotent (upsert by key) so duplicates don't corrupt state — this is more robust than relying on exactly-once alone.
- Event time + watermarks. Window by when the event happened, not when it was processed; a watermark bounds how long to wait for stragglers.
- Checkpoint so a failed job resumes from the last committed offset/state instead of reprocessing everything or losing data.
- Late data policy — allowed lateness updates windows; beyond it, drop or route to a side output/dead-letter for reconciliation.
Patterns
Idempotent consumer — key the sink write on a stable event id so replays upsert rather than duplicate:
# Structured Streaming: exactly-once-ish via idempotent upsert in foreachBatch
def upsert(batch_df, batch_id):
(delta_table.alias("t")
.merge(batch_df.dropDuplicates(["event_id"]).alias("s"), "t.event_id = s.event_id")
.whenNotMatchedInsertAll().execute())
(stream.writeStream.foreachBatch(upsert)
.option("checkpointLocation", "/chk/events").start())
Event-time windowing with watermark:
(events
.withWatermark("event_time", "10 minutes")
.groupBy(window("event_time", "5 minutes"), "user_id")
.count())
Kafka consumer basics — consumer group for scale-out; commit offsets after processing (not before) to avoid data loss; partition count caps parallelism.
Common pitfalls
- Relying on exactly-once instead of idempotent sinks — any at-least-once hop reintroduces duplicates; upsert by key.
- Processing-time windows — produce wrong results when events arrive late or the job lags; use event time + watermark.
- Committing offsets before processing — a crash then loses those messages.
- No checkpointing — recovery either reprocesses everything or loses state.
- Unbounded state (no watermark/TTL) — keyed state grows until the job OOMs.
- Too few partitions — caps consumer parallelism; you can't scale past partition count.
- Ignoring dead-letter/late paths — bad or very late events silently vanish.
Files in this skill
1processing-streaming-data skill questions
How do I install the processing-streaming-data skill in Claude Code?
Run npx degit Unknown-333/awesome-data-engineering-skills/skills/processing-streaming-data ~/.claude/skills/processing-streaming-data. Claude Code loads skills from ~/.claude/skills (all projects) or .claude/skills (one project) and uses a skill when its description matches the task.
Which clients can use the processing-streaming-data skill?
Any client that reads the Agent Skills format: Claude Code, Codex, Gemini CLI and claude.ai (upload as a ZIP). Scripts in the skill run only where the client allows code execution.