Enables AI agents to diagnose Apache Spark performance issues, summarize runs, and compare job executions using event logs.
Also known as shuffle-works/sparkforensics, sparkforensics
- Stars
- 0
- Tools
- 5
- Downloads, 30 days
- —
- Latest version
- 0.1.0
- Last commit
- Nov 13, 2026
- License
- MIT
What is SparkForensics MCP?
SparkForensics enables AI agents to diagnose Apache Spark performance issues, summarize runs, and compare job executions using event logs. It exposes 5 tools, such as diagnose_run, get_run_summary, compare_runs.
Summary generated from the project's README
About SparkForensics MCP server
SparkForensics provides a specialized way for AI agents to investigate Apache Spark performance issues using event logs. Instead of relying on raw metrics from a History Server, this tool parses logs to identify specific bottlenecks like data skew, shuffle issues, memory spills, and garbage collection pressure. It is designed for developers and data engineers who need to understand why a job is slow without installing anything on their existing clusters.
The MCP server allows an LLM to directly access tools for diagnosing, summarizing, and comparing different job runs. Users can run this in a local-server mode using a Node companion, which can also fetch runs from a Spark History Server via a base URL. This setup ensures that data remains on the local machine, as nothing is uploaded to external accounts. The server supports complex tasks such as structural A/B comparisons to detect regressions between job executions.
What you can do with it
- 1Compare job executionsHave the agent perform a structural comparison between a current job run and a baseline to identify regressions.
- 2Automate regression testingUse the headless CLI to evaluate a Spark run against specific budget thresholds like maximum runtime or skew.
Written from the project's README. Every use case is checked against a quote from it.
How to install SparkForensics MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall SparkForensics MCP in Claude Code
$ claude mcp add sparkforensics -- npx -y sparkforensics-mcpInstall SparkForensics MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"sparkforensics": {
"command": "npx",
"args": [
"-y",
"sparkforensics-mcp"
]
}
}
}Install SparkForensics MCP in VS Code
.vscode/mcp.json
{
"servers": {
"sparkforensics": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"sparkforensics-mcp"
]
}
}
}Install SparkForensics MCP in VS Code CLI
$ code --add-mcp '{"name":"sparkforensics","type":"stdio","command":"npx","args":["-y","sparkforensics-mcp"]}'Install SparkForensics MCP in Claude Desktop
claude_desktop_config.json
{
"mcpServers": {
"sparkforensics": {
"command": "npx",
"args": [
"-y",
"sparkforensics-mcp"
]
}
}
}Install SparkForensics MCP in Codex
$ codex mcp add sparkforensics -- npx -y sparkforensics-mcpInstall SparkForensics MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.sparkforensics] command = "npx" args = ["-y", "sparkforensics-mcp"]
Install SparkForensics MCP in Gemini CLI
$ gemini mcp add -s user sparkforensics npx -- -y sparkforensics-mcpInstall SparkForensics MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"sparkforensics": {
"command": "npx",
"args": [
"-y",
"sparkforensics-mcp"
]
}
}
}Started by HostedMCP on 16 Nov 2026
We ran this command for sparkforensics-mcp 0.1.0 in an isolated container with no network and no keys. It answered and listed 5 tools.
SparkForensics MCP tools 5
Read from the server with tools/listdiagnose_runDiagnose a Spark run: thresholded findings with remediation text, an impact-ranked fix recommendation rollup, and clean-check status.
get_run_summaryApp/stage/job/sql counts and duration for a run, no findings.
compare_runsCompare two runs: categorized findings delta and metric deltas.
evaluate_budgetsEvaluate a run (optionally against a second run for regression budgets) against pass/fail thresholds.
get_finding_evidenceRaw evidence bundle backing one finding, for drill-down after diagnose_run.
Automated checks
Ran Nov 14, 2026Full report →Passed our automated checks. That is not a security audit: read the code and give the server the narrowest access it needs. How checks work
- Source repository linked and reachableshuffle-works/sparkforensics is public and reachablegithub.com/shuffle-works/sparkforensics ↗Pass
- Package names this repository as its sourcesparkforensics-mcp names shuffle-works/sparkforensics as sourcenpmjs.com/package/sparkforensics-mcp ↗Pass
- Build provenance (Sigstore attestation)No build attestation published for this releaseNot checked
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-11-13github.com/shuffle-works/sparkforensics ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNo popular package one edit away from sparkforensics-mcpPass
- No critical advisory in the package or its dependencies99 resolved dependencies checked; no critical advisoryosv.dev ↗Pass
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubSparkForensics
Your Spark job is slow. SparkForensics tells you why, and what to do about it.
Drop a Spark event log in and get an evidence-backed breakdown of which stages are slow, why, and how to fix them. Nothing installs on your cluster and nothing leaves your machine.
Try it now → with your own event log. No install, no account, no upload. Read the documentation for a guided tour.
Spark's own History Server gives you raw metrics. It won't tell you that stage 14 spilled 40GB because of a skewed join key, or that cold start is eating a third of the job. SparkForensics reads the same event logs and turns them into findings.
Why SparkForensics
- Your data stays on your machine. Drop a log file and read the dashboard; nothing is uploaded anywhere. Local-server mode adds a small Node companion that can also pull runs from a Spark History Server.
- Event logs are streamed and parsed off the main thread in a Web Worker, so a 240MB+ (or multi-GB) NDJSON log doesn't freeze the tab.
- The finding types cover skew, shuffle, spill, GC pressure, stragglers, cold start, cache/memory utilization, autoscaling churn, job/stage failures and partition sizing, plus a SQL Plan Advisor (duplicate subtrees, small files, broadcast sizing) and a config audit. Each has a page in the detector catalog, and its thresholds are in the detector table.
- Fix These First orders every finding by its estimated wall-clock impact, not severity alone, so the top row is the fix worth doing first.
- Run comparison (A/B) mode structurally matches stages across two runs of the same job and shows which stages regressed, rather than every difference.
- Every finding tracks what it could and couldn't confirm from the log, so you know when to trust a flag and when it's a hint worth checking manually.
- A headless CLI (
sparkforensics-analyze) evaluates a run against budget thresholds (max runtime, skew, spill, failure rate...) and exits non-zero on violation, so a regression can fail a build. The run can come from a local event-log file/directory or, with--shs-base-url, fetched directly from a Spark History Server. - An MCP server (
sparkforensics-mcp) exposes tools for diagnosing, summarizing and comparing runs, so an LLM agent can investigate a run directly. The full tool list is in MCP tools. - For running the analyze CLI automatically after every Spark job instead of by hand, see sparkforensics-operator, an Airflow operator that runs SparkForensics on a job's event log and acts on the result.
Unlike tools that need a cluster-side plugin or a running server plus an LLM agent (Spark History Server MCP tools), SparkForensics needs nothing installed anywhere else: the event log and a browser tab are enough.
Get started
Run the dashboard locally:
npx sparkforensics-server
The Getting started guide covers what it serves, the live demo alternative, and loading a run. To work on SparkForensics itself, see Development.
For a one-shot check with no browser, the kind a CI pipeline can gate on:
npx -p sparkforensics-cli sparkforensics-analyze path/to/eventlog --max-runtime 3600000 --max-skew 3
The command ships in the sparkforensics-cli package. To install it once
instead: npm i -g sparkforensics-cli, then run sparkforensics-analyze
directly. The sparkforensics-analyze alias package also works:
npx sparkforensics-analyze.
That's a headless run against budget thresholds. See Command-line reference below for HTML export, Spark History Server fetching, baseline comparisons, and redaction. To run a detector check with your own threshold, see Tuning detector thresholds.
Development
Install and run
From a checkout:
git clone git@github.com:shuffle-works/sparkforensics.git
cd sparkforensics
npm install
npm run dev
Open the URL Vite prints. For a production bundle: npm run build, then serve dist/ with
any static file server. Asset URLs are relative, so the bundle also works
under a URL subpath.
Note: The app must be served over HTTP:
file://URLs block Web Worker module imports in Chrome.
Deploy modes
This app runs in two modes:
- Static mode (e.g. GitHub Pages, or any plain file server): Choose file
is the normal browser-local path for a single event log. Choose
rolling-log folder is available only for an
eventlog_v2_*directory. Deploy the output ofnpm run buildfromdist/. Fetching directly from a Spark History Server is not available: the browser's CORS policy blocks it and a static host can't proxy the request. If a History Server load cannot be used, choose a local event-log file instead. - Local-server mode is a small Node companion that serves this app and fetches from a Spark History Server on your behalf (server-to-server, so no browser CORS restriction). Useful when the SHS is behind a VPN or isn't CORS-configured.
Run local-server mode with:
npx sparkforensics-server
It listens on http://127.0.0.1:4173 by default (override with --port or
the PORT env var) and binds to localhost only. Open the URL in Chrome and
use the Fetch from Spark History Server disclosure on the intake
screen. The same server serves the MCP tools over streamable HTTP at /mcp,
for clients on the same machine; see
Connecting over HTTP.
History Server fetching requires local-server mode, a History Server reachable from the machine running that server, and a supported base application ID (see Getting started for the full list of accepted forms). Enter an optional attempt separately from the base application ID. The local server sends no credentials and follows no upstream redirects. Its recoverable errors intentionally omit upstream response details; edit the fields or choose a local event-log file to recover.
During development from a checkout, npm run local-server does the same
thing without publishing or installing the package.
Server configuration
All settings have working defaults. The port (PORT or --port) and the
SPARKFORENSICS_* environment variables are listed in
Server configuration.
SHS-proxy route test fixture
The /shs-proxy route (packages/core/src/proxy.js) is covered by
packages/server/test/shs-proxy-fixture.test.js, part of the normal npm run test:server suite. It loads a real event log from the
spark-event-corpus-data git submodule at dev/log-corpus/ and asserts
the route streams it back as a valid ZIP. No Docker, no live Spark
process, and no live History Server are needed for this.
To verify the full Fetch-from-SHS UI flow by hand, including its
unreachable-upstream and application-not-found recovery states, point
npm run local-server at a real Spark History Server you have access to
and use its Fetch from Spark History Server disclosure. Automated
coverage of /shs-proxy uses the spark-event-corpus-data logs, so no
Docker image or live Spark process is needed.
Running tests
npm test # single run
npm run test:watch # watch mode
npm test -- tests/finding-filter.test.js # single file
npm run test:core # packages/core: detectors, analyzer, parser, MCP tools
npm run test:cli # packages/cli
npm run test:mcp # packages/mcp
npm run test:server # packages/server
Command-line reference
Beyond the plain budget check in Get started, the same
sparkforensics-analyze CLI covers a few other jobs.
Write a self-contained HTML dashboard for the run instead of a JSON/Markdown
report, viewable offline over file:// with no server:
npx -p sparkforensics-cli sparkforensics-analyze path/to/eventlog --export-html path/to/output-dir
Fetch the run straight from a Spark History Server instead of a local file (mutually exclusive with the positional path):
npx -p sparkforensics-cli sparkforensics-analyze --shs-base-url http://history-server:18080 --app-id application_1234_0001 --max-skew 3
Compare a candidate run against a baseline (A/B) and fail the build on a regression or a newly introduced critical finding:
npx -p sparkforensics-cli sparkforensics-analyze path/to/candidate --baseline path/to/baseline --max-regression-pct 10 --fail-on-introduced critical
Several regression budgets in one call, and several candidates against one baseline (NDJSON, one line per candidate; the baseline is parsed once):
npx -p sparkforensics-cli sparkforensics-analyze cand-1 cand-2 cand-3 --baseline base --regression-budget wallClock:10 --regression-budget gcTime:25
Compare against a baseline that lives on a Spark History Server, from a local
candidate or a History Server one, with --baseline-shs-base-url,
--baseline-app-id and --baseline-attempt-id (the baseline can be on a
different server):
npx -p sparkforensics-cli sparkforensics-analyze --shs-base-url http://history-server:18080 --app-id application_1234_0002 --baseline-shs-base-url http://history-server:18080 --baseline-app-id application_1234_0001 --max-regression-pct 10
Several candidates from one History Server take repeated --app-id values
(--app-id <id>:<attempt> names an attempt) and print NDJSON, one line each.
See Several budgets and several candidates
for the --budgets file, the NDJSON line fields and the exit code rules.
List every finding type the detectors can report, without reading a log (the MCP
list_detectors tool returns the same records):
npx -p sparkforensics-cli sparkforensics-analyze detectors --format md
Redact the app id, the app name, host names, IP addresses and user names in URLs and
paths before sharing the output outside the environment that produced it. Other paths, table
names and SQL text stay as written; the
CI guide
lists what --redact covers and what it does not:
npx -p sparkforensics-cli sparkforensics-analyze path/to/eventlog --redact
Narrow the output's findings array to certain impact bands, types, or a
stage (recommendations/cleanChecks/notRunChecks and the summary counts stay on the
full, unfiltered set):
npx -p sparkforensics-cli sparkforensics-analyze path/to/eventlog --impact critical,warning --type stageSlowness --stage 12
Other budget flags round out the checks: --max-spill <gb>,
--max-failed-task-rate <pct>, and --min-efficiency <pct> (busy core time:
the share of executor core time that ran tasks, 100 minus the dashboard's
Unused core time, not its Efficiency tile). --attempt-id
pairs with --shs-base-url/--app-id for a non-default attempt;
--regression-metric picks which metric --max-regression-pct checks
(default wallClock); --conf-keys and --conf-redact-regex shape the JSON
output's effectiveConf block, next to its metrics block (both in
Getting started);
--format md switches output to Markdown (default
json); --out <path> writes it to a file instead of stdout (its folder must
exist, or the command exits 2). The JSON is one { schemaVersion, run, comparison, budgets }
object, described by packages/cli/schema/cli-envelope.schema.json
(JSON output);
Exit codes:
0 pass, 1 a budget was violated, 2 usage error (bad flags, arguments or
a --thresholds/--budgets file), 3 a budget was inconclusive (e.g. missing
evidence for a regression check), 4 the candidate log can't be read or
parsed, 5 the baseline log can't be read or parsed, 6 internal error. When
several apply, the worst wins in the order 6, 5, 4, 1, 3, 0; with
several candidates that means the worst line, and a candidate that can't be
read or parsed gives its own line 4.
Node version: the published
sparkforensics-cli,sparkforensics-mcp, andsparkforensics-serverpackages all pre-strip their vendored TypeScript to plain JS at publish time, sonpx-installed use only needs Node>=18(see each package'senginesfield). A checkout running any of the three straight offpackages/*/bin/againstpackages/core/src/*.ts(the monorepo dev-mode fallback) needs a Node version with native (unflagged) TypeScript type-stripping instead:>=22.18.0on the Node 22 LTS line, or>=23.6.0on Node 23+.
Learn more
The full documentation is published at shuffle-works.github.io/sparkforensics/docs.
- Getting started guide: reading the dashboard, run comparison, CI and automation
- Understanding findings
- MCP tools
- Detector table
- Worker protocol
- Streaming/parsing design
- Widget layout
Contributing
See CONTRIBUTING.md.
License
SparkForensics MCP questions
How do I add SparkForensics MCP to Claude Code?
Run claude mcp add sparkforensics -- npx -y sparkforensics-mcp in your terminal. Add --scope user to make it available in every project.
How do I add SparkForensics MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
What tools does SparkForensics MCP provide?
It exposes 5 tools, including diagnose_run, get_run_summary, compare_runs, evaluate_budgets, get_finding_evidence.
Can I use SparkForensics to compare two different Spark runs?
Yes, it features an A/B comparison mode that structurally matches stages across two runs to identify regressions.
Is my data safe when using the dashboard?
Yes, your data stays on your machine and nothing is uploaded anywhere.