An AI tool that measures silent join losses between data sources and suggests normalization rules to recover missing matches.
Also known as harsh-chandak/shadowjoin
- Stars
- 0
- Tools
- —
- Downloads, 30 days
- —
- Latest version
- 0.4.0
- Last commit
- Nov 12, 2026
- License
- MIT
What is shadowjoin MCP?
shadowjoin is an AI tool that measures silent join losses between data sources and suggests normalization rules to recover missing matches.
Summary generated from the project's README
About shadowjoin MCP server
Data engineers and developers who struggle with silent data loss during joins can use shadowjoin to identify exactly which records are failing to match between sources. Instead of relying on exact matches that ignore variations in spelling or formatting, this tool runs a shadow matcher to find missed connections and calculates a lower bound on lost matches. It works with CSV, JSON, Postgres, MySQL, and SQLite sources, allowing for local file analysis or remote database scanning.
When connected to an AI client via the MCP server, it provides specific tools like scan_join, explain_misses, and suggest_normalization. The server is designed with security in mind, requiring credentials to be stored in a local configuration file rather than passed through tool calls. It also uses read-only transactions with strict timeouts when accessing databases to ensure production stability. Users can receive a ranked list of normalization rules, such as stripping corporate suffixes or punctuation, to recover missing data and improve join precision.
What you can do with it
- 1Identify silent join lossesMeasure how many rows are being dropped during a join between two datasets due to formatting differences.
- 2Generate normalization rulesGet a ranked list of specific string transformations to recover missing matches in a data pipeline.
- 3Verify data recovery precisionUse ground truth columns to prove that suggested normalization rules actually result in correct matches.
- 4Automate data quality checksIntegrate join loss measurement into a continuous integration pipeline to fail builds if data loss exceeds a threshold.
Written from the project's README. Every use case is checked against a quote from it.
How to install shadowjoin MCP
Formats verified 10 Nov 2026Add to CursorAdd to GooseInstall shadowjoin MCP in Claude Code
$ claude mcp add --env 'SHADOWJOIN_CONFIG=<SHADOWJOIN_CONFIG>' shadowjoin -- npx -y shadowjoinInstall shadowjoin MCP in Cursor
~/.cursor/mcp.json
{
"mcpServers": {
"shadowjoin": {
"command": "npx",
"args": [
"-y",
"shadowjoin"
],
"env": {
"SHADOWJOIN_CONFIG": "<SHADOWJOIN_CONFIG>"
}
}
}
}Install shadowjoin MCP in VS Code
.vscode/mcp.json
{
"servers": {
"shadowjoin": {
"type": "stdio",
"command": "npx",
"args": [
"-y",
"shadowjoin"
],
"env": {
"SHADOWJOIN_CONFIG": "<SHADOWJOIN_CONFIG>"
}
}
}
}Install shadowjoin MCP in VS Code CLI
$ code --add-mcp '{"name":"shadowjoin","type":"stdio","command":"npx","args":["-y","shadowjoin"],"env":{"SHADOWJOIN_CONFIG":"<SHADOWJOIN_CONFIG>"}}'Install shadowjoin MCP in Claude Desktop
claude_desktop_config.json
{
"mcpServers": {
"shadowjoin": {
"command": "npx",
"args": [
"-y",
"shadowjoin"
],
"env": {
"SHADOWJOIN_CONFIG": "<SHADOWJOIN_CONFIG>"
}
}
}
}Install shadowjoin MCP in Codex
$ codex mcp add shadowjoin --env 'SHADOWJOIN_CONFIG=<SHADOWJOIN_CONFIG>' -- npx -y shadowjoinInstall shadowjoin MCP in Codex config.toml
~/.codex/config.toml
[mcp_servers.shadowjoin] command = "npx" args = ["-y", "shadowjoin"] [mcp_servers.shadowjoin.env] SHADOWJOIN_CONFIG = "<SHADOWJOIN_CONFIG>"
Install shadowjoin MCP in Gemini CLI
$ gemini mcp add -s user -e 'SHADOWJOIN_CONFIG=<SHADOWJOIN_CONFIG>' shadowjoin npx -- -y shadowjoinInstall shadowjoin MCP in Gemini settings.json
~/.gemini/settings.json
{
"mcpServers": {
"shadowjoin": {
"command": "npx",
"args": [
"-y",
"shadowjoin"
],
"env": {
"SHADOWJOIN_CONFIG": "<SHADOWJOIN_CONFIG>"
}
}
}
}Configuration
SHADOWJOIN_CONFIGRequiredAbsolute path to shadowjoin.config.json, which names your data sources. Connection strings live here and never pass through a tool call.
| Name | Description | |
|---|---|---|
| SHADOWJOIN_CONFIG | Absolute path to shadowjoin.config.json, which names your data sources. Connection strings live here and never pass through a tool call. | Required |
Automated checks
Ran Nov 14, 2026Full report →Passed our automated checks. That is not a security audit: read the code and give the server the narrowest access it needs. How checks work
- Source repository linked and reachableharsh-chandak/shadowjoin is public and reachablegithub.com/harsh-chandak/shadowjoin ↗Pass
- Package names this repository as its sourceshadowjoin names harsh-chandak/shadowjoin as sourcenpmjs.com/package/shadowjoin ↗Pass
- Build provenance (Sigstore attestation)Sigstore attestation for shadowjoinnpmjs.com/package/shadowjoin/v/0.4.0#provenance ↗Pass
- Open source license (OSI)MITspdx.org/licenses/MIT.html ↗Pass
- Commit or release in the last yearLast commit or release 2026-11-12github.com/harsh-chandak/shadowjoin ↗Pass
- OpenSSF ScorecardNot checked
- Package name not close to a popular packageNo popular package one edit away from shadowjoinPass
- No critical advisory in the package or its dependenciesthe package checked; no critical advisoryosv.dev ↗Pass
- No hidden instructions to the model0 tool descriptions and the README read; no override phrases, credential paths or hidden charactersPass
- Destructive tools are labelledNot checked
- Remote endpoint answers the MCP handshakeNot checked
README
View on GitHubshadowjoin
Measure the matches your exact join is silently dropping.
Your pipeline joins two sources on a name, an email, a SKU, a company. It matches what matches. Everything else just... isn't there. No error, no exception, no alert — the query returns, the dashboard renders, and a quarter of the rows that should have matched quietly don't.
shadowjoin runs a shadow matcher alongside your real join key and tells you how many matches you are losing, which normalization rule would recover each one, and — if you have any ground truth at all — proves the recoveries are real.
npx shadowjoin scan \
--left customers.csv --left-key company_name \
--right crm_export.csv --right-key account_name
Fourteen seconds, no sound. Every figure in it is verbatim CLI output from the committed example below — higher-resolution version.
The progress output above is real: shadowjoin shows what it is doing while it
works. All of it is suppressed automatically when stdout is not a terminal, so
piping to a file, to jq, or running in CI produces byte-identical output to
before — and there is no postinstall script, because a tool you point at a
production database has no business running code at install time.
What it found on public data
Joining the S&P 500 constituent list against NYSE + Nasdaq listings on company name. Both files carry ticker symbols, so every recovered match can be checked rather than believed:
left rows 503 right rows 6,709
exact join matches 1 0.2%
recovered by shadow join 428 85.09%
still unexplained 74
LOWER BOUND ON LOST MATCHES: 85.09% of left rows
verified against ground truth: 419 correct, 1 wrong, 8 ambiguous (precision 99.76%)
rules worth adopting, in order (greedy by marginal recovery):
1. strip security/listing boilerplate (common stock, ADS, ...) +68 68
2. strip corporate suffixes (inc, corp, ltd, plc, gmbh ...) +319 387
3. strip punctuation +33 420
4. sort tokens (order-insensitive match) +4 424
5. remove all whitespace +1 425
The exact join found one match. There were four hundred and twenty-eight, and 419 of them are confirmed correct by ticker.
Reproduce it yourself — the data ships with the repo:
git clone https://github.com/harsh-chandak/shadowjoin && cd shadowjoin
npm install && npm run build
node dist/cli.js scan \
--left examples/sp500.csv --left-key Security --left-id Symbol --left-truth Symbol \
--right examples/listings.json --right-key name --right-id symbol --right-truth symbol
A join you would actually ship
The example above is deliberately unflattering — a short index name against a full security description. Here is the realistic case.
Unicode CLDR country names (what your UI displays) joined to UN official country names (what your reference data uses), on the country name, with ISO 3166 alpha-3 as ground truth. That is a join people write every week.
exact join matches 205 / 249 82.33%
recovered by shadow join 5 2.01% precision 100%
still unexplained 39
rules worth adopting:
1. & -> and +4
2. remove all whitespace +1
It looks healthy. 82% matched, no errors, nobody investigating. And it is
silently dropping Antigua & Barbuda, Trinidad & Tobago,
São Tomé & Príncipe, Turks & Caicos Islands and Vietnam — the last one
because the UN spells it Viet Nam.
Two things this example shows that the first one does not:
Applying every rule finds nothing. The full rule set recovers zero here; the two-rule subset recovers five. Over-normalization collapses keys that should stay distinct, and shadowjoin names the better subset instead of letting you assume more rules is better.
It separates mechanical misses from semantic ones. The 39 it cannot explain are not normalization problems at all:
Vatican City vs Holy See
North Korea vs Democratic People's Republic of Korea
Cape Verde vs Cabo Verde
Bosnia vs Bosnia and Herzegovina
Hong Kong vs China, Hong Kong Special Administrative Region
No string transform recovers those. They need an alias table or a real entity resolution library. Knowing which of your misses are mechanical and which are semantic is the difference between a one-line fix and a project.
node dist/cli.js scan \
--left examples/countries_ui.csv --left-key country --left-truth iso3 \
--right examples/countries_reference.csv --right-key country --right-truth iso3
That particular join is deliberately unflattering — a short index name against a full security description. It is a worked example of the measurement, not a claim that anyone ships that join. Point it at your own data to get a figure that means something about you.
A join on more than one column
Real joins are rarely one column. They are (name, city), (sku, vendor),
(country, code) — and the most useful thing a join like that can tell you is
which part of the key disagreed. Comma-separate the columns:
node dist/cli.js scan \
--left examples/countries_ui.csv --left-key country,iso3 \
--right examples/countries_reference.csv --right-key country,iso3
left rows 249 right rows 249
join key country + iso3
exact join matches 205 82.33%
recovered by shadow join 5 2.01%
still unexplained 39
which key components agree, among the 44 row(s) the exact join missed:
components rows cands
----------------------------------------------------------------------------
= country + iso3 (every component) 5 1
iso3 39 1
Only "=" is counted in the floor. The rest agree on part of the key: a
dirty column, or a different record — never counted, always worth reading.
"cands" is the most right rows one left row agreed with: 1 identifies, many block.
examples of partial agreement (suggestions, NOT counted in the floor):
agrees on iso3, not on country
country Bolivia x Bolivia (Plurinational State of)
iso3 BOL = BOL
Thirty-nine rows where the ISO code agrees and the country name does not. That is neither a normalization problem nor a missing record: it is one column carrying the join while the other vetoes it. Add a column to a join "to be safe" and this is what it costs you — and an exact join reports all forty-four misses as the same silence.
Partial agreement is never counted in the floor. Only rows where every
component agrees are counted — the = line. Agreeing on a name is not evidence
that two cities are the same place, and a bound that counted maybes would not be
a bound. So partial agreement is reported beside the floor, never inside it.
Two things make a partial bucket readable rather than suggestive:
cands— the most right rows that agreed with any single left row.1means those components identify a row on their own, so a key of just them is worth measuring, and shadowjoin says so. A large number means they block rather than identify — half the table shares a city — and those rows are candidate sets, not pairs. shadowjoin will not suggest narrowing to them.- ground truth — supply a
truthcolumn on both sides and every bucket gains its ownokandwrongcounts. On a partial bucket,okmeans those rows really are the same entity, so the disagreeing column is dirty and the loss is real;wrongmeans they are different records that happen to share a component. Two buckets of identical size can mean opposite things, and the count alone cannot tell them apart.
The number is a floor, not an estimate
You cannot compute recall without labels. The entity-resolution literature calls the metric pair completeness, and it assumes an oracle you do not have in production.
So shadowjoin reports a lower bound: matches the shadow matcher found that exact matching missed. The shadow matcher has unknown recall of its own, so the true loss is at least this large and may be larger. Every figure ships with sample pairs so a human can verify them by hand, and when ground truth is available on both sides, the report carries a precision figure too.
This is deliberately the weaker claim. It is also the one that survives being questioned.
Why not just ask an LLM to check?
For a one-off look at a small table, do that — it is faster and this tool is redundant. It stops working when:
- You need the same number twice. An agent writes different SQL every run, so you cannot tell whether the pipeline changed or the query did.
- You need rules nobody thought of. Asked "is this join lossy," an agent tries
lower()andtrim(). It does not systematically try NFKD folding, diacritics, leading zeros, corporate suffixes, token reordering or listing boilerplate. The exhaustive rule set is the product. - You need it in CI.
--max-lossexits non-zero. "Ask Claude" has no exit code. - Exploratory SQL on production is expensive. This does hash joins on precomputed keys, under a statement timeout, in a read-only transaction.
Install
npm install shadowjoin # or run it with npx, no install
Zero runtime dependencies. pg and @modelcontextprotocol/sdk are optional
peers, pulled in only if you use the Postgres adapter or the MCP server.
Use it
CLI
# CSV or JSON, no database involved
npx shadowjoin scan --left a.csv --left-key name --right b.json --right-key title
# with ground truth on both sides, so recoveries are verified
npx shadowjoin scan \
--left a.csv --left-key name --left-truth customer_id \
--right b.csv --right-key title --right-truth customer_id
# straight at Postgres
npx shadowjoin scan \
--left 'postgres://ro@host/db' --left-table orders --left-key customer_name \
--right 'postgres://ro@host/db' --right-table customers --right-key name
# a composite key: comma-separate the columns, same order on both sides
npx shadowjoin scan \
--left orders.csv --left-key customer_name,city \
--right customers.csv --right-key name,town
# in CI: fail the build if the floor climbs above 2%
npx shadowjoin scan --left a.csv --left-key name --right b.csv --right-key name --max-loss 2
# MySQL and SQLite work the same way
npx shadowjoin scan \
--left 'mysql://ro@host/db' --left-table accounts --left-key company_name \
--right ./reference.sqlite --right-table vendors --right-key name
npx shadowjoin rules # list the normalization rules
Sources: CSV, JSON, Postgres, MySQL and SQLite. SQLite uses Node's built-in
node:sqlite (Node 22.5+), so it needs nothing installed; Postgres and MySQL
need pg or mysql2, which are optional peers.
Anything else works through the library in three lines, because the core takes two arrays rather than a connection:
import { scan } from "shadowjoin";
const [rows] = await anyDriver.query("SELECT id, company_name FROM accounts");
const left = rows.map((r) => ({ id: String(r.id), key: r.company_name }));
scan(left, right);
Library
import { scan, normalize, rulesByIds, toSql } from "shadowjoin";
const report = scan(leftRows, rightRows); // rows are { id, key, truth? }
console.log(report.lowerBoundLossPct, report.stack);
// a composite key: give each row `keys` instead of `key`, in column order
const composite = scan(
[{ id: "1", keys: ["Acme", "Boston"] }],
[{ id: "a", keys: ["Acme Inc.", "Boston"] }],
{ keyLabels: ["name", "city"] },
);
composite.agreement; // the missed rows, grouped by WHICH components agree
composite.partialPairs; // examples, component by component. Never counted.
// adopt the recommended rules in your own pipeline
const rules = rulesByIds(report.stack.map((s) => s.rule));
const key = (value) => normalize(value, rules);
toSql("company_name", rules); // the same thing as Postgres SQL
Claude Code plugin (easiest)
One command installs the skill and the MCP server together, and prompts for your config path:
claude plugin marketplace add harsh-chandak/harsh-chandak
claude plugin install shadowjoin@harsh-chandak
Any other MCP client
Codex, Copilot, Cursor, Zed and anything else that speaks the protocol take a stdio server rather than a Claude Code plugin:
{
"mcpServers": {
"shadowjoin": {
"command": "npx",
"args": ["-y", "-p", "shadowjoin", "-p", "@modelcontextprotocol/sdk", "shadowjoin-mcp"]
}
}
}
Nothing is installed up front; npx fetches both on first run. The second -p
is not optional — the SDK is a peer dependency, not a runtime one, so this
package stays dependency-free for anyone who only wants the CLI. Add -p pg or
-p mysql2 if you will scan a database.
Also listed in the official MCP registry
as io.github.harsh-chandak/shadowjoin.
MCP server
Lets an agent measure a join, look at real examples, propose a fix, and then prove the fix worked — a loop it cannot fake, because the measurement is not something it writes.
{
"mcpServers": {
"shadowjoin": {
"command": "npx",
"args": ["-y", "-p", "shadowjoin", "-p", "@modelcontextprotocol/sdk", "shadowjoin-mcp"],
"env": { "SHADOWJOIN_CONFIG": "/abs/path/to/shadowjoin.config.json" }
}
}
}
{
"sources": {
"prod": { "type": "postgres", "connectionString": "postgres://shadowjoin_ro@host/db" },
"crm": { "type": "csv", "path": "./crm_export.csv" }
}
}
Tools: scan_join, explain_misses, suggest_normalization, compare_runs.
Pass columns: ["name","city"] instead of column on either side to join on a
composite key; the summary then carries the agreement breakdown as well as the
floor.
Two things this MCP server deliberately does not have:
- No tool that executes arbitrary SQL. An agent can name a configured source, a table and a column. That is the entire surface.
- No tool that accepts a connection string. Credentials live in your config file and never enter a tool call, because tool calls end up in transcripts.
Pointing it at a real database
shadowjoin opens a READ ONLY transaction with statement_timeout,
lock_timeout and idle_in_transaction_session_timeout set, and it refuses to
run if the role it connected with can write to the table it is about to read.
Give it a role that makes trust unnecessary:
CREATE ROLE shadowjoin_ro LOGIN PASSWORD 'xxx';
GRANT CONNECT ON DATABASE yourdb TO shadowjoin_ro;
GRANT USAGE ON SCHEMA public TO shadowjoin_ro;
GRANT SELECT ON public.orders, public.customers TO shadowjoin_ro;
Point it at a replica where you can.
It makes no outbound network connection except to the source you give it. No telemetry, not even opt-out. Check the imports — there are three dependencies and none of them is an HTTP client.
Limits
Read these before quoting a number from it.
- The figure is a floor. The shadow matcher's own recall is unknown. True loss is at least this, possibly more.
- Recoveries can be wrong. In the bundled example, 1 of 428 was contradicted
by ground truth and 8 were ambiguous — mostly share classes, where stripping
"Class A" merges genuinely different securities. Ambiguous matches need a
tie-breaker, not a normalization rule. Always read
samplePairs. - Partial agreement is not part of the floor. With a composite key, rows
agreeing on some components and not others are reported separately and never
counted. They are the most useful output the tool has, and also the easiest to
over-read: a shared name is not a shared record. Check the
verifiedcount on the bucket, and readpartialPairs, before calling any of them a loss. - Per-rule counts overlap and will not sum to the total. Each is measured
independently against the exact baseline, deliberately, so the numbers are not
an artifact of the order rules happen to run in. Use
stack— the greedy ordering — to decide what to actually adopt. - More rules is not better. Applying every rule can recover fewer matches than the best subset, because over-normalization merges keys that should stay apart. shadowjoin warns when this happens and names the better subset.
- It reads your join keys, so those values are in memory while it runs.
--redacthashes them in sample output and server-side normalization keeps whole rows in the database, but neither makes the keys invisible. For regulated data, run it on a replica inside your own environment. - It does not write, fix or migrate anything. It measures.
unaccentin emitted SQL needs the Postgres extension of the same name.
Prior art
Blocking recall — pair completeness — is well established in the entity-resolution literature, along with its companions pairs quality and reduction ratio:
- A Survey of Blocking and Filtering Techniques for Entity Resolution
- Comparative Analysis of Approximate Blocking Techniques for Entity Resolution
- Supervised Meta-blocking
Those papers assume a labelled gold standard. shadowjoin's contribution is not the metric — it is computing a defensible version of it against a live pipeline with no labels at all, and reporting it as a bound rather than a point estimate.
For doing the matching rather than measuring what yours misses, use Splink, Dedupe or recordlinkage. shadowjoin is complementary: it tells you whether you need them.
License
MIT © Harsh N. Chandak
shadowjoin MCP questions
How do I add shadowjoin MCP to Claude Code?
Run claude mcp add --env 'SHADOWJOIN_CONFIG=<SHADOWJOIN_CONFIG>' shadowjoin -- npx -y shadowjoin in your terminal. Add --scope user to make it available in every project.
How do I add shadowjoin MCP to Cursor?
Open ~/.cursor/mcp.json and add the server under mcpServers, using the Cursor config shown in the install section above.
How does shadowjoin handle database security?
It opens a read-only transaction with timeouts and refuses to run if the connected role has write permissions. It also requires credentials to be stored in a config file rather than passed through tool calls.
Is the reported match count an exact estimate?
No, the figure is a lower bound. It represents the matches the shadow matcher found that the exact join missed, but the true loss may be larger because the shadow matcher's own recall is unknown.
