MCP server Code Crawler

Code-Crawler

Static + dynamic call tracing MCP server backed by a persistent property graph of your codebase.

Code-Crawler indexes a project into a SQLite-backed property graph — files, packages, modules, classes, interfaces, methods, fields, imports, annotations/decorators, call edges, virtual dispatch, aspect advice, and the wiring frameworks add (entry points, routes, injected dependencies, reflective lookups) — then exposes that graph to LLM clients as MCP tools: find callers, trace call paths, explore neighborhoods, follow runtime dispatch, and search symbols across the whole codebase.

Supports Java, Python, TypeScript, and JavaScript projects, and can index all of them into the same graph.

📖 Documentation site: https://nithin554.github.io/code-crawler/ 📘 Task-oriented guide: USAGE.md 🗺️ Roadmap: ROADMAP.md

        ┌───────────────────────  MCP (stdio)  ───────────────────────┐
        │  crawl_project  search_symbols  get_callers  get_callees     │
        │  trace_path     trace_execution  find_aspects  get_decorators│
        │  find_implementations  find_hotspots  find_dead_code  ...    │
        └───────────────┬──────────────────────────────┬───────────────┘
                        │                              │
                 ┌──────▼───────┐             ┌────────▼────────┐
                 │   crawler    │             │  graph queries  │
                 │ parse+resolve│             │ recursive CTEs  │
                 └──────┬───────┘             └────────┬────────┘
                        │                              │
                 ┌──────▼──────────────────────────────▼──────┐
                 │        SQLite property graph (WAL)          │
                 │   nodes (kind, fqn, props) + edges (kind)   │
                 └─────────────────────────────────────────────┘

Why a graph?

Plain-text grep can't answer questions like:

• "Who transitively calls OrderRepository#create?"
• "Is there a static path from Main#main to Tracking#dispatch?"
• "What is the impact radius of changing OrderService#place?"

These need graph traversal. Code-Crawler builds the graph once (idempotently, file-by-file) and answers them with bounded recursive SQL queries.

Features

• Four languages, one graph: Java (tree-sitter-java), Python (tree-sitter-python), TypeScript (tree-sitter-typescript, tree-sitter-tsx), and JavaScript (tree-sitter-javascript) — all WASM, no native build step. Language is detected per file, so a polyglot repo lands in a single graph.
• Dynamic tracing on a static graph — DYNAMIC_DISPATCH edges connect interface/base calls to every override in the project, and ADVISES edges weave AspectJ/Spring AOP advice into execution traces (before, after, around, after-returning, after-throwing, named @Pointcut references, &&/||/! and execution, within, @annotation, @within, target, this, args primitives).
• Run-time indirection is named, not lost — Java reflection/ServiceLoader, Python getattr/importlib, and JS require/dynamic import() produce REFLECTS edges to the resolved symbol when the target is statically determinable, or to a __reflect__: marker when it is not.
• Framework wiring — dependency injection (@Autowired, NestJS constructor injection, FastAPI Depends) becomes INJECTS edges, and HTTP routes (@GetMapping, @RequestMapping, NestJS @Controller/@Get, FastAPI/Flask/Quart decorators, Express app.get) become ROUTES edges to the handler, which is also marked as an entry point.
• Control flow on call edges — every CALLS edge records whether the call is conditional and the compact source text of its enclosing guards/loops, so an agent can tell a happy path from an error path without reading the file.
• External dependencies are explicit — third-party imports and runtime globals become package-level external_symbol nodes with REFERENCES edges and a find_external_dependencies summary, distinct from project code.
• Annotations as first-class nodes — @Transactional, @Cacheable, @Async, @Retryable, Python decorators, and TS decorators become annotation nodes with ANNOTATED_WITH edges, so you can query decorators on types, methods, and fields.
• Call resolution heuristics that go beyond naive string matching: receiver types inferred from locals, parameters, fields, new T() expressions, self/this scoping, dotted paths (service.repo.find), loop-variable element types, bare/this/super calls, static/qualified calls, and superclass chains. Unresolvable calls become unknown_symbol phantom nodes with resolved:false edges, so the graph stays traversable.
• Incremental, file-scoped indexing — each crawl records a per-file manifest (size, mtime, SHA-256) and replays the cached parse result for files whose content is unchanged, so a re-crawl re-parses only what actually changed. Changes are reported as filesAdded / filesUpdated / filesUnchanged / filesRemoved. Derived dynamic edges and orphan pruning are rebuilt only when something changed. A full re-crawl still removes files that disappeared, without duplicating the graph, and scoped crawls (fileFilter or a forced language) never remove files outside their scope. When a checkout, rsync -t, an mtime-preserving unzip, or clock skew restores an edit with the old size and mtime, the shortcut misses it — pass force (--force on the CLI) to re-parse everything.
• FTS5-backed symbol search — symbol lookup runs against an external-content FTS5 index with BM25 ranking and exact/prefix boosting, falling back to a LIKE scan only when FTS finds nothing. Search does not degrade as the graph grows.
• Versioned schema with automatic migration — the database carries a schema version and upgrades itself in place on open. A database written by a newer build is rejected with a clear message instead of being read as if the shape were known.
• A database per project — the index lives in <project>/.code-crawler/index.db, so two repositories can never collide. Pointing a populated database at a different project fails with a clear_index hint instead of silently wiping the first index.
• Analytics — hotspots (fan-in/fan-out/advice counts) and dead-code candidates.
• Self-contained: single SQLite file, no external services.
• LLM-friendly: every tool returns structured JSON, unknown symbols come back with fuzzy search suggestions.

Install & build

Requires Node.js ≥ 22.12 (better-sqlite3@13 and the Vitest toolchain both need it).

npm install
npm run build        # tsc → dist/
npm test             # vitest (full suite; set CODECRAWLER_BENCH=1 for the scale test)
npm run lint         # biome check (lint + format, fails CI)
npm run format       # biome format --write
npm run coverage     # vitest run --coverage (thresholds in vitest.config.ts)
npm run bench        # crawl/query benchmark on a generated ~137k-line project

CLI

codecrawler crawl <path>     # index a project
codecrawler crawl <path> --force  # re-parse every file, ignoring cached metadata
codecrawler status           # index summary
codecrawler query <symbol>   # fuzzy-lookup a symbol
codecrawler serve            # start the MCP server over stdio (default)

Environment variables:

Variable	Default	Purpose
CODECRAWLER_DB	<project>/.code-crawler/index.db	Database path. Resolved from the cwd upward, so each project gets its own file; falls back to ~/.code-crawler/index.db when no project encloses the cwd. The home directory is only that fallback, never an enclosing project
CODECRAWLER_EXCLUDE_DIRS	.git,node_modules,dist,build,target,...	Dirs skipped during crawl
CODECRAWLER_DEFAULT_LIMIT	50	Default result cap
CODECRAWLER_MAX_DEPTH	10	Traversal depth cap (1–100)
CODECRAWLER_MAX_FILE_BYTES	1000000	Files larger than this are skipped, not parsed
CODECRAWLER_RESPECT_GITIGNORE	true	Honor the project's .gitignore during discovery
CODECRAWLER_HELPFUL_ERRORS	true	Fuzzy suggestions on symbol misses
CODECRAWLER_PLUGIN	unset	Plugin mode: resolve one database per project at call time instead of opening one file at startup
CODECRAWLER_PLUGIN_ROOT	unset	Plugin install directory, never indexed as a project
CODECRAWLER_PLUGIN_DATA	temp dir	Writable directory for the no-project fallback database

Invalid values never crash the server: booleans fall back to their default and integers are clamped to their documented range, each with a one-line warning on stderr.

What gets indexed

crawl walks the project and skips .git, node_modules, build output, and other CODECRAWLER_EXCLUDE_DIRS entries. On top of that:

• the project's .gitignore is honored (disable with CODECRAWLER_RESPECT_GITIGNORE=false),

• an optional .crawlerrc in the project root can narrow or widen discovery with gitignore-style globs, and relocate where module ids start:

  {
    "include": ["src/", "packages/core/"],
    "exclude": ["**/generated/"],
    "moduleRoots": ["src"]
  }
  

  When include is non-empty, only matching files are indexed. A malformed .crawlerrc is ignored with a warning rather than failing the crawl.

  moduleRoots is opt-in: with ["src"], src/orders/service.ts becomes the module orders.service instead of src.orders.service, which is what a baseUrl-or-PYTHONPATH project actually imports. It is never inferred, because inferring it would silently renumber an existing index. TypeScript and JavaScript projects also get path aliases for free: compilerOptions.paths and baseUrl are read from tsconfig.json (or jsconfig.json), following extends within the project, so import { x } from "@app/orders/service" resolves to the real module instead of being recorded as external. Changing either setting re-parses cached files on the next crawl.

• files above CODECRAWLER_MAX_FILE_BYTES and files that look binary are skipped; crawl_project reports them in skippedFiles with a reason.

• per-file parse failures never abort the crawl; they are reported in parseErrors.

Only the root .gitignore is read — nested .gitignore files are not yet supported.

Incremental crawls

A crawl records a manifest row per file (size, mtime, SHA-256, language). On the next crawl a file whose size and truncated mtime both match is served from a cached parse result instead of being re-read and re-parsed; a file whose mtime changed but whose hash did not is also replayed from cache. Resolution, phantom-node creation and derived edges still run over the whole in-memory file set, so cross-file edges stay correct — only I/O, parsing and database writes are skipped.

crawl_project reports the split:

Field	Meaning
filesAdded	First seen by this crawl
filesUpdated	Content changed, re-parsed
filesUnchanged	Replayed from the parse cache
filesRemoved	Disappeared from disk and were pruned

Known limitation: because an unchanged file's edges are not rewritten, a call that becomes newly resolvable only because some other file changed (for example a method gained an overload, or a class was introduced that a previously-ambiguous call could now bind to) will not be upgraded in the untouched file. Touch that file, or run clear_index, to force full resolution.

Benchmarks

npm run bench generates a synthetic TypeScript project (575 modules, ~137k lines, ~9.8k methods), crawls it, and reports crawl phases and query latency percentiles. tests/perf.test.ts asserts the same budgets and is opt-in, because a realistic corpus takes time to crawl:

npm run bench                                        # human-readable report
CODECRAWLER_BENCH=1 npm test -- tests/perf.test.ts   # enforce the budgets
CODECRAWLER_BENCH_LINES=20000 npm run bench          # smaller corpus

Measured on a Windows dev laptop with a warm disk cache; absolute numbers vary, the ratios and the index usage are what matter:

Phase	Duration	Added	Updated	Unchanged
Cold crawl	22.5 s	575	0	0
Re-crawl, nothing changed	0.91 s	0	0	575
Re-crawl after a one-file edit	0.75 s	0	1	574

Query	p50	p95	Budget (p95)
search_symbols (exact)	0.69 ms	1.96 ms	50 ms
search_symbols (prefix)	1.60 ms	4.09 ms	50 ms
get_callers (depth 2)	0.16 ms	0.21 ms	50 ms
get_callees (depth 2)	0.11 ms	0.15 ms	50 ms
get_subgraph (radius 2)	0.16 ms	0.27 ms	50 ms
find_entry_points	0.12 ms	0.29 ms	50 ms
find_hotspots	145.5 ms	181.4 ms	1000 ms
find_dead_code	99.8 ms	290.3 ms	1000 ms

The index is 66 MB for those 14.4k nodes / 38.8k edges. find_hotspots and find_dead_code are whole-project aggregations over every edge and node, which is why their band is an order of magnitude looser than the symbol lookups; both were profiled with EXPLAIN QUERY PLAN and are served by idx_edges_target_kind.

Agent cost: graph versus grep-and-read

npm run bench:ergonomics measures ten structural questions (who calls this, what runs on this endpoint, what breaks if I change this, what is dead) on the four pinned benchmark projects, against the real MCP server and against a model of text search plus reading every file a search hit. Across the ten, the graph answered in 12 tool calls / 22,907 tokens where grep-and-read cost 120 calls / 105,758 tokens — and 8,553 tokens with compact: true. Reads are capped at 20 files and the dead-code baseline is handed the answer set for free, so the baseline is under-charged.

Where plain search wins is measured too: a question about template text or a config key has no graph answer, and a one-file lookup is cheaper as a search. Use the graph for structure, search for text. The full comparison, method and caveats are on the accuracy page.

CI & publishing

GitHub Actions runs on every pull request (pull-request.yml): npm ci → typecheck → tests → build → smoke-test of the built CLI.

Merges to main trigger publish.yml, which re-runs the same checks and then, if green, auto-releases:

1. Reads the most recent v* tag (or package.json version on the first run), computes the next patch version, and pushes a vX.Y.Z tag.
2. Creates a matching GitHub release.
3. Publishes the package to the npm registry with provenance (OIDC trusted publishing) — no token or repository secret is needed. The step is idempotent and retries: it skips versions npm already has, and a transient transparency-log or registry failure no longer aborts the release.
4. Publishes server.json to the official MCP Registry as io.github.nithin554/code-crawler from a separate, re-runnable job using ExaDev/mcp-registry-publish. The job runs inside the mcp-registry-publish environment, whose deployment branch policy is limited to main. It pins server.json to the release version, authenticates with GitHub OIDC, skips versions the registry already lists, and retries transient npm propagation or registry errors. Ownership is verified through the mcpName field in package.json.

Using as an MCP server

Install as an agent plugin

The repository is also an agent plugin, so Claude Code and Copilot CLI can install the server and the skill together:

# Claude Code
/plugin marketplace add nithin554/code-crawler
/plugin install code-crawler@code-crawler

# Copilot CLI
copilot plugin marketplace add nithin554/code-crawler
copilot plugin install code-crawler@code-crawler

The plugin runs npx -y @nithin554/code-crawler serve. Agent Plugins clients start a plugin's stdio servers inside the plugin directory, so the server does not treat that directory as your checkout: it resolves a project from crawl_project, the client's workspace roots, or the project this session already selected, and keeps that graph in <project>/.code-crawler/index.db — one database per repository, the same layout the CLI uses. CODECRAWLER_DB still pins a single file when you want one.

The same skill is published to ClawHub as code-crawler, and the plugin is published there too:

• Skill — clawhub install code-crawler. ClawHub assigns this version by bumping the patch digit of whatever it already published, so it advances one release at a time instead of copying the npm version; a release whose skill publish fails leaves the two numbers one apart until the next publish.
• Plugin — published as the code-crawler bundle plugin, versioned from the release tag: release v0.1.67 publishes the bundle at 0.1.67. The bundle's own plugins/code-crawler/package.json is packaging metadata — ClawHub needs a package manifest to inspect the bundle and build its ClawPack artifact, and warns when one is missing — so the release workflow stamps the tag's version into it, packs the bundle, and publishes that tarball; the committed version is only a placeholder.

The plugin publish runs the official clawhub CLI itself, under Node, rather than calling upstream's reusable package-publish.yml. That workflow cannot publish this bundle: it inspects the packed artifact with bun ... package validate, and the CLI refuses to run under Bun ("Plugin validation requires Node.js 22.13+ (22.x) or 24+"). A ClawPack always contains package/package.json and no input skips the step, so the call only ever succeeded while the bundle had no manifest — the very thing ClawHub warns about.

Publishing is restricted to main. The publishing jobs deploy through the clawhub-publish environment, whose deployment branch policy rejects any other ref, so a manual dispatch from a feature branch stops before it can upload. CLAWHUB_TOKEN is a repository secret, not an environment secret: the skill call hands it to upstream, which reads it as clawhub_token, and a job that calls a reusable workflow cannot declare an environment. Forwarding the value through a job output does not help either — GitHub refuses to emit a secret-valued output and logs Skip output 'clawhub_token' since it may contain secret.

One command

add-mcp registers the server in whichever client you name — Claude Code, Cursor, VS Code, Claude Desktop, Codex, OpenCode, Zed, and more (npx add-mcp list-agents). Run it from inside the repository you want indexed:

npx add-mcp @nithin554/code-crawler --args serve -a cursor

That writes project-scoped config next to the code (.mcp.json, .cursor/mcp.json, .vscode/mcp.json, …), so each repository carries its own registration and its own .code-crawler/index.db. The server name code-crawler is derived from the package.

--env CODECRAWLER_DB is optional — add it only to put the database somewhere else: --env CODECRAWLER_DB=/absolute/path/to/index.db. -g writes user-level config instead. Prefer that only for clients that launch the server with your checkout as its working directory: with -g and an unrelated working directory (Claude Desktop starts in its own install folder) the server falls back to ~/.code-crawler/index.db, and every repository then competes for that one file.

One server per project

Outside plugin mode a server process opens one database at startup and a database holds exactly one project, so the model is one registration per repository — which is what the per-project config above gives you. The plugin install instead resolves a database per project at call time, so one plugin serves every repository you open. Re-crawling a repository reuses its existing file and only re-parses what changed; two repositories never share a file, so they cannot clash. Only a few project databases stay open at once (idle ones are closed), and every handle is released when the client disconnects.

MCP Registry / npx

The server is listed in the MCP Registry as io.github.nithin554/code-crawler. Any npx-capable client can run it without a local checkout:

{
  "mcpServers": {
    "code-crawler": {
      "command": "npx",
      "args": ["-y", "@nithin554/code-crawler", "serve"]
    }
  }
}

Claude Desktop

claude_desktop_config.json:

{
  "mcpServers": {
    "code-crawler": {
      "command": "node",
      "args": ["/absolute/path/to/code-crawler/dist/index.js", "serve"],
      "env": { "CODECRAWLER_DB": "/absolute/path/to/your/index.db" }
    }
  }
}

Cursor / other stdio MCP clients

Point the client at the same command. The typical flow for an LLM agent:

1. crawl_project the repo once.
2. find_entry_points to see where execution actually starts (routes, listeners, mains).
3. search_symbols to locate the symbol of interest.
4. get_callers / get_callees / trace_path to reason about the call graph.
5. trace_execution to follow runtime dispatch and aspect advice; find_aspects, get_decorators, and find_annotated to inspect behaviour-changing annotations.
6. find_routes, find_injections, find_reflections, and find_external_dependencies to map framework wiring and third-party surface.

See USAGE.md for the full task-oriented walkthrough.