codeindex MCP server

codeindex

Self-contained, deterministic repo-indexing engine: file walking, language detection, symbol/import extraction (tree-sitter AST with a regex fallback), import resolution, a typed cross-file link-graph, and graph analytics — shipped as a single zero-dependency engine.mjs that consumer tools vendor (copy into their repo) instead of installing.

Designed for downstream tools — agent skills, CLIs, CI gates — that vendor the engine as a single file instead of taking an npm dependency. How it stacks up against universal-ctags, Serena and Graphify: How it compares.

What it does

• Walk a repo deterministically: ignore lists, .gitignore and .git/info/exclude, binary/lockfile skips, a size cap, symlink-cycle guard. A symlink that stays inside the repo, file or directory, is an alias: its target is indexed once, under its own path. build, out, target and tmp are skipped as build output unless git tracks files in them — they are ordinary package names too (a Go build package, com.acme.build). Nested repositories (a subdirectory with its own .git — linked worktrees, vendored clones, submodules) are skipped like git does, and .git itself — like the engine's own .codeindex — is never walked even when --ignore-dir replaces the default ignore list. No file-count cap unless you ask for one (--max-files), and asking sets the capped flag — never a silent truncation.
• Scan every file into a FileRecord: classification, language, symbols, imports, headings, hashes — with an incremental cache fastpath. Extraction runs across worker threads by default (--workers, CODEINDEX_WORKERS); artifacts are byte-identical either way, and anything that would make a worker's result differ falls back to the single-threaded path. JS/TS and Python imports are read from code only: comments, docstrings and string/template-literal text are masked first, so example code quoted in a JSDoc block, a docstring or a code generator's template never becomes an edge (JSDoc import("./x") types and @import tags, which are real type dependencies, are kept). The same scan runs with or without a grammar, so extractAst and the index report the same imports. Python from pkg import name also links pkg/name.py when name is a submodule (and nothing when it is a function or class); PHP group (use A\{B, C}) and comma use lists and __DIR__-anchored includes are followed, and a trait use inside a class is not an import. Build output committed under an ordinary name is recognised by its content: minified JavaScript (not only a .min.js name) and bundles (esbuild's // src/x.ts module banners, webpack's and ncc's module loader). It stays in the index with its summary and imports, flagged generated: "minified" or "bundle" on its FileRecord and graph node, but its symbols and call sites are not extracted: one-letter noise for the first, copies of the sources' definitions for the second.
• Extract symbols via tree-sitter (15 committed grammars, plus 6 more via grammars pull) or per-language regex rules (16 languages, always available). Each symbol carries its complete signature (parameters and return type, not the first physical line; one line, with no comment and no body — not an arrow's expression body, a Go interface's method list or a macro's expansion), its own doc comment, its qualified parent, and its line span — including the members a declaration-only walk misses: interface members, class fields, enum members, every declare/.d.ts declaration, Rust trait method signatures, Go interface method sets and type aliases, record components, constructor val parameters and their TypeScript (private readonly dep: Dep) and PHP 8 (promoted) twins, every name of a multi-name declaration (var a, b int, int x, y;, a, b = 1, 2), C #define macros and the members of a typedef struct, Python declarations under if TYPE_CHECKING: / try: / with blocks, Ruby class << self methods, private def x definitions and the block of Point = Struct.new(…) do, Elixir clauses with a when guard and defguard, and the members of a class bound by module.exports = or an anonymous export default class (a default export with no name of its own is named after the file stem). A doc comment is found across Rust attributes and TypeScript decorators. Visibility is read from a declaration's modifiers, never from its parameter names or default values; an export { … } list marks only the bindings of its own scope; and a Python module's __all__ (when written as literals) decides which of its top-level names are public, the names it imports and lists becoming reexport symbols. An out-of-line C++ definition (void Widget::draw()) belongs to its class, and a Lua function M.go() to its table; a .h header is parsed as C++ when its content is (a namespace, class or template), as C otherwise. Vue, Svelte and Astro single-file components are extracted from their <script> blocks (and Astro's frontmatter) as the JS/TS their lang names, at their real lines: symbols, imports, and calls from both the script and the template, bound in the JS/TS call family. A Svelte prop (export let) and an Astro frontmatter export are not module exports, so they are never reported as dead code. The regex tier (the only one for Swift and Dart, and for the extended languages until a pull) reads code only: comments and strings are masked, so an example in a doc comment or a code generator's template is not a declaration. It takes each declaration's doc comment from the lines above it and, in brace languages, its line span, but only where the body's braces close as a formatter puts them; otherwise the span is left out rather than guessed. Its signature is the declaration's first line, and it reports no parent. Each file's summary is the first leading comment that describes something: license and copyright text (MIT, BSD, Apache, GPL, MPL, the Go "governed by" line), linter and editor magic comments (frozen_string_literal, -*- coding -*-, go:build), Xcode's file stamp and bundler region markers are skipped, and # reads as a comment only in languages where it is one — never a C #include or a Rust #[attribute]. Docs get a title, section headings, a summary and doc-link refs: markdown (ATX and setext headings, inline and reference links) and reStructuredText (Sphinx section titles; toctree entries, :doc: roles and include/literalinclude targets as links). Other prose (.txt, .adoc) is indexed under its file name.
• Resolve imports across languages: tsconfig paths (tsc's precedence: exact alias, then longest prefix) and baseUrl, extends chains into workspace packages and ${configDir}, package exports and imports (#subpath), bundler ?query suffixes, Python import roots found the way mypy finds them (the dir holding each top-level package, so src layouts resolve and a package's own typing.py does not shadow the stdlib), go.mod (a package's representative file is never a _test.go when it has other files), Cargo ([lib] names, renamed dependencies, #[path] modules, and use paths read the way the crate's edition reads them), PSR-4, C# namespaces, and one JVM index for Java, Kotlin and Scala, so a Kotlin file importing a Java class (or the reverse, through the <File>Kt facade) links; Scala selector groups and package-relative imports resolve too. Dart (relative and package: URIs of an in-repo pubspec.yaml), Lua require (a/b.lua, a/b/init.lua), shell source/. of a literal path, and Elixir alias/import/use of a module the repo defines resolve as well. A markdown link starting with / is repo-root-relative, as GitHub renders it.
• Build a typed link-graph: import / call / extends / implements / use / doc-link / mention edges at file and module level, plus Louvain communities, PageRank/betweenness centrality, a tests→code map (a test covers what it imports, uses or calls, the file it is named after, and in Go its whole package), and surprise-edge detection. Inheritance also yields a type hierarchy (what a type extends and implements, and what extends and implements IT) and a symbol-level graph for bounded "what does this reach" neighborhoods. Go states no implementations, so the hierarchy adds them: a var _ I = (*T)(nil) assertion, or a type whose methods (its own, plus the ones embedding promotes) match the interface's by name and parameter count. The second kind is marked structural: true. These are answers to hierarchy and implementations only, never graph.json edges.
• Render byte-stable graph.json / symbols.json (two builds of an unchanged repo are byte-identical), plus a SCIP code-intelligence index (index.scip: nested symbols, package identity, implementation relationships) via a hand-rolled zero-dependency protobuf encoder — validated by the official scip CLI (stats/lint).

Measured against other indexers

"Finds better" is a claim, so the checks that count are the ones this project did not author. Four oracles score extraction against outside authorities — a real compiler, a mature indexer, and the grammars' own published queries and vocabulary:

oracle	what makes it independent	result
TypeScript compiler index (scip-typescript 0.4.0)	an index built by the real TypeScript compiler — authoritative where every other check here is syntactic	100% of its 93 named declarations, against ctags' 94.6% on the same files
universal-ctags differential (Universal Ctags 6.2.1)	an independent, mature indexer covering ~40 languages	reports 2,014 declarations ctags does not over 6 real repositories, and reproduces 61.7%–98.8% of ctags' names — what is left bucketed by kind, per repo below
Official tags.scm queries	the code-navigation patterns each grammar's own authors publish, and GitHub uses	1 adjudicated difference, over the 14 of 17 languages that publish one
Grammar vocabulary	each tree-sitter grammar's own declared node types, read at runtime from the parser	21 grammars audited, 209 declaration-ish node types still unhandled

The one head-to-head

Exactly one figure on this page is a score: the one where both tools are measured against the same third-party authority, rather than against each other. On the 53 files of create-t3-turbo that an index built by the real TypeScript compiler covers:

against the compiler's 93 named declarations	found
codeindex	100%
universal-ctags	94.6%

All 5 ctags missed are one construct (string-literal declaration names — module augmentations, quoted interface keys). That head-to-head is also the calibration for everything below: it is how far a syntactic oracle can be trusted on the ~40 languages no compiler here can check.

Every other percentage in this section is an overlap ratio between two tools that disagree about what counts as a declaration, which is a different thing and is not scored as one.

Per repository, against universal-ctags

Declaration names compared per file over real code, not fixtures. The percentage here is not a score, which is why it is not in the last column. It is |ours ∩ ctags| / |ctags| — the share of ctags' names this index also reports — so by construction it can only ever show where we lose: nothing in it measures what ctags omits. The two count columns are the directions that actually compare the tools; read those.

repo	files	both report	of ctags reproduced	ctags only	codeindex only
BurntSushi/ripgrep	107	3,189	98.8%	40	47
gin-gonic/gin	100	2,010	98.6%	29	15
pallets/flask	86	1,516	93.9%	98	6
t3-oss/create-t3-turbo	54	97	76.4%	30	15
nrwl/nx-examples	87	87	69.6%	38	27
socialgouv/code-du-travail-numerique	1,429	3,659	61.7%	2,271	1,904

So no, the low rows are not "ctags finds more" — and that is measured, not asserted. The differential records what the ctags only column is, bucketed by the kind ctags itself assigned (ctagsOnlyByKind in the same record). On code-du-travail, its 2,271 names are:

ctags kind	count	what they are
constant	2,020	all but a handful sit inside a function body, an object literal or a test block — read off the source, not assumed
variable	116	same story
property	107	object-literal keys (Conditions: ConditionsIcon)
alias	13	import aliases — import type Engine from "publicodes"
method / class / function / enumerator	15	object-literal methods and test-scope declarations

That is a definition gap, not a hole: a declaration index omits locals and config keys on purpose, which is the whole reason its output fits in a model's context. The same holds on the other repos — ripgrep's 40 are mostly variable and Rust implementation blocks, flask's include 21 that ctags itself labels unknown (its kind for an import alias), gin's are its synthetic anonMember/packageName.

And where a third tool can settle it, it does — create-t3-turbo is the 76.4% row above, and it is also the head-to-head at the top of this section, the one the real TypeScript compiler adjudicates 100% to 94.6% in our favour. A row that looks like a loss against ctags is a row the authority scores as a win over ctags. That is the whole reason the percentage is not in the score column.

And the residue is what the differential is genuinely for. Where it named real misses they were fixed, not explained away: Go package clauses, Python PEP 484 re-exports, Rust in-function const/static, and — in EXTRACTOR_VERSION 12 — declarations inside an IIFE, which is why code-du-travail moved to 3,659 here. The honest limit stands: on the languages no compiler-backed oracle covers, nothing proves the rest of that column is entirely surplus.

Refreshed by CODEINDEX_ORACLE=1 pnpm vitest run tests/oracles-external-diff.test.ts and by the weekly CI job; tool version, corpus and date sit next to the figures in tests/quality/external-oracles.json.

Against hand-labelled ground truth

The external indexers report declarations and nothing else, so doc comments, complete signatures and call edges cannot be checked against them at all. Those are covered by labels written here: tests/fixtures/quality/ holds every declaration a correct indexer should report for 17 languages, with its kind, visibility, doc and signature, plus a relevance-judged search corpus whose query terms live only in prose.

what is scored	score	measured on
symbol precision / recall	100% / 100%	346 labelled declarations in 23 files
kind accuracy	100%	the same 346 declarations
visibility accuracy	100% on 16 of 17 languages, 95.8% on Go	the same 346 declarations
doc comment attached	100%	the 188 declarations labelled with a doc
complete signature	100%	the 32 declarations labelled with a signature
call edges / inheritance (F1)	100% / 100%	54 labelled call sites, 24 relations
search MRR / nDCG@10 / recall@5	93.8% / 86.0% / 84.4%	16 relevance-judged queries

pnpm quality:report reproduces every number; tests/quality.test.ts enforces them as a ratchet in both directions — losing quality fails CI, and gaining it fails too until the baseline is refreshed in the same commit. Two builds of an unchanged repo stay byte-identical.

One judged query still returns nothing relevant, and the reason is honest: it asks for "authentication" against a file that never writes "auth" in any form. No lexical index can answer that; the semantic tier is what it is for.

Use as a library (the vendoring model)

Consumers commit scripts/engine.mjs + scripts/engine.d.mts (fetched at a pinned release tag) into src/vendor/ and import from it; their bundler inlines the engine so they still ship a single file:

import { buildIndexArtifacts, renderGraphJson } from "./vendor/engine.mjs";

const { scan, graph, symbols } = buildIndexArtifacts("/path/to/repo");

The AST tier is optional: without a grammars/ directory next to the bundle the engine silently uses its regex tier. Only tools that want AST precision also vendor scripts/grammars/ (~17 MiB of wasm).

Inventory consumers

walk(root, options) also serves exhaustive inventories. The default source-indexing policy is unchanged. Opt into includeBinary, includeLockfiles, includeOversize, includeMinified and includeFileSymlinks (in-repo file links, skipped by default as aliases), or replace binaryExtensions to keep textual SVG. These controls are independent: a .lock extension still follows the binary policy.

filter({ rel, abs, directory }) prunes consumer output and out-of-scope trees before descending. onSkip({ rel, reason, directory, size? }) records observed exclusions, including nested repositories and broken links, without allocating a second inventory for ordinary indexing. It does not enumerate pruned descendants; capped still means that an explicitly requested file budget cut the walk short.

readTextEx(path) distinguishes empty, binary and unreadable files, reports the encoding and original bytes, and marks whether decoded offsets address those bytes. OffsetMap converts JavaScript string positions to UTF-8 byte positions. Consumers keep their own classification and editing policy; readText remains the compatible string-only reader.

Two grammar tiers

tier	languages	how you get it
core (committed)	TypeScript, TSX, JavaScript, Python, Go, Rust, Java, C, C++, C#, Ruby, PHP, Scala, Bash, Lua	ships in the bundle — no network, no install
extended (pull-only)	Kotlin, Elixir, Zig, Solidity, HCL, Terraform	codeindex grammars pull

The extended set is not in git: it adds ~6 MiB of wasm for languages most repos do not contain, so committing it would grow every vendoring consumer's checkout for a benefit only some can use. It ships inside the per-release grammars-<version>.tar.gz asset instead. Without a pull those grammars are simply absent and the engine falls back to the regex tier, exactly as it does for a language it has no grammar for at all — codeindex grammars status reports resolved-vs-missing per tier (and extendedPullNeeded while any extended grammar is missing) so a Kotlin repo quietly indexed by regex is visible rather than guesswork.

Not included, and why: Swift publishes no prebuilt wasm at all, and Dart's does not load under web-tree-sitter 0.26 — shipping it would be dead bytes advertising precision that silently degrades. Both have regex extractors.

Slim grammars (pull instead of vendor)

Consumers that want AST precision but not the ~17 MiB of vendored wasm can codeindex grammars pull the grammars once into a shared, per-machine cache (<XDG_CACHE_HOME|~/.cache>/codeindex/grammars/<ENGINE_VERSION>) instead:

codeindex grammars status   # active tier (adjacent/env/cache/none) + whether a pull is needed
codeindex grammars pull     # fetch the per-release grammars asset, sha256-verified, into the cache

Resolution is adjacent > env > cache > regex, per grammar: a bundle-adjacent grammars/ still wins if present (offline setups are untouched), then CODEINDEX_GRAMMARS_DIR, then the pulled cache — and a grammar the winner lacks is looked up in the tiers below it. That is what lets the npm package (which ships only the core wasms) pick up the extended ones a pull put in the cache. The legacy CODEINDEX_GRAMMAR_DIR still pins one dir with nothing behind it. pull fetches the official grammars-<version>.tar.gz release asset (its .sha256 sidecar is verified before anything is written, and a pull that cannot fetch it fails instead of installing the archive unverified; a CODEINDEX_GRAMMARS_URL mirror carries no checksum) and extracts it atomically; the same wasm bytes produce byte-identical AST extraction from the cache as from a vendored dir. It is fully offline-safe: with no grammars resolvable anywhere — and after a failed or absent pull — the engine silently falls back to the regex tier exactly as it does today; a pull never throws into indexing.

Use from npm

For consumers who don't want to vendor the bundle, @maxgfr/codeindex also resolves as a regular package:

npm i @maxgfr/codeindex

import { scanRepo, ENGINE_VERSION } from "@maxgfr/codeindex";

const scan = scanRepo("/path/to/repo");

The CLI ships in the same package — see Use as a CLI below for the global install command. Consumer tools should still prefer vendoring: it keeps their own bundle single-file and pinned to an exact commit without an npm dependency.