합성 및 표준 SQL 벤치마크를 사용하여 데이터베이스 성능을 검증하기 위한 결정론적 오케스트레이션 하네스를 제공합니다.
다른 이름 rctruta/sql-benchmarks-dagster, sql-benchmarks-dagster, sqlbenchdag
- 스타
- 3
- Tools
- —
- 다운로드, 30일
- 279
- 최신 버전
- 0.1.2
- 마지막 커밋
- 2026년 10월 28일
- 라이선스
- Apache-2.0
SQL Benchmarks Lab MCP란?
합성 및 표준 SQL 벤치마크를 사용하여 데이터베이스 성능을 검증하기 위한 결정론적 오케스트레이션 하네스를 제공합니다.
프로젝트 README에서 생성된 요약
SQL Benchmarks Lab MCP server 정보
SQL Benchmarks Lab은 재현 가능한 과학을 통해 데이터베이스 성능을 검증해야 하는 개발자와 엔지니어를 위해 설계된 결정론적 오케스트레이션 하네스를 제공합니다. 이 시스템은 메커니즘 실험 중 변수를 격리하기 위해 합성 및 표준 TPC-H 데이터를 사용합니다. 구성, SQL 로직 및 측정 코드를 기반으로 생성된 8자리 SHA-256 해시인 Experiment ID를 통해 쿼리 튜닝을 반복 가능한 프로세스로 변환하는 전문 실험실 역할을 합니다.
사용자는 sqlbench CLI를 사용하는 명령줄 인터페이스나 Dagster 기반의 자동화된 워크플로우를 통해 플랫폼과 상호작용합니다. 설정에는 Python 3.11 이상과 의존성 관리를 위한 uv가 필요합니다. Postgres 또는 TypeDB와 같은 컨테이너화된 엔진의 경우 Docker가 필요하지만, DuckDB 사용자는 Docker 없이도 퀵스타트를 수행할 수 있습니다. 이 하네스는 쿼리 실행 전 엔진 레벨의 버퍼를 확실히 제거하기 위해 Postgres는 Docker 컨테이너를 재시작하고 DuckDB는 새로운 워커 프로세스를 사용하는 방식으로 다층 콜드 캐시(cold-cache) 격리를 관리합니다.
활용 사례
- 1SQL 성능 가설 검증개발자는 이 하네스를 사용하여 직관적인 쿼리 튜닝을 재현 가능한 과학적 실험으로 전환할 수 있습니다.
- 2병렬 벤치마크 매트릭스 실행엔지니어는 YAML에서 N차원 매트릭스를 정의하여 독립적인 Dagster 파티션 전체에서 벤치마크를 실행할 수 있습니다.
- 3깨끗한 캐시 측정 보장연구자는 메모리 버퍼가 IO 바운드 성능 결과를 가리는 것을 방지하기 위해 엔진 레벨의 버퍼 제거를 트리거할 수 있습니다.
- 4자율 에이전트와 통합AI 에이전트는 Experiment ID를 사용하여 결정론적 성능 API로서 실험실과 상호작용할 수 있습니다.
프로젝트의 README를 바탕으로 작성되었습니다. 모든 활용 사례는 해당 내용의 인용문과 대조하여 확인합니다.
SQL Benchmarks Lab MCP 설치 방법
형식 확인: 2026년 11월 10일Cursor에 추가Goose에 추가Claude Code에 SQL Benchmarks Lab MCP 설치
$ claude mcp add sql-benchmarks-dagster -- uvx sqlbenchdagCursor에 SQL Benchmarks Lab MCP 설치
~/.cursor/mcp.json
{
"mcpServers": {
"sql-benchmarks-dagster": {
"command": "uvx",
"args": [
"sqlbenchdag"
]
}
}
}VS Code에 SQL Benchmarks Lab MCP 설치
.vscode/mcp.json
{
"servers": {
"sql-benchmarks-dagster": {
"type": "stdio",
"command": "uvx",
"args": [
"sqlbenchdag"
]
}
}
}VS Code CLI에 SQL Benchmarks Lab MCP 설치
$ code --add-mcp '{"name":"sql-benchmarks-dagster","type":"stdio","command":"uvx","args":["sqlbenchdag"]}'Claude Desktop에 SQL Benchmarks Lab MCP 설치
claude_desktop_config.json
{
"mcpServers": {
"sql-benchmarks-dagster": {
"command": "uvx",
"args": [
"sqlbenchdag"
]
}
}
}Codex에 SQL Benchmarks Lab MCP 설치
$ codex mcp add sql-benchmarks-dagster -- uvx sqlbenchdagCodex config.toml에 SQL Benchmarks Lab MCP 설치
~/.codex/config.toml
[mcp_servers.sql-benchmarks-dagster] command = "uvx" args = ["sqlbenchdag"]
Gemini CLI에 SQL Benchmarks Lab MCP 설치
$ gemini mcp add -s user sql-benchmarks-dagster uvx -- sqlbenchdagGemini settings.json에 SQL Benchmarks Lab MCP 설치
~/.gemini/settings.json
{
"mcpServers": {
"sql-benchmarks-dagster": {
"command": "uvx",
"args": [
"sqlbenchdag"
]
}
}
}구성
SB_API_BASEOptional. Base URL of a running SQL Benchmarks REST API. Leave unset and the read tools mount the API in-process — no server needed. Set it only when the API runs elsewhere, or to use submit_experiment, which needs a process that outlives the request.
SB_RESULTS_DIRDirectory of experiment result capsules. The package ships the SQL suites, taxonomy and templates but NOT capsules, which are run output — so on a fresh install the catalog tools work and list_results is empty until this points at a results directory.
| 이름 | 설명 | 유형 |
|---|---|---|
| SB_API_BASE | Optional. Base URL of a running SQL Benchmarks REST API. Leave unset and the read tools mount the API in-process — no server needed. Set it only when the API runs elsewhere, or to use submit_experiment, which needs a process that outlives the request. | |
| SB_RESULTS_DIR | Directory of experiment result capsules. The package ships the SQL suites, taxonomy and templates but NOT capsules, which are run output — so on a fresh install the catalog tools work and list_results is empty until this points at a results directory. |
자동 검사
2026년 11월 13일에 실행됨전체 보고서 →자동 검사를 통과했습니다. 이는 보안 감사가 아닙니다. 코드를 직접 읽고 서버에 필요한 최소한의 권한만 부여하십시오. 검사 방식
- 소스 저장소 연결 및 접근 가능통과
- 오픈 소스 라이선스 (OSI)통과
- 최근 1년 내 커밋 또는 릴리스통과
- 패키지 이름이 인기 패키지와 유사하지 않음통과
- 원격 엔드포인트가 HTTPS 사용검사되지 않음
- 파괴적인 도구에 라벨 표시됨검사되지 않음
- 모델에 대한 숨겨진 지침 없음통과
- 패키지 및 해당 종속성에 중요한 보안 권고 없음통과
README
GitHub에서 보기SQL Benchmarking Laboratory
A deterministic, orchestrated harness for verifying database performance at scale. Developed by Ramona C. Truta
The Mission: "Ground Truth" as Code
This platform is a specialized laboratory for testing SQL performance hypotheses. It transforms query tuning from intuition into a reproducible science.
The core of the system is a Deterministic Orchestration Harness that guarantees that if the logic or the environment changes, the benchmark result changes. If they do not, the result is addressable and cached.
Scope: the focus to date is synthetic and canonical (TPC-H) data, which is the right instrument for mechanism experiments — where controlled, reproducible data isolates the variable under test. Real-data support exists but is experimental; see the FAQ for the synthetic-vs-real rationale, the container model, and the roadmap (AI-security testbed, real-data trust chain).
Key Features & Innovations
1. Context-Aware Semantic Hashing (The Experiment ID)
The Heart of the system is the Experiment ID, an 8-character hash that governs the entire lifecycle. This hash is a SHA-256 fingerprint generated from:
- The Config: Every dimension in your YAML (rows, skew, parameters).
- The SQL Logic: The actual content of the benchmarked scripts.
- The Code: All measurement-relevant Python — orchestration (
assets/), engine clients (resources/), and data generators (plugins/).
Semantic Normalization: The hashing engine distinguishes between a logic change and a formatting change.
- SQL: Comments, whitespace, and case are normalized before hashing.
- Python: Orchestration scripts are parsed into an Abstract Syntax Tree (AST) to strip docstrings and formatting variations, ensuring the Experiment ID only changes when execution logic changes.
2. Multi-Layer Cold-Cache Isolation
To ensure IO-bound performance is not masked by memory buffers, engine-level buffers are cleared before every query:
- Postgres: Mandatory Docker Container Restarts clear shared buffers.
- Versioned DuckDB (
duckdb_1_5,duckdb_2_0): every query runs in a fresh worker process, so DuckDB's buffer pool starts empty.
OS page cache: not evicted. Engines call thrash_os_cache, which dirties an anonymous memory region (half of available RAM, capped at 4 GB). On macOS this was measured to leave a cached 1 GB file fully resident, so measurements should be read as warm at the OS level. DuckDB clients take a required cold argument, and cold=True raises until a verified eviction step exists.
3. Agentic AI Integration
The platform is built for the future of Autonomous Engineering. The Experiment ID allows AI agents to treat the laboratory as a Deterministic Performance API.
- See AGENTS.md for the full Agentic Benchmarking Protocol.
4. Declarative Matrix Orchestration
Benchmarks are defined as N-dimensional matrices in YAML. The platform expands these into a Cartesian product of Independent Dagster Partitions. This allows for parallel dispatch and granular retries.
Usage & Technical Setup
Prerequisites
- uv: the project's Python environment & dependency manager (fast, modern).
setup.shuses it, and it can install Python 3.11 for you. - Python 3.11+: core runtime (uv provisions it if missing).
- Docker: for the containerized engines (Postgres, TypeDB), which the harness manages itself — not required for the DuckDB-only quickstart.
Installation & Setup
From PyPI — the lab is installable, which gives you the tooling and the sqlbench CLI:
pip install sqlbenchdag
From source — clone the repo to also get the published capsules (sql_benchmarks/experiments/results/) and the full harness. The laboratory includes a comprehensive setup script that manages virtual environments, dependencies, and directory initialization.
# 1. Automate Setup
chmod +x setup.sh && ./setup.sh
# 2. Activate Laboratory
source venv/bin/activate
The Execution Workflow (CLI)
While the system is powered by Dagster, the primary interface is the CLI for automated workflows.
# QUICKSTART: DuckDB only, no Docker required — runs in seconds
./run.sh sql_benchmarks/experiments/queue/quickstart.yaml --auto
# FULL BENCHMARK: Run all queued experiments (requires Docker for Postgres)
./run.sh queue --auto
# SINGLE EXPERIMENT: Pass a specific config path
./run.sh sql_benchmarks/experiments/queue/quack_execution_modes.yaml --auto
To write your own experiment, start from the annotated template and see the catalog of published runs in docs/experiments.md.
Project Structure
Managed with a strict separation between Harness and Scenario:
/sql_benchmarks
├── /assets # THE HARNESS: Dagster factories & pipeline logic
├── /resources # THE INFRASTRUCTURE: DB drivers & Docker management
├── /scripts/sql # THE SCENARIOS: Raw SQL partitioned by scenario
├── /plugins # THE DATA: Declarative generators & scenario providers
├── /utils # THE BRAIN: AST-hashing, common logic, & system primitives
└── /experiments # THE LABORATORY
├── active.yaml # LOCAL runtime staging — the coordinator overwrites
│ # this on every run (gitignored; not shared state)
├── /queue # Staging area for new experiment configs
├── /archive # Library of previously defined experiment templates
├── /configs # Registry of immutable, hash-addressed experiment capsules
└── /results # Data capsules: fragments, CSVs, and Dashboards
Published Experiments — the Quack investigation
The first published study with this lab: an independent measurement of DuckDB's Quack client-server protocol (beta, v1.5.3). Each row links the runnable config to its committed, verifiable capsule. Full numbers, scaling exponents, and verification steps in docs/published_capsules.md.
| Act | Question | Config | Capsule | Finding |
|---|---|---|---|---|
| I | What does the protocol cost? | quack_execution_modes.yaml | b8e2bfaf |
Attach-mode overhead grows with scan size (2.6× → 9.5× at 10M rows); pushdown stays flat at ~2×. |
| II | Why is pushdown ~2× and not 1×? | quack_residual_threads.yaml | 25b0e134 |
The residual tracks reduced server-side parallelism (~2–4 of 8 threads), not transport. |
| III | Does it generalize to joins? | tpch_quack_validation.yaml | b198363e |
On canonical TPC-H Q3, pushdown holds (~1.9×); attach mode cannot run the join at all (DNF). |
| IV | Does it beat the incumbent? | quack_vs_postgres.yaml | 902d1277 |
DuckDB-over-Quack (pushdown) beats PostgreSQL by up to 13.2×, and the gap widens with scale. |
Act 0 (b82b4eae) is the exploratory scout that started it — see docs/published_capsules.md.
The lab also ships scenario suites for other studies — selectivity/,
null_logic/, null_sentinel/, recursion/, sort_spill/, tpch/, and more
under sql_benchmarks/scripts/sql/ — each runnable across the engine matrix.
Naming & Provenance
The lab works at two layers, named two different ways. Keeping them distinct is the difference between a number you can trace and a number you have to trust.
| Layer | What it is | Named by | Example |
|---|---|---|---|
| Capsule | one experiment's complete results | its Experiment ID — an 8-char SHA-256 of the question (config + SQL + measurement code), assigned by the machine | 48c92f31 |
| Release | a signed set of capsules backing a publication | a human name: sqlbenchdag-<topic>-v<N>-<YYYYMMDD> |
sqlbenchdag-quack-v1-20260614 |
A release contains capsules; a capsule is one experiment. The machine names
capsules (hashes you can't forge); the author names releases (words that mean
something). sqlbenchdag is the lab's maker's mark — sql + bench +
dag (Dagster, the orchestration that distinguishes this lab from a bare query
timer). It is carried on every release this lab produces.
Two hashes, two jobs (a capsule contains both — they are not the same thing):
- the Experiment ID (8 chars) is an address — it names the capsule, and the same question always produces the same address;
- the integrity seal (64 chars, in
integrity.seal) is a checksum — it verifies the capsule's contents are untouched. You file by the address; the seal is the tamper-evident lid.
Reproducibility is pinned to the release. Because the Experiment ID hashes the measurement code, it is stable only at a fixed code revision — which is exactly what the signed release tag freezes. To reproduce a published capsule, check out its release tag and re-run its config; the same question yields the same ID, or the comparison is refused by construction.
Provenance roadmap: a per-capsule
generatorstamp (sqlbenchdag@<commit>) will record the exact build that produced each capsule. It writes through the finalization code, which lives inside the Experiment-ID hash, so it ships with a future release rather than re-numbering capsules already published.
See docs/published_capsules.md for the full trust chain (reproducibility, integrity, timestamp, authorship) and how to verify each. Reproducibility and integrity are automatic on every run; timestamp and signature are optional steps you add only when publishing — the producer workflow is docs/PUBLISHING.md.
License
Copyright 2025-2026 Ramona C. Truta. Licensed under the Apache License 2.0.
Built with Dagster, Polars, Docker, DuckDB, and Postgres.
SQL Benchmarks Lab MCP 질문
SQL Benchmarks Lab MCP를 Claude Code에 추가하려면 어떻게 하나요?
터미널에서 claude mcp add sql-benchmarks-dagster -- uvx sqlbenchdag를 실행합니다. 모든 프로젝트에서 사용할 수 있게 하려면 --scope user를 추가합니다.
SQL Benchmarks Lab MCP를 Cursor에 추가하려면 어떻게 하나요?
~/.cursor/mcp.json를 열고 mcpServers 아래에 서버를 추가합니다. 위 설치 섹션에 표시된 Cursor 구성을 사용합니다.
시스템에서 데이터베이스 캐시 격리를 어떻게 처리하나요?
Postgres의 경우 Docker 컨테이너 재시작을 사용하며, DuckDB의 경우 새로운 워커 프로세스를 사용하여 엔진 레벨의 버퍼를 제거합니다.
AI 에이전트가 이 플랫폼을 사용할 수 있나요?
네, 이 플랫폼은 자율 엔지니어링을 위해 설계되었으므로 AI 에이전트가 Experiment ID를 결정론적 성능 API로 사용할 수 있습니다.