MCP server SonicCraft

SonicCraft

A declarative, deterministic audio engine for AI agents. Describe a production as a YAML (or JSON) timeline: who says what, in which order, over which music, at what loudness. SonicCraft compiles it into TTS speech, music beds and sound effects, mixed with sidechain ducking, mastered to an EBU R128 loudness target, plus WebVTT/SRT subtitles that match the audio it actually rendered. It ships as a CLI, an MCP server and an Agent Skill.

Website · Quickstart · Use it from an AI agent · MCP server · Agent Skill

Use it from an AI agent

Give an agent (Claude Code, Codex, Cursor, Gemini CLI, ...) the link to this repository and ask for audio, for example "Use https://github.com/user-github-me/soniccraft to make a 30-second voiceover with background music for my product video." Everything it needs is below and in AGENTS.md.

1. Install the CLI (prebuilt and checksum-verified, no Rust needed). Homebrew, Scoop, Docker and Cargo work too: see Quickstart.

   curl -fsSL https://raw.githubusercontent.com/user-github-me/soniccraft/main/install.sh | sh   # macOS, Linux
   

   irm https://raw.githubusercontent.com/user-github-me/soniccraft/main/install.ps1 | iex        # Windows
   

2. Write a timeline. soniccraft init my-audio starts one that renders offline: macOS voices on a Mac, elsewhere a mock test voice (not real speech). Real voices anywhere: Kokoro or Piper (free, local) or an ElevenLabs, OpenAI or Cartesia key, see Providers. soniccraft init --list shows the templates, soniccraft providers the voices usable here, and soniccraft schema the JSON Schema; the Agent Skill explains the format.

3. Check it, see the cost, render it:

   soniccraft validate my-audio/timeline.yaml           # errors with a code, line, column and fix hint
   soniccraft plan my-audio/timeline.yaml               # free dry run: timing, ducking, characters to synthesize
   soniccraft render my-audio/timeline.yaml --preview   # mastered audio, subtitles, manifest, preview page
   

To keep SonicCraft in an agent for good, add the skill and the MCP server. Both use the CLI from step 1; with Node.js 20 or newer the MCP server also runs with nothing installed: use npx -y soniccraft-mcp in place of soniccraft mcp below.

Agent	Skill	MCP server
Claude Code	claude plugin marketplace add user-github-me/soniccraft, then claude plugin install soniccraft@soniccraft	included in the plugin, or claude mcp add soniccraft -- soniccraft mcp
Codex	npx skills add user-github-me/soniccraft -g -a codex	codex mcp add soniccraft -- soniccraft mcp
Cursor	npx skills add user-github-me/soniccraft -g -a cursor	{"mcpServers": {"soniccraft": {"command": "soniccraft", "args": ["mcp"]}}} in .cursor/mcp.json
Gemini CLI and others	npx skills add user-github-me/soniccraft -g (choose the agent)	the same mcpServers entry (Gemini CLI: ~/.gemini/settings.json)

The server is also in the official MCP Registry as io.github.user-github-me/soniccraft (container image). Details: MCP server and Agent Skill.

Why

Ask an agent for a two-voice explainer with background music and it will usually hand-write an ffmpeg filter graph: adelay offsets computed from guessed speech lengths, an amix, perhaps a half-remembered sidechaincompress. That fails in predictable ways:

• Timing drift and overlaps. Synthesized speech is never exactly as long as the agent guessed, so lines collide or leave dead air, and every script edit shifts every later offset.
• No ducking. Sidechain compression is fiddly to get right in a filter graph, so the music either drowns the voice or sits uniformly too low.
• Loudness disparity. Different providers and voices come out at different levels. Nothing measures integrated loudness or true peak, so the result is uneven and can clip.
• Drifting subtitles. Captions are written from estimated times instead of from the audio that was rendered.
• Opaque, expensive iteration. A broken graph is a cryptic ffmpeg error, and every retry re-bills every TTS call.

How SonicCraft works

HyperFrames showed that agents make better video when they write declarative HTML and a deterministic renderer does the frame-accurate work. SonicCraft applies the same idea to audio. The agent writes intent as data; a compiler validates it, measures the real durations, resolves all timing in integer sample frames, and renders natively in Rust. The same timeline, TTS cache and binary always produce byte-identical audio, subtitles and manifests.

version: "1.0"
output:
  format: "mp3"
  sample_rate: 48000
  bitrate: "320k"
  target_lufs: -16.0 # integrated loudness target, measured per EBU R128 / ITU-R BS.1770
  export_subtitles: true # Emits matching .vtt and .srt

tracks:
  - id: ambient_bed
    type: "music"
    source: "assets/audio/synthwave_loop.wav"
    loop: true
    volume: 0.65
    fade_in_ms: 1200
    fade_out_ms: 2500
    ducking:
      trigger_tracks: ["voice_primary", "voice_secondary"]
      reduction_db: -18.0
      attack_ms: 80
      release_ms: 350

  - id: voice_primary
    type: "speech"
    pan: -0.15 # Subtle left stereo placement
    sequence:
      - id: cue_01
        speaker: "narrator"
        provider: "elevenlabs"
        voice_id: "21m00Tcm4TlvDq8ikWAM"
        text: "Autonomous software development requires deterministic compilation targets."
      - pause_duration: 350ms
      - id: cue_03
        speaker: "narrator"
        provider: "elevenlabs"
        voice_id: "21m00Tcm4TlvDq8ikWAM"
        text: "By separating high-level planning from digital signal processing, systems eliminate runtime hallucinations."

  - id: voice_secondary
    type: "speech"
    pan: 0.15 # Subtle right stereo placement
    sequence:
      - id: cue_02
        start_relative:
          after_cue: "cue_01"
          offset_ms: 150
        speaker: "guest"
        provider: "local_kokoro"
        voice_id: "am_adam"
        text: "Exactly. SonicCraft treats audio composition the same way Hyperframes treated web-based video."

  - id: sfx_impact
    type: "sfx"
    source: "assets/sfx/whoosh.wav"
    volume: 0.8
    start_at_ms: 0

This is examples/tech-briefing/timeline.yaml. Here is what soniccraft render does with it:

1. Validates everything at once. Unknown keys, misspelled enum values, dangling anchors, missing voices, missing files and missing API keys are all reported together, each with a stable code, a path, a line and column, and a fix hint.
2. Synthesizes and caches. cue_01 and cue_03 go to ElevenLabs (with word timestamps); cue_02 goes to a local Kokoro-FastAPI server (local_kokoro is an alias of the built-in kokoro provider). Each result is cached by a content hash, so later renders only pay for lines that changed.
3. Levels the voices. Each cue is trimmed of leading and trailing silence and normalized to -20 LUFS, so the cloud voice and the local voice sit at the same level.
4. Lays out the timeline in sample frames. cue_01 starts at 0. cue_02 starts 150 ms after cue_01 ends. cue_03 is sequential, 350 ms after cue_01, but that slot collides with cue_02, so the default overlap policy (push) automatically moves cue_03 to 350 ms after cue_02 ends, keeping its pause as the gap, and reports an info cue_pushed. The result is a natural three-turn exchange with no timing arithmetic by the agent.
5. Places the music. The bed is a looping clip without a length, so it runs to the end of the timeline with a 1.2 s fade-in; the timeline ends 2.5 s after the last word (the bed's fade_out_ms) and the bed fades out over exactly that tail. The whoosh fires at 0.
6. Ducks the bed by 18 dB whenever either voice speaks. Detection runs on the voices' own signal with an 80 ms lookahead (it defaults to the attack time), so the bed is already down when the first syllable lands; it recovers over 350 ms, and the default 250 ms hold bridges the 150 ms turn gap.
7. Pans the two voices to -0.15 and +0.15 (constant power).
8. Masters the mix to -16 LUFS integrated with a -1.0 dBTP true-peak ceiling (from the default podcast preset), iterating loudness gain and a 4× oversampled true-peak limiter until within 0.2 LU (the closest pass is kept and loudness_target_missed says by how much when no pass gets there; an unreachable target gives the loudest master within 6 dB of average limiting, whatever the target).
9. Encodes a 48 kHz, 320 kbps MP3 natively, decodes and measures the delivered file (re-limiting if the codec pushed a peak over the ceiling), and writes timeline.vtt and timeline.srt from the placed word timings, labelled narrator and guest.

Before spending anything, soniccraft plan (alias dry-run) shows the same layout from estimated durations without calling a TTS API (abridged JSON, estimated numbers):

{ "valid": true, "duration_ms": 20616.9, "estimated": true,
  "items": [
    { "id": "cue_01", "kind": "cue", "start_ms": 0.0,     "end_ms": 4838.7,  "duration_source": "estimate" },
    { "id": "cue_02", "kind": "cue", "start_ms": 4988.7,  "end_ms": 10863.7, "duration_source": "estimate" },
    { "id": "cue_03", "kind": "cue", "start_ms": 11213.7, "end_ms": 18116.9, "duration_source": "estimate",
      "pushed_from_ms": 5188.7 } ],
  "synthesis": { "cues": 3, "cached": 0, "characters_to_synthesize": 276 } }

Features

• Timeline DSL v1.0 in YAML or JSON with a published JSON Schema. Strict validation: unknown keys are errors with did-you-mean hints; every problem is reported at once with a stable code, path, line/column and a mechanical fix.
• Timing without arithmetic: sequential cues and pauses, absolute starts, and relative anchors (after_cue, with_cue, before_cue, with signed offsets) on cues, clips or whole tracks. Speech overlaps are pushed apart automatically (or warned about, rejected, or allowed per cue for crosstalk).
• Eight TTS providers: ElevenLabs, any OpenAI-compatible /audio/speech server, Cartesia, Kokoro (Kokoro-FastAPI), Piper, macOS say, any local command, and a deterministic mock voice. Mix them freely in one timeline. Speakers registry with layered voice settings.
• Cheap iteration: a content-addressed TTS cache, dry runs that never call paid APIs (with estimated durations and per-provider character counts), bounded concurrency and retries with backoff.
• Music and sound effects: loops (seamless, or with loop crossfades), trims, fades with three curves, crossfades between clips, open-ended beds that end with an automatic fade-out tail.
• Sidechain ducking driven by the actual speech signal (or by the timeline), with attack, release, hold and lookahead.
• Loudness done properly: per-source BS.1770 normalization, then master normalization to a preset (podcast -16, streaming -14, EBU R128 broadcast -23, ATSC A/85 -24 LUFS) and an oversampled true-peak limiter. Lossy outputs are decoded and measured as delivered, with codec true-peak and loudness-drift compensation.
• Native DSP and encoders: wav, mp3 and flac with no ffmpeg; ogg, opus and m4a through ffmpeg when it is installed. 16/24-bit integer or 32-bit float, fixed-seed TPDF dither.
• Subtitles from the rendered audio: WebVTT, SRT, JSON with per-word timestamps, and a HyperFrames-compatible transcript; speaker labels; per-cue caption overrides.
• Side files: per-track stems, a deterministic manifest, and a self-contained HTML preview with the waveform, ducking envelopes and captions.
• Agent surfaces: machine-readable --json reports from validate, plan, render, providers and voices, an MCP server that speaks both protocol eras, an Agent Skill, and a Claude Code plugin.

Quickstart

Install a prebuilt binary (checksum-verified; no Rust or C toolchain needed):

curl -fsSL https://raw.githubusercontent.com/user-github-me/soniccraft/main/install.sh | sh   # macOS, Linux: ~/.local/bin
brew install user-github-me/soniccraft/soniccraft                                            # Homebrew (macOS, Linux)

irm https://raw.githubusercontent.com/user-github-me/soniccraft/main/install.ps1 | iex        # Windows
scoop bucket add soniccraft https://github.com/user-github-me/scoop-soniccraft; scoop install soniccraft

With Rust: cargo binstall soniccraft downloads the same release binary, and cargo install --locked soniccraft builds it from crates.io (requirements below).

From source. Requirements: Rust 1.89 or newer (rustup) and a C toolchain for the bundled LAME MP3 encoder (cc and make: Xcode Command Line Tools on macOS, build-essential on Linux; on Windows, the Visual Studio Build Tools, plus AWS_LC_SYS_PREBUILT_NASM=1 in the environment unless NASM is installed). ffmpeg is optional (ogg/opus/m4a output, extra input formats). Prebuilt binaries need none of this: see below.

git clone https://github.com/user-github-me/soniccraft
cd soniccraft
cargo install --locked --path crates/soniccraft-cli    # installs `soniccraft` and `soniccraft-mcp` into ~/.cargo/bin
soniccraft --version

--locked builds with the dependency versions in Cargo.lock, the ones CI tests.

Prebuilt binaries. Each GitHub release has archives for Linux (x86_64-unknown-linux-gnu, aarch64-unknown-linux-gnu; glibc 2.35 or newer), macOS (aarch64-apple-darwin, x86_64-apple-darwin; macOS 11 or newer) and Windows (x86_64-pc-windows-msvc). soniccraft-v<version>-<target>.tar.gz (.zip for Windows) holds soniccraft, soniccraft-mcp, the license texts and THIRD-PARTY-LICENSES.txt. The -no-mp3 archives leave out the LGPL MP3 encoder; mp3 output then needs ffmpeg. Check a download with sha256sum --check --ignore-missing SHA256SUMS (on macOS: shasum -a 256 --check --ignore-missing SHA256SUMS). The binaries are not notarized: on macOS a browser download is quarantined; clear it with xattr -d com.apple.quarantine soniccraft soniccraft-mcp (curl and the npm launcher do not set the attribute).

Container image. Each release is also published as ghcr.io/user-github-me/soniccraft (linux/amd64 and linux/arm64, with ffmpeg for ogg, opus and m4a output):

docker run --rm -v "$PWD":/work ghcr.io/user-github-me/soniccraft render timeline.yaml
docker run -i --rm -v "$PWD":/work ghcr.io/user-github-me/soniccraft mcp --base-dir /work   # MCP on stdio

Files written to the mounted directory belong to uid 1000; add --user "$(id -u):$(id -g)" to write them as yourself. Pass API keys with -e ELEVENLABS_API_KEY (never in the timeline). macOS say is not available in the container: use the mock voice, a cloud provider or a local server. docker build -t soniccraft . builds the same image from source.

Render the demo with zero API keys:

# macOS: Samantha and Daniel from macOS `say`, plus the built-in mock voice
soniccraft render examples/offline-demo/timeline.yaml --preview

# any OS: the same timeline with mock voices only
soniccraft render examples/offline-demo/timeline.mock.yaml

The macOS run writes examples/offline-demo/offline-demo.wav with .vtt, .srt, .words.json and .transcript.json subtitles, a .manifest.json, and offline-demo.preview.html (open it in a browser to see the waveform, the ducking envelope and the captions). The full loop an agent follows:

soniccraft validate examples/offline-demo/timeline.mock.yaml          # errors, warnings, fix hints
soniccraft plan     examples/offline-demo/timeline.mock.yaml          # dry run: layout, ducking, cost
soniccraft render   examples/offline-demo/timeline.mock.yaml --json   # one JSON report on stdout

With API keys and a local Kokoro server, render the README timeline:

export ELEVENLABS_API_KEY=...                                          # never put keys in timelines
docker run -d -p 8880:8880 ghcr.io/remsky/kokoro-fastapi-cpu:latest
soniccraft render examples/tech-briefing/timeline.yaml

Start your own project from a template (writes the timeline and generated stand-ins for its audio assets):

soniccraft init my-episode --template two-person-podcast

CLI

soniccraft validate <TIMELINE> [--json] [--check-credentials]
soniccraft plan <TIMELINE> [--json]                      (alias: dry-run)
soniccraft render <TIMELINE> [-o PATH] [--format F] [--subtitles] [--stems] [--manifest] [--preview]
                  [--no-cache] [--cache-dir DIR] [--jobs N] [--json]
soniccraft schema [-o FILE]
soniccraft init [DIR] [--template NAME] [--list] [--no-assets] [--force]
soniccraft providers [--json]
soniccraft voices <PROVIDER> [--json]
soniccraft cache <info|clear> [--cache-dir DIR] [TIMELINE]
soniccraft mcp [--base-dir DIR] [--allow-command]

Command	What it does
validate	Parses and checks the timeline and its environment (assets exist and decode, credentials present and allowed to go to their host, local TTS servers answer, macOS voices and piper models installed, ffmpeg when needed, the render fits the memory budget, cache status). --check-credentials also verifies API keys with a cheap authenticated request.
plan / dry-run	Validates, then lays out the timeline with cached or estimated durations (mock voices exactly): item times, overlap moves, ducking regions, characters to synthesize per provider, and an ASCII diagram. Never calls a paid API.
render	The full pipeline. -o sets the output path (its extension picks the format only when output.format is not set: for a format: wav timeline write -o out.mp3 --format mp3); --format is one of wav, mp3, flac, ogg, opus, m4a and overrides both. --subtitles, --stems and --manifest switch on those exports (like the output.export_* fields); --preview writes <stem>.preview.html. --no-cache ignores cached TTS audio (fresh results are still cached); --cache-dir picks the cache; --jobs N limits concurrent synthesis requests (default 4). Outputs are put in place only when the render succeeds, and existing files that are not SonicCraft outputs are never overwritten.
schema	Prints the timeline JSON Schema, or writes it with -o.
init	Writes DIR/timeline.yaml (default DIR: the current directory) from a template, plus the audio assets it references, generated procedurally (a seamless synth loop and three SFX, deterministic and royalty-free). Templates (--list): offline-demo (default on macOS: system voices + mock), offline-demo-mock (default elsewhere: renders anywhere), tech-briefing, two-person-podcast, voiceover-with-music, dialogue-crosstalk. --no-assets writes only the timeline; existing files are never overwritten without --force.
providers	Lists built-in providers with availability, credential status and default voices.
voices	Lists the voices of one provider (may call its API); the provider name ignores case and -/_.
cache	info shows the TTS cache location, entry count and size; clear empties it.
mcp	Runs the MCP server on stdio. --allow-command (or SONICCRAFT_ALLOW_COMMAND=1) lets timelines run command providers, which are refused by default.

Exit codes: 0 success, 1 timeline invalid (error diagnostics), 2 usage error, 3 runtime or render failure, 130 interrupted. With --json, stdout contains exactly one JSON document (the report, or {"message", "diagnostics", "hint"} when the command fails, usage errors included); progress and logs go to stderr. A render stopped by its environment or options (a missing key, a provider that cannot run, a voice that is not installed, the memory budget) also exits 1 and ends with error: render stopped: N errors. Timelines (files, FIFOs, - for stdin) are read up to the 5 MB document limit and no further. Run soniccraft <command> --help for every option.