Perform end-to-end smoke testing of CLI, API, and UI user flows using disposable sandboxed scripts against a real application.
- GitHub stars
- 1,663
- Files
- 4
- Last commit
- Nov 9, 2026
- License
- Apache-2.0
- Allowed tools
- —
What does the dev-smoke skill do?
dev-smoke is an agent skill. Perform end-to-end smoke testing of CLI, API, and UI user flows using disposable sandboxed scripts against a real application.
Summary generated from the skill's SKILL.md.
About the dev-smoke skill
Developers can validate changes by running end-to-end smoke tests across CLI, API, and UI user flows using disposable sandboxed scripts. This skill is intended for use after unit tests pass but before a feature is marked as complete, or when a user requests an impact check. It works by analyzing git diffs to map how code changes affect entry points like commands, routes, or screens, then generating scripts based on templates like references/smoke-template.sh or references/web-smoke-template.mjs.
An agent should load this skill to exercise a real application through its actual entry points, such as a built binary or a running server, rather than using mocks. The process relies on creating an isolated environment with a temporary directory, seeded test data, and dedicated ports to ensure no interference with the user's own servers. It produces a detailed report via the verify command that includes scenarios, exit codes, and per-check results. Once testing is finished, the sandbox and scripts are deleted unless the user chooses to promote them to permanent tests.
What an agent can do with it
- 1Isolate regressions from existing bugsDetermine if a test failure is a new regression by rerunning the check on the base commit in a separate worktree.
Written from the skill's SKILL.md. Every use case is checked against a quote from it.
How to install the dev-smoke skill
Each command copies the skill folder into the client's skills directory. Restart the client or start a new session to load it.Install the dev-smoke skill in Claude Code
~/.claude/skills/dev-smoke/SKILL.md
$ npx degit codeaholicguy/ai-devkit/skills/dev-smoke ~/.claude/skills/dev-smokeInstall the dev-smoke skill in Claude Code (project)
.claude/skills/dev-smoke/SKILL.md
$ npx degit codeaholicguy/ai-devkit/skills/dev-smoke .claude/skills/dev-smokeInstall the dev-smoke skill in Codex
~/.agents/skills/dev-smoke/SKILL.md
$ npx degit codeaholicguy/ai-devkit/skills/dev-smoke ~/.agents/skills/dev-smokeInstall the dev-smoke skill in Gemini CLI
~/.gemini/skills/dev-smoke/SKILL.md
$ npx degit codeaholicguy/ai-devkit/skills/dev-smoke ~/.gemini/skills/dev-smokeInstall the dev-smoke skill in Claude.ai
npx degit codeaholicguy/ai-devkit/skills/dev-smoke dev-smoke && zip -r dev-smoke.zip dev-smoke # claude.ai → Customize → Skills → + → Create skill → Upload a skill → dev-smoke.zip
Automated checks
Ran Nov 15, 2026Full report →Passed our automated checks. That is not a security audit: read the scripts before you let an agent run them. How checks work · What to fix for more points
- Source repository linked and reachablecodeaholicguy/ai-devkit/skills/dev-smoke is public and reachablegithub.com/codeaholicguy/ai-devkit/tree/HEAD/skills/dev-smok ↗Pass
- Open source license (OSI)Apache-2.0spdx.org/licenses/Apache-2.0.html ↗Pass
- Commit or release in the last yearLast commit 2026-11-09github.com/codeaholicguy/ai-devkit/commits ↗Pass
- No download-and-run or credential theft in scripts2 scripts read; no download-and-run, remote eval, base64 execution or credential uploadPass
- No hidden instructions in SKILL.mdNo override phrases, safeguard switches, credential requests or hidden charactersgithub.com/codeaholicguy/ai-devkit/blob/HEAD/skills/dev-smok ↗Pass
- Pre-approved tools are scopedNo pre-approved tools; the agent asks before using onegithub.com/codeaholicguy/ai-devkit/blob/HEAD/skills/dev-smok ↗Pass
- SKILL.md follows the Agent Skills formatthe description does not say when to use the skillgithub.com/codeaholicguy/ai-devkit/blob/HEAD/skills/dev-smok ↗Finding
SKILL.md
View on GitHubSmoke
Exercise the flows a change touches the way a user would, then throw the script away. Report with verify; use dev-testing for permanent tests.
Rules
- Drive the real entry point: built binary, running server, CLI, UI, or job. Imported functions and mocks are not smoke checks.
- Sandbox everything: temp dir, isolated
HOMEand config, empty git hooks, seeded test data, test accounts and tokens. - Never use production, shared environments, real accounts or credentials, or paid APIs. Ask before outward-facing side effects: messages, emails, webhooks, deploys, pushes.
- Start servers on a dedicated port, wait until they respond, and stop the whole process tree on exit. Never reuse or kill the user's own servers.
- Keep secrets out of the script and its output.
- Write the script outside the repo; commit it only if the user asks to promote it.
- Timebox setup. If the app cannot run locally, say what was not checked instead of faking a pass.
Workflow
- Read the change:
git diffagainst the base branch, plus untracked files. - Map impact outward: changed symbols -> callers -> entry points (commands, routes, screens, jobs, hooks, config, migrations) -> user flows. Add adjacent flows sharing the changed code, config, or data; note skipped flows and why.
- List scenarios as setup / action / expected result, ordered by risk. Wait for the user to add or cut flows; when running unattended, proceed and include the list in the report.
- Search memory for known setup:
npx ai-devkit@latest memory search --query "<project> smoke setup" --tags "smoke". Build once, then write the script fromreferences/smoke-template.sh(CLI, API) orreferences/web-smoke-template.mjs(browser). Each check asserts output, files, HTTP status, or exit code, not just "it ran"; API checks also assert the body and one error case (bad input, missing auth). Print PASS/FAIL per check and exit non-zero on any failure. - Run it and read the full output.
- On a failure, first rule out a wrong check (selector, timing, setup), then rerun it on the base commit in a separate worktree (
dev-worktree) to separate regressions from pre-existing failures. - Report with
verify: scenarios, command, exit code, per-check result, pre-existing failures, flows not covered. - Delete the script and sandbox, or offer to promote valuable checks into permanent tests or the project's release smoke suite. Store non-obvious setup:
npx ai-devkit@latest memory store --title "<project> smoke setup" --content "<start command, port, health URL, seeding, traps>" --tags "smoke".
UI and Web
- Use the harness's browser tool when available (for example Claude in Chrome or a Playwright MCP server); otherwise the web template, with Playwright installed in the sandbox.
- Use a fresh browser context per check, and act like a user: navigate, click, fill forms, submit.
- Assert visible content, URL, network status, and zero console errors, not just that the page loaded. Match visible elements by role or text; hidden duplicates (mobile menus, templates) cause false failures.
- UI impact includes shared components, layouts, global styles, and every route rendering the changed component. Check narrow viewports or dark mode when layout or theming changed.
- Save a screenshot per check as evidence, and read it before judging a failure.
- Not covered: native mobile apps, desktop apps, pixel-level visual diffs. Report them as unchecked.
Stop
Pause when impacted flows are unclear, a scenario needs anything the Rules forbid, setup needs destructive actions, or failures come from the environment rather than the change.
Files in this skill
4dev-smoke skill questions
How do I install the dev-smoke skill in Claude Code?
Run npx degit codeaholicguy/ai-devkit/skills/dev-smoke ~/.claude/skills/dev-smoke. Claude Code loads skills from ~/.claude/skills (all projects) or .claude/skills (one project) and uses a skill when its description matches the task.
Which clients can use the dev-smoke skill?
Any client that reads the Agent Skills format: Claude Code, Codex, Gemini CLI and claude.ai (upload as a ZIP). Scripts in the skill run only where the client allows code execution.
What kind of environments should I use for smoke testing?
You should use sandboxed environments with temporary directories, isolated configurations, and seeded test data, avoiding production or shared environments.
How does the tool handle UI testing?
It uses browser tools like Playwright to navigate, click, and fill forms, asserting visible content, URLs, and network status while saving screenshots as evidence.
What should I do if a smoke test fails?
First, rule out incorrect checks like selectors or timing, then rerun the test on the base commit in a separate worktree to distinguish new regressions from existing failures.