画面録画を分析して手動のワークフローを再構築し、ユーザーが現在使用しているローカルツールを活用して段階的な自動化スクリプトを生成します。
- GitHubスター
- 41,038
- ファイル
- 1
- 最終コミット
- 2026年11月11日
- ライセンス
- MIT
- 許可されたツール
- —
automate-this skillは何をしますか
画面録画を分析して手動のワークフローを再構築し、ユーザーが現在使用しているローカルツールを活用して段階的な自動化スクリプトを生成します。
skillのSKILL.mdから生成された要約。
automate-this skill について
手動の画面録画を機能的な自動化スクリプトに変換することが、このSKILLの主な機能です。繰り返しの多いタスクを録画し、それらの動画を既存のローカル環境で動作するコードに変換したいユーザー向けに設計されています。プロセスは、ffmpegを使用して視覚的なフレームを抽出し、Whisperを通じて音声ナレーションを文字起こしすることで、ユーザーの意図や課題を理解することから始まります。
ワークフローの再構築と確認が完了すると、このSKILLはローカルマシンを分析し、Python、Node、Playwright、AppleScriptなどの利用可能なツールを特定します。その後、シェルエイリアスのようなクイックウィン、エンドツーエンドの実行が可能なスタンドアロンスクリプト、そしてcronやlaunchdによるスケジュール実行を含む完全な自動化という、3つの異なるレベルの自動化を提案します。このアプローチにより、不要な新規インストールを必要とせず、生成されたスクリプトがユーザーの現在のセットアップと互換性を持つことを保証します。
エージェントができること
- 1動画から手動ワークフローを再構築AIが画面録画を解析し、プロセスで使用されている特定のアプリケーション、アクションの順序、およびデータフローを特定します。
- 2段階的な自動化案の生成開発者は、単純なキーボードショートカットから完全にスケジュールされたスクリプトまで、3つの異なるレベルの自動化案を受け取ることができます。
- 3ローカルツールの互換性確認このSKILLは、ユーザーのオペレーティングシステムとインストール済みソフトウェアを確認し、提案されたスクリプトが実際にマシン上で動作することを確認します。
- 4ブラウザおよびデータタスクの自動化Web操作にはPlaywrightを、複雑なスプレッドシート操作にはpandasを使用したPythonを使用するといった提案が可能です。
skill の SKILL.md から作成されています。すべてのユースケースは、その引用と照合されています。
automate-this skillのインストール方法
各コマンドはskillフォルダをクライアントのskillsディレクトリにコピーします。読み込むには、クライアントを再起動するか新しいセッションを開始してください。Claude Codeにautomate-this skillをインストールする
~/.claude/skills/automate-this/SKILL.md
$ npx degit github/awesome-copilot/skills/automate-this ~/.claude/skills/automate-thisClaude Code (project)にautomate-this skillをインストールする
.claude/skills/automate-this/SKILL.md
$ npx degit github/awesome-copilot/skills/automate-this .claude/skills/automate-thisCodexにautomate-this skillをインストールする
~/.agents/skills/automate-this/SKILL.md
$ npx degit github/awesome-copilot/skills/automate-this ~/.agents/skills/automate-thisGemini CLIにautomate-this skillをインストールする
~/.gemini/skills/automate-this/SKILL.md
$ npx degit github/awesome-copilot/skills/automate-this ~/.gemini/skills/automate-thisClaude.aiにautomate-this skillをインストールする
npx degit github/awesome-copilot/skills/automate-this automate-this && zip -r automate-this.zip automate-this # claude.ai → Customize → Skills → + → Create skill → Upload a skill → automate-this.zip
自動チェック
実行日 2026年11月15日完全なレポート →自動チェックに合格しました。これはセキュリティ監査ではありません。エージェントに実行させる前にスクリプトを読んでください。 チェックの仕組み · 点数を上げるための改善点
- ソースリポジトリがリンクされており、アクセス可能github/awesome-copilot/skills/automate-this is public and reachablegithub.com/github/awesome-copilot/tree/HEAD/skills/automate- ↗合格
- オープンソースライセンス (OSI)MITspdx.org/licenses/MIT.html ↗合格
- 過去1年以内にコミットまたはリリースありLast commit 2026-11-11github.com/github/awesome-copilot/commits ↗合格
- スクリプトにダウンロード実行や認証情報の窃取がないNo scripts in the skill folder合格
- SKILL.md に隠れた指示がないNo override phrases, safeguard switches, credential requests or hidden charactersgithub.com/github/awesome-copilot/blob/HEAD/skills/automate- ↗合格
- 事前許可ツールが限定されているNo pre-approved tools; the agent asks before using onegithub.com/github/awesome-copilot/blob/HEAD/skills/automate- ↗合格
- SKILL.md が Agent Skills の形式に従っているthe description does not say when to use the skillgithub.com/github/awesome-copilot/blob/HEAD/skills/automate- ↗検出事項
SKILL.md
GitHubで表示Automate This
Analyze a screen recording of a manual process and build working automation for it.
The user records themselves doing something repetitive or tedious, hands you the video file, and you figure out what they're doing, why, and how to script it away.
Prerequisites Check
Before analyzing any recording, verify the required tools are available. Run these checks silently and only surface problems:
command -v ffmpeg >/dev/null 2>&1 && ffmpeg -version 2>/dev/null | head -1 || echo "NO_FFMPEG"
command -v whisper >/dev/null 2>&1 || command -v whisper-cpp >/dev/null 2>&1 || echo "NO_WHISPER"
- ffmpeg is required. If missing, tell the user:
brew install ffmpeg(macOS) or the equivalent for their OS. - Whisper is optional. Only needed if the recording has narration. If missing AND the recording has an audio track, suggest:
pip install openai-whisperorbrew install whisper-cpp. If the user declines, proceed with visual analysis only.
Phase 1: Extract Content from the Recording
Given a video file path (typically on ~/Desktop/), extract both visual frames and audio:
Frame Extraction
Extract frames at one frame every 2 seconds. This balances coverage with context window limits.
WORK_DIR=$(mktemp -d "${TMPDIR:-/tmp}/automate-this-XXXXXX")
chmod 700 "$WORK_DIR"
mkdir -p "$WORK_DIR/frames"
ffmpeg -y -i "<VIDEO_PATH>" -vf "fps=0.5" -q:v 2 -loglevel warning "$WORK_DIR/frames/frame_%04d.jpg"
ls "$WORK_DIR/frames/" | wc -l
Use $WORK_DIR for all subsequent temp file paths in the session. The per-run directory with mode 0700 ensures extracted frames are only readable by the current user.
If the recording is longer than 5 minutes (more than 150 frames), increase the interval to one frame every 4 seconds to stay within context limits. Tell the user you're sampling less frequently for longer recordings.
Audio Extraction and Transcription
Check if the video has an audio track:
ffprobe -i "<VIDEO_PATH>" -show_streams -select_streams a -loglevel error | head -5
If audio exists:
ffmpeg -y -i "<VIDEO_PATH>" -ac 1 -ar 16000 -loglevel warning "$WORK_DIR/audio.wav"
# Use whichever whisper binary is available
if command -v whisper >/dev/null 2>&1; then
whisper "$WORK_DIR/audio.wav" --model small --language en --output_format txt --output_dir "$WORK_DIR/"
cat "$WORK_DIR/audio.txt"
elif command -v whisper-cpp >/dev/null 2>&1; then
whisper-cpp -m "$(brew --prefix 2>/dev/null)/share/whisper-cpp/models/ggml-small.bin" -l en -f "$WORK_DIR/audio.wav" -otxt -of "$WORK_DIR/audio"
cat "$WORK_DIR/audio.txt"
else
echo "NO_WHISPER"
fi
If neither whisper binary is available and the recording has audio, inform the user they're missing narration context and ask if they want to install Whisper (pip install openai-whisper or brew install whisper-cpp) or proceed with visual-only analysis.
Phase 2: Reconstruct the Process
Analyze the extracted frames (and transcript, if available) to build a structured understanding of what the user did. Work through the frames sequentially and identify:
- Applications used — Which apps appear in the recording? (browser, terminal, Finder, mail client, spreadsheet, IDE, etc.)
- Sequence of actions — What did the user do, in order? Click-by-click, step-by-step.
- Data flow — What information moved between steps? (copied text, downloaded files, form inputs, etc.)
- Decision points — Were there moments where the user paused, checked something, or made a choice?
- Repetition patterns — Did the user do the same thing multiple times with different inputs?
- Pain points — Where did the process look slow, error-prone, or tedious? The narration often reveals this directly ("I hate this part," "this always takes forever," "I have to do this for every single one").
Present this reconstruction to the user as a numbered step list and ask them to confirm it's accurate before proposing automation. This is critical — a wrong understanding leads to useless automation.
Format:
Here's what I see you doing in this recording:
1. Open Chrome and navigate to [specific URL]
2. Log in with credentials
3. Click through to the reporting dashboard
4. Download a CSV export
5. Open the CSV in Excel
6. Filter rows where column B is "pending"
7. Copy those rows into a new spreadsheet
8. Email the new spreadsheet to [recipient]
You repeated steps 3-8 three times for different report types.
[If narration was present]: You mentioned that the export step is the slowest
part and that you do this every Monday morning.
Does this match what you were doing? Anything I got wrong or missed?
Do NOT proceed to Phase 3 until the user confirms the reconstruction is accurate.
Phase 3: Environment Fingerprint
Before proposing automation, understand what the user actually has to work with. Run these checks:
echo "=== OS ===" && uname -a
echo "=== Shell ===" && echo $SHELL
echo "=== Python ===" && { command -v python3 && python3 --version 2>&1; } || echo "not installed"
echo "=== Node ===" && { command -v node && node --version 2>&1; } || echo "not installed"
echo "=== Homebrew ===" && { command -v brew && echo "installed"; } || echo "not installed"
echo "=== Common Tools ===" && for cmd in curl jq playwright selenium osascript automator crontab; do command -v $cmd >/dev/null 2>&1 && echo "$cmd: yes" || echo "$cmd: no"; done
Use this to constrain proposals to tools the user already has. Never propose automation that requires installing five new things unless the simpler path genuinely doesn't work.
Phase 4: Propose Automation
Based on the reconstructed process and the user's environment, propose automation at up to three tiers. Not every process needs three tiers — use judgment.
Tier Structure
Tier 1 — Quick Win (under 5 minutes to set up) The smallest useful automation. A shell alias, a one-liner, a keyboard shortcut, an AppleScript snippet. Automates the single most painful step, not the whole process.
Tier 2 — Script (under 30 minutes to set up) A standalone script (bash, Python, or Node — whichever the user has) that automates the full process end-to-end. Handles common errors. Can be run manually when needed.
Tier 3 — Full Automation (under 2 hours to set up) The script from Tier 2, plus: scheduled execution (cron, launchd, or GitHub Actions), logging, error notifications, and any necessary integration scaffolding (API keys, auth tokens, etc.).
Proposal Format
For each tier, provide:
## Tier [N]: [Name]
**What it automates:** [Which steps from the reconstruction]
**What stays manual:** [Which steps still need a human]
**Time savings:** [Estimated time saved per run, based on the recording length and repetition count]
**Prerequisites:** [Anything needed that isn't already installed — ideally nothing]
**How it works:**
[2-3 sentence plain-English explanation]
**The code:**
[Complete, working, commented code — not pseudocode]
**How to test it:**
[Exact steps to verify it works, starting with a dry run if possible]
**How to undo:**
[How to reverse any changes if something goes wrong]
Application-Specific Automation Strategies
Use these strategies based on which applications appear in the recording:
Browser-based workflows:
- First choice: Check if the website has a public API. API calls are 10x more reliable than browser automation. Search for API documentation.
- Second choice:
curlorwgetfor simple HTTP requests with known endpoints. - Third choice: Playwright or Selenium for workflows that require clicking through UI. Prefer Playwright — it's faster and less flaky.
- Look for patterns: if the user is downloading the same report from a dashboard repeatedly, it's almost certainly available via API or direct URL with query parameters.
Spreadsheet and data workflows:
- Python with pandas for data filtering, transformation, and aggregation.
- If the user is doing simple column operations in Excel, a 5-line Python script replaces the entire manual process.
csvkitfor quick command-line CSV manipulation without writing code.- If the output needs to stay in Excel format, use openpyxl.
Email workflows:
- macOS:
osascriptcan control Mail.app to send emails with attachments. - Cross-platform: Python
smtplibfor sending,imaplibfor reading. - If the email follows a template, generate the body from a template file with variable substitution.
File management workflows:
- Shell scripts for move/copy/rename patterns.
find+xargsfor batch operations.fswatchorwatchmanfor triggered-on-change automation.- If the user is organizing files into folders by date or type, that's a 3-line shell script.
Terminal/CLI workflows:
- Shell aliases for frequently typed commands.
- Shell functions for multi-step sequences.
- Makefiles for project-specific task sets.
- If the user ran the same command with different arguments, that's a loop.
macOS-specific workflows:
- AppleScript/JXA for controlling native apps (Mail, Calendar, Finder, Preview, etc.).
- Shortcuts.app for simple multi-app workflows that don't need code.
automatorfor file-based workflows.launchdplist files for scheduled tasks (prefer over cron on macOS).
Cross-application workflows (data moves between apps):
- Identify the data transfer points. Each transfer is an automation opportunity.
- Clipboard-based transfers in the recording suggest the apps don't talk to each other — look for APIs, file-based handoffs, or direct integrations instead.
- If the user copies from App A and pastes into App B, the automation should read from A's data source and write to B's input format directly.
Making Proposals Targeted
Apply these principles to every proposal:
-
Automate the bottleneck first. The narration and timing in the recording reveal which step is actually painful. A 30-second automation of the worst step beats a 2-hour automation of the whole process.
-
Match the user's skill level. If the recording shows someone comfortable in a terminal, propose shell scripts. If it shows someone navigating GUIs, propose something with a simple trigger (double-click a script, run a Shortcut, or type one command).
-
Estimate real time savings. Count the recording duration and multiply by how often they do it. "This recording is 4 minutes. You said you do this daily. That's 17 hours per year. Tier 1 cuts it to 30 seconds each time — you get 16 hours back."
-
Handle the 80% case. The first version of the automation should cover the common path perfectly. Edge cases can be handled in Tier 3 or flagged for manual intervention.
-
Preserve human checkpoints. If the recording shows the user reviewing or approving something mid-process, keep that as a manual step. Don't automate judgment calls.
-
Propose dry runs. Every script should have a mode where it shows what it would do without doing it.
--dry-runflags, preview output, or confirmation prompts before destructive actions. -
Account for auth and secrets. If the process involves logging in or using credentials, never hardcode them. Use environment variables, keychain access (macOS
securitycommand), or prompt for them at runtime. -
Consider failure modes. What happens if the website is down? If the file doesn't exist? If the format changes? Good proposals mention this and handle it.
Phase 5: Build and Test
When the user picks a tier:
- Write the complete automation code to a file (suggest a sensible location — the user's project directory if one exists, or
~/Desktop/otherwise). - Walk through a dry run or test with the user watching.
- If the test works, show how to run it for real.
- If it fails, diagnose and fix — don't give up after one attempt.
Cleanup
After analysis is complete (regardless of outcome), clean up extracted frames and audio:
rm -rf "$WORK_DIR"
Tell the user you're cleaning up temporary files so they know nothing is left behind.
このスキルのファイル
1automate-this skillに関する質問
Claude Codeにautomate-this skillをインストールするには
npx degit github/awesome-copilot/skills/automate-this ~/.claude/skills/automate-thisを実行してください。Claude Codeは~/.claude/skills(すべてのプロジェクト)または.claude/skills(単一プロジェクト)からskillsを読み込み、説明がタスクに一致する場合にskillを使用します。
automate-this skillはどのクライアントで使用できますか
Agent Skills形式を読み込むすべてのクライアントで使用できます: Claude Code、Codex、Gemini CLI、claude.ai(ZIPとしてアップロード)。skill内のスクリプトは、クライアントがコード実行を許可した場合にのみ実行されます。
録画を分析するために必要なツールは何ですか?
フレームと音声の抽出にはffmpegが必要です。より正確なコンテキストを把握するために、録画にナレーションが含まれている場合はWhisperの使用を推奨します。
長い動画ファイルはどのように処理されますか?
5分を超える録画の場合、コンテキストウィンドウの制限を管理するため、フレームのサンプリング間隔を4秒に1フレームに増やして処理します。