Screen2tutorial

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

Screen2Tutorial turns a short screen recording of a coding or terminal tutorial into a step-by-step written guide in Markdown, with the commands and code copied from what appeared on screen.

If you make coding videos, you know the chore of writing the blog-post version: , rewind, squint at the terminal, retype each command, and hope you didn't mistype one. Screen2Tutorial does that first draft for you. It is aimed at independent coding educators, YouTubers and developer-relations people who publish a video and a written version of the same lesson.

The part I care most about is trust. A model that writes tutorials will sometimes invent a command or change an argument. So every command it writes is checked against two sources: the spoken transcript and the text read from the video frames. Each command ends up marked as verified by the screen, by speech, by both, or UNVERIFIED, so the creator knows which lines to re-check before publishing.

Would someone pay for it? That's a hypothesis I haven't validated. My guess is that a creator would pay to save editing time, as long as they can see where each step came from.

Scope, on purpose: one English-speaking narrator, terminal or IDE recordings, short videos, Markdown output. No logins, payments or multiple languages.

This is the link of my video demo (link)

I tested it on two public videos. The main one is "10 Basic Ubuntu Linux Commands Every Beginner Should Know" by vCloudBitsBytes (link), about 4 minutes long. Here is one step of the output, with the verification label next to each command:

Step 4: Create Directories and Files using MKDIR and TOUCH

The author demonstrates how to use the mkdir command to create a new directory named test and the touch command to create a new file named test.txt...

mkdir test — verified_screen

ls — verified_both

touch test.txt — verified_screen

Turn a 5–10 minute English coding recording into a code-backed, step-by-step Markdown tutorial. Speech transcription and screen OCR are aligned by time The writing model organizes the explanation and extracts visible code examples each shell command is compared with both evidence sources, and commands found in neither are marked UNVERIFIED.

The supported scope is one English-speaking narrator, one terminal/IDE recording, and Markdown output. The application has a Streamlit upload page and the CLI python -m pipeline.run path/to/video.mp4.

Excerpt from a generated guide for a CSS random() tutorial:

The author demonstrates how to generate random values directly within CSS to avoid hard-coding manual values or using JavaScript for simple visual effects.

particle {
  scale: random(1px, 7px);
}

Generated code is included when it matches text recognized on screen. Review the extracted snippet against the recording before relying on it.

MIT licensed. The README has Windows install steps (including ffmpeg and Tesseract), configuration, limitations and tests.

The pipeline has six stages, one module each:

base model, int8, CPU) transcribes the audio with timestamps. The open AI: the writing step uses Gemma 4, an open-weight model, reached through Ollama's OpenAI-compatible endpoint. Whisper and Tesseract run entirely on my laptop. The endpoint and model name are two lines in config.yaml, so another open model, hosted or local, can be swapped in without touching the code.

Built for a weak laptop. My machine has 8 GB of RAM, an Intel i3 and no GPU. That decided most of the design: a small Whisper model, frame deduplication before OCR, caching of every stage on disk, and a hosted open model for the one step that needs real language quality.

I'm a beginner with Whisper, OCR and LLM APIs, so I built and tested one stage at a time. The terminal test taught me the most.

First run on the Ubuntu video: 20 commands found, 14 verified, 6 flagged UNVERIFIED. When I looked closer, four of the six flagged commands were verified in the neighboring step. The cause was in my own code: each command was checked only against the text of its own 30-second window, so a command typed near a window boundary was checked against the wrong text.

The fix: verify each step against the previous, current and next window together, and require whole-word matches for very short commands like ls and cd (otherwise "ls" matches inside ordinary words like "also").

All six earlier flags were false alarms from the window boundary, not real misses. All ten commands named in the video's own chapter list (cd, ls, pwd, mkdir, touch, cp, mv, rm, sudo, apt-get) appear in the tutorial.

The CSS video (a tutorial on CSS random(), (link)) produced an 8-step guide with 10 code blocks. Its verification summary showed 0 commands, which is correct, because the video was about CSS and not shell commands. It also shows the OCR limits: Tesseract read 0deg as @deg and :root as sroot, and some closing braces were lost. Each code block carries a "check unclear characters against the recording" note for this reason.

A 20/20 score on one video is not proof that the verifier catches mistakes, so here are the honest limits:

touch, rm and apt-get steps each appear twice in the Ubuntu output). 1. Privacy is a setting, not a promise. Tutorials often show half-built projects, internal tools or terminal history. With open weights, the same pipeline can point at a local model for recordings that must not leave the machine. Right now my config uses a cloud-hosted Gemma, which sends the transcript and OCR text to Ollama for the writing step, and the README says so plainly. Switching to a local model is a config change.

2. No lock-in. The model is a config value. If a better or cheaper open model appears, or one disappears, I change a line instead of rewriting the app. A creator who builds a workflow around the tool needs that stability.

3. I could see why it was wrong. The verification feature is only worth trusting if its parts can be inspected. Because the whole stack is open, readable code and components, I could trace the six false alarms to a single design decision, fix it, and measure the difference, which is the loop in the testing section above.