{"slug": "turning-video-into-line-art-in-the-browser-with-webgl-2-and-sobel", "title": "Turning Video into Line Art in the Browser with WebGL 2 and Sobel", "summary": "A developer added a line-art asset generator to Timeline Studio, an open-source browser video editor, running locally via WebGL 2 and Sobel edge extraction rather than a neural model. The pipeline applies weighted-luminance grayscale and a 3x3 smoothing pass into a GPU framebuffer texture, then a Sobel gradient pass with a soft threshold, keeping intermediate pixels on the GPU and reusing shader programs, buffers, and textures across frames. A shared renderer drives both preview and asset generation, with inspector controls for detail (mapped to an edge threshold from 0.42 to 0.035), line weight (scaled against a 1080px reference), and inversion.", "body_md": "Depth maps and line art preserve different parts of a video. Depth describes spatial relationships; contours emphasize shape, composition, and visible changes in pose.\n\nAfter adding depth-video conversion to **Timeline Studio**, my open-source browser video editor, I added a line-art asset generator. This version runs locally with WebGL 2 and Sobel edge extraction. It does not download or run a neural line-art model.\n\nHere is how the image-processing pipeline connects to actual video editing: previews, trimmed clips, speed curves, source audio, encoding, and cancellation.\n\nThe pipeline is deliberately small:\n\n```\nImage or decoded video frame\n          |\n   Grayscale + 3x3 smoothing\n          |\n   Intermediate GPU texture\n          |\n   Sobel + soft threshold\n          |\n    Output canvas\n          |\n      PNG or WebM\n```\n\nThe smoothing pass renders into a texture attached to a framebuffer. The edge pass samples that texture and draws to the output canvas. Intermediate pixels stay on the GPU rather than being read back into JavaScript between passes.\n\nShader programs, buffers, textures, and the framebuffer are reused across frames. That reduces setup work, although decoding and encoding still contribute to total processing time.\n\nThe shader computes a weighted RGB luminance:\n\n```\nfloat gray(vec2 p) {\n  return dot(\n    texture(source, p).rgb,\n    vec3(0.2126, 0.7152, 0.0722)\n  );\n}\n```\n\nThe first pass uses this weighted 3x3 kernel, divided by 16:\n\n```\n1  2  1\n2  4  2\n1  2  1\n```\n\nSmoothing suppresses some high-frequency texture before edge extraction. It is a tradeoff: small details can also become weaker.\n\nBecause detection operates on luminance, different colors with similar luminance may produce weak boundaries. Conversely, a shadow or reflection can produce a strong edge without marking an object boundary.\n\nThe second pass estimates horizontal and vertical gradients:\n\n```\nGx                  Gy\n-1  0  1            -1 -2 -1\n-2  0  2             0  0  0\n-1  0  1             1  2  1\n```\n\nTheir combined magnitude becomes the edge strength:\n\n```\nfloat gradient = length(vec2(\n  c + 2.0 * f + i - a - 2.0 * e - g,\n  g + 2.0 * h + i - a - 2.0 * b - c\n));\n```\n\nThe letters represent neighboring luminance samples. Each output pixel performs the same local calculation, which maps naturally to a fragment shader.\n\nThe inspector exposes detail, line weight, and inversion. Defaults are 55% detail, 1x line weight, and black lines on white.\n\nDetail maps to an edge threshold:\n\n```\nfloat threshold = mix(0.42, 0.035, detail / 100.0);\nfloat ink = smoothstep(threshold * 0.5, threshold, gradient);\nfloat value = inverted ? ink : 1.0 - ink;\n```\n\nHigher detail lowers the threshold, retaining weaker edges. The soft threshold avoids an abrupt binary transition around the cutoff. For busy footage, increasing detail can make the result less useful by retaining background texture.\n\nLine weight changes the sampling distance. It uses the longest dimension as a 1080px reference:\n\n``` js\nconst scaledThickness =\n  thickness * Math.max(width, height) / 1080;\n```\n\nThis helps keep the edge-sampling scale comparable across output sizes. It is not morphological dilation: changing the sampling distance also changes which details respond. The initial blur itself remains a one-pixel 3x3 neighborhood.\n\nBoth paths call the same implementation:\n\n``` js\nconst renderer = createLineArtRenderer(canvas, width, height);\nrenderer.draw(media, options);\n// Release resources when finished:\nrenderer.dispose();\n```\n\nThe preview redraws a sampled frame when parameters or the local playhead time change. Asset generation traverses output frames through that same renderer.\n\nSharing the implementation reduces differences in thresholds, sampling, and inversion between preview and generated assets. It does not make processing instantaneous.\n\nOutput dimensions preserve aspect ratio, cap the longest edge at 1080 by default, avoid enlarging smaller media, and normalize dimensions to even numbers.\n\nA selected clip may already be trimmed or retimed. Sampling the original file from zero would produce the wrong result.\n\nThere are separate coordinates for output time, clip-local time, and source-media time. For constant speed:\n\n```\nsourceTime = sourceIn + clipLocalTime * speed\n```\n\nFor speed curves, the implementation reuses the editor's mapping:\n\n```\ngetVisualSourceTime(segment, localTime)\n```\n\nThe generator outputs 24 fps. Each local sample time is i / 24; it is mapped to source time before decoding the frame. The processed frame is then encoded at its output timestamp.\n\n```\nOutput frame time\n       |\nSource-time mapping, including trim and retiming\n       |\nDecode source frame -> draw contours -> encode output\n```\n\nMediabunny handles media input and timestamp-based canvas sampling. The final frame duration is limited to the remaining clip duration.\n\n**24 fps is the output rate, not a claim of real-time generation speed.** Actual throughput depends on the browser, hardware, source media, and encoder.\n\nOne integration detail had nothing to do with Sobel: image orientation.\n\nThe image path decodes a Blob to an ImageBitmap, then draws it to a 2D canvas before uploading it to WebGL. ImageBitmap does not follow the same UNPACK_FLIP_Y_WEBGL behavior as the other upload path, so the intermediate canvas normalizes orientation.\n\nWithout that step, still images and video frames could disagree about which side is up.\n\nImages become PNG. Videos become WebM using VP9 at approximately 4 Mbps. When corresponding source audio exists, the output uses Opus at 192 kbps.\n\nAudio preparation reuses the editor's source-audio and offline-mixing helpers so trimming and retiming follow the selected segment. This preserves the segment's corresponding source sound; it does not mix every music and narration track from the whole project.\n\nThe completed asset goes to **My assets**. It is not automatically inserted into the timeline, making it possible to retain the original and compare versions before placement.\n\nAn AbortController carries cancellation through source reading, frame processing, and output finalization. The renderer releases its textures, framebuffer, shaders, programs, and vertex buffer. Media input resources are disposed, and encoding is canceled when necessary.\n\nFrame processing reports progress up to 99%. Finalization and the asset commit must finish before the UI reports completion. Processing the last frame is not the same as having a usable saved file.\n\nDepth video and contour video provide complementary reference assets:\n\n| Asset | Emphasizes | \n|---|---|\n| Depth video | Relative depth and foreground/background layout | \n| Line-art video | Visible contours, composition, and changes in shape | \n\nA downstream generation workflow can use these only if it accepts the corresponding conditioning input and its preprocessing, timing, resolution, and frame-rate requirements are satisfied.\n\nThis feature creates the reference asset. It does not imply automatic integration with every image or video generator, nor guarantee that a generator will preserve the desired structure.\n\nThe current Sobel route also lacks semantic simplification and explicit temporal tracking. Fine textures, noise, and motion can make lines fluctuate across frames. A learned line-art model or additional temporal processing would need separate evaluation rather than being claimed as part of this implementation.\n\nOpen **Effects -> Line art**, select an image or video clip, adjust parameters, and generate the asset. The feature includes 13 interface languages and a mobile layout.\n\nImplementation entry points:\n\nIf you work on browser graphics or video tools, I would welcome difficult examples: busy backgrounds, small contours, and fast motion are particularly useful for understanding where this approach needs improvement.\n\n[GitHub repository](https://github.com/MartinDelophy/ai-video-editor) · [Open the editor](https://video-editor.ai-creator.top/)\n\n*Disclosure: I maintain the project discussed here. This article was drafted by an AI assistant using the current repository source and implementation notes.*", "url": "https://wpnews.pro/news/turning-video-into-line-art-in-the-browser-with-webgl-2-and-sobel", "canonical_source": "https://dev.to/martindelophy/turning-video-into-line-art-in-the-browser-with-webgl-2-and-sobel-4cmm", "published_at": "2026-11-04 06:40:49+00:00", "updated_at": "2026-11-04 06:52:35.088320+00:00", "lang": "en", "topics": ["computer-vision", "developer-tools", "ai-tools"], "entities": ["Timeline Studio", "WebGL 2"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/turning-video-into-line-art-in-the-browser-with-webgl-2-and-sobel", "markdown": "https://wpnews.pro/news/turning-video-into-line-art-in-the-browser-with-webgl-2-and-sobel.md", "text": "https://wpnews.pro/news/turning-video-into-line-art-in-the-browser-with-webgl-2-and-sobel.txt", "jsonld": "https://wpnews.pro/news/turning-video-into-line-art-in-the-browser-with-webgl-2-and-sobel.jsonld"}}