{"slug": "seehere-making-educational-videos-more-accessible-with-ai-powered-audio", "title": "SeeHere: Making Educational Videos More Accessible with AI-Powered Audio Descriptions", "summary": "A developer built SeeHere, an AI-powered accessibility pipeline that automatically generates and inserts audio descriptions into educational lecture videos for blind and low-vision learners. The system uses Gemini for visual and contextual understanding of extracted frames, Tesseract OCR to capture on-screen labels and equations, Google OR-Tools' CP-SAT solver to schedule descriptions into non-interfering time windows, and smallest.ai for text-to-speech. The project targets visual information the lecturer's narration leaves unexplained, such as diagrams and annotations, rather than summarizing the whole video.", "body_md": "Educational videos have become an essential part of how people learn. From online courses and university lectures to tutorials and recorded presentations, a lot of knowledge is now delivered through video.\n\nBut there's a problem we don't talk about enough.\n\n**A significant amount of information in educational videos is visual.**\n\nThink about a biology lecture explaining photosynthesis. The instructor might be talking about chlorophyll while a diagram illustrates the light-dependent reactions, labels identify different structures, and arrows explain how energy flows through the process.\n\nSomeone listening to the lecture might hear the explanation, but what happens when important information is communicated primarily through visuals?\n\nFor blind and low-vision learners, that missing context can make learning unnecessarily difficult.\n\nThis is the problem we're exploring with **SeeHere**.\n\nSeeHere is an AI-powered accessibility project designed to make educational lecture videos more accessible through automatically generated audio descriptions.\n\nThe idea is straightforward:\n\nInstead of expecting learners to interpret every visual element themselves, SeeHere aims to describe important visual information through synchronized audio.\n\nThe goal isn't to replace the lecturer's narration or summarize the entire video.\n\nIt's to communicate the visual information that the original narration might leave unexplained.\n\nFor example, when a photosynthesis lecture displays a diagram of a chloroplast, an audio description could explain the visible structures and their relationships, giving the listener additional context about what's happening on screen.\n\nSeeHere brings together several technologies to approach the problem as a complete processing pipeline.\n\nThe architecture is designed around the following stages:\n\nThe process begins when a user uploads an educational video.\n\nThe backend handles the video and prepares it for visual analysis. Relevant frames can then be extracted for further processing.\n\nThe important part is identifying which visual moments actually deserve an audio description.\n\nDescribing every frame would create unnecessary interruptions and make the experience overwhelming.\n\nThis is where multimodal AI becomes useful.\n\nWe use Gemini for visual and contextual understanding.\n\nThe objective is to identify meaningful information in lecture frames, including diagrams, illustrations, visual demonstrations, and other educational content.\n\nA description should communicate what matters in the context of the lesson, rather than simply listing objects visible in an image.\n\nEducational slides frequently contain important information in the form of labels, equations, annotations, and diagrams.\n\nThis is why OCR is an important part of our approach.\n\nWe use Tesseract OCR to extract visible text and help verify information identified during visual analysis.\n\nCombining contextual understanding with extracted text helps us work towards more informative and grounded descriptions.\n\nGenerating an audio description is only part of the problem.\n\n**When should that description be played?**\n\nA description that overlaps with an important explanation from the lecturer could make the video harder to understand.\n\nThis introduces a scheduling problem.\n\nSeeHere explores Google OR-Tools and its CP-SAT solver to model the placement of descriptions within suitable time windows.\n\nThe intention is to find placements that respect timing constraints and reduce unnecessary interference with the original lecture.\n\nThis is one of the interesting engineering challenges in the project: descriptions need to be useful, but their timing matters just as much as their content.\n\nOnce the descriptions have been generated and scheduled, they need to become audio.\n\nWe use smallest.ai as the text-to-speech component in the proposed pipeline.\n\nThe generated speech provides the audio layer that communicates the missing visual context.\n\nThe final stage focuses on making the descriptions usable alongside the original video.\n\nWebVTT provides a timed format for representing synchronized description segments.\n\nThe objective is to produce an output where learners can understand not only what is happening visually, but also when that information is relevant.\n\nThe project brings together frontend development, backend processing, multimodal AI, optimization, and audio generation.\n\n| Component | Technology | \n|---|---|\n| Frontend | Next.js | \n| Backend | FastAPI | \n| Visual understanding | Google Gemini | \n| OCR | Tesseract | \n| Scheduling and optimization | Google OR-Tools CP-SAT | \n| Text-to-speech | smallest.ai | \n| Timed output | WebVTT | \n| Database | PostgreSQL | \n| Infrastructure | Vultr | \n\nEach component has a specific responsibility in the overall workflow.\n\nThe interesting part isn't simply using multiple AI services. It's coordinating them into a pipeline where visual understanding, description generation, timing, and audio output all need to work together.\n\nThere are three engineering challenges we're particularly interested in.\n\n**1. Context matters more than object detection.**\n\nKnowing that a frame contains a diagram isn't enough.\n\nAn educational description needs to communicate the meaning of the visual information in the context of the lecture.\n\n**2. Descriptions need to respect the original narration.**\n\nAudio descriptions should complement the lecturer, not compete with them.\n\nThis makes scheduling and synchronization an important part of the problem rather than an afterthought.\n\n**3. Accessibility requires an end-to-end experience.**\n\nGenerating a description is not the finish line.\n\nIt needs to be transformed into usable audio, synchronized with the relevant part of the video, and delivered in a way that supports the learning experience.\n\nThese considerations shape the direction of SeeHere.\n\nOur goal is to make accessibility a part of the educational video experience rather than something that needs to be manually added to every lecture.\n\nImagine uploading a recorded university lecture and receiving an accessible version that explains important diagrams, visual demonstrations, and other information that might otherwise be missed.\n\nThat's the experience we're working towards.\n\nSeeHere is our attempt to explore how multimodal AI and constraint-based scheduling can work together to address a real accessibility problem.\n\nThere is still plenty to improve, especially around description quality, contextual accuracy, timing, and the overall listening experience.\n\nBut we believe this is a problem worth solving.\n\n**Because understanding a lecture shouldn't depend entirely on being able to see it.**\n\n*SeeHere is a project developed as part of HackGlobal. We're exploring the intersection of AI, accessibility, and educational technology to make visual learning content more inclusive.*", "url": "https://wpnews.pro/news/seehere-making-educational-videos-more-accessible-with-ai-powered-audio", "canonical_source": "https://dev.to/priyansh_narang_c1b96ab75/seehere-making-educational-videos-more-accessible-with-ai-powered-audio-descriptions-5c12", "published_at": "2026-11-15 04:42:11+00:00", "updated_at": "2026-11-15 04:51:56.964973+00:00", "lang": "en", "topics": ["artificial-intelligence", "generative-ai", "computer-vision", "ai-tools", "natural-language-processing"], "entities": ["SeeHere", "Gemini", "Google", "Tesseract OCR", "OR-Tools", "smallest.ai"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/seehere-making-educational-videos-more-accessible-with-ai-powered-audio", "markdown": "https://wpnews.pro/news/seehere-making-educational-videos-more-accessible-with-ai-powered-audio.md", "text": "https://wpnews.pro/news/seehere-making-educational-videos-more-accessible-with-ai-powered-audio.txt", "jsonld": "https://wpnews.pro/news/seehere-making-educational-videos-more-accessible-with-ai-powered-audio.jsonld"}}