SeeHere: Making Educational Videos More Accessible with AI-Powered Audio Descriptions

A developer built SeeHere, an AI-powered accessibility pipeline that automatically generates and inserts audio descriptions into educational lecture videos for blind and low-vision learners. The system uses Gemini for visual and contextual understanding of extracted frames, Tesseract OCR to capture on-screen labels and equations, Google OR-Tools' CP-SAT solver to schedule descriptions into non-interfering time windows, and smallest.ai for text-to-speech. The project targets visual information the lecturer's narration leaves unexplained, such as diagrams and annotations, rather than summarizing the whole video.

Educational videos have become an essential part of how people learn. From online courses and university lectures to tutorials and recorded presentations, a lot of knowledge is now delivered through video. But there's a problem we don't talk about enough. A significant amount of information in educational videos is visual. Think about a biology lecture explaining photosynthesis. The instructor might be talking about chlorophyll while a diagram illustrates the light-dependent reactions, labels identify different structures, and arrows explain how energy flows through the process. Someone listening to the lecture might hear the explanation, but what happens when important information is communicated primarily through visuals? For blind and low-vision learners, that missing context can make learning unnecessarily difficult. This is the problem we're exploring with SeeHere . SeeHere is an AI-powered accessibility project designed to make educational lecture videos more accessible through automatically generated audio descriptions. The idea is straightforward: Instead of expecting learners to interpret every visual element themselves, SeeHere aims to describe important visual information through synchronized audio. The goal isn't to replace the lecturer's narration or summarize the entire video. It's to communicate the visual information that the original narration might leave unexplained. For example, when a photosynthesis lecture displays a diagram of a chloroplast, an audio description could explain the visible structures and their relationships, giving the listener additional context about what's happening on screen. SeeHere brings together several technologies to approach the problem as a complete processing pipeline. The architecture is designed around the following stages: The process begins when a user uploads an educational video. The backend handles the video and prepares it for visual analysis. Relevant frames can then be extracted for further processing. The important part is identifying which visual moments actually deserve an audio description. Describing every frame would create unnecessary interruptions and make the experience overwhelming. This is where multimodal AI becomes useful. We use Gemini for visual and contextual understanding. The objective is to identify meaningful information in lecture frames, including diagrams, illustrations, visual demonstrations, and other educational content. A description should communicate what matters in the context of the lesson, rather than simply listing objects visible in an image. Educational slides frequently contain important information in the form of labels, equations, annotations, and diagrams. This is why OCR is an important part of our approach. We use Tesseract OCR to extract visible text and help verify information identified during visual analysis. Combining contextual understanding with extracted text helps us work towards more informative and grounded descriptions. Generating an audio description is only part of the problem. When should that description be played? A description that overlaps with an important explanation from the lecturer could make the video harder to understand. This introduces a scheduling problem. SeeHere explores Google OR-Tools and its CP-SAT solver to model the placement of descriptions within suitable time windows. The intention is to find placements that respect timing constraints and reduce unnecessary interference with the original lecture. This is one of the interesting engineering challenges in the project: descriptions need to be useful, but their timing matters just as much as their content. Once the descriptions have been generated and scheduled, they need to become audio. We use smallest.ai as the text-to-speech component in the proposed pipeline. The generated speech provides the audio layer that communicates the missing visual context. The final stage focuses on making the descriptions usable alongside the original video. WebVTT provides a timed format for representing synchronized description segments. The objective is to produce an output where learners can understand not only what is happening visually, but also when that information is relevant. The project brings together frontend development, backend processing, multimodal AI, optimization, and audio generation. | Component | Technology | |---|---| | Frontend | Next.js | | Backend | FastAPI | | Visual understanding | Google Gemini | | OCR | Tesseract | | Scheduling and optimization | Google OR-Tools CP-SAT | | Text-to-speech | smallest.ai | | Timed output | WebVTT | | Database | PostgreSQL | | Infrastructure | Vultr | Each component has a specific responsibility in the overall workflow. The interesting part isn't simply using multiple AI services. It's coordinating them into a pipeline where visual understanding, description generation, timing, and audio output all need to work together. There are three engineering challenges we're particularly interested in. 1. Context matters more than object detection. Knowing that a frame contains a diagram isn't enough. An educational description needs to communicate the meaning of the visual information in the context of the lecture. 2. Descriptions need to respect the original narration. Audio descriptions should complement the lecturer, not compete with them. This makes scheduling and synchronization an important part of the problem rather than an afterthought. 3. Accessibility requires an end-to-end experience. Generating a description is not the finish line. It needs to be transformed into usable audio, synchronized with the relevant part of the video, and delivered in a way that supports the learning experience. These considerations shape the direction of SeeHere. Our goal is to make accessibility a part of the educational video experience rather than something that needs to be manually added to every lecture. Imagine uploading a recorded university lecture and receiving an accessible version that explains important diagrams, visual demonstrations, and other information that might otherwise be missed. That's the experience we're working towards. SeeHere is our attempt to explore how multimodal AI and constraint-based scheduling can work together to address a real accessibility problem. There is still plenty to improve, especially around description quality, contextual accuracy, timing, and the overall listening experience. But we believe this is a problem worth solving. Because understanding a lecture shouldn't depend entirely on being able to see it. SeeHere is a project developed as part of HackGlobal. We're exploring the intersection of AI, accessibility, and educational technology to make visual learning content more inclusive.