{"slug": "airsketch", "title": "AirSketch", "summary": "A team of student developers at HackGB 2026, hosted at the University of Wisconsin–Green Bay on October 17–18, built AirSketch, an AI-powered smart whiteboard that combines webcam-based hand-gesture drawing, speech recognition, and generative AI to convert spoken explanations into structured notes. The system runs two pipelines: a gesture pipeline that maps hand landmarks to a digital canvas, and a speech pipeline that transcribes audio and passes it to a model that organizes it into readable notes. The team said the hardest engineering problems were translating camera coordinates to canvas coordinates and stabilizing drawing against jittery landmark detection.", "body_md": "What if you could write on a whiteboard without touching it, explain your ideas out loud, and automatically turn everything into organized notes?\n\nThat's the idea behind our AI-powered smart whiteboard, a project we worked on at HackGB 2026, hosted at the University of Wisconsin–Green Bay on October 17–18.\n\nWe wanted to explore how computer vision, speech recognition, and generative AI could work together to create a more interactive learning experience.\n\nAnd we had just one hackathon weekend to bring it all together!\n\nThink about a typical classroom.\n\nA professor explains a complicated concept while writing on a whiteboard. Students try to listen, understand, and take notes simultaneously.\n\nSometimes, important explanations get missed because everyone is busy copying what's on the board.\n\nWe started with a simple question:\n\nWhat if a whiteboard could understand both what you're drawing and what you're saying?\n\nInstead of treating drawing and speaking as separate activities, we imagined a system that combines them into a single learning experience.\n\nOur goal was to build a whiteboard that could:\n\nOur project brings together three core capabilities.\n\nUsing computer vision, we created a virtual drawing interface controlled by hand movements.\n\nThe webcam captures the user's hand, and the gesture-recognition pipeline tracks hand landmarks to interpret movement.\n\nInstead of using a physical marker or touchscreen, users can interact with a digital canvas through gestures.\n\nThe interesting challenge wasn't simply detecting a hand. It was translating continuously changing hand positions into meaningful drawing actions.\n\nWe had to think about coordinate mapping, gesture states, and how to prevent unintended drawing.\n\nDrawing is only part of teaching. The explanation behind a diagram is often just as important as the diagram itself.\n\nWe incorporated speech recognition to capture spoken explanations and convert them into text.\n\nThis creates a textual record of the ideas being discussed, reducing the need to manually transcribe everything.\n\nRaw speech transcripts aren't always easy to study.\n\nThey can contain repetition, incomplete sentences, and conversational language.\n\nOur idea was to use generative AI to turn these transcripts into clearer, structured notes.\n\nFor example, imagine someone explaining machine learning while drawing a neural network.\n\nThe system can use the spoken explanation to produce organized notes covering concepts such as input layers, hidden layers, activation functions, and model training.\n\nThis is where the project becomes more than a virtual whiteboard: it becomes a potential AI-powered learning assistant.\n\nAt a high level, the system contains two main processing pipelines.\n\nGesture pipeline:\n\nWebcam → Hand Detection → Landmark Tracking → Gesture Interpretation → Digital Canvas\n\nSpeech pipeline:\n\nMicrophone → Speech Recognition → Transcript → AI Processing → Structured Notes\n\nThese pipelines handle different types of information but contribute to the same learning experience.\n\nThe gesture-controlled whiteboard uses a sequence of operations:\n\nOne important engineering consideration is the difference between camera coordinates and screen coordinates.\n\nThe camera captures images at one resolution, while the drawing canvas may have different dimensions.\n\nTo make the interaction feel natural, the system must correctly translate movement between these coordinate systems.\n\nAnother challenge is maintaining stable drawing behavior. Even small variations in landmark detection can make a line appear shaky.\n\nThese details matter because a gesture-controlled application must feel responsive, not just technically functional.\n\nThe speech-processing side follows a different workflow.\n\nFirst, spoken audio is converted into text.\n\nThen, the transcript is provided to an AI model with instructions to organize the content into readable notes.\n\nA useful note-generation pipeline should preserve the original meaning, identify important concepts, and avoid introducing unsupported information.\n\nThe goal isn't simply to summarize everything. It's to produce notes that someone can actually use when reviewing a lesson.\n\nOne of the most valuable parts of HackGB was collaborating under a tight deadline.\n\nWe divided the project into separate responsibilities so development could happen in parallel.\n\nI focused on the gesture-controlled whiteboard and computer vision functionality, while my teammate worked on speech transcription and AI-generated notes.\n\nThis separation allowed us to work independently before integrating the components.\n\nIt also taught us an important software engineering lesson:\n\nBuilding individual features is only half the challenge. Making them work together is where system design becomes important.\n\nWe had to consider how different components would communicate, how users would interact with the combined system, and how to keep the demonstration understandable.\n\nA system can detect hand landmarks correctly and still provide a frustrating user experience.\n\nResponsiveness, stability, and gesture interpretation are equally important.\n\nWe learned to think about the entire interaction pipeline rather than focusing only on detection.\n\nWhen multiple people build different components, consistent interfaces and clear responsibilities become essential.\n\nEven in a short hackathon, thinking about component boundaries early can save valuable development time.\n\nConverting speech to text is useful, but turning that text into educational notes requires additional design decisions.\n\nThe AI needs to identify the most important information without losing context.\n\nThis reinforced the importance of prompt design and output quality when building AI applications.\n\nHackathons aren't only about how much code you write.\n\nYou also need to explain the problem, demonstrate the solution, and communicate why the technology matters.\n\nWe learned that a focused, understandable demonstration can be more compelling than a long list of features.\n\nWe built this project with education in mind, but the underlying idea has applications beyond classrooms.\n\nImagine using a similar system for:\n\nThese are future possibilities rather than features we've fully implemented, but they show how the concept could evolve.\n\nA hackathon prototype is only the beginning.\n\nSome improvements we'd like to explore include:\n\nOne especially interesting direction is **multimodal understanding**.\n\nRather than processing drawings and speech separately, a future system could reason about both together.\n\nFor example, when an instructor points to a specific part of a diagram and says, \"This layer transforms the input,\" the AI could connect that explanation to the corresponding visual element.\n\nThat would move the project closer to a genuinely context-aware learning assistant.\n\nHackGB 2026 reminded me why I enjoy building AI applications.\n\nLearning about computer vision, machine learning, and generative AI is exciting, but applying those concepts to a real problem is a completely different experience.\n\nWorking under a deadline pushed us to prioritize, experiment, debug, collaborate, and communicate our ideas.\n\nAnd that's what made the experience memorable.\n\nThe most exciting part of AI isn't just what a model can do. It's what we can build when we connect different technologies to solve real problems.\n\nA big thank you to the HackGB organizers, mentors, sponsors, volunteers, and everyone who helped make the event possible. Hackathons are a great reminder that you don't need months to start building something meaningful.", "url": "https://wpnews.pro/news/airsketch", "canonical_source": "https://dev.to/lenin_goud_809af2a4a25dc0/airsketch-2pb9", "published_at": "2026-10-22 19:32:10+00:00", "updated_at": "2026-10-22 19:40:43.933145+00:00", "lang": "en", "topics": ["computer-vision", "generative-ai", "ai-tools", "natural-language-processing"], "entities": ["AirSketch", "HackGB 2026", "University of Wisconsin–Green Bay"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/airsketch", "markdown": "https://wpnews.pro/news/airsketch.md", "text": "https://wpnews.pro/news/airsketch.txt", "jsonld": "https://wpnews.pro/news/airsketch.jsonld"}}