What if you could write on a whiteboard without touching it, explain your ideas out loud, and automatically turn everything into organized notes?
That's the idea behind our AI-powered smart whiteboard, a project we worked on at HackGB 2026, hosted at the University of Wisconsin–Green Bay on October 17–18.
We wanted to explore how computer vision, speech recognition, and generative AI could work together to create a more interactive learning experience.
And we had just one hackathon weekend to bring it all together!
Think about a typical classroom.
A professor explains a complicated concept while writing on a whiteboard. Students try to listen, understand, and take notes simultaneously.
Sometimes, important explanations get missed because everyone is busy copying what's on the board.
We started with a simple question:
What if a whiteboard could understand both what you're drawing and what you're saying?
Instead of treating drawing and speaking as separate activities, we imagined a system that combines them into a single learning experience.
Our goal was to build a whiteboard that could:
Our project brings together three core capabilities.
Using computer vision, we created a virtual drawing interface controlled by hand movements. The webcam captures the user's hand, and the gesture-recognition pipeline tracks hand landmarks to interpret movement.
Instead of using a physical marker or touchscreen, users can interact with a digital canvas through gestures.
The interesting challenge wasn't simply detecting a hand. It was translating continuously changing hand positions into meaningful drawing actions.
We had to think about coordinate mapping, gesture states, and how to prevent unintended drawing.
Drawing is only part of teaching. The explanation behind a diagram is often just as important as the diagram itself.
We incorporated speech recognition to capture spoken explanations and convert them into text.
This creates a textual record of the ideas being discussed, reducing the need to manually transcribe everything.
Raw speech transcripts aren't always easy to study.
They can contain repetition, incomplete sentences, and conversational language.
Our idea was to use generative AI to turn these transcripts into clearer, structured notes.
For example, imagine someone explaining machine learning while drawing a neural network. The system can use the spoken explanation to produce organized notes covering concepts such as input layers, hidden layers, activation functions, and model training.
This is where the project becomes more than a virtual whiteboard: it becomes a potential AI-powered learning assistant.
At a high level, the system contains two main processing pipelines.
Gesture pipeline:
Webcam → Hand Detection → Landmark Tracking → Gesture Interpretation → Digital Canvas
Speech pipeline:
Microphone → Speech Recognition → Transcript → AI Processing → Structured Notes
These pipelines handle different types of information but contribute to the same learning experience.
The gesture-controlled whiteboard uses a sequence of operations:
One important engineering consideration is the difference between camera coordinates and screen coordinates.
The camera captures images at one resolution, while the drawing canvas may have different dimensions.
To make the interaction feel natural, the system must correctly translate movement between these coordinate systems.
Another challenge is maintaining stable drawing behavior. Even small variations in landmark detection can make a line appear shaky.
These details matter because a gesture-controlled application must feel responsive, not just technically functional.
The speech-processing side follows a different workflow.
First, spoken audio is converted into text.
Then, the transcript is provided to an AI model with instructions to organize the content into readable notes.
A useful note-generation pipeline should preserve the original meaning, identify important concepts, and avoid introducing unsupported information.
The goal isn't simply to summarize everything. It's to produce notes that someone can actually use when reviewing a lesson.
One of the most valuable parts of HackGB was collaborating under a tight deadline.
We divided the project into separate responsibilities so development could happen in parallel.
I focused on the gesture-controlled whiteboard and computer vision functionality, while my teammate worked on speech transcription and AI-generated notes.
This separation allowed us to work independently before integrating the components.
It also taught us an important software engineering lesson:
Building individual features is only half the challenge. Making them work together is where system design becomes important.
We had to consider how different components would communicate, how users would interact with the combined system, and how to keep the demonstration understandable.
A system can detect hand landmarks correctly and still provide a frustrating user experience.
Responsiveness, stability, and gesture interpretation are equally important.
We learned to think about the entire interaction pipeline rather than focusing only on detection.
When multiple people build different components, consistent interfaces and clear responsibilities become essential.
Even in a short hackathon, thinking about component boundaries early can save valuable development time.
Converting speech to text is useful, but turning that text into educational notes requires additional design decisions.
The AI needs to identify the most important information without losing context.
This reinforced the importance of prompt design and output quality when building AI applications.
Hackathons aren't only about how much code you write.
You also need to explain the problem, demonstrate the solution, and communicate why the technology matters.
We learned that a focused, understandable demonstration can be more compelling than a long list of features.
We built this project with education in mind, but the underlying idea has applications beyond classrooms.
Imagine using a similar system for:
These are future possibilities rather than features we've fully implemented, but they show how the concept could evolve.
A hackathon prototype is only the beginning.
Some improvements we'd like to explore include:
One especially interesting direction is multimodal understanding.
Rather than processing drawings and speech separately, a future system could reason about both together.
For example, when an instructor points to a specific part of a diagram and says, "This layer transforms the input," the AI could connect that explanation to the corresponding visual element. That would move the project closer to a genuinely context-aware learning assistant.
HackGB 2026 reminded me why I enjoy building AI applications.
Learning about computer vision, machine learning, and generative AI is exciting, but applying those concepts to a real problem is a completely different experience.
Working under a deadline pushed us to prioritize, experiment, debug, collaborate, and communicate our ideas.
And that's what made the experience memorable.
The most exciting part of AI isn't just what a model can do. It's what we can build when we connect different technologies to solve real problems.
A big thank you to the HackGB organizers, mentors, sponsors, volunteers, and everyone who helped make the event possible. Hackathons are a great reminder that you don't need months to start building something meaningful.