Clipto enables ChatGPT and Claude to search and analyze years of people’s meetings and video calls
Artificial intelligence assistants such as ChatGPT and Claude have become essential tools for millions of workers worldwide, helping them compose emails in seconds, generate complex code and draft reports by looking at whatever data that person has access to.
But there’s one thing that’s missing from those chatbots’ knowledge. Ask them what was discussed in the previous week’s team meeting or on yesterday’s call with a client, and it will be stumped.
It’s this shortcoming that a startup called Clipto Inc. is trying to solve with the launch of a new AI chatbot memory platform called Clipto Studio, which enables AI assistants to listen in on calls, presentations and meetings and search through them or analyze them upon request.
Clipto argues that this is going to be an important capability as AI becomes more pervasive in daily work. Though AI chatbots can sift through email inboxes and corporate databases in seconds, that’s not where many companies’ most valuable knowledge is actually stored. Rather, Clipto says the most important business insights are trapped in raw video and audio files.
Most of the big decisions are made during Zoom calls, customer interviews and team meetings, but the details of those discussions are stored in heavy, unstructured data formats that are incredibly expensive for AI to process. But AI can potentially provide much more meaningful help to workers if it has a way to access this knowledge.
That’s where Clipto Studio comes in. It’s a native skill set that integrates directly with ChatGPT, Claude and other AI assistants using the Model Context Protocol, powered by proprietary models that run locally on user’s Mac and Windows computers.
The software works by pre-indexing speech, individual speaker’s voices, facial expressions and other on-screen visuals before transforming the raw footage into a structured and easily searchable dataset that’s optimized for AI chatbots. Then, when the user asks a question about one of their recent conversations, or even one from two years ago, it can quickly find the answers it needs by searching through the metadata of those recordings.
Clipto founder and Chief Executive Henry Kang said AI knows basically everything that has been posted online, but has barely scratched the surface of the terabytes of audio and video data that’s sitting in people’s hard drives.
“The next generation of AI will need to understand the conversations, experiences and knowledge that never makes it into a text-based document,” he said. “That’s the future we’re building, giving AI the memory it needs to move beyond answering questions from the internet and start getting real world done with what we ourselves have seen and heard.”
Clipto Studio can be thought of as a kind of instant-replay service. For instance, a management consultant could ask ChatGPT to pinpoint the exact time when a project’s scope changed by looking at more than six month’s worth of calls and meetings, and it will surface the exact moment when that occurred. Alternatively, a researcher might prompt Claude to identify the most common problems with a particular product or service by looking at thousands of customer calls, and pull out direct quotes that illustrate the depth of those issues.
There are other use cases too. For instance, users could ask Clipto Studio to look at dozens of hours of event presentations they’ve attended and create a highlight reel that surfaces the most interesting or relevant moments from them.
Clipto Studio’s official launch is a big moment for the startup, which recently signaled its ambitions with the hiring of former Google LLC research scientist Dr. Mei Han as its new chief scientist. Han, who holds a doctorate in robotics from Carnegie Mellon University, has been tasked with leading the development of Clipto’s lightweight multimodal AI architecture.
Han said she believes in Clipto’s mission because it’s trying to help AI perceive the world in the same way as humans do, by seeing and listening to what’s going on around them.
“The challenge isn’t just recognizing speech or identifying what’s on screen,” she said. “It’s connecting that information across thousands of recordings so AI can understand the context and use it. We’re building the technology to make video and audio as accessible and as useful to AI as text, opening up an entirely new generation of applications.”
Images: Clipto
Support our mission to keep content open and free by engaging with theCUBE community. Join theCUBE’s Alumni Trust Network, where technology leaders connect, share intelligence and create opportunities.
- 15M+ viewers of theCUBE videos , powering conversations across AI, cloud, cybersecurity and more
- 11.4k+ theCUBE alumni — Connect with more than 11,400 tech and business leaders shaping the future through a unique trusted-based network
Are you an AWS customer? Support SiliconANGLE financially by buying your AWS services from our Marketplace portal page and links: https://siliconangle.com/aws-marketplace/
About SiliconANGLE Media
theCUBE AIand theCUBE SuperStudios — with flagship locations in Silicon Valley and the New York Stock Exchange — SiliconANGLE Media operates at the intersection of media, technology and AI.
Founded by tech visionaries John Furrier and Dave Vellante, SiliconANGLE Media has built a dynamic ecosystem of industry-leading digital media brands that reach 15+ million elite tech professionals. Our new proprietary theCUBE AI Video Cloud is breaking ground in audience interaction, leveraging theCUBEai.com neural network to help technology companies make data-driven decisions and stay at the forefront of industry conversations.