OpenCV explained: The computer vision library for AI apps

OpenCV, the computer vision library first released in 2000 as an Intel Research project, has reached version 5.0, a significant cleanup and reorganization of the toolkit with a modernized API structure and better compatibility with today's machine learning models and workflows. The library is now stewarded by a non-profit foundation and consists of Apache 2-licensed C++ libraries with Python, Java, and JavaScript interfaces, plus a separate wasmCV project providing a WebAssembly interface. OpenCV runs across a broad range of architectures and uses a separate hardware acceleration layer so programs automatically use available acceleration techniques such as AVX-512 without being aware of them.

Before the rise of all-in-one frontier models, artificial intelligence https://www.infoworld.com/article/4061121/a-brief-history-of-ai.html and machine learning https://www.infoworld.com/article/3214424/what-is-machine-learning-intelligence-derived-from-data.html libraries tended to focus on highly specialized jobs. Even today this is still a powerful and useful approach: the more specialized the tool, the better the job it will do for its narrowly defined purpose. One category for such tooling is computer vision https://www.infoworld.com/article/2259830/what-is-computer-vision-ai-for-images-and-video.html : real-time analysis of video or still images. Facial recognition, OCR, motion tracking, object detection, augmented reality, and gesture detection are all common examples of computer vision jobs. The OpenCV library https://opencv.org is easily the most commonly used and long-lived toolkit for computer vision work. OpenCV saves you the trouble of writing from scratch the software needed for any of the above-mentioned tasks. It also gives you a handy bundle of image and video processing tools that can be used in most any project that might need them. It’s also powerful way to take existing or newly trained machine learning models for vision tasks and put them to work for you. OpenCV’s original release dates back to the year 2000, when it was originally an Intel Research project. Today the project is under the stewardship of a non-profit foundation, and has gone through several major point revisions. The most recent, OpenCV 5.0 https://opencv.org/opencv-5/ , is a significant cleanup and reorganization of the toolkit, with a modernized API structure and better compatibility with the kinds of models and workflows used in machine learning today. OpenCV is at core a collection of Apache 2-licensed C++ libraries. You can link against those libraries in your own C++ application, or work a different language and use OpenCV through an interface. Python https://www.infoworld.com/article/2253770/what-is-python-powerful-intuitive-programming.html , Java https://www.infoworld.com/article/4202901/jdk-27-the-new-features-of-java-27.html , and JavaScript https://www.infoworld.com/article/2263137/what-is-javascript-the-full-stack-programming-language.html are all supported, and a separate wasmCV project https://github.com/wasmvision/wasmcv provides a WebAssembly https://www.infoworld.com/article/2255892/what-is-webassembly-the-next-generation-web-platform-explained.html interface. OpenCV is designed to run across a broad range of architectures. It also uses a separate hardware acceleration layer https://docs.opencv.org/5.0/main modules/hal.html , so every platform OpenCV runs on can use the acceleration techniques available for it e.g., AVX-512 . Programs written with OpenCV need not be aware of any hardware acceleration techniques; OpenCV automatically selects the best option on the current system. It’s also possible to build your own acceleration layer https://docs.opencv.org/5.0/main modules/hal.html implementing-a-custom-hal if you want to deploy OpenCV on custom silicon or tweak an existing deployment. OpenCV’s modules https://docs.opencv.org/5.0/main modules/api root.html are organized by major tasks in the general field of computer vision. Most of them are self-explanatory. For example, the objdetect module is the best place to start if you’re putting together a straightforward object-detection system. Some high-level OpenCV modules might appear to overlap, but they are meant for different tasks. For instance, the video I/O module https://docs.opencv.org/5.0/main modules/videoio.html api-videoio gives you a convenient interface for reading and writing video files or image sequences https://opencv.org/reading-and-writing-videos-using-opencv/ . But there’s also a video analysis https://docs.opencv.org/5.0/main modules/video.html module, which handles a different high-level use case: performing motion analysis or object tracking on video or image sets. If you want to generate video with or without a large language model https://www.infoworld.com/article/2335213/large-language-models-the-foundations-of-generative-ai.html , use the video I/O module; if you want to analyze video i.e., track the movement of objects , use the video analysis module. Many of OpenCV’s functions can be hooked into and customized. For instance, object detection https://docs.opencv.org/5.0/tutorials/objdetect/table of content objdetect.html is a high-level module that includes barcode detection https://docs.opencv.org/5.0/tutorials/objdetect/barcode detect and decode/barcode detect and decode.html . Support for common barcodes EAN-8, EAN-13, UPC-A, and UPC-E is already available. But the barcode detector can optionally use an ONNX-format model for detection, so you could have it detect a bespoke variety of barcode, or attempt to decipher damaged or distorted barcodes based on a model. OpenCV can work with external models for many other tasks, too, and without having to set up extra software to do it. The DNN module https://opencv.org/opencv-dnn-module/ lets you load third-party, pretrained models for tasks like object detection and use them to serve predictions from still-image or video input. You can use models created with TensorFlow or PyTorch, or any other framework that can export a model in the common ONNX format. OpenCV also curates its own collection of useful models, called the OpenCV zoo https://huggingface.co/collections/opencv/opencv-zoo-6842b10ecefbfc1e8aaa59ad , on Hugging Face. The zoo is another good source for models and examples for common tasks, such as deblurring images https://huggingface.co/opencv/deblurring nafnet , pose estimation https://huggingface.co/opencv/pose estimation mediapipe , and object detection https://github.com/opencv/opencv/wiki/TensorFlow-Object-Detection-API . One major advantage of OpenCV for serving inferences from these models is that you don’t need anything except the model and what OpenCV itself provides. You do not have to set up a separate runtime for the model to serve inferences from it. This makes it easier to stand up a focused pipeline for a particular task. This pipeline could in turn later be attached to, or built out into, a larger and more complex pipeline if needed. But for basic inference, you can start with OpenCV. OpenCV also comes with a large catalog of extra modules https://docs.opencv.org/5.0/extra modules/api root.html for other, highly specific tasks: direct access to encoding/decoding of video, adding text to images, video stabilization, text detection in scenes, and many more. One of these modules, datasets https://docs.opencv.org/5.0/extra modules/datasets.html , provides a way to load well-known, pretrained datasets for common tasks face recognition or gesture recognition, pedestrian detection, etc. that can be used to make predictions or to generate performance benchmarks for those tasks. Another convenient OpenCV feature is the high-level GUI module https://docs.opencv.org/5.0/main modules/highgui.html , which lets you create simple GUIs—windowing, buttons, images, mouse operations—with nothing but OpenCV. This is handy not only for quick prototypes, but even for creating a simple application without having to reach for an entire UI framework like GTK to create the front end. High-level GUI is no substitute for GTK or app frameworks like Electron, but it isn’t meant to be. OpenCV’s big benefits go beyond giving you a prepackaged set of algorithms to perform computer-vision work. You can use OpenCV also to serve predictions from models for related tasks, like object recognition, without having to spin up extra infrastructure. And you can use it to build basic GUIs for those tasks with no extra libraries—quite handy to have when working with images or video. Finally, OpenCV’s decades of development and ongoing advancement make it a solid base to build on going forward.