Fueled by thousands of photographs submitted by more than 100 students, Bruce Maxwell’s machine vision model automatically removes shadows from images to depict their true colors in any lighting condition.
SEATTLE – Despite rapid improvement in computer vision models, which can perform accurate facial recognition or read the text printed on signs, their ability to distinguish the subtle visual qualities of objects, colors and light remains no match for the human eye, technology experts say.
But new research from Bruce Maxwell, director of computing programs in the Khoury College of Computer Sciences at Northeastern University Seattle, is seeking to bring algorithms closer than ever before with a novel approach to image processing.
Fueled by thousands of photographs submitted to him by more than 100 of his computer vision students from campuses across Northeastern’s global network, Maxwell’s machine vision model accurately depicts the true colors of objects even when they are shrouded in shadows or captured in low lighting.
Northeastern experts in computer vision say this supremely accurate, automated object- and color-detection algorithm could be an antidote to some of the challenges in industries that rely on pixel-perfect images – from autonomous vehicles to 3D video game art.
On sunny days when the road is blanketed by shadows of nearby trees or buildings, the cameras that serve as the eyes for autonomous vehicles – like those from Tesla – can have trouble discerning the edges and colors of objects like curbs and cyclists. They brake early and often as a measure of safety, even when what they see in front of them isn’t an obstacle, but a shadow, according to Xian Li, professor of electrical and computer engineering on Northeastern’s Seattle campus.
“People don’t have this problem. We know a shady patch of road is still the same road,” said Li, who conducts his own image and sensor data research from behind the wheel of Northeastern Seattle’s electric Ford Mustang. “[Maxwell’s] approach teaches cameras the same thing by learning how light changes color between sun and shade.”
Part of the reason the human eye is seldom confused about an object’s true shape and color in shadow is because, unlike cameras, it is a logarithmic sensor, Maxwell said. When humans with normal vision register photons of light, they do so on a logarithmic curve, which prevents bright highlights from blowing out to complete whiteness and dark features from crushing into pure blackness.
Cameras take linear images, which artificially compress light intensity across the scene, giving darker shades and brighter highlights an unnaturally equal share of data.
“You either can’t see in the shadows, or all the stuff that’s fully lit is blown out,” Maxwell said.
The only way to get reasonably color-accurate pictures is with a device that uses multiple cameras to take photographs at different exposures and combine them together in a logarithmic way, he said.
Creating truly advanced machine vision models – rule-based processing systems that enable computers to capture, process and analyze images to automate decision-making in physical environments – means training them on the most accurate data, Maxwell said. He found that the most true-color results came when he trained his model using digital images with RGB (red, green and blue) values stored in a logarithmic scale, more closely mimicking the process of the human eye.
“For something like autonomous driving, where you’re outside in a high dynamic-range environment…you have to be able to use very efficient models,” he said. “So, if you can get a little bit better performance with the same model by just changing what data you’re feeding it, that’s useful.”
In his recent research, Maxwell investigated this model’s ability to look at an image with a shadowy object and, without any distortion or loss of fidelity, automatically brighten it to display the entire object as it would look in completely uniform light – something he said had not been achieved before.
The project utilized a dataset of approximately 25,000 logarithmic images captured by dozens of Maxwell’s computer vision students over the course of four years with their own smartphones and cameras. Students contributed from across Northeastern’s global network – including the Boston, Seattle, Vancouver, Silicon Valley, Miami, Arlington and Portland campuses – capturing photos of the same streets, pets, gardens or signs in the fall, spring and summer seasons, he said.
This automated shadow-removal algorithm could help photographers correct their images with more accuracy and authentic color richness than mainstream AI models and smartphone editing software, Maxwell said. For video game art designers who want to insert 3D objects from the real world into games – like a coffee cup – Maxwell’s model can also eliminate the need to take hundreds of photos from all angles in perfect lighting before stitching them together. The designer could simply take a short video of the cup, turning it around in their hand, then let the model remove all of the shadows itself and upload a game-ready recreation in perfect color, he said.
Maxwell said that he hopes to one day implement the model into autonomous vehicles and run it in real time, giving the cars a clearer picture of the road by completely removing the shadows from surrounding objects and people.
Li said that Maxwell’s work could help cameras perform more reliably in real driving conditions.
“I would love to test it on the street data we collect with our research car here in Seattle,” he said.
Peter Rubinstein is a news reporter at Northeastern Global News.