In my previous experiment I compared the performance of Vision Transformers (ViTs) against older and more modern CNN based models and the results were surprising. The modern ConvNeXt model surpassed all transformers by a huge margin on my 20 class food dataset. That brought up another more important question: if I have limited dataset and compute power, would a small VLM perform better than the ConvNeXt model? if not how would it add value to this application?
I already know that the modern CNN based models like ConvNeXt-Tiny can outperform small transformer models on my limited food 101 dataset. I wanted to know whether a vision-language model could contribute something the classifier could not.
That lead me to two questions:
1- How does the SmolVLM perform compared to the fine tuned ConvNeXt -Tiny?
2- If the VLM cannot beat the classifier, can the it help ConvNeXt correct some of its mistakes?
To answer these I kept the dataset similar to baseline: 20 reproducibly selected classes, 250 images per class, and a stratified 4,000-image training, 1,000-image validation split drawn from Food-101’s training data.
I fine tuned a ConvNeXt-Tiny model on my food 101 subset and used it as my primary classifier and the baseline to compare the SmolVLM-Instruct against it. This was because it already had beaten the ViTs and performed substantially better than them.
There is a difference in interface between ConvNeXt and VLM models where some approaches need to be taken in order for a transparent and meaningful comparison.
The output from a classifier is a N-class logits that is converted into class probability distribution by softmax. However, the output from a VLM is a generated text that has no probability distribution or a tensor shaped answer. In order to be able to compare the results of the VLM with the ConvNeXt I took two approaches:
First, have the VLM output the class prediction where it outputs one class in text format and I process it and compare it against the prediciton of ConvNeXt. This apprach only allowed me to compare the predictions of the SmolVLM with that of the ConvNeXt-Tiny but no information about how confident it was or what was the top choices if the first choice was not correct. Those metrics matter when you want to compare models in production more than just the accuracy.
Second, I forced the SmolVLM to output a 20 class probability distribution for each class. This way I get a probability distribution similar to that of the ConvNeXt-Tiny which you can calculate metrics from such as Calibration Error, cross Entropy amongst other metrics.
However, generating a probability for each 20 class for every single image using the VLM was a time consuming process. The VLM needs to generate 20,000 answers (20 probabilities for 1,000 images) plus some post processing of the probabilities so this part of the experiment was not feasible and the full run was not completed.
So far, I arrived at the result that my original fine tuned ConvNeXt-Tiny model still beats SmolVLMs even though they performed better than ViTs on my food dataset subset. However, as I mentioned in the beginning of this article and what the title asks is where I can use VLMs to add value. Now is the time to see how the SmolVLM can actually help boost the performance of my fine tuned ConvNeXt-Tiny in classifying the images.
For that, I first run the ConvNeXt-Tiny model on the images and then look at the confidence of the model and its prediction. If the model has predicted the class wrong or the confidence score is lower than 0.60 (that is the top-1 confidence score) the image and a prompt including 20 class names and the image along with a short text that ask VLM to return the class to the SmolVLM-Instruct model. The results are separated into their own groups. 81 cases amongst the 121 were corrected by the SmolVLM model which is a substantial boost in performance. However, 25 out of 73 correctly classified with low confidence score cases were reclassified incorrectly.
This shows that VLM can greatly benefit the results as a reviewer but on the downside it might turn some of the correctly classified cases into misclassification.
The results above is all good but it does not represent a realistic scenario where the dataset is not labeled or we don’t have access to the labels for a lot of the data which happens quite often in production. I need to measure the performance when the labels are not accessible.
To achieve this, labels were made unavailable to the router. The routing decision was now using only the values output by the ConvNeXt model. ConvNeXt’s top-1 confidence, the gap between its top two probabilities, and the entropy of its predicted distribution. An image was sent to the VLM if confidence was below 0.60, the top-two margin was below 0.20, or entropy was above 1.50.
ConvNeXt made 125 misclassification but only 94 met the low confidence criteria to be routed for review. 174 images out of 1000 were routed for review where 94 were misclassified and 80 were correctly classified meeting the low confidence criteria. SmolVLM corrected 66 out of the 94 mistakes and misclassified 28 images out of the 80 low confidence correct classifications.
The decision to know if SmolVLM actually added value as a reviewer depends on three questions:
1- Did the router find the mistakes? if the incorrectly classified images with high confidence never reached the reviewer then the existence of the reviewer does not matter.
2- Did the VLM fix more answers than it broke? Uncertain classifications were routed to the VLM for a second opinion. They were not necessarily wrong and we don’t want VLM to change a correct classification to a wrong one.
3- How much time did the review add? I added the VLM as a reviewer because I wanted to reduce the cost and complexity and boost correct prediction at the same time. This is inherently an important factor in my experiment.
As illustrated in the figures (a) and (b) above, routing the low confidence cases that met the criteria were reviewed and some were misclassified but more were saved. The boosted performance is 38 fixed cases (3.8%) overall. The added time for review was 226.4 seconds for all images, almost 1.30 seconds per routed image.
The figure above shows the performance boost of using SmolVLM as a reviewer for ConvNeXt. We have to keep in mind that SmolVLM did convert some of the correct classifications into misclassifications so there is room to improve the results potentially by using more advanced fine tuning and using more data which are avenues for investigation. However, for the scope of this experiment this shows that using a VLM model in addition to a classifier can be a powerful combo for limited data and compute power.
When Does a VLM Add Value to Your Project? was originally published in Towards AI on Medium, where people are continuing the conversation by highlighting and responding to this story.