# How to Distill a VLM Into a Small Object Detection Model

> Source: <https://roboflow.com/blog/distill-a-vlm-into-object-detection-model>
> Published: 2026-11-15 23:47:23+00:00

To get VLM-quality detections without paying VLM prices on every frame, use a model like Gemini to auto-label your images, correct the labels, and train a small detector to run next to your camera. In Roboflow, you can auto-label with a Gemini Workflow, review in Annotate, train RF-DETR Nano and deploy it with Inference.

A vision-language model (VLM) can identify cardboard boxes and pallets from a text description. That makes it a useful starting point for building a warehouse monitoring application.

For continuous detection, you can use its predictions to train a smaller model that runs on a computer beside the camera, reducing reliance on cloud model calls.

In this guide, I'll build on my model distillation article, where I introduced the teacher-student idea. Here you’ll learn how to distill a VLM into a small object detector with Roboflow. You’ll use Gemini to generate bounding-box annotations, review them in Roboflow Annotate, train an RF-DETR model, and deploy it locally with Roboflow Inference.

Hinton, Vinyals, and Dean (opens in a new tab) describe how to “compress the knowledge in an ensemble into a single model which is much easier to deploy.” Their method uses soft targets. This tutorial follows the Autodistill label-generation approach (opens in a new tab) i.e. teacher predictions become training annotations after human review. It transfers supervision for object detection; it does not transfer the VLM's language capabilities.

I’ll also show you how to evaluate the tradeoffs. You’ll learn how to compare detection quality and inference costs, then measure performance on your own deployment hardware.

Step 1: Auto-label with a foundation model

The teacher produces a class and bounding box for each target object. These are candidate annotations: a person will check them before they become student training data.

One bounding box for each identifiable shipping box, not one box around an entire stack.

pallet

One bounding box for each identifiable pallet, excluding the load above it.

For this example, annotate the visible extent of partially hidden objects. Use the exact same class names throughout labeling, training, and evaluation.

Expose predictions, error_status, input_tokens, and output_tokens from the corresponding outputs of teacher. For example, the predictions selector is $steps.teacher.predictions. The v6 block decodes detection responses itself; no separate conversion block is needed. error_status=True means decoding failed and must not be treated as an empty scene.

Save and publish the Workflow as dispatch-vlm-teacher. Test it in the editor (opens in a new tab) with a representative image, inspect the boxes and classes, and record the selected Gemini model and saved Workflow ID.

To test it in the editor, click Run, upload a representative JPG, PNG, or WEBP image, check the selected runtime, and run the Workflow. Inspect the structured predictions and error_status outputs. The Workflow testing guide (opens in a new tab) explains these controls.

The diagram shows the two inputs and outputs of this teacher. Only the boxes and classes become annotations, the error and token fields help validate the response and measure usage.

1.3 Generate the training labels

Open the training batch in Auto Label (opens in a new tab). Select Workflows, choose the saved teacher, and map its classes to the project classes. Check the preview and cost estimate, then start Auto Label With This Workflow. Unmapped classes are dropped, so confirm both class mappings.

Review a small pilot batch before labeling the full collection. Look for merged stacks, missed pallets, and duplicate boxes. Keep the original teacher output if you want to compare it with the corrected annotations later.

Other labeling routes:You can choose Gemini directly in Auto Label. If you already use Autodistill (opens in a new tab), use its Base Model to generate labels and train RF-DETR separately in Step 3. Existing Auto Annotate output can also be imported with its images in a supported annotation format (opens in a new tab). Each route must provide object boxes and class names; captions alone are insufficient.

Step 2: Review and fix the labels in Roboflow Annotate

The student learns from the annotations you accept. A missed pallet supplies no positive training example for that object. A box covering an entire stack teaches a different target from a box covering one carton. Review therefore determines what task the student learns.

Open an image in Roboflow Annotate (opens in a new tab). Use the selection tool to move boxes or resize their handles, the class editor to change labels, and the bounding box tool to draw missing annotations. Work through each image in this order:

Find missing objects: Scan the whole image, including borders and partially hidden areas. Do not restrict the review to regions the teacher already found.

Correct class labels: Apply the class definitions from Step 1.

Correct box boundaries: Follow the same visible-extent rule throughout the dataset.

Remove incorrect and duplicate boxes: Each object should contribute one annotation of its class.

Review empty results: Confirm whether the scene actually lacks the target classes or the teacher missed them.

For example, suppose the teacher finds one of two cardboard boxes and places two overlapping boxes around it. Delete the duplicate, adjust the remaining box, add the missed carton, and check the pallet below them. This is an illustrative correction, not a measured result from the teacher.

Where Review Mode is available and enabled, use its approval process. Otherwise, still perform the inspection manually. From the annotation job, select the checked images and use Add N Images to Dataset. Approval and dataset inclusion are described in Collaborate on Labeling (opens in a new tab).

Keep a small review log i.e. images checked, missed objects added, incorrect boxes removed, and review minutes. It helps identify a teacher that is inexpensive to call but costly to correct.

Step 3: Train RF-DETR

The reviewed dataset now represents a two-class detection task. We will train RF-DETR Nano, a compact option in the RF-DETR family (opens in a new tab), using Roboflow's hosted training flow.

3.1 Prepare a dataset version

Open Versions and thenGenerate New Version. Check the split assignments, select preprocessing and augmentation settings appropriate to the camera scenes, and generate the version. A dataset version (opens in a new tab) records the images, annotations, and processing settings used for training.

Preserve the session-based split and confirm that resizing leaves small objects recognizable. Record the version number and image counts.

Review Customize Training Recipe, including the checkpoint and hyperparameters. Start from the supported public checkpoint for the first experiment.

Review the duration and credit estimate, set a credit cap if appropriate and available, and select Start Training.

After training, save the recipe and use Copy Model ID (opens in a new tab) on the resulting model. Multiple training runs can use one dataset version, so copy the actual model ID.

Open the built-in Training Graphs (opens in a new tab) to inspect the learning curves and object detection metrics. These reports are the first evidence of how training behaved. Use validation examples while deciding whether to change the data or training settings.

Step 4: Deploy at the edge with Roboflow Inference

The trained student can now run on a computer beside the camera. We will create one student Workflow and use it for both a test image and a recorded video. This deployment does not include the Gemini teacher.

4.1 Create the student Workflow

Create a Workflow named rfdetr-student with an image input named image. Add these built-in blocks:

Object Detection Model, named student: select the trained model ID, connect $inputs.image, and set the validation-selected confidence threshold.

Bounding Box Visualization, named boxes: use the input image and student predictions.

Label Visualization, named labels: use the image returned by boxes and the same student predictions.

The Inference Server (opens in a new tab) uses http://localhost:9001 for local requests. Keep it running while executing the examples. Header authentication in the code requires Inference Server 1.5.0 or newer.

The first request can download and initialize the model weights. Local execution does not automatically mean disconnected operation; use the separate Offline Mode guide (opens in a new tab) if that is a requirement.

4.3 Run the Workflow on an image

Place a representative image at test-image.jpg. Replace the workspace and saved Workflow IDs below, then save the code as run_image_workflow.py. It calls the Workflow SDK API (opens in a new tab) and prints the detection output.

Python

``` python
import json
import os
from inference_sdk import InferenceHTTPClient, InferenceConfiguration

client = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key=os.environ["ROBOFLOW_API_KEY"],
).configure(InferenceConfiguration(api_key_transport="header"))

results = client.run_workflow(
    workspace_name="YOUR_WORKSPACE_ID",
    workflow_id="YOUR_SAVED_STUDENT_WORKFLOW_ID",
    images={"image": "test-image.jpg"},
    use_cache=False,
)

for result in results:
    print(json.dumps(result["predictions"], indent=2))
```

Shell

```
python run_image_workflow.py
```

The Workflow also returns output_image; the script prints only predictions to keep the terminal readable. The editor preview displays the annotated image. use_cache=False ensures this verification request uses your updated saved definition.

4.4 Run the same Workflow on a video

Place a short warehouse clip at test-video.mp4. The built-in VideoFileSource (opens in a new tab) sends the file to the local server; the Workflow returns annotated frames and prediction data.

Save this as run_video_workflow.py, using the same workspace and Workflow IDs:

Python

``` python
import cv2
import os
from inference_sdk import InferenceHTTPClient, InferenceConfiguration
from inference_sdk.webrtc import VideoFileSource, StreamConfig

client = InferenceHTTPClient(
    api_url="http://localhost:9001",
    api_key=os.environ["ROBOFLOW_API_KEY"],
).configure(InferenceConfiguration(api_key_transport="header"))

session = client.webrtc.stream(
    source=VideoFileSource("test-video.mp4", realtime_processing=False),
    workflow="YOUR_SAVED_STUDENT_WORKFLOW_ID",
    workspace="YOUR_WORKSPACE_ID",
    config=StreamConfig(
        stream_output=["output_image"],
        data_output=["predictions"],
        realtime_processing=False,
    ),
)

@session.on_frame
def show(frame, metadata):
    cv2.imshow("RF-DETR student - press q to stop", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        session.close()

@session.on_data("predictions")
def report(predictions, metadata):
    print({"frame_id": metadata.frame_id, "predictions": predictions})

@session.on_error
def report_error(errors, metadata):
    print({"frame_id": metadata.frame_id, "errors": errors})

try:
    session.run()
finally:
    session.close()
    cv2.destroyAllWindows()
```

Shell

```
python run_video_workflow.py
```

Run this preview script on the edge computer with a desktop display. Press q in the preview window to stop. It uses the Workflow's output_image, so the displayed frames include both boxes and class labels. Prediction data appears in the terminal. With real-time processing disabled, this file test is configured to process frames without live-preview pacing; it is a deployment demonstration, not a speed benchmark.

For a camera connected to the same computer, import WebcamSource and replace the source with WebcamSource(). Use realtime_processing=True in StreamConfig for a live preview. See the video Workflow guide (opens in a new tab) for source options.

Benchmark the student against the teacher: accuracy, FPS, cost per image

The final check is whether the student meets the application's needs. Check three things:

whether the student detects the required objects correctly,

how quickly it processes inputs, and

what inference costs.

For a direct teacher–student comparison, use the same held-out images, class definitions, and comparable Workflow settings.

Accuracy: compare detections with Workflow Evals

Accuracy evaluation checks the predicted classes and bounding boxes against independently reviewed reference labels. Missed objects and false detections both matter. Use Workflow Evals (opens in a new tab) as follows:

Open Workflows -> Evals -> Create Eval in your workspace.

Choose the Workflow. Use the server selector to choose where it runs.

Import your held-out images from a Dataset test set, the Asset Library, or your device. Select predictions as the output to evaluate.

Review the reference annotations, correcting missing boxes, incorrect classes, and box boundaries. Approve each Case with Add to Ground Truth. Imported labels require review before approval.

Add an object-detection Check and set its thresholds. Use Run preview to verify the Check on one Case.

Review the configuration and credit estimate, then start the evaluation. Evaluate both teacher and student against the same Cases and Checks.

Open Metrics to compare the detection results and inspect failing Cases. Report the metric name and aggregation used; a Check pass rate is different from detection accuracy.

FPS and latency: measure processing performance

Latency is the time taken to return a result; throughput is the number of requests or images processed per second. For video, FPS describes the measured number of frames processed per second.

For cloud measurements, use Benchmark Workflow in the saved Workflow's toolbar:

Select the image set, deployment service, and request load.

Run the benchmark and record its throughput, latency percentiles, and failed requests.

Repeat for the other Workflow. For a direct comparison, keep the images, service, request rate, duration, and warm-up settings the same, with comparable visualization and other processing blocks.

The completed reports below both used Cloud API, a configured load of 1 request/second, and 120 planned inferences over 2 minutes. The student report showed 409 ms median latency and 1,062 ms p95 latency; the teacher report showed 23.8 s median latency and 58.4 s p95 latency. The p95 value means approximately 95% of measured requests completed within that time.

Both Workflows used all 100 images in Pallet Monitoring. The results therefore cover the same image set and configured request load. They describe the complete Workflows under these cloud test conditions.

Throughput was 0.96 requests/second for the student and 0.72requests/second for the teacher under the configured one-request-per-second load. The teacher report flagged that it delivered 72% of the configured request rate and did not fully keep up with this load. It recorded two failed requests across the whole run, compared with zero for the student. These numbers describe Cloud API Workflow requests rather than isolated model inference or edge-device video FPS. A test configured for one request per second does not establish the student's maximum processing rate.

Start the Inference Server on the edge computer, using the deployment setup in Step 4.

Follow the API benchmark example in the linked guide, supplying the trained model ID, representative test-image directory, and report output directory.

Review the saved throughput, latency, errors, and platform details. Record the hardware and request settings, and identify whether timing covers a model endpoint or a complete Workflow.

Report an image benchmark as images/second. For video, measure processed frames per second over a stated interval and identify whether network transfer and visualization are included. Keep the video input rate separate from the processing rate.

Cost per image: use the billed benchmark report

The Cost Benchmark report displayed 5.19 credits per 1,000 inferences for the teacher and 0.12 credits per 1,000 inferences for the student. These correspond to approximately 0.0052 and 0.00012 credits per inference, respectively. For requests containing one image, they are also the displayed costs per image. These are the displayed rounded rates for the two Workflows on the Pallet Monitoring image set.

The teacher report attributed its measured cost to Workflows (3rd Party Blocks); the student report showed Cloud API charges. For additional attribution, use Credit Usage and its audit log (opens in a new tab), filtering by the experiment's time window and API key or folder.

Warm-up traffic is billed but excluded from the reports' steady-window rates, latency, and throughput. Total Billed covers the whole run, so keep it separate from normalized cost per inference. For edge deployment, state any hardware and electricity allocation separately. Keep auto-labeling, annotation review, and training costs separate from deployed inference.

The table records the displayed values from the two completed cloud runs, using the same Pallet Monitoring image set.

Measure

VLM teacher

RF-DETR student

Workflow

vlm-teacher

rfdetr-student

Image selection

All 100 images in Pallet Monitoring

All 100 images in Pallet Monitoring

Deployment service

Cloud API

Cloud API

Configured load

1 request/second; 120 planned inferences over 2 minutes

1 request/second; 120 planned inferences over 2 minutes

Measured throughput

0.72 requests/second

0.96 requests/second

Median request latency (p50)

23.8 s

409 ms

p95 request latency

58.4 s

1,062 ms

Cold start

13.6 s

860 ms

Failed requests, whole run

2

0

Displayed cost per 1,000 inferences

5.19 credits

0.12 credits

Approximate cost per image, single-image requests

0.0052 credits

0.00012 credits

Total Billed, whole run

0.9 credits

0.0216 credits

Conclusion

Build a detector for your camera footage by starting with an Object Detection project in Roboflow. Upload a small, representative set of images, use a Gemini teacher to generate annotations, and review the boxes before training your first RF-DETR model.

From there, use Roboflow Workflows to test the detector on new images and Roboflow Inference to run it on compatible edge hardware. Check missed objects, processing speed, and operating costs to decide what to improve before expanding the deployment.

Get started in Roboflow by uploading your first batch of images and previewing the teacher’s annotations.
