{"slug": "how-to-distill-a-vlm-into-a-small-object-detection-model", "title": "How to Distill a VLM Into a Small Object Detection Model", "summary": "Roboflow published a guide showing how to use Google's Gemini vision-language model as a teacher to auto-label images, then correct those labels and train a small RF-DETR Nano object detector that runs locally via Roboflow Inference instead of calling a cloud VLM on every frame. The workflow, saved as \"dispatch-vlm-teacher,\" exposes predictions, error_status, input_tokens and output_tokens, with error_status=True flagging failed decoding rather than an empty scene. The approach follows the Autodistill label-generation method and transfers supervision for object detection only, not the VLM's language capabilities.", "body_md": "To get VLM-quality detections without paying VLM prices on every frame, use a model like Gemini to auto-label your images, correct the labels, and train a small detector to run next to your camera. In Roboflow, you can auto-label with a Gemini Workflow, review in Annotate, train RF-DETR Nano and deploy it with Inference.\n\nA vision-language model (VLM) can identify cardboard boxes and pallets from a text description. That makes it a useful starting point for building a warehouse monitoring application.\n\nFor continuous detection, you can use its predictions to train a smaller model that runs on a computer beside the camera, reducing reliance on cloud model calls.\n\nIn this guide, I'll build on my model distillation article, where I introduced the teacher-student idea. Here you’ll learn how to distill a VLM into a small object detector with Roboflow. You’ll use Gemini to generate bounding-box annotations, review them in Roboflow Annotate, train an RF-DETR model, and deploy it locally with Roboflow Inference.\n\nHinton, Vinyals, and Dean (opens in a new tab) describe how to “compress the knowledge in an ensemble into a single model which is much easier to deploy.” Their method uses soft targets. This tutorial follows the Autodistill label-generation approach (opens in a new tab) i.e. teacher predictions become training annotations after human review. It transfers supervision for object detection; it does not transfer the VLM's language capabilities.\n\nI’ll also show you how to evaluate the tradeoffs. You’ll learn how to compare detection quality and inference costs, then measure performance on your own deployment hardware.\n\nStep 1: Auto-label with a foundation model\n\nThe teacher produces a class and bounding box for each target object. These are candidate annotations: a person will check them before they become student training data.\n\nOne bounding box for each identifiable shipping box, not one box around an entire stack.\n\npallet\n\nOne bounding box for each identifiable pallet, excluding the load above it.\n\nFor this example, annotate the visible extent of partially hidden objects. Use the exact same class names throughout labeling, training, and evaluation.\n\nExpose predictions, error_status, input_tokens, and output_tokens from the corresponding outputs of teacher. For example, the predictions selector is $steps.teacher.predictions. The v6 block decodes detection responses itself; no separate conversion block is needed. error_status=True means decoding failed and must not be treated as an empty scene.\n\nSave and publish the Workflow as dispatch-vlm-teacher. Test it in the editor (opens in a new tab) with a representative image, inspect the boxes and classes, and record the selected Gemini model and saved Workflow ID.\n\nTo test it in the editor, click Run, upload a representative JPG, PNG, or WEBP image, check the selected runtime, and run the Workflow. Inspect the structured predictions and error_status outputs. The Workflow testing guide (opens in a new tab) explains these controls.\n\nThe diagram shows the two inputs and outputs of this teacher. Only the boxes and classes become annotations, the error and token fields help validate the response and measure usage.\n\n1.3 Generate the training labels\n\nOpen the training batch in Auto Label (opens in a new tab). Select Workflows, choose the saved teacher, and map its classes to the project classes. Check the preview and cost estimate, then start Auto Label With This Workflow. Unmapped classes are dropped, so confirm both class mappings.\n\nReview a small pilot batch before labeling the full collection. Look for merged stacks, missed pallets, and duplicate boxes. Keep the original teacher output if you want to compare it with the corrected annotations later.\n\nOther labeling routes:You can choose Gemini directly in Auto Label. If you already use Autodistill (opens in a new tab), use its Base Model to generate labels and train RF-DETR separately in Step 3. Existing Auto Annotate output can also be imported with its images in a supported annotation format (opens in a new tab). Each route must provide object boxes and class names; captions alone are insufficient.\n\nStep 2: Review and fix the labels in Roboflow Annotate\n\nThe student learns from the annotations you accept. A missed pallet supplies no positive training example for that object. A box covering an entire stack teaches a different target from a box covering one carton. Review therefore determines what task the student learns.\n\nOpen an image in Roboflow Annotate (opens in a new tab). Use the selection tool to move boxes or resize their handles, the class editor to change labels, and the bounding box tool to draw missing annotations. Work through each image in this order:\n\nFind missing objects: Scan the whole image, including borders and partially hidden areas. Do not restrict the review to regions the teacher already found.\n\nCorrect class labels: Apply the class definitions from Step 1.\n\nCorrect box boundaries: Follow the same visible-extent rule throughout the dataset.\n\nRemove incorrect and duplicate boxes: Each object should contribute one annotation of its class.\n\nReview empty results: Confirm whether the scene actually lacks the target classes or the teacher missed them.\n\nFor example, suppose the teacher finds one of two cardboard boxes and places two overlapping boxes around it. Delete the duplicate, adjust the remaining box, add the missed carton, and check the pallet below them. This is an illustrative correction, not a measured result from the teacher.\n\nWhere Review Mode is available and enabled, use its approval process. Otherwise, still perform the inspection manually. From the annotation job, select the checked images and use Add N Images to Dataset. Approval and dataset inclusion are described in Collaborate on Labeling (opens in a new tab).\n\nKeep a small review log i.e. images checked, missed objects added, incorrect boxes removed, and review minutes. It helps identify a teacher that is inexpensive to call but costly to correct.\n\nStep 3: Train RF-DETR\n\nThe reviewed dataset now represents a two-class detection task. We will train RF-DETR Nano, a compact option in the RF-DETR family (opens in a new tab), using Roboflow's hosted training flow.\n\n3.1 Prepare a dataset version\n\nOpen Versions and thenGenerate New Version. Check the split assignments, select preprocessing and augmentation settings appropriate to the camera scenes, and generate the version. A dataset version (opens in a new tab) records the images, annotations, and processing settings used for training.\n\nPreserve the session-based split and confirm that resizing leaves small objects recognizable. Record the version number and image counts.\n\nReview Customize Training Recipe, including the checkpoint and hyperparameters. Start from the supported public checkpoint for the first experiment.\n\nReview the duration and credit estimate, set a credit cap if appropriate and available, and select Start Training.\n\nAfter training, save the recipe and use Copy Model ID (opens in a new tab) on the resulting model. Multiple training runs can use one dataset version, so copy the actual model ID.\n\nOpen the built-in Training Graphs (opens in a new tab) to inspect the learning curves and object detection metrics. These reports are the first evidence of how training behaved. Use validation examples while deciding whether to change the data or training settings.\n\nStep 4: Deploy at the edge with Roboflow Inference\n\nThe trained student can now run on a computer beside the camera. We will create one student Workflow and use it for both a test image and a recorded video. This deployment does not include the Gemini teacher.\n\n4.1 Create the student Workflow\n\nCreate a Workflow named rfdetr-student with an image input named image. Add these built-in blocks:\n\nObject Detection Model, named student: select the trained model ID, connect $inputs.image, and set the validation-selected confidence threshold.\n\nBounding Box Visualization, named boxes: use the input image and student predictions.\n\nLabel Visualization, named labels: use the image returned by boxes and the same student predictions.\n\nThe Inference Server (opens in a new tab) uses http://localhost:9001 for local requests. Keep it running while executing the examples. Header authentication in the code requires Inference Server 1.5.0 or newer.\n\nThe first request can download and initialize the model weights. Local execution does not automatically mean disconnected operation; use the separate Offline Mode guide (opens in a new tab) if that is a requirement.\n\n4.3 Run the Workflow on an image\n\nPlace a representative image at test-image.jpg. Replace the workspace and saved Workflow IDs below, then save the code as run_image_workflow.py. It calls the Workflow SDK API (opens in a new tab) and prints the detection output.\n\nPython\n\n``` python\nimport json\nimport os\nfrom inference_sdk import InferenceHTTPClient, InferenceConfiguration\n\nclient = InferenceHTTPClient(\n    api_url=\"http://localhost:9001\",\n    api_key=os.environ[\"ROBOFLOW_API_KEY\"],\n).configure(InferenceConfiguration(api_key_transport=\"header\"))\n\nresults = client.run_workflow(\n    workspace_name=\"YOUR_WORKSPACE_ID\",\n    workflow_id=\"YOUR_SAVED_STUDENT_WORKFLOW_ID\",\n    images={\"image\": \"test-image.jpg\"},\n    use_cache=False,\n)\n\nfor result in results:\n    print(json.dumps(result[\"predictions\"], indent=2))\n```\n\nShell\n\n```\npython run_image_workflow.py\n```\n\nThe Workflow also returns output_image; the script prints only predictions to keep the terminal readable. The editor preview displays the annotated image. use_cache=False ensures this verification request uses your updated saved definition.\n\n4.4 Run the same Workflow on a video\n\nPlace a short warehouse clip at test-video.mp4. The built-in VideoFileSource (opens in a new tab) sends the file to the local server; the Workflow returns annotated frames and prediction data.\n\nSave this as run_video_workflow.py, using the same workspace and Workflow IDs:\n\nPython\n\n``` python\nimport cv2\nimport os\nfrom inference_sdk import InferenceHTTPClient, InferenceConfiguration\nfrom inference_sdk.webrtc import VideoFileSource, StreamConfig\n\nclient = InferenceHTTPClient(\n    api_url=\"http://localhost:9001\",\n    api_key=os.environ[\"ROBOFLOW_API_KEY\"],\n).configure(InferenceConfiguration(api_key_transport=\"header\"))\n\nsession = client.webrtc.stream(\n    source=VideoFileSource(\"test-video.mp4\", realtime_processing=False),\n    workflow=\"YOUR_SAVED_STUDENT_WORKFLOW_ID\",\n    workspace=\"YOUR_WORKSPACE_ID\",\n    config=StreamConfig(\n        stream_output=[\"output_image\"],\n        data_output=[\"predictions\"],\n        realtime_processing=False,\n    ),\n)\n\n@session.on_frame\ndef show(frame, metadata):\n    cv2.imshow(\"RF-DETR student - press q to stop\", frame)\n    if cv2.waitKey(1) & 0xFF == ord(\"q\"):\n        session.close()\n\n@session.on_data(\"predictions\")\ndef report(predictions, metadata):\n    print({\"frame_id\": metadata.frame_id, \"predictions\": predictions})\n\n@session.on_error\ndef report_error(errors, metadata):\n    print({\"frame_id\": metadata.frame_id, \"errors\": errors})\n\ntry:\n    session.run()\nfinally:\n    session.close()\n    cv2.destroyAllWindows()\n```\n\nShell\n\n```\npython run_video_workflow.py\n```\n\nRun this preview script on the edge computer with a desktop display. Press q in the preview window to stop. It uses the Workflow's output_image, so the displayed frames include both boxes and class labels. Prediction data appears in the terminal. With real-time processing disabled, this file test is configured to process frames without live-preview pacing; it is a deployment demonstration, not a speed benchmark.\n\nFor a camera connected to the same computer, import WebcamSource and replace the source with WebcamSource(). Use realtime_processing=True in StreamConfig for a live preview. See the video Workflow guide (opens in a new tab) for source options.\n\nBenchmark the student against the teacher: accuracy, FPS, cost per image\n\nThe final check is whether the student meets the application's needs. Check three things:\n\nwhether the student detects the required objects correctly,\n\nhow quickly it processes inputs, and\n\nwhat inference costs.\n\nFor a direct teacher–student comparison, use the same held-out images, class definitions, and comparable Workflow settings.\n\nAccuracy: compare detections with Workflow Evals\n\nAccuracy evaluation checks the predicted classes and bounding boxes against independently reviewed reference labels. Missed objects and false detections both matter. Use Workflow Evals (opens in a new tab) as follows:\n\nOpen Workflows -> Evals -> Create Eval in your workspace.\n\nChoose the Workflow. Use the server selector to choose where it runs.\n\nImport your held-out images from a Dataset test set, the Asset Library, or your device. Select predictions as the output to evaluate.\n\nReview the reference annotations, correcting missing boxes, incorrect classes, and box boundaries. Approve each Case with Add to Ground Truth. Imported labels require review before approval.\n\nAdd an object-detection Check and set its thresholds. Use Run preview to verify the Check on one Case.\n\nReview the configuration and credit estimate, then start the evaluation. Evaluate both teacher and student against the same Cases and Checks.\n\nOpen Metrics to compare the detection results and inspect failing Cases. Report the metric name and aggregation used; a Check pass rate is different from detection accuracy.\n\nFPS and latency: measure processing performance\n\nLatency is the time taken to return a result; throughput is the number of requests or images processed per second. For video, FPS describes the measured number of frames processed per second.\n\nFor cloud measurements, use Benchmark Workflow in the saved Workflow's toolbar:\n\nSelect the image set, deployment service, and request load.\n\nRun the benchmark and record its throughput, latency percentiles, and failed requests.\n\nRepeat for the other Workflow. For a direct comparison, keep the images, service, request rate, duration, and warm-up settings the same, with comparable visualization and other processing blocks.\n\nThe completed reports below both used Cloud API, a configured load of 1 request/second, and 120 planned inferences over 2 minutes. The student report showed 409 ms median latency and 1,062 ms p95 latency; the teacher report showed 23.8 s median latency and 58.4 s p95 latency. The p95 value means approximately 95% of measured requests completed within that time.\n\nBoth Workflows used all 100 images in Pallet Monitoring. The results therefore cover the same image set and configured request load. They describe the complete Workflows under these cloud test conditions.\n\nThroughput was 0.96 requests/second for the student and 0.72requests/second for the teacher under the configured one-request-per-second load. The teacher report flagged that it delivered 72% of the configured request rate and did not fully keep up with this load. It recorded two failed requests across the whole run, compared with zero for the student. These numbers describe Cloud API Workflow requests rather than isolated model inference or edge-device video FPS. A test configured for one request per second does not establish the student's maximum processing rate.\n\nStart the Inference Server on the edge computer, using the deployment setup in Step 4.\n\nFollow the API benchmark example in the linked guide, supplying the trained model ID, representative test-image directory, and report output directory.\n\nReview the saved throughput, latency, errors, and platform details. Record the hardware and request settings, and identify whether timing covers a model endpoint or a complete Workflow.\n\nReport an image benchmark as images/second. For video, measure processed frames per second over a stated interval and identify whether network transfer and visualization are included. Keep the video input rate separate from the processing rate.\n\nCost per image: use the billed benchmark report\n\nThe Cost Benchmark report displayed 5.19 credits per 1,000 inferences for the teacher and 0.12 credits per 1,000 inferences for the student. These correspond to approximately 0.0052 and 0.00012 credits per inference, respectively. For requests containing one image, they are also the displayed costs per image. These are the displayed rounded rates for the two Workflows on the Pallet Monitoring image set.\n\nThe teacher report attributed its measured cost to Workflows (3rd Party Blocks); the student report showed Cloud API charges. For additional attribution, use Credit Usage and its audit log (opens in a new tab), filtering by the experiment's time window and API key or folder.\n\nWarm-up traffic is billed but excluded from the reports' steady-window rates, latency, and throughput. Total Billed covers the whole run, so keep it separate from normalized cost per inference. For edge deployment, state any hardware and electricity allocation separately. Keep auto-labeling, annotation review, and training costs separate from deployed inference.\n\nThe table records the displayed values from the two completed cloud runs, using the same Pallet Monitoring image set.\n\nMeasure\n\nVLM teacher\n\nRF-DETR student\n\nWorkflow\n\nvlm-teacher\n\nrfdetr-student\n\nImage selection\n\nAll 100 images in Pallet Monitoring\n\nAll 100 images in Pallet Monitoring\n\nDeployment service\n\nCloud API\n\nCloud API\n\nConfigured load\n\n1 request/second; 120 planned inferences over 2 minutes\n\n1 request/second; 120 planned inferences over 2 minutes\n\nMeasured throughput\n\n0.72 requests/second\n\n0.96 requests/second\n\nMedian request latency (p50)\n\n23.8 s\n\n409 ms\n\np95 request latency\n\n58.4 s\n\n1,062 ms\n\nCold start\n\n13.6 s\n\n860 ms\n\nFailed requests, whole run\n\n2\n\n0\n\nDisplayed cost per 1,000 inferences\n\n5.19 credits\n\n0.12 credits\n\nApproximate cost per image, single-image requests\n\n0.0052 credits\n\n0.00012 credits\n\nTotal Billed, whole run\n\n0.9 credits\n\n0.0216 credits\n\nConclusion\n\nBuild a detector for your camera footage by starting with an Object Detection project in Roboflow. Upload a small, representative set of images, use a Gemini teacher to generate annotations, and review the boxes before training your first RF-DETR model.\n\nFrom there, use Roboflow Workflows to test the detector on new images and Roboflow Inference to run it on compatible edge hardware. Check missed objects, processing speed, and operating costs to decide what to improve before expanding the deployment.\n\nGet started in Roboflow by uploading your first batch of images and previewing the teacher’s annotations.", "url": "https://wpnews.pro/news/how-to-distill-a-vlm-into-a-small-object-detection-model", "canonical_source": "https://roboflow.com/blog/distill-a-vlm-into-object-detection-model", "published_at": "2026-11-15 23:47:23+00:00", "updated_at": "2026-11-16 00:50:33.121918+00:00", "lang": "en", "topics": ["computer-vision", "ai-tools", "machine-learning", "ai-products"], "entities": ["Roboflow", "Gemini", "Google", "RF-DETR Nano", "Roboflow Inference", "Roboflow Annotate", "Autodistill", "Geoffrey Hinton"], "also_reported_by": [], "alternates": {"html": "https://wpnews.pro/news/how-to-distill-a-vlm-into-a-small-object-detection-model", "markdown": "https://wpnews.pro/news/how-to-distill-a-vlm-into-a-small-object-detection-model.md", "text": "https://wpnews.pro/news/how-to-distill-a-vlm-into-a-small-object-detection-model.txt", "jsonld": "https://wpnews.pro/news/how-to-distill-a-vlm-into-a-small-object-detection-model.jsonld"}}