How to Pre-Filter Images Before Sending Them to an LLM

Roboflow published a tutorial showing how to pre-filter images before sending them to an LLM by running an RF-DETR Nano object detector at a 0.3 confidence threshold inside Roboflow Workflows and gating the results with a Continue If block so only matching images reach Google Gemini. The two-stage pattern tunes the detector for high recall and the LLM for high-precision final decisions; in a hypothetical 10M-image sports-car campaign where about 1% of images show a sports car and the detector passes 1% of images, Gemini would process 100K candidates instead of all 10M. The tutorial also covers tuning the filter on labeled examples, measuring how many relevant images it retains, and processing a dataset with Python.

To pre-filter images before sending them to an LLM, run a fast RF-DETR detector on every image in Roboflow Workflows and add a Continue If gate, so only images that match your rule reach Gemini. Sending every image in a dataset to a multimodal LLM means spending time and tokens on images that may never meet your selection criteria. To pre-filter images before sending them to an LLM, run an object detection model first, apply a relevance rule, and send only the images that pass for deeper analysis. With Roboflow Workflows, you can connect these steps in one pipeline. An RF-DETR model detects objects, a Continue If block controls which images advance, and Google Gemini analyzes the remaining candidates and returns structured results. In this tutorial, you’ll build a workflow for finding sports-car photos in a larger image library. The detector identifies potential candidates, while Gemini checks whether the vehicle is a sports car, whether it is the main subject, and whether alcohol is visible. You’ll also learn how to tune the filter on labeled examples, measure how many relevant images it retains, and process a dataset with Python. The goal is to reduce unnecessary LLM calls while preserving the images your application needs. Why Pre-Filter Images Before an LLM? Imagine you need to select images for a sports-car marketing campaign from an existing media library of 10M images. The campaign requires photos that show a sports car with no visible alcohol, along with a short description of each selected image. A direct approach is to send every image to a vision-capable LLM and ask it to check these conditions. However, the model must process each submitted image before deciding whether it is relevant. An instruction such as “ignore images without a sports car” does not avoid that processing. When relevant images are rare, most requests analyze images that do not meet the selection criteria. Across millions of images, these requests consume inference budget and processing capacity without adding useful images to the campaign’s collection. The engineering goal is to reduce this unnecessary inference while retaining the relevant images. How Pre-Filtering Images Before an LLM Works The fix is to change when the LLM runs. In a dataset with rare positives, most images fail an easy check. An image with no car cannot contain a sports car, and a small object detector can run that check on every image at very low cost. Only the images that pass go on to the LLM, which handles the hard part of the task. The pattern turns this idea into two stages, each with a clear job. Stage 1, the detector, must keep almost every positive image, so it is tuned for high recall. Stage 2, the LLM, sees far fewer images, so you can afford a large model for the final high-precision decision. Because the jobs are separate, each stage can also be evaluated and tuned on its own. In the campaign example, a sports car is the positive condition and visible alcohol is the negative condition. For a hypothetical estimate, assume that about 1% of the 10M images in the library show a sports car. Without Stage 1, Gemini processes all 10M images. For this hypothetical calculation, assume that a custom Stage 1 detector passes 1% of the images while retaining the relevant examples. Gemini would then process 100K candidates. This assumed pass rate must be measured in practice; having 1% relevant images does not guarantee that only 1% will pass the filter. How to Pre-Filter Images Before Sending Them to an LLM Let us now build a pre-filter workflow in Roboflow Workflows. Our example workflow screens images where a sports car is the main subject and no alcohol is visible. It avoids Gemini LLM calls on images where the detector finds no car. The blocks run in a fixed order. First, the RF-DETR Nano detector looks for cars, bottles, and wine glasses at a confidence threshold of 0.3. A Property Definition block counts the cars, and an Expression block turns that count into a gate decision. The Continue If block then acts as the gate. An image moves on only if it has at least one car. Other images stop at the gate. Bottles and wine glasses do not stop an image, because detecting these objects does not establish that alcohol is present. Gemini checks for visible alcohol or alcohol branding in Stage 2. For images that pass, the Google Gemini block looks at the full photo. It decides if the vehicle is a sports car or supercar. It checks if any alcohol or alcohol branding is visible. It also checks if the car is the clear subject. Then it writes a one-sentence description. The JSON Parser block turns this answer into separate fields that your application can use. First Non Empty Or Default blocks fill any empty field with a default value. A Custom Python block then checks every field and gives each image one of four outcomes. The following diagram shows how the workflow works. The diagram groups the workflow into three stages, pre-filter the image, verify it with Gemini, and return a consistent result. Images without a detected car stop before Gemini is called. The four outcomes appear along the bottom. The steps below build this workflow one block at a time. Step 1. Create a Workflow In your Roboflow workspace, open the Workflows tab and click Create Workflow. This opens a blank workflow in the editor. In the default Auto mode, the editor connects compatible inputs and outputs as you add blocks. You can also describe Steps 2 to 5 to the Workflows AI Assistant in the chat panel of the editor. It adds the blocks for you. Check each block afterward, because some settings may need a manual fix. Each save creates a new version, so you can trace every run to its exact thresholds. Step 2. Add the Detector for the Cheap Annotation Pass Add an Object Detection Model block and connect it to the image input. Use RF-DETR Nano pretrained on COCO, with the model ID rfdetr-nano. COCO already includes the classes car, bottle, and wine glass. Then set two parameters. Class Filter: Set it to car, bottle, and wine glass. The gate uses only car, while the bottle and wine glass boxes stay in the output for the optional rule in Step 3 and for later reuse. Confidence Mode: Choose Custom and enter 0.3. A low threshold keeps recall high for cars, because an image with no detected car never reaches the LLM. For better recall on small objects, move to the Small or Medium size of RF-DETR. A COCO model knows the class car but not sports car. So this starter gate passes every car image, which is far more than the 1% of images that show a sports car. The share of images that pass the gate is the pass rate. To bring the pass rate closer to the positive rate, train a detector on a task class such as sports-car. A custom detector can take over the alcohol rule only if it learns alcohol-specific classes such as beer-bottle and beer-can. Validate it on labeled images before it drops any image. You can also train RF-DETR on a custom dataset or start from public datasets on Roboflow Universe. After that, change only the model ID in this step and the class name in the Step 3 count. Step 3. Count the Cars and Add the Continue If Gate The gate needs one clear yes or no for each image. Three small blocks create that decision and act on it. Add a Property Definition block named car count and set its data to the detector predictions. Give it two operations. The first is a Detections Filter that keeps only the car class. The second is SequenceLength, which counts the cars that remain. Add an Expression block named gate decision after car count. Give it one case that returns true when car count is 1 or more, and set the default result to false. This value is the gate decision, and the workflow returns it as gate pass. Add a Continue If block after gate decision. For each image, it checks a condition. Map gate pass in evaluation parameters to the gate decision output, and set the condition to gate pass equals true. If the condition is true, the steps in next steps run. If it is false, the block stops that branch. The gate does not judge alcohol. A detected bottle or wine glass tells you what the object is, not what it holds. A COCO detector gives a water bottle and a beer bottle the same bottle class. A higher confidence threshold only makes the class more certain. Gemini decides whether alcohol is visible in Step 4. Some campaigns ban every bottle or glass, whatever it holds. For those, add a second Property Definition block that counts bottle and wine glass detections with a confidence of 0.6 or higher. Pass that count to the Expression block too, and add a second statement with AND that requires it to be 0. Leave this rule out of the starter workflow, because it also drops images that meet the stated requirements. Step 2 covers the other option, a custom detector for alcohol-specific classes. Add a Google Gemini block, connect its image input to the workflow image, and select it as the next step of the Continue If block. Then set five parameters. Task Type. Choose Structured Output Generation structured-answering . Each value in the output structure is the instruction for its key, and the model returns JSON only. Output Structure. Use the four keys below. Model Version. Choose Gemini 3 Flash gemini-3-flash-preview . API Key. Keep rf key:account to use Roboflow's managed key with usage-based billing, or enter your own Google AI API key. Start each instruction with its type, such as Boolean or String, so Gemini knows which kind of value to return. JSON { "is sports car": "Boolean: true if the main vehicle is a sports car or supercar, otherwise false.", "alcohol visible": "Boolean: true only when visible evidence identifies an alcoholic drink, beer can, wine or liquor bottle, cocktail, or alcohol logo anywhere in the image; otherwise false. A generic bottle, water bottle, empty wine glass, or juice does not by itself establish alcohol. Assess visible evidence only, not hidden contents.", "car is subject": "Boolean: true if the car is the clear subject of the photo, otherwise false.", "description": "String: one sentence describing the car, its color, and the setting." } The alcohol visible instruction also says that a plain bottle or an empty glass is not proof of alcohol. Gemini judges only what the image shows. Stage 1 already removed the clear negatives, so the LLM task is small. It confirms the sports car class, checks every bottle, glass, can, and logo for alcohol, and writes a short caption you can search later. Step 5. Parse and Validate the Answer Add a JSON Parser block. Set raw json to the Gemini output and add the four keys as expected fields. Each key becomes a workflow output. The block sets error status to true when a key is missing, which gives you a parse error rate to monitor. When the gate stops an image, the Gemini and JSON Parser steps do not run, so their outputs are null. Add five First Non Empty Or Default blocks, one for each parsed field and one for error status. Each block passes its value on, or returns a default value when the input is null. Set the default to missing for the three Boolean fields, to an empty string for description, and to true for error status. None of these defaults can pass the check below, so a missing answer never becomes a match. Add a Custom Python block named final decision after these five blocks. Its inputs are gate pass and the five filled fields, and its outputs are outcome, is match, and response valid. The block lives in the workflow JSON as a dynamic Python block. Custom Python blocks also run on the Serverless API, so the Step 8 script can call this workflow. Its run function holds the whole check. Python python def run self, gate pass, sports car, alcohol, subject, description, parser error : if gate pass is not True: return {"outcome": "filtered out", "is match": False, "response valid": False} valid = parser error is False and all type v is bool for v in sports car, alcohol, subject and isinstance description, str and bool description.strip if not valid: return {"outcome": "error", "is match": False, "response valid": False} match = sports car is True and subject is True and alcohol is False return { "outcome": "llm match" if match else "llm rejection", "is match": match, "response valid": True, } The function returns filtered out when the gate stopped the image. It returns error when the parser reports a problem, when a Boolean field is not a real true or false, or when the description is empty. A complete answer becomes llm match only when is sports car and car is subject are true and alcohol visible is false. Every other complete answer becomes llm rejection. After Step 5, the editor shows the full chain from the image input to the final decision block. Your application can filter on outcome or is match without reading the raw LLM text. The Python script in Step 8 reads several of these outputs by name, so map them exactly as shown in the table. Output name Selector detections $steps.detector.predictions is sports car $steps.verdict.is sports car alcohol visible $steps.verdict.alcohol visible car is subject $steps.verdict.car is subject description $steps.verdict.description gemini output $steps.gemini.output gate pass $steps.gate decision.output car count $steps.car count.output parser error $steps.verdict.error status outcome $steps.final decision.outcome is match $steps.final decision.is match response valid $steps.final decision.response valid gemini input tokens $steps.gemini.input tokens gemini output tokens $steps.gemini.output tokens gemini inference id $steps.gemini.inference id The selectors assume the step names detector, car count, gate decision, gemini, verdict, and final decision, so change them if your blocks use other names. The workflow look like following. Try this workflow to view it, run a preview, or fork it into your own workspace. Step 6. Test the Gate in the Editor Click Test Workflow at the top of the editor, add your test images, and click Run to see the JSON output for each image. Use three images that cover every branch. The first shows only a car. The second shows a car with a wine glass, which should still reach Gemini. The third shows no car. The table shows how the gate handles typical detector outputs. Detections in the image Reaches Gemini Why car 0.91 Yes A car is present car 0.77, bottle 0.45 Yes A car is present, and Gemini checks the bottle car 0.88, wine glass 0.82 Yes A car is present, and Gemini checks the glass for alcohol car 0.35, bottle 0.70 Yes The car is above the 0.3 detector threshold wine glass 0.90 No The image has no car No detections No The image has no car When the gate pass output gives these results for your test images, the gate logic is correct and ready for tuning. Step 7. Tune the Gate on a Labeled Sample Without LLM Calls The 0.3 car threshold from Step 2 is a starting point, so choose the final value with data. Label a sample of your dataset as positive or negative. Run only Stage 1 on it once, with a low threshold, through the Serverless API, and cache the detections. Then replay the gate rule offline for a grid of thresholds. Each setting gives a pass rate, which sets the LLM cost, and a recall, which shows how many positives survive. This step makes no LLM calls. The labels go in a file named labels.csv, which is the answer key for the sample. It has one row per image. The image column holds the file name, and the match column holds yes when the image shows a sports car as the main subject with no visible alcohol. Every other image gets no. Decide each label by looking at the image, not from the workflow output. CSV image,match input 1.png,no input 2.png,no input 3.png,no input 4.png,yes input 5.png,yes input 6.png,no input 7.png,no input 8.png,yes input 9.png,yes input 10.png,yes input 11.png,no input 12.png,no input 13.png,yes input 14.png,no input 15.png,no Select the sample images at random, so the pass rate reflects your dataset. The sample also needs enough positives to measure recall well. With 10 positives, one missed image moves recall by 10 points. When positives are rare, add extra known positives to the sample, and expect the measured pass rate to come out a little high. Put the labeled images in a folder named sample. Then save the script below as tune gate.py, next to labels.csv and the sample folder. The script prints five lines, one for each car threshold. Each line shows the pass rate and recall for that threshold. You will see output similar to following. Pick the operating point with the lowest pass rate that still meets your recall target. Copy the car threshold into the detector confidence. Then confirm the choice with one end-to-end run on the same sample. If you use the optional exclusion rule from Step 3, add its threshold to the grid in the same way. Set workspace name and workflow id to your own values. The script reads the JPEG and PNG files in the images folder and keeps six of the workflow outputs from the table in Step 5. It sends eight requests at a time and writes one JSON line per image to results.jsonl. To use your own GPU server, change api url to its address, such as http://localhost:9001. Every line in results.jsonl has the same structure. The example below shows the saved result for input 4.png. JSON {"image": "input 4.png", "gate pass": true, "outcome": "llm match", "description": "A bright yellow supercar with its headlights on is captured driving through a dimly lit tunnel.", "gemini input tokens": 1295, "gemini output tokens": 57, "detections": {"image": {"width": 827, "height": 789}, "predictions": {"width": 553.0, "height": 345.0, "x": 426.5, "y": 439.5, "confidence": 0.9483212828636169, "class id": 3, "class": "car", "detection id": "c3fcc038-d388-43b0-9186-a1f07fa9744c", "parent id": "image"} }, "error": null} Each line has eight fields. image is the file name. gate pass is the gate decision. It is true when the image passed the gate, false when the gate stopped it, and null when the request failed. outcome is one of the four values from final decision. filtered out means the gate stopped the image. llm match means Gemini returned a complete answer with is sports car and car is subject as true and alcohol visible as false. llm rejection means the answer was complete but failed that rule. error means the answer was incomplete or the request failed. description is the one-sentence caption from Gemini. gemini input tokens and gemini output tokens count the tokens of the Gemini call. They are null when Gemini did not run. detections holds the image size and the Stage 1 boxes, in the same predictions format that the Object Detection Model block returns. In each box, x and y give the center point in pixels, and width and height give the size. Each box also has a confidence, a class name with its class id, a unique detection id, and a parent id for the source image. error is null unless the request failed. It then holds the error message, and every other field except image and outcome is null. Rerun only the images that show an error. The above result was for following image. In the 15-image sample, 13 images passed the car-only gate and two were filtered out. The gate therefore reduced the number of images sent to Gemini by 13.3%. Images containing both a detected car and bottles or wine glasses still passed, leaving Gemini to assess visible alcohol or alcohol branding. The outcome field keeps the four results apart, so a missing caption never hides an error. The pass rate is the share of lines where gate pass is true. This field comes straight from the gate decision, so an image with an incomplete Gemini answer still counts as a pass. Keep the results file after the run. Its detections field holds the Stage 1 annotations for every image, including the images that stopped at the gate. Load these annotations into a table or a database such as BigQuery. A later campaign can then filter them with a new rule and send only its own matches to the LLM, without running the detector again. The saved annotations cover only the classes in the Class Filter, so add any classes a later campaign needs before the first run. You can run the RF-DETR pre-filter on your own hardware with Roboflow’s free self-hosted inference. Local inference does not consume Roboflow credits; you cover the hardware and operating costs. Images that pass the filter still incur any applicable Gemini API charges. Measured Results on a Labeled Sample A labeled sample shows how the whole pipeline performs on your own data. Score it end to end with the Step 8 script and one short evaluation script. Run the Step 8 script on the labeled sample from Step 7. Change Path "images" to Path "sample" and results.jsonl to sample results.jsonl, so the results of the full run stay untouched. Then put the evaluation script below in a file named score.py, next to labels.csv and sample results.jsonl, and run it with python score.py. Python python import csv import json with open "labels.csv", newline="" as f: labels = {row "image" : row "match" == "yes" for row in csv.DictReader f } with open "sample results.jsonl" as f: rows = r for r in map json.loads, f if r "image" in labels relevant = r for r in rows if labels r "image" known = r for r in rows if r "gate pass" is not None passed = r for r in known if r "gate pass" matches = r for r in rows if r "outcome" == "llm match" def share part, whole : return f"{len part / max len whole , 1 :.1%}" def total field : return sum r field or 0 for r in rows print "sample size", len rows print "relevant images", len relevant print "pass rate", share passed, known print "filter recall", share r for r in relevant if r "gate pass" , relevant print "pipeline recall", share r for r in relevant if r "outcome" == "llm match" , relevant print "pipeline precision", share r for r in matches if labels r "image" , matches print "LLM requests", len passed print "Gemini input tokens", total "gemini input tokens" print "Gemini output tokens", total "gemini output tokens" print "errors", sum r "outcome" == "error" for r in rows input tokens = total "gemini input tokens" output tokens = total "gemini output tokens" estimated gemini credits = input tokens / 4 000 000 + output tokens / 666 667 print "Estimated Gemini credits", f"{estimated gemini credits:.6f}" Filter recall is the share of relevant images that pass the gate. Pipeline recall is the share that end as llm match, so it also counts the mistakes Gemini makes. Pipeline precision is the share of llm match images that are truly relevant. For the total cost, read the credits used during the run on the Credit Usage page in your workspace settings. With the managed rf key:account key, Gemini is billed per token, so the two token totals explain the Gemini part of that cost. If the Gemini block uses your Google key instead, add the Gemini charges from your Google account. Metric Measured value Sample size 15 images Relevant images 6 Pass rate 86.7% Filter recall 100.0% Pipeline recall 100.0% Pipeline precision 100.0% LLM requests 13 Gemini input tokens 16829 Gemini output tokens 812 Estimated Gemini Credits 0.005425 credits The evaluation metrics and token counts come from the 15-image sample. Gemini credits are estimated from the recorded tokens, while the earlier 10M-image figures are hypothetical. Gemini credit usage was estimated as input tokens ÷ 4,000,000 + output tokens ÷ 666,667 using the published Gemini 3 Flash Preview rates, excluding RF-DETR inference costs. These results demonstrate the workflow on a small 15-image sample. Evaluate it on a larger, representative dataset before scaling, and use a separate labeled test set to check the threshold selected during tuning. Step 9. Reuse the LLM Output to Improve Stage 1 Every full run produces new training data. For each candidate, the LLM output gives an is sports car answer. That answer is an image-level label, which fits an image classifier. A detector also needs a bounding box for each object, which takes one more labeling step. Train a custom detector. Connect the detector predictions to the same Dataset Upload block and turn on persist predictions, so the car boxes from RF-DETR arrive as pre-labels. In Roboflow Annotate, relabel each box as sports-car or car and fix missing or loose boxes. Then train RF-DETR on the reviewed boxes and swap its model ID into Step 2. Also review a random sample of images with the outcome filtered out. Gemini never sees these images, so its labels cannot show which relevant images Stage 1 misses. Every relevant image in this sample is a false negative to add to the training set. Both paths are a form of knowledge distillation. A large teacher model generates labels, which are reviewed and used to train a smaller student model. When deployed as Stage 1, the new model can reduce false positives while maintaining recall, allowing fewer irrelevant images to reach the LLM on comparable future batches. Another Example: Inspecting Finished-Product Packaging The same pre-filtering approach can help manufacturers find cartons with missing labels or visible damage. A production-line image archive may contain empty conveyor frames, unrelated products, and finished cartons. Only images containing the target carton need detailed inspection. Here's how the whole process works. In Roboflow Workflows, a custom RF-DETR model first detects the target carton. A Continue If block forwards images containing that detection to Gemini and skips the others before an LLM request is made. Gemini checks whether the required label is visible and looks for damage such as an open flap, tear, or crushed corner. It returns label present, visible damage, needs review, and a short inspection summary to support operator review. The gate checks carton presence, so a detected carton with a missing label still reaches Gemini. This focuses LLM processing on relevant images. Teams can measure the reduction in requests alongside how reliably the pre-filter retains cartons that need review. Conclusion Sign up for a free Roboflow account, build the workflow in the editor, and tune the gate on a labeled sample of your own data before you scale.