{"slug": "is-this-even-a-valid-card-zero-shot-image-classification-model-in-a-lambda", "title": "Is this even a valid card? Zero-shot image classification model in a lambda container", "summary": "An AWS developer built a zero-shot image classifier using a CLIP model inside a Lambda container to distinguish AWS Builder Cards from other images without any training data or GPU. The model, which embeds images and text into the same space, runs on CPU with 2 GB of memory and uses ONNX Runtime, allowing the developer to teach it new card types by simply writing additional English descriptions.", "body_md": "**No training, no training data and no GPU. This classifier's whole brain is a list of English sentences I typed by hand - and I can teach it a new card type by writing one more.**\n\nIn [Previous article](https://dev.to/aws-builders/from-anonymous-photo-to-a-published-page-an-event-driven-ai-image-processing-pipeline-on-aws-2m2n), I walked through the whole pipeline that turns a crooked phone photo of a collectible AWS Builder Card into a published page. That pipeline goes like this:\n\nToday's article goes deep into a **cheap image classifier**, the `image classification model`\n\ninside the lambda container. Specifically it's this part:\n\nBecause the upload is public and anonymous, the AWS Builder Cards may not be the only images that lands in the upload bucket.\n\nIt could be cat photos, Pokemon cards, poker cards, sports collectibles, etc... Telling those apart from a real AWS card sounds like easy for the human eye, but not so easy for the ML model, considering no model was specifically trained on the AWS Builder Cards.\n\nThe path *eye-brain* is the key concept here. Just as your brain works by distinguishing the AWS Builder Card by some specifications, same you can tell the model to do.\n\nThis is what a real AWS Builder Card look like:\n\nKeep that layout in mind, because the whole trick below is built on describing it in plain English:\n\n`card-detect`\n\nlambda\nOne of the first tasks in my image processing pipeline in to answer the question: *is this a valid AWS Builder Card?*.\n\nThis is the job of `card-detect`\n\n, which should filter valid cards from anything else, before expensive processing starts. It uses **vision classification model** and runs in lambda container.\n\nBut before I put (any) model into the container, I have to answer the 3 questions. Based on the answers, which comes from running the model locally (in my case on M4 macOS), I can tell if I can use this or that model and put in inside the container, to do this or that particular job.\n\nQ1: **How much CPU and memory does it need to do the job?**\n\nA1: `2048 MB`\n\nand plain CPU.\n\nQ2: **Where do the weights come from?**\n\nA2: I export them myself on my laptop and they are baked into the image and already sit at `/opt/model`\n\nat cold start.\n\nQ3:. **What runs at the inference?**\n\nA3: No `PyTorch`\n\n(unlike locally) anywhere in the image, but `onnxruntime`\n\n.\n\nIn other words: I have to find a model that is capable to do the job and still fits into lambda container\n\nWhy **lambda container** and not `Bedrock models`\n\n, `SageMaker`\n\nor GPU instances?\n\n`SageMaker`\n\ncan become expensive)Logically, only reasonable option was Lambda function deployed as `ECR`\n\ncontainer.\n\nBecause of the container limitations I can allocate maximum of 10 GB of memory, it has to be amodel capable of delivering the task within that hardware requirements.\n\nI found one, and there's something about this model that keep surprising me the most - it need **no training at all!**\n\nThe winner model for this case for me is from `Contrastive Language-Image Pre-training (CLIP)`\n\nfamily, which can do the image classification within 2 GB of memory ans still very fast.\n\nThe way it works is absolutely fascinating! `CLIP`\n\nmodels **embeds images and text into the same space**.\n\nThat means a *cat image* and the actual *word cat* end up close to each other. You describe what to look for and it looks for **that** in the picture. If it finds it, it knows the description and the picture goes close together. And that's exactly how I am using it here.\n\nIt compares my photo against a fixed set of *labels* which I created, and scores how close each one sits against the photo.\n\nThen it looks into the script and knows into which category (AWS computing trading card, Pokemon trading card, etc...) it score to.\n\n`softmax`\n\nturns those scores into percentages that sum to 100%, so when I upload an AWS card, the shares come out roughly like this:\n\n| # | Scored phrase | Share |\n|---|---|---|\n| 1 | an AWS cloud computing trading card | 34.0% |\n| 2 | a tech company card with a pixel-art icon, a service name and a QR code | 22.0% |\n| 3 | a software product promotional trading card | 9.0% |\n| 4 | a Pokemon trading card | 1.2% |\n| 5 | a Magic: The Gathering trading card | 0.8% |\n| 6 | a Yu-Gi-Oh trading card | 0.6% |\n| 7 | a standard playing card | 0.9% |\n| 8 | a poker playing card | 0.7% |\n| 9 | an ice hockey trading card | 0.3% |\n| 10 | a basketball trading card | 0.3% |\n| 11 | an American football trading card | 0.3% |\n| 12 | a baseball trading card | 0.4% |\n| 13 | a soccer trading card | 0.3% |\n| 14 | a sports trading card with a photo of an athlete | 0.5% |\n| 15 | a collectible game card with fantasy artwork | 1.7% |\n| 16 | a photo of a person | 0.2% |\n| 17 | a selfie | 0.1% |\n| 18 | an animal | 0.1% |\n| 19 | a landscape photo | 0.2% |\n| 20 | a screenshot of an app or website | 2.5% |\n| 21 | a piece of food | 0.2% |\n| 22 | an everyday object | 3.5% |\n| 23 | a sheet of paper or a document | 6.0% |\n| 24 | a business card | 12.5% |\n| 25 | a cartoon or meme image | 1.2% |\n| 26 | a photo of a room or building | 0.5% |\n\nLet's say I want to add another category - maybe a **monopoly card**. In the \"standard\" machine learning, doing that means collecting labeled photos of monopoly cards and hours of retraining.\n\nWith `CLIP`\n\nI just write a new `label`\n\nwhere I describe it and when it compares the label and the picture it finds the match. No retraining, no fine-tning at all!\n\nIt is me who decides which labels it should recognize, because I define them:\n\n```\nAWS_LABELS = [\n    \"an AWS cloud computing trading card\",\n    \"a tech company card with a pixel-art icon, a service name and a QR code\",\n    \"a software product promotional trading card\",\n]\nCOMPETITOR_LABELS = [\n    \"a Pokemon trading card\",\n    \"a Magic: The Gathering trading card\",\n    # ... (the other 10 competitor labels trimmed - see the table above) ...\n]\nNONCARD_LABELS = [\n    \"a photo of a person\",\n    \"a selfie\",\n    # ... (the other 9 non-card labels trimmed - see the table above) ...\n]\nALL_LABELS = AWS_LABELS + COMPETITOR_LABELS + NONCARD_LABELS\n```\n\nBut why do I list the **competitor cards** at all?\n\nBecause `softmax`\n\nis a **zero-sum game** - all 26 shares must sum to 100%, so every percent one label wins, another label loses.\n\n**It wasn't always like this!**\n\nMy first version had only generic labels like *\"a trading card\"* and it was a disaster! **Every** card scored ~1.0:\n\n*Pokemon card: ~1.0,\nPoker card: ~1.0,\nAWS card: ~1.0*\n\nThe fix was not a better AWS label, but rather giving a Pokemon and others **their own labels**. Now the Pokemon card's probability lands on *\"a Pokemon trading card\"* (where it fits best) instead of leaking into my AWS labels and its AWS score collapses to almost zero - **0.016**, when I measured it later.\n\n**Adding a card type as its own label is how you \"subtract\" points from its AWS score.**\n\nImagine, with all those labels I have, the Pokemon card scores 87% in the Pokemon category, but 0.016% in AWS Builder Cards category.\n\n**The model** I am actually using here specifically is `open_clip ViT-B/32`\n\n, pretrained `laion2b_s34b_b79k`\n\n.\n\nIt has probably never seen an AWS Builder Card, but it has seen plenty of \"*pixel art*\", \"*QR code*\" and \"*service icon*\", and those are exactly the words I packed into my three AWS sentences at the top of the list.\n\nNow here is the catch:\n\n`CLIP`\n\nlives inside`PyTorch`\n\n, and`PyTorch`\n\nturns a lambda into a~2 GB imagewith a slow cold start.\n\nYou definitely do not want it inside your lambda, because it makes it anything but fast.\n\nHere's the good news:\n\nIn zero-shot classification only the\n\nimage encoderhas to run for every upload, the other part (the labels) is constant, because they never change at runtime.\n\nIf I wanna make the pancake, I do not need to export whole kitchen where the recipe was created and the chef who created it. I just need **the** recipe.\n\nThere is a **one-time build script**, `build_onnx.py`\n\n, which I run (and you will too) locally and it does two things:\n\nIt **exports** the image encoder into a portable model format `Open Neural Network Exchange (ONNX)`\n\n.\n\nThe export works like watching a chef (`CLIP`\n\n) cook the dish in his super expensive fancy kitchen (`PyTorch`\n\n) for once and writing down every step. Then you just reproduce the steps in your own kitchen (`ONNX`\n\n) which is (sadly) way less fancy.\n\n`torch.onnx.export`\n\npushes one fake image through the encoder, records every math operation it performs plus all the learned weights, and saves the result to disk:\n\n``` python\ndef main():\n    # ... (docstring and model loading trimmed) ...\n    visual_path = os.path.join(OUT_DIR, \"visual.onnx\")\n    dummy = torch.zeros(1, 3, IMAGE_SIZE, IMAGE_SIZE, dtype=torch.float32)\n    torch.onnx.export(\n        model.visual,\n        dummy,\n        visual_path,\n        input_names=[\"image\"],\n        output_names=[\"embed\"],\n\n        # allow variable batch size (runtime sends 1 at a time)\n        dynamic_axes={\"image\": {0: \"batch\"}, \"embed\": {0: \"batch\"}},\n        opset_version=17,\n    )\n```\n\nThat produces two files:\n\n`visual.onnx`\n\n- the \"`visual.onnx.data`\n\n- the learned weights (~335 MB).From now on, the model can be executed by `onnxruntime`\n\n- a small engine that only knows how to follow `ONNX`\n\ninstructions. It cannot train anything, it cannot learn anything, and that is exactly why it fits in a lambda. No PyTorch or any other ML framework needed anymore at inference. It just follows your 335 MB pancake recipe.\n\nIt **vectorizes** the labels.\n\nThe text encoder runs exactly **once** - all 26 labels go in as one batch and only the answers (26 vectors of 512 numbers each) get saved into `text.npz`\n\n, together with the group counts:\n\n``` python\ndef main():\n    # ... (the ONNX export above and the aws_count/card_count comments trimmed) ...\n    with torch.no_grad():\n        tf = model.encode_text(tokenizer(ALL_LABELS))\n        tf = tf / tf.norm(dim=-1, keepdim=True)\n    text_embeds = tf.numpy().astype(np.float32)\n    logit_scale = float(model.logit_scale.exp())\n\n    text_path = os.path.join(OUT_DIR, \"text.npz\")\n    np.savez(\n        text_path,\n        embeds=text_embeds,\n        logit_scale=np.float32(logit_scale),\n        aws_count=np.int64(len(AWS_LABELS)),\n        card_count=np.int64(len(AWS_LABELS) + len(COMPETITOR_LABELS)),\n        labels=np.array(ALL_LABELS),\n    )\n```\n\nThat means the model running in the lambda doesn't have to create the vectors from labels every time. They were already created locally and shipped. This is something that you by far can do locally, because the labels do not change often, if at all.\n\nThis is also the \"regenerate the sentence points\" step from the monopoly example: add a labeled sentence, re-run `build_onnx.py`\n\n, run `terraform apply`\n\n-> done.\n\n`text.npz`\n\nchanges its hash, terraform rebuilds the container image automatically and no manual intervention is needed.\n\nThe Docker puts all three files into the container at `/opt/model`\n\n, and on cold start the lambda loads them into module globals:\n\n```\nFROM public.ecr.aws/lambda/python:3.14\n# ...ommited\nENV MODEL_DIR=/opt/model\nCOPY onnx_out/ /opt/model/\nRUN chmod -R a+rX /opt/model\n# ...ommited\n```\n\nLambda then simply loads them in to the code as:\n\n```\n_SESSION = ort.InferenceSession(VISUAL_ONNX, providers=[\"CPUExecutionProvider\"])\n_INPUT_NAME = _SESSION.get_inputs()[0].name\n_TEXT = np.load(TEXT_NPZ, allow_pickle=True)\n_TEXT_EMBEDS = _TEXT[\"embeds\"].astype(np.float32)\n_LOGIT_SCALE = float(_TEXT[\"logit_scale\"])\n_CARD_COUNT = int(_TEXT[\"card_count\"])\n_AWS_COUNT = int(_TEXT[\"aws_count\"]) if \"aws_count\" in _TEXT else _CARD_COUNT\n```\n\nThere is no torch at the runtime, no tokenizer, no text encoder. The heavy \"*PyTorch kitchen*\" was needed exactly once when 3 files were produced locally.\n\nAfter first run, you don't need to run it again until you change the labels (add monpoly card)\n\nThe whole **inference** is actually about ten lines:\n\n``` python\ndef p_card(raw_bytes):\n    # ... (docstring and section comments trimmed) ...\n    img = Image.open(io.BytesIO(raw_bytes))\n    x = preprocess_np(img)\n    emb = _SESSION.run(None, {_INPUT_NAME: x})[0]\n    emb = emb / np.linalg.norm(emb, axis=-1, keepdims=True)\n    logits = _LOGIT_SCALE * (emb @ _TEXT_EMBEDS.T)\n    z = logits - logits.max(axis=-1, keepdims=True)\n    e = np.exp(z)\n    probs = (e / e.sum(axis=-1, keepdims=True))[0]\n    p_aws = float(probs[:_AWS_COUNT].sum())\n    p_any_card = float(probs[:_CARD_COUNT].sum())(AWS+competitor): LABELS a rejection, never gates\n    return p_aws, p_any_card\n```\n\n...and **the decision** just one condition:\n\n``` python\ndef handler(event, context):\n    # ... (docstring, the threshold log line and the results list trimmed) ...\n    for record in event.get(\"Records\", []):\n        # ... (unwrapping and guards omitted) ...\n        try:\n            # ... (the S3 fetch and the scoring omitted) ...\n            if p_aws >= DROP_THRESHOLD:\n                verdict = \"aws-card\" if p_aws >= KEEP_THRESHOLD else \"aws-unsure\"\n                _forward_to_processor(record)\n                action = \"forwarded\"\n            else:\n                verdict = \"non-aws-card\" if p_any >= DROP_THRESHOLD else \"not-a-card\"\n                rejected_key = _quarantine(bucket, key)\n                _publish_reject(bucket, key, rejected_key, p_aws)\n                action = \"rejected\"\n            # ... (the structured log line and the results list omitted)\n```\n\nTwo thresholds (0,20 and 0,50) do the work.\n\nThe code looks only at `p_aws`\n\n- the sum of the three AWS labels:\n\n`p_aws`\n\n≥ 0,50 - AWS card`_forward_to_processor()`\n\nasynchronously invokes the next lambda with the same S3 event, so it starts processing it.`p_aws`\n\n< 0,50 - AWS unsure`p_aws`\n\n< 0,20 - not an AWS card`_quarantine()`\n\nmoves it to the `images/rejected/`\n\nprefix and `_publish_reject()`\n\nsends me an email about the rejected upload, so I can inspect it manually - just in case this was a valid card and the model got it wrong.That closes two of the three questions from the top.\n\n`onnxruntime`\n\n, four packages, no framework. Which leaves the first question - why this model gets a container of its own at all.Now only the third one remains, which also answers why do I need its own lambda.\n\nCould I run both vision models (image classification and image segmentation) in the same container?\n\nI did it locally and it worked perfectly.\n\nThat's right. Locally it worked perfectly as one entity, but going into the container there are some requirements I had to follow.\n\n**Memory cap**\n\nWhen tested locally, image classification model used 2 GB memory and image segmentation model almost 10 GB. This would me not possible in the container where the memory cap is 10 GB. Even though later I was able to make image segmentation model use only 7,5 GB by shrinking the images to a maximum pixel cap and use a lighter model, together it still would be 9,5 GB+, which is kinda risky.\n\n**Cheap gate, expensive processor**\n\nThis function is very fast - finishes literally in seconds. Imagine a user uploading 5 cat images. Even if I was able to run both models inside a single lambda, spinning up the whole 10 GB container just to tell *this is a cat -> go and quarantine it* would be an overkill.\n\nTherefore this lambda must be a cheap gate in front of (relatively) expensive workloads, the flood of uploads cannot end up in expensive invocations.\n\n**Least privilege**\n\nThis is already an evergreen, but separating lambdas allows me to assign them different IAM policies. The `card-detect`\n\nrole can read and delete in `images/raw/`\n\n, write to `images/rejected/`\n\n, invoke the `image-processor`\n\nand publish to one `sns`\n\n. No `Bedrock`\n\n, no `DynamoDB`\n\n, no secrets.\n\nThe threshold and the memory story came from the real local testing.\n\nMy local testset of different shapes and different file formats was this:\n\nand the results went like this:\n\n| Upload | `p_aws` |\nOutcome |\n|---|---|---|\n| Real AWS cards | 0,31-0,93 | forwarded |\n| Pokemon | 0,011-0,016 | rejected |\n| Sports cards | 0,0002-0,0004 | rejected |\n| Poker | 0,0001 | rejected |\n| Scenery | 0,0001 | rejected |\n| Cat | 0,0001 | rejected |\n\nThe lowest AWS Builder Cards scored 0,31 so it was below the confident threshold but still within unsure, thus forwarded for processing.\n\nThe highest ever scored non AWS card was a Pokemon with 0,016, so not even close to the unsure threshold, thus not forwarded\n\nBoth thresholds are lambda **env vars** set by `terraform`\n\n, so I can change them anytime and this re-tuning needs no rebuild at all.\n\nAt beginning I thought distinguishing a cat photo from an AWS Builder Card sounds like the easy part, and that it was the part I got wrong first.\n\nI thought my job was to describe an AWS Builder Card really well. It turned out, it works better when describing **everything else** well enough that it stops competing for the card's score.\n\nMy first version had a generic label and no Pokemon/sports/poker labels and it confidently called a Pokemon card an AWS card with ~1.0 score.\n\nAter what I've learned here, I can confidently say: **in a zero-sum score you do not win by describing your class better, you win by describing its neighbours at all**, which is btw a whole article in one sentence.\n\nSo yes - take my code, swap the labels, re-run `build_onnx.py`\n\n, `terraform apply`\n\n, and the gate scores whatever **you** describe. With one condition the article already paid for: your labels have to be **specific** and every category you want to keep out needs its own sentence.\n\n**You don't need to train model for something it was never trained for**\n\n`CLIP`\n\nfamily models probably never seen the AWS builder card, but it seen a lot of other stuff, including a pixel art, QR code, other cards, etc... You can use \"that other stuff\" to describe it in the labels and also to filter out what it knows. Then what stays is the thing it has never seen - exactly what you want it to know.\n\n** CLIP** - the idea that you can classify an image against sentences you make up at runtime, without training anything.\n\n** open_clip** - the open implementation I actually import, by\n\n** laion2b_s34b_b79k** - the weights, trained by\n\n`2B`\n\ndataset, `34B`\n\nsamples seen, batch `79k`\n\n.[From anonymous photo to a published page: An event-driven, AI, image processing pipeline on AWS](https://dev.to/aws-builders/from-anonymous-photo-to-a-published-page-an-event-driven-ai-image-processing-pipeline-on-aws-2m2n)\n\n[Where exactly is card in this photo? Image segmentation model inside a maxed-out lambda container](https://dev.tolink)\n\n[What does the card say? Text extraction using Amazon Nova Lite 2](https://dev.tolink)\n\n[Who reviews the reviewer? Building the human step in an AI pipeline](https://dev.tolink)", "url": "https://wpnews.pro/news/is-this-even-a-valid-card-zero-shot-image-classification-model-in-a-lambda", "canonical_source": "https://dev.to/aws-builders/is-this-even-a-valid-card-zero-shot-image-classification-model-in-a-lambda-container-58oj", "published_at": "2026-08-10 22:03:51+00:00", "updated_at": "2026-08-10 22:17:07.279212+00:00", "lang": "en", "topics": ["computer-vision", "machine-learning", "artificial-intelligence", "developer-tools"], "entities": ["AWS", "CLIP", "Lambda", "ONNX Runtime", "ECR", "SageMaker", "Bedrock"], "alternates": {"html": "https://wpnews.pro/news/is-this-even-a-valid-card-zero-shot-image-classification-model-in-a-lambda", "markdown": "https://wpnews.pro/news/is-this-even-a-valid-card-zero-shot-image-classification-model-in-a-lambda.md", "text": "https://wpnews.pro/news/is-this-even-a-valid-card-zero-shot-image-classification-model-in-a-lambda.txt", "jsonld": "https://wpnews.pro/news/is-this-even-a-valid-card-zero-shot-image-classification-model-in-a-lambda.jsonld"}}