{"slug": "where-exactly-is-the-card-in-this-photo-image-segmentation-model-inside-a-maxed", "title": "Where exactly is the card in this photo? Image segmentation model inside a maxed-out lambda container", "summary": "An AWS developer built an image processing pipeline that uses a segmentation model inside a Lambda container to straighten, crop, and remove backgrounds from photos of AWS Builder Cards. The model, BiRefNet-General-Lite, runs on CPU via the rembg library, avoiding GPU costs, and the pipeline also uses a Bedrock vision model to read text from the processed image. The developer faced memory constraints, as the initial model barely fit in a 10 GB Lambda container.", "body_md": "**A crooked phone photo goes in, a clean straight card image comes out - no GPU, and nothing running when nobody uploads. It runs on the biggest lambda AWS sells, and the biggest is not the same as enough.**\n\nThis is the 3rd article in the series about the image processing pipeline.\n\nIn two previous articles we went over the whole image processing pipeline, which process AWS Builder Cards images:\n\n[From anonymous photo to a published page: An event-driven, AI, image processing pipeline on AWS](https://dev.to/aws-builders/from-anonymous-photo-to-a-published-page-an-event-driven-ai-image-processing-pipeline-on-aws-2m2n)\n\nand also over image classification model running in lambda container, which filters out the uploaded images and filter only valid AWS Builder Card images:\n\n[Is this even a valid card? Zero-shot image classification model in a lambda container](https://dev.to/aws-builders/is-this-even-a-valid-card-zero-shot-image-classification-model-in-a-lambda-container-58oj)\n\nToday's article is about vision segmentation model, which also runs in a lambda container, but it's role is to process the filtered image - remove background, straighten it and crop it.\n\nSpecifically about this part:\n\nSomebody photographs an AWS Builder Card lying on their desk and uploads it to my collection. What arrives is not a card - it is a \"photo\" of a card. Tilted and with some background around it.\n\nOne lambda has already looked at it before - a small, container based lambda `card-detect`\n\n, which filtered it as valid AWS Builder Card.\n\nSee this [article](https://dev.to/aws-builders/is-this-even-a-valid-card-zero-shot-image-classification-model-in-a-lambda-container-58oj) for more info.\n\nNow we have to clean that picture (straighten, crop, remove background), which is a job of lambda function `image-processor`\n\n. That lambda answers the question: *Where exactly is the card in this photo?*\n\nThe coworking between the two lambdas is simple: after the image is put into S3 `images/raw/`\n\nand filtered by `card-detect`\n\nas a valid one, `image-processor`\n\novertakes and start to processing the image.\n\nTo process the image as describe above, I need an **image segmentation model**. The one that goes pixel by pixel and says: \"card, background, card, background...\"\n\nI need it to run on `CPU`\n\n, inside a lambda container from the same reasons as `card-detect`\n\n(inside AWS, cheap, serverless).\n\nTo find a right model to do the job I need to know the answers to the same questions, as with `card-detect`\n\nlambda running `CLIP ViT-B-32, laion2b_s34b_b79k`\n\n:\n\nIn this case, this is the easiest one to answer.\n\nIt runs on `onnxruntime`\n\nbehind a small library called `rembg`\n\n, which means no `PyTorch`\n\nand no training framework anywhere in the image.\n\nThe other two get a section each further down, and question 1 is where the story is: the model I picked first barely fit at `10 GB`\n\n. Lambda also ties vCPU to memory, so that number decides how fast this thing runs, not just whether it survives.\n\nSo this is an article about memory, about the bill, and about the geometry that happens after the model has stopped talking.\n\nOne thing before we start, so it does not ambush you later: there are **two** models in this lambda. `BiRefNet`\n\nfinds the card. Right at the end, a `Bedrock`\n\nvision model reads the text off the finished image.\n\nBut let's start from the beginning\n\n``` python\nfrom rembg import new_session\nSESSION = new_session(\"birefnet-general-lite\")\n```\n\nAs an image segmentation model, I decided to go with `BiRefNet-General-Lite`\n\n, loaded through `rembg`\n\n. The ** Lite** is also the answer to the memory question.\n\nRemember *it works on my machine* meme from previous [article](https://dev.to/aws-builders/is-this-even-a-valid-card-zero-shot-image-classification-model-in-a-lambda-container-58oj)?\n\nHere is where I **really** earned it.\n\nWhat I was working with locally was the `BiRefNet-General`\n\nmodel. It worked fine (on my machine 🤣), so I shipped it.\n\nWhat I never did was to watch how much memory it was eating while it worked.\n\nI started the container at 4 GB memory. The first couple of invocations showed me the problem: `Runtime.OutOfMemory, Max Memory Used: 4095 MB of 4096`\n\n.\n\nSo I went to maximum I could - 10 GB memory. It worked fine, but not as per `CloudWatch`\n\n: `Max Memory Used: 9930 MB`\n\n. That's wonderful 97% of maximum container's capacity.\n\nSo yet it works, but then I tested 125 MB image and it crashed. There is no 12 GB to escape to, no bigger instance type, no flag to ask for more. With standard <20 MB image 97% of the memory is out of the question.\n\nBecause I could not buy more memory, I had to need less of it. The answer was lighter model from the same family: `BiRefNet-General-Lite`\n\n| Setup | What happened |\n|---|---|\n4 GB + `BiRefNet-General`\n|\nDies at `OOM` - `4095/4096 MB` used |\n10 GB + `BiRefNet-General`\n|\nworks, but peaks at `9930/10240 MB`\n|\n10 GB + `BiRefNet-General-Lite`\n|\nmedian peak `7540 MB` , and results are not different from `non-Lite` model |\n\nThere was a second fix too to lower the memory consumption and that's **image resize**.\n\nEvery upload whose longer side is over `2048px`\n\ngets downscaled to maximum `2048px`\n\ncap.\n\nIt shrinks the whole process: smaller composite, smaller sample, smaller arrays, smaller encode. That bought me about 300-400 MB of memory, depending on a picture size.\n\nMaxing the memory does buy me one thing for free. Lambda ties vCPU to memory, so 10 GB also gets me the most vCPUs Lambda could offer (6vCPUs) and this model needs every one, because there is no GPU under a lambda container.\n\n**The come from the docker build**\n\nThere are two ways to get model weights into a container\"\n\n`COPY`\n\nthe files in, as I did with `card-detect`\n\nlambda (see In this case, I went for option 2 and I used `rembg`\n\nto do it.\n\n```\nFROM public.ecr.aws/lambda/python:3.14\n# ... (package and requirements installation omitted) ...\n\nENV U2NET_HOME=/opt/models \\\n    NUMBA_CACHE_DIR=/tmp \\\n    MPLCONFIGDIR=/tmp\n\nRUN python -c \"from rembg import new_session; new_session('birefnet-general-lite')\" \\\n    && chmod -R a+rX /opt/models\n\nCOPY app.py ${LAMBDA_TASK_ROOT}/\nCMD [\"app.handler\"]\n```\n\n`new_session()`\n\nnormally loads a model at runtime, but if the model is not already cached, `rembg`\n\ndownloads it first. As being called during Docker buildz the `BiRefNet`\n\nweights are downloaded into `/opt/models`\n\nand become part of the resulting image. When Lambda starts, the weights are already there and no model download is needed at the runtime.\n\nAt my volume of the cars this lambda stays still within the **free tier**. Above the free tier, it'd cost roughly **$0.006 per card**, but I would never reach it.\n\nA comparable GPU setup would cost hundreds of dollars per month even when nobody is uploading cards and that's exactly what this project is about - mostly idle.\n\nRoughly speaking, the lambda approach would start to lose its cost advantage somewhere around **tens of thousands of cards per month**, but in this case the lambda is clear winner.\n\nThe whole lambda is actually five steps:\n\n`PNG`\n\n.**Not every step here is done by the image segmentation model**. This lambda combines AI part - `BiRefNet-General-Lite`\n\nmodel to distinguish **card vs background pixels**, everything else is deterministic geometry using `OpenCV`\n\nand `numpy`\n\n.\n\nI will run the whole process on this card:\n\nFirst, `OpenCV`\n\ndecodes supported file image (`.jpg`\n\n, `.png`\n\n, `.webp`\n\n, `.tiff`\n\n, and `.bmp`\n\n) into an **OpenCV image**, while Apple's `HEIC`\n\nformat is decoded using `Pillow`\n\nwith HEIC support.\n\nThen the file size check (for the memory reasons above) happens. If the longest side is over `2048px`\n\n, lambda downcales it to max cap `2048px`\n\n, while smaller images are untouched.\n\nThis resized image goes into the `BiRefNet-general-lite`\n\nsegmentation model to generate the mask (see next step) and the **same** resized image is what `OpenCV`\n\nwarps later.\n\n``` python\nMAX_DIM = 2048\n\ndef process_image(raw_bytes):\n    img = _decode_bgr(raw_bytes)\n    # ... (error guard and size capture omitted) ...\n    img = downscale(img, MAX_DIM)\n    # ... (size logging omitted) ...\n\n    ok, buf = cv2.imencode(\".png\", img)\n    # ... (encode guard trimmed) ...\n    work_bytes = buf.tobytes()\n\n    # ... (segmentation and geometry processing omitted) ...\n```\n\nThe model has to decide what is the card and what is the background - so the expected output should be something, where every pixel is scored for being part of the card or part of the background.\n\nThat something is called the mask, which is nothing more than black and white alpha channel stencil.\n\nThat mask is produced by `BiRefNet-General-Lite`\n\nmodel, where each pixel receives a value on how strong it belongs to the backgrund.\n\n`rembg`\n\nthen returns the result as a mask.\n\nThe pixel score goes between 0 as most noncard pixel, to 255 as most card pixel. I set up the line on 127:\n\n`get_mask()`\n\nthen cleans up the result with morphological operations, filling small holes and removing small specks.\n\nThe result is a clean **black and white stencil** of the card. From this point on, the upcoming geometry operations (see next steps) works with that stencil, rather than the original photo.\n\n``` python\ndef get_mask(raw_bytes, session):\n    # ... (docstring trimmed) ...\n\n    # BiRefNet segmentation\n    cutout = Image.open(\n        io.BytesIO(remove(raw_bytes, session=session))\n    ).convert(\"RGBA\")\n\n    # Extract and threshold the foreground mask\n    alpha = np.array(cutout)[:, :, 3]\n    _, mask = cv2.threshold(alpha, 127, 255, cv2.THRESH_BINARY)\n\n    # Remove holes and noise\n    k = cv2.getStructuringElement(cv2.MORPH_ELLIPSE, (15, 15))\n    mask = cv2.morphologyEx(mask, cv2.MORPH_CLOSE, k)\n    mask = cv2.morphologyEx(mask, cv2.MORPH_OPEN, k)\n\n    # Keep only the largest connected component\n    n, labels, stats, _ = cv2.connectedComponentsWithStats(mask, 8)\n    if n > 1:\n        largest = 1 + int(np.argmax(stats[1:, cv2.CC_STAT_AREA]))\n        mask = np.where(labels == largest, 255, 0).astype(np.uint8)\n\n    return mask\n```\n\nThe returned mask for Golden Jacket card looks like this:\n\n**And this is where the segmentation model's job ends**. It has separated the card from the background and everything that follows is a deterministic math.\n\nWhen I was thinking about straightening the card, I also thought about the *eye-to-brain path*. Taking a photo of a card from an angle, turns its rectangular shape into a quadrilateral, as you can see in the mask above.\n\nHow do I perform a perspective rectification and turn it back into a rectangle?\n\nThe answer is in the corners.\n\nLambda needs to identify the card's four corners first. A standard approach would be use `OpenCV`\n\nfunction `approxPolyDP`\n\n, which simplifies the detected outline into four corner points.\n\nBut here's where Builder Cards fight back: their corners are **rounded**, so the contour does not have four **sharp** corners.\n\nFor this reason `approxPolyDP`\n\nsometimes mistakenly placed a corner somewhere inside the rounded section, like here:\n\nSo instead calling a `approxPolyDP`\n\n, the more logical step would be if lambda fits a straight line along each of the four sides of the card. Now **the mask** comes handy, because it tells exactly where the card's contour is.\n\nBefore fitting the lines, it temporarily rotate the **contour points** so the card is approximately upright. This lets it separate the contour points into **top, bottom, left and right** edges.\n\nBecause the **rounded** corners would distort the straight line, lambda excludes the outer 18% at each end, when selecting the contour points used for each edge.\n\nIn other words, it uses the middle part of each side, where the contour follows the card's straight edge.\n\nIt then fits a straight line to each of those four sections.\n\nWhere two neighbouring lines intersect gives us a corner. Because the lines represent the card's straight edges rather than its rounded physical corners, those intersections can lie slightly outside the visible card.\n\nThose four intersections become the card's four corner points. Lambda then transforms those coordinates back to the original image coordinate system. The card itself is not straightened yet. These points are passed to the next step, where they are used for the actual perspective rectification.\n\n``` python\ndef find_quad(mask):\n    # ... (docstring and contour extraction omitted) ...\n\n    # Rotate the contour so the card becomes approximately axis-aligned.\n    (cx, cy), (rw, rh), ang = cv2.minAreaRect(c)\n    M = cv2.getRotationMatrix2D((cx, cy), ang, 1.0)\n    P = _rotate_pts(pts, M)\n\n    # Compute the bounding box of the rotated contour.\n    minx, maxx = P[:, 0].min(), P[:, 0].max()\n    miny, maxy = P[:, 1].min(), P[:, 1].max()\n    W, H = maxx - minx, maxy - miny\n    bx, by = 0.18 * W, 0.18 * H\n\n    # Ignore the rounded corners when selecting edge points.\n    cxlo, cxhi = minx + bx, maxx - bx\n    cylo, cyhi = miny + by, maxy - by\n\n    # Split contour points into the four card edges.\n    top    = P[(P[:, 1] < miny + by) & (P[:, 0] > cxlo) & (P[:, 0] < cxhi)]\n    bottom = P[(P[:, 1] > maxy - by) & (P[:, 0] > cxlo) & (P[:, 0] < cxhi)]\n    # ... (left and right the same way) ...\n\n    # Fit a line to each edge and find their intersections.\n    if min(len(top), len(bottom), len(left), len(right)) >= 15:\n        lt, lb = _fit_line(top), _fit_line(bottom)\n        ll, lr = _fit_line(left), _fit_line(right)\n        tl, tr = _intersect(lt, ll), _intersect(lt, lr)\n        br, bl = _intersect(lb, lr), _intersect(lb, ll)\n\n    # ... (rotate corners back and fallback methods omitted) ...\n```\n\nBut wait, should I blindly trust four invisible points? Well, No.\n\nFor debugging reasons, `encode_corners()`\n\ndraws the detected quadrilateral and its four corner points onto the original image., which is saved as `images/raw/<stem>__corners.png`\n\n.\n\nIn case a card comes out cropped wrong, I open this overlay and see **immediately** whether the corners were the problem.\n\nNow that we have the four corners identified, `OpenCV`\n\ncalls `getPerspectiveTransform`\n\nto calculate a perspective transformation that maps those four points onto the four corners of a rectangle, and then `warpPerspective`\n\napplies that transformation to the original image. This is where the card is actually straightened and warped.\n\nThe result is a **rectangular image** containing the straightened card, **but the background is still present around the rounded corners.**\n\nThis is where the mask comes back in.\n\nIt goes through the exact same perspective transformation as the photo. Once it's straight and perfectly aligned into a rectangle, lambda can replace those corner pixels with the configured background color.\n\n``` python\ndef rectify(img_bgr, quad, mask, background=\"white\"):\n    # ... (docstring omitted) ...\n\n    # Order the detected corners consistently.\n    src = order_corners(quad)\n    tl, tr, br, bl = src\n\n    # Compute the dimensions of the output rectangle.\n    w = int(max(\n        np.linalg.norm(br - bl),\n        np.linalg.norm(tr - tl),\n    ))\n    h = int(max(\n        np.linalg.norm(tr - br),\n        np.linalg.norm(tl - bl),\n    ))\n\n    # ... (invalid-size guard omitted) ...\n\n    # Map the four card corners to the four corners of the rectangle.\n    dst = np.float32([\n        [0, 0],\n        [w - 1, 0],\n        [w - 1, h - 1],\n        [0, h - 1],\n    ])\n    M = cv2.getPerspectiveTransform(src, dst)\n\n    # Straighten the original image.\n    warped = cv2.warpPerspective(img_bgr, M, (w, h))\n\n    # Straighten the mask using exactly the same transformation.\n    warped_mask = cv2.warpPerspective(\n        mask, M, (w, h),\n        flags=cv2.INTER_NEAREST,\n    )\n\n    # Threshold and erode the transformed mask.\n    _, warped_mask = cv2.threshold(\n        warped_mask, 127, 255, cv2.THRESH_BINARY\n    )\n    er = cv2.getStructuringElement(\n        cv2.MORPH_ELLIPSE, (3, 3)\n    )\n    warped_mask = cv2.erode(\n        warped_mask, er, iterations=1\n    )\n\n    # ... (transparent-background branch omitted) ...\n\n   # Replace the background with the selected solid color\n    FILLS = {\"white\": 255, \"black\": 0}\n    if background not in FILLS:\n        logger.warning(\n            \"[ImageProcessor] unknown BACKGROUND=%r (expected white, black or none) - falling back to black\",\n            background,\n        )\n    fill = FILLS.get(background, 0)\n    outside = warped_mask == 0\n    warped[outside] = fill\n\n    return warped\n```\n\nIn this step, we still don't have a proper `png`\n\nas a result, because `rectify()`\n\n**returns raw pixels, not a file. **\n\nThe result is a straight rectangular card stored in the memory as an OpenCV image, with the remaining background replaced by the black, white, or made transparent when background is set none. It's still not a valid `png`\n\nfile yet.\n\nThe background color black, white, or none is defined as an environmental variable in terraform:\n\n```\nresource \"aws_lambda_function\" \"image_processor\" {\n  # ... (code omitted) ...\n\n  environment {\n    variables = {\n      BACKGROUND           = \"black\" # could be also \"white\" or \"none\"\n  # ... (code omitted) ...\n```\n\nNow the bytes resulted from `rectify()`\n\nare turned into proper `png`\n\nby `process_image()`\n\n.\n\n``` python\ndef process_image(raw_bytes):\n    # ... (image decoding, downscaling, mask generation, corner detection,\n    #      and rectification omitted) ...\n\n    out = rectify(img_bgr, quad, mask, background=BACKGROUND)\n\n    ok, png_buf = cv2.imencode(\".png\", out)\n    if not ok:\n        raise RuntimeError(\"PNG encoding failed\")\n\n    png = png_buf.tobytes()\n\n    # ... (corners overlay encoding omitted) ...\n\n    return png, corners_png\n```\n\nThe reason I choose a .png is because in case of the transparent background, only png supports the aplha channel option.\n\nFinally the handler() puts the image into the images/finished/ prefix of the S3 bucket and that's the final, straightened, backgroundless png image ready to be used.\n\nThose were the steps that made the card transform from left to right:\n\nThere are actually two other calls and that's where the second model finally shows up:\n\n`png`\n\nbytes to `Bedrock`\n\nfor a `multimodal model`\n\no extract the text from the card, such as `DynamoDB`\n\nas a card's metadata.\n\n``` python\ndef handler(event, context):\n    # ... (S3 event parsing, guards, download, and other unrelated code omitted) ...\n\n            png, corners_png = process_image(raw)\n\n            # ... (no-card guard, corners-overlay write, and filename\n            #      metadata parsing omitted) ...\n\n            fields = {\"title\": \"\", \"effect\": \"\", \"description\": \"\"}\n\n            try:\n                fields = extract_card_fields(png)\n            except Exception as ex:\n                logger.exception(\n                    \"[ImageProcessor] Bedrock extraction failed: %s\", ex\n                )\n\n            try:\n                write_card_item(\n                    card_id,\n                    event_name,\n                    year,\n                    key,\n                    out_key,\n                    fields,\n                    uploader,\n                )\n            except Exception as ex:  # noqa: BLE001\n                logger.exception(\n                    \"[ImageProcessor] DynamoDB write failed: %s\", ex\n                )\n\n            # Upload the rectified card image.\n            s3.put_object(\n                Bucket=bucket,\n                Key=out_key,\n                Body=png,\n                ContentType=\"image/png\",\n            )\n\n    # ... (remaining handler code omitted) ...\n```\n\nMore on both steps in **this article - TBD Article 4**\n\nI built this project for collectible (AWS builder) cards, but almost none of it is about it.\n\n**Running cheap**\n\nGPUs are not always necessary to run the custom or external model inside the AWS. Unless you can fit it under 10GB memory and CPUs are enough to do the job - you have yourself a model doing the job and running very cheap.\n\n**You don't need a model to do everything**\n\nWhat can be done deterministically, I'd say it safer to do it that way.\n\n**You don't need a model with PhD to do simple job**\n\nLite model is sometimes as good as general one, just test it locally and decide.\n\nExcept everything above, the `card-detect`\n\n+ `image-processor`\n\n+ `Bedrok`\n\n+ `DynamoDB`\n\ncombo can by used for any other image category, just fork the [REPO](https://dev.torepo), modify it o your needs and you are good to go.\n\n** BiRefNet** - the model that draws the mask, by\n\n** rembg** - the wrapper I load it through, by\n\nThe exact file I am running is `BiRefNet-general-bb_swin_v1_tiny-epoch_232.onnx`\n\n- and that `bb_swin_v1_tiny`\n\ninthe name is the whole memory story in one string. *Lite* means a Swin-v1-Tiny backbone.\n\n[From anonymous photo to a published page: An event-driven, AI, image processing pipeline on AWS](https://dev.tolink)\n\n[Is this even a valid card? Zero-shot image classification model in a lambda container](https://dev.tolink)\n\nWhat does the card say? Text extraction using Amazon Nova Lite 2 - TBD\n\nWho reviews the reviewer? Building the human step in an AI pipeline - TBD", "url": "https://wpnews.pro/news/where-exactly-is-the-card-in-this-photo-image-segmentation-model-inside-a-maxed", "canonical_source": "https://dev.to/aws-builders/where-exactly-is-the-card-in-this-photo-image-segmentation-model-inside-a-maxed-out-lambda-51da", "published_at": "2026-08-11 00:02:46+00:00", "updated_at": "2026-08-11 00:15:02.072557+00:00", "lang": "en", "topics": ["computer-vision", "machine-learning", "ai-infrastructure", "developer-tools"], "entities": ["AWS", "BiRefNet", "rembg", "Bedrock", "Lambda", "S3", "CLIP ViT-B-32"], "alternates": {"html": "https://wpnews.pro/news/where-exactly-is-the-card-in-this-photo-image-segmentation-model-inside-a-maxed", "markdown": "https://wpnews.pro/news/where-exactly-is-the-card-in-this-photo-image-segmentation-model-inside-a-maxed.md", "text": "https://wpnews.pro/news/where-exactly-is-the-card-in-this-photo-image-segmentation-model-inside-a-maxed.txt", "jsonld": "https://wpnews.pro/news/where-exactly-is-the-card-in-this-photo-image-segmentation-model-inside-a-maxed.jsonld"}}