What does the card say? Text extraction using Amazon Nova 2 Lite An AWS developer integrated Amazon Nova 2 Lite via Amazon Bedrock into an event-driven image processing pipeline to extract text and icon metadata from AWS Builder Cards. The multimodal model is invoked through a simple converse() API call, replacing Amazon Textract and Amazon Rekognition because those services cannot interpret the card's icons. The extracted data is stored in DynamoDB as card metadata. No container, no weights, no memory limit to worry about. The only thing I own in this model is a prompt and it did not always go well. Two models in this pipeline have already had their own article: card-detect https://dev.to/aws-builders/is-this-even-a-valid-card-zero-shot-image-classification-model-in-a-lambda-container-58oj lambda, which answers is this even an AWS Builder Card? , and image-processor https://dev.to/aws-builders/where-exactly-is-the-card-in-this-photo-image-segmentation-model-inside-a-maxed-out-lambda-51da lambda, which answers where exactly is the card in this photo? Whole pipeline in a nutshell is also described here https://dev.to/aws-builders/from-anonymous-photo-to-a-published-page-an-event-driven-ai-image-processing-pipeline-on-aws-2m2n . So what is the current situation? The card is straight and clean by now, but I still don't know what it says. This article describes what happens next. There is another model which is not part of any lambda, but is being called by image-processor . That model is Nova 2 Lite living in Amazon Bedrock and it extracts the card's text. Once extracted, the same lambda write that extracted text to DynamoDB as a card's metadata. Both previous articles run the same three questions before putting a model into a lambda. Since this model does not run in the lambda, I don't even care about the answers: Bedrock .No container to build, nothing to bake into an image, no ECR push, no cold start to measure, no 10240 MB ceiling to hit. The model is Amazon Nova 2 Lite and it doesn't even run in my account. The lambda make an API call towards Bedrock , where the text is extracted and later stored in DynamoDB as a card's metadata. The only thing I have to worry about here is IAM permissions and writing a good prompt. Reading a card is not just OCR Here is the actual problem and it is not just the text recognition: A player knows what an orange circle means, but model has to be told. Some cards carry all 3 icons, others just 1 or 2 and there are cards with none. AWS offer couple of resources capable of recognizing the text: Amazon Textract Amazon Reckognition But their inability to understand and interpret the icons made them unusable for this case. This is the only reason I used multimodal model, capable of both tasks. The whole thing is just a converse API call: python def extract card fields png bytes : ... docstring trimmed ... Invoke the Bedrock vision model. resp = bedrock.converse modelId=BEDROCK MODEL ID, system= {"text": EXTRACT SYSTEM} , messages= { "role": "user", "content": {"text": EXTRACT PROMPT}, {"image": {"format": "png", "source": {"bytes": png bytes}}}, , } , inferenceConfig={"maxTokens": 1200, "temperature": 0.0}, Parse the model response. text = resp "output" "message" "content" 0 "text" return parse json text I set up temperature to 0.0 because I want it to always return answer it is most confident about. I am expecting always the same answer, no matter how many times the same card goes in. That actually makes sense, since it is reading the text. In order for the model to do anything, it needs its instructions - the prompt. Especially in case of the icons. ... beginning and title omitted ... TASK 2: EFFECT The effect is the gameplay area in the middle of the card. The effect area may contain small icons. Not all icon types are always present on s csrd. A card may have: 0 icons, or 1 icon, or 2 icons or all 3 icons. Only use icons that are actually visible on this card. There are exactly 3 possible icon types: ICON TYPE 1: CREDIT Visual appearance: orange circle contains a white number. That number has NO plus sign. Example: orange circle with "1" This icon can be small, so look carefully. Meaning: "Get N credit", if N = 1. ICON TYPE 2: DRAW CARDS Visual appearance: black rounded rectangle contains a white number. the number ALWAYS has a plus sign, like "+1" or "+2" This icon can be small, so look carefully. Meaning: "Draw N card from your Resources Pile" if N = 1. "Draw N cards from your Resources Pile" if N 1. ICON TYPE 3: CLOUD ADOPTION EFFECT Visual appearance: small white cloud shape with black outline contains a black number. the number ALWAYS has a plus sign, like "+1" or "+2". This icon can be small, so look carefully. Meaning: "Use N cloud adoption effect" if N = 1. "Use N cloud adoption effects" if N 1. ... rest of the prompt omitted ... During the local and live testing, I came into several issues but all I was able to fix with tuning the prompt. The effects were where prompt was loosing to most. Sometimes it was ignoring the icons, other time it was adding them where they weren't, like for this card: I ran it 11 times and 7 times I got it wrong: Get 1 credit. . Draw 1 card from your Resources Pile. Use 1 cloud adoption effect ---- THIS IS NOT ON THE CARD The card has no cloud adoption effect , but yet it was fabricating it The solution is easier than you think. After I added section IMPORTANT ICON RULES into the prompt, the icon hallucination stopped. IMPORTANT ICON RULES: If a number has NO plus sign and is inside an orange circle, it means credits. If a number has a plus sign and is inside a dark rounded rectangle, it means draw cards. If a number has a plus sign and is inside a white cloud outline, it means cloud adoption effects. A plus sign never means credits. Do not invent missing icons. Do not mention credits unless an orange circle is visible. Do not mention drawing cards unless a dark rounded rectangle is visible. Do not mention cloud adoption effects unless a white cloud icon is visible. Another problem I had was with Credits a number in orange circle . With initial prompt, it was only interpreting it as: Get 1 credit , no matter the number in the circle. The solution was making the prompt into few-shot example prompt, adding examples like: "Get N credit", if N = 1. "Draw N card from your Resources Pile" if N = 1. "Draw N cards from your Resources Pile" if N 1. "Use N cloud adoption effect" if N = 1. "Use N cloud adoption effects" if N 1 I am expecting a model to read the text on the card, and return 3 key:value pairs: But here's where AWS Builder Cards fights back again - some of them have a "subtitle": Without considering that into the prompt, this was the result of the text extraction: | What model extracted | Full card's title | |---|---| AWS certified Solutions Architect | AWS certified Solutions Architect Associate | AWS certified Solutions Architect | AWS certified Solutions Architect Professional | AWS certified Developer | AWS certified Developer Associate | AWS certified Sysops Administrator | AWS certified Sysops Administrator Associate | Look at the first two rows. Those are two physically different cards - a Solutions Architect Associate and a Solutions Architect Professional , but the model returned the same title for both. If I had trusted the drafts without manual approvals , my catalog would hold the same card twice and be missing another one. I had 2 options how to deal with that: Of course I fixed the prompt - the less manual job for me during the approvals, the better EXTRACT PROMPT = """Return ONLY valid JSON. The JSON must have exactly these keys: { "title": "", "effect": "", "description": "" } Read the card from top to bottom. TASK 1: TITLE The title is the card name. It can have two parts. Part 1 - the text in the title bar at the top of the card. Always present. Read it in full, exactly as printed. Part 2 - a qualifier printed in its own banner inside the artwork, below the title bar. Only some cards have this. Examples of what it looks like: ASSOCIATE, PROFESSIONAL, FOUNDATIONAL, SPECIALTY. If a banner like that is visible, the title is Part 1 followed by Part 2, written in normal capitalisation: "AWS certified Solutions Architect" + "ASSOCIATE" - "AWS certified Solutions Architect Associate" If no such banner is visible, the title is Part 1 alone. Do not add a qualifier that is not printed on the card. Examples: "AWS Cloud Practitioner", "David", "AWS certified Solutions Architect Professional". If you cannot read it, use "". ... rest of the prompt omitted ... That's just enough for model to understand when the card has a "subtitle". That may seem like not important, but consider text on the the cards contain bold text, links, sepparate lines, etc... If I want final card page to look like the card itself, I have to follow that. Again, this is something that would take me 10 seconds during the manual approval, but why if I can do it with prompt? Few shots example will do the job If you spot text in bold, write it as for makrdown files - that means like this: this is bold text Any internet link URL you spot, you must write in this format: link link . Having implemented all prompt modifications, now I can say in most cases, this prompt works 100%. Occasionally there some some minimal hickups, but generally it works perfectly. As you can imagine, I did not write this prompt at once. At least 6 versions of it went live, after I was happy with the outputs. Bedrock returns title , description and effect back to image-processing lambda. To create a card's slug markdown file which is performed by lambda review-editor in the next steps LINK TO ARTICLE 5 - TBD , more values are actually needed. Therefore the image-processor lambda actually gathers a lot more values, before sending them to DynamoDB . python def write card item card id, event, year, raw key, fin key, fields, uploader="" : Generate a timestamp for the new record now = datetime.now timezone.utc .isoformat The year is only appended when the title does not already carry it title draft = fields.get "title", "" year str = str year .strip slug src = f"{title draft} {year str}" if year str and year str not in title draft else title draft slug draft = slugify slug src Store the pending card record in DynamoDB ddb.put item TableName=DDB TABLE, Item={ "cardId": {"S": card id}, --- from S3 key the uuid, via parse meta "slug": {"S": ""}, --- empty, for the human "slug ai draft": {"S": slug draft}, --- computed locally from Bedrock's title + year "weight": {"S": ""}, --- empty, for the human "event": {"S": event or ""}, --- from S3 key via parse meta "year": {"S": str year or "" }, --- from S3 key via parse meta "uploader": {"S": uploader or ""}, --- from S3 object metadata typed by the visitor "category": {"S": "collectibles"}, --- hardcoded constant "subcategory": {"S": ""}, --- empty, for the human "title": {"S": ""}, --- empty, for the human "title ai draft": {"S": fields.get "title", "" }, --- from Bedrock "effect": {"S": ""}, --- empty, for the human "effect ai draft": {"S": fields.get "effect", "" }, --- from Bedrock "description": {"S": ""}, --- empty, for the human "description ai draft": {"S": fields.get "description", "" }, --- from Bedrock "rawKey": {"S": raw key}, --- from the S3 event the uploaded object's key "finishedKey": {"S": fin key}, --- computed locally images/finished/