[GCP in Action] LINE Business Card Bot Evolution: Dual-Side Recognition and Merging with Gemini

LINE business card bot now recognizes front and back sides of dual-sided cards and merges them into a single record using Gemini. The bot asks users if there is a back side, then sends both images in one Gemini API call for semantic merging, avoiding manual rule-based integration.

Business cards in Taiwan often have a common design: Chinese printed on the front and English on the back or vice versa . Our LINE business card bot's original logic was very simple: receive an image, perform OCR once, and save one record. This is where the problem lies. When a user sends the front side, the bot saves a record with only the Chinese name. If the user then sends the back side, the bot treats it as "another new business card," resulting in two records for the same person in the database, each missing half the information. Users have to manually compare and delete duplicate data, which is a terrible experience. This article records how we taught the bot to recognize that "these are two sides of the same business card" and merge the information from both sides into a single complete record. Instead of writing rules to guess if two images are the same business card, we chose a more direct approach: Ask the user . The workflow is designed as follows: We manage this waiting state using the user states memory dictionary already present in the project, and add a 5-minute timeout. If a user clicks "Yes, there's a back side" but then ignores it or does something else, the process is treated as abandoned after 5 minutes, preventing the entire workflow from getting stuck. user states user id = { 'action': 'pending backside confirm', 'card obj': card obj, 'front image bytes': image content, 'expires at': time.time + PENDING BACKSIDE TIMEOUT SECONDS } The most critical technical decision was: should we recognize the front and back sides separately and then write code to merge them? We chose another path: wrapping both the front and back images in a single generate content request and letting Gemini handle the judgment directly. The reason is simple: merging Chinese and English names into a format like "Wang Daming David Wang" using string rules is prone to being messy and inaccurate. Semantic-level integration is more likely to fail with hardcoded rules, so letting Gemini handle it directly is much easier. In app/gemini utils.py , we added generate json from two images , which reuses the existing NAMECARD SCHEMA structured output, but this time the contents includes two image Parts: python def generate json from two images front img: PIL.Image.Image, back img: PIL.Image.Image, prompt: str - object: model = GenerativeModel "gemini-3-flash-preview", generation config={ "response mime type": "application/json", "response schema": NAMECARD SCHEMA }, front part = Part.from data data=pil to bytes front img , mime type="image/jpeg" back part = Part.from data data=pil to bytes back img , mime type="image/jpeg" response = model.generate content prompt, front part, back part , stream=False, labels={"client id": "namecard"} return response The prompt also just adds a merge instruction after the original IMGAGE PROMPT : DOUBLE SIDED IMAGE PROMPT = IMGAGE PROMPT + """ These two images are the front and back of the same business card; please integrate them into a single complete record. If both Chinese and English appear in the same field such as name or company , please present them merged e.g., "Wang Daming David Wang" ; if a field appears on only one side, use the value from that side; ignore obviously redundant information. """ One API call means we don't have to write or maintain any merging rules ourselves. During the overall code review before the feature went live, we caught two details that are easily overlooked but can really cause issues. Originally, the duplicate check comparing if the email already exists was done immediately after OCR. However, after the double-sided recognition went live, if the front side happened to have a duplicate email from an old record, the process would prematurely determine it "already exists" and end. This would mean any new email on the back side would never be seen. The fix was to move the duplicate check later, performing it only after "single-side chosen not to merge" or "double-sided merge complete." This ensures we are always comparing the final version of the data: php async def finalize and save card card obj, event, user id - None: existing card id = firebase utils.check if card exists card obj, user id if existing card id: ... Reply already exists return card id = firebase utils.add namecard card obj, user id ... Reply save successful Both the single-sided and double-sided merge workflows eventually converge to call this shared function, ensuring the duplicate check is executed only once when the data is finalized. The user states dictionary is actually shared by several features: editing memos, modifying fields, and this back-side recognition. The initial implementation, for convenience, would delete the entire state whenever a residual state was detected before processing a new event. The problem is: if a user is "editing the phone field" and waiting to input a new number, but accidentally sends an image, this logic would clear the editing field state as well, silently canceling the user's original editing operation. The fix was to only clear the two states related to the back-side recognition process and leave other states untouched: if state.get 'action' in 'pending backside confirm', 'awaiting backside image' : del user states user id We later found a missing branch in the same logic; the part handling "operation expired" replies also cleared everything initially. It was only truly fixed after unifying it with the same selective judgment. The awaiting backside image state stores not just text, but also the raw byte data of the front image. If a user disappears after being asked "Is there a back side?", this data would theoretically stay in the process memory because the original design only checked and cleared timeout states during the "user's next interaction." We added a sweep expired states function, which runs immediately when a Webhook comes in. It clears all expired temporary states for all users, so we don't have to wait for the specific user to return for passive cleanup: php def sweep expired states - None: now = time.time expired user ids = user id for user id, state in user states.items if 'expires at' in state and state 'expires at' <= now for user id in expired user ids: del user states user id Whenever any user sends a message, it performs garbage collection for all users, ensuring that those who abandoned the process don't leave behind memory-consuming remnants. This double-sided recognition and merging feature makes the LINE business card bot much more aligned with the actual usage habits of Taiwanese users: The complete code has been pushed to GitHub https://github.com/kkdai/linebot-namecard-python , feel free to check it out