# [GCP in Action] LINE Business Card Bot Evolution: Dual-Side Recognition and Merging with Gemini > Source: > Published: 2026-07-23 13:24:50+00:00 Business cards in Taiwan often have a common design: Chinese printed on the front and English on the back (or vice versa). Our LINE business card bot's original logic was very simple: receive an image, perform OCR once, and save one record. This is where the problem lies. When a user sends the front side, the bot saves a record with only the Chinese name. If the user then sends the back side, the bot treats it as "another new business card," resulting in two records for the same person in the database, each missing half the information. Users have to manually compare and delete duplicate data, which is a terrible experience. This article records how we taught the bot to recognize that "these are two sides of the same business card" and merge the information from both sides into a single complete record. Instead of writing rules to guess if two images are the same business card, we chose a more direct approach: **Ask the user**. The workflow is designed as follows: We manage this waiting state using the `user_states` memory dictionary already present in the project, and add a 5-minute timeout. If a user clicks "Yes, there's a back side" but then ignores it or does something else, the process is treated as abandoned after 5 minutes, preventing the entire workflow from getting stuck. ``` user_states[user_id] = { 'action': 'pending_backside_confirm', 'card_obj': card_obj, 'front_image_bytes': image_content, 'expires_at': time.time() + PENDING_BACKSIDE_TIMEOUT_SECONDS } ``` The most critical technical decision was: should we recognize the front and back sides separately and then write code to merge them? We chose another path: wrapping both the front and back images in a single `generate_content` request and letting Gemini handle the judgment directly. The reason is simple: merging Chinese and English names into a format like "Wang Daming David Wang" using string rules is prone to being messy and inaccurate. Semantic-level integration is more likely to fail with hardcoded rules, so letting Gemini handle it directly is much easier. In [app/gemini_utils.py], we added `generate_json_from_two_images` , which reuses the existing `NAMECARD_SCHEMA` structured output, but this time the `contents` includes two image Parts: ``` python def generate_json_from_two_images( front_img: PIL.Image.Image, back_img: PIL.Image.Image, prompt: str) -> object: model = GenerativeModel( "gemini-3-flash-preview", generation_config={ "response_mime_type": "application/json", "response_schema": NAMECARD_SCHEMA }, ) front_part = Part.from_data( data=pil_to_bytes(front_img), mime_type="image/jpeg") back_part = Part.from_data( data=pil_to_bytes(back_img), mime_type="image/jpeg") response = model.generate_content( [prompt, front_part, back_part], stream=False, labels={"client_id": "namecard"} ) return response ``` The prompt also just adds a merge instruction after the original `IMGAGE_PROMPT` : ``` DOUBLE_SIDED_IMAGE_PROMPT = IMGAGE_PROMPT + """ These two images are the front and back of the same business card; please integrate them into a single complete record. If both Chinese and English appear in the same field (such as name or company), please present them merged (e.g., "Wang Daming David Wang"); if a field appears on only one side, use the value from that side; ignore obviously redundant information. """ ``` One API call means we don't have to write or maintain any merging rules ourselves. During the overall code review before the feature went live, we caught two details that are easily overlooked but can really cause issues. Originally, the duplicate check (comparing if the email already exists) was done immediately after OCR. However, after the double-sided recognition went live, if the front side happened to have a duplicate email from an old record, the process would prematurely determine it "already exists" and end. This would mean any new email on the back side would never be seen. The fix was to move the duplicate check later, performing it only after "single-side chosen not to merge" or "double-sided merge complete." This ensures we are always comparing the final version of the data: ``` php async def _finalize_and_save_card(card_obj, event, user_id) -> None: existing_card_id = firebase_utils.check_if_card_exists(card_obj, user_id) if existing_card_id: # ... Reply already exists return card_id = firebase_utils.add_namecard(card_obj, user_id) # ... Reply save successful ``` Both the single-sided and double-sided merge workflows eventually converge to call this shared function, ensuring the duplicate check is executed only once when the data is finalized. The `user_states` dictionary is actually shared by several features: editing memos, modifying fields, and this back-side recognition. The initial implementation, for convenience, would delete the entire state whenever a residual state was detected before processing a new event. The problem is: if a user is "editing the phone field" and waiting to input a new number, but accidentally sends an image, this logic would clear the `editing_field` state as well, silently canceling the user's original editing operation. The fix was to only clear the two states related to the back-side recognition process and leave other states untouched: ``` if state.get('action') in ( 'pending_backside_confirm', 'awaiting_backside_image' ): del user_states[user_id] ``` We later found a missing branch in the same logic; the part handling "operation expired" replies also cleared everything initially. It was only truly fixed after unifying it with the same selective judgment. The `awaiting_backside_image` state stores not just text, but also the raw byte data of the front image. If a user disappears after being asked "Is there a back side?", this data would theoretically stay in the process memory because the original design only checked and cleared timeout states during the "user's next interaction." We added a `sweep_expired_states()` function, which runs immediately when a Webhook comes in. It clears all expired temporary states for all users, so we don't have to wait for the specific user to return for passive cleanup: ``` php def sweep_expired_states() -> None: now = time.time() expired_user_ids = [ user_id for user_id, state in user_states.items() if 'expires_at' in state and state['expires_at'] <= now ] for user_id in expired_user_ids: del user_states[user_id] ``` Whenever any user sends a message, it performs garbage collection for all users, ensuring that those who abandoned the process don't leave behind memory-consuming remnants. This double-sided recognition and merging feature makes the LINE business card bot much more aligned with the actual usage habits of Taiwanese users: The complete code has been pushed to [GitHub](https://github.com/kkdai/linebot-namecard-python), feel free to check it out!