Project Log #15: My AI Agent Just Completed Its First Multi-App Task

A developer building an AI phone agent achieved a milestone: the agent completed its first multi-app task by copying an account balance from a banking app and sending it via WhatsApp. The key enabler was a task memory system using a persistent key-value store that bridges data between apps. Challenges remain with app loading times, OCR accuracy, and screen state management.

Day 15. The agent switched between two apps, copied data from one, and pasted it into another. For 14 days, my AI agent could do one thing well: operate within a single app. Open WhatsApp. Find a contact. Send a message. All within one ecosystem. Today, that changed. The Milestone I gave the agent a command: "Copy my account balance from my banking app and send it to Mom on WhatsApp." It opened the banking app. It read the balance from the screen using OCR since banking apps have zero accessibility labels—F score, remember? . It copied the number. It switched to WhatsApp. It found Mom. It typed the balance. It hit send. Two apps. One task. Completed autonomously. What Made This Possible The key was building a task memory system. The agent now stores values extracted from one app in a simple key-value store a Python dictionary that persists for the duration of the task . When it switches to the next app, it retrieves that value and uses it. The flow works like this: account balance . account balance from memory. Type it. Send.The memory system is simple—just a dictionary that gets passed between steps. But it's the bridge between single-app automation and true multi-app workflows. Today's Progress | Task | Status | |---|---| | Built task memory system key-value store | ✅ Done | | Tested "Copy balance → send to Mom" | ✅ Success | | Tested "Copy address → paste into Maps" | ✅ Success | Updated agent.py with multi-app support | ✅ Done | | Added task memory to README | ✅ Done | What Still Breaks The banking app is slow to load on my phone. Sometimes the agent times out waiting for the balance screen to appear. I added a retry loop, but it's not perfect. App switching is clunky. The agent uses adb shell am start to launch apps, but if the app was already open in the background, it sometimes opens to the wrong screen. I'm working on a "reset to home screen" step between app switches. And the OCR on banking app text is unreliable. The font is small. The contrast is low. The agent sometimes misreads the balance by a digit or two. For a task like "send my balance to Mom," that's embarrassing but not catastrophic. For a task like "transfer money," that's dangerous. I need higher accuracy before I can trust this with anything financial. The Bigger Picture Multi-app workflows are the difference between a cool prototype and a useful tool. An agent that can only operate in one app is a novelty. An agent that can move data between apps is an assistant. The next step is chaining more complex workflows: "Find my last three transactions, summarise them, and email them to my accountant." That's three apps. That's the goal. What's Next Day 16 The Repo 👉 github.com/Dexter2344/phone-agent agent.py now includes a task memory system. Multi-app workflows are supported. README updated with Day 15 status. This is Day 15. The agent is no longer trapped in one app.