Agentation-iOS

Developer mearnest-dev released Agentation-iOS 0.1, an open-source Swift package that lets developers tap views in an iOS app running in the Simulator, attach notes, and copy structured markdown so an AI coding agent receives the view's identifier, type, frame, text and a cropped screenshot. The package has zero dependencies, runs only in DEBUG builds and by default only in the Simulator (with an allowOnDevice: true option for physical devices), and supports both SwiftUI and UIKit through a .agentation() modifier or Agentation.start(). Annotations can sync to the existing agentation-mcp server so agents read them without copy/paste, and the author states the 0.1 release is early and has not been tried on many apps.

Visual annotation tool for iOS apps running in the Simulator. Inspired by Agentation https://www.agentation.com/ for React and agentation-vanilla https://github.com/mearnest-dev/agentation-vanilla for plain JS. This is the native iOS version. Tap a view, write a note, copy structured markdown. Your AI coding agent gets the view's identifier, type, frame, text and a screenshot, so it can find the code you mean. Status: 0.1, early. It hasn't been tried on many apps yet. Please open an issue when it describes a view badly. One Swift package. Zero dependencies. Debug builds only. In Xcode: File → Add Package Dependencies… and enter: https://github.com/mearnest-dev/agentation-ios Or in Package.swift : .package url: "https://github.com/mearnest-dev/agentation-ios", from: "0.1.0" Link the Agentation library to your app target. SwiftUI python import SwiftUI import Agentation @main struct MyApp: App { var body: some Scene { WindowGroup { ContentView .agentation } } } UIKit python import Agentation func scene scene: UIScene, willConnectTo session: UISceneSession, options: UIScene.ConnectionOptions { // ...your window setup... Agentation.start } A floating button appears in the bottom-right corner. Tap it to open the toolbar. The toolbar only runs in DEBUG builds. In release builds start and all the modifiers do nothing, so you don't need if DEBUG around them. By default it also only runs in the Simulator; pass allowOnDevice: true to use it on a physical device. - Inspect mode : tap any view to see its type, identifier, frame, text, font and colors. Copy the details, or turn them into an annotation. - Annotate mode : tap a view to drop a numbered marker with a note - Area select : drag to annotate a region, even empty space - Intent tags : tag annotations as bug, style, feature, or content - Screenshots : each annotation saves a cropped PNG with the selection outlined. In the Simulator, that file lives on your Mac, so the agent can open it directly. - Copy as markdown : copies to the Simulator pasteboard, which syncs to your Mac clipboard. Paste into Claude Code, Cursor, or any AI tool. - MCP sync : sends annotations to the existing agentation-mcp https://www.npmjs.com/package/agentation-mcp server, so agents can read them without copy/paste - Agent round trip : when your agent resolves or dismisses an annotation through MCP, it disappears from the toolbar - Per-screen persistence : annotations survive relaunches, grouped by screen - Isolated overlay : draws in its own UIWindow , so it never changes your view hierarchy or layout - UIKit and SwiftUI : works with either, or both in the same app On the web, Agentation hands the agent a CSS selector to grep for. On iOS, the best grep targets are: | What it captures | Example | Where it comes from | |---|---|---| | Accessibility identifier | checkout.payButton | .accessibilityIdentifier ... / accessibilityIdentifier | | Agentation tag | PaymentSummaryCard | .agentationTag ... / view.agentationTag optional, see below | | Source location | MyApp/PaymentSummary.swift:42 | .agentationSource optional, see below | | View controller chain | UINavigationController CheckoutViewController | responder chain | | View path | UITableView PaymentCell 2 UIStackView UILabel | view hierarchy | | SwiftUI root view | UIHostingController<CheckoutView | hosting controller's generic type | | Visible text | "Pay $42.00" | label, button and text field contents | | Accessibility label | "Pay" | accessibility tree | Tip: the more views have an accessibilityIdentifier , the more precise the output. Identifiers also help your UI tests. SwiftUI views don't keep their type names at runtime, so the package adds two modifiers. Both do nothing in release builds. PaymentSummary order: order .agentationTag "PaymentSummaryCard" // a name the agent can grep for .agentationSource // records this file and line automatically .agentationSource uses fileID and line default arguments, so you don't type the path yourself. For UIKit views, set the tag as a property: headerView.agentationTag = "ProfileHeader" type: Button identifier: checkout.payButton tag: PaymentSummaryCard source: MyApp/PaymentSummary.swift:42 screen: CheckoutView path: UITabBarController UINavigationController UIHostingController<CheckoutView PaymentSummaryCard Button "Pay $42.00" frame: x=16 y=712 w=361 h=50 pt, window text: "Pay $42.00" nearby: "Total", "$42.00" UIKit views also get a style: line, e.g. font=SFUI-Semibold 17pt; foreground= FFFFFF; background= 0A84FF; cornerRadius=12 . iOS feedback: MyApp com.example.myapp Device: iPhone 16 Pro Simulator, iOS 26.0, 393×852pt @3x, light mode, Dynamic Type L style Annotation 1 Screen: CheckoutView Element: Button Identifier: checkout.payButton Tag: PaymentSummaryCard Source: MyApp/PaymentSummary.swift:42 Path: UITabBarController UINavigationController UIHostingController<CheckoutView PaymentSummaryCard Button "Pay $42.00" View controllers: UITabBarController UINavigationController UIHostingController<CheckoutView Frame: x=16 y=712 w=361 h=50 Text: "Pay $42.00" Nearby text: "Total", "$42.00" Screenshot: /Users/you/Library/Developer/CoreSimulator/Devices/…/tmp/agentation/annotation-3f2a9c1d.png Note: corners should be 12, match the cards above bug Annotation 2 Screen: CheckoutView Area: x=0 y=120 w=393 h=180 Inside: ShippingOptions Path: UITabBarController UINavigationController UIHostingController<CheckoutView ShippingOptions area Text in area: "Shipping", "Standard 3–5 days " Screenshot: /Users/you/Library/Developer/CoreSimulator/Devices/…/tmp/agentation/annotation-9b1e04aa.png Note: this section overlaps the header when Dynamic Type is XXL agentation-ios speaks the same HTTP API as the web toolbar. The Simulator shares your Mac's network, so localhost reaches a server running on the Mac. 1. Register the server with your agent: claude mcp add agentation -- npx -y agentation-mcp server 2. Point the toolbar at it: .agentation endpoint: "http://localhost:4747" // or Agentation.start endpoint: "http://localhost:4747" 3. Allow local networking in your app's Info.plist : <key NSAppTransportSecurity</key <dict <key NSAllowsLocalNetworking</key <true/ </dict 4. Add a test annotation and ask your agent to call agentation get all pending . Annotations are sent with a URL of the form ios-sim://<bundle id /<screen , so the app gets its own session next to any web sessions. The web schema has no fields for some iOS details screen, view class, identifier, tag, screenshot path, intent tag , so those go in the annotation's attributes . Intent tags map to the server's intents: bug → fix , the rest → change . The agent can acknowledge, reply to, and resolve them like any other Agentation annotation. While the toolbar is open, it checks the server every few seconds and removes annotations the agent resolved or dismissed. The dot in the toolbar shows sync status: green when connected, red when the server can't be reached. On a physical device, localhost is the phone itself. Run the server with --host 0.0.0.0 and use your Mac's LAN address instead. Agentation.start endpoint: "http://localhost:4747", // MCP server; nil = local copy only appName: "MyApp", // shown in copied output captureScreenshots: true, toolbarPosition: .bottomTrailing, // starting corner; drag to move enableKeyCommands: true, allowOnDevice: false, onAnnotationAdd: { annotation in }, // structured callback onCopy: { markdown in } Everything is optional. The SwiftUI .agentation ... modifier takes the same arguments. Other calls: Agentation.show opens the toolbar, Agentation.copyAll copies everything, Agentation.annotations returns what's saved, and Agentation.stop removes the toolbar. Matches the Agentation annotation schema https://agentation.com/schema where the fields make sense, with iOS additions: js public struct Annotation: Codable, Identifiable { public let id: String public var comment: String public var intent: Intent? // .bug, .style, .feature, .content public var timestamp: TimeInterval // seconds since 1970 public var screen: String // top view controller / SwiftUI root view public var element: String // "Button", "Text", "Area" public var elementPath: String // condensed view path public var viewClass: String? // "UIButton" UIKit selections public var viewControllers: String // owning view controllers, outermost first public var accessibilityIdentifier: String? public var accessibilityLabel: String? public var agentationTag: String? public var sourceLocation: String? // "MyApp/CheckoutView.swift:42" public var text: String? public var nearbyText: String // nearby text, or text inside an area public var containedTags: String // tags inside an area public var style: String? // font, colors, corner radius UIKit public var boundingBox: Rect? // window points public var x: Double // marker, % of window width public var y: Double // marker, pt from top of window public var isArea: Bool public var screenshotPath: String? public var url: String // ios-sim://<bundle id /<screen public var serverId: String? // set after syncing to agentation-mcp } | Action | How | |---|---| | Open or close the toolbar | Tap the floating button, or shake Device → Shake / ⌃⌘Z in the Simulator | | Inspect mode | Scope button, or I | | Annotate mode | Speech-bubble button, or A . Tap a view, or drag to select an area | | Edit or delete an annotation | Tap its numbered marker | | Copy all as markdown | Copy button, or C | | Clear annotations | Trash button this screen or all screens | | Leave mode / close | Esc , or tap the active button again | | Move the toolbar | Drag it | Keyboard shortcuts work on a hardware keyboard your Mac keyboard in the Simulator while the toolbar is open. They stop while one of your app's text fields has focus. For the Mac clipboard to receive copies, keep Edit → Automatically Sync Pasteboard turned on in the Simulator. While a mode is active, taps go to the overlay instead of your app. Turn the mode off to use the app normally. The toolbar never blocks touches outside its own bounds. - Overlay window. A UIWindow above your app's windows draws the toolbar, highlights and markers. It passes touches through unless a mode is active. This is the iOS version of agentation-vanilla's Shadow DOM isolation. - Hit testing. A tap is resolved by finding the front-most visible view under it in your app's window, including non-interactive views like labels. Inside SwiftUI hosting views, the accessibility tree is used to narrow it down, because SwiftUI draws many views without a matching UIView . - Tags. .agentationTag and .agentationSource place an invisible view behind the modified view, so a tap point can be matched to the innermost tag around it. - Context capture. The package walks up the view hierarchy and responder chain to build the path, screen name and view controller chain. It reads text, fonts, colors and layer properties from the views it finds. - Screenshots. The selected area plus some margin is rendered with UIGraphicsImageRenderer and saved to the app's tmp/agentation/ folder. In the Simulator, that folder is a normal path on your Mac. - Persistence. Annotations are stored as JSON in Application Support/Agentation/annotations.json , grouped by screen, and expire after 7 days. - Sync. With an endpoint , annotations are sent to agentation-mcp with URLSession using the same session and annotation routes as the web toolbar. Failed sends are retried while the toolbar is open. - Shake. UIWindow.motionEnded is swizzled once to detect shakes. This happens when Agentation starts, so never in release builds. - Simulator first. It works on a device, but the MCP endpoint and screenshot paths are less convenient there. - Markers don't follow scrolling. A marker stays where you tapped. The copied output is unaffected. - Screen names are a best guess. They come from the top view controller. In SwiftUI, pushed NavigationStack screens may show the navigation title instead of a type name. - SwiftUI is best effort. SwiftUI doesn't expose view types at runtime. Without identifiers, tags or .agentationSource , output falls back to hosting-controller types, accessibility info and visible text. Style details are only captured for UIKit views. - No content inside Metal, SpriteKit, SceneKit or video layers. You can annotate the containing view or an area. - Web views. For content inside a WKWebView , inject agentation-vanilla https://github.com/mearnest-dev/agentation-vanilla into the page instead. - Debug only. The package refuses to start in release builds, and the SwiftUI modifiers do nothing there. The code is still linked into release builds, just never run. - iOS 16+ - Xcode 15+ Swift 5.9+ - A Mac, because the iOS Simulator only runs on macOS - agentation-mcp optional needs Node.js 20+ on your Mac agentation-ios/ ├── Package.swift ├── Sources/Agentation/ │ ├── Agentation.swift // start , stop , Configuration │ ├── SwiftUI/ // .agentation , .agentationTag , .agentationSource │ ├── Overlay/ // overlay window, toolbar, markers, editor, shake │ ├── Capture/ // hit testing, view path, accessibility, styles, tags │ ├── Model/ // Annotation, type-name parsing │ ├── Output/ // markdown formatter, screenshot writer │ ├── Storage/ // JSON persistence │ └── Sync/ // agentation-mcp HTTP client ├── Tests/AgentationTests/ └── Demo/ // SwiftUI + UIKit demo app and UI tests XcodeGen The demo has a SwiftUI checkout screen and a UIKit settings screen. git clone https://github.com/mearnest-dev/agentation-ios.git cd agentation-ios/Demo brew install xcodegen once xcodegen open AgentationDemo.xcodeproj Run the AgentationDemo scheme on any iPhone Simulator. To try MCP sync, run npx -y agentation-mcp server first. The model, formatting, storage and sync payload tests don't need UIKit, so they run on macOS: swift test Or run them on a Simulator: xcodebuild test -scheme Agentation -destination 'platform=iOS Simulator,name=iPhone 16' The demo app has UI tests that drive the toolbar in the Simulator: inspect, annotate, area select, edit, delete, clear, copy, and persistence across relaunch. Start from a fresh install so no old annotations are left: cd Demo && xcodegen xcrun simctl uninstall booted dev.mearnest.AgentationDemo xcodebuild test -project AgentationDemo.xcodeproj -scheme AgentationDemo -destination 'platform=iOS Simulator,name=iPhone 17 Pro' SyncFlowTests also checks the round trip with agentation-mcp create, edit, agent resolves, toolbar drops it . It is skipped unless npx -y agentation-mcp server is running. The idea and the annotation format come from Agentation https://agentation.com by Benji Taylor. This project is an independent implementation and is not affiliated with Agentation. It talks to the published agentation-mcp server; it does not include its code. MIT