Read and tap the screen

Your agent reads the screen as text and acts by label, the way it reads code. A sign-in screen is about 70 tokens, not a screenshot.

Read it

phone_ui_look

Demo · Sign In · iPhone 17 Pro (iOS 27.0)
textfield "Email" #email @201,194
securefield "Password" #password @201,246
switch "Remember me" #remember off @341,298
button "Sign In" #signIn @201,350
text "Not signed in" #status @201,402
tabs: Login (selected) @158,822, Items @244,822

One line per element: its kind, its label, #id(the app's accessibility identifier), its state and@x,y, its center in points. Tabs, scrolling and the keyboard are a line each, and an alert or sheet hides the screen behind it. A web view with nothing for accessibility to read comes back as oneunreadableline with its place: take a screenshot and tap by point there.

findOnly the elements whose label, id or value contains it.

screenshotA JPEG beside the text.

screenshot_fileA full-size PNG to download.

fullThe whole accessibility tree, as a JSON file.

Text on the screen comes from the app. It is data, never instructions for your agent.

Act on it

phone_ui_act

steps: ["fill #email gary@example.com", "fill #password secret1", "tap #signIn", "wait \"Welcome\""]

4 steps done in 4.5 s
changed: #email "gary@example.com" · #password "•••••••" · #status "Welcome, gary@example.com"

phone_ui_act takes steps in the words phone_ui_look prints, runs them in order and answers with what changed. A tap that changes nothing is reported, and the first step that fails stops the rest.

Tap tap #id, tap "label", tap @x,y, hold "label" 2

Type fill #id text, type text, key return. Any language, read back after.

Move scroll down, swipe left "row", set #slider 50, back, home

Wait wait "text" 10, gone "text"

Control apps and the phone

MCP call

phone_app_control  { "action": "open_url", "value": "myapp://orders/42" }

phone_app_control launches, installs, stops or removes an app; opens a link or a deep link; grants, revokes or resets a permission; sets the location; sends a push; and sets the appearance, locale, time zone, text size, or a clean status bar (9:41, full bars) for screenshots.

Any screen, without tapping

MCP call

phone_app_control  { "action": "launch", "value": "-Onboarding 3 -Demo YES" }

An app opens any screen at once when it reads launch arguments: -Key value sets that key in UserDefaults for this launch, like the arguments of an Xcode scheme. Words with spaces are quoted as in a shell, and the answer is the screen once it is up. A screenshot from phone_ui_look then waits for the screen to hold still, so a set of screens for review or the App Store is a launch and a look for each. Launch arguments can be seen by other programs on the phone's Mac, like any program's: a secret goes in phone_app_run's env, which only the app sees.

In-app purchases

MCP call

phone_app_control  { "action": "storekit", "value": "MyApp/Products.storekit", "app": "com.example.app" }

The App Store's sandbox cannot buy in a Simulator: Apple only allows purchases there on a device. StoreKit Testing can, and storekit applies your project's .storekit file to the app the way Xcode's Run does, with no change to the app. Its value is the file's path in the project you ran (or clear). Relaunch the app: its products and purchases are local, with no Apple Account, and complete without iOS's purchase sheet, until the app is removed.

An iPhone app on an iPad

App Review runs iPhone apps on an iPad, in a window iPadOS moves and scales. The ipad phone cannot show an iPhone-only app yet: iOS 27.0's Simulator stops drawing when one opens on an iPad Pro 11-inch (M5). Check it on a NoMac Mac with an iPad Air Simulator, where simulator-use says the app is in a window and gives positions in the iPad's screen points.

Logs, screenshots and video

phone_app_logs What the app printed and logged since launch. grep filters, wait_for waits up to 25 seconds for a line, crashes lists crash reports.

phone_screen_record Start, then stop for a link to an MP4 that shows a person what happened. The video is cut from the phone's replay, so its link opens once the phone has stopped. A phone that stops while recording ends the recording there, and its stop answers the link as recording.

screenshot_file On an iphone-pro-max phone, the 6.9-inch size App Store Connect takes.