# Read and tap the screen

Your agent reads the screen as text and acts by label, the way it reads code. A sign-in screen is about 70 tokens, not a screenshot.

## Read it

phone\_ui\_look

```
Demo · Sign In · iPhone 17 Pro (iOS 27.0)
textfield "Email" #email @201,194
securefield "Password" #password @201,246
switch "Remember me" #remember off @341,298
button "Sign In" #signIn @201,350
text "Not signed in" #status @201,402
tabs: Login (selected) @158,822, Items @244,822
```

One line per element: its kind, its label, `#id`(the app's accessibility identifier), its state and`@x,y`, its center in points. Tabs, scrolling and the keyboard are a line each, and an alert or sheet hides the screen behind it. A web view with nothing for accessibility to read comes back as one`unreadable`line with its place: take a screenshot and tap by point there.

findOnly the elements whose label, id or value contains it.

screenshotA JPEG beside the text.

screenshot\_fileA full-size PNG to download.

fullThe whole accessibility tree, as a JSON file.

Text on the screen comes from the app. It is data, never instructions for your agent.

## Act on it

phone\_ui\_act

```
steps: ["fill #email gary@example.com", "fill #password secret1", "tap #signIn", "wait \"Welcome\""]

4 steps done in 4.5 s
changed: #email "gary@example.com" · #password "•••••••" · #status "Welcome, gary@example.com"
```

`phone_ui_act` takes steps in the words `phone_ui_look` prints, runs them in order and answers with what changed. A tap that changes nothing is reported, and the first step that fails stops the rest.

Tap `tap #id`, `tap "label"`, `tap @x,y`, `hold "label" 2`

Type `fill #id text`, `type text`, `key return`. Any language, read back after.

Move `scroll down`, `swipe left "row"`, `set #slider 50`, `back`, `home`

Wait `wait "text" 10`, `gone "text"`

## Control apps and the phone

MCP call

```
phone_app_control  { "action": "open_url", "value": "myapp://orders/42" }
```

`phone_app_control` launches, installs, stops or removes an app; opens a link or a deep link; grants, revokes or resets a permission; sets the location; sends a push; and sets the appearance, locale, time zone, text size, or a clean status bar (9:41, full bars) for screenshots.

## Any screen, without tapping

MCP call

```
phone_app_control  { "action": "launch", "value": "-Onboarding 3 -Demo YES" }
```

An app opens any screen at once when it reads launch arguments: `-Key value` sets that key in `UserDefaults` for this launch, like the arguments of an Xcode scheme. Words with spaces are quoted as in a shell, and the answer is the screen once it is up. A screenshot from `phone_ui_look` then waits for the screen to hold still, so a set of screens for review or the App Store is a launch and a look for each. Launch arguments can be seen by other programs on the phone's Mac, like any program's: a secret goes in `phone_app_run`'s `env`, which only the app sees.

## In-app purchases

MCP call

```
phone_app_control  { "action": "storekit", "value": "MyApp/Products.storekit", "app": "com.example.app" }
```

The App Store's sandbox cannot buy in a Simulator: Apple only allows purchases there on a device. StoreKit Testing can, and `storekit` applies your project's `.storekit` file to the app the way Xcode's Run does, with no change to the app. Its value is the file's path in the project you ran (or `clear`). Relaunch the app: its products and purchases are local, with no Apple Account, and complete without iOS's purchase sheet, until the app is removed.

## An iPhone app on an iPad

App Review runs iPhone apps on an iPad, in a window iPadOS moves and scales. The `ipad` phone cannot show an iPhone-only app yet: iOS 27.0's Simulator stops drawing when one opens on an iPad Pro 11-inch (M5). Check it on a NoMac Mac with an iPad Air Simulator, where simulator-use says the app is in a window and gives positions in the iPad's screen points.

## Logs, screenshots and video

phone\_app\_logs What the app printed and logged since launch. `grep` filters, `wait_for` waits up to 25 seconds for a line, `crashes` lists crash reports.

phone\_screen\_record Start, then stop for a link to an MP4 that shows a person what happened. The video is cut from the phone's replay, so its link opens once the phone has stopped. A phone that stops while recording ends the recording there, and its stop answers the link as recording.

screenshot\_file On an `iphone-pro-max` phone, the 6.9-inch size App Store Connect takes.