Skip to main content
A session is always the same four beats: acquire a lease, observe, act under that lease, release. This page walks one session end to end and shows what each call returns. Every example is a tool call. Your client decides how you phrase it; the parameters and the results below are what travels on the wire.
1

See what is reachable

list_devices needs no lease and takes no parameters. Call it first, every time.
Each entry carries an id, a status, and a capabilities block:
Every payload on this page is illustrative. Ids, timestamps, screen geometry and freshness windows come from your own devices, so read them out of the response rather than copying them from here.
Read capabilities before you plan anything. A device with uiTree: false has no element tree to query, and a device whose pressKey list omits a key refuses that key. See Capabilities.If you already know the id, get_device answers the same shape for one device without enumerating the fleet.
2

Take a lease

A device holds one lease at a time, across every org. While you hold one, no other agent can acquire the device, and every action and observation you send must carry the leaseId it returned. A signed-in person in your org may still act on the device from the console beside you; when they do, your results say so in others.
Keep leaseId. Read expiresAt rather than assuming a duration: a lease expires on its own, so a client that crashes cannot strand a device. You never pass fencingToken anywhere; the control plane uses it to make a resumed or duplicated caller lose to the current lease holder.Acquiring a device that is already leased fails. That is the point of a lease, and it is why repeating acquire_device is not idempotent.
3

Observe

The result carries the image itself plus the metadata you need next:
pixelSize is informational. Do not aim with it. Aim on the normalized grid, where both axes run 0 to 1000 whatever the real screen size is.Keep frameId. It is how the next step proves it was decided on this picture of the screen.On a device with a UI tree, ui_snapshot gives you the same screen as structured elements, and every bound inside that tree is already on the 0 to 1000 grid, so a coordinate you read there can be tapped directly.
4

Act

Two optional parameters are worth passing on every action.observedFrame binds the action to the frame you decided on. If the screen has moved on since that frame, the tap is refused with stale_frame rather than landing somewhere you did not intend. Observe again and decide again.A frame does not age out on our side, and a newer screenshot does not replace it. Take as long as you need between observing and acting, including time spent reading a large ui_snapshot or waiting for a person to approve the step, and observe again in between if that helps.What the frame is for is the others report that comes back on every action: how many actions another principal took on this device since you looked, when the newest was, and whether the hands were a person’s or an agent’s. With a frame named, an agent’s action is refused rather than taken on a screen somebody else has touched; pass refuseIfOthersActed: false to act anyway and decide from others.A device whose own resource expires frames can still answer expired; a non-null freshnessMs in its capabilities is how you know to expect it. Nothing is lost when a frame is refused for any reason: the action never began and your idempotencyKey was not consumed, so re-observe and act under the same key.idempotencyKey is your own request id. Reuse the same value when retrying the same intended action, and the recorded outcome is replayed instead of the gesture happening twice.The result tells you the device applied it:
committed means the device confirmed the action, not that the request was accepted. On the physical tier the actuator has finished moving by the time you read this.Because you passed an idempotencyKey, this outcome is also on the ledger: get_receipt with the same key returns it later, which is how you confirm an action whose response you never received. See Reading a trace.The other action tools take the same shape: swipe, long_press, type_text, press_key, and open_app. Every one of them is destructive by design, because a gesture on a real device cannot be undone.
5

Release

leaseId is required here, unlike on the action tools where it is optional: giving a device back is about one particular lease, so the id has to be the one you were given.Releasing a device you no longer hold is a no op, so a release in a finally block is safe.

​
What happens if you skip a step

Read Reading a trace next for how to confirm an action whose response you never received.