A valid request URL is required to generate request examples{
"deviceId": "<string>",
"leaseId": "<string>",
"coDriving": true,
"status": "committed",
"committedAt": "<string>",
"others": {
"actions": 123,
"lastAt": "2023-11-07T05:31:56Z",
"by": [
"person"
]
},
"idempotencyKey": "<string>",
"deduplicated": true,
"frameNotes": {
"ageMs": 1,
"newerFrames": 1,
"edgeMovedSince": true,
"settled": true
}
}{
"error": {
"code": "backend_resolution_failed",
"message": "<string>",
"retryable": true
}
}{
"error": {
"code": "unauthorized",
"reason": "missing_credentials",
"hint": "<string>"
}
}{
"error": {
"code": "backend_resolution_failed",
"message": "<string>",
"retryable": true
}
}{
"error": {
"code": "backend_resolution_failed",
"message": "<string>",
"retryable": true
}
}{
"error": {
"code": "backend_resolution_failed",
"message": "<string>",
"retryable": true
}
}{
"error": {
"code": "payload_too_large",
"message": "<string>"
}
}{
"error": {
"code": "rate_limited",
"message": "<string>"
}
}{
"error": {
"code": "backend_resolution_failed",
"message": "<string>",
"retryable": true
}
}Act on a device
One gesture per request: tap, swipe, long press, type text, press a key, open an app. Coordinates are in the normalized 0-1000 grid, so nothing has to know the device’s resolution.
WHOSE LEASE. Any caller may leave leaseId out. The gesture then runs under the lease your organisation currently holds on the device, whoever inside it took that lease (an agent’s run, a colleague, you), and the response says so in coDriving. When your organisation holds none, a lease is taken for you and the gesture runs under it; if that gesture then fails, the lease taken for it is given back before the error is answered, and your next gesture takes one again. Two people taking an idle device at the same moment both drive it: one took the lease, the other is co-driving under it. When another organisation holds the device the answer is the 409 the acquire route gives. Taking needs devices:lease: a key with only devices:act acts on a device its organisation already holds. A signed-in person’s gesture slides a lease THEY hold when it is running low; it never extends anybody else’s.
DRIVING BESIDE AN AGENT. A person’s tap does not interrupt a run, and does not wait behind it either: gestures on one device run one at a time, a signed-in person’s go ahead of every agent gesture still waiting, and the rest keep their arrival order. The lease is checked again when a gesture’s turn comes (one that ended while it waited answers lease_required), each one is receipted with who took it, and every result carries others: what principals other than you did since your anchor. The agent reads the same field on its own results, so it is informed rather than stopped. The only interruptions are explicit and yours: POST /v1/runs/{runId}/pause and POST /v1/runs/{runId}/cancel.
The response resolves only once the DEVICE applied the gesture. status has one value, and an action that did not commit is an error rather than a different status.
Every attempt is on the receipt ledger before it is dispatched, so GET /v1/receipts can answer what happened even when this response is lost. Send your own idempotencyKey and a retry replays that outcome instead of touching the phone twice.
A device may refuse a gesture it cannot perform; read its capabilities first. A robot arm, for instance, can tap and swipe and cannot type.
Hosted deployments only. A local checkout does not mount this route, so calling it there is a 404.
A valid request URL is required to generate request examples{
"deviceId": "<string>",
"leaseId": "<string>",
"coDriving": true,
"status": "committed",
"committedAt": "<string>",
"others": {
"actions": 123,
"lastAt": "2023-11-07T05:31:56Z",
"by": [
"person"
]
},
"idempotencyKey": "<string>",
"deduplicated": true,
"frameNotes": {
"ageMs": 1,
"newerFrames": 1,
"edgeMovedSince": true,
"settled": true
}
}{
"error": {
"code": "backend_resolution_failed",
"message": "<string>",
"retryable": true
}
}{
"error": {
"code": "unauthorized",
"reason": "missing_credentials",
"hint": "<string>"
}
}{
"error": {
"code": "backend_resolution_failed",
"message": "<string>",
"retryable": true
}
}{
"error": {
"code": "backend_resolution_failed",
"message": "<string>",
"retryable": true
}
}{
"error": {
"code": "backend_resolution_failed",
"message": "<string>",
"retryable": true
}
}{
"error": {
"code": "payload_too_large",
"message": "<string>"
}
}{
"error": {
"code": "rate_limited",
"message": "<string>"
}
}{
"error": {
"code": "backend_resolution_failed",
"message": "<string>",
"retryable": true
}
}Authorizations
The control surface credential. Send Authorization: Bearer <token>.
Two kinds of token are accepted and they are told apart by shape, not by a separate header. A token beginning pbk_ is an org scoped API key, whose public half and secret half are generated together and of which only a hash of the secret is ever stored; anything else is treated as an OAuth 2.1 access token and verified against the authorization server's keys.
Both resolve to the same context: an org, a principal and a set of scopes. Nothing downstream branches on which channel you used, with one deliberate exception, key management, which requires a signed-in person so that a key can never mint another key.
Scopes are enforced when MCP tools are REGISTERED rather than when they are called, so a tool your credential cannot use is absent from tools/list rather than refused mid gesture.
Path Parameters
The device to act on.
Body
One gesture.
- tap
- swipe
- long_press
- type_text
- press_key
- open_app
One gesture, discriminated by action. Coordinates are in the NORMALIZED 0-1000 grid with the origin at the top left of the screen in its current orientation, never in screenshot pixels: the same grid the tools use, so a coordinate read off a UI snapshot can be tapped directly and nothing has to know the device's resolution.
leaseId is OPTIONAL for every caller: the gesture runs under the lease your organisation currently holds on the device, whoever inside it took that lease, and when it holds none one is taken for you, provided your credential may take devices (devices:lease). A leaseId you send is only an address: your organisation holding the device is what authorizes the gesture. Watching a screen needs no lease; moving it still does, and the response tells you which one it ran under.
refuseIfOthersActed refuses the gesture if anybody else acted on the device after observedFrame was taken. It is on by default for an API key that sends observedFrame, off for a person, and off without a frame; send it explicitly to choose.
"tap"0 <= x <= 10000 <= x <= 1000Optional, for every caller. The gesture runs under the lease your organisation currently holds on this device, or has one taken for it when it holds none (a credential without devices:lease is refused lease_required instead of taking). A lease id you send is an address, not a credential: your organisation holding the device is what authorizes the gesture, and sending one while your organisation holds nothing is refused lease_required rather than turned into a new take.
Refuse this gesture (409 stale_frame, reason others_acted) if another principal acted on the device after observedFrame was taken. Sending true needs observedFrame; without it the body is refused as invalid_argument. Left out, it defaults by who is acting: on for an API key, a copilot or a run that sends observedFrame, off for a person acting directly, and off whenever no observedFrame is sent. You are told through others either way.
Your own id for this one action: 1 to 128 printable ASCII characters, and it must not start with "agent:", which is reserved for keys this service issues itself. Send the SAME value when retrying the SAME intended gesture and the recorded outcome is replayed instead of the phone being touched twice. Omit it and the service generates one, which cannot deduplicate your retries.
The frame your decision came from. Optional here, unlike on the worker surface: a person driving a live view is looking at the screen as they touch it. When you do send it, the action is refused if that frame is no longer the device's current one, so re-observe and decide again.
Response
The device applied it.
The device the gesture was applied to.
The lease the gesture ran under: the one you named, your organisation's current one, or the one taken for you. Keep it if you want to release a lease this call took for you; nothing else needs it.
True when the lease belongs to somebody else inside your organisation, an agent's run most often: you acted beside them, and they were informed through their own others. False when it is yours.
The device applied it. There is no other success.
"committed"ISO-8601 instant the device confirmed it, on this service's clock.
What everybody else did on this device since you last looked. On a gesture the anchor is the frame you bound it to, else your previous gesture that reached the device (a refused one never anchors), else the last sixty seconds. A person may drive the same device beside an agent's run; each is informed of the other through this and neither is stopped.
Show child attributes
Show child attributes
The key this action was deduplicated under, whether you sent it or the service generated it.
True when this outcome was replayed from the ledger and the device was NOT touched again. Absent means the gesture really happened just now.
What the robot arm noticed about the frame this action was bound to. The arm no longer refuses an old or overtaken frame: it performs the gesture and reports here, and you decide whether to take a new screenshot before the next one. Present only on a robot-driven phone whose controller reports it, and only on an action that was bound to a frame; any field may be missing.
Show child attributes
Show child attributes