Optimize warm CLI control path

Author: DjDeveloperr

README.md

@@ -58,6 +58,16 @@ The browser bootstrap comes from `GET /api/health`, which returns the WebTranspo
 certificate hash, and packet version needed by the client.
 The served browser UI receives the generated API access token automatically; direct HTTP callers can use the startup token with `X-SimDeck-Token` or `Authorization: Bearer`.
 
+For fastest agent control, keep `simdeck serve` or `simdeck service on` running and route hot CLI controls through the warm local service:
+
+```sh
+export SIMDECK_SERVER_URL=http://127.0.0.1:4310
+simdeck tap <udid> 0.5 0.5 --normalized
+simdeck describe-ui <udid> --format agent --max-depth 2
+```
+
+You can also pass `--server-url http://127.0.0.1:4310` on individual commands. Supported fast-path controls include launch/open-url, normalized touch/tap/swipe/gesture input, key/key-sequence/key-combo, hardware buttons, dismiss-keyboard, home/app-switcher, rotate, and appearance toggles.
+
 ## Service
 
 Enable the per-user background service with `launchd`:
@@ -139,7 +149,8 @@ UIKit in-app inspectors, then falls back to the built-in private CoreSimulator
 accessibility bridge. Use `--format agent` or `--format compact-json` for
 lower-token hierarchy dumps. Coordinate commands accept screen coordinates from
 the accessibility tree by default; pass `--normalized` to send `0.0..1.0`
-coordinates directly.
+coordinates directly. With `--server-url` or `SIMDECK_SERVER_URL`, normalized
+input commands use the warm service path to avoid repeated native setup.
 
 ## NativeScript Inspector
 

docs/api/rest.md

@@ -175,6 +175,26 @@ Content-Type: application/json
 
 Allowed `phase` values: `began`, `moved`, `ended`, `cancelled`.
 
+### `POST /api/simulators/{udid}/touch-sequence`
+
+Replays multiple normalized touch events through one native input session:
+
+```http
+POST /api/simulators/{udid}/touch-sequence
+Content-Type: application/json
+
+{
+  "events": [
+    { "x": 0.5, "y": 0.7, "phase": "began", "delayMsAfter": 25 },
+    { "x": 0.5, "y": 0.4, "phase": "moved", "delayMsAfter": 25 },
+    { "x": 0.5, "y": 0.2, "phase": "ended" }
+  ]
+}
+```
+
+This is the preferred API for agent gestures because it avoids one HTTP request
+per touch phase.
+
 ### `POST /api/simulators/{udid}/key`
 
 Replays a single keyboard event by HID key code:
@@ -188,6 +208,33 @@ Content-Type: application/json
 
 `keyCode` is the HID usage value. `modifiers` is a bitmask defined by the HID input subsystem (defaults to `0`).
 
+### `POST /api/simulators/{udid}/key-sequence`
+
+Replays multiple HID key codes through one native input session:
+
+```http
+POST /api/simulators/{udid}/key-sequence
+Content-Type: application/json
+
+{ "keyCodes": [11, 8, 15, 15, 18], "delayMs": 5 }
+```
+
+`delayMs` defaults to `0`.
+
+### `POST /api/simulators/{udid}/button`
+
+Presses a hardware button:
+
+```http
+POST /api/simulators/{udid}/button
+Content-Type: application/json
+
+{ "button": "lock", "durationMs": 50 }
+```
+
+Supported button names match the CLI: `home`, `lock`, `side-button`, `siri`,
+and `apple-pay`. `durationMs` defaults to `0`.
+
 ### `POST /api/simulators/{udid}/home`
 
 Presses the home button:

docs/cli/commands.md

@@ -118,6 +118,24 @@ Use `--format agent` for compact hierarchy text intended for LLM planning, and
 `--point` returns the native element at a screen point and uses the native point
 query directly.
 
+## Warm service fast path
+
+Most agent loops should keep `simdeck serve` or `simdeck service on` running and
+set:
+
+```sh
+export SIMDECK_SERVER_URL=http://127.0.0.1:4310
+```
+
+Supported hot controls then use the local HTTP service and avoid repeated native
+setup in short-lived CLI processes. This fast path covers `launch`, `open-url`,
+normalized `touch`, normalized coordinate `tap`, normalized `swipe`, normalized
+`gesture`, `key`, `key-sequence`, `key-combo`, `dismiss-keyboard`, `home`,
+`app-switcher`, `button`, `rotate-left`, `rotate-right`, and
+`toggle-appearance`. Commands that need local files, selector-to-point
+resolution, screenshots, pasteboard, or batch execution stay on the direct
+native path.
+
 ## `boot`
 
 Boot a simulator by UDID:

docs/cli/flags.md

@@ -75,6 +75,17 @@ Encoder used by the native bridge. See [Video Pipeline](/guide/video) for when t
 
 HTTP API and WebTransport access token. The served browser UI receives it automatically through a strict same-site cookie, so normal local use does not require copying the token. Direct API callers should send either `X-SimDeck-Token: <token>` or `Authorization: Bearer <token>`.
 
+## Global CLI flags
+
+### `--server-url <url>`
+
+| Default | unset                          |
+| ------- | ------------------------------ |
+| Env     | `SIMDECK_SERVER_URL`           |
+| Type    | `http://` URL for local server |
+
+When set, supported hot controls delegate to the warm local SimDeck service instead of starting a fresh native control path in the CLI process. This is fastest for agent-driven loops. Supported delegated controls include `launch`, `open-url`, normalized `touch`, normalized coordinate `tap`, normalized `swipe`, normalized `gesture`, `key`, `key-sequence`, `key-combo`, `button`, `dismiss-keyboard`, `home`, `app-switcher`, `rotate-left`, `rotate-right`, and `toggle-appearance`.
+
 ## Positional arguments
 
 Subcommands that take positionals expect them in the order shown:
@@ -88,15 +99,15 @@ Subcommands that take positionals expect them in the order shown:
 
 ## `describe-ui` flags
 
-| Flag                 | Default                         | Description                                                                      |
-| -------------------- | ------------------------------- | -------------------------------------------------------------------------------- |
-| `--format`           | `json`                          | Output format: `json`, `compact-json`, or `agent`.                               |
-| `--source`           | `auto`                          | Hierarchy source: `auto`, `nativescript`, `uikit`, or `native-ax`.               |
-| `--max-depth`        | unlimited native / `80` service | Trim descendants after the requested depth.                                      |
-| `--include-hidden`   | `false`                         | Include hidden in-app inspector views when supported.                            |
-| `--direct`           | `false`                         | Skip the local service and use the private native accessibility bridge directly. |
-| `--point <x>,<y>`    | unset                           | Return the native element at a screen point.                                     |
-| `--server-url <url>` | `http://127.0.0.1:4310`         | Local service URL used for source-aware hierarchy requests.                      |
+| Flag                 | Default                                | Description                                                                      |
+| -------------------- | -------------------------------------- | -------------------------------------------------------------------------------- |
+| `--format`           | `json`                                 | Output format: `json`, `compact-json`, or `agent`.                               |
+| `--source`           | `auto`                                 | Hierarchy source: `auto`, `nativescript`, `uikit`, or `native-ax`.               |
+| `--max-depth`        | unlimited native / `80` service        | Trim descendants after the requested depth.                                      |
+| `--include-hidden`   | `false`                                | Include hidden in-app inspector views when supported.                            |
+| `--direct`           | `false`                                | Skip the local service and use the private native accessibility bridge directly. |
+| `--point <x>,<y>`    | unset                                  | Return the native element at a screen point.                                     |
+| `--server-url <url>` | global flag or `http://127.0.0.1:4310` | Local service URL used for source-aware hierarchy requests.                      |
 
 ## Exit codes
 

docs/cli/index.md

@@ -5,9 +5,11 @@ The `simdeck` binary is the only entrypoint SimDeck ships. It hosts the HTTP ser
 ## Synopsis
 
 ```sh
-simdeck <COMMAND> [OPTIONS]
+simdeck [--server-url <url>] <COMMAND> [OPTIONS]
 ```
 
+Set `SIMDECK_SERVER_URL=http://127.0.0.1:4310` or pass `--server-url` to route supported hot controls through an already-running local service. That avoids repeated native setup for agent loops while preserving the same JSON command output.
+
 ## Top-level commands
 
 | Command                    | Purpose                                                                                     |

scripts/integration/cli.mjs

@@ -943,7 +943,7 @@ async function retrySimdeckText(args, label, options = {}) {
 
 async function cliStep(label, args, commandOptions = {}, verifyOptions = {}) {
   return measuredStep(label, async () => {
-    const result = await retrySimdeckJson(args, label, {
+    const result = await retrySimdeckJson(cliArgs(args), label, {
       maxElapsedMs: cliCommandBudgetMs,
       ...commandOptions,
     });
@@ -952,6 +952,10 @@ async function cliStep(label, args, commandOptions = {}, verifyOptions = {}) {
   });
 }
 
+function cliArgs(args) {
+  return serverProcess ? ["--server-url", serverUrl, ...args] : args;
+}
+
 async function httpStep(
   label,
   method,

server/src/api/routes.rs

@@ -57,13 +57,42 @@ struct TouchPayload {
     phase: String,
 }
 
+#[derive(Deserialize)]
+struct TouchSequencePayload {
+    events: Vec<TouchSequenceEvent>,
+}
+
+#[derive(Deserialize)]
+struct TouchSequenceEvent {
+    x: f64,
+    y: f64,
+    phase: String,
+    #[serde(rename = "delayMsAfter")]
+    delay_ms_after: Option<u64>,
+}
+
 #[derive(Deserialize)]
 struct KeyPayload {
     #[serde(rename = "keyCode")]
     key_code: u16,
     modifiers: Option<u32>,
 }
 
+#[derive(Deserialize)]
+struct KeySequencePayload {
+    #[serde(rename = "keyCodes")]
+    key_codes: Vec<u16>,
+    #[serde(rename = "delayMs")]
+    delay_ms: Option<u64>,
+}
+
+#[derive(Deserialize)]
+struct ButtonPayload {
+    button: String,
+    #[serde(rename = "durationMs")]
+    duration_ms: Option<u32>,
+}
+
 #[derive(Deserialize)]
 struct AccessibilityPointQuery {
     x: f64,
@@ -142,11 +171,20 @@ pub fn router(state: AppState) -> Router {
         .route("/api/simulators/{udid}/open-url", post(open_url))
         .route("/api/simulators/{udid}/launch", post(launch_bundle))
         .route("/api/simulators/{udid}/touch", post(send_touch))
+        .route(
+            "/api/simulators/{udid}/touch-sequence",
+            post(send_touch_sequence),
+        )
         .route("/api/simulators/{udid}/key", post(send_key))
+        .route(
+            "/api/simulators/{udid}/key-sequence",
+            post(send_key_sequence),
+        )
         .route(
             "/api/simulators/{udid}/dismiss-keyboard",
             post(dismiss_keyboard),
         )
+        .route("/api/simulators/{udid}/button", post(press_button))
         .route("/api/simulators/{udid}/home", post(press_home))
         .route(
             "/api/simulators/{udid}/app-switcher",
@@ -420,6 +458,46 @@ async fn send_touch(
     Ok(json(json_value!({ "ok": true })))
 }
 
+async fn send_touch_sequence(
+    State(state): State<AppState>,
+    Path(udid): Path<String>,
+    Json(payload): Json<TouchSequencePayload>,
+) -> Result<Json<Value>, AppError> {
+    if payload.events.is_empty() {
+        return Err(AppError::bad_request(
+            "Request body must include at least one touch event.",
+        ));
+    }
+    if payload.events.len() > 64 {
+        return Err(AppError::bad_request(
+            "Touch sequence cannot contain more than 64 events.",
+        ));
+    }
+    for event in &payload.events {
+        if !event.x.is_finite() || !event.y.is_finite() {
+            return Err(AppError::bad_request(
+                "`x` and `y` must be finite normalized numbers.",
+            ));
+        }
+    }
+    run_bridge_action(state, move |bridge| {
+        let input = bridge.create_input_session(&udid)?;
+        for event in payload.events {
+            input.send_touch(
+                event.x.clamp(0.0, 1.0),
+                event.y.clamp(0.0, 1.0),
+                &event.phase,
+            )?;
+            if let Some(delay_ms) = event.delay_ms_after.filter(|delay_ms| *delay_ms > 0) {
+                std::thread::sleep(Duration::from_millis(delay_ms));
+            }
+        }
+        Ok(())
+    })
+    .await?;
+    Ok(json(json_value!({ "ok": true })))
+}
+
 async fn send_key(
     State(state): State<AppState>,
     Path(udid): Path<String>,
@@ -432,6 +510,37 @@ async fn send_key(
     Ok(json(json_value!({ "ok": true })))
 }
 
+async fn send_key_sequence(
+    State(state): State<AppState>,
+    Path(udid): Path<String>,
+    Json(payload): Json<KeySequencePayload>,
+) -> Result<Json<Value>, AppError> {
+    if payload.key_codes.is_empty() {
+        return Err(AppError::bad_request(
+            "Request body must include at least one key code.",
+        ));
+    }
+    if payload.key_codes.len() > 512 {
+        return Err(AppError::bad_request(
+            "Key sequence cannot contain more than 512 key codes.",
+        ));
+    }
+    run_bridge_action(state, move |bridge| {
+        let input = bridge.create_input_session(&udid)?;
+        let delay_ms = payload.delay_ms.unwrap_or(0);
+        let key_count = payload.key_codes.len();
+        for (index, key_code) in payload.key_codes.into_iter().enumerate() {
+            input.send_key(key_code, 0)?;
+            if delay_ms > 0 && index + 1 < key_count {
+                std::thread::sleep(Duration::from_millis(delay_ms));
+            }
+        }
+        Ok(())
+    })
+    .await?;
+    Ok(json(json_value!({ "ok": true })))
+}
+
 async fn dismiss_keyboard(
     State(state): State<AppState>,
     Path(udid): Path<String>,
@@ -440,6 +549,21 @@ async fn dismiss_keyboard(
     Ok(json(json_value!({ "ok": true })))
 }
 
+async fn press_button(
+    State(state): State<AppState>,
+    Path(udid): Path<String>,
+    Json(payload): Json<ButtonPayload>,
+) -> Result<Json<Value>, AppError> {
+    if payload.button.trim().is_empty() {
+        return Err(AppError::bad_request("Request body must include `button`."));
+    }
+    run_bridge_action(state, move |bridge| {
+        bridge.press_button(&udid, &payload.button, payload.duration_ms.unwrap_or(0))
+    })
+    .await?;
+    Ok(json(json_value!({ "ok": true })))
+}
+
 async fn press_home(
     State(state): State<AppState>,
     Path(udid): Path<String>,