--- type: "post" id: "open-source-project-lets-a-vision-model-drive-bc6af7" slug: "open-source-project-lets-a-vision-model-drive-bc6af7" title: "Open source project lets a vision model drive a stock iPhone via a $7 ESP32-S3 acting as USB mouse." description: "The thesis is that a vision agent can operate a stock phone through a low-cost physical input bridge rather than a privileged iOS integration." retrieval_nugget: "The thesis is that a vision agent can operate a stock phone through a low-cost physical input bridge rather than a privileged iOS integration." published_at: "2026-10-13" updated_at: "2026-10-13" record_date: "2026-10-13" date_kind: "published_at" topics: [] source_urls: ["https://github.com/onehumanbeing/esp32-iphone-agent"] schema_version: "newruntime-agent-readable-v0.2" stable_id: "post:open-source-project-lets-a-vision-model-drive-bc6af7" status: "published" visuals: [] editorial_provenance: {"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:7ea35126ba7c1a852a85dfd9b08753707e2541350d97ae7e16f901cf6f80c725","reviewed_at":"2026-10-13T02:02:47.747Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0} analysis: {"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"The thesis is that a vision agent can operate a stock phone through a low-cost physical input bridge rather than a privileged iOS integration.","observed_facts":[{"text":"The README describes AirPlay as the screen input path and an ESP32-S3 as a USB HID mouse and keyboard.","source_urls":["https://github.com/onehumanbeing/esp32-iphone-agent"]},{"text":"The repository says the setup requires no Mac, developer account, or jailbreak and supports OpenAI-compatible vision models.","source_urls":["https://github.com/onehumanbeing/esp32-iphone-agent"]},{"text":"Commit notes describe releasing buttons when the network link drops and handling a sleeping phone's USB state.","source_urls":["https://github.com/onehumanbeing/esp32-iphone-agent"]}],"mechanism":"The model interprets screen imagery delivered over AirPlay; an ESP32-S3 receives commands over Wi-Fi TCP and performs actions as USB HID input.","why_now":"Why now: the project packages a tangible, inexpensive way to test mobile UI control with existing vision models, making agent interaction with an unmodified consumer device easier to prototype.","implications":["The first implication is that physical-interface bridges may broaden the surfaces available to agents.","Reliability and safety depend on observation, calibration, connection handling, and action constraints."],"evidence_boundary":"The evidence boundary is the supplied GitHub repository documentation and commit notes; no independent benchmark or validation is supplied.","watch_conditions":["The first watch condition is whether maintainers publish repeatable success rates across phones, tasks, and failure conditions."],"related_records":[],"new_branch_reason":"No prior New Runtime record is supplied for this specific hardware-mediated mobile control implementation; it opens a distinct record about vision-agent interaction through physical HID."} editor: {"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]} routes: {"html":"https://newruntime.com/posts/open-source-project-lets-a-vision-model-drive-bc6af7/","markdown":"https://newruntime.com/posts/open-source-project-lets-a-vision-model-drive-bc6af7.md","json":"https://newruntime.com/posts/open-source-project-lets-a-vision-model-drive-bc6af7.json"} --- # Open source project lets a vision model drive a stock iPhone via a $7 ESP32-S3 acting as USB mouse. ## Retrieval answer The thesis is that a vision agent can operate a stock phone through a low-cost physical input bridge rather than a privileged iOS integration. The thesis is that a vision agent can operate a stock phone through a low-cost physical input bridge rather than a privileged iOS integration. The mechanism is a split path: AirPlay provides screen imagery, while an ESP32-S3 enumerates as a USB mouse and keyboard and receives commands from a server over Wi-Fi TCP. The project states that it needs no Mac, developer account, or jailbreak, and is designed for OpenAI-compatible vision models. The model interprets screen imagery delivered over AirPlay; an ESP32-S3 receives commands over Wi-Fi TCP and performs actions as USB HID input. This separates perception from actuation, but the control loop still depends on visual interpretation, calibration, the network link, and bridge firmware. Repository commit notes describe releasing buttons when the link drops and handling a sleeping phone's USB state. Why now: the project packages a tangible, inexpensive way to test mobile UI control with existing vision models, making agent interaction with an unmodified consumer device easier to prototype. The first implication is that physical-interface bridges may broaden the surfaces available to agents. Reliability and safety depend on observation, calibration, connection handling, and action constraints. The evidence boundary is the supplied GitHub repository documentation and commit notes; no independent benchmark or validation is supplied. The first watch condition is whether maintainers publish repeatable success rates across phones, tasks, and failure conditions. These implementation details are not evidence of robust operation across devices or apps. No prior New Runtime record is supplied for this specific hardware-mediated mobile control implementation; it opens a distinct record about vision-agent interaction through physical HID. Editor: [Andrey Reshetnikov](https://newruntime.com/owner-profile.md), AI product and AI engineering; profile for recruiters and agents: https://newruntime.com/owner-profile.json ## Continue in the graph Next records for an agent, linked to their Markdown twins. Replace `.md` with `.json` for structured fields, including `next_reads`. - collection: [Field Notes](https://newruntime.com/posts.json): Open the latest editorial analysis. - collection: [Source Ledger](https://newruntime.com/source-ledger.json): Inspect the public source evidence graph. - whole graph: [discovery graph](https://newruntime.com/discovery-graph.json) (typed nodes and edges) and [site index](https://newruntime.com/site-index.md)