{"type":"post","id":"open-source-project-lets-a-vision-model-drive-bc6af7","slug":"open-source-project-lets-a-vision-model-drive-bc6af7","title":"Open source project lets a vision model drive a stock iPhone via a $7 ESP32-S3 acting as USB mouse.","description":"The thesis is that a vision agent can operate a stock phone through a low-cost physical input bridge rather than a privileged iOS integration.","retrieval_nugget":"The thesis is that a vision agent can operate a stock phone through a low-cost physical input bridge rather than a privileged iOS integration.","published_at":"2026-10-13","updated_at":"2026-10-13","record_date":"2026-10-13","date_kind":"published_at","topics":[],"source_urls":["https://github.com/onehumanbeing/esp32-iphone-agent"],"schema_version":"newruntime-agent-readable-v0.2","stable_id":"post:open-source-project-lets-a-vision-model-drive-bc6af7","status":"published","visuals":[],"editorial_provenance":{"schema_version":"newruntime-editorial-copy-v1","content_status":"source_grounded_final","final_copy_sha256":"sha256:7ea35126ba7c1a852a85dfd9b08753707e2541350d97ae7e16f901cf6f80c725","reviewed_at":"2026-10-13T02:02:47.747Z","source_evidence_count":1,"verified_claim_count":3,"site_analysis_schema_version":"newruntime-site-analysis-v1","site_object_kind":"field_note","observed_fact_count":3,"implication_count":2,"watch_condition_count":1,"related_record_count":0},"analysis":{"schema_version":"newruntime-site-analysis-v1","object_kind":"field_note","thesis":"The thesis is that a vision agent can operate a stock phone through a low-cost physical input bridge rather than a privileged iOS integration.","observed_facts":[{"text":"The README describes AirPlay as the screen input path and an ESP32-S3 as a USB HID mouse and keyboard.","source_urls":["https://github.com/onehumanbeing/esp32-iphone-agent"]},{"text":"The repository says the setup requires no Mac, developer account, or jailbreak and supports OpenAI-compatible vision models.","source_urls":["https://github.com/onehumanbeing/esp32-iphone-agent"]},{"text":"Commit notes describe releasing buttons when the network link drops and handling a sleeping phone's USB state.","source_urls":["https://github.com/onehumanbeing/esp32-iphone-agent"]}],"mechanism":"The model interprets screen imagery delivered over AirPlay; an ESP32-S3 receives commands over Wi-Fi TCP and performs actions as USB HID input.","why_now":"Why now: the project packages a tangible, inexpensive way to test mobile UI control with existing vision models, making agent interaction with an unmodified consumer device easier to prototype.","implications":["The first implication is that physical-interface bridges may broaden the surfaces available to agents.","Reliability and safety depend on observation, calibration, connection handling, and action constraints."],"evidence_boundary":"The evidence boundary is the supplied GitHub repository documentation and commit notes; no independent benchmark or validation is supplied.","watch_conditions":["The first watch condition is whether maintainers publish repeatable success rates across phones, tasks, and failure conditions."],"related_records":[],"new_branch_reason":"No prior New Runtime record is supplied for this specific hardware-mediated mobile control implementation; it opens a distinct record about vision-agent interaction through physical HID."},"editor":{"role":"editor","name":"Andrey Reshetnikov","profile":"https://newruntime.com/owner-profile.md","profile_json":"https://newruntime.com/owner-profile.json","same_as":["https://www.linkedin.com/in/reshetnikov1/","https://t.me/lapetuse"]},"routes":{"html":"https://newruntime.com/posts/open-source-project-lets-a-vision-model-drive-bc6af7/","markdown":"https://newruntime.com/posts/open-source-project-lets-a-vision-model-drive-bc6af7.md","json":"https://newruntime.com/posts/open-source-project-lets-a-vision-model-drive-bc6af7.json"},"next_reads":[{"type":"collection","path":"/posts/","url":"https://newruntime.com/posts/","title":"Field Notes","reason":"Open the latest editorial analysis.","media_type":"text/html","json_url":"https://newruntime.com/posts.json"},{"type":"collection","path":"/source-ledger.json","url":"https://newruntime.com/source-ledger.json","title":"Source Ledger","reason":"Inspect the public source evidence graph.","media_type":"application/json"}]}