Agentic Remote Control UX and IxD Patterns
Purpose
Many are still catching up to the usefulness and power of remote controlling agents. For those out of the loop 😉, agentic remote control means controlling agents remotely. It's about spawning and driving agentic work while untethered from a laptop or desktop machine. That's it.
Though the canonical example of a remote controller is a smartphone, it's worth mentioning you can go further upstream. I use the Index 01 ring as an agentic remote controller for example. That said, this article will focus on the visual UX and IxD patterns of smartphone interfaces.
The patterns below are ones that I've landed on over the past year of first-hand experience using Ora—my personal agent fleet management tool. She's a daily driver so I've had a lot of reps (both at home and work). Sampling from Ora's Manifests it looks like personal use with Anthropic models alone hits over 1K threads and 6K prompts. 1K+ of those prompts came from my smartphone. Reps.
Ora manifest sample
Though others have independently developed and applied some of the patterns, the below set is informed from my own design and engineering background in combo with daily Ora use.
Hopefully more mainstream agentic remote control products will address their shortcomings and implement a lot of the below low-hanging fruit improvements.
At last check, the underleveraged patterns are Collapsible Composer, Composer Snap, Docked Reference, Selection Expansion, Selection Actions, and Action Chaining.
Users win by gaining more control with less effort (see MIMO below). I bet existing product teams could even point to this article and one-shot them all or fill in their gaps.
Back-story
Myself and other early adopters in 2025 loved Claude Code but wanted an agentic engineering UX that was driven remotely. At the core, we wanted to talk to our repos without being anchored to a laptop or desktop machine.
As designers and engineers many of us created our own agent fleet management tools to accomplish this. Peter Steinberger built Clawdbot Moltbot OpenClaw to interact with agents through existing apps like WhatsApp and Telegram. Others just wanted raw Claude Code via a TUI on their phone (god bless these folks). I on the other hand had a ton of ideas for a first class rich UI—Ora was born.
A few—Ora included—even beat Anthropic's Claude Code and OpenAI's Codex remote control market offerings. Though it's not lost on me that they had a much bigger problem to solve (large user base), the fact stands. Many other products have since landed their remote control offerings throughout 2026.
In any case, feel free to jump to the patterns or continue for a UX and IxD terminology refresher that includes some insight into my IxD decision process.
Refresher
To start, here are some terms:
- UX - User Experience
- IxD - Interaction Design
- Control - An interactive UI element
- MIMO - Minimum Input Maximum Output
In the context of IxD we directly influence UX.
User experience is subjective and unique to each user and it can improve or degrade during each user session. A designer most directly influences it via their design decisions, but it's less recognized that all contributors to a product influence it (yes, even infrastructure, backend, security, marketing, etc.).
Interaction design is a subcategory of hardware and software product design, but we'll be focusing on the software side. IxD decisions are about graphics, controls, triggers, timing, and their interwoven application to aid or guide user task flows. Put simply, IxD is where designers think through a specific small task flow and make microdecisions that optimize a user's experience through said task flow. That's it.
Microdecisions
Concretely, these microdecisions involve:
- static graphics (attention and feedback)
- motion graphics (attention and feedback)
- controls (button, toggle switch, slider, etc.)
- triggers (click, tap, gesture, voice, etc.)
- timing (progressive disclosure and UX "feel")
These microdecisions typically result in microinteractions the user takes. For more on triggers, feedback, and doing more with less, see my notes on Dan Saffer's Microinteractions - Designing with Details.
Combined these microdecisions are intended to nudge a user toward the designer's intent—most often a control. The designed intent followed by user interaction is what enables this invisible dance, the interactive conversation between a user—or users—and the interface.
I use the personal acronym MIMO to guide my decision making. This is a shorthand that encapsulates other UX and IxD patterns for guiding my design decisions (topic for another article). A tactical shortcut process you can employ is:
- Go through a task flow
- Count how many trigger steps you took (click, tap, gesture, voice, etc.)
- Ask yourself, "can I delete this or replace it in favor of..."?
- Repeat
Through this process you can get to MIMO where a user provides minimum input (low effort) but attains maximum output (high outcome).
You can keep this framing in the back of your mind when exploring the patterns below. All of them were filtered through the MIMO lense in combo with the below concepts:
- visually constrained real-estate
- touch-based input and "thumb first" thinking
- virtual keyboard (and voice) input
A few additional notes:
- The desktop experience often benefits from or can extend the patterns
- The interaction trigger can be customized per pattern implementation (often with an accompanying keyboard shortcut on laptop and desktop)
- Some patterns often have emergent relationships enabling UX combos
Patterns
Nerd-out below.
- Collapsible Composer
- Composer Snap
- Slash Menu
- Docked Reference
- Quick Zoom
- Anchor Jump
- Selection Expansion
- Selection Actions
- Action Chaining
- Decision Chips
- Dictation
- Handedness Toggle
This is a non-exhaustive list.
Collapsible Composer
This is a common toggle IxD pattern applied specifically to a composer UI's screen real-estate.
Collapsible Composer
The user can't read, scan, or interact with content the composer UI obstructs. This user pain is exacerbated when a virtual keyboard is expanded.
Content is obstructed by the composer UI
Provide a thumb-accessible toggle switch control for contracting and expanding the composer UI
Add a single tap listener to a thumb-accessible toggle switch control that toggles between two states:
- Tap to contract (optionally use transition animation)
- Read, scan, or interact with previously obstructed content
- Tap to expand (optionally use transition animation)
Composer Snap
This is a common toggle IxD pattern applied specifically to a composer UI's height.
Composer Snap
The user's input is expansive enough to obstruct content. This user pain is exacerbated when a virtual keyboard is expanded.
The user has expansive prompt input and wants to quickly view obstructed content
Provide a thumb-accessible toggle switch control for contracting and expanding the composer UI's input height
Add a double tap listener to a thumb-accessible toggle switch control that toggles between two states:
- Tap to contract (optionally use transition animation)
- Read, scan, or interact with previously obstructed content
- Tap to expand back to previous height value (optionally use transition animation)
Slash Menu
This is a common command menu IxD pattern applied specifically to a composer UI. A thumb-accessible / button control makes commands available without typing. This and the Action Chaining pattern are one of the aforementioned emergent UX combos.
Slash Menu
The user can't quickly access commands without remembering and typing their names. This user pain is exacerbated when a virtual keyboard is required just to repeat a recently used command.
The composer supports reusable commands and the user wants to discover or repeat them with minimal input
Provide a thumb-accessible / button control that opens a command popover menu relative to the composer
Add a single tap listener to a thumb-accessible / button control in the composer that toggles a recent commands popover menu. Additionally support typing / in the composer to find commands:
- Tap the
/control to view recent commands without opening the virtual keyboard, or type/followed by part of a command name to filter available commands - Tap a command to insert it into the composer and close the menu
- Add any desired prompt context before submitting
- Allow dismissal without selecting a command
Ora on desktop expands via:
ctrl+/: toggle recent commandsup|down: navigate commandsenter|tab: insert the selected commandesc: dismiss the menu
Docked Reference
This is a useful variation of the common modal and lightbox UI patterns where a portion of content is desired to be unobstructed.
Docked Reference
The user can't remember the specific part of a reference (an image, document, code, etc.) they want to prompt about and it is hidden from view while prompting
A user wants to prompt about a specific part of a reference without having to open it, refresh their memory, close it, then resume prompting with refreshed memory
Design a UI that supports viewing references while keeping the composer unobstructed
Add a pane, modal, popover, or similar UI component that allows referencing without composer obstruction
- Tap to dock reference
- Don't obstruct the composer
- Compose while reference in view
Ora on desktop expands via:
ctrl+e: open docked reference if closed whereleft|rightnavigate prev|nextesc: close docked reference
Quick Zoom
This is a standard touch IxD pattern that toggles (or cycles) between different zoom levels. It is often accompanied with panning support. This and the Docked Reference pattern are one of the aforementioned emergent UX combos.
Quick Zoom
The user can't quickly and easily focus on a reference
The user wants a closer look at a particular reference
Provide a thumb-accessible hit area for touch zoom support
Add a double tap listener to a reference that toggles zoom levels. Optionally add a drag listener for panning the zoomed reference.
- Double tap to zoom in
- Touch drag to pan
- Double tap to zoom out
Ora on desktop expands via:
ctrl+up|down: zoom in|outmousewheel: zoom in|outclick+drag: pan
Anchor Jump
This expands upon the common "jump to" anchor navigation IxD pattern but instead supports a dynamic cycle instead of a one-shot.
Anchor Jump
The user can't quickly reference content not currently in view. This user pain is exacerbated when the content is far away (typically vertically) from their current viewing location.
Desired content can have static or dynamically generated anchors
Provide a thumb-accessible button control for cycling anchors
Add a single tap listener to a thumb-accessible button control that cycles the anchors. Add a long-press to conditionally anchor jump (canonically top ↔ bottom of a page)
- Tap to cycle to the next anchor
- Read, scan, or interact with previously out-of-frame content
Ora on desktop expands via:
ctrl+click+up|down: anchor jump to prev|next assistant bubblealt+click+up|down: anchor jump to prev|next assistant bubble and/or its headings
Selection Expansion
This is an uncommon text selection pattern optimized for applications that can intuit a user's next step following text-selection. MacOS and Windows support triple click to expand to a word's enclosing paragraph as an example application.
Selection Expansion
A user can't select a range of text easily. Only the cumbersome and slow native mobile text-selection tools are available.
The user wants to quickly select a text range with minimal effort
Provide a thumb-accessible alternative to native text-selection
Add a double tap listener to a thumb-accessible text container that intelligently expands the range from the target word to its enclosing sentence, statement, line, or paragraph
- Double tap a word
- Text-selection automatically expands the word selection to a sentence, statement, line, or paragraph selection
Ora on desktop expands via:
ctrl+click: lighter IxD alternative to desktop triple click (this is an example of MIMO via two actions vs. three)
Selection Actions
This is a fairly common variant of the Contextual Actions pattern whose prerequisite is a text-selection. This and the Selection Expansion pattern are one of the aforementioned emergent UX combos.
Selection Actions
The user can't quickly execute their known next action after a text-selection
The user has common next step use cases after a text-selection
Provide a thumb-accessible control set of quick actions
Add a selectionchange listener to a text container and conditionally position a contextual quick actions menu
- Text-selection (native or Selection Expansion)
- Calculate and conditionally display a contextual actions menu relative to the selected text
- Auto-close the contextual menu when the user taps an action or taps outside
Action Chaining
Action Chaining generalizes the Execute and Exit pattern I created, which you can learn about in the Interaction Design Pattern - Execute and Exit article. This and the Slash Menu pattern are one of the aforementioned emergent UX combos.
Action Chaining
There are a few scenarios where it's useful in Ora:
- Recent commands inline button control (double tap) - append the command to the input, automatically submit the prompt, and then navigate home (shown in the video above).
- Inline document commands button control (single tap) - same as Recent commands above, but triggered via a button control within a thread or document.
- Prompt submission button control (double tap) - execute the prompt and then return (or stay) home.
The user can't quickly execute an action and its known follow-up action(s) without multiple steps
The user commonly repeats the same follow-up action(s) after a primary action
Provide a thumb-accessible button control that executes a primary action and its follow-up action(s) in one gesture
Add a double tap listener to a thumb-accessible button control that chains two or more actions:
- Execute the primary action (ex. submit the prompt)
- Execute the follow-up action(s) (ex. return home)
Ora on desktop expands via:
ctrl+enter: prompt submission and then return (or stay) home
Decision Chips
The canonical example are one-click UI responses to tool calls like AskUserQuestionTool. These can be customized for other one-click action scenarios.
Dictation
Voice vs. typing—you get it. This is also where products like the aforementioned Index 01 come in.
Handedness Toggle
Not implemented in Ora but useful for applications where left-handed and right-handed users have opposite thumb-accessible needs.
Conclusion
As you can see these are all quite simple and straightforward patterns to implement. Additionally, many of them are useful outside agentic remote control scenarios.
If you are a designer, engineer, or builder working on remote control features I'd love to hear what you adopt or ignore (and why). If you're not working on remote control features but still learned a thing or two then I'd love to hear what resonated.
You can tweet me @derekknox.

