Category
Visual Design
Interaction Design
Software Engineering

Published
Nov 11, 2026

Updated
Nov 15, 2026

Read
10 min

Agentic Remote Control UX and IxD Patterns

Purpose

Many are still catching up to the usefulness and power of remote controlling agents. For those out of the loop 😉, agentic remote control means controlling agents remotely. It's about spawning and driving agentic work while untethered from a laptop or desktop machine. That's it.

Though the canonical example of a remote controller is a smartphone, it's worth mentioning you can go further upstream. I use the Index 01 ring as an agentic remote controller for example. That said, this article will focus on the visual UX and IxD patterns of smartphone interfaces.

The patterns below are ones that I've landed on over the past year of first-hand experience using Ora—my personal agent fleet management tool. She's a daily driver so I've had a lot of reps (both at home and work). Sampling from Ora's Manifests it looks like personal use with Anthropic models alone hits over 1K threads and 6K prompts. 1K+ of those prompts came from my smartphone. Reps.

Ora manifest sample

Ora manifest sample

Though others have independently developed and applied some of the patterns, the below set is informed from my own design and engineering background in combo with daily Ora use.

Hopefully more mainstream agentic remote control products will address their shortcomings and implement a lot of the below low-hanging fruit improvements.

At last check, the underleveraged patterns are Collapsible Composer, Composer Snap, Docked Reference, Selection Expansion, Selection Actions, and Action Chaining.

Users win by gaining more control with less effort (see MIMO below). I bet existing product teams could even point to this article and one-shot them all or fill in their gaps.

Back-story

Myself and other early adopters in 2025 loved Claude Code but wanted an agentic engineering UX that was driven remotely. At the core, we wanted to talk to our repos without being anchored to a laptop or desktop machine.

As designers and engineers many of us created our own agent fleet management tools to accomplish this. Peter Steinberger built Clawdbot Moltbot OpenClaw to interact with agents through existing apps like WhatsApp and Telegram. Others just wanted raw Claude Code via a TUI on their phone (god bless these folks). I on the other hand had a ton of ideas for a first class rich UI—Ora was born.

A few—Ora included—even beat Anthropic's Claude Code and OpenAI's Codex remote control market offerings. Though it's not lost on me that they had a much bigger problem to solve (large user base), the fact stands. Many other products have since landed their remote control offerings throughout 2026.

In any case, feel free to jump to the patterns or continue for a UX and IxD terminology refresher that includes some insight into my IxD decision process.

Refresher

To start, here are some terms:

  • UX - User Experience
  • IxD - Interaction Design
  • Control - An interactive UI element
  • MIMO - Minimum Input Maximum Output

In the context of IxD we directly influence UX.

User experience is subjective and unique to each user and it can improve or degrade during each user session. A designer most directly influences it via their design decisions, but it's less recognized that all contributors to a product influence it (yes, even infrastructure, backend, security, marketing, etc.).

Interaction design is a subcategory of hardware and software product design, but we'll be focusing on the software side. IxD decisions are about graphics, controls, triggers, timing, and their interwoven application to aid or guide user task flows. Put simply, IxD is where designers think through a specific small task flow and make microdecisions that optimize a user's experience through said task flow. That's it.

Microdecisions

Concretely, these microdecisions involve:

  • static graphics (attention and feedback)
  • motion graphics (attention and feedback)
  • controls (button, toggle switch, slider, etc.)
  • triggers (click, tap, gesture, voice, etc.)
  • timing (progressive disclosure and UX "feel")

These microdecisions typically result in microinteractions the user takes. For more on triggers, feedback, and doing more with less, see my notes on Dan Saffer's Microinteractions - Designing with Details.

Combined these microdecisions are intended to nudge a user toward the designer's intent—most often a control. The designed intent followed by user interaction is what enables this invisible dance, the interactive conversation between a user—or users—and the interface.

I use the personal acronym MIMO to guide my decision making. This is a shorthand that encapsulates other UX and IxD patterns for guiding my design decisions (topic for another article). A tactical shortcut process you can employ is:

  1. Go through a task flow
  2. Count how many trigger steps you took (click, tap, gesture, voice, etc.)
  3. Ask yourself, "can I delete this or replace it in favor of..."?
  4. Repeat

Through this process you can get to MIMO where a user provides minimum input (low effort) but attains maximum output (high outcome).

You can keep this framing in the back of your mind when exploring the patterns below. All of them were filtered through the MIMO lense in combo with the below concepts:

  • visually constrained real-estate
  • touch-based input and "thumb first" thinking
  • virtual keyboard (and voice) input

A few additional notes:

  • The desktop experience often benefits from or can extend the patterns
  • The interaction trigger can be customized per pattern implementation (often with an accompanying keyboard shortcut on laptop and desktop)
  • Some patterns often have emergent relationships enabling UX combos

Patterns

Nerd-out below.

This is a non-exhaustive list.

Collapsible Composer

This is a common toggle IxD pattern applied specifically to a composer UI's screen real-estate.

Name

Collapsible Composer

Example
Problem

The user can't read, scan, or interact with content the composer UI obstructs. This user pain is exacerbated when a virtual keyboard is expanded.

Use When

Content is obstructed by the composer UI

Solution

Provide a thumb-accessible toggle switch control for contracting and expanding the composer UI

Implementation

Add a single tap listener to a thumb-accessible toggle switch control that toggles between two states:

  1. Tap to contract (optionally use transition animation)
  2. Read, scan, or interact with previously obstructed content
  3. Tap to expand (optionally use transition animation)

Composer Snap

This is a common toggle IxD pattern applied specifically to a composer UI's height.

Name

Composer Snap

Example
Problem

The user's input is expansive enough to obstruct content. This user pain is exacerbated when a virtual keyboard is expanded.

Use When

The user has expansive prompt input and wants to quickly view obstructed content

Solution

Provide a thumb-accessible toggle switch control for contracting and expanding the composer UI's input height

Implementation

Add a double tap listener to a thumb-accessible toggle switch control that toggles between two states:

  1. Tap to contract (optionally use transition animation)
  2. Read, scan, or interact with previously obstructed content
  3. Tap to expand back to previous height value (optionally use transition animation)

Slash Menu

This is a common command menu IxD pattern applied specifically to a composer UI. A thumb-accessible / button control makes commands available without typing. This and the Action Chaining pattern are one of the aforementioned emergent UX combos.

Name

Slash Menu

Example
Problem

The user can't quickly access commands without remembering and typing their names. This user pain is exacerbated when a virtual keyboard is required just to repeat a recently used command.

Use When

The composer supports reusable commands and the user wants to discover or repeat them with minimal input

Solution

Provide a thumb-accessible / button control that opens a command popover menu relative to the composer

Implementation

Add a single tap listener to a thumb-accessible / button control in the composer that toggles a recent commands popover menu. Additionally support typing / in the composer to find commands:

  1. Tap the / control to view recent commands without opening the virtual keyboard, or type / followed by part of a command name to filter available commands
  2. Tap a command to insert it into the composer and close the menu
  3. Add any desired prompt context before submitting
  4. Allow dismissal without selecting a command

Ora on desktop expands via:

  • ctrl+/: toggle recent commands
  • up|down: navigate commands
  • enter|tab: insert the selected command
  • esc: dismiss the menu

Docked Reference

This is a useful variation of the common modal and lightbox UI patterns where a portion of content is desired to be unobstructed.

Name

Docked Reference

Example
Problem

The user can't remember the specific part of a reference (an image, document, code, etc.) they want to prompt about and it is hidden from view while prompting

Use When

A user wants to prompt about a specific part of a reference without having to open it, refresh their memory, close it, then resume prompting with refreshed memory

Solution

Design a UI that supports viewing references while keeping the composer unobstructed

Implementation

Add a pane, modal, popover, or similar UI component that allows referencing without composer obstruction

  1. Tap to dock reference
  2. Don't obstruct the composer
  3. Compose while reference in view

Ora on desktop expands via:

  • ctrl+e: open docked reference if closed where left|right navigate prev|next
  • esc: close docked reference

Quick Zoom

This is a standard touch IxD pattern that toggles (or cycles) between different zoom levels. It is often accompanied with panning support. This and the Docked Reference pattern are one of the aforementioned emergent UX combos.

Name

Quick Zoom

Example
Problem

The user can't quickly and easily focus on a reference

Use When

The user wants a closer look at a particular reference

Solution

Provide a thumb-accessible hit area for touch zoom support

Implementation

Add a double tap listener to a reference that toggles zoom levels. Optionally add a drag listener for panning the zoomed reference.

  1. Double tap to zoom in
  2. Touch drag to pan
  3. Double tap to zoom out

Ora on desktop expands via:

  • ctrl+up|down: zoom in|out
  • mousewheel: zoom in|out
  • click+drag: pan

Anchor Jump

This expands upon the common "jump to" anchor navigation IxD pattern but instead supports a dynamic cycle instead of a one-shot.

Name

Anchor Jump

Example
Problem

The user can't quickly reference content not currently in view. This user pain is exacerbated when the content is far away (typically vertically) from their current viewing location.

Use When

Desired content can have static or dynamically generated anchors

Solution

Provide a thumb-accessible button control for cycling anchors

Implementation

Add a single tap listener to a thumb-accessible button control that cycles the anchors. Add a long-press to conditionally anchor jump (canonically top ↔ bottom of a page)

  1. Tap to cycle to the next anchor
  2. Read, scan, or interact with previously out-of-frame content

Ora on desktop expands via:

  • ctrl+click+up|down: anchor jump to prev|next assistant bubble
  • alt+click+up|down: anchor jump to prev|next assistant bubble and/or its headings

Selection Expansion

This is an uncommon text selection pattern optimized for applications that can intuit a user's next step following text-selection. MacOS and Windows support triple click to expand to a word's enclosing paragraph as an example application.

Name

Selection Expansion

Example
Problem

A user can't select a range of text easily. Only the cumbersome and slow native mobile text-selection tools are available.

Use When

The user wants to quickly select a text range with minimal effort

Solution

Provide a thumb-accessible alternative to native text-selection

Implementation

Add a double tap listener to a thumb-accessible text container that intelligently expands the range from the target word to its enclosing sentence, statement, line, or paragraph

  1. Double tap a word
  2. Text-selection automatically expands the word selection to a sentence, statement, line, or paragraph selection

Ora on desktop expands via:

  • ctrl+click: lighter IxD alternative to desktop triple click (this is an example of MIMO via two actions vs. three)

Selection Actions

This is a fairly common variant of the Contextual Actions pattern whose prerequisite is a text-selection. This and the Selection Expansion pattern are one of the aforementioned emergent UX combos.

Name

Selection Actions

Example
Problem

The user can't quickly execute their known next action after a text-selection

Use When

The user has common next step use cases after a text-selection

Solution

Provide a thumb-accessible control set of quick actions

Implementation

Add a selectionchange listener to a text container and conditionally position a contextual quick actions menu

  1. Text-selection (native or Selection Expansion)
  2. Calculate and conditionally display a contextual actions menu relative to the selected text
  3. Auto-close the contextual menu when the user taps an action or taps outside

Action Chaining

Action Chaining generalizes the Execute and Exit pattern I created, which you can learn about in the Interaction Design Pattern - Execute and Exit article. This and the Slash Menu pattern are one of the aforementioned emergent UX combos.

Name

Action Chaining

Example

There are a few scenarios where it's useful in Ora:

  • Recent commands inline button control (double tap) - append the command to the input, automatically submit the prompt, and then navigate home (shown in the video above).
  • Inline document commands button control (single tap) - same as Recent commands above, but triggered via a button control within a thread or document.
  • Prompt submission button control (double tap) - execute the prompt and then return (or stay) home.
Problem

The user can't quickly execute an action and its known follow-up action(s) without multiple steps

Use When

The user commonly repeats the same follow-up action(s) after a primary action

Solution

Provide a thumb-accessible button control that executes a primary action and its follow-up action(s) in one gesture

Implementation

Add a double tap listener to a thumb-accessible button control that chains two or more actions:

  1. Execute the primary action (ex. submit the prompt)
  2. Execute the follow-up action(s) (ex. return home)

Ora on desktop expands via:

  • ctrl+enter: prompt submission and then return (or stay) home

Decision Chips

The canonical example are one-click UI responses to tool calls like AskUserQuestionTool. These can be customized for other one-click action scenarios.


Dictation

Voice vs. typing—you get it. This is also where products like the aforementioned Index 01 come in.


Handedness Toggle

Not implemented in Ora but useful for applications where left-handed and right-handed users have opposite thumb-accessible needs.


Conclusion

As you can see these are all quite simple and straightforward patterns to implement. Additionally, many of them are useful outside agentic remote control scenarios.

If you are a designer, engineer, or builder working on remote control features I'd love to hear what you adopt or ignore (and why). If you're not working on remote control features but still learned a thing or two then I'd love to hear what resonated.

You can tweet me @derekknox.