This blog post aims to foster inspirations for future UX design for AI systems by looking at the evolution of designs from the past to present. We will also look at some of the up coming product features to better understand the direction the industry is heading towards.

The current dominant interface for AI systems is the single-frame conversational chatbot. In the far future, we anticipate the emergence of artificial general intelligence (AGI) where we are all doomed. We will try to analyze the transitional UX that bridges between our current world and the future.

alt text

Genesis

The current wave of AI began with the launch of ChatGPT in November 2022. It featured a simple UX, designed to showcase the capabilities of large language models (LLMs). The main interface displays ongoing conversations, which scroll upward, with a chat box positioned at the bottom for user inputs. Responses from the LLM are clearly marked, with thumbs up or down feedback button.

All conversations are organized and stored into threads, visible on the left-hand side of the screen.

Features:

  • Chat UI
  • In-line Planning and Reasoning

alt text

Compound AI Systems

As LLMs evolved, it became apparent that they lacked access to all necessary information to answer queries. Thus, starting in 2022, engineers started to adopt Retrieval Augmented Generation (RAG) design, where system retrieves the relevant information from search engines and then generate answers based on the retrieved chunks.

These technologies ranged from vector searches to augmenting data from various sources, including search engines, APIs, SQL databases, and code execution environments. By 2024, researchers at Berkeley and Stanford formalize the concept and coined the term Compound AI Systems to describe these systems.

alt text

UX for Building Compound AI Systems

The emergence of conversational assistants built on compound AI systems necessitated UX designs for their construction. The first build system was OpenAI’s GPT Store, introduced in June 2023. GPT Store was later rebranded ChatGPT Assistant. Its interface includes tools like file upload for adding knowledge and code interpretation for runtime, along with the flexibility to integrate additional tools based on function definitions. This setup provides a simple and effective experience for adding external tools in LLMs.

alt text

Other sophisticated commercial systems are also being designed and could enter the market sometimes in 2024. For example, Twilio’s Assistant. Twilio’s Assistant enable users to name their assistant and select a Segment workspace to integrate customer data, add personalization and a personality prompt. We believe both the personalization and personality prompt are implemented as part of the system prompt.

alt text

During the build process, the Tools tab offers a detailed list of available tools, description of the tools, and their URLs.

alt text

Furthermore, users can add and define new tools by adding a tool’s description, URL, API schema, and authentication methods. For more information on tools for LLMs, visit Tools for Large Language Model Agents.

alt text

Twilio Assistant also supports the addition of new knowledge bases, allowing users to integrate websites, raw text, upload files, or link to a vector database.

alt text

UX for Compound AI Systems

An example of a compound AI system assistant is Presentation and Slides GPT: PowerPoints, PDFs, featuring Browsing and Actions capabilities. Notably, it uses a paperclip icon to access external data as a tool. Generated artifacts like text, code, graphs, or charts appear in-line within the chat window.

Features:

  • Chat UI
  • In-line Artifacts
  • In-line Planning and Reasoning

alt text

Industry progression has led to a split-screen UX design, with the left side dedicated to chat interactions and the right displaying results and artifacts, which may prompt further interactions.

Features:

  • Chat UI Panel
  • Artifacts Panel
  • In-line Planning and Reasoning

alt text

Applications of this design include OpenAI’s Wanderlust demo from the November 2023 OpenAI Dev Day, where left-side instructions generate maps on the right.

alt text

In June 2023, Anthropic introduced a similar design for its Artifacts functionality, rendering code artifacts on the right side of the screen.

alt text

Agentic Systems

The next evolution in Compound AI systems is agentic systems. Agentic systems use AI (LLM) as the control logic. The AI could be augmented with tools, memory, knowledge bases, and/or external search engines. These systems commonly use LLM to first Reason and then Act on the plan, or ReAct pattern, as described in Yao et al, 2023. This is in contrast with traditional systems that has developer-written control logic.

In other words, an agentic system has the agency to decide what to do.

alt text

To better serve this type of agentic systems, we have observed various different UX designs to enable some interaction between human and AI. We covered Parrallism and Interaction Material in the UX for Compound AI Systems section. A classic example of Overlap is Github Copilot where we provide interaction overlay directly inside VSCode. The classic example of Spatialty, or Post-It Notes, is ComfyUI. Comfy UI was the de facto format for users to use graphical GenAI models like Stable Diffusion.

alt text

Examples user journeys include common two-panel designs and Post-It flow designs, effectiveness depending on the usage model.

alt text

As an example Post-It design, here’s an agent built using Salt.ai interactive low-code agent builder tool. These type of Post-It design is rarely used for agentic based applications.

alt text

Agentic UX - Devin

One of well known agentic system out there is the AI Software Engineer, Devin. Devin uses the two-panel design with a conversational interface on the left and agent workspaces on the right. The conversational interface is where user is able to interact with Devin and issue commands or correcting Devin’s actions.

Devin’s work spaces are in the right hand side. Note that Devin has several workspaces, including shell for code executions, browser for web data, editor for code executions, and a planner for it to think and plan out its steps. One key feature is user can toggle on follow to follow Devin’s behaviors.

Features:

  • Chat UI (left panel)
  • Artifacts Tab (right panel)
  • Planning and Reasoning Tab (right panel)

From Devin’s initial demos, we note that it takes several minutes to hours for it to complete a task.

alt text

Agentic UX - Atlassian Rovo

Atlassian announced the development of its Rovo agents platform in 2Q 2024. One of the agent living on its platform is Backlog buddy. The Rovo platform uses a more integrated UX on top of its existing designs for Jira. Underneath the hood, it still follows the basic agentic system building blocks.

Features:

  • Chat UI (Widget)
  • Planning and Reasoning Tab (Popup Widget)

Here, user first chat with Backlog Buddy over a conversational UX.

alt text

Then Backlog Buddy displays its Plan in a pop up window. Through this pop up window, user can then decline or approve Backlog Buddy’s plans, which is to generate all of the Epics and Stories. This Plan window behaves similarily to the Plan in Devin’s workspaces.

alt text

Agentic UX - Zapier

As the agentic systems grows more powerful, the agents can be used for robotic process automation (RPA) type of tasks. For example, research the Internet to produce a daily newspaper to be emailed or monitoring Github for opened pull requests to automatically review the PRs.

Zapier’s AI Automation product is a good example. Zapier is an automation platform that provides many connectors to different data and services, so going to agentic system is a natural extension for its business.

Here’s an example on setting up an agent on Zapier’s platform. Users provides instructions, triggers, and actions to take.

Features:

  • Build UI
  • Chat UI for Testing (Left Panel)
  • Planning and Artifacts Tab (Right Panel)
  • Scheduling UI

alt text

Then, the agent conducts the actions as a rehersal and let user experience how the agent would function.

alt text

Then the User can setup the actions and schedule it.

alt text

Conclusion

This overview highlights the rapid evolution of AI system UX over 20 months since ChatGPT’s release. As LLMs become more capable, we will uncover more usecases and these UX will continue to evolve.

References

@article{
    leehanchung,
    author = {Lee, Hanchung},
    title = {Navigating Through Time: The Evolution of UX in AI Systems},
    year = {2024},
    month = {07},
    howpublished = {\url{https://leehanchung.github.io}},
    url = {https://leehanchung.github.io/blogs/2024/07/31/ai-system-ux/}
}