ChoiceMates

Demo Video

Play the video to see how ChoiceMates works!

Abstract

From deciding on a PhD program to buying a new camera, unfamiliar decisions — decisions without domain knowledge — are frequent and significant. The complexity and uncertainty of such decisions demand unique approaches to information seeking, understanding, and decision-making. Our formative study highlights that users want to start by discovering broad and relevant domain information evenly and simultaneously, quickly address emerging inquiries, and gain personalized standards to assess information found. We present ChoiceMates, an interactive multi-agent system designed to address these needs by enabling users to engage with a dynamic set of LLM agents, each presenting a unique experience in the domain. Unlike existing multi-agent systems that automate tasks with agents, the user orchestrates agents to assist their decision-making process. Our user evaluation (n=12) shows that ChoiceMates enables a more confident, satisfactory decision-making with better situation understanding than web search, and higher decision quality and confidence than a commercial multi-agent framework. This work provides insights into designing a more controllable and collaborative multi-agent system.

Figure 1. The user converses with agents in the conversation space (1) to gather diverse perspectives, tracks discovered criteria and options in the conversation history (2), and pins what matters to the preference space (3) to reach a confident final decision.

Motivation

Comparing perspectives in unfamiliar decision-making

Understanding an unfamiliar domain means assembling scattered sources yourself, one at a time. This is especially difficult for novices, who constantly have to juggle between general knowledge and one's own preferences in the domain without personalized standards. Existing tools introduce an easier way of comparing different options, provide an overview, or present a likely preferred choice, where they assume both knowledge and preferences are already constructed.

Our approach brings in conversational interactions with multiple agents, where multiple agents can surface diverse viewpoints in parallel, and conversation makes follow-up questions cheap.

Most existing multi-agent systems automate. Instead, we want the user to orchestrate agents to help them make a decision.

From a 14-participant formative study

Design goals

• DG1Provide diverse opinions and perspectives in the domain through experiences.
• DG2Provide options to explore broad and contextually relevant domain information.
• DG3Allow for direct exchanges of information needs through conversational interactions.
• DG4Facilitate users in building preferences relative to the domain.

System

ChoiceMates

Three regions — a conversation space, a conversation history, and a preference space — work together as users explore an unfamiliar domain.

Agents populate the conversation space (1) and identify criteria and options as they talk. Those keywords automatically stack into the conversation history (2), where (a) shows unfamiliar information accumulating there. The user pins what matters to the preference space (3), where (b) is the save action, (c) is that preference flowing back to shape how agents respond.

Agent: basic unit of information

Each agent contains a one-line descriptor (e.g. "professional photographer"), the criteria it weighs, and a single option it prefers. Hovering an agent's icon reveals this full profile. This was designed to connect lived perspectives to options through a bridging characteristic: criteria.

1Conversation space

In ChoiceMates, there are three unique interactions users can do with the agents. Users can (1) chat with any agents currently in the space, (2) select specific agents to respond, or (3) call new agents in. Beyond user-agent interaction, agents can also respond to each other - agreeing, disagreeing, or asking follow-up questions — including when the user asks them to debate.

2Conversation history

In order to help users organize the information found throughout the process, on the side of the conversation space we keep a conversation history section. As agents talk, ChoiceMates automatically lists every agent, criterion, and option mentioned, each with a running count. Hovering a criterion highlights the agents and options connected to it, and agents or keywords can be hidden or pinned straight to the preference space.

3Preference space

This is where saved agents, criteria, and options live — pinned from the conversation history, each with a short note the user can add to capture why it mattered.

With the toggle button off (1), the preference space is hidden from agents and "any other agents?" brings in more serendipitous, broader scope of agents. With it on (2), the same question brings in agents aligned to agents, criteria, and options already pinned. This allows the users to nagivate between broad and relevant information in the space (DG2).

Study & results

Within-subjects study, n = 12

12 participants compared ChoiceMates against Web search and a commercial multi-agent GPT ("MultiAgent") across three unfamiliar domains — climbing shoes, a fabric shaver, and a robot vacuum. Each condition ran within-subjects, 20 minutes per task, in counterbalanced order.

How people used the agents

Across the sessions, four dominant strategies emerged for orchestrating ChoiceMates. Early on, participants talked to all agents to get a first read on the domain, relying on ChoiceMates' response logic and on overhearing agent-agent exchanges to pick up context they hadn't asked for directly. Once they had their bearings, they tagged multiple agents at once to compare them quickly, often asking a tagged group to "debate" or "tell me more." When one agent's profile felt closest to their own situation, they conversed with that one agent in depth — P10 spent 4 of 10 turns with a single agent, asking follow-ups beyond its opening pitch. And whenever the current agents didn't reflect their preferences well enough, participants called in more agents by asking "any other agents?", both to widen the space and to double-check they hadn't missed an option.

Reliability

We also checked the reliability of the agents' responses with a small-scale hallucination analysis over the 439 agent messages from the study: 1.82% objective inaccuracy (8/439), 2.85% subjective inaccuracy (28/981 pieces of subjective information), 4.78% irrelevant responses (21/439), and 1.14% self-contradiction (5/439).

Discussion

Takeaways

• Agency, not automation. MultiAgent felt efficient but produced decisions people didn't stand behind. Orchestrating agents made people reason about their own priorities instead of accepting an answer handed to them.
• Can agents be the unit we understand and decide with? Not a source to read, but a viewpoint you can ask, compare, and pin. "Hearing from multiple friends" (P12) surfaced trade-offs and eased pressure to accept the first suggestion — but human-likeness cuts both ways: if agents become the unit of information, they persuade because they're human-like, and in deployment someone can pay to be one of them.

Cite this work

BibTeX

@inproceedings{10.1145/3742413.3789107,
        author = {Park, Jeongeon and Min, Bryan and Son, Kihoon and Song, Jean Y and Ma, Xiaojuan and Kim, Juho},
        title = {ChoiceMates: Supporting Unfamiliar Online Decision-Making with Multi-Agent Conversational Interactions},
        year = {2026},
        isbn = {9798400719844},
        publisher = {Association for Computing Machinery},
        address = {New York, NY, USA},
        url = {https://doi.org/10.1145/3742413.3789107},
        doi = {10.1145/3742413.3789107},
        abstract = {From purchasing a gift to deciding on a hobby, unfamiliar decisions—decisions without domain knowledge and experience—are frequent and significant. The complexity and uncertainty of such decisions demand unique approaches to information seeking, understanding, and decision-making. Our formative study highlights that in the current workflow, users want to start by discovering broad and relevant domain information evenly and simultaneously, quickly address emerging inquiries, and gain personalized standards to assess information found. We present ChoiceMates, an interactive multi-agent system designed to address these needs by enabling users to engage with a dynamic set of LLM agents each presenting a unique experience in the domain. Unlike existing multi-agent systems that automate tasks with agents, the user orchestrates agents to assist their decision-making process in each turn, through chatting with all agents, with a tagged subset of agents, or calling in new agents into the space. By comparing ChoiceMates with a web search condition and a multi-agent framework (n=12), we show that ChoiceMates enables a more confident, satisfactory decision-making with better situation understanding than web search, and higher decision quality than a commercial multi-agent framework. We further illustrate how participants utilized ChoiceMates to make unfamiliar decisions, providing insights into designing a more controllable and collaborative multi-agent system.},
        booktitle = {Proceedings of the 31st International Conference on Intelligent User Interfaces},
        pages = {1526–1550},
        numpages = {25},
        keywords = {multi-agent interactions, conversational user interface, decision-making support, large-language models},
        location = {
        },
        series = {IUI '26}
        }
}