Michael Widjaya
Hi

HCI + AI Research Paper

Investigating the value of references in a conversational context.

2023

User Research

Research Paper Cover

Story

The boom in generative artificial intelligence (Al) and continuing growth of Voice Assistants (VAs) suggests their trajectories will converge. However, design guidelines for VAs prioritise maximum efficiency by advocating for the use of concise answers. This poses a conflict with the challenges around generative Al, such as inaccuracies and misinterpretation, as shorter responses may not adequately provide users with meaningful information. A better understanding of user behaviour when using the system is needed to develop revised design recommendations for Al-powered VA systems.

Solution

This paper reports an online survey of 256 participants residing in the U.K and nine follow-up interviews, where user behaviour is investigated to identify drivers of trust in the context of obtaining digital information from a generative Al-based VA system. Adding references is promising as a tool for increasing trust in systems producing text, yet we found no evidence that the inclusion of references in a VA response contributed towards the perceived reliability or trust towards the system.

Website

www.scienceopen.com/hosted-document?doi=10.14236/ewic/BCSHCI2023.13

Relevant Skills

Alexa Developer Console, Voiceflow, JavaScript, SPSS.

Process

This research project started with a literature review on relevant articles pertaining to large language models (LLM), sociphonetics (the study of how voices affect behaviour), and the general development of trust in the human-computer interaction (HCI) space. It was found that literature was lacking especially in the voice user interface space, where guidelines on how best to design a voice assistant were scarce.

Literature Review Collection
Literature Review Collection Snippet

I then started to get a better understanding of voice assistant users through quantitative research. Creating a survey that aligned with an existing scale, the “Trust in Automation” scale then allowed me to conduct a series of data analyses to find any potential correlations to discuss/focus on in the interviews.

Survey Components Snippets
Survey Components Snippets

For the interviews, I developed a Generative AI-powered voice assistant prototype for participants to directly interact with. I developed an Amazon Alexa Skill, connected to Open AI’s Completions API - imagine Chat GPT but voice based. The conversation design was made on Voiceflow, allowing me to quickly prototype the Skill, using JavaScript to call, receive, and manipulate data from the API.

Voiceflow Conversation Design Snippet
Voiceflow Conversation Design Snippet

The interviews were then done in a semi-structured way to allow for an open conversation yet have consistency between participants. Through the interview, it was concluded that people did not really value references and prefer quick concise answers. A lot of the participants blindly trusted the answers coming from the Alexa, even when programmed to provide mistake-filled answers.

Interview Process with Amazon Alexa (participant consented)
Interview Process with Amazon Alexa (participant consented)

Michael Widjaya
Projects
Photography
About