Project Overview

Problem Statement

The AI Deployment Application, an LLM-powered tool designed to assist customers during software deployment, needed to be evaluated for its usability and effectiveness. To ensure it meets user needs and delivers a seamless experience, UX Research was asked to conduct evaluative research, which resulted in a heuristic evaluation and an AI model evaluation using a usability tests.

Goals & Objectives

The goal was to identify usability issues, assess the tool’s overall effectiveness, and provide actionable insights for improving the user experience. This research was driven by a request from the Product Manager, who wanted to optimize the tool before wider adoption.

To effectively evaluate the AI Deployment Application, we conducted a comprehensive assessment that focused on both the user interface and the AI model's performance. This meticulous approach not only helped us identify usability issues but also provided valuable insights for enhancing the overall user experience.

Methods

Heuristic Evaluation

Rationale

Usability Test (Unmoderated)

Rationale

Research Process

Planning

Dissect Problem Statement
Research Method Selection & Timeline Creation
Recruitment & Set Up
Research and Data Collection
Analysis & Reporting

Recruitment

Identify key SMEs for research
Set up an unmoderated usability test using Maze

Research Execution

Our research methodology involved several strategic steps:

Heuristic Evaluation: Engaged 3-5 UX experts to independently evaluate the interface using established heuristics.
Usability Testing: Recruited 8-10 participants with varying levels of deployment experience to interact with the AI Deployment Application.
Task Scenarios: Created realistic deployment scenarios for participants to complete using the tool.
Post-Test Questionnaire: Gathered quantitative data on user satisfaction and perceived usefulness of the AI assistant.
AI Performance Metrics: Collected data on response accuracy, time, and relevance throughout usability testing.

Data Analysis

In our analysis phase, we undertook several key activities:

Compiled findings from heuristic evaluations, identifying critical usability issues along with their severity.
Analyzed usability test data, including task completion rates, time on task, and user satisfaction scores.
Reviewed AI performance metrics to assess accuracy, response time, and effectiveness in assisting with deployment tasks.
Identified patterns in user behavior and feedback that highlighted strengths and weaknesses of the AI Deployment Application LLM.
Prioritized issues based on their impact on user experience and deployment success.
Developed actionable recommendations for improving both user interface design and AI model performance.
Created a comprehensive report detailing our findings, insights, and proposed solutions for the Product Manager.

Insights

Design Elements: Highlight key findings with graphs, charts, or annotated screenshots.
Text Layout: Use two columns – one for visuals and one for text. Each insight should be clearly labeled and paired with its visual.
Visual Tips: Use colors to highlight important data points. Add emphasis to key insights with icons like exclamation marks or light bulbs.

Business Impact

By conducting this thorough evaluation, we provided invaluable insights aimed at optimizing the AI Deployment Application. Our goal was to ensure it met user needs while delivering a seamless experience during software deployment.

These insights enhanced readability through structured sections, bolded key points, and a more engaging tone. It also increased the model effectiveness by 40% by the time of AI tool deployment and first release.

Learning & Reflection

Learning

Mauris id fermentum nulla. Phasellus sodales massa malesuada tellus fringilla, nec bibendum tellus blandit.

Reflection