Project Overview
The AI Deployment Application was an LLM-powered tool designed to support customers during software deployment. I led evaluative research to understand whether the experience was usable, useful, and trustworthy enough for broader adoption, resulting in a heuristic evaluation, unmoderated usability testing, and model-level feedback for the product team.
Evaluated both the user interface and the underlying AI model performance.
Tested realistic deployment scenarios with participants across experience levels.
Improved model effectiveness by the time the AI tool reached first release.
The AI Deployment Application needed to be evaluated for both usability and effectiveness before broader adoption. Because it relied on an LLM to assist with deployment tasks, the team needed clarity on more than interface quality. They also needed to know whether responses were accurate, relevant, and helpful enough to support real user work.
To evaluate the product thoroughly, I looked at the experience through both a UX and AI-performance lens. That meant reviewing the interface structurally, then testing how the product performed in realistic usage scenarios.
The Process
The product was still evolving in real time, so the research needed to move fluidly between early evaluative signals and concrete release guidance.
The strongest insights came from looking at interaction quality and model behavior together. The interface could not be evaluated in isolation because trust in the tool depended on both what it looked like and what it said.
Users did not just want answers quickly. They wanted clearer signals about what the bot could do, what it was uncertain about, and how much to rely on its outputs.
Even strong responses lost value when the interface made next steps, context, or interaction states feel ambiguous.
Scenario-based tasks showed where the assistant meaningfully supported deployment work and where it created friction or hesitation.
Working closely with product and ML stakeholders helped surface risk areas like hallucination and confidence scoring early enough to influence the roadmap.
I organized insights so product partners could clearly separate interface problems from model problems, while still seeing how they interacted in the full user experience.
Business Impact
This evaluation gave the team actionable guidance for improving the product before wider adoption, while also helping the AI experience feel more intuitive, explainable, and reliable.
Increase in model effectiveness by the time of deployment and first release.
Recommendations targeted both UI design and AI model performance rather than treating them separately.
Structured findings improved readability and made product tradeoffs easier for stakeholders to act on.
Research helped the team optimize the experience before wider rollout instead of reacting after adoption issues grew.
This project deepened my ability to design research that balances the complexity of AI systems with the everyday workflows of technical users. Working with a product still evolving in real time, I learned how to move between foundational inquiry and scrappy evaluative testing, often within the same week.
One of the biggest takeaways was that developers did not just want speed. They wanted clarity around what the bot could and could not do. Building user trust required transparency, not just better automation.
Collaborating with product and ML stakeholders early helped us identify risk areas around hallucination and confidence scoring before they became bigger product problems. It also reinforced how valuable mixed methods can be when the goal is to evaluate not just usability, but confidence in an AI system’s outputs.
Ultimately, this work reminded me that successful AI integration is not just about automation. It is about augmenting human work in ways that feel intuitive, explainable, and reliable.