Evaluating the usability of Google Cloud's AI/ML developer platform - Vertex AI
As a UX Researcher on a 4-person team, I helped plan and run an 8-week usability study on Vertex AI Studio's loginless desktop experience for Google. We tested with 8 developers across six tasks replicating the initial user journey, then presented 18 actionable insights with recommendations to Google's UXR team in Seattle.
- Company
Google- Role
- UX Researcher
- Timeline
- 8 Weeks (2025)
18
actionable insights presented to Google's team, along with specific suggestions to enhance the Vertex AI Studio experience.
Influenced
future platform priorities by identifying key usability issues in the loginless web experience.
What is Vertex AI?
Vertex AI is a cloud-based machine-learning development platform for building and using AI models, designed for enterprise developers to build and deploy AI models and applications at scale. Vertex AI's loginless experience is a free way to interact with Gemini models in Vertex AI Studio.
I worked with 3 of my classmates on this project, and we reported to 1 UX Research manager at Google.
We had weekly meetings with Katie O'Leary, the staff UXR manager at Google, to discuss our progress, clear doubts, and plan next steps. This study was part of our grad school class led by Dr. Katya Cherukumilli, who also mentored us throughout.
At the Google SLU office after presenting our findings!
Google wanted to uncover the usability barriers in Vertex AI's loginless experience for desktop, to improve its usability and retention.
Vertex AI is aimed at developers with a range of experience, from little to a good amount, building applications with AI models. Google asked us to usability-test its desktop, loginless experience directly with that audience.
What we did
This usability study assessed the loginless web experience of Google Vertex AI Studio, focusing on how developers navigate and engage with its features.
Participants were students and professional developers familiar with AI application development, all first-time users of Vertex AI. Sessions were in-person and moderated, on the University of Washington campus, with participants completing tasks while thinking aloud as we recorded audio and video of their reactions and navigation. Questionnaires after each task and after testing added supplemental quantitative data.
The tasks traced that journey step by step: from the marketing page, to the Vertex AI Freeform page, to finally getting the code for a model implementation.
0 min
per session
0
tasks
A hands-on end-to-end UX research experience with ample learnings!
In close collaboration with Google's research team, I worked hands-on across every part of this study: drafting the plan, building the test kit, managing recruitment, moderating sessions, taking notes, and analyzing the data to generate insights aimed at improving Vertex AI Studio's loginless web experience.
We started with the following research questions:
01
Is the loginless experience usable and satisfying?
02
What are the major frictions?
03
Does the loginless experience provide enough capabilities to engage and entice new users to sign up for more access?
Originally planned as a 10-week project, we swiftly adapted our approach to meet an accelerated 8-week timeline due to scheduling changes.
None of us had a development background, so before testing we ran our own cognitive walkthroughs and built interaction maps of Vertex AI Studio, picking up the "developer lingo" we'd need to read participants' reasoning during sessions.
A glimpse of how the interaction mapping was done to understand all the current flows & features
We leveraged personal connections and University of Washington's CS directories to recruit participants for our study.
We used convenience sampling from personal connections, plus large-scale email outreach to computer science students at University of Washington, to recruit participants with some level of experience in AI application development.
By sending out screening surveys, we were able to identify qualified participants and maintain integrity of our results by excluding those who had previous experience with Vertex AI and those who were only available for remote testing.
~0
outreach emails
0
participants recruited
Out of a total 21 responses, we narrowed down on 8 participants.
Below was our inclusion criteria:
01
The participant must have a developer background with familiarity with programming languages and some experience using AI platforms to build applications.
02
The participant must be willing to participate in-person on the University of Washington campus to ensure data collection accuracy and consistency across all participants.
Participants varied in occupation, familiarity with AI tools, the context in which they used them, and the AI development skills they had.
A rigorous multi-method testing protocol: creating a controlled environment for comprehensive user insights.
We conducted in-person usability sessions in isolated study rooms on the UW campus. Each session was recorded with participant consent, obtained through a signed consent form at the start. Sessions began with reading a standard scenario, followed by a set of tasks created specifically for each participant.
Each session involved a team of four
One moderator
One person collecting qualitative insights
One person tracking time to task completion
One person noting the number of clicks to success and identifying any click path errors
After each task
Post-task questionnaire
Quantitative scores based on standard NASA-TLX and CSAT questions, plus qualitative feedback on frustrations or satisfaction with the task.
After all tasks
Post-test questionnaire
Based on standard SUS and NPS questions, summarizing participants' overall experience with Vertex AI after completing all the tasks.
Here's how we streamlined our data collection.
Contents under NDA
We combined qualitative feedback with quantitative metrics to validate our findings.
After collecting the data, we conducted affinity mapping with qualitative notes from observations, interviews, and questionnaires to identify key patterns, and pulled representative quotes from session recordings across a diverse set of participants.
We analyzed quantitative metrics including task completion times and click counts, comparing these against our benchmarks, which lined up with what we were seeing qualitatively.
From post-task and post-test questionnaires, we visualized satisfaction (CSAT), usability (SUS), and cognitive load (NASA-TLX) scores across every session.
Affinity mapping, grouping of findings (under NDA)
We presented 18 findings along with our recommendations to Google's UXR team at the SLU Google office, Seattle.
These insights are expected to guide the prioritization of future Vertex AI Studio updates. We organized and presented our findings in the following categories:
Task-based findings
We organized results by the task each finding surfaced from, rated severity on a 4-point scale from Jeffrey Rubin's Handbook of Usability Testing, and grounded each one in direct user quotes.
Significant concerns beyond task-based issues
We discovered issues that were unrelated to the tasks but could directly be linked to our research questions around user retention.
Quantitative metrics
Task completion times, click counts, and the CSAT, SUS, and NASA-TLX scores from our questionnaires pointed in the same direction as what we saw qualitatively, without us treating an 8-participant sample as statistically conclusive on its own.
Below are snippets peeking at how we presented our quantitative findings. Due to an NDA clause, I'm unable to share the insights that came out of this study.
What would I do differently?
01
Increase dry runs before testing
Our initial dry runs helped us cut distracting elements from the test kit and sharpen the components that mattered. More of them would have let us optimize further.
02
Deepen technical domain knowledge prior to testing
Despite our interaction mapping and early research, we still hit unfamiliar developer terminology during sessions. It didn't compromise our results, but more prep in Vertex's own technical language would have avoided the momentary confusion.
03
Eye tracking
We tried to get eye tracking equipment for this study, but it fell through for technical reasons and there wasn't time to source a replacement. I'd want to bring it in for a future round: it would reveal natural click paths that qualitative observation alone can't capture.
04
Broaden recruitment beyond a single university
Our sample came almost entirely from UW's CS student network, and our in-person requirement filtered out anyone only available remotely. That skewed us toward students rather than the professional developers Vertex AI ultimately serves. It didn't undermine what we found, but a wider, less convenience-sampled pool would strengthen how far these findings generalize.