Background: Pantheon acquired StagingPilot (renamed to Autopilot) in early 2019. When I joined the design team, my first major project was to integrate this legacy product into Pantheon’s dashboard, and to identify the key features that would make it into a market-leading solution.
The Problem: Website owners need to update their sites to introduce new functionality and to stay secure. However, each update introduces the risk of breaking the site's visual appearance or business-critical functionality. Especially when it comes to security updates, many businesses choose to save development resources and defer for months or even years.
This leads many to get to a point where it would be easier to "start from scratch" rather than take on all of the incurred technical debt of updating properly. This cycle can stifle innovation and can leave sites outdated.
Getting to the problem statement
We began by conducting research—a heuristic evaluation of the legacy product, user interviews to better understand what potential customers are looking for, and an in-depth competitor analysis.
Screenshot of some of the analysis we did:
Through our interviews, I learned that website maintenance work requires expertise and continual effort, an especially daunting task when managing a large portfolio of sites. There are new updates and security disclosures every day, so sites are almost always out of date.
We also learned a bit more about how users traditionally handle website updates. They usually have to do one or more of the following.
With this happy path in mind, I was able to break down the solution into 7 key user flows. I captured the flows in a user story map (built with Miro), so that other stakeholders could easily digest the information and provide input.
Multiple rounds of design iteration, testing with users and presenting to stakeholders allowed us to take advantage of quick feedback loops and build confidence in the solution we were implementing. This product was highly anticipated within Pantheon, and we were facing pressure from stakeholders to deliver. Design at Pantheon runs parallel sprints to the development team, so we were consistently a month ahead of our engineers. This enabled us to break down designs into eng tasks right away. Every one of the features below was built into a Figma prototype and shared with real customers for validation.
This is a three step process which allows Autopilot users to customize the feature for their site's needs. They can set the desired scope of the tool, exclude specific themes or plugins from the automated updates, and define pages for visual review.
Iterations:
Latest version in production:
Automated regression testing for key pages ensures that UX is consistent while securing your site and implementing new features.
Iterations:
Latest version in production:
The Autopilot Activity Feed displays historical reporting of the updates and testing that were performed on a given website.
Iterations:
Latest version in production:
The Status page gives real-time information to users who are viewing Autopilot for a specific site.
Iterations:
Latest version in production:
In March 2021, Autopilot was ready to be shared with a small number of customers through our Early Access Program. As Design Lead, I was fortunate enough to partner with our sales engineers and product manager to hand select the first 10 customers. I set up hour-long calls with each of these customers, and alongside my product manager, we used this time to conduct moderated usability testing. We enabled Autopilot on each of their accounts, and observed them as they navigated the interface and shared their feedback.
All of our notes from these sessions (including the script, and our analysis) are captured in a sheet like the one below:
This process took a few weeks. At the end of it, we analyzed our notes and I created a research report that was presented to the entire executive team and sales organization.
Everyone was eager to ship the product to a larger audience, so it was important that we took a moment to share our learnings from the Early Access group. This gave stakeholders confidence that we are prioritizing urgent bugs and UX issues prior to releasing it to the general public.
Below are some screenshots of a few of the slides I presented:
From April - July 2021, we gradually released Autopilot to more customers (About 5 per week). Every week during this period, our scrum team reviewed and discussed the customer feedback, so that we could start implementing bug fixes and improvements.
I’ve highlighted a few of those UX Improvements below:
Autopilot launched into General Availability in early August. While the initial feature adoption rates was low due to the amount of undiscovered bugs, we saw a dramatic improvement after a few weeks.
After 3 months here is a summary:Autopilot is being used on 17,320 websites, across 300 different accounts. Of all the eligible customers, 10% are using Autopilot. On average, it saves each account roughly 4 hours per week of time. That extra time back empowers website teams to focus on more strategic work.
As part of the launch, myself and a few colleagues hosted an external webinar. I demo'd the product from end-to-end and answered questions in the chat from our customers.
As always, our plans didn’t go as expected. Below are a few of the major things I learned from the project.
Lack of design predictability Our product design team at this time was a small group of 3. All of us were working on at least 3 projects simultaneously. At the start of the Autopilot project, design work wasn’t being tracked in a formal way (no JIRA, n Asana..) and worse, we weren’t estimating our design tasks. This led to the inability to factor design work as part of the official project timelines. It sometimes felt as if I was holding back the engineers, or blocking them from being able to implement frontend work. Unfortunately, it also led to shortening some of our UX research activities so that I could speed up the design cycle. Luckily, we corrected this about 3 months into the project once we developed a project management playbook for design.
Overlapping projects
Aside from Autopilot, another major project I was working on at the time (2020) was a complete redesign of Pantheon’s dashboard UI and navigation. This project started to pick up around the same time that Autopilot was being developed. Midway through development for the MVP of Autopilot, the executive team recommended we pause all frontend work and restart in the new dashboard interface. If our legacy dashboard would get sunsetted soon, it would make more sense to develop all new features in the new UI. While this was great for our customers (aside from a more modern looking UI, the new dashboard performed much better) it did extend our release timelines by several month. This decision required me to redo several of the designs for the updated interface, using our new design system components.
Autopilot is projected to be a major contributor to upsell revenue in the following years, which means our work has just started. In December 2021, I partnered with our researcher and product manager and launched an in-app customer satisfaction survey for existing Autopilot users.
We’re hoping to use this data to better understand why customers aren’t using Autopilot for 100% of their website portfolio. Our hypothesis is that the lack of awareness and frequency of repeat bugs is causing some hesitation. Our goal is to present our findings to the company and adjust our roadmap based on our results.
