Role
I was responsible for problem framing, conducting user research, and the design of the new code review experience. Our team’s focus was to increase user adoption so that we could deprecate the legacy code review experience.
Timeline
Jan 2020 to Sept 2020
Overview
Bitbucket has a collection of experiences built on top of its code hosting core. Code review is one of those experiences that supports developers through their end to end journey from picking up a piece of work to shipping it.
Code review flow
Code
Review
Merge
Code review is a collaborative framework that supports the back and forth feedback loop between teammates to effectively get code reviewed and merged into the main codebase.
Bitbucket recently shipped a new Code review experience. To help teams reduce maintenance and focus on new features, it was a big team goal to deprecate the legacy Code review experience in favor of the new one. The team had a couple of obstacles in the way of deprecating the old experience.
Build up feature parity
Some users rely on features that aren’t available in the new experience. We don’t want to alienate these users by forcing them to switch before we provide alternatives to the features they rely on.
Stalled adoption at 70%
What can we do to motivate users to switchover to the new experience? We need to consider what are blocking users from switching over to the new experience.
Increasing user feedback
We needed to start iterating on some of the features that we have been building. Even though many of these features are relatively new, we can see that there are some problems to address.
To give us a basis for where we started, we monitored quantitative and qualitative data. On the metrics side, we looked at user adoption of the new experience. We also had a code review specific feedback collector which we reviewed as a team in a monthly cadence. We found patterns and organized the themes based on a graph with axis of impact to effort.
Addressing feature parity
Side-by-side is a viewing mode that is available amongst competitors and also available on our old experience. We needed to introduce it back into the new experience to win the trust of users who used that experience day to day.
Loading files individually is a popular way of reviewing code from Bitbucket Server (another version of Bitbucket). Many teams prefer this way of reviewing code because it allows users to focus on one file at a time. Although the concept of viewing a file one by one is new to Bitbucket Cloud, the old experience was able to view large pull requests without performance issues. The new experience is built on a different stack and wasn’t intended to be used to view extremely large pull requests.
Increasing adoption
In addition to matching features from the old experience, we also needed to provide some additional carrots to motivate users to switch over to the new experience. The new experience had to be better than the old experience. One popular request that hasn’t been available on Bitbucket Cloud was the ability to signal when a pull request has been reviewed but not approved.
Iterating on the new experience
When pull requests have gone through multiple revisions, it’s difficult to understand what has been addressed. Because the new experience has a less verbose activity feed, users began to struggle to understand what had recently changed. To help address this, we introduced filtering files by comments. This way, users can focus on the files that have active comments on them.
Research
We facilitated a couple of different rounds of research throughout this project. They mainly fell into two categories: problem framing to understand the user problem and concept testing to understand if our solutions would address the user problems.
Problem framing
Problem framing user interviews helped us discover and validate user goals, top tasks, and problem areas. As an example, when we were iterating on the side file tree view, our problem framing research gave us the information to synthesize the following:
User goals
- I want to see an overview of what files have changed
- I want to easily navigate to a specific file
- I want to see meta data on files
Top tasks
- Get an overview of what files have changed
- Prioritizing first files to review
- Navigate to specific files
Insights
- Mismatched mental model
- Importance of spatial understanding
- Difference in code review sizes and screen sizes
Concept testing
We went into concept testing with a lot more information from problem framing. Because we wanted these tests to help us make decisions, we recruited from both users who have expressed an interest in the feature we were building and those who were less biased. We wanted to get perspectives from both pools to make a more informed decision.
I’m stating my intention that I’m still working on this. I’m not really ready for it to be merged and then the system will enforce that constraint and prevent merging it. They know that the ball’s in my court.
Participant
Impact
- The new code review experience is now rolled out to all users which brings all of our users to the same experience. This means one set of functionality for users to learn.
- From a velocity perspective, having one experience makes shipping improvements easier and faster because there’s only one code base and one set of documentation to maintain.
- In the last year since we rolled out the new experience, we’ve seen impacts at a higher level. We grew our users by 5% more than expected and our annual recurring revenue grew to 52% more than expected.
