User Testing Case Study: Google Flights
Introduction
Google Flights is an online flight booking search service that facilitates the purchase of airline tickets through third-party suppliers. This platform allows open-ended searches based on criteria other than the destination; for example, a user may search for flights within a range of times and a budget and be offered various destination choices.
Alternatively, the user can select a destination, and Google Flights will calculate every price for each day of the next 12 months, visualized in a graph or table. This allows users to easily spot the cheapest date to fly to the destination.
The systems that I’d like to assess here is the core function of the website: Search flight and filter feature. Google Flights also allows user to set notification of the price change through email, the user just needs to login to the Gmail account and set the notification button.
To get specific insights about how the system works, I evaluate the website using scenarios in which participant try to accomplish specific tasks related to the website as follows:
- Choose one-way economy flights to a specific destination on specific dates.
- Utilize the filter feature to specify search results.
- Use track price features so the user can get notified through email once there’s a price change.
These specific tasks are meant to give understandings about how their target user uses the system. I also want to gain insights into where the design works well and where it needs better improvement. By watching a representative user trying to complete a task using the product.
Before I started to test the website with the participant, I asked them several pre-test questions about Google Flights, the last trip he planned, the most important information to plan a trip, and how often does he travel. Then after the testing, I use SUS post-test questionnaire to see how the participant feels about the website in quantifiable metrics.
Methods
Before I designed the test, I need to define Google Flights’ intended user. Which can be divided into expertise, characteristics, behavior, and attitude.
Therefore, I decided to choose the participant with the following profile.
To get the participant, I used convenience sampling, which involved my friend’s father that fits the profile, then explained to him the brief purpose of this testing and what he will get through during the test.
Instruments
I specifically use formative testing, a common user test that comes across UX research which evaluates and identifies what problems to fix. Formative tests are generally performed during the design process to find bugs to fix in the next iteration of the design. The general form of a formative test is to have representative users perform a set of tasks that have defined, then see how they do things and hear what they say. From this test, we can see where they struggle, and identify parts of the design that cause problems.
The testing consists of three phases.
1 — Pre-test interview
I asked the participant about Google Flights to see what he knows about the website, the last trip he planned, the most important information to plan a trip, and how often does he travel.
2 — Test
Conducting three scenarios in which the participant must complete each task, assess how he did things and what he reacts to the process (thinking-out-loud).
3 — Post-test
I used a SUS questionnaire* to see how the participant feels about the website in quantifiable metrics.
*SUS questionnaire: The System Usability Scale (SUS) provides a reliable tool for measuring usability. It consists of a 10-items questionnaire with five response options for respondents; from Strongly agree to Strongly disagree.
All the details of the questions and results can be seen in the appendix.
Analysis Methods
The goal of this testing is to see if the user who frequently uses online bookings can use Google Flights to search and utilize filters. To analyze the user test, I collate the baseline statistics. Such as observing which tasks have succeeded or failed, what kind of errors were made and how many errors were made. I also compare how long it takes for the participant to complete each task.
After collating the statistics, I review all the critical incidents that occurred. Observe where the breakdown occurred, where did the participant run into trouble, and why. It’s also important to collate the results from the questionnaire as well, then interpret the responses during the debrief and interview of the test session.
Addressing Bias
During all three tests, I tried as much to avoid biases. By giving honest feedback and notice if the answer seemed forced. I also have tests run by disinterested 3rd parties or someone who does not involve in the design process and don’t have a stake in the outcome of the test. It’s critical to be aware of potential biases that both me and participants bring to the user test. So that the overall tests can create the best insights possible.
Findings and Recommendations
Summary Results
To get brief insights about the findings, I’d like to present the following non-binary outcomes to see how a participant in a particular user test did on each of the tasks in this particular test.
I ranked the task completion into three levels where:
- 3=Succeed with no help
- 2=Struggle with some help or some hints
- 1=Failed
Key Findings
On each finding, I added the severity level from 1 to 4. Severity was judged based on a five-point rating scale taken from Nielsen’s “Heuristic Evaluation” chapter, Table 2.3 (1994):
- Cosmetic problem — no real usability impact;
- Minor usability problem — fix if there is time;
- Major usability problem — important to fix; and
- Usability catastrophe — imperative to fix.
Task 1: Choose a one-way economy flight from Jakarta to Gardermoen Airport, Oslo for October 13, 2020.
Completion time: 1 minute 24 seconds
Finding 1: Hard to switch from Round trip to One-way trip.
Severity: 2/4
At first, it was easy for the participant to find the destination airport since the departure airport is already set by default based on the current location. Using automatic search results, once he typed “garderm”, the result already appeared on suggestion. The trouble begins when the participant chooses the date of departure. He first observed the date, then realized that it was not yet switched to a One-way trip.
After realizing he failed to click the “Done” button, then he clicks the Departure date column again until realized there is a Round trip choice that can be switched to One-way. Once the “Done” button turned blue, he finally able to click the button and the search results immediately appeared.
Task 2: You only want to use either Qatar Airways or Emirates to get there, with a price under IDR 12.000.000, and arrive at least at 4 PM local time.
Completion time: 3 minutes 38 seconds
Finding 1: Confused to set detailed filters for different criteria.
Severity: 3/4
The participant quickly reacts to the Sort by button, then he tried to click the button to see if it works from there. After that, he clicks the Price choice, then scrolls up and down to see if the results change.
It took him around 30 seconds to realize that the search results are sorted by the cheapest price to the most expensive. Then scrolling up and down again, which indicates his confusion to find another filter.
Finding 2: Participant tries all criterion to see how it works.
Severity: 1/4
There are many other filter criteria offered by Google Flights. Such as Bags where user can set how many luggage they will carry. Stops to set up to two stops to the destination. Connecting airports which allow the user to choose at which airport to connect from departure airport to destination. The last one is More option where the user can set a filter based on flight duration and separate tickets.
It’s actually not a severe problem if the participant wants to know what every filter criterion looks like. But would be better if the participant can straight to the filter section that meets the criteria for task 2.
Finding 3: Struggle to filter arrival time.
Severity: 2/4
Again, he thought that the Sort by feature will work to set arrival time. Until he realized that it changes the search results from the earliest arrival to the last. He clicks several buttons once again until he finally realized the Times option, then he changed the arrival time to 16:00 following the task instruction.
Task 3: You want to keep up to date with the price. Track prices for this trip and receive price updates to your email.
Completion time: 2 minutes 47 seconds
This task has the highest severity. The participant can’t accomplish a task without guidance or hints. Therefore, I told the participant to try to sign into his Gmail account and refresh the Google Flights page.
Finding 1: Can’t find the Track prices button.
Severity: 4/4
As seen on this page, there is no Track prices button if the user hasn’t signed into the Gmail account. At first, the participant scrolls up and down several times to make sure there is a Track prices button hidden somewhere.
After he finished and say “nothing”, I told him to open a new tab, then sign into Gmail. Once signed in, I guide the participant to try refreshing the Google Flights page. But still, he didn’t quickly recognize the Track prices button on the left side of the page.
At first, he clicks the Track price text to see if there is any reaction. Then he recognized the switch button on the side. After switch the Track prices, the page looks like this.
We can see that there is a notification on the bottom. Allows the user to review all the tracked prices. To undo the tracking prices, the user can switch again the button.
Findings unrelated to the tasks
The feedback card pop-up on the corner page once users click “View all” on the Tracking prices section. I’m not really sure why this happened, it can be either Google Flights can detect if the user has completed all features available, or simply wait for the best moment to pop-up.
These are the choices the participant made and his written feedback about the website.
This pop-up surprisingly gives me additional insights that were never planned. Such as why he visits Google Flights, his future vacation plan, when, and for what purpose. We can see that participant is neutral about the satisfaction, he is neither satisfied nor dissatisfied with the experience.
From the post-test questionnaire, the participant also expresses a lack of satisfaction with how the website works. He said that the website is unnecessarily complex and needs guidance from a technical person to use the whole system. Therefore, he felt that he still needs a bit of learning to get going on what every feature does.
Recommendations
Here are recommendations based on these findings, using comparative analysis from other similar websites.
1. For Round trip, One-way, or Multi-city option
In Google Flights, we can see that the user must click the Round trip button to break down another option. It’s pretty hard if the user wants to choose a One-way or Multi-city trip as if there is no other option.
Before
After
Instead, Google Flights can display all options without hiding them all. Like the one that I tried to redesign here, the option button for One-way, Round trip, and Multi-city are all displayed. Makes it easier for user to identify option available and choose the option in just one click.
2. Put all kinds of Filters on the page
Currently, Google Flights are showcasing all filters hidden based on category, with labels represent each category as shown in Figure 14.
It does make it less clutter compared to other online booking websites. But if we see it from functionality perspectives, it gives user frustration especially to differentiate which button to sort the results, and which button to filter results.
Before
After
For better experience and reduce confusion, Google Flights can display all filters on the left side. Users can also hide each filter category by clicking the small triangle button if they think the category is unnecessary.
3. Spend less time scrolling the desired time
If the user wants to set departure or arrival, they must drag scroll the button to the left or right. The same interaction also applied to set price and stopover duration.
These are considered cumbersome for some people. Therefore, it would be much better if Google Flights use the checkboxes with a range number of each option. The user just needs to check the desired departure or arrival time, then the results will adjust immediately. Not only reduce confusion, but it will also reduce the time frame for the user to accomplish the task.
4. Display Track prices switch button without Gmail sign in
If we look back to Figure 6, when the user hasn’t signed into Gmail, the Track prices switch button is not available. Might be confusing, especially for new user and they will think that there is actually no option at all to track price changes.
What if Google Flights enable the Track prices to switch button without sign in? So, after the user switches the button, there will be a notification to insert the Gmail address. After the user inserts the address and proceed, the user can directly get a notification through their Gmail account.
Limitations
1. Include more participant from different populations
The output and analysis would be much more diverse if I also have the chance to get each population represented. Hopefully, I can compare how each participant is doing, and where they find trouble during the test.
2. Try on different devices
Google Flights system and interface might be different on other devices. Especially in a smartphone when the information is merged and reduced to enhance efficiency. Thus, the flow and interface may as well different. Also consider bandwidth capacity, especially for slower connectivity, the website may load longer.
3. Try as much not to give any hint during the test and let participant figure out what to do
As an observer, we should not overlook and not give responses immediately once the participant finds trouble during the test. I gave several hints for task 3 since it’s the most difficult task compared to the other two. Instead, we must observe what they do and how they react to the problem. Give the participant room to explore the whole page and system, let them figure out how to accomplish the task by themselves.
Conclusion
This user test report of Google Flights revealed that there are a number of issues that need to be fixed in order to improve user experience. Conclusions are made based on the tasks done by the participant. I also include the severity ratings so we can know at what urgency we need to fix the problem.
Here are some changes that should be made for the next version of the website:
1. Redesign the Round trip, One-way, and Multi-City button.
Make these options visible in just one click.
2. Make the Filters visible by displaying on the side of the page.
With the show and hide options so the user can hide them if they think the categories are unnecessary.
3. Instead of using a drag-scroll to choose the desired option would be much better if Google Flights use the checkboxes with a range number of each option.
4. Display the Track prices switch button without Gmail sign in and enable the user to insert an email address after they switch the Track prices button.
5. Participants expressed a lack of satisfaction with how the website works.
Therefore, he felt that he still need a bit of learning to get going on what every feature does.
References
Neil, T. (n.d.). 6 Tips for a Great Flex UX: Part 5. Designing Web Interfaces. Retrieved from https://designingwebinterfaces.com/6tipsforagreatflexuxpart5.
Nielsen, J. (1994). Heuristic Evaluation. In J. Nielsen. & R. L. Mack (Eds.) Usability Inspection Methods. New York, NY: John Wiley & Sons.
Nielsen, J. (n.d.). 10 Heuristics for User Interface Design. Nielsen Norman Group. Retrieved from https://www.nngroup.com/articles/tenusabilityheuristics/.
Appendix
Pre-test Interview
- What do you know about Google Flights?
It’s kind of a website to search flights, so there’s like filters and you can find information about cheap flights and other criteria.
2. What and when was the last trip you planned?
It was in November 2019 to Pekanbaru for business trip.
3. What do you think is the most important information to plan a trip?
Of course, the price, destination, and cheapest price per date.
4. How often do you travel a year?
Around five times a year, the purpose can be either for business trip or family vacation.
Usability Testing
Some parts on the video are removed to protect the participant’s private information.
Post-test Questionnaire
