UX Research — WhatsApp Preference Testing
“What if I modify the existing interface and see if it works?”
What is WhatsApp?
WhatsApp is a cross-platform messaging and Voice over IP (VoIP) service that allows users to send text and voice messages, make voice and video calls, share images, documents, user locations, and other media. WhatsApp runs on mobile devices but is also accessible from desktop computers, as long as the user’s mobile device remains connected to the Internet while they use the desktop app.
This app provides convenience for all smartphone users. It is easy to download, it’s free, fast, and safe. Even people with very basic devices can use WhatsApp. All these conveniences then became its competitive advantage.
Considering its diverse features and convenience, I am truly interested to conduct a preference testing for this app. By modified the layout of an existing page that I think will solve the problem, then compare the modified ones with the existing interface.
Preference testing vs. A/B testing
To compare the system, we mostly heard a method called A/B testing.
In fact, Preference testing and A/B testing are not the same.
A / B testing requires us to have a basic understanding of what is “normal” for the user experience and we want to test alternative approaches that will drive improvement in the specified metrics. That is, it is conducted exclusively on the live site.
On the other hand, Preference testing can be conducted at any development process once we have a concept, wireframe, or other representation of experience to show study participants. Occasionally, organizations will perform both preference and A/B testing; if there are too many options to test using our optimization solution, we can choose to do preference testing to override the in or out options for testing on the live site.
Study Design
In this study, I’m using a preference test approach by putting two treatments of some kind of asset in front of respondents to see which one they prefer.
The questionnaire is divided into three parts:
1 — Demographic Questions
Including occupational background, age group, and devices used.
2 — Preference Test
By giving two options for each feature, where respondents must choose which one they prefer. With the feature examined as follows:
- Chats
- New Chat
- Calls
- Status
- Camera
3 — Preference Questions
Refer to SUPR-Q (Standardized User Experience Percentile Rank Questionnaire) survey questions with adjusted wordings to tailor better with context. Using a 5-points Likert scale, this series of questions assessing four attributes: Usability, Preference, Loyalty, and Appearance.
To compare the treatments, I used a within-subjects design where each person is going to see two variations. Along with a question on each feature where the respondent must choose one choice. To complete the study, there is a post-test questionnaire or preference questions after the respondent completes the preference test.
Treatments
There are several things I modified on each feature page, such as button replacement, rearrange the layout, resizing, or simply remove the unnecessary elements. I used Nielsen’s 10 design heuristic principle as a guideline to redesign the overall pages, by identifying problems first then looking for a basic idea of how a good design might look like on another similar platform.
Chats
“What if we separate group chats and personal chats?”
I feel like the current interface seems a bit clutter, where I can’t define which are the groups, and which are the personal chats. Would be much better if the groups and personal chats are separated. On the section heading, the user can also sort chat based on unread messages, and time received. Which gives the user an option to move the unread messages all to the top.
I also replaced the New Chat button from the top corner to the down corner, plus making the button bigger so it’s visible for users. The consideration behind moving the button to the bottom area is to make it more easily reachable by the thumb, so users don’t need to move their thumb to the top of the phone.
New Chat
Once users click the New Chat button, here’s the page they’ll come up to. All the contact lists here are sorted in alphabetical order. So, if the user wants to search a contact based on the alphabet, they should click one of the blue buttons on the right side, which for some people is a bit too small to click. For example, if I want to click R, sometimes it comes up to its surrounding alphabet, in this case, it’s either Q or S. Quite frustrating for people with a bigger finger.
Therefore, I modified the alphabet category that enables the user to show/hide the alphabet category. Not only to make it less clutter, but also accessible for wider users scope. But the negative side is the user must scroll down a bit to find the alphabet category they’re looking for.
Calls/Video Call
There’s no significant change I made on this page. Just like in the Chats page, here I replaced the New Call button from the top to the down corner and make it bigger. I also moved the All/Missed switch button under the Search bar.
Status
In My Status panel, you can see that there’s a camera button next to the type button. But at the same time, there’s also a “+” button on the profile picture. This might create confusion for some users, which button should I touch if I want to update status?
So, I decided to delete the camera button if the user doesn’t have a status yet. But when users have status updates, the camera button can also appear in case they want to make a new status.
I also add the day status (Today and Yesterday) on the right side of each profile on the Recent Updates panel, just to give the user information about the day they posted since the status only lasts for 24 hours.
Camera
WhatsApp enables the user to open the camera straight from the app then share the picture directly to contacts, groups, or My Status. Users can also pick a picture from the gallery that they can scroll through or click the picture icon to pick from the whole gallery.
Here I modified the gallery and make one of the pictures focus on the middle. Hopefully, it helps the user to easily guide them on which picture to share or post.
Subjects
I took a sample of 30 respondents using convenience sampling which represents six occupational backgrounds, four age groups, and two types of devices used by participants (iOS and Android).
Findings
As explained in the study design, the test was conducted in two parts. Preference test to see which option or interface that people prefer and they think will solve the problem and preference questions consist of eight questions using the SUPR-Q survey to assess the overall new design using four attributes.
Preference Test
1. Chats
Question: For the Chats feature, which interface do you prefer?
At first, when the respondents were still around 15 to 20, the respondents who chose the current interface dominated the pie chart. But once the number moving up until 30, there’s a slight difference between the two options. This indicates that almost half of the respondents aren’t agreed to separate personal and group chats. People still comfortable with current Chats where both personal and group chats are mixed up into one single category.
2. New Chat
Question: For the New Chat feature, which interface do you prefer?
There are 57% (17 respondents) agree with the contact list layout redesign. On the other hand, the rest still feel like the current alphabet buttons work well. Maybe we can also consider making the alphabet buttons bigger or create a new panel on the right side.
3. Calls
Question: For the Calls feature, which interface do you prefer?
About 80% of respondents agree with the new Calls interface. This means that people agree to move the New Call button and make it bigger to enhance accessibility by the thumb.
4. Status
Question: For the Status feature, which interface do you prefer?
More than three-quarters of respondents were satisfied with the new Status page layout. Indicates that day status is necessary on each profile, and the camera button should be removed to reduce confusion.
5. Camera
Question: For the Camera feature, which interface do you prefer?
Here’s the highest presentation among other assessed features. Up to 87% of respondents agree if one of the pictures from the gallery stands out in the middle. Which then hopefully will help the user to easily guide them which picture to share or post.
Preference Questions
The majority of respondents choose 4 (agree) for 7 questions, followed by 3 (Neutral) for 6 questions. I calculated the score based on the average response of each question in a particular attribute, then calculate using the same method for all responses from the 30 respondents. Then average the scores to 30.
Here are the findings for the Preference Questions of each attribute, along with the score of each attribute.
1. Usability
=(total responses from attribute 1 + total responses from attribute 2)/2
=(3,80 + 3,93)/2
Score: 3,87 out of 5
2. Preference
=(total responses from attribute 3 + total responses from attribute 4)/2
=(3,70 + 3,83)/2
Score: 3,77 out of 5
3. Loyalty
For question 5, since it has 10 scales, first I divided the score by half, then calculate using the same method as other attributes. This question basically asks customers one simple question about whether they would recommend the new interface to other people.
=((total responses from attribute 5 x 1/2) + total responses from attribute 6)/2
=((7,33/2) + 3,87)/2
=(3,67 + 3,87)/2
Score: 3,77 out of 5
From attribute 5, we can also get the Net Promoter Score (NPS) using the following formula.
Before calculating the Net Promoter Score, there are three indicators based on participant’s responses.
- Promoters: They are those users who range their willingness of recommendation to 9 or 10 and very satisfied with the kind of user experience we are offering them. They are the repeat users, who recommend your system to other potential users.
- Detractors: They are those users who range their willingness of recommendation from 0 to 6 and considered to be very unsatisfied. They are less likely to remain for long and denied to recommend your system to other potential buyers.
- Passives: They are those users whose behavior falls between promoters and detractors. They range their willingness of recommendation to 7 or 8 and moderately satisfied with the kind of user experience you are offering them. These kinds of users never spread any negative word-of-mouth, but are not enthusiastic enough to promote your proposed system.
Based on the formula, we can then calculate the Net Promoter Score as follows.
=(13,30% + 3,30%) - (0% + 0% + 3,3% + 0% + 3,3% + 20%)
=16,6% - 26,6%
=-10 (NPS is always shown in integer and not persentage)
Well, the result is quite surprising. But since majority of participants choose 8 (36,70%) and IF, let’s say, we also include 8 as a promoters, we will get:
=53,3% - 26,6%
=26,7
Any NPS score above 0 is “good”. It means that the users is more loyal than not. Anything above 20 is considered “favourable”. Bain & Co, the source of the NPS system, suggests that above 50 is excellent, and above 80 is world class.
4. Appearance
=(total responses from attribute 7 + total responses from attribute 8)/2
=(3,93 + 3,90)/2
Score: 3,92 out of 5
Calculation
The score for Preference Questions must be reached using the following formula:
- Average the scores for questions 1,2,3,4,6,7, and 8.
- Then add half the score for question 5.
=((3,80 + 3,93 + 3,7 + 3,83 + 3,87 + 3,93 + 3,90)/7) +(7,33/2)
=3,85 + 3,67
=7,52
Based on these eight questions, after calculated the 30 responses, the SUPR-Q score is 7,52. It’s a relatively good percentile score considering 5 is an average score. From this score, we can also compare with the industry standard, in this context using communication & messenger industries.
Using the score on each attribute, we can also assess one attribute compare to another. For example, the Usability score is higher compared to Loyalty, which means that respondents are more confident with the usability side on new pages rather than their intention to use and recommend the new page. We can also decide the design priority for each attribute and see which one needs more effort to be developed further.
On the other hand, if we take a look at the Net Promoter Score, it shows a poor result (below 0), but if we also include response 8 as a promoter, then the result is considered “favourable” (above 20). Net Promoter Score gives us an opportunity to improve the system along with the user experience. Thus, Net Promoter Score provides us with the privilege to focus on the ‘Why’ behind the ‘What’ for both promoters and detractors to augment the health of the user experience.
References
About WhatsApp. Retrieved from https://www.whatsapp.com/about/.
A Net Promoter Score Form (NPS) in 5 easy steps. Netigate. Retrieved from https://www.netigate.net/articles/customer-satisfaction/net-promoter-score-form-5-easy-stages/
Nielsen, J. (2018). Between-Subjects vs. Within-Subjects Study Design. Nielsen Norman Group. Retrieved from https://www.nngroup.com/articles/between-within-subjects/
Travis, D. (2019). Preference Testing is Not A/B Testing. Userfocus. Retrieved from https://www.userfocus.co.uk/articles/preference-testing-is-not-A-B-testing.html
What is a good Net Promoter Score (NPS)?. Perceptive. Retrieved from https://www.customermonitor.com/
