While it’s great to innovate, UXers and designers can get too creative sometimes. We need to validate our ideas all the time. Innovate, Validate, Iterate! Research helps us make sure we are on the right track, helping us and the business making informed decisions.
SCROLL DOWN
A/B navigation in progress.
Aim.
During the early stages of The Daily Telegraph navigation redesign, a few concepts needed to be validated and compared against each other and the baseline (the existing nav).
What we did.
To make sure we had a good representative sample that captured success rates as well as the time spent per task, I organised a quantitative, task-based remote test.
For this, I turned the designs for two new concepts into clickable prototypes, then set two separate task-based tests in Loop11 and divided the participant traffic equally between the tests using a redirect page.
All users were taken through the exact same tasks on the A, B and C (existing) designs. The C design test results became the baseline. The A and B concepts’ results were compared to each other, and also the baseline success rate and time spent per task.
The winning concept undertook an in-lab usability test with 5 desktop and 5 mobile users to catch any usability issues that could be fixed to further improve the solution.
about
research.
Sample size: 250
Research type: Qualitative & Quantitative
Research Techniques: Remote, task-based A/B test
Tools used: Loop11
Subscription process.
Aim.
As part of a redesign of all our Mastheads subscription page, we wanted to ensure it was easy for users to compare the value for money of our subscription offers. There was an assumption that the existing pricing was somehow confusing and that the benefits of each offer weren’t clear to the users.
What we did.
We organised some qualitative remote usability testing that proved the assumption that the pricing was confusing. We were able to report the findings to the marketing team, supported by video snippets showing users struggling with the prices for about 10 minutes, with several even getting their calculators out to make sense of the pricing.
We also organised quantitative, task-based A/B testing to analyse the best possible layout for the offers. This involved us asking users to find a particular offer as we captured the fail vs success rates, as well as time spent per task on each design.
As a result of the tests we ran, we made recommendations and informed decisions on the “offer” layouts and "calls to action".
about
research.
Sample size: 58 participants (15 users for qualitative test and 43 users for quantitative test)
Research type: Qualitative & Quantitative
Research Techniques: Remote moderated and unmoderated, task-based tests
Tools used: Loop11, UserTesting
Panel size: 3 panels with 1000 users in each, rebuilt every 6 months
Tools used: SurveyMonkey, custom invitation
The panel used for: any research including remote and in-lab tests, surveys etc.
Breach page, paywall.
Aim.
When The Daily Telegraph and The Australian first introduced a paywall we wanted to see what users’ initial reaction was about the breach message.
What we did.
Knowing that users only spend about 5 seconds on a page before they make a decision to leave or stay on it, we created a 5 seconds test. We placed different breach messages in front of users for 5 seconds only and asked them to tell us what they remembered about the page, what options they had there and why they thought they might have been shown it.
The results we gathered helped us communicate to the marketing team the importance of the right messages, options and call to actions on the page.
about
research.
Sample size: 10 participants
Research type: Qualitative
Research Techniques: 5 seconds test, in-lab one-on-one test
Tools used: Loop11 to make a screen disappear after 5 seconds
Font study.
Aim.
How would scannability and findability be affected if we changed our headlines’ font from lowercase to uppercase on The Daily Telegraph and Herald Sun home pages?
What we did.
To make sure we had a good representative sample that captured success rates as well as time spent per task, I organised for a quantitative, task-based remote A/B test.
The participants were divided between the two concepts and were given a set of tasks to check for scannability and readability of the two fonts.
​
We asked the users to find a story on a homepage that contained the word ‘strawberry’ and the word ‘watermelon’, capturing the fail rate and the time spent per task as users scanned for the words.
We then asked them to find a story about postnatal depression, capturing the fail rate and the time spent per task as users read through the story snippets. For this test the participants actually needed to read - not scan - the headlines, as the story snippet did not actually contain the words ‘postnatal depression’.
about
research.
Sample size: 60 participants for each concept
Research type: Qualitative & Quantitative
Research Techniques: Remote, task-based test
Tools used: Loop11
Private research panel.
Aim.
What we did.
about
panels.
I created an invitation for our Daily Telegraph, The Australian and Herald Sun users to opt-in for research activities with us and placed it on our homepages.
Soon after we had accurate, pre-screened panels that we used for all sort of research activities. Having the panels ensured we captured accurate data from our real users and reduced the in-lab research cost by 50%
Building and using the panels involved working closely with the legal department to ensure we don’t breach privacy policy and comply with all the legal requirements while contacting our users, inviting them in and paying them for attendance. It also involved me in managing the panels, keeping them up to date and teaching the UX team members to wisely use the panels for their research activities.
Testing with users recruited by an agency was proving a little inaccurate in previous testing sessions. To avoid a ‘pretend user’ and ‘professional testers’ scenario, I decided to build our very own research panels for The Australian, The Daily Telegraph and Herald Sun.