Introduction to mobile analytics: Informing your test strategy
Mobile analytics are a powerful source of data to inform test strategies. Several kinds of analytics, or data, can be gathered about the users of an application. To list just a few: the numbers of new versus returning users, revenue generated from the app, user retention, number of downloads, geolocation, and custom in-app behavior events.
In addition to these readily quantifiable types of data, customer feedback about an application is useful, since it is the easiest way to learn about specific areas of frustration or joy that customers experience with your application. Feedback might be quantitative, such as a star-based rating on the Apple or Play store. Equally valuable feedback might be qualitative, such as the inclusion of your product in a blog review on tech-based websites, interactions with your customer service team, or even users' descriptions of their experiences with your app on social media such as Facebook, Twitter, and Instagram.
Where to find mobile analytics
Mobile analytics are everywhere. Many can be gathered for free. For example, every app developer or product manager can see the basic information about the number of downloads, amount of 5-star ratings, and customer experience stories on Apple and Play stores. Paid tools, or free-to-paid models such as Bitly, Amazon Pinpoint, or Firebase, help testers gather data from points within the mobile customer "journey" by using different methods to track customer behavior within an app once it has been downloaded.
What is the mobile customer journey? What kinds of information are measured and revealed throughout the journey, and more importantly, how can that help testers? Here are some pieces of information that can be collected about the mobile customer journey:
- How many first opens your app has
- Time spent in each part of your app (popular or engaging areas)
- User demographics given upon registration: interests, age, gender, location, language settings
- In-app purchases
- Total amount of time active in-app for each session
- Total number of users active in the app at a given time
Basic mobile analytics data and user responses can be gathered about competing applications via the Apple or Play stores as well, helping marketing, sales, product, and tech personnel better understand, create, and test the product based on consumer needs.
Gathering and reporting analytics data
Once testers have access to sources of mobile analytics data, they can review and synthesize the data in several ways. It is recommended that testers do the following to maximize the usefulness of their interaction with mobile data:
- Select a time period short enough to be manageable but long enough to gain insight about the specific feature/area you want to understand. An example might be to view data the week before and the week after the release of a new feature, or to select the two weeks of recent performance on an area of the mobile journey that is underperforming.
- Decide what data is valuable to know from that time period. For example, do you want to know customer opinions about app interactions, or are you more concerned with the number of downloads? Perhaps your biggest concern is how your customers respond to a specific feature of your application, such as the time it takes to download content or the way that certain UI features inform interactions.
- Organize multiple sources of analytics into one place (source of truth) so that broad patterns, if they exist, can be determined. For example:
- Customer opinions about an application can be reported by analyzing the number of 1-, 2-, 3-, 4-, or 5-star ratings over a given time, and placing these into a graph for easy comparison and reporting.
- Customer experiences as reported in the app store ratings, customer service feedback, or social media can be used to create infographics or word clouds to easily communicate the main takeaways about the customer experience to C-level, business development, and product/technical teams.
- Customer journey points might be plotted along a graph to note what parts of the app engage the most users, or perhaps to represent areas of risk in comparison to levels of usage and levels of testing.
Targeted test strategies from analytics data
The nature of mobile testing is very complicated because of mobile fragmentation, resulting in difficulty predicting how an application will behave on any given device or operating system.
Testers can use mobile analytics to develop specific and focused test strategies to mitigate high-risk areas of their mobile applications.
A/B test plans are often used as a first technique to try to gain information about apps.
- Use analytics data to create a beta version of a new feature or a redesign a low-performing feature. This can include testing headlines, UI design choices, or usability of features.
- Release beta to pre-determined customers. Gather their feedback early, and implement changes based on feedback. Test again.
- Release the feature for general audience; compare analytics from users of new feature to users of the old version. How do the analytics compare? Use these A/B test results, combined with analytics, to create a feedback loop and build testing into your mobile application.
Customer journey testing strategies are another key component.
- Use session-based and exploratory test strategies to investigate areas of the customer journey where there are complaints/difficulties/low metrics.
- Use results of exploratory testing to recommend testing the business scenarios/customer journey points that yield the highest value, such as areas of high customer interaction or areas that have a high-risk level.
Free analytics, such as app downloads and customer feedback, can be gained about products built by your competitors. Compare the journey of your product with the journey of another, similar product that has successful business results.
- How are the customer journey touchpoints in competing problems similar or different?
- What are customers of a competing product praising about that product?
- What do customers of a competing product find irritating about that product?
- Use these comparisons of your product to competing products to develop tests around customer journey touchpoints.
Performance testing, which includes data on loading/performance times, can be gathered through various monitoring tools in order to learn about your customers and improve their experience.
- Examine analytics about customer bounces and compare to page load times.
- Ask yourself: Are customers leaving the app because of poor performance?
- Gather analytics about the kinds of devices most used by your customers.
- Get live devices and test how performant the app is, especially in areas where customer bounces are high.
- Test the devices online and offline and determine if there are issues for customers related to how they are using the app (for example, in the subway with a poor signal or no signal at all, versus with Wi-Fi or 4G).
Additional resources
As more people rely on their phones for carrying out personal communications, doing shopping, making shopping comparisons, and juggling their busy work lives, the rates of app usage steadily increase, providing a constant stream of rich data that companies and development teams can use to make decisions. Testers who become familiar with the mobile customer journey can make targeted recommendations about what to test. To keep up with what's happening in the mobile landscape, testers may find these resources helpful: