Mobile Product Performance Testing

Performance testing is used to verify that an app or web page will display quickly to the user and will continue to function as the number of users increases to peak loads. Performance is an important consideration for mobile applications because the connection speed of users is often slower and more variable for mobile users than desktop users. Surveys have shown that users will often stop using applications or web sites that load slowly.

This article was developed as part of the Mobile Application Development Program. See our general guidelines to testing article for more resources on mobile product testing.

Performance testing will typically consider the following items:

• Performance when device has low battery
• Performance and download speeds on a variety of network speeds (e.g. 3G, 4G LTE)
• Performance and potential lagging on a variety of mobile devices and OS versions
• Performance when disconnected from network
• Performance under heavy load by a large number of users
• Battery consumption
• Response time
• Resource utilization (memory and processor)
• Binary size of native applications

Government Guidance

• Executive Order on Improving Customer Service in Government
• OMB Fast Track Process for Collecting Service Delivery Feedback
• OMB Guidance on Plain Language Writing Act
• OMB Memo on Information Collection under the Paperwork Reduction Act

Resources Available

Listed below are resources available that further describe and conduct performance testing. These services/companies or websites are offered as a sample of what is currently available for performance testing and do not indicate an endorsement of them or their products and/or services.

Web Resources

• Mobile Performance Testing In The Wild (recorded vendor webinar, registration required) – Discusses strategies and tools for mobile performance testing. Makes the point that mobile performance testing is very similar to conventional web performance testing.
• Mobile Performance Testing Overall Analysis Whitepaper – Comprehensive discussion of mobile performance testing techniques.
• Smashing Magazine: How To Make Your Websites Faster On Mobile Devices – Provides valuable statistics regarding the importance of performance on mobile, in addition the development techniques to improve performance.
• Smashing Magazine: Seven Guidelines For Designing High-Performance Mobile User Experiences – Explains performance in the overall context of mobile application success.
• Testing Performance of Mobile Apps – Provides detailed approach to mobile app performance testing.
• What Users Want From Mobile (requires registration) – Survey results that describe user expectations. For example, 71% of users expected mobile web sites to load as quickly as desktop web sites.

Testing Services

These services/companies or web sites are offered as a sample of what is currently available and do not indicate an endorsement of them or their products and/or services.

• Akamai Mobitest – A free mobile web performance tool from Akamai.
• Infostretch – Provide performance testing of mobile applications.
• mob4hire – Similar to uTest, provides “crowdsourcing” testing through third-party testers.
• uTest – Paid “crowdsourcing” test service that manages load testing for web and mobile applications via third-party testers.

Testing Tools

These tools are installed locally for mobile performance testing.

• Apache JMeter – Open source tool for performance testing web applications, including Mobile Web.
• Aurora Software Quadrant Standard Edition – Free Android app to benchmark the performance of Android mobile devices. This may be useful when testing a mobile app on a limited number of real devices, in order to determine how the app will perform on other devices with similar resources.
• Chrome Developer Tools – These tools are bundled in the Chrome Browser. The Network Panel page explains how to use Chrome Developer Tools to gather performance data about your mobile web page.
• NeoLoad by Neotys – Load and stress testing solution for mobile web and native mobile apps.
• SOASTA CloudTest Lite – Free downloadable tool for automated performance testing.
• Unicorn– Unicorn is an HTTP server for Rack applications
• Web Analytics – Services such as Google Analytics and Yahoo Web Analytics can be used to track mobile web site performance.

Test Plans/Checklists available on GitHub

The Mobile Code Sharing Catalog has test plans or cases or checklists that have been uploaded to GitHub and are available as samples and/or for use.

Ryan Day, GSA, and Ben Weaver, USDA, contributed to this post.