Mobile Application Validation: More than Manual and Automated Traditional Testing Activities

May 1, 2019

Eduardo Rodrigues

"Keep in mind that most mobile app failures will be found while performing field testing activities."

Developing and validating mobile applications are huge challenges due to mobile operating systems brand variations, including its versions, screen sizes, mobile manufacturers, user customizations, and intermittent mobile networks that significantly disrupt application communication with its servers and services. However, most organizations only validate their applications with traditional manual and automated testing activities, leaving aside one of the most important testing activities for mobile apps: field testing.


The Mobile Platform Challenges

Chris Karnacki and Rachel Obstler during the 2016 Keynote Webinar [1], stated that in 2015 there were about 24,000 variations of Android devices on the market, highlighting the need for automation to validate functionalities, business rules, and requirements on as many mobile devices as possible. Obviously, this validation process must be set within a proper testing strategy to cover the largest number of mobile devices that will be used by end users. Additionally, in 2015 about 227 billion downloads were made in online stores and only 16% of app users were willing to give more than one chance for a low-quality application.

If the user experience with the mobile app is not good, the user will simply delete the app from the mobile device and will look for alternatives, such as using the service on other platforms (WEB) or choosing a similar app (from competitor), etc. But be sure that before doing so, the user will register his dissatisfaction in the online store about the app and the app's rating will start to decrease, driving the organization's reputation down. These comments, in addition to the rating, influence the decision of other users to download the app.

There are several factors that may be associated with poor quality systems, as described in the article titled "The system works in the development environment, but once it is published, the users start to find failures. Is this familiar to you?" [2]. However, while manual and automated traditional testing activities are imperative in the mobile application validation process, it is necessary to go beyond these traditional activities to prevent users from finding app failures.


The Testing Environment in Organizations

The application validation process in the organizations follows much the same ritual. Once the app is released by the development team, quality analysts perform a series of manual and / or automated tests on emulators, simulators or real devices, where the first two being an inefficient approach (or not at all).

It is important to note that for political or bureaucratic organizational reasons, many analysts, and perhaps many developers, perform development and testing activities on their own personal mobile devices. At other times, they use company devices that are generally not being used on other projects or by other teams, destroying the testing strategy, since mobile devices selection should be based on those that are being most used by users or the most widespread ones on the market ("how to select these devices is for an upcoming article").

This approach or strategy is well represented in the "short blanket" story. Whenever the team covers the head, the feet are uncovered and vice-versa. In other words, while the application works on some models / operating systems range, it stops working on others and vice-versa.

The image below is an example of a testing environment for mobile application validation, using real devices connected to multiple servers, and accessed through interface tools for modeling and executing automated scripts or manual testing.

However, whether these test environments are perfect or not, testing activities in organizations are performed in controlled environments, because in laboratories everything works perfectly ("or should work"). The testing environment found in organizations is not the same of end users. If something is not working in the organization, there is an IT team that will make it work. Generally, tools are working fine, corporate internet is fast, devices are up to date, SIM cards have a data plan, and if the mobile phone network is bad in a particular lab location, the team moves to a better environment, "with more network signal strength". This scenario is very far from the real environment faced by end users.


The User Environment

According to the Brazilian Telecommunications Infrastructure Association (ABRINTEL), it would be necessary to triple the number of network antennas to meet the demand for mobile device connectivity in the city of Sao Paulo [3]. This demand is associated with areas that do not have carrier signal coverage, antennas with outdated technologies, overloaded antennas in neighborhoods or areas of large crowds such as football stadiums, concerts, shopping malls, airports, heavily populated neighborhoods, etc.

This study corroborates the image below. This map is a representation of TIM's 4G mobile carrier coverage areas [4] and its services in eastern Sao Paulo in 2018. 2G and 3G coverage follow the same pattern. In addition to the arrangement of the antennas on the map (ex. a single antenna to serve a large area), also note the various geographic areas, with large extensions, with no coverage or partial coverage of the mobile network signal:

Some regions, even containing a high concentration of antennas, make it difficult for electromagnetic waves to spread due to the number of hills, buildings, lakes, trees, etc.

In  this context, when the user opens the application on the mobile device and there is no communication with the servers due to infrastructure problems on the carrier's network, the application will experience a number of failures if they have not been addressed by the development team and also not identified by the quality and software testing team.

Last year I had the opportunity to watch a football game at the stadium. My anxiety made me arrive at the stadium one and a half hour before the match began. As soon as I settled in, I immediately tried to take a selfie and publish it on social networks. There were about 15,000 people so far, and this simple task seemed impossible to be accomplished. When opening the app, the "loading" message remained on the screen for about 30 seconds. However, after much sacrifice, I was able to publish the photo. As an Engineer, I soon realized that latency was high due to the large concentration of people and their devices attached on the same base transceiver station (BTS) - mobile phone antenna.

There were about 40,000 people in the stadium when the match began. Meanwhile, a great friend, who was by my side, was trying to post his selfie on social networks while his device had 100% 4G signal. The app started to present "weird" behaviors. When opening the application, for example, the "loading" icon remained on the screen for minutes. However, when touching anywhere on the screen, a white image was presented with absolutely no information. At this point, the app has slowed his device, practically crashing. His first comment was: "- This app sucks after the last update."

Even though the 4G signal was 100%, it was evident that there was no communication between the application and its servers, caused by problems with the carrier's infrastructure. There was a lot of packet loss due to the large number of people connected to the same BTS, inside and outside the stadium, and consequently the application did not work as expected, the process was looping.

However, the development team should have addressed these issues by displaying an alert message to the user after "x" seconds, saving content locally and synchronizing it when communicating with the servers, pinging before calling a service, checking signal strength before making a transaction, rolling back the database if something goes wrong, and so on.

On another occasion, while the user was traveling on public roads using a specific application, the mobile device automatically migrated to another BTS, keeping the application session opened at the previous BTS. This was due to the device's logical address change during the handover process managed by the mobile carrier's servers. Users simply had to wait about 20 minutes for the session, stuck in the previous BTS, to be released again. Whenever the user made a request in the application, an error message was displayed: "There is already a session open for that user." Believe me, this was part of the organization's business rule to avoid two active credential connections at the same time. However, this had not been implemented by the developers the right way.


The Field Testing

These problems could be reduced or even avoided by executing field testing activities. Field tests for mobile applications should be executed to represent the actual scenarios and the user experience in real environments. They are needed to identify the application's ability to maintain and manage data, voice communication, and app session with its servers based on a variety of telecommunication scenarios.

The focus of the activities is to evaluate the application behavior during the mobile device interaction with the BTS, against the scenarios faced by the users on public and private areas such as interference in the transmission, areas without mobile carrier network coverage, data exchange, telecommunication technologies, antenna and frequency exchanges, etc., while using the app.

The areas to be covered must be previously mapped in strategic locations, with the use and assistance of specific tools, in order to identify the above scenarios based on the field testing activities to be performed.
The test cases to be executed should also be part of the field testing strategy. Generally, the test cases to validate the main flows, business rules, functionalities, requirements, etc. most critical and most used by the end users are selected.

The following activities are some typical examples of field tests to validate mobile application when the mobile device:

Although some field test activities are executed dynamically, there are others that are executed statically such as Fallback testing that is summarized in the scenario when the mobile device is connected to the 4G data network and receives a call, forcing the device to auto-switch to 3G technology for data connection continuity to keep the application session active, as well as network compatibility testing, local access point testing, and network migration carrier and vice versa, etc.

In conclusion, I would like to emphasize the importance of field testing for mobile application validation. The goal, as already described in this article, is to find and address application failures caused by intermittent mobile phone networks that are found by end users.

While manual and automated traditional test activities are imperative for validating business rules, functionality, requirements, and application portability across multiple mobile devices, the greatest number of failures not addressed by the development team will be found during field testing activities. This is due to the proximity of the chaotic environment faced by end users.


Works Cited

[1] KARNACKI, C.; OBSTLER, R. Keynote Conference 2016. jan, 2017, Accessed Date: 09 December 2017.

[2] RODRIGUES, Eduardo. The system works in the development environment, but once it was published users start to find failures. Is this familiar to you? LinkedIn, São Paulo, Jan, 2018, Accessed Date: 06 January 2018.

[3] ABRINTEL. Project will change license for installation of Antennas in SP, Accessed Date 05 January 2018.

[4] TIM. TIM coverage map, <>. Accessed Date: 06 january 2018.


Back to Overview Previous Article Next Article