App Testing

The launch of any software product is a tense time for the company who has developed it. It is at this point of the software development life-cycle that is probably the most important. There are critical questions to be answered, for example; how much testing has been done?, what is the impact of anything going wrong?

To answer the question; how much testing has been performed can be fairly straight-forward to answer by reviewing the test documentation and conversing with the appropriate stakeholders. All too often though, the testing can be heavily focused on an environment that although was designed to replicate a customers environment, normally doesn't quite live up to expectations. This can result in a software product being released with a high risk of failing in the end-users environment, even though it was heavily tested. Companies who develop software should go the extra mile and ensure that sufficient Beta Testing is performed prior to the actual release of the product. Often, a company will simply send a Beta version of their application to a customer or two hoping that's enough. It almost certainly isn't enough, and careful planning should go into choosing the right people to perform the Beta Testing. If suitable existing customers do not exist, then the developing company should consider a professional Beta Testing company to do it for them. A good Beta Testing company should be able to replicate any kind of environment required and perform the kind tests that will flush out the issues that may have been missed earlier on in the software development life-cycle.

A good example of this kind of situation can be found in the "new wave" of software that is around today, which is of course iPhone Apps. So many potentially good iPhone Apps are released without Beta Testing them. The developers/owners look at their poor ratings and ever decreasing sales figures wondering why its happening as their App is great and offers something unique. While the latter may be true, just looking at the iPhone App reviews, you will see so many users complaining that the App crashes when using even the most basic functionality. If the App developers took the time to hire a professional iPhone App Beta Tester, they could have easily avoided the issues and started their App's shelf-life with positive reviews and subsequent high sales figures.

Beta Testers are not a specific kind of software tester, but merely a chosen group of individuals assigned with the task of looking for defects. The hardest task is choosing the right people to do the Beta Testing.  Careful consideration should be made as to what kind of defects are we trying to find. It could be that there is a risk of the server systems will fail if there is a high load on it. In this case, some kind of Public Beta Test would be in order. If some kind of sanity test before the product's release is required, then it could be simple as hiring a professional Beta Tester to do the testing.

The more consideration and planning that are put into a Beta Test, the chances of its effectiveness increase significantly. Consider hiring professionals where possible, and many will provide a free quote, fast turn-around and effective Beta Testing service. If you cannot afford a professional Beta Tester, then simply select appropriate personnel within your organization. If the selected people are given the right information about what they should be looking for, then this will often produce a surprisingly effective and in the long-term profitable exercise.

Early during the last decade, software development companies were producing applications by the bucket-load to keep up with the incredible customer demand. The role of the Software Tester then, was to (in my opinion) detect the critical and major issues, and most minor issues were simply brushed under the carpet to get the product out of the door on time. For the Software Testers, it was (to coin a phrase) "A much simpler time". During this time-frame the Software Testers actively working hard to get themselves recognized as a necessary part of the software development process, and not just some kine of 'bottle-neck' that delays a products release. Credit to the testers for actually achieving that goal!

Having worked so hard to achieve their aim, Software Testers found themselves under threat, a threat which in a way came from inside their own camp. The threat I am talking about was 'Test Automation'. Loose talk around the offices focused on how 'so much more useful' this kind of testing was. Testing could be ran 24 hours a day and repeated many times faster than a manual tester could perform. The humble Software testers were considered a dying breed. But, hang on a minute.....we are still here!

Rather than migrating the manual testing over to automated testing entirely. The move was actually more of a 'leaning' instead. A typical software development company will include only 'some' automated testing. Just tests where automation would enhance the manual testing or save a bit of time. Some testers have switched over to becoming devoted to test automation and that is now their primary job.

So while the road of a Software Tester has been 'uneven' to say the least. They have in fact, now firmly established their role as a necessary part of the software development cycle. Once considered to to 'dying out', the Software Testers have not only 'beaten' the threat of test automation, but actively use it alongside their own work to assist it and to enhance it.

So what it is store for the future of Software Testers? Well, software is still going strong and is now present on other platforms. The most popular 'new' platform has to be the 'iPhone'. It pretty much came from nowhere and has now topped 250,000 Apps that have been developed. These iPhone Apps are just regular software applications designed to work on a specific type of hardware. Many software testers are realizing that their software testing skills can be transferred over to this new kind of software development, and are becoming iPhone App Testers. There now exists iPhone App testing companies that are dedicated to providing iPhone App Testers to improve the quality of this rapidly growing phenomenon.

The future looks bright for Software Testers and iPhone App Testers alike, and that's good news for the people who want quality from the software they are using, whether its a PC application, web application or an iPhone App.

So you have developed a mobile application and you think its ready for release. You will have probably tested it on an emulator of some kind and possibly a browser or two, and so now you think its time to release it. Before this is done lets consider what state the mobile App is in. The mobile App only 'appears' to function correctly at this stage, which to all intensive purposes is a good thing right? but an emulator really only shows you how a real device 'should' work when used with your mobile application, it is certainly no guarantee. A web browser can certainly show you what your mobile App will look like on a real device, but there are many factors it cannot show you, for example; the touch screen functionality, CPU difference, memory usage and many more very important areas of functionality. It only takes a slight a small difference in what is 'expected' behavior to what is 'actual' behavior on a real device.

So once the decision is made to test on a real device, there are some important things to consider:

Which mobile devices do I need?

Lets for example use the most popular for of a mobile application, the iPhone. Currently there are several devices that fit into this category; the iPhone 4, iPhone 3GS, iPad, and iPod-Touch. In addition there are multiple hardware versions and multiple operating system versions associated with each device. This at first presents a dilemma,  but we can rule out older hardware and firmware(operating systems), why is this? well, 99% of users will upgrade to the latest firmware whenever possible to take advantage of bug fixes and new enhanced functionality. Again, you can rule out older versions of hardware, particularly if it is several years old and no longer supports the latest firmware available. Most users of these types of devices will upgrade when their device 'appears' out of date. In short, aim to test on the latest hardware and firmware, this in effect gives you a longer period of mobile App longevity.

I can't afford to buy real devices, what do I do?

Most carriers will offer you the mobile device cheaply if you sign a lengthy contract. This is not really a viable option if you just want to test with it. Most devices can also be bought without a contract, but for an expensive price. Again, this can be expensive, especially using our example of an iPhone App. Teh most cost-effective testing solution is to use an iPhone App testing company to do it for you. They have already made the expensive investment in purchasing the hardware so you don't have to. A good iPhone App testing company should offer you bundles for testing on multiple devices too.

Should I get a professional App test before release?

As the developer, you are essentially just one set of eyes. It is very important to get a new App tested to some level before its release, as once its out there you can't do anything about it, until its too late, i.e. you are receiving negative feedback and decreasing sales. You don't have to hire professional App testers to do the testing, you can get colleagues or friends to help you out. This should at least get some other sets of eyes looking at your App. The advantage of using a professional App testing service however, is they will have experience of a variety of software testing techniques, such as Usability, Functional, Performance testing etc. on many other Apps previously. Their software testers will know where to look for defects within your App and will provide you with an all important degree of confidence which you will only get with a professional App tester.

Which carriers/network providers should I use?

Depending on the country where the software testing is carried out, there will be a limit to which providers can be used. To be honest though, as long as the testing takes into account network specific testing, then the carrier should not matter at all. For example, an App tester will typically test on good, medium and poor signal strengths to test the behavior of your App under these conditions. So, as long as these tests are carried out not only on a wi-fi network but also on a 3G carrier providers network, then it really doesn't matter.

So to summarize then, you necessarily have to pay out a fortune in order to achieve a level of mobile application testing. With some forethought, you can effectively reduce the amount of real devices you need to test on. You can don't have to pay a fortune to ensure the quality of your mobile application.

If the App you are testing crashes, a well-designed App will record information within a crash log. This information can be invaluable to the developer responsible for the App. This log file contains information regarding what the App was doing before it crashed. The log files are copied to your Mac/PC when you sync your iPhone. Do the following to locate the log files;

1) Sync the iPhone

2) Locate the following folders. Note that DEVICE_NAME will be the name if your iPhone as shown in iTunes.
Mac OS X : /Library/Logs/CrashReporter/MobileDevice//
Windows XP: C:\Documents and Settings\\Application Data\Apple computer\Logs\CrashReporter\\
Windows Vista: C:\Users\\AppData\Roaming\Apple computer\Logs\CrashReporter\MobileDevice\\

3) Each log file will begin with the name of the application.

There appears to be a few iPhone App simulators out there on the net claiming to offer solutions for beta testing an iPhone App. Well, although technically(and I use this term very loosely) correct, the so called 'simulators' are nothing more than web browser screens made to be the same size an iPhone screen. This really cannot offer a real beta test of an application surely?

So what are the differences between testing on a simulator and a real device?

The CPU type
The screen (a real iphone has a touch-screen of course)
Installing the App
Memory usage
Zooming functionality
Network connectivity related issues

Remember, customers that buy the App from the App store won't be running it on a simulator!