Learning From Failure: The Tricky iOS Environment

"You build on failure. You use it as a stepping stone. Close the door on the past. You don't try to forget the mistakes, but you don't dwell on it. You don't let it have any of your energy, or any of your time, or any of your space." 
- Johnny Cash

There are times when our work, as a team, as testers on a team can give us insight into how we can improve things for the overall success of the product. Often those things are issues or defects which can present themselves at the most inconvenient times, in the most inconsistent ways. When we can learn from them and improve things for everyone on the team, the team benefits. When the team owns the defects and the testing strategy, not only the testers, then customers benefit as well.

I was working remotely verifying a release and only had simulators and emulators to work with as I was testing mobile builds. Often folks mistrust builds which only use emulated software and not real devices for fairly good reasons. It’s usually not the best indicator of how the hardware will work with the software. What we forget is that with users involved replicating a production environment is extremely hard. Even deploying to a real device doesn’t always inform us of interactions with other software or what users could be doing with it.  

Presenting the risk is part of the job. Testing needed to be done, and I did the best I could. All of the testing from my side of things went well. I reported my findings. The release went forward.

The Next Day....

The following day, we immediately heard from customers that there was an issue with iOS. I was surprised. iOS generally works the same on a Mac desktop as it does on an iPhone because of the closed proprietary nature of the system. I’ve rarely found something on a device that I wouldn’t have found on a simulator. Questions were asked immediately:

“Why didn’t you test on a real device?”

“Why didn’t you inform us that you were not testing on real devices?”

“Did you do a regular build or a development build?”

“Were you aware that development builds don’t strip editor code like regular builds?”

All the questions were fair, but also a bit worrisome. While I had tested with an emulator, the initial thought was that I would have caught the defect if I was testing with a real device. Later it was discovered that it had nothing to do with the device or the emulator, but how I had built the project for Xcode. I had built it in Developer Mode, which doesn’t deploy the same code stripping as a project built without this mode. The problem was with the stripping, which I had inadvertently bypassed because I did not know that there would be a difference in data handling with build types like these. I only thought I was getting a log output from the app versus not getting one.

Feedback Loops Are Hard

I made notes were to make it more clear in the future about how I tested and what I tested when. I worked with another team member to come up with a small test report to give a better indication on stories how things were tested. We wrote a test plan to fill out details a bit more, and to gather feedback. Better progress reports on how testing is going and what might be potential risks in the future. My hope was that if we had faster feedback cycles on automated testing, and for manual testing, it might help the team catch issues faster or suggest other things to test so that we aren’t all surprised by something when it’s released to production.

Learning about the data handling issue triggered a change in how I built my projects after that event. It also made it clear that we needed a solution people could use remotely. Another clear issue was that we should find a way to have the CI create a project, build it, and deploy it to a device where it could be tested with automation, rather than spend time testing it manually. We might have gotten faster feedback with the automated build process, especially for a build defect.

We, as a team, learned from the defect. The defect was corrected quickly and patched to production. The team has a plan in place to minimize these kinds of events in the future. We learned from the defect and used it as a way to create an action plan to move forward with better testing.

The Future Is Slow To Happen Sometimes

Have we been able to get all of our automated testing in place yet? No, it will take time and while progress is being made, we are aware that our manual testing needs to consider emulators versus real devices and how best to leverage those things. Nothing we have on our action plan is a simple fix, but when it is in place, it will give us more time to test other things and allow us to have more agility in our deployment cycles.

Issues, like the one I encountered with iOS, can give us an opportunity to change practices, habits, and better understand the system we are working with. The team can gauge risk better and move faster with code changes and deployments. Looking at the longer term risks, or even the trends, in application behaviors based on reported defects can allow teams to focus on testing in those areas instead of using a more scattershot approach, hoping to find something critical.

Amazing things happen when you can reliably, and rapidly test the functionality of the software. Testing can open into many other avenues like accessibility, security, and performance. System health and quality can be raised by everyone by stabilizing the functional feedback cycle. Manual testing is a good start, but it can’t be the only way and your development workflow should be evaluated and changed to bring new processes in which could help a team reach a fast functional feedback system.

Don't get frustrated when this doesn't happen overnight. This is a marathon not a sprint to change. Sometimes things can happen immediately, and that's good. When developing a plan, set them up as short, medium, and long-term goals. Set an overall timeline to help keep the team on track for making those goals. It's possible. Patience and dedication can lead everyone to a safer, more stable and happy release cycle.