Let’s start the year 2017 with some fresh and new mobile content. Today’s post is about the mobile bug matrix. Developing, testing and releasing mobile apps can be a challenging task. However, patching an app to fix bugs is not easy, too. How do you decide which bug is going to be fixed in an hotfix or not? Once a bug is online and installed on user’s mobile phones, there is a high likelihood that the bug will stay there for a longer time period, because not everyone is updating apps every day or has the auto update enabled.
There is no way in rolling back an app version once it has been uploaded. When I explain this problem to people who are not into mobile developing and testing I use the metaphor of a burned CD. Do you remember the good old times, when magazines added CDs with useful tools or CDs containing important drivers for your PC? Then you also remember how frustrating it was when the software on that CD had bugs or was completely broken. Once the CD is burned and delivered there is no way of rolling it back. The only way to solve the problem is to send a new CD ;). The same happens with native mobile apps. Once the app file was uploaded to an app store and users have installed the buggy version, you only have the chance to patch/ update this version with the next version.
Patching an app is time consuming, requires development and testing efforts and maybe even more alignments with other departments in the company. Furthermore, the planned development and testing in the current cycle will be slowed down or paused until the app is patched. And you should also keep in mind that some app stores have a review process in place, which may take some days until the patched app is available to the users.
One way to minimize the likelihood of doing a hotfix is an extended internal testing phase with your colleagues. Another way is to distribute the app to some beta testers to gain their feedback. But we all know that the real nasty bugs happen in the wild on the customer phones and companies need to handle them as soon as possible.
Mobile Bug Matrix
But how to decide which bug is worth doing a hotfix? Over the past years I have seen teams/ companies dealing with exact this question. Some teams tried to fix every bug in production but ended up in just doing fixes for the released app and stopped doing feature development. Teams/ companies need to find a way to handle the question and to find the right balance between doing a hotfix and to postpone the bug fix to the next upcoming release.
To make the decision of doing a hotfix easier, I created the mobile bug matrix. The mobile bug matrix has 2 axes. The x-axis is for the criticality/ severity for the user (affected users) and y-axis stands for the criticality/ severity for the business point of view (affected business).
This results in 4 areas/ categories as you can see in the picture.
- Hotfix: This is the area which has a high impact to both, the business and the user. If a bug fits into this bucket, it’s worth doing a hotfix.
- Next Release: Either affects more the business or the user, but it’s nothing critical that needs to be done now, but should be fixed with the next planned release.
- Next Release: Either affects more the business or the user, but it’s nothing critical that needs to be done now, but should be fixed with the next planned release.
- Upcoming Sprints: Bugs that need to be fixed but there is no rush. Add them to the backlog for further release planning.
Here are some bug examples* for every area:
Hotfix Worthy Bugs
- The app login is not working anymore.
- In-app purchase is not working anymore.
- Main app feature is broken.
Next Release Bugs
- Usability issues.
- Random app crashes.
- Tracking is broken.
Upcoming Sprints Bugs
- Typos or wrong translations.
- Wrong colors.
* Note: These are just examples and may look completely different for various apps. Please create your own bug examples that fit to your app and into the different areas.
Questions to Raise
Next to the 4 areas and the bug examples, it’s worth to raise the following questions. The questions will help you to get a discussion started, if the hotfix is needed.
- How many users are affected by the problem e.g. in %?
- Which app section is affected by the problem?
- Is the problem reproducible on all supported operating system versions?
- Are we losing money?
- Is the app losing activity?
- Is the company losing trust?
- Do we have a legal issue with this bug?
- Is the problem causing any privacy concerns?
- …
Having the matrix, the bug examples and the questions in place it’s way easier to decide if it is worth doing a hotfix or not.
Try to establish the mobile bug matrix in your team/ company and give it a try.
Let me know what do you think about the mobile bug matrix.
#HappyTesting