In one of my last blog posts, I wrote about the Mobile Bug Matrix and how to use it to identify bugs that are worth doing a hotfix. Today, I write about the mobile release train and how to use this approach to release apps to the different app stores.
The release train concept is nothing new and is part of the scaled agile framework. It describes how to deliver software in a specific way. The concept of the mobile release train can be used by small and/ or bigger distributed teams who work on one app, but in different teams. Usually smaller app developer teams/ companies release an app whenever there are enough bug fixes or new features that make sense for their users. However, apps that are developed across multiple teams need an aligned approach to plan releases in advance. If more than one team is developing features they may depend on each other and can’t release without the changes from other teams. In this case, it is not possible to just push the release button to upload an app, this approach will fail.
Let’s take a look at a mobile release train. In the following picture, you see a simple one. It has a defined development phase in most cases 2-4 weeks. On a defined day and time let’s say on a Monday at 3pm, there is a code freeze happening. Until this time, the teams have the time to review, test and merge the features to the master branch that should be part of the train. At 3pm someone will create a release branch from master branch, either manual or in an automated way. This release branch will get a final integration testing phase. In this phase, all team members should check that the new features are working as expected. If there is a problem on the release branch the bug will be fixed on the release branch and later merged back to the master branch.
Once the code freeze happened a new development cycle will start and all contributing teams can start a new sprint and continue with the development. The great thing about the mobile release train is, that everyone knows the next planned release and can plan their work accordingly.
However, there must be one team or person handling the release. Either it’s always the same person/ team or it’s a shared effort among all developers. In any case, one person must be the responsible, let’s say “Release Engineer”. This person is responsible to get all required information for the release, e.g. release notes, new store images and so on. Furthermore, this person can organize a weekly release standup to inform about ongoing releases and problems that occur on the current live version.
If a team is using internal testing with colleagues, beta testing with real customers and make use of a staged rollout, the mobile release train can look like this.
In this example, you can see a possible Android release train. The development and integration testing phases are the same as in the simple version of the train. Once the app has been tested internally, the release engineer uploads the app to the beta channel. In this channel, real customers will receive an update notification on their devices for testing. During the beta testing phase, beta testers can give direct feedback via the app store. Next to the feedback, the crashes can be monitored in tools such as TestFlight, HockeyApp or Crashlytics.
After the beta testing phase, the app will be pushed to the stage-rollout. In this stage, the release engineer can define how many percentage of the users will get the update. In this case the steps are 10%, 20% and 50% rollout. In each stage the reviews as well as the crash tools must be monitored to check for possible problems.
If a problem gets visible in the 20% stage, the team has the chance to react to the problem and use the mobile bug matrix to decide if the problem is worth a hotfix. If this is the case, the train should not reach the next rollout step of 50%. Instead the problem must be fixed for 20% of the users. Once the problem is fixed and verified the train can move on to 50%.
As in the simple version of the train, after the code freeze only the release engineer or the release team is taking care of the release process. All other teams continue with the “normal” development.
Before you start to implement the mobile release train in your company, you must define the time slots for each stage. In most cases the development phase is 2-4 weeks. But then you need to define how much time you want to spend doing the integration testing, beta testing and the rollout stages.
Here is an example:
- 10 days of development
- 2 days of integration testing
- 3 days beta testing
- 2 days 10% of the users
- 4 days 20% of the users
- 2 days 50% of the users
In total, it takes 23 days until a new app version is live for 100% of your customers. For some apps and customers that might be too fast for others this is way too long. Every team/ company need to define their own time slots.
If you want to establish a more advanced mobile release train for iOS, you can remove the stage-rollout and exchange it with an extended internal testing phase with your colleagues. You can distribute the app internally using tools such as TestFlight or HockeyApp. Once the internal testing phase is over, you can use the same tools to send the app to beta customers. Afterwards you can upload the app to the app store and release it straight to 100%.
Establishing and using a mobile release train sounds great, but keeping the process up and running can be tough. Here are some aspects you should follow:
- One team/ person must handle the release coordination and the release itself.
- Have a weekly/ bi-weekly release standup to exchange between the contributing teams.
- Schedule and communicate the train departure times some weeks/ months ahead.
- Only accept reviewed and tested code.
- Only accept code that has unit tests and end-2-end test automation in place (if needed).
- Don’t accept late passengers. Don’t merge code to the release branch which was still in development on code freeze day. This code hasn’t been reviewed and tested enough.
- Use stage-rollout if possible.
- Monitoring the released app version is crucial.
What do you think about the mobile release train concept? Are you using it in your team/ company? If yes, I would love to see a comment down the post to exchange on that topic.