Producing Frequent High Quality Updates
Since fall 2015 we have been updating SaaS environments regularly, providing small improvements in every update. These regular updates contain features and bug fixes we want to deliver to our customers as soon as they are available. To achieve these frequent high quality updates we had to create a new way of working.
We currently update most of the SaaS environments weekly. This means that the software that is developed now, is live in your SaaS TOPdesk environment a week later! So how do we work fast and take care of the quality of the software we deliver at the same time?
Interdisciplinary scrum team
As a first step, during development, there is always an interdisciplinary scrum team responsible for the quality of the product. The design, code and test work is done in the same room, at the same time, by a group of people that feels collectively responsible. It means that fewer mistakes are made. Four eyes see more than two, which is why pair programming is common practice for our developers. Another way to manage risk are the automated regression tests that run every day to check if something has changed in the software. Only after the automated tests succeed, a new version is approved for use.
Eat your own dogfood
The second step follows the ‘Eat your own dogfood’ principle. We deploy the latest version of our software to an internal test environment. After approval in this stage, the software is deployed to a production environment is vital for our own work process. You might know the public side of this environment as extranet.topdesk.com. Currently, this environment is updated at least once a week, sometimes even more frequently. Because the difference between two updates is quite small, it is more manageable and less risky than a large update. If we find that despite our efforts in the development phase we made a mistake, it’s easier to pinpoint where the mistake was made.
The third and most important step is to roll out the software. Only a version that does well on our internal environment is promoted to the customer deployment phase, which consist of multiple stages to do a controlled staged rollout. We begin by updating the test environments. Since these test environments are almost similar to production this tells us that an update to the production environments will most likely be successful as well.
The version is now ready to move to production, starting with only 10% of our live environments. By not releasing a version to all environments at the same time, we can see if we can put our trust in this update before making it available to a large group. This first group that receives a new version are also the first to see new features and receive bug fixes.
This group is closely followed by a second group which consists of all other environments that receive regular updates. Within a few days from the development step, the improvements are live in production for a majority of SaaS customers.
Customers who are not in these frequent update groups are part of a more conservative track and only receive versions that did well during these frequent updates. This way, they too benefit from the staged rollout process.
The fourth and final step is to actively search for feedback and respond to it. By keeping an eye on performance, memory use and other statistics, we quickly notice unexpected changes after an update. When we receive feedback or even bug reports on the functional changes we made, we can quickly start the process of responding to these issues. We can also choose not to promote a version to a deployment stage.
We don’t only receive feedback on the product, but also on the release steps. We try to learn and improve on the steps with every update.
These 4 steps describe how we have been updating SaaS environments with increased frequency since fall 2015 and how we have reduced risk at the same time. Updates comprise small features and bug fixes and are delivered quickly after they are developed. With these updates we make sure that our SaaS customers are up-to-date with the latest we have to offer.