|
Unlock access to the data your product needs (Sponsored)
 |
Most tools are still locked to their own database, blind to everything users already have in Slack, GitHub, Salesforce, Google Drive, and dozens of other apps. That's the ceiling on what you can build.
WorkOS Pipes removes it. One API call connects your product to the apps your users live in. Pull context from their tools, cross-reference data across silos, power AI agents that act across services. All with fresh, managed credentials you never have to think about.
Every Friday morning, a team at Spotify takes hundreds of code changes written by dozens of engineering teams and begins packaging them into a single app update. That update will eventually reach more than 675 million users on Android, iOS, and Desktop. They do this every single week. And somehow, more than 95% of those releases ship to every user without a hitch.
The natural assumption is that they’re either incredibly careful and therefore slow, or incredibly fast and therefore reckless. The truth is neither.
How do you ship to 675 million users every week, with hundreds of changes from dozens of teams running on thousands of device configurations, without breaking things?
The answer is not to test really hard. Spotify built a release architecture where speed and safety reinforce each other. In this article, we will take a look at this process in detail and attempt to derive learnings.
Disclaimer: This post is based on publicly shared details from the Spotify Engineering Team. Please comment if you notice any inaccuracies.
The Two-Week Journey of a Spotify Release
To see how this works, let us follow a single release from code merge to production.
Spotify practices trunk-based development, which means that all developers merge their code into a single main branch as soon as it’s tested and reviewed. There are no long-lived feature branches where code sits in isolation for weeks. Everyone pushes to the same branch continuously, which keeps integration problems small but requires discipline and solid automated testing.
See the diagram below that shows the concept of trunk-based development:
 |
Each release cycle starts on a Friday morning. The version number gets bumped on the main branch. From that point, nightly builds start going out to Spotify employees and a group of external alpha testers. During this first week, teams develop and merge new code freely. Bug reports flow in from internal and alpha users. Crash rates and other quality metrics are tracked for each build, both automatically and by human review. When a crash or issue crosses a predefined severity threshold, a bug ticket gets created automatically. When something looks suspicious but falls below that threshold, the Release Manager can create one manually.
 |
On the Friday of the second week, the release branch gets cut, meaning a separate copy of the codebase is created specifically for this release. This is the key moment in the release cycle. From this point, only critical bug fixes are allowed on the release branch. Meanwhile, the main branch keeps moving. New features and non-critical fixes continue to merge there, destined for next week’s release. This separation is the mechanism that lets Spotify develop at full speed while simultaneously stabilizing what’s about to ship.
Teams then perform regression testing, checking that existing features still work correctly after the new changes, and report their results. Teams with high confidence in their automated tests and pre-merge routines can opt out of manual testing entirely. Beta testers receive builds from the more stable release branch, providing additional real-world runtime over the weekend.
By Monday, the goal is to submit the app to the stores. By Tuesday, if the app store review passes and quality metrics look good, Spotify rolls it out to 1% of users. By Wednesday, if nothing alarming surfaces, they roll out to 100%.
The flow below shows all the steps in a typical release process:
 |