It seems completely incomprehensible today, but there was a time where automatically building your software every single day was a novel practice. Where deployments were done a few times a year with potentially tens, if not hundreds, of manual steps. But just read through Martin Fowler’s original version of Continuous Integration from 2000.
“A fully automated process that allows you to build several times a day is both achievable and well worth the effort.” - Martin Fowler (September 2000)
But I am also very convinced that 10 years from now we will look back at today and wonder how we were able to ship any products with such primitive tools.
But let’s start with the beginning.
A long long time ago...
First Generation: Continuous Integration
Throughout most of the late 90s, any build automation was usually done with custom scripting solutions. I (and many many other people) have written many a script that checked out all the code and build it overnight so that it was ready for testing the next morning. This is quite literally where the term “Nightly build” comes from.
But by the early 2000s, we started to see tooling appear in this space.
The main purpose of these tools was to do Continuous Integration: Build & test every commit and notify people when a build broke.
The main question that was answered by these tools was: “Is this still working?”
All of these products started as installable packages and were used on-prem.
The first open-source one was Cruisecontrol, released in 2001. Other notable CI tools in this space were Hudson/Jenkins, Teamcity, Atlassian’s Bamboo and the original Microsoft’s Team Foundation Server.
Second Generation: General purpose CI/CD Workflow
By the end of that decade, however, things looked very different: EC2 was launched by AWS in 2006, properly kicking off “The Cloud™” and tools like Puppet (started in 2005) and Chef (2009) popularised deployment automation.
Both of which led to an entirely new generation of tools by the early 2010s. These are cloud-based, SaaS products that provide building blocks and workflow automation. This allowed anyone to not just build and test on every commit, but to also automatically deploy versions to different environments for example.
The main question that this generation of tools tried to answer is: “Is this working as intended?”.
Popular products in this space are Travis, CircleCI, Atlassian Pipelines, Azure DevOps and Buildkite.
Which is here already a bit, just not very evenly distributed.
Third Generation: Delivery Pipeline as a Service
And just like last time, the seeds for a new generation of tools were sown in the second half of the previous decade. One of those trends is hosting providers starting to include some CI/CD-type functionality into their platforms. Heroku Review Apps, Netlify, and Zeit’s Now are just some examples.
But the really exciting development is that we are seeing an incredible convergence of execution platforms. A trend that is driven largely by the popularity of Docker. Every application that doesn’t require specialist hardware will end up being run on one or more of three platforms: Kubernetes, Serverless and Web Frontend.
And that means it is now possible to build tools that target one of these platforms and allow deployments to multiple hosting providers. This allows for a much larger investment than just that one provider can afford on a non-core business feature. And because these tools can now target a particular platform they can integrate much deeper and use all of the available features, rather than cater to the lowest common denominator.
These kinds of tools will make it possible to answer a much more pressing question: “Is this change valuable?”
There are of course no mature products in this generation, but mature internet-scale companies have entire teams dedicated to building this type of tooling. And commercially, it is going to be interesting to see where projects like Spinnaker, Harness and Serverless are going. Of course, Linc plays in this space too.
What will these tools look like?
It is hard to describe exactly what these tools will look like, but between the current trends and the internal tools that the current crop of internet-scale businesses have built we can make out the contours.
Easy testing in production
Every developer has at least once in their career exclaimed: “But it works on my machine!”. Most likely a lot more than once. But the same happens in a lot of postmortems: “But it worked on staging!”.
That is why most mature companies have been radically reducing the number of testing environments they have, often to the point that only production is left. And this tooling will need to support and coordinate running multiple versions/features in parallel and allow access depending on several factors.
Going from all or nothing production deploys to phased roll-outs.
There has been a massive amount of innovation in large organisations when it comes to safely releasing software at an incredible pace. Most of that innovation has centered around making releasing not a binary event, but a phase. Just some of the techniques used for that are Blue/Green deployments, Canary releases, A/B Testing, Traffic Shadowing, and Tap & Compare.
Controlling the pace of a particular release rollout over time will be a core feature of these pipelines. How do we safely test with production traffic?
Acting on real customer experience
By integrating with the monitoring system we can take the phased roll-out concept and start to do automatic rollbacks when issues are encountered such as increased error rates or an increase in response times.
Taking this concept further we can include access to analytics and automatically scale up and down canary releases and A/B tests based on things like usage or conversion metrics.
Developing products people love is incredibly hard and requires a team of people with different skills. And there is a lot of room to improve collaboration across the entire product development lifecycle. One area where there is currently an explosion of innovation is around design and frontend development with say Figma, Playroom, Design Systems and Storybook. How do we include product management, testing, operations, but also post-release marketing and sales?
Integrated into the entire product development ecosystem
Obviously, it would need to be highly integrated with the hosting platform, version control, monitoring, logging, error monitoring, and configuration systems. But also into most other applications that are in use across the product development ecosystem. Workflow management systems like Jira, external testing tools to run automated tests, analytics tools, communication software like Slack, release management tools. Maybe even IDEs and helpdesk & ticketing systems.
How will it be different?
Building and compiling software every day was once a challenge, now we do it on every keystroke. Running tests took the better part of the night once and now we run them on every save. Both radically changed the way we develop software for the better.
How would we change the way we write software if every time you saved a file, the resulting build was run against a sample of real requests in the last hour, highlighting any differences in the output?
How much safer would you feel if every deploy was tested in production before getting any traffic, slowly ramping up traffic if all goes well and removed immediately if error rates increase or business metrics go down?
How differently would we be organising our workflow if with a push to the repository that version was deployed to production with a link to it showing up in Jira and Slack for designers, product managers, and stakeholders to look at it in seconds?
How would it change the relationships with our customers if, with a push of a button on that Jira ticket, that feature would be enabled for all customers who requested it with a feedback button after they used it that feeds right back into that Jira ticket and the Github PR together with detailed analytics?
How would we change the way we organise our teams if after every merge an A/B test was immediately provisioned with real-time metrics rolling in immediately.
Right now building out the infrastructure to support all of this costs millions, a few years and a small army of extremely talented engineers. This new generation of tools will make that available to anyone for, comparatively, free.
I think we are going to be seeing a level of innovation in this space in the early 2020s that we have never seen before.