Replicated offers Independent Software Vendors (ISVs) the ability to easily install their software application into end-customer managed environments. While Replicated can provide a dedicated K8s cluster underneath that application to install onto a VM via kURL, a common install method involves utilizing the end-customer’s own managed Kubernetes infrastructure to install the vendor’s application.
In these “existing Kubernetes cluster” type of application install scenarios, vendors can leverage open source KOTS, a kubectl plugin, to automate the install with a point and click GUI. This means that KOTS needs to successfully install and run alongside the vendor’s application in the end-customer environment. Any unexpected bugs introduced into KOTS can have a negative impact on the perceived overall product experience of the vendor’s application.
As we’ve previously mentioned in the blog Testing Releases in Customer-representative Environments, not all K8s distros, versions, and platforms are the same, which can make it incredibly difficult to have confidence your software will behave consistently in these various environments, particularly with distributions like OpenShift that tend to have certain security constraints that can introduce unexpected complexity if not tested on that distribution.
Before Compatibility Matrix
Ahead of Replicated’s new Compatibility Matrix offering, KOTS testing was primarily limited to running a robust end-to-end test suite on top of K3s clusters. This test suite consisted of 32 tests (and growing!) with 9 of those tests highly impacted by upstream Kubernetes evolving functionality, meaning they each needed to be tested across multiple minor versions of Kubernetes to guarantee no unexpected issues with any supported Kubernetes minor version.
This meant we were spinning up at least 50 K3s clusters on every PR commit, gating our ability to commit to main. We chose K3s underneath the test suite because it was fast and lightweight, but it wasn’t truly representative of what our customers most often install on. Testing on a distribution like OpenShift was hard and expensive, so despite most unexpected bugs popping up on this distro, our OpenShift testing was typically limited to distro-specific features and fixes when they arose.
Now With Compatibility Matrix
Now with Replicated’s new Compatibility Matrix offering, we are able to confidently test KOTS compatibility, and the new Replicated SDK offering, on OpenShift, EKS, GKE, AKS, and more.
With every new KOTS PR, we now test the latest patch version of all available Kubernetes minor versions, for every distribution that Compatibility Matrix currently supports. Despite the fact this means we are spinning up significantly more testing clusters than before (existing tests now multiplied by each new distribution and version), our testing pipeline takes no more time to run than before (just a couple of hours), due to the speed at which Compatibility Matrix can spin up these various environments allowing us to run all distribution tests concurrently.
Not only do we get the benefit of testing KOTS on many more popular enterprise Kubernetes distributions, we’ve also been able to better unify our testing logic around Compatibility Matrix, allowing us to replace former bespoke testing infrastructure solutions for each distribution type.
With Compatibility Matrix still expanding the amount of Kubernetes cluster types and versions that it offers, KOTS testing continues to benefit with each new distribution and version added.
To automatically inherit new Compatibility Matrix distributions into our KOTS testing pipelines with no engineering effort required, the KOTS team integrated the ability to automatically determine which distributions are available to be tested on. The code that enables this “automatic inheritance” for KOTS compatibility testing is not yet public, but let us know if you’d be interested in learning how to do something similar in your own Compatibility Matrix-powered testing pipeline.
All in all, the ability to leverage the Compatibility Matrix to provide the necessary infrastructure to power KOTS distribution compatibility testing has been a significant engineering time saver for Replicated.
Our original k3s testing infrastructure build-out took two weeks. The internal estimates to expand that to include EKS and AKS testing were originally two additional weeks each, with the team saying that OpenShift in particular would likely take two to four weeks to operate reliably enough to not block our test and release process.
As a product leader, I love that our KOTS team can now focus even more development time on features, instead of devoting several weeks to building out more testing infrastructure. With the Compatibility Matrix, we get even more distributions to test against than we could have allocated time to build DIY. With those additional distributions available in our KOTS testing pipeline, I gain significantly more confidence that our product experience will work as intended on the most popular enterprise Kubernetes distributions.
Learn More
Interested in getting access to the Compatibility Matrix and giving us feedback? Sign up now to learn more and join our waitlist.
You can also check out the Compatibility Matrix docs or watch our short Compatibility Matrix intro video from RepliCon Q3.
Not a Replicated customer, but interested? Schedule a demo with us!