A common challenge for independent software vendors (ISVs) is that the wild diversity of customer-managed operating environments makes testing their application very complex and time consuming. They often are faced with an exponential number of combinations of:
- Kubernetes (K8s) distributions
- K8s platforms and cloud managed services
- K8s versions
- Underlying compute and storage infrastructure and instances
- Major and minor application releases and versions
The unfortunate truth is that K8s distros, versions, and platforms are NOT all the same. They can perform differently in different contexts. The differences may be subtle but can cause unexpected issues. You can read more about this challenge in our blog All K8s are not the same: How ISVs test compatibility.
Yet due to the sheer number of possible combinations, the de facto result is that most ISVs don’t test nearly enough different customer-representative environments.
Further complicating matters are the realities of how testing usually gets done, and the common barriers to more testing, which often include:
- The cost of infrastructure for testing
- The time to provision all those environments
- A lack of time to build processes and tests
- A lack of automation for repeatability
- And a lack of understanding of which combinations are the most common patterns or significant anti-patterns in real world customer environments
Due to these challenges, we again note that most ISVs will only test a few combinations, maybe the last three versions of one distro running in one managed service. And they only do those tests around a major version, not for the vast majority of smaller releases nor continuously as they are developing new features.
The Compatibility Testing Maturity Curve
The (not unexpected) impact of this lack of testing is that problems are usually discovered not in the lab, but in live customer accounts. For obvious reasons, this leads to customer frustration, growing dissatisfaction, and all too often, churn. At the same time the vendor will experience extra overhead in managing relationships, while facing an extra burden on support teams and core engineers to try to recreate and resolve unforeseen issues. It can become ugly, but what can vendors do about this situation?
To help vendors rate their own testing effectiveness against their peers in the industry, we have created a compatibility testing maturity curve (see diagram above.) This shows that the best vendors are comprehensively testing all their functionality in customer-representative environments, continuously for every new release and multiple recent versions, and that testing is fully automated, scalable, and repeatable. Mid-pack companies test common combinations, with 2-3 distros and versions supported, doing periodic testing with some automation. The bare minimum, where many vendors are today, is testing basic installation with a single distro and version, just before big releases, with a very manual effort. Not good.
Announcing the Replicated Compatibility Matrix (beta)
We are delighted to announce a new service from Replicated that provides programmatic access to test clusters of various Kubernetes distributions and versions, available in seconds (a few minutes at most) to help vendors provision many combinations of customer-representative environments on-demand.
Provision Resources Quickly for Testing, Development, and Support
With the Compatibility Matrix, your engineers can create test clusters in under 2 minutes using our warm pool of common cloud machine types on Amazon EKS or GKE with per-minute billing (AKS coming soon!) Or they can choose their preferred vCPUS, memory, and storage from VM-based clusters. The provisioned resources will have a default time-to-live (TTL) so clusters expire automatically if not deleted sooner, keeping costs down. Your team can leverage kind and K3s Kubernetes distributions. Red Hat OpenShift is available today, and we will continue to rapidly expand to more clouds and distros, based on our telemetry of what is in highest demand.
Now your team can spin up representative combinations programmatically for testing as they develop, before every release, and when needed to recreate customer environments for support. Then they can run any variety of tests you want to build and automate. We don't explicitly provide for different types of testing, that’s up to the vendor to suit their needs, but anyone can define their own smoke tests, policy tests, release tests, and canary tests. There is a wide variety of testing strategies that might be desirable, which are also covered in our blog about ISV testing strategies for reliability and compatibility.
Vendors can now efficiently check releases for compatibility, readiness, stability, whatever they need. They can explore a simple configuration, end-to-end configurations, or a broad matrix of combinations.
Testing Customer-representative Environments before Releases
An added benefit here is that with our Instance Insights, the Replicated vendor portal reports and graphs can actually show which combinations are in real world use by customers. For each of individual customer, Replicated can show the:
- Kubernetes distribution and version
- Cloud provider and region
- Application version
- Installation method and version
Even better, vendors can use this information to programmatically provision truly customer-representative environments, configure them, and set up a series of tests for each release. If the test passes, a script that can even automatically promote the release to your stable or latest release channels and notify customers that is available for upgrade. Maybe even take advantage of semantic versioning and automatic upgrades here!
The Replicated Effect from our Compatibility Matrix
There are a number of business benefits that vendors will enjoy with the Replicated Compatibility Matrix. Comprehensive testing will build confidence with customers, reduce costs and the effort to test, increase install success rates, reduce support incidents in the field, and reduce risk in diverse environments.
There are technical benefits too, including the ability to spin up test environments on-demand (useful for recreating support issues too), identify version incompatibilities, test more comprehensively, automate common tests to reduce effort, and ultimately catch issues before customers find them.
Interested to learn more? Let us know!
We’ll help you onboard and start testing. We want feedback too! Your input can help shape our roadmap and what we prioritize next. Sign up now to learn more and join our waitlist.
You can also checks out the Compatibility Matrix docs or watch our short Compatibility Matrix intro video from RepliCon Q2.