Compatibility Testing — or testing that your application can be successfully installed on unique combinations of Kubernetes distributions (Openshift, EKS, AKS, GKE, K3s, etc.), versions, and configurations — is a relatively new concept in software distribution. Why is it so important? Compatibility Testing empowers you to proactively prepare your application for complicated environments your customers might not even use yet, making troubleshooting environment-specific issues much easier.
Here’s a high-level example of Compatibility Testing integrated into a CI/CD pipeline, where the team will be certain that their application can be deployed on EKS, GKE, KiND, K3s, AKS, and OpenShift:
As a software engineer or engineering manager working with distributed software, you’re probably familiar with a situation like this:
- Your sales team sold to a big new customer - which is great! This big new customer wants to use a K8s version and configuration that your application has never been deployed on before. You tried to convince sales to get this information before the sale, but they never handled that request. You put it on your list to make sure your application works with this new customer’s distribution, but you have a lot on your plate and the ticket ends up deep in the backlog.
- Your field engineers start the new customer’s implementation sooner than you expected. They’re either physically on-site at the customer’s office or have lots of remote meetings scheduled and they’re already having a ton of deployment issues. You’re getting pinged constantly to jump in and fix what you can. Everyone is worried the new customer isn’t happy, and they aren’t even using your product yet.
- You (or your team) end up meeting with the customer directly to debug the deployment. There’s a ton of back and forth to figure out what’s broken, and the velocity of your current sprint is quickly slipping away. You feel like all you’re doing is debugging issues.
- Finally, after too many meetings and a lot of headaches, you, the field engineers, the sales team, and your customer have worked together to get the application fully deployed. You’re already thinking about the issues that will arise for them on the next release. You make a note to talk to your team about it.
- The sales team sells to another customer, and the whole cycle starts again! Fingers crossed they’re already using a distribution you know your application can handle…
If any of these steps sound familiar, then Compatibility Testing can make your life a whole lot easier. Compatibility testing allows you to fully test your application on a new distribution and version before the customer ever interacts with it. Even better, if your application is already deployed to your customer and they start having issues after the fact, you can use the same tools to help you quickly debug their deployment and eliminate back-and-forth communication to figure out what the issue is.
5 Major Reasons to Add Compatibility Testing to Your Distribution Stack:
- To make sure a new release will be successful in current customer-representative environments before you ever release it.
- To find and fix compatibility issues before your customers run into them.
- To prepare to release your application in new complex environments so you don't spend days (or weeks) reconfiguring your app to land a big new customer.
- To create validated environments that make it easier to communicate tested configurations.
- To troubleshoot issues in new and existing customer deployments.
Can you imagine a world where you can be confident that every new customer, and every new release, goes smoothly every time? Compatibility Testing can make that happen.
Interested in adding Compatibility Testing to your stack? Replicated’s Compatibility Matrix can help. You can also see how our own team used the Compatibility Matrix to fix a failed customer deployment in this blog!
Want to learn more about what Replicated does to help vendors distribute software to self-hosted environments? We would love to show you -- click here to schedule a demo.