The first metric we’ll dive into in the Instance Insights series is time to install. The time it takes for end users to get a vendor’s software up and running in a new environment is a key indicator of the quality of the packaging, configuration, testing, and documentation of the delivery of a software product. Lacking in one or more of these categories can result in a lengthy and cumbersome installation process.
Reducing time to install allows ISVs to complete more installations in less time, which allows customers to see value faster during crucial POC or onboarding phases. Improving this metric also creates value by reducing the time engineers spend assisting with installs, making them available to work on other things, like improving the product. Measuring this metric across all installation attempts allows software vendors to assess the overall performance of the delivery of applications into customer environments and identify key areas for improvement.
Best in Class
In our experience, best in class vendors will generally measure the 80th percentile for time to install performance, and will complete 80% of installs in under 2 hours.
Define the “Time to Install” Metric
Time to install is a multi-faceted metric and can be complicated to measure, so it’s important to define what it means to your business to ensure you measure what matters to you. While deciding on your definition of time to install, think about what outcomes you want to drive by focusing on this metric. Are you looking to reduce the support burden on engineering teams that are pulled in to support customer installations? Are you looking to reduce the time it takes to progress prospective customers through a proof of concept? Are you looking to reduce the minimum hours of a professional services package for new customers?
Starting Point
The specific start and end points for measuring time to install can vary. You can measure a few different combinations of timestamps to understand installation performance. In this article, we’ll consider two possible start points:
- Intent to deploy - this is usually marked by the issuing of a production license but could also be signified by a transition in your CRM system, or by an end customer downloading a trial license from an automated signup portal.
- Install attempt start - when an end customer initiates an install attempt, usually with a CLI command in provisioned infrastructure or launching an instance from a pre-baked image.
The graphic below shows a high-level summary of a common installation workflow:
Ending Point
While this article primarily focuses on time to install, the end goal of any customer installation effort is to get software not only up and running in a customer environment, but also getting to the point where the application is delivering value to the customer.
Time to value - this concept of delivering value may encompass the completion of any or all of the following:
- Application is installed, running, and healthy
- Customer data is connected into the instance
- Multiple customer users are authenticated into the instance
- Instance is connected to production infrastructure
- Instance is managing production builds/workloads
Time to install - While there are many steps on the journey from intent to deploy up to software delivering value, we find the most valuable place to focus measurement and improvement is time to install - that is, the time it takes to get software installed, running, and healthy.
While we primarily focus on time to install, it’s important to keep time to value in mind. If you speed up your time to install but somehow slow down the overall time to value by adding time to other steps in the process, this should be considered an overall regression in delivery performance.
Pros and Cons
1. Time to Install (license) - time from license creation to software live
- Pro -This tends to be the easiest definition to measure as both license creation and software live have data points that can be collected and measured.
- Con -If you don’t also look at what is occurring between the creation of the license and when the software is live, it’s hard to know where a bottleneck might be. The license could have been sent to a customer but they missed the email or it wasn’t an immediate priority, there could be delays in provisioning requirements, failed install attempts, etc.
2. Time to Install (instance) - time from installation attempt start (usually initiating a CLI command) to software live
- Pro - By targeting this specific step in the installation process and inspecting outliers, you can gain a lot of insight into what may be causing the install to take more time, making it easier to identify areas that can be improved.
This also tends to be the most effort-intensive part of the whole process for both vendors and end customers. It may require synchronous work and deep network/infrastructure debugging by skilled engineers from both teams.
- Con - Being so focused can result in the oversight of other areas that may be delaying the overall install process. If your time to install (instance) is quick and painless, that doesn’t necessarily mean the whole process is.
It can be hard to programmatically measure exactly when an installation attempt started, especially if an end customer ends up starting an attempt and then discarding the install environment and starting fresh with a new VM or cluster.
3. Time to Value - time from license creation to software delivering value
- Pro - Having the end point be value delivery can help uncover undiscovered problems preventing a customer from receiving value after the software is live.
- Con - Due to the nuance of value delivery, there may not be an associated data point that can be collected verifying the date, making it difficult to measure. In many cases, the point at which the software is delivering value will need to be manually recorded, making it potentially error-prone.
Where you start and end measuring time to install will be impacted by your business practices and installation methods. These are likely to change over time, so it’s important to reevaluate your definition regularly to make sure you’re still measuring what matters.
Evaluate Current Performance
Before setting a goal, you’ll want to gauge what your current time to install is so that you’re able to set a goal that is attainable. While you may not have been measuring time to install previously (or even have a method for measurement), you should minimally be able to estimate what your happy paths look like vs. unhappy paths.
- Happy path - when an install goes smoothly as planned with no major issues or delays.
- Unhappy path - when an install does not go as planned and bumps and roadblocks are encountered causing delays, multiple install attempts, and customer frustration.
Example:
- Happy path - primarily online installs with a single server where everyone is prepared and requirements have been reviewed and met. In these cases, installs are easy and only take about 2 hours.
- Unhappy path - complex air gapped installations take about 2 weeks due to the amount of requirements and the complexity of the environment.
Set a Goal
It’s recommended to set both a short and long term goal for time to install and each should be specific, measurable, attainable, relevant, and time bound. The goals should be based on current performance and are impacted by your ability to identify the why behind the installs that end up on the unhappy path. If you know the why, and it’s something that you’re able to address and improve, you’re more likely to attain a loftier goal.
Example:
- Short term: Reduce time to install to 80% in under 3 days within the next quarter.
- Long term: Reduce time to install to 80% in under 2 hours within the next year.
Measure Performance
Method of measurement may vary depending on how time to install was defined.
Starting point - if this was defined as the intent to deploy, this will have a defined date. Otherwise, unless your installation tooling tracks and reports this information, the point at which the customer starts the installation will need to be manually recorded and tracked.
Ending point - if this was defined as when the software is live, the end point can be captured manually (i.e. while on a call with the end customer helping them complete the install) or using tooling that is able to capture first ready data. High quality application health checks can help with the automated collection of this timestamp.
If you are measuring the end point of an air gap install or are using value delivery as the end point, the data will likely need to be manually recorded and tracked.
Scope - You should record the start and end points for each customer instance you deploy so that you can compute statistical summaries (median, mean, and percentiles) across your entire customer base. If you have multiple instances at a single customer site, it is worth recording each instance’s time to install separately. In general, if an instance never becomes live or never reaches a point of value delivery, it should not be included in your time to install data. We’ll go into more detail on measuring installation success rate in an upcoming blog post.
Examples of Measuring Performance Over Time
The graph below depicts the average time to install over time compared to the goal that was set. You can see where the goal was also adjusted after an initial short term goal was reached.
It can also be helpful to look at a histogram of all install attempts for a specific time period, with the median and 80th percentile identified, as depicted above. Tracking time to install using these data points can help your team understand the distribution of install times across instances and makes it easier to visualize if you’ve missed, met, or exceeded your goal(s).
Visualizations of the data can highlight the installations that end up on the unhappy path and help you understand if your goals are truly driving the outcomes that matter to your business. Looking at the example graph above, you might decide that you’re okay with 2 installs being in the 5+ days range, or you might decide to revise your performance metric to capture a maximum time to install instead of the 80th percentile.
Make Adjustments to Improve Your Performance
As you are measuring your time to install performance, it’s important to identify edge cases and pain points that are negatively impacting progress towards your goals. Below are our recommendations for identifying and making adjustments to improve those pain points:
- Environment testing - if you are installing your application into a new type of customer environment, it’s important to do a test install internally in a similar environment. Testing in advance of a customer install will help you identify additional requirements or limitations that may need to be addressed and uncover potential pitfalls that may be encountered. Being prepared for a potential failure and being able to quickly identify and resolve the issue reduces time to install and helps maintain customer confidence. We often find that by adding an hour of testing up front, vendors can reduce time to install (instance) by 24+ hours. This testing could involve a checklist for manual testing, or an automated tool or script provided to the customer along with the other installation assets.
- Diagnostic bundling - enabling end users to easily collect a bundle of log files, metrics, and other diagnostic information can help your team diagnose issues when an installation attempt doesn’t go as planned. The faster you can understand what’s wrong, the faster you can get things back on track, and sending logs back and forth via email or screen share can cause delays in identifying the root issue(s). This can be as simple as a shell script or as sophisticated as an OSS tool like Troubleshoot.
- Documentation - sharing documentation on prerequisites, installation process, and troubleshooting steps with your customer will help them be more prepared and agile during the install. Continuing to iterate on your documentation with insights from previous installs will help you continue to move towards your time to install goal. Having members of your team continuously test your documentation while thinking from the perspective of a customer helps ensure it’s clear and focused for first-time install attempts.
How Replicated Can Help
At Replicated, we are actively working on improving reporting and telemetry within our product to help ISVs understand the health and performance of customer instances and measure key indicators of performance like time to install.
Replicated can assist with the measurement of time to install by providing information on license creation dates for the starting point and the timestamp of first seen (for online installs) in the install information on Replicated’s new instance details page (see below). More detail on our improved customer instance telemetry can be found here.
Replicated also provides computations for instance and license time to install based on information about when the instance is first seen, and when it first reports a ready state during an update check. These are listed as “beta” because we’re still evolving how we compute and display this information for software vendors.
In addition to providing insight into the data, Replicated also has resources and features that can be used to make adjustments to improve time to install:
- Preflight checks - customizing preflight checks to ensure a customer’s infrastructure meets the needs of your application is an impactful way to reduce the time it takes to complete an install. Continuing to iterate on preflight checks as you find new limitations/requirements is key to continually reducing time to install as you scale to tens or hundreds of installations per year.
- Support bundles - in the scenario where something doesn’t go as planned, customizing your support bundles through collectors and analyzers can help you find the source of the issue and resolve it. You can also use the information gathered in support bundles to help identify additional items to add into your preflight checks and/or documentation.
- Basic dashboard customizations - application custom resources can help improve your customer’s experience and make it easier for them to understand application health during initial setup.
- Documentation - Replicated has a documentation starter repo that can be used as a starting point for creating effective documentation.
- Technical review - we recommend that all of our vendors go through a technical review with a member of our team. Technical reviews involve issuing a trial or development license to Replicated, and we will go through a test installation using your install documentation. During this process we look for areas in your documentation and in the application deployment process that can be improved. We check for best practices, ways to reduce your time to live, and advise on how to improve your overall install experience.
Continue iterating on the define-evaluate-goal-measure-adjust cycle as you make progress towards your ideal time to install. Once time to install is approaching best in class, vendors tend to look at improving the number of installations that can be performed fully self-service, without vendor assistance. If you measure time to install, we’d love to hear from you on what has worked for you and what your goals are.
Stay tuned for our next post in the Instance Insights series, install success rate.
Learn More
- Recent product enhancements: improved customer instance telemetry
- Follow our roadmap item for enhancing time to install reporting for more updates as Replicated’s efforts on measuring and reporting time to install evolve over time.
- Not a Replicated customer, but interested? Schedule a demo with us!