To learn more about Support Bundle Control for Troubleshoot.sh, and how to troubleshoot Kubernetes clusters, please sign up today for our webinar, Troubleshoot Your K8s Cluster, happening April 27th, 2022 at 10:00AM PST.
Sign up to attend here!
What it does
Regardless of your level of experience, whether you're a budding junior developer just starting out, a battle-worn DevOps Engineer who raises app uptime with their mere presence, or if you fall somewhere in between, reading raw JSON isn't what anyone would describe as a good time. Sure, you can just `grep` for a specific keyword or number, but that option is only helpful when you know exactly what you’re looking for. But, that may miss important supplementary information if said info happens to be a line down or up from the keyword you searched for.
How it works
At its core, a troubleshoot support-bundle is basically like running a few dozen `kubectl` commands in one go, with all of the output saved to compressed files that can be searched instead of having to scroll through terminal output - a Godsend for busy support teams. Believe it or not, our engineers have made troubleshoot.sh even better by creating a supplementary plugin that converts `support-bundle` output from JSON to standard `kubectl` output, saving even more time & flowing seamlessly into standard workflows for Kubernetes admins.
As of right now, only a few commands are supported with more being added every day, which means there’s ample opportunity to become a foundational contributor to what is sure to be an extremely popular open source project. If you’d like to contribute to sbctl, please feel free to open an issue with a short outline of what you would like to add to the project & we’ll help you along.
Installation
As a lover of anything that makes life easier, it is my pleasure to present a walkthrough on sbctl. Let’s get into it!
If you want to give sbctl a try, head over to the Releases page on GitHub & download the latest version for your operating system. As of now, 64-bit Linux & MacOS is supported but if you have suggestions or requests, feel free to head on over to the issues page to request support for other operating systems. I just so happen to have a single-node Kubernetes cluster running on Ubuntu, so I’m going to demonstrate installation on that. Both Linux & Mac users should be able to follow along using these commands.
[.pre]trilliams@replicated:~$ wget https://github.com/replicatedhq/sbctl/releases/latest/download/sbctl_linux_amd64.tar.gz--2022-03-24 14:47:01-- …2022-03-24 14:47:02 (32.9 MB/s) - ‘sbctl_linux_amd64.tar.gz’ saved [11738600/11738600][.pre]
Once the download completes, unpack the archive using your favorite archiving tool. I’m going with tar because it’s included with Ubuntu:
[.pre]trilliams@replicated:~$ tar xvf sbctl_linux_amd64.tar.gz README.mdsbctl[.pre]
That should leave you with two files: the `sbctl` executable & a README with usage instructions. Keep the README handy for later reference, but we’ll need to move `sbctl` to the system `$PATH` for easiest use. If you need help finding it, just run the following command to print the system path to STDOUT:
[.pre]trilliams@replicated:~$ echo $PATH/home/trilliams/.krew/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin[.pre]
I’m going to add `sbctl` to `/usr/local/bin` because I don’t want to `sudo`, but you’re free to store in whatever location you prefer. That can be done with the `mv` command:
[.pre]trilliams@replicated:~$ mv sbctl /usr/local/bin/trilliams@replicated:~$ which sbctl/usr/local/bin/sbctl[.pre]
Finally, let’s make sure that `sbctl` is running properly by executing the help flag:
[.pre]trilliams@replicated:~$ sbctl -hRun commands against a support bundleUsage:sbctl [command]Available Commands:completiongenerate the autocompletion script for the specified shellhelpHelp about any commandserve Start API serverFlags:-h, --help help for sbctlUse "sbctl [command] --help" for more information about a command.[.pre]
Installation is now complete! Now we’re ready to try it out.
Let's use it!
The first thing we’ll need to do is create a `support-bundle`. There are several example `support-bundle` templates available in troubleshoot docs. If you’d like use `sbctl` to do some serious investigating, feel free to create a new YAML definition with some custom Analyzers, Collectors &/or Redactors using the template below:
[.pre]apiVersion: troubleshoot.sh/v1beta2kind: SupportBundlemetadata:name: supportbundle-tutorialspec:collectors: []analyzers: [][.pre]
Or if you just want to play around, use the sample `support-bundle` hosted on GitHub:
[.pre]trilliams@replicated:~$ kubectl support-bundle \https://support-bundle.replicated.com[.pre]
Once you’ve generated a `support-bundle`, you’re ready to run the `sbctl` command, which will proxy a connection to the kubernetes API & also print out location of a new kubeconfig file for as long as the connection is open. If you’re unable to open a new terminal window to connect to the instance, you can shift the `sbctl serve` command to the background by appending an `&` to the end of the command, then hit the ENTER key to return to command line as shown below:
[.pre]trilliams@replicated:~$ sbctl serve --support-bundle-location=/home/trilliams/support-bundle-2022-03-24T15_12_24 &Server is runningexport KUBECONFIG=/tmp/local-kubeconfig-4178765460[.pre]
Now that `sbctl` proxy has connected, we are now able to pull `support-bundle` details directly from `kubectl`.
[.pre]trilliams@replicated:~$ kubectl get po -n schemahero-system -o wideNAME READY STATUSRESTARTSAGEIP NODE NOMINATED NODE READINESS GATESschemahero-0 1/1 Running 4 (12h ago) 6d8h 10.32.0.25 replicated <none><none>trilliams@replicated:~$ kubectl get po -n schemahero-system -o wideNAME READY STATUSRESTARTSAGEIP NODE NOMINATED NODE READINESS GATESschemahero-0 1/1 Running 4 (11h ago) 6d8h ***HIDDEN*** replicated <none><none>[.pre]
If you’d like to see more, have a feature request, or just want to connect with fellow Troublemakers, subscribe to the Replicated Community Calendar for invites to community meetings & events for troubleshoot.sh & SchemaHero.
TL;dr - support-bundle is the equivalent of running a couple dozen kubectl commands at once & saving to file. Support bundle control makes getting to that output way easier by reprinting the support bundle contents as kubectl output
Output bank
w/o sbctl
[.pre]trilliams@replicated:~$ kubectl get nsNAMESTATUS AGEdefault Active 5d22hkube-node-lease Active 5d22hkube-public Active 5d22hkube-system Active 5d22hkurlActive 5d22hlonghorn-system Active 5d22hminio Active 5d22hmonitoringActive 5d22hprojectcontourActive 5d22hschemahero-system Active 5d21htrilliams@replicated:~$ kubectl get pods -n kube-systemNAME READY STATUSRESTARTS AGEcoredns-64897985d-7qj481/1 Running 5 (39m ago)5d22hcoredns-64897985d-d4vjl1/1 Running 5 (39m ago)5d22hetcd-replicated1/1 Running 5 (39m ago)5d22hkube-apiserver-replicated1/1 Running 5 (39m ago)5d22hkube-controller-manager-replicated 1/1 Running 5 (39m ago)5d22hkube-proxy-p5t6n 1/1 Running 5 (39m ago)5d22hkube-scheduler-replicated1/1 Running 5 (39m ago)5d22hweave-net-6m7wf2/2 Running 16 (38m ago) 5d22htrilliams@replicated:~$ kubectl describe pod --namespace=schemahero-system schemahero-0Name: schemahero-0Namespace:schemahero-systemPriority: 0Node: replicated/192.168.1.127Start Time: Fri, 18 Mar 2022 18:00:33 +0000Labels: control-plane=schemaherocontroller-revision-hash=schemahero-77bcc67945statefulset.kubernetes.io/pod-name=schemahero-0Annotations:<none>Status: RunningIP: 10.32.0.25IPs:IP: 10.32.0.25Controlled By:StatefulSet/schemaheroContainers:manager:Container ID:containerd://3592b77cb960970ab6f70c51ce4274085aadf738be32e01731144dfe67a86c8fImage: schemahero/schemahero-manager:0.12.7Image ID:docker.io/schemahero/schemahero-manager@sha256:9516f0ee8b90ec53c9d61e5cef007b3bb49da60f771d5031052e83ea380a9165Port:9876/TCPHost Port: 0/TCPCommand:/managerrun--enable-database-controllerState:RunningStarted:Thu, 24 Mar 2022 14:37:34 +0000Last State: TerminatedReason: UnknownExit Code:255Started:Tue, 22 Mar 2022 13:29:30 +0000Finished: Thu, 24 Mar 2022 14:36:48 +0000Ready:TrueRestart Count:4Limits:cpu: 1memory:150MiRequests:cpu: 100mmemory:50MiEnvironment:POD_NAMESPACE:schemahero-system (v1:metadata.namespace)SECRET_NAME:webhook-server-secretMounts:/tmp/cert from cert (ro)/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b54q4 (ro)Conditions:TypeStatusInitialized True Ready True ContainersReady True PodScheduledTrue Volumes:cert:Type:Secret (a volume populated by a Secret)SecretName:webhook-server-secretOptional:falsekube-api-access-b54q4:Type:Projected (a volume that contains injected data from multiple sources)TokenExpirationSeconds:3607ConfigMapName: kube-root-ca.crtConfigMapOptional: <nil>DownwardAPI: trueQoS Class: BurstableNode-Selectors:<none>Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300sEvents:Type ReasonAgeFrom Message---- ---------- ---- -------WarningFailedMount 44m (x2 over 44m)kubeletMountVolume.SetUp failed for volume "cert" : failed to sync secret cache: timed out waiting for the conditionNormal SandboxChanged43m (x3 over 44m)kubeletPod sandbox changed, it will be killed and re-created.Normal Pulling 43mkubeletPulling image "schemahero/schemahero-manager:0.12.7"Normal Pulled43mkubeletSuccessfully pulled image "schemahero/schemahero-manager:0.12.7" in 638.523822msNormal Created 43mkubeletCreated container managerNormal Started 43mkubeletStarted container managertrilliams@replicated:~$ [.pre]
With sbctl
[.pre]trilliams@replicated:~$ kubectl describe pod --namespace=schemahero-system schemahero-0Name: schemahero-0Namespace:schemahero-systemPriority: 0Node: replicated/***HIDDEN***Start Time: Fri, 18 Mar 2022 18:00:33 +0000Labels: control-plane=schemaherocontroller-revision-hash=schemahero-77bcc67945statefulset.kubernetes.io/pod-name=schemahero-0Annotations:<none>Status: RunningIP: ***HIDDEN***IPs:IP: ***HIDDEN***Controlled By:StatefulSet/schemaheroContainers:manager:Container ID:containerd://3592b77cb960970ab6f70c51ce4274085aadf738be32e01731144dfe67a86c8fImage: schemahero/schemahero-manager:0.12.7Image ID:docker.io/schemahero/schemahero-manager@sha256:9516f0ee8b90ec53c9d61e5cef007b3bb49da60f771d5031052e83ea380a9165Port:9876/TCPHost Port: 0/TCPCommand:/managerrun--enable-database-controllerState:RunningStarted:Thu, 24 Mar 2022 14:37:34 +0000Last State: TerminatedReason: UnknownExit Code:255Started:Tue, 22 Mar 2022 13:29:30 +0000Finished: Thu, 24 Mar 2022 14:36:48 +0000Ready:TrueRestart Count:4Limits:cpu: 1memory:150MiRequests:cpu: 100mmemory:50MiEnvironment:POD_NAMESPACE:schemahero-system (v1:metadata.namespace)SECRET_NAME:webhook-server-secretMounts:/tmp/cert from cert (ro)/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-b54q4 (ro)Conditions:TypeStatusInitialized True Ready True ContainersReady True PodScheduledTrue Volumes:cert:Type:Secret (a volume populated by a Secret)SecretName:webhook-server-secretOptional:falsekube-api-access-b54q4:Type:Projected (a volume that contains injected data from multiple sources)TokenExpirationSeconds:3607ConfigMapName: kube-root-ca.crtConfigMapOptional: <nil>DownwardAPI: trueQoS Class: BurstableNode-Selectors:<none>Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s node.kubernetes.io/unreachable:NoExecute op=Exists for 300sEvents:Type ReasonAgeFrom Message---- ---------- ---- -------WarningFailedMount 56m (x2 over 56m)kubeletMountVolume.SetUp failed for volume "cert" : failed to sync secret cache: timed out waiting for the conditionNormal SandboxChanged55m (x3 over 55m)kubeletPod sandbox changed, it will be killed and re-created.Normal Pulling 55mkubeletPulling image "schemahero/schemahero-manager:0.12.7"Normal Pulled55mkubeletSuccessfully pulled image "schemahero/schemahero-manager:0.12.7" in 638.523822msNormal Created 55mkubeletCreated container managerNormal Started 55mkubeletStarted container manager[.pre]