Ship It and Sleep Well: How GitGuardian Built Bulletproof Support for On-Prem Security
“A good engineer is a lazy engineer.”
At GitGuardian, that idea shows up in a simple principle: automate everything that can be automated.
In a recent RepliCon talk, Jérémy Cotineau, Senior Kubernetes DevOps Engineer at GitGuardian, walked through how their team applied that principle to one of the hardest problems in self-hosted software: support.
Because in on-prem environments, the real challenge is not fixing issues. It is everything required to get to the fix.
The Problem: Supporting Systems You Cannot Access
GitGuardian operates in environments where direct access to customer infrastructure is not possible. Many of their customers run in highly secure, self-hosted environments with strict access controls.
That creates a familiar set of challenges:
- Debugging requires coordination across multiple roles
- Logs may be incomplete or unavailable
- Reproducing issues is difficult
In practice, support becomes a multi-step process involving different teams and tools, with delays at each step.
The Foundation: Support Bundles and AI
GitGuardian built their support workflow around Replicated Support Bundles, which collect logs and cluster state into a single package.
Out of the box, this provides a consistent way to capture data from a Kubernetes environment. GitGuardian extended this further by adding custom collectors.
One key addition was integrating logs from their internal log collection system. This ensures that even if a pod crashes and logs are lost from the cluster, they are still available in the bundle.
Because support bundles are configurable, GitGuardian can include exactly the data they need for debugging without relying on manual log collection.
The result is a repeatable starting point for every support case: a complete snapshot of the system at the time of the issue.
Once data collection was standardized, the next step was reducing the effort required to analyze it.
GitGuardian built an internal AI system that integrates with their support workflow. When a support ticket is created and a bundle is uploaded, the system:
- Receives the ticket context
- Processes the support bundle
- Identifies likely root causes
- Suggests fixes
- Returns a confidence score based on evidence in the logs
One of the key benefits is that the analysis is tied directly to observable data. The system references logs and signals from the bundle to justify its conclusions.
In many cases, this allows support engineers to resolve issues without escalation to the most advanced support tier.
The Bottleneck: Manual Collection
Even with bundles and AI in place, one major source of friction remained: generating the bundle itself.
The existing workflow required:
- A user with cluster access
- Installing a kubectl plugin
- Running the collection command locally
- Uploading the resulting archive
This introduced several issues:
- Some environments restrict installing additional tools
- Windows support was limited
- Large bundles had to be split into multiple uploads
- Multiple roles needed to coordinate
At the moment when speed matters most, the process still depended on manual steps.
The Breakthrough: One-Click Support Bundles
To remove this friction, GitGuardian moved bundle generation into their product.
They built a one-click workflow in their admin UI that:
- Creates a pod inside the customer’s cluster
- Runs the support bundle collection process
- Enriches the bundle with metadata such as a ticket ID
- Uploads the bundle automatically
- Triggers their AI analysis pipeline
This approach removes the need for local tooling and reduces coordination across teams.
It also enables additional controls, such as adjusting the log collection window dynamically. For example, users can limit the bundle to the last hour of logs instead of collecting a full day of data.
Under the hood, the system uses a dedicated pod with two containers:
- One to generate the bundle
- One to serve it for download if needed
This keeps the implementation simple and avoids introducing persistent storage complexity.
The Hard Part: Securing Dynamic Pod Creation
Creating pods dynamically introduces risk.
If not properly controlled, it could allow an attacker to run arbitrary containers, execute commands, and potentially take over the cluster.
To mitigate this, GitGuardian implemented Kubernetes Validating Admission Policies.
These policies enforce strict rules on pod creation:
- Only allowing requests from the GitGuardian application
- Validating the pod specification in detail
- Ensuring only approved images, commands, and configurations are used
This ensures that the application can only create support bundle pods and nothing else.
By separating permissions between the application and the pod, GitGuardian also limits the scope of what each component can do.
The Takeaways
GitGuardian’s support system builds in layers:
- Replicated Support Bundles provide structured data collection
- Custom collectors ensure completeness
- AI handles initial analysis
- A one-click workflow removes manual steps
- Admission policies enforce security constraints
Each layer addresses a specific source of friction, and together they create a support loop that is faster and more consistent.
A few key ideas stand out from this approach:
- Standardizing data collection is critical for reliable support
- AI is effective when applied to structured diagnostic data
- The biggest bottlenecks are often in collection, not analysis
- Automation needs to be paired with strong security controls
- Extending existing systems is often more effective than building from scratch
As Jérémy put it: “Everything that we can automate, we should automate.”
In environments where access is limited and stakes are high, that principle is not just about efficiency. It is what makes effective support possible.
Want to take a similar approach with your application support workflow? Learn more in Building A Support Bundle Workflow That Meets Your Engineers Where They Already Work.