How We Worklet: Certificate Configuration Assistant

Welcome to the sixth installment of our blog series that focused on the power of automated scripts: How We Worklet.

Today, we have the great pleasure of talking shop with Automox Senior Security Engineer, Randall Pipkin.

But before we dive in and discover what Randall’s Certificate Configuration Assistant Worklet is all about, let’s briefly revisit just what a Worklet is and how you can use them with Automox.

Recap: What’s a Worklet and how is it used?

Automox Worklets™️ are super helpful automation tools. Worklets hand over the reins so you can eliminate manual tasks across all your endpoints (regardless of OS or location) and improve your endpoint security posture and compliance. With Automox, there’s no need to spend hours combing through audit spreadsheets or applying fixes manually.

Automox Worklets are script-based units of work that consolidate your tooling and automate work – from configuration, vulnerability remediation, provisioning, and other routine and repetitive tasks across Windows, macOS, and Linux. Worklets save money, reduce cyber risk, and save your IT team members time (and their sanity).

Leverage our catalog of pre-built Worklets or customize them for your unique environment to strengthen endpoint security controls and meet compliance standards without breaking a sweat.

Today’s Winning Worklet: Certificate Configuration Assistant

Today's Winning Worklet is brought to you by Randall Pipkin.

Randall is a Senior Security Engineer with Automox focused on enterprise security and security automation.

He’s spent the last eight years as a security practitioner implementing and operationalizing firewalls, taps, IDS/IPS, VPN, proxy, MFA, RADIUS, AV, EDR, and vulnerability management for large enterprise data centers, hybrid-cloud, and cloud environments.

We sat down with Randall to learn more about his Worklet and the different use cases to which it can be applied. Here’s what he said:

Why did you build the Certificate Configuration Assistant Worklet? What problem does it solve?

In short, the Certificate Configuration Assistant Worklet helps unblock development teams while maturing your security program.

Like many security programs, as ours at Automox grew there came a point at which we really needed a layer of introspection for traffic leaving trusted networks to gain insight into threats. We also needed a method to implement controls around which machines should or should not be able to communicate with on the internet.

Technologies for gaining this visibility and control today span broadly from the suite of tools in a secure access service edge (SASE) to targeted protection provided by secure web gateways (SWG), and on the minimalist side of things can be implemented as forward proxies.

One of the more difficult aspects of deploying any of these technologies is that they typically use self-signed certificate authorities (CAs) for generating certificates, which are necessary for impersonating and intercepting secure TLS communication heading out to the internet. Some applications and many development tools are not configured by default for handling this impersonation for security reasons and will error when checking the network proxy certificate.

There are accessible certificate stores where you may add these CA certificates for many mainstream operating systems; however, not all command-line tools or applications respect the system certificate store.

For this reason, we created the Certificate Configuration Assistant Worklet to help mitigate common impacts on developer systems when dealing with proxies.

What task does this Worklet accomplish?

The Certificate Configuration Assistant Worklet is intended to allow system administrators to deal with two aspects of getting developer workstations functional from behind a proxy that typically requires manual tasks:

1. Configuring a list of common environment variables with the appropriate path pointing to the proxy CA public key

2. Updating a list of java-based application directories containing keystores by loading the proxy CA public key into them

How long did it take to build the Certificate Configuration Assistant Worklet?

Well, we had a few iterations of the script. All accounted for, it took a few hours of dedicated testing and research to determine the environment variables we needed and ways to deal with Java keystores. Then, it took a few more hours of dedicated time writing the Worklet and triaging with pilot groups before rolling it out.

The most challenging aspect of this endeavor was probably determining the various environment variables development tools use, as there does not appear to be great consistency between languages – or even with tools written in the same language.

How does it work?

First, the Worklet needs a set of variables provided. We recommend filling out the variables in the Remediation section first. Then, synchronize those variables up to the Evaluation section after. You’ll need to review and update the prefilled variables to work with your environment.

Next, for the tmp_certs variable, you may either upload certificates as attachments to the Worklet or reference a place on the disk where you’ve distributed them previously. Make sure the values for tmp_certs reflect where the certificates will be at runtime. We have included an example next to the variable that shows the use of Worklet-relative paths in case you want to attach them to the Worklet.

Once the variables are filled, you’re ready to run!

The Worklet references the certificates you provided to create a CA bundle file on the machine at the path you’ve specified. Since the CA bundle is referenced in system-wide environment variables, the file and directory must be located in a place that’s readable by non-privileged accounts.

It then configures the array of environment variables you provided to point to the CA bundle file that now includes both trusted CAs and your proxy CAs.

The macOS version of the worklet also creates a plist file that configures environment variables to be present when users login to their profiles. This has much broader coverage on MacOS than the bashrc and zshrc files.

After it has loaded environment variables, it will search for the presence of defined applications using Java keystores and load the CA bundle into the keystores.

In some cases these keystores use common passwords. You may need to add more dynamic handling for passwords if your environment requires it as this script does not currently account for other passwords.

That’s it!

Following this configuration, users of Windows systems may need to log out and log back in to benefit from the updates. MacOS users need only to quit the affected applications from the task bar and restart them.

The Worklet will also be available in our Automox Security public github for review.

Before you built this Worklet, how much time did it take to do the same task(s)?

Well, we’d have to evaluate which tools were running on every developer’s system. That could be a 20-minute-long engagement to step through broken workflows to identify what was failing. And while we fix that one machine, we’d need to disseminate configurations to the rest of the engineering team.

Of course, there’s no guarantee someone would see or take action on additional settings, so you could spend a similar amount of time troubleshooting the same problem over and over.

Now that you have this Worklet, how much time does the same task take?

Adding new environment variables or keystore paths is very quick, maybe a couple of minutes worth of work. We push out changes to the entire environment as we identify impacted applications, preventing other developers yet to adopt the tool from facing the same challenges.

The real value of this Worklet is that it prevents developers from having to go searching for solutions to known problems.

Is this Worklet device-specific?

The Worklet is provided for macOS, Windows, and Linux. So, depending on your operating system, you can select the version of the Worklet with the appropriate language for your OS. For Windows, the Worklet is written in Powershell. For macOS, the Worklet is written in Bash.

We know they cover common Debian and RHEL-based environments. They may need a little fine-tuning for others.

Which type of IT or security role might especially benefit from using the Certificate Configuration Assistant Worklet?

The Certificate Configuration Assistant Worklet may be useful to system administrators or security engineers managing a secure web gateway for workstations, or proxies delivering egress control on internet-bound traffic in server environments.

Have you been able to measure any quantitative outcomes as a result of implementing this Worklet?

Yes and no.

Because we had so many iterations early on, there was a very gradual decline in certificate-related tickets. In parallel, we were tuning various configurations elsewhere in the environment to better enable business processes. Once a developer workstation was configured appropriately they wouldn’t need further assistance unless they received a new machine or had a major update that caused the global variable configuration to reset.

What we are able to measure as a direct correlation to the impact of the Worklet is a before-and-after ratio of tickets where solutions would have been fixed by the Worklet.

We know that roughly 65% of all tickets before the Worklet were issues relating to these configurations, and in the last four weeks, it only accounted for roughly 10% of tickets.

And finally… just for fun, if this Worklet were a SitCom character, who would it be and why?

If I had to relate it to a SitCom character, I would say it’s somewhat akin to Kramer from Seinfeld. Like Kramer, the Worklet’s a bit quirky and functions in perpetually chaotic environments (development workstations).