The Secret Life of IT Agents: A Conversation with the Automox Agent Team

Summary

In this episode of Product Talk, host Peter Pflaster discusses the role of software agents in IT management with guests Carson Grogg and Tyler Schmidtke from the Automox engineering team. They explore the importance of agents in executing actions on endpoints, the engineering challenges in developing agents for various operating systems, and the rigorous testing and release processes that ensure high-quality software. The conversation also touches on future developments and the excitement surrounding upcoming features.

Transcript

Peter Pflaster: Hello and welcome back to the September edition of Product Talk on the Autonomous IT Podcast Network. I'm your host, Peter Pflaster from the Automox Product and Marketing team. Really excited to have on the show today, a couple of folks from our engineering team. We're going to meet them shortly. Tyler and Carson. Today, we have a really interesting topic.

We're going to be talking about agents. No, not James Bond, unfortunately, but the agents that help tools like Automox and other endpoint management platforms to actually execute actions and manage your endpoints. Many tools in this space use agents from endpoint management like us all the way over to endpoint detection and response. So it's gonna be a really great conversation today, but first let's meet our guests on the show. Let's start Carson with you, welcome. Tell us a little bit about your role here and your experience.

Carson Grogg: Sure. Hi, I'm Carson Grogg. I am a Software Engineer in Test here at Automox. I am the primary point of contact for testing both the backend and the agent.

Peter Pflaster: Awesome. Thank you, Carson. Perspective is going to be really interesting on the talk today. So looking forward to it. Tyler, Tyler is an Automox veteran. Tell us a little bit about what you do at Automox and what you did before.

Tyler Schmidtke: Yeah, so I am a software engineering manager here at Automox. I manage our Agent Team as well as our team that builds our platform connectivity for our agent. I've been at Automox for three years now. I started as a software engineer. If you have seen AVR, I worked on that.

Prior to Automox, I have worked in security, software engineering, IT. I really love being at a company where I get to do so much for our customers in that space. think IT and security is really close to my heart. It's where I started. So it's great to be here.

Peter Pflaster: Awesome. Yeah. Deep expertise and the problems that we're trying to solve for the IT teams that use Automox and the security teams that use it. So really looking forward to this today. think first I want to go ahead and just set the scene of like, for those of you that may be coming in with zero prior knowledge, What is an agent and why does it matter? Right? So I'll take a swing at this and then Tyler, Carson, if you have any additional points at a high level to add, feel free to hop in. But really what an agent is, is it's a piece of software, an executable that runs on a device that you're trying to control, right? So for us, that's macOS systems, it's Windows and Windows Server systems, and it's Linux as well. And really what that allows our platform to do is have a way to execute action, but then also report back into our platform. Same thing for a SaaS product like us, all the way over to a legacy on-prem product with a on-prem console that you manage entirely within your own network. So the agent is really like that agent of action. 

I think that's the market architecture term we would probably set to it, right? That exists on every single endpoint that you want to manage. It's really important for us to have that agent, right? But if you expand that, open the aperture a bit there to all the other IT and security products, right? Many of them, really, really an agent's essential to how they function, right? If you look at an endpoint detection response tool, that agent is absolutely critical

Tyler Schmidtke: Yeah.

Peter Pflaster: to be installed on all the devices that you want to be able to secure. So you can view telemetry, execute scans, et cetera. And it really enables you to have a lot of flexibility as well in how you manage. Many systems have an architecture, like an MDM type architecture, where you can manage them without an agent. But what you're able to do with that sort of an approach is a lot more rigid.

Tyler Schmidtke: yes

Peter Pflaster: than what you can actually do with an agent -based approach. And that's part of the reason why we use an agent. Anything glaring I missed there, guys?

Tyler Schmidtke: No, no. I mean, I think you cover a lot of the basis there. And if you think about MDMs and how they work, it's really like a very tight integration with the operating system. And you can kind of think of it as like the operating system doing a lot of the work that an agent could also do for you in a really particular way. I love that.

Referring to our agent as an agent of action on endpoints. I frequently refer to it as the execution engine for our platform. You schedule policies, you set up automations inside of the Automox console, and our agent is what powers that entire experience. It's how you do things in your environment, how you see your environment. And as you might expect, that necessitates us having a presence on those devices so that we can really get into the nitty gritty and control those devices. So over the years, we've seen a lot of investment in endpoint software for agents. If I look to my security background, there's also a period of time where we did a lot of scanning. I think it's been a big conversions on agent technology because you have so much control and capability that you can power by having software with administrative rights on a device that you're managing.

Peter Pflaster: Awesome. That's a great overview. And I promise I'll, I'll try to reduce my amount talking here. So it's not a, you know, watch the engineering guys laugh at the marketing guy try to explain how an agent works technically. although that probably would be a good podcast that yeah. so I think that, you know, when we talk about agents naturally, right? Like it's, it's a piece of software with.

Carson Grogg: You did a great job.

Tyler Schmidtke: Yeah.

Peter Pflaster: with a higher level of permissions than a lot of other software would have on an endpoint. security becomes pretty important there as does ensuring a really great release process. And we're gonna get into that. But I do wanna talk a little bit about just from an engineering perspective, some of the nuances and challenges associated with building a well-functioning agent for a tool in the IT or security space, right? We manage three different families of operating systems. There are lot of other tools that do similar stuff in the security space too. So I'm curious, just like what kind of considerations do we have to make as we kind of architect and build our agent to manage all of these diverse, pretty different architecture-based operating systems.

Tyler Schmidtke: Yeah, we have, with our agents built, we've got one code base, but we've got a bunch of platform specific stuff that we need to do on each of those operating systems that we support. So we build our agent for every single operating system and we've got a lot of the code that we share, but there's also platform specific stuff that is required because I need to do something on Windows and I need to get really deep into the APIs that Windows provides or, you know, macOS does something in a really particular way. great example of that is Apple Silicon patching and needing to have secure token access set up. And, you know, obviously there's a ton of idiosyncrasies and complexities to managing Linux devices as well. 

So from an engineering standpoint, there's a lot that we need to think about working at a lower level with systems. then there's a ton that we need to think about as well once we've engineered something when it comes to testing. So I don't know if you have anything you'd want to add there from a testing standpoint, Carson. But I think that that's really for delivering great endpoint software. That's one of the biggest components.

Carson Grogg: Yeah, yeah, I agree. One of the big challenges is just how much, how many different environments and different software we encounter on the agent. I think at last count, we support 33 different operating systems and they all have their own challenges. So when we test stuff, we try to go through a large breadth of operating systems, make sure that we didn't, that we account for everything on every OS and make sure we have tests for all of that and find everything we can before it goes out to the customers.

Peter Pflaster: Yeah, that's huge. Cause I think the real magic of what you guys do when you build an agent, right? Is going into that depth at the operating system level and doing what needs to be done to ensure it functions. But then figuring out how to abstract all of that and make the customer experience as seamless as possible and as uniform as possible, despite all these nuances. Once you really peel back kind of the first layer when you're trying to manage a macOS system versus a Windows 10 system. I was going to say Windows Vista, but nobody manages that anymore, hopefully, right? I guess if you're watching this on LinkedIn, you can comment how many Windows Vista systems you're still running in your enterprise. yeah, hopefully, of you have that out of your environments by now.

Tyler Schmidtke: You'd be surprised.

Peter Pflaster: Another really interesting question that we see a lot from customers evaluating us is, what level of permissions does the agent have and how does that compare to other types of solutions in the market, specifically on the security side with endpoint detection and response?

Tyler Schmidtke: Yeah. So the Automox agent is definitely different than what you find with an EDR, endpoint detection and response solution. So we do have administrative capabilities on a device. Obviously, we need that for doing patching. But we operate in user space. So that's a really keen distinction, that we don't have any kernel space components in our agent today, which means that the Automox agent doesn't have the ability to crash a system prior to startup as part of a startup sequence. So it still gives us a ton of capabilities, absolutely. But part of that's also what Automox does and the things that we integrate with in a system don't necessitate us having that really, really deep level of integration in the system. We're not capturing security events in real time on a system. We're collecting software. We're giving administrators the ability to run bash and PowerShell and all sorts of other stuff on their endpoints. But that doesn't necessitate us having that really, really low-level integration with the system with what we do with our agent today.

Peter Pflaster: Yeah, and think that's really important, right? We approach management with the lens of using the least amount of permission possible, which is a good security practice because we don't want to be in a position, especially if we don't need to be, to cause a potential blue screen type incident or cause kernel level problems as well.

So I think given that, that's kind of a good transition actually to move the spotlight over a little bit, I think, into testing and release, right? We've talked a little bit about what agents are, some of the nuances from a build perspective and what level we operate at. But I think another really interesting piece is how we actually bring that stuff into production, right? How we ensure that what we've built is functioning as we expect it to and doesn't cause any unexpected issues. Carson, I'd love to hear from you, just talk us a little bit through at a high level, you know, when you do get a new agent release that we're working through, you what's the approach to understand what's happening and what sort of tests we need to run to ensure, you know, it's working as we expect it to.

Carson Grogg: Yeah. So before it even gets to me, obviously we've gone through our unit testing and integration testing stuff that developers go through before they merge. And then once they merge, an agent build comes to me to test for QE. We have a couple different environments we go through. We've got a dev staging and prod environment. When it's in dev, we run through all of our existing regression tests, which covers a wide breadth of everything the agent does today, and then any new tests that we've written for whatever new feature we're working on. We run through that and over a certain set of OS, and then we go to staging, and we go through a larger set of OS to get more breadth. And then once we've gone through all of that, we have kind of a phased release process. And I think that might be one of your next questions. So I won't get too far into that yet. yeah.

Tyler Schmidtke: And one thing I'll just kind of add to Carson's is a big part of that testing process is automated. But we also do manual testing as well. We've got a big test matrix that we rely on that gives us the confidence that the software that we're putting into production out for our customers is up to the level of quality that our customers should expect from Automox.

Peter Pflaster: Yeah, I think that's really important distinction to make with the approach, right? Where like many other vendors, I would hope pairing kind of the scale and vast breadth and potentially depth of automated testing with the intelligence of someone like Carson or any other QE team in the loop to ensure that those tests are actually running as we expect it. So I think that's awesome. Yeah, Carson, you alluded a little bit to the phase approach. So I'd love to discuss that. We don't just release everything all at once for agents, that's for sure. So why don't you walk us through what that typically looks like, a new agent release, once the tests have all passed and we're ready to start getting this into the hands of the customers.

Carson Grogg: Yep, so the first thing we do is we put it on a few agent team member devices connected to prod, make sure everything's looking good there. After that, we do what's called an internal release where we push the new agent out to all devices for Automox. So we're dogfooding our own stuff and making sure that it works for us before it works for anybody else. After that, we do what's called a beta release. So we have some customers who have opted in to experience our new features for a specific release, and they can get those first and give us feedback. And then once we get the green light on that and everything's looking good, we do a phased release to the rest of prod where we release to a subset of the orgs at a time and we scale up from there depending on what we're seeing.

Tyler Schmidtke: Yeah, usually our release process to production is over the course of a few days. We're monitoring throughout that entire process and not flipping the switch and turning it on for everybody all at once. And that helps us with that monitoring effort. But there's a ton of stuff that we've done before we even get to that point. So a lot of rigorous testing.

It's also something that we're continuously improving, especially this year with some changes that we have coming in the Asia.

Peter Pflaster: Yeah. And the other, the other big thing to note, right? A couple of things. If you're a current customer today and you're interested in becoming a part of the beta program, definitely reach out to the CS team or your CS representative. We can, we can help you get looped in there.

The other thing to note is when we do releases, whether it's the agent or any really any other new feature enhancement to the platform, we have a great docs team here. We have release notes page within the help desk documentation. You can view that just ad hoc, but you can also actually subscribe to that page and get notifications when new stuff drops there. So that's a really important way to find out about new stuff as well, in addition to our customer newsletter that we send out on a monthly basis.

Awesome, guys. Well, we have discussed quite a bit in a very short amount of time. I really appreciate everyone giving your time today to talk a little bit more about the agent and the release process with us. Any closing thoughts from either of you?

Tyler Schmidtke: We've got a lot of cool stuff to come with the agent. So definitely keep an eye on that space. I can't say too much, but a lot of stuff that think everybody internally is incredibly excited about and our customers are going to be very excited about as well.

Peter Pflaster: That's awesome. Foreshadowing. I love it.

Carson Grogg: Yeah, thanks for having me. It was fun. I'm excited for all the stuff the agent's going to bring in the next year and we'll do everything we can to keep quality high.

Peter Pflaster: Yeah, love that. Thank you, Tyler, Carson. Maybe we'll have you guys on in a few months/quarters to talk about all the cool stuff that's dropped. Really looking forward to that. And stay tuned, folks. We got a lot of good stuff coming. Thanks, all.

Tyler Schmidtke: Yeah. Thank you.

Carson Grogg: Thank you.